Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.


Why robots.txt file is important to Website ?

jasminekour12 Offline referral

Posts: 32
Joined: Dec 2015
Reputation: 0

Junior Member
Hello Friends,

I want to know that Why robots.txt file is important to Website ? tell me guys,,
Victor Dub Offline referral

Posts: 552
Joined: Oct 2013
Reputation: 3

Senior Member
It's not too important most of the sites don't need it BUT if you need to prevent some files from being indexed then you have to use it.
chinomoreno Offline referral

Posts: 182
Joined: Jan 2015
Reputation: 1

Robots.txt file is use to tell search engines on how to behave when crawling website pages.
salenaadam Offline referral

Posts: 211
Joined: Dec 2016
Reputation: 0

The content of a robots.txt file consists of so-called "records". both "/support-desk/index.html" and "/support/index.html" as well as all other files in the "support" directory would not be indexed by search engines. If you leave the Disallow line blank, you're telling the search engine that all files may be indexed.
Brandfuge Offline referral

Posts: 18
Joined: Jan 2017
Reputation: 0

Junior Member
Robots.txt file is what tells the search engine which pages to access and index on your website on which pages not to.

If you want to hide some of the web pages from the Google crawler you can this text file.
Pick your Perfect Web hosting! Hosting reviews

User(s) browsing this thread: 1 Guest(s)