Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.


Why robots.txt file is important to Website ?

jasminekour12 Offline referral

Posts: 25
Joined: Dec 2015
Reputation: 0

Junior Member
Hello Friends,

I want to know that Why robots.txt file is important to Website ? tell me guys,,
Victor Dub Offline referral

Posts: 507
Joined: Oct 2013
Reputation: 1

Senior Member
It's not too important most of the sites don't need it BUT if you need to prevent some files from being indexed then you have to use it.
chinomoreno Offline referral

Posts: 182
Joined: Jan 2015
Reputation: 1

Robots.txt file is use to tell search engines on how to behave when crawling website pages.
salenaadam Offline referral

Posts: 126
Joined: Dec 2016
Reputation: 0

The content of a robots.txt file consists of so-called "records". both "/support-desk/index.html" and "/support/index.html" as well as all other files in the "support" directory would not be indexed by search engines. If you leave the Disallow line blank, you're telling the search engine that all files may be indexed.

User(s) browsing this thread: 1 Guest(s)