Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.

Thread: 

Why robots.txt file is important to Website ?

Ads
banner
banner
jasminekour12 Offline referral

Posts: 55
Joined: Dec 2015
Reputation: 0

#1
Member
Hello Friends,


I want to know that Why robots.txt file is important to Website ? tell me guys,,
Victor Dub Offline referral

Posts: 804
Joined: Oct 2013
Reputation: 3

#2
Posting Freak
It's not too important most of the sites don't need it BUT if you need to prevent some files from being indexed then you have to use it.
chinomoreno Offline referral

Posts: 332
Joined: Jan 2015
Reputation: 1

#3
Senior Member
Robots.txt file is use to tell search engines on how to behave when crawling website pages.
salenaadam Offline referral

Posts: 211
Joined: Dec 2016
Reputation: 0

#4
Member
The content of a robots.txt file consists of so-called "records". both "/support-desk/index.html" and "/support/index.html" as well as all other files in the "support" directory would not be indexed by search engines. If you leave the Disallow line blank, you're telling the search engine that all files may be indexed.
Brandfuge Offline referral

Posts: 18
Joined: Jan 2017
Reputation: 0

#5
Junior Member
Robots.txt file is what tells the search engine which pages to access and index on your website on which pages not to.

If you want to hide some of the web pages from the Google crawler you can this text file.
Pick your Perfect Web hosting! Hosting reviews
damponting44 Offline referral

Posts: 277
Joined: Oct 2016
Reputation: 0

#6
Senior Member
A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.
herlaiser Offline referral

Posts: 20
Joined: Mar 2017
Reputation: 0

#7
Junior Member
It's made to keep some of your pages to be indexed by search engines robots
admissioninfo Offline referral

Posts: 50
Joined: Feb 2017
Reputation: 0

#8
Member
A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.





User(s) browsing this thread: 1 Guest(s)