Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.

Thread: 

What is use of robot.txt in SEO ?

Ads
banner
banner
Shivam Tripathi Offline referral

Posts: 102
Joined: Sep 2017
Reputation: 0

#1
Member
Hello Friends,

Actually i want to know litle bit information about robot.txt in seo.so,please share with me.
HokuRaj Offline referral

Posts: 54
Joined: Dec 2017
Reputation: 0

#2
Member
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.
davidweb09 Offline referral

Posts: 314
Joined: Sep 2016
Reputation: 0

#3
Senior Member
Robots.txt file help to control our website indexing.
SaraWilliams Offline referral

Posts: 76
Joined: Oct 2017
Reputation: 0

#4
Member
The site's owner had included a disallow to “/”. They were telling the search engine robots not to crawl any part of the website. In another case, a SEO company edited the robots.txt file to disallow indexing of all parts of a website after the site's owner stopped paying the SEO company.
etdigital Offline referral

Posts: 40
Joined: Jun 2017
Reputation: 0

#5
Junior Member
Robot.txt file uses to restrict the search engines for crawling specific file or folder or webpage.
robot.txt file is for the search engine not for users.
to know more: moz.com/learn/seo/robotstxt
drnagwaniseo Offline referral

Posts: 101
Joined: Oct 2017
Reputation: 0

#6
Member
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
Hair transplant less than half cost | Doctor for hair transplant in West Bengal | Best Surgeon for hair transplant in Kolkata | Best and cheapest center for hair transplant in Kolkata | cheapest cost for hair transplant in Kolkata
worldimmigration21 Offline referral

Posts: 186
Joined: Dec 2017
Reputation: 0

#7
Member
robot txt fill help you regarding which URL want to index and which one URL not to want to index
iprism Offline referral

Posts: 111
Joined: Aug 2017
Reputation: 0

#8
Member
When search engine crawlers come to your site it will be looking for some special file. i.e, call robots.txt file. This file tells to search engine spiders which pages of your site should be index and which pages of your site should be ignored.
berto Offline referral

Posts: 185
Joined: Jul 2017
Reputation: 0

#9
Member
One of the best example of using robot.txt file is the log in module of the website in which not needed to be crawled and indexed by the search engine.
AaliyaAnubhav Offline referral

Posts: 46
Joined: Jul 2017
Reputation: 0

#10
Junior Member
The robots.txt is used to instruct robots (like Googlebot) what pages to index and which pages to ignore, on your website.
Rachana Offline referral

Posts: 26
Joined: Nov 2017
Reputation: 0

#11
Junior Member
Robots.txt is a text file created by webmasters that instruct web robots how to crawl pages in their website. This file also indicates whether the pages of their website should be crawled or not.
alicemark15 Offline referral

Posts: 296
Joined: Nov 2015
Reputation: 0

#12
Senior Member
Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.
astrologerhome Offline referral

Posts: 67
Joined: Apr 2017
Reputation: 0

#13
Member
Robot website is used for crawel the website
tgdevelopers Offline referral

Posts: 53
Joined: Nov 2017
Reputation: 0

#14
Member
the main use of robots.txt is to allow or dis allow google crawllers
carla172 Offline referral

Posts: 26
Joined: Nov 2017
Reputation: 0

#15
Junior Member
We need to hide some pages from Search Engine like login page, dashboards etc.
So put all such pages in Robots.txt

Alternatively you can put meta tag of NOFOLLOW NOINDEX in header of that page.
Estherjuly Offline referral

Posts: 149
Joined: Jul 2017
Reputation: 0

#16
Member
Robots.txt is the text file to inform search engine which is not index or access from our website.

you can control your website which is needed to access which need not access and index from your website.
sankalppatil Offline referral

Posts: 232
Joined: Jan 2018
Reputation: 0

#17
Member
Robot.txt is the text file you put on your site to tell search robots which pages you would like them not to visit. The robot.txt file is also known as the robot exclusion protocol. The robot.txt is main part of the text file.
amitkhare Offline referral

Posts: 99
Joined: Jul 2017
Reputation: 0

#18
Member
Robot.txt is generally use for block the duplicate URL
davidsmith0143 Offline referral

Posts: 217
Joined: Aug 2017
Reputation: 0

#19
Member
robot.txt is used to execute the robots exclusion protocol. means site owners use the robot.txt to command the site web robots.
amitkhare Offline referral

Posts: 99
Joined: Jul 2017
Reputation: 0

#20
Member
(02-07-2018 09:08 AM)AaliyaAnubhav Wrote:  The robots.txt is used to instruct robots (like Googlebot) what pages to index and which pages to ignore, on your website.

Exactly.You have explained very well in simple way.Really,Today i came to know about robot.txt...thanks for sharing





User(s) browsing this thread: 1 Guest(s)