Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.

What is use of robot.txt in SEO ?

make money online
Christi Smith Offline referral

Posts: 34
Joined: Apr 2017
Reputation: 0

Junior Member
(01-30-2018 07:39 AM)Shivam Tripathi Wrote:  Hello Friends,

Actually i want to know litle bit information about robot.txt in SEO.so,please share with me.

Robots.txt file is used to tell Google, which pages of the website need to be crawled or which file need to blocked from crawling. It is also recommended to mention Sitemap file in it.
Memorials4u - Cremation Urn | Adult Urns
Shilpa111 Offline referral


Robot.txt file is controlling our website in the search engines. Example if we will disable Robot.txt file means our website is not showing under search engines. By using this we control the indexing of our website in the search result.
fayeseom Offline referral

Posts: 142
Joined: Feb 2016
Reputation: 1

The robots.txt is used to instruct robots (like Googlebot) what pages to index and which pages to ignore, on your website.
moazzem28 Offline referral

Posts: 38
Joined: May 2018
Reputation: 1

Junior Member
When you want to block search engine to crawl and index your webpage, you need the robots.txt file.
allahbarkatlocket Offline referral

Posts: 52
Joined: Jun 2018
Reputation: 0

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.
keshacarey Offline referral

Posts: 25
Joined: Aug 2018
Reputation: 0

Junior Member
It is used to control the permissions of each pages and directories. By this we can tell search engines what are allowed to be shown in search results and what are not.
jacksmm Offline referral

Posts: 248
Joined: Jul 2018
Reputation: 0

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site.
Saravanan28 Offline referral

Posts: 131
Joined: Nov 2018
Reputation: 1

robots.txt file used to tell search engine crawlers to access which part of the website or if someone doesn’t want some part of the website to appear in search results.
medical Offline referral

Posts: 70
Joined: Sep 2018
Reputation: 0

Robots.txt is text files mainly tell to crawler that it is permitted any website directory. The use of robots.txt is to disallow crawlers from visit private folder or content that gives them no extra information.
Jennifer5 Offline referral

Posts: 35
Joined: Aug 2018
Reputation: 0

Junior Member
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
Unisecure Data Centers Since 1996
Cloud Hosting | Dedicated Server Hosting | VPS Hosting Servers
Ametuniversity Offline referral

Posts: 384
Joined: Feb 2019
Reputation: 0

Senior Member
A robots.txt is a text file that resides in the root directory of your website and gives search engines crawlers instructions as to which pages they can crawl and index, during the crawling and indexing process.
Naman Modi Offline referral

Posts: 39
Joined: Feb 2019
Reputation: 0

Junior Member
Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website.
Shoppingswag Offline referral

Posts: 59
Joined: Dec 2018
Reputation: 0

Robots.txt files are useful:

If you want search engines to ignore any duplicate pages on your website
If you don’t want search engines to index your internal search results pages
If you don’t want search engines to index certain areas of your website or a whole website
If you don’t want search engines to index certain files on your website (images, PDFs, etc.)
If you want to tell search engines where your sitemap is located
sinelogixtech Offline referral

Posts: 312
Joined: Mar 2017
Reputation: 1

Senior Member
Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.

Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.
andy123 Offline referral

Posts: 214
Joined: Jan 2019
Reputation: 1

Robots.txt file is use to instruct search engines which pages to crawl in the website & which not to crawl.
mayoclinic Offline referral

Posts: 117
Joined: Oct 2018
Reputation: 0

include or dis include urls in website , you can take help of robots.txt

Google this topic

User(s) browsing this thread: 1 Guest(s)