Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.

Thread: 

Importance of robot.txt

Ads
Alex Thompson Offline referral

Posts: 101
Joined: Aug 2013
Reputation: 9

#1
Member
Importance of robot.txt
KevinMax Offline referral

Posts: 573
Joined: Aug 2013
Reputation: 78

#2
Senior Member
Mainly In-tented for security to a website.

For Ex : Army people's also has some sites , But that will not be listed in the search engines. So a coder code the robot.txt file in a confidential way so that , it will not visible in any of the search engines.

Now It was used by website developer's & Seo people who want to block or give space for a search engine to crawl their site.

A website may have robot.txt file (It is not a compulsory one)
maya Offline referral

Posts: 1,058
Joined: May 2013
Reputation: 34

#3
Posting Freak
Use robot.txt whenever you want to block a URL from being indexed by search engine
chod Offline referral

Posts: 860
Joined: Jul 2013
Reputation: 85

#4
Posting Freak
(08-30-2013 08:11 AM)KevinMax Wrote:  Mainly In-tented for security to a website.

For Ex : Army people's also has some sites , But that will not be listed in the search engines. So a coder code the robot.txt file in a confidential way so that , it will not visible in any of the search engines.

This is wrong!
Robots adds no security to a site! Robots has nothing to do with security! robots.txt is visible by anyone and especially all bots. That would be like saying your Sitemap helps protect your site in a more secure manner Tongue

I really hope this is not the level of consulting you offer @KevinMax!!!!!!! Please learn why your example for "Army people" is really a bad one. Mainly because that is not how they prevent getting into SERPs.

Quote:Now It was used by website developer's & Seo people who want to block or give space for a search engine to crawl their site.

A website may have robot.txt file (It is not a compulsory one)

A robot.txt like maya said will prevent a good search bot from indexing files or tells it that its OK to index files or which files are OK. It will not stop scrapers, bad bots and similar. If you have a robots or not there is nothing that make you more secure.

Taken from wikipedia to help those grasp what this file actually does and most importantly what it does not do.

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
www.AdminEmpire.com
www.BlimptonTech.com - Best Free Online JavaScript tool to minify JavaScript files.
marcus_avrelius Offline referral

Posts: 2,424
Joined: May 2013
Reputation: 102

#5
Support Team
Once you submit your site to search engine, it's crawler doesn't know which URL you don't want to be indexed. When crawler arrives to your site first it checks if there it a robot.txt file and only then it begins indexing your site!
chod Offline referral

Posts: 860
Joined: Jul 2013
Reputation: 85

#6
Posting Freak
Search engine crawlers will crawl your site regardless of having a robots.txt or not. It has nothing to do with that. It is only for GOOD bots/crawlers like googlebot. Most crawlers other than major search engine ones do not even check for a robots.txt file.
www.AdminEmpire.com
www.BlimptonTech.com - Best Free Online JavaScript tool to minify JavaScript files.





User(s) browsing this thread: 1 Guest(s)