Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.

Thread: 

robots[.]txt is blocking all spiders

Ads
boopika Offline referral

Posts: 6
Joined: Jul 2014
Reputation: 0

#1
Junior Member
what does it meant?
robots[.]txt is blocking all spiders

how can i rectify this error
invisibe_dude Offline referral

Posts: 820
Joined: May 2013
Reputation: 24

#2
Posting Freak
This file is used by most major search engines. Spam-bots and other bogus ones will not be affected sorry.
marcus_avrelius Offline referral

Posts: 2,424
Joined: May 2013
Reputation: 102

#3
Support Team
Where do you get this message?

Don't use any auto generators do it manually!!!!

Make sure you don't have similar code in your file:

Code:
User-agent: Googlebot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
KevinMax Offline referral

Posts: 573
Joined: Aug 2013
Reputation: 78

#4
Senior Member
Marcus is right! Check your robots file first, Understand why it is intended for... I am sure you file is blocking all the web spiders to crawl.
Check also this! https://support.google.com/webmasters/an...ic=6061961
SimranKaur Offline referral

Posts: 7
Joined: Jun 2014
Reputation: 0

#5
Junior Member
(07-17-2014 03:24 PM)boopika Wrote:  what does it meant?
robots[.]txt is blocking all spiders

how can i rectify this error

Have you used right codes in Robots txt file?

Do check your file, might be there is some mistake on your file.
boopika Offline referral

Posts: 6
Joined: Jul 2014
Reputation: 0

#6
Junior Member
I got this message from Bing webmaster, yes of course I'm not using any tools.
marcus_avrelius Offline referral

Posts: 2,424
Joined: May 2013
Reputation: 102

#7
Support Team
Use Fetch as Bingbot: http://www.bing.com/webmaster/help/fetch...t-fe18fa0d to check if you are really blocking Bing crawlers.

If Google says nothing then it's probably a problem with Bing cause Google will be the first to warn you about that!!





User(s) browsing this thread: 1 Guest(s)