Google Product Forums

Re: Site not ranked, robots.txt 500 error


Becky Sharpe Mar 21, 2012 12:06 PM
Posted in group: Webmaster Central

Categories: Crawling, indexing & ranking :

I would delete the robots.txt as it isn't blocking anything and see if that helps. Really, I think you need to speak to your hosting service to establish if it is blocking Google crawling. Go to Webmaster Tools and diagnostics and test the Fetch as Googlebot tool -  http://support.google.com/webmasters/bin/answer.py?hl=en&answer=158587

The site certainly isn't indexed so there must be something technical preventing that happening.