Google Product Forums

Re: Google Webmaster Tools Detects Different Robots.txt

cristina May 5, 2012 9:55 AM
Posted in group: Webmaster Central

Categories: Crawling, indexing & ranking :

Do you see the robots.txt with 
Disallow: / 
in Google Webmaster Tools - Site configuration - Crawler access ?
Can you see the date when Googlebot last downloaded the robots.txt file?
It might have been an older version, still cached by Googlebot, if the downloaded date is sometime ago.
Did you look at robots.txt with Fetch as Googlebot? Click the "Success" link to see the content seen by Googlebot.
If all is OK with the robots.txt file as seen by Fetch as Googlebot, try to submit your home page to index with Fetch as Googlebot to indicate to Google that you need a recrawl.