Google Product Forums

Re: robots.txt


atensoftware.com May 11, 2011 6:13 AM
Posted in group: Google Merchant Center

Categories: Troubleshooting :

Regarding the images being blocked for Yahoo Stores, this is no longer an issue.  After what I imagine was a fuss raised by Yahoo store clients, Yahoo removed the line in their robots.txt that was preventing Google from indexing the images.  It looks like Yahoo caused this problem when they (inadvertently?) added the exclusions to their robots.txt file.

Note that Yahoo Store still creates new image links on every store publish, but this doesn't have any bearing on whether the images can be indexed or not.

Evogear: On (2), Google recommends, and I think you should follow, the practice of making your robots.txt as inclusive as possible.  Let Google index everything possible and then let their algorithm decide what's important and how much to index.  After all, they are changing their algorithms all the time, and you do not want to box yourself in based on presupposed assumptions about their 'limits'.

On (3), I don't believe a few minutes of downtime is not going to bother Google's crawlers.  You might get a handful of crawl errors, but that's about it.  I would worry more about the impression left on your customers of your site being down, even if it is just a minute.

Regarding the customers commenting about traffic declines, note that Google is constantly changing their algorithms, and they have been adding requirements such as unique identifiers.  In any case, merchants should expect ups and downs from this free advertising medium.