| GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | WebTravelLog | 4/13/10 6:36 AM | I have read the FAQs and checked for similar issues: YES My site's URL is: *.http://traveljournal.net & *.http://webtravellog.com Description: Google bot access /sitemap.xml for an excessive amount of times: +--------------+-------------------+---------------- |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | luzie | 4/13/10 7:26 AM | Hi WebTravelLog, hmm, looks weird ... ^^ 1. Are you sure it's really Googlebot who's doing this? (Do a reverse IP lookup to find out) 2. When you say, "sitemaps are dynamically generated" - how often does this happen? Do you (or does your software |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | WebTravelLog | 4/13/10 8:07 AM | The request where from (543 IPs in total): 66.249.65.* 66.249.66.* 66.249.67.* 66.249.68.* 66.249.71.* those are all within the Google netblock: 66.249.64.0 - 66.249.95.255 The sitemaps are generated on request; all information (including timestamps |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | luzie | 4/13/10 8:20 AM | Ok, I think we have all necessary information now to ask a Googler what's wrong here. -luzie- |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | JohnMu | 4/13/10 9:16 AM | Hi WebTravelLog Thanks for posting -- I'll check with the team and get back to you as soon as I know more :-) Cheers John |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | WebTravelLog | 4/19/10 11:07 PM | bump The problem persists. |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | JohnMu | 4/20/10 4:53 AM | Hi WebTravelLog Google's Sitemaps crawler usually reacts to the update frequency of your Sitemap files. If we find new content there every time we crawl, we may choose to crawl more frequently. If you can limit updates of the Sitemap files to daily ( |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | WebTravelLog | 4/20/10 9:53 AM | No, this does not help. The sitemap is advertised in the robots.txt, and it is not a problem that the sitemap is fetched separately for each subdomain. My _site_ is not crawled to frequently, only our sitemap is. The problem is that /sitemap.xml (l |
| Re: GoogleBot request /sitemap.xml up to 14,800 times a day, across 375 subdomains | ColinMcDermott | 4/21/10 9:24 AM | That does seem very excessive if true... http://www.webtravellog.com/Sitemap.xml The last mod date is the same on each file in the sitemap, and - if I am correct is displaying ... now? So everytime Google visits the sitemap, it is told that every |