Categories: Webmaster Tools :

Sitemaps are missing. Help, please

Showing 1-9 of 9 messages
Sitemaps are missing. Help, please justsandy 12/25/13 12:38 PM
I've read the FAQs and searched the help center. Yes
I submitted 2 sitemaps years ago, one is a video one, too. They were both fine for 6 years. 
Webmaster tools now tells me that I have not submitted any sitemaps. On Dec. 7, I got message that Google couldn't crawl my robots.txt file. Now it is being crawled. I made it years ago and this is what it says:

User-agent: *
Disallow: /cgi-bin/


So is this why my sitemap is not showing or being indexed? I don't recall putting the sitemap url in the robots.txt file but it's there. Also, the video sitemap isn't referred to in the robots.txt file, yet it is not showing up in the Sitemaps listing on Webmaster tools. Any help is appreciated. God bless. Sandy

Re: Sitemaps are missing. Help, please Eric.Kuan 1/2/14 6:39 PM
Hi justsandy,

Google not being able to crawl your robots.txt file is a separate issue from your Sitemaps in Webmaster Tools. If your Sitemap is missing in Webmaster Tools, you can simply re-add them in your Webmaster Tools account.

If Google is having trouble crawling your robots.txt file, it will stop crawling the rest of your site to prevent it from crawling pages that have been blocked by the robots.txt file. If this isn't happening frequently, then it's probably a one off issue you won't need to worry about. If it's happening frequently or if you're worried, you should consider contacting your hosting or service provider to see if they encountered any issues on the date that you saw the crawl error.

You can also find some helpful information here: ttps://support.google.com/webmasters/answer/2409682.

Eric
Re: Sitemaps are missing. Help, please Sean Van Guilder 1/3/14 9:12 AM
Can you clarify this statement: If Google is having trouble crawling your robots.txt file, it will stop crawling the rest of your site to prevent it from crawling pages that have been blocked by the robots.txt file.

If it's having trouble crawling the robots.txt how would it know what directives are in robots.txt?
Re: Sitemaps are missing. Help, please Grace Massa Langlois 1/3/14 9:36 AM
Good Afternoon Sean,

If Google can't crawl the robots.txt they won't know if you're blocking any urls therefore to be safe they stop crawling your site altogether just in case you have blocked any urls.

Grace

Re: Sitemaps are missing. Help, please Gaieus 1/3/14 9:36 AM
That1s obviously the point. If there is a problem, it's better to stay on the safer side and not crawl anything than crawl something that should not be crawled.
Re: Sitemaps are missing. Help, please Sean Van Guilder 1/3/14 1:23 PM
Sorry to sound obtuse about this, but it actually makes no sense as I have a client (new client at that) and their site has never had a robots.txt but have a sitemap.xml file submitted via WMT and Google does return results for their entire site. This is not the only time I've seen entire sites indexed just fine without a robots.txt and am quite leery of the statement that Google will stop indexing as I have not seen that be the case, but rather Google will over-index because there are no directives stating what not to index.
Re: Sitemaps are missing. Help, please Gaieus 1/3/14 1:56 PM
All right, there seems to be a misunderstanding here. What Eric wrote was "If Google is having trouble crawling your robots.txt file, it will stop crawling the rest of your site..." which means that no 404 (non-existent at all), no 200 (responses and downloads just fine) but some different problem (not loading, forbidden, server error whatnot).
Re: Sitemaps are missing. Help, please justsandy 1/4/14 1:58 PM
Thanks for your answer Eric, but Google has been able to crawl it again cuz it doesn't show up as a problem in WMT. However, I wanted to know if the robots.txt I have is coded wrong and that since my sitemap's url is listed, that's why the sitemap is missing. Here's what it says and again, I don't remember adding the sitemap url but I must have or how did it get there? Or is it correct since the cgi-bin page of every page listed on the sitemap is not to be crawled and that's why it's listed as such. Can you make it clear to me? Thanks.

User-agent: *
Disallow: /cgi-bin/

Re: Sitemaps are missing. Help, please Eric.Kuan 1/7/14 5:43 PM
Hi justsandy,

You can read more about the robots.txt specification on the official Web Robots page: http://www.robotstxt.org/.

From the robots.txt file you pasted, it looks like you're disallowing all Web Crawlers from crawling the /cgi-bin/ folder on your site. 

The Sitemap line in your robots.txt file tells Google or other search engines that your Sitemap is at the specified location. More information about options for Sitemap submission can be found here: https://support.google.com/webmasters/answer/183669?hl=en