Categories: Mobile websites :

Smartphone Crawl Errors in GWT - Blocked URLs

Showing 1-6 of 6 messages
Smartphone Crawl Errors in GWT - Blocked URLs netmeg3 12/4/13 12:31 PM
Ok, I'm referencing this blog post, released today:

Wherein it says and I quote:

Blocked URLs: A blocked URL is when the site's robots.txt explicitly disallows crawling by Googlebot for smartphones. Typically, such smartphone-specific robots.txt disallow directives are erroneous. You should investigate your server configuration if you see blocked URLs reported in Webmaster Tools.

Now, for one particular Magento site, I am showing 3200+ blocked URLs under the smartphone tab, and I do not have a separate robots.txt for the mobile site (it's a separate CSS template served on the same URLs), nor do I have any separate directives for smartphones in my robots.txt, and these blocked URL "errors" are URLs I have deliberately blocked (for all devices) in one single robots.txt for the entire site.

So I'm pretty sure they're not errors, they're not only being blocked for smartphones, and there's no point in marking them fixed, because they'll just come back.

So why do these show up as errors?

(Side note - IMHO every single Google error message should include a dismissal button that says "I meant to do that")

Re: Smartphone Crawl Errors in GWT - Blocked URLs nsapp 12/10/13 10:27 AM
I'm not offering a solution, but just giving an extended "me, too!" to your post.  I'm transitioning from one shopping cart solution (/shop/) to another (/cms/).  The new solution is responsive and there is nothing smartphone-specific there.  I disallowed the "shop" directory in my robots.txt file for obvious reasons - I don't want it crawled anymore - by desktops or smartphones.  In the near future, I'll delete that directory.  So I get smartphone errors for that path that I have disallowed.  I'm not getting errors on the non-smartphone tab, so I really don't know how to fix this.  Sometimes, I think it's OK to block URLs because we know what we are doing.  Hopefully, someone will help us with an answer.  I do like you dismissal button idea, however.
Re: Smartphone Crawl Errors in GWT - Blocked URLs JohnMu 12/13/13 2:58 AM
Hi Meg

Thanks for the feedback! TL;DR: It'll probably take a bit of time for things to stabilize in that regard. 
We have seen many sites try to use robots.txt to block our mobile crawlers from their mobile site (for various reasons). When they do that, this essentially makes it impossible for us to recognize that the mobile site exists at all, and that we should associate it to the desktop site. The goal of this section is to bubble that up. That said, it can take a bit of time for the data to settle down here - we don't want to show when a URL is just generally blocked by robots.txt, it should really just be for those cases when a URL is only blocked on the smartphone side. For that, it'll take a bit of time for the new data to be collected and shown there. 

Re: Smartphone Crawl Errors in GWT - Blocked URLs silverbling 12/14/13 1:30 AM

I have the opposite problem, I have 65 smart phone errors, I redirected them to the desktop page which did not work, I then removed the redirects and set up a new robots text to allow all crawling and the errors are still there.

I have an ajax site with .php pages and html snapshots added. The smart phone errors are .html not .php. I have spent 2 days working on this but now have no idea what to do next to get rid of the errors
Re: Smartphone Crawl Errors in GWT - Blocked URLs focusgroups 1/31/14 3:15 PM
me too

This has been occurring in both my websites and my blogger sites. Both 404 and 503 errors reappear ONLY in smartphone errors every day.

Re: Smartphone Crawl Errors in GWT - Blocked URLs ets 2/1/14 4:02 AM
focusgroups ==> Please open a new question (thread) with all the details of your site and your problem and we'll take a look. It helps everyone to keep each site in its own thread (or the questions and answers become very confused). Thanks :)