Categories: Webmaster Tools :

Serious bug in the "remove urls" tool

Showing 1-42 of 42 messages
Serious bug in the "remove urls" tool mvandemar 2/4/14 10:20 PM
I've read the FAQs and searched the help center. 
My URL is:

It appears as if the url removal tool is ignoring url parameters. I have a client who was hacked, and the serps are filled with spam. They asked for my help in cleaning out the serps. I was adding the urls to the url removal tool, when I got to one and I noticed that it suddenly thought I was wanting to remove the entire site:




It's obvious that the tool is ignoring everything from the question mark on, even though those parameters were what was causing the spam to show. Why on earth would the Google Webmaster Tool developers think that url parameters were irrelevant, and how do we get this fixed? And yes, I know that most people who help on these forums won't be able to assist with this kind of core level bug, but I am hoping that one of the Googlers might take notice on it. In the meantime, does anyone know if there is a workaround for it?

Thanks.

-Michael
Re: Serious bug in the "remove urls" tool Bilal Sarwari 2/4/14 10:33 PM
Open this page on your browsers and verify its opening or redirecting to your homepage
Re: Serious bug in the "remove urls" tool mvandemar 2/4/14 10:51 PM
@Bilal - it is not redirecting to the homepage, I assure you that is not the issue.

I just tested with another site and g
ot the exact same results. It assumed that asking to remove www.beginnerwriting.com?thisparameter=isafake was a request to remove the entire site from the serps.

-Michael
Re: Serious bug in the "remove urls" tool Bilal Sarwari 2/4/14 10:59 PM
Its strange, I hope anyone from Google will look at it in detail, 

If the pages are removed from site then take it easy, you can also check your internal links, if any link going to .co.uk/?p=bailey-bow-ugg.. .. ..   then remove it.
Re: Serious bug in the "remove urls" tool DonnaFontenot 2/6/14 5:48 AM
Whoa, this seems like a pretty important issue. Dangerous bug. 
Re: Serious bug in the "remove urls" tool Grace Massa Langlois 2/6/14 10:37 AM
Good Morning,

Have you cleared the malicious code, can you provide a url?

Grace
Re: Serious bug in the "remove urls" tool Grace Massa Langlois 2/6/14 10:56 AM
Good Afternoon,

Please disregard my last message, I was clearly confused.

I'm going to escalate the thread, I'll get back to you as soon as I have more information.

Grace
Re: Serious bug in the "remove urls" tool Lysis 2/6/14 11:05 AM
need url.

A thought: sometimes hacks do cloaking redirects when the useragent is googlebot, so possibly that URL is pointing to the home page.
Re: Serious bug in the "remove urls" tool mvandemar 2/6/14 11:07 AM
@Grace - Thank you, you rock!
Re: Serious bug in the "remove urls" tool mvandemar 2/6/14 11:11 AM
@Lysis - no, it's not that at all. If you notice in my second post I tested with a completely unrelated, flat html, non-hacked site and got the same results. Thanks though.

-Michael

Re: Serious bug in the "remove urls" tool Suzanneh 2/6/14 11:20 AM
Are you telling Google to ignore that parameter in WMT?  Crawl -> URL Parameters

Suzanne
Re: Serious bug in the "remove urls" tool mvandemar 2/6/14 11:29 AM
@Suzanneh - no, and again, this was tested on more than one site, using random parameters. Can you do me a favor though? Can you test it? Try tacking on any ?someparameter=anything to the root domain on any site in your account, and asking Google to remove that url from the serps, and see if they flag it as a request to remove the entire site. Don't worry, the request is not instantaneous, there is a Pending period and it is very easy to cancel the request afterwards.

-Michael
Re: Serious bug in the "remove urls" tool Ben Griffiths 2/6/14 11:30 AM
I can replicate this, FYI.


Re: Serious bug in the "remove urls" tool Suzanneh 2/6/14 12:44 PM
ROTFL!  Um, yeah, no.

If there is a bug, I wouldn't test it with a ten foot pole. ;-)

Suzanne


On Thursday, February 6, 2014 2:29:27 PM UTC-5, mvandemar wrote:
@Suzanneh - no, and again, this was tested on more than one site, using random parameters. Can you do me a favor though? Can you test it? Try tacking on any ?someparameter=anything to the root domain on any site in your account, and asking Google to remove that url from the serps, and see if they flag it as a request to remove the entire site. Don't worry, the request is not instantaneous, there is a Pending period and it is very easy to cancel the request afterwards.

-Michael
Re: Serious bug in the "remove urls" tool JohnMu 2/6/14 11:47 PM
Thanks for reporting this, Michael (and thanks for escalating, Grace!). I'll check with the engineering team to see what's happening here. It's great to see you help clean up this kind of hacked content!

Cheers
John
Re: Serious bug in the "remove urls" tool RyanJones 2/7/14 8:59 AM
Is it perhaps the case that the removal tool works at the page level, and does not factor in parameters? That would make perfect sense to me.  If the removal worked at the parameter level, then a page with 200 "sort by" parameters would need 200 different removal lines.

long term advice: I would strongly advise you take a different approach to your site architecture.  
Re: Serious bug in the "remove urls" tool Ben Griffiths 2/7/14 9:03 AM
I don't think this particular architecture was entirely planned for ;)
Re: Serious bug in the "remove urls" tool mvandemar 2/7/14 9:47 AM
Thanks John! Appreciate it.

Btw, you should check in on Twitter more often. :)

-Michael

Re: Serious bug in the "remove urls" tool mvandemar 2/7/14 9:59 AM
Is it perhaps the case that the removal tool works at the page level, and does not factor in parameters?

Considering that is what this entire thread is about, I would say yes, that is in fact the case.

long term advice: I would strongly advise you take a different approach to your site architecture.

@Ryan - I think you should learn to read more carefully before replying. These url parameters have nothing whatsoever to do with the site's architecture. They were appended via a hacked injection, and worked in conjunction with the injected code to display spammy text. They have nothing to do with the site's normal url structure.

You're other assertion that it needs to ignore the parameters in order to function is simply a lack of understanding of urls on the web or actual web development. It would be a relatively simple job for the GWT team to utilize wildcard matching in the url removal tool, which is the way I would expect developers of the caliber that Google hires to design things. Php content powered by url parameters way predates "pretty" urls, and for the team to simply ignore their existence is a bit sloppy.

-Michael
Re: Serious bug in the "remove urls" tool webado 2/8/14 11:07 AM
Sorry if I'm asking the obvious or what has already been asked and answered.

Is it possible the drop down list offered several types of removal only one of which is for directory removal (which would result in a full site removal if it's off of the root)?

Maybe you missed picking up the url removal option and accidentally picked directory removal.
Re: Serious bug in the "remove urls" tool webado 2/8/14 11:11 AM
Sorry, I see the tool, when used from within one's own Webmaster Tools account as opposed to the more public url removal tool (https://www.google.com/webmasters/tools/removals) does not offer any drop-down list of options for removal.





Re: Serious bug in the "remove urls" tool Ben Griffiths 2/8/14 11:13 AM
Yeah, it jumps straight to a Site Removal without displaying the options dialog.
Re: Serious bug in the "remove urls" tool mvandemar 2/8/14 11:23 AM
@webado - did you test it by the way, to see if you had the same issue?

Also, thanks for the link to that other version of the tool, I was unaware of that one. Apparently back in 2009 someone tried to remove a bunch of pages on my blog from the serps, it would be great if they showed the ip address of all of those denied requests.

-Michael
Re: Serious bug in the "remove urls" tool webado 2/8/14 11:31 AM
I didn't go all the way, I only noticed there were no options offered. Then I cancelled.

But I will make a full test. Just need to pick a site where it wouldn't hurt too much if I do that and if the cancel fails ;)




Re: Serious bug in the "remove urls" tool Ben Griffiths 2/8/14 11:35 AM
webado,

I already did it on one of mine and was able to replicate. Whole site removal was submitted no matter what string.
Re: Serious bug in the "remove urls" tool webado 2/8/14 11:37 AM
Then I'd venture that requesting a removal for /subfolder/?querystring will also result in a full directory removal of /subfolder/ .


Re: Serious bug in the "remove urls" tool Ben Griffiths 2/8/14 11:50 AM
Nope - 'Web page' removal:

vs



Re: Serious bug in the "remove urls" tool mvandemar 2/8/14 12:00 PM
@Ben - there isn't actually a designation for "directory removal" as far as I can see, webado might be right. We would need to test it to find out. I will locate a directory on one of my own sites that is indexed with more than one page and test if removing a parameter-appended directory removes that whole directory as well.

-Michael
Re: Serious bug in the "remove urls" tool webado 2/8/14 12:00 PM
Even the bug is buggy ;)
Re: Serious bug in the "remove urls" tool webado 2/8/14 12:11 PM
Until the bug is fixed, I suggest you use the public url removal tool. It rather looks more robust.

Or you could strip off the dodgy query strings if you are hosted on an Apache server.

Here's how to strip unwanted query strings on an Apache server:
Re: Serious bug in the "remove urls" tool mvandemar 2/8/14 12:14 PM
Ok, I take that back... if the parameter is on the root domain it assumes it is a site removal, and I don't get a dropdown. If it is on a specific page or directory, then it gives the dropdown with the option of removing the page or the whole directory. I am testing now on a site with a subdirectory that I am not that concerned about that has 4 pages total. I blocked this in robots.txt:

/articles/?thisisnot=arealparameter

which doesn't actually exist, and then I submitted a request to remove site.com/articles/?thisisnot=arealparameter. We'll see if the directory itself survives the removal. Technically if the tool were working correctly nothing should be removed, since the specific page I am requesting be removed doesn't exist, but my guess is it will remove the /articles/ page and leave the other 3 under it intact.

-Michael
Re: Serious bug in the "remove urls" tool mvandemar 2/8/14 12:20 PM
Or you could strip off the dodgy query strings if you are hosted on an Apache server.

@webado - there are already redirects in place. The issue is the spammy stuff that is cached in Google. If Google never revisits those pages (which it won't, since they don't exist anymore) then it can take a very, very long time for them to disappear from Google's cache (Google is a bit of a hoarder, honestly). Thanks though. :)

-Michael
Re: Serious bug in the "remove urls" tool webado 2/8/14 12:21 PM
Ben said it didn't do that for the subdirectory.

Which is why I said even the bug is buggy - for being inconsistent ;)
Re: Serious bug in the "remove urls" tool webado 2/8/14 12:35 PM
>>Google is a bit of a hoarder, honestly

Understatement LOL

You could try Fetch as Googlebot for them.
Re: Serious bug in the "remove urls" tool webado 2/8/14 12:38 PM
Or list them in a plain text file and submit it like a sitemap. 


Re: Serious bug in the "remove urls" tool mvandemar 2/9/14 6:42 PM
For what it's worth, I tested and I can confirm that this issues appears to be specific to the root url only. GWT says that it processed the removal of this url:

site.com/articles/?thisisnot=arealparameter

And site.com/articles/ and all 3 pages underneath it all are still indexed.

-Michael
Re: Serious bug in the "remove urls" tool Amlan Maiti 2/10/14 9:23 PM
Hello Michael,

This is a big problem. "www.yoursite..co.uk/?p=bailey-bow-ugg " this is not a non-root url, this is root url with query string. For this google webmaster removed your full site. When your try to removed "www.yoursite..co.uk/marketing/?bsa.html" google show you "web page removal" that means google don't removed only "www.yoursite..co.uk/marketing/?bsa.html", google removed your full "www.yoursite..co.uk/marketing/" page.
Re: Serious bug in the "remove urls" tool mvandemar 2/12/14 7:32 AM
Or list them in a plain text file and submit it like a sitemap.

@webado, I do not see a way to do that. Are you sure that's possible? And if so, how?

-Michael
Re: Serious bug in the "remove urls" tool webado 2/12/14 5:55 PM
You make a plain text file using Notepad. Save it as  e.g. list-of-urls.txt, with one url per line. Upload it to your root folder. Submit it to Webmaster Tools like any other sitemap.
Re: Serious bug in the "remove urls" tool mvandemar 2/12/14 6:01 PM
Why would Google assume that was a list of urls I wanted removed...?

-Michael
Re: Serious bug in the "remove urls" tool webado 2/12/14 6:34 PM
It wouldn't assume that, but it will attempt to crawl them and thus discover whatever is happening to them: redirection or 404 or noindex (I don't remember what you're doing to those urls).
Re: Serious bug in the "remove urls" tool webado 2/12/14 6:42 PM
>>there are already redirects in place. The issue is the spammy stuff that is cached in Google. If Google never revisits those pages (which it won't, since they don't exist anymore) then it can take a very, very long time for them to disappear from Google's cache (Google is a bit of a hoarder, honestly)

Either they are 301 redirected elsewhere or they don't exist, so they respond with 404. Which is it?
More topics »