Google Product Forums

Re: Ondemand removal on random/individual urls is made easier

omr Oct 23, 2011 11:11 PM
Posted in group: Google Custom Search

Categories: Indexing and Results :

@kuch11:

Currently it appears that your web server does not use the appropriate HTTP status code of either 404 (Not Found) or 410 (Gone), in responses to requests for URLs of pages that no longer exist on your site.

For such requests, currently your web server issues a 301 (Moved Permanently) response, redirecting the browser to your "Page Not Found" help page; and when the browser requests that page, the web server issues a response with HTTP status code 200 (OK).

Here is a sequence of events observed during my test:

  1:
  I searched for "asp" on your site,
  and I clicked a search result leading to:
  http://www.spray.com/ispray/ispray.asp
    
  2:
  The browser issued a GET request for that page.
    
  3:
  The web server issued a response with
  HTTP status code 301 (Moved Permanently),
  redirecting the browser to your
  "Page Not Found" help page URL:
  http://www.spray.com/help/?404;http://www.spray.com:80/ispray/ispray.asp
    
  4:
  The browser issued a GET request for that page.
    
  5:
  The server issued a response with
  HTTP status code 200 (OK).

  ...

I think you'll need to configure your web server to issue an appropriate HTTP status code (either 404 or 410) for 'Page Not Found' responses.

After completing that configuration change, and after testing* to verify that the web server issues an appropriate status code for such responses, then try Custom Search index removal procedures:

   http://www.google.com/support/customsearch/bin/answer.py?hl=en&answer=1351673

   http://code.google.com/apis/customsearch/docs/indexing.html#removal


* Note:  HTTP status codes can be observed by using appropriate diagnostic software.  For example, if you use the Mozilla Firefox browser, you can use the "Live HTTP Headers" add-on, or the "HttpFox" add-on, or the network monitoring panel of the "Firebug" debugger, to observe HTTP traffic and check the response status codes.

-- omr