| Can I restrict Google from crawling my site on a specific day of the week? | rcneeman | 9/8/09 2:31 AM | Is it possible to restrict Google from crawling my website on Saturdays or Sundays routinely? I'd like it to crawl my site on weekdays only. Thanks |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Phil Payne | 9/8/09 2:38 AM | There is a defined interface - sniff out the Googlebot user-agent in a server-side script and serve it a 503 response code with a Retry-After header set for Monday. The Googlebot is very, very likely to ignore the Retry-After - some years ago people |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | rcneeman | 9/8/09 2:53 AM | Thanks for your answer. I'm not trying to trick Googlebot (but try explain that to a bot). Since our site is geared towards Sabbath observing people (Orthodox Jews), there's a demand that the site will be down on Saturday, because we are not allowed |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Phil Payne | 9/8/09 3:06 AM | Yes - a 503 is the correct server response for "We're closed". If you substitute a normal HTML page saying "We're closed" and serve a 200 it's very likely to get indexed by Google. If you give the Googlebot a 503, it will just go away and come back |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Mark Dalton | 9/8/09 3:13 AM | That's an interesting problem.... think I would consider an image with a phrase you consider appropriate that indicates you are closed for business and only serve that image up at the top of the page to visitors on a Saturday for instance... any oth |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Mark Dalton | 9/8/09 3:17 AM | p.s. BTW you would still deliver the usual page too, but insert the "closed" image using the s/w you have identified! Robots can't read images so should be a safe method. |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Phil Payne | 9/8/09 3:26 AM | I don't think you can do that - I don't think you can show any business information at all. The 503 custom page, as I understand Orthodox rules, may not even say what the business does. Some of these people are very strict. There was a case of a c |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Mark Dalton | 9/8/09 6:08 AM | WOW that's going to be tough .... unless a new code is defined for this type of thing... but no doubt someone would find a way to abuse it! From what you are saying... a physical shop would need shutters over windows & signs etc .... The only metho |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Sebastian | 9/8/09 9:57 AM | I understood that you've a tool in place to 503 visitors based on their IPs. Why not just perform crawler detection using the recommended methods to exclude bots? Is having computers running unattended against the rules? If so, why is it ok to run a |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | rustybrick | 9/9/09 6:16 AM | Sebastian, I think it depends on the Rabbi you ask. rcneeman, is it about making money on Shabbat or about the servers working on Shabbat? The easy solution, if it was money making is to just drop the "add to cart" and shopping cart feature. Th |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | isaac11 | 1/10/10 1:30 PM | I also don't want to sell products on Saturday so i'm changing the "add to cart" button with phrase saying "buying unavailable now" and a link to a page that states the schedule of shabbos, but i notice that google bot crawls my site a Saturday and p |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Autocrat | 1/10/10 1:37 PM | Sorry - but did the info above (noted as Best Answer) not make it clear for you? You send a 503 response on the days/dates/times you wish to avoid being indexed. Whether you do this at the Server level, or via a script... Whether you do this Manua |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | isaac11 | 1/10/10 1:50 PM | thanks Autocart. does it mean i need a script to control the server to give the 503 response "only to the Google bot etc., and to any other requester it should serve the normal page, thanks |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | JohnMu | 1/10/10 1:59 PM | Hi Isaac11 The interesting thing about a 503 HTTP result code (or most others) is that you can serve normal content to your users and it will only be recognized by those that explicitly watch out for the result codes, usually only search engine crawl |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Autocrat | 1/10/10 2:12 PM | If you are on an apache based server - you can setup the htaccess to send a 503 response header for any URL. Alternatively - you can script in a header response into your site (if it's dynamic). Both these options mean that you can let your site be |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | isaac11 | 1/10/10 2:31 PM | can i implant the 503 code only for that phrase at the <P>Buying unavailable Now</P> level, or it goes only in page header? thanks Isaac |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | JohnMu | 1/10/10 2:39 PM | Hi Isaac The HTTP result code is on a page level, so it would cover any and all content that you have on your page. Cheers John |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Autocrat | 2/25/10 10:25 AM | 1) You should really create your own topic and post all relevant details there. 2) Maybe the AdSense/AdWords groups would be better places to ask than here? |
| Re: Can I restrict Google from crawling my site on a specific day of the week? | Mark Dalton | 2/26/10 1:51 AM | I can see 2 options here... if you are using a programming language like php then you can work out the date or day of the week... and where it matches your criteria the you don't insert the adwords code.... it could probably be done with javascript |