Categories: Crawling, indexing & ranking :

Re: DDOS by google bots

Showing 1-29 of 29 messages
matww 12/11/12 5:05 AM <This message has been deleted.>
Re: DDOS by google bots Lysis 12/11/12 6:30 AM
Google will crawl at a higher rate when changes like this occur. Are you logging bot visits to a table or doing any kind of custom logging for reports? I've seen Google hit sites a 3 visits per second and it didn't affect anything, but people who are trying to sniff and do custom logging with poor performing code can kill their own site.
Re: DDOS by google bots Ashley 12/11/12 7:41 AM
They'll also crawl at a higher rate if your code or site structure has issues. The bot can get stuck. 

What pages are they hitting the most? What are the responses you serve? How many a day are you getting?
Re: DDOS by google bots matww 12/11/12 9:07 AM

all requests are coming from 74.125.0.0/16 IP range

from different IP to different page up to 100 connections per second

example: 74.125.16.211 - - [11/Dec/2012:14:47:11 +0100] "GET /captcha.png.php?667 HTTP/1.1" 200 294

Please someone from Google team look into this.
Re: DDOS by google bots Ashley 12/11/12 9:08 AM
Escalated. 
Re: DDOS by google bots matww 12/11/12 9:18 AM
just a tip to further explain, all these problems started after I switched to Cloudflare CDN
Re: DDOS by google bots JohnMu 12/11/12 11:45 AM
Hi Matww

The fastest way to give us this information is to use the "report a problem with Googlebot" link next to the crawl rate settings. The reports you submit there go directly to the Googlebot-team, and they'll be able to take direct action based on the information you give them there. I'll also forward this to them, but going directly is generally faster :)

Cheers
John
Re: DDOS by google bots matww 12/11/12 6:17 PM
JohnMu,

I've followed your guideline and report is also submitted too. Please forward this forum request also as to escalate the matter as it is badly hammering my server and I got no resort so far.

Thanks
Re: DDOS by google bots webado 12/11/12 8:25 PM
Shouldn't you block anything to do with your Captcha so it won't get requested even?
Re: DDOS by google bots webado 12/11/12 8:30 PM
You have links on your site which result in in a page that claims to be a 404 like: http://www.expatads.com/18-Canada/posts/5-/200-House-Exchange/ but whch in fact responds with a 200.

Big issues I call this.

First off your site navigation should not lead to anything that's not there. Second of  all anything that's not there should respond with 404, not 200.
No wonder you get hammered by bots.


Re: DDOS by google bots matww 12/11/12 8:45 PM
webado,

Where did you get this link from?

http://www.expatads.com/18-Canada/posts/5-/200-House-Exchange/

Can you please tell me source of this link, I can't find this link generated at my site. Unless you tear off a part of it then it will return 404.
Re: DDOS by google bots webado 12/11/12 8:54 PM
 attempted to add an ad for Canada > housing > house exchange

I think. Not sure what sequence of clicks bought me there. But here is somehtign like it:


and now I get this message : 

This website is offline

No cached version is available


Re: DDOS by google bots webado 12/11/12 8:56 PM
>>Unless you tear off a part of it then it will return 404

Nope, all invalid urls, impossible gibberish urls respond with 200 while claiming to be 404. test with web-sniffer.net .
Re: DDOS by google bots matww 12/11/12 8:58 PM
Yes, that's the problem right now google bots are bombarding and it happens servet times a day and will make server literally dead. That's why now you see website offline.
Re: DDOS by google bots webado 12/11/12 9:02 PM
Well as I told you the problem is your bad server responses besides generating urls in your navigation which don't exist (but respond with 200 not 404).

Unless you fix this your site will get hammered.
Re: DDOS by google bots matww 12/11/12 9:03 PM
Yes, thanks for bring that into my notice, I got it. But that's indeed not the cause of google bot attack or overwhelming crawling it only started to happen since I switched to CloudFlare.com a reputed CDN. Interestly earlier I had option to manually set crawling frequencey in webmaster tools since I switched to CDN that option is no long avaiable instead I found message we have set a speical crawling rate and it cannot be changed.
Re: DDOS by google bots webado 12/11/12 9:10 PM
SO I managed to get to the page where I can post an add and found your captcha.

Your robots.txt file is worse than useless.

You need to use it to block what should nto get crawled.

Your action script for the form should no tget crawled.
Your captahc image should not get crawled. 

Probably many other things should not get crawled.

You currently have 2 captcha images indexed:



Re: DDOS by google bots webado 12/11/12 9:17 PM
Reputed or not, the server is not configured correctly.
Re: DDOS by google bots Free2Write 12/11/12 10:06 PM
http://web-sniffer.net/?url=http://www.expatads.com/18-Canada/posts/5-/200-House-Exchange/&uak=9&type=HEAD

http://web-sniffer.net/?url=http://www.expatads.com/18-Canada/posts/5-/jibberish/&uak=9&type=HEAD

http://web-sniffer.net/?url=http://www.expatads.com/18-Canada/more-jibberish/&uak=9&type=HEAD

http://web-sniffer.net/?url=http://www.expatads.com/proper-jibberish/&uak=9&type=HEAD

http://web-sniffer.net/?url=http://www.google.com/18-Canada/posts/proper-jibberish/&uak=9&type=HEAD

http://www.iana.org/assignments/http-status-codes/http-status-codes.xml

http://en.wikipedia.org/wiki/List_of_rules_of_inference#Rules_of_classical_predicate_calculus


Re: DDOS by google bots webado 12/12/12 4:41 AM
Awesome! Nothing like a bit of predicate calculus sprinkled with  Latin to drive the point home ;) :)
Re: DDOS by google bots JohnMu 12/13/12 7:58 AM
Hi Matww

It looks like you or another verified owner have over 90'000 variations of your website listed in your Webmaster Tools account. As we regularly check for verification elements on these pages, we have to fetch them regularly -- which is, to some extent, the traffic that you're seeing. I would strongly recommend listing domains only once or twice (perhaps www/non-www), it makes it much easier to maintain things on your side. 

Hope it helps!
John
Re: DDOS by google bots Lysis 12/13/12 8:03 AM
holy crap LMAO
Re: DDOS by google bots travler 12/13/12 8:30 AM
just when you think you've heard it all, along comes John!
Re: DDOS by google bots Lysis 12/13/12 8:39 AM
And people wonder why we're cynics...

There's always a reason. ALWAYS. And it's usually some dope SEO behind it all.
Re: DDOS by google bots luzie 12/13/12 3:30 PM
>>> t looks like you or another verified owner
>>> have over 90'000 variations of your website
>>> listed in your Webmaster Tools account.

*scream* - This is as freaky as anything ever since Wrong Turn IV , worse than Texas Chain Saw Massacre ?
Re: DDOS by google bots luzie 12/13/12 3:33 PM
Boy, you have created a monster, an alien, an ugly species ... :-/
Re: DDOS by google bots VenomVoid 12/14/12 6:45 AM
Just go and try to manually     change the rate, they are known to KILL sites dead.
for them is an experiment, for site owners is a nightmare
Re: DDOS by google bots VenomVoid 12/14/12 6:48 AM
and then you have those strange bots that will not tell you who they are eating up your bandwith.
you should be ware of those and try and knock'em using url_rewrite..

Re: DDOS by google bots Seo100 12/14/12 7:04 AM
LOL, Google Bot rocks this time. I think this is case of using too much WMT. These days every one use WTM for ranking and links as well, but this is ex-team :P
More topics »