|matww||12/11/12 5:05 AM||<This message has been deleted.>|
|Re: DDOS by google bots||Lysis||12/11/12 6:30 AM|
Google will crawl at a higher rate when changes like this occur. Are you logging bot visits to a table or doing any kind of custom logging for reports? I've seen Google hit sites a 3 visits per second and it didn't affect anything, but people who are trying to sniff and do custom logging with poor performing code can kill their own site.
|Re: DDOS by google bots||Ashley||12/11/12 7:41 AM|
They'll also crawl at a higher rate if your code or site structure has issues. The bot can get stuck.
What pages are they hitting the most? What are the responses you serve? How many a day are you getting?
|Re: DDOS by google bots||matww||12/11/12 9:07 AM|
all requests are coming from 188.8.131.52/16 IP range
from different IP to different page up to 100 connections per second
example: 184.108.40.206 - - [11/Dec/2012:14:47:11 +0100] "GET /captcha.png.php?667 HTTP/1.1" 200 294
Please someone from Google team look into this.
|Re: DDOS by google bots||Ashley||12/11/12 9:08 AM|
|Re: DDOS by google bots||matww||12/11/12 9:18 AM|
just a tip to further explain, all these problems started after I switched to Cloudflare CDN
|Re: DDOS by google bots||JohnMu||12/11/12 11:45 AM|
The fastest way to give us this information is to use the "report a problem with Googlebot" link next to the crawl rate settings. The reports you submit there go directly to the Googlebot-team, and they'll be able to take direct action based on the information you give them there. I'll also forward this to them, but going directly is generally faster :)
|Re: DDOS by google bots||matww||12/11/12 6:17 PM|
I've followed your guideline and report is also submitted too. Please forward this forum request also as to escalate the matter as it is badly hammering my server and I got no resort so far.
|Re: DDOS by google bots||webado||12/11/12 8:25 PM|
Shouldn't you block anything to do with your Captcha so it won't get requested even?
|Re: DDOS by google bots||webado||12/11/12 8:30 PM|
You have links on your site which result in in a page that claims to be a 404 like: http://www.expatads.com/18-Canada/posts/5-/200-House-Exchange/ but whch in fact responds with a 200.
Big issues I call this.
First off your site navigation should not lead to anything that's not there. Second of all anything that's not there should respond with 404, not 200.
No wonder you get hammered by bots.
|Re: DDOS by google bots||matww||12/11/12 8:45 PM|
Where did you get this link from?
Can you please tell me source of this link, I can't find this link generated at my site. Unless you tear off a part of it then it will return 404.
|Re: DDOS by google bots||webado||12/11/12 8:54 PM|
attempted to add an ad for Canada > housing > house exchange
I think. Not sure what sequence of clicks bought me there. But here is somehtign like it:
and now I get this message :
This website is offline
No cached version is available
|Re: DDOS by google bots||webado||12/11/12 8:56 PM|
>>Unless you tear off a part of it then it will return 404
Nope, all invalid urls, impossible gibberish urls respond with 200 while claiming to be 404. test with web-sniffer.net .
|Re: DDOS by google bots||matww||12/11/12 8:58 PM|
Yes, that's the problem right now google bots are bombarding and it happens servet times a day and will make server literally dead. That's why now you see website offline.
|Re: DDOS by google bots||webado||12/11/12 9:02 PM|
Well as I told you the problem is your bad server responses besides generating urls in your navigation which don't exist (but respond with 200 not 404).
Unless you fix this your site will get hammered.
|Re: DDOS by google bots||matww||12/11/12 9:03 PM|
Yes, thanks for bring that into my notice, I got it. But that's indeed not the cause of google bot attack or overwhelming crawling it only started to happen since I switched to CloudFlare.com a reputed CDN. Interestly earlier I had option to manually set crawling frequencey in webmaster tools since I switched to CDN that option is no long avaiable instead I found message we have set a speical crawling rate and it cannot be changed.
|Re: DDOS by google bots||webado||12/11/12 9:10 PM|
SO I managed to get to the page where I can post an add and found your captcha.
Your robots.txt file is worse than useless.
You need to use it to block what should nto get crawled.
Your action script for the form should no tget crawled.
Your captahc image should not get crawled.
Probably many other things should not get crawled.
You currently have 2 captcha images indexed:
|Re: DDOS by google bots||webado||12/11/12 9:17 PM|
Reputed or not, the server is not configured correctly.
|Re: DDOS by google bots||Free2Write||12/11/12 10:06 PM|
|Re: DDOS by google bots||webado||12/12/12 4:41 AM|
Awesome! Nothing like a bit of predicate calculus sprinkled with Latin to drive the point home ;) :)
|Re: DDOS by google bots||JohnMu||12/13/12 7:58 AM|
It looks like you or another verified owner have over 90'000 variations of your website listed in your Webmaster Tools account. As we regularly check for verification elements on these pages, we have to fetch them regularly -- which is, to some extent, the traffic that you're seeing. I would strongly recommend listing domains only once or twice (perhaps www/non-www), it makes it much easier to maintain things on your side.
Hope it helps!
|Re: DDOS by google bots||Lysis||12/13/12 8:03 AM|
holy crap LMAO
|Re: DDOS by google bots||travler||12/13/12 8:30 AM|
just when you think you've heard it all, along comes John!
|Re: DDOS by google bots||Lysis||12/13/12 8:39 AM|
And people wonder why we're cynics...
There's always a reason. ALWAYS. And it's usually some dope SEO behind it all.
|Re: DDOS by google bots||luzie||12/13/12 3:30 PM|
|Re: DDOS by google bots||luzie||12/13/12 3:33 PM|
Boy, you have created a monster, an alien, an ugly species ... :-/
|Re: DDOS by google bots||VenomVoid||12/14/12 6:45 AM|
Just go and try to manually change the rate, they are known to KILL sites dead.
for them is an experiment, for site owners is a nightmare
|Re: DDOS by google bots||VenomVoid||12/14/12 6:48 AM|
and then you have those strange bots that will not tell you who they are eating up your bandwith.
you should be ware of those and try and knock'em using url_rewrite..
|Re: DDOS by google bots||Seo100||12/14/12 7:04 AM|
LOL, Google Bot rocks this time. I think this is case of using too much WMT. These days every one use WTM for ranking and links as well, but this is ex-team :P