Categories: Chit-chat :

lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler???

Showing 1-22 of 22 messages
lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/3/13 8:23 PM
lately i see a lot of request with USER-AGENT set to "rarely used".

looks like a crawler or bot. there is never any referer passed.

any idea?

PS: check your server log and look for it.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? webado 6/3/13 8:28 PM
The user agent can be set to anything. I've not seen any like that, but I don't look too often in my logs.
 
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/3/13 8:37 PM
The user agent can be set to anything.

i know all about that. but it's good manners to pass descriptive user
agents, unless you do something shady...
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? webado 6/3/13 9:04 PM
Manners on the internet? Surely you jest ;)


Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? Lysis 6/4/13 6:15 AM
>> lately i see a lot of request with USER-AGENT set to "rarely used".

lolol that's funny.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/4/13 1:51 PM
the only reference i found about this "rarely used" user agent is in this forum page:

it's very hard to use google to search infos about it, because there is no way to tell google to only search pages with the words "rarely used" in double-quotes. google will return all the pages with the words "rarely used" appearing on the page, together but not enclosed by quotes.
loupiote 6/7/13 10:25 PM <This message has been deleted.>
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/7/13 10:27 PM
i did a bit more digging about this new "rarely used" user agent.  here is what i found:

1) it fetches only images, and passes no referrer.

2) it originates from a wide range of IP addresses, mostly from the US, and those IP often sit on major networks, like comcast

3) it is not rarely used (i get thousands of requests from it every day).

4) it is definitely not a robot or a crawler. it appears to be used by either a mobile app, or maybe a browser extension?

i'd love to know what is the perpetrator...
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? Zakkie 6/27/13 6:26 AM
Looks to be a mobile thing. I get thousands of requests and they are always for images. No referrer is ever set, and they seldom request more than 3 or 4 images from each IP.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/27/13 10:53 AM
Looks to be a mobile thing. I get thousands of requests and they are always for images. No referrer is ever set, and they seldom request more than 3 or 4 images from each IP. 
yes, that's exactly what i observe.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? Ashley 6/27/13 1:14 PM
Perpetrator? What is it doing that is so negative?
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/27/13 1:23 PM
Perpetrator? What is it doing that is so negative?

using the user-agent "rarely used" is the problem.

the user agent is ment to give useful information about the software or browser or crawler that makes the http requests.

it is not illegal, but it is very bad "netiquette", and it will cause a lot of webmasters to block requests with this user-agent, since it appears suspicious for not telling what it actually is.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? webado 6/27/13 3:59 PM
And do you always believe the user agent string? ANybody can fake it for their own browser even, let alone robots.
You can be using Firefox but pretend you are using IE.

Is that any bettter?
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/27/13 4:45 PM
yes, it is better.  i know you can spoof user-agents, but most major search engine crawlers (googlebot, bingbot, yandex etc) and most browsers (chrome, ff, ie, safari) use a correct user-agent.

i know that anyone can spoof if they want to, but the majority of people don't.

in any case, i am blocking "rarely used" until we know what app is using this user agent.  i don't see what they gain by using a non-descriptive user-agent.  they just have more chances to be blocked as being "suspicious".
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? webado 6/27/13 4:47 PM
Googlebot spoofs the UA as well when it tries to discover cloaking or malware.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/27/13 4:52 PM
Googlebot spoofs the UA as well when it tries to discover cloaking or malware.

i'm sure they do (and hopefully they also use an IP address that doesn't look like a google IP).  but they do that only for a fraction of their requests.  otherwise it would screw-up the analytic statistics pretty good!

and google has many other ways to discover cloaking or malware.  for example they can do that in chrome, and in that case, no need to spoof their UA.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? webado 6/27/13 5:02 PM
Analytics doesn't pick up hits from robots - only javascript enabled critters.

What is reported in Chrome for malware usually comes from earlier discoveries done by Googlebot. But I think it's got a little something extra  as well to discover new stuff.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 6/27/13 5:15 PM
Analytics doesn't pick up hits from robots - only javascript enabled critters.

a lot of bots are able to execute javascript.  that includes googlebot. i received hits on URL's that only appear in some javascript code on my pages, and those hits are from googlebot . i can tell not only from the user-agent (which can be spoofed), but also from the google IP address (much harder to spoof, right?).

Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? webado 6/27/13 5:47 PM
They don't exactly execute javascript - those that are so enabled simply parse it to discover links or wonky redirections. It's limited.

The  Analytics script will not log hits from robots.

Google seems to have some robot-look-alikes which are more akin to browsers. FAIK they are actual computers that run a browser and visit sites. Anyway, they would be for specific tasks.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? Zakkie 7/5/13 4:02 AM
Webado, you're not fully informed. There are javascript parsers out there that can be fully automated & scripted, and they execute javascript as well, probably better then, most javascript-enabled browsers. There is no barrier to automated javascript execution at all.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? Zakkie 7/5/13 4:19 AM
s/better then/better than/

No idea why I can't edit my post, but please make the above change. Thanks.
Re: lately i see a lot of request with USER-AGENT set to "rarely used". is that a new crawler??? loupiote 7/5/13 9:12 PM
> Webado, you're not fully informed. There are javascript parsers out there that can be fully automated & scripted,\

that's correct.

however, the "rarely used" user agent it not one of them.

it appears to be used by a mobile app that fetches and displays images. maybe it's used by some sort of image-search app?
More topics »