Blocking Israel is also not enough. At a minimum, one
must also do this:
Block all image requests with blank user agents. Picscouts firefox add on will visit with blank user agent after someone uses it. But also, I see *many* visits to images using blank user agents only. So images should never be served to these things. This should very, very rarely interfere with any honest visits because everything should present a user agent. (Some people who are both stupid and paranoid buy privacy software that presents a blank user agent. But it's very, very rare. And those people should be told to turn on their user agent.)
These visits do not come from Israel. They are generally coming from larger servers.
Oh...today... I saw something try to visit
rankexploits.com/protect/wp-content/themes/
imagesThis is definitely an attempt to find images that might have been uploaded to the 'theme' folder. There is nothing at that location. Because I partially 'broke' my wordpress redirects, that attempt was sent to the "file missing" bin (404). I made my 404 page dynamic, and also run a script to ban things that are looking for <i>missing</i> things in 'wp-content/themes/'. So, that IP got banned.
Here's the record of the 'ban'
#: 111365 @: Thu, 15 Nov 2012 07:09:38 -0800 Running: 0.4.10a1
Host: 91.229.125.213
IP: 91.229.125.213
Score: 2
Violation count: 1
Why blocked: ; It looks like you are trying to call the theme directly. (404) Fingerprint, scrape or hack behavior. (2: wp-content/theme ) || ( ax=0) [GB] ( 404=1 ) ; ( 0 )
Query:
Referer:
User Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7
Reconstructed URL: http:// rankexploits.com /protect/wp-content/themes/images/arrows-ffffff.png
That image is not on my site. But the fact that something tried to load it means either
a) They were trying to figure out if I have a particular image directory at that location or
b) They were hunting for a vulnerability -- that would be a threat unrelated to image sniffing.
I can't know which. But those worried about image scrapers should be worried about both. (To some extent, it is pointless to worry only about image scraping. You simply
cannot block those things without blocking all the other stuff too. Leaving avenues for other types of scraping or hacking open will leave avenues for scraping open. Period.)
Anyway, here's are some details on the IP:
IP=91.229.125.213 is associated with
netname: ITLAB-NET
descr: IT Lab Limited
That looked like an interesting name, so I googled and discovered:
http://www.itlab.co.uk/That is
a) a cloud service and
b) one that provides custom coding
http://www.itlab.co.uk/services/consultancy-services/codelab/I think the probability that IP was someone was devoting a lot of effort to hunt for images is near 50%. But that's a guess. The alternative is it's a penetration tester-- which is just as bad.
Whois gives their domain range as
91.229.124.0 - 91.229.127.255
They are located in Great Britain and an Israel ban would not keep them out.
Based on watching my logs have every reason to believe that crawlers looking for images are:
1) using a vast number of proxies-- both anonymous and transparent. I often see blocked israeli IPs using non-transparent proxies.
2) using specialized cloud services located all over the world.
I see lots of hits from "The Planet", "Hostgator", "BlueHost", "voxility" and all sorts of dedicated servers all over the world. These are cheap services picscout or any picscout-like entity (e.g. tineye, idee etc.) could chose to run a scraper. I suspect they do so. I'm reasonably certain that if you think blocking Israel is enough, you are likely wrong.
I have a blog. I post lots of graphs-- most created by me-- but some by co-bloggers. All are constantly visited by agents with
a) referrers I
know to be wrong.
b) blank or weird user agents.
To catch the scrapers, I'm doing rather complicated stuff. Specifically:
1) In .htaccess, I divert image requests with whacked out referrers or user agents to a script. I also send requests for any images that are more than 3 months old to the script.
2) That script processes the request. If it fails quality checks, that IP is banned-- at Cloudflare. That means that IP cannot crawl anything at my site. If it passes quality checks, it is shown the image. If it's in between, it is shown a substitute image of a cat.
(Note the potential downsite of blocking a Hostgator/Bluehost/etc. IP at cloudflare is that if a blog on those services sent me a 'ping' I wouldn't see it. I'm willing to sacrifice that-- as it's really the only reasonable incoming connection from those services my blog might expect. )
This system is:
1) resource intensive because you have to run a script rather than just deliver an image.
2) requires willingness to fiddle with your .htaccess and customize in a way that makes sense. (In fact, you must edit each month if you want to check all requests for images older than 3 months.)
In addition, you need to make some decisions about what you are going to permit. For example: My /feed/ addresses do not display links to images if one loads those addresses directly in the browser. So, things that try to load images from
http://myblogdomain/blogpost/feed are banned. I've been doing this for months. I ban at least 20 things a day with that referrer and you know what? I have never, ever, ever had a human complain. That tells me those requests are not people.
I don't think there is any other way to block image scrapers. Moreover, I doubt if I even succeed in blocking all of them. If someone or something wants to visit each post
promptly providing a not obviously wrong user agent, and a not-obviously-wrong referrer, that someone or something will be indistinguishable from a real visitor. My system will let them view the images. So, an image scraper could see every single image. (You might wonder why their bot doesn't do this? It's because it takes more resources to monitor a blog for posting, and load a whole page rather than just try to watch what shows up on google image search and then load that image without knowing what the "right" referrer would be. My guess is they show some obviously wrong referrers because they don't know the right one for the image.)
Oh-- and I'm doing more stuff. For example: I'm blocking all connections from TOR, and I"m blocking lots of connections from free public proxies listed on various web sites. Doing the latter has drastically cut into the rate at which I see requests for images with blank user agents. (This is why I feel rather certain that lots of the image scrapers are using public proxies.)
Anyway... unfortunately, I can't give good easy advice on how to block picscout and picscout like scrapers. I know I've made it expensive for them to operate on my site-- that's the best anyone could do.
Topic: Picscout sighting. (Read 11651 times)