New image bot Pixray

[lucia]

New image bot
Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; [email protected])
IP: http://whois.domaintools.com/176.9.31.202

node-176-9-31-202.cluster.eu.webcrawler.pixray.com
I'm going to block "Pixray-Seeker"  by user agent the this entire webcrawler in various ways.

[lucia]

More on pixray
http://www.pixray.com/products
"The PIXRAY Seeker™ helps image copyright holders and brand managers to track their assets on the web." This seems to be the bot that stopped by.  I'm banning it for a variety of reasons with which ELI members will sympathize, but also because I don't want to spend my money to provide computer resources to let others crawl for their own business purposes.

On the other hand "The PIXRAY Filter™ helps web site operators to retain control of image content uploaded by their users." seems to be a very useful product that anyone who lets users upload images should look into.

[SoylentGreen]

Good work, Lucia!!
Very interesting info; we'll be hearing more about this in the near future, I'm sure.

S.G.

[lucia]

At least for the time being I seem to have gotten the bots that were racing through all my images on a daily basis to go away.   This one visited robots.txt.  I guess it's worth checking who visits that!

[Matthew Chan]

I notice that Pixray is from Germany.  I am guessing Pixray will be just as intrusive and invasive as Picscout.  Apparently, folks in Europe don't have to play by the same rules as those in the U.S.

[Khan]

The ip location is Hetzner Online AG  in Germany (biggest internet hosting company in Germany). This Company had a severe security issue last year October. During weeks password lists, bank data and even police emails were open and accessible for almost everybody.
 
Normally the German data law is very strict. They always picking on facebook data gathering.

[Robert Krausankas (BuddhaPi)]

@Lucia, please do share your methods of blocking, ie htaccess, or robots.txt, do we know a user-agent for this one? I'm seeing a spike from 67.54.19.69 which goes back to a savvis communications here in the states..

[Robert Krausankas (BuddhaPi)]

Heres another bandwidth sucking thief worth blocking.

Host: 220.181.94.214
Http Version: HTTP/1.1
Referer: http://pic.sogou.com
Agent: Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

[SoylentGreen]

Anyone here familiar with the concept of "TressPass to Chattels"?
Spiders that are consuming the resources of another's system(s) might apply here.

http://www.chillingeffects.org/linking/faq.cgi#QID460

Question: What are the "trespass to chattels" claims some companies or website owners have brought?

Answer: Some Internet companies have claimed that unauthorized use of their servers, such as unsolicited email or robot-generated hits to websites, are a "trespass" to those servers by depriving the owners of the full use of their machines. eBay won an injunction stopping Bidder's Edge from automatically spidering the eBay site to generate auction comparison listings, because the robotic crawler used eBay system resources. The caselaw is far from settled in this area, and some commentators argue that technical means to block the use are more appropriate than legal action.

---

Example:

eBay, Inc. v. Bidder's Edge, Inc.

Wiki:

http://en.wikipedia.org/wiki/EBay_v._Bidder%27s_Edge

eBay v. Bidder's Edge, 100 F.Supp.2d 1058 (N.D. Cal. 2000), was a leading case applying the trespass to chattels doctrine to online activities. In 2000, eBay, an online auction company, successfully used the 'trespass to chattels' theory to obtain a preliminary injunction preventing Bidder’s Edge, an auction data aggregator, from using a 'crawler' to gather data from eBay’s website.[1] The opinion was a leading case applying 'trespass to chattels' to online activities, although its analysis has been criticized in more recent jurisprudence.

http://pub.bna.com/lw/21200.htm
IN THE UNITED STATES DISTRICT COURT, FOR THE NORTHERN DISTRICT OF CALIFORNIA, NO. C-99-21200 RMW, ORDER GRANTING PRELIMINARY
INJUNCTION [Docket Nos. 6, 12]

S.G

[lucia]

Budhappi-
I will be.  I now run access to my blog through cloudflare-- so I block there.  The delay in releasing my "methods" is that I keep adding little tweaks to my module based on what I see. I also know I need to add a module to *unblock* somethings because I end up blocking both hack attempts and images scraping attempts. The hack attempts change IPs, the image scraping tends to be pretty static.

But there are lots of ways to block:
1) Use ZBblock and add something to the custom sigs.
2) Use htaccess.  But that's flexible, so it's sometimes difficult to tell someone precisely what to do.
3) If you run through cloudflare, block there. That's sort of like blocking in .htaccess, but you don't even see the server load.

I do all 3!! (But if you use cloudflare, you often can't block by IP in .htacesss.)

Since the method will vary depending on how you run your blog, I'm now just saying *what* to block.   If you want to know .htaccess commands to block something in particular, I can give those. But I'm not an .htaccess guru. And I know enough to know that there are considerations in .htaccess that one needs to be aware of-- so I'll ask if you are running a WP blog in a subdomain and other questions.

[lucia]

Budhappi--
As I mentioned, my method is complicated. But, now write every access to .php file (i.e. Wordpress) to a a 15 minute log file. At the end of 15 minutes, I runs a cron job that looks at the logs and bans every IP with these things in it:

$bannedSpiders=array("brandwatch.net","huaweisymantec","Ahrefs","SiteBot","baidu","yandex","CydralSpider","Cydral Image","Aghaven/Nutch","/Nutch-1.2 ","aghaven.com","linkdex.com","bazqux.com","getty","googlealerts","  spider ","http://intensedebate.com/","gettywan","CoverScout","tineye","cmsworldmap.com","web-sniffer.net","(IBM EVV/3.0/EAK01AG9/LE)","Windows NT;","Windows 3","Windows 95","webinator","thunderstone","t-h-u-n-d-e-r-s-t-o-n-e","proximic","picsearch","java","Extreme Picture Finder","BPImageWalker","doubanbot","curl","reverseget.com","RGAnalytics","CoverScout","wikio","python-urllib","Powermarks","KeywordSearchTool","Chilkat","ahrefs.com","panscient","liperhey","Webster","sosoimagespider","bpimagewalker","CCBot/1.0","Daylife","xrumer","xpymep","mail.ru","mozilla/picgrabber","psbot","NSPlayer","vlc/","TraumaCadX","upictoBot","PHP/5.2.10","http://mattters.com/","commoncrawl","pixmatch","Copyright","abot","aipbot","image","pics","pict","SNAPSHOT","naver","sogou","soso","magpie","Netseer","Mozilla/0","Bitvo","pipl.com","www.80legs.com","benderthewebrobot","Semrush","WordPress/3.4-alpha","Wget","Pixray","mlbot","MJ12bot","www.accelobot.com");

I think you can see that quite a few are obviously image bots.  Heck if it's got "pics" or "pict" in it, it's out of here! "pix" onl y gets caught in pixmatch and pixray.   Note also things containing 'getty', tineye,  (Note some bans are redundant. I've done little to make this particularly efficient.)

Some are things that showed up when I started banning picscout-- and which turn out to be particularly effective at processing images. These include ""NSPlayer","vlc/","TraumaCadX","

I'm tempted to ban anything with "alpha" in it. (That Wordpress/3.4-alpha thing was really going to town asking for the same post thousands of times at a rate of... on... more than 1 per second. Banned!)  The code also bans anything that give NO useragent. (This bans IPs for the imageExchange add on.) 

Some of the banned agents are SEO bots. They can be voracious; example: brandwatch.

Some are generic copyright bots: googlealerts (which is copyscape, not google). 

I also have bans for certain referrers. 

I also have a real time module.... And some bans in .htaccess!  But the list above is a good start for things to block by useragent. 

[lucia]

The caselaw is far from settled in this area, and some commentators argue that technical means to block the use are more appropriate than legal action.
The main thing is technical means have the potential to work immediately. If used widely, they will also raise the cost of operation for the trolls.

Legal means are costly, difficult and time consuming.  Running software to collect data to prove trespass in court would be no easier than blocking.  (In fact, to detect trespass in real time one would likely just add a module to the exact same software that blocks. Why not block while you are at it.)

That the law is unsettled makes the idea of using legal means unrealistic for a small time blogger or small time business owner who is running a blog or web site.

This is not to say that one should rule out legal action. But for most of us it may not be a convenient option relative to blocking.

[Robert Krausankas (BuddhaPi)]

Thanx for response Lucia, I'm currently blocking using different methods as well,  .htaccess, blacklisting IPs at the firewall & server level, I haven't had any issues as of late, I think I got a pretty good handle on it at this point, tho every once in a while I'll add an IP or 2. I'm not doing business worldwide, so it's easier for me to just ban IP Country blocks, such as Israel, china, nigeria, ect..

[lucia]

I have a blog with visitors from all over. So I don't want to block whole countries. I am currently requiring China and Israel to use a captcha at cloudflare. I needed to stop the incessant hammering while getting my scripts tweaked.  I'm going to lift that soon.  I can still catch Bezequint and have many of the Chinese spiders under control.