Is there a definitive IP range for PICSCOUT - i want to kill the spider!

[Robert Plant]

For your typical small non-retail business, would you really want anything cataloguing your site besides google?

[jot]

I digress... would blocking these IPs in .htaccess prevent them from impacting server resources?  I assume they would completely ignore robots.txt.

If blocked in .htaccess, blocking in robots.txt becomes superfluous. However, if you are blocking by IP and you miss an IP block or a robot changes IP ranges, it won't work. robots.txt might-- if the bot obeys it (which it may not.)

It was Getty Images.  I am already blocking a good range of IP addresses and two domain names with Picscout.

[lucia]

Depend what's "typical".  You might want yahoo search, bing and a few others. If you are a blogger you might want incoming pings from blogs. You might  want feed readers visiting.  If you sell advertizing, there might be a few bots that are worth letting visit. (I don't know what they are, but they may exist.)  Many people like the wayback. (Some don't.)

But really, there are a stupendous number of things visiting. I have no idea why *anyone* outside China would want baidu spider to visit. Similar for yandex but with Russia. I don't know why anyone who is not retail wants a "shopping bot" (the kind that find good prices on retail items for people to compare) to visit. 

So, I can't say "only google" categorically. But I'd say if you pick a 'mystery' bot that visits a lot at random, chances it does you no good exceed 90%.

[ws2001]

Here is what I have currently on Pic-Scout and others.  I have not updated my file in a while so others may have additional info.

One of our sites was pounded (mini DDOS) through multiple IPs. The IPs lead to Bezeq International-Ltd, and information on them lead me/us here.

- Hello - Wave -

Thought the following might help those who want to block Getty Image's rogue BI bots.

Bezeq International-Ltd
http://www.nirsoft.net/countryip/il.html  <-- no connection to them 
31.168.0.0-31.168.255.255
62.219.0.0-62.219.255.255
79.176.0.0-79.183.255.255
81.218.0.0-81.218-255.255
82.80.0.0-82.81.255.255
84.108.0.0-84.111.255.255
85.130.128.0-85.130.255.255
109.64.0.0-109.67.255.255
212.5.64.0-212.25.127.255
212.179.0.0-212.179.255.255
217.22.112.0-217.22.127.255

We were hit by bots from three of the above blocks, so it appears ok. But you should double-check. Especially since we're new here.

Best of luck.

[brianjclark]

There is a good bit on this forum about using your htaccess and basically saying to the bots "don't go to these areas please". If they ignore your warning you ban them!

http://forums.eukhost.com/newreply.php?do=newreply&p=87709

Just applied it to all my sites plus the little addition of emailing me when one is banned. Within 20 seconds of putting it in place, blinkin Googlebot came along, completely disregarded the robots.txt file and got itself banned. Well there ya go

[Robert Krausankas (BuddhaPi)]

There is a good bit on this forum about using your htaccess and basically saying to the bots "don't go to these areas please". If they ignore your warning you ban them!

http://forums.eukhost.com/newreply.php?do=newreply&p=87709

Just applied it to all my sites plus the little addition of emailing me when one is banned. Within 20 seconds of putting it in place, blinkin Googlebot came along, completely disregarded the robots.txt file and got itself banned. Well there ya go
[/quote

Googlebot DOES adhere to robots.txt, chances are good this bot was simply masking itself as googlebot , If it were me, I would be doing some further digging into this...IP address, ect....

[brianjclark]

Well it did resolve back to Google, but maybe they were going by what the robot.txt said a few hours ago. I can understand they might want to cache it etc.

"The IP 66.249.75.237 (crawl-66-249-75-237.googlebot.com) has been blocked for an invalid access attempt to a file, directory, or a scanning attempt."

I think I can let them off maybe once or twice and take them back out of my block list

[brianjclark]

Over time there seems to be a gradual trickle of bots poking about. Some tell the truth and say who they are while others quite clearly try to cover their tracks and make out they are Mozilla browsers. I have looked up the IPs to some of there and they resolve to things that quite clearly are not browsers, as there are no ways to link to the "booby traps" on my sites with a normal browser.

Remains an interesting subject!

## Banned IPs
Deny from 220.181.108.158
# baiduspider-220-181-108-158.crawl.baidu.com

Deny from 94.242.198.110
# Agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0)
# on 2013-07-03 (Wed) 02:38:11 IP: 94.242.198.110 (static-198-110.softronics.ch)

Deny from 27.45.240.84
# Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729))
# on 2013-07-03 (Wed) 06:33:30 IP: 27.45.240.84 (27.45.240.84) <-- China Unicom Guangdong Province Network

Deny from 27.45.240.82
# Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729))
# on 2013-07-07 (Sun) 12:45:13 IP: 27.45.240.82 (27.45.240.82) <-- China Unicom Guangdong Province Network

Deny from 175.42.90.137
# Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729))
# on 2013-07-09 (Tue) 06:29:28 IP: 175.42.90.137 (175.42.90.137) <-- China Unicom Fujian Province Network

Deny from 76.94.95.83
# Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0)
# on 2013-07-11 (Thu) 18:52:09 IP: 76.94.95.83 (cpe-76-94-95-83.socal.res.rr.com) <-- Road Runner / Time Warner Cable

Deny from 183.234.49.109
# Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729))
# on 2013-07-13 (Sat) 09:59:59 IP: 183.234.49.109 (183.234.49.109) <-- China Mobile communications corporation

Deny from 14.211.88.3
# Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729))
# on 2013-07-13 (Sat) 14:32:40 IP: 14.211.88.3 (14.211.88.3) <-- CHINANET Guangdong province network

Deny from 50.19.165.99
# Agent: Test Spider 0.2)
# on 2013-07-14 (Sun) 01:25:24 IP: 50.19.165.99 (ec2-50-19-165-99.compute-1.amazonaws.com)

Deny from 188.143.234.127
# Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1))
# on 2013-07-08 (Mon) 13:29:20 IP: 188.143.234.127 (188.143.234.127) <-- ToussaintDesaulniers-net

Deny from 192.114.71.13
# Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0)
# on 2013-07-10 (Wed) 17:40:51 IP: 192.114.71.13 (bzq-114-71-13.static.bezeqint.net) <-- Bastards

Deny from 89.75.96.207
# Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36)
# on 2013-07-11 (Thu) 08:39:09 IP: 89.75.96.207 (89-75-96-207.dynamic.chello.pl) <-- PL-UPC-20060222 in Warsaw

Deny from 5.10.83.73
# Agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/))
# on 2013-07-14 (Sun) 05:41:12 IP: 5.10.83.73 (5.10.83.73-static.reverse.softlayer.com) <-- Ahrefs Pte Ltd Singapore

Deny from 188.143.234.127
# Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1))
# on 2013-07-08 (Mon) 13:29:20 IP: 188.143.234.127 (188.143.234.127)<-- ToussaintDesaulniers-net

Deny from 192.114.71.13
# Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0)
# on 2013-07-10 (Wed) 17:40:51 IP: 192.114.71.13 (bzq-114-71-13.static.bezeqint.net)  <-- Bastards

Deny from 89.75.96.207
# Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36)
# on 2013-07-11 (Thu) 08:39:09 IP: 89.75.96.207 (89-75-96-207.dynamic.chello.pl) <-- PL-UPC-20060222 in Warsaw

Deny from 5.10.83.73
# Agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/))
# on 2013-07-14 (Sun) 05:41:12 IP: 5.10.83.73 (5.10.83.73-static.reverse.softlayer.com) <-- Ahrefs Pte Ltd Singapore

Deny from 37.59.202.77
# Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0.1) Gecko/20100101 Firefox/5.0.1)
# on 2013-07-07 (Sun) 14:08:04 IP: 37.59.202.77 (37.59.202.77) <-- Str Miron Costin, Brasov, France

Deny from 207.189.121.44
# Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2. Gecko/20100721 Firefox/3.6.
# on 2013-07-08 (Mon) 01:31:17 IP: 207.189.121.44 (207.189.121.44) <-- VIAWEST-NETBLOCK-207.189.96.0/19

Deny from 5.135.47.74
# Agent: Mozilla/5.0 (Windows NT 6.1; rv:2.0b7pre) Gecko/20100921 Firefox/4.0b7pre)
# on 2013-07-08 (Mon) 11:00:01 IP: 5.135.47.74 (5.135.47.74) <-- 2 rue Kellermann, France

--- end ---

[brianjclark]

The Bezeq International IPs now appear to start "192."   

That's a nice touch to anyone thinking that's ur local internal network (192.168 etc etc.)

[ws2001]

For Bezeq International add 192.114.64.0-192.114.79.255

Looks like some cough lawyer is reading the posts and passing the news of exposed IPs.

[Robert Krausankas (BuddhaPi)]

I knew it was only a matter of time, before new IP's started appearing...hence I simply block Israel as a whole..at the server firewall and the on the server.

[jbigfoot]

TinEye.com

You may also wish to exclude TinEye.com. TinEye is a program like pic Scout that crawls the web taking samples of images off of page webpages. It then stores these images and you can go to the website and upload an image and it will show you all other instances where it has found this image on the Internet. Getty has also been known to use TinEye as a quick and easy method of locating webpages in which to send demand letters to.

I've been using TinEye, on the recommendation of ELI, to check and see if images I have or others are using are public domain or copyrighted. What are the liabilities for me?

[Robert Krausankas (BuddhaPi)]

there are no "liabilities for you using tineye to research images...whoever can you explain how tineye is telling you whether images are public domain or copyrighted?

FYI: Copyright exists at the moment of creation, so a good majority of images are copyrighted..

[jbigfoot]

can you explain how tineye is telling you whether images are public domain or copyrighted?

I didn't explain myself fully on this one.

The images in question have been on my web site for years, as a tribute to my father. They are pictures of the Distinguished Flying Cross and the Air Medal. I didn't remember when or where I got them, so I used TinEye to see if these pictures appeared anywhere else on the web. They did; and the links that TinEye came back with were federal government web sites (if I recall), and these web sites clearly state that there is no copyright restriction for the use of these images.

[Robert Krausankas (BuddhaPi)]

can you explain how tineye is telling you whether images are public domain or copyrighted?

I didn't explain myself fully on this one.

The images in question have been on my web site for years, as a tribute to my father. They are pictures of the Distinguished Flying Cross and the Air Medal. I didn't remember when or where I got them, so I used TinEye to see if these pictures appeared anywhere else on the web. They did; and the links that TinEye came back with were federal government web sites (if I recall), and these web sites clearly state that there is no copyright restriction for the use of these images.

Thanks for the clarification! Getty has been known to accuse folks of infringement over public domain images.. If the images on those government sites you mention are indeed the same, you could sternly point this out to getty, and advise them if they continue their harassment, you will file complaints with the Washington Attorney General, and you will also invoice them for the time you have been wasting on this matter.