Thoughts on Picscouts ignoring Robots.txt

[Robert Krausankas (BuddhaPi)]

Here's some reading regarding legal ramifications of ignoring robots.txt... basically what I'm getting at is stopping picscount from "trespassing"...and abiding by rules set forth by webmasters. my heads all a jumble at this point..

Keep in mind some/most of this is fairly old, but I wondering if any progress has been made from a legal standpoint..

http://pubs.cli.vu/pub238.php - what caught my eye here was section #5

http://en.wikipedia.org/wiki/Web_scraping#Legal_issues - again could this be enforceable??

http://www.chillingeffects.org/linking/faq.cgi#QID460 -

http://en.wikipedia.org/wiki/Trespass_to_chattels - interesting concept...

Sorry for the really long reads, i'm sure none of this would stop the letters, but if picscout itself can be stopped, GI would then have to resort to manually looking for infringments, which would crastically slow the process or not even allow it to be profitable anymore.. not to mention the money GI just spent purchasing picscout..

[Matthew Chan]

My practical position is if anyone is depending on robots.txt to ensure any privacy of their content on web servers, it is a foolhardy exercise.

That is the equivalent of telling strangers not to go inside or intrude into your car if you are not there but the doors are unlocked.  Or telling strangers not to enter your house but you leave your doors unlocked.

Robots.txt was mostly designed as a "courtesy" measure to instruct search engines and other benevolent Internet entities to not scour, scrape, or index specific locations and content on your web server.

I don't care what the law says about it because it is contingent upon legally enforcing it (which is not always easy or cheap).

Unfortunately nowadays, if you want to ensure people stay out of your car or home, you have to lock your doors each and every time.  The same goes with content on your webserver.  If you want someone to stay out, lock it down. Robots.txt only "work" for people who want to follow the rules of etiquette. I think we can clearly establish that the stock photo companies don't follow that otherwise they would engage in cease or desist letters or make more reasonable settlement amounts at least.

If you want to keep robots, search engines, PicScout, or any other person/entity out of your content, you have to secure your servers with at least password access (if not more).

Matthew

[Robert Krausankas (BuddhaPi)]

I agree 1--% that robots.txt cannot be relied upon, and while I also agree we need to lock our doors, I think there needs to be something changed in the laws to make all agents obey robots.txt or face the music..

If you lock your door, and I come behind you and break in, and you're neighbor ( or someone else) sees me, chances are high that i'll be busted for B & E..
Yes enforcing it would be a difficult proposition, but if something were to be put in place, it's easy enough to look at logs, ect..

[Matthew Chan]

I am sure you understand where I am coming from when I say I will be making zero time and effort in trying to change legislation.  That means potentially dealing with Congress and other legislators.  And the political, law-making game is left for those who have the stomach for it.

We can sue different people for all kinds of reasons but there is a line where you have to decide whether it is worth it.  I make decision like that nearly every other month.

I know this might upset some people but I have begun losing sympathy for many extortion letter recipients mainly because of their attitudes, ignorance, or lack of spine.  For some, I don't really feel sorry they had to pay because they didn't care enough about themselves to do anything about it.  They were almost asking to be taken advantaged of.  If paying someone to go away is the way some people want to operate even when they have a good case, then that is their right.  I just don't want to hear them griping how they lost the money then.

Matthew

[Robert Krausankas (BuddhaPi)]

I completely understand, and it's not something I have the stomach or patience for..but maybe someone will  come along and jump in, I think it makes for not only good dialog, but also gets more info out there with perhaps new options...

[Crying-out-Loud]

not sure if this helps - but this site lists all IP addresses by country:

http://www.countryipblocks.net/country-blocks/htaccess-deny-format/

[Biopharm]

SO I guess that this plugin http://wordpress.org/extend/plugins/pc-robotstxt/ for Wordpress designer is useless then? It claims you can control images and other indexing from your website.

[Robert Krausankas (BuddhaPi)]

yup pretty much useless, the plug-in is just creating a regular robots.txt file, and giving you the option to change/edit via the wordpress admin dashboard... You'd have much more control by creating your own.

[Biopharm]

yup pretty much useless, the plug-in is just creating a regular robots.txt file, and giving you the option to change/edit via the wordpress admin dashboard... You'd have much more control by creating your own.

Thanks. I created my own robot.txt file and have it uploaded to all my websites. I'll be submitting them to wayback.

[Robert Krausankas (BuddhaPi)]

you can request removal from archive.org as opposed to submitting it and waiting longer..

http://www.archive.org/about/exclude.php

Make sure to include the disallow tag for the domaintools user agent, as they abide by robots.txt

[Biopharm]

you can request removal from archive.org as opposed to submitting it and waiting longer..

http://www.archive.org/about/exclude.php

Make sure to include the disallow tag for the domaintools user agent, as they abide by robots.txt

can you share the exact coding you recommend in the robots.txt file?
I currently have this:

User-agent: ia_archiver
Disallow: /

[Oscar Michelen]

I'll let the technical folks try and figure out how bets to make this workable Thanks for all the great info.  I just love this forum!