You are here:  » robots.txt


robots.txt

Submitted by support on Tue, 2006-02-07 19:28 in

ia_archiver is currently busy taking up bandwidth on WebPriceCheck.co.uk; so I would recommend anyone running Price Tapestry to block this particular crawler. I don't normally block ia_archiver on my other sites because it can come in handy; but on a site with thousands of pages I don't see any point in having pages stored by archive.org.

I don't include a robots.txt in the distribution because it's down to personal preference what you do and don't block, but here's what i'm using at the moment:

http://www.webpricecheck.co.uk/robots.txt

More info on ia_archiver:

http://www.archive.org/about/exclude.php