On the 16th and 17th, I got crawled by a very poorly written bot. It sucked down over 5GB alone on the 17th as it got caught in some sort of recursive loop. It requested a single photo 37,969 times… not cool.
I’d like to set a sane transfer limit (10-20mb) per visitor/day, unfortunately it doesn’t look like there is a mod_throttle for Apache 2 yet. I’ve banned the IP, but if anyone has any other bright ideas, send them over.
That absolutely, utterly sucks, man.
Try bwshare (for Apache 1.3.x and 2.0.44):
http://www.topology.[...]/README.html
I’ve found it to be much more stable than mod_throttle, although it has less features.
Whoa! 5GB?! I watch some of the bots crawling from Google, MSN and Yahoo. They usually take minimal bandwidth from my sites. Nothing like what you’re talking about.
Tim