On January 1, I received a bill from my web hosting provider for a bandwidth overage for $155. I’ve never had this happen before. For comparison, I pay about $400/year for the hosting service, and usually the limitation is disk space.
Turns out, on December 17, my bandwidth usage jumped dramatically - see the attached graph.
I run a few different sites, but tech support was able to help me narrow it down to one site. This is a hobbyist site, with a small phpBB forum, for a very specific model of motorhome that hasn’t been built in 25 years. This is NOT a high traffic site; we might get a new post once a week…when it’s busy. I run it on my own dime; there are no ads, no donation links, etc.
Tech support found that AI bots were crawling the site repeatedly. In particular, OpenAI’s bot was hitting it extremely hard.
Here’s an example: There are about 1,500 attachments to posts (mostly images), totaling about 1.5 GB on the disc. None of these are huge; a few are into the 3-4 megabyte range, probably larger than necessary, but not outrageously large either. The bot pulled 1.5 terabytes on just those pictures. It kept pulling the same pictures repeatedly and only stopped because I locked the site down. This is insane behavior.
I locked down the pictures so you had to be logged in to see them, but the attack continued. This morning I took the site offline to stop the deluge.
My provider recommended implementing Cloudflare, which initially irritated me, until I realized there was a free tier. Cloudflare can block bots, apparently. I’ll re-enable the site in a few days after the dust settles.
I contacted OpenAI, arguing with their bot on the site, demanding the bug that caused this be fixed. The bot suggested things like “robots.txt”, which I did, but…come on, the bot shouldn’t be doing that, and I shouldn’t be on the hook to fix their mistake. It’s clearly a bug. Eventually the bot gave up talking to me, and an apparent human emailed me with the same info. I replied, trying to tell them that their bot has a bug to cause this. I doubt they care, though.
I also asked for their billing address, so I can send them a bill for the $155 and my consulting fee time. I know it’s unlikely I’ll ever see a dime. Fortunately my provider said they’d waive the fee as a courtesy, as long as I addressed the issue, but if OpenAI does end up coming through, I’ll tell my provider not to waive it. OpenAI is responsible for this and should pay for it.
This incident reinforces all of my beliefs about AI: Use everyone else’s resources and take no responsibility for it.
Yup… I just had to read your title to know how it happened. In fact more than a year ago at OFFDEM (the off discussion parallel to FOSDEM in Brussels) we discussed how to mitigate such practices because at least 2 of us self-hosting had this problem. I had problem with my own forge because AI crawlers generate archives and that quickly generate quite a bit of space. It’s a well known problem that’s why there are quite a few “mazes” out there or simply blocking rules for HTTPS or reverse proxies.
AI hype is so destructive for the Web.
There has to be a better way to do this. Like using a hash or something to tell if a bot even Need to scrape again.
No doubt there are better ways … but I believe pure players, e.g. OpenAI or Anthropic, or resellers who get paid with scaling, e.g AWS, equate very large scale with moat. So they get so much funding that they have ridiculous computing resources, probably way WAY cheaper for “old” cloud (i.e. anything but GPUs) than new cloud (GPUs) so basically they put 0 effort to optimize anything. They probably even brag about how large their “dataset” is despite it being full of garbage data. They don’t care because in their marketing materials they claim to train over Exabytes of data or whatever.