When AI Bots Flood Your Website, Who Pays the Bill?

A business owner got a piece of software featured somewhere popular recently, and within hours their site was buckling. Not from real visitors. From AI bots, crawling every page, every link, every variation of a URL, over and over, within minutes of the post going up. Their hosting bill for the month arrived looking like a different business entirely.

This isn't a one-off. If your site or app gets mentioned anywhere with an audience, whether that's a forum, a newsletter, or a review site, you can expect a wave of automated traffic to follow. Some of it is AI crawlers hoovering up content for training or search. Some of it is scrapers built by nobody in particular, just running because someone left a script going. None of it cares whether your server can take the load.

Why this is different from a traffic spike you'd expect

A normal traffic spike, say from a marketing push, follows human patterns. People browse a handful of pages, read for a bit, leave. Bots don't behave like that. They can hit every page on your site in parallel, follow every link including ones meant for internal use, and repeat the whole crawl multiple times a day from different IP ranges. A site that comfortably handles a thousand real visitors an hour can fall over under a few hundred bot requests a minute, because the pattern of access is completely different from what the system was built to expect.

If your hosting bills by usage, by requests, by bandwidth, by compute time, that traffic costs you money whether or not a single customer ever sees your site. If your hosting is fixed but under-provisioned, the bots just slow everything down for the people who actually matter: your customers trying to check out, your staff trying to log in and get work done.

Robots.txt won't save you

A lot of business owners assume a robots.txt file sorts this out. It tells well-behaved crawlers what not to touch. The key word is well-behaved. Plenty of scrapers ignore it entirely, and the ones built for speed rather than politeness aren't checking your file before they start hammering your product pages or search endpoints. Robots.txt is a request, not a lock on the door.

What actually helps

I'd rather build a site so that a bot swarm is a non-event than hope one never turns up. A few things make the biggest difference:

• Cache anything that doesn't change per visitor. If a product page, blog post or pricing page looks the same to everyone, it should be served from a cache, not regenerated from the database on every single request. That's the difference between a bot crawl costing you nothing and costing you a server.
• Rate limit by behaviour, not just identity. A single IP address hitting your site thirty times a second is not a customer, whatever its user agent string claims. Rate limiting at the application or CDN level catches this without needing to identify every bot by name.
• Keep expensive operations away from public, unauthenticated routes. Search boxes, filters and anything that triggers a database query per keystroke are exactly what crawlers love to batter. If those routes are cheap to run, a crawl is a shrug. If they're expensive, a crawl is an incident.
• Put a CDN or reverse proxy in front of the site. This absorbs a huge amount of repeat traffic before it ever reaches your actual server, and it's usually the cheapest single thing you can do.
• Set a cost alert, not just a performance alert. If your hosting can scale automatically to handle load, that's good for uptime and bad for your bank balance if nobody's watching the bill. I've written before about why automation needs a hard spending limit, not just an alert, and the same logic applies to infrastructure that auto-scales under bot load. An alert tells you after the damage. A cap stops it happening.

The testing gap

Most businesses test their site or app against the traffic they expect: real users, browsing normally, at a volume roughly matching what marketing projects. That's sensible, but it misses the scenario that actually causes outages: a burst of automated, non-human traffic hitting every URL at once. I've covered this before when it comes to why software passes testing but still falls over on launch day, and bot traffic is one of the clearest examples. The load that breaks a system is rarely the load anyone planned for.

You can't block your way out of this entirely

It's tempting to think the answer is just blocking bots outright. Be careful here. Some of that traffic is legitimate search indexing you want, and some AI crawlers now drive real referral traffic through AI-powered search and chat tools. Block too aggressively and you disappear from places your customers are actually looking. The goal isn't to shut the door on everything automated, it's to make sure your site can absorb a crawl without the cost or the slowdown landing on you and your customers.

What to check now, before it happens to you

If you've never had a traffic spike you didn't expect, that's not evidence it won't happen, it's evidence you haven't been tested yet. Worth checking: does your hosting bill have any ceiling at all? Is anything on your public site hitting the database on every single request? Do you have any visibility into traffic spikes beyond "the site feels slow today"? These are quick things to check and far cheaper to fix in advance than to untangle during an actual spike, usually at the exact moment you can least afford the distraction.