TL;DR — My database is designed to sleep when nobody's looking. Last month it never slept, and the bill more than doubled. The cause wasn't visitors — it was SEO crawlers (Ahrefs and friends) trawling thousands of pages a day to sell what they read. Sorting humans from machines turns out to be a core job of running a site now: search engines and genuinely useful AI stay welcome — ChatGPT sends me real travellers — extractors now get a 403 at the gate, and my suppliers never feel any of it.
---
My database bill is meant to be small. The database itself is not — it holds a proper catalogue of hotels across six continents and a price history for every night anyone has asked about — but on serverless Postgres you don't pay for what you store, you pay for compute, and mine is designed to sleep: a modest fixed floor to exist, and if nobody asks it anything for five minutes, it suspends and the meter stops.
Last month's bill was more than double that floor.
The difference — week after week of a database that never slept — was not my visitors. Analytics counts them without cookies, so I know exactly how many there were: a handful, most days. The difference was companies reading my site so that they could sell what they read to other people.
What the meter recorded
Two clients dominated August, and I can name them because they announced themselves in every request.
AhrefsBot is the crawler behind Ahrefs, an SEO-tools company. It fetched 13,606 social-share card images in one 24-hour window — each one a ~600ms server render of an image that exists so a human can paste a link into a group chat. No human was in a group chat. It was indexing my pages so that Ahrefs subscribers — a group that includes any competitor curious about me — can read about my site in their dashboards. Google does not consult Ahrefs. Being crawled by them has no effect on where I rank. The relationship is purely extractive: they take, they sell, I pay the compute.
Lightpanda is an open-source headless browser built for AI agents — a tool, not a company, so someone was driving it. Whoever it was rendered my hotel pages 18,945 times in 24 hours, executing the JavaScript on each one, firing thirteen thousand broken-image reports from pages no person was looking at. A headless browser defeats every cheap path I have: it can't be served the lightweight HTML a simple crawler gets, so every visit was a full server render, and every render woke the database.
Add the smaller regulars of the same class — Semrush and its peers run the same business — and the arithmetic is blunt. A database that suspends after five minutes of quiet needs only one request every four minutes to stay awake forever. The crawlers above managed one every few seconds.
The page is the cheap part
It's worth saying what a "page render" actually is, because the word makes it sound like the cost is the page. It isn't. The front end of this site — the HTML, the styling, the pictures — is a thin skin, and serving it is nearly free. The work lives underneath: a catalogue of hotels accreted from supplier feeds, a price tape recording what every room cost on every night anyone asked, cached live rates, exchange-rate handling, geocoding so "Back Bay, Boston" means the place and not the words. Almost everything I have built here is data machinery; the visible site is the last inch of it.
That's what a crawler spins up. When a person loads a hotel page, the machine earns its keep — the price it shows might become a booking. When a headless browser loads the same page nineteen thousand times, the whole apparatus runs — database, tape, caches — for a reader that will never book anything, on the crawler's schedule rather than any human's. A static brochure site could ignore all of this; the crawl would cost pennies. The sites that get hurt are exactly the ones that do real work under the page — and from the outside, the crawler can't tell the difference and doesn't care.
There's one cost I refuse to let them touch at all: my suppliers. Live room prices come from booking systems that meter every request, and my standing with those suppliers is part of the business. So the gate does double duty — a machine asking for a hotel page gets prices from my own cache, and a live supplier request only ever happens for a person actually choosing dates. A crawl wave can waste my compute; it cannot spend my supplier quota, and it cannot make me a noisy neighbour to the systems my bookings depend on. If you run anything that sits on top of a metered upstream, this is the fence to build first.
The part that took me too long to see
None of this shows up anywhere you'd normally look. The site was fast. The pages were fine. Human analytics looked normal, because these clients don't run analytics. The only place the cost surfaced was an invoice, weeks later, twice the size it should have been — and at small-site scale that's exactly the kind of number a busy person shrugs at.
That is the hidden part. A big company genuinely wouldn't notice this; the same crawlers hitting a big OTA are a rounding error inside a rounding error. On a small site the same absolute traffic is most of the bill. The costs of being trawled scale down much more slowly than revenue does, which means the smaller you are, the larger the share of your infrastructure you donate to companies that sell data about you.
What I did about it
I sorted crawlers into three tiers, by one question: does this client's reading benefit anyone but the client?
Search engines earn their keep. Googlebot reads my pages and sends me guests. It gets everything, served from a cache so it costs me almost nothing.
Honest tools get honesty back. Any bot that names itself gets cached copies too — I won't punish a crawler for telling the truth, because that only teaches crawlers to lie.
Extractors get nothing. AhrefsBot and its class are refused at the edge — a 403 before any page renders, before the database is ever consulted. They can still read robots.txt, which asks them politely to stop, because a compliant crawler that reads the rule is cheaper than one I have to keep refusing. My morning report now counts their knocks; the day the number reaches zero is the day they took the hint.
And for AI agents acting on behalf of an actual person, I built a proper front door — a machine-readable site description and an API that answers "where should I stay for the Boston Marathon?" in one cheap, structured call instead of fifty page renders. Agents that serve people are welcome. The door is only closed to clients that have already shown they're strip-mining.
Here's the detail that removes any sympathy I had left: that front door was already there while the trawling happened, published in the same robots.txt every one of these crawlers fetched. A machine that wanted my data could have asked for it in a form that costs me almost nothing — and they read the sign on the gate, then climbed over the fence and dug anyway. Rendering nineteen thousand pages when a structured endpoint sits advertised beside them isn't a client that couldn't do better. It's one that didn't care.
The wider point
The unglamorous truth is that sorting humans from machines is now a core part of running a website — as fundamental as backups. And it is emphatically not the same thing as being anti-AI. Some machines bring people: ChatGPT is one of my biggest referrers, and my first fully organic customer signup arrived through it — a real traveller, planning a real trip, sent by an AI that read my pages and pointed a person at them. That's the machine equivalent of Googlebot: it digs, and it pays for the privilege in humans. The line I draw isn't human-versus-machine. It's between clients that bring or serve people, and clients that only carry data away.
Every small independent site pays this tax. The SEO-tools industry runs on crawling millions of sites that get nothing back, and its costs land hardest on exactly the sites least able to absorb them — the personal projects, the independents, the one-person businesses running on serverless tiers where compute is metered by the second. If that's you: read your infrastructure bill, not your analytics. The visitors who cost you most are the ones no dashboard shows.
— Nick, Founder