TL;DR — Ten days after I fenced off the SEO crawlers, my database still never slept: awake for 259 of the month's first 260 hours, on course to cost double its floor again. The names had changed. Meta's AI-training crawler was running my site about fifteen thousand times a day, executing the pages and writing tens of thousands of review rows for a reader that will never book. Amazonbot was walking ten thousand pages a day for an Alexa that books hotels through Expedia. Nine more names are refused at the gate now. The rule is the same one as before: if a machine never sends a person back, it doesn't get a render.
---
On the first of the month I wrote about the crawlers that cost me money: an SEO index and a headless browser that between them kept my database awake all of August, on a site with a handful of human visitors a day. I refused them at the edge and expected the meter to slow down.
It didn't. This month the database has been awake for 259 of the first 260 hours. The bill is on course to be double its floor again. When I went looking, the two clients I'd named had gone. Their places had been taken.
Who replaced them
Every request carries a name, and my morning report samples them. On a typical day this week, rounded:
- Meta's AI-training crawler, `meta-externalagent`: about fifteen thousand page renders.
- Amazonbot: about ten thousand.
- SE Ranking's backlink crawler: three to five thousand.
- Majestic's MJ12bot and Semrush: around a thousand each.
One name is missing from that list on purpose, and it's the biggest: ClaudeBot, Anthropic's crawler, read about twenty-two thousand pages the same day. It stays welcome. Claude is an answer engine that cites and links the pages it draws on, which puts it in the same tier as ChatGPT, and its renders come from the crawler cache. I've put it in the chart because leaving it out would make the picture look tidier than it is. Set against all of that, the search engines and ChatGPT's own fetchers came to about a thousand requests between them.
Meta ran my site as if it were a guest
There are two Meta crawlers, and it matters which is which. `facebookexternalhit` fetches a page once when someone shares a link into WhatsApp, Facebook or Instagram, so the preview card shows a picture and a title. It's welcome; that's a person sharing my site with another person. `meta-externalagent` is different. Meta's own description says it exists to train the models behind Meta AI and Llama and to index content for Meta's products.
It also executes JavaScript, which is the part that hurt. A simple crawler reads a page and stops. This one *runs* it: my own front end, which on a hotel page asks for the guest reviews, which — if they're more than a month old — asks a supplier feed for fresh ones and writes them to disk. So for every hotel it walked, the crawler operated my site exactly as a guest would, and the site did what it does for guests. In one day that came to about two and a half thousand review refreshes and sixty-odd thousand rows written to a table that was already the second biggest I have. All of it for a reader with no dates, no card, and no intention.
What comes back? I've looked. No WhatsApp answer links to a hotel page. No Instagram surface cites a source. The reading is one-directional by design.
What Amazonbot is for
Amazonbot is Amazon's web crawler, and Amazon is clear about its purpose: content it gathers may be used to train Amazon's models and to improve Alexa. In June Amazon split its crawling into three agents. `Amazonbot` is the training one. `Amzn-SearchBot` feeds search-style answers in Alexa, and `Amzn-User` fetches a page when a specific person asks Alexa something. Amazon says the last two are not used for training, and that all three now honour robots.txt.
That split is honest and I've respected it: only the training crawler is refused. The two that could conceivably send a person to my site are still let in, cached like any other named bot. If Alexa ever answers "find me somewhere near the ground on Saturday" with a link, I'd like to be in it.
But I don't expect it to. This year Amazon announced hotel search and booking inside Alexa+, and it runs on Expedia's inventory of seven hundred thousand properties, with the booking closed inside Amazon's own wallet. That is a partnership feeding a walled catalogue, not an assistant browsing the open web. Nothing Amazonbot reads on my site earns me a place in it. The crawl trains the model; the money goes through the partner.
The SEO indexes, again
SE Ranking, Majestic and Semrush run the same business as Ahrefs, the company in the first note: crawl the web, index who links to whom, sell the index to subscribers. Google doesn't consult any of them. Being read by them has no effect on where I rank. They are the clearest case of the lot, and they were still walking the site a week after the gate went up, because the gate only had one of their names on it.
What I changed
Nine more names are refused at the edge — a 403 before any page renders, before the database is consulted — and listed in robots.txt so the compliant ones stop asking at all. Amazon says its crawlers now honour robots.txt, so for them the polite request should be enough; the 403 is for the rest.
Two changes underneath matter more than the list. A machine reader now gets the reviews I already have on file and never triggers a supplier fetch or a write — a crawler cannot book, so it never earns one. And when a real guest does trigger a refresh, the rows go in as one batch instead of one at a time. That last one was overdue regardless; the crawler just made it visible.
The test I apply hasn't changed since the first note, and I'll write it down plainly because it's the whole policy: does this reader ever send a person back? Search engines do. ChatGPT does — it's still my biggest referrer. Share-card fetchers do, one preview at a time. Backlink indexes don't. Training crawlers don't. The first group gets everything, from a cache. The second gets a closed door.
What it can't fix
I should be honest about the limit. The database still won't sleep. Even the welcome crawlers walk enough pages that the five quiet minutes it needs never arrive — ClaudeBot alone was a page every four seconds — and I'm not going to refuse the machines that send people in order to save a few pounds. The floor is the price of being readable. What the gate does is stop me paying *above* the floor for readers who give nothing back — and stop my site doing real work, with real supplier feeds, at the request of a machine that will never book.
The wider point
Two of the largest companies on earth read my site more than every search engine put together this month, by a factor of about twenty, and both are building products that will never link to it. Meta trains a model. Amazon books hotels through Expedia. The open web is the training set; the revenue runs through partners and closed apps; the compute for the reading is paid by whoever owns the pages. On a big site that's a rounding error. On a one-person site it's most of the infrastructure bill.
If you run something small: pull your server logs, not your analytics, and look for these five names. Put the ones that honour robots.txt in there. Refuse the rest at the edge. And keep the door open for the machines that pay in people — they're the reason the rest of the traffic is worth having at all.
— Nick, Founder