The AI crawler that asked for a private key

Last month, on a small estate agency website we look after, something calling itself an AI crawler asked for a copy of a private SSH key.

Not once. Eighty-nine times.

That is the sentence this whole article exists for, so it is worth sitting with for a moment. A private key is the thing that lets someone log in to a server as though they were you. It has no business being requested by anything, let alone by a visitor introducing itself as an artificial intelligence that has come to read your property listings.

The dull question that found it

We log AI crawler traffic on client sites, in the way that anybody with a modern site might. The usual output of that is a number: this many visits from this many engines this month, up or down on last.

What we did differently was ask a duller question than the number invites. Not how many hits, but which pages they were actually taking.

The answer was not pages. Over thirty days, from requests arriving with PerplexityBot in the user agent, the requested paths included /.ssh/id_rsa, /.ssh/authorized_keys, /.ssh/config, /.htpasswd, /backend/.env, /.env.backup, /secrets.yml, /.aws/config, /.s3cfg, /.npmrc, /.git/HEAD, /docker-compose.yaml, /terraform.tfstate and /rclone.conf. Three hundred and forty five distinct paths in all, across seven thousand three hundred and fifty seven requests.

There was also a path nobody could ever link to, an invented string of characters followed by the site's own domain name. That is a fingerprinting trick: you ask for a URL that cannot exist, and how the site answers tells you what it is built on and how it handles things it does not have.

None of the files above exist on that site. It is a modern build with no WordPress and no PHP anywhere on it, so the entire list was a miss. That is a relief rather than a defence, because the request pattern is the same whoever it lands on.

What the pattern actually is

Read the list again and notice what it is not. It is not a crawler that has misunderstood a sitemap. It is not an engine indexing content in an unusual order. Every one of those paths is a file that, if it existed and was readable, would hand over credentials of some kind: a login key, a password file, a set of cloud keys, a database connection string, the state file describing an entire piece of infrastructure.

That is not reading. That is checking the doors.

The question nobody asks

Which raises something we had not been asking, and neither, as far as we can tell, had our tooling. How do you know a visitor is who it says it is?

A user agent, the little bit of text that says "I am GPTBot" or "I am ClaudeBot", is not a credential. It is a self-declaration typed into a request by whoever sent it. Anyone can put any name in that field. It costs nothing and it is checked by nothing, unless somebody decides to check it.

There is a way to check, and it is not obscure. The AI vendors publish the ranges of internet addresses their crawlers genuinely operate from, precisely so that site owners can verify a caller. OpenAI publishes a file per identity, gptbot.json, searchbot.json and chatgpt-user.json, so that a training crawler, a search crawler and a live fetch on behalf of a user can be told apart. Anthropic publishes its ranges at claude.com/crawling/bots.json. Perplexity publishes perplexitybot.json and perplexity-user.json. Google publishes common-crawlers.json alongside separate files for its user-triggered fetchers. Apple publishes at search.developer.apple.com/applebot.json, Amazon at developer.amazon.com/amazonbot/ip-addresses/, and Bing at bing.com/toolbox/bingbot.json, though that last one is stamped January 2024, so it is worth pairing with a reverse DNS lookup rather than trusting alone.

So we added address verification to our own logging. Every claimed identity gets checked against the ranges its vendor publishes, and anything that fails is recorded as failing rather than counted as a visit.

What the check found

Over the same thirty days, on the same site, 1,054 page requests claimed an AI crawler identity and failed that address check.

The breakdown is the part worth sitting with. Requests calling themselves GPTBot: 149 failed the check, against 107 that passed it. Claude-User: 128 failed, against 34 that passed. ChatGPT-User was the other way round, 353 failed against 768 genuine.

For two of those three identities, in that window, on that site, the traffic wearing the name outnumbered the traffic entitled to it.

One caution on those figures, because it matters. Our forged detection was added after the logging itself, so any all-time column understates the problem twice over. Everything here is thirty days, on one site, and it is not a survey of the internet. It is one site, measured properly.

The limit of what we know, which is the actual point

Now the uncomfortable part, and we would rather print it than leave it out.

We verify OpenAI's and Anthropic's crawlers today. We do not yet verify every vendor. Which means that for those requests at the top of this article, the ones asking for the private key, we cannot tell you whether that was genuinely Perplexity's crawler behaving in a way we would not expect, or somebody else entirely wearing the name as a coat.

We are not going to accuse a company on evidence we do not have. What we can say is narrower and, we think, more useful: nobody looking at a normal traffic report could tell either. The name in the log is the only thing most tooling has to go on, and the name is the one part of the request that anybody can type.

That is the whole argument. Not that AI crawlers are dangerous, because mostly they are not and we want them reading our clients' sites. The point is that a name is not an identity until something checks it, and the check exists, published by the vendors themselves, sitting there unused.

What we would suggest

If somebody is reporting AI crawler numbers to you, ask one question: are those verified against the published address ranges, or are they counts of what the requests claimed to be? Both are legitimate things to measure. They are not the same measurement, and only one of them can tell you whether a name has been borrowed.

And if you are wondering whether the engines can read your site at all, which is the more common problem by a distance, that is a different check and a five minute one. We wrote it up separately in how to tell whether your website is turning the AI engines away.

None of this came out of a security review. It came out of asking what was actually being requested rather than how much of it there was, on an ordinary Tuesday, on an ordinary client site. Most of what is worth knowing turns up that way.