Is your website blocking the AI engines? How to check in five minutes

We spend a lot of time measuring how agencies show up in AI answers, and most of the reasons an agency goes unnamed are the ones you would expect. Thin reviews. A Google Business Profile nobody has touched in three years. Rivals who have simply been recommended more often.

Then there is the dull one. Sometimes the engines never named the agency because they were never allowed to read the site. Nobody decided this. It arrived in a plugin, or a hosting default, or a tick box somebody pressed when blocking AI was in the news, and it has been quietly working ever since. It is worth five minutes of your morning to rule out.

The two places it hides

The first is a file called robots.txt, which sits at the root of your website and tells automated visitors what they may read. Yours is public, and so is everybody else's. Type your domain followed by /robots.txt into a browser and you will see it. What you are looking for is any line naming an AI crawler followed by Disallow. The names worth knowing, as at August 2026: OpenAI runs GPTBot for training, OAI-SearchBot for search, and ChatGPT-User when somebody in ChatGPT asks a question and it goes and fetches a page. Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User, split the same way. Perplexity runs PerplexityBot for search indexing and Perplexity-User when somebody's question sends it to fetch a page; it has no training crawler at all, and its documentation says neither is used to collect content for foundation models, so on Perplexity's own account blocking PerplexityBot costs you visibility and buys you nothing on training. Google runs Google-Extended, which is not the same thing as Googlebot and does not do what most people think, and Apple's equivalent is Applebot-Extended. CCBot, which turns up in a lot of copied and pasted robots files, is Common Crawl: a public dataset rather than an engine, which many model builders draw on.

The split matters more than the names. Blocking ClaudeBot stops Anthropic training on your content. It does not stop Claude-SearchBot indexing you, and it does not stop Claude-User fetching your valuation page for somebody who is asking about you right now. A site that blocks all three has removed itself from the answer. A site that blocks only the training bot has made a defensible choice and kept its seat. The second place is your hosting or your CDN, which is where this gets less obvious, because that setting is not in your robots.txt and you cannot see it by looking at your website.

What changed on 15 September

Cloudflare sits in front of a large share of the web, very possibly yours, and it offered a one click "Block AI bots" option for a couple of years. Plenty of people pressed it. It felt like the responsible thing to do at the time. In July 2026 Cloudflare said that from 15 September a crawler doing more than one job would be judged on all of its behaviours, with the most restrictive rule winning, and it named Googlebot, Applebot and BingBot as crawlers that would be blocked for anyone who had selected to block training, including through the legacy Block AI bots service. Your Google visibility, in other words, not only your AI visibility.

What Cloudflare actually did on the day was gentler than that, and better. It split the control into three, Search, Training and Agent, acting independently, and added a Training option called Disallow AI Training which publishes a preference into robots.txt while leaving accountable crawlers free to carry on indexing you for search. Existing selections were migrated into it rather than enforced as blocks: in Cloudflare's words, previous Training selections of Block, or Block on pages with ads, migrate to Disallow AI Training, and its answer on what customers need to do is "Nothing, in almost every case." It also named the operators meeting its four transparency conditions, with Apple, Google and Microsoft among them, which settles a question that was open when this article was first written.

The reason to check your setting anyway is that Block is still there, and it still does what the July warning described. Cloudflare's own description of it is that it will stop Applebot, Bingbot and Googlebot reaching your site, search included. It is now a deliberate choice rather than an inherited one, sitting next to the option most people actually want and sounding like the stronger of the two. The whole thing is written up in what changed, and the switch that still costs you Google.

So, the five minutes

One. Open yourdomain.co.uk/robots.txt and read it. If you see GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot, CCBot or anything similar sitting above a Disallow line, somebody made that decision. Find out who, and whether they would make it again today. One caveat if you are behind Cloudflare: Cloudflare announced Bot Preference Sync on 21 August, rolling out to all plans, which generates AI crawler lines in robots.txt from your dashboard settings and puts them above whatever is already there. It will be on by default for new customers, and existing ones running Cloudflare's managed robots.txt will be prompted to review and confirm. It is in use now, because the Disallow AI Training setting introduced on 15 September works by publishing a preference into that file. So check the dashboard as well as the file: the line you are reading may have been generated rather than chosen, and editing the file by hand will not remove it unless you turn the sync off. Two. Ask whoever hosts your site one question: are we behind Cloudflare, and is "Block AI bots" or any AI bot blocking switched on? If the answer is yes, ask them which of the three settings is actually in force, because Block and Disallow AI Training are not the same thing and only one of them keeps you in Google. Three. If you have a security plugin on a WordPress site, check whether it has an AI or bot blocking feature and what it is set to, because several enable something by default and describe it as protection. Four. Check you are not blocking the fetch bots, and check that at the firewall as well as in robots.txt. Those are the ones that turn up when a real person has asked a real question about you, which is the closest thing to a warm lead in this whole subject, and a text file does not reliably control them: OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them, precisely because a person asked for the page. Anthropic's Claude-User does respect robots.txt. In practice it is a CDN or firewall rule that stops these, not a line in a file.

The Google one that confuses everybody

Google-Extended is not a search setting. It governs whether your content can be used to train the models behind Gemini apps and the Vertex AI API for Gemini, and for grounding in Gemini apps and in Grounding with Google Search on Vertex AI. Google's own documentation says it does not affect inclusion in Search and is not a ranking signal. Which means blocking it does not remove you from AI Overviews or AI Mode, because those are features inside Google Search rather than separate products, and they draw on the ordinary index. That used to mean you could not hide from Google's AI answers and stay in Google, because the only other levers were the snippet controls, nosnippet, data-nosnippet and max-snippet, or noindex, and every one of those costs you something in ordinary search results too.

That changed in June 2026, and UK sites got it first. The Competition and Markets Authority imposed a binding conduct requirement on Google on 3 June, and Google responded with a site-level control in Search Console that takes a property out of AI Overviews, AI Mode and the AI features in Discover while leaving ordinary Search results alone. It is called the Search generative AI control and it sits under Settings in Search Console. It is property-level for now: the CMA's notice gives Google six months from 3 June for a directory-level control and nine months for a page-level one, which works out at 3 December 2026 and 3 March 2027. The rollout began with a subset of UK site owners, and Google said on 31 August 2026 that it had reached all websites worldwide, so it should now be in your Search Console. Google's own help page says content is excluded within a day or two of the setting going live.

So the honest position has moved. You can now opt out of Google's AI answers without leaving Google, which is worth knowing about. Whether an estate agent should is a different question, and we would say almost certainly not: an AI answer that names you is the one that puts you in front of somebody already asking who to trust. The choice worth making is not whether to appear, it is whether what appears is accurate and current, which is a content and structured data question rather than a blocking one.

Where we land on it

Our own position, for what it is worth, is that search and fetch crawlers should be allowed to read an estate agency website, because those are the ones that put your name in front of somebody who is asking. Training crawlers are a fair debate, and we do not think an agency is wrong to block them. What is worth avoiding is blocking all of it by accident, then paying somebody to work out why the engines never mention you. If the engines are quiet about you and your robots.txt is clean, the reasons are usually elsewhere, and we have written up the five that come up most.