Is your website blocking the AI engines? How to check in five minutes

We spend a lot of time measuring how agencies show up in AI answers, and most of the reasons an agency goes unnamed are the ones you would expect. Thin reviews. A Google Business Profile nobody has touched in three years. Rivals who have simply been recommended more often.

Then there is the dull one. Sometimes the engines never named the agency because they were never allowed to read the site. Nobody decided this. It arrived in a plugin, or a hosting default, or a tick box somebody pressed when blocking AI was in the news, and it has been quietly working ever since. It is worth five minutes of your morning to rule out.

The two places it hides

The first is a file called robots.txt, which sits at the root of your website and tells automated visitors what they may read. Yours is public, and so is everybody else's. Type your domain followed by /robots.txt into a browser and you will see it. What you are looking for is any line naming an AI crawler followed by Disallow. The names worth knowing, as at August 2026: OpenAI runs GPTBot for training, OAI-SearchBot for search, and ChatGPT-User when somebody in ChatGPT asks a question and it goes and fetches a page. Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User, split the same way. Perplexity runs PerplexityBot for search indexing and Perplexity-User when somebody's question sends it to fetch a page; it has no training crawler at all, and its documentation says neither is used to collect content for foundation models, so on Perplexity's own account blocking PerplexityBot costs you visibility and buys you nothing on training. Google runs Google-Extended, which is not the same thing as Googlebot and does not do what most people think, and Apple's equivalent is Applebot-Extended. CCBot, which turns up in a lot of copied and pasted robots files, is Common Crawl: a public dataset rather than an engine, which many model builders draw on.

The split matters more than the names. Blocking ClaudeBot stops Anthropic training on your content. It does not stop Claude-SearchBot indexing you, and it does not stop Claude-User fetching your valuation page for somebody who is asking about you right now. A site that blocks all three has removed itself from the answer. A site that blocks only the training bot has made a defensible choice and kept its seat. The second place is your hosting or your CDN, which is where this gets less obvious, because that setting is not in your robots.txt and you cannot see it by looking at your website.

The 15 September change worth knowing about

Cloudflare sits in front of a large share of the web, very possibly yours, and it has offered a one-click "Block AI bots" option for a couple of years. Plenty of people pressed it. It felt like the responsible thing to do at the time. On 1 July 2026 Cloudflare published its new AI traffic options and set a date of 15 September 2026 for two changes. The first is a new default for domains onboarding from that date, and Cloudflare softened it on 21 August: a new customer will get no blocks or disallows at all unless they choose them, and a customer who ticks "I monetize from pages with ads on this domain" at onboarding gets training set to Disallow, which writes a no-training line into robots.txt, so crawlers meeting Cloudflare's transparency conditions keep their search access while the ones that do not are still blocked at the edge. Cloudflare's changelog puts the same change slightly more widely: from that date new domains have both training and agent traffic blocked on pages that display ads, with search left allowed. Most agency sites carry no advertising, so this is not the part to worry about.

The second change is the one to pay attention to. From the same date, crawlers that do more than one job are judged on all of their behaviours, and the most restrictive rule wins. In Cloudflare's own words, multi-purpose crawlers "such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)". Read that again with your own site in mind. If anyone at your agency, or at your web company, ever ticked "Block AI bots", then from 15 September Googlebot could be on the blocked list too. Not your AI visibility. Your Google visibility. Cloudflare has said owners can change this in their security settings at any point before 15 September, and that it will keep notifying customers meanwhile. Those notifications go to the email address on the Cloudflare account, which in a lot of agencies is a developer who left, or an address nobody reads.

Two things to be clear about, because this is where the headlines have run ahead of the facts. The reason Googlebot is caught at all is that Cloudflare classifies it as a crawler that does more than one job. Google describes it differently, and points to a separate control, Google-Extended, for training. On this particular question it is Cloudflare's classification that decides whether the request gets through, which is why it matters even though Google would dispute the label.

And Cloudflare has already softened it. On 21 August it published Bot Preference Sync, under which operators whose crawlers do both search and training can avoid the block by meeting four transparency conditions, with the warning that crawlers which do not provide that transparency "will not get the benefit of the doubt". It also added a Disallow setting for training under which cooperating crawlers honour robots.txt and, in Cloudflare's words, "your Search visibility for cooperating crawlers is unaffected". Whether Googlebot ends up qualifying has not been confirmed either way. And note what is not changing: the legacy "Block AI bots" toggle is being replaced by the new controls, but an existing legacy selection is carried across rather than cancelled, which is precisely why it needs looking at. None of which changes what an agent should do, which is find out what their own setting is rather than assume.

So, the five minutes

One. Open yourdomain.co.uk/robots.txt and read it. If you see GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot, CCBot or anything similar sitting above a Disallow line, somebody made that decision. Find out who, and whether they would make it again today. One caveat if you are behind Cloudflare: Cloudflare announced Bot Preference Sync on 21 August, rolling out to all plans, which generates AI crawler lines in robots.txt from your dashboard settings and puts them above whatever is already there. It will be on by default for new customers, and existing ones running Cloudflare's managed robots.txt will be prompted to review and confirm. It still had not reached Cloudflare's changelog when we last checked on 1 September, so watch for it rather than assume it is already there. So check the dashboard as well as the file: the line you are reading may have been generated rather than chosen, and editing the file by hand will not remove it unless you turn the sync off. Two. Ask whoever hosts your site one question: are we behind Cloudflare, and is "Block AI bots" or any AI bot blocking switched on? If the answer is yes, ask them to look at it this week rather than nearer the deadline. Three. If you have a security plugin on a WordPress site, check whether it has an AI or bot blocking feature and what it is set to, because several enable something by default and describe it as protection. Four. Check you are not blocking the fetch bots, and check that at the firewall as well as in robots.txt. Those are the ones that turn up when a real person has asked a real question about you, which is the closest thing to a warm lead in this whole subject, and a text file does not reliably control them: OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them, precisely because a person asked for the page. Anthropic's Claude-User does respect robots.txt. In practice it is a CDN or firewall rule that stops these, not a line in a file.

The Google one that confuses everybody

Google-Extended is not a search setting. It governs whether your content can be used to train the models behind Gemini apps and the Vertex AI API for Gemini, and for grounding in Gemini apps and in Grounding with Google Search on Vertex AI. Google's own documentation says it does not affect inclusion in Search and is not a ranking signal. Which means blocking it does not remove you from AI Overviews or AI Mode, because those are features inside Google Search rather than separate products, and they draw on the ordinary index. That used to mean you could not hide from Google's AI answers and stay in Google, because the only other levers were the snippet controls, nosnippet, data-nosnippet and max-snippet, or noindex, and every one of those costs you something in ordinary search results too.

That changed in June 2026, and UK sites got it first. The Competition and Markets Authority imposed a binding conduct requirement on Google on 3 June, and Google responded with a site-level control in Search Console that takes a property out of AI Overviews, AI Mode and the AI features in Discover while leaving ordinary Search results alone. It is called the Search generative AI control and it sits under Settings in Search Console. It is property-level for now: the CMA's notice gives Google six months from 3 June for a directory-level control and nine months for a page-level one, which works out at 3 December 2026 and 3 March 2027. The rollout began with a subset of UK site owners, and Google said on 31 August 2026 that it had reached all websites worldwide, so it should now be in your Search Console. Google's own help page says content is excluded within a day or two of the setting going live.

So the honest position has moved. You can now opt out of Google's AI answers without leaving Google, which is worth knowing about. Whether an estate agent should is a different question, and we would say almost certainly not: an AI answer that names you is the one that puts you in front of somebody already asking who to trust. The choice worth making is not whether to appear, it is whether what appears is accurate and current, which is a content and structured data question rather than a blocking one.

Where we land on it

Our own position, for what it is worth, is that search and fetch crawlers should be allowed to read an estate agency website, because those are the ones that put your name in front of somebody who is asking. Training crawlers are a fair debate, and we do not think an agency is wrong to block them. What is worth avoiding is blocking all of it by accident, then paying somebody to work out why the engines never mention you. If the engines are quiet about you and your robots.txt is clean, the reasons are usually elsewhere, and we have written up the five that come up most.