Field note
AI Crawlers, robots.txt and llms.txt for SMEs
Training crawlers and search crawlers are different bots; block the first if you want, but think hard before blocking the second.
Training crawlers and search crawlers are different bots; block the first if you want, but think hard before blocking the second.
Why one "block AI" switch is the wrong question
Most small business owners ask whether they should block AI. The better question is which job each bot does. The main AI providers now publish separate user agents for model training, for their search features, and for fetching a page because a user asked a question. Blocking all of them together can remove you from the answers your prospects are reading, while doing little to protect anything that was not already public.
For a UK consultancy, trades business or professional practice, the public site exists to be found. Service pages, location pages and contact details are marketing. The content you might reasonably keep out of training is original long-form work such as guides, reports or course material.
Know the main user agents
OpenAI documents four bots. GPTBot is used for training, and OpenAI says disallowing it "indicates a site's content should not be used in training generative AI foundation models". OAI-SearchBot surfaces sites in ChatGPT search, and sites opted out of it "will not be shown in ChatGPT search answers". ChatGPT-User visits pages when a user asks, and OpenAI notes that robots.txt rules may not apply to these user-initiated visits. OAI-AdsBot checks pages submitted as ChatGPT ads.
Anthropic documents three. ClaudeBot collects content that could contribute to model training. Claude-SearchBot improves search result quality. Claude-User fetches pages when someone asks Claude a question. Anthropic says its bots honour robots.txt and also support the non-standard Crawl-delay directive.
Google-Extended is not a separate crawler but a robots.txt token. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Blocking it does not remove you from AI Overviews, because those are part of Search and use Googlebot.
PerplexityBot and other providers publish their own names. Check each provider's current documentation before adding a rule, because names and purposes change.
A sensible robots.txt for most SMEs
If you want to opt out of training but stay visible in AI search, a starting point looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
Remember three limits. Robots.txt is a request, not access control; anything confidential should sit behind authentication. Rules only affect future crawling, not content already collected. And user-triggered fetchers may not follow the file at all. If a page must not be read, do not publish it.
What llms.txt is and is not
llms.txt is a proposal first published by Jeremy Howard on 3 September 2024, with a second version dated 10 August 2026. It is a markdown file at /llms.txt that gives language models a short summary of the site and links to clean markdown versions of key pages. The proposal site describes it as not an official web standard.
It is a signpost, not a permission file. It does not block or allow any crawler, and it is not a replacement for robots.txt. Google's guidance on AI features says you do not need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode. For most SMEs, llms.txt is optional. If your site has substantial documentation, a well-maintained one costs little. If it only has ten pages, spend the time on those pages instead.
Cloudflare's AI crawler controls
If your site runs behind Cloudflare, the dashboard offers AI bot policies under Security settings. Cloudflare classifies AI bots by behaviour as Search, Agent or Training, and lets you block each category across the whole zone, block only on pages that show ads, or allow. Cloudflare's documentation states that the legacy "Block AI bots" setting is deprecating on 15 September 2026, and that from that date new domains default to blocking Training and Agent bots on pages that display ads.
This is enforcement at the edge, which is stronger than robots.txt. It also means a single toggle can quietly block search bots you wanted. After changing a policy, check which categories are blocked and test that your pages still load for the search user agents you intend to allow.
Deciding the trade-off
A short decision path works for most small firms:
- List the pages that win you work. Keep those open to search bots from Google, OpenAI, Anthropic and Perplexity.
- Decide whether you mind your public content being used for model training. If you do, disallow the training agents and Google-Extended.
- Move anything genuinely private behind a login. Do not rely on robots.txt.
- Review robots.txt and any Cloudflare AI policy every quarter, and after any platform migration.
- Measure outcomes rather than guessing. See our note on AI Overviews measurement in Search Console and what content earns AI citations.