Decide which AI tools may read your website

Most AI companies now run more than one crawler. One collects pages to train models, one finds pages for search answers, and one opens a page when a user asks the assistant to. Each has its own name, so robots.txt can treat them differently.

Two names in the list are not crawlers at all. Google-Extended and Applebot-Extended are names you use in robots.txt to tell Googlebot and Applebot how the pages they already collect may be used for AI.

For most businesses that want customers, it is a sound starting point. The search and user crawlers are how ChatGPT, Claude and Perplexity find and cite you when someone asks for a recommendation; the training crawlers only decide whether your words feed future models.

There is a case for allowing training too. A model that has read your pages may describe your business more accurately even when it does not search, so firms that want to be widely known often allow everything.

Blocking everything makes sense for content you sell, such as paid courses or research, and for pages that should never appear in answers. Whatever you choose, check the file afterwards: one stray line can block more than you meant.

• Want to be recommended: allow the search and user crawlers.
• Want your words kept out of training: block GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot.
• Selling the content itself: block all of them, and keep paid pages behind a login.
• Not sure yet: allow everything and revisit, since blocking later stops future crawling only.

robots.txt is a plain text file at the root of your domain, such as www.yoursite.co.uk/robots.txt. It covers only that host, so a shop on a subdomain needs its own file.

The file is made of groups. Each starts with one or more User-agent lines and lists Disallow and Allow rules beneath them. A crawler follows the group that names it and ignores the general group marked with an asterisk, so a group for one bot must repeat any private folders it should also stay out of.

Disallow: / blocks the whole site, and an empty Disallow: line allows everything. To block GPTBot, the file needs two lines: User-agent: GPTBot, then Disallow: / on the next line. Google reads up to 500 KiB of the file and usually picks up changes within a day; other crawlers keep their own timetable.

It cannot force anyone to obey it. The large AI companies say their crawlers follow it, but any bot can ignore the file or borrow another bot's name. OpenAI and Perplexity publish the IP addresses their crawlers use, so server logs can tell a real visit from an impostor.

It cannot take back what has already been collected. Blocking a training crawler keeps new and changed pages out of future training; models already trained keep what they learned.

It may not be the only thing deciding, either. Some hosts and firewalls, Cloudflare among them, offer a setting that blocks AI crawlers before they read robots.txt. If AI tools never seem to read your site, check there too.

• Not a lock: private pages need a login.
• Not retroactive: past crawls stay where they are.
• Not a ranking tool: Googlebot's rules decide Google Search.
• Not the same as llms.txt, which points AI tools at your best pages but blocks nothing.