Home Free toolsAI readiness checker

AI searchFree · no sign-upReads live data

Is your website ready for AI search?

AI assistants can only recommend what they can read. This checker opens your home page the way an AI crawler does and runs eleven checks: whether AI crawlers are allowed in, whether your words are there before JavaScript runs, whether your business facts are marked up, and whether you answer the questions customers ask.

A score out of 100
Eleven checks, passed or failed
The fixes, most important first

1Fill in a few boxes
2See your result straight away
3Unlock the full report, free

Rather have a person look at it? We will go through your numbers with you on a free 30-minute call.

Book a free call

Which AI crawlers should you allow in robots.txt?

Each AI company sends more than one crawler, and they do different jobs. Search crawlers fetch pages so the tool can answer questions now. User crawlers open a page when a person asks the assistant to look at it. Training crawlers collect text for future models.

To appear in AI answers, the search and user crawlers matter most. Blocking training crawlers is a fair choice some publishers make: it keeps your words out of future models but does not remove you from live answers.

Many sites block these crawlers by accident, through a security plugin, a hosting default or a copied robots.txt. A Disallow: / under User-agent: * blocks every crawler that has no group of its own, AI crawlers included.

The AI crawlers this checker looks for
Crawler	Company	What it does	If you block it
OAI-SearchBot	OpenAI	Finds pages for ChatGPT's search answers	You drop out of ChatGPT search results
ChatGPT-User	OpenAI	Opens a page when a ChatGPT user asks it to	ChatGPT cannot open your page for that person
GPTBot	OpenAI	Collects text to train future models	Future models know less about you; live answers are unaffected
Claude-SearchBot	Anthropic	Finds pages for Claude's search answers	Claude's answers cannot draw on your site
Claude-User	Anthropic	Opens a page when a Claude user asks it to	Claude cannot open your page for that person
ClaudeBot	Anthropic	Collects text to train future models	Future models know less about you; live answers are unaffected
PerplexityBot	Perplexity	Finds pages for Perplexity's answers	You drop out of Perplexity's results
Google-Extended	Google	A robots.txt setting, not a crawler: whether Gemini may use what Googlebot read	Gemini uses less of your site; Google Search and AI Overviews are unaffected

Crawler names as each company publishes them in 2026. Companies add and rename crawlers, so check their own pages when you edit robots.txt.

Can AI crawlers read websites built with JavaScript?

Google's crawler runs JavaScript before it indexes a page, but many AI crawlers fetch the raw HTML and read only what is already there. If your text is added by scripts after the page loads, those crawlers see a near-empty page with nothing to quote.

The checker counts the words in your page before any script runs. About 250 or more is a pass, 80 to 250 is a warning, and fewer than 80 is a fail, because by then an AI sees little beyond a menu and a footer.

Most builders and WordPress themes already send the text in the HTML. The problem is commonest on single-page apps, where a developer can switch on server-side rendering.

What makes a website easy for AI to understand?

Beyond access, AI tools need facts they can lift without guessing: structured data naming your business type, address, phone and hours, a clear title and description, and your name, address and phone in the page text.

Questions and answers help too. An assistant answering 'how much is a boiler service in Oxford' looks for a page that asks and answers that plainly, so a short FAQ on each service page, in your customers' words, gives it something to quote. The eleven checks, heaviest first:

AI search crawlers allowed in robots.txt (20 points).
Search engines may index the page, and words readable without JavaScript (15 points each).
Business facts marked up with structured data (12 points).
Questions and answers on the page (8 points).
AI training crawlers, llms.txt, a sitemap, a title and description, name, address and phone, and HTTPS (5 points each).

How do you fix a low AI readiness score?

Work in order of weight. First make sure robots.txt lets the AI search crawlers in and that no noindex tag sits on the home page; either can hide a good site completely. Then check your text is in the HTML, not added by scripts.

Next, add LocalBusiness or Organization structured data and a short FAQ. The smaller checks, such as llms.txt, a sitemap and HTTPS, take minutes each.

Run the check again after each change. Robots.txt and structured data changes show here as soon as they are live, though AI tools may take days or weeks to read the site again.

How the tool works

Type your website address.
We read your home page, robots.txt, llms.txt and sitemap, as an AI crawler would.
Eleven checks, each passed, failed or could not be read, with the reason.
A score out of 100 and the fixes in the order that matters most.

How Scale Oxford helps

Make your whole site readable to AI with Scale Oxford

This checker reads your home page; we check every page that matters for what stops AI tools reading it, then fix it with you or your developer, so every key page can be read, understood and quoted.

Audit robots.txt, rendering and indexing across your key pages, and fix anything that blocks AI or search crawlers.
Add accurate structured data for your business, services and FAQs, and test it in Google's tools.
Rewrite the opening of each key page so it states in plain text what you do, where and for whom.

Book a free call See our AI SEO service

Case study · BitdefenderScale Oxford got Bitdefender named in 417% more AI answers to the questions UK buyers ask. Its product pages were too vague for ChatGPT or Google’s AI Overviews to quote. We rewrote them plainly and got it mentioned where those tools look.↑ 417%More AI answers naming the company↑ 29Tracked questions where it is named↑ 38More enquiries from AI searchRead the case study →

Questions people ask

How much does llms.txt count in the AI readiness score?

A little: 5 points out of 100. Not every AI tool reads llms.txt yet, so it counts for far less than letting AI crawlers in (20 points) and having pages they can read (15). Our llms.txt generator writes the file for you.

Which AI crawlers should I allow?

GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot and Claude-SearchBot (Claude), PerplexityBot, and Google-Extended for Gemini. Blocking the search crawlers keeps you out of those tools' answers.

Why does content without JavaScript matter?

Many AI crawlers do not run JavaScript. If your words only appear after scripts load, those crawlers see an empty page.

Does a high score mean AI will recommend me?

No. It means AI can read you. Being recommended also depends on what other sites say about you; the AI visibility checker measures that part.

Will blocking AI training crawlers stop me being recommended?

Not on its own. Training crawlers such as GPTBot and ClaudeBot gather text for future models, while search crawlers such as OAI-SearchBot and Claude-SearchBot fetch pages for live answers. Blocking only the training ones keeps you in live answers; blocking the search ones takes you out.

Does Google's AI Overview need a separate crawler?

No. AI Overviews are built from Google's normal search index, which Googlebot fills. If Googlebot can crawl and index your page, it can appear in AI Overviews, and the Google-Extended setting does not change that.

How do I check my robots.txt?

Open yoursite.co.uk/robots.txt in a browser. Look for any group naming an AI crawler with Disallow: /, and for a User-agent: * group that disallows everything. Google Search Console's robots.txt report also shows the file Google last fetched.

My site is on Wix or Squarespace. Can AI read it?

Usually yes. The main builders send page text in the HTML, but some offer a switch to block AI crawlers, so check it is off if you want to be found. Run the checker to see what an AI crawler actually gets from your site.

Does page speed affect AI readiness?

Indirectly. AI crawlers give up on pages that load too slowly or fail, so a very heavy page risks not being read at all. The website grader checks page weight.

What is the difference between this and the AI visibility checker?

This checker asks whether AI tools can read your website. The AI visibility checker asks whether they recommend you when a customer asks, which also depends on what other sites say about you.

How often should I run an AI readiness check?

After any change to your website, theme, hosting or security plugin, and otherwise every few months. Updates often change robots.txt without anyone noticing, and one wrong line can hide the whole site from AI tools.