HomeFree toolsRobots.txt tester

Website and SEOFree · no sign-upReads live data

Robots.txt tester: can Google read this page?

Google retired the robots.txt Tester in Search Console in December 2023. Type your website, or paste a robots.txt, then a page path, and we test it for Googlebot, Bingbot and the main AI crawlers using Google's own matching rules, and show whether each may read the page and the exact rule that decides it.

  • Allowed or blocked, per crawler
  • The exact rule that decides
  • Test changes before you publish
  1. 1Fill in a few boxes
  2. 2See your result in under a minute
  3. 3Copy it, or ask us to look at it
Robots.txt testerReads live data

Testing /

Load a website's robots.txt above, or paste one into the box, to see what each crawler may read.

Rather have a person look at it? We will go through your numbers with you on a free 30-minute call.

Book a free call

How does Google read a robots.txt file?

A robots.txt file sits at the root of a site and is split into groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules for addresses on that site.

A crawler follows only one group: the one whose User-agent most specifically names it. If Googlebot has its own group, it ignores the rules under User-agent: *. Within that group, the matching rule with the longest path wins, and if an Allow and a Disallow are equally long, Allow wins.

Paths are case sensitive and match from the start of the address, so Disallow: /admin also blocks /admin-login and /administrator. This tester applies those rules for Googlebot, Bingbot and the main AI crawlers.

What do common robots.txt rules do?

Most mistakes come from a rule that matches more, or less, than its author meant. The examples below show how Google applies each one.

Robots.txt rules, examples and what Google does
RuleExample addressOutcome
Disallow: //any-pageBlocked: the whole site is closed to that crawler
Disallow: (empty)/any-pageAllowed: an empty rule blocks nothing
Disallow: /admin/administratorBlocked: rules match the start of the path
Disallow: /admin//administratorAllowed: only addresses inside the /admin/ folder are blocked
Disallow: /*.pdf$/files/menu.pdfBlocked: * matches any characters and $ marks the end
Disallow: /*?/shop?colour=redBlocked: any address with a query string
Disallow: / with Allow: /blog//blog/our-newsAllowed: the longer rule wins
Disallow: /page with Allow: /page/pageAllowed: equally specific, so allow wins the tie
Outcomes follow Google's published robots.txt rules; other crawlers may differ in edge cases.

What replaced Google's robots.txt Tester?

Google retired the robots.txt Tester in Search Console in late 2023. In its place is a robots.txt report, under Settings, that shows the robots.txt files Google has found for your site, when it last fetched them and any problems reading them.

The report tells you what Google fetched, but not whether a particular address is blocked. This tester fills that gap: test any path against your live file, or paste a draft and test it before you publish.

Should you use robots.txt or noindex to hide a page?

Use noindex to keep a page out of Google, and robots.txt to stop crawlers spending time on pages that do not need reading, such as search results or filters on a shop.

Do not use both on the same page. A page blocked in robots.txt cannot be read, so Google never sees its noindex, and the address can still be listed from links elsewhere. Google stopped supporting noindex lines inside robots.txt in 2019.

How should you handle AI crawlers in robots.txt?

AI companies publish the names their crawlers use, so you can allow or block each one with its own User-agent group. Some crawl to train models, such as GPTBot, and some fetch pages to answer questions in AI search, such as OAI-SearchBot.

Google-Extended is a separate token: it controls whether Google may use your pages for its Gemini models, and blocking it does not affect Google Search. Decide crawler by crawler, then test the result here for each one.

How the tool works

  1. Type your website to read its live robots.txt, or paste one to test changes before you publish.
  2. Type the page path you want to test, for example /services/plumbing.
  3. We apply Google's matching rules for each crawler: the most specific rule wins, and allow wins a tie.
  4. See allowed or blocked for every crawler, and the line that decided it.

How Scale Oxford helps

Not sure what your robots.txt is blocking? We check the whole site

Scale Oxford runs technical SEO audits that read a site the way Google does. We check robots.txt, noindex tags, canonicals and the sitemap together, because a rule in one often undoes another.

  1. We test every important page against your robots.txt and list anything Google or AI search cannot read.
  2. We rewrite the file so it blocks what should be blocked and nothing else, and test it before it goes live.
  3. We set the AI crawlers one by one, so you stay in AI answers while choosing what is used for training.
Case study · BitdefenderScale Oxford got Bitdefender named in 417% more AI answers to the questions UK buyers ask. Its product pages were too vague for ChatGPT or Google’s AI Overviews to quote. We rewrote them plainly and got it mentioned where those tools look.↑ 417%More AI answers naming the company↑ 29Tracked questions where it is named↑ 38More enquiries from AI searchRead the case study →

Questions people ask

What happened to Google's robots.txt Tester?

Google retired the robots.txt Tester in Search Console in late 2023 and replaced it with a robots.txt report that shows the files Google has fetched. The report does not let you test a path, which this tool does.

Does blocking a page in robots.txt remove it from Google?

No. Robots.txt stops crawling, not indexing: a blocked page can still be listed if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex tag.

Which rule wins in robots.txt?

For Google, the most specific matching rule wins, measured by the length of the path. If an allow and a disallow rule are equally specific, allow wins.

Should I block AI crawlers?

Blocking AI search crawlers such as OAI-SearchBot keeps you out of AI answers. Blocking training crawlers such as GPTBot does not. The tester shows each crawler on its own line, so you can see which you allow.

Where does a robots.txt file go?

At the root of the host, for example yoursite.co.uk/robots.txt. Each subdomain and protocol needs its own, so shop.yoursite.co.uk is not covered by the main site's file.

What happens if a site has no robots.txt?

If the file is missing and the server answers not found, Google treats the whole site as open to crawling. Nothing is blocked.

What happens if robots.txt returns a server error?

Google treats a server error (or a 429, too many requests) on robots.txt as a reason to pause crawling the site for a while, as if it were fully blocked. A robots.txt that errors can stop Google reading a working site.

Does Google follow Crawl-delay?

No. Google ignores the Crawl-delay line. Bing does read it, so it can slow Bingbot down.

Do rules for User-agent: * apply to Googlebot?

Only if there is no group naming Googlebot. Once Googlebot has its own group, it follows that group alone and ignores the general one.

Are robots.txt paths case sensitive?

Yes. Disallow: /Admin does not block /admin. The User-agent name, by contrast, is matched without regard to case.

Can I test a robots.txt before publishing it?

Yes. Paste the draft instead of typing your website, then test the paths you care about. Nothing changes on your site until you upload the file.

Should I block CSS and JavaScript files?

No. Google renders pages much as a browser does, and blocking the files a page needs can stop Google seeing it properly. Leave them open.

Read next

Free 30-minute call

Want a person to look at this with you?

We go through your result, show you what we would do first and what it would cost. No obligation.

Book a free call

Where we can help

More free tools