Free robots.txt checker for AI crawlers

How the checker decides allowed or blocked

It applies the Robots Exclusion Protocol as written in RFC 9309, the same rules Google documents for its crawlers. Consecutive User-agent lines form one group. A bot obeys only the group that names it and falls back to * only when nothing names it. Groups naming the same bot are merged.

Inside a group the longest matching path wins, and on a tie Allow beats Disallow. * matches any run of characters and $ anchors the end, so Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?v=2. The Status column shows the exact line that decided each bot, so you can find it in your file.

Worked example: The Guardian lets ChatGPT in and keeps Claude out

The Guardian's live robots.txt is a good lesson in how much a file can say without saying it. The top carries a comment that LLM and AI uses of its content aren't permitted. Then comes one long group of named bots, ending in a single rule:

User-Agent: PerplexityBot
User-Agent: yacy
User-agent: anthropic-ai
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
...
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: DuckAssistBot
Disallow: /

Paste the full file above and test an article path. The checker reports 10 of 21 AI user agents allowed. PerplexityBot, all three Anthropic agents, DuckAssistBot and Amazon's bots are blocked by that group. GPTBot, OAI-SearchBot and ChatGPT-User aren't named anywhere, so they fall through to the * group, which only blocks paths like /search and /discussion/. Google-Extended isn't named either.

That's a business decision written as a robots.txt. Guardian Media Group signed a content partnership with OpenAI in February 2025, with Guardian reporting shown in ChatGPT with attribution. Two things to take from it. The comment at the top has no effect on any crawler, since only User-agent and Allow/Disallow lines count. And silence is permission. A bot you don't name gets your * rules.

Compare The New York Times, whose file gives GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User and Google-Extended each their own Disallow: /. That blocks the search agents too, which means opting out of being a cited source in ChatGPT search and Perplexity. For a publisher negotiating licences that can be the point. For a SaaS company it would be self-harm.

Which blocked rows should you fix first?

Fix a blocked AI search agent or a blocked Googlebot or Bingbot first; a blocked training crawler can wait, because it may be a deliberate choice. The table mixes three kinds of agent, a split documented by OpenAI, Anthropic and Perplexity.

Don't test only /. Put your pricing page, a top blog post and a docs page in the path box. Old rules such as Disallow: /blog/*? or a blanket block on /docs/ tend to hit exactly the pages engines would cite.

Should you block GPTBot? A decision rule

Block training crawlers if your content is the product: you sell subscriptions or licences, or a model that memorized your pages would replace a visit. Allow them if you're a brand that wants models to describe your category with you in it, which covers most B2B software companies. Either way, keep the search agents open.

That's what most large sites that block training already do. In our robots.txt study of 261 top sites, 31 of the 59 that disallow GPTBot still allow OAI-SearchBot, and every site that blocked an AI search bot also blocked a training bot. B2B SaaS sites barely block at all: 11 of 95 block any training bot.

Four mistakes that block AI search by accident

A leftover staging rule. User-agent: * plus Disallow: / shipped to production blocks every bot you haven't named, including all the AI search agents. The checker shows this as a wall of red rows reading "Disallow: / (* group)".

Giving a bot its own group and losing your other rules. Add User-agent: GPTBot with Allow: / and GPTBot stops reading your * group, so it can now crawl /admin/ and /cart. The recommended config above copies your * rules into the named group for that reason.

Expecting Google-Extended to keep you out of AI Overviews. It doesn't. Google says the token only covers Gemini training and grounding and has no effect on Search. AI Overviews and AI Mode are part of Search, and Google's AI features documentation points to nosnippet, data-nosnippet, max-snippet or noindex to limit what appears there.

A clean file and a blocking firewall. robots.txt is a request, and your CDN may refuse bots before they ever read it. Cloudflare began blocking known AI crawlers by default for new domains on July 1, 2025. If the checker says allowed but you never see the bot, search your access logs for the user agent (for example grep -c "OAI-SearchBot" access.log) and check your bot-management settings.

Once access is right, the next questions are whether a page is worth quoting, which the citability grader scores, and whether engines name you at all. The free Arobis AI visibility checker tests the second one against real buyer prompts, and our comparison of 10 free AI visibility checkers covers the alternatives. The full user-agent list, with IP verification, is in our AI crawler and robots.txt guide, and the OAI-SearchBot glossary entry has the short version.

Frequently asked questions