Which AI crawlers do top websites block? A study of 261 sites

How many top websites block AI crawlers?

About one in three blocks something, and about one in five blocks a bot that decides whether the site can be cited in AI answers. We fetched robots.txt from 285 domains and could read 261. Of those, 91 (35%) disallow at least one of the eight training agents we tested, and 56 (21%) disallow at least one of the three AI search agents: OAI-SearchBot, Claude-SearchBot and PerplexityBot.

Blocking tends to be all or nothing. 26 sites block all eight training agents, while 170 block none. The middle is thin: just 12 sites block exactly one training agent, and for eight of those the one bot is Bytespider or CCBot.

Googlebot is our control, and it behaves like one. Exactly 1 of 261 robots.txt files we received blocks Googlebot at the homepage, and that file (Reddit's) blocks everyone. More on that below.

Which AI crawlers get blocked most often?

Training crawlers top the list, led by Common Crawl's CCBot and ByteDance's Bytespider at 75 sites each. The three AI search crawlers sit near the bottom, and OpenAI's OAI-SearchBot is the least blocked AI agent of all.

The table splits each count by how the block happens. "Named group" means the file has a User-agent line for that bot. "Fallback" means the bot isn't named, so under RFC 9309 it obeys the * group, and that group says Disallow: /8.

ClaudeBot (67) is blocked more often than GPTBot (59), even though both are training crawlers and both operators document a separate search bot. The fallback column is large for the newer agents: 11 of the 38 Claude-SearchBot blocks and 12 of the 40 Perplexity-User blocks come from a site that never named the bot at all. Those sites wrote rules for the older tokens and a catch-all deny for everything else.

Do sites block training bots but allow AI search bots?

Yes, and that's the most useful pattern in the data. OpenAI, Anthropic and Perplexity each run separate agents for training, search indexing and user-requested fetches, and they document them separately123. A site can refuse training and still be eligible for citations.

• 59 sites block GPTBot. 31 of them (53%) still allow OAI-SearchBot.
• 67 sites block ClaudeBot. 31 of them (46%) still allow Claude-SearchBot.
• 91 sites block at least one training agent. 35 of them (38%) allow all three AI search agents.
• 56 sites block at least one AI search agent, and all 56 also block a training agent. We found zero sites that allow training and block search.
• 21 sites (8%) block all three AI search agents.

The vendors are not treated equally. 13 sites block PerplexityBot but allow GPTBot, and another 13 block Claude-SearchBot but allow OAI-SearchBot. Only 5 sites do the reverse (block OAI-SearchBot, allow Claude-SearchBot): Gartner, CNBC, bbc.com, bbc.co.uk and the Sydney Morning Herald. Across the three search agents, PerplexityBot is blocked on 50 sites, Claude-SearchBot on 38 and OAI-SearchBot on 30.

Which kinds of websites block AI crawlers?

News and social platforms block the most; developer, government and B2B SaaS sites block almost nothing. The gap is large enough that a single blended number hides it, so here it is per category.

Percentages use the sites we could read in each category as the base. Small categories (8 to 11 sites) swing a lot on one site, so read them as direction, not precision.

Nearly all the blocking happens in news. Of the 48 news sites we read, only six block none of the 14 AI agents: Fox News, Time, The Independent, Sky, Hindustan Times and the South China Morning Post. Nine block training while leaving all three search agents open, including CBS News, ABC News, Business Insider, Axios, the Los Angeles Times and TechCrunch.

B2B SaaS is the opposite. 84 of the 95 SaaS sites we read block none of the AI agents. The 11 that block something mostly target scrapers: Bytespider alone at Amplitude, Postman, G2 and Capterra, CCBot at Calendly, Amazonbot at Notion. Canva and Figma block several training agents. Gartner, which we grouped with B2B software, is the only one in the group that blocks an AI search agent (OAI-SearchBot).

Among tech and developer sites, the only block is GitHub's on Bytespider, and among government sites it's the World Health Organization's on CCBot. Wikipedia, Microsoft, Mozilla and every US and UK government domain we read allow all 14.

How many top websites have an llms.txt file?

34%: 81 of the 241 domains that gave us a definite answer. llms.txt is a proposed standard for a Markdown file at the site root that points language models to a site's key content10. We counted it only if the response was HTTP 200, the content type was text/plain or text/markdown, and the file opened with a Markdown H1 (# followed by a space). That rules out sites that redirect /llms.txt to an HTML page.

"Definite answer" means the server returned 200, 404 or 410. We left out 44 domains that answered with 403, 429, another error or a timeout, because a refusal tells you nothing about whether the file exists. OpenAI's and Perplexity's own domains returned 403 to our fetcher, so they are not in the base.

Adoption tracks the people who sell to developers and marketers. 59 of 92 SaaS sites have one, including Stripe, HubSpot, Atlassian, Vercel and Shopify. Six more SaaS sites (Salesforce, Notion, Twilio, Greenhouse, BILL and PagerDuty) serve a plain text file at /llms.txt that doesn't start with an H1, so our rule didn't count them. Among news sites, only Fox News, The Times of India and Hindustan Times pass.

Ten sites publish an llms.txt and also block at least one training crawler, including GitHub, Coursera, ZoomInfo and Loom. Those aren't contradictory. llms.txt tells a model where the useful pages are; robots.txt decides which bots may fetch them.

Want one for your own site? Our llms.txt guide covers what the file does and doesn't do, and the llms.txt generator builds a valid file from your page list.

What do the actual robots.txt rules look like?

Here's how ten well-known sites express their policy. Every rule below is quoted from the file our fetcher received.

• The New York Times gives each AI agent its own group ending in Disallow: /, including User-agent: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Googlebot shares the * group, which only blocks paths like /ads/. Result: all 14 AI agents blocked, Google allowed.
• The Guardian lists about 30 agents in a single group that ends in Disallow: /: ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Applebot-Extended, meta-externalagent, Amazonbot and others. GPTBot, OAI-SearchBot and ChatGPT-User are not in that group, so they fall back to the permissive * rules and can crawl.
• BBC (bbc.com) blocks GPTBot and OAI-SearchBot with Disallow: / plus Allow: /storyworks, and blocks ClaudeBot outright. It never names Claude-SearchBot or Claude-User, so both are allowed through the * group.
• Bloomberg labels its groups with comments, # Training above GPTBot and # Search indexing above OAI-SearchBot. The search group reads Disallow: / followed by Allow: /$ and a few section allows. The homepage is open to ChatGPT search; articles are not.
• Facebook and Instagram end with User-agent: * / Disallow: /. Both name GPTBot, ClaudeBot and PerplexityBot with path-level rules, so those can reach the homepage. OAI-SearchBot, Claude-SearchBot and even Meta's own meta-externalagent aren't named, so the catch-all blocks them.
• Netflix runs default-deny (User-agent: * / Disallow: /), then opens a group for GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, ClaudeBot, Claude-User, Claude-SearchBot and PerplexityBot with Allow: /. Agents it didn't list, such as Perplexity-User and Applebot-Extended, stay blocked.
• LinkedIn blocks GPTBot, ClaudeBot and PerplexityBot with Disallow: /, but gives OAI-SearchBot its own group of path-level disallows such as /public-profile/. ChatGPT search can reach the homepage; OpenAI's trainer cannot.
• Reddit served our fetcher a two-line policy: User-agent: * / Disallow: /. Taken literally that blocks Googlebot too, and it's the only Googlebot block in the dataset. Reddit pages still rank in Google, which tells you the file a generic fetcher sees isn't always the whole access story.
• Canva groups GPTBot, ClaudeBot, CCBot and Applebot-Extended under Disallow: / with Allow: /create/, while OAI-SearchBot, Claude-SearchBot and PerplexityBot stay allowed. A clean training-only block.
• GitHub names GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot together with Crawl-delay: 1 and a list of allowed sections. It slows them down instead of shutting them out; Bytespider is the only agent it blocks at the homepage.

The fallback trap shows up again and again. 15 of the 261 sites run a default-deny * group, and on those sites any AI agent the file forgets to name is blocked. Facebook names three AI crawlers and blocks the search agents it left out. That might be intentional. It's also exactly what happens when a file gets a rule for GPTBot and then nobody touches it again.

What should you do with your own robots.txt?

Start by checking what your file does today: paste it into our robots.txt AI crawler checker, which uses the same parser as this study. Then apply these rules.

1. If you sell software or services and want AI engines to recommend you, allow all three search agents and the user fetchers. 94 of the 95 SaaS sites we read leave OAI-SearchBot, Claude-SearchBot and PerplexityBot open. Blocking them puts you behind every competitor that doesn't.
2. Block training agents only if your content is the product. That's the pattern among publishers whose pages are what they sell. If your pages exist to win customers, a training block protects nothing you'd charge for.
3. If you block a vendor, name all of its agents explicitly, in both directions. BBC blocks ClaudeBot but, by leaving Claude-SearchBot unnamed, allows it. If that's what you want, write User-agent: Claude-SearchBot / Allow: / so the next person editing the file can see it.
4. If your * group says Disallow: /, list every AI agent you want to allow. Under default-deny, every new agent is blocked until you name it. Re-check the list whenever OpenAI, Anthropic or Perplexity publish a new token.
5. Test a deep URL, not just the homepage. Allow: /$ opens the homepage and nothing else. When we re-ran every check against a generic article path, OAI-SearchBot blocks rose from 30 to 34 sites.
6. Don't rely on robots.txt for user fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page13. If you must stop those, do it at the CDN or firewall.

For the full list of AI user agents and what each one does, see our AI crawler user agent reference. Access is only step one. To see whether ChatGPT, Perplexity and Gemini actually mention your brand once they can reach you, run your buyer prompts through Arobis AI's free AI visibility checker, or compare the free and paid options in our roundup of AI visibility checkers.

How we collected and parsed the data

Sample. We built the list ourselves rather than downloading a ranking: 285 domains in 10 categories. 130 are large global sites across search and portals (11), social and community (17), video and streaming (10), e-commerce and marketplaces (23), travel and local (8), reference and education (25), tech and developer (22) and government and public sector (14). 102 are well-known B2B SaaS companies (the group also includes G2, Capterra and Gartner). 53 are news and media publishers from the US, UK, Europe, Asia and Australia. Each domain appears once. The full list with categories is in the dataset.

Collection. Data collected October 2026. A Python script requested https://<domain>/robots.txt and https://<domain>/llms.txt with the user agent CitedResearch/1.0 (robots.txt study) (the publication's working name before AI Ranked), a 10-second timeout, redirects followed and at most 8 requests in flight. Failed requests were retried once, including on the www. host.

What counted as read. A 200 response with a text body was parsed. A 404 or 410 means the site has no robots.txt, which RFC 9309 treats as permission to crawl8; two sites (disneyplus.com and telegram.org) fell here. 24 domains were excluded from every percentage: 17 refused our fetcher with an HTTP error (403, 405, 418 or 429), 3 returned an HTML challenge or redirect page instead of a text file, 3 timed out and 1 refused the connection. That leaves n = 261.

Parser. We ported the parser from our robots.txt checker to Python and evaluated each agent against the path /. Consecutive User-agent lines form one group, groups naming the same agent are merged, an agent with no named group falls back to *, the longest matching rule wins, Allow wins a tie, and * and $ work as wildcards. Agent names are matched case-insensitively. An agent counts as blocked if / is disallowed.

Agents. Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Amazonbot. AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. User fetch: ChatGPT-User, Claude-User, Perplexity-User. Control: Googlebot. The groupings follow each operator's documentation123456711. ByteDance doesn't publicly document Bytespider's purpose, so we grouped it with training crawlers based on common practice.

Limitations

• robots.txt is a request, not an enforcement mechanism. A site that allows a bot here may still block it at the firewall or CDN. Cloudflare, for example, began blocking AI crawlers by default for new customers on 1 July 20259, and that kind of block never shows up in robots.txt.
• Some sites serve different robots.txt files to different requesters. We saw one file per site, the one served to a self-identified research fetcher. We can't see what Reddit serves to Googlebot, for example.
• We tested the homepage path only. Sites that block sections (or allow only /$) are under- or over-counted for deep pages; the article-path rerun raised each agent's count by 1 to 4 sites.
• The 24 excluded domains skew toward sites with aggressive bot protection, so the true blocking rate among very large sites is probably somewhat higher than what we measured.
• This is a curated list of well-known sites, not a random sample of the web. Our numbers describe big brands, not the average website. For wider context, see our AI search statistics.

Frequently asked questions

Sources

Dataset: ai-crawler-access-data.csv (285 rows, one per domain, including the 24 we could not read).