What is ClaudeBot? Anthropic's crawler, Claude-SearchBot and Claude-User explained

What is ClaudeBot?

ClaudeBot is the crawler Anthropic uses to collect public web content that could go into training its Claude models. Anthropic's crawler help page describes it as "collecting web content that could potentially contribute" to model training, and says a site that restricts it is signalling that its future materials should be left out of training datasets. It is a training crawler, so it doesn't decide whether Claude can cite you in a live answer.

The name causes some confusion in search results. Type "claudebot" into Google and you'll also see Clawdbot, a self-hosted personal assistant that runs on your own machine and is now called OpenClaw. It's a separate open-source project that can use Claude as its model. It isn't a crawler and has nothing to do with your robots.txt file. Everything below is about Anthropic's crawler.

ClaudeBot is also a crawler in the strict sense: it requests pages on its own schedule, with no person waiting on the other end. That separates it from the two newer Anthropic agents, which matter more for whether Claude can find and quote you today.

What are Anthropic's three crawlers?

Anthropic documents three user agents: ClaudeBot for model training, Claude-SearchBot for search indexing, and Claude-User for fetches a person triggers inside a Claude conversation. Each one reads its own group in robots.txt, so you can block training while staying visible in Claude's search answers.

Two older tokens still show up in a lot of robots.txt files: anthropic-ai and Claude-Web. Neither appears on Anthropic's current crawler page, which lists only the three above. Robots.txt matching works on the exact product token (more on that below), so a rule written for anthropic-ai does nothing to ClaudeBot. We found four large sites making exactly that mistake.

Anthropic also commits to a few behaviours on the same page. Its bots honor robots.txt, support the non-standard Crawl-delay extension, and "will not attempt to bypass CAPTCHAs". It asks you to put the robots.txt rules on every subdomain you want covered, because each host reads its own file. And it doesn't publish the full user agent strings, only the product tokens, so match on the token and on the IP list rather than on a copied header from someone's blog.

The OpenAI and Perplexity equivalents line up role for role, which makes cross-vendor rules easier to reason about. OpenAI's bot documentation pairs GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (user actions), says each setting is independent, and warns that for ChatGPT-User, which a person sets off, robots.txt rules "may not apply". Perplexity's crawler docs say Perplexity-User "generally ignores robots.txt rules" for the same reason. Anthropic is the stricter of the three on paper: its page says all three of its bots respect robots.txt, Claude-User included.

How much does ClaudeBot crawl?

ClaudeBot is one of the two busiest AI crawlers on the web, alongside GPTBot. Cloudflare reported that in the first week of August 2025, ClaudeBot and GPTBot together made up nearly half of the AI crawling it observed across its network, according to its crawl purpose and industry analysis.

Its share moves a lot depending on the window you pick. Comparing May 2024 with May 2025, Cloudflare's crawler share report had ClaudeBot falling from 27% to 21% of AI-only crawling, and its raw requests dropping 46%. Comparing July 2024 with July 2025, a later Cloudflare post had it rising from 15% to 23.3% of AI-only crawling. The two aren't in conflict so much as they show a bot whose volume swings month to month. Cloudflare's 2025 Year in Review describes the full year: ClaudeBot crawling roughly doubled through the first half of 2025, then declined through the second half, ending about 10% above where it started.

Those are network averages. On a single site, ClaudeBot can be far heavier. In July 2024 iFixit's CEO said ClaudeBot hit the site's servers almost a million times in 24 hours, as The Verge reported. iFixit added a Crawl-delay rule to robots.txt, and Wiens said that based on the logs the crawling stopped after it went in. Read the Docs and Freelancer.com reported similar experiences in the same thread.

The practical lesson from the iFixit case is that Crawl-delay is the tool Anthropic itself points to for load problems, and it worked there. A blanket block is the heavier option and also ends your participation in training, which you may or may not care about.

Does ClaudeBot send any traffic back?

Very little, relative to how much Anthropic crawls. Cloudflare's crawl-to-refer ratio divides the HTML pages a platform's bots fetch by the visits its products send back. Anthropic had the highest ratio of any major AI or search platform in 2025: about 38,000 crawls per referred visit in July 2025, against roughly 1,091 for OpenAI and 195 for Perplexity in the same month.

The trend is improving fast. Anthropic's ratio fell 87% between January and July 2025, and Cloudflare tied the timing to Claude getting web search in March 2025, which produces clickable citations. Anthropic's announcement is dated March 20, 2025, and an update on the same page says search reached all Claude plans globally on May 27, 2025. The Year in Review adds the longer view: ratios as high as 500,000 to 1 early in 2025, erratic through May because referrals were sparse, then settling between about 25,000 and 100,000 to 1.

The ratio also depends heavily on what kind of site you run. Cloudflare split it by industry for the first week of August 2025:

News sites get the best deal from every platform, because people ask assistants about current events and click through to read more. Even there, Anthropic's ratio is about 16 times OpenAI's and 76 times Perplexity's.

Who blocks ClaudeBot? Our robots.txt test of 251 sites

About a quarter of large websites block ClaudeBot. In our test, 65 of 251 major sites (25.9%) disallow it at the homepage path, which makes it more widely blocked than GPTBot (58 sites, 23.1%). The search and user bots get far more room: 36 sites (14.3%) block Claude-SearchBot and 40 (15.9%) block Claude-User.

Anthropic gets blocked slightly more often than OpenAI in two of the three roles. The difference is small for search (14.3% against 11.2%) and flips for user fetchers, where ChatGPT-User is blocked by 41 sites and Claude-User by 40. Here are the full counts, including the two legacy Anthropic tokens:

The "blocked" count can be higher than the "naming it" count because some sites block every bot through the * group and then allow a short list by name. Eight sites block ClaudeBot that way without ever writing its name: Baidu, X, Reddit, Pinterest, IMDb, The Wall Street Journal, Reuters and The Times.

Of the 66 sites that block at least one Anthropic bot, most pick one of two patterns. 36 block all three, which reads as "no Anthropic access at all". 26 block only ClaudeBot, which reads as "no training, but you can still find and fetch us".

Industry decides most of it. News publishers block ClaudeBot at nearly three times the overall rate, and B2B software companies almost never do:

The picture barely moved in the weeks since our original crawl. Across the 248 sites readable both times, ClaudeBot blockers went from 63 to 64. Facebook and AliExpress added ClaudeBot to their robots.txt blocks, Forbes dropped it, and CNBC newly blocked Claude-SearchBot. Policy changes at this level happen a handful at a time, not in waves.

Why do publishers block ClaudeBot but allow GPTBot?

Mostly because they have a content deal with OpenAI and none with Anthropic. In our test, 15 sites block ClaudeBot while leaving GPTBot allowed, and 14 of them belong to publishers that have announced licensing or content partnerships with OpenAI. The 15th is Tumblr, where we couldn't find a confirmed deal.

The News Corp, Vox Media, Atlantic and Axel Springer deals are listed in the same Engadget report as the Condé Nast agreement. The pattern is consistent enough to read as policy: these publishers didn't stop blocking AI crawlers, they made an exception for the company paying them. Anthropic gets the default treatment.

The reverse case is smaller and more interesting. Eight sites block GPTBot but not ClaudeBot. Three of them, ABC News, ESPN and CBC, block the legacy anthropic-ai token and never name ClaudeBot, so ClaudeBot falls through to their permissive * group. That looks like a list copied from an older template rather than a decision to welcome Anthropic.

Does blocking anthropic-ai or Claude-Web block ClaudeBot?

No. Robots.txt groups match on the bot's own product token, and ClaudeBot's token is ClaudeBot. A group for anthropic-ai or Claude-Web applies only to a bot that identifies with that token, and neither is on Anthropic's current list.

RFC 9309, the robots.txt standard, says crawlers must use case-insensitive matching to find the group for their product token, combine every group that matches, and fall back to the * group only when nothing matches. So claudebot and ClaudeBot are the same, but anthropic-ai and ClaudeBot are strangers.

In our sample, 59 sites still name anthropic-ai and 46 name Claude-Web. Most of them also name ClaudeBot, so no harm done. Four don't. Netflix, ABC News, CBC and ESPN block anthropic-ai while ClaudeBot stays allowed. The bigger gap is on the search side: 38 of the 80 sites that name ClaudeBot never mention Claude-SearchBot. For those sites, Claude-SearchBot gets whatever the * group says, which is usually "allowed". That may be what they want. It's rarely a choice anyone made on purpose.

Should you block ClaudeBot?

Block ClaudeBot if your content is what you sell, or if you're negotiating a licensing deal and want a stronger hand. Keep it allowed if you sell software or services and want Claude's underlying model to know your company when it answers without searching. Either way, think separately about Claude-SearchBot and Claude-User, because they control whether Claude can find and quote you live.

Two facts from our data back the software row. First, only 2 of 93 B2B SaaS sites in our test block ClaudeBot, so allowing it is the norm in that category, not a risk you're taking alone. Second, training is how Claude's model learns about you for the answers it gives without searching, and a block only affects future training runs. Anthropic's wording is that a block signals that "future materials" should be excluded, so nothing already collected comes back out.

The publisher row reflects the clearest split in the data: 26 sites block ClaudeBot alone and leave the two live bots open. If your traffic depends on referrals, blocking Claude-SearchBot trades a small and growing referral source for very little, because training isn't affected by that rule at all.

How to block or allow ClaudeBot in robots.txt

Add a group for each Anthropic token you want to control in the robots.txt at the root of each host. A group starts with one or more User-agent lines and applies its Disallow and Allow lines to those bots. These three templates cover the common cases.

Block training only, keep Claude search and user fetches:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

Block every Anthropic bot:

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /

Slow ClaudeBot down without blocking it, while keeping your private paths closed:

User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /admin/
Disallow: /cart/

User-agent: *
Disallow: /admin/
Disallow: /cart/

The third template carries the most common robots.txt mistake we see. Once you write a group for ClaudeBot, ClaudeBot stops reading the * group entirely, because RFC 9309 sends a crawler to its own group when one exists. A ClaudeBot group that contains only Crawl-delay: 1 therefore gives ClaudeBot access to every path you disallowed for everyone else. Repeat your private-path rules inside every named group.

A few other details trip people up:

• Subdomains. docs.example.com and help.example.com each need their own file. Anthropic's help page says to add the rules for every subdomain you want to opt out.
• File size. RFC 9309 says crawlers must parse at least 500 KiB of a robots.txt file. Lines beyond whatever limit a crawler sets can be ignored, so keep AI bot groups near the top of very long files.
• Status codes. A robots.txt that returns 404 tells crawlers there are no rules. If your firewall serves the file only to browsers, a bot sees an error instead of your rules.
• IP blocks. Anthropic warns that blocking its IP addresses instead of using robots.txt "may not work correctly or persistently", partly because it then can't read your robots.txt at all.

For the full list of AI user agents beyond Anthropic, with rules for OpenAI, Google, Perplexity and the rest, see our AI crawlers list for robots.txt.

How do you verify that a request is really ClaudeBot?

Check the request's IP address against Anthropic's published list at claude.com/crawling/bots.json. The user agent header proves nothing, since anyone can send "ClaudeBot". Reverse DNS doesn't work for Anthropic either, because its addresses belong to Amazon and Google Cloud.

We downloaded the list and looked up every entry. It holds 14 IPv4 prefixes and no IPv6. One is a /22 block of 1,024 addresses registered to Amazon. The other 13 are single addresses registered to Google, and each reverse-resolves to a generic bc.googleusercontent.com hostname. Addresses we sampled in the Amazon block returned no reverse DNS name at all. That's 1,037 addresses in total, 98.7% of them in the single Amazon range.

This matters for two reasons. Reverse DNS checks, which work for Googlebot and bingbot, can't confirm Anthropic. And a lazy check can actively mislead: Google's verification guide lists googleusercontent.com among the domains its own fetchers can resolve to, so a script that treats any googleusercontent.com hostname as Google would label some of Anthropic's traffic as Google's.

Anthropic's list next to the other published crawler lists, all fetched the same day:

Anthropic publishes one list for all three of its bots, while OpenAI and Perplexity publish one file per bot. That means an IP match tells you a request came from Anthropic, but the user agent token is what tells you which of the three it was. Check both. A few lines of Python do the IP half:

import ipaddress, json, urllib.request

req = urllib.request.Request("https://claude.com/crawling/bots.json",
                             headers={"User-Agent": "Mozilla/5.0"})
nets = [ipaddress.ip_network(p["ipv4Prefix"])
        for p in json.load(urllib.request.urlopen(req))["prefixes"]]

def is_anthropic(ip):
    addr = ipaddress.ip_address(ip)
    return any(addr in n for n in nets)

print(is_anthropic("216.73.216.10"))   # True: inside the Amazon /22

Refresh the list on a schedule rather than hard-coding it. The file carries a creationTime field, and the copy we read had been regenerated nine days before we fetched it. Requests that claim a Claude token from an address outside the list are spoofed, and you can treat them like any other unknown scraper.

What happens when a firewall challenges ClaudeBot?

It fails quietly. Anthropic says its bots won't try to bypass CAPTCHAs, so a challenge page or a bot-management block means the request ends there, whatever your robots.txt says. For Claude-User that's a failed fetch in the middle of someone's conversation about your page.

This is where robots.txt and reality drift apart. Cloudflare's Year in Review found AI crawlers were the user agents most often fully disallowed in robots.txt in 2025, with GPTBot, ClaudeBot and CCBot named among them. Many sites go further and block AI bots at the edge, through a CDN setting, a WAF rule or an Amazon or Google Cloud ASN block aimed at scrapers. Because Anthropic's addresses sit in exactly those cloud ranges, a broad "block cloud hosting" rule catches Claude-SearchBot and Claude-User along with the scrapers it was written for.

To check, look in your server or CDN logs for requests whose IP is on Anthropic's list and see what status code you returned. A 200 is fine. A 403, a 429 or a challenge page on a request from a listed IP is your firewall overriding the allow you set in robots.txt. If your CDN has an allowlist for verified bots, add Anthropic's ranges there instead of loosening the whole rule.

How does Claude search use your pages?

When Claude searches the web, it shows citations to the pages it used, and those citations are where Anthropic's referrals come from. Anthropic says Claude "provides direct citations so you can easily fact check sources" when it brings web information into a response. Pages that Claude-SearchBot has indexed and that Claude-User can fetch are the ones eligible to appear there.

Developers see the same rules through the API. Claude's web search tool costs $10 per 1,000 searches plus token costs, and the web fetch tool lists robots.txt among the Anthropic-side restrictions that can return a "url_not_allowed" error. In other words, apps built on Claude inherit your robots.txt choices for Anthropic's bots, not just the Claude app itself.

Allowing the bots is necessary but not sufficient for a citation. What gets picked from the pages Claude can read is a separate question, which our guide to how AI engines choose sources covers.

Five ClaudeBot mistakes we found in real robots.txt files

1. Blocking the legacy token instead of the current one. Four sites in our sample, including Netflix and ESPN, block anthropic-ai and leave ClaudeBot allowed. Fix: add a User-agent: ClaudeBot line to the same group.
2. Naming ClaudeBot and forgetting Claude-SearchBot. 38 of 80 sites that name ClaudeBot never mention Claude-SearchBot, so its access depends on the * group. Fix: decide on the search bot explicitly.
3. A Crawl-delay-only group. A ClaudeBot group with no Disallow lines overrides your * rules and opens private paths. Fix: repeat path rules inside named groups.
4. Blocking by IP or ASN and calling it an opt-out. Anthropic says IP blocks may not persist and stop it reading your robots.txt. Fix: use robots.txt for policy and the firewall for abuse.
5. Trusting the user agent header. Any scraper can claim to be ClaudeBot. Fix: match the IP against bots.json before you allow, block or count it.

Robots.txt rules are only half of whether Claude can reach you. If you want to see which AI bots your current file lets through, run it through our free robots.txt AI checker.

Frequently asked questions