Cloudflare and AI crawlers: how to check, allow and monitor them

Does Cloudflare block AI crawlers by default?

For AI training crawlers, often yes; for AI search, not by default. On July 1, 2025 Cloudflare changed its default to block AI crawlers unless they pay creators, and it now sets policy per crawler behavior, so what your zone does depends on when it was set up and what someone chose.

The current controls are AI bot policies under Security Settings > Configure AI bot policies, available on every plan including Free. There are three behaviors, Search, Agent and Training, and each gets its own setting: Allow, Block, or Block on pages with ads. A September 2026 Cloudflare post adds a fourth option for Training only, Disallow AI Training.

Older zones may still show the legacy Block AI bots toggle, which targets verified bots classed as crawling for AI training. Either way, Cloudflare's docs are explicit that each blocking option blocks verified bots with that behavior plus unverified bots that look similar. A verified bot isn't exempt from a policy you set.

Which Cloudflare settings can block AI search agents?

Five settings can stop an AI search agent or user fetcher, and they live in different parts of the dashboard. When a crawler is blocked and nobody remembers blocking it, it's usually one of these.

Order matters too. Cloudflare notes that a crawler set to Allow in AI Crawl Control can still be blocked by security rules that run before the AI Crawl Control rule. Allowing a crawler in one screen doesn't undo a block in another.

Two features look like blocks but work differently. AI Labyrinth adds hidden links that lead crawlers ignoring your no-crawl rules into generated pages; Cloudflare says its actions aren't mitigations. Bot Preference Sync edits your robots.txt, covered below.

How do I refuse AI training without losing AI search?

Set the Training policy to Disallow AI Training rather than Block, and leave Search and Agent on Allow. That combination keeps the crawlers that decide citations working while telling training crawlers no.

The distinction matters more than it used to. In that same September 2026 post, Cloudflare says Block and Block on pages with ads for Training now apply to mixed-use crawlers, including Applebot, Bingbot and Googlebot. Those crawlers also feed classic search and AI answers: Google's AI features, for example, draw on pages indexed by normal Search crawling. A hard block on Training can therefore reach well past model training.

The same thinking applies in robots.txt: GPTBot and OAI-SearchBot do different jobs, and blocking one shouldn't block the other. Our AI crawler list sorts every agent into training, search and user fetch.

Why is Perplexity treated differently on Cloudflare?

Because Cloudflare doesn't verify it. In August 2025 Cloudflare published evidence that Perplexity used undeclared crawlers to get around no-crawl directives and de-listed its bot from the verified bot list, a step it restated in October 2026.

In practice, any rule written as "allow verified bots" won't cover PerplexityBot or Perplexity-User. If you want Perplexity to cite you, check that nothing challenges those user agents, and match them explicitly in any allow rule. Perplexity publishes the IP ranges for both agents in its crawler docs, which is how Monitor tells real Perplexity visits from fakes.

How do I check what Cloudflare does to AI crawlers?

Use three views, because each one sees a different slice of the problem.

1. AI Crawl Control. The Crawlers tab lists the AI crawlers hitting your zone with request counts, robots.txt violations and the action applied to each. On the Free plan detection is by user agent, with a 24-hour window.
2. Cloudflare Radar. The bots directory shows whether Cloudflare treats an agent as verified and which behavior it's classed under, which tells you which policy applies to it.
3. An outside test. Request your homepage as the agents and compare with a browser. The free AI access scan does this for seven AI agents and flags any that get a challenge page or an error.

Read outside tests with one caveat. A test request carries the crawler's user agent but not its IP address, so Cloudflare sees an unverified bot. If your zone lets verified OAI-SearchBot through but challenges a lookalike, the test shows a block the real crawler never hits. The way to settle it is real visit data, which is what the Worker connector collects.

How do I make sure AI search agents are allowed?

Work through the settings in the order requests meet them, and confirm with a fresh test at the end.

1. AI bot policies. In Security Settings > Configure AI bot policies, set Search to Allow. Set Agent to Allow if you want ChatGPT, Claude and Perplexity to fetch pages for people asking about you. Choose Training to taste, preferring Disallow AI Training over Block.
2. AI Crawl Control. In the Crawlers tab, check the Action column for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. Anything you want cited should say Allow.
3. WAF custom rules. Read each of your WAF custom rules and rate limiting rules for conditions that match crawlers: user agent strings, data center ASNs, countries or tight rate limits. Add the skip rule below.
4. Bot Fight Mode. If it's on and the tests still show challenges for AI agents, try a test with it off. It can't be skipped by rule, so off is the only switch on a Free plan.
5. Retest. Run the scan again, or click Check now in Monitor, and confirm each agent gets the same page a browser gets.

How do I write a skip rule for verified bots?

Create a WAF custom rule with the expression (cf.client.bot), the action Skip, and All remaining custom rules ticked, then place it first. This is Cloudflare's own recipe in its bot protection guide, and the cf.client.bot field is true for requests from known good bots.

On Pro and above you can also tick All Super Bot Fight Mode rules in the same rule. Bot Fight Mode on the Free plan isn't affected by skip rules at all.

Want finer control than all verified bots? The cf.verified_bot_category field lets a rule match categories such as AI Search or AI Assistant, which Cloudflare's verified bots docs keep for backward compatibility while new crawlers are classed by behavior.

How does Cloudflare change my robots.txt?

If you turn on Bot Preference Sync, the successor to managed robots.txt, Cloudflare prepends rules to your robots.txt: a Content Signals policy plus Disallow: / groups for training agents such as GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider. It's available on all plans, and it adds nothing when every policy is set to Allow.

The result is a robots.txt that differs from the file in your repository or CMS. Fetch /robots.txt from the live domain to see what crawlers get, and test it with our robots.txt checker. Monitor always reads the live file, so prepended rules show up in its agent list like any other rule.

How do I install the Monitor Worker on Cloudflare?

The connector is a Worker that sits in front of your site, passes every request through unchanged and reports AI agent requests to Monitor in the background. It needs the domain to be proxied through your Cloudflare account (the orange cloud), because Workers routes only run on proxied hostnames.

1. Copy your snippet. The Cloudflare tab on the site's Connect logs section in Monitor has the code with your ingest key filled in. The code below is identical except for the placeholder YOUR_INGEST_KEY.
2. Create the Worker. In Cloudflare, go to Workers & Pages, create a new Worker, open its code editor, replace the starter code with the snippet and deploy.
3. Add a route. In the Worker, open Settings > Domains & Routes, choose Add, then Route. Pick your zone and enter example.com/* with your domain. Add a second route for www.example.com/* if both hosts serve pages.
4. Set the failure mode. The Workers Free plan has a daily limit of 100,000 requests, and every request on the route counts. Choose fail open so traffic over the limit bypasses the Worker instead of seeing Cloudflare's error 1027. Busy sites can move to the Workers Paid plan, which starts at $5 a month with 10 million requests included.
5. Test it. Request a page on your site with an AI user agent, for example curl -A "Claude-User/1.0" https://example.com/. Reload your site in Monitor and the visit appears under Latest visits.

// Cloudflare Worker: add a route for ${host}/* to this Worker.
// It passes every request through untouched and reports AI agent visits in the background.
const ENDPOINT = "https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY";
const AI = /OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-externalfetcher|DuckAssistBot|MistralAI-User|Amazonbot|CCBot|Bytespider/i;

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const ua = request.headers.get("user-agent") || "";
    if (AI.test(ua)) {
      const u = new URL(request.url);
      ctx.waitUntil(fetch(ENDPOINT, {
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({ p: u.pathname, ua, ip: request.headers.get("cf-connecting-ip"), s: response.status, t: Date.now() }),
      }).catch(() => {}));
    }
    return response;
  },
};

How it works: the Worker fetches the original request and keeps the response. If the user agent matches one of the 16 AI agents Monitor tracks, it posts the path, user agent, status and the visitor IP from the CF-Connecting-IP header to Monitor. That post runs inside ctx.waitUntil(), which lets work continue after the response is returned, so visitors never wait for it. Errors in the report are swallowed, so a Monitor outage can't break your site.

The test uses Claude-User because Monitor verifies OpenAI and Perplexity visits against their published IP ranges. A request you send as GPTBot comes from your own IP, fails that check and is dropped, which is the point. More on that in how visits are verified.

How does Monitor watch Cloudflare settings every day?

Cloudflare settings change without a deploy: a teammate enables a policy, a plan upgrade switches on new defaults, a WAF rule meant for scrapers starts matching a crawler. Monitor checks for that every day.

• Every day it requests your homepage as seven AI agents and as a browser. If OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot or Perplexity-User starts getting a challenge or an error twice in a row, you get a critical alert naming the agent and what it received.
• It reads the live robots.txt, including anything Bot Preference Sync prepends, and alerts when a change blocks a search agent.
• With the Worker installed, real visits that pass IP verification within three days override a blocked probe, so a zone that only lets the real crawler through isn't flagged. If an agent that visited every day goes quiet for three days, you hear about it.

Each alert is listed in the alerts reference, and the AI Access Score shows how many points the firewall part is costing you. You can try all of it on the 14-day trial, no card needed.

Frequently asked questions