AI crawler blocking and AI Overview sources: what changed this month

This is the first AI Ranked monthly report. It reruns two of our studies against fixed panels: the robots.txt and llms.txt files of the 285 sites in our AI crawler access study, and a 50-query slice of the queries behind our AI Overview citations study. Each month we compare the new pull with the one before it.

How much did AI crawler blocking change in a month?

Very little in total, and a lot at a few sites. Of the 259 major websites whose robots.txt we read in October and again on 1 November 2026, 8 changed a rule for at least one of the 14 AI user agents we test. Those 8 sites added 22 agent blocks and lifted 2, so the headline rates moved by about one percentage point.

The share of sites that block one or more AI training crawlers went from 34.7% to 35.9% (90 to 93 sites). The share blocking at least one AI search crawler (OAI-SearchBot, Claude-SearchBot or PerplexityBot) did not move at all: 55 sites in October, 55 in November. Sites that block none of the 14 agents fell from 169 to 166, still 64.1% of the panel.

The pattern matches what we saw when we first published the study. Blocking is mostly all or nothing, and the sites that move tend to move several agents at once. Facebook, Instagram and AliExpress each added six blocks. Only two changes went the other way, and both were single agents at news sites.

All rates in this report use the same parser and the same rule as the original study: an agent counts as blocked when the path / is disallowed for it under RFC 9309. Comparisons use only the 259 sites we could read in both months, so a site that refused our fetcher this month cannot fake a change in the rate. If you want the same check on your own file, our robots.txt AI crawler checker uses the parser behind these numbers.

Which AI crawlers gained blocks this month?

GPTBot, ClaudeBot and Amazonbot gained the most, with three new blocking sites each. Google-Extended, Applebot-Extended and PerplexityBot gained two. OAI-SearchBot, ChatGPT-User and Claude-User did not change. ClaudeBot remains the most blocked training crawler run by a major AI lab, at 69 sites, and Common Crawl's CCBot and ByteDance's Bytespider still top the list at 75 each.

The order of the list barely changed. Training crawlers still sit at the top and the three AI search crawlers near the bottom, with OAI-SearchBot the least blocked AI agent of all at 11.6%. That gap is the single most useful number in this panel for a marketer. Blocking GPTBot keeps your pages out of OpenAI's training data. Blocking OAI-SearchBot keeps them out of ChatGPT search answers. The big sites in our panel overwhelmingly accept the second while refusing the first: 31 of the 61 sites that block GPTBot still let OAI-SearchBot in.

Googlebot is the control and it stayed flat. One site in the panel disallows it at the homepage, the same as last month. If your crawler report ever shows Googlebot blocked at /, treat it as a mistake until proven otherwise. For the roles of each Anthropic bot, see our ClaudeBot guide; for the full list of tokens and copy-paste rules, see AI crawlers and robots.txt.

Which sites changed their robots.txt rules?

Eight sites, and three of them did most of the work. Facebook, Instagram and AliExpress each added six AI agents to their disallow lists. IKEA, CNBC and Hindustan Times added one or two. CNN and Der Spiegel each removed one block. We confirmed each change by fetching the live file a second time.

Meta's two apps. In October, Facebook and Instagram already blocked OAI-SearchBot, Claude-SearchBot and ChatGPT-User, but our parser found GPTBot, ClaudeBot and PerplexityBot allowed at the homepage. Their November files name each of those three in its own group with Disallow: /, alongside Google-Extended, Applebot-Extended and Amazonbot. Both files open with the same notice that automated data collection is prohibited without written permission.

AliExpress. The marketplace went from allowing every AI agent to blocking six training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot. It left OAI-SearchBot, Claude-SearchBot and PerplexityBot open. That is the exact split our October study recommended for sites that want to stay out of training data but still show up in AI search. Together with IKEA, it lifted the e-commerce category from 5 to 7 training blockers.

CNBC. CNBC already blocked all three OpenAI agents and PerplexityBot. It now puts Claude-SearchBot and Perplexity-User in a group that allows only one section and disallows the rest, which makes CNBC one of 15 sites that block all 14 agents at the homepage. In October CNBC was one of five sites that blocked OAI-SearchBot while allowing Claude-SearchBot. That asymmetry is gone.

The two removals. CNN no longer names Google-Extended, and Der Spiegel no longer names Amazonbot, so both now fall back to their general rules, which allow the homepage. We can't tell from a robots.txt file why a rule was removed. In CNN's case the timing is interesting, because Google said on 30 October that it will read a shared AI training opt-out as a Google-Extended block (more in the news section below), but nothing in the file links the two.

Smaller edits. IKEA added a named Amazonbot group with Disallow: /. Hindustan Times added meta-externalagent. Neither touched any AI search crawler.

Did any site block AI search crawlers while allowing training?

No. In both months, every site that blocks at least one AI search crawler also blocks at least one training crawler. The reverse is common: 38 sites block training while leaving all three AI search crawlers open. Nobody in our panel shuts out search and invites training.

That ordering tells you how large sites think about AI access. Training is the thing they refuse first. Search access, which is what puts a page in a cited answer, is the thing they give up last, and almost only news publishers give it up at all. In November, 33 of the 55 sites that block an AI search crawler were news and media sites. B2B SaaS, the category most of our readers sit in, has exactly 1 such site out of 94.

If your own file blocks OAI-SearchBot, Claude-SearchBot or PerplexityBot and you sell software, you're in a group of one. Check whether that was a decision or an accident, for example a blanket Disallow: / under User-agent: * with a short allow list that forgot the search bots.

How did blocking change by category?

News publishers still block far more than anyone else, and the month's movement came from social platforms and e-commerce. News sites blocking a training crawler went from 42 to 43 of 48. E-commerce went from 5 to 7 of 20. B2B SaaS, tech and government did not change.

Social and community looks flat on the training column because Facebook and Instagram were already counted there in October, through other agents. The GPTBot column shows the real move: 8 to 10 of 16 social sites now block it. B2B SaaS stays the most open category in the panel, with 4 of 94 blocking GPTBot and 1 blocking an AI search crawler.

Did llms.txt adoption grow?

Slightly. 83 of the 244 sites that gave a definite answer in November serve a valid llms.txt, or 34.0%. On the 240 sites with a definite answer in both months, the count went from 80 to 82: Substack and Business Insider published files, and no site took one down.

We count a file only when the response is HTTP 200, the content type is text/plain or text/markdown, and the body opens with a Markdown H1. "Definite answer" means the server returned 200, 404 or 410; the rest refused or failed, so we leave them out. Four sites moved from no answer to a definite one this month and Writer went the other way, which is why the November denominator is larger.

B2B SaaS still leads by a wide margin. The proposal behind the file is published at llmstxt.org; whether any AI engine reads it is a separate question, and our study of AI content signals found the newer Markdown and Content-Signal mechanisms used mostly by SaaS sites as well.

Which sites stopped answering our crawler?

Two. LinkedIn now answers our robots.txt request with a reCAPTCHA challenge page instead of a text file, and Writer redirects it to a "Request blocked" page from its web application firewall. Both were readable in October. Every other site that was readable last month was readable again.

This is the limit of any robots.txt panel. A firewall decision is invisible in the file, and it can change without a single line of robots.txt changing. If the AI crawler you care about gets the same treatment as our fetcher, it never reads your rules at all. That's the gap AI Ranked Monitor is built to catch: it checks your key pages every day the way AI crawlers reach them and sends an alert when something blocks them. We exclude both sites from this month's comparisons, so the rates above are on the 259 sites read twice.

How much did AI Overview citations change in a month?

A lot. We pulled the latest Ahrefs snapshot for 50 US B2B software queries from our AI Overview panel. 27 had been recaptured since our October pull, and on the 25 of those with an on-topic AI Overview in both months, Google cited 151 sources in October and 180 in November. Only 64 URLs, or 42.4%, appeared both times.

That turnover lines up with what others have measured over shorter windows. Ahrefs' AI Overview volatility study of 43,000+ keywords found a 54.5% URL overlap between consecutive AI Overview responses for the same keyword, with checks spaced days apart. Our gap is about a month, so a lower overlap is what you'd expect.

Presence held steady. 45 of the 50 queries returned AI Overview rows in November, the same count as in October, but not the same queries. "bamboohr vs rippling" showed an AI Overview for the first time in our data, and "email marketing software for ecommerce" lost the one it had. Of the rest, 18 queries had no new snapshot since October, and for each of those, our November pull matched the October citations URL for URL. That check matters: it tells us the churn above comes from Google, not from our method.

The average refreshed AI Overview also got longer. It cited 6.0 sources in October and 7.2 in November on the same 25 queries. More slots, and most of them filled by pages that weren't there a month ago, is the mechanism behind the lower overlap.

Which source types gained and lost AI Overview citations?

YouTube and Reddit gained, and software vendors and publishers lost share. On the same 25 queries, YouTube went from 24 citations (15.9%) to 40 (22.2%) and Reddit from 6 to 17. Vendors' own sites still earned the most citations, 70, but their share fell from 43.7% to 38.9%. Publishers fell from 21.9% to 16.1%.

Vendors losing share isn't the same as vendors losing slots: their count went from 66 to 70. The longer answers made room for everyone, and video and forum threads took most of the net growth: the panel gained 29 citations, and YouTube plus Reddit, forums and social gained 25 of them. Reddit, forums and social together rose from 11 to 20 citations (7.3% to 11.1%).

Across all 50 panel queries, counting the snapshots that didn't change, the most cited domains in November were YouTube (60 citations) and Reddit (25), followed by gartner.com (11), zapier.com (9), g2.com (7), linkedin.com (4). In October the same 50 queries gave YouTube 40 and Reddit 15. Our full AI Overview study has the source breakdown for all 150 queries and the method behind the source types.

This is the main reason we track AI Overview sources monthly rather than once. A study tells you who Google cited during one window. A tracker tells you how fast that list goes stale. If you report AI Overview visibility to anyone, a monthly AI Overview tracker will catch losses that a quarterly audit would miss entirely.

Which queries changed the most?

Three queries kept none of their October sources: "crm for startups", "trello alternatives" and "freshdesk alternatives". Two of the three are alternatives queries. At the stable end, "clickup vs notion" and "best cybersecurity software" kept all 3 of their October sources, and "accounting software for nonprofits" kept 8 of 13.

One query fell out of the comparison for a different reason. In October, the AI Overview for "intercom alternatives" cited Crisp, Decagon and a Reddit thread about replacing Intercom, the customer messaging software. The November snapshot answers a different question: it cites an Amazon search for home intercom systems, an electrician's page about replacing old home intercoms, a building-entry vendor and a Reddit thread from a low-voltage wiring forum. Google read the query as hardware. We flagged that snapshot as off-intent and left it out of every source share above, but it is a real result, and a useful warning if your brand name is also a common noun.

Which AI search changes mattered in October 2026?

Three announcements in the last days of October change how robots.txt and AI access work, and a handful of studies changed what we know about AI Overview coverage and Claude's search. Each item links to its source.

• Google will read a shared AI training opt-out as a Google-Extended block. On 30 October, Google said it plans to support a shared AI training "policy token" in robots.txt, based on work from the IETF's AIPREF working group. When a site opts out of training with that token and does not name Google-Extended, Google plans to treat it as if Google-Extended were disallowed. Company-specific controls keep working alongside it. Google Search Central
• Bing splits AI training from AI answer grounding. Microsoft announced separate training and grounding preferences, including a new DisallowExtended option that limits grounding to Microsoft-operated experiences, such as Copilot and Microsoft 365, and excludes grounding services Microsoft provides to third parties. It says it will honor the new preferences as it crawls starting 1 February 2027. Bing blog
• Google's Gary Illyes says AI training scrapers made fake Googlebot traffic worse. Companies gathering training data pose as Googlebot to get past sites that would block them, he said, and firewalls catch most but not all. It's a reminder that a user-agent string proves nothing without an IP check. Search Engine Roundtable
• AI Overviews now appear on most searches, by several counts. Search Engine Roundtable rounded up new data from Semrush, SE Ranking and seoClarity on 26 October: AI Overviews show for over 80% of Google searches in Semrush's tracking, SE Ranking found them on 82.5% of US brand searches, and seoClarity puts them above 62% of queries, varying by industry. Search Engine Roundtable
• Rank trackers are fighting "poisoned" Google results. Ahrefs estimates that 5% to 20% of search results have been affected over recent weeks, depending on the data provider, and says it filters abnormal results and requests replacements. Our AI Overview panel uses Ahrefs snapshots, so we note it as a limitation below. Search Engine Journal
• Claude's default web search appears to have moved to a new index. Writesonic reviewed 648 Claude search calls and found that from 5 October every call carries a standard or extended mode, and that the two modes share just 10% of cited domains. Writesonic
• Google moves inline links higher in AI Mode and AI Overviews. Google says it has substantially increased inline links in its AI answers and will show them higher in responses more often, and is testing a section at the top of some answers for the official pages a task needs. Google
• Ads in AI Mode reach 10 more countries. On 28 October, Google said ads in AI Mode are expanding to Australia, Canada, India, Kenya, Malaysia, New Zealand, Nigeria, Pakistan, the Philippines and Singapore, with Brazil, Mexico and the UK planned for Q4, and that multiturn ads on follow-up questions are launching in the US. Google
• Google's "control what you share" page now lists the Search generative AI control. The help page groups the robots meta tag, data-nosnippet, X-Robots-Tag and the generative AI control in Search Console as the ways to limit what Google shows. Google Search Central

The first two items matter most for anyone who maintains a robots.txt file. Until now, opting out of AI training meant naming every crawler you could find. A shared token, if more providers adopt it, makes the training decision one line, and Bing's grounding split makes the search decision separate from it.

What should you do with this month's numbers?

1. Split your robots.txt decision in two. Decide training and search separately. The big sites in our panel block training first and search last, and AliExpress made exactly that split this month. If you want to be cited in ChatGPT, Claude and Perplexity answers, keep OAI-SearchBot, Claude-SearchBot and PerplexityBot open even if you block GPTBot and ClaudeBot.
2. Check your file against the live response, not your CMS. LinkedIn and Writer show that the thing a crawler gets can be a challenge page while the file itself is unchanged. Fetch /robots.txt as each AI agent and confirm you get text back, or run the free checker on the file and a firewall test on the homepage.
3. Plan for the shared training token. When Google and Bing start honoring the new preferences, a single training opt-out can cover several providers at once. Write down today which providers you block by name, so you can tell whether adopting the token would block more than you intend.
4. Track AI Overview sources monthly, per query. With fewer than half of last month's sources still cited, a snapshot from one quarter is close to useless by the next. Track the queries that matter to you every month and record which URLs appear, not just whether an AI Overview showed.
5. Earn a place on YouTube and Reddit for your category terms. On our panel those two gained more citations in a month than any other source type. A product walkthrough video and honest participation in the threads that rank for "X alternatives" are now part of AI Overview visibility, not a side channel.
6. Watch for query drift on ambiguous names. If your brand or category shares a word with something else, as Intercom does, check what Google thinks your query means before reading anything into citation counts.

How did we collect the data?

Crawler panel. The panel is the same 285 domains in 10 categories as our October AI crawler access study. On 1 November 2026, a Python script requested https://<domain>/robots.txt and /llms.txt with the same research user agent as the original crawl, a 10-second timeout, redirects followed (including 307 and 308), gzip bodies decoded, at most 8 requests in flight and one retry, then a try on the www host. A 200 response with a text body was parsed; a 404 or 410 counts as no robots.txt, which RFC 9309 treats as permission to crawl. 259 domains were readable both months; LinkedIn and Writer were readable only in October, and the remaining 24 were unreadable in both.

Parser. Unchanged from the original study. Consecutive User-agent lines form one group, groups naming the same agent are merged, an agent with no named group falls back to *, the longest matching rule wins, Allow wins a tie, and * and $ work as wildcards. An agent counts as blocked if / is disallowed. Before comparing, we recomputed the October figures from the published October dataset and matched every headline number in the original study. Every site-level change listed above was confirmed with a second fetch.

Agents. Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Amazonbot. AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. User fetch: ChatGPT-User, Claude-User, Perplexity-User. Control: Googlebot.

llms.txt. Counted as valid when the response was HTTP 200, the content type text/plain or text/markdown, and the body opened with a Markdown H1. Sites that refused or failed are left out of the denominator.

AI Overview panel. 50 of the 150 queries in our AI Overview study, every third query in the original list, which keeps all 10 software categories and all six query patterns. On 1 November 2026 we pulled the latest US snapshot for each from the Ahrefs API v3 serp-overview endpoint, filtered to AI Overview and AI Overview sitelink rows. Snapshot dates run from 21 September to 31 October 2026. A query counts as refreshed when its snapshot is newer than the one we used in October; 27 were, 18 were not, and 5 returned no AI Overview rows. URLs were deduplicated per query, with YouTube links compared without their timestamp parameter, and one Google Shopping link was dropped.

Source types. Domains already classified in the October study keep their type. The 40 domains new to the panel were classified by hand from their homepages; the full list and the type of every citation are in the CSV.

What are the limitations?

• robots.txt is a request, not an enforcement mechanism. A site that allows a crawler here may block it at the firewall or CDN, and that never shows up in the file.
• We see one robots.txt per site, the one served to our research fetcher. Some sites serve different files to different requesters.
• We test the homepage path only. Sites that block sections are under- or over-counted for deep pages.
• The crawler panel is a curated list of well-known sites, not a random sample of the web. A month-over-month change of one or two sites is a real event at that site, not a trend.
• The AI Overview panel depends on when Ahrefs recaptures each SERP. 18 of the 50 queries had no new snapshot since October, so this month's churn figures rest on 25 queries. Ahrefs has also said some scraped results across the industry were "poisoned"; it filters abnormal results, and we can't rule out that a snapshot was affected.
• An AI Overview can differ between two searches minutes apart. Each snapshot is one observation, not the answer every user saw that day.
• Source types are our classification. A different scheme would move individual citations between buckets, though not the YouTube and Reddit counts, which are by domain.

Frequently asked questions

Datasets: ai-search-monthly-report-crawlers.csv (285 rows, one per site, same columns as the October file) and ai-search-monthly-report-ai-overviews.csv (300 citation rows for the 50-query panel). Licensed CC BY 4.0. The October baselines are on the crawler study and AI Overview study pages.