Markdown for agents and AI content signals: who actually uses them?

How many websites serve Markdown to AI agents?

About one in eight that we could test. We requested each homepage a second time with the header Accept: text/markdown, text/html;q=0.9, which is what an agent sends when it prefers Markdown but will take HTML. 218 homepages answered that request with HTTP 200, and 28 of them (12.8%) sent back a Markdown document instead of a web page. The other 190 ignored the header and returned their normal HTML.

The idea got its push from Cloudflare. In February 2026 it launched Markdown for Agents, an edge feature that converts HTML to Markdown on the fly when a request asks for it, free in beta on Pro, Business and Enterprise plans. Cloudflare's own example made the case in tokens: the launch post took 16,180 tokens as HTML and 3,150 as Markdown, an 80% cut. It also said coding agents such as Claude Code and OpenCode already send these headers. Anything that saves an agent tokens makes your page cheaper to read and easier to quote whole.

The bars above put every signal we checked on one scale. The grey bar is the older tool, a named robots.txt rule that shuts out an AI training crawler, and it is still about three times as common as any of the newer ones. Markdown on request comes next, ahead of the Content-Signal line that Cloudflare also promotes. Everything after that is close to zero.

Which kinds of sites serve Markdown to agents?

Software companies, almost exclusively. 25 of the 28 Markdown responders are in our B2B SaaS group, and the other 3 are developer platforms: GitLab, Cloudflare and WordPress.org. We got no Markdown from any of the 36 news homepages that answered, none from search engines, social networks, retailers, travel sites, universities, streaming services or government sites.

The SaaS names read like a list of companies that sell to developers or to teams buying AI tooling: Asana, Zendesk, Intercom, Okta, Miro, Pipedrive, Webflow, Amplitude, Mixpanel, Deel, Lattice, Brex, Vercel, Netlify, Sentry, PagerDuty, Postman, Wrike, Typeform, Sanity, Algolia, Jasper, Outreach, ZoomInfo and Hootsuite. Their buyers increasingly ask an assistant to compare tools before they open a browser tab, so the incentive is plain.

Sentry did the most interesting thing with it. Its Markdown homepage is a short note telling the agent it has reached a page meant for humans, followed by the configuration for Sentry's MCP server. That's a homepage written for a machine reader, and at 976 bytes it was the smallest response in the set.

Which platforms are serving the Markdown?

Cloudflare, for most of it. The response headers of 16 of the 28 Markdown sites name Cloudflare, 6 name Vercel and 2 name Netlify. The last 4 came from nginx, Google's load balancer or CloudFront, which points to a custom build. Being on Cloudflare made a large difference: 16 of 61 Cloudflare-served homepages (26.2%) returned Markdown, against 12 of 157 elsewhere (7.6%).

You can often tell which route a site took. Cloudflare's converted responses add a Content-Signal header, and 15 of the 28 Markdown responses carried one. Three sites (Cloudflare, Netlify and Hootsuite) also sent an x-markdown-tokens header with a token count. The Vercel-hosted sites, including Vercel itself, Intercom, Brex and Sanity, sent clean Markdown with neither header, so their conversion runs somewhere other than Cloudflare's feature.

How much smaller is the Markdown version of a page?

Very much smaller. The median Markdown homepage was 2.6% of the byte size of the same homepage served as HTML. Across all 28 sites, 17.5 MB of HTML shrank to 0.49 MB of Markdown, or 2.8%. Bytes aren't tokens, and much of a modern homepage's weight is scripts and styling that a parser drops anyway, so treat this as a ceiling on the saving rather than the token bill an agent pays.

The spread says something about what each team decided to keep. Webflow's Markdown homepage was the largest at about 64 KB because it keeps most of the page copy. Cloudflare's own was under 3 KB. Neither choice is wrong, but a very short Markdown file can strip out the product detail an AI answer needs. If you serve one, read it the way an agent would and check that your category, buyer, pricing model and main features survive the conversion.

How many sites use Cloudflare's Content-Signal in robots.txt?

21 of 250 readable robots.txt files (8.4%). Cloudflare introduced the Content Signals Policy in September 2025 as an addition to robots.txt with three yes or no values: search (building a search index and showing links and snippets), ai-input (feeding content into a model's answer, as in grounding or retrieval) and ai-train (training or fine-tuning a model). A line looks like Content-Signal: search=yes, ai-input=yes, ai-train=no.

The split surprised us. More sites used the line to say yes to training (11) than to refuse it (9). The yes group is all software: Cloudflare, monday.com, Intercom, Webflow, Hootsuite, Deel, Netlify, Sentry, PagerDuty, Sanity and Jasper. The refusals come from Stack Overflow, Freshworks, Rippling, Wrike, The Atlantic, Corriere della Sera and three Twilio brands (twilio.com, sendgrid.com and segment.com), which ship the identical line. Amplitude says yes to every crawler it names except Bytespider, which gets no on all three signals plus a disallow.

Almost nobody turns off the answer side. Only Stack Overflow said search=no to every crawler, and only The Atlantic and Corriere della Sera said ai-input=no. Both of those also block several AI crawlers by name, so the signal backs up a rule they already had. The common SaaS pattern, ai-train=no with search=yes and ai-input=yes, reads as "quote us in answers, don't train on us", which is a coherent position for a vendor that wants citations.

None of the 250 files contained Cloudflare's managed block, which Cloudflare's documentation says it prepends only when a site turns on Bot Preference Sync and sets at least one AI policy to block. Cloudflare says the managed robots.txt feature is on for more than 3.8 million domains, but in a sample of large, hand-run sites every Content-Signal line we found was written in by the site itself. Only The Atlantic also pasted the policy's legal preamble into its file.

Do sites still block AI crawlers the old way?

Yes, and by a wide margin. 94 of 250 robots.txt files (37.6%) disallow the homepage for at least one of seven AI training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent or Bytespider), and 64 (25.6%) do the same to at least one AI search or user agent. Only 4 of the 94 training blockers also write a Content-Signal line. Blockers block by name; the new signals are mostly used by sites that want to be read.

The two groups at the ends of the chart behave like mirror images. 45 of 50 news publishers (90.0%) block an AI training bot, and 2 of 50 (4.0%) send any of the new signals. Among B2B SaaS sites with a readable robots.txt, 10 of 93 (10.8%) block a training bot, while 31 of 93 (33.3%) send a Content-Signal line, serve Markdown, or both. Every other category sits close to the news pattern on new signals (4 of 107, 3.7%) and in between on blocking (39 of 107, 36.4%).

That fits what we found in our robots.txt study of the same 285 domains earlier this month: news publishers make up close to half of the blockers (45 of the 94 here), and for them the content is the product. Software companies sell something else, so being quoted by an assistant costs them nothing and can bring them buyers. The new signals give that second group a way to say so out loud.

Do any sites send contradictory signals?

Yes, and the cause is defaults. 15 Markdown responses carried a Content-Signal header with ai-train=yes, the value Cloudflare's Markdown for Agents adds to converted pages. For 8 of those sites (GitLab, Zendesk, Okta, Outreach, Pipedrive, Lattice, Typeform and Algolia) that header is the only content signal they send anywhere, so their only stated position on training may be one they never chose.

Wrike is the clearest conflict. Its robots.txt says Content-Signal: ai-train=no, search=yes, ai-input=yes. Ask its homepage for Markdown and the response header says ai-train=yes, search=yes, ai-input=yes. An agent that reads the header and never fetches robots.txt gets the opposite answer from one that does. Cloudflare's launch post says custom Content Signal policies for Markdown responses are planned; until a site sets one, the default speaks for it.

A second mismatch is in the names. The IETF's AI preferences working group drafts a vocabulary with the labels train-ai, ai-use and search, and a separate attachment draft that puts them in a Content-Usage robots.txt rule or HTTP header, such as Content-Usage: train-ai=n. Cloudflare's line says ai-train, the IETF says train-ai. We found the Cloudflare form 21 times and the IETF form zero times.

Does anyone use noai, ai.txt or TDMRep?

Barely. These are the older proposals for telling AI systems what they may do with a page, and our sample shows they never caught on outside a few publishers.

• noai and noimageai. DeviantArt introduced these robots directives in November 2022. We found them in the meta robots tags or X-Robots-Tag headers of 0 of 219 homepages.
• ai.txt. Spawning proposed a robots-style file at /ai.txt for text and image training permissions. 0 of 285 sites served one (several returned a normal web page at that path, which we did not count).
• TDMRep. The W3C community group's TDM Reservation Protocol lets a rightsholder reserve text and data mining rights, a technical answer to Article 4 of the EU copyright directive. 4 sites use it: Nature and ScienceDirect, and the European newspapers Le Figaro and Corriere della Sera. Three publish a valid /.well-known/tdmrep.json file, ScienceDirect also sends a tdm-reservation header, and Corriere uses the meta tag.
• Content-Usage. The IETF form described above: 0 of 250 robots.txt files and 0 homepage headers.

The pattern across all four is the same. Proposals that a single company or community put forward without a large platform switching them on stayed at or near zero. Content-Signal and Markdown for Agents both had Cloudflare behind them and made it into double digits among software companies within about a year.

Do sites use snippet controls to keep content out of AI Overviews?

Not on their homepages. Google's robots meta tag documentation says nosnippet applies to AI Overviews and AI Mode and stops the content being used as a direct input for them, and that data-nosnippet can fence off parts of a page. 0 of 219 readable homepages used nosnippet. 16 (7.3%) used data-nosnippet on at least one element, usually once, with Linear the outlier at 32 marked elements.

The snippet directive we did see points the other way. 17 homepages (7.8%) set max-snippet:-1, which tells Google there's no length limit on snippets, a permission rather than a restriction. They include Salesforce, Zendesk, CrowdStrike, Rippling, Harvard, NASA and TechCrunch. It may well come from a template or plugin rather than a decision anyone made about AI.

Homepages are the wrong place to look for page-level opt-outs, to be fair. A publisher that wants to keep paywalled articles out of AI Overviews would mark the articles, not the front page. What the homepage data does show is that no large site in our sample uses snippet controls as a blanket AI opt-out.

Which AI signals should your site send?

It depends on whether you want to be read or left alone, and the data gives a fairly clear rule for each case.

• Pick Markdown on request if you sell to buyers who research with AI assistants and your site runs on a platform that makes it a toggle or a small route. Then fetch your own homepage with curl -H "Accept: text/markdown" https://yoursite.com/ and read what comes back.
• Pick a Content-Signal line if you want a written record of your position on training and grounding. The SaaS default we saw most among refusers was search=yes, ai-input=yes, ai-train=no. If you also serve Markdown through Cloudflare, check the response header says the same thing as robots.txt.
• Pick named Disallow rules if you need a crawler to stay out. That's what 90% of news publishers in our sample do, and it's the mechanism the crawler operators document. Our guide to AI crawlers and robots.txt lists the user agents and which ones serve search versus training.
• Skip noai, ai.txt and Content-Usage for now unless a partner asks for them. Nobody in our sample uses the first two, and the IETF syntax is still a draft.
• Pick TDMRep if you publish in the EU and want a machine-readable text and data mining reservation. Academic and news publishers are the only users we found.

Whatever you send, send it once and consistently. The most common fault we found was a site saying one thing in robots.txt and another in a header its CDN added on its behalf.

How we collected the data

Sample. The same 285 domains in 10 categories we used for our robots.txt study: 102 B2B SaaS companies (including G2, Capterra and Gartner), 53 news and media publishers from several countries, and 130 large sites across search and portals, social and community, video and streaming, e-commerce, travel, reference and education, tech and developer, and government. The list was built by hand, one entry per domain. Reusing it lets you compare the two datasets row by row.

Collection. On 28 October 2026 a Python script ran curl against every domain with a desktop Chrome user agent, a 25-second timeout, redirects followed and 16 requests in flight. Each domain got five requests: /robots.txt, the homepage as HTML, the homepage with Accept: text/markdown, text/html;q=0.9, /.well-known/tdmrep.json and /ai.txt. We kept the final response headers and bodies. We did not retry blocked requests or try to get past bot protection.

What counted. A robots.txt file was readable if it came back as plain text (not an HTML page) with HTTP 200, or with another status but a body that contained a User-agent line: 250 of 285. A homepage was readable if it returned HTTP 200 with HTML: 219. Most of the rest were 403, 429 or 202 responses from bot protection, concentrated among publishers and retailers. Markdown counted when the response to the Markdown request was HTTP 200 with a text/markdown content type; we checked by hand that all 28 bodies were Markdown and not HTML. Content-Signal and Content-Usage were read from uncommented robots.txt lines. Meta directives came from meta tags named robots, googlebot, bingbot or a named AI crawler, plus the X-Robots-Tag header. ai.txt counted only as plain text with at least one User-agent, Allow or Disallow line. TDMRep counted as a valid JSON array at the well-known path, a tdm-reservation meta tag, or a tdm-reservation header.

Blocking. We parsed each robots.txt with Python's standard robots parser and asked whether each of 13 AI user agents could fetch the homepage. Training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent and Bytespider. Search and user agents: OAI-SearchBot, Claude-SearchBot and PerplexityBot, plus the user-triggered ChatGPT-User, Perplexity-User and Claude-User.

Platform and size. The serving platform comes from response headers (server, cf-ray, x-vercel-id, x-nf-request-id). Sizes are uncompressed body bytes of the two homepage responses fetched minutes apart. Every number on this page was computed by script from the CSV.

Limitations

• We tested homepages. Some sites may serve Markdown or apply nosnippet on documentation, blog or article pages but not on the front page, so page-level figures are a floor.
• 66 homepages didn't return a readable page to our fetcher, most of them publishers and retailers behind bot protection. Shares are computed on sites that answered, so the news and e-commerce figures rest on smaller groups.
• Requests came from one network location with a browser user agent. A site could treat real AI crawlers, which fetch from their operators' published IP ranges, differently from us.
• A Content-Signal or Content-Usage line records a preference. We measured what sites say, not whether any AI company honors it.
• The sample leans toward large, well-known sites. Small sites on default Cloudflare settings may look quite different, which is why the managed robots.txt figure here (0 of 250) shouldn't be read as a web-wide rate.

Frequently asked questions

Dataset: markdown-for-agents-study-data.csv (285 rows, one per domain, collected 28 October 2026). The license is CC BY 4.0: quote, chart or rework the data freely, with credit to AI Ranked and a link to this study.