Webflow and AI crawlers: what you can control and how to monitor it

Can you see AI crawler visits on a Webflow site?

Not directly. Webflow hosts and serves your site, delivered through Cloudflare, and its help center doesn't describe any way for customers to download server or access logs. Without logs, nothing on a standard plan records a visit from GPTBot or ChatGPT-User, and analytics tags can't either, since AI crawlers don't run the JavaScript those tags rely on.

Webflow does read its own server logs for one feature: LLM bot insights, which reports visits from agents such as ChatGPT-User, Claude-User, Perplexity-User, GPTBot and OAI-SearchBot. It requires a Team or Enterprise plan with the newest version of the Enterprise Analyze add-on.

For everyone else there are two routes, and they combine well: monitoring from outside, which works on any plan, and a Cloudflare Worker if your domain can run through your own Cloudflare zone.

How do Webflow's bot toggles and robots.txt rules work?

Both live in Site settings > SEO > Indexing. Webflow's indexing docs describe traffic control toggles that write robots.txt disallow rules for groups of bots. The AI group covers agents including GPTBot, ChatGPT-User, ClaudeBot, Claude-User and PerplexityBot, while OAI-SearchBot sits in the search engine group. Both groups are allowed by default.

That grouping is worth knowing before you flip a switch. Turning off the AI group to refuse training also blocks ChatGPT-User, Claude-User and PerplexityBot, which fetch pages for answers and citations. If you only want to refuse training, write targeted rules in the robots.txt box instead, following the robots.txt rules guide: add the rules, save and publish. Our AI crawler list has rules that block training while keeping AI search open.

Two details from Webflow's docs: once a robots.txt exists it can't be removed completely, only replaced with allow rules, and custom robots.txt rules need a Site plan or a paid Workspace plan.

Which Webflow settings can add noindex by mistake?

Two switches add a noindex to pages on your live domain, and each one hides those pages from AI search as well as Google. Both are described in the same Webflow indexing guide:

• Sitemap indexing in each page's SEO settings adds <meta content="noindex" name="robots"> to that page when switched off. It's easy to switch off on a draft and forget when the page goes live.
• Sitemap Indexing on CMS items does the same per collection item on paid Site plans, so a single blog post or case study can drop out.

A third setting, Staging indexing, only covers the webflow.io staging domain, so it won't hide your custom domain.

Noindex can also arrive through custom code in the page head. Monitor reads the robots meta tag and the X-Robots-Tag header on every key page, so a noindex from any of these sources shows up as a critical alert on the next check.

How do I add llms.txt to a Webflow site?

Upload it under Site settings > SEO > LLMs.txt. Webflow's llms.txt docs say the file must be UTF-8 and under 100 KB, and that it's served at /llms.txt on your custom domain but not on the staging domain.

An llms.txt file is a short markdown map of your most important pages. You can draft one with our llms.txt generator. Monitor checks that the file still loads every day and alerts you if it starts returning an error page.

What can you monitor on Webflow without logs?

Everything about access, because access is visible from outside. AI Ranked Monitor needs nothing installed in Webflow: you add your domain and it checks what crawlers can reach.

• robots.txt, read live every day and evaluated for 18 AI agents. A toggle flipped in Site settings shows up as "robots.txt now blocks" with the agents named.
• Firewall response, by requesting your homepage as seven AI agents and comparing each answer with a browser's.
• Your key pages, as many as 100 taken from the Webflow sitemap, checked for status, redirects, noindex, canonical, words in the raw HTML and JSON-LD types. A page republished with indexing off, or a CMS template that loses its schema, is caught the next day.
• llms.txt, checked for availability and changes.

What you don't get without logs is visit data: which agents actually came and which pages ChatGPT, Claude and Perplexity fetched for real people. That needs the next section.

How do I log real AI visits on Webflow through Cloudflare?

Run your domain through your own Cloudflare zone with Orange-to-Orange (O2O), then add the Monitor Worker on that zone. It's the one way to see each request on a Webflow site without Webflow's own add-on.

The standard Webflow connection doesn't allow it. Webflow's Cloudflare connection guide says to set the proxy status to DNS only for standard hosting, since proxying the usual way causes a 525 handshake error. Webflow supports proxying through Cloudflare O2O instead: proxied CNAME records for @ and www pointing to cdn.webflow.com, on a paid Webflow Site plan and any Cloudflare zone plan, including Free.

In O2O your zone's settings apply first, and Cloudflare's O2O compatibility table lists Workers as supported on the customer zone, with a note that a Worker matching the hostname can interfere with visitors. The Monitor Worker forwards every request unchanged, so it passes traffic through to Webflow and only adds a background report for AI agents.

How do I set up the Worker for a Webflow site?

1. Set up O2O. Follow Webflow's O2O guide so your domain's CNAME records are proxied through your Cloudflare zone, with SSL set the way the guide describes, and confirm the site loads.
2. Create the Worker. Copy the Cloudflare snippet, key included, from your site's connector section in Monitor. In Cloudflare, create a Worker, paste it and deploy. The full walkthrough is in our Cloudflare guide.
3. Add routes. Add routes for example.com/* and www.example.com/* on your zone, and set them to fail open so the site keeps working if the free Workers limit is reached.
4. Test. Run curl -A "Claude-User/1.0" https://example.com/, then reload your site in Monitor and check Latest visits.

Once your zone proxies the site, Cloudflare's bot settings apply to it as well. Check your AI bot policies and Bot Fight Mode so the new layer doesn't start blocking the agents you want to see.

What does Monitor watch on a Webflow site every day?

Webflow sites change through the Designer and CMS rather than code deploys, and anyone with editor access can flip an SEO setting. Daily checks catch those changes no matter who made them:

• A bot group toggled off or a Disallow rule added in Site settings becomes an alert naming each agent that lost access.
• A page or CMS item published with indexing off becomes a critical noindex alert for that path.
• A template change that removes JSON-LD, moves copy into a script-rendered embed or changes canonicals shows up as a warning on the pages affected.
• With O2O and the Worker in place, you also get verified AI visit data and an alert when a regular visitor goes quiet.

Try it on the 14-day trial, with no card, or start with the setup guide.

Frequently asked questions