How much AI bot traffic does a new site get? We logged every request

How much AI bot traffic does a small site get?

About 150 requests a day, every day, to a site with 83 URLs and few links. Between 25 October and 3 November 2026 we logged 1,453 requests from 11 AI agents. On the 9 full days in that window the count ran from 116 to 220, with a median of 150.

That volume surprised us because the site is so small. AI Ranked launched in October and has logged its own AI bot requests since 25 October. You don't need a big brand to attract these bots. We did nothing special for them beyond an open robots.txt, a sitemap and IndexNow pings, and the count was steady from the first full day of logging.

Plenty has been written about bot traffic at internet scale. Thales' 2025 Imperva report put automated traffic at 51% of all web traffic, and Cloudflare publishes crawler shares across its network. Those figures tell you about the web. They can't tell you what happens on one site that isn't a household name, so we published ours request by request.

Do AI bots crawl more than Googlebot?

On this site, yes, by a wide margin. In the 8 days from 26 October to 2 November, AI bots made 1,155 requests and Googlebot made 296, a ratio of 3.9 to 1. AI bots came out ahead on all 8 days.

The comparison leans in Google's favor. Search Console's Crawl stats count every file Googlebot pulls, and only 46% of the requests in that report were HTML. Our log skips the /assets/ folder altogether. If we strip robots.txt and the sitemap from the AI side as well, 830 page requests remain, which is still 2.8 times Googlebot's total.

Google's AI products aren't in the AI count. AI Overviews and AI Mode work from the Search index, so their reading shows up under Googlebot, and Google-Extended is a robots.txt token with no crawler of its own. So this compares Google's one crawler with everyone else's AI agents added together. Bingbot isn't in either number.

Which AI bots send the most traffic?

OpenAI's two search-side agents lead: OAI-SearchBot made 406 requests (27.9%) and ChatGPT-User 311 (21.4%). ClaudeBot (240) and PerplexityBot (228) come next, then Amazonbot (127). OpenAI's three agents together account for 734 requests, or 50.5% of everything we logged.

Five of the 16 agents we track never appeared: Claude-SearchBot, Perplexity-User, meta-externalfetcher, DuckAssistBot and CCBot. A silent agent in a log this short proves little, but site owners who write rules for every bot on a list should know that a third of the list may not be visiting at all.

What kind of AI bot traffic is it: training, search or user fetches?

Most of it is search and answer traffic. AI search crawlers made 675 requests (46.5%), training crawlers 453 (31.2%) and user-triggered fetchers 325 (22.4%). That's close to the reverse of what Cloudflare reported across its network.

The gap at agent level is just as wide. GPTBot held 28.1% of AI-only crawler traffic on Cloudflare in July 2025, and OAI-SearchBot 2.2%. Here GPTBot is 1.2% and OAI-SearchBot 27.9%. These aren't like-for-like numbers. Cloudflare's come from a year earlier, cover its whole network and may sort agents differently, and a page about AI search is exactly the kind of page an answer engine would look up. Still, if your mental picture of AI crawlers is a training bot vacuuming your archive, that picture didn't describe this site.

How much AI bot traffic actually reads your pages?

About three quarters. 391 of the 1,453 requests (26.9%) were for robots.txt or the sitemap, and the other 1,062 (73.1%) were for pages and files. The average hides how differently each bot behaves.

ClaudeBot is the clearest case. 224 of its 240 requests (93.3%) were control files, one robots.txt and one sitemap fetch roughly every two hours. A bot that looks like the third busiest on a dashboard read 16 pages and files in 10 days. ClaudeBot is Anthropic's training crawler, and what it does read is telling: 9 of those 16 requests went to pages and CSV files we had published the same day. It behaves like a bot watching the sitemap for changes, though we can only infer that from the timing.

GPTBot is quieter still. It took the sitemap once a day at about 07:00 UTC and fetched pages in one minute of the whole window, 8 URLs at 22:21 UTC on 26 October, all of them the homepage, docs or integrations pages. It didn't request a single guide or study. If you run a robots.txt rule for GPTBot and expect to see it working through your site, check your log first.

OAI-SearchBot goes the other way. 84.2% of its requests were pages, and it comes back to the same ones. It fetched one tool roundup 50 times in 10 days and our Semrush review 27 times, while the median page it touched got 5 visits. Two pages took 22.5% of its page requests. We don't know why those two, and repeat fetching isn't proof of citation, but it does show that OpenAI's crawler keeps a short list of pages it re-checks.

How much of a site do AI bots reach?

All of it, once you add them up. Every one of the 83 sitemap URLs was fetched by at least one AI bot, and PerplexityBot fetched all 83 by itself. Coverage per bot ranges from 100% to under 10%.

PerplexityBot got there in a sweep. On 3 November it fetched 79 different URLs in 49 minutes, from 02:02 to 02:50 UTC, one request each. That single pass is 34.6% of all its requests in the window. Amazonbot reached 79 URLs at a slower pace. Among the big crawlers, the lowest coverage belongs to the two you hear about most, ClaudeBot at 9 URLs and GPTBot at 8.

By section, guides took 478 of the 1,062 page requests (45.0%) and the homepage 242 (22.8%). Adjusted for size, research did best at 15.6 requests per URL, ahead of guides at 11.4 and tools at 7.6. Our news page drew 5 requests and the glossary 4, even though both change often. Only 4 requests in the whole log asked for llms.txt, one each from ChatGPT-User, Amazonbot, PerplexityBot and Bytespider.

The 31 malformed requests are a small, odd finding. About two hours after Amazonbot fetched our llms.txt, it started requesting addresses like /guides/profound-vs-semrush):, with the closing bracket and colon from the Markdown link still attached. All 16 of those paths match links in that file. Whatever parses llms.txt for that user agent (Amazonbot's IPs aren't verified in this log), it isn't reading Markdown, and those requests make up 31 of the 36 non-200 responses in the log.

Does ChatGPT-User traffic mean people are reading your site?

Not on its own. OpenAI says ChatGPT-User may visit a page when someone asks ChatGPT a question and isn't used for automatic crawling, so it's tempting to treat every hit as a person. On our site 219 of 311 ChatGPT-User requests (70.4%) were for the homepage, at a pace of 17 to 29 a day on every full day, at all hours, often with two requests in the same second.

People don't ask about a small software company 22 times a day at 03:00 and 15:00 alike. Something scripted on the user side is much more likely: a scheduled task, a monitoring product or an AI visibility tracker running a fixed list of prompts. AI Ranked's own prompts run in a tracker, and that may be part of it. The IP check passes on all of these requests because they really do come from OpenAI, and that's the point. Verification tells you who sent the request. It doesn't tell you why.

The other 92 requests look more like real curiosity, spread over 42 URLs, with short runs of several pages inside a minute. The tool page, our About page and the crawler study led that list. Even there we can't separate a person from a script.

How fast do AI bots find a new page?

Within 90 minutes in all four cases we could time. We published four pages during the window, and the first AI bot arrived 22 to 86 minutes after each deploy, with a median of 58 minutes. No page waited longer than an hour and a half.

Each deploy also pinged IndexNow and updated the sitemap, so these are times for a site that announces its changes. Timing is measured from the deploy record to the first logged request, and the GPTBot guide had less than a day of data. PerplexityBot's times swing the most because it works in batches. OAI-SearchBot and ClaudeBot were the dependable early visitors, and GPTBot fetched none of the four.

How much AI bot traffic turns into visits?

Very little. In the 8 days from 26 October to 2 November, GA4 recorded 321 sessions on the site, 9 of them from AI engines, against 1,155 AI bot requests. That's about 128 bot requests for each session that came from an AI engine, and 3.6 bot requests for every session of any kind.

Published crawl-to-referral ratios run higher, because Cloudflare's July 2025 numbers were about 1,091 to 1 for OpenAI and 195 to 1 for Perplexity. Those count crawls against referrals per platform across a whole network, and our figure pools every bot on one site with 9 sessions behind it, so read it as an order of magnitude. Nine sessions is too few to split by engine.

None of this traffic reaches your analytics. GA4 excludes known bots automatically and doesn't show you how many it removed, and most crawlers don't run its script anyway. A site can have hundreds of AI requests a week and see nothing of them in GA4. You have to read them at the server or the edge.

What should you do with your AI bot traffic data?

Start logging it, then report it by what each agent does. Raw hit counts mislead in both directions, as ClaudeBot and ChatGPT-User show. A short routine covers most of what we learned.

• Log at the edge or server. Filter your access logs by user agent, or forward matching requests from your CDN. GA4 can't do this job.
• Check the IP before you count. OpenAI, Anthropic and Perplexity publish IP lists. A user agent string alone can be faked by anyone.
• Separate control files from pages. Report robots.txt and sitemap fetches on their own line, or ClaudeBot will look like your most engaged reader.
• Group agents by type. Search, user-triggered and training agents call for different decisions, and your robots.txt may already treat them differently without anyone having chosen that.
• Time your new pages. If a page gets no search-agent visit within a day or two on a site that usually sees one in hours, check whether a firewall rule is turning bots away. A page your robots.txt allows can still return 403 to an AI agent.

We built AI Ranked Monitor for this job, and the numbers on this page came straight from its log. You can see the same view for our own site on the live demo. Any log pipeline with an IP check gets you the same data if you'd rather build it yourself.

How we collected the data

The sample is every request to airanked.ai from 16 tracked AI user agents between 10:55 UTC on 25 October 2026, when logging began, and 23:59 UTC on 3 November 2026. The edge worker that serves the site forwards each matching request to AI Ranked Monitor with its path, status code, user agent and connecting IP. Monitor checks the IP against the vendor's published ranges and stores the result. We don't publish IPs.

• Agents tracked: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Applebot, meta-externalagent, meta-externalfetcher, DuckAssistBot, MistralAI-User, Amazonbot, CCBot and Bytespider.
• IP verification: 1,207 of 1,453 rows (83.1%) passed a check against the vendor's published ranges. The other 246 couldn't be checked. Amazonbot, Bytespider, Applebot, meta-externalagent and MistralAI-User have no list loaded in Monitor, and 5 Anthropic rows came before our ranges loaded.
• Spoofed requests: requests claiming to be an OpenAI, Anthropic or Perplexity agent from an IP outside the vendor's list are discarded before storage, so we can't report a spoofing rate, and a real bot on an unlisted IP would be missed.
• Agent types: Monitor's labels, based on each vendor's description, and listed in the CSV. Applebot is counted as search and Amazonbot as training, and you may sort them differently.
• Googlebot: daily totals read from Search Console Crawl stats for airanked.ai. Sessions: GA4 Traffic acquisition for the same 8 dates, with AI engines as a custom channel. Those two tools use their own time zones, so daily comparisons can be off by a few hours.
• New pages: timed from the Cloudflare deployment record to the first logged request for that URL. The monthly report went live under an earlier address that now redirects, and we counted requests for both.

What this data can't tell you

This is one site over 10 days. It's small and new, it writes about AI search, and its robots.txt welcomes every AI agent, so a large publisher or a site that blocks bots will look different. Treat the shares as a case study. The request counts are exact for this log, but they won't generalize. The log also leaves out /assets/ requests and agents outside the 16 we track, and it can't show whether a fetched page was later cited in an answer.

Frequently asked questions

Dataset: ai-bot-traffic-data.csv (1,453 rows, one per request, 25 October to 3 November 2026 UTC). The license is CC BY 4.0.