What is GPTBot? OpenAI's training crawler, its IP ranges and robots.txt rules

What is GPTBot?

GPTBot is the crawler OpenAI uses to gather public web content for training its models. OpenAI's crawler documentation says it is "used to crawl content that may be used in training our generative AI foundation models", and that disallowing it "indicates a site's content should not be used in training". It visits on its own schedule, with no ChatGPT user waiting on the result, which makes it a crawler in the classic sense.

OpenAI introduced it in August 2023. Search Engine Roundtable's write-up from August 7, 2023 quoted the user agent as GPTBot/1.0 and the original documentation's promise that crawled pages would be filtered to drop paywalled sources, sources known to gather personally identifiable information, and text that violates OpenAI's policies. That filtering language no longer appears on OpenAI's current crawler page, which describes GPTBot in two sentences and shows version 1.4 in its example string. Treat the 2023 description as history, not as a current commitment.

The search results for "gptbot" mix in a few unrelated things, including a commercial product that uses the same name. Everything on this page is about OpenAI's crawler, the one that shows up in your logs with GPTBot in its user agent.

How is GPTBot different from OAI-SearchBot and ChatGPT-User?

GPTBot feeds model training, OAI-SearchBot builds the index ChatGPT search draws on, and ChatGPT-User fetches a page when a person's question needs it. Each one follows a separate robots.txt group, so blocking GPTBot keeps your pages out of training while ChatGPT can still find, fetch and cite them in search answers.

The key sentence on OpenAI's page is short: "Each setting is independent of the others". OpenAI gives the exact example most publishers want, allowing OAI-SearchBot to appear in search results while disallowing GPTBot. Two more details on the same page are easy to miss. If a site allows both bots, OpenAI "may use the results from just one crawl for both use cases" to avoid crawling twice. And after a robots.txt change, OpenAI says it can take about 24 hours for its search systems to adjust.

ChatGPT-User is the odd one out. OpenAI says it doesn't crawl automatically and plays no part in deciding what appears in search, but since a person set off the visit, OpenAI's wording is that robots.txt rules "may not" be followed for it. To keep ChatGPT from reading a page at all, robots.txt isn't the right lever for that agent. A login is.

The same three-way split exists at Anthropic, whose training crawler we covered in our ClaudeBot guide. For every other AI user agent, from Google-Extended to PerplexityBot, see our full AI crawler list for robots.txt.

What is the GPTBot user agent string?

OpenAI publishes this example and notes that the version number may change:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Robots.txt only cares about the product token, GPTBot. The version suffix moved from 1.0 at launch to 1.4 in OpenAI's current example, and that's why rules and log filters should match the token, never the whole string. When GPTBot fetches robots.txt itself, OpenAI says it may add a robots.txt marker to the user agent so you can tell those requests apart in logs that don't record paths. OpenAI shows the marked format only for OAI-SearchBot (a robots.txt; segment before the URL), so match GPTBot loosely there too.

For comparison, the other OpenAI strings as published:

OAI-SearchBot (example, version may change):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

ChatGPT-User:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

OAI-AdsBot:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Notice that OAI-SearchBot's string starts like a Mac Chrome browser. A log filter that drops anything containing "Macintosh" as human traffic will hide it. Filter on the token instead.

How much does GPTBot crawl?

More than any other AI-only crawler on Cloudflare's network. In Cloudflare's crawl-to-click analysis, GPTBot's slice of AI-only crawling grew from 11.9% (July 2024) to 28.1% (July 2025), and it passed Amazonbot and Bytespider, both of which shrank sharply over those twelve months.

An earlier Cloudflare post, From Googlebot to GPTBot, measured it against all crawlers, search engines included. Between May 2024 and May 2025 GPTBot's share of that wider pool went from 2.2% to 7.7%, its requests rose 305%, and it climbed from ninth to third place. Cloudflare's 2025 Year in Review later named it the most active verified bot after Googlebot, at about 7.5% of verified bot traffic.

Most of that activity is training. Cloudflare says training drove nearly 80% of AI bot activity in the twelve months to July 2025, up from 72% a year before, and its Year in Review says the training traffic figure "is heavily influenced by OpenAI's GPTBot". So when OpenAI requests jump in your own logs, split them by token before you read the spike as ChatGPT users finding you.

Does GPTBot send any traffic back?

Not directly. GPTBot collects training data and has no link back to your site, so the visits OpenAI sends come from ChatGPT search and user-triggered fetches. Cloudflare measures this as a crawl-to-refer ratio, HTML pages crawled per referred visit, across all of a platform's bots. For OpenAI it was about 1,091 to 1 in July 2025, down 10% from 1,217 in January.

The industry rows come from Cloudflare's crawl purpose and industry breakdown. The Year in Review adds that OpenAI's ratio was "quite spiky" during 2025 and reached as much as 3,700 to 1 in March. OpenAI sits between Anthropic and Perplexity: far more referrals per crawl than Anthropic, far fewer than Perplexity.

Who blocks GPTBot? Our robots.txt test of 251 sites

Just under a quarter of large websites. In our test, 59 of 251 major sites (23.5%) disallow GPTBot at the homepage path, and 84 (33.5%) name it in robots.txt at all. That makes GPTBot the most-named OpenAI agent and the most-blocked one, well ahead of ChatGPT-User (41 sites) and OAI-SearchBot (28).

The figure matches what the original study found. In its 261-site set, 59 sites (22.6%) disallowed GPTBot. Across the 248 sites readable in both crawls, GPTBot blockers went from 55 to 57: Facebook and AliExpress added a block, and no site lifted one. GPTBot policy at this level moves a site or two at a time.

The more useful number is how the blockers combine their OpenAI rules. 62 sites block at least one OpenAI agent, and they split into two main camps:

28 sites shut OpenAI out completely. 31 block GPTBot and keep OAI-SearchBot open, which is the "no training, but you can still cite us" position OpenAI's documentation describes. The direction never reverses: no site in the sample blocks OAI-SearchBot while letting GPTBot in. Our original study found the same split at the same size, with 31 of its 59 GPTBot blockers still allowing OAI-SearchBot.

Six of the blockers never type the word GPTBot. Baidu, X, Reddit, Pinterest, IMDb and Reuters shut the default * group and open the door only for a handful of crawlers they name, so GPTBot is caught by the catch-all. If you're reading competitors' robots.txt files to see who blocks GPTBot, a text search for "GPTBot" misses this whole pattern.

Industry explains most of the variation. Social platforms and news publishers block GPTBot at more than twice the overall rate, and the software sites in the sample barely touch it:

News sites make up 28 of the 59 blockers, nearly half. The four B2B SaaS blockers are Canva, Figma, Loom and Gartner. Every other software company in the sample, 89 of 93, leaves GPTBot allowed at the homepage.

Robots.txt files also show the licensing deals. 15 sites in the sample block Anthropic's ClaudeBot but allow GPTBot, and our ClaudeBot guide traced 14 of them to publishers with announced OpenAI content partnerships, including the Financial Times, Condé Nast and News Corp. Some of those publishers go further and name GPTBot with an explicit allow: The Verge and Vox give it a group with no disallow lines at all.

Robots.txt counts are a floor, not a ceiling. Our markdown and AI usage signals study read the same domains' files and found user-agent rules blocking at least one of seven training bots on 94 of 250 sites (37.6%), led by news at 45 of 50, against 10 of 93 B2B SaaS sites. None of the 250 files carried Cloudflare's managed AI block, and only 21 used the newer Content-Signal line. The rest of the blocking happens at the firewall, where no robots.txt study can see it.

What are GPTBot's IP ranges?

OpenAI publishes GPTBot's addresses as JSON at openai.com/gptbot.json. The copy we fetched for this guide held 18 IPv4 prefixes and no IPv6: four /24 blocks, thirteen /25 blocks and one /28, 2,704 addresses in total. Every prefix sits in AS8075, Microsoft's network.

We looked up all four of OpenAI's lists the same day:

Three things in that table change how you should handle GPTBot traffic.

First, six of GPTBot's eighteen prefixes also appear, unchanged, in OAI-SearchBot's list. That fits OpenAI's note that one crawl can serve both purposes when a site allows both bots. So an address on the list proves a request is OpenAI's, while only the user agent token says which bot sent it. Check both.

Second, the lists get regenerated. Each file carries a creationTime field, and the four we fetched had different ones, months apart. Pull the JSON on a schedule, daily is plenty, instead of pasting addresses into a firewall rule once and forgetting it.

Third, every range belongs to Microsoft's cloud. A firewall rule that blocks Azure or AS8075 to stop scrapers also blocks GPTBot, OAI-SearchBot and ChatGPT-User, which is probably not what whoever wrote it intended.

How can you tell a real GPTBot request from a fake one?

Match the request's IP against openai.com/gptbot.json, and confirm the user agent contains the GPTBot token. A request that claims to be GPTBot from an address outside the list is fake. The user agent alone proves nothing, because any scraper can send it.

Reverse DNS won't help here. We ran a reverse lookup on an address in each of the 18 GPTBot prefixes, and none returned a hostname. That's the opposite of Googlebot, where Google's verification guide tells you to run a reverse DNS lookup and check for a Google domain. OpenAI's documentation offers only the IP lists, so the IP match is the whole test.

A short Python check:

import ipaddress, json, urllib.request

req = urllib.request.Request("https://openai.com/gptbot.json",
                             headers={"User-Agent": "Mozilla/5.0"})
nets = [ipaddress.ip_network(p["ipv4Prefix"])
        for p in json.load(urllib.request.urlopen(req))["prefixes"]]

def is_gptbot(ip, user_agent):
    addr = ipaddress.ip_address(ip)
    return "gptbot" in user_agent.lower() and any(addr in n for n in nets)

print(is_gptbot("20.171.207.5", "Mozilla/5.0 ...; compatible; GPTBot/1.4; +https://openai.com/gptbot"))
# True: 20.171.207.0/24 is on the list

Cache the parsed list and refresh it once a day. Requests that fail the check aren't OpenAI's problem or yours to honor; treat them like any other unidentified scraper. If you sit behind Cloudflare, GPTBot is already on its verified bots list (the Year in Review counts it in verified bot traffic), so Cloudflare can tell the real crawler from an impostor at the edge.

GPTBot robots.txt: how to block, allow or split training from search

Give GPTBot its own User-agent: GPTBot group in the robots.txt served at the root of every host, then write Disallow: / to keep it out or Allow: / to let it in. Three templates handle nearly every case we see.

Block training only, keep ChatGPT search and user fetches:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /

Block every OpenAI agent:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
Disallow: /

Allow GPTBot everywhere except a paid or licensed section, while keeping your private paths closed:

User-agent: GPTBot
Disallow: /reports/
Disallow: /admin/
Disallow: /cart/
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/

That last template guards against the error we run into most often. As soon as GPTBot has a group of its own, it ignores the * group completely, because RFC 9309 sends a crawler to the groups that name it and only falls back to * when none do. A GPTBot group that says only Allow: / opens every private path you closed for everyone else. Copy the private-path lines into every group you write.

Path-level rules are common among large sites. 22 of the 251 sites name GPTBot, leave the homepage open and disallow specific sections, including Netflix, GitHub, Coursera, G2 and the Wall Street Journal. That's a better fit than a blanket block for most businesses with a mix of marketing pages and gated content.

A few rules from RFC 9309 that change real outcomes:

• Matching is case-insensitive. gptbot, GPTBot and GPTBOT are the same token. GPT-Bot is not.
• A 404 on robots.txt means no rules. The standard says a crawler may access anything when the file is "unavailable" (a 4xx status). A 5xx is the reverse: crawlers must assume a complete disallow.
• Caching. Crawlers shouldn't use a cached copy for more than 24 hours, which lines up with OpenAI's "about 24 hours" for search changes. Don't expect an edit to take effect in minutes.
• Size. The standard only obliges crawlers to read the first 500 KiB, so on a huge file put the GPTBot group early.
• Every host needs its own file. docs.example.com and blog.example.com don't inherit the rules on example.com.

To see how your current file treats GPTBot next to 20 other AI crawlers, paste it into our free robots.txt checker for AI crawlers. It runs in your browser and applies the same matching rules.

Should you block GPTBot?

Block GPTBot when the words on your pages are the product, as with paywalled journalism, paid research or course material. Allow it when the pages exist to sell a product or service, because training is how OpenAI's models learn what your company does for the answers ChatGPT gives without searching. In both cases, leave OAI-SearchBot open unless you want out of ChatGPT search entirely.

The software row is where most of our readers sit, and the data backs it. Of the 93 software companies we checked, 89 leave GPTBot allowed, so allowing it is the category norm. OpenAI frames a block as a signal that content "should not be used in training", and nothing on its crawler page says a block removes what earlier crawls already gathered, so treat it as forward-looking.

The publisher row mirrors the 31 sites in our test that block GPTBot and keep OAI-SearchBot open. If your business depends on readers clicking through, blocking OAI-SearchBot gives up a citation source and buys no training protection at all, because training is GPTBot's job.

Two cases deserve a second look before you block. If you're negotiating with OpenAI, a block strengthens your hand, and our data shows several deal partners now allow GPTBot by name. And if you publish research or statistics that you want quoted, a training opt-out makes it less likely the model repeats your figures from memory, though ChatGPT can still cite the live page through search.

How to find GPTBot in your server logs and Cloudflare

Filter your access logs on the GPTBot token and look at the status codes your server sent back. 200s mean the crawler got the page. 403s, 429s or challenge pages served to an address on OpenAI's list mean a firewall is quietly overruling your robots.txt.

On a server with the common combined log format, where the client IP is the first field, this counts GPTBot requests by status code and lists the busiest IPs:

# GPTBot requests by HTTP status
grep -i "gptbot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# Top claimed-GPTBot IPs (check each against openai.com/gptbot.json)
grep -i "gptbot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

# Pages GPTBot fetched most
grep -i "gptbot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Run the IP list through the Python check above. Any address that fails is a scraper borrowing GPTBot's name, and those shouldn't count toward your read of what OpenAI is doing.

Firewalls and CDNs are where GPTBot most often gets blocked by accident, and the block is silent: GPTBot doesn't email you when it gets a 403. Cloudflare is the most common case. It added a one-click setting to block AI crawlers in September 2024, said more than one million customers had turned it on by July 2025, and since July 1, 2025 asks every new domain whether to allow AI crawlers at all. Its AI Crawl Control page says you can "set allow or block rules for individual crawlers", available on all plans, which is the setting to check if GPTBot is blocked and you didn't decide that.

The result is a gap between policy and reality. A site's robots.txt can say Allow: / for GPTBot while its edge returns a challenge to every request from Microsoft's IP space. Robots.txt studies, ours included, can't see that layer, so the only way to know is to send a request as GPTBot and look at what comes back, or to read your edge logs.

We built AI Ranked Monitor to do that daily. It fetches your homepage seven times, posing as GPTBot, OAI-SearchBot, ChatGPT-User and four other AI agents, sets each response beside an ordinary browser fetch, checks what your robots.txt says to 18 AI user agents, and alerts you in Slack or by email once something starts turning them away. If you add its optional log connector, it also counts real GPTBot visits, discarding any that arrive from outside OpenAI's published ranges.

GPTBot mistakes we found in real robots.txt files

1. Naming GPTBot and forgetting OAI-SearchBot. 29 of the 84 sites that name GPTBot never mention OAI-SearchBot, so ChatGPT search access falls to whatever the * group says. Fix: write an explicit OAI-SearchBot group, whichever way you decide.
2. A GPTBot group that overrides the * rules. Any named group replaces the default group for that bot, so a GPTBot group with only Allow: / or only a crawl delay opens your private paths to it. Fix: repeat path rules inside every named group.
3. Counting on Crawl-delay. GitHub and The Atlantic set a crawl delay in the group that covers GPTBot. OpenAI's crawler page doesn't mention Crawl-delay at all, so there's no documented promise it's honored. Fix: rate-limit at the server or CDN if load is the real problem.
4. Writing rules for agents OpenAI doesn't list. Four sites name OpenAI-sounding agents that aren't on OpenAI's crawler page. BuzzFeed and HuffPost list WebGPTBot, Le Figaro lists ChatGPT Agent, and USA Today lists seven, from OpenAI-SnippetBot to ChatGPT Operator. Harmless if GPTBot is also named, useless if it's the only line. Fix: use the four documented tokens.
5. Blocking Microsoft's cloud to stop scrapers. All 18 GPTBot prefixes are in AS8075. Fix: allowlist OpenAI's published ranges above any ASN-wide block, or use your CDN's verified bot setting.
6. Serving robots.txt only to browsers. If a bot-protection layer challenges the request for robots.txt itself, GPTBot gets an error instead of your rules. A 4xx tells it there are no rules; a 5xx tells it to stay out of everything. Fix: always serve robots.txt with a 200.

Frequently asked questions