- Only AI search agents and user-triggered fetchers count toward the robots.txt and firewall parts; blocking training crawlers costs no points.
- The firewall part compares five probe agents with a normal browser request to your homepage.
- Parts we couldn't measure are left out and the rest is scaled to 100, which the dashboard labels a partial score.
- Caps come from homepage findings: critical caps at 60, a homepage that needs JavaScript at 40, a warning at 95.
- The dashboard scores every monitored key page, 100 at most; the free scan scores the homepage only.
What are the six parts of the score?
The score adds six parts that together total 100 points. Each part is the share of a check that passed, multiplied by its weight and rounded, so every point on the report traces back to a specific agent or page.
| Part | Points | What earns the points |
|---|---|---|
| robots.txt access | 35 | AI search agents and user fetchers allowed at your homepage path |
| Firewall and CDN | 25 | Probe agents that load your homepage like a normal browser does |
| Key pages | 20 | Key pages that return 2xx and aren't set to noindex |
| Content without JavaScript | 10 | Pages whose copy is in the raw HTML |
| Structured data | 5 | Pages with at least one JSON-LD type |
| llms.txt | 5 | An llms.txt file that loads as text |
The weights follow how badly each problem hurts. A blocked search crawler takes you out of that engine's answers entirely, so robots.txt and the firewall carry 60 points between them. Markup helps engines understand a page they can already read, so it carries 10.
How is robots.txt access scored?
Monitor checks your robots.txt rules for the homepage path for 10 agents, the five AI search agents and five user-triggered fetchers, and awards 35 points times the share that are allowed. Allow all 10 and you get 35; block two and you get 28.
The search agents are OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and DuckAssistBot. The fetchers are ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher and MistralAI-User. The eight training agents, including GPTBot, ClaudeBot and Google-Extended, are shown in the dashboard but never cost points.
Rules are read the way RFC 9309 describes, with two details that often surprise people:
- A robots.txt that answers with a server error (5xx) or doesn't respond means "disallow everything" under the standard, so every agent counts as blocked until the file loads again.
- Apple documents that Applebot follows your Googlebot rules when your file has no Applebot group, and Monitor scores it the same way.
If robots.txt comes back as an HTML page or a bot challenge, we can't tell which agents it allows. That part scores 0 with the note "We could not read your robots.txt", and the agents show as Unknown. Monitor reads files up to 512 KB, the minimum parse size the standard asks crawlers to support.
How is the firewall part scored?
The firewall part is 25 points times the share of five probe agents that get your homepage the way a normal browser does. Monitor requests the homepage once as a browser and once as each of seven agents, then compares the answers.
The five that count are OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot and Perplexity-User. GPTBot and ClaudeBot are probed too, but as training crawlers they're reported without costing points.
A probe counts as blocked when the browser got the page and the agent got something else: a 4xx or 5xx status, no response, a challenge page (for example Cloudflare's "Just a moment" interstitial), or a 200 response that's a much smaller page with a different title, which is how many block pages look.
A probe is a request from our servers that names itself as the agent, so it isn't the real crawler, and some CDNs let the real one through by checking its IP address. If you connect logs and we've recorded IP-verified visits from that agent in the last 72 hours, the blocked probe is overridden and the points come back. That applies to the OpenAI and Perplexity agents, the two vendors whose IP lists Monitor checks.
How are key pages scored?
The pages part is 20 points times the share of checked key pages that load with a 2xx status and aren't set to noindex. Noindex is read from both the robots meta tag and the X-Robots-Tag response header.
A key page that returns 404, a 5xx error, redirects into a loop or carries noindex counts against this part. In the free scan only the homepage is judged, so this part is either 20 or 0.
How is content without JavaScript scored?
This part is 10 points times the share of loaded pages whose content is in the raw HTML. A page is flagged as JavaScript-only when its HTML holds fewer than 120 words and it either has an empty app container (such as <div id="root"></div> or __next) or loads four or more scripts.
It matters because most AI crawlers fetch HTML without rendering it. Vercel's crawler analysis found none of the major AI crawlers it studied render JavaScript, so text that only appears after scripts run is invisible to them.
How are JSON-LD and llms.txt scored?
Structured data is 5 points times the share of loaded pages with at least one JSON-LD type, and llms.txt is 5 points if /llms.txt returns a text file with real content. An HTML page served at that address, which many sites return as a soft 404, doesn't count.
These are the smallest parts on purpose. Schema markup and an llms.txt file can help engines understand your site, but neither makes a blocked page readable.
When is the score capped at 60, 40 or 95?
The score is capped whenever an open finding on your homepage is serious enough that a high number would mislead. The final score is the lower of the raw sum and the cap.
- 60 when any critical finding is open: an AI search crawler blocked in robots.txt, a firewall that rejects one of the counted agents, a homepage set to noindex, a homepage that returns an error or doesn't respond, a robots.txt that fails with a server error, or bot protection that blocked our server.
- 40 when the homepage needs JavaScript to show its content. This cap wins over the other two.
- 95 when any warning is open, such as user fetchers disallowed in robots.txt, a robots.txt we couldn't read, no JSON-LD on the homepage or a JSON-LD block that doesn't parse.
Low-severity findings never cap the score. A missing llms.txt, a missing meta description, no robots.txt at all (crawlers treat that as allow-all) or blocked training crawlers stay informational. When a cap applies, the report shows both numbers and the reason, for example "a critical issue is open".
What is a partial score?
A partial score means we couldn't measure every part, so the missing parts were left out and the rest scaled to 100. The usual cause is bot protection that blocked our normal browser request to your homepage, which leaves no clean baseline to compare the probe agents against.
In that case the firewall part isn't scored, the dashboard says "Partial score: your bot protection blocked our server", and the critical finding "Your bot protection blocked our server" caps the score at 60. Verified crawlers may still be getting through; connecting logs shows whether they are.
Two things help. Look at your bot protection settings for rules that challenge unknown automated traffic, since that is what our servers are to your CDN; the Cloudflare guide lists the usual suspects. And connect logs: real crawler visits are recorded on your side, so you see whether OpenAI, Anthropic and Perplexity get through whatever our server gets.
What does a worked example look like?
Take a site with 50 key pages. Its robots.txt disallows ChatGPT-User and Perplexity-User but allows the other eight counted agents, all five firewall probes pass, 49 pages load (one returns 404), 48 of those are indexable, 47 have their copy in raw HTML, 30 carry JSON-LD and there's no llms.txt.
| Part | Result | Points |
|---|---|---|
| robots.txt | 8 of 10 agents allowed | 28 of 35 |
| Firewall | 5 of 5 probes pass | 25 of 25 |
| Key pages | 48 of 50 healthy | 19 of 20 |
| Content | 47 of 49 loaded pages readable | 10 of 10 |
| JSON-LD | 30 of 49 loaded pages | 3 of 5 |
| llms.txt | Not found | 0 of 5 |
| Total | Warning open (user fetchers disallowed), cap 95 | 85 |
The warning cap doesn't bite here, because 85 is already under 95. Now suppose the same robots.txt also blocked OAI-SearchBot. The robots part would drop to 25, the raw total to 82, and the critical finding would cap the score at 60.
How do I raise my score?
Fix the findings in severity order, because caps decide more than points. Clearing a critical finding can lift a site from 60 to its raw score in one step, while adding an llms.txt is worth 5 points at most.
- Unblock AI search crawlers. Remove or narrow the robots.txt rules that block search agents. Our AI crawler list has rules that block training while keeping search open, and the robots.txt checker tests a draft before you deploy it.
- Let the agents through your firewall. Check your CDN's bot settings for challenges or blocks on AI agents. The Cloudflare guide walks through each setting.
- Fix broken and noindexed pages. The key pages table lists the status, noindex flag and word count for each page.
- Put copy in the HTML. Server-render or pre-render pages that show fewer than 120 words to crawlers.
- Add markup last. JSON-LD on key pages and an llms.txt file are worth 10 points together.
Frequently asked questions
What is a good AI Access Score?
There's no industry benchmark, because the score is Monitor's own measure. Read it through the caps: anything at 60 or below means a critical finding is open, such as a blocked AI search crawler. A score in the high 80s or 90s with no open warnings means AI crawlers can get to your important pages and read them today.
Does blocking GPTBot or ClaudeBot lower my score?
No. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot don't count toward the robots.txt or firewall points, because blocking them doesn't stop AI search answers from citing you. They still appear in the agent list so you can confirm the blocks you meant to set.
Why did my score drop when nothing changed on my site?
Something reachable from outside usually did change: a CDN rule, a robots.txt deploy, a redirect or a page that now returns an error. Open the alerts feed and the score breakdown, which shows the points lost in each part. If our server was blocked on that run, the score also turns partial.
Why is my free scan score different from my dashboard score?
The free scan only looks at your homepage, so the pages, content and JSON-LD parts are judged on that one page. The dashboard scores the same parts across up to 100 key pages, so a broken pricing page or a JavaScript-only product page moves the dashboard number but not the free scan.
Does a higher score mean AI engines will cite me more?
Not by itself. The score measures access: whether crawlers can fetch and read your pages. Being cited also depends on your content, your authority and what each engine retrieves for a question. A low score is a reason pages can't be cited; a high score only removes that reason.
How often is the score updated?
The score is recalculated after every daily site check and after every Check now, using the latest result for each key page. Pages are rechecked about once every 24 hours on their own schedule, so a page fix shows up in the pages, content and JSON-LD parts at the next site check, or straight away if you click Check now.