- Firewall blocks are confirmed with a second request before an alert goes out, and timeouts or 5xx errors on key pages are retried once about 20 minutes later.
- Four or more pages failing the same way in one run produce one summary alert, not dozens.
- Check now reruns the checks on demand, once every 10 minutes per site, which suits a post-deploy check.
- Log connectors exist for Cloudflare Workers, Next.js, WordPress and Express, and anything else can post to the ingest endpoint.
- robots.txt is parsed following RFC 9309, including the rule that a 5xx response means disallow everything.
Who is this for?
It's for the developers, platform engineers and webmasters who run a company's marketing site: the people who own the CDN configuration, the framework, the build and the deploy pipeline. Marketing may own the goal of being cited by AI assistants, but every way of breaking it runs through your stack.
Monitor treats AI crawler access as something to test continuously, like uptime or performance budgets, rather than a one-off SEO audit.
What does Monitor check, technically?
Three layers, each run from outside your infrastructure the way a crawler would see it.
| Layer | What's checked | How often |
|---|---|---|
| robots.txt | Rules for 18 AI agents, parsed per RFC 9309, evaluated for the homepage and every key page path | Daily |
| Edge and firewall | Homepage requested as a Chrome browser and as 7 AI agents; status, challenge pages and block pages compared | Daily |
| Key pages | Status after up to 5 redirect hops, final URL, robots meta and X-Robots-Tag noindex, canonical, title, words in raw HTML, JSON-LD types | About every 24 hours per page |
| llms.txt | Loads as text (an HTML soft 404 doesn't count), content changes | Daily |
The robots.txt behavior follows RFC 9309: a file that fails with a 5xx or can't be reached means "disallow everything", a 4xx means no restrictions, and files are read up to 512 KB. Applebot falls back to Googlebot rules when it has no group of its own, as Apple documents.
How are alerts built for debugging?
Each alert says what changed, where and compared with what, so it can go straight into a ticket.
- "Your firewall started blocking OAI-SearchBot, PerplexityBot" with the response each got, such as a challenge page or a 403, seen twice in a row.
- "/pricing now needs JavaScript to show content" with the new raw-HTML word count.
- "/docs/api now redirects" with the old and new landing URLs.
- "robots.txt now blocks Claude-SearchBot" with the change compared against the previous copy of the file.
Noise control is built in. A probe that flips to blocked is retried once before alerting; a key page that times out, returns a 5xx or a 429 is retried about 20 minutes later; four or more pages failing the same way in one run collapse into one summary listing the paths. The alerts reference lists every alert with its trigger.
Which regressions does it catch?
Three examples of regressions Monitor is designed to catch. They illustrate the mechanics, not specific customers.
A WAF rule with a wide match
A new rate limiting rule meant for a scraping incident matches on data center ASNs, which is where crawler traffic comes from. The next site check sees ChatGPT-User and Claude-User get 429s on the homepage while the browser gets a 200, confirms it with a second request and raises a critical alert. Rolling the rule back clears it on the next check with a Fixed notice.
A staging header in production
A header rule that adds X-Robots-Tag: noindex to preview builds is promoted with the rest of the config. Every key page now carries noindex. As the day's page checks run, Monitor groups pages that fail the same way into summary alerts that list the affected paths, and holds further messages for 30 minutes after each one, so you get a few clear messages instead of 60.
A migration that broke canonicals
A framework migration changes how canonical URLs are generated, and product pages now point at the homepage. Monitor compares each page's canonical with its last check and raises a warning per page, or a summary if many changed at once.
How does it fit into a deploy workflow?
Route alerts to the channel your team already watches for production issues. Each site takes a Slack incoming webhook, and critical alerts and warnings land there a few minutes after the check runs. Email recipients can be saved too, but alert emails only start once email delivery is live.
After a risky release, click Check now on the site. It reruns the site checks and the first batch of key pages immediately, and the rest follow on their schedule. It's limited to once every 10 minutes per site. Everything else happens without anyone remembering to run it, which is the point: the regressions that cost AI visibility are the ones nobody thought to test.
Monitor's page requests identify themselves with a user agent containing AIRankedMonitor, so they're easy to find in your logs or exclude from analytics.
Can I send our own logs instead of using a connector?
Yes. The connectors are thin wrappers around one endpoint, POST /api/hit/<ingest key>, which accepts one event or up to 100 per request with the path, user agent, client IP, status and timestamp. Anything that can post JSON can feed it: a log drain, an edge function on another CDN or a batch job over access logs.
OpenAI and Perplexity visits are checked against the IP ranges those vendors publish, so send the real client IP or those visits are dropped. The ingest API reference has the field list and response codes, and the Cloudflare, Next.js and WordPress guides cover the ready-made connectors.
How do I set it up?
- Scan the domain with the free scan to see what's broken today.
- Start the trial, add the site and review the key page list. Add the routes that matter commercially and remove ones that don't.
- Point alerts at your team's Slack channel.
- Install a connector at the edge if you want visit data and fewer probe false positives; recent verified visits beat a blocked probe result.
- Fix the baseline findings, then let the daily checks run. The getting started guide covers each step.
Frequently asked questions
How does Monitor detect a firewall block?
It requests your homepage once with a normal Chrome user agent and once as each of seven AI agents, from the same servers, and compares the results. An agent counts as blocked when the browser gets the page and the agent gets a 4xx or 5xx status, no response, a challenge page, or a much smaller page with a different title.
Will Monitor's requests trigger our rate limits?
It's light. A site check is a handful of requests: robots.txt, llms.txt, one browser request and seven probe requests to the homepage. Key pages are spread across the day in small batches, each page about once every 24 hours. Requests to key pages identify themselves with a user agent containing AIRankedMonitor.
Can I trigger a check from our CI pipeline?
Not through an API today. Check now in the dashboard reruns the site checks and the first batch of key pages on demand, once every 10 minutes per site. Many teams click it after a release; everything else is picked up by the daily checks without anyone doing anything.
How does Monitor handle robots.txt edge cases?
It parses robots.txt following RFC 9309, reads up to 512 KB, treats a 5xx or unreachable file as disallow everything and a 4xx as allow everything. If robots.txt returns an HTML or challenge page, it retries on the www or bare host and with a browser user agent, then marks the agents Unknown rather than guessing.
What does Monitor need from our side?
Nothing for daily monitoring, which works from outside the way a crawler does. The optional log connector, available for Cloudflare Workers, Next.js, WordPress and Express, only reports AI agent requests, and for each one sends five fields: path, user agent, IP, status and time.