WordPress and AI crawlers: the settings that matter and how to log visits

Where does a WordPress site's robots.txt come from?

From one of two places. If a real robots.txt file sits in your web root, the server sends that. If not, and WordPress is installed at the domain root, WordPress generates a virtual robots.txt on request, and the Reading Settings docs confirm the virtual file only works when no physical one exists.

The virtual file is easy to change without anyone noticing, because SEO and security plugins edit it through the robots_txt filter. A plugin update can add a "block AI bots" list that also catches the agents deciding citations, such as OAI-SearchBot or PerplexityBot.

Always judge the live file at yoursite.com/robots.txt, not a settings screen. Our robots.txt checker shows which AI agents it allows, and the AI crawler list explains which ones you can block without losing AI search.

What does the Discourage search engines checkbox do to AI crawlers?

It asks every crawler not to index any page. Since WordPress 5.3, Settings > Reading > Discourage search engines from indexing this site outputs a robots meta tag with noindex, nofollow on every page, according to the same Reading Settings documentation. Older versions served a robots.txt Disallow: / instead.

That box is meant for staging sites, and it's a classic launch mistake: a site goes live with it still ticked, or a staging database gets copied over production. Crawlers can still fetch the pages, but compliant search and AI search crawlers won't index them, so they can't be cited. Monitor checks every key page for a noindex in the meta tag or the X-Robots-Tag header, and a new one is a critical alert.

What else can block AI crawlers on WordPress?

Anything between the crawler and PHP. Check these when an AI agent can load a page in a test but your logs show it failing, or the other way round:

• Security plugins with bot blocking, rate limiting or country rules that match crawler traffic.
• Your host's firewall, which many managed WordPress hosts run in front of every site.
• A CDN or proxy such as Cloudflare. If your domain runs through Cloudflare, its bot settings apply before WordPress sees anything; our Cloudflare guide walks through them.
• Content behind JavaScript. Page builders and plugins that load text in the browser leave crawlers that don't run scripts with little to read. Monitor counts the words in each key page's raw HTML.

How do I install the Monitor plugin on WordPress?

The connector is a single PHP file you upload as a must-use plugin. Must-use plugins live in wp-content/mu-plugins, are enabled automatically, load before normal plugins and can't be disabled from wp-admin, so a teammate can't switch it off by accident.

1. Copy your snippet. Take the WordPress version from the connector section of your site in Monitor. It already contains your ingest key; the code below uses YOUR_INGEST_KEY as a placeholder.
2. Create the file. Save the code as airanked-monitor.php. Start the file with <?php exactly as shown, with nothing before it.
3. Upload it. Using SFTP, your host's file manager or your deployment process, put it in wp-content/mu-plugins/. Create the folder if it doesn't exist. The file has to sit directly in that folder, since WordPress doesn't load must-use plugins from subfolders automatically.
4. Confirm. In wp-admin, Plugins > Must-Use lists "AI Ranked Monitor connector". It's active as soon as it appears.
5. Test it. Run curl -A "Claude-User/1.0" https://example.com/ with your domain, then reload your site in Monitor and look under Latest visits. If you run a page cache, test a page it hasn't stored yet, for the reason explained below.

<?php
/**
 * Plugin Name: AI Ranked Monitor connector
 * Description: Reports AI agent visits (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot...) to AI Ranked Monitor.
 * Save as wp-content/mu-plugins/airanked-monitor.php
 */
add_action('shutdown', function () {
  $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
  if (!preg_match('/OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-externalfetcher|DuckAssistBot|MistralAI-User|Amazonbot|CCBot|Bytespider/i', $ua)) return;
  $ip = $_SERVER['HTTP_CF_CONNECTING_IP'] ?? ($_SERVER['REMOTE_ADDR'] ?? '');
  wp_remote_post('https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY', [
    'blocking' => false,
    'timeout'  => 1,
    'headers'  => ['content-type' => 'application/json'],
    'body'     => wp_json_encode([
      'p'  => parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH),
      'ua' => $ua, 'ip' => $ip, 's' => http_response_code(), 't' => (int) (microtime(true) * 1000),
    ]),
  ]);
});

How does the plugin work?

It hooks into shutdown, which fires just before PHP stops, so the page has already been generated. If the user agent matches one of the 16 AI agents Monitor tracks, it sends the path, user agent, client IP, response status and time; every other request returns without doing anything.

The report is a wp_remote_post call with blocking set to false, which per the WP_Http reference sends the request and returns to your code immediately without waiting for a response. The status comes from PHP's http_response_code(), which returns the current response code, so 404s and errors served to crawlers are counted too.

The plugin creates no database tables, options, cron events or admin screens. Removing it means deleting the file.

Does page caching hide AI visits?

It can, depending on where the cache sits. A full-page cache that answers before WordPress runs never executes the plugin. The WP Super Cache plugin page, for example, says its expert mode serves cached files in a way that completely bypasses PHP, and caches that answer at the web server or proxy level behave the same way.

Caching plugins that serve pages through PHP still load early: WordPress includes the advanced-cache.php drop-in before it loads must-use plugins, as you can see in wp-settings.php. A page served from there has usually ended the request before the plugin exists.

In practice you'll see AI visits to pages that missed the cache, which still shows which agents come and roughly where, but undercounts popular cached pages. If you need complete numbers and your domain runs through Cloudflare, use the Cloudflare Worker connector, which sees every request before any cache.

Which IP address does the plugin report?

It sends the CF-Connecting-IP header when one is present, which Cloudflare sets to the visitor's address, and the server's remote address otherwise. That's the right IP on a site served directly or through Cloudflare.

Behind another proxy or load balancer, the remote address is the proxy's own, so OpenAI and Perplexity visits fail Monitor's IP verification and are dropped while other vendors are still counted. If your AI visit counts show Anthropic and others but never OpenAI, that's the usual cause. Report from the proxy instead, or make sure your host passes the real client IP through. How visits are verified has the details.

How does Monitor watch a WordPress site every day?

WordPress changes through plugin updates, theme changes and settings screens as much as through deploys, which is why the checks are daily.

• robots.txt is read live every day, so a plugin that rewrites the virtual file triggers an alert naming the agents that lost access.
• Every key page is checked for status, noindex, redirects, canonical changes, words in the raw HTML and JSON-LD types. A theme update that drops your schema or a staging setting that adds noindex shows up the next day.
• The homepage is requested as seven AI agents and compared with a browser, which catches a security plugin or host firewall that starts blocking them.
• With the plugin installed, you see which pages ChatGPT, Claude and Perplexity fetch for real people.

Start with the 14-day trial, no card needed, and use the setup guide to pick your key pages.

Frequently asked questions