Connect your logs to see real AI crawler visits

Why connect logs at all?

Connecting logs shows what AI agents actually do on your site, which outside checks can't. Daily monitoring tells you whether crawlers can reach a page; logs show whether they do, how often, and which pages assistants fetch while answering someone's question.

Analytics tools can't do this, because AI crawlers fetch HTML without running the JavaScript that analytics tags need. The request only exists in your server or CDN logs, so the connector reads it there.

With logs connected you get three things in the dashboard: AI agent visits for the last 30 days, the pages that ChatGPT, Claude and Perplexity fetched on behalf of people, and a silence alert when an agent that visited every day stops. Verified visits also correct firewall probe results.

Which connector should I use?

Use the one closest to the edge of your stack, because it sees every request, including cached pages that never reach your app server.

Where do I find my ingest key?

Each site has its own ingest key, a random string that ties reported visits to that site. Open the site in the dashboard and go to Connect logs: every snippet there already contains your key and endpoint, so you can copy it as is.

The snippets on this page are the same code with the key replaced by YOUR_INGEST_KEY. If you copy from here, swap in your key. Treat it like a webhook URL: anyone holding it can send visits to your site's stats, so keep it out of public repositories.

How do I install the Cloudflare Worker?

Create a Worker in your Cloudflare account, paste this code, deploy it and add a route such as example.com/* so it runs in front of your site. It passes every request through untouched. The Cloudflare guide has each dashboard step.

// Cloudflare Worker: add a route for ${host}/* to this Worker.
// It passes every request through untouched and reports AI agent visits in the background.
const ENDPOINT = "https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY";
const AI = /OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-externalfetcher|DuckAssistBot|MistralAI-User|Amazonbot|CCBot|Bytespider/i;

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const ua = request.headers.get("user-agent") || "";
    if (AI.test(ua)) {
      const u = new URL(request.url);
      ctx.waitUntil(fetch(ENDPOINT, {
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({ p: u.pathname, ua, ip: request.headers.get("cf-connecting-ip"), s: response.status, t: Date.now() }),
      }).catch(() => {}));
    }
    return response;
  },
};

How do I install the Next.js middleware?

Save this as middleware.ts in your project root (or in src/ if you use one) and deploy. If you already have middleware, merge the AI check into your existing function. On Next.js 16, where middleware was renamed to proxy, the Next.js guide explains what to change.

// middleware.ts (Next.js on Vercel or any host). Merge into your existing middleware if you have one.
import { NextResponse, type NextRequest, type NextFetchEvent } from "next/server";

const ENDPOINT = "https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY";
const AI = /OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-externalfetcher|DuckAssistBot|MistralAI-User|Amazonbot|CCBot|Bytespider/i;

export function middleware(req: NextRequest, event: NextFetchEvent) {
  const ua = req.headers.get("user-agent") || "";
  if (AI.test(ua)) {
    const ip = (req.headers.get("x-forwarded-for") || "").split(",")[0].trim();
    event.waitUntil(fetch(ENDPOINT, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ p: req.nextUrl.pathname, ua, ip, t: Date.now() }),
    }).catch(() => {}));
  }
  return NextResponse.next();
}

export const config = { matcher: "/((?!_next/static|_next/image|favicon.ico).*)" };

How do I install the WordPress plugin?

Save this as wp-content/mu-plugins/airanked-monitor.php (create the mu-plugins folder if it doesn't exist). Must-use plugins run without activation, so it's live as soon as the file is uploaded. Caching caveats are in the WordPress guide.

<?php
/**
 * Plugin Name: AI Ranked Monitor connector
 * Description: Reports AI agent visits (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot...) to AI Ranked Monitor.
 * Save as wp-content/mu-plugins/airanked-monitor.php
 */
add_action('shutdown', function () {
  $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
  if (!preg_match('/OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-externalfetcher|DuckAssistBot|MistralAI-User|Amazonbot|CCBot|Bytespider/i', $ua)) return;
  $ip = $_SERVER['HTTP_CF_CONNECTING_IP'] ?? ($_SERVER['REMOTE_ADDR'] ?? '');
  wp_remote_post('https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY', [
    'blocking' => false,
    'timeout'  => 1,
    'headers'  => ['content-type' => 'application/json'],
    'body'     => wp_json_encode([
      'p'  => parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH),
      'ua' => $ua, 'ip' => $ip, 's' => http_response_code(), 't' => (int) (microtime(true) * 1000),
    ]),
  ]);
});

How do I install the Node and Express middleware?

Add the function to your server and register it with app.use(aiRankedMonitor) before your routes. It needs Node 18 or later for the built-in fetch.

// Express / Node middleware. Add before your routes: app.use(aiRankedMonitor);
const ENDPOINT = "https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY";
const AI = /OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-externalfetcher|DuckAssistBot|MistralAI-User|Amazonbot|CCBot|Bytespider/i;

function aiRankedMonitor(req, res, next) {
  const ua = req.get("user-agent") || "";
  if (AI.test(ua)) {
    res.on("finish", () => {
      fetch(ENDPOINT, {
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({ p: req.path, ua, ip: req.ip, s: res.statusCode, t: Date.now() }),
      }).catch(() => {});
    });
  }
  next();
}

How do I test the connection?

Send one fake visit with curl and check that it appears under Latest visits in the dashboard. Replace the key with yours first.

curl -X POST https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY \
  -H "content-type: application/json" \
  -d '{"p":"/pricing","ua":"Mozilla/5.0 (compatible; Claude-User/1.0)","ip":"203.0.113.9","s":200}'

A working endpoint answers 204 No Content. Other answers tell you what's wrong:

The test uses Claude-User on purpose. A test sent as GPTBot or PerplexityBot from your laptop would be dropped by the IP check described next.

How are OpenAI and Perplexity visits verified?

Every visit that claims to be an OpenAI or Perplexity agent is checked against the IP ranges that company publishes, and visits from any other IP are discarded as spoofed. Verified visits get a "verified" label in the dashboard.

Monitor refreshes these lists every day from the vendors' own files: searchbot.json, chatgpt-user.json and gptbot.json from OpenAI, as documented on its crawler page, and the PerplexityBot and Perplexity-User lists linked from Perplexity's crawler docs. IPv4 and IPv6 are both supported.

Visits from other vendors are counted without a verified label. That includes Anthropic's agents, even though Anthropic now publishes a crawler IP list: Monitor doesn't check against it yet. Treat unverified counts as an upper bound, because anyone can send a request with a borrowed user agent.

Verification depends on the connector reporting the real client IP. The Cloudflare connector reads CF-Connecting-IP, the Next.js one the first address in x-forwarded-for, the WordPress one CF-Connecting-IP when present and the server's remote address otherwise, and the Node one req.ip.

Which agents does Monitor track?

Monitor knows 18 AI agents from 11 companies. Sixteen of them send real requests, so the connectors match those 16 and ignore everything else. Google-Extended and Applebot-Extended are robots.txt tokens only; they never visit, so they appear in robots.txt checks but never in logs.

"Firewall probe" means Monitor also requests your homepage as that agent every day to catch CDN blocks. The AI crawler list explains what each agent does.

Can I send visits from my own log pipeline?

Yes. The connectors are thin wrappers around one endpoint, so you can post from a log shipper, an edge function on another CDN or a batch job instead.

Send a POST to https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY with a JSON body. The body is either one event or {"events": [...]} with up to 100 events per request. Each event has these fields:

What data is sent and how long is it kept?

Only AI agent requests are sent, and only five fields per request: path, user agent, IP, status and time. Requests from people are filtered out inside the snippet, before anything leaves your server.

Monitor keeps individual visits for 14 days (the Latest visits table) and daily totals per agent and path for 120 days (the charts). Removing a site from your account deletes its visit data along with its pages and alerts. Our privacy policy covers the rest.

Frequently asked questions