Vercel and Next.js: keep AI crawlers in and log their visits

Does Vercel block AI crawlers?

Not unless someone tells it to. The Vercel Firewall has an AI Bots managed ruleset that can deny AI crawlers, but it's available on all plans and inactive by default, shown as Allow in the dashboard. Blocks on Vercel usually come from that ruleset being switched to Deny, or from a custom rule written for scrapers that also matches crawlers.

The rulesets live under your project's Firewall tab, in Rules, in the Bot Management section. Changes take effect after you review and publish them, so check that page first when an AI agent suddenly can't reach a Vercel site.

How do Vercel's bot settings treat AI agents?

Three settings decide what bots get on Vercel, and only one of them targets AI crawlers on purpose.

The exemptions depend on Vercel's verified bots directory, which recognizes bots by IP range, reverse DNS or signed requests. Its list covers the OpenAI, Anthropic and Perplexity agents, from OAI-SearchBot and Claude-SearchBot to GPTBot, ClaudeBot and Perplexity-User. So switching on Bot Protection or Attack Mode during an incident won't lock out the AI agents that matter, while setting AI Bots to Deny will.

One catch from the same page: Bot Protection doesn't work behind a reverse proxy such as Cloudflare. If Cloudflare sits in front of Vercel, bot decisions happen there, and our Cloudflare guide covers them.

Can AI crawlers read your Next.js pages?

Usually, as long as the content is in the HTML the server sends. Vercel's own analysis of AI crawler traffic found that none of the major AI crawlers render JavaScript, naming OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot among them. Gemini, which uses Googlebot's infrastructure, and Applebot were the exceptions.

The App Router helps by default. The Next.js docs say layouts and pages are Server Components by default, rendered on the server, and Client Components are prerendered to HTML on first load as well. The problem cases are narrower:

• Copy fetched in the browser after the page loads, for example in a useEffect hook, so the server HTML only holds a loading state.
• Pricing tables, feature lists or FAQs pulled from an API on the client.
• Pages built as a single-page app around an empty root element.

To check a page yourself, look at its HTML without a browser: curl -s https://example.com/pricing and search the output for a sentence from the page. If it isn't there, crawlers don't see it either. Monitor runs the same test on every key page every day and counts the words in the raw HTML.

How do robots.txt and noindex work on Vercel?

In the App Router, robots.txt is either a static app/robots.txt or generated by an app/robots.ts file that returns rules per user agent, as the Next.js robots.txt reference shows. Generated files make it easy for a refactor to change rules nobody reviews, so treat that file like production config.

Noindex has a Vercel-specific twist. Vercel adds an X-Robots-Tag: noindex header to preview deployments, and to a production deployment once a newer one replaces it. That's what you want on previews. It goes wrong when a production domain ends up pointing at the wrong deployment, or when headers set in next.config.js for staging leak into production. Monitor reads both the robots meta tag and the X-Robots-Tag header on every key page, and a new noindex is a critical alert.

How do I add the Monitor middleware to Next.js?

The connector is a middleware file that checks each request's user agent and, for AI agents only, reports the visit to Monitor in the background. It works on Vercel and on any other host that runs Next.js middleware.

1. Copy your snippet. Monitor's Connect logs section has a Next.js tab with your ingest key already in the code. The code below is the same with YOUR_INGEST_KEY as a placeholder.
2. Add the file. Save it as middleware.ts in the project root, or in src/ if your app lives there, as the Next.js middleware docs describe. If you already have a middleware file, merge the AI check into your function and keep one config export.
3. Deploy. Push as usual. No environment variables or packages are needed.
4. Test it. Run curl -A "Claude-User/1.0" https://example.com/ against your production domain, then reload your site in Monitor. The visit shows under Latest visits.

// middleware.ts (Next.js on Vercel or any host). Merge into your existing middleware if you have one.
import { NextResponse, type NextRequest, type NextFetchEvent } from "next/server";

const ENDPOINT = "https://monitor.airanked.ai/api/hit/YOUR_INGEST_KEY";
const AI = /OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-externalfetcher|DuckAssistBot|MistralAI-User|Amazonbot|CCBot|Bytespider/i;

export function middleware(req: NextRequest, event: NextFetchEvent) {
  const ua = req.headers.get("user-agent") || "";
  if (AI.test(ua)) {
    const ip = (req.headers.get("x-forwarded-for") || "").split(",")[0].trim();
    event.waitUntil(fetch(ENDPOINT, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ p: req.nextUrl.pathname, ua, ip, t: Date.now() }),
    }).catch(() => {}));
  }
  return NextResponse.next();
}

export const config = { matcher: "/((?!_next/static|_next/image|favicon.ico).*)" };

The matcher skips _next/static, _next/image and the favicon, so static assets never run the check. Vercel's docs note that middleware runs before the cache, so requests for prerendered and cached pages are still seen. The report goes out through event.waitUntil(), which keeps the work alive after the response is returned, and errors are swallowed so a Monitor outage can't affect your site.

One difference from the other connectors: middleware runs before the page renders, so it can't know the response status. Visits from this connector are stored without a status code.

What changes on Next.js 16?

Next.js 16 deprecated middleware.ts and renamed it proxy.ts. According to the Next.js 16 release post, middleware.ts still works for Edge runtime use cases but will be removed in a future version, and proxy.ts runs on the Node.js runtime.

The Monitor snippet works in both. To move to the new convention:

• Rename the file to proxy.ts and the exported function from middleware to proxy, or run npx @next/codemod@canary middleware-to-proxy ., which does both.
• Keep the second parameter: proxy receives the same NextFetchEvent with waitUntil().
• Leave out any runtime option. Proxy files reject it and always use Node.js.

Which client IP does the middleware report?

The snippet reports the first address in x-forwarded-for. On Vercel that's reliable: Vercel's request header docs say the header holds the public IP of the client and that Vercel overwrites it rather than forwarding external values, to prevent spoofing.

It matters because Monitor verifies OpenAI and Perplexity visits against the IP ranges those companies publish, and drops visits that don't match. Two setups break that:

• A proxy in front of Vercel. Vercel warns the header can be overwritten when another proxy sits on top. If Cloudflare proxies your domain, install the Cloudflare Worker connector instead, which reads Cloudflare's own client IP header.
• Self-hosted Next.js behind a load balancer that doesn't set x-forwarded-for to the client IP. Fix the header at the load balancer, or report from the edge instead.

How does Monitor watch a Vercel site every day?

Deploys are where Next.js sites usually break for crawlers, and Vercel makes deploying easy, so the checks run daily rather than once.

• Every key page is fetched as raw HTML and compared with the previous check. You hear about a page that drops below 120 words, loses more than half its text, gains a noindex, changes its canonical or loses a JSON-LD type such as FAQPage.
• The homepage is requested as seven AI agents and as a browser, so a firewall rule that starts denying AI bots raises a critical alert naming the agents.
• robots.txt is read from the live domain, so a change made through app/robots.ts is caught even when nobody edits a text file.
• With the middleware installed, you see the pages AI assistants load while answering real people, and get an alert if a regular visitor stops coming.

The alerts reference lists every alert, and the 14-day trial covers up to 10 sites, free, with no card needed.

Frequently asked questions