AI search optimization: what the engines actually ask for, and what to fix first

Most advice on this topic is written by people selling a new acronym. The useful sources are thinner and more boring: documentation from the companies that run the engines, plus a few large studies. This page reads those sources side by side, shows where they disagree, and turns them into an order of work you can defend to a boss. If you want the vocabulary first, our explainer on generative engine optimization covers the term most people use for the same discipline, and AEO vs SEO vs GEO sorts out the naming.

What is AI search optimization, in plain terms?

It's SEO extended to engines that answer instead of listing links. An AI search engine takes a question, runs one or more searches behind the scenes, reads a handful of the pages it finds and writes a reply that cites some of them. Google calls the first part grounding and the second query fan-out, where the model generates related queries to gather more results1.

OpenAI describes the same mechanics for ChatGPT. Its help center gives the example of a researcher asking about CCR8 cancer drugs: ChatGPT first sends a search partner a rewritten query, reviews the results, then sends more specific follow-up queries3 (OpenAI Help Center). So you're optimizing for searches the user never typed. That single fact explains most of what follows.

There are two outcomes to chase, and they're different. A citation is your URL appearing as a source. A mention is your brand named in the answer text, often because some other page you don't own named you. A B2B software company usually cares more about mentions on "best X for Y" prompts. A publisher cares more about citations. Decide which one you're paying for before you start, because the work diverges fast.

What do the companies behind the engines actually tell you?

Five operators publish guidance that matters here. Google wrote a full guide. Microsoft's Bing team wrote a long blog post. OpenAI, Perplexity and Anthropic mostly document their crawlers. Put next to each other, they agree on more than the hype suggests and disagree on two specific points.

Sources: Google's AI optimization guide1, Microsoft's post on inclusion in AI answers2, OpenAI's crawler overview4, Perplexity's crawler docs5, Anthropic's help center6 and Bing's AI Performance announcement7. "Not stated" means we found nothing in that company's documentation, which is different from "no".

Google: it's still SEO, and skip the hacks

Google's guide to generative AI features is blunt. It treats "AEO" and "GEO" as ordinary SEO, tells you to create "non-commodity" content with first-hand experience, and lists things you don't need to do: llms.txt files, special markup, chunking, rewriting for synonyms, chasing inauthentic mentions, and piling on structured data1. It also warns that mass-producing pages to sway AI answers can fall under its scaled content abuse policy.

Two lines in that guide deserve more attention than they get. Pages must be eligible to show with a snippet, so a nosnippet rule quietly takes you out. And Google says no third-party tool has access to its ranking or AI systems, which is a polite warning about vendors promising "internal" metrics.

Microsoft: write sections that survive being quoted alone

Krishna Madhavan, a principal product manager at Bing, published the most practical document any operator has released (Microsoft Advertising blog). It asks for one- or two-sentence direct answers, sentences that make sense out of context, descriptive H2 and H3 headings, Q&A pairs, comparison tables, and claims anchored in measurable facts. Its own example swaps "quiet dishwasher" for a 42 dB dishwasher designed for open-concept kitchens2.

The same post lists what hurts: long walls of text, answers hidden in tabs or expandable menus that AI systems may not render, core facts locked in PDFs, details that exist only in images, and vague claims like "next-gen" with nothing behind them.

OpenAI, Perplexity and Anthropic: mostly about access

These three say little about content and a lot about crawlers. OpenAI says sites that opt out of OAI-SearchBot won't appear in ChatGPT search answers, that robots.txt changes take about 24 hours to reach search results, and that GPTBot (training) is a separate setting4. Its help center adds that a site's host or CDN must allow traffic from OpenAI's published search bot IP addresses3. That second point catches more sites than robots.txt does, because bot-fighting rules at the CDN are often switched on by someone outside marketing.

Perplexity separates PerplexityBot, which surfaces sites in results, from Perplexity-User, which fetches pages on a user's request and "generally ignores robots.txt rules"5. Anthropic runs three agents: ClaudeBot for training, Claude-User for fetches a user asks for, and Claude-SearchBot for search indexing. Its help center says disabling Claude-SearchBot "may reduce your site's visibility" in search results6. Our AI crawlers list has every user agent string.

The four layers of AI search optimization, with a test for each

Everything practical fits into four layers. Work them in order, because a failure low down makes the layers above it worthless. Each one has a test you can run this week.

1. Access: can the search bots reach the page?

Pass if OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot all get a 200 response on your ten most important URLs, and none of those URLs carry noindex or nosnippet. Check robots.txt with our robots.txt AI checker, then check the CDN, because a robots.txt that allows a bot means nothing if a firewall rule returns 403 to it. Your server logs are the only proof; filter them for each user agent and look at status codes.

Decide training and search separately. In our study of 261 sites, 31 of the 59 sites that disallow GPTBot still allow OAI-SearchBot, and every site that blocked an AI search bot also blocked at least one training bot. A template that keeps search open while closing training looks like this:

# Keep AI search visibility, opt out of model training
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Be careful with the last block. Google-Extended controls whether your content trains future Gemini models and grounds answers in the Gemini app and Vertex AI. Google says it doesn't affect inclusion in Google Search or act as a ranking signal14 (Google crawler docs), and AI Overviews and AI Mode run on Search's own systems. If you want the Gemini app to ground answers in your pages, leave that line out, and drop the training lines entirely if you want your content in future models.

2. Retrieval: are you in the index each engine searches?

Pass if your key pages are indexed in both Google and Bing. That sounds like old SEO, and it is, but the two indexes matter for different engines. Google's AI features draw on Google's index1. Copilot's citations are reported inside Bing Webmaster Tools7, which tells you which index it leans on. ChatGPT partners with "other search providers" and requires access for its own crawler3. Plenty of sites have never opened Bing Webmaster Tools, which means they've never checked whether half the AI market can find them.

Don't read your Google rankings as a proxy. In Ahrefs' study of 15,000 long-tail prompts, only about 12% of links cited by ChatGPT, Gemini and Copilot sat in Google's top 10 for the same prompt. Perplexity matched Google most closely at 28.6%, and ChatGPT's in-text citations matched just 8%8. Fan-out explains the gap: the engine searched for something other than your keyword.

3. Selection: is there a passage worth quoting?

Pass if a stranger can copy any single section of your page into a chat window and it still makes sense, names the subject, and contains at least one checkable fact. This is where Microsoft's advice earns its keep. The academic evidence points the same way: the GEO paper by Aggarwal and colleagues, accepted to KDD 2024, found that adding citations, quotations and statistics could raise a source's visibility in generated answers by up to 40%, with results varying by domain11 (arXiv).

A before-and-after makes the standard concrete. This is a hypothetical payroll company we've called Ledgerline; the product and its numbers are invented for illustration.

Run your own pages through the citability grader for a quick read on how quotable each section is, then fix the worst three pages by hand.

4. Mentions: are you named on the pages that get cited?

Pass if you appear on at least half of the third-party URLs that engines cite for your top ten buyer prompts. This layer moves recommendations more than anything on your own site. In Ahrefs' study of 75,000 brands, branded web mentions correlated with AI Overview visibility at 0.664, against 0.218 for the number of backlinks and 0.326 for Domain Rating10. Ahrefs says plainly that correlation isn't causation, and stronger brands get both. Still, it's the best large-sample signal anyone has published.

Google's guide warns against "inauthentic mentions"1, and it means it. The legitimate route is slow: get reviewed, get included in roundups that are already cited, answer real questions in the communities your buyers read, and publish data others want to quote. Our guide on how AI engines choose sources shows how to find the cited URLs for a prompt.

Why one optimization doesn't cover every engine

Engines built by the same company still disagree about sources. Ahrefs compared Google's two AI products on 540,000 query pairs and found their answers 86% semantically similar on average, yet only 13.7% of the cited URLs overlapped9 (Ahrefs). Across companies the gap is wider, which is why per-engine playbooks exist. Ranking in ChatGPT depends on its search partners and the lists it retrieves, Perplexity tracks Google's top 10 more closely than the others (28.6% in the Ahrefs data above), and AI Overviews sit on the Google index.

Reach decides where to start. Google says AI Overviews have over 2.5 billion monthly active users and AI Mode passed 1 billion within a year of launch13 (Google I/O 2026). For many B2B categories, though, ChatGPT and Claude send the referrals that convert. Check your own analytics referrals before you pick a favorite engine; our AI search statistics page collects the published traffic splits.

How to decide what to fix first: a scoring model with a worked example

Teams waste months on tactics that are cheap and visible (an llms.txt file, a schema plugin) while the slow, decisive work (getting named on cited pages) never starts. A simple score fixes the order. We use:

priority = engine reach x prompt coverage / effort in days

engine reach    = share of your buyers' AI usage the fix touches (0 to 1)
prompt coverage = share of your tracked prompts the fix can affect (0 to 1)
effort          = working days to ship the fix

Here's the model applied to the hypothetical Ledgerline, tracking 40 buyer prompts. It assumes ChatGPT carries 45% of its buyers' AI usage, which is the kind of number you'd estimate from referral data and customer interviews. Every figure in the table is illustrative.

Read both numeric columns. The score tells you the order: unblock the bot this afternoon, then rewrite the money pages. The impact column tells you where the size is: fix D is second-biggest but scores last because it takes weeks of outreach, so it needs a named owner and a start date, or it never happens. Fix C ties with B on score only because we were generous; Google says it doesn't use llms.txt at all1. Our llms.txt explainer covers where it might still help.

For a full sequence over a quarter, the 90-day GEO strategy turns this kind of list into a calendar, and the GEO audit checklist has 32 pass/fail checks to find the fixes in the first place.

How do you measure AI search optimization?

Use the two free first-party reports first, then a fixed prompt set. Google's guide points to the Generative AI performance report in Search Console1. Bing's AI Performance report, in public preview since February 2026, shows total citations across Copilot, AI summaries in Bing and some partner integrations, the pages cited, and sampled "grounding queries", the phrases the AI used to retrieve your content7 (Bing Webmaster blog). Neither report covers ChatGPT, Perplexity or Claude.

For those, track mention rate and citation rate on a fixed set of prompts, run several times per engine. Mind the sample size. With 40 prompts run 3 times each you have 120 answers per engine; if your mention rate is 20%, the standard error is about 3.7 points, so the 95% margin is roughly plus or minus 7 points. A move from 20% to 25% is noise. A move from 20% to 35% is real. That arithmetic, more than any dashboard, decides whether last month's work did anything. Our guide to measuring AI visibility covers the metrics, and the AI visibility tools roundup compares the trackers that automate the runs.

Clicks are a weaker signal than they used to be. Pew Research Center tracked 900 US adults in March 2025 and found people clicked a regular result on 8% of visits when Google showed an AI summary, against 15% without one, and clicked a link inside the summary on just 1% of visits12 (Pew). Measure presence in the answer, not just sessions.

Mistakes we see on real sites, and the fix for each

• Training blocks that also kill search. A blanket User-agent: * disallow added to stop AI scraping takes OAI-SearchBot and PerplexityBot with it. Fix: name each bot explicitly, as in the template above.
• The answer lives in an accordion. Pricing details or specs sit behind a click-to-expand tab. Microsoft says AI systems may not render those2. Fix: put the key facts in visible HTML and use the accordion for detail.
• Spec sheets as PDFs. The only place the product's limits appear is a PDF download. Fix: publish an HTML version with headings.
• Reporting Google rank as AI visibility. With 12% overlap8, a rank report and an AI report measure different things. Fix: report them separately.
• Chasing one screenshot. Someone asks ChatGPT once, sees no mention, and starts a project. AI answers vary run to run. Fix: decide from 3 or more runs across the prompt set.
• Schema as the whole plan. Google says no special schema is needed for its AI features1. Add Organization and Product markup with our schema generator because it's cheap, then move on to work that matters more.

If you'd rather go deeper on the model side (training data, entity consistency, how LLMs form a view of a brand), our LLM optimization guide picks up where this one stops.

Frequently asked questions

Sources