Anything Engine Optimization The rolling record of the AI-search industry.

Reference

Current Best Practices

What the record supports, practice by practice. Every entry carries the date it was last verified and the references behind it — because in this field, an undated best-practices page is fiction within a quarter.

Last updated November 12, 2026.

Maintained by the editor; practices are re-verified against the record and re-dated, revised, or retired. This page sells nothing — the wire offers no optimization services.

Level 1 Foundations

Be readable, be quotable, be unambiguous, be honestly dated. Nothing below this line substitutes for it — 76% of the passages Google's AI Mode reuses most come from pages already ranking first organically.

  1. Decide your AI crawler policy deliberately — and per bot

    robots.txt, per user-agent

    AI companies run separate crawlers for model training, search indexing, and live retrieval, and the trade-offs differ: blocking a training bot withholds content from future models, while blocking an index or retrieval bot removes you from today's answers. Set robots.txt per user-agent with that distinction in mind, and re-check the strings periodically — companies rename crawlers as products rebrand, and a rule pointed at yesterday's string silently stops applying. IAB Australia's Bots and Crawler Guidance formalizes the same call into four verdicts per crawler category — allow, allow with conditions, require licensing, or block — a workable checklist for making the decision deliberately rather than defaulting to one blanket rule.

    A robots.txt block is a request, not a lock. TollBit found roughly 15% of AI page fetchers it tracked in the first half of 2026 reached European publisher URLs marked disallowed, with OpenAI's ChatGPT-User among the bots most often crossing blocks aimed specifically at it — treat a block as a policy statement worth logging compliance against, not a guarantee.

    For Cloudflare-proxied sites, part of that policy is now set in a dashboard rather than hand-edited into the file. Bot Preference Sync, rolling out across every plan from free to Enterprise starting August 21, 2026, keeps robots.txt in step with an AI-crawler policy configured in the dashboard. It prepends a managed block rather than replacing the file — Cloudflare says any existing Disallow directives are maintained — and the sync is opt-in and can be switched off at any time. It also changes what a no-training preference actually buys: a crawler that does both search and training keeps access to a site marked no-training only if it meets four conditions — honoring the preference, offering an AI-summary opt-out, reporting URL-level training use alongside search metrics, and publicly showing the block doesn't hurt search visibility. Continued access now turns on public disclosure, not a crawler's self-declared promise.

    Checked against Google, OpenAI and Anthropic's own documentation on August 27, 2026, none of the three publicly meets all four conditions — Google clears three, OpenAI two, Anthropic none. The condition every operator fails is the third: URL-level reporting of which pages were used for training, alongside search metrics. Cloudflare has published no compliance list of its own, and an operator could satisfy a condition through a private arrangement it never discloses — but on the public record, a dual-purpose crawler reaching a Cloudflare site that disallows training is doing so without publicly clearing the bar Cloudflare set for it.

    That fourth condition now has teeth on new domains. Starting September 15, 2026, Cloudflare's own default blocks Training and Agent crawlers on any page that displays ads, leaving Search allowed — but Cloudflare's post also states that "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training," because those crawlers carry search and training under one user-agent. A site that takes the default therefore loses Google's crawler on every ad-bearing page — the exact harm the fourth condition requires an operator to show doesn't happen. The default applies only to new domains onboarding to Cloudflare; an existing domain keeps its current configuration, and owners who don't want it can switch it off in Security settings any time before September 15. What "pages that display ads" resolves to in practice, and whether the change also reaches existing free-plan sites, is not addressed in Cloudflare's post.

    The user-agent string this whole policy runs on is itself forgeable. GreyNoise found 824 addresses across 795 networks sending traffic with forged AI-crawler names between July 28 and August 23, 2026 — including 263,849 sessions claiming to be Google-Extended, a token Google's own documentation says has no separate user-agent string at all, meaning none of those sessions could have been genuine. The forged traffic was scanning for credential and configuration files (.env, .aws/credentials, .git/config). Verify the connecting IP against each company's published crawler ranges before trusting what a User-Agent header claims.

    That policy differentiation is now visible in practice, not just in Cloudflare's disclosure requirement. The wire's own crawler-access panel — 55 publisher domains probed daily — caught three People Inc. properties (Allrecipes, Investopedia, People) and The Atlantic making an identical change on September 13, 2026: OpenAI's GPTBot and a plain browser request both went from a 200 to a Cloudflare-managed 403, while Anthropic's ClaudeBot and Perplexity's crawler kept drawing the same Pay Per Crawl 402 as before — the same site charging two crawlers for the access it now refuses a third outright. Not every large site is taking that route: Quora hardened the same day to a flat 403 for GPTBot, ClaudeBot, PerplexityBot and a browser alike, and its newly-readable robots.txt disallows all three crawlers by name and bars AI-training use outright.

    Three days later the same panel caught a second large publisher rejecting the split outright and one moving in the other direction. El País's homepage flipped from a 200 to a 403 for GPTBot, ClaudeBot and PerplexityBot alike between the September 13 and 16 captures. Stack Overflow moved just one crawler: it took Anthropic's ClaudeBot off Cloudflare's Pay Per Crawl paid track and onto a flat 403 block, while GPTBot and PerplexityBot stayed behind the same Cloudflare challenge as before — the panel's first capture of a crawler moving from paid access to blocked rather than the reverse.

    A block set one week is not necessarily the block still in place the next. A third capture, September 20, moved two of those September 16 findings in different directions. El País's homepage returned to a 200 for both a browser request and ClaudeBot — reversing its own September 16 block — even though El País's robots.txt still disallows ClaudeBot by name, so the file and the enforcement behind it now disagree. The Atlantic moved PerplexityBot off Cloudflare's Pay Per Crawl track onto a flat 403, while keeping Anthropic's ClaudeBot on the paid track unchanged — the first capture of a publisher ending one crawler's paid access while leaving another's in place. Treat any snapshot of a site's crawler policy as dated the moment it's taken, and re-check enforcement rather than assuming a robots.txt file describes it.

    A fourth capture, September 27, caught the divergence running the other way: a robots.txt more permissive than the enforcement in front of it. The Atlantic's robots.txt became readable again for the first time since September 13 and now names GPTBot, OAI-SearchBot and ChatGPT-User as allowed — but its live homepage still returns Cloudflare's 403 to GPTBot itself, unchanged since September 13. Neither document alone is reliable: check the file and the live response, because they can disagree in either direction. The same capture found Search Engine Land's robots.txt, readable throughout the panel's tracking, start returning the same Cloudflare challenge as its homepage — a trade publication covering this beat losing machine-readable access to its own stated policy.

    A separate measurement adds scale to the compliance question itself, and a new failure mode to the Cloudflare migration above it. HasData's AI Crawler Block Index tested 10,894 domains directly and found 592 with a declared GPTBot ban in robots.txt — of those, 234 (39.5%) still served GPTBot a live page when fetched, a declaration-level echo of TollBit's per-fetch 15% crossing rate. The same study found the migration itself cost 118 sites their existing opt-out outright: when Cloudflare retired its older "Managed Robots.txt" and "Block AI Bots" features on September 15, 2026, the replacement left robots.txt blank until the customer reviewed and confirmed new rules — a step Cloudflare's own announcements did not flag to existing customers. Named among the sites that lost their opt-out: the Smithsonian, Ko-fi, Find a Grave, Vatican News, National Review, Bellingcat and Patreon.

    Three more October readings sharpen what a robots.txt line can and cannot do. Google's Gary Illyes said in a Search Central video that robots.txt, as RFC 9309 defines it, governs which URLs a crawler may request, not what happens to the content afterward; keeping one AI crawler out means naming that crawler's own user-agent token, and an invented directive would simply be ignored by Google's parser. Google-Extended, the token that controls whether Google can use crawled content to train future Gemini models or for grounding, does not affect inclusion in Search. On how many sites actually do this, ContextFetch, a web-retrieval API vendor, read robots.txt on 7,897 of the 10,000 most-linked-to domains on October 19, 2026: GPTBot was blocked on 13.6% (Bytespider most, at 14.4%; Claude-SearchBot least, at 4.6%), and bigger sites block more, with 17.4% of the top 1,000 blocking GPTBot against 12.1% of those ranked 5,001-10,000. Vercatus, an AI-visibility vendor, probed 64 well-known sites on September 18 and found access sorting by niche rather than by a single norm: software and SaaS let GPTBot, ClaudeBot and PerplexityBot in on 7 of 8 sites, travel and hospitality on 1 of 8. Both are vendor measurements, and Vercatus tested eight sites per niche. Perplexity's updated crawler documentation also still says Perplexity-User, which runs when a person's question prompts a page visit, generally ignores robots.txt, so for that bot a firewall rule, not the file, is the control.

    A November 2 scan puts a scale on the same gap from the other side. SearchDock, an SEO platform vendor, requested the homepages of the Tranco top 1,000 domains on October 20, 2026 as a browser and as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Googlebot. Of the 555 that served a homepage, 27.2% blocked at least one AI crawler: 20.5% in robots.txt and 10.3% at the server or CDN. GPTBot was turned away at the server by 9% of the sites tested (49 of 542) against 14.4% that disallow it in robots.txt, and 35 sites (6.5%) turned it away while their robots.txt allowed it. A robots.txt file read alone therefore misses some blocks; check the live response as well. SearchDock's requests came from one location using crawler user-agent strings rather than the vendors' own IP ranges, so sites that admit only verified crawler IPs can look blocked, and it fetched each site once.

    On October 30 the training half of this policy started to converge on one line. Google, Microsoft, OpenAI and Anthropic each said they will honor a single robots.txt policy token, `__usage_ai-training`, defined in an individual IETF Internet-Draft by Anthropic's Kevin Kelley, so one rule could opt a site out of training across all four instead of one rule per company's crawler. None of it is live yet, and the dates differ: Microsoft says it will honor the preference as it crawls from February 1, 2027, Google says early 2027, OpenAI "in the coming months", and Anthropic is reported to have started. OpenAI's and Anthropic's positions come from secondary reports; the desk could not retrieve OpenAI's own post. The token is not a crawl directive: the draft says it states a preference about training only, wildcard `User-agent: *` groups do not match it, and it does not reach content already collected or answer-time use of a page. Google says its existing controls keep working beside it, and plans to treat the shared opt-out, where Google-Extended is not named, as a Google-Extended disallow. Microsoft separates the second decision as well: AI Grounding, the use of a page to inform generated answers, becomes a per-page setting independent of training, with Microsoft warning that disallowing it means reduced brand visibility in its AI experiences. Until the dates arrive, the per-crawler rules above remain the working control.

    Common questions

    Should I block AI crawlers in robots.txt?

    The decision is per bot, not blanket. Blocking a training crawler withholds content from future models; blocking an index or retrieval crawler removes you from the answers engines give today.

    How do I decide which AI crawlers to allow?

    IAB Australia's guidance sorts every crawler category into one of four verdicts — allow, allow with conditions, require licensing, or block. Re-check the user-agent strings periodically: companies rename crawlers as products rebrand, and a rule pointed at yesterday's string silently stops applying.

    Do AI crawlers actually respect robots.txt blocks?

    Not reliably. TollBit found about 15% of AI page fetchers it tracked reached European publisher URLs marked disallowed in the first half of 2026, with OpenAI's ChatGPT-User among the bots most often crossing blocks aimed specifically at it — a block is worth monitoring, not just setting.

    Does Cloudflare change my robots.txt for AI crawlers now?

    Only if you switch it on. Bot Preference Sync, rolling out from August 21, 2026 across every plan, keeps robots.txt in step with an AI-crawler policy set in the Cloudflare dashboard. It prepends a managed block and maintains any existing Disallow directives rather than replacing the file, and it is opt-in and reversible at any time. It also conditions continued access for dual-purpose search-and-training crawlers on public disclosure rather than a self-declared promise.

    Does any major AI crawler actually meet Cloudflare's four disclosure conditions?

    Not on the public record as of August 27, 2026. Google meets three of four, OpenAI two, Anthropic none it documents — every operator fails the requirement to report URL-level training use alongside search metrics, which makes the no-training preference a block rather than a standard for now.

    Does Cloudflare's September 15 default also block Google's search crawler?

    Yes, on pages that display ads, if the default is taken. Cloudflare says its own multi-purpose crawlers — Googlebot, Applebot and BingBot — are blocked by customers who block Training, because they carry both behaviors under one user-agent. The default applies to new domains onboarding to Cloudflare; existing domains keep their current settings, and owners can opt out in Security settings any time before September 15.

    Can I trust the User-Agent string to identify a real AI crawler?

    No. GreyNoise found 824 addresses forging AI-crawler names, including sessions claiming to be Google-Extended — a token with no user-agent string at all — while scanning for credential files. Verify the connecting IP against each company's published crawler ranges instead of trusting the header.

    Are publishers now charging some AI crawlers while blocking others?

    On some Cloudflare-proxied publishers, yes as of September 13, 2026: Allrecipes, Investopedia, People and The Atlantic block OpenAI's GPTBot outright while still billing Anthropic's ClaudeBot and Perplexity's crawler through Cloudflare's Pay Per Crawl. Other large sites reject that split — Quora and, as of September 16, El País block all three crawlers and a browser alike, regardless of payment. The split can also run in reverse: Stack Overflow moved ClaudeBot off its Pay Per Crawl track onto a flat block on September 16, while leaving GPTBot and PerplexityBot behind an unpaid Cloudflare challenge. None of it is necessarily stable: El País reopened to ClaudeBot and browser traffic on September 20, even though its robots.txt still disallows ClaudeBot by name, while The Atlantic ended PerplexityBot's paid access the same day and left ClaudeBot's unchanged.

    Can a site's robots.txt say a crawler is allowed while Cloudflare still blocks it?

    Yes. As of September 27, 2026, The Atlantic's robots.txt became readable again and names GPTBot, OAI-SearchBot and ChatGPT-User as allowed — but its live homepage still returns Cloudflare's 403 to GPTBot itself, unchanged since September 13. Check the live response as well as the file; they can disagree in either direction, as El País's September 20 reversal showed the opposite way.

    Did Cloudflare's robots.txt changes silently remove existing AI opt-outs?

    Yes, for some sites. HasData found 118 sites — among them the Smithsonian and Vatican News — lost their Cloudflare-managed AI-training opt-out and GPTBot-ban lines when Cloudflare retired its older Managed Robots.txt and Block AI Bots features on September 15, 2026. The replacement, Bot Preference Sync, leaves the file blank until the site owner reviews and confirms new rules, a step Cloudflare's own announcements didn't flag to existing customers.

    Do sites that declare a GPTBot ban in robots.txt actually block it?

    Not always. HasData tested 592 sites with a declared GPTBot ban directly and found 234 of them (39.5%) still served GPTBot a live page — a declaration-level version of the fetch-level gap TollBit measured above.

    Does blocking an AI crawler in robots.txt stop my content being used for training?

    Not by itself. Google's Gary Illyes said robots.txt governs which URLs a crawler may request, not what happens to the content afterward, and that excluding one AI crawler means naming its specific user-agent token. Google-Extended controls whether content trains future Gemini models or grounds answers, and does not affect inclusion in Search.

    How many websites block GPTBot?

    About 13.6% of the sites ContextFetch could read, per its October 19, 2026 check of the 10,000 most-linked-to domains, with larger sites blocking more: 17.4% of the top 1,000 against 12.1% of those ranked 5,001-10,000. It is a vendor measurement, and the sites it could not read are not counted.

    How many sites block AI crawlers at the server instead of in robots.txt?

    In SearchDock's October 20, 2026 scan of the Tranco top 1,000 domains, 9% of the sites tested (49 of 542) turned GPTBot away at the server, against 14.4% that disallow it in robots.txt, and 35 sites (6.5%) did so while their robots.txt allowed it. Overall 27.2% of the 555 sites that served a homepage blocked at least one AI crawler. It is a vendor scan from one location using crawler user-agent strings, so sites that admit only verified crawler IPs can look blocked.

    Which AI companies will honor a shared robots.txt AI training opt-out?

    OpenAI, Anthropic, Google and Microsoft said on October 30, 2026 that they will honor one robots.txt token, `__usage_ai-training`, defined in an IETF Internet-Draft. It is not live yet: Microsoft says from February 1, 2027, Google early 2027, OpenAI in the coming months. The token states a training preference only, not a crawling rule, and the draft has no formal standing as a standard.

    Verified 2026-11-02 · References: AI crawlers — glossary · Google robots.txt documentation · NotebookLM crawler renamed (wire) · IAB Australia sorts AI crawlers into four access verdicts (wire) · ChatGPT ads expand to Europe as crawler bypasses blocks (wire) · Cloudflare syncs robots.txt, sets four conditions for crawlers that also train (wire) · No major AI crawler publicly meets all four of Cloudflare's conditions (wire) · Cloudflare's September 15 default blocks Googlebot on ad-bearing pages (wire) · GreyNoise finds forged ClaudeBot and GPTBot traffic hunting for credentials (wire) · People Inc. sites and The Atlantic block GPTBot, still bill Claude, Perplexity (wire) · El País blocks GPTBot, ClaudeBot and PerplexityBot (wire) · El País reopens to ClaudeBot; The Atlantic ends Perplexity's paid access (wire) · The Atlantic's robots.txt allows GPTBot, but Cloudflare still blocks it (wire) · HasData: Cloudflare's robots.txt change erased GPTBot bans at 118 sites (wire) · Google: robots.txt limits AI crawling, not AI training use (wire) · ContextFetch finds bigger sites block AI crawlers more often (wire) · Vercatus: AI crawlers read SaaS sites, not travel sites (wire) · Perplexity updates PerplexityBot and Perplexity-User crawler documentation (wire) · OpenAI and Anthropic join Google, Microsoft in backing one robots.txt AI-training opt-out (wire) · Bing splits AI training from AI grounding in new publisher controls (wire) · OpenAI and Anthropic join Google, Microsoft in backing one robots.txt AI-training opt-out (wire) · SearchDock scan finds top sites block AI crawlers at the server, not just robots.txt (wire)

  2. Answer the reader clearly, with evidence

    Clear answers, original evidence, descriptive headings

    Lead with a clear answer when that helps the reader, then give evidence and limits. Google specifies no ideal page length or special AI-search format. Pillarbase reconstructed 528 client passages with a median length of 117 words, separate from its high-reuse third-party cohort. This is an observation about a selected sample, not an ideal length for earning citations.

    Wording matters as much as shape. A preprint analyzing roughly 2 million citations from ChatGPT, Claude, Google AI and Gemini, joined to 10,000 pages across 19 B2B SaaS workspaces, found prompt-content alignment, the word overlap between a page and the wording of real prompts, was the strongest page-level predictor of citation (beta +0.37), while FAQ blocks, structured data and Core Web Vitals stopped predicting it once domain-level effects were controlled for. The authors run an AEO agency and the paper is not peer reviewed, but it points the same way as the passage data: answer in the words people actually ask with.

    A later study supplies the mechanism behind that domain-level finding. Tracking 90 days of ChatGPT, Gemini and Perplexity answers (158,099 answers, 468,037 citations), University of Toronto researchers found that whether ChatGPT's own rewritten search queries retrieved a page explained 32% of which pages it cited within a given site, against just 2.8% for page content and structure — a gap the study calls statistically decisive. Content still mattered, but mainly through matching the wording of those rewritten queries, not through FAQ blocks, structured data or page speed, which only showed an effect when comparing across different websites rather than between pages of the same one. A separate experiment in the same study found the models' own prior knowledge skews results too: with every version of a product holding identical facts, revealing its brand name changed ChatGPT's top-three ranking rate 2.1 times as much as merely repeating the prompt, and 3.3 times as much for Gemini.

    Heading structure and question-phrased titles show the shape this practice recommends in a newer study — on the figure, not the mechanism this practice tracked before. Sistrix matched 48,527 AI-cited pages against uncited pages on the same domain across 96,418 URLs and found pages with five or more H2 headings were cited 1.6 times as often as matched pages with fewer; on ChatGPT specifically, a question-phrased title carried a separate 1.9x factor. The same study found FAQPage schema and other markup also correlate with citation once domain is controlled for — in tension with the domain-authority study above, which found markup's apparent effect disappear under the same control. See the structured-data practice for that disagreement; it doesn't touch the heading-count and question-title findings here, which have no competing null result on the record.

    A direct measurement of the answer-first opening, with a limit on what it shows. TJ Digital, an SEO and generative engine optimization agency, took blind ratings of 195 page pairs drawn from 193,507 answers by ChatGPT, Google AI Mode and AI Overviews for 78 client businesses: 20% of cited pages opened with a direct answer against 5% of Google top-10 pages the AI never pulled up. Against pages the same answer retrieved but didn't cite, the gap narrows to 23% versus 14%. Opening with the answer therefore separates cited pages from ignored ones far more than from pages the AI read and still skipped. The agency sells these services, calls its findings associations rather than causes, and its Google ranking data is a single snapshot.

    Common questions

    How do I get my content quoted by AI search?

    Make the page accessible, answer a real reader question, and support the answer with original evidence and clear sources. Citation is not guaranteed. Passage and wording studies describe associations; they do not establish a universal format that earns citations.

    How long should an answer passage be?

    As long as the reader needs to understand the answer and its limits. Pillarbase reconstructed 528 client passages with a median length of 117 words, separate from its high-reuse third-party cohort. This is an observation about a selected sample, not an ideal length for earning citations.

    Does matching the wording of real prompts help a page get cited by AI?

    One preprint found an association: prompt-content word overlap was its strongest page-level predictor (beta +0.37). It does not establish that rewriting wording causes more citations. The authors run an AEO agency, and the paper is not peer reviewed.

    Does matching an AI engine's own search queries matter more than content quality for getting cited?

    In one University of Toronto study, retrieval explained more variation in citations within a site than the page features measured. That is not a test showing content quality is unimportant, nor a guarantee that matching query wording will improve citations.

    How do I get my page cited in Google AI Overviews?

    Google applies its normal search foundations: make the page crawlable and indexable, provide useful original content, and qualify for a search snippet. There is no special AI schema, fixed passage length, or required question-heading format. Eligibility does not guarantee a citation.

    How do I show up in AI search?

    Check the relevant search crawler and firewall access, then provide a useful, accurate page with clear ownership and evidence. Different engines retrieve and cite differently; indexing, retrieval, citation and a referral click are separate outcomes.

    How do I get cited by AI search engines?

    Provide something worth citing and make it accessible to the engine. Wording overlap was associated with citations in one agency-authored preprint, but it is not a universal strongest signal or a proven intervention. Measure changes across repeated runs.

    Does the number of headings on a page affect AI citation?

    Sistrix found an association: pages with five or more H2 headings were cited 1.6 times as often as matched uncited pages in its dataset. That does not show that adding headings causes citations. Use headings to organize the answer for readers.

    Does phrasing a page title as a question help it get cited by ChatGPT?

    Sistrix reported a 1.9x citation-odds association for question titles on ChatGPT in its sample. It was not a randomized test of title changes. Choose a question title when it accurately expresses the page's purpose.

    How do I show up in Google AI Overviews?

    Use the same search foundations as for Google Search: accessible, indexable pages with helpful original information and snippet eligibility. Google does not require special AI files, small content chunks or prescribed wording.

    Do pages cited by AI open with a direct answer?

    More often than pages the AI never pulled up, per TJ Digital: in blind ratings, 20% of cited pages opened with a direct answer against 5% of Google top-10 pages the AI never retrieved. Against pages retrieved but not cited, the split was 23% to 14%. The agency sells SEO and GEO services and calls the findings associations, not causes.

    Why aren't my top-ranking Google pages getting cited by ChatGPT?

    Ranking isn't the gate; being retrieved is. A University of Toronto study of 468,037 ChatGPT citations found whether ChatGPT's own rewritten search queries retrieved a page explained 32% of which pages on a site it cited, against 2.8% for page content and structure. TJ Digital also found that only 5% of Google top-10 pages the AI never pulled up opened with a direct answer, against 20% of cited pages. Both are associations, and the second comes from an agency that sells these services.

    Verified 2026-11-07 · References: Google: optimizing for generative AI features · RAG — glossary (why retrieval units matter) · AEO — glossary · Pillarbase: 15.7M AI Mode citations show what Google reuses (wire) · Study of 2M AI citations: domain authority beats SEO signals (wire) · Study: ChatGPT's own search explains citations better than page content (wire) · Sistrix ranks GEO tactics by how often AI engines cite them (wire) · TJ Digital study: AI-cited pages more often open with a direct answer (wire)

  3. Ship structured data for what the page actually is

    Schema.org JSON-LD

    Google retired FAQ rich results in May 2026 and removed their documentation in June. FAQPage remains a schema.org type, but adding it does not earn a Google FAQ rich result or special AI-search eligibility. Useful visible answers and accurate markup are separate decisions.

    Schema.org markup — NewsArticle for news, FAQPage for question-and-answer content, DefinedTerm for definitions, Organization and Person for who stands behind it — is machine-readable self-description, and it is cheap. Mark up what the page genuinely is; structured data that misdescribes a page is a trust liability, not an optimization. That framing now has a direct test behind it: an Ahrefs analysis of 1,885 pages found adding schema markup produced no measurable increase in AI citations after 30 days — treat it as descriptive infrastructure, not a lever for citation volume.

    A separate study widens that finding past schema specifically. Re-testing the original 2023 GEO citation-lever effect sizes against ten modern AI engine families found none of them moved citations any longer, and a deterministic content score built from those levers correlated with actual citations at just 0.11 within-query — weak enough that the authors position such scores as content-quality filters, not citation predictors. The on-page levers that still measure are structural (crawlability, recency, passage shape), not scored content attributes.

    A third study explains why schema keeps testing null: across roughly 2 million citations tied to 10,000 pages, domain-level AI authority outweighed every page-level feature — schema included — sixfold in SHAP value, and FAQ blocks, structured data and Core Web Vitals all showed a positive correlation with citations that reversed or fell to zero once that domain effect was controlled for. The strongest surviving page-level predictor was how closely a page's wording matched real user prompts, not any traditional SEO signal.

    One case now makes markup a gate rather than a lever. Google's UGC Fresh Data Program, documented October 8, 2026, lets qualifying forum and social platforms submit fresh user-generated content directly to Search outside normal crawling — but only if the platform uses schema.org markup such as SocialMediaPosting or DiscussionForumPosting, alongside OAuth 2.0 authentication and active moderation. That's a different role for structured data than the citation-ranking tests above: here it's an eligibility requirement for a faster ingestion channel, not something Google claims moves citations once content is in.

    A fourth study complicates the null result rather than confirming it. Sistrix matched 48,527 pages cited by ChatGPT, Google AI Mode and AI Overviews against uncited pages on the same domain — controlling for domain-level authority by the same logic as the domain-authority study above — across 96,418 URLs and 17,380 domains, and still found Article schema markup carried a 1.6x citation-odds factor and FAQPage schema a separate 1.5x, the third- and fifth-strongest of 43 features tested (the retrospectively added title-year feature had an observational odds ratio of 2.7x; a table of contents, used by only 14% of pages, carried 1.7x). That is a disagreement between two domain-controlled studies, not a resolution of the earlier one: the record holds both rather than picking a winner. Schema's citation effect, if any, is contested, not settled either way. None of these associations establishes that adding the feature causes citations.

    Common questions

    Which schema types should I use for AI search?

    Mark up what the page genuinely is: NewsArticle for news, FAQPage for question-and-answer content, DefinedTerm for definitions, and Organization and Person for who stands behind it.

    Can structured data hurt my site?

    Yes. Structured data that misdescribes a page is a trust liability, not an optimization — and FAQ markup whose text a reader cannot see on the page violates Google's policy outright.

    Does adding schema markup increase AI citations?

    Contested. A study of 2 million citations found schema's apparent positive correlation with citations collapsed to zero once domain-level AI authority was controlled for; a later Sistrix study of 96,418 URLs, also matching pages against an uncited page on the same domain, still found a 1.6x citation-odds factor for Article schema and 1.5x for FAQPage schema. Mark up what the page genuinely is regardless — the citation-lever question is unresolved, not debunked.

    Does a table of contents help a page get cited by AI search engines?

    In one study, yes. Sistrix found pages with a table of contents were cited 1.7 times as often as matched, uncited pages on the same domain — though only 14% of pages in its 96,418-URL dataset used one.

    Do GEO content-scoring tools predict which pages AI engines cite?

    Barely. A study re-testing 2023-era GEO content levers against ten modern AI engine families found none of them moved citations, and a content score built from those levers correlated with actual citations at just 0.11 within-query — treat such scores as quality filters, not citation predictors.

    Does domain authority matter more than on-page SEO for AI citations?

    Per a study of roughly 2 million citations across 10,000 pages, yes — domain-level AI authority outweighed the next-strongest page feature sixfold in SHAP value, and traditional SEO signals like schema, FAQ blocks and Core Web Vitals lost their apparent citation lift once domain was controlled for.

    Do Core Web Vitals have any impact on visibility in AI search results?

    Not on their own, in the one study that tested it. A preprint of roughly 2 million citations across 10,000 pages from 19 B2B SaaS workspaces found Core Web Vitals showed a positive correlation with citations that fell to zero or reversed once the analysis controlled for which domain the page sat on. It measured citations in AI engines, not Google ranking, and it is a single preprint on B2B SaaS sites.

    Does Google ever require structured data rather than just recommend it?

    In one narrow case, yes. Google's UGC Fresh Data Program requires forum and social platforms to use schema.org markup like SocialMediaPosting or DiscussionForumPosting to qualify for a direct content-submission pipeline — a gate for faster ingestion, not a claim that markup moves citations.

    Verified 2026-11-06 · References: Google Search documentation updates: FAQ rich results retired · Google structured data documentation · Schema.org · Ahrefs study: llms.txt files mostly unread by AI crawlers (wire) · GEO content scores barely predict AI citations, study finds (wire) · Study of 2M AI citations: domain authority beats SEO signals (wire) · Google launches UGC Fresh Data Program for forums and social platforms (wire) · Sistrix ranks GEO tactics by how often AI engines cite them (wire)

  4. Be unambiguous about your entities

    Canonical hub pages, sameAs

    Knowledge graphs and answer engines resolve content to entities — companies, products, people. Give each entity you own a canonical hub page, keep names consistent, and declare external identities with sameAs links to official sites and reference pages. Ambiguity about which thing you mean is a retrieval failure you inflicted on yourself.

    The pages other people keep about you are part of your entity footprint. On searches for a brand's name alone, Ahrefs found Google's AI Overviews built mostly from third-party pages: of 57 brand-name overviews whose sources it could see, 41 cited Wikipedia, 22 YouTube and 13 LinkedIn, and only one a competitor's site (vendor-published). Wikipedia carries similar weight in ChatGPT, where it alone made up 10 to 14% of the domains cited in answers to SaaS buying queries, per a Growth Memo analysis. Check what the reference pages you declare with sameAs say about you, because answer engines read them as you.

    Name variants are an entity problem you can see in the counts. BrightEdge's AI Hyper Cube, covering thirteen US streaming brands in September 2026, found a year after the HBO Max rename that Google AI Overviews still mentioned "Max" more often than "HBO Max" (5.05% of brand mentions against 2.33%), while ChatGPT ranked HBO first and the two newer names below it. Engines count each name separately, so a company with a short name, a legacy name and a product name has its visibility split across several lines. BrightEdge says it could not identify why one name is captured over another, and does not publish mention or prompt counts (vendor-published). SE Ranking, a vendor, checked 1,000 US brand keywords in October 2026 and found an AI Overview on an average of 82.5%, with 61% of the citations in those overviews pointing to sources a brand manages or can post on (40.5% brand-managed, 20.8% social, video and review sites), and homepages only 36.8% of brand-site citations, against product, service and category pages at 22.9%. The study is desktop only; it counts where the citations point, not why.

    Common questions

    What is entity optimization?

    Knowledge graphs and answer engines resolve content to entities — companies, products, people. Entity work means giving each entity you own a canonical hub page and keeping its name consistent everywhere it appears.

    How do I tell AI engines which company my brand is?

    Declare external identities with sameAs links to official sites and reference pages. Ambiguity about which thing you mean is a retrieval failure you inflicted on yourself.

    What sources does Google's AI Overview cite for a brand's name?

    Mostly third-party pages, per Ahrefs: of 57 brand-name AI Overviews whose sources it could see, 41 cited Wikipedia, 22 YouTube and 13 LinkedIn, and only one cited a competitor's site.

    If my company renamed, will AI engines use the new name?

    Not necessarily. BrightEdge's AI Hyper Cube found a year after the HBO Max rename that Google AI Overviews still mentioned "Max" more often than "HBO Max" (5.05% of brand mentions against 2.33%). Engines count each name separately, so visibility can split across old and new names; BrightEdge could not say why one name is captured over another.

    Verified 2026-10-31 · References: The record — this site's entity hubs · schema.org/sameAs · Ahrefs: AI Overviews now appear on 83% of branded searches (wire) · UGC platforms outcite review sites in ChatGPT answers (wire) · SE Ranking: AI Overviews appear on 82.5% of US brand searches (wire) · BrightEdge: AI Overviews still mention "Max" more than HBO Max, a year after the rename (wire)

  5. Give readers a one-click way to name you a Preferred Source

    Preferred Sources embed button

    Google's Preferred Sources feature lets a reader mark a site as preferred across Search, AI Overviews and AI Mode, and Google says a preferred pick makes that site surface more often in all three. As of August 2026, publishers can embed a one-click "Add to Preferred Sources" button directly on their own pages rather than relying on readers to find the pick buried in search settings, and Google has now extended the same mechanism into AI Mode's new developing-topic link carousel. Adoption is real and growing: unique sources selected as Preferred rose from over 345,000 at the feature's May 2026 launch to over 600,000 by August, and Google said in October 2026 that use is up more than 7x since the start of the year — though it gave no absolute count alongside that multiple.

    Google names only Top Stories, AI Overviews and AI Mode as surfaces a preferred pick affects — Discover is not on that list, even though Discover now sends news publishers more than twice the Google traffic Web Search does (67.5% versus 27% in Q4 2025, reversed from 2023), per publishers tracked by NewzDash. A reader-facing pick does not substitute for Discover-specific work.

    Common questions

    What is Google's Preferred Sources feature?

    A reader-facing pick that Google says makes a site surface more often in Top Stories, AI Overviews and AI Mode. Publishers can now embed a one-click button on their own pages that adds the site as a Preferred Source.

    Is Google's Preferred Sources feature gaining adoption?

    Yes — unique sources picked as Preferred rose from over 345,000 at the feature's May 2026 launch to over 600,000 by August 2026, and Google has extended the mechanism into AI Mode's developing-topic link carousel. Google said in October 2026 that use is up more than 7x since the start of the year, without giving an absolute count to go with it.

    Does being a Preferred Source help a site show up in Google Discover?

    Google doesn't say. Its own documentation names only Top Stories, AI Overviews and AI Mode as surfaces a Preferred Source pick affects, not Discover — even though Discover now sends news publishers more than twice the Google traffic Web Search does (67.5% versus 27% in Q4 2025), per publishers tracked by NewzDash.

    Verified 2026-10-16 · References: Google: personalize your search and Discover with preferred sources · Google lets publishers embed a Preferred Sources button (wire) · Google's AI Mode gets link carousels for developing topics (wire) · Google Discover now sends news publishers more than twice what Search does (wire) · Google adds sources section, more links to AI Overviews and AI Mode (wire)

  6. Date everything, visibly and in markup

    Visible dates, honest dateModified

    Answer engines weigh freshness and increasingly surface dates alongside citations. Publish visible dates, keep dateModified truthful in markup, and never launder old content as new — an engine that catches a laundered date has a durable reason to distrust the domain. Honest dating is both a trust signal and a defense. The effect is measurable: a controlled Otterly experiment adding the current year to 11 page titles lifted their AI-search citations 61% — with the obvious corollary that the year must be true, and the content actually current.

    Seer classified the same 4,124 cited pages using two dates: 72% were less than a year old by last update, compared with 42% by original publication. This does not establish that refreshed pages outcite newly published ones or that an update causes a citation.

    The same signal predicts traditional Google ranking, not just AI citation: a Growth Memo analysis of nearly 60,000 search results pages found a listicle with an old or unreadable publish date had 56% lower odds of reaching the top 3 — the single strongest factor measured, ahead of author bios, schema markup and word count.

    Seasonal content shows the same failure at the category level, and on a deadline. A BrightEdge analysis of 13,915 holiday prompts against ChatGPT across six major US retailers (June-August 2026) found dated press releases outciting a retailer's own current, correct seasonal page: at Best Buy, a dated release drew 71.7% of Black Friday citations versus 10.5% for the live store hub; Home Depot split 63.1% to 13.3%. Weighted across all six retailers, about 11% of cited holiday pages carried a year that had already passed — nothing ages a stale press release out of the citation pool until the retailer replaces it, so a seasonal page needs updating before peak demand, not just published once and left.

    A third, independent measurement points the same way, with a caveat on its strength. TJ Digital, an SEO and GEO agency, compared 21,874 cited and 4,507 passed-over pages across 434 questions in ChatGPT, Google AI Mode and AI Overviews answers, adjusting for each site's Ahrefs Domain Rating and page type: a year in the title carried more than twice the odds of being cited, and a publish or update date within the past 12 months 88% higher. TJ Digital sells those services and describes the results as associations, not proof that a date causes a citation.

    Common questions

    Does updating old content help AI visibility?

    Seer classified the same 4,124 cited pages using two dates: 72% were less than a year old by last update, compared with 42% by original publication. This does not establish that refreshed pages outcite newly published ones or that an update causes a citation.

    Should I put the current year in my page titles?

    Only when it is true. Otterly reports a controlled experiment where adding the year to 11 page titles lifted AI-search citations 61%; that is the vendor's own figure, not one this wire has verified.

    Does a stale publish date hurt traditional Google ranking too, not just AI citation?

    Yes — a Growth Memo analysis of nearly 60,000 search results pages found an old or unreadable publish date was the strongest predictor of a listicle failing to reach the Google top 3, ahead of author bios, schema markup and word count.

    Does ChatGPT cite outdated seasonal press releases over a retailer's current page?

    Yes — per BrightEdge, at Best Buy dated press releases draw 71.7% of ChatGPT's Black Friday citations versus 10.5% for the current store hub page; Home Depot splits a similar 63.1% to 13.3%, across 13,915 holiday prompts tracked June-August 2026.

    How many AI-cited holiday pages are out of date?

    About 11% of cited holiday pages carry a year that has already passed, a weighted average BrightEdge measured across six major US retailers — a dated page doesn't age out of the citation pool on its own; the retailer has to replace it.

    How long does outdated product info stick around in AI answers?

    Indefinitely, unless someone replaces it. BrightEdge found dated holiday press releases kept outciting a retailer's own current page for as long as the retailer left the stale release live — about 11% of cited holiday pages weighted across six major US retailers still carried a year that had already passed.

    Are pages with a recent date or the year in the title cited more by ChatGPT and Google AI?

    Per TJ Digital's study of 21,874 cited and 4,507 passed-over pages, a year in the title carried more than twice the odds of being cited and a publish or update date in the past 12 months 88% higher, adjusted for Domain Rating and page type. TJ Digital sells SEO and GEO services and calls these associations, not causes.

    Verified 2026-11-07 · References: This site's editorial standards · Otterly title-year experiment — vendor blog, not wire-verified · Seer: content recency's impact on AI visibility (wire) · Growth Memo: freshness and Google listicle ranking (wire) · BrightEdge: ChatGPT's Black Friday citations still favor dated press releases (wire) · TJ Digital study: AI-cited pages more often open with a direct answer (wire)

  7. Know the AI Overviews opt-out — and what it doesn't reach

    Search Console → Settings → Search generative AI control

    Search Console carries a Search generative AI control, under Settings, that is the only lever separating appearance in AI Overviews, AI Mode and Discover's generative features from ordinary Search — unlike noindex, nosnippet or Google-Extended, which each take something else with them. Google says the control "isn't used as a ranking or inclusion signal affecting other parts of Search" and takes one to two days to apply. It rolled out to all sites worldwide as of August 31, 2026, but Google's own AI-features-specific guidance for site owners still does not mention it at all, naming nosnippet 18 times instead — unchanged when rechecked October 21, 2026. A separate, general Search Central page, "Control what you share with Google Search," was updated that same day to list the control alongside the robots meta tag, data-nosnippet and X-Robots-Tag, so the control is now documented somewhere in Google's help center — just not in the page aimed at this question.

    The control covers appearance, not training — Google's documentation sends publishers to Google-Extended for that, so it answers only one of the three uses (training, fine-tuning, grounding) the UK CMA's June 3, 2026 Publisher Conduct Requirement ordered Google to let publishers separately opt out of. There's a further caveat for news publishers specifically: Google renders Top Stories carousels inside AI Overviews on roughly 15.5% of US news searches, per an unconfirmed NewzDash analysis, so opting a news site out of AI Overviews may cost the Top Stories placement rendered within them, not just the summary.

    The control is set per Search Console property, not per domain. A site managing subdomains or subfolders as separate properties could previously have a child property quietly opt back into AI features after the parent domain opted out, with no visible flag. Search Console's documentation now says the top-level domain property shows a notification when a child property overrides the parent's setting, per Search Engine Roundtable.

    The first published before-and-after of using the control is one site, so treat it as a lead rather than a benchmark. SEO consultant Cyrus Shepard opted his Zyppy site out of Google's AI features from September 1 to September 29, 2026, then back in for another 28 days, and published the results on November 5. He estimates AI Overviews and AI Mode drove about 21% of the site's Google search clicks (23% counting a possible lift on regular listings) from about 10% of its impressions, at roughly 0.9% click-through on the AI impressions against about 0.4% for the site overall. The clicks are inferred: Search Console counts AI-feature clicks inside its standard totals and does not report them separately. Shepard calls it a single experiment on a site optimized to win AI citations, which may explain the high share; an earlier write-up of the same test from a LinkedIn post, per Search Engine Roundtable, also reported that regular organic clicks fell after the opt-out. He says the site left AI features within an hour, while per Search Engine Journal Google's help center says exclusions generally take several days, so the one-to-two-day figure above and his observation do not agree.

    Common questions

    How do I opt a site out of Google's AI Overviews without leaving Search entirely?

    Search Console's Settings carry a Search generative AI control that removes a site from AI Overviews, AI Mode and Discover's generative features while Google says it isn't used as a ranking or inclusion signal elsewhere in Search. It rolled out to all sites worldwide as of August 31, 2026, and takes one to two days to apply.

    Does opting out of AI Overviews also stop Google from training on my content?

    No. The control covers appearance only; Google's documentation points to Google-Extended for training, fine-tuning and grounding — the other two uses the UK CMA's June 2026 order requires publishers be able to opt out of separately.

    Is there a downside to opting a news site out of AI Overviews?

    Possibly. Google places Top Stories carousels inside AI Overviews on roughly 15.5% of US news searches per an unconfirmed NewzDash analysis, so excluding a site from AI Overviews may also remove it from that carousel placement.

    If my site has subdomains or subfolders set up as separate Search Console properties, can one of them silently override the main domain's AI opt-out?

    It could before this update. Search Console now shows the top-level domain property a notification when a child property's setting overrides the parent's, so the divergence is no longer invisible.

    How many clicks do AI Overviews and AI Mode send to a site that opts out?

    One published test estimates about 21% of one site's Google search clicks (23% counting a possible lift on regular listings), at roughly 0.9% click-through on AI impressions versus about 0.4% for the site overall. It is a single AI-optimized site, the clicks are inferred because Search Console does not report AI-feature clicks separately, and the author calls it a directional yardstick rather than a benchmark.

    Verified 2026-11-12 · References: Google Search Console Help: Search generative AI control · Google: AI features and your website · What Google's AI Overviews opt-out does, and two things it does not (wire) · Search Console's AI Overviews opt-out control rolls out to all sites (wire) · Search Console flags child-property overrides of its AI opt-out control (wire) · Google adds AI Overviews opt-out to 'Control what you share' guide (wire) · Zyppy opt-out test estimates AI Overviews and AI Mode drove 21% of one site's clicks (wire)

Level 2 Measurement

Knowing whether any of it worked, and noticing when the engines change underneath you. Only 16% of brands track their AI visibility at all.

  1. Wire the freshness signals: sitemaps, IndexNow, WebSub

    Sitemaps, IndexNow, WebSub

    Engines can only be as current as their discovery. Maintain an XML sitemap (a news sitemap if you publish news), ping IndexNow on every publish for the engines that accept it, and declare a WebSub hub on your feeds so subscribers are pushed rather than left to poll. None of this improves content; all of it shortens the gap between publishing and being retrievable.

    A sitemap "couldn't fetch" error in Search Console doesn't always mean the file is broken. Google's John Mueller and Martin Splitt said on the Oct. 1, 2026 "Search Off the Record" podcast that the error can also reflect Google's systems being too busy, or a judgment that a site's perceived quality doesn't justify more crawl budget — check the file is actually valid before treating the error as a technical fault to fix. Mueller also said RSS feeds can double as short-form sitemaps, typically listing 10-20 dated entries, which is a lighter-weight option than a full XML sitemap for a feed that updates often. Neither Google statement extends any role to llms.txt: Mueller said Google's search systems don't process llms.txt files as sitemaps at all and that site owners shouldn't treat it as a priority, consistent with llms.txt's standing as a bet on other engines, not Google.

    PDF files carry their own freshness ceiling. Google's documentation caps Googlebot's PDF fetch at the first 64MB of a file — a limit Google's own search advocates Splitt and Mueller couldn't recall on their own podcast despite it appearing in Google's crawler documentation. Converted PDFs rank in AI Overviews and AI Mode the same as standard HTML pages, but Google refetches PDFs far less often than HTML; Mueller's advice for updating one is to give it a new filename rather than edit the file at the same URL, since an edit left at the same address may sit unrefetched for a long stretch.

    Common questions

    How do I get new pages into AI answers faster?

    Maintain an XML sitemap — a news sitemap if you publish news — ping IndexNow on every publish for the engines that accept it, and declare a WebSub hub on your feeds so subscribers are pushed rather than left to poll.

    Does IndexNow help with AI search?

    It shortens the gap between publishing and being retrievable for the engines that accept it. None of this plumbing improves content; all of it speeds discovery.

    Why does Search Console show a sitemap 'couldn't fetch' error even though the file is valid?

    Google's John Mueller says the error doesn't always signal a technical fault — it can mean Google's systems are too busy, or that Google has decided a site's perceived quality doesn't justify more crawl budget.

    Does Google treat llms.txt as a sitemap?

    No. Mueller said Google's search systems don't process llms.txt files as sitemaps and that site owners shouldn't treat it as a priority — consistent with llms.txt being a bet on other engines, not a Google lever.

    Does Google have a file-size limit on crawling PDFs?

    Yes — Google's documentation caps Googlebot's PDF fetch at the first 64MB of the file, a limit its own search advocates couldn't recall on their own podcast. Converted PDFs rank in AI Overviews and AI Mode the same as standard HTML pages.

    Should I edit an existing PDF in place to update it, or upload a new file?

    Upload under a new filename, per Google's John Mueller — Google refetches PDFs far less often than HTML pages, so an edit left at the same URL may go unseen for a long stretch.

    Verified 2026-10-14 · References: IndexNow · WebSub (W3C) · Google sitemap documentation · Google: sitemap "couldn't fetch" errors can mean low crawl demand (wire) · Google's own hosts can't name its documented 64MB PDF crawl limit (wire)

  2. Get your first-party AI numbers where they exist

    Search Console, Bing Webmaster Tools

    The engines have started shipping first-party AI-surface data: Search Console added generative-AI performance reports, and Bing Webmaster Tools previews citation-share and AI-visibility insights. First-party beats scraped estimates wherever both exist — verify your site in both consoles and take the numbers the engines will actually give you. The surface keeps widening: Microsoft Clarity now splits AI citation data into branded and non-branded views, and Search Console's platform properties extend first-party measurement to social and video accounts. Clarity's Bot Analytics dashboard added an AI Scrape-to-Referral Ratio card on August 13, 2026, ranking which AI crawlers send return visitors against those that mostly extract content without driving traffic back — though Clarity itself cautions a high ratio doesn't by itself signal failing AI visibility, since many AI platforms strip referrer data and likely undercount real referrals. Baidu joined with an AI Traffic Analysis section in its own webmaster tools around September 15, 2026, but its numbers aren't directly comparable to the others: Baidu's AI CTR divides clicks by citations rather than by impressions, which can understate real link performance, per Search Engine Journal; the dashboards were still empty as the rollout continued in phases, and the tool covers only citations inside Baidu's traditional search results, not its standalone AI products or rival Chinese platforms. Treat each engine's first-party AI metric as defined by that engine, not as a common unit across consoles.

    Alex Galinos’s single-brand study found 22.4% of tracked text-fragment events assigned to Direct. Events are not unique clicks or sessions, and text fragments are not exclusive to AI Overviews. Cross-check distinct provider metrics without turning this single-site result into a universal adjustment.

    Gemini briefly looked like it was fixing a piece of that gap itself: Search Engine Journal reported on October 1, 2026 that it appended UTM parameters to the links it sends users to, which would have let site owners separate Gemini referrals from the Direct bucket in GA4 for the first time, a change Google's John Mueller said he'd seen too. The tag mostly survived only on desktop web, with roughly 9% pass-through from Gemini's mobile app. Search Engine Roundtable later reported, from analyst Natzir's chart, that Google stopped appending `utm_source=gemini` around September 28, after about three weeks, and a second analyst saw the same drop; Google has not commented. Desktop visits still carry a gemini.google.com referrer, but the iOS app sends none, so those visits land in GA4 as Direct and the only trace is the GeminiiOS user agent in server logs. Don't build Gemini attribution on the UTM tag; fall back to the referrer and server logs.

    ChatGPT moved the opposite direction the same month. Around October 9, 2026 it stopped appending a UTM tag to its outbound links and started opening them with `noopener,noreferrer`, which also blocks the referrer header — so visits that used to register as ChatGPT traffic now show up as Direct, per Search Engine Roundtable. Adobe's Cedric Huesler, analyzing roughly 500 pages that got at least 5% of their traffic from ChatGPT, found ChatGPT-attributed visits fell by about 31,000 a day from October 9-13 while direct traffic on the same pages rose by about 26,000 a day. Profound ties the cause to its newer Intelligent UI answer format rather than a bug: the share of ChatGPT answers carrying a tagged citation link fell from about 98% on October 8 to 66% by October 14, tracking Intelligent UI's share of answers rising from 2% to roughly 35% over the same days. If a site's GA4 Direct channel is growing unexpectedly alongside ChatGPT citations elsewhere, this is a likely cause, not a traffic-quality problem to chase.

    But don't assume every rise in Direct traffic hides AI referrals. A Growth Memo analysis of GA4 data across 12 sites from January through September 2026 found raw Direct traffic rose 140.9%, while engaged (non-bot) Direct traffic fell 1.0% in raw monthly counts (up 2.3% once normalized for days per month) — one site in the sample logged 409,000 Direct sessions of which only 198 counted as engaged, illustrating how much of the apparent growth was bot activity rather than real visits. Only 0.003% of Direct sessions carried an identifiable AI trace, and testing 632 comparisons of AI-visibility metrics against branded-search and Direct traffic across four brands turned up correlations as high as 0.72 that didn't hold up against randomized control data, suggesting the pattern was likely coincidental rather than causal. Check engaged sessions and bot filtering before reading a Direct-traffic increase as a sign of unmeasured AI traffic.

    Google's own results page may be testing the same fix a day later. Marketer Tom Pool spotted a `?st_source=ai_overview` parameter on links inside AI Overviews and AI Mode, which would let site owners separate that click traffic from ordinary search clicks the same way Gemini's UTM tag did. Treat it as a sighting, not a scheme to build reporting around yet: Google hasn't confirmed the test, and Search Engine Roundtable's own attempt to reproduce it across several browsers didn't surface the parameter.

    First-party doesn't mean link-level, though. Responding to a Reddit thread, Google's John Mueller said Search Console reports AI Overview position "as a block" — the same way it handles other search features — rather than isolating a per-link position, which he said "is hard to do." Links revealed only via a "Show More" expansion are undercounted, and an impression doesn't confirm a user actually saw the link. Mueller asked for input on what better tracking would look like but described no fix in progress. Per Search Engine Journal's follow-up, that block flattening typically reports every link in the box at position 1 regardless of whether it's shown prominently or buried behind "Show More" — so treat a Search Console AI Overview position number as directional rather than a rank tracker's link-level reading, and weigh actual clicks and conversions over the position figure for AI Overview traffic.

    Bing's first-party view is moving from single queries toward portfolio-level strategy. Topics Portfolio, a beta added to its AI Performance report, groups a site's AI Search citations into thematic clusters, each scored by Topic Size (total citations across all sites, a demand signal), Sites Cited (competing sites) and Queries — and separates a site's own "grounding queries" from "Opportunity Queries," where competitors are cited and the site is not. Microsoft says the beta appears only for sites with "sufficient data" and names no threshold, so smaller sites may see little or nothing yet.

    Common questions

    Where can I see my own AI search performance data?

    Search Console added generative-AI performance reports and Bing Webmaster Tools previews citation-share and AI-visibility insights. Verify your site in both consoles: first-party beats scraped estimates wherever both exist.

    Does Baidu offer first-party AI citation data too?

    It's rolling one out. Baidu's AI Traffic Analysis dashboard, added to its webmaster tools around September 15, 2026, tracks AI citations, impressions and clicks, but its AI CTR divides clicks by citations rather than by impressions — which can understate real link performance — and the dashboards were still empty as the phased rollout continued.

    Can I see which AI answers cite my site?

    Bing Webmaster Tools previews citation-level insights, and Microsoft Clarity now splits AI citation data into branded and non-branded views. Search Console reports impressions from AI surfaces, but no AI-feature clicks and not which pages an answer cited.

    Does Google Analytics correctly attribute AI Overview traffic?

    Alex Galinos’s single-brand study found 22.4% of tracked text-fragment events assigned to Direct. Events are not unique clicks or sessions, and text fragments are not exclusive to AI Overviews. Cross-check distinct provider metrics without turning this single-site result into a universal adjustment.

    Can I tell whether an AI crawler that scrapes my content ever sends traffic back?

    Microsoft Clarity's Bot Analytics dashboard added an AI Scrape-to-Referral Ratio card in August 2026 that ranks AI sources by crawl volume against referral visits — though Clarity cautions a high ratio doesn't by itself mean failing AI visibility, since many AI platforms strip referrer data and likely undercount real referrals.

    Does Search Console show the exact position of my link inside an AI Overview?

    No. Google's John Mueller says Search Console tracks AI Overview position as a block, the same way it handles other search features, rather than for each link individually, and links behind a "Show More" expansion are undercounted — every link in the box typically reports as position 1 regardless of where it actually sits, so weigh clicks and conversions over the position figure.

    Does Gemini tag its outgoing links so I can separate that traffic from Direct in GA4?

    Not any more, apparently. Search Engine Journal reported on October 1, 2026 that Gemini appended UTM parameters to outgoing links, but per Search Engine Roundtable, analyst Natzir's chart shows Google stopped appending utm_source=gemini around September 28 after about three weeks, and Google has not commented. Desktop visits still carry a gemini.google.com referrer; the iOS app sends none, so those visits land in GA4 as Direct.

    Why is my Direct traffic growing in GA4 while my ChatGPT-sourced clicks look flat or declining?

    ChatGPT stopped tagging its outbound links with a UTM parameter around October 9, 2026, and now opens them with noopener,noreferrer, which also blocks the referrer header — so those visits now land in GA4's Direct bucket instead of being attributable to ChatGPT. Adobe measured roughly 31,000 fewer ChatGPT-attributed visits a day and about 26,000 more direct visits a day across ~500 affected pages in the days just after the change. Profound ties the cause to ChatGPT's Intelligent UI answer format rather than a bug.

    Does Google tag AI Overview and AI Mode links so I can track their clicks separately?

    Maybe — a marketer spotted a ?st_source=ai_overview parameter on AI Overview and AI Mode links, the day after Gemini's own UTM tagging was confirmed, but Google hasn't confirmed the test and Search Engine Roundtable's own attempt to reproduce it across several browsers didn't surface the parameter.

    Is AI driving a surge in my site's Direct traffic?

    Probably not as much as it looks. A Growth Memo analysis of GA4 data across 12 sites found raw Direct traffic rose 140.9% from January to September 2026 while engaged (non-bot) Direct traffic was roughly flat — down 1.0% in raw monthly counts, up 2.3% once normalized for days per month — and only 0.003% of Direct sessions carried an identifiable AI trace; one site in the sample logged 409,000 Direct sessions of which only 198 counted as engaged. Most of the apparent growth was bot activity, not hidden AI referrals.

    Can Bing Webmaster Tools group my AI citations by topic instead of by query?

    Yes, in beta — Topics Portfolio clusters a site's AI Search citations thematically, scoring each topic by total citations across all sites, competing sites cited, and query volume, and flags "Opportunity Queries" where a competitor is cited and your site isn't. Microsoft says it only appears for sites with "sufficient data," so smaller sites may see little or nothing.

    Bing's AI Performance report shows far more citations, but my GA4 sessions from AI assistants barely moved. Why?

    The record doesn't explain any one site's gap, but it documents two places the numbers can part. A citation is not a visit, and the visits that do happen are often misfiled: ChatGPT stopped tagging its outbound links around October 9, 2026 and blocks the referrer header, so Adobe measured about 31,000 fewer ChatGPT-attributed visits a day and about 26,000 more direct visits a day across roughly 500 affected pages. Microsoft Clarity likewise cautions that many AI platforms strip referrer data and likely undercount real referrals. Cross-check GA4 against the consoles' citation and click counts rather than reading either alone.

    Verified 2026-11-07 · References: Search Console adds generative-AI reports (wire) · Bing WMT previews AI visibility insights (wire) · Clarity splits AI citations branded/non-branded (wire) · Platform properties roll out globally (wire) · Study: text-fragment events assigned to Direct (wire) · Microsoft Clarity adds an AI scrape-to-referral ratio (wire) · Google: Search Console tracks AI Overviews position only as a block (wire) · Baidu adds AI citation and CTR tracking to webmaster tools (wire) · Search Console's block flattening masks AI Overview link rank (wire) · Gemini quietly adds UTM parameters to outgoing links (wire) · Google tests a tracking parameter on AI Overview and AI Mode links (wire) · Bing Webmaster Tools adds Topics Portfolio to AI Performance (wire) · ChatGPT removes UTM tracking from its outbound links (wire) · Profound ties missing ChatGPT UTM tags to Intelligent UI (wire) · Gemini stops adding UTM parameters to cited links, tracking shows (wire) · Analysis: GA4 Direct traffic shows little hidden AI activity (wire)

  3. Track citation share, and expect it to move when models do

    Per-engine citation tracking

    Citation share — the fraction of an engine's citations your domain earns — is the discipline's emerging visibility metric. Measure it per engine, because the same content earns different shares in different engines; and treat every model swap inside an engine as a potential discontinuity in your numbers, because the model doing the citing just changed. Not every citation names the source, either: a Writesonic analysis of roughly 16 million brand appearances found engines omit the brand name anywhere from 19% (Microsoft Copilot) to 52% (Perplexity) of the time, so a citation-share number built only on named mentions will undercount actual retrieval. And citation volume alone doesn't establish category leadership: a Kevin Indig analysis of Semrush's ChatGPT citation data found the most-cited domain matched the most-mentioned brand only 20.8% of the time across 1,094 categories.

    Only 16% of brands track this at all, and the vendors selling the other 84% a dashboard often disagree on the same brand's numbers. The IAB's "Measuring Visibility in the AI Era" framework gives buyers a checklist for that gap: don't trust a citation-share number until the vendor discloses platform coverage, prompt-library construction, and query volume — the IAB sets 50 queries as the floor before a program counts as more than exploratory — and separates hallucination rate from plain factual-accuracy error.

    The coverage gap the IAB asks vendors to disclose now has a number attached. A study replaying the same ChatGPT Web queries through OpenAI's Responses API and Google Search — the two external proxies most GEO tracking tools lean on in place of real ChatGPT Web citations — found the two combined recovered only about a quarter of the exact URLs ChatGPT Web actually cited. A model combining both signals tracked hostname- and entity-level visibility far better (AUC above 0.74 for GPT-5.3 and GPT-5.5), which argues for weighting domain- and entity-level visibility over exact-URL counts in any tool built on these proxies.

    SISTRIX measured weekly domain churn-in, not URL loss: 74% for ChatGPT Search in Germany and 56% for AI Mode there. Rates and stable-core behavior differ by engine and country. SISTRIX reports 0.17 Jaccard overlap between the cited domain sets of AI Overviews and AI Mode: shared domains divided by their union. It is not the percentage of prompts on which they agree. The 1.4% core rate belongs to the News/Media domain category in SISTRIX’s AI Mode table. It is not the lifespan of every news URL across AI search.

    The model-swap risk above isn't hypothetical. Two days after Google rolled Gemini 3.8 Flash out to AI Mode, SEO practitioners found citations vanishing from top-of-funnel queries entirely, and Google's Search VP confirmed it as a bug rather than a design change. A citation tracker that only samples periodically reads an incident like that as a sudden collapse in visibility, indistinguishable at a glance from an actual ranking loss — check for an engine-side incident before concluding a citation drop reflects on your content.

    Retrieval changes can shift which sources an engine sees. Seer Interactive measured site: use in 76% of 346 matched fan-out queries for ChatGPT-5.6 Luna, against 21.6% for GPT-5.5. Ahrefs measured site: in 6.8% of 15 million fan-out queries from 66.6 million logged-out responses in August–September 2026; the studies use different prompt panels. Ahrefs also corrected its report: 14.4% of responses contained fan-out queries, not 14.4% searching the web. Do not infer that the remainder used no web retrieval. A change in targeted domain searches is worth measuring separately from citation counts.

    A citation also decays on its own measurable schedule, separate from weekly churn. Profound's Citation Decay tracker found that across 883,000 pages on seven engines over a year, the median page's citation share loses half its peak within 11 days, with the full rise-to-half-peak arc running 42 days. Durability is engine-specific — a sustained Claude citation is 2.8 times as likely as a ChatGPT one to still hold half its peak two months out — and a page cited by five or more engines is 5.5 times as likely as a single-engine page to stay above half its peak after eight weeks. Treat a citation as a position that decays on a schedule, not a one-time win: refreshing and re-promoting cited content is what keeps it above half-peak.

    The per-engine split isn't just how much gets cited — it's which pages. BrightEdge compared the top 50 cited rows per domain on ChatGPT and Google AI Overviews for holiday-shopping prompts across four US retailers: AI Overviews put 64.6% of its citations on commercial product pages versus 35.4% for ChatGPT, while ChatGPT sent 51.5% of its citation weight to newsroom and customer-support pages versus 18.1% for AI Overviews. A page built to rank for commercial intent can still miss an engine's citations if that engine is drawing from a different part of the site.

    How many prompts to track matters more than how often to repeat them. Sistrix analysed 10,826 daily measurement series across four AI models and eight countries: for a brand that truly appears in 25% of answers, one prompt queried on 20 days could read anywhere from 0% to 59%, against 16% to 35% for 20 different prompts and 21% to 29% for 100. Typical error was ±34 points for one prompt, ±9.5 for 20 and ±4.2 for 100, and Sistrix puts roughly 70 prompts at ±5 points and about 200 at ±3 — so the IAB's 50-query floor buys a rough reading, not a precise one. Repetition still earns its place for spotting change: with 20 prompts, a weekly average is accurate to ±1.8 points against ±4.8 for a single day. Sistrix also found a brand beat its rivals more often in prompt sets the brand chose than in its market-wide check, in three of four cases, so build the prompt set from the category's questions rather than the ones you hope to win. This is Sistrix's own data, and Sistrix sells AI-visibility tracking.

    The churn also runs at a daily timescale, and much of it reverses. Ahrefs asked five AI tools the same questions every day for 20 days and found 50% of day-one citations on Google AI Overviews, 58% on AI Mode and 59% on ChatGPT were gone by day two, yet roughly 75% of citations that disappeared at least once came back within the 20 days, about half within three. So a single day's check of whether a site is cited measures noise as much as visibility; read a window of days rather than one. Ahrefs also says brand mentions were more stable than website citations and may be a better KPI. This is Ahrefs' own data, and Ahrefs sells AI visibility tracking.

    The split can also flip which brand leads a category outright. In a separate BrightEdge analysis of holiday-travel queries, Hilton led hotel brands on Google AI Overviews visibility (28.3%) while Marriott led instead on ChatGPT (29.4%) — the two engines didn't just weight the same leader differently, they disagreed on who the leader was. A citation-share report that tracks only one engine can name the wrong brand as the category winner.

    Common questions

    How should I track citation share?

    Measure it per engine, because the same content earns different shares in different engines, and treat every model swap inside an engine as a potential discontinuity in your numbers.

    How do I track my brand's AI visibility?

    Citation share — the fraction of an engine's citations your domain earns — is the record's current visibility metric, measured per engine rather than as one number. Only 16% of brands track this at all, per the IAB, whose framework says not to trust a vendor's number until it discloses platform coverage, prompt-library construction and query volume.

    Why do AI visibility tools disagree about the same brand?

    The IAB's framework says not to trust a citation-share number until the vendor discloses platform coverage, prompt-library construction and query volume — with 50 queries as the floor before a program counts as more than exploratory.

    Does a high citation count mean my brand is winning the category?

    Not reliably. An analysis of Semrush's ChatGPT citation data found the most-cited domain matched the most-mentioned brand only 20.8% of the time across 1,094 categories, and Writesonic found engines omit the source brand 19% to 52% of the time depending on the engine.

    How much does AI citation drift week to week?

    SISTRIX measured weekly domain churn-in, not URL loss: 74% for ChatGPT Search in Germany and 56% for AI Mode there. Rates and stable-core behavior differ by engine and country. The stable-core finding concerns AI Mode; ChatGPT’s median prompt had no domain present in every sampled week. Domain churn and individual URL survival are different measures.

    Should I track Google AI Overviews and AI Mode as one citation number?

    SISTRIX reports 0.17 Jaccard overlap between the cited domain sets of AI Overviews and AI Mode: shared domains divided by their union. It is not the percentage of prompts on which they agree.

    Can a citation drop be a bug rather than a real visibility loss?

    Yes. When Google rolled Gemini 3.8 Flash out to AI Mode on September 2, 2026, AI Mode responses briefly stopped showing citations on some queries; Google's Search VP Robby Stein confirmed it was a bug two days later. Check for an engine-side incident before concluding a citation drop reflects your content.

    Does switching ChatGPT's default model change how many sources it cites?

    Yes — per Seer Interactive, ChatGPT-5.6 Luna, OpenAI's new default model, consulted 43% fewer sources and generated about 13% fewer inline citations than GPT-5.5 across 346 matched prompts, while using the site: operator in 76% of its fan-out queries versus 21.6% before. A much larger Ahrefs sample — 15 million fan-out queries from 66.6 million responses — measured site: operator use at just 6.8%, a gap Ahrefs attributes to sample scope and prompt mix rather than either study being wrong.

    How often does ChatGPT actually search the web?

    Ahrefs’ corrected report says 14.4% of sampled responses contained fan-out queries. This does not measure all web retrieval or establish that the remaining answers used only prior knowledge.

    How long does an AI citation last before it fades?

    Not long, typically. Per Profound's tracking of 883,000 pages across seven AI engines over a year, the median page loses half its peak citation share within 11 days, with the full rise-to-half-peak arc running 42 days — treat a citation as a decaying position to refresh, not a one-time win.

    If an AI engine stops citing my site, does the citation come back?

    Often. Ahrefs tracked five AI tools daily for 20 days: 50% to 59% of day-one citations on Google AI Overviews, AI Mode and ChatGPT were gone by day two, but roughly 75% of citations that disappeared at least once returned within the 20 days, about half within three days. A single day's check measures noise as much as visibility. This is Ahrefs' own data, and Ahrefs sells AI visibility tracking.

    Does it matter which engine cites me, for how long the citation lasts?

    Yes. Profound found a sustained page on Claude is 2.8 times as likely as one on ChatGPT to still hold half its peak citation share two months later, and a page cited by five or more engines is 5.5 times as likely as a single-engine page to stay above half its peak after eight weeks.

    Can a GEO tracking tool built on OpenAI's API accurately show what ChatGPT cites?

    Not at the URL level. A study found OpenAI's Responses API and Google Search, the two external proxies most GEO tools lean on, recovered only about a quarter of ChatGPT Web's actual citations combined — weight hostname- and entity-level visibility over exact-URL counts if a tool depends on these proxies.

    Does ChatGPT cite a retailer's own product pages for holiday shopping questions?

    Less than Google does. Per BrightEdge, only 35.4% of ChatGPT's holiday citations go to commercial product pages, versus 64.6% for Google AI Overviews on the same retailers and queries; 51.5% of ChatGPT's citations instead go to newsroom and customer-support pages, versus 18.1% for AI Overviews.

    Which hotel brand gets cited most by AI Overviews for holiday travel searches?

    Hilton leads Google AI Overviews among hotel brands at 28.3% visibility share for holiday-travel queries, while Marriott leads on ChatGPT instead, at 29.4%, per BrightEdge — the two engines disagreed on which brand led the category, not just by how much.

    How many prompts do I need to track to measure AI visibility reliably?

    Per Sistrix, 20 different prompts give a rough estimate of a brand's mention rate (about ±9.5 points), roughly 70 are needed for ±5 points and about 200 for ±3. One prompt queried for 20 days scatters by ±34 points. The finding is from Sistrix's own 10,826 measurement series, and Sistrix sells AI-visibility tracking.

    What is the optimal frequency of running AI visibility analysis?

    Per Sistrix, more different prompts beat more repeats: 140 prompts run once are accurate to ±3.6 points, against ±7.9 for 20 prompts run daily over a 7-day window. Repetition is better for spotting change — with 20 prompts a weekly average is accurate to ±1.8 points against ±4.8 for a single day. Separately, Profound's data has a typical page losing half its peak citation share within 11 days.

    Verified 2026-11-07 · References: Citation share — glossary · Ahrefs' four-surface citation rankings (wire) · Search now runs Gemini 3.5 Flash-Lite (wire) · Writesonic: AI engines often skip naming source brands (wire) · Analysis: most AI search demand has no clear ChatGPT category leader (wire) · IAB sets first measurement standard for AI-visibility tools (wire) · SISTRIX: weekly churn in cited domains (wire) · Google confirms AI Mode citation bug in Gemini 3.8 Flash (wire) · ChatGPT-5.6 Luna shifts to site: search, cites fewer sources (wire) · Profound: AI citation half-life is 11 days (wire) · Ahrefs: 14.4% of responses contained fan-out queries (wire) · Study: OpenAI's Responses API captures only a quarter of ChatGPT's citations (wire) · BrightEdge: Google AI Overviews, ChatGPT cite different kinds of holiday pages (wire) · BrightEdge: Hilton leads AI Overviews, Marriott leads ChatGPT on holiday travel (wire) · Sistrix: querying more prompts beats repeating one for measuring AI visibility (wire) · Ahrefs: AI citation stability study finds half of citations vanish within a day, most return (wire)

  4. Watch the engines' documentation, not just their announcements

    Scheduled documentation diffing

    The engines' most consequential changes ship quietly — help-page edits, crawler renames, policy language — hours or days before anyone announces them, if anyone ever does. Watch the primary documentation pages that govern your visibility and diff them on a schedule. The announcements are marketing; the documentation is policy.

    Late September into October 2026 ran five such changes in two weeks, none announced separately from the page itself: Search Console's help page split its "Web" search-type filter into text-based and multimodal (image-in-query) sub-types; Google's VideoObject structured data documentation added a recommended creator property and spelled out which interactions count toward interactionStatistic; Gemini began tagging outgoing links with UTM parameters, confirmed only after someone noticed on Reddit; Merchant Center's Loyalty Customer Match documentation named AI Mode and Gemini for the first time while expanding to 14 countries; and a new UGC Fresh Data Program for forums and social platforms surfaced only in a Search Central changelog.

    Three more in mid-to-late October fit the same pattern. Google removed the quota-increase form and the approval-process language from its Indexing API documentation on October 16, without addressing months of reported unanswered requests; a developer who tested afterward still got the standard 200-publish-requests-per-day default, so the edit may trim the paperwork rather than the quota system itself (per Search Engine Roundtable). Perplexity rewrote its crawler documentation around October 29, now telling site owners to match each bot's user agent against that bot's own published IP ranges rather than the user-agent string alone, a change the wire's page watcher caught and the change date is approximate. And the documentation can hold facts the engine's own staff do not carry in their heads: Google's crawler documentation puts the PDF fetch limit at 64MB, a figure two Google search advocates could not recall on their own podcast (per PPC Land).

    Common questions

    How do I keep up with changes to AI search?

    The engines' most consequential changes ship quietly — help-page edits, crawler renames, policy language. Watch the primary documentation pages that govern your visibility and diff them on a schedule.

    Are an engine's announcements or its documentation more reliable?

    The documentation. Google's AI Overviews help page spelled out how signed-in search history trains its generative models with no announcement at all.

    Does Google announce every documentation change that affects AI search visibility?

    No — in late September and early October 2026 alone, a Search Console multimodal-search split, a VideoObject creator property, Gemini's new UTM link tagging, an AI Mode/Gemini mention added to Loyalty Customer Match documentation, and a new UGC Fresh Data Program all shipped with no separate announcement, visible only to a page watch.

    Did Perplexity change how to allow its crawlers in a firewall?

    Its crawler documentation was edited around October 29, 2026 (change date approximate, per the wire's page watcher). The Cloudflare example now says to match each bot's user agent against that bot's own IP ranges and to use a Skip action above any custom rules that block or challenge requests, and it notes Cloudflare's Bot Fight Mode cannot be skipped by custom rules. The page still says Perplexity-User generally ignores robots.txt.

    Verified 2026-10-31 · References: Help page spells out AI Overviews training (wire exclusive) · Search Console now separates text and multimodal web search (wire) · Google adds creator property to VideoObject structured data (wire) · Gemini quietly adds UTM parameters to outgoing links (wire) · Google adds AI Mode and Gemini to Loyalty Customer Match, expands to 14 countries (wire) · Google launches UGC Fresh Data Program for forums and social platforms (wire) · Google drops quota-increase form from Indexing API docs (wire) · Perplexity updates PerplexityBot and Perplexity-User crawler documentation, including WAF rules (wire) · Google's own hosts can't name its documented 64MB PDF crawl limit (wire) · Sources & method — how this site watches

Level 3 Bets

Cheap options on where this goes, labeled as bets rather than dressed as practice. The agent-protocol layer belongs here too — this page will carry it once the wire has reported it, and not before.

  1. Publish llms.txt — as a bet on every surface except Google

    llms.txt, markdown twins

    llms.txt is a proposed standard: a machine-readable index of what your site offers, with llms-full.txt carrying the content itself for systems that want the corpus. Google has now answered its half of the open question on the record: its AI-features documentation states that machine-readable files, AI text files, and markup aren't needed to appear in Search or its AI features. An Ahrefs analysis spanning roughly 15 million data points adds evidence for the file's uncertain value even on the surfaces it's meant for: AI crawlers never read 97% of sites' llms.txt files at all. A Common Crawl analysis of 584,107 llms.txt files in its July 2026 archive suggests one reason why: 68% were produced by templates or SEO plugins rather than hand-authored — Wix alone accounts for 41% of the corpus — and only about half fully follow the spec's structure of a title, summary line, and link sections. So publish llms.txt for what it is — a near-zero-cost bet on the standalone assistants and agents beyond Google, deliberately authored rather than left to a plugin default — never as a Google lever, and never instead of the schema and crawlability every engine actually reads.

    The same bet now extends past the index file: publishers have begun serving agent-readable markdown versions of pages outright — Time went as far as selling ads inside its agent-readable markdown — and for markdown-native sites, page-level .md twins cost nearly nothing. This site serves one for every item and reference page.

    Google's John Mueller has pushed back on the premise itself: on his own test sites, "the only crawlers who claim to accept markdown are SEO tools" — not the major AI bots. His advice before building a markdown version on the assumption AI crawlers want it: log the accept header on incoming requests and check whether anything is actually asking for one.

    That markdown surface is not exempt from trust rules. Perplexity has since blocked Time's markdown ads outright, calling the sponsored FAQ content deceptive and applying a trust-score downgrade to publishers that run it — an agent-facing page answers to the same trust standard as the human-facing one.

    The spec itself got its first revision since launch on August 10: llms.txt version 2 adds rel="alternate" type="text/markdown" and rel="describedby" link relations so agents can discover a page's markdown twin and the llms.txt file governing it — formalizing, as a standard, the kind of Link-header discovery a markdown-twin setup should already expose.

    Adoption is still rare among the sites with the most to gain. LintLab crawled the 1,000 most-visited sites on Chrome's August 2026 top-sites list on October 1 and found only 66 of them, 6.6%, serving a valid llms.txt file; nearly two-thirds returned a 404, per PPC Land's report of the vendor-published crawl. Read alongside Ahrefs' finding that 97% of published files go unread, the file remains a cheap bet rather than a convention the large sites have adopted. A second crawl counts higher but points the same way: ContextFetch, a web-retrieval API vendor, asked the 10,000 most-linked-to domains in the Majestic Million for /llms.txt on October 19 and found 11.6% of the 7,462 that answered serving one. The two used different site lists and different tests (ContextFetch's report does not say it checked the file for validity), so the 6.6% and 11.6% are not in conflict; neither shows a convention.

    Common questions

    Does Google read llms.txt?

    No. Google's AI-features documentation states that machine-readable files, AI text files, and markup aren't needed to appear in Search or its AI features.

    Is llms.txt worth publishing?

    As a near-zero-cost bet on the standalone assistants and agents beyond Google — never as a Google lever, and never instead of the schema and crawlability every engine actually reads.

    Do AI crawlers actually read llms.txt files?

    Rarely. An Ahrefs analysis of roughly 15 million data points found AI crawlers never read 97% of sites' llms.txt files at all, even where the file exists.

    Are most llms.txt files hand-authored?

    No. Common Crawl's analysis of 584,107 files in its July 2026 archive found 68% were produced by templates or SEO plugins — Wix alone accounts for 41% of the corpus — with only about half fully following the spec's structure.

    Are agent-readable markdown pages held to the same trust rules as regular pages?

    Yes. Perplexity blocked Time's markdown ads as deceptive and downgraded the publisher's trust score, treating the agent-facing page the same as the visible one.

    Does llms.txt support linking to a page's markdown version?

    As of version 2 — the spec's first revision since it launched in 2024 — yes: rel="alternate" type="text/markdown" and rel="describedby" link relations let agents find a page's markdown twin and the llms.txt file that covers it.

    Do AI crawlers actually request markdown versions of pages?

    Not the major ones, per Google's John Mueller: on his own test sites, only SEO tools claimed to accept markdown. He recommends logging accept headers before building a markdown version on the assumption AI crawlers want it.

    How many major websites have an llms.txt file?

    Few. LintLab found only 66 of the 1,000 most-visited sites on Chrome's August 2026 top-sites list, 6.6%, serving a valid llms.txt file on October 1, 2026, in a vendor-published crawl.

    What share of websites have an llms.txt file?

    It depends on the list and the test. ContextFetch, a vendor, found 11.6% of the 7,462 answering sites among the 10,000 most-linked-to domains serving one on October 19, 2026; LintLab found 6.6% of the 1,000 most-visited sites serving a valid one on October 1. Different site lists and checks, so the figures are not directly comparable.

    Verified 2026-11-01 · References: Google: AI features and your website · llmstxt.org — the proposal · llms.txt — the record · Ahrefs ships a free llms.txt generator (wire) · Time sells ads inside agent-readable markdown (wire) · Perplexity blocks Time's markdown ads (wire) · llms.txt spec adds link relations for Markdown discovery (wire) · Ahrefs study: llms.txt files mostly unread by AI crawlers (wire) · Google's Mueller: AI crawlers aren't requesting markdown (wire) · Common Crawl: two-thirds of llms.txt files are templates (wire) · LintLab: sitemap errors common, llms.txt adoption still rare (wire) · ContextFetch finds bigger sites block AI crawlers more often (wire)