Anything Engine Optimization The rolling record of the AI-search industry.
A single symbolic glyph on a deep color field.
Illustration: AEO Wire
single-source — confirmation pending
PPC Land · Oct 21 Google SearchSEOAEO

Google: robots.txt limits what gets crawled, not what AI does with the data

Corrections

  • 2026-10-31: The wire covered the same October 21 Google Search Central video twice. The October 23 copy now redirects here; its additional details and Search Engine Journal attribution have been incorporated. This is a historical account, not a statement of current training-control standards.

What this answers

Does blocking an AI crawler in robots.txt stop it from using already-crawled data?
No — per Google Search Relations, robots.txt is an access protocol that only controls whether a page gets crawled, not what happens to data already collected from it.
Can you use emoji in a robots.txt user-agent line?
No — per Google, user-agent tokens only allow letters, underscores and hyphens, so emoji and other custom characters are silently dropped rather than flagged as errors.

Robots.txt is “an access preference protocol” governed by RFC 9309 — it controls whether a page may be fetched, and “nothing in it expresses a purpose limitation,” Google Search Relations’ Gary Illyes and Cherry Prommawin said in a Google Search Central video published October 21, per PPC Land. Disallowing a crawler “has nothing to do with” what happens to data already collected from a page it was allowed to crawl, they said. The pair also confirmed robots.txt silently drops any line outside RFC 9309’s syntax — including emoji in a user-agent token, since the field allows only letters, underscores and hyphens — with no warning or error shown to the site owner.

The video did not address Google-Extended, the separate token Google says site owners can use to opt out of having crawled content used for AI training without affecting Search inclusion — leaving open how that purpose-specific control fits inside a protocol its own hosts describe as access-only.

A second account of the same video, Search Engine Journal, reported the need to target a specific crawler token, Google-Extended’s separate training and grounding controls without affecting Search inclusion, and an IETF Content-Usage draft that had not yet reached steering-body review at the time of reporting. These details were also carried in the wire’s October 23 duplicate.

Why it matters: Robots.txt is a search-era protocol being read as an AI-training opt-out, and Google's own description of what it legally does under RFC 9309 — access only, no purpose limitation — doesn't match that use without a separate signal like Google-Extended.

The record: Google

Glossary: Grounding

Via PPC Land ↗ · Google Search Central (YouTube) ↗

Posted to the wire October 21, 2026. Updated November 5, 2026. Edited by Joe Balewski.