# Google: robots.txt limits what gets crawled, not what AI does with the data

Published: 2026-10-21T21:35:22.973Z · Source: PPC Land (https://ppc.land/google-says-robots-txt-limits-ai-crawling-not-what-bots-do-with-data/)
Modified: 2026-11-05T19:49:43.000Z
Second source: Google Search Central (YouTube) (https://www.youtube.com/watch?v=-jHONLQgE-s)
Source date: 2026-10-21
Flags: single-source — confirmation pending
Entities: google

> **Correction, 2026-10-31:** The wire covered the same October 21 Google Search Central video twice. The October 23 copy now redirects here; its additional details and Search Engine Journal attribution have been incorporated. This is a historical account, not a statement of current training-control standards.

Robots.txt is "an access preference protocol" governed by RFC 9309 — it controls whether a page may be fetched, and "nothing in it expresses a purpose limitation," Google Search Relations' Gary Illyes and Cherry Prommawin said in a Google Search Central video published October 21, per PPC Land. Disallowing a crawler "has nothing to do with" what happens to data already collected from a page it was allowed to crawl, they said. The pair also confirmed robots.txt silently drops any line outside RFC 9309's syntax — including emoji in a user-agent token, since the field allows only letters, underscores and hyphens — with no warning or error shown to the site owner.

The video did not address Google-Extended, the separate token Google says site owners can use to opt out of having crawled content used for AI training without affecting Search inclusion — leaving open how that purpose-specific control fits inside a protocol its own hosts describe as access-only.

A second account of the same video, Search Engine Journal (https://www.searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training/593957/), reported the need to target a specific crawler token, Google-Extended's separate training and grounding controls without affecting Search inclusion, and an IETF Content-Usage draft that had not yet reached steering-body review at the time of reporting. These details were also carried in the wire's October 23 duplicate.

Why it matters: Robots.txt is a search-era protocol being read as an AI-training opt-out, and Google's own description of what it legally does under RFC 9309 — access only, no purpose limitation — doesn't match that use without a separate signal like Google-Extended.

## What this answers

**Does blocking an AI crawler in robots.txt stop it from using already-crawled data?**

No — per Google Search Relations, robots.txt is an access protocol that only controls whether a page gets crawled, not what happens to data already collected from it.

**Can you use emoji in a robots.txt user-agent line?**

No — per Google, user-agent tokens only allow letters, underscores and hyphens, so emoji and other custom characters are silently dropped rather than flagged as errors.


Canonical: https://anythingengineoptimization.com/item/2026-10-21-google-robots-txt-limits-what-gets-crawled-not-what-ai-does-with-the/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/
