Anything Engine Optimization The rolling record of the AI-search industry.
A single wall lantern emitting a warm glow against a dark green wall, creating a cozy atmosphere.
Photo by Nguyễn Khánh on Pexels
single-probe — panel capture
Crawler access panel · Nov 15 Tools & VendorsAEO

TechCrunch's robots.txt no longer disallows ChatGPT-User or Google-Extended

What this answers

Does TechCrunch still block ChatGPT-User and Google-Extended?
Not in its robots.txt. The file the wire archived on November 8 disallowed both across the whole site; the file captured on November 15 has no rule for either token, so both fall under the general rules that block only a few paths such as /wp-admin/ and /search/.
Which AI crawlers does TechCrunch's robots.txt still disallow?
As of the November 15 capture it still disallows GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Applebot-Extended, cohere-ai, Diffbot, FacebookBot and omgili across the whole site.

The wire’s crawler access panel found eight changes between its November 8 and November 15 captures. In TechCrunch’s robots.txt, the site-wide Disallow: / rules for ChatGPT-User (OpenAI’s agent for fetches a user requests) and Google-Extended (Google’s token for opting out of Gemini use) are gone. The wire’s archived copy of the file from November 8 carries both; the November 15 copy carries neither, and the file read at curation time matches the November 15 copy. The panel recorded both tokens as disallowed in every capture back to September 12, so the change landed between November 8 and 15.

Comparing the two archived files shows a third removal the panel does not track: a rule for Claude-Web, an older Anthropic token. TechCrunch’s file still disallows GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Applebot-Extended, cohere-ai, Diffbot, FacebookBot and omgili across the whole site. It has never named OAI-SearchBot, Claude-SearchBot or PerplexityBot in either copy. The file does not say why the rules were removed, and the wire has no statement from TechCrunch.

The other five changes are homepage status codes, each a single probe with no retries. El País returned a 200, up from a 403, to a browser-style request, GPTBot and ClaudeBot, while PerplexityBot moved from a 200 to a 403. The New York Times returned a 200 to the browser-style request, up from a 403. WikiHow returned a 403 to GPTBot, up from a 200. A flip that does not hold on the next capture counts as transient rather than a change.

Why it matters: A publisher that had blocked OpenAI's user-initiated fetcher and Google's Gemini opt-out token for at least two months has dropped both while keeping its training-crawler blocks, so its stated policy now separates those tokens from GPTBot and ClaudeBot.

The record: Google

Via Crawler access panel ↗

Posted to the wire November 15, 2026. Edited by Joe Balewski.