# TechCrunch's robots.txt no longer disallows ChatGPT-User or Google-Extended

Published: 2026-11-15T22:47:15.478Z · Source: Crawler access panel (https://anythingengineoptimization.com/crawl-ledger/)
Source date: 2026-11-15
Flags: single-probe — panel capture
Entities: google

The wire's crawler access panel found eight changes between its November 8 and November 15 captures. In TechCrunch's robots.txt, the site-wide Disallow: / rules for ChatGPT-User (OpenAI's agent for fetches a user requests) and Google-Extended (Google's token for opting out of Gemini use) are gone. The wire's archived copy of the file from November 8 carries both; the November 15 copy carries neither, and the file read at curation time matches the November 15 copy. The panel recorded both tokens as disallowed in every capture back to September 12, so the change landed between November 8 and 15.

Comparing the two archived files shows a third removal the panel does not track: a rule for Claude-Web, an older Anthropic token. TechCrunch's file still disallows GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Applebot-Extended, cohere-ai, Diffbot, FacebookBot and omgili across the whole site. It has never named OAI-SearchBot, Claude-SearchBot or PerplexityBot in either copy. The file does not say why the rules were removed, and the wire has no statement from TechCrunch.

The other five changes are homepage status codes, each a single probe with no retries. El País returned a 200, up from a 403, to a browser-style request, GPTBot and ClaudeBot, while PerplexityBot moved from a 200 to a 403. The New York Times returned a 200 to the browser-style request, up from a 403. WikiHow returned a 403 to GPTBot, up from a 200. A flip that does not hold on the next capture counts as transient rather than a change.

Why it matters: A publisher that had blocked OpenAI's user-initiated fetcher and Google's Gemini opt-out token for at least two months has dropped both while keeping its training-crawler blocks, so its stated policy now separates those tokens from GPTBot and ClaudeBot.

## What this answers

**Does TechCrunch still block ChatGPT-User and Google-Extended?**

Not in its robots.txt. The file the wire archived on November 8 disallowed both across the whole site; the file captured on November 15 has no rule for either token, so both fall under the general rules that block only a few paths such as /wp-admin/ and /search/.

**Which AI crawlers does TechCrunch's robots.txt still disallow?**

As of the November 15 capture it still disallows GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Applebot-Extended, cohere-ai, Diffbot, FacebookBot and omgili across the whole site.


Canonical: https://anythingengineoptimization.com/item/2026-11-15-techcrunch-s-robots-txt-no-longer-disallows-chatgpt-user-or-google/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/
