The Lead
TechCrunch's robots.txt no longer disallows ChatGPT-User or Google-Extended
The wire’s crawler access panel found eight changes between its November 8 and November 15 captures. In TechCrunch’s robots.txt, the site-wide Disallow: / rules for ChatGPT-User (OpenAI’s agent for fetches a user requests) and Google-Extended (Google’s token for opting out of Gemini use) are gone. The wire’s archived copy of the file from November 8 carries both; the November 15 copy carries neither, and the file read at curation time matches the November 15 copy. The panel recorded both tokens as disallowed in every capture back to September 12, so the change landed between November 8 and 15.
Comparing the two archived files shows a third removal the panel does not track: a rule for Claude-Web, an older Anthropic token. TechCrunch’s file still disallows GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Applebot-Extended, cohere-ai, Diffbot, FacebookBot and omgili across the whole site. It has never named OAI-SearchBot, Claude-SearchBot or PerplexityBot in either copy. The file does not say why the rules were removed, and the wire has no statement from TechCrunch.
The other five changes are homepage status codes, each a single probe with no retries. El País returned a 200, up from a 403, to a browser-style request, GPTBot and ClaudeBot, while PerplexityBot moved from a 200 to a 403. The New York Times returned a 200 to the browser-style request, up from a 403. WikiHow returned a 403 to GPTBot, up from a 200. A flip that does not hold on the next capture counts as transient rather than a change.
Via Crawler access panel ↗ · Nov 15 · Tools & Vendors, AEO