Anything Engine Optimization The rolling record of the AI-search industry.

Anything Engine Optimization — The rolling record of the AI-search industry.

The news wire of answer engine optimization — also called AI engine optimization: the work of being retrieved and cited by ChatGPT, Perplexity, Gemini and Google's AI Overviews, and the industry forming around it.

The Lead

single-probe — panel capture

TechCrunch's robots.txt no longer disallows ChatGPT-User or Google-Extended

The wire’s crawler access panel found eight changes between its November 8 and November 15 captures. In TechCrunch’s robots.txt, the site-wide Disallow: / rules for ChatGPT-User (OpenAI’s agent for fetches a user requests) and Google-Extended (Google’s token for opting out of Gemini use) are gone. The wire’s archived copy of the file from November 8 carries both; the November 15 copy carries neither, and the file read at curation time matches the November 15 copy. The panel recorded both tokens as disallowed in every capture back to September 12, so the change landed between November 8 and 15.

Comparing the two archived files shows a third removal the panel does not track: a rule for Claude-Web, an older Anthropic token. TechCrunch’s file still disallows GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Applebot-Extended, cohere-ai, Diffbot, FacebookBot and omgili across the whole site. It has never named OAI-SearchBot, Claude-SearchBot or PerplexityBot in either copy. The file does not say why the rules were removed, and the wire has no statement from TechCrunch.

The other five changes are homepage status codes, each a single probe with no retries. El País returned a 200, up from a 403, to a browser-style request, GPTBot and ClaudeBot, while PerplexityBot moved from a 200 to a 403. The New York Times returned a 200 to the browser-style request, up from a 403. WikiHow returned a 403 to GPTBot, up from a 200. A flip that does not hold on the next capture counts as transient rather than a change.

Via Crawler access panel ↗ · Nov 15 · Tools & Vendors, AEO

Research you can check

Our own measurements and a dated record of the evidence behind AI search.

All original reporting →

Friday, November 13, 2026

Thursday, November 12, 2026

Wednesday, November 11, 2026

Tuesday, November 10, 2026

Monday, November 9, 2026

Sunday, November 8, 2026

Friday, November 6, 2026

Thursday, November 5, 2026

Wednesday, November 4, 2026

Tuesday, November 3, 2026

Monday, November 2, 2026

All 520 items in the archive