# Cloudflare — the record The infrastructure layer where crawler access is actually enforced. Its defaults reach a large share of the web, so its policy changes are the industry's, whether or not anyone voted. Cloudflare is where crawler policy stops being an opinion and becomes a setting. Its data underpins much of this beat — automated requests crossed 57.5% of web traffic in June 2026, with 52% of AI crawler requests serving model training against 2.6% for live queries — and its products increasingly set the terms: Pay Per Crawl evolved into usage-based payments for AI search, and an Agent Readiness dashboard now scores domains on crawler rules, machine-readable content and agent-callable APIs. Because its defaults reach a large share of the web, a change to them is an industry event; IAB Australia built its four-verdict crawler matrix explicitly around Cloudflare's September 15 changes. The record is thin on one thing: what those defaults do to sites that are already customers, which Cloudflare's public documentation does not clearly say. Also known as: https://www.cloudflare.com/ · https://en.wikipedia.org/wiki/Cloudflare · https://www.wikidata.org/wiki/Q2140714 - 2026-09-16: El País blocks GPTBot, ClaudeBot and PerplexityBot (https://anythingengineoptimization.com/item/2026-09-16-el-pa-s-blocks-gptbot-claudebot-and-perplexitybot/) The wire's crawler access panel caught El País's homepage flip from a 200 to a 403 for GPTBot, ClaudeBot and PerplexityBot alike between the September 13… - 2026-09-13: People Inc. sites and The Atlantic block GPTBot, still bill Claude, Perplexity (https://anythingengineoptimization.com/item/2026-09-13-people-inc-sites-and-the-atlantic-block-gptbot-still-bill-claude/) The wire's crawler access panel — 55 publisher domains probed daily as GPTBot, ClaudeBot, PerplexityBot and a browser — caught three People Inc.… - 2026-08-30: ACP, AP2 and x402 are three layers of one stack, not three rivals (https://anythingengineoptimization.com/item/2026-08-30-acp-ap2-and-x402-are-three-layers-of-one-stack-not-three-rivals/) An agent arriving at a site raises three questions in order: who are you, may you read this, and may you buy. - 2026-08-30: Cloudflare's September 15 default blocks Googlebot on ad-bearing pages (https://anythingengineoptimization.com/item/2026-08-30-cloudflare-s-september-15-default-blocks-googlebot-on-ad-bearing-pages/) On September 15 Cloudflare sets new defaults for the three crawler classifications it introduced in July. - 2026-08-27: No major AI crawler publicly meets all four of Cloudflare's conditions (https://anythingengineoptimization.com/item/2026-08-27-no-major-ai-crawler-publicly-meets-all-four-of-cloudflare-s-conditions/) When Cloudflare launched Bot Preference Sync on August 21 it set four conditions a crawler doing both search and training must meet to keep reaching sites… - 2026-08-23: Cloudflare syncs robots.txt, sets four conditions for crawlers that also train (https://anythingengineoptimization.com/item/2026-08-23-cloudflare-rewrites-robots-txt-gates-access-on-disclosure/) Cloudflare on August 21 launched Bot Preference Sync, which keeps a site's robots.txt in step with the AI-crawler policy set in its dashboard. - 2026-08-08: Donor-database site blocks Amazon's AI crawler after logging 117,000 daily requests (https://anythingengineoptimization.com/item/2026-08-08-donor-database-site-blocks-amazon-s-ai-crawler-after-logging-117-000/) PatronView, a database of U.S. museum and cultural-institution donor records, blocked Amazon's Amzn-SearchBot crawler in early August after founder Nick… - 2026-08-06: Cloudflare launches dashboard tools to measure agent access and AI citations (https://anythingengineoptimization.com/item/2026-08-06-cloudflare-launches-dashboard-tools-to-measure-agent-access-and-ai/) Cloudflare added two tools to its dashboard on August 6: Agent Readiness, a technical audit of how well AI crawlers and agents can discover and read a… - 2026-08-01: Bipartisan bill would force AI crawlers to identify themselves (https://anythingengineoptimization.com/item/2026-08-01-bipartisan-bill-would-force-ai-crawlers-to-identify-themselves/) Reps. Laurel Lee (R-FL), Valerie Foushee (D-NC), and Gus Bilirakis (R-FL) introduced the Stealth Bot Prohibition Act, H.R. - 2026-08-01: Cloudflare evolves Pay Per Crawl into usage-based AI search payments (https://anythingengineoptimization.com/item/2026-08-01-cloudflare-evolves-pay-per-crawl-into-usage-based-ai-search-payments/) Cloudflare announced two initiatives on July 1, 2026 to reshape AI-era crawling economics. - 2026-08-01: IAB Australia sorts AI crawlers into four access verdicts (https://anythingengineoptimization.com/item/2026-08-01-iab-australia-sorts-ai-crawlers-into-four-access-verdicts/) IAB Australia published version 1.0 of its Bots and Crawler Guidance and Decision Matrix on July 22, 2026, unveiling it at the Discovery: AI and Search… - 2026-08-01: Time starts selling ads inside its AI-agent-readable markdown pages (https://anythingengineoptimization.com/item/2026-08-01-time-starts-selling-ads-inside-its-ai-agent-readable-markdown-pages/) Time has converted its web pages into markdown — stripped-down, text-only copies without design or images — to make its content easier for AI crawlers and… Canonical: https://anythingengineoptimization.com/entity/cloudflare/ From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/entity/