Anything Engine Optimization The rolling record of the AI-search industry.
A striking visual of light beams illuminating particles in a dark setting, creating an abstract effect.
Photo by Ananya Jain on Pexels
Crawl Ledger: crawler roster · Oct 9 Tools & VendorsAEO

Common Crawl, Mistral, Cohere bots visit, then vanish from logs

What this answers

Does Common Crawl's bot crawl a site every week once it shows up?
Not necessarily — on the wire's own site, Common Crawl's CCBot logged requests in the week of September 21, 2026, then zero the following week, per the site's own Cloudflare data.
What crawlers besides the major AI chatbots hit AI-search sites?
The wire's Cloudflare logs caught dozens of unregistered scanners in the week of September 21, 2026, led by ShapBot at 872 requests, alongside SEO tools like AhrefsBot and SemrushBot.

The wire’s own Cloudflare logs recorded four registry crawlers for the first time in the week of September 21, 2026: Common Crawl’s CCBot, Mistral’s MistralAI-User, Cohere’s cohere-ai and Huawei’s PetalBot, alongside dozens of unregistered scanners led by ShapBot’s 872 requests. One week later, CCBot, MistralAI-User and cohere-ai logged zero requests, and Google-Extended — present in the ledger since early September — also fell to zero; only PetalBot’s traffic held. The same week’s data recorded 3,519 AI-crawler requests against zero AI-referral pageloads, down from 10 the week before, on the wire’s own Crawl Ledger.

Why it matters: It is a first-party reminder that a registry crawler's appearance in a given week can be a one-off rather than a standing presence, which weakens any single week's crawl count as a signal of real training or answer use.

The record: CloudflareGoogle

Via Crawl Ledger: crawler roster ↗

Posted to the wire October 9, 2026. Edited by Joe Balewski.