BBC's robots.txt now disallows Claude-SearchBot, Anthropic's search crawler
The wire's crawler access panel found seven changes between its November 1 and November 8 captures.
Via Crawler access panel ↗ · Nov 8 · · Tools & Vendors, AEO
The record
Anthropic is the maker of Claude. Its release notes and crawler policies matter to anyone deciding which AI systems may read their site.
Four items, and Anthropic appears in each as the subject of someone else's measurement rather than as a shipper of news. YouGov BrandIndex put Gen Z consideration of Claude at 28.1% in Q2 2026, up from 14.2% in Q1 — the largest single move in the consideration figures the wire has logged. Claude was one of three models in geoSurge's test of 66 U.S. buyer prompts, and one of three assistants named in an Optimove survey where 72% of U.S. holiday shoppers said they consult AI tools for gift ideas. On the access side, PatronView says it blocked Claude-SearchBot in June 2026 after measuring a lopsided ratio of crawls to human-referred visitors. The record carries no Anthropic product or crawler-policy announcement of its own.
45 items on the wire, newest first. Elsewhere: official · Wikipedia · Wikidata
The wire's crawler access panel found seven changes between its November 1 and November 8 captures.
Via Crawler access panel ↗ · Nov 8 · · Tools & Vendors, AEO
Anthropic licensed content and data from the Associated Press shortly before the U.S. midterm elections, its first licensing deal with a news publisher…
Via Adweek ↗ · Nov 5 · · Engines, Law & Money
The New York Times sued Brave on Nov 2 in federal court in Manhattan, alleging the browser and search company scraped its copyrighted content and…
Via TechCrunch AI ↗ · Nov 2 · · Law & Money, Engines
SearchDock, an SEO platform vendor, requested the homepages of the Tranco top 1,000 domains on October 20, 2026 as a browser and as GPTBot, ClaudeBot…
Via SearchDock ↗ · Oct 20 · · Research, AEO
Corrected — read the correction
OpenAI, Anthropic, Google and Microsoft said on October 30 that they will honor a single robots.txt policy token, usage_ai-training, so a site can opt out…
Via IETF Datatracker (draft-kelley-aipref-training-preference-signal-00) ↗ · Oct 30 · · Engines, AEO, Research, Google Search
ContextFetch, a web-retrieval API vendor, fetched robots.txt as "ContextFetchBot" from the 10,000 most-linked-to domains in the Majestic Million on…
Via ContextFetch ↗ · Oct 19 · · Research, AEO
Vercatus, an AI-visibility platform, probed the home page and one deep page of 64 well-known sites across eight niches — eight sites each — with GPTBot…
Via Vercatus ↗ · Sep 18 · · Research, AEO
The wire's crawler access panel caught shifts at three publishers between its October 18 and October 25 captures.
Via Crawler access panel ↗ · Oct 25 · · Tools & Vendors, AEO
Judge Eumi Lee heard cross-motions for summary judgment Oct. 21 in Concord Music Group v. Anthropic, the music publishers' suit over Claude's training on…
Via Courthouse News Service ↗ · Oct 21 · · Law & Money
The Korean Publishers Society, a trade group representing roughly 500 trade-book publishers, issued a public appeal to South Korea's government, National…
Via Seoul Economic Daily ↗ · Oct 20 · · Law & Money
Corrected — read the correction
Which AI search bot a site lets in depends, on AEO Wire's crawler panel, on whether the site has a licensing deal with OpenAI.
Via AEO Wire crawler panel ↗ · Oct 19 · · AEO, Engines, Law & Money
OpenAEO, an AI-visibility vendor, measured 131 publisher, SaaS, ecommerce and agency domains for how they treat AI crawlers by name, narrowing to 12 sites…
Profound ran 711 prompts through Anthropic's API 20 times each on Claude Sonnet 5 and Sonnet 5.5 with web search enabled — 28,440 answers total — and…
Via Profound research blog ↗ · Oct 13 · · Research, AEO
Search Engine Journal contributor Reza Moaiandin tested Claude Sonnet 5 on six shopping and travel scenarios across five languages — English, Arabic…
Via Search Engine Journal ↗ · Oct 13 · · Research, AEO, GEO
Anthropic's usage policy, effective November 12, 2026, now prohibits using Claude to "manipulate the sources from which search engines or AI systems draw…
Via Anthropic ↗ · Oct 9 · · Engines, AEO
Profound tracked 883,000 pages across seven AI engines — ChatGPT, Perplexity, Claude, Google AI Overviews, Google AI Mode, Gemini, and Microsoft Copilot…
Via Profound research blog ↗ · Sep 30 · · Research, AEO
Discovered Labs co-founders Ben Moore and Liam Dunne, in an arXiv preprint, analyzed roughly 2 million citations from ChatGPT, Claude, Google AI and…
Via arXiv AI-search citations ↗ · Sep 28 · · Research, AEO, GEO
The wire's crawler access panel caught four verified shifts between its September 20 and 27 captures.
Via Crawler access panel ↗ · Sep 27 · · Tools & Vendors, AEO
Adobe has put Photoshop, Lightroom, Adobe Express and Firefly tools directly into Gemini on every subscription plan, in a global rollout that began…
Via PPC Land ↗ · Sep 25 · · Engines, Tools & Vendors
ChatGPT's share of AI assistant prompt volume fell from 70% in January 2026 to 50% by June, while Gemini nearly doubled from 17% to 30% and Claude climbed…
Via Digiday Media ↗ · Sep 22 · · Engines, Research
The wire's crawler access panel caught four shifts between its September 16 and 20 captures.
Via Crawler access panel ↗ · Sep 20 · · Tools & Vendors, AEO
Anthropic has moved to dismiss copyright claims brought by authors who opted out of the Bartz class-action settlement in Shakespeare v. Anthropic, arguing…
Via Chat GPT Is Eating the World ↗ · Sep 19 · · Law & Money
The wire's crawler access panel caught El País's homepage flip from a 200 to a 403 for GPTBot, ClaudeBot and PerplexityBot alike between the September 13…
Via Crawler access panel ↗ · Sep 16 · · Tools & Vendors, AEO
Authors and publishers who declined the Bartz v. Anthropic book-piracy settlement will get their own trial — but not until 2028.
Via Chat GPT Is Eating the World ↗ · Sep 15 · · Law & Money
Summary judgment briefing — the filings asking a judge to rule without a trial — is complete in Concord Music v. Anthropic I, with a hearing set for Oct.…
Via Chat GPT Is Eating the World ↗ · Sep 15 · · Law & Money
Anthropic filed its opposition to Daniel Benjamin Gilbert's motion to dismiss the company's fair-use counterclaim in his pro se copyright suit over his…
Via Chat GPT Is Eating the World ↗ · Sep 11 · · Law & Money
Profound tested ChatGPT, Gemini, and Claude with personas built from combinations of income, age, gender, and occupation, asking each about credit cards…
Via Profound research blog ↗ · Sep 10 · · Research, AEO
Class counsel in the Bartz v. Anthropic book-piracy settlement says the first payout to class members will be $2,203.56 per work, sent on or before Nov.…
Via Chat GPT Is Eating the World ↗ · Sep 2 · · Law & Money
GreyNoise reported on August 28 that scanners forging AI-crawler names probed the open web for credential files from 824 addresses across 795 separate…
Via GreyNoise ↗ · Aug 30 · · Engines, Tools & Vendors, AEO
Sony Music Publishing sued Anthropic in the Northern District of California on August 28, alleging direct copyright infringement via torrenting…
Via Chat GPT Is Eating the World ↗ · Aug 29 · · Law & Money
Daniel Benjamin Gilbert, proceeding pro se in his copyright suit against Anthropic after opting out of the Bartz class settlement, has filed a covenant…
Via Chat GPT Is Eating the World ↗ · Aug 28 · · Law & Money
Judge Rita Lin granted Anthropic summary judgment on its First Amendment challenge to the U.S. Department of War's designation of the company as a "supply…
Via Chat GPT Is Eating the World ↗ · Aug 28 · · Law & Money
The Bartz v. Anthropic book-piracy class settlement became effective Aug. 20, 2026, the day after the deadline to appeal the settlement itself passed, per…
Via Chat GPT Is Eating the World ↗ · Aug 27 · · Law & Money
Seer Interactive tracked 1,562 branded prompts about its own company across six AI platforms — ChatGPT, Claude, Gemini, Perplexity, Google AI Overviews…
Via Seer Interactive ↗ · Aug 27 · · Research, AEO
Testing GPT-4o-mini, Claude Sonnet and Gemini on skincare product recommendations, researchers Xi Chu and Yupeng Hou found that when competing products…
Via arXiv GEO ↗ · Jun 16 · · Research, GEO, AEO
Profound's analysis of 24,135 responses collected July 13–23 found Claude searched the web in over 93% of responses versus just 13% for Claude Code, and…
Via Profound research blog ↗ · Aug 24 · · Research, AEO, Tools & Vendors
Anthropic has filed an amended Answer to the copyright complaint from Daniel Gilbert, author of "Hacking World of Warcraft," who opted out of the Bartz…
Via Chat GPT Is Eating the World ↗ · Aug 20 · · Law & Money
A First Amended Complaint in Shakespeare v. Anthropic adds several new plaintiffs and, for the first time in the case, names CEO Dario Amodei directly…
Via Chat GPT Is Eating the World ↗ · Aug 19 · · Law & Money
Profound analyzed more than 158,000 AI-generated brand claims through its FactCheck tool and found inaccuracy rates of 4.3-6.8% across engines, with…
Via Profound research blog ↗ · Aug 17 · · Research, AEO
The Publishers' Coordination Counsel has filed a notice of appeal to the Ninth Circuit challenging the denial of its attorneys' fee award in the Bartz v.…
Via Chat GPT Is Eating the World ↗ · Aug 19 · · Law & Money
Anthropic began embedding invisible watermarks in Claude-generated text and signed provenance metadata in generated files on August 2, 2026, to comply…
Via Chat GPT Is Eating the World ↗ · Aug 11 · · Engines, Law & Money
PatronView, a database of U.S. museum and cultural-institution donor records, blocked Amazon's Amzn-SearchBot crawler in early August after founder Nick…
Via PPC Land ↗ · Aug 8 · · Tools & Vendors, AEO
Gen Z consideration of Claude climbed from 14.2% to 28.1% and of OpenAI from 10.1% to 22.1% between Q1 and Q2 2026, per YouGov BrandIndex data reported by…
Via Search Engine Journal ↗ · Aug 7 · · Research, AEO, GEO
Optimove's survey of 648 U.S. consumers found 72% regularly or occasionally use AI tools such as ChatGPT, Claude or Gemini for gift ideas and shopping…
Via PPC Land ↗ · Aug 5 · · Research, AEO
A geoSurge study found AI models search for familiar brands 3.2 times more often than unfamiliar ones — 55.7% versus 17.4% of the time — per Search Engine…
Via Search Engine Land ↗ · Jul 30 · · Research, AEO, GEO
Glossary: AI crawlers