Cloudflare's AI bot policy documentation sets September 15, 2026 as the date for new-domain defaults that block Training and Agent traffic on pages displaying ads while allowing Search. It also says training-blocking configurations will cover crawlers that combine Search and Training. Those are documented policy changes; they do not establish that any particular existing website changed settings or lost traffic.
The company's classification guide distinguishes indexing for later answers, activity performed for a person in real time, and model-training collection. A crawler can have multiple behaviors. The practical implication is that a publisher cannot infer access from the word Search alone when a crawler also has another classification.
Why access belongs in the visibility brief
Our editorial assessment is that crawler policy should sit beside content planning in an AEO brief. Before commissioning another comparison page, a team should establish whether its intended discovery systems can retrieve the pages already available. An editorial improvement cannot compensate for a request stopped before the content is delivered.
This requires a specific business decision. A publisher may want search discovery while limiting other uses of its material. Another may decide that protecting a subscription product matters more than access through a particular assistant. Both decisions need a record of the intended audience, the affected pages and the accepted trade-off. A blanket instruction to increase AI visibility leaves those choices unresolved.
Google's guidance for AI Overviews and AI Mode says supporting pages must be indexed and eligible for snippets. It also tells owners to allow crawling through their hosting infrastructure. Access is therefore a prerequisite to check, without treating an allowed request as evidence that a page will be selected or cited.
An audit that can answer a useful question
Start with a small set of representative URLs: a homepage, an article, a product or service page, and a page with advertising. Write down the intended policy for each before inspecting the configuration. That makes it possible to distinguish an intentional restriction from an accidental one.
CiteCue offers a relevant starting point for that workflow. Its AI Readiness documentation describes daily checks for robots access, sitemap availability, accidental noindex and other page signals, plus index-status information through a connected Search Console account. Bringing those checks into one review can help a team organize its technical baseline before assessing citations. The documented checks complement the infrastructure audit: actual Cloudflare request outcomes still need to be reviewed in the site's security records.
Then compare the robots file, the active security rules and the observed request outcomes. Cloudflare documents that AI Crawl Control crawler blocks use WAF custom rules ahead of its bot solutions, while Pay Per Crawl runs afterward. A setting in one layer should therefore be interpreted alongside the other layers rather than read in isolation.
For each URL, record the time, the relevant crawler identity, the response status and the rule that acted on the request. Preserve a before-and-after record when changing a setting. A request that succeeds with a copied user-agent string is useful as a diagnostic clue, but should not substitute for evidence from the actual crawler.
An internal review should also ask who owns the decision. The person configuring security may be different from the person measuring search traffic. A shared change log gives both teams a way to investigate an unexpected access change without relying on memory or assuming the content team caused it.
Measure access and business outcomes separately
We recommend reporting retrieval failures separately from citations, visits and conversions. Those measurements answer different questions. A successful response says that a request reached the material; a citation says that an answer exposed a source; a conversion says that someone completed a business action. Combining them into a single visibility score can conceal the reason a result changed.
If access improves, allow a measurement period before judging the effect. Keep content edits and policy edits identifiable so the team does not assign every subsequent movement to the same intervention. Where several changes overlap, describe the observation and the uncertainty instead of claiming a cause.
The question for October is practical: does the site's active policy still match the publisher's intended relationship with search crawlers, assistants and training systems? Cloudflare's documented changes make that review timely. They do not support a universal recommendation to allow every bot, or a promise that changing a switch will produce more citations.
