If your website uses Cloudflare, a setting labelled “Block” deserves a careful look before anyone switches it on. Cloudflare has changed its controls for automated visitors, with a new option intended to refuse AI training while keeping search engines able to crawl your pages.
The useful distinction is between Disallow AI Training and Block. They now have different consequences for crawlers that serve more than one purpose. For a Bath shop, Somerset charity or South West professional practice, confusing the two could interfere with the search access that helps potential customers find the business.
What Cloudflare has changed
In its 15 September announcement, Cloudflare explained that some crawlers combine search indexing with other uses of website content. Blocking such a crawler entirely can therefore affect search discovery as well as AI training.
Its new Disallow AI Training setting publishes the relevant preference in robots.txt, the file through which websites give instructions to crawlers. Under this setting, the mixed-use crawlers Cloudflare designates “Accountable” can continue accessing pages for search. Other training crawlers are blocked.
Choosing Block under the Training controls is different. Cloudflare says it now includes mixed-use crawlers such as Googlebot, Bingbot and Applebot. Its “Block on pages with ads” option also includes those crawlers, for the pages concerned. These are choices with search consequences, not simply stronger versions of the same training preference.
Existing settings are being carried across
This is not an announcement that every business must urgently change its configuration. Cloudflare says it is migrating existing preferences to preserve their practical effect. Previous Training selections of Block or Block on pages with ads move to Disallow AI Training under the new definitions.
The older Block AI Bots and Managed Robots.txt features are being replaced by more detailed controls and Bot Preference Sync. That makes a brief review worthwhile, especially where different people manage the website, security and marketing. An old screenshot or remembered setting name may no longer explain what the current configuration does.
For organisations investing in SEO in Bath and the surrounding area, the immediate question is straightforward: can the search engines you rely on still reach the public pages you want customers to discover?
There is an important Microsoft qualification
Cloudflare’s “Accountable” designation includes both capabilities available now and commitments to deliver others. It should not be read as a promise that every operator implements every preference in the same way today.
In particular, Cloudflare says Microsoft is targeting early 2027 for a site-level no-training preference through robots.txt. Until that support arrives, Disallow AI Training does not automatically communicate that preference to Bing through robots.txt. Businesses with a firm requirement to restrict that use should review Microsoft’s current controls with their website specialist, including their wider consequences, rather than assume this single switch settles the question.
Google’s own documentation provides another useful distinction. Google-Extended controls specified Gemini uses of content, including training and grounding, while Google says it does not affect inclusion or ranking in Google Search. The purpose of each control matters more than an umbrella label such as “AI bots”.
Training and AI search appearances are separate decisions
Deciding whether content may help build a model is different from deciding whether it can appear in an AI-generated answer. Cloudflare discusses further summary controls as future work; the training setting should not be treated as a universal opt-out from every AI experience.
We have previously covered Google’s Search Console AI reports and controls. Those choices deserve their own review. A local service business seeking discovery may reach a different decision from a publisher whose articles are its main product. Neither should make that decision accidentally while changing a crawler setting.
A sensible check for your website team
- Confirm whether Cloudflare handles the site’s traffic. Ask your hosting provider or website manager if you are unsure. This particular dashboard change will not apply to every website.
- Record the current choices. Check the correct domain and note the Search, Training and Agent settings, along with whether Bot Preference Sync is enabled.
- Agree the intended outcome. Being discoverable in search, accepting user-directed visits and permitting model training are different choices.
- Check the whole configuration. Existing robots.txt instructions and separate security rules can still affect access. An Allow selection in one place does not override every other restriction.
- Verify any authorised change. Keep the previous settings, record when the change was made and have the website team check crawler access and relevant search diagnostics afterwards.
There is no need to switch hosting providers or rewrite good website content because of this announcement. If you already use Cloudflare, the useful next step is a short, documented conversation with whoever maintains it. Make sure the settings express your business’s actual preference, and check the result before drawing conclusions from traffic changes.
