On September 15, something changed about your website that had nothing to do with your website. Cloudflare’s new crawler controls went live, and with them a quiet shift in power: whether AI systems can read your pages is now decided in a dashboard most business owners have never opened.
Here is what happened. Cloudflare had planned something blunt, as PPC Land reported: sites that refused AI training would also have lost Googlebot, Bingbot and Applebot, search indexing included. Owners would have had to choose between protecting their content and being found at all. Cloudflare backed off and shipped a separate “Disallow AI Training” setting instead. Reasonable outcome. But the new defaults still bite. New domains and free-tier customers now get training and agent crawlers blocked by default on pages that display ads, with search crawlers still allowed.
Cloudflare’s own numbers show how uneven this already is. Roughly 17 percent of sites on its network use some mechanism to block AI training. Fewer than 1 percent block search crawlers. A lot of businesses are sitting in that 17 percent without having made any decision at all.
Blocked is bad. Unreadable is worse.
Access is only half the problem. A crawler can reach your homepage and still fail to find a single service page.
A 41-day crawler test published by Search Engine Land ran on a 2,400 page site. Researchers split 1,062 pages into two groups: 748 linked with plain HTML, 293 linked with JavaScript. Sitemaps and breadcrumbs were switched off so navigation was the only variable. GPTBot and ClaudeBot found 100 percent of the HTML-linked pages. They found zero of the JavaScript-linked ones. Not a small number. Zero. GoogleOther managed 66 percent of the HTML pages and 48 percent of the JavaScript ones. When those links were converted to plain HTML, GPTBot came back within about 48 hours and picked up roughly 250 pages it had never seen.
Think about where that lands for a service business. Your service pages behind an animated mega menu. Your city pages loaded by a “view more” button. Your pricing inside a tabbed widget. All of it looks perfect to you, because your browser runs JavaScript. GPTBot does not. To the assistant answering your buyer’s question, those pages do not exist.
What actually decides who gets named
When someone asks an AI assistant who to hire, the model is not ranking ten blue links. It is building one answer out of the few businesses it can reach, read, understand and verify. That comes down to a handful of things working together: machine access at the server and CDN level, a machine-readable site where links are real links and text is real text (not words baked into images), entity clarity so a model can tell exactly who you are and what you do, structured signals that state your services and service area in a format nothing has to guess at, outside corroboration that backs up your claims, and reviews with enough volume and recency to count as evidence.
None of that is a plugin you switch on. The signals depend on each other, which is the part most owners miss. Perfect schema on pages a crawler cannot reach is worth nothing. Clean crawl access to thin, vague pages is worth nearly as little. This work sits closer to engineering than to content marketing, which is why our process audits the whole chain before anyone touches a page.
The uncomfortable part: the competitors pulling your leads right now are not necessarily better at the actual job. They are more legible. The model recommends what it can read and confirm. You can see what that shift looks like in practice on our results page, and the usual questions about scope, timing and what we need from you are answered on our FAQ.
The cost of waiting is not flat
Old search was forgiving about delay. You slipped a few spots, you climbed back. AI recommendation is less forgiving, because citation compounds. Every time a model names a business as the answer, that business picks up more of the signals that make it the answer next time. Six months of absence is not six months of lost leads. It is six months of a competitor hardening into the default.
And the ground keeps moving without telling you. A CDN default flipped on September 15. A theme update swaps your navigation for a JavaScript component. A host turns on bot protection and starts serving 403s to crawlers your robots.txt says are welcome. None of it shows up in your traffic report for weeks, and by then the answer has already been given to someone else.
Related reading: How AI reads your website (and what it ignores).
Andy Rodriguez is the founder of Agentic Growth Engine, where he helps service businesses across the US get found, trusted and recommended by AI search. More about Andy.
When AI is asked who to recommend in your market, does it name you or your competitor? Most owners have never checked, and the answer is rarely what they expect. Before you assume your site is readable, find out: request your free AEO and SEO visibility audit and we will show you exactly what AI can and cannot see on your website.