If your website content drives awareness, leads, or conversions, there’s a change on the horizon worth paying attention to. Cloudflare, the security company quietly protecting much of the internet, has started blocking AI crawlers from accessing websites in its care.
Depending on your goals, this could be a step towards protecting your content, or a missed chance to be seen in AI-driven searches.
So, what’s changed?
In short, Cloudflare now blocks AI companies like OpenAI and Google’s Gemini from crawling websites unless the site explicitly opts in. If you don’t want AI models harvesting your hard-earned content to fuel their chatbots, Cloudflare is offering a way to say no.
The clampdown comes as AI models hoover up vast amounts of data from websites, often without permission. The models get smarter, but website owners often see fewer clicks and less traffic. It feels like a raw deal for those investing time and budget into creating useful, original content.
The disappearing clicks problem
The way people discover and engage with content is shifting. AI chatbots and Google’s AI overviews serve up answers directly, often without the need to visit the source website. That means fewer clicks, but it doesn’t mean your content becomes invisible.
In fact, your blog posts, guides, and resources might still be the foundation of those AI-generated answers, quietly building brand recognition even if the visitor never lands on your site. It’s a change in behaviour, and for marketers, it could mean lower traffic but more focused, ready-to-convert visitors.
Why this matters for marketers and businesses
For businesses that invest in content to build visibility, AI brings both risk and opportunity. Blocking AI crawlers might protect proprietary research, sensitive resources, or specialist content. But for many, the goal is to be found, whether through Google search or AI-generated answers.
If your content isn’t visible to AI, someone else’s will be. Like it or not, AI discovery is now part of how people research, learn, and make buying decisions.
A new way to measure content value?
Cloudflare has floated the idea of a new content value metric, based not on clicks or traffic, but on how much your content contributes to improving AI knowledge.
It’s an interesting thought, but let’s be realistic. Most marketers still need results they can track, like leads, enquiries, and sales. While the philosophical debate continues, the practical advice remains simple.
What to do next
If your site runs behind Cloudflare, review your settings. Some businesses will want to block AI crawlers to protect their content. Others will want to allow access, making sure their brand and expertise appear in AI answers and chat responses.
The key is making an informed choice, not leaving it to default settings.
Final thoughts
Cloudflare’s move doesn’t fix the disappearing clicks issue, but it does give businesses an option. You can block AI crawlers to protect your content, or you can lean into the shift and make sure your content is part of AI-generated answers.
For many marketers, being part of those AI insights is where the real opportunity lies. Your content might not always bring in direct clicks, but it can still shape what people see, hear, and learn when they search with AI tools.
The key is making a conscious decision. Understand how AI fits into your content strategy, decide what should be protected and what should stay discoverable, and make sure your content keeps working for you, even when the clicks don’t come straight through.
AI-driven discovery isn’t going anywhere. Making your content part of that conversation might just be the smartest option on the table.