Cloudflare announced a major policy shift that could reshape the dynamics between content creators and AI companies.
Beginning Tuesday, the company will automatically block AI crawlers from scraping the websites it powers, unless site owners explicitly opt in. This makes Cloudflare the first major internet infrastructure provider to enforce a default permission-based model for AI content access.
The move comes amid growing concerns from content creators and publishers that AI giants are exploiting their work without consent or compensation.
Historically, search engines have indexed content in a way that drives traffic and ad revenue back to original sources. But AI bot crawlers, used for training large language models, harvest vast amounts of data and send much less traffic back to the original creators of this content. These bots are used by industry giants including Google, Meta, OpenAI, and Anthropic.
Cloudflare, which manages and protects traffic for about 20% of the web, initially rolled out an optional one-click AI crawler blocking system in 2024. More than a million customers enabled it. Now, this becomes the default: AI companies will now be required to obtain explicit permission from websites before scraping, flipping the script on passive data harvesting.
“Original content is what makes the internet one of the greatest inventions in the last century, and we have to come together to protect it,” Cloudflare CEO Matthew Prince said.
“AI crawlers have been scraping content without limits,” he added. “Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”
Pay Per Crawl
As part of its broader push toward a permission-based internet, Cloudflare also rolled out Pay Per Crawl on Tuesday. This new feature lets select publishers and creators charge AI companies for accessing their content. Participants can set pricing for individual crawlers, granting full control over how and whether their work is used in AI model training.
Cloudflare hopes to create a transparent, consent-driven marketplace that helps creators decide whether to allow all AI crawlers, permit specific ones, or set their own access fees, turning previously unmonetized content usage into new revenue streams.
For AI companies, Pay Per Crawl offers a streamlined interface to browse access terms, view pricing, and choose whether to pay or walk away without the data.
Pay Per Crawl is currently available to a select group of partners, with broader access available through this sign-up page.
Major publishers have signed on or expressed support for Cloudflare’s latest move to block AI bot crawlers by default, including Gannett, Time, BuzzFeed, The Atlantic, and the Associated Press. Other companies, such as Stack Overflow, have endorsed the initiative.
At its core, Cloudflare’s moves are an attempt to reset the internet’s economic model for the new age of generative AI. This initiative doesn’t halt AI innovation, but encourages it to grow responsibly by paying for intellectual property and rewarding the creators behind the data.
Sign up for BI’s Tech Memo newsletter here. Reach out to me via email at abarr@businessinsider.com.
