Cloudflare’s Vision for a Fairer Agentic Internet
Cloudflare is introducing new tools, analytics, and partnerships designed to help website owners control how AI systems access their content, improve discoverability, and receive compensation when their work contributes to AI-generated answers. The company’s guiding principle is simple: your content, your rules.
Cloudflare says automated agents and bots now generate more than half of all web requests. As AI becomes a primary way people search for information and complete online transactions, website owners face a difficult choice: make their content visible to AI systems or risk allowing it to be used without compensation.
The company’s new initiatives aim to create a more transparent ecosystem by giving publishers and creators greater control over AI crawling, improving visibility into how AI companies use content, making AI search more efficient, and developing systems that compensate publishers when their content generates value.
A major part of the announcement is Cloudflare’s plan to change default settings for certain customers on September 15, 2026. New customers and new sites will be configured to allow search access while blocking AI training and agent use on pages containing advertisements, unless site owners change those settings. Mixed-use crawlers that do not clearly distinguish between search, AI agents, and training will also be blocked on pages with ads.
Why the Internet Is Changing
Traditional search engines primarily direct users to websites. AI answer engines, by contrast, can summarize information directly within an AI-generated response. This shift changes how content is discovered, consumed, and monetized.
For publishers that rely on advertising or subscriptions, increased visibility does not necessarily translate into revenue. Their content may be used to train AI models or generate answers without sending meaningful traffic back to the original site.
Cloudflare argues that website owners should not have to choose between being discoverable and protecting their intellectual property. Site owners should be able to specify whether their content may be accessed for:
- Search and discovery.
- AI agent interactions.
- AI model training.
- Commercial use.
The company’s proposed approach is intended to separate these purposes more clearly and give publishers more meaningful control.
New Rules for Mixed Crawlers
Cloudflare is particularly focused on crawlers that combine search, agent activity, and AI training under a single identity. These mixed-use crawlers make it difficult for site owners to permit useful discovery while blocking activities they do not approve.
According to the announcement, Cloudflare plans to test and refine new classifications with industry feedback before a September 15, 2026 deadline. For applicable new customers and sites, the expected defaults will:
- Allow search access.
- Block AI training on pages with advertisements.
- Block AI agent use on pages with advertisements.
- Block mixed crawlers that do not provide separate controls for different purposes.
Customers will still be able to modify these settings through the Cloudflare dashboard. Existing free customers who have not changed their settings by the deadline will also be affected by the proposed defaults.
Greater Visibility Through Analytics
Cloudflare is launching an Attribution Business Insights dashboard to help organizations understand how AI bots interact with their websites.
Historically, AI traffic data has largely been handled by security and IT teams. Cloudflare says its new analytics are designed to make this information useful to business leaders, publishers, and content owners as well.
The dashboard is intended to help organizations track:
- Which AI companies crawl their content.
- How frequently their pages are accessed.
- What content AI bots consume.
- How much human traffic AI companies send back.
- How content appears in AI-generated answers.
This supports a growing practice known as Answer Engine Optimization, or AEO. While search engine optimization focuses on ranking in traditional search results, AEO focuses on how frequently and prominently a brand or publisher is cited in AI-generated responses.
Making AI Crawling More Efficient
Cloudflare also wants to reduce unnecessary crawling. The company says more than 50% of AI crawler traffic is spent re-fetching pages that have not changed.
Repeatedly downloading unchanged pages creates unnecessary costs for both publishers and AI companies. It uses bandwidth and computing resources without improving the information available to users.
Cloudflare is testing signals that can tell AI systems when a page has changed and should be fetched again. When broadly available, these signals could help:
- Reduce bandwidth consumption.
- Lower AI infrastructure costs.
- Limit unnecessary server requests.
- Improve the freshness of AI-generated answers.
- Make crawling more efficient for publishers and AI companies.
From Pay Per Crawl to Pay Per Use
Cloudflare is also evolving its Pay Per Crawl model into Pay Per Use. The distinction is important: publishers would be compensated not merely when an AI company downloads their content, but when that content is actually used to create value.
Cloudflare is working with partners including Ceramic.ai and You.com on different versions of this approach.
With Ceramic.ai, participating publishers can be paid when their content appears in AI search results. They can also receive information about the queries, citations, and rankings associated with that use.
You.com is exploring an on-demand model in which an AI agent can pay for access to a specific piece of premium content when it needs it. Cloudflare’s role is to provide a common infrastructure layer that supports these transactions across different AI companies and publishers.
Building on Content Independence Day
The announcement builds on Cloudflare’s earlier work to give website owners more control over AI access.
Over the past year, the company has introduced or supported several initiatives, including:
- AI Crawl Control, which lets site owners manage AI crawler permissions.
- Partnerships with platforms such as beehiiv to help creators track, block, or permit access by specific AI models.
- Web Bot Auth, an open framework designed to help AI bots and autonomous shopping agents verify their identity.
- Payment infrastructure intended to support secure AI-powered commerce.
- Collaboration with content platforms and publishers seeking more control over AI training and discovery.
- Cloudflare describes these efforts as three foundations for the agentic Internet: control, transparency, and monetization.
Industry Support
Several publishers and AI companies cited in the announcement support Cloudflare’s approach.
beehiiv emphasized the importance of giving independent creators control over how their work is discovered and used. Ceramic.ai highlighted the need for transparent compensation when content appears in AI search results. Condé Nast supported efforts to ensure premium publishers are paid for their content.
Patreon also described a model in which creators should be able to grow their audiences through search while restricting known AI training crawlers. This reflects the broader distinction Cloudflare is promoting: content can remain discoverable without automatically being available for every form of AI use.
What It Means for Website Owners
Cloudflare’s announcement suggests that website owners will need to make more deliberate decisions about AI access. Rather than treating all automated traffic the same, businesses may increasingly define separate policies for search, training, agent activity, and commercial use.
Website owners should consider:
- Reviewing their current bot and AI crawler settings.
- Identifying whether their business depends on advertising, subscriptions, licensing, or direct sales.
- Deciding which AI uses provide value and which create unacceptable risks.
- Monitoring how AI systems cite and refer traffic to their content.
- Evaluating whether compensation models such as Pay Per Use fit their publishing strategy.
For publishers, the goal is not necessarily to block AI altogether. Instead, Cloudflare’s model is based on allowing owners to choose the conditions under which AI systems may access and use their content.
A New Contract Between Content and AI
Cloudflare’s proposal points toward a new relationship between publishers and AI companies. In the traditional web model, search engines indexed content and sent users back to websites. In the agentic Internet, AI systems may interpret, summarize, and act on information on behalf of users.
That new model requires clearer identity, better measurement, more efficient crawling, and fairer compensation. Cloudflare’s latest tools are designed to provide the infrastructure for those relationships.
Whether the proposed defaults and payment systems become widely adopted will depend on participation from publishers, creators, AI companies, and other technology providers. However, the central idea is clear: AI access should be transparent, configurable, and aligned with the rights and business models of content owners.
Source