Cloudflare gives AI crawlers three lanes

Cloudflare is giving every customer three controls for AI traffic—Search, Agent, and Training—and will block Agent and Training by default on ad-displaying pages for new domains from September 15. Its new AI traffic taxonomy says multi-purpose crawlers are governed by all their behaviors, while Cloudflare says more than 20% of web domains sit behind its network. Content teams should audit robots.txt and agent access against the site’s analysis of AI search referral economics before a crawler-policy default becomes a broken acquisition path.