Cloudflare and GoDaddy partner to help websites fight back against AI crawlers

Cloudflare and GoDaddy have announced a partnership to address how AI crawlers from Big Tech companies are affecting website traffic and publisher revenue. The move focuses on the growing problem of AI answer engines reusing web content without sending users back to the original source. As part of the deal, Cloudflare’s AI Crawl Control will be integrated into GoDaddy’s hosting platform, letting site owners allow, block, or require payment from AI bots.

The partnership is a response to the rise of AI answer engines like Google's AI Overview, which increasingly give users direct responses instead of linking them back to websites. That shift has increased tensions between content creators and AI companies, as publishers see less traffic and significantly weaker returns while their content continues to be used, and often not even credited.

Cloudflare and GoDaddy are also backing standards such as Agent Name Service and Web Bot Auth to verify AI agent identity through cryptographically signed credentials. Alongside audit logs that give site owners more visibility into bot activity, the broader goal is to build a permission based system where identity, access control, and compensation help protect human created content online, which in my personal opinion feels more necessary than ever right now.