Patreon, the membership platform for creators, is taking a harder line against AI companies that scrape its content for training purposes. On Thursday, the company announced it is partnering with internet infrastructure provider Cloudflare to directly block access to AI bots designed to train their models on creators' work without permission.
Stricter measures needed
The strengthened measures were necessary because AI scraping has become more sophisticated since Patreon first put measures in place to deter AI crawlers in 2023, the company said. Patreon's paywall has long kept much of creators' content out of reach of crawlers. But more recently, the company introduced new discovery tools like a redesigned Home Feed and its tweet-like Quips feature, which could expose more content to crawlers.
The changes come as more online publishers and content creators are grappling with how AI is ingesting their work to make AI models smarter. To address this, Cloudflare now offers tools that allow website publishers to restrict AI bots, including a marketplace that lets websites charge AI bots for scraping, called Pay Per Crawl. Earlier this month, Cloudflare changed its policies so that mixed-use crawlers, meaning those that both index and train on a website's content, are blocked by default on any pages that host ads.
From asking to blocking
Patreon says it is extending its existing work with Cloudflare to use the company's AI Crawl Control technology to update its AI policies and enforcement tools. The difference is that instead of simply asking AI crawlers not to scrape content using robots.txt files, a standard way to provide bots with instructions on how they can use a site, Patreon is now actively blocking AI training bots.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
"Consent shouldn't depend on whether a scraper chooses to behave," a Patreon blog post explains, referencing the stricter measures.
When testing the features, individual AI training crawlers' weekly attempts to access Patreon went from "thousands of attempts to zero," the post noted. That indicates that the AI scrapers were ignoring Patreon's robots.txt file and scraping the site anyway, despite its requests.
Allowing indexing bots
However, the company said it will allow bots that index pages and organize information that can be used to send users back to Patreon.
"As AI agents become increasingly powerful and popular, creators deserve a meaningful say in how their work is used by AI companies," remarked Patreon's product chief Drew Rowny in the announcement. "On most of the Internet, creators have to accept AI training on their work just to reach and grow an audience. Patreon has a different vision: creators should be able to grow their audience and control how their work is used."