Why publishers must act now to protect content from AI scrapers

AI is the latest and potentially most disruptive wave. AI companies have spent four years crawling the web at industrial scale, ingesting decades of journalism to train models that answer questions directly instead of sending readers back to the source. Content that publishers built and funded is being taken without permission, compensation or attribution. AI companies have little incentive to pay while they can access publisher content for free. 

Publishers can retain revenue and control — if they move first

Media leaders understand the risk. Yet research from digital analytics and customer engagement solutions provider Piano shows that fewer than 20% of the company’s 500 publisher customers use aggressive AI-bot blocking, while 30% use none at all. Without blocking, there’s no leverage to negotiate. 

Some publishers worry that blocking AI means losing the chance to be discovered through AI. But less than 0.1% of publisher traffic comes from AI tools, according to data from Piano. Large language models keep users inside their own environments, so most readers never click through to the source. Anthropic sends back one visit for every 8,692 pages it scrapes; Perplexity, one for every 369. Until those numbers change, appearing in AI results doesn’t bring meaningful value.

Google’s pivot to AI Overviews has added to the pressure. Since AI Overviews launched in May 2024, clicks to publisher sites are down 49%. At Google I/O 2026 in May, Google unveiled a more AI-centric search experience that puts its chatbot-style AI Mode at the center of the search box, potentially pushing the traditional 10 blue links further down the page. 

If organic search — currently 25% of traffic to Piano customers, according to internal benchmark data — continues to erode, the economic case for producing original content could collapse. Likewise, as long as AI companies can scrape content for free, they have no incentive to pay. 

The path to compensation in the AI era runs through scarcity — and scarcity requires taking action like blocking bots and locking down content to create leverage. Then, publishers can negotiate.

Here are four steps publishers can take now to create a path to compensation in the AI era.

1. Protect content

Publishers can limit AI companies’ access to their content by declaring exclusions in robots.txt, blocking at the content delivery network (CDN) level, and locking paywalls to recognize only human visitors.

Some AI companies respect publisher rules, but many crawlers don’t. Piano’s analysis of Cloudflare data shows 70% of bot traffic is unverified. Perplexity has been documented circumventing blocking by impersonating Chrome browsers. Tavily openly admits its crawler skips a distinctive user agent specifically to dodge rules that let only Google crawl a site.

Basic bot-blocking may not be enough. Content protection ultimately means locking content for everyone, human or not, because the industry can’t reliably distinguish AI agents from human readers. One of the most secure approaches is requiring all readers to register, using server-side or CDN edge protection to block content from any unknown user.

Some users will leave rather than register, but Piano data shows those are low-value users, while registered users are far more likely to return and subscribe.

2. Know the audience

Owning a direct relationship with readers has always been valuable, but in the AI era it’s essential. The publishers who survive will be the ones with direct audience connections: users who log in, subscribe and consent to be known. That makes registration more than a way to restrict access. It gives publishers a direct relationship with readers.

3. Monetize on one’s own terms

An agentic commerce future in which AI systems pay publishers per crawl, per answer or per piece of content is real, but it’s not here yet. Platforms like TollBit and Pro Rata and standards like x402 and IAB’s Content Monetization Protocols are developing the frameworks. But no one is making meaningful money from bot marketplaces yet.

Publishers should build that infrastructure now: tracking systems to show what content was accessed, an identity layer to identify subscribers and AI agents, and tools to convert users who arrive.

4. Use AI to create value while retaining content ownership

A defensive strategy won’t drive growth on its own. The same AI companies threatening publishers also enable valuable tools, including personalization at scale and AI-powered subscription monetization. The next generation of predictive systems will auto-optimize in real time, cutting manual work.

At the same time, an industry that spent 20 years unable to coordinate is starting to unify around how its content is used. The Standards for Publisher Usage Rights (SPUR) initiative is the clearest signal: 30 new members joined at WAN-IFRA’s World News Media Congress in June. The Real Simple Licensing Collective and the News Media Alliance are building similar infrastructure.

Publishers are also taking a harder line on how AI companies access their content. The Atlantic and People Inc. have put aggressive bot-blocking into practice. At the WAN-IFRA June conference, The New York Times’ publisher called on the industry to stop giving content away and demand fair terms. 

Publishers should use AI where it creates value while retaining control over the content that powers these systems. The right move is the same one the best publishers made when subscriptions became necessary: be decisive, be clear about the value offered and build a business that doesn’t depend on someone else’s goodwill.

The Publisher’s Content Protection Playbook from Piano covers practical guidance on blocking scrapers, how to know what content is worth before licensing it to AI companies, how to build leverage in AI licensing conversations and, how Piano protects content, creates personalized experiences and builds stronger reader relationships. 

Partner insights from Piano