The coalition of the willing (and unable): publishers rally to wall off AI’s free ride

“Publishers are still very much concerned about whether those companies are even going to participate, which is why the consensus in the room was we have to block them,” said Anthony Katsur, CEO of the IAB Tech Lab. 

The hope? Put up a wall and force a conversation. That wall is taking shape in the form of a technical framework – the LLM Content Ingest API – designed to let publishers control who can access their content, and under what terms. The idea is to bake in permission and payments from the start. A formal spec is expected this fall. 

If it holds, it could slow down the firehose of free content feeding AI models — and turn access into something that has to be earned, or at least paid for. 

“Most of those companies aren’t profitable so is there a market where if we block crawlers at large and start to starve them, will it kind of force some of them to come to the table again?,” said Katsur. 

That’s the bet. A coalition is forming around it, with plans to go public, and formal, by the end of the month. 

“From announcement to publication of a specification within five to six months, that’s lighting fast given how fast we moved on the Privacy Sandbox,” said Katsur. 

But for all its momentum, this coalition of the willing is also defined by a different truth: it is equally a coalition of the unable – at least for now. 

The reality is many AI companies have already demonstrated their willingness to ignore publisher signals. Some treat publisher guidance – like robots.txt – as advisory not authoritative. Others scrape content while masking their identities or using obfuscated crawlers. Even publishers that have implemented technical blocks still see their content siphoned.

“What’s notable is that not all the LLMs are obeying block orders,” said Katsur. “They’ll find other ways to surreptitiously continue to crawl.”

Which is why this effort isn’t about airtight enforcement. It’s about building enough leverage to make ignoring publishers no longer worth it. 

As several execs who were at the event put it: the time to lock arms is now while there’s still something left to protect. Whether that unity holds is another matter. The temptation for some publishers to cut a deal – to trade access for short-term relief – will be hard to resist, especially in a market where pressure is mounting and margins are thin. 

“Industry bodies cannot dictate commercial decisions by its members, but it can create buy-in and adoption of the standard, or the line in the sand that we’re all going to act within,” said Amanda Martin, CRO at Mediavine, who was at the event. “That’s what I’m looking to see come out of the IAB’s role.”

No one left the room thinking this would be easy. There are still too many moving parts. But the point of the meeting was never to solve the problem. It was to draw a line and, crucially, to agree on how. 

First: that the line only holds if everyone uses the same framework, one that works across the infrastructure and provides consistent tracking, control and enforcement. Without that consistency, the wall falls before its built, and publishers are back to playing whack-a-mole with anonymous crawlers. 

Second: attribution isn’t optional – its table stakes. No attribution means no recognition, no relationship, no visibility. Just summaries, regurgitations and AI-generated blurbs that borrow the information, context and credibility of the original – albeit without the source. 

And third: there won’t be a single monetization model. What’s likely to emerge is a hybrid approach – some form of pay-per-crawl, where AI companies are charred each time a bot ingests content, paired with models that pay publishers each time their content powers an AI-generated response. However the IAB Tech Lab’s framework evolves, it’s expected to lean toward the latter. 

Or, as one exec framed it, this entire push is about four 4Cs: control, consent, credit and compensation. 

“It’s important that the industry is coming together to think through ways to put friction into the system to make it harder for the AI systems to steal everybody’s content,” said Paul Bannister, chief strategy officer at Raptive, which sells ads for independent sites. “We’re seeing the industry come together to really think through what an ongoing payment structure between publishers and AI companies looks like.” 

Still, even the most aggressive blocking effort is just one front in a much bigger battle. The larger reckoning will play out across courtrooms, regulator chambers and publisher boardrooms. That much was clear at the IAB Tech Lab meeting, where a rare thing happened: publishers walked away with something resembling hope. The no-shows from the major AI platforms didn’t go unnoticed – and shouldn’t be excused. But the presence of Google and Meta sent a different kind of signal: these companies are seasoned enough to know that ecosystems break when one side takes everything and gives nothing. 

“If we do this right, everyone who creates information can take share in a growing market,” said a publishing exec who attended the meeting but asked to remain anonymous because they were not authorized to speak to Digiday.

That’s the hope, anyway. That common sense eventually catches up to disruption.