Inside IAB Tech Lab’s meeting with publishers to confront the AI era

“It was focused on how the industry can respond to AI companies scraping their content, often for little or no money,” said Joseph.

That response has largely revolved around publishers’ blocking AI companies’ crawlers from accessing their web pages, but it has expanded to IAB Tech Lab’s LLM Content Ingest API that is designed to be a technical framework for publishers to give AI companies access to their content and, crucially, to control that access. But that would require AI companies adopting the IAB Tech Lab’s framework. While Google and Meta were in attendance, many of the major AI companies were absent from the meeting. So how likely is anything to come of the coming-together?

“I posed that exact question [to publishers], ‘Is this dead in the water without these big players even here, even willing to listen to these conversations?’ And I was kind of surprised by how optimistic some people were for now. ‘This was just the start’ is really the way that it was framed,” said Guaglione.

Here are a few highlights from the conversation, which have been edited for length and clarity.

The LLM Content Ingest API

Joseph: It’s basically a way to block AI platforms from scraping content unless they’re paying for it in some way. We’re expecting the technical spec in November, but whether the AI companies will accept it is an open question. OpenAI, Perplexity, Anthropic – they weren’t at the meeting last week despite the Tech Lab reaching out.

Publishers’ leverage

Guaglione: Now that the structure or model of these deals has really transitioned from training data to needing access to real-time, reliable content because of the boom of these AI search engines, it really puts a little bit more power in publishers’ hands. It allows them to have loftier demands. I don’t know how far they’re going to get with these, but I do think that they have a stronger case for the value of that real-time content that AI systems need in order to be something that people actually want to use.

A united front

Joseph: Given the existential issues at stake for the publishing industry, this idea of a united front seems more urgent than in past times where we have seen pleas for publishers to band together and then break up because of their own commercial interests. Time will tell how much of that is actually true. But the fact that this thing is moving so quickly – the fact that there is such a firm consensus around a specific direction to go – I’m less skeptical of that than I have been when we’ve heard about other united fronts.

‘That’s just the beginning’

Guaglione: Another publisher was telling me, “Some people are saying all those big deals [with AI companies] have been signed; what’s left?” And their perspective was, “No no, that’s just the beginning. The marketplace for this is just going to evolve and expand, and we need things like this to have some sort of standard in place for when that happens.”