How Time and others are rebuilding parts of the web for AI agents 

Time, for example, is converting all its webpages from HTML into markdown versions, a simplified format that AI systems and agents can process more efficiently. Delivering content to AI agents in HTML is inefficient because it bundles layout, style and navigation information designed for humans and browsers. Markdown strips that away, giving agents the content and metadata, not the surrounding page design. 

Last month, Time decided to block all AI bots by default, then created whitelists of approved bots that can access its content. Those approved bots are redirected to markdown versions of all of Time’s webpages, according to Mark Howard, chief operating officer at Time. 

Making it easier for AI agents to access Time’s content should help improve the publisher’s visibility in AI search, noted Howard – and in turn, strengthen Time’s pitch for the GEO product it’s selling to brands to help shape their messaging in AI search.

“[The bots are] just getting the content itself and the metadata, but they’re not getting the full page experience, and we’re routing all the humans to the full page experience. So we’re separating out that traffic,” Howard said. 

“Now we’re starting to think about, as the volume of bot traffic continues to increase significantly – and we see through a number of our vendor partners that we have very high domain authority with AI bot traffic – there’s value in that,” he added.

Time is using TollBit, a marketplace for publishers and AI companies, to convert HTML webpages into markdown. TollBit claims that scraping and processing large HTML pages can take over a minute, while AI systems can fetch structured content in 0.25 seconds through TollBit. Not only is this process more efficient, according to TollBit, but it could also lead to fewer hallucinations by AI systems, because they are getting data that is easier to understand.

“Part of onboarding to TollBit is we create your agent site for you,” said Toshit Panigrahi, co-founder and CEO of TollBit. “It really comes down to the token economy. Websites have a lot of HTML tags and JavaScript and CSS and things that don’t have to do with the content. That creates a big bloat in the actual size of the page.” 

Markdown can make websites “friendlier” to agents, he added. “AI can comprehend more of your article because they’re not spending money parsing out other HTML that’s on the page. We see, on average, a 90% reduction in tokens, because we have converted the content to markdown.”

The Economist is also experimenting with agent‑readable versions of content that already sit outside its paywall, Digiday previously reported. But unlike Time’s blanket move to create markdown versions of all of its website content, The Economist is starting with a narrower slice: marketing copy and B2B sales material. As a subscription-based publisher, it has to weigh the upside of exposing content to AI agents, against the risk of eroding the value of its paid offering.

A third major news publisher is taking a different tack again, experimenting with Web Model Context Protocol (WebMCP), a web standard co-developed by Google and Microsoft, according to a publishing exec at the company, who spoke on condition of anonymity. WebMCP is designed to let websites share structured data directly with AI agents, rather than being scraped or clicked through – a slightly different web standard from Anthropic’s original MCP,  according to a Google blog post.

Not only could WebMCP help improve the publisher’s visibility and citation rate in AI search tools, but it could also reduce the costs associated with bot traffic by making it cheaper and more efficient to serve those requests, the exec said. 

“Google is proposing it as a whole agentic layer. We are looking at how we can create that layer. There are some cost advantages there. When bots hit your human site there’s a cost associated with serving those pages, so if you’re making it easier [and] quicker [for bots], the CDN cost is advantageous for us.”

There are also internal benefits to creating versions of webpages for agents, the exec noted. Standardizing the process of connecting webpage data with AI systems means the publisher’s own AI tools can more efficiently access its content, potentially improving its internal and external-facing AI products, they said.

It’s one of several variations on the same theme. Elsewhere, French daily Le Monde is experimenting with how it surfaces content to agents and detects whether those agents are acting on behalf of paying subscribers. 

Yet, even as more publishers quietly spin up agent-friendly feeds, stripped down pages and custom schemes, not everyone is convinced they should be racing to re-architect the web for bots. Independent publisher consultant Scott Messer, principal of Messer Media, argues that building for agents should be a highly qualified decision, not the default. His reasoning: traffic isn’t the reward in an agentic environment – if there is no click, no ad impression and no check, the build is pure cost. 

He pushes back on the idea that publishers should optimize for agents purely to avoid “disappearing” from AI-generated answers. In his view, you only expose content to agents if you genuinely believe there’s long-term value in being discovered and cited in those systems. 

“If you believe there’s a value to being discovered by these bots and agents, then you should build them. If you don’t believe [that], I would ask, why would you build them?,” he said.

The answer will look different for a hard paywalled news brand than for a scale, ad-funded lifestyle publisher, he noted. “There’s a philosophical debate around like: is that a visit? Is that valid traffic under, like MRC [Media Rating Council] terms of valid or invalid? Did a human see it? Can a bot be influenced by ads? I don’t know, probably not.”

Jessica Davies, senior media editor, contributed to this story.