Media Briefing: The case for and against markdown for AI bots

For years, publishing meant optimizing for Google. That’s shifting to optimizing for AI. Some publishers — such as Time — think the fix is as simple as markdown pages.

So much so that they’re creating markdown versions of their websites, a simplified format that strips away page elements like visuals and JavaScript to make it easier for AI systems and bots to understand. The thinking behind this is these pages can improve the likelihood that AI systems accurately process — and, potentially, more likely cite — their content. 

But not everyone is convinced markdown is necessary to do that. AI bots can already interpret standard webpages, skeptics argue. The debate also highlights a broader question facing publishers: is better visibility in AI worth making their content even easier for bots to scrape?

“In the short term, having a markdown copy of a webpage makes sense, in theory. But I don’t think it’s a great long-term solution. In the publishing world, we saw this with Google Accelerated Mobile Pages,” said Leigh McKenzie, director of online visibility at analytics firm Semrush. “Nobody’s doing that anymore. The web just kind of caught up.”

The case for

Markdown makes it easier for AI to read content

HTML pages are built for humans, with JavaScript, ads, design elements, video and photos to improve a user experience. But AI bots are looking for information that is clearly structured and easy to understand to complete tasks like answer a user’s question, index the information or use it to improve an AI model. 

The AI systems have to do less work to interpret the content on the page when it’s in a markdown file. This means fewer opportunities for mistakes and misrepresentations of the content in AI answer engines.

“Agents prefer markdown because it’s structured. They don’t have to get through all the HTML because what they normally do on the traditional page is they say, “OK, disable JavaScript, turn off image.’ It has to get through all the human stuff in order to get to what it actually wants, which is the underlying content, the references, the links,” said Jonah Goodhart, co-founder and CEO of AI-powered ad tech platform Mobian. 

Markdown could improve AI visibility

Markdown makes it easier for bots to pull a publisher’s content. The easier it is to do that, the more likely that content gets surfaced in an AI answer engine.

Several SEO and GEO vendors back this up. They argue that markdown can improve how reliably AI models ingest and retrieve content. The cleaner the inputs, goes the thinking, the easier to cite, with lower processing costs for the AI system. And the more visible a publisher is in an AI answer engine, the more likely the publisher is to reach a user with their content, improve its brand awareness and authority. That visibility also doubles as leverage in licensing talks with the platforms. In an ideal world, it drives some traffic back to their sites too.

Faster, cheaper AI crawling

When something is more efficient, it means it’s cheaper. Markdown can reduce how many tokens — or small chunks of text — an AI system has to process to understand a webpage. Fewer tokens can mean lower compute costs. And some publishers are betting that efficiency will eventually reward them with better visibility in AI search.

“It really comes down to the token economy. HTML has a lot of tags and JavaScript and CSS and things that don’t have to do with the content. That creates a big bloat in the actual size of the page. But in terms of markdown, it makes it friendlier, the inference costs drop. Crawl budgets last longer. AI can comprehend more of your article because they’re not spending money parsing out other HTML that’s on the page,” said Toshit Panigrahi, co-founder and CEO of TollBit, a data marketplace for publishers and AI companies. 

TollBit is one of the companies that helps publishers convert their webpages into markdown. TollBit sees about an average 90% reduction in tokens when content is converted to markdown, Panigrahi said.

Potential revenue opportunities

This is an emerging area, but one Time is experimenting in. The publisher began serving ads to AI agents in its markdown files last month. Mobian’s Goodhart said he prompted ChatGPT and Perplexity to pull a Time webpage and see if the AI agent came across the ads, which the agents confirmed and shared information in the ads back to Goodhart. He considers this proof that the AI agents were seeing and interacting with the markdown ads.

Rather than trying to keep content away from AI scraping, publishers like Time are investing around the influx of AI bot traffic it’s getting and figuring out how to monetize that traffic, not unlike the way the publishers monetize impressions through digital ads on webpages.

The case against

Markdown makes it easier for AI to read content

For the same reasons this is in the “for” bucket, it’s also in the “against.”

Not all publishers want AI bots scraping their content for free, without control of where that content is going and what it’s used for.

Many publishers have invested in efforts to slow or block AI bot scraping, enlisting the help of bot detection tools and Cloudflare protections, for example. Markdown removes a lot of that friction by making it even easier for AI bots to access content. Unless there’s some sort of monetization agreement in place, the value of letting bots scrape content is still in the works. Why would an AI company pay to use their content, if they can get it for free?

“When you take a website, you convert it to markdown, and you offer it to OpenAI or to Claude, it now has that data. You’ve now lost control. Essentially, you’ve leaked your information into that model,” said Ed Zyszkowski, co-founder and CEO of software development platform Personal Digital Spaces, which launched an AI licensing protocol for publishers in May. “It’s like saying, ‘We’re going to make the house more suited to a commercial user. We’re going to tear out the walls and make it a little different… [But you could] just rent out specific spaces in your house as needed.’”

Better AI visibility is hard to prove

And even if the argument is that better AI visibility can lead to improved mentions or citations in AI answer engines — which can help a publisher’s authority in the space and hopefully even drive back some traffic — there’s little data that shows markdown does help improve AI visibility and citations. GEO platform Promptwatch analyzed over 1.6 million citations in AI answers and found that markdown did not increase the likelihood of getting content cited.

“I think it’s more of a marketing thing than a technical thing,” Zyszkowski said.

Publishers and brands creating markdown versions of their sites are taking a bet on where the agentic web is going to go, but it’s not a necessary step yet, said Choy Travers, co-founder of Oasy, a startup building an AI advertising layer for publishers.

“I think it’s a good bet to do… It’s great for the big players to add this and not have any revenue for it in the near to mid long term, but for most publishers they’re way more concerned about having revenue streams. Where can they make money now? Where can I diversify the business? And it doesn’t make sense when there’s no proof there,” Travers said. “I don’t think it’s going to move the needle in the short term.”

Oasy has tested putting ads in markdown and in HTML webpages, and found that ads in markdown did not change advertisers’ results or win rate yet, Travers said.

Google guidelines: markdown isn’t necessary

Google recently published GEO guidelines to communicate what is and isn’t needed to optimize for generative AI search. In it, Google stated that best practices for SEO continue to be relevant because Google’s generative AI features on Google Search are rooted in its core search ranking systems.

One of the things not needed is markdown, according to the guidelines. Google search doesn’t use them to discover, crawl or index information, it reads.

Google did later add a note to this that offered further clarification, softening its language around markdown: “It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”

What we’ve heard

“It’s playing a relatively minor role in the growth in advertising so far, and you’re going to see us really focus on scaling production, scaling engagement, and then scaling monetization.”

– Meredith Kopit Levien, CEO of The New York Times Company, on video’s role in ad growth during Wednesday’s Q2 earnings call.

Numbers to know

4X: European sites’ median AI scraping per site, compared with North American sites, in the first half of 2026, according to a new TollBit report.

1:33: The ratio of AI bot scrapes to human visits on European sites in H1 2026.

0.16%: The share of external referrals to North American sites that came from AI apps in H1 2026, compared with 0.05% for European sites.

8.3%: The percentage of U.S. newspaper jobs now based in Manhattan in 2025, up from 3.3% in 2016.

What we’ve covered

People Inc CEO: We’re not turning off Google crawlers yet

• People Inc. CEO Neil Vogel said the publisher wants to block Google crawlers to protect its content from being used by AI, but can’t do so yet because it would cut off valuable search traffic.
• 21% of People Inc.’s traffic still coming from Google search, despite a 40% year-over-year decline in Google search traffic.

Read more here.

WTF is a stealth crawler?

• Publishers and lawmakers are pushing back against stealth crawlers by introducing new bills that require crawlers to identify themselves and their purpose.
• Stealth crawlers are AI bots that disguise themselves and can undermine publishers’ control over their content.

Read more here.

As Hollywood studios chase creator culture, ownership gets complicated

• After the runaway success of creator-led films this year, Hollywood is hungry for creators and is looking to gobble up more and more of their IP.
• But there could be bumps ahead. Acquiring IP that has been planted, cultivated, and propagated online calls into question the nature of its ownership.

Read more here.

Time has started serving ads to AI agents

• Time has begun serving ads to AI agents, by adding sponsored content to versions of its websites created for AI bots. Ally Bank and Project Management Institute are among the first buyers. 
• Time and Mobian claim this is the first time a publisher has served ads specifically targeting AI agents.

Read more here.

What we’re reading

Business Insider names new CEO

Christian Baesler, an Axel Springer advisor who joined Business Insider as interim CEO in June, will now be the permanent chief executive, Axios reported. He was the chief operating officer of BuzzFeed until 2023.

How Reuters, the BBC, and The Guardian are deploying AI

Nieman Lab conducted a study to analyze how three different media outlets were using AI in their workflows, ranging from speeding up processes to reformatting local news stories to fit style guides.

Right-wing media is struggling

The audience for pro-Trump content is declining, and some creators are reaching new lows in viewership, according to The New York Times, which analyzed audience data across social platforms for a dozen political content creators.  

Paramount CEO says he won’t bend CNN to his views

In a guest essay for The New York Times, Paramount CEO David Ellison assured critics that CNN would remain editorially independent if the proposed Paramount-Warner Bros. Discovery merger goes through, CNN reported.

Media’s New York bubble grows

New York City is becoming more dominant than ever as the center of American journalism — due to a combination of consolidation, layoffs and the decline of local newspapers — giving outsized influence to the Manhattan media bubble, Bloomberg reported.