WTF is headless browsing, and how are AI agents fueling it?

These players are using headless browsers to power AI agents that need to click, scroll and interact with websites as a human would, to retrieve information. 

For media companies, that raises questions: How much traffic is real vs. automated? How should analytics platforms account for agent-driven browsing? And for advertisers, it raises concerns about whether AI-driven sessions will distort measurement or expose new vulnerabilities to fraud.

Read on to find out what this means for publishers, at a time when many are working to block AI bot traffic and protect their content from getting scraped without their control or compensation.

What is headless browsing in the AI era?

Here’s how people typically interact with AI engines: A user goes to an AI application (like ChatGPT for example) and types in a prompt. That AI system either operates on its trained knowledge, or goes out to retrieve fresh information from websites (known as retrieval-augmented generation.) These AI bots identify themselves to websites through their user-agent strings, such as ChatGPT-User or PerplexityBot. The AI then synthesizes the retrieved information with its own training to generate the final answer to the user’s prompt.

But a newer wave of AI interfaces — including autonomous agents and AI browsers — works differently. Instead of making targeted retrieval calls and flagging themselves as bots, they rely on headless browsers that actually peruse sites the way a human would: clicking, scrolling and loading pages. Often, these sessions present themselves as standard visitors (for example, using a Chromium string) rather than as a named bot. That makes it much harder for publishers to distinguish automated browsing from real human traffic, according to a new report from Tollbit, a data marketplace for publishers and AI companies, published on Sept. 8.

The dynamic can be seen in AI web browsers. These tools use AI agents that operate inside a browser to visit websites, without the user ever needing to see webpages. And rather than identifying themselves by their user agent name (ChatGPT-User, for example), they present as a standard visitor (such as Chromium, Google’s open-source web browser engine). The result: publishers can’t easily tell what’s human and what’s AI-driven.

According to Tollbit’s report, AI headless browsers can bypass common bot-blocking techniques. Headless browsers like Browserless claim they can evade these detection tools, and solve CAPTCHAs (which are meant to weed out humans from bots).

“In the case of Perplexity Comet, the agent appears to use the user’s own desktop to request webpages when automating workflows. This makes the requests appear to come from a user’s residential IP address, which can confuse [content delivery network] companies into thinking it is legitimate,” Tollbit’s report reads. The report found a 336% increase in websites blocking or redirecting AI bots over the past year.

How big of a problem are AI headless browsers?

It’s still early days, but these new AI headless browsers do pose a threat to publishers’ businesses, execs told Digiday.

The adoption of AI browsers is only going to increase. PayPal and Venmo users (so, millions of people) were just offered access to the Comet browser for free this month (previously, it was available to Perplexity subscribers and a small group of early users on a waitlist).

In addition to new AI browsers on the market, OpenAI is rumored to be developing one, and Google is currently developing one through its Project Mariner. 

Publishers have already moved to take a stronger stance against AI bot traffic and content scraping. AI headless browsers could be the next evolution of that battle.

“There’s the notion that whatever work we’re doing here on blocking traffic and crawlers and robots… that gets circumvented all together,” said an exec at a business publisher, who asked to remain anonymous. But the fact that new challenges are cropping up in this area shows that these efforts are worthwhile, the exec added. “It’s a reinforcement of, we’re doing the right thing [by] blocking as much as we can.”

A different exec at a large digital publisher, who requested anonymity, said they believed that, for now, headless browsers posed a minor issue. But it could become a significant problem if major players like Google or OpenAI adopt them.

What does it mean for publishers?

It’s the next step in a broader shift: AI systems are starting to replace the need for users to search or visit publisher sites directly, chipping away at publishers’ (human) traffic. TollBit data suggests that human traffic to publisher websites is beginning to decline, while bot traffic is growing. There was a 9.4% decline in the number of human visitors to websites between Q1 and Q2 2025, according to the Tollbit report. Meanwhile, the ratio of site visits coming from AI bots compared to human visitors was 1 in 50, up from 1 in 200 in Q1 2025.

First of all, this is a transparency issue. Publishers can’t have control over something they can’t fully see or measure. It’s also a business issue. Ads are being served on publishers’ sites to AI agents, rather than humans. According to Toshit Panigrahi, co-founder and CEO of TollBit, ads are being served in OpenAI’s agent mode in ChatGPT, which can complete similar tasks to an AI headless browser.

“If this is triggering an actual ad serving call… multiply that by a million and it can basically be… unbelievable scale ad fraud,” the business publishing exec said, who added that this has the potential to drive down CPMs. 

“The first [next] step will be information,” the exec said. But it wasn’t like the exec planned to bring this up at the next advertiser client meeting. “As a publisher that relies on advertising to keep the lights on, we can’t be going out and beating the drum saying, ‘This is bad.’ We’ll let them come to us on that one.”

The cost of loading webpages in response to AI agent requests can add up for publishers too, because they have to pay for better cybersecurity tools, Panigrahi said. 

Jesse Dwyer, head of communication at Perplexity, said the costs were far greater for Perplexity. “This does create some real cost considerations for publishers and AI companies to tackle together. When a user sends their Comet assistant to a site, it costs Perplexity 350X to 2300X more than the publisher’s cost to serve that user a single page view. The new costs of giving people a better internet are proof that users have been frustrated by the old internet. So publishers and AI companies are together at the forefront of creating a better, healthier internet,” Dwyer said.

How can publishers detect headless browsing?

There are some signals publishers can look for that indicate they’re receiving traffic from AI headless browsers, Panigrahi said.

If the request isn’t rendering JavaScript, it’s likely not from a human – because most people have JavaScript in their browsers, Panigrahi said. If the IP address is coming from a data center, that can also be a red flag. The problem is that many of these requests come from residential IP addresses.

The digital publishing exec said there are a few signs they are looking for now, such as how many people are browsing its sites using the Comet browser. But that still doesn’t make it clear if the browsing is being done by an AI agent or human.

“At the moment, there’s no flag getting passed through from the browser to the website that tells you whether the browser is being human-controlled or AI-controlled,” the exec said.