Media Briefing: The transatlantic push to rein in stealth bots is gathering pace

AI scraping is growing at a rapid clip, and publishers are already fighting it at an enormous scale. 

Scraping accounted for 70.9% of malicious bot traffic analyzed by DataDome per its September report, and rose 185.2% over the study’s 12-month period. 

Individual publishers continue to wrestle with the volume of unwanted crawls. 

People Inc. now allows through 38 bots a day among “tens of millions” of AI crawler requests that are blocked, Jon Roberts, chief innovation officer at People Inc., told delegates at Digiday’s Media Strategies London event last week.

“They come in with a named crawler; we block it. They come in with an anonymous crawler; we block it. They pretend to be human; we block it. Then they try to be Google’s crawler… we block that, and then they come through people’s home proxies. They come through individual citizens’ internet. And this is where governments start getting involved,” he said.

At the same Digiday event, the Financial Times’ md of customer revenue Fiona Spooner said the paywalled title is plagued by bots on a daily basis, and is dedicating resources to determine how to distinguish between bad and good bots. To highlight the scale, she pointed to how one account alone tried to access about a million articles in eight hours from roughly 700,000 IP addresses. “We are being scraped and content taken and attempted all the time, as well as humans trying to get around it and access it in different ways,” said Spooner.

Last week, Reach CEO Piers North told Digiday it has seen over 100 million successful scrapes of its content over the last six months.

The figures build on Digiday’s April reporting, which documented a surge in AI bot activity and a growing ecosystem of third-party scrapers harvesting and selling publishers’ content without permission or compensation, while bot-blocking measures struggled to keep pace.

But blocking crawlers is only part of the problem: publishers also want to know who is accessing their sites and why. That demand is pushing the fight beyond technical defenses and into policymaking.

The U.S. publishing industry’s push to crack down on stealth crawlers is growing into a transatlantic effort. The News/Media Alliance organized more than 300 publishing executives to lobby Congress in September to support the federal Stealth Bot Prohibition Act, which would force stealth crawlers to clearly identify themselves and their purpose. Now, publishers in the U.K. are pushing for similar legislation.

A coalition of British and American publishers, including The Times, Financial Times and Condé Nast, urged the U.K. government to back a similar bill on Oct. 15. Danielle Coffey, CEO of the News/Media Alliance, told Digiday the bill is modeled after the Stealth Bot Prohibition Act. The effort in the U.K. is spearheaded by the U.K.’s News Media Association.

“We work closely with our U.K. counterparts,” Coffey said. The coordinated efforts are helping to gain international momentum on the issue of unfettered AI bot scraping of publishers’ content, she noted.

It reflects growing international pressure from publishers for greater transparency and control over how AI companies and third-party scrapers access and use their content. (The News/Media Alliance also helped draft and promote a New York bill cracking down on stealth crawlers.)

Pushing for regulation on AI bot scraping is one method publishers have to try to maintain control over how their content is accessed, used and monetized — and retain some leverage around AI content licensing deals. 

“I expect this issue will only keep growing,” said Gary Kibel, partner at law firm Davis+ Gilbert, which advises media and advertising clients.

The push for disclosure comes as publishers and their vendors report that crawlers can misrepresent who they are. A recent Piano analysis of Cloudflare data found that 70% of bot traffic is unverified. Meanwhile, Centinel, which examined crawler identity claims across the news and media sites it protects from late June through Sept. 21 for its Fake Googlebot and AI Crawlers report, found that crawlers were impersonating other bots.

It found that on a typical site, one in seven requests claiming to be Googlebot failed verification. And 84% of requests claiming to be GPTBot were identified as fake. Meanwhile, 99% of requests claiming to be Perplexity-User were identified as fake. And 94% of fake Googlebot requests copied Google’s actual crawler version string.

Publishers using their own methods to block stealth crawlers have mixed results. Earlier this year, Reuters adopted a block-by-default approach to bots to crack down on bot scraping, but Phil Andraos, gm for digital at Reuters, said on stage at the Digiday Publishing Summit in September that these efforts aren’t ironclad.

“We do our best to block crawlers and bots unless they identify themselves and we authorize them. We do our best to block those that are not authorized. It’s not always 100% foolproof,” he said.

People Inc. initially blocked known AI crawlers but allowed access for partners. When some crawlers hid or changed their identity to get around the blocks, the company found it couldn’t keep up by blocking them one at a time. After months of that back-and-forth, it switched to a default-block approach in January. “It has to be the CDN level. If you just have a list of like: ‘these are the names we can’t come through,’ that is as useful as a chocolate teapot. Robots.txt does nothing,” he said. 

For publishers, the concern is not only access to their sites, but who captures the value created by their content. People Inc. is determined to crack into the multi-million-dollar industry that has piggybacked on the back of the phenomenon of illicit scraping. 

Roberts argued that there’s strong investor appetite for businesses that package and sell access to online content, even as the publishers who created that content often receive little or nothing. “That is the broken dichotomy we’re living with at the moment,” he said at Digiday’s event. He cited Exa’s fundraising as an example of the imbalance, saying it raised $85 million at a $900 million valuation last September, followed by a $250 million Series C at a $2.2 billion valuation this May. “There are about 80 of these Napsters of content,” he said. 

“If I go to a venture capitalist and say I want to sign a publisher because AI uses good information, I will get laughed out there with a door slammed in my face,” added Roberts. “If I go to a venture capitalist and say ‘I want to steal the world’s information and sell it as a service; no, I’m not an AI company, but please can I have a quarter of a billion dollars’, they’re like, ‘where do I sign’?” 

For Roberts, while the current piracy is frustrating, the level of VC investment proves there is a “huge market and demand for content” that didn’t exist a year ago. “Now we just sort of wire it up. First step of wiring it up is do not give it away for free: have a block.”

For Roberts, blocking is the first step in stopping publishers’ content from being taken for free. But a publisher’s own defenses can only go so far when crawlers conceal or change their identities. That is the gap publishers want policymakers to address: replacing voluntary signals such as robots.txt with rules that require crawlers to identify themselves.

“I think it’s clear that many content rightsholders around the world are looking to implement a stronger framework than the current one, which is largely based on voluntary measures like robots.txt directives,” said Aaron Rubin, partner in the strategic transactions & licensing group at law firm Gunderson Dettmer. “There is certainly precedent for parallel movement on these kinds of digital-economy issues on both sides of the Atlantic. For instance, GDPR was followed by a wave of U.S. state privacy laws.”

What we’ve heard

“We urge you to back Damian Hinds MP’s Automated Online Software (Access and Transparency) Bill or introduce equivalent provisions in this parliamentary session. This is a narrow, practical measure that would require bots to identify who operates them and why they are accessing UK websites. With bots now accounting for more than 50% of global web traffic, such measures are vital to protect the open web, British interests, and creators’ ability to control and commercialize their content.”

– A joint statement published on Oct. 14 from top execs at major U.K. and U.S. media companies like Condé Nast and The Guardian, urging the British government to back legislation requiring AI bots to disclose their operators and purposes when scraping websites.

Numbers to know

36%: The year-over-year decline in Condé Nast’s U.K. profits last year, from £28 million to about £18 million.

15%: The percentage of Americans getting news from AI chatbots, up from 9% in 2025, according to Pew Research Center.

1 in 5: The share of U.S. adults who regularly get news on TikTok, according to Pew Research Center.

More than 400: The number of publishers worldwide participating in Google’s paid News AI pilot program. 

What we’ve covered

Forbes joins Google’s paid AI pilot as publishers seek value beyond the click

• Forbes has joined Google’s paid News AI pilot, which compensates publishers for expanded use of their content in experimental AI features.
• Beyond licensing revenue, Forbes hopes to gain insights into how its content performs across Google’s AI-powered experiences — data that could inform its own products and audience strategies – as referral traffic dips roughly 45% year over year.

Read more here.

The Verge restructures its audience team to prepare for the post-search era

• The Verge has overhauled its audience development team, shifting away from generalist roles focused on SEO, social distribution and referral traffic toward specialists in newsletters, platform engagement and audience analytics. 
• The goal is to turn casual visitors into loyal, direct readers as publishers prepare for a future with less Google search traffic.

Read more here.

Brands are betting on microseries, but the business model is still taking shape

• Brands are increasingly experimenting with creator-led microseries — short-form, episodic videos on platforms like TikTok and YouTube — either by sponsoring existing hits with built-in audiences or producing their own branded series for greater creative control and potentially lower costs. 
• But the economics remain uncertain. While some microseries can cost a fraction of traditional advertising campaigns, brands and creators are still navigating questions around ROI, intellectual property ownership and profit margins.

Read more here.

Reach touts Google AI licensing deal as a model while escalating pressure on OpenAI

• U.K. publisher Reach has struck a paid AI content licensing deal with Google, which CEO Piers North hopes will serve as a template for future agreements with AI companies.
•  The deal gives Google expanded rights to use content from Reach’s U.K. and Ireland publications in experimental AI features, while providing Reach with compensation and greater visibility into how its content is used. Meanwhile, Reach has sent OpenAI a cease-and-desist letter over unauthorized scraping.

Read more here.

Social commerce overtakes SEO at The Independent for the first time

• The Independent’s social-commerce revenue surpassed its SEO-driven commerce revenue for the first time in September, marking a milestone in the publisher’s efforts to reduce its reliance on search traffic. 
• While the gap remains narrow, CEO Christian Broughton said it signals a broader shift toward building revenue around trusted brands and audiences across platforms. The publisher is betting on four growth areas — AI, creator-led content, e-commerce and membership — which are expected to account for nearly half of its revenue in 2026, growing close to 30% year over year.

Read more here.

What we’re reading

Meta’s new terms still allow broad scraping for AI

Despite removing explicit references to AI training from its updated terms of service, Meta still claims broad rights to scrape publishers’ websites for its advertising and business products, raising concerns among legal experts that Meta could use that content to develop AI tools without publishers’ consent, Nieman Lab reported.

Axel Springer launches creator platform

Axel Springer announced it has launched a creator platform that allows journalists to build their own media brands and formats with greater editorial independence and a share of revenue, as the publisher looks to capitalize on journalists’ own brands and direct audience relationships.

The Wall Street Journal plans big streaming TV push

The Wall Street Journal is planning a major expansion into streaming video, exploring a slate of six shows featuring its own journalists and outside creators to attract younger audiences, grow ad revenue and expand its presence on YouTube and other platforms, Business Insider reported.

Bari Weiss faces internal backlash over CBS News reporting

CBS News editor-in-chief Bari Weiss is facing internal criticism over her hands-on involvement in reporting on Michigan Democratic Senate candidate Abdul El-Sayed’s alleged ties to the Muslim Brotherhood, with some staff accusing her of allowing her views to influence the network’s editorial coverage, Semafor reported.

BBC News channel tests streaming live on YouTube

The BBC News channel is trialling a live stream on YouTube for U.K. and U.S. viewers in order to reach more digitally-native new and younger audiences, the BBC reported.