Reuters and Time adopt bot-blocking whitelists to rein in AI crawlers

“We saw that there was an imbalance between the value that publishers like Reuters provide and the value… that Reuters receives in kind, and so instead we went from a default allow-all to a default disallow all,” said Josh London, head of Reuters Professional, who oversees the direct-to-consumer and direct-to-professional businesses. “Our content costs money to create. It has significant value, and the access to it, we feel, must be earned.”

Time now allows about 70 bots to access onsite content, according to its COO Mark Howard. Those bots range from crawlers run by big AI labs and social platforms, to the automated systems Time uses to keep its own website running. Time uses ScalePost to manage AI bots.

“Now we’re starting to think about: as the volume of bot traffic continues to increase significantly, and we see through a number of our vendor partners that we have very high domain authority with AI bot traffic, there’s value in that,” Howard said. That perceived value, he added, can help support the AI visibility product Time is developing to sell GEO insights to brand clients.

Reuters is blocking bots by default using robots.txt files. That method is far from foolproof. It tells web crawlers which URLs they can, but it relies on voluntary compliance — and many AI bots ignore it. A Tollbit report found that 30% of total AI bot scrapes in Q4 2025 did not abide by explicit robots.txt permissions.

But the point is to create friction, which is especially critical given the recent explosion of bot traffic publishers have seen, according to Alphonse Hardel, head of agency at Reuters, who leads the content licensing and solutions business.

“[Robots.txt] is non-binding, and so it’s something that is just a part of the funnel but it strengthens a message to say, ‘if you want this, let’s have a conversation and then we can allow you to access’,” Hardel said. “We don’t have resources to monitor everything around the wall. There are very many bots that exist today.”

That friction can act as a deterrent to AI bots scraping publishers’ sites. 

At an IAB Tech Lab event on May 28, Lindsay Van Kirk, People’s Inc.’s svp of innovation, said when the company moved from a bot “block list” to an “allow list,” People Inc. went from blocking roughly 2,100 user agents to over 30,000.

“It gives you a sense of just how big of a scale this challenge really is,” she said onstage. 

“There’s this sense that, ‘if there are all of these ways [bots are] getting around these blocks, why would I even do it? Why should I even try’? And I encourage everybody to… think about it as: ‘if we don’t put friction into this side of the economy now, it makes it harder for us to recapture value on the other side of it’. Friction is really the goal.”

Bot-blocking mechanisms, while not entirely effective, can slow down unauthorized scrapers and increase the cost of scraping, thereby incentivizing companies to come to the negotiating table, Van Kirk said.

That’s worked for Reuters before, according to Hardel. Reuters has AI licensing agreements with companies like Microsoft and Meta.

“Adding two full seconds of latency to the majority of scrapers when you implement a block-all-bots approach is a really good thing, even if they have to go through. Every scraper who has to pay a home proxy network in order to get access to the content is margin that you are taking out of their business. That is good for publishing,” Van Kirk said.

Also, by publishing a publicly available approved-bot list through robots.txt, it creates a benchmark for spotting unauthorized scraping of Reuters content and supporting discussions around enforcement, said Phil Andraos, general manager of Reuters Digital.

Reuters is also working with bot monitoring tools, server-level enforcement, commercial licensing arrangements and legal protections, London said.

“There’s no perfect solution here, but there is value in every little bit of friction that you have that forces people to have good behavior,” Hardel said. “This is just one other thing we can do to coerce people to have the right behavior.”

The bots allowed through must offer a “fair value exchange,” London said. That boils down to four categories: licensing, traffic, site function and monetization. If a bot helps drive traffic to the site, that can help Reuters’ subscription and advertising businesses, according to Andraos. Other bots are necessary to support Reuters’ ad tech stack, like those owned by demand and supply side platforms.

So far, the change hasn’t led to a decline in site traffic, London noted. After monitoring bot activity for the past two years, Reuters had gathered enough data to make a list of which bots Reuters could block without impacting revenue, Hardel said. Reuters’ robots.txt file shows the publisher is allowing some bots from Amazon, Google, Bing, Yahoo and ChatGPT’s user agent to access its site.

“It’s not a set it and forget it approach. It’s something that needs constant care and feeding, as the industry and bots themselves evolve,” London said. “The value of content is something that we ignore at our own peril, especially as AI scales. It’s important for us and for the industry to safeguard that.”

There’s also a cost consideration to now blocking “hundreds” of bots, Hardel added.

“It is also saving us quite a bit of cost from serving the content bots. That’s an important consideration. The cost of the bot-blocking vendor can be almost covered by the decrease in non-human traffic that you experience when you stop that,” Hardel said. “It was high enough that it had a positive impact on our cost of serving the site.”

The IAB Tech Lab published a bot blocking guide last month to help website owners make these assessments, which is especially important when testing a block-all-bots approach can trigger unwanted consequences. The SPUR Coalition, a publisher group formed earlier this year with major news organizations, announced it added 30 members last week to create technical standards for AI licensing and content protection.

“When we moved to our approach, we broke all of our email syndications for about five minutes, because we forgot that they all do live link checking before they actually publish something,” Van Kirk said.

The growth of the third-party scraper economy bubbling up beneath big AI companies is making it harder than ever for publishers to know who is taking their content, let alone stop them. 

“There are companies that are out there who don’t disclose themselves…and they’re doing that for an economic benefit on the other side of it…” Van Kirk said. “So I think a huge part of this discussion isn’t just about who blocks or not… it is, someone is stealing from you and reselling it. How do you think about that as opposed to whether or not you want to show up in ChatGPT? Those are two fundamentally different questions.”