WTF is a stealth crawler?

The timing tracks. Bot traffic is no longer a fringe problem. Cloudflare data shows more than half of all web traffic is now bot based. And a growing share of that is crawlers pulling content off publisher sites without identifying themselves or their purpose. Publishers are left in the dark on both counts: who’s taking their content, and what it’s being used for.

WTF is a stealth crawler?

A stealth crawler is a web crawler that scrapes content from sites without identifying itself or its purpose. 

These crawlers often ignore or circumvent robots.txt files (which websites use to communicate to crawlers what they can and cannot scrape) that prohibit scraping or scrape sites through third-party services. A publisher might see the visit from a stealth crawler as an ordinary site visitor. In other words, the crawler won’t self-identify and mirror browsing patterns that make it look like a human visitor. 

While not all stealth crawlers are nefarious, it’s very hard for publishers to weed them out. And that makes it difficult for them to be stopped, since publishers can’t protect themselves from something they can’t see.

How widespread are stealth crawlers?

As Digiday previously reported, a third-party scraper economy is emerging. Media analyst Matthew Scott Goldstein’s report on the “scraper economy” argued that this has developed into a $1 billion industry. Danielle Coffey, president and CEO of the trade organization News/Media Alliance, said she’s seen data that makes it seem more like a multi-billion dollar market.

Either way, these crawlers are growing at a clip. Cybersecurity company Human Security found AI scraper traffic grew 597% from January to December 2025, and AI-driven traffic overall grew 187% in 2025, nearly tripling year over year, per the report. Scraping attacks now affect nearly 20% of site traffic for the median organization, almost double 2022 levels.

How are these crawlers able to keep their identities hidden?

Stealth crawlers can mask their identities by using generic browser agents or cloud infrastructures instead of identifying themselves as AI crawlers. For example, a stealth crawler can present itself as a normal Chrome browser or use a residential IP address.

By not identifying themselves, stealth crawlers make it difficult for publishers to know if the traffic is coming from an AI company or another intermediary that collects data to then resell it.

Lindsay Van Kirk, People Inc’s svp of innovation, spoke onstage at an IAB Tech Lab event in May and outlined how some scrapers are getting through publishers’ attempts to protect their content.

“We’ve seen things like our content showing up in applications where we do not have a relationship with an application, and it sort of begged the question to us: how are you getting access to that? How are you taking this information, repackaging it, and sending it through to a consumer? And the answer is through these unauthorized usages, most of which default through to home proxy networks as their last fallback solution,” Van Kirk said. “We see how they’re getting through, and so we feel like that’s impacting our brand, it’s impacting how consumers are seeing it.”

Why is this bad for publishers?

When a crawler identifies itself, publishers can see which crawlers are scraping their content and then decide whether they want to block them. But if the crawler masks its identity, publishers are shooting in the dark. They can’t see which AI companies are interested in their content, and can’t attribute traffic to that source.

It also goes back to the whole conversation around publishers trying to protect their content from being scraped and used without permission or compensation – an issue that has kicked off a number of AI lawsuits and licensing deals. Many publishers don’t want their content used for free to train AI models or be sold through a secondary market that they aren’t benefitting from. 

How are publishers trying to wrangle these stealth crawlers?

Publishing execs have told Digiday their servers have been overloaded with millions of bot scrapes over time. Van Kirk said People Inc. went from blocking roughly 2,100 user agents to over 30,000 when it adopted a block-all bots approach. That amounts to tens of millions of scrapes a day. At the IAB Tech Lab event, she said this strategy can help publishers in the near term, even if stealth crawlers can get through.

“If we don’t put friction into this side of the economy now, it makes it harder for us to recapture value on the other side of it. Friction is really the goal. The slower that you can make any of these scrapers for hires, the crawlers who are reselling, the folks who are not using it for legitimate purposes – the more effective we become at being able to not only provide value on the other side that has some economic value and attribute to it, and it impedes the user experience in such a way that it will bring parties to the table,” Van Kirk said.

“Adding two full seconds of latency to the majority of scrapers when you implement a block-all bot approach is a really good thing, even if they have to go through. Every scraper who has to pay a home proxy network in order to get access to the content is margin that you are taking out of their business. That is good for publishing,” she added.

How is the U.S. cracking down on stealth crawlers?

New York passed the Stealth Crawler Prohibition Act in June, which requires bots to disclose their identity and purpose. The bill defines a stealth crawler as any software that retrieves, scrapes or accesses a website, including AI agents. The New York attorney general’s office would be able to sue companies that fail to disclose this activity, with civil penalties that could reach $15,000 per day “for each violation,” according to the bill.

That was the precursor to the bill that was introduced to the U.S. House of Representative last month. News/Media Alliance has been actively lobbying for these measures.

“Wagging your finger isn’t going to work. They’re bad actors. They behave as such. They’re hiding themselves for a reason. So you have to nip it in the bud. You have to start it at the top first line of defense: Make them identify themselves,” News/Media Alliance’s Coffey said. “Bad actors, bad bots must identify themselves, and then when they do, we can stop them. And if we can’t stop them, or even if we stop them, we can sue them… The AG and FTC have enforcement authority to hold them accountable.”

The laws would act as a deterrent to bad actors, she said. Goldstein believes these regulations would set a foundation for publishers’ licensing agreements with AI companies.

“Transparency is not the endgame, it is the precondition. Every licensing conversation I sit in stalls at the same place, which is that publishers have no reliable way to prove what was taken, by whom, at what volume. Identification changes the negotiating table. Once a crawler has to say its name, scraping stops being free and starts being a line item. The companies with the most to lose would rather keep the meter unreadable,” Goldstein wrote in a LinkedIn post.

What happens now?

The New York bill has to be signed by Governor Kathy Hochul before it becomes law. The next step for the bill in Congress is to attract sponsors. 

The process could take a while, Coffey predicted. However, she believes these laws will help publishers gain the visibility they need by cracking down on stealth crawler activity.

“It’s like if you pull up your roof and you have termites,” Coffey said. “You have to dig in sometimes and you have to do the basics.”