WTF are shadow blocklists and how they can take a bite out of publisher revenue

WTF is a shadow blocklist?

The first thing to know is that shadow blocklists aren’t a literal list. They are a catch-all term to describe those instances where a buyer — in a bid to avoid funding rogue publishers — uses algorithms to do so without letting the publisher know the reason. A publisher either knows that a buyer has stopped bidding on their inventory but doesn’t know why or they don’t know it’s happening because the ad tech vendors they sell through aren’t flagging the issue. Either way, publishers lose money. 

Tech site XDA Developers recently saw revenue drop 30% overnight when Google’s demand-side platform DV360 stopped buying ads on its site, said Boris Shterev, head of marketing at PubGalaxy, the ad management firm the publisher hired to fix the problem. The investigation started in October and it took five months of trial and error before Google’s DSP started bidding again in February. Even then, the investigators couldn’t say for sure what they did to get off the shadow blocklist, said Shterev.

Why are shadow blocklists a tough nut to crack for publishers?

Many publishers aren’t close enough to demand-side platforms to ask why they’ve stopped bidding on inventory. Getting those answers isn’t straightforward, as XDA Developers’ execs discovered. “Google may own both sides of the marketplace, but they’re separate entities so a publisher can’t just go to the side they sell ads through and ask why the demand side isn’t working,” said Shterev. 

And even if publishers were able to talk to both sides of Google’s marketplace, there’s a chance that they still wouldn’t be able to. “Google doesn’t want to give out the algorithm for how their systems work because people would find ways to cheat it,” said Shterev. 

Shadow blocklists aren’t just a Google problem, however. It’s just easier for publishers to spot when Google does stop bidding on their ads because it accounts for a large slice of their online ad revenues.

“It’s why whenever we’re analyzing drops in performance for publishers we’re looking at all the different factors that could impact it because sometimes the impact [of shadow blocklists] isn’t visible,” said Mirela Kolarova, manager of yield and account management teams at PubGalaxy.

What should I do if I find my ads are being inexplicably blocked?

The first thing to do is to check that the site isn’t doing anything visibly wrong or nefarious to win ad revenue. Its a long and arduous process that saw PubGalaxy testing a medley of theories, from analyzing bot traffic coming to the XDA Developers site, to getting agencies to bid on traffic in order to see how the pages were being labeled by Google’s DSP.

Eventually, the investigators found what they think caused the problem. The two companies that managed the affiliate and comment aspects of the site were also serving ads on it as part of the deal. That meant that whenever Google — or another ad tech vendor — decided not to bid on ads sold by those businesses because the page contained objectionable content they were sent a notice. Usually, those notices are passed on to the publisher. But neither the affiliate or comment companies usually help sell publisher inventory and so hadn’t thought to pass those notices on to XDA Developers, said Kolarova. They could have been receiving those notifications “for years” and not know it, she said.

“Once the publisher was aware of the page violations they resolved everything,” she said. “We had a day of no page violations and then after that, we got our demand back. 

Start with who is selling

It’s a problem that could be stunting revenue for many unsuspecting publishers. In fact, there are two other publishers working with PubGalaxy on similar investigations, said Kolarova. Other ad tech execs say the XDA Developers investigation spotlights concerns on the publisher’s side. The term is new, said one exec, but the problem is a “real thing.”

“Ultimately, the story of XDA was over monetization and a lack of governance and controls on who is selling,” said John Donahue, CEO at media consultancy WLxJS. “If a publisher is concerned over who is buying them, they should start with who is selling them and place governance on that.”

Should advertisers care?

It depends on how much an advertiser cares about the impressions they’ve won and lost. Most just aren’t close enough to those impressions for shadow blocklists to be a concern now. As Shterev explained: “I don’t think many advertisers really know how to evaluate the sites they’re buying from. They rely on platforms to do it for them.”