WTF is dark traffic?

So WTF is dark traffic?
For the most part, publishers and marketers can tell where their Web traffic is coming from, be it direct or through search engines or a social network like Facebook or Twitter. But then there’s all the under-the-radar sharing that takes place — on chat, messaging apps, IM and in email — outside the mainstays of the social network ecosystem. Then there are sites that are secure, which means that they don’t collect information on users. That’s the dark traffic part. “There’s a general move to people going to secure [sites], and as a general rule, when people go to secure, referral data is lost,” said Danny Sullivan, founding editor of Search Engine Land.

OK. So how much of Web traffic is dark?
Impossible to say for sure, because it varies so much by publisher. The Atlantic, which is credited with coining the term dark social back in 2012, estimated that it doesn’t know where as much as 40 percent of its traffic comes from, up from that 25 percent figure a year ago. The Guardian, which has been particularly revealing about its dark traffic, estimated its unknown traffic at 7-8 percent. That has grown in the last year, especially as mobile traffic has risen. It’s a safe bet that the more a publisher is reliant on side-door traffic, the more of its traffic is dark. Chartbeat estimates that about one third of traffic to article pages these days falls into the “dark social” bucket. That number climbs to around 45 percent on mobile. All that means that traffic that’s been counted as coming from direct, social and search isn’t as high as we thought it was.

Is this a new thing?
Email has long been a source of dark traffic, as URLs shared through email don’t come with referral data. But dark traffic has been on the rise, for a few reasons. There’s the growing use of messaging apps like WhatsApp. The rise of privacy concerns has led sites like Yahoo Search and Google News to make traffic anonymous to protect users’ privacy. Traffic from search engine Duck Duck Go, while tiny compared to Google, also doesn’t send referral data (giving rise to the term “dark search”). “For as long as we’ve had email, there’s been an attribution problem,” said Danny Wong, marketing manager at Shareaholic. “The way communication has changed makes it a bigger problem for businesses and publishers.”

He said “problem.” Why is this a concern?
In the wake of scary NSA revelations, the ability for consumers to read and share without leaving footprints is seen as a good thing for user privacy. But publishers and marketers that don’t want to tread on users’ privacy still want to know as much as they can about where visitors are coming from so they can learn what appeals to users and give them more of it. “The main problem is not really knowing where your audience is coming from, so you’re not understanding your audience as well as you’d like to,” said The Atlantic’s Cohn. “Are they loyal readers? Avid users of social media? Meaning, it’s working in a sharing economy. Those are both good things. But the trick for publishers is to know where your traffic is coming from.”

So where is it coming from?
As people shift their media consumption to mobile from desktop, dark traffic will likely only get bigger. Mobile traffic is more likely to be socially driven, and social is the traffic source that’s most affected by dark traffic. Dark also can affect the sharing of links through certain mobile apps, which may not send referral data. Visits that come through Facebook’s mobile app do not consistently show up as being a Facebook referral, which underestimates the impact of Facebook on mobile, the Atlantic’s Cohn said.

What can publishers do?
They already tag links that go in their email newsletters, so a publisher can tell if a visit came via a newsletter link as opposed to another source. But the tools are limited. They can watch to see if, say, a spike in mentions of their brand on a dark social platform like Instagram correlates with increased traffic to their site, but causality is hard to determine. BuzzFeed once put a WhatsApp share button on its articles, which would tell the publisher how many people clicked (although not how many people they shared the article with, or if they completed the sharing action at all).

Is anyone doing it right?
One publisher that seems to be getting a handle on the issue is the Guardian. It has an in-house tool, Ophan, that pinpoints where traffic is coming from on a minute-by-minute basis. By overlaying data from dark social sources with traffic to its own site, the Guardian can infer the relationship between unknown traffic and specific referrers. So, for example, if a story starts to rise in popularity on a subreddit, but traffic to the Guardian from reddit itself didn’t spike, the Guardian could infer that the traffic is coming from a third-party reddit app that doesn’t send referral traffic to the Guardian. “Thanks to this tool, we are growing our understanding of how unknown traffic mirrors specific referrers,” said Tanya Cordrey, chief digital officer for the Guardian.

Where do the big platforms sit in all this?
As publishers’ sideways (non-homepage) traffic increases, their distributors would seem to have an incentive to provide as much referral data as possible, pointed out Josh Schwartz, chief data scientist at Chartbeat. “Without that data, it’s hard for a publisher to care as much about Google News if they can’t determine the ROI.”