‘Value is a proxy’: Publishers are soliciting feedback from readers with one-question surveys

The asks for direct feedback, along with more established efforts like The Athletic’s, are part of a yearslong transition away from scale-oriented metrics such as pageviews or unique visitors in favor of numbers that measure the scope of the audience’s engagement with their content. Today, most of that measurement is still done in the background: measuring things like how long site visitors spend on pages, how frequently a story is shared, or whether it converts readers into newsletter subscribers.

But volunteered feedback is different enough that it needs to be gathered and treated differently. Some publishers such as Axios are funneling the data to their product teams, hoping to set a baseline that can be used to see a trend line in how people regard what they make. Others, such as The Athletic, use reader feedback about stories to help figure out how to distribute or package content on a personalized basis.

But in all cases, the feedback publishers solicit from readers is being thrown into a larger stew of metrics. Though a Net Promoter Score might be easy to understand, or a list of the stories readers deemed most valuable is easy to compile, all of the publishers using user-provided feedback say that they cannot use it as a standalone signal that shapes product, editorial or audience decisions.

“I think [this trend] makes sense with revenue, but you’re not just trying to attract drive-by traffic anymore,” said Gwen Vargo, the director of reader revenue at the American Press Institute. “Value is a proxy of if someone’s going to pay for it or not.”

“Because it’s a singular number, it’s easy to understand, but you don’t want it to become the new pageviews of 2019,” Vargo went on, talking about Net Promoter Scores. “You don’t want them to be not looking at other things.”

Though the one-question surveys mentioned above bear superficial similarities, the language of each of publisher’s message differs in a meaningful way.

BI Prime members are asked how “valuable” the story was, while Athletic readers are asked, “What did you think of this story?” Back in 2015, Mic readers that got to the bottom of stories saw a widget that asked, “Was this story worth your time?”

That people were identifying with the icons’ faces made the feedback less valuable for its intended purpose. But the replies remain one of the signals The Athletic uses in its strategy. This year, the site began using the data to figure out which kinds of stories specific readers like, then serving more of those kinds of stories to subscribers using newsletters or mobile push notifications. A reader that prefers biographical back stories on athletes, for example, might be sent more of those than a newsier or more analytical item, using their story feedback as a signal.

The key, Hansmann said, is to measure and analyze lots of signals at once, including comments, scroll depth, time spent, shares and other things. Finding the balance, Hansmann said, without offering specifics on the Athletic’s process, is the key.

“If you over-optimize for one variable, then everything becomes the same,” Adam Hansmann, Athletic co-founder, said. “If you over-optimize for raw clicks, then you get a pile of mock drafts. If you optimize for emoji, then you get positive backstories. No one piece of data is the end-all.”

In the case of BI Prime, the purpose of its survey is to measure what readers find valuable and to reinforce the message that their input, along with their satisfaction with the product, matters.

Claudius Senst, the head of consumer subscriptions at Business Insider, said BI will take time gathering data before using it to inform any product changes. But it has already gotten one unexpected benefit out of it. BI Prime readers who answer are given an anonymous feedback form where they can leave additional details, and many respondents have used that field to share questions that the story they read did not answer. That has yielded several ideas for followup stories, Senst said.

Ad-supported publishers have tried to incorporate that kind of feedback into their strategies too. Mic first put a one-question survey at the foot of stories in 2015, and within a few months of deploying it, people discovered there was little correlation between how valuable the story was deemed and how many pageviews it got, or how often it was shared, one former employee said.

That seemed important enough that a small team began trying to figure out a new aggregate metric that could be shared with Mic’s newsroom, a combination of pageviews, share counts, how valuable the content was deemed. Persistent pressure to grow revenue made it hard to make the project a priority, and a metric was never ultimately shipped, that source said.

As publishers focus more on concepts such as consumer journey or consumer experience, Vargo said she expects publishers will work to gather more of these kinds of insights from their audience. They will have to do so with a light touch. “I always say, ‘Don’t make them work too hard,'” Vargo said.

An earlier version of this story said that BI Prime users who responded neutrally were given a chance to follow up. All respondents are given that opportunity.