The subscribe page has become a laboratory for news publishers experimenting with revenue stream options

But unlike the A/B tests that have become a fixture in an audience development team’s workday, the tests require input from many different stakeholders, including marketing, audience development, product and sometimes even editorial.

They also take longer to measure. While an audience team might be able to conclude which headline/image combination is more effective within a matter of minutes, it can take weeks, or even months, to figure out which subscribe page decision drove more conversions — and how many of those subscribers churned out.

And while some of the tests are cosmetic, around color or placement of buttons, most are grounded in hypotheses built out of user insights or audience feedback about what they want, which needs to be gathered continuously from readers.

“Some of it is, ‘You want the buy now button bigger,’ but it’s more getting to those customer insights,” said Matt Nowen, the director of customer experience practice at House of Kaizen, an agency that helps clients optimize their subscription products.  

For a while, a news publisher’s subscription pages were often adjusted or updated using less than rigorous criteria. “Those tended to be sort of one-off projects,” said Erik Zenhausern, the director of subscriber acquisition and retention marketing at Newsday. “Someone [in the office] would see another newspaper’s site and say, ‘We should be doing that!’ So we’d spend two or three months redesigning it, and that would sit there for six months until someone saw something else.”

Today, Zenhausern said, a cross-functional team at Newsday has a pipeline of A/B or multi-variate tests running on its subscriptions page, along with other surfaces including its paywall; the results of one test will determine which experiment is conducted next. The team that works on it includes members of Newsday’s audience development, product, user experience and marketing groups.  

This has been a good time to run tests. Publishers have seen record-breaking subscriber growth over the past several months, thanks to news-hungry consumers spending more time on publishers’ sites. That influx of visitors means tests that might have taken weeks to run have, over the past several months, been completed in a matter of days.  

But even with traffic up year over year, the pace of testing for subscribe page changes can vary widely. While publishers typically begin subscription operations focused on customer acquisition, most news publishers are now focused on retention, which affects how long it takes to determine impact.

If a publisher tests something brand new, such as a new offer, the results need to be measured over much longer periods of time, to account for how well the publisher retains any subscribers converted with the test.

“In the early days of optimizing commerce flows or shop pages, it was about, ‘Can you get a better conversion rate if you change something?’” said Karl Wells, the gm of membership at the Wall Street Journal. Today, he and his colleagues evaluate changes they make based on their effect on the Journal’s projected annualized revenue, or PAR; a change that attracts lots of customers who churn right out of their subscriptions makes less sense than a change that attracts fewer customers who stick around and renew.

For tests around things that have proxy, such as a change in price, publishers that have accumulated retention data can draw conclusions much more quickly. “In the old days we’d have to wait six months to get a read,” Wells said. “But now we understand the early signals of good retention.”

For now, the changes publishers are testing on their sites are being seen by everybody. While more news publishers have begun experimenting more with tailoring or personalizing the content that their readers see, few have begun showing their visitors different subscribe pages or offers.

“You get to segment-level package and pricing when you’ve exhausted all the low-hanging fruit,” Wells said.

“If you’ve spent five years on it and you can’t squeeze any more juice out of the orange, you’d say ‘We’ve done enough,’” Wells said. “But we haven’t reached that point yet.”