‘Levers being pulled that are unseen’: Measurement errors inside Amazon’s OSP program setting publishers on edge

While some participating publishers have continued to see revenue from OSP trend upward overall, many have seen it decline precipitously. Sources at two publishers participating in OSP said their revenues had declined so much that they were considering abandoning the program altogether. A third said they worried that they would have to abandon it next year.   

“The future of OSP’s pretty murky for us,” said a source at a fourth publisher. “Amazon provides enough reporting on the affiliate side [with Amazon Associates] where you can hire staff, forecast performance, you can follow best practices. With OSP, it feels like there are levers being pulled that are unseen by the content creators.” 

Reached for comment, Amazon sent a statement in lieu of responding to a set of questions Digiday sent. 

“At Amazon, we are constantly experimenting with our programs to both create the best shopping experience for customers and to help publishers find success on Amazon,” the statement read. “Onsite Publishing is still in an invite-only phase and we will continue to test and learn with how we are sharing this content with customers —testing new placements across Amazon, making it even easier for customers to discover great content and introduce publishers to a new audience. Our goals for the program are aligned with the goals of our publishers, and we’re committed to helping them succeed.” 

A loss of standing

While all affiliate programs are at least somewhat dependent on the retailers offering the commissions, OSP requires the publishers participating to invest a lot of faith in Amazon. OSP allows publishers to distribute product guides through Amazon’s site and app, then earn commissions if a person who reads those guides buys one of the featured products. 

Though publishers can house all their guides on profile pages inside Amazon’s properties, the overwhelming majority of the guides’ views come from Amazon search queries; for most of the past two years, a person who entered a search query such as “best air fryer,” for example, or “best pillows,” would likely find excerpts from a guide written by a participating OSP publisher among the first few search results. 

Like any digital media company, Amazon regularly tinkered with what it put in its search results. Those tests sometimes had catastrophic results for OSP publishers, who said Amazon rarely warned them that tests were coming, though sources contacted for this story said it was perfectly reasonable for them to conduct them. “They wouldn’t be doing their job if they weren’t trying to optimize,” a sixth source said. 

But this spring, many publishers noticed Amazon started to lean into one experiment that they found especially troubling. Instead of showing publishers’ guides in many search results, Amazon was instead showing users products that Amazon itself recommended, or products that were highly rated by Amazon customers. 

Those recommendation widgets, which featured many of the same design elements as publishers’ guides, felt like an ominous step toward an outcome many participants had feared since the program started: That Amazon, after using publishers’ content to learn how content might compel a user to purchase something, would simply cut publishers out of the equation. 

“They’ve basically replaced most of the OSP [content] I could see with ‘Highly Rated’ [results],” said a source at a fifth participating publisher, whose revenues from the program have fallen by close to 90% from their peak.

Opinions differ on just how much share of search results OSP content has lost. But beyond the loss of revenue, the change has set off a wave of suspicion among several participants that Amazon could be using the data created by OSP conversions to inform which items they are recommending to consumers. Amazon has, in several different instances, taken data generated by sellers and other partners and used it to spin up features or products that directly compete with those offered by those sellers and partners. 

“There are not industrial-strength MSAs [master service agreements] governing the use of [OSP] data,” the third source said. 

A change in standards

But Amazon’s support of its own recommendations was especially galling to some publishers in light of a separate change it began making in the spring, when it told publishers they would have to make changes to their guides. 

In its earliest incarnation, most of the content inside OSP came from well-known, heritage media brands, much of it lightly edited versions of content they’d already published on their respective sites. But within a few months, a new crop of publishers had come aboard, many of them much more obscure, some of questionable provenance and making recommendations based on much less rigorous evaluations. 

“They did not go out to try and achieve an environment that rewards great content; they went out to drive sales,” said a sixth source. “On the one hand, they expected the quality to go down [when they let smaller publishers in]. I don’t think they anticipated how quickly people would push the boundaries.”

Earlier this year, OSP participants got a big dose of guidance. In late spring, Amazon began telling participants that it was changing the guidelines and list of attributes that each guide needed to include. 

To several sources, the guidance was welcome, but the timeframe was not: Many were told they would have 30 days to get their guides, which in some cases amounted to hundreds of pieces of content, into compliance. 

“This is hundreds of articles. This is stuff that’s [been] previously approved,” the second executive fumed. 

Many made the changes, not just to avoid a significant drop in revenue. but because some of the changes would might lead to better performance. For some, that didn’t bear fruit.

“We never saw material improvement from those changes,” the third source said. “There was a bitter taste from that.”