Which Facebook metrics flub are you? Ad buyers rate Facebook’s 10 measurement errors

Facebook’s latest error stems from a bug that led it to miscalculate how often users clicked through its carousel ad units, which let brands show multiple images and link to their properties. Facebook declined interview requests for this story.

The error comes during the upfronts, which TV networks are using to push back against their digital competitors. For example, during a Fox presentation on Monday, Family Guy creator Seth MacFarlane made a joke about YouTube ads “shown on ISIS vids.” Buyers said the error will give networks ammo for their presentations, but it won’t impact budgets.

“I’m sure there will be keynotes at the upfront who bring this up,” said an ad buyer requesting anonymity. “But Facebook has such scale. As long as people keep visiting the platform, advertisers will still have an appetite for Facebook regardless of what gets said this week.”

Facebook claims that the error, which only occurred within the mobile web, affected just 0.04 percent of its ad impressions. But it went undiscovered for nearly a year, and Facebook is refunding affected advertisers, which it hasn’t done after its other errors.

“If there are mishaps, we want to address them with Facebook,” said Jake Rosenberg, senior digital media strategist at Red Tettemer O’Connell + Partners. “It is better [for Facebook] to address it head on and control a narrative because it works in their best interest to call it out and say this is what we’re doing to fix it.”

Digiday asked buyers to anonymously rate each of Facebook’s measurement errors on a scale of 1 to 10, with higher numbers representing more egregious errors. The errors are ordered by recency.

Wrongly charged advertisers for video carousel clicks, reported in May
A bug caused Facebook to miscalculate how often users clicked through its carousel ads. But the error applied to just 0.04 percent of the platform’s impressions, according to Facebook. One ad buyer called the carousel units “fringe.”

Rating: 3.3

Undercounted iPhone traffic for Instant Articles, reported in December
From Sept. 20 to Nov. 30, Facebook was undercutting iPhone traffic for Instant Articles publishers who had not yet moved to HTTPS protocol. “This is more detrimental to publishers,” said an ad buyer. “It is not much of a concern of ours.”

Rating: 4.5

Miscalculated likes for Live videos, reported in December
Users were only supposed to be counted once whenever they engaged with a video, but people who responded with multiple likes to shared videos were counted multiple times. “While reactions are a nice metric to have, they are not the base of success for any of our campaigns,” said an ad buyer.

Rating: 4.2

Inflated like and share counts for links posted on Facebook, reported in December
Facebook miscalculated how often users reacted to Live videos and how often users liked and shared links posted on Facebook. “The errors were ridiculous by this point, but the metrics are fluff,” said an ad buyer.

Rating: 3.1

Overestimated referral traffic, reported in November
Rather than only measuring how often Facebook users were driven to an advertiser’s own properties, Facebook also counted clicks that were driven to the advertiser’s Facebook page. “When they got referral traffic wrong, they messed up metrics that really matter to our clients,” said an ad buyer.

Rating: 5.3

Inflated average time spent on Instant Articles, reported in November
Facebook had trouble properly calculating how much time users were spending on Instant Articles per visit, which led the platform to overestimate average time spent by about 7 percent. “It is a value metric for publishers, but it doesn’t hold much weight for us,” an ad buyer said.

Rating: 4.3

Overstated video ad completion rates, reported in November
Facebook didn’t properly the sync the length of its video ads, which led the platform to undercount “video watches at 100 percent.” One buyer said, “If you were running video ads, you severely misled.”

Rating: 5.2

Organic users counted multiple times, reported in November
Facebook counted users multiple times when they were viewing a page’s organic posts. For some Facebook pages, this went on for 28 days. Buyers said they were not affected by the error.

Rating: 3.3

Organic users counted before content was on screen, reported in November
While already counting users multiple times while they viewed organic posts, Facebook also counted users before the organic posts appeared on their screens. Like the other organic-page error, buyers were not affected.

Rating: 3.7

Inflated video views, reported in September
For two years, Facebook overestimated the average time users spent watching video on Facebook. In some cases, Facebook’s numbers were 80 percent off. “In terms of the magnitude of the discrepancy, no platform should be that off,” said an ad buyer.

Rating: 7