For nearly a year, advertisers using Facebook’s conversion lift studies tool were unknowingly being fed faulty data.
A code error caused the reached conversions metric to be undercounted for conversion lift tests conducted on Facebook by several thousand advertisers between Aug. 15, 2019, and Aug. 31 of this year.
Advertisers use conversion lift studies, a free tool provided by Facebook, to measure the incrementality of their Facebook ads and make decisions about their Facebook spending. For affected advertisers, this information was erroneous.
Facebook fixed the bug on Sept. 1, and is “working with impacted advertisers,” a Facebook spokesperson told AdExchanger.
But Facebook didn’t alert buyers to the issue until last week.
This is far from Facebook’s first measurement-related misstep, which include a high-profile case of miscalculated video metrics in 2016 and an overestimation of its reach the following year.
What happened?
Specifically, the bug was caused by a data pipeline migration in August of last year that affected the way in which certain impressions were logged into Facebook’s conversion lift systems.
As a result, Facebook undercounted the number of conversions from people who were exposed to impressions on Facebook apps.
In other words, Facebook miscalculated the number of sales that came from people that saw an ad, which is a key ratio necessary to measure incrementality, because it’s used to calculate other metrics, including conversion lift percentage, which is the difference in conversions between the people who did and didn’t see ads during a test.
A token
Facebook is offering makegoods in the form of ad credits – or, in Facebook’s parlance, a “one-time coupon” – that must be used by March 31, 2021, but only for advertisers that were “meaningfully affected” by the data pipeline migration issue.
The coupons are being handed out on a case-by-case basis depending on an advertiser’s use of the conversion lift product and how much the bug impacted the results of their conversion testing.
Advertisers that were determined by Facebook not to have been meaningfully affected by the error aren’t getting anything.
According to a FAQ appended to an email AdExchanger obtained that was sent by Facebook measurement reps to affected advertisers last week, Facebook arrived at the specific amount for each coupon by “evaluating the potential incremental spend following study results,” taking into account how “incremental conversions may have shifted as a result of these issues” and how this may have affected marketing efforts on Facebook.
But without actually talking to the advertisers themselves to understand their marketing strategies, and it doesn’t appear that Facebook did, “I’m not really sure how their data science team would actually know that,” an agency representing an affected advertiser told AdExchanger.
This lack of transparency is par for the course for Facebook’s lift conversion tool, which has a reputation of being a bit of a black box, said the executive, who was actually more flummoxed by the “token” amount of the coupon received by one of the agency’s clients – tens of thousands of dollars back for an advertiser that had conducted multiple conversion lift tests and spent millions of dollars on Facebook media between August 2019 and August 2020.
Although Facebook’s conversion lift studies tool is free, the outcome of these studies often directly influences how much money an advertiser spends on Facebook.
“If you’re providing a tool for free as a media company and your data is being used to buy ads, then your liability should be the cost of the media – not what you decide the measurement errors are worth,” the executive said. “In this case, the tool is free and so I guess you got what you paid for.”
Sorry ‘bout that
According to the email Facebook sent affected account holders last week, the delay between discovering the conversion lift issue in late August and mid-November, which is when it gave advertisers the heads up, was because Facebook wanted to do its “due diligence to better understand the scope of the issue” and assess the impact on individual studies and metrics.
But that argument isn’t flying with some buyers, even though the bug didn’t directly relate to a billable metric.
“People continued to spend money after Facebook knew there was a bug, because they weren’t told,” said one account holder who received Facebook’s apologia email and was deeply frustrated by its “vague and corporate tone,” but asked to remain anonymous.
“The subject line of the email was very Facebook,” the account holder said. “It wasn’t, ‘Oh, my god, we effed up and we’re sorry!’ It was ‘Upcoming update to Facebook conversion lift studies.’”
During Facebook’s investigation into the original bug, it also came across and patched two additional, separate and, it says, “smaller” technical issues that affected conversion-based metrics for some lift tests.
The first was to do with the winning probability between Sept. 21 and Nov. 13 for multi-cell studies, which is when a target audience is split into multiple cells, each with its own test and control groups.
A second glitch caused the undercounting of the data that informs Facebook’s statistical models, which led Facebook to underestimate web conversions for some studies that started by June 15 and ended by July 26.
Facebook says that since uncovering the bugs, it’s implemented “additional measures” to better monitor and detect issues with its lift measurement systems going forward.
