“Data-Driven Thinking” is written by members of the media community and contains fresh ideas on the digital revolution in media.
Today’s column is written by Dan Hill, senior data scientist at Integral Ad Science.
How effective was the last ad campaign you worked on? What was the return on investment?
Chances are you don’t know. It’s all too common to leave ROI performance unmeasured. Advertisers often have no idea whether their budget was spent wisely or if it was even profitable to advertise in the first place.
Attribution providers can help answer these questions. They’re paid to estimate the effectiveness of ad campaigns. Each attribution provider has its own proprietary model for how to divide up credit for every conversion to the ad impressions that touched it. The most famous of these models is called last-touch attribution, where all credit is given to the last impression that the customer saw before converting. More advanced models use sophisticated sets of equations to assign credit along the entire path that the customer takes through the campaign, from touchpoint to touchpoint.
Simple or complex, the problem with these models is that they only measure how many conversions were touched by the campaign rather than how many were caused by the campaign. Unless you can tell the difference, it’s impossible to evaluate how successful the campaign was.
Imagine there was a mix-up at the office where someone accidentally linked ads for pizza to conversions for a blue jeans campaign. The attribution provider is then asked to report on which ads in this campaign were the most effective. We know the impossibility of selling blue jeans with pizza ads, but how would some attribution models handle this situation?
If it’s a large campaign, we would expect to see overlap between people who were advertised pizza and those who bought jeans. The attribution provider would apply their analysis and report which publisher served ads that, coincidentally, touched the most customers who bought blue jeans. Some publishers would be chosen as winners and others as losers. No alarm would go off screaming, “Hey, these ads are doing nothing! Something is wrong!” The problem is that these reports don’t show how many conversions were actually caused by the ads.
Bias And Baseline
The way out of this scenario is for marketers to establish a baseline. How many conversions would have occurred if the ad campaign had not happened at all? Let’s call these natural conversions. Those natural converters didn’t need any ads to make their decision, so money spent on advertising to them was wasted. However, if we find that customers are converting more often than their natural rate, then the ads are working.
To get to this baseline scientifically, we could perform an A/B test where we randomly give 10% of our audience a placebo, such as a public service announcement (PSA). Any difference between the ad exposure and PSA group could be attributed to the campaign. However, in this scenario, 10% of the ad spend is thrown out on PSAs. That’s a rather expensive option.
As an alternative, one could compare conversions by those who received ads vs. those who did not. This is cheaper than buying PSAs, but this exposes one to a whole array of selection biases. Users who receive ads are just different from those who did not. These targeted users were specially selected to receive ads, usually by some type of purchase-intent modeling, and so cannot be compared to the general population. Research has established that correcting for this bias is possible, but extreme care must be taken.
Moving Forward
Measuring true campaign performance is clearly difficult, but also too important to leave undone. It is widely known that today’s attribution systems are imperfect. An attribution model that can’t figure out whether pizza ads can sell blue jeans is hardly useful at all.
But, if more ad professionals apply a critical eye, then we can push the industry towards better and more reliable measurements of performance.
Follow Integral Ad Science (@Integralads) and AdExchanger (@adexchanger) on Twitter.
