“Data-Driven Thinking” is written by members of the media community and contains fresh ideas on the digital revolution in media.
Today’s column is written by Cameron Wertheimer, director of corporate development and strategy at Vertical Mass.
As budgets continue to shift toward programmatic, it is more important than ever for marketers to use statistics when analyzing their campaigns. Even a moderate amount of statistical analysis can have an outsized impact, given the amount of data a single spend can generate.
One little-known statistical fallacy called Simpson’s paradox deserves special attention in digital advertising because it can cause marketers to misinterpret the results of their campaigns and waste money.
“Simpson’s paradox, or the Yule–Simpson effect, is a phenomenon in probability and statistics in which a trend appears in several different groups of data but disappears or reverses when these groups are combined,” according to Wikipedia.
Don’t worry if you had trouble following that. Simpson’s paradox is much easier to understand when illustrated with an example.
Suppose an agency is running a campaign with click-through rate (CTR) as the main objective. The campaign manager pulls the first weekly performance report and breaks out the results by gender.
| Female | Male | |
| Clicks | 500 | 750 |
| Impressions | 50,000 | 50,000 |
| CTR | 1% | 1.5% |
The chart shows that males have a 50% greater CTR. The campaign manager concludes that more budget should be allocated to males. However, that would be a mistake if other variables aren’t taken into account.
Here is the same data set broken down by age.
| Female | Male | |||
| 18-24 | 25-34 | 18-24 | 25-34 | |
| Clicks | 470 | 30 | 740 | 10 |
| Impressions | 25,000 | 25,000 | 40,000 | 10,000 |
| CTR | 1.88% | 0.12% | 1.85% | 0.10% |
The female grouping still has an aggregate 1% CTR, and the male grouping still has a 1.5% CTR. However, this new data indicates that the campaign manager should increase spending on the female 18-to-24-year-old cohort.
The age grouping is a confounding variable. It plays a major role in determining CTR, but it was not observable in the first table due to how the data was broken out. There are two lessons we can take away from this.
First, planners should not rely on overly broad audience segments. Many buyers use reach as their primary criterion to ensure that their campaigns scale. This approach leaves them susceptible to Simpson’s paradox.
In the case of a brand trying to reach pop fans, it would most likely be helpful to break out those pop fans by their passions, such as whether they are concertgoers, heavy music streamers, merchandise purchasers or social engagers.
These distinct groups of pop fans may react differently to different messaging styles. For example, concertgoers may be more likely to engage with an ad showing people at a concert, whereas heavy streamers may be partial to seeing imagery featuring someone listening to music at home.
The buyer will not understand these distinct groups unless they plan ahead of the campaign to surface those groups. Working with granular, robust audiences is key to combating this common mistake.
Second, planners should work with their campaign managers to break out other targeting variables as much as possible to surface other confounding variables. Time of day, day of week, ad exchange and browser are some variables that often are not surfaced in campaign reports and therefore muddle the conclusions drawn from campaigns.
These variables should be considered not only for reporting but also during the campaign setup process. This entails creating targeting groups that correspond to the variables. Campaign managers tend to check the performance of targeting groups on a daily basis, and they tend to pull more detailed reporting on a weekly basis. Matching targeting groups to key reporting variables during the campaign setup will make the campaign manager more aware of the key variables on a day-to-day basis.
There are drawbacks to setting up campaigns with too many data sets and targeting groups, however. The added complexity increases the likelihood of mistakes and makes optimizations more cumbersome. Having too many targeting groups can be also inefficient when there is not enough budget to deliver a statistically significant number of impressions against each group.
Planners should try to deploy appropriately granular audiences, and the campaign setup should aim to surface confounding variables without being disproportionately complex relative to the overall budget.
Follow Vertical Mass (@VerticalMass) and AdExchanger (@adexchanger) on Twitter.
