“Data-Driven Thinking” is written by members of the media community and contains fresh ideas on the digital revolution in media.
Today’s column is written by Marilois Snowman, CEO and founder of Mediastruction.
The MRC is taking on the task of accrediting media mix modeling. It’s a laudable effort, especially with the deprecation of cookies.
But with that said, the industry needs an education initiative before accreditation can take place. We need to agree on definitions at the very least, then align on methodology. But it will be some time before this happens.
In the meantime, here’s how to evaluate the effectiveness and impact of existing models.
Is the attribution measuring incrementality or lift?
There are two primary types of attribution models: those that measure incrementality and those that measure lift.
In any given year, about 75% of a brand’s sales (or other KPIs) are attributable to “history.” That includes all the years of marketing, word of mouth, referrals, customer experience and squishier marketing levers that contribute to sales. Incremental models are important here, because they are the ones that measure everything outside of history.
Meanwhile, other attribution models measure lift – the marginal difference of sales before, during and after advertising exposure. I would argue that measuring lift is far less insightful and actionable than incrementality. That’s mostly because incrementality delivers a much more accurate ROI gauge.
One other important note: There are always exogenous variables that contribute to KPIs. These variables can be difficult to quantify. Examples of exogenous variables include weather, stock prices, overall economic confidence and competitive share of voice. These variables are typically lumped in “history,” but they are typically factors for which we can’t account. A good scientific model notes that such metrics exist.
Is the methodology holistic or singular?
Many singular attribution methodologies use identifiers to connect an exposed consumer to a desired action and attribute it to a single media touch point. For example, did the consumer get exposed to an ad on her mobile device and then visit a store, where a geolocation identifier attributed the visit to the same device the ads were served to?
Sometimes, these identifiers do reveal valuable insights. However, this methodology doesn’t account for all the other marketing touch points that also may have contributed to that same store visit. In the previous example, did our consumer also pass a billboard or see an ad on linear TV?
With the singular digital breadcrumb methodology of attribution, digital advertising receives disproportionate credit for the sale.
Too often, media companies and digital ad firms try to fit a square digital attribution peg into a round cross-channel, “tradigital” hole. Yes, Virginia, there are holistic attribution models, namely media mix modeling, that can align online with offline channels and level the playing field in measurement.
Is the solution provider a black box or is the model custom to the brand?
There are hundreds if not thousands of potential attribution math models.
Remember, modeling is also done in other industries like economics, weather, travel, etc. If a vendor’s model breaks with a faulty forecast, is the model to blame? Or is the advertising ineffective? When the solution operates like a black box, there would be no way to know.
We like to say “let the data decide” which model is best suited for a brand. But there exists a fairly elegant way to prove the model using a technique called the “holdout.” It works like this: Your model partner should extract all available historical data and train the model on just a portion of it, holding out the last section.
If the model’s predicted KPIs are in alignment with the actual KPI performance of the holdout section, then the model “fits.” There, of course, will be a margin of error. As long as it’s fairly standard, you can proceed with confidence.
As the MRC continues its accreditation process, there will be questions around customization, opaque black boxes, differing definitions/misunderstanding and bias. But that’s a good thing.
If more of us ask questions, collaborate and educate, the effort will be a huge win.
Follow Mediastruction (@Mediastruction) and AdExchanger (@adexchanger) on Twitter.
For more articles featuring Marilois Snowman, click here.
