Marketing measurement is really about answering two questions: What happened? And what should I do about it?
Most marketing mix models are built to answer the first one, but the second is where things get harder and also more interesting.
On Friday, Lifesight – a marketing measurement platform that combines MMM, incrementality testing and attribution – open-sourced Horizon, its tool for predicting future demand. Everything – the code, the underlying methodology, supporting documentation and benchmarking materials – is now available on GitHub.
“The reason we made it open source is to build trust and also to establish that it’s good practice to bring forecasting into your measurement system,” said Rajeev Nair, Lifesight’s co-founder and chief product officer. “By making this public, our customers and prospects can see the results for themselves and know this is something that’s been proven on real data.”
Open-source measurement has been having a moment. Google’s Meridian, Meta’s Robyn and PyMC-Marketing, an independent open-source marketing analytics library, all arrived in recent years with a pitch to offer more open, inspectable measurement.
The motivations may not always be purely altruistic. PyMC-Marketing aside, the big platforms have their own reasons for wanting your data in their models. But the effect of these launches has been real. Measurement is more open and transparent than it was five years ago.
MMM, meet tomorrow
But those tools explain the past, like showing which channel drove what action or how much revenue can be tied to media versus pricing versus promotions. MMMs aren’t built for looking forward.
The thing is, a significant chunk of an established brand’s revenue – sometimes 50% to 60% – comes from factors that have nothing to do with marketing, including seasonality, brand loyalty, word of mouth and repeat purchases. This demand would exist even if a brand didn’t run ads, didn’t change its prices – did nothing, basically.
Modelers call this the “baseline,” and it’s where MMM runs out of road.
“Extrapolating from measurement models is usually accurate only for very small changes,” said Dr. Ron Berman, an associate professor of marketing at the Wharton School and the first member of Lifesight’s newly created Scientific Advisory Council, an independent body that the company created to peer review its research and methodologies.
“If you double your budget on something you’ve never observed in the past, the extrapolation is going to be really off – and in marketing, things change,” Berman said. “Take seasonality. My forecast from holiday season data is going to be way off for January or February.”
Most companies don’t really solve for this. They often either lean on MMM to do the forecasting, or they forecast without solid measurement behind it.
“But without a forecasting system, even if you have accurate measurement, it doesn’t help your business beyond a point,” Nair said.
Horizon, which works by running multiple forecasting models simultaneously and combining their outputs, rather than relying on any single one, sits alongside whatever measurement setup a brand is already running – Lifesight’s MMM, Google’s Meridian or something built in-house – and handles the forecasting side separately.
Because the question isn’t really about which forecasting model to use. The answer is to use all of them.
All together now
That approach has a name: ensemble forecasting, a statistical method that runs multiple models at the same time and combines their outputs.
Different models catch different signals, and no single model gets everything right. But, averaged together, they tend to get closer than any one of them would alone.
This thinking is well established in academic forecasting research. It’s yet to catch on in the business world, however.
“A lot of people just assume there’s a best method, that you can compare them and simply pick the best one – it’s instinctive,” Berman said. “But we know academically that combining them is generally better, and I think it’s important to evolve people’s thinking about that.”
But why open-source it?
Helping practitioners rethink measurement is part of what the new Scientific Advisory Council exists to do, and it starts with taking academic research out of the ivory tower.
Most forecasting research has been applied to things like predicting exchange rates or disease diffusion – worlds away from what marketers need to know before spending their next incremental media dollar.
“I’m always trying to put myself in the mind not of an engineer,” Berman said, “but of a marketer who needs to make a decision.”
Publishing Horizon openly, with its benchmarks and documentation there to be poked and prodded, is one way to make that research usable – and credible. In an ideal world, though, none of this would be remarkable. It’s just how measurement would work, Nair said.
And if a competitor wants to build on top of Horizon, that’s fine by him. They’re welcome to it. Horizon is a layer over existing open-source libraries, not a proprietary breakthrough. The differentiation, Nair said, is in the execution.
“There’s no need for any of this to be a black box,” he said. “The true value is in making it easy for brands to orchestrate, calibrate and then make decisions.”
