Home Data-Driven Thinking The Truth About Cross-Device Data Quality

The Truth About Cross-Device Data Quality


johndempseyData-Driven Thinking” is written by members of the media community and contains fresh ideas on the digital revolution in media.

Today’s column is written by John Dempsey, senior director of mobile, Oracle Data Cloud, at Oracle.

“Deterministic” isn’t the same as “true.”

The industry has reduced cross-device accuracy to a single question: “How many of your IDs are deterministic and how many are probabilistic?”

Unfortunately, this attempt to quantify accuracy is based on the flawed assumption that any link that is directly observed is 100% correct. So-called “deterministic” links come from users who have logged in to a device – a straightforward concept that’s become the industry shorthand for high-quality matching. But a login doesn’t always tell the truth.

What about when your kid logs in to Facebook using your phone? It’s a login, but it’s not you.

What about when users don’t want to provide their real information and fill in bogus emails, such as [email protected]? That particular email address matches to hundreds of thousands of US households in some databases. Same for [email protected], [email protected] and [email protected]. Before long, you can build a “deterministic” set of millions of devices tied to real devices.

Unfortunately, these “bad data issues” are common. Before applying quality controls, as many as 15% of emails in industry databases link with two or more households. This means that for many databases, this deterministic data is only 85% accurate.

Companies may claim they have a lot of deterministic matches. If they have no way of cleaning up the bad data, the solution starts to fall apart.

A better way to talk about data quality is to acknowledge that everything is probabilistic. Any single observed link can be thought of as a fact – sure, that email was seen logging in to that device at that time – but not all facts are true.

All links, including deterministic links, have a range of probability, from zero to 100%. Login data tends to be on the higher end of the scale, but rarely is it 100% right.


AdExchanger Daily

Get our editors’ roundup delivered to your inbox every weekday.

It’s All Probabilistic … Until You Have A Truth Set

There simply isn’t enough high-quality deterministic data available to build a cross-device graph at meaningful scale, so even large players like Facebook and Google rely on so-called “probabilistic” techniques.

A probabilistic match means an association is inferred without being directly observed. In other words, you’re making a prediction about the likelihood that two bits of information are connected in real life. With data, technology and math, you can tease out these connections very well – even if they weren’t directly observed.

How does it work?

A probabilistic model has three components:

First, there is observation data – what signals are you using to predict a match?

For a cross-device solution, it’s best to observe each device as many times as possible, with associated metadata such as IP address, time stamp, application and device type. More data tends to be better.

Second, there is the algorithm – how are you making your prediction?

This is the math used to identify the correct connections. The methodology can vary but what matters most is how well the algorithm can identify correct matches. Statistician George Box famously said, “All models are wrong but some models are useful.”

How are you going to judge the usefulness of your model?

Finally, there is the truth set – what will you use to train and validate your algorithm?

A truth set – a set of links assumed to be 100% true – is required for any evaluation of a probabilistic solution. Careful consideration must be given when obtaining a truth set: If this is the data an algorithm will be judged against, you first have to make sure your truth set is, in fact, TRUE.

How True Is Your Truth Set?

It’s not good enough to say, “All my data is scored against a truth set,” without first understanding what true means.

Ask the right questions and you’ll quickly find that most companies have no way to validate their so-called truth sets.

The typical approach looks like this: A company licenses anonymized login emails from a third party without knowledge of how the data was collected or whether it is accurate. These “truth sets” are often small – a million or fewer cross-device pairs – and not representative of the general population.

Deterministic data cannot be blindly trusted to be accurate. If you train your model against a “dirty” truth set, you’re bound to get bad results. It’s garbage in, garbage out.

The source of the truth set matters: You can have far more confidence in truth set data sourced directly from companies that charge customers’ credit cards on a monthly basis and regularly ship items to the same address associated with their devices.

Truth Be Told

For companies like Facebook, Google and Twitter, an email address alone can be good enough because it’s more likely to be “you” when using their services. But when it comes to the open web, which relies on demand-side platforms and exchanges, a much higher standard is required.

Accuracy matters. Cross-device mapping impacts everything from retargeting and frequency capping to attribution. It pays to get it right.

So how do you assess the quality of a cross-device solution?

Don’t bother asking how many deterministic IDs there are. Instead, ask about the trade-off between scale and accuracy. And make sure the truth set is actually “true.”

Follow John Dempsey (@johnfdempsey), Oracle Data Cloud (@OracleDataCloud) and AdExchanger (@adexchanger) on Twitter.

Must Read

Advertible Makes Its Case To SSPs For Running Native Channel Extensions

Companies like TripleLift that created the programmatic native category are now in their awkward tween years. Cue Advertible, a “native-as-a-service” programmatic vendor, as put by co-founder and CEO Tom Anderson.

Mozilla acquires Anonym

Mozilla Acquires Anonym, A Privacy Tech Startup Founded By Two Top Former Meta Execs

Two years after leaving Meta to launch their own privacy-focused ad measurement startup in 2022, Graham Mudd and Brad Smallwood have sold their company to Mozilla.

Nope, We Haven’t Hit Peak Retail Media Yet

The move from in-store to digital shopper marketing continues, as United Airlines, Costco, PayPal, Chase and Expedia make new retail media plays. Plus: what the DSP Madhive saw in advertising sales software company Frequence.

Privacy! Commerce! Connected TV! Read all about it. Subscribe to AdExchanger Newsletters
Comic: Ad-ception

The New York Times And Instacart Integrate For Shoppable Recipes

The New York Times and Instacart are partnering for shoppable recipe videos.

Experian Enters The Third-Party Data Onboarding Business

Experian entered the third-party data onboarder market on Tuesday with a new product based on its Tapad acquisition.

Albertsons Takes Its First Steps Into Non-Endemic Advertising, Retail Media’s Next Frontier

Albertsons is taking that first step into non-endemic advertising next week via a partnership with Rokt to serve ads to people who have already purchased groceries.