Fact Checking Data

This data was originally featured in the January 31, 2024 newsletter found here: INBOX INSIGHTS, JANUARY 31, 2024: NEW TECH DOESN’T SOLVE OLD PROBLEMS, FACT CHECKING DATA

One of the things that is common in any news cycle, but especially during political cycles like elections, is bold claims made with numbers and data. “Restaurant prices have doubled since 2019!” and “Inflation has gone sky high!” and other claims. We aren’t going to deal with politics at all, but this is a good opportunity for us to discuss the citizen analyst – and how it can inform and improve your marketing.

When I hear someone make a claim using data, my first instinct is to ask for a source. What’s the source of that data? And if it’s a person or source that I don’t feel comfortable asking (because who has time to argue on the Internet all day?), then I’ll go get the data myself.

Let’s tackle the claim about restaurant prices doubling since 2019. Is that actually true? How would we know? With the help of generative AI-powered search engines like Microsoft Bing or Perplexity.ai, we can identify credible sources of data. Here’s one from The Economist, the price of a Big Mac in every nation over the last quarter century.

About a decade ago, IBM pioneered the concept of the citizen analyst with tools like IBM Watson Analytics, which unfortunately never took off to the extent they wanted it to. However, in the era of generative AI, doing “citizen analysis” is much more practical because AI tools can perform complex analysis without requiring you and I to code or perform other complex analysis.

Let’s take the example of the Big Mac pricing data from The Economist:

You’ll note that the prompt doesn’t contain any advanced statistical knowledge or techniques, just a clear statement of what we want to achieve. After doing its analysis, ChatGPT had this to say:

Broadly, then, the prices of a signature dish at the McDonald’s restaurant chain have NOT doubled since 2019. Is there anywhere this claim that restaurant prices have doubled is true?

Yes, in Argentina, from the dataset – at least in terms of the price of a Big Mac, restaurant prices have doubled since 2019.

What this exercise shows is that, with the help of data you acquire from credible sources and tools like ChatGPT’s Advanced Data Analysis, you can investigate claims made about data.

This is a great exercise to do, not only for fact checking what news sources and public figures say, but for also proving or disproving claims about numbers in general. For example, you’ll often hear marketing claims like “email has the highest ROI of any channel” or “the average engagement rate on YouTube is XYZ” or “XYZ has the highest CTR of any ad category”. With the right data and tools you have access to, you can investigate this for yourself and see if the claims are actually true, then change your marketing strategy based on your findings.