Statistics in the Age of Big Data

Much of modern life is recorded: searches, purchases, clicks, sensor readings, location pings. This flood of data has made possible things earlier statisticians could only dream of, from real-time maps of traffic to language models trained on billions of sentences. The temptation is to conclude that with enough data, analysis becomes easy and old cautions no longer apply.

The opposite is often true. Big data brings genuine new powers, but it also amplifies old dangers and adds fresh ones. A dataset can be enormous and still be biased, misleading, or dangerous to individuals' privacy. The skills you have already built, thinking about how data was generated and whether a pattern is real, matter more at scale, not less.

What actually changes with scale

With huge datasets you can detect rare events, study fine-grained subgroups, and update estimates almost in real time. But large is not the same as complete or representative. Data that captures only the people who use a particular app or platform can be vast and still leave out whole populations. Size reduces one kind of error, random noise, while doing nothing about bias in how the data was collected.

Spurious correlations at scale

Search a big enough dataset for patterns and you will find them, whether or not they mean anything. With thousands of variables, some will correlate strongly by pure chance. This is why famous spurious correlations, such as cheese consumption tracking an unrelated statistic across years, look so convincing. The more variables you comb through, the more coincidences you are guaranteed to surface. Correlation found by dredging is not evidence of a real relationship.

Bias baked in, and privacy

If the data reflects unequal past decisions, a model trained on it can learn and repeat those inequalities, a problem often called algorithmic bias. And because big data is often about people, it raises privacy stakes: individuals can sometimes be re-identified even from data that was supposedly anonymised, by combining it with other sources.

Suppose you have web logs for millions of users and you test 1,000 unrelated variables to see which predict whether someone buys a product. Even if none truly matter, at a standard 5 percent significance threshold you would expect about 50 of them to look statistically significant by chance alone. If you then report the strongest few as discoveries, you are reporting noise. The huge sample did not protect you; the large number of tests created the illusion. The fix is to hold out fresh data and see whether the pattern survives.

A small, carefully designed survey of a few thousand people, drawn to represent a whole country, can predict an election better than a massive dataset of millions of self-selected online responses. In 2016 several large but non-representative online panels misfired, while properly weighted smaller samples did better. More rows lost to the wrong people beat fewer rows chosen well.

In 2008 and 2009 Google researchers described Google Flu Trends, a system that estimated influenza activity from the frequency of flu-related search queries. Their paper in Nature reported that from hundreds of billions of searches they selected a set of query terms whose volume closely tracked official flu data from the US Centers for Disease Control and Prevention, and it could produce estimates faster than the CDC's own reporting. It was hailed as a landmark for big data in public health.

Then it drifted. Google Flu Trends missed the non-seasonal 2009 H1N1 outbreak and, in later seasons, badly overestimated flu. For the 2012 to 2013 season it estimated roughly double the CDC's figure. In a 2014 Science article, The Parable of Google Flu, David Lazer and colleagues coined the phrase big data hubris to describe the assumption that big data can substitute for, rather than supplement, traditional analysis. Search behaviour shifted, Google's own autocomplete nudged queries, and the model was never retuned to match. Google stopped publishing the estimates in 2015. The lesson is not that search data is useless, but that scale without careful, updated modelling fails.

• "With enough data, bias washes out." More data shrinks random noise, not systematic bias in who is included.
• "A strong correlation in a huge dataset must be real." Testing many variables guarantees chance correlations.
• "Anonymised big data is automatically safe." People can often be re-identified by linking datasets.
• "Big data replaces sampling and modelling." Google Flu Trends shows the opposite: it still needs both.

You are shown several strong correlations pulled from a giant dataset. For each, decide whether there is a plausible mechanism or whether it is likely a product of data dredging, then propose one test to tell the difference.

Think Like a Maester: More data is not automatically more truth, because size cures random noise but not bias, drift, or the coincidences that hide in a haystack of variables.

1. What kind of error does a very large sample reduce, and what kind does it leave untouched?
2. Why does testing 1,000 unrelated variables produce apparently significant results even when none truly matter?
3. What does big data hubris mean, as described in the Google Flu Trends case?
4. Give two specific reasons Google Flu Trends grew inaccurate over time.
5. Why can a small representative survey beat a massive self-selected dataset?

Big data opens real possibilities: rare events become visible, estimates arrive in near real time, and fine subgroups can be studied. But size is not truth. Large datasets can be unrepresentative, riddled with chance correlations, biased by the past, and hazardous to privacy. Google Flu Trends captured this perfectly: an impressive start that decayed into large overestimates once its makers assumed volume could replace careful, updated modelling. The old questions, where did this data come from and is this pattern real, are the ones that keep big data honest.

• Correlation and Regression
• Statistics for an Informed Life