The Data Analysis Workflow

The earlier modules taught the pieces: charts, averages, spread, correlation, probability. This one is about the job those pieces belong to. Real analysis is a process, and the tools only earn their keep when they sit inside it. Skip the process and you can produce a beautiful chart that answers a question nobody asked, or a confident number built on data that was never fit for the purpose.

The biggest mistake beginners make is reaching for a method too soon. They open the spreadsheet and start plotting before they know what decision the work is meant to inform. A calm, ordered workflow protects you from that. It keeps the question in charge, treats the data with suspicion until earned, and leaves you with an answer you can explain to someone who was not in the room.

The question comes first

Every worthwhile analysis starts with a question specific enough to be answered and to be wrong. "How are sales doing?" is a mood, not a question. "Did revenue per customer fall after the price change in March?" can be checked against data and can come back no. A sharp question tells you which data you need, which comparison to make, and when you are finished. Vague questions expand to fill all available time.

Get and prepare the data

Once the question is clear, you find data that can actually answer it, which is rarely the data you already have. Then comes preparation: fixing formats, handling missing values, removing duplicates, and reconciling sources that disagree. This stage is unglamorous and it is where the hours go. The next lesson is devoted to it. The point here is simply that preparation is part of the workflow, not an annoyance before the workflow.

Explore before you analyse

Before any formal method, look at the data. Plot distributions, check ranges, eyeball a few rows, and see whether the numbers are even plausible. This exploratory step catches errors, reveals surprises, and tells you which analysis is sensible. Only then do you run the actual analysis, an average, a comparison, a model, whatever the question demands.

Communicate, and loop back

An answer nobody understands is not an answer. The final stage translates the result into plain language and a clear picture for the people who must act on it. And the whole thing is a loop, not a line: exploring the data reshapes the question, cleaning reveals you need different data, a first result raises a better question. Good analysts expect to go round several times.

A cafe owner asks whether Tuesday's new loyalty card lifted spending. Question first: did average spend per visit rise after the card launched, compared with before? Next, data: till records for the weeks either side. Cleaning: strip out refunds and staff test transactions. Exploring: a quick chart shows a school holiday landed in the "after" weeks, inflating footfall. That surprise sends the owner back to compare like with like, matching term-time weeks only. The final answer, spend rose modestly, is now honest, and it took a loop to get there.

Contrast that with jumping straight to modelling. An eager analyst pulls the whole year of sales, fits a fancy trend line, and reports that spending is "up 12 percent, statistically significant." But nobody defined the question, so the 12 percent silently mixes the price rise, the holiday, and seasonal weather. The maths is flawless and the conclusion is meaningless, because the workflow was skipped. Sophistication downstream cannot rescue a missing question upstream.

A widely repeated finding is that data professionals spend most of their time finding and preparing data rather than analysing it. A 2014 New York Times article by Steve Lohr, "For Big-Data Scientists, 'Janitor Work' Is Key Hurdle to Insights," reported estimates that data scientists spend from 50 to 80 percent of their time collecting and cleaning data. A 2016 CrowdFlower Data Science Report similarly found respondents spending around 60 percent of their time cleaning and organising data and another 19 percent collecting datasets. These figures are self-reported survey estimates and the exact numbers vary and are debated, so treat them as a well-documented pattern rather than a precise law. The direction, though, is consistent across sources: preparation dominates, and modelling is the smaller slice.

• "Analysis means running a model." Modelling is one late step; framing, sourcing, cleaning, and exploring are most of the work.
• "Start with the data you have." Start with the question, then find data that fits it; convenient data often cannot answer it.
• "The workflow runs in a straight line." It loops. Findings reshape the question and send you back for better data.
• "A polished chart proves the work was sound." Presentation says nothing about whether the question or data were right.

Given a messy real project, drag the stages into order and spot where the analyst should loop back. Decide which questions are answerable, which datasets actually fit, and where a surprise in the data should send the work upstream again.

Think Like a Maester: The question is the compass; without it, every chart points somewhere, but none point at the answer.

1. Why should a specific question come before choosing any data or method?
2. Which stage of the workflow typically consumes the most time, and why?
3. In the cafe example, what did the exploratory step reveal, and how did it change the plan?
4. What does the counterexample show about starting analysis without a defined question?
5. How should you present the survey figures on time spent preparing data, and why hedge them?

Data analysis is a process, not a single trick. It begins with a question sharp enough to be answered and to be wrong, then moves through sourcing, cleaning, exploring, analysing, and communicating, looping back whenever the data teaches you something new. The unglamorous middle, finding and preparing data, is where most of the effort goes, a pattern reported across industry surveys. Keep the question in charge, respect the preparation, and your answers will be ones you can defend.

• What Is Statistics?
• Cleaning and Preparing Data