The Data Science Workflow

The ten steps, and where you’ll learn each one

Almost every data analysis you will ever do, in this course and in your career, is a path through the same ten steps, listed below. Most analyses use only some of the ten steps, and the ones an analysis does use may not always run in the order they are numbered. Many analyses will actually iterate across steps as the work becomes more focused on a specific question or set of results. Despite the fact that each workflow is going to be somewhat unique, like snowflakes, they often generally take on the same general shape and patterns. For this reason, we’re using this 10-step process as a framing for our short course and as a general heuristic for how a data scientist progresses from raw data to insight.

On Day 1 we run a short path end to end, from a data file to a finished chart. Over the following days we work through each step in depth.

flowchart LR
    A["1. Import πŸ“‚"] --> B["2. Explore πŸ”Ž"] --> C["3. Clean 🧼"]
    C --> D["4. Filter 🎯"] --> E["5. Sort πŸ₯‡"] --> F["6. Transform βž•"]
    F --> G["7. Group πŸ—‚οΈ"] --> H["8. Aggregate πŸ“Š"] --> I["9. Join / Reshape πŸ”—"]
    I --> J["10. Visualize πŸ–ΌοΈ"]

The ten steps

  1. Import. Load your data from a file or URL into a pandas DataFrame, which is a table.
  2. Explore. Look at the table’s shape, columns, types, and summary statistics so you understand what you actually have.
  3. Clean. Fix missing values, wrong data types, and duplicate rows so the data is trustworthy.
  4. Filter. Keep only the rows that match a condition you care about.
  5. Sort. Reorder the rows by the values in one or more columns.
  6. Transform. Build new columns from existing ones, such as calculations, conversions, or categories.
  7. Group. Split the rows into buckets that share a common key.
  8. Aggregate. Collapse each bucket down to a single summary number, like a mean, sum, or count.
  9. Join / Reshape. Combine multiple tables, and pivot or stack the data into the shape your analysis needs. Working with dates belongs to step 9 as well, because a date column is what lets you reshape a table along time.
  10. Visualize. Turn your numbers into charts, so that a reader can see the result rather than work it out from a table. Saving what you made belongs to step 10 as well, whether that is writing the figure to a file or exporting the summary table you built it from.

Where each step is taught

Days 2 through 7 each teach one or two of the steps. The teaching order is not the workflow order. Filter and Sort are easier to learn first, so Day 3 teaches them, and Day 4 comes back for Clean afterwards.

# Step What it does Taught on
1 Import Load data into a DataFrame Day 2: Import + Explore
2 Explore Get to know the table Day 2: Import + Explore
3 Clean Fix missing values, types, duplicates Day 4: Clean + Transform
4 Filter Keep only the rows you want Day 3: Filter + Sort
5 Sort Order rows by a column Day 3: Filter + Sort
6 Transform Build new columns Day 4: Clean + Transform
7 Group Split rows into buckets by a key Day 5: Group + Aggregate
8 Aggregate Collapse each bucket to a number Day 5: Group + Aggregate
9 Join / Reshape Combine tables; add a time dimension Day 6: Join + Reshape + Dates
10 Visualize Turn numbers into pictures Day 7: Visualize

Days 8 and 9 put the whole workflow together in your own project.

The Data Biography

Exploring a dataset, step 2, has a written half that the step list does not show. A data biography is a short account of a dataset: where it came from, what it covers, what condition it is in, and what a reader needs to know before trusting anything computed from it. Working data scientists write a biography for most datasets they touch, usually in the first cells of a notebook. The alternative is discovering a flaw in the data after an analysis has already been built on it.

One rule makes a biography worth reading: every claim in it must sit directly below the cell that produced the number behind it. Not β€œthe data covers about thirty years”, which is too vague to check, but β€œthe data covers 1988 to 2018”, with the cell that printed those two years directly above the sentence.

Day 2’s end-of-day practice is writing a data biography of the Toolik Field Station weather record.

Key Points

  • Real analyses are paths through the ten steps, always in the numbered workflow order rather than the order the course teaches them in, though few analyses use every step.
  • Days 2 through 7 each teach one or two steps, in an order chosen for learning rather than for the workflow. Day 1’s run through the complete workflow gives you the big picture first.
  • When you feel lost later in the course, come back here and ask yourself which step you are on.

Resources