A crowd of pandas, in need of some grouping. MidJourney 5
Day 5 is all about one code pattern, and it is the one that forms the basis for many, many data science analyses. It has a common name, split-apply-combine, which defines the steps necessary to break a table into groups, run a calculation on each group, and then summarize the results across each group. Almost every real question about data turns out to be that shape: not βwhat is the averageβ, but βwhat is the average for eachβ. This approach is similar to the even more general map-reduce pattern of data analysis popularized by Google, where the map step corresponds to split-apply, and reduce takes the place of combine. Generally, reduce encompasses a larger range of more sophisticated analyses than combine, and the map phase is implicitly structured around parallel processing workflows.
This afternoon you will address some questions the pattern cannot answer, and learn some older, slower tools that can handle them.
Class materials
| Session | Session 1 | Session 2 |
|---|---|---|
| day 5 / morning | ποΈ The Split-Apply-Combine Pattern | π Several Answers at Once |
| day 5 / afternoon | π The Comparison You Made Yesterday, Properly | π When a Pattern Is Not Enough |
End-of-day practice
At the end of the day today we will use a month of hourly air quality readings from three monitoring stations on the South Coast to explore patterns across the dataset. Our end goal is a result that depends on the split-apply-combine workflow we have learned today.