A panda in a room where everything has been sorted and framed.MidJourney 5
Yesterday afternoon, in the last coding task of the colab, you were asked to compare the mean dissolved oxygen at two different sites. That took coding up two different filters and two .mean() calls. At the time, the instructions mentioned that today you would learn how to do analyze all six sites at once.
This morning we will learn how. We will write one line of code that produces analysis across all six sites simultaneously.
Most of the real questions you will ask about data are per-group questions. Not βwhat is the averageβ, but βwhat is the average for each Xβ: for each site, for each species, for each year, for each ocean. This morning we will learn the code pattern that answers all of those, and we will write it in both of the forms you will meet in other peopleβs code. In doing so, we will revisit commands and tools we havenβt seen since the first day of class when we were doing our data science workflow demo.
By the end of this session you will be able to:
describe what .groupby() does in terms of split, apply, combine
write the split-apply-combine pattern in its two-step form and its one-line form, and explain why they are the same pattern
learn to choose aggregations based on the question you need to answer
read the result of a grouped calculation and explain what its index means
explain why a grouped answer is often more useful than a whole-column answer
Getting Started
Create the file. In the Explorer, hover over the EDS217 heading and click New Fileβ¦, then type the name in full, extension included: Session_5A_Grouping_Data.ipynb
Check the kernel. The Kernel Selector in the notebookβs action bar should read Python 3.11.15 (Conda: eds217). If it reads anything else, click it, choose Change Kernel, and pick the eds217 entry.
Add a title cell. Click + Markdown in the action bar, and give it this content, with todayβs date:
# Day 5: Session 5A - The Split-Apply-Combine Pattern[Session Webpage](https://eds-217-essential-python.github.io/course-materials/interactive-sessions/5a_grouping_data.html)Date: 09/04/2026
Save with Ctrl + S (Cmd + S on macOS), and keep saving as you go.
Rebuild yesterdayβs clean stream-chemistry table. Every line below is one you wrote in yesterday afternoonβs colab, collected here so you can start from a table that is clean and well-organized:
Two filters, two means, two answers. Six sites would be six filters and six means, twelve lines to get one small table. And if a seventh site were added to the file next season, your twelve lines would silently keep reporting six.
Split, apply, combine
The pattern in those twelve lines has a name, and pandas has a single method that does all of it.
Split the table into one group per site. Apply the same calculation to each group. Combine the answers into a single result, labelled by group.
Split, apply and combine are what .groupby() does. We will write it in two steps first, because splitting the code up keeps the split and the apply visible separately:
Code
grouped = survey.groupby('site')grouped
<pandas.core.groupby.generic.DataFrameGroupBy object at 0x119059410>
Look carefully at the output. The grouped object is not a table. .groupby() on its own does the split and stops there, holding pointers to six different piles of rows. No arithmetic has happened yet, and none will happen until you name a calculation.
The second step picks a column and names the calculation:
Six sites, six means, one line of arithmetic. site_d and site_f are the two numbers you computed yesterday, and the other four came along for free.
Note
π The object .groupby() returns is a real object that you can keep in a variable and reuse. Ask it for a different column and the table is not split again, because the split has already happened:
groupby('key') says which piles to make. The key is the column whose values name the groups.
['column'] says which column to do analysis on.
.mean() says what arithmetic.
Learn it as one unit, the way you learned the top-N pattern on Wednesday, because you will be writing it for the rest of this course (and for most of your time in MEDS!).
βοΈ Test your knowledge
Write the two-line form to find the mean pH at each site (you can use the grouped object that already exists, or create a new variable if you want). Then write the same question in the one-step form. Confirm you get identical output.
Reading the result
The result is a Series, which is the one-column object we met on Day 2 and have been using all week. It has an index, just like all Series. However, this time, the index is made of the group labels:
The index matters, because it is now something you can look values up by. The site labels have stopped being data inside a column and have become the labels of rows:
Code
mean_do['site_c']
np.float64(6.873111111111111)
Everything you already know about a Series still works here: All Series objects will always share the same methods and attributes! For example, .idxmax(), from Wednesday afternoon, gives you the label of the largest value, and the label is now a site name:
Group survey by site and take the mean of temperature_c, then use .idxmax() on the result to name the warmest site. Compare it with the worst site for dissolved oxygen above. Are they the same site? Write one sentence saying whether you would expect them to be.
Choosing the aggregation
.mean() is not special. Any of the summaries you have been running on Series (whole DataFrame columns) since Tuesday will run on each group instead:
.count() is the one worth pausing on, because it answers a different kind of question. Instead of summarising the measurement, it tells you how many rows are in each pile:
Forty-five samples at site_c and thirty-nine at site_d, which is close enough to even that comparing their means is reasonable. When the counts are lopsided, .count() is the line that tells you so, before you have written the comparison into a report. Get in the habit of asking for it.
Aggregation
Answers
.mean()
what is typical in each group
.median()
what is typical, when a few extreme values would drag the mean
.sum()
how much in total, per group
.count()
how many rows went into each answer
.min(), .max()
the extremes within each group
.std()
how spread out each group is
.count() is not an afterthought
A group mean computed from four rows and a group mean computed from four hundred are printed in exactly the same font. Nothing in the output warns you. Whenever you report a grouped mean, run .count() on the same grouping and look at it, even if it never reaches your report.
βοΈ Test your knowledge
Answer both of these with one sentence each:
What is the highest pH recorded at each site?
How many bottles were filled in total at each site? (n_replicates counts bottles.)
Then say, in a markdown cell, why question 2 needs .sum() and not .count().
What can be a key
Any column whose values repeat can be a grouping key. The values do not have to be text, and they do not have to be tidy, but they do have to mean something.
site works because six labels describe all 255 rows. pH would not work, because its 255 readings take 138 different values, and grouping on it would give you 138 groups of which 67 hold a single row. A group of one tells you nothing you did not already have.
Yesterday you built a column of your own that groups beautifully:
Code
def classify_ph(value):"""Label a pH value as acidic, neutral, or alkaline."""if value <6.5:return'acidic'elif value >7.5:return'alkaline'else:return'neutral'survey['ph_class'] = survey['pH'].apply(classify_ph)survey['ph_class'].value_counts()
The derived-column pattern and the split-apply-combine pattern are working together here, and one part of it is easy to miss. ph_class was not in the file. You invented the categories yesterday with a function of your own, and this morning they are a grouping key. Many of the most useful groupings you will make are ones you build rather than ones the file hands you.
Note
π .value_counts(), which you have used since Day 2, is a grouped count in disguise. The two lines below produce the same numbers:
Read the three outputs together and there is a result in them. The sites with the warmest water have the least oxygen in it, in order, without a single exception. Warm water holds less dissolved oxygen than cold water, which is a fact about gases, so we should expect to see this pattern. But itβs nice to confirm that our data are showing us a pattern we expect!
The file was the same yesterday, and so was the ordering. Splitting by site and using aggregation turns data into information; the ordering only shows up once we have used our split-apply-combine workflow to put these results side by side.
Key points
Split, apply, combine: .groupby() splits the table into piles, an aggregation runs on each pile, and the answers come back as one labelled result.
The split-apply-combine pattern is df.groupby('key')['column'].aggregation().
The two-step form and the one-line form are the same pattern. The two-step version keeps the split in a variable so you can reuse it.
.groupby() on its own calculates nothing. The arithmetic happens when you name an aggregation.
The result is a Series indexed by the group labels, so .idxmax(), .idxmin() and label lookup all still work.
Any aggregation you can run on a column can run on a group: .mean(), .sum(), .count(), .min(), .max(), .median(), .std().
Always look at .count() alongside a grouped mean. Unequal groups are invisible otherwise.
Grouping keys are often columns you derived, not columns you were given.
