Yesterday we asked you to determine the varying distribution of plastic in the Pacific and the Atlantic. You built two tables with two filters, ran .describe() on each, and then wrote a paragraph to an imaginary journalist.
You gave them an answer about two oceans because two oceans were all we asked about. However, the raw file has four oceans, thirty-seven contributing studies and nineteen different ways of taking a sample.
This afternoon you will go back into the microplastics data and do a more comprehensive analysis: all four oceans in one line instead of two filters, and then a look at whether the sampling methods behind those four numbers are comparable enough to put them side by side.
Work in pairs, in one shared notebook, taking turns at the keyboard. Swap every time you finish a numbered task. The person not typing should read the code through (quietly!) and explain what it should produce or calculate before you run it.
We have about an hour for this activity: roughly 45 minutes on the four parts, and the rest on setting up and on the wrap-up. Please let Cella or Kelly know if you get stuck, rather than spending your time on one line!
Both new patterns are from this morning. Hopefully, you will not need to look them up more than once!
df.groupby('key')['column'].mean() # split, apply, combine
df.groupby('key')['column'].agg(['count', 'mean']) # several summaries at once
df.groupby('key').agg({'a': 'mean', 'b': 'count'}) # different summaries, different columnsYou will also need the top-N pattern from Wednesday:
df.sort_values('column', ascending=False).head(n)Create a notebook named Colab_5C_Grouped_Comparisons.ipynb, with both partnersβ names in the title cell, then rebuild yesterdayβs positive table. Every line below should be one you wrote yesterday afternoon:
import pandas as pd
import numpy as np
url = 'https://eds-217-essential-python.github.io/data/marine_microplastics.csv'
df = pd.read_csv(url)
df = df.dropna(subset=['Measurement'])
samples = df[df['Unit'] == 'pieces/m3'].copy()
samples['ocean'] = samples['Oceans'].str.replace(' Ocean', '')
positive = samples[samples['Measurement'] > 0].copy()
positive.shape(7091, 23)You should end up with 7,091 water samples, each with a measured, non-zero concentration of plastic in pieces per cubic metre.
Answer each question with code, then write the answer in a markdown cell underneath, in a complete sentence with the numbers in it.
Yesterday you computed the mean Measurement for the Atlantic and the Pacific with two filters. Now do all four oceans in one line, and rank the result from dirtiest to cleanest.
Do the same with the median instead of the mean. Does the order change? If so, which two oceans swap?
How many samples went into each of those four numbers? (can/should be a single line of code!)
In a markdown cell: two of the four oceans have numbers you should refuse to report. Name those two oceans and explain why in one sentence.
Replace questions 1 through 3 with a single .agg() call that reports the count, median, mean and max of Measurement for each ocean.
Look at the Atlantic rowβs median and maximum. In a markdown cell, explain what a single large number does to a mean computed from 6,216 values. Which of the two measures of central tendency (mean vs. median) would you send to the journalist?
Use the dictionary form of .agg(), grouped by ocean, to report the mean Measurement, the number of distinct Sampling Method values, and the number of distinct Organization values in each ocean.
In a markdown cell explain the value of asking for these non-measurement summary data. What might it mean if one ocean had been sampled by only one organization using a single method?
Group by Sampling Method instead of by ocean, and report the count and median of Measurement for each method. Rank by median, largest first.
Write down the largest and smallest medians in that table. How many orders of magnitude separate them? (np.log10 of the ratio will tell you, or count the zeros.)
In a markdown cell, before you write any more code: if the oceans were not sampled with the same mix of methods, what might that imply regarding your answer from question 1? Two or three sentences.
Letβs see if we can determine the impact of measurement method. Group by both ocean and Sampling Method at once and count the samples. Then call .reset_index() on the result and look at the Atlantic and Pacific rows.
Hereβs a simpler and more direct approach: run .value_counts() on Sampling Method for the Atlantic samples and again for the Pacific samples. Which method dominates the Atlantic? What fraction of the Pacific samples used it?
Now letβs control for these differences. Filter positive to the rows collected with a 'Neuston net', end the line with .copy(), then group that by ocean and report count, median and mean.
In a markdown cell, write three or four sentences that compare this table with your answer to question
They should! A difference that survives the control but changes size is a different claim from a difference that goes away completely, and the two need different interpretations in a data analysis report. Which one do you have?
The journalist replies: βThanks. My editor wants one sentence with a number in it.β Write that sentence in a markdown cell. It must contain a number, must name the sampling method you controlled for, and must not overstate what 158 samples can support.
In a markdown cell of four or five sentences: what changed between yesterdayβs answer and this one, and was it the data or the question? Name the specific line of code that made the difference visible.
The Regions column is finer-grained than Oceans. Group by it, report count and median, and rank by count to find the ten most-sampled regions. One of them has a median four hundred thousand times larger than another. Look at which sampling methods were used there before you conclude anything about the water.
Do not try to compare regions across years. Comparing across years needs dates⦠and we get to those on Tuesday!
If we have time, we will hear from two or three pairs on question 17. Be ready to say which grouped table changed your mind (if one of them did!), and not just what your final answer was.