Coding Colab

Two pandas, collaborating to combine their datasets. MidJourney 5

Two of the most important time series in environmental science are global surface temperature and atmospheric concentration of carbon dioxide. The two datasets are collected by different people/institutions, are used for different reasons, and are stored/maintained in different locations.

NASA’s GISTEMP record estimates how much warmer or cooler each month’s global surface temperature was than a twentieth-century baseline. It starts in 1880, which is the earliest that sufficient observations of temperature existed to estimate a global average.

The concentration of carbon dioxide is measured in the atmosphere directly, in parts per million, from an observatory near the summit of Mauna Loa, a Hawaiian volcano. It starts in 1958, because that is when Charles Keeling built the system that first started recording these concentrations (initially as air samples collected in flasks and then returned to his lab for analysis).

The most famous graph in climate science puts those two time series on one pair of axes. We do not draw it today, because plotting is tomorrow’s work, but everything that has to happen to the data before anybody can draw it is today’s, and it takes every pattern we have learned along with some good decisions about how to organize these datasets.

Work in pairs, in one shared notebook, taking turns at the keyboard. Swap every time you finish a numbered task. The person not typing should read through the output (quietly) and explain what the code should produce/calculate before you run it.

We have about 45 minutes for this activity, so please let Cella or Kelly know if you and your partner are getting stuck on any of the tasks!

The patterns you need

The patterns that are new today came from this morning and from the session you just finished.

pd.merge(left, right, on='key')                       # the join pattern
pd.merge(left, right, on='key', how='left')           # keep every row on the left

pd.to_datetime(column, format='%Y-%m-%d')             # the parsing pattern
column.dt.year                                        # and its accessors

df.pivot_table(index=, columns=, values=)             # the pivot pattern

Plus two more code patterns you have been using all week:

df.groupby('key')['column'].agg(['count', 'mean'])
df.sort_values('column', ascending=False).head(n)

Setup

Create a notebook named Colab_6D_Two_Records.ipynb, with both partners’ names in the title cell, then read the two files:

Code

import pandas as pd

base = 'https://eds-217-essential-python.github.io/data/'

temp = pd.read_csv(base + 'monthly_temperature_data.csv')
co2 = pd.read_csv(base + 'monthly_co2_concentration.csv')

Part 1: Making one table out of two (about 12 minutes)

Answer each question with code, then write the answer in a markdown cell underneath, in a complete sentence with the numbers in it.

How many rows does each table have, and what columns? What is the earliest and latest Date in each? (.min() and .max() on the Date column will do it, because these dates are text in %Y-%m-%d form and text in that form sorts correctly.)
Merge the two tables on Date, with the default how=. How many rows come back?
Merge them again with how='left', putting temp on the left. How many rows now, and how many nulls, in which column?
In a markdown cell: the two merges differ by 940 rows. Provide a sentence explaining what those 940 rows contain (your answer to question 1 should already have this).
Suppose you are writing one of the following two papers. For each one, say which merge you would use and why:
- “Atmospheric CO₂ and global temperature move together”
- “The global surface temperature record, 1880 to the present”

Part 2: Making the dates real (about 12 minutes)

Work from the inner merge for the rest of the exercise; make sure to give this new dataframe a name you will not confuse with anything else.

Copy the inner merge into a table called climate, ending the line with .copy(). Then parse its Date column into a new column called date. Demonstrate that the parse worked by printing the new column’s dtype.
Add year and month columns using the .dt accessors.
Build a table of annual means: group by year and report the count of months, the mean of MonthlyAnomaly, and the mean of CO2Concentration. Show the first three rows and the last three rows.
Two of the sixty-seven years in the table are not twelve months long. Which two, and why? Filter the data into a new table called full_years. How many years survive?

Part 3: The size of the annual cycle (about 12 minutes)

Build a wide table with year down the rows, month across the columns, and CO2Concentration in the cells. Show the first three rows and the last three rows.
The wide table holds two different patterns at once. Read it down a single column, then read it across a single row. In a markdown cell, describe both in one sentence each.
For every year, the difference between its largest monthly value and its smallest is the size of the annual cycle. Compute that difference for two specific years, 1959 and 2023, and report both. (You will need two lines of code; one for each year)
In a markdown cell: the annual cycle in CO₂ is caused by vegetation (mainly in the northern hemisphere), which removes carbon from the atmosphere during the growing season and releases it back in the autumn/winter. Given that context, explain what your two numbers from question 12 might indicate. Are two years sufficient to make that inference?
Build the same wide table for MonthlyAnomaly. Does the temperature record have a comparable seasonal cycle in it? Say why or why not in one sentence.

Part 4: Making a statement about change (about 9 minutes)

Using full_years, compare the 1960s with the 2010s. Compute the mean of each column for year between 1960 and 1969, and again for year between 2010 and 2019, and report the change in each.
In a markdown cell of four or five sentences: state what these calculations indicate in terms of changes in the temperature and CO2 concentrations, with units. You have put two variables in the same table and found that both varied together; explain what a data scientist would need before making a causal claim about this covariation.

Before you stop

Check that:

you can say, without peeking at your notes, what how='inner' did to your row count and why!
every mean you reported has a count beside it, or a filter that guaranteed the count
you supplied format= to every pd.to_datetime() call
your notebook reads top to bottom as a document, not as a mishmash of cells