Interactive Session

A panda, investigating. MidJourney 5

In the last session we read the Goleta air quality file into a DataFrame and pulled a few columns back out of it. This morning we will look at what is actually in the DataFrame: how big the table is, what the columns are called, what data type pandas assigned to each column, where the missing values are, and what the numbers look like.

Exploring is the easiest step in the whole workflow to skip, because skipping it costs you nothing right away. We are going to do it properly, on the same Goleta file, and along the way we will find three columns with nothing in them at all, a negative pollutant concentration, and a mean that was computed by averaging two different units together.

Every one of those would have been a real problem for an analysis built on this file, so the lesson here is that you should always spend time at the beginning of a data science workflow exploring your datasets and examining them closely. This will make all subsequent analyses easier!

By the end of this session you will be able to:

size up a DataFrame with .shape, .columns, and .dtypes
run a health check with .info() and .isnull().sum()
summarize numbers with .describe(), and know when not to trust it
count categories with .value_counts()
rename columns using a Python dictionary

Getting Started

Create the file. In the Explorer, hover over the EDS217 heading and click New File…, then type the name in full, extension included: Session_2B_Exploring_Data.ipynb
Check the kernel. The Kernel Selector in the notebook’s action bar should read Python 3.11.15 (Conda: eds217). If it reads anything else, click it, choose Change Kernel, and pick the eds217 entry.
Add a title cell. Click + Markdown in the action bar, and give it this content, with today’s date:

# Day 2: Session 2B - Exploring a DataFrame

[Session Webpage](https://eds-217-essential-python.github.io/course-materials/interactive-sessions/2b_exploring_data.html)

Date: 09/01/2026

Save with Ctrl + S (Cmd + S on macOS), and keep saving as you go.
Read the data. Add a code cell, and read in the same Goleta air quality file we used in the last session:

Code

import pandas as pd

url = 'https://eds-217-essential-python.github.io/data/openaq_goleta_measurments.csv'
goleta = pd.read_csv(url)

Look at it

Always start by looking. .head() shows the first five rows, .tail() the last five rows:

Code

goleta.head()

	location_id	location_name	parameter	value	unit	datetimeUtc	datetimeLocal	timezone	latitude	longitude	country_iso	isMobile	isMonitor	owner_name	provider
0	1186	Goleta	o3	0.025	ppm	2024-07-12T01:00:00+00:00	2024-07-11T18:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1	1186	Goleta	o3	0.028	ppm	2024-07-12T02:00:00+00:00	2024-07-11T19:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
2	1186	Goleta	o3	0.029	ppm	2024-07-12T03:00:00+00:00	2024-07-11T20:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
3	1186	Goleta	o3	0.027	ppm	2024-07-12T04:00:00+00:00	2024-07-11T21:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
4	1186	Goleta	o3	0.026	ppm	2024-07-12T05:00:00+00:00	2024-07-11T22:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow

Code

goleta.tail()

	location_id	location_name	parameter	value	unit	datetimeUtc	datetimeLocal	timezone	latitude	longitude	country_iso	isMobile	isMonitor	owner_name	provider
1957	1186	Goleta	pm25	10.0	µg/m³	2024-08-11T20:00:00+00:00	2024-08-11T13:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1958	1186	Goleta	pm25	8.0	µg/m³	2024-08-11T21:00:00+00:00	2024-08-11T14:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1959	1186	Goleta	pm25	8.0	µg/m³	2024-08-11T22:00:00+00:00	2024-08-11T15:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1960	1186	Goleta	pm25	9.0	µg/m³	2024-08-11T23:00:00+00:00	2024-08-11T16:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1961	1186	Goleta	pm25	5.0	µg/m³	2024-08-12T00:00:00+00:00	2024-08-11T17:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow

Both take a number if you want more or fewer:

Code

goleta.head(3)

	location_id	location_name	parameter	value	unit	datetimeUtc	datetimeLocal	timezone	latitude	longitude	country_iso	isMobile	isMonitor	owner_name	provider
0	1186	Goleta	o3	0.025	ppm	2024-07-12T01:00:00+00:00	2024-07-11T18:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1	1186	Goleta	o3	0.028	ppm	2024-07-12T02:00:00+00:00	2024-07-11T19:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
2	1186	Goleta	o3	0.029	ppm	2024-07-12T03:00:00+00:00	2024-07-11T20:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow

How big is it?

.shape gives you (rows, columns):

Code

goleta.shape

(1962, 15)

The result is a pair of numbers, and you pull either one out by position the same way you would from a list. It is not a list, though: the round brackets mean it is a tuple, which is a list you are not allowed to change. Indexing works, .append() does not:

Code

print("rows:", goleta.shape[0])
print("columns:", goleta.shape[1])

rows: 1962
columns: 15

What are the columns, and what type is each?

Code

goleta.columns.tolist()

['location_id',
 'location_name',
 'parameter',
 'value',
 'unit',
 'datetimeUtc',
 'datetimeLocal',
 'timezone',
 'latitude',
 'longitude',
 'country_iso',
 'isMobile',
 'isMonitor',
 'owner_name',
 'provider']

Code

goleta.dtypes

location_id        int64
location_name     object
parameter         object
value            float64
unit              object
datetimeUtc       object
datetimeLocal     object
timezone          object
latitude         float64
longitude        float64
country_iso      float64
isMobile         float64
isMonitor        float64
owner_name        object
provider          object
dtype: object

object almost always means text. float64 and int64 are numbers. Types matter because they determine what you can do with a column: you can average a float64, and you cannot average an object.

✏️ Test your knowledge

How many columns in goleta are stored as object? Use .dtypes to answer, and say how you counted. Then check the three empty columns against your answer: what type did pandas give a column it never found a single value in, and is that what you expected?

The health check: `.info()`

.info() is the most useful single line in this morning’s lesson, because it combines the shape, the column names, the types, and, critically, how many non-null values each column has:

Code

goleta.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1962 entries, 0 to 1961
Data columns (total 15 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   location_id    1962 non-null   int64  
 1   location_name  1962 non-null   object 
 2   parameter      1962 non-null   object 
 3   value          1962 non-null   float64
 4   unit           1962 non-null   object 
 5   datetimeUtc    1962 non-null   object 
 6   datetimeLocal  1962 non-null   object 
 7   timezone       1962 non-null   object 
 8   latitude       1962 non-null   float64
 9   longitude      1962 non-null   float64
 10  country_iso    0 non-null      float64
 11  isMobile       0 non-null      float64
 12  isMonitor      0 non-null      float64
 13  owner_name     1962 non-null   object 
 14  provider       1962 non-null   object 
dtypes: float64(6), int64(1), object(8)
memory usage: 230.1+ KB

Read the Non-Null Count column carefully. Most columns have 1962 non-null values, one for every row. Three of them have 0.

country_iso, isMobile, and isMonitor are completely empty. They exist as column headers in the file, and there is not one single value in any of them!

Important

Empty columns are exactly what exploring turns up. The three columns are blank in every row head() shows you, but five blank rows out of 1,962 is not evidence of much, and plenty of perfectly good columns have gaps near the top. .info() is what tells you these three are empty all the way down.

You can ask the same question directly. .isnull() marks every missing value, and .sum() counts them per column:

Code

goleta.isnull().sum()

location_id         0
location_name       0
parameter           0
value               0
unit                0
datetimeUtc         0
datetimeLocal       0
timezone            0
latitude            0
longitude           0
country_iso      1962
isMobile         1962
isMonitor        1962
owner_name          0
provider            0
dtype: int64

We won’t fix any of this today, because deciding what to do about missing data is a Day 4 problem. This morning we only need to know that it is there.

✏️ Test your knowledge

The Santa Barbara station’s file sits beside the Goleta one, at https://eds-217-essential-python.github.io/data/openaq_santa_barbara_measurments.csv (the spelling of measurments is the file’s, not a mistake here). Read it into santa_barbara and run .info() on it. Does it have the same three empty columns?

Summarizing numbers with `.describe()`

.describe() gives you count, mean, standard deviation, min, max, and quartiles for every numeric column:

Code

goleta.describe()

	location_id	value	latitude	longitude	country_iso	isMobile	isMonitor
count	1962.0	1962.000000	1.962000e+03	1.962000e+03	0.0	0.0	0.0
mean	1186.0	6.378173	3.444530e+01	-1.198278e+02	NaN	NaN	NaN
std	0.0	7.313763	2.132172e-14	2.842896e-14	NaN	NaN	NaN
min	1186.0	-4.000000	3.444530e+01	-1.198278e+02	NaN	NaN	NaN
25%	1186.0	0.027000	3.444530e+01	-1.198278e+02	NaN	NaN	NaN
50%	1186.0	5.000000	3.444530e+01	-1.198278e+02	NaN	NaN	NaN
75%	1186.0	10.000000	3.444530e+01	-1.198278e+02	NaN	NaN	NaN
max	1186.0	40.000000	3.444530e+01	-1.198278e+02	NaN	NaN	NaN

Look at the value column. The mean is about 6.4 and the minimum is −4. Both of those numbers are a problem, for two different reasons.

First, a negative concentration. You cannot have less than zero of a pollutant in the air. A negative reading is usually either an instrument artifact or a sentinel value standing in for something the instrument could not report, and it is the kind of thing you flag now and decide what to do about later.

Second, the mean itself is meaningless. Look at what is in the unit column:

Code

goleta['unit'].value_counts()

unit
µg/m³    1251
ppm       711
Name: count, dtype: int64

Some rows are in micrograms per cubic metre and some are in parts per million. The mean of 6.4 was computed by averaging those two units together, so it does not describe any real quantity that anyone measured in Goleta.

.describe() will average anything you give it

.describe() averages whatever numbers are in a column, with no check on whether they belong together, and this table stacks several different pollutants, in two different units, into a single value column. Data arranged like this, one row per measurement with a column naming which kind of measurement it is, is extremely common in environmental data. Getting a real answer out of it means grouping first, which is Day 5.

Counting categories with `.value_counts()`

.describe() is for numbers. For text columns, the question is usually “what values appear, and how often?”, and .value_counts() answers it:

Code

goleta['parameter'].value_counts()

parameter
pm25    734
o3      711
pm10    517
Name: count, dtype: int64

Three pollutants, then: pm25, o3, and pm10. The unit result above makes sense now, because ozone is reported in ppm and particulates in µg/m³.

.value_counts() is also how you spot a column that tells you nothing:

Code

goleta['location_name'].value_counts()

location_name
Goleta    1962
Name: count, dtype: int64

Every row says Goleta, so the station name is a constant. A constant column is not wrong, and this one will matter on Day 6 when we stack several stations on top of each other, but within this one file it tells you nothing you did not already know.

Add .head(n) when a column has many distinct values:

Code

goleta['datetimeLocal'].value_counts().head()

datetimeLocal
2024-07-27T21:00:00-07:00    3
2024-07-28T13:00:00-07:00    3
2024-07-27T01:00:00-07:00    3
2024-07-27T00:00:00-07:00    3
2024-07-26T23:00:00-07:00    3
Name: count, dtype: int64

✏️ Test your knowledge

Use .value_counts() on the provider and owner_name columns. Is either one a constant, like location_name?

Renaming columns with a dictionary

datetimeUtc and datetimeLocal are in camelCase, while everything else uses lowercase with underscores. Mixed naming styles are an easy way to introduce typos, because you have to remember which style each column happens to use, so let’s rename those two columns to match the rest.

Renaming columns needs Python’s other essential container, the dictionary.

A dictionary stores pairs. Each pair has a key and a value, written key: value, and the whole thing goes in curly braces:

Code

new_names = {
    'datetimeUtc': 'datetime_utc',
    'datetimeLocal': 'datetime_local',
}
new_names

{'datetimeUtc': 'datetime_utc', 'datetimeLocal': 'datetime_local'}

You look up a value by its key, using square brackets:

Code

new_names['datetimeUtc']

'datetime_utc'

A list is ordered, and you get things out of it by position. A dictionary is a set of labelled pairs, and you get things out of it by name. Lookup by name is exactly what a rename needs, because a rename is a set of “this becomes that” pairs:

Code

goleta_renamed = goleta.rename(columns=new_names)
goleta_renamed.columns.tolist()

['location_id',
 'location_name',
 'parameter',
 'value',
 'unit',
 'datetime_utc',
 'datetime_local',
 'timezone',
 'latitude',
 'longitude',
 'country_iso',
 'isMobile',
 'isMonitor',
 'owner_name',
 'provider']

You can write the dictionary inline if it’s short:

Code

goleta.rename(columns={'value': 'concentration'}).columns.tolist()

['location_id',
 'location_name',
 'parameter',
 'concentration',
 'unit',
 'datetimeUtc',
 'datetimeLocal',
 'timezone',
 'latitude',
 'longitude',
 'country_iso',
 'isMobile',
 'isMonitor',
 'owner_name',
 'provider']

Note

🐍 .rename() returns a new DataFrame and leaves the original alone. If you want to keep the change, assign it to a name, as we did with goleta_renamed. Almost every pandas method behaves this way, and forgetting it is a very common source of “why didn’t my change stick?” (It still catches Cella and Kelly out, so ask if a cell seems to do nothing!)

✏️ Test your knowledge

Build a dictionary that renames value to concentration and parameter to pollutant, apply it, and confirm with .columns.tolist() that both changes took effect.

The exploration ritual

Here are the six lines to run every time you open a new dataset, in the order we ran them this morning:

df.head()                  # what does a row look like
df.shape                   # how big
df.info()                  # types, and how much of each column is filled in
df.isnull().sum()          # how many values are missing, column by column
df.describe()              # what do the numbers look like
df['col'].value_counts()   # what is in the text columns

Run them before you compute anything, because a problem you find here is quick to deal with, while the same problem found in your final figure means unpicking everything you built on top of it.

Running the five lines is the easy half. Reading what comes back is the half that finds the empty column and the mixed units, so give the output a proper look before you move on!

Key points

.shape is (rows, columns). .columns and .dtypes say what’s in the table.
.info() is the health check. Read the non-null counts, because a completely empty column looks perfectly normal everywhere else.
.isnull().sum() counts missing values per column.
.describe() summarizes numeric columns, and it will average values in different units together without any warning. Check what a column actually contains before you trust its mean.
.value_counts() counts categories, and shows you constants and unexpected values.
A dictionary maps keys to values, written {key: value}, looked up by name.
.rename(columns={...}) returns a new DataFrame. Assign it to keep it.

Getting Started

Look at it

How big is it?

What are the columns, and what type is each?

The health check: .info()

Summarizing numbers with .describe()

Counting categories with .value_counts()

Renaming columns with a dictionary

The exploration ritual

Key points

Resources

The health check: `.info()`

Summarizing numbers with `.describe()`

Counting categories with `.value_counts()`