Interactive Session

A panda at a farmers market, choosing his vegetables. MidJourney 5

Earlier this morning you wrote a question and asked it of an entire column of the Toolik weather table. The answer came back as a boolean mask: a True or a False for every one of its 11,171 rows.

A mask on its own does not do much for us. The rows it says True about are what you are after, and getting them takes one line of code, which we will write in nearly every session from here on.

By the end of this session you will be able to:

write the filter pattern, df[df['col'] > value], without looking it up
filter on text as well as on numbers
combine two conditions with & and |, and get the parentheses right
invert a condition with ~
match against a list of values with .isin()
say what .copy() is for, and use it when you intend to keep a result

Getting Started

Create the file. In the Explorer, hover over the EDS217 heading and click New File…, then type the name in full, extension included: Session_3B_Filtering_Data.ipynb
Check the kernel. The Kernel Selector in the notebook’s action bar should read Python 3.11.15 (Conda: eds217). If it reads anything else, click it, choose Change Kernel, and pick the eds217 entry.
Add a title cell. Click + Markdown in the action bar, and give it this content, with today’s date:

# Day 3: Session 3B - The Filter Pattern

[Session Webpage](https://eds-217-essential-python.github.io/course-materials/interactive-sessions/3b_filtering_data.html)

Date: 09/02/2026

Save with Ctrl + S (Cmd + S on macOS), and keep saving as you go.
Read in the Goleta air quality data you explored yesterday:

Code

import pandas as pd

url = 'https://eds-217-essential-python.github.io/data/openaq_goleta_measurments.csv'
goleta = pd.read_csv(url)
goleta.shape

(1962, 15)

The pattern

Here it is, whole:

Code

goleta[goleta['value'] > 20]

	location_id	location_name	parameter	value	unit	datetimeUtc	datetimeLocal	timezone	latitude	longitude	country_iso	isMobile	isMonitor	owner_name	provider
720	1186	Goleta	pm10	21.0	µg/m³	2024-07-17T14:00:00+00:00	2024-07-17T07:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
746	1186	Goleta	pm10	21.0	µg/m³	2024-07-22T20:00:00+00:00	2024-07-22T13:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
795	1186	Goleta	pm10	38.0	µg/m³	2024-07-24T21:00:00+00:00	2024-07-24T14:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
796	1186	Goleta	pm10	36.0	µg/m³	2024-07-24T22:00:00+00:00	2024-07-24T15:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
797	1186	Goleta	pm10	23.0	µg/m³	2024-07-24T23:00:00+00:00	2024-07-24T16:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
1226	1186	Goleta	pm10	27.0	µg/m³	2024-08-11T23:00:00+00:00	2024-08-11T16:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1227	1186	Goleta	pm10	23.0	µg/m³	2024-08-12T00:00:00+00:00	2024-08-11T17:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1376	1186	Goleta	pm25	22.0	µg/m³	2024-07-18T08:00:00+00:00	2024-07-18T01:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1712	1186	Goleta	pm25	22.0	µg/m³	2024-08-01T08:00:00+00:00	2024-08-01T01:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1873	1186	Goleta	pm25	21.0	µg/m³	2024-08-08T08:00:00+00:00	2024-08-08T01:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow

115 rows × 15 columns

Read it from the inside out. goleta['value'] > 20 is the same kind of mask you wrote this morning, on a different table: True on every row where the value is above 20, False everywhere else. Wrapping it in goleta[ ... ] says “give me the rows where that was True.”

The line we just ran is the filter pattern. The general form is:

df[df['column'] <comparison> value]

The df appears twice, which looks redundant until you see what each one is doing. The inner one builds the question, the outer one applies the answer, and they are the same table both times.

Give the result a name if you want to use it later:

Code

high_readings = goleta[goleta['value'] > 20]
high_readings.shape

(115, 15)

115 rows out of 1,962. Compare the shapes before and after, and you can see exactly how many rows the filter kept:

Code

print('before:', goleta.shape)
print('after: ', high_readings.shape)

before: (1962, 15)
after:  (115, 15)

Note

🐍 Filtering never changes the original. goleta still has all 1,962 rows. You get back a new, smaller table, and if you don’t give it a name it is displayed once and then discarded. .rename() followed the same rule yesterday.

✏️ Test your knowledge

Filter goleta to the rows where value is below zero. How many are there? A concentration below zero is not physically possible, so make a note of what you find; we will deal with values like these on Day 4.

Filtering on text

The same pattern works on text columns. Use == for an exact match:

Code

ozone = goleta[goleta['parameter'] == 'o3']
ozone.head()

	location_id	location_name	parameter	value	unit	datetimeUtc	datetimeLocal	timezone	latitude	longitude	country_iso	isMobile	isMonitor	owner_name	provider
0	1186	Goleta	o3	0.025	ppm	2024-07-12T01:00:00+00:00	2024-07-11T18:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1	1186	Goleta	o3	0.028	ppm	2024-07-12T02:00:00+00:00	2024-07-11T19:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
2	1186	Goleta	o3	0.029	ppm	2024-07-12T03:00:00+00:00	2024-07-11T20:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
3	1186	Goleta	o3	0.027	ppm	2024-07-12T04:00:00+00:00	2024-07-11T21:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
4	1186	Goleta	o3	0.026	ppm	2024-07-12T05:00:00+00:00	2024-07-11T22:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow

Code

ozone.shape

(711, 15)

711 ozone readings. Text comparisons are exact and case-sensitive, so 'O3' would have matched nothing at all:

Code

goleta[goleta['parameter'] == 'O3'].shape

(0, 15)

An empty result is not an error. Zero rows is a real answer, and it usually means a spelling or capitalization problem rather than a fact about the data.

✏️ Test your knowledge

Filter goleta to the pm25 readings and store them in a variable called pm25. How many rows? Check your number against goleta['parameter'].value_counts() from yesterday.

Why filtering matters for the Goleta data

Yesterday you found that goleta['value'] stacks three different pollutants, in two different units, in one column. Mixing three pollutants together made .describe() on the whole column meaningless.

Filtering is the fix:

Code

goleta['value'].describe()

count    1962.000000
mean        6.378173
std         7.313763
min        -4.000000
25%         0.027000
50%         5.000000
75%        10.000000
max        40.000000
Name: value, dtype: float64

Code

goleta[goleta['parameter'] == 'pm25']['value'].describe()

count    734.000000
mean       6.480926
std        3.651091
min       -4.000000
25%        4.000000
50%        6.000000
75%        8.000000
max       22.000000
Name: value, dtype: float64

The first summary averages micrograms per cubic metre together with parts per million, which gives a mean that describes no substance at all. The second one describes 734 particulate readings, all in µg/m³, and only the second one tells you anything about the air in Goleta.

Two conditions at once

& means “and”. Every comparison gets its own parentheses:

Code

goleta[(goleta['parameter'] == 'pm25') & (goleta['value'] > 9)]

	location_id	location_name	parameter	value	unit	datetimeUtc	datetimeLocal	timezone	latitude	longitude	country_iso	isMobile	isMonitor	owner_name	provider
1261	1186	Goleta	pm25	14.0	µg/m³	2024-07-13T10:00:00+00:00	2024-07-13T03:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1265	1186	Goleta	pm25	10.0	µg/m³	2024-07-13T14:00:00+00:00	2024-07-13T07:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1270	1186	Goleta	pm25	10.0	µg/m³	2024-07-13T19:00:00+00:00	2024-07-13T12:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1313	1186	Goleta	pm25	12.0	µg/m³	2024-07-15T14:00:00+00:00	2024-07-15T07:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1314	1186	Goleta	pm25	11.0	µg/m³	2024-07-15T15:00:00+00:00	2024-07-15T08:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
1953	1186	Goleta	pm25	16.0	µg/m³	2024-08-11T16:00:00+00:00	2024-08-11T09:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1954	1186	Goleta	pm25	16.0	µg/m³	2024-08-11T17:00:00+00:00	2024-08-11T10:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1955	1186	Goleta	pm25	13.0	µg/m³	2024-08-11T18:00:00+00:00	2024-08-11T11:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1956	1186	Goleta	pm25	14.0	µg/m³	2024-08-11T19:00:00+00:00	2024-08-11T12:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1957	1186	Goleta	pm25	10.0	µg/m³	2024-08-11T20:00:00+00:00	2024-08-11T13:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow

128 rows × 15 columns

128 hours where fine particulate matter was above 9 µg/m³, which is the annual air quality standard the US EPA set in 2024, tightened from the 12 µg/m³ that had stood since 2012.

| means “or”:

Code

particulates = goleta[(goleta['parameter'] == 'pm25') | (goleta['parameter'] == 'pm10')]
particulates.shape

(1251, 15)

Each side of an & or a | can itself be a two-part condition. When it is, wrap that whole side in a further pair of parentheses so the two questions stay separate:

Code

either_pollutant_high = goleta[
    ((goleta['parameter'] == 'pm25') & (goleta['value'] > 9))
    | ((goleta['parameter'] == 'pm10') & (goleta['value'] > 20))
]
either_pollutant_high.shape

(240, 15)

240 rows: 128 hours of pm25 above 9, plus 112 hours of pm10 above 20. Read the line from the inside out. Each innermost pair is one comparison, each middle pair is one complete question about one pollutant, and the | joins the two questions.

The parentheses are not optional

Leave them out and you get an error that has nothing to do with what you meant:

goleta[goleta['parameter'] == 'pm25' & goleta['value'] > 9]

& binds more tightly than == and >, so Python combines the wrong pieces before it ever gets to your comparison.

The rule that avoids all of this: put parentheses around each comparison you join with & or |. A comparison is something with ==, >, <, >=, <= or != in it. A method call like .isin([...]) is not a comparison and does not need its own pair, though adding them does no harm if you would rather not think about which is which.

✏️ Test your knowledge

Find the ozone readings above 0.04 ppm. Then, using the two-level shape above, find the readings that are either ozone above 0.04 ppm or pm25 above 20 µg/m³. How many rows does each give you?

Turning a condition around with `~`

~ inverts a mask: every True becomes False and every False becomes True.

Code

not_ozone = goleta[~(goleta['parameter'] == 'o3')]
not_ozone.shape

(1251, 15)

1,251 rows, which is 1,962 minus the 711 ozone readings. You could have written != here and gotten the same answer:

Code

goleta[goleta['parameter'] != 'o3'].shape

(1251, 15)

~ is most useful when the condition is more complicated than a single comparison, and you would rather say “not that” than rewrite the whole condition backwards.

Matching a list of values with `.isin()`

Writing (col == 'a') | (col == 'b') | (col == 'c') gets tedious fast. .isin() takes a list and matches any of its members:

Code

particulates = goleta[goleta['parameter'].isin(['pm25', 'pm10'])]
particulates.shape

(1251, 15)

Same 1,251 rows as the | version above, in a line that stays readable when the list grows.

The argument is an ordinary Python list, so you can build it first and pass it by name:

Code

of_interest = ['pm25', 'pm10']
goleta[goleta['parameter'].isin(of_interest)].shape

(1251, 15)

Note

🐍 Call .isin() on the column, not on the DataFrame: df['col'].isin([...]). It returns a mask, exactly like a comparison does, so it goes in the same place inside the brackets.

✏️ Test your knowledge

Build a list called keep_parameters holding 'o3' and 'pm10', then use .isin() to filter goleta down to those two pollutants. Check your row count against the two numbers in goleta['parameter'].value_counts(); they should add up.

Keeping a result with `.copy()`

Once you have a subset you like, you will usually want to keep working on it, and often that means adding a column to it later this week.

When you plan to keep and modify a filtered result, say .copy():

Code

pm25 = goleta[goleta['parameter'] == 'pm25'].copy()
pm25.shape

(734, 15)

.copy() gives you a table of your own, with no remaining connection to goleta. Without it, the subset you get back may still be tied to the original table. When you then try to change the subset, you get either a warning that is hard to interpret or a change that silently never happens.

The habit is easy to state:

If the filtered result is going to be a thing you keep and change, end the line with .copy(). If you are just looking at it, don’t bother.

Code

pm25.head()

	location_id	location_name	parameter	value	unit	datetimeUtc	datetimeLocal	timezone	latitude	longitude	country_iso	isMobile	isMonitor	owner_name	provider
1228	1186	Goleta	pm25	3.0	µg/m³	2024-07-12T01:00:00+00:00	2024-07-11T18:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1229	1186	Goleta	pm25	8.0	µg/m³	2024-07-12T02:00:00+00:00	2024-07-11T19:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1230	1186	Goleta	pm25	6.0	µg/m³	2024-07-12T03:00:00+00:00	2024-07-11T20:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1231	1186	Goleta	pm25	4.0	µg/m³	2024-07-12T04:00:00+00:00	2024-07-11T21:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow
1232	1186	Goleta	pm25	9.0	µg/m³	2024-07-12T05:00:00+00:00	2024-07-11T22:00:00-07:00	America/Los_Angeles	34.445301	-119.827797	NaN	NaN	NaN	Unknown Governmental Organization	AirNow

A warning you will meet

SettingWithCopyWarning shows up when a change you have written could apply either to the subset or to the original table, and the code as written does not say which one you meant. It is not always a real problem, but it is always worth reading! Adding .copy() at the point where you made the subset is usually the fix.

✏️ Test your knowledge

Make a .copy() of the ozone readings called ozone, then confirm with .shape and .head() that you have what you expect. Keep it in your notebook; you’ll use it in the next session.

Putting it together

A real filter is usually two or three of these ideas in one line:

Code

summer_particulates = goleta[
    goleta['parameter'].isin(['pm25', 'pm10']) & (goleta['value'] > 15)
].copy()

summer_particulates.shape

(246, 15)

Code

summer_particulates[['parameter', 'value', 'unit', 'datetimeLocal']].head()

	parameter	value	unit	datetimeLocal
711	pm10	20.0	µg/m³	2024-07-16T22:00:00-07:00
720	pm10	21.0	µg/m³	2024-07-17T07:00:00-07:00
721	pm10	20.0	µg/m³	2024-07-17T08:00:00-07:00
746	pm10	21.0	µg/m³	2024-07-22T13:00:00-07:00
748	pm10	16.0	µg/m³	2024-07-22T15:00:00-07:00

Notice that the filter is broken across several lines inside the square brackets. Python allows a line break anywhere inside a pair of brackets, and a long filter is much easier to read that way.

Key points

The filter pattern is df[df['col'] <comparison> value]. The inner df builds the mask; the outer one applies it.
Filtering returns a new DataFrame. Assign it to a name to keep it.
== matches text exactly, and is case-sensitive. An empty result is an answer, not an error.
Combine conditions with & for and, | for or, ~ for not. Every comparison needs its own parentheses.
df['col'].isin([...]) matches any value in a list, and replaces a pile of |.
End a filter with .copy() when you plan to keep the result and change it later.

Getting Started

The pattern

Filtering on text

Why filtering matters for the Goleta data

Two conditions at once

Turning a condition around with ~

Matching a list of values with .isin()

Keeping a result with .copy()

Putting it together

Key points

Resources

Turning a condition around with `~`

Matching a list of values with `.isin()`

Keeping a result with `.copy()`