Cleaning and organising data takes 60% of a data scientistβs time. Survey of data scientists, CrowdFlower, 2016.
So far this week, weβve been working with tables that behave themselves. Today we will learn what to do with messy data (i.e. most data!).
Day 4 has three new code patterns, and we meet them in the order the day runs. This morning weβll learn a missing-data pattern that helps you address data gaps. In addition, we get formally introduced to functions, which provide a tool for reusing code.
This afternoonβs first session covers the two remaining patterns together. First the derived-column pattern, which creates a new column in a DataFrame, because what you actually need is almost never what is contained in the raw data! Then the string-cleaning pattern, which tidies up messy text so that comparisons can work.
Class materials
| Session | Session 1 | Session 2 |
|---|---|---|
| day 4 / morning | π§Ό Missing, Duplicated, Miscast | βοΈ Write It Once, Name It, Use It |
| day 4 / afternoon | β The Derived-Column Pattern | π From Field Sheet to DataFrame |
End-of-day practice
At the end of the day today we will analyze NOAAβs marine microplastics archive, going from the raw download to a defensible answer about whether the patterns of microplastics in the Atlantic Ocean and the Pacific Ocean are different.