The DataFrame — a spreadsheet you drive with code. Loading a file, then selecting and filtering to the rows and columns you need.
Programming for Data Science · University of the Philippines Cebu
Read a CSV into a DataFrame and take a first, honest look.
Pull out columns, and rows by label or by position.
Keep only the rows that match a condition you write.
pandas is the tool you'll spend the most hours in. It's NumPy with column names and an index attached.
Load a real CSV and answer a question by filtering it.
NumPy gave you speed, but column 2 means nothing to a reader. A DataFrame keeps that speed and adds names and an index so the data explains itself.
One labelled column — a NumPy array with an index (a label for each row).
Many Series side by side — the whole table.
pandas’ table: rows and named columns, like a spreadsheet you control with code.
e.g. df = pd.read_csv("cities.csv")
One column of a DataFrame: a NumPy array plus a label for each row.
e.g. df["pop"]
One named field that every row has — a vertical strip of the table.
e.g. city, region, pop
One record: all the values about one thing (one city, one project).
e.g. Cebu · Visayas · 964,000
The labels down the left side that name each row. By default 0, 1, 2… and they stay with their row.
e.g. df.loc[0] is the row labelled 0
The type of one column. object is pandas’ word for text.
e.g. pop is int64, city is object
Two ways to pick rows and columns: .loc by label, .iloc by position (counting from 0).
e.g. df.loc[0, "city"], df.iloc[0, 0]
A True/False Series used inside df[ ] to keep only the rows where it is True.
e.g. df[df["pop"] > 1_000_000]
“Not a Number”: pandas’ marker for an empty cell. Summaries skip it; you count it with .isna().
e.g. a project with no start date
Writing several methods in a row, each working on the result of the one before.
e.g. df.sort_values("pop").head(3)
Load a file; look before you leap.
Last week you did maths on whole arrays. pandas wraps those arrays in a table with column names and row labels — a spreadsheet you control with code.
read_csv does the heavy liftingA CSV file (comma-separated values) is a plain-text table: one row per line,
commas between the columns. Point pandas at one and it parses the header (the first line of column
names), infers each column's type, and hands back a DataFrame — a table of
rows and named columns. .head() shows the first five rows.
import pandas as pdload pandas under its usual nickname pddf = pd.read_csv("cities.csv")read the file into a DataFrame and call it df (short for data frame)df.head()a method: show the first 5 rows as a tabledf.shapean attribute (no brackets): (16, 3) = 16 rows, 3 columnsNames, types, size, and a summary — before any analysis. Wrong types here cause every bug later, exactly as in the Data Preparation phase of DS 227, the companion course.
df.columnsthe column names, in orderdf.dtypeseach column’s type: int64 = whole numbers, object = textdf.info()types plus how many values are present in each column, so missing ones show updf.describe()count, mean, spread, min, quartiles (the 25/50/75% marks) and max of every numeric column.head() shows you the values. .info() shows you what pandas thinks they are — which is what every later bug comes from.
Anything numeric showing object is text. Sorting, summing and comparing will all be wrong, and none of them will raise.
RangeIndex: 34079 entries, 0 to 3407834,079 rows, labelled 0 to 34,078Non-Null Counthow many cells in that column are filled in: start_date has 33,106, so 973 are missingstart_date … object <-the dates are stored as text, not as dates — the arrow marks a column to fix (two slides on). budget is float64: already numbersmemory usage: 3.9+ MBa quick estimate of the computer memory the table takes (MB = millions of bytes); the + means text was only partly countedAn object column stores a separate Python string per row. A category column stores each distinct value once, plus small integer codes.
18 regions across 34,079 rows means each region name is stored 1,893 times as an object — and once as a category.
df.memory_usage(deep=True).sum() / 1e6total bytes used by every column (deep=True counts the text fully), divided by a million = megabytes: 28.5 MBdf[c] = df[c].astype("category").astype converts a column to another dtype; the loop does it for three columns# and groupby gets faster toogroupby = summarising per group (e.g. per region) — next week’s topicA NaN (“Not a Number”) is pandas’ marker for a missing value. Convert the types immediately after reading the file. Every line you write after that point can then assume the types are right.
Bad values become NaN instead of raising, so you can count them and go look rather than guessing.
df.budgeta shortcut for df["budget"]; it works when the column name has no spacespd.to_numeric(…, errors="coerce")turn text into numbers; anything that is not a number becomes NaN instead of crashingpd.to_datetime(…)the same for dates: text becomes a real date type (datetime64).isna().sum().isna() is True where a value is missing; summing counts them: (0, 973)The index labels rows; the columns name what each holds; a row is one record (one city). You'll select along both.
A DataFrame is a spreadsheet you control with code. The column names are the header row and the index is the row numbers down the left — except the labels travel with their row.
A single column pulled out is a Series — the NumPy array from last week, labelled.
Columns, and rows by label or position.
You know the parts of a DataFrame. This part pulls pieces out — one column, several columns, or rows by label or position — like list indexing from week 3, but with names.
One name in single brackets gives a Series. A list of names in double brackets gives a smaller DataFrame. The double bracket is a list inside the select.
df["pop"]one name in the brackets: one column back, as a Seriesdf[["city", "pop"]]the inner brackets are a Python list of names; a list in → a smaller DataFrame out.loc by label, .iloc by position.loc | .iloc | |
|---|---|---|
| Reaches by | index label | integer position |
| First row | df.loc[0] | df.iloc[0] |
| A slice | df.loc[0:2] — inclusive | df.iloc[0:2] — exclusive |
| Accepts a mask | yes — for filtering | no |
The slice rule differs: .loc[0:2] includes row 2;
.iloc[0:2] stops before it, like a normal Python slice.
.loc finds a guest by the name on their badge; .iloc counts seats from the door. Shuffle the guests and the seat numbers change, but the badges do not.
Know the label → .loc. Know the position → .iloc.
The inclusive/exclusive difference is the one that bites. .loc[0:2] gives three rows; .iloc[0:2] gives two. Both are correct — they are answering different questions.
region and 4 is budgetAfter filtering, the index keeps the original labels. This is a feature — you can always trace a row back — but it surprises people who expect 0, 1, 2.
Leave off drop=True and the old index is kept as a new column, which is occasionally what you want and usually not.
big = df[df.budget > 1e9]keep the rows with a budget over one billionIndex([18777, 28049, 33185], …)the three rows kept their original labels (int64 = the labels are whole numbers)big.loc[0]raises KeyError: the last line would read KeyError: 0 — no row is labelled 0 any morereset_index(drop=True)give the rows fresh labels 0, 1, 2 and throw the old ones away (stop=3 means up to, not including, 3)Grab a single row by its label, or hand .loc a condition to keep
every row where it's true. That second form is filtering — Part C.
df.loc[0]the row labelled 0, returned as a Series whose labels are the column namesdf.iloc[-1]the row in the last position (-1 counts from the end, as with lists)df.loc[df["city"] == "Cebu"]a True/False Series inside .loc keeps only the matching rowsWhat type does df["pop"] return?
B — a Series.
Single brackets with one name give a Series.
For a one-column DataFrame instead, use double brackets:
df[["pop"]].
One bracket → Series. Two brackets → DataFrame. The shape follows the brackets.
df["c"] Series · df[["c"]] DataFrame.
Back to keep only the rows that answer your question.
Keep the rows that match a condition.
Last week you kept array elements with a boolean mask. The same move works on a DataFrame — it keeps whole rows.
Boolean filtering keeps rows by a rule. A comparison on a column gives a True/False Series. Put it in the
brackets and pandas keeps the rows where it's True — the NumPy mask, on a
table.
df["pop"] > 1_000_000one True/False per row; the underscores in 1_000_000 are only there to make it readabledf[df["pop"] > 1_000_000]that Series inside df[ ]: keep the rows marked True, drop the restIt is Excel’s filter button, except the rule is written down, so anyone can re-run it and get the same rows.
.str is an accessor: a doorway to a group of methods for one kind of column (text). Put .str in front and the familiar Python string methods work on all 34,079 rows at once — and skip NaN instead of crashing.
Without it, missing values come back as NaN rather than False, and using the result as a mask raises.
.str.contains("FLOOD", case=False, na=False)True where the description mentions flood, in any capitals; missing descriptions count as False.sum()True counts as 1, so the sum is how many rows matched: 22,022.str.len().mean()the length of every description, then their average: about 139 charactersOnce a column is datetime64, .dt gives you year, month, weekday and the rest — each as a normal column you can group by.
On an object column .dt raises. That error is usually telling you the pd.to_datetime step is missing.
df.start_date.min(), .max()the earliest and latest start dates in the table.dt.monththe month number (1–12) of each date.value_counts().sort_index()method chaining: count each month, then order the result by month number instead of by count| Month | Projects started | Share |
|---|---|---|
| February | 9,416 | 28.4% |
| March | 7,122 | 21.5% |
| April | 3,972 | 12.0% |
| every other month | 12,596 | 38.0% |
That is not weather. Construction season would spread across the dry months; this spikes in two. It is almost certainly the budget release cycle — the pattern is in the administration, not the rivers.
Convert to datetime, then .dt.month.value_counts(). Finding shapes like this is what the accessor is for.
& and |Use & for and, | for or — not the words
and/or. Wrap each condition in parentheses, or precedence bites you.
big = df["pop"] > 500_000store the first rule as a True/False Series called bigvis = df["region"] == "Visayas"the second rule; == asks “is it equal?”df[big & vis]& = and: rows where both are Truedf[big | vis]| (the vertical bar key) = or: rows where at least one is Trueand and missing parentheses fail&, not andPython's and works on single True/False values, not on a whole Series. On a
Series it raises a ValueError.
& binds tighter than > (Python does & first), so
a > 5 & b < 3 groups wrongly. Always
(a > 5) & (b < 3).
These two mistakes account for most early pandas errors. Fix them once and filtering becomes reliable.
Parenthesise each condition; join with & or |.
pandas 3.0 made Copy-on-Write permanent. The old SettingWithCopyWarning is gone.
So far you have only read from a DataFrame. This part is about changing one. Copy-on-Write means every filtered result behaves as its own copy. (The in-browser labs still run pandas 2.2, so you may meet the old warning there.)
Filter, assign, and the frame you filtered from is untouched. No warning, no ambiguity — the copy is guaranteed.
Filtering is photocopying the pages you need: scribbling on the photocopy never changes the book. To change the book, write in it directly — that is the next slide’s .loc.
Under pandas 2.x this same code printed a SettingWithCopyWarning and might or might not have written through. That uncertainty is what 3.0 removed.
dfa tiny 4-row frame with a text column g and a number column asub = df[df.g == "x"]the rows where g is "x" (two of them), as a new framesub["a"] = 99set column a to 99 — in sub only.tolist()turn a column into a plain Python list for printing; df.a is still [1, 2, 3, 4]If the goal is to change the original, say so explicitly: one .loc with the mask and the column, on df itself.
Filtering into a variable and editing that variable now provably does nothing to the original. That is safer, but only if you know it.
df.loc[df.g == "x", "a"] = 77rows where the mask is True, column a, on df itself: set to 77[77, 77, 3, 4]the two "x" rows changed in the originalpandas could not tell whether your object was a view or a copy, so it warned that your assignment might silently do nothing.
Copy-on-Write removed the ambiguity: a filter is always a copy, always.
Use one .loc[mask, col] = value on the frame you intend to modify. Correct on both versions.
A SettingWithCopyWarning is pandas 2.x’s message that a change might not reach the original table. Older notebooks, Stack Overflow answers (a question-and-answer website for programmers) and blog posts are full of .copy() calls added to silence this warning. They are harmless — just no longer necessary.
value_counts() tallies a text column; .mean() and
describe() summarise numbers. Chain them onto a filtered frame to answer real
questions.
df["region"].value_counts()how many rows have each region, most common firstdf["pop"].mean()the average population across the rowsRaw counts are hard to compare across datasets. normalize=True gives proportions instead.
"80.8% of projects are completed" travels. "27,534 projects" makes the reader do the division.
df.status.value_counts()a Series: each status (as the label) with its countnormalize=Truegive fractions of the total instead: 0.8079 = 80.79% of projectssort_values(...).head(n) is method chaining — sort, then take the first n — and it sorts all 34,079 rows to show you three. nlargest does not.
Sorting a DataFrame keeps whole rows, so you still have the region and the description next to the number.
df.nlargest(3, "budget")the 3 whole rows with the biggest budget[["budget", "region"]]then keep just two columns to displaydf.nsmallest(3, "budget")the other end: the 3 smallest budgetsA readability choice, not a speed one. Long boolean expressions get hard to scan; query keeps them close to the sentence you would say out loud.
No repeated df., no brackets around every clause. Use @name to refer to a Python variable.
df.query("region == 'Region III' and …")the filter written as one string; inside it, and is allowed"…" "…"Python joins strings that sit side by side, so a long condition can span lines@floor@ means “the Python variable called floor”, not a columnPython’s and wants a single True/False. A pandas mask is 34,079 of them, so it cannot decide — and says so in a way that reads like nonsense until you have seen it once.
Use & | ~, and parenthesise every clause — & binds tighter than >.
ValueError: The truth value of a Series is ambiguousread the last line: and asked the whole Series for one True/False, and pandas cannot choose oneUse a.empty, a.bool(), a.item(), a.any() or a.all()Python’s suggestions are not what you want here — the fix is & with parenthesesdf[(…) & (…)]each condition in its own brackets, joined by &On numbers you get quartiles. On an object or category column you get the four things that actually matter for text.
18 unique regions is a groupable column. 34,079 unique descriptions is free text — treat it with .str, not groupby.
counthow many values are present (not missing)uniquehow many different values: 18 regionstop / freqthe most common value and how many times it appearsfillna replaces missing values (NaN) with something you choose. Whether to fill at all is a judgement call (DS 227, the companion course, debates it). This slide is how — and the choice of fill value is itself a claim about the data.
Or a category with the mean. A fill must be a plausible member of that column, or it will distort every summary that follows.
fillna("Not yet awarded")every missing contractor gets the same textfillna(df.budget.median())missing budgets get the middle budget of the whole tablegroupby("region").budget.transform("median")each missing budget gets its own region’s median (groupby is next week)isna().sum()count the gaps before and after, so you know how many you changed.apply() calls your Python function once per row. A lambda is a one-line function with no name: lambda x: x / 1e6 takes x and gives back x ÷ 1,000,000. A vectorised expression hands the whole column to NumPy’s fast C code at once (week 5).
When there is genuinely no vectorised equivalent — complex string parsing, a lookup with branching. Reach for it second, not first.
df.budget.apply(lambda x: x / 1e6)run the little function 34,079 times, once per budgetdf.budget / 1e6the same division as one vectorised operation (week 5): the whole column at once2.33 ms vs 0.044 msmilliseconds: the vectorised version is about 52 times fasterRead the flood CSV. Run .info(). List every column pandas guessed as object that should be numeric or a date.
Convert them with errors="coerce". Count what became NaN in each.
Filter to one region into sub. Assign a new column on sub. Check df — is it changed? Now do it properly with .loc.
Which region has the highest median budget? Use groupby, then nlargest. Is it the same region as the highest total?
Step 4 is the interesting one: biggest total and biggest typical project are usually different regions, and the difference is the story.
df.groupby("region").budget.median() — one median per region, as a SeriesYou filter sub = df[df.region == "Region III"] then run sub["flag"] = 1. On pandas 3.x, what happens to df?
df is untouched — with no warning. To change the original, use one statement: df.loc[df.region == "Region III", "flag"] = 1. (On pandas 2.x this same code raised SettingWithCopyWarning.)df.loc[0:2] and df.iloc[0:2] on the same frame return different numbers of rows. Why?
.loc[0:2] gives 3 rows, .iloc[0:2] gives 2.Filter to one region, then sort and take the top row. Load, select, filter, summarise — the whole loop of this week, in four lines.
vis = df[df["region"] == "Visayas"]filter: only the Visayan rowsvis["pop"].mean()the average population of those citiesvis.sort_values("pop").tail(1)sort smallest to biggest, then .tail(1) keeps the last row: the biggest cityWhich keeps rows that are both big and Visayan?
df[big and vis]df[big & vis]df[big, vis]df[big | vis]B — df[big & vis].
Element-wise AND on Series uses
&. The word and (A) raises an error; | (D)
is OR, not AND.
& = and, | = or, each condition in parentheses.
Symbols, not words, for Series logic.
You'll build a small table of Philippine cities, inspect its types, select columns, filter with single and combined conditions, sort, and count to answer set questions. ~45 minutes.
.describe, column selection, boolean filters, isin, sort_values,
.loc/.iloc, value_counts, and a first taste of groupby.
Use .query("pop > 500000") as a readable filter alternative.
read_csv, then .head/.info/.describe before anything else.
Columns by name; rows by .loc (label) or .iloc (position).
A condition in brackets; combine with & / | and
parentheses.
One sentence: read the file, look at it, then keep the rows and columns your question needs.
Turn a raw CSV into exactly the slice you want to study.
.isna().sum()df[mask]: keep the rows where the mask is True.sort_values().head().locFinish the Week 6 lab and submit it. Be able to filter with a combined condition from memory.
pandas "10 minutes to pandas" — pandas.pydata.org/docs/user_guide/10min.
Everything here is linked on the course page beside this deck.
Next week: grouping, merging, and reshaping — where pandas earns its keep.
Split-apply-combine, joining two tables on a shared key, and pivoting long data into wide — the moves behind every real analysis.
DS 208 · Programming for Data Science