Week 3: Data Scraping I — Web Fundamentals & HTML Parsing

The process assumed you had data

Weeks 1–2 gave you the map: definition and process. But CRISP-DM's very first phases need data in hand. Scraping is one way to get it when nobody hands you a CSV (a plain-text table file: commas between columns).

Scraping is the last resort, not the first

A scraper is code you must maintain against a page you don't control. Check the cheaper doors first — in this order.

Every response carries a status code

Before you trust the body (the content itself, after the status and headers), check the code. A page that "looks empty" is often a 404 or 403 you didn't read.

A status code is the courier’s delivery slip: “delivered” (200), “moved, new address attached” (301), “recipient refused” (403), “no such address” (404). Read the slip before you open the box.

Requests and responses carry headers

Headers are key–value metadata (labelled facts about the message, like Content-Type: text/html) riding along with every request and response. Two matter to scrapers from day one.

The body is just text — HTML text

"View Source" in any browser shows it. It looks like chaos, but it's a strict, nested structure a program can walk. That structure is Part B.

Here li marks one item of a list and span marks a small labelled piece of text; class is the label. Part B explains every tag.

We scrape the site you're looking at

portal.latarak.com — this course portal. Target: the academic calendar page, a real table of real dates you actually need.

Two installs — on your own laptop

Scraping the open internet needs your machine — the in-portal sandbox can only reach this same site (the lab uses exactly that). One-time setup:

• pip install …typed in a terminal (a window for typed commands), not in Python: download and install three libraries
• import requests, bs4Python: load the two libraries so your code can use them
• print(requests.__version__, …)show each library’s version number. 2.34.2 4.15.0 means both are installed; yours may be newer

Your first fetch — three lines

This output is real — captured against the live portal while building this deck. The calendar has been edited since, so the live numbers may differ; the frozen snapshot on the course page still gives exactly these.

• import requestsload the requests library (ready-made code for fetching pages)
• r = requests.get("https://…")call the function get with the URL as a string: it sends the request and gives back the response, stored in the variable r
• r.status_code → 200the status code: 200 means OK
• r.headers["Content-Type"]look up one header by its name, in square brackets: the server says it sent HTML
• len(r.text) → 24337r.text is the whole page as one string; len counts its characters

Classes and ids are your handles

A page has hundreds of <div>s. What makes one findable is its class or id — the labels the site's own designers added for their CSS (the style rules that make a page look the way it does).

An HTML table is already a dataset

thead holds the header row; tbody holds data rows; each tr is a row of td cells. Your DataFrame (pandas’ table of rows and named columns) is pre-drawn.

Text in, tree out

Hand BeautifulSoup the HTML string and a parser name. You get back a soup object you can search — no string slicing, ever.

• from bs4 import BeautifulSoupload just the BeautifulSoup tool from the bs4 library
• BeautifulSoup(r.text, "html.parser")parse the page text from the fetch, using Python’s built-in HTML reader; the searchable tree goes into the variable soup
• soup.find("title")the first <title> element in the tree
• .get_text()its text without the tags: the words on the browser tab
• strip=Truea keyword argument (an input given by name): trim spaces and line breaks from both ends

get_text(strip=True) — almost always

Pretty-printed HTML is full of newlines and indentation. They all count as text. strip=True trims the edges before you store the value.

• td.get_text()all the text in the cell, including the line breaks and spaces from the HTML file
• '\n …'the quotes mean the value is a string; \n is how Python shows a line break (a newline)
• td.get_text(strip=True)the same text with spaces and newlines trimmed from both ends

.select() speaks CSS

If you know the CSS a page uses, select() takes the same selector syntax — often shorter than chained find calls. A CSS selector is a short pattern describing which elements to pick: . means class, # means id, a space means “inside”.

Fetch, then check before parsing

raise_for_status() converts a bad code into a loud error right here — not a mystery three functions later.

• three import linesload requests (fetch), BeautifulSoup (parse) and pandas, nicknamed pd (tables)
• URL = "https://…"a variable holding the address as a string; capitals are a habit for values that never change
• r.raise_for_status()a method (a function that belongs to a value, after a dot): silent if the status is 2xx, otherwise stops with an error on this line
• 200 24337status OK, and the page is 24,337 characters of HTML

Parse, then locate the table

DevTools told us the class: cal-table. Always count what you found before trusting [0] — the page has two such tables.

• soup.find_all("table", class_="cal-table")a list of every table whose class is cal-table; class_ has an underscore because plain class is reserved by Python
• len(tables) → 2the list holds two tables
• cal = tables[0]take the first item: an index is a position number, counted from 0

Extract the header row first

Column names come from thead's th cells. A list comprehension (a one-line loop that collects its results into a new list) over find_all.

• cal.find("thead")inside this table only, the header section
• .find_all("th")every header cell in it, as a list
• [th.get_text(strip=True) for th in …]for each cell (called th in turn), take its trimmed text; the square brackets collect the answers into a list
• the outputa list of 4 strings: the table’s column names

Loop the body rows into dicts

One tr at a time: grab its td texts, zip them with the headers, keep the dict. The next slide reads it line by line.

Store — and you're back in pandas-land

A list of dicts drops straight into a DataFrame. From here it's the pandas you met in the Week 1–2 labs: clean, filter, analyse.

• pd.DataFrame(rows)each dictionary becomes a row; each key becomes a column name
• df.shape → (13, 4)13 rows, 4 columns
• df.head(3)the first 3 rows; ... means pandas hid some columns to fit the screen
• df.to_csv(…, index=False)save the table as a CSV file; index=False leaves out the 0, 1, 2 row labels

For pure tables, pandas can do it alone

pd.read_html finds every <table> in the HTML and returns a list of DataFrames. When it works, it's four lines total.

Most web data isn't in a <table>

The portal's holiday list is ul + li + spans. read_html returns nothing for it. Listings, cards, and directories all look like this.

No table this time: the holiday list

Same page, different shape. Each li holds two labelled spans — and sometimes a <small> note (small print). Optional fields are where scrapers die.

Handle the optional field explicitly

find("small") returns None when absent — so test it, and store None as the honest value.

• soup.select("ul.holiday-list li")every li inside the list with class holiday-list
• small = li.find("small")this item’s note, or None if it has none
• hols.append({ … })add a dictionary with two keys, "date" and "note", to the list
• … if small else Noneif a note was found, use its text; otherwise store None
• the outputthe first holiday has a note; the second’s note is None: recorded as missing instead of crashing

Scrapers break — because pages change

Your loop assumes every row has the tags you saw today. The day the site redesigns — or one row is different — the whole loop stops.

Freeze a snapshot; parse the file

Live pages change under you. For homework and debugging, save the HTML once, then parse the file — same code from Step 2 onward.

• open("calendar.html", "w")open a file for writing ("w" erases it first); the name is its file path
• .write(r.text)write the page’s text into the file
• open("calendar.html").read()open it for reading (the default) and read all its text into html

Your turn: two scrapes

Laptops out, same pipeline, new targets. Outputs to beat are on the right — they're real.