Contents
S3/Session 3–5/Foundations/~3 weeks

Python for data

By the end you canYou handle data with NumPy and Pandas without thinking about the syntax, so you spend your time on the model, not on how to write a filter.

All modulesS3 · Python for data▾

NumPy and Pandas are the tools you use on every problem, whatever the model. If you trip over them, you lose precious time. The goal here isn't just to write code that works, it's to write it fast and by reflex. You learn them once, well, and use them the rest of the year.

01

Why plain Python isn't enough

In ordinary Python, a list of a million numbers you want to add up element by element is done with a for loop. It works, but it's slow, because Python checks the type of every element at every step. With real data, that becomes unbearable.

NumPy solves it with a new structure: the ndarray, a grid of numbers of the same type, with a fixed shape. Operations apply to the whole array at once, in fast compiled code, with no Python loop. The idea is called vectorization and it's tens of times faster.

import numpy as np

# slow, in pure Python
total = 0
for x in range(1_000_000):
    total += x * x

# fast, vectorized
v = np.arange(1_000_000)
total = (v * v).sum()
The same sum, but the second version is much faster.
02

ndarray: shape, axes, indexing

An ndarray has a shape: how many rows, how many columns, how many dimensions. A grayscale image is a 2D matrix, a color one is 3D (height, width, channels). Your first move on any bug is to print the shape and check it's what you thought.

The axes are the directions you operate along. axis=0 goes down the rows (vertically, you get one value per column), axis=1 goes across the columns (horizontally). Mixing up the axes is one of the most common beginner mistakes.

x = np.array([[1, 2, 3],
              [4, 5, 6]])

x.shape          # (2, 3): 2 rows, 3 columns
x.mean(axis=0)   # mean of each column -> [2.5, 3.5, 4.5]
x.mean(axis=1)   # mean of each row    -> [2.0, 5.0]

Indexing with a boolean mask is a tool you use daily: you build a vector of True/False and select only the elements where it's True. That's how you filter data with no loop.

x[x > 3]         # only the elements greater than 3 -> [4, 5, 6]
x[x > 3] = 0     # and you can change them in place too
03

Broadcasting: how NumPy matches different shapes

Broadcasting is the rule by which NumPy does operations between arrays of different shapes, automatically stretching the smaller one to fit. It's gold once you understand it, and a source of strange bugs when you don't. Half the beginner errors come from shapes that don't match the way you thought.

The rule, simply: NumPy compares shapes from right to left. Two dimensions match if they're equal or if one of them is 1 (that one stretches). A scalar matches anything.

X = np.array([[1, 2, 3],
              [4, 5, 6]])      # shape (2, 3)
means = X.mean(axis=0)         # shape (3,): [2.5, 3.5, 4.5]
X_centered = X - means         # (2,3) - (3,) matches, subtracts per column
Centering a matrix on its means, with no loop at all.
04

Pandas: data with labels

NumPy is good with numbers, but real data has names: an age column, a score column, a class column. Pandas adds labels on top of NumPy. A DataFrame is a table with column names and a row index. It's the structure most contest datasets arrive in.

You read a CSV file in one line. Then you look at it before anything: the first rows, the column types, how many values are missing.

import pandas as pd

df = pd.read_csv("data.csv")
df.head()        # first 5 rows
df.info()        # types and non-null counts
df.describe()    # stats on the numeric columns
05

Selection: loc vs iloc

There are two ways to select from a DataFrame, and mixing them is a classic mistake. loc selects by label (the column name, the index value). iloc selects by position (which row, which column, counting from 0).

df.loc[10, "score"]        # value at index 10, column "score"
df.iloc[0, 2]              # row 0, column 2, by position
df.loc[df["score"] > 8]    # all rows with score above 8
06

groupby, merge, pivot: the three you always use

groupby splits the data into groups and computes something on each group: the mean per class, the sum per category. It's the split-apply-combine pattern: you split, you apply a function, you stitch the results back together.

df.groupby("class")["score"].mean()      # mean score per class
df.groupby("class").size()               # how many rows each class has

merge stitches two tables together on a common column, exactly like a JOIN in databases. how says what you do with the rows that have no match: left keeps everything from the left, inner keeps only the matches.

df.merge(other_table, on="id", how="left")

pivot_table reshapes a long table into a wide one, with a column turned into a header. It's handy for reports and for spotting patterns across two dimensions at once.

Remember
  • Vectorization replaces loops: you operate on the whole array at once, much faster.
  • Shape and axes are the first thing to check on a NumPy bug.
  • Broadcasting matches shapes by comparing right to left; one of the dimensions has to be equal or 1.
  • loc by label, iloc by position, don't mix them.
  • groupby, merge, pivot_table show up in almost every tabular problem.
Index
Lists, dictionaries, sets, functions, classesNumPy: ndarray, indexing, broadcasting, axis operations, boolean masksPandas: loc/iloc, groupby, merge, pivot_tableReading text, CSV, imagesMatplotlib and Seaborn
Checkpoint

Full pipeline: once you have the Python basics, one run from reading the data to a submission, end to end.

If you want more