Why pandas and how we will work

Spreadsheets are great for looking at data and making small changes. They struggle when files get large, when the same cleaning steps must be repeated every month, or when you need to combine several sources reliably. pandas solves these problems by turning your analysis into code: a series of steps you can rerun, review and share.

What pandas gives you

• DataFrame: a table with named columns, each with its own data type. Think of a spreadsheet tab, but with clear rules.
• Series: a single column, with an index.
• Fast operations on whole columns at once, instead of looping over rows.
• Readers and writers for CSV, Excel, JSON, Parquet, SQL databases and more.
• Grouping, joining, reshaping, date handling and quick plotting.

pandas is built on NumPy and is the foundation for much of Python's data ecosystem. Libraries for charts, statistics and machine learning all accept DataFrames.

pandas and newer tools

You may hear about Polars and DuckDB, which can be much faster for large datasets. They are excellent, and I use them for big files. But pandas remains the most widely used and documented tool, most tutorials and codebases use it, and the concepts you learn here transfer directly. At the end of the course we look briefly at when to switch.

The analysis workflow

Every analysis in this course follows the same loop:

1. Question: write down what you want to know, as specifically as possible.
2. Load: read the data and look at it before doing anything.
3. Clean: fix types, missing values and inconsistencies, documenting each decision.
4. Explore: group, filter and chart to understand patterns.
5. Check: sanity check results against known totals or simple calculations.
6. Explain: write the answer in plain words, with its limitations.

The checking step is the one most people skip. We will not.

The datasets

We use three public datasets included in the course files: trips from a city bike share system, a year of anonymised retail sales with product and store tables, and daily weather observations. They are messy in realistic ways, which is exactly what we want.

In the next lesson we install the tools and open the first notebook.