The job in one small pipeline

Ask ten data engineers what they do and you will hear ten tool lists. Strip the tools away and almost every pipeline does the same four things.

1. Extract: get data from a source system. That could be an API, a production database, files dropped by a partner or events from an app.
2. Load: store the raw data somewhere you control, unchanged, so you can always go back to it.
3. Transform: clean, join and aggregate the raw data into tables that answer questions.
4. Serve: make the result available, for example to a dashboard, a report, a machine learning model or another team.

Around those steps sits the part that turns a script into engineering: running on a schedule, handling failures, rerunning safely, knowing when data is late or wrong, and documenting what each table means.

ETL or ELT

You will hear both terms. In ETL you transform data before loading it into the final store. In ELT you load raw data first and transform it inside the analytical database. Modern teams mostly do ELT because storage is cheap and SQL engines are fast, and because keeping the raw data lets you fix transformation bugs by simply rerunning them.

Our pipeline

In this course we build an ELT pipeline for public weather data:

• Extract hourly observations from an HTTP API.
• Load the raw JSON responses to a folder, one file per day.
• Transform them into a clean Parquet table and a daily summary using DuckDB.
• Serve the summary as a small table you could point a dashboard at.

We deliberately use small, free tools: Python, the requests library, DuckDB and Parquet files. Each of them has a big brother used in larger companies, an orchestrator instead of cron, a cloud warehouse instead of DuckDB, object storage instead of a local folder. The ideas carry over unchanged, and you will understand why those tools exist once you have built the simple version yourself.