Giorgi T.
Sample Free and genuinely useful
The idempotency and backfill lessons fixed problems in scripts I already run at work.
Data engineering foundations: your first pipeline in Python
Extract data from an API, store it as Parquet, transform it with DuckDB SQL and run it on a schedule, all on your own laptop
Created by Priya Raghavan
- Updated August 2026
- English
- Beginner
What you will learn
- Describe the extract, load and transform steps of a data pipeline
- Fetch paginated data from an HTTP API with retries and timeouts
- Store data as Parquet and explain why columnar formats matter
- Transform and aggregate data with SQL in DuckDB
- Make a pipeline idempotent so reruns are safe
- Schedule a pipeline and add basic logging and alerts
Course content
What data engineers actually do
Transforming data with DuckDB
- Video: DuckDB in five minutes 8 min
- Video: Flattening raw JSON into clean rows 12 min
- Article: Handling missing and suspicious values 10 min
- Video: Building the daily summary table 10 min
Making the pipeline repeatable
- Video: Idempotency: safe to run twice 10 min
- Article: Backfilling a date range 9 min
- Video: Logging that helps at 3am 9 min
- Video: Simple data quality checks 10 min
Testing your pipeline
- Article: Testing transformations with pytest 11 min
- Video: Testing extraction without the real API 9 min
- Video: Keeping SQL readable and reviewed 8 min
- Article: Running checks before every commit 8 min
Scheduling and running unattended
- Video: Running on a schedule with cron 9 min
- Video: Running on a schedule in GitHub Actions 10 min
- Article: Alerts when something goes wrong 8 min
Where to go next
- Video: When you need an orchestrator 9 min
- Video: From local files to a cloud warehouse 9 min
- Article: Building a portfolio project 8 min
- Quiz: Final check 6 min
Requirements
- Basic Python: variables, functions, loops
- Very basic SQL is helpful but not required
- A computer with Python 3.11 or newer
Description
Data engineering can look like a wall of tool names. Underneath, the work is simple to describe: get data from somewhere, store it safely, reshape it into something useful and do that reliably every day. This free course teaches those foundations with a small, complete pipeline you build yourself.
What you build
A pipeline that downloads public weather observations from an HTTP API, saves the raw responses, converts them to Parquet, cleans and aggregates them with SQL in DuckDB and produces a daily summary table. Then we make it repeatable: configuration, logging, safe reruns and a schedule.
- Python scripts organised as a small project, not one long notebook
- Raw, clean and summary layers on disk
- SQL transformations you can read and test
- A pipeline that can run twice without breaking anything
Who it is for
This course is for people who know a little Python and a little SQL and want to understand what data engineers do before investing in heavier tools. Everything runs locally and for free. At the end you will know which problems Airflow, dbt, Spark or Kafka solve, because you will have felt those problems on a small scale.
Who this course is for
- Analysts and developers curious about data engineering
- Students choosing between data roles
- Anyone who maintains a few fragile data scripts at work
Instructor
Priya Raghavan
Streaming and data platform engineer. Kafka, Python pipelines and DuckDB explained plainly
- 4.4 instructor rating
- 21 reviews
- 7,170 students
- 3 courses
- Sample
I work on data platforms where events arrive all day and someone always wants yesterday's numbers fixed. Most of my time goes into Kafka topics, Python ingestion services and the small analytical tools that make debugging them faster.
I started teaching because new data engineers kept asking me the same questions: why is this consumer lagging, why did this file load twice, why is this query slow. My courses answer those questions with small, complete examples rather than slides.
Outside work I maintain a few open source notebooks for teaching SQL over Parquet files, and I like explaining file formats more than I probably should.
Student reviews
Aylin D.
Sample Perfect first data engineering course
I knew some Python and pandas. Building one small pipeline end to end showed me what the job is really about.
Laura P.
Sample Made me confident to apply
I built my own version with transport data, and it became the main project on my CV.
Marcus J.
Sample The format lesson was eye opening
Seeing a CSV become a much smaller Parquet file and query faster made columnar storage click instantly.
Noor H.
Sample Clear and honest about tools
I liked that it explains why Airflow and dbt exist instead of throwing them at you on day one.
Omar F.
Sample DuckDB is a great teaching choice
Running real SQL on files with no server to install kept the focus on the concepts.
Emma C.
Sample Great structure, small gaps
The project layout and logging advice were excellent. I wanted a bit more on testing extraction, but it is a free course and very good.
Kenji M.
Sample Good basics, slow for me
Solid and accurate, but I already knew most of section one. Sections three and four were the most useful for me.
Keep exploring
All Data Engineering courses
More in Data Engineering
Free
Enroll now