Kwame A.
Sample Good Airflow 3 migration notes
We are moving from Airflow 2 and the upgrade lesson listed exactly the things that broke in our repo, especially direct database access from tasks.
Apache Airflow 3: build and run reliable data pipelines
Write DAGs with the TaskFlow API, schedule them safely, test them and run Airflow 3 locally and in production
Created by Tomas Herrera
- Updated September 2026
- English
- Intermediate
What you will learn
- Write Airflow 3 DAGs with the TaskFlow API and the Task SDK
- Choose schedules, start dates and catchup settings that do not surprise you
- Design idempotent tasks that can be retried and backfilled safely
- Connect pipelines with assets and data aware scheduling
- Test DAGs and task logic with pytest before deploying
- Configure retries, alerts and secrets for a production deployment
Course content
Airflow 3 essentials
Running Airflow 3 locally
- Video: Starting Airflow with the official Docker Compose file 12 min
- Video: A project layout that scales past five DAGs 9 min
- Article: Connections and variables without hard coded secrets 10 min
- Video: Finding your way around the new UI 8 min
Moving data reliably
- Video: Operators, hooks and providers 11 min
- Article: Extracting from an HTTP API with pagination and rate limits 13 min
- Video: Loading into PostgreSQL with upserts 12 min
- Video: Retries, timeouts and failure callbacks 9 min
- Article: Sensors and deferrable operators 10 min
Assets, mapping and backfills
- Video: Assets and data aware scheduling 11 min
- Video: Dynamic task mapping 10 min
- Article: Backfills the Airflow 3 way 9 min
- Video: Event driven runs with asset watchers 10 min
Testing and data quality
- Video: Testing that every DAG imports cleanly 8 min
- Article: Unit testing task logic 11 min
- Video: Data quality checks inside the pipeline 10 min
- Video: Running dag.test and CI on every pull request 9 min
Production and operations
- Article: Choosing an executor 10 min
- Video: Secrets backends and configuration per environment 9 min
- Video: Monitoring, SLAs and alerting that people read 10 min
- Article: Upgrading from Airflow 2 to Airflow 3 12 min
- Quiz: Final check 6 min
Requirements
- Comfortable writing Python functions and using pip
- Basic SQL (SELECT, INSERT, CREATE TABLE)
- Docker Desktop or another Docker engine installed
Description
Apache Airflow is the scheduler most data teams reach for when cron jobs stop being enough. Airflow 3 changed a lot: a new UI, DAG versioning, assets instead of datasets, a Task SDK and stricter rules about how tasks talk to the metadata database. This course teaches Airflow the way it works today.
What you will build
You will build one pipeline across the course. It pulls data from a public HTTP API, lands raw files in object storage, loads them into PostgreSQL and triggers a downstream reporting step when new data arrives. Every lesson adds one piece and explains why it is there.
- DAGs written with the TaskFlow API and
airflow.sdk - Safe schedules, catchup and backfills
- Idempotent loads that can run twice without duplicating rows
- Asset based scheduling between pipelines
- Tests that run in CI before anything reaches the scheduler
How the course works
Section one is fully written so you can read it at your own pace. Later sections are short videos with a runnable repository. You run Airflow 3 locally with the official Docker Compose setup, so nothing needs a cloud account. The final section covers executors, alerting, secrets and upgrading from Airflow 2.
Who this course is for
- Data engineers moving from cron scripts to an orchestrator
- Analytics engineers who need to own their own pipelines
- Airflow 2 users who want to understand what changed in Airflow 3
Instructor
Tomas Herrera
Data engineer teaching Airflow, dbt and warehouse modelling with real, testable pipelines
- 4.4 instructor rating
- 15 reviews
- 4,150 students
- 2 courses
- Sample
I build batch data pipelines for a living: ingestion jobs, orchestration with Apache Airflow, and warehouse models in dbt that analysts can trust on Monday morning.
My courses follow the same order I use at work. First we make a pipeline run, then we make it fail loudly, then we make it boring. Boring is the goal: idempotent tasks, clear ownership, tests that catch bad data before a dashboard does.
I keep lessons short and code heavy. Every course ships with a repository you can run on a laptop with Docker, and I update it when Airflow or dbt release changes that affect what you write.
Student reviews
Marta S.
Sample Finally understand data intervals
The lesson on logical dates and data intervals cleared up something I had been getting wrong for a year. My backfills now do what I expect.
Daniel P.
Sample Testing section is excellent
The import test and data quality checks are now in our CI. Short lessons, real code, no filler.
Elif Y.
Sample Practical and well paced
The idempotency lesson alone was worth it. I would have liked a little more on the Kubernetes executor, but the rest was solid.
Lucia F.
Sample Exactly the level I needed
I knew Python and SQL but not orchestration. The repository grows lesson by lesson and I never felt lost.
Irina V.
Sample Clear explanation of assets
Splitting the pipeline into producer and consumer DAGs with assets made data aware scheduling click for me.
Sam O.
Sample Solid but assumes Docker comfort
Content is accurate, but I struggled with the Docker Compose setup at first and had to look up a few things outside the course.
Keep exploring
All Data Engineering courses
More in Data Engineering
$69.99
Enroll now