What Airflow does, and what it does not

Airflow is a workflow orchestrator. It decides when a piece of work should run, in what order, and what happens when it fails. It is not a data processing engine. The heavy lifting happens elsewhere: in your warehouse, in Spark, in a Python process, in a container. Airflow starts that work, watches it and records the result.

The core idea is the DAG, a directed acyclic graph. Each node is a task, each edge says "this must finish before that starts", and there are no cycles. A DAG has a schedule, and every scheduled run covers a specific interval of time called the data interval.

The pieces you will meet

• Scheduler: reads your DAG files, decides which runs and tasks are due and queues them.
• API server: in Airflow 3 this serves the new React UI and the REST API, and it is also how running tasks talk back to Airflow.
• DAG processor: parses DAG files in its own process, separate from the scheduler.
• Executor and workers: run the tasks. Locally this can be the LocalExecutor. In production it is often Celery or Kubernetes.
• Metadata database: usually PostgreSQL, storing runs, task states and history.

One important Airflow 3 change: task code no longer talks to the metadata database directly. Tasks use the Task SDK and the Task Execution API instead. This makes remote and isolated workers possible, and it means old tricks like opening a database session inside a task will break.

When Airflow is the wrong tool

If you need sub second event processing, use a streaming system such as Kafka with a stream processor. If you only have one script that runs nightly, cron or a scheduled cloud function may be simpler. Airflow earns its place when you have many dependent jobs, retries, backfills and people who need to see what ran and why.

In this course we build one realistic pipeline and use it to learn each of these ideas in order.