Chapter 21: Why Pipelines Need a Scheduler
Apache Airflow
Chapter 22: Airflow Architecture
Airflow is not a single program you run. It is a system of cooperating components. Each one has exactly one job. They communicate through a central Metadata Database, so no component talks to another directly. When something goes wrong, understanding this separation is what tells you which component to look at. When you are designing a deployment, it is what tells you which component to scale.
Chapter 23: Running Airflow with Docker
The cleanest way to run Airflow locally is with Docker. All six components from the previous chapter — Scheduler, Webserver, Worker, Triggerer, PostgreSQL, and Redis — run as separate containers managed by Docker Compose. This matches how Airflow runs in most production environments, so what you learn here carries forward directly.