Chapter 2: The Modern Data Stack
"Just give me a report of yesterday's revenue by region" sounds like a five-minute ask.
It isn't. Before that number reaches a dashboard, it has to be produced by a source system, pulled out of that system, landed somewhere, cleaned up, joined against other tables, computed on a schedule, served to whatever renders the dashboard, and watched closely enough that someone finds out if it's wrong before the CEO does.
Seven jobs, hiding inside one sentence. This chapter names them, and shows you where each one is taught in this book.
2.1 Seven Layers, One Pipeline
Every data platform, from a two-person startup to a bank, is doing the same seven jobs. The tools change constantly. The seven jobs don't.
| Layer | The job it does | The question it answers |
|---|---|---|
| Source Systems | Where data is born: apps, databases, third-party APIs, IoT devices, event streams | Where does this value actually come from? |
| Ingestion | Moving data out of source systems and into your platform | How does data get here, and how often? |
| Storage | Landing and holding data at a cost and access pattern that fits its use | Where does it live, and how cheaply can I keep it? |
| Transformation | Cleaning, joining, and reshaping raw data into something usable | Is this actually correct and ready to use? |
| Orchestration | Scheduling, sequencing, and retrying every step above | What runs when, in what order, and what happens on failure? |
| Serving | Exposing finished data to dashboards, applications, APIs, and models | Who consumes this, and how fast do they need it? |
| Observability | Watching all of the above for failure, drift, and cost | How would anyone know if any of this quietly broke? |
Orchestration and observability aren't steps in the line. They wrap around every other layer: scheduling and watching all five of the others, rather than sitting between two of them in sequence.
2.2 The Roadmap This Book Follows
Every part of this book is one or more of those seven layers, taught with the specific tools the industry actually uses in production today.
| Part | Layer(s) it covers | Tools |
|---|---|---|
| 1. SQL | Storage, Transformation | Querying, joins, aggregation, window functions, schema design |
| 2. Python | Ingestion, Transformation | APIs, file handling, Pandas, NumPy |
| 3. Airflow | Orchestration | DAGs, scheduling, retries, dependency management |
| 4. Engineering Practices | Observability, delivery | Docker, CI/CD, Bash |
| 5. Agentic Data Engineering | Cuts across every layer | AI-assisted pipeline development and coding agents |
| 6. Snowflake + dbt | Storage, Transformation, Serving | Cloud warehousing, ELT, production dbt |
| 7. Kafka | Ingestion | Real-time streaming, change data capture |
| 8. AWS | Storage, Transformation, Serving, Orchestration | S3, Glue, Athena, Redshift, Lambda, Step Functions |
| 9. Azure | Storage, Transformation, Serving | ADF, Databricks, Fabric, Synapse |
Notice that no single part owns a layer outright. Storage shows up in SQL, Snowflake, AWS, and Azure, because in a real job you'll meet all four, often in the same quarter. That repetition is deliberate. By the time you reach Part 8, storage design won't be a new idea, just a new tool applied to an idea you've already used three times.
2.3 One Pipeline, Walked End to End
Take a concrete example: a ride-sharing app that pays drivers nightly based on completed trips.
| Step | What happens | Layer | Where it's taught |
|---|---|---|---|
| 1 | The app's production database logs every completed trip | Source Systems | Chapter 1 |
| 2 | A nightly job pulls new trip records via API or database connection | Ingestion | Part 2 (Python), Part 7 (Kafka for real-time) |
| 3 | Raw trip records land in a cloud warehouse, unmodified | Storage | Part 1 (SQL), Part 6 (Snowflake) |
| 4 | Fares, tips, and platform fees are calculated per driver | Transformation | Part 1 (SQL), Part 6 (dbt) |
| 5 | The whole sequence runs automatically every night at 2am, with retries if step 2 fails | Orchestration | Part 3 (Airflow) |
| 6 | Driver payout totals are exposed to the payments app | Serving | Part 6, Part 8, Part 9 |
| 7 | An alert fires if fewer trips were processed than the night before | Observability | Part 4 |
That's the whole book, in seven rows. Every remaining chapter is one row, done properly.
2.4 Why No Single Tool Owns This Anymore
A decade ago, one vendor's monolithic platform tried to do all seven jobs at once. That world is mostly gone.
| The old monolith approach | The modern data stack | |
|---|---|---|
| Tooling | One vendor, one platform, for everything | Best tool per layer, connected together |
| Storage cost | Compute and storage bundled and expensive | Storage is cheap; compute scales separately |
| Change | A platform upgrade touches everything at once | Swap one layer without rebuilding the rest |
| Talent | Deep expertise in one proprietary system | Fluency across widely-used, transferable tools |
This is also why this book is structured the way it is. You aren't learning "the one tool." You're learning the seven jobs well enough that any specific tool, current or the one that replaces it in five years, is just a new way to do a job you already understand.
Summary
Every data platform does the same seven jobs: source systems produce data, ingestion moves it, storage holds it, transformation shapes it, orchestration schedules all of it, serving exposes it, and observability watches all of it. This book teaches those seven jobs through the tools the industry actually runs in production: SQL, Python, Airflow, Docker and CI/CD, AI-assisted engineering, Snowflake and dbt, Kafka, AWS, and Azure. No single tool owns a layer, and by design, you'll practice storage, transformation, and serving more than once, with different tools, because that's exactly how it works on a real team.
What's Next
Before touching SQL itself, Chapter 3 walks through installing SQL Server, Python, and Git/GitHub, the tools every remaining chapter in this book assumes are already on your machine. Then Part 1 starts with SQL, the layer underneath almost everything else in this book. Chapter 4 begins with querying and filtering, the foundation everything from Chapter 5 onward builds on.