At this point, you should be well on your way to mastering Airflow—being able to write complex pipelines and knowing how to deploy Airflow in a production-like setting.

So far, we’ve focused on running Airflow on a local system, either natively or using container technologies such as Docker. A common question is how to run and use Airflow in cloud settings, as many modern technological landscapes involve cloud platforms. Part 4 focuses entirely on running Airflow in the clouds, including topics such as designing architectures for Airflow deployments and leveraging the built-in functionality of Airflow to call various cloud services.

First, in chapter 15, we give a short introduction into the different components involved in designing cloud-based Airflow deployments. We’ll also briefly discuss Airflow’s built-in functionality for interacting with various cloud services and touch on vendor-managed Airflow deployments, which may save you from rolling your own deployment in the cloud.

After this introduction, we’ll dive into specific Airflow implementations for several cloud platforms: Amazon AWS (chapter 16), Microsoft Azure (chapter 17), and Google Cloud Platform (chapter 18). In each of these chapters, we’ll design architectures for deploying Airflow using services from the corresponding platform and discuss built-in Airflow functionality for interacting with platform-specific services. Finally, we’ll close each chapter with an example use case.

After completing part 4, you should have a good idea of how to design an Airflow deployment for your cloud of interest. You should also be able to build pipelines that smoothly integrate with cloud services to leverage the scale of the clouds in your workflows.