Cloud Composer supports Apache Airflow 3.1

In a world of fragmented data stacks, you need workflow orchestration that is innovative, portable, and extensible. Cloud Composer, our fully managed data and AI/ML workflow orchestration service, relies on Apache Airflow to provide a universal control plane. And today, we’re excited to share that Cloud Composer now supports Airflow 3.1 in preview.

Released in November, this update marks a significant milestone for the platform, and the first time a hyperscaler is offering Airflow 3.1.

Built on the foundation of Airflow 3

The new features in Airflow 3.1 are built on top of the revolutionary architecture introduced in Airflow 3.0:

• Decoupled architecture: Robust separation between the scheduler and execution layer for better scalability and security

• DAG versioning: Native support for automated DAG versioning, retaining the historical structure and run history even after you remove tasks or change logic

• Powerful managed backfills: A redesigned backfill system that is now a first-class citizen, fully managed by the scheduler

• Event-driven scheduling and data assets: Enhanced capabilities for triggering workflows based on assets as well as external events, like messages arriving in a message queue

• and many more…

For a deeper dive into the architectural shifts of Airflow 3, you can read our previous announcement: Next Gen Data Pipelines: Airflow 3 Arrives on Google Cloud Composer.

What’s new in Airflow 3.1 on Composer?

We’ve invested heavily to ensure the new capabilities in Airflow 3.1 are robust and reliable on Google Cloud. Specifically, Airflow 3.1 introduces enhancements designed to increase oversight, improve reliability, and support global teams:

1. Human-in-the-Loop (HITL) workflows

As AI agents and automated pipelines become more complex, the need for human oversight increases. Airflow 3.1 introduces powerful Human-in-the-Loop (HITL) functionality that allows workflows to pause execution in a deferred state and wait for a person to make a decision via the Airflow UI or API call.

HITL puts you in control, whether you are approving a deployment, reviewing a generative AI model’s output, or providing feedback to steer a pipeline.

To make this process even smoother, HITL operators integrate natively with Airflow Notifiers. You can configure a notifier to automatically send a message via Slack, email, or PagerDuty the moment a task pauses for input. Using these helper methods, the notification can include a direct link to the specific approval page, allowing stakeholders to respond immediately without searching through the UI.