When an Airflow pipeline fails in production, the hard part is rarely the fix. It’s finding out what to fix. A task dies without a traceback. Is it the code, the data, or the environment? Hard to say, as clues are scattered across task logs, scheduler logs, and worker metrics. And once you know what the problem is, the question becomes whether to add memory, rewrite the DAG, or finally take on the major version upgrade you’ve been putting off. Answering these questions takes time and deep Airflow expertise, usually while you’re on call in the middle of an incident.
AI agents can increasingly do much of this legwork. Managed Service for Apache Airflow (formerly Cloud Composer) now offers agentic help across a pipeline’s entire lifecycle. In the Google Cloud console, Managed Airflow Agent investigates failed tasks, recommends improvements to your environment, and checks whether your DAGs are ready for a newer Airflow version. In your IDE or coding agent, Data Agent Kit writes, migrates, and deploys DAGs. In this post, we take you on a tour of these features through a story of a fictional company, Cymbal Shops.
Meet Cymbal Shops, where every click counts
Amira is a data engineer at Cymbal Shops, an online retailer running cymbal-prod, a production Managed Airflow (Gen 3) environment whose workers have 2 vCPUs, 4 GB of memory, and that autoscale between 1 and 6. Every click, search, and add-to-cart on Cymbal’s website and mobile app goes to a Pub/Sub topic. A Cloud Storage subscription writes the events as newline-delimited JSON files into hourly folders under gs://cymbal-clickstream-landing/events/. It writes files in parallel and starts new ones every 5 minutes or every 128 MB, whichever comes first.
Analysts, meanwhile, want these events in BigQuery, cleaned and with personal data hashed — and they want them fast. Ideally, each file would be processed the moment it lands. But Cymbal’s Airflow environment runs Airflow 2.11.1, and Airflow 2 has no native way to start a DAG run when something happens outside Airflow. You can get close: Schedule a DAG every few minutes and let a sensor wait for new files, or build an extra service that triggers runs through the Airflow REST API. However, the team doesn’t want all those moving parts, so they settle for the next best thing: an hourly batch.
The pipeline that worked (until it didn’t)
Amira writes her DAGs with an agent. She works in Antigravity with the Data Agent Kit plugin, which adds Google Data Cloud skills and tools to the agent, and she uses Gemini as her model. Setup takes a few minutes: She installs the plugin, signs in with her Google account, and selects her project and region.
In the agent panel, she describes what she needs with this prompt:
“Write a DAG named clickstream_hourly for the Managed Airflow environment cymbal-prod in us-central1. Every hour, load the previous hour of clickstream events into the BigQuery table clickstream.events. The events are newline-delimited JSON files in gs://cymbal-clickstream-landing/events/YYYY-MM-DD/HH/. Transform them with pandas: read each file into a DataFrame with the nested context object kept as a JSON string, combine the files with pd.concat, then drop malformed records, deduplicate on event_id (Pub/Sub delivers at least once), and hash user_email and ip_address. Re-running an hour must not create duplicates. If a run fails, email data-oncall@example.com. Validate the DAG and deploy it.”
Source Credit: https://cloud.google.com/blog/products/data-analytics/scaling-airflow-pipelines-with-ai-agents-on-managed-airflow/
