Enterprise AgentOps: Decoupling Terraform Infrastructure from Google ADK Agent Deployments on Vertex AI
How to bootstrap placeholder Reasoning Engines with Terraform and patch real agent code in CI/CD — plus couple of battle-tested lessons learned in production.
Building production-grade AI agents requires bridging two very different worlds: Infrastructure as Code (IaC) and Continuous Integration/Continuous Deployment (CI/CD).
In enterprise Google Cloud environments, infrastructure teams use Terraform to provision VPCs, Cloud SQL instances, Cloud Run batch invokers, Cloud Scheduler triggers, and IAM permissions. Application developers, meanwhile, build and iterate on their AI agents using frameworks like the Google Agent Development Kit (ADK) and deploy them to Vertex AI Agent Engine (Reasoning Engine).
Here lies the classic chicken-and-egg dilemma: – Downstream resources (such as Cloud Run invoker jobs, Cloud Scheduler, and Web UIs) need the Vertex AI Reasoning Engine Resource ID (projects/…/locations/…/reasoningEngines/<id>) at Terraform apply time. – But application logic evolves rapidly. Packaging, testing, and deploying prompt engineering changes and tool definitions should happen in application CI/CD pipelines, without running terraform apply or needing infrastructure credentials on every code push.
In this post, we will walk through the Bootstrapped Placeholder Agent Pattern that solves this decoupling problem, and share four crucial lessons learned debugging production Vertex AI Agent Engine deployments.
The Architecture: Two-Phase Deployment
The solution is a clean separation of concerns:
+-------------------------------------------------------------------------------+
| PHASE 1: INFRASTRUCTURE (Terraform / cloud-infrastructure repo) |
| |
| 1. Package minimal dummy_agent_source.tar.gz |
| 2. Provision google_vertex_ai_reasoning_engine (source_code_spec) |
| 3. ignore_changes = [spec] <-- CRITICAL |
| 4. Output reasoning_engine_id --> Wire into Cloud Run Jobs & Scheduler |
+-------------------------------------------------------------------------------+
|
v Stable Reasoning Engine ID
+-------------------------------------------------------------------------------+
| PHASE 2: APPLICATION CI/CD (GitLab CI / GitHub Actions / app repo) |
| |
| 1. Read reasoning_engines.json mapping (Target ID per environment) |
| 2. Run pytest & python3.11 verification |
| 3. Execute adk deploy agent_engine --agent_engine_id=<ID> |
| 4. In-place patch running container on Vertex AI without modifying ID |
+-------------------------------------------------------------------------------+
- Phase 1 (Infrastructure): Terraform provisions a lightweight "placeholder" (dummy) Reasoning Engine instance using a minimal source archive. It captures the generated Engine ID and passes it to all downstream infrastructure (Cloud Run, Scheduler, IAM).
- Phase 2 (Application CI/CD): The application pipeline takes the pre-existing Engine ID and uses the ADK CLI (adk deploy agent_engine –agent_engine_id=<ID>) to patch the running engine with real code, prompts, and tools in place.
Phase 1: Bootstrapping the Placeholder Agent in Terraform
1. The Minimal Placeholder Code
Inside your infrastructure workspace, create a lightweight directory: dummy_agent_source/.
dummy_agent_source/requirements.txt:
google-cloud-aiplatform[adk,agent_engines]
dummy_agent_source/agent.py:
"""Bootstrapped placeholder agent for Terraform provisioning."""
from google.adk.agents import Agent
root_agent = Agent(
name="root_agent",
model="gemini-2.5-flash",
instruction="Bootstrap placeholder agent runtime initialized via Terraform.",
)
Package this directory into a tarball:
tar -czf dummy_agent_source.tar.gz -C dummy_agent_source .
2. The Terraform Resource Definition
The key to enabling future in-place updates from ADK is using source_code_spec (not package_spec), paired with Terraform's lifecycle { ignore_changes = [spec] }:
# infrastructure/reasoning_engine.tf
resource "google_vertex_ai_reasoning_engine" "agents" {
for_each = toset(["customer_support", "order_routing"])
display_name = "${each.key}-agent-${var.env}"
project = var.project_id
region = var.location
spec {
source_code_spec {
inline_source {
# Base64-encode the local placeholder archive
source_archive = filebase64("${path.module}/dummy_agent_source.tar.gz")
}
python_spec {
version = "3.11"
}
}
}
# CRITICAL: Prevent Terraform from reverting future ADK code deployments
lifecycle {
ignore_changes = [
spec
]
}
}
output "reasoning_engine_ids" {
description = "Stable Resource IDs to configure downstream Cloud Run jobs"
value = {
for k, v in google_vertex_ai_reasoning_engine.agents : k => split("/", v.name)[5]
}
}
Why lifecycle { ignore_changes = [spec] } is Essential
When your application pipeline later patches the agent via the ADK CLI, Vertex AI updates the engine's underlying source code, image digest, and class methods.
Without ignore_changes = [spec], the next time someone runs terraform apply, Terraform would detect a diff between the live engine spec and the dummy archive, and promptly overwrite your production code back to the dummy placeholder!
Phase 2: Updating the Agent in Application CI/CD
Once Terraform outputs the Engine IDs, your application repository stores these IDs as the target destination.
1. The Single Source of Truth (reasoning_engines.json)
{
"dev": {
"customer_support": "6448157959103971328",
"order_routing": "9054053293491224576"
},
"qa": {
"customer_support": "7819230192830192831",
"order_routing": "8912389128391283912"
}
}
2. The Deployment Script (deploy.sh)
The deployment script resolves the target ID and executes adk deploy agent_engine targeting the specific –agent_engine_id:
#!/usr/bin/env bash
set -euo pipefail
AGENT_NAME="$1" # e.g. "customer_support"
TARGET_ENV="$2" # e.g. "dev"
# 1. Read the target Reasoning Engine ID from JSON
CONFIG_JSON="agent_deploy/reasoning_engines.json"
AGENT_ID=$(python3 -c "
import json
with open('${CONFIG_JSON}') as f:
data = json.load(f)
print(data['${TARGET_ENV}']['${AGENT_NAME}'])
")
# 2. Build deployment config
CONFIG_FILE=$(mktemp)
cat <<EOF > "${CONFIG_FILE}"
{
"display_name": "${AGENT_NAME}-agent-${TARGET_ENV}",
"description": "Acme Corp ${AGENT_NAME} Agent",
"agent_framework": "google-adk",
"service_account": "svc-agent-engine@${PROJECT_ID}.iam.gserviceaccount.com",
"env_vars": {
"GOOGLE_GENAI_USE_VERTEXAI": "True",
"GOOGLE_CLOUD_LOCATION": "us-central1",
"GOOGLE_CLOUD_AGENT_ENGINE_LOCATION": "us-central1",
"GOOGLE_CLOUD_MODEL_LOCATION": "global"
}
}
EOF
# 3. In-place patch the running Reasoning Engine
adk deploy agent_engine \
--project="${PROJECT_ID}" \
--region="us-central1" \
--display_name="${AGENT_NAME}-agent-${TARGET_ENV}" \
--otel_to_cloud \
--agent_engine_config_file="${CONFIG_FILE}" \
--agent_engine_id="${AGENT_ID}" \
"${AGENT_NAME}"
3. The GitLab CI / GitHub Actions Job
In .gitlab-ci.yml, the deploy job simply authenticates and triggers the script:
.adk_deploy:
stage: deploy
image: python:3.11-slim
before_script:
- python3 -m venv .venv && source .venv/bin/activate
- pip install google-adk
script:
- chmod +x agent_deploy/deploy.sh
- ./agent_deploy/deploy.sh "${AGENT_DIR}" "${CI_ENVIRONMENT_NAME}"
adk_update_customer_support_dev:
extends: .adk_deploy
when: manual
variables:
AGENT_DIR: "customer_support"
environment:
name: dev
4 Battle-Tested Lessons Learned in Production
During our implementation, we hit several subtle edge cases between Google Cloud Vertex AI, the ADK CLI, and Python runtimes. Here are the four biggest takeaways:
Lesson 1: Avoid package_spec — It Cannot Be Converted to ADK Source Code
If you initially provision a Reasoning Engine in Terraform using package_spec (the pickled Python object specification used by older Vertex AI samples), you will be locked out of ADK patching:
Error: Error updating ReasoningEngine "projects/.../locations/.../reasoningEngines/...":
googleapi: Error 400: package_spec cannot be updated to empty.
Vertex AI Agent Engine treats spec.package_spec and spec.source_code_spec as mutually exclusive deployment types. You cannot transition an existing engine from one to the other.
Rule: Always bootstrap placeholder engines using source_code_spec so that adk deploy agent_engine can patch it seamlessly.
Lesson 2: The Python 3.11 Grammar Trap (PEP 701 Backslashes in F-Strings)
When testing queries on our newly deployed agent, the container crashed on boot with:
File "/app/agents/customer_support/tools/bigquery/customer_lookup.py", line 149
raw_where.append(f"customer_id = '{customer_id.strip().replace('\'', '\\\'')}'")
^
SyntaxError: Fail to load 'customer_support' module. f-string expression part cannot include a backslash
Why did this happen?
When deploying via CI/CD runners (typically standard python:3.11-slim images) or when deploying with python_spec.version = "3.11", Vertex AI compiles and runs your source code against Python 3.11.
Developers writing code on local Python 3.12+ workstations often rely on PEP 701 (syntactic formalization of f-strings introduced in Python 3.12), which allows escape sequences and backslashes inside f-string expressions. In Python 3.11 and earlier, embedding a backslash inside { … } triggers a fatal SyntaxError.
The Fix:
Always sanitize string expressions outside the f-string curly brackets:
# ❌ FAILS on Python 3.11:
f"customer_id = '{customer_id.strip().replace('\'', '\\\'')}'"
# COMPATIBLE with all Python versions:
clean_id = customer_id.strip().replace("'", "\\'")
raw_where.append(f"customer_id = '{clean_id}'")
Conclusion
By adopting the Bootstrapped Placeholder Agent Pattern: – Your infrastructure team provisions stable, immutable Vertex AI Reasoning Engine IDs in Terraform and wires them into downstream Cloud Run and Cloud Scheduler jobs on Day 1. – Your application team iterates and pushes agent code, prompt instructions, and tool integrations via lightweight CI/CD pipelines without ever needing to touch Terraform. – You avoid runtime surprises by keeping your code strictly Python 3.11 compatible and cleanly separating regional infrastructure endpoints from global Gemini model routing.
Happy building with Google ADK and Vertex AI!
Enterprise AgentOps: Decoupling Terraform Infrastructure from Google ADK Agent Deployments on… was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/enterprise-agentops-decoupling-terraform-infrastructure-from-google-adk-agent-deployments-on-dcfefab99c3d?source=rss—-e52cf94d98af—4
About the Author
