Google Cloud CLI Remote MCP Server (Part 4): Handling & Verifying Long-Running gcloud Commands (async)
How to prevent HTTP and agent timeouts using the 3-Phase Async Submit, Operation Polling, and State Verification pattern in Google ADK and Gemini Enterprise Managed Agents.
Google Cloud CLI Remote MCP Server Series
- Part 1: Getting Started with the Google Cloud CLI Remote MCP Server — Run gcloud and bq commands from AI agents with zero local SDK install across curl, Local IDEs, Google ADK, and Managed Agents.
- Part 2: Building a Multi-Node ADK Workflow for Cloud Audit & Remediation — Move from a single LLM prompt to a predictable, multi-step ADK Workflow that discovers stopped VMs and tags non-compliant resources.
- Part 3: Human-in-the-Loop (HITL) Cloud Remediation on Cloud Run — Pause a workflow for human approval before restarting a VM, persist state with VertexAiSessionService on scale-to-zero Cloud Run, and enforce approval expiry and command checks.
- Part 4 (This Post): Handling & Verifying Long-Running gcloud Commands (–async) — Prevent HTTP timeouts on slow cloud operations using a 3-step Submit → Poll → Verify pattern in ADK and Managed Agents.
In Part 1, Part 2 and Part 3, webuilt a Human-in-the-Loop (HITL) ADK Workflow on Cloud Run backed by VertexAiSessionService and the Google Cloud CLI Remote MCP Server.
In Part 3, once the human operator approved remediation, Node 5A (remediation_executor_agent) started the stopped Compute Engine VM using the –async flag (gcloud compute instances start demo-web-server … –async) and returned the operation id.
That leads directly to the next production engineering question:
Once an agent submits a long-running gcloud command with –async, how do we reliably poll that operation until it finishes and verify that the underlying cloud resource actually reached the desired state?
In this Part 4 guide, we walk through why running slow or shall we say, time consuming gcloud commands synchronously causes timeouts, and build two complete, live-tested implementations of the 3-Phase Submit → Poll → Verify Pattern:
- Approach 1 — Google ADK (Workflow Graph with a Polling Loop): A 7-node ADK Workflow (examples/07-long-running-commands/app/agent.py) that uses a plain Python router node (evaluate_poll_status) and loop edges ("continue", "done", "failed") to submit a –async command, poll the operation until DONE, and verify the final VM status.
- Approach 2 — Gemini Enterprise Managed Agents & Interactions API: A server-managed agent and stateful Interactions API runner (examples/07-long-running-commands/run_managed_lro.py) that stores the 3-step instructions on the Managed Agents Control Plane and runs the multi-turn polling and verification loop on the Interactions API Data Plane.
Companion Code on GitHub: All complete, runnable files for this guide are available in examples/07-long-running-commands in the rominirani/gcloud-remote-mcp-guide repository.
Why Synchronous gcloud Commands Time Out in Agent Workflows
When you run gcloud compute instances stop demo-web-server in a local terminal, the gcloud CLI sends the request to the Compute Engine API, gets back a compute#operation ID, and waits in your terminal while polling that operation in the background for 30–90 seconds (or 10+ minutes for GKE and Cloud SQL).
When an AI agent calls run_gcloud_command over the Cloud CLI Remote MCP Server , waiting inside a single MCP tool call causes practical issues:
- HTTP & Proxy Timeouts: MCP HTTP clients (McpToolset), Cloud Run, and corporate proxies usually close HTTP requests that take longer than 30–120 seconds. If stopping a VM or creating a database takes 125 seconds, the HTTP connection drops with a timeout error even though the operation is still running in Google Cloud.
- Not Knowing Whether the Command Worked: When an HTTP timeout cuts off a synchronous gcloud call, the agent gets no operation ID and has no way of knowing if the change failed, is still running, or finished a second after the connection closed. Retrying blindly can cause duplicate actions or 409 Conflict errors.
- “Done” Operation does not mean “Ready” Resource: Even when an operation reports status: "DONE", an agent shouldn't assume the resource is ready without checking the resource itself and also checking that the the operation didn't finish with an error.errors payload.
The 3-Phase Pattern: Submit, Poll, Verify
The official tool description for run_gcloud_command on the Cloud CLI MCP Server, explicitly recommends passing –async for slow commands:
The Phase 1 looks like this:
The arrow then leads to Phase 2, where we run a loop to verify if the desired operation is DONE.
And if DONE and no error, then we lead to Phase 3 as shown below, where we are verifying the resource state.
Note: I have split the diagram into 3 parts but you can conceptually join them together in a linear fashion to see the whole flow.
To summarize again, here is what each of the three phases executes over the Remote Server.
Approach 1: ADK Workflow with Cyclic Polling
In Google ADK (google-adk >= 2.0.0), a Workflow graph is not limited to a straight-line DAG, it can include conditional cycles powered by a deterministic Python FunctionNode router.
Rather than asking an LLM to keep calling operations describe in a loop and wait a few seconds between calls, which models cannot time reliably and may abandon early, we put Python in charge of the polling loop, backoff timer, and timeout ceiling, and use focused LlmAgent nodes for the MCP tool calls.
The 7-Node Cyclic Graph Architecture
The flowchart below shows how the 7 nodes in examples/07-long-running-commands/app/agent.py implement the 3-phase flow, including the cyclic edge from Node 5 (evaluate_poll_status) back to Node 4 (poll_operation_agent):
Let’s walk through the code and examine how each phase is implemented in examples/07-long-running-commands/app/agent.py.
Phase 1: Submitting with –async and Extracting the Operation ID (Nodes 1–3)
First, init_lro_context seeds the workflow state with:
- target VM (demo-web-server)
- requested action (start or stop)
- expected final state (RUNNING or TERMINATED)
- poll_interval_seconds (5)
- max_poll_attempts (10).
Next, Node 2 (submit_async_gcloud_agent) executes the mutation with –async –format=json –quiet and writes the raw JSON output to async_submission_result:
submit_async_gcloud_agent = LlmAgent(
name="submit_async_gcloud_agent",
model=os.environ.get("ADK_MODEL", "gemini-3-flash-preview"),
include_contents="none",
description="Submits a long-running gcloud compute instances start/stop command with --async.",
instruction="""You are the Async Command Submission Node in an ADK Workflow.
Task:
1. Call `run_gcloud_command` with:
- `project`: "projects/{target_project}"
- `command`: "gcloud compute instances {desired_action} {target_vm_name} --zone={default_zone} --project={target_project} --async --format=json --quiet"
2. Return ONLY a valid JSON object containing the operation fields (`name`, `operationType`, `status`, `targetLink`).
""",
tools=[gcloud_mcp_tools],
output_key="async_submission_result",
)
Immediately after, Node 3 (extract_operation_metadata) deterministically parses async_submission_result in Python and stores the canonical operation-… identifier into state["operation_name"].
Phase 2: The Cyclic Polling Loop (Nodes 4 & 5)
Node 4 (poll_operation_agent) reads {operation_name} from state and calls gcloud compute operations describe {operation_name} –zone={default_zone} –project={target_project} –format=json –quiet, storing the result in latest_operation_poll.
Node 5 (evaluate_poll_status) is important here. It parses latest_operation_poll in Python, checks for operation errors or completion (status == "DONE"), and if the operation is still PENDING or RUNNING, awaits asyncio.sleep(poll_interval_seconds) and emits route="continue" to jump right back to Node 4:
async def evaluate_poll_status(
node_input: str,
operation_name: str,
poll_attempt: int,
max_poll_attempts: int,
poll_interval_seconds: int,
latest_operation_poll: str = "",
) -> Event:
"""Inspects the polled operation status and routes to 'continue', 'done', or 'failed'."""
attempt = int(poll_attempt) + 1
op_data = _parse_json_dict(latest_operation_poll or str(node_input))
op_status = str(op_data.get("status", "UNKNOWN")).upper()
op_progress = op_data.get("progress", 0)
op_error = op_data.get("error")
if op_error or (op_status != "DONE" and attempt >= int(max_poll_attempts)):
return Event(
output={"poll_attempt": attempt, "operation_status": op_status},
route="failed",
state={"poll_attempt": attempt, "operation_status": op_status},
)
if op_status == "DONE":
return Event(
output={"poll_attempt": attempt, "operation_status": "DONE", "progress": 100},
route="done",
state={"poll_attempt": attempt, "operation_status": "DONE"},
)
# Still PENDING or RUNNING -> sleep and loop back to poll_operation_agent!
await asyncio.sleep(int(poll_interval_seconds))
return Event(
output={"poll_attempt": attempt, "operation_status": op_status, "progress": op_progress},
route="continue",
state={"poll_attempt": attempt, "operation_status": op_status},
)
Phase 3: Post-Operation Resource State Verification (Node 6)
Once evaluate_poll_status emits route="done", execution moves to Node 6 (verify_resource_state_agent). It calls gcloud compute instances describe on the VM and checks that actual_vm_status == expected_vm_status (RUNNING or TERMINATED).
Watch Out for Transitional States (PROVISIONING, STAGING, STOPPING, SUSPENDING): In Compute Engine, a VM does not jump straight from TERMINATED to RUNNING. While starting up, its status moves through PROVISIONING and STAGING; while shutting down, it moves through STOPPING. Verification should only pass when actual_vm_status reaches a settled terminal state (RUNNING or TERMINATED). Likewise, if an agent inspects a VM before acting and catches it mid-transition (PROVISIONING, STAGING, or STOPPING), waiting a few seconds and re-checking instances describe until the transition settles is much safer than firing a competing start or stop command.
Wiring the Cyclic Graph in Workflow
Finally, root_agent wires all 7 nodes together, including the cyclic edge "continue": poll_operation_agent:
root_agent = Workflow(
name="gcloud_async_lro_verification_workflow",
edges=[
(
"START",
init_lro_context,
submit_async_gcloud_agent,
extract_operation_metadata,
poll_operation_agent,
evaluate_poll_status,
{
"continue": poll_operation_agent, # <-- Cyclic polling loop
"done": verify_resource_state_agent, # <-- Exit on DONE
"failed": handle_lro_failure, # <-- Exit on error/timeout
},
),
],
)
Running the ADK LRO Workflow (run_adk_lro_workflow.py)
export GOOGLE_CLOUD_PROJECT="your-project-id"
uv run --project examples/07-long-running-commands \
examples/07-long-running-commands/run_adk_lro_workflow.py \
--project="${GOOGLE_CLOUD_PROJECT}" \
--zone=us-central1-a \
--vm-name=demo-web-server \
--action=start \
--poll-interval=3
The execution output that we observed:
==============================================================================
ADK 7-Node LRO Workflow : Async Submit -> Operation Polling -> Verification
Target Project : your-project-id
Target Zone : us-central1-a
Target VM : demo-web-server
Requested LRO Action : start (--async) -> Expect 'RUNNING'
Poll Policy : every 3s (max 20 polls)
==============================================================================
[init_lro_context]
text : [Node 1: init_lro_context] Target: 'demo-web-server' (us-central1-a in your-project-id) | Action: `gcloud compute instances start --async` | Expected Final State: RUNNING | Poll Policy: every 3s (max 20 attempts)
[submit_async_gcloud_agent]
tool : gcloud_mcp_run_gcloud_command({'command': 'gcloud compute instances start demo-web-server --zone=us-central1-a --project=your-project-id --async --format=json --quiet', 'project': 'projects/your-project-id'})
[extract_operation_metadata]
text : [Node 3: extract_operation_metadata] Captured Operation: {"operation_name": "operation-1792805117422-65e8bf82ced56-2c38e2a0-87561087", "operation_type": "start", "initial_status": "RUNNING", "progress": 0, "target_link": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/instances/demo-web-server"}
[poll_operation_agent]
tool : gcloud_mcp_run_gcloud_command({'command': 'gcloud compute operations describe operation-1792805117422-65e8bf82ced56-2c38e2a0-87561087 --zone=us-central1-a --project=your-project-id --format=json --quiet', 'project': 'projects/your-project-id'})
[evaluate_poll_status]
route : 'done'
text : [Node 5: evaluate_poll_status] Poll #1: Operation 'operation-1792805117422-65e8bf82ced56-2c38e2a0-87561087' reached status=DONE (progress=100%) -> route='done'
[verify_resource_state_agent]
tool : gcloud_mcp_run_gcloud_command({'command': 'gcloud compute instances describe demo-web-server --zone=us-central1-a --project=your-project-id --format="json(name,status,lastStartTimestamp,lastStopTimestamp)" --quiet', 'project': 'projects/your-project-id'})
text : {
"operation_name": "operation-1792805117422-65e8bf82ced56-2c38e2a0-87561087",
"operation_status": "DONE",
"vm_name": "demo-web-server",
"zone": "us-central1-a",
"expected_vm_status": "RUNNING",
"actual_vm_status": "RUNNING",
"verification_passed": true,
"last_start_timestamp": "2026-10-24T01:25:21.564-07:00",
"last_stop_timestamp": "2026-10-23T20:27:25.197-07:00",
"summary": "The asynchronous start operation (operation-1792805117422-65e8bf82ced56-2c38e2a0-87561087) completed with status DONE, and the Compute Engine instance 'demo-web-server' in zone 'us-central1-a' has been verified to be in the RUNNING state."
}
==============================================================================
Final Session State & Verification Summary:
==============================================================================
Operation Name : operation-1792805117422-65e8bf82ced56-2c38e2a0-87561087
Operation Final State : DONE (progress=100%)
Total Poll Attempts : 1
Expected VM Status : RUNNING
Verified VM Status : RUNNING
Verification Passed : True
==============================================================================
Approach 2: Managed Agents & Interactions API
What if you are using Managed Agents and the Gemini Interactions API on the Gemini Enterprise Agent Platform (aiplatform.googleapis.com/v1beta1) instead of hosting your own ADK workflow?
In examples/07-long-running-commands/run_managed_lro.py, long-running operations are coordinated across two layers:
- Control Plane (Managed Agents API): We provision (or PATCH with ?update_mask=system_instruction) a stateful Agent resource (cloud-cli-lro-agent) at https://aiplatform.googleapis.com/v1beta1/projects/{PROJECT_ID}/locations/global/agents whose system_instruction codifies the 3-Phase LRO & Verification Protocol.
- Data Plane (Gemini Interactions API): We execute the multi-turn conversation via client.interactions.create(…, store=True) using the unified Google Gen AI SDK (google-genai >= 2.3.0), chaining turns via previous_interaction_id.
Walking Through run_managed_lro.py
In run_managed_lro.py, we retrieve the provisioned cloud-cli-lro-agent from the Control Plane (client.agents.get("cloud-cli-lro-agent")) and drive a stateful client.interactions.create(…) loop across the 3 phases:
from google import genai
client = genai.Client(
enterprise=True,
project=project_id,
location="global",
)
# 1. Fetch the Managed Agent definition from the Control Plane
managed_agent = client.agents.get("cloud-cli-lro-agent")
# 2. Start Turn 1 on the Stateful Interactions API Data Plane (store=True)
interaction = client.interactions.create(
model="gemini-3-flash-preview",
system_instruction=managed_agent.system_instruction,
tools=[run_gcloud_tool],
input=prompt,
store=True,
)
# 3. Process `function_call` steps across turns using `previous_interaction_id`
for turn_idx in range(2, max_turns + 1):
function_calls = [
s for s in (interaction.steps or [])
if getattr(s, "type", None) == "function_call"
]
if not function_calls:
break
tool_results = []
for fc in function_calls:
cmd = (fc.arguments or {}).get("command", "")
mcp_output = invoke_remote_mcp_gcloud(project_id, cmd)
# Backoff between operation polls while status is PENDING or RUNNING
if "operations describe" in cmd:
parsed_op = parse_gcloud_mcp_json(mcp_output)
if parsed_op.get("status") in ("PENDING", "RUNNING"):
time.sleep(poll_interval)
tool_results.append(
{
"type": "function_result",
"name": fc.name,
"call_id": fc.id,
"result": [{"type": "text", "text": mcp_output}],
}
)
interaction = client.interactions.create(
model="gemini-3-flash-preview",
system_instruction=managed_agent.system_instruction,
tools=[run_gcloud_tool],
input=tool_results,
previous_interaction_id=interaction.id,
store=True,
)
Running run_managed_lro.py & Live Execution Output
export GOOGLE_CLOUD_PROJECT="your-project-id"
uv run --project examples/07-long-running-commands \
examples/07-long-running-commands/run_managed_lro.py \
--project="${GOOGLE_CLOUD_PROJECT}" \
--zone=us-central1-a \
--instance=demo-web-server \
--action=stop \
--poll-interval=3
We observed the following output:
==============================================================================
Managed Agent & Interactions API : LRO Execution & Verification
Target Project : your-project-id
Target Zone : us-central1-a
Target VM : demo-web-server
Requested LRO Action : stop (--async) -> Expect 'TERMINATED'
Data Plane Mode : interactions
==============================================================================
Updating existing Managed Agent 'cloud-cli-lro-agent' on Control Plane (?update_mask=system_instruction)...
Control Plane Managed Agent Ready: projects/your-project-id/locations/global/agents/cloud-cli-lro-agent (base_agent=antigravity-preview-05-2026)
=== Data Plane: Starting Stateful Interaction (store=True) ===
[Turn 1] interaction.id=v1_CiQ2ZWI2ODU0MC0wMDAwLTJmYTgtODMzNy1mYzQxMTY4YzhhNTkSJDZlYjY4NTQxLTAwMDAtMmZhOC04MzM3LWZjNDExNjhjOGE1OQ | status=requires_action
[Turn 1 -> Phase 1 Async Submit] Remote MCP: `gcloud compute instances stop demo-web-server --zone=us-central1-a --project=your-project-id --async --format=json --quiet`
-> Captured Operation: operation-1792811585390-65e8d79b245f2-a32ef294-dca90f15 (operationType=stop, status=RUNNING)
[Turn 2] interaction.id=v1_CiQ2ZWI2ODU0MC0wMDAwLTJmYTgtODMzNy1mYzQxMTY4YzhhNTkSJDZlYzg2OGE4LTAwMDAtMjYzYi05OTgwLTk4OThmYmQ0ZDIxZA | status=requires_action
[Turn 2 -> Phase 2 Poll #1] Remote MCP: `gcloud compute operations describe operation-1792811585390-65e8d79b245f2-a32ef294-dca90f15 --zone=us-central1-a --project=your-project-id --format=json --quiet`
-> Operation Status: DONE (progress=100%)
[Turn 3] interaction.id=v1_CiQ2ZWI2ODU0MC0wMDAwLTJmYTgtODMzNy1mYzQxMTY4YzhhNTkSJDZmNDUxYmM2LTAwMDAtMjk5MC04Y2JjLTA0MDA2ZTZlNmEwZA | status=requires_action
[Turn 3 -> Phase 3 Verify Resource] Remote MCP: `gcloud compute instances describe demo-web-server --zone=us-central1-a --project=your-project-id --format="json(name,status,lastStartTimestamp,lastStopTimestamp)" --quiet`
-> Target VM 'demo-web-server' actual status: TERMINATED
[Turn 4] interaction.id=v1_CiQ2ZWI2ODU0MC0wMDAwLTJmYTgtODMzNy1mYzQxMTY4YzhhNTkSJDZmMGU2M2M2LTAwMDAtMjkzNC05OTRlLTA0MDA2ZTZmMjNlZA | status=completed
=== Final Agent Response ===
The 3-Phase LRO Protocol for stopping the VM 'demo-web-server' in zone 'us-central1-a' has been successfully completed.
1. **Phase 1 (Submit):** The `stop` command was initiated with `--async`, returning operation `operation-1792811585390-65e8d79b245f2-a32ef294-dca90f15`.
2. **Phase 2 (Poll):** The operation was polled and reached the `DONE` status successfully.
3. **Phase 3 (Verify):** The VM 'demo-web-server' was inspected and confirmed to be in the `TERMINATED` state.
==============================================================================
Managed Agent & Interactions API LRO Verification Summary:
=============================================================================
Managed Agent Resource: projects/your-project-id/locations/global/agents/cloud-cli-lro-agent
Final Interaction ID : v1_CiQ2ZWI2ODU0MC0wMDAwLTJmYTgtODMzNy1mYzQxMTY4YzhhNTkSJDZmMGU2M2M2LTAwMDAtMjkzNC05OTRlLTA0MDA2ZTZmMjNlZA
Operation Name : operation-1792811585390-65e8d79b245f2-a32ef294-dca90f15
Operation Final State : DONE
Expected VM Status : TERMINATED
Verified VM Status : TERMINATED
Verification Passed : True
==============================================================================
Production Design Notes: 30-Second Operations vs. 30-Minute Operations
A common question when designing polling loops for cloud operations is:
“Sleeping for 3–5 seconds inside a workflow loop (await asyncio.sleep(…)) works well for a 20-second VM start or stop. What if an operation takes 20 minutes—like creating a Cloud SQL database or upgrading a GKE cluster—on Cloud Run?"
In production, you match the polling mechanism to how long the operation takes:
Catching a Resource Mid-Transition Before Submitting
As noted in Phase 3, a cloud resource can also be caught in a transitional state (PROVISIONING, STAGING, STOPPING, or REPAIRING) before your workflow even submits its –async command (for example, if another operator or an autoscaler triggered an action seconds earlier).
- If instances describe shows the VM is already transitioning toward your goal (PROVISIONING or STAGING when you want RUNNING), skip Phase 1 (start –async) and jump straight to polling instances describe until it settles into RUNNING.
- If it is transitioning the other way (STOPPING when you want RUNNING), wait and re-check until it settles into TERMINATED before deciding whether to start it, calling start while a VM is STOPPING will fail with a GCP resource-not-ready error.
Series Wrap-Up
Across this 4-part series and the companion repository, we have taken Google Cloud's stateless gcloud and bq Remote MCP Server from a first curl test all the way to production cloud operations:
- We started off with understanding the Cloud CLI MCP Server and connected it to curl, local IDEs, a single-agent ADK service, and Gemini Enterprise Managed Agents, without installing the local Cloud SDK.
- We then moved moved from a single prompt to a predictable ADK Workflow graph (FunctionNode + LlmAgent + Event routing).
- Having humans approve commands that have cost implications and otherwise is important. So we added Human-in-the-Loop (HITL) approval gates (@node(rerun_on_resume=True) + RequestInput) deployed on Cloud Run and backed by VertexAiSessionService, complete with approval expiry, exact command pinning, and Python before_tool_callback checks.
- Finally, several resource intensive commands take a long time to execute. This led us to handling slow infrastructure changes using the 3-Phase Submit (–async) → Poll (operations describe) → Verify (instances describe) pattern across both ADK workflows and Gemini Enterprise Managed Agents.
Google Cloud CLI Remote MCP Server (Part 4): Handling & Verifying Long-Running gcloud Commands… was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/google-cloud-cli-remote-mcp-server-part-4-handling-verifying-long-running-gcloud-commands-7b1ebc34dadb?source=rss—-e52cf94d98af—4
About the Author
