
For the past decade, running workloads on Google Kubernetes Engine (GKE) has rested on one foundational assumption: we know the code running inside our pods.
Think about how a traditional microservice reaches a node. An engineer writes the code, a peer reviews the pull request, CI/CD compiles it, Artifact Analysis scans for CVEs, and Binary Authorization verifies container signatures. Because that code was vetted before deployment, standard Linux cgroups and namespaces sharing the host kernel satisfied our security and operational models.
With AI agents things change – fast
With AI agents, those assumptions collapse. When an AI agent writes a Python script, runs Bash commands, or calls tools on the fly it executes arbitrary, untrusted code at runtime. None of which has gone through your CI/CD pipeline. Just as challenging is that agents often introduce erratic operational patterns: lengthy idle periods waiting on LLM inference, paired with requirements for instant, sub-second execution that standard Pod scheduling cannot deliver.
For GKE administrators, this shift raises the question — Why isn’t a standard pod enough for AI agents, and what needs to sit on top of GKE to run them safely at scale?
In this blog post, we will look at the agent runtime-focussed progressions that Google has built on top of GKE – from Sandboxes (gVisor & MicroVMs) for kernel isolation, to Agent Sandbox Warm Pools for subsecond provisioning, to Agent Substrate for suspend/resume lifecycle management . We will also dive into some hands-on to see these technologies in action.
GKE platform agentic additions
To understand why Google built new primitives on top of GKE for AI agents, we need to follow the risks that exist the moment an application evolves from a traditional microservice into an autonomous agent. The key issues/risks and GKE solutions can be represented with the chart below:

Risk 1: Untrusted code
If the code is generated on the fly, sharing the host kernel is a big risk. Mitigation is to have a hard isolation boundary between the container and the GKE node via either GKE Sandbox (gvisor) to intercept syscalls in user space, or GKE MicroVM (microvm) to put a full hardware VM boundary around the Pod. gVisor acts as a lightweight sandbox for fast, dense, general-purpose tasks by intercepting system calls before they reach the host kernel. MicroVMs act as hardware-isolated mini virtual machines that give each agent its own dedicated OS kernel for maximum isolation and full tool compatibility.
Risk 2: Speed
Once every untrusted task or session requires its own sandboxed pod (1:1 pod:agent), standard pod scheduling can become a bottleneck: waiting seconds to schedule a pod and initialize a runtime can break interactive agent loops. Agent Sandbox on top of GKE provides pre-warmed sandbox pools (SandboxWarmPool) and a declarative Claim Model (SandboxClaim) that hands a running, network-jailed sandbox to an agent in under 1 second (<1s).
Risk 3: Costly Idle time
Agents are mostly stateful (they hold in-memory context and local workspace files over hours or days), yet they can spend the majority of their time idle waiting for the user’s next prompt.
As standard GKE pods cannot “hibernate”, keeping 1000s of idle sandbox pods running wastes compute (also exhausting cluster IP space). But deleting the pods destroys the agent’s working memory.
Enter Agent Substrate on GKE — an architecture that runs agents as logical Actors instead of standard Deployments. Active agent sessions are dynamically assigned to a shared fleet of warm Worker pods, bypassing the pod lifecycle and eliminating the cost of idle containers. When an agent goes idle, Agent Substrate snapshots its active RAM and filesystem to Cloud Storage and frees the Worker pod — and then restores that exact RAM state onto a warm worker in sub 500ms when the next request arrives.
Building the Demo
In this demo, we will deploy a Share Price Analyst Agent (built with Google ADK and FastAPI) side by side on regular GKE and Agent Substrate on GKE. This agent adds the name of every stock a trader has asked to analyze to their running watchlist in-memory, and checks which Pod IP it’s currently running on compared to the last turn.
Side by side deployments mean we can directly watch Agent Substrate claim warm workers in milliseconds, trap untrusted code inside gVisor, and freeze and restore live memory across different Worker Pods
The tests we will run will showcase::
- How long it takes to spin up a new trader session on a cold regular GKE Pod versus claiming a warm Worker Pod on Agent Substrate ( the speed risk)
- What happens when the agent session goes idle between prompts ( the idle risk)
- What happens when a trader (or a prompt injection) asks the agent to execute a custom Python financial formula on the fly that also attempts to probe the host kernel (the isolation risk)
1. Deployed Architecture
Here is what the deployed architecture we are looking to deploy:

All source code is available in the companion GitHub repository: github.com/opowbow/gke_substrate_stocks. To follow along, download the code:
git clone https://github.com/opowbow/gke_substrate_stocks.git
cd gke_substrate_stocks
2. The Agent
Our agent (agent/agent.py) calculates both stock price for an input list of companies; and 30-day annualized stock volatility:
# Excerpt from agent/agent.py
SYSTEM_INSTRUCTION = """You are a Senior Quantitative Financial Analyst running on GKE.
When asked about a stock ticker:
1. Call 'get_stock_quote' to retrieve the current share price.
2. Call 'calculate_volatility' to compute 30-day annualized volatility.
3. Provide a concise risk summary and reference any tickers analyzed earlier in this session.
"""
The agent keeps the trader’s session history, a resident_secret token, and its previous/current Pod IPs entirely in process RAM (no external database — see main.py)
# Excerpt from agent/main.py -- pure in-memory session state
PORTFOLIO_STATE = {
"resident_secret": None,
"boot_pod_ip": get_current_ip(),
"last_pod_ip": None,
"tickers_analyzed": [],
"turn_count": 0,
}
3. Build and Push the Agent
Build from the agent/ folder, and retrieve the image digest (needed later for ActorTemplate)
export IMAGE="${REGION}-docker.pkg.dev/${PROJECT_ID}/agent-demo/stock-analyst:v1"
gcloud artifacts repositories create agent-demo \
--repository-format=docker \
--location=${REGION} || true
gcloud builds submit --tag ${IMAGE} ./agent
# Retrieve the @sha256 digest-pinned image URI required by ActorTemplate
export IMAGE_DIGEST=$(gcloud artifacts docker images describe ${IMAGE} \
--format='value(image_summary.fully_qualified_digest)')
echo "Digest-pinned image: ${IMAGE_DIGEST}"
4. Deploy Agent Substrate
a. Start Agent Substrate install: (Note — Agent substrate install notes are documented here)
curl -sSL https://raw.githubusercontent.com/ai-on-gke/substrate-gke/main/install.sh | bash
b. Select [1] Quickstart:

c. Make sure all prerequisite tests pass:

d. Select Install pre-built images for the Substrate control plane:

e. Select a current cluster or new one (I selected [2] Create a new cluster):

f. Make sure all the clusters create and GCP resource check steps pass:

g. For Ingress clients (required) enter the SPIFFE IDs of the workloads that send requests to your Actors:
spiffe://cluster.local/ns/default/sa/default,spiffe://cluster.local/ns/ate-system/sa/atenet-client
(Note — These two identities allow client applications running in the default namespace (our agent) to send requests to Actors, while also letting the Substrate CLI proxy (kubectl ate proxy) in ate-system connect for administrative tasks.)
h. Turn on the Substrate:

i. Skip the Filestore CSI install:

j. We don’t need autoscaling , so skip to keep a fixed node count:

k. We will be using gVisor in this demo, so select option 1:

l. The Agent Substrate is now installed:

5. Grant SA permissions
Set up project and cluster variables, and grant the newly created substrate-node-pool SA permission to pull from the agent-demo repo
export NODE_SA=$(SA=$(gcloud container node-pools describe substrate-node-pool \
--cluster=${CLUSTER_NAME} --zone=${ZONE} --project=${PROJECT_ID} \
--format='value(config.serviceAccount)'); \
[ "$SA" = "default" ] && echo "$(gcloud projects describe ${PROJECT_ID} --format='value(projectNumber)')-compute@developer.gserviceaccount.com" || echo "$SA")
echo "NODE_SA=${NODE_SA}"
gcloud artifacts repositories add-iam-policy-binding agent-demo \
--location=${REGION} \
--project=${PROJECT_ID} \
--member="serviceAccount:${NODE_SA}" \
--role="roles/artifactregistry.reader"
6. Install the Kubectl ate plugin
While WorkerPool is a standard Kubernetes CRD applied via kubectl apply, Agent Substrate resources like atespace and ActorTemplate are managed through its control-plane API using the kubectl ate plugin. We need to install this from the Substrate repository:
git clone --depth 1 https://github.com/agent-substrate/substrate.git /tmp/substrate-repo
# 2. Build and install the plugin
cd /tmp/substrate-repo
go install ./cmd/kubectl-ate
# 3. Verify the plugin is installed
kubectl ate --help
# 4. Clean up source
rm -rf /tmp/substrate-repo
7. Deploy Agent Substrate resources
Next, deploy the 2-Pod gVisor WorkerPool and create the ActorTemplate. Registering the template instructs Agent Substrate to boot your container image once on a warm worker, verify its readiness, and capture a golden full-memory snapshot to Cloud Storage.
a. Prepare the Manifests
Before applying the manifests, replace the placeholder values in the YAML files under manifests/ with your environment’s values
(Note: you can view fully populated examples in the samples/ folder):
- IMAGE_DIGEST_PLACEHOLDER (in substrate-actor-template.yaml): The @sha256 digest-pinned Artifact Registry image from Step 2 (${IMAGE_DIGEST}).
- IMAGE_PLACEHOLDER (in standard-gke-pod.yaml): The Artifact Registry image from Step 2 (${IMAGE}).
- WORKER_IMAGE_PLACEHOLDER (in substrate-workerpool.yaml): The ateworker-gvisor image from the installer (e.g., ghcr.io/agent-substrate/substrate/ateworker-gvisor:v1.0.0, This can be checked via:
kubectl get ds atelet -n ate-system -o wide
b. Deploy the Manifests
# 1. Apply the Agent Substrate WorkerPool
kubectl apply -f manifests/substrate-workerpool.yaml
# 2. Create the trading-desk atespace and ActorTemplate (creates golden RAM snapshot)
kubectl ate create atespace trading-desk
kubectl ate create actor-template -f manifests/substrate-actor-template.yaml
c. Check the Resources
The resources applied at this stage can be viewed as follows:

Check Agent Substrate resources:
kubectl ate get atespace
kubectl ate get actor-template -a trading-desk
kubectl ate get actor -a trading-desk

Check Kubernetes Resources:
kubectl get workerpools -n default
kubectl get pods -n default -o wide

Testing
With our cluster resources running, we can evaluate each of the three agent risks discussed in isolation. We will use a script – run_demo.py – to help with this. This script automates kubectl, kubectl ate, and live HTTP requests against both the regular GKE pod and our Agent Substrate actors. But crucially, it gives us extra capability around extracting and presenting the output details for comparison.
Let’s prepare for the tests by creating our first trader’s dedicated agent session: trader-alice, from the stock-analyst golden memory snapshot inside the trading-desk atespace, and then suspend:
kubectl ate create actor trader-alice -a trading-desk --template stock-analyst
kubectl ate suspend actor trader-alice -a trading-desk
Test 1: Speed – Regular GKE Pod vs. Warm Agent Substrate Worker
In this first test, we will deploy our agent pod onto GKE and check the timing for that; while on the Agent Substrate side, we will resume the trader-Alice session and issue a NVDA stock query – and check the timing for that. The manual commands for these are as below, for reference:
kubectl apply -f manifests/standard-gke-pod.yaml
kubectl wait --for=condition=Ready pod/stock-analyst-cold-pod --timeout=120s
and
# 1. Open the mTLS proxy tunnel and send /analyze (wakes Alice on a warm worker + runs NVDA analysis)
kubectl ate proxy actor trader-alice 8081:8080 -a trading-desk &
curl "http://127.0.0.1:8081/analyze?symbol=NVDA"
# 2. Check which warm WorkerPool Pod & IP Alice claimed
kubectl ate get actor -a trading-desk
The ./run_demo.py will run these commands and output the captured timings. It will also capture the stock ticker queried and its 30-day volatility, and the worker pod IP – all of which will be shown later as stored memory for the session. All of this and some pretty icons!
Run the test:
python3 run_demo.py --test 1

On the Regular GKE side, a new pod is created and checked for /readyz
On the Agent Substrate side, the trader-alice session sends an /analyze?symbol=NVDA request through kubectl ate proxy — prompting the Agent Substrate to match the stock-analyst template’s selector to a warm worker Pod in stock-analyst-pool, restore the golden RAM snapshot into its gVisor sandbox, and return the NVDA volatility analysis.
Times from each test are noted and presented in the output results. In this demo environment, Agent Substrate is 2.4 times faster.
Test 2: Idle – What Happens When the Agent Session Goes Idle Between Prompts
In this test we check if we can free the underlying Worker Pod for another trader without losing the first trader’s in-memory session state. This means we suspend the current running session (with NVDA as the stock queried), and start a new session with a different stock request (MSFT). We then check that the worker pod is reclaimed (same IP).
Finally we will re-start the first session with a different stock query ) GOOGL) and show that the memory from the previous (first) session is retained (NVDA query).
The process flow looks like this:
trader-alice → ACTOR_STATE_SUSPENDED (WORKER POD IP disappears)
trader-bob → ACTOR_STATE_RUNNING (claims the WORKER POD (same IP))
trader-alice → ACTOR_STATE_RUNNING (reappears on the other WORKER POD (different IP – as we have 2 worker pod replicas but stock ticker memory))
The manual steps are:
kubectl ate suspend actor trader-alice -a trading-desk
kubectl ate create actor trader-bob -a trading-desk — template stock-analyst
kubectl ate proxy actor trader-bob 8082:8080 -a trading-desk &
curl “http://127.0.0.1:8082/analyze?symbol=MSFT"
kubectl ate proxy actor trader-alice 8081:8080 -a trading-desk &
curl “http://127.0.0.1:8081/analyze?symbol=GOOGL")
Run the test:
python3 run_demo.py --test 2

As we can see, on Agent Substrate, trader-alice’s live RAM freezes directly to Cloud Storage in 3936ms — freeing Worker pod stock-analyst-pool-8b5b86c66–4rqlp,IP=10.12.3.20. trader-bob claims that Pod (= 0ms hardware provisioning).
When the trader-alice session resumes, Agent Substrate restores her RAM snapshot onto a different Worker Pod (stock-analyst-pool-8b5b86c66-h6x6d,IP=10.12.2.9) in 3415ms, with her FOLIO-BHIJR2 token and [‘NVDA’,’GOOGL’] watchlist intact in memory — with zero external database.
Sandbox states stored in GCS
As an extra, let’s check the sandbox state in Cloud Storage. First, suspend the trader-alice session and inspect the UID and active snapshot URI in JSON. Use this to list the snapshots in GCS ( in gcloud):
kubectl ate suspend actor trader-alice -a trading-desk
kubectl ate get actor trader-alice -a trading-desk -o json
gcloud storage ls -l --readable-sizes -r gs://ate-snapshots-op-agents-001-gke-substrate-001-us-west1-c
or in the console

Test 3: Isolation — On-the-Fly Python Code Execution & Host Kernel Probe
In this test we check what happens when, instead of calling a fixed CI/CD-reviewed endpoint, the trader asks the AI agent to generate and execute a custom code on the fly.
Note: In a real production deployment, Google Gemini’s built-in safety filters and core protections actively refuse to generate exploit payloads or destructive system commands.
To test execution of malicious code in this blog , we will bypass Gemini and send a pre-written Python payload directly to the ./run_fly_script. This allows us to focus on NOT how the code got created but how the container runtime isolates it the moment exec() within the container runs.
The test script does 2 things: 1. run a known financial calculation (Sharpe Ratio) on the NDVA stock, and 2. run some malicious behavior i.e. checking the OS kernel (os.uname()), attempting to read /dev/kmsg, opening a raw packet-sniffing socket (AF_PACKET), and looking for a mounted Kubernetes ServiceAccount token. We can then compare what a Regular GKE Pod exposes versus what GKE Agent Substrate’s gVisor sandbox blocks.
The code used in this test lives in two files:
1. Fly_body defined in test_3_isolation_risk in ./run_demo.py
def test_3_isolation_risk(router_url: str = ""):
banner("TEST 3 (THE ISOLATION RISK): On-the-Fly Python Code & Host Kernel Probe")
print(f"\n{CYAN}[Prompt Injection Simulation]{RESET} Trader submits on-the-fly Python script (/run_fly_script):")
print(' "Calculate NVDA Sharpe Ratio AND probe host kernel, /dev/kmsg, raw sockets & KSA token"')
fly_body = {
"symbol": "NVDA",
"code": (
"returns = [0.018, -0.024, 0.021, -0.014, 0.011]\n"
"mean_r = sum(returns) / len(returns)\n"
"sq_diff = 0.0\n"
"for r in returns:\n"
" sq_diff += (r - mean_r) ** 2\n"
"std_r = (sq_diff / (len(returns) - 1)) ** 0.5\n"
"sharpe_ratio = round((mean_r / std_r) * (252 ** 0.5), 2)"
),
2. A. OS probes and B. Container execution defined in agent/main.py
A.
def run_sandbox_isolation_audit() -> dict:
"""Audits kernel isolation & escape vectors when executing untrusted code on the fly."""
kernel_release = os.uname().release
is_gvisor = "gvisor" in kernel_release.lower() or kernel_release.startswith("4.4.")
# Probe 1: Attempt to read host kernel ring buffer (/dev/kmsg)
try:
with open("/dev/kmsg", "r", encoding="utf-8") as f:
f.read(16)
kmsg_status = "EXPOSED (Read /dev/kmsg succeeded)"
except Exception as e:
kmsg_status = f"BLOCKED ({type(e).__name__}: {e})"
# Probe 2: Attempt to open a raw packet socket (CAP_NET_RAW / host network sniff)
try:
s = socket.socket(socket.AF_PACKET, socket.SOCK_RAW)
s.close()
raw_sock_status = "EXPOSED (AF_PACKET raw socket opened)"
except Exception as e:
raw_sock_status = f"BLOCKED ({type(e).__name__}: {e})"
# Probe 3: Check if a Kubernetes ServiceAccount token is mounted in the filesystem
ksa_path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
ksa_mounted = os.path.exists(ksa_path)
return {
"kernel_release": kernel_release,
"isolation_boundary": (
f"gVisor User-Space Kernel (runsc Sentry -- Host Kernel Hidden) ({kernel_release})"
if is_gvisor
else f"Shared Host Linux Kernel (runc -- Node Kernel Exposed) ({kernel_release})"
),
"is_sandboxed": is_gvisor,
"kmsg_probe": kmsg_status,
"raw_socket_probe": raw_sock_status,
"ksa_token_mounted": ksa_mounted,
}
B.
@app.post("/run_fly_script")
async def run_fly_script(req: FlyScriptRequest):
t0 = time.perf_counter()
exec_scope = {
"__builtins__": {"sum": sum, "len": len, "round": round, "min": min, "max": max, "abs": abs},
"math": math,
"symbol": req.symbol,
}
exec(req.code, exec_scope) # Dynamically compiles & executes incoming Python code
return {
"symbol": req.symbol,
"sharpe_ratio": exec_scope.get("sharpe_ratio"),
"exec_ms": round((time.perf_counter() - t0) * 1000, 2),
"sandbox_audit": run_sandbox_isolation_audit(),
}
Run the test:
python3 run_demo.py --test 3

Both environments compute the on-the-fly Python risk script (NVDA Sharpe Ratio = 1.89) in ~0.2ms, but the Regular GKE Pod executes untrusted syscalls directly against the GKE node’s shared Linux 6.18.55+ kernel with a mounted Kubernetes ServiceAccount token (KSA Token Mounted: True).
On the Agent Substrate side, the ateworker-gvisor worker traps every syscall inside the user-space gVisor Sentry (4.19.0-gvisor) – hiding the host kernel completely, omitting /dev/kmsg, blocking AF_PACKET raw sockets, and not mounting any Kubernetes ServiceAccount credentials inside the actor (KSA Token Mounted: False).
Same GKE Control Plane, now built for Untrusted Agents
Moving from traditional microservices to autonomous AI agents flips a decade-old Kubernetes assumption on its head: you can no longer assume every line of code executing inside a container went through Git, code review, and CI/CD. At the same time, interactive agents behave nothing like stateless web servers – they need lightning fast startup, hold state in memory, and sit idle a lot of the time.
With GKE Agent Substrate, you can build on GKE infrastructure and management investments, to help solve this. By layering GKE Sandbox (gvisor), Agent Sandbox warm pools, and Agent Substrate directly onto the clusters you already run, GKE gives you one unified control plane for every piece of an agentic architecture.
Next Steps: Try It Yourself
- Run the side-by-side benchmark on your own cluster: Clone github.com/opowbow/gke_substrate_stocks and run python3 run_demo.py – test all to see the spin-up race, stateful RAM teleportation across Worker Pods, and gVisor isolation audit live.
- Explore the GKE documentation: Dive deeper into GKE Sandbox & MicroVMs, Agent Sandbox on GKE, Agent Substrate on GKE, and Model Armor.
Running Untrusted AI Agents on GKE Agent Substrate: Sub-Second, Zero-Idle, and Sandboxed was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/running-untrusted-ai-agents-on-gke-agent-substrate-sub-second-zero-idle-and-sandboxed-c80dcd2aad09?source=rss—-e52cf94d98af—4
