Accelerate agentic RL with GKE Agent Sandbox

When scaling up agentic reinforcement learning (RL) and evaluation across massive parallel rollouts, frontier AI labs inevitably hit a bottleneck: Expensive GPU clusters sit idle, waiting minutes for CPU sandbox cold-starts, plus thousands of multi-gigabyte SWE-bench-style image pulls and scheduling backlogs. It’s a sandbox infrastructure problem that silently slows down your research and burns your training budget.

To solve this fundamental infrastructure bottleneck, today we are introducing GKE Agent Sandbox optimized for RL along with the Agent Sandbox RL orchestration SDK, plus native integrations for popular RL gyms and harnesses, now generally available.

As the operating system for modern AI, Kubernetes has evolved to power massive GPU/TPU training clusters and distributed inference. Now Kubernetes is expanding to drive the next AI compute frontier: agents. But unlike static workloads, agentic workloads evolve rapidly, so infrastructure must evolve just as fast. Rather than guessing at what RL researchers needed, we placed Kubernetes itself on an auto-research and verification loop driven by performance benchmarks and evaluations. We used heavy agentic benchmarks like SWE-bench to intentionally stress-test and break our own clusters. Every bottleneck that surfaced — from etcd timeouts to GPU idle spikes — was fed back into our development cycle to refine GKE’s core primitives.

This resulted in a purpose-built sandbox layer for agentic RL and eval workloads that features:

• 10x – 45x faster time-to-first-command: GKE can spin up a sandbox environment in 1–9 seconds instead of 45–85 seconds, keeping your expensive GPUs fully in use.
• Reduced tail latency: We reduced the worst-case sandbox wait times from 7.5 minutes down to under 10 seconds.
• 3x less control-plane churn: Each RL training step triggers a rollout burst, where thousands of sandboxes are requested at once. The SDK has an in-place recycling strategy that reuses pods across rollouts instead of deleting and re-creating them. That means 3x fewer pods being created, which keeps the Kubernetes API server stable under the churn.

With this new primitive, AI labs and agent-native startups can now reliably run large scale agentic RL trajectories and evals simultaneously, minimizing accelerator idle time and drastically accelerating their research velocity.

The real-world challenges of agentic RL infrastructure

Before we talk about the solution, let’s be precise about what makes agentic RL so demanding for infrastructure in the first place. In a standard agentic RL loop, an LLM policy generates actions like code snippets on GPUs and executes them inside isolated CPU sandboxes to observe a reward signal. However, when scaling up this loop to support tens of thousands of parallel rollouts, three critical infrastructure bottlenecks emerge:

1. Accelerator idle costs, i.e., time-to-first-command (TTFC) and the tail latency trap: Agentic RL is a batch workload, and in a synchronous RL step, training cannot proceed until the slowest sandbox in the batch is ready. If standing up CPU sandboxes takes minutes to provision, pull images, and execute initial startup scripts, expensive accelerator capacity sits wasted.
2. Massive image cardinality: Standard container caching assumes a handful of static base images. However, in agentic RL, every single task (like thousands of GitHub repositories in SWE-bench or R2E-Gym) often requires a completely distinct OCI container image. Managing thousands of unique, large images per run can lead to severe image-pulling bottlenecks and storage friction.
3. Control-plane saturation: As a bursty batch workload, agentic RL typically executes batch-size image tasks and multiple rollouts per task (e.g. 4, 8, 16 rollouts per task), pushing an already large number of images to the extreme. For instance, one public dataset we tested against has 4,578 R2E images. Assuming four rollouts per image, that means 18,312 tasks, which require 18,312 sandboxes simultaneously. Standard Kubernetes control planes degrade significantly under these burst loads. Churning tens of thousands of ephemeral pods per minute creates API-server queue bottlenecks, pod state errors and triggers false node-health evictions.

These are not hypothetical problems. They are the daily reality for frontier AI labs training state-of-the-art agents. 

GKE Agent Sandbox optimized for RL

To fan-out the code execution sandboxes, we set up a relatively modest cluster — a 10-node gVisor sandbox pool with GKE image streaming enabled, the Agent Sandbox controller, and an in-cluster SDK driver to claim warm pods. We tested various strategies and setups including a large number of images and high cardinality.