Coding agents do heavy engineering work. The agent harness manages runtime state, sandbox execution, and prompt preparation. Providing static surrounding context on every turn creates a multi-turn scaling bottleneck, wastes token and increases cost.
There are a few ways to prevent this. One way is through Gemini Enterprise Agent Platform Context Caching, which helps eliminate this waste. By keeping the static codebase in server cache and sending only the dynamic updates on subsequent turns, the harness cuts transmitted input volume.
Here are the reasons why naive harness implementations can incur unnecessary costs and how to avoid it by building a production caching pipeline with Google ADK 2.0 and the Gemini Enterprise Agent Platform.
https://medium.com/media/d77cda9dc2720e6b1f6b1299c2aabd7b/href
Cost Drivers of Naive Harness Implementations
Multi-turn agent loops can unlock powerful capabilities but understanding token acclamation dynamics and optimizing prompt layout are critical steps towards building cost-effective, scalable agent architectures.
The Token Accumulation Problem
Large language models are stateless. They process an input token sequence and return generated text. The agent harness maintains conversational memory and serializes the full context window on each model invocation.
Consider the arithmetic of a typical multi-turn coding loop. If your static reference code contains 37,500 tokens and each turn appends a 300-token error traceback, a 5-turn run transmits 37,800 tokens on turn one, 37,800 on turn two, and so on. By the fifth turn, you have billed 189,000 prompt tokens. A ten-turn refactoring task against a large codebase consumes nearly 400,000 prompt tokens. The network spends time re-uploading identical files, and the server spends compute re-tokenizing them.
The Prefix-Breaking Trap in Harness Design
Most developers who add context caching to an agent loop fail to see cache hits. The cause is almost always Prefix Invariance Violation.
Context caching requires an exact, byte-for-byte token match starting from token zero of the prompt. If the harness places any mutable runtime metadata before or inside the static text, the prompt hash changes. The server cannot match the request to the pre-computed KV cache. It drops into a cache miss and bills the full prompt at standard rates.
Dynamic variables belong at the end of the prompt, never at the beginning.
How Server-Side Context Caching Operates
Transformer models process input prompts by calculating Key-Value (KV) attention matrices for every token across each layer. In standard inference, the server recalculates these KV matrices from scratch on every turn. Gemini Context Caching creates a persistent, addressable server-side representation of these KV states. The harness writes the static codebase once. The server tokenizes the text, computes the attention states, and returns a unique cache reference.
On subsequent turns, the harness passes only the dynamic suffix (such as a unit test traceback or compiler error) along with the cache reference. The model loads the pre-computed KV activations directly into its attention layers.
Understanding the Dual Savings Model
Context caching delivers two distinct types of efficiency:
- Bandwidth and Ingestion Savings: On a 5-turn workflow, you transmit the 37,500-token codebase once instead of five times. The remaining four turns transmit only the short 300-token execution diffs. This cuts physical network data transmission by roughly 80%.
- Billing and Cost Savings: Under Agent Platform, cached content reads are billed at a 75% discount, meaning you pay 0.25x the standard prompt rate for cached tokens. Accounting for the initial 1.0x write cost, a 4-turn run cuts total prompt token costs by 50%, and a 5-turn run cuts costs by 55%.
Implementing a Successful Caching Pipeline
A successful context caching architecture relies on deterministic payload structure and centralized lifecycle management. Below is the implementation breakdown of core components to build a caching pipeline.
Enforcing Prefix Invariance in the Harness
To prevent prefix corruption, the harness uses CachePayloadBuilder to strictly isolate immutable prompt headers from dynamic execution suffixes. Here are the key methods from CachePayloadBuilder:
Implementing the Cache Manager in Your Harness
The ContextCacheManager handles the lifecycle of cached resources on Agent Platform. It computes content hashes to reuse existing active caches, extends time-to-live (TTL) settings when needed, and dispatches cached inference requests.
Core Lifecycle Methods
Here are the key lifecycle operations from ContextCacheManager:
End-to-End Walkthrough: Multi-Target Modernization
To analyze caching in a realistic scenario, we constructed a five-module batch modernization workload. The harness modernizes five legacy Python 2.7 modules against a shared 37,659-token monorepo reference prefix. The target modules contain semantic traps including integer division changes, binary versus text CSV parsing, and deprecated dictionary iterators.
See context-caching/multi_target_suite.py for the complete target definitions and sandbox test harness.
Measured Results Across Three Topologies
We executed empirical analyses across three distinct multi-agent topologies in Python 3.11 sandbox environments on Google Cloud Run.
Scenario 1: Multi-Target Batch Modernization (5 Turns)
Modernizing five independent legacy Python files against a 37,659-token monorepo prefix.
- Uncached Baseline: 190,139 input tokens
- Cached Transmitted Tokens: 39,503 tokens (37,659 write + 1,844 dynamic suffixes)
- Transmitted Reduction: 79.46% (150,636 tokens eliminated)
Scenario 2: Adversarial SQLi Red/Blue Debate (4 Turns)
A four-turn debate between a Red Team exploiter agent and a Blue Team fixer agent evaluating database security against a 38,367-token OWASP specification.
- Uncached Baseline: 154,008 input tokens
- Cached Transmitted Tokens: 38,907 tokens
- Transmitted Reduction: 74.69% (115,101 tokens eliminated)
Scenario 3: Multi-File Dependency Graph Modernization (4 Turns)
Cascading refactoring across four interdependent microservice layers against a 38,441-token Object-Relational Mapping (ORM) database SDK.
- Uncached Baseline: 154,479 input tokens
- Cached Transmitted Tokens: 39,156 tokens
- Transmitted Reduction: 74.56% (115,323 tokens eliminated)
Platform Engineering Decision Framework
Context caching provides massive structural advantages, but platform engineers must apply it selectively based on workflow characteristics.
Follow these operational rules:
- If context size is under 32,768 tokens, do not cache. Below this threshold, prompt transmission costs are minor, and cache management overhead is not justified.
- If the turn count is under three, do not cache. Single-turn and two-turn calls cannot amortize the 1.0x cache creation cost.
- If prompt headers change every cycle, do not cache. Refactor prompt construction to isolate static files before attempting to cache.
- If running iterative test-and-repair loops against large repositories, always cache. Enforce prefix invariance in your harness using CachePayloadBuilder and route calls through ContextCacheManager.
Running the Reproducible Analyses
You can run these empirical analyses locally using the test suite and CLI runner:
Get started on your own
Context caching can turn multi-turn agent loops from an expensive token drain into a predictable and scalable architecture. The rules are simple, if you pass large static codebases across three or more turns, isolate your dynamic context to suffixes, pin a server-side cache and let the model reuse it.
Use the code and test suites in this repository to get started.
How to Slash Token Costs with Context Caching in Agent Harnesses was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/how-to-slash-token-costs-with-context-caching-in-agent-harnesses-6431ba16d931?source=rss—-e52cf94d98af—4
About the Author
