The Software Factory needs an Eval Factory : Benchmark Coding Agents on your own repo with repo2bench & Harbor, and Optimize with Harness²
When everyone has access to the same frontier models and agent harnesses, your unique advantage is your engineering judgment — and how systematically you turn that judgment into a custom benchmark that evaluates and optimizes agents on your own codebase.
Step 1: The Anatomy of a Coding Agent Benchmark — What Drives Quality, Cost, and Speed
You never deploy a raw model in production — you deploy a Coding Agent (Harness × Model × Thinking Budget).
We ran 160 container-isolated Harbor trials on WorkBuddy Bench across 8 Gemini configurations:
- 2 Harnesses: Antigravity (v0.1.22) vs. Pi (v0.73.1)
- 2 Models: Gemini 4 Argon vs. Gemini 3.8 Flash
- 2 Thinking Budgets: medium vs. high
Swapping the harness or thinking budget shifts quality by 3× (18% to 50%) and cost by 15× (\$0.11 to \$1.73 per task):
Why does the exact same task cost so much more under certain settings? Two factors control your token bill:
Model Choice: Backend Token Framing (Flash vs. Argon)
Send the exact same Turn-1 request (System Prompt + 7 Tools + Task Prompt) to both models. Gemini 4 Argon bills 24% more Turn-1 tokens (1,909 vs. 1,538) because its backend injects extra thinking headers (+44 tok) and per-tool schema wrappers (+48 tok/tool):
Harness Choice: Turn-1 Cache Floor & Long bash Stdout/Error Spikes (Pi vs. Antigravity)
Pi sends a tiny Turn-1 prompt (1.5K tokens) that misses Gemini's implicit cache floor, and its raw bash commands frequently dump 20K+ tokens of uncached stdout/error logs into the prompt at once. Antigravity sends a larger structured Turn-1 prompt (8.8K tokens) that caches immediately and uses bounded view_file reads—making Antigravity cheaper on tasks with heavy tool output (\$0.30 vs. \$0.37) despite using nearly 2\times more total tokens:
Step 2: Build Your Eval Factory (repo2bench + Harbor) — Turn Git Activity into Harbor Benchmark Tasks
We built repo2bench to turn a repository's Git history into runnable Harbor benchmark tasks—the standard sandboxed container evaluation framework for coding agents. In this post we run it on pydantic/pydantic-ai and marimo-team/marimo, and you can run the exact same CLI on your own private repo:
- Filter Git Signals: repo2bench scans merged PRs, keeps commits that touch both source code and unit tests, and drops noisy commits (git revert, force-pushes, docs/typos, or giant refactors).
- Verify the Test Signal (Fail → Pass): Every candidate task is tested in a clean container: the PR's unit test must fail before the fix (score = 0.0) and must pass after the developer's fix (score = 1.0).
- Package into a Harbor Task (Zero Solution Leakage): Each verified PR is packaged into a standard Harbor task folder (task.toml, environment/Dockerfile, instruction.md, and tests/test.sh) split into a Starting Workspace, a Goal-Oriented Prompt (instruction.md) that strips file paths and commit-message solution spoilers so the prompt reads as if written before the commit existed, and a Hidden Test Grader:
One command builds your entire dataset:
repo2bench mine --repo /path/to/your-repo --limit-num-tasks 20 --output ./datasets/my-repo-bench
Step 3: Run the Benchmark on Your Codebase — Watch the Winner Change
We mined 30 verified October 2026 tasks (repo2bench-v2.0, split chronologically into 10 Train PRs and 20 Held-Out Test PRs across pydantic/pydantic-ai and marimo-team/marimo), ran all 8 baseline Gemini configurations blind on the 20 Held-Out Test tasks (160 test trials), and compared the ranking against the public benchmark:
Decision Flip: Pi + Argon 4 (med) Jumps from #6 on the Public Leaderboard to #1 on Your Codebase (70% Pass@1 at \$0.22/task)
Step 4: Close the Loop with Harness² — Upgrade 3.8 Flash from 50% to 70% (75% Reward)
Why did stock Antigravity + 3.8 Flash (med) stall at 50% (50.1 turns, \$0.38/task) on the Held-Out Test set? Trajectory logs revealed a classic harness mismatch: without repo-specific localization and verification rules, 3.8 Flash spent 25+ turns searching multi-module directories for virtualenvs and running full-monorepo pytest suites that flooded its context window.
Instead of abandoning a fast, low-cost model, we plugged repo2bench into Harness² (google-research/harness2)—Google Research's framework for Recursive Agent Harnessing (published October 2026).
What Is Harness²?
Instead of retraining model weights, Harness² keeps your Gemini model frozen and automatically upgrades the agent's harness—recursively refining its system prompt (GEMINI.md), guardrails, repo memory, tool rules, subagents, skills (.agents/skills/), and verification scripts based on what passes your unit tests:
How repo2bench + Harness² Work Together on Your Repo
To prevent data leakage, repo2bench splits mined PRs chronologically so Harness² optimizes your harness only on older Train PRs (10 tasks, where Harness² lifted Pass@1 to 90%) and evaluates blind on unseen Held-Out Test PRs (20 tasks):
Live Empirical Breakthrough on Unseen Held-Out Test PRs: 50.0% → 70.0% Pass@1 (75.0% Reward, #1 Overall)
Using AntigravitySubstrate, Harness² evolved a 3-turn multi-module localization workflow in GEMINI.md, a targeted .agents/skills/pytest-loop/SKILL.md skill, and a lightweight scripts/verify_diff.py helper on the 10 Train tasks. Evaluated blind across all 20 unseen repo2bench-v2.0-test tasks with frozen Gemini 3.8 Flash (medium) weights:
Summary: Build Your Own Software Factory
- Benchmark the Full Agent (Harness × Model × Thinking): Your harness and thinking budget change cost by 15\times and quality by 3\times.
- Mine Your Own Repo (repo2bench): Turn your team's merged PRs into verified, container-isolated Harbor tasks in one command—revealing the #6 → #1 ranking flips that public leaderboards hide.
- Continuously Upgrade Your Harness (Harness²): Evolve repo-specific GEMINI.md rules and skills (AntigravitySubstrate) on historical Train PRs to lift Gemini 3.8 Flash (med) from 50% to 70% Pass@1 (75.0% reward, #1 overall) on unseen Held-Out Test PRs while cutting cost (-18%) and execution time (-20%).
Disclaimer: This code is provided “as-is” as a demonstration only to illustrate a potential solution. The code does not constitute a Google product or service of any kind, and Google offers no support, warranties, or liability of any kind with its regard. Whoever chooses to use this code accepts all responsibility related to it, including for its implementation, use, and ongoing maintenance. For the avoidance of doubt, this code is not eligible for the Google Open Source Software Vulnerability Rewards Program.
Build your own custom code benchmark was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/build-your-own-custom-code-benchmark-12b2673f1906?source=rss—-e52cf94d98af—4
About the Author
