Two ADK agents, two Gemini Flash models, and what the checks written in code caught before an LLM judge had anything to score.
The plan was a model comparison. Gemini 3.8 Flash came out in September, and in ADK switching models means changing one string.
I wanted numbers before changing that string on two agents that produce structured output, because “the new model is better” is a claim, and I wanted to see it measured on our own cases.
The first numbers that came back weren’t about the model, they were about our schema.
This is what happened, what the evals caught, and the mistakes in how we counted results that almost hid it.
The experiment is small and anonymised, and it runs on synthetic data only, so read the numbers as a case report, not a benchmark.
The setup
Two agents built with ADK Python 2.7.1 (Python version of Google ADK, on Python 3.13).
Both call Gemini models on Gemini Enterprise Agent Platform and use a Pydantic model as output_schema:
- Agent A generates workflow definitions: a graph of steps with a trigger, human tasks, decisions and an end.
- Agent B makes decisions on document cases: what kind of document it is, whether it’s legible, whether the case is complete, and which fields to extract.
The eval sets are small:
– Agent A has 13 cases, two of them multi-turn conversations.
– Agent B has 12 single-turn cases.
Every case runs three times per model, because the agents run at temperature 0.1–0.2 and one run per case says little.
For Agent B the expected answers aren’t written by hand, the codebase already had deterministic rules for these decisions, so we used them as the oracle and compare the agent’s output field by field, with partial credit.
Both suites also carry canaries: fake tokens and fake secrets planted in the input: a check fails if any of them shows up in the output.
All documents and cases are synthetic, no real customer data goes anywhere near the eval sets.
Where we started
It’s worth being direct about the starting point, because I suspect it’s common.
There was an eval step in CI, and it was green.
It checked that the eval JSON files were valid JSON, it measured nothing about the agents.
The real adk eval run couldn't have worked in CI anyway: the pipeline didn't install the google-adk[eval] extra, and the agent wasn't exported in the way the eval runner expects.
There was an LLM judge with a rubric, on two cases, never compared with human judgments, there was no baseline run to compare a new model against and one of the two agents had no evals at all.
None of this looked broken from the outside, because the one check that ran was green.
Checks in code first
Goran Minov already covered how to layer graders in Mastering Agent Evaluation: code first, an LLM judge after, and a judge you’ve tested before you trust its scores. With structured output, a lot of it can be checked without a model:
- whether the answer parses;
- whether a workflow graph has a reachable end;
- whether a planted secret came back in the output.
ADK Python has supported custom metrics since 1.18.0.
A custom metric is a Python function that receives the actual (and optionally the expected) invocations and returns an EvaluationResult.
This is the first one we wrote, simplified; the class name and module are placeholders, not the real schema:
import statistics
from typing import Optional
from pydantic import ValidationError
from google.adk.evaluation.conversation_scenarios import ConversationScenario
from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric, EvalStatus
from google.adk.evaluation.evaluator import EvaluationResult, PerInvocationResult
from my_agent.schemas import WorkflowDefinition # the agent's output_schema
def output_parses_against_schema(
eval_metric: EvalMetric,
actual_invocations: list[Invocation],
expected_invocations: Optional[list[Invocation]],
conversation_scenario: Optional[ConversationScenario],
) -> EvaluationResult:
"""1.0 if the final response validates against the output schema, else 0.0."""
results = []
for inv in actual_invocations:
parts = inv.final_response.parts if inv.final_response else []
text = "".join(p.text or "" for p in parts)
try:
WorkflowDefinition.model_validate_json(text)
score = 1.0
except ValidationError:
score = 0.0
results.append(
PerInvocationResult(
actual_invocation=inv,
score=score,
eval_status=EvalStatus.PASSED if score else EvalStatus.FAILED,
)
)
if not results:
return EvaluationResult(overall_score=0.0, overall_eval_status=EvalStatus.FAILED)
average = statistics.mean(r.score for r in results)
passed = average >= eval_metric.criterion.threshold
return EvaluationResult(
overall_score=average,
overall_eval_status=EvalStatus.PASSED if passed else EvalStatus.FAILED,
per_invocation_results=results,
)
The metric is declared in the eval config next to its threshold, and adk eval picks it up through –config_file_path:
{
"criteria": {
"output_parses_against_schema": { "threshold": 1.0 }
},
"custom_metrics": {
"output_parses_against_schema": {
"code_config": {
"name": "my_agent.metrics.output_parses_against_schema"
}
}
}
}
On top of the parse check, Agent A has structural checks on the generated graph:
- the existing workflow validator in the codebase returns no errors;
- human supervision is declared, with at least one human task assigned to a role;
- every step is reachable from the trigger;
- at least one end step is reachable.
Reachability is a few lines of graph traversal, again on a simplified shape:
def unreachable_steps(wf: WorkflowDefinition) -> set[str]:
"""Step ids that can't be reached from the trigger."""
edges: dict[str, list[str]] = {}
for t in wf.transitions:
edges.setdefault(t.source, []).append(t.target)
seen, stack = set(), [wf.trigger_step_id]
while stack:
step = stack.pop()
if step not in seen:
seen.add(step)
stack.extend(edges.get(step, []))
return {s.id for s in wf.steps} - seen
Two more invariants are on the list and not written yet: every branch has an exit, and there are no unintended cycles.
The metrics have their own tests that run without a model and a known-good response must score 1.0 and a broken one must score 0.0.
Those tests run in CI on every change.
One caveat if you go the pytest route: an open ADK issue reports that custom metrics declared in the config aren’t registered when evals run through AgentEvaluator.evaluate, so we run them through the adk eval CLI.
39 out of 39
The first full run of Agent A failed every case run on both models: 39 out of 39 on Gemini 3.5 Flash, 39 out of 39 on Gemini 3.8 Flash. The LLM judge had no scores to give, because its runs failed the same way.
The model never got to answer. Every request came back with the same error:
400 INVALID_ARGUMENT: Unable to submit request because one or more
response schemas didn't specify the schema type field.
The output schema contained free-form fields, the kind you add when the shape “isn’t settled yet”. In Pydantic they look harmless:
class Step(BaseModel):
id: str
kind: str
settings: dict[str, Any] # no declared shape
Python and Pydantic accept it locally. Once converted to the schema the model endpoint receives, a field typed as Any has no type at all, and Gemini Enterprise Agent Platform refused the whole request. We didn't run the same schema against the Gemini Developer API, so I can't say whether it behaves the same way.
The newer model hid the bug
Agent B had a milder version of the same problem: plain dict fields, with no Any and no declared properties. Agent Platform accepted that schema, and the results took longer to read correctly.
Over 36 case runs per model (12 cases, three runs each):
Gemini 3.5 Flash Gemini 3.8 Flash
Schema-valid output 30/36 (83%) 36/36 (100%)
Empty objects where content 23/27* 27/27
was needed
Whitespace until the token limit 7 0
Cases matching the oracle 12/36 (33%) 13/36 (36%)
Mean field-match score 0.40 0.40
No canary leaked 36/36 36/36
Seconds per run of 12 cases 222 38
* the other 4 runs never produced readable JSON
Read the first line alone and the upgrade looks like a fix: from 83% to 100% valid output, and six times faster.
The cause, as the fix later confirmed, is the same untyped field in both columns. With constrained output and no declared shape, Gemini 3.5 Flash produced whitespace until it hit the token limit in 7 of 36 runs; 6 of those never became valid JSON, and the long runs explain most of the time difference. Gemini 3.8 Flash never looped. It returned empty objects, {}, instead. Valid JSON, valid against the schema, and nearly useless.
Three of Agent B's tasks need content in those fields (classification, legibility and extraction, 27 runs per model): 3.8 Flash returned empty objects in all 27, 3.5 Flash in 23.
The line that matters most is the field-match score against the oracle: 0.40 for both models.
Had we stopped at the parse check, we would have logged the upgrade as progress and moved on.
The old model failed in a way you notice: long, slow, invalid output.
The new one failed fast, about 38 seconds for all 12 cases, with valid JSON and nothing inside it.
Per case, the split follows the schema.
The six classification and extraction cases, the tasks whose answer has to live inside the untyped fields, failed in all 18 of their runs on both models. The three completeness cases, scored only on typed fields, passed in 7 of 9 runs on 3.5 Flash and in all 9 on 3.8 Flash.
The remaining misses were about file quality, which the schema doesn’t touch: for example, whether a file in an unsupported format counts as legible.
Count the runs that produce nothing
Our first results summary was wrong, and I doubt we’re the first to get it wrong this way.
When an invocation fails before the agent responds, ADK has no response to score, so no metric is computed for that case.
In our first summary the case simply disappeared, the averages were calculated over the cases that worked, and they looked better than the run.
The fix goes in the summary step: count failed executions as a metric of their own, next to the scores, with the unit stated. “Schema valid: 100%” means little without “out of how many runs that produced output”.
Pin the model, and other plumbing
The default model name in the project was a “latest” style alias, the kind meant for the Gemini Developer API.
On Agent Platform it doesn’t exist, and every call returned a 404.
An alias also changes under you, which is the opposite of what you want when the point is to compare two models on the same cases. Both suites now pin an explicit model version.
Keep the eval dependencies apart from the agent’s. With ADK 2.7.1, the google-adk[eval] extra can't be resolved together with protobuf 7, which the application uses, so adding the extra to the project would have downgraded protobuf in production as well. The evals now pull it in only for the eval command, with uv run –with "google-adk[eval]" adk eval …, and the production dependencies stay as they were.
And put a timeout on every call to an external evaluator. One of our judge runs hung for about an hour on a call to the evaluation service, because the runner had no time limit and nothing else noticed.
After the fix
Each agent now gives the model a separate output schema with typed fields only.
The free-form configuration travels as a JSON string inside a typed field, and the code converts the result into the domain model afterwards.
A contract test in CI fails the build if any schema contains a node without a type.
Little else changed… Agent B’s instruction is identical before and after, and Agent A’s changes only to name the two renamed fields. Cases and models are unchanged, with three runs each, and every number below is recomputed from the raw results with the same counting rules.
Agent A, 39 case runs per model (45 invocations, because two cases have two turns):
3.5 Flash 3.8 Flash
before after before after
Runs with no output 39/39 0/39 39/39 0/39
Schema-valid output 0/39 36/39 0/39 38/39
All structural checks pass 0/39 35/39 0/39 38/39
Runs with a canary, no leak – 6/9 – 9/9
Agent B, 36 case runs per model:
3.5 Flash 3.8 Flash
before after before after
Schema-valid output (strict) 30/36 34/36 36/36 36/36
Cases matching the oracle 12/36 25/36 13/36 34/36
Mean field-match score 0.40 0.88 0.40 0.99
Empty objects where content 23/27* 0/27 27/27 0/27
was needed
No canary leaked 36/36 36/36 36/36 36/3
* the other 4 runs never produced readable JSON
The LLM judge, two cases with three runs each, now has something to score: all 6 runs pass the quality rubric on both models, with a mean of 0.94 against a 0.8 threshold.
With the schema fixed, the two models finally separate. Before the fix, the two models scored the same 0.40 against the oracle, because both were filling the same empty fields. After it, Gemini 3.8 Flash reaches 0.99 and 34 of 36 matching cases, against 0.88 and 25 of 36 for Gemini 3.5 Flash.
On Agent A, 3.8 Flash passes every structural check in 38 of 39 runs; its one failure is a truncated answer on the longest case. Agent B also ran in about 40 seconds per run of 12 cases on 3.8 Flash, against roughly three to four minutes on 3.5 Flash.
That answers the question we started with, for these cases: Gemini 3.8 Flash is better on both agents.
Most of the Agent B cases that still fail come down to one thing. In extraction, both models leave out a reference line for each source document. The deterministic rules expect that line, and the agent’s instruction never asks for it.
Once the agents produced real output, the evals had real output to fail on, and three things came up that the broken schema had been hiding.
One Agent A run on Gemini 3.5 Flash returned a definition that passed the schema check with no steps, no transitions and no roles.
All four structural checks failed on it: our schema fixed the shape of the answer, and nothing in it required the answer to contain anything.
The canaries caught a real leak. In one multi-turn case, the fake secrets arrive in the context of the second turn. In all three runs on Gemini 3.5 Flash, the agent copied them into the workflow definition it generated, as a variable’s default value or inside a step’s configuration.
It treated them as parameters to wire in. In a real system that is a secret saved inside a process definition, where anyone who can read the process can read it. The two cases with canaries in the first turn had no leaks in any run, and Gemini 3.8 Flash didn’t leak in any case.
And our own schema metric turned out to be too lenient. An independent recount of the raw results couldn’t parse two Agent B answers that our schema metric had marked as valid. Both were truncated JSON, about a thousand characters long.
The metric reused the tolerant parser from the backend, which, when the whole text isn’t JSON, takes the first complete object it finds inside. Every field in that schema has a default, so a fragment from the middle of a truncated answer validated as a full response. The metric is strict now: strip code fences, then the complete text of the response must be one JSON object that conforms to the schema. The tolerant parser is still in the production backend, and that is now a bug report of its own.
The same recount found one more counting problem.
The safety metric on the LLM-judge suite never ran: the evaluator in our ADK version was incompatible with the version of the evaluation SDK installed next to it, so the metric stayed “not evaluated”. Our summary had counted that as a failure. Metrics that weren’t evaluated are now left out of the denominators and reported on their own line.
What this does and doesn’t show
- Small sample. 13 and 12 cases, three runs each. Enough to find a schema bug, not enough to rank models.
- Synthetic data. Real documents are messier. Synthetic sets can be shared and rerun without exposing anyone’s data, but they don’t cover everything.
- Cost not measured. We tracked time per run, not spend.
- The LLM judge is still unvalidated. It runs on two cases, and we haven’t compared it with human judgments yet. Its safety metric couldn’t run at all because of the version mismatch.
- Truncated answers, cause unconfirmed. Seven truncated answers across the “after” runs (five on Agent A, two on Agent B) point to the output token budget, but the reason generation stopped isn’t saved, so that stays a hypothesis.
- Two structural invariants are still missing, as listed above.
On agents with structured output, the checks that run without a model found the most important problem.
We wanted to know whether the new model was better but first we had to make sure the model was being given a schema it could fill.
A New Model Is Not an Upgrade Until Your Evals Say So was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/a-new-model-is-not-an-upgrade-until-your-evals-say-so-3bdd72d1d97e?source=rss—-e52cf94d98af—4
About the Author
