For the past few weeks, I’ve been running an automated GitHub Action (run-antigravity-cli) powered by Gemini 3.8 Flash to review pull requests in CI/CD. To sanity-check the bot’s feedback, I started pasting its review comments into Fable for a second opinion. That’s when I noticed a frustrating pattern. Roughly half of the issues flagged by our CI reviewer were flat-out wrong. Looking at Fable’s spot-on rebuttals alongside Gemini’s noisy comments, I jumped to the obvious conclusion: Fable is just fundamentally better at code review than Gemini 3.8 Flash.
Anatomy of a “Whack-a-Mole” Code Review
Across three back-to-back pull requests — PR #17, PR #18, and PR #19 — the CI reviewer kept trapping me in a repetitive loop where fixing one valid comment triggered a fresh wave of false alarms on the next push:
1. Repeating the Same False Alarms (PR #17): While hardening input sanitization, the bot repeatedly warned that passing –files $FILES unquoted in bash would break argument parsing if $FILES was empty, and worried that a boolean $DRY_RUN variable might be capitalized as "True" instead of "true". Neither was an actual bug as Python’s argparse handled the flag boundary cleanly, and GitHub Actions booleans always serialize as lowercase strings. Even after I replied “Invalid”, pushing a new commit caused the bot to post the exact same warnings again.
2. Misreading the Repo’s Own Purpose (PR #18): On a security fix for fork PR gating, the bot insisted twice that gh pr edit –remove-label would fail with a 403 Forbidden unless we granted issues: write permissions (in reality, pull-requests: write is completely sufficient). It also claimed my default branch was main when it was master, and warned me against checking out the PR’s head.sha in test.yml — missing the fact that test.yml is the repository’s own self-test workflow, so checking out master would mean CI only ever tested old code.
3. Arguing Against Empirical Runner Behavior (PR #19): When I wired up cm find … | tee -a scan_output.log followed by SCAN_EXIT_CODE=${PIPESTATUS[0]}, the bot insisted three separate times that GitHub Actions runs bash -e -o pipefail by default, claiming the pipeline would abort before PIPESTATUS[0] was ever reached. In reality, when shell: bash isn’t explicitly set on a step, GitHub runs /user/bin/bash -e {0} without pipefail, and the bot’s suggested one-click fix ( || SCAN_EXIT_CODE=${PIPESTATUS[0]} ) would have actually reintroduced the exit-code masking bug. Most maddeningly, after I added a comment in the YAML explaining this exact behavior, the bot left a new review comment claiming my explanation comment was wrong.
The Realization: I Had Blindfolded the Reviewer
When I kicked off PR #20 to stop the noise, Conductor helped me step back and diagnose why the failures were happening. That’s when it clicked: I wasn’t comparing Fable and Gemini 3.8 Flash on a level playing field.
When I asked Fable to evaluate a comment, it had access to the full files and surrounding logic. Meanwhile, my GitHub Action in review_engine.py was handicapping Gemini 3.8 Flash in two ways:
- Diff Myopia: I only sent the raw git diff — just the changed lines and three lines of surrounding text. The model couldn’t see how variables were initialized earlier in the file, what guard clauses already existed, or how the rest of the workflow was structured.
- Single-Shot “Find a Bug” Prompting: I gave the model a single pass with a prompt telling it to act like an elite security auditor and return a JSON list of flaws, with no confidence score, no severity threshold, and no chance to double-check its work. I learned that when you hand any capable LLM a tiny code snippet and tell it to find bugs, it will bend over backward to invent theoretical edge cases just to be helpful.
How Conductor Helped Me Fix the Architecture in PR #20
Using Conductor’s structured spec-and-plan workflow, I redesigned the review engine around four practical improvements:
1. Curing Diff Myopia with Full File Context ( context_resolver.py ): Conductor helped me built ContextResolver to automatically pull the full source file (for files under 1,000 lines) or a ±100-line window around every modified section. Instead of guessing what happens outside a 6-line diff hunk, the reviewer now sees the surrounding setup, guards, and comments.
2. A Two-Pass “Devil’s Advocate” Verifier ( review_engine.py ): Instead of posting first-pass brainstorms directly to GitHub, Pass 1 generates candidate findings. Pass 2 ( _run_verifier_pass ) acts as a skeptical second reviewer armed with the full file context from ContextResolver. Its sole job is to play devil’s advocate, asking whether each candidate is genuinely reachable and broken in practice, or just a pedantic nitpick on a safe shell expansion. Unverified findings are discarded before they ever touch the PR. If Pass 1 finds nothing, Pass 2 is skipped entirely.
3. Severity & Confidence Thresholding: Every candidate finding now requires a severity rating ( LOW to CRITICAL ) and a confidence score (1 to 5). By filtering out anything below HIGH severity and 4/5 confidence via _filter_threshold_comments, speculative “what-ifs” get cut out right.
4. Shifting Left with a Local Review CLI ( local_review.py ): To break the cycle of pushing commits just to see what CI will say, Conductor helped me add a standalone CLI ( python scripts/local_review.py –staged ) and a one-command Git pre-push hook installer ( –install-hook ). Now I can run the exact same two-pass review locally in seconds before pushing. Hooray!
The Takeaway
When an AI integration produces noisy results, it is easy to blame the model and assume another tool is simply smarter. What PR #17 through PR #20 have taught me is that pipeline architecture matters just as much as model choice. Once I stopped starving Gemini 3.8 Flash of surrounding file context and gave it a second pass to sanity-check its own claims, the false positives vanished — proving the flaw was not necessarily in the model, but in how I asked it to work.
It Wasn’t the Model, It Was the Harness: How I Fixed my AI Code Reviewer was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/it-wasnt-the-model-it-was-the-harness-how-i-fixed-my-ai-code-reviewer-86a9457eb431?source=rss—-e52cf94d98af—4
About the Author
