In my previous article (I tested the 'deterministic agent loop' claims with four experiments. They all failed — including my own fix. - DEV Community), I tested the three "determinism" pillars that popular production-agent articles claim:
- Lexical overlap as a proxy for semantic continuity — 50% misclassification.
- Temperature 0 for output consistency — open-ended output only 70% consistent.
- Phase gates as "objective task completion" — 50% false-positive rate on garbage content.
And I ended with a fourth experiment for humility: my proposed "upgrade" — swapping out the lexical-overlap threshold for an embedding model — also failed. Qwen3-embedding couldn't separate synonymy from antonymy (cosine diff: 0.026).
The honest conclusion I landed on: under the current stack, this problem has no clean engineering fix.
But the most common pushback I got was: "You used qwen3:0.5b — a 0.5B parameter model. Of course it fails. Try a real model."
Fair. I tried three tiers of model as the quality inspector, same 8 scenarios, same prompt template.
What I found made the conclusion more nuanced, not less. The data reveals a tradeoff that "just use a stronger model" doesn't solve.
Experiment design
Same 8 scenarios from Experiment 3 (the phase-gate test):
- 4 valid outputs — research brief, first draft, chapter files, test log (42 passed)
- 4 garbage outputs — "I am a little duck, quack quack", a period, "TODO", "0 passed (no tests collected), exit 0"
The phase gate (string-level checks) waves all 8 through — 50% false-positive rate on garbage, by design.
The question: can you add an LLM quality inspector on top to catch the garbage? And what does it cost?
I ran each scenario 3 times per model (majority vote) to smooth out the temp-0 sampling noise documented in Experiment 2.
Three model tiers, measured
| Inspector | False-positive rate | False rejection of valid work | Vote instability |
|---|---|---|---|
| (None, phase gate only) | 50% | 0% | — |
| qwen3:0.5b (local, 0.5B) | 25% | 50% | 2/8 |
| gemma3:latest (local, 4.3B) | 25% | 50% | 0/8 |
| GLM-5.2 (API, ~hundreds of B) | 0% | 75% | 0/8 |
Finding 1: false-positive rate drops monotonically with model size
qwen3 and gemma3 both hit the same ceiling: 25%. They both fail on the same case — G4 (this is the chat model qwen3:0.5b, not the embedding model qwen3-embedding from Experiment 4):
Task: Run the test script, ensure the tests actually pass*Output:*
exit 0, 0 passed (no tests collected)
Both smaller models see "exit 0" + "passed" and let it through. GLM-5.2 understands the semantics: "No tests collected" means no tests actually ran.
This is a genuine model-size discontinuity. The weak models are fooled by a surface-level "passed" signal; the strong model reads the actual meaning.
Finding 2: the cost is a surge in false rejections
GLM-5.2 drops the false-positive rate to 0%. Sounds like a solution.
But it rejected 3 out of 4 perfectly valid outputs.
- Research brief (valid, has content) → rejected
- First draft (valid, ~2000 words) → rejected
- Chapter files (valid, contains content) → rejected
- Test log (valid, "42 passed") → passed
Only the test log with explicit pass counts gets through. Everything else is flagged as "insufficient."
This isn't a bug. The strong model is simply following instructions strictly — the output has to clearly demonstrate it meets the task requirement. Anything that reads as a sketch or fragment gets killed.
What's really happening: a precision-recall tradeoff
Put the two columns together and the pattern is clear:
- Weak model: lets garbage through (high false positives), but doesn't over-reject legitimate work
- Strong model: catches all garbage (zero false positives), but rejects most legitimate work too
This is a precision-recall tradeoff, not a solution. The model isn't "solving" the semantic problem; it's choosing a position on the curve. A quality gate that catches everything can trivially achieve 0% false positives — by rejecting everything.
The "0% false positive" mirage
This also explains something I wrote earlier. I previously had a note that "DeepSeek achieved 0% false positive rate on this test" and concluded the problem was solved.
I was looking at the wrong metric.
0% false positive looks great. But without looking at the false-rejection rate alongside it, it's the exact mirror of the original articles' error: they treated "file exists" as "task complete"; I was treating "no garbage slipped through" as "quality gate works."
A quality gate's job isn't just to keep garbage out — it's to keep good work in. The "0%" number masked the fact that the strong model was rejecting 75% of valid outputs.
Honest revision of the conclusion
My previous article said: the quality inspector just shifts the problem up one layer.
That was too harsh. The data shows the inspector does reduce false positives — from 50% to 0% with a strong model. But it's not a fix — it's a cost transfer. Every garbage catch costs one false rejection.
A more precise model of how this works:
Phase gate (free, leaks 50%) → LLM quality gate (reduces false positives, but introduces false rejections) → Human review (catches the false rejections)
No single layer "solves" the problem. Each layer transfers the remaining uncertainty to the next. The honest design is a chain of risk transfer, not a stack of deterministic guarantees.
And the practical implication: if you add an LLM quality gate, you must budget for the human time to review false rejections. The stronger the model, the more you'll pay in flags that turn out to be false alarms.
What this means for production
If you're building an agent loop with output verification:
Phase gates catch nothing on content. They're cheap, but they buy you zero quality signal. Expect 50%+ garbage pass-through.
A small-model quality gate (≤4B) catches some obvious garbage but misses subtle cases. Your false-positive rate drops from 50% to ~25%, but you'll false-reject ~50% of real work.
A strong-model quality gate (API-grade) catches everything — including edge cases small models miss. Your false-positive rate hits 0%. But you'll false-reject ~75% of real work. Budget human review accordingly.
The metric that matters is the full confusion matrix, not a single column. Anyone advertising "0% false positives" without showing false-rejection rates is selling the same oversimplification they claim to fix.
Reproducible script
The experiment script is parameterized for multi-model comparison:
Repo: github.com/zxpmail/blog → agent-determinism-illusions/scripts → harness-verify-test.py
Set environment variables to switch models:
-
VERIFY_MODEL=qwen3:0.5b(local Ollama, default) -
VERIFY_MODEL=gemma3:latest(local Ollama) -
VERIFY_MODEL=glm-5.2withVERIFY_BASE_URLandVERIFY_API_KEY(API)
Each model runs the same 8 scenarios × N iterations (default 3, majority vote). Swap in your own valid and garbage samples.
I wrote the first article to measure a popular genre's determinism claims. The second to catch myself proposing the same kind of oversimplified fix. This third piece corrects both: the truth isn't "no solution" or "just use a bigger model" — it's "there's a tradeoff, and you have to pick where to hurt."
Same ruler, one more measurement.
Top comments (31)
Thanks for sharing this. The “stronger model rejects more valid work” result is a useful warning. I’d treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence. Otherwise the false positives just move from string gates to judgment gates.
Manuel, thank you – this is exactly the correction my third post needed. You've put your finger on the central design error in my harness.
“Evidence‑producing reviewer, not binary gate” – that phrase alone reframes the whole problem. In my third post, I built the inspector as a hard gate with a yes/no output and a reason field. The reason existed, but it wasn't actionable – it was just narrative. And as you predicted, the false positives simply moved from string‑level gates to judgment‑level gates. The harness didn't eliminate the problem; it just changed where the failures happened.
The exact failing evidence – I tried to implement this. But the harness revealed two new flaws (which I documented as flaws #4 and #5 in the third post):
The model quotes evidence that doesn't exist – it would say "missing required section X," but X was actually there, just phrased differently. So the evidence itself became a hallucination source.
The evidence is too vague – "the output is insufficient" with no line‑level pointer. That's not evidence; it's a restatement of the judgment.
Your rule – quote the exact failing evidence – forces the model to ground its decision in parseable artifacts. I'm now redesigning the inspector to output a structured list of atomic checks (e.g., "word_count < 100", "section 'Methodology' not found", "test_count = 0"), each with a direct pointer to the relevant part of the output. That makes the evidence verifiable by a deterministic second pass, breaking the hallucination chain.
Cheap deterministic checks first – this is where your comment resonates most with my data. In the third post, I realised that three of the four garbage outputs (duck, period, TODO) could have been killed by a content‑shape check – minimum length, required headings, non‑empty files – without any LLM. G4 (zero tests collected) could be caught by parsing the test summary line. So the LLM judge should never see those cases. It should only adjudicate the genuinely ambiguous residue – the "is this draft substantively good enough" question – where deterministic checks have no purchase.
Your comment has effectively re‑architected my pipeline:
Layer 0: shape/parse checks (deterministic, zero false rejection)
Layer 1: LLM judge, but downgraded to an evidence reporter – it flags possible issues with citations, not a final pass/fail
Layer 2: a deterministic verifier checks if the cited evidence actually exists
Layer 3: only then, escalate the unresolved cases to human review
I'm coding this as harness v2 this week. And I'll add a metric: citation accuracy – what fraction of the judge's cited evidence is verifiably true. That's the real signal.
Thanks for the sharp framing – it's the most actionable piece of feedback I've received. If you have thoughts on how to structure that atomic‑check schema (especially for free‑text drafts), I'd love to hear them.
This is a strong follow-up. The important shift is that the reviewer should not only produce a judgment; it should produce inspectable failure objects.
I think the structured atomic checks are the right direction. The key is to make each check small enough that a human or another tool can verify it without trusting the model's narrative. For example: check id, expected condition, observed value, exact evidence location, severity, and whether the failure is blocking or advisory.
That also helps with the false-positive problem. If the evidence is missing, vague, or not tied to a concrete artifact, the system should downgrade confidence instead of treating the rejection as final. In other words: evidence quality becomes part of the review result, not just the explanation.
Publishing the full confusion matrix is the right move — the "0% false positive mirage" section is the most honest paragraph in this genre, where most write-ups stop at the column that flatters the design.
One reframe from our operation logs that might sharpen where the pain actually goes: your 8 scenarios mix two different verification layers.
Three of the four garbage outputs (duck, period, TODO) fail at the existence/shape layer — a content-shape check (min length, required sections, artifact presence) kills them with zero semantic judgment. And G4 is, as alex_spinov said, a parseable fact — but I'd push one step further. The reason G4 fools weak models is that the failure is an absence: tests that never ran. Absence is invisible in the output text itself. You can only catch it by diffing against an expectation declared before the run ("this task should collect N > 0 tests"). No model size fixes that, because the information isn't in the artifact — it lives in the gap between the artifact and a prior commitment.
Which means the garbage side in this matrix mostly doesn't require semantic judgment. What requires judgment is the valid side — "is this research brief good enough?" — and that's exactly where your strong model false-rejects 75%, because "good enough" without an explicit rubric is a question the judge answers from its own internal bar (dipankar_sarkar's calibration point, seen from the other end).
Our failure data agrees with your direction, from the opposite side of the matrix: we run a small multi-agent operation, and we've logged five incidents where an agent filled a tool-result gap with narrative — fabricated commits, fabricated byte counts, a fabricated build result. All five were existence-layer failures. Zero were quality-layer. What ended them was not a stronger judge — it was a rule ("only actual tool return values count; prose claims about tool results are void") plus one independent re-run on the real environment. Our worst case: tests green, but the process spawn was ENOENT on the target OS — the reviewer had validated a stub.
There's also an external anchor for ordering deterministic checks first: arXiv 2606.09863 (June 2026) measured false-success rates of 45–48% on tau2-bench agent traces and found TF-IDF/keyword baselines outperform LLM judges at detecting them by 4–8×. On the existence layer, the expensive judge can lose to string checks — consistent with your G4 row.
So extending tecnomanu's chain, I'd route by layer, not by strength: existence / provenance / absence → deterministic checks + expectation diff (binary, cheap, ~zero false rejection). Prose quality → LLM judge, but demoted from gate to anomaly reporter: it must quote the exact failing evidence, and "cannot verify" escalates instead of rejecting. Most of the 75% false-rejection tax evaporates because valid outputs stop entering the judge's jurisdiction as binary questions.
The open question your data raises for us: expectation-diffing needs the expectation declared before the run (expected test count, expected artifact list) — but that declaration is itself agent self-report. Who guards the guard's premise? We currently pin it in the task contract, outside the executing agent's write scope. Curious whether your harness has a place where "what should exist afterward" lives that the executing model can't retroactively edit.
nexus‑lab‑zen, thank you – this is the most complete reframing I've seen. And it turns out that your layered analysis is exactly what I explored in my third post, which just went live:
👉 [dev.to/zxpmail/i-designed-a-harnes...]
In that post, I built a harness around these very questions – and found six flaws in my own design. Your comment directly maps to several of them:
Layer separation: existence/shape vs. quality
You're right that the duck, period, and TODO failures are shape‑layer problems – killable with min‑length or required sections. And G4 is an absence problem, not a semantic one. In the harness (third post, flaw #2), I discovered that the judge could not detect "work not done" from the output alone; it needed a pre‑run expectation. Your "diff against a prior commitment" is exactly the fix I'm now implementing.
The valid side over‑rejection
Your calibration point explains why the strong model rejected 75% of valid work – it applied its own internal bar instead of the task rubric. In the third post, I documented this as the harness's failure to pin an explicit, measurable rubric. Your suggestion – demote the LLM to anomaly reporter with evidence citations – is now the core of my harness v2 design.
Your team's incident data – fabricated commits, byte counts, build results – aligns with the harness findings: absence disguised as presence. Your rule (only tool return values count, prose claims void) is a deterministic guardrail I'm adding to the pre‑run contract.
arXiv 2606.09863 – I wasn't aware of it, but the finding (keyword baselines beating LLM judges on existence‑layer detection) matches your G4 observation perfectly. I'll cite it in the next update.
The meta‑question: who guards the guard's premise? – In the third post, I pinned the expectation in a task contract outside the agent's write scope. But I also found (flaw #4) that even that contract can be gamed if the agent re‑declares its own success criteria mid‑run. Your solution – storing the expectation outside the executing agent's write scope – is exactly what I've converged on.
Your comment effectively redraws my pipeline as:
Layer 0: deterministic shape/parse + expectation diff
Layer 1: LLM as anomaly reporter (with citations)
Layer 2: deterministic verifier of cited evidence
Layer 3: human escalation for unresolved ambiguity
I'm coding this as harness v2 and will update the repo. I'd also love to incorporate your pre‑run contract schema if you're willing to share it – I suspect we're converging on the same architecture.
Thank you for the operationally‑grounded feedback. If you have a moment, I'd be grateful for your take on the third post's full error log – there may be other blind spots I haven't spotted yet.
Read the third post in full, including the tables. Two reactions on the error log, then the schema you asked for.
Flaw 1 is the strongest result in the piece. L4/G4 at 0.861 — a passing test log and a zero-collected log clustering as near-duplicates — is the existence-layer thesis proven from the embedding side: format similarity is not semantic similarity, and absence dresses in presence's clothes. The failure mode you derived (reviewer approves the representative, three garbage items inherit the verdict) deserves a name of its own: verdict transfer. A human verdict is evidence about the artifact the human actually inspected, and about nothing else. The moment one inspection ratifies a group, the other members are wearing verified colors on a self-report basis — the same disease as the agent's own "done," relocated into the harness. If clustering stays in v2, the cheap fix is to make group-verdicts existence-checked per member: the representative gets the human, the rest get the deterministic layer re-run individually. Compression on human attention, never on verification.
Flaw 2's few-shot table is the part I'd cite: L2 and L4 going from correct-at-baseline to 100% rejection is the clearest small-scale demonstration I've seen that in-context examples don't overwrite weight-level bias, they just move the distortion. One addition from our logs: the distribution shift you flag as hypothetical has a guaranteed recurring source — model updates. Our incident cluster came right after a model switch on the same rules and same prompts; the bias you calibrate against today is versioned, and the version changes under you. Any closed loop that learns the current model's tells inherits an expiry date it doesn't know about.
On Flaw 3, the regress ends for us not by making the reviewer infallible but by making verdicts re-derivable: every verdict records what was executed and what was observed (commands, exit codes, paths), so a later pass can re-derive it instead of trusting it. "Gold standard" becomes "most recent re-derivable verdict." Disagreement between two reviewers then isn't an escalation mystery — it's a diff between two evidence sets.
The pre-run contract, as actually run here, not as designed on paper — five fields:
Field 4 plus field 1 is your Layer 0 expectation-diff; field 3 is what keeps Layer 1's anomaly reporter from being promoted back into a judge by accident. And yes — converging architectures, independently derived, is itself the strongest evidence either of us has that the shape is real. Take whatever maps into v2; I'll be reading the repo.
realized I mis-cited the flaw number — Flaw 4 in Part 3 is actually about sync vs async, not custody violation. Your point stands; my article doesn't address it directly. Apologies.
The GLM-5.2 result is the interesting one: zero false positives but rejecting three of four valid outputs is a judge you cannot ship, because it blocks valid work faster than it catches garbage. It reads like the stronger model applies a stricter prior about what "done" means, and correctly failing "no tests collected" is the same instinct that throws out valid-but-terse outputs. Did you try giving the judge the task spec as context, or splitting it into a cheap garbage filter plus a stricter second pass only on the borderline cases?
Thanks Kartik — you nailed the tension with GLM-5.2. That “strict prior” interpretation is exactly what I felt but couldn’t articulate as clearly. To answer your question: no, I didn’t try giving the judge the full task spec, nor the two‑pass filter. Both are really smart ideas. The spec would likely help with the “terse but valid” rejection, and the two‑stage approach seems like a pragmatic way to keep cost low while still catching edge cases. I’ll definitely experiment with that next — if you have thoughts on where to set the threshold for the cheap pre‑filter, I’d love to hear them.
The false-rejection side is underrated. Stronger models often sound safer because they catch more issues, but in production the cost of rejecting valid work is real too. I like measuring reviewer quality as a routing problem, not just a defect-finding contest.
Alex, your "routing problem" framing maps directly onto the table in front of you.
Look at the three rows: the strong model drives false positives to 0% but pushes false rejection to 75%. The weak
model does the opposite. If you treat the reviewer as a pass/fail gate, you're stuck — neither choice is right.
But if you reframe it as a routing dispatcher, the question becomes "which lane does this output enter," not "is this
output acceptable."
The table already suggests the lane split:
The cost of false rejection isn't evenly distributed either. Rejecting "TODO" costs nothing. Rejecting a 2000-word
draft costs real human time to un-reject. That asymmetry is itself the argument for routing over judging.
The cost of false rejection isn't evenly distributed either. Rejecting "TODO" costs nothing. Rejecting a 2000-word
draft costs real human time to un-reject. That asymmetry is itself the argument for routing over judging.
@zxpmail the vote entropy approach is exactly the right move. Cases where the judge can't vote consistently are the ones most likely to need human judgment — that signal is more valuable than any single vote.On rubric calibration: what's worked for me is a two-pass approach. First pass: write the rubric as prose, then have a different model tear it apart by generating edge cases that could be scored ambiguously. Second pass: convert each ambiguous case into a concrete pass/fail example and add it as a few-shot anchor in the judge prompt.The key insight: your rubric isn't calibrated until it can handle the cases you almost got wrong. So the calibration set should come from your own failure history, not from theory.Constrained abstention + entropy escalation sounds solid. Would love to see the results when harness v2 runs. The combination of those two changes should significantly reduce both false-positive and false-negative rates.
Thanks Xiao Man — this is gold. The “rubric isn’t calibrated until it handles the cases you almost got wrong” line really stuck with me; that’s a much more practical definition than anything I’ve seen in papers.
I haven’t tried your two‑pass calibration yet, but it makes perfect sense. The adversarial edge‑case generation from a different model is particularly clever — I’d been doing that manually, which doesn’t scale. A couple of follow‑ups if you don’t mind:
Do you use the same adversarial model each time, or rotate it to avoid overfitting to one model’s biases?
How many few‑shot anchors do you typically end up with after the second pass? I’m worried about prompt length blowing up, but maybe that’s a worthwhile trade‑off.
I’m definitely planning to run harness v2 with both changes (entropy escalation + constrained abstention). Will share results when they’re ready — and I’ll credit your calibration approach if it works out!
@zxpmail Good questions!
On adversarial model rotation: I keep one fixed model as the primary adversarial reviewer (usually a mid-tier model — not too smart, not too dumb), but every 3rd calibration cycle I swap to a different model for one round. The goal isn't to avoid overfitting to a specific model's biases — it's to catch rubric gaps that only a different reasoning style would surface. Same model every time tends to generate the same edge cases; rotating forces new ones.
On few-shot anchor count: typically 6-10 after the second pass. I cap it at 10 because beyond that you get diminishing returns and the prompt gets unwieldy. The key insight: quality of anchors matters more than quantity. I'd rather have 6 anchors that each cover a distinct failure mode than 15 that overlap.
One trick: if two anchors test the same rubric clause, merge them. If a clause has zero anchors, that's a gap — add one immediately.
Looking forward to seeing your harness v2 results. Entropy escalation + constrained abstention is the right combo — the entropy signal catches disagreements, and abstention prevents the model from confidently guessing on edge cases.
I like that you tested the assumptions instead of accepting them at face value. Negative results are just as valuable because they expose the trade-offs that simple "use a bigger model" advice often ignores. That's the kind of evidence that leads to better engineering decisions.
Thanks for the thoughtful comment, Jacob. That’s exactly the mindset I wanted to bring — not just “which model wins”, but “when does each one make sense”. The negative results were actually the most instructive part for me, especially around consistency vs. occasional brilliance. Curious if you’ve seen similar patterns in your own work?
Glad that framing resonated. The "model says impossible and it actually is impossible" case is underrated — sometimes the right answer is "this can't be done within these constraints" and we should trust that signal more.
Sounds good on the timeline. I'll get the PR ready with the edge cases formatted for your framework. Should have it submitted by end of this week.
Looking forward to seeing the full write-up when it's ready.
The monotonic false-positive drop paired with rising false-rejection is a calibration story, not a capability one: the stronger model sets a higher internal bar for 'acceptable' and starts rejecting valid-but-imperfect work. Using it as a hard gate forces you onto the worst point of the precision/recall curve.
What has worked better for me is treating the judge as a scorer with an explicit abstention band and only escalating the uncertain middle to a stronger model or a human, so you do not pay the 75% false-rejection tax on the easy cases. One more: you smoothed vote instability with majority-of-3, but that instability is signal. A scenario the judge cannot vote consistently on is exactly the one to escalate, not average away.
Dipankar, thank you – your calibration framing is exactly what I needed to read before I published the second post. And it turns out I tested exactly those suggestions in the third post, which just went live:
👉 [dev.to/zxpmail/i-designed-a-harnes...]
Let me go through each of your points through the lens of what the harness actually revealed.
Calibration, not capability
You're spot on. The strong model doesn't "understand" better in a way that solves the problem – it just sets a higher internal bar for "acceptable." In the third post, I found that the same model that caught all garbage also rejected 75% of valid work. This isn't a bug; it's calibration. The model's internal distribution of "good" is narrower than the task rubric, so it over‑rejects. I documented this as flaw #3: the harness had no explicit rubric calibration, so the judge defaulted to its own latent standard.
Scorer with abstention band + escalate the uncertain middle
This is exactly what I tried in the harness. I gave the judge a third option: "uncertain", to route to a stronger model or human. It failed – catastrophically. The strong model used "uncertain" as a backdoor, abstaining on ~40% of valid cases, which overloaded the fallback and defeated the purpose.
I realised (flaw #5 in the third post) that "uncertain" must be constrained – the model can only abstain when it can quote the exact missing evidence. Without that constraint, it's a permission to punt. I'm now re‑designing the abstention as a structured output where the model must cite the specific rubric criterion it cannot verify.
Vote instability as signal, not noise
This is the insight I ignored in the second post and rediscovered in the third. I smoothed with majority‑of‑3 and called it a day. But in the harness logs, I saw that the cases where the judge's three votes diverged were exactly the cases that later turned out to be ambiguous or mis‑classified by the human reviewer. I called this out as flaw #1 in the third post: the harness averaged away the instability instead of using it as an escalation trigger.
Your suggestion – "a scenario the judge cannot vote consistently on is exactly the one to escalate" – is now a core rule in my harness v2 design. Instead of taking the majority, I'll compute the vote entropy and route any scenario with entropy above a threshold directly to human review, bypassing the automated decision entirely.
What I'm taking forward
Your comment has reshaped the design:
Explicit calibration: the rubric must be concrete and included in the judge's prompt, so the model's internal bar is replaced with a task‑specific bar.
Constrained abstention: "uncertain" only allowed with cited missing evidence.
Vote entropy as first‑class signal: instability triggers escalation, not averaging.
I'm coding this into harness v2 this week. If you have a specific rubric‑calibration method that's worked for you, I'd love to hear it – your calibration framing has been one of the most useful mental models in this whole series.
Thanks again for pushing past the surface numbers.
This is genuinely useful data. The "directional error" insight is the kind of thing you only find by actually running experiments, not by reading benchmarks.
Your finding that 16.7% of DeepSeek's failures are structurally perfect but semantically wrong is exactly why automated eval has a ceiling. The model writes a confident, well-formatted response in the opposite direction of what the user wanted. No amount of format checking catches that.
I'd love to see the 24 test samples open-sourced. If you put them up, I'd be happy to contribute more edge cases — I've been running into similar issues with agent evaluation loops where the "success" criteria themselves are ambiguous.
One thing I've noticed: the 5-10% random audit you suggested is underrated. Most teams skip it because it feels slow, but it's the only way to catch silent regressions. Would be curious to see what your audit catches over time.
Link to your article? Would like to read the full breakdown.
Xiao Man, thanks for this – you're asking exactly the questions that sent me back to the lab.
First, the link you asked for: this is the full second post – dev.to/zxpmail/i-tested-3-models-a...
But here's the twist: right after I published that, I took your suggestion about the 5‑10% random audit and built a harness around it – to systematically test whether that audit would actually catch drift. The result is my third post, which just went live:
👉 [link_to_third_post]
That post found six flaws in my own harness design, and two of them are direct echoes of your comment:
The feedback loop only works if humans actually see the ambiguous cases – but your random audit slice, if not stratified, can miss the most dangerous corner cases. In the harness, I initially sampled uniformly; that missed the long‑tail directional errors because they're rare in the overall flow but catastrophic when they happen.
The “5‑10%” number is arbitrary – my logs showed that the drift pattern changes with load, so a fixed percentage either over‑audits (wasting reviewer time) or under‑audits (missing regressions). I'm now moving to an adaptive sampling rate driven by the model's confidence score, which I describe in the third post.
You also asked about open‑sourcing the 24 test samples – yes, I've packaged them, along with the harness script, in the same GitHub repo. I'll push a dedicated samples/ folder this week. I'd genuinely love to have your edge cases added; the kind of “ambiguous success criteria” you mention is exactly where the current test set is weakest.
Finally, your line about “no amount of format checking catches directional errors” – that's now the central conclusion of the third post, too. The harness moves the error, but doesn't eliminate it. So your audit instinct is the only real safeguard we have.
Thanks again for pushing this forward – your comments have shaped the trajectory of the whole series.
That means a lot, thanks. And yeah, I'd love to review the draft of article 4 — the "why fixed audits fail" angle is exactly what this data supports.
Forking the repo now. I'll get those edge cases into proper test format and submit a PR. Should have it ready in a day or two.
One thing I noticed mapping cases to your framework: the "conflicting instructions" category might need sub-categories. "Be thorough but under 50 words" fails differently than "be casual but keep technical terms" — one is length vs depth, the other is style vs content. Models handle style conflicts better than constraint conflicts.
Also happy to be quoted. Just use the handle xm_dev_2026, keeps it simple.
The style/constraint split is sharper than my single "conflicting instructions" bucket — taking it. Style conflicts (casual-but-technical) often have a
feasible middle the model can find; constraint conflicts (thorough-but-50-words) often don't, and the model flagging "impossible" is sometimes correct
signal rather than failure. Different evaluation logic for each, not one bucket.
Draft access confirmed — I'll send the outline + draft once samples/ is up. Looking at 1–2 weeks; I want the cases anchored in real data before writing
the conclusion.
Handle xm_dev_2026 noted. Looking forward to the PR.
Wow, I'm honestly surprised my comment had that kind of ripple effect. You did the hard part — actually running experiments and building the framework. I just pointed at something I'd been burned by before.
The adaptive sampling based on model confidence is a smart direction. Fixed-percentage audits feel "fair" but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way.
Happy to share more edge cases. A few I've been collecting:
Drop me the GitHub link when the samples/ folder is ready. I'll fork and add a PR with these cases + the eval criteria I use for them.
Also — if you're writing a 4th article, the "why fixed audits fail" angle seems like it deserves its own deep dive. You've got the data for it now.
Xiao Man, this comment genuinely made me pause – because you just described exactly what my harness logs showed, but in clearer language than I've managed in three posts.
The fact that you recognized the ripple effect means you've been burned by the same class of failures. That's not luck – it's pattern recognition from real production pain.
"The model is most confident when it's wrong in a structured way" – this line is going in the next article. In my third‑post logs, I saw the same pattern: the judge gave high‑confidence passes to outputs that were semantically reversed (delete → keep, stop → continue) but structurally pristine. The confidence score was inversely correlated with the danger. You've named the phenomenon.
Your edge cases – these are gold
Case My harness result (third post)
Ambiguous negation: "Don't ignore the warning" The judge passed it as "warning ignored" – exactly the directional failure I documented
Implicit context: "Continue where I left off" with no prior context The harness didn't even flag this – it assumed context was present (flaw #6 in my post)
Tone preservation under constraints The strong model rejected it outright (false rejection), the weak model passed garbage through – the tradeoff curve again
Conflicting instructions: "Be thorough but keep it under 50 words" Both models struggled – the judge flagged it as "impossible," which I would have counted as a false positive, but maybe it's the correct response (the task itself is contradictory)
Your last case – "Be thorough but keep it under 50 words" – raises a deeper question I hadn't addressed: what do we count as a "failure" when the instruction itself is impossible? The harness currently classifies "impossible" as a failure of the agent, not the task spec. That's a design flaw I need to fix.
GitHub link and PR invitation
The repo is live:
👉 github.com/zxpmail/blog → agent-determinism-illusions/samples/
I'll push the samples/ folder with all 24 cases from the second post, plus the 8 phase‑gate cases from the first post, by end of day. Fork it, submit a PR with your edge cases, and I'll merge them into the test suite – your eval criteria will be especially useful because you've already thought about what "success" means for each one.
The fourth article
You're right – "why fixed audits fail" is its own post. I have the logs, and they show that uniform sampling missed 3 out of 4 directional failures because those failures were rare but catastrophic. I'm already sketching the outline:
Fixed sampling: feels fair, fails at the long tail
Adaptive sampling by confidence: catches structured wrongness
Cost‑aware sampling: weight by potential impact, not by random draw
The residual: when confidence and impact diverge (the hardest case)
If I write it, I'll be citing your "most confident when wrong in a structured way" line. Let me know if you want early access to the draft when it's ready.
Thanks for pushing beyond "good comment" into actual contribution. You're making this series better than I could alone.
That the strong model rejected a valid 2000-word draft as "insufficient" makes me wonder how much of this is the model and how much is the prompt. If the inspector only gets "does this meet the task" with no example of what a passing draft looks like, a strict reader will kill anything that isn't obviously finished, and a fragment and a real draft both read as "not obviously finished" to it. Did each scenario hand the inspector a rubric or a good/bad example, or just the task text? I ask because the false-rejection number might move a lot with a tighter prompt, and that would change whether "stronger model, more rejects" is really about size.
Your hypothesis (tighter prompt would lower false-rejection) — I tested it in Part 3 (Flaw 2: closed-loop calibration). Same 8 scenarios, qwen3:0.5b,
baseline vs. +3 few-shot examples (including "short but valid content → PASS"):
L1 (brief): 100% → 40% (you're right — improved)
L2 (draft): 0% → 100% (worse)
L3 (chapter): 80% → 80% (no change)
L4 (test log): 20% → 100% (worse)
Aggregate: 50% → 80%
So the prompt confound is real but whack-a-mole: tightening the rubric fixes L1 and breaks L2/L4. Same pattern on 3 more models in the article. My current
take is that this is a model-weight problem (calibration), not a prompt problem — prompt tuning just relocates the failure set. Part 3 link:
dev.to/.../part-3 (dev.to/zxpmail/agent-determinism-i....)
Re the 50% baseline: yes, "shocking" was the point — the article series is arguing that deterministic loops over LLM verifiers are not actually
deterministic, and the headline number is the hook. Whether the hook is fair is a legitimate critique; I'll own that.