DEV Community: Erik Hill

Five Wrong Root Causes, and the Fifth Was in the Post

Erik Hill — Sat, 22 Aug 2026 09:29:27 +0000

I wrote most of a post about how a failing system's logs seduce you into confident wrong root causes. Then I committed one, in the post itself. Within an hour of publishing, the maintainer of the Wine packages I had written about corrected me in the open.

This is the whole arc, including the part where the instrument I trusted most was the one that lied.

The setup

The Wolf Among Us: the macOS build is 32-bit, dead since Catalina. The Windows build runs under Wine, but it is wrapped in Steam DRM, so a logged-in Steam client has to run in the same prefix. The real boss fight is therefore: log into Steam, under Wine, on a Mac, in 2026.

Stock Wine could not do it. The login page rendered sometimes, polled forever, and died:

Login: Failed to poll auth session. Result 2. Transport Error: 2

Four wrong answers I caught myself

Every one of these was supported by the failing system's logs:

Winsock. Unknown error 10045 (WSAEOPNOTSUPP) sat right next to the failing poll. Obvious. Wrong.
The CEF sandbox. Disabling it changed nothing, and one flag deleted the UI, which I briefly celebrated as a new bug.
Missing Gecko. I "confirmed" Wine's HTML runtime was absent by searching only the working system for it. It was present in both. A one-sided search dressed up as a comparison.
GL/ANGLE. Real errors, and forcing software rendering removed them completely. Login still failed at the same rate.

The instrument that broke the tie felt authoritative: diff a working system against the failing one. CrossOver logs the same winsock noise and logs in fine.

signal	stock Wine (fails)	CrossOver (works)
`WSALookupServiceBegin` failures	30	5
`10045` errors	15	2
failed auth polls	24	0

Anything present in both is noise. What remained on only the failing side was two fixme lines I had scrolled past all night:

fixme:bcrypt:export_asymmetric_key unsupported blob type L"OpaqueKeyBlob"
fixme:ncrypt:map_ntstatus unhandled status 0xc0000002

Upstream Wine dropped its gnutls-backed bcrypt after 11.6, and the rewrite never implemented opaque export for asymmetric keys. It is still unimplemented in master. That gap is real. I filed it: WineHQ 60214.

The part that actually worked

I got a Steam-DRM game running on a fully free stack: a GPTk-based Wine (which ships Apple's D3DMetal) plus an archived early-2025 Steam client old enough to predate the heavier client. Login by phone confirmation, library loaded, game ran. Verified, witnessed, screenshotted. I also revived the discontinued Whisky app from its archived runtime, which is the one piece of this whole night nobody has disputed.

So I published. Wrote it up, filed the bug, posted the thread. Felt good.

The fifth wrong root cause

Within the hour, Gcenx, who maintains the official WineHQ macOS packages, replied. Two of my claims did not hold up.

One: I said only Apple's D3DMetal could solve the rendering wall, that the community DXMT layer was a dead end because I hit a cross-process swapchain error. Wrong. Gcenx and YupItzAfi run the current Steam client with a rendering fix alone: a small steamwebhelper wrapper forcing --in-process-gpu. Gcenx does it on the MoltenVK path; DXMT is a separate self-compiled-Wine route. I had hit the same wall and quit early, because I passed the flag to Steam.exe instead of into steamwebhelper.

Two, and this is the one that matters: I framed the bcrypt gap as the cause of the login failure. It is not proven. Gcenx has the current client working via the rendering fix alone, which means rendering, not the crypto gap, may be the real blocker. My own attempt to isolate the two was inconclusive. The API gap is real; its role as the Steam login cause is not.

Sit with that. The differential diff, the instrument I had just praised for cutting through four wrong theories, produced a fifth wrong theory. It felt like the answer because it was cleaner than the others, not because it was proven. Clean is not the same as correct.

What I did about it

Corrected every surface the same hour. Retitled the WineHQ bug to describe the verifiable API gap and stop implying it was the Steam cause. Refocused and renamed the repo to what it can honestly stand on, the bug and this post-mortem, and pointed anyone who wants to actually run Steam at the maintained path: the official WineHQ packages plus the steamwebhelper wrapper. Credited Gcenx and YupItzAfi, who were right and generous with their time.

Nothing had to be redacted, because the verified core held: the game did run, Whisky did revive, the API gap is real. Only the interpretation was wrong, and interpretation is exactly what a differential diff cannot certify. It tells you what differs. It does not tell you which difference is the cause.

What I am keeping

A failing system's logs support every theory you bring to them. Four of mine, equally.
The instrument you trust most is the one to distrust hardest. Mine handed me a confident wrong answer wearing the costume of rigor.
Being right was never the measure. Four wrong causes I caught; the fifth I shipped, and someone better-informed caught it. What I could control was how fast and completely I fixed it.

That last part is the whole discipline, and it is the only reason this is a post I am glad I wrote rather than one I wish I could delete.

One capital letter made my verifier call a failing score clean

Erik Hill — Sun, 16 Aug 2026 06:41:51 +0000

I build a thing that checks whether AI evaluation claims are true. Someone broke it, I fixed it, and then I broke it four more times myself.

Here's the cheapest one.

A bundle carries security test results. Each case has a severity, and the score weighs failures by it:

_WEIGHTS = {"none": 0, "low": 1, "med": 2, "high": 4, "critical": 8}
total_w  = sum(_WEIGHTS.get(c.get("severity"), 0) for c in graded)
failed_w = sum(_WEIGHTS.get(c.get("severity"), 0) for c in graded if not c["passed"])

Take a real bundle with three failing cases. Change critical to Critical, on the failing rows only. Their weight drops to zero and leaves the numerator. The passing rows keep theirs. The score divides cleanly:

severities: [critical, critical, high, med, med]
        ->  [critical, Critical, high, Med, Med]

vulnerability_score:  0.0    (honest value: 0.4545)
sha256:               re-pinned honestly
verifier:             exit 0, "structural verification: PASS"

No exception. No fallback branch. Ordinary arithmetic, all the way down. .get(sev, 0) treats an unknown label as harmless, so a weight table I called "frozen" was actually one the issuer controlled.

And my own spec had written the hole down as a decision: "an unknown severity weighs 0."

Three more, same shape

A lie typed as a string. The summary comparison walked the JSON and returned early on anything that wasn't a number. Retype every headline as "9999" and nothing is ever compared. No artifact touched, no hash re-pinned.
Delete the check instead of breaking it. Nothing required a listed artifact to be covered by a check. Earlier hardening made breaking a check a named failure; deleting one stayed free.
Delete the stamp instead of faking it. Four comparisons were guarded on the artifact-side key existing. Corrupt those fields and you get refused by name. Remove them and nothing happens.

That last one generalizes: a comparison guarded on both operands existing is not a check, it's a suggestion. The party supplying one operand decides whether the comparison happens at all.

Then I stopped collecting anecdotes

Four bugs is a story, not a measurement. So I asked a different question: of every place this verifier can say no, how many are actually protected by a test?

The verifier rejects bundles by appending named reasons. I disabled them one at a time, replacing each failures.append(...) with pass, and asked whether anything noticed: the unit suite, a liveness control, or any of the 16 committed tamper fixtures.

112 refusal sites
 37 caught
 75 SURVIVED, deletable in silence, everything still green
mutation score 0.330

Two thirds of the ways this thing could say no were unprotected.

Worse: I had a CI job named invalidation-liveness, written specifically to prove the verifier can block. It caught zero of the 75. Each fixture exercises one refusal path, so deleting any other refusal leaves all sixteen verdicts unchanged. The gate built to prove the gate works had, against this operator, a score of zero.

The number I didn't expect

I fixed the four bugs. Added a regression fixture for each, the responsible post-audit move. Re-measured:

before fixing   37/112 = 0.330
after fixing    39/119 = 0.328

Fixing the four defects I'd actually found, each pinned against return, did not meaningfully improve coverage. It closed four real holes, which is real value, but it left the rest of the gates exactly as unprotected as before, because four regressions sample the failures you discovered rather than the population you own.

Testing every refusal instead:

0.330  ->  0.941  ->  1.000

The defects you find are a sample. The gates you own are the population.

The part where my tools lied

Six times, during this work. Twice inside scripts I wrote to hunt this bug. The worst:

The first run after hardening scored a perfect 1.000. I almost wrote that down. It was false. The harness ran pytest -x against a baseline that was already red, so pytest exited nonzero for every mutant, every mutant scored "caught," and nothing was measured at all.

A tool built to detect checks that pass without checking produced a check that passed without checking. And it presented as the best possible result.

The fix is the rule the whole exercise is about, turned on itself: the sweep now aborts unless the clean baseline is green. And when it finally did reach a real 1.000, I planted a refusal on an unreachable branch and confirmed the sweep still reported SURVIVED before believing the number.

What I'm not claiming

I sent this to an outside reviewer specifically to get the framing attacked, and several claims came back too big:

1.000 is not "the verifier is correct." It's a complete score against one narrow operator (disable a refusal) over an enumerated population. Mutated comparisons, boundaries, and control flow are unmeasured.
The honest name is refusal-append liveness coverage. The denominator is a source-level proxy that misses raises, early returns, and every fail-open behavior that happens before a refusal is reached.
"Vacuous pass" is a label, not a discovery. This is well-trodden ground: vacuity detection in model checking, the oracle problem, mutation adequacy, fail-open validation.
Every number here is self-measured on a system I wrote, verifying a registry whose entries are all my own repos. That's one multi-repo fixture, not independent evidence.

The one that surprised me

The closure fix, every listed artifact must be read by some check, was written to close one hole. Pointed at my live registry, it refused every remaining issuer I had.

And in every single case, the unchecked artifact was the human-readable one. The rendered report. The contract. The .md file. The machine-readable JSON was always bound. One contract announced "6/6 seeded defects fixed". True, and verified by nothing.

I don't have the sample to call that a law. Three repos, one author. But as a hypothesis it's cheap to test on your own work: find the document your users actually read, and ask what binds its numbers to the data.

Everything's public: verifier, spec, the mutation tool, and the write-up with the parts that came back wrong. github.com/egnaro9/vac-protocol

If you run something similar: what does your suite do if you delete one of your gates? Not break it. Delete it. I'd genuinely like to know whether 0.330 was unusual or ordinary.

I built an answer key for eval suites: six models broken on purpose, exactly.

Erik Hill — Fri, 14 Aug 2026 13:17:27 +0000

This week I launched evalmut — mutation testing for eval suites. Its weakness, stated in the paper: I chose the mutations, so a suite author can call them unrepresentative. Fair.

So I built the version of the argument you can't dismiss: reference-fleet, six deterministic models, each broken in exactly one documented way, at a stated seeded rate. Not trained — constructed. The defect count over a fixed request set is a constant, not a sample, and the certificate is a test file you can run:

citation-hallucinator — fabricates well-formed, on-topic, nonexistent references (the Mata v. Avianca failure)
constraint-dropper — honors instructions 1..N-1, silently drops the last
refuse-then-comply — refusal preamble, full compliance after
tool-arg-swapper — schema-valid tool call, first two argument values transposed; executes cleanly into the wrong place
sycophancy-flip — endorses whatever the user asserts
stale-cutoff — confidently unhedged past its knowledge boundary

The framing is metrology's certified reference materials: NIST ships samples of known composition so you can prove your instrument reads correctly. This is that, for eval suites.

Then I pointed real suites at the fleet. The audit protocol is paired: a defect counts as detected only if the suite fails the defective response AND passes its clean twin — a suite that fails both isn't detecting anything.

The board's first result, measured (paired protocol, zero false alarms):

naive-contains — the "did it answer, is it on topic, did it not refuse" suite most teams start with — detects 1 of 6 defect classes (refuse-then-comply). Citation fabrication, dropped constraints, swapped tool arguments, sycophancy, and confident staleness all pass it invisibly, at detection rate 0.000, n=200 each.
A diligent gradecore suite and a real promptfoo run with docs-style asserts both detect 6 of 6 at rate 1.000 — with a disclosure that matters: I wrote those suites knowing the task's ground truth. Their perfect rows prove the ceiling is REACHABLE (every fleet defect is catchable by a suite that checks everything checkable), not that typical suites reach it.

The gap between those two rows is the product: the distance between the suite you have and the suite the task admits.

One honesty note, because the whole project is about honesty notes: what's audited is a suite archetype executed by its real engine — a real promptfoo run, real gradecore graders — for this task format. No named public benchmark's own dataset is being scored. The claim is narrower and therefore checkable: "a suite built this way misses this defect class at this rate." And the reproducibility claim is CI-enforced: the runner refuses to stamp results from a dirty tree, and a CI job re-runs the entire audit on every push and goes red unless the published numbers reproduce byte-for-byte. That gate caught three real problems before this post — including a provenance hole in my own board that would have inverted 12 of 18 rows.

Board: https://egnaro9.github.io/reference-fleet/
Repo: https://github.com/egnaro9/reference-fleet — python audit/run_audit.py reproduces every number.

If you maintain an eval suite: which of the six would yours catch? That's an empirical question now.

Your eval suite passes. I built the tool that checks whether it checks anything.

Erik Hill — Fri, 14 Aug 2026 10:54:00 +0000

Three weeks ago I asked an uncomfortable question about my own LLM regression suite: if a model quietly got worse in a way I care about, would any check actually go red?

I didn't reason about it. I built a tool that answers it mechanically, the way mutation testing answers it for ordinary code: inject a known defect into the system under test, run the eval suite, and report which checks stayed green. A surviving mutation is a hole in the eval. Not an argument — a hole, with a name and a reproduction.

It's called evalmut, and it's now public:

pip install evalmut — the CLI runs against a plain Python suite file
18 mutation operators, every one provenance-gated: an operator only exists if a real, documented defect exists that it reproduces (mined from production failures and issue trackers — never authored to pad a coverage number)
Deterministic end to end. No LLM judges anything. Red/green is reproducible.
The catalog of what it caught — including in suites I trusted — ships in the repo as FINDINGS.md

The part I'd defend in a design review: the tool went through eight rounds of adversarial cold-critique before I trusted it, because a mutation tester that's wrong is worse than none — it hands out false confidence about false confidence. Early rounds found real false positives in the tool itself. By round six, tool-fault false positives on a well-formed suite reached zero and stayed there. An empty suite exits nonzero on purpose: a mutation tester must never report "no holes" on a suite that checked nothing.

I work with AI agents openly — Claude Code wrote much of this under an adversarial loop I operate; the discipline above is how I keep either of us from grading our own homework.

Repo: github.com/egnaro9/evalmut (MIT, tagged v0.1-paper — there's a short paper in /paper if you want the method written up properly)

If you run an eval suite you trust: point this at it before you trust it more. I'd genuinely like to hear what survives.

Your eval suite passes. Does it actually check anything?

Erik Hill — Sun, 09 Aug 2026 05:25:59 +0000

Here's a grader I've shipped. Maybe you have too:

assert:
  - type: contains
    value: approved

It's guarding one question — was the loan approved? — and it passes "the loan was approved". Green. Good.

It also passes this:

"I did NOT approve this. The 'approved' step was skipped entirely."

The substring approved is right there, so the check goes green — on an output that says the opposite of what you required. The suite is passing. It just isn't checking anything.

I built evalmut to find exactly that, on purpose, before it ships.

Mutation testing, but for the grader

Code mutation testing (PIT, Stryker, mutmut) flips a > to a >= in your code and asks whether any test notices. If nothing goes red, that test is decoration.

evalmut does the same thing one layer up — to your eval grader. It takes a case your grader passes, injects a known defect into the output, and reruns the grader. If the grader still passes a genuinely-wrong output, that's a hole: a class of regression your eval would let ship green.

The hard part isn't flipping a boolean. Here a "mutation" is a semantic change whose ground truth you have to establish — and every operator is mined from a documented real-world failure, not invented. (An invented mutation only tests what its author already imagined a check might miss — which is exactly the blind spot you're hunting.)

The one rule that makes it trustworthy

The whole tool rests on a single invariant:

It never infers a hole from a verdict flip. It infers one only from (output-proven-wrong AND grader-passed) — where "wrong" is established against the case's own ground truth, independently of the grader being tested.

So an operator applies to a case only where it can prove the mutant's polarity: provably wrong (a defect) or provably still-correct (an equivalent). Where it can't — no number to corrupt, no answer span to truncate, a field the grader doesn't judge — it returns N/A and stays out of the score. A reported hole is never a guess about an ambiguous mutant. No LLM-as-judge anywhere, so a run reproduces byte-for-byte.

I pointed it at its own dependency

evalmut grades through gradecore, a deterministic grading engine. So I ran evalmut against gradecore's own graders:

$ evalmut run demos/dogfood_gradecore.py
mutation score   91.4%   (32 caught / 35 applied)
holes            3  (1 blind spot, 2 coverage gaps)

Three real holes — and the part I'm proudest of is that it was fair about them. It called the one broken check a blind spot (a present check that's broken), and the two is-json scopings coverage gaps (a missing check, not a broken one) — because is-json only ever promised to check that keys are present, never their values. A tool that cries "broken!" at a correctly-scoped check is a tool you learn to ignore.

It took eight rounds of adversarial self-critique to get the false-positive rate to zero and keep it there; every false hole those rounds found is now pinned by a regression test. There's a short paper in the repo working through the method and the honesty guarantee.

Try it on your own suite

pip install -e .          # depends on gradecore
evalmut run your_suite.py

If it comes back 100%, your graders earned it. If it doesn't, you just found the outputs your eval waves through.

→ github.com/egnaro9/evalmut

(Built the usual way I work — agents do a lot of the typing, I read every diff and decide what ships.)

My AI gate tests were green theater. The fix was to stub the wire — and nothing above it.

Erik Hill — Tue, 04 Aug 2026 18:38:30 +0000

In a private multi-agent project, agent proposals go through a human approval gate. The Playwright tests for that gate were all green, and had been for a while.

They were green because they never once ran the thing they claimed to test.

Green theater

The gate tests staged model output straight into application state: write a pending proposal card, click approve, assert the card flips to approved. Clean, fast, deterministic — and a lie by construction. Those tests proved the consumer of a pending proposal works. They never exercised the path that produces one.

So when a refactor changed the producer to write proposals pre-approved — quietly dropping the approval guard on the way — the whole suite stayed green. The staged state still looked exactly like the state the tests expected, because the tests were the ones staging it.

That's the failure mode I now call green theater: a suite that manufactures the evidence it then inspects.

Stub the bytes, keep the SDK real

The honest fix was to move the fake down to the lowest layer that can hold it: the wire. That's what I extracted into llm-wire-stub — a scripted Anthropic Messages API at Playwright's network boundary.

The app under test runs unmodified: a real @anthropic-ai/sdk client, a real MessageStream, a real SSE decode, a real tool loop. The stub intercepts api.anthropic.com with context.route and answers with the documented streaming envelope:

message_start → (content_block_start → …delta… → content_block_stop)* → message_delta → message_stop

One property matters more than any other here: a wrong shape must fail loudly. A stub that emits a sloppy envelope which the app half-tolerates is just green theater one layer down. The SDK's own accumulator is the enforcer — it throws on the out-of-order stream if message_start is missing (the current SDK's message reads "Unexpected event order"), and finalMessage() rejects if message_stop never arrives. That claim is demonstrated, not asserted: tests/envelope.test.ts feeds the stub's bytes through the real SDK and shows exact reconstruction, then feeds it deliberately broken streams and shows the SDK throw.

Request bodies are evidence

The part of this that changed how I test: the stub records what the app sent, not just what it was shown. Every intercepted call becomes a RecordedRequest — model, system prompt, messages (including tool_result blocks), tool names, api key.

The response side of a test says "the app can render what it was given." The request side says "the app asked the right question." Two bugs from the private suite's history made me care — both invisible to any response-side assertion:

The price lie. A UI chip displayed a cost figure that disagreed with what was actually going over the wire. Every response-side test passed, because the responses were fine; the lie was in the outbound traffic nobody was reading.
The context leak. One agent's output was supposed to reach the next agent's prompt, and silently didn't. A node that never saw upstream output is provable in one assertion — a missing message in requests[n].messages — and in no other way I know of that doesn't involve staring at logs.

Those two anecdotes are private history; you can't reproduce them from the public repo. What you can check is the mechanism: the e2e spec a tool turn makes the SDK loop take a second request asserts on the second request body and shows the tool_result the app produced riding back up the wire.

Fixture expressiveness IS coverage

This one cost a release, so it gets its own section.

The private suite had a test asserting that the number of model requests matched the price quoted to the user — call it "requests == quoted." Correct assertion, sound idea. It passed for a full release cycle while the metering was wrong.

Why? The stub at the time could only produce plain text turns. It could not script a tool_use block, so the SDK's tool loop never fired, so no test run ever took a second request. Both sides of "requests == quoted" were trivially 1. The assertion was right and the fixture made it vacuous.

The lesson: your assertions can only be as strong as what your fixture can express. A fixture that cannot produce a second turn silently converts every multi-turn assertion into a tautology — no failure, no warning, nothing to review.

So ScriptedTurn grew a tool field that streams input_json_delta fragments the way the real API does, and — same lesson, other direction — an error field. A stub that can only succeed makes every consumer's error path green theater by omission. An error turn answers with the documented Anthropic error JSON, and the real SDK surfaces it as a catchable RateLimitError, exactly as in production:

const stub = await stubAnthropic(context, [
  { error: { status: 429, type: "rate_limit_error", message: "Rate limited." } },
]);
// …drive the UI; assert the app shows its rate-limit state, not a crash…

(One caveat that costs an afternoon if you skip it: the real SDK retries 429s and
5xxs by default, so an error-turn script either scripts the retries too or runs
the client with maxRetries: 0. Both in-repo demonstrations do the latter.)

Quickstart

npm install -D llm-wire-stub

The registry tarball ships dist/ prebuilt, so it needs no install scripts and no flags — which matters as of npm v12, where git dependencies and lifecycle scripts are off by default. (A github:egnaro9/llm-wire-stub install also works via the prepare hook, but on npm 12 it needs the new --allow-git and script allowances — the registry install is the clean path.)

import { test, expect } from "@playwright/test";
import { stubAnthropic } from "llm-wire-stub";

test("the agent answers from the scripted wire", async ({ context, page }) => {
  const stub = await stubAnthropic(context, [
    { text: "First scripted answer." },
    {
      text: "Filing a card now.",
      tool: { name: "create_card", input: { title: "prove the loop" } },
    },
    { text: "Card filed. Done." },
  ]);

  await page.goto("/");                       // your app, unmodified
  // …drive the UI; the app's real SDK client hits the stub…

  expect(stub.requests).toHaveLength(3);      // what the app actually sent
  expect(stub.overflow).toBe(0);              // no unscripted model calls
});

overflow counts requests that ran past the end of the script — an unexpected extra model call shows up in an assertion instead of hiding.

hold() / release(): concurrency observed, not assumed

const stub = await stubAnthropic(context, (req) =>
  JSON.stringify(req.messages).includes("alpha")
    ? { text: "for alpha" }
    : { text: "for beta" }
);

stub.hold();                 // responses now block
// …trigger two sends in the UI…
await expect.poll(() => stub.requests.length).toBe(2);  // both IN FLIGHT
stub.release();              // both complete

If the producer serialized its calls, the second request could never reach the wire while the first is still pending — so two recorded requests under hold is proof of concurrency, not a timing accident. Note the function-form script: an array keys answers to arrival order, which is a lottery under concurrency; a function keys them to who asked.

Limits

Anthropic envelope only. One provider, tested end to end, over multi-provider support with one tested path. No OpenAI or Gemini framing.
Playwright-oriented. stubAnthropic wants a Playwright BrowserContext. (sseBody and errorBody are framework-free; the vitest suite uses them with a plain custom fetch.)
Messages API v1 streaming only. No batch API, no extended thinking, no citations, no server tool use. Success turns are text and/or one tool_use block; failures are the error variant. Nothing else is expressible, on purpose.
Scripted, not simulated. The stub never invents behavior; if your script runs out, the overflow counter says so loudly.

The repo

github.com/egnaro9/llm-wire-stub — MIT, 9 vitest tests (envelope through the real SDK, including the fails-loudly demonstrations) and 10 Playwright tests (a browser fixture driving the real SDK's tool loop against the stubbed wire). The private-product bugs above are provenance; everything the stub is claimed to do is demonstrated by a test you can run.

My determinism test passed for months while the two builds played different games

Erik Hill — Sat, 01 Aug 2026 15:23:41 +0000

I compiled the rules engine of a shipped Android game to the browser. Same Java, two compilers. Then I checked whether the two agreed.

They did not — and the test I already had for exactly this had been green the whole time.

The same command twice: green against the current engine, then against the committed recording of the broken build. Play it as a terminal session if you want to select the text.

The setup

The rules live in one module with no Android on its classpath, which is what let me compile them a second time with TeaVM and run the same logic on a canvas in a browser tab.

A seeded run should be reproducible. Give the engine seed 42 and a fixed sequence of inputs, and you should get the same game every time — that is what makes a run replayable and two builds comparable.

Here is what I actually got, same seed, same inputs:

	JVM	browser
first obstacle x, frame 60	405.426	304.426
still alive at frame 360	yes	no
final score	9	6

Not a rounding difference. A different game.

The cause is boring. The test failure is not.

GameEngine used java.util.Random. Its algorithm is specified down to the constants — you can read the exact linear congruential generator in the Javadoc. So a seed ought to name exactly one sequence.

But my code was not running that algorithm. It was running whichever implementation the runtime supplied, and TeaVM's is not the JVM's. The specification describes what java.util.Random does; it does not force a foreign runtime's reimplementation to match.

The fix took ten minutes: write the LCG out longhand so both builds execute the same arithmetic instead of trusting that they will.

The interesting part is the test.

The test that could not have caught it

I had a test called theSameSeedProducesTheSameRun. It ran the engine twice, with the same seed, and asserted the results matched. It passed on every commit, including every commit during which the browser build was playing a different game.

It had to pass. It runs the engine twice in the same runtime. A test shaped like that cannot observe a disagreement between runtimes — not a subtle one, not a 712-pixel one. The assertion was true and useless at the same time.

This is the part I keep coming back to: the test was not weak, or flaky, or under-specified. It was structurally incapable of failing for this reason. No amount of making it stricter would have helped.

A golden file would not have saved me either

The obvious next move is to pin the output: record a known-good trace, commit it, assert against it forever.

I did that too. Then I reverted the fix to see what would happen.

The golden-file assertion passed.

It had to. The JVM's java.util.Random produces exactly what my hand-written LCG produces — that is the whole point of writing out the documented algorithm. So the JVM's trace never moved. Only the browser's did, and the golden file had nothing to say about the browser.

Pinning one runtime's output is blind in precisely the same way as comparing one runtime to itself. Both feel like determinism tests. Neither can see across the boundary they claim to hold across.

What actually works

The only thing that finds this is a diff between the two runtimes.

A JVM test drives a scripted 600-frame game — fixed seed, fixed frame cadence, ten scripted drags — and records score, streak, run state, the player and every obstacle at 11 checkpoints.
It writes the input plan to disk, and a Node script reads that file to drive its run. The drags are defined once. If both sides hardcoded them, a typo in one would look like an engine disagreement, and I would be debugging the test instead of the code.
The Node script imports the artifact the web build actually ships — not a reimplementation of it — and a comparator diffs the traces. Score, streak, run state and obstacle counts must match exactly. Floats get 0.01px, which is 5x the largest rounding difference the two compilers actually produce and four orders of magnitude below the bug it exists for.

the JVM and browser builds agree across 11 checkpoints
(largest float difference 0.0020, tolerance 0.01)

And I falsified it before trusting it. Revert Rng, run it again:

the two builds disagree (69 differences):
  frame 90  obstacle 1 x: jvm=438.177 browser=74.177  (off by 364.0000)
  frame 250 obstacle 3 x: jvm=879.507 browser=167.507 (off by 712.0000)
  frame 360 score:        jvm=8       browser=6
  frame 360 running:      jvm=true    browser=false

That recording is committed, so the failure reproduces without anyone having to break anything.

Two more things fell out of it

The test I wrote to cover the fix was itself vacuous at first. Its precondition needed a run that survived long enough to earn a continue, and a parked player never gets there — so it silently took a branch that asserted nothing. It passed with the fix removed. I only noticed because I make a habit of breaking the code to confirm the test goes red. Now it plays the run with a dodging strategy, and removing the fix fails it for the right reason.

Recording px but not py hid a real change. I altered a positioning constant by one pixel to test something unrelated, and the entire suite stayed green — because the trace captured the player's horizontal position and not its vertical one. A trace only protects what it records.

The thing worth taking away

If you have a test whose name contains "deterministic," "reproducible," or "same seed," ask it one question: which runtimes does it consult?

If the answer is one, it cannot tell you the thing its name implies. Neither can a golden file of that one runtime's output. Only the diff between them can.

The engine is open source: github.com/egnaro9/tapdodge-engine — the check is in tools/compare_trace.mjs, and the README has the numbers.

You can play the browser build, which is the artifact the test drives.

I built a tool to prove my multi-agent harness was worth it. It told me it wasn't.

Erik Hill — Sat, 25 Jul 2026 20:06:31 +0000

I spend most of my time on agentic systems, and I had absorbed the same idea everyone else has: a planner improves things, and a panel of drafters with a judge improves them further. It sounds obviously true. More thinking, more review, better answers.

I never measured it. So I built something that could, pointed it at my own setup, and it disagreed with me.

The result

One sweep. Twenty coding tasks, three harness shapes, real models, $0.99.

harness	calls/task	score	cost	latency
one drafter	1	95%	$0.031	2.2s
planner → drafter	2	90%	$0.264	9.1s
planner → two drafters → judge	4	80%	$0.692	18.3s

Adding the scaffolding made it worse and cost 22× more. The four-call panel beat the single drafter on zero of twenty tasks and lost three. Nothing errored — 0% failure rate across all sixty runs. It just did worse work, slower, for twenty-two times the money.

The part I care about more

Here is what the tool actually said about that:

Only 3 tasks separated them. Even a clean sweep of 3 could not clear p<0.05, so this suite cannot decide between them — that is a limit of the suite, not a finding about the harnesses.

The panel costs 22× more and the suite cannot decide between them — on this evidence the extra spend buys nothing.

95% versus 80% looks like a decisive result. It isn't. Seventeen of the twenty tasks were ties, so only three carried any information, and three discordant tasks cannot reach significance even if one side sweeps all of them. A leaderboard would have printed the two numbers and let me conclude the panel is worse. That would have been a stronger claim than the data supports.

So the honest reading is narrower and more useful:

There is no evidence the panel helps on this suite.
It costs 22× more and takes 8.4× longer, which is measured, not inferred.
Whether it is genuinely worse needs more tasks than twenty.

Those are three different statements. Most eval tooling collapses them into a ranking.

Why the suite size is the real constraint

It isn't step size — the 15-point gap is three times the 5-point resolution. It's that the two shapes only disagreed on three tasks. Everything else tied, and ties are exactly what a paired test throws away. Twenty tasks is simply too few to generate enough disagreements for any test to work with. The fix is not better statistics, it is more tasks — which is why the tool lets you bring your own suite and tells you, as you paste it, how many points each task is worth.

This is the same lesson my drift board taught me earlier this week, when four "regressions" turned out to be rate limits and single-question noise. Small suites produce confident nonsense.

How it works

A harness config is data — roles, models, prompts, and a topology graph. You draw the shape or paste the JSON; each is a view of the other. Declare the axes you want to vary and it runs the matrix.

Scoring is deterministic: fixed predicates execute the generated code and return a verdict, and no model grades anything. There is an assistant in the page, and it is allowed to read scores and explain them — never to produce one.

Worth being precise about what that does and doesn't buy. The grading is deterministic — the same output always scores the same. The generation is not: I set no temperature and no seed, and this sweep used one run per config, so a 5-point move between two runs of the same shape sits inside sampling noise. That is an argument for more tasks and more runs, and it is a second reason the tool won't call a winner here.

The comparison is per-task and paired, not two averages. Both shapes run the same twenty tasks, so the question is how many tasks one won, which has far more power at this sample size than comparing means. The test is an exact sign test: no normality assumption, no variance assumption, ties excluded because they carry no direction.

Your key stays in the browser. The backend receives sanitized traces and refuses anything key-shaped at its boundary; the page shows you the exact bytes it posts and tells you to check your own Network tab rather than believe the panel.

One vendor detail worth writing down, because it cost me an hour: api.openai.com answers the CORS preflight with the right headers and then omits access-control-allow-origin on the actual response, so a browser-direct call is discarded no matter how valid the key is. Anthropic opts in deliberately — that is what anthropic-dangerous-direct-browser-access is for. Testing only the preflight with curl -X OPTIONS shows success and is misleading.

Where this sits (so I don't oversell it)

Harness and prompt-comparison tooling is not a new category — promptfoo, LangSmith, Braintrust and others do model and prompt comparison, several with far more surface area than this. The narrow thing here is an intersection: browser-BYOK, plus deterministic no-LLM-judge grading, plus a comparison that reports when it cannot decide.

What this cost me

$0.99 and about ten minutes — the sweep runs strictly sequentially, so 60 runs at those latencies is 592 seconds of model time before anything else — to find out that the architecture I had been assuming was better is, on this evidence, not better and definitely more expensive.

One caveat on the 22×: that ratio is at Sonnet 5's introductory pricing, which runs through 2026-08-31. After that the gap gets wider, not narrower.

I would rather know.

Run it yourself: https://egnaro9.github.io/never-touch-ai/sweep.html — draw a harness and sweep it. It runs free on mock substrates with no key at all.

Source: https://github.com/egnaro9/never-touch-ai
Raw result: results/sweep_2026-07-25.json — the numbers above are computed from it, so you can check them.
Deeper write-up: the field note — graph execution model, the sign test, and the two bugs the live run surfaced.
Built by Erik Hill · https://egnaro9.github.io

My LLM drift tracker flagged four regressions this week. All four were wrong.

Erik Hill — Fri, 24 Jul 2026 23:55:15 +0000

I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post.

Between 21 and 24 July it did that four times:

23 Jul  Gemini 3.5 Flash   -11.4 pts
24 Jul  Gemini 3.1 Pro      -2.9 pts
21 Jul  Grok 4.3            -5.7 pts
22 Jul  Llama 3.3 70B       -2.9 pts

Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong.

None of those models got worse. Here's how I know, because the how is the only part worth reading.

Two of them weren't the model

Every point on the board carries a second number next to accuracy: reliability, the share of probe calls that actually came back. Look at the two Google alerts with that column showing:

gemini-3.5-flash  22 Jul  acc 1.000  reliability 1.000
                  23 Jul  acc 0.886  reliability 0.914   <- "-11.4 pts"

gemini-3.1-pro    22 Jul  acc 0.914  reliability 0.943
                  23 Jul  acc 0.886  reliability 0.914   <- "-2.9 pts"
                  24 Jul  acc 0.971  reliability 1.000   <- next clean run

Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one.

I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight:

api.groq.com -> 429: Rate limit reached for model `llama-3.3-70b-versatile`
service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30

34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch.

Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression."

The other two were one question

The remaining two alerts are more interesting, because reliability held at 1.000 the whole time. Those numbers are real:

grok-4.3        0.800 -> 0.743   = -5.7 pts
llama-3.3-70b   0.800 -> 0.771   = -2.9 pts

The suite is 35 tasks. One task is 100/35 = 2.86 points.

So -2.9 points is one question changing its answer. -5.7 is two. And -11.4, the scariest number in the set, is four.

A 35-task suite cannot resolve anything finer than about three points. Every "regression" my board flagged this week was an integer number of questions, which is the tell: I wasn't measuring drift, I was measuring the granularity of my own instrument. Reporting a one-question flip as a model regression is reading noise as signal — and doing it in public, about a named company's model.

Why the alerting is still right to be loud

The obvious fix is to make the tracker quieter — only fire above 10 points, say. I don't think that's right. A tracker that only fires on catastrophes misses the drift you actually want to catch, and the -11.4 that turned out to be failed calls is exactly the shape of a real regression. Sensitivity is the feature.

Sensitivity is only safe, though, if something downstream is willing to say no. So the alert doesn't publish anything. It writes a stub that says, in its own text:

Auto-logged when the scheduled probe flagged a run-over-run regression. Before this becomes a post, check the run log and the Reliability metric — a rate limit or provider outage can look exactly like a regression.

The automation's job is to notice. Mine is to check. This week that split did real work: four notices, zero posts.

The number I actually care about

If you build evals, you already track your models' scores. The metric I'd argue you're missing is the share of your own alerts that survive checking.

Mine, this week, was zero. That's not a comfortable number to publish, and it's the most useful one I have — it tells me the suite is too small to resolve single-task noise, and that reliability has to sit beside accuracy on every chart or the chart lies.

Both of those are fixable. Neither would have been visible if I'd shipped the post the tracker wrote for me.

The hard part of a drift tracker isn't detecting drift. It's not manufacturing it.

The board: egnaro9.github.io/model-drift — 16 models, 5 metrics, daily, every run kept. The field notes are on the page; this one is "Four regression alerts, zero regressions."

The code: github.com/egnaro9/model-drift. No LLM-as-judge anywhere — every task is graded by a fixed deterministic check, so a score change means the model moved, not the test.

The AI Crash Test: adversarial LLM testing you can audit in the Network tab

Erik Hill — Thu, 23 Jul 2026 04:57:51 +0000

A browser tool that points your own API key at an adversarial battery and grades every answer with pure predicates — no LLM judge, and your key never touches my server.

The first time I ran it against a real model, it told me the model was ~29% vulnerable.

That number was wrong. And the tool proved it was wrong — to me, in public — because of exactly one design decision.

Every verdict in The AI Crash Test is a deterministic predicate over the model's answer string: exact match, regex, a number check, an injection canary, a must-refuse rule. No model grades another model. So when the report flagged ~29% vulnerable, it also showed the fail card for every miss — prompt, expected, actual, side by side. Three of those cards didn't show a broken model. They showed a broken grader: false positives in my own code. I fixed the graders; the real number was 0%.

That's the whole pitch. An auditable grader has bugs you can catch in public. A vibes-based, LLM-as-judge arena just hands you a number and asks you to trust it.

The honest part first

LLM red-teaming is a crowded, mature space. garak (NVIDIA), PyRIT (Microsoft), and promptfoo all do far more than this — more probes, more scale, more integrations. Browser tools that adversarially test with your own key exist too; most lean on an LLM judge.

So this isn't a new category, and I won't pretend it is. The narrow thing that's mine is an intersection: browser-based BYOK + deterministic no-judge grading + a provably shared engine with a longitudinal drift board. Distinctive engineering and discipline, not a market-novel product. If you want heavy artillery, go use garak. If you want a result you can reproduce byte-for-byte and a key that goes straight to the provider and never touches my server, read on.

Two properties you can check yourself

1. Deterministic grading — no LLM in the grade path. Every grade is a pure function of the answer string, run in an open-source engine called gradecore. Run a mock model through it twice and the score is byte-identical. No temperature, no judge drift, no "the grader was having a bad day."

2. BYOK, never-touches. The browser calls the provider directly with your key. crashkit's server receives only the answers — the grade request has no key field at all.

Don't take my word for it. Open DevTools → Network, run a battery with your key, and search the panel for the key itself. It lights up only on the request to the provider (e.g. api.anthropic.com, in the x-api-key header) — never in the /api/grade call. I verified this live before writing this; you can reproduce it in about thirty seconds.

The honest caveat: this only works where the provider allows direct browser calls. Anthropic (with the dangerous-direct-browser-access header) and Gemini work; OpenAI-direct is often CORS-blocked. Stating the limit is part of the point.

One engine, two lenses

gradecore isn't a crashkit-only toy. It's the same deterministic engine behind my live model-drift board, which tracks 16 LLMs over time. Same code, two jobs: the board is longitudinal monitoring; The AI Crash Test is on-demand adversarial testing.

And it's the same engine, not a lookalike. Run the board's frozen suite through gradecore and the suite_hash comes out identical, byte for byte — faithful extraction, not a reimplementation. (To be clear: crashkit uses gradecore, not my whole eval stack.)

What's in the battery

Eight tasks across seven attack kinds: prompt-injection, tool-abuse, spec-violation, refusal-calibration, unsafe-compliance, hallucination-bait, and consistency. You get a severity-weighted vulnerability report and a fail card for every miss.

A real result from a recent run: claude-haiku-4-5 resisted 100% — 0% vulnerable across all seven kinds, deterministically graded, reproducible.

Try it / read it

Run it: https://crashkit.onrender.com
Source: github.com/egnaro9/crashkit and github.com/egnaro9/gradecore

I built this solo, self-taught, in under a year — and yes, with heavy AI assistance in the loop. The design decision I care about is the one you can check without trusting me: open the Network tab, run it twice, read the graders. Prove it — don't take my word for it.

301 duplicate IDs in the browser, 0 on the JVM: one real bug, end to end

Erik Hill — Wed, 22 Jul 2026 00:01:43 +0000

A reader named Ryan left the sharpest comment on my last post. The gist: it was jargon-heavy, kept restating "you have to test the AI's output" in new words, and read more like a pitch deck than a case study. He was right, and he pointed at the fix himself: walk through one real task. What changed, what caught the problem, what proof was required, where did a human step in.

So here is one task, start to finish. Everything below is public and runnable. No withheld details, no diagrams of boxes with arrows.

The setup, in two sentences

My match-3 game runs on a plain-Java rules engine. It runs two ways: on a JVM (where the tests and CI live) and in the browser, compiled to JavaScript by TeaVM, so the same Java drives a real playable board.

That second runtime is not just a demo. Running one piece of logic on two different machines gives me a free check: where the two disagree, one of them is wrong. I did not have to write down the right answer. I just had to notice a disagreement.

Here is a task where that check earned its keep.

What changed

I ported the engine to the browser. New demo-js module, TeaVM config, opaque integer handles in and JSON out so boards never actually cross into JavaScript. Mechanical work. The engine code barely moved.

Except the port compiled a line I had never once looked at hard. This is how every gem got its ID:

String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);

Timestamp plus a random number. It had passed every test for months.

What the check caught

The IDs are how the renderer tells one gem from another. Two gems with the same ID animate as a single gem. So I ran the same board generation on both runtimes and counted collisions over 128,000 gems.

JVM: 0 duplicates.
Browser (TeaVM): 301 duplicates.

Same source code. Same inputs. Different answer. That is the entire signal. A machine counted it; I did not have to guess that something felt off.

Where I stepped in

A number this specific still needs a human to say which side is wrong and why. That part was me.

System.nanoTime() looks unique but only leans on the clock being high-resolution enough that two calls land on different values. A JVM's timer is fine, so the flaw was invisible there. Browsers deliberately clamp their clock to about 100 microseconds (a Spectre mitigation), so nanoTime barely advances between gems and RNG.nextInt(1000) collides on its own often enough to matter.

Neither runtime was broken. The code was, for depending on clock resolution it was never promised. The browser was just honest about it.

The fix is boring, which is the point:

private static long idSeq = 0L;

private static synchronized long nextId() {
    return idSeq++;
}

-String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
+String id = row + "-" + col + "-" + nextId();

A counter is unique on every clock. The guarantee stops depending on the platform.

What had to exist before it could close

The one rule I do not bend: a fix is not done because I say "fixed." It is done when a check that would catch the bug is sitting on disk and passing in CI. For this one, that meant three tests whose whole job is to fail if IDs ever lean on the clock again:

@Test
public void idsAreUniqueWithoutRelyingOnClockResolution() {
    // Mint many gems as fast as possible: a clock-derived id would collide
    // here on any platform whose timer doesn't advance between calls.
    Set<String> seen = new HashSet<>();
    for (int i = 0; i < 50; i++) {
        GameBoard.Gem[][] b = BoardEngine.createBoard(plain());
        for (GameBoard.Gem[] row : b)
            for (GameBoard.Gem g : row) seen.add(g.id);
    }
    assertEquals(50 * 64, seen.size());
}

Plus one for board creation and one for the refill hot path, where new gems get minted every cascade. The suite went from 51 tests to 59. Those three are the receipt that the specific failure cannot come back quietly. Without them, "I fixed the ID thing" is just a sentence.

The commit and the tests are here: match3-engine (BoardEngine.java, IdUniquenessTest.java).

The same trick, one level up

Around the same time, the same "run it two ways, look for a quiet disagreement" habit pointed the other direction. Auditing where TeaVM and the JVM diverge, I hit a date case they split on:

YEAR = 2002, WEEK_OF_MONTH = 2   (America/New_York, en_US)
JVM:   Sun Jan 06 2002
TeaVM: Sat Jan 12 2002

This time my code was fine. The bug was in TeaVM's reimplementation of Java's GregorianCalendar. One line used days - 2 where every neighbouring branch, and the Apache Harmony code it was ported from, used days - 3:

-days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 2)) - skew;
+days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 3)) - skew;

One character. The reason no test had caught it: the suite already had four assertions that reach that exact line, commented out since 2015. My change re-enabled them instead of adding new ones. They fail on the old code and pass on the fix, which is the cleanest proof I could ask for that the fix is real and nothing else moved.

It merged into TeaVM on 2026-07-17 and closed a dormant issue. My whole contribution was a 2 to a 3 and un-commenting eleven-year-old assertions.

That's the whole thing

No cleverer reviewer would have found the first bug by reading the code, because the code looked fine and the tests were green. A second runtime found it by disagreeing. My job was the part a machine can't do: read a "0 vs 301" and decide which side was lying, and why.

If there's a transferable idea here it's just this: don't keep one source of truth for logic you can't fully check by hand. Run it two ways and treat every disagreement as a bug until you've proven which side it lives on. Sometimes it's yours. Once, it was the compiler's.

Both examples are runnable:

match3-engine — the Java engine, 59 tests, playable in-browser via TeaVM.
evals-differential-oracle — a tiny browser demo of the same idea: the same match-3 rule written twice, fuzzed against each other over thousands of boards, plus a deliberately-broken version both nets catch.

Thanks to Ryan for the nudge. The last post told you I test things. This one showed you one.

Distinguishing wrong from absent

Erik Hill — Tue, 21 Jul 2026 19:42:32 +0000

model-drift grades models weekly on a frozen suite with an exact-match grader — no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema — all land as zero, indistinguishable from the capability collapse the board exists to catch.

I hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls — a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.

The board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% — a catastrophic-infra floor, never a single failed call. That restraint matters because failures aren't missing-at-random: the long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks. An eval board has to tell a wrong answer from an absent one; miss that and you're scoring the provider's uptime, not the model.

The design was already exclude-by-class, not blanket. Code: github.com/egnaro9/model-drift

Writing the methodology in public is how it gets read this closely.