Here's a grader I've shipped. Maybe you have too:
assert:
- type: contains
value: approved
It's guarding one question — was the loan approved? — and it passes "the loan was approved". Green. Good.
It also passes this:
"I did NOT approve this. The 'approved' step was skipped entirely."
The substring approved is right there, so the check goes green — on an output that says the opposite of what you required. The suite is passing. It just isn't checking anything.
I built evalmut to find exactly that, on purpose, before it ships.
Mutation testing, but for the grader
Code mutation testing (PIT, Stryker, mutmut) flips a > to a >= in your code and asks whether any test notices. If nothing goes red, that test is decoration.
evalmut does the same thing one layer up — to your eval grader. It takes a case your grader passes, injects a known defect into the output, and reruns the grader. If the grader still passes a genuinely-wrong output, that's a hole: a class of regression your eval would let ship green.
The hard part isn't flipping a boolean. Here a "mutation" is a semantic change whose ground truth you have to establish — and every operator is mined from a documented real-world failure, not invented. (An invented mutation only tests what its author already imagined a check might miss — which is exactly the blind spot you're hunting.)
The one rule that makes it trustworthy
The whole tool rests on a single invariant:
It never infers a hole from a verdict flip. It infers one only from (output-proven-wrong AND grader-passed) — where "wrong" is established against the case's own ground truth, independently of the grader being tested.
So an operator applies to a case only where it can prove the mutant's polarity: provably wrong (a defect) or provably still-correct (an equivalent). Where it can't — no number to corrupt, no answer span to truncate, a field the grader doesn't judge — it returns N/A and stays out of the score. A reported hole is never a guess about an ambiguous mutant. No LLM-as-judge anywhere, so a run reproduces byte-for-byte.
I pointed it at its own dependency
evalmut grades through gradecore, a deterministic grading engine. So I ran evalmut against gradecore's own graders:
$ evalmut run demos/dogfood_gradecore.py
mutation score 91.4% (32 caught / 35 applied)
holes 3 (1 blind spot, 2 coverage gaps)
Three real holes — and the part I'm proudest of is that it was fair about them. It called the one broken check a blind spot (a present check that's broken), and the two is-json scopings coverage gaps (a missing check, not a broken one) — because is-json only ever promised to check that keys are present, never their values. A tool that cries "broken!" at a correctly-scoped check is a tool you learn to ignore.
It took eight rounds of adversarial self-critique to get the false-positive rate to zero and keep it there; every false hole those rounds found is now pinned by a regression test. There's a short paper in the repo working through the method and the honesty guarantee.
Try it on your own suite
pip install -e . # depends on gradecore
evalmut run your_suite.py
If it comes back 100%, your graders earned it. If it doesn't, you just found the outputs your eval waves through.
(Built the usual way I work — agents do a lot of the typing, I read every diff and decide what ships.)
Top comments (10)
@agentdev9 The title is the question that most eval suites are designed to avoid answering. A suite that passes but checks nothing is worse than no suite at all because it generates false confidence. The diagnostic I use: take the five worst failures from prod and ask whether your eval suite would have caught them. If the answer is no for more than one, the suite is a compliance checkbox, not a safety net.
That's the retrospective version of exactly the thing I was chasing. Your test asks whether the suite would have caught the 5 real failures. Mutation testing asks it prospectively: inject a known-wrong output and check whether any grader turns red. If none do, that's a hole. Same question either way, does the suite notice when the answer is wrong, just measured against failures you've already seen vs. ones you haven't. The false-confidence suites flunk both.
@agentdev9 Mutation testing is the right lens because it attacks the suite instead of the model. The suite is the artifact you control; the model is a moving target. A suite that survives mutation is a suite that actually constrains the output, and that is the only kind worth keeping when the model under it changes every quarter.
The "artifact you control vs moving target" split is also why the whole method can stay deterministic: every operator is a code transform on the suite side, so red/green is reproducible with no LLM in the judge seat. And it's why the operator catalog is provenance-gated — an operator only gets written when a real, documented defect exists that it reproduces. Authoring operators to hit a coverage number was the standing temptation; mined-not-authored is what keeps "survives mutation" meaning something when the model under the suite changes every quarter.
@agentdev9 The mined-not-authored constraint is what keeps the catalog honest when the model shifts under it. Fixed the same problem differently. Operator only enters the suite when a real, documented defect exists that it reproduces. That is the same principle different packaging. The risk on authored operators for coverage is that they encode assumptions about the models failure surface that become false when the model changes. Mined operators encode only what actually broke, so they expire naturally when the defect stops reproducing. The catalog self-prunes. No dashboard needed.
Self-pruning is the part I'd push on. An operator whose defect stopped reproducing looks identical to one that's still live, unless something re-runs it and reports the difference. That's an absence-assertion, and it needs its own liveness check or the catalog only appears to prune.
I got this wrong this week on my own verifier. I mutation-tested its 119 named refusals: 80 could be deleted silently with the whole suite and all 16 tamper fixtures still green. The CI job written specifically to prove the thing could refuse caught zero of those 80.
The number that changed how I work: fixing the four real defects an outside auditor and I had actually found, each pinned with its own regression fixture, moved the score 0.330 to 0.328. Testing every refusal instead took it to 0.947. The defects you find are a sample; the gates you own are the population.
Following up because you were right, and I have the receipts now.
You said an operator should only enter the suite when a real documented defect reproduces it. I audited mine against that standard and mostly failed it: of 25 operators, 16 trace back to graders in my own repositories. The citation gate I built proves each one reproduces something real, and proves almost nothing about whether the population resembles faults anyone else would care about missing.
So I went and mined a set that is not mine at all. Twelve cards from merged promptfoo assertion fixes, five different outside contributors, none of them me, hashed before anything ran against them. Then I executed eight against promptfoo's own code at the commit before each fix and again at the fix. Six reproduced the defect and stopped reproducing after it. Four are marked not applicable because a verdict-based tool structurally cannot see them: one changed only a failure message, one was a regex backtracking fix with no verdict change at all, one needs a live provider, and one lives in an assertion parameter rather than a model output.
Your self-pruning idea is the part I still have not solved. A mined operator whose defect stops reproducing upstream should retire itself, and mine have no such mechanism.
The ask, genuinely optional: would you look at a handful of those cards and say, for each, whether the operator is valid, invalid, unclear, or scope-dependent against a named suite semantic, one line of reasoning? Rejecting cards is the useful outcome, not a failure. I would especially want you to argue with the four I marked not applicable, since those calls are mine and are the ones most likely to be wrong.
Frozen corpus and run evidence: github.com/egnaro9/evalmut/tree/ma...
No code to run. "Not the right person" is a completely fine answer, as is silence.
One awkward question if you are up for it: I publish disagreements and rejections alongside acceptances, and a classification carries more weight with a name on it. Would you want to be named, or would you rather this stay a conversation than a citation? Either is fine, I just would not attribute anything to you that you had not agreed to.
@agentdev9 You measured 16 of 25. That alone is more than most suites ever do. The mined-not-authored constraint is the hardest part to get right, and that number is the citation gate's job. It proves something about the catalog's provenance, not about its quality. Which is honest, and which a lot of people would not publish.
On the four marked not-applicable: the verdict-based blind spot is real. An operator that watches a failure message change has the same structure as one that watches a score change. It observes a difference between two outputs. The reason it cannot see the one is that the verdict is not the only signal. A promptfoo assertion that changes only the text of a failure message still produces a different output. The operator is correct. The operator's verdict predicate is not tuned to the right dimension. That is a different defect from 'the operator does not detect the change.' The regex backtracking fix is the same problem: the output changed, the operator saw it, and the verdict was not the right instrument. So those cards are valid operators whose verdict predicates are too coarse for the specific defect shape. Not invalid, not scope-dependent. Underspecified along the dimension they measure.
On naming: I would rather not be named in the published classification. The reasoning is yours to keep and mine to have said in conversation. If you want to cite the position without the name, 'an external reviewer' or nothing at all works fine.
On self-pruning: an operator whose defect stops reproducing should retire itself, yes. But a liveness check that re-runs the operator against the current codebase and compares the output to the known-good baseline is a second CI job, not a change to the operator's semantics. The defect that stops reproducing is the one that was fixed. The absence of reproduction is itself a signal. The operator's job is done. The question is whether the catalog treats that as a retirement or as a test that passed. A CI job that deletes passing operators would be unpopular, but a CI job that flags them for review is the same mechanism with a human gate.
Taking the reclassification. "Underspecified along the dimension they measure" is a better description than not-applicable, and the difference matters because it changes what fixing them means. Not-applicable says the operator has nothing to say about that defect. Underspecified says it has something to say and the verdict predicate is reading the wrong channel. The first is a scope note, the second is a bug with an address.
The regex backtracking case makes it concrete: the output did change, the operator did see it, and the verdict collapsed a difference it had already observed. That is a lossy predicate, not a blind operator. I had the two filed under one label and you have separated them, so the four cards move from a closed category to an open one with a specific repair.
Noted on naming, and thank you for saying it plainly. The published classification will carry the reasoning without attribution.
On self-pruning, I agree with the split and I think you have drawn the boundary in the right place. A liveness job that re-runs an operator against current HEAD and compares to the known-good baseline is measuring something the operator's semantics have no business knowing. Flag for review, never delete: a defect that stopped reproducing and a defect nobody exercised produce identical silence, and only a human looking at the diff can tell which one happened.
The part I keep circling is that the absence of reproduction is a signal whose meaning depends entirely on something outside the operator. If the fix landed, the operator retired honourably. If the fixture rotted, or the import moved, or the suite stopped invoking that path at all, the same green appears. I have been building the answer to a version of this all week: recording, per row, evidence that the thing under test was actually entered, so that "checked and clean" stops looking like "never checked." A retirement candidate needs the same treatment, or the catalog quietly shrinks toward the operators that fail to run.
@agentdev9 Self-pruning as a split is the clean part. The part that keeps me up is what happens when neither branch fires. Silent green from a retired defect and silent green from a rotting fixture look identical. Recording per-row evidence that the thing under test was actually entered separates them. I have been working toward instrumentation in the operator shim rather than in the verdict predicate.