DEV Community: Maya Andersson

A Higher Pass Rate Can Mean a Worse Model. The Math Is Simpson's Paradox.

Maya Andersson — Tue, 25 Aug 2026 20:52:04 +0000

We shipped a model update last quarter that moved our aggregate pass rate from 81.2% to 83.6%. Everyone was happy. I was not, because I had seen this shape before. When I split the eval set by the four traffic slices we actually serve, the new model was worse on three of them. The aggregate went up anyway. This is not a paradox in the mystical sense. It is arithmetic, and it has a name.

What Simpson's paradox actually is

Simpson's paradox is when a trend that holds in every subgroup reverses once you pool the subgroups. The reversal comes from the subgroup sizes, not the subgroup rates. If your eval set has a different slice mix than it had last time, or a different mix than production, the pooled average is a weighted sum where the weights are doing the talking.

Here is the case that ruined my afternoon, with numbers close to the real ones.

import pandas as pd

data = {
    "slice":       ["short_factual", "multi_turn", "tool_use", "long_context"],
    "v1_pass":     [855, 560, 408, 10],
    "v1_total":    [900, 700, 600, 50],
    "v2_pass":     [792, 532, 390, 950],
    "v2_total":    [900, 700, 600, 1000],
}
df = pd.DataFrame(data)
for v in ["v1", "v2"]:
    df[f"{v}_rate"] = df[f"{v}_pass"] / df[f"{v}_total"]
print(df[["slice", "v1_rate", "v2_rate"]].round(3))

agg_v1 = df["v1_pass"].sum() / df["v1_total"].sum()
agg_v2 = df["v2_pass"].sum() / df["v2_total"].sum()
print(f"aggregate v1: {agg_v1:.3f}, aggregate v2: {agg_v2:.3f}")

Output:

            slice  v1_rate  v2_rate
0   short_factual    0.950    0.880
1      multi_turn    0.800    0.760
2        tool_use    0.680    0.650
3    long_context    0.200    0.950
aggregate v1: 0.815, aggregate v2: 0.833

v2 lost on three of four slices: short_factual (0.950 to 0.880), multi_turn (0.800 to 0.760), tool_use (0.680 to 0.650). It won only on long_context, and it won catastrophically large there (0.200 to 0.950, almost certainly because someone fixed a truncation bug in that eval fixture, not the model itself). The aggregate moved 1.8 points in v2's favor anyway. How? The v2 eval set has 1000 long_context rows against v1's 50. Twenty times the representation for the one slice where v2 wins. The aggregate is not measuring "is the model better". It is measuring "is the model better, weighted by whatever slice mix happened to be in this run".

Why the average is the wrong unit

The pooled pass rate answers a question almost nobody asks: what is the probability a uniformly random row from this particular eval set passes? Your users are not uniformly random rows from your eval set. They arrive in a fixed mix, and that mix is a property of your product, not your test harness. The fix is to stop comparing pooled numbers and compare slice by slice, then recombine using a single fixed weighting that you control. Standardize both models to the same slice weights (production traffic share is the honest choice).

w = {"short_factual": 0.45, "multi_turn": 0.30, "tool_use": 0.20, "long_context": 0.05}
df["w"] = df["slice"].map(w)
std_v1 = (df["v1_rate"] * df["w"]).sum()
std_v2 = (df["v2_rate"] * df["w"]).sum()
print(f"production-weighted v1: {std_v1:.3f}, v2: {std_v2:.3f}")

Output:

production-weighted v1: 0.814, v2: 0.802

Under the weighting that matches who we actually serve, v2 is 1.2 points worse. The naive aggregate said plus 1.8. The gap between those two numbers is 3.0 points, and it is invisible if you only log one scalar per run. This is the machinery epidemiologists call direct standardization, and the warning is old: Blyth's 1972 JASA paper "On Simpson's Paradox and the Sure-Thing Principle" laid out exactly how a treatment can look better marginally while being worse in every stratum. Nothing about LLMs changed the math.

What I do now

One number per eval run is a liability. The slice vector is the artifact. I store per-slice pass counts and totals every run, I fix a weighting derived from production and apply it to every model, and I never let two runs with different slice mixes be compared by their pooled rate. When the pooled number and the standardized number disagree, the standardized one is the quality signal and the pooled one is a sampling artifact.

FAQ

Is this just a sample-size problem I can fix with more eval data? No. More data shrinks variance, not bias. Simpson's reversal is a weighting effect, so a bigger eval set with the same skewed slice mix gives you a tighter estimate of the wrong number. Stratify, then standardize.

My slices have wildly different sizes. Won't the tiny ones be noisy under production weights? Yes, and that is information, not an objection. If long_context is 5% of traffic but swings your decision, you have too few long_context examples to decide. Put a confidence interval on each slice rate and widen your eval set where the weight times the uncertainty is largest.

Can I just always weight by production share and forget the pooled number? Production-weighted standardization is the right default for "which model serves users better", but keep the per-slice rates visible. A model that improves the average while regressing your highest-stakes slice (say, tool_use that triggers real actions) can still be the wrong ship.

Open question

Production-weighting assumes the slice definitions are fixed and the traffic mix is roughly stationary. Neither holds cleanly. Slices are a human carving of a continuous input space, and a model change can shift the traffic distribution itself (better long-context handling pulls in more long-context users). When the act of shipping moves the weights you standardized on, what is the honest denominator? I do not have a clean answer for evaluating under distribution shift that you yourself caused, and I have not seen one that survives contact with a fast-moving product.

Dividing your RAG score by retrieval recall overstates your generation quality, and here is by how much

Maya Andersson — Tue, 25 Aug 2026 20:45:03 +0000

Almost every RAG eval writeup I read, including several I have recommended, uses the same mental model. The system is two stages in series. Retrieval either finds the right context or it does not, with probability r. Given the right context, generation either produces the right answer or it does not, with probability g. So end-to-end accuracy is r times g, and if you want to know how good your generator is you divide.

The multiplication is wrong, the division is worse, and the error runs in a consistent direction. It makes your generator look better than it is, and it makes retrieval regressions look like generation regressions.

None of this is deep. It is one term that the product model sets to zero and that is not zero in any system I have measured.

The term everyone drops

The product model says a RAG system answers correctly only when retrieval succeeds. That is not how a language model behaves. It can answer correctly with bad context, because the answer is in its parameters, or because the question is inferable, or because the wrong document happens to contain the right fact. (The parametric-versus-contextual tension has its own literature; Longpre et al., "Entity-Based Knowledge Conflicts in Question Answering", arXiv:2109.05052, is the cleanest construction of it I know.) Call that probability g_bad.

The honest structure is a two-by-two, and end-to-end accuracy is:

e2e = r * g_good + (1 - r) * g_bad

where g_good is P(correct | good context) and g_bad is P(correct | bad context). The product model is the special case g_bad = 0.

If you estimate generation quality by dividing, you get

g_hat = e2e / r  =  g_good + (1 - r) * g_bad / r

so the bias is exactly (1 - r) * g_bad / r, and it is upward whenever g_bad is above zero. It grows as retrieval gets worse, which is the opposite of reassuring.

Worked, at r = 0.72 and a true g_good of 0.88:

g_bad = 0.00: e2e 0.6336, g_hat 0.8800, bias 0.0000
g_bad = 0.05: e2e 0.6476, g_hat 0.8994, bias +0.0194
g_bad = 0.10: e2e 0.6616, g_hat 0.9189, bias +0.0389
g_bad = 0.20: e2e 0.6896, g_hat 0.9578, bias +0.0778
g_bad = 0.30: e2e 0.7176, g_hat 0.9967, bias +0.1167
g_bad = 0.40: e2e 0.7456, g_hat 1.0356, bias +0.1556

At g_bad = 0.20, which I do not think is high for a general-knowledge domain, you report your generator at 0.958 when it is at 0.880. Just under eight points, in the flattering direction, on a metric people use to decide that generation is fine and retrieval is the problem.

The last row is the useful diagnostic. When g_bad is high enough, g_hat exceeds 1.0. If you have ever divided end-to-end by retrieval recall and got a number above one and quietly clipped it, that clip was the model telling you it was misspecified.

What a single end-to-end number pins down

Bias is the smaller of the two problems. The larger one is that a single end-to-end number does not pin down the system that produced it. You have one equation and, if you count honestly, three unknowns, and note that what follows is the generous version, since I hand you one of them for free.

Fix g_bad at 0.20 and take the end-to-end score from the table above, 0.6896. Here are the (r, g_good) pairs that produce exactly it:

r = 0.55: g_good = 1.0902, impossible
r = 0.65: g_good = 0.9532
r = 0.72: g_good = 0.8800
r = 0.85: g_good = 0.7760
r = 0.95: g_good = 0.7154

Every one of those except the first is a perfectly ordinary system, and they describe completely different engineering situations. At r = 0.65 you have a near-perfect generator starved of context. At r = 0.95 you have excellent retrieval and a generator that fails nearly three times in ten on good context. The end-to-end number is identical.

The impossible row is worth a second look, because it is the one thing a single score does pin down. g_good cannot exceed 1, so r is bounded below by (e2e - g_bad) / (1 - g_bad), which here is 0.612. One number gives you one inequality on one parameter. That is the whole of what it gives you.

So an end-to-end score does not summarise a RAG system's health. It places one constraint on a surface. Watching that constraint move over time tells you the surface shifted, and nothing about which axis it shifted along.

The misattribution, worked

This is the failure I most want people to be able to recognise, because it produces a confident wrong diagnosis rather than an absence of one.

Take a real regression: retrieval degrades from 0.72 to 0.60 because an index rebuild changed chunking. Generation is untouched at g_good = 0.88, g_bad = 0.20.

End-to-end falls from 0.6896 to 0.6080, which is 8.16 points, and you will see that on the dashboard.

Now suppose your retrieval recall figure is stale, as it usually is, because it needs relevance labels and gets recomputed quarterly while end-to-end runs nightly. You divide by 0.72, the number you have.

Your inferred generation quality falls from 0.9578 to 0.8444. That is 11.33 points, and it is entirely an artifact. Generation did not move at all. Worse, the inferred generation drop is larger than the end-to-end drop, so the natural reading is that the generator is the epicentre and end-to-end was partly cushioned.

A team reading that dashboard will spend its next sprint on the generator. I did exactly that in July, in a post about an average that was hiding two failure modes, and I described editing prompts to fix what turned out to be a retrieval gap. This is the mechanism underneath that mistake.

The minimum instrumentation that resolves it

Three quantities, and they have to be measured on the same evaluation set on the same day.

r, retrieval success. Whatever your definition, hit-rate at k or a graded relevance judgement thresholded, it needs to be recomputed on every run, not quarterly. This is the expensive one because it needs relevance labels, and it is the one whose staleness causes the misattribution above.
g_good, accuracy restricted to the subset where retrieval succeeded. Conditional, not marginal.
g_bad, accuracy restricted to the subset where retrieval failed. This is the one nobody computes, it is free once you have the split, and it is the term that makes the arithmetic honest.

It really is a groupby, so here it is rather than described:

import pandas as pd

def decompose(df):                      # columns: retrieval_ok (bool), correct (bool)
    r = df.retrieval_ok.mean()
    by = df.groupby("retrieval_ok").correct.agg(["mean", "size"])
    return {"r": r,
            "g_good": by.loc[True, "mean"],  "n_good": int(by.loc[True, "size"]),
            "g_bad":  by.loc[False, "mean"], "n_bad":  int(by.loc[False, "size"]),
            "e2e_check": r * by.loc[True, "mean"] + (1 - r) * by.loc[False, "mean"]}

e2e_check is there because it should equal your measured end-to-end score. If it does not, one of the three is being computed on a different slice from the other two, which is a bug I have shipped.

With those three, end-to-end is a derived quantity rather than a measured one, and a movement decomposes into the terms that produced it. One caveat, and it is not small: g_good and g_bad are conditional on a split that retrieval itself defines, so when r moves the two subsets are re-drawn and the conditional numbers move even if the generator is byte-identical. On a two-stratum example I built, and it is the same index rebuild as the section above with the population resolved rather than assumed away: short factual questions that retrieve well and long synthesis questions that retrieve badly. The rebuild hurts the long ones, r drops from 0.720 to 0.600, and measured g_good rises from 0.8894 to 0.9050 with the generator byte-identical. Above I held g_good flat at 0.88 to isolate the division error on its own. In a real rebuild both effects are live at once and they push the inferred number in opposite directions. The effect that motivates this post shows up inside its own fix. So report the three terms with their split sizes beside them, and treat a g_good change that arrives in the same run as an r change as uninterpretable until you re-measure on a fixed question set.

There is one more line worth putting on the same page, because it is the ceiling and it is one multiplication:

r = 0.60, perfect generation on good context, g_bad 0.20: end-to-end cannot exceed 0.6800
r = 0.72: cannot exceed 0.7760
r = 0.85: cannot exceed 0.8800

That is the number to put beside any target somebody proposes, because it is the one the target has to fit under.

FAQ

Is r really binary?
No, and the two-by-two is a simplification I am using because it is the smallest structure that exposes the error. Graded retrieval quality turns the two-by-two into a curve and the bias result above survives, with the sums becoming integrals and the arithmetic becoming less legible. I would rather publish the version people can check by hand.

Does the bias not vanish if retrieval is good?
It shrinks. The bias is (1-r) * g_bad / r, so at r = 0.95 and g_bad = 0.20 it is 0.0105, which is negligible. At r = 0.72 it is 0.0778. The systems that most need a correct diagnosis are exactly the ones where the correction is largest, which is an unhelpful property of the world rather than of the method.

Is g_bad not just measuring that your relevance labels are wrong?
Sometimes, and you should check that first. If g_bad is very high, the likely explanation is that your "retrieval failed" bucket contains documents that were actually sufficient and your labeller disagreed. That is a real and common cause and it is worth ruling out before treating g_bad as parametric knowledge. Either way it is not zero, which is the point.

Why not just use a faithfulness metric on the retrieved context?
Faithfulness measures whether the answer is supported by what was retrieved, which is a different question from whether the answer is correct. A perfectly faithful answer to a wrong document is wrong. Faithfulness is a good instrument for one component of g_good and it says nothing about g_bad, because in the g_bad case the answer is correct while being unsupported by the retrieved context, which faithfulness scores as a failure.

Are r and g_good not correlated? Hard questions fail retrieval and also get answered badly.
They are, and it turns on a definition rather than on an assumption. Write the system as a mixture over question types and the bias is E[(1-r) g_bad] / E[r]. If g_bad is the failure-weighted rate, meaning accuracy on the subset where retrieval actually failed, then E[(1-r) g_bad] = g_bad times E[1-r] by definition, and the whole thing collapses back to (1 - r_bar) g_bad / r_bar. I checked this in exact rational arithmetic over mixtures of two to four strata with deliberately strong dependence between r and both generation rates, and the two sides are the same number rather than two numbers that agree to fifteen places. It is an identity.

The condition is on both conditionals, not just one. g_good has to be the retrieval-success-weighted rate and g_bad the retrieval-failure-weighted rate, which is what steps 2 and 3 above measure. Average the per-type rates any other way and the identity goes: on my four-stratum case, substituting a plain weight-weighted g_bad of 0.187 for the failure-weighted 0.095 turns a true bias of 0.1012 into a computed 0.1986.

Where dependence does bite is if you substitute a closed-book rate for it. At r = 0.72, if the true failure-subset g_bad is 0.20 and you plug in a closed-book number of 0.35, you compute a bias of 0.1361 against a true 0.0778. That is the error, and it is an error of measurement rather than of model.

Open question

The closest thing I have seen published is the no-retrieval ablation most RAG papers report, and that is a different quantity. A closed-book baseline is accuracy with no context. g_bad is accuracy with wrong context, and wrong context is not neutral: Shi and colleagues showed that irrelevant material in the prompt actively degrades accuracy on problems the model otherwise solves (Shi et al., "Large Language Models Can Be Easily Distracted by Irrelevant Context", arXiv:2302.00093, 2023). I would therefore expect g_bad to sit below the closed-book number, though their setting is grade-school arithmetic with an inserted distractor rather than retrieval, and I have not seen no-context and wrong-context measured side by side on the same knowledge task. Nobody reports that gap. Reporting it needs no new labels beyond the ones a retrieval evaluation already requires. It is one groupby.

My suspicion, and it is only that, is that it is much higher than people expect for general-knowledge benchmarks and much lower for genuinely proprietary corpora, and that the gap explains a chunk of why RAG results published on public benchmarks do not reproduce inside companies. On a public benchmark the model may already know the answer, so g_bad is large, the product model's bias is large, and the reported generation quality is inflated. On an internal corpus of last quarter's contracts, g_bad is near zero, the product model is nearly correct, and the same pipeline reports a much worse generator.

That is a testable claim and I do not have the data to test it. If you have both kinds of corpus and the same pipeline over them, the groupby is four lines and I would very much like to see the two numbers side by side.

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

Maya Andersson — Tue, 25 Aug 2026 20:42:48 +0000

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision boundary. In a simulation where the judge is a clean, unbiased, well-behaved instrument, overall agreement of 0.92 comes with agreement of 0.60 on the examples within 0.05 of the gate's threshold. Nothing is wrong with the judge. The summary statistic is answering a question you are not asking.

Every judge validation writeup I have read, including two of my own, reports agreement as a scalar. We sampled n examples, humans labelled them, the judge labelled them, they agreed 87 percent of the time, ship it.

The number is real. It is also computed over a population that is mostly irrelevant to what the judge is for.

The mismatch, stated plainly

If you use a judge to gate, the judge is a classifier with a threshold. Its job is to sort examples into pass and fail. Almost all of its value, and all of its risk, lives in a thin band around that threshold, because that is the only region where a small error changes an outcome.

An example whose true quality is far above the cut gets passed by any judge that is not broken. An example far below gets failed by any judge that is not broken. Those examples cost you nothing when the judge is noisy, and they are the overwhelming majority of your validation set. So they set your headline agreement number, and they tell you almost nothing about the decisions you actually delegated.

This is not a claim about any real judge. It follows from the geometry.

A simulation, and what it is and is not

I want to be careful about what follows, because a number that reproduces from your own simulation is not thereby right. What is below is a simulation. It measures no real model. It is a demonstration that a mechanism exists and is large, not evidence about the size of the effect in your system.

The construction is deliberately friendly to the judge. Latent quality is uniform on 0 to 1. The human label is the ground truth by definition: pass if quality exceeds 0.5. The judge sees quality plus symmetric Gaussian noise and applies the same 0.5 threshold. So the judge's latent score is unbiased and, near the threshold, calibrated. It has no position bias, no verbosity bias, no prompt sensitivity, and agrees with the human criterion exactly. Its only flaw is noise.

I hold that construction fixed and sweep one parameter, the noise scale.

import numpy as np

def run(sd, n=200_000, thr=0.5, seed=0):
    rng = np.random.default_rng(seed)
    q = rng.uniform(0, 1, n)                  # latent quality
    human = q > thr                           # ground truth by construction
    judge = (q + rng.normal(0, sd, n)) > thr  # quality plus noise, same threshold
    agree = human == judge
    out = {"overall": agree.mean()}
    for lo, hi in [(0.0, 0.05), (0.05, 0.15), (0.15, 0.30), (0.30, 0.50)]:
        band = (np.abs(q - thr) >= lo) & (np.abs(q - thr) < hi)
        out[f"{lo:.2f}-{hi:.2f}"] = agree[band].mean()
    return out

for sd in [0.05, 0.10, 0.15, 0.20, 0.30]:
    print(sd, {k: round(v, 3) for k, v in run(sd).items()})

Rows are the noise scale. Columns are distance from the threshold.

judge noise sd 0.05: overall 0.960, within 0.05 of threshold 0.685, 0.05 to 0.15 0.958, 0.15 to 0.30 1.000, 0.30 to 0.50 1.000
judge noise sd 0.10: overall 0.919, within 0.05 of threshold 0.600, 0.05 to 0.15 0.827, 0.15 to 0.30 0.981, 0.30 to 0.50 1.000
judge noise sd 0.15: overall 0.879, within 0.05 of threshold 0.566, 0.05 to 0.15 0.740, 0.15 to 0.30 0.925, 0.30 to 0.50 0.994
judge noise sd 0.20: overall 0.840, within 0.05 of threshold 0.549, 0.05 to 0.15 0.686, 0.15 to 0.30 0.864, 0.30 to 0.50 0.973
judge noise sd 0.30: overall 0.772, within 0.05 of threshold 0.532, 0.05 to 0.15 0.627, 0.15 to 0.30 0.773, 0.30 to 0.50 0.904

Read the second row. Overall agreement is 0.919, which is the number that goes in the writeup and gets called strong. In the band where the gate is actually deciding, it is 0.600, against a coin-flip floor of 0.500.

Read the Overall column downward, then the near-threshold column. As the judge gets noisier, the headline degrades gently, from 0.96 to 0.77. The near-threshold column has already collapsed by the second row and then barely moves, because it is pinned near 0.5. Once noise dominates the signal at the boundary, more noise cannot make it much worse. The headline keeps a lot of room to look respectable while the only region you care about has been at chance for some time.

Why the headline is so insensitive

I have made a version of this argument before, about how an aggregate pass rate hides the slice that matters. The difference here is that the slice is not one I picked. It is wherever your threshold happens to sit.

The bands are not equally populated. With uniform quality, examples within 0.05 of the threshold are 10 percent of the set. The outer band, 0.30 to 0.50 away, is 40 percent, and agreement there is 1.000 in the two lowest-noise rows.

So the reported scalar is roughly a weighted average in which the hardest decile carries a tenth of the weight and the trivial 40 percent carries four times that. If your true quality distribution is not uniform but concentrated near the threshold, which is what I have watched happen once a team has been optimising against the gate for a while, the picture gets worse rather than better, because the population moves into the band where the judge is weakest.

This is a cousin of a known problem, not the same one

The classic reference for "a single agreement summary can mislead" is Feinstein and Cicchetti, High agreement but low kappa: I. The problems of two paradoxes, J Clin Epidemiol 1990;43(6):543-9, doi:10.1016/0895-4356(90)90158-L. Worth reading if you have not.

I want to be precise about the relationship, because they are not the same mechanism and I do not want to borrow authority I have not earned. Their paradoxes are about marginal imbalance: with skewed marginal totals, a high raw agreement can produce a low kappa, and asymmetry can raise kappa relative to symmetry. That is a property of the fourfold table.

What I am describing is different. It is heterogeneity of agreement across the latent scale, where the summary is fine as arithmetic and simply averages over a region that matters and a region that does not. Both land in the same practical place, which is that one number is not enough, but for different reasons, and a fix for one will not fix the other.

What to report instead

Three changes, in increasing order of effort.

Report agreement in bands of distance from the threshold, not just overall. This costs one group-by and it is the whole finding. If you only do one thing, do this.
Report the population of each band. The near-threshold share tells you how much of your traffic the judge is genuinely deciding, and it moves over time as your system improves and as your team optimises against the gate.
Sample your validation set by band rather than uniformly. If you are going to spend human labels, spend them where the judge is uncertain. Uniform sampling buys you a precise estimate of the agreement rate on examples whose outcome was never in doubt. I have argued for stratifying eval samples before, on score class. The band is a different axis, and the reason I prefer it is that it falls out of the gate's geometry rather than out of somebody's judgement about which classes are interesting.

The second one is the one I underrate. A judge validated when 8 percent of traffic sat near the threshold is not validated for a system where 30 percent does, and no property of the judge changed in between.

FAQ

Does this mean judge validation is useless?
No. It means the scalar is a summary of the wrong population for a gating decision. The same labels you already collected can be re-cut into bands at no extra labelling cost.

Is this just the same as reporting precision and recall instead of accuracy?
Related but not the same. Precision and recall also aggregate over the whole set. You can have good precision, good recall, and near-chance agreement in the boundary band, because that band is small.

My judge outputs a discrete 1 to 5, not a continuous score. Does this apply?
Yes, and it is harder to see, because the band structure is coarse. Use the distance from the cut in score units, and if your gate is "3 or above", the 3s are your near-threshold band. They are usually the largest single bucket and the least reliable.

Does a stronger judge model fix it?
Not across the range I swept. A stronger judge is a smaller sd, which is the top row rather than the bottom, and agreement within 0.05 of the threshold there is still 0.685 while the headline reads 0.96. Push sd small enough and the gap does close, and it is not even monotone: extending the sweep, the gap between overall and near-threshold agreement peaks around sd=0.10 at 0.32 and falls to 0.04 by sd=0.005. But sd=0.05 is already a very quiet judge and the gap there is 0.28.

Should I move the threshold to where the judge is confident?
That is choosing your decision boundary to flatter your instrument rather than to match your risk. If you genuinely have freedom in where the cut goes, set it on cost, then measure agreement there.

Open question

The band analysis tells you where the judge is unreliable. It does not tell you what to do with those examples at run time.

The obvious move is routing: when a score lands within some distance of the threshold, send it to a human, or to a second judge, or to a stronger model. But the distance is measured in the judge's score space, and the judge's score near the boundary is exactly the quantity we just established is noise-dominated. So the router is deciding what is uncertain using the measurement whose uncertainty is the problem.

I do not have a clean answer. Judge-reported confidence is one candidate and I distrust it for the usual reasons. Disagreement across a small ensemble is another, and it costs what it costs. If you have shipped a boundary router that survived contact with production, I would like to know what you keyed it on.

Six prompt-optimization frameworks: what matters when you run them on the same task

Maya Andersson — Tue, 18 Aug 2026 18:31:23 +0000

TL;DR: I ran six prompt-optimization frameworks against the same task and the same eval metric over a few weeks (DSPy, GEPA, TextGrad, agent-opt, Arize Prompt Learning, and MLflow's optimizer). They are not interchangeable: some are full programming models, some are single algorithms, some are platform features. The one that mattered for me was whichever optimized against MY metric on MY dataset and let me swap the search algorithm without rewriting the harness. Here is the rundown as of June 2026.

"Prompt optimization" is at least three different things

The term covers a programming model where you declare structure and an optimizer compiles the prompts (DSPy), a single search algorithm you point at a prompt (GEPA, TextGrad), and a platform feature that wraps one of those. Comparing them as if they are the same product is the first mistake. The real axis is narrower: what objective does it optimize, and can you plug in your own metric and data.

The six, and what each actually optimizes

DSPy: the ecosystem standard. You write declarative LM programs (signatures, modules) and an optimizer (MIPRO, BootstrapFewShot) compiles the prompts and few-shot examples. Most mature, biggest community. The cost is buying into the DSPy programming model; it is a framework, not a drop-in.
GEPA: a standalone evolutionary-Pareto optimizer, strong when the solution space is complex and you want a diverse set of candidates rather than one. It is an algorithm, so you bring the harness.
TextGrad: treats the prompt as a text variable and runs "textual gradient" descent, an LLM critiques the output and proposes edits. Elegant for iterative refinement; you supply the loss.
Future AGI agent-opt: an Apache-2.0 library (github.com/future-agi/agent-opt) that puts six optimizers behind one optimize() call, Random Search, Bayesian (Optuna), ProTeGi, Meta-Prompt, PromptWizard, and GEPA, so you can swap the algorithm without touching your dataset or evaluator. It scores against any metric (it reuses the metrics from their ai-evaluation SDK, or your own) and any LLM via LiteLLM, as of June 2026. The draw for me was the swap: most tools lock you into one algorithm, and the right algorithm depends on the problem shape.
Arize Prompt Learning: a feedback-loop approach that optimizes prompts from production signals. Fits if you already live in the Arize observability stack.
MLflow: the MLOps platform added prompt optimization tooling. Useful if MLflow is already your tracking backbone, less so as a standalone.

I am not crowning one. DSPy if you want the full programming model and ecosystem; GEPA if your space is complex and multi-objective; agent-opt if you want to try several algorithms against your own metric without rewriting the harness. They optimize different things.

The methodologist's test: what is the objective?

The question I care about most is whether it optimizes against a real metric computed on YOUR data, or a generic proxy. A tool that "improves your prompt" without naming the objective is just reshuffling. All six can optimize against a metric; the difference is how hard it is to plug in YOUR metric, your judge, your pass rate, your cost-adjusted score. That integration, not the search algorithm, is where most of the value and most of the friction live.

FAQ

Is this just for DSPy users? No. DSPy is a programming model; GEPA, TextGrad, and agent-opt are usable without adopting DSPy.
Which algorithm should I pick? It depends on the space: Bayesian or random for small and cheap, ProTeGi or TextGrad for iterative refinement, GEPA for complex multi-objective. This is exactly why a library that lets you swap is convenient, you do not have to commit up front.
Does prompt optimization overfit? Easily, if you optimize and evaluate on the same set. Hold out a test set, same as any ML.

Open question

Every one of these optimizes the prompt against a fixed metric, but the metric is the thing I am least sure of. If my eval metric is slightly wrong, the optimizer will faithfully exploit its flaws and hand me a prompt that games the metric. I do not have a clean way to optimize the prompt while staying robust to my own metric being imperfect. If you have, that is the comment I want.

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

Maya Andersson — Tue, 18 Aug 2026 18:28:02 +0000

The number is real. It is also computed over a population that is mostly irrelevant to what the judge is for.

The mismatch, stated plainly

This is not a claim about any real judge. It follows from the geometry.

A simulation, and what it is and is not

I hold that construction fixed and sweep one parameter, the noise scale.

import numpy as np

def run(sd, n=200_000, thr=0.5, seed=0):
    rng = np.random.default_rng(seed)
    q = rng.uniform(0, 1, n)                  # latent quality
    human = q > thr                           # ground truth by construction
    judge = (q + rng.normal(0, sd, n)) > thr  # quality plus noise, same threshold
    agree = human == judge
    out = {"overall": agree.mean()}
    for lo, hi in [(0.0, 0.05), (0.05, 0.15), (0.15, 0.30), (0.30, 0.50)]:
        band = (np.abs(q - thr) >= lo) & (np.abs(q - thr) < hi)
        out[f"{lo:.2f}-{hi:.2f}"] = agree[band].mean()
    return out

for sd in [0.05, 0.10, 0.15, 0.20, 0.30]:
    print(sd, {k: round(v, 3) for k, v in run(sd).items()})

Rows are the noise scale. Columns are distance from the threshold.

Why the headline is so insensitive

This is a cousin of a known problem, not the same one

What to report instead

Three changes, in increasing order of effort.

Report agreement in bands of distance from the threshold, not just overall. This costs one group-by and it is the whole finding. If you only do one thing, do this.
Report the population of each band. The near-threshold share tells you how much of your traffic the judge is genuinely deciding, and it moves over time as your system improves and as your team optimises against the gate.
Sample your validation set by band rather than uniformly. If you are going to spend human labels, spend them where the judge is uncertain. Uniform sampling buys you a precise estimate of the agreement rate on examples whose outcome was never in doubt. I have argued for stratifying eval samples before, on score class. The band is a different axis, and the reason I prefer it is that it falls out of the gate's geometry rather than out of somebody's judgement about which classes are interesting.

FAQ

Open question

The band analysis tells you where the judge is unreliable. It does not tell you what to do with those examples at run time.

Run forty experiments against one eval set and you will find an improvement that is not there

Maya Andersson — Wed, 12 Aug 2026 14:07:15 +0000

You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an upward-biased estimate of the thing you care about. The size of that bias is computable, it is larger than most of the "wins" teams celebrate, and there is a decade-old literature on how to reuse a holdout without fooling yourself that I almost never see cited in LLM eval writeups.

The selection machine

The workflow looks responsible. A fixed eval set of 250 examples, a binary pass judgment per example, every candidate prompt measured on the same set. Best score ships. It looks like discipline.

The problem is the word "best." Each measurement is the true pass rate plus sampling noise. Pick the maximum of forty measurements and you have preferentially picked positive noise. The winner's measured score overstates its true score even when some candidates are genuinely better than others; the statistics community has called this the winner's curse for decades.

How big the bias is

For a pass rate near p on n examples, one measurement has standard error sqrt(p(1-p)/n). At p = 0.85 and n = 250 that is 2.26 points. Now suppose, worst case, all k candidates are actually identical: every measured difference is pure noise. The expected maximum of k standard normal draws is easy to compute numerically:

5 experiments: expected max 1.16 sigma, so +2.6 points at n = 250
10 experiments: 1.54 sigma, +3.5 points
40 experiments: 2.16 sigma, +4.9 points
100 experiments: 2.51 sigma, +5.7 points

Forty tries against a 250-example set buys you an expected phantom improvement of about five points. If your quarterly review says the prompt work moved the pass rate from 85 to 89, the null hypothesis says: that is what selection looks like when nothing improved at all.

Twelve lines to check my arithmetic:

import numpy as np

rng = np.random.default_rng(0)
n, p, k, sims = 250, 0.85, 40, 100_000

# k identical candidates, each measured once on n examples
runs = rng.binomial(n, p, size=(sims, k)) / n
best = runs.max(axis=1)

print(f"true rate: {p:.3f}")
print(f"mean best-of-{k}: {best.mean():.3f}")
print(f"inflation: {(best.mean() - p) * 100:.1f} points")

This prints about +4.7, just under the normal-approximation figure above for k = 40 (the binomial's discreteness and ties shave off a couple of tenths). Change k and re-run: the bias keeps growing, roughly like sqrt(2 ln k), so it slows with persistence but never stops.

The part the field already solved

Dwork, Feldman, Hardt, Pitassi, Reingold, and Roth worked out the general problem in 2015 under the name adaptive data analysis (the reusable holdout line of work; arXiv:1506.02629). Their observation: the moment your next experiment depends on the last measurement, the holdout stops being fresh, and classical guarantees quietly void themselves. Their mechanism, Thresholdout, answers holdout queries through a noise-adding gate so the set survives many more adaptive looks.

You do not need to implement Thresholdout to benefit from the diagnosis. Four practices capture most of the value:

Two sets, two roles. Iterate freely against a dev set. Confirm on a frozen set you touch only when shipping. The confirm set's job is to be boring: it answers a handful of times per quarter, not forty.
Selection-adjusted bars. If you compared k variants, the winner must clear the baseline by more than the expected max of k noise draws, not by more than zero. For k = 40 at n = 250, that bar is about five points, which is sobering and correct.
Count your touches. k is the one quantity in this whole analysis you control and can log exactly. Teams version their prompts and their datasets and keep no record of how many times the eval set has been queried. Make k a first-class number in the eval report.
Retire burned sets. After enough adaptive looks, refresh the eval set from new traffic and re-baseline. Not because the data drifted (that is a separate failure with its own essay) but because your process has memorized this particular sample's noise.

FAQ

Does a bigger eval set fix this? It helps: sigma scales as 1/sqrt(n), so quadrupling the set halves the inflation. It does not change the shape: the bias still grows with k. A bigger set lowers the noise floor. The selection effect sitting on top of it remains.

Is this just p-hacking with a new name? It is the same statistical family. P-hacking selects among analyses of one experiment; this selects among experiments measured on one reused set. The adaptive data analysis literature treats both as instances of the same leakage.

Has anyone measured whether this happens in practice, or is it theory? Measured, with a twist. Recht, Roelofs, Schmidt, and Shankar built a fresh CIFAR-10 test set in 2018 (arXiv:1806.00451) expecting to catch years of test-set reuse as overfitting. The drop they found pointed at distribution shift instead, and adaptivity-driven overfitting was smaller than theory allows. Two honest readings: benchmark-scale selection has correction mechanisms (many teams, public leaderboards, diverse models), and a single team hammering a private 250-example set with forty variants of one prompt is a much purer selection machine than ImageNet ever was. The math above is the worst case; your k and your n decide how close you sit to it.

Open question

Is there a principled touch budget for an eval set: a k beyond which refresh is mandatory, stated in advance the way significance levels are? I have a heuristic (refresh when the selection-adjusted bar exceeds the effect sizes you care about) but I have not seen a treatment that turns it into a rule teams actually adopt. If you have seen one in the wild, I want to read it.

I read the metric libraries of five widely-used eval tools. The metric was never the hard part.

Maya Andersson — Tue, 11 Aug 2026 18:53:06 +0000

Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. The pitch works because it is true, and because it quietly relocates the hard part of evaluation to somewhere you cannot see it.

I spent a week reading the actual metric catalogs of five widely-used eval libraries, not the landing pages. Arize Phoenix, DeepEval, Future AGI, Langfuse, and Ragas. I wanted to know two things. What do you get for free, and what does the free part let you stop thinking about. The answer to the second question is: almost nothing. The metric is the easy 20 percent. The two hard parts, choosing a metric that matches your failure mode and putting an error bar on the result, are still yours, and none of these tools does them for you.

Here is the survey, then the argument.

What five catalogs actually ship

I ordered these alphabetically on purpose. I am not ranking them, and I built a comparison table precisely so I would not be tempted to.

Tool	License	Built-in metrics (documented)	LLM-as-judge	Aggregation
Arize Phoenix	Elastic License 2.0 (source-available)	~16 pre-built evaluators: Faithfulness, Correctness, Q&A, Tool Selection, Hallucination, Toxicity	Yes, judge templates use function calling to extract structured verdicts	Datasets and Experiments: per-example scores, aggregate metrics, cross-experiment comparison
DeepEval	Apache-2.0	"50+" metrics: G-Eval, DAG, Answer Relevancy, Faithfulness, Tool Correctness, Hallucination, Bias	Yes, most metrics are LLM-judged (QAG, DAG, G-Eval)	Pytest-style pass/fail against thresholds; `-r` flag repeats a test case
Future AGI	Apache-2.0	72 local metrics: BLEU, ROUGE, embedding similarity, JSON-schema, plus judged faithfulness and function-call accuracy	Yes, local heuristics augmentable with a judge via `augment=True`	Aggregated metric type; datasets in the platform
Langfuse	MIT (core; `ee/` is commercial)	Managed evaluator catalog: Hallucination, Context-Relevance, Toxicity, Helpfulness (some maintained with Ragas)	Yes, numeric, categorical, or boolean judge scores	Dataset runs scored at aggregate level, side-by-side experiment comparison
Ragas	Apache-2.0	"30+" metrics: Faithfulness, Context Precision, Context Recall, Response Relevancy, Tool Call Accuracy	Yes, LLM-based metrics issue one or more judge calls	`evaluate()` returns mean scores across the dataset; `to_pandas()` for per-sample

A few honest notes before anyone quotes this table at me.

The metric counts are each project's own stated approximation, not a hand count. "50+", "30+", and "~16" are marketing-adjacent numbers from their docs, and "72" is a figure from Future AGI's SDK README. I did not recount anyone's catalog line by line, so treat the column as order-of-magnitude, not a leaderboard. Phoenix is worth a license asterisk: Elastic License 2.0 is source-available, not OSI-approved open source, which matters if you are redistributing, less so if you are just running it. Langfuse is MIT at the core with a separately licensed enterprise directory. The other three are Apache-2.0.

The overlap is the real story. Faithfulness shows up in four of the five. Tool-calling correctness shows up in four. Every one of them supports LLM-as-a-judge, and every one of them returns you a score. If you were choosing on metric coverage alone, you would have a hard time telling them apart, because for the common cases they are converging on the same list. That convergence is a signal. The catalog is becoming a commodity. Which means the catalog is not where your evaluation succeeds or fails.

The hard part number one: the metric that matches your failure

A built-in metric encodes a definition somebody else wrote. "Faithfulness" in a RAG library means the answer is grounded in the retrieved context. That is a real and useful thing to measure. It is also completely silent about whether your agent picked the right document to retrieve in the first place, whether it called the refund tool instead of the lookup tool, or whether it was confidently, groundedly wrong because the context itself was stale.

I have watched teams adopt a library, wire up five of its metrics, get a dashboard full of 0.9s, and ship a regression anyway, because none of the five metrics measured the thing that actually broke. The tool did its job. The scores were accurate. They were just answers to questions nobody's users were asking.

The move that works is unglamorous and the tools cannot do it for you. Write down your failure taxonomy first. Go read fifty real production traces, cluster the ways your system actually fails, and only then go shopping in the catalog for metrics that map onto those clusters. Most of the time you will find that two or three built-in metrics cover your top failure modes and the rest are noise you were about to pay judge-call money to compute. Occasionally you will find that your worst failure mode has no built-in metric at all, and you will write a custom judge. Every one of these five tools lets you write a custom metric, which tells you the authors already know the catalog does not cover you.

So the first hard part is selection, and selection is a function of your traces, not their docs.

The hard part number two: the error bar nobody ships

Here is the finding that sent me down this whole path. I read the aggregation behavior of all five tools carefully, because aggregation is where statistics is supposed to happen. What I found is that all five will happily give you a mean. Ragas returns {'faithfulness': 0.892}. Phoenix and Langfuse show aggregate scores across an experiment. DeepEval gives you a pass rate against a threshold. Future AGI has an aggregated-metric type.

Not one of them, as of July 2026, computes a confidence interval for you by default.

I want to be precise, because this is the kind of claim that gets misquoted into "tool X can't do statistics." That is not what I am saying. I am saying I went looking in the documented, out-of-the-box behavior of each library and did not find confidence-interval computation in any of them, Future AGI included. DeepEval's -r repeat flag gets you closest to the raw material, since you can run a case multiple times, but the docs stop at repeating, not at turning the repeats into an interval. The others aggregate to a mean and stop.

This matters more than the metric-count column everyone stares at. A faithfulness pass rate of 0.88 measured on 50 examples (44 of 50 above threshold) is not 0.88. It is 0.88 with a 95 percent Wilson interval running from roughly 0.76 to 0.94 (Wilson, 1927, the standard small-sample interval for a proportion). If your last release scored 0.82 (41 of 50) on the same 50 and you are about to write "faithfulness improved," you are reading noise as signal. The tool told you 0.82 and 0.88. It did not tell you those two numbers are statistically indistinguishable, because computing that was never its job, and it did not warn you that you were about to assume it was.

Here are the ten lines the catalog leaves to you.

from statsmodels.stats.proportion import proportion_confint

def pass_rate_with_ci(passes, n, alpha=0.05):
    rate = passes / n
    lo, hi = proportion_confint(passes, n, alpha=alpha, method="wilson")
    return rate, (lo, hi)

# 44 of 50 examples passed the metric's threshold
rate, (lo, hi) = pass_rate_with_ci(44, 50)
print(f"{rate:.2f}  95% CI [{lo:.2f}, {hi:.2f}]")
# 0.88  95% CI [0.76, 0.94]

That interval is the difference between "we improved" and "we cannot tell yet, run more examples." For continuous judge scores rather than pass/fail, bootstrap the mean instead of using Wilson, but the discipline is identical: never report a point estimate from an eval set without the interval around it, and never compare two releases whose intervals overlap as if one won.

So what is the tool for

This is not an argument against any of these libraries. I use them. The catalog genuinely saves you from reimplementing BLEU, ROUGE, embedding similarity, and a dozen judge prompts, and standardizing on one of them is a good decision. Phoenix rides OpenTelemetry if that is your stack. DeepEval's pytest ergonomics are the cleanest if you live in CI. Future AGI if you want eval, tracing, and guardrails in one place. Langfuse if you want tracing and evals in one product. Ragas is still the sharpest on RAG. They are all fine, and they are all converging.

The mistake is thinking the tool finished your evaluation. It started it. It computed the metric, which was the part a library can do, and it handed you back a number stripped of the two things that make the number mean anything: whether the metric matches your failure, and how much of the number is noise. Those two stay on your side of the API no matter which logo is on the dashboard.

FAQ

Which of these five should I pick?
Whichever fits your stack and license constraints, because on metric coverage they are close to interchangeable for common cases. Decide on ergonomics (pytest vs platform vs OTel), license (Apache-2.0 vs Elastic vs MIT-core), and whether you need the surrounding tracing and datasets, not on the metric count.

Is a bigger metric library better?
No. A bigger library raises the chance that a metric matching your failure mode exists, but it does nothing to raise the chance that you picked it. Ten well-chosen metrics beat seventy defaults every time.

Do I really need confidence intervals for eval?
If you are making ship/no-ship decisions on the difference between two scores, yes, always. The interval is what tells you whether the difference is real. Ten extra lines of code, and it is the highest-leverage code in your eval suite.

Does any tool do the interval for me?
Not by default, in any of the five I read, as of July 2026. If that changes I will happily update this. Until then, wrap your own.

Open question

If the metric catalog is commoditizing, the differentiation moves up a level, to selection and uncertainty. So why does no eval tool ship a "your last two runs are statistically indistinguishable" warning yet? It is a small feature. It would prevent a large fraction of the false "we improved" claims I see in eval writeups. My guess is that a tool that keeps telling you your result is inconclusive is a harder product to sell than one that always returns a confident number. I would love to be proven wrong by whichever project ships it first.

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

Maya Andersson — Tue, 11 Aug 2026 18:50:05 +0000

A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate drops below 88 percent. Their baseline is around 92. The monitor paged on four separate days last week. Four investigations, four shrugs, four "it recovered on its own."

Nobody chose a false-alarm rate for that monitor. But it has one, and it is not small. With roughly 150 judge scores an hour, a fresh window each check, and a true pass rate of 0.92, the chance that at least one hourly check dips below 0.88 at some point in a day is 53 percent. That makes the expected number of alert days in a week 3.7, and the single most likely weekly count exactly four. Their monitor was not detecting regressions. It was sampling noise on a schedule, and the on-call rotation was the readout.

An alert threshold on an eval score is a hypothesis test. Every tool that ships eval monitoring runs that test for you. What none of the configuration surfaces I read this month asks you for is the one number the test depends on: how many scores are in the window.

The test you are actually running

The setup is ordinary binomial arithmetic, which is what makes it checkable. Each check looks at a fresh window of n judge verdicts. The true pass rate is p. The monitor fires when the observed rate falls below a threshold t. Then:

import math
from scipy.stats import binom

def alpha_per_check(n, p0=0.92, t=0.88):
    """P(false alarm): windowed rate < t although nothing changed."""
    k = math.ceil(t * n) - 1          # largest count that fires
    return binom.cdf(k, n, p0)

def p_fire_today(n, checks=24, **kw):
    a = alpha_per_check(n, **kw)
    return 1 - (1 - a) ** checks      # tumbling windows, independent checks

def power_per_check(n, p1=0.85, t=0.88):
    k = math.ceil(t * n) - 1
    return binom.cdf(k, n, p1)        # P(one check catches a real drop)

Run that for a 0.88 threshold against a 0.92 baseline, across realistic window sizes, and add the probability that a single check catches a genuine regression to 0.85:

Scores per window	Fires per check on noise	Fires some time today	Expected alert days per week	Catches a real drop to 0.85, per check
25	13.5%	96.9%	6.8	52.9%
50	10.2%	92.4%	6.5	63.9%
100	5.6%	74.8%	5.2	75.3%
150	3.1%	53.3%	3.7	81.9%
200	1.8%	35.0%	2.5	86.3%
400	0.2%	4.9%	0.3	94.9%

Read the 25-per-window row twice. That monitor pages nearly every day on pure noise, and a single check still misses a real seven-point regression about half the time. It manages to be trigger-happy and insensitive simultaneously, which is what happens when the threshold is chosen as a round number four points under baseline instead of as a quantile of anything.

The window size is not an abstract parameter. It is your traffic times your sampling rate. Sample 10 percent of 500 traces an hour and n is 50. The sampling knob every tool gives you for cost control is also, silently, the monitor's sensitivity knob: cutting sampling from 100 percent to 10 percent widens the score's confidence interval by a factor of about 3.2, the square root of ten. None of this is exotic. It is the standard behaviour of binomial proportions, catalogued in detail in Brown, Cai and DasGupta's "Interval Estimation for a Binomial Proportion" (Statistical Science, 2001).

So the question I brought to the current tooling was not "who has alerting." Most do. It was three narrower questions.

Question one: which traces does the score come from?

This decides n, so it decides everything above.

Langfuse runs judge evaluators at ingest time on observations that match your filters, with a sampling percentage you set; matching observations enter an evaluation queue and are scored asynchronously. Filters cover observation type, trace name, tags, user, session and metadata, so the scored population is well defined, and the docs are explicit that sampling exists to manage evaluation cost.

LangSmith does the equivalent through automation rules: a filter selects runs, a sampling rate decides what fraction of the filtered runs the evaluator sees, and the scores attach back to the traces.

Opik's online evaluation rules score live production traces with LLM-as-judge metrics and write results back as feedback scores on each trace.

Future AGI's continuous eval tasks take the same two controls; a forward-only cursor means history is never backfilled.

Phoenix evaluates production traces when you run evals over them by SDK or in the app; for continuously scheduled scoring of live traffic its own docs point you to Arize AX, the commercial platform. DeepEval sits at the same boundary from the other side: the framework is offline-first, and production scoring is the companion Confident AI platform's job.

Question two: what turns a score into a page?

Langfuse has the most developed vocabulary here of the set I read. Monitors watch numeric, categorical or boolean scores; you set an operator, an alert threshold and an optional warning threshold over a lookback window, and route through Slack, webhooks or GitHub Actions. It is also the only config I found that makes you decide what an empty window means: treat missing data as zero, hold the previous severity, record NO_DATA silently, or page after sustained NO_DATA. How many monitors you can create depends on plan tier; the features themselves do not.

LangSmith alerts on five metric types, run count, cost, errors, latency and feedback score, that last one being where online eval results live, and it routes to Slack, PagerDuty, Dynatrace or any webhook.

Future AGI's monitors take a static or percentage-change threshold, or learn one from the historical mean, with separate warning and critical levels and a check frequency in minutes.

Opik writes rule scores onto traces and tracks them on dashboards; its online-rules documentation describes score computation and display, and no notification configuration appears on that page.

Phoenix routes threshold-based triggers on production traffic to Arize AX, per its own docs. I did not audit AX's alert configuration, nor Confident AI's, so the two commercial companions stay out of the comparison below.

Question three: where does sample size enter?

I read three alerting configuration surfaces end to end this month: Langfuse's monitor settings, LangSmith's alert settings and Future AGI's monitor model. In all three, the vocabulary is drawn from the same short list: a value, a direction, a window, sometimes a check frequency. None of the three has a field for the number of scores the window is expected to contain, and none derives the threshold from one. The window is specified in time, five minutes, an hour, a day, and how many scores fall inside it is whatever traffic and sampling happen to produce.

Two of the threshold types deserve a specific caution. A percentage-change threshold compares two noisy window estimates, and the difference of two independent windowed rates carries roughly twice the variance of either one, so at small n it is noisier than the static threshold it replaces. And an auto or anomaly threshold that learns the historical mean solves a real problem, baseline drift, but the alert it fires is still a point-estimate comparison; learning where the baseline sits is not the same as knowing how far a healthy window wanders from it.

To be precise about scope: this is what the configuration surfaces expose as of early August 2026, from each vendor's public docs and, where the code is public, its source tree. Any of them could ship a sample-size-aware policy tomorrow, and the gap is easy to work around today, which is the last section.

Setting the threshold from n instead

You do not need any vendor to fix this. Invert the binomial: decide the false-alarm rate you can staff, then compute the threshold your window size implies.

from scipy.stats import binom

def threshold_from_n(n, alpha, p0=0.92):
    """Largest rate threshold whose per-check false alarm stays <= alpha."""
    k = int(binom.ppf(alpha, n, p0))
    if binom.cdf(k, n, p0) > alpha:
        k -= 1
    return k / n

Scores per window	Page below (1% per check)	Warn below (10% per check)
25	0.720	0.800
50	0.800	0.840
100	0.840	0.870
150	0.860	0.887
200	0.870	0.890
400	0.885	0.900

Two things fall out. First, at 25 scores a window, an honest 1-percent page threshold is 0.72, a full sixteen points below the 0.88 the team "felt" was right; the small-n rows are exactly where intuition overshoots most. Second, the pairing maps cleanly onto the warning-plus-critical structure that Langfuse and Future AGI both expose and that LangSmith approximates with two rules: put the warning at the 10 percent quantile and the page at the 1 percent quantile, and both levels inherit a false-alarm ceiling you actually chose.

If your platform only takes time-based windows, fix the count instead: check every N scores rather than every hour. A count-based window makes n a constant, which makes the table above exact instead of approximate, and it stops quiet hours from paging you simply because n collapsed overnight.

FAQ

Is 0.92 a magic baseline? No. Every number above recomputes for your baseline and your window in the four lines of code shown; the shape of the conclusion, thresholds must move with n, survives any realistic parameter choice.

My monitor uses a rolling window checked every five minutes. Does the daily 53 percent still apply? Not directly, and the direction matters: the five-minute checks include the hourly windows among them, so the daily probability is at least the 53 percent, not less. Plugging 288 checks into the independence formula overstates it badly, though; overlapping windows are correlated, the per-check false alarm is unchanged, and correlated checks cluster their false alarms into the same bad hour.

Why not just require two consecutive breaches? That is a legitimate sample-size-aware policy, and on tumbling windows it squares the per-check false-alarm rate. You pay in detection delay, one extra check period, and in power against short-lived regressions. It is the cheapest fix on this page; the quantile threshold is the principled one.

Open question

Auto thresholds that learn the baseline mean already read the score history. The same history contains everything needed to learn the window-to-window variance and set the alert line at a chosen quantile of it, which would make the monitor's false-alarm rate a configured property instead of an accident of traffic. None of the docs I read says whether any of the auto modes does this today. If someone from one of these teams can point me at a sample-size-aware or variance-aware alert policy in their product, mine is exactly the kind of dashboard it would quiet, and I will happily run the comparison again.

Sources for the tool claims, all read in the first week of August 2026: Langfuse docs, "Monitors and Alerts" and the LLM-as-a-judge evaluator pages; LangSmith docs, "Online evaluations" and "Alerts"; Opik repository README and its online evaluation rules doc (comet-ml/opik); Phoenix repository docs, llm-evals page (Arize-ai/phoenix); DeepEval repository README (confident-ai/deepeval); Future AGI repository, tracer models eval_task.py and monitor.py (future-agi/future-agi). The binomial arithmetic is standard; see Brown, Cai and DasGupta, "Interval Estimation for a Binomial Proportion," Statistical Science 16(2), 2001.

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

Maya Andersson — Fri, 07 Aug 2026 19:01:00 +0000

The test you are actually running

import math
from scipy.stats import binom

def alpha_per_check(n, p0=0.92, t=0.88):
    """P(false alarm): windowed rate < t although nothing changed."""
    k = math.ceil(t * n) - 1          # largest count that fires
    return binom.cdf(k, n, p0)

def p_fire_today(n, checks=24, **kw):
    a = alpha_per_check(n, **kw)
    return 1 - (1 - a) ** checks      # tumbling windows, independent checks

def power_per_check(n, p1=0.85, t=0.88):
    k = math.ceil(t * n) - 1
    return binom.cdf(k, n, p1)        # P(one check catches a real drop)

Run that for a 0.88 threshold against a 0.92 baseline, across realistic window sizes, and add the probability that a single check catches a genuine regression to 0.85:

Scores per window	Fires per check on noise	Fires some time today	Expected alert days per week	Catches a real drop to 0.85, per check
25	13.5%	96.9%	6.8	52.9%
50	10.2%	92.4%	6.5	63.9%
100	5.6%	74.8%	5.2	75.3%
150	3.1%	53.3%	3.7	81.9%
200	1.8%	35.0%	2.5	86.3%
400	0.2%	4.9%	0.3	94.9%

So the question I brought to the current tooling was not "who has alerting." Most do. It was three narrower questions.

Question one: which traces does the score come from?

This decides n, so it decides everything above.

LangSmith does the equivalent through automation rules: a filter selects runs, a sampling rate decides what fraction of the filtered runs the evaluator sees, and the scores attach back to the traces.

Opik's online evaluation rules score live production traces with LLM-as-judge metrics and write results back as feedback scores on each trace.

Future AGI's continuous eval tasks take the same two controls; a forward-only cursor means history is never backfilled.

Question two: what turns a score into a page?

Future AGI's monitors take a static or percentage-change threshold, or learn one from the historical mean, with separate warning and critical levels and a check frequency in minutes.

Opik writes rule scores onto traces and tracks them on dashboards; its online-rules documentation describes score computation and display, and no notification configuration appears on that page.

Question three: where does sample size enter?

Setting the threshold from n instead

You do not need any vendor to fix this. Invert the binomial: decide the false-alarm rate you can staff, then compute the threshold your window size implies.

from scipy.stats import binom

def threshold_from_n(n, alpha, p0=0.92):
    """Largest rate threshold whose per-check false alarm stays <= alpha."""
    k = int(binom.ppf(alpha, n, p0))
    if binom.cdf(k, n, p0) > alpha:
        k -= 1
    return k / n

Scores per window	Page below (1% per check)	Warn below (10% per check)
25	0.720	0.800
50	0.800	0.840
100	0.840	0.870
150	0.860	0.887
200	0.870	0.890
400	0.885	0.900

FAQ

Open question

Upgrading the judge ends one score series and starts another

Maya Andersson — Thu, 06 Aug 2026 19:17:16 +0000

There is a mature literature on what happens when you swap one measuring instrument for another, and it is not in machine learning.

The standard treatment is Bland and Altman, "Statistical methods for assessing agreement between two methods of clinical measurement", Lancet 1986; i: 307-310. Their core move is the one worth borrowing: do not correlate the two instruments. Plot their difference against their average, and look at the shape.

Correlation is the wrong tool because it measures association rather than agreement, and it inflates with the spread of whatever sample you happened to use. Two instruments can correlate at 1.00 while one of them reads a fifth of the scale high everywhere it has room to. A constant offset is a perfect linear relationship, so correlation reports agreement that is not there.

An LLM-as-judge is an instrument. When we replaced ours with its successor and the weekly quality score moved from 0.78 to 0.83 in the same week, we had changed the system being measured and the thing measuring it at once, and the honest position was that we did not know which had moved. Our scores run 0 to 1.

The usual response is to estimate an offset and subtract it. That is a reasonable instinct and it is also a model, with assumptions, and the assumptions are checkable. Three checks, and the third one I got wrong the first time I ran it.

The apparatus

An anchor set: a fixed collection of (input, output) pairs, with the outputs frozen, scored under both judges. Paired, same items, same text, two raters.

Ours had 60 items. On it, sd(d) was 0.12, where d is the per-item difference, new judge minus old. Both judges had a score sd of 0.22 on the anchor set, so the ratio sd(new)/sd(old) came out at 1.0 to the precision I can report from two figures rounded to two decimals. That ratio matters more than it looks, and check two is where I explain why. The scores themselves are continuous, a weighted rubric rolled up to 0 to 1, not a grid of discrete levels; that turns out to matter for check three.

Check one: the offset, and how badly you know it

The offset is the mean of d, and the useful number is not the offset but its confidence interval:

half-width = 1.96 x sd(d) / sqrt(n)

Rearranged, that gives the anchor set size you need for a given precision:

±0.05 on the offset: 10 items at sd(d) = 0.08, 23 at 0.12, 50 at 0.18
±0.03 on the offset: 28 items at sd(d) = 0.08, 62 at 0.12, 139 at 0.18
±0.02 on the offset: 62 items at sd(d) = 0.08, 139 at 0.12, 312 at 0.18
±0.01 on the offset: 246 items at sd(d) = 0.08, 554 at 0.12, 1,245 at 0.18

That table uses z = 1.96, the large-sample form. Under the t distribution, which is what Bland and Altman specify because sd(d) is estimated rather than known, every cell rises by two or three items. Proportionally it only matters at the top: the 10 becomes 13 and the 23 becomes 25, while the 554 becomes 556.

At 60 items and sd(d) = 0.12, our offset is pinned to ±0.030.

Now put the two numbers next to each other. We were trying to detect a product improvement of about 0.05, using a correction whose own uncertainty was ±0.030. The error bar on the correction was about 60% of the thing being measured. Subtracting that offset relabels the confounding rather than removing it, and it does so in a way that looks precise because a single number has been subtracted.

Note which interval that is, because I conflated the two for longer than I would like. The ±0.030 is the precision of the mean offset, which is the right number for correcting an aggregate. Bland and Altman's headline statistic is the limits of agreement, mean ± 1.96 x sd(d), which for us is ±0.235, and at 60 items each of those limits is itself uncertain by roughly ±0.05. Any individual rescaled score carries the wide interval, not the narrow one. If you plan to correct a per-item score and act on it, you are working with something close to eight times the width you computed.

A word on why the paired design is worth the trouble, since the reason usually given is wrong. Scoring the same frozen outputs twice does not merely make sd(d) small. Against two disjoint samples of frozen outputs, one per judge, the paired design needs 2/(1 - rho) times fewer items and 1/(1 - rho) times fewer judge calls, so it wins on items even when the two judges are uncorrelated. What rho buys is the judge calls. Our sd(d) of 0.12 against a score sd of 0.22 implies rho of about 0.85, and at ±0.01 that is 3,719 items per arm unpaired, 7,438 in total, against 554 scored by both judges: thirteen times the items, or 6.7 times the judge calls.

(The identity sd(d)² = 2 x sd(score)² x (1 - rho) assumes both judges have the same score sd, which ours did. If yours do not, use rho = (sd_new² + sd_old² - sd_d²) / (2 x sd_new x sd_old), because the shortcut will quietly mis-state rho and everything downstream of it.)

Check two: did the spread change, or only the location?

Subtracting an offset assumes the new judge is the old judge shifted. In what I have seen, it is at least as common for a new judge to be compressed: more agreeable, more clustered in the middle, fewer confident lows. The mean moves and the standard deviation shrinks.

The diagnostic is the ratio sd(new) / sd(old) on the anchor set, plus the difference-against-average plot. If the difference d trends with the average score, or fans out at one end, the relationship is not a shift and no single constant will fix it.

Ours came back at 1.00, which is the boring answer and the one that let me use the simple rho identity above. When it does not come back near 1, the consequence is specific: a compressed judge attenuates every future comparison toward zero, your real improvements shrink, your real regressions shrink, and the series goes quiet. This is the same shape as the attenuation I wrote about last week, arriving by a different route. There the effect shrank because a binary judge's flips are bounded. Here it would shrink because the judge's scale is narrower.

I owe this check the same precision arithmetic I gave check one, and I have not done it. There is a proper paired-variance test (Pitman and Morgan's, which works by testing the correlation between the sum and the difference), and I have not worked out what interval 60 items buys on the ratio. So the point estimate is 1.00 and I am not going to pretend I know how tightly. I would rather say that than quote a number I have not derived.

Check three: did the ranking change, and what would "unchanged" have looked like?

The offset and the spread are both about the scale. The third question is about order: does the new judge rank items the way the old one did? Kendall's tau or Spearman's rho on the pairs.

Here is the mistake I made, and it is the reason this section exists.

I computed tau, got 0.65, and read that as substantial rank movement. It is not. Before reading tau you have to work out what tau you would get if nothing but noise had changed. For jointly normal scores, tau = (2/pi) x arcsin(rho), and our implied rho of 0.85 gives tau = 0.6483 on its own. So the gap I thought I was reading was 0.002, which at 60 items is not a gap at all. The arcsin value is the floor and I had mistaken it for a finding.

And then the uncomfortable corollary, which is the reason I now treat this check as the weakest of the three. Under joint normality, rho and tau carry the same information, so comparing observed tau against the rho-implied floor is not really a rank-change detector at all. It is a test of the normality and linearity assumption. It fires when the relationship between the judges is monotone but curved, or when a handful of extreme pairs are propping up the Pearson correlation.

One caveat on the floor itself, which I got wrong on the first attempt. The arcsin identity is a continuous-normal result. Our anchor scores are continuous, so it applies. If yours sit on a discrete rubric grid, ties break it in both directions at once: tau-a is dragged down by tied pairs, while tau-b's tie correction over-restores and lands above the continuous value. On a grid the arcsin number is simply the wrong null, and the floor has to be recomputed on your actual grid before any of this means anything. Say which variant you computed, too.

So the check is worth running, and what it earns you is a reason to look harder at the difference-against-average plot. It does not on its own earn you the conclusion that you have two series.

When you do conclude the ranks have moved, it decides whether splicing is possible. If ranks are preserved, some monotone transformation maps one scale to the other, and with enough anchor items you can estimate it. If they have moved, there is no item-level rescaling that makes a score from March comparable to a score from August. You can still equate the two score distributions and keep the aggregate series roughly comparable, at the cost of no individual rescaled score being trustworthy.

Anchor-set construction: freeze the outputs

The outputs must be frozen artefacts, not re-generated.

It is tempting to take last quarter's 200 eval inputs, run today's system on them, and score that with both judges. It feels like the same thing. It is not: the model, the prompt and the retrieval index have all moved, so the paired differences now contain both judge drift and system drift, and the whole point of the exercise was to separate those. Store the text, score the text, and accept that the anchor set stops being a test suite the moment you freeze it.

When the ranks have moved

I have found only one honest option, and it is expensive: run both judges in parallel on a sample of traffic for a transition window, publish both series side by side, and let the old series end rather than pretending it continues. Annotate the chart at the switch date. Anyone who reads across the boundary is then doing so knowingly.

The alternative, which I have watched happen, is that the offset gets subtracted, the chart stays smooth, and six weeks later a genuine regression is invisible because it is the same size as the correction nobody re-examined.

FAQ

How often does the judge actually change? More often than the calendar suggests. A version pin protects you from silent provider updates but not from your own team changing the rubric text, the temperature, the few-shot examples, or the parser. Every one of those is a new instrument by this argument, and the rubric edit is the most frequent and least announced.

Can I skip this if I only look at relative comparisons within a week? Partly. The judge's offset cancels out of a comparison where both arms were scored by the same judge in the same window. Its scale does not: a compressed judge attenuates the contrast by the same factor in both weeks, so the comparison stays valid in sign and direction but not in magnitude. That is still a good reason to structure reporting around within-week contrasts rather than a long-running level, because the level is where the trouble is worst, and levels are what end up on dashboards.

Is 554 anchor items realistic? For ±0.01, at our sd(d), that is what the arithmetic says. Most teams do not need ±0.01, and the table above lets you buy the precision you actually need.

Does a deterministic metric avoid all of this? For the metric itself, yes. Exact match does not drift. But most people are not upgrading exact match, they are upgrading the thing they use because exact match was inadequate, and that thing has a version.

Open question

I have been assuming the anchor set is representative of production traffic, and everything above inherits that assumption. But an anchor set is frozen by construction, and traffic is not. Six months in, the anchor set describes a distribution that no longer arrives, so the offset you measured is the offset on old traffic, which may not be the offset on current traffic.

Refreshing the anchor set restores representativeness and destroys comparability with everything measured before the refresh, which is the same problem one level up.

I do not have a clean answer. The best I have managed is overlapping anchor sets, retiring and adding a slice at a time so consecutive versions share most of their items, which is the common-item nonequivalent-groups design from the test-equating literature (Kolen and Brennan, Test Equating, Scaling, and Linking). Whether that is worth the machinery for an internal quality metric, I genuinely do not know.

A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.

Maya Andersson — Wed, 05 Aug 2026 19:28:06 +0000

Most of what I have written about LLM judges argues in one direction: your improvement is probably not real. Small eval sets, selection across many experiments, position bias, clustered examples. All of that pushes toward scepticism and I still believe it.

This post argues the other way, because there is a failure mode that runs opposite and I do not see it discussed. Judge noise biases your measured improvement toward zero. The interval gets wider, which everyone expects, and the point estimate also moves, systematically, in the same direction every time. A change that genuinely helped can read as flat, you will record a null result, and the noisier your judge the more confident that mistake becomes.

This is old statistics wearing new clothes. Spearman named it attenuation in 1904: an observed correlation is bounded above by the geometric mean of the reliabilities of the two instruments measuring it, so unreliable measurement makes real relationships look weaker than they are ("The proof and measurement of association between two things", American Journal of Psychology 15, 72-101, whose section II.2 is titled "'Attenuation' by errors"). An LLM judge is an instrument with reliability well under 1, and we report its output as though it were a ruler.

The arithmetic, which is embarrassingly simple

Take a binary judge: pass or fail. Suppose its flip probability is the same conditional on either true label, so it is as likely to turn a true pass into a reported fail as the reverse. Call that flip probability e.

If a model's true pass rate is p, the rate your judge reports is:

p_observed = p(1 - e) + (1 - p)e
           = e + p(1 - 2e)

Now measure two models, true rates p_A and p_B, with the same judge. Subtract:

p_observed(B) - p_observed(A) = (1 - 2e) * (p_B - p_A)

The e term cancels. What survives is a multiplier on your effect: every measured gap is the true gap times (1 - 2e).

That factor bites at values of e that are entirely ordinary:

Judge flip rate e	Disagreement with your labels	Attenuation factor	A true 5-point gap measures as
0.00	0%	1.00	5.0 points
0.10	10%	0.80	4.0 points
0.15	15%	0.70	3.5 points
0.20	20%	0.60	3.0 points
0.25	25%	0.50	2.5 points

I checked the algebra against simulation rather than trusting it: 6,000 simulated eval runs of 1,600 examples each, true rates 0.70 and 0.75, e = 0.15, mean observed gap 3.5 points against a predicted 3.5. Note what that does and does not establish. The bias is the same at every eval-set size, so it confirms the algebra and says nothing about how many examples you need. That question comes next, and it is worse than the table suggests.

It costs you twice, and the second cost is a noise term with two sources

Shrinking the effect is the visible cost. The second depends on which test you are running, and I had it wrong at first.

The intuitive story is that noise pushes both observed rates toward 0.5, since p_observed - 0.5 = (1 - 2e)(p - 0.5), and 0.5 is where per-example variance is greatest. That is true and it is the right picture to hold. But it is the noise term of an unpaired comparison, and unpaired is the wrong test here. Eval examples are not independent draws: some prompts are hard for every system, that shared difficulty correlates the two arms, and I have argued before that comparing two runs by their average pass rate is the wrong test for exactly this reason.

So what does judge noise do to a paired comparison? The arithmetic is prettier than I expected. Write the reported label as A' = A + F_A(1 - 2A), with F_A a Bernoulli flip at rate e independent of everything else. Expand E[A'B'] and E[A']E[B'] and the cross terms are identical, so they cancel:

Cov(A', B') = (1 - 2e)^2 * Cov(A, B)

The factor that attenuates your effect appears squared in the covariance between arms. That is an identity rather than a simulation result, and the ratio does not depend on how correlated your arms were to begin with. I checked it at three different difficulty spreads and got covariance ratios of 0.495, 0.490 and 0.493 against a predicted (1 - 2e)^2 = 0.490, with the correlation itself falling from 0.27 to 0.12 in the middle case.

It is worth saying up front what that identity is for, because it does more work later than it looks like it is doing here. Pairing helps you only through the covariance, so a judge flipping labels at rate e keeps exactly (1 - 2e)^2 of whatever pairing was buying you. That one fact is why every number in this post moves when your examples get more or less uniform in difficulty, and it is the last section of this piece.

Which is why there are two costs on the statistic and not three. McNemar's test looks only at the discordant pairs, so its noise term is the discordant-pair rate, and that rate is

p_discordant = p_A + p_B - 2 p_A p_B - 2 Cov(A, B)

so judge noise inflates it from both directions at once: the marginal rates slide toward 0.5, and the covariance is destroyed. I decomposed it one factor at a time, in both orders, and they come out close to even: of the total inflation, about 48 percent is the marginal shift and about 52 percent is the covariance loss. Which means the "toward 0.5" intuition is not superseded by the covariance identity, it is one of two roughly equal ingredients in the same noise term. On my model that rate went from 0.294 to 0.400 at e = 0.15, and the whole effect on the test statistic is:

z' / z = (1 - 2e) * sqrt(p_discordant / p_discordant')
       = 0.70 * sqrt(0.294 / 0.400)  =  0.60

Two factors on the statistic: the effect shrinks, and the discordance inflates. What I got wrong at first was adding a third, by counting the marginal-variance story separately from the discordance it is half responsible for.

So here is the power calculation for the design this actually is: a per-example difficulty term, marginal pass rates calibrated to 0.70 and 0.75 (0.699 and 0.749 after calibration), McNemar's exact test on the discordant pairs, 6,000 simulated runs per cell.

Eval set size	Power, perfect judge	Power, judge at e = 0.15
200	0.21	0.10
400	0.41	0.19
800	0.72	0.32
1,600	0.95	0.58
3,200	1.00	0.87

Those cells carry about a point of Monte Carlo error, so read them to two figures and no further. To reach 80 percent power on that effect I needed roughly 1,000 examples with a perfect judge and roughly 2,750 with the noisy one, about 2.75 times the eval set to buy back what a 15 percent flip rate cost.

At n = 800, a size that felt generous to me for years, that puts a real 5-point improvement below a one-in-three chance of surfacing. Most teams who try that change will shelve it on the evidence.

One caveat, because that table has a knob in it I nearly published without stating. Everything above assumes one level of between-arm correlation, which is to say one answer to how much your examples vary in difficulty. Change it and the whole table moves: at n = 800 a perfect judge's power runs from 0.59 at uncorrelated arms to 0.90 at a correlation of 0.57.

I expected the ratio between the two columns to be the stable thing. It is not, and it moves in the direction that makes this worse rather than better. Eval-set sizes for 80 percent power, across the same range:

Between-arm correlation	Perfect judge	Judge at e = 0.15	Ratio
0.00	1,254	2,888	2.3x
0.10	1,129	2,763	2.5x
0.27	918	2,551	2.8x
0.45	693	2,327	3.4x
0.57	544	2,178	4.0x

Those are closed-form rather than simulated, so the trend is exact rather than approximate. The first row is worth checking by hand, because at zero correlation pairing buys nothing and McNemar reduces to the unpaired test: the ratio there is just the variance inflation times the attenuation term, 1.13 * 2.04 = 2.3, which is the two factors from the previous section multiplied together.

So the more your examples differ in difficulty, the more a noisy judge costs you, and the covariance identity says why: pairing pays off precisely when the arms are correlated, and (1 - 2e)^2 is the fraction of that payoff you keep. At a correlation of 0.57 a perfect judge clears 80 percent power on about 540 examples and a judge flipping 15 percent of labels needs almost exactly four times as many. The 2.75x I quoted above is the middle of that range, not a constant. Measure your own arm correlation, which you can do directly from a paired run, and read the ratio off the row you land on.

What I do about it

Estimate e, and be honest about what you are estimating. You probably have a human-labelled set from validating the judge. What that gives you is the disagreement between two fallible instruments, which is a different quantity from e. If your humans flip at rate e_h and the judge at e_j, then for the observed disagreement d:

1 - 2d = (1 - 2*e_h) * (1 - 2*e_j)

I confirmed that on four million simulated labels and it holds to four decimal places. Which means dividing your gap by (1 - 2d) over-corrects by exactly 1/(1 - 2*e_h). Humans flipping 5 percent of labels inflates your effect by 11 percent; at 10 percent human error you are inflating by 25 percent. Two annotators agreeing does not make them right, which I have written about before, and it is precisely the assumption this correction smuggles in. So treat the corrected number as an upper bound unless you truly believe your labels are error-free.

Report the disattenuated effect only next to the raw one. Divide the observed gap by (1 - 2e): an observed 3.5 points at e = 0.15 implies a true gap near 5. The uncertainty is computable and belongs in the report. The standard error on (1 - 2e) from m human labels is 2*sqrt(e(1-e)/m), so at e = 0.15 with 100 labels the correction factor alone carries about 20 percent relative error, putting your corrected 3.5 points somewhere near 4.2 to 6.3 before any sampling error in the gap itself. At 500 labels that tightens to roughly 4.6 to 5.5.

Spearman's own worked example in the 1904 paper takes an observed 0.38 and corrects it to approximately 1, which he calls absolute and complete. The abuse is as old as the method.

Check whether you have noise or bias, and use the right null. Attenuation assumes the judge's errors do not depend on what you are measuring. If it prefers longer answers and model B is more verbose, that is a confound with a direction, and dividing by reliability amplifies it. I will not re-derive that argument here beyond the one thing people get wrong in code: the expected split of the disagreements is not even. Under symmetric noise E[flip_up] = (1-p)*e*n and E[flip_down] = p*e*n, so the expected ratio is (1-p) : p. At a pass rate of 0.70, perfectly symmetric noise produces a 30:70 split, and any test against 50/50 will call that lopsided and wrongly tell you not to correct. I ran exactly that mistake against simulated symmetric noise at pass rates from 0.5 to 0.9 and it produced a false "do not correct" verdict at every rate above 0.5.

import numpy as np
from scipy.stats import binomtest

def diagnose(human, judge, alpha=0.01):
    """human, judge: equal-length boolean arrays of pass/fail labels.
    Returns e_hat, its standard error, and whether the disagreement is
    consistent with symmetric noise (attenuation) rather than a directional bias."""
    human, judge = np.asarray(human, bool), np.asarray(judge, bool)
    m = human.size
    flip_up   = int((~human & judge).sum())      # human fail, judge pass
    flip_down = int((human & ~judge).sum())      # human pass, judge fail
    e_hat = (flip_up + flip_down) / m
    se_e  = np.sqrt(e_hat * (1 - e_hat) / m)

    # The null is the HUMAN FAIL RATE, not 0.5: symmetric noise flips
    # (1-p) of the failures up and p of the passes down.
    n_disagree = flip_up + flip_down
    pval = binomtest(flip_up, n_disagree, 1 - human.mean()).pvalue if n_disagree else 1.0
    return {"e_hat": e_hat, "se_e": se_e, "attenuation_ok": pval > alpha, "p_value": pval}

rng   = np.random.default_rng(0)
truth = rng.random(20_000) < 0.70                          # symmetric noise, e = 0.15
judge = np.where(rng.random(20_000) < 0.15, ~truth, truth)
print(diagnose(truth, judge))    # e_hat ~ 0.15, attenuation_ok True

Do not assume cutting e beats buying examples. I believed this and it is wrong. Going from e = 0.15 to e = 0.08 is worth about 1.7 times the eval set, and at a fixed budget doubling the examples beat cutting e at every size I tested: at n = 800 that is 0.57 power against 0.49. The honest reason to attack e is that tightening a rubric or splitting a compound criterion is usually cheaper than labelling another 800 cases, and it also recovers some of the pairing benefit noise destroyed. Both interventions are legitimate. And the sample-size requirement scales as the inverse square of (1 - 2e), not of e, which is worth getting right before quoting it at anyone.

Stop reading a flat result as evidence of no effect unless you know your power at the effect size you cared about. A null with unknown power carries no information, and attenuation biases even that absence toward the conclusion you are about to draw.

FAQ

Does this apply to continuous scores, not just pass or fail?
Not in the same way, and this is the part I had wrong until I simulated it. Under classical additive error, where the measured score is the true score plus independent noise, a raw mean difference is not attenuated at all, because the noise has mean zero. What attenuates is a correlation or a standardised effect size, and there the correction is division by the square root of the reliability when the other variable is measured without error, or by the geometric mean of both reliabilities when neither is. The binary case above is not a special case of that. It attenuates the raw gap precisely because a flip is bounded rather than additive, so the expected judge score is 0.64 where the true rate is 0.70. Identify your error process before reaching for a correction.

My judge disagrees with humans 5 percent of the time. Am I fine?
Mostly. The factor is 0.90, so you lose about a tenth of every effect. Worth knowing when you are arguing over a 1-point difference, not worth restructuring around. Confirm the 5 percent was measured on traffic-like cases rather than your cleanest examples.

Can I average several judge calls and skip the maths?
Averaging independent calls reduces the random component of e, so it is a real intervention, and it recovers some of the lost pairing correlation as well. The systematic component is untouched by averaging, and it was a bias rather than attenuation to begin with.

Does a paired test fix it?
No, and that is the finding that surprised me most. Pairing remains the right design and it still helps. But the (1 - 2e) multiplier is a property of the instrument rather than of the comparison, so pairing cannot reach it, and per-judgment noise actively erodes the between-arm correlation pairing depends on.

Were my past null results wrong?
Some, and you can work out which. Take the effect you were hoping for, your e, and your eval-set size, and compute the power you actually had. Any null where power was under about 0.5 tells you nothing in either direction. Three of mine qualified.

Open question

The awkward case is a judge whose error rate depends on what it is reading. Everything above assumes one e applied to both arms. If model B produces output that is easier to grade correctly then e_B < e_A, the arms attenuate by different factors, and the observed gap mixes a real quality difference with a difference in how legibly each model fails. Those two are not separable from the judge's output alone.

Estimating e per model on a small human-labelled slice was my first attempt, and at the sample sizes I could afford the two estimates were too noisy to propagate, which the standard error above explains: separating a 3-point difference in e needs a couple of thousand labels per arm, not the few hundred I assumed when I started. If you have a design that identifies model-dependent judge reliability without a full human pass over both arms, I would like to read it.

Your eval's confidence interval assumes independent examples. Yours are clustered.

Maya Andersson — Tue, 28 Jul 2026 21:24:36 +0000

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't.

I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong.

Why clustering shrinks your real sample size

Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half.

The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately:

Deff = 1 + (m̄ - 1) · ICC

where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is:

n_eff = n / Deff

That is the number of independent examples your clustered set is actually worth.

The number

Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher).

Deff  = 1 + (5 - 1) · 0.3 = 2.2
n_eff = 200 / 2.2 ≈ 91

Your 200-example eval is worth about 91 independent examples. The correct confidence interval is √2.2 ≈ 1.48 times wider than the naive one. So the interval you proudly reported as plus or minus 3.5 points is really plus or minus 5.2. The 4-point improvement you shipped last sprint, the one that "cleared the CI," may not clear the corrected interval at all.

And note the direction of the error. Clustering never makes your interval too wide. It always makes it too narrow. So the bias is always toward false confidence, toward shipping a change that didn't actually beat baseline.

The three clusterings I check for

Source clustering. Multiple examples generated from or grounded in the same document, table, or context. This is the big one for RAG evals, where synthetic questions are minted per-document. ICC here is often 0.2 to 0.4.
Conversation clustering. Multiple turns scored from the same multi-turn session. Turns within a session share the same user, same goal, same accumulated context, so they correlate hard. Scoring 10 turns from 20 conversations is not 200 independent points.
Template clustering. Paraphrases or perturbations of the same underlying case. If you augmented 50 seed cases into 150 by rewording, your n is closer to 50 for the purpose of the interval. The rewordings measure robustness to phrasing, not 150 independent capabilities.

The fix: resample clusters, not rows

The cleanest correction that does not require you to estimate ICC by hand is a cluster bootstrap. Instead of resampling individual examples (which assumes independence, reintroducing the exact bug), you resample whole clusters with replacement.

import numpy as np

def cluster_bootstrap_ci(scores, cluster_ids, B=5000, alpha=0.05, seed=0):
    rng = np.random.default_rng(seed)
    clusters = {}
    for s, c in zip(scores, cluster_ids):
        clusters.setdefault(c, []).append(s)
    keys = list(clusters)
    means = np.empty(B)
    for b in range(B):
        drawn = rng.choice(keys, size=len(keys), replace=True)   # resample CLUSTERS
        pooled = [s for k in drawn for s in clusters[k]]
        means[b] = np.mean(pooled)
    lo, hi = np.percentile(means, [100*alpha/2, 100*(1-alpha/2)])
    return float(lo), float(hi)

Resampling at the cluster level automatically bakes in the correlation structure. You do not have to estimate the ICC. The interval it returns is the honest one, and it will be wider than the binomial interval your CI currently prints. The wider interval is not an artifact of the bootstrap. The correlation in your data was always there, and this is just the first interval that stops ignoring it.

FAQ

What if my examples really are independent? Then the cluster bootstrap and the binomial interval will roughly agree, and you have lost nothing by checking. The check costs a few seconds of compute. Shipping a change that never actually beat baseline costs a lot more than that.

Do I need to know the ICC? No. The cluster bootstrap sidesteps it. You only need the design-effect formula if you want a back-of-envelope sense of how bad the problem is before you code anything.

Does this change my point estimate? No. The mean pass rate is unchanged. Only the interval around it widens. This is purely about how much you should trust the number, not the number itself.

Is this the same as just using a bigger eval set? No, and this is the trap. Adding 50 more questions from the same 8 documents barely helps, because you are adding within-cluster examples that carry little new information. To tighten a clustered interval you need more clusters (more documents, more conversations), not more examples per cluster.

Open question

I do not have a good rule of thumb for the minimum number of clusters before a cluster bootstrap itself becomes unstable. With very few clusters (say under 15) the bootstrap distribution gets lumpy and the interval is itself uncertain. I have been using a soft floor of 20 clusters and falling back to reporting the design-effect-adjusted interval below that, but I have not seen a principled threshold for eval-sized data. If you know the literature here better than I do, point me at it.