I wanted to disagree with 'AI made me a worse reviewer' from Michael Amachree (@dev_michael) . Instead I counted 204 of my own guards — and 89 % of them have never been asked to prove they can fail.
Michael wrote something that I couldn't put down: AI didn't make me a worse coder, it made me a worse reviewer. Here is the number, and it's worse than his thesis: of the 204 automated checks in my repositories that draw a conclusion, only 22 can prove they are able to fail. That's 11 %. The other 89 % have never once been shown a known-bad input. They are green. Whether they are green because everything is fine, or green because they are incapable of finding anything - I could not have told you last week. And I'm the person who wrote them.
What I actually counted
First the definition, so you can reject it or reuse it.
A conclusion-bearing guard is any test that reads source code, config, or system state and asserts a claim about it. Not "does this function return 4" - but "no workflow downloads its cache over the network", "every page passes the same quarter filter", "this feature flag matches the deployed spec". The tests that stand in for a human reviewer.
A negative control is a probe that feeds that guard a known-bad input and asserts it gets rejected for the expected reason. Our convention marks them KONTROLLE: in the test name.
Counting is mechanical: 204 guard files across three repositories, 22 with at least one control probe, 54 probes total. The counter is a proxy - marker-based, so unmarked controls and false-positive guard files put the true number at plus or minus a few points. The shape survives any correction: most of my reviewers have never been reviewed.
Three green-and-blind checks, one ordinary week
This isn't theoretical. All three of these happened to me in the last seven days, in production tooling.
The deploy gate that died of its own medicine. A pipeline step existed specifically to catch a silent failure mode - a missing tool falling back to an empty result. It called node -e to parse a health response. The deploy runner has no Node. Six consecutive deployments failed with exit 127 - the check against missing tools failed on a missing tool, and nothing shipped for six hours. The step had been green in review because nobody had ever run it where it actually runs.
The harvester that threw away its own work. An autonomous job collected data from public repositories and judged each run by exit code. One run wrote seven perfectly good records, then hit a non-fatal warning and exited non-zero. The machine booked its own completed work as "failed, retry later" - because interrupted-with-partial-results had no representation, only success and failure. We caught it because the result file was sitting on disk right next to the exit code that denied its existence.
The pattern that matched the wrong 500. An error classifier looked for server errors with the pattern 50[024] - anywhere in the output. It matched the "500" inside "4258 of 5000 quota points remaining" and classified a successful run as a server failure. Every field it read was real. It was answering a different question than the one asked.
Three different systems. One shape: the check watched a messenger - an exit code, a pattern, a status - while the artifact that mattered told a different story.
What this has to do with AI making you a worse reviewer
Here's where I think Michael's post lands harder than he says.
AI moved my job. I used to spend most of my day producing artifacts and a little of it verifying them. Now an agent produces most of the artifacts, and my job is verification. Which means my real codebase - the one my judgment actually ships through - is those 204 guards.
And that codebase is held to a standard I would reject in application code. No test coverage (11 %). No review of the reviewer. Green as the default state, silence booked as success.
When Michael says AI made him a worse reviewer, I'd sharpen it: AI promoted us all to reviewers, and none of us tested the reviewer. The model isn't the weak link. The unfalsifiable green checkmark is.
The rule that survived the week
Everything above collapses into one sentence we now apply mechanically:
Judge the artifact, not the messenger.
Exit codes are messengers. Summaries are messengers. The agent's own "done" is a messenger. Green badges are messengers. The artifact is the diff, the file on disk, the served response body, the row in the database. When a messenger and an artifact disagree, the artifact is right - and a check that only ever reads messengers should be treated as unverified, however green it is.
The corollary for guards: a green zero is the most dangerous answer a check can give. "Found no violations" and "is incapable of finding violations" produce identical output. Only a negative control separates them.
Count your own ratio (60 seconds)
This is the part you can use without believing me. Drop this in your repo root - it counts test files that read source or state, and how many carry a marked negative control (adjust the marker to your convention):
// count-controls.mjs — node count-controls.mjs
import { readFileSync, readdirSync, statSync } from "node:fs";
import { join } from "node:path";
const files = [];
(function walk(d) {
for (const n of readdirSync(d)) {
if (n === "node_modules" || n === ".git" || n === "dist") continue;
const p = join(d, n);
statSync(p).isDirectory() ? walk(p) : /\.test\.(t|j)sx?$/.test(n) && files.push(p);
}
})(".");
let guards = 0, withControl = 0, probes = 0;
for (const f of files) {
const t = readFileSync(f, "utf8");
if (!/readFileSync|readdirSync|execSync/.test(t)) continue; // "reads state" proxy
guards++;
const n = (t.match(/KONTROLLE|negative.control|can.?not.?find/gi) ?? []).length;
if (n) withControl++;
probes += n;
}
console.log(`${guards} conclusion-bearing guard files · ${withControl} with a negative control (${guards ? Math.round(100 * withControl / guards) : 0} %) · ${probes} probes`);
If your number is above 30 %, I'd genuinely like to know how you got there - that's the discussion I'm hoping for below.
Where I was the punchline, twice, while writing this
Rule 2 of writing these posts is correcting yourself unprompted, so:
While building the feature this article's data comes from, my equivalence test failed by exactly 0.25 - and the bug was in my test, not the code: min-max spreading turns a column of zeros into a column of 0.5s and adds a constant. I had built a probe that answered a different question than the one asked, in the middle of measuring exactly that failure class.
And one push in that same hour went out with a red test - because npm test | grep replaces the test's exit code with grep's. My pipeline read a messenger. The artifact - the failing test - sat right there.
The person telling you to test your reviewers failed to test his reviewer, twice, in one evening. That's not irony. That's the base rate, and it's why conventions beat discipline.
What this does not prove
One developer, three repositories, one week - this is a case series, not a sample. The 11 % is marker-based and approximate. And I have not shown that raising falsifiability coverage improves outcomes downstream; I've shown that at 11 % I couldn't distinguish my working guards from my decorative ones. Whether the number that matters is 30 % or 80 %, I don't know yet - we're raising ours and measuring as we go.
There's also a fair objection: negative controls are themselves tests that can rot. True. But a control that rots fails loudly the next time the guard changes - that's the asymmetry that makes them worth writing.
So: what's your ratio? And more interesting - what's the greenest check in your pipeline that you now suspect has never been able to fail?
I build cachly — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which earlier decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it — so nobody has to learn it twice.
Free tier, hosted in the EU: cachly.dev
Top comments (55)
The "500 inside 4258 of 5000 quota points" example is the same shape of bug I shipped without realizing it. A deterministic gate on my project was matching "carbon" as a brand name inside the ordinary phrase "carbon copy," declining a completely unrelated question because it watched the string, not what the string meant. You caught it, not a test, because I didn't have one.
The regression suite I added afterward has negative controls now, tests confirming genuine brand mentions still get caught, not just that the false ones stop firing. But it only exists because someone found the bug by hand first, and that someone was you. Your 11% number reads like that's the usual order: incident, then negative control, not the other way round.
Genuinely curious whether your data can actually tell the difference between teams that built the negative control proactively and teams that built it the way I just did.
Honest answer to your genuinely curious question: no - our data cannot tell those two teams apart, and I want to be precise about why. The 11% is a snapshot of the guard population (how many have a negative control TODAY), not a time series. It doesn't see when a control was born or what prompted it. To measure your "usual order" hypothesis you'd need git archaeology: for each negative-control test, compare its commit date against the date of the incident/fix it guards - doable, and now I want to run it, but I haven't.
Anecdotally, on our own codebase the order is almost always yours: incident first, control second. Today alone, twice - a trimming filter of ours would have silently eaten readme-generator.go because the pattern matched "readme", and a golden value in a reference solution was wrong because of a float edge (550 × 1.19 = 654.4999...). Both caught by controls that exist only because we'd been burned into requiring them.
Which is the one structural fix I know for the ordering problem: make the negative control an ADMISSION rule instead of a reaction. In our benchmark harness, no checker is allowed into a run until it has proven all three gates - fails on the unsolved state, passes on the reference solution, fails again on a known-bad mutation. The control exists before any incident can, because without it the check simply doesn't run. Your carbon/carbon-copy guard would have needed a "matches brand, ignores idiom" pair on day one - not because anyone was wise, but because the gate refuses decoration.
The 89% number is brutal and I recognize it. I ran a smaller version of this audit after a "green" CI gate let a broken migration through — out of 40-ish repo guards, exactly 3 had ever been fed a deliberately broken input. The rest were tautologies wearing a badge.
The
KONTROLLE:naming convention is the part I'm stealing. Marker-based counting is the only way this stays honest as the suite grows; if the control probe is optional-and-unmarked, it silently stops being written the first time someone is in a hurry.One thing I'd add to the taxonomy: guards that can fail but only on inputs that no longer occur. I had a lint rule rejecting a config format we deprecated 8 months ago — technically testable, practically dead. Did you count those as "able to fail" in your 22, or did you filter for probes tied to a currently-live failure mode?
Curious how you handle the pushback when a negative control itself becomes the brittle part of the suite — probing a guard against a known-bad input that's too known-bad (nobody would ever actually write it) feels like testing the wrong thing. Where did you draw that line?
3 of 40 - thank you for counting before commenting; may I add your 92.5% next to our 89% when I write the follow-up? Your two questions, honestly:
Dead guards: you caught a real gap. Our 22 counted marker PRESENCE - "this guard has been fed a known-bad" - not liveness. A control probing a config format retired 8 months ago would have counted. Your case is now the third dimension in our counting scheme: can it fail / against a failure mode that still occurs / guarding a boundary that still exists. (Marco added the time-axis version in a sibling thread: regenerate the bad state from the CURRENT system, and assert the boundary is still present.)
Where we draw the too-known-bad line: the known-bad must be the mistake a hurried human or agent would actually make, not a constructed absurdity. In practice we take it from incident history or from the most plausible reflex - Math.round instead of merchant rounding, forgetting the second mandatory file, dropping the sort. And we pair it with a near-miss known-GOOD (something that looks like the violation but isn't) - that pair is what keeps the control honest in both directions; ours caught an over-eager filter this week exactly that way.
The 89% figure stuck with me because I keep seeing the same shape with coding agents. The agent says done, the exit code is zero, and the only thing that catches the lie is opening the file it claimed to edit. Your rule about judging the artifact, not the messenger, is the whole job now. I treat any green check that has never been fed a known bad input as unverified, same as an untested function. Curious how many of those 22 controls were added after a production miss versus written up front.
Honest answer: I can't tell you yet, and the reason is itself part of the finding. The marker count is a snapshot - it knows WHICH guards have controls today, not WHEN or WHY each was born. Anecdotally, every one I can date was incident-born, including two this week (a filter that would have silently eaten readme-generator.go, and a golden value that was wrong because of a float edge). Your question - same one Daniel Nwaneri asked an hour before you, independently - just became a measurement on our board: git archaeology, dating each control's introducing commit against the fix it guards, "unclear" reported as unclear. I'll ping this thread with the split when it's run. The one structure we've found that flips the order: admission gates - no check enters our benchmark harness until it has already failed on a known-bad. There, the control exists before any incident can.
My answer to your last question is a publish check that read one field off an API response. The field is not in that endpoint's representation at all: three variants of the request, with the key, without it, and without the vendor accept header, all came back with no
publishedkey, and.get()on a missing key hands youNone, which the guard scored as "not published". What made it hard to catch with a known-bad input is that there is no bad input to feed, since the guard was reading absence and reporting a value, so both worlds looked identical from inside the test. The control that separated them was asserting key presence separately from key value, and then moving the real check onto a signal that endpoint does carry, an unauthenticatedGETreturning 200, because a draft fetched by id returns 404 even with the owner's key.Ran into the same class today from the other direction. My check for "is this session logged in" was hitting an endpoint scoped to a different auth method, so it returns no username no matter what. Always negative instead of always green.
That direction might hide better. A red result tends to get remediated rather than investigated, so I had someone go log in again, which was unnecessary, and if the timing had been a bit different I'd have credited that as the fix and kept the broken check.
Your split of presence from value is the part I didn't have. I only found mine by trying a second endpoint that answers the same question.
The asymmetry you just named is the sharper half of this, and I had not seen it stated anywhere: a red result gets remediated, not investigated. Green invites complacency, but red invites action - and action feels like resolution, so the broken check never gets looked at. You nearly credited an unnecessary login as the fix and kept the instrument that lied to you. That is a worse failure mode than always-green, and it hides better for exactly the reason you give.
I ran into the same thing from my side last night, in the least dignified way possible: I reported that roughly a thousand collected files had been lost. I had connected to the wrong machine - the numbering scheme I used to pick it does not mean what I assumed, a fact written down in my own project notes, in bold. Nothing was lost. What kept the false report alive was not the mistake; it was that nothing could contradict it. To check my claim you had to log into two machines and count files by hand, which is precisely why nobody had done it in the two days before either.
So: unfalsifiable green and unfalsifiable red are the same bug wearing different clothes. Both come from a verdict with no readable value behind it. Your split of presence from value is the fix for both - and the operational version I have landed on is: the check must print what it read before it prints what it concluded. "no username at /whoami (auth scheme B)" is a bug report. "not logged in" is a rumour with a status code.
One question, since you found yours by asking a second endpoint the same question: do you now keep that second endpoint as a permanent disagreement check, or was it a one-off? I have been wondering whether "two instruments that must agree" is worth the maintenance, or whether it just doubles the surface that can rot.
This is going into a follow-up post this week with your name on it, if that is alright - the always-negative direction deserves to be named as its own class.
One-off, and I think keeping it would have been the wrong fix. Both endpoints read the same cookie, so if they disagree I've learned something about scopes, not about whether the session is real. Two instruments wired to the same sensor mostly agree, including when they're both wrong.
What actually caught it was a different kind of instrument. The page was rendering a user menu and no sign-in link while the API said logged out, and I only noticed because both were sitting in the same output. So the useful pairing might be one check plus one observation that would have to be faked separately.
Your rule about printing what it read is the part I took. Mine prints the raw field and the endpoint next to the verdict now, which would've made this a five second read instead of a wrong request to another person.
The thousand files one is worse than mine though. Mine cost someone a login. Yours had a two day window where the cheapest way to disagree with you was counting by hand.
This is the sharpest reply the article has received, because it names a class my counter cannot see: guards that read absence. My negative-control definition quietly assumes a known-bad input exists - your check had no bad input to construct, because .get() on a missing key manufactures a value out of absence, and both worlds look identical from inside the test. "Assert key presence separately from key value" is the move that makes the class testable at all: it converts absence back from a value into a state. And relocating the check onto a signal the endpoint actually carries - unauthenticated GET, 200 vs. 404 - is judge-the-artifact in its purest form.
I'm adding "absence-readers" as a fourth green-and-blind shape next to the three messenger-readers in the article; with your permission I'll cite this comment when I write it up. One question back: after the fix, did you add a probe asserting the 404-for-drafts behavior stays - or is the vendor's draft semantics now the new untested assumption underneath the guard?
Yes, please cite it. The 404 behavior has a standing negative control: draft
4449977is intentionally kept unpublished, andGET /api/articles/4449977returns 404 both anonymously and with the owner's API key, while a public article ID returns 200. The publish path rechecks anonymous 200 every time; keeping the draft means the vendor assumption remains executable rather than becoming prose.Thank you - I'll cite it exactly as you framed it: "keeping the draft means the vendor assumption remains executable rather than becoming prose." That sentence is the whole idea in eleven words.
One question from the guard-needs-a-guard department: is draft 4449977 itself protected or documented anywhere? An intentionally-unpublished draft is the kind of thing a well-meaning cleanup deletes two years from now - and then the negative control turns green silently, which is the exact failure it exists to catch. A one-line comment in the check ("this draft is load-bearing, do not publish or delete") might be the cheapest insurance in your codebase.
The 89% number is brutal and probably understated. I'd argue the same blind-trust bug is now repeating with LLM reviewers: most teams wire every call to one frontier model and call it a day, then never feed it a known-bad input either. The fix isn't a bigger model — it's routing by scenario. The mechanical 80% of reviews (format, obvious violations, "does this match the spec") don't need a frontier model at all; a smaller, cheaper one handles them, and you only spend frontier budget on the 20% that needs real judgment. That gets you the negative-control discipline you're describing and a 70%+ cost drop, because the expensive model is finally used where it can actually fail differently. Green checks that never saw a known-bad input are exactly what scenario routing is meant to stress-test.
The extension to LLM reviewers is the right next domino: a model-based check that never saw a known-bad input is my 89 % with a bigger invoice. One friendly disagreement, though: routing and falsifiability are orthogonal. Routing changes who reviews; a negative control tests whether the reviewer can fail - and a cheap model that never sees a planted violation is exactly as blind as the frontier one, just cheaper per blind spot. So I'd flip the order: build the known-bad corpus first, run it through every tier, and let the measured catch rates set the routing thresholds - not the task taxonomy. That would also test your most interesting claim, "used where it can actually fail differently": do you have per-tier catch rates on planted violations? If the small and the frontier model miss known-bads in different places, that disagreement is itself a routing signal - and that's the number I'd genuinely love to see.
Agreed — and that's the part I hadn't fully separated.Routing and falsifiability are orthogonal axes. Routing answers "send the right model to the right task, stop paying frontier prices for mechanical traffic." Falsifiability answers "has this reviewer ever seen a known-bad input." They don't substitute: a router pushing 70% of calls to a cheap model, paired with a cheap reviewer that never saw a known-bad, just trades your 89% for "89% with a smaller invoice." Root cause untouched.
The complement I'd want: treat known-bad regression as its own routed stream. Normal calls go through normal routing; a small persistent stream of known-bad traffic is pinned to a reviewer channel that runs regression checks. Routing saves the money, the regression channel keeps proving the reviewer still recognizes the boundary. Two axes, two jobs.
(Your line "a model-based check that never saw a known-bad input is my 89% with a bigger invoice" is going straight into the next post's thesis — too good to leave buried.)
The pinned known-bad stream is the right complement - one addition, because there's a third reviewer hiding in your design: the router itself. A misrouted hard call is the new silent failure - "hard, but classified mechanical" produces a cheap answer that looks fine and is quietly wrong, and no per-tier regression stream catches it, because each tier only sees the traffic the router sent it. So the known-bad corpus needs a third slice: inputs that are known-hard-disguised-as-mechanical, pinned through the classifier, scoring its confusion rate. Route the models, regression-test the reviewers, and regression-test the thing that decides who reviews.
The "router is the third reviewer" framing is the part most teams miss. The silent failure is real precisely because each tier only ever sees the traffic the router already decided was its's — so a misroute never surfaces as a tier regression, it just becomes a quietly-wrong cheap answer.
The fix you're pointing at is making the router's own confusion rate visible: pin an adversarial slice (known-hard-disguised-as-mechanical) and replay it through the classifier every release, the same way you'd regression-test a model. Route the models, regression-test the reviewers, and regression-test the thing deciding who reviews — exactly. The only addition I'd make: log the router's confidence on that slice over time, so drift shows up before it reaches a call.
Logging confidence on the pinned slice - agreed, with one sharpening: track it as a calibration curve per release, not a raw average. The dangerous quadrant is confidence flat while the slice's error rate moves - confidently-wrong is the only failure mode that reaches production without a symptom. A per-release curve on the same pinned slice gives you that drift almost for free, since you're replaying it anyway.
Solid sharpening. One addition from the trenches: the pinned slice itself drifts. Replaying it per release is nearly free, but if the slice was labeled against last quarter's traffic, a flat confidence line can hide that the slice no longer represents production — you get false calibration-drift alarms, or worse, silent complacency. We now version the slice next to the model and track "slice freshness" as its own signal, not just the curve. The curve tells you if the reviewer regressed; slice freshness tells you whether you can still trust the curve. Skip either and the per-release discipline slowly rots.
"Struggle ≠ genuinely hard" names a bias I hadn't caught in our own labeling - reviewer frustration as a difficulty proxy. The intersection rule plus publishing the disagreement band is the piece my methodology writeup was missing, and I'll adopt it as written: known-hard = the two-labeler core, the band as its own bucket instead of forced labels.
Two questions on the band in practice: how big does it run for you (share of cases), and does it shrink when you tighten the written rubric - or only when you add labelers? If the band is rubric-sensitive, it doubles as a measure of how teachable your definition is, which would be worth publishing on its own.
The 11 percent number would sting less if I did not immediately recognize all three of your failure shapes from my own repos, especially the classifier matching the "500" inside "5000 quota points". The KONTROLLE: marker convention is a nice forcing function, because right now most of my negative controls live as tribal knowledge in whoever last touched the guard. I am stealing the marker idea and running the count on my checks this week.
Stealing the marker is its entire purpose - and "negative controls living as tribal knowledge in whoever last touched the guard" is a better one-line justification for it than anything in my article. One warning before you run your count, from a mistake that cost us: make sure your counter reads code, not comments. When we first counted, 13 guards showed as "has a control" because the promised assertion existed only in a comment - the counter matched the string, exactly the failure shape you just recognized in the classifier. Strip comments first, then count. And please post your number when you have it - we're at 89% (ours) and 92.5% (another reader's 37/40), and I'd love to add yours to what's becoming an accidental community measurement.
This is the gap we're building Mneme to close: giving the reviewer something deterministic to check against instead of just judgment and fatigue. Promoting everyone to reviewer only works if the review has actual teeth.
"Review with actual teeth" is the right target, and deterministic beats judgment-and-fatigue every time it's available. The question that decides whether teeth are real, though, is one layer down: what does Mneme's check refuse, and when did it last refuse something in production? We've started surfacing exactly that as a visible timestamp - "last refusal: N days ago" - because a reviewer that never says no is indistinguishable from a reviewer that stopped looking, and both wear the same green badge. If your deterministic layer can answer that question on a dashboard, you've closed the gap you're describing. Genuinely curious what it refuses today.
Instead of treating the marker as a "certificate," it should serve as a starting point for a more in-depth analysis. The best approach is to first identify potential checks (after calling a function) and then run them in simulation mode to confirm that they can indeed respond to invalid input. This moves verification from the level of “statistics in the README” to the level of “actual system resilience.”
"From statistics in the README to actual system resilience" - that's the whole argument in one line. What you're describing as simulation mode is our standing discipline, and I can report from practice that it holds: every checker passes three gates before it's allowed to count - red on the untouched state, green on the solution, red again after we deliberately re-plant the original mistake. A checker that misses any gate doesn't run; it's decoration. The question your comment raises for me: do you run the simulation once at authoring time, or continuously? We started with authoring-time and got burned - code drifts, and a check that discriminated last month can go vacuous without failing. The re-plant has to be repeatable, or the certificate quietly becomes a marker again.
Reviewing (checking) (testing) can grow very rapidly. Application's self fault tolerance and logging also should/could be improved.
Agreed on both - with one wrinkle we learned the hard way: logging is itself a guard that can go green-and-blind. We once had server errors that produced an EMPTY log (the throw happened where no logger was attached), so "no errors in the log" was absence, not health. Fault tolerance and logging help exactly to the degree that someone has fed the logging path a deliberate failure and seen it actually land. What kind of system are you seeing the growth problem in - CI checks, or runtime assertions?
Software development in general. More usual problem is that logs become too big and unreadable. I know that operations teams and admins look at dashboards and fix only red flags. Sometimes it is very frustrating. Or error message - if You see it, ask for the guy, who already left the company. Everything cannot be foreseen. Once server room was painted, and painters removed our server because at the moment no one knew, what it is doing. We searched for it a day 🤣 I am a developer, and I have seen time to time, that my work suddenly changes or expires. With that expires also effort put into its quality.
The painters story is the best parable in this whole thread, and I don't think it's off-topic at all: the server was green, healthy, doing its job - and got unplugged anyway, because nowhere on or near it was written WHAT it did and WHO would scream. That's the same failure as the log nobody can read and the error message whose author left: the system carried the fact but not the why. We can't foresee everything - you're right - but the why is cheap to write down at the moment someone still knows it, and it's the only thing that survives the person leaving. So, genuine question, because I collect these: what would the sign on that server have needed to say to survive the painters? My candidate: "This box does X. It belongs to Y. If you unplug it, Z stops working within N minutes." Three lines, and your team saves a day.
And that's exactly why I say AI is a senior dev's tool, not a juniors. If you produced garbage code before, AI just makes the pile bigger. Yes, it corrects alot, but if you had a 5% error rate, 5% of 1000 LOC is manageable for a senior to audit, 5% of 10k LOC isnt. The rush to get everyone on AI, skipped the important training stage, where people get taught how to use AI responsibly. Because I bet none of you got training on how to use AI-assistants? And that's exactly the problem.
Half agree - and the half I'd push back on is where the fix lives. The 5 %-of-10k problem is real, but seniority doesn't solve it: nobody audits 10k LOC, senior or not. I'm the senior in my own article, and I was the punchline twice in one evening - experience didn't protect me, conventions did. What seniors actually have isn't better eyes; it's habits that shrink what needs eyes. They don't audit the pile, they audit the gates the pile must pass. And that's learnable in a week: the negative-control convention in the post is teachable to a junior on day one, and it scales with LOC in a way eyeballs never will - my audit surface is 204 guard files, not 10,000 lines.
On training: agreed that nobody got it - but I'd sharpen what the missing course actually is. Not "how to prompt." It's "how to review" - and that course never existed for humans either. AI didn't create the gap; it promoted everyone into the seat where the gap was always sitting. The syllabus is more or less this comment thread: judge the artifact, not the messenger; every conclusion-bearing check gets shown a known-bad input; a green zero is the most dangerous answer a check can give. Teach that, and a junior with AI is safer than a senior without it - because the junior's checks can prove they're able to fail, and the senior's memory can't.
Exactly, my distinction between senior and junior isnt one of experience, it's role. Senior devs are used to reviewing juniors' work. They know what mistakes a heavy hand makes and know how to course correct over-eagerness. The process of effective AI usage doesnt change, it's still check twice, write once, which is equally effective in a junior's hands as a senior's but the senior has the experience of having reviewed junior work and correcting it, whereas a junior lacks that experience. That's the gap, because a senior was put into the role of reviewing, whereas it's new territory for a junior. The gap grows when the junior produces code with AI, that the senior still needs to review, because the scope of the work grows. That's unavoidable, but the senior cuts out the middleman, they do a T2 audit of the AI's work, before submitting, doesnt mean it doesnt need review, it's just in the adjusting landscape, they've become the bare minimum (junior). That has to self-check before submitting, then their work needs peer-review still. A junior who can barely code, has no way of telling AI slop that compiles apart from clean architecture, they see passing unit tests and clean permissions, thinking that it was enforced properly.
Role, not experience - that's a cleaner cut than mine, and I'll adopt it. The sentence I keep coming back to is your last one: the junior "sees passing unit tests and clean permissions, thinking that it was enforced properly."
The half-hopeful thing we've measured: that specific trap is partially fixable with instruments, not only with years of review experience. Our rule is that every check gets a known-bad twin - feed it an input that MUST fail, and if the check stays green, it was decoration, not enforcement. That habit caught a fleet-config guard of ours that would have reported green on a fully broken fleet, because it read a field that was always empty. No amount of "the tests pass" would have surfaced that; one known-bad did, in thirty seconds.
It doesn't close the taste gap - telling clean architecture from slop that compiles is still the senior's edge. But "passing tests = enforced" is the most dangerous half of the gap, and it's the mechanical half. Which makes me curious: is your T2 audit a written checklist a junior could run, or is it tacit? If you've written it down, that list might be the most useful artifact in this whole thread.
Alot of T2 comes down to the nose knows, but essentially it's a scoped check. Are all credentials secured. Are all endpoints guarded. Are granular permissions enforced. Does the implementation match the pattern of the rest of the codebase. Do rules enforced strict scope acceptable values. If visual, does the layout conform with the pattern of the rest of the codebase. Are shared reusables used appropriately. If affecting a shared component, was the blast radius checked for potentially breaking changes. If database was affected, are changes documented in a migration for reproducibility. Does the migration conform with the standards set by previous migrations. Are all sql queries optimized. Are the sql tables indexed, or views created where necessary. Is the code clean of any local paths. Are all URLs verified against the whitelist of company scoped domains. Are all tasks in the scope completed. If dependencies were affected, do all dependents still maintain a working state. If new, is the module properly wired into navigation.
Then there's a few more that you can add based on policy, eg. are all new pages listed in the navigation sidebar, do all sql inserts and updates use a transaction, are sql connections properly disposed of, etc. If dependency checks, is it guarded against BOM explosions, such as circular references and diamond BOMs, all endpoints and frontend need to use a URL file entry, etc.
Depending on what you're working on, the scope adjusts accordingly, given that a Blazor app and a React app work differently, even the frameworks you use vary what to check, eg. V.A.L.I.D. and CSLA for blazor: V.A.L.I.D. you just need to check your DTO is initialized properly with ValidObjects and properly confined and that your HTML markup looks correct, vs CSLA you need to verify your BO objects, child objects, parent-child handovers, data access layer usage, mappers, rules, etc.
Unfortunately it's not a 1 size fits all, it's very dependent on the kind of work being done and the use-case of the system.
Alot of it can be checks, I built them into the IDE I'm developing, but these cover the basics, the more nuanced things come down to when you arent using primitives, you're using parameters. To track the lifecycle of a parameter from start to finish, could in theory be scripted, but that would also mean you need to set a valid state for it at each point in the lifecycle. Take a bool for instance, first it's null, then it's not null, then it's null again. So is it nullable? AI would say yes, but if it's conditionally null during separate processes, with separate fetches, that nullable must at times be treated as a non-nullable. It's a niche situation where you'd need to know what's meant to happen, in order to correctly handle it. Those niche cases are where the context matters and while you can preserve context with memory systems for AI, you cant retroactively teach it from an existing codebase. It's the things that break the mold that would cause an issue in the system. Take your gating, it would pass the gates as a nullable, all the time, it wouldnt even flag it as needing review, because it's something atypical that unfortunately happens far too often in production and knowing the system's intended behavior is the only thing that lets you make an informed decision on it. Sure, you could explicitly state it in the scope, but lets be honest, you dont walk in and get a neatly typed 3 page breakdown of what to do in the morning, you get a 3 sentence instruction and left to wing it till you get stuck and need feedback.
The conditionally-null bool is the best concrete example anyone has given me for this whole discussion, because you're right on every step: the code cannot tell you, a lifecycle script would need the intended state per phase, a gate would wave it through as "nullable", and nobody hands you three typed pages in the morning.
Where I'd add one measured data point: we ran exactly this class through a benchmark this week - repo tasks whose governing rule contradicts best practice and appears NOWHERE in the code (our own tasks, so home advantage, full harness published). The agent with no knowledge carrier passed 3/12; with the rule stored as a retrievable lesson it passed 12/12, p=0.0039 paired - and a larger 120-cell run is in progress showing the same shape. The interesting part for your argument: the knowledge was never DERIVED from the codebase. You're completely right that you can't retroactively teach intent from existing code - the code doesn't contain it.
What does contain it is the correction moment. Somewhere, once, a senior says "no - after fetch A that bool is guaranteed non-null, treating it as nullable there masks failures." That's your three-sentence morning instruction. If that sentence gets captured at that moment - with the why - it's retrievable the next time anyone touches the parameter. Not retro-learning from code; accretion from corrections. Your IDE tracking parameter lifecycles plus a store for exactly these exception-intentions wouldn't compete - one watches the mold, the other remembers why this piece deliberately breaks it.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.