Context that survives
Answer it without opening a chat window. What does your assistant know this morning that it learned yesterday?
For most setups the honest answer is nothing, and nothing has a price. But that is the easy version of the problem. The one that actually cost me an afternoon is worse: it knew, and it did not tell me. The fact was stored, it was displayed, and I made the mistake anyway. More on that below, because it is the part everybody gets wrong on the first try — and the part nobody warns you about.
The cost is a number, not a feeling
Take one week and count the first messages of your sessions. Not all messages, just the opening one of each.
In my own week, seventeen of twenty-two opening messages were setup. Which servers exist, which one runs the database, which deploy path is the real one, which trap I fell into last month. Roughly ten minutes each before any work started.
That is nearly three hours a week spent re-teaching facts that had not changed. Not learning anything. Re-typing.
Before you take that number anywhere: it is one person, one week, twenty-two sessions. It is not a study and I would not defend it as one. The reason I am handing it to you is that it took twenty minutes to produce, and you can produce your own by tomorrow morning. That is the only part of it I would argue for.
The worse half is invisible: the sessions where I did not bother re-explaining, and the assistant confidently used the wrong server name because nobody had told it otherwise.
Why a longer context window does not fix it
A bigger window makes one conversation smarter. It does nothing across conversations, and across conversations is where the work actually lives.
The knowledge you need tomorrow was produced today, at the end of a debugging session, in the moment somebody said out loud why the thing broke. That sentence is the asset. It exists for about thirty seconds and then the window closes.
Documentation is supposed to catch it and mostly does not, because writing docs is a separate task with separate motivation, performed at the exact moment you least want another task.
So the sentence needs to be captured by the participant who is already typing: the assistant.
What changes when the memory outlives the session
Two things get better, and one gets uncomfortable.
Better: the first ten minutes disappear. The assistant reads what it wrote before, and starts from the state of the world instead of from zero. You notice this as the absence of an annoyance, which is a strange kind of win to measure but a real one.
Also better: repeated mistakes get expensive to repeat. When the reason a thing broke is written down where the next session reads it, the second occurrence of that mistake stops being free.
Uncomfortable: you find out how many of your problems were repeats. In my own recorded set, several entries described mistakes I had already made twice. The memory did not make me smarter. It made me stop paying twice.
The part everyone gets wrong on the first try
Storing is easy. Delivering is where this fails, and it fails quietly.
I had a lesson stored that contained the exact address I needed, along with a warning against the exact mistake I was about to make. It was displayed to me at the start of the session. I made the mistake anyway.
The preview showed the first hundred characters. The address sat at character three hundred and twenty-three. The warning lived in a field that was never rendered at all.
The store was perfect. The delivery was a sieve with a hundred-character mesh. If you build this, spend your effort on what gets shown, not on what gets saved.
Build the smallest version this week
You can test the idea in an afternoon without adopting anything. Two habits and a file.
First, at the end of any session where something broke and got fixed, write one line: what broke, what fixed it, which file. One line, not a document.
Second, at the start of the next session, paste the lines that match what you are about to do. That is the whole loop, performed by hand.
Third, and this is the part that decides whether it works: put the runnable thing first in the line. Not the backstory, the command.
A file and two shell functions are enough to try it:
mem=~/.memory.tsv
# save: one line, the fact FIRST, prose after
remember() {
printf '%s\t%s\n' "$1" "$2" >> "$mem" # remember "staging-db" "port 5433, NOT 5432 - 5432 is prod"
}
# recall: grep, then read the whole line — not the first 100 chars
recall() {
grep -i -- "$1" "$mem" | cut -f2
}
# Try it for a week. If recall returns something you would otherwise
# have retyped, you have your answer. If it returns nothing, you learned
# that cheaply.
Two rules make the difference between this working and this rotting. Write the decisive fact in the first ten words. And when you truncate for display, truncate the prose, never the identifiers.
What changes for you
Before: every morning you type the same three paragraphs about your own infrastructure, and on the mornings you skip it, you get confident answers built on the wrong assumptions.
After: the assistant opens with what it learned last time, including the mistake it made and how that got fixed. Your first message is about today's work, not about your server names.
A memory that stores everything and shows a hundred characters is not a memory. It is a filing cabinet nobody can open.
I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.
Free tier, hosted in the EU: cachly.dev
Top comments (4)
The distinction between storing memory and actually delivering the right memory is probably the most important part here.
I’ve seen a similar failure mode with AI-assisted development: the system technically has the context, but the relevant piece doesn’t reach the model at the moment it matters.
That makes me think memory quality shouldn’t only be measured by “what did we successfully store?” but also by “did the right piece of context change the next decision?”
Otherwise we’re just building increasingly sophisticated filing cabinets.
I’m curious how you think about that second metric — should memory systems eventually evaluate themselves based on whether retrieved context actually improved the outcome?
Yes - and I want to give you the half I now have and the half I still don't, because the gap between them is the whole problem.
What I built this weekend: the retrieval path now records, per answer, the query, the lesson it returned, whether it helped, and at which rank - with rank 0 meaning "was not in the answer at all". The last one is the valuable case and the one everybody drops, because a miss leaves no row anywhere. It's derived server-side rather than self-reported: the tool doesn't get to grade its own homework.
That still only measures delivery, not outcome. It tells me the right lesson was in position two. It does not tell me the answer changed because of it.
What I don't have: the counterfactual. The only honest instrument I've found is withholding - hold back the top record on a random half of eligible turns and compare outcomes in aggregate. It costs real sessions, because half of my own turns get a deliberately worse answer for the duration. I haven't found a cheaper version that isn't a self-report in disguise.
The trap I'd warn you about before you build it, because it took me a while to see: an outcome-based metric is gameable by the memory becoming conservative. If the system only surfaces what it's confident about, "did retrieved context help" goes up - while the number of turns that got any help at all goes down. You'd be optimising toward a memory that says less and is right more often, and the graph would look like progress.
So the denominator has to be eligible turns, not surfaced turns. helped / turns where a relevant lesson existed, not helped / turns where we showed one. The second is the number that's easy to collect and it rewards silence.
Which lands on the thing your last line already says: a filing cabinet that is never opened scores perfectly on precision.
That denominator distinction is excellent. “Helpful / surfaced turns” really could create a misleading feedback loop where the system learns that saying less is safer.
I especially like the counterfactual approach, even though it’s expensive. It changes the question from “did retrieval look good?” to “did retrieval actually make a measurable difference?”
It also makes me think that memory systems may eventually need to optimize for useful intervention, not just retrieval precision. Knowing when not to surface a memory could be just as important as knowing which memory to surface.
The “filing cabinet that is never opened scores perfectly on precision” line is probably the simplest way to describe that problem.
"Knowing when not to surface a memory could be just as important" - I agree with the principle, and our own numbers say it is not our bottleneck. Which is a more useful answer than agreement, so let me show the numbers.
First, the honest structural admission: our system has no "nothing here is good enough" state. It ranks candidates and returns the top three. There is no abstention. That is the same missing third state I've hit four times this week - found / found nothing, and no way to say I have nothing worth showing.
Second, why fixing that would buy us almost nothing right now. From our benchmark, on 100 questions against a real store:
So in 97 of 100 cases we have the thing. Our failure is almost never "we didn't have it" - it is "we had it and put it second". An abstention mechanism improves the 3 %. Ranking improves the other 57.
That's worth stating precisely because it cuts against the intuition. Abstention feels like the sophisticated move. For us it would be sophistication applied to the smallest term.
Third, and this is the part I'd hand you before you build it: abstention is unmeasurable in production by the same argument we just made about the denominator.
To know whether a silence was correct, you need to know whether a relevant lesson existed - which is ground truth. You have it in a benchmark. You do not have it in production, ever. So "was this abstention right?" collapses into the same counterfactual as "did this recall help?", and the only instrument is still the hold-out.
Which means abstention has a nastier property than ranking: a wrong abstention leaves no trace at all. A wrong recall at least produces a visible bad answer someone can complain about. A wrong silence produces a session that was merely slightly worse, and nobody files a ticket for that.
So my order would be: fix ranking while you can measure it, and only add abstention once you have the hold-out running - because abstention without the hold-out is exactly the "say less, look better" loop you named.