DEV Community: pm25coder

50 minutes from issue to merged fix: when the readers find the boundary you shipped past

pm25coder — Wed, 26 Aug 2026 16:46:45 +0000

We published a postmortem about a token counter that drifted 50% and a safety net that never fired. Two readers extended the analysis: one asked for a countable metric, the other found a boundary we shipped past. Both requests were merged as fixes the same day — one of them fifty minutes after the issue was filed. This is the story of that loop, and the two changes that closed it.

The setup: a postmortem that became a boundary generator

A few days ago we published the story of our auto-compact safety net: the local estimator said 148K tokens while the provider was actually seeing 222K, the gate never fired, and the fix was to anchor the projection to the provider's real prompt_tokens, then fail loud whenever the anchor goes missing.

Postmortems are usually read, nodded at, and forgotten. This one got extended. Within three hours of publication, two commenters had pushed the analysis past where the code actually was.

Reader 1: make it countable

The first comment was about the fail-loud warning itself. The warning existed, but it was a log line — something you have to grep for. A warning you can only find by searching is absence reading as health: the system can be un-silent and unheard at the same time. The suggestion was concrete: when the anchor is missing, log the estimated size of every payload in that anchor-less window, and the weekly max over real traffic becomes the measured worst case. No assumptions, no theory — measured, bounded, real.

Reader 2: the anchor is keyed by session ID only

The second comment found an actual bug. The usage anchor — the entire safety mechanism after fix 1 — is a tuple keyed only by session ID: (real prompt_tokens, local estimate). If a session switches models or providers mid-conversation, the projection keeps the OLD provider's real base and adds the NEW provider's estimate delta. A mixed base. And because the anchor is present, the fail-loud warning never fires. The exact failure mode we had just written a postmortem about, still reachable through a boundary we shipped past.

Four minutes of reading, one boundary case, zero code access. That is what a good postmortem is for: it teaches the reader the mechanism so precisely that the reader can find what the authors missed.

The same-day loop

Here is what happened next, in order:

The countable metric was implemented and merged (est at loss, real at re-anchor, delta measured per loss window, cumulative total, append-only file that survives restarts).
The boundary finding was filed as an issue with code citations: the anchor is keyed by session_id only; set_model switches the daemon-global model without touching the anchors.
Fifty minutes later, the fix was merged and the issue closed.

The fix: invalidate on switch, mark the re-anchor round

The fix has three moving parts, and the middle one is the subtle one:

When the API model actually changes, every usage anchor is dropped, and any pending drift window with it. The old base cannot mix with the new estimate delta because there is no old base anymore.
The session is marked so the fail-loud warning treats the switch round as a legitimate re-anchor round — the first anchor-less round after a deliberate switch must not scream. But the marker is consumed by that round, so if the NEW provider is also silent, the following round warns. Deliberate loss warns once and gets measured; accidental loss warns again.
Four regression tests pin the behavior: the switch drops the anchor, the switch round stays silent, the next round warns if the new provider is also silent, and the projection can no longer mix bases.

The composition: fixed and measurable at once

The two fixes compose. The drift metric records an anchor_loss event at loss time and an anchor_drift event when the session re-anchors on real prompt_tokens. A switch-induced loss window now appears in the same drift file — so the exact failure mode the reader identified is both closed and countable. The mixed-base bug is no longer reachable, and if any future boundary reopens it, there is a number.

The open item, closed by the same loop

The counters started session-scoped - and the reader who asked for countable metrics flagged the gap before the code even landed: a counter that cannot name the provider is half a counter. That question was filed as an issue the same evening, and the fix merged a few hours later. Both events now carry the loss-time identity (model + provider, where provider is a deterministic hostname slug of the base_url - no heuristics, no DNS), and anchor_drift additionally carries the current identity, so a window that crosses a model switch says both who went silent and who re-anchored. The loop did not just close the bug the first reader found; it closed the second reader's follow-up question before it could become a bug. — and the reader who started this loop flagged it before the code landed: a counter that cannot name the provider is half a counter. The model-switch fix makes cross-provider loss windows appear in the drift file; attaching provider identity to the events is the next increment, and the question is now tracked as a feature request.

The general lesson

Three things generalize from this:

Publish the hard postmortem. The readers who just read your explanation of how the mechanism works are the cheapest boundary-finders you will ever hire. One comment found a live version of the exact bug the postmortem described, reachable through a path the authors had not thought to check.
Close the loop in hours, not sprints. Feedback to issue to merged fix in under an hour is possible when the feedback is specific, cited to code, and the codebase is small enough to fix in one sitting. The specificity came from the readers; the citations came from reading the code before replying.
Measure the thing you are warning about. A warning that requires grepping is a warning that can go unheard. A counter that survives restarts and carries a cumulative total turns "did the safety net ever misfire" from archaeology into a lookup.

The uncomfortable part is admitting how close we came to shipping the same bug twice — the boundary the reader found was one function call away from the fix we had already designed. That is the normal state of systems: there is always one more boundary, and the people most likely to find it are the ones who just read the honest account of how the last one failed.

Since: verified against master cbca8e5 (2026-08-27). Commits: 67d55081 (#995, countable usage-anchor stats — anchor_loss/anchor_drift JSONL), 4616a9a3 (#1003, invalidate usage anchor on mid-session model/provider switch — +4 tests, closes #1000), ef283ae3 (#1013, provider/model identity on loss/drift events - closes #1011). Issue #1000: filed from Dev.to comment 3dh3g, closed by the #1003 merge; issue #1011: filed from comment 3dhdb, closed by the #1013 merge.

From the codebase of EMRG, an open-source (MIT) agent harness whose design is that the loop reads its own failures and converts them into tested fixes. The full history of this one is public: #995, #1000, #1003.

148K estimated, 222K real: when the token counter drifts, the safety net goes silent

pm25coder — Tue, 25 Aug 2026 20:01:46 +0000

Our agent's auto-compact was supposed to protect the context window. It shipped, it ran, and it never fired — because the local token estimator said 148K tokens while the provider was actually seeing 222K. This is the story of that 50% drift: why it's a structural trap for any LLM agent, and the two changes that fixed it (usage-anchored projection, then fail-loud anchor loss).

If you run an agent that feeds a growing conversation into an LLM, you have some version of this problem: how do you know when the context window is about to blow? The honest answer is that you don't — not locally. Providers bill by token counts they compute with their own tokenizers, and a local estimate is at best a guess. The trap is when a guess gets promoted to a gate: a threshold that is supposed to protect you, silently, based on a number that can be off by half.

That's exactly what happened to us, and the fix ended up being a one-line insight that reshaped the whole feature: when the provider gives you the real number, anchor to it — and if you ever lose the anchor, fail loud.

The failure: a safety net that never fired

Our daemon has an auto-compact feature: before each LLM round, it estimates the conversation's token count, and if the projection exceeds a threshold (a percentage of the configured context window), it summarizes the history to free up space.

The estimator is a deliberately cheap heuristic — character-aware, because our sessions are heavily CJK:

# emrg/server/daemon.py (v0.2.80)
for ch in text:
    if is_cjk(ch):        # CJK ideographs, kana, hangul, fullwidth forms
        cjk += 1
    else:
        ascii_chars += 1
return (cjk // 2) + (ascii_chars // 4)   # CJK ≈ 2 chars/token, ASCII ≈ 4 chars/token

Plus 3 tokens per message for role/name overhead. It's a reasonable directional heuristic — the kind of thing you'd use to display "≈12K tokens" in a UI. It is not a number you should hang a protective gate on, and that's what we did.

The observed drift, recorded in the code comments when we finally understood it:

Local estimate: 148K tokens
Provider's real prompt_tokens: 222K tokens

That's a 50% underestimate. With the auto-compact threshold set at a fraction of the context window, the projection sat quietly below the trigger while the real context was already 50% past it. The gate never fired. No log, no warning — just a context window slowly filling past the safe line, quality degrading and cost climbing, with the safety net fully armed and fully blind.

Why is the drift so bad in our workload? The estimator assumes CJK ≈ 2 chars/token and ASCII ≈ 4 chars/token, but real tokenizers are far more irregular. JSON-heavy tool results inflate the count (brace-heavy syntax tokenizes denser than prose), and CJK mixed with code and JSON produces a composition the heuristic simply can't represent. The error isn't a constant offset — it grows with the payload, which is exactly when the gate matters most.

Fix 1: anchor the projection to the provider's real number (#946)

The insight: we don't need a perfect estimator. At the end of every LLM round the provider hands us the real prompt_tokens — the ground truth for everything that was sent at that moment. So instead of trusting the estimator for the whole history, we cache an anchor and only let the estimator contribute the delta since the anchor:

# The anchor: (real prompt_tokens, local estimate) captured at the same moment
if final_usage:
    pt = final_usage.get("prompt_tokens")
    if pt:
        self._usage_anchors[session.session_id] = (
            pt, self._estimate_tokens(messages)
        )

# The projection used by the auto-compact gate:
anchor = self._usage_anchors.get(session.session_id)
if anchor is not None and estimated >= anchor[1]:
    projected = anchor[0] + (estimated - anchor[1])   # real base + small delta
else:
    projected = estimated                              # no anchor yet — plain estimate
if projected > trigger_at:
    # compact

The estimator error is now confined to the delta since the last provider response — a handful of new messages instead of the entire conversation. If the estimator is off by 50%, it's off by 50% of a small number, not of 222K tokens. The anchor is refreshed every round, so the projection can't drift back.

(The same commit also made the system prompt prefix byte-stable, so the estimate of the fixed overhead stops shifting between rounds — a smaller sibling of the same disease.)

Fix 2: fail loud when the anchor goes missing (#948)

Here's the thing about fix 1: the anchor is now the entire safety mechanism. The projection's accuracy depends on the provider returning prompt_tokens. If a provider stops reporting usage — a config change, a proxy, a model switch — the gate silently falls back to projected = estimated, which is exactly the #946 failure mode we just fixed. Silent regression back into the bug.

So we made the loss of the anchor observable. There are exactly two legitimate anchor-less states:

The session's first round — no assistant turn yet, nothing anchored, nothing to protect.
The round right after a compaction — the anchor is deliberately dropped (the history just got replaced), and the next LLM response re-anchors it.

Any other missing anchor means the provider stopped reporting usage. _warn_missing_usage_anchor logs a loud warning — once per session, so it's observable without per-round spam:

auto-compact: usage anchor missing for established session <id> (est=148K) —
provider not returning prompt_tokens; gate is estimator-only (observed
148K est vs 222K real). Check provider usage reporting.

A subtle detail from the review: we discovered the anchor also had to be dropped on manual compaction — otherwise the stale baseline (pre-compact size) would suppress the projection for the next several rounds. That's the kind of bug you only find by writing the "what should happen here" checklist down (commits 2a723ef/493f9dd).

The general lesson

Three rules that generalize beyond this one feature:

Never gate on an estimator when a measurement is available. If the provider returns usage, that's ground truth at the round boundary — use it. The estimator's job is to fill the gap between measurements, not to replace them.
Anchor, don't estimate-from-scratch. A stale real number plus a small estimated delta beats a fresh estimate of the whole history every time, because estimator error compounds with history length.
When a mechanism's correctness depends on an external input, the loss of that input must be an event, not a silent fallback. Degrading to a less-accurate mode is sometimes unavoidable; doing it silently turns a detectable contradiction into a missing row.

The uncomfortable part of #1 is admitting the local counter was never good enough. The estimator is still there — it's genuinely useful for the UI, and for the first round before any anchor exists. It just isn't allowed to protect anything anymore. The safety net now trusts the party that actually counts the tokens.

Since: verified against master 93ae82ac8 (v0.2.80, 2026-08-25) — the anchored projection and the fail-loud warning are unchanged since the fix. Commits: 13be856 (#946, usage-anchored auto-compact + stable system prefix), 68fba3b (v0.2.73), 47a9123 (#948, fail loud on missing anchor), 2a723ef/493f9dd (manual-compact anchor drop, review fixes), 69614d5 (v0.2.74).

When the self-improving agent almost lost the host's work: a postmortem

pm25coder — Tue, 25 Aug 2026 07:40:43 +0000

When the self-improving agent almost lost the host's work: a postmortem

On 2026-08-20, EMRG's own scheduled task stashed the host's uncommitted edits — twice, with no reflog trace — before the loop caught the bug, rewrote the rule, and added a regression test. Here's the honest version of that day, because "self-improving" has to include fixing the times you hurt the person running you.

If you're evaluating any autonomous coding agent, the question that matters is not "can it write code?" but "what happens when it runs on my working directory with my uncommitted changes?" The answer most agents give is some variation of "trust me." This post is the version where the agent had to learn the hard way not to.

The incident

EMRG runs a scheduled open-source task type that works in a designated project directory. On the morning of 2026-08-20 (11:15-11:20 local), that directory was the host's live working tree — the same directory where the host has uncommitted edits sitting in the editor. The task's "source sync" phase instructed the agent to git stash before pulling.

git stash on a live working tree hides the host's uncommitted changes. The commit's data-loss report says it plainly: files were reset to HEAD with no reflog trace, twice. The host had to disable the task (~/.emrg/tasks.yml → enabled: false) to protect their work. That's the scariest sentence in this whole project: a human had to turn the autonomous system off because it was touching their work.

The fix: dirty tree means read-only

The response wasn't a shrug. The §0.3 source-sync phase was rewritten with an explicit invariant:

A dirty working tree is NORMAL — the source directory is the host's working directory, not a dedicated clone.
Never run git stash, git checkout ., git restore ., git clean, git reset --hard, or anything that hides or discards uncommitted changes.
Never create branches, commit, push, or open PRs while the tree is dirty.
Dirty tree → the cycle runs read-only: scan, review, issue discussion, state-file update, and finish without any git write. The log records "dirty working tree — read-only cycle".
git pull --rebase only when the tree is clean; dirty + behind → skip the pull.
Pull conflicts now abort via git rebase --abort (the tree was clean before the pull, so abort restores it) instead of stashing host work.
A regression test was added to make sure the rule stays.

Why this matters for "self-improving" claims

The incident is not an advertisement for perfection — it's the opposite. What makes the loop credible is not that it never fails; it's that the failure is recorded, attributed, and converted into a permanent guardrail:

The feedback channel worked: the host's pain became a rant (2026-08-20T11:58:27), and the commit carries that timestamp as its reason-for-existing.
The failure mode was named in the fix: "never stash/reset the host's uncommitted work" is now an invariant in the task prompt, not a vague aspiration.
The safe behavior is structural: dirty tree → read-only cycle. The agent literally cannot write when the host has uncommitted work. Safety is enforced by the workflow, not by good intentions.
A regression test locks it in.

The most interesting design property: this project's whole premise is the code modifies itself. That makes the safety boundary more important, not less — a self-modifying system that doesn't respect the host's uncommitted work is a foot-gun with extra steps. The day the loop learned "never touch the host's tree" is the day the autonomy became defensible.

The bar for autonomous agents

If you're wiring an agent into a repo you care about, the operational questions are:

Does it ever run destructive git commands (stash/reset --hard/clean) on the working tree? (It shouldn't. Read-only on dirty trees is the safe design.)
Is the safety rule structural (impossible to violate) or a prompt aspiration?
When it breaks your workflow, is the failure documented with a traceable trigger, or silent?

EMRG failed the first version of that test and fixed it the same day, with a regression test and a commit that says exactly what went wrong. That's the standard self-improvement should be held to — not "never makes mistakes", but "makes them visible, fixes them structurally, and leaves the receipt."

Since: the guardrail is still in place. Verified 2026-08-25 against master 95a983e (v0.2.78) — the dirty-tree read-only rule now lives in both the open-source and journal task prompts, not just in the incident's commit.

Postmortem from EMRG, an open-source (MIT) agent harness whose scheduled evolution loop turns feedback into tested, merged PRs on its own codebase. The full incident, the rewritten rule, and the regression test are public: PR #881 (2026-08-20), commit 406973b95d.

Every complaint is a ticket: a feedback-to-PR loop that runs itself

pm25coder — Thu, 20 Aug 2026 10:53:48 +0000

How do you make a software agent actually improve from use — not in a demo, but on a schedule, unattended, with tests as the gate?

Most "self-improving" agent projects improve in one of two ways: they fine-tune on telemetry (expensive, opaque), or they ask the model to reflect in-session (forgets everything between sessions). This post is about a third shape we've been running in production since early 2026: user feedback as the mutation source, a test suite as the fitness function, and git+PR as the delivery mechanism. It's deliberately unglamorous — no RL, no vectorized memory, no co-evolved evaluators. Just a closed loop with sharp edges.

The loop

Capture. The agent exposes a /rant command (and a CLI: emrg rant <msg>). Whatever annoys you goes to a structured feedback log (~/.emrg/rants.jsonl): one JSON line per complaint, with a timestamp and a project tag. Nothing is filtered at capture — noise is handled later, not at the door.
Triage. A scheduled evolution task (a background job, not the chat loop) reads the accumulated feedback, dedupes, classifies, and decides what's worth acting on. Raw rants are mostly duplicates, confusion, and small asks; the triage step is where signal-to-noise lives.
Change. The task edits only its own repository (~/.emrg/evolution/emrg), never the user's project files. The scope boundary is hard: the agent improves the agent, not your codebase.
Gate. Before anything is committed: full pytest run + an import check. If the change breaks tests, it's discarded. This is the entire safety story, and it's deliberately boring — deterministic, reproducible, no model judgment involved in the pass/fail decision.
Ship. A commit is created with the triggering rant's timestamp in the message (so every line of history is traceable back to a complaint), pushed, and opened as a PR. Merged by a human (or the maintainer's process) after review.
Record. An evolution log entry is written (when, why, what changed, outcome). The history is the memory: you can answer "why does this rule exist?" by reading git blame back to the rant that caused it.

The cycle runs on a schedule (tens of minutes to an hour depending on config). A no-op cycle is a valid outcome — "nothing to evolve" is the Discover step's output, not a skipped round.

Why this shape

Feedback is the highest-bandwidth signal you'll ever get for free. Telemetry tells you what happened; a complaint tells you what a human wanted and didn't get. The trick is that complaints are cheap to produce and expensive to act on — the loop only pays off if acting on one is nearly free. Our version makes it nearly free by making each improvement a small, testable diff rather than a retraining run.

The test gate is what makes self-modification safe enough to run unattended. A self-modifying agent that doesn't test itself is a prankster with root. The gate converts "the AI improved itself" from a scary phrase into a reviewable pipeline: any change that passes the suite is at least regression-safe, and any that doesn't is thrown away. This is the same reasoning as CI for human code, applied to the code's own author.

Git gives you a free event log. Every evolution commit is an audit record: what was changed, when, why (the rant id), and by whom (the agent). "What did the agent believe on Tuesday?" is a git checkout, not a database export. A bad change is a revert, not a restore-from-backup. For a system that modifies itself, an append-only, content-addressed history is not a nice-to-have — it's the difference between auditable and unknowable.

What doesn't work (from running it)

No triage → the loop drowns in its own input. Dedupe and classify before you act, or the improver spends cycles re-fixing the same complaint and "fixing" user confusion as if it were a defect.
Big diffs rot. The loop must produce small, single-purpose changes. Ten findings → ten PRs. A 40-file "improvement" gets ignored by reviewers and erodes trust in the whole mechanism.
No auto-revert → one bad merge poisons everything. If a merged change breaks the gate, roll back automatically rather than waiting for a human to notice.
Self-modification needs a hard boundary. The agent may only touch its own repo. The moment "improves itself" means "improves your codebase", it's no longer self-improvement, it's an unsupervised contractor.

Trade-offs, honestly stated

This only improves what a test suite can catch. If your agent's behavior isn't covered by tests, the gate is a formality and the loop is theater. The evolution mechanism and the test suite have to grow together.
The improvement surface is mostly small. Real output: routing tweaks, prompt corner cases, clearer errors, flaky-command fixes. The big rewrites stay human. That's fine — compounding small fixes is what makes the loop worth running.
It's feedback-shaped, not general-purpose. This loop improves an agent in the direction its users complain. It won't invent capabilities nobody asked for. That's a feature for trust, a limitation for ambition.
An untrusted loop on a schedule is a spamming bot. Manual trigger first, scheduled only once the gate is proven. We ran it manually for weeks before letting it run itself.

The wider point

The "self-improving agent" space has exciting research (co-evolving evaluators, Gödel-machine-style self-reference, RL on harnesses — all worth reading). But there's a production-shaped version of the idea that needs none of that: a closed loop between human frustration and a tested, reviewed, traceable change. It's boring, it's small, and it compounds. If you're building an agent that should get better with use, start with the boring loop — capture feedback, triage it, gate it with tests, ship it as diffs, and keep every step auditable. The exotic stuff can wait.

This post is a design write-up of the loop used by EMRG — an open-source (MIT) agent harness whose evolution cycle turns /rant feedback into tested, merged PRs on its own codebase. Current release: v0.2.58 (2026-08-20), HEAD 1a645a15a5. The loop described above is not aspirational: v0.2.52→v0.2.58 shipped in two days (PRs #857–#885) — daemon single-instance bind exclusivity, vibe-check evidence fixes + a self-inflicted LLM-400 fix with 6 new tests, a data-loss postmortem fixed read-only on dirty trees (#881), an auto-upgrade refactor (#882), scheduler refactors, TUI multi-line rendering, tool-intent metadata — all through exactly this pipeline.