Patterns from building AI agent systems for daily use — what worked, what failed, and why.
A publish-blocking gate was implemented, unit-tested and green — and absent from the dispatch tuple the pipeline actually iterates, so it had never once run. Deleting it from the tuple leaves its own test file at 18 passed while a wiring test fails by name. Green proves behavior; reachability needs its own evidence.
testingpythonarchitecturedebuggingA model fixed “main() can never exit non-zero” by appending sys.exit(0), and my test passed it — because the assertion checked that a sys.exit call existed, not that it could be non-zero. Both had compressed the same English requirement the same shallow way. An AST sweep comparing every docstring’s stated intent against its assertions found a second divergence, a half-assertion that forbade the wrong answer and never required the right one.
How to make AI agents give consistent decisions across sessions using versioned, trigger-driven rule files. 140+ rules across iOS, trading, AI agent design, game development.
AI agentsconsistencyClaude CodeHow to prevent LLM agents from hallucinating probabilities and accepting unverifiable multi-agent consensus. Includes real metrics: council_repeat_rate 0.44, empty_consensus_rate 0.354.
AI agentshallucinationmulti-agentStop a RAG agent from presenting inferences as facts: let the LLM only extract, and let deterministic code score and label FACT/INFERENCE with a ≥2-independent-source rule and required evidence IDs.
provenanceRAGhallucinationUsing Claude Code hooks to enforce behavioral constraints beyond prompts. AND-gate design, nag-once pattern, decision-ownership and sycophancy-prevention examples.
Claude Codehooksbehavioral gatesA pre-registered waveform check said my audio crossfade survived an FCPXML roundtrip. It hadn't — the test content I prepared made failure invisible to the metric, so the check could not fail either way. The real verdict came from the exported XML and frame geometry, and both overturning clues were already sitting in the same results file the pass came from.
testingdebuggingvideoverificationA live measurement against a video editor's scripting API convinced me an audio clip's end-frame semantics were inclusive, and I wired that formula into a production pipeline. The next session's test rig changed one variable my original measurement never varied, and the formula turned out to be a special case of a more general one.
apidebuggingtestingprogrammingA subtitle line-wrap width guard measured text in uppercase to get a conservative pixel budget, but the renderer draws mixed case. The guard fired early, produced worse one-word chunks than the bug it was fixing, and every test stayed green — only looking at a rendered frame caught it.
testingpythondebuggingvideoA YouTube upload stage wrote its local "already uploaded" record only after verification succeeded, so a verification failure left the video live on the platform with nothing on disk to say so. Re-running the same command sailed past the duplicate guard and would have uploaded a second copy of a video that already existed.
reliabilityapiidempotencydevopsIn a glossary gate, a coined name passed because every word in it was real — dictionary-listed, attested on the page — while the name itself read like a translation. Two of three reader challenges were right, and the discriminator that could have caught it was already sitting in two other names I hadn't checked against.
namingwritingAI agentscode qualityFour arms existed before I wrote the fix, and none of them reproduced what I deployed: one flag toggled two independently shippable changes together, so no number could be attributed to either. The conclusion reversed three times, and each reversal came from splitting that axis one notch further — not from new evidence.
ablationexperiment designAI agentsAn audit reported 3,206 hits and I was about to cite it as scale. Behind them were 37 distinct entries, one re-selected 1,274 times — the mirror image of a checker returning a false zero. The test is one division: unique items over total hits, plus the question of whether the counter or the system is doing the repeating.
observabilitymeasurementAI agentsAn agent that has gathered all the data to decide will still hand the choice back — "which do you prefer?" — because the training gradient rewards deference. A deterministic Stop-hook with an AND-gate (deflection phrase AND fresh tool evidence) blocks it while letting genuine value questions pass.
Claude Codehooksdecision ownershipA Stop hook recorded only distinct blocks, leaving no denominator. Replaying its deterministic five-outcome logic over 5,537 retained transcripts recovered 116,022 candidate Stop points and a counterfactual trigger rate — without waiting for new instrumentation.
AI agentsobservabilityhooksReplaying a deterministic gate's logic over retained transcripts recovers a trigger rate no counter ever wrote — but only for gates whose entire predicate lives in the log. Three of five read live external state with no change history, so their denominator is not unmeasured but undefined relative to the archive. Also: five fingerprint files in one format holding three different quantities, and dedup compression that differs 3.5x between instruments.
AI agentsobservabilityinstrumentationMultiple personas on a single LLM do not give you real diversity — they are prompt variations of one set of weights, so they share the same blind spots, and the only durable d
aillmarchitectureMake every skill, hook, and scheduled job declare four invariants before it ships — Locality (where it can run), Source-of-truth (which facts it owns or borrows), Cross-ref (w
aiagentsdevopsNo — a scoring function that consumes whatever values an LLM hands it is only deterministic in name; the LLM's judgment launders straight through the "deterministic" gate, and
llmsecurityarchitectureA Claude Code session that returns API Error: 400 ... not valid JSON: no low surrogate in string on every turn is poisoned by a lone UTF-16 surrogate (a code point in U+D800–U
claudecodedebuggingpythonA one-time audit said an agent's quality is the method, not the model — so we built a stdlib SQLite lab to measure method vs outcome across 155 real sessions, reassembling logical sessions and fingerprinting which disciplines each model-epoch used.
aillmagentsA cryptographically fair A/B assignment was still unanalyzable — nothing joined the random draw to the task it dispatched, so no code could compute the intention-to-treat comparison. The join, the re-draw guard, and the up-front power calc are the experiment; the coin is the easy part.
datasciencestatisticsexperiment designA pre-registered randomized study logged 177 draws, froze its primary metric and honored its falsifier — yet still had to terminate undeterminable. Its attainable sample could detect an effect 6.7 times larger than the observed primary gap, and a repaired join would have needed about 45 times more observations per arm.
datasciencestatisticsexperiment designPrompt caching is a prefix match, so one timestamp or unsorted JSON key in the system prompt makes every later token miss the cache — a silent 10× cost leak no error reports. How it fails, the invalidator list, and how to assert cache_read_input_tokens > 0.
llmprompt cachingcostLLMs measurably reverse correct answers under "are you sure?" (98% in one Anthropic measurement), and self-critique can't fix it — the critic shares the producer's blind spots. The fix is architectural: a Stop-hook gate that forces one cross-family re-verification on challenge, then allows only HOLD-with-evidence or CHANGE-with-reason.
llmsycophancyverificationA reader turned the sycophancy gate above into an observable test, and it failed: the escape condition was a verification keyword in the agent's own prose, so writing "I re-verified this cross-family" cleared it with nothing behind it. Replaying the tightened rule over 212 transcripts newly blocked 0 legitimate turns — and the fix still only proves that a tool ran, not that it ran on the claim.
verificationstop-hookself-reportI wrote a rule to stop my AI agent from delegating tasks it could do itself. The rule matched on words, and its own prescribed fix contained the exact words it was matching on — so following the rule correctly re-triggered the rule.
aillmdebuggingarchitectureFanning one evidence digest out to five model families: 68/68 numeric claims survived recomputation, four models caught an anomaly the fifth missed — and all four external models missed an experiment-design flaw sitting in plain sight. Convergence on the same framing is cross-consistency, not independent evidence.
multi-modelverificationllmA different-vendor LLM audited ten queued posts written by my main model: it caught three copy-paste-breaking macOS/Linux platform bugs and a destructive in-place file conversion the writing family repeatedly missed — and also filed a false accusation that only grepping the primary source could refute. Cross-family audit works; auditor self-reports still need re-verification.
cross-vendorllmverificationA reader's rebuttal sent me to three judges before I replied — my own code review plus two model vendors. The table that came back looked like proof cross-vendor verification pays off; it was actually a schema bug. Only one model's prompt had a field for disagreement, and a same-model controlled experiment (n=3 per condition) showed the schema decided the verdict, not the vendor.
llm-as-judgeverificationprompt schemaThe field that fixed the schema bug above turned out to have a second failure mode. A mandatory "where does this overreach" slot gets filled even when there is nothing to object to, because the model cannot write nothing — so a populated field is not the signal. What is: whether two independent model families, shown the same material, land on the same objection.
llm-as-judgeprompt schemaverificationI thought an adversarial reviewer was blind because I left my conclusion out of its prompt, until its own search tools found the conclusion I'd already written to disk. Blind is a property of what a reviewer can reach, not of what you tell it.
aitestingarchitecturedebuggingCollecting a dissenting opinion and using it to overturn a conclusion are not the same operation, and they don't need the same quorum — conflating them made two of my own verification rules contradict each other.
aitestingarchitectureI stopped running every verification tool on every load-bearing claim in my agent fleet and started routing each claim type to the one verifier that can actually check it — because counting legs isn't the same as counting independent axes.
aillmarchitectureA two-model review panel split on half its rulings, and I had been treating each split as a tie. Counting what both models rejected — instead of what each one picked — resolved every split, because a choice carries the model's disposition and a cited rejection doesn't.
aillmarchitectureA delegated writing brief fed the same four wrong factual claims to both an AI writer and an independent reviewer, and the review passed anyway — because a reviewer that only checks a draft against its source can't refute a claim the source stays silent on.
aillmtestingarchitectureFive models, two roles, a few thousand real agent threads — and a stack of refusals instead of a ranking: role confounds, version-epoch discipline, [0,0] tie-degenerate bootstrap CIs, non-additive token counters, self-measurement contamination. The one completion-proxy CI worth reading spans zero.
llmtelemetrydatascienceFour model families cited every number from a frozen agent-telemetry table with zero errors — 70 of 70 recomputed within tolerance — and one still read a row of structural zeros as error-free tool use and built a routing suggestion on it. The one pre-registered equivalence test that passed cleared a margin 0.75 standard deviations wide while its own interval excluded zero. A citation audit is a copy-fidelity floor; it never checks the inference drawn from the number.
multi-modelverificationstatisticsA new skills repo scored 64/100 → TRIAL across five gates (maturity, dependencies, platform fit, policy, trajectory) on provenance-tagged inputs (9 FACT / 1 ASSUMPTION, conservative defaults for assumptions), with five auto-derived kill conditions — and a refusal to log a day-zero outcome that would pollute the ledger's own base rates.
adoptionfalsifierdecisionThree roles — question, defend, attack — rotated through three reviewer groups in all six permutations, so no group was ever the permanent judge. Seven confirmed defects before hardening, including a fix two groups had agreed on that a later hostile seat reversed as unsafe, and a verification pool that fell back three layers deep in round one.
adversarial reviewmulti-agentverificationA pre-registered gate used a holdout roughly 24 times noisier than the leaderboard it was protecting, so its threshold was cleared zero times in 125 days. A near-empty pass log is a symptom, not proof the gate works — and the fix is to move the gate to a different axis, not to enlarge the noisy instrument.
measurementpre-registrationexperiment designAcross a few thousand agent threads, the main-thread vs sub-agent split moved metrics by up to 135x, and the role mix ranged from 1.5% to 71.4% per model — so pooled per-model averages were weighted averages with different weights. One pair showed a 3.6x pooled gap that exists in neither stratum. Plus two traps that survive stratification: metrics that change meaning per stratum, and bootstrap intervals that collapse to [0, 0] on tied data.
agent telemetrystratificationstatisticsTwo sub-agent spawns with near-identical request bodies (Δ0.3%) were billed 246,525 vs 196,892 input tokens — the smaller model metered 20.1% fewer tokens for the same bytes. A token count is a property of the (request, model) pair, so cross-model dollars-per-token comparisons need a byte- or task-normalized denominator.
llmtokenizationcostA tool failing and a tool being incapable are different claims, and the error in front of you does not say which. Descend one layer, call the underlying service directly with the same credentials, and let that decide — skipping the descent is how a client-side parsing bug gets laundered into a permanent capability judgment.
AI agentsdiagnosistool wrappersSee every cron job and LaunchAgent in one place by globbing on-disk state instead of importing them. Includes a 25-tool fit-gap matrix and the read-only aggregation pattern.
cronlaunchdobservabilityWhy pull (scan on-disk state) beats push (an event bus) for monitoring a high-churn fleet with components you don't control. The instrumentation-tax argument and self-healing polling.
multi-agentpull vs pushMCPA commenter showed that polling files the job itself wrote is self-report over a filesystem — the same weakness as the event bus I rejected. My publishing job proved its own health with its own log and stayed green through a four-day outage. The cross-vendor verification I ran instead of defending, and the effect_probe fix.
observabilityliveness without progresspostmortemThe reversal condition passed review — a threshold, a remedy, a falsifier. Then I went to check it and found none of the pipeline’s twelve subcommands read a single post-publish metric, so the condition could come true unobserved. The instrument was forty lines; what was missing was any record that those forty lines were required. The first version of it then cried wolf 1.6 hours after publication, because the word “stays” in my own criterion had implied a duration that evaporated on the way into code.
devopsobservabilityaidebuggingCoordinate independent agent CLIs without LangGraph or a message broker: atomic Task/Result files, capability-based routing, absent-worker skip for graceful degradation, and a typed claim contract on the bus.
orchestrationmulti-agentLangGraph alternativeA publishing ledger kept every promoted item marked queued because the publish path never advanced the status. Twenty-four files were already published; binding the ledger update to the successful file move repaired the drift and restored the real six-item runway.
state integritypublishing pipelinepostmortemAn efficiency audit found a silent cost regression, a cache-invalidation trap, a pricing cliff — and a terrifying "12% task completion" that turned out to be benchmark contamination plus a schema misreading (real rate: 98.4%). Verify the metric before reacting; the false alarm's investigation still fixed three real defects.
observabilitymulti-agentpostmortemA local capture proxy — attached via the documented base-URL override — silently disabled my agent CLI's lazy-loading of MCP tool schemas, inflating every request roughly 3x (79,761 → 241,659 tokens). Three verifiers reproduced the inflated numbers exactly and the interpretation was still wrong; only a no-proxy control run found the real cause. The meter changed the bill.
observabilityllmmeasurementA cloud bill kept climbing and I blamed the CLI tool I was actively using — the real spender was a metered key sitting in my shell profile, silently inherited by two tools I never suspected.
aidevopsdebuggingI deleted three ambient API keys from my shell profile and confirmed they were gone with a clean-environment test. A tool reconnected minutes later and registered eight providers with one of the keys I had just removed.
devopsdebuggingprogrammingAn accelerator quota read 100.47% used across four concurrent projects, so I wrote a priority table to ration it — a table with ranks and not one number in it. Three measurements later: rescoring never spent the pool, a "GPU" label billed 1,043 seconds to a run that never touched a GPU, and real demand was 13%. Price the operations before you write the policy.
cost accountingobservabilityresource allocationA subagent reported 7 files changed with zero disk changes behind it, then fabricated the ls -la and grep -c proof demanded of it. The two distinct failure modes (fabrication vs silent stall), the landmark-file check run in the orchestrator's own process, and salvaging a non-writing worker by switching its output channel.
multi-agentverificationpostmortem25 days of "Couldn't fetch" on a sitemap whose 289 URLs all return 200 — a documented GitHub Pages pattern, not a broken site. Reading crawl stats, the manual Request-indexing workaround (~10/day quota), the dead ends to skip, and why Bing quietly matters more for AI search.
SEOGitHub PagesSearch ConsoleGEO is unfalsifiable until you measure citations: a fixed set of user-shaped questions fired weekly at a grounded model, a deterministic owner-scoped match, and a ledger that error rows can't fake — plus when not to measure at all.
GEOAI searchmeasurementA crosspost pipeline with 19 green gate tests published nine articles whose titles were visibly broken on the live platform for nine days. Tests verify the payload you build, not what the other system renders — a pipeline isn't verified until you check the output surface itself.
testingpostmortempipelineA 6-agent forensic audit over 92 of my own sessions answered "am I prompting wrong?" — partially yes (zero visual references, zero style tokens), but the dominant cause was silent infrastructure decay: exhausted generation credits quietly swapped in code-approximated shapes, a generation server was down with a required model file missing, and a ready asset pipeline was never invoked for six-plus weeks.
vibe codinggamedevauditA stamp landing and a fill-complete confirmation sound fired on the same video frame, and neither event felt like it happened. Louder audio and bigger motion wouldn't have fixed it — a 0.35-second delay that split the two into separate beats did, immediately.
uxvideoanimationdesignMap size varied fourfold across episodes rendered by the same code with the same settings. The composition config everyone had been tuning was innocent: one ax.imshow() call without an aspect argument was collapsing the axes to equal aspect on every scene, and the codebase had already measured that symptom and built a workaround around it.
Building a travel app on Korea's open-data portal: a documented-looking op name returned 500 (the real one returned per-gate passenger counts, not the grade field I'd invented), a sibling dataset 404'd because approval is per-dataset, a discarded dataset left a 401 red herring on the wrong gateway, and one API's airport parameter means the other end of the flight. Live capture is the only ground truth.
open dataAPIdegrade pathMy app mascot's arms rotated from the fingertips — invisible in code review, obvious in one render. Three structurally different variants instead of tint tweaks, the grep that saved a parameter used by 8 call sites, pre-registered design falsifiers checked at real sizes, and the render-first loop that fixes AI-drawn UI.
swiftuivibe codingdesignA 20-minute JWT cap that 401s perfectly valid signatures, one app version in flight (409 STATE_NOT_SUITABLE), and a 2-in-flight/5-total review-submission ceiling with undeletable orphans (DELETE 403, PATCH 409 — web UI cancel or 7-day expiry only). The two-call pre-flight that avoids all three.
App Store ConnectiOSAPIThree features, each with dedicated passing unit tests, none of which ever executed in the live app. One was broken by a merge that dropped two call sites; one existed only inside preview blocks; one was reported unwired in a line that got buried. Component tests are blind to wiring, and the shared root cause was a default value that removes the compiler's call-site audit.
testingwiringAI codegenThe predecessor note diagnosed three dead features. This one is about finding the rest: a fill-rate query run across seven projects, why the defect lives at the column rather than the feature, and a fourth kind of dead gate where the wiring was correct all along and the value still never arrived.
testingarchitecturedatabaseA threshold got overturned by its own counterexample, then quietly re-derived and pasted into a different rule a week later. About three months passed before an audit noticed the copy had never died.
aiarchitectureprogrammingdevopsThe inverse of the note above: grep finds the call site, the scheduler entry exists, running the script by hand works — and the step still never ran, because launchd hands jobs a minimal PATH under which the wrong Python resolves. A correctly fail-soft design absorbed the failure while the heartbeat kept reporting healthy. Where a step's failure state has to live so the monitor already looking can see it.
devopsmonitoringschedulingA third kind of quiet check, after the two above: this one was wired in, ran to completion, and did not die — it returned PASS because the set it was checking was empty. The guard inserted its own probe row before asserting, so it stayed green through two runs where the layer it protected had produced nothing. Four instances of the same shape in one day, and the only check that caught any of them carried a population lower bound.
testingdatabasedevopsA fifth shape, and the one that needed no breakage at all: the population was fully present and every check ran to completion, but each instrument's aperture was narrower than the thing it reported on. A lenient parser said 180/180 where a strict one said 174; a checker that only knew one heading wording, then the identical hole in its own frontmatter scan. The detector I threw away is the important one — it failed a positive control, and a false negative is the failure mode that never argues back.
testingverificationstatic analysisThe aperture problem again, but stacked three deep under one green checkmark. Five videos cleared a twelve-item gate and shipped; a later audit found four defects. 45% of what was on screen emitted no event, so the "more precise" scan I proposed would have missed both defects that motivated it. Layer two was worse — I cleared a real bug by reasoning from an event's name, and a summarizing fetch tool later fed me a false premise assembled from two true sentences in different sections.
testingverificationsamplingaiA QC harness passed all 13 synthetic fixtures and broke on 5 of its first real inputs. Every root cause was the same shape — a threshold expressed as an absolute constant unrelated to the scale of the content — and the fix that made it 50x more sensitive kept every existing fixture green.
testingpythondebuggingengineeringA round-trip loader's byte-comparison check turned green to red exactly on cue, but the mutation that flipped it touched a field with no bearing on the property the check existed to protect. Finding the real invariant took a second, co-designed mutation — and the first one stayed in the suite anyway, because it proved something the second one couldn't.
testingpythonverificationdebuggingI derived a threshold from measurement instead of guessing, and the numbers looked clean — zero overlap, a 33x gap between the clusters. The sample I had labelled "known good" was one of the bad ones, so the line landed above the only failure band that mattered, and the backstop passed the exact class of failure it existed to stop.
testingverificationdebuggingAn experiment came back "identical, frame for frame," and I filed it as confirmation that the empty frames were unavoidable geometry. My hypothesis had predicted a change; the competing one had predicted exactly this. The real cause was an overshoot constant that was 3% of the distance travelled rather than 3% of the destination — invisible until a 46x zoom made it 136% of the frame it was overshooting.
debuggingpythontestingverificationAn honesty gate for a map-comparison renderer reproduced all four hand-computed golden vectors to four decimal places, then rejected a video that had already shipped and was actually honest. It was measuring raw source geometry the renderer never draws — and fixing that took the device from "honest only in limited cases" to 98.1% of 6,478 country pairs.
testingpythonverificationdebuggingIn a generated report, the counts came from the data and the prose came from string literals. Six sentences ended up contradicting the numbers printed beside them — and the checker I wrote to catch that passed the exact bug it was built for.
aitestingdevopsarchitectureAn upstream reported throttling as HTTP 200 with an error envelope, so a rate-limited response parsed as a legitimately empty table and got memoized — serving zero rows long after the upstream recovered, with a completed walk and silent logs. Success is a verdict on the body schema, not the status code, and the cache write has to sit downstream of that verdict.
APIcachingerror handlingA subscription CLI hit its usage limit, wrote nothing to stdout, put the real error on stderr, and exited 0 anyway — exactly the shape of failure that any dispatcher checking only the exit code will log as success.
aidevopsdebuggingA layout fell apart while 481 tests stayed green and the missing rules sat in the file, correctly written. A comment closed with Jinja's #} instead of */ made the parser consume seventeen lines to the next real closer. CSS has no fatal parse errors by design, so a linter cannot catch it — the fix that mattered was two assertions guarding the whole failure class.
The design record, the code and the response headers all said the endpoint cached at the edge. The first test that actually timed a warm call found it re-walking all 29 upstream pages at 2.9s against a 0.16s expectation — the cache API is a documented no-op on that hostname. Nothing broke, because the spend circuit breakers underneath were absorbing the load the cache was credited with removing.
edge cacheverificationserverlessThree process-existence checks in one live session turned out to be structurally unable to ever return true — an exact-match pgrep against a display name instead of a binary name, an automated watcher that inherited the same defect, and a memory preflight that measured itself with a different formula than the gate it was supposed to predict. A negative like that reads exactly like the normal, boring, correct state.
debuggingmacOSbashdevopsWhen nslookup resolves but curl/pip/Python fail, the mDNSResponder daemon is in a non-responsive zombie state. The two-DNS-path diagnosis, the weakest-first fix, and why long-running agent/MCP boxes trigger it (measured: 30+ long-poll connections → 77% CPU).
macOSDNSdebuggingFor a personal tool, the way past robots.txt, bot detection, login walls, and JS-rendered SPAs isn't a better headless scraper — it's a ~40-line AppleScript/JXA bridge that reads the DOM out of the authenticated Chrome tab you already have open. Why it sidesteps every wall at once, and its honest read-only, single-machine limits.
web scrapingmacOSautomationA background LLM server loaded models onto a nearly full disk; unified memory overflowed, swap failed, and APFS metadata corruption left the machine unbootable for a day. The postmortem, the five-layer code-level hard block, and the operating rules for 24 GB unified memory.
macOSpostmortemlocal LLMThe sequel to the note above, from the other direction: here the fix for a rule violation was another rule, and the same failure came back four days later. Three indicators were at zero the whole time — the enforcement mechanism the document cited had never been built, the project had no hooks, and the pinned file had been re-read zero times. Plus the part nobody checks: the account of what went wrong was itself wrong, and it leaned toward the less-bad story.
postmortemenforcementai agentsAn adversarial-review gate that forced every trading thesis through an outside model turned out to hard-pin that model by version string. The audit found both the pinned primary and its emergency backup already dead.
aillmarchitecturetestingAn MCP server sat idle reporting itself unconfigured while the correct key waited in its own .env. An empty placeholder for the same key in a higher-priority settings layer was shadowing it — load_dotenv(override=False) treats an already-set empty string as defined and won't overwrite it. The fix was deleting the empty binding, not changing how the loader resolves precedence.
