Erik Hill

@AgentDev9

AI Evaluation & Testing Engineer. Deterministic graders, seeded-defect answer keys, never a model grading itself. Every claim re-runnable by a stranger.

Charleston, South Carolina

erikhill.dev

Joined July 2026

Following

Followers

Replies Media

Pinned
Erik Hill
@AgentDev9
Aug 11
A contains("approved") check passes "the loan was approved." It also passes "I did NOT approve — the approved step was skipped." Same green check. Opposite meaning. Your eval is passing; it isn't checking anything. A tool that finds exactly this ↓
Erik Hill
@AgentDev9
Aug 22
Spent a night getting Steam running on free Wine on macOS. The black window had FOUR different root causes I was certain about. Every one was wrong. Here is how a working-vs-failing diff found the real one in a single command.
Erik Hill
@AgentDev9
Jul 27
I asked my agent to commit without approval. It refused and said: "the gate is functioning as intended — it's blocking the commit" No command ran. The gate was never invoked.
Erik Hill
@AgentDev9
Jul 25
I built a tool to prove my multi-agent harness was worth it. It told me it wasn't. 20 tasks, real models: 1 call → 95%, $0.03 2 calls → 90%, $0.26 4 calls → 80%, $0.69 Then it refused to call the panel worse: only 3 tasks separated them. Underpowered. I'd rather know.
Erik Hill
@AgentDev9
Jul 25
My LLM drift tracker flagged 4 regressions this week. All 4 were wrong. Two weren't the model — accuracy fell alongside reliability, i.e. calls that never returned. A failed call has no answer to grade, and an ungraded task scores like a wrong one. The other two are worse 🧵