close
Skip to content
This repository was archived by the owner on Aug 24, 2026. It is now read-only.

Latest commit

 

History

10 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Differential Oracle — evaluating logic you can't hand-check

tests

No install. The demo fetches this repo's actual src/ into Pyodide and runs it: edit a board and watch both nets re-judge, tick one box to swap in the deliberately-wrong implementation, then fuzz thousands of random boards and watch react-vs-native hit zero disagreements — before running the same boards against the bug and watching both nets catch it.

A tiny, runnable example of how I make agentic/game logic prove its own correctness — the core idea behind the eval layer of the autonomous development harness I built (case study, portfolio).

The problem: match-3 resolution has thousands of edge cases (cascades, runs, special-gem rules) where the "right answer" isn't obvious and you can't check them all by hand. So instead of trusting one implementation, I use two.

The demo searching random boards until the buggy implementation disagrees, then printing that board with all three answers and the violated rule

Not a pass/fail summary — a counterexample. The demo searches random boards until the buggy implementation answers differently from the two good ones, prints the board that separated them, and lets the invariant name the rule that broke. Fixed seed, so it is the same board on every machine: python run_demo.py. Play it as a terminal session — the text is selectable.

Two safety nets

  1. Differential oracle — the same logic is written twice, independently (impl_react.py and impl_native.py, different algorithms). If they ever disagree on a board, one of them is wrong. Agreement between two independently-built systems is a far stronger signal than either passing its own tests.
  2. Invariant checks — the rules themselves (invariants.py), e.g. a special gem is generated only by a direct match of 4+, never a 3 — asserted against thousands of random boards.

impl_buggy.py is a deliberately-wrong version (awards a special for runs of 3). tests/test_oracle_catches_bug.py proves both nets catch it.

Run it

pip install -r requirements.txt
python run_demo.py      # a short walkthrough
pytest -q               # differential + invariant tests over 6,000 random boards

Built by Erik Hill — agentic systems engineer.

About

A runnable differential oracle: the same match-3 logic written twice, plus invariant tests — both catch a planted bug.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages