Case study · the extension
The only test that exercises software the way people use it — click the button, look at the screen, hear the sound — is the one teams abandon first, because it breaks constantly. This case study rebuilds it on a simple split: a fixed script decides everything; a model is used only to perceive — and the verdict never rests on perception at all.
§1 · the problem
The setting
You want to test an application end to end: open it, run through a real user's task across a desktop window and two different browsers signed in as two different people, with live audio and a real backend, and confirm everything actually worked. The two standard approaches both fail:
The split
The resolution: keep what to do deterministic (a fixed, reviewable step list) and delegate only what things mean — find the button described as “the red End button,” decide whether the screen shows a live presentation, read the 8-character join code off a dialog — to a vision model. And crucially: a screen that merely looks right must never be able to pass the test.
§2 · the kimiya reading
act effects; waiting for a screen is settle … within — re-observe and judge until the state appears or a real deadline passes, never a blind sleep.In code — one interaction, then the verdict
a := observe screen<A>() btns := select<0.97>("the Create Group button", a) under k_ui by L b := first(btns) act<A> screen.click(b.x, b.y) settle<A> until check group_exists("the group") within 12 -- THE VERDICT: kernel first, judged second, and only then commit. b3 := observe screen<B>() if check( group_members("the group") == 2 ): -- kernel: certainty 1 if judge<3,2/3> shows(b3, "the chat displays the message") under k_state panel [J1, J2]: commit(trace) else: abstain else: abstain
Reading the verdict
The ordering is the design. The kernel fact comes first — the database really does contain
two members, certainty 1, no instrument involved — because it carries the substance: the feature
worked. The judged claim comes second and adds only what a kernel cannot see: that the user-visible
screen showed it. commit fires only past both; and every other path — count
wrong, screen unconvincing — lands on abstain, a visible refusal rather than
a quiet pass. A test that cannot meet its bar says so.
§3 · the safety fact
The structure
Perception gates progress; the kernel gates the verdict. If the model misreads a screen and lets the script continue, the lie is caught at the end by a database count or a waveform peak that perception cannot forge. So a judge's mistake converts into a wasted retry or an honest abstention — never into a false pass. Across the entire evaluation campaign of the tool this study draws on, the vision layer produced zero false passes.
The yield
On a real, deployed application, eighteen such scenarios found ten genuine bugs that a 65-test conventional suite had missed — every one an integration failure, and three of them invisible on screen: a counter that lied, a feature whose code was never wired, and a backend fault returning an error to every signed-in user. Screens looked perfect; the kernel oracles did not blink.
The numbers
Two of those numbers repay a closer look. The α bound is loose for an honest reason: in a campaign of mostly passing runs, the judge is rarely shown a wrong screen, so there were only nineteen true-negative trials to measure it on — and a bound from nineteen trials cannot be tight. Purpose-built calibration suites exist to pin exactly this number. And the 0.879 is not a grade anyone assigned: it is the invoice multiplication from the tutorial’s §6 — eight locates at recall ≥ 0.975, the judged gates at β ≥ 0.976, one judged final claim — multiplied out, with the kernel conjuncts contributing exactly 1.
Key insight
That 0.879 is the tutorial's reliability invoice on a
real system: eight locates, four judged gates, one judged claim, kernel conjuncts at certainty 1.
And note which number is weakest — the false-accept bound α — and that the design is robust
to exactly that number, because tests end in check. The architecture spends
its trust where its measurement is strongest.
§4 · why it matters
A conventional test suite answers “pass or fail.” This program answers with a certificate: which claims hold at certainty 1 (the counts, the waveform, the backend state), which hold at the instrument's measured rate (the screens looked right, at β ≥ 0.976), what it cost, and which datasheets it all cites. A reviewer — or another AI agent deciding whether the feature is safe to build on — reads the invoice instead of re-running the test. That is the extension's promise kept: the world got touched, and the accounting never changed.
Next case
Hiding in plain sight →
Here the world-effects were recoverable — a stray click can be walked back. What happens when they aren't?