kimiya

Case study · the extension

Script as brain, model as eyes end-to-end software testing as a Kimiya program

The only test that exercises software the way people use it — click the button, look at the screen, hear the sound — is the one teams abandon first, because it breaks constantly. This case study rebuilds it on a simple split: a fixed script decides everything; a model is used only to perceive — and the verdict never rests on perception at all.

§1 · the problem

Two bad options, and the gap between them

The setting

You want to test an application end to end: open it, run through a real user's task across a desktop window and two different browsers signed in as two different people, with live audio and a real backend, and confirm everything actually worked. The two standard approaches both fail:

brittle scriptsClassical automation clicks by invisible internal names ("button #submit-3"). Rename a widget and the test breaks; surfaces you don't control — a system file dialog, another vendor's page — have no names to click at all.
improvising agentsAn autonomous AI agent can operate any interface — but it never runs the same test twice. A test that changes each run cannot be reviewed, diffed, or trusted as a regression check.

The split

The resolution: keep what to do deterministic (a fixed, reviewable step list) and delegate only what things mean — find the button described as “the red End button,” decide whether the screen shows a live presentation, read the 8-character join code off a dialog — to a vision model. And crucially: a screen that merely looks right must never be able to pass the test.

§2 · the kimiya reading

The eyes are instruments; the verdict is kernel

selectFinding a control from a description, on a screenshot, is retrieval with measured recall — and a miss is loud (the step retries or the test visibly stops), never silent.
judge“Does the screen show X?” is a judgment under a purpose: preserve the key widgets and text, allow loss of theme, cursor, clock, unrelated windows. That purpose is exactly why one assertion works in light mode and dark mode alike.
genReading a value off the screen (that join code) is untrusted generation — never relied on directly, always confirmed downstream by a database check.
checkThe verdict. Every test ends in non-visual kernel facts: database counts, the public backend, audio waveform peaks. Certainty 1, no instrument error.
act · settleThe extension's part. Clicks and keystrokes are act effects; waiting for a screen is settle … within — re-observe and judge until the state appears or a real deadline passes, never a blind sleep.

In code — one interaction, then the verdict

a := observe screen<A>()
btns := select<0.97>("the Create Group button", a) under k_ui by L
b := first(btns)
act<A> screen.click(b.x, b.y)
settle<A> until check group_exists("the group") within 12

-- THE VERDICT: kernel first, judged second, and only then commit.
b3 := observe screen<B>()
if check( group_members("the group") == 2 ):     -- kernel: certainty 1
    if judge<3,2/3> shows(b3, "the chat displays the message") under k_state panel [J1, J2]:
        commit(trace)
    else: abstain
else: abstain

Reading the verdict

The ordering is the design. The kernel fact comes first — the database really does contain two members, certainty 1, no instrument involved — because it carries the substance: the feature worked. The judged claim comes second and adds only what a kernel cannot see: that the user-visible screen showed it. commit fires only past both; and every other path — count wrong, screen unconvincing — lands on abstain, a visible refusal rather than a quiet pass. A test that cannot meet its bar says so.

§3 · the safety fact

Why a wrong glance cannot fake a green

The structure

Perception gates progress; the kernel gates the verdict. If the model misreads a screen and lets the script continue, the lie is caught at the end by a database count or a waveform peak that perception cannot forge. So a judge's mistake converts into a wasted retry or an honest abstention — never into a false pass. Across the entire evaluation campaign of the tool this study draws on, the vision layer produced zero false passes.

The yield

On a real, deployed application, eighteen such scenarios found ten genuine bugs that a 65-test conventional suite had missed — every one an integration failure, and three of them invisible on screen: a counter that lied, a feature whose code was never wired, and a backend fault returning an error to every signed-in user. Screens looked perfect; the kernel oracles did not blink.

The numbers

▪ the eyes, measured — from the campaign's own run artifacts
β  screen-judgment true-accept≥ 0.976
ρr  per-call locate recall≥ 0.975
α  false-accept (few chances to err — bound is loose)≤ 0.16
certified reliability of a representative passing test≈ 0.879

Two of those numbers repay a closer look. The α bound is loose for an honest reason: in a campaign of mostly passing runs, the judge is rarely shown a wrong screen, so there were only nineteen true-negative trials to measure it on — and a bound from nineteen trials cannot be tight. Purpose-built calibration suites exist to pin exactly this number. And the 0.879 is not a grade anyone assigned: it is the invoice multiplication from the tutorial’s §6 — eight locates at recall ≥ 0.975, the judged gates at β ≥ 0.976, one judged final claim — multiplied out, with the kernel conjuncts contributing exactly 1.

Key insight

That 0.879 is the tutorial's reliability invoice on a real system: eight locates, four judged gates, one judged claim, kernel conjuncts at certainty 1. And note which number is weakest — the false-accept bound α — and that the design is robust to exactly that number, because tests end in check. The architecture spends its trust where its measurement is strongest.

§4 · why it matters

A green run becomes a priced guarantee

A conventional test suite answers “pass or fail.” This program answers with a certificate: which claims hold at certainty 1 (the counts, the waveform, the backend state), which hold at the instrument's measured rate (the screens looked right, at β ≥ 0.976), what it cost, and which datasheets it all cites. A reviewer — or another AI agent deciding whether the feature is safe to build on — reads the invoice instead of re-running the test. That is the extension's promise kept: the world got touched, and the accounting never changed.

Next case

Hiding in plain sight →

Here the world-effects were recoverable — a stray click can be walked back. What happens when they aren't?