Testing & benchmarks

One set of focused runners covers fast correctness checks, real-browser behavior, performance regressions, and the benchmark harness behind the live benchmarks page.

This page is the source of truth for running and maintaining Minnow's tests and benchmarks.

Choose the smallest runner that proves the change

Install the browser binaries once before using a browser runner:

npx playwright install chromium firefox webkit

Everything above also runs in CI: .github/workflows/ci.yml on every push and pull request, and .github/workflows/performance.yml nightly. The performance gate is deliberately kept off the merge path — it measures a machine as much as it measures the code, and a merge gate that fails on a noisy runner is one people learn to re-run rather than read.

Each browser runner owns its test directory, server, port, and build prerequisites. Shared browser defaults live in playwright.shared.mjs; runner-specific configuration stays in its named playwright.*.config.ts file. This keeps every runner independently callable without hiding which application it starts.

What each layer proves

• Unit and conformance tests stay beside the source they exercise. They cover deterministic behavior, SQL and mutation conformance, differential execution, fault injection, and the benchmark generator/oracle contracts. npm test picks up apps/site/bench alongside packages/**, so the dataset generator, the oracles, and the suite definitions are checked on every unit run.
• Library browser tests exercise IndexedDB and transaction behavior that a Node substitute cannot prove, and drive a database through a real module worker: the published entry booting, transferred buffers arriving intact, and a second worker reopening what the first one wrote.
• Site browser tests execute the public examples and drive the benchmarks page itself: one case runs a suite end to end in each browser and asserts it verified against the oracles, another asserts the /benchmarks route is cross-origin isolated. Documentation cannot drift into non-running sample code, and the harness cannot drift from the page that runs it.
• The performance gate detects regressions on stable seeded query shapes, reads and writes alike. It is a guardrail, not a published cross-engine benchmark: the comparison engines run without indexes, which is fine for noticing that Minnow got slower and useless as a fair cross-engine claim.

When adding a runner, give it one responsibility and make it independently runnable. When adding a suite case, derive smoke-test counts from the suite itself instead of copying totals into tests or HTML.

Seeds, and how a soak failure becomes a permanent test

The generative suites — SQL conformance, DML conformance, and the columnar-versus-row differential — build their corpora from a seeded generator. A committed run is deterministic: it uses the checked-in seeds plus every seed that has ever failed, listed in packages/core/regression-seeds.json. That makes the suite a reliable regression net, and on its own it would be nothing else, because the questions it asks never change.

npm run soak is the other half. It runs the same suites on seeds nobody has tried and stops at the first failure, printing the seed:

npm run soak -- --rounds 200

A failing seed is the whole artifact. Replay it directly:

MINNOW_SEED=1476318588 npx vitest run packages/core/src/engine/sql-conformance.test.ts

Then add it to regression-seeds.json under the suite that failed, where every future run picks it up. Never remove one: a seed in that file is a bug that used to exist, and the entry is what stops it coming back. The explored space only grows, and it grows by exactly what the soak found.

Format compatibility

A browser database's data lives in the user's browser, so it outlives every deploy. There is no migration window and no way to reach back and rewrite it: if a format change makes yesterday's blocks unreadable, the first anyone hears is a user whose application will not open.

packages/core/format-fixtures/ holds one frozen database per released format version — a snapshot, which carries the raw block bytes verbatim, plus the answers that build gave to a fixed set of queries. format-compatibility.test.ts opens all of them on every run, checks the answers still hold, and checks that writes into a restored database still work.

It also fails when the current BLOCK_FORMAT_VERSION or SNAPSHOT_FORMAT_VERSION has no fixture behind it. That failure is the useful one, because it fires before the damage:

npm run fixture:format

Run it on the build that still writes the old format, commit the fixture, and only then change the version. A fixture can only be produced by the build that writes it — once that code is gone, the format it wrote cannot be regenerated. For the same reason, never delete one.

Running out of quota

quota.test.ts covers what happens when the browser refuses a write because the origin is out of space — the characteristic way a browser database fails, and the one an application most needs to handle deliberately. It pins four things: the write fails rather than half-landing, the error keeps its QuotaExceededError identity so an application can branch on it, everything committed earlier stays readable, and the same write succeeds once there is room, with no repair step.

Property-based tests

block-format/properties.test.ts states the format's contract as properties and checks them against generated inputs rather than chosen ones: a value written and read back is the same value, both codecs agree, a zone map contains everything it summarizes, and a flipped byte is detected rather than decoded into plausible rows. The generators reach for what breaks encoders — negative zero, the extremes of the double range, subnormals, empty and astral-plane strings, all-null and empty columns. A failure shrinks to a minimal case and prints the seed.

Soaks

Two suites cover accumulation rather than a single operation, which is where a fold that loses a row or a compactor that stops folding actually shows up:

• compaction-soak.test.ts runs 1,500 interleaved mutations against a reference Map, compacting at checkpoints, and compares the whole table to the reference each time. Compaction is bounded and incremental — one call folds a limited number of blocks — so the contract asserted is monotone progress, not a fixed point reached in one call.
• concurrency-simulation.test.ts runs twelve independent databases over one shared store, the shape of twelve browser tabs on one origin, issuing a seeded random schedule. It checks that the outcome is explicable: every visible row was written by some tab, acknowledged writes are present, no key appears twice, the write the store acknowledged last is the one visible, and every tab sees the same database afterwards.

Concurrent writes

write-contention.test.ts documents a sharp edge. Writes commit optimistically and rebase on conflict, up to maxCommitRetries attempts. Issued sequentially, they all land. Issued concurrently against IndexedDB, at most maxCommitRetries + 1 land — and that ceiling does not move with the number of writers, because each commit that wins costs every other in-flight writer one retry. Sixty-four parallel writes leave the same nine winners as sixteen do.

The tests guarantee the losses are clean: a rejected write leaves nothing behind, an accepted one is fully present, and which is which is deterministic. An application issuing parallel writes to one table should await them, or retry the rejections.

The fault sweep

packages/core/src/testing/fault-sweep.test.ts runs one fixed write workload many times, failing a different storage operation each time — the first block write, then the second, then the first block read, and so on through every operation a clean run performs. After each interruption it reopens the store and asks whether what survived is coherent.

The invariant is atomicity, not durability. A write interrupted mid-flight may be present or absent; both are correct. What is never correct is a torn state — a row nobody wrote, a duplicated key, a database that will not reopen. Every assertion is phrased as "the surviving state is one a caller could have observed", never "the write landed".

The set of fault points the workload reaches is pinned in the test. If a change starts or stops routing writes through one, the sweep fails and someone decides whether it should follow.

Benchmark workload model

The benchmarks page keeps four workload classes apart. They are never combined into one score:

A read workload keeps every query visible rather than collapsing to one figure: each cell is the median of the timed windows for that query, after one untimed warm-up. Writes keep every operation and batch size visible the same way. Nothing is reported unless it was verified — every read must match an independent JavaScript oracle and every write is read back and compared row for row, and a query an engine got wrong or could not run prints a dash instead of a timing.

There is nothing to publish or regenerate. The benchmarks page has no checked-in numbers: it builds every engine and runs the suites in the visitor's browser, on the engines, suites, and dataset size they pick, and explains the measurement methodology, storage differences, caching, and memory caveats as it reports them. The suites themselves live in apps/site/bench and the page that drives them in apps/site/app/benchmarks.

To iterate on the harness, run the unit tests (npm test) for the generator, oracles, and suite contracts, then npm run test:browser:site to run a suite in real browsers. To try a change by hand, npm run site:dev and open /benchmarks.

Update the performance-gate baseline

After an intentional executor change, inspect the performance-gate output before updating its thresholds:

npm run benchmark:gate -- --update

Treat that update as a code change: explain it in review and run the gate again without --update.

Refresh the download-size comparison

The benchmarks page opens with what a browser downloads to run each engine. It is measured from the installed packages rather than quoted, so refresh it whenever a dependency version or the public entry point changes:

npm run benchmark:sizes

Each engine's browser entry is bundled and minified with identical esbuild settings, and the WebAssembly and data files it fetches at run time are added at their shipped size. The result lands in apps/site/components/bench/library-sizes.json, which is a scratch output — nothing imports it. The figure each engine shows on the page is the download field in apps/site/components/bench/config.ts, updated by hand from that run.