Scale & coverage lane (suites/scale)

# Scale & coverage lane (`suites/scale`)

Pillar P5 of the benchmark program: every user-facing VCS operation timed for

git and oak, on deterministic synthetic repositories from "fresh" to XL, with

git-sizer-style statistics recorded as provenance on every row.

```bash

python3 suites/scale/run_scale_suite.py list

python3 suites/scale/run_scale_suite.py generate --tier default            # fixture + sizer only

python3 suites/scale/run_scale_suite.py run --tier fresh --tier default    # A/A-interleaved, runs=5

python3 suites/scale/run_scale_suite.py run --tier shapes --shape-scale 0.05

python3 suites/scale/run_scale_suite.py generate --allow-xl --tier xl-history --tier xl-refs

```

Set `--work` (fixtures and workspaces; large) and `--results` (rows) to a

directory on the benchmark volume, and `OAK_BENCH_LOCK_PATH` when several

lanes share a host. Raw rows are never committed (ADR-0003).

## Status of the numbers

`evidence_class` and `claim_eligible` are derived, never set by hand, and are

verified by recomputation (`scripts/oakbench/scale_calibration.py`):

1. `run_scale_suite.py calibrate --tier <t> --out rec.json` runs an

   identical-binary A/A campaign (git, git_aa, oak_installed,

   oak_installed_aa) plus host probes (load average, F_FULLFSYNC latency,

   small-file creation rate) and writes a record naming the campaign JSONL by

   content sha256.

2. `run --calibration-record rec.json` recomputes the campaign from that JSONL

   and stamps rows `calibrated_quiet_host` only if (policy `2026-09-29.3`,

   provisional):

   - the campaign has >= 10 runs per arm; A/A floors come from all runs PER ARM

     (git and oak separately): each arm's median floor <= 10% and at most 10%

     of its cells over 10%; identical-binary claims are counted on a held-out

     half with floors from the other half (a floor computed from the pair it

     judges can never produce a claim), and must be zero;

   - the host is quiet: load/core <= 0.10; the flush probe uses the platform's

     full-flush primitive (F_FULLFSYNC on macOS) with median <= 2.5 ms; >= 2000

     small files/s;

   - the work volume is physical media (statfs/`df` device and mount point,

     `diskutil` bus protocol of the APFS physical store): RAM disks, disk

     images and tmpfs are refused because they skip the flush cost Oak's

     clone/switch pay (fb-549);

   - calibration and run share runner id, host cpu count and memory, the work

     volume, the Oak/Git binary sha256, and registry-pinned fixtures; the run's

     own start/end disk probes and each row's load sample are quiet too.

   No idle baseline was obtainable on the shared dev host (its internal SSD

   read F_FULLFSYNC 3.7-3.9 ms under load); the 2.5 ms threshold may be

   unreachable even idle and must be re-derived on the dedicated runner (P6).

3. `scripts/publish_gate.py` refuses scale rows unless the campaign JSONL is

   among its inputs (matched by sha256) and repeats the same recomputation; the

   record's stated numbers are never trusted.

This catches edited, resealed, copied or invented records (QA's forged record

is the regression fixture `tests/fixtures/scale/qa-r2-forgedC.fresh.jsonl.gz`).

It does not authenticate the campaign JSONL itself: editing its load/probe

values (a few dozen numbers) and resealing a record to the new digest passes,

as would a fabricated campaign; that needs a signed runner. Without a calibration, rows are

`development_only_loaded_host`.

## Tiers (`scalesuite/tiers.py`)

The t/perf repository-tier concept (fresh / default / large; Git's

`test_perf_*_repo` helpers, GPL: concept only) with synthetic repositories:

| tier | files | depth | commits | branches | tags | binaries |

|---|---:|---:|---:|---:|---:|---|

| fresh | 100 | 3 | 1 | 1 | 0 | - |

| default | 5,000 | 6 | 1,000 | 50 | 20 | - |

| large | 100,000 | 8 | 10,000 | 1,000 | 200 | 1 x 256 MiB, 3 revisions |

| xl-history | 10,000 | 5 | 100,000 | 1 | 0 | - |

| xl-tree | 1,000,000 | 6 | 3 | 1 | 0 | - |

| xl-deep | 50,000 | 40 | 100 | 1 | 0 | - |

| xl-refs | 2,000 | 4 | 1,000 | 10,000 | 2,000 | - |

| xl-bigblob | 16 | 2 | 3 | 1 | 0 | 1 x 3 GiB, 2 revisions |

| xl-combined | 1,000,000 | 32 | 100,000 | 10,000 | 2,000 | 1 x 3 GiB |

XL tiers need `--allow-xl` (`xl-combined` also `--allow-xl-combined`). Each XL

tier pushes ONE axis past real-world "large" so a scaling defect is

attributable; `xl-combined` exists for the calibrated campaign. XL tiers skip

the per-run `clone.full` and `gc` where one run costs tens of minutes (skip

rows say so).

`shape:<id>` (or `shapes` for all 14) turns every scenario of

`scenarios/core.yaml` — the 14-shape catalog no lane used to run — into a

runnable tier from its `fixture_shape` (counts, sizes, depth, fan-out,

history, branches, ignore patterns, conflict and dirty shapes).

`--shape-scale` shrinks counts (never depth) for development runs; the scale

is part of the fixture identity.

## Generator (`scalesuite/generator.py`)

One `git fast-import` stream per fixture, written by one Python process:

1M files in ~16 s, 100k commits in ~40 s on the dev host. Fixed identity,

timestamps (epoch + 60 s per commit) and seeded RNG make the stream, and so

every commit id, byte-identical for a given spec. Identity recorded per row:

`fixture_spec_sha256` (the recipe) and `fixture_manifest_sha256` (sha256 of

the sorted `<oid> <ref>` list). Big binaries are a 4 KiB random block per

MiB followed by zeros: small in a pack, dedupable in a chunk store, full size

in a checkout.

Oak ingest uses Oak's own git importer, `oak clone <path>.git` (it runs

`git clone`, then replays the HEAD history into Oak storage; other refs are

not imported, so side branches are recreated with `oak switch -c` + commit).

## Operations (`scalesuite/ops.py`, catalog `scenarios/scale.yaml`)

clone.full, status.clean, status.dirty, diff.dirty, commit.dirty, push.branch,

pull.ff, pull.diverged, merge.preview, merge.preview.conflict, merge.local,

merge.conflict, log.full, log.n, log.path, blame, switch.existing,

switch.back, branch.create, branch.list, gc, review.branch_diff, file.show.

Each op cites the git t/perf script names it corresponds to (names only).

Committed-state oracles never trust the subject: git's are read with the

oracle's own `git cat-file`; Oak's with the independent stdlib reader

(`scripts/oakbench/oak_sqlite_reader.py`, `committed_payloads`). When the

reader cannot decode a store (its size bound, or the storage format written by

`oak maintenance compact`, which the reader does not support yet) the oracle is

recorded as unmeasured (`oracle_ok: null`, `reader_unavailable`), never passed.

Only `ctx.timed(...)` calls are measured. Setup (dirty shapes, sync-client

commits, branch switches for Oak's merge preview) and every oracle run

outside the timed path. Oracles compare against facts read from the fixture

with git plumbing and the generator's own content function (worktree bytes,

history and path-touch counts, branch paths, conflict counts), or parse the

subject's output where only the subject can answer (labelled

`oracle_reader`). An op with no equivalent is a skip row naming why

(`oak_cli_has_no_blame_or_annotate_command`, ...).

Transport: both subjects sync over loopback HTTP — git through

`oakbench.gitserver` (CGI `git http-backend`), oak through `oak serve`. When

Oak's push admission caps refuse the full history (see findings), the server

is seeded with a one-commit snapshot of main and the oak sync rows say

`oak_server_seed: snapshot_1_commit`; those cells are marked

`oak_vs_git_comparability: not_comparable:oak_server_seed_snapshot_1_commit` and

the summary suppresses their Oak-vs-Git delta and MDE (oak-vs-oak only).

Timing precision: timed commands are bounded by a watchdog timer that kills the

process group while the harness blocks in one `wait4`

(`run_timed(watchdog_seconds=...)`), because `Popen.wait(timeout)` on Python

< 3.11 polls with a sleep that doubles up to 50 ms and quantizes short timings.

Teardown: branches the harness creates are deleted in git and closed in Oak (no

local delete); Oak's `branch.list` lists open branches, so counts stay fixed.

Fixture identity is checked against the `config/fixtures.json` pin at

generation time (a mismatch aborts the tier).

## Statistics

Subjects default to `git, git_aa, oak_installed, oak_installed_aa`. The A/A

arms are the same binaries under a second name, interleaved in a fresh

seeded order per unit and run, so every (tier, op) cell has its own A/A floor

(`oakbench.cellstats`, ADR-0011). The summary reports, per cell: medians,

git and oak A/A floors, the dev-only oak/git ratio, the oak-vs-git minimum

detectable effect at 80% power, and the oak-vs-oak MDE (what an Oak

changeset comparison could resolve on this host). Verdicts are computed but

suppressed: nothing in this lane is a claim until the calibrated campaign.

## git-sizer statistics (`scalesuite/sizer.py`)

The git-sizer metric set (MIT; see `third_party/git-sizer/NOTICE`)

re-implemented on git plumbing, with level-of-concern ratios. Stored in

`<fixture>/SIZER.json`; a compact subset is copied into every row's

`fixture_stats`.