Scale & coverage lane (suites/scale)

# Scale & coverage lane (`suites/scale`)

Pillar P5 of the benchmark program: every user-facing VCS operation timed for

git and oak, on deterministic synthetic repositories from "fresh" to XL, with

git-sizer-style statistics recorded as provenance on every row.

```bash

python3 suites/scale/run_scale_suite.py list

python3 suites/scale/run_scale_suite.py generate --tier default            # fixture + sizer only

python3 suites/scale/run_scale_suite.py run --tier fresh --tier default    # A/A-interleaved, runs=5

python3 suites/scale/run_scale_suite.py run --tier shapes --shape-scale 0.05

python3 suites/scale/run_scale_suite.py generate --allow-xl --tier xl-history --tier xl-refs

```

Set `--work` (fixtures and workspaces; large) and `--results` (rows) to a

directory on the benchmark volume, and `OAK_BENCH_LOCK_PATH` when several

lanes share a host. Raw rows are never committed (ADR-0003).

## Status of the numbers

`evidence_class` and `claim_eligible` are derived, never set by hand

(`scripts/oakbench/scale_calibration.py`). A row is `calibrated_quiet_host` /

claimable only when the run was given `--calibration-record` produced by

`run_scale_suite.py calibrate` on the same runner, the record validates (content

digest, quiet host: load/core <= 0.25, >= 20 A/A cells, A/A floor median <= 10%,

zero identical-binary claims) and the row's own load sample is quiet. Calibrated

rows embed the record; `scripts/publish_gate.py` re-derives eligibility from each

row and refuses every other scale row, so a hand-edited `claim_eligible: true`

fails. Without a record, rows are `development_only_loaded_host`.

## Tiers (`scalesuite/tiers.py`)

The t/perf repository-tier concept (fresh / default / large; Git's

`test_perf_*_repo` helpers, GPL: concept only) with synthetic repositories:

| tier | files | depth | commits | branches | tags | binaries |

|---|---:|---:|---:|---:|---:|---|

| fresh | 100 | 3 | 1 | 1 | 0 | - |

| default | 5,000 | 6 | 1,000 | 50 | 20 | - |

| large | 100,000 | 8 | 10,000 | 1,000 | 200 | 1 x 256 MiB, 3 revisions |

| xl-history | 10,000 | 5 | 100,000 | 1 | 0 | - |

| xl-tree | 1,000,000 | 6 | 3 | 1 | 0 | - |

| xl-deep | 50,000 | 40 | 100 | 1 | 0 | - |

| xl-refs | 2,000 | 4 | 1,000 | 10,000 | 2,000 | - |

| xl-bigblob | 16 | 2 | 3 | 1 | 0 | 1 x 3 GiB, 2 revisions |

| xl-combined | 1,000,000 | 32 | 100,000 | 10,000 | 2,000 | 1 x 3 GiB |

XL tiers need `--allow-xl` (`xl-combined` also `--allow-xl-combined`). Each XL

tier pushes ONE axis past real-world "large" so a scaling defect is

attributable; `xl-combined` exists for the calibrated campaign. XL tiers skip

the per-run `clone.full` and `gc` where one run costs tens of minutes (skip

rows say so).

`shape:<id>` (or `shapes` for all 14) turns every scenario of

`scenarios/core.yaml` — the 14-shape catalog no lane used to run — into a

runnable tier from its `fixture_shape` (counts, sizes, depth, fan-out,

history, branches, ignore patterns, conflict and dirty shapes).

`--shape-scale` shrinks counts (never depth) for development runs; the scale

is part of the fixture identity.

## Generator (`scalesuite/generator.py`)

One `git fast-import` stream per fixture, written by one Python process:

1M files in ~16 s, 100k commits in ~40 s on the dev host. Fixed identity,

timestamps (epoch + 60 s per commit) and seeded RNG make the stream, and so

every commit id, byte-identical for a given spec. Identity recorded per row:

`fixture_spec_sha256` (the recipe) and `fixture_manifest_sha256` (sha256 of

the sorted `<oid> <ref>` list). Big binaries are a 4 KiB random block per

MiB followed by zeros: small in a pack, dedupable in a chunk store, full size

in a checkout.

Oak ingest uses Oak's own git importer, `oak clone <path>.git` (it runs

`git clone`, then replays the HEAD history into Oak storage; other refs are

not imported, so side branches are recreated with `oak switch -c` + commit).

## Operations (`scalesuite/ops.py`, catalog `scenarios/scale.yaml`)

clone.full, status.clean, status.dirty, diff.dirty, commit.dirty, push.branch,

pull.ff, pull.diverged, merge.preview, merge.preview.conflict, merge.local,

merge.conflict, log.full, log.n, log.path, blame, switch.existing,

switch.back, branch.create, branch.list, gc, review.branch_diff, file.show.

Each op cites the git t/perf script names it corresponds to (names only).

Committed-state oracles never trust the subject: git's are read with the

oracle's own `git cat-file`; Oak's with the independent stdlib reader

(`scripts/oakbench/oak_sqlite_reader.py`, `committed_payloads`). When the

reader cannot decode a store (its size bound, or the storage format written by

`oak maintenance compact`, which the reader does not support yet) the oracle is

recorded as unmeasured (`oracle_ok: null`, `reader_unavailable`), never passed.

Only `ctx.timed(...)` calls are measured. Setup (dirty shapes, sync-client

commits, branch switches for Oak's merge preview) and every oracle run

outside the timed path. Oracles compare against facts read from the fixture

with git plumbing and the generator's own content function (worktree bytes,

history and path-touch counts, branch paths, conflict counts), or parse the

subject's output where only the subject can answer (labelled

`oracle_reader`). An op with no equivalent is a skip row naming why

(`oak_cli_has_no_blame_or_annotate_command`, ...).

Transport: both subjects sync over loopback HTTP — git through

`oakbench.gitserver` (CGI `git http-backend`), oak through `oak serve`. When

Oak's push admission caps refuse the full history (see findings), the server

is seeded with a one-commit snapshot of main and the oak sync rows say

`oak_server_seed: snapshot_1_commit`; those cells are marked

`oak_vs_git_comparability: not_comparable:oak_server_seed_snapshot_1_commit` and

the summary suppresses their Oak-vs-Git delta and MDE (oak-vs-oak only).

Timing precision: timed commands are bounded by a watchdog timer that kills the

process group while the harness blocks in one `wait4`

(`run_timed(watchdog_seconds=...)`), because `Popen.wait(timeout)` on Python

< 3.11 polls with a sleep that doubles up to 50 ms and quantizes short timings.

Teardown: branches the harness creates are deleted in git and closed in Oak (no

local delete); Oak's `branch.list` lists open branches, so counts stay fixed.

Fixture identity is checked against the `config/fixtures.json` pin at

generation time (a mismatch aborts the tier).

## Statistics

Subjects default to `git, git_aa, oak_installed, oak_installed_aa`. The A/A

arms are the same binaries under a second name, interleaved in a fresh

seeded order per unit and run, so every (tier, op) cell has its own A/A floor

(`oakbench.cellstats`, ADR-0011). The summary reports, per cell: medians,

git and oak A/A floors, the dev-only oak/git ratio, the oak-vs-git minimum

detectable effect at 80% power, and the oak-vs-oak MDE (what an Oak

changeset comparison could resolve on this host). Verdicts are computed but

suppressed: nothing in this lane is a claim until the calibrated campaign.

## git-sizer statistics (`scalesuite/sizer.py`)

The git-sizer metric set (MIT; see `third_party/git-sizer/NOTICE`)

re-implemented on git plumbing, with level-of-concern ratios. Stored in

`<fixture>/SIZER.json`; a compact subset is copied into every row's

`fixture_stats`.