Scale & coverage lane (suites/scale)

# Scale & coverage lane (`suites/scale`)

Pillar P5 of the benchmark program: every user-facing VCS operation timed for

git and oak, on deterministic synthetic repositories from "fresh" to XL, with

git-sizer-style statistics recorded as provenance on every row.

```bash

python3 suites/scale/run_scale_suite.py list

python3 suites/scale/run_scale_suite.py generate --tier default            # fixture + sizer only

python3 suites/scale/run_scale_suite.py run --tier fresh --tier default    # A/A-interleaved, runs=5

python3 suites/scale/run_scale_suite.py run --tier shapes --shape-scale 0.05

python3 suites/scale/run_scale_suite.py generate --allow-xl --tier xl-history --tier xl-refs

```

Set `--work` (fixtures and workspaces; large) and `--results` (rows) to a

directory on the benchmark volume, and `OAK_BENCH_LOCK_PATH` when several

lanes share a host. Raw rows are never committed (ADR-0003).

## Status of the numbers

Every row carries `evidence_class: development_only_loaded_host` and

`claim_eligible: false`. The lane was built and validated on a heavily loaded

shared workstation; its timings estimate cost and resolution (MDE) only. The

calibrated Oak-vs-Git campaign re-runs the same machinery on a quiet host.

## Tiers (`scalesuite/tiers.py`)

The t/perf repository-tier concept (fresh / default / large; Git's

`test_perf_*_repo` helpers, GPL: concept only) with synthetic repositories:

| tier | files | depth | commits | branches | tags | binaries |

|---|---:|---:|---:|---:|---:|---|

| fresh | 100 | 3 | 1 | 1 | 0 | - |

| default | 5,000 | 6 | 1,000 | 50 | 20 | - |

| large | 100,000 | 8 | 10,000 | 1,000 | 200 | 1 x 256 MiB, 3 revisions |

| xl-history | 10,000 | 5 | 100,000 | 1 | 0 | - |

| xl-tree | 1,000,000 | 6 | 3 | 1 | 0 | - |

| xl-deep | 50,000 | 40 | 100 | 1 | 0 | - |

| xl-refs | 2,000 | 4 | 1,000 | 10,000 | 2,000 | - |

| xl-bigblob | 16 | 2 | 3 | 1 | 0 | 1 x 3 GiB, 2 revisions |

| xl-combined | 1,000,000 | 32 | 100,000 | 10,000 | 2,000 | 1 x 3 GiB |

XL tiers need `--allow-xl` (`xl-combined` also `--allow-xl-combined`). Each XL

tier pushes ONE axis past real-world "large" so a scaling defect is

attributable; `xl-combined` exists for the calibrated campaign. XL tiers skip

the per-run `clone.full` and `gc` where one run costs tens of minutes (skip

rows say so).

`shape:<id>` (or `shapes` for all 14) turns every scenario of

`scenarios/core.yaml` — the 14-shape catalog no lane used to run — into a

runnable tier from its `fixture_shape` (counts, sizes, depth, fan-out,

history, branches, ignore patterns, conflict and dirty shapes).

`--shape-scale` shrinks counts (never depth) for development runs; the scale

is part of the fixture identity.

## Generator (`scalesuite/generator.py`)

One `git fast-import` stream per fixture, written by one Python process:

1M files in ~16 s, 100k commits in ~40 s on the dev host. Fixed identity,

timestamps (epoch + 60 s per commit) and seeded RNG make the stream, and so

every commit id, byte-identical for a given spec. Identity recorded per row:

`fixture_spec_sha256` (the recipe) and `fixture_manifest_sha256` (sha256 of

the sorted `<oid> <ref>` list). Big binaries are a 4 KiB random block per

MiB followed by zeros: small in a pack, dedupable in a chunk store, full size

in a checkout.

Oak ingest uses Oak's own git importer, `oak clone <path>.git` (it runs

`git clone`, then replays the HEAD history into Oak storage; other refs are

not imported, so side branches are recreated with `oak switch -c` + commit).

## Operations (`scalesuite/ops.py`, catalog `scenarios/scale.yaml`)

clone.full, status.clean, status.dirty, diff.dirty, commit.dirty, push.branch,

pull.ff, pull.diverged, merge.preview, merge.preview.conflict, merge.local,

merge.conflict, log.full, log.n, log.path, blame, switch.existing,

switch.back, branch.create, branch.list, gc, review.branch_diff, file.show.

Each op cites the git t/perf script names it corresponds to (names only).

Only `ctx.timed(...)` calls are measured. Setup (dirty shapes, sync-client

commits, branch switches for Oak's merge preview) and every oracle run

outside the timed path. Oracles compare against facts read from the fixture

with git plumbing and the generator's own content function (worktree bytes,

history and path-touch counts, branch paths, conflict counts), or parse the

subject's output where only the subject can answer (labelled

`oracle_reader`). An op with no equivalent is a skip row naming why

(`oak_cli_has_no_blame_or_annotate_command`, ...).

Transport: both subjects sync over loopback HTTP — git through

`oakbench.gitserver` (CGI `git http-backend`), oak through `oak serve`. When

Oak's push admission caps refuse the full history (see findings), the server

is seeded with a one-commit snapshot of main and the oak sync rows say

`oak_server_seed: snapshot_1_commit`; they are then not comparable with git's

full-history clone.

## Statistics

Subjects default to `git, git_aa, oak_installed, oak_installed_aa`. The A/A

arms are the same binaries under a second name, interleaved in a fresh

seeded order per unit and run, so every (tier, op) cell has its own A/A floor

(`oakbench.cellstats`, ADR-0011). The summary reports, per cell: medians,

git and oak A/A floors, the dev-only oak/git ratio, the oak-vs-git minimum

detectable effect at 80% power, and the oak-vs-oak MDE (what an Oak

changeset comparison could resolve on this host). Verdicts are computed but

suppressed: nothing in this lane is a claim until the calibrated campaign.

## git-sizer statistics (`scalesuite/sizer.py`)

The git-sizer metric set (MIT; see `third_party/git-sizer/NOTICE`)

re-implemented on git plumbing, with level-of-concern ratios. Stored in

`<fixture>/SIZER.json`; a compact subset is copied into every row's

`fixture_stats`.