mrmrs-d5ae98

Closed into main

10 commits · 99 files +69 added ~30 modified

Description

P5 scale & coverage: every user-facing op timed (git vs oak), fresh/default/large/XL tiers, 14 core.yaml shapes runnable, git-sizer stats per fixture

Lane P5 of the benchmark program (worker /Users/mrmrs/o/worker-mrmrs-20260928T133154-p5-scale). New suite suites/scale plus a scale lane in the row contract. ALL timings are DEVELOPMENT-ONLY (evidence_class development_only_loaded_host, claim_eligible false, derived: no calibration record): the calibrated Oak-vs-Git campaign re-runs this machinery on a quiet host.

What it adds

23 ops timed for git and oak with untimed oracles: clone.full, status.clean/dirty, diff.dirty, commit.dirty, push.branch, pull.ff, pull.diverged, merge.preview(+conflict), merge.local, merge.conflict, log.full, log.n, log.path, blame, switch.existing/back, branch.create, branch.list, gc (git gc / oak maintenance compact), review.branch_diff, file.show. Only ctx.timed() calls are measured; setup, dirty shapes and every oracle run outside the timed path. No equivalent -> skip row with a reason (oak has no blame; oak local merge unreachable; oak serve lacks merge-to-main).
Oracles read the fixture with git plumbing + the generator's own content function (worktree bytes, history/path-touch counts, branch paths, conflict counts) and label their evidence class (oracle_reader). Committed-state oracles (commit.dirty, pull.ff, pull.diverged, gc) use git cat-file for git and the independent stdlib reader (oak_sqlite_reader.committed_payloads) for Oak; a reader bound or unsupported format => oracle_ok null (unmeasured), never a pass. Oak rows (incl. ingest rows) carry binary_sha256 for publish_gate.
Tiers (t/perf fresh/default/large concept, re-implemented): fresh, default, large; XL axes behind --allow-xl: xl-history (100k commits), xl-tree (1M files), xl-deep (depth 40), xl-refs (10k branches + 2k tags), xl-bigblob (3 GiB, sparse-friendly), xl-combined (all axes; --allow-xl-combined).
shape:<id> / shapes: the 14 scenarios/core.yaml shapes (never run before) mapped to runnable fixtures via their fixture_shape; --shape-scale for dev runs.
Deterministic fast-import generator (1M files 10-16 s, 100k commits 23-39 s); identity = refs_sha256 pinned in config/fixtures.json for every tier except xl-combined, with a regeneration test.
git-sizer metric set re-implemented on git plumbing (MIT attribution: suites/scale/third_party/git-sizer/{NOTICE,LICENSE-MIT}, docs/prior-art.md); compact stats on every row.
Statistics: git, git_aa, oak_installed, oak_installed_aa interleaved (seeded shuffle per unit per run), per-cell A/A floors (oakbench.cellstats), MDE at 80% power for oak-vs-git and oak-vs-oak per cell; verdicts suppressed.
Transport: loopback HTTP both sides (oakbench.gitserver vs oak serve). Oak sync rows say oak_server_seed when the server had to be seeded with a one-commit snapshot (push caps; below).
Docs: suites/scale/README.md, docs/prior-art.md (git-sizer + t/perf op list/tiers, file names only), docs/benchmark-coverage.md, scenarios/scale.yaml (generated op catalog with tperf_refs).

Tests: python3 -m unittest discover -s tests -> 1453 tests OK (7 skipped) on the combined tree (main 76308956 + this branch; conflicts in rows.py / lane_contract.py / docs/prior-art.md resolved keeping both robustness and scale); tests/test_scale_suite.py 69 tests, tests/test_oak_sqlite_reader.py +4 (generator determinism + registry pins, sizer vs git, tiers/shapes, every op oracle pass+fail with canned subject output, row contract, git-only end-to-end schedule).

Oak findings (filed; standalone repros in the worker's repros/)

fb-549 HIGH clone/switch wall-bound ~10-25 ms per file with little CPU: every materialized file does 2x F_FULLFSYNC (materialize.rs:464 -> atomic_file.rs:113 temp-file sync_all + :143 parent-dir sync_all). Dev rows: clone 5k files 42-64 s vs git 0.5-1.3 s; 50k files depth 40 537 s vs 4 s. repros/oak-clone-switch-wall-vs-cpu.sh
fb-550 HIGH push admission caps: resolved manifest entries summed over every pushed commit (commits x files, cap 1M); a single 50,000-file commit is refused ("blobs total 50002"; real ceiling ~49,998 files); no split. repros/oak-push-admission-cap.sh
fb-551 MEDIUM oak log <path> / oak log -v blind on history imported by oak clone <x.git>. repros/oak-log-path-imported-history.sh
fb-552 MEDIUM peer pull exits 5 (incomplete_ancestry): first-parent-only fetch leaves a merge parent missing (not a reparent problem). repros/oak-peer-pull-missing-merge-parent.sh
fb-553 MEDIUM depth/large-file scaling: merge.preview 3.1 s and review.branch_diff 3.0 s at 100k commits (git 14-18 ms); clean status 370 ms with one 3 GiB file (git 6 ms).
fb-554 LOW a refused push still persists the remote linkage (remote_url set after "no remote state was mutated"). repros/oak-refused-push-persists-remote-link.sh. The earlier "log hides local history" claim was a repro artifact and is withdrawn; the dev-run log.full "expected 1002, got 3" was the same artifact (work repo linked to a snapshot-seeded server) and is gone since the server is seeded from a separate import.

QA round 1 fixes (verdict FIX @ c44c9111)

Timing: timed commands use run_timed(watchdog_seconds=...) (timer thread kills the process group; single blocking wait4) instead of Popen.wait(timeout) polling; core lane unchanged. Test: 13 ms child with vs without the bound, |median delta| < 2 ms.
Comparability: cells whose Oak server was snapshot-seeded are oak_vs_git_comparability=not_comparable:oak_server_seed_snapshot_1_commit; Oak-vs-Git delta/MDE suppressed (kept out of the BH family), summary shows n/c + oak-vs-oak only. Tested for push/pull/clone on default/xl-deep/xl-history/xl-refs.
Dev-only enforced in data, verified by recomputation (QA r2 blocker + r3 holes fixed): evidence_class/claim_eligible are derived by scripts/oakbench/scale_calibration.py. run_scale_suite.py calibrate (>= 10 runs) writes the A/A campaign + host probes to a JSONL named in the record by content sha256. run --calibration-record and publish_gate both require that JSONL (gate: among its inputs) and RECOMPUTE everything from it; the record's numbers are never trusted. Policy 2026-09-29.3 (provisional, re-derive on the dedicated runner, P6):
- A/A per arm (git and oak separately): median floor <= 10% and <= 10% of cells over 10%; identical-binary claims counted on a HELD-OUT half with floors from the other half, must be 0 (QA r3: a 50%-slow oak_aa arm in 11/12 cells now FAILS; a same-pair floor could never produce a claim).
- Quiet host: load/core <= 0.10; flush probe must be the platform's full-flush primitive (F_FULLFSYNC on macOS; plain fsync rejected there), median <= 2.5 ms; >= 2000 files/s. No idle baseline was obtainable on this shared host (SSD reads 3.7-3.9 ms loaded).
- Work volume recorded on every row and probe (df device/mount point, fstype, diskutil bus protocol of the APFS physical store); RAM disks / disk images / tmpfs / unknown refused (a RAM disk read 0.19 ms and was classified ram_or_disk_image); calibration and run must be on the same volume.
- Shared runner_id, host_cpu_count, host_mem_bytes, Oak/Git binary sha256, registry-pinned fixtures; the run's own disk probes and each row's load must be quiet. Tests FAIL: QA's forged record (fixture tests/fixtures/scale/qa-r2-forgedC.fresh.jsonl.gz), resealed edits, copied runner, host mismatch, JSONL sha mismatch, missing campaign JSONL, broken A/A arm, held-out A/A shift, < 10 runs, RAM/image volume, volume mismatch, non-full-flush probe, contended disk. Stated limit: the campaign JSONL itself is not authenticated; editing its ~40 load/probe numbers and resealing (or fabricating a campaign) needs a signed runner to detect. Real smoke here: calibrate -> invalid (git 6/21 and oak 3/18 cells over 10%, load/core 0.25, F_FULLFSYNC 3.9 ms).
Findings above corrected with fb IDs; repros renamed/re-headed.
Oak harness branches are closed on teardown (git deletes); oak branch.list lists open branches: counts now constant across runs (3 fresh, 52 default).
Fixture pin checked at generation (fixture_pin_status on rows; mismatch aborts the tier); results files named <bench_id>..jsonl.
oak_sqlite_reader: tree_entries optional (dropped by oak maintenance compact by design); trees.content is the only tree form then, NULL content is Invalid. New real-store tests: compacted read, root/nested tree tamper, nulled content. P2 tamper tests green. gc oracles now measured (0 reader-unavailable rows).

Re-measure (dev-only, host load ~3): fresh + default, 5 runs x 4 arms, 941 rows; only failures are Oak log.path oracles (fb-551). oak-vs-oak MDE median 16% (fresh) / 6.5% (default); git small-op rows no longer cluster at poll steps (fresh status.clean 4.4-5.8 ms).

Left for later: calibrated campaign on a quiet host; xl-tree and xl-combined runs (1M-file worktrees are hours on this host; generation + sizer only); oak serve merge-to-main (merge.local/server merge skips); 16 agent-task catalog (P7); cold-cache mode for this lane.

Commits 10

f08b4a249972