Commit b00752e4 · oak/benchmarks

· Browse files at this commit

b00752e49ee2

Merged from mrmrs-1b7902

mrmrs · 1 month ago · 65 files +20 added ~45 modified

Description

Honest measurement (P1): per-cell verdicts, real Git, recorded aborts, reader parity

Revives mrmrs-1b7902 (13 re-review fixes, gate 2026-09-19 FIX) on main e9b767a51662 and fixes its gate blockers, then lands program pillar P1 (ADR-0011).

Stage A - gate blockers

Campaign report no longer crashes on a default Git-vs-Oak campaign: incompatible paired evidence refuses the PAIR (comparison_refused, incompatible_evidence_fields, claim reason paired_evidence_incompatible) and the JSON/Markdown report renders. CLI test runs a real default-config git+oak mock campaign through report.
Git/Oak committed-tree parity: one committed_scope(scenario) feeds both Git ls-tree and the independent Oak reader (--scope); out-of-scope entries are never decoded, in-scope non-regular entries fail exactly as Git's mode check; rows and COMPARISON_FIELDS carry workflow_integrity_committed_scope; reason strings aligned.
Reader budgets (object/total/entry bounds, depth, deadline, oversized trees) are Unavailable (77) -> unmeasured, never an Oak correctness failure. Format pin: required schema columns + HEAD preimage + root tree serialization (mismatch = unrecognized format, unmeasured); deeper mismatches remain corrupt. Reader 2026-09-27.2, oracle 2026-09-27.1.
The 77 sentinel is harness-owned: bugfix_test_loop behavior tests (agent-edited code) treat any nonzero, including 77, as a measured failure; only a harness launch failure is unmeasured. Real-producer test with os._exit(77) at import.
Abandoned operations are recorded: after a failed/mutating timed op every remaining planned op is a skip row (aborted_after_failed_prerequisite:); dirty-replay setup failure emits one skip per phase op. Oracle test commands_total counts both bugfix harness commands; agent summary averages exclude harness-aborted rows; the M3 HOME test asserts against the production home_root.

Stage B - honest measurement

Git baseline: macOS xcrun shim detected and resolved to the real Git (xcrun --find git / developer dir / CLT); refused unless --allow-git-shim. Rows record subject_bin, subject_version, binary_sha256, git_requested_bin, git_shim_measured, git_config_profile. Oracle instrument and auxiliary lanes resolve the same way. New --git-modes scalar subject (Scalar recommended local config), labelled scalar_recommended vs stock.
Statistics (oakbench/cellstats.py): in-lane A/A control subject oak_installed_aa, randomized interleaved runs, default runs 5; per-cell A/A floor, bootstrap CI of relative median delta, exact Mann-Whitney permutation p, BH-FDR across the family. Change only when BH rejects AND CI excludes 0 AND the effect exceeds the cell floor (see Round 2 for the confirmation stage). Unconfirmed candidates are never claimed (unconfirmed Oak-baseline regression -> UNMEASURED). Scorecard cells take at least their A/A floor. Legacy global micro floor is informational (--global-noise-floor).
Timed sequence: harness attestation reads are named per row (harness_reads_before_op); patch/manifest oracles stay in the isolated replay; failed ops abort with recorded rows; page_cache_state on every timed row; opt-in --cache-mode cold purges immediately before each timed command (macOS purge needs root / sudo -n; Linux drop_caches), refusing to start when the preflight purge fails.
Metrics: cpu_user_ms, cpu_sys_ms, io_read_blocks, io_write_blocks, major_page_faults from wait4 rusage.
Provenance: timing_contract_version, fixture_sha256 + fixture_stats per row, plus per-row binary identity above and existing runner_profile.

Evidence (macOS 26.6.2, Apple M4 Max; oak 0.105.0 frozen copy sha256 4dc220e1f089; real Git = Xcode git 2.50.1 sha256 10f9c1df8945; shim /usr/bin/git sha256 b8763cf250e6; all runs under the measurement lock; results in the worker's results/, not committed):

Identical-binary A/A (devloop --lanes core --core-profiles micro,smoke --runs 5, baseline = A/A control = local = same frozen binary): PASS, 0 Oak-vs-Oak regressions, 0 Oak-vs-Oak improvements, 0 unconfirmed. micro: 14 cells tested, all no_change (per-cell floors median 14.8%, max 41.9%; max |delta| 8.7%; min p 0.056). smoke: 42 cells tested, all no_change (floors median 8.0%, max 27.1%; max |delta| 22.2%; min p 0.032); 12 unmeasured cells are Git-only local remote ops. The 8 listed improvements are Oak-vs-Git guardrail cells (e.g. few_large_binaries/snapshot.dirty -74.8%, CI [-76.0, -67.1], p=0.0079, BH-confirmed), not self-comparisons.
Shim before/after (micro, runs=5, separate randomized runs, medians ms git -> oak): shim /usr/bin/git: proc.spawn 7.70 (oak/git 0.65), status.clean 8.09 (0.85), status.dirty 8.24 (0.79), diff.dirty 8.52 (0.80). Real Git: proc.spawn 3.87 (1.34), status.clean 4.59 (1.51), status.dirty 4.49 (1.46), diff.dirty 4.69 (1.48). The shim adds about 3.7-3.9 ms per call and inverts every small-op Oak-vs-Git verdict on this host. These are separate runs of a smoke-sized sample, not a published speed claim.
Micro smoke (real oak + real git + git_scalar): 204 rows, 0 failures, 0 skips; every timed row carries page_cache_state, rusage CPU/IO, subject_bin/version/sha256, fixture_sha256. Scalar config is slower than stock on this 24-file fixture (status.clean 8.96 vs 5.32 ms, fsmonitor daemon start), reported as a separate labelled subject.
Tests: full bare suite python3 -m unittest discover -s tests: 1,184 OK, 32 skipped (gate baseline 1,156 / 25; the 7 extra skips are new optional-reader tests). With a blake3/zstandard reader venv: oak_sqlite_reader 25/25; workflow parity with reader + Xcode Git instrument 6/6, 0 skipped. New: tests/test_cellstats.py (includes the gate's -25.9% identical-binary samples, now no_change), tests/test_honest_measurement.py (real bench.py producer rows: cache/rusage/provenance, cold-mode purge ordering, scalar subject, shim resolution).

Round 2 (independent QA FIX on 1c9b819a; /Users/mrmrs/o/worker-mrmrs-20260927T102405-qa-p1/QA-P1.md)

P1-1 fail closed: any gated Oak-vs-Oak cell that an Oak arm measured but that has no verdict (no A/A floor from --skip-noise-floor, fewer than 5 runs/arm) makes devloop UNMEASURED (exit 2). Git-only cells (local remote ops) are exempt.
P1-2 guardrail: a Git-guardrail latency breach is "new" only when the same cell's Oak-vs-Oak verdict is a confirmed regression; otherwise it is a known gap. Regression test replays the reviewer's identical-binary aa2 rows (few_large_binaries/branch.create crossing) -> PASS.
A1: the Oak export arm and the independent reader classify like the Git arm. A subject oak status/oak export that exits nonzero, a removed or linked .oak, or a repo with no committed HEAD is a measured failure; unmeasured only for a missing instrument, a signal/timeout, a reader bound or an unrecognized format. Parity-matrix test on real git + oak (+ reader when configured) across correct/wrong/exec/oversized/never-committed/metadata-removed, plus failing status/export stubs in all 6 payload workflows and the oracle CLI exit code.
Detection power: no fixed 5 ms min-delta (default 0; per-cell floor + CI decide). Adaptive two-stage confirmation: cells screened at runs=5 (p<=0.2, CI excluding 0, any candidate, or MDE above --target-mde-pct 20) have only their scenarios re-run at --confirm-runs 20 within --confirm-budget-s 900, all subjects incl. the A/A control, and are decided on the fresh data alone (fresh floor, BH over the candidate family). Every no_change cell reports "no change detected >= X%" (MDE = max(floor, CI half-width)); cells.json persists every decision. Runs-to-confirm guidance derives from the exact minimum p and BH (n=5 cannot confirm one cell in a family of >= 7).
P1-5 public claim path: Baseline Books record noise_floor_source; baseline_campaign runs an interleaved git_aa control so each book cell gets its own A/A floor; claim-mode latency targets require per_cell_aa; decisive claim verdicts carry a one-sided bootstrap p and are BH-controlled across targets (unconfirmed -> AMBER).
P1-4 Scalar fairness: fsmonitor daemon started and untracked cache warmed in recorded untimed setup rows (also git_fsmonitor); git_scalar applies only to the large-tree scenarios; the bench summary compares Oak with the strongest Git per cell.
A2 refused comparisons publish no success/efficiency delta. A3 reader format pin: newer schema migration, unfindable HEAD row, or a blob-hash convention no sampled blob satisfies -> unmeasured. A4 in-scope over-limit blobs and colliding paths fail with Git's reasons before reader bounds. A5 tests isolated from the reader env; default-campaign test non-vacuous. A6 dirty-replay aborts record every abandoned op (incl. spectrum names). A7 a subject exiting 77 is a failure (returncode 1, process_returncode 77). P1-8 platform_lifecycle times the resolved Git (incl. gh's git via PATH). AGENTS.md updated (defaults, exit codes, detection limits).

Round-2 evidence (same host/binaries; results in the worker's results2/):

Identical binary A/A, defaults, micro+smoke: PASS; Oak-vs-Oak 0 regressions, 0 improvements, 0 unconfirmed (45 no_change at screen + 11 decided at confirmation, runs=20); A/A control 0 changes; MDE median 9.7%, p90 16.7%, max 24.1%. Wrapper-vs-wrapper A/A: PASS, 0 claims, MDE median 8.5%, max 19.4%.
Single-cell positive control (zero-delay exec wrapper as baseline and A/A; +2.5 ms busy-wait only on oak switch): REGRESSED on exactly the 4 branch.create cells and no other: micro tiny_text +15.7% (13.9 ms, CI [+13.4, +21.2], floor 8.0, n=20; the only shifted cell of its 14-cell family), smoke +13.9/+18.4/+16.6% on 14-16 ms ops; 52 other cells no_change (MDE median 6.4%).
Uniform exec-wrapper overhead (+1.2 ms/call) plus +3 ms on switch: 48/56 cells confirmed regressions, including +5.6% to +9.9% cells.
Fail closed: +10 ms slowdown with --skip-noise-floor -> UNMEASURED (exit 2); with --runs 2 -> UNMEASURED (exit 2) (were PASS exit 0).
Tests (round 2): bare 1,205 OK / 39 skipped; deps 1,205 / 10; reader 1,205 / 8. Round 3: bare 1,209 OK / 39 skipped; with blake3+zstandard, reader unset: 1,209 OK / 10 skipped; with the reader configured: 1,209 OK / 8 skipped.

Round 3 (QA round 2 PASS on 290f1164; pre-land fixes)

R2-1: the minimum detectable effect is stated at 80% power: MDE80 = max(floor, h) + 1.0*h (h = CI half-width). The old max(floor, h) was the ~50%-power point. Calibrated on resampled real A/A rows with injected shifts: multiplier 0 detected 45-50%, 0.84 80-88%, 1.0 92-95% for confirmation-stage cells (n=20, families of 10 and 30), and 100% for the screen(5)->confirm(20) pipeline behind screen-stage cells. tests/test_cellstats_power.py re-checks >= 80% on the committed real rows (28/30 confirmation-stage). Output reads "no change >= X% at 80% power"; with --no-confirm, screen-stage cells whose family makes a lone change BH-infeasible carry no power statement. AGENTS.md/README/ADR wording updated.
R2-2: the A/A self-check also covers confirmation-stage rows (a confirmed A/A change there makes the run UNMEASURED); those A/A decisions are in cells.json.
R2-3: with the format pinned by schema and migrations, an Oak-written HEAD naming a missing commit row is an Oak failure (dangling HEAD), not unmeasured. Reader 2026-09-27.3; cellstats 2026-09-27.3.
Real identical-binary A/A with the final code (micro+smoke, defaults): PASS; Oak-vs-Oak 0 claims (23 screen + 33 confirmation-stage no_change); A/A control 0 changes in both stages. MDE at 80% power over 56 no_change cells: min 4.7%, p25 7.4%, median 9.5%, p75 14.4%, p90 18.3%, max 22.8% (screen-stage median 12.7%, confirmation-stage median 8.7%).

Remaining (deliberate, with owner = next P1/P2 lane): agent campaign WORKFLOWS still 2; publish-gate coverage of task_loop/parallel/mount ops (M1); bare setup exceptions and outcome_attribution "subject" (B1); dirty-replay disk footprint; campaign_report harness-vs-instrument branch ordering; the workflow lane has no confirmation stage (its unconfirmed candidates make the verdict UNMEASURED); Git-guardrail cells use the Oak A/A floor. Snapshot attestation still reads before the timed command (labelled; purged in cold mode). Cold purge not verified here (needs root).

Worker: /Users/mrmrs/o/worker-mrmrs-20260927T090644-p1-honest/benchmarks. No merge performed.

Files 65

config

M command_semantics.json M git_modes.json M subjects.toml

docs

adr

A 0011-honest-measurement-contract.md

A independent-workflow-readers.md A qa-integrity-2026-09-18.md M runner-discipline.md M statistical-methodology.md M tuned-git-baselines.md

scripts

oakbench

M agent_campaign.py M baseline_book.py M cachectl.py M campaign_report.py A cellstats.py M command_semantics.py M execution.py A oak_sqlite_reader.py M output_semantics.py M populations.py M public_output.py M rows.py M scorecard.py M subjects.py M workflow_oracles.py

M agent_metrics_schema.py M agent_workflow.py M baseline_campaign.py M bench.py M conflict_resolution_lane.py M devloop.py M parallel_contention.py M platform_lifecycle.py M publish_gate.py M task_loop.py M workflow_ab.py

tests

fixtures

devloop_real_rows

A README.md A aa-micro.jsonl.gz A aa-smoke.jsonl.gz A aa2-micro.jsonl.gz A aa2-smoke.jsonl.gz

M test_agent_campaign.py M test_agent_campaign_cli.py M test_agent_execution_limits.py A test_agent_outcome_attribution.py M test_baseline_campaign.py M test_campaign_report.py A test_cellstats.py A test_cellstats_power.py M test_claim_eligibility.py M test_command_semantics.py A test_core_oracle_sequence.py A test_devloop_real_rows.py A test_honest_measurement.py M test_large_binary_lfs.py A test_oak_sqlite_reader.py A test_oracle_parity_matrix.py M test_public_output.py M test_rows.py M test_schema_and_oracle.py M test_workflow_oracles.py A test_workflow_parity.py A test_workflow_publication_qualification.py

M AGENTS.md M CONTEXT.md M README.md

Details

Commitb00752e49ee2

Parente9b767a51662

Merge parent5bd54e7b7233

M AGENTS.md

@@ -61,16 +61,25 @@

6161 the latest results. Pick a target from the top; don't re-derive priorities

6262 from raw JSONL.

6363 2. Edit Oak in `../oak`, build it (`cargo build --release` there).

64-3. `python3 scripts/devloop.py` — one command, one verdict: PASS, REGRESSED,

65- or UNMEASURED for your changeset vs the Oak baseline.

66-4. Only claim a win that devloop's printed detection limit supports.

64+3. `python3 scripts/devloop.py` — one command, one verdict: PASS (exit 0),

65+ REGRESSED (1), or UNMEASURED (2) for your changeset vs the Oak baseline.

66+4. Only claim a win that devloop's printed per-cell detection limit supports.

6767

6868 Claim discipline (non-negotiable):

6969

70-- **Never claim a delta below the noise floor.** devloop prints its own

71- per-lane detection limits; the default `--runs 2` can only catch large

72- changes. Chasing a 3% win? Bump `--runs` until the limit is below the

73- effect you claim, or the claim is noise.

70+- **Never claim a delta below the noise floor.** devloop judges every

71+ scenario/operation cell on its own interleaved A/A floor (`oak_installed_aa`),

72+ a bootstrap CI and Benjamini-Hochberg across the family, then re-measures

73+ screened cells (`--confirm-runs 20`, within `--confirm-budget-s`) and decides

74+ them on fresh data (ADR-0011). Every no-change cell prints "no change >= X%

75+ at 80% power": a true change of X% or more would have been detected at least

76+ 80% of the time (calibrated on real A/A rows). Smaller changes may pass

77+ undetected. Chasing a smaller win? Raise `--runs`/`--confirm-runs` until X is

78+ below it, or the claim is noise.

79+- **Defaults are `--runs 5`; fewer, or `--skip-noise-floor`, is UNMEASURED

80+ (exit 2), never PASS.** An unconfirmed Oak-baseline regression is also

81+ UNMEASURED with the runs needed to confirm it. A Git-guardrail latency breach

82+ fails the changeset only when the same cell is a confirmed Oak-vs-Oak regression.

7483 - **A cheaper output that loses information is a regression, not a win.**

7584 The gates enforce this: `information_recall` < baseline or lost

7685 `pipe_compatible_unified` structure fails the verdict (RECALL /