ADR-0011: Honest measurement — per-cell verdicts, real Git, labelled caches

# ADR-0011: Honest measurement — per-cell verdicts, real Git, labelled caches

- Status: accepted

- Date: 2026-09-27

## Context

Five measurement defects made small Oak-vs-Git and Oak-vs-Oak claims

indefensible on the benchmark workstation:

1. `/usr/bin/git` on macOS is the xcrun trampoline. It re-resolves the

   developer directory on every call (about 3 ms here, roughly 40% of an 8 ms

   status/diff median), and `binary_sha256` hashed the stub, not the Git that

   ran. The committed-tree oracle resolved the same stub.

2. devloop judged every cell with one threshold, `max(cfg, 2 x A/A floor)`,

   where the floor came from the micro profile. High-variance cells cleared

   it by chance: identical binaries published "improvements" of -38.9% and

   -25.9% (gate on mrmrs-1b7902, 2026-09-19). Default runs were 2.

   Bootstrap and BH-FDR helpers existed but were unwired.

3. A failed prerequisite aborted a subject's scenario and emitted no rows for

   the abandoned operations, so a consumer saw "fewer samples", not a skip.

4. Cache state was a label on some rows only; no core-lane cold mode existed.

5. Rows carried wall time and peak RSS only; no CPU or block I/O.

## Decision

**Git baseline.** Subject loading detects the xcrun shim (`/usr/bin/git` on

Darwin) and resolves the real binary (`xcrun --find git`, then the selected

developer directory, then Command Line Tools). The run is refused when the

shim cannot be resolved, unless `--allow-git-shim` / `OAK_BENCH_ALLOW_GIT_SHIM=1`

explicitly measures it (rows then say `git_shim_measured: true`). Every git

row records `subject_bin`, `subject_version`, `binary_sha256`,

`git_requested_bin` and `git_config_profile` (`stock`, `scalar_recommended`,

`tuned_<mode>`). The oracle's Git instrument resolves the shim the same way.

`--git-modes scalar` adds `git_scalar`, configured with Scalar's recommended

local-performance keys, with the fsmonitor daemon started and the untracked

cache warmed in recorded untimed setup rows (see `docs/tuned-git-baselines.md`).

It applies to the large-tree scenarios Scalar targets; on small fixtures it is

slower than stock Git. The bench summary compares Oak with the strongest Git

per cell, so a weaker tuned mode never flatters Oak.

**Per-cell statistics** (`scripts/oakbench/cellstats.py`, version

`2026-09-27.1`). devloop runs an A/A control subject `oak_installed_aa` (the

baseline binary under a second name) interleaved with every other subject in

randomized order within each run; default runs are 5. Each (scenario,

operation) cell gets:

- its own A/A floor: the largest |relative median delta| inside the A/A

  bootstrap interval, at least 3%;

- a 95% percentile-bootstrap CI of the relative median delta (paired by run

  index when both arms share runs);

- an exact two-sided permutation p-value of the Mann-Whitney rank statistic;

- Benjamini-Hochberg (q = 0.05 by default) across every tested cell of the

  comparison family.

A latency change is claimed only when BH rejects AND the CI excludes zero AND

the point effect exceeds the cell's A/A floor (the floor is already the outer

bound of the A/A interval, so requiring the target CI to clear it too would

count noise twice). There is no fixed absolute minimum delta (`--min-delta-ms`

defaults to 0); the floor and CI are per cell.

*Adaptive confirmation (two-stage).* Stage 1 (screen) runs every cell at

`--runs` (default 5). Cells that look suggestive (p <= 0.2, CI excluding zero,

any stage-1 change or unconfirmed candidate) or too imprecise

(minimum detectable effect above `--target-mde-pct`, default 20%) are

re-measured: only their scenarios are re-run, with all subjects including the

A/A control, at `--confirm-runs` (default 20) within `--confirm-budget-s`

(default 900 s). Those cells are then decided on the FRESH data alone, with a

fresh A/A floor and BH over the candidate family only. This is independent

replication, so screening does not bias the confirmation test.

*Fail closed.* A gated Oak-vs-Oak cell that either Oak arm measured but that

has no verdict (fewer than 5 runs per arm, no A/A floor from

`--skip-noise-floor`) makes the run UNMEASURED (exit 2), never PASS. So does an

unconfirmed Oak-baseline regression after confirmation (the report gives the

runs per arm to confirm, from both the CI width and the exact-p/BH bound:

at n=5 the smallest exact p is 2/252, so one change cannot pass BH in a family

of 7 or more), and an A/A control that itself shows a confirmed change.

*Detection limits.* Every no-change cell reports its minimum detectable effect

at 80% power, max(floor, h) + 1.0 * h with h the CI half-width: "no change >=

X% at 80% power", never a bare no_change. max(floor, h) alone is only the ~50%

power point; the 1.0 * h margin was calibrated on resampled real A/A rows with

injected shifts (92-95% detection for confirmation-stage cells, 100% for the

screen-then-confirm pipeline; tests/test_cellstats_power.py). Without the

confirmation stage, screen-stage cells whose family makes a lone change

BH-infeasible carry no power statement.

*Git guardrail.* A Git-guardrail latency breach counts as introduced by the

changeset only when the SAME cell's Oak-vs-Oak comparison is a confirmed

regression; otherwise it is a known Oak-vs-Git gap.

The legacy global micro A/A is informational only (`--global-noise-floor`).

Scorecard cells in devloop take at least their own A/A floor.

*Public claim path.* Baseline Books record `noise_floor_source`:

`per_cell_aa` when `baseline_campaign.py` ran the interleaved `git_aa` control

(same Git, second subject name), else `operator_global`. Claim-mode latency

targets require `per_cell_aa`. Decisive claim verdicts carry a one-sided

bootstrap p-value and are Benjamini-Hochberg controlled across all claim

targets in one evaluation; unconfirmed GREEN/RED become AMBER.

**Recorded aborts.** When a timed operation fails (nonzero exit or failed

attestation), that subject's cell timing stops and every abandoned operation

is emitted as a skip row (`returncode: 77`, `skip_reason:

aborted_after_failed_prerequisite:<op>`).

**Cache labels and cold mode.** Every timed core row carries

`page_cache_state` (`warm` | `purged` | `unknown`), `page_cache_reason` and

`cache_mode_requested`; `cache_state` is set to `warm`/`cold` where a caller

did not already label it (remote and replay phases keep their labels).

`--cache-mode cold` purges the OS page cache immediately before each timed

command, after every harness read for that command. macOS needs root for

`purge(8)` (`sudo -n` with a NOPASSWD sudoers entry); Linux needs write access

to `/proc/sys/vm/drop_caches`. A failed preflight purge refuses the run; a

later failed purge labels the row `unknown`, never `cold`. Limits: purge

drops the unified buffer cache, not SSD controller caches or Oak/Git

in-process caches; APFS clones share blocks with the fixture.

**Harness reads.** Snapshot and branch attestation still hash repository

metadata and a payload sample before the timed command (the attestation

needs a before-state). Rows name those reads in `harness_reads_before_op`;

in cold mode the purge follows them. Patch/manifest oracles and determinism

/ information probes stay outside the main timed sequence (isolated replay

repo, null elapsed for diagnostics).

**Rusage.** Every timed row records `cpu_user_ms`, `cpu_sys_ms`,

`io_read_blocks`, `io_write_blocks`, `major_page_faults` from the same

`wait4` rusage as peak RSS (`rusage_source: os_wait4_child_rusage`). Block

counts are real device I/O, so a warm run's measured value is 0, not null.

**Provenance.** Core rows add `timing_contract_version` (`2026-09-27.1`),

`fixture_sha256` and `fixture_stats` (files, bytes, largest file, max path

depth), alongside the existing `runner_profile` (OS, CPU model) and

`subject_details`.

All row changes are additive; scenario and operation names are unchanged

(ADR-0005). `workflow_integrity_committed_scope` is added to agent/workflow

rows and to `COMPARISON_FIELDS`.

## Consequences

- Stock-Git numbers recorded before this ADR on macOS include the

  trampoline and are not comparable with new rows (different

  `binary_sha256`).

- devloop's core lane takes about 1.3x longer (one extra subject) plus the

  bounded confirmation stage.

- The workflow lane has no confirmation stage yet: its unconfirmed candidates

  make the verdict UNMEASURED.

## Addendum (2026-09-27, QA P1 re-review)

Subject-controlled state is a measured failure in both committed-tree readers:

a removed or linked `.oak`, a repository with no committed HEAD, or a subject

`oak status`/`oak export` that exits nonzero fails the Oak arm exactly as Git's

`ls-tree` failure fails the Git arm. Unmeasured is reserved for a missing

instrument, a signal/timeout, a reader resource bound, or an unrecognized

storage format (newer schema migration, commit preimage,

root tree serialization, or blob-hash convention: at least one of the 8

smallest stored blobs must verify). In-scope blobs over the task's own size

limit, colliding paths and non-regular entries are reported by the reader and

fail with Git's reasons before any reader bound applies. With the format pinned,

a HEAD naming a missing commit row is a dangling HEAD: an Oak failure. A subject

process that exits 77 is recorded as a failure (`returncode` 1, `process_returncode` 77).

A refused campaign comparison reports no success or efficiency delta.