ADR-0011: Honest measurement — per-cell verdicts, real Git, labelled caches

# ADR-0011: Honest measurement — per-cell verdicts, real Git, labelled caches

- Status: accepted

- Date: 2026-09-27

## Context

Five measurement defects made small Oak-vs-Git and Oak-vs-Oak claims

indefensible on the benchmark workstation:

1. `/usr/bin/git` on macOS is the xcrun trampoline. It re-resolves the

   developer directory on every call (about 3 ms here, roughly 40% of an 8 ms

   status/diff median), and `binary_sha256` hashed the stub, not the Git that

   ran. The committed-tree oracle resolved the same stub.

2. devloop judged every cell with one threshold, `max(cfg, 2 x A/A floor)`,

   where the floor came from the micro profile. High-variance cells cleared

   it by chance: identical binaries published "improvements" of -38.9% and

   -25.9% (gate on mrmrs-1b7902, 2026-09-19). Default runs were 2.

   Bootstrap and BH-FDR helpers existed but were unwired.

3. A failed prerequisite aborted a subject's scenario and emitted no rows for

   the abandoned operations, so a consumer saw "fewer samples", not a skip.

4. Cache state was a label on some rows only; no core-lane cold mode existed.

5. Rows carried wall time and peak RSS only; no CPU or block I/O.

## Decision

**Git baseline.** Subject loading detects the xcrun shim (`/usr/bin/git` on

Darwin) and resolves the real binary (`xcrun --find git`, then the selected

developer directory, then Command Line Tools). The run is refused when the

shim cannot be resolved, unless `--allow-git-shim` / `OAK_BENCH_ALLOW_GIT_SHIM=1`

explicitly measures it (rows then say `git_shim_measured: true`). Every git

row records `subject_bin`, `subject_version`, `binary_sha256`,

`git_requested_bin` and `git_config_profile` (`stock`, `scalar_recommended`,

`tuned_<mode>`). The oracle's Git instrument resolves the shim the same way.

`--git-modes scalar` adds `git_scalar`, configured with Scalar's recommended

local-performance keys (see `docs/tuned-git-baselines.md`); it is reported

separately from stock Git.

**Per-cell statistics** (`scripts/oakbench/cellstats.py`, version

`2026-09-27.1`). devloop runs an A/A control subject `oak_installed_aa` (the

baseline binary under a second name) interleaved with every other subject in

randomized order within each run; default runs are 5. Each (scenario,

operation) cell gets:

- its own A/A floor: the largest |relative median delta| inside the A/A

  bootstrap interval, at least 3%;

- a 95% percentile-bootstrap CI of the relative median delta (paired by run

  index when both arms share runs);

- an exact two-sided permutation p-value of the Mann-Whitney rank statistic;

- Benjamini-Hochberg (q = 0.05 by default) across every tested cell of the

  comparison family.

A latency change is claimed only when BH rejects AND the CI excludes zero AND

the whole CI lies beyond the cell's A/A floor AND the absolute delta clears

`--min-delta-ms`. A candidate that passes CI and floor but not BH is

UNCONFIRMED: never published; an unconfirmed Oak-baseline regression makes

the verdict UNMEASURED (exit 2) with the runs needed to confirm. Cells with

fewer than 3 runs per arm or no A/A floor are UNMEASURED. An A/A control that

itself shows a confirmed change makes the lane UNMEASURED. The legacy global

micro A/A is informational only (`--global-noise-floor`). Scorecard cells in

devloop take at least their own A/A floor.

**Recorded aborts.** When a timed operation fails (nonzero exit or failed

attestation), that subject's cell timing stops and every abandoned operation

is emitted as a skip row (`returncode: 77`, `skip_reason:

aborted_after_failed_prerequisite:<op>`).

**Cache labels and cold mode.** Every timed core row carries

`page_cache_state` (`warm` | `purged` | `unknown`), `page_cache_reason` and

`cache_mode_requested`; `cache_state` is set to `warm`/`cold` where a caller

did not already label it (remote and replay phases keep their labels).

`--cache-mode cold` purges the OS page cache immediately before each timed

command, after every harness read for that command. macOS needs root for

`purge(8)` (`sudo -n` with a NOPASSWD sudoers entry); Linux needs write access

to `/proc/sys/vm/drop_caches`. A failed preflight purge refuses the run; a

later failed purge labels the row `unknown`, never `cold`. Limits: purge

drops the unified buffer cache, not SSD controller caches or Oak/Git

in-process caches; APFS clones share blocks with the fixture.

**Harness reads.** Snapshot and branch attestation still hash repository

metadata and a payload sample before the timed command (the attestation

needs a before-state). Rows name those reads in `harness_reads_before_op`;

in cold mode the purge follows them. Patch/manifest oracles and determinism

/ information probes stay outside the main timed sequence (isolated replay

repo, null elapsed for diagnostics).

**Rusage.** Every timed row records `cpu_user_ms`, `cpu_sys_ms`,

`io_read_blocks`, `io_write_blocks`, `major_page_faults` from the same

`wait4` rusage as peak RSS (`rusage_source: os_wait4_child_rusage`). Block

counts are real device I/O, so a warm run's measured value is 0, not null.

**Provenance.** Core rows add `timing_contract_version` (`2026-09-27.1`),

`fixture_sha256` and `fixture_stats` (files, bytes, largest file, max path

depth), alongside the existing `runner_profile` (OS, CPU model) and

`subject_details`.

All row changes are additive; scenario and operation names are unchanged

(ADR-0005). `workflow_integrity_committed_scope` is added to agent/workflow

rows and to `COMPARISON_FIELDS`.

## Consequences

- Stock-Git numbers recorded before this ADR on macOS include the

  trampoline and are not comparable with new rows (different

  `binary_sha256`).

- At runs=5 a single-cell change cannot survive BH across a large family

  (minimum exact p is 2/252); such changes surface as UNCONFIRMED with a

  required run count rather than as silent passes or false claims.

- devloop's core lane takes about 1.3x longer (one extra subject).