Measurement methodology review — 2026-09-09

# Measurement methodology review — 2026-09-09

Reviewed base: `a9e8fa39d0a097e03fba83c6fcd0029865e2387373e19b57c8f23da03ec173cd`.

Worker: `/Users/mrmrs/o/worker-mrmrs-20260909T081049-benchmark-populations/bench`.

Branch: `mrmrs-9df99c`. References below describe the reviewed base unless marked fixed.

The repository has unusually strong measurement building blocks, but their

presence does not establish end-to-end scientific validity. In particular,

human usability is currently unmeasured, incompatible populations could be

pooled, and several statistical protections exist only as tested helpers.

This review makes no empirical Oak-versus-Git performance claim.

## Current strategies and strengths

- Shared measurement policy in `scripts/oakbench/`, versioned fixture and

  command semantics, and explicit null/skip contracts (AGENTS.md; ADR-0002,

  ADR-0005) provide a sound reproducibility foundation.

- Core operation timing, deterministic workflows, real CLI agent adapters,

  contention and lazy-mount workloads measure different layers. These must

  remain distinct populations; none substitutes for human participants.

- Statistical helpers implement seeded percentile bootstrap, ratio intervals,

  robust outlier flags, sample planning, and multiple-testing math

  (`scripts/oakbench/stats.py`). A/A and embedded null measurements estimate

  host noise; runner identity and portability are explicit in ADR-0007.

- Scorecards gate failures, information recall/precision and strong content

  integrity evidence, propagate baseline interval uncertainty, and cap partial

  metric coverage (`scripts/oakbench/scorecard.py:87-169,177-215`).

- Publication gates check provenance, explicit skips, transports, instruction

  levels and minimum samples for public core rows

  (`scripts/publish_gate.py:297-336,428-439`). Token calibration distinguishes

  estimates from reported usage. Git tuned modes address baseline fairness.

- Tests include synthetic adapter traces and malformed/null observations,

  golden reporting fixtures plus Node/Python parity, gate boundary tests,

  integrity attestation tests, and local protocol/server integration tests.

  Relevant files include `tests/test_agent_workflow.py`,

  `tests/test_reporting_golden.py`, `tests/test_scorecard.py`,

  `tests/test_public_claim_gate.py`, and `tests/test_integrity_attestation.py`.

  Statistical tests check helper outputs; they do not demonstrate empirical

  confidence-interval coverage or false-positive rates across distributions.

- Existing external cross-checks drive hyperfine and Git t/perf separately;

  their minima are diagnostic sanity checks, not equivalent estimands to

  median latency. `docs/prior-art.md` documents provenance boundaries.

## Findings and acceptance criteria

### P1 — incompatible measurement populations (partially fixed here)

`scripts/benchmark_stats.py:82-94` grouped only by subject/scenario/operation.

`scripts/oakbench/scorecard.py:227-238` matched lane/profile but omitted

instruction level, model, runner and semantics. Multiple input files could

therefore create precise-looking aggregates from distinct populations.

Publication instruction-level checks are per file, so they do not prevent a

later report from pooling separate valid files (`publish_gate.py:428-439`).

**Fixed:** shared `scripts/oakbench/populations.py` rejects conflicting known

identity metadata in each report cell before statistics or scorecard verdicts.

It distinguishes present from missing metadata. New tests exercise conflicting

runner, track, cache, fixture, transport, model, instruction and binary identity

through both consumers. Uniformly missing legacy metadata remains usable for

diagnostics; this guard is not a public provenance attestation.

**Remaining:** `BaselineCell.key` and `Target.baseline_key` cannot express

runner/semantics/model identity (`baseline_book.py:21-58`). A homogeneous target

population may still be compared with an incompatible saved baseline. Other

report consumers need the same policy. Add a versioned comparison fingerprint

to rows and baseline books, reject duplicate keys with conflicting metadata,

and require full compatibility for public comparisons. Negative tests must

cover baseline-versus-target mismatch and every dashboard ingestion path.

### P1 — human usability has no empirical instrument (open)

The lane glossary in CONTEXT.md and coding-agent scenarios in

`scenarios/agent.yaml:1-6` describe automated workloads. No participant

protocol, human trial schema, counterbalancing or participant-cluster analysis

was found in the scenario/config/agent methodology inventory. Human-readable

command output is not a measurement of human performance.

Add a human crossover protocol using the same task-state oracles as agents.

Predeclare skill and VCS-familiarity strata, training budget, task variants,

counterbalanced tool/task order, timeouts and assistance rules. Record task

success, elapsed time, destructive mistakes, recovery time, help use and a

defined workload questionnaire. Report participant-cluster intervals and

learning curves separately from first-use performance. Acceptance requires a

pilot with actual participants and explicit uncertainty, not synthetic humans.

### P1 — scorecard certainty exceeds evidence at small n (open)

`stats.py:32-33` gives a singleton a zero-width bootstrap interval.

`scorecard.py:142` treats a missing noise floor as zero, and no minimum sample

gate precedes GREEN/RED. `tests/test_scorecard.py:78-85` deliberately uses two

samples. A single very small timing can receive GREEN against a fixed baseline.

Keep diagnostic scoring available, but add explicit claim eligibility requiring

minimum independent runs, measured noise, baseline provenance and replication.

Singletons and unknown calibration must never be publicly decisive. Test that

large effect sizes cannot bypass missing-evidence guards. Tail claims also

need adequate baseline and target sample counts.

### P1 — advertised statistical protections are not wired (open)

`benjamini_hochberg[_detail]`, `adaptive_rep_plan`, and

`two_stage_confirmation` are defined in `stats.py:98,184,260,294`; repository

search finds callers only in tests. `scorecard.py:18-37` independently scores

every target. The FDR prose in `docs/statistical-methodology.md` can therefore

be mistaken for a production guarantee. Two-stage wiring is explicitly

documented as future work, which is accurate.

Define the hypothesis family and valid p-value procedure before wiring FDR;

do not apply BH directly to interval bounds or arbitrary scores. Persist raw

and adjusted decisions, family membership and independent confirmation data.

For adaptive sampling, predeclare stopping rules and evaluate coverage under

those rules. Until implemented, label all three as library capabilities, not

campaign protections. End-to-end tests must demonstrate that the report/runner

actually invokes each promised control.

### P1 — agent oracle can certify insufficient evidence (owned by oracle review)

`agent_workflow.py:492-499` falls back to an always-success command for unknown

workflow validators. Status parsing ignores exit status and can treat empty

failed output as clean (`475-489`). `snapshot_created` accepts clean state as

an alternative to an observed snapshot (`1329-1334`), making it redundant with

the mandatory clean-state check.

Require explicit per-workflow validators and structured, successful VCS state

probes. Verify the intended persisted content and history operation, not just

a clean working directory. Acceptance includes mutation tests that replace

validators with no-ops, cause empty failed status, or leave correct workspace

bytes without the requested snapshot. These changes are outside this worker.

### P1 — real-agent reproducibility and serialization (open)

All real adapters default to an empty model in `config/agents.toml`; the row

records configured model, CLI version, and null sampling/budget settings

(`agent_workflow.py:1475-1484`). Provider defaults can change without a resolved

model identity. Randomized order is optional (`159`, `1846-1854`). There is no

measurement-lock context in the agent runner; its lock-wait metric is null

(`1609`), despite the general serialization policy.

Require resolved model/version and declared budgets for public trials, retain

request and resolved identities, fingerprint customizations, persist the trial

order/seed, and randomize or counterbalance by default in campaign mode. Wrap

timed agent trials in the shared measurement lock and exclude queue time.

Acceptance: two concurrent runner processes cannot overlap measured intervals;

unresolved models cannot receive public-claim eligibility.

### P2 — intervals and required-n omit experimental structure (open)

`stats.py:68-95` independently resamples numerator/denominator rows.

Repeated measurements from one machine/session/fixture do not necessarily

represent independent experimental units. `required_n_for_relative_delta`

(`111-117`) is a square-ratio heuristic without alpha, target power or an

estimated sampling distribution; it is not formal power analysis.

Record block/session/fixture identities and support paired or cluster bootstrap

where the design supplies pairing. Replicate across sessions and runners rather

than assuming more within-session repetitions remove systematic bias. Rename

the n estimate as a planning heuristic or implement validated power planning.

Acceptance: simulation tests measure interval coverage and false positives for

skew, ties, outliers, temporal drift and correlated blocks, with fixed seeds and

statistical tolerances rather than exact implementation-mirroring assertions.

### P2 — failure-conditioned summaries can mislead (open)

`benchmark_stats.py:84-85` drops unsuccessful rows without printing their

denominator. Agent `metric_average` and `summary_text` (`1755-1782`) instead

average available values across outcomes; the estimand is not consistently

named. Failed cheap attempts or expensive retries can distort efficiency.

Always report attempted/successful/failed/timed-out/skipped counts next to cost

and latency, with binomial uncertainty for task success. Separate successful

latency from all-attempt resource cost and a defined cost-to-success measure.

Do not silently discard censored timeouts or recategorize measured failures as

skips to enable publication. Negative tests must keep failure-only cells visible.

### P2 — tail resolution is not tail precision (open)

`oakbench/reporting.py:44-70` gates p95/p99 at n=20/100. This prevents a trivial

max relabeling at smaller n, but does not establish a precise tail estimate.

The sample can still have only about one observation in the relevant tail.

Report interval widths and expected/existing tail counts; predeclare the

precision required for a p99 claim. Use enough independent samples to meet

that criterion and report timeout censoring. Tests should distinguish

"computable percentile" from "eligible precise tail claim".

## Recommended sequence

1. Complete oracle fail-closed behavior and comparison fingerprint coverage.

2. Separate diagnostic scores from publishable conclusions; wire sample,

   provenance, calibration and failure reporting requirements across all lanes.

3. Run instrument-validation simulations and replicated A/A campaigns before

   interpreting small improvements. Add planned paired/cluster analysis.

4. Build a human pilot protocol and held-out agent tasks, with measured model

   identity and success-cost curves. Keep development and confirmation sets

   distinct to limit benchmark-specific optimization.

5. Wire multiplicity and confirmation controls, then scale representative

   upstream-inspired behavioral scenarios under the same oracles and contracts.

Industry leadership requires both semantic coverage and demonstrated validity

of the instrument. Passing helper tests or adding many scenarios alone does not

establish either cross-tool fairness or precise human/agent outcome claims.