Statistical Methodology

# Statistical Methodology

The benchmark suite should be useful locally, but public claims require a higher

bar than smoke runs.

## Runner Discipline

- Randomize subject order per scenario repetition with

  `--randomize-subject-order`.

- Use at least `n=10` for standard local comparisons and `n=30` for public

  reports.

- Run cold-cache and warm-cache lanes separately. Do not mix them in one median.

- Pin runner hardware, OS, filesystem, CPU governor/power mode, and available

  disk space.

- Record background load notes or runner telemetry when possible.

- Keep fixture generation, binary build/install, dependency cache population,

  and cleanup outside the timed operation unless the scenario explicitly

  measures setup.

## Reported Statistics

For each subject/scenario/operation group:

- sample count

- failures and failure rate

- median

- p90

- p99 for larger sample counts

- mean as secondary context

- bootstrap 95% confidence intervals for median, p90, and mean

- raw stdout/stderr bytes

- admitted-output truncation count

`scripts/benchmark_stats.py` computes bootstrap intervals from JSONL files:

```bash

python3 scripts/benchmark_stats.py results/latest.jsonl

```

## JSONL Fields To Preserve

Rows should include:

- `benchmark_track`

- `command_semantics_version`

- `subject_details`

- `elapsed_ms`

- `returncode`

- `estimated_tokens_total`

- `raw_output_bytes`

- `stdout_bytes`

- `stderr_bytes`

- `output_truncated`

- `tool_call_count`

- host/platform/machine/Python metadata

Agent-adapter rows should additionally include provider-reported token fields

when available, normalized transcript character counts, and `agent.environment`

so `minimal` runs are not mixed with `local-default` runs.

## Token Estimate Calibration

char/4 is biased for VCS output and the bias differs by output style: hex

hashes tokenize at roughly 2-3 chars/token (char/4 underestimates by up to

~50% on hash-heavy logs) while prose runs slightly over 4. Because Oak and Git

emit different output styles, raw char/4 deltas between them carry a styling

bias. Before publishing cross-subject token claims:

```bash

python3 scripts/token_calibration.py results/agent-workflow/artifacts --label <subject>

```

Report the calibration factor per subject/operation alongside the delta, or

use provider-reported tokens. Install `tiktoken` for exact counts; without it

the tool reports a documented composition heuristic.

## Cost-Weighted Tokens

Model output tokens cost ~5x input tokens, and cache reads ~0.1x. Harness rows

carry `estimated_cost_weighted_tokens` (and agent rows

`token_metrics.cost_weighted_total`) using billing direction: command text an

agent types is model output; stdout it reads is model input. Rank

optimizations by cost-weighted deltas, not raw token totals — shortening a

command an agent must type every call is worth ~5x the same characters of

stdout.

## Efficiency Regression Gates

`scripts/regression_report.py` gates the Oak-baseline comparison on exact

metrics by default: any tool-call increase and >25% output-byte increase flag.

Token-estimate gates are off by default (estimates are advisory); enable

`--tokens-threshold-pct` once provider-reported rows dominate the lane.

## Overclaiming Rules

- A single smoke run can justify “caught a regression,” not “Oak is X% faster.”

- Wide confidence intervals should be shown, not hidden.

- Token estimates are transcript-pressure proxies, not billing tokens.

- If a row is truncated, raw output bytes matter more than admitted token count.

- If command semantics differ, label the row diagnostic or agent-default.

- Null metric values are unmeasured, not zero; never chart them as wins.

- Status/diff speed claims against Git require at least one tuned mode from

  `--git-modes`; stock-Git-only comparisons must say so.

- Never aggregate rows across `agent.instruction_level` values.

## Noise-Floor Doctrine (A/A and Embedded Nulls)

A benchmark that does not know its own noise will happily report it as signal.

The suite measures zero before it measures anything else:

- **Per-cell A/A control** (`scripts/devloop.py`, `oakbench/cellstats.py`,

  ADR-0011): the SAME baseline binary runs as `oak_installed_aa`, interleaved

  with the other subjects in every randomized run of every lane. Each

  (scenario, operation) cell's floor is the largest |relative median delta|

  inside its own A/A bootstrap interval (min 3%). A change requires BH-FDR

  rejection (exact Mann-Whitney permutation p), a CI excluding zero, and the

  effect beyond the cell's floor. Screened cells get an independent

  confirmation run (default 20 runs/arm) decided on fresh data only; every

  no-change cell reports its minimum detectable effect at 80% power,

  max(floor, h) + h (h = CI half-width; calibrated on real A/A rows). Missing A/A evidence or fewer than 5 runs is UNMEASURED. The former global micro-profile floor

  (`--global-noise-floor`) is informational only; it produced "improvements"

  of a binary over itself because one number cannot describe every cell.

- **Embedded nulls** (per lane): search/read/edit/test steps run identical

  commands for every subject (zero VCS calls), so any spread there is live

  host noise inside that very lane. Lane thresholds rise to clear it too.

- High-jitter operations (process spawn, sub-steps) are excluded from the

  floor because the verdict never gates on them.

## Bootstrap and Ratio Confidence Intervals

`oakbench.stats.bootstrap_ci` resamples with replacement (seeded, default

2000 draws) and reports a percentile interval for a named statistic (median,

mean, p90/p95/p99). `ratio_ci` bootstraps numerator and denominator samples

independently and reports the interval of the statistic ratio — the portable

cross-runner claim (see docs/runner-discipline.md: latency never pools across

runner classes; the same-run ratio with its CI does travel). Both intervals

capture within-sample variance only; they know nothing about run-to-run drift,

which is why the next section exists.

## Noise-Inflated GREEN/RED (ADR-0008)

Scorecard verdicts inflate the bootstrap ratio bound by the baseline's A/A

noise floor before comparing against the goal. A decisive result survives the

inflation and stays GREEN or RED; a narrow interval that clears the goal by

less than the floor falls through to AMBER (with a printed required `n`) or

UNDECIDABLE (when the goal's margin from parity is itself below the floor —

a 1.0x parity goal is undecidable on any instrument with a nonzero floor).

Guard metrics (integrity, failure rate, recall/precision, pipe compatibility,

output stability, transport comparability) must hold before any efficiency

win can be GREEN. See ADR-0008.

## Outlier Policy: MAD Modified-Z, Flag-Never-Delete

`oakbench.stats.mad_outliers` computes modified z-scores

`z = (x - median) / (1.4826 * MAD)` where `MAD = median(|x - median|)` and

1.4826 is the consistency constant that turns the MAD into a standard-

deviation estimate on normal data. The default threshold 14.826 (hyperfine's)

means ~14.8 robust standard deviations from the median — equivalently

`|x - median| / MAD > 14.826 * 1.4826 ~= 22` — far more conservative than the

textbook 3.5 cut, so only blatant escapees are flagged. `MAD == 0` (scale

collapse) flags nothing.

Policy is **flag, never delete**: flagged samples stay in every aggregate, as

with suspect environment windows (docs/runner-discipline.md). Deleting data

hides exactly the noise the flag exists to expose. **First-run special case**:

if index 0 is the sole outlier and it is slow, that is a warmup/cold-cache

signature (`first_run_warmup_hint`) — the right response is warmup runs or a

proper cold-cache lane, not discarding the sample.

## Adaptive Repetitions: Floor, Top-Up, Cap

`oakbench.stats.adaptive_rep_plan` is pure planning math:

It is currently exercised by instrument tests, not invoked by production lane

runners. Existing campaigns do not automatically receive adaptive top-ups.

- **Floor** (hyperfine-style): at least `min_runs` (default 10) repetitions

  per cell, paid for inside a wall-time budget.

- **Top-up** (PTS-style): once the floor is met, more runs are requested in

  floor-sized batches while %stddev >= the threshold (default 5%) or any

  sample escapes beyond 3 sigma.

- **Cap**: total runs never exceed `min(min_runs * cap_multiplier,

  budget / per_run_estimate)` — long trials shrink the cap. When a trigger

  wants runs the cap will not grant, the plan says `budget_exhausted` and the

  cell is reported with the variance it has, never silently blessed.

## BH-FDR for the Full Matrix

A full subject x scenario x operation matrix makes hundreds of win/tie/loss

comparisons; at p < 0.05, a wall of green is expected by chance alone.

`oakbench.stats.benjamini_hochberg_detail` implements the Benjamini-Hochberg

step-up procedure for a supplied family of valid p-values:

sort the m p-values, find the largest rank k with `p_(k) <= (k/m) * fdr`,

reject everything at or below that critical p-value (ties included). devloop

wires it through `oakbench.cellstats`: the family is every tested latency cell

of one comparison (target vs baseline) in one lane, p-values are exact

permutation p-values of the Mann-Whitney statistic, and the adjusted decision

is persisted per cell (`fdr_rejected`). Scorecard GREEN/RED verdicts against

Baseline Books still do not claim matrix-wide FDR control.

Production integration needs a declared hypothesis family, valid p-values,

applicable dependence assumptions, and persisted adjusted decisions; applying

the helper to arbitrary scores or confidence bounds would not establish FDR

control.

## Planning Heuristic and Required n

`required_n_for_relative_delta` scales the sample size by the square of the

noise-floor-to-target ratio: resolving a 5% delta on a 10% floor needs ~4x

the runs. AMBER scorecard cells print this number so an agent knows whether

to rerun with more reps or declare the goal undecidable on this instrument.

This is a planning heuristic, not a power calculation or a precision guarantee:

it has no specified alpha, target power, or model of dependence. Confirmatory

campaigns need sample planning for their actual experimental units and effect.

## Min-of-N Is a Sanity Band, Never a Headline

External cross-checks (`scripts/tperf_runner.py` driving git's own t/perf,

`scripts/crosscheck_hyperfine.py`) report min-of-N, while the suite's lanes

report medians with CIs. Min-of-N answers "best case the hardware permits"

and is kept as a sanity band around our numbers — agreement is reassuring,

disagreement is a bug hunt — but it is never the published statistic.

## Two-Stage Confirmation for Borderline Verdicts

`oakbench.stats.two_stage_confirmation`: when a devloop regression lands

within +/-25% of the verdict threshold, it is borderline and earns an

independent second measurement; the regression is confirmed only if both

stages exceed the threshold. One borderline excursion is never a verdict.

(devloop wiring is future work; the decision rule is the pure function.)

## Provenance

The methods are deliberately stolen from tools that earned them:

- **hyperfine**: minimum-runs floor with a wall-time budget; conservative

  MAD modified-Z outlier *warnings* (threshold 14.826) with the slow-first-run

  warmup hint; warn-don't-discard.

- **Phoronix Test Suite**: variance-triggered dynamic run top-up

  (%stddev threshold) with a cap scaled by trial length.

- **git t/perf**: min-of-N reporting, used here only as an external sanity

  band (GPLv2 boundary: driven at arm's length, never vendored).

- Benjamini & Hochberg (1995) for FDR control; Iglewicz & Hoaglin (1993) for

  the modified z-score the outlier flag is built on.