Statistical Methodology

# Statistical Methodology

The benchmark suite should be useful locally, but public claims require a higher

bar than smoke runs.

## Runner Discipline

- Randomize subject order per scenario repetition with

  `--randomize-subject-order`.

- Use at least `n=10` for standard local comparisons and `n=30` for public

  reports.

- Run cold-cache and warm-cache lanes separately. Do not mix them in one median.

- Pin runner hardware, OS, filesystem, CPU governor/power mode, and available

  disk space.

- Record background load notes or runner telemetry when possible.

- Keep fixture generation, binary build/install, dependency cache population,

  and cleanup outside the timed operation unless the scenario explicitly

  measures setup.

## Reported Statistics

For each subject/scenario/operation group:

- sample count

- failures and failure rate

- median

- p90

- p99 for larger sample counts

- mean as secondary context

- bootstrap 95% confidence intervals for median, p90, and mean

- raw stdout/stderr bytes

- admitted-output truncation count

`scripts/benchmark_stats.py` computes bootstrap intervals from JSONL files:

```bash

python3 scripts/benchmark_stats.py results/latest.jsonl

```

## JSONL Fields To Preserve

Rows should include:

- `benchmark_track`

- `command_semantics_version`

- `subject_details`

- `elapsed_ms`

- `returncode`

- `estimated_tokens_total`

- `raw_output_bytes`

- `stdout_bytes`

- `stderr_bytes`

- `output_truncated`

- `tool_call_count`

- host/platform/machine/Python metadata

Agent-adapter rows should additionally include provider-reported token fields

when available, normalized transcript character counts, and `agent.environment`

so `minimal` runs are not mixed with `local-default` runs.

## Token Estimate Calibration

char/4 is biased for VCS output and the bias differs by output style: hex

hashes tokenize at roughly 2-3 chars/token (char/4 underestimates by up to

~50% on hash-heavy logs) while prose runs slightly over 4. Because Oak and Git

emit different output styles, raw char/4 deltas between them carry a styling

bias. Before publishing cross-subject token claims:

```bash

python3 scripts/token_calibration.py results/agent-workflow/artifacts --label <subject>

```

Report the calibration factor per subject/operation alongside the delta, or

use provider-reported tokens. Install `tiktoken` for exact counts; without it

the tool reports a documented composition heuristic.

## Cost-Weighted Tokens

Model output tokens cost ~5x input tokens, and cache reads ~0.1x. Harness rows

carry `estimated_cost_weighted_tokens` (and agent rows

`token_metrics.cost_weighted_total`) using billing direction: command text an

agent types is model output; stdout it reads is model input. Rank

optimizations by cost-weighted deltas, not raw token totals — shortening a

command an agent must type every call is worth ~5x the same characters of

stdout.

## Efficiency Regression Gates

`scripts/regression_report.py` gates the Oak-baseline comparison on exact

metrics by default: any tool-call increase and >25% output-byte increase flag.

Token-estimate gates are off by default (estimates are advisory); enable

`--tokens-threshold-pct` once provider-reported rows dominate the lane.

## Overclaiming Rules

- A single smoke run can justify “caught a regression,” not “Oak is X% faster.”

- Wide confidence intervals should be shown, not hidden.

- Token estimates are transcript-pressure proxies, not billing tokens.

- If a row is truncated, raw output bytes matter more than admitted token count.

- If command semantics differ, label the row diagnostic or agent-default.

- Null metric values are unmeasured, not zero; never chart them as wins.

- Status/diff speed claims against Git require at least one tuned mode from

  `--git-modes`; stock-Git-only comparisons must say so.

- Never aggregate rows across `agent.instruction_level` values.

## Noise-Floor Doctrine (A/A and Embedded Nulls)

A benchmark that does not know its own noise will happily report it as signal.

The suite measures zero before it measures anything else:

- **A/A null test** (`scripts/devloop.py`): the SAME baseline binary runs as

  two subjects on the micro profile. The p95 spread between them is what zero

  looks like on this machine right now. Latency deltas smaller than the floor

  are never claimed; verdict thresholds rise to at least 2x the floor.

- **Embedded nulls** (per lane): search/read/edit/test steps run identical

  commands for every subject (zero VCS calls), so any spread there is live

  host noise inside that very lane. Lane thresholds rise to clear it too.

- High-jitter operations (process spawn, sub-steps) are excluded from the

  floor because the verdict never gates on them.

## Bootstrap and Ratio Confidence Intervals

`oakbench.stats.bootstrap_ci` resamples with replacement (seeded, default

2000 draws) and reports a percentile interval for a named statistic (median,

mean, p90/p95/p99). `ratio_ci` bootstraps numerator and denominator samples

independently and reports the interval of the statistic ratio — the portable

cross-runner claim (see docs/runner-discipline.md: latency never pools across

runner classes; the same-run ratio with its CI does travel). Both intervals

capture within-sample variance only; they know nothing about run-to-run drift,

which is why the next section exists.

## Noise-Inflated GREEN/RED (ADR-0008)

Scorecard verdicts inflate the bootstrap ratio bound by the baseline's A/A

noise floor before comparing against the goal. A decisive result survives the

inflation and stays GREEN or RED; a narrow interval that clears the goal by

less than the floor falls through to AMBER (with a printed required `n`) or

UNDECIDABLE (when the goal's margin from parity is itself below the floor —

a 1.0x parity goal is undecidable on any instrument with a nonzero floor).

Guard metrics (integrity, failure rate, recall/precision, pipe compatibility,

output stability, transport comparability) must hold before any efficiency

win can be GREEN. See ADR-0008.

## Outlier Policy: MAD Modified-Z, Flag-Never-Delete

`oakbench.stats.mad_outliers` computes modified z-scores

`z = (x - median) / (1.4826 * MAD)` where `MAD = median(|x - median|)` and

1.4826 is the consistency constant that turns the MAD into a standard-

deviation estimate on normal data. The default threshold 14.826 (hyperfine's)

means ~14.8 robust standard deviations from the median — equivalently

`|x - median| / MAD > 14.826 * 1.4826 ~= 22` — far more conservative than the

textbook 3.5 cut, so only blatant escapees are flagged. `MAD == 0` (scale

collapse) flags nothing.

Policy is **flag, never delete**: flagged samples stay in every aggregate, as

with suspect environment windows (docs/runner-discipline.md). Deleting data

hides exactly the noise the flag exists to expose. **First-run special case**:

if index 0 is the sole outlier and it is slow, that is a warmup/cold-cache

signature (`first_run_warmup_hint`) — the right response is warmup runs or a

proper cold-cache lane, not discarding the sample.

## Adaptive Repetitions: Floor, Top-Up, Cap

`oakbench.stats.adaptive_rep_plan` is pure planning math:

It is currently exercised by instrument tests, not invoked by production lane

runners. Existing campaigns do not automatically receive adaptive top-ups.

- **Floor** (hyperfine-style): at least `min_runs` (default 10) repetitions

  per cell, paid for inside a wall-time budget.

- **Top-up** (PTS-style): once the floor is met, more runs are requested in

  floor-sized batches while %stddev >= the threshold (default 5%) or any

  sample escapes beyond 3 sigma.

- **Cap**: total runs never exceed `min(min_runs * cap_multiplier,

  budget / per_run_estimate)` — long trials shrink the cap. When a trigger

  wants runs the cap will not grant, the plan says `budget_exhausted` and the

  cell is reported with the variance it has, never silently blessed.

## BH-FDR for the Full Matrix

A full subject x scenario x operation matrix makes hundreds of win/tie/loss

comparisons; at p < 0.05, a wall of green is expected by chance alone.

`oakbench.stats.benjamini_hochberg_detail` implements the Benjamini-Hochberg

step-up procedure for a supplied family of valid p-values:

sort the m p-values, find the largest rank k with `p_(k) <= (k/m) * fdr`,

reject everything at or below that critical p-value (ties included). This

helper is currently exercised by tests, not wired into campaign scorecards.

Existing GREEN/RED verdicts therefore do not claim matrix-wide FDR control.

Production integration needs a declared hypothesis family, valid p-values,

applicable dependence assumptions, and persisted adjusted decisions; applying

the helper to arbitrary scores or confidence bounds would not establish FDR

control.

## Planning Heuristic and Required n

`required_n_for_relative_delta` scales the sample size by the square of the

noise-floor-to-target ratio: resolving a 5% delta on a 10% floor needs ~4x

the runs. AMBER scorecard cells print this number so an agent knows whether

to rerun with more reps or declare the goal undecidable on this instrument.

This is a planning heuristic, not a power calculation or a precision guarantee:

it has no specified alpha, target power, or model of dependence. Confirmatory

campaigns need sample planning for their actual experimental units and effect.

## Min-of-N Is a Sanity Band, Never a Headline

External cross-checks (`scripts/tperf_runner.py` driving git's own t/perf,

`scripts/crosscheck_hyperfine.py`) report min-of-N, while the suite's lanes

report medians with CIs. Min-of-N answers "best case the hardware permits"

and is kept as a sanity band around our numbers — agreement is reassuring,

disagreement is a bug hunt — but it is never the published statistic.

## Two-Stage Confirmation for Borderline Verdicts

`oakbench.stats.two_stage_confirmation`: when a devloop regression lands

within +/-25% of the verdict threshold, it is borderline and earns an

independent second measurement; the regression is confirmed only if both

stages exceed the threshold. One borderline excursion is never a verdict.

(devloop wiring is future work; the decision rule is the pure function.)

## Provenance

The methods are deliberately stolen from tools that earned them:

- **hyperfine**: minimum-runs floor with a wall-time budget; conservative

  MAD modified-Z outlier *warnings* (threshold 14.826) with the slow-first-run

  warmup hint; warn-don't-discard.

- **Phoronix Test Suite**: variance-triggered dynamic run top-up

  (%stddev threshold) with a cap scaled by trial length.

- **git t/perf**: min-of-N reporting, used here only as an external sanity

  band (GPLv2 boundary: driven at arm's length, never vendored).

- Benjamini & Hochberg (1995) for FDR control; Iglewicz & Hoaglin (1993) for

  the modified z-score the outlier flag is built on.