Log in
benchmarksPublic

Commit b297e8a2

· Browse files at this commit

b297e8a25d08

Merged from mrmrs-7776e1
mrmrs · 1 month ago · 11 files +2 added ~9 modified
A docs/adr/0009-baseline-comparison-eligibility.md
1+# ADR-0009: Saved baseline identity and claim eligibility
2+
3+- Status: accepted
4+- Date: 2026-09-10
5+
6+## Context
7+
8+A homogeneous candidate population can still be incomparable with its saved
9+baseline. Earlier book cells retained a median and uncertainty but omitted
10+semantics/fixture/runner identity and outcome denominators. Singleton bootstrap
11+intervals and missing noise floors could yield a diagnostic GREEN that looked
12+like public evidence. Counting rows also permitted duplicated trials to create
13+artificial precision.
14+
15+## Decision
16+
17+Baseline Books now emit schema version 2. Existing immutable version 1 books
18+remain readable for diagnostics; never rewrite them to imply evidence that was
19+not collected. Each new cell records:
20+
21+- `comparison_identity`, independently versioned as schema 1;
22+- `independent_n`, the number of distinct recorded `(bench_id, run)` trials;
23+- `attempted_n`, including skipped rows, and `failure_n`/`skipped_n`;
24+- the existing measured metric count `n`, confidence interval and noise floor,
25+ plus the interval's recorded `confidence` level.
26+
27+`independent_n` is an operational distinct-trial count, not proof of statistical
28+independence. Repeated sessions and valid experiment design remain necessary.
29+Missing/empty campaign IDs or missing/noninteger trial indices leave this count
30+null. Candidate duplicate trial IDs are rejected for claims; a baseline's
31+distinct count must equal its metric sample count.
32+
33+Common identity includes command track/semantics, fixture/scenario/oracle
34+versions, environment isolation, runner class, cache/load state, remote
35+transport/server/shaping, token calibration, and controlled agent identity.
36+Actual nested fixture/instruction aliases are normalized. Subject binary hashes,
37+VCS versions and subject-specific prompt labels deliberately differ between
38+comparators and are not equality requirements. Within-subject population checks
39+still reject mixed binary identities. The full field list is centralized in
40+`oakbench.populations.COMPARISON_FIELDS`.
41+
42+Known-versus-missing or unequal common identity invalidates comparisons in both
43+modes. Books with duplicate cell keys or unknown identity versions fail loading.
44+Book construction rejects incompatible populations rather than silently pooling
45+them. It preserves failed/skipped/unmeasured outcome denominators alongside the
46+successful metric summary.
47+
48+The scorecard CLI defaults to `--mode claim`; `--mode diagnostic` is explicit.
49+The Python API retains its diagnostic default for existing local tooling.
50+Every result names its mode and `claim_eligible`; diagnostic results always set
51+that field false and carry a diagnostic-only notice. Existing diagnostic
52+GREEN/RED semantics remain useful locally but cannot establish a public claim.
53+
54+Claim mode requires complete applicable identity, at least 30 distinct trials
55+in both populations (100 for p99), no duplicate candidate trials, no hidden
56+failures/skips/unmeasured metrics, a measured finite nonnegative baseline noise
57+floor, and a valid baseline confidence interval. Unknown cache/load/model
58+sentinels are not measured identity. Recall/precision guards require versioned
59+output oracles; a `patch_apply_ok` guard requires a versioned patch oracle and
60+every successful row to explicitly pass it.
61+
62+Both candidate and recorded baseline confidence levels must be at least 0.95
63+and below 1. Legacy missing confidence does not imply 95%. Candidate results
64+retain their confidence level and bootstrap draw count. Nonfinite/out-of-range
65+confidence and nonpositive/noninteger draw counts are rejected before analysis.
66+No arbitrary bootstrap-draw floor substitutes for assessing precision.
67+
68+Absent eligibility evidence yields AMBER with explicit reasons and a nonzero
69+claim-mode CLI exit; missing inputs remain UNMEASURED and also exit nonzero.
70+Incompatible evidence is INVALID. Evidence eligibility does not force GREEN:
71+the existing statistical verdict still decides distance from the target.
72+
73+## Consequences and limits
74+
75+The eligibility floor is intentionally conservative even for exact resource
76+metrics. It is not a power calculation, tail-precision guarantee, multiplicity
77+correction, or claim that arbitrary row IDs prove independent sampling.
78+
79+`claim_eligible` means this scorecard's comparison-evidence prerequisites passed.
80+It does not replace publication gates, provenance/content attestation,
81+preregistered analysis, public failure reporting, or scientific review. A
82+baseline noise-floor value is a recorded measurement, not an authenticated
83+calibration provenance record. Cross-runner-class ratios still require a
84+separate same-run analysis; saved incompatible medians are rejected.
85+
86+Older rows missing required identity need a fresh measured campaign. The
87+baseline builder must not manufacture source-run identity from the host on
88+which an old JSONL file happens to be summarized. Human/agent study protocol,
89+hierarchical uncertainty and production FDR wiring remain separate work.
90+
91+`benchmark_stats.py` now prints all outcome denominators before its
92+success-conditioned statistics, including failure-only and unmeasured cells.
93+Unknown outcomes remain distinct from observed nonzero exits. Skips are
94+excluded from the attempts denominator and reported separately.