ADR-0009: Saved baseline identity and claim eligibility

• Status: accepted
• Date: 2026-09-10

Context

A homogeneous candidate population can still be incomparable with its saved baseline. Earlier book cells retained a median and uncertainty but omitted semantics/fixture/runner identity and outcome denominators. Singleton bootstrap intervals and missing noise floors could yield a diagnostic GREEN that looked like public evidence. Counting rows also permitted duplicated trials to create artificial precision.

Decision

Baseline Books now emit schema version 2. Existing immutable version 1 books remain readable for diagnostics; never rewrite them to imply evidence that was not collected. Each new cell records:

• comparison_identity, independently versioned as schema 1;
• independent_n, the number of distinct recorded (bench_id, run) trials;
• attempted_n, including skipped rows, and failure_n/skipped_n;
• the existing measured metric count n, confidence interval and noise floor, plus the interval's recorded confidence level.

independent_n is an operational distinct-trial count, not proof of statistical independence. Repeated sessions and valid experiment design remain necessary. Missing/empty campaign IDs or missing/noninteger trial indices leave this count null. Candidate duplicate trial IDs are rejected for claims; a baseline's distinct count must equal its metric sample count.

Common identity includes command track/semantics, fixture/scenario/oracle versions, environment isolation, runner class, cache/load state, remote transport/server/shaping, token calibration, and controlled agent identity. Actual nested fixture/instruction aliases are normalized. Subject binary hashes, VCS versions and subject-specific prompt labels deliberately differ between comparators and are not equality requirements. Within-subject population checks still reject mixed binary identities. The full field list is centralized in oakbench.populations.COMPARISON_FIELDS.

Known-versus-missing or unequal common identity invalidates comparisons in both modes. Books with duplicate cell keys or unknown identity versions fail loading. Book construction rejects incompatible populations rather than silently pooling them. It preserves failed/skipped/unmeasured outcome denominators alongside the successful metric summary.

The scorecard CLI defaults to --mode claim; --mode diagnostic is explicit. The Python API retains its diagnostic default for existing local tooling. Every result names its mode and claim_eligible; diagnostic results always set that field false and carry a diagnostic-only notice. Existing diagnostic GREEN/RED semantics remain useful locally but cannot establish a public claim.

Claim mode requires complete applicable identity, at least 30 distinct trials in both populations (100 for p99), no duplicate candidate trials, no hidden failures/skips/unmeasured metrics, a measured finite nonnegative baseline noise floor, and a valid baseline confidence interval. Unknown cache/load/model sentinels are not measured identity. Recall/precision guards require versioned output oracles; a patch_apply_ok guard requires a versioned patch oracle and every successful row to explicitly pass it.

Both candidate and recorded baseline confidence levels must be at least 0.95 and below 1. Legacy missing confidence does not imply 95%. Candidate results retain their confidence level and bootstrap draw count. Nonfinite/out-of-range confidence and nonpositive/noninteger draw counts are rejected before analysis. No arbitrary bootstrap-draw floor substitutes for assessing precision.

Absent eligibility evidence yields AMBER with explicit reasons and a nonzero claim-mode CLI exit; missing inputs remain UNMEASURED and also exit nonzero. Incompatible evidence is INVALID. Evidence eligibility does not force GREEN: the existing statistical verdict still decides distance from the target.

Consequences and limits

The eligibility floor is intentionally conservative even for exact resource metrics. It is not a power calculation, tail-precision guarantee, multiplicity correction, or claim that arbitrary row IDs prove independent sampling.

claim_eligible means this scorecard's comparison-evidence prerequisites passed. It does not replace publication gates, provenance/content attestation, preregistered analysis, public failure reporting, or scientific review. A baseline noise-floor value is a recorded measurement, not an authenticated calibration provenance record. Cross-runner-class ratios still require a separate same-run analysis; saved incompatible medians are rejected.

Older rows missing required identity need a fresh measured campaign. The baseline builder must not manufacture source-run identity from the host on which an old JSONL file happens to be summarized. Human/agent study protocol, hierarchical uncertainty and production FDR wiring remain separate work.

benchmark_stats.py now prints all outcome denominators before its success-conditioned statistics, including failure-only and unmeasured cells. Unknown outcomes remain distinct from observed nonzero exits. Skips are excluded from the attempts denominator and reported separately.