Log in
benchmarksPublic

Commit 2caecb1e

· Browse files at this commit

2caecb1e6549

Merged on mrmrs-7776e1
mrmrs · 1 month ago · 6 files ~6 modified
M docs/adr/0009-baseline-comparison-eligibility.md
@@ -21,7 +21,8 @@
2121 - `comparison_identity`, independently versioned as schema 1;
2222 - `independent_n`, the number of distinct recorded `(bench_id, run)` trials;
2323 - `attempted_n`, including skipped rows, and `failure_n`/`skipped_n`;
24-- the existing measured metric count `n`, confidence interval and noise floor.
24+- the existing measured metric count `n`, confidence interval and noise floor,
25+ plus the interval's recorded `confidence` level.
2526
2627 `independent_n` is an operational distinct-trial count, not proof of statistical
2728 independence. Repeated sessions and valid experiment design remain necessary.
@@ -58,6 +59,12 @@
5859 output oracles; a `patch_apply_ok` guard requires a versioned patch oracle and
5960 every successful row to explicitly pass it.
6061
62+Both candidate and recorded baseline confidence levels must be at least 0.95
63+and below 1. Legacy missing confidence does not imply 95%. Candidate results
64+retain their confidence level and bootstrap draw count. Nonfinite/out-of-range
65+confidence and nonpositive/noninteger draw counts are rejected before analysis.
66+No arbitrary bootstrap-draw floor substitutes for assessing precision.
67+
6168 Absent eligibility evidence yields AMBER with explicit reasons and a nonzero
6269 claim-mode CLI exit; missing inputs remain UNMEASURED and also exit nonzero.
6370 Incompatible evidence is INVALID. Evidence eligibility does not force GREEN: