Log in
benchmarksPublic

Commit 290f1164

· Browse files at this commit

290f11640be1

Merged on mrmrs-1b7902
mrmrs · 1 month ago · 33 files +7 added ~26 modified
M AGENTS.md
@@ -61,16 +61,23 @@
6161 the latest results. Pick a target from the top; don't re-derive priorities
6262 from raw JSONL.
6363 2. Edit Oak in `../oak`, build it (`cargo build --release` there).
64-3. `python3 scripts/devloop.py` — one command, one verdict: PASS, REGRESSED,
65- or UNMEASURED for your changeset vs the Oak baseline.
66-4. Only claim a win that devloop's printed detection limit supports.
64+3. `python3 scripts/devloop.py` — one command, one verdict: PASS (exit 0),
65+ REGRESSED (1), or UNMEASURED (2) for your changeset vs the Oak baseline.
66+4. Only claim a win that devloop's printed per-cell detection limit supports.
6767
6868 Claim discipline (non-negotiable):
6969
70-- **Never claim a delta below the noise floor.** devloop prints its own
71- per-lane detection limits; the default `--runs 2` can only catch large
72- changes. Chasing a 3% win? Bump `--runs` until the limit is below the
73- effect you claim, or the claim is noise.
70+- **Never claim a delta below the noise floor.** devloop judges every
71+ scenario/operation cell on its own interleaved A/A floor (`oak_installed_aa`),
72+ a bootstrap CI and Benjamini-Hochberg across the family, then re-measures
73+ screened cells (`--confirm-runs 20`, within `--confirm-budget-s`) and decides
74+ them on fresh data (ADR-0011). Every no-change cell prints "no change
75+ detected >= X%": that X is the effect you could have seen. Chasing a smaller
76+ win? Raise `--runs`/`--confirm-runs` until X is below it, or the claim is noise.
77+- **Defaults are `--runs 5`; fewer, or `--skip-noise-floor`, is UNMEASURED
78+ (exit 2), never PASS.** An unconfirmed Oak-baseline regression is also
79+ UNMEASURED with the runs needed to confirm it. A Git-guardrail latency breach
80+ fails the changeset only when the same cell is a confirmed Oak-vs-Oak regression.
7481 - **A cheaper output that loses information is a regression, not a win.**
7582 The gates enforce this: `information_recall` < baseline or lost
7683 `pipe_compatible_unified` structure fails the verdict (RECALL /