Log in
benchmarksPublic

Commit 2735d7f5

· Browse files at this commit

2735d7f5558e

Merged on mrmrs-eec7ec
mrmrs · 1 month ago · 24 files +1 added ~23 modified
M CONTEXT.md
@@ -19,7 +19,7 @@
1919 | **Agent-emitted / agent-ingested** | Billing-direction token accounting: text the model emits (commands, prose) is output-priced; text it reads back (tool results) is input-priced. `cost_weighted_total` combines them. |
2020 | **Admitted output** | The capped portion of command stdout/stderr treated as agent-visible context for token estimates. Full byte counts are always recorded alongside. |
2121 | **Determinism probe** | A diagnostic row pair checking byte-stability of identical-state output (prompt-cache friendliness) and ANSI pollution in non-TTY output. |
22-| **Skip row** | A row with `returncode: 77` and `skip_reason`, recording that a measurement could not run (missing capability, no remote). Skips are recorded, never hidden. |
22+| **Skip row** | A row with `returncode: 77`, `skipped: true` and `skip_reason`, recording that a measurement could not run (missing capability, no remote). Skips are recorded, never hidden. |
2323 | **Measurement source** | Per-metric-group string saying how values were obtained (adapter stream, transcript classification, filesystem probe). Accompanies the null-honesty contract. |
2424 | **Baseline** | The comparison anchor: Git baseline (`git`), Oak baseline (`oak_main` if present, else `oak_installed`). The release-blocking comparison is target (`oak_local`) vs Oak baseline. |
2525 | **Efficiency gate** | A CI regression threshold on exact metrics (tool calls, output bytes) applied to the Oak-baseline comparison only (ADR-0004). |