/oak
Log in
benchmarks
docs/adr/0008-target-scorecards.md 59 lines · 2.7 KB

ADR-0008: Versioned targets drive beat-Git scorecards

  • Status: accepted
  • Date: 2026-06-12

Context

devloop.py answers whether an Oak changeset regressed relative to an Oak baseline. It does not answer how far Oak is from a published Git target, nor whether a target is statistically resolvable on the current instrument. A benchmark intended to drive VCS development needs red/green targets that agents can consume without human interpretation.

Decision

Targets live in config/targets.yaml or a successor versioned config. Each target binds an immutable lane/profile/scenario/operation/metric cell to a goal such as "Oak median latency <= 0.5x git_fsmonitor" or "tokens <= 0.3x Git". Targets reference the fair comparator, not necessarily stock Git.

Every target carries guard metrics by policy: integrity, failure rate, information recall, information precision, pipe compatibility, output stability, and transport comparability must remain valid before an efficiency win can be green.

Scorecard verdicts are:

  • GREEN: confidence interval excludes the goal in the desired direction.
  • RED: measured distance misses the goal.
  • AMBER: confidence interval straddles the goal and prints required n.
  • UNDECIDABLE: the goal is below the measured noise floor.
  • UNMEASURED: rows are absent or skipped with recorded reasons.
  • INVALID: a guard metric failed.

GREEN and RED are decided against noise-inflated confidence bounds: the bootstrap interval only captures within-sample variance, so the ratio bound is widened by the baseline's A/A noise floor before comparison. A decisive result survives the inflation and stays green or red; a narrow interval that clears the goal by less than the noise floor cannot be a verdict — it falls through to UNDECIDABLE (when the goal margin from parity is itself below the noise floor) or AMBER. The goal margin is measured as distance from parity in either direction (a 1.5x goal demands the same 50% resolution a 0.5x goal does), and a parity goal (1.0x) is undecidable on any instrument with a nonzero noise floor.

Partial metric coverage caps decisive verdicts: when the metric is unmeasured on some successful rows, the reported n is the rows that contributed to the statistic (with n_successful alongside), and neither GREEN nor RED may be issued — the best verdict is AMBER, since dropping information can never show green and missing data cannot fail a target either.

Consequences

  • Optimizers cannot make a target green by dropping information or hiding failures.
  • Goals tighter than the instrument's noise floor are called undecidable, not failed or passed.
  • Adding targets is a benchmark-contract change and should be reviewed like a scenario addition.