ADR-0007: Latency comparisons require runner identity and portability class

# ADR-0007: Latency comparisons require runner identity and portability class

- Status: accepted

- Date: 2026-06-12

## Context

Rows already preserve raw measurements, but latency depends on CPU, disk,

kernel, filesystem, power state, and cache discipline. Publishing cross-machine

latency medians as if they were one population hides runner effects. At the

same time, many benchmark metrics are machine-independent: output bytes, token

counts, tool calls, information recall, pipe compatibility, and integrity

checks should reproduce exactly given the same fixture and subject behavior.

## Decision

Every campaign records a runner profile and a stable `runner_id`; rows may

stamp `runner_id`, `runner_class`, and `runner_profile` additively. Latency

summaries and Git Baseline Books declare their `runner_class`.

Metric portability is explicit:

- Machine-independent metrics compare by equality or exact deterministic

  summary within the fixture/semantics/calibration version.

- Machine-dependent metrics compare inside one runner class. Across runner

  classes, the portable latency claim is the same-run subject ratio with a

  confidence interval.

## Consequences

- Dashboards must not pool latency rows across runner classes.

- Public reports name the runner class beside latency claims.

- New rows can add runner fields without changing scenario or operation

  identity.