Runner discipline: identity, cache state, environment, calibration

# Runner discipline: identity, cache state, environment, calibration

This document describes how the harness keeps latency rows honest about the

machine they ran on. The underlying contracts are ADR-0007 (runner identity

and portability) and ADR-0002 (null means unmeasured).

## Runner identity (ADR-0007 recap)

Every campaign captures a machine profile (`oakbench.runner.machine_profile`)

and derives a stable `runner_id` from its stable fields plus a `runner_class`

from `config/runner_classes.json`. Rows stamp these fields additively.

- Machine-independent metrics (bytes, tokens, tool calls, integrity) compare

  by equality across runners.

- Machine-dependent metrics (latency) compare only inside one runner class.

  Across classes, the portable claim is the same-run subject ratio with a

  confidence interval. Dashboards must never pool latency across classes.

## Cache discipline (`oakbench.cachectl`)

Cache state is a measurement, not a wish. `purge_fs_caches()` attempts an OS

page-cache purge (`/usr/sbin/purge` on macOS, `/proc/sys/vm/drop_caches` on

Linux) and reports `{attempted, ok, method, error}` without ever raising.

`cache_fields(requested, purge_result, *, warmed_by_scenario=False)` turns the

request plus the evidence into row fields:

- `requested="cold"` + successful purge → `cache_state="cold"`.

- `requested="cold"` + failed or missing purge → `cache_state="unknown"` with

  a `cache_state_reason` naming the purge failure. Unknown is honest; the run

  still happened, the row just refuses to claim coldness.

- `requested="warm"` → `cache_state="warm"` only when the caller passes

  `warmed_by_scenario=True`, i.e. it actually ran a warming pass. The warming

  claim belongs to the scenario, never to the bookkeeping.

- No request → `cache_state="unknown"`, reason `cache_state_unset`.

- Rows written before this discipline existed are labeled

  `unspecified_legacy` by consumers; they are never retro-claimed cold or warm.

## Environment boundary sampling (`oakbench.envwatch`)

`sample_environment()` snapshots load average, available memory, thermal state

(macOS `pmset -g therm` CPU_Speed_Limit, Linux thermal zones), and the CPU

governor. Every field is nullable: a host that cannot provide a signal yields

`None`, never a guess. Callers sample before and after the measurement window

and stamp timestamps themselves.

`environment_suspect(before, after)` returns human-readable suspicion reasons

(empty list = clean window):

- load average per core above the threshold (default 1.5) at either boundary;

- thermal throttling (CPU_Speed_Limit below 100);

- Linux CPU governor not `performance` (informational).

Flagged rows are never deleted — suspicion is metadata, and deleting data

would hide the very noise the flag exists to expose. As a policy hook for the

publish gate: if more than 10% of a campaign's rows are suspect, the campaign

should fail publication and be re-run, not trimmed.

## Calibration (`scripts/runner_calibration.py`)

A fixed ~20-second micro-suite scores the runner with three probes: sha256

over 64 MiB of deterministic in-process bytes (CPU), write+fsync+read of a

64 MiB temp file in the probe directory (IO), and the median of 50

`python -c pass` spawns. Output is one JSON object with `runner_id` and

`runner_class` from `oakbench.runner`:

```

python3 scripts/runner_calibration.py --path /probe/dir

```

`--quick` scales sizes down 8x for tests. Calibration is advisory only: it

contextualizes latency rows and supports sanity checks across runner classes,

but no normalization of measured latencies is ever derived from it and no

threshold turns it into a gate.

## fsync policy

The harness never adds or removes `fsync` around the subject under test.

Durability behavior is part of what a VCS or tool is — flushing for it (or

suppressing its flushes) would measure a different subject. The only fsync the

harness issues is inside its own IO calibration probe, which measures the

runner, not the subject.