Adding Benchmarks: The Playbook

# Adding Benchmarks: The Playbook

The mechanical checklist for adding or changing a scenario, workflow, or lane.

Written so an agent can follow it without judgment calls; the judgment is

already encoded in `CONTEXT.md` and `docs/adr/`.

## Before you start

1. Read `CONTEXT.md` (vocabulary) and `docs/adr/` (decisions you must not

   re-litigate). Use the glossary terms exactly.

2. Shared measurement policy lives in `scripts/oakbench/`. If your change

   touches how anything is measured (clocks, tokens, environment, row fields,

   commands), it goes there — never copied into a lane.

## Adding a scenario or workflow to an existing lane

1. **Name it forever.** Scenario and operation names are immutable identity

   (ADR-0005). Pick names that won't need to change; never reuse a retired

   name.

2. **Fixture**: deterministic bytes only. Generation is untimed. Use

   `oakbench.fixtures.write_pattern_file` for bulk content. Bump the lane's

   fixture version (`FIXTURE_VERSION` / `fixture_version`) if you change any

   existing fixture's bytes.

3. **Commands through the contract.** VCS commands come from

   `oakbench.command_semantics` (backed by `config/command_semantics.json`).

   If the scenario needs a command the contract lacks, add it to the JSON

   (both tracks, both subject kinds, a `compatibility_note` where one subject

   lacks the capability) and bump the contract version. Capability gaps run

   the closest available command and are recorded, never hidden.

4. **Snapshot accounting**: git pays stage+commit as two recorded calls, oak

   one. Use `snapshot_steps` / `semantics().snapshot_commands`; never

   hand-roll a combined `add && commit` call.

5. **Tolerated failures**: steps that are SUPPOSED to fail use

   `expected_returncodes`; the row keeps `process_returncode` as the truth.

6. **Oracle**: deterministic only — validators, changed-file shape

   (`expected_change_ok` for the agent lane), clean final VCS state. The

   agent's final message is never an oracle. Read-only scenarios must require

   an untouched tree.

7. **Null honesty** (ADR-0002): any signal your scenario cannot observe is

   `null` with a `measurement_source`, never a fabricated zero.

8. **Spec entry**: add the scenario to the lane's YAML under `scenarios/`

   (match the existing anchor structure) and a row to

   `docs/benchmark-coverage.md` with an honest status badge.

9. **Tests**: add or extend a test in `tests/`. Minimum: the scenario appears

   in the catalog, its oracle shape accepts/rejects correctly, and (for new

   step recipes) the operations list matches expectations. If it parses new

   external output, add a golden fixture under `tests/fixtures/`.

## Changing measurement behavior

1. If the commands, fixture bytes, or semantics of an existing

   scenario/operation change materially, that is a NEW name or a version

   bump — not an edit in place (ADR-0005).

2. If only harness code changed, prove measurement identity:

   ```bash

   # before your change

   python3 scripts/workflow_ab.py --workflows all --results /tmp/parity-old

   # after your change

   python3 scripts/workflow_ab.py --workflows all --results /tmp/parity-new

   python3 scripts/row_parity.py /tmp/parity-old/latest.jsonl /tmp/parity-new/latest.jsonl

   ```

   Zero invariant drift required. Repeat for any other lane you touched

   (`bench.py --profile micro` is the fastest core-lane parity source).

3. Run the instrument tests: `python3 -m unittest discover -s tests`.

## Adding a new lane

1. Define the lane's row shape in `oakbench.rows.LANE_REQUIRED_KEYS` and

   write results through `oakbench.results.ResultsStore(results_dir, lane=...)`

   so the contract is enforced at write time.

2. Build on the oakbench primitives: `subjects.load_subjects`,

   `environment.base_env`, `execution.run_timed`,

   `tokens.interaction_token_fields`, `reporting` math.

3. Standard CLI flags: `--subjects`, `--config`, `--git-bin`,

   `--oak-installed-bin`, `--oak-local-bin`, `--results`, `--runs`,

   `--keep-workdirs`, `--randomize-subject-order` where ordering matters.

4. Skip rows (`returncode: 77`, `skip_reason`) for anything the environment

   cannot run — recorded, never hidden.

5. Add: a glossary entry in `CONTEXT.md`, a coverage row in

   `docs/benchmark-coverage.md`, a README section, an integration test in

   `tests/` (tiny scale, contract-validated), and devloop wiring if the lane

   should gate Oak development.

## Adding an agent CLI adapter

1. New entry in `config/agents.toml` plus a command builder in

   `agent_workflow.build_agent_command`.

2. New `StreamAdapter` subclass in `oakbench/stream_adapters.py` for the CLI's

   stream shape — never widen the shared heuristics for one CLI's quirk.

3. Capture a real transcript, sanitize it (paths, ids), and add it as a golden

   fixture under `tests/fixtures/` with assertions on turns, usage, tool

   outcomes, and truncation semantics.

## Before pushing

```bash

python3 -m unittest discover -s tests        # instruments

python3 scripts/devloop.py --lanes core ...  # the loop your change serves

oak status                                   # nothing generated is staged

```

Raw results, transcripts, and workdirs never get committed (ADR-0003).

## Adding a new subject (VCS)

Adding a VCS subject is data, not lane code: one plugin JSON under

`config/subject_kinds/`, one `[subjects.<name>]` entry in

`config/subjects.toml`, and golden output samples under `tests/fixtures/`.

Capability gaps are `null` commands that become skip rows

(`capability_gap:<kind>:<operation>`), never substituted commands. The git

and oak plugin JSONs are a parallel read of `config/command_semantics.json`

(which lanes still load unchanged), proven measurement-identical by

`oakbench.subject_kinds.parity_report` — an empty report is the proof, and

`tests/test_subject_kinds.py` enforces it. Full recipe with the jujutsu

pilot as the worked example: `docs/adding-a-subject.md`.