AGENTS.md

# AGENTS.md

Guidance for AI coding agents working in this benchmark repository.

## This repo uses Oak

Use `oak`, not Git, for version control in this repository.

```bash

oak status

oak diff

oak commit          # local checkpoint only; does not publish

oak desc --file /tmp/oak-branch-desc.txt

oak push --repo oak/benchmarks

```

Do not mutate canonical checkouts such as `/Users/mrmrs/o/oak`,

`/Users/mrmrs/o/benchmarks`, or `/Users/mrmrs/o/oakspace`. Work in an isolated

`worker-mrmrs-*` checkout and report the full worker path with every branch,

commit, and validation result.

## Working On The Harness

- Shared measurement policy (subjects, env, tokens, row contract, command

  semantics, stream adapters, reporting math) lives in `scripts/oakbench/`.

  Change it there, once — never copy a policy into a lane script.

- Read `CONTEXT.md` for vocabulary and `docs/adr/` for decisions before

  proposing structural changes.

- Run the instrument tests before pushing: `python3 -m unittest discover -s tests`.

- Null means unmeasured (ADR-0002): never emit a fabricated zero for a signal

  the runner did not observe.

- Scenario and operation names are immutable identity (ADR-0005): changed

  semantics require new names or a version bump.

- Before running remote-backed lanes, source the repo-local remote config:

  `. config/bench-env.sh`. Do not rely on `~/.zshrc`; non-interactive agent

  shells often do not load it.

## Finishing A Task

Before handing work back, leave the repo in a reviewable, server-visible

state:

```bash

oak status

python3 -m unittest discover -s tests

oak commit

oak desc --file /tmp/oak-branch-desc.txt

oak push --repo oak/benchmarks

```

`oak commit` is local-only. Use `oak push` after validation so reviewers can

see the branch. Do not merge your own branch unless the user explicitly asks

for a safe-merge/land flow.

## Improving Oak Against These Benchmarks (swarm runbook)

You are probably here to make Oak (`../oak`) faster and cheaper for agents.

The loop:

1. `python3 scripts/opportunities.py` — the ranked attack list generated from

   the latest results. Pick a target from the top; don't re-derive priorities

   from raw JSONL.

2. Edit Oak in `../oak`, build it (`cargo build --release` there).

3. `python3 scripts/devloop.py` — one command, one verdict: PASS (exit 0),

   REGRESSED (1), or UNMEASURED (2) for your changeset vs the Oak baseline.

4. Only claim a win that devloop's printed per-cell detection limit supports.

Claim discipline (non-negotiable):

- **Never claim a delta below the noise floor.** devloop judges every

  scenario/operation cell on its own interleaved A/A floor (`oak_installed_aa`),

  a bootstrap CI and Benjamini-Hochberg across the family, then re-measures

  screened cells (`--confirm-runs 20`, within `--confirm-budget-s`) and decides

  them on fresh data (ADR-0011). Every no-change cell prints "no change >= X%

  at 80% power": a true change of X% or more would have been detected at least

  80% of the time (calibrated on real A/A rows). Smaller changes may pass

  undetected. Chasing a smaller win? Raise `--runs`/`--confirm-runs` until X is

  below it, or the claim is noise.

- **Defaults are `--runs 5`; fewer, or `--skip-noise-floor`, is UNMEASURED

  (exit 2), never PASS.** An unconfirmed Oak-baseline regression is also

  UNMEASURED with the runs needed to confirm it. A Git-guardrail latency breach

  fails the changeset only when the same cell is a confirmed Oak-vs-Oak regression.

- **A cheaper output that loses information is a regression, not a win.**

  The gates enforce this: `information_recall` < baseline or lost

  `pipe_compatible_unified` structure fails the verdict (RECALL /

  PIPE-COMPAT lines). Don't try to win tokens by omitting changed-file

  names or unified-diff hunks.

- **Never aggregate across instruction levels or transports.**

  `remote.net.*` rows (real server, network) are oak-vs-previous-oak trend

  evidence only — they are deliberately named differently from git's

  local-file `remote.*` ops and must never be compared against them.

- **Measurement serializes; building parallelizes.** Lanes take a

  cross-process measurement lock so concurrent agents don't poison each

  other's timings. If your run waits on the lock, that's correct behavior —

  do your editing/building while waiting, never set `OAK_BENCH_LOCK=off` on

  a shared machine.

- **Run with your own results dir when working in parallel**

  (`--results results/<your-task-slug>`), and leave `results/latest.*` to

  serialized verdict runs.

- **Correctness is gated, not only speed.** devloop's `conformance` lane runs

  the roundtrip conformance matrix (`scripts/vcs_conformance.py`, see

  `docs/vcs-conformance.md`) against your Oak build. Oak state is checked by the

  independent stdlib reader (`oakbench/oak_sqlite_reader.py`), never by the

  binary under test. A behaviour that newly fails is REGRESSED; if you fix a

  listed failure, remove its entry from `config/conformance_expected_failures.json`.

- **Skips are work items, not noise.** A returncode-77 row names exactly

  what infrastructure would unlock a measurement (see the "Unmeasured"

  section of opportunities.py). Unlocking coverage is as valuable as

  improving a number.

## Benchmark Data Hygiene

- Do not commit raw real-agent transcripts, generated result JSONL, run

  artifacts, or temporary workspaces.

- Keep benchmark scenarios, scripts, configs, dashboard schema, and docs in the

  repo.

- Put large/generated benchmark outputs in external storage or ignored local

  directories.

- Treat `--agent-environment minimal` as the default for publishable token

  comparisons; use `local-default` only when intentionally measuring local agent

  customization overhead.