Measure Your Oak Changes

# Measure Your Oak Changes

You are working in a clone of **oak/oak** (the Oak VCS source, a Rust

workspace — verify: `Cargo.toml` exists in your working directory). You are

changing Oak's code, and as you work you measure the effect with the

oak-benchmarks harness. This file tells you how.

Vocabulary used below:

- `OAK_SRC` = your oak clone = the directory you are working in. Never

  another checkout, never an absolute path from someone else's machine.

- `BENCH` = your own clone of the benchmark harness, sitting NEXT TO your

  oak clone. Create it once if it doesn't exist:

      cd "$OAK_SRC/.." && oak clone oak/benchmarks "$(basename "$OAK_SRC")-bench"

  Yours alone: never use another agent's benchmarks checkout, never edit

  files in BENCH (you only run its scripts), and your results land in

  BENCH/results without colliding with anyone.

Both repos use **oak, not git** (`oak status / diff / commit / desc / push`).

## The loop

1. Edit code in `OAK_SRC`.

2. Build: `cargo build --release` (in `OAK_SRC`).

3. Correctness first: `cargo test` (in `OAK_SRC`). A benchmark win that

   breaks a test is a loss — stop and fix.

4. Measure (in `BENCH`). ALWAYS pass your binary explicitly — never rely on

   any default path:

       cd "$BENCH"

       python3 scripts/devloop.py --skip-build \

         --oak-local-bin "$OAK_SRC/target/release/oak"

   One command, one verdict: PASS or REGRESSED vs the installed Oak

   baseline, with its own printed noise floor.

5. Iterate. When done, commit in `OAK_SRC` with `oak desc` carrying your

   numbers (before → after, n, noise floor cleared), then `oak push`.

## When you want the git comparison too

Git is a control group: its numbers only change when the harness changes,

so measure it ONCE per session into a baseline file, then never again:

    cd "$BENCH"

    python3 scripts/bench.py --profile smoke --runs 5 \

      --subjects git --git-modes untracked_cache --results results/baselines

    cp results/baselines/latest.jsonl results/baselines/git-smoke.jsonl

Per experiment, run only the oak subjects (fast):

    python3 scripts/bench.py --profile smoke --runs 5 \

      --subjects oak_installed,oak_local \

      --oak-local-bin "$OAK_SRC/target/release/oak" \

      --randomize-subject-order

Then one report merges both files — git vs installed vs your changeset:

    python3 scripts/regression_report.py \

      results/baselines/git-smoke.jsonl results/latest.jsonl \

      --target oak_local

## Other lanes (run the ones your change touches)

All from `BENCH`, all with your explicit binary:

    python3 scripts/workflow_ab.py --workflows all --runs 3 \

      --subjects oak_installed,oak_local --oak-local-bin "$OAK_SRC/target/release/oak"

    python3 scripts/parallel_contention.py --workers 2,8 \

      --subjects oak_installed,oak_local --oak-local-bin "$OAK_SRC/target/release/oak"

    python3 scripts/mount_probe.py --oak-bin "$OAK_SRC/target/release/oak"

    # mount caveat: your CLI talks to the INSTALLED Mount app; if numbers

    # look insane, suspect a CLI/app version mismatch and say so.

    python3 scripts/opportunities.py   # ranked view of current gaps

## Rules that invalidate your results if broken

- Claims must clear devloop's printed noise floor. n=1 is a smoke signal,

  not a result; use `--runs 5` minimum for claims.

- Lower `information_recall`, broken `pipe_compatible_unified`, unstable

  determinism, or contention integrity failures = REGRESSED, regardless of

  any token/latency win.

- Measurement is machine-serialized via a lock; `[runlock]` waits are

  normal. Never set `OAK_BENCH_LOCK=off`. Build while you wait.

- Never compare `remote.net.*` (network) against `remote.*` (local-file).

- Never edit the harness (BENCH) to improve a number. If you find a harness

  bug, write it up; don't fix it in your oak changeset.

## Reporting format

Per metric: scenario/operation, baseline → new value, n, noise floor

cleared yes/no, and the bench_id of the run. Put it in your `oak desc`.