Benchmark workstation

# Benchmark workstation

Use a fresh isolated worker checkout. The harness requires Python 3.9+ and the

subjects under test; its core has no required third-party Python packages.

Tests with local HTTP fixtures require permission to bind loopback sockets.

They do not require production remotes.

## Create an isolated checkout

Read the workspace and repository `AGENTS.md` instructions first. In the

standard Oak workspace, use its required worker naming convention:

```sh

: "${WORKSPACE_ROOT:?Set WORKSPACE_ROOT to the registry allowed by your AGENTS.md}"

WORKER="$WORKSPACE_ROOT/worker-mrmrs-$(date -u +%Y%m%dT%H%M%S)-benchmark-workstation"

mkdir -p "$WORKER"

oak clone oak/benchmarks "$WORKER/bench"

cd "$WORKER/bench"

pwd

oak status --json

```

Set `WORKSPACE_ROOT` before running this example. Before edits or generated runs, confirm the exact

worker path, selected branch, head and clean/dirty state. Canonical repository

checkouts are not disposable workers.

## Configure subjects and serialization

The default subject configuration finds Git and installed Oak on `PATH`. Copy

it outside the repository and set the selected subjects' `bin` entries to the

absolute executable paths you intend to test:

```sh

export BENCH_SUBJECTS="$WORKER/subjects.toml"

cp config/subjects.toml "$BENCH_SUBJECTS"

command -v git

command -v oak

```

For repeated measurements, place candidate/released Oak binaries in stable

worker-owned paths. Record executable hashes, versions, source revisions and

host state. Copying an executable does not pin its libraries, system helpers

or services; recheck provenance after host updates.

The harness forces `OAK_NO_UPDATE_CHECK=1` so Oak's first command in a fresh

benchmark HOME does not charge an update-server request to whichever subject

runs first. Measure that cold-home user experience separately if it is a study

objective.

Set the same explicit lock path in every process measuring on this machine.

Keep it outside individual workers so deleting a worker does not replace the

shared lock. The location is configurable:

```sh

export OAK_BENCH_LOCK_PATH="$WORKSPACE_ROOT/oak-bench-measurement.lock"

```

The default lock lives under the process temporary directory, which can differ

between environments. It does not establish a machine-wide shared lock by

itself. Keep locking enabled and run measurements after builds and tests have

stopped. Do not stop another worker's campaign. Campaign plans pin the canonical

lock path and refuse a different path on resume.

## Local validation

From the new worker checkout:

```sh

python3 -m unittest discover -s tests

# Diagnostic smoke; one repetition cannot establish a public speedup.

python3 scripts/bench.py --profile smoke --runs 1 \

  --config "$BENCH_SUBJECTS" --subjects git,oak_installed \

  --skip-remote --randomize-subject-order \

  --workdir "$WORKER/workdirs/core-smoke" --results "$WORKER/results/core-smoke"

```

Retain logs, raw rows and generated artifacts outside source control. A smoke

failure is a development signal, not evidence to discard or relabel as a skip.

A successful setup is not certification of a dedicated performance runner.

Historical review findings and validation counts belong to the dated review

documents, not assumptions about the current machine.

## Agent campaigns

Follow the [agent campaign plan](agent-campaign-plan.md) to copy and configure

the smoke manifest, freeze a schedule, and use the plan/run/status/report CLI.

Point its `subjects_config` at the same intended subject configuration and

select the mock agent for initial validation. Real agents require explicit

enablement and worker attestation; that attestation does not provide an OS

sandbox for unrestricted agent CLIs.

Preserve the campaign directory for status, reports and resume. Do not edit

its frozen plan or delete failed trials to improve a result. A changed model,

executable, instruction condition, configuration, lock path or harness requires

a new plan. A finished mock campaign verifies harness execution and accounting;

it does not estimate an agent's success rate or a VCS speedup.

## Expand to a performance campaign

Clone Oak source into another isolated worker before building a candidate.

`scripts/prepare-subjects.sh` invokes Cargo and writes its target directory;

its `OAK_REPO` must therefore identify an isolated source checkout, never a

canonical repository. Set `OUT` to a stable path owned by the task.

Use pinned baseline/candidate binaries and source provenance, equal semantic

tracks, matched transport, seeded randomized/interleaved repetitions, recorded

cache evidence and runner state, and a separate A/A noise campaign. Choose

sample size for the effect and uncertainty needed; a minimum count alone is

not sufficient precision. See [runner discipline](runner-discipline.md),

[statistical methodology](statistical-methodology.md), and the dated reviews

for current limitations.

Source `config/bench-env.sh` only for deliberately configured remote-backed

lanes. It supplies existing named disposable remotes, not per-run isolation.

Allocate and track run-owned repositories before lifecycle or contention

campaigns. Missing credentials, FUSE, Linux network privileges, LFS,

real-agent binaries/models and hosted CI are capability gaps, not zero cost

or successful measurements.

For human usability, pilot the separate

[human study protocol](human-vcs-study-protocol.md). Neither a scripted

workflow nor a mock-agent row substitutes for a participant observation.