Log in
benchmarksPublic

Viewing files at commit 538751fb07e6Commit detailsView latest

Browse files
AGENTS.md 115 lines · 5.2 KB

AGENTS.md

Guidance for AI coding agents working in this benchmark repository.

This repo uses Oak

Use oak, not Git, for version control in this repository.

oak status
oak diff
oak commit          # local checkpoint only; does not publish
oak desc --file /tmp/oak-branch-desc.txt
oak push --repo oak/benchmarks

Do not mutate canonical checkouts such as /Users/mrmrs/o/oak, /Users/mrmrs/o/benchmarks, or /Users/mrmrs/o/oakspace. Work in an isolated worker-mrmrs-* checkout and report the full worker path with every branch, commit, and validation result.

Working On The Harness

  • Shared measurement policy (subjects, env, tokens, row contract, command semantics, stream adapters, reporting math) lives in scripts/oakbench/. Change it there, once — never copy a policy into a lane script.
  • Read CONTEXT.md for vocabulary and docs/adr/ for decisions before proposing structural changes.
  • Run the instrument tests before pushing: python3 -m unittest discover -s tests.
  • Null means unmeasured (ADR-0002): never emit a fabricated zero for a signal the runner did not observe.
  • Scenario and operation names are immutable identity (ADR-0005): changed semantics require new names or a version bump.
  • Before running remote-backed lanes, source the repo-local remote config: . config/bench-env.sh. Do not rely on ~/.zshrc; non-interactive agent shells often do not load it.

Finishing A Task

Before handing work back, leave the repo in a reviewable, server-visible state:

oak status
python3 -m unittest discover -s tests
oak commit
oak desc --file /tmp/oak-branch-desc.txt
oak push --repo oak/benchmarks

oak commit is local-only. Use oak push after validation so reviewers can see the branch. Do not merge your own branch unless the user explicitly asks for a safe-merge/land flow.

Improving Oak Against These Benchmarks (swarm runbook)

You are probably here to make Oak (../oak) faster and cheaper for agents. The loop:

  1. python3 scripts/opportunities.py — the ranked attack list generated from the latest results. Pick a target from the top; don't re-derive priorities from raw JSONL.
  2. Edit Oak in ../oak, build it (cargo build --release there).
  3. python3 scripts/devloop.py — one command, one verdict: PASS (exit 0), REGRESSED (1), or UNMEASURED (2) for your changeset vs the Oak baseline.
  4. Only claim a win that devloop's printed per-cell detection limit supports.

Claim discipline (non-negotiable):

  • Never claim a delta below the noise floor. devloop judges every scenario/operation cell on its own interleaved A/A floor (oak_installed_aa), a bootstrap CI and Benjamini-Hochberg across the family, then re-measures screened cells (--confirm-runs 20, within --confirm-budget-s) and decides them on fresh data (ADR-0011). Every no-change cell prints "no change >= X% at 80% power": a true change of X% or more would have been detected at least 80% of the time (calibrated on real A/A rows). Smaller changes may pass undetected. Chasing a smaller win? Raise --runs/--confirm-runs until X is below it, or the claim is noise.
  • Defaults are --runs 5; fewer, or --skip-noise-floor, is UNMEASURED (exit 2), never PASS. An unconfirmed Oak-baseline regression is also UNMEASURED with the runs needed to confirm it. A Git-guardrail latency breach fails the changeset only when the same cell is a confirmed Oak-vs-Oak regression.
  • A cheaper output that loses information is a regression, not a win. The gates enforce this: information_recall < baseline or lost pipe_compatible_unified structure fails the verdict (RECALL / PIPE-COMPAT lines). Don't try to win tokens by omitting changed-file names or unified-diff hunks.
  • Never aggregate across instruction levels or transports. remote.net.* rows (real server, network) are oak-vs-previous-oak trend evidence only — they are deliberately named differently from git's local-file remote.* ops and must never be compared against them.
  • Measurement serializes; building parallelizes. Lanes take a cross-process measurement lock so concurrent agents don't poison each other's timings. If your run waits on the lock, that's correct behavior — do your editing/building while waiting, never set OAK_BENCH_LOCK=off on a shared machine.
  • Run with your own results dir when working in parallel (--results results/<your-task-slug>), and leave results/latest.* to serialized verdict runs.
  • Skips are work items, not noise. A returncode-77 row names exactly what infrastructure would unlock a measurement (see the "Unmeasured" section of opportunities.py). Unlocking coverage is as valuable as improving a number.

Benchmark Data Hygiene

  • Do not commit raw real-agent transcripts, generated result JSONL, run artifacts, or temporary workspaces.
  • Keep benchmark scenarios, scripts, configs, dashboard schema, and docs in the repo.
  • Put large/generated benchmark outputs in external storage or ignored local directories.
  • Treat --agent-environment minimal as the default for publishable token comparisons; use local-default only when intentionally measuring local agent customization overhead.