Oak Benchmarks

# Oak Benchmarks

Oak is version control designed for coding agents: fewer state-management

steps, task-oriented branches and spaces, compact machine-readable state, and

lazy workspaces that let an agent inspect and edit before paying full checkout

cost.

This repository is Oak's evidence system. It compares Oak with Git on work that

agents actually do: status, diff, snapshots, branch/task isolation, recovery

from bad states, real-agent tool loops, hosted integration, contention, lazy

mounts, output bytes, tool calls, token pressure, turn count, and correctness.

The marketing rule is simple: Oak wins where rows prove it. Gaps become

explicit skip rows and roadmap items. Public claims must cite raw JSONL, tuned

Git baselines, runner identity, sample counts, command track, source

provenance, recall/pipe-compatibility checks, and the measured noise floor.

This repo is intentionally separate from the Oak source checkout so agents can

change Oak while benchmark scaffolding stays isolated.

## Subjects

The default subjects are:

- `git`: stock Git on the same machine.

- `oak_installed`: the Oak binary found on `PATH`.

- `oak_local`: an optional Oak binary built from a local Oak source checkout.

- `oak_main`: an optional clean-main Oak binary for exact main-vs-local

  regression checks.

The suite tracks two comparisons:

- **Oak vs Git**: whether Oak's agent-shaped workflow is cheaper, faster, or

  more reliable than Git for the same task.

- **Oak vs previous Oak**: whether a local Oak changeset improved or regressed

  the current Oak baseline.

When an Oak source checkout is available, rows include source metadata from

`oak hash` and `oak status`. Set `OAK_REPO` or pass `--oak-repo` when the Oak

checkout is not a sibling directory named `oak`.

## Quick Start

```bash

oak clone oak/benchmarks benchmarks

cd benchmarks

python3 scripts/bench.py --profile smoke

python3 scripts/regression_report.py results/latest.jsonl

```

The core harness needs Python 3.9+ and the benchmarked VCS binaries. Other

lanes may need credentials or platform capabilities: Git LFS for `git_lfs`,

Linux `netns`/`tc` privileges for shaped-network runs, GitHub/Oak disposable

repos for hosted/platform lanes, and model CLIs for real-agent campaigns.

Missing capabilities should produce skip rows, not silent omissions.

Before adjudicating or starting a cross-repo campaign, render the read-only

field map from isolated clones:

```bash

python3 scripts/oak_field_map.py \

  --repo /path/to/worker-.../oak \

  --repo /path/to/worker-.../benchmarks

```

## End-to-End Agent Campaigns

For repeatable end-to-end agent experiments, use the manifest-driven

[`agent_campaign.py`](scripts/agent_campaign.py) runner. It freezes a paired

schedule, preserves interrupted attempts, and reports failures and missing

measurements alongside successful-task efficiency. Start with the free mock

campaign; mock results verify the instrument, not agent performance.

```bash

python3 scripts/agent_campaign.py plan \

  --manifest config/agent-campaign-smoke.json \

  --campaign-dir results/agent-campaign-smoke

python3 scripts/agent_campaign.py run --campaign-dir results/agent-campaign-smoke

python3 scripts/agent_campaign.py report --campaign-dir results/agent-campaign-smoke

```

The [campaign plan](docs/agent-campaign-plan.md) describes the experiment

contract, implementation status, and launch criteria for real-agent pilots

and confirmatory studies. Campaign outputs and transcripts stay uncommitted.

## The Dev Loop

For agents developing Oak, `devloop.py` answers one question: is this Oak

changeset better, same, or worse?

```bash

python3 scripts/devloop.py                  # cargo-builds ../oak, runs core/workflow/contention

python3 scripts/devloop.py --lanes core     # fastest signal

python3 scripts/devloop.py --oak-local-bin ../oak/target/release/oak --skip-build

```

devloop measures before it judges. An A/A null test sets the host's noise

floor, embedded null controls catch lane noise, and the report prints detection

limits. A delta below the measured floor is not a claim.

The verdict gates on the Oak-baseline comparison: latency above noise, tool-call

increases, output-byte growth, new failures, information-recall loss,

pipe-compatibility loss, and contention integrity. New Git-guardrail breaches

are called out; pre-existing Oak-vs-Git gaps are tracked without failing a

changeset. Exit 0 means PASS, exit 1 means REGRESSED.

## What The Suite Measures

Oak's pitch is not just "faster command." The suite measures the full cost an

agent feels:

- Wall-clock latency for core VCS operations.

- Tool calls and terminal calls.

- Agent-emitted and agent-ingested token pressure, including tool-call

  envelope costs.

- Output bytes, truncation, ANSI pollution, determinism, and prompt-cache

  friendliness.

- Information recall and unified-diff pipe compatibility, so compact output

  cannot win by dropping facts.

- Provider-reported tokens, turn counts, recovery cost, and follow-up calls

  for real coding-agent runs.

- Contention, branch/task isolation, merge throughput, and payload integrity.

- Mount/lazy-hydration time to first useful work and disk/network work avoided.

- Hosted integration waits, API/tool round trips, poll quantization, and branch

  triage outcomes.

Fixture generation, subject binary discovery, and directory copying are outside

the timed window.

## Current Coverage

The implemented and partially implemented lanes are:

| Lane | Entry point | Status |

| --- | --- | --- |

| Core VCS | `scripts/bench.py` | Implemented for init, snapshot, status, diff, branch, task snapshot, remote rows, tuned Git modes, RSS, determinism, recall, pipe compatibility, stats, and regression gates. |

| Scripted workflows | `scripts/workflow_ab.py` | Implemented for deterministic bugfix, wide refactor, large asset, history archaeology, error recovery, and sync/divergence recovery workflows. |

| Real agents | `scripts/agent_workflow.py` | Implemented for mock plus local Codex, Claude, and Cursor Agent stream adapters. Use real rows only when the installed CLI stream has campaign evidence. |

| Parallel contention | `scripts/parallel_contention.py` | Implemented for Git shared/workspace modes; Oak workspace-per-task rows require a disposable Oak remote and otherwise skip honestly. |

| End-to-end task loop | `scripts/task_loop.py` | Git local-file cold-clone/warm-worktree rows, successful Oak two-fresh-checkout work against owned loopback `oak serve`, and actual Oak loopback interrupted-publication evidence are runnable without production writes. Local Oak CI/review and transport-matched hosted phases remain unmeasured. |

| Mount/lazy hydration | `scripts/mount_probe.py`, `scripts/mount_vs_clone.py` | Dry probe and partial remote-backed coverage. Real mount timing needs `OAK_BENCH_MOUNT_REPO`; large-mirror Oak-vs-Git acquisition is not publishable yet. |

| Platform lifecycle | `scripts/platform_lifecycle.py` | Implemented for capability probes, hosted integration anatomy, poll-until-merged, race scenarios, and fake-provider branch triage. Real hosted rows need credentials and disposable repos. Excluded from devloop. |

| Netshape | `scripts/netshape_bench.py` | Implemented where Linux network namespaces and `tc` are available. Cross-subject latency claims require the same shaped pipe and server identity. |

| Dashboard/control plane | `cloudflare/` | Worker/D1/R2 scaffold exists; live historical dashboard claims require deployed ingest and archived rows. |

Coverage details and public-claim rules live in

`docs/benchmark-coverage.md`, `docs/publish-checklist.md`, and

`docs/statistical-methodology.md`.

The [September 2026 quality review](docs/review-2026-09-09.md) records current

testing strategies, major evidence gaps, and independently authored behavioral

ports from upstream VCS test suites. Start local development with the

[workstation guide](docs/benchmark-workstation.md). Human usability remains

unmeasured; the [proposed participant protocol](docs/human-vcs-study-protocol.md)

defines the separate study needed to evaluate it.

The [gap-closure follow-up](docs/gap-closure-2026-09-10.md) adds trusted task

validators, exact output/patch checks, and saved-baseline claim eligibility.

Scorecard users should read [ADR-0009](docs/adr/0009-baseline-comparison-eligibility.md)

before comparing legacy books or publishing a verdict.

## In Development And Roadmap

The benchmark suite is also a product map for Oak. Current gaps are intentional

work items, not hidden caveats:

- Compact agent-facing commands: Oak has verified core-equivalent `diff --stat`

  and `diff --print` mappings in `config/command_semantics.json`; still needed

  are short/porcelain or JSON status, diff name-only, and quiet commit output.

- Remote-backed Oak rows: Git local-file `remote.*` rows are implemented; Oak

  network rows use `remote.net.*` when a disposable remote is configured. Local

  file and network transports are never latency-comparable.

- Lazy acquisition story: the suite is built to show time-to-first-read and

  bytes-to-first-read for Oak mount versus Git clone variants, but no public

  Oak-vs-Git acquisition claim exists until Oak mount succeeds on the large

  mirror and enough interleaved repetitions are captured.

- Workspace-per-agent fleets: Git worktree contention is implemented. Oak

  workspace-per-task needs real disposable remotes, high-concurrency mount/push

  behavior, branch cleanup, and merge/integration plumbing.

- Full hosted task loop: local Git cold-clone/warm-worktree and Oak

  interrupted-publication cells are implemented. A transport-matched Oak/GitHub

  acquisition -> publish -> exact-CI -> review comparison still needs disposable

  provider fixtures; missing provider phases stay explicit skip rows.

- Hosted workflow evidence: platform rows need GitHub org assets, PATs or app

  credentials, webhooks, disposable repos, and real Oak/GitHub provider wiring

  before public hosted-branch claims.

- XL and monorepo evidence: generators and registry entries exist, but planned

  or null-manifest fixtures are not performance evidence until consumed by

  active profiles on pinned runners.

- Product dependencies exposed by skip rows include a self-hostable Oak server,

  bulk history import, webhooks, branch TTL/cleanup, high-concurrency mounts and

  pushes, 100k-branch scale, and 20 GB repository support.

- Additional subject families such as Jujutsu are pilot configuration only

  until golden outputs and lane wiring land.

The eventual public roadmap should make each of these measurable: a runner

class, a disposable repo or fixture, an operation vocabulary, raw JSONL, a

fair comparator, and a publish gate.

## Profiles

| Profile | Intended use | Shapes |

| --- | --- | --- |

| `micro` | Noise-floor A/A probes, integration tests, fastest regression signal | Tiny text repo only, 3 reps |

| `smoke` | Every local change or push | Tiny text repo, one medium binary, a few large binaries |

| `standard` | Nightly or pre-merge | Many small files, wide dirty tree, 128 MB single file, many 8 MB files |

| `large` | Dedicated runner | 50k small files, 1 GB single file, multi-GB many-large-file repo |

Smoke catches regressions. Public speed claims need pinned hardware, enough

repetitions, randomization, tuned Git rows, and confidence intervals.

## Methodology Guardrails

The machine-readable row is the durable source of truth. Summaries and

dashboards are derived views.

Key contracts:

- `agent-default` measures what an agent would naturally call today.

- `core-equivalent` measures the closest equivalent semantic output level

  across subjects. Rows with compatibility notes are diagnostic, not proof.

- Null means unmeasured, never zero.

- Returncode 77 means skipped with a `skip_reason`; skip rows are coverage

  work items.

- Scenario and operation names are immutable identity. Changed semantics need a

  new name or version.

- Network rows and local-file rows are different measurements and are never

  aggregated together.

- Real-agent rows are never aggregated across instruction levels.

- Latency compares inside one runner class; portable cross-runner claims use

  same-run ratios and confidence intervals.

- Byte/token wins must cite information recall and pipe compatibility from the

  same run.

Methodology docs:

- `docs/command-semantics.md`: `agent-default` versus `core-equivalent`.

- `docs/benchmark-coverage.md`: implemented evidence versus planned coverage.

- `docs/publish-checklist.md`: public claim gates.

- `docs/statistical-methodology.md`: repetitions, confidence intervals, and

  overclaiming rules.

- `docs/tuned-git-baselines.md`: credible Git modes beyond stock Git.

- `docs/runner-discipline.md`: runner identity, cache state, environment

  sampling, and calibration.

## What Is Timed

Core VCS rows time operations such as:

- `repo.init`

- `snapshot.initial`

- `status.clean`

- `status.dirty`

- `diff.dirty`

- `snapshot.dirty`

- `branch.create`

- `task.snapshot`

- `remote.push.first`, `remote.clone.cold`, `remote.pull.uptodate` for Git

  local-file remotes

- `remote.net.push.first`, `remote.net.clone.cold`,

  `remote.net.fetch.uptodate` for Oak network remotes when configured

For Git, staging is part of snapshot timing because agents pay that step. For

Oak, `oak commit --no-verify` is the snapshot operation. Tuned Git modes are

set up outside the timed region after `repo.init`.

Remote rows record `remote_transport` and `remote_server`. Same-machine Git

remote rows isolate VCS transfer cost from network jitter; Oak network rows

measure the real Oak server path. Reports must not subtract one from the other

as a latency delta.

## Agent-Efficiency Metrics

The direct CLI harness records more than wall-clock time:

- `tool_call_count`, `vcs_tool_call_count`, and `terminal_tool_call_count`.

- `estimated_tokens_agent_emitted`, `estimated_tokens_agent_ingested`, and

  `estimated_cost_weighted_tokens`.

- Tool-call envelope fields, because a terminal call includes model-emitted

  tool-use JSON and model-ingested tool-result framing.

- `raw_output_bytes`, admitted output counts, truncation flags, and byte

  counts.

- `proc.spawn` rows for binary startup overhead.

- Output determinism and ANSI-in-pipe probes.

- `peak_rss_bytes` on platforms where child resource usage is observable.

- `status.dirty.inforecall`, `diff.dirty.inforecall`, and

  `diff.full.inforecall` probe rows for bytes saved versus information lost.

- Tail latency summaries with sample-count honesty: p95 requires n>=20 and p99

  requires n>=100.

Tool calls are exact for this harness: each terminal command counts once. Git

snapshot operations intentionally count as two calls (`git add .` plus

`git commit`), while Oak snapshots count as one `oak commit`.

Token counts in direct CLI rows are estimates, not provider billing tokens.

Use `scripts/token_calibration.py` before publishing cross-subject token deltas.

End-to-end agent rows should prefer provider totals from

`token_metrics.total_tokens_reported` when adapters expose them.

## Tuned Git Baselines

Stock Git alone can overstate Oak wins. Tuned Git modes are wired as derived

subjects:

```bash

python3 scripts/bench.py --git-modes untracked_cache,split_index,fsmonitor,lfs

```

Public wide-tree status/diff tables should include at least

`git_untracked_cache` and `git_fsmonitor`. `git_lfs` applies only to configured

binary scenarios and emits explicit skip rows when Git LFS is unavailable or

not applicable.

## Running Lanes

Core:

```bash

python3 scripts/bench.py --profile smoke

python3 scripts/regression_report.py results/latest.jsonl

python3 scripts/benchmark_stats.py results/latest.jsonl

```

Scripted workflows:

```bash

python3 scripts/workflow_ab.py --workflows all --runs 3

```

Real-agent workflow validation:

```bash

python3 scripts/agent_workflow.py --list-agents

python3 scripts/agent_workflow.py --agents mock --subjects git,oak_installed

```

Parallel contention:

```bash

python3 scripts/parallel_contention.py --subjects git,oak_installed \

  --workers 2,8,32 --commits-per-worker 5

```

End-to-end task cost without production writes:

```bash

# Fresh full clone followed by a shared-object Git worktree. Hosted CI remains

# an explicit unmeasured phase on the local-file transport.

python3 scripts/task_loop.py --subjects git --git-remote local \

  --scenarios agent_task_baseline_v1 --runs 10

# Successful Oak clone/edit/commit/push in two independent fresh checkouts,

# backed by an owned loopback `oak serve`. Hosted CI/review remain explicit skips.

python3 scripts/task_loop.py --subjects oak_local \

  --oak-local-bin /path/to/current-oak \

  --scenarios agent_task_oak_local_v1 --runs 10

# Actual Oak HTTP publication attempt whose loopback fixture verifies the

# declared request bytes, then drops the reply. Use pinned released/current

# binaries; no live remote is used.

python3 scripts/task_loop.py --subjects oak_installed,oak_local \

  --oak-installed-bin /path/to/released-oak \

  --oak-local-bin /path/to/current-oak \

  --scenarios interrupted_publication_after_request_v1 \

  --runs 10 --randomize-subject-order

# Legacy selected-branch clone materialization. Run separate A/A and A/B

# configs with the same immutable Serve binary and at least 10 randomized

# pairs. Its fixed fixture has 192 regular 16 KiB files (eight executable),

# one empty file, and one selected-branch-only file: 194 paths total. Fresh

# destinations do not imply cold OS cache.

OAK_BENCH_LOCK_PATH=/abs/shared/oak-bench-measurement.lock \

python3 scripts/task_loop.py \

  --config /abs/path/subjects-ab.toml \

  --subjects oak_baseline,oak_candidate \

  --oak-serve-bin /abs/immutable/fixed-oak-serve \

  --oak-serve-source-head 64_HEX_HEAD \

  --materialization-baseline-subject oak_baseline \

  --scenarios legacy_selected_clone_materialization_v1 \

  --runs 10 --randomize-subject-order --random-seed 46802 \

  --materialization-timing-controls-met \

  --workdir /abs/worker-owned/materialization-ab/work \

  --results /abs/worker-owned/materialization-ab/results

```

Add `--oak-serve-source /abs/read-only/exact-head-checkout` when that immutable

checkout is retained; otherwise rows explicitly record that only the supplied

source head and binary hash were available for provenance.

The scenario records raw proxy byte/connection totals, but its correctness gate

uses normalized pull phases, response/object/chunk counters, and connection

count; it does not pretend variable HTTP framing is product work. Clone, first

post-clone status, and first no-op commit are separate rows. The harness marks

only that its local timing controls are met when

`--materialization-timing-controls-met` is explicit, the shared lock is held,

order is randomized, and at least ten runs are present. That marker is not a

standalone speedup claim: the documented quiet-host window and a separate

interleaved A/A noise campaign are still required evidence.

These cells are different evidence classes. Local Git wall time is not a

network comparison, and the Oak local-server row is loopback-only. Its second

checkout is fresh and does not claim shared-object or OS-cache warmth. The

interruption fixture reports framed-request delivery

knowledge separately from the CLI's own outcome claim and verifies exact local

head, unpublished state, and payload preservation; it does not measure

successful publication latency. Its call counts are scripted command

equivalents, not model-observed tool calls. Local Oak CI/review are unconfigured

and stay explicit skips rather than synthetic passes. Use

`parallel_contention.py --modes workspace_per_task` for concurrent isolated

tasks instead of duplicating its worker implementation here.

The materialization cell is Oak-before versus Oak-after only. Its setup probe

must first observe the exact legacy two-pull/two-pass route; one fixed Serve

and one repo are then shared by all subjects. Receipt phases, proxy byte counts,

HEAD, history, file bytes, and modes are correctness gates. Establish the A/A

noise floor before interpreting A/B timing.

Mount and acquisition probes:

```bash

. config/bench-env.sh  # remote-backed lanes; safe no-secret defaults

python3 scripts/mount_probe.py

OAK_BENCH_MIRROR_REPO=oak/bench-large-mirror \

  [email protected]:oakdotspace/bench-large-mirror.git \

  python3 scripts/mount_vs_clone.py --reps 5

```

Platform capability or fake-provider branch triage:

```bash

python3 scripts/platform_lifecycle.py --scenario platform_capability_probe

python3 scripts/platform_lifecycle.py \

  --platform github \

  --driver fake-provider \

  --scenario branch_triage_n4

```

Use `--track agent-default` for current CLI/agent UX and

`--track core-equivalent` when making VCS-mechanics claims.

## Results

Each run writes raw JSONL and, when available, a Markdown summary. Core rows use:

- `results/<timestamp>.jsonl`

- `results/<timestamp>.summary.md`

- `results/latest.jsonl`

- `results/latest.summary.md`

Other lanes may write suffixed latest files such as `latest.mount.jsonl` or

`latest.platform.jsonl`, and long campaigns can flush

`<timestamp>.<lane>.partial.jsonl` as crash insurance before final validation.

Never commit raw real-agent transcripts, generated result JSONL, temporary

workspaces, or large benchmark artifacts. Archive publishable raw rows outside

the repo and link them from reports.

## Subject Configuration

By default, `git` and `oak_installed` resolve from `PATH`. Optional local-build

subjects are disabled until explicitly requested. Edit `config/subjects.toml`

or pass overrides:

```bash

python3 scripts/bench.py \

  --subjects git,oak_installed,oak_local \

  --oak-local-bin ../oak/target/release/oak \

  --oak-repo ../oak

```

To benchmark exact `main` against local changes, build or place both binaries at

stable paths and enable `oak_main` and `oak_local`, or pass the paths with CLI

flags. The harness does not mutate the Oak source checkout.

## Architecture

Shared measurement policy lives in `scripts/oakbench/`: subject identity,

environment control, timed execution, token accounting and cost weights,

remotes, run locks, row contracts, command semantics, stream adapters,

fixture/config IO, runner identity, and reporting math.

Lane scripts stay thin over that core:

- `bench.py`

- `workflow_ab.py`

- `agent_workflow.py`

- `parallel_contention.py`

- `mount_probe.py`

- `mount_vs_clone.py`

- `platform_lifecycle.py`

- `netshape_bench.py`

- `task_loop.py`

Domain vocabulary is in `CONTEXT.md`; load-bearing decisions are ADRs under

`docs/adr/`.

## Testing The Instruments

A benchmark that measures with untested instruments cannot make accuracy

claims. The suite ships tests for row contracts, adapter streams, token

accounting, command semantics, skip-row coherence, fixture registries,

calibration math, oracles, and reporting:

```bash

python3 -m unittest discover -s tests

```

Run the test suite before pushing harness changes. For measurement-policy

refactors, use `scripts/row_parity.py` to prove the new rows are

measurement-identical before trusting the refactor.

```bash

python3 scripts/row_parity.py old/latest.jsonl new/latest.jsonl

```

## Cloud And CI

Cloudflare is the control and publishing plane, not the latency measurement

plane. Workers, D1, and R2 receive and index raw rows; dedicated runners execute

benchmarks on pinned hardware.

See `docs/cloud.md` and `docs/infrastructure-plan.md` for the recommended

setup: per-push smoke, nightly standard, dedicated large-file/monorepo runners,

Linux shaped-network runners, macOS mount/agent runners, archived raw rows, and

a small dashboard for Oak-vs-Git and Oak-vs-previous-Oak deltas.