Benchmark Coverage

# Benchmark Coverage

This page separates implemented evidence from planned coverage. Public reports

should cite only implemented rows and should link to the raw JSONL used for the

claim.

| Area | Status | Evidence | Public Claim Allowed |

| --- | --- | --- | --- |

| Core smoke VCS operations | Implemented | `scripts/bench.py --profile smoke` | Local smoke/regression signal only. |

| Core standard/large profiles | Implemented in harness, needs dedicated runs | `scripts/bench.py --profile standard|large` | Cite only when run on pinned hardware with enough repetitions. Large binary scenarios include `single_huge_binary_1gb` and `many_large_binaries_multigb`; `git_lfs` only applies to the named binary scenarios in `config/git_modes.json` and emits skip rows when unavailable. |

| XL/monorepo fixture family | Registered generator, not consumed by active `bench.py` profiles | `scripts/make_monorepo_fixture.py`, `config/fixtures.json`, `tests/test_make_monorepo_fixture.py` | No monorepo performance claim yet. Registry entries are `status: planned` with null manifests; missing/unverified generated trees are coverage gaps, not evidence. |

| Scripted agent-shaped workflows | Implemented | `scripts/workflow_ab.py` | Deterministic workflow cost, not real agent behavior. |

| History archaeology + error recovery workflows | Implemented | `scripts/workflow_ab.py --workflows history_archaeology,vcs_error_recovery` | Deterministic; note closest-available-command substitutions for Oak. |

| Sync/divergence recovery workflows | Implemented | `scripts/workflow_ab.py --workflows sync_push_divergence,sync_pull_upstream,sync_pull_dirty,sync_non_ff_after_amend` | Git uses a local bare remote; oak needs `OAK_BENCH_SYNC_REPO` (skip rows otherwise). Compare tokens/calls/recovery steps only — transports differ by construction. Failure rows carry `error_mentions_recovery_command`. |

| Real agent adapters | claude + codex verified against real runs (2026-06-10), re-verified plus cursor verified (2026-06-11: success, provider tokens, turn metrics on all three) | `scripts/agent_workflow.py`, `tests/fixtures/*_real.jsonl` | Cite all three adapters' rows with raw JSONL. |

| Turn metrics + recovery cost | Implemented for stream-json adapters | `agent_workflow.py` turn timeline, `turn_metrics`, `turns_to_recovery` | Cite per adapter; mock rows are unmeasured (null). |

| Instruction-level familiarity lane | Implemented, needs repeated paid runs | `scripts/agent_workflow.py --instruction-level` | Cite only with per-level success rates and n>=10 per level. |

| Parallel contention lane | Implemented for git modes; Oak per-task mode skip-rows | `scripts/parallel_contention.py` | Local signal; Oak workspace-per-task claims blocked until a disposable remote is wired. |

| End-to-end task cost | Partial, runnable local evidence | `scripts/task_loop.py --scenarios agent_task_baseline_v1,agent_task_oak_local_v1,interrupted_publication_after_request_v1` | Git local-file cold clone/warm worktree, successful Oak two-fresh-checkout work against owned loopback `oak serve`, and actual Oak loopback interruption evidence. Scripted call equivalents are not model-observed calls. Local Oak CI/review and transport-matched Oak-vs-Git wall time remain explicitly unmeasured. |

| Output determinism / ANSI probes | Implemented | `bench.py` `*.determinism` rows | Diagnostic rows; cite stability percentage with scenario shape. |

| Oak mount remote-backed benchmarks | Dry probe/partial | `scripts/mount_probe.py`, `scenarios/mount.yaml` | Capability/skip reporting only unless a disposable remote is configured. |

| Mount-vs-clone acquisition comparison | Git side measured (2026-06-11, n=3: full 28.1s/670MB, shallow 21.1s/574MB, blobless 24.2s/583MB, sparse-task 7.1s/66MB/3 calls). Oak large-mirror mount startup is no longer a timeout after the M7 server metadata repair + root-only lazy startup: launch proof has a successful default-root large-mirror smoke at 3889.945ms mount / 164.322ms root readdir / 1473.293ms stat-all, plus a cold custom-root handoff proxy A/B showing 33107.974ms main vs 6038.319ms candidate (-81.762%, 6/6 wins). | `scripts/mount_vs_clone.py` (`OAK_BENCH_MIRROR_REPO`, `GIT_BENCH_MIRROR_URL` — SSH URL, mirror is private); M7 launch proof artifacts under `/Users/mrmrs/o/launch-proof-20260618/raw/` | Oak mount startup claims are allowed only with the M7 caveats: the default-root large-mirror smoke is a successful mount proof; the custom-root A/B is a handoff/failure proxy, not a successful mount; cold unhydrated file attrs may report size 0. Do not publish a precise "~1s" large-repo mount claim from this evidence. Git-only acquisition costs remain citable with the different-networks caveat. |

| Tuned Git baselines | Wired for untracked-cache, split-index, fsmonitor, and Git LFS on binary fixtures | `scripts/bench.py --git-modes ...` | Required for status/diff claims; `git_lfs` emits explicit skip rows on non-binary scenarios or hosts without Git LFS. Sparse and worktree remain declarative-only modes. |

| Token estimate calibration | Implemented as standalone tool | `scripts/token_calibration.py` | Report the calibration factor with any cross-subject token delta. |

| Instrument test suite | Implemented | `tests/`, golden adapter-stream fixtures, `python3 -m unittest discover -s tests` | Cite as methodology evidence: the measurement code itself is regression-tested. |

| Per-lane row contract | Implemented, enforced at write time | `scripts/oakbench/rows.py`, `scripts/oakbench/results.py` | Contract violations fail the producing run, not the dashboard. |

| Dev-loop verdict with measured noise floor | Implemented | `scripts/devloop.py` (A/A null test + embedded null controls) | Verdicts state their own detection limits; deltas below the measured floor are never claimed. |

| Measurement-identity parity tool | Implemented | `scripts/row_parity.py` | Required evidence for harness refactors (ADR-0005). |

| Statistical confidence intervals | Implemented as standalone tool | `scripts/benchmark_stats.py` | Use for publishable reports with adequate sample counts. |

| Efficiency regression gates | Implemented (exact metrics, Oak-baseline comparison) | `scripts/regression_report.py` | CI gate for oak_local vs released Oak. |

| Cloud dashboard | Scaffolded | `cloudflare/`, `docs/results-and-dashboard.md` | Ingest architecture only until live historical data exists. |

| Peak memory (RSS) per VCS command | Implemented (core lane) | `bench.py` `peak_rss_bytes` via `oakbench/execution.py` (os.wait4 rusage) | Per-command child peak RSS; null on platforms without wait4. |

| Tool-call envelope token cost | Implemented (additive fields) | `oakbench/tokens.py` `*_with_envelope` fields; calibrate with `token_calibration.py --envelope` | Cite with the envelope constants and the calibration output; historical fields exclude envelopes. |

| Tail latency p95/p99 with sample honesty | Implemented | `oakbench/reporting.py` `tail_latency_summary`, `benchmark_stats.py` p95/p99 columns | Percentiles are null below n=20 (p95) / n=100 (p99); never cite a relabelled max as a tail. |

| Output information recall + bytes-per-file | Implemented and gated | `bench.py` `*.inforecall` rows, `oakbench/output_semantics.py`; gated in `regression_report.py` (`information-recall`) and `devloop.py` (`RECALL`) | Bytes-saved claims must cite recall alongside: compact-but-lossy is not a win, and the gates now fail it. |

| Piped-diff format compatibility | Implemented and gated | `bench.py` `diff.full.inforecall` rows (`pipe_compatible_unified`, hunk counts); gated in `regression_report.py` (`pipe-compat`) and `devloop.py` (`PIPE-COMPAT`) | Cite when claiming oak diff works with patch/grep-style tooling; a changeset that loses unified structure fails the verdict. |

| Remote cold clone / push / pull | Implemented for git (local file remote); Oak skip-rows | `bench.py` `remote.*` rows, `--skip-remote` | Same-machine remote isolates VCS transfer cost from network; Oak claims blocked until a disposable remote is wired. |

| Shaped network operations and first-payload timing | Implemented for netshape lane where Linux netns/tc is available | `scripts/netshape_bench.py` `netshape_clone_cold`, `netshape_log_follow_cold`, byteproxy `ttfd_ms` fields | Cross-subject latency claims require the same `netshape_profile` and server host. `netshape_log_follow_cold` measures cold network acquire plus file-history query (`git.log.follow`/`oak.log.follow`) with TTFD fields. Non-Linux or missing Oak local server paths emit skip rows. |

| Mount lifecycle (desc, finish, space clean, task loop) | Implemented handlers; needs remote to run | `mount_probe.py` `oak.desc`/`oak.finish --desc-file`/`space.clean`/`loop.push_desc*`, `task_lifecycle_loop` scenario | `oak finish` capability-gated per binary; skip rows until OAK_BENCH_MOUNT_REPO set. |

| Compact-output follow-up cost | Implemented for streams exposing per-call outcomes | `tool_call_metrics.vcs_info_followup_calls_total` via `oakbench/stream_adapters.py` | Each follow-up means the first output was insufficient; null when commands are not adapter-visible. |

| Contention commit-latency tail + retry depth | Implemented | `parallel_contention.py` `commit_latency_tail`, `retry_depth_histogram`, per-worker samples | Cite p95 only when `tail_supported` includes it. |

Known unmeasured-by-design (null in v2 rows until instrumented): hydration

bytes outside the mount lane, tool wait vs model time inside agent CLIs, and

Oak conflict counts in real-agent rows.

## Git `t/perf` Clean-Room Cross-Reference

This table maps upstream Git `t/perf` behavioral areas to this repository's

coverage. It is a clean-room index: the mapping uses Git perf-test names and

titles only, not GPL test bodies, shell helpers, fixtures, or expected output.

External prior-art policy is in `docs/prior-art.md`.

| Git `t/perf` behavioral area | Representative Git perf tests | Current Oak benchmark coverage | Phase 0 status |

| --- | --- | --- | --- |

| Perf harness sanity and non-VCS helper cost | `perf-lib.sh`, `p0000-perf-lib-sanity.sh`, `p0071-sort.sh` | Instrument tests cover row contracts, adapters, calibration, run locks, and reporting helpers. | Excluded from VCS coverage: Git's perf harness internals and generic `sort(1)` cost are not Oak-vs-Git behavior. |

| Revision walking, graph queries, and history lookup | `p0001-rev-list.sh`, `p1500-graph-walks.sh`, `p1501-rev-parse-oneline.sh`, `p6010-merge-base.sh`, `p6011-rev-list-maximal.sh`, `p6100-describe.sh` | Core spec has `long-history` with history operations; workflow lane has history archaeology. | Partial/planned. Merge-base, describe, maximal-independent sets, and regex commit lookup are explicit gaps. |

| Index, checkout, status, and working-tree scans | `p0002-read-cache.sh`, `p0004-lazy-init-name-hash.sh`, `p0005-status.sh`, `p0006-read-tree-checkout.sh`, `p0007-write-cache.sh`, `p0090-cache-tree.sh`, `p5314-name-hash.sh` | Core lane measures `status.clean`, `status.dirty`, `diff.dirty`, snapshots, tuned Git modes, peak RSS, output determinism, and information recall. | Implemented for user-visible status/diff/snapshot pressure; Git internal index/cache-tree/name-hash microbenchmarks are excluded unless exposed as user-visible regressions. |

| Object database durability and fsync policy | `p0008-odb-fsync.sh` | No dedicated durability lane. Snapshot rows record latency and output, not persistence-policy variants. | Explicit gap. Needs a durability semantics contract before public comparisons. |

| Refs, branch/tag enumeration, and ref globbing | `p0100-globbing.sh`, `p1400-update-ref.sh`, `p6300-for-each-ref.sh` | Core spec includes branch create/list/switch and many-branches; contention lane measures branch-per-task behavior. | Partial/planned. Raw ref update/globbing and tag formatting internals are gaps. |

| Object inspection and repository verification | `p1006-cat-file.sh`, `p1450-fsck.sh`, `p1451-fsck-skip-list.sh` | Integrity attestation tests validate the harness; benchmark lanes do not time object inspection or fsck equivalents. | Excluded for Phase 0 unless Oak exposes comparable user-facing object verification commands. |

| Sparse checkout, partial clone, clone reference, Scalar, and acquisition | `p2000-sparse-operations.sh`, `p5600-partial-clone.sh`, `p5601-clone-reference.sh`, `p9210-scalar.sh` | Mount lane, mount-vs-clone comparison, tuned Git sparse-checkout mode, and remote clone/pull rows cover acquisition and lazy-use behavior. | Partial. Git's full sparse-operation matrix, Scalar setup, and clone-reference optimization remain gaps. |

| Rebase, stash, reset, clean, and history rewriting | `p3400-rebase.sh`, `p3404-rebase-interactive.sh`, `p3904-stash-patch.sh`, `p7000-filter-branch.sh`, `p7102-reset.sh`, `p7300-clean.sh` | Workflow lane covers scripted recovery, sync divergence, cleanup, and branch lifecycle; mount lane covers teardown and `space.clean`. | Partial/excluded. Reset/cleanup behavior is agent-relevant; rebase, interactive rebase, stash patch UI, and filter-branch are not Phase 0 goals. |

| Diff algorithms, log formats, pickaxe, line-log, grep, and last-modified queries | `p4000-diff-algorithms.sh`, `p4001-diff-no-index.sh`, `p4002-diff-color-moved.sh`, `p4205-log-pretty-formats.sh`, `p4209-pickaxe.sh`, `p4211-line-log.sh`, `p4220-log-grep-engines.sh`, `p7810-grep.sh`, `p7820-grep-engines.sh`, `p8020-last-modified.sh` | Core lane measures dirty diff output, full-diff pipe compatibility, information recall, and output bytes; agent lane has file exploration scenarios. | Partial. Agent-facing diff and exploration cost is covered; Git-specific algorithm and grep-engine matrices are gaps/exclusions. |

| Packfiles, bitmaps, multi-pack indexes, pruning, fetch-pack, import/export | `lib-bitmap.sh`, `lib-pack.sh`, `p5302-pack-index.sh`, `p5303-many-packs.sh`, `p5304-prune.sh`, `p5310-pack-bitmaps.sh`, `p5311-pack-bitmaps-fetch.sh`, `p5312-pack-bitmaps-revs.sh`, `p5313-pack-objects.sh`, `p5315-pack-objects-filter.sh`, `p5326-multi-pack-bitmaps.sh`, `p5332-multi-pack-reuse.sh`, `p5333-pseudo-merge-bitmaps.sh`, `p5550-fetch-tags.sh`, `p5551-fetch-rescan.sh`, `p9300-fast-import-export.sh` | Core remote rows cover same-machine push/clone/pull; mount-vs-clone records acquisition bytes and wall time for Git clone variants. | Explicit gap/exclusion. Add only if public claims require storage-transfer mechanics beyond current remote/acquisition rows. |

| Fsmonitor, untracked cache, split index, and watcher-backed status | `p7519-fsmonitor.sh`, `p7527-builtin-fsmonitor.sh` | Tuned Git baselines are wired for `untracked_cache`, `split_index`, and `fsmonitor`. | Implemented as comparison modes where platform support exists; builtin fsmonitor lifecycle details remain a Git-specific exclusion. |

## Public Claim Rules

- Say whether a result is `agent-default` or `core-equivalent`.

- Say whether a workflow is scripted or produced by a real agent adapter.

- Say the exact subject roles: Git baseline, Oak release installed, Oak local

  changeset, optional Oak main.

- Include raw output bytes and truncation count for token-related claims.

- Include hardware, OS, filesystem, repetitions, randomization, and confidence

  intervals for public performance claims.

- Never cite planned scenario YAML as evidence.

## Suggested Badges

- `implemented`: runnable and verified locally

- `partial`: runnable but missing a fairness mode, tuned baseline, or oracle

- `dry-probe`: records capability/skip rows only

- `planned`: design exists but no benchmark evidence yet