Plan: Accuracy & Insight Push (June 2026)

# Plan: Accuracy & Insight Push (June 2026)

Six tasks to take the suite from "instrumented" to "publishable evidence" for

oak's core pitch. Ordering is driven by one shared dependency: **remote

plumbing**. Tasks 1, 3, and 5 all need the harness to stand up disposable

remotes (oak server-side, git local-bare or GitHub). Build that once in

`oakbench`, then the lanes are thin.

## Repos required (blocking — request before Phase 1)

| Repo | Where | Purpose | Lifecycle |

| --- | --- | --- | --- |

| `oak/bench-fleet-tmp` | oak org | Branch storms from the agent-fleet lane (N workers × M commits × runs). Kept separate from `oak/oak-benchmarks-tmp` so fleet garbage never pollutes the mount lane's timing repo. | Disposable; deletable/recreatable any time |

| `oak/bench-sync-tmp` | oak org | Push-divergence / pull-with-upstream-changes scenarios (task 5) and the full task-loop pushes (task 3). | Disposable |

| `oak/bench-large-mirror` | oak org | Large-repo fixture for mount-vs-clone (task 2). Seeded once from a generated fixture; read-mostly afterward. | Long-lived, read-only after seeding |

| `bench-large-mirror` | GitHub | Byte-identical mirror of the above, the git side of mount-vs-clone. | Long-lived, read-only after seeding |

| `bench-task-loop-tmp` | GitHub | Optional but recommended: git's agent-default push target for the task-loop lane, so git pays real `git push` like oak pays real `oak push`. Local-bare remains the core-equivalent track. | Disposable |

Large-mirror fixture spec (generated by a new `scripts/make_large_fixture.py`,

deterministic seed so both mirrors are identical): ~30k files across a

realistic tree (src/, docs/, vendored deps), ~300 MB working tree including a

handful of 20–100 MB binaries, ~500 commits of synthetic history. Big enough

that full `git clone` visibly pays for bytes an agent never reads; small

enough to re-seed in minutes.

## Phase 0 — shared remote plumbing (`oakbench/remotes.py`)

Lift what already exists in `bench.py` (OAK_BENCH_REMOTE resolution at

`bench.py:400`, disposable branch namespacing at `bench.py:431-449`,

local-bare git remote setup at `bench.py:566-576`) into a shared module:

- `resolve_oak_remote(purpose)` → owner/repo from env

  (`OAK_BENCH_FLEET_REPO`, `OAK_BENCH_SYNC_REPO`, falling back to

  `OAK_BENCH_REMOTE`), or a structured skip reason. Skips stay returncode-77

  rows naming the env var (skips are work items).

- `make_git_bare_remote(tmpdir)` — untimed local bare repo (core-equivalent

  track) and `resolve_git_github_remote()` from `GIT_BENCH_REMOTE` env

  (agent-default track).

- `disposable_branch(bench_id, scenario, subject, run)` — the existing

  collision-safe namescheme, shared by all lanes.

- Transport honesty: every row touching a remote records

  `transport: local_file | network` and the server identity. Network rows are

  never aggregated with local-file rows (AGENTS.md already mandates this for

  `remote.net.*`; the new lanes inherit the rule).

Tests: `tests/test_remotes.py` (resolution, skip reasons, branch-name

uniqueness). Run `row_parity.py` against a pre-change `bench.py` run to prove

the lift is measurement-identical (ADR-0005 discipline).

## Task 1 — agent-fleet lane: oak workspace-per-task

**Today:** `parallel_contention.py:425-439` hard-skips non-git subjects in

`workspace_per_task` mode. Git mode already emits `setup_ms`, `merge_ms`,

`marginal_workspace_disk_kb`, commit throughput, lost-update detection, and

integrity (`parallel_contention.py:535-554`).

**Build:**

1. Untimed setup: push the contention fixture to `oak/bench-fleet-tmp` on a

   disposable branch (one per run).

2. Per worker *i*: timed `oak mount`/space creation of workspace *i* →

   record per-workspace `setup_ms` **and** a new run-level

   `time_to_nth_workspace_ms` curve (cumulative wall time until N workspaces

   are usable — this is the fleet-spinup headline number). Git comparator

   gets the same curve computed from its existing worktree setup timings, so

   the metric lands on both subjects simultaneously (ADR-0005: new metric,

   additive, no renames).

3. Marginal disk: reuse the mount lane's `disk.usage` probe

   (`mount_probe.py:439-511` — visible vs allocated bytes, entry count) so

   lazy hydration's near-zero marginal disk is measured the same way in both

   lanes. Git worktrees keep `workspace_disk_kb` as-is.

4. Commit storm: same disjoint-file edit/commit loop; oak workers commit and

   push their workspace branch.

5. Merge throughput: merge N branches back (git: existing merge step; oak:

   its merge/integrate flow against the remote), emit `merge_ms`,

   `merges_clean`, `merge_conflicts` for both.

6. Integrity pass: payload survival in the merged result + oak's closest

   fsck analog, `measurement_source` named accordingly.

**Honesty constraint:** git worktrees are local; oak workspaces hit the

network. Rows carry `transport`; the summary presents the comparison with the

asymmetry stated, and optionally adds a `clone_per_task` git mode (clone from

local bare per worker) as the transport-comparable third column. Do not bury

this — it is the first question a skeptical reader asks.

**Flags/env:** `OAK_BENCH_FLEET_REPO=oak/bench-fleet-tmp`, existing

`--workers 2,8,32 --modes workspace_per_task`.

**Deliverable claim shape:** "N agents × isolated workspaces:

time-to-32nd-workspace X s vs Y s; marginal disk per workspace X MB vs Y MB;

merge throughput X/s vs Y/s (oak network-backed, git local — see methodology)."

## Task 2 — mount-vs-clone with a git comparator

**Today:** mount lane is oak-only (`scenarios/mount.yaml:45-174`, no git

scenarios); `docs/benchmark-coverage.md` lists no mount-vs-clone plan.

**Build:**

1. Seed `oak/bench-large-mirror` and GitHub `bench-large-mirror` from the

   same generated fixture (Phase 0 fixture script). Verify byte-identity with

   a manifest hash committed to both.

2. New lane scenarios — new names, not edits to existing mount ops

   (ADR-0005): `acquire.cold` with subjects:

   - `oak mount oak/bench-large-mirror` → first targeted file read

   - `git clone <github-url>` (stock) → same file read

   - `git clone --filter=blob:none` (partial clone, tuned-git credibility)

   - `git clone --depth 1` (shallow)

   - optionally `git clone` + `sparse-checkout set <task-path>`

3. Metrics per subject: `time_to_first_read_ms` (command start → target file

   bytes in hand), `bytes_transferred` (oak: existing `bytes_hydrated`

   parsing at `mount_probe.py:327-347`; git: pack bytes from clone stderr

   `Receiving objects` / `.git` size delta), `disk_total_bytes`,

   `time_to_task_ready_ms` (tree listable + task path readable).

4. Follow-up scenario `acquire.task_scoped`: agent needs 3 files in one

   subdirectory — oak hydrates ~those bytes; git pays the whole clone (or the

   partial-clone fetch chatter). This is the marketing picture.

5. Network honesty: GitHub's CDN vs oak.space are different networks.

   Mitigate: interleave reps, record server + timestamps, report medians of

   ≥5 reps, and state the caveat in the summary. The claim is about the

   *acquisition model* (lazy vs eager), which dominates network variance at

   this repo size — but say so rather than assume it.

**Deliverable claim shape:** "Start working on a 300 MB repo: oak mount

X s / Y MB hydrated vs git clone X s / 300 MB (and partial clone X s) —

time-to-first-read and bytes-to-first-read, 5 reps, medians."

## Task 3 — full oak task-loop end-to-end

**Today:** `workflow_ab.py` workflows are single-checkout, no remote, no

mount; the signature motion (mount → edit → commit → push → desc → end →

remount for follow-up) is only exercised piecemeal by the mount lane's

`task_lifecycle_loop`.

**Build:** new workflow-ab-style lane (likely its own script,

`scripts/task_loop.py`, sharing `oakbench` policy) with one scenario per

subject family, agent-default track:

- **oak native:** `oak mount` (from `oak/bench-sync-tmp`) → edit → `oak

  commit` → `oak push` → `oak desc` → `oak end` → remount for a follow-up

  task → second edit/commit/push.

- **git native:** `git clone` (GitHub `bench-task-loop-tmp`) or `git worktree

  add` from an existing checkout → branch → edit → commit → push → PR-less

  desc analog (commit message / `git push` output) → `git worktree remove` →

  new worktree for follow-up.

- Core-equivalent variant runs git against a local bare remote so the

  mechanics claim survives without GitHub network in the loop.

**Metrics:** per-phase operation rows plus a `task_loop.total` row: wall

time, tool-call count (envelope-priced — this lane is where oak's

fewer-calls-per-motion compounds), estimated cost-weighted tokens, and

`followup_warm_delta_ms` (second task on warm caches vs first — the remount

story). Oracle: payload present on the remote branch, workspace cleanly

ended, follow-up edit visible.

**Why a separate lane:** the command sequences are not semantically

equivalent step-by-step (desc/end have no git twin), so this is explicitly

agent-default-track, whole-task-cost evidence — exactly the framing

`docs/command-semantics.md` exists to police.

## Task 4 — real-agent runs

**Today:** adapters exist (`agent_workflow.py:570-642`), token extraction

handles all provider field variants (`agent_workflow.py:737-786`), but only

`mock` has ever run; stream formats for current CLI versions are unverified.

**Build (no new repos needed):**

1. **Adapter shakeout:** one manual run per installed CLI

   (`--agents available`), single rep, `bugfix_test_loop`, both subjects.

   Verify: stream parses, `usage` events found (provider-reported tokens

   non-null), turn_metrics populated, oracle passes. Fix adapter drift

   against current CLI flags before burning tokens on reps.

2. **Token-extraction goldens:** capture one real transcript per CLI into

   `tests/` golden fixtures (scrubbed) so adapter drift is caught by

   `test_stream_adapters.py` instead of mid-campaign.

3. **The campaign:** `--agents claude[,codex,cursor_agent] --subjects

   git,oak_installed --workflows bugfix_test_loop --runs 5

   --agent-environment minimal --randomize-agent-order

   --randomize-subject-order --instruction-level zero-shot` then the same at

   `cheat-sheet`. Never aggregate across levels (existing contract). Budget

   estimate before launch: runs × agents × subjects × levels ≈ 40 tasks; cap

   with `--timeout-seconds 900` and a per-campaign dollar estimate printed

   up front.

4. **The story metrics:** zero-shot vs cheat-sheet deltas in success rate,

   `help_calls_total`, `unknown_command_failures_total`,

   `turns_to_recovery` / `tokens_to_recovery` — the "git is in pretraining,

   oak isn't, and AGENTS.md closes the gap" measurement.

5. Stretch (post-campaign): run task 5's divergence workflow and task 3's

   loop under real agents — recovery-cost numbers from real transcripts are

   the credibility ceiling for the whole drop.

Transcripts/JSONL stay out of the repo (data-hygiene rules); keep run

artifacts under an external/ignored dir with the run manifest.

## Task 5 — sync & divergence recovery

**Today:** `vcs_error_recovery` covers only empty-commit and

unknown-subcommand (`workflow_ab.py:413-433`). No remote plumbing exists in

workflow_ab. ADR-0005 forbids changing `vcs_error_recovery`'s meaning — these

are **new workflows**, not extensions.

**Build:** new workflow `vcs_sync_recovery` (uses Phase 0 remotes; git →

local bare for core-equivalent + optional GitHub for agent-default; oak →

`oak/bench-sync-tmp`):

- `sync.push_divergence`: checkout A and B from the same remote; A edits +

  pushes; B edits + pushes → rejected. Then the recovery arc: whatever the

  error output tells you to do (git: pull/rebase or

  `--force-with-lease` decision; oak: its divergence flow — the exact trap

  hit three times in real use on 2026-06-11). Steps carry

  `expected_returncodes` admitting the failure; recovery steps follow.

- `sync.pull_upstream_changes`: clean pull with upstream commits, and dirty

  pull (local uncommitted edits) variant.

- `sync.non_ff_after_amend`: amend/rewrite local tip after pushing, push

  again — the second-most-common agent trap.

**Metrics (this is the "error output is a prompt" lane):** per-failure

`error_output_tokens` (the rejected push's stderr, cost-weighted as model

input), `recovery_steps_count`, `recovery_tokens_total`, and an

`error_actionability` check: does the error text literally contain the

command that fixes it? (boolean per subject, `measurement_source:

stderr_contains_recovery_command`). Real-agent `turns_to_recovery` already

exists for task 4 to consume on these same workflows.

## Task 6 — pre-publish calibration & tuned-git (gate, runs last)

1. **Calibration:** capture real per-style outputs from a standard run

   (status, diff, log, push-rejection stderr — per subject), run

   `token_calibration.py` with `tiktoken` available (optional import, allowed

   under ADR-0001) per style group, plus `--envelope` to validate the

   envelope constants. Output: calibration factors table committed to

   `docs/` and cited next to every published token delta; if any factor moves

   a published cross-subject delta's sign or order, the table uses calibrated

   numbers, not char/4.

2. **Tuned-git everywhere public:** every public table includes

   `git_untracked_cache` AND `git_fsmonitor` rows (verify

   `fsmonitor--daemon` health on the darwin runner; it is the load-bearing

   comparator for wide-tree claims). Wire `--git-modes

   untracked_cache,fsmonitor` into the canonical publish-run invocation.

3. **Publish checklist:** small `docs/publish-checklist.md` (or

   `scripts/publish_gate.py` that greps result sets): calibration run ✓,

   tuned modes present ✓, no aggregation across instruction levels or

   transports ✓, nulls rendered as unmeasured ✓, every skip row accounted

   for ✓, tail-percentile sample-count honesty ✓.

## Sequencing

```

Phase 0  oakbench/remotes.py + fixture generator + repo creation   ← blocks 1,2,3,5

Phase 1  Task 1 (fleet lane)        Task 5 (sync recovery)         [parallel]

Phase 2  Task 2 (mount-vs-clone)    Task 3 (task loop)             [parallel]

Phase 3  Task 4 (real agents: shakeout → goldens → campaign,

                 then rerun tasks 3/5 workflows under real agents)

Phase 4  Task 6 (calibration + tuned-git + publish gate)

```

Each phase lands with instrument tests (`python3 -m unittest discover -s

tests`) and, where existing lanes are touched, a `row_parity.py` proof.