Commit dc75e12c · oak/benchmarks

benchmarksPublic

· Browse files at this commit

dc75e12cb299

Merged on mrmrs-0a9c5c

mrmrs · 1 month ago · 4 files ~4 modified

M README.md

@@ -117,6 +117,7 @@

117117

| Scripted workflows | `scripts/workflow_ab.py` | Implemented for deterministic bugfix, wide refactor, large asset, history archaeology, error recovery, and sync/divergence recovery workflows. |

118118

| Real agents | `scripts/agent_workflow.py` | Implemented for mock plus local Codex, Claude, and Cursor Agent stream adapters. Use real rows only when the installed CLI stream has campaign evidence. |

119119

| Parallel contention | `scripts/parallel_contention.py` | Implemented for Git shared/workspace modes; Oak workspace-per-task rows require a disposable Oak remote and otherwise skip honestly. |

120+

| End-to-end task loop | `scripts/task_loop.py` | Git local-file cold-clone/warm-worktree rows and actual Oak loopback interrupted-publication evidence are runnable without production writes. Hosted Oak/GitHub publication and exact-CI phases remain unmeasured without disposable provider fixtures. |

120121

| Mount/lazy hydration | `scripts/mount_probe.py`, `scripts/mount_vs_clone.py` | Dry probe and partial remote-backed coverage. Real mount timing needs `OAK_BENCH_MOUNT_REPO`; large-mirror Oak-vs-Git acquisition is not publishable yet. |

121122

| Platform lifecycle | `scripts/platform_lifecycle.py` | Implemented for capability probes, hosted integration anatomy, poll-until-merged, race scenarios, and fake-provider branch triage. Real hosted rows need credentials and disposable repos. Excluded from devloop. |

122123

| Netshape | `scripts/netshape_bench.py` | Implemented where Linux network namespaces and `tc` are available. Cross-subject latency claims require the same shaped pipe and server identity. |

@@ -144,9 +145,10 @@

144145 - Workspace-per-agent fleets: Git worktree contention is implemented. Oak

145146 workspace-per-task needs real disposable remotes, high-concurrency mount/push

146147 behavior, branch cleanup, and merge/integration plumbing.

147-- Full Oak task loop: the roadmap is mount -> edit -> commit -> push -> desc

148- -> finish/clean -> remount follow-up, compared with Git's clone/worktree,

149- branch, commit, push, cleanup, and follow-up flow.

148+- Full hosted task loop: local Git cold-clone/warm-worktree and Oak

149+ interrupted-publication cells are implemented. A transport-matched Oak/GitHub

150+ acquisition -> publish -> exact-CI -> review comparison still needs disposable

151+ provider fixtures; missing provider phases stay explicit skip rows.

150152 - Hosted workflow evidence: platform rows need GitHub org assets, PATs or app

151153 credentials, webhooks, disposable repos, and real Oak/GitHub provider wiring

152154 before public hosted-branch claims.

@@ -307,6 +309,29 @@

307309 --workers 2,8,32 --commits-per-worker 5

308310 ```

309311

312+End-to-end task cost without production writes:

313+

314+```bash

315+# Fresh full clone followed by a shared-object Git worktree. Hosted CI remains

316+# an explicit unmeasured phase on the local-file transport.

317+python3 scripts/task_loop.py --subjects git --git-remote local \

318+ --scenarios agent_task_baseline_v1 --runs 10

319+

320+# Actual Oak HTTP publication attempt whose loopback fixture reads one request

321+# then drops the reply. Use pinned released/current binaries; no live remote is used.

322+python3 scripts/task_loop.py --subjects oak_installed,oak_local \

323+ --oak-installed-bin /path/to/released-oak \

324+ --oak-local-bin /path/to/current-oak \

325+ --scenarios interrupted_publication_after_request_v1 \

326+ --runs 10 --randomize-subject-order

327+```

328+

329+These cells are different evidence classes. Local Git wall time is not a

330+network comparison, and the interruption fixture measures preservation and

331+outcome knowability rather than successful publication latency. Use

332+`parallel_contention.py --modes workspace_per_task` for concurrent isolated

333+tasks instead of duplicating its worker implementation here.

334+

310335 Mount and acquisition probes:

311336

312337 ```bash