Log in
benchmarksPublic

Commit 25881039

· Browse files at this commit

25881039a2bb

Merged on mrmrs-0a9c5c
mrmrs · 1 month ago · 6 files ~6 modified
M README.md
@@ -117,7 +117,7 @@
117117 | Scripted workflows | `scripts/workflow_ab.py` | Implemented for deterministic bugfix, wide refactor, large asset, history archaeology, error recovery, and sync/divergence recovery workflows. |
118118 | Real agents | `scripts/agent_workflow.py` | Implemented for mock plus local Codex, Claude, and Cursor Agent stream adapters. Use real rows only when the installed CLI stream has campaign evidence. |
119119 | Parallel contention | `scripts/parallel_contention.py` | Implemented for Git shared/workspace modes; Oak workspace-per-task rows require a disposable Oak remote and otherwise skip honestly. |
120-| End-to-end task loop | `scripts/task_loop.py` | Git local-file cold-clone/warm-worktree rows and actual Oak loopback interrupted-publication evidence are runnable without production writes. Hosted Oak/GitHub publication and exact-CI phases remain unmeasured without disposable provider fixtures. |
120+| End-to-end task loop | `scripts/task_loop.py` | Git local-file cold-clone/warm-worktree rows, successful Oak two-fresh-checkout work against owned loopback `oak serve`, and actual Oak loopback interrupted-publication evidence are runnable without production writes. Local Oak CI/review and transport-matched hosted phases remain unmeasured. |
121121 | Mount/lazy hydration | `scripts/mount_probe.py`, `scripts/mount_vs_clone.py` | Dry probe and partial remote-backed coverage. Real mount timing needs `OAK_BENCH_MOUNT_REPO`; large-mirror Oak-vs-Git acquisition is not publishable yet. |
122122 | Platform lifecycle | `scripts/platform_lifecycle.py` | Implemented for capability probes, hosted integration anatomy, poll-until-merged, race scenarios, and fake-provider branch triage. Real hosted rows need credentials and disposable repos. Excluded from devloop. |
123123 | Netshape | `scripts/netshape_bench.py` | Implemented where Linux network namespaces and `tc` are available. Cross-subject latency claims require the same shaped pipe and server identity. |
@@ -317,6 +317,12 @@
317317 python3 scripts/task_loop.py --subjects git --git-remote local \
318318 --scenarios agent_task_baseline_v1 --runs 10
319319
320+# Successful Oak clone/edit/commit/push in two independent fresh checkouts,
321+# backed by an owned loopback `oak serve`. Hosted CI/review remain explicit skips.
322+python3 scripts/task_loop.py --subjects oak_local \
323+ --oak-local-bin /path/to/current-oak \
324+ --scenarios agent_task_oak_local_v1 --runs 10
325+
320326 # Actual Oak HTTP publication attempt whose loopback fixture verifies the
321327 # declared request bytes, then drops the reply. Use pinned released/current
322328 # binaries; no live remote is used.
@@ -328,11 +334,14 @@
328334 ```
329335
330336 These cells are different evidence classes. Local Git wall time is not a
331-network comparison. The interruption fixture reports framed-request delivery
337+network comparison, and the Oak local-server row is loopback-only. Its second
338+checkout is fresh and does not claim shared-object or OS-cache warmth. The
339+interruption fixture reports framed-request delivery
332340 knowledge separately from the CLI's own outcome claim and verifies exact local
333341 head, unpublished state, and payload preservation; it does not measure
334342 successful publication latency. Its call counts are scripted command
335-equivalents, not model-observed tool calls. Use
343+equivalents, not model-observed tool calls. Local Oak CI/review are unconfigured
344+and stay explicit skips rather than synthetic passes. Use
336345 `parallel_contention.py --modes workspace_per_task` for concurrent isolated
337346 tasks instead of duplicating its worker implementation here.
338347