mrmrs-0a9c5c

Closed into main

3 commits · 6 files ~6 modified

Description

Measure bounded end-to-end agent task costs without production mutations.

Adds a successful owned-loopback Oak task cell: seed oak serve, acquire a fresh full clone, inspect/edit/commit/push, acquire a second independent fresh clone, repeat, then verify exact file bytes and selected-branch head from a third fresh clone and the server branch endpoint. Every process and command is bounded and owned; local CI/review remain explicit unmeasured rows. The cell does not claim shared-object warmth, known OS-cache state, network-byte measurement, hosted semantics, or a latency comparison with local-file Git.

Independent QA correction: interruption evidence certifies a request only when one strict ASCII-decimal, bounded Content-Length (with valid HTTP OWS) exactly matches observed bytes. Signed/duplicate lengths are rejected. Deadline coverage handles partial senders, and owned POSIX groups are killed even when the leader exits but a descendant ignores SIGTERM. Fixture delivery knowledge stays separate from the CLI's own reported outcome. Exact worktree payload hash, local head, and unpublished state must all survive; a generic CLI error is unreported rather than inferred unknown.

Negative fixtures cover truncated, missing, invalid, signed, duplicate, oversized, and stalled request bodies, timed-out children and POSIX descendants, outcome separation, and payload mismatch. Call counts are explicitly scripted command equivalents, not model-observed calls. A corrected n=10 matrix for released 0.102.1 and preserved 0.103.0 passed request/head/payload/unpublished state/no-replay gates. A quiet-window n=10 successful local Oak cell passed exact payload/head/server-cleanup gates; local CI/review remain unconfigured.

Extends the existing task-loop lane with a local-file Git cold full clone plus warm shared-object worktree scenario, explicit unmeasured hosted-CI rows, per-phase/tool/output/setup/cleanup/cache/storage accounting, and remote payload oracles. Adds an actual Oak loopback interrupted-publication scenario that reads exactly one complete push request, drops the response, and verifies the client preserves local head/work without blind replay. Fixes the local bare remote HEAD seed that previously broke the second worktree acquisition.

Reuses parallel_contention.py for concurrent isolation rather than adding a second worker framework. Documents evidence limits: local Git is not a network comparison; the fault fixture is not accepted-then-lost durability; hosted CI, network bytes, OS cache state, partial clone, and Oak shared-cache warmth stay unmeasured.

Verification: full 896 unittest cases pass with two intentional skips; focused actual-Oak task/instrument tests 51 pass; n=10 successful local Oak, n=10 Git task, n=10 each released/current Oak interruption, and n=10 Git workspace isolation at workers 2 and 4 all passed their applicable oracles. Independent review of this final amendment remains required.

Commits 3

Files 6

docs

M benchmark-coverage.md

scripts

oakbench

M execution.py

M task_loop.py

tests

M test_new_instruments.py M test_task_loop.py

M README.md

@@ -117,6 +117,7 @@

117117

| Scripted workflows | `scripts/workflow_ab.py` | Implemented for deterministic bugfix, wide refactor, large asset, history archaeology, error recovery, and sync/divergence recovery workflows. |

118118

| Real agents | `scripts/agent_workflow.py` | Implemented for mock plus local Codex, Claude, and Cursor Agent stream adapters. Use real rows only when the installed CLI stream has campaign evidence. |

119119

| Parallel contention | `scripts/parallel_contention.py` | Implemented for Git shared/workspace modes; Oak workspace-per-task rows require a disposable Oak remote and otherwise skip honestly. |

120+

| End-to-end task loop | `scripts/task_loop.py` | Git local-file cold-clone/warm-worktree rows, successful Oak two-fresh-checkout work against owned loopback `oak serve`, and actual Oak loopback interrupted-publication evidence are runnable without production writes. Local Oak CI/review and transport-matched hosted phases remain unmeasured. |

120121

| Mount/lazy hydration | `scripts/mount_probe.py`, `scripts/mount_vs_clone.py` | Dry probe and partial remote-backed coverage. Real mount timing needs `OAK_BENCH_MOUNT_REPO`; large-mirror Oak-vs-Git acquisition is not publishable yet. |

121122

| Platform lifecycle | `scripts/platform_lifecycle.py` | Implemented for capability probes, hosted integration anatomy, poll-until-merged, race scenarios, and fake-provider branch triage. Real hosted rows need credentials and disposable repos. Excluded from devloop. |

122123

| Netshape | `scripts/netshape_bench.py` | Implemented where Linux network namespaces and `tc` are available. Cross-subject latency claims require the same shaped pipe and server identity. |

@@ -144,9 +145,10 @@

144145 - Workspace-per-agent fleets: Git worktree contention is implemented. Oak

145146 workspace-per-task needs real disposable remotes, high-concurrency mount/push

146147 behavior, branch cleanup, and merge/integration plumbing.

147-- Full Oak task loop: the roadmap is mount -> edit -> commit -> push -> desc

148- -> finish/clean -> remount follow-up, compared with Git's clone/worktree,

149- branch, commit, push, cleanup, and follow-up flow.

148+- Full hosted task loop: local Git cold-clone/warm-worktree and Oak

149+ interrupted-publication cells are implemented. A transport-matched Oak/GitHub

150+ acquisition -> publish -> exact-CI -> review comparison still needs disposable

151+ provider fixtures; missing provider phases stay explicit skip rows.

150152 - Hosted workflow evidence: platform rows need GitHub org assets, PATs or app

151153 credentials, webhooks, disposable repos, and real Oak/GitHub provider wiring

152154 before public hosted-branch claims.

@@ -307,6 +309,42 @@

307309 --workers 2,8,32 --commits-per-worker 5

308310 ```

309311

312+End-to-end task cost without production writes:

313+

314+```bash

315+# Fresh full clone followed by a shared-object Git worktree. Hosted CI remains

316+# an explicit unmeasured phase on the local-file transport.

317+python3 scripts/task_loop.py --subjects git --git-remote local \

318+ --scenarios agent_task_baseline_v1 --runs 10

319+

320+# Successful Oak clone/edit/commit/push in two independent fresh checkouts,

321+# backed by an owned loopback `oak serve`. Hosted CI/review remain explicit skips.

322+python3 scripts/task_loop.py --subjects oak_local \

323+ --oak-local-bin /path/to/current-oak \

324+ --scenarios agent_task_oak_local_v1 --runs 10

325+

326+# Actual Oak HTTP publication attempt whose loopback fixture verifies the

327+# declared request bytes, then drops the reply. Use pinned released/current

328+# binaries; no live remote is used.

329+python3 scripts/task_loop.py --subjects oak_installed,oak_local \

330+ --oak-installed-bin /path/to/released-oak \

331+ --oak-local-bin /path/to/current-oak \

332+ --scenarios interrupted_publication_after_request_v1 \

333+ --runs 10 --randomize-subject-order

334+```

335+

336+These cells are different evidence classes. Local Git wall time is not a

337+network comparison, and the Oak local-server row is loopback-only. Its second

338+checkout is fresh and does not claim shared-object or OS-cache warmth. The

339+interruption fixture reports framed-request delivery

340+knowledge separately from the CLI's own outcome claim and verifies exact local

341+head, unpublished state, and payload preservation; it does not measure

342+successful publication latency. Its call counts are scripted command

343+equivalents, not model-observed tool calls. Local Oak CI/review are unconfigured

344+and stay explicit skips rather than synthetic passes. Use

345+`parallel_contention.py --modes workspace_per_task` for concurrent isolated

346+tasks instead of duplicating its worker implementation here.

347+

310348 Mount and acquisition probes:

311349

312350 ```bash