Log in
benchmarksPublic

Commit d3edd8c5

· Browse files at this commit

d3edd8c5e361

Merged on mrmrs-516758
mrmrs · 1 month ago · 11 files +8 added ~3 modified
M README.md
@@ -141,6 +141,7 @@
141141 | Mount/lazy hydration | `scripts/mount_probe.py`, `scripts/mount_vs_clone.py` | Dry probe and partial remote-backed coverage. Real mount timing needs `OAK_BENCH_MOUNT_REPO`; large-mirror Oak-vs-Git acquisition is not publishable yet. |
142142 | Platform lifecycle | `scripts/platform_lifecycle.py` | Implemented for capability probes, hosted integration anatomy, poll-until-merged, race scenarios, and fake-provider branch triage. Real hosted rows need credentials and disposable repos. Excluded from devloop. |
143143 | Netshape | `scripts/netshape_bench.py` | Implemented where Linux network namespaces and `tc` are available. Cross-subject latency claims require the same shaped pipe and server identity. |
144+| Hosted agent loop | `scripts/agent_loop_hosted.py` | `agent_loop_hosted_v1`: per-step time, HTTP requests/bytes/serial rounds, tool calls and output bytes for one agent task loop against hosted oak.space vs GitHub on identically seeded private scratch fixtures (`scripts/seed_agent_loop_fixture.py`). Read-side companion on real history: `scripts/agent_loop_readonly_probe.py`. Hosted CI excluded. Not in devloop. |
144145 | Dashboard/control plane | `cloudflare/` | Worker/D1/R2 scaffold exists; live historical dashboard claims require deployed ingest and archived rows. |
145146
146147 Coverage details and public-claim rules live in
@@ -414,6 +415,20 @@
414415 HEAD, history, file bytes, and modes are correctness gates. Establish the A/A
415416 noise floor before interpreting A/B timing.
416417
418+Hosted agent loop gap map (real network; needs write access to one private
419+scratch repo per host; `--mode proxied` counts requests through a local
420+plaintext->TLS proxy, `--mode direct` is timing-only):
421+
422+```bash
423+python3 scripts/seed_agent_loop_fixture.py --source /path/to/oak-tree \
424+ --git-dir /abs/seed/git # then push to GitHub; see --publish-oak-clone
425+python3 scripts/agent_loop_hosted.py --oak-bin /abs/frozen/oak \
426+ --oak-repo ORG/SCRATCH --git-url https://github.com/USER/SCRATCH.git \
427+ --fixture-git /abs/seed/git --review-branch review-target \
428+ --workdir /abs/worker/runs/loop --results /abs/worker/results \
429+ --runs 3 --modes direct,proxied # add --part acquire for acquisition variants
430+```
431+
417432 Mount and acquisition probes:
418433
419434 ```bash