mrmrs-516758

Open into main

1 commit · 11 files +8 added ~3 modified · 6 behind main ·Seen by zach 10 days ago

Description

agent_loop_hosted_v1: hosted agent-loop gap map lane (Oak vs GitHub vs ideal)

New lane agent_loop_hosted measuring each step of one agent task loop (acquire, branch, commit, inspect, publish, review, main advances k=1/k=20 -> sync, merge, next task warm) against hosted oak.space vs GitHub on identically seeded private scratch fixtures. Per step: wall time, tool calls, output bytes, and (proxied mode) HTTP requests / bytes / serial request rounds.

scripts/agent_loop_hosted.py: lane (--part loop|acquire|review; --modes direct,proxied; --git-merge push_ff|pr_squash); independent oracles (blob-id workspace digest, REST/ls-remote heads, on-disk content); holds OAK_BENCH_LOCK.
scripts/oakbench/httpcount.py: stdlib plaintext->TLS forwarding proxy that counts requests (method/route/status/bytes/TTFB, never headers or query strings) plus CONNECT tunnels (bytes/connections) for CDN and GitHub.
scripts/oakbench/rttprobe.py: baseline TCP RTT and warm-request probe.
scripts/seed_agent_loop_fixture.py: deterministic 1000-commit fixture and the Oak squash-merge replay (documents the 128-blob push cap met while seeding).
scripts/agent_loop_readonly_probe.py: read-only probes on real hosted history.
Lane contract entry + scenarios/agent_loop_hosted.yaml catalog; tests.
docs/findings-2026-09-27-agent-loop-gap-map.md: the measured gap map.

Headline (measured): first oak pull after oak clone --shallow = ~1080 serial one-hash commits/info requests, 19 MB, 190-216 s vs Git 1.3 s (sync.rs backfill_parent_ancestor_chain); steady-state sync k=20 45 serial requests 7.8 s vs 1.5 s; review/log --remote serial walks + O(branches) listing; shallow acquire 3.8 s of which ~2.6 s is per-file F_FULLFSYNC; full clone fails the 45 s server integrity budget. Oak wins merge (1.3 s vs 7.2 s GitHub PR path), publish, local commit.

Validation: python3 -m unittest discover -s tests -> 1112 tests OK (3 skipped). Not in devloop (real network/hosted services). Results are not committed (ADR-0003). Do not merge without review.

Commits 1

d3edd8c5e361

mrmrs 1 month ago

Files 11

docs

A findings-2026-09-27-agent-loop-gap-map.md

scenarios

A agent_loop_hosted.yaml

scripts

oakbench

A httpcount.py M lane_contract.py A rttprobe.py

A agent_loop_hosted.py A agent_loop_readonly_probe.py A seed_agent_loop_fixture.py

tests

A test_agent_loop_hosted.py M test_lane_contract.py

M README.md

@@ -141,6 +141,7 @@

141141

| Mount/lazy hydration | `scripts/mount_probe.py`, `scripts/mount_vs_clone.py` | Dry probe and partial remote-backed coverage. Real mount timing needs `OAK_BENCH_MOUNT_REPO`; large-mirror Oak-vs-Git acquisition is not publishable yet. |

142142

| Platform lifecycle | `scripts/platform_lifecycle.py` | Implemented for capability probes, hosted integration anatomy, poll-until-merged, race scenarios, and fake-provider branch triage. Real hosted rows need credentials and disposable repos. Excluded from devloop. |

143143

| Netshape | `scripts/netshape_bench.py` | Implemented where Linux network namespaces and `tc` are available. Cross-subject latency claims require the same shaped pipe and server identity. |

144+

| Hosted agent loop | `scripts/agent_loop_hosted.py` | `agent_loop_hosted_v1`: per-step time, HTTP requests/bytes/serial rounds, tool calls and output bytes for one agent task loop against hosted oak.space vs GitHub on identically seeded private scratch fixtures (`scripts/seed_agent_loop_fixture.py`). Read-side companion on real history: `scripts/agent_loop_readonly_probe.py`. Hosted CI excluded. Not in devloop. |

144145

| Dashboard/control plane | `cloudflare/` | Worker/D1/R2 scaffold exists; live historical dashboard claims require deployed ingest and archived rows. |

145146

146147 Coverage details and public-claim rules live in

@@ -414,6 +415,20 @@

414415 HEAD, history, file bytes, and modes are correctness gates. Establish the A/A

415416 noise floor before interpreting A/B timing.

416417

418+Hosted agent loop gap map (real network; needs write access to one private

419+scratch repo per host; `--mode proxied` counts requests through a local

420+plaintext->TLS proxy, `--mode direct` is timing-only):

421+

422+```bash

423+python3 scripts/seed_agent_loop_fixture.py --source /path/to/oak-tree \

424+ --git-dir /abs/seed/git # then push to GitHub; see --publish-oak-clone

425+python3 scripts/agent_loop_hosted.py --oak-bin /abs/frozen/oak \

426+ --oak-repo ORG/SCRATCH --git-url https://github.com/USER/SCRATCH.git \

427+ --fixture-git /abs/seed/git --review-branch review-target \

428+ --workdir /abs/worker/runs/loop --results /abs/worker/results \

429+ --runs 3 --modes direct,proxied # add --part acquire for acquisition variants

430+```

431+

417432 Mount and acquisition probes:

418433

419434 ```bash