Viewing files at commit d3edd8c5e361Commit detailsView latest

Browse files
docs/findings-2026-09-27-agent-loop-gap-map.md 327 lines · 22.7 KB

Agent-loop gap map: hosted Oak vs GitHub vs the physical ideal

Scenario agent_loop_hosted_v1 (oak/benchmarks lane agent_loop_hosted), measured 2026-09-27 08:50-10:10 UTC on skullcaacStudio.lan (macOS 26.6.2, shared build host; 1-minute load average 1.8-15.8 during runs, recorded per row; other sessions' benchmark campaigns were serialized by the same lock). Hosted CI is excluded by construction (fixture carries no CI config).

Gap table (scratch fixtures, identical content)

Time = --mode direct median [min-max] and n; requests / serial rounds / bytes = --mode proxied median. Git request counts include GitHub's 401 credential challenge (one per Git process). Git request counts were recomputed from the raw rows: the harness's GIT_TRACE_CURL parser counted every HTTP/2 request twice (fixed in the committed harness, with a regression test; the correction is exact). Git acquire.cold in this table is the loop's clone --filter=blob:none; other variants are below. inspect.review_remote uses a freshly forked review branch (the loop's stale-branch rows, with failures, are in the appendix). Ratio = Oak/Git direct median; ideal = physical minimum round trips (0 for local steps, 1 for anything that must consult the server).

stepOak time s: direct median [min-max] nOak requests / serial rounds / bytesOak calls / out BGit time s: direct median [min-max] nGit requests / bytesGit calls / out Bideal RTOak/Git timeOak rounds / ideal
acquire.cold3.76 [3.61-4.00] n=104 / 4 / 2.1MB1 / 3851.77 [1.47-2.68] n=137 / 2.2MB1 / 10412.1x4/1
branch.create0.20 [0.18-0.29] n=51 / 1 / 1KB1 / 7311ms [10ms-12ms] n=80 / 0B1 / 41019.2x1/0
edit.commit23ms [20ms-32ms] n=50 / 0 / 0B1 / 059ms [44ms-67ms] n=80 / 0B2 / 000.4x0/0
inspect.status17ms [16ms-18ms] n=50 / 0 / 0B1 / 110226ms [11ms-27ms] n=80 / 0B1 / 8200.7x0/0
inspect.diff_main9ms [9ms-11ms] n=50 / 0 / 0B1 / 11910ms [9ms-11ms] n=80 / 0B1 / 24000.9x0/0
inspect.log6ms [6ms-7ms] n=50 / 0 / 0B1 / 1709ms [9ms-11ms] n=80 / 0B1 / 60800.7x0/0
publish.branch1.22 [1.07-1.28] n=55 / 4 / 147KB1 / 5901.56 [1.29-1.71] n=83 / 8KB1 / 19410.8x4/1
inspect.review_remote1.80 [1.73-2.68] n=52 / 2 / 20KB1 / 40720.57 [0.53-1.47] n=53 / 6KB2 / 14613.2x2/1
inspect.main_since_branch.k11.80 [1.12-2.08] n=54 / 4 / 69KB1 / 97020.79 [0.63-1.18] n=84 / 8KB2 / 2912.3x4/1
sync.k1208.62 [189.65-216.00] n=51082 / 1082 / 19.1MB1 / 1991.30 [1.04-1.80] n=86 / 12KB1 / 01160.0x1082/1
sync.k207.80 [7.72-8.16] n=545 / 45 / 730KB1 / 2281.47 [1.20-1.90] n=87 / 20KB1 / 015.3x45/1
merge.main1.34 [1.29-1.43] n=57 / 6 / 128KB1 / 4691.62 [1.21-1.67] n=53 / 7KB1 / 010.8x6/1
next_task.warm55ms [49ms-72ms] n=50 / 0 / 0B1 / 360.70 [0.52-0.85] n=83 / 6KB2 / 00-10.1x0/0-1

k=1 vs k=20: sync.k1 is the first sync after a shallow acquire (it includes the one-time history backfill); sync.k20 is steady state in the same workspace. Cold vs warm: acquire.cold is a fresh directory (no shared store; OS and DNS warm); next_task.warm is the second task in the same workspace. oak mount time-to-first-read: 2.6-2.8 s mount + 0.39-0.50 s first read (n=5, every mount ended cleanly). It works, but it is no faster than a complete git clone --depth 1 workspace (1.1 s).

Read side on real history: hosted oak/oak (Oak only, read-only)

oak/oak stepdirect s (each run)proxied s (each run)requests / serial roundsbytes (API + CDN)failed / attempted
clone_shallow3.4, 3.43.3, 3.64/4, 4/42.1MB, 2.1MB0/4
log_remote5.7, 5.95.4, 6.121/21, 21/210.6MB, 0.6MB0/4
review_remote7.3, 7.06.5, 7.528/28, 28/280.8MB, 0.8MB0/4
diff_remote2.2, 2.22.0, 2.61/1, 1/10.2MB, 0.2MB0/4
fetch159.0, 163.1146.1, 181.6989/989, 989/98910.8MB, 10.8MB0/4
pull_after_shallow0.6, 0.40.52/20.0MB0/3
pull_noop0.4, 0.40.4, 0.52/2, 2/20.0MB, 0.0MB0/4
clone_full45.4 FAIL, 45.2 FAIL55.24/448.1MB2/3

diff_remote is a single GET /branches (all 441 branches, 218 KB, ~2 s) and nothing else: listing every branch to pin one head is O(branches) server time (branch.rs:723-746 fetch_remote_branches, called from review.rs:3726 and log_remote.rs:261; GET /branches/{name} exists and answers in ~80 ms).

Variants and failures (all attempts)

operationsubjectvariantmodeattemptedokoracle okmedian s [min-max]requestsbytes
acquire.coldgitgit_clone_bloblessdirect1313131.77 [1.47-2.68]--
acquire.coldgitgit_clone_bloblessproxied1313131.76 [1.56-5.49]72.2MB
acquire.coldgitgit_clone_depth1direct5551.10 [0.93-1.42]--
acquire.coldgitgit_clone_depth1proxied6661.22 [0.98-1.58]41.9MB
acquire.coldgitgit_clone_fulldirect5551.04 [0.91-1.55]--
acquire.coldgitgit_clone_fullproxied6661.23 [0.86-1.56]42.2MB
acquire.coldgitgit_worktree_from_warmdirect5550.85 [0.53-1.34]--
acquire.coldgitgit_worktree_from_warmproxied6650.79 [0.60-0.99]36KB
acquire.coldoakoak_clone_fulldirect200---
acquire.coldoakoak_clone_fullproxied300---
acquire.coldoakoak_clone_shallowdirect1010103.76 [3.61-4.00]--
acquire.coldoakoak_clone_shallowproxied1010103.81 [3.61-3.99]42.1MB
inspect.review_remotegitfresh_review_branchdirect5550.57 [0.53-1.47]--
inspect.review_remotegitfresh_review_branchproxied3330.67 [0.55-1.23]36KB
inspect.review_remoteoakdirect5207.30 [5.53-9.07]--
inspect.review_remoteoakproxied4204.47 [1.66-7.27]19329KB
inspect.review_remoteoakfresh_review_branchdirect5551.80 [1.73-2.68]--
inspect.review_remoteoakfresh_review_branchproxied3331.71 [1.63-2.22]220KB
merge.maingitgit_gh_pr_squashdirect3337.15 [6.56-7.31]--
merge.maingitgit_gh_pr_squashproxied3336.73 [6.44-6.86]1130KB
merge.maingitgit_push_ff_to_maindirect5551.62 [1.21-1.67]--
merge.maingitgit_push_ff_to_mainproxied4441.50 [1.19-1.67]37KB
merge.mainoakoak_merge_hosted_squashdirect5551.34 [1.29-1.43]--
merge.mainoakoak_merge_hosted_squashproxied4441.26 [1.19-1.35]7128KB

Notes: Oak loop inspect.review_remote rows (empty variant) review the stale review-target branch; every failure is review preparation incomplete: cumulative exact-commit budget exceeded (review.rs:3804 MAX_EXACT_COMMITS = 256) once main was ~130+ commits past the branch's fork point; their oracle ok is 0 because the review oracle was added after those rows (the extra-run Git rows pass it). One git_worktree_from_warm oracle miss is the harness's own bug (worktree .git file counted as an extra file), fixed before the remaining runs.

Subjects, fixtures, instruments

  • Oak 0.105.0, frozen copy bin/oak, sha256 4dc220e1f089845ca1ada8da4025b18b37f7823e266989e757302b8fa8bc6647, against https://oak.space.
  • Git 2.50.1 (Apple Git-155) against https://github.com, auth via gh auth git-credential (so every Git process pays GitHub's 401 challenge round trip — that is what a stock credential-helper setup costs; it is counted, not hidden).
  • Fixtures (write side, both private, identical content): oak/oak tree at d4d8062d minus .github/ and .oak/workflows/ (314 files incl. 50 synthetic), plus 1000 deterministic commits (seed 20260927; 36 --no-ff merges).
    • GitHub mrmrs/agent-loop-bench-20260927: plain git push of that history.
    • oak.space mrmrs/agent-loop-bench-20260927: same 1000 commits replayed as branch commits + 100 hosted squash merges (~10 commits per merge; the first import slice was the repo-creating push). Final tree verified identical to the Git fixture (blob-id digest, 314/314 files). Oak main's first-parent chain is therefore ~100 long vs Git's 928; the reachable DAG (parent + merge_parent) holds all ~1100 commits, which is what the O(history) walks traverse. Each loop trial adds 1 task + 21 actor merges.
    • review-target branch (3 files x 15 lines) on both, unmerged.
  • Read-side realistic history: hosted oak/oak (read-only probes; 441 branches; no GitHub mirror exists, so Git has no oak/oak row — the Git comparison is the scratch fixture).
  • Instruments. Oak: a local plaintext->TLS forwarding proxy (scripts/oakbench/httpcount.py) counts every oak.space API request with method/route/status/bytes/TTFB; CDN chunk traffic (cdn.oak.space) goes through the same proxy as CONNECT tunnels (bytes + connections measured; CDN request count unmeasured). Git: CONNECT tunnel for bytes + GIT_TRACE_CURL for request count (headers only; Authorization redacted by Git; trace deleted after parse). "Bytes" = HTTP/1.1 plaintext bytes for Oak API + TLS tunnel bytes (CDN, GitHub). Timing comes from --mode direct (no proxy, Oak on HTTP/2); counts from --mode proxied. Proxy distortion check: Oak sync.k1 189.6 s direct vs 206.6/224.6 s proxied (+9-18%); per-request TTFB through the proxy (~170 ms for commits/info) matches direct per-request cost (189.6 s / ~1080 req = 176 ms).
  • Baseline RTT (scripts/oakbench/rttprobe.py, n=20): TCP connect oak.space median 21.0 ms (16.9-108.9), cdn.oak.space 23.2 ms, github.com 46.4 ms (43.0-133.5). Warm-connection trivial HTTPS GET: oak.space 27.0 ms (edge-served robots.txt), github.com 47.1 ms. Oak API requests are slower than RTT by server time: trivial GET /api/{repo} TTFB ~87-116 ms, POST commits/info TTFB ~170 ms (measured through proxy).
  • Oracles (independent of CLI success output): workspace blob-id digest vs the local Git fixture tree (acquire), remote heads via Oak REST GET /api/{repo}/branches and git ls-remote (publish, merge), on-disk file content (sync: actor file has exactly k lines and the task patch survives; next task: merged patch + k=20 actor file present), squash tree == branch tree (gh PR path).
  • Ideal = the physical minimum for the step: 0 round trips for local steps, 1 request round trip for any step that must consult/update the server (excluding connection setup), bytes = content that actually changed (compressed).
  • Measurement lock OAK_BENCH_LOCK held for every timed run (runlock default path).

Verdict on the hypothesis

Confirmed (measured), with one addition. The dominant agent-time cost in the Oak loop is the client reconstructing history locally, one commit per serial request: the first oak pull after oak clone --shallow issued ~1060-1100 sequential POST /commits/info (one hash each), downloaded ~18-19 MB (97% of it tree objects the client did not ask for) and took 189.6-216.0 s direct (median 208.6 s, n=5), vs 1.30 s for git pull --rebase on the same content. The same per-commit pattern (not the O(history) part) makes every steady-state remote read O(k) or O(depth) round trips: sync.k20 45 requests, log --remote -n 20 21, branch review --remote 6-32. On real oak/oak, oak fetch after a shallow clone was 989 serial requests / 146 s / 10.4 MB. The addition: the second-largest acquire cost is not network at all but two F_FULLFSYNCs per materialized file.

Top gaps, ranked by agent-time impact

1. First sync after a shallow acquire walks the whole history, one commit per request (~190 s vs 1.5 s; ~1080 RTs vs ideal 1)

  • Measured: sync.k1 Oak direct 208.6 s median [189.6-216.0], n=5; proxied 1061-1104 requests, every one its own serial round, 18-19 MB down. Git 1.30 s [1.04-1.80], n=8, 6 requests (incl. one 401 credential challenge), 12 KB. oak/oak: oak fetch on a shallow clone 989 serial requests, 146 s, 10.4 MB; afterwards oak pull is 2 requests / 0.4 s. Also triggered (by source reading, not measured here) by oak fetch, oak merge finish/recovery, and oak switch -c whenever main has moved (the "head moved" branch runs the backfill for HeadOnly callers too).
  • Root cause (oak/oak @ d4d8062d): cli/src/commands/sync.rs:116-121 (pull's parent sync asks fetch_parent_from_server(.., AncestryNeed::Complete)) -> sync.rs:1652 (moved head: backfill parent chain) -> sync.rs:1688-1778 backfill_parent_ancestor_chain: BFS over parent + merge_parent edges; for a locally present commit it keeps walking its parents (1719-1726, deliberately crossing the shallow boundary), and for each missing commit awaits fetch_commit_metadata (sync.rs:2124-2138) whose body is hashes: vec![head] — exactly one hash per HTTP request, strictly serial. Server side (oakspace, inferred from the wire): commits/info ignores metadata_only: true and always attaches the commit's tree objects (49 KB uncompressed / 17.5 KB gzip per commit vs ~0.6 KB of commit metadata; verified by a direct API call with the flag on and off — identical 49,443-byte responses; log_remote.rs:34 documents "the trees the endpoint always attaches"). Each request costs ~170 ms TTFB against a 21 ms network RTT, so the gap is server time x serialization, not bandwidth.
  • Smallest change that closes it: make the pull-side need a merge-base need: stop the backfill at commits already present locally (the shallow clone's boundary is the task branch's fork point, so the LCA is always above it) and fetch only the new segment; batch whatever frontier remains into one commits/info call (the endpoint already accepts hashes[] — push's edge-proof sends up to 256). Expected: sync.k1 from ~1080 serial requests to ~k (then ~1 with #2). Server: honor metadata_only (-97% bytes). The server-authoritative version: commits/info gains an ancestors_of=HEAD&until=[haves] mode that returns the closure in one response (Git's want/have negotiation, 1 RT).

2. Every remote read or sync is O(new commits) or O(depth) serial round trips

  • Measured: steady-state sync.k20 (history already backfilled) Oak 7.80 s [7.72-8.16], n=5, 45 requests = 2 per landed squash merge (squash + its branch commit), all serial; Git 1.47 s [1.20-1.90], n=8, 7 requests. inspect.review_remote Oak 1.80 s [1.73-2.68] (n=5) on a fresh branch, 5.5-9.1 s on a branch ~130+ commits behind main (6-32 serial commits/info rounds searching for the merge base) vs Git fetch+diff 0.57 s [0.53-1.47] (n=5). On the stale branch it then fails in 6 of 9 loop attempts with review preparation incomplete: cumulative exact-commit budget exceeded (review.rs:3804, MAX_EXACT_COMMITS = 256); Git has no such cliff. Even the fresh-branch review spends ~1.4 s of its 1.8 s in one GET /branches listing every branch (O(branches) server time) just to pin two heads. log --remote -n 20 on oak/oak: 21 serial requests, 630 KB, 5.4 s for 20 one-line entries. inspect.main_since_branch.k1 Oak 1.80 s [1.12-2.08] (n=5) (3-22 requests depending on what is already local) vs Git fetch+log 0.79 s [0.63-1.18] (n=8).
  • Root cause: the same one-hash-per-request pattern in the ancestry walks: sync.rs:1688-1778 (sync), review.rs:3797-3897 prepare_remote_review_ancestry (a loop that recomputes the merge base locally, then fetches only the currently blocking gap each round), log_remote.rs:246 (first-parent walk: "the next first parent is only known from the previous commit"). The client cannot know hashes ahead of time, so it cannot parallelize; only the server can answer in one trip. Head pinning uses branch.rs:723-746 fetch_remote_branches (full listing) from review.rs:3726 and log_remote.rs:261.
  • Smallest change: server-side walks on the endpoint that already exists: commits/info with {walk: "first_parent"|"ancestors", limit, until} (log and sync), and a merge-base/compare answer for review (the server already computes merges for oak merge). Client change is replacing each loop with one call, and pinning heads with GET /branches/{name} (~80 ms) instead of the full listing (0.7-2 s). Ideal = 1 RT each; today 20-45.

3. Acquire: two F_FULLFSYNCs per file, and full clone is O(history) on the server

  • Measured: acquire.cold Oak shallow 3.76 s [3.61-4.00] (n=10) vs Git 1.10 s [0.93-1.42] (n=5) (depth 1) / 1.77 s [1.47-2.68] (n=13) (blobless). Oak's network part is fast — 4 serial API requests done at ~0.7 s, 322 chunks via 9 parallel CDN connections (batch endpoint, cf-cache HIT) done at ~1.0 s — but a sample of the process shows ~90% of the remaining ~2.6 s inside File::sync_all (two per file). Full clone: fails on the 1000-commit scratch fixture after the server integrity proof exhausts its 45 s budget: 5/5 attempts failed at 45.2 s. On oak/oak, 2 attempts succeeded (72.6 s manual, 55.2 s proxied with 48 MB of CDN traffic; 08:58-09:10 UTC) and the 2 later ones failed the same way at 45.2-45.4 s (10:01-10:09 UTC). git clone (full history) of the scratch fixture: 1.04 s [0.91-1.55], 2.2 MB.
  • Root cause: cli/src/materialize.rs:464 writes every worktree file via atomic_file::write_atomic -> cli/src/atomic_file.rs:113 file.sync_all()
    • :117/:142-143 sync_parent_dir (fsync of the directory). On macOS sync_all is F_FULLFSYNC (~4 ms each). Git never fsyncs worktree files. Full clone: the pre-download integrity proof walks the whole history server-side (GET /integrity, 45 s wall budget, integrity.rs:379), and Oak ships whole chunk content per historical blob (no delta compression): 46 MB vs Git's 2.2 MB pack.
  • Smallest change: materialize worktree files without per-file durability (write temp + rename; one directory fsync per directory or none) — the object store/SQLite remain the durable copy, so a crash loses nothing that is not re-materializable. Expected (inferred: removes the ~2.6 s sampled in fsync): shallow acquire ~3.7 s -> ~1.1 s, on par with git clone --depth 1. Full clone: make the integrity proof incremental (cache/prove per-commit once, not per clone) and default agents to shallow.

Where Oak already beats (or ties) Git — measured

  • Merging to main vs the GitHub PR path: 5x faster, 1 call vs 3. oak merge 1.34 s [1.29-1.43] (n=5; server squash + branch close + CI gate in one call) vs git push -f + gh pr create + gh pr merge --squash 7.15 s [6.56-7.31] (n=3, 11 requests). Against a bare fast-forward git push origin HEAD:main (no review record, no squash) Oak still ties or wins: 1.34 vs 1.62 s.
  • Publishing a branch: 1.22 s [1.07-1.28] vs 1.56 s [1.29-1.71] (Git pays a 401 credential round trip per process).
  • Local commit: 23 ms, 1 call vs 59 ms, 2 calls (add + commit).
  • Local inspection (status/diff/log): parity, all under 30 ms.
  • Next task in the same workspace: 55 ms, 0 requests vs 0.70 s (Git fetch + switch). Caveat: Oak's switch -c here did not consult the server (main had not moved); Git without the fetch would also be ~10 ms. Call it parity with a better default.
  • Network distance: oak.space is 2x closer than github.com (21 vs 46 ms TCP RTT; Cloudflare edge). Oak currently spends that advantage many times over in server time per request (87-170 ms TTFB; GET /branches 0.7-2 s) and in serial request chains.
  • Acquire bytes: shallow clone 2.1 MB (322 chunks, batched, 9 parallel CDN connections, cf-cache HIT) vs 1.9 MB for git clone --depth 1: parity.

Everything else

  • branch.create: Oak makes 1 network request (GET /api/{repo} from switch.rs HeadOnly parent refresh) — 0.20 s [0.18-0.29] vs Git 11 ms (local only). Small absolute cost, but ideal is 0.
  • publish.branch: Oak 5 requests / 4 serial rounds (repo GET, branch GET, 2x blobs/check, push) vs ideal 1; Git 3 requests (incl. one 401 challenge). Oak uploads 135-140 KB for a ~2 KB patch (inferred: the push body carries tree objects); Git sends a 7 KB pack.
  • merge.main: Oak 7 requests / 6 rounds (branch GET, repo GET, commits/info, 2x blobs/check, push, merge) vs ideal 1; still ties or beats Git's push.
  • Output bytes: Oak's JSON surfaces are 10-100x Git's text for the same step (status 1.1 KB vs 82 B, review 4-6 KB vs 143 B, log --remote --json ~10 KB vs 29 B). These are recall-rich machine formats; bytes are not tokens and this map does not claim a token cost.

Seeding failures preserved (write-side product gaps found during setup)

  1. oak push --repo of a 1000-commit imported history: refused client-side, "exceeds the safe ordinary publication envelope and requires staged_v1, but the server has not advertised the complete staged capability set". Cause: push.rs:942-947 requires staged push for >128 blobs (MAPPING_PROOF_MAX_BLOBS) or >500 commits, and oak.space returns 404 for /api/capabilities. Observed with a 349-commit / 264-file import (under the 500-commit cap, so the blob cap is what refused it). Consequence (inferred from the same rule): no push may carry more than 128 new blobs today, so a fresh ~270-file repository cannot be published in one push.
  2. Extending a remote main from a fresh oak clone file://<git> import: "push admission phase could not prove older remote commit edge(s)"; main only advances by hosted squash merge. The fixture was therefore replayed as 100 branch+merge cycles (~3.5 s each).

Limits

  • One runner, one network (home ISP), shared build host (load 1.8-15.8). n per cell is in the tables (loop: 5 direct + 4 proxied Oak trials, 8 direct Git; acquire: 5-13 per variant; fresh review 5 + 3). Treat sub-100 ms differences as noise.
  • Proxied Oak runs use HTTP/1.1 through a local hop; direct Oak uses HTTP/2. Counts come from proxied runs, time from direct runs.
  • CDN request counts (Oak) and per-request timing (Git) are unmeasured.
  • Oak's history grows by ~22 merges per trial, so later sync.k1 walks are a little longer; the trend is visible in the rows.
  • The Git merge row uses a fast-forward push to main (competent single-agent path); the gh pr create + gh pr merge --squash row is reported separately.
  • Hosted CI excluded by design; oak merge gating on CI therefore did not wait.