mrmrs-b26e28

Open into main

1 commit · 4 files +3 added ~1 modified · 6 behind main ·Seen by zach 9 hours ago

Description

Add stale_branch_land_v1: baseline for landing branches that are behind main

New lane scripts/stale_land.py (contention row contract, scenario stale_branch_land_v1, fixture stale_land_fixture_v1). Each trial creates a fresh repo, forks N ordinary agent branches (disjoint files) plus one seeded semantic-conflict pair (pair_rename / pair_consumer: green alone, red combined), moves main k commits, then releases all agents at once.

Arms (contention_mode):

oak_safe: pull converge (fresh shallow worker + reapply fallback) -> local check -> push -> exact-commit CI -> main-unchanged recheck -> hosted merge -> verify
oak_naive: oak merge on the stale branch
git_competent: bare origin + clone per agent over file transport; fetch -> rebase -> CI on combined -> push, with retry on non-ff
git_fifo: the same flow, serialized through a FIFO lock

scripts/oakbench/ci_stub.py: a loopback-only CI execution backend that speaks oakspace's signed dispatch/callback protocol. It handles /jobs, /capabilities, and /cancel, fetches through the token-scoped git export, and posts step and run conclusions. It adds a fixed, labeled delay and writes a ledger that records each run's tree digest.

Independent oracles:

Every main head created during the trial is materialized (server token export / git archive) and checked, giving red_main_events.
The final tree must equal the intended set, with exactly one pair member (intended_changes_exact).
combined_tree_tested_before_land compares the landed commit's tree digest with the digests this agent's checks and CI actually tested.

Oak rows are loopback / hosted-equivalent-local: a local oakspace server built from main e2c41b75 (tree-identical to reviewed 488976f8), with its own Postgres and a local chunk store. Oak tool counts include one explicit oak ci trigger per CI cycle. The local server's publication-triggered CI cannot read workflow blobs from a local chunk store (ci.rs:2687 passes local_chunks=None), so tool_call_count_hosted_equivalent subtracts those calls. HTTP 429 (concurrent-clone integrity proof) and explicitly retryable 409s (push blob-mapping race, merge main_moved) are retried with a bound and counted.

Tests: tests/test_stale_land.py covers the fixture oracle (pair red combined), final-state classification, retry rules, null honesty, git-arm end-to-end, and the CI stub protocol (signature, steps, conclusion, delay, fetch failure). Docs: coverage row in docs/benchmark-coverage.md.

Not merged. No tracker writes.

Commits 1

3d84c25e26db

mrmrs 1 month ago

Files 4

docs

M benchmark-coverage.md

scripts

oakbench

A ci_stub.py

A stale_land.py

tests

A test_stale_land.py

M docs/benchmark-coverage.md

@@ -38,6 +38,7 @@

3838

| Shaped network operations and first-payload timing | Implemented for netshape lane where Linux netns/tc is available | `scripts/netshape_bench.py` `netshape_clone_cold`, `netshape_log_follow_cold`, byteproxy `ttfd_ms` fields | Cross-subject latency claims require the same `netshape_profile` and server host. `netshape_log_follow_cold` measures cold network acquire plus file-history query (`git.log.follow`/`oak.log.follow`) with TTFD fields. Non-Linux or missing Oak local server paths emit skip rows. |

3939

| Mount lifecycle (desc, finish, space clean, task loop) | Implemented handlers; needs remote to run | `mount_probe.py` `oak.desc`/`oak.finish --desc-file`/`space.clean`/`loop.push_desc*`, `task_lifecycle_loop` scenario | `oak finish` capability-gated per binary; skip rows until OAK_BENCH_MOUNT_REPO set. |

4040

| Compact-output follow-up cost | Implemented for streams exposing per-call outcomes | `tool_call_metrics.vcs_info_followup_calls_total` via `oakbench/stream_adapters.py` | Each follow-up means the first output was insufficient; null when commands are not adapter-visible. |

41+

| Stale-branch landing (`stale_branch_land_v1`) | Implemented; baseline before a server-side land primitive | `scripts/stale_land.py` (arms `oak_safe`, `oak_naive`, `git_competent`, `git_fifo`), `scripts/oakbench/ci_stub.py` (loopback CI executor speaking oakspace's signed dispatch/callback protocol), `tests/test_stale_land.py` | Oak rows need an owned local oakspace server (not `oak serve`: no hosted merge/CI gate) and are labeled `loopback` / `hosted-equivalent-local`; Git rows are `local-file`. CI duration is a fixed labeled delay (`ci_added_seconds`) plus a real check script. `combined_tree_tested_before_land` and `red_main_events` come from independent tree oracles (server token-scoped export / `git archive`), never agent reports. Oak `tool_call_count` includes one explicit `oak ci trigger` per CI cycle that hosted auto-triggering would not need (`tool_call_count_hosted_equivalent` subtracts it). Not a transport-matched Oak-vs-Git latency claim. |

4142

| Contention commit-latency tail + retry depth | Implemented | `parallel_contention.py` `commit_latency_tail`, `retry_depth_histogram`, per-worker samples | Cite p95 only when `tail_supported` includes it. |

4243

4344 Known unmeasured-by-design (null in v2 rows until instrumented): hydration