oakmark-throughput-benchmark-transplant

Open into main

1 commit · 17 files +16 added ~1 modified · 11 behind main ·Seen by zach 9 hours ago

Description

Add oakmark: per-repo push/clone throughput benchmark for oak (ForgeMark analogue)

Measures how many pushes/hour and clones/hour a single oak repo sustains — oak's answer to Entire's ForgeMark and its per-repo throughput claims. ForgeMark can't be pointed at oak (it drives go-git/smart-HTTP receive-pack; oak has its own HTTP/JSON content-addressed protocol), so this forks ForgeMark's measurement harness and swaps the transport.

New tool at benchmarks/oakmark/:

Go harness forked from ForgeMark: stats.go kept VERBATIM (levelResult schema, nearest-rank percentiles, ops/sec, warm-up exclusion) for cross-tool comparability; agent loops made generic over a Transport seam.
Two transports, reported side by side:
- cli: shells to real oak commit + oak push — honest real-agent floor (two steps deliberately; oak commit --push wrongly needs creds on loopback serve, filed as fb-64).
- protocol: in-process oak-engine (new Rust crate on oakvcs-core) that builds correct-by-construction commit/tree/blob hashes and sends the push HTTP directly — the ForgeMark-comparable ceiling.
Strategies mirror ForgeMark: branch (headline per-repo push throughput), repo (horizontal), session (clones/hour; cli-only).
Markdown report leads with pushes/hour & clones/hour per repo next to Entire's and Cursor's published numbers.

Validated against a local oak serve: both transports pass a cross-client correctness gate (a real oak clone pulls pushed branches with matching payload). Local-serve ceiling ~4.7M pushes/hr/repo (protocol) vs ~0.4M (cli floor, generator-bound). Point -remote https://oak.space with OAK_API_KEY for the real hosted headline.

oakvcs-core is consumed by path (not confirmed on crates.io) via a rendered, gitignored Cargo.toml (build-engine.sh); build artifacts added to .oakignore.

Transplanted byte-for-byte from oak/benchmarks branch oakmark-throughput-benchmark@10ae8d75ec7d onto rebuilt main@88e86b3faccd; preserved only, not merged.

Commits 1

c10cefe7abe5