oakmark — per-repo push/clone throughput benchmark for oak

# oakmark — per-repo push/clone throughput benchmark for oak

`oakmark` measures how many **pushes/hour and clones/hour a single oak repo can

sustain** — oak's answer to [Entire's ForgeMark](https://github.com/entireio/forgemark)

and the per-repo throughput numbers in

[their blog post](https://entire.io/blog/an-entirely-new-git-hosting-network)

(Entire 21M pushes/hr · 570K clones/hr; Cursor Origin 81K · 296K).

## Why a fork, not ForgeMark itself

ForgeMark drives real pushes through **go-git against smart-HTTP git forges**

(`git-receive-pack`). Oak does not speak the git wire protocol — it has its own

HTTP/JSON protocol (`POST /api/{owner}/{name}/push`, content-addressed BLAKE3

chunks, flat branches). So ForgeMark **cannot be pointed at oak.space or

`oak serve`**. `oakmark` keeps ForgeMark's measurement harness — the concurrency

sweep, warm-up exclusion, nearest-rank latency percentiles, `ops/sec → ops/hour`,

and the `levelResult` JSON schema (`stats.go`, kept **verbatim** for

comparability) — and swaps the transport for oak's protocol.

## Two transports — report both

Both are shipped because they answer different questions and publishing both is

more credible than either alone:

| `-transport` | What it is | Reads as |

| --- | --- | --- |

| `cli` | Shells out to the real `oak` binary (`oak commit` + `oak push`) | **Honest real-agent floor** — includes client-side BLAKE3 chunking + one process spin-up per commit and push. Can become *generator-bound* before the server does at high concurrency. |

| `protocol` | Drives the in-process `oak-engine` (Rust, built on `oakvcs-core`) — builds correct oak objects and sends the push HTTP directly, no subprocess | **ForgeMark-comparable ceiling** — analogous to ForgeMark driving go-git in-process. |

`cli` uses two steps (`oak commit` then `oak push`) deliberately: on an open

loopback `oak serve`, `oak commit --push` wrongly demands credentials while

standalone `oak push` succeeds anonymously (filed as oak friction fb-64).

## Strategies (mirror ForgeMark)

- **branch** — N agents, each pushing to its own branch off `main`, in a tight

  loop. Oak's flat-branch model means each agent owns an independent head, so

  pushes never lose a CAS race: this isolates the server's per-repo push

  throughput. The headline single-repo number.

- **repo** — agents spread across N repos (`-repo-count`); horizontal ceiling.

- **session** — each agent shallow-clones the base repo, pushes `-session-commits`

  checkpoints to an ephemeral branch, then abandons it: the realistic agent

  lifecycle that interleaves read (clone) and write (push) load. This yields the

  **clones/hour** number. `cli`-only (clone latency is server-dominated, so the

  CLI is representative for the read side).

## Build

```bash

go build -o oakmark .                 # the harness (Go)

cargo build --release --manifest-path oak-engine/Cargo.toml   # the protocol engine (Rust)

```

## Run

Against a local `oak serve` (no auth, reproducible — best for iterating/CI):

```bash

oak serve --dir /tmp/oakmark-data --port 8096 &

# real-agent floor:

./oakmark -remote http://127.0.0.1:8096 -transport cli      -strategy branch \

  -owner bench -repo tp -concurrency 1,8,32 -duration 30s -warmup 5s -report cli.md

# in-process ceiling:

./oakmark -remote http://127.0.0.1:8096 -transport protocol -strategy branch \

  -owner bench -repo tp -concurrency 1,8,32,128 -duration 30s -warmup 5s -report proto.md

# clones/hour:

./oakmark -remote http://127.0.0.1:8096 -transport cli -strategy session \

  -owner bench -repo tp -session-commits 5 -clone-depth 1 -concurrency 8,32 -duration 30s

```

Against **oak.space** (the real per-repo headline; needs a token + an existing

org). `OAK_API_KEY` or `-token-file` supplies the bearer token; mint one with

`oak login` once. Use a dedicated throwaway org/repo and `-branch-prefix` for

cleanup — **this writes to production**:

```bash

OAK_API_KEY=$(cat ~/.oak-token) ./oakmark -remote https://oak.space -transport protocol \

  -strategy branch -owner <benchorg> -repo tp -concurrency 1,8,32,128 \

  -duration 60s -warmup 10s -branch-prefix oakmark-$(date +%s) -out oakspace.json -report oakspace.md

```

`run-local.sh` runs the whole local matrix and drops reports in `results/`.

## Output

Per concurrency level: `ops_per_sec`, `p50/p95/p99/p999/max_ms`, `cas_failures`,

`other_errors` (ForgeMark's schema), plus clone stats in session mode. The

markdown report leads with **peak pushes/hour and clones/hour per repo**, next to

Entire's and Cursor's published numbers.

## Correctness gates

- `cli`: after a run, `oak clone` the repo and `oak switch` to an agent branch —

  the pushed `file-*.txt` payload must be present.

- `protocol`: a branch pushed by `oak-engine` must be pullable by the real `oak`

  CLI with matching file content (the server re-verifies commit/tree hashes and

  rejects mismatches, so a wrong object build fails the push outright).

## Honesty note

Oak's push is not a git `receive-pack`; it is a content-addressed, chunked,

flat-branch protocol. `oakmark` measures at the **workload** level (an agent

publishing a changeset of N files) — the same level the referenced per-repo

claims live at — and always records the transport, warm-up, and noise so a number

is never quoted out of context.