Infrastructure & External Assets Plan

# Infrastructure & External Assets Plan

What the suite needs from the outside world to go from "all code landed, skip

rows everywhere" to "publishing real numbers." Each item names the skip gate

it unblocks (the code already exits 3 with these exact reasons today).

## The split that decides everything: control plane vs measurement plane

The suite separates **machine-independent** metrics (tokens, bytes on wire,

tool calls, API round trips, poll counts — reproduce exactly anywhere) from

**machine-dependent** metrics (latency, RSS — valid only within a declared

runner class, ADR-0007). That split answers the Cloudflare question:

**Cloudflare is the control/publishing plane, not the measurement plane.**

- Workers + D1 + R2 (already built: ingest, scorecard/trends/baseline-book

  endpoints, webhook ground truth) — deploy and use. ✓

- Cloudflare Containers/Sandboxes are **not suitable for latency lanes**:

  shared multi-tenant CPU with no governor/pinning control (the noise floor

  would swallow most targets), instance sizes far below the 32-core/128 GB

  monorepo tier, small ephemeral disk vs ~200 GB scratch per monorepo run,

  no CAP_NET_ADMIN for netns/tc (netshape lane), and no macOS (mount lane

  is FSKit). Our own runner-discipline doc would flag every row

  environment_suspect.

- Cloudflare Containers **are acceptable** for: scheduled orchestration

  (cron Worker triggering campaigns on real runners), corpus

  verification/import jobs, GitGoodBench CSV materialization, repro-bundle

  verification of machine-independent metrics, and CI-grade smoke of

  skip-row paths. Anything where wall-clock is not the measurand.

Measurement plane = real machines (below). The portable latency claim is the

same-run oak/git ratio per runner class; we need few but quiet boxes, not a

fleet of cloud instances.

## 1. Hardware (measurement plane)

| Runner | Spec | Lanes | Suggested source |

| --- | --- | --- | --- |

| perf-linux-1 | 32-core, 128 GB, 4 TB NVMe, Ubuntu LTS, performance governor, bare metal | core xl/monorepo, Baseline Book campaigns, fixture generation, fleet tiers (timeshared off-hours via runlock) | Hetzner dedicated (AX102-class) or OVH/Latitude bare metal — avoid shared-vCPU clouds |

| netshape-linux-1 | 8–16 core, 64 GB, root/CAP_NET_ADMIN, otherwise idle | netshape lane + local git/oak servers | second small Hetzner box, or timeshare perf box initially (runlock serializes) |

| mac-agent-1 | Apple Silicon, 64 GB (exists — current dev machine class) | agent lane, mount lane (FSKit), mac Baseline Book | already have |

Provisioning checklist (perf box): Ubuntu LTS, `cpupower frequency-set -g

performance`, sudoers rule for `/proc/sys/vm/drop_caches` (cachectl cold

discipline), git + build toolchain + a built git source tree (t/perf

cross-checks; unblocks `git_src_missing`), hyperfine, stress-ng, GNU time,

node, python3.11+, oak binary, `OAK_BENCH_FIXTURE_CACHE` on the NVMe.

Run `scripts/runner_calibration.py` and commit the runner profile; add the

box to `config/runner_classes.json` (config-only).

## 2. GitHub org assets (platform lane)

Unblocks `github_token_missing` / `gh_cli_missing` skip rows.

- Repos in `oakdotspace`: bench-platform-tmp, bench-platform-aa-tmp (A/A

  twin), bench-platform-mq-tmp (merge queue), bench-xl-mirror (~4.5 GiB,

  chunked pushes ≤1.5 GB), bench-pr-race, bench-conflict-corpus, corpus-*

  mirrors. Reset automation = a small script deleting branches/PRs between

  runs (to write).

- Machine accounts: `oakbench-bot` + `oakbench-reviewer` (reviewer exists

  because a bot cannot approve its own PR), fine-grained PATs scoped to the

  bench repos; export as `GITHUB_TOKEN` on runners. GitHub App is the

  upgrade path if secondary limits bind.

- Webhook: point org webhooks at the Worker `/webhook/github` with

  `GITHUB_WEBHOOK_SECRET` (already implemented + HMAC-verified).

- Budget reality: secondary content limits ≈500 content ops/hr — the

  budgeter already paces and records `race.pacing.injected`; campaigns for

  integration_race_n10 at n=30 must spread over ≥2 h by design.

## 3. Cloudflare (control plane) — deploy steps

1. `wrangler d1 execute <db> --file schema.sql` then `--file schema_v2.sql`

   (additive, verified against sqlite locally).

2. `wrangler deploy` the worker; set secrets: ingest bearer token,

   `GITHUB_WEBHOOK_SECRET`.

3. R2 buckets/prefixes: `fixtures/` (master tarballs), `corpus/`,

   `raw-rows/` (JSONL linked from baseline-book manifests by sha256),

   `bundles/` (repro bundles). ~60–80 GB.

4. Wire `scripts/` upload: results push with the bearer token (existing

   upload path in worker.js).

## 4. External binaries & datasets

| Asset | Where | Unblocks |

| --- | --- | --- |

| hyperfine | brew/apt (MIT) | crosscheck instrument rows (skip today) |

| stress-ng | brew/apt (GPL-2, external) | load tiers + live falsification test (currently the 1 skipped test) |

| GNU time + built git tree | apt / `make` from git.git | tperf_runner run mode |

| tiktoken | `pip install tiktoken` (optional import) | token calibration `--method tiktoken` |

| jj | brew (pilot subject) | subject-kind pilot golden samples |

| GitGoodBench Lite CSV | github.com/JetBrains-Research (Apache-2.0) | `dataset_missing` skip in import_gitgoodbench |

| GitGoodBench Lite repos (~100) | mirror into corpus org FIRST (pointer datasets rot) | `mirror_missing:*` skips |

| corpus-nextjs/react/rails/linux/gltf | clone at frozen tags → org mirrors + R2 tarballs | corpus scale lanes |

## 5. oak.space product dependencies (file as handoff, not purchasable)

The skip rows are the feature requests: self-hostable oak server

(`OAK_BENCH_LOCAL_SERVER_CMD` — blocks netshape oak side), `oak import`

bulk history ingestion (blocks history mirrors), ≥128 concurrent

mounts/pushes + branch TTL/cleanup (fleet tiers), 100k-branch + 20 GB repo

support, webhooks.

## 6. Budgets

- Model API: ~$1.5–3k per quarterly full real-agent campaign; $200–500/mo

  spot; mock adapter remains CI-only.

- Hardware: one Hetzner AX102-class ≈ $100–130/mo; small netshape box

  ≈ $40–60/mo.

- Cloudflare: existing account suffices (Workers paid plan + R2 storage).

## 7. Execution order (each step unblocks the next)

1. **Today, this Mac, $0**: install hyperfine/stress-ng/jj/tiktoken; run

   `baseline_campaign.py run` (smoke profile) → first mac-class Baseline

   Book → set `baselines/git/CURRENT` → `devloop --scorecard` shows the

   first real red/green board. Generate conflict corpus + quick xl

   fixtures. Token calibration campaign with tiktoken samples.

2. **Cloudflare deploy** (hours): schema_v2 + worker + secrets + R2; start

   ingesting devloop/baseline rows.

3. **GitHub org setup** (a day): repos, machine accounts, PATs, webhook;

   first credentialed platform run: pr_single_anatomy n=5 both drivers,

   then integration_race_n1.

4. **Order perf-linux-1** (lead time days): provision per checklist; Linux

   runner class + Linux Baseline Book at n=50; xl/monorepo fixture

   generation; t/perf cross-check; fleet tiers 16→128 off-hours.

5. **Datasets** (parallel): GitGoodBench mirrors, corpus mirrors at frozen

   tags, R2 tarballs.

6. **netshape** (after #4): either CAP_NET_ADMIN on the perf box

   (timeshared, runlock) or the dedicated small box; git side fully

   measurable; oak side stays skip-rows until oakd ships (the pressure).

7. **Agent campaigns** (after #1–#3, budget approval): real-agent lane with

   `--vcs-shim` on the Mac; platform_integrate_10 once the org exists.