Continuous Benchmarking Plan

# Continuous Benchmarking Plan

The useful benchmark loop has three layers.

## One-Shot AWS Runs

`scripts/aws_bench.sh` provisions an ephemeral EC2 instance, runs `bench.py`

on it, downloads `results/` to `results/aws/<run-id>/`, and destroys

everything it created (instance, security group, key pair) — including on

failure, via an exit trap.

```bash

aws login                                       # refresh credentials first

scripts/aws_bench.sh                            # smoke profile, c6id.2xlarge

scripts/aws_bench.sh --profile standard

scripts/aws_bench.sh --profile large -- --skip-diff

scripts/aws_bench.sh --keep                     # leave the box up for debugging

```

The default instance type (`c6id.2xlarge`) has local NVMe instance-store,

which the setup step mounts at `/mnt/bench` and exports as `TMPDIR` so

fixture generation hits local SSD rather than the EBS root volume. The

benchmark runs detached (`nohup`) on the box and the script polls, so a

dropped SSH connection does not kill a long `standard`/`large` run.

Subjects are the harness defaults: platform `git` plus `oak_installed`

from `curl -fsSL oak.space/install | sh`.

## 1. Per-Push Smoke

Run on every push/change set:

```bash

python3 scripts/bench.py --profile smoke

```

This should complete quickly and catch obvious regressions in:

- Initial snapshot cost.

- Clean and dirty status.

- Dirty diff.

- Commit after text and binary mutations.

- Branch-per-task cycle.

Store `results/latest.jsonl` and `results/latest.summary.md` as build artifacts.

## 2. Nightly Standard

Run once per day on pinned hardware:

```bash

python3 scripts/bench.py --profile standard

```

This is the better regression signal because it includes many-file and

large-file shapes where VCS internals matter.

## 3. Dedicated Large-File Runner

Run weekly or manually:

```bash

python3 scripts/bench.py --profile large --skip-diff

```

Use a machine with fast local SSD and plenty of free disk. Large diffs are often

less interesting than init/snapshot/status for binary-heavy repos, so

`--skip-diff` is usually the right default.

## Cloudflare Fit

Cloudflare Workers are not the right place to execute the heavy benchmarks:

they do not provide the persistent filesystem and CPU profile we need.

Cloudflare is a good fit for coordination and reporting:

- **R2**: raw JSONL result storage.

- **D1**: indexed medians/deltas by commit, profile, scenario, operation.

- **Workers**: receive uploaded result files and update D1.

- **Pages**: dashboard showing Oak-vs-Git and Oak-vs-previous deltas.

Recommended runner setup:

- GitHub Actions, Buildkite, Depot, Fly machines, or dedicated bare-metal

  runners execute `scripts/bench.py`.

- A post-step uploads JSONL to R2 or directly to a Worker endpoint.

- Dashboard reads D1 for trend charts and links to raw artifacts in R2.

## Subject Strategy

Always record at least:

- `git`: platform Git.

- `oak_main` or `oak_installed`: previous stable Oak.

- `oak_local`: current change set.

For exact main-vs-local comparisons, build a clean-main Oak binary into a stable

path and enable the `oak_main` subject in `config/subjects.toml`.

## Metrics To Trend

Use median and p90 per scenario/operation:

- `elapsed_ms`

- average tokens per operation or agent run

- average tool calls per operation or agent run

- failure rate

- fixture shape metadata

- Oak-vs-Git latency, token, and tool-call deltas

- Oak-local-vs-Oak-main latency, token, and tool-call deltas

The dashboard should sort by regression magnitude first. A small text-repo

regression is useful, but a large-file or many-file regression should page

louder because those are Oak's core claims.