Results, Regression Tracking, and Dashboard

# Results, Regression Tracking, and Dashboard

This document defines the reporting contract for Oak benchmark results. The

raw JSONL file is the durable source of truth. D1 tables are derived indexes

for dashboard queries and can always be rebuilt from R2.

## Raw JSONL Storage

Each benchmark run writes one immutable JSONL object. Each non-empty line is one

JSON object for one timed operation or sub-operation.

Required fields currently emitted by `scripts/bench.py`:

| Field | Meaning |

| --- | --- |

| `bench_id` | UTC run identifier, formatted as `YYYYMMDDTHHMMSSZ`. |

| `timestamp_utc` | UTC timestamp for the run. Today this matches `bench_id`. |

| `profile` | Benchmark profile, such as `smoke`, `standard`, or `large`. |

| `scenario` | Fixture shape, such as `tiny_text` or `single_large_binary`. |

| `operation` | Timed operation, such as `status.clean` or `snapshot.dirty`. |

| `run` | Scenario repetition index. |

| `subject` | Logical subject name, such as `git`, `oak_installed`, or `oak_local`. |

| `subject_kind` | Tool family, currently `git` or `oak`. |

| `subject_label` | Human display label for the subject. |

| `subject_versions` | Run-scoped map of subject name to version string. |

| `subject_details` | Run-scoped map with subject role, label, kind, binary path, and version. |

| `benchmark_track` | Comparison track, such as `agent-default` or `core-equivalent`. |

| `command_semantics_version` | Version of the command semantics contract used for the run. |

| `elapsed_ms` | Wall-clock elapsed milliseconds for this operation. |

| `returncode` | Process exit code. Zero is success. |

| `command` | Command or command sequence that was timed. |

| `tool_call_count` | Total agent-visible tool calls represented by this row. |

| `vcs_tool_call_count` | VCS command calls represented by this row. |

| `terminal_tool_call_count` | Terminal command calls represented by this row. |

| `estimated_tokens_total` | Estimated command-plus-output tokens for direct CLI rows. |

| `estimated_tokens_input` | Estimated command/input tokens for direct CLI rows. CLI-perspective naming, kept for compatibility. |

| `estimated_tokens_output` | Estimated admitted stdout/stderr tokens for direct CLI rows. CLI-perspective naming, kept for compatibility. |

| `estimated_tokens_agent_emitted` | Billing direction: text the model emits (the command). Weighted as output tokens. |

| `estimated_tokens_agent_ingested` | Billing direction: text the model reads back (stdout/stderr). Weighted as input tokens. |

| `estimated_cost_weighted_tokens` | `emitted * 5 + ingested * 1`. Use this, not raw totals, when ranking optimizations. |

| `git_mode` | Tuned Git mode for derived subjects (`untracked_cache`, `split_index`, `fsmonitor`), null for stock subjects. |

| `output_stable` | Determinism probe rows only: identical state produced byte-identical output twice. Stability is what makes provider prompt caches hit. |

| `ansi_escape_count_non_tty` | Determinism probe rows only: count of ESC-initiated sequences emitted to a pipe. Non-zero means token waste and parse risk. |

| `raw_output_bytes` | Full stdout plus stderr byte count, independent of admitted-output caps. |

| `stdout_bytes` | Full stdout byte count. |

| `stderr_bytes` | Full stderr byte count. |

| `output_truncated` | True when admitted stdout/stderr text was capped for token estimation. |

| `token_estimate` | Nested estimate detail, including character counts, byte counts, truncation flags, and method. |

| `tool_calls` | Nested tool-call counts for consumers that prefer grouped counters. |

| `host` | Hostname reported by the runner. |

| `platform` | OS/platform string. |

| `machine` | Machine architecture. |

| `python` | Python runtime version used by the harness. |

| `source` | Oak source metadata gathered with Oak commands. |

| `stderr` | Optional truncated stderr when a command fails. |

The dashboard treats operations ending in `.add` or `.commit` as raw

sub-steps. They remain available for debugging, but default comparisons use the

combined canonical operations, for example `snapshot.initial` instead of

`snapshot.initial.add` and `snapshot.initial.commit`.

Future-compatible rows may add fixture shape fields such as `file_count`,

`file_size`, `dirty_count`, `binary`, `fixture_bytes`, `runner_id`, `cpu_model`,

or `disk_model`. Consumers must ignore unknown fields.

End-to-end agent workflow rows should include `token_metrics`, `turn_metrics`,

and `tool_call_metrics` from `scripts/agent_metrics_schema.py` (schema v2).

Reporting code prefers provider-reported totals from

`token_metrics.total_tokens_reported`, then falls back to direct CLI token

estimates, then to normalized transcript characters divided by four. This lets

direct VCS benchmarks and real agent benchmarks share the same dashboard views

while keeping provider billing tokens distinct from proxy estimates.

**Null means unmeasured.** Schema v2 forbids fabricated zeros: a null metric

value means the runner could not observe the signal, and each metric group

carries a `measurement_source`. Dashboards must render null as an explicit

"unmeasured" state — never as zero, never as a pass. Historical v1 rows

(schema_version 1) may contain placeholder zeros in `bytes_hydrated`,

`tool_wait_ms`, `context_truncation_events`, and `parallel_metrics`; treat

those fields as unmeasured for v1 rows.

Turn-level views matter more than per-call views for agent cost:

`turn_metrics.assistant_turns_total` counts model round-trips, and

`cumulative_input_tokens_reported` captures context-residency cost (verbose

output early in a session is re-paid on every later turn). Chart turns and

cumulative input next to tool calls.

## Identity

Benchmark identity is:

```text

bench_id + profile + timestamp_utc + source.oak_hash

```

`bench_id` is unique enough for local files, but cloud storage also records the

SHA-256 of the uploaded JSONL body. If the same `bench_id` is uploaded with

different bytes, both raw objects are retained under distinct hash-prefixed R2

keys and the D1 `uploads` table records each upload.

Scenario and operation identity is:

```text

profile + scenario + operation

```

The benchmark suite should not silently change what an existing scenario name

means. If fixture shape or operation semantics change materially, use a new

scenario or operation name so historical charts stay meaningful.

Subject identity has two layers:

| Layer | Example | Use |

| --- | --- | --- |

| Logical subject | `oak_local` | Regression comparisons and dashboard grouping. |

| Build subject | `oak_local` + `oak 0.95.0` + `source.oak_hash` | Exact result provenance. |

The default comparison subjects are:

- Git baseline: `git`.

- Oak baseline: `oak_main` when present, otherwise `oak_installed`.

- Target: `oak_local`.

Host identity is normalized separately from benchmark identity because timing

comparisons are only trustworthy on matching hardware. The D1 `hosts.host_id`

is derived from host, platform, machine, Python version, and optional runner

metadata. The dashboard should default to comparing rows from the same host or

same pinned runner class.

## R2 Object Naming

Raw JSONL uploads use this key shape:

```text

raw/v1/profile=<profile>/date=<YYYY-MM-DD>/bench_id=<bench_id>/sha256=<first12>/results.jsonl

```

Companion objects should use parallel prefixes:

```text

summaries/v1/profile=<profile>/date=<YYYY-MM-DD>/bench_id=<bench_id>/sha256=<first12>/summary.md

reports/v1/profile=<profile>/date=<YYYY-MM-DD>/bench_id=<bench_id>/sha256=<first12>/regression.md

ingest-errors/v1/profile=<profile>/date=<YYYY-MM-DD>/bench_id=<bench_id>/sha256=<first12>.json

```

Recommended R2 custom metadata:

- `bench_id`

- `profile`

- `timestamp_utc`

- `sha256`

- `row_count`

- `source_oak_hash`

- `host`

- `machine`

R2 is append-only from the dashboard's perspective. Fixes to aggregation logic

should replay raw objects into D1 instead of rewriting the source JSONL.

## D1 Normalized Schema

`cloudflare/schema.sql` defines the first normalized schema:

- `uploads`: one row per raw object accepted by the Worker.

- `hosts`: normalized runner metadata.

- `benchmark_runs`: one row per logical benchmark run.

- `subjects`: run-scoped subject records and version strings.

- `measurements`: one row per JSONL measurement, including token and tool-call

  counters when present.

- `aggregates`: p50, p90, p99, mean, min, max, sample count, failure rate,

  average token count, and average tool-call count by run, subject, scenario,

  and operation.

- `comparisons`: precomputed target-vs-baseline latency, token, tool-call, and

  failure-rate deltas for dashboard sorting.

D1 is allowed to lag R2. The Worker scaffold records upload and run metadata;

a follow-up ingest step can stream rows into `measurements`, compute aggregates,

and populate comparisons.

## Percentiles

Aggregate percentiles are computed per:

```text

bench_id + subject + scenario + operation

```

Only successful measurements (`returncode == 0`) contribute to latency

percentiles. Failed measurements contribute to `failure_count` and

`failure_rate`, but not to p50/p90/p99.

Rules:

- `p50_ms`: median of successful elapsed times.

- `p90_ms`: nearest-rank p90 of successful elapsed times.

- `p99_ms`: nearest-rank p99 of successful elapsed times.

- `avg_tokens`: arithmetic mean of the best available token total per row.

- `avg_tool_calls`: arithmetic mean of total tool calls per row.

- `failure_rate`: `failure_count / total_count` for that grouped operation.

- If a group has no successful samples, p50/p90/p99 are `NULL` and the failure

  rate carries the signal.

For smoke runs with one repetition, p50/p90/p99 will often be identical. The

dashboard should show that honestly rather than smoothing it.

## Regression Thresholds

Lower latency is better. A positive slower percentage means the target is worse

than the baseline:

```text

slower_pct = ((target_ms - baseline_ms) / baseline_ms) * 100

delta_ms = target_ms - baseline_ms

```

Default thresholds:

| Comparison | Threshold |

| --- | --- |

| Target vs Oak baseline | p50 or p90 is at least 10 percent and 5 ms slower. |

| Target vs Git | p50 or p90 is at least 25 percent and 10 ms slower. |

| p99 severe tail | p99 is at least 50 percent and 50 ms slower. |

| Failure rate | Any increase over a zero-failure baseline, or configured pp delta. |

Token and tool-call deltas use the same "lower is better" direction:

```text

higher_pct = ((target_value - baseline_value) / baseline_value) * 100

delta = target_value - baseline_value

```

These are reported as optimization metrics by default. CI can later add hard

thresholds for token or tool-call regressions once the suite has enough

historical variance data.

The local report script exposes configurable thresholds. CI can run:

```bash

python3 scripts/regression_report.py results/latest.jsonl --fail-on-regression

```

Publish/readiness workflows can generate the matching ranked opportunity

backlog from the same result files, including machine-readable ranks:

```bash

python3 scripts/opportunities.py results/latest.jsonl results/latest.mount.jsonl \

  --json results/opportunity-backlog.json

```

Failure-rate handling is intentionally stricter than latency handling. A new

nonzero failure rate should be shown above latency regressions even when the

failure is rare, because failed benchmark operations often mean the comparison

row is no longer semantically valid.

## Dashboard Sorting

The dashboard should default to the latest run per profile and host class, then

sort rows in this order:

1. Failed operations or failure-rate regressions.

2. Target-vs-Oak-baseline regressions above threshold.

3. Target-vs-Git guardrail regressions above threshold.

4. Highest p90 or p99 severity ratio.

5. Largest absolute `delta_ms`.

6. Profile priority: `large`, `standard`, then `smoke`.

7. Scenario priority: many-file and large-binary scenarios before tiny smoke

   scenarios.

8. Operation priority: `snapshot.*`, `status.*`, `diff.*`, `branch.create`,

   then `repo.init`.

Recommended row states:

| State | Meaning |

| --- | --- |

| `fail` | Failure rate increased or no successful samples exist. |

| `regression` | Threshold exceeded against the Oak baseline. |

| `guardrail` | Threshold exceeded against Git. |

| `watch` | Slower than baseline but below threshold. |

| `ok` | At or faster than baseline. |

Dashboard charts should keep raw p50, p90, p99, failure rate, and sample count

visible. For agent-efficiency views, keep raw average tokens, average tool

calls, and their absolute deltas visible too. Percent deltas alone hide small

noisy differences and large absolute regressions.

## Upload API

The Worker scaffold accepts JSONL uploads:

```bash

curl -X POST \

  -H "content-type: application/x-ndjson" \

  -H "authorization: Bearer $UPLOAD_TOKEN" \

  --data-binary @results/latest.jsonl \

  https://<worker>/upload

```

The Worker validates that every non-empty line is JSON, derives the R2 key from

the first row, writes raw JSONL to R2 when `OAK_BENCH_RESULTS` is bound, and

records upload metadata in D1 when `OAK_BENCH_DB` is bound. Without those

bindings it returns a successful stub response that documents what would have

been stored.