/oak
Log in
benchmarks
docs/publish-checklist.md 107 lines · 4.8 KB

Publish Checklist

Every public Oak-vs-Git claim walks this list. An unchecked box is a blocked publish, not a footnote. Complements docs/benchmark-coverage.md (what may be cited) and docs/statistical-methodology.md (how many runs).

Per-table gates

  • Claim numbers are reproduced mechanically. For every public number, write a scripts/public_claim_gate.py claim spec that filters the cited raw JSONL rows, recomputes the stated value/ratio/delta, and fails when drift exceeds the stated tolerance. publish_gate.py says rows are publishable; public_claim_gate.py says the specific number in the copy is still true.
  • Tuned git present. Wide-tree status/diff tables include git_untracked_cache AND git_fsmonitor rows (--git-modes untracked_cache,fsmonitor; fsmonitor--daemon verified healthy on the runner — it is, on this darwin host, 2026-06-11). Stock git alone overstates oak wins.
  • Oak source provenance clean. Every published Oak subject row carries source.oak_hash and a clean source.oak_status; dirty or unknown Oak source state blocks publication.
  • Core sample count sufficient. Public core tables use at least n=30 per subject/scenario/operation group (standard, large, and micro profiles). Smoke rows catch regressions only; they do not support public speed claims.
  • Randomization recorded. Public core rows state randomized_subject_order; when true, rows include the per-run subject_order used to reduce order/cache bias.
  • Public rows are failure-free. Failed measured rows block publication; failures must be fixed, quarantined as explicit skip rows, or reported in a non-published investigation table.
  • One benchmark track per file. agent-default, core-equivalent, and other tracks are never mixed in the same published JSONL file.
  • Calibration cited. Any cross-subject token delta names its char/4 calibration factors from docs/token-calibration-factors.md (recaptured at scale for the drop, not reused from the working table) or uses exact tokenizer counts. Log/stat-family deltas are never published raw.
  • Envelope constants current. token_calibration.py --envelope drift vs oakbench/tokens.py constants checked; rows state their constants via ENVELOPE_NOTE.
  • No transport mixing. No cell aggregates or deltas local-file vs network rows (remote.* vs remote.net.*; sync workflows: git local-bare vs oak server — tokens/calls only, never latency). Symmetric-network lanes (task loop, mount-vs-clone) state both server identities.
  • No instruction-level mixing. Real-agent tables report zero-shot and cheat-sheet separately, n stated per level (n≥10 for success-rate claims).
  • Nulls are unmeasured. No blank-as-zero: every null renders as "unmeasured"; every returncode-77 skip appears with its skip_reason or the row set is stated as filtered.
  • Tails are honest. p95 only with n≥20, p99 only with n≥100 (benchmark_stats.py enforces blank cells — do not fill them in by hand).
  • Recall beside bytes. Output-byte/token wins cite information_recall and pipe_compatible_unified from the same run — compact-but-lossy is a regression, not a win.
  • Noise floor cleared. Every claimed delta exceeds the run's printed detection limit (devloop A/A floor or CI from benchmark_stats.py).

Per-drop gates

  • Raw JSONL for every cited table is archived and linked.
  • Hardware, OS, filesystem, git/oak versions, repetitions, randomization stated once per drop.
  • Scripted vs real-agent provenance labeled per table; real-agent rows use provider-reported tokens (token_metrics.total_tokens_reported).
  • Track labeled per table (agent-default vs core-equivalent).
  • Known oak losses present, not filtered: the suite's credibility is that it reports both directions (e.g. pipe-compat 0%, divergence-recovery data loss, wide-tree mount findings — whatever the current run shows).
  • Fixture identity stated (name, version, seed) so third parties can regenerate (scripts/make_large_fixture.py for the mirrors).

Claim gate spec sketch

Run:

python3 scripts/public_claim_gate.py --claims claim.json raw/*.jsonl

Minimal claim file:

{
  "schema_version": 1,
  "claims": [
    {
      "id": "branch_fleet_n100_wall",
      "row_filter": {
        "scenario": "branch_fleet_n100",
        "operation": "fleet.total",
        "platform": "oak"
      },
      "metric": "elapsed_ms",
      "statistic": "median",
      "guards": {
        "fields_true": ["json_oracle_passed"],
        "fields_zero": ["cleanup_failure_count"]
      },
      "expected": {
        "field": "value",
        "value": 122000,
        "tolerance_pct": 5
      }
    }
  ]
}