/oak
Log in
benchmarks
docs/qa-follow-up-2026-09-11.md 97 lines · 5.6 KB

Benchmark QA follow-up — 2026-09-11

Historical snapshot: review rejected 2735d7f5558e. In particular, the claim below that non-trusted workflows retained other correctness publication checks was wrong; failed workflow summaries could publish. Cache validation also admitted invalid trials. See the next correction for the replacement invariants and new producer-level evidence.

This review corrects defects found in mrmrs-eec7ec at 2310099b696d. Main remains b297e8a25d08. The branch stays open for independent re-review; this document is not a merge receipt.

QA worker: /Users/mrmrs/o/worker-mrmrs-20260911T191215-bench-regate-fixes/bench. Generated logs and subject identities live in the worker parent or ignored results directories. The original review remains in the prior author's QA-FEEDBACK.md and the worker-mrmrs-20260911T184303-bench-regate logs.

Acceptance boundaries

  • Both workflow producers must write contract-valid skip rows when a correctness instrument is unavailable. A skipped required check is unmeasured; an independently failed check still identifies the failure.
  • The scripted Git subject must not select itself as the trusted HEAD reader. The correctness instrument has its own configuration and provenance.
  • Publication must not compare source labels as byte-identical proof. The stricter committed-tree check applies to trusted config/asset workflows. Git/Oak comparisons for those workflows remain blocked until independent Oak committed-tree evidence exists. An Oak-produced export does not provide that independence. Other workflows, including bugfix_test_loop, retain their existing publication checks rather than acquiring this blanket block.
  • Optional patch proof may produce a diagnostic note without blocking devloop. Missing required proof remains UNMEASURED; missing target operations and independently observed failures remain regressions. Unmeasured baseline instruments cannot invalidate a fully proven target operation.
  • Agent rows carry observed runner identity and explicit cache/calibration status. Absent cache control and absent token calibration must not be relabeled as known measurements merely to make a claim eligible.
  • Definite worktree mutation during a diff carries its own tree-comparison evidence and no contradictory instrument-skip reason.
  • Campaigns reuse fixtures outside trial worktrees, verify their persisted tree identity on resume, and reject corrupted cached fixtures. Trial progress goes to stderr so run/resume stdout remains JSON. Provenance drift names the changed key, including the measurement lock path.

The workflow oracle version is 2026-09-11.1. Claim-eligibility tests now use produced agent rows: observed runner and oracle identity are present, while unknown cache state blocks latency claims and null calibration identity blocks token claims. Emitting those fields is not certification of their values.

Verification

Final instrument suite: python3 -m unittest discover -s tests completed successfully with 1,072 tests, three skips, in 136.731 seconds under the shared measurement lock configuration. Full output: ../qa-tests-final.log.

Live producer checks:

  • Missing Git instrument: scripted and standalone mock-agent CLI runs exit without tracebacks and persist returncode: 77, skipped: true and a reason. All 15 scripted rows and the agent row pass validate_rows.
  • Injected Oak export failure: both CLI runs likewise persist contract-valid skips (13 scripted rows and one agent row), with null correctness evidence.
  • Real bugfix_test_loop: Git and Oak both succeed; the 30 rows pass the row contract and the workflow evidence gate. The publication-entrypoint regression tests additionally exercise these real producer labels.

Evidence: ../qa-cli-summary.json, ../qa-row-contract-summary.json, ../qa-export-failure-summary.json and the associated CLI logs.

The final identical-binary default-core devloop ran two repetitions with two A/A repetitions, unmodified default thresholds and the pinned Oak 0.103.0 binary recorded in ../subject-identities.json. It returned PASS, with zero baseline regressions, informational optional binary patch gaps and the known EOF defect classified PREEXISTING. See ../qa-devloop-final.log and results/qa-regate-core-final/20260911T192415Z/. This verifies the verdict's reachability; two repetitions do not establish performance superiority.

The live check found an additional first-use bias: Oak 0.103.0 performs a synchronous update check after its first successful command in a fresh HOME. Whichever subject ran first paid that external network cost. The benchmark environment now forces OAK_NO_UPDATE_CHECK=1 (environment isolation version 3). Cold-home updater UX is not part of these engine comparisons.

The A/A floor now uses 100 * (max(a, b) / min(a, b) - 1) for positive medians, so reversing subject order cannot shrink the same observed spread. Ordinary candidate-versus-baseline regression percentages remain directional. Two earlier two-repetition runs reported initialization regressions; a five-repetition run passed before the updater fix. All are retained in ../qa-devloop*.log; they are diagnostic evidence, not a performance claim.

Remaining qualification work

No provider campaign or human study is performed in this correction. Mock execution validates instrumentation only. Independent Oak durability evidence, controlled agent-cache measurements and calibrated token estimates remain qualification work. Product defect fb-521 remains a product issue. The Oakmark preservation branch is unchanged.