Benchmark evidence and admission invariants — 2026-09-12 UTC

This correction follows the FIX review of mrmrs-eec7ec at fcd8a785ee1a. Main remains b297e8a25d08. It is pending independent review, not integrated. The worker is /Users/mrmrs/o/worker-mrmrs-20260911T191215-bench-regate-fixes/bench; new evidence uses the qa5- prefix in its parent, preserving earlier reports.

Publication and evidence

Scripted workflow summaries must report successful, measured correctness. workflow.skipped summaries cannot disappear from the publication check or trusted subject pairing. Agent trials instead retain fully measured failures as success-rate evidence: a failed task is not a broken benchmark. Skipped or unmeasured trials remain ineligible. This includes single subjects and non-trusted scenarios such as bugfix_test_loop. Trusted config/asset success comparisons additionally require independent committed-tree evidence. A trusted worktree validation failure establishes task failure without a successful HEAD read; it cannot claim persisted completion.

Reader classes describe evidence actually obtained. A failed worktree check or unavailable reader cannot claim committed-tree verification. A successful Git HEAD read made by the same binary bytes as the Git subject is labeled subject_binary_committed_tree, including the default PATH configuration. Executed failed HEAD checks carry an attempt class and source, never a verified committed-tree class. Non-trusted agent checks use tests_and_subject_status, or subject_status_only when the required check was unavailable. repository_metadata is reserved for the scripted lane that actually reads repository metadata. Non-trusted workflows with narrower committed-payload checks are labeled as such. Resolved instrument identity is passed through the oracle, avoiding duplicate resolution and binary hashing. Different executable hashes identify distinct selected instruments; they do not prove independent implementation or certify the operator's tool choice. The workflow oracle is version 2026-09-11.2, with a literal regression pin. Unavailable check pass/fail counts are null rather than fabricated zeroes.

Campaign admission and fixture integrity

Fixture preparation and integrity validation precede admission. A corrupt or incomplete cache aborts the invocation with pending trials intact. Partial first-time generation is preserved for investigation; it cannot burn pending trials as task errors.

Cache permissions discourage accidental writes but are not a security boundary against a process with the same UID. The runner validates the cache before each copy, stages and verifies the copied bytes before journaling a trial start, and checks the cache again after the trial. A detected drift invalidates the affected measurement, retains the rejected row, stops admission, and excludes the invalidated attempt from complete paired comparisons. Invalidations are visible in reports and CLI diagnostics. Copy faults with an intact cache leave the trial pending. Cache files lose write bits, while directories retain owner write access for cleanup and rebuilds. Trial copies remain writable for task execution. Resume binds the cache and its mutable record to the digests already observed in trial journal events. Rewriting both cache and record cannot replace that baseline. Synthetic uncertain torn starts do not claim an observed fixture; ordinary trial events require their recorded binding. These checks do not establish OS containment or protect the trusted runner itself from a fully hostile host.

Shared measurement policy

Both dedicated A/A and embedded-null noise estimates use the same symmetric ratio-spread helper. Candidate regression percentages remain directional. Git local-file and Oak network remote operations are excluded from cross-kind guardrails with an explicit transport note, not treated as equivalent tasks. Missing comparable target operations still regress. Lost or failed optional patch proof remains visible as informational evidence.

Patch tree hashing and mutation classification now belong to the shared output oracle, with one manifest canonicalization. The core lane consumes that policy.

Mount, interleaved command and external-timer processes sanitize Git settings and suppress Oak's updater through the shared environment helper. They retain the caller's HOME for authentication/configuration, so their environment policy is 3-host-home, distinct from the isolated-HOME core policy 3. In particular, mount rows no longer claim the core isolation contract. Host-home configuration remains a qualification limitation.

Verification

The final full instrument suite passed: 1,102 tests, three skips, in 197.279 seconds (python3 -m unittest discover -s tests with the shared measurement lock configured). Complete output is ../qa5-tests.log.

The identical-binary core devloop used the shared measurement lock, default thresholds and pinned Oak 0.103.0 in ../subject-identities.json. The required two-repetition/two-A/A smoke returned REGRESSED (exit 1) on one latency check: few_large_binaries/snapshot.initial, 24.3 to 43.3 ms (+78.1%) against a 65.6% effective threshold. Correctness classification remained correct, including optional binary proof and the pre-existing EOF patch defect. This result is preserved in ../qa5-devloop.log and results/qa5-core/20260912T065948Z/.

One follow-up with eight repetitions and eight A/A repetitions passed (exit 0), without code or threshold changes. Its A/A p95 spread was 30.9%, with a 61.7% effective threshold. Evidence: ../qa5-devloop-8.log and results/qa5-core-8/20260912T070041Z/. Both runs are reported; these noisy instrument checks establish no latency advantage. No remote was configured, so network operations remain unmeasured. The Low transport diagnostic finding remains deferred: messages use the subject-kind heuristic and repeat per operation; devloop.py is unchanged in this correction.

Producer-level regression tests exercise failed and repaired bugfix summaries, cache corruption before admission, interrupted generation, cache poisoning by a real mock-agent trial, copy corruption before a subsequent agent can run, and a cache-plus-record rewrite rejected against the original journal digest. Environment boundary tests exercise an actual mount subprocess and the interleaved/external-timer execution APIs under poisoned environment settings.

This round additionally exercises real whole-workflow skip rows, measured agent failures (including trusted pre-HEAD failures), cache copy faults followed by successful resume, ordinary deletion/rebuild, missing/corrupt cache records during a real mock-agent trial, and instrument resolution/hash call counts. Linux chmod capability absence is simulated; no native Linux run was performed.