Log in
benchmarksPublic

Commit 2e081b95

· Browse files at this commit

2e081b95a118

Merged on mrmrs-eec7ec
mrmrs · 1 month ago · 13 files ~13 modified
M docs/qa-invariants-2026-09-12.md
@@ -1,24 +1,33 @@
11 # Benchmark evidence and admission invariants — 2026-09-12 UTC
22
3-This correction follows the FIX review of `mrmrs-eec7ec` at `2735d7f5558e`.
3+This correction follows the FIX review of `mrmrs-eec7ec` at `fcd8a785ee1a`.
44 Main remains `b297e8a25d08`. It is pending independent review, not integrated.
55 The worker is `/Users/mrmrs/o/worker-mrmrs-20260911T191215-bench-regate-fixes/bench`;
6-new evidence uses the `qa4-` prefix in its parent, preserving earlier reports.
6+new evidence uses the `qa5-` prefix in its parent, preserving earlier reports.
77
88 ## Publication and evidence
99
10-Every workflow summary must report successful, measured correctness and a
11-successful non-skipped operation before publication. This includes single
12-subjects and non-trusted scenarios such as `bugfix_test_loop`. The separate
13-trusted config/asset policy adds committed-tree evidence requirements; it does
14-not replace the general correctness gate.
10+Scripted workflow summaries must report successful, measured correctness.
11+`workflow.skipped` summaries cannot disappear from the publication check or
12+trusted subject pairing. Agent trials instead retain fully measured failures
13+as success-rate evidence: a failed task is not a broken benchmark. Skipped or
14+unmeasured trials remain ineligible. This includes single subjects and
15+non-trusted scenarios such as `bugfix_test_loop`. Trusted config/asset success
16+comparisons additionally require independent committed-tree evidence. A trusted
17+worktree validation failure establishes task failure without a successful HEAD
18+read; it cannot claim persisted completion.
1519
1620 Reader classes describe evidence actually obtained. A failed worktree check or
1721 unavailable reader cannot claim committed-tree verification. A successful Git
1822 HEAD read made by the same binary bytes as the Git subject is labeled
1923 `subject_binary_committed_tree`, including the default PATH configuration.
20-Non-trusted workflows with narrower committed-payload checks are labeled as
21-such. Different executable hashes identify distinct selected instruments; they
24+Executed failed HEAD checks carry an attempt class and source, never a verified
25+committed-tree class. Non-trusted agent checks use `tests_and_subject_status`,
26+or `subject_status_only` when the required check was unavailable.
27+`repository_metadata` is reserved for the scripted lane that actually reads
28+repository metadata. Non-trusted workflows with narrower committed-payload
29+checks are labeled as such. Resolved instrument identity is passed through the
30+oracle, avoiding duplicate resolution and binary hashing. Different executable hashes identify distinct selected instruments; they
2231 do not prove independent implementation or certify the operator's tool choice.
2332 The workflow oracle is version `2026-09-11.2`, with a literal regression pin.
2433 Unavailable check pass/fail counts are null rather than fabricated zeroes.
@@ -32,10 +41,14 @@
3241
3342 Cache permissions discourage accidental writes but are not a security boundary
3443 against a process with the same UID. The runner validates the cache before each
35-copy, verifies the copied bytes before agent launch, and checks the cache again
44+copy, stages and verifies the copied bytes before journaling a trial start, and
45+checks the cache again
3646 after the trial. A detected drift invalidates the affected measurement, retains
3747 the rejected row, stops admission, and excludes the invalidated attempt from
38-complete paired comparisons. Trial copies remain writable for task execution.
48+complete paired comparisons. Invalidations are visible in reports and CLI
49+diagnostics. Copy faults with an intact cache leave the trial pending. Cache
50+files lose write bits, while directories retain owner write access for cleanup
51+and rebuilds. Trial copies remain writable for task execution.
3952 Resume binds the cache and its mutable record to the digests already observed
4053 in trial journal events. Rewriting both cache and record cannot replace that
4154 baseline. Synthetic uncertain torn starts do not claim an observed fixture;
@@ -64,16 +77,26 @@
6477
6578 ## Verification
6679
67-The final full instrument suite passed: 1,092 tests, three skips, in 130.519
80+The final full instrument suite passed: 1,102 tests, three skips, in 197.279
6881 seconds (`python3 -m unittest discover -s tests` with the shared measurement
69-lock configured). Complete output is `../qa4-tests.log`.
82+lock configured). Complete output is `../qa5-tests.log`.
7083
71-The identical-binary core devloop passed with two repetitions and two A/A
72-repetitions under the shared measurement lock, with default thresholds and
73-the pinned Oak 0.103.0 binary in `../subject-identities.json`. It reports zero
74-baseline regressions, the known EOF patch defect as `PREEXISTING`, and
75-incompatible remote operations as informational transport exclusions. Evidence:
76-`../qa4-devloop.log` and `results/qa4-core/20260912T012121Z/`.
84+The identical-binary core devloop used the shared measurement lock, default
85+thresholds and pinned Oak 0.103.0 in `../subject-identities.json`. The required
86+two-repetition/two-A/A smoke returned REGRESSED (exit 1) on one latency check:
87+`few_large_binaries/snapshot.initial`, 24.3 to 43.3 ms (+78.1%) against a 65.6%
88+effective threshold. Correctness classification remained correct, including
89+optional binary proof and the pre-existing EOF patch defect. This result is
90+preserved in `../qa5-devloop.log` and `results/qa5-core/20260912T065948Z/`.
91+
92+One follow-up with eight repetitions and eight A/A repetitions passed (exit 0),
93+without code or threshold changes. Its A/A p95 spread was 30.9%, with a 61.7%
94+effective threshold. Evidence: `../qa5-devloop-8.log` and
95+`results/qa5-core-8/20260912T070041Z/`. Both runs are reported; these noisy
96+instrument checks establish no latency advantage. No remote was configured,
97+so network operations remain unmeasured. The Low transport diagnostic finding
98+remains deferred: messages use the subject-kind heuristic and repeat per
99+operation; `devloop.py` is unchanged in this correction.
77100
78101 Producer-level regression tests exercise failed and repaired bugfix summaries,
79102 cache corruption before admission, interrupted generation, cache poisoning by
@@ -81,3 +104,9 @@
81104 and a cache-plus-record rewrite rejected against the original journal digest.
82105 Environment boundary tests exercise an actual mount subprocess and the
83106 interleaved/external-timer execution APIs under poisoned environment settings.
107+
108+This round additionally exercises real whole-workflow skip rows, measured agent
109+failures (including trusted pre-HEAD failures), cache copy faults followed by
110+successful resume, ordinary deletion/rebuild, missing/corrupt cache records
111+during a real mock-agent trial, and instrument resolution/hash call counts.
112+Linux chmod capability absence is simulated; no native Linux run was performed.