benchmarksPublic
benchmarksPublic
Commit 1c9b819a
M
CONTEXT.md
@@ -23,6 +23,8 @@
2323
| **Measurement source** | Per-metric-group string saying how values were obtained (adapter stream, transcript classification, filesystem probe). Accompanies the null-honesty contract. |2424
| **Baseline** | The comparison anchor: Git baseline (`git`), Oak baseline (`oak_main` if present, else `oak_installed`). The release-blocking comparison is target (`oak_local`) vs Oak baseline. |2525
| **Efficiency gate** | A CI regression threshold on exact metrics (tool calls, output bytes) applied to the Oak-baseline comparison only (ADR-0004). |26+
| **A/A control** | The Oak baseline binary run under a second subject name (`oak_installed_aa`), interleaved with the other subjects; its difference from `oak_installed` is pure noise and sets each cell's floor (ADR-0011). |27+
| **Page cache state** | `page_cache_state` on every timed core row: `warm` (no purge), `purged` (verified OS purge immediately before the command), `unknown` (purge failed). |2628
2729
## Architecture vocabulary2830
M
README.md
Loading diff…
M
config/git_modes.json
Loading diff…
M
config/subjects.toml
Loading diff…
A
docs/adr/0011-honest-measurement-contract.md
Loading diff…
M
docs/independent-workflow-readers.md
Loading diff…
M
docs/runner-discipline.md
Loading diff…
M
docs/statistical-methodology.md
Loading diff…
M
docs/tuned-git-baselines.md
Loading diff…
M
scripts/agent_metrics_schema.py
Loading diff…
M
scripts/agent_workflow.py
Loading diff…
M
scripts/baseline_campaign.py
Loading diff…
M
scripts/bench.py
Loading diff…
M
scripts/conflict_resolution_lane.py
Loading diff…
M
scripts/devloop.py
Loading diff…
M
scripts/oakbench/cachectl.py
Loading diff…
M
scripts/oakbench/campaign_report.py
Loading diff…
A
scripts/oakbench/cellstats.py
Loading diff…
M
scripts/oakbench/execution.py
Loading diff…
M
scripts/oakbench/oak_sqlite_reader.py
Loading diff…
M
scripts/oakbench/populations.py
Loading diff…
M
scripts/oakbench/subjects.py
Loading diff…
M
scripts/oakbench/workflow_oracles.py
Loading diff…
M
scripts/parallel_contention.py
Loading diff…
M
scripts/task_loop.py
Loading diff…
M
scripts/workflow_ab.py
Loading diff…
M
tests/test_agent_campaign_cli.py
Loading diff…
M
tests/test_agent_outcome_attribution.py
Loading diff…
M
tests/test_baseline_campaign.py
Loading diff…
M
tests/test_campaign_report.py
Loading diff…
A
tests/test_cellstats.py
Loading diff…
M
tests/test_core_oracle_sequence.py
Loading diff…
A
tests/test_honest_measurement.py
Loading diff…
M
tests/test_large_binary_lfs.py
Loading diff…
M
tests/test_oak_sqlite_reader.py
Loading diff…
M
tests/test_workflow_oracles.py
Loading diff…