Log in
benchmarksPublic

Commit 878fc46b

· Browse files at this commit

878fc46b2f31

Merged on mrmrs-250abb
mrmrs · 1 month ago · 4 files ~4 modified
M docs/vcs-conformance.md
@@ -70,8 +70,12 @@
7070 the same failure there is a regression). Rows mixing profiles are UNMEASURED;
7171 - coverage: `coverage.<profile>.expected_skips` lists the skips a recorded
7272 profile may have. Any other skip (a pass or xfail that turned into a skip) is a
73- coverage loss (UNMEASURED). Unrecorded profiles accept capability/gated skips
74- and say so; record the profile once its first full run is reviewed;
73+ coverage loss (UNMEASURED). On an unrecorded profile only behaviours needing a
74+ known filesystem-dependent capability (case sensitivity, Unicode
75+ normalization, non-UTF-8 names) or gated behaviours may skip; any other skip
76+ (e.g. a broken symlink/exec probe) is UNMEASURED. A row without `fs_profile`
77+ is UNMEASURED, and a run with zero measured Oak behaviours is always
78+ UNMEASURED. Record a profile once its first full run is reviewed;
7579 - harness errors, reader-unmeasured legs and missing behaviours are UNMEASURED.
7680 Git rows are the reference arm and never gate.
7781
@@ -86,7 +90,8 @@
8690 binary; regressions make the verdict REGRESSED, unmeasured UNMEASURED, and
8791 FIXED entries are printed as notes.
8892 - `publish_gate.py` check 14: every published Oak row must carry binary identity
89- (`binary_sha256`, or `subject_details[subject].binary_sha256`); for each
93+ (`binary_sha256`, or `subject_details[subject].binary_sha256`); the host's
94+ `fs_profile` must have a recorded `coverage` entry; for each
9095 published Oak binary the gate is evaluated on that binary's conformance rows
9196 only (rows of another build can neither satisfy nor fail it) and must be PASS
9297 with no FIXED entries.