Exact output and patch correctness
Core probes now use output_oracle_version: exact-paths-v2. Historical
substring-based recall is not comparable with this version. Core run metadata
also identifies patch_oracle_version: apply-tree-v1, so runs using the new
diagnostic sequence remain distinguishable from older campaigns.
information_recall and information_precision use the same parsed metadata
path set. Sources are exact_metadata_path_set. Diff bodies, diagnostics on
stderr, path suffixes inside longer names and prose no longer recover a path.
Ground truth is deduplicated. No known paths still means null recall; no
reported paths means null precision and byte efficiency. Truncation remains
explicit. Failed commands do not become successful evidence.
The parser supports status, unified diff, diffstat, NUL-delimited Git
porcelain, name-only and Oak JSON grammars. The current core information probes
select display status, unified diff and diffstat through the command contract;
porcelain, name-only and Oak JSON support is not an active core probe selection.
Git C-quoted strings and octal UTF-8 bytes are decoded;
status paths retain meaningful whitespace and a/ prefixes. Unified headers
remove only the protocol's single a/ or b/ prefix. Renames report both
endpoints, using quote-aware delimiters; ambiguous display encodings are not
guessed. Both diff --git and diff --oak delimit file sections, and hunk
content cannot add path identities. Lossy diffstat abbreviations cannot certify
full paths; machine-readable output is preferable for unusual names.
Applying the actual full diff
diff.full.inforecall also checks actual output applicability. The original
generated fixture supplies the pristine tree; a complete manifest of the
subject tree before the diff supplies expected bytes, types, link targets and
executable bits. The probe uses the same subprocess capture's raw stdout bytes,
not a lossy decode/re-encode. Raw capture happens after the timer stops.
The untimed oracle makes a disposable pristine copy, invokes the installed Git
binary as git apply --whitespace=nowarn -, and compares the complete result
manifest. It does not repair whitespace, recount malformed hunks, allow unsafe
paths or accept fuzzy application. Subject worktree changes during diff fail.
The core fixture's root .fixture-ready marker is excluded consistently with
copy_fixture; VCS metadata is excluded and ancestor Git discovery is bounded.
The original fixture and subject tree are not modified by the oracle.
Evidence fields:
| Field | Meaning |
|---|---|
patch_apply_ok | True only when application succeeds and the entire resulting tree matches. False means failed evidence; null means unavailable. |
patch_tree_matches | Exact comparison of paths, content, types, executable bits and link targets. |
patch_apply_source | git_apply_exact_tree. |
patch_apply_returncode | Actual application process exit; null when not attempted. |
patch_instrument_binary, patch_instrument_sha256, patch_instrument_version | Identity of the external correctness instrument, distinct from subject identity. |
patch_skip_reason, patch_failure_reason | Why applicability was unmeasured or failed. |
patch_application_required | True for text scenarios whose full diff should reconstruct the expected tree. |
Copying is bounded to fixtures at most 16 MiB by declared size. Larger fixtures
and Git LFS filter scenarios explicitly lack patch evidence. Missing Git is an
unmeasured instrument capability, never a pass. Raw output above 16 MiB is
unmeasured with patch_output_exceeds_capture_limit. Empty/partial patches and binary notices do
not certify the intended changes.
pipe_compatible_unified remains a legacy structural indicator. In
particular, a binary notice can make it true even though it contains no
applicable binary patch. A true structural field alone is not application
proof. Binary-summary scenarios retain their valid summary contract while
reporting false applicability; do not call that a data-corruption failure.
For text scenarios, failed actual applicability marks the core probe failed.
Devloop gates precision loss, output-oracle version mismatches and loss of previously proven patch application, in addition to recall. Text patch failure also fails without requiring a previously successful baseline. Historical structural results remain readable but cannot prove actual applicability.
Verification and observed gap
tests/test_output_exact.py covers exact names, quoting, renames, Unicode/NUL
records, multiple Oak file sections, malformed/truncated/missing diff sections,
binary notices, wrong bytes/modes/path sets, unsafe paths, raw invalid UTF-8 and
real Git/Oak newline-terminated multifile output. A generated core Git fixture
test exercises marker exclusion through the real lane.
The micro core diagnostic exposed a distinct Oak v0.103.0 limitation: its full diff omits the no-final-newline marker for the generated 1024-byte text fixture. Git emits the marker and its patch applies; Oak's patch is rejected by Git apply. Both subjects recover all changed paths exactly. This is patch-output compatibility evidence, not a latency claim or data-loss claim. The synthetic negative control checks missing markers without requiring future Oak versions to retain the defect. Rerun against exact binaries before publishing conclusions.