Log in
docs/output-correctness-oracles.md 87 lines · 5.3 KB

Exact output and patch correctness

Core probes now use output_oracle_version: exact-paths-v2. Historical substring-based recall is not comparable with this version. Core run metadata also identifies patch_oracle_version: apply-tree-v1, so runs using the new diagnostic sequence remain distinguishable from older campaigns.

information_recall and information_precision use the same parsed metadata path set. Sources are exact_metadata_path_set. Diff bodies, diagnostics on stderr, path suffixes inside longer names and prose no longer recover a path. Ground truth is deduplicated. No known paths still means null recall; no reported paths means null precision and byte efficiency. Truncation remains explicit. Failed commands do not become successful evidence.

The parser supports status, unified diff, diffstat, NUL-delimited Git porcelain, name-only and Oak JSON grammars. The current core information probes select display status, unified diff and diffstat through the command contract; porcelain, name-only and Oak JSON support is not an active core probe selection. Git C-quoted strings and octal UTF-8 bytes are decoded; status paths retain meaningful whitespace and a/ prefixes. Unified headers remove only the protocol's single a/ or b/ prefix. Renames report both endpoints, using quote-aware delimiters; ambiguous display encodings are not guessed. Both diff --git and diff --oak delimit file sections, and hunk content cannot add path identities. Lossy diffstat abbreviations cannot certify full paths; machine-readable output is preferable for unusual names.

Applying the actual full diff

diff.full.inforecall also checks actual output applicability. The original generated fixture supplies the pristine tree; a complete manifest of the subject tree before the diff supplies expected bytes, types, link targets and executable bits. The probe uses the same subprocess capture's raw stdout bytes, not a lossy decode/re-encode. Raw capture happens after the timer stops.

The untimed oracle makes a disposable pristine copy, invokes the installed Git binary as git apply --whitespace=nowarn -, and compares the complete result manifest. It does not repair whitespace, recount malformed hunks, allow unsafe paths or accept fuzzy application. Subject worktree changes during diff fail. The core fixture's root .fixture-ready marker is excluded consistently with copy_fixture; VCS metadata is excluded and ancestor Git discovery is bounded. The original fixture and subject tree are not modified by the oracle.

Evidence fields:

FieldMeaning
patch_apply_okTrue only when application succeeds and the entire resulting tree matches. False means failed evidence; null means unavailable.
patch_tree_matchesExact comparison of paths, content, types, executable bits and link targets.
patch_apply_sourcegit_apply_exact_tree.
patch_apply_returncodeActual application process exit; null when not attempted.
patch_instrument_binary, patch_instrument_sha256, patch_instrument_versionIdentity of the external correctness instrument, distinct from subject identity.
patch_skip_reason, patch_failure_reasonWhy applicability was unmeasured or failed.
patch_application_requiredTrue for text scenarios whose full diff should reconstruct the expected tree.

Copying is bounded to fixtures at most 16 MiB by declared size. Larger fixtures and Git LFS filter scenarios explicitly lack patch evidence. Missing Git is an unmeasured instrument capability, never a pass. Raw output above 16 MiB is unmeasured with patch_output_exceeds_capture_limit. Empty/partial patches and binary notices do not certify the intended changes.

pipe_compatible_unified remains a legacy structural indicator. In particular, a binary notice can make it true even though it contains no applicable binary patch. A true structural field alone is not application proof. Binary-summary scenarios retain their valid summary contract while reporting false applicability; do not call that a data-corruption failure. For text scenarios, failed actual applicability marks the core probe failed.

Devloop gates precision loss, output-oracle version mismatches and loss of previously proven patch application, in addition to recall. Text patch failure also fails without requiring a previously successful baseline. Historical structural results remain readable but cannot prove actual applicability.

Verification and observed gap

tests/test_output_exact.py covers exact names, quoting, renames, Unicode/NUL records, multiple Oak file sections, malformed/truncated/missing diff sections, binary notices, wrong bytes/modes/path sets, unsafe paths, raw invalid UTF-8 and real Git/Oak newline-terminated multifile output. A generated core Git fixture test exercises marker exclusion through the real lane.

The micro core diagnostic exposed a distinct Oak v0.103.0 limitation: its full diff omits the no-final-newline marker for the generated 1024-byte text fixture. Git emits the marker and its patch applies; Oak's patch is rejected by Git apply. Both subjects recover all changed paths exactly. This is patch-output compatibility evidence, not a latency claim or data-loss claim. The synthetic negative control checks missing markers without requiring future Oak versions to retain the defect. Rerun against exact binaries before publishing conclusions.