VCS behavioral conformance and upstream review

# VCS behavioral conformance and upstream review

`scripts/vcs_conformance.py` runs five independently authored correctness

diagnostics against local Git and Oak binaries. It does not measure human or

agent success, tokens, latency, or throughput. Its records deliberately use

`conformance_schema_version`, not the benchmark row schema, and

`evidence_class: diagnostic_conformance`. **Do not ingest these records into

performance reports, devloop baselines, or public comparison tables.** They are

development prerequisites and expose unsupported behavior without concealing it.

Run from an isolated worker checkout:

```sh

python3 scripts/vcs_conformance.py --workdir workdirs/conformance

python3 scripts/vcs_conformance.py --workdir workdirs/conformance --subjects git

python3 -m unittest discover -s tests -p test_vcs_conformance.py

```

Optional `--git-bin`, `--oak-bin`, and comma-separated `--scenarios` select exact

binaries and scenarios. stdout is diagnostic JSONL; archive it outside source

control. Records include binary path, SHA-256, reported version, command argv,

exit codes, and captured output. Each scenario has a fresh disposable repository

and isolated home. No remote commands run. The CLI exits 1 on any failed

scenario and 0 otherwise; individual missing-capability records use code 77

and null integrity. All-skipped output is not a successful conformance result.

Do not run these subprocess-heavy checks during latency campaigns.

## Implemented scenarios and their exact contracts

| Scenario | Required behavior | Independent oracle |

| --- | --- | --- |

| `path_roundtrip_v1` | Checkpoint and restore space, tab, newline, quote, leading-dash, Unicode, and nested paths. | Delete every fixture file before restore; compare exact path set, SHA-256, size, type, executable bit, HEAD and branch. |

| `byte_roundtrip_v1` | Preserve empty files, all byte values, mixed CR/LF bytes and absent final LF. | Same destructive roundtrip oracle; no whitespace normalization. |

| `mode_symlink_roundtrip_v1` | Preserve executable bit, regular file, relative link and dangling link. | `lstat` and link target inspection; no symlink traversal; full roundtrip and identity check. |

| `scoped_restore_preservation_v1` | Restore one tracked path while leaving unrelated dirty and untracked payloads intact. | Full-tree expected-state comparison plus unchanged logical repository identity. |

| `failed_restore_preservation_v1` | Invalid source must fail normally and preserve all user files and logical identity. | Positive nonzero exit required; success, signal termination, any changed/extra/missing path, or identity change fails. |

All scenario and fixture identities are version 1. Git identity includes HEAD,

branch and exact index entries (NUL-delimited path/mode/blob/stage projection).

Oak identity includes HEAD and branch from JSON status. This is logical-state

preservation, not proof of byte-for-byte metadata atomicity, transaction-log

durability, full-history preservation, or crash recovery. The tree oracle

excludes root `.git`/`.oak`, empty directories, timestamps, ACLs/xattrs and

non-executable mode bits. Those require separate contracts. The manifest

records symlinks themselves, including broken links, without following them.

The existing shared environment helper is reused, with inherited Git plumbing

variables removed to prevent commands being redirected outside their fixture.

Git uses explicit HEAD worktree restore; Oak uses its documented default HEAD

source and `--force` to avoid a confirmation prompt. Installed Oak v0.103.0

does not accept literal `--source HEAD`; that spelling is not assumed equivalent.

Missing binaries and unavailable filesystem symlink/executable support are

explicit skips. Product rejection of a valid fixture or command remains a

failure, not a capability skip.

Negative controls inject a no-op restore, lost untracked file, changed bytes,

renamed/extra paths, changed mode/link target/type, successful invalid-source

response, signal termination, and unexpected HEAD movement. An instrument

that accepts any of these must fail its own test suite.

## Reviewed upstream sources and port mapping

Reviewed 2026-09-09. These are deliberately pinned historical releases, not

claims about the latest suite. Source inspection used official Git/Jujutsu

GitHub repositories and Mercurial's official repository. No external test

script, helper, fixture, expected output, or generated repository was copied or

translated. This follows [prior-art policy](prior-art.md). Local fixture bytes,

paths, Python implementations and oracles are independently authored.

Git v2.50.1 resolves to commit

`d82adb61ba2fd11d8f2587fca1b6bd7925ce4044`; Jujutsu v0.31.0 to

`312c610c592b41ec81f0661e124fd660cb510ab0`; Mercurial 6.9 to

`b267c5764cc6b804c619a42067405f27e8705beb`.

| Inspected primary source | Behavior learned | Local use or next port |

| --- | --- | --- |

| [Git t3902 quoted paths](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t3902-quoted.sh) | Path names with whitespace, line separators, quotes and non-ASCII content must remain unambiguous across command output. | `path_roundtrip_v1` covers storage/restoration; exact JSON/NUL output parsing remains a separate gap. |

| [Git t2070 restore](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t2070-restore.sh) | Restoring a chosen file/source must respect worktree/index scope; invalid option combinations must fail. | Scoped restore and invalid-source preservation probes; byte roundtrip extends the exact-file comparison principle. Index-specific restore/merge-conflict behavior deferred. |

| [Git t2003 checkout-index](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t2003-checkout-cache-mkdir.sh) | Symlinks, file/directory collisions and output prefixes require file-type checks, not content alone. | Mode/link roundtrip is the prerequisite; replacement of symlinked parent directories and no-write-outside-checkout checks remain deferred. |

| [Git t7519 fsmonitor](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t7519-status-fsmonitor.sh) | Tuned status must preserve full-index answers; sparse cases may expand the index. | P1: compare exact dirty-path sets against untuned independently generated manifests across rename/delete/untracked changes, watcher reset and sparse boundaries. |

| [Jujutsu restore tests](https://github.com/jj-vcs/jj/blob/312c610c592b41ec81f0661e124fd660cb510ab0/cli/tests/test_restore_command.rs) | `test_restore` checks source/destination behavior; interactive path selection must not restore unselected edits. | Scoped preservation is a common behavioral contract; no Jujutsu subject adapter is claimed. |

| [Jujutsu undo tests](https://github.com/jj-vcs/jj/blob/312c610c592b41ec81f0661e124fd660cb510ab0/cli/tests/test_undo.rs) | `test_undo_rewrite_with_child`, `test_git_push_undo`, `test_git_push_undo_with_import` and repeated-undo cases expose graph and remote-tracking consequences. | P1: local-history recovery with descendant and branch-head oracles. P2: undo/restore after publish must distinguish local state from external remote state; never assume local undo retracts a push. |

| [Mercurial change/delete conflicts](https://repo.mercurial-scm.org/hg/file/b267c5764cc6b804c619a42067405f27e8705beb/tests/test-merge-changedelete.t) | Noninteractive conflicts preserve unresolved entries; malformed/insufficient input stays recoverable; transitions among resolution policies retain consistent content and merge state. | P1: independently generated edit/delete conflicts in both directions, retry/abort and re-resolution matrix. Exact file state and unresolved-path state must be checked separately. Not implemented by the simple invalid-source scenario. |

The repository clean-room restriction applies regardless of upstream license.

Git/Mercurial copyleft suites and Jujutsu's Apache-2.0 headers are provenance

context, not permission to vendor tests under this policy. There are no imported

upstream files to relicense. Preserve these links when evolving the scenarios;

new semantics require new scenario identities.

## Local diagnostic observation

The implementation run on macOS with installed Oak v0.103.0 passed byte,

mode/symlink, scoped-restore and failed-restore checks. The path roundtrip failed

at checkpoint because Oak rejects tab-containing filenames. Git passed all five.

This is a concrete path-support gap against the declared fixture, not evidence

of data corruption, slower performance, or a general reliability ranking. Later

path entries are not independently certified by a scenario that stopped early.

The runner intentionally retains the failure. Generated raw records are not

committed; rerun with pinned binaries before using the observation in a decision.

## Remaining priorities

1. Add merge edit/delete, rename/edit and rename/rename cases with exact parent,

   unresolved-path, result-tree and abort-preservation oracles; adapt these to

   the existing conflict corpus lane instead of duplicating that runner.

2. Add metamorphic tuned-status/sparse/full-tree parity and safe symlink-parent

   replacement. Distinguish absent tracked entries from absent hydrated data.

3. Add graph-aware recovery after rewrite and push, with fresh second-reader

   remote validation and failure injection at request/acknowledgment boundaries.

4. Apply proven oracles to varied agent tasks and controlled human studies.

   Upstream command tests do not establish discoverability, human task time,

   agent success rate, prompt familiarity, or statistical power.

This bounded port does not represent exhaustive upstream coverage. Submodules,

partial-clone object absence, filesystem normalization/case collisions, hostile

paths, fsync/crash durability, broad graph topologies and large-repo performance

need independently specified fixtures and supported comparator semantics.