# Robustness suite (program pillar P4)

Crash points, network fault injection, concurrency, and environment faults for

Oak and Git, judged by **independent readers**. Rows are the `robustness` lane

(`scripts/oakbench/rows.py`, `scenarios/robustness.yaml`). Nothing here is a

timing measurement: `elapsed_ms` is the faulted operation's wall time, recorded

for context only.

```bash

# the independent Oak tree reader needs blake3 + zstandard in a pinned Python

python3 -m venv /tmp/readerenv && /tmp/readerenv/bin/pip install blake3 zstandard

export OAK_BENCH_OAK_READER_PYTHON=/tmp/readerenv/bin/python

python3 suites/robustness/run_robustness.py --quick                 # smoke, ~10 min

python3 suites/robustness/run_robustness.py                         # full matrix

python3 suites/robustness/run_robustness.py --subjects oak --families proxy --ops push

```

Output: `results/robustness/<bench_id>.robustness.jsonl` (+ `.summary.md`, a

`.partial.jsonl` flushed per case, and `<bench_id>.robustness-evidence/` with

the trial directories of violated / harness_error cases). Results are never

committed (ADR-0003). Cases are not timing measurements, so the runner does not

hold the shared benchmark measurement lock by default (a long run would block

every other lane); the fsync storm and N-client runs do perturb timing, so pass

`--measurement-lock` to serialize behind a running campaign, and never run the

full suite beside one.

Safety: every server is local — a loopback `oak serve` (fixed per-fixture port,

restarted from a copied data dir per trial), a local bare Git repository, or the

stdlib smart-HTTP Git server. The fault proxy refuses non-loopback upstreams and

the hosted spec refuses non-loopback URLs; nothing here ever targets oak.space.

## Classification

| class | row | meaning |

|---|---|---|

| `held` | returncode 0 | every measured invariant held |

| `violated` | returncode 1 | at least one invariant is `false` (a subject bug; see `violations`) |

| `unsupported` | returncode 77, `skipped: true`, `skip_reason` | the case cannot run for this subject/host (a work item, ADR-0002) |

| `harness_error` | returncode 2, `harness_error` | the harness failed (incl. an independent reader raising); never counted as a subject result |

| `fault_not_fired` | returncode 77, `skipped: true`, `skip_reason`, `fault_fired: false` | the fault never took effect (the op exited before the kill; the disk was not short of the op's need; no proxy rule matched). A clean run is never `held` |

Invariants are `true` / `false` / `null` (unmeasured, e.g. the tree reader is

unavailable). `null` never becomes a pass or a fail. The lane validator

(`oakbench.rows.validate_rows(..., "robustness")`) rejects unknown classes,

a classification whose returncode does not match, and `held`/`violated` rows

with `fault_fired: false`.

Fired evidence: a time or syscall kill fired only if the client's exit status

is SIGKILL (`rc == -9`); a proxy fault only if its rule matched a request; disk

full only if the measured free space was below the op's need, or the op failed

/ reported ENOSPC. A time kill that lands after the op exited is re-aimed at

the same fraction of the duration the op actually took (up to 2 retries, fresh

fixture copy each; `observations.time_attempts`), then recorded

`fault_not_fired`.

Known bugs: violated rows whose violations match `robustness.KNOWN_BUGS` carry

`known_bugs` (e.g. `fb-535` torn switch/pull worktree, `fb-534` lost

acknowledged push); unmatched violations are new findings.

## Independent readers

- **Oak committed graph**: stdlib `sqlite3` on a private copy of `.oak/oak.db`

  (local) or `<owner>/<repo>.oakdb` (serve data dir). The copy's WAL is folded in

  by SQLite itself; the subject's files are never opened for write. Refs =

  `branch_heads` + `metadata.head` + `tags`; reachability walks

  `commits.parent_hash/merge_parent_hash`. `PRAGMA integrity_check` is the

  "store opens" oracle.

- **Oak committed trees**: `scripts/oakbench/oak_sqlite_reader.py` (P1's reader,

  blake3/zstd hash-verified) run on the same private copy; for a non-HEAD commit

  the copy's metadata is pointed at it.

- **Git**: the real Git plumbing (`for-each-ref --contains`, `merge-base

  --is-ancestor`, `fsck`, `archive`) — Git is the fixture/oracle subject.

## Families and invariants

**crash_time_v1 / crash_syscall_v1** (`robustness/crash.py`). Ops: `commit`,

`push`, `pull_ff`, `pull_diverged` (re-parent / merge), `clone`, `switch`,

`merge_local` (Git), `merge_hosted` and `mount_finish` (unsupported rows, with the

reason). Each (subject, op) fixture is built once and copied per trial. A clean

calibration run records the post-state, the duration, and the number of

durability boundaries. `crash_time` kills at 11 fractions of the duration;

`crash_syscall` kills immediately **before the N-th fsync / F_FULLFSYNC /

rename** of the subject process (ALICE/CrashMonkey method, process-crash model:

the page cache survives; power loss needs LazyFS on Linux). The interposer is a

DYLD library loaded into an ad-hoc re-signed copy of the same binary (hardened

runtime ignores DYLD_INSERT_LIBRARIES); both sha256s are recorded.

After each kill:

- `op_no_hang`, `repo_opens` (integrity_check / fsck),

- `acked_local_reachable`: every commit a local ref named before the op is still

  reachable from a ref (checked again after the next op: `acked_after_next`),

- `server_intact`: remote store opens, pre-op heads reachable, branch head is

  the pre-op head or the op's intended head (no torn / regressed ref),

- `worktree_consistent`: worktree equals the pre/post state or a committed tree,

  or status reports dirty / refuses actionably — never "clean" while it differs,

- `worktree_not_torn`: the worktree is compared file by file against the op's

  pre side (pristine client) and target side (calibrated post-op worktree), and

  against where the ref is (HEAD unchanged = pre side, moved = post side). If it

  holds the op's output out of step with the ref (target-side files while HEAD

  is old, pre-side files while HEAD moved, or the op's temp files) and status

  lists those as ordinary uncommitted changes with nothing in progress, it

  fails: committing would record the op's partial output as the user's work.

  Passes when the state is flagged (MERGE_HEAD, `progress_state.in_progress`, a

  stale git lock that makes the next `git commit` refuse with "remove the file

  manually": `index.lock`, `HEAD.lock` or the current branch's

  `refs/heads/<branch>.lock`; `config.lock`, other refs' locks,

  `packed-refs.lock` do not block a commit and do not count), status refuses, or the

  failing op itself names a recovery command,

- `partial_clone_detected` (clone): a torn clone directory must be recognised by

  the subject's status with an actionable error,

- `next_op_ok_or_actionable`: rerunning the op succeeds or fails with an

  actionable message: a concrete recovery command (a real `oak`/`git`

  subcommand or `--continue/--abort`), an explicit manual step ("remove the

  file manually"), a documented retryable condition, `recommended_next_commands`,

  or a specific (non-generic) JSON error code. Mentioning "oak"/"git" is not

  enough. Corruption / internal / opaque-store wording (malformed, corrupt,

  bad or invalid object, unable to read sha1 file/tree, early EOF, segfault,

  empty object file, HTTP 500, "unable to open database file", "not a git

  repository") is checked first and makes the failure non-actionable even

  when it also names a recovery command. Git's normal outcomes "Automatic merge failed; fix conflicts and

  then commit the result." and "Not possible to fast-forward, aborting." are

  actionable only for the git subject and only when every output line is one

  of git's ordinary merge/pull lines (Auto-merging, CONFLICT, hint:, From,

  FETCH_HEAD, Updating): any other line (`fatal: bad object HEAD`, a

  segfault, "unable to open database file") makes the hint not count;

  corruption / internal / panic text and generic codes (`error`,

  `internal`, ...) are never actionable; hang, panic or signal is a violation,

- `no_false_success`: after a successful next op the state is exactly the

  calibrated outcome (trees compared by the independent readers).

**fault_proxy_v1** (`robustness/faultproxy.py`, `proxycases.py`). A clean record

run enumerates the op's requests; each trial injects one fault on the first /

last occurrence of each request key: `fail_status` (503), `drop_after_send`

(server applied it, reply lost: unknown outcome), `truncate_reply`,

`kill_before_forward` / `kill_after_upstream` (network-phase crash points), plus

`slow_reply` and seeded `jitter` (reordering of concurrent replies). Per-request

trials pin `OAK_UPLOAD_CONCURRENCY=OAK_DOWNLOAD_CONCURRENCY=1` for a

deterministic request sequence. Extra invariants: `op_success_is_true` (exit 0

implies the calibrated outcome — an acknowledged push is on the server) and

`unknown_not_reported_success` (a lost reply to a mutating request is never

followed by exit 0 without further server contact). Git runs the same plan over

`git http-backend`. `applied_but_reported_failure` is recorded as an observation.

**conc_same_branch_v1 / conc_distinct_branches_v1 / env_fsync_load_v1**

(`concurrency.py`, `lincheck.py`). N checkouts (copies of one seed, the jj

"two machines" pattern) commit and push, pulling and retrying on rejection.

Each push is an operation with invoke/return times and outcome ok / rejected /

unknown. `lincheck.py` is a stdlib Wing–Gong/Lowe search (the Porcupine

approach, re-implemented; no Porcupine code) over a fast-forward register

model, with ancestry from the union of independently-read commit graphs.

Invariants: `linearizable`, `no_lost_acknowledged_write`, `acked_content_on_server`

(every acknowledged file is in the final server tree), `server_opens`,

`rejections_actionable`, `no_hang`. `env_fsync_load_v1` repeats same-branch under

4 background F_FULLFSYNC writers.

**lock_shared_commit_v1 / lock_shared_push_v1 / lock_stale_owner_v1**: 8 processes

commit or push concurrently in one checkout (workdir / publication locks);

`lock_stale_owner_v1` first SIGKILLs a commit at its first durability boundary

(owner dead while holding the lock) — contenders must reap it (`stale_owner_reaped`).

`lock_credentials_v1` is unsupported (no non-interactive credential writer).

Oak pushes run with `--json`; each acknowledged push's reported `pushed_head` is

cross-checked against the local head read before the push

(`acked_head_matches_reported`).

**env_disk_full_v1 / env_readonly_v1** (`envfaults.py`). The trial lives on a

96 MiB RAM disk (`hdiutil attach -nomount ram://`, `newfs_hfs`, mounted inside

the work dir; no sudo) filled until writes fail at 1-byte granularity (1 MiB,

64 KiB, 4 KiB, 512 B, 1 B rounds, fsync between), then 0 / 256 KiB / 4 MiB are

released. The free space is measured (`statvfs`) right before the op and

recorded as `free_bytes_measured`, beside `op_need_bytes` (net allocation of a

clean run on an ordinary disk); or `.oak`/`.git`

(store) or one worktree subdir is made read-only. The op must fail actionably

(`op_failure_actionable`) or succeed exactly (`op_success_is_true`); after the

fault is removed the crash invariants and next-op convergence apply.

`fault_fired` (reason in `observations.fault_fired_evidence`): the disk was

really filled, so an op that fails, errors or hangs is always judged: an opaque

error (fb-548 "unable to open database file", a panic) violates

`op_failure_actionable`, and a hang violates `op_no_hang`. Only a successful op is

decided by evidence: fired if it reported ENOSPC / "No space left" / SQLITE_FULL

or the measured free space was below `op_need_bytes`, else `fault_not_fired`

(it had room). A failed calibration run leaves the need unmeasured and the row

is `harness_error`, never need=0. Read-only works the same way: a failing op is

judged; a successful op counts as fired only if the path was really unwritable

(not as root) and the op wrote there: it printed a permission / read-only

error, or a clean probe run of the same op (`observations.readonly_exercise`)

changes something under the locked path (store: any file; subdir: the

directory's own entries). Otherwise it is `fault_not_fired` ("not exercised").

If the probe itself failed, the row is `harness_error`.

**hosted_merge_invariants_v1** (`hosted.py`): the merge owner's invariants

(acknowledged merge is on main; main never loses a landed change; push during

merge not dropped; concurrent merges serialize; ambiguous merge never reported

as success). Spec only: loopback `oak serve` returns 404 for merge publication,

so these are unsupported rows until a local oakspace driver exists

(`OAK_BENCH_ROBUSTNESS_OAKSPACE_URL`, loopback only).

## Provenance and attribution

Ideas, re-implemented here (no code copied): ALICE (OSDI'14) and CrashMonkey/ACE

(OSDI'18) crash-state enumeration; Porcupine (MIT) linearizability checking;

jj's `test_bad_locking` (Apache-2.0) copied-workspace pattern; Jepsen nemeses.

Seed harnesses: the 2026-09-26 L9 pull-loss kill-9 matrix, the L8 lock/fsync

harnesses, and the W3a QA fault proxy.