Log in
benchmarksPublic

Viewing files at commit f2acb751aa96Commit detailsView latest

..repro2 filesrobustness14 filesREADME.md8.2 KBrun_robustness.py22.5 KB
suites/robustness/README.md 140 lines · 8.2 KB

Robustness suite (program pillar P4)

Crash points, network fault injection, concurrency, and environment faults for Oak and Git, judged by independent readers. Rows are the robustness lane (scripts/oakbench/rows.py, scenarios/robustness.yaml). Nothing here is a timing measurement: elapsed_ms is the faulted operation's wall time, recorded for context only.

# the independent Oak tree reader needs blake3 + zstandard in a pinned Python
python3 -m venv /tmp/readerenv && /tmp/readerenv/bin/pip install blake3 zstandard
export OAK_BENCH_OAK_READER_PYTHON=/tmp/readerenv/bin/python

python3 suites/robustness/run_robustness.py --quick                 # smoke, ~10 min
python3 suites/robustness/run_robustness.py                         # full matrix
python3 suites/robustness/run_robustness.py --subjects oak --families proxy --ops push

Output: results/robustness/<bench_id>.robustness.jsonl (+ .summary.md, a .partial.jsonl flushed per case, and <bench_id>.robustness-evidence/ with the trial directories of violated / harness_error cases). Results are never committed (ADR-0003). Cases are not timing measurements, so the runner does not hold the shared benchmark measurement lock by default (a long run would block every other lane); the fsync storm and N-client runs do perturb timing, so pass --measurement-lock to serialize behind a running campaign, and never run the full suite beside one.

Safety: every server is local — a loopback oak serve (fixed per-fixture port, restarted from a copied data dir per trial), a local bare Git repository, or the stdlib smart-HTTP Git server. The fault proxy refuses non-loopback upstreams and the hosted spec refuses non-loopback URLs; nothing here ever targets oak.space.

Classification

classrowmeaning
heldreturncode 0every measured invariant held
violatedreturncode 1at least one invariant is false (a subject bug; see violations)
unsupportedreturncode 77, skipped: true, skip_reasonthe case cannot run for this subject/host (a work item, ADR-0002)
harness_errorreturncode 2, harness_errorthe harness failed; never counted as a subject result

Invariants are true / false / null (unmeasured, e.g. the tree reader is unavailable). null never becomes a pass or a fail.

Independent readers

  • Oak committed graph: stdlib sqlite3 on a private copy of .oak/oak.db (local) or <owner>/<repo>.oakdb (serve data dir). The copy's WAL is folded in by SQLite itself; the subject's files are never opened for write. Refs = branch_heads + metadata.head + tags; reachability walks commits.parent_hash/merge_parent_hash. PRAGMA integrity_check is the "store opens" oracle.
  • Oak committed trees: scripts/oakbench/oak_sqlite_reader.py (P1's reader, blake3/zstd hash-verified) run on the same private copy; for a non-HEAD commit the copy's metadata is pointed at it.
  • Git: the real Git plumbing (for-each-ref --contains, merge-base --is-ancestor, fsck, archive) — Git is the fixture/oracle subject.

Families and invariants

crash_time_v1 / crash_syscall_v1 (robustness/crash.py). Ops: commit, push, pull_ff, pull_diverged (re-parent / merge), clone, switch, merge_local (Git), merge_hosted and mount_finish (unsupported rows, with the reason). Each (subject, op) fixture is built once and copied per trial. A clean calibration run records the post-state, the duration, and the number of durability boundaries. crash_time kills at 11 fractions of the duration; crash_syscall kills immediately before the N-th fsync / F_FULLFSYNC / rename of the subject process (ALICE/CrashMonkey method, process-crash model: the page cache survives; power loss needs LazyFS on Linux). The interposer is a DYLD library loaded into an ad-hoc re-signed copy of the same binary (hardened runtime ignores DYLD_INSERT_LIBRARIES); both sha256s are recorded. After each kill:

  • op_no_hang, repo_opens (integrity_check / fsck),
  • acked_local_reachable: every commit a local ref named before the op is still reachable from a ref (checked again after the next op: acked_after_next),
  • server_intact: remote store opens, pre-op heads reachable, branch head is the pre-op head or the op's intended head (no torn / regressed ref),
  • worktree_consistent: worktree equals the pre/post state or a committed tree, or status reports dirty / refuses actionably — never "clean" while it differs,
  • partial_clone_detected (clone): a torn clone directory must be recognised by the subject's status with an actionable error,
  • next_op_ok_or_actionable: rerunning the op succeeds or fails with a typed / actionable message (JSON error code, or a recovery command); hang, panic or signal is a violation,
  • no_false_success: after a successful next op the state is exactly the calibrated outcome (trees compared by the independent readers).

fault_proxy_v1 (robustness/faultproxy.py, proxycases.py). A clean record run enumerates the op's requests; each trial injects one fault on the first / last occurrence of each request key: fail_status (503), drop_after_send (server applied it, reply lost: unknown outcome), truncate_reply, kill_before_forward / kill_after_upstream (network-phase crash points), plus slow_reply and seeded jitter (reordering of concurrent replies). Per-request trials pin OAK_UPLOAD_CONCURRENCY=OAK_DOWNLOAD_CONCURRENCY=1 for a deterministic request sequence. Extra invariants: op_success_is_true (exit 0 implies the calibrated outcome — an acknowledged push is on the server) and unknown_not_reported_success (a lost reply to a mutating request is never followed by exit 0 without further server contact). Git runs the same plan over git http-backend. applied_but_reported_failure is recorded as an observation.

conc_same_branch_v1 / conc_distinct_branches_v1 / env_fsync_load_v1 (concurrency.py, lincheck.py). N checkouts (copies of one seed, the jj "two machines" pattern) commit and push, pulling and retrying on rejection. Each push is an operation with invoke/return times and outcome ok / rejected / unknown. lincheck.py is a stdlib Wing–Gong/Lowe search (the Porcupine approach, re-implemented; no Porcupine code) over a fast-forward register model, with ancestry from the union of independently-read commit graphs. Invariants: linearizable, no_lost_acknowledged_write, acked_content_on_server (every acknowledged file is in the final server tree), server_opens, rejections_actionable, no_hang. env_fsync_load_v1 repeats same-branch under 4 background F_FULLFSYNC writers.

lock_shared_commit_v1 / lock_shared_push_v1 / lock_stale_owner_v1: 8 processes commit or push concurrently in one checkout (workdir / publication locks); lock_stale_owner_v1 first SIGKILLs a commit at its first durability boundary (owner dead while holding the lock) — contenders must reap it (stale_owner_reaped). lock_credentials_v1 is unsupported (no non-interactive credential writer).

env_disk_full_v1 / env_readonly_v1 (envfaults.py). The trial lives on a 96 MiB RAM disk (hdiutil attach -nomount ram://, newfs_hfs, mounted inside the work dir; no sudo) filled to 0 / 256 KiB / 4 MiB free; or .oak/.git (store) or one worktree subdir is made read-only. The op must fail actionably (op_failure_actionable) or succeed exactly (op_success_is_true); after the fault is removed the crash invariants and next-op convergence apply.

hosted_merge_invariants_v1 (hosted.py): the merge owner's invariants (acknowledged merge is on main; main never loses a landed change; push during merge not dropped; concurrent merges serialize; ambiguous merge never reported as success). Spec only: loopback oak serve returns 404 for merge publication, so these are unsupported rows until a local oakspace driver exists (OAK_BENCH_ROBUSTNESS_OAKSPACE_URL, loopback only).

Provenance and attribution

Ideas, re-implemented here (no code copied): ALICE (OSDI'14) and CrashMonkey/ACE (OSDI'18) crash-state enumeration; Porcupine (MIT) linearizability checking; jj's test_bad_locking (Apache-2.0) copied-workspace pattern; Jepsen nemeses. Seed harnesses: the 2026-09-26 L9 pull-loss kill-9 matrix, the L8 lock/fsync harnesses, and the W3a QA fault proxy.