Log in
benchmarksPublic

Viewing files at commit ef78687eef1aCommit detailsView latest

..repro3 filesrobustness14 filesREADME.md12.9 KBrun_robustness.py23.9 KB
suites/robustness/README.md 204 lines · 12.9 KB

Robustness suite (program pillar P4)

Crash points, network fault injection, concurrency, and environment faults for Oak and Git, judged by independent readers. Rows are the robustness lane (scripts/oakbench/rows.py, scenarios/robustness.yaml). Nothing here is a timing measurement: elapsed_ms is the faulted operation's wall time, recorded for context only.

# the independent Oak tree reader needs blake3 + zstandard in a pinned Python
python3 -m venv /tmp/readerenv && /tmp/readerenv/bin/pip install blake3 zstandard
export OAK_BENCH_OAK_READER_PYTHON=/tmp/readerenv/bin/python

python3 suites/robustness/run_robustness.py --quick                 # smoke, ~10 min
python3 suites/robustness/run_robustness.py                         # full matrix
python3 suites/robustness/run_robustness.py --subjects oak --families proxy --ops push

Output: results/robustness/<bench_id>.robustness.jsonl (+ .summary.md, a .partial.jsonl flushed per case, and <bench_id>.robustness-evidence/ with the trial directories of violated / harness_error cases). Results are never committed (ADR-0003). Cases are not timing measurements, so the runner does not hold the shared benchmark measurement lock by default (a long run would block every other lane); the fsync storm and N-client runs do perturb timing, so pass --measurement-lock to serialize behind a running campaign, and never run the full suite beside one.

Safety: every server is local — a loopback oak serve (fixed per-fixture port, restarted from a copied data dir per trial), a local bare Git repository, or the stdlib smart-HTTP Git server. The fault proxy refuses non-loopback upstreams and the hosted spec refuses non-loopback URLs; nothing here ever targets oak.space.

Classification

classrowmeaning
heldreturncode 0every measured invariant held
violatedreturncode 1at least one invariant is false (a subject bug; see violations)
unsupportedreturncode 77, skipped: true, skip_reasonthe case cannot run for this subject/host (a work item, ADR-0002)
harness_errorreturncode 2, harness_errorthe harness failed (incl. an independent reader raising); never counted as a subject result
fault_not_firedreturncode 77, skipped: true, skip_reason, fault_fired: falsethe fault never took effect (the op exited before the kill; the disk was not short of the op's need; no proxy rule matched). A clean run is never held

Invariants are true / false / null (unmeasured, e.g. the tree reader is unavailable). null never becomes a pass or a fail. The lane validator (oakbench.rows.validate_rows(..., "robustness")) rejects unknown classes, a classification whose returncode does not match, and held/violated rows with fault_fired: false.

Fired evidence: a time or syscall kill fired only if the client's exit status is SIGKILL (rc == -9); a proxy fault only if its rule matched a request; disk full only if the measured free space was below the op's need, or the op failed / reported ENOSPC. A time kill that lands after the op exited is re-aimed at the same fraction of the duration the op actually took (up to 2 retries, fresh fixture copy each; observations.time_attempts), then recorded fault_not_fired.

Known bugs: violated rows whose violations match robustness.KNOWN_BUGS carry known_bugs (e.g. fb-535 torn switch/pull worktree, fb-534 lost acknowledged push); unmatched violations are new findings.

Independent readers

  • Oak committed graph: stdlib sqlite3 on a private copy of .oak/oak.db (local) or <owner>/<repo>.oakdb (serve data dir). The copy's WAL is folded in by SQLite itself; the subject's files are never opened for write. Refs = branch_heads + metadata.head + tags; reachability walks commits.parent_hash/merge_parent_hash. PRAGMA integrity_check is the "store opens" oracle.
  • Oak committed trees: scripts/oakbench/oak_sqlite_reader.py (P1's reader, blake3/zstd hash-verified) run on the same private copy; for a non-HEAD commit the copy's metadata is pointed at it.
  • Git: the real Git plumbing (for-each-ref --contains, merge-base --is-ancestor, fsck, archive) — Git is the fixture/oracle subject.

Families and invariants

crash_time_v1 / crash_syscall_v1 (robustness/crash.py). Ops: commit, push, pull_ff, pull_diverged (re-parent / merge), clone, switch, merge_local (Git), merge_hosted and mount_finish (unsupported rows, with the reason). Each (subject, op) fixture is built once and copied per trial. A clean calibration run records the post-state, the duration, and the number of durability boundaries. crash_time kills at 11 fractions of the duration; crash_syscall kills immediately before the N-th fsync / F_FULLFSYNC / rename of the subject process (ALICE/CrashMonkey method, process-crash model: the page cache survives; power loss needs LazyFS on Linux). The interposer is a DYLD library loaded into an ad-hoc re-signed copy of the same binary (hardened runtime ignores DYLD_INSERT_LIBRARIES); both sha256s are recorded. After each kill:

  • op_no_hang, repo_opens (integrity_check / fsck),
  • acked_local_reachable: every commit a local ref named before the op is still reachable from a ref (checked again after the next op: acked_after_next),
  • server_intact: remote store opens, pre-op heads reachable, branch head is the pre-op head or the op's intended head (no torn / regressed ref),
  • worktree_consistent: worktree equals the pre/post state or a committed tree, or status reports dirty / refuses actionably — never "clean" while it differs,
  • worktree_not_torn: the worktree is compared file by file against the op's pre side (pristine client) and target side (calibrated post-op worktree), and against where the ref is (HEAD unchanged = pre side, moved = post side). If it holds the op's output out of step with the ref (target-side files while HEAD is old, pre-side files while HEAD moved, or the op's temp files) and status lists those as ordinary uncommitted changes with nothing in progress, it fails: committing would record the op's partial output as the user's work. Passes when the state is flagged (MERGE_HEAD, progress_state.in_progress, a stale git lock that makes the next git commit refuse with "remove the file manually": index.lock, HEAD.lock or the current branch's refs/heads/<branch>.lock; config.lock, other refs' locks, packed-refs.lock do not block a commit and do not count), status refuses, or the failing op itself names a recovery command,
  • partial_clone_detected (clone): a torn clone directory must be recognised by the subject's status with an actionable error,
  • next_op_ok_or_actionable: rerunning the op succeeds or fails with an actionable message: a concrete recovery command (a real oak/git subcommand or --continue/--abort), an explicit manual step ("remove the file manually"), a documented retryable condition, recommended_next_commands, or a specific (non-generic) JSON error code. Mentioning "oak"/"git" is not enough. Corruption / internal / opaque-store wording (malformed, corrupt, bad or invalid object, unable to read sha1 file/tree, early EOF, segfault, empty object file, HTTP 500, "unable to open database file", "not a git repository") is checked first and makes the failure non-actionable even when it also names a recovery command. Git's normal outcomes "Automatic merge failed; fix conflicts and then commit the result." and "Not possible to fast-forward, aborting." are actionable only for the git subject and only when every output line is one of git's ordinary merge/pull lines (Auto-merging, CONFLICT, hint:, From, FETCH_HEAD, Updating): any other line (fatal: bad object HEAD, a segfault, "unable to open database file") makes the hint not count; corruption / internal / panic text and generic codes (error, internal, ...) are never actionable; hang, panic or signal is a violation,
  • no_false_success: after a successful next op the state is exactly the calibrated outcome (trees compared by the independent readers).

fault_proxy_v1 (robustness/faultproxy.py, proxycases.py). A clean record run enumerates the op's requests; each trial injects one fault on the first / last occurrence of each request key: fail_status (503), drop_after_send (server applied it, reply lost: unknown outcome), truncate_reply, kill_before_forward / kill_after_upstream (network-phase crash points), plus slow_reply and seeded jitter (reordering of concurrent replies). Per-request trials pin OAK_UPLOAD_CONCURRENCY=OAK_DOWNLOAD_CONCURRENCY=1 for a deterministic request sequence. Extra invariants: op_success_is_true (exit 0 implies the calibrated outcome — an acknowledged push is on the server) and unknown_not_reported_success (a lost reply to a mutating request is never followed by exit 0 without further server contact). Git runs the same plan over git http-backend. applied_but_reported_failure is recorded as an observation.

conc_same_branch_v1 / conc_distinct_branches_v1 / env_fsync_load_v1 (concurrency.py, lincheck.py). N checkouts (copies of one seed, the jj "two machines" pattern) commit and push, pulling and retrying on rejection. Each push is an operation with invoke/return times and outcome ok / rejected / unknown. lincheck.py is a stdlib Wing–Gong/Lowe search (the Porcupine approach, re-implemented; no Porcupine code) over a fast-forward register model, with ancestry from the union of independently-read commit graphs. Invariants: linearizable, no_lost_acknowledged_write, acked_content_on_server (every acknowledged file is in the final server tree), server_opens, rejections_actionable, no_hang. env_fsync_load_v1 repeats same-branch under 4 background F_FULLFSYNC writers.

lock_shared_commit_v1 / lock_shared_push_v1 / lock_stale_owner_v1: 8 processes commit or push concurrently in one checkout (workdir / publication locks); lock_stale_owner_v1 first SIGKILLs a commit at its first durability boundary (owner dead while holding the lock) — contenders must reap it (stale_owner_reaped). lock_credentials_v1 is unsupported (no non-interactive credential writer). Oak pushes run with --json; each acknowledged push's reported pushed_head is cross-checked against the local head read before the push (acked_head_matches_reported).

env_disk_full_v1 / env_readonly_v1 (envfaults.py). The trial lives on a 96 MiB RAM disk (hdiutil attach -nomount ram://, newfs_hfs, mounted inside the work dir; no sudo) filled until writes fail at 1-byte granularity (1 MiB, 64 KiB, 4 KiB, 512 B, 1 B rounds, fsync between), then 0 / 256 KiB / 4 MiB are released. The free space is measured (statvfs) right before the op and recorded as free_bytes_measured, beside op_need_bytes (net allocation of a clean run on an ordinary disk); or .oak/.git (store) or one worktree subdir is made read-only. The op must fail actionably (op_failure_actionable) or succeed exactly (op_success_is_true); after the fault is removed the crash invariants and next-op convergence apply. fault_fired (reason in observations.fault_fired_evidence): the disk was really filled, so an op that fails, errors or hangs is always judged: an opaque error (fb-548 "unable to open database file", a panic) violates op_failure_actionable, and a hang violates op_no_hang. Only a successful op is decided by evidence: fired if it reported ENOSPC / "No space left" / SQLITE_FULL or the measured free space was below op_need_bytes, else fault_not_fired (it had room). A failed calibration run leaves the need unmeasured and the row is harness_error, never need=0. Read-only works the same way: a failing op is judged; a successful op counts as fired only if the path was really unwritable (not as root) and the op wrote there: it printed a permission / read-only error, or a clean probe run of the same op (observations.readonly_exercise) changes something under the locked path (store: any file; subdir: the directory's own entries). Otherwise it is fault_not_fired ("not exercised"). If the probe itself failed, the row is harness_error.

hosted_merge_invariants_v1 (hosted.py): the merge owner's invariants (acknowledged merge is on main; main never loses a landed change; push during merge not dropped; concurrent merges serialize; ambiguous merge never reported as success). Spec only: loopback oak serve returns 404 for merge publication, so these are unsupported rows until a local oakspace driver exists (OAK_BENCH_ROBUSTNESS_OAKSPACE_URL, loopback only).

Provenance and attribution

Ideas, re-implemented here (no code copied): ALICE (OSDI'14) and CrashMonkey/ACE (OSDI'18) crash-state enumeration; Porcupine (MIT) linearizability checking; jj's test_bad_locking (Apache-2.0) copied-workspace pattern; Jepsen nemeses. Seed harnesses: the 2026-09-26 L9 pull-loss kill-9 matrix, the L8 lock/fsync harnesses, and the W3a QA fault proxy.