/oak
Log in
benchmarks
docs/core-vcs-benchmarks.md 113 lines · 6.4 KB

Core VCS Benchmark Scenarios

scenarios/core.yaml defines the benchmark scenario layer for core version-control behavior. It is intentionally data-only: the current scripts/bench.py remains untouched, and a later harness can load this file to generate fixtures, choose operations, run modes, and apply correctness checks.

Goals

The matrix covers the workload dimensions that usually decide VCS performance:

  • Small fixed overhead: tiny.
  • Normal developer work: normal.
  • File-count pressure: large-tree, huge-tree, deep-tree.
  • Working-tree noise: ignored-heavy.
  • Structural churn: rename-heavy, delete-heavy.
  • Metadata scale: long-history, many-branches.
  • Binary storage and checkout: single-huge-blob, many-large-blobs, binary-history.
  • Merge behavior: conflict-heavy.

Fixture generation, manifest writing, and directory copying should not be timed. Timed operations should begin only after the fixture is fully materialized in the subject run directory.

YAML Shape

The file uses a simple YAML structure:

  • runtime_classes: smoke, standard, and large scheduling guidance.
  • profiles: named scenario sets.
  • operation_catalog: stable operation IDs for the harness to map to Git and Oak command sequences.
  • check_catalog: stable correctness check IDs.
  • mode_catalog: VCS-specific execution modes.
  • scenarios: the scale matrix, one object per benchmark scenario.

Each scenario includes purpose, fixture_shape, operations_to_time, correctness_checks, git_modes, oak_modes, runtime_class, and disk_requirements.

Treat the YAML as declarative input. If a VCS cannot perform an operation or mode, the harness should emit an explicit unsupported result row instead of silently dropping the case.

Runtime Classes

ClassUseExpected Size
smokeEvery local change or pushFast, low disk, catches command overhead regressions
standardNightly or pre-mergeBroad signal for file-count, ignored-tree, rename/delete, and conflict behavior
largeDedicated runnerMulti-GB or metadata-heavy validation of major performance claims

The disk numbers in YAML are deliberately conservative because Git and Oak may both create metadata, temporary files, and checkout copies. Runners should check recommended_free_bytes before generating a scenario.

Mode Semantics

Modes are execution configurations, not binary versions. The existing subject layer still decides which binary to run, such as git, oak_installed, oak_local, or optional oak_main.

Recommended harness behavior:

  1. Select scenarios from a profile.
  2. Select enabled subjects from config/subjects.toml.
  3. For each scenario, run every compatible git_modes entry against Git subjects and every compatible oak_modes entry against Oak subjects.
  4. Record scenario, operation, subject, mode, run, command, elapsed time, return code, and fixture metadata in each JSONL row.

Git notes:

  • Disable auto-gc during timed runs so background maintenance does not dominate variance.
  • git_split_index and git_untracked_cache are optimized comparisons for large working trees.
  • git_sparse_cone is valid only for scenarios with sparse_visible_files; it should not replace the full-tree baseline.
  • git_worktree should be measured separately from plain branch checkout because it represents a different isolation strategy.
  • git_lfs is wired in the active runner only for generated binary fixture scenarios; unsupported rows are explicit skips, never stock Git under an LFS label.

Oak notes:

  • oak_default is the materialized-repository baseline.
  • oak_virtual_branch represents branch-per-task behavior through oak switch -c.
  • oak_space_mount is for lazy mount or space workflows and should be marked unsupported until the harness has a fixture strategy for that mode.

Correctness

Every fixture should write a deterministic manifest outside the timed repository metadata. At minimum, include:

  • Scenario ID and generation parameters.
  • Tracked path list.
  • Content hash per tracked path at each relevant revision.
  • Expected dirty counts.
  • Expected branch names and branch tips.
  • Expected conflict paths and resolution hashes for conflict-heavy.

Correctness checks should run after the timed command they validate. The checks themselves are not benchmark measurements; record their pass/fail state separately so failures cannot be mistaken for slow timings.

For binary scenarios, diff operations must be binary-safe summaries. A harness should fail the correctness check if a command emits full binary payloads into captured output.

Operation Mapping

The operation IDs are stable API names for a future harness. A mapping layer should translate them to concrete command sequences per VCS and mode. For example:

  • snapshot.initial: Git pays add -A plus commit; Oak pays its snapshot command.
  • status.clean and status.dirty: same command shape, different fixture state.
  • branch.create_many: setup many branches from one base and time the creation loop as one operation unless a later runner wants per-branch rows.
  • merge.conflict: time only the merge attempt that produces conflicts; divergent branch setup is covered by merge.clean_base and is not timed.

Where a command has human-oriented output by default, prefer machine-readable or porcelain output if the VCS supports it. Use the same semantic output level across modes.

Implementation Notes

Keep the first harness pass dependency-free:

  • Use deterministic byte generators instead of random files from the OS.
  • Store generated fixtures under the existing benchmark work directory, not under the Oak source tree.
  • Include a fixture-ready marker only after the manifest has been written and verified.
  • Do not time fixture generation, manifest validation, subject binary discovery, or cleanup.
  • Capture stderr tails for failures, but avoid storing large diffs or binary output in JSONL.
  • Mark skipped modes as unsupported with a reason, not as successful zero-time rows.

Open questions for the harness phase:

  • Whether oak_space_mount should mount from a generated local repository fixture or from a synthetic remote-like namespace.
  • Whether branch enumeration should use a dedicated Oak machine-readable command if one exists, or a lower-level metadata query.
  • Whether large binary scenarios should run with cold cache, warm cache, or both on dedicated runners.