Human VCS study protocol v1 (proposed, not measured)
The current scripted and real-agent lanes do not establish human usability. This protocol specifies a separate evidence stream. No participant sessions have been collected or validated by this change; implementing a recorder and running a pilot are prerequisites to a human-performance claim.
Questions and experimental units
Measure whether a person can reach a correct, durable repository state, how long it takes, and what recovery/help costs they incur. The primary outcome is task success within a preregistered time budget. The co-reported resource outcome is total elapsed time to a verified result, including mistakes and recovery. Never rank a tool on successful attempts alone.
A participant is the independent sampling unit. Tasks and repeated attempts are observations nested inside a participant, not additional participants. Keep human and model-agent results in separate tables. Scripted command counts are neither observed human actions nor observed model tool calls.
Assignment and controls
- Recruit across declared VCS experience and tool-familiarity strata; record prior Git, Oak, and Jujutsu experience separately. A convenience sample of tool developers must be labeled as such.
- Use a crossover design with balanced subject order. Assign distinct but structurally equivalent fixture variants to each tool so participants do not solve the same exact conflict twice. Record the assignment seed and task/variant order before running sessions.
- Separate first-use and trained-use campaigns. Give a fixed training budget and equivalent documentation access. Record training time and competency; do not silently exclude participants who fail training.
- Keep the task objective, starting state, time budget, hardware, network, editor, and allowed help identical. Use isolated disposable repositories. CLI-only and GUI-assisted conditions are different tool surfaces.
- Pin binary hashes, versions, subject configuration, task/fixture/oracle versions, and documentation hashes. Log deviations and interruptions.
- Obtain consent for recorded interactions, use pseudonymous participant IDs, and keep raw recordings outside this repository. Permit withdrawal.
Initial task matrix
| Task family | Required end state | Specific loss/error check |
|---|---|---|
| Inspect an unfamiliar checkout | Correct inventory of edits, branch and unpublished work | Exact facts; distinguish omissions from invented facts |
| Save a scoped fix | Intended files durably snapshotted; unrelated edits preserved | Exact committed tree and unchanged out-of-scope bytes/modes |
| Switch between two tasks | Both task states independently recoverable | Dirty and untracked files preserved; correct parent/history |
| Undo a mistaken change | Requested state restored without collateral loss | Full tree manifest, symlink targets, executable bits, and unrelated edits |
| Resolve overlapping work | Expected merged content and declared history relationships | No dropped edits, conflict markers, duplicate application, or fabricated success |
| Recover from failed publication | Correct understanding of local and remote publication state | Inspect both sides independently; ambiguity remains explicit |
| Handoff to another person | Fresh recipient can locate and continue exact task state | Fresh-checkout bytes, history, description, and pending work |
Reuse a task's independent oracle across human and agent conditions when semantics really match. A subject's success message and a clean worktree are insufficient evidence of a correct commit. Local failure preservation and roundtrip conformance tests are building blocks, not a usability score.
Event and outcome record
The future recorder must retain an append-only event stream with participant, session, trial, subject, condition, task, variant, and assignment IDs. Capture monotonic event times, start/end markers, commands or UI actions, process exit status, help requests, retries, interruptions, and independent oracle results. Sensitive content belongs in access-controlled external storage.
Derive these fields without using unobserved zero defaults:
- Attempted, verified-success, timeout, abandoned, infrastructure-failure, and unobservable counts, with denominators and reasons.
- End-to-end elapsed time and time to first useful correct action; command execution time separately from human deliberation and help time when the recorder can distinguish them.
- Observed commands/actions, errors, recovery attempts, help usage, and collateral changes. Batch shell commands are one submitted interaction but may invoke several VCS processes; retain both units.
- Post-task ease/confidence on a declared fixed scale, and a short explanation of the hardest step. These are self-reports, not correctness evidence.
- Oracle evidence hashes and unknown/failed subchecks. A timeout is an outcome, not a row to delete. An infrastructure failure cannot become a tool success.
Analysis and publication gate
Preregister primary outcomes, practical effect thresholds, comparison family,
timeout rules, exclusions, and stopping policy. Use a pilot to estimate variance
and participant/task effects before determining the confirmatory sample size.
Do not present the harness's heuristic required_n as a human-study power
calculation. Freeze confirmatory tasks before evaluating candidates repeatedly;
retain held-out task families and fixture seeds for generalization checks.
Report success differences and uncertainty, restricted completion time within the common budget, and full attempt cost. Every uncompleted attempt, including timeout, abandonment or an incorrect terminal state, contributes the full budget to restricted completion time; retain actual time/resource use separately so early abandonment cannot look like fast completion. Show paired participant-level differences and participant-cluster uncertainty; use task-stratified analysis to expose order, learning, and task-family effects. Predeclare pairing and effect estimation for incomplete crossover sessions. Treat missing observations and participant attrition explicitly and include sensitivity analyses. Correct the declared comparison family for multiplicity or label exploration as such.
A publishable result needs audited event-to-metric reconstruction, oracle falsification tests, a completed counterbalanced pilot, a frozen analysis plan, adequate precision for the claimed effect, all attempted sessions accounted for, and an independently reproduced analysis. This protocol alone satisfies none of those gates.