Driver Loop Reliability
Driver Loop Stress Reliability
A flaky driver-loop class became a reproducible stress case, hardened git path, and documented closeout.
0 milestones 5 tasks 14 agents
3 plan 5 execute 5 review 1 fix
222k peak context
156k median execute
3h 5m total 31m 47s per task
4m 24s plan 1h 3m execute 18m 47s review
49 edits 454 commands
codex harness
This run took a concurrency failure that had been under-sampled and made it a first-class reliability problem. The plan did not start by editing loop behavior; it first built the stress shape that could make the failure visible, then used that same shape to guide git subprocess hardening and clearer failure classification.
The result is split between code and evidence. The live tree still has the driver-loop stress preset, the shared git command hardening, infrastructure-aware loop and gate behavior, and the reliability record that tells future work which sample is meaningful for this class.
How this walkthrough is structured
Feature
What did the run build and ship?
Gantry now carries a whole-module driver-loop stress preset, hardened shared git invocation, infrastructure-aware control flow, and a durable reliability closeout.
Build
How did Gantry structure the work?
The cut put the oracle first, absorbed the only red boundary there, and then let the git, verdict, audit, and closeout work proceed against a stable reproduction.
Stress Repro Harness
4 agents 33m 32s wall time 81k peak context 1 fix 1 replan
The live `bin/stress` command still has the driver-loop concurrency preset. It materializes a frozen checkout, isolates cargo artifacts, fixes the scope to the driver-loop module, and routes analysis through the stress report script. The original integration test file for the runner no longer stands in the current tree, but the preset, analyzer, pre-fix snapshot, and reliability guidance remain as the usable artifact.
This sprint had to make the failure reproducible before any product fix could claim success. Its brief was narrow enough for a fresh worker because the deliverable was a stress runner and a captured pre-fix sample, not a diagnosis of every loop path. The run facts put the only red gate here, followed by a fix and then a re-plan of the remaining work. That boundary matters: the acceptance oracle itself needed correction before the rest of the plan leaned on it.
Git Subprocess Hardening
2 agents 31m 47s wall time 164k peak context
The shared worktree helper now builds hardened git commands with automatic maintenance disabled and read-only operations running without optional locks. It classifies mutating versus read-only verbs and retries only stderr shapes that look like transient lock contention. Many original call sites have moved during later refactors, but current run, registry, reconstruction, TUI, lock, and command paths still use the helper rather than spawning unconstrained git directly.
After the oracle was in place, this sprint took the suspected shared amplifier: Gantry-owned git subprocesses under parallel disposable worktrees. The work was a suitable single slice because it centered on one common command path and its callers, with tests for mode selection and retry behavior rather than broad driver rewrites. Its gates and review stayed green after the re-plan, which suggests the corrected reproduction gave the worker a stable target. The boundary left later sprints free to reason about verdict semantics instead of git process mechanics.
Verdict Isolation
2 agents 40m 27s wall time 222k peak context
The current loop implementation distinguishes a gate that runs red from a gate or git operation that cannot run as infrastructure. Heavy driver-loop coverage still checks that a failed git baseline stops as infrastructure and does not mark the iteration complete, while other loop tests preserve red streak and revert behavior. The files were later renamed from `engine/build` into `engine/run`, but the distinction survives in the migrated adapters and tests.
This sprint handled the product behavior behind the incident: a loop must not confuse a failed surrounding operation with an actual red invariant verdict. The brief gave the worker a crisp contract, preserving ordinary red repair and systemic halt behavior while changing baseline, execution, rollback, and cleanup failures. The run record shows no additional repair loop here, so the cut appears to have fit within one worker's context once git hardening was already present. Its boundary was semantic, not mechanical.
Driver Path Audit
2 agents 30m wall time 156k peak context
The current tree still carries stricter git reads for decision-critical cases and infrastructure-specific post-merge gate classification. A post-merge gate that cannot launch reports infrastructure instead of being treated as a red product verdict, and baseline gate tests keep real red output in the red branch. Some original audited files were reorganized later, but the surviving `engine/run` tests show the classification rule is still part of the driver surface.
With the loop path fixed, this sprint audited adjacent driver and gate surfaces for the same category error. The brief explicitly limited the work to practical shared-layer fixes, so a fresh agent could inspect the nearby paths without trying to redesign every driver. Its gates stayed green, matching a propagation slice rather than another discovery phase. The boundary was the confusion class: infrastructure failures became clearer, while real red verdicts and existing retry ladders were left alone.
Stress Evidence Closeout
2 agents 22m 4s wall time 68k peak context
The reliability guide and closeout note still point future workers at the driver-loop concurrency preset and explain why narrow focused samples are not closeout evidence for this class. Durable pre-fix and post-fix stress reports remain under the analysis snapshot directory, and the final flaky-test closeout links this incident into the broader reliability record. This sprint's surviving feature is the operational memory that keeps the same mistake from being re-measured the old way.
The final sprint returned to the first sprint's oracle and used it as the closeout instrument. Its job was not another code change but an evidence and documentation pass, comparing the corrected system against the same whole-module stress shape that exposed the failure. The run facts show a clean finish through gate and review, so the closeout did not force another change to the cut. It closed the plan by replacing the older narrow-filter story with the new reliability standard.