skip to content
Replays

Ghost Reaping

Run Ownership And Process Reaping

How Gantry learned to identify live owners, record child processes, and clean them up safely.

Gantry milestones

3 milestones 13 tasks 31 agents

6 plan 13 execute 18 review 2 fix

239k peak context

137k median execute

How Gantry learned to identify live owners, record child processes, and clean them up safely.

33h 13m total 16m 13s per task

38s plan 2h 16m execute 2h 5m review

189 edits 1.2k commands

codex harness

This run turned a process-lifetime incident into a run-management contract. The plan was split into ownership truth, durable subprocess accounting, and the operator commands that act on those facts.

The work matters because cleanup is destructive when it guesses. Gantry now has a verified owner record, an append-only process roster, stop and remove paths that share cleanup outcomes, and reporting surfaces that show the same process model without signalling anything.

How this walkthrough is structured

Feature

What did the run build and ship?

The current tree still carries the run lock, process roster, stop and remove safety paths, process readouts, and nested-start guards as one connected process-control layer.

Build

How did Gantry structure the work?

The cut moved from ownership truth to process accounting and then to operator commands, with the hard boundaries surfacing in parent-death cleanup and the first stop/reporting passes.

Milestone 1

Honest Run Ownership

3 tasks 9 agents 47m 46s wall time 187k peak context

The tree still treats `run.pid` as the flocked rendezvous file and the owner record, not as a disposable pid note. Owner queries verify process identity before reporting a run as live, and detached starts publish through the same engine path. The first removal guard is now part of a broader remove flow that consults ownership and cleanup results before deleting run data.

This milestone carried the trust foundation for every later cleanup action. Its sprints were small enough for separate agents because the first established mutual exclusion, the next strengthened the owner record, and the last consumed that query at the deletion boundary. The milestone stayed green, but its review left a plan-impact note: one remove path still needed the same live-owner protection. That note shaped the later stop and remove milestone instead of being hidden inside this foundation.

Exclusive Lock Paths

2 agents 11m 20s wall time 76k peak context

`src/engine/run_lock.rs` still keeps the run lock keyed to the resolved runtime home and keeps `run.pid` in place across release. The guard releases the flock rather than unlinking the path, so a successor reuses the same rendezvous and overwrites the stale contents only after acquiring the lock. Crash release and clean release therefore converge on the same acquisition path.

This sprint had one narrow job: make the lock path and the actual exclusive owner stop diverging during release and acquire races. That was an appropriate first slice because no later owner record can be trusted if two contenders can look at different lock targets. Its gate and review stayed green, so the boundary held without forcing a redesign.

Reliable Owner Records

2 agents 13m 34s wall time 94k peak context

Owner records now include process identity alongside the pid, and reused or unreadable identities resolve to unowned. The detach path also documents that the detached child, not its parent, writes the trusted record once it reaches engine startup. That behavior is still visible in `src/detach.rs`, `src/sys.rs`, and the owner-query code.

After exclusivity existed, this sprint could focus only on what the owner record means. The brief made a clean contract for callers: ask the ownership query and do not perform bare pid checks elsewhere. The sprint passed through the gate and review cleanly, which let later registry and remove code rely on the query rather than repeat liveness logic.

Protected Live Removal

2 agents 16m 14s wall time 187k peak context

Removal now has separate live-owner and process-safety override concepts, and ordinary confirmation does not stand in for either one. The current command path runs through shared stop cleanup before final deletion when a live or ghosting run is involved. The initial live-removal guard is therefore still present, but it is folded into the later destructive-command flow.

This sprint put the verified-owner query at the first destructive boundary. It was scoped to refusal and override semantics, leaving roster cleanup and stop-before-remove behavior for later work. The sprint itself stayed green, but the milestone review found the remaining remove entry point that still bypassed the check, so the run carried that gap forward explicitly.

Milestone 2

Durable Process Roster

4 tasks 11 agents 1h 10m wall time 173k peak context

The current tree has an append-only `process-roster.jsonl` model with launch and reap records, verified process identity, run association checks, and cleanup reports. The roster is used by stop, reporting, and MCP process views, while the sandbox documentation still names parent-death handling as secondary to the durable roster. That split between authority and backstop remains intact.

This milestone made cleanup possible after the owner record was honest. The cut separated recording from acting, then put the reaper on engine exits, then closed the parent-death investigation as a backstop rather than the authority. Most of the work stayed green; the parent-death sprint hit a red gate and needed a fix, locating the risk in the place where operating-system behavior met Gantry's cleanup guarantees.

Durable Process Roster

1 agents 16m 11s wall time 72k peak context

`src/engine/process_roster.rs` still defines launch records, structured attribution, process identity, worktree association, and append-only replay. Agent, gate, harness, and pattern-script launch sites write into that roster before continuing. The file is explicitly a join key for cleanup and reporting, not a metrics stream.

This sprint was the storage contract for process cleanup. It did not try to kill anything; it only required every Gantry-owned subprocess group to be recorded before the caller moved past spawn. That made the work self-contained for a fresh agent and gave later sprints a replayable source of truth.

Verified Reaping

2 agents 13m 57s wall time 113k peak context

The reaper still derives unreaped groups by replaying the roster and appends matching reap records when cleanup settles. It skips mismatched process identities and run-association mismatches instead of signalling a process that merely inherited a number. Cleanup reports preserve skipped entries and errors so callers can refuse to claim a run is clean.

With launch records in place, this sprint could define what it means to act on them safely. The brief centered on identity and run-association checks, bounded graceful termination, forced cleanup, and appended reap records. It passed cleanly, so later exit and stop paths could call one verified reaper rather than maintain per-spawn cleanup code.

Exit Sweep Bounds

2 agents 20m 41s wall time 173k peak context

Engine paths still record spawned agent, gate, and pattern-script groups and sweep them through the shared roster cleanup on exit. The process roster exposes bounded graceful-then-forced cleanup, and the run paths call into it instead of relying on scattered per-child cleanup. Remaining unreaped entries after an unclean engine death are treated as facts for later stop/reporting paths.

This sprint moved the reaper onto engine shutdown paths and the aborted-run hang class. The work was larger because it had to cover success, abort, signal shutdown, timeouts, and error returns without letting an ignored graceful signal block forever. Its boundary held through review and gate, making clean engine exit a caller-visible contract for later commands.

Parent Death Check

3 agents 13m 4s wall time 98k peak context 1 fix

The sandbox documentation still says parent-death behavior is a backstop, and the code sets a Linux parent-death signal for agent launches where that applies. Tests in the agent-launch area cover the thread-scoped behavior instead of treating it as a process-wide guarantee. The current docs point maintainers back to the roster and verified reaping as the cleanup authority.

This sprint carried the investigation that could not be reduced to code plumbing alone. Its brief required executable or written evidence about sandboxed and unsandboxed parent-death behavior, while preserving the roster as the authoritative safety mechanism. The gate went red after review and needed a fix, which fits the task: the brittle part was proving the backstop without overstating what it guarantees.

Milestone 3

Stop And Start Guards

6 tasks 20 agents 3h 41m wall time 239k peak context

The tree now exposes `gantry stop`, guarded `gantry remove`, `gantry ps`, status process lines, MCP process payloads, and start guards from the same underlying owner and roster facts. Live and dead-owner cases share structured stop outcomes, while discovery paths render process state without acting on it. Nested starts are refused before the engine begins unless the scoped override is explicit.

This milestone put operator workflows on top of the ownership and roster layers. It began with the live stop command, completed the shared outcome vocabulary, added dead-owner cleanup, routed remove through the same semantics, surfaced process facts, and finally blocked accidental nested starts. The first stop-control unit needed repeated review and replan attention, and reporting later hit a red gate, showing that the hard work was in making the new safety model visible and resumable without inventing side channels.

Live Stop Control

7 agents 54m 43s wall time 239k peak context 1 replan

`gantry stop <name>` still sends a correlated stop request through the attach endpoint when a live owner exists. The domain layer carries a stop request id, safety policy, owner identity, and structured outcome, and the engine command channel has a stop command distinct from local attach quit. A lost reply and a non-contactable owner are reported as different failures.

This sprint had the riskiest interface boundary in the final milestone: a new operator command had to reach the live owner through the attach-control channel and stop resumably. The run facts show repeated attempts around this unit, including review errors and a replan error before it settled. That stress belongs here because the sprint was defining transport, correlation, and engine wind-down semantics that every later stop case would inherit.

Shared Stop Result

2 agents 46m 8s wall time 233k peak context

`StopOutcome` still includes stopped, already-stopped, refused, partially-cleaned, and failed cases, all carrying cleanup facts. The process roster converts cleanup reports into those outcomes, including surviving processes, errors, and next operator action. Live stop therefore reports settled cleanup state instead of an intention to clean later.

Once the live transport existed, this sprint completed the result vocabulary rather than reopening the transport. Its brief made cleanup truth the contract: a stop outcome could not say success while child process cleanup was partial or failed. The clean boundary was that later dead-owner and remove work would consume the same shape, so this agent only had to finish the live-owner truth model.

Dead Owner Stop

2 agents 23m 30s wall time 145k peak context

When no live owner is recorded, `gantry stop` now calls the roster reaper directly and reports already-stopped, partial, failed, or stopped outcomes from the cleanup report. Identity mismatches and stale roster entries are skipped rather than signalled. That makes stop the command for both live shutdown and dead-owner ghost cleanup.

This sprint extended stop from live-owner communication to offline cleanup when the owner was gone. The brief was sharply scoped around the roster: verify each surviving group, reap what still belongs to the run, and report the same outcome vocabulary as the live path. It stayed green because the previous sprints had already provided both the reaper and the result shape.

Remove Usage Stop

2 agents 37m 9s wall time 191k peak context

The remove command path now calls stop cleanup before deleting run data and bases the proceed-or-refuse decision on the structured outcome. Live-owner override and process-safety override are separate flags, and cleanup failure blocks deletion unless the relevant safety override is explicit. Both by-name removal and plan removal flow through the same safety model.

This sprint returned to the deletion gap noted after the first milestone and made remove consume the shared stop operation. Its slice was destructive-command policy: refuse live-owned runs by default, run cleanup before deletion, and require explicit process-safety override when survivors remain. The run stayed green because stop semantics were already defined and could be reused instead of restated.

Running Status Reporting

3 agents 32m 20s wall time 164k peak context 1 fix

`src/process_view.rs` is now the shared process model for terminal status, `gantry ps`, the build-run process overlay, and MCP payloads. It verifies recorded group identity before calling a process live, distinguishes no process data from a real empty roster, and renders `gantry stop <name>` only as guidance. `gantry list` remains intentionally free of process columns.

This sprint made the safety model observable while keeping discovery read-only. Its brief required human and machine-readable status to show owner identity, surviving tracked processes, cleanup status, and stop guidance without signalling anything. The gate went red and then fixed, locating the difficulty in renderer parity and non-destructive reporting rather than in the underlying reaper.

Start Guard Checks

1 agents 18m 36s wall time 61k peak context

Startup now checks the current directory and Gantry's agent marker before building a new run. Agents receive the marker in their environment, and `--allow-nested-run` bypasses only the start guard rather than live-owner or process-safety checks. The CLI docs and diagnostics both say what was detected and how to override intentional nesting.

The final sprint prevented the incident class from recurring after operators had a cleanup path. Its brief was intentionally narrow: detect starts under managed worktree ancestry or from a Gantry-launched agent environment, fail before the engine starts, and document a scoped override. That made it independent of stop internals, and it finished without further stress.