The Run That Never Finished
💥 The problem: One-shot-orchestra rarely finished a build clean. It lost track of its own agents halfway through and shipped at half score.
🔧 The fix: Stop making the AI steer the run by hand. One command,
orchestra drive, runs the whole state machine to the finish.✅ The result: A run drove through all 10 phases untouched, then auto-healed the one stall that still snuck through at the end.
One-shot-orchestra is our heaviest execution mode. You hand it a task, and it breaks the work across a fleet of fresh AI workers: one to diagnose, several to research and build in parallel, more to test, harden, and score the result.
It produced great work when it finished. The problem was the "when." A user summed it up in one sentence: it rarely makes it to the end of a build completely clean.
Why a build rarely reached the finish line
The receipts were in the logbook. Orchestra records every completed run to a file, and the two full real-world runs on record both told the same story.
Both shipped at a composite score of 0.50 out of 1.00, flagged "max-attempts-reached." That is the system's way of saying it gave up and shipped whatever it had.
| Run | What the log said |
|---|---|
| Migration build | Shipped 0.50 — "spawn pipeline silently failed, no worker process" |
| Three.js game build | Shipped 0.50 — "refusing to spawn worker" under hardware throttle |
| The symptom | Loses track of agents and build stages, rarely finishes clean |
| The demand | Work cleanly every time |
A half-score ship is worse than a crash. A crash tells you something broke. A half-score ship hands you work that looks finished, wears a passing label the system stopped believing, and quietly buries the fact that it gave up. The run did not fail loudly; it faded out.
The AI was steering by hand
Under the hood, orchestra is a small program (a daemon) that owns all the mechanical work. The AI session on top is supposed to be a thin narrator: it calls the program, reads back a status, and reacts.
But there was no single command that said "run this to the end." So the narrator had to drive the whole thing one tick at a time: read the state, decide whether to spawn the next worker or wait, call the right command, keep the run's ID straight, and never stop early.
A full run is 100 to 300 of those manual ticks. Every tick spends a little of the AI's working memory. By tick 200, that memory is crowded with status lines, and the narrator loses the thread, exactly the "loses track" symptom.
↓
🤔 is this a "spawn" state or a "wait" state?
↓
🖥️ call the right command, keep the run-id straight
↓
🔁 repeat 100 to 300 times without losing count
↓
💥 working memory fills, the narrator loses its place
It is like reading a 300-line recipe out loud, one line at a time, and doing exactly what each line says without ever losing your place. Miss one line near the end and dinner is ruined, not because the recipe was wrong, but because a human (or an AI) was asked to be a machine.
Drive the run yourself below. Each click is one manual tick the old narrator had to make by hand. Watch the counter climb, then see what one command does instead.
A run has 10 build phases plus a scoring gate. Click Tick to advance it the old way.
One command drives the whole run
The fix was to move the loop off the AI and into code. A new command, orchestra drive, does inside the program exactly what the narrator used to do by hand, but perfectly, every time.
It classifies each state with one authoritative rule set, spawns or waits as needed, and keeps going until the run is done. The narrator now makes a handful of calls instead of hundreds.
orchestra drive <run>
↓
🤖 drive loops internally: status → spawn → wait → repeat
↓
⏳ every ~4 minutes it hands back "still running, call me again"
↓
🛡️ if it stalls, it diagnoses and heals itself first
↓
✅ it returns only on delivered, failed, or a question for you
The narrator went from driver to passenger. Old way: hold the wheel for all 300 turns and pray you do not blink. New way: tell the car the destination and let it drive, checking in every few minutes. Same road, but the machine handles the road.
The old, hand-written instructions had a second, quieter bug: the list of "spawn" states was incomplete. It left out the pre-delivery quality gate that every run passes through. A narrator following it to the letter would stall there on every single run. Owning one rule set in code deleted that whole class of mistake.
Then a run hung at the very end
To prove the fix, we ran a real build and drove it. It cruised through all 10 phases: diagnose, research, build, test, harden, document, verify, visual, polish. Then it stopped dead at the scoring gate.
Scoring works by spawning one grader per quality dimension. Seven graders fired. Six came back. The seventh left no result file at all, and the run sat there waiting for a file that would never arrive.
Here is why the file vanished. Every worker has a watchdog whose job is to notice if the worker dies and write a "failed" note in its place, so the run can move on. This machine's background cleanup swept away both the grader and its watchdog at the same moment.
No worker, no watchdog, no note. The gate waited on a ghost.
Teaching it to heal the stall
Drive already noticed the freeze: it watches for on-disk progress and flags a run that has gone quiet for too long. But noticing is not fixing. So we taught the healer to recognize this exact shape and repair it.
When drive finds the gate stalled on an absent grader, it writes the note the dead watchdog never got to write. That unblocks the gate, and the run's normal retry-and-finish logic takes over from there.
↓
🩺 healer: the scoring gate is missing one grader's note
↓
✍️ repair: write the note the dead watchdog never wrote
↓
🔁 gate unblocks, normal retry logic finishes the run
↓
✅ delivered — no human touched it
Think of a relay race where a runner collapses and drops the baton. The old system froze at the empty lane, waiting for a runner who would never come. Now a judge steps in, marks that leg "did not finish," and lets the race conclude, instead of leaving everyone standing on the track forever.
We also removed the hidden dependency that made the watchdog fragile: its "failed" note used to require Python to be installed. If Python was missing, no note, silent hang. It now falls back to the runtime the program already ships with, so the note always gets written.
What shipped
Two releases, both proven against real runs and a full offline test suite before they went out. Here is the receipt, not a brag.
| Metric | Value |
|---|---|
| Releases | v0.32.0 (the drive command) + v0.33.0 (scoring self-heal) |
| Files touched | 16 across the two commits |
| Lines added | ~1,013 (much of it tests) |
| New test checks | 74 (drive: 51, scoring-stall: 18, watchdog fallback: 5) |
| Prior test suites | All green — zero regressions |
| Live proof | Drove all 10 phases, then auto-recovered the scoring stall |
The honest part: we scoped the self-heal to the scoring gate, because that is the one place the freeze actually reproduced. The same "worker and watchdog both swept" freeze could, in theory, hit other phases too.
What we fixed for real
- The hand-cranked loop, replaced by one deterministic command.
- The incomplete state list that stalled every run at the quality gate.
- The scoring freeze, now detected and auto-healed end to end.
- The watchdog's hidden Python dependency.
What we did not pretend to fix
- Auto-heal for the same freeze in other phases — not reproduced, so not risked.
- Drive still surfaces those rarer stalls to you rather than hiding them.
- We named the gap out loud instead of quietly shipping around it.
That gap is a clean, separate job for a day it actually bites. Shipping the fix we could prove, and naming the one we could not, is the whole point of auditing our own tools the way we audit a build.
Max-effort execution that finishes clean
One-shot-orchestra is the heavy end of the Godmode toolkit. Start with the free tier and feel the difference a real quality loop makes.
Get Godmode Lite (Free) How the orchestrator became a daemon