muse-crew 0.7.5 → 0.7.7

package/API.md

@@ -293,6 +293,11 @@ Returns `{ ok: true, runs: [...], events: [...] }`.
 
 The platform records workflow run status in `runtime.workflow_runs` — separate from our telemetry. When a workflow dies on a fatal `agent()` failure (e.g. "subagent bootstrap is no longer authorized"), our telemetry shows a run with no events, but the platform knows why. This monitor correlates platform failures with our tasks.
 
+Two layers, in order of preference:
+
+1. **Prevention — the launcher stays alive.** Async workflow `agent()` authorization is tied to the launcher's lifetime: if the cron tick ends while a workflow is still running, the workflow's next `agent()` call fails. So the poll tick runs with a 90-minute execution timeout (`timeout_secs: 5400` in `seed/crons.json`) and its Step 5 monitor stays alive until every launched run reaches a terminal state. A launcher that outlives its runs never triggers the failure in the first place.
+2. **Recovery — per-tick correlation through the durable mapping.** If the launcher dies early anyway (platform kill, cell recycle), the next tick's Step 0 finds the dead run, `record-platform-failure` correlates it via the `platform_run_tasks` mapping, and `retry-platform-failure` requeues the task — at most ~3 minutes later, with the ghost session settled so the dispatcher treats it as a retry candidate immediately.
+
 ### `record-platform-failure`
 
 Record a platform workflow run failure. Correlates the task in this order: (1) the durable `platform_run_tasks` mapping written by `acknowledge-dispatch-run`, (2) timestamp proximity (±60s) between `platform_created_at` and crew telemetry launch time (fallback for runs launched before the mapping existed), (3) an explicit `crew_run_id`/`task_id` when provided. A failure that correlates to no task is kept with `task_id: null` and reported as skipped by retry.
@@ -324,6 +329,12 @@ Returns `{ ok: true, failures: [...] }`.
 
 Retry a task whose workflow died on a platform failure. Clears the stale reservation, re-queues the task to `todo`, and increments the retry count. If `retry_count` >= `max_retries` (default 3), parks the task instead.
 
+**Ghost-session settlement:** the dead workflow's `agent_sessions` row still says `running`, and the dispatcher skips any task whose latest session is running ("work in flight"). The retry settles it atomically with the state change, so the task is redispatchable on the very next tick instead of waiting out the 1-hour zombie-session sweep:
+- on `requeued`: running sessions → `stalled` (the dispatcher already treats `stalled` as a retry candidate);
+- on `parked`: running sessions → `failed` (what `recover-task` accepts), same as `park-task`.
+
+Returns `{ ok: true, action: "requeued"|"parked"|"skipped", ..., settled_sessions: <n> }`.
+
 **Supersede guard:** if a newer platform run has been linked for this task since this failure's run, the task is already owned by the successor — the retry is skipped instead of clobbering live work (a late detection of an old dead run never resets a redispatched task).
 
 | Field | Type | Notes |

package/lib/crew-api.js

@@ -847,14 +847,32 @@ commands["retry-platform-failure"] = (db, args) => {
     if (task && task.state !== "parked") {
       const msg = "Platform workflow failed " + (failure.retry_count + 1) + "x: " +
         failure.error_message.substring(0, 200);
-      db.prepare("UPDATE tasks SET state = 'parked', updated_at = datetime('now') WHERE id = ?")
-        .run(failure.task_id);
-      db.prepare("DELETE FROM dispatch_reservations WHERE task_id = ?").run(failure.task_id);
-      db.prepare(
-        `INSERT INTO events (task_id, event_type, message) VALUES (?, 'parked', ?)`
-      ).run(failure.task_id, msg);
+      const ts = new Date().toISOString();
+      db.exec("BEGIN");
+      try {
+        db.prepare("UPDATE tasks SET state = 'parked', updated_at = datetime('now') WHERE id = ?")
+          .run(failure.task_id);
+        db.prepare("DELETE FROM dispatch_reservations WHERE task_id = ?").run(failure.task_id);
+        // Settle the ghost session: a parked task never keeps a running
+        // session, and 'failed' is what recover-task accepts.
+        const settled = db.prepare(
+          `UPDATE agent_sessions
+           SET status = 'failed', ended_at = ?,
+               notes = COALESCE(notes, '') || ' Parked: platform retry exhausted.'
+           WHERE task_id = ? AND status = 'running'`
+        ).run(ts, failure.task_id);
+        db.prepare(
+          `INSERT INTO events (id, type, task_id, identity, message, timestamp)
+           VALUES (?, 'note', ?, NULL, ?, ?)`
+        ).run(uuid(), failure.task_id, "Parked: " + msg, ts);
+        db.exec("COMMIT");
+        return { ok: true, action: "parked", reason: "max retries exceeded", retry_count: failure.retry_count, settled_sessions: Number(settled.changes) };
+      } catch (e) {
+        db.exec("ROLLBACK");
+        throw e;
+      }
     }
-    return { ok: true, action: "parked", reason: "max retries exceeded", retry_count: failure.retry_count };
+    return { ok: true, action: "parked", reason: "max retries exceeded", retry_count: failure.retry_count, settled_sessions: 0 };
   }
   // Clear reservation and re-queue for retry.
   db.exec("BEGIN");
@@ -863,22 +881,36 @@ commands["retry-platform-failure"] = (db, args) => {
     db.prepare(
       "UPDATE tasks SET state = 'todo', updated_at = datetime('now') WHERE id = ? AND state != 'done'"
     ).run(failure.task_id);
+    // Settle the ghost session atomically with the requeue: the platform
+    // workflow is dead, but its agent_sessions row still says 'running' —
+    // and the dispatcher skips any task whose latest session is running
+    // (crew-dispatch.js "work in flight"). Without this, the requeued task
+    // would sit until the 1-hour zombie-session sweep. 'stalled' is what
+    // the dispatcher already treats as a retry candidate.
+    const ts = new Date().toISOString();
+    const settled = db.prepare(
+      `UPDATE agent_sessions
+       SET status = 'stalled', ended_at = ?,
+           notes = COALESCE(notes, '') || ' Platform retry: launcher died; session settled for redispatch.'
+       WHERE task_id = ? AND status = 'running'`
+    ).run(ts, failure.task_id);
     db.prepare(
       `UPDATE platform_run_failures
        SET retry_count = retry_count + 1, last_retry_at = datetime('now')
        WHERE platform_run_id = ?`
     ).run(args.platform_run_id);
     db.exec("COMMIT");
+    return {
+      ok: true,
+      action: "requeued",
+      task_id: failure.task_id,
+      retry_count: failure.retry_count + 1,
+      settled_sessions: Number(settled.changes),
+    };
   } catch (e) {
     db.exec("ROLLBACK");
     throw e;
   }
-  return {
-    ok: true,
-    action: "requeued",
-    task_id: failure.task_id,
-    retry_count: failure.retry_count + 1,
-  };
 };
 
 commands["park-task"] = (db, args) => {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "muse-crew",
-  "version": "0.7.5",
+  "version": "0.7.7",
   "description": "Opinionated orchestration for Muse — workflows, identities, and tooling for autonomous software development.",
   "license": "UNLICENSED",
   "private": false,

package/seed/cron-body-template.md

@@ -13,7 +13,7 @@ You are the dispatch trigger for Muse Crew. Run the authoritative dispatcher wor
      WHERE w.created_at > now() - interval '15 minutes' AND w.status = 'failed'
      ORDER BY w.created_at DESC
      ```
-   - For each failed run, record it in the crew DB (correlates by timestamp with our telemetry):
+   - For each failed run, record it in the crew DB (correlates via the durable platform run -> task mapping written at acknowledge time; timestamp proximity is the fallback for pre-mapping runs):
      `node {crewHome}/lib/crew-api.js --crew-home {crewHome} record-platform-failure --json '{"platform_run_id": "<run_id>", "error_message": "<error>", "platform_created_at": "<created_at>"}'`
    - Then retry each failure (clears stale reservation, re-queues task, or parks after 3 attempts):
      `node {crewHome}/lib/crew-api.js --crew-home {crewHome} retry-platform-failure --json '{"platform_run_id": "<run_id>"}'`
@@ -36,24 +36,22 @@ You are the dispatch trigger for Muse Crew. Run the authoritative dispatcher wor
 
    If the dispatcher returned no claims or the claims array is empty, report: NO_DISPATCH and exit.
 
-5. **Monitor launched workflows (launcher stays alive — platform workaround):** The platform ties async workflow subagent authorization to the launcher's lifetime. If you exit now, the workflow's next `agent()` call may fail with "subagent bootstrap is no longer authorized." You are the monitor — stay alive until each workflow reaches a terminal state.
-   - For each launched run_id, poll its status every 2 minutes via muse.db:
+5. **Monitor launched workflows until terminal (stay-alive — 2026-09-13):** The platform ties async workflow `agent()` authorization to the launcher's lifetime: if THIS tick ends while a workflow is still running, the workflow's next `agent()` call fails with "subagent bootstrap is no longer authorized" / "subagent reservation owner is terminal". Prevention beats recovery here, so this tick is configured with a 90-minute execution timeout (`timeout_secs: 5400` in seed/crons.json) and you MUST stay alive until every launched run reaches a terminal state. Do not exit early while a launched run is still `running` — your death is what kills it.
+   - For each launched run_id, poll its status every ~2 minutes via muse.db:
      ```sql
      SELECT status, error FROM runtime.workflow_runs WHERE run_id = '<run_id>'
      ```
    - **If status is `completed`:** Done. Log success and stop monitoring this run.
    - **If status is `failed`:** Check if it's a platform `agent()` error (error contains "subagent bootstrap", "reservation owner is terminal", "bootstrap was cancelled", or "workflow agent call failed"):
-     - **Platform error:** Record it and retry:
-       1. `node {crewHome}/lib/crew-api.js --crew-home {crewHome} record-platform-failure --json '{"platform_run_id": "<run_id>", "error_message": "<error>"}'`
-       2. `node {crewHome}/lib/crew-api.js --crew-home {crewHome} retry-platform-failure --json '{"platform_run_id": "<run_id>"}'`
-       3. If the retry result says `requeued` and you have retries remaining (max 3 per task per tick):
+     - **Platform error:** Record it and retry immediately (same two commands as Step 0). If the retry says `requeued` and retries remain (max 3 per task):
           - Re-acquire the reservation: `node {crewHome}/lib/crew-api.js --crew-home {crewHome} reserve-dispatch --json '{"task_id": "<task_id>"}'`
           - If `acquired` is true, re-launch via workflow_launch_async with the same scriptPath and args, acknowledge with the new run_id, and continue monitoring the NEW run_id.
           - If `acquired` is false, stop — another dispatcher claimed it.
-       4. If the retry result says `parked` (3 attempts exhausted), stop monitoring this task.
+       If the retry result says `parked` (attempts exhausted), stop monitoring this task.
      - **Task-level error (not a platform error):** The workflow's own error handling applies. Stop monitoring this run.
    - **If status is `running` or `paused`:** Continue polling.
-   - **Timeout:** If a workflow hasn't reached terminal state after 90 minutes, log it and stop monitoring (the next tick's Step 0 will catch it if it failed). The monitor timeout MUST exceed the longest workflow `agent()` timeout (currently 60 minutes for the work phase) — otherwise the launcher shuts down mid-workflow and the bug recurs.
+   - **Monitor ceiling:** 90 minutes from tick start. If a run is still not terminal then (pathological — the work phase caps each `agent()` call at 60 minutes), exit; the next tick's Step 0 continues recovery through the durable mapping.
+   - **Backstop (not the plan, the insurance):** If THIS launcher dies early for any reason (platform kill, cell recycle), the next tick's Step 0 detects the dead run through the durable platform run -> task mapping and retries it — at most ~3 minutes later. The mapping exists so a dead launcher never strands a task for an hour; the monitor exists so the launcher rarely dies mid-run in the first place.
    - When all launched workflows are terminal or retry-exhausted, exit silently.
 
 6. Exit silently.

package/seed/crons.json

@@ -10,7 +10,7 @@
         "every": "3m",
         "kind": "interval"
       },
-      "timeout_secs": 120,
+      "timeout_secs": 5400,
       "title": "Muse Crew polling loop"
     }
   ],