muse-crew 0.7.1 → 0.7.3

package/API.md

@@ -190,6 +190,151 @@ Two namespaces, not one: the vocabulary above is the **dashboard API** (task sta
 
 Record that a poll tick occurred. Takes no arguments. Used by the dispatcher at the end of each cycle to update the last-polled timestamp.
 
+### `reserve-dispatch`
+
+Create a dispatch reservation for a task. Called by the dispatcher workflow BEFORE the worker launches the workflow — the reservation is the atomic claim on the task.
+
+**Atomic:** Uses `INSERT ... ON CONFLICT DO NOTHING`. Returns `acquired: true` if this call created the reservation, `acquired: false` if one already exists. The dispatcher only recommends tasks where `acquired` is true — fail closed on contention.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `task_id` | string | Required. The task being dispatched. |
+| `run_id` | string | Required. Use `"pending-dispatch"` as the placeholder; the worker updates it with the real platform run ID via `acknowledge-dispatch-run`. |
+| `workflow` | string | Required. Which workflow will be launched (`standard`, `bugfix`, `chore`). |
+| `ttl_seconds` | integer | Optional. Reservation lifetime, clamped to 60–3600 seconds. Default 900 (15 minutes). |
+
+Returns `{ ok: true, acquired: <boolean>, task_id, run_id, workflow, dispatched_at, expires_at }`.
+
+### `acknowledge-dispatch-run`
+
+Update a dispatch reservation with the real platform workflow run ID. Called by the cron worker immediately after `workflow_launch_async` returns.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `task_id` | string | Required. The task that was dispatched. |
+| `run_id` | string | Required. The real platform run ID (e.g. `workflow-run-xxx`). Must not be `"pending-dispatch"`. |
+
+Fails closed: throws if no reservation exists for the task (launching without a reservation is forbidden). Returns `{ ok: true, task_id, run_id }`.
+
+### `clear-reservation`
+
+Remove a task's dispatch reservation. Called by the workflow after a successful self-claim. Takes `task_id` (required). Returns `{ ok: true, cleared: <boolean> }` — `cleared` is false if no reservation existed (idempotent).
+
+### `list-reservations`
+
+List all active (unexpired) dispatch reservations, ordered by dispatch time. Takes no arguments. The dispatcher consumes this via `getdispatchstate` (which includes `reservations`) rather than calling it directly.
+
+---
+
+## Workflow Telemetry
+
+Structured timeline for each launched workflow run. The workflow records its start, milestones, and terminal state. Timestamps come from SQLite `datetime('now')` — workflow JS never reads the wall clock.
+
+**Blast radius note (2026-09-13):** Every `agent()` call risks a fatal platform subagent-spawning failure. Telemetry batches non-critical events in memory and flushes them in as few `agent()` calls as possible. Critical events (start, claim, end) flush immediately.
+
+### `record-run-start`
+
+Record the start of a workflow run. The API mints the `run_id` (UUID, SQLite-side) — the workflow doesn't know its platform run ID.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `task_id` | string | Required. |
+| `workflow` | string | Required. |
+| `launched_by` | string | Optional. Defaults to `"unknown"`. |
+
+Returns `{ ok: true, run_id }`.
+
+### `record-run-event`
+
+Record a single telemetry event. Prefer `record-run-events-batch` for multiple events.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `run_id` | string | Required. |
+| `task_id` | string | Required. |
+| `event_name` | string | Required. |
+| `detail` | string | Optional. |
+
+### `record-run-events-batch`
+
+Record multiple telemetry events in a single call. Used by the workflow's batched flush.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `run_id` | string | Required. |
+| `task_id` | string | Required. |
+| `events` | array | Required. Each element: `{ event_name: string, detail?: string }`. |
+
+Returns `{ ok: true, count }`.
+
+### `record-run-end`
+
+Record the terminal state of a workflow run.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `run_id` | string | Required. |
+| `status` | string | Required. One of `completed`, `failed`, `parked`, `timed_out`. |
+
+### `get-run-timeline`
+
+Get a workflow run (or all runs for a task) with its events.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `run_id` | string | Optional. Exactly one of `run_id` or `task_id` is required. |
+| `task_id` | string | Optional. |
+
+Returns `{ ok: true, runs: [...], events: [...] }`.
+
+---
+
+## Platform Failure Monitor
+
+The platform records workflow run status in `runtime.workflow_runs` — separate from our telemetry. When a workflow dies on a fatal `agent()` failure (e.g. "subagent bootstrap is no longer authorized"), our telemetry shows a run with no events, but the platform knows why. This monitor correlates platform failures with our tasks.
+
+### `record-platform-failure`
+
+Record a platform workflow run failure. Correlates with a crew telemetry run by timestamp proximity (±60s) if `crew_run_id` is not provided.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `platform_run_id` | string | Required. E.g. `workflow-run-xxx`. |
+| `error_message` | string | Required. The platform error. |
+| `platform_created_at` | string | Required. ISO timestamp of the platform run. |
+| `crew_run_id` | string | Optional. Skips timestamp correlation if provided. |
+| `task_id` | string | Optional. |
+| `workflow` | string | Optional. |
+
+Returns `{ ok: true, platform_run_id, crew_run_id, task_id }`.
+
+### `get-platform-failures`
+
+List recorded platform failures.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `task_id` | string | Optional. Filter by task. |
+| `platform_run_id` | string | Optional. Get a specific failure. |
+| `limit` | integer | Optional. Default 50, max 200. |
+
+Returns `{ ok: true, failures: [...] }`.
+
+### `retry-platform-failure`
+
+Retry a task whose workflow died on a platform failure. Clears the stale reservation, re-queues the task to `todo`, and increments the retry count. If `retry_count` >= `max_retries` (default 3), parks the task instead.
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `platform_run_id` | string | Required. |
+| `max_retries` | integer | Optional. Default 3. |
+
+Returns `{ ok: true, action: "requeued"|"parked"|"skipped", ... }`.
+
+### `mark-failure-retried`
+
+Manually increment a failure's retry count without re-queuing. Takes `platform_run_id` (required).
+
 ---
 
 ## Projects

package/lib/crew-api.js

@@ -371,6 +371,15 @@ commands["get-dispatch-state"] = (db) => {
       total_sessions: sessionCount,
       total_events: eventCount,
     },
+    // Active dispatch reservations: tasks the worker launched but whose
+    // workflow has not yet self-claimed. The dispatcher skips these.
+    reservations: db.prepare(
+      `SELECT task_id, run_id, workflow, dispatched_at, expires_at
+       FROM dispatch_reservations WHERE expires_at > ? ORDER BY dispatched_at`
+    ).all(now()).map((r) => ({
+      task_id: r.task_id, run_id: r.run_id, workflow: r.workflow,
+      dispatched_at: r.dispatched_at, expires_at: r.expires_at,
+    })),
   };
 };
 
@@ -562,6 +571,270 @@ commands["claim-task"] = (db, args) => {
   return { ok: true, claimed: false, reason: "already_claimed", existing_session_id: existing.id };
 };
 
+// Dispatch reservations: the cron worker creates one after launching a
+// workflow for a task; the dispatcher skips reserved tasks on the next tick;
+// the workflow clears it on self-claim. Expired reservations are inert.
+commands["reserve-dispatch"] = (db, args) => {
+  if (!args.task_id) throw usageError("task_id is required.");
+  if (!args.run_id) throw usageError("run_id is required.");
+  if (!args.workflow) throw usageError("workflow is required.");
+  requireTask(db, args.task_id);
+  const ttl = Math.max(60, Math.min(3600, Number(args.ttl_seconds) || 900));
+  const dispatchedAt = now();
+  const expiresAt = new Date(Date.now() + ttl * 1000).toISOString();
+  // ATOMIC: INSERT with ON CONFLICT DO NOTHING. If a reservation already
+  // exists (even expired — cleanup is separate), we do NOT overwrite it.
+  // The caller must check `acquired` and fail closed if false.
+  // This prevents the last-write-wins race where two dispatchers both
+  // think they own the task.
+  const info = db.prepare(
+    `INSERT INTO dispatch_reservations (task_id, run_id, workflow, dispatched_at, expires_at)
+     VALUES (?, ?, ?, ?, ?)
+     ON CONFLICT(task_id) DO NOTHING`
+  ).run(args.task_id, args.run_id, args.workflow, dispatchedAt, expiresAt);
+  const acquired = info.changes > 0;
+  return {
+    ok: true,
+    acquired,
+    task_id: args.task_id,
+    run_id: args.run_id,
+    workflow: args.workflow,
+    dispatched_at: dispatchedAt,
+    expires_at: expiresAt,
+  };
+};
+
+commands["acknowledge-dispatch-run"] = (db, args) => {
+  // The worker calls this after workflow_launch_async returns the real
+  // platform run_id. Updates the placeholder reservation with the actual
+  // run ID. Fails closed if no reservation exists (worker must not have
+  // launched without acquiring one).
+  if (!args.task_id) throw usageError("task_id is required.");
+  if (!args.run_id) throw usageError("run_id is required.");
+  if (args.run_id === "pending-dispatch") {
+    throw usageError("run_id must be the real platform run ID, not the placeholder.");
+  }
+  const info = db.prepare(
+    `UPDATE dispatch_reservations SET run_id = ? WHERE task_id = ?`
+  ).run(args.run_id, args.task_id);
+  if (info.changes === 0) {
+    throw usageError("no reservation exists for task_id — launch without reservation is forbidden.");
+  }
+  return { ok: true, task_id: args.task_id, run_id: args.run_id };
+};
+
+commands["clear-reservation"] = (db, args) => {
+  if (!args.task_id) throw usageError("task_id is required.");
+  const info = db.prepare("DELETE FROM dispatch_reservations WHERE task_id = ?").run(args.task_id);
+  return { ok: true, cleared: info.changes > 0 };
+};
+
+commands["list-reservations"] = (db) => {
+  const rows = db.prepare(
+    `SELECT task_id, run_id, workflow, dispatched_at, expires_at
+     FROM dispatch_reservations WHERE expires_at > ? ORDER BY dispatched_at`
+  ).all(now());
+  return { ok: true, reservations: rows };
+};
+
+// Workflow run telemetry (2026-09-13): structured timeline for each launched
+// run. The workflow records its start, key milestones (pin, claim, phase
+// boundaries), and its terminal state. Timestamps come from SQLite
+// (datetime('now')) — workflow JS never reads the wall clock.
+commands["record-run-start"] = (db, args) => {
+  if (!args.task_id) throw usageError("task_id is required.");
+  if (!args.workflow) throw usageError("workflow is required.");
+  // The workflow doesn't know its platform run_id; the telemetry run_id is
+  // minted here (SQLite-side) so workflow JS never touches randomness.
+  const runId = args.run_id || uuid();
+  db.prepare(
+    `INSERT INTO workflow_runs (run_id, task_id, workflow, launched_by)
+     VALUES (?, ?, ?, ?)
+     ON CONFLICT(run_id) DO NOTHING`
+  ).run(runId, args.task_id, args.workflow, args.launched_by || "unknown");
+  return { ok: true, run_id: runId };
+};
+
+commands["record-run-event"] = (db, args) => {
+  if (!args.run_id) throw usageError("run_id is required.");
+  if (!args.task_id) throw usageError("task_id is required.");
+  if (!args.event_name) throw usageError("event_name is required.");
+  db.prepare(
+    `INSERT INTO run_events (run_id, task_id, event_name, detail)
+     VALUES (?, ?, ?, ?)`
+  ).run(args.run_id, args.task_id, args.event_name, args.detail || "");
+  return { ok: true };
+};
+
+commands["record-run-events-batch"] = (db, args) => {
+  if (!args.run_id) throw usageError("run_id is required.");
+  if (!args.task_id) throw usageError("task_id is required.");
+  if (!Array.isArray(args.events)) throw usageError("events must be an array.");
+  const stmt = db.prepare(
+    `INSERT INTO run_events (run_id, task_id, event_name, detail)
+     VALUES (?, ?, ?, ?)`
+  );
+  db.exec("BEGIN");
+  try {
+    for (const e of args.events) {
+      if (!e.event_name) continue;
+      stmt.run(args.run_id, args.task_id, e.event_name, e.detail || "");
+    }
+    db.exec("COMMIT");
+  } catch (err) {
+    db.exec("ROLLBACK");
+    throw err;
+  }
+  return { ok: true, count: args.events.length };
+};
+
+commands["record-run-end"] = (db, args) => {
+  if (!args.run_id) throw usageError("run_id is required.");
+  const status = args.status || "completed";
+  if (!["completed", "failed", "parked", "timed_out"].includes(status)) {
+    throw usageError("status must be completed, failed, parked, or timed_out.");
+  }
+  db.prepare(
+    `UPDATE workflow_runs SET status = ?, completed_at = datetime('now')
+     WHERE run_id = ?`
+  ).run(status, args.run_id);
+  return { ok: true, run_id: args.run_id, status };
+};
+
+commands["get-run-timeline"] = (db, args) => {
+  if (!args.run_id && !args.task_id) throw usageError("run_id or task_id is required.");
+  let runs;
+  if (args.run_id) {
+    runs = db.prepare("SELECT * FROM workflow_runs WHERE run_id = ?").all(args.run_id);
+  } else {
+    runs = db.prepare(
+      "SELECT * FROM workflow_runs WHERE task_id = ? ORDER BY launched_at"
+    ).all(args.task_id);
+  }
+  const runIds = runs.map((r) => r.run_id);
+  let events = [];
+  if (runIds.length > 0) {
+    const placeholders = runIds.map(() => "?").join(",");
+    events = db.prepare(
+      `SELECT run_id, event_name, detail, created_at FROM run_events
+       WHERE run_id IN (${placeholders}) ORDER BY created_at`
+    ).all(...runIds);
+  }
+  return { ok: true, runs, events };
+};
+
+commands["record-platform-failure"] = (db, args) => {
+  if (!args.platform_run_id) throw usageError("platform_run_id is required.");
+  if (!args.error_message) throw usageError("error_message is required.");
+  if (!args.platform_created_at) throw usageError("platform_created_at is required.");
+  // Try to correlate with a crew telemetry run by timestamp proximity (±60s).
+  // The workflow doesn't know its platform run_id, so we match on launch time.
+  let crewRunId = args.crew_run_id || null;
+  let taskId = args.task_id || null;
+  let workflow = args.workflow || null;
+  if (!crewRunId) {
+    const match = db.prepare(
+      `SELECT run_id, task_id, workflow FROM workflow_runs
+       WHERE ABS(strftime('%s', launched_at) - strftime('%s', ?)) < 60
+       ORDER BY ABS(strftime('%s', launched_at) - strftime('%s', ?))
+       LIMIT 1`
+    ).get(args.platform_created_at, args.platform_created_at);
+    if (match) {
+      crewRunId = match.run_id;
+      taskId = taskId || match.task_id;
+      workflow = workflow || match.workflow;
+    }
+  }
+  db.prepare(
+    `INSERT INTO platform_run_failures
+     (platform_run_id, crew_run_id, task_id, workflow, error_message, platform_created_at)
+     VALUES (?, ?, ?, ?, ?, ?)
+     ON CONFLICT(platform_run_id) DO UPDATE SET
+       error_message = excluded.error_message,
+       detected_at = datetime('now')`
+  ).run(args.platform_run_id, crewRunId, taskId, workflow, args.error_message, args.platform_created_at);
+  return { ok: true, platform_run_id: args.platform_run_id, crew_run_id: crewRunId, task_id: taskId };
+};
+
+commands["get-platform-failures"] = (db, args) => {
+  let failures;
+  if (args.task_id) {
+    failures = db.prepare(
+      "SELECT * FROM platform_run_failures WHERE task_id = ? ORDER BY detected_at DESC"
+    ).all(args.task_id);
+  } else if (args.platform_run_id) {
+    failures = db.prepare(
+      "SELECT * FROM platform_run_failures WHERE platform_run_id = ?"
+    ).all(args.platform_run_id);
+  } else {
+    const limit = Math.min(parseInt(args.limit) || 50, 200);
+    failures = db.prepare(
+      "SELECT * FROM platform_run_failures ORDER BY detected_at DESC LIMIT ?"
+    ).all(limit);
+  }
+  return { ok: true, failures };
+};
+
+commands["mark-failure-retried"] = (db, args) => {
+  if (!args.platform_run_id) throw usageError("platform_run_id is required.");
+  db.prepare(
+    `UPDATE platform_run_failures
+     SET retry_count = retry_count + 1, last_retry_at = datetime('now')
+     WHERE platform_run_id = ?`
+  ).run(args.platform_run_id);
+  return { ok: true, platform_run_id: args.platform_run_id };
+};
+
+commands["retry-platform-failure"] = (db, args) => {
+  if (!args.platform_run_id) throw usageError("platform_run_id is required.");
+  const maxRetries = parseInt(args.max_retries) || 3;
+  const failure = db.prepare(
+    "SELECT * FROM platform_run_failures WHERE platform_run_id = ?"
+  ).get(args.platform_run_id);
+  if (!failure) throw usageError("platform_run_id not found.");
+  if (!failure.task_id) {
+    return { ok: true, action: "skipped", reason: "no task correlated" };
+  }
+  if (failure.retry_count >= maxRetries) {
+    // Park the task — transient failures are not resolving.
+    const task = db.prepare("SELECT * FROM tasks WHERE id = ?").get(failure.task_id);
+    if (task && task.state !== "parked") {
+      const msg = "Platform workflow failed " + (failure.retry_count + 1) + "x: " +
+        failure.error_message.substring(0, 200);
+      db.prepare("UPDATE tasks SET state = 'parked', updated_at = datetime('now') WHERE id = ?")
+        .run(failure.task_id);
+      db.prepare("DELETE FROM dispatch_reservations WHERE task_id = ?").run(failure.task_id);
+      db.prepare(
+        `INSERT INTO events (task_id, event_type, message) VALUES (?, 'parked', ?)`
+      ).run(failure.task_id, msg);
+    }
+    return { ok: true, action: "parked", reason: "max retries exceeded", retry_count: failure.retry_count };
+  }
+  // Clear reservation and re-queue for retry.
+  db.exec("BEGIN");
+  try {
+    db.prepare("DELETE FROM dispatch_reservations WHERE task_id = ?").run(failure.task_id);
+    db.prepare(
+      "UPDATE tasks SET state = 'todo', updated_at = datetime('now') WHERE id = ? AND state != 'done'"
+    ).run(failure.task_id);
+    db.prepare(
+      `UPDATE platform_run_failures
+       SET retry_count = retry_count + 1, last_retry_at = datetime('now')
+       WHERE platform_run_id = ?`
+    ).run(args.platform_run_id);
+    db.exec("COMMIT");
+  } catch (e) {
+    db.exec("ROLLBACK");
+    throw e;
+  }
+  return {
+    ok: true,
+    action: "requeued",
+    task_id: failure.task_id,
+    retry_count: failure.retry_count + 1,
+  };
+};
+
 commands["park-task"] = (db, args) => {
   if (!args.task_id) throw usageError("task_id is required.");
   const message = (args.message ?? "").trim();

package/lib/schema.sql

@@ -99,3 +99,65 @@ CREATE INDEX IF NOT EXISTS events_task_id_idx ON events(task_id);
 -- another session holds the claim.
 CREATE UNIQUE INDEX IF NOT EXISTS agent_sessions_one_running_per_task_idx
   ON agent_sessions(task_id) WHERE status = 'running';
+
+-- Dispatch reservations: the dispatcher recommends, the cron worker launches,
+-- and the launched workflow self-claims. The claim can take 15+ minutes for
+-- cron-launched runs (canary 2026-09-13), far longer than the 3-minute poll
+-- interval — without a reservation, every tick re-dispatches the same task.
+-- The worker creates a reservation after launching; the dispatcher skips
+-- reserved tasks; the workflow clears it on self-claim. Expired reservations
+-- are ignored (the workflow died before claiming).
+CREATE TABLE IF NOT EXISTS dispatch_reservations (
+  task_id       TEXT PRIMARY KEY REFERENCES tasks(id) ON DELETE CASCADE,
+  run_id        TEXT NOT NULL,
+  workflow      TEXT NOT NULL,
+  dispatched_at TEXT NOT NULL,
+  expires_at    TEXT NOT NULL
+);
+CREATE INDEX IF NOT EXISTS dispatch_reservations_expires_idx
+  ON dispatch_reservations(expires_at);
+
+-- Workflow run telemetry (2026-09-13): the crew had no visibility into what
+-- a launched workflow was doing — a task could sit in_progress for hours
+-- with only a timed_out session as evidence. These tables give every run
+-- a structured timeline: when it started, what phases it entered/exited,
+-- when it claimed the task, and how it ended. Timestamps are written by
+-- SQLite (datetime('now')), never by workflow JS — the determinism guard
+-- forbids wall-clock in workflow scripts.
+CREATE TABLE IF NOT EXISTS workflow_runs (
+  run_id      TEXT PRIMARY KEY,
+  task_id     TEXT NOT NULL REFERENCES tasks(id) ON DELETE CASCADE,
+  workflow    TEXT NOT NULL,
+  launched_by TEXT NOT NULL DEFAULT 'unknown',
+  launched_at TEXT NOT NULL DEFAULT (datetime('now')),
+  completed_at TEXT,
+  status      TEXT NOT NULL DEFAULT 'running'
+              CHECK (status IN ('running', 'completed', 'failed', 'parked', 'timed_out'))
+);
+CREATE TABLE IF NOT EXISTS run_events (
+  id         INTEGER PRIMARY KEY AUTOINCREMENT,
+  run_id     TEXT NOT NULL REFERENCES workflow_runs(run_id) ON DELETE CASCADE,
+  task_id    TEXT NOT NULL,
+  event_name TEXT NOT NULL,
+  detail     TEXT NOT NULL DEFAULT '',
+  created_at TEXT NOT NULL DEFAULT (datetime('now'))
+);
+CREATE INDEX IF NOT EXISTS workflow_runs_task_id_idx ON workflow_runs(task_id);
+CREATE INDEX IF NOT EXISTS run_events_run_id_idx ON run_events(run_id);
+
+-- Platform workflow run failures (from runtime.workflow_runs).
+-- The platform records workflow run status separately from our telemetry.
+-- This table correlates platform failures with our tasks for visibility.
+CREATE TABLE IF NOT EXISTS platform_run_failures (
+  platform_run_id TEXT PRIMARY KEY,
+  crew_run_id     TEXT REFERENCES workflow_runs(run_id) ON DELETE SET NULL,
+  task_id         TEXT REFERENCES tasks(id) ON DELETE SET NULL,
+  workflow        TEXT,
+  error_message   TEXT NOT NULL,
+  platform_created_at TEXT NOT NULL,
+  detected_at     TEXT NOT NULL DEFAULT (datetime('now')),
+  retry_count     INTEGER NOT NULL DEFAULT 0,
+  last_retry_at   TEXT
+);
+CREATE INDEX IF NOT EXISTS platform_run_failures_task_id_idx ON platform_run_failures(task_id);
+CREATE INDEX IF NOT EXISTS platform_run_failures_detected_at_idx ON platform_run_failures(detected_at);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "muse-crew",
-  "version": "0.7.1",
+  "version": "0.7.3",
   "description": "Opinionated orchestration for Muse — workflows, identities, and tooling for autonomous software development.",
   "license": "UNLICENSED",
   "private": false,

package/seed/cron-body-template.md

@@ -4,6 +4,21 @@ You are the dispatch trigger for Muse Crew. Run the authoritative dispatcher wor
 
 ### Steps
 
+0. **Check for platform workflow failures (opacity killer):** The platform records workflow run failures in `runtime.workflow_runs` — these are invisible in the crew DB unless you check. A workflow that dies on a fatal `agent()` error (e.g. "subagent bootstrap is no longer authorized") leaves its task stranded with no explanation.
+   - Use `muse.db` to find failed platform runs in the last 15 minutes:
+     ```sql
+     SELECT w.run_id, w.created_at, c.error
+     FROM runtime.workflow_runs w
+     LEFT JOIN runtime.workflow_agent_calls c ON c.run_id = w.run_id AND c.status = 'failed'
+     WHERE w.created_at > now() - interval '15 minutes' AND w.status = 'failed'
+     ORDER BY w.created_at DESC
+     ```
+   - For each failed run, record it in the crew DB (correlates by timestamp with our telemetry):
+     `node {crewHome}/lib/crew-api.js --crew-home {crewHome} record-platform-failure --json '{"platform_run_id": "<run_id>", "error_message": "<error>", "platform_created_at": "<created_at>"}'`
+   - Then retry each failure (clears stale reservation, re-queues task, or parks after 3 attempts):
+     `node {crewHome}/lib/crew-api.js --crew-home {crewHome} retry-platform-failure --json '{"platform_run_id": "<run_id>"}'`
+   - Log the results. This ensures transient platform failures don't strand tasks.
+
 1. **Load tools:** Call tool_search_load_tool_namespace with paths ["workflow_launch"].
 
 2. **Load the workflow registry:** Read the file "{crewHome}/workflows/registry.json" with the read tool and parse it as JSON. If the file does not exist (the live release predates the registry), proceed without it — omit the `registry` arg and the dispatcher will load the registry the slow way and log a warning.
@@ -13,9 +28,32 @@ You are the dispatch trigger for Muse Crew. Run the authoritative dispatcher wor
    Wait for it to complete. It reads the crew's task state, determines eligibility, claims tasks, acknowledges the poll, and returns structured results.
 
 4. **Launch claims autonomously:** Extract the `claims` array from the dispatcher result. For each claim (up to 3 per tick — if more than 3, launch the first 3 and log the rest as deferred):
-   - Call workflow_launch_async with scriptPath={claim.scriptPath} and args={claim.args}.
-   - The launched workflow self-claims the task as its first action. If the task was already claimed or is done, the claim fails closed and the run stands down quietly — this is the mechanical duplicate protection, not an error.
-
-   If the dispatcher returned no claims or the claims array is empty, report: NO_DISPATCH
-
-5. Exit silently.
+   - Call workflow_launch_async with scriptPath={claim.scriptPath} and args={claim.args}. Record the returned run_id.
+   - Immediately acknowledge the dispatch reservation with the real run_id (the dispatcher already created a placeholder reservation — this updates it). Run in shell:
+     `node {crewHome}/lib/crew-api.js --crew-home {crewHome} acknowledge-dispatch-run --json '{"task_id": "{claim.task_id}", "run_id": "<run_id>"}'`
+   - If the acknowledge fails (no reservation exists), DO NOT LAUNCH — the dispatcher did not acquire this task. This is a safety invariant.
+   - The launched workflow self-claims the task and clears the reservation as its first actions. If the task was already claimed or is done, the claim fails closed and the run stands down quietly — this is the mechanical duplicate protection, not an error.
+
+   If the dispatcher returned no claims or the claims array is empty, report: NO_DISPATCH and exit.
+
+5. **Monitor launched workflows (launcher stays alive — platform workaround):** The platform ties async workflow subagent authorization to the launcher's lifetime. If you exit now, the workflow's next `agent()` call may fail with "subagent bootstrap is no longer authorized." You are the monitor — stay alive until each workflow reaches a terminal state.
+   - For each launched run_id, poll its status every 2 minutes via muse.db:
+     ```sql
+     SELECT status, error FROM runtime.workflow_runs WHERE run_id = '<run_id>'
+     ```
+   - **If status is `completed`:** Done. Log success and stop monitoring this run.
+   - **If status is `failed`:** Check if it's a platform `agent()` error (error contains "subagent bootstrap", "reservation owner is terminal", "bootstrap was cancelled", or "workflow agent call failed"):
+     - **Platform error:** Record it and retry:
+       1. `node {crewHome}/lib/crew-api.js --crew-home {crewHome} record-platform-failure --json '{"platform_run_id": "<run_id>", "error_message": "<error>"}'`
+       2. `node {crewHome}/lib/crew-api.js --crew-home {crewHome} retry-platform-failure --json '{"platform_run_id": "<run_id>"}'`
+       3. If the retry result says `requeued` and you have retries remaining (max 3 per task per tick):
+          - Re-acquire the reservation: `node {crewHome}/lib/crew-api.js --crew-home {crewHome} reserve-dispatch --json '{"task_id": "<task_id>"}'`
+          - If `acquired` is true, re-launch via workflow_launch_async with the same scriptPath and args, acknowledge with the new run_id, and continue monitoring the NEW run_id.
+          - If `acquired` is false, stop — another dispatcher claimed it.
+       4. If the retry result says `parked` (3 attempts exhausted), stop monitoring this task.
+     - **Task-level error (not a platform error):** The workflow's own error handling applies. Stop monitoring this run.
+   - **If status is `running` or `paused`:** Continue polling.
+   - **Timeout:** If a workflow hasn't reached terminal state after 90 minutes, log it and stop monitoring (the next tick's Step 0 will catch it if it failed). The monitor timeout MUST exceed the longest workflow `agent()` timeout (currently 60 minutes for the work phase) — otherwise the launcher shuts down mid-workflow and the bug recurs.
+   - When all launched workflows are terminal or retry-exhausted, exit silently.
+
+6. Exit silently.

package/workflows/bugfix.js

@@ -1060,6 +1060,7 @@ while (i < STEPS.length) {
       "Claim this task for the " + step.name + " step.\n" +
       "Run in shell and return the stdout verbatim:\n" + crewCmd("update-task", firstClaimUpdateArgs) + "\n" +
       "Then run in shell and return the stdout verbatim:\n" + crewCmd("claim-task", { task_id: taskId, identity: step.identity, step: step.name, notes: step.name + " step started" }) + "\n" +
+      "If the claim response has claimed=true, then run in shell and return the stdout verbatim:\n" + crewCmd("clear-reservation", { task_id: taskId }) + "\n" +
       "Do not interpret the claim response. It already contains an explicit \"claimed\" field — copy it verbatim.\n" +
       "Return { claimed: <verbatim>, session_id: \"<...>\" }. If claimed is false there is no session_id; return { claimed: false, session_id: \"\" }.",
       {

package/workflows/chore.js

@@ -50,6 +50,66 @@ function crewCmd(command, args) {
   var json = JSON.stringify(args || {}).replace(/'/g, "'\\''");
   return "node " + CREW_API + " --crew-home " + crewHome + " " + command + " --json '" + json + "'";
 }
+// Telemetry (2026-09-13): structured run timeline. The workflow mints a
+// telemetry run_id at startup via record-run-start (the API generates it
+// server-side — workflow JS never touches the clock or randomness) and
+// records milestone events. Timestamps come from SQLite datetime('now').
+// Fire-and-forget: telemetry must never break the run (no schema — the
+// run-5/run-6 fire-and-forget rule).
+//
+// BLAST RADIUS REDUCTION (2026-09-13): Every agent() call is a chance for
+// the platform to kill the workflow with a subagent-spawning failure
+// ("bootstrap is no longer authorized", "reservation owner is terminal").
+// These failures are FATAL to the workflow — try/catch does not save us.
+// So we batch non-critical telemetry events in memory and flush them in
+// as few agent() calls as possible. Critical events (start, claim,
+// completion, park) flush immediately.
+let telemetryRunId = null;
+let telemetryBuffer = [];
+async function telemetryStart(workflowName) {
+  try {
+    const out = await agent(crewCmd("record-run-start", {
+      task_id: taskId, workflow: workflowName, launched_by: "cron"
+    }), { key: "telemetry-start", label: "Recording run start" });
+    const parsed = typeof out === "string" ? JSON.parse(out) : out;
+    if (parsed && parsed.run_id) telemetryRunId = parsed.run_id;
+  } catch (e) {
+    log("Telemetry start failed (non-fatal): " + (e && e.message ? e.message : e));
+  }
+}
+async function telemetryEvent(eventName, detail) {
+  // Buffer non-critical events; flush at critical points.
+  if (!telemetryRunId) return;
+  telemetryBuffer.push({ event_name: eventName, detail: detail || "" });
+  // Auto-flush if buffer gets large (avoid unbounded memory).
+  if (telemetryBuffer.length >= 10) {
+    await telemetryFlush();
+  }
+}
+async function telemetryFlush() {
+  if (!telemetryRunId || telemetryBuffer.length === 0) return;
+  const events = telemetryBuffer.splice(0, telemetryBuffer.length);
+  try {
+    // Send all buffered events in a single agent() call via a batch command.
+    await agent(crewCmd("record-run-events-batch", {
+      run_id: telemetryRunId, task_id: taskId, events: events
+    }), { key: "telemetry-flush", label: "Flushing " + events.length + " telemetry events" });
+  } catch (e) {
+    log("Telemetry flush failed (non-fatal): " + events.length + " events lost");
+  }
+}
+async function telemetryEnd(status) {
+  if (!telemetryRunId) return;
+  // Flush any buffered events before recording the end.
+  await telemetryFlush();
+  try {
+    await agent(crewCmd("record-run-end", {
+      run_id: telemetryRunId, status: status
+    }), { key: "telemetry-end", label: "Recording run end" });
+  } catch (e) {
+    log("Telemetry end failed (non-fatal)");
+  }
+}
 const ORCH_PATH = crewHome + "/.orchestration";
 // Pin lifecycle scripts to this run
 const LIFECYCLE_SRC = crewHome + "/lib/worktree-lifecycle.sh";
@@ -846,16 +906,23 @@ async function parkTask(reason) {
     return { status: "failed", task_id: taskId, reason: "park failed: " + reason, park_failed: true };
   }
   await terminalCleanup();
+  await telemetryEnd("parked");
   return { status: "parked", task_id: taskId, reason: reason };
 }
 let i = startStepIndex;
 
+// Telemetry: mark run start before anything else (pin timing is the
+// 2026-09-13 mystery — the pin agent took 15m with no visibility).
+await telemetryStart("chore");
+
 // ── Pin lifecycle scripts ────────────────────────────────────────────
 // Copy lifecycle scripts into a per-task temp dir so this run is immune
 // to upgrades that land while it's in flight. Verified mechanically:
 // workflow code asserts the four basenames from the verbatim listing —
 // the agent cannot self-certify. Any miss parks the task before Triage.
+await telemetryEvent("pin_start");
 const initialPins = parsePinListing(await pinLifecycle("pin-lifecycle"));
+await telemetryEvent("pin_end", "pinned " + initialPins.length + " files");
 const missingInitialPins = PIN_BASENAMES.filter(function (b) { return initialPins.indexOf(b) === -1; });
 if (missingInitialPins.length > 0) {
   return await parkTask("Lifecycle pin incomplete before Triage — missing " + missingInitialPins.join(", ") + " in " + RUN_LIB + ".");
@@ -874,6 +941,7 @@ while (i < STEPS.length) {
 
   phase(step.name);
   log(step.name + " step (" + step.identity + ") for task " + taskId);
+  await telemetryEvent("phase_start", step.name);
 
   // ── Project-change guard ───────────────────────────────────────────
   // A task moved to another project mid-run must not keep working in the
@@ -1017,6 +1085,7 @@ while (i < STEPS.length) {
       "Claim this task for the " + step.name + " step.\n" +
       "Run in shell and return the stdout verbatim:\n" + crewCmd("update-task", firstClaimUpdateArgs) + "\n" +
       "Then run in shell and return the stdout verbatim:\n" + crewCmd("claim-task", { task_id: taskId, identity: step.identity, step: step.name, notes: step.name + " step started" }) + "\n" +
+      "If the claim response has claimed=true, then run in shell and return the stdout verbatim:\n" + crewCmd("clear-reservation", { task_id: taskId }) + "\n" +
       "Do not interpret the claim response. It already contains an explicit \"claimed\" field — copy it verbatim.\n" +
       "Return { claimed: <verbatim>, session_id: \"<...>\" }. If claimed is false there is no session_id; return { claimed: false, session_id: \"\" }.",
       {
@@ -1031,9 +1100,14 @@ while (i < STEPS.length) {
     );
     if (!claimResult.claimed) {
       log("Standing down — task " + taskId + " was already claimed by another run");
+      await telemetryEnd("duplicate");
       return { status: "duplicate", task_id: taskId, reason: "task already claimed by another run" };
     }
     activeSessionId = claimResult.session_id;
+    await telemetryEvent("claim", step.name + " claimed");
+    // Flush telemetry before proceeding — the claim is a critical point.
+    // If a subsequent agent() fails, we want the claim event persisted.
+    await telemetryFlush();
   } else {
     const claimResult = await agent(
       "Claim a session for this task step.\n" +
@@ -2212,5 +2286,6 @@ await agent(
 );
 
 log("Chore workflow complete for task " + taskId);
+await telemetryEnd("completed");
 await agent("Clean up pinned lifecycle scripts: rm -rf " + RUN_LIB, { key: "cleanup-pins", label: "Cleaning pinned scripts" });
 return { status: "ok", task_id: taskId, message: "Chore workflow complete for " + taskTitle };

package/workflows/crew-dispatch.js

@@ -237,6 +237,15 @@ const allTasks = boardData.ready_tasks.map(projectTaskRecord).filter(function (t
 const config = boardData.config || {};
 const projects = boardData.projects || [];
 
+// Dispatch reservations: tasks the worker launched on a previous tick whose
+// workflow has not yet self-claimed. The claim can take 15+ minutes for
+// cron-launched runs (canary 2026-09-13) — far longer than the poll interval —
+// so without this filter every tick would re-dispatch the same task.
+const reservedTaskIds = new Set((boardData.reservations || []).map(function (r) { return r.task_id; }));
+if (reservedTaskIds.size > 0) {
+  log("Reservations: skipping " + reservedTaskIds.size + " task(s) with an active dispatch reservation");
+}
+
 // Default project: explicit arg, or first registered project
 const DEFAULT_PROJECT = inputs.defaultProject || (projects.length > 0 ? projects[0].id : "");
 
@@ -383,6 +392,14 @@ for (var t = 0; t < allTasks.length; t++) {
   var task = allTasks[t];
   if (task.blocked) continue;
 
+  // Skip tasks with an active dispatch reservation: the worker launched a
+  // workflow for this task on a previous tick, but the workflow has not yet
+  // self-claimed (claims can take 15+ minutes for cron-launched runs).
+  if (reservedTaskIds.has(task.id)) {
+    log("Skipped \"" + task.title + "\" — active dispatch reservation (workflow launched, claim pending)");
+    continue;
+  }
+
   // Skip tasks from quiesced projects
   var taskProject = task.project || DEFAULT_PROJECT;
   if (quiescedProjects[taskProject]) {
@@ -790,6 +807,46 @@ results.forEach(function(r) {
   recommended.push(r);
 });
 var completed = results.filter(function(r) { return r.action === "completed"; });
+
+// Atomic reservation: create a dispatch reservation for each recommended task
+// BEFORE returning. This closes the launch→reservation race — the worker no
+// longer needs to create reservations (it updates the run_id after launch).
+// If the worker dies or times out, the reservation expires via TTL.
+// Note: run_id uses a deterministic placeholder (no wall-clock — see
+// tests/determinism.test.js); the worker updates it with the real run_id
+// via acknowledge-dispatch-run after workflow_launch_async returns.
+if (recommended.length > 0) {
+  var acquired = [];
+  for (var i = 0; i < recommended.length; i++) {
+    var rec = recommended[i];
+    var reserveCmd = crewCmd("reserve-dispatch", {
+      task_id: rec.task_id,
+      run_id: "pending-dispatch",
+      workflow: rec.workflow
+    });
+    try {
+      var reserveOut = await agent(
+        "Create a dispatch reservation.\nRun in shell and return the stdout verbatim:\n" + reserveCmd,
+        { key: "reserve-" + rec.task_id.slice(0, 8), label: "Reserving dispatch for " + rec.task_id.slice(0, 8) }
+      );
+      var reserveParsed = typeof reserveOut === "string" ? JSON.parse(reserveOut) : reserveOut;
+      if (reserveParsed && reserveParsed.acquired) {
+        log("Reserved task " + rec.task_id.slice(0, 8));
+        acquired.push(rec);
+      } else {
+        // FAIL CLOSED: Another dispatcher owns this task. Do not recommend it.
+        log("SKIPPED task " + rec.task_id.slice(0, 8) + " — reservation not acquired (another dispatcher owns it)");
+      }
+    } catch (e) {
+      // FAIL CLOSED: If we cannot reserve, we cannot safely recommend.
+      // The task stays eligible for the next poll.
+      log("SKIPPED task " + rec.task_id.slice(0, 8) + " — reservation failed: " + e.message);
+    }
+  }
+  // Only recommend tasks we actually acquired.
+  recommended = acquired;
+}
+
 var msg = "Dispatch complete.";
 if (recommended.length > 0) {
   msg += " Recommended: " + recommended.map(function(r) { return r.workflow + "/" + r.step + " for " + r.task_id; }).join(", ") + ".";

package/workflows/standard.js

@@ -1059,6 +1059,7 @@ while (i < STEPS.length) {
       "Claim this task for the " + step.name + " step.\n" +
       "Run in shell and return the stdout verbatim:\n" + crewCmd("update-task", firstClaimUpdateArgs) + "\n" +
       "Then run in shell and return the stdout verbatim:\n" + crewCmd("claim-task", { task_id: taskId, identity: step.identity, step: step.name, notes: step.name + " step started" }) + "\n" +
+      "If the claim response has claimed=true, then run in shell and return the stdout verbatim:\n" + crewCmd("clear-reservation", { task_id: taskId }) + "\n" +
       "Do not interpret the claim response. It already contains an explicit \"claimed\" field — copy it verbatim.\n" +
       "Return { claimed: <verbatim>, session_id: \"<...>\" }. If claimed is false there is no session_id; return { claimed: false, session_id: \"\" }.",
       {