muse-crew 0.7.11 → 0.7.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/API.md +17 -3
- package/docs/guide.md +4 -4
- package/docs/ooda-report.md +32 -5
- package/docs/publish-verification.md +46 -22
- package/lib/AGENTS.md +2 -1
- package/lib/crew-api.js +186 -41
- package/lib/read-ooda-verdict.js +94 -0
- package/lib/readback-disk.js +186 -0
- package/lib/render-html.js +2 -6
- package/lib/see-act.js +2 -6
- package/lib/verify-publish.js +62 -4
- package/lib/write-ooda-verdict.js +19 -2
- package/package.json +5 -2
- package/seed/cron-body-template.md +7 -3
- package/seed/crons.json +1 -1
- package/workflows/bugfix.js +302 -12
- package/workflows/chore.js +236 -10
- package/workflows/crew-dispatch.js +71 -12
- package/workflows/crew-init.js +26 -32
- package/workflows/docs.js +10 -1
- package/workflows/standard.js +237 -11
package/API.md
CHANGED
|
@@ -50,10 +50,11 @@ Atomically claim a task for a workflow step. Used by the dispatcher and by workf
|
|
|
50
50
|
| `identity` | string (1–80) | yes | The crew identity performing the step (e.g. `sage`, `wren`) |
|
|
51
51
|
| `step` | string (1–120) | no | The workflow phase name |
|
|
52
52
|
| `notes` | string (≤ 3000) | no | Defaults to `""` |
|
|
53
|
+
| `expected_next_phase` | string (1–120) | no | The `next_phase` routing the dispatcher launched this run for. When the claim wins, the matching `next_phase` is cleared in the **same transaction** as the winning session insert — there is no claim→consume window for a platform death to replay the routing through. Only an exact match clears; a stale or superseded routing survives. Returned as `next_phase_consumed: true/false` on the winning response (absent when the field was not passed). A losing claim never touches `next_phase`. |
|
|
53
54
|
|
|
54
55
|
Returns one of:
|
|
55
56
|
|
|
56
|
-
- `{ "ok": true, "claimed": true, "session_id": "<uuid>", "session": {...} }`
|
|
57
|
+
- `{ "ok": true, "claimed": true, "session_id": "<uuid>", "session": {...}, "next_phase_consumed": true/false }`
|
|
57
58
|
- `{ "ok": true, "claimed": false, "reason": "already_claimed", "existing_session_id": "<uuid>" }`
|
|
58
59
|
|
|
59
60
|
`claimed` is the single field callers branch on. `reason` is present only when `claimed` is false.
|
|
@@ -66,9 +67,22 @@ Send a stuck or failed task to a specific workflow phase for recovery.
|
|
|
66
67
|
|-------|------|----------|-------|
|
|
67
68
|
| `task_id` | uuid | yes | |
|
|
68
69
|
| `action` | `"send_to"` | yes | Only `send_to` is supported |
|
|
69
|
-
| `target_phase` | string (1–120) | yes | The phase to send the task to |
|
|
70
|
+
| `target_phase` | string (1–120) | yes | The phase to send the task to — must be a phase of the task's workflow (validated against the workflow's phase registry; unknown phases fail closed) |
|
|
70
71
|
| `updated_description` | string (≤ 5000) | no | Optional revised description |
|
|
71
72
|
|
|
73
|
+
Recovery intent, not a verdict: accepts a latest session of `failed`, `timed_out`, or `stalled` — or no session at all (pre-claim platform death). Rejects a `running` latest session (work in flight) and terminal task states. Sets `tasks.next_phase` to the target and moves `parked`/`todo` tasks to `in_progress`. Inserts **no** session: recovery never consumes retry budget. The dispatcher routes the task to `next_phase` one-shot (bypassing the retry cap); the launched workflow's successful self-claim clears it atomically (`claim-task` with `expected_next_phase`). Returns the task and `valid_phases`.
|
|
74
|
+
|
|
75
|
+
### `consume-next-phase`
|
|
76
|
+
|
|
77
|
+
One-shot consumption of a routed `next_phase`. Retained as an idempotent public helper for out-of-band recovery tooling — workflows no longer call this; `claim-task` with `expected_next_phase` clears the routed `next_phase` in the same transaction as the winning session insert, so the old claim→consume death window is closed.
|
|
78
|
+
|
|
79
|
+
| Field | Type | Required | Notes |
|
|
80
|
+
|-------|------|----------|-------|
|
|
81
|
+
| `task_id` | uuid | yes | |
|
|
82
|
+
| `expected` | string (1–120) | yes | The `next_phase` value the dispatcher routed on |
|
|
83
|
+
|
|
84
|
+
Clears `tasks.next_phase` only when the stored value still matches `expected` (atomic conditional update). Returns `{ "consumed": true/false }`. A newer `recovertask` written in the race window survives a stale consumer.
|
|
85
|
+
|
|
72
86
|
### `parktask`
|
|
73
87
|
|
|
74
88
|
Atomically park a task for human attention: sets `state` to `parked`, writes the explanatory `note` event, and settles every `running` session for the task — in one transaction. Either all land or none do: callers never see a note without a park, a park without a note, or a parked task with a ghost `running` session left behind (a terminal task transition settles its sessions). Settled sessions move to `failed` — the status `recovertask` accepts, so recovery works immediately after a park — with `ended_at` stamped and `Parked: <message>` appended to their notes; `failure_reason` stays null.
|
|
@@ -306,7 +320,7 @@ The platform records workflow run status in `runtime.workflow_runs` — separate
|
|
|
306
320
|
Two layers, in order of preference:
|
|
307
321
|
|
|
308
322
|
1. **Prevention — the launcher stays alive.** Async workflow `agent()` authorization is tied to the launcher's lifetime: if the cron tick ends while a workflow is still running, the workflow's next `agent()` call fails. So the poll tick runs with a 90-minute execution timeout (`timeout_secs: 5400` in `seed/crons.json`) and its Step 5 monitor stays alive until every launched run reaches a terminal state. A launcher that outlives its runs never triggers the failure in the first place.
|
|
309
|
-
2. **Recovery — per-tick correlation through the durable mapping.** If the launcher dies early anyway (platform kill, cell recycle), the next tick's Step 0 finds the dead run, `record-platform-failure` correlates it via the `platform_run_tasks` mapping, and `retry-platform-failure` requeues the task — at most ~
|
|
323
|
+
2. **Recovery — per-tick correlation through the durable mapping.** If the launcher dies early anyway (platform kill, cell recycle), the next tick's Step 0 finds the dead run, `record-platform-failure` correlates it via the `platform_run_tasks` mapping, and `retry-platform-failure` requeues the task — at most ~15 minutes later, with the ghost session settled so the dispatcher treats it as a retry candidate immediately.
|
|
310
324
|
|
|
311
325
|
### `record-platform-failure`
|
|
312
326
|
|
package/docs/guide.md
CHANGED
|
@@ -17,7 +17,7 @@ This is the full setup and operations reference. If you're new, start with the [
|
|
|
17
17
|
- A **release** — the first immutable snapshot of the crew's runtime code.
|
|
18
18
|
- An **`.orchestration/` directory** in the crew home with identities, personas, workflow docs, and feedback conventions.
|
|
19
19
|
- A **project registration** — the task service registered as its own first project.
|
|
20
|
-
- **Cron jobs** — the scheduler state declared in `seed/crons.json`: a polling loop (every
|
|
20
|
+
- **Cron jobs** — the scheduler state declared in `seed/crons.json`: a polling loop (every 15 minutes via Muse's scheduling, even when nobody's in the conversation). Owner is `space:<slug>`, so deleting the task service also removes the crons.
|
|
21
21
|
|
|
22
22
|
The agent running in the main chat receives the dispatcher's claims and launches each task workflow. Workflows can't launch workflows, so this handoff is structural.
|
|
23
23
|
|
|
@@ -197,7 +197,7 @@ A task in `todo` state becomes eligible for dispatch on the next polling tick. A
|
|
|
197
197
|
|
|
198
198
|
## How the loop works
|
|
199
199
|
|
|
200
|
-
Every
|
|
200
|
+
Every 15 minutes, the `crew-poll` cron fires:
|
|
201
201
|
|
|
202
202
|
1. **Cron runs the dispatcher** — `crew-dispatch.js` with the task service slug and `crewHome` as arguments.
|
|
203
203
|
|
|
@@ -302,7 +302,7 @@ For tasks that change anything rendered and visible in the project's artifact ("
|
|
|
302
302
|
|
|
303
303
|
### Publish content verification
|
|
304
304
|
|
|
305
|
-
The artifact builder's `applied` report is derived from the diff the workflow carries to it, so comparing the report to the diff is circular — canary run 8 (2026-09-11) stamped provenance on a hollow build and every phase went green. The workflow therefore never stamps provenance itself: after the build lands it parks with `publish: verification-requested <commit> (build <agent_id|agent_id unobserved>)`. The parent protocol owns the independent content confirmation (parent-driven — see `docs/publish-verification.md`); the park message records the observed builder build identifier as `(build <agent_id|agent_id unobserved>)`, and the parent correlates the read-back's live build agent_id against it — a mismatch logs `publish: build-mismatch <commit> …`, stays parked, and is never stamped (parent-driven — see `docs/publish-verification.md` step 4b). The independent read-back step is currently unavailable: `artifact_inspect` was removed by the platform (2026-09-14) and no agent-callable replacement exists (`artifact.inspect` is malfunction diagnosis, not a read-back tool), so the parent cannot confirm content independently and tasks stay parked at verification-requested until a read-back path exists. QA's provenance check then enforces the stamp mechanically, so an unverified publish fails loudly in QA instead of passing silently.
|
|
305
|
+
The artifact builder's `applied` report is derived from the diff the workflow carries to it, so comparing the report to the diff is circular — canary run 8 (2026-09-11) stamped provenance on a hollow build and every phase went green. The workflow therefore never stamps provenance itself: after the build lands it parks with `publish: verification-requested <commit> (build <agent_id|agent_id unobserved>)`. "Landed" requires positive evidence (canary 2026-09-15, task `1d692d91`): the build poll must have positively observed our build — a running build with the receipt `agent_id`, or a completed-build record matching it. Absence of a running build is not evidence our build ran; an unobserved "done" is an unknown outcome, parked fail-closed with an append-only `unknown` ledger entry — never parked as verification-requested. The parent protocol owns the independent content confirmation (parent-driven — see `docs/publish-verification.md`); the park message records the observed builder build identifier as `(build <agent_id|agent_id unobserved>)`, and the parent correlates the read-back's live build agent_id against it — a mismatch logs `publish: build-mismatch <commit> …`, stays parked, and is never stamped (parent-driven — see `docs/publish-verification.md` step 4b). The independent read-back step is currently unavailable: `artifact_inspect` was removed by the platform (2026-09-14) and no agent-callable replacement exists (`artifact.inspect` is malfunction diagnosis, not a read-back tool), so the parent cannot confirm content independently and tasks stay parked at verification-requested until a read-back path exists. QA's provenance check then enforces the stamp mechanically, so an unverified publish fails loudly in QA instead of passing silently.
|
|
306
306
|
|
|
307
307
|
## Identities
|
|
308
308
|
|
|
@@ -424,7 +424,7 @@ crew-release.sh current
|
|
|
424
424
|
|
|
425
425
|
2. **Stuck sessions.** If a workflow dies mid-run, its session stays `running` indefinitely and the cron skips it every tick. Manual cleanup is currently required.
|
|
426
426
|
|
|
427
|
-
3. **No streaming.** The polling loop checks every
|
|
427
|
+
3. **No streaming.** The polling loop checks every 15 minutes. There is no webhook or event-driven dispatch.
|
|
428
428
|
|
|
429
429
|
4. **Single dispatcher.** Only one cron runs the dispatcher. Concurrent ticks are prevented by the session-claiming atomicity, but there is no explicit distributed lock.
|
|
430
430
|
|
package/docs/ooda-report.md
CHANGED
|
@@ -63,12 +63,13 @@ Written at phase end via `lib/write-ooda-verdict.js --dir <phase-dir>
|
|
|
63
63
|
|
|
64
64
|
```json
|
|
65
65
|
{
|
|
66
|
-
"verdict": "
|
|
66
|
+
"verdict": "FAIL",
|
|
67
67
|
"attempt": "1",
|
|
68
|
-
"summary": "
|
|
69
|
-
"expected": "
|
|
70
|
-
"actual": "
|
|
71
|
-
"missing_evidence": []
|
|
68
|
+
"summary": "Login button missing on desktop header.",
|
|
69
|
+
"expected": "login button on header",
|
|
70
|
+
"actual": "no login control in the ARIA tree",
|
|
71
|
+
"missing_evidence": ["mobile viewport not checked"],
|
|
72
|
+
"reason": "Header renders but no login control present in the ARIA tree on desktop."
|
|
72
73
|
}
|
|
73
74
|
```
|
|
74
75
|
|
|
@@ -76,6 +77,12 @@ Written at phase end via `lib/write-ooda-verdict.js --dir <phase-dir>
|
|
|
76
77
|
was not checked — an honest gap is recorded, never hidden. Unknown/
|
|
77
78
|
inconclusive is neither PASS nor FAIL.
|
|
78
79
|
|
|
80
|
+
`reason` is REQUIRED and must be non-empty when the verdict is `FAIL` or
|
|
81
|
+
`NOT_POSSIBLE`: the writer rejects a reason-less negative verdict with exit
|
|
82
|
+
2 and writes nothing. A FAIL verdict without a machine-readable reason is
|
|
83
|
+
not writable — the defect that parked the 2026-09-15 canary at Publish was a
|
|
84
|
+
bare `VERDICT: FAIL` with an all-positive report and no recorded reason.
|
|
85
|
+
|
|
79
86
|
Two records are written:
|
|
80
87
|
|
|
81
88
|
- `verdict.json` — the LATEST verdict, what the workflow closeout reads.
|
|
@@ -121,3 +128,23 @@ No nested agents — the depth-1 work agent runs the browser steps itself.
|
|
|
121
128
|
3. Open the archived PNGs and compare against the written observations. Never trust prose alone.
|
|
122
129
|
|
|
123
130
|
The report is the evidence. A QA claim without its OODA report is an unverified claim.
|
|
131
|
+
|
|
132
|
+
## Reading the verdict back (2026-09-15)
|
|
133
|
+
|
|
134
|
+
`lib/read-ooda-verdict.js --dir <phase-dir> --expect <PASS|FAIL>` is the
|
|
135
|
+
deterministic cross-checker the bugfix QA closeout runs after extracting the
|
|
136
|
+
prose `VERDICT:` line. It prints one JSON line to stdout and never touches
|
|
137
|
+
the clock or randomness:
|
|
138
|
+
|
|
139
|
+
- Exit 0, `{ok:true, verdict, reason, summary, expected, actual, attempt}` —
|
|
140
|
+
the record exists, parses, carries a `verdict` field, agrees with the
|
|
141
|
+
prose expectation, and a FAIL carries a non-empty reason.
|
|
142
|
+
- Exit 2, `{ok:false, code}` — `missing` (no verdict.json), `corrupt`
|
|
143
|
+
(unparseable or no verdict field), `contradiction` (record disagrees with
|
|
144
|
+
the prose line), `no_reason` (FAIL with no machine-readable reason).
|
|
145
|
+
|
|
146
|
+
The prose `VERDICT:` line routes, but `verdict.json` carries the reason the
|
|
147
|
+
workflow reads. On a cross-check failure the QA phase is recorded as failed
|
|
148
|
+
and retried at the same step — an unreasoned or contradictory verdict never
|
|
149
|
+
routes to rework. No LLM judges report-prose consistency; the machine only
|
|
150
|
+
enforces that the reason is present and the records agree.
|
|
@@ -1,11 +1,21 @@
|
|
|
1
1
|
# Publish content verification — parent protocol
|
|
2
2
|
|
|
3
|
-
> **
|
|
4
|
-
>
|
|
5
|
-
>
|
|
6
|
-
>
|
|
7
|
-
>
|
|
8
|
-
>
|
|
3
|
+
> **UNBLOCKED (2026-09-15):** the platform's `artifact_inspect` is still
|
|
4
|
+
> gone, but no platform tool is needed anymore. The platform's artifact
|
|
5
|
+
> edits land in its on-disk working copy of the artifact source
|
|
6
|
+
> (`~/workspace/ts-spaces/<slug>/` — verified empirically 2026-09-15:
|
|
7
|
+
> added lines present, removed lines absent across real platform commits),
|
|
8
|
+
> so `lib/readback-disk.js` performs the read-back deterministically: it
|
|
9
|
+
> reads the working copy and emits the exact machine-readable findings
|
|
10
|
+
> block `lib/verify-publish.js` already parses. No LLM, no async handoff,
|
|
11
|
+
> no prose to parse. The verifier is unchanged — the sensor changed, the
|
|
12
|
+
> judge didn't.
|
|
13
|
+
>
|
|
14
|
+
> Authority boundary: the sensor reads the platform's working copy — the
|
|
15
|
+
> tree the hosted artifact is built/served from. A working copy that is
|
|
16
|
+
> stale relative to a just-applied edit yields honest ABSENT findings and
|
|
17
|
+
> the verifier fails CLOSED (parked). Staleness can only park a task,
|
|
18
|
+
> never stamp provenance.
|
|
9
19
|
|
|
10
20
|
Provenance is the artifact's claim that its live content came from a specific
|
|
11
21
|
repo commit. The workflow never stamps it. This document is the parent-side
|
|
@@ -120,10 +130,16 @@ LLM never judges. The division:
|
|
|
120
130
|
and never uses `commit^1` as the base: push-time reconcile merges put
|
|
121
131
|
the task's own changes behind an intermediate merge, so `commit^1`
|
|
122
132
|
covers only the reconcile delta (2026-09-14, task `0c53af4e`).
|
|
123
|
-
3. **Ferry (tick worker):**
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
133
|
+
3. **Ferry (tick worker):** runs the deterministic sensor
|
|
134
|
+
`lib/readback-disk.js` (`--repo-path`, `--commit`, `--base` — the same
|
|
135
|
+
base as step 2 — `--slug` from the project's `deploy_slug`, `--task-id`)
|
|
136
|
+
and saves its stdout to the result file. The sensor exits 0 only when it
|
|
137
|
+
actually read the working copy; on a non-zero exit the tick must NOT
|
|
138
|
+
save stdout — log `publish: verification-procedural-error <commit>
|
|
139
|
+
<stderr>` and leave the task parked for the next tick to retry (a sensor
|
|
140
|
+
failure is procedural — the read could not be performed — not a content
|
|
141
|
+
verdict). `build-readback-request.js` is retained for the manual LLM
|
|
142
|
+
fallback below.
|
|
127
143
|
4. **Certify (code):** `verify-publish.js` parses the inspector's
|
|
128
144
|
machine-readable findings block, compares every added/removed diff line
|
|
129
145
|
against the reported present/absent verdicts, checks build-ID correlation
|
|
@@ -194,15 +210,15 @@ For a task parked with `publish: verification-requested <commit>`:
|
|
|
194
210
|
reconcile merges violate the `merge^1 == previously-published tree`
|
|
195
211
|
invariant, so `commit^1..commit` can omit the task's own fix
|
|
196
212
|
(2026-09-14, task `0c53af4e`).
|
|
197
|
-
3. **Actual content.**
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
213
|
+
3. **Actual content.** Run `lib/readback-disk.js` with `--repo-path`,
|
|
214
|
+
`--commit`, `--base` (the same base as step 2), `--slug`, and
|
|
215
|
+
`--task-id`, and save its stdout to the result file — this is the
|
|
216
|
+
deterministic read-back; it emits the machine-readable findings block
|
|
217
|
+
directly. (LLM fallback: if the disk working copy is unavailable, call
|
|
218
|
+
the artifact inspector with `repair_authorized: false` and the
|
|
219
|
+
`verbatim_request` built by `lib/build-readback-request.js`, passing
|
|
220
|
+
`--build-agent-id` from the park message's `(build …)` suffix when it
|
|
221
|
+
is not `agent_id unobserved`.) The findings block grammar:
|
|
206
222
|
```
|
|
207
223
|
FILE: <path>
|
|
208
224
|
ADDED: <exact added line> :: PRESENT|ABSENT
|
|
@@ -215,9 +231,17 @@ For a task parked with `publish: verification-requested <commit>`:
|
|
|
215
231
|
4. **Compare mechanically.** For every added (`+`) line in the diff, the
|
|
216
232
|
read-back's machine-readable block must report it PRESENT in the
|
|
217
233
|
artifact's current source. For every removed (`-`) line, it must report
|
|
218
|
-
it ABSENT
|
|
219
|
-
|
|
220
|
-
|
|
234
|
+
it ABSENT — with one mechanical exemption: a removed line that also
|
|
235
|
+
occurs verbatim in untouched code has zero discriminating power (its
|
|
236
|
+
presence proves nothing about whether the old block survived), so the
|
|
237
|
+
verifier exempts it instead of failing a good publish. The exemption is
|
|
238
|
+
computed, never judged: a removed line L in file F is exempt iff L
|
|
239
|
+
occurs in F's old tree (at `<base>`) strictly more times than the diff
|
|
240
|
+
removes it (2026-09-15, task `00bca4b8` — a valid publish parked because
|
|
241
|
+
two removed lines occurred identically in the untouched WorkflowSteps
|
|
242
|
+
component). The comparison is computed by `lib/verify-publish.js` —
|
|
243
|
+
never by eyeballing prose. A missing or malformed findings block fails
|
|
244
|
+
closed as `unreadable-result`, never as a pass.
|
|
221
245
|
4b. **Build-ID correlation.** The read-back may have inspected a different
|
|
222
246
|
build's output than this publish attempt's:
|
|
223
247
|
1. **Expected** = the agent_id in the park message's `(build …)` suffix.
|
package/lib/AGENTS.md
CHANGED
|
@@ -19,5 +19,6 @@ Shell scripts for the crew's infrastructure. Called by workflow scripts, cron, a
|
|
|
19
19
|
- `render-html.js` — deterministic HTML evidence composition (2026-09-15, headless Chromium): renders a local HTML layout to PNG (fixed viewport width, device scale 1, full-page screenshot). Hermetic: remote HTTP(S) assets are blocked and fail the render loudly; relative image paths resolve against the HTML file; local/system fonts only. Exit 3 reports NOT POSSIBLE when Chromium is unavailable. The composition layer above edit-image.py's pixel layer.
|
|
20
20
|
- `see-act.js` — single-step browser driver for experiential QA (2026-09-14): one browser action per invocation (`aria`, `shot`, `click`, `scroll`, `type`), one JSON line on stdout, exit 0/2/3 (3 = NOT POSSIBLE: missing playwright-core or Chromium). Spawns its own loopback forward proxy on an ephemeral port (Chromium blocks direct loopback). The QA work agent closes the OODA loop: run a step, read the screenshot/aria, decide the next. Determinism: no wall-clock reads, no randomness. `SEE_ACT_ARCHIVE_DIR=<phase-dir>` (2026-09-14): every screenshot is archived automatically as `001-shot-desktop.png`, `002-click-mobile.png`, ... (counter in `<dir>/.seq`); `--out` becomes optional; JSON carries `screenshot` + `archived`; explicit `--out` + archive copies into the archive; unusable dir is NOT POSSIBLE exit 3.
|
|
21
21
|
- `append-ooda-step.js` — deterministic writer for the OODA report log (2026-09-14, attempt identity 2026-09-15): `node append-ooda-step.js --log <path> --attempt <id> --step <n> --action <a> --exit <code> [--args <json>] [--screenshot <path>] [--observation <text>]` appends one JSON line to `<phase-dir>/ooda-log.jsonl` (`{step, attempt, action, args, exit, screenshot|null, observation}`). `--attempt` is required; steps must be strictly monotonic within an attempt (a rerun is a new attempt at step 1 — attempts accumulate, never overwrite). Actions: browser (`aria|shot|click|scroll|type`) and image (`crop|zoom|label|nup|compose` — the last five log evidence derivatives made with `lib/edit-image.py` / `lib/render-html.js`; frame-producing actions require `--screenshot`). Corrupt logs or sequence gaps fail loudly (exit 2). No wall-clock reads, no randomness.
|
|
22
|
-
- `write-ooda-verdict.js` — deterministic writer for the OODA terminal verdict (2026-09-14, append-only ledger 2026-09-15): `node write-ooda-verdict.js --dir <phase-dir> --attempt <id> --verdict <PASS|FAIL|NOT_POSSIBLE> --summary <text> --expected <text> --actual <text> --missing <json-array
|
|
22
|
+
- `write-ooda-verdict.js` — deterministic writer for the OODA terminal verdict (2026-09-14, append-only ledger 2026-09-15): `node write-ooda-verdict.js --dir <phase-dir> --attempt <id> --verdict <PASS|FAIL|NOT_POSSIBLE> --summary <text> --expected <text> --actual <text> --missing <json-array> [--reason <text>]` writes `<phase-dir>/verdict.json` (the latest verdict) and appends one JSON line to `<phase-dir>/verdicts.jsonl` — the append-only ledger: every attempt's verdict is preserved with a mechanical `seq`, never overwritten; corrupt or non-contiguous ledgers fail loudly. `--reason` is REQUIRED and must be non-empty for `FAIL` and `NOT_POSSIBLE` — a reason-less negative verdict fails with exit 2 before anything is written (2026-09-15). Exit 2 on bad input. See `docs/ooda-report.md`.
|
|
23
|
+
- `read-ooda-verdict.js` — deterministic cross-checker for the OODA terminal verdict (2026-09-15): `node read-ooda-verdict.js --dir <phase-dir> --expect <PASS|FAIL>` reads `<dir>/verdict.json`, prints one JSON line to stdout, exits 0 with `{ok:true, verdict, reason, summary, expected, actual, attempt}` when the record agrees with the prose expectation and a FAIL carries a non-empty reason, or exits 2 with `{ok:false, code}` — `missing|corrupt|contradiction|no_reason`. No wall-clock reads, no randomness. The bugfix QA closeout runs it against the prose `VERDICT:` line before any rework routing: a failed cross-check records the phase as failed for retry, never routes to rework.
|
|
23
24
|
- `serve-artifact.js` — local server for a built TS space for experiential QA (2026-09-14): serves `<space-dir>/client/dist` statically and dispatches POST `*/actions` to the compiled server actions with a locally-built Ctx. Prints `READY port=<n>` then serves until killed. Read-only w.r.t. the space directory. Fidelity: the served client and action handlers are the artifact's own built code; the Ctx is locally built (privileged handlers run from the space's own `server/dist/privileged.js` when present; blobs are stored in a per-run temp dir and served back at `/__blobs/<key>`); environment is inherited from the caller. It is not the hosted runtime — tasks that cannot be judged under it must report `NOT POSSIBLE: <reason>`.
|
package/lib/crew-api.js
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
//
|
|
13
13
|
// Commands (kebab-case) map to API.md actions:
|
|
14
14
|
// get-dispatch-state, get-state, create-task, update-task, claim-task,
|
|
15
|
-
// park-task, recover-task, upsert-session, log-event, get-events,
|
|
15
|
+
// park-task, recover-task, consume-next-phase, upsert-session, log-event, get-events,
|
|
16
16
|
// create-project, update-project, delete-project, list-projects,
|
|
17
17
|
// get-project, kill-switch, get-config, update-config,
|
|
18
18
|
// set-provenance, get-provenance, acknowledge-poll,
|
|
@@ -260,6 +260,53 @@ function workflowExists(crewHome, slug) {
|
|
|
260
260
|
);
|
|
261
261
|
}
|
|
262
262
|
|
|
263
|
+
// Workflow step names for a slug, from the release registry when present,
|
|
264
|
+
// falling back to the workflow file's own `export const meta` block (the
|
|
265
|
+
// single source of truth the registry is built from). Returns null when the
|
|
266
|
+
// workflow file cannot be found or parsed. Used by recover-task to validate
|
|
267
|
+
// target_phase, and by the dispatcher-side contract tests.
|
|
268
|
+
function workflowStepNames(crewHome, slug) {
|
|
269
|
+
if (!slug) return null;
|
|
270
|
+
const candidates = [
|
|
271
|
+
join(crewHome, "current", "workflows", "registry.json"),
|
|
272
|
+
join(crewHome, "workflows", "registry.json"),
|
|
273
|
+
];
|
|
274
|
+
for (const regPath of candidates) {
|
|
275
|
+
try {
|
|
276
|
+
const reg = JSON.parse(readFileSync(regPath, "utf8"));
|
|
277
|
+
const entry = reg && reg[slug];
|
|
278
|
+
if (entry && Array.isArray(entry.steps)) {
|
|
279
|
+
const names = entry.steps.map((s) => s && s.name).filter((n) => typeof n === "string");
|
|
280
|
+
if (names.length > 0) return names;
|
|
281
|
+
}
|
|
282
|
+
} catch (e) { /* fall through to the meta-block parse */ }
|
|
283
|
+
}
|
|
284
|
+
const wfCandidates = [
|
|
285
|
+
join(crewHome, "current", "workflows", `${slug}.js`),
|
|
286
|
+
join(crewHome, "workflows", `${slug}.js`),
|
|
287
|
+
];
|
|
288
|
+
for (const wfPath of wfCandidates) {
|
|
289
|
+
let source;
|
|
290
|
+
try { source = readFileSync(wfPath, "utf8"); } catch (e) { continue; }
|
|
291
|
+
const marker = "export const meta = ";
|
|
292
|
+
const start = source.indexOf(marker);
|
|
293
|
+
if (start < 0) continue;
|
|
294
|
+
const bodyStart = start + marker.length;
|
|
295
|
+
const end = source.indexOf("\n};", bodyStart);
|
|
296
|
+
if (end < 0) continue;
|
|
297
|
+
try {
|
|
298
|
+
// The meta block is a static literal (no function calls); evaluating
|
|
299
|
+
// the object literal in isolation is safe.
|
|
300
|
+
const meta = new Function("return (" + source.slice(bodyStart, end + 2) + ");")();
|
|
301
|
+
if (meta && Array.isArray(meta.steps)) {
|
|
302
|
+
const names = meta.steps.map((s) => s && s.name).filter((n) => typeof n === "string");
|
|
303
|
+
if (names.length > 0) return names;
|
|
304
|
+
}
|
|
305
|
+
} catch (e) { /* try the next candidate */ }
|
|
306
|
+
}
|
|
307
|
+
return null;
|
|
308
|
+
}
|
|
309
|
+
|
|
263
310
|
// Minimal git-repo validation (ported from the dashboard's validateRepoPath):
|
|
264
311
|
// accepts a repo root (.git dir) or a linked worktree (.git file).
|
|
265
312
|
function isGitRepoPath(p) {
|
|
@@ -536,6 +583,17 @@ commands["claim-task"] = (db, args) => {
|
|
|
536
583
|
if (!args.task_id) throw usageError("task_id is required.");
|
|
537
584
|
const identity = (args.identity ?? "").trim();
|
|
538
585
|
if (identity.length < 1 || identity.length > 80) throw usageError("identity is required (1-80 chars).");
|
|
586
|
+
// ATOMIC NEXT_PHASE CONSUMPTION (2026-09-15): expected_next_phase is the
|
|
587
|
+
// routing the dispatcher launched this run for. When the claim wins, the
|
|
588
|
+
// matching next_phase is cleared in the SAME transaction as the winning
|
|
589
|
+
// session insert — there is no claim→consume window for a platform death
|
|
590
|
+
// to replay the routing through. The separate consume-next-phase command
|
|
591
|
+
// is retained only as an idempotent public helper (out-of-band recovery);
|
|
592
|
+
// workflows no longer call it.
|
|
593
|
+
const rawExpected = args.expected_next_phase;
|
|
594
|
+
const expectedNextPhase = (rawExpected === undefined || rawExpected === null) ? null : String(rawExpected).trim();
|
|
595
|
+
if (expectedNextPhase !== null && (expectedNextPhase.length < 1 || expectedNextPhase.length > 120))
|
|
596
|
+
throw usageError("expected_next_phase must be 1-120 chars when provided.");
|
|
539
597
|
const task = requireTask(db, args.task_id);
|
|
540
598
|
const project = requireProject(db, task.project);
|
|
541
599
|
if (project.quiesced) throw conflict("This project is paused. Resume it before claiming tasks.");
|
|
@@ -553,23 +611,41 @@ commands["claim-task"] = (db, args) => {
|
|
|
553
611
|
notes: (args.notes ?? "").trim().slice(0, 3000), caveats: "[]",
|
|
554
612
|
last_heartbeat: now(),
|
|
555
613
|
};
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
614
|
+
const timestamp = now();
|
|
615
|
+
db.exec("BEGIN");
|
|
616
|
+
try {
|
|
617
|
+
// Atomic: the partial unique index on (task_id) WHERE status='running'
|
|
618
|
+
// turns a duplicate claim into a no-op insert.
|
|
619
|
+
const info = db.prepare(
|
|
620
|
+
`INSERT INTO agent_sessions (id, task_id, identity, step, status, started_at,
|
|
621
|
+
ended_at, notes, caveats, last_heartbeat)
|
|
622
|
+
VALUES (@id, @task_id, @identity, @step, @status, @started_at,
|
|
623
|
+
@ended_at, @notes, @caveats, @last_heartbeat)
|
|
624
|
+
ON CONFLICT DO NOTHING`).run(row);
|
|
625
|
+
// Same transaction: only the claim winner consumes, and only when the
|
|
626
|
+
// stored routing still matches what the dispatcher launched on — a newer
|
|
627
|
+
// recover-task written in the race window survives untouched.
|
|
628
|
+
let nextPhaseConsumed = false;
|
|
629
|
+
if (info.changes > 0 && expectedNextPhase) {
|
|
630
|
+
const cleared = db.prepare(
|
|
631
|
+
"UPDATE tasks SET next_phase = NULL, updated_at = ? WHERE id = ? AND next_phase = ?"
|
|
632
|
+
).run(timestamp, args.task_id, expectedNextPhase);
|
|
633
|
+
nextPhaseConsumed = cleared.changes > 0;
|
|
634
|
+
}
|
|
635
|
+
db.exec("COMMIT");
|
|
636
|
+
if (info.changes > 0) {
|
|
637
|
+
return { ok: true, claimed: true, session_id: row.id, session: mapSession({ ...row, failure_reason: null }),
|
|
638
|
+
next_phase_consumed: expectedNextPhase ? nextPhaseConsumed : undefined };
|
|
639
|
+
}
|
|
640
|
+
const existing = db.prepare(
|
|
641
|
+
`SELECT id FROM agent_sessions
|
|
642
|
+
WHERE task_id = ? AND status = 'running'
|
|
643
|
+
ORDER BY started_at DESC LIMIT 1`).get(args.task_id);
|
|
644
|
+
if (!existing) throw new CrewError("claim_unresolved", "Task claim could not be resolved. Please retry.", 4);
|
|
645
|
+
// Lost claim: the winner (or an earlier winner) owns next_phase
|
|
646
|
+
// consumption — a loser must never clear it.
|
|
647
|
+
return { ok: true, claimed: false, reason: "already_claimed", existing_session_id: existing.id };
|
|
648
|
+
} catch (e) { db.exec("ROLLBACK"); throw e; }
|
|
573
649
|
};
|
|
574
650
|
|
|
575
651
|
// Dispatch reservations: the cron worker creates one after launching a
|
|
@@ -845,6 +921,9 @@ commands["retry-platform-failure"] = (db, args) => {
|
|
|
845
921
|
if (failure.retry_count >= maxRetries) {
|
|
846
922
|
// Park the task — transient failures are not resolving.
|
|
847
923
|
const task = db.prepare("SELECT * FROM tasks WHERE id = ?").get(failure.task_id);
|
|
924
|
+
if (task && task.state === "done") {
|
|
925
|
+
return { ok: true, action: "skipped", reason: "task already done — a platform retry never parks a done task", retry_count: failure.retry_count };
|
|
926
|
+
}
|
|
848
927
|
if (task && task.state !== "parked") {
|
|
849
928
|
const msg = "Platform workflow failed " + (failure.retry_count + 1) + "x: " +
|
|
850
929
|
failure.error_message.substring(0, 200);
|
|
@@ -875,19 +954,47 @@ commands["retry-platform-failure"] = (db, args) => {
|
|
|
875
954
|
}
|
|
876
955
|
return { ok: true, action: "parked", reason: "max retries exceeded", retry_count: failure.retry_count, settled_sessions: 0 };
|
|
877
956
|
}
|
|
878
|
-
// Clear reservation and re-queue for retry
|
|
957
|
+
// Clear the reservation and re-queue for retry — preserving the task's
|
|
958
|
+
// phase. Canary 2026-09-15 (task 1d692d91): the old code forced
|
|
959
|
+
// state='todo', so a platform failure during QA restarted the entire
|
|
960
|
+
// workflow at Triage; Build reimplemented an already-merged and
|
|
961
|
+
// already-verified task and produced an unrequested variant. The
|
|
962
|
+
// dispatcher already resumes a failed/timed_out/stalled session at that
|
|
963
|
+
// session's own step — so the fix is to leave the task's state alone and
|
|
964
|
+
// settle the ghost session as 'stalled', letting the dispatcher resume at
|
|
965
|
+
// the failed phase instead of restarting at Triage. 'done' is never
|
|
966
|
+
// resurrected and 'parked' is never unparked: parked is the designed
|
|
967
|
+
// human decision point.
|
|
968
|
+
const retryTask = db.prepare("SELECT id, state FROM tasks WHERE id = ?").get(failure.task_id);
|
|
969
|
+
if (!retryTask) {
|
|
970
|
+
return { ok: true, action: "skipped", reason: "task not found" };
|
|
971
|
+
}
|
|
972
|
+
if (retryTask.state === "done") {
|
|
973
|
+
return { ok: true, action: "skipped", reason: "task already done — never resurrected by a platform retry" };
|
|
974
|
+
}
|
|
975
|
+
if (retryTask.state === "parked") {
|
|
976
|
+
return { ok: true, action: "skipped", reason: "task parked — the human decision point, never unparked by a platform retry" };
|
|
977
|
+
}
|
|
978
|
+
// The failed phase is the ghost session's step: the dispatcher resumes
|
|
979
|
+
// failed/timed_out/stalled sessions at that step. Capture it before
|
|
980
|
+
// settling so the requeue reports where the run will resume.
|
|
981
|
+
const ghost = db.prepare(
|
|
982
|
+
`SELECT step FROM agent_sessions WHERE task_id = ? AND status = 'running'
|
|
983
|
+
ORDER BY started_at DESC LIMIT 1`
|
|
984
|
+
).get(failure.task_id);
|
|
879
985
|
db.exec("BEGIN");
|
|
880
986
|
try {
|
|
881
987
|
db.prepare("DELETE FROM dispatch_reservations WHERE task_id = ?").run(failure.task_id);
|
|
882
988
|
db.prepare(
|
|
883
|
-
"UPDATE tasks SET
|
|
989
|
+
"UPDATE tasks SET updated_at = datetime('now') WHERE id = ?"
|
|
884
990
|
).run(failure.task_id);
|
|
885
991
|
// Settle the ghost session atomically with the requeue: the platform
|
|
886
992
|
// workflow is dead, but its agent_sessions row still says 'running' —
|
|
887
993
|
// and the dispatcher skips any task whose latest session is running
|
|
888
994
|
// (crew-dispatch.js "work in flight"). Without this, the requeued task
|
|
889
995
|
// would sit until the 1-hour zombie-session sweep. 'stalled' is what
|
|
890
|
-
// the dispatcher already treats as a retry candidate
|
|
996
|
+
// the dispatcher already treats as a retry candidate, resuming at the
|
|
997
|
+
// session's own step.
|
|
891
998
|
const ts = new Date().toISOString();
|
|
892
999
|
const settled = db.prepare(
|
|
893
1000
|
`UPDATE agent_sessions
|
|
@@ -905,6 +1012,8 @@ commands["retry-platform-failure"] = (db, args) => {
|
|
|
905
1012
|
ok: true,
|
|
906
1013
|
action: "requeued",
|
|
907
1014
|
task_id: failure.task_id,
|
|
1015
|
+
state: retryTask.state,
|
|
1016
|
+
resume_step: ghost ? ghost.step : null,
|
|
908
1017
|
retry_count: failure.retry_count + 1,
|
|
909
1018
|
settled_sessions: Number(settled.changes),
|
|
910
1019
|
};
|
|
@@ -951,41 +1060,77 @@ commands["recover-task"] = (db, args, ctx) => {
|
|
|
951
1060
|
const targetPhase = (args.target_phase ?? "").trim();
|
|
952
1061
|
if (targetPhase.length < 1 || targetPhase.length > 120) throw usageError("target_phase is required (1-120 chars).");
|
|
953
1062
|
const task = requireTask(db, args.task_id);
|
|
1063
|
+
if (!task.workflow) throw conflict("This task does not have a workflow with selectable phases.");
|
|
1064
|
+
if (!workflowExists(ctx.crewHome, task.workflow)) throw notFound("Workflow not found.");
|
|
1065
|
+
// Validate the target against the workflow's own phase registry — an
|
|
1066
|
+
// unknown phase fails closed here, never as a dispatcher skip or a
|
|
1067
|
+
// workflow launched at the wrong step.
|
|
1068
|
+
const stepNames = workflowStepNames(ctx.crewHome, task.workflow);
|
|
1069
|
+
if (!stepNames) throw conflict(`Could not read the phase registry for workflow "${task.workflow}".`);
|
|
1070
|
+
if (!stepNames.includes(targetPhase)) {
|
|
1071
|
+
throw conflict(`Unknown phase "${targetPhase}" for workflow "${task.workflow}". Valid phases: ${stepNames.join(", ")}.`);
|
|
1072
|
+
}
|
|
1073
|
+
// Recovery intent, not a verdict: the latest session may be failed,
|
|
1074
|
+
// timed_out, stalled — or absent entirely (the platform died before the
|
|
1075
|
+
// first claim). A running session means work is in flight; completed,
|
|
1076
|
+
// rejected, passed, and superseded sessions have their own normal paths
|
|
1077
|
+
// (next step, rework) and are not recovery targets.
|
|
954
1078
|
const latest = db.prepare(
|
|
955
1079
|
`SELECT * FROM agent_sessions WHERE task_id = ? ORDER BY started_at DESC LIMIT 1`
|
|
956
1080
|
).get(args.task_id);
|
|
957
|
-
if (
|
|
958
|
-
throw conflict(
|
|
1081
|
+
if (latest && !["failed", "timed_out", "stalled"].includes(latest.status)) {
|
|
1082
|
+
throw conflict(`The latest session is ${latest.status}; recover-task accepts only failed, timed_out, stalled, or no session.`);
|
|
959
1083
|
}
|
|
960
|
-
if (
|
|
961
|
-
|
|
1084
|
+
if (["done", "cancelled"].includes(task.state)) {
|
|
1085
|
+
throw conflict(`Task is ${task.state}; recovery applies to todo, in_progress, or parked tasks.`);
|
|
1086
|
+
}
|
|
1087
|
+
// No synthetic session: recovery intent must not consume retry budget.
|
|
1088
|
+
// The dispatcher routes on tasks.next_phase directly (one-shot, consumed
|
|
1089
|
+
// by the claiming workflow), so no failed-session marker is needed to
|
|
1090
|
+
// make the task eligible.
|
|
962
1091
|
const timestamp = now();
|
|
963
|
-
const
|
|
964
|
-
|
|
965
|
-
identity: latest.identity, step: targetPhase, status: "failed",
|
|
966
|
-
started_at: timestamp, ended_at: timestamp,
|
|
967
|
-
notes: (args.updated_description ?? "").trim().slice(0, 5000),
|
|
968
|
-
caveats: "[]", failure_reason: null, last_heartbeat: timestamp,
|
|
969
|
-
};
|
|
1092
|
+
const newState = (task.state === "parked" || task.state === "todo") ? "in_progress" : task.state;
|
|
1093
|
+
const note = (args.updated_description ?? "").trim().slice(0, 5000);
|
|
970
1094
|
db.exec("BEGIN");
|
|
971
1095
|
try {
|
|
972
|
-
db.prepare("UPDATE tasks SET next_phase = ?, updated_at = ? WHERE id = ?")
|
|
973
|
-
.run(targetPhase, timestamp, task.id);
|
|
974
|
-
db.prepare("UPDATE agent_sessions SET ended_at = ? WHERE id = ?").run(timestamp, latest.id);
|
|
1096
|
+
db.prepare("UPDATE tasks SET next_phase = ?, state = ?, updated_at = ? WHERE id = ?")
|
|
1097
|
+
.run(targetPhase, newState, timestamp, task.id);
|
|
975
1098
|
db.prepare(
|
|
976
|
-
`INSERT INTO
|
|
977
|
-
|
|
978
|
-
|
|
979
|
-
|
|
1099
|
+
`INSERT INTO events (id, type, task_id, identity, message, timestamp)
|
|
1100
|
+
VALUES (?, 'note', ?, 'recover-task', ?, ?)`
|
|
1101
|
+
).run(uuid(), task.id,
|
|
1102
|
+
`Recovery: next phase set to ${targetPhase}` +
|
|
1103
|
+
(task.state !== newState ? ` (state ${task.state} -> ${newState})` : "") +
|
|
1104
|
+
(note ? `. ${note}` : ""),
|
|
1105
|
+
timestamp);
|
|
980
1106
|
db.exec("COMMIT");
|
|
981
1107
|
} catch (e) { db.exec("ROLLBACK"); throw e; }
|
|
982
1108
|
const taskStates = new Map(db.prepare("SELECT id, state FROM tasks").all().map((r) => [r.id, r.state]));
|
|
983
1109
|
return {
|
|
984
|
-
task: mapTask({ ...task, next_phase: targetPhase, updated_at: timestamp }, taskStates),
|
|
985
|
-
|
|
1110
|
+
task: mapTask({ ...task, next_phase: targetPhase, state: newState, updated_at: timestamp }, taskStates),
|
|
1111
|
+
valid_phases: stepNames,
|
|
986
1112
|
};
|
|
987
1113
|
};
|
|
988
1114
|
|
|
1115
|
+
// Retained as an idempotent public helper for out-of-band recovery tooling.
|
|
1116
|
+
// Workflows no longer call this: claim-task with expected_next_phase clears
|
|
1117
|
+
// the routed next_phase in the same transaction as the winning session
|
|
1118
|
+
// insert, so the old claim→consume death window is closed. The conditional
|
|
1119
|
+
// UPDATE keeps this exactly-once for any other caller: only the claim winner
|
|
1120
|
+
// clears, and only when the value still matches what the dispatcher routed
|
|
1121
|
+
// on — a newer recover-task written in the race window survives untouched.
|
|
1122
|
+
commands["consume-next-phase"] = (db, args) => {
|
|
1123
|
+
if (!args.task_id) throw usageError("task_id is required.");
|
|
1124
|
+
const expected = (args.expected ?? "").trim();
|
|
1125
|
+
if (expected.length < 1 || expected.length > 120) throw usageError("expected is required (1-120 chars).");
|
|
1126
|
+
requireTask(db, args.task_id);
|
|
1127
|
+
const timestamp = now();
|
|
1128
|
+
const info = db.prepare(
|
|
1129
|
+
"UPDATE tasks SET next_phase = NULL, updated_at = ? WHERE id = ? AND next_phase = ?"
|
|
1130
|
+
).run(timestamp, args.task_id, expected);
|
|
1131
|
+
return { consumed: info.changes > 0, task_id: args.task_id };
|
|
1132
|
+
};
|
|
1133
|
+
|
|
989
1134
|
commands["upsert-session"] = (db, args) => {
|
|
990
1135
|
if (!args.task_id) throw usageError("task_id is required.");
|
|
991
1136
|
requireTask(db, args.task_id);
|