pi-durable-subagents 1.0.23 → 1.0.25

package/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 1.0.25
+
+- `restart` refusals show what a fence would cut short: each lease a running
+  call holds, with its mode, how long it has been held, its command and note.
+  Leases held outside those executions (a shell, a `systemd-run` unit) are
+  listed apart, since a restart leaves them held.
+- The waiting/moving check period is configured as `k.waitCheckMs` (was
+  `k.r7Ms` in 1.0.24; the old key is not accepted).
+- README: deduplicate events by `id`, not cursor; a call with no next
+  execution has no `fenced`; an open question keeps the orchestrator from its
+  idle exit (compact the event log now with a non-force `restart`); leases
+  outside calls survive restarts; why a restart cannot hand running
+  executions to the new orchestrator.
+
+## 1.0.24
+
+- `events --all [--since <cursor>] [--limit <n>]`: one durable log of
+  milestones across all workflows (`submitted`, `started`, `asking` with the
+  full question, `answered` without the text, `sealed` with status and data,
+  `fenced` with the reason of an interruption, `workflow-done`), read with an
+  `<epoch>:<seq>` cursor in pages of at most 1000. Delivery is at least once
+  without gaps, also across `kill -9` and restarts (re-derived events keep
+  their `id`); events are kept at least 7 days and never while their workflow
+  is unfinished or asking; an older cursor gets `cursor-expired` (exit 4). A
+  prune whose events cannot be logged first is rejected (`event-log: …`).
+- `run --labels <json>` (tool: `labels`): caller labels, part of the spec
+  digest, returned by `describe` and echoed on every event of the run.
+- Why a call does not move: `describe` adds `reason`, `detail` and
+  `since` to each waiting call, and the event log gets `waiting`/`moving`
+  when the reason changes: `unconfirmed-stop`, `provider-exhausted`,
+  `writer-lock`, `lease`, `slot`, `silent`.
+- README: the subagent force-restart guard is a rail against accidents, not a
+  security boundary.
+
 ## 1.0.23
 
 - An asker cut off by a restart (also `restart --force`) now hibernates and

package/README.md

@@ -235,12 +235,14 @@ pi-durable-subagents smoke              check this machine and this pi (offline,
 pi-durable-subagents chaos              run the fault suite (offline, about 2 minutes)
 pi-durable-subagents status [wid] [--json]
 pi-durable-subagents events <wid> [--json]   the meaningful timeline of one workflow
+pi-durable-subagents events --all [--since <cursor>] [--limit <n>] [--json]
+                                        milestones of every workflow, read with a cursor (see below)
 pi-durable-subagents tail [wid] [--json]
 pi-durable-subagents start              start the orchestrator if work is pending; sends nothing
 pi-durable-subagents resume [wid]       continue unfinished or parked work (undoes drain / stop-all)
 pi-durable-subagents drain              hold existing workflows: running calls finish, nothing new starts in them
 pi-durable-subagents stop <wid|call>
-pi-durable-subagents run --request <id> --spec <file|-> [--cwd <dir>] [--json] [--wait-ms <n>]
+pi-durable-subagents run --request <id> --spec <file|-> [--labels <json>] [--cwd <dir>] [--json] [--wait-ms <n>]
                                         start a run under a caller-chosen id; safe to retry (see below)
 pi-durable-subagents send --request <id> --to <run-id|wid/key> --kind follow-up|answer|steer|model
                          [--call <key>] [--qid <qid> --rev <n>] --message <text|@file> [--model <m>] [--json]
@@ -310,8 +312,9 @@ full text, `qid`, `rev` and the `to` address to answer), `sealed` (finished:
 `status` plus every call's unclipped `output`, `error` and schema `data`) or
 `pruned` (`pruned: {status, endedAt}`; workflows pruned before 1.0.21 have
 only `endedAt`), with `wid`, `request` and
-`spec_digest`. Live calls also show what they wait for (slot, writer lock,
-lease, exhausted provider). `lastFence: {at, exec, reason}` appears only
+`spec_digest`, and `labels` when the run has any. Live calls also show what
+they wait for (slot, writer lock, lease, exhausted provider; see "Labels and
+why a call does not move"). `lastFence: {at, exec, reason}` appears only
 when an execution was cut off: it had not ended its turn when it was fenced,
 did not hibernate on a question (also when recovery finds it cut off while only
 its question's `ask` ran), and was not ended on purpose (stop, timeout,
@@ -329,6 +332,118 @@ pi-durable-subagents send --request build-42-a1 --to build-42 --kind answer \
   --qid <qid> --rev <rev> --message "yes"
 ```
 
+#### Events across workflows
+
+`events --all` reads one durable log of milestones of every workflow
+(`$DSA_HOME/events.jsonl`, written only by the orchestrator), so a program
+without a daemon can poll it and react to completions and questions without
+reading every workflow. Output is JSON lines (with or without `--json`).
+
+```sh
+pi-durable-subagents events --all                          # {"head":"<epoch>:<seq>","more":false}
+pi-durable-subagents events --all --since <cursor> --limit 500
+```
+
+Every event has `id`, `cursor`, `ts` (when the milestone happened), `type`,
+`wid`, `request` (the run id, when the run was created by `run --request`),
+`labels` (the run's labels, when it has any), and for call events `key`,
+`gen` and `call` (`<wid>@<rev>/<key>@<gen>`):
+
+| type | fields |
+| --- | --- |
+| `submitted` | `name?` — the workflow was created |
+| `started` | `exec` — the first execution of a call (generation) began |
+| `asking` | `qid`, `rev`, `question` (full text), `to` (`<wid>/<key>`, the answer address) |
+| `answered` | `qid`, `rev`, `by`, `via?`, `digest` (sha256 hex of the UTF-8 answer), `length` (its length in UTF-16 code units, as JavaScript counts) — never the text; `describe` has it |
+| `sealed` | `status` (`ok`, `failed`, `gate-failed`, `stopped`, `timeout`, `budget`, `unknown`, …), `error?` (unclipped), `data` when its JSON is at most 16 KiB, else `data_omitted: <bytes>` (read it with `describe`) |
+| `fenced` | `exec` (the execution cut off), `reason` (`restart-force`, `orchestrator-crash`, `process-died`), `at` — an execution was interrupted and the call resumed in a new one: the processes its tools had started are gone |
+| `workflow-done` | `status`, `error?` |
+
+Readers must ignore types they do not know (`waiting`/`moving` follow).
+`by` is the sender of the answer: `session:<id>` for a pi session (with
+`via: "ui"` when it came from the subagent list), `cli:<user>@<host>` for the
+CLI (a subagent answering through the CLI also shows as `cli:…`), else
+`unknown`. `fenced` is emitted when the call's next execution begins (right
+after recovery, before it waits for a slot) and only when the fence
+interrupted work, exactly as `describe`'s `lastFence`: a turn that had ended,
+a hibernated question, an answer's resume or a seal are no `fenced`. A `once`
+call cut off in a tool is never resumed: it gets `sealed` with status
+`unknown` and no `fenced`. A call with no next execution has no `fenced`; its
+`sealed` carries the outcome (`describe`'s `lastFence` still names the fence).
+
+Cursors are `<epoch>:<seq>`; `--since c` returns the events after `c` in log
+order, at most `--limit` (default and maximum 1000), then
+`{"head": …, "more": …}`. With `more: true`, `head` is the cursor of the last
+event printed: pass it as the next `--since`. With `more: false`, `head` is
+the log's head; it may name a seq that no event has (every orchestrator start
+skips 1000 seqs, so a seq you saw in a write that a power cut undid is never
+reused), and it is still a valid cursor. Without `--since` only the head is
+printed. When no log exists yet, the command starts the orchestrator (which
+creates it from everything still on disk) and waits up to `--wait-ms`
+(default 60 s), else prints `{"pending": true}` and exits 75; when the log
+exists it never starts anything.
+
+Delivery is at least once, without gaps: after a crash the orchestrator
+derives again from its last durable watermark, and an event derived again has
+the same `id` (a new cursor). Deduplicate by `id`, not by cursor (keeping
+ids for the retention window is enough), and persist your cursor only after
+you applied the events of a page.
+
+Retention: an event is dropped only when it was logged more than 7 days ago
+(`"k": { "eventRetentionMs": … }` in `$DSA_HOME/config.json`) and its workflow is
+finished in its current revision (done, failed or stopped — not parked) with
+no open question and no unsealed call, or was pruned. The log is compacted at
+orchestrator start and at most hourly; to compact now while a question is open
+(which keeps the orchestrator from idle exit), run `restart` without `--force`. A cursor of another epoch (the log was
+replaced: a corrupt log is kept aside as `events.jsonl.corrupt-<ms>` and a new
+one starts), below the highest dropped seq, or beyond the head gets exit 4
+and one line `{"error":"cursor-expired","head":"…","oldest":"…"}` (`oldest`
+is the smallest cursor still accepted). To recover, run `describe --key` for
+every run you have not closed (it reports `sealed`, `asking` with the full
+question, `pruned`, …), rebuild your state from those answers, then continue
+with `--since <head>` from that reply. A malformed cursor or option exits 1
+with `{"error":"invalid-arguments","message":…}`.
+
+#### Labels and why a call does not move
+
+`run --request <id> --spec <file> --labels '{"node":"n1","attempt":"2"}'`
+(tool: `labels: {…}`) attaches your own labels to a run: a flat JSON object
+of at most 32 keys `[A-Za-z0-9_.:-]{1,64}` with string values of at most 256
+characters, at most 4096 bytes of JSON. They are part of the content: the
+same id with other labels (or none) is a `request-conflict` (exit 3). Give
+them only with `--labels`; a `labels` field in the spec file is refused.
+Invalid labels exit 1 and submit nothing, and the orchestrator rejects a
+request that carries invalid ones (`invalid-labels: …`). `describe` returns
+them as `labels` (also after `prune`), and every event of the run carries
+them.
+
+When an unsealed call does not move, its `waiting` in `describe` adds
+`reason`, `detail` (the status line for that cause, e.g. `waiting for a slot:
+probe 1/1`) and `since` (ms: when that cause started). The event log has the
+same: `waiting {reason, detail, since}` when the reason appears or changes,
+`moving {after}` when it clears (also when the call ends), checked every
+`k.waitCheckMs` (default 5 s; read when the orchestrator starts, unlike the other
+`k` settings a `config.json` change does not apply it until a restart); a
+change of detail alone is no event. The first reason
+that applies wins:
+
+| reason | the call … |
+| --- | --- |
+| `unconfirmed-stop` | had processes that did not exit after SIGKILL; it starts nothing until they are gone (look at them) |
+| `provider-exhausted` | runs on, or can only be admitted to, providers whose usage window is used up |
+| `writer-lock` | waits for another call that writes in the same worktree |
+| `lease` | waits for a resource lease (`hold`, below) |
+| `slot` | is queued for a provider slot or memory headroom (at once when its providers are full, else after 3 s) |
+| `silent` | is running (launched, not stopped) without visible activity: the stall notice of that execution, with the command running and for how long |
+
+A call whose current execution asked a question (also while it hibernates
+until the answer) is `asking`, not waiting. Once the answer arrives the call
+launches again, and from then on it waits like any call (for a slot, the
+writer lock, …) even though `describe` still lists the question as open until
+the new execution reads it (its `state` stays `asking`). A drained call
+(no execution running) is never `silent`; a sealed call never waits. `provider-exhausted`, `writer-lock`, `lease` and `slot` are queues
+that clear by themselves; `silent` and `unconfirmed-stop` may need a look.
+
 ### Housekeeping
 
 Journals are never compacted, so state only grows. `prune` removes finished
@@ -349,7 +464,7 @@ command:
 ```sh
 pi-durable-subagents hold machine -- make bench          # exclusive
 pi-durable-subagents hold machine --shared -- npm test   # with other shared holders, never with an exclusive one
-pi-durable-subagents hold machine --max-wait 600 --note "frame phase" -- ./measure.sh
+pi-durable-subagents hold machine --max-wait 600 --note "profile" -- ./measure.sh
 ```
 
 - The lease covers one command, not a whole call: a subagent that thinks
@@ -370,6 +485,10 @@ pi-durable-subagents hold machine --max-wait 600 --note "frame phase" -- ./measu
   ended). State is one small file per request under
   `$DSA_HOME/leases/<resource>/`; no orchestrator is needed, and the user's
   own shell can take part.
+- A lease taken outside any call (your shell, a `systemd-run --user` unit)
+  does not depend on the orchestrator: a restart, forced or not, leaves it
+  held, and it is released when its `hold` and command end. A lease taken
+  inside a call ends with that call's processes when the call is fenced.
 - Subagents find the command on their `PATH` (the orchestrator puts a shim
   in `$DSA_HOME/bin`), and their leases are tagged with their call:
   `status` shows `lease: machine held by <wid>/<key> …; waiting: …` and
@@ -453,8 +572,11 @@ On load, it checks the pi exports and API methods it uses.
 Running work stays on the version it started with until you restart the
 orchestrator. When the orchestrator runs another version than the one a pi
 session loaded, that pi says so once, and `status` shows the running version
-with a note. The orchestrator exits about 10 s after all work ends, and the
-next start runs the new version. To switch sooner:
+with a note. The orchestrator exits about 10 s (`k.idleExitMs`) after all
+work ends: every workflow is finished in its current revision (done, failed,
+stopped or parked) or held by `drain`. A workflow with an open question is not
+finished, so a call hibernated on its question keeps the orchestrator running
+(it holds no slot and costs little). The next start runs the new version. To switch sooner:
 
 ```sh
 pi-durable-subagents restart          # or the subagents tool: action "restart"
@@ -462,7 +584,9 @@ pi-durable-subagents restart          # or the subagents tool: action "restart"
 
 The orchestrator refuses while any execution runs (a subagent process, or a
 gate before a call's seal). The refusal groups executions by session with ages,
-lease annotations and a token for that exact set; no new execution starts while
+the leases each one holds (mode, how long, command, note: what a fence would cut
+short) and a token for that exact set. Leases held outside those executions are
+listed apart, since the restart leaves them held; no new execution starts while
 it decides, so nothing slips in between. Calls waiting
 for your answer (hibernated), waiting for a provider slot, or held by a drain
 do not block it. Otherwise it exits and its successor starts at once from the
@@ -481,9 +605,21 @@ pi-durable-subagents restart --force <token> --reason "<why>"
 A changed execution set is refused with a fresh list and token. With no live
 executions no token is needed. Bare force cannot fence live executions, and the
 tool rejects `force:true`. Subagents cannot force a restart, even from bash:
-it would fence themselves and other sessions' work. Force fences running
+it would fence themselves and other sessions' work. That guard reads the
+environment on purpose (`DSA_EXEC`/`DSA_CALL` and the request's initiator
+call): it is a rail against accidents and instructions, not a security
+boundary — a subagent runs as the same OS user and could signal the
+orchestrator anyway. Force fences running
 executions; they resume on the new version from their sessions, like after a
 crash, so a tool call that was running is repeated or reported as interrupted.
+A running execution cannot be handed over to the new orchestrator: each
+subagent is a pi process the orchestrator drives over its stdin and stdout, and
+those pipes end with the old process. A crash is no different: the successor
+fences every execution that still runs (an execution that had already ended is
+not counted as interrupted). Work that must survive a forced restart, such as a
+long measurement, belongs outside the subagent's processes (for example
+`systemd-run --user … pi-durable-subagents hold machine -- …`), with the
+subagent only watching it.
 The restart ledger records the reason and initiator; after the next start,
 `status` shows who forced it and why for 24 hours.
 

package/dist/agent/main/schema.js

@@ -16,10 +16,11 @@ export const parameters = Type.Object({
     usageBudget: Type.Optional(Type.Object({ tokens: Type.Optional(Type.Number()), costUsd: Type.Optional(Type.Number()) })),
     maxCalls: Type.Optional(Type.Integer({ minimum: 1 })), inputs: Type.Optional(Type.Record(Type.String(), Type.String())),
     name: Type.Optional(Type.String()),
+    labels: Type.Optional(Type.Record(Type.String(), Type.String(), { description: "run: your labels, e.g. {node, attempt}: at most 32 keys [A-Za-z0-9_.:-]{1,64}, string values of at most 256 characters, 4096 bytes of JSON; part of the request's content (spec_digest), shown by describe and on its events." })),
     timeoutMs: Type.Optional(Type.Number({ description: "Per-call limit on active time in milliseconds (a number). Omit unless a hard limit is needed; prefer budgets." })),
     key: Type.Optional(Type.String({ description: "A single agent/task run: the call's key. status with wid: that call's full result." })),
     full: Type.Optional(Type.Boolean({ description: "status: with wid, the complete workflow detail including every output." })),
-    force: Type.Optional(Type.Union([Type.String(), Type.Boolean()], { description: "restart: the token shown by a refusal. Show the user the list and obtain explicit approval first; boolean true is refused. Subagents cannot force a restart." })),
+    force: Type.Optional(Type.Union([Type.String(), Type.Boolean()], { description: "restart: the token shown by a refusal. Show the user the list and obtain explicit approval first; boolean true is refused. Subagents cannot force a restart (an environment-based rail against accidents, not a security boundary)." })),
     reason: Type.Optional(Type.String({ description: "restart: non-empty reason, at most 500 characters; required with force." })),
     request: Type.Optional(Type.String({ description: "run/send/stop: your own request id (1-124 chars [A-Za-z0-9][A-Za-z0-9._:-]*) making a retry safe: the same id with the same content gets the first outcome; other content is refused (request-conflict)." })),
 }, { additionalProperties: true });

package/dist/agent/main/tool.js

@@ -2,6 +2,7 @@ import { restartInputError } from "../../orchestrator/restart.js";
 import { resolve } from "node:path";
 import { validateCallSpec } from "../../compat/spec.js";
 import { compileFanout } from "../../compat/fanout.js";
+import { checkLabels } from "../../events/labels.js";
 /** Call fields a tasks/chain run applies to every step that does not set its own. */
 export const stepDefaults = ["model", "timeoutMs", "budget", "isolation", "context", "tools", "skills", "once", "writer"];
 function string(args, name) {
@@ -20,7 +21,7 @@ function call(value, cwd, where) {
     return spec;
 }
 /** v12 §2: Reject unknown explicit call agents before starter or outbox publication; scripts remain call-local. Shared by
- *  the tool and the CLI `run --request` (R2). */
+ *  the tool and the CLI `run --request`. */
 export function checkAgents(body, available) {
     const names = [...(body.call ? [body.call] : []), ...(body.tasks ?? []), ...(body.chain ?? [])].map(call => call.agent);
     if (!names.length)
@@ -45,7 +46,7 @@ export function request(args, cwd) {
     if (typeof action !== "string" || !action)
         throw new Error("action is required: run, agents, send, stop, revise, status, resume, drain, restart");
     if (action === "run") {
-        const { action: _, workflow, source, tasks, chain, args: inputs, name, usageBudget, maxCalls, inputs: files, by: _by, request: _request, ...spec } = args;
+        const { action: _, workflow, source, tasks, chain, args: inputs, name, usageBudget, maxCalls, inputs: files, labels, by: _by, request: _request, ...spec } = args;
         const choices = [workflow, source, tasks, chain, spec.agent === undefined && spec.task === undefined ? undefined : spec];
         if (choices.filter(v => v !== undefined).length !== 1)
             throw new Error("run requires exactly one of workflow, source, tasks, chain, or agent/task");
@@ -84,6 +85,9 @@ export function request(args, cwd) {
             body.args = inputs;
         if (name !== undefined)
             body.name = string(args, "name");
+        // Part of the spec digest; an empty object is the same as none.
+        if (labels !== undefined && Object.keys(checkLabels(labels)).length)
+            body.labels = labels;
         // P31a, P36, P11: workflow-level limits and declared input files (absolute paths, pinned at admission).
         if (usageBudget !== undefined) {
             const b = usageBudget;

package/dist/agent/main.js

@@ -297,7 +297,7 @@ export function registerMain(pi, ui) {
         if (args.action === "resume" && args.wid === undefined)
             args = { ...args, origin: sender };
         const normalized = request(args, cwd);
-        // R1: a caller-chosen request id names a run, send or stop; a retry with the same content gets the first outcome.
+        // A caller-chosen request id names a run, send or stop; a retry with the same content gets the first outcome.
         if (args.request !== undefined && (typeof args.request !== "string" || !["run", "send", "stop"].includes(normalized.kind) || normalized.replaces?.length))
             throw new Error(REQUEST_USE);
         const rid = typeof args.request === "string" ? requestRid(args.request) : undefined;
@@ -378,7 +378,7 @@ export function registerMain(pi, ui) {
     pi.registerTool(defineTool({
         name: "subagents", label: "Subagents", description: [
             "Durable asynchronous subagents; run returns {wid} when created (or {submitted:{rid}} while pending). A finished workflow (its notice carries every agent's result) or a question wakes you, so after starting work end your turn: never poll with sleep or repeated status. Crash recovery resumes sessions, not external side effects. Background helper processes (orchestrator, evaluator) exit by themselves about 10 s after all work ends: never kill processes or delete files to 'clean up'. When the user quits pi, this session's running workflows pause (nothing is spent); resume continues them.",
-            "run (action optional for exactly one launch form): agent+task; tasks:[call specs] parallel; chain:[call specs] sequential ({previous}); workflow:'./script.js' or source (runs.run(key,spec), runs.all([...]), emit(value), args, runs.input(name)). Optional name, cwd, usageBudget, maxCalls, inputs. With tasks/chain, top-level model, timeoutMs, budget, isolation, context, tools, skills, once are defaults for every step (a step's own value wins); a workflow/source script sets them per runs.run call. timeoutMs is milliseconds of active time (a number); omit it unless a hard limit is needed. Explicit unknown agents are rejected BEFORE creation, with available names; unknown script agents fail only their call.",
+            "run (action optional for exactly one launch form): agent+task; tasks:[call specs] parallel; chain:[call specs] sequential ({previous}); workflow:'./script.js' or source (runs.run(key,spec), runs.all([...]), emit(value), args, runs.input(name)). Optional name, cwd, usageBudget, maxCalls, inputs, labels. With tasks/chain, top-level model, timeoutMs, budget, isolation, context, tools, skills, once are defaults for every step (a step's own value wins); a workflow/source script sets them per runs.run call. timeoutMs is milliseconds of active time (a number); omit it unless a hard limit is needed. Explicit unknown agents are rejected BEFORE creation, with available names; unknown script agents fail only their call.",
             "agents: list names, descriptions, default models and source for this cwd; use these names for run.",
             "send to:'<wid>/<key>' (bare '<wid>' only for a single-call workflow): steer on a running call delivers at the next safe point (receipt in status/UI); a steer to a call waiting on its question interrupts the question and the subagent usually asks again — use answer to answer it; sealed → finished:<status> — use kind 'follow-up'. follow-up continues a sealed call as generation g+1 or queues after a running turn; follow-up model:'provider/id' or a pool name runs that generation on it. answer: give the qid (or just the call, or nothing when one question is open); to and rev are filled in. A question that needs the user's decision goes to the user; if you answer one yourself, tell the user what you chose. model ('provider/id' or a pool name — its first model not used up): a running call switches at its next provider request; an asking, hibernated or queued call launches on it when it runs again; the reply's model/effect (next-request|next-execution|next-generation) says which. status model = model actually used by the last request; switching = requested, not used yet; switchFailed = refused. A provider content refusal (ToS/usage policy) fails the call at once, not retried. Unknown targets list valid addresses. replaces:[rid] supersedes an earlier send.",
             "stop target:<wid|<wid>/<key>> is terminal stopped (usage and partial edits kept); a sealed call → already-sealed:<status>, a finished workflow → terminal:<status>. drain holds existing workflows reversibly (new runs unaffected); resume [wid] releases held workflows. restart (after an update) replaces the orchestrator with the installed version: refused with busy:<running executions> while any runs. Never force without the user's explicit approval: show the user the refusal's list first, then supply force:'<token>' and reason. Subagents cannot force; hibernated askers and queued calls do not block it. Never kill the orchestrator process. Commands that need the machine (benchmarks, timing) take a lease: tell the subagent to run them as `pi-durable-subagents hold machine [--shared] -- <command>` (FIFO; status lists lease holders and waiters). status: without wid, what runs, asks (with its answer address; hibernated:true holds no slot) or failed, writerWait: a call queued for its git worktree's writer lock (one call whose tools include edit/write runs per worktree; spec writer:false or isolation:'worktree' opts out), sharedWorktree names calls sharing observed edit/write roots (reminder), lease: a call holding or waiting for a resource lease, finished workflows one line each, provider slots held/limit, the config in effect and providers whose usage window is used up (avoided until a probe finds them answering again), and the orchestrator version (versionNote when it differs from the loaded one); wid: one workflow, outputs clipped; wid+key: one call's full result; full:true: everything. A run's rid from {submitted:{rid}} works wherever a wid is expected. revise wid + workflow/source/args starts a revision.",

package/dist/cli/control.js

@@ -104,7 +104,7 @@ export async function submit(home, command, target, env = process.env, options =
         return requests;
     });
 }
-/** R1: Submit a request named by a caller-chosen id through the CLI sender: a retry with the same content republishes
+/** Submit a request named by a caller-chosen id through the CLI sender: a retry with the same content republishes
  *  (or reuses) the recorded envelope, other content is a conflict and publishes nothing. Starts the orchestrator
  *  unless the request conflicts. */
 export async function submitIdentified(home, rid, kind, body, cond, env = process.env, starter = startOrchestrator) {

package/dist/cli/main.js

@@ -132,7 +132,7 @@ export function serviceEntryError(entry) {
         return `install-service refuses to run from an npx cache (${entry}); the cache can be pruned and the service would break. Install the CLI with \`npm i -g pi-durable-subagents\` and run \`pi-durable-subagents install-service\` again.`;
     return undefined;
 }
-export const HELP = "pi-durable-subagents: smoke | status [wid] [--json] | events <wid> [--json] | tail [wid] [--json] | start | resume [wid] | drain | stop <wid|callId> | stop-all | run --request <id> --spec <file|-> [--cwd <dir>] [--json] [--wait-ms <n>] | send --request <id> --to <run-id|wid/key> [--call <key>] --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] [--message <text|@file>] [--model <m>] [--json] [--wait-ms <n>] | stop --request <id> <run-id|wid|wid/key> [--json] [--wait-ms <n>] | describe --key <id> | describe <wid> [--json] | prune [wid] [--older-than <days>] | restart [--force <token> --reason <text>] | hold <resource> [--shared] [--max-wait <s>] [--note <text>] -- <command…> | leases [--json] | doctor [--json] | install-service [--dry-run] | uninstall-service [--dry-run] | chaos [--scenario <1-9>] [--keep] [--json]";
+export const HELP = "pi-durable-subagents: smoke | status [wid] [--json] | events <wid> [--json] | events --all [--since <cursor>] [--limit <n>] [--json] [--wait-ms <n>] | tail [wid] [--json] | start | resume [wid] | drain | stop <wid|callId> | stop-all | run --request <id> --spec <file|-> [--labels <json>] [--cwd <dir>] [--json] [--wait-ms <n>] | send --request <id> --to <run-id|wid/key> [--call <key>] --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] [--message <text|@file>] [--model <m>] [--json] [--wait-ms <n>] | stop --request <id> <run-id|wid|wid/key> [--json] [--wait-ms <n>] | describe --key <id> | describe <wid> [--json] | prune [wid] [--older-than <days>] | restart [--force <token> --reason <text>] | hold <resource> [--shared] [--max-wait <s>] [--note <text>] -- <command…> | leases [--json] | doctor [--json] | install-service [--dry-run] | uninstall-service [--dry-run] | chaos [--scenario <1-9>] [--keep] [--json]";
 /** Restart: the orchestrator exits when no execution runs (or `force`) and the installed version takes over. */
 async function restartCommand(home, env, write, options) {
     const body = { ...(typeof options.force === "string" ? { token: options.force } : options.force === true ? { force: true } : {}), ...(options.reason !== undefined ? { reason: options.reason } : {}), initiator: cliInitiator(env) };
@@ -194,7 +194,12 @@ export async function main(args = process.argv.slice(2), options = {}) {
     }
     if (args[0] === "chaos")
         return (await import("./chaos/index.js")).chaos(args.slice(1), options.env ?? process.env, options.write);
-    // R1–R3: program-facing commands named by request ids (strict flags of their own).
+    // The cross-workflow event log (strict flags of its own); `events <wid>` stays below.
+    if (args[0] === "events" && args.includes("--all")) {
+        const env = options.env ?? process.env;
+        return (await import("../events/cli.js")).eventsAll(args.slice(1), { home: dsaHome(env), env, write: options.write ?? ((line) => console.log(line)), starter: options.starter ?? startOrchestrator, waitMs: options.waitMs });
+    }
+    // Program-facing commands named by request ids (strict flags of their own).
     if (["run", "send", "describe"].includes(args[0]) || (args[0] === "stop" && args.includes("--request"))) {
         const env = options.env ?? process.env, requests = await import("./requests.js");
         const ctx = { home: dsaHome(env), env, write: options.write ?? ((line) => console.log(line)), starter: options.starter, waitMs: options.waitMs, cwd: options.cwd, stdin: options.stdin };

package/dist/cli/requests.js

@@ -1,12 +1,12 @@
-// R1–R4: Program-facing commands named by caller-chosen request ids — `run|send|stop --request <id>` and `describe`.
+// Program-facing commands named by caller-chosen request ids — `run|send|stop --request <id>` and `describe`.
 // Exit codes: 0 decided (applied/created), 1 rejected or invalid, 3 request-conflict (the id names other content),
 // 75 not decided within --wait-ms (retry with the same id and content: safe).
 import { existsSync, readFileSync } from "node:fs";
-import { resolve } from "node:path";
+import { join, resolve } from "node:path";
 import { setTimeout as delay } from "node:timers/promises";
 import { readJournalSnapshot } from "../kernel/journal.js";
 import { reduceLifecycle } from "../kernel/lifecycle.js";
-import { journalPath, orchLedger } from "../paths.js";
+import { journalPath, orchLedger, pinnedDir } from "../paths.js";
 import { findRequest, REQUEST_ID, requestId, requestRid, RequestsBusy, specDigest } from "../requests.js";
 import { isLive, slotsView, workflowSnapshot } from "../orchestrator/snapshot.js";
 import { leaseCalls, leaseState } from "../platform/lease.js";
@@ -14,6 +14,10 @@ import { checkAgents, request } from "../agent/main/tool.js";
 import { discoverAgents } from "../compat/agents.js";
 import { JT } from "../types.js";
 import { startOrchestrator, submitIdentified } from "./control.js";
+import { endedExecs, fenceReason } from "../events/fence.js";
+import { parseLabels } from "../events/labels.js";
+import { foldWaits, leaseWaits, waitsOf } from "../events/waiting.js";
+import { emptyLedger, foldLedger } from "../orchestrator/ledger.js";
 export const EXIT = { ok: 0, rejected: 1, conflict: 3, pending: 75 };
 /** Strict `--name value` / `--flag` parsing; unknown or repeated options are errors. */
 function flags(args, spec) {
@@ -53,7 +57,7 @@ function decision(entries, rid) {
 const createdBy = (entries, rid) => entries.find(e => e.type === JT.created && e.rid === rid);
 async function stdin() { const chunks = []; for await (const chunk of process.stdin)
     chunks.push(chunk); return Buffer.concat(chunks).toString("utf8"); }
-/** R3: The state of a request id (`{request}`) or a workflow (`{wid}`), with full texts (no clipping). */
+/** The state of a request id (`{request}`) or a workflow (`{wid}`), with full texts (no clipping). */
 export async function describe(home, key, now = Date.now()) {
     const entries = ledger(home);
     if ("wid" in key)
@@ -61,7 +65,7 @@ export async function describe(home, key, now = Date.now()) {
     const rid = requestRid(key.request), found = await findRequest(home, rid);
     if (!found)
         return { state: "absent", request: key.request };
-    const head = { request: key.request, kind: found.request.kind, spec_digest: specDigest(found.request) };
+    const head = { request: key.request, kind: found.request.kind, spec_digest: specDigest(found.request), ...labelsOf(found.request) };
     const decided = decision(entries, rid), created = createdBy(entries, rid);
     if (found.request.kind === "run" && created)
         return { ...await describeWorkflow(home, String(created.wid), entries, now), ...head };
@@ -72,24 +76,57 @@ export async function describe(home, key, now = Date.now()) {
         return { state: "applied", ...head };
     return { state: "pending", ...head };
 }
+/** `{labels}` of a run request that has any. */
+function labelsOf(request) {
+    const labels = request?.kind === "run" ? request.body?.labels : undefined;
+    return labels && typeof labels === "object" && Object.keys(labels).length ? { labels } : {};
+}
+/** The admitted run request that created `wid` (the ledger keeps it after a prune). */
+function runOf(entries, wid) {
+    const created = entries.find(e => e.type === JT.created && e.wid === wid);
+    return { created, run: created ? entries.find(e => e.type === "request" && e.request.rid === created.rid)?.request : undefined };
+}
+/** The pinned agents of a workflow revision, read once on demand (the waiting check needs the model of a call that names none). */
+function pinnedAgentModel(home, wid, rev) {
+    let agents;
+    return name => {
+        if (!agents) {
+            try {
+                agents = JSON.parse(readFileSync(join(pinnedDir(home, wid), rev === 1 ? "" : `r${rev}`, "agents.json"), "utf8"));
+            }
+            catch {
+                agents = [];
+            }
+        }
+        return Array.isArray(agents) ? agents.find(a => a?.name === name)?.model : undefined;
+    };
+}
 function describeWorkflow(home, wid, entries, now) {
     const pruned = entries.find(e => e.type === "pruned" && e.wid === wid);
     if (pruned)
         return { state: "pruned", wid, pruned: { ...(pruned.status !== undefined ? { status: String(pruned.status) } : {}), endedAt: Number(pruned.endedAt) },
-            ...(typeof pruned.request === "string" ? { request: pruned.request } : {}), ...(typeof pruned.spec_digest === "string" ? { spec_digest: pruned.spec_digest } : {}) };
+            ...(typeof pruned.request === "string" ? { request: pruned.request } : {}), ...(typeof pruned.spec_digest === "string" ? { spec_digest: pruned.spec_digest } : {}), ...labelsOf(runOf(entries, wid).run) };
     if (!/^[^/\\\0]+$/.test(wid) || wid === "." || wid === ".." || !existsSync(journalPath(home, wid)))
         return { state: "absent", wid };
-    const wf = workflowSnapshot(home, wid), journal = readJournalSnapshot(journalPath(home, wid));
-    const created = entries.find(e => e.type === JT.created && e.wid === wid), id = created ? requestId(String(created.rid)) : undefined;
-    const admitted = id ? entries.find(e => e.type === "request" && e.request.rid === created.rid)?.request : undefined;
-    const slots = slotsView(home, now), leases = leaseCalls(leaseState(home), now);
+    // The snapshot and the journal the wait fold reads must be the same bytes (a writer-wait appended between the two reads
+    // would give a reason without its writerWait): read again until the journal did not move around the snapshot.
+    let journal = readJournalSnapshot(journalPath(home, wid)), wf = workflowSnapshot(home, wid);
+    for (let i = 0, again = readJournalSnapshot(journalPath(home, wid)); again !== journal && i < 5; i++, again = readJournalSnapshot(journalPath(home, wid))) {
+        journal = again;
+        wf = workflowSnapshot(home, wid);
+    }
+    const { created, run } = runOf(entries, wid), id = created ? requestId(String(created.rid)) : undefined;
+    const admitted = id ? run : undefined;
+    const lstate = leaseState(home), slots = slotsView(home, now), leases = leaseCalls(lstate, now);
+    // The same fold and decision the orchestrator's collector uses, from the disk snapshots.
+    const waits = waitsOf(foldWaits(wid, journal), { now, ledger: foldLedger(emptyLedger(), entries), leases: leaseWaits(lstate, now) }, pinnedAgentModel(home, wid, wf.rev));
     const line = (lines, model) => { const provider = model?.split("/")[0]; return provider ? lines?.find(l => l.startsWith(`${provider} `)) : undefined; };
     const latest = [...new Map(wf.calls.map(c => [c.key, c])).values()];
     const calls = latest.map((c) => {
         const r = c.result, waiting = r ? undefined : {
             ...(c.writerWait ? { writerWait: c.writerWait } : {}), ...(leases.get(c.callId) ? { lease: leases.get(c.callId) } : {}),
             ...(line(slots.slots, c.model) ? { slot: line(slots.slots, c.model) } : {}), ...(line(slots.exhausted, c.model) ? { exhausted: line(slots.exhausted, c.model) } : {}),
-            ...(c.hibernated ? { hibernated: true } : {})
+            ...(c.hibernated ? { hibernated: true } : {}), ...waits.get(c.callId)
         };
         return { key: c.key, gen: c.gen, phase: c.phase, agent: c.agent, ...(c.model ? { model: c.model } : {}),
             ...(r ? { status: r.status, ok: r.ok, ...(r.error ? { error: r.error } : {}), output: r.output, ...(r.data !== undefined ? { data: r.data } : {}) } : {}),
@@ -100,48 +137,20 @@ function describeWorkflow(home, wid, entries, now) {
     const attention = wf.attention.filter(a => a.kind !== "question").map(a => ({ id: a.id, rev: a.rev, kind: a.kind, ...(a.call ? { call: a.call } : {}), text: a.text }));
     const state = questions.length ? "asking" : isLive(wf) || wf.status === "parked" ? "running" : "sealed";
     const fence = lastFence(journal, entries);
-    return { state, wid, ...(id ? { request: id } : {}), ...(admitted ? { spec_digest: specDigest(admitted) } : {}), status: wf.status, ...(wf.error ? { error: wf.error } : {}),
+    return { state, wid, ...(id ? { request: id } : {}), ...(admitted ? { spec_digest: specDigest(admitted) } : {}), ...labelsOf(run), status: wf.status, ...(wf.error ? { error: wf.error } : {}),
         calls, ...(questions.length ? { questions } : {}), ...(attention.length ? { attention } : {}), ...(fence ? { lastFence: fence } : {}) };
 }
-/** R3, best effort: why the latest fence that interrupted work happened. Every execution ends with a fence; one interrupted
- *  work only when the execution neither settled (its turn ended) before it nor hibernated (it waits for an answer), and
- *  was not sealed on purpose: a seal ends an execution on purpose unless its outcome is `unknown` (a `once` call cut off
- *  in a tool) or the execution was recorded as lost (the loss bound sealed it), which are interruptions themselves.
- *  A seal for an execution that never ran (a launch failure) or that the call's stop, timeout or budget ended is on
- *  purpose. restart-force: a forced restart listed the execution as live;
- *  orchestrator-crash: the execution was launched before an orchestrator start that is not preceded by a clean exit and
- *  fenced after it (startup recovery); otherwise process-died (the child or its host went away, or a drain fenced it). */
+/** Best effort: why the latest fence that interrupted work happened. The per-execution classification and the
+ *  reason are shared with the event log's `fenced` events (src/events/fence.ts). */
 export function lastFence(journal, orch) {
-    const lost = new Set(journal.filter(e => e.type === "loss").map(e => String(e.exec)));
-    const fencedAt = new Map(journal.filter(e => e.type === JT.fenced).map(e => [String(e.exec), Number(e.seq)]));
-    const ended = new Set(journal.filter(e => {
-        const exec = String(e.exec);
-        // Recovery records `hibernated` after the fence for an execution cut off while only its question's ask ran (P28):
-        // it was waiting, not working, so that is no interruption either.
-        if (e.type === "hibernated")
-            return true;
-        if (e.type === "settled")
-            return Number(e.seq) < (fencedAt.get(exec) ?? Infinity);
-        return e.type === JT.sealed && e.result?.status !== "unknown" && !lost.has(exec);
-    }).map(e => String(e.exec)));
+    const ended = endedExecs(journal);
     const fence = journal.findLast(e => e.type === JT.fenced && !ended.has(String(e.exec)));
-    if (!fence)
-        return undefined;
-    const exec = String(fence.exec), at = Number(fence.ts);
-    if (orch.some(e => e.type === "restart" && e.force === true && Array.isArray(e.live) && e.live.includes(exec)))
-        return { at, exec, reason: "restart-force" };
-    const launched = journal.find(e => e.type === JT.exec && e.exec === exec), starts = orch.filter(e => e.type === "orchestrator");
-    const recovery = starts.findLast(s => Number(s.ts) <= at);
-    if (launched && recovery && Number(launched.ts) < Number(recovery.ts)) {
-        const prior = orch.filter(e => Number(e.seq) < Number(recovery.seq));
-        const lastStart = prior.findLast(e => e.type === "orchestrator"), cleanExit = lastStart && prior.some(e => e.type === "orchestrator-exit" && Number(e.seq) > Number(lastStart.seq));
-        if (!cleanExit)
-            return { at, exec, reason: "orchestrator-crash" };
-    }
-    return { at, exec, reason: "process-died" };
+    return fence ? { at: Number(fence.ts), exec: String(fence.exec), reason: fenceReason(journal, orch, fence) } : undefined;
 }
 export function renderDescription(d) {
     const lines = [`${d.request ?? d.wid}: ${d.state}${d.reason ? ` (${d.reason})` : ""}${d.wid && d.request ? ` — ${d.wid}` : ""}${d.status && d.state !== d.status ? ` · ${d.status}` : ""}`];
+    if (d.labels)
+        lines.push(`  labels: ${Object.entries(d.labels).map(([k, v]) => `${k}=${v}`).join(" ")}`);
     if (d.pruned)
         lines.push(`  pruned: ${d.pruned.status ?? "?"} at ${new Date(d.pruned.endedAt ?? 0).toISOString()}`);
     if (d.error)
@@ -201,7 +210,7 @@ function pending(ctx, id, json, why = "") {
 }
 /** Submit, then wait; a decision whose admitted envelope has other content (another sender won the id) is a conflict. */
 async function submitAndWait(ctx, seen, id, kind, body, cond, wait, json) {
-    // R2: `created` is false only when the id was already decided before this invocation submitted (decisions are
+    // `created` is false only when the id was already decided before this invocation submitted (decisions are
     // monotonic); racing first attempts may all report created — the wid is what identifies the run.
     const rid = requestRid(id), earlier = Boolean(await outcome(ctx.home, rid, kind === "run", 0));
     let sent;
@@ -227,7 +236,7 @@ async function submitAndWait(ctx, seen, id, kind, body, cond, wait, json) {
         return { code: await conflict(ctx, id, specDigest(admitted), json) };
     return { sent, outcome: result, earlier };
 }
-/** R2: a failure once this invocation submitted, or once the id is found recorded with this content, is "not decided
+/** A failure once this invocation submitted, or once the id is found recorded with this content, is "not decided
  *  yet" (75), never a refusal. Otherwise, with --json, a request refused before submission (a usage error, an invalid
  *  spec, an unknown agent, no open question) answers `{request, applied:false, reason, spec_digest?}` with exit 1;
  *  spec_digest is present once the content was complete enough to hash. This invocation submitted nothing. */
@@ -250,17 +259,18 @@ async function refusable(args, ctx, run) {
 }
 const forks = (spec) => [spec, ...["tasks", "chain"].flatMap(k => Array.isArray(spec[k]) ? spec[k] : [])]
     .some(s => s && typeof s === "object" && s.context === "fork");
-/** R2: `run --request <id> --spec <file|-> [--cwd <dir>] [--json] [--wait-ms <n>]`. The spec is the `subagents` run
+/** `run --request <id> --spec <file|-> [--cwd <dir>] [--json] [--wait-ms <n>]`. The spec is the `subagents` run
  *  form ({agent,task,…} or {tasks|chain:[…],…}); it is validated by the tool's own normalizer and agent check. */
 export const runCommand = (args, ctx) => refusable(args, ctx, runRequest);
 async function runRequest(args, ctx, seen) {
-    const { values, positionals } = flags(args, { request: "value", spec: "value", cwd: "value", json: "flag", "wait-ms": "value" });
+    const { values, positionals } = flags(args, { request: "value", spec: "value", cwd: "value", labels: "value", json: "flag", "wait-ms": "value" });
     const id = text(values, "request"), file = text(values, "spec"), json = values.json === true, wait = waitMs(values, ctx);
     seen.id = id;
     seen.json = json;
     if (!id || !file || positionals.length)
-        throw new Error("usage: run --request <id> --spec <file|-> [--cwd <dir>] [--json] [--wait-ms <n>]");
+        throw new Error("usage: run --request <id> --spec <file|-> [--labels <json>] [--cwd <dir>] [--json] [--wait-ms <n>]");
     requestRid(id);
+    const raw = text(values, "labels"), labels = raw !== undefined ? parseLabels(raw) : undefined;
     const bytes = file === "-" ? await (ctx.stdin ?? stdin)() : readFileSync(resolve(ctx.cwd ?? process.cwd(), file), "utf8");
     let spec;
     try {
@@ -275,11 +285,13 @@ async function runRequest(args, ctx, seen) {
         throw new Error("--spec describes a run; action must be absent or \"run\"");
     if (spec.request !== undefined)
         throw new Error("the request id is --request, not a spec field");
+    if (spec.labels !== undefined)
+        throw new Error("labels are given with --labels <json>, not in the spec");
     if (forks(spec))
         throw new Error("context \"fork\" needs a pi session to fork; it is not available to run --request");
     // RunBody.cwd = spec.cwd ?? --cwd ?? the current directory, absolute before the digest.
     const base = resolve(ctx.cwd ?? process.cwd(), text(values, "cwd") ?? "."), dir = typeof spec.cwd === "string" && spec.cwd ? resolve(base, spec.cwd) : base;
-    const normalized = request({ ...spec, action: "run", ...(typeof spec.cwd === "string" && spec.cwd ? { cwd: dir } : {}) }, dir);
+    const normalized = request({ ...spec, action: "run", ...(typeof spec.cwd === "string" && spec.cwd ? { cwd: dir } : {}), ...(labels ? { labels } : {}) }, dir);
     const body = normalized.body;
     seen.digest = specDigest({ kind: "run", body });
     // An id already recorded is decided by that record: the same content gets its first outcome (the agents it named may
@@ -308,7 +320,7 @@ async function widOf(home, head) {
     const found = await findRequest(home, rid);
     return found?.request.kind === "run" && decision(ledger(home), rid)?.type !== "rejected" ? { pending: true } : { wid: head };
 }
-/** R2: `--to <run-id>[/<key>] | <wid>/<key>` (+ `--call <key>`) → `<wid>/<key>`; a run of one call implies its key. */
+/** `--to <run-id>[/<key>] | <wid>/<key>` (+ `--call <key>`) → `<wid>/<key>`; a run of one call implies its key. */
 async function target(home, to, call, prior) {
     const cut = to.indexOf("/"), head = cut < 0 ? to : to.slice(0, cut), key = cut < 0 ? call : to.slice(cut + 1);
     if (cut >= 0 && call !== undefined)
@@ -329,7 +341,7 @@ async function target(home, to, call, prior) {
         return { to: `${resolved.wid}/${keys[0]}` };
     throw new Error(`${to} has ${keys.length ? `calls ${keys.join(", ")}` : "no calls yet"}; name one with --call <key> or --to <wid>/<key>`);
 }
-/** R2: `send --request <id> --to <…> --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] --message <text|@file> [--model <m>]`. */
+/** `send --request <id> --to <…> --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] --message <text|@file> [--model <m>]`. */
 export const sendCommand = (args, ctx) => refusable(args, ctx, sendRequest);
 async function sendRequest(args, ctx, seen) {
     const { values, positionals } = flags(args, { request: "value", to: "value", call: "value", kind: "value", qid: "value", rev: "value", message: "value", model: "value", json: "flag", "wait-ms": "value" });
@@ -366,7 +378,7 @@ async function sendRequest(args, ctx, seen) {
         return done.code;
     return decided(ctx, id, done, json);
 }
-/** R2: `stop --request <id> <run-id|wid|wid/key|callId>`. */
+/** `stop --request <id> <run-id|wid|wid/key|callId>`. */
 export const stopCommand = (args, ctx) => refusable(args, ctx, stopRequest);
 async function stopRequest(args, ctx, seen) {
     const { values, positionals } = flags(args, { request: "value", json: "flag", "wait-ms": "value" });