pi-durable-subagents 1.0.24 → 1.0.27
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +47 -1
- package/README.md +45 -16
- package/dist/agent/main/tool.js +2 -2
- package/dist/agent/main.js +1 -1
- package/dist/cli/control.js +1 -1
- package/dist/cli/hold.js +17 -6
- package/dist/cli/main.js +3 -3
- package/dist/cli/requests.js +17 -15
- package/dist/events/cli.js +1 -1
- package/dist/events/derive.js +9 -5
- package/dist/events/fence.js +4 -4
- package/dist/events/labels.js +1 -1
- package/dist/events/log.js +2 -2
- package/dist/events/pump.js +4 -4
- package/dist/events/types.js +2 -3
- package/dist/events/{r7.js → waiting.js} +15 -15
- package/dist/kernel/lifecycle.js +1 -1
- package/dist/orchestrator/config.js +3 -3
- package/dist/orchestrator/engine.js +17 -17
- package/dist/orchestrator/executor/session.js +4 -2
- package/dist/orchestrator/restart.js +13 -2
- package/dist/orchestrator/snapshot.js +1 -1
- package/dist/orchestrator/store.js +1 -1
- package/dist/paths.js +1 -1
- package/dist/platform/lease.js +20 -3
- package/dist/requests.js +10 -6
- package/dist/ui/session.js +2 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,51 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.0.27
|
|
4
|
+
|
|
5
|
+
- `hold --no-wait` (or `--max-wait 0`) takes the lease at once or exits 75
|
|
6
|
+
without ever being queued: the decision is made under the resource's lock
|
|
7
|
+
and a refused request writes nothing, so `leases`, `status` and other
|
|
8
|
+
waiters never see it. Before, `--max-wait 0` queued a ticket, checked, and
|
|
9
|
+
removed it. A no-wait request also no longer ends processes left by a
|
|
10
|
+
killed holder (a queued waiter still does); it is refused while they remain.
|
|
11
|
+
- Release check: `pack:smoke` waits for its orchestrator to exit before
|
|
12
|
+
removing its temporary directory, and a failed removal is a warning. The
|
|
13
|
+
1.0.26 release stopped at this step on macOS (`ENOTEMPTY` after
|
|
14
|
+
`pack-smoke ok`), so 1.0.26 was never published; 1.0.27 includes its
|
|
15
|
+
changes.
|
|
16
|
+
|
|
17
|
+
## 1.0.26
|
|
18
|
+
|
|
19
|
+
- A daily quota message in Chinese ("remaining quota is 0, resets at 00:00 the
|
|
20
|
+
next day") is a used-up usage window: a pool call moves to its next candidate
|
|
21
|
+
and a single-model call waits, instead of failing at once as a balance error
|
|
22
|
+
(the word for "balance" occurs inside "remaining quota"). Checked against the 7,941 provider errors
|
|
23
|
+
recorded on a working machine: this was the only usage-window text misread,
|
|
24
|
+
and no rate limit or transient error is read as one.
|
|
25
|
+
- `answered.by` is `call:<wid>/<key>` for a subagent answering through the CLI
|
|
26
|
+
(the CLI now sends its `DSA_CALL` as `caller`). The caller is provenance and
|
|
27
|
+
not part of `spec_digest`: retrying a request id with or without it is the
|
|
28
|
+
same request. A pi session still running an older extension computes the
|
|
29
|
+
digest with the caller included and would see such a retry as a conflict.
|
|
30
|
+
- Tests: an end-to-end pool failover through the CLI and a detached
|
|
31
|
+
orchestrator; the effects fixtures use a unique workflow id, since gate
|
|
32
|
+
processes are found by tag across the whole machine and parallel test files
|
|
33
|
+
shared one; the CI runner also reruns files reported in nested tests.
|
|
34
|
+
|
|
35
|
+
## 1.0.25
|
|
36
|
+
|
|
37
|
+
- `restart` refusals show what a fence would cut short: each lease a running
|
|
38
|
+
call holds, with its mode, how long it has been held, its command and note.
|
|
39
|
+
Leases held outside those executions (a shell, a `systemd-run` unit) are
|
|
40
|
+
listed apart, since a restart leaves them held.
|
|
41
|
+
- The waiting/moving check period is configured as `k.waitCheckMs` (was
|
|
42
|
+
`k.r7Ms` in 1.0.24; the old key is not accepted).
|
|
43
|
+
- README: deduplicate events by `id`, not cursor; a call with no next
|
|
44
|
+
execution has no `fenced`; an open question keeps the orchestrator from its
|
|
45
|
+
idle exit (compact the event log now with a non-force `restart`); leases
|
|
46
|
+
outside calls survive restarts; why a restart cannot hand running
|
|
47
|
+
executions to the new orchestrator.
|
|
48
|
+
|
|
3
49
|
## 1.0.24
|
|
4
50
|
|
|
5
51
|
- `events --all [--since <cursor>] [--limit <n>]`: one durable log of
|
|
@@ -13,7 +59,7 @@
|
|
|
13
59
|
prune whose events cannot be logged first is rejected (`event-log: …`).
|
|
14
60
|
- `run --labels <json>` (tool: `labels`): caller labels, part of the spec
|
|
15
61
|
digest, returned by `describe` and echoed on every event of the run.
|
|
16
|
-
- Why a call does not move
|
|
62
|
+
- Why a call does not move: `describe` adds `reason`, `detail` and
|
|
17
63
|
`since` to each waiting call, and the event log gets `waiting`/`moving`
|
|
18
64
|
when the reason changes: `unconfirmed-stop`, `provider-exhausted`,
|
|
19
65
|
`writer-lock`, `lease`, `slot`, `silent`.
|
package/README.md
CHANGED
|
@@ -55,7 +55,7 @@ the npx cache, so `install-service` refuses to run from there.
|
|
|
55
55
|
| You steer a subagent while it is asking you a question | Your message reaches it, in order. Nothing is rejected or lost. |
|
|
56
56
|
| Two steers arrive out of order and the second replaces the first | Only the second one applies. |
|
|
57
57
|
| A step is refused, or a dependency fails | The workflow stops that branch cleanly. Nothing is retried in vain. |
|
|
58
|
-
| A provider's usage window runs out (`No available accounts`, usage limit, quota exceeded) | Found at the second refusal in a row, while pi is still retrying. A call in a pool continues **in the same session** on the pool's next model (within pi's next retry or two); new calls skip that provider. After 15 minutes the next call that wants it tries it once; when it answers, new calls and new generations use it again. A call with a single model waits for it instead of failing. Billing errors (402, insufficient balance) still fail at once. |
|
|
58
|
+
| A provider's usage window runs out (`No available accounts`, usage limit, quota exceeded, a daily quota at 0) | Found at the second refusal in a row, while pi is still retrying. A call in a pool continues **in the same session** on the pool's next model (within pi's next retry or two); new calls skip that provider. After 15 minutes the next call that wants it tries it once; when it answers, new calls and new generations use it again. A call with a single model waits for it instead of failing. Billing errors (402, insufficient balance) still fail at once. |
|
|
59
59
|
| Two subagents would write in the same worktree | Only one runs there at a time. A call that can write (its tools include `edit` or `write`, which pi's default tools do) holds its git worktree's writer lock from its launch until it ends, also while it waits for an answer. Another writer for that worktree waits in order, and status shows `waiting for writer lock: <root> held by <wid>/<key>`. `writer: false` (a call that does not write there), `isolation: "worktree"` and `"writerLock": "off"` opt out. |
|
|
60
60
|
| A subagent waits for an answer for a long time | It releases its model slot and memory, then resumes exactly once when you answer. The question survives orchestrator restarts (also forced ones) and crashes, including one that hits before the subagent released its slot. |
|
|
61
61
|
| A subagent's work ends (finished, stopped, or cut off) | Every process its tools started ends with that execution, also ones started with `nohup`, `setsid` or `&`: they carry the execution's tag (see the limit below). Anything that must outlive the subagent has to be started by you or the parent session. A command run under `hold` is no exception: a forced restart stops it and its lease is released. |
|
|
@@ -254,7 +254,7 @@ pi-durable-subagents stop-all pause every existing workflow now; journ
|
|
|
254
254
|
pi-durable-subagents prune [wid] [--older-than <days>]
|
|
255
255
|
delete finished workflows (done, failed, stopped); prints count and bytes freed
|
|
256
256
|
pi-durable-subagents restart [--force <token> --reason <text>] switch to the installed version (see "Updating Durable Subagents")
|
|
257
|
-
pi-durable-subagents hold <resource> [--shared] [--max-wait <s>] [--note <text>] -- <command…>
|
|
257
|
+
pi-durable-subagents hold <resource> [--shared] [--max-wait <s> | --no-wait] [--note <text>] -- <command…>
|
|
258
258
|
run one command while holding a resource lease (see below)
|
|
259
259
|
pi-durable-subagents leases [--json] who holds and who waits for each resource
|
|
260
260
|
pi-durable-subagents doctor [--json] read-only health check; exits 1 when something needs you
|
|
@@ -361,14 +361,15 @@ Every event has `id`, `cursor`, `ts` (when the milestone happened), `type`,
|
|
|
361
361
|
|
|
362
362
|
Readers must ignore types they do not know (`waiting`/`moving` follow).
|
|
363
363
|
`by` is the sender of the answer: `session:<id>` for a pi session (with
|
|
364
|
-
`via: "ui"` when it came from the subagent list), `
|
|
365
|
-
|
|
366
|
-
`unknown`. `fenced` is emitted when the call's next execution begins (right
|
|
364
|
+
`via: "ui"` when it came from the subagent list), `call:<wid>/<key>` for a
|
|
365
|
+
subagent answering through the CLI (its `DSA_CALL`; provenance, not
|
|
366
|
+
authority), `cli:<user>@<host>` for any other CLI use, else `unknown`. `fenced` is emitted when the call's next execution begins (right
|
|
367
367
|
after recovery, before it waits for a slot) and only when the fence
|
|
368
368
|
interrupted work, exactly as `describe`'s `lastFence`: a turn that had ended,
|
|
369
369
|
a hibernated question, an answer's resume or a seal are no `fenced`. A `once`
|
|
370
370
|
call cut off in a tool is never resumed: it gets `sealed` with status
|
|
371
|
-
`unknown` and no `fenced`.
|
|
371
|
+
`unknown` and no `fenced`. A call with no next execution has no `fenced`; its
|
|
372
|
+
`sealed` carries the outcome (`describe`'s `lastFence` still names the fence).
|
|
372
373
|
|
|
373
374
|
Cursors are `<epoch>:<seq>`; `--since c` returns the events after `c` in log
|
|
374
375
|
order, at most `--limit` (default and maximum 1000), then
|
|
@@ -384,14 +385,16 @@ exists it never starts anything.
|
|
|
384
385
|
|
|
385
386
|
Delivery is at least once, without gaps: after a crash the orchestrator
|
|
386
387
|
derives again from its last durable watermark, and an event derived again has
|
|
387
|
-
the same `id` (a new cursor). Deduplicate by `id`,
|
|
388
|
-
|
|
388
|
+
the same `id` (a new cursor). Deduplicate by `id`, not by cursor (keeping
|
|
389
|
+
ids for the retention window is enough), and persist your cursor only after
|
|
390
|
+
you applied the events of a page.
|
|
389
391
|
|
|
390
392
|
Retention: an event is dropped only when it was logged more than 7 days ago
|
|
391
|
-
(`"k": { "eventRetentionMs": … }` in
|
|
393
|
+
(`"k": { "eventRetentionMs": … }` in `$DSA_HOME/config.json`) and its workflow is
|
|
392
394
|
finished in its current revision (done, failed or stopped — not parked) with
|
|
393
395
|
no open question and no unsealed call, or was pruned. The log is compacted at
|
|
394
|
-
orchestrator start and at most hourly
|
|
396
|
+
orchestrator start and at most hourly; to compact now while a question is open
|
|
397
|
+
(which keeps the orchestrator from idle exit), run `restart` without `--force`. A cursor of another epoch (the log was
|
|
395
398
|
replaced: a corrupt log is kept aside as `events.jsonl.corrupt-<ms>` and a new
|
|
396
399
|
one starts), below the highest dropped seq, or beyond the head gets exit 4
|
|
397
400
|
and one line `{"error":"cursor-expired","head":"…","oldest":"…"}` (`oldest`
|
|
@@ -419,7 +422,7 @@ When an unsealed call does not move, its `waiting` in `describe` adds
|
|
|
419
422
|
probe 1/1`) and `since` (ms: when that cause started). The event log has the
|
|
420
423
|
same: `waiting {reason, detail, since}` when the reason appears or changes,
|
|
421
424
|
`moving {after}` when it clears (also when the call ends), checked every
|
|
422
|
-
`k.
|
|
425
|
+
`k.waitCheckMs` (default 5 s; read when the orchestrator starts, unlike the other
|
|
423
426
|
`k` settings a `config.json` change does not apply it until a restart); a
|
|
424
427
|
change of detail alone is no event. The first reason
|
|
425
428
|
that applies wins:
|
|
@@ -461,7 +464,7 @@ command:
|
|
|
461
464
|
```sh
|
|
462
465
|
pi-durable-subagents hold machine -- make bench # exclusive
|
|
463
466
|
pi-durable-subagents hold machine --shared -- npm test # with other shared holders, never with an exclusive one
|
|
464
|
-
pi-durable-subagents hold machine --max-wait 600 --note "
|
|
467
|
+
pi-durable-subagents hold machine --max-wait 600 --note "profile" -- ./measure.sh
|
|
465
468
|
```
|
|
466
469
|
|
|
467
470
|
- The lease covers one command, not a whole call: a subagent that thinks
|
|
@@ -470,6 +473,14 @@ pi-durable-subagents hold machine --max-wait 600 --note "frame phase" -- ./measu
|
|
|
470
473
|
everything before it, and keeps later shared requests out (no starvation).
|
|
471
474
|
A waiting `hold` prints who holds the resource; `--max-wait` gives up with
|
|
472
475
|
exit 75 without running the command.
|
|
476
|
+
- `--no-wait` (same as `--max-wait 0`) takes the lease now or not at all.
|
|
477
|
+
It decides under the resource's lock. If it can run now, its request is
|
|
478
|
+
written already granted. Otherwise nothing is written, and it exits 75
|
|
479
|
+
naming who holds or waits, without running the command. It is never
|
|
480
|
+
listed as a waiter, even for a moment, so it can probe a resource whose
|
|
481
|
+
owner treats any queued request as interference. It also leaves the
|
|
482
|
+
resource alone: unlike a queued waiter, it does not end processes left
|
|
483
|
+
by a holder whose `hold` was killed, and is refused while they remain.
|
|
473
484
|
- The command runs without a shell (write `-- sh -c '…'` for one) in its
|
|
474
485
|
own process group; signals to `hold` go to it and its exit status is
|
|
475
486
|
returned. When it exits, whatever it left in its process group is ended
|
|
@@ -482,6 +493,10 @@ pi-durable-subagents hold machine --max-wait 600 --note "frame phase" -- ./measu
|
|
|
482
493
|
ended). State is one small file per request under
|
|
483
494
|
`$DSA_HOME/leases/<resource>/`; no orchestrator is needed, and the user's
|
|
484
495
|
own shell can take part.
|
|
496
|
+
- A lease taken outside any call (your shell, a `systemd-run --user` unit)
|
|
497
|
+
does not depend on the orchestrator: a restart, forced or not, leaves it
|
|
498
|
+
held, and it is released when its `hold` and command end. A lease taken
|
|
499
|
+
inside a call ends with that call's processes when the call is fenced.
|
|
485
500
|
- Subagents find the command on their `PATH` (the orchestrator puts a shim
|
|
486
501
|
in `$DSA_HOME/bin`), and their leases are tagged with their call:
|
|
487
502
|
`status` shows `lease: machine held by <wid>/<key> …; waiting: …` and
|
|
@@ -507,7 +522,8 @@ State lives in `~/.pi/durable-subagents`; set `DSA_HOME` to move it.
|
|
|
507
522
|
}
|
|
508
523
|
```
|
|
509
524
|
|
|
510
|
-
- **Pools:** a model can name a pool
|
|
525
|
+
- **Pools:** a model can name a pool (the `model` of a call, a `run --spec`
|
|
526
|
+
file or a model send). The first candidate with a free slot is
|
|
511
527
|
used, and a candidate that keeps failing is skipped for 10 minutes. The
|
|
512
528
|
order is the preference: list the provider you want to use first.
|
|
513
529
|
- **A used-up provider** is not sent new calls until its next try, 15 minutes
|
|
@@ -565,8 +581,11 @@ On load, it checks the pi exports and API methods it uses.
|
|
|
565
581
|
Running work stays on the version it started with until you restart the
|
|
566
582
|
orchestrator. When the orchestrator runs another version than the one a pi
|
|
567
583
|
session loaded, that pi says so once, and `status` shows the running version
|
|
568
|
-
with a note. The orchestrator exits about 10 s after all
|
|
569
|
-
|
|
584
|
+
with a note. The orchestrator exits about 10 s (`k.idleExitMs`) after all
|
|
585
|
+
work ends: every workflow is finished in its current revision (done, failed,
|
|
586
|
+
stopped or parked) or held by `drain`. A workflow with an open question is not
|
|
587
|
+
finished, so a call hibernated on its question keeps the orchestrator running
|
|
588
|
+
(it holds no slot and costs little). The next start runs the new version. To switch sooner:
|
|
570
589
|
|
|
571
590
|
```sh
|
|
572
591
|
pi-durable-subagents restart # or the subagents tool: action "restart"
|
|
@@ -574,7 +593,9 @@ pi-durable-subagents restart # or the subagents tool: action "restart"
|
|
|
574
593
|
|
|
575
594
|
The orchestrator refuses while any execution runs (a subagent process, or a
|
|
576
595
|
gate before a call's seal). The refusal groups executions by session with ages,
|
|
577
|
-
|
|
596
|
+
the leases each one holds (mode, how long, command, note: what a fence would cut
|
|
597
|
+
short) and a token for that exact set. Leases held outside those executions are
|
|
598
|
+
listed apart, since the restart leaves them held; no new execution starts while
|
|
578
599
|
it decides, so nothing slips in between. Calls waiting
|
|
579
600
|
for your answer (hibernated), waiting for a provider slot, or held by a drain
|
|
580
601
|
do not block it. Otherwise it exits and its successor starts at once from the
|
|
@@ -600,6 +621,14 @@ boundary — a subagent runs as the same OS user and could signal the
|
|
|
600
621
|
orchestrator anyway. Force fences running
|
|
601
622
|
executions; they resume on the new version from their sessions, like after a
|
|
602
623
|
crash, so a tool call that was running is repeated or reported as interrupted.
|
|
624
|
+
A running execution cannot be handed over to the new orchestrator: each
|
|
625
|
+
subagent is a pi process the orchestrator drives over its stdin and stdout, and
|
|
626
|
+
those pipes end with the old process. A crash is no different: the successor
|
|
627
|
+
fences every execution that still runs (an execution that had already ended is
|
|
628
|
+
not counted as interrupted). Work that must survive a forced restart, such as a
|
|
629
|
+
long measurement, belongs outside the subagent's processes (for example
|
|
630
|
+
`systemd-run --user … pi-durable-subagents hold machine -- …`), with the
|
|
631
|
+
subagent only watching it.
|
|
603
632
|
The restart ledger records the reason and initiator; after the next start,
|
|
604
633
|
`status` shows who forced it and why for 24 hours.
|
|
605
634
|
|
package/dist/agent/main/tool.js
CHANGED
|
@@ -21,7 +21,7 @@ function call(value, cwd, where) {
|
|
|
21
21
|
return spec;
|
|
22
22
|
}
|
|
23
23
|
/** v12 §2: Reject unknown explicit call agents before starter or outbox publication; scripts remain call-local. Shared by
|
|
24
|
-
* the tool and the CLI `run --request
|
|
24
|
+
* the tool and the CLI `run --request`. */
|
|
25
25
|
export function checkAgents(body, available) {
|
|
26
26
|
const names = [...(body.call ? [body.call] : []), ...(body.tasks ?? []), ...(body.chain ?? [])].map(call => call.agent);
|
|
27
27
|
if (!names.length)
|
|
@@ -85,7 +85,7 @@ export function request(args, cwd) {
|
|
|
85
85
|
body.args = inputs;
|
|
86
86
|
if (name !== undefined)
|
|
87
87
|
body.name = string(args, "name");
|
|
88
|
-
//
|
|
88
|
+
// Part of the spec digest; an empty object is the same as none.
|
|
89
89
|
if (labels !== undefined && Object.keys(checkLabels(labels)).length)
|
|
90
90
|
body.labels = labels;
|
|
91
91
|
// P31a, P36, P11: workflow-level limits and declared input files (absolute paths, pinned at admission).
|
package/dist/agent/main.js
CHANGED
|
@@ -297,7 +297,7 @@ export function registerMain(pi, ui) {
|
|
|
297
297
|
if (args.action === "resume" && args.wid === undefined)
|
|
298
298
|
args = { ...args, origin: sender };
|
|
299
299
|
const normalized = request(args, cwd);
|
|
300
|
-
//
|
|
300
|
+
// A caller-chosen request id names a run, send or stop; a retry with the same content gets the first outcome.
|
|
301
301
|
if (args.request !== undefined && (typeof args.request !== "string" || !["run", "send", "stop"].includes(normalized.kind) || normalized.replaces?.length))
|
|
302
302
|
throw new Error(REQUEST_USE);
|
|
303
303
|
const rid = typeof args.request === "string" ? requestRid(args.request) : undefined;
|
package/dist/cli/control.js
CHANGED
|
@@ -104,7 +104,7 @@ export async function submit(home, command, target, env = process.env, options =
|
|
|
104
104
|
return requests;
|
|
105
105
|
});
|
|
106
106
|
}
|
|
107
|
-
/**
|
|
107
|
+
/** Submit a request named by a caller-chosen id through the CLI sender: a retry with the same content republishes
|
|
108
108
|
* (or reuses) the recorded envelope, other content is a conflict and publishes nothing. Starts the orchestrator
|
|
109
109
|
* unless the request conflicts. */
|
|
110
110
|
export async function submitIdentified(home, rid, kind, body, cond, env = process.env, starter = startOrchestrator) {
|
package/dist/cli/hold.js
CHANGED
|
@@ -1,13 +1,14 @@
|
|
|
1
1
|
// `pi-durable-subagents hold <resource> [--shared] [--max-wait <s>] [--note <text>] -- <command> [args…]`
|
|
2
2
|
// Waits for the lease (strict FIFO), runs the command in its own process group, ends what the command left in that
|
|
3
|
-
// group, then releases.
|
|
3
|
+
// group, then releases. `--max-wait 0` (or `--no-wait`) takes the lease at once or exits 75 without ever being queued.
|
|
4
|
+
// See src/platform/lease.ts for the ticket protocol.
|
|
4
5
|
import { spawn } from "node:child_process";
|
|
5
6
|
import { watch } from "node:fs";
|
|
6
7
|
import { constants } from "node:os";
|
|
7
8
|
import { dsaHome } from "../paths.js";
|
|
8
9
|
import { captureStart } from "../platform/proctable.js";
|
|
9
|
-
import { blockers, enqueue, groupAlive, leaseDir, liveTickets, orphaned, removeTicket, RESOURCE, who, writeTicket } from "../platform/lease.js";
|
|
10
|
-
export const HOLD_USAGE = "usage: pi-durable-subagents hold <resource> [--shared] [--max-wait <seconds>] [--note <text>] -- <command> [args…]";
|
|
10
|
+
import { blockers, enqueue, groupAlive, tryGrant, leaseDir, liveTickets, orphaned, removeTicket, RESOURCE, who, writeTicket } from "../platform/lease.js";
|
|
11
|
+
export const HOLD_USAGE = "usage: pi-durable-subagents hold <resource> [--shared] [--max-wait <seconds> | --no-wait] [--note <text>] -- <command> [args…]";
|
|
11
12
|
/** Exit status when --max-wait expires before the lease is granted (EX_TEMPFAIL). */
|
|
12
13
|
export const WAIT_EXPIRED = 75;
|
|
13
14
|
export function parseHold(args) {
|
|
@@ -20,6 +21,8 @@ export function parseHold(args) {
|
|
|
20
21
|
const a = opts[i];
|
|
21
22
|
if (a === "--shared")
|
|
22
23
|
mode = "shared";
|
|
24
|
+
else if (a === "--no-wait")
|
|
25
|
+
maxWaitMs = 0;
|
|
23
26
|
else if (a === "--max-wait" || a === "--note") {
|
|
24
27
|
const v = opts[++i];
|
|
25
28
|
if (v === undefined)
|
|
@@ -54,13 +57,21 @@ const SIGNALS = ["SIGINT", "SIGTERM", "SIGHUP"];
|
|
|
54
57
|
export async function hold(args, options = {}) {
|
|
55
58
|
const env = options.env ?? process.env, home = dsaHome(env), say = options.stderr ?? (l => process.stderr.write(`${l}\n`));
|
|
56
59
|
const pollMs = options.pollMs ?? 250, graceMs = options.graceMs ?? 2000;
|
|
57
|
-
const
|
|
60
|
+
const request = {
|
|
58
61
|
resource: args.resource, mode: args.mode,
|
|
59
62
|
wrapper: { pid: process.pid, start: (await captureStart(process.pid)) || undefined },
|
|
60
63
|
argv: args.argv.map(a => a.length > 200 ? `${a.slice(0, 199)}…` : a).slice(0, 32), cwd: process.cwd(),
|
|
61
64
|
...(args.note ? { note: args.note.slice(0, 300) } : {}), since: Date.now(),
|
|
62
65
|
...(env.DSA_EXEC ? { exec: env.DSA_EXEC } : {}), ...(env.DSA_CALL ? { call: env.DSA_CALL } : {}),
|
|
63
|
-
}
|
|
66
|
+
};
|
|
67
|
+
// No wait: granted now, or refused without a ticket — never listed as a waiter, even for a moment.
|
|
68
|
+
const granted = args.maxWaitMs === 0 ? await tryGrant(home, request) : undefined;
|
|
69
|
+
if (granted && "busy" in granted) {
|
|
70
|
+
const now = Date.now(), by = granted.busy.map(t => `${who(t)} (${t.mode}${t.grantedAt === undefined ? ", waiting" : ""}, ${age(now - (t.grantedAt ?? t.since))})`).join(", ");
|
|
71
|
+
say(`hold: ${args.resource} is not free now (${by}); not running the command (exit ${WAIT_EXPIRED})`);
|
|
72
|
+
return WAIT_EXPIRED;
|
|
73
|
+
}
|
|
74
|
+
const ticket = granted ?? await enqueue(home, request);
|
|
64
75
|
// Signals: while waiting they withdraw the request; while the command runs they go to its process group.
|
|
65
76
|
let child, interrupted, wake = () => { };
|
|
66
77
|
const onSignal = (signal) => {
|
|
@@ -125,7 +136,7 @@ export async function hold(args, options = {}) {
|
|
|
125
136
|
}
|
|
126
137
|
watcher?.close();
|
|
127
138
|
watcher = undefined;
|
|
128
|
-
ticket.grantedAt
|
|
139
|
+
ticket.grantedAt ??= Date.now();
|
|
129
140
|
await writeTicket(home, ticket);
|
|
130
141
|
if (interrupted) {
|
|
131
142
|
removeTicket(home, ticket);
|
package/dist/cli/main.js
CHANGED
|
@@ -132,7 +132,7 @@ export function serviceEntryError(entry) {
|
|
|
132
132
|
return `install-service refuses to run from an npx cache (${entry}); the cache can be pruned and the service would break. Install the CLI with \`npm i -g pi-durable-subagents\` and run \`pi-durable-subagents install-service\` again.`;
|
|
133
133
|
return undefined;
|
|
134
134
|
}
|
|
135
|
-
export const HELP = "pi-durable-subagents: smoke | status [wid] [--json] | events <wid> [--json] | events --all [--since <cursor>] [--limit <n>] [--json] [--wait-ms <n>] | tail [wid] [--json] | start | resume [wid] | drain | stop <wid|callId> | stop-all | run --request <id> --spec <file|-> [--labels <json>] [--cwd <dir>] [--json] [--wait-ms <n>] | send --request <id> --to <run-id|wid/key> [--call <key>] --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] [--message <text|@file>] [--model <m>] [--json] [--wait-ms <n>] | stop --request <id> <run-id|wid|wid/key> [--json] [--wait-ms <n>] | describe --key <id> | describe <wid> [--json] | prune [wid] [--older-than <days>] | restart [--force <token> --reason <text>] | hold <resource> [--shared] [--max-wait <s>] [--note <text>] -- <command…> | leases [--json] | doctor [--json] | install-service [--dry-run] | uninstall-service [--dry-run] | chaos [--scenario <1-9>] [--keep] [--json]";
|
|
135
|
+
export const HELP = "pi-durable-subagents: smoke | status [wid] [--json] | events <wid> [--json] | events --all [--since <cursor>] [--limit <n>] [--json] [--wait-ms <n>] | tail [wid] [--json] | start | resume [wid] | drain | stop <wid|callId> | stop-all | run --request <id> --spec <file|-> [--labels <json>] [--cwd <dir>] [--json] [--wait-ms <n>] | send --request <id> --to <run-id|wid/key> [--call <key>] --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] [--message <text|@file>] [--model <m>] [--json] [--wait-ms <n>] | stop --request <id> <run-id|wid|wid/key> [--json] [--wait-ms <n>] | describe --key <id> | describe <wid> [--json] | prune [wid] [--older-than <days>] | restart [--force <token> --reason <text>] | hold <resource> [--shared] [--max-wait <s> | --no-wait] [--note <text>] -- <command…> | leases [--json] | doctor [--json] | install-service [--dry-run] | uninstall-service [--dry-run] | chaos [--scenario <1-9>] [--keep] [--json]";
|
|
136
136
|
/** Restart: the orchestrator exits when no execution runs (or `force`) and the installed version takes over. */
|
|
137
137
|
async function restartCommand(home, env, write, options) {
|
|
138
138
|
const body = { ...(typeof options.force === "string" ? { token: options.force } : options.force === true ? { force: true } : {}), ...(options.reason !== undefined ? { reason: options.reason } : {}), initiator: cliInitiator(env) };
|
|
@@ -194,12 +194,12 @@ export async function main(args = process.argv.slice(2), options = {}) {
|
|
|
194
194
|
}
|
|
195
195
|
if (args[0] === "chaos")
|
|
196
196
|
return (await import("./chaos/index.js")).chaos(args.slice(1), options.env ?? process.env, options.write);
|
|
197
|
-
//
|
|
197
|
+
// The cross-workflow event log (strict flags of its own); `events <wid>` stays below.
|
|
198
198
|
if (args[0] === "events" && args.includes("--all")) {
|
|
199
199
|
const env = options.env ?? process.env;
|
|
200
200
|
return (await import("../events/cli.js")).eventsAll(args.slice(1), { home: dsaHome(env), env, write: options.write ?? ((line) => console.log(line)), starter: options.starter ?? startOrchestrator, waitMs: options.waitMs });
|
|
201
201
|
}
|
|
202
|
-
//
|
|
202
|
+
// Program-facing commands named by request ids (strict flags of their own).
|
|
203
203
|
if (["run", "send", "describe"].includes(args[0]) || (args[0] === "stop" && args.includes("--request"))) {
|
|
204
204
|
const env = options.env ?? process.env, requests = await import("./requests.js");
|
|
205
205
|
const ctx = { home: dsaHome(env), env, write: options.write ?? ((line) => console.log(line)), starter: options.starter, waitMs: options.waitMs, cwd: options.cwd, stdin: options.stdin };
|
package/dist/cli/requests.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Program-facing commands named by caller-chosen request ids — `run|send|stop --request <id>` and `describe`.
|
|
2
2
|
// Exit codes: 0 decided (applied/created), 1 rejected or invalid, 3 request-conflict (the id names other content),
|
|
3
3
|
// 75 not decided within --wait-ms (retry with the same id and content: safe).
|
|
4
4
|
import { existsSync, readFileSync } from "node:fs";
|
|
@@ -16,7 +16,7 @@ import { JT } from "../types.js";
|
|
|
16
16
|
import { startOrchestrator, submitIdentified } from "./control.js";
|
|
17
17
|
import { endedExecs, fenceReason } from "../events/fence.js";
|
|
18
18
|
import { parseLabels } from "../events/labels.js";
|
|
19
|
-
import { foldWaits, leaseWaits, waitsOf } from "../events/
|
|
19
|
+
import { foldWaits, leaseWaits, waitsOf } from "../events/waiting.js";
|
|
20
20
|
import { emptyLedger, foldLedger } from "../orchestrator/ledger.js";
|
|
21
21
|
export const EXIT = { ok: 0, rejected: 1, conflict: 3, pending: 75 };
|
|
22
22
|
/** Strict `--name value` / `--flag` parsing; unknown or repeated options are errors. */
|
|
@@ -57,7 +57,7 @@ function decision(entries, rid) {
|
|
|
57
57
|
const createdBy = (entries, rid) => entries.find(e => e.type === JT.created && e.rid === rid);
|
|
58
58
|
async function stdin() { const chunks = []; for await (const chunk of process.stdin)
|
|
59
59
|
chunks.push(chunk); return Buffer.concat(chunks).toString("utf8"); }
|
|
60
|
-
/**
|
|
60
|
+
/** The state of a request id (`{request}`) or a workflow (`{wid}`), with full texts (no clipping). */
|
|
61
61
|
export async function describe(home, key, now = Date.now()) {
|
|
62
62
|
const entries = ledger(home);
|
|
63
63
|
if ("wid" in key)
|
|
@@ -76,7 +76,7 @@ export async function describe(home, key, now = Date.now()) {
|
|
|
76
76
|
return { state: "applied", ...head };
|
|
77
77
|
return { state: "pending", ...head };
|
|
78
78
|
}
|
|
79
|
-
/**
|
|
79
|
+
/** `{labels}` of a run request that has any. */
|
|
80
80
|
function labelsOf(request) {
|
|
81
81
|
const labels = request?.kind === "run" ? request.body?.labels : undefined;
|
|
82
82
|
return labels && typeof labels === "object" && Object.keys(labels).length ? { labels } : {};
|
|
@@ -86,7 +86,7 @@ function runOf(entries, wid) {
|
|
|
86
86
|
const created = entries.find(e => e.type === JT.created && e.wid === wid);
|
|
87
87
|
return { created, run: created ? entries.find(e => e.type === "request" && e.request.rid === created.rid)?.request : undefined };
|
|
88
88
|
}
|
|
89
|
-
/** The pinned agents of a workflow revision, read once on demand (
|
|
89
|
+
/** The pinned agents of a workflow revision, read once on demand (the waiting check needs the model of a call that names none). */
|
|
90
90
|
function pinnedAgentModel(home, wid, rev) {
|
|
91
91
|
let agents;
|
|
92
92
|
return name => {
|
|
@@ -108,7 +108,7 @@ function describeWorkflow(home, wid, entries, now) {
|
|
|
108
108
|
...(typeof pruned.request === "string" ? { request: pruned.request } : {}), ...(typeof pruned.spec_digest === "string" ? { spec_digest: pruned.spec_digest } : {}), ...labelsOf(runOf(entries, wid).run) };
|
|
109
109
|
if (!/^[^/\\\0]+$/.test(wid) || wid === "." || wid === ".." || !existsSync(journalPath(home, wid)))
|
|
110
110
|
return { state: "absent", wid };
|
|
111
|
-
// The snapshot and the journal the
|
|
111
|
+
// The snapshot and the journal the wait fold reads must be the same bytes (a writer-wait appended between the two reads
|
|
112
112
|
// would give a reason without its writerWait): read again until the journal did not move around the snapshot.
|
|
113
113
|
let journal = readJournalSnapshot(journalPath(home, wid)), wf = workflowSnapshot(home, wid);
|
|
114
114
|
for (let i = 0, again = readJournalSnapshot(journalPath(home, wid)); again !== journal && i < 5; i++, again = readJournalSnapshot(journalPath(home, wid))) {
|
|
@@ -118,7 +118,7 @@ function describeWorkflow(home, wid, entries, now) {
|
|
|
118
118
|
const { created, run } = runOf(entries, wid), id = created ? requestId(String(created.rid)) : undefined;
|
|
119
119
|
const admitted = id ? run : undefined;
|
|
120
120
|
const lstate = leaseState(home), slots = slotsView(home, now), leases = leaseCalls(lstate, now);
|
|
121
|
-
//
|
|
121
|
+
// The same fold and decision the orchestrator's collector uses, from the disk snapshots.
|
|
122
122
|
const waits = waitsOf(foldWaits(wid, journal), { now, ledger: foldLedger(emptyLedger(), entries), leases: leaseWaits(lstate, now) }, pinnedAgentModel(home, wid, wf.rev));
|
|
123
123
|
const line = (lines, model) => { const provider = model?.split("/")[0]; return provider ? lines?.find(l => l.startsWith(`${provider} `)) : undefined; };
|
|
124
124
|
const latest = [...new Map(wf.calls.map(c => [c.key, c])).values()];
|
|
@@ -140,7 +140,7 @@ function describeWorkflow(home, wid, entries, now) {
|
|
|
140
140
|
return { state, wid, ...(id ? { request: id } : {}), ...(admitted ? { spec_digest: specDigest(admitted) } : {}), ...labelsOf(run), status: wf.status, ...(wf.error ? { error: wf.error } : {}),
|
|
141
141
|
calls, ...(questions.length ? { questions } : {}), ...(attention.length ? { attention } : {}), ...(fence ? { lastFence: fence } : {}) };
|
|
142
142
|
}
|
|
143
|
-
/**
|
|
143
|
+
/** Best effort: why the latest fence that interrupted work happened. The per-execution classification and the
|
|
144
144
|
* reason are shared with the event log's `fenced` events (src/events/fence.ts). */
|
|
145
145
|
export function lastFence(journal, orch) {
|
|
146
146
|
const ended = endedExecs(journal);
|
|
@@ -210,7 +210,7 @@ function pending(ctx, id, json, why = "") {
|
|
|
210
210
|
}
|
|
211
211
|
/** Submit, then wait; a decision whose admitted envelope has other content (another sender won the id) is a conflict. */
|
|
212
212
|
async function submitAndWait(ctx, seen, id, kind, body, cond, wait, json) {
|
|
213
|
-
//
|
|
213
|
+
// `created` is false only when the id was already decided before this invocation submitted (decisions are
|
|
214
214
|
// monotonic); racing first attempts may all report created — the wid is what identifies the run.
|
|
215
215
|
const rid = requestRid(id), earlier = Boolean(await outcome(ctx.home, rid, kind === "run", 0));
|
|
216
216
|
let sent;
|
|
@@ -236,7 +236,7 @@ async function submitAndWait(ctx, seen, id, kind, body, cond, wait, json) {
|
|
|
236
236
|
return { code: await conflict(ctx, id, specDigest(admitted), json) };
|
|
237
237
|
return { sent, outcome: result, earlier };
|
|
238
238
|
}
|
|
239
|
-
/**
|
|
239
|
+
/** A failure once this invocation submitted, or once the id is found recorded with this content, is "not decided
|
|
240
240
|
* yet" (75), never a refusal. Otherwise, with --json, a request refused before submission (a usage error, an invalid
|
|
241
241
|
* spec, an unknown agent, no open question) answers `{request, applied:false, reason, spec_digest?}` with exit 1;
|
|
242
242
|
* spec_digest is present once the content was complete enough to hash. This invocation submitted nothing. */
|
|
@@ -259,7 +259,7 @@ async function refusable(args, ctx, run) {
|
|
|
259
259
|
}
|
|
260
260
|
const forks = (spec) => [spec, ...["tasks", "chain"].flatMap(k => Array.isArray(spec[k]) ? spec[k] : [])]
|
|
261
261
|
.some(s => s && typeof s === "object" && s.context === "fork");
|
|
262
|
-
/**
|
|
262
|
+
/** `run --request <id> --spec <file|-> [--cwd <dir>] [--json] [--wait-ms <n>]`. The spec is the `subagents` run
|
|
263
263
|
* form ({agent,task,…} or {tasks|chain:[…],…}); it is validated by the tool's own normalizer and agent check. */
|
|
264
264
|
export const runCommand = (args, ctx) => refusable(args, ctx, runRequest);
|
|
265
265
|
async function runRequest(args, ctx, seen) {
|
|
@@ -320,7 +320,7 @@ async function widOf(home, head) {
|
|
|
320
320
|
const found = await findRequest(home, rid);
|
|
321
321
|
return found?.request.kind === "run" && decision(ledger(home), rid)?.type !== "rejected" ? { pending: true } : { wid: head };
|
|
322
322
|
}
|
|
323
|
-
/**
|
|
323
|
+
/** `--to <run-id>[/<key>] | <wid>/<key>` (+ `--call <key>`) → `<wid>/<key>`; a run of one call implies its key. */
|
|
324
324
|
async function target(home, to, call, prior) {
|
|
325
325
|
const cut = to.indexOf("/"), head = cut < 0 ? to : to.slice(0, cut), key = cut < 0 ? call : to.slice(cut + 1);
|
|
326
326
|
if (cut >= 0 && call !== undefined)
|
|
@@ -341,7 +341,7 @@ async function target(home, to, call, prior) {
|
|
|
341
341
|
return { to: `${resolved.wid}/${keys[0]}` };
|
|
342
342
|
throw new Error(`${to} has ${keys.length ? `calls ${keys.join(", ")}` : "no calls yet"}; name one with --call <key> or --to <wid>/<key>`);
|
|
343
343
|
}
|
|
344
|
-
/**
|
|
344
|
+
/** `send --request <id> --to <…> --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] --message <text|@file> [--model <m>]`. */
|
|
345
345
|
export const sendCommand = (args, ctx) => refusable(args, ctx, sendRequest);
|
|
346
346
|
async function sendRequest(args, ctx, seen) {
|
|
347
347
|
const { values, positionals } = flags(args, { request: "value", to: "value", call: "value", kind: "value", qid: "value", rev: "value", message: "value", model: "value", json: "flag", "wait-ms": "value" });
|
|
@@ -373,12 +373,14 @@ async function sendRequest(args, ctx, seen) {
|
|
|
373
373
|
const normalized = request({ action: "send", to: where.to, kind, ...(message !== undefined ? { message } : {}), ...(text(values, "model") !== undefined ? { model: text(values, "model") } : {}),
|
|
374
374
|
...(qid !== undefined ? { qid } : {}), ...(revision !== undefined ? { rev: revision } : {}) }, ctx.cwd ?? process.cwd());
|
|
375
375
|
seen.digest = specDigest({ kind: "send", body: normalized.body, cond: normalized.cond });
|
|
376
|
-
|
|
376
|
+
// Inside a subagent the CLI names its call (provenance for `answered.by`; spec_digest ignores it).
|
|
377
|
+
const body = ctx.env.DSA_CALL ? { ...normalized.body, caller: ctx.env.DSA_CALL } : normalized.body;
|
|
378
|
+
const done = await submitAndWait(ctx, seen, id, "send", body, normalized.cond, wait, json);
|
|
377
379
|
if ("code" in done)
|
|
378
380
|
return done.code;
|
|
379
381
|
return decided(ctx, id, done, json);
|
|
380
382
|
}
|
|
381
|
-
/**
|
|
383
|
+
/** `stop --request <id> <run-id|wid|wid/key|callId>`. */
|
|
382
384
|
export const stopCommand = (args, ctx) => refusable(args, ctx, stopRequest);
|
|
383
385
|
async function stopRequest(args, ctx, seen) {
|
|
384
386
|
const { values, positionals } = flags(args, { request: "value", json: "flag", "wait-ms": "value" });
|
package/dist/events/cli.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// `events --all [--since <cursor>] [--limit <n>] [--json] [--wait-ms <n>]` — read the cross-workflow event log.
|
|
2
2
|
// Output is JSON lines (with or without --json). Without --since: `{"head","more":false}`. With --since: the events
|
|
3
3
|
// after the cursor (at most --limit, default and max EVENTS_PAGE_MAX), then `{"head","more"}`: more:true → head is the
|
|
4
4
|
// cursor of the last event printed; more:false → the log head. Exit 0; 4 cursor-expired (other epoch, seq below
|
package/dist/events/derive.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Journal → event drafts. Pure functions of durable sources: the entry at journal index i (seq i+1) is derived from
|
|
2
2
|
// that journal's entries up to it and from orchestrator-ledger entries durable before it, so a re-derivation after a
|
|
3
3
|
// crash gives the same events with the same ids (`<wid>:<journal seq>:<type>`, `<wid>:submitted`).
|
|
4
4
|
import { createHash } from "node:crypto";
|
|
@@ -14,7 +14,7 @@ export function callParts(call) {
|
|
|
14
14
|
const echo = (id) => ({ ...(id.request !== undefined ? { request: id.request } : {}), ...(id.labels ? { labels: id.labels } : {}) });
|
|
15
15
|
const onCall = (call) => { const p = callParts(call); return p ? { key: p.key, gen: p.gen, call } : { call }; };
|
|
16
16
|
/** The labels of a run body: a non-empty plain object of strings (anything else, and `{}`, is not echoed — as describe
|
|
17
|
-
* and the
|
|
17
|
+
* and the wait collector treat them). */
|
|
18
18
|
export function labelsOf(body) {
|
|
19
19
|
const labels = body?.labels;
|
|
20
20
|
if (!labels || typeof labels !== "object" || Array.isArray(labels))
|
|
@@ -23,11 +23,15 @@ export function labelsOf(body) {
|
|
|
23
23
|
return entries.length && entries.every(([, v]) => typeof v === "string") ? Object.fromEntries(entries) : undefined;
|
|
24
24
|
}
|
|
25
25
|
/** `answered.by` from the answer request's sender: a pi session `main:<id>` → `session:<id>` (+ via "ui" when the
|
|
26
|
-
* subagent list sent it, SendBody.by "user"); the CLI
|
|
26
|
+
* subagent list sent it, SendBody.by "user"); the CLI run inside a subagent (SendBody.caller `<wid>@<rev>/<key>@<gen>`)
|
|
27
|
+
* → `call:<wid>/<key>`; any other CLI sender `cli:<user>@<host>` as is; anything else `unknown`. */
|
|
27
28
|
export function answeredBy(req) {
|
|
28
|
-
const from = req?.from ?? "";
|
|
29
|
+
const from = req?.from ?? "", body = req?.body;
|
|
29
30
|
if (from.startsWith("main:"))
|
|
30
|
-
return { by: `session:${from.slice(5)}`, ...(
|
|
31
|
+
return { by: `session:${from.slice(5)}`, ...(body?.by === "user" ? { via: "ui" } : {}) };
|
|
32
|
+
const caller = typeof body?.caller === "string" ? /^([^/@]+)@\d+\/(.+)@\d+$/.exec(body.caller) : null;
|
|
33
|
+
if (from.startsWith("cli:") && caller)
|
|
34
|
+
return { by: `call:${caller[1]}/${caller[2]}` };
|
|
31
35
|
if (/^cli:[^@]+@.+$/.test(from))
|
|
32
36
|
return { by: from };
|
|
33
37
|
return { by: "unknown" };
|
package/dist/events/fence.js
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Which fences interrupted work, shared by `describe` (lastFence) and the event deriver (`fenced`). Read-only over
|
|
2
2
|
// committed journal entries; `limit` restricts the view to the first `limit` entries (the deriver may read only entries
|
|
3
3
|
// before the one it derives, so a re-derivation after a crash sees the same history).
|
|
4
4
|
import { JT } from "../types.js";
|
|
5
|
-
/**
|
|
5
|
+
/** Executions whose fence did NOT interrupt work: every execution ends with a fence; one interrupted work only when
|
|
6
6
|
* the execution neither settled (its turn ended) before it nor hibernated (it waits for an answer), and was not sealed
|
|
7
7
|
* on purpose: a seal ends an execution on purpose unless its outcome is `unknown` (a `once` call cut off in a tool) or
|
|
8
8
|
* the execution was recorded as lost (the loss bound sealed it), which are interruptions themselves. A seal for an
|
|
@@ -31,7 +31,7 @@ export function endedExecs(journal, limit = journal.length) {
|
|
|
31
31
|
}
|
|
32
32
|
return ended;
|
|
33
33
|
}
|
|
34
|
-
/**
|
|
34
|
+
/** Best effort: why `fence` (a `fenced` entry that interrupted work) happened. restart-force: a forced restart listed
|
|
35
35
|
* the execution as live; orchestrator-crash: the execution was launched before an orchestrator start that is not
|
|
36
36
|
* preceded by a clean exit and fenced after it (startup recovery); otherwise process-died (the child or its host went
|
|
37
37
|
* away, or a drain fenced it). */
|
|
@@ -49,7 +49,7 @@ export function fenceReason(journal, orch, fence) {
|
|
|
49
49
|
}
|
|
50
50
|
return "process-died";
|
|
51
51
|
}
|
|
52
|
-
/**
|
|
52
|
+
/** The fence of `exec` among the first `limit` entries when it interrupted work, else undefined. */
|
|
53
53
|
export function interruptingFence(journal, exec, limit = journal.length) {
|
|
54
54
|
let fence;
|
|
55
55
|
for (let i = limit - 1; i >= 0 && !fence; i--) {
|
package/dist/events/labels.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Caller labels of a run — `run --labels <json>`, the tool's `labels`, and the
|
|
2
2
|
// orchestrator's admission all validate with this one function. Labels are part of RunBody, so they are part of the
|
|
3
3
|
// request's spec_digest (the same request id with other labels is a request-conflict) and are echoed by `describe` and
|
|
4
4
|
// on every event of the workflow.
|
package/dist/events/log.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// The cross-workflow event log, `<home>/events.jsonl`. Single writer: the orchestrator (under its OS lock). Each line
|
|
2
2
|
// is CRC-framed like the journals (`<crc32 hex 8> <json>\n`, kernel/journal.ts); a reader ignores a torn final line, a
|
|
3
3
|
// bad line before the last one is corruption. Records (`k`):
|
|
4
4
|
// log {v, epoch, dropped} first line, written when the file is created or compacted. `epoch` (16 hex chars, random)
|
|
@@ -120,7 +120,7 @@ function scanFile(fd, size) {
|
|
|
120
120
|
throw new LogCorrupt("event log has no header");
|
|
121
121
|
return { epoch: header.epoch, dropped: header.dropped, head: Math.max(head, header.dropped), marks, end: bad ?? end, records, events };
|
|
122
122
|
}
|
|
123
|
-
/**
|
|
123
|
+
/** The writer. Appends, compaction and close are serialized; an append resolves only after fsync. */
|
|
124
124
|
export class EventLog {
|
|
125
125
|
path;
|
|
126
126
|
epoch;
|
package/dist/events/pump.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
|
-
//
|
|
1
|
+
// The orchestrator's event pump. Derives events from the durable sources (workflow journals, the orchestrator
|
|
2
2
|
// ledger's `created`) after their appends and writes them to the event log with the watermarks of the sources derived
|
|
3
3
|
// (same fsync), so a restart re-derives only after them (at least once). It is also the EventSink other producers
|
|
4
|
-
// (
|
|
4
|
+
// (waiting/moving) emit through. One chain serializes passes, emits, compaction and close. A pass that cannot log
|
|
5
5
|
// what it derived retries in the background, and `flush` (prune) rejects; a broken log (a failure after a compaction's
|
|
6
6
|
// rename) is reopened by the next pass or emit.
|
|
7
7
|
// Memory: per unpruned workflow one watermark and one cached identity; reads the journals' committed arrays from their
|
|
@@ -123,7 +123,7 @@ export class EventPump {
|
|
|
123
123
|
async emit(drafts) {
|
|
124
124
|
await this.ready;
|
|
125
125
|
return this.serial(async () => {
|
|
126
|
-
// After close (the orchestrator exits) nothing is logged:
|
|
126
|
+
// After close (the orchestrator exits) nothing is logged: waiting state is derived again by the next orchestrator.
|
|
127
127
|
if (!this.log || this.closed || !drafts.length)
|
|
128
128
|
return;
|
|
129
129
|
const log = await this.writer();
|
|
@@ -209,7 +209,7 @@ export class EventPump {
|
|
|
209
209
|
}
|
|
210
210
|
/** The writer; a broken one (a failure after a compaction's rename) is reopened first: the open applies the start
|
|
211
211
|
* skip. A log it had to create anew (corrupt or gone) is backfilled from everything on disk: the caller's drafts
|
|
212
|
-
* were derived for the old one, so this throws (a pass retries from the cleared watermarks,
|
|
212
|
+
* were derived for the old one, so this throws (a pass retries from the cleared watermarks, waiting/moving on its next tick). */
|
|
213
213
|
async writer() {
|
|
214
214
|
const log = this.log;
|
|
215
215
|
if (!log.broken)
|
package/dist/events/types.js
CHANGED
|
@@ -1,10 +1,9 @@
|
|
|
1
|
-
//
|
|
2
|
-
// batch 2; both leaves build on it. Changing a field here changes the public contract: do it only with the parent.
|
|
1
|
+
// The public shape of the cross-workflow event log. Changing a field here changes the public contract.
|
|
3
2
|
//
|
|
4
3
|
// One durable sequence, single writer (the orchestrator). Each record is one event; `seq` grows strictly (gaps allowed:
|
|
5
4
|
// retention drops records and every orchestrator start skips ahead, see EVENT_SEQ_SKIP). The public cursor is
|
|
6
5
|
// `<epoch>:<seq>`; `--since c` returns events with a larger seq of the same epoch.
|
|
7
|
-
/**
|
|
6
|
+
/** Wait reasons, one per waiting call, in this precedence when several apply (first wins). */
|
|
8
7
|
export const WAIT_REASONS = ["unconfirmed-stop", "provider-exhausted", "writer-lock", "lease", "slot", "silent"];
|
|
9
8
|
/** `data` of a `sealed` event is inlined up to this many bytes of JSON; larger → `data_omitted`. */
|
|
10
9
|
export const EVENT_DATA_INLINE_MAX = 16 * 1024;
|
|
@@ -1,7 +1,7 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Why a call is not moving. One pure decision (`whyWaiting`) from one
|
|
2
2
|
// unsealed call's observable state; `describe` (CLI, from disk snapshots) and the orchestrator's collector (from its
|
|
3
3
|
// in-memory journals) both build that state with the same fold (`foldWaits`) and the same inputs (`waitsOf`), so they
|
|
4
|
-
// agree. The tracker turns the per-tick waits into `waiting`/`moving` event drafts for the log's EventSink; `
|
|
4
|
+
// agree. The tracker turns the per-tick waits into `waiting`/`moving` event drafts for the log's EventSink; `startWaiting`
|
|
5
5
|
// runs it on a timer. No durable state of its own: the log holds what was emitted, and seeds the tracker after a start.
|
|
6
6
|
//
|
|
7
7
|
// Sources, per reason (precedence WAIT_REASONS, first wins):
|
|
@@ -31,7 +31,7 @@ import { leaseCalls, leaseState } from "../platform/lease.js";
|
|
|
31
31
|
import { requestId } from "../requests.js";
|
|
32
32
|
/** A queued call whose providers have a free slot (or are unknown) counts as waiting for a slot only after this long. */
|
|
33
33
|
export const SLOT_GRACE_MS = 3000;
|
|
34
|
-
/**
|
|
34
|
+
/** Why the call does not move, or undefined when it moves (or is asking or sealed). Pure. */
|
|
35
35
|
export function whyWaiting(c, now) {
|
|
36
36
|
if (c.sealed || c.asking)
|
|
37
37
|
return undefined;
|
|
@@ -246,7 +246,7 @@ export function waitInput(f, c, env, agentModel) {
|
|
|
246
246
|
export function liveWaitCalls(f) {
|
|
247
247
|
return [...f.calls.values()].filter(c => !f.done || c.generation);
|
|
248
248
|
}
|
|
249
|
-
/**
|
|
249
|
+
/** The waits of a folded workflow's live calls (calls that move are absent). */
|
|
250
250
|
export function waitsOf(f, env, agentModel) {
|
|
251
251
|
const out = new Map();
|
|
252
252
|
if (!f.calls.size)
|
|
@@ -263,8 +263,8 @@ const base = (m) => ({ wid: m.wid, ...(m.request ? { request: m.request } : {}),
|
|
|
263
263
|
* (also when the call seals or disappears while waiting); a change of detail or age alone emits nothing. Ids carry the
|
|
264
264
|
* observing tick: the same cause can recur with the same `since` (a used-up provider whose probe is refused again keeps
|
|
265
265
|
* its first `since`), and a reader deduplicating on the id must still see it. A draft that failed to log is retried with
|
|
266
|
-
* its id (
|
|
267
|
-
export class
|
|
266
|
+
* its id (startWaiting), and the seed keeps a restarted tracker from repeating a logged transition. */
|
|
267
|
+
export class WaitTracker {
|
|
268
268
|
last = new Map();
|
|
269
269
|
/** Start from the latest `waiting`/`moving` per call id (the log's view at an orchestrator start). */
|
|
270
270
|
seed(latest) {
|
|
@@ -299,11 +299,11 @@ export class R7Tracker {
|
|
|
299
299
|
return out;
|
|
300
300
|
}
|
|
301
301
|
}
|
|
302
|
-
/** Every `intervalMs` (k.
|
|
302
|
+
/** Every `intervalMs` (k.waitCheckMs, default 5000): collect, diff, emit. Never overlaps itself; an emit failure is logged and
|
|
303
303
|
* its drafts are emitted first on the next tick (in order, same ids), so no transition is lost. `tick()` runs one now
|
|
304
304
|
* (or joins the one running). */
|
|
305
|
-
export function
|
|
306
|
-
const tracker = options.tracker ?? new
|
|
305
|
+
export function startWaiting(options) {
|
|
306
|
+
const tracker = options.tracker ?? new WaitTracker(), now = options.now ?? Date.now, log = options.log ?? (line => console.error(line));
|
|
307
307
|
let pending = [], running, stopped = false, timer, epoch = options.epoch?.();
|
|
308
308
|
const once = async () => {
|
|
309
309
|
try {
|
|
@@ -315,7 +315,7 @@ export function startR7(options) {
|
|
|
315
315
|
pending.push(...tracker.diff(at, current, meta));
|
|
316
316
|
}
|
|
317
317
|
catch (error) {
|
|
318
|
-
log(`durable-subagents:
|
|
318
|
+
log(`durable-subagents: wait collection failed: ${String(error)}`);
|
|
319
319
|
}
|
|
320
320
|
while (pending.length) {
|
|
321
321
|
const batch = pending.slice(0, EVENT_SEQ_SKIP);
|
|
@@ -324,7 +324,7 @@ export function startR7(options) {
|
|
|
324
324
|
pending = pending.slice(batch.length);
|
|
325
325
|
}
|
|
326
326
|
catch (error) {
|
|
327
|
-
log(`durable-subagents: ${pending.length}
|
|
327
|
+
log(`durable-subagents: ${pending.length} waiting/moving event(s) not logged, retried next tick: ${String(error)}`);
|
|
328
328
|
return;
|
|
329
329
|
}
|
|
330
330
|
}
|
|
@@ -339,13 +339,13 @@ export function startR7(options) {
|
|
|
339
339
|
schedule();
|
|
340
340
|
return { tick, async stop() { stopped = true; clearTimeout(timer); await running; } };
|
|
341
341
|
}
|
|
342
|
-
/** The waits of the orchestrator's live calls, for `
|
|
343
|
-
* const collect =
|
|
344
|
-
*
|
|
342
|
+
/** The waits of the orchestrator's live calls, for `startWaiting`'s `collect`:
|
|
343
|
+
* const collect = waitCollector({ home, workflows: () => engine.store.workflows.values(), orch: ledgers.orch, config: ledgers.config });
|
|
344
|
+
* startWaiting({ collect: () => collect(), sink, intervalMs: config.k?.waitCheckMs, tracker });
|
|
345
345
|
* Per tick: one `entries()` per workflow (a cached view unless it was appended to), each changed journal folded only
|
|
346
346
|
* past what was folded before; a workflow without unsealed calls costs a map lookup. The ledger is folded the same way
|
|
347
347
|
* (slots, used-up providers, settings, and each workflow's request id and labels). Call it from one place at a time. */
|
|
348
|
-
export function
|
|
348
|
+
export function waitCollector(src) {
|
|
349
349
|
const folds = new Map();
|
|
350
350
|
const ledger = emptyLedger();
|
|
351
351
|
// Run requests not yet created (rid → labels) and the request id and labels of each workflow: bounded by the
|
package/dist/kernel/lifecycle.js
CHANGED
|
@@ -35,7 +35,7 @@ export function planDecisions(records, candidates, decide) {
|
|
|
35
35
|
if (bound) {
|
|
36
36
|
if (bound.hash === hash)
|
|
37
37
|
envelopes.set(req.rid, req);
|
|
38
|
-
// A1
|
|
38
|
+
// A1: a conflicting duplicate of a resolved rid is only dropped (its file still goes); it never records a decision.
|
|
39
39
|
else if (!view.resolved.has(req.rid) && !conflicts.has(req.rid)) {
|
|
40
40
|
conflicts.add(req.rid);
|
|
41
41
|
emit({ type: 'rejected', rid: req.rid, reason: 'identity-conflict' });
|
|
@@ -3,14 +3,14 @@
|
|
|
3
3
|
// Orchestrator ledger entries: config{hash,config} — the settings in effect from then on (at start, or after a change);
|
|
4
4
|
// config-rejected{hash,error} — a changed file that was not applied; the settings before it stay in effect.
|
|
5
5
|
// A reload changes the shared config object in place: every later read sees it (the next slot acquisition, model
|
|
6
|
-
// resolution or check). Slots already held are kept when a limit drops; timers of running executions keep their period, and the
|
|
7
|
-
// period (k.
|
|
6
|
+
// resolution or check). Slots already held are kept when a limit drops; timers of running executions keep their period, and the waiting check
|
|
7
|
+
// period (k.waitCheckMs) is read once at orchestrator start.
|
|
8
8
|
import { readFile } from "node:fs/promises";
|
|
9
9
|
import { join } from "node:path";
|
|
10
10
|
import { contentHash } from "../kernel/ids.js";
|
|
11
11
|
/** The keys the orchestrator reads; config.json also holds pi-side settings (ui, onQuit) that it ignores. */
|
|
12
12
|
const KEYS = ["defaultModel", "pools", "providers", "memory", "writerLock", "k"];
|
|
13
|
-
const K = ["lossBound", "checkpointMs", "stallMs", "progressMs", "switchTimeoutMs", "idleExitMs", "trackerMs", "hibernateMs", "spawnBudget", "probeMs", "eventRetentionMs", "
|
|
13
|
+
const K = ["lossBound", "checkpointMs", "stallMs", "progressMs", "switchTimeoutMs", "idleExitMs", "trackerMs", "hibernateMs", "spawnBudget", "probeMs", "eventRetentionMs", "waitCheckMs"];
|
|
14
14
|
export const configPath = (home) => join(home, "config.json");
|
|
15
15
|
/** The orchestrator's part of a parsed config.json. */
|
|
16
16
|
export function orchestratorSettings(raw) {
|
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
// stop-requested {rid,call?} marks a call or workflow stop as taking effect, so its replay is applied, not already-sealed.
|
|
10
10
|
// Orch entry pruned {rid,wid,endedAt,bytes,status,request?,spec_digest?} is the decisive record of a prune: appended before
|
|
11
11
|
// the journal handle is closed and w/<wid> and its staging dirs are removed (bytes = footprint measured just before;
|
|
12
|
-
// status = the final workflow status; request/spec_digest when the workflow was created by a request id,
|
|
12
|
+
// status = the final workflow status; request/spec_digest when the workflow was created by a request id, request-id tombstone).
|
|
13
13
|
// Nothing is rewritten (A1): the admitted request and created entries stay, so a retried id still resolves to this wid.
|
|
14
14
|
import { watch } from 'node:fs';
|
|
15
15
|
import { mkdir, readdir, unlink } from 'node:fs/promises';
|
|
@@ -28,7 +28,7 @@ import { formatUsage, holdOf, refusedResult, snapshotFromEntries } from "./snaps
|
|
|
28
28
|
import { validateCallSpec } from "../compat/spec.js";
|
|
29
29
|
import { EventPump } from "../events/pump.js";
|
|
30
30
|
import { labelsProblem } from "../events/labels.js";
|
|
31
|
-
import {
|
|
31
|
+
import { WaitTracker, waitCollector, startWaiting } from "../events/waiting.js";
|
|
32
32
|
import { readPage } from "../events/log.js";
|
|
33
33
|
import { eventsLog } from "../paths.js";
|
|
34
34
|
import { parseModel } from "../compat/model.js";
|
|
@@ -97,9 +97,9 @@ export function finishedText(wid, entries, call) {
|
|
|
97
97
|
/** A1, P2, P10, P11: Serialize decisions while executions run independently. */
|
|
98
98
|
export class Engine {
|
|
99
99
|
store;
|
|
100
|
-
/**
|
|
100
|
+
/** The cross-workflow event log's writer (derives milestones from the journals; waiting/moving emits through it). */
|
|
101
101
|
events;
|
|
102
|
-
|
|
102
|
+
waiting;
|
|
103
103
|
ledgers;
|
|
104
104
|
executor;
|
|
105
105
|
evaluator;
|
|
@@ -152,7 +152,7 @@ export class Engine {
|
|
|
152
152
|
/** A2, P10: Recover executor authority before replaying each unfinished workflow. */
|
|
153
153
|
async recover() {
|
|
154
154
|
await this.store.recover();
|
|
155
|
-
//
|
|
155
|
+
// Before any recovery append, so every journal entry from here on is derived promptly (and a new log derives
|
|
156
156
|
// everything still on disk).
|
|
157
157
|
await this.events.open();
|
|
158
158
|
for (const wf of this.store.workflows.values()) {
|
|
@@ -186,14 +186,14 @@ export class Engine {
|
|
|
186
186
|
this.dispatchGeneration(wf, entry);
|
|
187
187
|
}
|
|
188
188
|
await this.intake();
|
|
189
|
-
this.
|
|
189
|
+
this.startWaiting();
|
|
190
190
|
}
|
|
191
|
-
/**
|
|
191
|
+
/** Every k.waitCheckMs (read here, at orchestrator start), why each live call does not move, as `waiting`/`moving`
|
|
192
192
|
* events through the pump. The tracker starts from the log's latest transition per call, so a restart repeats none
|
|
193
193
|
* (and a call that ended or started moving meanwhile gets its `moving`). The seed reads from seq 0: retention keeps
|
|
194
|
-
* events below `dropped`. A seed that cannot be read is logged and
|
|
195
|
-
|
|
196
|
-
if (this.closed || this.
|
|
194
|
+
* events below `dropped`. A seed that cannot be read is logged and the tracker starts empty (recovery never fails on it). */
|
|
195
|
+
startWaiting() {
|
|
196
|
+
if (this.closed || this.waiting)
|
|
197
197
|
return;
|
|
198
198
|
const latest = new Map(), head = this.events.head;
|
|
199
199
|
try {
|
|
@@ -211,13 +211,13 @@ export class Engine {
|
|
|
211
211
|
}
|
|
212
212
|
}
|
|
213
213
|
catch (error) {
|
|
214
|
-
console.error(`durable-subagents:
|
|
214
|
+
console.error(`durable-subagents: waiting seed from the event log failed, starting without it: ${String(error)}`);
|
|
215
215
|
latest.clear();
|
|
216
216
|
}
|
|
217
|
-
const tracker = new
|
|
217
|
+
const tracker = new WaitTracker();
|
|
218
218
|
tracker.seed(latest);
|
|
219
|
-
const collect =
|
|
220
|
-
this.
|
|
219
|
+
const collect = waitCollector({ home: this.ledgers.home, workflows: () => this.store.workflows.values(), orch: this.ledgers.orch, config: this.ledgers.config });
|
|
220
|
+
this.waiting = startWaiting({ collect: () => collect(), sink: this.events, intervalMs: this.ledgers.config.k?.waitCheckMs, tracker, epoch: () => this.events.head?.epoch });
|
|
221
221
|
}
|
|
222
222
|
async startHost() {
|
|
223
223
|
await this.evaluator.start(message => this.background(() => this.message(message)), () => this.background(async () => {
|
|
@@ -342,7 +342,7 @@ export class Engine {
|
|
|
342
342
|
}
|
|
343
343
|
if (req.kind === 'run') {
|
|
344
344
|
const created = this.ledgers.orch.entries().find(e => e.type === JT.created && e.rid === req.rid);
|
|
345
|
-
//
|
|
345
|
+
// Senders validate labels; a hand-written request must not bypass that.
|
|
346
346
|
const labels = req.body?.labels, invalid = !created && labels !== undefined ? labelsProblem(labels) : undefined;
|
|
347
347
|
if (invalid)
|
|
348
348
|
return { action: 'reject', reason: `invalid-labels: ${invalid}` };
|
|
@@ -590,7 +590,7 @@ export class Engine {
|
|
|
590
590
|
const createdBy = String(entries.find(e => e.type === JT.created && e.wid === wf.wid)?.rid ?? '');
|
|
591
591
|
const admitted = requestId(createdBy) !== undefined ? entries.find(e => e.type === 'request' && e.request.rid === createdBy)?.request : undefined;
|
|
592
592
|
const identity = admitted ? { request: requestId(createdBy), spec_digest: specDigest(admitted) } : {};
|
|
593
|
-
//
|
|
593
|
+
// Its events are derived and logged before the journal can go (recovery removes it once `pruned` is committed);
|
|
594
594
|
// when they cannot be, the prune is rejected and the caller retries later.
|
|
595
595
|
try {
|
|
596
596
|
await this.events.flush();
|
|
@@ -857,7 +857,7 @@ export class Engine {
|
|
|
857
857
|
this.closed = true;
|
|
858
858
|
this.watcher?.close();
|
|
859
859
|
clearInterval(this.poll);
|
|
860
|
-
await this.
|
|
860
|
+
await this.waiting?.stop();
|
|
861
861
|
await this.queue;
|
|
862
862
|
await this.evaluator.close();
|
|
863
863
|
await this.executor.shutdown();
|
|
@@ -136,7 +136,9 @@ export function evidence(entries, exec) {
|
|
|
136
136
|
/** Only explicit payment failures are terminal; rate limits, overload and transport errors still retry, and a used-up
|
|
137
137
|
* usage window (`quotaExhausted`) waits for the provider or moves to another one. */
|
|
138
138
|
export function fatalProviderError(text) {
|
|
139
|
-
|
|
139
|
+
// Chinese "balance", but not inside "remaining quota": "remaining quota is 0, resets at 00:00 the next day" is a daily
|
|
140
|
+
// window. CJK text is written as \u escapes (the repository is ASCII-only English).
|
|
141
|
+
return /\b402\b|insufficient[_ ]?(quota|balance|funds)|billing|credit balance|(?<!\u5269)\u4f59\u989d/i.test(text);
|
|
140
142
|
}
|
|
141
143
|
/** A provider's usage window is used up: its requests are refused (and not counted) until the window resets, hours
|
|
142
144
|
* later. Seen as a gateway's `503 No available accounts` once pi's own retries are spent, or a usage-limit message.
|
|
@@ -150,7 +152,7 @@ export function quotaExhausted(text) {
|
|
|
150
152
|
// per minute"): pi's retries and the lost-execution path handle it; it must not take the provider out for minutes.
|
|
151
153
|
if (/rate.?limit|too many requests|request limit|per (second|minute)|\b[RT]PM\b|resets? in \d+ ?(ms|s|secs?|seconds?|minutes?)\b/i.test(text))
|
|
152
154
|
return false;
|
|
153
|
-
return /usage limit|quota (exceeded|exhausted)|exceeded your (current )?(usage|quota)|limit (reached|exceeded)[^.]*resets?\b
|
|
155
|
+
return /usage limit|quota (exceeded|exhausted)|exceeded your (current )?(usage|quota)|limit (reached|exceeded)[^.]*resets?\b|\u989d\u5ea6/i.test(text);
|
|
154
156
|
}
|
|
155
157
|
/** A refusal of the request's content (terms of service, usage or content policy): the same request is refused again,
|
|
156
158
|
* on this provider and usually on another, so it is reported at once instead of retried as a lost execution. */
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// Restart guards shared by the orchestrator and the legacy client. The caller owns the launch gate;
|
|
2
2
|
// these checks never change durable state. A1: only an accepted restart is appended to the ledger.
|
|
3
3
|
import { contentHash } from "../kernel/ids.js";
|
|
4
|
-
import {
|
|
4
|
+
import { holdDetail, leaseState, who } from "../platform/lease.js";
|
|
5
5
|
export const subagentRestartError = "a subagent cannot force a restart: it would fence itself and other sessions' work; ask the user";
|
|
6
6
|
export const restartToken = (live) => contentHash(live.map(l => l.exec).sort()).slice(0, 12);
|
|
7
7
|
/** Old senders' boolean force is read only to refuse it when executions are live. */
|
|
@@ -27,7 +27,17 @@ export function restartRefusal(home, live, body, tool = false, now = Date.now())
|
|
|
27
27
|
const token = restartToken(live);
|
|
28
28
|
if (body.token === token)
|
|
29
29
|
return undefined;
|
|
30
|
-
|
|
30
|
+
// What a fence would cut short: each lease a listed call holds, with its age and command. Holders outside these
|
|
31
|
+
// executions (a shell, a systemd unit, another call) keep their lease across the restart; they are listed so the
|
|
32
|
+
// machine is not mistaken for free.
|
|
33
|
+
const leases = new Map(), others = [], calls = new Set(live.map(l => l.callId));
|
|
34
|
+
for (const { resource, holders } of leaseState(home))
|
|
35
|
+
for (const t of holders) {
|
|
36
|
+
if (t.call && calls.has(t.call))
|
|
37
|
+
leases.set(t.call, [leases.get(t.call), `holds lease ${resource} (${holdDetail(t, now)})`].filter(Boolean).join(", "));
|
|
38
|
+
else
|
|
39
|
+
others.push(` ${resource} held by ${who(t)} (${holdDetail(t, now)})`);
|
|
40
|
+
}
|
|
31
41
|
const groups = new Map();
|
|
32
42
|
for (const l of live) {
|
|
33
43
|
const origin = l.origin ?? "unknown";
|
|
@@ -40,6 +50,7 @@ export function restartRefusal(home, live, body, tool = false, now = Date.now())
|
|
|
40
50
|
...[...groups].sort(([a], [b]) => a.localeCompare(b)).flatMap(([origin, executions]) => [
|
|
41
51
|
`${origin}:`, ...executions.map(l => ` ${l.wid}/${l.key} ${age(now - l.since)}${l.phase === "gate" ? " gate" : ""}${leases.has(l.callId) ? ` ${leases.get(l.callId)}` : ""}`),
|
|
42
52
|
]),
|
|
53
|
+
...(others.length ? ["not fenced (a restart leaves these leases held):", ...others] : []),
|
|
43
54
|
`token: ${token}`,
|
|
44
55
|
tool ? `to fence exactly these: subagents {action:"restart", force:"${token}", reason:"<why>"}` : `to fence exactly these: pi-durable-subagents restart --force ${token} --reason "<why>"`,
|
|
45
56
|
].join("\n");
|
|
@@ -543,7 +543,7 @@ export function runningOrchestrator(home) {
|
|
|
543
543
|
ledgerStates.set(path, state);
|
|
544
544
|
return orchestratorView(state);
|
|
545
545
|
}
|
|
546
|
-
/** One `exhausted` line of the status views (
|
|
546
|
+
/** One `exhausted` line of the status views (the `provider-exhausted` wait uses it as its detail). */
|
|
547
547
|
export function exhaustedLine(provider, x, now) {
|
|
548
548
|
return `${provider} exhausted since ${age(now - x.since)} ago (${clip(x.error, 80)}), ` +
|
|
549
549
|
(x.probe ? `probing with ${x.probe.split("#")[0]}` : x.nextTry > now ? `next try in ${age(x.nextTry - now)}` : "next call probes it");
|
|
@@ -180,7 +180,7 @@ export async function diskUsage(path) {
|
|
|
180
180
|
/** A1, P11: Own shared workflow handles and reconcile create intents after a crash. */
|
|
181
181
|
export class Store {
|
|
182
182
|
workflows = new Map();
|
|
183
|
-
/**
|
|
183
|
+
/** Called after each append to a workflow journal (the event pump derives from it). */
|
|
184
184
|
appended;
|
|
185
185
|
ledgers;
|
|
186
186
|
constructor(ledgers) { this.ledgers = ledgers; }
|
package/dist/paths.js
CHANGED
|
@@ -6,7 +6,7 @@ import { ENV } from "./types.js";
|
|
|
6
6
|
export const dsaHome = (env = process.env) => env[ENV.home] || path.join(os.homedir(), ".pi", "durable-subagents");
|
|
7
7
|
export const orchLedger = (home) => path.join(home, "orchestrator.jsonl");
|
|
8
8
|
export const orchLock = (home) => path.join(home, "orchestrator.lock");
|
|
9
|
-
/**
|
|
9
|
+
/** The cross-workflow event log (single writer: the orchestrator; read by `events --all`). */
|
|
10
10
|
export const eventsLog = (home) => path.join(home, "events.jsonl");
|
|
11
11
|
/** Executables for subagents: the orchestrator writes a `pi-durable-subagents` shim here and children get it on PATH. */
|
|
12
12
|
export const binDir = (home) => path.join(home, "bin");
|
package/dist/platform/lease.js
CHANGED
|
@@ -159,6 +159,16 @@ export function removeTicket(home, t) {
|
|
|
159
159
|
/** Queue a request: number it under the resource's kernel lock and write its ticket before the lock is released, so a
|
|
160
160
|
* later request always sees it. */
|
|
161
161
|
export async function enqueue(home, ticket, lock = new OsLock()) {
|
|
162
|
+
return (await numbered(home, ticket, false, lock));
|
|
163
|
+
}
|
|
164
|
+
/** Take the lease now or not at all (`hold --max-wait 0`): under the resource's kernel lock, either no live ticket
|
|
165
|
+
* blocks it and its ticket is written already granted, or nothing is written and the blockers are returned. A refused
|
|
166
|
+
* request is never visible as a waiter, and a granted one never was one. Granting needs no more than the lock: a later
|
|
167
|
+
* request gets a higher number and waits behind this ticket, and earlier tickets can only disappear. */
|
|
168
|
+
export async function tryGrant(home, ticket, lock = new OsLock()) {
|
|
169
|
+
return numbered(home, ticket, true, lock);
|
|
170
|
+
}
|
|
171
|
+
async function numbered(home, ticket, now, lock) {
|
|
162
172
|
const dir = leaseDir(home, ticket.resource);
|
|
163
173
|
mkdirSync(dir, { recursive: true });
|
|
164
174
|
let handle = await lock.tryAcquire(path.join(dir, ".lock"));
|
|
@@ -171,6 +181,12 @@ export async function enqueue(home, ticket, lock = new OsLock()) {
|
|
|
171
181
|
const last = Number.parseInt(await readFile(counter, "utf8").catch(() => "0"), 10) || 0;
|
|
172
182
|
const highest = readTickets(home, ticket.resource).reduce((m, t) => Math.max(m, t.seq), last);
|
|
173
183
|
const t = { ...ticket, seq: highest + 1 };
|
|
184
|
+
if (now) {
|
|
185
|
+
const busy = blockers(t, liveTickets(home, ticket.resource));
|
|
186
|
+
if (busy.length)
|
|
187
|
+
return { busy };
|
|
188
|
+
t.grantedAt = Date.now();
|
|
189
|
+
}
|
|
174
190
|
await writeFile(`${counter}.tmp`, String(t.seq));
|
|
175
191
|
renameSync(`${counter}.tmp`, counter);
|
|
176
192
|
writeTicketSync(home, t);
|
|
@@ -201,12 +217,13 @@ const clip = (s, n) => s.length > n ? `${s.slice(0, n - 1)}…` : s;
|
|
|
201
217
|
export const callAddress = (call) => { const m = /^([^@/]+)@\d+\/(.+)@\d+$/.exec(call); return m ? `${m[1]}/${m[2]}` : call; };
|
|
202
218
|
/** Who holds or waits: the subagent call when there is one, else the pid and command. */
|
|
203
219
|
export const who = (t) => t.call ? callAddress(t.call) : `pid ${t.wrapper.pid} \`${clip(t.argv.join(" "), 60)}\``;
|
|
220
|
+
/** A granted ticket's mode, how long it has been held, the command (unless `who` already names it) and the note:
|
|
221
|
+
* "exclusive, 12m, `make bench`, nightly". */
|
|
222
|
+
export const holdDetail = (t, now = Date.now(), command = !!t.call) => `${t.mode}, ${age(now - (t.grantedAt ?? t.since))}${command ? `, \`${clip(t.argv.join(" "), 60)}\`` : ""}${t.note ? `, ${clip(t.note, 80)}` : ""}`;
|
|
204
223
|
/** One status line per resource: "machine held by <who> (exclusive, 12m, `make bench`); waiting: <who> 3m, …". */
|
|
205
224
|
export function leaseLines(state, now = Date.now()) {
|
|
206
225
|
return state.map(({ resource, holders, waiters }) => {
|
|
207
|
-
const held = holders.length
|
|
208
|
-
? `held by ${holders.map(t => `${who(t)} (${t.mode}, ${age(now - (t.grantedAt ?? t.since))}${t.call ? `, \`${clip(t.argv.join(" "), 60)}\`` : ""}${t.note ? `, ${clip(t.note, 80)}` : ""})`).join(", ")}`
|
|
209
|
-
: "free";
|
|
226
|
+
const held = holders.length ? `held by ${holders.map(t => `${who(t)} (${holdDetail(t, now)})`).join(", ")}` : "free";
|
|
210
227
|
return `${resource} ${held}${waiters.length ? `; waiting: ${waiters.map(t => `${who(t)} ${t.mode === "shared" ? "shared " : ""}${age(now - t.since)}`).join(", ")}` : ""}`;
|
|
211
228
|
});
|
|
212
229
|
}
|
package/dist/requests.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Caller-chosen request ids. A program names a run/send/stop with `<id>`; its kernel rid is `req:<id>` (fits the
|
|
2
2
|
// mailbox rid pattern), unique per DSA_HOME across kinds and senders. A retry with the same id and the same content
|
|
3
3
|
// (spec_digest) gets the first outcome; a different content is a request-conflict and is never published.
|
|
4
4
|
// The check-and-send is serialized per home by the OS lock <home>/requests.lock (CLI and tool senders alike); no
|
|
@@ -13,15 +13,15 @@ import { OsLock } from "./platform/lock.js";
|
|
|
13
13
|
import { orchInbox, orchLedger, outboxRoot } from "./paths.js";
|
|
14
14
|
export const REQUEST_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,123}$/;
|
|
15
15
|
const PREFIX = 'req:';
|
|
16
|
-
/**
|
|
16
|
+
/** `<id>` → its rid `req:<id>`; ids are 1–124 chars `[A-Za-z0-9][A-Za-z0-9._:-]*`. */
|
|
17
17
|
export function requestRid(id) {
|
|
18
18
|
if (!REQUEST_ID.test(id))
|
|
19
19
|
throw new Error(`invalid request id ${JSON.stringify(id)}: 1-124 characters [A-Za-z0-9][A-Za-z0-9._:-]*`);
|
|
20
20
|
return PREFIX + id;
|
|
21
21
|
}
|
|
22
|
-
/**
|
|
22
|
+
/** The request id of a `req:<id>` rid; undefined for any other rid (ULIDs never contain ':'). */
|
|
23
23
|
export function requestId(rid) { return rid.startsWith(PREFIX) ? rid.slice(PREFIX.length) : undefined; }
|
|
24
|
-
/**
|
|
24
|
+
/** Spec_digest = contentHash({kind, body, cond}) (cond omitted when absent). A run's body.origin (the pi session
|
|
25
25
|
* branch offered for context:"fork") is delivery metadata, not spec: it differs on every turn, so it is not hashed. */
|
|
26
26
|
export function specDigest(req) {
|
|
27
27
|
let body = req.body;
|
|
@@ -29,9 +29,13 @@ export function specDigest(req) {
|
|
|
29
29
|
const { origin: _, ...rest } = body;
|
|
30
30
|
body = rest;
|
|
31
31
|
}
|
|
32
|
+
if (req.kind === 'send' && body && typeof body === 'object' && !Array.isArray(body)) {
|
|
33
|
+
const { caller: _, ...rest } = body;
|
|
34
|
+
body = rest;
|
|
35
|
+
}
|
|
32
36
|
return contentHash({ kind: req.kind, body, cond: req.cond });
|
|
33
37
|
}
|
|
34
|
-
/**
|
|
38
|
+
/** The envelope recorded for `rid`: the orchestrator's admitted copy (ledger `request`, kept after prune), else any
|
|
35
39
|
* sender's outbox `sent` entry (published or about to be). Read-only. */
|
|
36
40
|
export async function findRequest(home, rid) {
|
|
37
41
|
const admitted = readJournalSnapshot(orchLedger(home)).find(e => e.type === 'request' && e.request.rid === rid);
|
|
@@ -51,7 +55,7 @@ export async function findRequest(home, rid) {
|
|
|
51
55
|
export class RequestsBusy extends Error {
|
|
52
56
|
name = 'RequestsBusy';
|
|
53
57
|
}
|
|
54
|
-
/**
|
|
58
|
+
/** P5: Check-then-send under the home-wide request-id lock (innermost: taken after a sender's own lock), so no two
|
|
55
59
|
* senders publish one rid and no second envelope with an existing rid and other content reaches the inbox (it would
|
|
56
60
|
* stall its sender's sequence). Same content: the recorded envelope stands (republished when it is this sender's and
|
|
57
61
|
* pending; `sent` false). */
|
package/dist/ui/session.js
CHANGED
|
@@ -86,8 +86,8 @@ export function thoughtSummary(text) {
|
|
|
86
86
|
return (headings.at(-1)[1] ?? headings.at(-1)[2]).trim();
|
|
87
87
|
return lastSentence(text);
|
|
88
88
|
}
|
|
89
|
-
const STOPS = new Set([..."
|
|
90
|
-
/** The last match of /[
|
|
89
|
+
const STOPS = new Set([...".!?\u3002\uff01\uff1f"]);
|
|
90
|
+
/** The last match of /[^.!?\u3002\uff01\uff1f]+[.!?\u3002\uff01\uff1f](?=\s|$)/gu without the regex: a long thought with no sentence end made
|
|
91
91
|
* that regex retry from every position (quadratic), which stalled pi's startup on long histories. Such a match is a
|
|
92
92
|
* whole run of non-stop characters followed by one stop that ends the text or precedes whitespace. */
|
|
93
93
|
function lastSentence(text) {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-durable-subagents",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.27",
|
|
4
4
|
"description": "Subagents for pi that never lose work and never do it twice. Crash-safe workflows, automatic recovery, and a live view just like the main agent.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|