pi-durable-subagents 1.0.24 → 1.0.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -1
- package/README.md +30 -10
- package/dist/agent/main/tool.js +2 -2
- package/dist/agent/main.js +1 -1
- package/dist/cli/control.js +1 -1
- package/dist/cli/main.js +2 -2
- package/dist/cli/requests.js +14 -14
- package/dist/events/cli.js +1 -1
- package/dist/events/derive.js +2 -2
- package/dist/events/fence.js +4 -4
- package/dist/events/labels.js +1 -1
- package/dist/events/log.js +2 -2
- package/dist/events/pump.js +4 -4
- package/dist/events/types.js +2 -3
- package/dist/events/{r7.js → waiting.js} +15 -15
- package/dist/kernel/lifecycle.js +1 -1
- package/dist/orchestrator/config.js +3 -3
- package/dist/orchestrator/engine.js +17 -17
- package/dist/orchestrator/restart.js +13 -2
- package/dist/orchestrator/snapshot.js +1 -1
- package/dist/orchestrator/store.js +1 -1
- package/dist/paths.js +1 -1
- package/dist/platform/lease.js +4 -3
- package/dist/requests.js +6 -6
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.0.25
|
|
4
|
+
|
|
5
|
+
- `restart` refusals show what a fence would cut short: each lease a running
|
|
6
|
+
call holds, with its mode, how long it has been held, its command and note.
|
|
7
|
+
Leases held outside those executions (a shell, a `systemd-run` unit) are
|
|
8
|
+
listed apart, since a restart leaves them held.
|
|
9
|
+
- The waiting/moving check period is configured as `k.waitCheckMs` (was
|
|
10
|
+
`k.r7Ms` in 1.0.24; the old key is not accepted).
|
|
11
|
+
- README: deduplicate events by `id`, not cursor; a call with no next
|
|
12
|
+
execution has no `fenced`; an open question keeps the orchestrator from its
|
|
13
|
+
idle exit (compact the event log now with a non-force `restart`); leases
|
|
14
|
+
outside calls survive restarts; why a restart cannot hand running
|
|
15
|
+
executions to the new orchestrator.
|
|
16
|
+
|
|
3
17
|
## 1.0.24
|
|
4
18
|
|
|
5
19
|
- `events --all [--since <cursor>] [--limit <n>]`: one durable log of
|
|
@@ -13,7 +27,7 @@
|
|
|
13
27
|
prune whose events cannot be logged first is rejected (`event-log: …`).
|
|
14
28
|
- `run --labels <json>` (tool: `labels`): caller labels, part of the spec
|
|
15
29
|
digest, returned by `describe` and echoed on every event of the run.
|
|
16
|
-
- Why a call does not move
|
|
30
|
+
- Why a call does not move: `describe` adds `reason`, `detail` and
|
|
17
31
|
`since` to each waiting call, and the event log gets `waiting`/`moving`
|
|
18
32
|
when the reason changes: `unconfirmed-stop`, `provider-exhausted`,
|
|
19
33
|
`writer-lock`, `lease`, `slot`, `silent`.
|
package/README.md
CHANGED
|
@@ -368,7 +368,8 @@ after recovery, before it waits for a slot) and only when the fence
|
|
|
368
368
|
interrupted work, exactly as `describe`'s `lastFence`: a turn that had ended,
|
|
369
369
|
a hibernated question, an answer's resume or a seal are no `fenced`. A `once`
|
|
370
370
|
call cut off in a tool is never resumed: it gets `sealed` with status
|
|
371
|
-
`unknown` and no `fenced`.
|
|
371
|
+
`unknown` and no `fenced`. A call with no next execution has no `fenced`; its
|
|
372
|
+
`sealed` carries the outcome (`describe`'s `lastFence` still names the fence).
|
|
372
373
|
|
|
373
374
|
Cursors are `<epoch>:<seq>`; `--since c` returns the events after `c` in log
|
|
374
375
|
order, at most `--limit` (default and maximum 1000), then
|
|
@@ -384,14 +385,16 @@ exists it never starts anything.
|
|
|
384
385
|
|
|
385
386
|
Delivery is at least once, without gaps: after a crash the orchestrator
|
|
386
387
|
derives again from its last durable watermark, and an event derived again has
|
|
387
|
-
the same `id` (a new cursor). Deduplicate by `id`,
|
|
388
|
-
|
|
388
|
+
the same `id` (a new cursor). Deduplicate by `id`, not by cursor (keeping
|
|
389
|
+
ids for the retention window is enough), and persist your cursor only after
|
|
390
|
+
you applied the events of a page.
|
|
389
391
|
|
|
390
392
|
Retention: an event is dropped only when it was logged more than 7 days ago
|
|
391
|
-
(`"k": { "eventRetentionMs": … }` in
|
|
393
|
+
(`"k": { "eventRetentionMs": … }` in `$DSA_HOME/config.json`) and its workflow is
|
|
392
394
|
finished in its current revision (done, failed or stopped — not parked) with
|
|
393
395
|
no open question and no unsealed call, or was pruned. The log is compacted at
|
|
394
|
-
orchestrator start and at most hourly
|
|
396
|
+
orchestrator start and at most hourly; to compact now while a question is open
|
|
397
|
+
(which keeps the orchestrator from idle exit), run `restart` without `--force`. A cursor of another epoch (the log was
|
|
395
398
|
replaced: a corrupt log is kept aside as `events.jsonl.corrupt-<ms>` and a new
|
|
396
399
|
one starts), below the highest dropped seq, or beyond the head gets exit 4
|
|
397
400
|
and one line `{"error":"cursor-expired","head":"…","oldest":"…"}` (`oldest`
|
|
@@ -419,7 +422,7 @@ When an unsealed call does not move, its `waiting` in `describe` adds
|
|
|
419
422
|
probe 1/1`) and `since` (ms: when that cause started). The event log has the
|
|
420
423
|
same: `waiting {reason, detail, since}` when the reason appears or changes,
|
|
421
424
|
`moving {after}` when it clears (also when the call ends), checked every
|
|
422
|
-
`k.
|
|
425
|
+
`k.waitCheckMs` (default 5 s; read when the orchestrator starts, unlike the other
|
|
423
426
|
`k` settings a `config.json` change does not apply it until a restart); a
|
|
424
427
|
change of detail alone is no event. The first reason
|
|
425
428
|
that applies wins:
|
|
@@ -461,7 +464,7 @@ command:
|
|
|
461
464
|
```sh
|
|
462
465
|
pi-durable-subagents hold machine -- make bench # exclusive
|
|
463
466
|
pi-durable-subagents hold machine --shared -- npm test # with other shared holders, never with an exclusive one
|
|
464
|
-
pi-durable-subagents hold machine --max-wait 600 --note "
|
|
467
|
+
pi-durable-subagents hold machine --max-wait 600 --note "profile" -- ./measure.sh
|
|
465
468
|
```
|
|
466
469
|
|
|
467
470
|
- The lease covers one command, not a whole call: a subagent that thinks
|
|
@@ -482,6 +485,10 @@ pi-durable-subagents hold machine --max-wait 600 --note "frame phase" -- ./measu
|
|
|
482
485
|
ended). State is one small file per request under
|
|
483
486
|
`$DSA_HOME/leases/<resource>/`; no orchestrator is needed, and the user's
|
|
484
487
|
own shell can take part.
|
|
488
|
+
- A lease taken outside any call (your shell, a `systemd-run --user` unit)
|
|
489
|
+
does not depend on the orchestrator: a restart, forced or not, leaves it
|
|
490
|
+
held, and it is released when its `hold` and command end. A lease taken
|
|
491
|
+
inside a call ends with that call's processes when the call is fenced.
|
|
485
492
|
- Subagents find the command on their `PATH` (the orchestrator puts a shim
|
|
486
493
|
in `$DSA_HOME/bin`), and their leases are tagged with their call:
|
|
487
494
|
`status` shows `lease: machine held by <wid>/<key> …; waiting: …` and
|
|
@@ -565,8 +572,11 @@ On load, it checks the pi exports and API methods it uses.
|
|
|
565
572
|
Running work stays on the version it started with until you restart the
|
|
566
573
|
orchestrator. When the orchestrator runs another version than the one a pi
|
|
567
574
|
session loaded, that pi says so once, and `status` shows the running version
|
|
568
|
-
with a note. The orchestrator exits about 10 s after all
|
|
569
|
-
|
|
575
|
+
with a note. The orchestrator exits about 10 s (`k.idleExitMs`) after all
|
|
576
|
+
work ends: every workflow is finished in its current revision (done, failed,
|
|
577
|
+
stopped or parked) or held by `drain`. A workflow with an open question is not
|
|
578
|
+
finished, so a call hibernated on its question keeps the orchestrator running
|
|
579
|
+
(it holds no slot and costs little). The next start runs the new version. To switch sooner:
|
|
570
580
|
|
|
571
581
|
```sh
|
|
572
582
|
pi-durable-subagents restart # or the subagents tool: action "restart"
|
|
@@ -574,7 +584,9 @@ pi-durable-subagents restart # or the subagents tool: action "restart"
|
|
|
574
584
|
|
|
575
585
|
The orchestrator refuses while any execution runs (a subagent process, or a
|
|
576
586
|
gate before a call's seal). The refusal groups executions by session with ages,
|
|
577
|
-
|
|
587
|
+
the leases each one holds (mode, how long, command, note: what a fence would cut
|
|
588
|
+
short) and a token for that exact set. Leases held outside those executions are
|
|
589
|
+
listed apart, since the restart leaves them held; no new execution starts while
|
|
578
590
|
it decides, so nothing slips in between. Calls waiting
|
|
579
591
|
for your answer (hibernated), waiting for a provider slot, or held by a drain
|
|
580
592
|
do not block it. Otherwise it exits and its successor starts at once from the
|
|
@@ -600,6 +612,14 @@ boundary — a subagent runs as the same OS user and could signal the
|
|
|
600
612
|
orchestrator anyway. Force fences running
|
|
601
613
|
executions; they resume on the new version from their sessions, like after a
|
|
602
614
|
crash, so a tool call that was running is repeated or reported as interrupted.
|
|
615
|
+
A running execution cannot be handed over to the new orchestrator: each
|
|
616
|
+
subagent is a pi process the orchestrator drives over its stdin and stdout, and
|
|
617
|
+
those pipes end with the old process. A crash is no different: the successor
|
|
618
|
+
fences every execution that still runs (an execution that had already ended is
|
|
619
|
+
not counted as interrupted). Work that must survive a forced restart, such as a
|
|
620
|
+
long measurement, belongs outside the subagent's processes (for example
|
|
621
|
+
`systemd-run --user … pi-durable-subagents hold machine -- …`), with the
|
|
622
|
+
subagent only watching it.
|
|
603
623
|
The restart ledger records the reason and initiator; after the next start,
|
|
604
624
|
`status` shows who forced it and why for 24 hours.
|
|
605
625
|
|
package/dist/agent/main/tool.js
CHANGED
|
@@ -21,7 +21,7 @@ function call(value, cwd, where) {
|
|
|
21
21
|
return spec;
|
|
22
22
|
}
|
|
23
23
|
/** v12 §2: Reject unknown explicit call agents before starter or outbox publication; scripts remain call-local. Shared by
|
|
24
|
-
* the tool and the CLI `run --request
|
|
24
|
+
* the tool and the CLI `run --request`. */
|
|
25
25
|
export function checkAgents(body, available) {
|
|
26
26
|
const names = [...(body.call ? [body.call] : []), ...(body.tasks ?? []), ...(body.chain ?? [])].map(call => call.agent);
|
|
27
27
|
if (!names.length)
|
|
@@ -85,7 +85,7 @@ export function request(args, cwd) {
|
|
|
85
85
|
body.args = inputs;
|
|
86
86
|
if (name !== undefined)
|
|
87
87
|
body.name = string(args, "name");
|
|
88
|
-
//
|
|
88
|
+
// Part of the spec digest; an empty object is the same as none.
|
|
89
89
|
if (labels !== undefined && Object.keys(checkLabels(labels)).length)
|
|
90
90
|
body.labels = labels;
|
|
91
91
|
// P31a, P36, P11: workflow-level limits and declared input files (absolute paths, pinned at admission).
|
package/dist/agent/main.js
CHANGED
|
@@ -297,7 +297,7 @@ export function registerMain(pi, ui) {
|
|
|
297
297
|
if (args.action === "resume" && args.wid === undefined)
|
|
298
298
|
args = { ...args, origin: sender };
|
|
299
299
|
const normalized = request(args, cwd);
|
|
300
|
-
//
|
|
300
|
+
// A caller-chosen request id names a run, send or stop; a retry with the same content gets the first outcome.
|
|
301
301
|
if (args.request !== undefined && (typeof args.request !== "string" || !["run", "send", "stop"].includes(normalized.kind) || normalized.replaces?.length))
|
|
302
302
|
throw new Error(REQUEST_USE);
|
|
303
303
|
const rid = typeof args.request === "string" ? requestRid(args.request) : undefined;
|
package/dist/cli/control.js
CHANGED
|
@@ -104,7 +104,7 @@ export async function submit(home, command, target, env = process.env, options =
|
|
|
104
104
|
return requests;
|
|
105
105
|
});
|
|
106
106
|
}
|
|
107
|
-
/**
|
|
107
|
+
/** Submit a request named by a caller-chosen id through the CLI sender: a retry with the same content republishes
|
|
108
108
|
* (or reuses) the recorded envelope, other content is a conflict and publishes nothing. Starts the orchestrator
|
|
109
109
|
* unless the request conflicts. */
|
|
110
110
|
export async function submitIdentified(home, rid, kind, body, cond, env = process.env, starter = startOrchestrator) {
|
package/dist/cli/main.js
CHANGED
|
@@ -194,12 +194,12 @@ export async function main(args = process.argv.slice(2), options = {}) {
|
|
|
194
194
|
}
|
|
195
195
|
if (args[0] === "chaos")
|
|
196
196
|
return (await import("./chaos/index.js")).chaos(args.slice(1), options.env ?? process.env, options.write);
|
|
197
|
-
//
|
|
197
|
+
// The cross-workflow event log (strict flags of its own); `events <wid>` stays below.
|
|
198
198
|
if (args[0] === "events" && args.includes("--all")) {
|
|
199
199
|
const env = options.env ?? process.env;
|
|
200
200
|
return (await import("../events/cli.js")).eventsAll(args.slice(1), { home: dsaHome(env), env, write: options.write ?? ((line) => console.log(line)), starter: options.starter ?? startOrchestrator, waitMs: options.waitMs });
|
|
201
201
|
}
|
|
202
|
-
//
|
|
202
|
+
// Program-facing commands named by request ids (strict flags of their own).
|
|
203
203
|
if (["run", "send", "describe"].includes(args[0]) || (args[0] === "stop" && args.includes("--request"))) {
|
|
204
204
|
const env = options.env ?? process.env, requests = await import("./requests.js");
|
|
205
205
|
const ctx = { home: dsaHome(env), env, write: options.write ?? ((line) => console.log(line)), starter: options.starter, waitMs: options.waitMs, cwd: options.cwd, stdin: options.stdin };
|
package/dist/cli/requests.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Program-facing commands named by caller-chosen request ids — `run|send|stop --request <id>` and `describe`.
|
|
2
2
|
// Exit codes: 0 decided (applied/created), 1 rejected or invalid, 3 request-conflict (the id names other content),
|
|
3
3
|
// 75 not decided within --wait-ms (retry with the same id and content: safe).
|
|
4
4
|
import { existsSync, readFileSync } from "node:fs";
|
|
@@ -16,7 +16,7 @@ import { JT } from "../types.js";
|
|
|
16
16
|
import { startOrchestrator, submitIdentified } from "./control.js";
|
|
17
17
|
import { endedExecs, fenceReason } from "../events/fence.js";
|
|
18
18
|
import { parseLabels } from "../events/labels.js";
|
|
19
|
-
import { foldWaits, leaseWaits, waitsOf } from "../events/
|
|
19
|
+
import { foldWaits, leaseWaits, waitsOf } from "../events/waiting.js";
|
|
20
20
|
import { emptyLedger, foldLedger } from "../orchestrator/ledger.js";
|
|
21
21
|
export const EXIT = { ok: 0, rejected: 1, conflict: 3, pending: 75 };
|
|
22
22
|
/** Strict `--name value` / `--flag` parsing; unknown or repeated options are errors. */
|
|
@@ -57,7 +57,7 @@ function decision(entries, rid) {
|
|
|
57
57
|
const createdBy = (entries, rid) => entries.find(e => e.type === JT.created && e.rid === rid);
|
|
58
58
|
async function stdin() { const chunks = []; for await (const chunk of process.stdin)
|
|
59
59
|
chunks.push(chunk); return Buffer.concat(chunks).toString("utf8"); }
|
|
60
|
-
/**
|
|
60
|
+
/** The state of a request id (`{request}`) or a workflow (`{wid}`), with full texts (no clipping). */
|
|
61
61
|
export async function describe(home, key, now = Date.now()) {
|
|
62
62
|
const entries = ledger(home);
|
|
63
63
|
if ("wid" in key)
|
|
@@ -76,7 +76,7 @@ export async function describe(home, key, now = Date.now()) {
|
|
|
76
76
|
return { state: "applied", ...head };
|
|
77
77
|
return { state: "pending", ...head };
|
|
78
78
|
}
|
|
79
|
-
/**
|
|
79
|
+
/** `{labels}` of a run request that has any. */
|
|
80
80
|
function labelsOf(request) {
|
|
81
81
|
const labels = request?.kind === "run" ? request.body?.labels : undefined;
|
|
82
82
|
return labels && typeof labels === "object" && Object.keys(labels).length ? { labels } : {};
|
|
@@ -86,7 +86,7 @@ function runOf(entries, wid) {
|
|
|
86
86
|
const created = entries.find(e => e.type === JT.created && e.wid === wid);
|
|
87
87
|
return { created, run: created ? entries.find(e => e.type === "request" && e.request.rid === created.rid)?.request : undefined };
|
|
88
88
|
}
|
|
89
|
-
/** The pinned agents of a workflow revision, read once on demand (
|
|
89
|
+
/** The pinned agents of a workflow revision, read once on demand (the waiting check needs the model of a call that names none). */
|
|
90
90
|
function pinnedAgentModel(home, wid, rev) {
|
|
91
91
|
let agents;
|
|
92
92
|
return name => {
|
|
@@ -108,7 +108,7 @@ function describeWorkflow(home, wid, entries, now) {
|
|
|
108
108
|
...(typeof pruned.request === "string" ? { request: pruned.request } : {}), ...(typeof pruned.spec_digest === "string" ? { spec_digest: pruned.spec_digest } : {}), ...labelsOf(runOf(entries, wid).run) };
|
|
109
109
|
if (!/^[^/\\\0]+$/.test(wid) || wid === "." || wid === ".." || !existsSync(journalPath(home, wid)))
|
|
110
110
|
return { state: "absent", wid };
|
|
111
|
-
// The snapshot and the journal the
|
|
111
|
+
// The snapshot and the journal the wait fold reads must be the same bytes (a writer-wait appended between the two reads
|
|
112
112
|
// would give a reason without its writerWait): read again until the journal did not move around the snapshot.
|
|
113
113
|
let journal = readJournalSnapshot(journalPath(home, wid)), wf = workflowSnapshot(home, wid);
|
|
114
114
|
for (let i = 0, again = readJournalSnapshot(journalPath(home, wid)); again !== journal && i < 5; i++, again = readJournalSnapshot(journalPath(home, wid))) {
|
|
@@ -118,7 +118,7 @@ function describeWorkflow(home, wid, entries, now) {
|
|
|
118
118
|
const { created, run } = runOf(entries, wid), id = created ? requestId(String(created.rid)) : undefined;
|
|
119
119
|
const admitted = id ? run : undefined;
|
|
120
120
|
const lstate = leaseState(home), slots = slotsView(home, now), leases = leaseCalls(lstate, now);
|
|
121
|
-
//
|
|
121
|
+
// The same fold and decision the orchestrator's collector uses, from the disk snapshots.
|
|
122
122
|
const waits = waitsOf(foldWaits(wid, journal), { now, ledger: foldLedger(emptyLedger(), entries), leases: leaseWaits(lstate, now) }, pinnedAgentModel(home, wid, wf.rev));
|
|
123
123
|
const line = (lines, model) => { const provider = model?.split("/")[0]; return provider ? lines?.find(l => l.startsWith(`${provider} `)) : undefined; };
|
|
124
124
|
const latest = [...new Map(wf.calls.map(c => [c.key, c])).values()];
|
|
@@ -140,7 +140,7 @@ function describeWorkflow(home, wid, entries, now) {
|
|
|
140
140
|
return { state, wid, ...(id ? { request: id } : {}), ...(admitted ? { spec_digest: specDigest(admitted) } : {}), ...labelsOf(run), status: wf.status, ...(wf.error ? { error: wf.error } : {}),
|
|
141
141
|
calls, ...(questions.length ? { questions } : {}), ...(attention.length ? { attention } : {}), ...(fence ? { lastFence: fence } : {}) };
|
|
142
142
|
}
|
|
143
|
-
/**
|
|
143
|
+
/** Best effort: why the latest fence that interrupted work happened. The per-execution classification and the
|
|
144
144
|
* reason are shared with the event log's `fenced` events (src/events/fence.ts). */
|
|
145
145
|
export function lastFence(journal, orch) {
|
|
146
146
|
const ended = endedExecs(journal);
|
|
@@ -210,7 +210,7 @@ function pending(ctx, id, json, why = "") {
|
|
|
210
210
|
}
|
|
211
211
|
/** Submit, then wait; a decision whose admitted envelope has other content (another sender won the id) is a conflict. */
|
|
212
212
|
async function submitAndWait(ctx, seen, id, kind, body, cond, wait, json) {
|
|
213
|
-
//
|
|
213
|
+
// `created` is false only when the id was already decided before this invocation submitted (decisions are
|
|
214
214
|
// monotonic); racing first attempts may all report created — the wid is what identifies the run.
|
|
215
215
|
const rid = requestRid(id), earlier = Boolean(await outcome(ctx.home, rid, kind === "run", 0));
|
|
216
216
|
let sent;
|
|
@@ -236,7 +236,7 @@ async function submitAndWait(ctx, seen, id, kind, body, cond, wait, json) {
|
|
|
236
236
|
return { code: await conflict(ctx, id, specDigest(admitted), json) };
|
|
237
237
|
return { sent, outcome: result, earlier };
|
|
238
238
|
}
|
|
239
|
-
/**
|
|
239
|
+
/** A failure once this invocation submitted, or once the id is found recorded with this content, is "not decided
|
|
240
240
|
* yet" (75), never a refusal. Otherwise, with --json, a request refused before submission (a usage error, an invalid
|
|
241
241
|
* spec, an unknown agent, no open question) answers `{request, applied:false, reason, spec_digest?}` with exit 1;
|
|
242
242
|
* spec_digest is present once the content was complete enough to hash. This invocation submitted nothing. */
|
|
@@ -259,7 +259,7 @@ async function refusable(args, ctx, run) {
|
|
|
259
259
|
}
|
|
260
260
|
const forks = (spec) => [spec, ...["tasks", "chain"].flatMap(k => Array.isArray(spec[k]) ? spec[k] : [])]
|
|
261
261
|
.some(s => s && typeof s === "object" && s.context === "fork");
|
|
262
|
-
/**
|
|
262
|
+
/** `run --request <id> --spec <file|-> [--cwd <dir>] [--json] [--wait-ms <n>]`. The spec is the `subagents` run
|
|
263
263
|
* form ({agent,task,…} or {tasks|chain:[…],…}); it is validated by the tool's own normalizer and agent check. */
|
|
264
264
|
export const runCommand = (args, ctx) => refusable(args, ctx, runRequest);
|
|
265
265
|
async function runRequest(args, ctx, seen) {
|
|
@@ -320,7 +320,7 @@ async function widOf(home, head) {
|
|
|
320
320
|
const found = await findRequest(home, rid);
|
|
321
321
|
return found?.request.kind === "run" && decision(ledger(home), rid)?.type !== "rejected" ? { pending: true } : { wid: head };
|
|
322
322
|
}
|
|
323
|
-
/**
|
|
323
|
+
/** `--to <run-id>[/<key>] | <wid>/<key>` (+ `--call <key>`) → `<wid>/<key>`; a run of one call implies its key. */
|
|
324
324
|
async function target(home, to, call, prior) {
|
|
325
325
|
const cut = to.indexOf("/"), head = cut < 0 ? to : to.slice(0, cut), key = cut < 0 ? call : to.slice(cut + 1);
|
|
326
326
|
if (cut >= 0 && call !== undefined)
|
|
@@ -341,7 +341,7 @@ async function target(home, to, call, prior) {
|
|
|
341
341
|
return { to: `${resolved.wid}/${keys[0]}` };
|
|
342
342
|
throw new Error(`${to} has ${keys.length ? `calls ${keys.join(", ")}` : "no calls yet"}; name one with --call <key> or --to <wid>/<key>`);
|
|
343
343
|
}
|
|
344
|
-
/**
|
|
344
|
+
/** `send --request <id> --to <…> --kind follow-up|answer|steer|model [--qid <qid> --rev <n>] --message <text|@file> [--model <m>]`. */
|
|
345
345
|
export const sendCommand = (args, ctx) => refusable(args, ctx, sendRequest);
|
|
346
346
|
async function sendRequest(args, ctx, seen) {
|
|
347
347
|
const { values, positionals } = flags(args, { request: "value", to: "value", call: "value", kind: "value", qid: "value", rev: "value", message: "value", model: "value", json: "flag", "wait-ms": "value" });
|
|
@@ -378,7 +378,7 @@ async function sendRequest(args, ctx, seen) {
|
|
|
378
378
|
return done.code;
|
|
379
379
|
return decided(ctx, id, done, json);
|
|
380
380
|
}
|
|
381
|
-
/**
|
|
381
|
+
/** `stop --request <id> <run-id|wid|wid/key|callId>`. */
|
|
382
382
|
export const stopCommand = (args, ctx) => refusable(args, ctx, stopRequest);
|
|
383
383
|
async function stopRequest(args, ctx, seen) {
|
|
384
384
|
const { values, positionals } = flags(args, { request: "value", json: "flag", "wait-ms": "value" });
|
package/dist/events/cli.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// `events --all [--since <cursor>] [--limit <n>] [--json] [--wait-ms <n>]` — read the cross-workflow event log.
|
|
2
2
|
// Output is JSON lines (with or without --json). Without --since: `{"head","more":false}`. With --since: the events
|
|
3
3
|
// after the cursor (at most --limit, default and max EVENTS_PAGE_MAX), then `{"head","more"}`: more:true → head is the
|
|
4
4
|
// cursor of the last event printed; more:false → the log head. Exit 0; 4 cursor-expired (other epoch, seq below
|
package/dist/events/derive.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Journal → event drafts. Pure functions of durable sources: the entry at journal index i (seq i+1) is derived from
|
|
2
2
|
// that journal's entries up to it and from orchestrator-ledger entries durable before it, so a re-derivation after a
|
|
3
3
|
// crash gives the same events with the same ids (`<wid>:<journal seq>:<type>`, `<wid>:submitted`).
|
|
4
4
|
import { createHash } from "node:crypto";
|
|
@@ -14,7 +14,7 @@ export function callParts(call) {
|
|
|
14
14
|
const echo = (id) => ({ ...(id.request !== undefined ? { request: id.request } : {}), ...(id.labels ? { labels: id.labels } : {}) });
|
|
15
15
|
const onCall = (call) => { const p = callParts(call); return p ? { key: p.key, gen: p.gen, call } : { call }; };
|
|
16
16
|
/** The labels of a run body: a non-empty plain object of strings (anything else, and `{}`, is not echoed — as describe
|
|
17
|
-
* and the
|
|
17
|
+
* and the wait collector treat them). */
|
|
18
18
|
export function labelsOf(body) {
|
|
19
19
|
const labels = body?.labels;
|
|
20
20
|
if (!labels || typeof labels !== "object" || Array.isArray(labels))
|
package/dist/events/fence.js
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Which fences interrupted work, shared by `describe` (lastFence) and the event deriver (`fenced`). Read-only over
|
|
2
2
|
// committed journal entries; `limit` restricts the view to the first `limit` entries (the deriver may read only entries
|
|
3
3
|
// before the one it derives, so a re-derivation after a crash sees the same history).
|
|
4
4
|
import { JT } from "../types.js";
|
|
5
|
-
/**
|
|
5
|
+
/** Executions whose fence did NOT interrupt work: every execution ends with a fence; one interrupted work only when
|
|
6
6
|
* the execution neither settled (its turn ended) before it nor hibernated (it waits for an answer), and was not sealed
|
|
7
7
|
* on purpose: a seal ends an execution on purpose unless its outcome is `unknown` (a `once` call cut off in a tool) or
|
|
8
8
|
* the execution was recorded as lost (the loss bound sealed it), which are interruptions themselves. A seal for an
|
|
@@ -31,7 +31,7 @@ export function endedExecs(journal, limit = journal.length) {
|
|
|
31
31
|
}
|
|
32
32
|
return ended;
|
|
33
33
|
}
|
|
34
|
-
/**
|
|
34
|
+
/** Best effort: why `fence` (a `fenced` entry that interrupted work) happened. restart-force: a forced restart listed
|
|
35
35
|
* the execution as live; orchestrator-crash: the execution was launched before an orchestrator start that is not
|
|
36
36
|
* preceded by a clean exit and fenced after it (startup recovery); otherwise process-died (the child or its host went
|
|
37
37
|
* away, or a drain fenced it). */
|
|
@@ -49,7 +49,7 @@ export function fenceReason(journal, orch, fence) {
|
|
|
49
49
|
}
|
|
50
50
|
return "process-died";
|
|
51
51
|
}
|
|
52
|
-
/**
|
|
52
|
+
/** The fence of `exec` among the first `limit` entries when it interrupted work, else undefined. */
|
|
53
53
|
export function interruptingFence(journal, exec, limit = journal.length) {
|
|
54
54
|
let fence;
|
|
55
55
|
for (let i = limit - 1; i >= 0 && !fence; i--) {
|
package/dist/events/labels.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Caller labels of a run — `run --labels <json>`, the tool's `labels`, and the
|
|
2
2
|
// orchestrator's admission all validate with this one function. Labels are part of RunBody, so they are part of the
|
|
3
3
|
// request's spec_digest (the same request id with other labels is a request-conflict) and are echoed by `describe` and
|
|
4
4
|
// on every event of the workflow.
|
package/dist/events/log.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// The cross-workflow event log, `<home>/events.jsonl`. Single writer: the orchestrator (under its OS lock). Each line
|
|
2
2
|
// is CRC-framed like the journals (`<crc32 hex 8> <json>\n`, kernel/journal.ts); a reader ignores a torn final line, a
|
|
3
3
|
// bad line before the last one is corruption. Records (`k`):
|
|
4
4
|
// log {v, epoch, dropped} first line, written when the file is created or compacted. `epoch` (16 hex chars, random)
|
|
@@ -120,7 +120,7 @@ function scanFile(fd, size) {
|
|
|
120
120
|
throw new LogCorrupt("event log has no header");
|
|
121
121
|
return { epoch: header.epoch, dropped: header.dropped, head: Math.max(head, header.dropped), marks, end: bad ?? end, records, events };
|
|
122
122
|
}
|
|
123
|
-
/**
|
|
123
|
+
/** The writer. Appends, compaction and close are serialized; an append resolves only after fsync. */
|
|
124
124
|
export class EventLog {
|
|
125
125
|
path;
|
|
126
126
|
epoch;
|
package/dist/events/pump.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
|
-
//
|
|
1
|
+
// The orchestrator's event pump. Derives events from the durable sources (workflow journals, the orchestrator
|
|
2
2
|
// ledger's `created`) after their appends and writes them to the event log with the watermarks of the sources derived
|
|
3
3
|
// (same fsync), so a restart re-derives only after them (at least once). It is also the EventSink other producers
|
|
4
|
-
// (
|
|
4
|
+
// (waiting/moving) emit through. One chain serializes passes, emits, compaction and close. A pass that cannot log
|
|
5
5
|
// what it derived retries in the background, and `flush` (prune) rejects; a broken log (a failure after a compaction's
|
|
6
6
|
// rename) is reopened by the next pass or emit.
|
|
7
7
|
// Memory: per unpruned workflow one watermark and one cached identity; reads the journals' committed arrays from their
|
|
@@ -123,7 +123,7 @@ export class EventPump {
|
|
|
123
123
|
async emit(drafts) {
|
|
124
124
|
await this.ready;
|
|
125
125
|
return this.serial(async () => {
|
|
126
|
-
// After close (the orchestrator exits) nothing is logged:
|
|
126
|
+
// After close (the orchestrator exits) nothing is logged: waiting state is derived again by the next orchestrator.
|
|
127
127
|
if (!this.log || this.closed || !drafts.length)
|
|
128
128
|
return;
|
|
129
129
|
const log = await this.writer();
|
|
@@ -209,7 +209,7 @@ export class EventPump {
|
|
|
209
209
|
}
|
|
210
210
|
/** The writer; a broken one (a failure after a compaction's rename) is reopened first: the open applies the start
|
|
211
211
|
* skip. A log it had to create anew (corrupt or gone) is backfilled from everything on disk: the caller's drafts
|
|
212
|
-
* were derived for the old one, so this throws (a pass retries from the cleared watermarks,
|
|
212
|
+
* were derived for the old one, so this throws (a pass retries from the cleared watermarks, waiting/moving on its next tick). */
|
|
213
213
|
async writer() {
|
|
214
214
|
const log = this.log;
|
|
215
215
|
if (!log.broken)
|
package/dist/events/types.js
CHANGED
|
@@ -1,10 +1,9 @@
|
|
|
1
|
-
//
|
|
2
|
-
// batch 2; both leaves build on it. Changing a field here changes the public contract: do it only with the parent.
|
|
1
|
+
// The public shape of the cross-workflow event log. Changing a field here changes the public contract.
|
|
3
2
|
//
|
|
4
3
|
// One durable sequence, single writer (the orchestrator). Each record is one event; `seq` grows strictly (gaps allowed:
|
|
5
4
|
// retention drops records and every orchestrator start skips ahead, see EVENT_SEQ_SKIP). The public cursor is
|
|
6
5
|
// `<epoch>:<seq>`; `--since c` returns events with a larger seq of the same epoch.
|
|
7
|
-
/**
|
|
6
|
+
/** Wait reasons, one per waiting call, in this precedence when several apply (first wins). */
|
|
8
7
|
export const WAIT_REASONS = ["unconfirmed-stop", "provider-exhausted", "writer-lock", "lease", "slot", "silent"];
|
|
9
8
|
/** `data` of a `sealed` event is inlined up to this many bytes of JSON; larger → `data_omitted`. */
|
|
10
9
|
export const EVENT_DATA_INLINE_MAX = 16 * 1024;
|
|
@@ -1,7 +1,7 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Why a call is not moving. One pure decision (`whyWaiting`) from one
|
|
2
2
|
// unsealed call's observable state; `describe` (CLI, from disk snapshots) and the orchestrator's collector (from its
|
|
3
3
|
// in-memory journals) both build that state with the same fold (`foldWaits`) and the same inputs (`waitsOf`), so they
|
|
4
|
-
// agree. The tracker turns the per-tick waits into `waiting`/`moving` event drafts for the log's EventSink; `
|
|
4
|
+
// agree. The tracker turns the per-tick waits into `waiting`/`moving` event drafts for the log's EventSink; `startWaiting`
|
|
5
5
|
// runs it on a timer. No durable state of its own: the log holds what was emitted, and seeds the tracker after a start.
|
|
6
6
|
//
|
|
7
7
|
// Sources, per reason (precedence WAIT_REASONS, first wins):
|
|
@@ -31,7 +31,7 @@ import { leaseCalls, leaseState } from "../platform/lease.js";
|
|
|
31
31
|
import { requestId } from "../requests.js";
|
|
32
32
|
/** A queued call whose providers have a free slot (or are unknown) counts as waiting for a slot only after this long. */
|
|
33
33
|
export const SLOT_GRACE_MS = 3000;
|
|
34
|
-
/**
|
|
34
|
+
/** Why the call does not move, or undefined when it moves (or is asking or sealed). Pure. */
|
|
35
35
|
export function whyWaiting(c, now) {
|
|
36
36
|
if (c.sealed || c.asking)
|
|
37
37
|
return undefined;
|
|
@@ -246,7 +246,7 @@ export function waitInput(f, c, env, agentModel) {
|
|
|
246
246
|
export function liveWaitCalls(f) {
|
|
247
247
|
return [...f.calls.values()].filter(c => !f.done || c.generation);
|
|
248
248
|
}
|
|
249
|
-
/**
|
|
249
|
+
/** The waits of a folded workflow's live calls (calls that move are absent). */
|
|
250
250
|
export function waitsOf(f, env, agentModel) {
|
|
251
251
|
const out = new Map();
|
|
252
252
|
if (!f.calls.size)
|
|
@@ -263,8 +263,8 @@ const base = (m) => ({ wid: m.wid, ...(m.request ? { request: m.request } : {}),
|
|
|
263
263
|
* (also when the call seals or disappears while waiting); a change of detail or age alone emits nothing. Ids carry the
|
|
264
264
|
* observing tick: the same cause can recur with the same `since` (a used-up provider whose probe is refused again keeps
|
|
265
265
|
* its first `since`), and a reader deduplicating on the id must still see it. A draft that failed to log is retried with
|
|
266
|
-
* its id (
|
|
267
|
-
export class
|
|
266
|
+
* its id (startWaiting), and the seed keeps a restarted tracker from repeating a logged transition. */
|
|
267
|
+
export class WaitTracker {
|
|
268
268
|
last = new Map();
|
|
269
269
|
/** Start from the latest `waiting`/`moving` per call id (the log's view at an orchestrator start). */
|
|
270
270
|
seed(latest) {
|
|
@@ -299,11 +299,11 @@ export class R7Tracker {
|
|
|
299
299
|
return out;
|
|
300
300
|
}
|
|
301
301
|
}
|
|
302
|
-
/** Every `intervalMs` (k.
|
|
302
|
+
/** Every `intervalMs` (k.waitCheckMs, default 5000): collect, diff, emit. Never overlaps itself; an emit failure is logged and
|
|
303
303
|
* its drafts are emitted first on the next tick (in order, same ids), so no transition is lost. `tick()` runs one now
|
|
304
304
|
* (or joins the one running). */
|
|
305
|
-
export function
|
|
306
|
-
const tracker = options.tracker ?? new
|
|
305
|
+
export function startWaiting(options) {
|
|
306
|
+
const tracker = options.tracker ?? new WaitTracker(), now = options.now ?? Date.now, log = options.log ?? (line => console.error(line));
|
|
307
307
|
let pending = [], running, stopped = false, timer, epoch = options.epoch?.();
|
|
308
308
|
const once = async () => {
|
|
309
309
|
try {
|
|
@@ -315,7 +315,7 @@ export function startR7(options) {
|
|
|
315
315
|
pending.push(...tracker.diff(at, current, meta));
|
|
316
316
|
}
|
|
317
317
|
catch (error) {
|
|
318
|
-
log(`durable-subagents:
|
|
318
|
+
log(`durable-subagents: wait collection failed: ${String(error)}`);
|
|
319
319
|
}
|
|
320
320
|
while (pending.length) {
|
|
321
321
|
const batch = pending.slice(0, EVENT_SEQ_SKIP);
|
|
@@ -324,7 +324,7 @@ export function startR7(options) {
|
|
|
324
324
|
pending = pending.slice(batch.length);
|
|
325
325
|
}
|
|
326
326
|
catch (error) {
|
|
327
|
-
log(`durable-subagents: ${pending.length}
|
|
327
|
+
log(`durable-subagents: ${pending.length} waiting/moving event(s) not logged, retried next tick: ${String(error)}`);
|
|
328
328
|
return;
|
|
329
329
|
}
|
|
330
330
|
}
|
|
@@ -339,13 +339,13 @@ export function startR7(options) {
|
|
|
339
339
|
schedule();
|
|
340
340
|
return { tick, async stop() { stopped = true; clearTimeout(timer); await running; } };
|
|
341
341
|
}
|
|
342
|
-
/** The waits of the orchestrator's live calls, for `
|
|
343
|
-
* const collect =
|
|
344
|
-
*
|
|
342
|
+
/** The waits of the orchestrator's live calls, for `startWaiting`'s `collect`:
|
|
343
|
+
* const collect = waitCollector({ home, workflows: () => engine.store.workflows.values(), orch: ledgers.orch, config: ledgers.config });
|
|
344
|
+
* startWaiting({ collect: () => collect(), sink, intervalMs: config.k?.waitCheckMs, tracker });
|
|
345
345
|
* Per tick: one `entries()` per workflow (a cached view unless it was appended to), each changed journal folded only
|
|
346
346
|
* past what was folded before; a workflow without unsealed calls costs a map lookup. The ledger is folded the same way
|
|
347
347
|
* (slots, used-up providers, settings, and each workflow's request id and labels). Call it from one place at a time. */
|
|
348
|
-
export function
|
|
348
|
+
export function waitCollector(src) {
|
|
349
349
|
const folds = new Map();
|
|
350
350
|
const ledger = emptyLedger();
|
|
351
351
|
// Run requests not yet created (rid → labels) and the request id and labels of each workflow: bounded by the
|
package/dist/kernel/lifecycle.js
CHANGED
|
@@ -35,7 +35,7 @@ export function planDecisions(records, candidates, decide) {
|
|
|
35
35
|
if (bound) {
|
|
36
36
|
if (bound.hash === hash)
|
|
37
37
|
envelopes.set(req.rid, req);
|
|
38
|
-
// A1
|
|
38
|
+
// A1: a conflicting duplicate of a resolved rid is only dropped (its file still goes); it never records a decision.
|
|
39
39
|
else if (!view.resolved.has(req.rid) && !conflicts.has(req.rid)) {
|
|
40
40
|
conflicts.add(req.rid);
|
|
41
41
|
emit({ type: 'rejected', rid: req.rid, reason: 'identity-conflict' });
|
|
@@ -3,14 +3,14 @@
|
|
|
3
3
|
// Orchestrator ledger entries: config{hash,config} — the settings in effect from then on (at start, or after a change);
|
|
4
4
|
// config-rejected{hash,error} — a changed file that was not applied; the settings before it stay in effect.
|
|
5
5
|
// A reload changes the shared config object in place: every later read sees it (the next slot acquisition, model
|
|
6
|
-
// resolution or check). Slots already held are kept when a limit drops; timers of running executions keep their period, and the
|
|
7
|
-
// period (k.
|
|
6
|
+
// resolution or check). Slots already held are kept when a limit drops; timers of running executions keep their period, and the waiting check
|
|
7
|
+
// period (k.waitCheckMs) is read once at orchestrator start.
|
|
8
8
|
import { readFile } from "node:fs/promises";
|
|
9
9
|
import { join } from "node:path";
|
|
10
10
|
import { contentHash } from "../kernel/ids.js";
|
|
11
11
|
/** The keys the orchestrator reads; config.json also holds pi-side settings (ui, onQuit) that it ignores. */
|
|
12
12
|
const KEYS = ["defaultModel", "pools", "providers", "memory", "writerLock", "k"];
|
|
13
|
-
const K = ["lossBound", "checkpointMs", "stallMs", "progressMs", "switchTimeoutMs", "idleExitMs", "trackerMs", "hibernateMs", "spawnBudget", "probeMs", "eventRetentionMs", "
|
|
13
|
+
const K = ["lossBound", "checkpointMs", "stallMs", "progressMs", "switchTimeoutMs", "idleExitMs", "trackerMs", "hibernateMs", "spawnBudget", "probeMs", "eventRetentionMs", "waitCheckMs"];
|
|
14
14
|
export const configPath = (home) => join(home, "config.json");
|
|
15
15
|
/** The orchestrator's part of a parsed config.json. */
|
|
16
16
|
export function orchestratorSettings(raw) {
|
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
// stop-requested {rid,call?} marks a call or workflow stop as taking effect, so its replay is applied, not already-sealed.
|
|
10
10
|
// Orch entry pruned {rid,wid,endedAt,bytes,status,request?,spec_digest?} is the decisive record of a prune: appended before
|
|
11
11
|
// the journal handle is closed and w/<wid> and its staging dirs are removed (bytes = footprint measured just before;
|
|
12
|
-
// status = the final workflow status; request/spec_digest when the workflow was created by a request id,
|
|
12
|
+
// status = the final workflow status; request/spec_digest when the workflow was created by a request id, request-id tombstone).
|
|
13
13
|
// Nothing is rewritten (A1): the admitted request and created entries stay, so a retried id still resolves to this wid.
|
|
14
14
|
import { watch } from 'node:fs';
|
|
15
15
|
import { mkdir, readdir, unlink } from 'node:fs/promises';
|
|
@@ -28,7 +28,7 @@ import { formatUsage, holdOf, refusedResult, snapshotFromEntries } from "./snaps
|
|
|
28
28
|
import { validateCallSpec } from "../compat/spec.js";
|
|
29
29
|
import { EventPump } from "../events/pump.js";
|
|
30
30
|
import { labelsProblem } from "../events/labels.js";
|
|
31
|
-
import {
|
|
31
|
+
import { WaitTracker, waitCollector, startWaiting } from "../events/waiting.js";
|
|
32
32
|
import { readPage } from "../events/log.js";
|
|
33
33
|
import { eventsLog } from "../paths.js";
|
|
34
34
|
import { parseModel } from "../compat/model.js";
|
|
@@ -97,9 +97,9 @@ export function finishedText(wid, entries, call) {
|
|
|
97
97
|
/** A1, P2, P10, P11: Serialize decisions while executions run independently. */
|
|
98
98
|
export class Engine {
|
|
99
99
|
store;
|
|
100
|
-
/**
|
|
100
|
+
/** The cross-workflow event log's writer (derives milestones from the journals; waiting/moving emits through it). */
|
|
101
101
|
events;
|
|
102
|
-
|
|
102
|
+
waiting;
|
|
103
103
|
ledgers;
|
|
104
104
|
executor;
|
|
105
105
|
evaluator;
|
|
@@ -152,7 +152,7 @@ export class Engine {
|
|
|
152
152
|
/** A2, P10: Recover executor authority before replaying each unfinished workflow. */
|
|
153
153
|
async recover() {
|
|
154
154
|
await this.store.recover();
|
|
155
|
-
//
|
|
155
|
+
// Before any recovery append, so every journal entry from here on is derived promptly (and a new log derives
|
|
156
156
|
// everything still on disk).
|
|
157
157
|
await this.events.open();
|
|
158
158
|
for (const wf of this.store.workflows.values()) {
|
|
@@ -186,14 +186,14 @@ export class Engine {
|
|
|
186
186
|
this.dispatchGeneration(wf, entry);
|
|
187
187
|
}
|
|
188
188
|
await this.intake();
|
|
189
|
-
this.
|
|
189
|
+
this.startWaiting();
|
|
190
190
|
}
|
|
191
|
-
/**
|
|
191
|
+
/** Every k.waitCheckMs (read here, at orchestrator start), why each live call does not move, as `waiting`/`moving`
|
|
192
192
|
* events through the pump. The tracker starts from the log's latest transition per call, so a restart repeats none
|
|
193
193
|
* (and a call that ended or started moving meanwhile gets its `moving`). The seed reads from seq 0: retention keeps
|
|
194
|
-
* events below `dropped`. A seed that cannot be read is logged and
|
|
195
|
-
|
|
196
|
-
if (this.closed || this.
|
|
194
|
+
* events below `dropped`. A seed that cannot be read is logged and the tracker starts empty (recovery never fails on it). */
|
|
195
|
+
startWaiting() {
|
|
196
|
+
if (this.closed || this.waiting)
|
|
197
197
|
return;
|
|
198
198
|
const latest = new Map(), head = this.events.head;
|
|
199
199
|
try {
|
|
@@ -211,13 +211,13 @@ export class Engine {
|
|
|
211
211
|
}
|
|
212
212
|
}
|
|
213
213
|
catch (error) {
|
|
214
|
-
console.error(`durable-subagents:
|
|
214
|
+
console.error(`durable-subagents: waiting seed from the event log failed, starting without it: ${String(error)}`);
|
|
215
215
|
latest.clear();
|
|
216
216
|
}
|
|
217
|
-
const tracker = new
|
|
217
|
+
const tracker = new WaitTracker();
|
|
218
218
|
tracker.seed(latest);
|
|
219
|
-
const collect =
|
|
220
|
-
this.
|
|
219
|
+
const collect = waitCollector({ home: this.ledgers.home, workflows: () => this.store.workflows.values(), orch: this.ledgers.orch, config: this.ledgers.config });
|
|
220
|
+
this.waiting = startWaiting({ collect: () => collect(), sink: this.events, intervalMs: this.ledgers.config.k?.waitCheckMs, tracker, epoch: () => this.events.head?.epoch });
|
|
221
221
|
}
|
|
222
222
|
async startHost() {
|
|
223
223
|
await this.evaluator.start(message => this.background(() => this.message(message)), () => this.background(async () => {
|
|
@@ -342,7 +342,7 @@ export class Engine {
|
|
|
342
342
|
}
|
|
343
343
|
if (req.kind === 'run') {
|
|
344
344
|
const created = this.ledgers.orch.entries().find(e => e.type === JT.created && e.rid === req.rid);
|
|
345
|
-
//
|
|
345
|
+
// Senders validate labels; a hand-written request must not bypass that.
|
|
346
346
|
const labels = req.body?.labels, invalid = !created && labels !== undefined ? labelsProblem(labels) : undefined;
|
|
347
347
|
if (invalid)
|
|
348
348
|
return { action: 'reject', reason: `invalid-labels: ${invalid}` };
|
|
@@ -590,7 +590,7 @@ export class Engine {
|
|
|
590
590
|
const createdBy = String(entries.find(e => e.type === JT.created && e.wid === wf.wid)?.rid ?? '');
|
|
591
591
|
const admitted = requestId(createdBy) !== undefined ? entries.find(e => e.type === 'request' && e.request.rid === createdBy)?.request : undefined;
|
|
592
592
|
const identity = admitted ? { request: requestId(createdBy), spec_digest: specDigest(admitted) } : {};
|
|
593
|
-
//
|
|
593
|
+
// Its events are derived and logged before the journal can go (recovery removes it once `pruned` is committed);
|
|
594
594
|
// when they cannot be, the prune is rejected and the caller retries later.
|
|
595
595
|
try {
|
|
596
596
|
await this.events.flush();
|
|
@@ -857,7 +857,7 @@ export class Engine {
|
|
|
857
857
|
this.closed = true;
|
|
858
858
|
this.watcher?.close();
|
|
859
859
|
clearInterval(this.poll);
|
|
860
|
-
await this.
|
|
860
|
+
await this.waiting?.stop();
|
|
861
861
|
await this.queue;
|
|
862
862
|
await this.evaluator.close();
|
|
863
863
|
await this.executor.shutdown();
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// Restart guards shared by the orchestrator and the legacy client. The caller owns the launch gate;
|
|
2
2
|
// these checks never change durable state. A1: only an accepted restart is appended to the ledger.
|
|
3
3
|
import { contentHash } from "../kernel/ids.js";
|
|
4
|
-
import {
|
|
4
|
+
import { holdDetail, leaseState, who } from "../platform/lease.js";
|
|
5
5
|
export const subagentRestartError = "a subagent cannot force a restart: it would fence itself and other sessions' work; ask the user";
|
|
6
6
|
export const restartToken = (live) => contentHash(live.map(l => l.exec).sort()).slice(0, 12);
|
|
7
7
|
/** Old senders' boolean force is read only to refuse it when executions are live. */
|
|
@@ -27,7 +27,17 @@ export function restartRefusal(home, live, body, tool = false, now = Date.now())
|
|
|
27
27
|
const token = restartToken(live);
|
|
28
28
|
if (body.token === token)
|
|
29
29
|
return undefined;
|
|
30
|
-
|
|
30
|
+
// What a fence would cut short: each lease a listed call holds, with its age and command. Holders outside these
|
|
31
|
+
// executions (a shell, a systemd unit, another call) keep their lease across the restart; they are listed so the
|
|
32
|
+
// machine is not mistaken for free.
|
|
33
|
+
const leases = new Map(), others = [], calls = new Set(live.map(l => l.callId));
|
|
34
|
+
for (const { resource, holders } of leaseState(home))
|
|
35
|
+
for (const t of holders) {
|
|
36
|
+
if (t.call && calls.has(t.call))
|
|
37
|
+
leases.set(t.call, [leases.get(t.call), `holds lease ${resource} (${holdDetail(t, now)})`].filter(Boolean).join(", "));
|
|
38
|
+
else
|
|
39
|
+
others.push(` ${resource} held by ${who(t)} (${holdDetail(t, now)})`);
|
|
40
|
+
}
|
|
31
41
|
const groups = new Map();
|
|
32
42
|
for (const l of live) {
|
|
33
43
|
const origin = l.origin ?? "unknown";
|
|
@@ -40,6 +50,7 @@ export function restartRefusal(home, live, body, tool = false, now = Date.now())
|
|
|
40
50
|
...[...groups].sort(([a], [b]) => a.localeCompare(b)).flatMap(([origin, executions]) => [
|
|
41
51
|
`${origin}:`, ...executions.map(l => ` ${l.wid}/${l.key} ${age(now - l.since)}${l.phase === "gate" ? " gate" : ""}${leases.has(l.callId) ? ` ${leases.get(l.callId)}` : ""}`),
|
|
42
52
|
]),
|
|
53
|
+
...(others.length ? ["not fenced (a restart leaves these leases held):", ...others] : []),
|
|
43
54
|
`token: ${token}`,
|
|
44
55
|
tool ? `to fence exactly these: subagents {action:"restart", force:"${token}", reason:"<why>"}` : `to fence exactly these: pi-durable-subagents restart --force ${token} --reason "<why>"`,
|
|
45
56
|
].join("\n");
|
|
@@ -543,7 +543,7 @@ export function runningOrchestrator(home) {
|
|
|
543
543
|
ledgerStates.set(path, state);
|
|
544
544
|
return orchestratorView(state);
|
|
545
545
|
}
|
|
546
|
-
/** One `exhausted` line of the status views (
|
|
546
|
+
/** One `exhausted` line of the status views (the `provider-exhausted` wait uses it as its detail). */
|
|
547
547
|
export function exhaustedLine(provider, x, now) {
|
|
548
548
|
return `${provider} exhausted since ${age(now - x.since)} ago (${clip(x.error, 80)}), ` +
|
|
549
549
|
(x.probe ? `probing with ${x.probe.split("#")[0]}` : x.nextTry > now ? `next try in ${age(x.nextTry - now)}` : "next call probes it");
|
|
@@ -180,7 +180,7 @@ export async function diskUsage(path) {
|
|
|
180
180
|
/** A1, P11: Own shared workflow handles and reconcile create intents after a crash. */
|
|
181
181
|
export class Store {
|
|
182
182
|
workflows = new Map();
|
|
183
|
-
/**
|
|
183
|
+
/** Called after each append to a workflow journal (the event pump derives from it). */
|
|
184
184
|
appended;
|
|
185
185
|
ledgers;
|
|
186
186
|
constructor(ledgers) { this.ledgers = ledgers; }
|
package/dist/paths.js
CHANGED
|
@@ -6,7 +6,7 @@ import { ENV } from "./types.js";
|
|
|
6
6
|
export const dsaHome = (env = process.env) => env[ENV.home] || path.join(os.homedir(), ".pi", "durable-subagents");
|
|
7
7
|
export const orchLedger = (home) => path.join(home, "orchestrator.jsonl");
|
|
8
8
|
export const orchLock = (home) => path.join(home, "orchestrator.lock");
|
|
9
|
-
/**
|
|
9
|
+
/** The cross-workflow event log (single writer: the orchestrator; read by `events --all`). */
|
|
10
10
|
export const eventsLog = (home) => path.join(home, "events.jsonl");
|
|
11
11
|
/** Executables for subagents: the orchestrator writes a `pi-durable-subagents` shim here and children get it on PATH. */
|
|
12
12
|
export const binDir = (home) => path.join(home, "bin");
|
package/dist/platform/lease.js
CHANGED
|
@@ -201,12 +201,13 @@ const clip = (s, n) => s.length > n ? `${s.slice(0, n - 1)}…` : s;
|
|
|
201
201
|
export const callAddress = (call) => { const m = /^([^@/]+)@\d+\/(.+)@\d+$/.exec(call); return m ? `${m[1]}/${m[2]}` : call; };
|
|
202
202
|
/** Who holds or waits: the subagent call when there is one, else the pid and command. */
|
|
203
203
|
export const who = (t) => t.call ? callAddress(t.call) : `pid ${t.wrapper.pid} \`${clip(t.argv.join(" "), 60)}\``;
|
|
204
|
+
/** A granted ticket's mode, how long it has been held, the command (unless `who` already names it) and the note:
|
|
205
|
+
* "exclusive, 12m, `make bench`, nightly". */
|
|
206
|
+
export const holdDetail = (t, now = Date.now(), command = !!t.call) => `${t.mode}, ${age(now - (t.grantedAt ?? t.since))}${command ? `, \`${clip(t.argv.join(" "), 60)}\`` : ""}${t.note ? `, ${clip(t.note, 80)}` : ""}`;
|
|
204
207
|
/** One status line per resource: "machine held by <who> (exclusive, 12m, `make bench`); waiting: <who> 3m, …". */
|
|
205
208
|
export function leaseLines(state, now = Date.now()) {
|
|
206
209
|
return state.map(({ resource, holders, waiters }) => {
|
|
207
|
-
const held = holders.length
|
|
208
|
-
? `held by ${holders.map(t => `${who(t)} (${t.mode}, ${age(now - (t.grantedAt ?? t.since))}${t.call ? `, \`${clip(t.argv.join(" "), 60)}\`` : ""}${t.note ? `, ${clip(t.note, 80)}` : ""})`).join(", ")}`
|
|
209
|
-
: "free";
|
|
210
|
+
const held = holders.length ? `held by ${holders.map(t => `${who(t)} (${holdDetail(t, now)})`).join(", ")}` : "free";
|
|
210
211
|
return `${resource} ${held}${waiters.length ? `; waiting: ${waiters.map(t => `${who(t)} ${t.mode === "shared" ? "shared " : ""}${age(now - t.since)}`).join(", ")}` : ""}`;
|
|
211
212
|
});
|
|
212
213
|
}
|
package/dist/requests.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
//
|
|
1
|
+
// Caller-chosen request ids. A program names a run/send/stop with `<id>`; its kernel rid is `req:<id>` (fits the
|
|
2
2
|
// mailbox rid pattern), unique per DSA_HOME across kinds and senders. A retry with the same id and the same content
|
|
3
3
|
// (spec_digest) gets the first outcome; a different content is a request-conflict and is never published.
|
|
4
4
|
// The check-and-send is serialized per home by the OS lock <home>/requests.lock (CLI and tool senders alike); no
|
|
@@ -13,15 +13,15 @@ import { OsLock } from "./platform/lock.js";
|
|
|
13
13
|
import { orchInbox, orchLedger, outboxRoot } from "./paths.js";
|
|
14
14
|
export const REQUEST_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,123}$/;
|
|
15
15
|
const PREFIX = 'req:';
|
|
16
|
-
/**
|
|
16
|
+
/** `<id>` → its rid `req:<id>`; ids are 1–124 chars `[A-Za-z0-9][A-Za-z0-9._:-]*`. */
|
|
17
17
|
export function requestRid(id) {
|
|
18
18
|
if (!REQUEST_ID.test(id))
|
|
19
19
|
throw new Error(`invalid request id ${JSON.stringify(id)}: 1-124 characters [A-Za-z0-9][A-Za-z0-9._:-]*`);
|
|
20
20
|
return PREFIX + id;
|
|
21
21
|
}
|
|
22
|
-
/**
|
|
22
|
+
/** The request id of a `req:<id>` rid; undefined for any other rid (ULIDs never contain ':'). */
|
|
23
23
|
export function requestId(rid) { return rid.startsWith(PREFIX) ? rid.slice(PREFIX.length) : undefined; }
|
|
24
|
-
/**
|
|
24
|
+
/** Spec_digest = contentHash({kind, body, cond}) (cond omitted when absent). A run's body.origin (the pi session
|
|
25
25
|
* branch offered for context:"fork") is delivery metadata, not spec: it differs on every turn, so it is not hashed. */
|
|
26
26
|
export function specDigest(req) {
|
|
27
27
|
let body = req.body;
|
|
@@ -31,7 +31,7 @@ export function specDigest(req) {
|
|
|
31
31
|
}
|
|
32
32
|
return contentHash({ kind: req.kind, body, cond: req.cond });
|
|
33
33
|
}
|
|
34
|
-
/**
|
|
34
|
+
/** The envelope recorded for `rid`: the orchestrator's admitted copy (ledger `request`, kept after prune), else any
|
|
35
35
|
* sender's outbox `sent` entry (published or about to be). Read-only. */
|
|
36
36
|
export async function findRequest(home, rid) {
|
|
37
37
|
const admitted = readJournalSnapshot(orchLedger(home)).find(e => e.type === 'request' && e.request.rid === rid);
|
|
@@ -51,7 +51,7 @@ export async function findRequest(home, rid) {
|
|
|
51
51
|
export class RequestsBusy extends Error {
|
|
52
52
|
name = 'RequestsBusy';
|
|
53
53
|
}
|
|
54
|
-
/**
|
|
54
|
+
/** P5: Check-then-send under the home-wide request-id lock (innermost: taken after a sender's own lock), so no two
|
|
55
55
|
* senders publish one rid and no second envelope with an existing rid and other content reaches the inbox (it would
|
|
56
56
|
* stall its sender's sequence). Same content: the recorded envelope stands (republished when it is this sender's and
|
|
57
57
|
* pending; `sent` false). */
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-durable-subagents",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.25",
|
|
4
4
|
"description": "Subagents for pi that never lose work and never do it twice. Crash-safe workflows, automatic recovery, and a live view just like the main agent.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|