@agfpd/iapeer 0.4.24 → 0.4.26
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -1
- package/README.md +15 -9
- package/package.json +13 -7
- package/src/connect/index.ts +17 -0
- package/src/daemon/index.ts +22 -5
- package/src/daemon/main.ts +14 -0
- package/src/lifecycle/index.ts +178 -154
- package/src/transport/index.ts +180 -69
package/CHANGELOG.md
CHANGED
|
@@ -10,6 +10,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
10
10
|
|
|
11
11
|
## [Unreleased]
|
|
12
12
|
|
|
13
|
+
## [0.4.25] - 2026-06-22
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
|
|
17
|
+
- **`connect telegram` now eager-reindexes the registry** (#1). `interface bot` merges
|
|
18
|
+
interfaces.telegram into the per-cwd profile (source of truth), but the derived registry
|
|
19
|
+
(peers-profiles.json) is a projection that only refreshed on specific verbs — so until
|
|
20
|
+
then the peer's telegram FACE was invisible to the sender-guard (`hasTelegramPresence`
|
|
21
|
+
reads the REGISTRY passport), **blocking the peer's outbound telegram** (observed live:
|
|
22
|
+
impact-finder "no telegram face"). connect now calls `reindexFromLocals` after a
|
|
23
|
+
successful `interface bot` (best-effort), so the face lands in the registry immediately —
|
|
24
|
+
no manual `iapeer verify --fix`.
|
|
25
|
+
- **Launchd-revive delivery retry** (#2): a MISS on a launchd-managed target — which the
|
|
26
|
+
daemon can't wake (H4) but launchd KeepAlive revives in ~1s — no longer fails terminally
|
|
27
|
+
in ~16ms. During a router restart (on `connect` / `iapeer update`) an in-flight delivery
|
|
28
|
+
was lost (observed: `natalya→arthur ok=false ms=16` mid-restart — a message to a HUMAN).
|
|
29
|
+
routeSend now retries-resolve for a bounded window (`IAP_LAUNCHD_REVIVE_GRACE_MS`, ~4s)
|
|
30
|
+
and delivers the instant the target revives; it re-resolves each poll (verify-before-act,
|
|
31
|
+
so the ok reflects a CONFIRMED delivery — no false-OK), and a genuinely-down peer still
|
|
32
|
+
fails LOUD and retryable (never silent). `isLaunchdManaged` injected via deps (no
|
|
33
|
+
transport→lifecycle coupling).
|
|
34
|
+
|
|
13
35
|
## [0.4.24] - 2026-06-22
|
|
14
36
|
|
|
15
37
|
### Fixed
|
|
@@ -366,7 +388,8 @@ Initial public release — the foundation core of the iapeer multi-agent ecosyst
|
|
|
366
388
|
- **Heterogeneous peers** — AI agents (Claude, Codex), humans (Telegram), and services
|
|
367
389
|
(timer = cron, watcher = event) are all first-class and addressed the same way.
|
|
368
390
|
|
|
369
|
-
[Unreleased]: https://github.com/agfpd/iapeer/compare/v0.4.
|
|
391
|
+
[Unreleased]: https://github.com/agfpd/iapeer/compare/v0.4.25...HEAD
|
|
392
|
+
[0.4.25]: https://github.com/agfpd/iapeer/compare/v0.4.24...v0.4.25
|
|
370
393
|
[0.4.24]: https://github.com/agfpd/iapeer/compare/v0.4.23...v0.4.24
|
|
371
394
|
[0.4.23]: https://github.com/agfpd/iapeer/compare/v0.4.22...v0.4.23
|
|
372
395
|
[0.4.22]: https://github.com/agfpd/iapeer/compare/v0.4.21...v0.4.22
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# iapeer
|
|
2
2
|
|
|
3
|
-
**
|
|
3
|
+
**Run Claude Code and Codex CLI as local background agents that message each other, wake on demand, and share one memory.**
|
|
4
4
|
|
|
5
5
|
[](https://github.com/agfpd/iapeer/actions/workflows/ci.yml)
|
|
6
6
|
[](https://www.npmjs.com/package/@agfpd/iapeer)
|
|
@@ -32,7 +32,7 @@ recipient peer — asleep? the daemon wakes it and delivers on wake
|
|
|
32
32
|
## What makes it different
|
|
33
33
|
|
|
34
34
|
- **Real interactive CLI sessions, not headless one-shots.** Each AI peer is a full Claude Code or Codex CLI session, not a disposable `claude -p` / `codex exec` call.
|
|
35
|
-
- **Warm
|
|
35
|
+
- **Warm on demand, always reachable.** A live session answers fast; when it goes idle the daemon parks it and brings it back — context resumed — on the next message. Availability never depends on a human keeping the session open. [Lifecycle and daemon](docs/04-lifecycle-and-daemon.md).
|
|
36
36
|
- **One identity across runtimes, with one memory.** A peer's `personality` is decoupled from its runtime: the same peer runs on Claude or Codex, switched with a command. By default (set up by `iapeer onboard`) it carries one shared memory (iapeer-memory) keyed to the personality, not the runtime — one personality, one memory, whatever runtime it runs on.
|
|
37
37
|
- **Heterogeneous peers, one protocol.** AI agents, humans (Telegram), and services (timer = cron, watcher = event) are all first-class, addressed the same way over an always-on MCP daemon that exposes exactly one tool, `send_to_peer`.
|
|
38
38
|
|
|
@@ -58,16 +58,22 @@ iapeer onboard
|
|
|
58
58
|
|
|
59
59
|
Either path runs `onboard` interactively: it surfaces the security checks (runtime sign-in, macOS Full Disk Access) and asks before installing the memory provider — nothing is auto-approved behind your back. You can read [`install.sh`](./install.sh) before running it, or preview every step with `IAPEER_INSTALL_DRYRUN=1`.
|
|
60
60
|
|
|
61
|
-
**Then create your
|
|
61
|
+
**Then create your agents and put them in touch.** A Claude peer and a Codex peer talk as equals — create one on each runtime:
|
|
62
62
|
|
|
63
63
|
```sh
|
|
64
|
-
# create
|
|
65
|
-
iapeer create
|
|
64
|
+
# create two peers, one per runtime
|
|
65
|
+
iapeer create architect --runtime claude
|
|
66
|
+
iapeer create implementer --runtime codex
|
|
66
67
|
|
|
67
|
-
#
|
|
68
|
-
iapeer
|
|
68
|
+
# message one from your terminal
|
|
69
|
+
iapeer send implementer --message "Draft the parser API; the architect will review."
|
|
70
|
+
|
|
71
|
+
# or join a peer's live session — detach with Ctrl-] (or just close it; the agent keeps running)
|
|
72
|
+
iapeer attach architect
|
|
69
73
|
```
|
|
70
74
|
|
|
75
|
+
Inside a peer's own session the agent reaches others with `send_to_peer(<name>, <text>)`; `iapeer send` is the terminal equivalent.
|
|
76
|
+
|
|
71
77
|
**Already have a project folder?** In step 3, use `init` instead of `create`: `cd` into the project and register that folder itself as a peer.
|
|
72
78
|
|
|
73
79
|
```sh
|
|
@@ -79,11 +85,11 @@ The peer's name is the folder name, normalized (`myagent`). `create` sets up a f
|
|
|
79
85
|
|
|
80
86
|
Two ways to work with an agent locally in your terminal:
|
|
81
87
|
|
|
82
|
-
- **Join its live session** — `iapeer attach
|
|
88
|
+
- **Join its live session** — `iapeer attach architect`. The daemon wakes the peer first if it is asleep. Detach with `Ctrl-]` — or just close the terminal — and the session keeps running in the background; reattach with `iapeer attach` whenever you need it.
|
|
83
89
|
- **Open a fresh session in the peer's folder** — go to the peer's directory and launch its runtime:
|
|
84
90
|
|
|
85
91
|
```sh
|
|
86
|
-
cd ~/.iapeer/peers/
|
|
92
|
+
cd ~/.iapeer/peers/architect
|
|
87
93
|
iapeer claude
|
|
88
94
|
```
|
|
89
95
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agfpd/iapeer",
|
|
3
|
-
"version": "0.4.
|
|
4
|
-
"description": "
|
|
3
|
+
"version": "0.4.26",
|
|
4
|
+
"description": "Run Claude Code and Codex CLI as local background agents that message each other, wake on demand, and share one memory.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"author": "agfpd",
|
|
7
7
|
"homepage": "https://github.com/agfpd/iapeer#readme",
|
|
@@ -13,12 +13,18 @@
|
|
|
13
13
|
"url": "https://github.com/agfpd/iapeer/issues"
|
|
14
14
|
},
|
|
15
15
|
"keywords": [
|
|
16
|
-
"
|
|
17
|
-
"
|
|
16
|
+
"claude-code",
|
|
17
|
+
"codex-cli",
|
|
18
|
+
"codex",
|
|
18
19
|
"mcp",
|
|
19
|
-
"
|
|
20
|
-
"
|
|
21
|
-
"
|
|
20
|
+
"shared-memory",
|
|
21
|
+
"peer-to-peer",
|
|
22
|
+
"background-agents",
|
|
23
|
+
"ai-agents",
|
|
24
|
+
"multi-agent",
|
|
25
|
+
"telegram",
|
|
26
|
+
"macos",
|
|
27
|
+
"launchd"
|
|
22
28
|
],
|
|
23
29
|
"type": "module",
|
|
24
30
|
"bin": {
|
package/src/connect/index.ts
CHANGED
|
@@ -25,6 +25,7 @@ import { spawnSync } from 'child_process'
|
|
|
25
25
|
import { normalizeIntelligenceValue } from '../core/constants.ts'
|
|
26
26
|
import { runtimeRoot } from '../storage/index.ts'
|
|
27
27
|
import { findPeer, readPeersIndex } from '../registry/index.ts'
|
|
28
|
+
import { reindexFromLocals } from '../identity/profileStandard.ts'
|
|
28
29
|
import { readRuntimeManifest } from '../runtime/index.ts'
|
|
29
30
|
|
|
30
31
|
export interface TgRunResult {
|
|
@@ -229,6 +230,22 @@ export async function connectTelegram(opts: ConnectTelegramOptions): Promise<Con
|
|
|
229
230
|
return { state: 'interface-failed', peer, username, detail: (iface.stderr || iface.stdout || `exit ${iface.status}`).trim() }
|
|
230
231
|
}
|
|
231
232
|
|
|
233
|
+
// (2b) EAGER reindex — `interface bot` merged interfaces.telegram into the per-cwd
|
|
234
|
+
// profile (source of truth), but the derived registry (peers-profiles.json) is a
|
|
235
|
+
// PROJECTION that only refreshes on specific verbs (verify --fix, default-runtime…).
|
|
236
|
+
// Until then the peer's telegram FACE is INVISIBLE to the sender-guard
|
|
237
|
+
// (hasTelegramPresence reads the REGISTRY passport) — so the peer's OUTBOUND telegram
|
|
238
|
+
// is blocked ("no telegram face") + list/attach are stale. Project it now (same
|
|
239
|
+
// reindexFromLocals the default-runtime verb uses). Best-effort: a reindex hiccup must
|
|
240
|
+
// not fail a connect whose profile + bot are already in place (registry catches up on
|
|
241
|
+
// the next reindex verb). Runs on EVERY connect, incl. the idempotent no-op below — so
|
|
242
|
+
// a registry that drifted on a prior (pre-fix) connect self-heals here too.
|
|
243
|
+
try {
|
|
244
|
+
await reindexFromLocals({ env })
|
|
245
|
+
} catch {
|
|
246
|
+
/* registry lazily catches up on the next reindex verb; profile + bot are correct */
|
|
247
|
+
}
|
|
248
|
+
|
|
232
249
|
// (3) Idempotency gate: the SAME token leaves bots/<alias>/.env byte-stable →
|
|
233
250
|
// the live poller already loaded these credentials → NO restart, clean no-op.
|
|
234
251
|
const after = readBotEnv(alias, env)
|
package/src/daemon/index.ts
CHANGED
|
@@ -194,6 +194,10 @@ export interface CallToolDeps {
|
|
|
194
194
|
noteLiveTopic?: (identity: string, topic: string) => void
|
|
195
195
|
/** Busy-human-composer queue — see transport.createComposerDeliveryQueue. */
|
|
196
196
|
composerQueue?: ComposerQueueRouteDeps
|
|
197
|
+
/** H4 launchd-managed detector — see transport.RouteDeps.isLaunchdManaged (the
|
|
198
|
+
* launchd-revive delivery retry). Injected so the daemon never imports lifecycle;
|
|
199
|
+
* the production main wires the real isLaunchdManaged. */
|
|
200
|
+
isLaunchdManaged?: (personality: string) => boolean
|
|
197
201
|
}
|
|
198
202
|
|
|
199
203
|
export async function callTool(
|
|
@@ -216,6 +220,7 @@ export async function callTool(
|
|
|
216
220
|
ephemeral: deps.ephemeral,
|
|
217
221
|
noteLiveTopic: deps.noteLiveTopic,
|
|
218
222
|
composerQueue: deps.composerQueue,
|
|
223
|
+
isLaunchdManaged: deps.isLaunchdManaged,
|
|
219
224
|
})
|
|
220
225
|
// Ф-#8a: ONE durable outcome line per delivery attempt (delivery.log, sibling
|
|
221
226
|
// to lifecycle.log) — metadata only, never the body. Both branches of the
|
|
@@ -348,7 +353,7 @@ export interface StartDaemonOptions extends StorageOptions {
|
|
|
348
353
|
* owns fleet supervision (idle-reap / zombie-sweep) instead of launchd. The
|
|
349
354
|
* caller wires `tick: () => superviseTick(cfg)` (H4-guarded inside).
|
|
350
355
|
*/
|
|
351
|
-
supervise?: { intervalMs: number; tick: () => void | Promise<void
|
|
356
|
+
supervise?: { intervalMs: number; tick: () => void | Promise<void>; onError?: (err: unknown) => void }
|
|
352
357
|
/**
|
|
353
358
|
* Optional bearer token (H8). When set, EVERY request must carry
|
|
354
359
|
* `Authorization: Bearer <token>` or it is rejected 401 BEFORE any MCP dispatch.
|
|
@@ -392,6 +397,9 @@ export interface StartDaemonOptions extends StorageOptions {
|
|
|
392
397
|
* senders on shutdown/restart.
|
|
393
398
|
*/
|
|
394
399
|
composerQueue?: ComposerQueueRouteDeps
|
|
400
|
+
/** H4 launchd-managed detector — see CallToolDeps.isLaunchdManaged / the launchd-revive
|
|
401
|
+
* delivery retry. OFF by default (library/tests); production main wires the real one. */
|
|
402
|
+
isLaunchdManaged?: (personality: string) => boolean
|
|
395
403
|
/**
|
|
396
404
|
* Write the discovery file (router.json) at <root>/state/iapeer/router.json with
|
|
397
405
|
* the active addresses `{sock, tcp}` — atomically on listen, removed on close.
|
|
@@ -426,6 +434,7 @@ export async function startDaemon(opts: StartDaemonOptions = {}): Promise<Daemon
|
|
|
426
434
|
ephemeral: opts.ephemeral,
|
|
427
435
|
noteLiveTopic: opts.noteLiveTopic,
|
|
428
436
|
composerQueue: opts.composerQueue,
|
|
437
|
+
isLaunchdManaged: opts.isLaunchdManaged,
|
|
429
438
|
})
|
|
430
439
|
const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined })
|
|
431
440
|
res.on('close', () => {
|
|
@@ -447,12 +456,20 @@ export async function startDaemon(opts: StartDaemonOptions = {}): Promise<Daemon
|
|
|
447
456
|
// timer regardless of how many listeners are bound).
|
|
448
457
|
let supervisor: ReturnType<typeof setInterval> | undefined
|
|
449
458
|
if (opts.supervise) {
|
|
450
|
-
const { intervalMs, tick } = opts.supervise
|
|
459
|
+
const { intervalMs, tick, onError } = opts.supervise
|
|
460
|
+
// A supervise tick must NEVER crash the daemon — but its error must NOT be SWALLOWED either: a
|
|
461
|
+
// silently-swallowed throw is a sweep that stops reaping INVISIBLY (no log, no stderr — the class
|
|
462
|
+
// that made a stuck reaper undiagnosable for hours). Route BOTH an async rejection and a sync throw
|
|
463
|
+
// to onError (the production main records ev=supervise-error in lifecycle.log); reporting itself
|
|
464
|
+
// never throws.
|
|
465
|
+
const report = (err: unknown): void => {
|
|
466
|
+
try { onError?.(err) } catch { /* a reporter must never crash the daemon */ }
|
|
467
|
+
}
|
|
451
468
|
supervisor = setInterval(() => {
|
|
452
469
|
try {
|
|
453
|
-
void Promise.resolve(tick()).catch(
|
|
454
|
-
} catch {
|
|
455
|
-
|
|
470
|
+
void Promise.resolve(tick()).catch(report)
|
|
471
|
+
} catch (err) {
|
|
472
|
+
report(err)
|
|
456
473
|
}
|
|
457
474
|
}, intervalMs)
|
|
458
475
|
supervisor.unref?.()
|
package/src/daemon/main.ts
CHANGED
|
@@ -35,6 +35,7 @@ import {
|
|
|
35
35
|
drainEphemeralQueue,
|
|
36
36
|
enqueueEphemeralTask,
|
|
37
37
|
isEphemeralPeer,
|
|
38
|
+
isLaunchdManaged,
|
|
38
39
|
addTopic,
|
|
39
40
|
loadLifecycleConfig,
|
|
40
41
|
processEagerRelaunches,
|
|
@@ -332,6 +333,10 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
332
333
|
// Busy-human-composer queue: fast queued ack, async drain, fail all pending
|
|
333
334
|
// on daemon close/update so a queued envelope is never silently lost.
|
|
334
335
|
composerQueue,
|
|
336
|
+
// H4 launchd-revive delivery retry (see RouteDeps.isLaunchdManaged): a MISS on a
|
|
337
|
+
// launchd-managed target (the daemon can't wake it, but KeepAlive revives it) → retry-
|
|
338
|
+
// resolve for a bounded window instead of failing in ~16ms — bridges a router restart.
|
|
339
|
+
isLaunchdManaged: (personality: string) => isLaunchdManaged(personality, env),
|
|
335
340
|
supervise: {
|
|
336
341
|
intervalMs: opts.superviseIntervalMs ?? DEFAULT_SUPERVISE_INTERVAL_MS,
|
|
337
342
|
// idle-reap / zombie-sweep, THEN the eager fresh re-launch for any peer whose
|
|
@@ -350,6 +355,15 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
350
355
|
// head). Identities without a queue or with a live session are no-ops.
|
|
351
356
|
await drainAllEphemeralQueues(cfg, { env })
|
|
352
357
|
},
|
|
358
|
+
// Surface a supervise-tick throw that the daemon timer would otherwise SWALLOW (the class that
|
|
359
|
+
// hid a stuck reaper for hours): record ev=supervise-error in lifecycle.log, flattened to one
|
|
360
|
+
// logfmt line. Best-effort — a reporter must never fail the daemon.
|
|
361
|
+
onError: (err: unknown) => {
|
|
362
|
+
try {
|
|
363
|
+
const detail = err instanceof Error ? (err.stack ?? err.message) : String(err)
|
|
364
|
+
appendLifecycleEvent(cfg.eventLogDir, { ev: 'supervise-error', error: detail.replace(/\s+/g, ' ').slice(0, 600) }, { env })
|
|
365
|
+
} catch { /* best-effort */ }
|
|
366
|
+
},
|
|
353
367
|
},
|
|
354
368
|
bearerToken,
|
|
355
369
|
port: opts.port ?? DEFAULT_DAEMON_PORT,
|
package/src/lifecycle/index.ts
CHANGED
|
@@ -1144,7 +1144,7 @@ export function killSession(_sock: string, identity: string): void {
|
|
|
1144
1144
|
|
|
1145
1145
|
export interface SuperviseOutcome {
|
|
1146
1146
|
identity: string
|
|
1147
|
-
action: 'reaped-idle' | 'reaped-gone' | 'reaped-ephemeral' | 'skipped-launchd' | 'skipped-stopped' | 'alive' | 'needs-eager-fresh'
|
|
1147
|
+
action: 'reaped-idle' | 'reaped-gone' | 'reaped-ephemeral' | 'skipped-launchd' | 'skipped-stopped' | 'alive' | 'needs-eager-fresh' | 'skipped-error'
|
|
1148
1148
|
reason?: string
|
|
1149
1149
|
/** For 'needs-eager-fresh': the peer to EAGERLY re-launch fresh (its session died
|
|
1150
1150
|
* carrying a .new-eager mark). The daemon timer drives the async relaunch.
|
|
@@ -1168,6 +1168,172 @@ export interface SuperviseDeps {
|
|
|
1168
1168
|
paneLogMtime?: (identity: string) => number | null
|
|
1169
1169
|
}
|
|
1170
1170
|
|
|
1171
|
+
/** Resolved per-tick context threaded into superviseOnePeer (deps already defaulted by superviseTick). */
|
|
1172
|
+
interface SuperviseCtx {
|
|
1173
|
+
cfg: LifecycleConfig
|
|
1174
|
+
env: NodeJS.ProcessEnv
|
|
1175
|
+
nowMs: number
|
|
1176
|
+
activityMtime: (runtime: string, cwd: string) => number | null
|
|
1177
|
+
isAlive: (sock: string, identity: string) => boolean
|
|
1178
|
+
paneLogMt: (identity: string) => number | null
|
|
1179
|
+
verbose: boolean
|
|
1180
|
+
trace: (fields: Record<string, string | number | undefined>) => void
|
|
1181
|
+
}
|
|
1182
|
+
|
|
1183
|
+
/**
|
|
1184
|
+
* Evaluate ONE daemon-owned session and take its at-most-one reap action, returning the outcome.
|
|
1185
|
+
* Extracted from superviseTick so a throw on a SINGLE peer can be ISOLATED by the caller: a malformed
|
|
1186
|
+
* peer-state must never abort the whole sweep (that is the silent fleet-wide reap-outage class — one bad
|
|
1187
|
+
* peer and NO peer gets reaped, the error swallowed by the daemon's supervise-tick catch). Every branch
|
|
1188
|
+
* returns exactly one outcome; the trace side-effects mirror the prior inline behaviour.
|
|
1189
|
+
*/
|
|
1190
|
+
function superviseOnePeer(s: SessionState, ctx: SuperviseCtx): SuperviseOutcome {
|
|
1191
|
+
const { cfg, env, nowMs, activityMtime, isAlive, paneLogMt, verbose, trace } = ctx
|
|
1192
|
+
// H4 — FIRST, before any reap. A launchd-managed peer is read-only.
|
|
1193
|
+
if (isLaunchdManaged(s.personality, env)) {
|
|
1194
|
+
if (verbose) trace({ identity: s.identity, action: 'skipped-launchd', outcome: 'read-only-h4' })
|
|
1195
|
+
return { identity: s.identity, action: 'skipped-launchd' }
|
|
1196
|
+
}
|
|
1197
|
+
const sock = buildSocketPath(s.runtime, s.personality, cfg.sockDir)
|
|
1198
|
+
if (!isAlive(sock, s.identity)) {
|
|
1199
|
+
// A DELIBERATE stop is not a death: the stop verb parks clean
|
|
1200
|
+
// (.stopped + .idle-reaped) and kills the session itself. Catch-up branch for
|
|
1201
|
+
// a stop that raced this tick (or a pre-fix stop): drop the state quietly —
|
|
1202
|
+
// no crash-loop entry, no death class — and ensure the clean-park marker so
|
|
1203
|
+
// the post-`start` wake RESUMES (stop→start must survive ≥ idle-reap).
|
|
1204
|
+
if (isStopped(cfg, s.identity)) {
|
|
1205
|
+
removeSessionState(cfg, s.identity)
|
|
1206
|
+
clearEphemeralArmed(cfg, s.identity)
|
|
1207
|
+
setIdleReaped(cfg, s.identity) // idempotent with the stop verb's own park
|
|
1208
|
+
trace({ identity: s.identity, action: 'skipped-stopped', outcome: 'resume-on-start' })
|
|
1209
|
+
return { identity: s.identity, action: 'skipped-stopped', reason: 'deliberate stop — parked clean, resumes on start' }
|
|
1210
|
+
}
|
|
1211
|
+
// A dead session: record a death for crash-loop accounting, then branch on the
|
|
1212
|
+
// .new-eager mark. This death was NOT daemon-initiated (the daemon only initiates
|
|
1213
|
+
// the idle-reap below) → it died on its own → do NOT write .idle-reaped here.
|
|
1214
|
+
recordDeath(cfg, s.identity, nowMs)
|
|
1215
|
+
removeSessionState(cfg, s.identity)
|
|
1216
|
+
// The .ephemeral-armed mark belongs to THIS (now dead) session — it armed on its
|
|
1217
|
+
// outbound reply. Clear it with the session, so a successor session can never be
|
|
1218
|
+
// quiet-reaped on a stale mark before answering its own task. No-op otherwise.
|
|
1219
|
+
clearEphemeralArmed(cfg, s.identity)
|
|
1220
|
+
// A session that died carrying a .new-eager mark is an owner /new: re-launch
|
|
1221
|
+
// EAGERLY as fresh (not lazily on the next message). The mark is LEFT for the
|
|
1222
|
+
// eager relaunch (processEagerRelaunches) to consume; the daemon timer drives it.
|
|
1223
|
+
if (hasNewEager(cfg, s.identity)) {
|
|
1224
|
+
trace({ identity: s.identity, action: 'needs-eager-fresh', reason: '/new eager mark', outcome: 'eager-fresh' })
|
|
1225
|
+
return {
|
|
1226
|
+
identity: s.identity,
|
|
1227
|
+
action: 'needs-eager-fresh',
|
|
1228
|
+
reason: '/new eager mark — eager fresh re-launch',
|
|
1229
|
+
personality: s.personality,
|
|
1230
|
+
runtime: s.runtime,
|
|
1231
|
+
}
|
|
1232
|
+
}
|
|
1233
|
+
// Crash / self-close: NO marker written, NO eager relaunch — the peer stays
|
|
1234
|
+
// asleep and wakes FRESH lazily on the next message (resolveWakeMode branch 3a).
|
|
1235
|
+
// The death-class tag (classifyGoneSession) makes the two gone-classes
|
|
1236
|
+
// distinguishable in lifecycle.log: `session-gone` (pane died, server alive →
|
|
1237
|
+
// exits.log should have the cause) vs `server-dead` (whole tmux server died →
|
|
1238
|
+
// exits.log structurally empty; this line is the only durable trace).
|
|
1239
|
+
const gone = classifyGoneSession(sock)
|
|
1240
|
+
trace({ identity: s.identity, action: 'reaped-gone', death: gone.death, reason: gone.reason, outcome: 'fresh-next-msg' })
|
|
1241
|
+
return { identity: s.identity, action: 'reaped-gone', reason: gone.reason }
|
|
1242
|
+
}
|
|
1243
|
+
// Idle accounting via the PANE-LOG (TUI render-stream) mtime — the RELIABLE idle
|
|
1244
|
+
// signal: it advances ~1s while the TUI renders ANY turn (model generation OR a tool
|
|
1245
|
+
// run — the "esc to interrupt" elapsed counter ticks) and goes quiet ONLY at the
|
|
1246
|
+
// prompt. NOT the transcript file mtime: a live claude session RE-TOUCHES its session
|
|
1247
|
+
// .jsonl ~every 30-45min WITHOUT a real turn (a system/bridge re-save), so the
|
|
1248
|
+
// transcript proxy is perpetually < idleSecs and a claude peer NEVER idle-reaps — the
|
|
1249
|
+
// fleet-reap incident: a warm claude pile, last real turn hours ago, never reaped;
|
|
1250
|
+
// codex's session file is NOT re-touched, so codex reaps (the claude-vs-codex split).
|
|
1251
|
+
// The transcript stays the launch ready-gate's signal (it needs "a real turn produced
|
|
1252
|
+
// output"); only this idle proxy moves to the pane-log. FLOORED at wokeAt: a session
|
|
1253
|
+
// cannot be idle longer than it has been awake (a wake that produced no turn →
|
|
1254
|
+
// mt=wokeAt → reaped idleSecs after the wake, never out from under an attaching op).
|
|
1255
|
+
const ephemeral = isEphemeralPeer(s.cwd)
|
|
1256
|
+
const plMt = paneLogMt(s.identity)
|
|
1257
|
+
let mt: number
|
|
1258
|
+
if (plMt !== null) {
|
|
1259
|
+
mt = Math.max(plMt, s.wokeAt)
|
|
1260
|
+
} else {
|
|
1261
|
+
// Pane-log MISSING (a legacy pre-pane-log supervisor) → fall back to the transcript
|
|
1262
|
+
// proxy, NEVER to wokeAt-as-activity: a wokeAt-only age would read an ACTIVE legacy
|
|
1263
|
+
// session's idle from its hours-old wokeAt and reap it MID-WORK. The transcript is
|
|
1264
|
+
// false-fresh at idle (so an IDLE legacy peer lingers warm until its next respawn
|
|
1265
|
+
// earns a pane-log — bounded, self-heals), but an ACTIVE legacy peer keeps a
|
|
1266
|
+
// genuinely-fresh transcript and is correctly protected. Safe direction preserved.
|
|
1267
|
+
let tx = 0
|
|
1268
|
+
try { tx = activityMtime(s.runtime, s.cwd) ?? 0 } catch { /* no adapter → wokeAt */ }
|
|
1269
|
+
mt = Math.max(tx, s.wokeAt)
|
|
1270
|
+
}
|
|
1271
|
+
const ageSecs = Math.floor((nowMs - mt) / 1000)
|
|
1272
|
+
// wake_policy:ephemeral M2 — die-after-reply: an ARMED ephemeral session (the
|
|
1273
|
+
// daemon routed its outbound reply) is reaped after a QUIET window, checked
|
|
1274
|
+
// BEFORE the idle branch (quiet ≪ idle). Quiet = the pane-log silent for
|
|
1275
|
+
// ephemeralQuietSecs (the reliable render signal above) — a still-rendering turn
|
|
1276
|
+
// keeps it fresh, so the worker finishes its turn first; post-reply housekeeping
|
|
1277
|
+
// (operative-note writes) also advances it.
|
|
1278
|
+
// NOT armed (still mid-task) → the ordinary idle bound below is its only reaper.
|
|
1279
|
+
// Deliberate, policy-driven death: NO .idle-reaped (an ephemeral peer never resumes)
|
|
1280
|
+
// and NO recordDeath (the crash-loop ring counts faults, not policy reaps).
|
|
1281
|
+
if (ephemeral && hasEphemeralArmed(cfg, s.identity) && ageSecs > cfg.ephemeralQuietSecs) {
|
|
1282
|
+
killSession(sock, s.identity)
|
|
1283
|
+
clearEphemeralArmed(cfg, s.identity)
|
|
1284
|
+
removeSessionState(cfg, s.identity)
|
|
1285
|
+
trace({ identity: s.identity, action: 'reaped-ephemeral', age: `${ageSecs}s`, outcome: 'ephemeral-done' })
|
|
1286
|
+
return {
|
|
1287
|
+
identity: s.identity,
|
|
1288
|
+
action: 'reaped-ephemeral',
|
|
1289
|
+
reason: `armed, quiet ${ageSecs}s`,
|
|
1290
|
+
personality: s.personality,
|
|
1291
|
+
runtime: s.runtime,
|
|
1292
|
+
}
|
|
1293
|
+
}
|
|
1294
|
+
// UNARMED ephemeral idle bound (live case): a worker that ended
|
|
1295
|
+
// SILENTLY (no final outbound → never armed; or its arm was lost to a CLI/
|
|
1296
|
+
// daemon-restart window) used to wait out the FULL generic idleSecs (1 h) —
|
|
1297
|
+
// and the M3 serial drain waits for the session's death, so ONE silent worker
|
|
1298
|
+
// stalled its whole conveyor. This bound is the defense-in-depth backstop:
|
|
1299
|
+
// ≫ the legitimate silent-tool case (sleep-180), ≪ idleSecs. The ШТАТНЫЙ
|
|
1300
|
+
// silent-finish path is `iapeer self-done` (arm without waking anyone —
|
|
1301
|
+
// the invariant «нет пустых пробуждений» stays intact); this branch only
|
|
1302
|
+
// bounds the damage when a worker does neither. Policy reap: NO .idle-reaped
|
|
1303
|
+
// (ephemeral never resumes), NO recordDeath (the ring counts faults). The pane-log
|
|
1304
|
+
// fold-in above also covers this branch: a STILL-WORKING unarmed worker (a silent
|
|
1305
|
+
// tool, no reply yet) keeps the pane-log fresh via the TUI render, so it is no longer
|
|
1306
|
+
// reaped mid-task — only a genuinely idle/wedged unarmed session crosses this bound.
|
|
1307
|
+
if (ephemeral && ageSecs > cfg.ephemeralUnarmedIdleSecs) {
|
|
1308
|
+
killSession(sock, s.identity)
|
|
1309
|
+
clearEphemeralArmed(cfg, s.identity)
|
|
1310
|
+
removeSessionState(cfg, s.identity)
|
|
1311
|
+
trace({ identity: s.identity, action: 'reaped-ephemeral', age: `${ageSecs}s`, outcome: 'ephemeral-unarmed-bound' })
|
|
1312
|
+
return {
|
|
1313
|
+
identity: s.identity,
|
|
1314
|
+
action: 'reaped-ephemeral',
|
|
1315
|
+
reason: `unarmed idle ${ageSecs}s (silent-finish backstop; штатный путь — iapeer self-done)`,
|
|
1316
|
+
personality: s.personality,
|
|
1317
|
+
runtime: s.runtime,
|
|
1318
|
+
}
|
|
1319
|
+
}
|
|
1320
|
+
if (ageSecs > cfg.idleSecs) {
|
|
1321
|
+
// THE ONLY place .idle-reaped is written: this is the one death the daemon
|
|
1322
|
+
// INITIATES. Its presence on the next wake = the session was parked cleanly =
|
|
1323
|
+
// RESUME-eligible (resolveWakeMode branch 3b). A crash/self-close (the dead
|
|
1324
|
+
// branch above) never writes it → that wakes FRESH (branch 3a).
|
|
1325
|
+
killSession(sock, s.identity)
|
|
1326
|
+
setIdleReaped(cfg, s.identity)
|
|
1327
|
+
removeSessionState(cfg, s.identity)
|
|
1328
|
+
trace({ identity: s.identity, action: 'reaped-idle', age: `${ageSecs}s`, outcome: 'resume-eligible' })
|
|
1329
|
+
return { identity: s.identity, action: 'reaped-idle', reason: `idle ${ageSecs}s` }
|
|
1330
|
+
}
|
|
1331
|
+
// ALIVE and not idle. (pty-only: no tmux server-death canary — a hosted death is recorded by the
|
|
1332
|
+
// supervisor's exits.log and classified by classifyGoneSession on the next sweep.)
|
|
1333
|
+
if (verbose) trace({ identity: s.identity, action: 'alive', age: `${ageSecs}s` })
|
|
1334
|
+
return { identity: s.identity, action: 'alive' }
|
|
1335
|
+
}
|
|
1336
|
+
|
|
1171
1337
|
export function superviseTick(cfg: LifecycleConfig, deps: SuperviseDeps = {}): SuperviseOutcome[] {
|
|
1172
1338
|
const env = deps.env ?? process.env
|
|
1173
1339
|
const nowMs = deps.nowMs ?? Date.now()
|
|
@@ -1190,160 +1356,18 @@ export function superviseTick(cfg: LifecycleConfig, deps: SuperviseDeps = {}): S
|
|
|
1190
1356
|
const trace = (fields: Record<string, string | number | undefined>): void =>
|
|
1191
1357
|
appendLifecycleEvent(cfg.eventLogDir, { ev: 'supervise', ...fields }, { env, nowMs })
|
|
1192
1358
|
const out: SuperviseOutcome[] = []
|
|
1359
|
+
const ctx: SuperviseCtx = { cfg, env, nowMs, activityMtime, isAlive, paneLogMt, verbose, trace }
|
|
1193
1360
|
for (const s of readSessionStates(cfg)) {
|
|
1194
|
-
//
|
|
1195
|
-
|
|
1196
|
-
|
|
1197
|
-
|
|
1198
|
-
|
|
1199
|
-
|
|
1200
|
-
|
|
1201
|
-
|
|
1202
|
-
|
|
1203
|
-
|
|
1204
|
-
// a stop that raced this tick (or a pre-fix stop): drop the state quietly —
|
|
1205
|
-
// no crash-loop entry, no death class — and ensure the clean-park marker so
|
|
1206
|
-
// the post-`start` wake RESUMES (stop→start must survive ≥ idle-reap).
|
|
1207
|
-
if (isStopped(cfg, s.identity)) {
|
|
1208
|
-
removeSessionState(cfg, s.identity)
|
|
1209
|
-
clearEphemeralArmed(cfg, s.identity)
|
|
1210
|
-
setIdleReaped(cfg, s.identity) // idempotent with the stop verb's own park
|
|
1211
|
-
out.push({ identity: s.identity, action: 'skipped-stopped', reason: 'deliberate stop — parked clean, resumes on start' })
|
|
1212
|
-
trace({ identity: s.identity, action: 'skipped-stopped', outcome: 'resume-on-start' })
|
|
1213
|
-
continue
|
|
1214
|
-
}
|
|
1215
|
-
// A dead session: record a death for crash-loop accounting, then branch on the
|
|
1216
|
-
// .new-eager mark. This death was NOT daemon-initiated (the daemon only initiates
|
|
1217
|
-
// the idle-reap below) → it died on its own → do NOT write .idle-reaped here.
|
|
1218
|
-
recordDeath(cfg, s.identity, nowMs)
|
|
1219
|
-
removeSessionState(cfg, s.identity)
|
|
1220
|
-
// The .ephemeral-armed mark belongs to THIS (now dead) session — it armed on its
|
|
1221
|
-
// outbound reply. Clear it with the session, so a successor session can never be
|
|
1222
|
-
// quiet-reaped on a stale mark before answering its own task. No-op otherwise.
|
|
1223
|
-
clearEphemeralArmed(cfg, s.identity)
|
|
1224
|
-
// A session that died carrying a .new-eager mark is an owner /new: re-launch
|
|
1225
|
-
// EAGERLY as fresh (not lazily on the next message). The mark is LEFT for the
|
|
1226
|
-
// eager relaunch (processEagerRelaunches) to consume; the daemon timer drives it.
|
|
1227
|
-
if (hasNewEager(cfg, s.identity)) {
|
|
1228
|
-
out.push({
|
|
1229
|
-
identity: s.identity,
|
|
1230
|
-
action: 'needs-eager-fresh',
|
|
1231
|
-
reason: '/new eager mark — eager fresh re-launch',
|
|
1232
|
-
personality: s.personality,
|
|
1233
|
-
runtime: s.runtime,
|
|
1234
|
-
})
|
|
1235
|
-
trace({ identity: s.identity, action: 'needs-eager-fresh', reason: '/new eager mark', outcome: 'eager-fresh' })
|
|
1236
|
-
continue
|
|
1237
|
-
}
|
|
1238
|
-
// Crash / self-close: NO marker written, NO eager relaunch — the peer stays
|
|
1239
|
-
// asleep and wakes FRESH lazily on the next message (resolveWakeMode branch 3a).
|
|
1240
|
-
// The death-class tag (classifyGoneSession) makes the two gone-classes
|
|
1241
|
-
// distinguishable in lifecycle.log: `session-gone` (pane died, server alive →
|
|
1242
|
-
// exits.log should have the cause) vs `server-dead` (whole tmux server died →
|
|
1243
|
-
// exits.log structurally empty; this line is the only durable trace).
|
|
1244
|
-
const gone = classifyGoneSession(sock)
|
|
1245
|
-
out.push({ identity: s.identity, action: 'reaped-gone', reason: gone.reason })
|
|
1246
|
-
trace({ identity: s.identity, action: 'reaped-gone', death: gone.death, reason: gone.reason, outcome: 'fresh-next-msg' })
|
|
1247
|
-
continue
|
|
1248
|
-
}
|
|
1249
|
-
// Idle accounting via the PANE-LOG (TUI render-stream) mtime — the RELIABLE idle
|
|
1250
|
-
// signal: it advances ~1s while the TUI renders ANY turn (model generation OR a tool
|
|
1251
|
-
// run — the "esc to interrupt" elapsed counter ticks) and goes quiet ONLY at the
|
|
1252
|
-
// prompt. NOT the transcript file mtime: a live claude session RE-TOUCHES its session
|
|
1253
|
-
// .jsonl ~every 30-45min WITHOUT a real turn (a system/bridge re-save), so the
|
|
1254
|
-
// transcript proxy is perpetually < idleSecs and a claude peer NEVER idle-reaps — the
|
|
1255
|
-
// fleet-reap incident: a warm claude pile, last real turn hours ago, never reaped;
|
|
1256
|
-
// codex's session file is NOT re-touched, so codex reaps (the claude-vs-codex split).
|
|
1257
|
-
// The transcript stays the launch ready-gate's signal (it needs "a real turn produced
|
|
1258
|
-
// output"); only this idle proxy moves to the pane-log. FLOORED at wokeAt: a session
|
|
1259
|
-
// cannot be idle longer than it has been awake (a wake that produced no turn →
|
|
1260
|
-
// mt=wokeAt → reaped idleSecs after the wake, never out from under an attaching op).
|
|
1261
|
-
const ephemeral = isEphemeralPeer(s.cwd)
|
|
1262
|
-
const plMt = paneLogMt(s.identity)
|
|
1263
|
-
let mt: number
|
|
1264
|
-
if (plMt !== null) {
|
|
1265
|
-
mt = Math.max(plMt, s.wokeAt)
|
|
1266
|
-
} else {
|
|
1267
|
-
// Pane-log MISSING (a legacy pre-pane-log supervisor) → fall back to the transcript
|
|
1268
|
-
// proxy, NEVER to wokeAt-as-activity: a wokeAt-only age would read an ACTIVE legacy
|
|
1269
|
-
// session's idle from its hours-old wokeAt and reap it MID-WORK. The transcript is
|
|
1270
|
-
// false-fresh at idle (so an IDLE legacy peer lingers warm until its next respawn
|
|
1271
|
-
// earns a pane-log — bounded, self-heals), but an ACTIVE legacy peer keeps a
|
|
1272
|
-
// genuinely-fresh transcript and is correctly protected. Safe direction preserved.
|
|
1273
|
-
let tx = 0
|
|
1274
|
-
try { tx = activityMtime(s.runtime, s.cwd) ?? 0 } catch { /* no adapter → wokeAt */ }
|
|
1275
|
-
mt = Math.max(tx, s.wokeAt)
|
|
1276
|
-
}
|
|
1277
|
-
const ageSecs = Math.floor((nowMs - mt) / 1000)
|
|
1278
|
-
// wake_policy:ephemeral M2 — die-after-reply: an ARMED ephemeral session (the
|
|
1279
|
-
// daemon routed its outbound reply) is reaped after a QUIET window, checked
|
|
1280
|
-
// BEFORE the idle branch (quiet ≪ idle). Quiet = the pane-log silent for
|
|
1281
|
-
// ephemeralQuietSecs (the reliable render signal above) — a still-rendering turn
|
|
1282
|
-
// keeps it fresh, so the worker finishes its turn first; post-reply housekeeping
|
|
1283
|
-
// (operative-note writes) also advances it.
|
|
1284
|
-
// NOT armed (still mid-task) → the ordinary idle bound below is its only reaper.
|
|
1285
|
-
// Deliberate, policy-driven death: NO .idle-reaped (an ephemeral peer never resumes)
|
|
1286
|
-
// and NO recordDeath (the crash-loop ring counts faults, not policy reaps).
|
|
1287
|
-
if (ephemeral && hasEphemeralArmed(cfg, s.identity) && ageSecs > cfg.ephemeralQuietSecs) {
|
|
1288
|
-
killSession(sock, s.identity)
|
|
1289
|
-
clearEphemeralArmed(cfg, s.identity)
|
|
1290
|
-
removeSessionState(cfg, s.identity)
|
|
1291
|
-
out.push({
|
|
1292
|
-
identity: s.identity,
|
|
1293
|
-
action: 'reaped-ephemeral',
|
|
1294
|
-
reason: `armed, quiet ${ageSecs}s`,
|
|
1295
|
-
personality: s.personality,
|
|
1296
|
-
runtime: s.runtime,
|
|
1297
|
-
})
|
|
1298
|
-
trace({ identity: s.identity, action: 'reaped-ephemeral', age: `${ageSecs}s`, outcome: 'ephemeral-done' })
|
|
1299
|
-
continue
|
|
1300
|
-
}
|
|
1301
|
-
// UNARMED ephemeral idle bound (live case): a worker that ended
|
|
1302
|
-
// SILENTLY (no final outbound → never armed; or its arm was lost to a CLI/
|
|
1303
|
-
// daemon-restart window) used to wait out the FULL generic idleSecs (1 h) —
|
|
1304
|
-
// and the M3 serial drain waits for the session's death, so ONE silent worker
|
|
1305
|
-
// stalled its whole conveyor. This bound is the defense-in-depth backstop:
|
|
1306
|
-
// ≫ the legitimate silent-tool case (sleep-180), ≪ idleSecs. The ШТАТНЫЙ
|
|
1307
|
-
// silent-finish path is `iapeer self-done` (arm without waking anyone —
|
|
1308
|
-
// the invariant «нет пустых пробуждений» stays intact); this branch only
|
|
1309
|
-
// bounds the damage when a worker does neither. Policy reap: NO .idle-reaped
|
|
1310
|
-
// (ephemeral never resumes), NO recordDeath (the ring counts faults). The pane-log
|
|
1311
|
-
// fold-in above also covers this branch: a STILL-WORKING unarmed worker (a silent
|
|
1312
|
-
// tool, no reply yet) keeps the pane-log fresh via the TUI render, so it is no longer
|
|
1313
|
-
// reaped mid-task — only a genuinely idle/wedged unarmed session crosses this bound.
|
|
1314
|
-
if (ephemeral && ageSecs > cfg.ephemeralUnarmedIdleSecs) {
|
|
1315
|
-
killSession(sock, s.identity)
|
|
1316
|
-
clearEphemeralArmed(cfg, s.identity)
|
|
1317
|
-
removeSessionState(cfg, s.identity)
|
|
1318
|
-
out.push({
|
|
1319
|
-
identity: s.identity,
|
|
1320
|
-
action: 'reaped-ephemeral',
|
|
1321
|
-
reason: `unarmed idle ${ageSecs}s (silent-finish backstop; штатный путь — iapeer self-done)`,
|
|
1322
|
-
personality: s.personality,
|
|
1323
|
-
runtime: s.runtime,
|
|
1324
|
-
})
|
|
1325
|
-
trace({ identity: s.identity, action: 'reaped-ephemeral', age: `${ageSecs}s`, outcome: 'ephemeral-unarmed-bound' })
|
|
1326
|
-
continue
|
|
1327
|
-
}
|
|
1328
|
-
if (ageSecs > cfg.idleSecs) {
|
|
1329
|
-
// THE ONLY place .idle-reaped is written: this is the one death the daemon
|
|
1330
|
-
// INITIATES. Its presence on the next wake = the session was parked cleanly =
|
|
1331
|
-
// RESUME-eligible (resolveWakeMode branch 3b). A crash/self-close (the dead
|
|
1332
|
-
// branch above) never writes it → that wakes FRESH (branch 3a).
|
|
1333
|
-
killSession(sock, s.identity)
|
|
1334
|
-
setIdleReaped(cfg, s.identity)
|
|
1335
|
-
removeSessionState(cfg, s.identity)
|
|
1336
|
-
out.push({ identity: s.identity, action: 'reaped-idle', reason: `idle ${ageSecs}s` })
|
|
1337
|
-
trace({ identity: s.identity, action: 'reaped-idle', age: `${ageSecs}s`, outcome: 'resume-eligible' })
|
|
1338
|
-
} else {
|
|
1339
|
-
// Server-death canary retrofit (canary.ts): ensure a canary is watching
|
|
1340
|
-
// every ALIVE daemon-owned server — covers a fleet launched by older code
|
|
1341
|
-
// within one tick of a deploy, no session restarts. Idempotent (pgrep on
|
|
1342
|
-
// the per-identity wait-for channel), best-effort, pure observability.
|
|
1343
|
-
// pty-only: no tmux server-death canary — a hosted death is recorded by the supervisor's
|
|
1344
|
-
// exits.log (host=supervisor, resilience-b), classified by classifyGoneSession on the next sweep.
|
|
1345
|
-
out.push({ identity: s.identity, action: 'alive' })
|
|
1346
|
-
if (verbose) trace({ identity: s.identity, action: 'alive', age: `${ageSecs}s` })
|
|
1361
|
+
// Per-peer ISOLATION: a throw while evaluating ONE peer must NOT abort the whole sweep — that is
|
|
1362
|
+
// the silent fleet-wide reap-outage class (one malformed peer-state and NO peer gets reaped, the
|
|
1363
|
+
// error swallowed by the daemon's supervise-tick catch). Catch it, log it loudly to lifecycle.log,
|
|
1364
|
+
// and continue to the next peer — the rest of the fleet is still swept.
|
|
1365
|
+
try {
|
|
1366
|
+
out.push(superviseOnePeer(s, ctx))
|
|
1367
|
+
} catch (e) {
|
|
1368
|
+
const reason = e instanceof Error ? e.message : String(e)
|
|
1369
|
+
out.push({ identity: s.identity, action: 'skipped-error', reason })
|
|
1370
|
+
trace({ identity: s.identity, action: 'skipped-error', reason, outcome: 'peer-error-isolated' })
|
|
1347
1371
|
}
|
|
1348
1372
|
}
|
|
1349
1373
|
return out
|
package/src/transport/index.ts
CHANGED
|
@@ -152,54 +152,65 @@ export function resolvePeerDeliveryTarget(
|
|
|
152
152
|
return resolveDeliveryTarget({ personality })
|
|
153
153
|
}
|
|
154
154
|
|
|
155
|
-
//
|
|
156
|
-
//
|
|
157
|
-
// is not rendered). Env-tunable for LIVE calibration (busy/idle/cold). Conservative
|
|
158
|
-
// default — long enough that a busy-but-alive session reliably shows an advance,
|
|
159
|
-
// short enough that a genuinely dead session is failed promptly. Prefer a false-
|
|
160
|
-
// FAIL (sender retries) over a false-OK (silent loss, which the contract forbids).
|
|
155
|
+
// Confirm poll interval — how often deliverViaHost re-reads the transcript for a NEW record carrying
|
|
156
|
+
// our envelope while waiting out the grace.
|
|
161
157
|
const LIVENESS_POLL_MS = 100
|
|
162
|
-
|
|
163
|
-
|
|
158
|
+
|
|
159
|
+
// Confirm grace — how long deliverViaHost polls for a transcript record CARRYING our envelope before
|
|
160
|
+
// declaring a false-FAIL. Per-RUNTIME because the runtimes log an accepted injection at different
|
|
161
|
+
// times: claude writes the queue-operation (busy) / user-turn (idle) record PROMPTLY (sub-second), so
|
|
162
|
+
// a short grace suffices; codex writes its session-jsonl user-input only at model-turn-START (TTFT
|
|
163
|
+
// measured ~4-6s after submit), so it needs a longer window or every codex delivery would false-FAIL
|
|
164
|
+
// before the first write. `IAP_HOST_LIVENESS_GRACE_MS` overrides BOTH (operator calibration; tests set
|
|
165
|
+
// 0 for an immediate deterministic fail). Prefer a false-FAIL (sender retries) over a false-OK (silent
|
|
166
|
+
// loss, which the contract forbids).
|
|
167
|
+
const CONFIRM_GRACE_CLAUDE_MS = 4000
|
|
168
|
+
const CONFIRM_GRACE_CODEX_MS = 8000
|
|
169
|
+
function confirmGraceMs(runtime: string): number {
|
|
170
|
+
const raw = process.env.IAP_HOST_LIVENESS_GRACE_MS
|
|
164
171
|
const n = raw === undefined ? NaN : Number(raw)
|
|
165
|
-
|
|
172
|
+
if (Number.isFinite(n) && n >= 0) return n
|
|
173
|
+
return runtime === 'codex' ? CONFIRM_GRACE_CODEX_MS : CONFIRM_GRACE_CLAUDE_MS
|
|
166
174
|
}
|
|
167
175
|
|
|
168
|
-
//
|
|
169
|
-
//
|
|
170
|
-
//
|
|
171
|
-
//
|
|
172
|
-
|
|
173
|
-
function
|
|
174
|
-
const raw = process.env.
|
|
176
|
+
// Launchd-revive retry — a MISS on a launchd-managed target the daemon can't wake (H4)
|
|
177
|
+
// but launchd KeepAlive WILL revive (router restart / crash-revive window). routeSend
|
|
178
|
+
// re-resolves for up to this window before failing, bridging the gap (silent-loss guard).
|
|
179
|
+
// Env-tunable; default ~4s (a bootout→bootstrap router restart settles well under it).
|
|
180
|
+
const LAUNCHD_REVIVE_POLL_MS = 300
|
|
181
|
+
function launchdReviveGraceMs(): number {
|
|
182
|
+
const raw = process.env.IAP_LAUNCHD_REVIVE_GRACE_MS
|
|
175
183
|
const n = raw === undefined ? NaN : Number(raw)
|
|
176
|
-
return Number.isFinite(n) && n >= 0 ? n :
|
|
184
|
+
return Number.isFinite(n) && n >= 0 ? n : 4000
|
|
177
185
|
}
|
|
178
186
|
|
|
179
187
|
// ─── Warm delivery (pty-only — supervisor socket) ────────────────────────────────────────────────
|
|
180
188
|
|
|
181
|
-
/** Test seam for `deliverWarm`. Production defaults route on the real supervisor
|
|
182
|
-
*
|
|
183
|
-
* Omitting all fields = pure production
|
|
189
|
+
/** Test seam for `deliverWarm`. Production defaults route on the real supervisor + the real
|
|
190
|
+
* message-specific transcript confirm; tests inject a fake host-deliver + a synthetic landed-confirm,
|
|
191
|
+
* so the deliver path is covered with no live daemon. Omitting all fields = pure production. */
|
|
184
192
|
export interface WarmDeliverSeam {
|
|
185
193
|
/** Deliver to the hosted session over its socket. Default: the real `deliverHosted` (Ф0a leaf). */
|
|
186
194
|
deliverHosted?: (identity: string, envelope: string) => Promise<DeliverResult>
|
|
187
|
-
/**
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
*
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
*
|
|
194
|
-
|
|
195
|
+
/** MESSAGE-SPECIFIC landed-confirm: did a NEW transcript/session-jsonl record CARRYING this envelope
|
|
196
|
+
* appear since the pre-deliver baseline? Default: `transcriptCarriesEnvelope` over a file-size
|
|
197
|
+
* baseline captured before the send. A bare mtime bump (the receiver's OWN turn) is NOT a confirm —
|
|
198
|
+
* that was the false-OK class this replaces. */
|
|
199
|
+
confirmLanded?: (envelope: string) => boolean
|
|
200
|
+
/** Injectable async sleep for the confirm poll (tests). Default: setTimeout — it YIELDS the event
|
|
201
|
+
* loop (the grace is up to several seconds for codex, so a sync block would stall the whole router
|
|
202
|
+
* daemon for the duration of one delivery). */
|
|
203
|
+
sleep?: (ms: number) => Promise<void>
|
|
195
204
|
}
|
|
196
205
|
|
|
197
206
|
/**
|
|
198
207
|
* Warm-deliver `envelope` to a LIVE local target over its supervisor socket (pty-only). Landing is
|
|
199
|
-
* CONFIRMED by a transcript
|
|
200
|
-
* CR proves the bytes left us,
|
|
201
|
-
*
|
|
202
|
-
*
|
|
208
|
+
* CONFIRMED MESSAGE-SPECIFICALLY — by a NEW transcript record CARRYING this envelope appearing past the
|
|
209
|
+
* pre-deliver baseline, NOT the socket-ack (a flushed CR proves the bytes left us, not that the session
|
|
210
|
+
* accepted them) and NOT a bare mtime advance (the receiver's OWN concurrent turn bumps mtime even when
|
|
211
|
+
* our paste was swallowed — the false-OK class the contract forbids). A router (telegram/notifier) has
|
|
212
|
+
* no transcript → confirmed by the socket-ack (liveness is structural via launchd KeepAlive). See
|
|
213
|
+
* `deliverViaHost`.
|
|
203
214
|
*/
|
|
204
215
|
export async function deliverWarm(
|
|
205
216
|
target: DeliveryTarget,
|
|
@@ -217,13 +228,10 @@ async function deliverViaHost(
|
|
|
217
228
|
seam: WarmDeliverSeam,
|
|
218
229
|
): Promise<Result<void>> {
|
|
219
230
|
const adapter = getAdapter(target.runtime)
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
//
|
|
223
|
-
|
|
224
|
-
// proves the session is alive and took our message.
|
|
225
|
-
const baseline = cwd ? mtimeOf(cwd) ?? 0 : 0
|
|
226
|
-
const paneBaseline = paneMtimeOf(target.address) ?? 0
|
|
231
|
+
// Baseline the transcript/session-jsonl file SIZES BEFORE delivery (per-runtime candidate files).
|
|
232
|
+
// The landed-confirm reads ONLY the bytes appended past this baseline, so a record carrying our
|
|
233
|
+
// envelope can only be one written in RESPONSE to this delivery — never a pre-existing copy.
|
|
234
|
+
const baseline = cwd ? compactDoneBaseline(target.runtime, cwd) : null
|
|
227
235
|
const send = seam.deliverHosted ?? deliverHosted
|
|
228
236
|
const sent = await send(target.address, envelope)
|
|
229
237
|
if (!sent.ok) {
|
|
@@ -232,45 +240,42 @@ async function deliverViaHost(
|
|
|
232
240
|
`message NOT delivered`,
|
|
233
241
|
)
|
|
234
242
|
}
|
|
235
|
-
// Router (telegram/notifier): no transcript proxy
|
|
236
|
-
//
|
|
237
|
-
//
|
|
238
|
-
// delivery-level confirm; a router's liveness is structural via launchd KeepAlive, not a model
|
|
239
|
-
// turn). See deliverViaTmux router branch (paste-buffer -r + C-j, no transcript confirm) + its
|
|
240
|
-
// header note "router … liveness is structural".
|
|
243
|
+
// Router (telegram/notifier): no transcript proxy → confirmed by the socket-ack just obtained. A
|
|
244
|
+
// router's liveness is structural via launchd KeepAlive (not a model turn), so the ack IS the
|
|
245
|
+
// delivery-level confirm (parity with the legacy tmux router C-j path).
|
|
241
246
|
if (adapter.kind === 'router') return ok(undefined)
|
|
242
|
-
// socket-ack ≠ landed. With no
|
|
243
|
-
//
|
|
244
|
-
if (!cwd) return ok(undefined)
|
|
245
|
-
//
|
|
246
|
-
//
|
|
247
|
-
//
|
|
248
|
-
//
|
|
249
|
-
//
|
|
250
|
-
//
|
|
251
|
-
//
|
|
252
|
-
//
|
|
253
|
-
//
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
// during a turn). Prefer a false-FAIL (sender retries) over a false-OK (silent loss).
|
|
258
|
-
const graceMs = hostLivenessGraceMs()
|
|
247
|
+
// socket-ack ≠ landed. With no transcript (direct callers/tests without a cwd) we can only trust the
|
|
248
|
+
// flushed submit — confirmed-only.
|
|
249
|
+
if (!cwd || !baseline) return ok(undefined)
|
|
250
|
+
// MESSAGE-SPECIFIC landed-confirm — the false-OK killer. A bare transcript/pane-log mtime BUMP is
|
|
251
|
+
// NOT proof the session took THIS message: a receiver in an active turn advances both mtimes with
|
|
252
|
+
// its OWN rendering even when our paste was swallowed at the turn boundary (incident 2026-06-23:
|
|
253
|
+
// ok=true, message gone). The ONLY proof is a NEW transcript record CARRYING our envelope —
|
|
254
|
+
// claude: a queue-operation `content` (busy → enqueued) or the user-turn message (idle),
|
|
255
|
+
// codex: a user-input response_item (written at model-turn-start, ~4-6s — hence the codex grace).
|
|
256
|
+
// The receiver's own assistant/tool turn never reproduces the `<iap from-personality=…>` wrapper, so
|
|
257
|
+
// it cannot forge a confirm. No such record within the grace → false-FAIL (the sender retries) — NOT
|
|
258
|
+
// a false-OK (silent loss the contract forbids).
|
|
259
|
+
const confirm = seam.confirmLanded ?? ((payload: string) => transcriptCarriesEnvelope(baseline, payload))
|
|
260
|
+
const sleep = seam.sleep ?? ((ms: number) => new Promise<void>(r => setTimeout(r, ms)))
|
|
261
|
+
const graceMs = confirmGraceMs(target.runtime)
|
|
259
262
|
const graceDeadline = monotonicMs() + graceMs
|
|
260
|
-
|
|
261
|
-
if ((
|
|
262
|
-
if ((
|
|
263
|
-
|
|
264
|
-
}
|
|
263
|
+
while (true) {
|
|
264
|
+
if (confirm(envelope)) return ok(undefined)
|
|
265
|
+
if (monotonicMs() >= graceDeadline) break
|
|
266
|
+
await sleep(LIVENESS_POLL_MS)
|
|
267
|
+
}
|
|
265
268
|
return err(
|
|
266
269
|
`hosted peer "${target.personality}" (${target.runtime}) is listed live but did not accept the message ` +
|
|
267
|
-
`(no transcript
|
|
270
|
+
`(no transcript record carrying the message within ${graceMs}ms) — live but unresponsive to this delivery (not dead); message NOT delivered`,
|
|
268
271
|
)
|
|
269
272
|
}
|
|
270
273
|
|
|
271
274
|
/** Pane-log (TUI render-stream) mtime for a process address, or null when absent/unreadable.
|
|
272
275
|
* The pane-log is the canonical `<root>/logs/lifecycle/<address>.log` the supervisor appends raw
|
|
273
|
-
* pty bytes to (§1 contract). Used
|
|
276
|
+
* pty bytes to (§1 contract). Used by `resolveLiveRuntime` (freshest-pane-log among alive sessions).
|
|
277
|
+
* NB: deliberately NOT used to confirm delivery — a pane-log mtime bump is not message-specific (the
|
|
278
|
+
* receiver's own turn bumps it too), which was exactly the false-OK class deliverViaHost now avoids. */
|
|
274
279
|
function defaultPaneLogMtime(address: string): number | null {
|
|
275
280
|
try {
|
|
276
281
|
return statSync(join(resolveGlobalRoot(), 'logs', 'lifecycle', `${address}.log`)).mtimeMs
|
|
@@ -481,6 +486,65 @@ export function compactTranscriptHasDone(
|
|
|
481
486
|
return false
|
|
482
487
|
}
|
|
483
488
|
|
|
489
|
+
/** Deep-search every string value of a parsed JSON record for `needle` (CR-stripped on the haystack
|
|
490
|
+
* side; the needle is pre-stripped by the caller). */
|
|
491
|
+
function jsonStringValuesContain(value: unknown, needle: string): boolean {
|
|
492
|
+
if (typeof value === 'string') return value.replace(/\r/g, '').includes(needle)
|
|
493
|
+
if (Array.isArray(value)) return value.some(v => jsonStringValuesContain(v, needle))
|
|
494
|
+
if (value && typeof value === 'object') return Object.values(value).some(v => jsonStringValuesContain(v, needle))
|
|
495
|
+
return false
|
|
496
|
+
}
|
|
497
|
+
|
|
498
|
+
/**
|
|
499
|
+
* Did a NEW transcript/session-jsonl record CARRYING `envelope` appear since `baseline` (the
|
|
500
|
+
* per-runtime absolute file paths + SIZES captured immediately before delivery)? This is the
|
|
501
|
+
* MESSAGE-SPECIFIC landed-confirm `deliverViaHost` uses to kill the false-OK class — a bare mtime
|
|
502
|
+
* advance from the receiver's OWN turn is NOT a confirm; only a record echoing our envelope is.
|
|
503
|
+
*
|
|
504
|
+
* Reads ONLY the bytes appended past each baseline offset, JSON-parses each new jsonl line, and
|
|
505
|
+
* deep-searches its string values for the envelope (CR-normalized on both sides — a stored line may
|
|
506
|
+
* carry CR the envelope never had; CR is never semantically meaningful inside an IAP envelope). A
|
|
507
|
+
* match means the SESSION recorded our injection:
|
|
508
|
+
* claude — busy: a `queue-operation` whose `content` is the pasted envelope verbatim; idle: the
|
|
509
|
+
* user-turn message.
|
|
510
|
+
* codex — a user-input `response_item` (written at model-turn-start).
|
|
511
|
+
* The receiver's own assistant/tool records never reproduce the `<iap from-personality=…>` wrapper, so
|
|
512
|
+
* the full envelope is UNFORGEABLE by a concurrent turn — no false-OK. Because only bytes PAST the
|
|
513
|
+
* baseline offset are read, a pre-existing copy of an identical earlier message cannot pass either.
|
|
514
|
+
*
|
|
515
|
+
* Iterates the baseline file LIST verbatim — it does NOT re-scan the runtime's session dir each poll
|
|
516
|
+
* (unlike the minutes-long compact-done gate). A delivery's confirm window is seconds, so a brand-new
|
|
517
|
+
* session file appearing mid-delivery (a resume mid-flight) is negligible, and missing it fails in the
|
|
518
|
+
* SAFE direction (false-FAIL → the sender retries, never a silent loss). For codex this also avoids a
|
|
519
|
+
* full recursive `~/.codex/sessions` re-scan on every ~100ms poll (the active session file is already
|
|
520
|
+
* in the baseline), keeping the hot delivery path cheap.
|
|
521
|
+
*/
|
|
522
|
+
export function transcriptCarriesEnvelope(baseline: CompactDoneBaseline, envelope: string): boolean {
|
|
523
|
+
const needle = envelope.replace(/\r/g, '')
|
|
524
|
+
if (!needle) return false
|
|
525
|
+
for (const { path, size: offset } of baseline.files) {
|
|
526
|
+
let buf: Buffer
|
|
527
|
+
try {
|
|
528
|
+
buf = readFileSync(path)
|
|
529
|
+
} catch {
|
|
530
|
+
continue
|
|
531
|
+
}
|
|
532
|
+
const start = Math.min(Math.max(0, offset), buf.length)
|
|
533
|
+
const tail = buf.subarray(start).toString('utf8')
|
|
534
|
+
for (const line of tail.split(/\r?\n/)) {
|
|
535
|
+
if (!line.trim()) continue
|
|
536
|
+
let obj: unknown
|
|
537
|
+
try {
|
|
538
|
+
obj = JSON.parse(line)
|
|
539
|
+
} catch {
|
|
540
|
+
continue
|
|
541
|
+
}
|
|
542
|
+
if (jsonStringValuesContain(obj, needle)) return true
|
|
543
|
+
}
|
|
544
|
+
}
|
|
545
|
+
return false
|
|
546
|
+
}
|
|
547
|
+
|
|
484
548
|
function compactDoneTimeoutMs(env: NodeJS.ProcessEnv | undefined): number {
|
|
485
549
|
const raw = env?.IAPEER_COMPACT_DONE_TIMEOUT_MS
|
|
486
550
|
const n = raw === undefined ? NaN : Number(raw)
|
|
@@ -793,6 +857,17 @@ export interface RouteDeps {
|
|
|
793
857
|
* imports lifecycle (layering — main.ts is where the two meet); call sites
|
|
794
858
|
* swallow hook errors (a hook must never fail a delivered message). */
|
|
795
859
|
noteLiveTopic?: (identity: string, topic: string) => void
|
|
860
|
+
/** H4 launchd-managed detector (injected so transport never imports lifecycle —
|
|
861
|
+
* main.ts wires the real `isLaunchdManaged`). A MISS on a launchd-managed target is
|
|
862
|
+
* almost always a TRANSIENT restart window (router restart on connect / `iapeer
|
|
863
|
+
* update`, or a crash that KeepAlive is reviving), NOT a real outage: the daemon
|
|
864
|
+
* can't wake it (H4) but launchd WILL revive it in ~1s. So routeSend retries the
|
|
865
|
+
* RESOLVE for a bounded window before failing, bridging the gap (prevents the
|
|
866
|
+
* silent-loss class: an in-flight delivery — esp. to a HUMAN over a telegram router —
|
|
867
|
+
* dropped during a restart). Absent → no retry (legacy behaviour). */
|
|
868
|
+
isLaunchdManaged?: (personality: string) => boolean
|
|
869
|
+
/** Injectable async sleep for the launchd-revive retry poll (tests). Default: setTimeout. */
|
|
870
|
+
sleep?: (ms: number) => Promise<void>
|
|
796
871
|
}
|
|
797
872
|
|
|
798
873
|
/** Best-effort noteLiveTopic invocation — only on a non-empty topic, never throws. */
|
|
@@ -1159,6 +1234,42 @@ export async function routeSend(
|
|
|
1159
1234
|
|
|
1160
1235
|
// MISS — peer offline.
|
|
1161
1236
|
if (!deps.wake) return target // Ф1: explicit offline, no wake
|
|
1237
|
+
// LAUNCHD-REVIVE RETRY: a launchd-managed target the daemon can't wake (H4) but launchd
|
|
1238
|
+
// KeepAlive WILL revive — a MISS here is almost always a transient RESTART WINDOW (router
|
|
1239
|
+
// restart on connect / `iapeer update`, or a crash-revive), not a real outage. Wake would
|
|
1240
|
+
// just refuse (launchd-managed) and the delivery would fail in ~16ms — losing an in-flight
|
|
1241
|
+
// message (observed: natalya→arthur ok=false ms=16 during a connect router restart; a
|
|
1242
|
+
// message to a HUMAN, the silent-loss-adjacent class). Bridge it: re-resolve for a bounded
|
|
1243
|
+
// window and deliver the instant it revives. Each poll RE-RESOLVES the live target
|
|
1244
|
+
// (verify-before-act) → the ok reflects a CONFIRMED delivery (no false-OK); a genuinely
|
|
1245
|
+
// down peer still fails LOUD after the window (retryable, never silent).
|
|
1246
|
+
if (deps.isLaunchdManaged?.(personality)) {
|
|
1247
|
+
const sleep = deps.sleep ?? ((ms: number) => new Promise<void>(r => setTimeout(r, ms)))
|
|
1248
|
+
const reviveDeadline = monotonicMs() + launchdReviveGraceMs()
|
|
1249
|
+
while (monotonicMs() < reviveDeadline) {
|
|
1250
|
+
await sleep(LAUNCHD_REVIVE_POLL_MS)
|
|
1251
|
+
const revived = resolvePeerDeliveryTarget(personality, runtime, peer)
|
|
1252
|
+
if (!revived.ok) continue // not back yet — keep polling until the deadline
|
|
1253
|
+
if (revived.value.address === caller.address) return err('cannot send to self')
|
|
1254
|
+
const reviveGuard = telegramSenderGuard(caller, revived.value.runtime, personality)
|
|
1255
|
+
if (!reviveGuard.ok) return reviveGuard
|
|
1256
|
+
const delivered = await deliverWarm(revived.value, envelope, peer.cwd)
|
|
1257
|
+
if (delivered.ok) {
|
|
1258
|
+
noteTopic(deps, revived.value.address, topic)
|
|
1259
|
+
return ok({
|
|
1260
|
+
ok: true,
|
|
1261
|
+
delivered_to: { personality: revived.value.personality, runtime: revived.value.runtime },
|
|
1262
|
+
woke: false,
|
|
1263
|
+
ts: new Date().toISOString(),
|
|
1264
|
+
})
|
|
1265
|
+
}
|
|
1266
|
+
// resolved but the deliver didn't land (session still settling) → keep polling
|
|
1267
|
+
}
|
|
1268
|
+
return err(
|
|
1269
|
+
`peer "${personality}" (launchd-managed) offline and did not revive within ${launchdReviveGraceMs()}ms ` +
|
|
1270
|
+
`(restart window outlasted the grace?) — message NOT delivered; retry`,
|
|
1271
|
+
)
|
|
1272
|
+
}
|
|
1162
1273
|
const woke = await deps.wake({ personality, runtime, topic, task: envelope })
|
|
1163
1274
|
if (woke.status === 'FAILED') {
|
|
1164
1275
|
// C1 — a durably STOPPED peer is a deliberate halt, not a transient miss:
|