@agfpd/iapeer 0.4.102 → 0.4.104
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/08-peer-management.md +2 -0
- package/docs/19-notices.md +60 -24
- package/package.json +1 -1
- package/src/cli/index.ts +1 -1
- package/src/daemon/main.ts +32 -1
- package/src/daemon/mutewatch.ts +146 -23
- package/src/daemon/profilesync.ts +112 -0
|
@@ -55,6 +55,8 @@ iapeer verify --fix # repair
|
|
|
55
55
|
|
|
56
56
|
`verify` checks two things: profiles' conformance to the standard, and divergences between the registry and the local profiles. With `--fix` the command self-heals the registry from the profiles (and migrates profiles from the old runtime-field format to the current one along the way). It's the standard way to put the registry in order if it's gone out of sync.
|
|
57
57
|
|
|
58
|
+
**The daemon reconciles continuously (0.4.104).** The local `peer-profile.json` is the source of truth and the registry only its projection — but until 0.4.104 a direct edit of the source (the owner's legitimate move) propagated only when some reindexing verb happened to run afterwards; the router reads the INDEX per-request, so a stale projection mis-routed deliveries (measured live 17.07.2026: a peer's local profile said `default_runtime: claude` for days while the index kept `codex`, and deliveries went to a mute codex session). The daemon now runs a profile-sync sweep (`daemon/profilesync.ts`, 60 s, mtime/set-gated — steady state is stats only, zero writes): any change to a registered peer's local profile triggers the same locked `reindexFromLocals` the verbs use, and a boot sweep heals drift accumulated while the daemon was down. Heals are logged to `lifecycle.log` as `profile-reindex` events. `verify` remains the read-only report and the manual lever; the daemon closes the propagation gap.
|
|
59
|
+
|
|
58
60
|
## Stop and start: stop, start
|
|
59
61
|
|
|
60
62
|
```bash
|
package/docs/19-notices.md
CHANGED
|
@@ -53,19 +53,41 @@ enum, so no prose is matched.
|
|
|
53
53
|
|
|
54
54
|
### codex — the rollout
|
|
55
55
|
|
|
56
|
-
`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl
|
|
56
|
+
`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`. **The primary signal is the death
|
|
57
|
+
fingerprint, established on the real incident of 17.07.2026** (quota genuinely exhausted,
|
|
58
|
+
both codex peers mute; forensics: `docs/internals/forensics/model-limit-2026-07-17/`). The
|
|
59
|
+
rollout records **no error event**; what the refused API call leaves instead is:
|
|
57
60
|
|
|
58
61
|
```json
|
|
59
|
-
{ "
|
|
60
|
-
"
|
|
61
|
-
"
|
|
62
|
-
|
|
62
|
+
{ "payload": { "type": "token_count",
|
|
63
|
+
"info": null-or-cumulative-totals-UNCHANGED,
|
|
64
|
+
"rate_limits": { "limit_id": "premium", "primary": null, "secondary": null,
|
|
65
|
+
"credits": { "has_credits": false, "unlimited": false, "balance": "0" },
|
|
66
|
+
"rate_limit_reached_type": null } } }
|
|
67
|
+
{ "payload": { "type": "task_complete", "last_agent_message": null, "completed_at": 1784309462 } }
|
|
63
68
|
```
|
|
64
69
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
70
|
+
The detector requires the CONJUNCTION: both windows null AND no new usage recorded (`info`
|
|
71
|
+
null in a fresh session, or cumulative totals identical to the previous `token_count` — a
|
|
72
|
+
refused call consumes nothing) AND the turn closes with `task_complete{last_agent_message:
|
|
73
|
+
null}`. Each conjunct is load-bearing: a lone null-window snapshot occurs transiently while
|
|
74
|
+
the turn SURVIVES (measured 10.07.2026 — totals advanced, the turn lived on), and a
|
|
75
|
+
message-less `task_complete` alone is an absence, not a signal (§8). The event time is
|
|
76
|
+
`completed_at` (the original epoch) — never the line timestamp, which a session resume
|
|
77
|
+
REWRITES when it replays history into a fresh rollout (measured 16.07: a 14:31:58 death
|
|
78
|
+
re-recorded at 18:49:23).
|
|
79
|
+
|
|
80
|
+
`errorType` is `usage_limit_exceeded` — codex's own error-code vocabulary (0.144.1 binary) —
|
|
81
|
+
when the tail holds a window at `used_percent ≥ 100` (the evidence this is the usage-limit
|
|
82
|
+
class); that window's `resets_at` becomes the notice's reset. Without such evidence the
|
|
83
|
+
honest fallback is `api-refusal` and no reset is invented. The model comes from the dead
|
|
84
|
+
turn's own `turn_context` (real bytes name it: `gpt-5.6-sol`).
|
|
85
|
+
|
|
86
|
+
`rate_limit_reached_type` non-null is KEPT as a forward-compat second path — but the real
|
|
87
|
+
incident **refuted it as the primary signal**: with the quota exhausted and turns dying, it
|
|
88
|
+
stayed `null` in every snapshot of every rollout. When it does fire, the detector keys on
|
|
89
|
+
**non-null**, never on the variant strings (`rate_limit_reached`,
|
|
90
|
+
`workspace_owner_usage_limit_reached`, … — read out of the 0.144.1 binary).
|
|
69
91
|
|
|
70
92
|
## 2a. The signal: `peer-goal-stalled` (codex only)
|
|
71
93
|
|
|
@@ -331,14 +353,20 @@ there is nothing to derive from. Reading the file's own statement works for both
|
|
|
331
353
|
- **claude — proven live.** The transcript signal, the missing reset, the whole path
|
|
332
354
|
(detector → board → fleet surface) were exercised against a genuinely exhausted fable
|
|
333
355
|
bucket on 15.07.2026. Tests replay the captured line byte-for-byte.
|
|
334
|
-
- **codex —
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
356
|
+
- **codex — the 15.07 replay acceptance was REFUTED by the first real incident, and the
|
|
357
|
+
detector was rebuilt on the real bytes (17.07.2026).** The original detector keyed on
|
|
358
|
+
`rate_limit_reached_type` non-null — a field whose reached VALUE was synthesized at
|
|
359
|
+
acceptance (honestly flagged here at the time). When the quota genuinely ran out on
|
|
360
|
+
17.07.2026 the field stayed `null` through the entire incident and the detector missed
|
|
361
|
+
both mute peers; the owner found out by asking. The current death-fingerprint path (§2)
|
|
362
|
+
is built from and tested against the incident's verbatim bytes (four independent dying
|
|
363
|
+
sessions + a negative transient + the resume-replay trap), and proven live on the real
|
|
364
|
+
mute fleet. The reached path is retained as forward-compat only.
|
|
365
|
+
- **The blocking-window heuristic (reached path only) is unverified.** Codex does not say
|
|
366
|
+
WHICH window blocked; that path reports the **fullest** one. The fingerprint path has no
|
|
367
|
+
such ambiguity in practice: the real incident carried exactly one exhausted window, and
|
|
368
|
+
with several the LATEST reset is reported (blocked until the last wall lifts). Windows
|
|
369
|
+
ride in `content`, so a human sees the raw fact either way.
|
|
342
370
|
- **`peer-goal-stalled` — the state read is proven on real data; the CAUSE of one instance is
|
|
343
371
|
not.** The detector was run against the live goal store on 16.07.2026 and returned exactly
|
|
344
372
|
the real stall (`zapret2-oneclick`, `blocked`, goal `c25d2378…`, 09:54:32, attributed via the
|
|
@@ -356,9 +384,12 @@ Forensic evidence for all of the above: `docs/internals/forensics/model-limit-20
|
|
|
356
384
|
|
|
357
385
|
## 8. Known hole: codex generic API errors — measured, not assumed
|
|
358
386
|
|
|
359
|
-
**A codex peer left mute by
|
|
360
|
-
|
|
361
|
-
|
|
387
|
+
**A codex peer left mute by a cause that emits NO rate_limits snapshot is not detected.**
|
|
388
|
+
The usage-limit class — the actual incident class — IS covered, by the §2 death fingerprint
|
|
389
|
+
(the refused call's own snapshot; `rate_limit_reached_type` proved inert on the real
|
|
390
|
+
incident and is forward-compat only). Mute from a cause whose refusal leaves no
|
|
391
|
+
`token_count` at all (this section's fault-injected 429 on a custom provider left none) is
|
|
392
|
+
not.
|
|
362
393
|
|
|
363
394
|
This is a property of codex, not an omission here. Measured on 15.07.2026:
|
|
364
395
|
|
|
@@ -379,10 +410,15 @@ This is a property of codex, not an omission here. Measured on 15.07.2026:
|
|
|
379
410
|
So codex's only structural error surface is the `rate_limits` snapshot this detector already
|
|
380
411
|
reads. The claude/codex asymmetry in §2 is that fact, not a shortcut.
|
|
381
412
|
|
|
382
|
-
**Rejected, deliberately:** `task_complete` with `last_agent_message: null` is
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
413
|
+
**Rejected, deliberately:** `task_complete` with `last_agent_message: null` **alone** is an
|
|
414
|
+
**absence, not a signal**. A detector on it by itself would fire on every turn that
|
|
415
|
+
legitimately produced no agent message; that is not built, and the 17.07 rebuild did not
|
|
416
|
+
change it. What the real incident ADDED (this probe could not see it — the custom provider
|
|
417
|
+
emitted no `token_count` at all) is that the real ChatGPT provider's refused call leaves a
|
|
418
|
+
POSITIVE artifact: the null-window, no-new-usage `token_count` snapshot. The §2 conjunction
|
|
419
|
+
keys on that artifact and uses the message-less `task_complete` only to scope it to a turn
|
|
420
|
+
that actually died. A mute from a cause that emits NO snapshot (as in this probe) remains
|
|
421
|
+
undetected — this hole is narrowed, not closed.
|
|
386
422
|
|
|
387
423
|
**Not answered:** whether `stream_error` — a real serde variant of codex's `EventMsg`, next to
|
|
388
424
|
`deprecation_notice` and `patch_apply_begin` — is persisted to the rollout. Codex filters
|
package/package.json
CHANGED
package/src/cli/index.ts
CHANGED
|
@@ -1392,7 +1392,7 @@ const VERBS: ReadonlyArray<{ sig: string; desc: string }> = [
|
|
|
1392
1392
|
},
|
|
1393
1393
|
{
|
|
1394
1394
|
sig: 'default-runtime <runtime> (--peer <p> | --all)',
|
|
1395
|
-
desc: '
|
|
1395
|
+
desc: 'switch/set the PRIMARY runtime of peer(s) (routing/wake default) — the fleet-switch lever; refuses an undeclared runtime; registry self-healed in the same command',
|
|
1396
1396
|
},
|
|
1397
1397
|
{ sig: 'attach <peer> [runtime]', desc: 'ensure-live + resume, then attach to the pty session (Ctrl-] detaches)' },
|
|
1398
1398
|
{ sig: 'interrupt <peer> [runtime]', desc: 'interrupt the current turn (Escape) — context intact' },
|
package/src/daemon/main.ts
CHANGED
|
@@ -69,6 +69,7 @@ import { appendDeliveryEvent } from './deliverylog.ts'
|
|
|
69
69
|
import { buildFleetHandler } from './fleet.ts'
|
|
70
70
|
import { NoticeBoard } from './notices.ts'
|
|
71
71
|
import { startMuteWatch } from './mutewatch.ts'
|
|
72
|
+
import { DEFAULT_PROFILE_SYNC_INTERVAL_MS, startProfileSync } from './profilesync.ts'
|
|
72
73
|
import { startGoalWatch } from './goalwatch.ts'
|
|
73
74
|
import { ActivityBoard, startTurnWatch } from './turnwatch.ts'
|
|
74
75
|
import { defaultDaemonSocketPath, startDaemon, type DaemonHandle } from './index.ts'
|
|
@@ -348,6 +349,8 @@ export interface ConfiguredDaemonOptions {
|
|
|
348
349
|
goalWatchIntervalMs?: number
|
|
349
350
|
/** Turn-watch sweep cadence (default DEFAULT_TURNWATCH_INTERVAL_MS). */
|
|
350
351
|
turnWatchIntervalMs?: number
|
|
352
|
+
/** Profile-sync sweep cadence (default DEFAULT_PROFILE_SYNC_INTERVAL_MS). */
|
|
353
|
+
profileSyncIntervalMs?: number
|
|
351
354
|
/** Write the router.json discovery file (default true for production). */
|
|
352
355
|
discovery?: boolean
|
|
353
356
|
rootDir?: string
|
|
@@ -417,6 +420,34 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
417
420
|
} catch { /* best-effort */ }
|
|
418
421
|
},
|
|
419
422
|
})
|
|
423
|
+
// Profile-sync (identity self-heal, continuous half) — the local peer-profile.json is
|
|
424
|
+
// the SOURCE OF TRUTH and the registry only its projection; this sweep is what makes
|
|
425
|
+
// that invariant operationally true (an owner's hand-edit of the source propagates
|
|
426
|
+
// within a minute instead of waiting for someone to run a reindexing verb — the
|
|
427
|
+
// 17.07.2026 routing-desync incident class). mtime-gated: steady state is stats only.
|
|
428
|
+
const stopProfileSync = startProfileSync({
|
|
429
|
+
env,
|
|
430
|
+
intervalMs: opts.profileSyncIntervalMs ?? DEFAULT_PROFILE_SYNC_INTERVAL_MS,
|
|
431
|
+
onHealed: r => {
|
|
432
|
+
try {
|
|
433
|
+
appendLifecycleEvent(
|
|
434
|
+
cfg.eventLogDir,
|
|
435
|
+
{
|
|
436
|
+
ev: 'profile-reindex',
|
|
437
|
+
...(r.healed.length > 0 ? { healed: r.healed.join('; ').slice(0, 600) } : {}),
|
|
438
|
+
...(r.missing.length > 0 ? { missing: r.missing.join(', ').slice(0, 300) } : {}),
|
|
439
|
+
},
|
|
440
|
+
{ env },
|
|
441
|
+
)
|
|
442
|
+
} catch { /* best-effort */ }
|
|
443
|
+
},
|
|
444
|
+
onError: (err: unknown) => {
|
|
445
|
+
try {
|
|
446
|
+
const detail = err instanceof Error ? (err.stack ?? err.message) : String(err)
|
|
447
|
+
appendLifecycleEvent(cfg.eventLogDir, { ev: 'profilesync-error', error: detail.replace(/\s+/g, ' ').slice(0, 600) }, { env })
|
|
448
|
+
} catch { /* best-effort */ }
|
|
449
|
+
},
|
|
450
|
+
})
|
|
420
451
|
const handle = await startDaemon({
|
|
421
452
|
wake: makeWakeFn(cfg, env),
|
|
422
453
|
// M2 arm-on-outbound (see makeArmEphemeralOnDelivered): ephemeral caller's ok
|
|
@@ -491,7 +522,7 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
491
522
|
// The mute-watch timer is OURS, not startDaemon's — so its teardown must hang off the
|
|
492
523
|
// handle we return, or a closed daemon would leave a live timer sweeping the disk (and
|
|
493
524
|
// every test that starts a daemon would hang on exit).
|
|
494
|
-
return { ...handle, close: async () => { stopMuteWatch(); stopGoalWatch(); stopTurnWatch(); await handle.close() } }
|
|
525
|
+
return { ...handle, close: async () => { stopMuteWatch(); stopGoalWatch(); stopTurnWatch(); stopProfileSync(); await handle.close() } }
|
|
495
526
|
}
|
|
496
527
|
|
|
497
528
|
// ─────────────────────────────────────────────────────────────────────────────
|
package/src/daemon/mutewatch.ts
CHANGED
|
@@ -24,12 +24,40 @@
|
|
|
24
24
|
// notice omits the reset rather than extrapolating from the 5h/7d buckets (a DIFFERENT
|
|
25
25
|
// limit: measured live at 5h 11% / 7d 66% while fable was fully exhausted).
|
|
26
26
|
//
|
|
27
|
-
// • codex → the rollout (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl)
|
|
28
|
-
//
|
|
29
|
-
//
|
|
30
|
-
//
|
|
31
|
-
//
|
|
32
|
-
//
|
|
27
|
+
// • codex → the rollout (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl). TWO detections:
|
|
28
|
+
//
|
|
29
|
+
// (a) `rate_limit_reached_type` non-null in a token_count's RateLimitSnapshot. Kept for
|
|
30
|
+
// forward-compat, but REFUTED as the primary signal by the real incident of
|
|
31
|
+
// 17.07.2026: with the account's codex quota genuinely exhausted (both codex peers
|
|
32
|
+
// mute, turns dying on delivery), every snapshot in every rollout still said
|
|
33
|
+
// `rate_limit_reached_type: null` — the field simply does not fire on this class.
|
|
34
|
+
// The 15.07 acceptance replayed real bytes but SYNTHESIZED this field (docs/19 §7
|
|
35
|
+
// said so); the synthesis diverged from reality and the detector missed the real
|
|
36
|
+
// incident. We still key on NON-NULL (never the variant strings) in case a future
|
|
37
|
+
// codex starts populating it.
|
|
38
|
+
//
|
|
39
|
+
// (b) The REAL fingerprint of a limit-death, read out of the incident's own bytes
|
|
40
|
+
// (doc, linus, zapret2-oneclick and a minimal `codex exec` probe, 17.07.2026 —
|
|
41
|
+
// fixtures in docs/internals/forensics/model-limit-2026-07-17/): the rollout has NO
|
|
42
|
+
// error event at all; the refused API call instead emits a token_count whose
|
|
43
|
+
// rate_limits snapshot has BOTH windows null (healthy snapshots carry a windowed
|
|
44
|
+
// primary), and whose `info` recorded no new usage (null in a fresh session, or
|
|
45
|
+
// cumulative totals identical to the previous token_count — the call consumed
|
|
46
|
+
// nothing because it was refused), immediately followed by `task_complete` with
|
|
47
|
+
// `last_agent_message: null` (the turn died). All three conjuncts are required:
|
|
48
|
+
// – a lone null-window snapshot happens transiently and the turn SURVIVES
|
|
49
|
+
// (observed 10.07.2026: null windows but totals advanced, turn continued);
|
|
50
|
+
// – `task_complete{last_agent_message:null}` alone is an absence, not a signal
|
|
51
|
+
// (docs/19 §8 rejected it deliberately — legitimate no-message turns exist).
|
|
52
|
+
// The event time is `completed_at` (epoch seconds), NOT the line timestamp: a
|
|
53
|
+
// resumed session REPLAYS history with fresh line timestamps but keeps the original
|
|
54
|
+
// completed_at, so a replayed old death does not re-raise after recovery.
|
|
55
|
+
// errorType is `usage_limit_exceeded` — codex's own error-code vocabulary (read out
|
|
56
|
+
// of the 0.144.1 binary) — when the tail holds an exhausted window (used_percent
|
|
57
|
+
// ≥ 100, the evidence this is the usage-limit class); `api-refusal` (our fallback,
|
|
58
|
+
// mirroring claude's `api-error`) when it does not. resets_at comes from that
|
|
59
|
+
// exhausted window — the refusal snapshot itself carries none. The model comes from
|
|
60
|
+
// the dead turn's own turn_context (real bytes name it, e.g. `gpt-5.6-sol`).
|
|
33
61
|
//
|
|
34
62
|
// PATHS ARE NOT DERIVED FROM cwd — but NOT for the reason first claimed here. The original note
|
|
35
63
|
// said this host carries BOTH `-Users-macmini-Projects-IAPeer` and `-Users-macmini-Projects-iapeer`
|
|
@@ -156,9 +184,17 @@ function describeWindow(name: string, w: CodexWindow): string | null {
|
|
|
156
184
|
return `${name} ${w.used_percent}% (${win}${reset})`
|
|
157
185
|
}
|
|
158
186
|
|
|
187
|
+
/** A window object out of a rate_limits snapshot, or null when absent/malformed. */
|
|
188
|
+
function asWindow(w: unknown): CodexWindow | null {
|
|
189
|
+
return w && typeof w === 'object' ? (w as CodexWindow) : null
|
|
190
|
+
}
|
|
191
|
+
|
|
159
192
|
/**
|
|
160
|
-
* The codex rollout's newest limit-
|
|
161
|
-
*
|
|
193
|
+
* The codex rollout's newest limit-death evidence newer than `sinceMs`, or null. Two paths
|
|
194
|
+
* (see the header): (a) `rate_limit_reached_type` NON-NULL — kept forward-compat, refuted as
|
|
195
|
+
* the primary signal by the 17.07.2026 incident; (b) the real-bytes fingerprint — a refused
|
|
196
|
+
* API call's token_count (both windows null, no new usage) as the dying turn's last snapshot,
|
|
197
|
+
* closed by `task_complete{last_agent_message:null}`.
|
|
162
198
|
* `cwd` comes from session_meta, which is the file's FIRST line — pass the file HEAD in
|
|
163
199
|
* `head` when the tail alone may have cut it away.
|
|
164
200
|
*/
|
|
@@ -184,34 +220,121 @@ export function parseCodexRollout(text: string, sinceMs: number, head = ''): Raw
|
|
|
184
220
|
if (!cwd) return null // unattributable to a peer
|
|
185
221
|
|
|
186
222
|
let found: RawDetection | null = null
|
|
223
|
+
// Path-(b) rolling state, in file order (= time order):
|
|
224
|
+
/** Latest snapshot that still HAD a window — the exhausted-window/resets evidence. */
|
|
225
|
+
let lastWindowed: { primary: CodexWindow | null; secondary: CodexWindow | null; planType?: string } | null = null
|
|
226
|
+
/** Previous token_count's CUMULATIVE total — the refused-call (no new usage) discriminator. */
|
|
227
|
+
let prevTotal: number | null = null
|
|
228
|
+
/** True while the current turn's last token_count is refusal-shaped. */
|
|
229
|
+
let refusalPending = false
|
|
230
|
+
let lastModel: string | undefined
|
|
231
|
+
const modelByTurn = new Map<string, string>()
|
|
232
|
+
|
|
187
233
|
for (const line of text.split('\n')) {
|
|
188
234
|
const s = line.trim()
|
|
189
|
-
if (!s.startsWith('{')
|
|
190
|
-
|
|
235
|
+
if (!s.startsWith('{')) continue
|
|
236
|
+
if (!s.includes('rate_limits') && !s.includes('turn_context') && !s.includes('task_started') && !s.includes('task_complete')) continue
|
|
237
|
+
let d: {
|
|
238
|
+
timestamp?: unknown
|
|
239
|
+
type?: unknown
|
|
240
|
+
payload?: {
|
|
241
|
+
type?: unknown
|
|
242
|
+
turn_id?: unknown
|
|
243
|
+
model?: unknown
|
|
244
|
+
last_agent_message?: unknown
|
|
245
|
+
completed_at?: unknown
|
|
246
|
+
info?: { total_token_usage?: { total_tokens?: unknown } } | null
|
|
247
|
+
rate_limits?: Record<string, unknown>
|
|
248
|
+
}
|
|
249
|
+
}
|
|
191
250
|
try {
|
|
192
251
|
d = JSON.parse(s)
|
|
193
252
|
} catch {
|
|
194
253
|
continue
|
|
195
254
|
}
|
|
196
|
-
const
|
|
255
|
+
const p = d.payload
|
|
256
|
+
if (!p) continue
|
|
257
|
+
|
|
258
|
+
// The dead turn's own turn_context names the model (real bytes: "gpt-5.6-sol").
|
|
259
|
+
if (d.type === 'turn_context') {
|
|
260
|
+
if (typeof p.model === 'string' && p.model) {
|
|
261
|
+
lastModel = p.model
|
|
262
|
+
if (typeof p.turn_id === 'string') modelByTurn.set(p.turn_id, p.model)
|
|
263
|
+
}
|
|
264
|
+
continue
|
|
265
|
+
}
|
|
266
|
+
|
|
267
|
+
if (p.type === 'task_started') {
|
|
268
|
+
refusalPending = false // a refusal snapshot never crosses turn boundaries
|
|
269
|
+
continue
|
|
270
|
+
}
|
|
271
|
+
|
|
272
|
+
if (p.type === 'task_complete') {
|
|
273
|
+
// `=== null` (strict): the field is PRESENT and null in every real death; a turn that
|
|
274
|
+
// produced a message carries the message. The conjunction with refusalPending is what
|
|
275
|
+
// keeps docs/19 §8 honest — null alone is an absence, not a signal.
|
|
276
|
+
const died = refusalPending && p.last_agent_message === null
|
|
277
|
+
refusalPending = false
|
|
278
|
+
if (!died) continue
|
|
279
|
+
// Event time = completed_at (epoch s), NOT the line timestamp: a resumed session
|
|
280
|
+
// REPLAYS history with fresh line timestamps but keeps the original completed_at —
|
|
281
|
+
// keying on it stops a replayed old death from re-raising after recovery.
|
|
282
|
+
const atMs = typeof p.completed_at === 'number' ? p.completed_at * 1000 : toMs(d.timestamp)
|
|
283
|
+
if (atMs === null || atMs <= sinceMs) continue
|
|
284
|
+
const exhausted = [lastWindowed?.primary, lastWindowed?.secondary].filter(
|
|
285
|
+
(w): w is CodexWindow => !!w && typeof w.used_percent === 'number' && w.used_percent >= 100,
|
|
286
|
+
)
|
|
287
|
+
const resets = exhausted
|
|
288
|
+
.map(w => (typeof w.resets_at === 'number' ? w.resets_at * 1000 : null))
|
|
289
|
+
.filter((n): n is number => n !== null)
|
|
290
|
+
// usage_limit_exceeded = codex's own error-code vocabulary (0.144.1 binary), applied
|
|
291
|
+
// only when an exhausted window evidences the class; else our honest fallback.
|
|
292
|
+
const errorType = exhausted.length ? 'usage_limit_exceeded' : 'api-refusal'
|
|
293
|
+
const parts = lastWindowed
|
|
294
|
+
? [describeWindow('primary', lastWindowed.primary ?? {}), describeWindow('secondary', lastWindowed.secondary ?? {})].filter(Boolean)
|
|
295
|
+
: []
|
|
296
|
+
const plan = lastWindowed?.planType ? ` plan=${lastWindowed.planType}` : ''
|
|
297
|
+
found = {
|
|
298
|
+
errorType,
|
|
299
|
+
model: (typeof p.turn_id === 'string' ? modelByTurn.get(p.turn_id) : undefined) ?? lastModel,
|
|
300
|
+
content: `Codex turn died on a refused API call (${errorType})${plan}${parts.length ? ` — ${parts.join('; ')}` : ''}`,
|
|
301
|
+
resetsAtMs: resets.length ? Math.max(...resets) : undefined,
|
|
302
|
+
sessionId,
|
|
303
|
+
atMs,
|
|
304
|
+
cwd,
|
|
305
|
+
}
|
|
306
|
+
continue
|
|
307
|
+
}
|
|
308
|
+
|
|
309
|
+
const rl = p.rate_limits
|
|
197
310
|
if (!rl) continue
|
|
311
|
+
const primary = asWindow(rl.primary)
|
|
312
|
+
const secondary = asWindow(rl.secondary)
|
|
313
|
+
if (primary || secondary) lastWindowed = { primary, secondary, planType: typeof rl.plan_type === 'string' ? rl.plan_type : undefined }
|
|
314
|
+
// Did this snapshot's call consume anything? info:null (fresh session, refused outright)
|
|
315
|
+
// or cumulative totals identical to the previous token_count = NO — the refused-call
|
|
316
|
+
// discriminator. Unknown previous (tail cut) counts as usage: conservative, no guess.
|
|
317
|
+
const total = typeof p.info?.total_token_usage?.total_tokens === 'number' ? p.info.total_token_usage.total_tokens : null
|
|
318
|
+
const noUsage = p.info === null ? true : prevTotal !== null && total !== null && total === prevTotal
|
|
319
|
+
if (total !== null) prevTotal = total
|
|
320
|
+
const unlimited = (rl.credits as { unlimited?: unknown } | null | undefined)?.unlimited === true
|
|
321
|
+
refusalPending = !primary && !secondary && !unlimited && noUsage
|
|
322
|
+
|
|
323
|
+
// Path (a) — forward-compat: NON-NULL reached type, never the variant strings.
|
|
198
324
|
const reached = rl.rate_limit_reached_type
|
|
199
|
-
// NON-NULL is the signal. An unknown future variant must still detect, so we never
|
|
200
|
-
// compare against the variant strings read out of the binary.
|
|
201
325
|
if (reached === null || reached === undefined || reached === '') continue
|
|
202
326
|
const atMs = toMs(d.timestamp)
|
|
203
327
|
if (atMs === null || atMs <= sinceMs) continue
|
|
204
|
-
const
|
|
205
|
-
const
|
|
328
|
+
const pWin = primary ?? {}
|
|
329
|
+
const sWin = secondary ?? {}
|
|
206
330
|
// Which window is blocking? Codex does not say — reported for the FULLEST one, the
|
|
207
|
-
// only defensible read.
|
|
208
|
-
//
|
|
209
|
-
|
|
210
|
-
const
|
|
211
|
-
const
|
|
212
|
-
const blocking = su > pu ? secondary : primary
|
|
331
|
+
// only defensible read. Both windows ride in `content`, so the human sees the raw
|
|
332
|
+
// fact even if the heuristic picked wrong.
|
|
333
|
+
const pu = typeof pWin.used_percent === 'number' ? pWin.used_percent : -1
|
|
334
|
+
const su = typeof sWin.used_percent === 'number' ? sWin.used_percent : -1
|
|
335
|
+
const blocking = su > pu ? sWin : pWin
|
|
213
336
|
const resetsAtMs = typeof blocking.resets_at === 'number' ? blocking.resets_at * 1000 : undefined
|
|
214
|
-
const parts = [describeWindow('primary',
|
|
337
|
+
const parts = [describeWindow('primary', pWin), describeWindow('secondary', sWin)].filter(Boolean)
|
|
215
338
|
const plan = typeof rl.plan_type === 'string' ? ` plan=${rl.plan_type}` : ''
|
|
216
339
|
found = {
|
|
217
340
|
errorType: String(reached),
|
|
@@ -220,7 +343,7 @@ export function parseCodexRollout(text: string, sinceMs: number, head = ''): Raw
|
|
|
220
343
|
sessionId,
|
|
221
344
|
atMs,
|
|
222
345
|
cwd,
|
|
223
|
-
|
|
346
|
+
model: lastModel,
|
|
224
347
|
}
|
|
225
348
|
}
|
|
226
349
|
return found
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
// Profile-sync — the daemon's CONTINUOUS half of the registry self-heal invariant.
|
|
2
|
+
//
|
|
3
|
+
// THE INVARIANT (identity/profileStandard.ts; contract zone «Идентичность пира — модель,
|
|
4
|
+
// профиль, реестр»): the per-cwd peer-profile.json is the SOURCE OF TRUTH and
|
|
5
|
+
// peers-profiles.json is a regenerable PROJECTION of it — "the index is never the source
|
|
6
|
+
// of truth". Before this module that sentence was true only lexically. reindexFromLocals
|
|
7
|
+
// ran on a handful of verbs (add-runtime / default-runtime / connect / verify --fix), so
|
|
8
|
+
// an edit of the source file ITSELF — the owner's legitimate edit, on the very file the
|
|
9
|
+
// contract names as the truth — propagated NEVER unless one of those verbs happened to
|
|
10
|
+
// run afterwards. Measured on the real fleet (17.07.2026): linus's local profile had said
|
|
11
|
+
// default_runtime=claude for days; the index still said codex; the router, which reads
|
|
12
|
+
// the INDEX per-request, kept delivering into a mute codex session until an operator ran
|
|
13
|
+
// the verb by hand. A projection invariant without a continuous reconciler is a promise
|
|
14
|
+
// the system does not keep.
|
|
15
|
+
//
|
|
16
|
+
// MECHANISM. Every tick (60 s default) stat each registered peer's peer-profile.json.
|
|
17
|
+
// Only when a profile mtime advanced past the previous sweep's maximum — or the set of
|
|
18
|
+
// profile paths changed — run reindexFromLocals: the SAME locked writer the verbs use,
|
|
19
|
+
// so the registry single-writer invariant is untouched. The steady state is N stats and
|
|
20
|
+
// ZERO writes; the index is rewritten only when a source actually changed. Reindex
|
|
21
|
+
// writes the index and never the profiles, so there is no feedback loop. A corrupt or
|
|
22
|
+
// missing profile is preserved-and-reported by reindexFromLocals itself (it treats both
|
|
23
|
+
// as "leave the record, surface in `missing`" — a half-saved editor buffer must never
|
|
24
|
+
// wipe a live record).
|
|
25
|
+
//
|
|
26
|
+
// The boot tick ALWAYS reconciles once (the gate starts empty): drift accumulated while
|
|
27
|
+
// the daemon was down — exactly the incident shape — heals on startup, not on the next
|
|
28
|
+
// hand-run verb.
|
|
29
|
+
|
|
30
|
+
import { statSync } from 'node:fs'
|
|
31
|
+
import { reindexFromLocals } from '../identity/profileStandard.ts'
|
|
32
|
+
import { readPeersIndex } from '../registry/index.ts'
|
|
33
|
+
import { peerProfilePath } from '../storage/index.ts'
|
|
34
|
+
|
|
35
|
+
export const DEFAULT_PROFILE_SYNC_INTERVAL_MS = 60_000
|
|
36
|
+
|
|
37
|
+
export interface ProfileSyncResult {
|
|
38
|
+
healed: string[]
|
|
39
|
+
missing: string[]
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
export interface ProfileSyncDeps {
|
|
43
|
+
/** Registered peers' profile paths. Default: the live registry (readPeersIndex). */
|
|
44
|
+
listProfilePaths?: () => string[]
|
|
45
|
+
/** The locked self-heal writer. Default: reindexFromLocals({ env }). */
|
|
46
|
+
reindex?: () => Promise<ProfileSyncResult>
|
|
47
|
+
/** Fired ONLY when a reindex actually changed/flagged something (healed or missing
|
|
48
|
+
* non-empty) — the caller's audit hook; quiet reconciles stay quiet. */
|
|
49
|
+
onHealed?: (r: ProfileSyncResult) => void
|
|
50
|
+
/** Reported, never thrown — a reconciler must never take the daemon down. */
|
|
51
|
+
onError?: (err: unknown) => void
|
|
52
|
+
env?: NodeJS.ProcessEnv
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/** The sweep gate: the profile set + the newest profile mtime, folded into one string.
|
|
56
|
+
* A changed gate = some source changed (edit, new peer, dropped profile) → reindex. */
|
|
57
|
+
function computeGate(paths: string[]): string {
|
|
58
|
+
let maxMtime = 0
|
|
59
|
+
const present: string[] = []
|
|
60
|
+
for (const p of paths) {
|
|
61
|
+
try {
|
|
62
|
+
const m = statSync(p).mtimeMs
|
|
63
|
+
present.push(p)
|
|
64
|
+
if (m > maxMtime) maxMtime = m
|
|
65
|
+
} catch {
|
|
66
|
+
// Missing profile: participates via the PATH SET (its appearance/disappearance
|
|
67
|
+
// flips the gate once), not via mtime. reindexFromLocals handles the rest.
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
return `${maxMtime}|${present.sort().join(',')}`
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* One sweep: reindex IFF the gate moved since `lastGate`. Returns the new gate.
|
|
75
|
+
* Exported for tests; the timer below is just this on an interval.
|
|
76
|
+
*/
|
|
77
|
+
export async function profileSyncTick(deps: ProfileSyncDeps, lastGate: string): Promise<string> {
|
|
78
|
+
try {
|
|
79
|
+
const env = deps.env ?? process.env
|
|
80
|
+
const paths = deps.listProfilePaths
|
|
81
|
+
? deps.listProfilePaths()
|
|
82
|
+
: readPeersIndex({ env }).peers.map(rec => peerProfilePath(rec.cwd))
|
|
83
|
+
const gate = computeGate(paths)
|
|
84
|
+
if (gate === lastGate) return lastGate
|
|
85
|
+
const reindex = deps.reindex ?? (() => reindexFromLocals({ env }))
|
|
86
|
+
const result = await reindex()
|
|
87
|
+
if (result.healed.length > 0 || result.missing.length > 0) deps.onHealed?.(result)
|
|
88
|
+
return gate
|
|
89
|
+
} catch (e) {
|
|
90
|
+
deps.onError?.(e)
|
|
91
|
+
return lastGate // a failed sweep retries the same gate next tick — never silently skips
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
/** Start the profile-sync timer. Returns its stop function — the caller OWNS teardown. */
|
|
96
|
+
export function startProfileSync(opts: ProfileSyncDeps & { intervalMs?: number } = {}): () => void {
|
|
97
|
+
let gate = '' // empty ≠ any computed gate → the FIRST tick always reconciles (boot heal)
|
|
98
|
+
let running = false
|
|
99
|
+
const sweep = async (): Promise<void> => {
|
|
100
|
+
if (running) return // a slow reindex must never overlap itself
|
|
101
|
+
running = true
|
|
102
|
+
try {
|
|
103
|
+
gate = await profileSyncTick(opts, gate)
|
|
104
|
+
} finally {
|
|
105
|
+
running = false
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
void sweep() // boot heal — drift accumulated while the daemon was down heals NOW
|
|
109
|
+
const timer = setInterval(() => void sweep(), opts.intervalMs ?? DEFAULT_PROFILE_SYNC_INTERVAL_MS)
|
|
110
|
+
;(timer as { unref?: () => void }).unref?.()
|
|
111
|
+
return () => clearInterval(timer)
|
|
112
|
+
}
|