@agfpd/iapeer 0.4.96 → 0.4.97
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/19-notices.md +78 -8
- package/package.json +1 -1
- package/src/daemon/goalwatch.ts +385 -0
- package/src/daemon/main.ts +24 -1
package/docs/19-notices.md
CHANGED
|
@@ -29,7 +29,8 @@ renders approvals renders notices with the same plumbing.
|
|
|
29
29
|
|
|
30
30
|
## 2. The signal: `peer-mute`
|
|
31
31
|
|
|
32
|
-
The
|
|
32
|
+
The first notice kind (§2a adds `peer-goal-stalled`). It means: *a structural API error left
|
|
33
|
+
this peer unable to answer.*
|
|
33
34
|
|
|
34
35
|
The daemon does not infer this from silence — silence is unfalsifiable. It reads the fact
|
|
35
36
|
**structurally** out of the runtime's own session file. Neither runtime is scraped from the
|
|
@@ -66,6 +67,63 @@ detector keys on **non-null**, never on the variant strings (`rate_limit_reached
|
|
|
66
67
|
`workspace_owner_usage_limit_reached`, `workspace_member_credits_depleted`, … — read out of
|
|
67
68
|
the 0.144.1 binary), so a variant we have never seen still detects and reports itself.
|
|
68
69
|
|
|
70
|
+
## 2a. The signal: `peer-goal-stalled` (codex only)
|
|
71
|
+
|
|
72
|
+
The second notice kind. It means: *this peer's pinned objective stopped being worked, and
|
|
73
|
+
will not resume on its own.* Same class as `peer-mute` — peer alive, every health signal
|
|
74
|
+
green, nobody told — different evidence and a different cause.
|
|
75
|
+
|
|
76
|
+
**What the harness is.** Codex 0.144.x pins an objective to a thread (`create_goal`) and then
|
|
77
|
+
continues it AUTOMATICALLY: when the thread goes idle it injects a synthetic
|
|
78
|
+
`<codex_internal_context source="goal">` user message and runs another turn. iapeer owns no
|
|
79
|
+
part of this and writes nothing into it.
|
|
80
|
+
|
|
81
|
+
**Why it stalls.** `ext/goal/src/extension.rs::on_turn_error` maps every terminal turn error
|
|
82
|
+
that is not a usage limit to `ActiveGoalStopReason::TurnError`, which `ext/goal/src/runtime.rs`
|
|
83
|
+
writes as status `blocked`. That is the **only** path that reaches `blocked` without the model
|
|
84
|
+
calling `update_goal`. Once the status leaves `active`, `clear_active_goal()` runs and
|
|
85
|
+
automatic continuation stops **permanently**: the objective is abandoned while the peer stays
|
|
86
|
+
healthy and keeps taking unrelated turns.
|
|
87
|
+
|
|
88
|
+
**Where the truth is — the state DB, not the rollout.** Unlike `peer-mute`, this fact is *not*
|
|
89
|
+
in the session file. A live capture (16.07.2026, `zapret2-oneclick`, thread `019f6709…`) holds
|
|
90
|
+
exactly one `thread_goal_updated` event in 1971 rollout lines — the `active` one at resume —
|
|
91
|
+
and **none** for the transition to `blocked`. The transition exists only in codex's goal store:
|
|
92
|
+
|
|
93
|
+
```
|
|
94
|
+
~/.codex/goals_<n>.sqlite thread_goals(thread_id, goal_id, objective, status,
|
|
95
|
+
token_budget, tokens_used, time_used_seconds,
|
|
96
|
+
created_at_ms, updated_at_ms)
|
|
97
|
+
~/.codex/state_<n>.sqlite threads(id, cwd, …) ← the only place a thread's cwd lives
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Both are read **read-only**, and the filename generation is globbed (highest `_<n>` wins) —
|
|
101
|
+
`state_5` says codex has rolled it before, so a pinned literal name would silently read a
|
|
102
|
+
stale DB the day it rolls again. Attribution is the §6 rule unchanged: `thread_id` → `cwd`
|
|
103
|
+
(state DB) → personality (registry). A thread whose cwd is no peer's is a human's own codex
|
|
104
|
+
and is never notified.
|
|
105
|
+
|
|
106
|
+
**The reported statuses** are codex's own values, verbatim on `errorType`:
|
|
107
|
+
|
|
108
|
+
| status | meaning | notified |
|
|
109
|
+
|---|---|---|
|
|
110
|
+
| `blocked` | a turn error stopped continuation | **yes** |
|
|
111
|
+
| `usage_limited` | a usage wall stopped continuation | **yes** |
|
|
112
|
+
| `budget_limited` | the token budget stopped continuation | **yes** |
|
|
113
|
+
| `active` | being worked | no — healthy |
|
|
114
|
+
| `complete` | the model proved it done | no — success is not news |
|
|
115
|
+
| `paused` | the human's own choice | no — his own click is not news |
|
|
116
|
+
|
|
117
|
+
`blocked` and `usage_limited` stay distinct rather than flattening into one "stalled": they
|
|
118
|
+
are different facts with different remedies, and they carry different dedup identities.
|
|
119
|
+
|
|
120
|
+
**One-shot without a state file.** A stalled goal's row is never touched again — its
|
|
121
|
+
accounting is `ActiveOnly`, measured live: through 17 further minutes of turns the blocked
|
|
122
|
+
row's `tokens_used`/`updated_at_ms` stayed frozen at the block instant. So `updated_at_ms` is
|
|
123
|
+
a stable *transition* clock, and the `updated_at_ms > since` boundary fires exactly once and
|
|
124
|
+
then falls behind forever. A goal blocked while the daemon was down is missed — the same
|
|
125
|
+
honest boundary `peer-mute` already has (§7).
|
|
126
|
+
|
|
69
127
|
## 3. What the owner is told — and what is deliberately withheld
|
|
70
128
|
|
|
71
129
|
A notice states who, which runtime, which error type, which model, and when the wall lifts.
|
|
@@ -108,15 +166,15 @@ resets, the plan) — data, never our interpretation of the cause.
|
|
|
108
166
|
| field | type | notes |
|
|
109
167
|
|---|---|---|
|
|
110
168
|
| `id` | string | `n1`, `n2`, … Unique while live; **not** stable across a daemon restart. |
|
|
111
|
-
| `personality` | string | the
|
|
169
|
+
| `personality` | string | the affected peer |
|
|
112
170
|
| `runtime` | string | `claude` \| `codex` |
|
|
113
|
-
| `kind` | string | `peer-mute`
|
|
114
|
-
| `errorType` | string | the runtime's OWN value (`rate_limit`, `overloaded`, `rate_limit_reached`,
|
|
115
|
-
| `model` | string? | **absent** when the runtime did not name it |
|
|
171
|
+
| `kind` | string | `peer-mute` \| `peer-goal-stalled` (codex only). **Treat unknown kinds as renderable** — new kinds ride this surface. |
|
|
172
|
+
| `errorType` | string | the runtime's OWN value (`rate_limit`, `overloaded`, `rate_limit_reached`, …; for `peer-goal-stalled` the goal status: `blocked` \| `usage_limited` \| `budget_limited`). Free-form: do not switch exhaustively. |
|
|
173
|
+
| `model` | string? | **absent** when the runtime did not name it (always absent on `peer-goal-stalled` — the goal store names no model) |
|
|
116
174
|
| `resetsAtMs` | number? | epoch-ms; **absent** when the runtime did not state it → render "unknown", never substitute |
|
|
117
|
-
| `summary` | string | one line, e.g. `boris · claude — rate_limit (Fable 5)` |
|
|
118
|
-
| `content` | string | verbatim (claude) / rendered from typed fields (codex) |
|
|
119
|
-
| `sessionId` | string? | correlates with the on-disk transcript |
|
|
175
|
+
| `summary` | string | one line, e.g. `boris · claude — rate_limit (Fable 5)`, `zapret2-oneclick · codex — goal blocked: <objective>` |
|
|
176
|
+
| `content` | string | verbatim (claude) / rendered from typed fields (codex, both kinds) |
|
|
177
|
+
| `sessionId` | string? | correlates with the on-disk transcript. On `peer-goal-stalled` this is the codex **thread id** — the rollout's `session_meta.session_id`. |
|
|
120
178
|
| `createdMs` | number | when the notice was raised (board clock — the TTL anchor) |
|
|
121
179
|
| `lastMs` | number | when the latest occurrence was folded in (board clock). A sweep that merely re-reads an already-counted event does not move it. |
|
|
122
180
|
| `expiresMs` | number | TTL boundary |
|
|
@@ -241,6 +299,18 @@ there is nothing to derive from. Reading the file's own statement works for both
|
|
|
241
299
|
- **The blocking-window heuristic is unverified.** Codex does not say WHICH window blocked;
|
|
242
300
|
the notice reports the **fullest** one. Both windows ride in `content`, so a human sees the
|
|
243
301
|
raw fact even if the heuristic picks wrong.
|
|
302
|
+
- **`peer-goal-stalled` — the state read is proven on real data; the CAUSE of one instance is
|
|
303
|
+
not.** The detector was run against the live goal store on 16.07.2026 and returned exactly
|
|
304
|
+
the real stall (`zapret2-oneclick`, `blocked`, goal `c25d2378…`, 09:54:32, attributed via the
|
|
305
|
+
real registry), and returned nothing with the boundary at `now` — both directions on real
|
|
306
|
+
bytes, not fixtures. What is **not** proven is which turn error blocked that particular goal:
|
|
307
|
+
codex logged nothing in the 28 s spanning the transition and the rollout carries no error
|
|
308
|
+
event. That gap does not weaken the detector, which watches the **state** and is deliberately
|
|
309
|
+
cause-agnostic — it is why the notice reports codex's status verbatim and never names a cause.
|
|
310
|
+
- **The `blocked` mechanism itself is proven by source + live state**, not inferred from
|
|
311
|
+
silence: `on_turn_error` → `TurnError` → `Blocked` is the only non-`update_goal` writer in
|
|
312
|
+
the upstream tree, the model called `update_goal` zero times in the captured rollout, and the
|
|
313
|
+
blocked row's counters stayed frozen through 17 further minutes of turns.
|
|
244
314
|
|
|
245
315
|
Forensic evidence for all of the above: `docs/internals/forensics/model-limit-2026-07-15/`.
|
|
246
316
|
|
package/package.json
CHANGED
|
@@ -0,0 +1,385 @@
|
|
|
1
|
+
// Goal-watch — the daemon's detector for a codex peer whose OBJECTIVE died while the peer
|
|
2
|
+
// itself stayed perfectly healthy (docs/19). Sibling of mutewatch.ts: same class of failure
|
|
3
|
+
// (peer alive, every health signal green, nothing anywhere saying why), different evidence.
|
|
4
|
+
//
|
|
5
|
+
// WHAT CODEX'S GOAL HARNESS IS. Codex 0.144.x ships a "thread goal": an objective pinned to a
|
|
6
|
+
// thread (`create_goal`), after which codex AUTOMATICALLY continues working it — when the
|
|
7
|
+
// thread goes idle the TUI injects a synthetic `<codex_internal_context source="goal">` user
|
|
8
|
+
// message and runs another turn. That loop is the harness's own; iapeer owns NO part of it and
|
|
9
|
+
// this module WRITES nothing — it only reads what codex already wrote about itself.
|
|
10
|
+
//
|
|
11
|
+
// THE FAILURE. `ext/goal/src/extension.rs::on_turn_error` maps EVERY terminal turn error that
|
|
12
|
+
// is not a usage limit to `ActiveGoalStopReason::TurnError`, and `runtime.rs` turns that into
|
|
13
|
+
// status `blocked`. Upstream calls this deliberate ("Block the goal to prevent automatic
|
|
14
|
+
// continuation from looping and consuming tokens, as can happen with compaction errors"), and
|
|
15
|
+
// it is the ONLY path that writes `blocked` without the model calling `update_goal`. Once the
|
|
16
|
+
// status leaves `active`, `clear_active_goal()` runs and automatic continuation STOPS FOREVER.
|
|
17
|
+
// The objective is abandoned. Nothing dies, nothing errors, no exit code, no IAP message —
|
|
18
|
+
// the peer just never works the objective again. Measured live on this host 16.07.2026
|
|
19
|
+
// (zapret2-oneclick, thread 019f6709…, goal c25d2378…): status `blocked` at 09:54:32 with no
|
|
20
|
+
// `update_goal` call anywhere in the rollout, while the peer went on running unrelated turns
|
|
21
|
+
// for another 17 minutes with its goal counters FROZEN at the block instant.
|
|
22
|
+
//
|
|
23
|
+
// WHY SQLITE AND NOT THE ROLLOUT. mutewatch reads the rollout because the rate-limit fact is
|
|
24
|
+
// written there. The block is NOT: the same live capture shows exactly ONE `thread_goal_updated`
|
|
25
|
+
// event in 1971 rollout lines (the `active` one at resume) and NONE for the transition to
|
|
26
|
+
// `blocked`. The transition is written ONLY to codex's goals DB. Read the state or miss it.
|
|
27
|
+
//
|
|
28
|
+
// WHY TWO DATABASES. The goals DB keys on thread_id and carries no cwd, so a goal row alone
|
|
29
|
+
// cannot be attributed to a peer. Codex's state DB (`threads`) carries `id` + `cwd`. Joining
|
|
30
|
+
// the two yields cwd, and cwd → personality is the SAME attribution rule mutewatch already
|
|
31
|
+
// uses: a thread whose cwd is no peer's (a human's own codex) is ignored, never notified.
|
|
32
|
+
//
|
|
33
|
+
// WHY THE FILENAMES ARE GLOBBED. These are codex's private files and their names carry a
|
|
34
|
+
// schema generation (`goals_1.sqlite`, `state_5.sqlite` — the 5 says it has rolled before).
|
|
35
|
+
// Pinning the literal name would silently read a stale DB the day codex rolls it, so the
|
|
36
|
+
// highest generation present wins. Absent/unreadable/renamed → the detector no-ops. Reading
|
|
37
|
+
// another product's internal store is a real coupling; it is accepted for the same reason
|
|
38
|
+
// mutewatch parses codex's rollout — there is no other place the fact exists — and it is
|
|
39
|
+
// confined to this module, guarded, and READ-ONLY.
|
|
40
|
+
//
|
|
41
|
+
// ONE-SHOT, WITHOUT A STATE FILE. A blocked goal's row is never touched again (its accounting
|
|
42
|
+
// is ActiveOnly — proven live: counters frozen through 17 minutes of later turns), so its
|
|
43
|
+
// `updated_at_ms` is a STABLE transition clock. Filtering on `updated_at_ms > sinceMs` there-
|
|
44
|
+
// fore fires exactly once per transition and then falls behind the boundary forever — no
|
|
45
|
+
// repeat, no TTL re-raise, no persisted "already told him" set. A goal blocked while the
|
|
46
|
+
// daemon was down is missed, exactly as mutewatch misses an error it was not up for; the
|
|
47
|
+
// boundary is the contour's existing honesty, not a new compromise.
|
|
48
|
+
|
|
49
|
+
import { existsSync, readdirSync } from 'node:fs'
|
|
50
|
+
import { join } from 'node:path'
|
|
51
|
+
import type { LifecycleConfig } from '../lifecycle/index.ts'
|
|
52
|
+
import type { NoticeBoard } from './notices.ts'
|
|
53
|
+
|
|
54
|
+
/** Overlap on the since-boundary so a transition landing between two sweeps is never missed.
|
|
55
|
+
* Re-reads are free: the board folds them by `eventAtMs` (see notices.ts). */
|
|
56
|
+
const SCAN_OVERLAP_MS = 15_000
|
|
57
|
+
/** Objective text is user-authored and unbounded — bound it before it reaches a notice. */
|
|
58
|
+
const OBJECTIVE_MAX = 300
|
|
59
|
+
|
|
60
|
+
/**
|
|
61
|
+
* The goal statuses that mean "this objective is not being worked and will not resume on its
|
|
62
|
+
* own". CODEX'S OWN taxonomy values, never our invention (docs/19 §2).
|
|
63
|
+
*
|
|
64
|
+
* Deliberately excluded:
|
|
65
|
+
* • `active` — working; the healthy case.
|
|
66
|
+
* • `complete` — the model proved the objective done. Success is not news.
|
|
67
|
+
* • `paused` — a human's own choice. Notifying him of his own click is noise.
|
|
68
|
+
* `usage_limited` / `budget_limited` ride along because they are the same shape: continuation
|
|
69
|
+
* stopped, objective unfinished, nobody told. They are NOT the same CAUSE as `blocked`, which
|
|
70
|
+
* is why the status travels verbatim as the notice's errorType instead of being flattened.
|
|
71
|
+
*/
|
|
72
|
+
export const STALLED_GOAL_STATUSES = new Set(['blocked', 'usage_limited', 'budget_limited'])
|
|
73
|
+
|
|
74
|
+
/** A row of codex's `thread_goals`, as codex wrote it. */
|
|
75
|
+
export interface GoalRow {
|
|
76
|
+
threadId: string
|
|
77
|
+
goalId: string
|
|
78
|
+
objective: string
|
|
79
|
+
status: string
|
|
80
|
+
tokensUsed: number
|
|
81
|
+
timeUsedSeconds: number
|
|
82
|
+
updatedAtMs: number
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
export interface GoalDetection {
|
|
86
|
+
personality: string
|
|
87
|
+
runtime: 'codex'
|
|
88
|
+
/** Codex's own status value — `blocked` | `usage_limited` | `budget_limited`. */
|
|
89
|
+
status: string
|
|
90
|
+
goalId: string
|
|
91
|
+
threadId: string
|
|
92
|
+
objective: string
|
|
93
|
+
tokensUsed: number
|
|
94
|
+
timeUsedSeconds: number
|
|
95
|
+
/** The row's `updated_at_ms` — the TRANSITION's own clock, not our sweep's. */
|
|
96
|
+
atMs: number
|
|
97
|
+
cwd: string
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
101
|
+
// Pure selection — rows in, detections out (fs/sqlite-free: tests need no DB)
|
|
102
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
103
|
+
|
|
104
|
+
function clip(s: string, max: number): string {
|
|
105
|
+
const t = s.replace(/\s+/g, ' ').trim()
|
|
106
|
+
return t.length <= max ? t : `${t.slice(0, max - 1)}…`
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
/** The human-readable line, rendered from codex's TYPED fields — codex writes no prose for a
|
|
110
|
+
* goal transition, the same situation as its rate-limit snapshot, and the same answer:
|
|
111
|
+
* render the facts, never our reading of the cause. */
|
|
112
|
+
export function renderGoalContent(d: Omit<GoalDetection, 'personality' | 'runtime'>): string {
|
|
113
|
+
return [
|
|
114
|
+
`Codex goal ${d.status} — automatic goal continuation has stopped; this objective will not`,
|
|
115
|
+
`resume without an explicit resume. Objective: "${d.objective}".`,
|
|
116
|
+
`goal=${d.goalId} thread=${d.threadId} accounted=${d.tokensUsed} tokens / ${d.timeUsedSeconds}s.`,
|
|
117
|
+
].join(' ')
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
/**
|
|
121
|
+
* Every peer-attributable goal that stalled STRICTLY after `sinceMs`.
|
|
122
|
+
*
|
|
123
|
+
* `cwdByThread` and `peerByCwd` are injected so this stays pure: the whole selection rule is
|
|
124
|
+
* testable without a codex install.
|
|
125
|
+
*/
|
|
126
|
+
export function selectStalledGoals(
|
|
127
|
+
goals: GoalRow[],
|
|
128
|
+
cwdByThread: Map<string, string>,
|
|
129
|
+
peerByCwd: (cwd: string) => string | undefined,
|
|
130
|
+
sinceMs: number,
|
|
131
|
+
): GoalDetection[] {
|
|
132
|
+
const out: GoalDetection[] = []
|
|
133
|
+
for (const g of goals) {
|
|
134
|
+
if (!STALLED_GOAL_STATUSES.has(g.status)) continue
|
|
135
|
+
if (!Number.isFinite(g.updatedAtMs) || g.updatedAtMs <= sinceMs) continue
|
|
136
|
+
const cwd = cwdByThread.get(g.threadId)
|
|
137
|
+
if (!cwd) continue // thread not in the state DB → unattributable; ignore rather than guess
|
|
138
|
+
const personality = peerByCwd(cwd)
|
|
139
|
+
if (!personality) continue // a human's own codex thread — never notify
|
|
140
|
+
out.push({
|
|
141
|
+
personality,
|
|
142
|
+
runtime: 'codex',
|
|
143
|
+
status: g.status,
|
|
144
|
+
goalId: g.goalId,
|
|
145
|
+
threadId: g.threadId,
|
|
146
|
+
objective: clip(g.objective, OBJECTIVE_MAX),
|
|
147
|
+
tokensUsed: g.tokensUsed,
|
|
148
|
+
timeUsedSeconds: g.timeUsedSeconds,
|
|
149
|
+
atMs: g.updatedAtMs,
|
|
150
|
+
cwd,
|
|
151
|
+
})
|
|
152
|
+
}
|
|
153
|
+
return out
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
157
|
+
// The codex-state reader — the only part that touches another product's files
|
|
158
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
159
|
+
|
|
160
|
+
export interface GoalWatchPaths {
|
|
161
|
+
/** ~/.codex — INJECTED, never re-resolved from env inside a sweep. */
|
|
162
|
+
codexHome: string
|
|
163
|
+
}
|
|
164
|
+
|
|
165
|
+
export function resolveGoalWatchPaths(env: NodeJS.ProcessEnv): GoalWatchPaths {
|
|
166
|
+
const home = env.HOME?.trim() || ''
|
|
167
|
+
return { codexHome: env.IAPEER_CODEX_HOME?.trim() || join(home, '.codex') }
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
/**
|
|
171
|
+
* The highest-generation `<prefix>_<n>.sqlite` in `dir`, or null.
|
|
172
|
+
* Codex rolls the generation when it changes schema; the newest is the live one.
|
|
173
|
+
*/
|
|
174
|
+
export function findVersionedDb(dir: string, prefix: string): string | null {
|
|
175
|
+
if (!existsSync(dir)) return null
|
|
176
|
+
let best: { n: number; name: string } | null = null
|
|
177
|
+
let names: string[]
|
|
178
|
+
try {
|
|
179
|
+
names = readdirSync(dir)
|
|
180
|
+
} catch {
|
|
181
|
+
return null
|
|
182
|
+
}
|
|
183
|
+
const re = new RegExp(`^${prefix}_(\\d+)\\.sqlite$`)
|
|
184
|
+
for (const name of names) {
|
|
185
|
+
const m = re.exec(name)
|
|
186
|
+
if (!m) continue
|
|
187
|
+
const n = parseInt(m[1] as string, 10)
|
|
188
|
+
if (!Number.isFinite(n)) continue
|
|
189
|
+
if (!best || n > best.n) best = { n, name }
|
|
190
|
+
}
|
|
191
|
+
return best ? join(dir, best.name) : null
|
|
192
|
+
}
|
|
193
|
+
|
|
194
|
+
/** Open a codex DB READ-ONLY. Never creates, never migrates, never writes. */
|
|
195
|
+
async function openReadOnly(path: string): Promise<{ query: (sql: string) => { all: (...a: unknown[]) => unknown[] }; close: () => void } | null> {
|
|
196
|
+
try {
|
|
197
|
+
const { Database } = (await import('bun:sqlite')) as typeof import('bun:sqlite')
|
|
198
|
+
return new Database(path, { readonly: true }) as unknown as {
|
|
199
|
+
query: (sql: string) => { all: (...a: unknown[]) => unknown[] }
|
|
200
|
+
close: () => void
|
|
201
|
+
}
|
|
202
|
+
} catch {
|
|
203
|
+
return null // absent / locked / rolled schema → the detector no-ops, never throws
|
|
204
|
+
}
|
|
205
|
+
}
|
|
206
|
+
|
|
207
|
+
function num(v: unknown): number {
|
|
208
|
+
return typeof v === 'number' && Number.isFinite(v) ? v : 0
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
function str(v: unknown): string {
|
|
212
|
+
return typeof v === 'string' ? v : ''
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
/** Codex's goal rows. Empty on any shape we do not recognise — never a throw. */
|
|
216
|
+
export async function readGoalRows(codexHome: string): Promise<GoalRow[]> {
|
|
217
|
+
const path = findVersionedDb(codexHome, 'goals')
|
|
218
|
+
if (!path) return []
|
|
219
|
+
const db = await openReadOnly(path)
|
|
220
|
+
if (!db) return []
|
|
221
|
+
try {
|
|
222
|
+
const rows = db
|
|
223
|
+
.query('SELECT thread_id, goal_id, objective, status, tokens_used, time_used_seconds, updated_at_ms FROM thread_goals')
|
|
224
|
+
.all() as Array<Record<string, unknown>>
|
|
225
|
+
return rows.map(r => ({
|
|
226
|
+
threadId: str(r.thread_id),
|
|
227
|
+
goalId: str(r.goal_id),
|
|
228
|
+
objective: str(r.objective),
|
|
229
|
+
status: str(r.status),
|
|
230
|
+
tokensUsed: num(r.tokens_used),
|
|
231
|
+
timeUsedSeconds: num(r.time_used_seconds),
|
|
232
|
+
updatedAtMs: num(r.updated_at_ms),
|
|
233
|
+
}))
|
|
234
|
+
} catch {
|
|
235
|
+
return []
|
|
236
|
+
} finally {
|
|
237
|
+
try {
|
|
238
|
+
db.close()
|
|
239
|
+
} catch {
|
|
240
|
+
/* best-effort */
|
|
241
|
+
}
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
/** thread_id → cwd, for the given threads only. Empty on any unrecognised shape. */
|
|
246
|
+
export async function readThreadCwds(codexHome: string, threadIds: string[]): Promise<Map<string, string>> {
|
|
247
|
+
const out = new Map<string, string>()
|
|
248
|
+
if (threadIds.length === 0) return out
|
|
249
|
+
const path = findVersionedDb(codexHome, 'state')
|
|
250
|
+
if (!path) return out
|
|
251
|
+
const db = await openReadOnly(path)
|
|
252
|
+
if (!db) return out
|
|
253
|
+
try {
|
|
254
|
+
const holes = threadIds.map(() => '?').join(',')
|
|
255
|
+
const rows = db.query(`SELECT id, cwd FROM threads WHERE id IN (${holes})`).all(...threadIds) as Array<Record<string, unknown>>
|
|
256
|
+
for (const r of rows) {
|
|
257
|
+
const id = str(r.id)
|
|
258
|
+
const cwd = str(r.cwd)
|
|
259
|
+
if (id && cwd) out.set(id, cwd)
|
|
260
|
+
}
|
|
261
|
+
} catch {
|
|
262
|
+
/* unrecognised schema → no attribution, no notice */
|
|
263
|
+
} finally {
|
|
264
|
+
try {
|
|
265
|
+
db.close()
|
|
266
|
+
} catch {
|
|
267
|
+
/* best-effort */
|
|
268
|
+
}
|
|
269
|
+
}
|
|
270
|
+
return out
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
export interface GoalScanDeps extends GoalWatchPaths {
|
|
274
|
+
/** personality ← cwd. Built from the registry by the composition point. */
|
|
275
|
+
peerByCwd: (cwd: string) => string | undefined
|
|
276
|
+
}
|
|
277
|
+
|
|
278
|
+
/** One sweep: every peer-attributable goal that stalled since `sinceMs`. */
|
|
279
|
+
export async function scanStalledGoals(deps: GoalScanDeps, sinceMs: number): Promise<GoalDetection[]> {
|
|
280
|
+
const floor = sinceMs - SCAN_OVERLAP_MS
|
|
281
|
+
const goals = await readGoalRows(deps.codexHome)
|
|
282
|
+
// Only the stalled rows are worth a threads lookup — an active goal is the common case.
|
|
283
|
+
const candidates = goals.filter(g => STALLED_GOAL_STATUSES.has(g.status) && g.updatedAtMs > floor)
|
|
284
|
+
if (candidates.length === 0) return []
|
|
285
|
+
const cwds = await readThreadCwds(
|
|
286
|
+
deps.codexHome,
|
|
287
|
+
candidates.map(g => g.threadId).filter(Boolean),
|
|
288
|
+
)
|
|
289
|
+
return selectStalledGoals(candidates, cwds, deps.peerByCwd, floor)
|
|
290
|
+
}
|
|
291
|
+
|
|
292
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
293
|
+
// The tick
|
|
294
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
295
|
+
|
|
296
|
+
export interface GoalWatchDeps extends GoalScanDeps {
|
|
297
|
+
board: NoticeBoard
|
|
298
|
+
now?: () => number
|
|
299
|
+
/** Reported, never thrown — a detector must never take the daemon down. */
|
|
300
|
+
onError?: (err: unknown) => void
|
|
301
|
+
}
|
|
302
|
+
|
|
303
|
+
/** Run one watch tick: sweep, raise. Returns notices actually RAISED + the new boundary. */
|
|
304
|
+
export async function goalWatchTick(deps: GoalWatchDeps, sinceMs: number): Promise<{ raised: number; sinceMs: number }> {
|
|
305
|
+
const now = deps.now ?? Date.now
|
|
306
|
+
const startedMs = now()
|
|
307
|
+
let raised = 0
|
|
308
|
+
try {
|
|
309
|
+
for (const d of await scanStalledGoals(deps, sinceMs)) {
|
|
310
|
+
const { deduped } = deps.board.raise({
|
|
311
|
+
personality: d.personality,
|
|
312
|
+
runtime: d.runtime,
|
|
313
|
+
kind: 'peer-goal-stalled',
|
|
314
|
+
// Codex's own status is the taxonomy value — `blocked` and `usage_limited` are
|
|
315
|
+
// different facts for the owner and must not collapse into one dedup identity.
|
|
316
|
+
errorType: d.status,
|
|
317
|
+
content: renderGoalContent(d),
|
|
318
|
+
// The thread IS the session: it correlates the notice with the rollout on disk.
|
|
319
|
+
sessionId: d.threadId,
|
|
320
|
+
// The TRANSITION's own clock — see notices.ts on why this is not the sweep's.
|
|
321
|
+
eventAtMs: d.atMs,
|
|
322
|
+
summary: `${d.personality} · codex — goal ${d.status}: ${clip(d.objective, 80)}`,
|
|
323
|
+
})
|
|
324
|
+
if (!deduped) raised += 1
|
|
325
|
+
}
|
|
326
|
+
} catch (e) {
|
|
327
|
+
deps.onError?.(e)
|
|
328
|
+
}
|
|
329
|
+
return { raised, sinceMs: startedMs }
|
|
330
|
+
}
|
|
331
|
+
|
|
332
|
+
export interface StartGoalWatchOptions {
|
|
333
|
+
env?: NodeJS.ProcessEnv
|
|
334
|
+
intervalMs?: number
|
|
335
|
+
paths?: GoalWatchPaths
|
|
336
|
+
/** Override the registry read (tests). Default: a lazy import of the CLI's listPeers. */
|
|
337
|
+
listPeers?: () => Array<{ personality: string; cwd: string }>
|
|
338
|
+
now?: () => number
|
|
339
|
+
onError?: (err: unknown) => void
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
/**
|
|
343
|
+
* Start the goal-watch timer. Returns its stop function — the caller OWNS teardown.
|
|
344
|
+
* Mirrors startMuteWatch: registry re-read every sweep, `listPeers` imported lazily.
|
|
345
|
+
*/
|
|
346
|
+
export function startGoalWatch(_cfg: LifecycleConfig, board: NoticeBoard, opts: StartGoalWatchOptions = {}): () => void {
|
|
347
|
+
const env = opts.env ?? process.env
|
|
348
|
+
const paths = opts.paths ?? resolveGoalWatchPaths(env)
|
|
349
|
+
const now = opts.now ?? Date.now
|
|
350
|
+
let listPeers = opts.listPeers
|
|
351
|
+
let since = now()
|
|
352
|
+
let running = false
|
|
353
|
+
|
|
354
|
+
const sweep = async (): Promise<void> => {
|
|
355
|
+
if (running) return // a slow sweep must never overlap itself into a stampede
|
|
356
|
+
running = true
|
|
357
|
+
try {
|
|
358
|
+
if (!listPeers) {
|
|
359
|
+
const cli = await import('../cli/index.ts')
|
|
360
|
+
listPeers = () => cli.listPeers({ env }).map(p => ({ personality: p.personality, cwd: p.cwd }))
|
|
361
|
+
}
|
|
362
|
+
const rows = listPeers()
|
|
363
|
+
// Case-insensitive, trailing-slash-tolerant — same rule and same reason as mutewatch:
|
|
364
|
+
// macOS paths are case-insensitive and codex records the cwd STRING as launched.
|
|
365
|
+
const byCwd = new Map<string, string>()
|
|
366
|
+
for (const r of rows) if (r.cwd) byCwd.set(r.cwd.replace(/\/+$/, '').toLowerCase(), r.personality)
|
|
367
|
+
const deps: GoalWatchDeps = {
|
|
368
|
+
...paths,
|
|
369
|
+
board,
|
|
370
|
+
now,
|
|
371
|
+
onError: opts.onError,
|
|
372
|
+
peerByCwd: (cwd: string) => byCwd.get(cwd.replace(/\/+$/, '').toLowerCase()),
|
|
373
|
+
}
|
|
374
|
+
since = (await goalWatchTick(deps, since)).sinceMs
|
|
375
|
+
} catch (e) {
|
|
376
|
+
opts.onError?.(e)
|
|
377
|
+
} finally {
|
|
378
|
+
running = false
|
|
379
|
+
}
|
|
380
|
+
}
|
|
381
|
+
|
|
382
|
+
const timer = setInterval(() => void sweep(), opts.intervalMs ?? 20_000)
|
|
383
|
+
;(timer as { unref?: () => void }).unref?.()
|
|
384
|
+
return () => clearInterval(timer)
|
|
385
|
+
}
|
package/src/daemon/main.ts
CHANGED
|
@@ -69,6 +69,7 @@ import { appendDeliveryEvent } from './deliverylog.ts'
|
|
|
69
69
|
import { buildFleetHandler } from './fleet.ts'
|
|
70
70
|
import { NoticeBoard } from './notices.ts'
|
|
71
71
|
import { startMuteWatch } from './mutewatch.ts'
|
|
72
|
+
import { startGoalWatch } from './goalwatch.ts'
|
|
72
73
|
import { defaultDaemonSocketPath, startDaemon, type DaemonHandle } from './index.ts'
|
|
73
74
|
|
|
74
75
|
/** Default TCP loopback port for the always-on router. Real http MCP clients
|
|
@@ -100,6 +101,12 @@ export const DEFAULT_SUPERVISE_INTERVAL_MS = 60_000
|
|
|
100
101
|
* to spare. The sweep is cheap — it stats only files that CHANGED since the last pass. */
|
|
101
102
|
export const DEFAULT_MUTEWATCH_INTERVAL_MS = 20_000
|
|
102
103
|
|
|
104
|
+
/** Goal-watch cadence (docs/19) — how often the daemon reads codex's goal store for an
|
|
105
|
+
* objective that stopped continuing. Same 20 s and the same reason as the mute-watch: the
|
|
106
|
+
* owner must learn within a minute. Cheaper than that sweep — two indexed reads of a small
|
|
107
|
+
* DB, and the threads join runs ONLY when a stalled row is actually present. */
|
|
108
|
+
export const DEFAULT_GOALWATCH_INTERVAL_MS = 20_000
|
|
109
|
+
|
|
103
110
|
// This module's own path — the launchd plist runs `bun <this>` as the daemon.
|
|
104
111
|
const DAEMON_MAIN_PATH = fileURLToPath(import.meta.url)
|
|
105
112
|
|
|
@@ -330,6 +337,8 @@ export interface ConfiguredDaemonOptions {
|
|
|
330
337
|
superviseIntervalMs?: number
|
|
331
338
|
/** Mute-watch sweep cadence (default DEFAULT_MUTEWATCH_INTERVAL_MS). */
|
|
332
339
|
muteWatchIntervalMs?: number
|
|
340
|
+
/** Goal-watch sweep cadence (default DEFAULT_GOALWATCH_INTERVAL_MS). */
|
|
341
|
+
goalWatchIntervalMs?: number
|
|
333
342
|
/** Write the router.json discovery file (default true for production). */
|
|
334
343
|
discovery?: boolean
|
|
335
344
|
rootDir?: string
|
|
@@ -369,6 +378,20 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
369
378
|
} catch { /* best-effort */ }
|
|
370
379
|
},
|
|
371
380
|
})
|
|
381
|
+
// Goal-watch (docs/19) — the SAME board and the same class as the mute-watch: a peer left
|
|
382
|
+
// alive and green while the thing it was told to do quietly stopped. Separate detector
|
|
383
|
+
// because the evidence is elsewhere (codex's goal store, not a transcript) and it is
|
|
384
|
+
// codex-only; sharing the board is what makes it one owner-facing surface.
|
|
385
|
+
const stopGoalWatch = startGoalWatch(cfg, board, {
|
|
386
|
+
env,
|
|
387
|
+
intervalMs: opts.goalWatchIntervalMs ?? DEFAULT_GOALWATCH_INTERVAL_MS,
|
|
388
|
+
onError: (err: unknown) => {
|
|
389
|
+
try {
|
|
390
|
+
const detail = err instanceof Error ? (err.stack ?? err.message) : String(err)
|
|
391
|
+
appendLifecycleEvent(cfg.eventLogDir, { ev: 'goalwatch-error', error: detail.replace(/\s+/g, ' ').slice(0, 600) }, { env })
|
|
392
|
+
} catch { /* best-effort */ }
|
|
393
|
+
},
|
|
394
|
+
})
|
|
372
395
|
const handle = await startDaemon({
|
|
373
396
|
wake: makeWakeFn(cfg, env),
|
|
374
397
|
// M2 arm-on-outbound (see makeArmEphemeralOnDelivered): ephemeral caller's ok
|
|
@@ -443,7 +466,7 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
443
466
|
// The mute-watch timer is OURS, not startDaemon's — so its teardown must hang off the
|
|
444
467
|
// handle we return, or a closed daemon would leave a live timer sweeping the disk (and
|
|
445
468
|
// every test that starts a daemon would hang on exit).
|
|
446
|
-
return { ...handle, close: async () => { stopMuteWatch(); await handle.close() } }
|
|
469
|
+
return { ...handle, close: async () => { stopMuteWatch(); stopGoalWatch(); await handle.close() } }
|
|
447
470
|
}
|
|
448
471
|
|
|
449
472
|
// ─────────────────────────────────────────────────────────────────────────────
|