@agfpd/iapeer 0.4.91 → 0.4.93
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/15-fleet-api.md +12 -4
- package/docs/19-notices.md +195 -0
- package/docs/README.md +1 -0
- package/package.json +1 -1
- package/src/approval/brokerClient.ts +10 -0
- package/src/daemon/fleet.ts +77 -1
- package/src/daemon/main.ts +34 -2
- package/src/daemon/mutewatch.ts +438 -0
- package/src/daemon/notices.ts +190 -0
- package/src/daemon/noticeslog.ts +54 -0
- package/src/supervisor/approvalRoute.ts +75 -7
- package/src/supervisor/daemon.ts +11 -0
package/docs/15-fleet-api.md
CHANGED
|
@@ -63,6 +63,13 @@ The full fleet state. The peer rows are produced by the **same in-process functi
|
|
|
63
63
|
{ "id": "a1", "personality": "boris", "runtime": "claude", "kind": "circuit-breaker",
|
|
64
64
|
"tool": "dangerous-rm", "summary": "rm -rf /tmp/x",
|
|
65
65
|
"content": "cmd=\"rm -rf /tmp/x\" target=\"/tmp/x\"", "createdMs": 1751731100000, "expiresMs": 1751731400000 }
|
|
66
|
+
],
|
|
67
|
+
"notices": [
|
|
68
|
+
{ "id": "n1", "personality": "boris", "runtime": "claude", "kind": "peer-mute",
|
|
69
|
+
"errorType": "rate_limit", "model": "Fable 5", "summary": "boris · claude — rate_limit (Fable 5)",
|
|
70
|
+
"content": "You've reached your Fable 5 limit. Run /usage-credits to continue or switch models with /model.",
|
|
71
|
+
"sessionId": "85ea79c9-…", "createdMs": 1751731100000, "lastMs": 1751731160000,
|
|
72
|
+
"expiresMs": 1751734700000, "count": 3 }
|
|
66
73
|
]
|
|
67
74
|
}
|
|
68
75
|
```
|
|
@@ -81,6 +88,7 @@ Field notes:
|
|
|
81
88
|
- The daemon answering **is** the health signal: there is no `healthy` boolean. A dashboard that cannot reach any advertised address should render the daemon as down (this is also why a purely API-fed dashboard is insufficient when the daemon itself is broken — the built-in TUI keeps its direct in-process read path for exactly that case).
|
|
82
89
|
- The peer's **LLM model is deliberately absent**: the effective model of a live session is not an observable registry fact (static launch pins exist, but reporting a pin as «the model» would lie under runtime auto-switching). If it ever becomes observable, it will appear as an additive field.
|
|
83
90
|
- `approvals` (top-level, docs/17-approval) — the pending human-approval queue, the SAME items `GET /fleet/v1/approvals` returns, carried in the snapshot so a client that already re-fetches `/snapshot` on every event renders the queue with no extra call. **ADDITIVE + OMITTED when the queue is empty** — a client MUST treat its absence as an empty queue (the same rule as `approval_mode`). Each item: `id` (broker id, the target of `POST /fleet/v1/approvals/<id>/(approve|deny)`), `personality` + `runtime` (whose action), `kind` (`tool`|`plan`|`question`|`circuit-breaker`), `tool` (the tool / breaker name), `summary` (a one-line badge string), `content` (the FULL verbatim action — command / diff / plan; criterion #7), `createdMs` / `expiresMs`. The broker's item is a superset — unknown extra fields (`title`, `approvers`) MUST be ignored (obligation 2).
|
|
91
|
+
- `notices` (top-level, docs/19-notices) — live OWNER-facing notices: things the daemon noticed that the peer itself could not report, the v1 case being a structural API error that left it MUTE (`kind: peer-mute`). The SAME items `GET /fleet/v1/notices` returns. **ADDITIVE + OMITTED when the board is empty** — absence ⇒ empty board (the same rule as `approvals`). **READ-ONLY**: a notice is one-way, there is nothing to approve or deny. Each item: `id`, `personality` + `runtime` (who went mute), `kind` (`peer-mute`; treat unknown kinds as renderable), `errorType` (the RUNTIME's own value — `rate_limit`, `overloaded`, `rate_limit_reached`, … — free-form, do not switch exhaustively), `model?`, `resetsAtMs?`, `summary`, `content` (verbatim for claude), `sessionId?`, `createdMs` / `lastMs` / `expiresMs`, `count` (detections folded in — render as `×N`). **`model` and `resetsAtMs` are ABSENT when the runtime did not state them** — absence means *the runtime did not say*, NOT *there is no limit*; a client MUST render unknown and MUST NOT substitute the 5h/7d reset, which belongs to a different limit (docs/19 §3).
|
|
84
92
|
|
|
85
93
|
## GET /fleet/v1/peers/<personality>
|
|
86
94
|
|
|
@@ -95,7 +103,7 @@ One peer's card: the same object as the snapshot row, plus its recent history.
|
|
|
95
103
|
|
|
96
104
|
## GET /fleet/v1/events — SSE stream
|
|
97
105
|
|
|
98
|
-
`text/event-stream` of fleet events, sourced by tail-following the daemon's **durable logs** (`lifecycle.log`, `delivery.log`, `exits.log`). Those files are written by *every* participating process — the daemon, CLI verbs (attach/stop/wake), the pty supervisors — so the stream covers events an in-daemon bus would miss.
|
|
106
|
+
`text/event-stream` of fleet events, sourced by tail-following the daemon's **durable logs** (`lifecycle.log`, `delivery.log`, `exits.log`, `approvals.log`, `notices.log`). Those files are written by *every* participating process — the daemon, CLI verbs (attach/stop/wake), the pty supervisors — so the stream covers events an in-daemon bus would miss.
|
|
99
107
|
|
|
100
108
|
```
|
|
101
109
|
GET /fleet/v1/events?replay=50
|
|
@@ -110,11 +118,11 @@ GET /fleet/v1/events?replay=50
|
|
|
110
118
|
data: {"src":"lifecycle","ev":"wake","ts":"2026-07-05T18:00:00.123Z","personality":"boris","mode":"resume"}
|
|
111
119
|
```
|
|
112
120
|
|
|
113
|
-
`data` is the full parsed log line: `src` (`lifecycle` | `delivery` | `exits`) + `ev` + every logfmt field with quoted values decoded. `id` = epoch-ms of the event.
|
|
121
|
+
`data` is the full parsed log line: `src` (`lifecycle` | `delivery` | `exits` | `approvals` | `notices`) + `ev` + every logfmt field with quoted values decoded. `id` = epoch-ms of the event.
|
|
114
122
|
- A comment `: connected` marks the transition from replay to live; `: hb` comments every ~15 s keep the connection alive.
|
|
115
123
|
- Latency: live events surface within ~0.5 s (log-tail poll).
|
|
116
124
|
|
|
117
|
-
Current `ev` vocabulary (grows over time — see obligation 1): `wake`, `supervise` (with `action`: `reaped-idle` = parked, `reaped-ephemeral`, `dead`→fresh/resume classification, `eager-orphan-fresh`, `skipped-error`…), `stopped` / `started` (the stop/start verbs — an operator changed a peer's ✕/○ state; `action` is `stopped`/`started` for warm runtimes, `bootout`/`bootstrap` for always-on, with `reason` on failure), `delivery`, `topic-note`, `ephemeral-drain`, `attach` / `attach-end`, `hosted-deliver`, `pty-host-failed`, `supervise-error`, `composer-queue-failed-notify`, `memory-provision`, `session-exit` (from `exits.log`: `cause=child-exit` with `dead_status`/`dead_signal` when the session's process died on its own, `cause=shutdown-sigterm`/`shutdown-sigint` when the supervisor was torn down — reap/stop/new), `supervisor-uncaught` (a survivable supervisor fault left as forensics; the session keeps serving).
|
|
125
|
+
Current `ev` vocabulary (grows over time — see obligation 1): `wake`, `supervise` (with `action`: `reaped-idle` = parked, `reaped-ephemeral`, `dead`→fresh/resume classification, `eager-orphan-fresh`, `skipped-error`…), `stopped` / `started` (the stop/start verbs — an operator changed a peer's ✕/○ state; `action` is `stopped`/`started` for warm runtimes, `bootout`/`bootstrap` for always-on, with `reason` on failure), `delivery`, `topic-note`, `ephemeral-drain`, `attach` / `attach-end`, `hosted-deliver`, `pty-host-failed`, `supervise-error`, `composer-queue-failed-notify`, `memory-provision`, `session-exit` (from `exits.log`: `cause=child-exit` with `dead_status`/`dead_signal` when the session's process died on its own, `cause=shutdown-sigterm`/`shutdown-sigint` when the supervisor was torn down — reap/stop/new), `supervisor-uncaught` (a survivable supervisor fault left as forensics; the session keeps serving), `notice-raised` (docs/19 — the daemon told the OWNER something a peer could not tell him itself: a structural API error left it mute. One line per condition, not per occurrence — repeats fold into the live notice silently).
|
|
118
126
|
|
|
119
127
|
What the headline states look like on the stream: a peer **waking** is `ev=wake`; a peer **parked to sleep** is `ev=supervise action=reaped-idle`; an operator **stop/start** is `ev=stopped` / `ev=started`; a **death** is `ev=session-exit` (immediately, from the supervisor) followed by the daemon's classification on its next pass.
|
|
120
128
|
|
|
@@ -140,7 +148,7 @@ What the headline states look like on the stream: a peer **waking** is `ev=wake`
|
|
|
140
148
|
|
|
141
149
|
`from` is optional: a full `<runtime>-<personality>`, or a bare personality (its default runtime). Default: the **single natural-intelligence peer** of the registry — a GUI-originated message is the human's. With zero or several natural peers the request is refused (`400`) with an instruction to pass `from`.
|
|
142
150
|
|
|
143
|
-
Not part of the *command* API, deliberately: **attach** (a terminal handoff — a client opens the system terminal with `iapeer attach <peer>`), registry/host **admin verbs** (`create`/`remove`/`update`/`uninstall` — will be added when a client actually needs them). The **human-approval broker** now lives under `/fleet/v1/approvals*` (`GET` the queue / one item, `POST /<id>/(approve|deny)`) with the `approval-request` / `approval-resolved` SSE `ev` kinds and the `approvals` snapshot field above — the full contract is **docs/17-approval**.
|
|
151
|
+
Not part of the *command* API, deliberately: **attach** (a terminal handoff — a client opens the system terminal with `iapeer attach <peer>`), registry/host **admin verbs** (`create`/`remove`/`update`/`uninstall` — will be added when a client actually needs them). The **human-approval broker** now lives under `/fleet/v1/approvals*` (`GET` the queue / one item, `POST /<id>/(approve|deny)`) with the `approval-request` / `approval-resolved` SSE `ev` kinds and the `approvals` snapshot field above — the full contract is **docs/17-approval**. The **notice board** lives under `/fleet/v1/notices*` (`GET` the board / one item — **read-only**: a notice is one-way and has nothing to resolve) with the `notice-raised` SSE `ev` kind and the `notices` snapshot field above — the full contract is **docs/19-notices**.
|
|
144
152
|
|
|
145
153
|
Errors are always `{"error": "…"}` with an honest status: `400` malformed request, `401` bearer required, `404` unknown peer/endpoint, `405` wrong method, `502` the verb itself failed, `500` internal.
|
|
146
154
|
|
|
@@ -0,0 +1,195 @@
|
|
|
1
|
+
# 19 — Notices: telling the owner a peer went mute
|
|
2
|
+
|
|
3
|
+
A peer can fail in a way that leaves it **alive, healthy by every signal the daemon has, and
|
|
4
|
+
unable to say a word**. Nobody inside that session can report it — that is precisely what
|
|
5
|
+
broke. So the daemon reports it, on the owner's own surfaces, by the same principle an
|
|
6
|
+
approval request travels.
|
|
7
|
+
|
|
8
|
+
This note is the normative contract for the daemon side and for any face that renders
|
|
9
|
+
notices. Client obligations are in §4.
|
|
10
|
+
|
|
11
|
+
## 1. Why a notice is not an approval
|
|
12
|
+
|
|
13
|
+
The approval broker (docs/17) and the notice board look alike and are not.
|
|
14
|
+
|
|
15
|
+
| | Approval | Notice |
|
|
16
|
+
|---|---|---|
|
|
17
|
+
| Direction | request → decision → unblock | one-way |
|
|
18
|
+
| Someone waits | yes — a hook long-polls, a tool is blocked | no |
|
|
19
|
+
| A human must act | yes, or a timeout default-denies | no — it is information |
|
|
20
|
+
| Faces can write | `POST …/approve` \| `…/deny` | nothing to write; **read-only** |
|
|
21
|
+
| Lifetime | until answered or timed out | until TTL |
|
|
22
|
+
|
|
23
|
+
Reusing the broker would drag a promise/timeout/fail-safe-deny machine into a problem with
|
|
24
|
+
none of those, and would invite a face to "resolve" something that has no resolution.
|
|
25
|
+
|
|
26
|
+
What the two DO share is the **surface**: a durable log tailed into the fleet SSE stream, an
|
|
27
|
+
additive snapshot field, and a GET endpoint for the verbatim content. A face that already
|
|
28
|
+
renders approvals renders notices with the same plumbing.
|
|
29
|
+
|
|
30
|
+
## 2. The signal: `peer-mute`
|
|
31
|
+
|
|
32
|
+
The v1 notice kind. It means: *a structural API error left this peer unable to answer.*
|
|
33
|
+
|
|
34
|
+
The daemon does not infer this from silence — silence is unfalsifiable. It reads the fact
|
|
35
|
+
**structurally** out of the runtime's own session file. Neither runtime is scraped from the
|
|
36
|
+
pane: the refusal there is prose that line-wraps, while both runtimes write the fact as data.
|
|
37
|
+
|
|
38
|
+
### claude — the transcript
|
|
39
|
+
|
|
40
|
+
`~/.claude/projects/<slug>/<sessionId>.jsonl`, on the line the runtime wrote when it refused:
|
|
41
|
+
|
|
42
|
+
```json
|
|
43
|
+
{ "isApiErrorMessage": true, "error": "rate_limit", "apiErrorStatus": 429,
|
|
44
|
+
"message": { "model": "<synthetic>", "content": [{ "type": "text",
|
|
45
|
+
"text": "You've reached your Fable 5 limit. Run /usage-credits to continue or switch models with /model." }] } }
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The detector keys on **`isApiErrorMessage: true`** — the CLASS — and reports `error`
|
|
49
|
+
verbatim as `errorType`. `rate_limit` is one value; `overloaded`, an expired auth and any
|
|
50
|
+
future member arrive on the same field and take the same path. `error` is a locale-stable
|
|
51
|
+
enum, so no prose is matched.
|
|
52
|
+
|
|
53
|
+
### codex — the rollout
|
|
54
|
+
|
|
55
|
+
`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`, in an `event_msg` / `token_count` payload:
|
|
56
|
+
|
|
57
|
+
```json
|
|
58
|
+
{ "rate_limits": { "limit_id": "codex", "plan_type": "plus",
|
|
59
|
+
"primary": { "used_percent": 1.0, "window_minutes": 300, "resets_at": 1783114795 },
|
|
60
|
+
"secondary": { "used_percent": 17.0, "window_minutes": 10080, "resets_at": 1783412915 },
|
|
61
|
+
"rate_limit_reached_type": null } }
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
`rate_limit_reached_type` is `null` while healthy and non-null once a wall is hit. The
|
|
65
|
+
detector keys on **non-null**, never on the variant strings (`rate_limit_reached`,
|
|
66
|
+
`workspace_owner_usage_limit_reached`, `workspace_member_credits_depleted`, … — read out of
|
|
67
|
+
the 0.144.1 binary), so a variant we have never seen still detects and reports itself.
|
|
68
|
+
|
|
69
|
+
## 3. What the owner is told — and what is deliberately withheld
|
|
70
|
+
|
|
71
|
+
A notice states who, which runtime, which error type, which model, and when the wall lifts.
|
|
72
|
+
Two of those are **absent by design rather than guessed**:
|
|
73
|
+
|
|
74
|
+
- **`resetsAtMs` on claude.** Claude does not say when a per-model bucket lifts. Its
|
|
75
|
+
`errorDetails` is a raw 429 reading *"Please try again later"* — no stamp. The 5h/7d
|
|
76
|
+
`resets_at` in the statusline blob belongs to a **different limit** and must never be
|
|
77
|
+
substituted: measured live on 15.07.2026, the 5h bucket sat at **11 %** and the 7d at
|
|
78
|
+
**66 %** while fable was fully exhausted. Codex does state its reset, so its notices carry
|
|
79
|
+
one. The asymmetry is real and stays visible.
|
|
80
|
+
- **`model` on codex.** The rollout snapshot names no model. Claude names it only in prose,
|
|
81
|
+
so it is lifted from the runtime's own sentence when no real model reply precedes the
|
|
82
|
+
error; a miss omits the field.
|
|
83
|
+
|
|
84
|
+
Absent field ⇒ *the runtime did not say*. Never ⇒ *there is no limit*.
|
|
85
|
+
|
|
86
|
+
`content` is the runtime's **verbatim** refusal for claude. For codex the rollout carries no
|
|
87
|
+
prose, so `content` is rendered from its typed fields (both windows, their percentages and
|
|
88
|
+
resets, the plan) — data, never our interpretation of the cause.
|
|
89
|
+
|
|
90
|
+
## 4. Fleet API
|
|
91
|
+
|
|
92
|
+
### `GET /fleet/v1/notices`
|
|
93
|
+
|
|
94
|
+
```json
|
|
95
|
+
{ "api": 1, "version": "0.4.93", "ts": "2026-07-15T21:10:00.000Z", "notices": [ { … } ] }
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### `GET /fleet/v1/notices/<id>`
|
|
99
|
+
|
|
100
|
+
```json
|
|
101
|
+
{ "api": 1, "version": "0.4.93", "ts": "…", "notice": { … } }
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
404 when the id is unknown or its TTL has passed. There is **no** POST on either route — see §1.
|
|
105
|
+
|
|
106
|
+
### The notice object
|
|
107
|
+
|
|
108
|
+
| field | type | notes |
|
|
109
|
+
|---|---|---|
|
|
110
|
+
| `id` | string | `n1`, `n2`, … Unique while live; **not** stable across a daemon restart. |
|
|
111
|
+
| `personality` | string | the mute peer |
|
|
112
|
+
| `runtime` | string | `claude` \| `codex` |
|
|
113
|
+
| `kind` | string | `peer-mute` in v1. **Treat unknown kinds as renderable** — new kinds ride this surface. |
|
|
114
|
+
| `errorType` | string | the runtime's OWN value (`rate_limit`, `overloaded`, `rate_limit_reached`, …). Free-form: do not switch exhaustively. |
|
|
115
|
+
| `model` | string? | **absent** when the runtime did not name it |
|
|
116
|
+
| `resetsAtMs` | number? | epoch-ms; **absent** when the runtime did not state it → render "unknown", never substitute |
|
|
117
|
+
| `summary` | string | one line, e.g. `boris · claude — rate_limit (Fable 5)` |
|
|
118
|
+
| `content` | string | verbatim (claude) / rendered from typed fields (codex) |
|
|
119
|
+
| `sessionId` | string? | correlates with the on-disk transcript |
|
|
120
|
+
| `createdMs` | number | first detection |
|
|
121
|
+
| `lastMs` | number | most recent detection folded in |
|
|
122
|
+
| `expiresMs` | number | TTL boundary |
|
|
123
|
+
| `count` | number | detections folded into this notice (≥1) — render as `×N` |
|
|
124
|
+
|
|
125
|
+
### Snapshot
|
|
126
|
+
|
|
127
|
+
`GET /fleet/v1/snapshot` carries `notices?: FleetNotice[]` — the same items, so a client that
|
|
128
|
+
already fetches the snapshot on every event needs no extra call. **ADDITIVE and omitted when
|
|
129
|
+
the board is empty**: absence ⇒ empty board (the same rule as `approvals`).
|
|
130
|
+
|
|
131
|
+
### Events (SSE)
|
|
132
|
+
|
|
133
|
+
`notices.log` is one of the tailed durable logs, so `GET /fleet/v1/events` pushes:
|
|
134
|
+
|
|
135
|
+
```
|
|
136
|
+
ev=notice-raised id=n1 personality=boris runtime=claude kind=peer-mute error_type=rate_limit
|
|
137
|
+
model="Fable 5" session=… summary="…" created=… expires=…
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
`resets_at` + `resets_at_iso` appear **only** when the runtime stated a reset. At-least-once;
|
|
141
|
+
ignore unknown `ev` kinds as always.
|
|
142
|
+
|
|
143
|
+
### Client obligations
|
|
144
|
+
|
|
145
|
+
1. **Render, never resolve.** No approve/deny.
|
|
146
|
+
2. **Absent `resetsAtMs` ⇒ unknown.** Never substitute the 5h/7d reset — different limit.
|
|
147
|
+
3. **Absent `notices` ⇒ empty board.**
|
|
148
|
+
4. **Unknown `kind` / `errorType` ⇒ still render.** Both are growth seams.
|
|
149
|
+
5. **Do not key on `id` across daemon restarts** — the board is in-memory; a restart
|
|
150
|
+
re-detects from the on-disk evidence and issues new ids.
|
|
151
|
+
|
|
152
|
+
## 5. Dedup and TTL — one mechanism
|
|
153
|
+
|
|
154
|
+
A mute peer re-emits its error on **every** attempted turn. So a notice carries a dedup key —
|
|
155
|
+
`personality | runtime | kind | errorType | model` — and while a notice with that key is live,
|
|
156
|
+
a repeat detection only bumps `count` and `lastMs`: **no new id, no new log line, no new SSE
|
|
157
|
+
event**. The owner sees one card saying `×7`, not seven cards.
|
|
158
|
+
|
|
159
|
+
When the TTL passes (`IAPEER_NOTICE_TTL_MS`, default 1 h) the notice expires silently. If the
|
|
160
|
+
peer is still broken, the next detection raises a **fresh** notice — a deliberate periodic
|
|
161
|
+
reminder rather than one card that scrolls away forever. Expiry is not an event: either the
|
|
162
|
+
peer recovered (nothing to say) or it is still broken and will re-raise.
|
|
163
|
+
|
|
164
|
+
The board is in-memory; `notices.log` is the audit trace, not a recovery store. A restart
|
|
165
|
+
re-detects from the transcript evidence still on disk.
|
|
166
|
+
|
|
167
|
+
## 6. Detection, and why paths are not derived from `cwd`
|
|
168
|
+
|
|
169
|
+
The daemon sweeps on its own timer (`DEFAULT_MUTEWATCH_INTERVAL_MS`, 20 s — detection latency
|
|
170
|
+
is the product here, and the 60 s supervise tick would put the worst case at the budget with
|
|
171
|
+
nothing to spare). A sweep only opens files whose mtime moved since the last pass.
|
|
172
|
+
|
|
173
|
+
The obvious route — slugify the peer's registry `cwd` into `~/.claude/projects/<slug>` — is a
|
|
174
|
+
**trap**. The slug mirrors the cwd *string as the process was launched*, not the real path.
|
|
175
|
+
This host carries BOTH `-Users-macmini-Projects-IAPeer` and `-Users-macmini-Projects-iapeer`
|
|
176
|
+
for one case-insensitive directory. So attribution runs the other way: read the cwd each file
|
|
177
|
+
states **about itself** (claude: the `cwd` field on every line; codex: `session_meta.payload.cwd`)
|
|
178
|
+
and match it against the registry, case-insensitively. A file whose cwd belongs to no peer is
|
|
179
|
+
ignored — a human's own session never notifies.
|
|
180
|
+
|
|
181
|
+
## 7. What is proven, and what is not
|
|
182
|
+
|
|
183
|
+
- **claude — proven live.** The transcript signal, the missing reset, the whole path
|
|
184
|
+
(detector → board → fleet surface) were exercised against a genuinely exhausted fable
|
|
185
|
+
bucket on 15.07.2026. Tests replay the captured line byte-for-byte.
|
|
186
|
+
- **codex — proven by replay, not live.** The rollout structure, windows, resets and the
|
|
187
|
+
healthy (`null`) path are real bytes from this host. Codex's quota never ran out here, so
|
|
188
|
+
**no real reached-bytes exist**: the tests flip that one field to a variant read out of the
|
|
189
|
+
binary. The detector keys on non-null precisely so this synthesis cannot flatter it. A real
|
|
190
|
+
codex reach should be captured opportunistically and folded in.
|
|
191
|
+
- **The blocking-window heuristic is unverified.** Codex does not say WHICH window blocked;
|
|
192
|
+
the notice reports the **fullest** one. Both windows ride in `content`, so a human sees the
|
|
193
|
+
raw fact even if the heuristic picks wrong.
|
|
194
|
+
|
|
195
|
+
Forensic evidence for all of the above: `docs/internals/forensics/model-limit-2026-07-15/`.
|
package/docs/README.md
CHANGED
|
@@ -80,6 +80,7 @@ The sections are layered: from "install and use" to "understand the architecture
|
|
|
80
80
|
- [16 — Tray](16-tray.md) — the macOS menu-bar fleet dashboard (SwiftBar plugin), the first external Fleet API client
|
|
81
81
|
- [17 — Human approval](17-approval.md) — per-peer gated/yolo mode: routing a runtime's blocking approvals to a human through the daemon broker (hook interception, the broker, Fleet API + CLI)
|
|
82
82
|
- [18 — Origin guard](18-origin-guard.md) — an agent's reply to a human must target the channel the human wrote from: the armed-pair model, the hold, `confirm-send`
|
|
83
|
+
- [19 — Notices](19-notices.md) — telling the owner a peer went mute: detecting the structural API error that leaves a session alive, green and speechless, and routing it to the owner's surfaces one-way (not an approval)
|
|
83
84
|
|
|
84
85
|
## Status
|
|
85
86
|
|
package/package.json
CHANGED
|
@@ -57,6 +57,10 @@ export interface RequestApprovalDeps {
|
|
|
57
57
|
fetch?: typeof fetch
|
|
58
58
|
/** Injectable client timeout (tests). */
|
|
59
59
|
timeoutMs?: number
|
|
60
|
+
/** External cancel: aborting this signal aborts the long-poll fetch (the daemon sees the requester
|
|
61
|
+
* transport close and settles the pending request cancel/default-deny, clearing it from every
|
|
62
|
+
* face). The supervisor uses it to close a request whose modal disappeared before a decision. */
|
|
63
|
+
signal?: AbortSignal
|
|
60
64
|
}
|
|
61
65
|
|
|
62
66
|
/**
|
|
@@ -73,6 +77,11 @@ export async function requestApproval(base: string, body: ApprovalRequestBody, d
|
|
|
73
77
|
const controller = new AbortController()
|
|
74
78
|
const timer = setTimeout(() => controller.abort(), timeoutMs)
|
|
75
79
|
;(timer as { unref?: () => void }).unref?.()
|
|
80
|
+
const onExternalAbort = (): void => controller.abort()
|
|
81
|
+
if (deps.signal) {
|
|
82
|
+
if (deps.signal.aborted) controller.abort()
|
|
83
|
+
else deps.signal.addEventListener('abort', onExternalAbort, { once: true })
|
|
84
|
+
}
|
|
76
85
|
try {
|
|
77
86
|
const headers: Record<string, string> = { 'content-type': 'application/json' }
|
|
78
87
|
const bearer = env.IAPEER_BEARER_TOKEN?.trim()
|
|
@@ -88,5 +97,6 @@ export async function requestApproval(base: string, body: ApprovalRequestBody, d
|
|
|
88
97
|
return data.decision === 'allow' ? { decision: 'allow' } : { decision: 'deny', reason: data.reason || 'denied' }
|
|
89
98
|
} finally {
|
|
90
99
|
clearTimeout(timer)
|
|
100
|
+
deps.signal?.removeEventListener('abort', onExternalAbort)
|
|
91
101
|
}
|
|
92
102
|
}
|
package/src/daemon/fleet.ts
CHANGED
|
@@ -62,6 +62,8 @@ import { heartbeatAgeSecs, probeFullDiskAccess, readMemoryProvider, readVoicePro
|
|
|
62
62
|
import { deliveryLogPath } from './deliverylog.ts'
|
|
63
63
|
import { approvalsLogPath } from './approvalslog.ts'
|
|
64
64
|
import { ApprovalBroker } from './approvals.ts'
|
|
65
|
+
import { noticesLogPath } from './noticeslog.ts'
|
|
66
|
+
import { NoticeBoard } from './notices.ts'
|
|
65
67
|
import { callTool, type CallToolDeps } from './index.ts'
|
|
66
68
|
|
|
67
69
|
/** The API generation baked into every snapshot (additive evolution within v1 —
|
|
@@ -128,6 +130,35 @@ export interface FleetApproval {
|
|
|
128
130
|
expiresMs: number
|
|
129
131
|
}
|
|
130
132
|
|
|
133
|
+
/** A live owner-facing notice as the snapshot exposes it (docs/19). The SAME items
|
|
134
|
+
* `GET /fleet/v1/notices` returns (board.list()) — carried in the snapshot so a client that
|
|
135
|
+
* already fetches `/snapshot` on every event renders the board with no extra call.
|
|
136
|
+
*
|
|
137
|
+
* ONE-WAY: there is nothing to answer. A face RENDERS a notice; it never resolves one (no
|
|
138
|
+
* approve/deny sibling exists on purpose — see docs/19 §1).
|
|
139
|
+
*
|
|
140
|
+
* `resetsAtMs` is present ONLY when the runtime itself stated when the wall lifts (codex
|
|
141
|
+
* does; claude does not for a per-model bucket). Its ABSENCE means "the runtime did not
|
|
142
|
+
* say" — a client MUST render that as unknown and MUST NOT substitute the 5h/7d reset,
|
|
143
|
+
* which belongs to a different limit. */
|
|
144
|
+
export interface FleetNotice {
|
|
145
|
+
id: string
|
|
146
|
+
personality: string
|
|
147
|
+
runtime: string
|
|
148
|
+
kind: string
|
|
149
|
+
errorType: string
|
|
150
|
+
model?: string
|
|
151
|
+
resetsAtMs?: number
|
|
152
|
+
summary: string
|
|
153
|
+
content: string
|
|
154
|
+
sessionId?: string
|
|
155
|
+
createdMs: number
|
|
156
|
+
lastMs: number
|
|
157
|
+
expiresMs: number
|
|
158
|
+
/** Detections folded into this notice while it stayed live (≥1) — render as "×N". */
|
|
159
|
+
count: number
|
|
160
|
+
}
|
|
161
|
+
|
|
131
162
|
export interface FleetSnapshot {
|
|
132
163
|
api: typeof FLEET_API_VERSION
|
|
133
164
|
version: string
|
|
@@ -137,6 +168,9 @@ export interface FleetSnapshot {
|
|
|
137
168
|
/** Pending human-approval requests (docs/17). ADDITIVE + omitted when the queue is empty —
|
|
138
169
|
* a client MUST treat its absence as an empty queue (same rule as `approval_mode`). */
|
|
139
170
|
approvals?: FleetApproval[]
|
|
171
|
+
/** Live owner-facing notices (docs/19). ADDITIVE + omitted when the board is empty — a
|
|
172
|
+
* client MUST treat its absence as an empty board (same rule as `approvals`). */
|
|
173
|
+
notices?: FleetNotice[]
|
|
140
174
|
}
|
|
141
175
|
|
|
142
176
|
// The CLI op functions, injectable for tests. Defaults LAZY-load cli/index.ts at
|
|
@@ -173,6 +207,7 @@ export function buildFleetSnapshot(
|
|
|
173
207
|
startedAtMs: number,
|
|
174
208
|
nowMs: number = Date.now(),
|
|
175
209
|
approvals: FleetApproval[] = [],
|
|
210
|
+
notices: FleetNotice[] = [],
|
|
176
211
|
): FleetSnapshot {
|
|
177
212
|
const peers: FleetPeer[] = ops.listPeers({ env }).map(r => {
|
|
178
213
|
const runtimes: FleetRuntimeStatus[] = r.runtimes.map(s => {
|
|
@@ -233,6 +268,8 @@ export function buildFleetSnapshot(
|
|
|
233
268
|
// superset of FleetApproval — the named fields are the contract; extras ride along and
|
|
234
269
|
// clients ignore them (obligation 2).
|
|
235
270
|
...(approvals.length ? { approvals } : {}),
|
|
271
|
+
// Same rule for the notice board (docs/19).
|
|
272
|
+
...(notices.length ? { notices } : {}),
|
|
236
273
|
}
|
|
237
274
|
}
|
|
238
275
|
|
|
@@ -255,6 +292,10 @@ export function fleetEventFiles(cfg: LifecycleConfig): Array<{ path: string; src
|
|
|
255
292
|
// Human-approval broker events (docs/17): approval-request / approval-resolved.
|
|
256
293
|
// A pre-approval daemon simply never writes this file (readLogTail tolerates absence).
|
|
257
294
|
{ path: approvalsLogPath(cfg.eventLogDir), src: 'approvals' },
|
|
295
|
+
// Notice-board events (docs/19): notice-raised. Tailing it here is the WHOLE push
|
|
296
|
+
// path — a face subscribed to /fleet/v1/events receives owner notices with no
|
|
297
|
+
// further wiring, exactly as it already receives approvals.
|
|
298
|
+
{ path: noticesLogPath(cfg.eventLogDir), src: 'notices' },
|
|
258
299
|
]
|
|
259
300
|
}
|
|
260
301
|
|
|
@@ -472,6 +513,10 @@ export interface FleetHandlerOptions {
|
|
|
472
513
|
pollMs?: number
|
|
473
514
|
/** Injectable approval broker (tests — e.g. a short timeout). Default: a fresh one. */
|
|
474
515
|
broker?: ApprovalBroker
|
|
516
|
+
/** The notice board (docs/19). The composition point passes the SAME instance the
|
|
517
|
+
* mute-watch raises into — that shared instance IS the wiring: the watch writes, this
|
|
518
|
+
* handler reads. Default: a fresh one (a handler with no watch simply stays empty). */
|
|
519
|
+
board?: NoticeBoard
|
|
475
520
|
}
|
|
476
521
|
|
|
477
522
|
export type FleetHandler = (
|
|
@@ -494,6 +539,9 @@ export function buildFleetHandler(opts: FleetHandlerOptions = {}): FleetHandler
|
|
|
494
539
|
// The approval broker — one per daemon; the ASK side (POST /approvals long-poll) and
|
|
495
540
|
// the ANSWER side (GET/POST /approvals/*) are all interfaces to this one queue.
|
|
496
541
|
const broker = opts.broker ?? new ApprovalBroker({ logDir: cfg.eventLogDir, env })
|
|
542
|
+
// The notice board — one per daemon; the mute-watch RAISES into it and the /notices reads
|
|
543
|
+
// below serve it. Same shape as the broker above: one in-memory store, many interfaces.
|
|
544
|
+
const board = opts.board ?? new NoticeBoard({ logDir: cfg.eventLogDir, env })
|
|
497
545
|
let opsPromise: Promise<FleetOps> | undefined
|
|
498
546
|
const getOps = (): Promise<FleetOps> => {
|
|
499
547
|
if (opts.ops) return Promise.resolve(opts.ops)
|
|
@@ -511,7 +559,13 @@ export function buildFleetHandler(opts: FleetHandlerOptions = {}): FleetHandler
|
|
|
511
559
|
if (method !== 'GET') return sendJson(res, 405, { error: 'method not allowed — GET /fleet/v1/snapshot' })
|
|
512
560
|
// broker.list() is PendingApproval[] — structurally a superset of FleetApproval, so it
|
|
513
561
|
// satisfies the snapshot contract (extra fields serialize + are ignored by clients).
|
|
514
|
-
|
|
562
|
+
// broker.list()/board.list() are supersets of FleetApproval/FleetNotice — the named
|
|
563
|
+
// fields are the contract; extras serialize and clients ignore them (obligation 2).
|
|
564
|
+
return sendJson(
|
|
565
|
+
res,
|
|
566
|
+
200,
|
|
567
|
+
buildFleetSnapshot(env, cfg, await getOps(), startedAtMs, undefined, broker.list(), board.list()),
|
|
568
|
+
)
|
|
515
569
|
}
|
|
516
570
|
if (path === '/fleet/v1/events') {
|
|
517
571
|
if (method !== 'GET') return sendJson(res, 405, { error: 'method not allowed — GET /fleet/v1/events' })
|
|
@@ -647,6 +701,26 @@ export function buildFleetHandler(opts: FleetHandlerOptions = {}): FleetHandler
|
|
|
647
701
|
if (!ok) return sendJson(res, 404, { error: `no pending approval "${id}" (already resolved, expired, or unknown)` })
|
|
648
702
|
return sendJson(res, 200, { id, action, ok: true })
|
|
649
703
|
}
|
|
704
|
+
|
|
705
|
+
// ── notices (docs/19) — READ-ONLY BY DESIGN ───────────────────────────
|
|
706
|
+
// No approve/deny sibling: a notice is one-way. Nothing is blocked on it and there
|
|
707
|
+
// is no decision to make, so there is nothing for a face to POST back.
|
|
708
|
+
if (path === '/fleet/v1/notices') {
|
|
709
|
+
if (method !== 'GET') return sendJson(res, 405, { error: 'method not allowed — GET /fleet/v1/notices' })
|
|
710
|
+
return sendJson(res, 200, {
|
|
711
|
+
api: FLEET_API_VERSION,
|
|
712
|
+
version: IAPEER_VERSION,
|
|
713
|
+
ts: new Date().toISOString(),
|
|
714
|
+
notices: board.list(),
|
|
715
|
+
})
|
|
716
|
+
}
|
|
717
|
+
const noticeMatch = /^\/fleet\/v1\/notices\/([^/]+)$/.exec(path)
|
|
718
|
+
if (noticeMatch) {
|
|
719
|
+
if (method !== 'GET') return sendJson(res, 405, { error: 'method not allowed — GET /fleet/v1/notices/<id>' })
|
|
720
|
+
const item = board.get(noticeMatch[1]!)
|
|
721
|
+
if (!item) return sendJson(res, 404, { error: `no live notice "${noticeMatch[1]}" (expired or unknown)` })
|
|
722
|
+
return sendJson(res, 200, { api: FLEET_API_VERSION, version: IAPEER_VERSION, ts: new Date().toISOString(), notice: item })
|
|
723
|
+
}
|
|
650
724
|
return sendJson(res, 404, {
|
|
651
725
|
error: `unknown fleet endpoint ${method} ${path}`,
|
|
652
726
|
endpoints: [
|
|
@@ -659,6 +733,8 @@ export function buildFleetHandler(opts: FleetHandlerOptions = {}): FleetHandler
|
|
|
659
733
|
'POST /fleet/v1/approvals (hook long-poll)',
|
|
660
734
|
'GET /fleet/v1/approvals/<id>',
|
|
661
735
|
'POST /fleet/v1/approvals/<id>/(approve|deny)',
|
|
736
|
+
'GET /fleet/v1/notices',
|
|
737
|
+
'GET /fleet/v1/notices/<id>',
|
|
662
738
|
],
|
|
663
739
|
})
|
|
664
740
|
} catch (e) {
|
package/src/daemon/main.ts
CHANGED
|
@@ -67,6 +67,8 @@ import {
|
|
|
67
67
|
import { iapeerBinPath } from '../install/index.ts'
|
|
68
68
|
import { appendDeliveryEvent } from './deliverylog.ts'
|
|
69
69
|
import { buildFleetHandler } from './fleet.ts'
|
|
70
|
+
import { NoticeBoard } from './notices.ts'
|
|
71
|
+
import { startMuteWatch } from './mutewatch.ts'
|
|
70
72
|
import { defaultDaemonSocketPath, startDaemon, type DaemonHandle } from './index.ts'
|
|
71
73
|
|
|
72
74
|
/** Default TCP loopback port for the always-on router. Real http MCP clients
|
|
@@ -91,6 +93,13 @@ export function parseDaemonPort(raw: string | undefined, dflt: number = DEFAULT_
|
|
|
91
93
|
* is the reap threshold; this is just how often the timer checks. */
|
|
92
94
|
export const DEFAULT_SUPERVISE_INTERVAL_MS = 60_000
|
|
93
95
|
|
|
96
|
+
/** Mute-watch cadence (docs/19) — how often the daemon sweeps the runtimes' session files
|
|
97
|
+
* for a structural API error that left a peer unable to answer. Its OWN timer rather than a
|
|
98
|
+
* ride on the 60 s supervise tick: detection latency IS the product here (the owner must
|
|
99
|
+
* learn within a minute), and a 60 s sweep puts the worst case AT the budget with nothing
|
|
100
|
+
* to spare. The sweep is cheap — it stats only files that CHANGED since the last pass. */
|
|
101
|
+
export const DEFAULT_MUTEWATCH_INTERVAL_MS = 20_000
|
|
102
|
+
|
|
94
103
|
// This module's own path — the launchd plist runs `bun <this>` as the daemon.
|
|
95
104
|
const DAEMON_MAIN_PATH = fileURLToPath(import.meta.url)
|
|
96
105
|
|
|
@@ -319,6 +328,8 @@ export interface ConfiguredDaemonOptions {
|
|
|
319
328
|
/** H8 bearer token; falls back to env.IAPEER_BEARER_TOKEN. Off when neither set. */
|
|
320
329
|
bearerToken?: string
|
|
321
330
|
superviseIntervalMs?: number
|
|
331
|
+
/** Mute-watch sweep cadence (default DEFAULT_MUTEWATCH_INTERVAL_MS). */
|
|
332
|
+
muteWatchIntervalMs?: number
|
|
322
333
|
/** Write the router.json discovery file (default true for production). */
|
|
323
334
|
discovery?: boolean
|
|
324
335
|
rootDir?: string
|
|
@@ -341,7 +352,24 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
341
352
|
const armEphemeralOnDelivered = makeArmEphemeralOnDelivered(cfg)
|
|
342
353
|
const ephemeral = makeEphemeralRouteDeps(cfg, env)
|
|
343
354
|
const composerQueue = makeComposerQueueRouteDeps(cfg, env)
|
|
344
|
-
|
|
355
|
+
// Notice board (docs/19) — ONE instance shared by the mute-watch (raises) and the
|
|
356
|
+
// fleet handler (serves /notices + the snapshot field). Constructed HERE, at the
|
|
357
|
+
// composition point, because that sharing is the whole wiring.
|
|
358
|
+
const board = new NoticeBoard({ logDir: cfg.eventLogDir, env })
|
|
359
|
+
const stopMuteWatch = startMuteWatch(cfg, board, {
|
|
360
|
+
env,
|
|
361
|
+
intervalMs: opts.muteWatchIntervalMs ?? DEFAULT_MUTEWATCH_INTERVAL_MS,
|
|
362
|
+
// Same stance as the supervise tick's onError: a detector throw must be VISIBLE, not
|
|
363
|
+
// swallowed into a silently-dead watch — the exact failure mode this feature exists to
|
|
364
|
+
// end. Best-effort; a reporter must never fail the daemon.
|
|
365
|
+
onError: (err: unknown) => {
|
|
366
|
+
try {
|
|
367
|
+
const detail = err instanceof Error ? (err.stack ?? err.message) : String(err)
|
|
368
|
+
appendLifecycleEvent(cfg.eventLogDir, { ev: 'mutewatch-error', error: detail.replace(/\s+/g, ' ').slice(0, 600) }, { env })
|
|
369
|
+
} catch { /* best-effort */ }
|
|
370
|
+
},
|
|
371
|
+
})
|
|
372
|
+
const handle = await startDaemon({
|
|
345
373
|
wake: makeWakeFn(cfg, env),
|
|
346
374
|
// M2 arm-on-outbound (see makeArmEphemeralOnDelivered): ephemeral caller's ok
|
|
347
375
|
// send ⇒ armed for the supervise quiet-reap. No-op for every other caller.
|
|
@@ -367,7 +395,7 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
367
395
|
// (the same listPeers truth as `iapeer list`), SSE events (tail of the durable
|
|
368
396
|
// logs), commands over the existing verb functions. Same listeners, same bearer
|
|
369
397
|
// gate; advertised in router.json as fleet:1. Contract: docs/15-fleet-api.md.
|
|
370
|
-
fleet: buildFleetHandler({ env }),
|
|
398
|
+
fleet: buildFleetHandler({ env, board }),
|
|
371
399
|
supervise: {
|
|
372
400
|
intervalMs: opts.superviseIntervalMs ?? DEFAULT_SUPERVISE_INTERVAL_MS,
|
|
373
401
|
// idle-reap / zombie-sweep, THEN the eager fresh re-launch for any peer whose
|
|
@@ -412,6 +440,10 @@ export async function startConfiguredDaemon(opts: ConfiguredDaemonOptions = {}):
|
|
|
412
440
|
env,
|
|
413
441
|
rootDir: opts.rootDir,
|
|
414
442
|
})
|
|
443
|
+
// The mute-watch timer is OURS, not startDaemon's — so its teardown must hang off the
|
|
444
|
+
// handle we return, or a closed daemon would leave a live timer sweeping the disk (and
|
|
445
|
+
// every test that starts a daemon would hang on exit).
|
|
446
|
+
return { ...handle, close: async () => { stopMuteWatch(); await handle.close() } }
|
|
415
447
|
}
|
|
416
448
|
|
|
417
449
|
// ─────────────────────────────────────────────────────────────────────────────
|