@cohortapp/agent-sdk 2.18.36 → 2.18.37
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/runbooks/cadence-requeue-storm-recovery.md +186 -0
- package/lib/assurance/first-reply.mjs +16 -0
- package/lib/assurance/notice-plane.mjs +218 -0
- package/lib/assurance/notice-voice.mjs +35 -2
- package/lib/budget-escalate.mjs +42 -4
- package/lib/cadence-bus.mjs +359 -46
- package/lib/comms/machine-status.mjs +377 -0
- package/package.json +1 -1
- package/scripts/ci/check-durable-write-seam.mjs +3 -3
- package/scripts/daemon/assurance.mjs +391 -15
- package/scripts/daemon/cadence-consumer.mjs +165 -9
- package/scripts/daemon/deliver.mjs +104 -0
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# Recovery: the cadence requeue storm (W19)
|
|
2
|
+
|
|
3
|
+
What this runs after: the fix on `cohort/w19-cadence-hotloop` is **merged and the
|
|
4
|
+
daemon restarted**. The queue a storming seat leaves behind does not drain
|
|
5
|
+
itself, because every pending copy of a periodic cadence is a legitimate tick
|
|
6
|
+
that the new hold slows down rather than removes.
|
|
7
|
+
|
|
8
|
+
Do these in order. Steps 1–3 are read-only.
|
|
9
|
+
|
|
10
|
+
## 0. Is this seat affected?
|
|
11
|
+
|
|
12
|
+
```
|
|
13
|
+
cat ~/<agent>/state/cadence-bus/health.json | python3 -m json.tool
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Affected if `stats.deferred` is in the hundreds of thousands, or `stats.retries`
|
|
17
|
+
is within a rounding error of `stats.deferred` (that equality is the old
|
|
18
|
+
conflation — see below). On this seat on 2026-10-01 it read `deferred: 1186383`,
|
|
19
|
+
`retries: 1186419`, `depth.inbox: 93`.
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
ls ~/<agent>/state/cadence-bus/inbox | wc -l
|
|
23
|
+
for f in ~/<agent>/state/cadence-bus/inbox/*.json; do
|
|
24
|
+
python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["cadence"])' "$f"
|
|
25
|
+
done | sort | uniq -c | sort -rn
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
One cadence holding nearly every pending tick is the signature. Here it was 87
|
|
29
|
+
of 88 `meeting-action-capture`. Split them by `source` too — it tells you which
|
|
30
|
+
lane is producing them, and both lanes are deduped now:
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
for f in ~/<agent>/state/cadence-bus/inbox/*.json; do
|
|
34
|
+
python3 -c 'import json,sys;d=json.load(open(sys.argv[1]));print(d["cadence"],d["source"])' "$f"
|
|
35
|
+
done | sort | uniq -c | sort -rn
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
On 2026-10-01 that read 47 `handoff-timeout` and 40 `launchd` for the one
|
|
39
|
+
cadence. Post-fix each lane holds at a single pending tick, so a steady state of
|
|
40
|
+
one or two per cadence is correct, not a leak.
|
|
41
|
+
|
|
42
|
+
## 1. Confirm the fix is actually running
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
grep -c '"stage":"requeued_deferred"' ~/<agent>/logs/cadence-bus/$(date -u +%F).jsonl
|
|
46
|
+
grep -m1 '"stage":"requeued_deferred"' ~/<agent>/logs/cadence-bus/$(date -u +%F).jsonl
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Post-fix rows carry `not_before`. If they do not, the running daemon predates
|
|
50
|
+
the fix — stop here and finish the restart. `stats.held` present in
|
|
51
|
+
`health.json` is the other tell: the field did not exist before this change.
|
|
52
|
+
|
|
53
|
+
## 2. Do NOT hand-delete the queue
|
|
54
|
+
|
|
55
|
+
The consumer is the single writer to `claimed/` and renames out of `inbox/`.
|
|
56
|
+
Deleting files underneath it races the claim and loses ticks silently. There is
|
|
57
|
+
nothing to delete anyway: see step 3.
|
|
58
|
+
|
|
59
|
+
## 3. The inbox will NOT drain, and that is the expected observation
|
|
60
|
+
|
|
61
|
+
Read this before you watch anything, because the obvious success criterion is
|
|
62
|
+
the wrong one and will make a working merge look like a failed one.
|
|
63
|
+
|
|
64
|
+
`enqueueTick`'s pending-dedupe stops NEW copies being created; it does not
|
|
65
|
+
remove the ones already queued. Nor will those drain on their own: only one tick
|
|
66
|
+
of a cadence can run at a time (`cadence-in-flight`), and for
|
|
67
|
+
`meeting-action-capture` a sub-session or an unacked handoff has been open
|
|
68
|
+
essentially permanently — that is the separate open item, and it is the reason
|
|
69
|
+
the copies accumulated. Until it is resolved, those ticks are deferred, held,
|
|
70
|
+
and deferred again. **Nothing is lost** (a deferral never touches `attempts` and
|
|
71
|
+
cannot reach `dlq/`), and the cost of holding them is now one requeue per 30 s
|
|
72
|
+
per held cadence instead of eight per second.
|
|
73
|
+
|
|
74
|
+
So: **a flat inbox count after the restart is success, not failure.** What this
|
|
75
|
+
change fixes is the COST of the backlog, not the backlog.
|
|
76
|
+
|
|
77
|
+
### The metric that does move
|
|
78
|
+
|
|
79
|
+
`deferred` in `state/cadence-bus/health.json`, per second of uptime:
|
|
80
|
+
|
|
81
|
+
```
|
|
82
|
+
python3 - <<'EOF'
|
|
83
|
+
import json,datetime
|
|
84
|
+
h=json.load(open('/Users/<you>/<agent>/state/cadence-bus/health.json'))
|
|
85
|
+
s=h['stats']; up=(datetime.datetime.fromisoformat(h['ts'].replace('Z','+00:00'))
|
|
86
|
+
- datetime.datetime.fromisoformat(s['started_at'].replace('Z','+00:00'))).total_seconds()
|
|
87
|
+
print('deferred/s', round(s['deferred']/up,3), '| held', s.get('held'), '| retries', s['retries'])
|
|
88
|
+
EOF
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
| | before | after |
|
|
92
|
+
|---|---|---|
|
|
93
|
+
| `deferred/s` | 4.8 measured on this seat (8.0 inside a drain) | **≤ 0.05** |
|
|
94
|
+
| `retries` | tracked `deferred` exactly (1,186,419 against 36 real) | the real retry count, tens |
|
|
95
|
+
| `held` | absent | the number of cadences currently waiting on a gate |
|
|
96
|
+
|
|
97
|
+
Give it 10 minutes of uptime before reading, so the denominator is meaningful.
|
|
98
|
+
If `deferred/s` is still above ~0.1, the fix is not doing its job and that is
|
|
99
|
+
worth escalating. If it is at or below 0.05 while `inbox` stays flat, the merge
|
|
100
|
+
worked and the remaining backlog is the in-flight-handoff item.
|
|
101
|
+
|
|
102
|
+
### Why the per-cadence hold is what makes this true at depth
|
|
103
|
+
|
|
104
|
+
The first cut of this fix held each TICK for 30 s, which sounds equivalent and
|
|
105
|
+
is not: `depth` independent deadlines expire inside one window, so the rate
|
|
106
|
+
stayed proportional to depth. Measured over 15 min of 2 s polls behind a gate
|
|
107
|
+
that never opens:
|
|
108
|
+
|
|
109
|
+
| inbox depth | before | per-tick hold | per-cadence hold (shipped) |
|
|
110
|
+
|---|---|---|---|
|
|
111
|
+
| 1 | 0.5/s | 0.03/s | 0.03/s |
|
|
112
|
+
| 87 (the live inbox) | 8.0/s | 2.9/s | 0.03/s |
|
|
113
|
+
| 240 | 8.0/s | 8.0/s — nothing | 0.03/s |
|
|
114
|
+
| 1,000 | 8.0/s | 8.0/s — nothing | 0.03/s |
|
|
115
|
+
| 15,674 (the write-up's depth) | 8.0/s | — | 0.03/s |
|
|
116
|
+
|
|
117
|
+
### Reading the reasons
|
|
118
|
+
|
|
119
|
+
If you want to know WHICH gate is shut — and for `meeting-action-capture` you
|
|
120
|
+
already do — tally them:
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
tail -20000 ~/<agent>/logs/cadence-bus/$(date -u +%F).jsonl \
|
|
124
|
+
| grep '"stage":"escalate_deferred"' \
|
|
125
|
+
| python3 -c 'import sys,json,collections
|
|
126
|
+
c=collections.Counter(json.loads(l).get("reason") for l in sys.stdin if l.strip())
|
|
127
|
+
print(c.most_common(10))'
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
`cadence-in-flight:sub-session` or `cadence-in-flight:handoff` dominating is the
|
|
131
|
+
known open item, not a regression in this change.
|
|
132
|
+
|
|
133
|
+
## 4. `processed/` needs nothing
|
|
134
|
+
|
|
135
|
+
`sweepRetention` already prunes `processed/<date>/` at a 14-day TTL on the
|
|
136
|
+
consumer's recovery timer, and the ~59k/62k files reported in the incident were
|
|
137
|
+
14 days of ticks at the storm's rate, not unpruned debris. Once deferrals fall
|
|
138
|
+
the directory shrinks on its own as old date dirs age out. Confirm:
|
|
139
|
+
|
|
140
|
+
```
|
|
141
|
+
ls ~/<agent>/state/cadence-bus/processed # expect <= 14 date dirs
|
|
142
|
+
find ~/<agent>/state/cadence-bus/processed -type f | wc -l
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Do **not** `rm -rf` it. `isProcessedId` reads it to suppress a replayed
|
|
146
|
+
explicit-id enqueue; emptying it re-opens that hole for 14 days.
|
|
147
|
+
|
|
148
|
+
To reclaim disk sooner, lower the TTL for one sweep rather than deleting by hand
|
|
149
|
+
— it is the same code path, with the guards:
|
|
150
|
+
|
|
151
|
+
```
|
|
152
|
+
MAESTRO_RETENTION_DAYS=3 <restart the daemon> # then put it back
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
## 5. The old logs are the real disk
|
|
156
|
+
|
|
157
|
+
The audit log, not the queue, is the 245 MB+: `logs/cadence-bus/<date>.jsonl`
|
|
158
|
+
ran 443 MB/day during the storm. It is pruned at the same 14-day TTL, so it
|
|
159
|
+
clears itself.
|
|
160
|
+
|
|
161
|
+
Scaling this seat's own bytes-per-deferral (443 MB/day at 4.8 deferrals/s) to
|
|
162
|
+
the post-fix rate puts a quiet day in the low single-digit MB. Do not hold the
|
|
163
|
+
fix to a specific byte figure: the deferral rows are only part of the file, the
|
|
164
|
+
rest scales with how much work the seat actually does, and the number that
|
|
165
|
+
matters is `deferred/s` in step 3. Nothing to do here unless the volume is
|
|
166
|
+
tight, in which case the oldest files are safe to delete — they are an audit
|
|
167
|
+
trail, not state.
|
|
168
|
+
|
|
169
|
+
## 6. Reset the misleading counter
|
|
170
|
+
|
|
171
|
+
`stats.retries` used to be incremented by every deferral, so a storming seat
|
|
172
|
+
reported over a million retries against a few dozen real ones. The counters are
|
|
173
|
+
in-memory and reset on restart; after the restart in step 1, `retries` reads
|
|
174
|
+
true. Nothing to clear.
|
|
175
|
+
|
|
176
|
+
## What NOT to do, and why
|
|
177
|
+
|
|
178
|
+
- **Do not `maestro emergency-stop`.** It halts every cadence on the seat to fix
|
|
179
|
+
a bus that is merely noisy. The daemon's measured cost during the storm was
|
|
180
|
+
7.2% of one core.
|
|
181
|
+
- **Do not restart repeatedly.** Circuit-breaker state is in-memory and a
|
|
182
|
+
restart means "try again now", so each restart re-opens every gate at once and
|
|
183
|
+
produces a thundering herd against whatever was failing.
|
|
184
|
+
- **Do not clear `dlq/`.** Nothing in this incident reached it — attempt
|
|
185
|
+
accounting is in `failTick` alone and the deferral path never calls it. A
|
|
186
|
+
non-empty `dlq/` is a different fault, and the files are the only record of it.
|
|
@@ -28,6 +28,22 @@
|
|
|
28
28
|
* (`scripts/daemon/assurance.generateAck`); this module never
|
|
29
29
|
* composes prose, it only decides whether prose is owed.
|
|
30
30
|
*
|
|
31
|
+
* THE FAILURE NOTICES IN THIS REGISTER NO LONGER GO TO PEOPLE AT ALL.
|
|
32
|
+
*
|
|
33
|
+
* Two of `RETIRED_FIRST_REPLIES`'s five entries are failure notices, and the
|
|
34
|
+
* register is the reason they cannot be re-emitted. But a register can only
|
|
35
|
+
* refuse what has already shipped: each fix added the screenshotted sentence
|
|
36
|
+
* here and shipped a new one, and the owner received the new one — three times,
|
|
37
|
+
* the last on 2026-10-01, twice in eleven minutes. The register is not the thing
|
|
38
|
+
* that stops the next one. `lib/assurance/notice-plane.mjs` (the notices do not
|
|
39
|
+
* reach a conversation) and `lib/comms/machine-status.mjs` (a sentence the tree
|
|
40
|
+
* has never contained is refused by SHAPE) are. Read those before adding a sixth
|
|
41
|
+
* entry here; if you are adding one, the structural fix has already failed.
|
|
42
|
+
*
|
|
43
|
+
* The register keeps its job for the ACK lines — "On it." and friends — which a
|
|
44
|
+
* stale seat still emits and which a shape test cannot safely catch, because a
|
|
45
|
+
* short human-sounding line is also what a real terse reply looks like.
|
|
46
|
+
*
|
|
31
47
|
* THE REGISTER, AND WHY IT IS A LIST OF EXACT SENTENCES
|
|
32
48
|
*
|
|
33
49
|
* `RETIRED_FIRST_REPLIES` holds the four sentences the owner screenshotted, in
|
|
@@ -0,0 +1,218 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* lib/assurance/notice-plane.mjs — WHERE a machine's status about itself goes.
|
|
3
|
+
*
|
|
4
|
+
* READ THIS BEFORE YOU EDIT A NOTICE'S WORDING. The grammar rules in
|
|
5
|
+
* `notice-voice.mjs` are downstream of this file, and this file says the
|
|
6
|
+
* notices do not go to people. If you are here to improve a sentence a human
|
|
7
|
+
* reads, you are in the wrong subsystem: no human reads these.
|
|
8
|
+
*
|
|
9
|
+
* WHY THIS MODULE EXISTS — THE THREE FIXES THAT WERE ALL ON THE WRONG AXIS
|
|
10
|
+
*
|
|
11
|
+
* The outcome notices have been iterated on hard. There is a register of
|
|
12
|
+
* retired sentences (`first-reply.RETIRED_FIRST_REPLIES`), a test that fails if
|
|
13
|
+
* a new line reproduces a retired one (`notice-emission.test.mjs`), a
|
|
14
|
+
* `claimsSuccess` guard, an `inventsFacts` guard, a per-ask generated voice, a
|
|
15
|
+
* quoted-topic fix so an arbitrary subject line stays grammatical, two dedupe
|
|
16
|
+
* layers, and a document-surface withholding. Every one of those improved the
|
|
17
|
+
* WORDING or the VOLUME of a message sent to a human. None asked whether it
|
|
18
|
+
* should be sent to a human at all.
|
|
19
|
+
*
|
|
20
|
+
* So the owner kept receiving a new variant after each fix. On 2026-10-01 he
|
|
21
|
+
* received, eleven minutes after complaining about receiving it:
|
|
22
|
+
*
|
|
23
|
+
* The work on "Cohort message" broke before it finished: the work ran past
|
|
24
|
+
* its time limit and got cut off. I'm running it again now.
|
|
25
|
+
*
|
|
26
|
+
* and a second seat posted the same shape about a different ask in the same
|
|
27
|
+
* screenshot, which establishes it as fleet behaviour rather than one desk.
|
|
28
|
+
*
|
|
29
|
+
* THE DECISION: MACHINE STATUS GOES TO THE OPERATOR PLANE, NEVER TO A PERSON.
|
|
30
|
+
*
|
|
31
|
+
* A notice about the machine's own internal failure — a session that died, a
|
|
32
|
+
* time limit that was hit, a retry that is now running — belongs in logs, the
|
|
33
|
+
* `needs-attention` escalation record, the board row and the counters. It does
|
|
34
|
+
* not belong in a conversation. Five reasons, in the order they are decisive:
|
|
35
|
+
*
|
|
36
|
+
* 1. IT IS NOT THE THING THEY ARE WAITING FOR. The legitimate need behind
|
|
37
|
+
* these notices — "do not leave someone waiting in silence" — is a need
|
|
38
|
+
* for the ANSWER. A bulletin about the absence of the answer does not
|
|
39
|
+
* advance the asker's goal; it transfers the machine's internal state to
|
|
40
|
+
* them and asks them to hold it.
|
|
41
|
+
* 2. IT IS UNACTIONABLE. "I'm running it again now" asks nothing of the
|
|
42
|
+
* reader and gives them nothing to decide. It is the one that was sent
|
|
43
|
+
* twice, and there was no action available either time.
|
|
44
|
+
* 3. THE COSTS ARE ASYMMETRIC AND THEY COMPOUND. Silence costs one
|
|
45
|
+
* unanswered ask. A notice costs the ask, plus credibility, plus a
|
|
46
|
+
* fan-out loop — all three measured in this repo's own comments: 11 of 14
|
|
47
|
+
* comments on one shared alerting doc were assurance notices burying a
|
|
48
|
+
* colleague's three real ones (`assurance.isDocumentSurface`); 142 of 188
|
|
49
|
+
* notices on 2026-09-30 came from seats acking each other's notices
|
|
50
|
+
* (`narratorPatterns`); the 25/27/30 Aug loop was the same shape.
|
|
51
|
+
* A notice is not a leaf — it is an inbound at every other seat.
|
|
52
|
+
* 4. THE OPERATOR SINK ALREADY WORKS AND WAS ALWAYS WRITTEN IN PARALLEL.
|
|
53
|
+
* `assurance.escalate` wrote a `needs-attention` row on every one of these
|
|
54
|
+
* failures, carrying the key, the cause, the sender, the trace and the
|
|
55
|
+
* session id — strictly more than the DM said. The DM was redundant with a
|
|
56
|
+
* sink that already existed. What was missing was never information; it
|
|
57
|
+
* was a person looking at the sink.
|
|
58
|
+
* 5. THE EMPIRICAL ARGUMENT IS DECISIVE. Three fixes on the wording axis
|
|
59
|
+
* produced three new variants and three more complaints. A wording axis
|
|
60
|
+
* cannot converge on a defect that is not on it.
|
|
61
|
+
*
|
|
62
|
+
* WHAT ABOUT THE NARROW CASE — "a human is blocked waiting on an ask they made
|
|
63
|
+
* and nothing else will reach them"? It does not survive as MACHINE PROSE, and
|
|
64
|
+
* the reason is worth stating because it is the case every previous author
|
|
65
|
+
* assumed:
|
|
66
|
+
*
|
|
67
|
+
* - If the retry succeeds, they get the answer and the notice bought nothing.
|
|
68
|
+
* - If it fails for good, what they need is a PERSON dispatched, not a
|
|
69
|
+
* sentence. That is an escalation, and escalation is this plane's job.
|
|
70
|
+
* - And if something must be said to them in words, the thing that says it is
|
|
71
|
+
* a SESSION that has read the thread and composed a reply — not the daemon
|
|
72
|
+
* stamping a cause label into a template. That distinction is the whole
|
|
73
|
+
* content of this module: a human-destined message is composed by something
|
|
74
|
+
* that read the conversation. A notice is not.
|
|
75
|
+
*
|
|
76
|
+
* So the default inverts: SILENT TO THE HUMAN, LOUD IN THE OPERATOR PLANE. The
|
|
77
|
+
* obligation is still never dropped — it still lands in `needs-attention`, in a
|
|
78
|
+
* counter and in the log — and `escalate`'s `requesterWasTold` is now honestly
|
|
79
|
+
* `false` instead of recording a message nobody wanted as a success.
|
|
80
|
+
*
|
|
81
|
+
* WHY THIS IS STRUCTURAL AND NOT A WORDING CHANGE
|
|
82
|
+
*
|
|
83
|
+
* ── AND IT IS ABOUT THE BODY, NOT ABOUT THE FUNCTION ────────────────────────
|
|
84
|
+
*
|
|
85
|
+
* The first pass of this fix read the question as "which composer produced this",
|
|
86
|
+
* and got one through its own gates on 2026-10-02: `assurance.composeSilentSuccess`
|
|
87
|
+
* called with no result text, which is a pure template ("I don't have a clean
|
|
88
|
+
* result to show you, though. Want me to run it again?") delivered as a
|
|
89
|
+
* `kind:"reply"`. It was excluded from the refusal on the reading that this
|
|
90
|
+
* composer "carries the answer" — true of one of its two shapes. A composer is
|
|
91
|
+
* not a plane. ASK WHAT THE BODY CARRIES: a work product is a reply, a template
|
|
92
|
+
* about the run is a notice, and the same function emits both.
|
|
93
|
+
*
|
|
94
|
+
* `noticePlane` is TOTAL over notice ids and its default is a REFUSAL, not a
|
|
95
|
+
* human send. A future author who adds a fourth notice kind, or renames one,
|
|
96
|
+
* gets `{plane:"operator", reason:"unregistered"}` — the new kind is silent to
|
|
97
|
+
* people until somebody deliberately registers it here, and registering it as
|
|
98
|
+
* human-facing requires editing this file, under this header. There is no
|
|
99
|
+
* sentence anyone can write that reaches a person by accident.
|
|
100
|
+
*
|
|
101
|
+
* PURE. No clock, no disk, no env, no network.
|
|
102
|
+
*
|
|
103
|
+
* @module lib/assurance/notice-plane
|
|
104
|
+
*/
|
|
105
|
+
|
|
106
|
+
"use strict";
|
|
107
|
+
|
|
108
|
+
/**
|
|
109
|
+
* The two planes a status message can land in.
|
|
110
|
+
*
|
|
111
|
+
* `OPERATOR` is logs, the `needs-attention` escalation record, the board row
|
|
112
|
+
* and the counters — surfaces an operator sweeps on purpose. `HUMAN` is a
|
|
113
|
+
* conversation someone is reading: a DM, a channel, a thread, a doc comment.
|
|
114
|
+
*/
|
|
115
|
+
export const NOTICE_PLANE = Object.freeze({
|
|
116
|
+
OPERATOR: "operator",
|
|
117
|
+
HUMAN: "human",
|
|
118
|
+
});
|
|
119
|
+
|
|
120
|
+
/**
|
|
121
|
+
* Every notice id this subsystem emits, and the plane it is allowed to reach.
|
|
122
|
+
*
|
|
123
|
+
* ALL OF THEM ARE OPERATOR, and that is the point of the table rather than an
|
|
124
|
+
* accident of its current contents. The table exists so that the plane is a
|
|
125
|
+
* DECLARED property of each notice, checked by a test, rather than a default
|
|
126
|
+
* that falls out of which function the author happened to call.
|
|
127
|
+
*
|
|
128
|
+
* Mirrors `assurance.NOTICE`. `notice-plane.test.mjs` pins that the two lists
|
|
129
|
+
* are the same set, so a notice added there without a row here fails the build
|
|
130
|
+
* rather than quietly inheriting a human destination.
|
|
131
|
+
*
|
|
132
|
+
* @type {Readonly<Record<string, string>>}
|
|
133
|
+
*/
|
|
134
|
+
export const NOTICE_PLANES = Object.freeze({
|
|
135
|
+
"failure:retrying": NOTICE_PLANE.OPERATOR,
|
|
136
|
+
"failure:final": NOTICE_PLANE.OPERATOR,
|
|
137
|
+
"interrupted": NOTICE_PLANE.OPERATOR,
|
|
138
|
+
// ── THE TWO THAT HAVE BOTH SHAPES ────────────────────────────────────────
|
|
139
|
+
//
|
|
140
|
+
// `silent-success` and `send-refused` are emitted by composers that ALSO
|
|
141
|
+
// produce a real reply. When the session's result text exists, the body is the
|
|
142
|
+
// work product plus one sentence of context and it is delivered as a `reply`,
|
|
143
|
+
// never as a notice, so it never consults this table. When it does not exist,
|
|
144
|
+
// what is left is a template about this machine's own run — "I don't have a
|
|
145
|
+
// clean result to show you, though. Want me to run it again?" — and that is
|
|
146
|
+
// a notice under these ids, hence operator.
|
|
147
|
+
//
|
|
148
|
+
// That distinction is the one the first pass of this fix got wrong: it read
|
|
149
|
+
// "which composer made it" as the question, and the honest question is "does
|
|
150
|
+
// this body carry the answer". A composer is not a plane.
|
|
151
|
+
"silent-success": NOTICE_PLANE.OPERATOR,
|
|
152
|
+
"send-refused": NOTICE_PLANE.OPERATOR,
|
|
153
|
+
});
|
|
154
|
+
|
|
155
|
+
/**
|
|
156
|
+
* The `kind` values `deliver()` receives for a machine-status emission.
|
|
157
|
+
*
|
|
158
|
+
* THIS IS THE ORIGIN GATE'S KEY, and it is the half of the fix that no wording
|
|
159
|
+
* can get around. `sayOutcomeNotice` passes `kind: "failure"` or
|
|
160
|
+
* `kind: "notice"` to the transport; those two strings say "a machine composed
|
|
161
|
+
* this about itself", independently of what the sentence says. A refusal keyed
|
|
162
|
+
* on them cannot be defeated by writing a better sentence, which is exactly the
|
|
163
|
+
* failure mode every previous fix had.
|
|
164
|
+
*
|
|
165
|
+
* Kept as data, in this module, next to the plane table, so the thing that
|
|
166
|
+
* decides and the thing that enforces read one list.
|
|
167
|
+
*
|
|
168
|
+
* @type {readonly string[]}
|
|
169
|
+
*/
|
|
170
|
+
export const MACHINE_STATUS_DELIVERY_KINDS = Object.freeze(["failure", "notice"]);
|
|
171
|
+
|
|
172
|
+
/** @param {unknown} kind @returns {boolean} */
|
|
173
|
+
export function isMachineStatusDeliveryKind(kind) {
|
|
174
|
+
return MACHINE_STATUS_DELIVERY_KINDS.includes(String(kind == null ? "" : kind).trim().toLowerCase());
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
/**
|
|
178
|
+
* Which plane may this notice reach?
|
|
179
|
+
*
|
|
180
|
+
* TOTAL, and the default is a refusal. An id this table does not know is
|
|
181
|
+
* `operator` with `reason:"unregistered"` — never `human`. That asymmetry is
|
|
182
|
+
* the anti-undo property: forgetting to register a new notice makes it quiet,
|
|
183
|
+
* not loud at the CEO.
|
|
184
|
+
*
|
|
185
|
+
* @param {unknown} notice a NOTICE id
|
|
186
|
+
* @returns {{plane:string, human:boolean, reason:string}}
|
|
187
|
+
*/
|
|
188
|
+
export function noticePlane(notice) {
|
|
189
|
+
const id = String(notice == null ? "" : notice).trim().toLowerCase();
|
|
190
|
+
if (!id) return { plane: NOTICE_PLANE.OPERATOR, human: false, reason: "no-notice-id" };
|
|
191
|
+
const declared = Object.prototype.hasOwnProperty.call(NOTICE_PLANES, id) ? NOTICE_PLANES[id] : null;
|
|
192
|
+
if (declared == null) return { plane: NOTICE_PLANE.OPERATOR, human: false, reason: "unregistered" };
|
|
193
|
+
if (declared === NOTICE_PLANE.HUMAN) return { plane: NOTICE_PLANE.HUMAN, human: true, reason: "declared-human" };
|
|
194
|
+
return { plane: NOTICE_PLANE.OPERATOR, human: false, reason: "declared-operator" };
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
/**
|
|
198
|
+
* May this notice be sent to a human conversation? Never, as the table stands.
|
|
199
|
+
*
|
|
200
|
+
* A separate named predicate rather than `noticePlane(x).human` at the call
|
|
201
|
+
* sites, so the one question the subsystem keeps getting wrong has one name and
|
|
202
|
+
* is greppable.
|
|
203
|
+
*
|
|
204
|
+
* @param {unknown} notice
|
|
205
|
+
* @returns {boolean}
|
|
206
|
+
*/
|
|
207
|
+
export function mayReachHuman(notice) {
|
|
208
|
+
return noticePlane(notice).human === true;
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
export default {
|
|
212
|
+
NOTICE_PLANE,
|
|
213
|
+
NOTICE_PLANES,
|
|
214
|
+
MACHINE_STATUS_DELIVERY_KINDS,
|
|
215
|
+
isMachineStatusDeliveryKind,
|
|
216
|
+
noticePlane,
|
|
217
|
+
mayReachHuman,
|
|
218
|
+
};
|
|
@@ -1,6 +1,39 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* lib/assurance/notice-voice.mjs — a failure
|
|
3
|
-
*
|
|
2
|
+
* lib/assurance/notice-voice.mjs — the words a failure notice wears, IN A LOG.
|
|
3
|
+
*
|
|
4
|
+
* ════════════════════════════════════════════════════════════════════════════
|
|
5
|
+
* READ `lib/assurance/notice-plane.mjs` FIRST. NOTHING THIS FILE COMPOSES IS
|
|
6
|
+
* SENT TO A PERSON.
|
|
7
|
+
*
|
|
8
|
+
* Everything below — the generated voice, `claimsSuccess`, `inventsFacts`, the
|
|
9
|
+
* register check, the quoted-topic grammar, the degraded fallback — was written
|
|
10
|
+
* for a message a human would read, and that premise is retired. These sentences
|
|
11
|
+
* are operator-plane artifacts now: one readable line per dead ask in
|
|
12
|
+
* `state/obligations/notice-journal.jsonl`, beside the `needs-attention` row.
|
|
13
|
+
*
|
|
14
|
+
* WHY THAT MATTERS FOR ANYONE EDITING THIS FILE. The notices were iterated on
|
|
15
|
+
* three times and each pass improved the wording; the owner received a new
|
|
16
|
+
* variant after each one, and on 2026-10-01 received the same one twice, eleven
|
|
17
|
+
* minutes apart, the second time after complaining. If you are here because a
|
|
18
|
+
* notice read badly to a human, the human should not have had it — the bug is a
|
|
19
|
+
* routing bug, and `notice-plane.mjs` is where it is answered. If you are here
|
|
20
|
+
* because an OPERATOR could not tell two journal rows apart, this is the right
|
|
21
|
+
* file.
|
|
22
|
+
*
|
|
23
|
+
* Two consequences, both already wired:
|
|
24
|
+
* - `assurance.generateFailureNotice` no longer spawns a model for these. A
|
|
25
|
+
* cold `claude --print` per dead obligation per sweep, to make a log line
|
|
26
|
+
* read nicely, is cost for nobody's benefit; the deterministic
|
|
27
|
+
* `fallbackNotice` is the right artifact for the operator plane and always
|
|
28
|
+
* was. The generator's guards stay — they are the defence for any future
|
|
29
|
+
* caller that is genuinely human-facing.
|
|
30
|
+
* - `lib/comms/machine-status.mjs` refuses every sentence this file can
|
|
31
|
+
* produce at the send chokepoint, and `machine-status.test.mjs` pins the
|
|
32
|
+
* whole `fallbackNotice` cross product (132 sentences), so a NEW cause or a
|
|
33
|
+
* NEW wording is refused too rather than discovered in a screenshot.
|
|
34
|
+
* ════════════════════════════════════════════════════════════════════════════
|
|
35
|
+
*
|
|
36
|
+
* HISTORICAL RECORD BELOW — kept because it explains why the guards exist.
|
|
4
37
|
*
|
|
5
38
|
* WHAT WAS WRONG, IN THE OWNER'S OWN SCREENSHOTS (2026-09-25)
|
|
6
39
|
*
|
package/lib/budget-escalate.mjs
CHANGED
|
@@ -41,6 +41,46 @@ import { maybeNotify } from "./budget-guard.mjs";
|
|
|
41
41
|
* @returns {Promise<{ok:boolean, notified:boolean, band:number, delivered:boolean,
|
|
42
42
|
* recipient:object|null, reason:string|null, status?:object}>}
|
|
43
43
|
*/
|
|
44
|
+
/**
|
|
45
|
+
* The supervisor's band notice.
|
|
46
|
+
*
|
|
47
|
+
* ── A DELIBERATE EXCLUSION FROM THE MACHINE-STATUS ROUTING LAW ───────────────
|
|
48
|
+
*
|
|
49
|
+
* `lib/assurance/notice-plane.mjs` says status about this machine's own apparatus
|
|
50
|
+
* goes to the operator plane and never to a conversation. This IS a templated
|
|
51
|
+
* message about the machine, it IS addressed to a human DM, and it does NOT pass
|
|
52
|
+
* through `scripts/daemon/deliver.mjs` — so neither the origin gate nor the shape
|
|
53
|
+
* gate sees it. An adversarial pass on 2026-10-02 found it unmapped and asked
|
|
54
|
+
* whether it was an oversight. It is not, and the reason is the test the plane
|
|
55
|
+
* law actually applies:
|
|
56
|
+
*
|
|
57
|
+
* IT IS ADDRESSED TO THE PERSON WHO CAN FIX IT. A failure notice went to the
|
|
58
|
+
* requester, who can do nothing about a dead session. This goes to the SEAT'S
|
|
59
|
+
* SUPERVISOR — the one party with the authority to raise the envelope.
|
|
60
|
+
* IT NAMES THE REMEDY. "Raise `payBasis.meteredBudget` on the member's
|
|
61
|
+
* Employee record" is an action, in the reader's own hands, that clears the
|
|
62
|
+
* state the message is about. `notice-plane` reason #2 — unactionable — does
|
|
63
|
+
* not apply.
|
|
64
|
+
* IT IS NOT ABOUT A DEAD RUN. A spend band is a standing budget fact, not a
|
|
65
|
+
* bulletin about the absence of an answer somebody is waiting for, so reason
|
|
66
|
+
* #1 does not apply either. Nothing fans out: it is one DM per band per day,
|
|
67
|
+
* dedupe-keyed server-side, to one person.
|
|
68
|
+
*
|
|
69
|
+
* Pinned rather than merely asserted: `budget-escalate.test.mjs` runs this body
|
|
70
|
+
* through `machineStatusNarration` and requires it to pass, so the exclusion is a
|
|
71
|
+
* passing test instead of an omission from a map. If a future edit makes this
|
|
72
|
+
* read as apparatus status, that test fails and the wording — or the exclusion —
|
|
73
|
+
* has to be reconsidered deliberately.
|
|
74
|
+
*
|
|
75
|
+
* @param {{band:number, message:string}} notice
|
|
76
|
+
* @returns {string}
|
|
77
|
+
*/
|
|
78
|
+
export function composeBudgetNotice(notice) {
|
|
79
|
+
// The body names the seat, the band, the money and the enforcement now in
|
|
80
|
+
// force — a supervisor must be able to act on it without opening a dashboard.
|
|
81
|
+
return `**Seat budget — band ${notice.band}%**\n\n${notice.message}\n\n_This notice goes to you rather than to the seat: an agent must not be handed its own meter. Raise the envelope on the member's Employee record (payBasis.meteredBudget) to clear it._`;
|
|
82
|
+
}
|
|
83
|
+
|
|
44
84
|
export async function escalateBudget(o = {}, deps = {}) {
|
|
45
85
|
const log = typeof deps.log === "function"
|
|
46
86
|
? deps.log
|
|
@@ -89,9 +129,7 @@ export async function escalateBudget(o = {}, deps = {}) {
|
|
|
89
129
|
return { ok: false, notified: true, band: notice.band, delivered: false, recipient: notice.recipient, reason: "dm-resolve-threw", status: notice.status };
|
|
90
130
|
}
|
|
91
131
|
|
|
92
|
-
|
|
93
|
-
// force — a supervisor must be able to act on it without opening a dashboard.
|
|
94
|
-
const body = `**Seat budget — band ${notice.band}%**\n\n${notice.message}\n\n_This notice goes to you rather than to the seat: an agent must not be handed its own meter. Raise the envelope on the member's Employee record (payBasis.meteredBudget) to clear it._`;
|
|
132
|
+
const body = composeBudgetNotice(notice);
|
|
95
133
|
|
|
96
134
|
try {
|
|
97
135
|
const send = deps.sendImpl || (await import("./org/messaging.mjs")).sendMessage;
|
|
@@ -130,4 +168,4 @@ export async function escalateBudget(o = {}, deps = {}) {
|
|
|
130
168
|
return { ok: true, notified: true, band: notice.band, delivered: true, recipient: notice.recipient, reason: null, status: notice.status };
|
|
131
169
|
}
|
|
132
170
|
|
|
133
|
-
export default { escalateBudget };
|
|
171
|
+
export default { escalateBudget, composeBudgetNotice };
|