projmux 0.14.2 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,555 @@
1
+ # Replacement completion and restorability
2
+
3
+ Three layers of this application hold an execution image, and every consumer of
4
+ `make install`, `npm install`, or a provider generation upgrade believes one
5
+ sentence about all three at once. That sentence is false in a different way on
6
+ each layer. This document fixes what is guaranteed, what is not, and how a
7
+ report can be recovered from the tokens the diagnosis prints.
8
+
9
+ The layers are named `L1`, `L2`, and `L3` throughout, and they are the rows of
10
+ the `projmux doctor` replacement table:
11
+
12
+ | Layer | Subject | What holds an image here |
13
+ | --- | --- | --- |
14
+ | `L1` | `installed-executable-image` | the file the installed path publishes |
15
+ | `L2` | `long-lived-projmux-processes` | every running child of that file |
16
+ | `L3` | `provider-sessions-and-generations` | provider sessions and the managed generation pool |
17
+
18
+ `projmux doctor` renders the table on an unfiltered run and under
19
+ `projmux doctor --section replacement`. It is read-only in the same sense as
20
+ every other section: the Registry read is the zero-write snapshot read, the
21
+ process table is two file reads per process, and nothing on this path starts,
22
+ signals, stops, or repairs anything it reports.
23
+
24
+ ## C-1 Replacement completion and restorability
25
+
26
+ **Assumption.** A consumer — an operator, an owning session, or CI — believes
27
+ that *from the moment `install` or `update` returns success, the old execution
28
+ image accepts no new work on any of the three layers*.
29
+
30
+ **Guarantee.** Per layer, what is actually true:
31
+
32
+ - `L1` publishes the new binary with a rename, so a process that starts after
33
+ the rename gets the new image. The *install* around that rename is not
34
+ atomic: config convergence runs before the rename and again after it, and a
35
+ failure between the two leaves a published binary whose live config was never
36
+ converged. There is no atomic install; there is one atomic step inside a
37
+ non-atomic install.
38
+ - `L2` is replaced for one role and reported for the rest. Replacing a file does
39
+ not replace the image of a process already running it, so an install has to
40
+ ask. `broker-runtime` is asked, through the drain this application already
41
+ ships: a runtime whose own image has been unlinked drains at the next session
42
+ that reaches it, refuses new work with `drain-required`, and closes when its
43
+ last binding goes. Every other role is reported with the route that does
44
+ replace it and is left alone — see *The L2 replacement policy* below. A
45
+ replacement that has not finished within the drain cutoff is reported as
46
+ `replacement-cutoff-reached` and is still not ended.
47
+ - `L3` cannot enter `draining` without a qualified version pair. Both doors into
48
+ a generation switch require a measured receipt for exactly the pair they are
49
+ about: managed activation refuses before it writes the journal, and a handover
50
+ resume refuses before it drives any effect — the one exception being a
51
+ generation nothing is bound to, which the planner has always been allowed to
52
+ retire because a receipt proves a thread survives a cross-version resume and
53
+ such a generation has no thread to carry. Each refusal names the action that
54
+ clears it, and that action is now runnable: `scripts/test-generation-pool-qualification.sh`
55
+ measures a declared pair in isolation and
56
+ `projmux agent app-server upgrade qualify --receipt <absolute-json>` installs
57
+ the receipt it writes where both doors read it. **What is not guaranteed is
58
+ the measurement's honesty.** The gate checks that the receipt's coverage
59
+ counters can have come from a run — an acceptance claim with no observation
60
+ behind it is refused as forged — but a determined forger who writes a coverage
61
+ count of one is not contradicted by anything else in the receipt. The gate
62
+ makes the receipt state its coverage; it does not attest to it.
63
+
64
+ **Non-Guarantee.** Explicitly outside this contract:
65
+
66
+ - Darwin observation of `L1` and `L2`. There is no `/proc/<pid>/exe`, so neither
67
+ the image link nor the process census can be taken, and both rows report
68
+ `unsupported-platform` with the `unknown` verdict on both axes. This contract
69
+ does not build a substitute observation for that platform; it states the gap.
70
+ - The provider's own upstream seamless-upgrade contract. `L3` reasons about the
71
+ pool this application manages, never about what the provider promises.
72
+ - An externally owned (`unmanaged`) app-server. Its state is not this
73
+ application's to observe or restore.
74
+ - Automatic classification of which changes form one invariant. See C-2.
75
+
76
+ **Scope.** In space: one machine's projmux installation, the long-lived
77
+ processes it owns, and the generation pool it manages. In time: from the return
78
+ of one `install` or `update` to the next. **Expiry: a verdict expires when the
79
+ installed binary's SHA changes.** A table printed before an install describes an
80
+ installation that no longer exists.
81
+
82
+ **Failure and recovery.** Per layer:
83
+
84
+ | Layer | Detection signal | Recovery route |
85
+ | --- | --- | --- |
86
+ | `L1` | this reader's own executable link carries the kernel's `(deleted)` suffix | none for the binary. No copy of the replaced image is retained, and the recovery text the install prints on failure is config convergence, never a rollback. Restorability is `not-restorable` on every supported path. |
87
+ | `L2` | the whole-fleet vintage census, the newest `install-residue.jsonl` record whose own census observed anything, and the last `install-replacement.json` pass | end and relaunch the residual process through the routes this application already ships. The drain cutoff bounds the wait, and reaching it changes the row rather than the fleet, so a failed replacement leaves every process exactly where a successful one would have found it. |
88
+ | `L3` | generation-pool status, its qualification result, and the Running-versus-live-session census | produce a receipt for the pair with `scripts/test-generation-pool-qualification.sh <old> <new> <output-dir>` and install it with `projmux agent app-server upgrade qualify --receipt <absolute-json>`. **The producer is the recovery route**: a pool that can be qualified can be entered, and one that cannot is refused before it enters. The 2026-09-07 measurement — no route back once `draining` was entered without a verdict — held because nothing read a produced receipt; the entry paths now refuse rather than reach that state. |
89
+
90
+ **Enforcement.** `TestDoctorReplacementLayerVerdictsAreFixedByInputCombination`
91
+ fixes every input combination to its two verdicts and its token.
92
+ `TestDoctorReplacementUnknownVerdictsNameTheEvidenceGap` requires each `unknown`
93
+ to name the evidence that was missing.
94
+ `TestDoctorReplacementDarwinReportsUnsupportedPlatformForImageAndProcessLayers`
95
+ holds the platform branch.
96
+ `TestDoctorReplacementQualificationMissingMakesGenerationPoolNotRestorable`
97
+ holds the `L3` mapping and
98
+ `TestDoctorReplacementL3RestorationMovesWhenTheQualificationLaneOpens` holds the
99
+ transition across it. For the `L3` guarantee above,
100
+ `TestActivateManagedCurrentRefusesEveryUnqualifiedRequestBeforeDraining` and
101
+ `TestResumeRefusesAnUnqualifiedPairBeforeAnyHandoverEffect` hold the two doors,
102
+ `TestResumeStillRetiresAVacantGenerationWithoutAReceipt` holds the vacancy
103
+ exception, and `TestQualificationGateRefusesEvidenceCountersNoObservationBacks`
104
+ holds the forgery refusal. For the `L2` guarantee above,
105
+ `TestBrokerRuntimeDrainsWhenItsOwnImageWasReplaced` holds the vintage entry
106
+ condition and that a runtime on the installed image is not drained by it,
107
+ `TestReplacementRolePoliciesMatchTheContractDocument` and
108
+ `TestReplacementSessionClientIsNeverADrainTarget` hold the policy table, and
109
+ `TestInstallReplacementPassOutcomesAreFixedByFleetAndRequest` fixes every
110
+ outcome of the install pass.
111
+
112
+ ## C-2 The unit of replacement is an invariant, not a file
113
+
114
+ **When one invariant is held by code in several files, installing part of that
115
+ set is an atomicity violation.** The file boundary is not the unit; the
116
+ invariant is. A build that carries half of one invariant is not a smaller
117
+ version of the change — it is a state neither the old code nor the new code was
118
+ written to produce.
119
+
120
+ **The 2026-09-08 split install.** Removing a gate in
121
+ `claude_explicit_reply.go` and routing `PutReply` through the target-derived
122
+ adapter in `store.go` were one invariant. Only the first was installed. The
123
+ intermediate build wrote one Agent-message record with `adapter=codex-inbox`
124
+ against `target.provider=claude`, and the strict record validator then refused
125
+ the whole store: **46 records became unreadable, and `agent message send` failed
126
+ in every session on the machine.** Expiry and prune both run after a load, so
127
+ the store could not clear the bad record by itself — the failure was not
128
+ self-healing. The recovery was not deleting the file; it was rewriting the
129
+ mismatched `adapter` to the value the record's own target implies.
130
+
131
+ **And `L2` widens that window.** `make install` does not replace the image of a
132
+ running helper, so a process started from the intermediate build keeps writing
133
+ bad records after the source file has been corrected. That is exactly what
134
+ happened: the record was corrected once and reappeared. The three layers do not
135
+ fail independently — an `L1` partial publication is multiplied by the `L2`
136
+ processes still running the partial image.
137
+
138
+ **Non-Guarantee.** This contract does **not** decide automatically which change
139
+ set forms one invariant. There is no analysis here that groups files by the
140
+ invariants they hold, and none is implied. The contract says only that a partial
141
+ install of such a set is an atomicity violation, and that `L2` extends how long
142
+ that violation stays observable.
143
+
144
+ **Enforcement.** The `L2` row is what makes the widened window visible at all:
145
+ `TestDoctorReplacementLayerVerdictsAreFixedByInputCombination` fixes
146
+ `residual-processes-present` and `install-residue-recorded` to `not-replaced`,
147
+ so a report taken after an install states how much of the fleet that install did
148
+ not reach.
149
+
150
+ ## C-3 A failure path preserves its discriminant
151
+
152
+ **Assumption.** A consumer believes that *when a refusal or failure is reported,
153
+ the reason token arrives together with enough of the underlying evidence to
154
+ identify which case produced it*.
155
+
156
+ **Guarantee.** This is a **partial** guarantee today, and points where the
157
+ underlying evidence is discarded are known. On the surface this document owns,
158
+ the guarantee is total: **every row carries at least one discriminant, and a row
159
+ with a token and no discriminant is a contract violation**.
160
+
161
+ **Non-Guarantee.** Exposing the *whole* underlying text on a user-facing
162
+ surface. What is required is a **recoverable discriminant**, not a dump, and
163
+ this contract does not dictate a logging surface.
164
+
165
+ **The 2026-09-08 case.** The only symptom was the single line
166
+ `malformed Agent message store`. It named neither the offending record nor the
167
+ condition that rejected it. A peer session had to re-implement the record
168
+ validator and walk all 46 records to identify the one. The `status` route took
169
+ the same load, so the fallback diagnosis path was dead too. **Failing closed was
170
+ correct. A fail-closed with no discriminant, combined with a deadlock, is
171
+ indistinguishable from unrecoverable.**
172
+
173
+ **How this surface preserves the discriminant.** Each row carries a `signals`
174
+ list of `key=value` pairs beside its reason token. The keys come from a closed
175
+ inventory and the values are either counters this surface formats or tokens
176
+ drawn from another closed vocabulary. `residual-processes-present` and
177
+ `install-residue-recorded` both say an install left work behind; only
178
+ `processes.residual`, `residual.role.*`, `residual.oldest-seconds`, and
179
+ `ledger.latest.residual` say how much, of what, how old, and whether the
180
+ reading was live or from the ledger. A `residual.role.*` key is emitted only
181
+ for a role that has at least one residual process, so the row states what an
182
+ install did not replace and never carries a zero.
183
+
184
+ **Where the discriminant is published, and why.** The signals are serialized in
185
+ `--json`, unlike `doctorFinding.Details`, which is `json:"-"` in every format.
186
+ That difference is deliberate and rests on one fact: a stored Registry refusal
187
+ reason quotes absolute paths, so it cannot be serialized without the support
188
+ report depending on a redaction allowlist. **No signal value here is free-form.**
189
+ Every value is a counter or a closed token, a regular expression rejects
190
+ anything else at construction, and a value that fails it is replaced with
191
+ `unclassified` rather than serialized. No path, argv word, pid, Agent UID, Pane
192
+ UID, thread id, or prose sentence can reach the JSON on this path. Because the
193
+ discriminant is closed, it needs no terminal-only channel, and putting it in
194
+ JSON is what lets CI and a support report recover a verdict at all.
195
+
196
+ **Registry `Running` is not a provider session.** `Running` is a statement about
197
+ a Pane's managed activation. It is not a statement about the provider session
198
+ behind it: a Pane can be present in the runtime while the session it was bound
199
+ to is gone, and the Registry spells both `Running`. The discriminant differs by
200
+ provider and that difference is structural — a Claude Pane carries a real
201
+ process handle at `status.activation.claude.process`, so liveness is a direct
202
+ question about a pid; a Codex Pane carries no pid at all, only a thread and a
203
+ composite authority, so the only checkable fact is whether that authority still
204
+ belongs to the runtime currently serving the state domain. The census therefore
205
+ reports three counts — `sessions.live`, `sessions.dead`, `sessions.unobservable`
206
+ — and the third is not a rounding bucket: it is the count of Running Agents
207
+ resting on the Registry alone.
208
+
209
+ The census is **count-only**. It names no Agent and no Pane, for the reason the
210
+ managed-authority census states: a diagnostics surface that names Agents is one
211
+ that cannot be shared. It also **repairs nothing**. A mismatch is reported and
212
+ left exactly as it was found.
213
+
214
+ **Enforcement.**
215
+ `TestDoctorReplacementEveryReasonTokenCarriesARecoverableDiscriminant` requires
216
+ zero rows with a token and no discriminant across every reachable input
217
+ combination. `TestDoctorReplacementSignalValuesCarryNoPathProcessOrFreeFormText`
218
+ holds the serialization rule, and
219
+ `TestDoctorReplacementSignalKeysMatchTheContractDocument` holds the key
220
+ inventory equal to the one published below.
221
+ `TestDoctorReplacementCensusSeparatesRegistryRunningFromLiveProviderSession`
222
+ fixes the Running-versus-live-session split against a fixture.
223
+ `TestDoctorRendersReplacementTableInDefaultRunAndSectionFilter` holds the output
224
+ contract for the unfiltered run, the section filter, and the JSON surface, and
225
+ `TestDoctorReplacementSectionCreatesNothing` holds the read-only claim by
226
+ running the section against an empty home and requiring it to stay empty.
227
+
228
+ ## Reason token vocabulary
229
+
230
+ Exactly one token governs each row: the first entry of its layer's list below
231
+ that the evidence selects. Everything else the layer observed stays in that
232
+ row's signals, so a fact that loses the precedence contest is never lost from
233
+ the report. The lists are in precedence order.
234
+
235
+ `TestDoctorReplacementReasonTokensMatchTheContractDocument` holds this table and
236
+ the token constants in the code equal, in both directions.
237
+
238
+ | Layer | Token | Replacement | Restoration | Selected when |
239
+ | --- | --- | --- | --- | --- |
240
+ | `L1` | `unsupported-platform` | `unknown` | `unknown` | the platform exposes no executable link |
241
+ | `L1` | `image-unresolved` | `unknown` | `unknown` | this reader's own image could not be resolved back to its executable path |
242
+ | `L1` | `image-unlinked` | `not-replaced` | `not-restorable` | this reader's image was unlinked out from under it, so the diagnosis is taken from a superseded build |
243
+ | `L1` | `image-current` | `replaced` | `not-restorable` | the installed path still publishes the image this diagnosis runs |
244
+ | `L2` | `unsupported-platform` | `unknown` | `unknown` | the platform exposes no process table |
245
+ | `L2` | `replacement-cutoff-reached` | `not-replaced` | `restorable` | a residual process has outlived the drain cutoff, so this replacement is not going to complete on its own |
246
+ | `L2` | `residual-processes-present` | `not-replaced` | `restorable` | a live child is still running the image from before the last install, and none of them is past the cutoff |
247
+ | `L2` | `no-residual-processes` | `replaced` | `restorable` | live children were observed and every one runs the installed image |
248
+ | `L2` | `install-residue-recorded` | `not-replaced` | `restorable` | no live child was observable, and the newest ledger record whose own census observed anything says that install left residue |
249
+ | `L2` | `install-residue-clean` | `replaced` | `restorable` | no live child was observable, and the newest ledger record whose own census observed anything says that install left none |
250
+ | `L2` | `no-observed-processes` | `unknown` | `unknown` | neither the live census nor any ledger record observed anything. Every reading is silent; none says the fleet is clean |
251
+ | `L3` | `generation-pool-unobserved` | `unknown` | `unknown` | no pool diagnosis was read |
252
+ | `L3` | `registry-running-provider-session-dead` | pool-derived | `not-restorable` | provider evidence contradicts a Registry `Running` Agent |
253
+ | `L3` | `qualification-missing` | pool-derived | `not-restorable` | the pool is installed and carries no qualification result. A pool in this state predates the entry gates or had its receipt removed; the row's `pool.action` names the producer that clears it |
254
+ | `L3` | `generation-pool-blocked` | pool-derived | `not-restorable` | the pool diagnosis is blocked |
255
+ | `L3` | `generation-handover-required` | pool-derived | `restorable` | the pool has a pending operation with a handover route |
256
+ | `L3` | `registry-running-session-unobserved` | pool-derived | `unknown` | Running Agents rest on the Registry alone, with no provider handle to check |
257
+ | `L3` | `generation-pool-not-installed` | `not-replaced` | `unknown` | no generation journal exists. Absence of a journal is absence of evidence about a restore route, not evidence of one |
258
+ | `L3` | `generation-pool-ready` | pool-derived | `restorable` | the pool is installed, qualified, and settled |
259
+
260
+ `pool-derived` means the replacement axis follows the pool rather than the
261
+ token: `replaced` once a generation is draining, handover-pending, or retired —
262
+ because such a generation accepts no new admission, which is exactly the
263
+ sentence C-1's Assumption makes — and `not-replaced` otherwise.
264
+
265
+ ## Process role vocabulary
266
+
267
+ `L2` counts every running child of the installed executable, and it names them
268
+ by **route** — the internal command words of the argv, with `argv[0]` and
269
+ everything behind a bare `--` discarded. That is the whole classifier. No pid,
270
+ no path, and no argv word reaches a diagnostics surface from it; the role name
271
+ is a token from the closed list below, and the value beside it is a count.
272
+
273
+ The same vocabulary is used by `projmux doctor`'s `L2` row and by every
274
+ `install-residue.jsonl` record, so the two surfaces name the same processes with
275
+ the same words.
276
+
277
+ | Role | Route | Cardinality | What ending it costs |
278
+ | --- | --- | --- | --- |
279
+ | `broker-runtime` | `internal codex-broker serve` | one per machine | nothing directly: a new broker starts on the installed image when a binding needs one |
280
+ | `lifecycle-observer` | `internal agent-hook ingest codex-broker-watch` | one per Codex pane | that pane's lifecycle ingestion until it is relaunched |
281
+ | `supervisor` | `internal supervise` | one per pane | the pane it supervises |
282
+ | `session-client` | `shell` | one per attached client | **the operator's whole attached session.** This role is the reason a drain cannot be uniform: every other role is background work, and this one is the person at the terminal |
283
+ | `agent-endpoint` | `internal claude-endpoint-helper` | one per registered agent activation | that agent's messaging endpoint until its pane is recreated |
284
+ | `usage-watcher` | `internal status usage --watch-codex-rate-limits` | one per machine, lease-held | rate-limit sampling until the lease is taken again |
285
+ | `other` | every route not named above | unbounded | — |
286
+
287
+ **`other` is a total guard, not a count of replacement targets.** It exists so
288
+ that a route this census has no name for cannot make the fleet look smaller than
289
+ it is, and naming more roles shrinks it without ever removing it. What remains
290
+ in it is the **short-lived invocation**: a status or statusbar render, a preview,
291
+ a picker, a hook callback, a `config apply`, the residue census itself. Those
292
+ processes are observed by whichever census happens to overlap them and are gone
293
+ seconds later. They are counted because the total has to be whole, and they are
294
+ not drain candidates because nothing has to end them.
295
+
296
+ That distinction is what the named roles buy. Before them, `other` was the
297
+ second-largest bucket on this repository's own ledger — 271 of 866 residual
298
+ observations — and it mixed the two kinds, so no reader could tell a long-lived
299
+ process an install failed to replace from a two-second render that happened to
300
+ be running. A residual `other` count is now read as measurement noise unless it
301
+ is large or persistent, in which case it names a long-lived route this list is
302
+ missing.
303
+
304
+ **Adding a role is not a change to the reason-token vocabulary.** The reason
305
+ tokens above are the closed list of *verdicts*; roles are the subjects those
306
+ verdicts are counted over. A role added here reaches the signal key inventory
307
+ automatically, because that inventory expands the census role order, and
308
+ `TestReplacementProcessRolesMatchTheContractDocument` holds this table and the
309
+ code's role order to each other in both directions.
310
+
311
+ ## The L2 replacement policy
312
+
313
+ An install may end a long-lived process only where this application already
314
+ ships a way to ask one to stand down, and only where ending it costs nothing an
315
+ operator did not choose. Both halves of that sentence are decisions taken from
316
+ measurement, and they are fixed here rather than at a call site.
317
+
318
+ ### Per-role disposition
319
+
320
+ `drain` means the install asks through a shipped path and the process decides
321
+ when it actually goes. `report-only` means no install touches it; the route that
322
+ does replace it is named instead, so a reader of the row knows what would.
323
+
324
+ `TestReplacementRolePoliciesMatchTheContractDocument` holds this table and the
325
+ code's policy map equal, in both directions.
326
+
327
+ | Role | Disposition | Replacement route |
328
+ | --- | --- | --- |
329
+ | `broker-runtime` | `drain` | `broker-drain` |
330
+ | `lifecycle-observer` | `report-only` | `pane-relaunch` |
331
+ | `supervisor` | `report-only` | `pane-relaunch` |
332
+ | `session-client` | `report-only` | `operator-reattach` |
333
+ | `agent-endpoint` | `report-only` | `pane-relaunch` |
334
+ | `usage-watcher` | `report-only` | `lease-expiry` |
335
+ | `other` | `report-only` | `process-exit` |
336
+
337
+ **Exactly one role drains, and that is a fact about what exists rather than a
338
+ first instalment.** A drain is a protocol: the runtime has to be able to hear
339
+ the request, refuse new work without failing anonymously, and carry the work it
340
+ already accepted to its end. `broker-runtime` is the only role that ships one.
341
+ Every other role is replaced by an event that already happens — a pane
342
+ recreated, an activation restarted, a lease taken again — and an install that
343
+ signalled them would sever live work to save an operator an action they can take
344
+ themselves.
345
+
346
+ **`session-client` is the entry that is a decision.** It is the operator's own
347
+ attached tmux session, the longest-lived role on this repository's ledger at
348
+ 169h29m, and a uniform cutoff applied to it ends the terminal the install was
349
+ typed into. It is report-only and no cutoff reaches it.
350
+ `TestReplacementSessionClientIsNeverADrainTarget` holds that as a property of
351
+ the table rather than of any call site.
352
+
353
+ **No disposition severs work.** The one hard kill this repository has measured —
354
+ 2026-09-05, four panes — brought three of the four back automatically and left
355
+ the fourth wedged on `backlog-overflow`. A recovery asymmetry that large is what
356
+ a default has to be chosen against, so `drain` asks and waits, and the cutoff
357
+ below reports rather than escalates.
358
+
359
+ ### The vintage trigger
360
+
361
+ The drain already had one entry condition: a client arrives speaking a protocol
362
+ version this runtime cannot negotiate. That condition misses the case an install
363
+ produces. A new binary usually speaks the same protocol, so a compatible
364
+ handshake proves nothing about which image is behind it.
365
+
366
+ The second entry condition is the runtime's own image. A runtime reads
367
+ `/proc/self/exe` once per arriving session until the answer is yes; the kernel's
368
+ `(deleted)` suffix says the file it is running was unlinked, which is exactly
369
+ what a publication leaves behind. The runtime then enters **the same drain by
370
+ the same door** — the same `drain-required` refusal, the same wire frames, the
371
+ same "live work is carried to its end". No new protocol, no new refusal token,
372
+ and nothing added to the wire.
373
+
374
+ Two consequences follow from putting the trigger there rather than in the
375
+ installer. Any client of the newly installed binary drains a superseded runtime,
376
+ so the npm install path is covered without a second implementation. And a
377
+ platform with no executable link to read declines to drain rather than draining
378
+ on a guess: `defaultProjmuxImageReplaced` answers false on darwin, and the
379
+ absence is stated by this table's `unsupported-platform` row.
380
+
381
+ ### The install pass
382
+
383
+ `projmux internal install-replace` runs as a step of `make install`, immediately
384
+ before the residue census so the census measures the fleet the pass left. It
385
+ takes the census, splits the residual processes by disposition, dials the
386
+ published broker runtime for this state domain, waits a bounded moment, and
387
+ writes `install-replacement.json`.
388
+
389
+ It **starts nothing** — a replacement pass that launched what it was sent to
390
+ replace would leave more behind than it found, so it dials and never ensures.
391
+ It **signals nothing**: the only request it makes is a socket handshake. And it
392
+ never fails an install: it runs after the install has already succeeded, and
393
+ everything it could not do is on the record it writes.
394
+
395
+ Its outcome vocabulary is closed, and it reaches the `L2` row as
396
+ `replacement.outcome`:
397
+
398
+ | Outcome | Meaning |
399
+ | --- | --- |
400
+ | `replacement-unsupported-platform` | no process table to take a census from |
401
+ | `replacement-no-target` | no residual process sits in a drainable role |
402
+ | `replacement-complete` | the drain was asked for and finished inside the settle window |
403
+ | `replacement-drain-pending` | the drain was accepted and is still carrying work |
404
+ | `replacement-target-unreachable` | a residual target the shipped path could not reach; `replacement.refusal` says which door was closed |
405
+ | `replacement-not-attempted` | written by no pass — what the row says when no record exists at all |
406
+
407
+ ### The drain cutoff
408
+
409
+ A drain that carries live work to its end has no bound of its own: the runtime
410
+ stays up while it still has bindings, and a binding that is never released keeps
411
+ it up forever. **The install has to decide when to stop calling such a
412
+ replacement "in progress" and start calling it unfinished.**
413
+
414
+ `projmux internal install-residue --survival` answers that from measurement.
415
+ On this repository's ledger at 2026-09-08 — 55 records, 884 residual
416
+ observations, **137 distinct processes** — the fraction of residual processes
417
+ still running at each cutoff is:
418
+
419
+ | Cutoff | Residual processes still running | Truncation rate |
420
+ | --- | --: | --: |
421
+ | 1h | 84 of 137 | 61.3% |
422
+ | 4h | 38 of 137 | 27.7% |
423
+ | 12h | 19 of 137 | 13.9% |
424
+ | **24h** | **13 of 137** | **9.5%** |
425
+ | 72h | 7 of 137 | 5.1% |
426
+ | 168h | 4 of 137 | 2.9% |
427
+
428
+ These are the **process-identity** numbers, not the observation numbers. The
429
+ section below says why the two bases disagree in both directions at once and why
430
+ a cutoff chosen from the observation basis would be wrong at both ends.
431
+
432
+ **Adopted: 24h. Truncation rate 9.5% — 13 of 137 by process identity.**
433
+
434
+ The reasoning is the shape of the curve against the cost of being wrong in
435
+ either direction. 1h and 4h report a majority and a large minority of ordinary
436
+ drains as failures, which turns the token into noise rather than a discriminant.
437
+ Past 24h the curve flattens: 24h to 72h buys 4.4 points and costs two more days
438
+ before an operator learns a replacement is never going to complete. 24h is also
439
+ the horizon the answer is read on — a residual process a day old is not a drain
440
+ still finishing its work, it is a drain that is not going to.
441
+
442
+ **The truncation rate is what gets reported, not what gets severed.** Reaching
443
+ the cutoff ends the install's claim about the replacement and never the process:
444
+ the `L2` row turns to `replacement-cutoff-reached`, `replacement.beyond-cutoff`
445
+ counts how many are past it, and every one of them keeps running. That is the
446
+ whole difference between this bound and a kill timer, and it is why 9.5% is a
447
+ reporting rate rather than a severed-session rate.
448
+
449
+ The cutoff is `PROJMUX_REPLACEMENT_CUTOFF`, a Go duration, and it exists so the
450
+ branch is reachable in a bounded test and in an isolated-fleet smoke. An unset,
451
+ unparsable, or non-positive value is the adopted 24h. **There is no value that
452
+ disables it**: an unbounded drain is the state this bound rules out, and offering
453
+ a spelling for it would put that state one environment variable away.
454
+
455
+ **Re-evaluation.** The identity basis rests on 137 processes of which 128 carry a
456
+ start reconstructed by subtraction, exact to one second. As records written with
457
+ a recorded start instant accumulate, that proportion falls and the tail of this
458
+ distribution sharpens. The cutoff is re-derived from the same command when it
459
+ does.
460
+
461
+ ## Reading residual survival: observations are not processes
462
+
463
+ The ledger records one census per install, and installs on a development
464
+ machine land minutes apart. A long-lived process is therefore recorded again in
465
+ record after record, once per install it survived. **Counting ledger samples
466
+ weights each process by its own longevity, which is the very quantity being
467
+ measured.**
468
+
469
+ `projmux internal install-residue --survival` reads the accumulated ledger and
470
+ answers each cutoff `T` twice: over samples, and over distinct processes. On
471
+ this repository's ledger at 2026-09-08 (54 records with a non-empty census, 866
472
+ residual samples, 128 distinct processes) the two bases disagree by several
473
+ times:
474
+
475
+ | Cutoff | By observation | By process identity |
476
+ | --- | --: | --: |
477
+ | 1h | 72.2% | 60.9% |
478
+ | 4h | 53.3% | 25.8% |
479
+ | 12h | 49.0% | 13.3% |
480
+ | 24h | 40.9% | 8.6% |
481
+ | 72h | 24.6% | 3.9% |
482
+ | 168h | 0.3% | **2.3%** |
483
+
484
+ The disagreement runs in both directions and neither one is a correction of the
485
+ other. Below 72h the sample basis is inflated, because the processes that
486
+ survive many installs contribute many samples each. At 168h it is *deflated*,
487
+ because the handful of week-old processes are diluted across a sample count they
488
+ did not generate. **A drain cutoff chosen from the sample basis would be wrong
489
+ in both directions at once.**
490
+
491
+ Identity is the pair `(role, start instant)`. The ledger has no pid, path, or
492
+ argv to key on, and does not acquire one for this: a start instant is when
493
+ something began, not which process it was. Records written from this change on
494
+ carry the start instants directly. Older records carry only ages, from which a
495
+ start is reconstructed by subtraction; both the instant and the age are whole
496
+ truncated seconds, so a reconstructed start lands on the true second or the one
497
+ after it, and the report says how many of its identities rest on that
498
+ reconstruction.
499
+
500
+ ## Signal key inventory
501
+
502
+ | Key | Layer | Value |
503
+ | --- | --- | --- |
504
+ | `platform.observable` | `L1`, `L2` | `true` / `false` |
505
+ | `image.link` | `L1` | `current` / `unlinked` / `unresolved` |
506
+ | `retained.previous-image` | `L1` | `none` |
507
+ | `processes.observed` | `L2` | counter |
508
+ | `processes.residual` | `L2` | counter |
509
+ | `residual.role.broker-runtime` | `L2` | counter |
510
+ | `residual.role.lifecycle-observer` | `L2` | counter |
511
+ | `residual.role.supervisor` | `L2` | counter |
512
+ | `residual.role.session-client` | `L2` | counter |
513
+ | `residual.role.agent-endpoint` | `L2` | counter |
514
+ | `residual.role.usage-watcher` | `L2` | counter |
515
+ | `residual.role.other` | `L2` | counter |
516
+ | `residual.oldest-seconds` | `L2` | counter |
517
+ | `ledger.records` | `L2` | counter |
518
+ | `ledger.latest.installer` | `L2` | install path token, or `unclassified` |
519
+ | `ledger.latest.observed` | `L2` | counter |
520
+ | `ledger.latest.residual` | `L2` | counter |
521
+ | `replacement.cutoff-seconds` | `L2` | counter |
522
+ | `replacement.beyond-cutoff` | `L2` | counter |
523
+ | `replacement.outcome` | `L2` | install replacement pass outcome token |
524
+ | `replacement.refusal` | `L2` | broker refusal token |
525
+ | `replacement.attempted` | `L2` | counter |
526
+ | `replacement.drained` | `L2` | counter |
527
+ | `replacement.reported` | `L2` | counter |
528
+ | `registry.observed` | `L3` | `true` / `false` |
529
+ | `pool.status` | `L3` | pool status token, or `unobserved` when no pool diagnosis was read |
530
+ | `pool.reason` | `L3` | pool reason token |
531
+ | `pool.action` | `L3` | pool action token |
532
+ | `pool.generations.live` | `L3` | counter |
533
+ | `pool.generations.draining` | `L3` | counter |
534
+ | `qualification.verdict` | `L3` | qualification verdict token |
535
+ | `qualification.reason` | `L3` | qualification reason token |
536
+ | `sessions.running` | `L3` | counter |
537
+ | `sessions.live` | `L3` | counter |
538
+ | `sessions.dead` | `L3` | counter |
539
+ | `sessions.unobservable` | `L3` | counter |
540
+
541
+ ## What this surface does not do
542
+
543
+ It does not replace anything, restore anything, end a process, start a process,
544
+ or repair a Registry. It reports, and a reported mismatch is left exactly as it
545
+ was found. `TestReplacementMeasurementPathEndsNoProcess` holds that as a
546
+ property of the source: the census, the residue ledger, the survival report, and
547
+ this table are checked to carry no process termination, signalling, or restart.
548
+
549
+ The replacement policy above is the one thing on this page that acts, and it
550
+ acts through a strictly narrower door. `TestReplacementPathEndsNoProcess` holds
551
+ the same guard over the policy table and the install pass: they carry no
552
+ termination, no signalling, and no restart either. **The whole of the action is
553
+ a socket handshake, and the runtime decides.** `L1` atomicity and the `L3`
554
+ qualification gate are separate work on separate doors, and no change on this
555
+ path may alter their behavior.
@@ -33,6 +33,14 @@ direct shell as the optional default, and mint identities only for missing
33
33
  items. An Agent-only Window is valid with an empty default. Repeating the same
34
34
  projection is a Registry zero-diff.
35
35
 
36
+ Resource metadata records the Registry schema that produced it. A v3 snapshot
37
+ is projected through the same root-wide same-kind duplicate-group and
38
+ destination-closure rule as Registry migration: affected resources receive
39
+ their exact UID names, unique names outside the closure are preserved, and no
40
+ numeric suffix is minted. A current-v4 snapshot containing a root-wide
41
+ same-kind collision is rejected as damaged before trust authorization,
42
+ Registry/tmux/provider mutation, or any snapshot write.
43
+
36
44
  The committed Registry is then converged by the ordinary Project materializer.
37
45
  For a restored offline Agent-anchor Window, `Continue project` visibly plans a
38
46
  lazy default shell, creates the Window from that shell, and stages the Agent on
@@ -53,17 +61,19 @@ A closed Project has exactly two actions:
53
61
  materializer. A retained graph keeps its Project, Window, Pane, and Agent
54
62
  UIDs. A zero-Window Project keeps its Project UID and atomically receives one
55
63
  new canonical Window and shell UID before materialization.
56
- - `Open fresh` atomically replaces the same-root graph with a new Project UID
57
- and one new canonical Window/shell UID chain. It does not archive or retain
58
- the old generation.
64
+ - `Recreate Project` atomically replaces the same-root graph with a new Project
65
+ UID and one new canonical Window/shell UID chain, after a confirmation naming
66
+ the exact old Project UID and its Window/Pane/Agent counts. Declining returns
67
+ to the startup rows and writes nothing. It does not archive or retain the old
68
+ generation.
59
69
 
60
70
  Esc/cancel returns to Projects; it is not an action row. Picker failure falls
61
71
  back to the non-destructive `Continue project` action.
62
72
 
63
- `Open fresh` never deletes or overwrites autosave or named snapshot files. It
73
+ `Recreate Project` never deletes or overwrites autosave or named snapshot files. It
64
74
  preserves the root, Git/worktrees, trust decision, and all unrelated Registry
65
75
  graphs while changing the Project identity. A rejected commit retains the
66
- exact old Registry preimage. Repeating `Open fresh` replaces identity again;
76
+ exact old Registry preimage. Repeating `Recreate Project` replaces identity again;
67
77
  each successful result has exactly one Project claiming the root.
68
78
 
69
79
  ## Snapshot contents and diagnostics
@@ -69,15 +69,16 @@ step, never a silent no-op.
69
69
  `get runtime` never read it. An unrecognized saved value applies the default
70
70
  without writing and shows an invalid source.
71
71
  `Closed Project startup` shows exactly two actions when no preference file
72
- exists and reports `Continue project / Open fresh - default` (`이어서 열기 /
73
- 새로 열기 - 기본값` in ko-KR). Saved `on` keeps those choices and reports
74
- `Continue project / Open fresh - on - saved`. Saved `off` reports
72
+ exists and reports `Continue project / Recreate Project - default` (`이어서 열기 /
73
+ Project 다시 만들기 - 기본값` in ko-KR). Saved `on` keeps those choices and reports
74
+ `Continue project / Recreate Project - on - saved`. Saved `off` reports
75
75
  `Continue project - off - saved`, skips the picker, and retains the
76
76
  registered-Continue/unregistered-Fresh automatic
77
77
  adjudication. Reading the row or opening/cancelling the picker never writes
78
78
  the default or changes saved preference bytes/mtime.
79
79
  `Continue project` materializes the Project's current Registry desired state
80
- and then moves the client. `Open fresh` confirms exact counts,
80
+ and then moves the client. `Recreate Project` confirms the exact old Project
81
+ UID and per-kind counts before writing anything,
81
82
  atomically replaces the old Project graph with a new Project UID and a new
82
83
  canonical Window/shell UID pair, and leaves exactly one same-root claimant
83
84
  before ordinary materialization. Esc returns to Projects. Neither action
package/docs/testing.md CHANGED
@@ -29,10 +29,12 @@ and humans run the same entrypoints.
29
29
  replays one scenario, `PROJMUX_E2E_LINUX_SHARD=<shard>` runs exactly one
30
30
  Linux fixture with the terminal inventory its `linux-shards.tsv` row owns,
31
31
  and `PROJMUX_E2E_SUITE=codex-lifecycle|npm-staging` runs one non-Linux suite.
32
- Setting none keeps the default: four Linux shards in parallel plus both
33
- suites. CI uses the shard/suite selectors to give every suite its own runner,
34
- so container isolation and schedule isolation are the same unit there; local
35
- `make test-e2e` still runs the whole matrix on one machine.
32
+ Setting none keeps the local acceptance profile: all four Linux shards run
33
+ serially in canonical `fixture-1` through `fixture-4` order on the one host,
34
+ followed by both suites. CI uses the shard/suite selectors to give every
35
+ suite its own runner, so each runner likewise has capacity for one shard.
36
+ `PROJMUX_E2E_LINUX_MODE=parallel make test-e2e` is the explicit same-host
37
+ stress profile; `serial` and `parallel` are the only accepted mode values.
36
38
  - Every scenario wait states a budget in seconds rather than a loop count, and
37
39
  `E2E_WAIT_SCALE=<factor>` multiplies all of them at once. Raise it when a
38
40
  runner is slow or loaded; a wait that expires still fails with the description
@@ -42,9 +44,12 @@ and humans run the same entrypoints.
42
44
  `make test-e2e-shards` validate typed attempt evidence, bounded semantic
43
45
  waits/owned cleanup, and exhaustive four-shard isolation without rerunning
44
46
  the full product matrix. The shard target also pins the CI job list to the
45
- manifest: one non-fail-fast job per shard and per suite, each with its own
46
- runner, its own timeout and its own uniquely named evidence artifacts, all of
47
- them required children of the aggregate `Test` gate. It also pins the thin
47
+ manifest. A deterministic fake-runner start/release barrier proves local
48
+ default overlap is one and explicit parallel overlap is four while both use
49
+ one immutable binary, the exact 21-scenario inventory, and the same result
50
+ hash. The target also pins one non-fail-fast CI job per shard and per suite,
51
+ each with its own runner, timeout, and uniquely named evidence artifacts, all
52
+ of them required children of the aggregate `Test` gate. It also pins the thin
48
53
  `E2E Tests` job, which exists because the branch ruleset requires a status
49
54
  check under that exact name; a required context that is never reported stays
50
55
  pending rather than failing, so dropping that job would deadlock merges. The