cursedops 0.11.0 → 0.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -14,18 +14,18 @@ bun add cursedops
14
14
  | `cursedops/smoke` | the scaffolding of a deployed smoke — the ledger, the fetch, the DNS hint, the exit code — the one check no app owns (every address of a deployment serving the same built client), and since 0.5.0 its VERDICTS: the origin asked on loopback, the smoke's own environment, a network that lies about DNS, and a version that has settled |
15
15
  | `cursedops/worker-deploy` | the sequence that ships a Cloudflare Worker — clean tree, stage first, schema, `--var` stamp, secrets, smoke — and the readers it needs (0.5.0); `readWranglerFile(root)` since 0.9.0, lifted from four apps' `readWrangler` (task 2153); since 0.10.25 every run, red or green, appends a `kind: "deploy"` line with each leg's wall clock (`install`, `stage`, `walk`, `build`, `schema`, `upload`, `secrets`, `smoke`) to the generation's ledger (task 2246) |
16
16
  | `cursedops/worker-deploy` (CPU) | a `wrangler tail` around the traffic a deploy already sends, each request's BILLED CPU charged to its declared route, red on an undeclared, over-budget or uncapped route — and `runWorkerDeploy`'s `stageWalk` + `cpuTail` that wrap the walk and the smoke in it (0.7.0, task 2134, lifted from collections) |
17
- | `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) |
17
+ | `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) — and read again, up to five times, before a set that is not exact is believed: the list lags the puts (0.11.1) |
18
18
  | `cursedops/worker-rollback` | a hostname back on its Mac origin: origin first, route second, the route found rather than typed (0.5.0) |
19
- | `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0) |
19
+ | `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0); a request's own `maxTimeSec` over the instance's (0.11.1) |
20
20
  | `cursedops/d1-import` | a cutover's data proof — SQL literals and the row-for-row comparison, never a count (0.5.0); and `d1SchemaText`, the generated `db/schema.sql` that refuses a statement spanning lines, because D1's exec splits on newlines (0.9.0, lifted from five apps, task 2153) — and its `forge-d1-schema` bin (0.9.1), so no app carries its own schema script at all |
21
- | `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153) |
21
+ | `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153); `BACKUP_STAMP_PATTERN`, `backupStamp` and its inverse `stampTime` — the local-time run name every backup script writes and station's audit reads (0.11.2) |
22
22
  | `cursedops/serve` | the static tier's four helpers — the path-traversal guard, the MIME table, the hashed-asset test, the crash handlers |
23
23
  | `cursedops/api-floor` | the rule that an unmatched `/api/...` is a phrase and never the app shell — the namespace predicates, the trailing-slash normaliser and the default 404 body. No `node:` import, so it mounts inside a Worker |
24
24
  | `cursedops/build-info` | which commit a checkout-served process is running and whether its tree was dirty — read once at load, and a `null` retried in the background rather than cached for the life of the process. `node:child_process`, so never in a Worker |
25
25
  | `cursedops/bound-lists` | the static check that no SQL builds an `IN (…)` list one `?` per value — the shape D1 500s on past 100 ids. Test-time only; the recorded third exception, see below |
26
26
  | `cursedops/staged-client` | the IMMUTABLE client a checkout-served app serves — a build staged into `<APP_DATA_DIR>/client/<commit>/` behind `CURRENT`, so another agent's `vite build` in the checkout never changes live bytes — and its `forge-client stamp` / `forge-client stage` bin (0.6.0, lifted from family, flix, roms, station) |
27
27
  | `cursedops/deploy-tree` | the two refusals a checkout-served deploy needs — never deploy a dirty tree, never `git revert` in one (`readTreeStatus`, `refuseToDeploy`, `refuseToRevert`; 0.6.0, lifted from nine apps) |
28
- | `cursedops/ship` | the owner's FAST SHIPPING switch as every deploy reads it — `readShipSetting` (`$FORGE_STATE/state/autopilot/ship.json`; absent or unreadable is `fast`, no generation is `each`), `deferral`/`shipDeferral` (the sentence a production deploy prints instead of deploying, or `null`), `FORGE_SHIP_NOW=1` to ship now whatever it says. `runWorkerDeploy` asks it before installing anything; a Mac-side `bun run deploy` calls `shipDeferral` first (0.11.0) |
28
+ | `cursedops/ship` | the owner's FAST SHIPPING switch as every deploy reads it — `readShipSetting` (`$FORGE_STATE/state/autopilot/ship.json`; absent or unreadable is `fast`, no generation is `each`), `deferral`/`shipDeferral` (the sentence a production deploy prints instead of deploying, or `null`), `FORGE_SHIP_NOW=1` to ship now whatever it says. `runWorkerDeploy` asks it before installing anything; a Mac-side `bun run deploy` calls `shipDeferral` first (0.11.0). Since 0.12.0 `shipNowVerdict` honours ship-now under `$AUTOPILOT_BATCH` only for a batch holding a live claim on the ship task or a `prod-down:` incident, with `FORGE_SHIP_NOW_WHY` set, and every ask is a `kind: "ship-now"` ledger row (task 2384) |
29
29
  | `cursedops/relay` | the WORKER half of a relay — a Worker in front of a Mac-bound app over ONE outbound WebSocket: the frames (station's wire, byte for byte), the key-digest door (`acceptLink`), and `RelayLink`, the Durable Object an app subclasses with its name, body ceiling, timeout and (optionally) station's per-cookie offline cache. No `node:` import (0.10.0, lifted from station for roms, task 065) |
30
30
  | `cursedops/relay-link` | the MAC half — `dialLink` (answer each frame with the app's own `fetch`, redial 1 s → 60 s on close, no timer otherwise), `readLinkSettings` for `<APP>_LINK_URL`/`<APP>_LINK_KEY`, and `rotateLinkKey` (digest to the Worker FIRST, the key to the 0600 file only if it took) (0.10.0) |
31
31
  | `cursedops/relay-stage` | the signed-in STAGE WALK's skeleton both relay apps shared — `stageByteCheck` (SHA-256 of every built file, a stale byte re-asked 4 × 10 s), `putStageLinkDigest` (`--env stage` only), `grantStageOwner` (auth-stage's 2FA grant, never production's), `oldKeyRefused` (the rotation leg: 20 s refused window, 3 min ceiling), `awaitSystemResolver`, `servedPath`, `STAGE_AUTH_URL`. With `relayWorkerDeployCli` in `cursedops/worker-deploy`, a relay app's whole `worker:deploy` as data (0.10.6) |
@@ -304,6 +304,12 @@ pins an address** — the deliverable is the distinction, not a way around the n
304
304
  `steadyHealth(ask, ready)` is the Worker smoke's wait: `ready` must hold for THREE reads in a
305
305
  row, because on `music`'s first deploy (2026-09-23, `a54ef71`) one good `/healthz` was followed
306
306
  by thirteen routes answering the pre-secrets version. It returns what answered and never judges.
307
+ A read that THROWS (0.11.1) is a bad read asked again, bounded by `deadlineMs` (240 s), and only
308
+ a LAST read that threw is thrown to the caller: the 2026-09-29 ship reported family and music red
309
+ on a first `/healthz` that timed out twice while the edge was between versions (task 2291).
310
+ `runWorkerSmoke` asks each of those reads with a 10 s curl `--max-time`
311
+ (`HEALTH_READ_MAX_TIME_SEC`, passed as `EdgeRequestInit.maxTimeSec`), so the window costs 20 s a
312
+ read instead of 120.
307
313
 
308
314
  `awaitConverged(probes)` (0.10.19) is the same wait over SEVERAL paths: every probe must agree
309
315
  three rounds in a row, bounded (120 s), before any leg is judged — a Worker's code and its static
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "cursedops",
3
- "version": "0.11.0",
3
+ "version": "0.12.0",
4
4
  "description": "The build-and-ops answers this generation's apps wrote independently and identically: finding a generation's roots — and printing a command that runs when pasted — without knowing a path, the generation's whole-tree laws run over one repo from a checkout or a worktree, macOS launchd agent install/replace/remove and the live port a job serves, the scaffolding and verdicts of a deployed smoke (origin probe, the smoke's own environment, a network that lies about DNS, a settled version), the static-serving helpers eight apps copied — the path-traversal guard among them — the API floor that keeps an unmatched /api/... from ever being answered with the app shell, the commit and dirty flag a checkout-served process reports, the Cloudflare Worker deploy toolkit four apps copied (the deploy sequence, exact-set secrets over a pipe, origin-first rollback, the curl edge fetch, the row-for-row D1 import proof, the billed-CPU tail check around a deploy's walk and smoke, and each app's worker:secrets and worker:smoke main as one function of its data), the relay a Worker fronts a Mac-bound app with (the Durable Object, the frames, the Mac's dialer and key rotation — lifted from station for roms — and the signed-in stage walk's skeleton and the relay app's whole worker:deploy), the Worker import-graph and await-port checks every Worker app's suite runs over its own source, and the public-surface ratchet three published libraries each carried a forked copy of. Mechanism only — no app knows its name from here. Bun, zero runtime dependencies (typescript is an optional peer, for public-surface only), ships source.",
5
5
  "type": "module",
6
6
  "scripts": {
package/src/backups.ts CHANGED
@@ -7,3 +7,28 @@
7
7
  * whatever was decided. So it is one number here, and each app's prune takes it.
8
8
  */
9
9
  export const BACKUP_KEEP = 14;
10
+
11
+ /**
12
+ * The name of one dated run: `YYYYMMDD-HHMMSS`, in the Mac's LOCAL time.
13
+ *
14
+ * 🔴 A contract with two sides, so it is spelled once. Every app's own backup script WRITES
15
+ * these directory names and prunes only what matches the pattern; station's backup audit READS
16
+ * them back into a time. Until cursedops 0.11.2 the writers held nine copies of the pattern and
17
+ * two of the stamp, and the reader held its own pair that wrote and read the stamp as UTC — so a
18
+ * run the apps named `20260929-100000` at 10:00 CDT was read as 10:00Z, five hours off.
19
+ */
20
+ export const BACKUP_STAMP_PATTERN = /^\d{8}-\d{6}$/;
21
+
22
+ const pad = (n: number): string => String(n).padStart(2, "0");
23
+
24
+ /** A `Date` → its run name, in local time. {@link stampTime} is its inverse. */
25
+ export const backupStamp = (d: Date): string =>
26
+ `${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}`;
27
+
28
+ /** A run name → epoch ms, read as local time — or `null` for anything that is not one. */
29
+ export function stampTime(stamp: string): number | null {
30
+ if (!BACKUP_STAMP_PATTERN.test(stamp)) return null;
31
+ const n = (from: number, to: number): number => Number(stamp.slice(from, to));
32
+ const at = new Date(n(0, 4), n(4, 6) - 1, n(6, 8), n(9, 11), n(11, 13), n(13, 15));
33
+ return Number.isNaN(at.getTime()) ? null : at.getTime();
34
+ }
package/src/edgeFetch.ts CHANGED
@@ -58,6 +58,12 @@ export interface EdgeFetchOptions {
58
58
  retryDelayMs?: (retry: number) => number;
59
59
  }
60
60
 
61
+ /**
62
+ * A `RequestInit` this fetch reads one more field from: `maxTimeSec`, curl's `--max-time` for
63
+ * THIS request, over the instance's {@link EdgeFetchOptions.maxTimeSec}. A plain `fetch` ignores it.
64
+ */
65
+ export type EdgeRequestInit = RequestInit & { maxTimeSec?: number };
66
+
61
67
  /**
62
68
  * 🔴 How many MORE times a request is sent after a connection-level curl failure that carried
63
69
  * no answer. Measured 2026-09-24: vault's prod deploy of `953e19a` reported "smoke FAILED" on one
@@ -125,7 +131,7 @@ export function parseCurlResponse(raw: Buffer, origin = "the edge", exit: number
125
131
  }
126
132
 
127
133
  /** A `fetch`-shaped function over curl — see the header. Each host is resolved once per instance — once it has an answer. */
128
- export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?: RequestInit) => Promise<Response> {
134
+ export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?: EdgeRequestInit) => Promise<Response> {
129
135
  const pinned = new Map<string, string>();
130
136
  const resolve = options.resolve === undefined ? digCloudflare : options.resolve;
131
137
  const curl =
@@ -155,7 +161,7 @@ export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, i
155
161
  const port = target.port || (target.protocol === "http:" ? "80" : "443");
156
162
  const head = () => {
157
163
  const ip = pinFor(target.hostname);
158
- return ["-s", "-i", "--max-time", String(options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
164
+ return ["-s", "-i", "--max-time", String(init.maxTimeSec ?? options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
159
165
  };
160
166
  const args: string[] = [];
161
167
  for (const [k, v] of Object.entries(options.headersFor?.(target.hostname) ?? {})) args.push("-H", `${k}: ${v}`);
package/src/ship.ts CHANGED
@@ -38,10 +38,17 @@
38
38
  * deploys, an outage fix, and a deploy the owner asked for in words (his sentence beats a switch he
39
39
  * flipped yesterday — the precedence the retired `fastMode.ts` settled on 2026-08-13). A stage
40
40
  * deploy is never deferred: it IS the fast way to see a change on a real edge.
41
+ *
42
+ * 🔴 **Under a batch (`$AUTOPILOT_BATCH` set) it is honoured only for the ship task or an incident**
43
+ * ({@link shipNowVerdict}), and only with a reason in `FORGE_SHIP_NOW_WHY`. Until task 2384 any agent
44
+ * could prefix one variable and turn the owner's switch off for itself — the thing
45
+ * `~/.claude/CLAUDE.md` §2 says an agent may never do — and no record said it had. Now every ask,
46
+ * honoured or refused, is a `kind: "ship-now"` ledger row, and `autopilot doctor` lists the ones
47
+ * with no reason. In the owner's own shell (no batch) it is honoured exactly as before, and recorded.
41
48
  */
42
- import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
49
+ import { appendFileSync, existsSync, mkdirSync, readdirSync, readFileSync } from "node:fs";
43
50
  import { dirname, join } from "node:path";
44
- import { forgeState } from "cursedops/roots";
51
+ import { forgeCodeRoot, forgeState } from "cursedops/roots";
45
52
 
46
53
  export const SHIP_MODES = ["fast", "each"] as const;
47
54
  export type ShipMode = (typeof SHIP_MODES)[number];
@@ -85,12 +92,152 @@ export function readShipSetting(state: string | null): ShipSetting {
85
92
  }
86
93
  }
87
94
 
88
- /** Is `FORGE_SHIP_NOW` set to something that means yes? */
95
+ /** Is `FORGE_SHIP_NOW` set to something that means yes? Asked is not honoured — see {@link shipNowVerdict}. */
89
96
  export function shipNowAsked(env: NodeJS.ProcessEnv = process.env): boolean {
90
97
  const v = env[SHIP_NOW_VAR]?.trim().toLowerCase();
91
98
  return v === "1" || v === "true" || v === "yes";
92
99
  }
93
100
 
101
+ /** Why ship-now was asked — required under a batch, recorded always. */
102
+ export const SHIP_NOW_WHY_VAR = "FORGE_SHIP_NOW_WHY";
103
+
104
+ /** The variable the runner sets on its batch agent — present means an agent, not the owner. */
105
+ export const BATCH_VAR = "AUTOPILOT_BATCH";
106
+
107
+ /**
108
+ * The `escalation-key:` values a claimed task may carry for its batch to ship now: autopilot's
109
+ * `SHIP_KEY` (`ship.ts`) and the prod watch's `prod-down:<host>` (`prod.ts`). The file shape is
110
+ * the interface, exactly as `ship.json` is — this module never imports the runner.
111
+ */
112
+ export const SHIP_NOW_KEYS: readonly RegExp[] = [/^ship$/, /^prod-down:\S+$/];
113
+
114
+ export interface ShipNowVerdict {
115
+ /** `FORGE_SHIP_NOW` means yes. */
116
+ asked: boolean;
117
+ /** Ship now: asked, and (outside a batch, or on a claim that earns it). */
118
+ honoured: boolean;
119
+ /** One sentence: what made it honoured or refused. */
120
+ why: string;
121
+ /** `FORGE_SHIP_NOW_WHY`, trimmed — `""` when not given. */
122
+ reason: string;
123
+ batch?: string;
124
+ /** The claimed task file that earned it. */
125
+ task?: string;
126
+ }
127
+
128
+ interface BoardClaim {
129
+ kind?: unknown;
130
+ pid?: unknown;
131
+ task?: unknown;
132
+ batch?: unknown;
133
+ expiresAt?: unknown;
134
+ }
135
+
136
+ function pidAlive(pid: number): boolean {
137
+ try {
138
+ process.kill(pid, 0);
139
+ return true;
140
+ } catch (err) {
141
+ return (err as NodeJS.ErrnoException).code === "EPERM";
142
+ }
143
+ }
144
+
145
+ /** The `escalation-key:` in a task file's front matter, lower-cased, or `null`. */
146
+ export function escalationKey(text: string): string | null {
147
+ const head = /^---\n([\s\S]*?)\n---/.exec(text.replace(/\r\n/g, "\n"));
148
+ const m = head ? /^escalation-key:\s*(.+?)\s*$/im.exec(head[1] ?? "") : null;
149
+ return m?.[1] ? m[1].toLowerCase() : null;
150
+ }
151
+
152
+ export interface ShipNowWhere {
153
+ /** `$FORGE_STATE` — the board lives at `state/board/`. */
154
+ state: string | null;
155
+ /** `$FORGE` — claimed task files live at `tasks/In_Progress/`. */
156
+ forge: string | null;
157
+ now?: number;
158
+ isAlive?: (pid: number) => boolean;
159
+ }
160
+
161
+ /**
162
+ * Is `FORGE_SHIP_NOW` honoured here? Outside a batch, whenever it is asked: that is the owner's own
163
+ * shell. Under `$AUTOPILOT_BATCH`, only when that batch holds a live `kind: "task"` board claim
164
+ * (`$FORGE_STATE/state/board/*.json`, the pid alive and not expired) on a task in
165
+ * `$FORGE/tasks/In_Progress/` whose `escalation-key:` is one of {@link SHIP_NOW_KEYS}, AND
166
+ * `FORGE_SHIP_NOW_WHY` says why. Anything it cannot read is a refusal — a refused ship-now only
167
+ * makes the deploy later, which is the safe direction.
168
+ */
169
+ export function shipNowVerdict(where: ShipNowWhere, env: NodeJS.ProcessEnv = process.env): ShipNowVerdict {
170
+ const asked = shipNowAsked(env);
171
+ const reason = env[SHIP_NOW_WHY_VAR]?.trim() ?? "";
172
+ const batch = env[BATCH_VAR]?.trim() || undefined;
173
+ const base = { asked, reason, ...(batch ? { batch } : {}) };
174
+ if (!asked) return { ...base, honoured: false, why: `${SHIP_NOW_VAR} is not set` };
175
+ if (!batch) return { ...base, honoured: true, why: `no ${BATCH_VAR} — the owner's own shell` };
176
+ const refuse = (why: string): ShipNowVerdict => ({ ...base, honoured: false, why });
177
+ if (!reason) return refuse(`under batch ${batch}, ${SHIP_NOW_VAR} needs ${SHIP_NOW_WHY_VAR}="<why this cannot wait for the ship>"`);
178
+ if (!where.state || !where.forge) return refuse(`under batch ${batch}, no generation to read the board from`);
179
+ const board = join(where.state, "state", "board");
180
+ const now = where.now ?? Date.now();
181
+ const isAlive = where.isAlive ?? pidAlive;
182
+ let files: string[] = [];
183
+ try {
184
+ files = readdirSync(board).filter((f) => f.startsWith("task-") && f.endsWith(".json"));
185
+ } catch {
186
+ /* no board → no claim */
187
+ }
188
+ const held: string[] = [];
189
+ for (const f of files) {
190
+ let c: BoardClaim;
191
+ try {
192
+ c = JSON.parse(readFileSync(join(board, f), "utf8")) as BoardClaim;
193
+ } catch {
194
+ continue;
195
+ }
196
+ if (c.kind !== "task" || c.batch !== batch || typeof c.task !== "string" || typeof c.pid !== "number") continue;
197
+ if (typeof c.expiresAt === "string" && Date.parse(c.expiresAt) < now) continue;
198
+ if (!isAlive(c.pid)) continue;
199
+ held.push(c.task);
200
+ let key: string | null = null;
201
+ try {
202
+ key = escalationKey(readFileSync(join(where.forge, "tasks", "In_Progress", c.task), "utf8"));
203
+ } catch {
204
+ continue;
205
+ }
206
+ if (key && SHIP_NOW_KEYS.some((k) => k.test(key))) return { ...base, honoured: true, why: `batch ${batch} holds ${c.task} (escalation-key: ${key})`, task: c.task };
207
+ }
208
+ return refuse(
209
+ `batch ${batch} holds ${held.length ? held.join(", ") : "no live task claim"} — ${SHIP_NOW_VAR} is honoured under a batch only for the ship task or a prod incident (escalation-key: ship | prod-down:<host>)`,
210
+ );
211
+ }
212
+
213
+ /** The ledger line every ship-now ask leaves — honoured or refused. */
214
+ export interface ShipNowEventLine {
215
+ at: string;
216
+ kind: "ship-now";
217
+ /** The deploy or publish it was asked for. */
218
+ what: string;
219
+ ok: boolean;
220
+ reason: string;
221
+ detail: string;
222
+ batch?: string;
223
+ task?: string;
224
+ }
225
+
226
+ /** Record one ship-now ask in the ledger. Never throws. */
227
+ export function recordShipNow(state: string | null, what: string, v: ShipNowVerdict, at: Date = new Date()): boolean {
228
+ const line: ShipNowEventLine = {
229
+ at: at.toISOString(),
230
+ kind: "ship-now",
231
+ what,
232
+ ok: v.honoured,
233
+ reason: v.reason,
234
+ detail: v.why,
235
+ ...(v.batch ? { batch: v.batch } : {}),
236
+ ...(v.task ? { task: v.task } : {}),
237
+ };
238
+ return appendLedger(state, line);
239
+ }
240
+
94
241
  /**
95
242
  * The sentence a production deploy prints INSTEAD of deploying, or `null` to deploy now.
96
243
  *
@@ -98,16 +245,21 @@ export function shipNowAsked(env: NodeJS.ProcessEnv = process.env): boolean {
98
245
  * and the runner's ship task — filed when the queue empties or the oldest owed commit is too old —
99
246
  * deploys every app that is behind, once. `autopilot ship` lists what is owed.
100
247
  */
101
- export function deferral(what: string, setting: ShipSetting, env: NodeJS.ProcessEnv = process.env): string | null {
102
- if (setting.mode !== "fast" || shipNowAsked(env)) return null;
248
+ export function deferral(what: string, setting: ShipSetting, env: NodeJS.ProcessEnv = process.env, where: ShipNowWhere = { state: null, forge: null }): string | null {
249
+ if (setting.mode !== "fast") return null;
250
+ const now = shipNowVerdict(where, env);
251
+ if (now.asked) recordShipNow(where.state, what, now);
252
+ if (now.honoured) return null;
253
+ const refused = now.asked ? [`🔴 ${SHIP_NOW_VAR} REFUSED: ${now.why}.`] : [];
103
254
  const by = setting.setBy === "default" ? "the default" : `set by ${setting.setBy}${setting.at ? ` at ${setting.at}` : ""}`;
104
255
  return [
256
+ ...refused,
105
257
  `🚢 ${what}: DEFERRED — fast shipping is on (${by}${setting.unreadable ? `; ship.json unreadable: ${setting.unreadable}` : ""}).`,
106
258
  " Nothing was deployed, and that is the intended result, not a failure: this app is now owed to",
107
259
  " the ship, which deploys every app behind its HEAD once, when the queue empties. Carry on with",
108
260
  " the next task — file this one as done once it is landed and gated, saying the deploy is owed.",
109
- ` \`autopilot ship\` lists what is owed. To deploy now anyway (an outage fix, the ship itself, or`,
110
- ` the owner asked for it): ${SHIP_NOW_VAR}=1 <the same command>.`,
261
+ ` \`autopilot ship\` lists what is owed. To deploy now anyway (the ship task or a prod incident`,
262
+ ` your batch holds, or the owner in his own shell): ${SHIP_NOW_VAR}=1 ${SHIP_NOW_WHY_VAR}="<why>" <the same command>.`,
111
263
  ].join("\n");
112
264
  }
113
265
 
@@ -118,7 +270,8 @@ export function deferral(what: string, setting: ShipSetting, env: NodeJS.Process
118
270
  * if (deferred) { console.log(deferred); process.exit(0); }
119
271
  */
120
272
  export function shipDeferral(cwd: string, what: string, env: NodeJS.ProcessEnv = process.env): string | null {
121
- return deferral(what, readShipSetting(forgeState(cwd, env)), env);
273
+ const state = forgeState(cwd, env);
274
+ return deferral(what, readShipSetting(state), env, { state, forge: forgeCodeRoot(cwd, env) });
122
275
  }
123
276
 
124
277
  /** The ledger line a deferred production deploy leaves — what `autopilot ship` counts. */
package/src/smoke.ts CHANGED
@@ -83,6 +83,8 @@
83
83
  import { spawnSync } from "node:child_process";
84
84
  import { lookup } from "node:dns/promises";
85
85
  import { isIP } from "node:net";
86
+ // The package's own subpath, never `./edgeFetch.ts` — see `publishShape.test.ts`.
87
+ import type { EdgeRequestInit } from "cursedops/edge-fetch";
86
88
 
87
89
  export interface Failure {
88
90
  check: string;
@@ -840,12 +842,18 @@ export interface SteadyOptions {
840
842
  reads?: number;
841
843
  /** Reads before giving up and judging whatever answered. Default 40. */
842
844
  tries?: number;
845
+ /**
846
+ * Stop asking after this long, whatever `tries` says. Default 240 s — a read that THROWS can
847
+ * cost a whole curl `--max-time`, so a count alone does not bound the wait.
848
+ */
849
+ deadlineMs?: number;
843
850
  /** Pause after a GOOD read, before the next. Default 2 s. */
844
851
  pauseMs?: number;
845
852
  /** Pause after a BAD read. Default 5 s. */
846
853
  retryMs?: number;
847
854
  /** Injected for tests. */
848
855
  sleep?: (ms: number) => Promise<void>;
856
+ now?: () => number;
849
857
  }
850
858
 
851
859
  export interface SteadyRead<T> {
@@ -868,6 +876,14 @@ export interface SteadyRead<T> {
868
876
  * answering `503` (the pre-secrets version) and `404`, because one isolate had the new version
869
877
  * and the next did not. The copies in `vault` and `collections` still judged after one read.
870
878
  *
879
+ * 🔴 **A read that THROWS is a bad read, not the end of the wait.** Measured 2026-09-29 (task
880
+ * 2291): family's and music's production smokes each went red on their FIRST `/healthz`, straight
881
+ * after the secrets were put — `no answer … (curl exit 28)`, twice 60 s — and both hosts answered
882
+ * in 0.2 s moments later. The edge was between versions; rethrowing that first silence charged a
883
+ * green deploy with a red smoke. A throw now resets the count and is asked again until `tries`
884
+ * or `deadlineMs`; only when the LAST read threw is that error thrown to the caller, so a Worker
885
+ * that never answers is still a red that names why.
886
+ *
871
887
  * It never decides pass or fail: it returns what answered, and the caller judges that — a wait
872
888
  * that swallowed its own timeout would be a check that cannot go red.
873
889
  */
@@ -878,22 +894,44 @@ export async function steadyHealth<T = Record<string, unknown>>(
878
894
  ): Promise<SteadyRead<T>> {
879
895
  const reads = options.reads ?? 3;
880
896
  const tries = options.tries ?? 40;
897
+ const deadlineMs = options.deadlineMs ?? 240_000;
881
898
  const sleep = options.sleep ?? ((ms: number) => new Promise<void>((done) => setTimeout(done, ms)));
899
+ const now = options.now ?? Date.now;
900
+ const started = now();
882
901
  let steady = 0;
883
902
  let asked = 0;
884
903
  for (;;) {
885
- const response = await ask();
904
+ let read: { response: Response; body: T } | null = null;
905
+ let thrown: unknown = null;
906
+ try {
907
+ const response = await ask();
908
+ const body = (await response
909
+ .clone()
910
+ .json()
911
+ .catch(() => ({}))) as T;
912
+ read = { response, body };
913
+ } catch (error) {
914
+ thrown = error;
915
+ }
886
916
  asked++;
887
- const body = (await response
888
- .clone()
889
- .json()
890
- .catch(() => ({}))) as T;
891
- steady = ready(body, response) ? steady + 1 : 0;
892
- if (steady >= reads || asked >= tries) return { response, body, steady, asked };
917
+ steady = read && ready(read.body, read.response) ? steady + 1 : 0;
918
+ const out = asked >= tries || now() - started >= deadlineMs;
919
+ if (read && (steady >= reads || out)) return { ...read, steady, asked };
920
+ if (!read && out) throw thrown;
893
921
  await sleep(steady > 0 ? (options.pauseMs ?? 2_000) : (options.retryMs ?? 5_000));
894
922
  }
895
923
  }
896
924
 
925
+ /**
926
+ * curl's `--max-time` for ONE `/healthz` read of a settling Worker, in seconds. Short on purpose:
927
+ * {@link steadyHealth} asks again after a silence, so a long timeout only makes the edge's
928
+ * between-versions window cost more — the 2026-09-29 reds spent 120 s on two reads of it.
929
+ */
930
+ export const HEALTH_READ_MAX_TIME_SEC = 10;
931
+
932
+ /** The init a smoke's `/healthz` read carries — `cursedops/edge-fetch` honours it; a plain `fetch` ignores it. */
933
+ const healthRead: EdgeRequestInit = { maxTimeSec: HEALTH_READ_MAX_TIME_SEC };
934
+
897
935
  /** One question a deploy must answer the NEW way — `null` when it does, else why not. */
898
936
  export interface ConvergenceProbe {
899
937
  name: string;
@@ -1065,7 +1103,7 @@ export async function runWorkerSmoke(spec: WorkerSmokeSpec): Promise<number> {
1065
1103
  let status: number;
1066
1104
  try {
1067
1105
  const read = await steadyHealth<Record<string, unknown>>(
1068
- () => ask("/healthz"),
1106
+ () => ask("/healthz", healthRead),
1069
1107
  (body) => body.commit === spec.head && body.apiServed === true,
1070
1108
  spec.steady,
1071
1109
  );
@@ -1155,7 +1193,7 @@ async function accessDoorSmoke(
1155
1193
  let health: Record<string, unknown>;
1156
1194
  let status: number;
1157
1195
  try {
1158
- const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz"), (body) => body.commit === spec.head, spec.steady);
1196
+ const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz", healthRead), (body) => body.commit === spec.head, spec.steady);
1159
1197
  health = read.body;
1160
1198
  status = read.response.status;
1161
1199
  } catch (error) {
@@ -55,8 +55,8 @@
55
55
  import { spawn, spawnSync } from "node:child_process";
56
56
  import { existsSync, readdirSync, readFileSync } from "node:fs";
57
57
  import { join } from "node:path";
58
- import { forgeState } from "cursedops/roots";
59
- import { appendLedger, deferral, readShipSetting, type ShipEventLine, type ShipSetting } from "cursedops/ship";
58
+ import { forgeCodeRoot, forgeState } from "cursedops/roots";
59
+ import { appendLedger, deferral, readShipSetting, type ShipEventLine, type ShipNowWhere, type ShipSetting } from "cursedops/ship";
60
60
 
61
61
  /** The two deployments every Worker app here has. */
62
62
  export type WorkerEnv = "production" | "stage";
@@ -327,6 +327,12 @@ export interface DeployDeps {
327
327
  ship?: () => ShipSetting;
328
328
  /** Where `FORGE_SHIP_NOW` is read from. Default `process.env`. */
329
329
  env?: NodeJS.ProcessEnv;
330
+ /**
331
+ * The generation roots `FORGE_SHIP_NOW` is judged against under a batch — the board, the claimed
332
+ * task, and the ledger its `ship-now` row goes to (`shipNowVerdict` in `./ship.ts`). Absent, a batch's
333
+ * ship-now is refused and nothing is recorded.
334
+ */
335
+ shipWhere?: ShipNowWhere;
330
336
  /** Where a deferred production deploy's `ship` line goes. {@link workerDeployDeps} appends it to the ledger. */
331
337
  recordShip?: (line: ShipEventLine) => void;
332
338
  }
@@ -408,6 +414,7 @@ export function workerDeployDeps(cwd: string): DeployDeps {
408
414
  recordDeployEvent(forgeState(cwd), line);
409
415
  },
410
416
  ship: () => readShipSetting(forgeState(cwd)),
417
+ shipWhere: { state: forgeState(cwd), forge: forgeCodeRoot(cwd) },
411
418
  recordShip: (line) => {
412
419
  appendLedger(forgeState(cwd), line);
413
420
  },
@@ -699,7 +706,7 @@ export function runWorkerDeploy(spec: WorkerDeploySpec, deps: DeployDeps): Deplo
699
706
  // runner's ship instead — asked before anything installs or builds, so deferring costs nothing.
700
707
  // Its `ship` line, not a `deploy` one: a deferral is not a deploy, and would drag every median.
701
708
  if (spec.env === "production" && deps.ship) {
702
- const deferred = deferral(`${spec.app} production deploy`, deps.ship(), deps.env ?? process.env);
709
+ const deferred = deferral(`${spec.app} production deploy`, deps.ship(), deps.env ?? process.env, deps.shipWhere);
703
710
  if (deferred) {
704
711
  log(`\n${deferred}`);
705
712
  try {
@@ -92,6 +92,12 @@ export interface SecretSync {
92
92
  /** Tries per put. A put is idempotent, so a retry can only turn a network blip into a success. */
93
93
  export const SECRET_PUT_ATTEMPTS = 3;
94
94
 
95
+ /**
96
+ * Reads of the account after the puts before a set that is not exact is believed. A read changes
97
+ * nothing, so asking again can only replace a stale answer with a current one — see {@link syncSecrets}.
98
+ */
99
+ export const SECRET_READBACK_ATTEMPTS = 5;
100
+
95
101
  /**
96
102
  * Apply `plan`: put every value, delete every extra, then READ THE ACCOUNT BACK and fail unless it
97
103
  * holds exactly the set. Refuses without touching anything when the plan has `absent` keys.
@@ -100,11 +106,17 @@ export const SECRET_PUT_ATTEMPTS = 3;
100
106
  * `wrangler secret put` of three hit `fetch failed` and the whole deploy stopped with
101
107
  * collections-stage holding a Worker and no secrets — serving nothing — for a blip the next
102
108
  * attempt would have cleared. The read-back is still the verdict.
109
+ *
110
+ * 🔴 **The read-back is asked again, up to {@link SECRET_READBACK_ATTEMPTS} times, before a set
111
+ * that is not exact is believed.** Measured 2026-09-29 on family-stage (task 2291): the account
112
+ * listed both secrets, both puts said ok, and the list read straight after answered NOTHING — a
113
+ * green deploy stopped as "secrets NOT attached". The same script run again a minute later read
114
+ * both. The LAST read is the verdict, so a set that is really wrong is still a red.
103
115
  */
104
116
  export async function syncSecrets(
105
117
  plan: SecretPlan,
106
118
  ops: SecretOps,
107
- options: { attempts?: number; pause?: (ms: number) => Promise<void> } = {},
119
+ options: { attempts?: number; readbacks?: number; pause?: (ms: number) => Promise<void> } = {},
108
120
  ): Promise<SecretSync> {
109
121
  const attempts = Math.max(1, options.attempts ?? SECRET_PUT_ATTEMPTS);
110
122
  const pause = options.pause ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)));
@@ -130,9 +142,20 @@ export async function syncSecrets(
130
142
  lines.push(`DELETE ${name} — not in the set — ${ok ? "gone" : "FAILED"}`);
131
143
  if (!ok) bad++;
132
144
  }
133
- const after = await ops.held();
134
- const exact = after.length === plan.wanted.length && plan.wanted.every((name) => after.includes(name));
135
- lines.push(exact ? `holds exactly: ${after.join(", ")}` : `🔴 holds ${after.join(", ") || "(nothing)"} — not exactly ${plan.wanted.join(", ")}`);
145
+ const readbacks = Math.max(1, options.readbacks ?? SECRET_READBACK_ATTEMPTS);
146
+ const isExact = (held: readonly string[]) => held.length === plan.wanted.length && plan.wanted.every((name) => held.includes(name));
147
+ let after = await ops.held();
148
+ let reads = 1;
149
+ while (!isExact(after) && reads < readbacks) {
150
+ await pause(2_000 * reads);
151
+ after = await ops.held();
152
+ reads++;
153
+ }
154
+ const exact = isExact(after);
155
+ const reread = reads > 1 ? ` (read ${reads} times)` : "";
156
+ lines.push(
157
+ exact ? `holds exactly: ${after.join(", ")}${reread}` : `🔴 holds ${after.join(", ") || "(nothing)"} — not exactly ${plan.wanted.join(", ")}${reread}`,
158
+ );
136
159
  return { ok: exact && bad === 0, lines, after };
137
160
  }
138
161