cursedops 0.10.26 → 0.11.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -14,17 +14,18 @@ bun add cursedops
14
14
  | `cursedops/smoke` | the scaffolding of a deployed smoke — the ledger, the fetch, the DNS hint, the exit code — the one check no app owns (every address of a deployment serving the same built client), and since 0.5.0 its VERDICTS: the origin asked on loopback, the smoke's own environment, a network that lies about DNS, and a version that has settled |
15
15
  | `cursedops/worker-deploy` | the sequence that ships a Cloudflare Worker — clean tree, stage first, schema, `--var` stamp, secrets, smoke — and the readers it needs (0.5.0); `readWranglerFile(root)` since 0.9.0, lifted from four apps' `readWrangler` (task 2153); since 0.10.25 every run, red or green, appends a `kind: "deploy"` line with each leg's wall clock (`install`, `stage`, `walk`, `build`, `schema`, `upload`, `secrets`, `smoke`) to the generation's ledger (task 2246) |
16
16
  | `cursedops/worker-deploy` (CPU) | a `wrangler tail` around the traffic a deploy already sends, each request's BILLED CPU charged to its declared route, red on an undeclared, over-budget or uncapped route — and `runWorkerDeploy`'s `stageWalk` + `cpuTail` that wrap the walk and the smoke in it (0.7.0, task 2134, lifted from collections) |
17
- | `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) |
17
+ | `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) — and read again, up to five times, before a set that is not exact is believed: the list lags the puts (0.11.1) |
18
18
  | `cursedops/worker-rollback` | a hostname back on its Mac origin: origin first, route second, the route found rather than typed (0.5.0) |
19
- | `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0) |
19
+ | `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0); a request's own `maxTimeSec` over the instance's (0.11.1) |
20
20
  | `cursedops/d1-import` | a cutover's data proof — SQL literals and the row-for-row comparison, never a count (0.5.0); and `d1SchemaText`, the generated `db/schema.sql` that refuses a statement spanning lines, because D1's exec splits on newlines (0.9.0, lifted from five apps, task 2153) — and its `forge-d1-schema` bin (0.9.1), so no app carries its own schema script at all |
21
- | `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153) |
21
+ | `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153); `BACKUP_STAMP_PATTERN`, `backupStamp` and its inverse `stampTime` — the local-time run name every backup script writes and station's audit reads (0.11.2) |
22
22
  | `cursedops/serve` | the static tier's four helpers — the path-traversal guard, the MIME table, the hashed-asset test, the crash handlers |
23
23
  | `cursedops/api-floor` | the rule that an unmatched `/api/...` is a phrase and never the app shell — the namespace predicates, the trailing-slash normaliser and the default 404 body. No `node:` import, so it mounts inside a Worker |
24
24
  | `cursedops/build-info` | which commit a checkout-served process is running and whether its tree was dirty — read once at load, and a `null` retried in the background rather than cached for the life of the process. `node:child_process`, so never in a Worker |
25
25
  | `cursedops/bound-lists` | the static check that no SQL builds an `IN (…)` list one `?` per value — the shape D1 500s on past 100 ids. Test-time only; the recorded third exception, see below |
26
26
  | `cursedops/staged-client` | the IMMUTABLE client a checkout-served app serves — a build staged into `<APP_DATA_DIR>/client/<commit>/` behind `CURRENT`, so another agent's `vite build` in the checkout never changes live bytes — and its `forge-client stamp` / `forge-client stage` bin (0.6.0, lifted from family, flix, roms, station) |
27
27
  | `cursedops/deploy-tree` | the two refusals a checkout-served deploy needs — never deploy a dirty tree, never `git revert` in one (`readTreeStatus`, `refuseToDeploy`, `refuseToRevert`; 0.6.0, lifted from nine apps) |
28
+ | `cursedops/ship` | the owner's FAST SHIPPING switch as every deploy reads it — `readShipSetting` (`$FORGE_STATE/state/autopilot/ship.json`; absent or unreadable is `fast`, no generation is `each`), `deferral`/`shipDeferral` (the sentence a production deploy prints instead of deploying, or `null`), `FORGE_SHIP_NOW=1` to ship now whatever it says. `runWorkerDeploy` asks it before installing anything; a Mac-side `bun run deploy` calls `shipDeferral` first (0.11.0) |
28
29
  | `cursedops/relay` | the WORKER half of a relay — a Worker in front of a Mac-bound app over ONE outbound WebSocket: the frames (station's wire, byte for byte), the key-digest door (`acceptLink`), and `RelayLink`, the Durable Object an app subclasses with its name, body ceiling, timeout and (optionally) station's per-cookie offline cache. No `node:` import (0.10.0, lifted from station for roms, task 065) |
29
30
  | `cursedops/relay-link` | the MAC half — `dialLink` (answer each frame with the app's own `fetch`, redial 1 s → 60 s on close, no timer otherwise), `readLinkSettings` for `<APP>_LINK_URL`/`<APP>_LINK_KEY`, and `rotateLinkKey` (digest to the Worker FIRST, the key to the 0600 file only if it took) (0.10.0) |
30
31
  | `cursedops/relay-stage` | the signed-in STAGE WALK's skeleton both relay apps shared — `stageByteCheck` (SHA-256 of every built file, a stale byte re-asked 4 × 10 s), `putStageLinkDigest` (`--env stage` only), `grantStageOwner` (auth-stage's 2FA grant, never production's), `oldKeyRefused` (the rotation leg: 20 s refused window, 3 min ceiling), `awaitSystemResolver`, `servedPath`, `STAGE_AUTH_URL`. With `relayWorkerDeployCli` in `cursedops/worker-deploy`, a relay app's whole `worker:deploy` as data (0.10.6) |
@@ -95,6 +96,13 @@ Workers — so the choice was two more copies of the scanner or one here. Copies
95
96
  It is mechanism only: each app passes its own root and directories and keeps its own
96
97
  "not vacuous" assertion about which files the scan must reach.
97
98
 
99
+ 🔴 **`ship` is the fourth recorded exception: no app wrote it, because it is the switch the
100
+ deploys themselves obey.** The owner's ruling (2026-09-28) was that agents run the same commands
101
+ whatever the switch says and the commands deal with it — so it has to live in the one seam every
102
+ production deploy passes through, which is `runWorkerDeploy` here and `deploy-tree`'s callers.
103
+ It reads a file the runner owns and knows no app: rule 2 holds, and the header of `src/ship.ts`
104
+ says why every failure reads as `fast`.
105
+
98
106
  ### What was deliberately left in the apps
99
107
 
100
108
  The comparison that produced this library found three areas that are genuinely
@@ -296,6 +304,12 @@ pins an address** — the deliverable is the distinction, not a way around the n
296
304
  `steadyHealth(ask, ready)` is the Worker smoke's wait: `ready` must hold for THREE reads in a
297
305
  row, because on `music`'s first deploy (2026-09-23, `a54ef71`) one good `/healthz` was followed
298
306
  by thirteen routes answering the pre-secrets version. It returns what answered and never judges.
307
+ A read that THROWS (0.11.1) is a bad read asked again, bounded by `deadlineMs` (240 s), and only
308
+ a LAST read that threw is thrown to the caller: the 2026-09-29 ship reported family and music red
309
+ on a first `/healthz` that timed out twice while the edge was between versions (task 2291).
310
+ `runWorkerSmoke` asks each of those reads with a 10 s curl `--max-time`
311
+ (`HEALTH_READ_MAX_TIME_SEC`, passed as `EdgeRequestInit.maxTimeSec`), so the window costs 20 s a
312
+ read instead of 120.
299
313
 
300
314
  `awaitConverged(probes)` (0.10.19) is the same wait over SEVERAL paths: every probe must agree
301
315
  three rounds in a row, bounded (120 s), before any leg is judged — a Worker's code and its static
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "cursedops",
3
- "version": "0.10.26",
3
+ "version": "0.11.2",
4
4
  "description": "The build-and-ops answers this generation's apps wrote independently and identically: finding a generation's roots — and printing a command that runs when pasted — without knowing a path, the generation's whole-tree laws run over one repo from a checkout or a worktree, macOS launchd agent install/replace/remove and the live port a job serves, the scaffolding and verdicts of a deployed smoke (origin probe, the smoke's own environment, a network that lies about DNS, a settled version), the static-serving helpers eight apps copied — the path-traversal guard among them — the API floor that keeps an unmatched /api/... from ever being answered with the app shell, the commit and dirty flag a checkout-served process reports, the Cloudflare Worker deploy toolkit four apps copied (the deploy sequence, exact-set secrets over a pipe, origin-first rollback, the curl edge fetch, the row-for-row D1 import proof, the billed-CPU tail check around a deploy's walk and smoke, and each app's worker:secrets and worker:smoke main as one function of its data), the relay a Worker fronts a Mac-bound app with (the Durable Object, the frames, the Mac's dialer and key rotation — lifted from station for roms — and the signed-in stage walk's skeleton and the relay app's whole worker:deploy), the Worker import-graph and await-port checks every Worker app's suite runs over its own source, and the public-surface ratchet three published libraries each carried a forked copy of. Mechanism only — no app knows its name from here. Bun, zero runtime dependencies (typescript is an optional peer, for public-surface only), ships source.",
5
5
  "type": "module",
6
6
  "scripts": {
@@ -122,6 +122,12 @@
122
122
  "source": "./src/deployTree.ts",
123
123
  "import": "./src/deployTree.ts"
124
124
  },
125
+ "./ship": {
126
+ "types": "./src/ship.ts",
127
+ "bun": "./src/ship.ts",
128
+ "source": "./src/ship.ts",
129
+ "import": "./src/ship.ts"
130
+ },
125
131
  "./relay": {
126
132
  "types": "./src/relay.ts",
127
133
  "bun": "./src/relay.ts",
package/src/backups.ts CHANGED
@@ -7,3 +7,28 @@
7
7
  * whatever was decided. So it is one number here, and each app's prune takes it.
8
8
  */
9
9
  export const BACKUP_KEEP = 14;
10
+
11
+ /**
12
+ * The name of one dated run: `YYYYMMDD-HHMMSS`, in the Mac's LOCAL time.
13
+ *
14
+ * 🔴 A contract with two sides, so it is spelled once. Every app's own backup script WRITES
15
+ * these directory names and prunes only what matches the pattern; station's backup audit READS
16
+ * them back into a time. Until cursedops 0.11.2 the writers held nine copies of the pattern and
17
+ * two of the stamp, and the reader held its own pair that wrote and read the stamp as UTC — so a
18
+ * run the apps named `20260929-100000` at 10:00 CDT was read as 10:00Z, five hours off.
19
+ */
20
+ export const BACKUP_STAMP_PATTERN = /^\d{8}-\d{6}$/;
21
+
22
+ const pad = (n: number): string => String(n).padStart(2, "0");
23
+
24
+ /** A `Date` → its run name, in local time. {@link stampTime} is its inverse. */
25
+ export const backupStamp = (d: Date): string =>
26
+ `${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}`;
27
+
28
+ /** A run name → epoch ms, read as local time — or `null` for anything that is not one. */
29
+ export function stampTime(stamp: string): number | null {
30
+ if (!BACKUP_STAMP_PATTERN.test(stamp)) return null;
31
+ const n = (from: number, to: number): number => Number(stamp.slice(from, to));
32
+ const at = new Date(n(0, 4), n(4, 6) - 1, n(6, 8), n(9, 11), n(11, 13), n(13, 15));
33
+ return Number.isNaN(at.getTime()) ? null : at.getTime();
34
+ }
package/src/edgeFetch.ts CHANGED
@@ -58,6 +58,12 @@ export interface EdgeFetchOptions {
58
58
  retryDelayMs?: (retry: number) => number;
59
59
  }
60
60
 
61
+ /**
62
+ * A `RequestInit` this fetch reads one more field from: `maxTimeSec`, curl's `--max-time` for
63
+ * THIS request, over the instance's {@link EdgeFetchOptions.maxTimeSec}. A plain `fetch` ignores it.
64
+ */
65
+ export type EdgeRequestInit = RequestInit & { maxTimeSec?: number };
66
+
61
67
  /**
62
68
  * 🔴 How many MORE times a request is sent after a connection-level curl failure that carried
63
69
  * no answer. Measured 2026-09-24: vault's prod deploy of `953e19a` reported "smoke FAILED" on one
@@ -125,7 +131,7 @@ export function parseCurlResponse(raw: Buffer, origin = "the edge", exit: number
125
131
  }
126
132
 
127
133
  /** A `fetch`-shaped function over curl — see the header. Each host is resolved once per instance — once it has an answer. */
128
- export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?: RequestInit) => Promise<Response> {
134
+ export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?: EdgeRequestInit) => Promise<Response> {
129
135
  const pinned = new Map<string, string>();
130
136
  const resolve = options.resolve === undefined ? digCloudflare : options.resolve;
131
137
  const curl =
@@ -155,7 +161,7 @@ export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, i
155
161
  const port = target.port || (target.protocol === "http:" ? "80" : "443");
156
162
  const head = () => {
157
163
  const ip = pinFor(target.hostname);
158
- return ["-s", "-i", "--max-time", String(options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
164
+ return ["-s", "-i", "--max-time", String(init.maxTimeSec ?? options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
159
165
  };
160
166
  const args: string[] = [];
161
167
  for (const [k, v] of Object.entries(options.headersFor?.(target.hostname) ?? {})) args.push("-H", `${k}: ${v}`);
package/src/ship.ts ADDED
@@ -0,0 +1,149 @@
1
+ /**
2
+ * 🚢 FAST SHIPPING — the owner's switch for when a production deploy happens: after each task
3
+ * (`each`), or once, at the end of a run of tasks (`fast`, the default).
4
+ *
5
+ * The owner's ruling, 2026-09-28:
6
+ *
7
+ * > *"While on, agents do not publish or do long running things at all if they block task
8
+ * > completions. Instead they keep doing tasks until the end and do a final test run and if
9
+ * > everything is perfect then they publish … While the toggle is off, things work like today …
10
+ * > I will generally want it on because a few hours of a broken deploy is generally not a deal
11
+ * > breaker if we can get a big increase in tasks completed."*
12
+ *
13
+ * and, on how agents should meet it: *"the same commands no matter what the toggle is set to and
14
+ * the automation commands deal with the changes automatically."* So this module is read by the
15
+ * COMMANDS — `runWorkerDeploy` and every Mac-side `bun run deploy` — not by an agent deciding
16
+ * whether to call them. Measured 2026-09-28 (`autopilot phases`): deploys ~110 min/day, publishes
17
+ * 45–120 min/day and the waits on them ~115 min/day had become the slowest things a task does,
18
+ * and a batch that touches one app three times deployed it three times.
19
+ *
20
+ * ## The contract — one small file the runner owns
21
+ *
22
+ * `$FORGE_STATE/state/autopilot/ship.json`, written only by `autopilot ship-mode` (the station
23
+ * console's switch runs it). This module only READS it, the same way `recordDeployEvent` writes the
24
+ * runner's `events.jsonl` without importing the runner: the file shape is the interface.
25
+ *
26
+ * { "mode": "fast" | "each", "setBy": "owner", "at": "<ISO>", "reason"?: "…" }
27
+ *
28
+ * - **Absent → `fast`.** The owner asked for it on by default.
29
+ * - **Unreadable or a mode it does not know → `fast`**, the same answer as absent. Fast never puts
30
+ * anything unproven on a production URL — it only makes the deploy LATER — so there is no
31
+ * direction of a bad read that ships something it should not.
32
+ * - **No generation at all → `each`.** A checkout outside any `forge.env` has no runner to ship
33
+ * for it later, so deferring there would defer for ever.
34
+ *
35
+ * ## The overrides, both directions
36
+ *
37
+ * `FORGE_SHIP_NOW=1` in the environment ships now whatever the switch says: the ship task's own
38
+ * deploys, an outage fix, and a deploy the owner asked for in words (his sentence beats a switch he
39
+ * flipped yesterday — the precedence the retired `fastMode.ts` settled on 2026-08-13). A stage
40
+ * deploy is never deferred: it IS the fast way to see a change on a real edge.
41
+ */
42
+ import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
43
+ import { dirname, join } from "node:path";
44
+ import { forgeState } from "cursedops/roots";
45
+
46
+ export const SHIP_MODES = ["fast", "each"] as const;
47
+ export type ShipMode = (typeof SHIP_MODES)[number];
48
+
49
+ /** The mode when the file is absent or unreadable — the owner's ruling, see the header. */
50
+ export const DEFAULT_SHIP_MODE: ShipMode = "fast";
51
+
52
+ /** `$FORGE_STATE/<this>` is the switch. The runner's `paths.ts` names the same file. */
53
+ export const SHIP_FILE = join("state", "autopilot", "ship.json");
54
+
55
+ /** The variable that ships now regardless of the switch. */
56
+ export const SHIP_NOW_VAR = "FORGE_SHIP_NOW";
57
+
58
+ export interface ShipSetting {
59
+ mode: ShipMode;
60
+ /** Who set it — `default` when no file says. */
61
+ setBy: string;
62
+ /** ISO, or `null` when nothing was ever written. */
63
+ at: string | null;
64
+ /** Why it reads as it does when the file could not be believed — shown, never thrown. */
65
+ unreadable?: string;
66
+ }
67
+
68
+ /** The switch as the file says it, with every failure read as {@link DEFAULT_SHIP_MODE}. */
69
+ export function readShipSetting(state: string | null): ShipSetting {
70
+ if (state === null) return { mode: "each", setBy: "no generation", at: null };
71
+ const file = join(state, SHIP_FILE);
72
+ if (!existsSync(file)) return { mode: DEFAULT_SHIP_MODE, setBy: "default", at: null };
73
+ try {
74
+ const raw = JSON.parse(readFileSync(file, "utf8")) as { mode?: unknown; setBy?: unknown; at?: unknown };
75
+ if (!SHIP_MODES.includes(raw.mode as ShipMode)) {
76
+ return { mode: DEFAULT_SHIP_MODE, setBy: "default", at: null, unreadable: `mode ${JSON.stringify(raw.mode)} is not one of ${SHIP_MODES.join(", ")}` };
77
+ }
78
+ return {
79
+ mode: raw.mode as ShipMode,
80
+ setBy: typeof raw.setBy === "string" && raw.setBy ? raw.setBy : "unknown",
81
+ at: typeof raw.at === "string" ? raw.at : null,
82
+ };
83
+ } catch (cause) {
84
+ return { mode: DEFAULT_SHIP_MODE, setBy: "default", at: null, unreadable: (cause as Error).message };
85
+ }
86
+ }
87
+
88
+ /** Is `FORGE_SHIP_NOW` set to something that means yes? */
89
+ export function shipNowAsked(env: NodeJS.ProcessEnv = process.env): boolean {
90
+ const v = env[SHIP_NOW_VAR]?.trim().toLowerCase();
91
+ return v === "1" || v === "true" || v === "yes";
92
+ }
93
+
94
+ /**
95
+ * The sentence a production deploy prints INSTEAD of deploying, or `null` to deploy now.
96
+ *
97
+ * The deploy then exits 0: it did what the mode asks of it. The app stays behind prod on purpose,
98
+ * and the runner's ship task — filed when the queue empties or the oldest owed commit is too old —
99
+ * deploys every app that is behind, once. `autopilot ship` lists what is owed.
100
+ */
101
+ export function deferral(what: string, setting: ShipSetting, env: NodeJS.ProcessEnv = process.env): string | null {
102
+ if (setting.mode !== "fast" || shipNowAsked(env)) return null;
103
+ const by = setting.setBy === "default" ? "the default" : `set by ${setting.setBy}${setting.at ? ` at ${setting.at}` : ""}`;
104
+ return [
105
+ `🚢 ${what}: DEFERRED — fast shipping is on (${by}${setting.unreadable ? `; ship.json unreadable: ${setting.unreadable}` : ""}).`,
106
+ " Nothing was deployed, and that is the intended result, not a failure: this app is now owed to",
107
+ " the ship, which deploys every app behind its HEAD once, when the queue empties. Carry on with",
108
+ " the next task — file this one as done once it is landed and gated, saying the deploy is owed.",
109
+ ` \`autopilot ship\` lists what is owed. To deploy now anyway (an outage fix, the ship itself, or`,
110
+ ` the owner asked for it): ${SHIP_NOW_VAR}=1 <the same command>.`,
111
+ ].join("\n");
112
+ }
113
+
114
+ /**
115
+ * {@link deferral} for the checkout at `cwd` — what a Mac-side `bun run deploy` asks first:
116
+ *
117
+ * const deferred = shipDeferral(appRoot(), "roms production deploy");
118
+ * if (deferred) { console.log(deferred); process.exit(0); }
119
+ */
120
+ export function shipDeferral(cwd: string, what: string, env: NodeJS.ProcessEnv = process.env): string | null {
121
+ return deferral(what, readShipSetting(forgeState(cwd, env)), env);
122
+ }
123
+
124
+ /** The ledger line a deferred production deploy leaves — what `autopilot ship` counts. */
125
+ export interface ShipEventLine {
126
+ at: string;
127
+ kind: "ship";
128
+ what: "deferred";
129
+ repo: string;
130
+ env: string;
131
+ commit: string;
132
+ dirty: boolean;
133
+ }
134
+
135
+ /**
136
+ * Append one JSON line to the generation's ledger (`$FORGE_STATE/state/autopilot/events.jsonl`).
137
+ * Never throws; `false` when it did not write — a ledger that can fail a deploy is worse than none.
138
+ */
139
+ export function appendLedger(state: string | null, line: object): boolean {
140
+ if (!state) return false;
141
+ try {
142
+ const file = join(state, "state", "autopilot", "events.jsonl");
143
+ mkdirSync(dirname(file), { recursive: true });
144
+ appendFileSync(file, `${JSON.stringify(line)}\n`);
145
+ return true;
146
+ } catch {
147
+ return false;
148
+ }
149
+ }
package/src/smoke.ts CHANGED
@@ -83,6 +83,8 @@
83
83
  import { spawnSync } from "node:child_process";
84
84
  import { lookup } from "node:dns/promises";
85
85
  import { isIP } from "node:net";
86
+ // The package's own subpath, never `./edgeFetch.ts` — see `publishShape.test.ts`.
87
+ import type { EdgeRequestInit } from "cursedops/edge-fetch";
86
88
 
87
89
  export interface Failure {
88
90
  check: string;
@@ -840,12 +842,18 @@ export interface SteadyOptions {
840
842
  reads?: number;
841
843
  /** Reads before giving up and judging whatever answered. Default 40. */
842
844
  tries?: number;
845
+ /**
846
+ * Stop asking after this long, whatever `tries` says. Default 240 s — a read that THROWS can
847
+ * cost a whole curl `--max-time`, so a count alone does not bound the wait.
848
+ */
849
+ deadlineMs?: number;
843
850
  /** Pause after a GOOD read, before the next. Default 2 s. */
844
851
  pauseMs?: number;
845
852
  /** Pause after a BAD read. Default 5 s. */
846
853
  retryMs?: number;
847
854
  /** Injected for tests. */
848
855
  sleep?: (ms: number) => Promise<void>;
856
+ now?: () => number;
849
857
  }
850
858
 
851
859
  export interface SteadyRead<T> {
@@ -868,6 +876,14 @@ export interface SteadyRead<T> {
868
876
  * answering `503` (the pre-secrets version) and `404`, because one isolate had the new version
869
877
  * and the next did not. The copies in `vault` and `collections` still judged after one read.
870
878
  *
879
+ * 🔴 **A read that THROWS is a bad read, not the end of the wait.** Measured 2026-09-29 (task
880
+ * 2291): family's and music's production smokes each went red on their FIRST `/healthz`, straight
881
+ * after the secrets were put — `no answer … (curl exit 28)`, twice 60 s — and both hosts answered
882
+ * in 0.2 s moments later. The edge was between versions; rethrowing that first silence charged a
883
+ * green deploy with a red smoke. A throw now resets the count and is asked again until `tries`
884
+ * or `deadlineMs`; only when the LAST read threw is that error thrown to the caller, so a Worker
885
+ * that never answers is still a red that names why.
886
+ *
871
887
  * It never decides pass or fail: it returns what answered, and the caller judges that — a wait
872
888
  * that swallowed its own timeout would be a check that cannot go red.
873
889
  */
@@ -878,22 +894,44 @@ export async function steadyHealth<T = Record<string, unknown>>(
878
894
  ): Promise<SteadyRead<T>> {
879
895
  const reads = options.reads ?? 3;
880
896
  const tries = options.tries ?? 40;
897
+ const deadlineMs = options.deadlineMs ?? 240_000;
881
898
  const sleep = options.sleep ?? ((ms: number) => new Promise<void>((done) => setTimeout(done, ms)));
899
+ const now = options.now ?? Date.now;
900
+ const started = now();
882
901
  let steady = 0;
883
902
  let asked = 0;
884
903
  for (;;) {
885
- const response = await ask();
904
+ let read: { response: Response; body: T } | null = null;
905
+ let thrown: unknown = null;
906
+ try {
907
+ const response = await ask();
908
+ const body = (await response
909
+ .clone()
910
+ .json()
911
+ .catch(() => ({}))) as T;
912
+ read = { response, body };
913
+ } catch (error) {
914
+ thrown = error;
915
+ }
886
916
  asked++;
887
- const body = (await response
888
- .clone()
889
- .json()
890
- .catch(() => ({}))) as T;
891
- steady = ready(body, response) ? steady + 1 : 0;
892
- if (steady >= reads || asked >= tries) return { response, body, steady, asked };
917
+ steady = read && ready(read.body, read.response) ? steady + 1 : 0;
918
+ const out = asked >= tries || now() - started >= deadlineMs;
919
+ if (read && (steady >= reads || out)) return { ...read, steady, asked };
920
+ if (!read && out) throw thrown;
893
921
  await sleep(steady > 0 ? (options.pauseMs ?? 2_000) : (options.retryMs ?? 5_000));
894
922
  }
895
923
  }
896
924
 
925
+ /**
926
+ * curl's `--max-time` for ONE `/healthz` read of a settling Worker, in seconds. Short on purpose:
927
+ * {@link steadyHealth} asks again after a silence, so a long timeout only makes the edge's
928
+ * between-versions window cost more — the 2026-09-29 reds spent 120 s on two reads of it.
929
+ */
930
+ export const HEALTH_READ_MAX_TIME_SEC = 10;
931
+
932
+ /** The init a smoke's `/healthz` read carries — `cursedops/edge-fetch` honours it; a plain `fetch` ignores it. */
933
+ const healthRead: EdgeRequestInit = { maxTimeSec: HEALTH_READ_MAX_TIME_SEC };
934
+
897
935
  /** One question a deploy must answer the NEW way — `null` when it does, else why not. */
898
936
  export interface ConvergenceProbe {
899
937
  name: string;
@@ -1065,7 +1103,7 @@ export async function runWorkerSmoke(spec: WorkerSmokeSpec): Promise<number> {
1065
1103
  let status: number;
1066
1104
  try {
1067
1105
  const read = await steadyHealth<Record<string, unknown>>(
1068
- () => ask("/healthz"),
1106
+ () => ask("/healthz", healthRead),
1069
1107
  (body) => body.commit === spec.head && body.apiServed === true,
1070
1108
  spec.steady,
1071
1109
  );
@@ -1155,7 +1193,7 @@ async function accessDoorSmoke(
1155
1193
  let health: Record<string, unknown>;
1156
1194
  let status: number;
1157
1195
  try {
1158
- const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz"), (body) => body.commit === spec.head, spec.steady);
1196
+ const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz", healthRead), (body) => body.commit === spec.head, spec.steady);
1159
1197
  health = read.body;
1160
1198
  status = read.response.status;
1161
1199
  } catch (error) {
@@ -53,9 +53,10 @@
53
53
  * refusal without wrangler, git or a network.
54
54
  */
55
55
  import { spawn, spawnSync } from "node:child_process";
56
- import { appendFileSync, existsSync, mkdirSync, readdirSync, readFileSync } from "node:fs";
57
- import { dirname, join } from "node:path";
56
+ import { existsSync, readdirSync, readFileSync } from "node:fs";
57
+ import { join } from "node:path";
58
58
  import { forgeState } from "cursedops/roots";
59
+ import { appendLedger, deferral, readShipSetting, type ShipEventLine, type ShipSetting } from "cursedops/ship";
59
60
 
60
61
  /** The two deployments every Worker app here has. */
61
62
  export type WorkerEnv = "production" | "stage";
@@ -318,6 +319,16 @@ export interface DeployDeps {
318
319
  * A throw here is swallowed: a ledger that can fail a deploy is worse than none.
319
320
  */
320
321
  record?: (line: DeployEventLine) => void;
322
+ /**
323
+ * The owner's fast-shipping switch ({@link readShipSetting}). {@link workerDeployDeps} reads the
324
+ * generation's `ship.json`; absent here, the deploy behaves as it always did — so a suite that
325
+ * builds its own deps never reads this Mac's switch.
326
+ */
327
+ ship?: () => ShipSetting;
328
+ /** Where `FORGE_SHIP_NOW` is read from. Default `process.env`. */
329
+ env?: NodeJS.ProcessEnv;
330
+ /** Where a deferred production deploy's `ship` line goes. {@link workerDeployDeps} appends it to the ledger. */
331
+ recordShip?: (line: ShipEventLine) => void;
321
332
  }
322
333
 
323
334
  /**
@@ -353,15 +364,7 @@ export interface DeployEventLine {
353
364
 
354
365
  /** Append one line to the ledger under a state root. Never throws; `false` when it did not write. */
355
366
  export function recordDeployEvent(state: string | null, line: DeployEventLine): boolean {
356
- if (!state) return false;
357
- try {
358
- const file = join(state, "state", "autopilot", "events.jsonl");
359
- mkdirSync(dirname(file), { recursive: true });
360
- appendFileSync(file, `${JSON.stringify(line)}\n`);
361
- return true;
362
- } catch {
363
- return false;
364
- }
367
+ return appendLedger(state, line);
365
368
  }
366
369
 
367
370
  export interface DeployResult {
@@ -404,6 +407,10 @@ export function workerDeployDeps(cwd: string): DeployDeps {
404
407
  record: (line) => {
405
408
  recordDeployEvent(forgeState(cwd), line);
406
409
  },
410
+ ship: () => readShipSetting(forgeState(cwd)),
411
+ recordShip: (line) => {
412
+ appendLedger(forgeState(cwd), line);
413
+ },
407
414
  };
408
415
  }
409
416
 
@@ -688,6 +695,21 @@ export function runWorkerDeploy(spec: WorkerDeploySpec, deps: DeployDeps): Deplo
688
695
  const unmeasured: string[] = [];
689
696
 
690
697
  if (!commit) return stop("head", "this checkout has no readable HEAD — there is no commit to stamp, so nothing could prove the deploy.");
698
+ // 🚢 Fast shipping (`./ship.ts`): a production deploy made while the switch is on is owed to the
699
+ // runner's ship instead — asked before anything installs or builds, so deferring costs nothing.
700
+ // Its `ship` line, not a `deploy` one: a deferral is not a deploy, and would drag every median.
701
+ if (spec.env === "production" && deps.ship) {
702
+ const deferred = deferral(`${spec.app} production deploy`, deps.ship(), deps.env ?? process.env);
703
+ if (deferred) {
704
+ log(`\n${deferred}`);
705
+ try {
706
+ deps.recordShip?.({ at: new Date(started).toISOString(), kind: "ship", what: "deferred", repo: spec.app, env: spec.env, commit: commit.slice(0, 8), dirty });
707
+ } catch {
708
+ // never the deploy's verdict — see DeployDeps.record
709
+ }
710
+ return { code: 0, step: "deferred", detail: deferred, commit, dirty };
711
+ }
712
+ }
691
713
  if (spec.env === "production" && dirty) {
692
714
  return stop("clean-tree", `the tree is dirty — production would report ${commit.slice(0, 8)} and run something else:\n${porcelain}`);
693
715
  }
@@ -92,6 +92,12 @@ export interface SecretSync {
92
92
  /** Tries per put. A put is idempotent, so a retry can only turn a network blip into a success. */
93
93
  export const SECRET_PUT_ATTEMPTS = 3;
94
94
 
95
+ /**
96
+ * Reads of the account after the puts before a set that is not exact is believed. A read changes
97
+ * nothing, so asking again can only replace a stale answer with a current one — see {@link syncSecrets}.
98
+ */
99
+ export const SECRET_READBACK_ATTEMPTS = 5;
100
+
95
101
  /**
96
102
  * Apply `plan`: put every value, delete every extra, then READ THE ACCOUNT BACK and fail unless it
97
103
  * holds exactly the set. Refuses without touching anything when the plan has `absent` keys.
@@ -100,11 +106,17 @@ export const SECRET_PUT_ATTEMPTS = 3;
100
106
  * `wrangler secret put` of three hit `fetch failed` and the whole deploy stopped with
101
107
  * collections-stage holding a Worker and no secrets — serving nothing — for a blip the next
102
108
  * attempt would have cleared. The read-back is still the verdict.
109
+ *
110
+ * 🔴 **The read-back is asked again, up to {@link SECRET_READBACK_ATTEMPTS} times, before a set
111
+ * that is not exact is believed.** Measured 2026-09-29 on family-stage (task 2291): the account
112
+ * listed both secrets, both puts said ok, and the list read straight after answered NOTHING — a
113
+ * green deploy stopped as "secrets NOT attached". The same script run again a minute later read
114
+ * both. The LAST read is the verdict, so a set that is really wrong is still a red.
103
115
  */
104
116
  export async function syncSecrets(
105
117
  plan: SecretPlan,
106
118
  ops: SecretOps,
107
- options: { attempts?: number; pause?: (ms: number) => Promise<void> } = {},
119
+ options: { attempts?: number; readbacks?: number; pause?: (ms: number) => Promise<void> } = {},
108
120
  ): Promise<SecretSync> {
109
121
  const attempts = Math.max(1, options.attempts ?? SECRET_PUT_ATTEMPTS);
110
122
  const pause = options.pause ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)));
@@ -130,9 +142,20 @@ export async function syncSecrets(
130
142
  lines.push(`DELETE ${name} — not in the set — ${ok ? "gone" : "FAILED"}`);
131
143
  if (!ok) bad++;
132
144
  }
133
- const after = await ops.held();
134
- const exact = after.length === plan.wanted.length && plan.wanted.every((name) => after.includes(name));
135
- lines.push(exact ? `holds exactly: ${after.join(", ")}` : `🔴 holds ${after.join(", ") || "(nothing)"} — not exactly ${plan.wanted.join(", ")}`);
145
+ const readbacks = Math.max(1, options.readbacks ?? SECRET_READBACK_ATTEMPTS);
146
+ const isExact = (held: readonly string[]) => held.length === plan.wanted.length && plan.wanted.every((name) => held.includes(name));
147
+ let after = await ops.held();
148
+ let reads = 1;
149
+ while (!isExact(after) && reads < readbacks) {
150
+ await pause(2_000 * reads);
151
+ after = await ops.held();
152
+ reads++;
153
+ }
154
+ const exact = isExact(after);
155
+ const reread = reads > 1 ? ` (read ${reads} times)` : "";
156
+ lines.push(
157
+ exact ? `holds exactly: ${after.join(", ")}${reread}` : `🔴 holds ${after.join(", ") || "(nothing)"} — not exactly ${plan.wanted.join(", ")}${reread}`,
158
+ );
136
159
  return { ok: exact && bad === 0, lines, after };
137
160
  }
138
161