cursedops 0.10.26 → 0.11.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -3
- package/package.json +7 -1
- package/src/backups.ts +25 -0
- package/src/edgeFetch.ts +8 -2
- package/src/ship.ts +149 -0
- package/src/smoke.ts +47 -9
- package/src/workerDeploy.ts +33 -11
- package/src/workerSecrets.ts +27 -4
package/README.md
CHANGED
|
@@ -14,17 +14,18 @@ bun add cursedops
|
|
|
14
14
|
| `cursedops/smoke` | the scaffolding of a deployed smoke — the ledger, the fetch, the DNS hint, the exit code — the one check no app owns (every address of a deployment serving the same built client), and since 0.5.0 its VERDICTS: the origin asked on loopback, the smoke's own environment, a network that lies about DNS, and a version that has settled |
|
|
15
15
|
| `cursedops/worker-deploy` | the sequence that ships a Cloudflare Worker — clean tree, stage first, schema, `--var` stamp, secrets, smoke — and the readers it needs (0.5.0); `readWranglerFile(root)` since 0.9.0, lifted from four apps' `readWrangler` (task 2153); since 0.10.25 every run, red or green, appends a `kind: "deploy"` line with each leg's wall clock (`install`, `stage`, `walk`, `build`, `schema`, `upload`, `secrets`, `smoke`) to the generation's ledger (task 2246) |
|
|
16
16
|
| `cursedops/worker-deploy` (CPU) | a `wrangler tail` around the traffic a deploy already sends, each request's BILLED CPU charged to its declared route, red on an undeclared, over-budget or uncapped route — and `runWorkerDeploy`'s `stageWalk` + `cpuTail` that wrap the walk and the smoke in it (0.7.0, task 2134, lifted from collections) |
|
|
17
|
-
| `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) |
|
|
17
|
+
| `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) — and read again, up to five times, before a set that is not exact is believed: the list lags the puts (0.11.1) |
|
|
18
18
|
| `cursedops/worker-rollback` | a hostname back on its Mac origin: origin first, route second, the route found rather than typed (0.5.0) |
|
|
19
|
-
| `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0) |
|
|
19
|
+
| `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0); a request's own `maxTimeSec` over the instance's (0.11.1) |
|
|
20
20
|
| `cursedops/d1-import` | a cutover's data proof — SQL literals and the row-for-row comparison, never a count (0.5.0); and `d1SchemaText`, the generated `db/schema.sql` that refuses a statement spanning lines, because D1's exec splits on newlines (0.9.0, lifted from five apps, task 2153) — and its `forge-d1-schema` bin (0.9.1), so no app carries its own schema script at all |
|
|
21
|
-
| `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153) |
|
|
21
|
+
| `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153); `BACKUP_STAMP_PATTERN`, `backupStamp` and its inverse `stampTime` — the local-time run name every backup script writes and station's audit reads (0.11.2) |
|
|
22
22
|
| `cursedops/serve` | the static tier's four helpers — the path-traversal guard, the MIME table, the hashed-asset test, the crash handlers |
|
|
23
23
|
| `cursedops/api-floor` | the rule that an unmatched `/api/...` is a phrase and never the app shell — the namespace predicates, the trailing-slash normaliser and the default 404 body. No `node:` import, so it mounts inside a Worker |
|
|
24
24
|
| `cursedops/build-info` | which commit a checkout-served process is running and whether its tree was dirty — read once at load, and a `null` retried in the background rather than cached for the life of the process. `node:child_process`, so never in a Worker |
|
|
25
25
|
| `cursedops/bound-lists` | the static check that no SQL builds an `IN (…)` list one `?` per value — the shape D1 500s on past 100 ids. Test-time only; the recorded third exception, see below |
|
|
26
26
|
| `cursedops/staged-client` | the IMMUTABLE client a checkout-served app serves — a build staged into `<APP_DATA_DIR>/client/<commit>/` behind `CURRENT`, so another agent's `vite build` in the checkout never changes live bytes — and its `forge-client stamp` / `forge-client stage` bin (0.6.0, lifted from family, flix, roms, station) |
|
|
27
27
|
| `cursedops/deploy-tree` | the two refusals a checkout-served deploy needs — never deploy a dirty tree, never `git revert` in one (`readTreeStatus`, `refuseToDeploy`, `refuseToRevert`; 0.6.0, lifted from nine apps) |
|
|
28
|
+
| `cursedops/ship` | the owner's FAST SHIPPING switch as every deploy reads it — `readShipSetting` (`$FORGE_STATE/state/autopilot/ship.json`; absent or unreadable is `fast`, no generation is `each`), `deferral`/`shipDeferral` (the sentence a production deploy prints instead of deploying, or `null`), `FORGE_SHIP_NOW=1` to ship now whatever it says. `runWorkerDeploy` asks it before installing anything; a Mac-side `bun run deploy` calls `shipDeferral` first (0.11.0) |
|
|
28
29
|
| `cursedops/relay` | the WORKER half of a relay — a Worker in front of a Mac-bound app over ONE outbound WebSocket: the frames (station's wire, byte for byte), the key-digest door (`acceptLink`), and `RelayLink`, the Durable Object an app subclasses with its name, body ceiling, timeout and (optionally) station's per-cookie offline cache. No `node:` import (0.10.0, lifted from station for roms, task 065) |
|
|
29
30
|
| `cursedops/relay-link` | the MAC half — `dialLink` (answer each frame with the app's own `fetch`, redial 1 s → 60 s on close, no timer otherwise), `readLinkSettings` for `<APP>_LINK_URL`/`<APP>_LINK_KEY`, and `rotateLinkKey` (digest to the Worker FIRST, the key to the 0600 file only if it took) (0.10.0) |
|
|
30
31
|
| `cursedops/relay-stage` | the signed-in STAGE WALK's skeleton both relay apps shared — `stageByteCheck` (SHA-256 of every built file, a stale byte re-asked 4 × 10 s), `putStageLinkDigest` (`--env stage` only), `grantStageOwner` (auth-stage's 2FA grant, never production's), `oldKeyRefused` (the rotation leg: 20 s refused window, 3 min ceiling), `awaitSystemResolver`, `servedPath`, `STAGE_AUTH_URL`. With `relayWorkerDeployCli` in `cursedops/worker-deploy`, a relay app's whole `worker:deploy` as data (0.10.6) |
|
|
@@ -95,6 +96,13 @@ Workers — so the choice was two more copies of the scanner or one here. Copies
|
|
|
95
96
|
It is mechanism only: each app passes its own root and directories and keeps its own
|
|
96
97
|
"not vacuous" assertion about which files the scan must reach.
|
|
97
98
|
|
|
99
|
+
🔴 **`ship` is the fourth recorded exception: no app wrote it, because it is the switch the
|
|
100
|
+
deploys themselves obey.** The owner's ruling (2026-09-28) was that agents run the same commands
|
|
101
|
+
whatever the switch says and the commands deal with it — so it has to live in the one seam every
|
|
102
|
+
production deploy passes through, which is `runWorkerDeploy` here and `deploy-tree`'s callers.
|
|
103
|
+
It reads a file the runner owns and knows no app: rule 2 holds, and the header of `src/ship.ts`
|
|
104
|
+
says why every failure reads as `fast`.
|
|
105
|
+
|
|
98
106
|
### What was deliberately left in the apps
|
|
99
107
|
|
|
100
108
|
The comparison that produced this library found three areas that are genuinely
|
|
@@ -296,6 +304,12 @@ pins an address** — the deliverable is the distinction, not a way around the n
|
|
|
296
304
|
`steadyHealth(ask, ready)` is the Worker smoke's wait: `ready` must hold for THREE reads in a
|
|
297
305
|
row, because on `music`'s first deploy (2026-09-23, `a54ef71`) one good `/healthz` was followed
|
|
298
306
|
by thirteen routes answering the pre-secrets version. It returns what answered and never judges.
|
|
307
|
+
A read that THROWS (0.11.1) is a bad read asked again, bounded by `deadlineMs` (240 s), and only
|
|
308
|
+
a LAST read that threw is thrown to the caller: the 2026-09-29 ship reported family and music red
|
|
309
|
+
on a first `/healthz` that timed out twice while the edge was between versions (task 2291).
|
|
310
|
+
`runWorkerSmoke` asks each of those reads with a 10 s curl `--max-time`
|
|
311
|
+
(`HEALTH_READ_MAX_TIME_SEC`, passed as `EdgeRequestInit.maxTimeSec`), so the window costs 20 s a
|
|
312
|
+
read instead of 120.
|
|
299
313
|
|
|
300
314
|
`awaitConverged(probes)` (0.10.19) is the same wait over SEVERAL paths: every probe must agree
|
|
301
315
|
three rounds in a row, bounded (120 s), before any leg is judged — a Worker's code and its static
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cursedops",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.2",
|
|
4
4
|
"description": "The build-and-ops answers this generation's apps wrote independently and identically: finding a generation's roots — and printing a command that runs when pasted — without knowing a path, the generation's whole-tree laws run over one repo from a checkout or a worktree, macOS launchd agent install/replace/remove and the live port a job serves, the scaffolding and verdicts of a deployed smoke (origin probe, the smoke's own environment, a network that lies about DNS, a settled version), the static-serving helpers eight apps copied — the path-traversal guard among them — the API floor that keeps an unmatched /api/... from ever being answered with the app shell, the commit and dirty flag a checkout-served process reports, the Cloudflare Worker deploy toolkit four apps copied (the deploy sequence, exact-set secrets over a pipe, origin-first rollback, the curl edge fetch, the row-for-row D1 import proof, the billed-CPU tail check around a deploy's walk and smoke, and each app's worker:secrets and worker:smoke main as one function of its data), the relay a Worker fronts a Mac-bound app with (the Durable Object, the frames, the Mac's dialer and key rotation — lifted from station for roms — and the signed-in stage walk's skeleton and the relay app's whole worker:deploy), the Worker import-graph and await-port checks every Worker app's suite runs over its own source, and the public-surface ratchet three published libraries each carried a forked copy of. Mechanism only — no app knows its name from here. Bun, zero runtime dependencies (typescript is an optional peer, for public-surface only), ships source.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"scripts": {
|
|
@@ -122,6 +122,12 @@
|
|
|
122
122
|
"source": "./src/deployTree.ts",
|
|
123
123
|
"import": "./src/deployTree.ts"
|
|
124
124
|
},
|
|
125
|
+
"./ship": {
|
|
126
|
+
"types": "./src/ship.ts",
|
|
127
|
+
"bun": "./src/ship.ts",
|
|
128
|
+
"source": "./src/ship.ts",
|
|
129
|
+
"import": "./src/ship.ts"
|
|
130
|
+
},
|
|
125
131
|
"./relay": {
|
|
126
132
|
"types": "./src/relay.ts",
|
|
127
133
|
"bun": "./src/relay.ts",
|
package/src/backups.ts
CHANGED
|
@@ -7,3 +7,28 @@
|
|
|
7
7
|
* whatever was decided. So it is one number here, and each app's prune takes it.
|
|
8
8
|
*/
|
|
9
9
|
export const BACKUP_KEEP = 14;
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* The name of one dated run: `YYYYMMDD-HHMMSS`, in the Mac's LOCAL time.
|
|
13
|
+
*
|
|
14
|
+
* 🔴 A contract with two sides, so it is spelled once. Every app's own backup script WRITES
|
|
15
|
+
* these directory names and prunes only what matches the pattern; station's backup audit READS
|
|
16
|
+
* them back into a time. Until cursedops 0.11.2 the writers held nine copies of the pattern and
|
|
17
|
+
* two of the stamp, and the reader held its own pair that wrote and read the stamp as UTC — so a
|
|
18
|
+
* run the apps named `20260929-100000` at 10:00 CDT was read as 10:00Z, five hours off.
|
|
19
|
+
*/
|
|
20
|
+
export const BACKUP_STAMP_PATTERN = /^\d{8}-\d{6}$/;
|
|
21
|
+
|
|
22
|
+
const pad = (n: number): string => String(n).padStart(2, "0");
|
|
23
|
+
|
|
24
|
+
/** A `Date` → its run name, in local time. {@link stampTime} is its inverse. */
|
|
25
|
+
export const backupStamp = (d: Date): string =>
|
|
26
|
+
`${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}`;
|
|
27
|
+
|
|
28
|
+
/** A run name → epoch ms, read as local time — or `null` for anything that is not one. */
|
|
29
|
+
export function stampTime(stamp: string): number | null {
|
|
30
|
+
if (!BACKUP_STAMP_PATTERN.test(stamp)) return null;
|
|
31
|
+
const n = (from: number, to: number): number => Number(stamp.slice(from, to));
|
|
32
|
+
const at = new Date(n(0, 4), n(4, 6) - 1, n(6, 8), n(9, 11), n(11, 13), n(13, 15));
|
|
33
|
+
return Number.isNaN(at.getTime()) ? null : at.getTime();
|
|
34
|
+
}
|
package/src/edgeFetch.ts
CHANGED
|
@@ -58,6 +58,12 @@ export interface EdgeFetchOptions {
|
|
|
58
58
|
retryDelayMs?: (retry: number) => number;
|
|
59
59
|
}
|
|
60
60
|
|
|
61
|
+
/**
|
|
62
|
+
* A `RequestInit` this fetch reads one more field from: `maxTimeSec`, curl's `--max-time` for
|
|
63
|
+
* THIS request, over the instance's {@link EdgeFetchOptions.maxTimeSec}. A plain `fetch` ignores it.
|
|
64
|
+
*/
|
|
65
|
+
export type EdgeRequestInit = RequestInit & { maxTimeSec?: number };
|
|
66
|
+
|
|
61
67
|
/**
|
|
62
68
|
* 🔴 How many MORE times a request is sent after a connection-level curl failure that carried
|
|
63
69
|
* no answer. Measured 2026-09-24: vault's prod deploy of `953e19a` reported "smoke FAILED" on one
|
|
@@ -125,7 +131,7 @@ export function parseCurlResponse(raw: Buffer, origin = "the edge", exit: number
|
|
|
125
131
|
}
|
|
126
132
|
|
|
127
133
|
/** A `fetch`-shaped function over curl — see the header. Each host is resolved once per instance — once it has an answer. */
|
|
128
|
-
export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?:
|
|
134
|
+
export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?: EdgeRequestInit) => Promise<Response> {
|
|
129
135
|
const pinned = new Map<string, string>();
|
|
130
136
|
const resolve = options.resolve === undefined ? digCloudflare : options.resolve;
|
|
131
137
|
const curl =
|
|
@@ -155,7 +161,7 @@ export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, i
|
|
|
155
161
|
const port = target.port || (target.protocol === "http:" ? "80" : "443");
|
|
156
162
|
const head = () => {
|
|
157
163
|
const ip = pinFor(target.hostname);
|
|
158
|
-
return ["-s", "-i", "--max-time", String(options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
|
|
164
|
+
return ["-s", "-i", "--max-time", String(init.maxTimeSec ?? options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
|
|
159
165
|
};
|
|
160
166
|
const args: string[] = [];
|
|
161
167
|
for (const [k, v] of Object.entries(options.headersFor?.(target.hostname) ?? {})) args.push("-H", `${k}: ${v}`);
|
package/src/ship.ts
ADDED
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* 🚢 FAST SHIPPING — the owner's switch for when a production deploy happens: after each task
|
|
3
|
+
* (`each`), or once, at the end of a run of tasks (`fast`, the default).
|
|
4
|
+
*
|
|
5
|
+
* The owner's ruling, 2026-09-28:
|
|
6
|
+
*
|
|
7
|
+
* > *"While on, agents do not publish or do long running things at all if they block task
|
|
8
|
+
* > completions. Instead they keep doing tasks until the end and do a final test run and if
|
|
9
|
+
* > everything is perfect then they publish … While the toggle is off, things work like today …
|
|
10
|
+
* > I will generally want it on because a few hours of a broken deploy is generally not a deal
|
|
11
|
+
* > breaker if we can get a big increase in tasks completed."*
|
|
12
|
+
*
|
|
13
|
+
* and, on how agents should meet it: *"the same commands no matter what the toggle is set to and
|
|
14
|
+
* the automation commands deal with the changes automatically."* So this module is read by the
|
|
15
|
+
* COMMANDS — `runWorkerDeploy` and every Mac-side `bun run deploy` — not by an agent deciding
|
|
16
|
+
* whether to call them. Measured 2026-09-28 (`autopilot phases`): deploys ~110 min/day, publishes
|
|
17
|
+
* 45–120 min/day and the waits on them ~115 min/day had become the slowest things a task does,
|
|
18
|
+
* and a batch that touches one app three times deployed it three times.
|
|
19
|
+
*
|
|
20
|
+
* ## The contract — one small file the runner owns
|
|
21
|
+
*
|
|
22
|
+
* `$FORGE_STATE/state/autopilot/ship.json`, written only by `autopilot ship-mode` (the station
|
|
23
|
+
* console's switch runs it). This module only READS it, the same way `recordDeployEvent` writes the
|
|
24
|
+
* runner's `events.jsonl` without importing the runner: the file shape is the interface.
|
|
25
|
+
*
|
|
26
|
+
* { "mode": "fast" | "each", "setBy": "owner", "at": "<ISO>", "reason"?: "…" }
|
|
27
|
+
*
|
|
28
|
+
* - **Absent → `fast`.** The owner asked for it on by default.
|
|
29
|
+
* - **Unreadable or a mode it does not know → `fast`**, the same answer as absent. Fast never puts
|
|
30
|
+
* anything unproven on a production URL — it only makes the deploy LATER — so there is no
|
|
31
|
+
* direction of a bad read that ships something it should not.
|
|
32
|
+
* - **No generation at all → `each`.** A checkout outside any `forge.env` has no runner to ship
|
|
33
|
+
* for it later, so deferring there would defer for ever.
|
|
34
|
+
*
|
|
35
|
+
* ## The overrides, both directions
|
|
36
|
+
*
|
|
37
|
+
* `FORGE_SHIP_NOW=1` in the environment ships now whatever the switch says: the ship task's own
|
|
38
|
+
* deploys, an outage fix, and a deploy the owner asked for in words (his sentence beats a switch he
|
|
39
|
+
* flipped yesterday — the precedence the retired `fastMode.ts` settled on 2026-08-13). A stage
|
|
40
|
+
* deploy is never deferred: it IS the fast way to see a change on a real edge.
|
|
41
|
+
*/
|
|
42
|
+
import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
|
|
43
|
+
import { dirname, join } from "node:path";
|
|
44
|
+
import { forgeState } from "cursedops/roots";
|
|
45
|
+
|
|
46
|
+
export const SHIP_MODES = ["fast", "each"] as const;
|
|
47
|
+
export type ShipMode = (typeof SHIP_MODES)[number];
|
|
48
|
+
|
|
49
|
+
/** The mode when the file is absent or unreadable — the owner's ruling, see the header. */
|
|
50
|
+
export const DEFAULT_SHIP_MODE: ShipMode = "fast";
|
|
51
|
+
|
|
52
|
+
/** `$FORGE_STATE/<this>` is the switch. The runner's `paths.ts` names the same file. */
|
|
53
|
+
export const SHIP_FILE = join("state", "autopilot", "ship.json");
|
|
54
|
+
|
|
55
|
+
/** The variable that ships now regardless of the switch. */
|
|
56
|
+
export const SHIP_NOW_VAR = "FORGE_SHIP_NOW";
|
|
57
|
+
|
|
58
|
+
export interface ShipSetting {
|
|
59
|
+
mode: ShipMode;
|
|
60
|
+
/** Who set it — `default` when no file says. */
|
|
61
|
+
setBy: string;
|
|
62
|
+
/** ISO, or `null` when nothing was ever written. */
|
|
63
|
+
at: string | null;
|
|
64
|
+
/** Why it reads as it does when the file could not be believed — shown, never thrown. */
|
|
65
|
+
unreadable?: string;
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
/** The switch as the file says it, with every failure read as {@link DEFAULT_SHIP_MODE}. */
|
|
69
|
+
export function readShipSetting(state: string | null): ShipSetting {
|
|
70
|
+
if (state === null) return { mode: "each", setBy: "no generation", at: null };
|
|
71
|
+
const file = join(state, SHIP_FILE);
|
|
72
|
+
if (!existsSync(file)) return { mode: DEFAULT_SHIP_MODE, setBy: "default", at: null };
|
|
73
|
+
try {
|
|
74
|
+
const raw = JSON.parse(readFileSync(file, "utf8")) as { mode?: unknown; setBy?: unknown; at?: unknown };
|
|
75
|
+
if (!SHIP_MODES.includes(raw.mode as ShipMode)) {
|
|
76
|
+
return { mode: DEFAULT_SHIP_MODE, setBy: "default", at: null, unreadable: `mode ${JSON.stringify(raw.mode)} is not one of ${SHIP_MODES.join(", ")}` };
|
|
77
|
+
}
|
|
78
|
+
return {
|
|
79
|
+
mode: raw.mode as ShipMode,
|
|
80
|
+
setBy: typeof raw.setBy === "string" && raw.setBy ? raw.setBy : "unknown",
|
|
81
|
+
at: typeof raw.at === "string" ? raw.at : null,
|
|
82
|
+
};
|
|
83
|
+
} catch (cause) {
|
|
84
|
+
return { mode: DEFAULT_SHIP_MODE, setBy: "default", at: null, unreadable: (cause as Error).message };
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/** Is `FORGE_SHIP_NOW` set to something that means yes? */
|
|
89
|
+
export function shipNowAsked(env: NodeJS.ProcessEnv = process.env): boolean {
|
|
90
|
+
const v = env[SHIP_NOW_VAR]?.trim().toLowerCase();
|
|
91
|
+
return v === "1" || v === "true" || v === "yes";
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
/**
|
|
95
|
+
* The sentence a production deploy prints INSTEAD of deploying, or `null` to deploy now.
|
|
96
|
+
*
|
|
97
|
+
* The deploy then exits 0: it did what the mode asks of it. The app stays behind prod on purpose,
|
|
98
|
+
* and the runner's ship task — filed when the queue empties or the oldest owed commit is too old —
|
|
99
|
+
* deploys every app that is behind, once. `autopilot ship` lists what is owed.
|
|
100
|
+
*/
|
|
101
|
+
export function deferral(what: string, setting: ShipSetting, env: NodeJS.ProcessEnv = process.env): string | null {
|
|
102
|
+
if (setting.mode !== "fast" || shipNowAsked(env)) return null;
|
|
103
|
+
const by = setting.setBy === "default" ? "the default" : `set by ${setting.setBy}${setting.at ? ` at ${setting.at}` : ""}`;
|
|
104
|
+
return [
|
|
105
|
+
`🚢 ${what}: DEFERRED — fast shipping is on (${by}${setting.unreadable ? `; ship.json unreadable: ${setting.unreadable}` : ""}).`,
|
|
106
|
+
" Nothing was deployed, and that is the intended result, not a failure: this app is now owed to",
|
|
107
|
+
" the ship, which deploys every app behind its HEAD once, when the queue empties. Carry on with",
|
|
108
|
+
" the next task — file this one as done once it is landed and gated, saying the deploy is owed.",
|
|
109
|
+
` \`autopilot ship\` lists what is owed. To deploy now anyway (an outage fix, the ship itself, or`,
|
|
110
|
+
` the owner asked for it): ${SHIP_NOW_VAR}=1 <the same command>.`,
|
|
111
|
+
].join("\n");
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/**
|
|
115
|
+
* {@link deferral} for the checkout at `cwd` — what a Mac-side `bun run deploy` asks first:
|
|
116
|
+
*
|
|
117
|
+
* const deferred = shipDeferral(appRoot(), "roms production deploy");
|
|
118
|
+
* if (deferred) { console.log(deferred); process.exit(0); }
|
|
119
|
+
*/
|
|
120
|
+
export function shipDeferral(cwd: string, what: string, env: NodeJS.ProcessEnv = process.env): string | null {
|
|
121
|
+
return deferral(what, readShipSetting(forgeState(cwd, env)), env);
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
/** The ledger line a deferred production deploy leaves — what `autopilot ship` counts. */
|
|
125
|
+
export interface ShipEventLine {
|
|
126
|
+
at: string;
|
|
127
|
+
kind: "ship";
|
|
128
|
+
what: "deferred";
|
|
129
|
+
repo: string;
|
|
130
|
+
env: string;
|
|
131
|
+
commit: string;
|
|
132
|
+
dirty: boolean;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
/**
|
|
136
|
+
* Append one JSON line to the generation's ledger (`$FORGE_STATE/state/autopilot/events.jsonl`).
|
|
137
|
+
* Never throws; `false` when it did not write — a ledger that can fail a deploy is worse than none.
|
|
138
|
+
*/
|
|
139
|
+
export function appendLedger(state: string | null, line: object): boolean {
|
|
140
|
+
if (!state) return false;
|
|
141
|
+
try {
|
|
142
|
+
const file = join(state, "state", "autopilot", "events.jsonl");
|
|
143
|
+
mkdirSync(dirname(file), { recursive: true });
|
|
144
|
+
appendFileSync(file, `${JSON.stringify(line)}\n`);
|
|
145
|
+
return true;
|
|
146
|
+
} catch {
|
|
147
|
+
return false;
|
|
148
|
+
}
|
|
149
|
+
}
|
package/src/smoke.ts
CHANGED
|
@@ -83,6 +83,8 @@
|
|
|
83
83
|
import { spawnSync } from "node:child_process";
|
|
84
84
|
import { lookup } from "node:dns/promises";
|
|
85
85
|
import { isIP } from "node:net";
|
|
86
|
+
// The package's own subpath, never `./edgeFetch.ts` — see `publishShape.test.ts`.
|
|
87
|
+
import type { EdgeRequestInit } from "cursedops/edge-fetch";
|
|
86
88
|
|
|
87
89
|
export interface Failure {
|
|
88
90
|
check: string;
|
|
@@ -840,12 +842,18 @@ export interface SteadyOptions {
|
|
|
840
842
|
reads?: number;
|
|
841
843
|
/** Reads before giving up and judging whatever answered. Default 40. */
|
|
842
844
|
tries?: number;
|
|
845
|
+
/**
|
|
846
|
+
* Stop asking after this long, whatever `tries` says. Default 240 s — a read that THROWS can
|
|
847
|
+
* cost a whole curl `--max-time`, so a count alone does not bound the wait.
|
|
848
|
+
*/
|
|
849
|
+
deadlineMs?: number;
|
|
843
850
|
/** Pause after a GOOD read, before the next. Default 2 s. */
|
|
844
851
|
pauseMs?: number;
|
|
845
852
|
/** Pause after a BAD read. Default 5 s. */
|
|
846
853
|
retryMs?: number;
|
|
847
854
|
/** Injected for tests. */
|
|
848
855
|
sleep?: (ms: number) => Promise<void>;
|
|
856
|
+
now?: () => number;
|
|
849
857
|
}
|
|
850
858
|
|
|
851
859
|
export interface SteadyRead<T> {
|
|
@@ -868,6 +876,14 @@ export interface SteadyRead<T> {
|
|
|
868
876
|
* answering `503` (the pre-secrets version) and `404`, because one isolate had the new version
|
|
869
877
|
* and the next did not. The copies in `vault` and `collections` still judged after one read.
|
|
870
878
|
*
|
|
879
|
+
* 🔴 **A read that THROWS is a bad read, not the end of the wait.** Measured 2026-09-29 (task
|
|
880
|
+
* 2291): family's and music's production smokes each went red on their FIRST `/healthz`, straight
|
|
881
|
+
* after the secrets were put — `no answer … (curl exit 28)`, twice 60 s — and both hosts answered
|
|
882
|
+
* in 0.2 s moments later. The edge was between versions; rethrowing that first silence charged a
|
|
883
|
+
* green deploy with a red smoke. A throw now resets the count and is asked again until `tries`
|
|
884
|
+
* or `deadlineMs`; only when the LAST read threw is that error thrown to the caller, so a Worker
|
|
885
|
+
* that never answers is still a red that names why.
|
|
886
|
+
*
|
|
871
887
|
* It never decides pass or fail: it returns what answered, and the caller judges that — a wait
|
|
872
888
|
* that swallowed its own timeout would be a check that cannot go red.
|
|
873
889
|
*/
|
|
@@ -878,22 +894,44 @@ export async function steadyHealth<T = Record<string, unknown>>(
|
|
|
878
894
|
): Promise<SteadyRead<T>> {
|
|
879
895
|
const reads = options.reads ?? 3;
|
|
880
896
|
const tries = options.tries ?? 40;
|
|
897
|
+
const deadlineMs = options.deadlineMs ?? 240_000;
|
|
881
898
|
const sleep = options.sleep ?? ((ms: number) => new Promise<void>((done) => setTimeout(done, ms)));
|
|
899
|
+
const now = options.now ?? Date.now;
|
|
900
|
+
const started = now();
|
|
882
901
|
let steady = 0;
|
|
883
902
|
let asked = 0;
|
|
884
903
|
for (;;) {
|
|
885
|
-
|
|
904
|
+
let read: { response: Response; body: T } | null = null;
|
|
905
|
+
let thrown: unknown = null;
|
|
906
|
+
try {
|
|
907
|
+
const response = await ask();
|
|
908
|
+
const body = (await response
|
|
909
|
+
.clone()
|
|
910
|
+
.json()
|
|
911
|
+
.catch(() => ({}))) as T;
|
|
912
|
+
read = { response, body };
|
|
913
|
+
} catch (error) {
|
|
914
|
+
thrown = error;
|
|
915
|
+
}
|
|
886
916
|
asked++;
|
|
887
|
-
|
|
888
|
-
|
|
889
|
-
|
|
890
|
-
|
|
891
|
-
steady = ready(body, response) ? steady + 1 : 0;
|
|
892
|
-
if (steady >= reads || asked >= tries) return { response, body, steady, asked };
|
|
917
|
+
steady = read && ready(read.body, read.response) ? steady + 1 : 0;
|
|
918
|
+
const out = asked >= tries || now() - started >= deadlineMs;
|
|
919
|
+
if (read && (steady >= reads || out)) return { ...read, steady, asked };
|
|
920
|
+
if (!read && out) throw thrown;
|
|
893
921
|
await sleep(steady > 0 ? (options.pauseMs ?? 2_000) : (options.retryMs ?? 5_000));
|
|
894
922
|
}
|
|
895
923
|
}
|
|
896
924
|
|
|
925
|
+
/**
|
|
926
|
+
* curl's `--max-time` for ONE `/healthz` read of a settling Worker, in seconds. Short on purpose:
|
|
927
|
+
* {@link steadyHealth} asks again after a silence, so a long timeout only makes the edge's
|
|
928
|
+
* between-versions window cost more — the 2026-09-29 reds spent 120 s on two reads of it.
|
|
929
|
+
*/
|
|
930
|
+
export const HEALTH_READ_MAX_TIME_SEC = 10;
|
|
931
|
+
|
|
932
|
+
/** The init a smoke's `/healthz` read carries — `cursedops/edge-fetch` honours it; a plain `fetch` ignores it. */
|
|
933
|
+
const healthRead: EdgeRequestInit = { maxTimeSec: HEALTH_READ_MAX_TIME_SEC };
|
|
934
|
+
|
|
897
935
|
/** One question a deploy must answer the NEW way — `null` when it does, else why not. */
|
|
898
936
|
export interface ConvergenceProbe {
|
|
899
937
|
name: string;
|
|
@@ -1065,7 +1103,7 @@ export async function runWorkerSmoke(spec: WorkerSmokeSpec): Promise<number> {
|
|
|
1065
1103
|
let status: number;
|
|
1066
1104
|
try {
|
|
1067
1105
|
const read = await steadyHealth<Record<string, unknown>>(
|
|
1068
|
-
() => ask("/healthz"),
|
|
1106
|
+
() => ask("/healthz", healthRead),
|
|
1069
1107
|
(body) => body.commit === spec.head && body.apiServed === true,
|
|
1070
1108
|
spec.steady,
|
|
1071
1109
|
);
|
|
@@ -1155,7 +1193,7 @@ async function accessDoorSmoke(
|
|
|
1155
1193
|
let health: Record<string, unknown>;
|
|
1156
1194
|
let status: number;
|
|
1157
1195
|
try {
|
|
1158
|
-
const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz"), (body) => body.commit === spec.head, spec.steady);
|
|
1196
|
+
const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz", healthRead), (body) => body.commit === spec.head, spec.steady);
|
|
1159
1197
|
health = read.body;
|
|
1160
1198
|
status = read.response.status;
|
|
1161
1199
|
} catch (error) {
|
package/src/workerDeploy.ts
CHANGED
|
@@ -53,9 +53,10 @@
|
|
|
53
53
|
* refusal without wrangler, git or a network.
|
|
54
54
|
*/
|
|
55
55
|
import { spawn, spawnSync } from "node:child_process";
|
|
56
|
-
import {
|
|
57
|
-
import {
|
|
56
|
+
import { existsSync, readdirSync, readFileSync } from "node:fs";
|
|
57
|
+
import { join } from "node:path";
|
|
58
58
|
import { forgeState } from "cursedops/roots";
|
|
59
|
+
import { appendLedger, deferral, readShipSetting, type ShipEventLine, type ShipSetting } from "cursedops/ship";
|
|
59
60
|
|
|
60
61
|
/** The two deployments every Worker app here has. */
|
|
61
62
|
export type WorkerEnv = "production" | "stage";
|
|
@@ -318,6 +319,16 @@ export interface DeployDeps {
|
|
|
318
319
|
* A throw here is swallowed: a ledger that can fail a deploy is worse than none.
|
|
319
320
|
*/
|
|
320
321
|
record?: (line: DeployEventLine) => void;
|
|
322
|
+
/**
|
|
323
|
+
* The owner's fast-shipping switch ({@link readShipSetting}). {@link workerDeployDeps} reads the
|
|
324
|
+
* generation's `ship.json`; absent here, the deploy behaves as it always did — so a suite that
|
|
325
|
+
* builds its own deps never reads this Mac's switch.
|
|
326
|
+
*/
|
|
327
|
+
ship?: () => ShipSetting;
|
|
328
|
+
/** Where `FORGE_SHIP_NOW` is read from. Default `process.env`. */
|
|
329
|
+
env?: NodeJS.ProcessEnv;
|
|
330
|
+
/** Where a deferred production deploy's `ship` line goes. {@link workerDeployDeps} appends it to the ledger. */
|
|
331
|
+
recordShip?: (line: ShipEventLine) => void;
|
|
321
332
|
}
|
|
322
333
|
|
|
323
334
|
/**
|
|
@@ -353,15 +364,7 @@ export interface DeployEventLine {
|
|
|
353
364
|
|
|
354
365
|
/** Append one line to the ledger under a state root. Never throws; `false` when it did not write. */
|
|
355
366
|
export function recordDeployEvent(state: string | null, line: DeployEventLine): boolean {
|
|
356
|
-
|
|
357
|
-
try {
|
|
358
|
-
const file = join(state, "state", "autopilot", "events.jsonl");
|
|
359
|
-
mkdirSync(dirname(file), { recursive: true });
|
|
360
|
-
appendFileSync(file, `${JSON.stringify(line)}\n`);
|
|
361
|
-
return true;
|
|
362
|
-
} catch {
|
|
363
|
-
return false;
|
|
364
|
-
}
|
|
367
|
+
return appendLedger(state, line);
|
|
365
368
|
}
|
|
366
369
|
|
|
367
370
|
export interface DeployResult {
|
|
@@ -404,6 +407,10 @@ export function workerDeployDeps(cwd: string): DeployDeps {
|
|
|
404
407
|
record: (line) => {
|
|
405
408
|
recordDeployEvent(forgeState(cwd), line);
|
|
406
409
|
},
|
|
410
|
+
ship: () => readShipSetting(forgeState(cwd)),
|
|
411
|
+
recordShip: (line) => {
|
|
412
|
+
appendLedger(forgeState(cwd), line);
|
|
413
|
+
},
|
|
407
414
|
};
|
|
408
415
|
}
|
|
409
416
|
|
|
@@ -688,6 +695,21 @@ export function runWorkerDeploy(spec: WorkerDeploySpec, deps: DeployDeps): Deplo
|
|
|
688
695
|
const unmeasured: string[] = [];
|
|
689
696
|
|
|
690
697
|
if (!commit) return stop("head", "this checkout has no readable HEAD — there is no commit to stamp, so nothing could prove the deploy.");
|
|
698
|
+
// 🚢 Fast shipping (`./ship.ts`): a production deploy made while the switch is on is owed to the
|
|
699
|
+
// runner's ship instead — asked before anything installs or builds, so deferring costs nothing.
|
|
700
|
+
// Its `ship` line, not a `deploy` one: a deferral is not a deploy, and would drag every median.
|
|
701
|
+
if (spec.env === "production" && deps.ship) {
|
|
702
|
+
const deferred = deferral(`${spec.app} production deploy`, deps.ship(), deps.env ?? process.env);
|
|
703
|
+
if (deferred) {
|
|
704
|
+
log(`\n${deferred}`);
|
|
705
|
+
try {
|
|
706
|
+
deps.recordShip?.({ at: new Date(started).toISOString(), kind: "ship", what: "deferred", repo: spec.app, env: spec.env, commit: commit.slice(0, 8), dirty });
|
|
707
|
+
} catch {
|
|
708
|
+
// never the deploy's verdict — see DeployDeps.record
|
|
709
|
+
}
|
|
710
|
+
return { code: 0, step: "deferred", detail: deferred, commit, dirty };
|
|
711
|
+
}
|
|
712
|
+
}
|
|
691
713
|
if (spec.env === "production" && dirty) {
|
|
692
714
|
return stop("clean-tree", `the tree is dirty — production would report ${commit.slice(0, 8)} and run something else:\n${porcelain}`);
|
|
693
715
|
}
|
package/src/workerSecrets.ts
CHANGED
|
@@ -92,6 +92,12 @@ export interface SecretSync {
|
|
|
92
92
|
/** Tries per put. A put is idempotent, so a retry can only turn a network blip into a success. */
|
|
93
93
|
export const SECRET_PUT_ATTEMPTS = 3;
|
|
94
94
|
|
|
95
|
+
/**
|
|
96
|
+
* Reads of the account after the puts before a set that is not exact is believed. A read changes
|
|
97
|
+
* nothing, so asking again can only replace a stale answer with a current one — see {@link syncSecrets}.
|
|
98
|
+
*/
|
|
99
|
+
export const SECRET_READBACK_ATTEMPTS = 5;
|
|
100
|
+
|
|
95
101
|
/**
|
|
96
102
|
* Apply `plan`: put every value, delete every extra, then READ THE ACCOUNT BACK and fail unless it
|
|
97
103
|
* holds exactly the set. Refuses without touching anything when the plan has `absent` keys.
|
|
@@ -100,11 +106,17 @@ export const SECRET_PUT_ATTEMPTS = 3;
|
|
|
100
106
|
* `wrangler secret put` of three hit `fetch failed` and the whole deploy stopped with
|
|
101
107
|
* collections-stage holding a Worker and no secrets — serving nothing — for a blip the next
|
|
102
108
|
* attempt would have cleared. The read-back is still the verdict.
|
|
109
|
+
*
|
|
110
|
+
* 🔴 **The read-back is asked again, up to {@link SECRET_READBACK_ATTEMPTS} times, before a set
|
|
111
|
+
* that is not exact is believed.** Measured 2026-09-29 on family-stage (task 2291): the account
|
|
112
|
+
* listed both secrets, both puts said ok, and the list read straight after answered NOTHING — a
|
|
113
|
+
* green deploy stopped as "secrets NOT attached". The same script run again a minute later read
|
|
114
|
+
* both. The LAST read is the verdict, so a set that is really wrong is still a red.
|
|
103
115
|
*/
|
|
104
116
|
export async function syncSecrets(
|
|
105
117
|
plan: SecretPlan,
|
|
106
118
|
ops: SecretOps,
|
|
107
|
-
options: { attempts?: number; pause?: (ms: number) => Promise<void> } = {},
|
|
119
|
+
options: { attempts?: number; readbacks?: number; pause?: (ms: number) => Promise<void> } = {},
|
|
108
120
|
): Promise<SecretSync> {
|
|
109
121
|
const attempts = Math.max(1, options.attempts ?? SECRET_PUT_ATTEMPTS);
|
|
110
122
|
const pause = options.pause ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)));
|
|
@@ -130,9 +142,20 @@ export async function syncSecrets(
|
|
|
130
142
|
lines.push(`DELETE ${name} — not in the set — ${ok ? "gone" : "FAILED"}`);
|
|
131
143
|
if (!ok) bad++;
|
|
132
144
|
}
|
|
133
|
-
const
|
|
134
|
-
const
|
|
135
|
-
|
|
145
|
+
const readbacks = Math.max(1, options.readbacks ?? SECRET_READBACK_ATTEMPTS);
|
|
146
|
+
const isExact = (held: readonly string[]) => held.length === plan.wanted.length && plan.wanted.every((name) => held.includes(name));
|
|
147
|
+
let after = await ops.held();
|
|
148
|
+
let reads = 1;
|
|
149
|
+
while (!isExact(after) && reads < readbacks) {
|
|
150
|
+
await pause(2_000 * reads);
|
|
151
|
+
after = await ops.held();
|
|
152
|
+
reads++;
|
|
153
|
+
}
|
|
154
|
+
const exact = isExact(after);
|
|
155
|
+
const reread = reads > 1 ? ` (read ${reads} times)` : "";
|
|
156
|
+
lines.push(
|
|
157
|
+
exact ? `holds exactly: ${after.join(", ")}${reread}` : `🔴 holds ${after.join(", ") || "(nothing)"} — not exactly ${plan.wanted.join(", ")}${reread}`,
|
|
158
|
+
);
|
|
136
159
|
return { ok: exact && bad === 0, lines, after };
|
|
137
160
|
}
|
|
138
161
|
|