cursedops 0.11.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -4
- package/package.json +1 -1
- package/src/backups.ts +25 -0
- package/src/edgeFetch.ts +8 -2
- package/src/ship.ts +161 -8
- package/src/smoke.ts +47 -9
- package/src/workerDeploy.ts +10 -3
- package/src/workerSecrets.ts +27 -4
package/README.md
CHANGED
|
@@ -14,18 +14,18 @@ bun add cursedops
|
|
|
14
14
|
| `cursedops/smoke` | the scaffolding of a deployed smoke — the ledger, the fetch, the DNS hint, the exit code — the one check no app owns (every address of a deployment serving the same built client), and since 0.5.0 its VERDICTS: the origin asked on loopback, the smoke's own environment, a network that lies about DNS, and a version that has settled |
|
|
15
15
|
| `cursedops/worker-deploy` | the sequence that ships a Cloudflare Worker — clean tree, stage first, schema, `--var` stamp, secrets, smoke — and the readers it needs (0.5.0); `readWranglerFile(root)` since 0.9.0, lifted from four apps' `readWrangler` (task 2153); since 0.10.25 every run, red or green, appends a `kind: "deploy"` line with each leg's wall clock (`install`, `stage`, `walk`, `build`, `schema`, `upload`, `secrets`, `smoke`) to the generation's ledger (task 2246) |
|
|
16
16
|
| `cursedops/worker-deploy` (CPU) | a `wrangler tail` around the traffic a deploy already sends, each request's BILLED CPU charged to its declared route, red on an undeclared, over-budget or uncapped route — and `runWorkerDeploy`'s `stageWalk` + `cpuTail` that wrap the walk and the smoke in it (0.7.0, task 2134, lifted from collections) |
|
|
17
|
-
| `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) |
|
|
17
|
+
| `cursedops/worker-secrets` | a Worker holding EXACTLY its deployment's secrets, uploaded over a pipe, read back (0.5.0) — and read again, up to five times, before a set that is not exact is believed: the list lags the puts (0.11.1) |
|
|
18
18
|
| `cursedops/worker-rollback` | a hostname back on its Mac origin: origin first, route second, the route found rather than typed (0.5.0) |
|
|
19
|
-
| `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0) |
|
|
19
|
+
| `cursedops/edge-fetch` | a request to a deployed Worker as a script must make it — curl pinned past the Mac's negative DNS cache (0.5.0); a request's own `maxTimeSec` over the instance's (0.11.1) |
|
|
20
20
|
| `cursedops/d1-import` | a cutover's data proof — SQL literals and the row-for-row comparison, never a count (0.5.0); and `d1SchemaText`, the generated `db/schema.sql` that refuses a statement spanning lines, because D1's exec splits on newlines (0.9.0, lifted from five apps, task 2153) — and its `forge-d1-schema` bin (0.9.1), so no app carries its own schema script at all |
|
|
21
|
-
| `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153) |
|
|
21
|
+
| `cursedops/backups` | `BACKUP_KEEP`, the fleet's one retention for dated snapshots — six apps each exported `KEEP = 14` (0.9.0, task 2153); `BACKUP_STAMP_PATTERN`, `backupStamp` and its inverse `stampTime` — the local-time run name every backup script writes and station's audit reads (0.11.2) |
|
|
22
22
|
| `cursedops/serve` | the static tier's four helpers — the path-traversal guard, the MIME table, the hashed-asset test, the crash handlers |
|
|
23
23
|
| `cursedops/api-floor` | the rule that an unmatched `/api/...` is a phrase and never the app shell — the namespace predicates, the trailing-slash normaliser and the default 404 body. No `node:` import, so it mounts inside a Worker |
|
|
24
24
|
| `cursedops/build-info` | which commit a checkout-served process is running and whether its tree was dirty — read once at load, and a `null` retried in the background rather than cached for the life of the process. `node:child_process`, so never in a Worker |
|
|
25
25
|
| `cursedops/bound-lists` | the static check that no SQL builds an `IN (…)` list one `?` per value — the shape D1 500s on past 100 ids. Test-time only; the recorded third exception, see below |
|
|
26
26
|
| `cursedops/staged-client` | the IMMUTABLE client a checkout-served app serves — a build staged into `<APP_DATA_DIR>/client/<commit>/` behind `CURRENT`, so another agent's `vite build` in the checkout never changes live bytes — and its `forge-client stamp` / `forge-client stage` bin (0.6.0, lifted from family, flix, roms, station) |
|
|
27
27
|
| `cursedops/deploy-tree` | the two refusals a checkout-served deploy needs — never deploy a dirty tree, never `git revert` in one (`readTreeStatus`, `refuseToDeploy`, `refuseToRevert`; 0.6.0, lifted from nine apps) |
|
|
28
|
-
| `cursedops/ship` | the owner's FAST SHIPPING switch as every deploy reads it — `readShipSetting` (`$FORGE_STATE/state/autopilot/ship.json`; absent or unreadable is `fast`, no generation is `each`), `deferral`/`shipDeferral` (the sentence a production deploy prints instead of deploying, or `null`), `FORGE_SHIP_NOW=1` to ship now whatever it says. `runWorkerDeploy` asks it before installing anything; a Mac-side `bun run deploy` calls `shipDeferral` first (0.11.0) |
|
|
28
|
+
| `cursedops/ship` | the owner's FAST SHIPPING switch as every deploy reads it — `readShipSetting` (`$FORGE_STATE/state/autopilot/ship.json`; absent or unreadable is `fast`, no generation is `each`), `deferral`/`shipDeferral` (the sentence a production deploy prints instead of deploying, or `null`), `FORGE_SHIP_NOW=1` to ship now whatever it says. `runWorkerDeploy` asks it before installing anything; a Mac-side `bun run deploy` calls `shipDeferral` first (0.11.0). Since 0.12.0 `shipNowVerdict` honours ship-now under `$AUTOPILOT_BATCH` only for a batch holding a live claim on the ship task or a `prod-down:` incident, with `FORGE_SHIP_NOW_WHY` set, and every ask is a `kind: "ship-now"` ledger row (task 2384) |
|
|
29
29
|
| `cursedops/relay` | the WORKER half of a relay — a Worker in front of a Mac-bound app over ONE outbound WebSocket: the frames (station's wire, byte for byte), the key-digest door (`acceptLink`), and `RelayLink`, the Durable Object an app subclasses with its name, body ceiling, timeout and (optionally) station's per-cookie offline cache. No `node:` import (0.10.0, lifted from station for roms, task 065) |
|
|
30
30
|
| `cursedops/relay-link` | the MAC half — `dialLink` (answer each frame with the app's own `fetch`, redial 1 s → 60 s on close, no timer otherwise), `readLinkSettings` for `<APP>_LINK_URL`/`<APP>_LINK_KEY`, and `rotateLinkKey` (digest to the Worker FIRST, the key to the 0600 file only if it took) (0.10.0) |
|
|
31
31
|
| `cursedops/relay-stage` | the signed-in STAGE WALK's skeleton both relay apps shared — `stageByteCheck` (SHA-256 of every built file, a stale byte re-asked 4 × 10 s), `putStageLinkDigest` (`--env stage` only), `grantStageOwner` (auth-stage's 2FA grant, never production's), `oldKeyRefused` (the rotation leg: 20 s refused window, 3 min ceiling), `awaitSystemResolver`, `servedPath`, `STAGE_AUTH_URL`. With `relayWorkerDeployCli` in `cursedops/worker-deploy`, a relay app's whole `worker:deploy` as data (0.10.6) |
|
|
@@ -304,6 +304,12 @@ pins an address** — the deliverable is the distinction, not a way around the n
|
|
|
304
304
|
`steadyHealth(ask, ready)` is the Worker smoke's wait: `ready` must hold for THREE reads in a
|
|
305
305
|
row, because on `music`'s first deploy (2026-09-23, `a54ef71`) one good `/healthz` was followed
|
|
306
306
|
by thirteen routes answering the pre-secrets version. It returns what answered and never judges.
|
|
307
|
+
A read that THROWS (0.11.1) is a bad read asked again, bounded by `deadlineMs` (240 s), and only
|
|
308
|
+
a LAST read that threw is thrown to the caller: the 2026-09-29 ship reported family and music red
|
|
309
|
+
on a first `/healthz` that timed out twice while the edge was between versions (task 2291).
|
|
310
|
+
`runWorkerSmoke` asks each of those reads with a 10 s curl `--max-time`
|
|
311
|
+
(`HEALTH_READ_MAX_TIME_SEC`, passed as `EdgeRequestInit.maxTimeSec`), so the window costs 20 s a
|
|
312
|
+
read instead of 120.
|
|
307
313
|
|
|
308
314
|
`awaitConverged(probes)` (0.10.19) is the same wait over SEVERAL paths: every probe must agree
|
|
309
315
|
three rounds in a row, bounded (120 s), before any leg is judged — a Worker's code and its static
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cursedops",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.12.0",
|
|
4
4
|
"description": "The build-and-ops answers this generation's apps wrote independently and identically: finding a generation's roots — and printing a command that runs when pasted — without knowing a path, the generation's whole-tree laws run over one repo from a checkout or a worktree, macOS launchd agent install/replace/remove and the live port a job serves, the scaffolding and verdicts of a deployed smoke (origin probe, the smoke's own environment, a network that lies about DNS, a settled version), the static-serving helpers eight apps copied — the path-traversal guard among them — the API floor that keeps an unmatched /api/... from ever being answered with the app shell, the commit and dirty flag a checkout-served process reports, the Cloudflare Worker deploy toolkit four apps copied (the deploy sequence, exact-set secrets over a pipe, origin-first rollback, the curl edge fetch, the row-for-row D1 import proof, the billed-CPU tail check around a deploy's walk and smoke, and each app's worker:secrets and worker:smoke main as one function of its data), the relay a Worker fronts a Mac-bound app with (the Durable Object, the frames, the Mac's dialer and key rotation — lifted from station for roms — and the signed-in stage walk's skeleton and the relay app's whole worker:deploy), the Worker import-graph and await-port checks every Worker app's suite runs over its own source, and the public-surface ratchet three published libraries each carried a forked copy of. Mechanism only — no app knows its name from here. Bun, zero runtime dependencies (typescript is an optional peer, for public-surface only), ships source.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"scripts": {
|
package/src/backups.ts
CHANGED
|
@@ -7,3 +7,28 @@
|
|
|
7
7
|
* whatever was decided. So it is one number here, and each app's prune takes it.
|
|
8
8
|
*/
|
|
9
9
|
export const BACKUP_KEEP = 14;
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* The name of one dated run: `YYYYMMDD-HHMMSS`, in the Mac's LOCAL time.
|
|
13
|
+
*
|
|
14
|
+
* 🔴 A contract with two sides, so it is spelled once. Every app's own backup script WRITES
|
|
15
|
+
* these directory names and prunes only what matches the pattern; station's backup audit READS
|
|
16
|
+
* them back into a time. Until cursedops 0.11.2 the writers held nine copies of the pattern and
|
|
17
|
+
* two of the stamp, and the reader held its own pair that wrote and read the stamp as UTC — so a
|
|
18
|
+
* run the apps named `20260929-100000` at 10:00 CDT was read as 10:00Z, five hours off.
|
|
19
|
+
*/
|
|
20
|
+
export const BACKUP_STAMP_PATTERN = /^\d{8}-\d{6}$/;
|
|
21
|
+
|
|
22
|
+
const pad = (n: number): string => String(n).padStart(2, "0");
|
|
23
|
+
|
|
24
|
+
/** A `Date` → its run name, in local time. {@link stampTime} is its inverse. */
|
|
25
|
+
export const backupStamp = (d: Date): string =>
|
|
26
|
+
`${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}`;
|
|
27
|
+
|
|
28
|
+
/** A run name → epoch ms, read as local time — or `null` for anything that is not one. */
|
|
29
|
+
export function stampTime(stamp: string): number | null {
|
|
30
|
+
if (!BACKUP_STAMP_PATTERN.test(stamp)) return null;
|
|
31
|
+
const n = (from: number, to: number): number => Number(stamp.slice(from, to));
|
|
32
|
+
const at = new Date(n(0, 4), n(4, 6) - 1, n(6, 8), n(9, 11), n(11, 13), n(13, 15));
|
|
33
|
+
return Number.isNaN(at.getTime()) ? null : at.getTime();
|
|
34
|
+
}
|
package/src/edgeFetch.ts
CHANGED
|
@@ -58,6 +58,12 @@ export interface EdgeFetchOptions {
|
|
|
58
58
|
retryDelayMs?: (retry: number) => number;
|
|
59
59
|
}
|
|
60
60
|
|
|
61
|
+
/**
|
|
62
|
+
* A `RequestInit` this fetch reads one more field from: `maxTimeSec`, curl's `--max-time` for
|
|
63
|
+
* THIS request, over the instance's {@link EdgeFetchOptions.maxTimeSec}. A plain `fetch` ignores it.
|
|
64
|
+
*/
|
|
65
|
+
export type EdgeRequestInit = RequestInit & { maxTimeSec?: number };
|
|
66
|
+
|
|
61
67
|
/**
|
|
62
68
|
* 🔴 How many MORE times a request is sent after a connection-level curl failure that carried
|
|
63
69
|
* no answer. Measured 2026-09-24: vault's prod deploy of `953e19a` reported "smoke FAILED" on one
|
|
@@ -125,7 +131,7 @@ export function parseCurlResponse(raw: Buffer, origin = "the edge", exit: number
|
|
|
125
131
|
}
|
|
126
132
|
|
|
127
133
|
/** A `fetch`-shaped function over curl — see the header. Each host is resolved once per instance — once it has an answer. */
|
|
128
|
-
export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?:
|
|
134
|
+
export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, init?: EdgeRequestInit) => Promise<Response> {
|
|
129
135
|
const pinned = new Map<string, string>();
|
|
130
136
|
const resolve = options.resolve === undefined ? digCloudflare : options.resolve;
|
|
131
137
|
const curl =
|
|
@@ -155,7 +161,7 @@ export function createEdgeFetch(options: EdgeFetchOptions = {}): (url: string, i
|
|
|
155
161
|
const port = target.port || (target.protocol === "http:" ? "80" : "443");
|
|
156
162
|
const head = () => {
|
|
157
163
|
const ip = pinFor(target.hostname);
|
|
158
|
-
return ["-s", "-i", "--max-time", String(options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
|
|
164
|
+
return ["-s", "-i", "--max-time", String(init.maxTimeSec ?? options.maxTimeSec ?? 60), ...(ip ? ["--resolve", `${target.hostname}:${port}:${ip}`] : [])];
|
|
159
165
|
};
|
|
160
166
|
const args: string[] = [];
|
|
161
167
|
for (const [k, v] of Object.entries(options.headersFor?.(target.hostname) ?? {})) args.push("-H", `${k}: ${v}`);
|
package/src/ship.ts
CHANGED
|
@@ -38,10 +38,17 @@
|
|
|
38
38
|
* deploys, an outage fix, and a deploy the owner asked for in words (his sentence beats a switch he
|
|
39
39
|
* flipped yesterday — the precedence the retired `fastMode.ts` settled on 2026-08-13). A stage
|
|
40
40
|
* deploy is never deferred: it IS the fast way to see a change on a real edge.
|
|
41
|
+
*
|
|
42
|
+
* 🔴 **Under a batch (`$AUTOPILOT_BATCH` set) it is honoured only for the ship task or an incident**
|
|
43
|
+
* ({@link shipNowVerdict}), and only with a reason in `FORGE_SHIP_NOW_WHY`. Until task 2384 any agent
|
|
44
|
+
* could prefix one variable and turn the owner's switch off for itself — the thing
|
|
45
|
+
* `~/.claude/CLAUDE.md` §2 says an agent may never do — and no record said it had. Now every ask,
|
|
46
|
+
* honoured or refused, is a `kind: "ship-now"` ledger row, and `autopilot doctor` lists the ones
|
|
47
|
+
* with no reason. In the owner's own shell (no batch) it is honoured exactly as before, and recorded.
|
|
41
48
|
*/
|
|
42
|
-
import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
|
|
49
|
+
import { appendFileSync, existsSync, mkdirSync, readdirSync, readFileSync } from "node:fs";
|
|
43
50
|
import { dirname, join } from "node:path";
|
|
44
|
-
import { forgeState } from "cursedops/roots";
|
|
51
|
+
import { forgeCodeRoot, forgeState } from "cursedops/roots";
|
|
45
52
|
|
|
46
53
|
export const SHIP_MODES = ["fast", "each"] as const;
|
|
47
54
|
export type ShipMode = (typeof SHIP_MODES)[number];
|
|
@@ -85,12 +92,152 @@ export function readShipSetting(state: string | null): ShipSetting {
|
|
|
85
92
|
}
|
|
86
93
|
}
|
|
87
94
|
|
|
88
|
-
/** Is `FORGE_SHIP_NOW` set to something that means yes? */
|
|
95
|
+
/** Is `FORGE_SHIP_NOW` set to something that means yes? Asked is not honoured — see {@link shipNowVerdict}. */
|
|
89
96
|
export function shipNowAsked(env: NodeJS.ProcessEnv = process.env): boolean {
|
|
90
97
|
const v = env[SHIP_NOW_VAR]?.trim().toLowerCase();
|
|
91
98
|
return v === "1" || v === "true" || v === "yes";
|
|
92
99
|
}
|
|
93
100
|
|
|
101
|
+
/** Why ship-now was asked — required under a batch, recorded always. */
|
|
102
|
+
export const SHIP_NOW_WHY_VAR = "FORGE_SHIP_NOW_WHY";
|
|
103
|
+
|
|
104
|
+
/** The variable the runner sets on its batch agent — present means an agent, not the owner. */
|
|
105
|
+
export const BATCH_VAR = "AUTOPILOT_BATCH";
|
|
106
|
+
|
|
107
|
+
/**
|
|
108
|
+
* The `escalation-key:` values a claimed task may carry for its batch to ship now: autopilot's
|
|
109
|
+
* `SHIP_KEY` (`ship.ts`) and the prod watch's `prod-down:<host>` (`prod.ts`). The file shape is
|
|
110
|
+
* the interface, exactly as `ship.json` is — this module never imports the runner.
|
|
111
|
+
*/
|
|
112
|
+
export const SHIP_NOW_KEYS: readonly RegExp[] = [/^ship$/, /^prod-down:\S+$/];
|
|
113
|
+
|
|
114
|
+
export interface ShipNowVerdict {
|
|
115
|
+
/** `FORGE_SHIP_NOW` means yes. */
|
|
116
|
+
asked: boolean;
|
|
117
|
+
/** Ship now: asked, and (outside a batch, or on a claim that earns it). */
|
|
118
|
+
honoured: boolean;
|
|
119
|
+
/** One sentence: what made it honoured or refused. */
|
|
120
|
+
why: string;
|
|
121
|
+
/** `FORGE_SHIP_NOW_WHY`, trimmed — `""` when not given. */
|
|
122
|
+
reason: string;
|
|
123
|
+
batch?: string;
|
|
124
|
+
/** The claimed task file that earned it. */
|
|
125
|
+
task?: string;
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
interface BoardClaim {
|
|
129
|
+
kind?: unknown;
|
|
130
|
+
pid?: unknown;
|
|
131
|
+
task?: unknown;
|
|
132
|
+
batch?: unknown;
|
|
133
|
+
expiresAt?: unknown;
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
function pidAlive(pid: number): boolean {
|
|
137
|
+
try {
|
|
138
|
+
process.kill(pid, 0);
|
|
139
|
+
return true;
|
|
140
|
+
} catch (err) {
|
|
141
|
+
return (err as NodeJS.ErrnoException).code === "EPERM";
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
/** The `escalation-key:` in a task file's front matter, lower-cased, or `null`. */
|
|
146
|
+
export function escalationKey(text: string): string | null {
|
|
147
|
+
const head = /^---\n([\s\S]*?)\n---/.exec(text.replace(/\r\n/g, "\n"));
|
|
148
|
+
const m = head ? /^escalation-key:\s*(.+?)\s*$/im.exec(head[1] ?? "") : null;
|
|
149
|
+
return m?.[1] ? m[1].toLowerCase() : null;
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
export interface ShipNowWhere {
|
|
153
|
+
/** `$FORGE_STATE` — the board lives at `state/board/`. */
|
|
154
|
+
state: string | null;
|
|
155
|
+
/** `$FORGE` — claimed task files live at `tasks/In_Progress/`. */
|
|
156
|
+
forge: string | null;
|
|
157
|
+
now?: number;
|
|
158
|
+
isAlive?: (pid: number) => boolean;
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/**
|
|
162
|
+
* Is `FORGE_SHIP_NOW` honoured here? Outside a batch, whenever it is asked: that is the owner's own
|
|
163
|
+
* shell. Under `$AUTOPILOT_BATCH`, only when that batch holds a live `kind: "task"` board claim
|
|
164
|
+
* (`$FORGE_STATE/state/board/*.json`, the pid alive and not expired) on a task in
|
|
165
|
+
* `$FORGE/tasks/In_Progress/` whose `escalation-key:` is one of {@link SHIP_NOW_KEYS}, AND
|
|
166
|
+
* `FORGE_SHIP_NOW_WHY` says why. Anything it cannot read is a refusal — a refused ship-now only
|
|
167
|
+
* makes the deploy later, which is the safe direction.
|
|
168
|
+
*/
|
|
169
|
+
export function shipNowVerdict(where: ShipNowWhere, env: NodeJS.ProcessEnv = process.env): ShipNowVerdict {
|
|
170
|
+
const asked = shipNowAsked(env);
|
|
171
|
+
const reason = env[SHIP_NOW_WHY_VAR]?.trim() ?? "";
|
|
172
|
+
const batch = env[BATCH_VAR]?.trim() || undefined;
|
|
173
|
+
const base = { asked, reason, ...(batch ? { batch } : {}) };
|
|
174
|
+
if (!asked) return { ...base, honoured: false, why: `${SHIP_NOW_VAR} is not set` };
|
|
175
|
+
if (!batch) return { ...base, honoured: true, why: `no ${BATCH_VAR} — the owner's own shell` };
|
|
176
|
+
const refuse = (why: string): ShipNowVerdict => ({ ...base, honoured: false, why });
|
|
177
|
+
if (!reason) return refuse(`under batch ${batch}, ${SHIP_NOW_VAR} needs ${SHIP_NOW_WHY_VAR}="<why this cannot wait for the ship>"`);
|
|
178
|
+
if (!where.state || !where.forge) return refuse(`under batch ${batch}, no generation to read the board from`);
|
|
179
|
+
const board = join(where.state, "state", "board");
|
|
180
|
+
const now = where.now ?? Date.now();
|
|
181
|
+
const isAlive = where.isAlive ?? pidAlive;
|
|
182
|
+
let files: string[] = [];
|
|
183
|
+
try {
|
|
184
|
+
files = readdirSync(board).filter((f) => f.startsWith("task-") && f.endsWith(".json"));
|
|
185
|
+
} catch {
|
|
186
|
+
/* no board → no claim */
|
|
187
|
+
}
|
|
188
|
+
const held: string[] = [];
|
|
189
|
+
for (const f of files) {
|
|
190
|
+
let c: BoardClaim;
|
|
191
|
+
try {
|
|
192
|
+
c = JSON.parse(readFileSync(join(board, f), "utf8")) as BoardClaim;
|
|
193
|
+
} catch {
|
|
194
|
+
continue;
|
|
195
|
+
}
|
|
196
|
+
if (c.kind !== "task" || c.batch !== batch || typeof c.task !== "string" || typeof c.pid !== "number") continue;
|
|
197
|
+
if (typeof c.expiresAt === "string" && Date.parse(c.expiresAt) < now) continue;
|
|
198
|
+
if (!isAlive(c.pid)) continue;
|
|
199
|
+
held.push(c.task);
|
|
200
|
+
let key: string | null = null;
|
|
201
|
+
try {
|
|
202
|
+
key = escalationKey(readFileSync(join(where.forge, "tasks", "In_Progress", c.task), "utf8"));
|
|
203
|
+
} catch {
|
|
204
|
+
continue;
|
|
205
|
+
}
|
|
206
|
+
if (key && SHIP_NOW_KEYS.some((k) => k.test(key))) return { ...base, honoured: true, why: `batch ${batch} holds ${c.task} (escalation-key: ${key})`, task: c.task };
|
|
207
|
+
}
|
|
208
|
+
return refuse(
|
|
209
|
+
`batch ${batch} holds ${held.length ? held.join(", ") : "no live task claim"} — ${SHIP_NOW_VAR} is honoured under a batch only for the ship task or a prod incident (escalation-key: ship | prod-down:<host>)`,
|
|
210
|
+
);
|
|
211
|
+
}
|
|
212
|
+
|
|
213
|
+
/** The ledger line every ship-now ask leaves — honoured or refused. */
|
|
214
|
+
export interface ShipNowEventLine {
|
|
215
|
+
at: string;
|
|
216
|
+
kind: "ship-now";
|
|
217
|
+
/** The deploy or publish it was asked for. */
|
|
218
|
+
what: string;
|
|
219
|
+
ok: boolean;
|
|
220
|
+
reason: string;
|
|
221
|
+
detail: string;
|
|
222
|
+
batch?: string;
|
|
223
|
+
task?: string;
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
/** Record one ship-now ask in the ledger. Never throws. */
|
|
227
|
+
export function recordShipNow(state: string | null, what: string, v: ShipNowVerdict, at: Date = new Date()): boolean {
|
|
228
|
+
const line: ShipNowEventLine = {
|
|
229
|
+
at: at.toISOString(),
|
|
230
|
+
kind: "ship-now",
|
|
231
|
+
what,
|
|
232
|
+
ok: v.honoured,
|
|
233
|
+
reason: v.reason,
|
|
234
|
+
detail: v.why,
|
|
235
|
+
...(v.batch ? { batch: v.batch } : {}),
|
|
236
|
+
...(v.task ? { task: v.task } : {}),
|
|
237
|
+
};
|
|
238
|
+
return appendLedger(state, line);
|
|
239
|
+
}
|
|
240
|
+
|
|
94
241
|
/**
|
|
95
242
|
* The sentence a production deploy prints INSTEAD of deploying, or `null` to deploy now.
|
|
96
243
|
*
|
|
@@ -98,16 +245,21 @@ export function shipNowAsked(env: NodeJS.ProcessEnv = process.env): boolean {
|
|
|
98
245
|
* and the runner's ship task — filed when the queue empties or the oldest owed commit is too old —
|
|
99
246
|
* deploys every app that is behind, once. `autopilot ship` lists what is owed.
|
|
100
247
|
*/
|
|
101
|
-
export function deferral(what: string, setting: ShipSetting, env: NodeJS.ProcessEnv = process.env): string | null {
|
|
102
|
-
if (setting.mode !== "fast"
|
|
248
|
+
export function deferral(what: string, setting: ShipSetting, env: NodeJS.ProcessEnv = process.env, where: ShipNowWhere = { state: null, forge: null }): string | null {
|
|
249
|
+
if (setting.mode !== "fast") return null;
|
|
250
|
+
const now = shipNowVerdict(where, env);
|
|
251
|
+
if (now.asked) recordShipNow(where.state, what, now);
|
|
252
|
+
if (now.honoured) return null;
|
|
253
|
+
const refused = now.asked ? [`🔴 ${SHIP_NOW_VAR} REFUSED: ${now.why}.`] : [];
|
|
103
254
|
const by = setting.setBy === "default" ? "the default" : `set by ${setting.setBy}${setting.at ? ` at ${setting.at}` : ""}`;
|
|
104
255
|
return [
|
|
256
|
+
...refused,
|
|
105
257
|
`🚢 ${what}: DEFERRED — fast shipping is on (${by}${setting.unreadable ? `; ship.json unreadable: ${setting.unreadable}` : ""}).`,
|
|
106
258
|
" Nothing was deployed, and that is the intended result, not a failure: this app is now owed to",
|
|
107
259
|
" the ship, which deploys every app behind its HEAD once, when the queue empties. Carry on with",
|
|
108
260
|
" the next task — file this one as done once it is landed and gated, saying the deploy is owed.",
|
|
109
|
-
` \`autopilot ship\` lists what is owed. To deploy now anyway (
|
|
110
|
-
` the owner
|
|
261
|
+
` \`autopilot ship\` lists what is owed. To deploy now anyway (the ship task or a prod incident`,
|
|
262
|
+
` your batch holds, or the owner in his own shell): ${SHIP_NOW_VAR}=1 ${SHIP_NOW_WHY_VAR}="<why>" <the same command>.`,
|
|
111
263
|
].join("\n");
|
|
112
264
|
}
|
|
113
265
|
|
|
@@ -118,7 +270,8 @@ export function deferral(what: string, setting: ShipSetting, env: NodeJS.Process
|
|
|
118
270
|
* if (deferred) { console.log(deferred); process.exit(0); }
|
|
119
271
|
*/
|
|
120
272
|
export function shipDeferral(cwd: string, what: string, env: NodeJS.ProcessEnv = process.env): string | null {
|
|
121
|
-
|
|
273
|
+
const state = forgeState(cwd, env);
|
|
274
|
+
return deferral(what, readShipSetting(state), env, { state, forge: forgeCodeRoot(cwd, env) });
|
|
122
275
|
}
|
|
123
276
|
|
|
124
277
|
/** The ledger line a deferred production deploy leaves — what `autopilot ship` counts. */
|
package/src/smoke.ts
CHANGED
|
@@ -83,6 +83,8 @@
|
|
|
83
83
|
import { spawnSync } from "node:child_process";
|
|
84
84
|
import { lookup } from "node:dns/promises";
|
|
85
85
|
import { isIP } from "node:net";
|
|
86
|
+
// The package's own subpath, never `./edgeFetch.ts` — see `publishShape.test.ts`.
|
|
87
|
+
import type { EdgeRequestInit } from "cursedops/edge-fetch";
|
|
86
88
|
|
|
87
89
|
export interface Failure {
|
|
88
90
|
check: string;
|
|
@@ -840,12 +842,18 @@ export interface SteadyOptions {
|
|
|
840
842
|
reads?: number;
|
|
841
843
|
/** Reads before giving up and judging whatever answered. Default 40. */
|
|
842
844
|
tries?: number;
|
|
845
|
+
/**
|
|
846
|
+
* Stop asking after this long, whatever `tries` says. Default 240 s — a read that THROWS can
|
|
847
|
+
* cost a whole curl `--max-time`, so a count alone does not bound the wait.
|
|
848
|
+
*/
|
|
849
|
+
deadlineMs?: number;
|
|
843
850
|
/** Pause after a GOOD read, before the next. Default 2 s. */
|
|
844
851
|
pauseMs?: number;
|
|
845
852
|
/** Pause after a BAD read. Default 5 s. */
|
|
846
853
|
retryMs?: number;
|
|
847
854
|
/** Injected for tests. */
|
|
848
855
|
sleep?: (ms: number) => Promise<void>;
|
|
856
|
+
now?: () => number;
|
|
849
857
|
}
|
|
850
858
|
|
|
851
859
|
export interface SteadyRead<T> {
|
|
@@ -868,6 +876,14 @@ export interface SteadyRead<T> {
|
|
|
868
876
|
* answering `503` (the pre-secrets version) and `404`, because one isolate had the new version
|
|
869
877
|
* and the next did not. The copies in `vault` and `collections` still judged after one read.
|
|
870
878
|
*
|
|
879
|
+
* 🔴 **A read that THROWS is a bad read, not the end of the wait.** Measured 2026-09-29 (task
|
|
880
|
+
* 2291): family's and music's production smokes each went red on their FIRST `/healthz`, straight
|
|
881
|
+
* after the secrets were put — `no answer … (curl exit 28)`, twice 60 s — and both hosts answered
|
|
882
|
+
* in 0.2 s moments later. The edge was between versions; rethrowing that first silence charged a
|
|
883
|
+
* green deploy with a red smoke. A throw now resets the count and is asked again until `tries`
|
|
884
|
+
* or `deadlineMs`; only when the LAST read threw is that error thrown to the caller, so a Worker
|
|
885
|
+
* that never answers is still a red that names why.
|
|
886
|
+
*
|
|
871
887
|
* It never decides pass or fail: it returns what answered, and the caller judges that — a wait
|
|
872
888
|
* that swallowed its own timeout would be a check that cannot go red.
|
|
873
889
|
*/
|
|
@@ -878,22 +894,44 @@ export async function steadyHealth<T = Record<string, unknown>>(
|
|
|
878
894
|
): Promise<SteadyRead<T>> {
|
|
879
895
|
const reads = options.reads ?? 3;
|
|
880
896
|
const tries = options.tries ?? 40;
|
|
897
|
+
const deadlineMs = options.deadlineMs ?? 240_000;
|
|
881
898
|
const sleep = options.sleep ?? ((ms: number) => new Promise<void>((done) => setTimeout(done, ms)));
|
|
899
|
+
const now = options.now ?? Date.now;
|
|
900
|
+
const started = now();
|
|
882
901
|
let steady = 0;
|
|
883
902
|
let asked = 0;
|
|
884
903
|
for (;;) {
|
|
885
|
-
|
|
904
|
+
let read: { response: Response; body: T } | null = null;
|
|
905
|
+
let thrown: unknown = null;
|
|
906
|
+
try {
|
|
907
|
+
const response = await ask();
|
|
908
|
+
const body = (await response
|
|
909
|
+
.clone()
|
|
910
|
+
.json()
|
|
911
|
+
.catch(() => ({}))) as T;
|
|
912
|
+
read = { response, body };
|
|
913
|
+
} catch (error) {
|
|
914
|
+
thrown = error;
|
|
915
|
+
}
|
|
886
916
|
asked++;
|
|
887
|
-
|
|
888
|
-
|
|
889
|
-
|
|
890
|
-
|
|
891
|
-
steady = ready(body, response) ? steady + 1 : 0;
|
|
892
|
-
if (steady >= reads || asked >= tries) return { response, body, steady, asked };
|
|
917
|
+
steady = read && ready(read.body, read.response) ? steady + 1 : 0;
|
|
918
|
+
const out = asked >= tries || now() - started >= deadlineMs;
|
|
919
|
+
if (read && (steady >= reads || out)) return { ...read, steady, asked };
|
|
920
|
+
if (!read && out) throw thrown;
|
|
893
921
|
await sleep(steady > 0 ? (options.pauseMs ?? 2_000) : (options.retryMs ?? 5_000));
|
|
894
922
|
}
|
|
895
923
|
}
|
|
896
924
|
|
|
925
|
+
/**
|
|
926
|
+
* curl's `--max-time` for ONE `/healthz` read of a settling Worker, in seconds. Short on purpose:
|
|
927
|
+
* {@link steadyHealth} asks again after a silence, so a long timeout only makes the edge's
|
|
928
|
+
* between-versions window cost more — the 2026-09-29 reds spent 120 s on two reads of it.
|
|
929
|
+
*/
|
|
930
|
+
export const HEALTH_READ_MAX_TIME_SEC = 10;
|
|
931
|
+
|
|
932
|
+
/** The init a smoke's `/healthz` read carries — `cursedops/edge-fetch` honours it; a plain `fetch` ignores it. */
|
|
933
|
+
const healthRead: EdgeRequestInit = { maxTimeSec: HEALTH_READ_MAX_TIME_SEC };
|
|
934
|
+
|
|
897
935
|
/** One question a deploy must answer the NEW way — `null` when it does, else why not. */
|
|
898
936
|
export interface ConvergenceProbe {
|
|
899
937
|
name: string;
|
|
@@ -1065,7 +1103,7 @@ export async function runWorkerSmoke(spec: WorkerSmokeSpec): Promise<number> {
|
|
|
1065
1103
|
let status: number;
|
|
1066
1104
|
try {
|
|
1067
1105
|
const read = await steadyHealth<Record<string, unknown>>(
|
|
1068
|
-
() => ask("/healthz"),
|
|
1106
|
+
() => ask("/healthz", healthRead),
|
|
1069
1107
|
(body) => body.commit === spec.head && body.apiServed === true,
|
|
1070
1108
|
spec.steady,
|
|
1071
1109
|
);
|
|
@@ -1155,7 +1193,7 @@ async function accessDoorSmoke(
|
|
|
1155
1193
|
let health: Record<string, unknown>;
|
|
1156
1194
|
let status: number;
|
|
1157
1195
|
try {
|
|
1158
|
-
const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz"), (body) => body.commit === spec.head, spec.steady);
|
|
1196
|
+
const read = await steadyHealth<Record<string, unknown>>(() => ask("/healthz", healthRead), (body) => body.commit === spec.head, spec.steady);
|
|
1159
1197
|
health = read.body;
|
|
1160
1198
|
status = read.response.status;
|
|
1161
1199
|
} catch (error) {
|
package/src/workerDeploy.ts
CHANGED
|
@@ -55,8 +55,8 @@
|
|
|
55
55
|
import { spawn, spawnSync } from "node:child_process";
|
|
56
56
|
import { existsSync, readdirSync, readFileSync } from "node:fs";
|
|
57
57
|
import { join } from "node:path";
|
|
58
|
-
import { forgeState } from "cursedops/roots";
|
|
59
|
-
import { appendLedger, deferral, readShipSetting, type ShipEventLine, type ShipSetting } from "cursedops/ship";
|
|
58
|
+
import { forgeCodeRoot, forgeState } from "cursedops/roots";
|
|
59
|
+
import { appendLedger, deferral, readShipSetting, type ShipEventLine, type ShipNowWhere, type ShipSetting } from "cursedops/ship";
|
|
60
60
|
|
|
61
61
|
/** The two deployments every Worker app here has. */
|
|
62
62
|
export type WorkerEnv = "production" | "stage";
|
|
@@ -327,6 +327,12 @@ export interface DeployDeps {
|
|
|
327
327
|
ship?: () => ShipSetting;
|
|
328
328
|
/** Where `FORGE_SHIP_NOW` is read from. Default `process.env`. */
|
|
329
329
|
env?: NodeJS.ProcessEnv;
|
|
330
|
+
/**
|
|
331
|
+
* The generation roots `FORGE_SHIP_NOW` is judged against under a batch — the board, the claimed
|
|
332
|
+
* task, and the ledger its `ship-now` row goes to (`shipNowVerdict` in `./ship.ts`). Absent, a batch's
|
|
333
|
+
* ship-now is refused and nothing is recorded.
|
|
334
|
+
*/
|
|
335
|
+
shipWhere?: ShipNowWhere;
|
|
330
336
|
/** Where a deferred production deploy's `ship` line goes. {@link workerDeployDeps} appends it to the ledger. */
|
|
331
337
|
recordShip?: (line: ShipEventLine) => void;
|
|
332
338
|
}
|
|
@@ -408,6 +414,7 @@ export function workerDeployDeps(cwd: string): DeployDeps {
|
|
|
408
414
|
recordDeployEvent(forgeState(cwd), line);
|
|
409
415
|
},
|
|
410
416
|
ship: () => readShipSetting(forgeState(cwd)),
|
|
417
|
+
shipWhere: { state: forgeState(cwd), forge: forgeCodeRoot(cwd) },
|
|
411
418
|
recordShip: (line) => {
|
|
412
419
|
appendLedger(forgeState(cwd), line);
|
|
413
420
|
},
|
|
@@ -699,7 +706,7 @@ export function runWorkerDeploy(spec: WorkerDeploySpec, deps: DeployDeps): Deplo
|
|
|
699
706
|
// runner's ship instead — asked before anything installs or builds, so deferring costs nothing.
|
|
700
707
|
// Its `ship` line, not a `deploy` one: a deferral is not a deploy, and would drag every median.
|
|
701
708
|
if (spec.env === "production" && deps.ship) {
|
|
702
|
-
const deferred = deferral(`${spec.app} production deploy`, deps.ship(), deps.env ?? process.env);
|
|
709
|
+
const deferred = deferral(`${spec.app} production deploy`, deps.ship(), deps.env ?? process.env, deps.shipWhere);
|
|
703
710
|
if (deferred) {
|
|
704
711
|
log(`\n${deferred}`);
|
|
705
712
|
try {
|
package/src/workerSecrets.ts
CHANGED
|
@@ -92,6 +92,12 @@ export interface SecretSync {
|
|
|
92
92
|
/** Tries per put. A put is idempotent, so a retry can only turn a network blip into a success. */
|
|
93
93
|
export const SECRET_PUT_ATTEMPTS = 3;
|
|
94
94
|
|
|
95
|
+
/**
|
|
96
|
+
* Reads of the account after the puts before a set that is not exact is believed. A read changes
|
|
97
|
+
* nothing, so asking again can only replace a stale answer with a current one — see {@link syncSecrets}.
|
|
98
|
+
*/
|
|
99
|
+
export const SECRET_READBACK_ATTEMPTS = 5;
|
|
100
|
+
|
|
95
101
|
/**
|
|
96
102
|
* Apply `plan`: put every value, delete every extra, then READ THE ACCOUNT BACK and fail unless it
|
|
97
103
|
* holds exactly the set. Refuses without touching anything when the plan has `absent` keys.
|
|
@@ -100,11 +106,17 @@ export const SECRET_PUT_ATTEMPTS = 3;
|
|
|
100
106
|
* `wrangler secret put` of three hit `fetch failed` and the whole deploy stopped with
|
|
101
107
|
* collections-stage holding a Worker and no secrets — serving nothing — for a blip the next
|
|
102
108
|
* attempt would have cleared. The read-back is still the verdict.
|
|
109
|
+
*
|
|
110
|
+
* 🔴 **The read-back is asked again, up to {@link SECRET_READBACK_ATTEMPTS} times, before a set
|
|
111
|
+
* that is not exact is believed.** Measured 2026-09-29 on family-stage (task 2291): the account
|
|
112
|
+
* listed both secrets, both puts said ok, and the list read straight after answered NOTHING — a
|
|
113
|
+
* green deploy stopped as "secrets NOT attached". The same script run again a minute later read
|
|
114
|
+
* both. The LAST read is the verdict, so a set that is really wrong is still a red.
|
|
103
115
|
*/
|
|
104
116
|
export async function syncSecrets(
|
|
105
117
|
plan: SecretPlan,
|
|
106
118
|
ops: SecretOps,
|
|
107
|
-
options: { attempts?: number; pause?: (ms: number) => Promise<void> } = {},
|
|
119
|
+
options: { attempts?: number; readbacks?: number; pause?: (ms: number) => Promise<void> } = {},
|
|
108
120
|
): Promise<SecretSync> {
|
|
109
121
|
const attempts = Math.max(1, options.attempts ?? SECRET_PUT_ATTEMPTS);
|
|
110
122
|
const pause = options.pause ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)));
|
|
@@ -130,9 +142,20 @@ export async function syncSecrets(
|
|
|
130
142
|
lines.push(`DELETE ${name} — not in the set — ${ok ? "gone" : "FAILED"}`);
|
|
131
143
|
if (!ok) bad++;
|
|
132
144
|
}
|
|
133
|
-
const
|
|
134
|
-
const
|
|
135
|
-
|
|
145
|
+
const readbacks = Math.max(1, options.readbacks ?? SECRET_READBACK_ATTEMPTS);
|
|
146
|
+
const isExact = (held: readonly string[]) => held.length === plan.wanted.length && plan.wanted.every((name) => held.includes(name));
|
|
147
|
+
let after = await ops.held();
|
|
148
|
+
let reads = 1;
|
|
149
|
+
while (!isExact(after) && reads < readbacks) {
|
|
150
|
+
await pause(2_000 * reads);
|
|
151
|
+
after = await ops.held();
|
|
152
|
+
reads++;
|
|
153
|
+
}
|
|
154
|
+
const exact = isExact(after);
|
|
155
|
+
const reread = reads > 1 ? ` (read ${reads} times)` : "";
|
|
156
|
+
lines.push(
|
|
157
|
+
exact ? `holds exactly: ${after.join(", ")}${reread}` : `🔴 holds ${after.join(", ") || "(nothing)"} — not exactly ${plan.wanted.join(", ")}${reread}`,
|
|
158
|
+
);
|
|
136
159
|
return { ok: exact && bad === 0, lines, after };
|
|
137
160
|
}
|
|
138
161
|
|