@futurelastic/colab-handbook 1.12.0-rc.30 → 1.12.0-rc.32

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -27,5 +27,5 @@
27
27
  "engines": {
28
28
  "node": ">=18"
29
29
  },
30
- "version": "1.12.0-rc.30"
30
+ "version": "1.12.0-rc.32"
31
31
  }
@@ -24,8 +24,8 @@ Starting points you **copy into your own repo**. That is the entire model.
24
24
  | `dist-refs.yml` | `.github/workflows/dist-refs.yml` | A **private** repo shipping a **compiled** tool, installed as `npx github:<org>/<repo>#vX.Y.Z` ([CONVENTIONS §6, *Distribution*](../CONVENTIONS.md#distribution--one-install-surface-npx-442)) | `workflow_call`, called from the same run that cut the release tag (a tag pushed with `GITHUB_TOKEN` triggers nothing else). Takes one build artifact per `<os>-<arch>` and pushes each as an orphan commit at `refs/tags/dist/vX.Y.Z/<os>-<arch>` with a `SHA256SUMS`. Refuses a public or unreadable repository (a public compiled tool ships npm platform packages instead), a version that is not an existing tag, and a missing platform — before pushing any. Never moves a dist ref that exists. |
25
25
  | `npx-launcher.mjs` | anywhere in-repo (e.g. `bin/launch.mjs`), named by the root `package.json`'s `bin` | Pairs with `dist-refs.yml` | Node ≥ 18, zero dependencies, needs `git`. Reads the `#vX.Y.Z` npx installed, fetches that one dist ref with the same URL and credentials npx cloned with, verifies `SHA256SUMS`, installs to a stable per-user path (`$XDG_DATA_HOME`/`%LOCALAPPDATA%`), and execs it; `--print-path` prints that path for a service's `init`. `*.mjs`, so `colab template npx-launcher.mjs --dest <path>` copies it stamped, `--dest` required. One edit point: `BIN` (defaults to the package name without its scope). |
26
26
  | `deploy-xserver.yml` | `.github/workflows/deploy-xserver.yml` | PHP-framework + Vite apps shipped to **shared hosting over SSH** (no root, no Docker): build on a runner, rsync, migrate on the server | Derived from three independently-written copies. Resolves Node the same way the CI templates do — all three hardcoded it, and one shipped on a different major than its CI built on. Migrates **production**; keeps a **mandatory** smoke test. Does **not** change your tier. |
27
- | `deploy-container.yml` | `.github/workflows/deploy-container.yml` | A **container** app on a host whose platform has an API (Portainer today), `deploy: tag` ([CONVENTIONS §6, *A container deploy*](../CONVENTIONS.md#6-releases), #452) | One contract on every host: build each listed image **once** per final tag (`vX.Y.Z` + commit sha), run the pre-deploy hook, have the platform run exactly that tag (all images, one call), green only on a **verified running version** (stack settled, commit and container images checked, then `release.health-url` reporting `X.Y.Z`), roll back to the previous final on a failure after the platform accepted. Triggers: a final tag push (route `deploy-tag`), `workflow_dispatch` with a tag (a manual deploy or rollback), and `workflow_call` — route `deploy-tag-fast` reaches it from `release-auto.yml`'s commented `deploy` job, like `dist-refs.yml` from its release run. Needs its two scripts below. One secret: `PORTAINER_API_KEY`, a non-admin key owning only this app's stack. **Every redeploy recreates the containers.** Does **not** change your exposure. |
28
- | `deploy-container-run.mjs` | `.github/deploy/deploy-container-run.mjs` | Pairs with `deploy-container.yml` | The driver: builds, runs the hook, calls the adapter, verifies, rolls back, records. Node ≥ 18, zero dependencies; shells out to `git`, `docker`, `gh`. `colab template deploy-container-run.mjs --dest <path>`, `--dest` required, stamped on line 2. |
27
+ | `deploy-container.yml` | `.github/workflows/deploy-container.yml` | A **container** app on a host whose platform has an API (Portainer today), `deploy: tag` ([CONVENTIONS §6, *A container deploy*](../CONVENTIONS.md#6-releases), #452) | One contract on every host: build each listed image **once** per final tag (`vX.Y.Z` + commit sha), run the pre-deploy hook, have the platform run exactly that tag (all images, one call), green only on a **verified running version** (stack settled, commit and container images checked, then `release.health-url` reporting `X.Y.Z`), roll back to the previous final on a failure after the platform accepted. **Two jobs (#460):** `publish` builds and pushes on **every** final, ungated — so a repo not yet cut over still gets each final's image in the registry; `deploy` `needs:` it, never builds (a missing image fails before the platform is touched), and is the one job you gate on "is the platform switched on". Images and build args sit in the workflow-level `env`, written once for both jobs. Triggers: a final tag push (route `deploy-tag`), `workflow_dispatch` with a tag (a manual deploy or rollback), and `workflow_call` — route `deploy-tag-fast` reaches it from `release-auto.yml`'s commented `deploy` job, like `dist-refs.yml` from its release run. Needs its two scripts below. One required secret: `PORTAINER_API_KEY`, a non-admin key owning only this app's stack; optional `HEALTH_AUTH_HEADER` (`Name: value`) or `HEALTH_BASIC_AUTH` (`user:password`) when the version URL sits behind auth — masked, never printed. **Every redeploy recreates the containers.** Does **not** change your exposure. |
28
+ | `deploy-container-run.mjs` | `.github/deploy/deploy-container-run.mjs` | Pairs with `deploy-container.yml` | The driver: builds, runs the hook, calls the adapter, verifies, rolls back, records. `--publish-only` = build and push only (no platform config, no version URL needed); `--deploy-only` = everything after, refusing a tag whose image is not in the registry. **Build args:** `DEPLOY_BUILD_ARGS`, one per line — `KEY=VALUE` for every image or `<image-ref> KEY=VALUE` for one; `${TAG}` (`vX.Y.Z`), `${VERSION}` (`X.Y.Z`) and `${SHA}` are filled in by the driver, so a static site can bake public config and its version into a served file. They end up in the image — public values only, never a secret. Node ≥ 18, zero dependencies; shells out to `git`, `docker`, `gh`. `colab template deploy-container-run.mjs --dest <path>`, `--dest` required, stamped on line 2. |
29
29
  | `deploy-adapter-portainer.mjs` | `.github/deploy/deploy-adapter-portainer.mjs` — **beside** the driver, which loads `./deploy-adapter-<DEPLOY_ADAPTER>.mjs` | Pairs with `deploy-container.yml` | The Portainer adapter: git stacks redeploy `refs/tags/vX.Y.Z`, file stacks update with the compose file; the stack's **whole** `Env` is sent back with `IMAGE_TAG` replaced; 409 waited out and retried, any other error a hard failure (nothing changed). Another platform = another file exporting the same five names (`name`, `configFromEnv`, `snapshot`, `deploy`, `restore`). |
30
30
  | `repo-CLAUDE-block.md` | *paste into* `CLAUDE.md` | Every adopting repo | The discovery hook — how an agent finds the handbook at all. |
31
31
  | `gotchas-d-README.md` | `docs/gotchas.d/README.md` | Any repo starting a `docs/gotchas.d/` directory ([`code-wrap`](../skills/code-wrap/SKILL.md)'s A2 step) | Naming rule (`<issue>-<slug>.md`) and the don't-copy-back rule against an existing `docs/gotchas.md`. Optional — writing the first gotcha entry does not require copying this in first. |
@@ -19,15 +19,30 @@
19
19
  // CONFIGURATION (env; the workflow's one env block):
20
20
  // DEPLOY_ADAPTER portainer (default) — loads ./deploy-adapter-<name>.mjs beside this file
21
21
  // DEPLOY_IMAGES one image per line: `<image-ref> <build-context> [<dockerfile>]`
22
+ // DEPLOY_BUILD_ARGS optional, one build arg per line: `KEY=VALUE` for every image, or
23
+ // `<image-ref> KEY=VALUE` for one. A value may use ${TAG} (vX.Y.Z),
24
+ // ${VERSION} (X.Y.Z) and ${SHA} (the tag's commit) — substituted here,
25
+ // not by a shell. Build args are baked into the image: never a secret.
22
26
  // PRE_DEPLOY optional shell command; gets TAG and PREVIOUS_TAG in its env
23
27
  // HEALTH_URL the version URL; else --health-url; else release.health-url in
24
28
  // .github/project.yml
29
+ // HEALTH_AUTH_HEADER optional, `Name: value` sent with every version-URL request — or
30
+ // HEALTH_BASIC_AUTH optional, `user:password` sent as HTTP Basic. From a secret; masked
31
+ // in the log and never printed. Set at most one.
25
32
  // VERIFY_TIMEOUT_SECONDS 600 · POLL_INTERVAL_MS 5000 · DEPLOY_TIMEOUT_SECONDS 600 ·
26
33
  // BUSY_TIMEOUT_SECONDS 600
27
34
  // GH_TOKEN, GITHUB_REPOSITORY — to record on the release issue (optional)
28
35
  //
29
36
  // Usage: node deploy-container-run.mjs --tag vX.Y.Z [--health-url URL] [--tag-sha SHA]
30
- // Exit: 0 = running vX.Y.Z, verified · 1 = not deployed (rolled back where something changed).
37
+ // [--publish-only | --deploy-only]
38
+ // --publish-only step 1 alone: build and push every image, then stop. Needs no platform
39
+ // configuration and no version URL — so a final publishes its images even
40
+ // before the platform is switched on (the workflow's ungated `publish` job).
41
+ // --deploy-only steps 2-5 alone: every image must already be in the registry for the tag
42
+ // (the `publish` job pushed it); a missing one fails before the platform is
43
+ // touched, never a rebuild here.
44
+ // Exit: 0 = running vX.Y.Z, verified (or, --publish-only, every image pushed) · 1 = not deployed
45
+ // (rolled back where something changed).
31
46
 
32
47
  import { spawnSync } from 'node:child_process';
33
48
  import fs from 'node:fs';
@@ -49,6 +64,60 @@ export function parseImages(text) {
49
64
  return out;
50
65
  }
51
66
 
67
+ /**
68
+ * DEPLOY_BUILD_ARGS → per-image `--build-arg` lists. `KEY=VALUE` applies to every image,
69
+ * `<image-ref> KEY=VALUE` to that one (a later line for the same key wins, per-image over shared).
70
+ * ${TAG} / ${VERSION} / ${SHA} are substituted; any other ${NAME} is a typo and refused.
71
+ */
72
+ export function parseBuildArgs(text, images, release) {
73
+ const vars = { TAG: release.tag, VERSION: String(release.tag).replace(/^v/, ''), SHA: release.sha };
74
+ const refs = new Set(images.map((i) => i.ref));
75
+ const shared = new Map();
76
+ const per = new Map(images.map((i) => [i.ref, new Map()]));
77
+ for (const raw of String(text || '').split('\n')) {
78
+ const line = raw.trim();
79
+ if (!line || line.startsWith('#')) continue;
80
+ let target = null; let pair = line;
81
+ const first = line.split(/\s+/, 1)[0];
82
+ if (!first.includes('=')) {
83
+ target = first;
84
+ pair = line.slice(first.length).trim();
85
+ if (!refs.has(target)) throw new Error(`DEPLOY_BUILD_ARGS line "${line}" names ${target}, which DEPLOY_IMAGES does not list`);
86
+ }
87
+ const m = /^([A-Za-z_][A-Za-z0-9_]*)=(.*)$/.exec(pair);
88
+ if (!m) throw new Error(`DEPLOY_BUILD_ARGS line "${line}" needs KEY=VALUE or <image-ref> KEY=VALUE`);
89
+ const value = m[2].replace(/\$\{([A-Za-z_][A-Za-z0-9_]*)\}/g, (_, k) => {
90
+ if (!(k in vars)) throw new Error(`DEPLOY_BUILD_ARGS: \${${k}} is not one of \${TAG}, \${VERSION}, \${SHA}`);
91
+ return vars[k];
92
+ });
93
+ (target ? per.get(target) : shared).set(m[1], value);
94
+ }
95
+ const out = {};
96
+ for (const img of images) {
97
+ const merged = new Map([...shared, ...per.get(img.ref)]);
98
+ out[img.ref] = [...merged].flatMap(([k, v]) => ['--build-arg', `${k}=${v}`]);
99
+ }
100
+ return out;
101
+ }
102
+
103
+ /** HEALTH_AUTH_HEADER / HEALTH_BASIC_AUTH → { headers, secret }; the secret is what to mask. */
104
+ export function healthAuth(env) {
105
+ const header = (env.HEALTH_AUTH_HEADER || '').trim();
106
+ const basic = env.HEALTH_BASIC_AUTH || '';
107
+ if (header && basic) throw new Error('set HEALTH_AUTH_HEADER or HEALTH_BASIC_AUTH, not both');
108
+ if (basic) {
109
+ if (!basic.includes(':')) throw new Error('HEALTH_BASIC_AUTH must be user:password');
110
+ const token = Buffer.from(basic, 'utf8').toString('base64');
111
+ return { headers: { Authorization: `Basic ${token}` }, secrets: [basic, token] };
112
+ }
113
+ if (header) {
114
+ const m = /^([A-Za-z0-9-]+):\s*(.+)$/.exec(header);
115
+ if (!m) throw new Error('HEALTH_AUTH_HEADER must be "Name: value"');
116
+ return { headers: { [m[1]]: m[2] }, secrets: [m[2]] };
117
+ }
118
+ return { headers: {}, secrets: [] };
119
+ }
120
+
52
121
  /** Does `body` report `version` — as a whole version, so 1.2.1 never matches 1.2.10 or 11.2.1? */
53
122
  export function reportsVersion(body, version) {
54
123
  const v = String(version).replace(/^v/, '').replace(/\./g, '\\.');
@@ -96,6 +165,7 @@ function defaultIo(env) {
96
165
  sleep: (ms) => new Promise((r) => setTimeout(r, ms)),
97
166
  now: () => Date.now(),
98
167
  log: (m) => console.log(m),
168
+ mask: (v) => { if (env.GITHUB_ACTIONS) console.log(`::add-mask::${v}`); },
99
169
  error: (m) => console.log(`::error::${m}`),
100
170
  sh,
101
171
  summary: (line) => { if (env.GITHUB_STEP_SUMMARY) fs.appendFileSync(env.GITHUB_STEP_SUMMARY, `${line}\n`); },
@@ -103,12 +173,12 @@ function defaultIo(env) {
103
173
  };
104
174
  }
105
175
 
106
- async function waitForVersion(url, version, io, timeoutS) {
176
+ async function waitForVersion(url, version, io, timeoutS, auth = {}) {
107
177
  const until = io.now() + timeoutS * 1000;
108
178
  let last = '';
109
179
  for (;;) {
110
180
  try {
111
- const res = await io.fetch(url, { headers: { 'Cache-Control': 'no-cache' } });
181
+ const res = await io.fetch(url, { headers: { 'Cache-Control': 'no-cache', ...auth } });
112
182
  last = await res.text();
113
183
  if (res.ok && reportsVersion(last, version)) return true;
114
184
  } catch (e) { last = String(e && e.message); }
@@ -128,14 +198,14 @@ function record(io, tag, line) {
128
198
  if (c.status !== 0) io.log(`could not comment on release issue #${n}: ${c.stderr.trim()}`);
129
199
  }
130
200
 
131
- function buildOnce(images, tag, sha, io) {
201
+ function buildOnce(images, tag, sha, io, buildArgs = {}) {
132
202
  for (const img of images) {
133
203
  if (io.sh('docker', ['manifest', 'inspect', `${img.ref}:${tag}`]).status === 0) {
134
204
  io.log(`${img.ref}:${tag} is already pushed — built once per tag, not rebuilt`);
135
205
  continue;
136
206
  }
137
207
  const steps = [
138
- ['build', '-f', img.dockerfile, '-t', `${img.ref}:${tag}`, '-t', `${img.ref}:${sha}`, img.context],
208
+ ['build', '-f', img.dockerfile, ...(buildArgs[img.ref] || []), '-t', `${img.ref}:${tag}`, '-t', `${img.ref}:${sha}`, img.context],
139
209
  ['push', `${img.ref}:${tag}`],
140
210
  ['push', `${img.ref}:${sha}`],
141
211
  ];
@@ -151,27 +221,55 @@ export async function run(opts, io) {
151
221
  const env = io.env;
152
222
  const tag = opts.tag;
153
223
  const fail = (m) => { io.error(m); return 1; };
224
+ if (opts.publishOnly && opts.deployOnly) return fail('--publish-only and --deploy-only together would do nothing — pass at most one');
154
225
  if (!FINAL_TAG.test(tag || '')) return fail(`"${tag}" is not a final tag vX.Y.Z — this deploys finals only`);
155
226
  const version = tag.slice(1);
156
- const healthUrl = opts.healthUrl || env.HEALTH_URL || readHealthUrl(io.readFile('.github/project.yml'));
157
- if (!healthUrl) return fail('no version URL — set release.health-url in .github/project.yml (or HEALTH_URL); a deploy is green only on a verified running version');
227
+ const healthUrl = opts.publishOnly ? null : opts.healthUrl || env.HEALTH_URL || readHealthUrl(io.readFile('.github/project.yml'));
228
+ if (!opts.publishOnly && !healthUrl) return fail('no version URL — set release.health-url in .github/project.yml (or HEALTH_URL); a deploy is green only on a verified running version');
158
229
 
159
- let images; let adapter; let cfg;
230
+ let images; let adapter; let cfg; let auth = {};
160
231
  try {
161
232
  images = parseImages(env.DEPLOY_IMAGES);
162
233
  if (!images.length) throw new Error('DEPLOY_IMAGES lists no image — list every image this stack runs');
163
- adapter = await loadAdapter(env.DEPLOY_ADAPTER || 'portainer');
164
- cfg = adapter.configFromEnv(env);
234
+ if (!opts.publishOnly) {
235
+ const a = healthAuth(env);
236
+ for (const s of a.secrets) if (io.mask) io.mask(s);
237
+ auth = a.headers;
238
+ adapter = await loadAdapter(env.DEPLOY_ADAPTER || 'portainer');
239
+ cfg = adapter.configFromEnv(env);
240
+ }
165
241
  } catch (e) { return fail(e.message); }
166
242
 
167
243
  const tagSha = opts.tagSha || io.sh('git', ['rev-parse', `${tag}^{commit}`]).stdout.trim();
168
244
  if (!/^[0-9a-f]{40}$/.test(tagSha)) return fail(`could not resolve ${tag} to a commit`);
169
- const tags = io.sh('git', ['tag', '--list', 'v*']).stdout.split('\n').map((t) => t.trim()).filter(Boolean);
170
- const previous = previousFinalTag(tags, tag);
171
245
  const at = () => new Date(io.now()).toISOString();
172
246
 
173
- // 1. Build once.
174
- try { buildOnce(images, tag, tagSha, io); } catch (e) { record(io, tag, `deploy of ${tag} failed at ${at()}: ${e.message} — nothing deployed`); return fail(e.message); }
247
+ // 1. Build once — or, --deploy-only, check the publish job left every image in the registry.
248
+ if (opts.deployOnly) {
249
+ const missing = images.filter((img) => io.sh('docker', ['manifest', 'inspect', `${img.ref}:${tag}`]).status !== 0).map((img) => `${img.ref}:${tag}`);
250
+ if (missing.length) {
251
+ const m = `not in the registry: ${missing.join(', ')} — the publish job pushes every image before the deploy; nothing deployed`;
252
+ record(io, tag, `deploy of ${tag} failed at ${at()}: ${m}`);
253
+ return fail(m);
254
+ }
255
+ } else {
256
+ let buildArgs;
257
+ try { buildArgs = parseBuildArgs(env.DEPLOY_BUILD_ARGS, images, { tag, sha: tagSha }); } catch (e) { return fail(e.message); }
258
+ try { buildOnce(images, tag, tagSha, io, buildArgs); } catch (e) {
259
+ const what = opts.publishOnly ? 'publish' : 'deploy';
260
+ if (opts.publishOnly) io.summary(`${what} of ${tag} failed at ${at()}: ${e.message}`);
261
+ else record(io, tag, `${what} of ${tag} failed at ${at()}: ${e.message} — nothing deployed`);
262
+ return fail(e.message);
263
+ }
264
+ if (opts.publishOnly) {
265
+ const line = `published ${tag} at ${at()} — ${images.map((i) => `${i.ref}:${tag}`).join(', ')}`;
266
+ io.log(line);
267
+ io.summary(line);
268
+ return 0;
269
+ }
270
+ }
271
+ const tags = io.sh('git', ['tag', '--list', 'v*']).stdout.split('\n').map((t) => t.trim()).filter(Boolean);
272
+ const previous = previousFinalTag(tags, tag);
175
273
 
176
274
  // 2. Pre-deploy hook — before anything on the platform changes.
177
275
  if (env.PRE_DEPLOY && env.PRE_DEPLOY.trim()) {
@@ -195,7 +293,7 @@ export async function run(opts, io) {
195
293
  let reason = null;
196
294
  try {
197
295
  await adapter.deploy(cfg, { tag, tagSha, images: refs, fileContent }, io, snap);
198
- if (!(await waitForVersion(healthUrl, version, io, io.timing.verifyTimeoutS))) {
296
+ if (!(await waitForVersion(healthUrl, version, io, io.timing.verifyTimeoutS, auth))) {
199
297
  throw Object.assign(new Error(`${healthUrl} did not report ${version} within ${io.timing.verifyTimeoutS}s`), { changed: true });
200
298
  }
201
299
  } catch (e) {
@@ -226,7 +324,7 @@ export async function run(opts, io) {
226
324
  const prevSha = io.sh('git', ['rev-parse', `${previous}^{commit}`]).stdout.trim();
227
325
  await adapter.deploy(cfg, { tag: previous, tagSha: prevSha, images: refs, fileContent: snap.fileContent }, io, snap);
228
326
  }
229
- if (back) verified = await waitForVersion(healthUrl, back.slice(1), io, io.timing.verifyTimeoutS);
327
+ if (back) verified = await waitForVersion(healthUrl, back.slice(1), io, io.timing.verifyTimeoutS, auth);
230
328
  } catch (e) {
231
329
  io.log(`rollback to ${back} failed: ${e.message}`);
232
330
  }
@@ -237,10 +335,12 @@ export async function run(opts, io) {
237
335
  return fail(line);
238
336
  }
239
337
 
240
- function parseArgs(argv) {
338
+ export function parseArgs(argv) {
241
339
  const o = {};
242
340
  for (let i = 0; i < argv.length; i++) {
243
341
  const a = argv[i];
342
+ if (a === '--publish-only') { o.publishOnly = true; continue; }
343
+ if (a === '--deploy-only') { o.deployOnly = true; continue; }
244
344
  const m = /^--([a-z-]+)(?:=(.*))?$/.exec(a);
245
345
  if (!m) continue;
246
346
  const v = m[2] !== undefined ? m[2] : argv[++i];
@@ -43,7 +43,15 @@
43
43
  # that owns only this app's stack. Never an admin key: a leaked deploy key must reach one app.
44
44
  # - `secrets.REGISTRY_PASSWORD` — optional; the registry login falls back to the job's
45
45
  # `github.token`, which is enough for GHCR with `packages: write`.
46
- # Everything else is configuration in the env block of the last step.
46
+ # - `secrets.HEALTH_AUTH_HEADER` (`Name: value`) or `secrets.HEALTH_BASIC_AUTH` (`user:password`)
47
+ # — optional, only when the version URL sits behind auth. Masked; never printed.
48
+ # Everything else is configuration: the workflow-level `env` (images, build args — both jobs read
49
+ # them) and the env block of the deploy job's last step (the platform).
50
+ #
51
+ # TWO JOBS (#460). `publish` builds and pushes every image on EVERY final, whatever the platform's
52
+ # state — so a repo not yet cut over still has each final's image in the registry, ready to stage.
53
+ # `deploy` needs it and only talks to the platform; it never builds (a missing image fails before
54
+ # the platform is touched). Gate `deploy`, never `publish`, on "is the platform switched on".
47
55
  #
48
56
  # TOPOLOGY. Every server runs its own Portainer — there is no central panel and no Portainer
49
57
  # Agent. So the per-app configuration is THAT host's Portainer URL + endpointId + stack id + key.
@@ -84,6 +92,10 @@ on:
84
92
  required: false
85
93
  REGISTRY_PASSWORD:
86
94
  required: false
95
+ HEALTH_AUTH_HEADER:
96
+ required: false
97
+ HEALTH_BASIC_AUTH:
98
+ required: false
87
99
 
88
100
  # One deploy at a time per repo, never cancelled half-way: a cancelled deploy is a stack in an
89
101
  # unknown state. A fixed group, because under workflow_call `github.workflow` is the CALLER's name.
@@ -94,16 +106,28 @@ concurrency:
94
106
  permissions:
95
107
  contents: read
96
108
 
109
+ # Read by both jobs. Workflow-level, so the list is written once: `publish` builds exactly what
110
+ # `deploy` then checks is in the registry.
111
+ env:
112
+ # EDIT: every image this stack runs, one per line: <image-ref> <build-context> [<dockerfile>].
113
+ DEPLOY_IMAGES: |
114
+ ghcr.io/${{ github.repository }} .
115
+ # EDIT: optional build args, one per line — `KEY=VALUE` for every image, `<image-ref> KEY=VALUE`
116
+ # for one. ${TAG} (vX.Y.Z), ${VERSION} (X.Y.Z) and ${SHA} (the tag's commit) are filled in by the
117
+ # driver. Baked into the image, so public values only (`vars.*`), never a secret.
118
+ DEPLOY_BUILD_ARGS: ""
119
+
97
120
  jobs:
98
- deploy:
99
- name: Deploy the final
100
- runs-on: ubuntu-latest # EDIT: a runner that reaches your platform (LAN or VPN) and has docker.
101
- environment: production # EDIT: the GitHub environment holding PORTAINER_API_KEY, or remove.
121
+ publish:
122
+ name: Publish the final's images
123
+ # Never gated on the platform: a final publishes its images before cutover too.
124
+ runs-on: ubuntu-latest # EDIT: a runner with docker.
102
125
  permissions:
103
126
  contents: read # check out the tag
104
127
  packages: write # push the images (GHCR); drop it for another registry
105
- issues: write # record the outcome on the release issue
106
128
  timeout-minutes: 45
129
+ outputs:
130
+ tag: ${{ steps.tag.outputs.tag }}
107
131
  steps:
108
132
  - name: Resolve the tag to deploy
109
133
  id: tag
@@ -120,7 +144,42 @@ jobs:
120
144
 
121
145
  - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
122
146
  with:
123
- ref: refs/tags/${{ steps.tag.outputs.tag }}
147
+ ref: refs/tags/${{ steps.tag.outputs.tag }} # the tag ref resolves to the commit it builds
148
+ persist-credentials: false # nothing here pushes to git
149
+
150
+ - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
151
+ with:
152
+ node-version: "22"
153
+
154
+ - name: Log in to the image registry
155
+ env:
156
+ REGISTRY: ghcr.io # EDIT: your registry host
157
+ REGISTRY_USER: ${{ github.actor }} # EDIT: the registry user, when not GHCR
158
+ REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD || github.token }}
159
+ run: |
160
+ set -euo pipefail
161
+ printf '%s' "$REGISTRY_PASSWORD" | docker login "$REGISTRY" -u "$REGISTRY_USER" --password-stdin
162
+
163
+ - name: Build once and push (deploy-container-run --publish-only)
164
+ env:
165
+ TAG: ${{ steps.tag.outputs.tag }}
166
+ run: node .github/deploy/deploy-container-run.mjs --tag "$TAG" --publish-only
167
+
168
+ deploy:
169
+ name: Deploy the final
170
+ needs: publish
171
+ # EDIT: gate the platform half here, e.g. `if: vars.PLATFORM_LIVE == 'true'` — never `publish`.
172
+ runs-on: ubuntu-latest # EDIT: a runner that reaches your platform (LAN or VPN) and has docker.
173
+ environment: production # EDIT: the GitHub environment holding PORTAINER_API_KEY, or remove.
174
+ permissions:
175
+ contents: read # check out the tag
176
+ packages: read # confirm the images are in the registry (GHCR); drop it for another registry
177
+ issues: write # record the outcome on the release issue
178
+ timeout-minutes: 45
179
+ steps:
180
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
181
+ with:
182
+ ref: refs/tags/${{ needs.publish.outputs.tag }}
124
183
  fetch-depth: 0 # every tag: the rollback target is the previous final
125
184
  persist-credentials: false # nothing here pushes to git
126
185
 
@@ -137,12 +196,9 @@ jobs:
137
196
  set -euo pipefail
138
197
  printf '%s' "$REGISTRY_PASSWORD" | docker login "$REGISTRY" -u "$REGISTRY_USER" --password-stdin
139
198
 
140
- - name: Build once, run exactly this tag, verify (deploy-container-run)
199
+ - name: Run exactly this tag, verify (deploy-container-run --deploy-only)
141
200
  env:
142
201
  DEPLOY_ADAPTER: portainer
143
- # EDIT: every image this stack runs, one per line: <image-ref> <build-context> [<dockerfile>].
144
- DEPLOY_IMAGES: |
145
- ghcr.io/${{ github.repository }} .
146
202
  # EDIT: optional — runs before the platform is called, with TAG and PREVIOUS_TAG set; a
147
203
  # non-zero exit stops the deploy before anything changes (a database snapshot, say).
148
204
  PRE_DEPLOY: ""
@@ -153,9 +209,11 @@ jobs:
153
209
  PORTAINER_COMPOSE_FILE: docker-compose.yml # file stacks only: sent as the stack content
154
210
  PORTAINER_API_KEY: ${{ secrets.PORTAINER_API_KEY }}
155
211
  HEALTH_URL: ${{ inputs.health-url }} # empty → release.health-url in .github/project.yml
212
+ HEALTH_AUTH_HEADER: ${{ secrets.HEALTH_AUTH_HEADER }} # optional: `Name: value`
213
+ HEALTH_BASIC_AUTH: ${{ secrets.HEALTH_BASIC_AUTH }} # optional: `user:password`
156
214
  VERIFY_TIMEOUT_SECONDS: "600" # EDIT: how long the new version may take to report itself
157
215
  DEPLOY_TIMEOUT_SECONDS: "600"
158
216
  BUSY_TIMEOUT_SECONDS: "600"
159
217
  GH_TOKEN: ${{ github.token }}
160
- TAG: ${{ steps.tag.outputs.tag }}
161
- run: node .github/deploy/deploy-container-run.mjs --tag "$TAG"
218
+ TAG: ${{ needs.publish.outputs.tag }}
219
+ run: node .github/deploy/deploy-container-run.mjs --tag "$TAG" --deploy-only
@@ -167,6 +167,33 @@ env:
167
167
  # a template
168
168
  HANDBOOK_REF: "stable"
169
169
  HANDBOOK_REPO: "https://github.com/futurelastic/colab-handbook.git"
170
+ # A TRANSIENT UPSTREAM ERROR NEVER TURNS MAIN RED (#464). This run fires on every landing, and a
171
+ # red run at main's head blocks every `colab ship` — so a GitHub / registry 5xx, a 429, a timeout
172
+ # or a dropped connection must not fail a step that a plain re-run would pass. Every step that
173
+ # talks to the network runs its call through `retry_transient` (each step `eval`s this block):
174
+ # the command is re-run with backoff while its output names a transient error, up to
175
+ # RETRY_ATTEMPTS times, and the last attempt's output and exit code are what the step sees. A
176
+ # persistent transient error still fails, with its message; anything else fails at once.
177
+ # Retrying is only safe because every retried call is idempotent — a read, or a write whose
178
+ # re-run after success is a no-op (`colab release cut/finalize/promote` read a head already done
179
+ # as a no-op; a Release or npm version that already exists is a skip). Keep that true if you add one.
180
+ RETRY_ATTEMPTS: "4" # EDIT: tries per call; backoff 5s, 10s, 20s between them.
181
+ RETRY_TRANSIENT: |
182
+ exec 3>&2
183
+ TRANSIENT_PATTERN='HTTP (5[0-9][0-9]|429)|returned error: (5[0-9][0-9]|429)|No server is currently available|Service Unavailable|Bad Gateway|Gateway Time-?out|Internal Server Error|secondary rate limit|rate limit exceeded|ETIMEDOUT|ECONNRESET|EAI_AGAIN|socket hang up|i/o timeout|TLS handshake timeout|connection reset|Could not resolve host|remote end hung up|early EOF|RPC failed|code E(5[0-9][0-9]|429)'
184
+ retry_transient() {
185
+ local attempt=1 max="${RETRY_ATTEMPTS:-4}" delay="${RETRY_DELAY:-5}" rc out err
186
+ out="$(mktemp)"; err="$(mktemp)"
187
+ while :; do
188
+ rc=0
189
+ "$@" > "$out" 2> "$err" || rc=$?
190
+ if [ "$rc" -eq 0 ] || [ "$attempt" -ge "$max" ] || ! cat "$err" "$out" | grep -Eiq "$TRANSIENT_PATTERN"; then
191
+ cat "$out"; cat "$err" >&2; rm -f "$out" "$err"; return "$rc"
192
+ fi
193
+ echo "::warning::$1: transient error (attempt $attempt/$max), retrying in ${delay}s — $(cat "$err" "$out" | grep -Eim1 "$TRANSIENT_PATTERN")" >&3
194
+ sleep "$delay"; attempt=$((attempt + 1)); delay=$((delay * 2))
195
+ done
196
+ }
170
197
 
171
198
  jobs:
172
199
  release:
@@ -227,7 +254,10 @@ jobs:
227
254
  - name: Fetch the colab CLI
228
255
  run: |
229
256
  set -euo pipefail
230
- git clone --quiet --depth 1 --branch "$HANDBOOK_REF" "$HANDBOOK_REPO" "$RUNNER_TEMP/colab-handbook"
257
+ eval "$RETRY_TRANSIENT"
258
+ # A clone that died half-way leaves a directory a second clone refuses — start clean each try.
259
+ fetch_cli() { rm -rf "$RUNNER_TEMP/colab-handbook"; git clone --quiet --depth 1 --branch "$HANDBOOK_REF" "$HANDBOOK_REPO" "$RUNNER_TEMP/colab-handbook"; }
260
+ retry_transient fetch_cli
231
261
  echo "COLAB=$RUNNER_TEMP/colab-handbook/tools/colab" >> "$GITHUB_ENV"
232
262
  # The tags are annotated; git needs an identity to write one.
233
263
  git config user.name "github-actions[bot]"
@@ -245,9 +275,10 @@ jobs:
245
275
  CI_WORKFLOW: "CI"
246
276
  run: |
247
277
  set -uo pipefail
278
+ eval "$RETRY_TRANSIENT"
248
279
  OUT="$RUNNER_TEMP/promote.json"
249
280
  rc=0
250
- node "$COLAB" promote --auto --json > "$OUT" || rc=$?
281
+ retry_transient node "$COLAB" promote --auto --json > "$OUT" || rc=$?
251
282
  if ! jq -e 'has("promoted")' "$OUT" > /dev/null 2>&1; then
252
283
  echo "::warning::colab promote --auto gave no verdict (exit $rc; a HANDBOOK_REF older than #440?) — nothing was promoted"
253
284
  exit 0
@@ -261,7 +292,7 @@ jobs:
261
292
  if [ "$(jq -r '.promoted' "$OUT")" = "true" ]; then
262
293
  echo "promoted=true" >> "$GITHUB_OUTPUT"
263
294
  echo "Promoted to main @ $(jq -r '.sha' "$OUT")" | tee -a "$GITHUB_STEP_SUMMARY"
264
- if gh workflow run "$CI_WORKFLOW" --ref main; then
295
+ if retry_transient gh workflow run "$CI_WORKFLOW" --ref main; then
265
296
  echo "Dispatched $CI_WORKFLOW on main — its green run cuts the candidate." | tee -a "$GITHUB_STEP_SUMMARY"
266
297
  else
267
298
  echo "::warning::promoted, but could not dispatch $CI_WORKFLOW on main (does it declare workflow_dispatch:?) — run it by hand to cut the candidate"
@@ -281,12 +312,14 @@ jobs:
281
312
  if: steps.gate.outputs.cut == 'true' && (github.event_name == 'workflow_run' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule') && steps.promote.outputs.promoted != 'true'
282
313
  run: |
283
314
  set -uo pipefail
315
+ eval "$RETRY_TRANSIENT"
284
316
  OUT="$RUNNER_TEMP/cut.json"
285
317
  rc=0
286
- node "$COLAB" release cut --auto --json > "$OUT" || rc=$?
318
+ retry_transient node "$COLAB" release cut --auto --json > "$OUT" || rc=$?
287
319
  # A verdict is JSON carrying `ok`; anything else is a run that never decided.
288
320
  if ! jq -e 'has("ok")' "$OUT" > /dev/null 2>&1; then
289
321
  echo "::error::colab release cut produced no verdict (exit $rc)"
322
+ echo "colab release cut produced no verdict (exit $rc) — the reason is in this step's log." >> "$GITHUB_STEP_SUMMARY"
290
323
  cat "$OUT"
291
324
  exit 1
292
325
  fi
@@ -319,11 +352,13 @@ jobs:
319
352
  if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
320
353
  run: |
321
354
  set -uo pipefail
355
+ eval "$RETRY_TRANSIENT"
322
356
  OUT="$RUNNER_TEMP/finalize.json"
323
357
  rc=0
324
- node "$COLAB" release finalize --auto --json > "$OUT" || rc=$?
358
+ retry_transient node "$COLAB" release finalize --auto --json > "$OUT" || rc=$?
325
359
  if ! jq -e 'has("state")' "$OUT" > /dev/null 2>&1; then
326
360
  echo "::error::colab release finalize produced no verdict (exit $rc)"
361
+ echo "colab release finalize produced no verdict (exit $rc) — the reason is in this step's log." >> "$GITHUB_STEP_SUMMARY"
327
362
  cat "$OUT"
328
363
  exit 1
329
364
  fi
@@ -385,6 +420,7 @@ jobs:
385
420
  TAGS: ${{ steps.cut.outputs.tag }} ${{ steps.finalize.outputs.tag }}
386
421
  run: |
387
422
  set -euo pipefail
423
+ eval "$RETRY_TRANSIENT"
388
424
  for TAG in $TAGS; do
389
425
  if gh release view "$TAG" > /dev/null 2>&1; then
390
426
  echo "$TAG already has a Release — skipping"
@@ -406,9 +442,10 @@ jobs:
406
442
  # The view above is a check, not a lock: `colab release cut` publishes a candidate's
407
443
  # pre-release itself, and the API can still answer 404 for it seconds later (#462 — a
408
444
  # trunk went red on exactly that 422). So a create refused because the Release already
409
- # exists is the skip case too, decided from the create's own answer.
445
+ # exists is the skip case too, decided from the create's own answer. That also covers a
446
+ # create the API applied but answered 5xx for: the retry's 422 is that skip (#464).
410
447
  ERR="$RUNNER_TEMP/create-$TAG.err"
411
- if ! gh release create "$TAG" --verify-tag --title "$TAG" --notes-file "$NOTES" --generate-notes "${KIND[@]}" 2> "$ERR"; then
448
+ if ! retry_transient gh release create "$TAG" --verify-tag --title "$TAG" --notes-file "$NOTES" --generate-notes "${KIND[@]}" 2> "$ERR"; then
412
449
  cat "$ERR" >&2
413
450
  if grep -qiE 'already[ _]exists' "$ERR"; then
414
451
  echo "$TAG already has a Release (published while this step ran) — skipping"
@@ -461,10 +498,14 @@ jobs:
461
498
  - name: Refuse a private repository, and any npm token
462
499
  run: |
463
500
  set -uo pipefail
464
- # Visibility from the API, never assumed: unknown is refused like private.
465
- PRIVATE="$(gh api "repos/$REPO" --jq '.private' 2> /dev/null || echo unknown)"
501
+ eval "$RETRY_TRANSIENT"
502
+ # Visibility from the API, never assumed: unknown is refused like private. A transient
503
+ # error on this read is retried (#464) — one 5xx here once turned a trunk red.
504
+ ERR="$RUNNER_TEMP/visibility.err"
505
+ PRIVATE="$(retry_transient gh api "repos/$REPO" --jq '.private' 2> "$ERR")" || PRIVATE=unknown
466
506
  if [ "$PRIVATE" != "false" ]; then
467
- echo "::error::$REPO is private (or its visibility could not be read: $PRIVATE) — a private repository never publishes to public npm (#432). Remove release.npm, or install from git."
507
+ WHY="$(grep -v '^::warning::' "$ERR" 2> /dev/null | tail -n 1)"
508
+ echo "::error::$REPO is private (or its visibility could not be read: $PRIVATE${WHY:+ — $WHY}) — a private repository never publishes to public npm (#432). Remove release.npm, or install from git."
468
509
  exit 1
469
510
  fi
470
511
  if [ -n "${NODE_AUTH_TOKEN:-}${NPM_TOKEN:-}" ]; then
@@ -484,8 +525,9 @@ jobs:
484
525
  - name: npm new enough for trusted publishing (>= 11.5.1)
485
526
  run: |
486
527
  set -euo pipefail
528
+ eval "$RETRY_TRANSIENT"
487
529
  if ! node -e 'const [a,b,c]=process.argv[1].split(".").map(Number);process.exit(a>11||(a===11&&(b>5||(b===5&&c>=1)))?0:1)' "$(npm --version)"; then
488
- npm install --global npm@11
530
+ retry_transient npm install --global npm@11
489
531
  fi
490
532
  npm --version
491
533
 
@@ -494,13 +536,15 @@ jobs:
494
536
  - name: Publish each tag (npm publish --provenance)
495
537
  run: |
496
538
  set -euo pipefail
539
+ eval "$RETRY_TRANSIENT"
497
540
  for TAG in $TAGS; do
498
541
  VERSION="${TAG#v}"
499
542
  case "$VERSION" in
500
543
  *-*) DIST=next ;;
501
544
  *) DIST=latest ;;
502
545
  esac
503
- if npm view "$PACKAGE@$VERSION" version > /dev/null 2>&1; then
546
+ # A 404 is "not published" and answers at once; a transient error is retried first (#464).
547
+ if retry_transient npm view "$PACKAGE@$VERSION" version > /dev/null 2>&1; then
504
548
  echo "$PACKAGE@$VERSION is already on npm — skipping" | tee -a "$GITHUB_STEP_SUMMARY"
505
549
  continue
506
550
  fi
@@ -508,7 +552,18 @@ jobs:
508
552
  git clean -fdxq
509
553
  (cd "$PKG_DIR" && npm pkg set "version=$VERSION")
510
554
  sh -c "$GATE"
511
- (cd "$PKG_DIR" && npm publish --tag "$DIST" --provenance --access public --ignore-scripts)
555
+ # The view above is a check, not a lock: a publish the registry applied but answered 5xx
556
+ # for, or one that landed between the view and here, is refused as a republish — the skip
557
+ # case again, decided from the publish's own answer (#464, the same shape as #462).
558
+ ERR="$RUNNER_TEMP/npm-publish-$VERSION.err"
559
+ if ! (cd "$PKG_DIR" && retry_transient npm publish --tag "$DIST" --provenance --access public --ignore-scripts) 2> "$ERR"; then
560
+ cat "$ERR" >&2
561
+ if grep -qiE 'EPUBLISHCONFLICT|cannot publish over (the )?previously published' "$ERR"; then
562
+ echo "$PACKAGE@$VERSION is already on npm (published while this step ran) — skipping" | tee -a "$GITHUB_STEP_SUMMARY"
563
+ continue
564
+ fi
565
+ exit 1
566
+ fi
512
567
  echo "Published $PACKAGE@$VERSION to npm (dist-tag $DIST)" | tee -a "$GITHUB_STEP_SUMMARY"
513
568
  done
514
569
 
@@ -533,10 +588,17 @@ jobs:
533
588
  - name: Record the npm outcome
534
589
  run: |
535
590
  set -euo pipefail
591
+ eval "$RETRY_TRANSIENT"
536
592
  for TAG in $TAGS; do
537
- BODY="$(gh release view "$TAG" --repo "$REPO" --json body --jq '.body' 2> /dev/null)" || {
538
- echo "$TAG has no Release — nothing to record on"
539
- continue
593
+ ERR="$RUNNER_TEMP/record-$TAG.err"
594
+ BODY="$(retry_transient gh release view "$TAG" --repo "$REPO" --json body --jq '.body' 2> "$ERR")" || {
595
+ # Only a Release that is not there is nothing to record on; a read that kept failing is a failure.
596
+ if grep -qiE 'release not found|HTTP 404|Not Found' "$ERR"; then
597
+ echo "$TAG has no Release — nothing to record on"
598
+ continue
599
+ fi
600
+ cat "$ERR" >&2
601
+ exit 1
540
602
  }
541
603
  VERSION="${TAG#v}"
542
604
  case "$VERSION" in *-*) DIST=next ;; *) DIST=latest ;; esac
@@ -548,7 +610,8 @@ jobs:
548
610
  NOTES="$RUNNER_TEMP/notes-$TAG.md"
549
611
  printf '%s\n' "$BODY" | sed '/^<!-- colab:npm -->$/,/^<!-- \/colab:npm -->$/d' > "$NOTES"
550
612
  printf '\n<!-- colab:npm -->\n%s\n<!-- /colab:npm -->\n' "$LINE" >> "$NOTES"
551
- gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES" > /dev/null
613
+ # Idempotent: the block between the colab:npm markers is replaced, never appended twice.
614
+ retry_transient gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES" > /dev/null
552
615
  echo "$TAG: $LINE" | tee -a "$GITHUB_STEP_SUMMARY"
553
616
  done
554
617
 
@@ -504,10 +504,12 @@ const WORKFLOW_FINGERPRINTS = [
504
504
  // overwrite. Verified against all three before being added here.
505
505
  { marker: 'Resolve deploy toolchain', kind: 'step', template: 'deploy-xserver' },
506
506
  { marker: 'Verify the deployed site answers', kind: 'step', template: 'deploy-xserver' },
507
- // deploy-container (#452): its one coined step. "Resolve the tag to deploy" is deliberately not a
508
- // marker — a phrase a hand-written deploy workflow could carry, and an over-eager marker invites
509
- // an overwrite.
510
- { marker: 'Build once, run exactly this tag, verify (deploy-container-run)', kind: 'step', template: 'deploy-container' },
507
+ // deploy-container (#452, split into publish + deploy jobs by #460): its two coined steps.
508
+ // "Resolve the tag to deploy" is deliberately not a marker — a phrase a hand-written deploy
509
+ // workflow could carry, and an over-eager marker invites an overwrite. A copy taken before #460
510
+ // (one step, "Build once, run exactly this tag, verify …") still matches on the header text marker.
511
+ { marker: 'Build once and push (deploy-container-run --publish-only)', kind: 'step', template: 'deploy-container' },
512
+ { marker: 'Run exactly this tag, verify (deploy-container-run --deploy-only)', kind: 'step', template: 'deploy-container' },
511
513
  // branch-name (#348): its one step.
512
514
  { marker: 'Check branch name (CONVENTIONS §4)', kind: 'step', template: 'branch-name' },
513
515
  // Shared by all three ci-* templates: proves derivation, cannot say from which. Note the exact