@futurelastic/colab-handbook 1.16.0-rc.16 → 1.16.0-rc.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json
CHANGED
package/templates/README.md
CHANGED
|
@@ -86,6 +86,7 @@ header naming its new home ([`docs/compat-surfaces.md`](../docs/compat-surfaces.
|
|
|
86
86
|
| `ci-node.yml`, `ci-python.yml`, `ci-laravel.yml` (with the #625 `Migration round-trip` job), `ci-*-heavy.yml` | your forge's CI; `colab ship` / `promote` read or dispatch the runs; `colab template --roundtrip` writes the round-trip job | the adopter's CI layer; until it ships (#608), an optional starting file here |
|
|
87
87
|
| `release-tag.yml`, `release-auto.yml` | your forge's CI; `release-auto.yml` runs `colab release cut` / `finalize --auto` | the release commands' new home (#608); an optional starting file here until then |
|
|
88
88
|
| `dist-refs.yml`, `npx-launcher.mjs`, `npx-service.mjs` | your own release workflow and package `bin` | stays here, as an optional starting file |
|
|
89
|
+
| `publish-downloads.yml`, `publish-downloads.mjs` | your own release workflow, as its last job | stays here, as an optional starting file |
|
|
89
90
|
| `deploy-xserver.yml`, `deploy-container.yml`, `deploy-container-run.mjs`, `deploy-adapter-portainer.mjs` | your repo's deploy workflow | the adopter's deploy tooling; here as an optional starting file |
|
|
90
91
|
| `pre-push-guard` | git, honouring `COLAB_SHIP=1` from `colab ship` | the tool that lands branches (#608); an optional starting file here until then |
|
|
91
92
|
| `pre-commit-dispatch`, `pre-commit-identity`, `identity-vocabulary.example` | git (this repo's own `.githooks/` run them in place) | stays here, as an optional starting file |
|
|
@@ -109,6 +110,8 @@ check that reads one of these files is that tool's behaviour, documented with th
|
|
|
109
110
|
| `dist-refs.yml` | `.github/workflows/dist-refs.yml` | A **private** repo shipping a **compiled** tool, installed as `npx github:<org>/<repo>#vX.Y.Z` ([CONVENTIONS §6, *Distribution*](../CONVENTIONS.md#distribution--one-install-surface-npx-442)) | `workflow_call`, called from the same run that cut the release tag (a tag pushed with `GITHUB_TOKEN` triggers nothing else). Takes one build artifact per `<os>-<arch>` and pushes each as an orphan commit at `refs/tags/dist/vX.Y.Z/<os>-<arch>` with a `SHA256SUMS`. Refuses a public or unreadable repository (a public compiled tool ships npm platform packages instead), a version that is not an existing tag, and a missing platform — before pushing any. Never moves a dist ref that exists. |
|
|
110
111
|
| `npx-launcher.mjs` | anywhere in-repo (e.g. `bin/launch.mjs`), named by the root `package.json`'s `bin` | Pairs with `dist-refs.yml` | Node ≥ 18, zero dependencies, needs `git`. Reads the `#vX.Y.Z` npx installed, fetches that one dist ref with the same URL and credentials npx cloned with, verifies `SHA256SUMS`, installs to a stable per-user path (`$XDG_DATA_HOME`/`%LOCALAPPDATA%`), and execs it; `--print-path` prints that path. A **channel** committish (`#stable`, `#next`, #465) is resolved to the release tag on the commit npx installed, then installed as that version; any other branch is refused. Also the library `npx-service.mjs` imports. `*.mjs`, so `colab template npx-launcher.mjs --dest <path>` copies it stamped, `--dest` required. One edit point: `BIN` (defaults to the package name without its scope). |
|
|
111
112
|
| `npx-service.mjs` | the same directory as your `npx-launcher.mjs` copy (e.g. `bin/service.mjs`), named by the root `package.json`'s `bin` | A tool that runs as a **per-machine service** (a dashboard, a daemon, a JS app a launch agent serves), installed with npx ([CONVENTIONS §6, *Services over npx*](../CONVENTIONS.md#services-over-npx--init-update-rollback-465)) | `npx github:<org>/<repo>#vX.Y.Z init [--auto-update 6h \| --no-auto-update]`, then `<tool> update [--check] [--version vX.Y.Z] [--channel next\|stable]`, `<tool> rollback`, `<tool> status`, `<tool> uninstall [--purge]` (#472: unit, timer, shim and every version go; config and state stay unless `--purge`); anything else runs the current version's payload. Versions side by side under `$XDG_DATA_HOME/<tool>/` with `current`/`previous` links the unit and the `~/.local/bin` shim go through; update installs beside, switches, restarts, health-checks, and **switches back** on failure. Config/secrets under `$XDG_CONFIG_HOME/<tool>/`, state and logs under `$XDG_STATE_HOME/<tool>/` — the unit carries their locations, never a value. launchd agent on macOS, systemd user unit on Linux, Windows refused; `<TOOL>_SERVICE_MANAGER=none` skips the manager. `--auto-update` (#471) adds a timer (launchd `StartInterval` agent / systemd user `.timer`) that runs the same `update`. Node ≥ 18, zero dependencies; needs `git`, `npm` for `update`, `tar` for `dist`. **Imports `./npx-launcher.mjs`** — copy both, or edit the import. Edit points: `KIND` (`binary` = a compiled tool from dist refs, `package` = a JS app run with node, `dist` = a prebuilt JS app unpacked from one verified tarball in the `dist/vX.Y.Z/any` ref, #470), `LABEL`, `SERVICE_ARGS`, `ENTRY`, `DIST_FILE`, `HEALTH_URL`, `KEEP`, `AUTO_UPDATE`, `VERBS`. The package's `files` must include both copies (and a `package` app's built output). `colab template npx-service.mjs --dest <path>`, `--dest` required. |
|
|
113
|
+
| `publish-downloads.yml` | `.github/workflows/publish-downloads.yml` | A repo whose **finals** carry files people install on their own machines, published to a **self-hosted downloads site** ([CONVENTIONS §6, *Downloads*](../CONVENTIONS.md#downloads--finals-to-a-self-hosted-site-648); contract in [`docs/downloads.md`](../docs/downloads.md)) | `workflow_call` from your release workflow's last job (a tag or Release made with `GITHUB_TOKEN` triggers nothing else), and `workflow_dispatch` with a tag to retry, finish a partial run, or `dry_run`. Finals only. Downloads the Release's files the `downloads:` block in `project.yml` names and runs the script below. Inputs: `runs-on` — a runner that can **reach the site** (a private-network site is unreachable from a GitHub-hosted runner; an unresolvable name fails the run before anything is sent) — and `reserved`, names the site keeps for itself. Secrets `DOWNLOADS_SSH_KEY` / `_SSH_HOST` / `_SSH_PORT` / `_SSH_USER` / `DOWNLOADS_KNOWN_HOSTS`, variable `DOWNLOADS_URL` (the site root). Edit point: `PUBLISH_SCRIPT`. |
|
|
114
|
+
| `publish-downloads.mjs` | `.github/downloads/publish-downloads.mjs` (the workflow's `PUBLISH_SCRIPT`) | Pairs with `publish-downloads.yml` | `check` · `names vX.Y.Z` · `publish vX.Y.Z <dir> [--dry-run]`. rsync over SSH in a fixed order — the version folder, each file read back over HTTPS against its checksum, `SHA256SUMS`, installers (`@DOWNLOADS_URL@` stamped), `manifest.json`, `latest.json` **last** (never moved back to an older final) — then reads it all back. Never replaces a published file; a different `SHA256SUMS` for a published version refuses; a re-run sends nothing. Validates the manifest the portal lists. Node ≥ 18, zero dependencies. `colab template publish-downloads.mjs --dest <path>`, `--dest` required. |
|
|
112
115
|
| `deploy-xserver.yml` | `.github/workflows/deploy-xserver.yml` | PHP-framework + Vite apps shipped to **shared hosting over SSH** (no root, no Docker): build on a runner, rsync, migrate on the server | Derived from three independently-written copies. Resolves Node the same way the CI templates do — all three hardcoded it, and one shipped on a different major than its CI built on. Migrates **production**; keeps a **mandatory** smoke test. Does **not** change your tier. |
|
|
113
116
|
| `deploy-container.yml` | `.github/workflows/deploy-container.yml` | A **container** app on a host whose platform has an API (Portainer today), `deploy: tag` ([CONVENTIONS §6, *A container deploy*](../CONVENTIONS.md#6-releases), #452) | One contract on every host: build each listed image **once** per final tag (`vX.Y.Z` + commit sha), run the pre-deploy hook, have the platform run exactly that tag (all images, one call), green only on a **verified running version** (stack settled, commit and container images checked, then `release.health-url` reporting `X.Y.Z`), roll back to the previous final on a failure after the platform accepted. **Two jobs (#460):** `publish` builds and pushes on **every** final, ungated — so a repo not yet cut over still gets each final's image in the registry; `deploy` `needs:` it, never builds (a missing image fails before the platform is touched), and is the one job you gate on "is the platform switched on". Images and build args sit in the workflow-level `env`, written once for both jobs. Triggers: a final tag push (route `deploy-tag`), `workflow_dispatch` with a tag (a manual deploy or rollback), and `workflow_call` — route `deploy-tag-fast` reaches it from `release-auto.yml`'s commented `deploy` job, like `dist-refs.yml` from its release run. Needs its two scripts below. One required secret: `PORTAINER_API_KEY`, a non-admin key owning only this app's stack; optional `HEALTH_AUTH_HEADER` (`Name: value`) or `HEALTH_BASIC_AUTH` (`user:password`) when the version URL sits behind auth — masked, never printed. **Every redeploy recreates the containers.** Does **not** change your exposure. |
|
|
114
117
|
| `deploy-container-run.mjs` | `.github/deploy/deploy-container-run.mjs` | Pairs with `deploy-container.yml` | The driver: builds, runs the hook, calls the adapter, verifies, rolls back, records. `--publish-only` = build and push only (no platform config, no version URL needed); `--deploy-only` = everything after, refusing a tag whose image is not in the registry. **Build args:** `DEPLOY_BUILD_ARGS`, one per line — `KEY=VALUE` for every image or `<image-ref> KEY=VALUE` for one; `${TAG}` (`vX.Y.Z`), `${VERSION}` (`X.Y.Z`) and `${SHA}` are filled in by the driver, so a static site can bake public config and its version into a served file. They end up in the image — public values only, never a secret. Node ≥ 18, zero dependencies; shells out to `git`, `docker`, `gh`. `colab template deploy-container-run.mjs --dest <path>`, `--dest` required, stamped on line 2. |
|
|
@@ -272,6 +275,8 @@ directory into ~45 repos and was wrong — fixed 2026-08-20.
|
|
|
272
275
|
colab template deploy-container --dest .github/workflows/deploy-container.yml
|
|
273
276
|
colab template deploy-container-run.mjs --dest .github/deploy/deploy-container-run.mjs
|
|
274
277
|
colab template deploy-adapter-portainer.mjs --dest .github/deploy/deploy-adapter-portainer.mjs
|
|
278
|
+
colab template publish-downloads --dest .github/workflows/publish-downloads.yml
|
|
279
|
+
colab template publish-downloads.mjs --dest .github/downloads/publish-downloads.mjs
|
|
275
280
|
colab template docs-lint.mjs --dest tools/docs-lint.mjs # --dest is required for this one
|
|
276
281
|
```
|
|
277
282
|
|
|
@@ -0,0 +1,574 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Publish one FINAL version's files to a self-hosted downloads site — TEMPLATE (colab-handbook #648).
|
|
3
|
+
// Copy me into your repo beside the workflow (`colab template publish-downloads.mjs --dest
|
|
4
|
+
// .github/downloads/publish-downloads.mjs`); from then on YOU OWN THIS FILE. Node >= 18, zero dependencies,
|
|
5
|
+
// shells out to `rsync` (and `ssh` through it). Docs: docs/downloads.md in the handbook.
|
|
6
|
+
//
|
|
7
|
+
// node publish-downloads.mjs check validate the downloads: block and the manifest
|
|
8
|
+
// node publish-downloads.mjs names vX.Y.Z the Release files this publishes, one per line
|
|
9
|
+
// node publish-downloads.mjs publish vX.Y.Z <assetsDir> [--dry-run] stage, upload, read back
|
|
10
|
+
// (every command also takes --root <repo root>, default the current directory)
|
|
11
|
+
//
|
|
12
|
+
// WHAT IS PUBLISHED comes from the repo's `.github/project.yml`:
|
|
13
|
+
//
|
|
14
|
+
// downloads:
|
|
15
|
+
// app: myapp # the app's folder on the site: [a-z0-9][a-z0-9-]*
|
|
16
|
+
// files: # every final carries these; {v} is X.Y.Z
|
|
17
|
+
// - myapp-{v}.tgz
|
|
18
|
+
// - MyApp_{v}_aarch64.dmg
|
|
19
|
+
// optional: [MyApp_{v}_x64.msi] # published when the Release has it (optional key)
|
|
20
|
+
// installers: # repo paths, uploaded to /<app>/ with @DOWNLOADS_URL@ stamped (optional key)
|
|
21
|
+
// - scripts/install.sh
|
|
22
|
+
// manifest: .github/downloads/manifest.json # optional key, this is the default path
|
|
23
|
+
//
|
|
24
|
+
// Only that subset of YAML is read here (scalars, `- item` lists, `[a, b]` lists, comments): this script has no
|
|
25
|
+
// YAML library on purpose, and refuses — naming the line — anything else inside the block.
|
|
26
|
+
//
|
|
27
|
+
// EVERYTHING ABOUT THE SITE COMES FROM THE ENVIRONMENT; nothing about it is written in the repository:
|
|
28
|
+
//
|
|
29
|
+
// DOWNLOADS_URL the site's root address (a repo variable); /<app> is appended here
|
|
30
|
+
// DOWNLOADS_SSH_HOST/PORT/USER where rsync goes (repo secrets). The account is restricted to the app's
|
|
31
|
+
// folder server-side and cannot delete: its root IS /<app>/
|
|
32
|
+
// DOWNLOADS_KEY_FILE the private key file the workflow wrote from DOWNLOADS_SSH_KEY
|
|
33
|
+
// DOWNLOADS_KNOWN_HOSTS_FILE the known-hosts file the workflow wrote from DOWNLOADS_KNOWN_HOSTS
|
|
34
|
+
// DOWNLOADS_RESERVED names the site keeps for itself, refused as an app slug (space/comma list)
|
|
35
|
+
//
|
|
36
|
+
// THE ORDER ON THE SITE, which is what makes a half-finished run harmless:
|
|
37
|
+
// 1. <X.Y.Z>/ with every file except SHA256SUMS (rsync --ignore-existing: a published file is never replaced)
|
|
38
|
+
// 2. every file read back over HTTPS against the SHA256SUMS this build computed
|
|
39
|
+
// 3. <X.Y.Z>/SHA256SUMS (its presence = the folder is complete)
|
|
40
|
+
// 4. the installers (address stamped in), then manifest.json
|
|
41
|
+
// 5. latest.json LAST (unless the site already names a newer version)
|
|
42
|
+
// 6. SHA256SUMS and every file sent in 4–5 read back from the site
|
|
43
|
+
// A published version is immutable: a folder whose SHA256SUMS differs from this build's is an error and nothing is
|
|
44
|
+
// changed. A complete folder is verified and left alone; a partial one gets only its missing files. Installers,
|
|
45
|
+
// manifest.json and latest.json are sent only when the site's copy differs, so a re-run is a verified no-op.
|
|
46
|
+
// Never a candidate: the tag must be `vX.Y.Z`.
|
|
47
|
+
import { spawnSync } from 'node:child_process'
|
|
48
|
+
import { createHash } from 'node:crypto'
|
|
49
|
+
import { lookup } from 'node:dns/promises'
|
|
50
|
+
import {
|
|
51
|
+
copyFileSync,
|
|
52
|
+
createReadStream,
|
|
53
|
+
existsSync,
|
|
54
|
+
mkdirSync,
|
|
55
|
+
mkdtempSync,
|
|
56
|
+
readFileSync,
|
|
57
|
+
readdirSync,
|
|
58
|
+
realpathSync,
|
|
59
|
+
statSync,
|
|
60
|
+
writeFileSync,
|
|
61
|
+
} from 'node:fs'
|
|
62
|
+
import { tmpdir } from 'node:os'
|
|
63
|
+
import { basename, join, resolve } from 'node:path'
|
|
64
|
+
import { pathToFileURL } from 'node:url'
|
|
65
|
+
|
|
66
|
+
const FINAL_TAG = /^v(\d+\.\d+\.\d+)$/
|
|
67
|
+
const SLUG = /^[a-z0-9][a-z0-9-]*$/
|
|
68
|
+
const PLATFORMS = ['macos', 'windows', 'linux']
|
|
69
|
+
export const PLACEHOLDER = '@DOWNLOADS_URL@'
|
|
70
|
+
export const DEFAULT_MANIFEST = '.github/downloads/manifest.json'
|
|
71
|
+
/** Names the script itself writes at the app's root; an installer may not take one. */
|
|
72
|
+
const ROOT_NAMES = new Set(['latest.json', 'manifest.json'])
|
|
73
|
+
|
|
74
|
+
/** The version of a final tag. A candidate (`vX.Y.Z-rc.N`) and anything else is refused. */
|
|
75
|
+
export function finalVersion(tag) {
|
|
76
|
+
const m = typeof tag === 'string' ? FINAL_TAG.exec(tag) : null
|
|
77
|
+
if (!m)
|
|
78
|
+
throw new Error(
|
|
79
|
+
`${JSON.stringify(tag)} is not a final tag: expected vX.Y.Z (a candidate is never published)`,
|
|
80
|
+
)
|
|
81
|
+
return m[1]
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
// ── the downloads: block ─────────────────────────────────────────────────────────────────────────
|
|
85
|
+
|
|
86
|
+
function scalar(raw, where) {
|
|
87
|
+
let s = raw.trim()
|
|
88
|
+
if (s.startsWith('"') || s.startsWith("'")) {
|
|
89
|
+
const q = s[0]
|
|
90
|
+
const end = s.indexOf(q, 1)
|
|
91
|
+
if (end < 0) throw new Error(`${where}: unterminated quote`)
|
|
92
|
+
const rest = s.slice(end + 1).trim()
|
|
93
|
+
if (rest && !rest.startsWith('#')) throw new Error(`${where}: text after the closing quote`)
|
|
94
|
+
return s.slice(1, end)
|
|
95
|
+
}
|
|
96
|
+
const hash = s.search(/\s#/)
|
|
97
|
+
if (hash >= 0) s = s.slice(0, hash).trim()
|
|
98
|
+
if (s.startsWith('#')) s = ''
|
|
99
|
+
return s
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
function flowList(raw, where) {
|
|
103
|
+
const s = scalar(raw.replace(/\s+#.*$/, ''), where)
|
|
104
|
+
if (!s.startsWith('[') || !s.endsWith(']')) return null
|
|
105
|
+
const inner = s.slice(1, -1).trim()
|
|
106
|
+
return inner ? inner.split(',').map((x) => scalar(x, where)) : []
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
/**
|
|
110
|
+
* The `downloads:` block of project.yml text, as a plain object — or null when the repo declares none. Reads the
|
|
111
|
+
* subset documented in the header and throws, naming the line, on anything else inside the block.
|
|
112
|
+
*/
|
|
113
|
+
export function parseDownloadsBlock(text) {
|
|
114
|
+
const lines = text.split(/\r?\n/)
|
|
115
|
+
const start = lines.findIndex((l) => /^downloads:\s*(#.*)?$/.test(l))
|
|
116
|
+
if (start < 0) {
|
|
117
|
+
const inline = lines.findIndex((l) => /^downloads:\s*\S/.test(l))
|
|
118
|
+
if (inline >= 0)
|
|
119
|
+
throw new Error(
|
|
120
|
+
`project.yml line ${inline + 1}: write downloads: as a block (one key per line), not inline`,
|
|
121
|
+
)
|
|
122
|
+
return null
|
|
123
|
+
}
|
|
124
|
+
const out = {}
|
|
125
|
+
let keyIndent = null
|
|
126
|
+
let listKey = null
|
|
127
|
+
for (let i = start + 1; i < lines.length; i++) {
|
|
128
|
+
const line = lines[i]
|
|
129
|
+
const where = `project.yml line ${i + 1}`
|
|
130
|
+
if (/^\s*(#.*)?$/.test(line)) continue
|
|
131
|
+
const indent = line.match(/^ */)[0].length
|
|
132
|
+
if (indent === 0) break // the next top-level key ends the block
|
|
133
|
+
if (line.includes('\t')) throw new Error(`${where}: a tab in the downloads: block`)
|
|
134
|
+
const body = line.slice(indent)
|
|
135
|
+
if (keyIndent === null) keyIndent = indent
|
|
136
|
+
if (indent > keyIndent && listKey && body.startsWith('- ')) {
|
|
137
|
+
out[listKey].push(scalar(body.slice(2), where))
|
|
138
|
+
continue
|
|
139
|
+
}
|
|
140
|
+
if (indent === keyIndent && listKey && body.startsWith('- ')) {
|
|
141
|
+
// `- item` at the key's own indent is valid YAML for a list under that key
|
|
142
|
+
out[listKey].push(scalar(body.slice(2), where))
|
|
143
|
+
continue
|
|
144
|
+
}
|
|
145
|
+
const m = indent === keyIndent ? /^([a-z][a-z-]*):(.*)$/.exec(body) : null
|
|
146
|
+
if (!m) throw new Error(`${where}: not a form this script reads inside downloads: (see its header)`)
|
|
147
|
+
const [, key, rest] = m
|
|
148
|
+
if (key in out) throw new Error(`${where}: downloads.${key} is given twice`)
|
|
149
|
+
const value = scalar(rest, where)
|
|
150
|
+
if (value === '') {
|
|
151
|
+
out[key] = []
|
|
152
|
+
listKey = key
|
|
153
|
+
continue
|
|
154
|
+
}
|
|
155
|
+
listKey = null
|
|
156
|
+
const list = flowList(rest, where)
|
|
157
|
+
out[key] = list ?? value
|
|
158
|
+
}
|
|
159
|
+
return out
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
const KEYS = ['app', 'files', 'optional', 'installers', 'manifest']
|
|
163
|
+
|
|
164
|
+
function fileName(pattern, where) {
|
|
165
|
+
if (typeof pattern !== 'string' || pattern === '') throw new Error(`${where}: an empty name`)
|
|
166
|
+
if (/[/\\]/.test(pattern) || pattern.startsWith('.') || pattern === 'SHA256SUMS')
|
|
167
|
+
throw new Error(`${where}: ${JSON.stringify(pattern)} is not a plain file name`)
|
|
168
|
+
return pattern
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
/** Validates a parsed block (see parseDownloadsBlock) and returns it normalised. */
|
|
172
|
+
export function readConfig(block, { reserved = [] } = {}) {
|
|
173
|
+
if (!block) throw new Error('.github/project.yml has no downloads: block — this repo publishes nothing')
|
|
174
|
+
for (const k of Object.keys(block))
|
|
175
|
+
if (!KEYS.includes(k)) throw new Error(`downloads.${k} is not a key (known: ${KEYS.join(', ')})`)
|
|
176
|
+
const app = block.app
|
|
177
|
+
if (typeof app !== 'string' || app === '') throw new Error('downloads.app is required')
|
|
178
|
+
if (app.startsWith('_')) throw new Error(`downloads.app ${JSON.stringify(app)}: names starting with _ belong to the site`)
|
|
179
|
+
if (!SLUG.test(app)) throw new Error(`downloads.app ${JSON.stringify(app)} is not a slug: [a-z0-9][a-z0-9-]*`)
|
|
180
|
+
if (reserved.includes(app)) throw new Error(`downloads.app ${JSON.stringify(app)} is a name the site reserves for itself`)
|
|
181
|
+
const list = (k, required) => {
|
|
182
|
+
const v = block[k]
|
|
183
|
+
if (v === undefined && !required) return []
|
|
184
|
+
if (!Array.isArray(v) || (required && v.length === 0))
|
|
185
|
+
throw new Error(`downloads.${k} must be a ${required ? 'non-empty ' : ''}list`)
|
|
186
|
+
return v
|
|
187
|
+
}
|
|
188
|
+
const files = list('files', true).map((p) => fileName(p, 'downloads.files'))
|
|
189
|
+
const optional = list('optional', false).map((p) => fileName(p, 'downloads.optional'))
|
|
190
|
+
const all = [...files, ...optional]
|
|
191
|
+
const dup = all.find((n, i) => all.indexOf(n) !== i)
|
|
192
|
+
if (dup) throw new Error(`downloads: ${dup} is listed twice`)
|
|
193
|
+
const installers = list('installers', false)
|
|
194
|
+
const seen = new Set()
|
|
195
|
+
for (const p of installers) {
|
|
196
|
+
const b = basename(String(p))
|
|
197
|
+
if (!p || p.startsWith('/') || String(p).split(/[/\\]/).includes('..'))
|
|
198
|
+
throw new Error(`downloads.installers: ${JSON.stringify(p)} is not a path inside the repo`)
|
|
199
|
+
if (ROOT_NAMES.has(b) || /^\d+\.\d+\.\d+$/.test(b) || seen.has(b))
|
|
200
|
+
throw new Error(`downloads.installers: ${b} would collide with another file at the app's root`)
|
|
201
|
+
seen.add(b)
|
|
202
|
+
}
|
|
203
|
+
const manifest = block.manifest ?? DEFAULT_MANIFEST
|
|
204
|
+
if (typeof manifest !== 'string' || manifest.startsWith('/') || manifest.split(/[/\\]/).includes('..'))
|
|
205
|
+
throw new Error('downloads.manifest must be a path inside the repo')
|
|
206
|
+
return { app, files, optional, installers, manifest }
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
export const namesFor = (patterns, v) => patterns.map((p) => p.split('{v}').join(v))
|
|
210
|
+
|
|
211
|
+
// ── manifest.json ────────────────────────────────────────────────────────────────────────────────
|
|
212
|
+
|
|
213
|
+
/** A text the portal shows: a string, or one string per language code (`{"en": "…", "vi": "…"}`). */
|
|
214
|
+
function isText(v, oneLine = false) {
|
|
215
|
+
const ok = (s) => typeof s === 'string' && s.trim() !== '' && (!oneLine || !/[\r\n]/.test(s))
|
|
216
|
+
if (typeof v === 'string') return ok(v)
|
|
217
|
+
return (
|
|
218
|
+
v !== null &&
|
|
219
|
+
typeof v === 'object' &&
|
|
220
|
+
!Array.isArray(v) &&
|
|
221
|
+
Object.keys(v).length > 0 &&
|
|
222
|
+
Object.entries(v).every(([k, s]) => /^[a-z]{2}(-[A-Z]{2})?$/.test(k) && ok(s))
|
|
223
|
+
)
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
/** Throws naming every problem with a manifest object; returns it unchanged when valid. Fields: docs/downloads.md. */
|
|
227
|
+
export function validateManifest(m) {
|
|
228
|
+
const problems = []
|
|
229
|
+
if (m === null || typeof m !== 'object' || Array.isArray(m)) throw new Error('manifest.json is not a JSON object')
|
|
230
|
+
for (const k of Object.keys(m))
|
|
231
|
+
if (!['schema', 'name', 'summary', 'install'].includes(k)) problems.push(`unknown field "${k}"`)
|
|
232
|
+
if (m.schema !== undefined && m.schema !== 1) problems.push('schema must be 1')
|
|
233
|
+
if (!isText(m.name, true)) problems.push('name: a one-line text is required')
|
|
234
|
+
if (!isText(m.summary, true)) problems.push('summary: a one-line text is required')
|
|
235
|
+
const inst = m.install
|
|
236
|
+
if (inst === null || typeof inst !== 'object' || Array.isArray(inst) || Object.keys(inst).length === 0) {
|
|
237
|
+
problems.push(`install: an object keyed by platform (${PLATFORMS.join(', ')}) is required`)
|
|
238
|
+
} else {
|
|
239
|
+
for (const [os, how] of Object.entries(inst)) {
|
|
240
|
+
if (!PLATFORMS.includes(os)) {
|
|
241
|
+
problems.push(`install.${os}: not a platform (${PLATFORMS.join(', ')})`)
|
|
242
|
+
continue
|
|
243
|
+
}
|
|
244
|
+
if (how === null || typeof how !== 'object' || Array.isArray(how)) {
|
|
245
|
+
problems.push(`install.${os}: an object with steps (and optional prompts)`)
|
|
246
|
+
continue
|
|
247
|
+
}
|
|
248
|
+
for (const k of Object.keys(how))
|
|
249
|
+
if (!['steps', 'prompts'].includes(k)) problems.push(`install.${os}.${k}: unknown field`)
|
|
250
|
+
if (!Array.isArray(how.steps) || how.steps.length === 0 || !how.steps.every((s) => isText(s)))
|
|
251
|
+
problems.push(`install.${os}.steps: a non-empty list of texts`)
|
|
252
|
+
if (how.prompts !== undefined && (!Array.isArray(how.prompts) || !how.prompts.every((s) => isText(s))))
|
|
253
|
+
problems.push(`install.${os}.prompts: a list of texts`)
|
|
254
|
+
}
|
|
255
|
+
}
|
|
256
|
+
if (problems.length) throw new Error(`manifest.json: ${problems.join('; ')}`)
|
|
257
|
+
return m
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
/** Reads and validates the repo's config + manifest. */
|
|
261
|
+
export function loadRepo(root, env = {}) {
|
|
262
|
+
const yml = join(root, '.github', 'project.yml')
|
|
263
|
+
if (!existsSync(yml)) throw new Error(`${yml} does not exist`)
|
|
264
|
+
const reserved = String(env.DOWNLOADS_RESERVED ?? '')
|
|
265
|
+
.split(/[\s,]+/)
|
|
266
|
+
.filter(Boolean)
|
|
267
|
+
const config = readConfig(parseDownloadsBlock(readFileSync(yml, 'utf8')), { reserved })
|
|
268
|
+
const mPath = join(root, config.manifest)
|
|
269
|
+
if (!existsSync(mPath)) throw new Error(`${config.manifest} does not exist — every app on the site carries a manifest`)
|
|
270
|
+
let manifest
|
|
271
|
+
try {
|
|
272
|
+
manifest = JSON.parse(readFileSync(mPath, 'utf8'))
|
|
273
|
+
} catch (error) {
|
|
274
|
+
throw new Error(`${config.manifest} is not JSON: ${error.message}`)
|
|
275
|
+
}
|
|
276
|
+
validateManifest(manifest)
|
|
277
|
+
for (const p of config.installers) {
|
|
278
|
+
const f = join(root, p)
|
|
279
|
+
if (!existsSync(f)) throw new Error(`downloads.installers: ${p} does not exist`)
|
|
280
|
+
if (!readFileSync(f, 'utf8').includes(PLACEHOLDER))
|
|
281
|
+
throw new Error(`${p} has no ${PLACEHOLDER} to stamp the downloads address into`)
|
|
282
|
+
}
|
|
283
|
+
return { config, manifest }
|
|
284
|
+
}
|
|
285
|
+
|
|
286
|
+
// ── staging ──────────────────────────────────────────────────────────────────────────────────────
|
|
287
|
+
|
|
288
|
+
export function sha256File(path) {
|
|
289
|
+
return new Promise((resolveHash, reject) => {
|
|
290
|
+
const h = createHash('sha256')
|
|
291
|
+
createReadStream(path)
|
|
292
|
+
.on('data', (c) => h.update(c))
|
|
293
|
+
.on('error', reject)
|
|
294
|
+
.on('end', () => resolveHash(h.digest('hex')))
|
|
295
|
+
})
|
|
296
|
+
}
|
|
297
|
+
|
|
298
|
+
/** Compares X.Y.Z versions: negative when a < b. */
|
|
299
|
+
export function compareVersions(a, b) {
|
|
300
|
+
const pa = a.split('.').map(Number)
|
|
301
|
+
const pb = b.split('.').map(Number)
|
|
302
|
+
for (let i = 0; i < 3; i++) if (pa[i] !== pb[i]) return pa[i] - pb[i]
|
|
303
|
+
return 0
|
|
304
|
+
}
|
|
305
|
+
|
|
306
|
+
export function siteRoot(baseUrl) {
|
|
307
|
+
if (typeof baseUrl !== 'string' || !/^https:\/\/[^\s'"\\]+$/.test(baseUrl)) {
|
|
308
|
+
throw new Error('DOWNLOADS_URL must be an https address (the repo variable is missing or malformed)')
|
|
309
|
+
}
|
|
310
|
+
return baseUrl.replace(/\/+$/, '')
|
|
311
|
+
}
|
|
312
|
+
|
|
313
|
+
/**
|
|
314
|
+
* Lays out what goes to the site under `outDir`: `<X.Y.Z>/` (the files + a combined SHA256SUMS), the installers with
|
|
315
|
+
* the site root stamped in, manifest.json and latest.json. Throws, uploading nothing, when a required file is missing.
|
|
316
|
+
*/
|
|
317
|
+
export async function stage({ tag, assetsDir, outDir, baseUrl, root, config, manifest }) {
|
|
318
|
+
const version = finalVersion(tag)
|
|
319
|
+
const base = siteRoot(baseUrl)
|
|
320
|
+
const present = new Set(readdirSync(assetsDir))
|
|
321
|
+
const missing = namesFor(config.files, version).filter((n) => !present.has(n))
|
|
322
|
+
if (missing.length > 0) {
|
|
323
|
+
throw new Error(
|
|
324
|
+
`the Release of ${tag} lacks ${missing.join(', ')} — nothing was published; finish the builds, then dispatch the publish again`,
|
|
325
|
+
)
|
|
326
|
+
}
|
|
327
|
+
const names = [...namesFor(config.files, version), ...namesFor(config.optional, version).filter((n) => present.has(n))]
|
|
328
|
+
const folder = join(outDir, version)
|
|
329
|
+
mkdirSync(folder, { recursive: true })
|
|
330
|
+
const files = {}
|
|
331
|
+
for (const name of names) {
|
|
332
|
+
copyFileSync(join(assetsDir, name), join(folder, name))
|
|
333
|
+
files[name] = { sha256: await sha256File(join(folder, name)), size: statSync(join(folder, name)).size }
|
|
334
|
+
}
|
|
335
|
+
const sums = Object.keys(files)
|
|
336
|
+
.sort()
|
|
337
|
+
.map((n) => `${files[n].sha256} ${n}\n`)
|
|
338
|
+
.join('')
|
|
339
|
+
writeFileSync(join(folder, 'SHA256SUMS'), sums)
|
|
340
|
+
const rootFiles = []
|
|
341
|
+
for (const p of config.installers) {
|
|
342
|
+
const source = readFileSync(join(root, p), 'utf8')
|
|
343
|
+
writeFileSync(join(outDir, basename(p)), source.split(PLACEHOLDER).join(base))
|
|
344
|
+
rootFiles.push(basename(p))
|
|
345
|
+
}
|
|
346
|
+
writeFileSync(join(outDir, 'manifest.json'), `${JSON.stringify(manifest, null, 2)}\n`)
|
|
347
|
+
rootFiles.push('manifest.json')
|
|
348
|
+
writeFileSync(join(outDir, 'latest.json'), `${JSON.stringify({ version, files }, null, 2)}\n`)
|
|
349
|
+
return { version, files, sums, rootFiles }
|
|
350
|
+
}
|
|
351
|
+
|
|
352
|
+
// ── the site ─────────────────────────────────────────────────────────────────────────────────────
|
|
353
|
+
|
|
354
|
+
function runRsync({ rsync, sshArgs, args, log }) {
|
|
355
|
+
const full = [...(sshArgs ? ['-e', sshArgs] : []), ...args]
|
|
356
|
+
log(`$ rsync ${full.join(' ')}`)
|
|
357
|
+
const r = spawnSync(rsync, full, { encoding: 'utf8' })
|
|
358
|
+
if (r.status !== 0) throw new Error(`rsync failed (${r.status ?? r.signal}): ${(r.stderr || r.stdout || '').trim()}`)
|
|
359
|
+
}
|
|
360
|
+
|
|
361
|
+
async function get(fetchImpl, url, attempts = 3) {
|
|
362
|
+
let last
|
|
363
|
+
for (let i = 0; i < attempts; i++) {
|
|
364
|
+
try {
|
|
365
|
+
const res = await fetchImpl(url, { headers: { 'cache-control': 'no-cache' } })
|
|
366
|
+
if (res.status < 500) return res
|
|
367
|
+
last = new Error(`${url} answered ${res.status}`)
|
|
368
|
+
} catch (error) {
|
|
369
|
+
last = new Error(`${url} could not be read (${error.cause?.code ?? error.message}) — is the site reachable from this runner?`)
|
|
370
|
+
}
|
|
371
|
+
if (i + 1 < attempts) await new Promise((r) => setTimeout(r, 1000 * (i + 1)))
|
|
372
|
+
}
|
|
373
|
+
throw last
|
|
374
|
+
}
|
|
375
|
+
|
|
376
|
+
/** The bytes at `url`, or null on 404. Any other failure throws. */
|
|
377
|
+
async function readBytes(fetchImpl, url) {
|
|
378
|
+
const res = await get(fetchImpl, url)
|
|
379
|
+
if (res.status === 404) return null
|
|
380
|
+
if (!res.ok) throw new Error(`${url} answered ${res.status}`)
|
|
381
|
+
return Buffer.from(await res.arrayBuffer())
|
|
382
|
+
}
|
|
383
|
+
|
|
384
|
+
async function hashUrl(fetchImpl, url) {
|
|
385
|
+
const res = await get(fetchImpl, url)
|
|
386
|
+
if (!res.ok) return { status: res.status }
|
|
387
|
+
const h = createHash('sha256')
|
|
388
|
+
for await (const chunk of res.body) h.update(chunk)
|
|
389
|
+
return { status: res.status, sha256: h.digest('hex') }
|
|
390
|
+
}
|
|
391
|
+
|
|
392
|
+
/**
|
|
393
|
+
* Fails clearly, before anything else is tried, when the site's host does not resolve from this runner — the site
|
|
394
|
+
* may be reachable only on a private network, and only a runner on it can publish and read back.
|
|
395
|
+
*/
|
|
396
|
+
export async function assertResolves(baseUrl, resolver = lookup) {
|
|
397
|
+
const host = new URL(siteRoot(baseUrl)).hostname
|
|
398
|
+
try {
|
|
399
|
+
await resolver(host)
|
|
400
|
+
} catch (error) {
|
|
401
|
+
throw new Error(
|
|
402
|
+
`${host} does not resolve from this runner (${error.code ?? error.message}) — the site may be private-network only: run the job on a runner that can reach it (the workflow's runs-on input)`,
|
|
403
|
+
)
|
|
404
|
+
}
|
|
405
|
+
}
|
|
406
|
+
|
|
407
|
+
/**
|
|
408
|
+
* Uploads a staged tree (see `stage`) in the order the header gives. `dest` is the rsync destination (`user@host:`
|
|
409
|
+
* in production — the account's root is the app's folder), `sshArgs` the `-e` command line.
|
|
410
|
+
*/
|
|
411
|
+
export async function publish({
|
|
412
|
+
outDir,
|
|
413
|
+
app,
|
|
414
|
+
dest,
|
|
415
|
+
sshArgs,
|
|
416
|
+
baseUrl,
|
|
417
|
+
dryRun = false,
|
|
418
|
+
rsync = 'rsync',
|
|
419
|
+
fetchImpl = fetch,
|
|
420
|
+
log = console.log,
|
|
421
|
+
}) {
|
|
422
|
+
const root = `${siteRoot(baseUrl)}/${app}`
|
|
423
|
+
const latestBytes = readFileSync(join(outDir, 'latest.json'))
|
|
424
|
+
const { version, files } = JSON.parse(latestBytes.toString('utf8'))
|
|
425
|
+
const localSums = readFileSync(join(outDir, version, 'SHA256SUMS'))
|
|
426
|
+
const rootFiles = readdirSync(outDir).filter((n) => n !== 'latest.json' && n !== version && n !== 'manifest.json')
|
|
427
|
+
rootFiles.sort()
|
|
428
|
+
rootFiles.push('manifest.json') // installers first, manifest after them, latest.json last
|
|
429
|
+
const out = { version, uploaded: [], unchanged: [], skippedLatest: false, dryRun }
|
|
430
|
+
const slash = /[/:]$/.test(dest) ? '' : '/'
|
|
431
|
+
|
|
432
|
+
const remoteSums = await readBytes(fetchImpl, `${root}/${version}/SHA256SUMS`)
|
|
433
|
+
if (remoteSums !== null && !remoteSums.equals(localSums)) {
|
|
434
|
+
throw new Error(
|
|
435
|
+
`${version} is already published and its SHA256SUMS differs from this build's — versions are immutable, nothing was changed; publish a new version instead`,
|
|
436
|
+
)
|
|
437
|
+
}
|
|
438
|
+
log(remoteSums === null ? `${version}: not complete on the site yet` : `${version}: complete on the site — verifying only`)
|
|
439
|
+
|
|
440
|
+
const remoteLatest = await readBytes(fetchImpl, `${root}/latest.json`)
|
|
441
|
+
let remoteVersion = null
|
|
442
|
+
if (remoteLatest !== null) {
|
|
443
|
+
try {
|
|
444
|
+
const v = JSON.parse(remoteLatest.toString('utf8')).version
|
|
445
|
+
if (typeof v === 'string' && /^\d+\.\d+\.\d+$/.test(v)) remoteVersion = v
|
|
446
|
+
} catch {
|
|
447
|
+
remoteVersion = null
|
|
448
|
+
}
|
|
449
|
+
}
|
|
450
|
+
out.skippedLatest = remoteVersion !== null && compareVersions(remoteVersion, version) > 0
|
|
451
|
+
|
|
452
|
+
const common = { rsync, sshArgs, log }
|
|
453
|
+
if (dryRun) {
|
|
454
|
+
log('dry run — nothing is uploaded. Would send, in this order:')
|
|
455
|
+
if (remoteSums === null) {
|
|
456
|
+
for (const n of Object.keys(files)) log(` ${version}/${n}`)
|
|
457
|
+
log(` ${version}/SHA256SUMS`)
|
|
458
|
+
}
|
|
459
|
+
for (const n of rootFiles) log(` ${n} (when the site's copy differs)`)
|
|
460
|
+
log(out.skippedLatest ? ` (latest.json stays: the site names ${remoteVersion})` : ' latest.json (when the site\'s copy differs)')
|
|
461
|
+
return out
|
|
462
|
+
}
|
|
463
|
+
|
|
464
|
+
if (remoteSums === null) {
|
|
465
|
+
runRsync({ ...common, args: ['-rt', '--ignore-existing', '--exclude', 'SHA256SUMS', join(outDir, version), dest] })
|
|
466
|
+
// Check every file as a visitor would get it before the folder is declared complete.
|
|
467
|
+
for (const [name, want] of Object.entries(files)) {
|
|
468
|
+
const got = await hashUrl(fetchImpl, `${root}/${version}/${name}`)
|
|
469
|
+
if (got.sha256 !== want.sha256) {
|
|
470
|
+
throw new Error(
|
|
471
|
+
`${version}/${name} on the site is ${got.status === 200 ? 'a different file' : `answering ${got.status}`} than this build's — the folder was left without SHA256SUMS; a person removes that file on the server, then dispatch the same tag again`,
|
|
472
|
+
)
|
|
473
|
+
}
|
|
474
|
+
out.uploaded.push(`${version}/${name}`)
|
|
475
|
+
}
|
|
476
|
+
runRsync({ ...common, args: ['-t', join(outDir, version, 'SHA256SUMS'), `${dest}${slash}${version}/`] })
|
|
477
|
+
out.uploaded.push(`${version}/SHA256SUMS`)
|
|
478
|
+
} else {
|
|
479
|
+
for (const [name, want] of Object.entries(files)) {
|
|
480
|
+
const got = await hashUrl(fetchImpl, `${root}/${version}/${name}`)
|
|
481
|
+
if (got.sha256 !== want.sha256)
|
|
482
|
+
throw new Error(`${version}/${name} on the site does not match its SHA256SUMS (${got.status})`)
|
|
483
|
+
}
|
|
484
|
+
}
|
|
485
|
+
|
|
486
|
+
// The app-root files are replaced only when they differ, so a re-run sends nothing.
|
|
487
|
+
const mutable = out.skippedLatest ? rootFiles : [...rootFiles, 'latest.json']
|
|
488
|
+
for (const name of mutable) {
|
|
489
|
+
const local = readFileSync(join(outDir, name))
|
|
490
|
+
const remote = await readBytes(fetchImpl, `${root}/${name}`)
|
|
491
|
+
if (remote !== null && remote.equals(local)) {
|
|
492
|
+
out.unchanged.push(name)
|
|
493
|
+
continue
|
|
494
|
+
}
|
|
495
|
+
runRsync({ ...common, args: ['-t', join(outDir, name), dest] })
|
|
496
|
+
out.uploaded.push(name)
|
|
497
|
+
}
|
|
498
|
+
if (out.skippedLatest) log(`latest.json stays: the site already names ${remoteVersion}, newer than ${version}`)
|
|
499
|
+
|
|
500
|
+
// Read back what a visitor reads.
|
|
501
|
+
const sumsBack = await readBytes(fetchImpl, `${root}/${version}/SHA256SUMS`)
|
|
502
|
+
if (sumsBack === null || !sumsBack.equals(localSums))
|
|
503
|
+
throw new Error(`${root}/${version}/SHA256SUMS does not read back as this build's`)
|
|
504
|
+
for (const name of mutable) {
|
|
505
|
+
const back = await readBytes(fetchImpl, `${root}/${name}`)
|
|
506
|
+
if (back === null || !back.equals(readFileSync(join(outDir, name))))
|
|
507
|
+
throw new Error(`${root}/${name} does not read back as the file just sent`)
|
|
508
|
+
}
|
|
509
|
+
return out
|
|
510
|
+
}
|
|
511
|
+
|
|
512
|
+
function sshCommand(env) {
|
|
513
|
+
const need = (k) => {
|
|
514
|
+
if (!env[k]) throw new Error(`${k} is not set — the upload secrets are missing on this repo`)
|
|
515
|
+
return env[k]
|
|
516
|
+
}
|
|
517
|
+
const port = need('DOWNLOADS_SSH_PORT')
|
|
518
|
+
if (!/^\d+$/.test(port)) throw new Error('DOWNLOADS_SSH_PORT is not a port number')
|
|
519
|
+
const key = need('DOWNLOADS_KEY_FILE')
|
|
520
|
+
const known = need('DOWNLOADS_KNOWN_HOSTS_FILE')
|
|
521
|
+
for (const f of [key, known]) if (!existsSync(f)) throw new Error(`${f} does not exist`)
|
|
522
|
+
return {
|
|
523
|
+
dest: `${need('DOWNLOADS_SSH_USER')}@${need('DOWNLOADS_SSH_HOST')}:`,
|
|
524
|
+
sshArgs: `ssh -p ${port} -i ${key} -o UserKnownHostsFile=${known} -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes`,
|
|
525
|
+
}
|
|
526
|
+
}
|
|
527
|
+
|
|
528
|
+
async function main(argv, env) {
|
|
529
|
+
const rootAt = argv.indexOf('--root')
|
|
530
|
+
const root = resolve(rootAt >= 0 ? argv[rootAt + 1] ?? '' : '.')
|
|
531
|
+
const args = argv.filter((a, i) => !(rootAt >= 0 && (i === rootAt || i === rootAt + 1)))
|
|
532
|
+
const [command, tag, ...rest] = args
|
|
533
|
+
const usage = 'usage: publish-downloads.mjs check | names <tag> | publish <tag> <assetsDir> [--dry-run] [--root <repo>]'
|
|
534
|
+
if (!['check', 'names', 'publish'].includes(command)) throw new Error(usage)
|
|
535
|
+
const { config, manifest } = loadRepo(root, env)
|
|
536
|
+
if (command === 'check') {
|
|
537
|
+
console.log(`downloads: ${config.app} — ${config.files.length} file(s), ${config.optional.length} optional, ${config.installers.length} installer(s), manifest ${config.manifest} valid`)
|
|
538
|
+
return 0
|
|
539
|
+
}
|
|
540
|
+
if (command === 'names') {
|
|
541
|
+
const v = finalVersion(tag)
|
|
542
|
+
console.log([...namesFor(config.files, v), ...namesFor(config.optional, v)].join('\n'))
|
|
543
|
+
return 0
|
|
544
|
+
}
|
|
545
|
+
const dryRun = rest.includes('--dry-run')
|
|
546
|
+
const assetsDir = resolve(rest.filter((a) => !a.startsWith('--'))[0] ?? '')
|
|
547
|
+
const baseUrl = env.DOWNLOADS_URL
|
|
548
|
+
const outDir = mkdtempSync(join(tmpdir(), 'publish-downloads-'))
|
|
549
|
+
const staged = await stage({ tag, assetsDir, outDir, baseUrl, root, config, manifest })
|
|
550
|
+
console.log(`Staged ${config.app} ${staged.version}: ${Object.keys(staged.files).join(', ')}`)
|
|
551
|
+
await assertResolves(baseUrl)
|
|
552
|
+
const target = dryRun ? { dest: 'DRY-RUN:', sshArgs: undefined } : sshCommand(env)
|
|
553
|
+
const result = await publish({ outDir, app: config.app, ...target, baseUrl, dryRun })
|
|
554
|
+
if (dryRun) console.log(`Dry run of ${result.version} done — nothing was uploaded`)
|
|
555
|
+
else
|
|
556
|
+
console.log(
|
|
557
|
+
`Published ${config.app} ${result.version}: sent ${result.uploaded.length ? result.uploaded.join(', ') : 'nothing (a verified no-op)'}` +
|
|
558
|
+
`${result.unchanged.length ? `; unchanged ${result.unchanged.join(', ')}` : ''}${result.skippedLatest ? ' (latest.json names a newer version)' : ''}`,
|
|
559
|
+
)
|
|
560
|
+
return 0
|
|
561
|
+
}
|
|
562
|
+
|
|
563
|
+
// realpath: a script run through a symlinked path (macOS /var → /private/var) is still this script.
|
|
564
|
+
if (process.argv[1] && import.meta.url === pathToFileURL(realpathSync(process.argv[1])).href) {
|
|
565
|
+
main(process.argv.slice(2), process.env).then(
|
|
566
|
+
(code) => {
|
|
567
|
+
process.exitCode = code
|
|
568
|
+
},
|
|
569
|
+
(error) => {
|
|
570
|
+
console.error(error instanceof Error ? error.message : String(error))
|
|
571
|
+
process.exitCode = 1
|
|
572
|
+
},
|
|
573
|
+
)
|
|
574
|
+
}
|
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# Publish downloads — TEMPLATE. Copy me into your repo at .github/workflows/publish-downloads.yml.
|
|
2
|
+
#
|
|
3
|
+
# This is a starting point, not a dependency. Once you copy it, YOU OWN THIS FILE.
|
|
4
|
+
# Edit it freely; nothing calls this file remotely and nothing will update it for you.
|
|
5
|
+
#
|
|
6
|
+
# EDIT POINTS are marked `# EDIT:` — walk them once after copying.
|
|
7
|
+
#
|
|
8
|
+
# It needs its script in your repo (copy-and-own as well), at the path PUBLISH_SCRIPT names below:
|
|
9
|
+
# colab template publish-downloads.mjs --dest .github/downloads/publish-downloads.mjs
|
|
10
|
+
#
|
|
11
|
+
# For: a repo whose FINAL releases carry files people install on their own machines (a .dmg, a .deb, an
|
|
12
|
+
# .msi, a tarball …), published to a self-hosted downloads site — one folder per app, the newest finals kept
|
|
13
|
+
# by the site (colab-handbook #648; docs/downloads.md has the whole contract). What is published comes from
|
|
14
|
+
# the repo's `.github/project.yml` `downloads:` block plus a manifest.json the site's portal lists the app
|
|
15
|
+
# from; the script's header documents both.
|
|
16
|
+
#
|
|
17
|
+
# WHAT IT DOES for a final tag vX.Y.Z whose files are on its GitHub Release: uploads them with rsync over SSH
|
|
18
|
+
# in the order that keeps a half-finished run harmless — the version folder (with a combined SHA256SUMS last
|
|
19
|
+
# inside it, written only after every file read back over HTTPS against it), then the installers with the
|
|
20
|
+
# site address stamped in, then manifest.json, then latest.json LAST — and reads it all back. A published
|
|
21
|
+
# version is immutable: nothing is ever replaced, and a build whose SHA256SUMS differs from the published
|
|
22
|
+
# one is refused. A re-run of a complete version verifies and sends nothing. NEVER a candidate (-rc.N).
|
|
23
|
+
#
|
|
24
|
+
# WHEN IT RUNS.
|
|
25
|
+
# - Chained: your release/build workflow calls this one as its LAST job for a final tag, after the jobs
|
|
26
|
+
# that attach the files to the Release, and only when none of them failed:
|
|
27
|
+
# downloads:
|
|
28
|
+
# needs: [build-macos, build-linux] # the jobs that upload the Release's files
|
|
29
|
+
# if: ${{ !contains(github.ref_name, '-') }} # finals only
|
|
30
|
+
# uses: ./.github/workflows/publish-downloads.yml
|
|
31
|
+
# with: { tag: '${{ github.ref_name }}' }
|
|
32
|
+
# secrets: inherit
|
|
33
|
+
# (A tag pushed or a Release published with GITHUB_TOKEN starts no other workflow, so a chained call is
|
|
34
|
+
# the reliable trigger — not `on: release`.) Nothing is published from an incomplete Release: the script
|
|
35
|
+
# names what is missing and uploads nothing.
|
|
36
|
+
# - By hand (Actions → Run workflow) with a tag: retries a failed publish, finishes a partial one (only the
|
|
37
|
+
# missing files are sent), re-checks a complete one. `dry_run` stages and lists — it uploads nothing.
|
|
38
|
+
# The project.yml, manifest, installers and script are read from THIS run's checkout: the tag's commit when
|
|
39
|
+
# chained, the branch you dispatch from when run by hand.
|
|
40
|
+
#
|
|
41
|
+
# THE RUNNER. The read-back goes over HTTPS to the site, which may be reachable only on a private network —
|
|
42
|
+
# a GitHub-hosted runner then cannot read anything back. `runs-on` is an input: a bare label or a JSON label
|
|
43
|
+
# list of a runner that can reach the site. If the site's name does not resolve there, the run fails saying
|
|
44
|
+
# so, before anything is sent. The runner needs `gh`, `rsync` and `ssh`; Node is set up by the job.
|
|
45
|
+
#
|
|
46
|
+
# CONFIGURATION — per repo, never shared between repos:
|
|
47
|
+
# secrets DOWNLOADS_SSH_KEY · DOWNLOADS_SSH_HOST · DOWNLOADS_SSH_PORT · DOWNLOADS_SSH_USER ·
|
|
48
|
+
# DOWNLOADS_KNOWN_HOSTS — an upload account restricted to this app's folder, that cannot delete
|
|
49
|
+
# variable DOWNLOADS_URL — the site's root address (https). The script appends /<app>; no host is ever
|
|
50
|
+
# written in the repository.
|
|
51
|
+
#
|
|
52
|
+
# THIS FILE DOES NOT CHANGE YOUR REPO'S EXPOSURE (OR LEGACY TIER): it publishes files a final already made.
|
|
53
|
+
name: Publish downloads
|
|
54
|
+
|
|
55
|
+
on:
|
|
56
|
+
workflow_call:
|
|
57
|
+
inputs:
|
|
58
|
+
tag:
|
|
59
|
+
type: string
|
|
60
|
+
required: true
|
|
61
|
+
dry_run:
|
|
62
|
+
type: boolean
|
|
63
|
+
default: false
|
|
64
|
+
runs-on:
|
|
65
|
+
type: string
|
|
66
|
+
default: '["self-hosted","linux"]' # EDIT: a runner that can reach the downloads site
|
|
67
|
+
reserved:
|
|
68
|
+
type: string
|
|
69
|
+
default: '' # EDIT: names the site keeps for itself, refused as an app slug (space-separated)
|
|
70
|
+
workflow_dispatch:
|
|
71
|
+
inputs:
|
|
72
|
+
tag:
|
|
73
|
+
description: 'Existing FINAL tag to publish, e.g. v1.4.0 (a candidate is refused)'
|
|
74
|
+
type: string
|
|
75
|
+
required: true
|
|
76
|
+
dry_run:
|
|
77
|
+
description: 'Dry run: stage and list what would be sent — nothing is uploaded'
|
|
78
|
+
type: boolean
|
|
79
|
+
default: false
|
|
80
|
+
runs-on:
|
|
81
|
+
description: 'Runner: a bare label or a JSON label list of a runner that can reach the site'
|
|
82
|
+
type: string
|
|
83
|
+
default: '["self-hosted","linux"]' # EDIT: keep in step with the workflow_call default
|
|
84
|
+
reserved:
|
|
85
|
+
description: 'Names the site reserves for itself, refused as an app slug (space-separated)'
|
|
86
|
+
type: string
|
|
87
|
+
default: '' # EDIT: keep in step with the workflow_call default
|
|
88
|
+
|
|
89
|
+
permissions: {}
|
|
90
|
+
|
|
91
|
+
concurrency:
|
|
92
|
+
group: publish-downloads
|
|
93
|
+
cancel-in-progress: false
|
|
94
|
+
|
|
95
|
+
env:
|
|
96
|
+
PUBLISH_SCRIPT: .github/downloads/publish-downloads.mjs # EDIT: where you copied the script
|
|
97
|
+
|
|
98
|
+
jobs:
|
|
99
|
+
publish:
|
|
100
|
+
name: Publish ${{ inputs.tag }}
|
|
101
|
+
runs-on: ${{ startsWith(inputs.runs-on, '[') && fromJSON(inputs.runs-on) || inputs.runs-on }}
|
|
102
|
+
timeout-minutes: 45 # EDIT: above your own p99 — large installers over a slow link take a while
|
|
103
|
+
permissions:
|
|
104
|
+
contents: read
|
|
105
|
+
env:
|
|
106
|
+
GH_TOKEN: ${{ github.token }}
|
|
107
|
+
TAG: ${{ inputs.tag }}
|
|
108
|
+
DRY_RUN: ${{ inputs.dry_run }}
|
|
109
|
+
DOWNLOADS_URL: ${{ vars.DOWNLOADS_URL }}
|
|
110
|
+
DOWNLOADS_RESERVED: ${{ inputs.reserved }}
|
|
111
|
+
DOWNLOADS_SSH_HOST: ${{ secrets.DOWNLOADS_SSH_HOST }}
|
|
112
|
+
DOWNLOADS_SSH_PORT: ${{ secrets.DOWNLOADS_SSH_PORT }}
|
|
113
|
+
DOWNLOADS_SSH_USER: ${{ secrets.DOWNLOADS_SSH_USER }}
|
|
114
|
+
steps:
|
|
115
|
+
- name: A final tag only
|
|
116
|
+
run: |
|
|
117
|
+
set -euo pipefail
|
|
118
|
+
[[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] || { echo "::error::$TAG is not a final tag (vX.Y.Z): only finals are published"; exit 1; }
|
|
119
|
+
|
|
120
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
121
|
+
with:
|
|
122
|
+
persist-credentials: false
|
|
123
|
+
|
|
124
|
+
- name: Setup Node
|
|
125
|
+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
|
|
126
|
+
with:
|
|
127
|
+
node-version: '22' # EDIT: any Node >= 18 runs the script
|
|
128
|
+
|
|
129
|
+
- name: The tools the run needs
|
|
130
|
+
run: |
|
|
131
|
+
set -euo pipefail
|
|
132
|
+
need="gh"; [ "$DRY_RUN" = "true" ] || need="gh rsync ssh"
|
|
133
|
+
for t in $need; do command -v "$t" >/dev/null || { echo "::error::$t is not on this runner — pick one that has it (the runs-on input)"; exit 1; }; done
|
|
134
|
+
node "$PUBLISH_SCRIPT" check
|
|
135
|
+
|
|
136
|
+
- name: Download the Release's files
|
|
137
|
+
id: assets
|
|
138
|
+
run: |
|
|
139
|
+
set -euo pipefail
|
|
140
|
+
DIR="$RUNNER_TEMP/assets"; rm -rf "$DIR"; mkdir -p "$DIR"
|
|
141
|
+
gh release view "$TAG" --repo "$GITHUB_REPOSITORY" --json assets -q '.assets[].name' > "$RUNNER_TEMP/release-assets.txt"
|
|
142
|
+
while IFS= read -r name; do
|
|
143
|
+
if grep -qxF -- "$name" "$RUNNER_TEMP/release-assets.txt"; then
|
|
144
|
+
gh release download "$TAG" --repo "$GITHUB_REPOSITORY" --pattern "$name" --dir "$DIR"
|
|
145
|
+
fi
|
|
146
|
+
done < <(node "$PUBLISH_SCRIPT" names "$TAG")
|
|
147
|
+
ls -l "$DIR"
|
|
148
|
+
echo "dir=$DIR" >> "$GITHUB_OUTPUT"
|
|
149
|
+
|
|
150
|
+
- name: Write the upload key (this run only)
|
|
151
|
+
if: inputs.dry_run != true
|
|
152
|
+
env:
|
|
153
|
+
KEY: ${{ secrets.DOWNLOADS_SSH_KEY }}
|
|
154
|
+
KNOWN_HOSTS: ${{ secrets.DOWNLOADS_KNOWN_HOSTS }}
|
|
155
|
+
run: |
|
|
156
|
+
set -euo pipefail
|
|
157
|
+
umask 077
|
|
158
|
+
[ -n "$KEY" ] && [ -n "$KNOWN_HOSTS" ] || { echo "::error::the downloads upload secrets are not set on this repo"; exit 1; }
|
|
159
|
+
mkdir -p "$RUNNER_TEMP/downloads-ssh"
|
|
160
|
+
printf '%s\n' "$KEY" > "$RUNNER_TEMP/downloads-ssh/key"
|
|
161
|
+
printf '%s\n' "$KNOWN_HOSTS" > "$RUNNER_TEMP/downloads-ssh/known_hosts"
|
|
162
|
+
chmod 600 "$RUNNER_TEMP/downloads-ssh/key" "$RUNNER_TEMP/downloads-ssh/known_hosts"
|
|
163
|
+
|
|
164
|
+
- name: Publish
|
|
165
|
+
env:
|
|
166
|
+
ASSETS: ${{ steps.assets.outputs.dir }}
|
|
167
|
+
DOWNLOADS_KEY_FILE: ${{ runner.temp }}/downloads-ssh/key
|
|
168
|
+
DOWNLOADS_KNOWN_HOSTS_FILE: ${{ runner.temp }}/downloads-ssh/known_hosts
|
|
169
|
+
run: |
|
|
170
|
+
set -euo pipefail
|
|
171
|
+
FLAG=""; [ "$DRY_RUN" = "true" ] && FLAG="--dry-run"
|
|
172
|
+
# The script's own status is kept: a `{ …; echo; } | tee` group would end on the echo and go green.
|
|
173
|
+
status=0
|
|
174
|
+
{ echo "### Downloads → $TAG"; echo; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
|
|
175
|
+
node "$PUBLISH_SCRIPT" publish "$TAG" "$ASSETS" $FLAG 2>&1 | tee -a "$GITHUB_STEP_SUMMARY" || status=$?
|
|
176
|
+
echo '```' >> "$GITHUB_STEP_SUMMARY"
|
|
177
|
+
exit "$status"
|
|
178
|
+
|
|
179
|
+
- name: Remove the upload key
|
|
180
|
+
if: always()
|
|
181
|
+
run: rm -rf "$RUNNER_TEMP/downloads-ssh"
|