@futurelastic/colab-handbook 1.20.4-rc.4 → 1.20.4-rc.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/templates/README.md +3 -3
- package/templates/ci-laravel-heavy.yml +132 -2
- package/templates/ci-laravel.yml +36 -2
- package/templates/ci-node-heavy.yml +136 -2
- package/templates/ci-node.yml +36 -3
- package/templates/ci-python-heavy.yml +136 -2
- package/templates/ci-python.yml +36 -3
- package/templates/publish-downloads.mjs +30 -1
- package/templates/publish-downloads.yml +5 -0
- package/templates/release-auto.yml +47 -3
- package/tools/lib/stamp.js +1 -1
package/package.json
CHANGED
package/templates/README.md
CHANGED
|
@@ -27,8 +27,8 @@ or warn. The handbook enforces only what `colab ship` reads from your CI
|
|
|
27
27
|
|
|
28
28
|
| Lane | Template → copy | Runs | What |
|
|
29
29
|
|---|---|---|---|
|
|
30
|
-
| **light** | `ci-<stack>.yml` → `ci.yml` | every push (branch and trunk); the lane `colab ship` reads | secret scan, build, typecheck/lint, the fast tests — aim for under 5 minutes at p50 |
|
|
31
|
-
| **heavy** | `ci-<stack>-heavy.yml` → `ci-heavy.yml` | nightly on trunk's tip (skipped when trunk has not moved since the last green nightly), at every release candidate and tag, and by hand — **never per merge** | the whole suite including heavy-marked tests, integration, browser/e2e, load |
|
|
30
|
+
| **light** | `ci-<stack>.yml` → `ci.yml` | every push (branch and trunk); the lane `colab ship` reads | secret scan of the commits the run brings (#716), build, typecheck/lint, the fast tests — aim for under 5 minutes at p50 |
|
|
31
|
+
| **heavy** | `ci-<stack>-heavy.yml` → `ci-heavy.yml` | nightly on trunk's tip (skipped when trunk has not moved since the last green nightly), at every release candidate and tag, and by hand — **never per merge** | the whole suite including heavy-marked tests, integration, browser/e2e, load; the secret scan of trunk's whole history (job `secrets`, #716) |
|
|
32
32
|
|
|
33
33
|
- **The adopter marks which tests are heavy; an unmarked test is light**, so nothing stops
|
|
34
34
|
running per merge until someone decides it should.
|
|
@@ -104,7 +104,7 @@ check that reads one of these files is that tool's behaviour, documented with th
|
|
|
104
104
|
|---|---|---|---|
|
|
105
105
|
| `ci-node.yml` | `.github/workflows/ci.yml` | Pure Node repos: Vite SPA, node libs, Astro static, **Capacitor apps** | Resolves Node version from `project.yml` → `.nvmrc`/`engines` → **fails**. Never a default. Opt-in `Migration round-trip` job — `--roundtrip` writes it live, see *Migration round-trip* below. |
|
|
106
106
|
| `ci-laravel.yml` | `.github/workflows/ci.yml` | Laravel + Inertia + Vite fullstack (with route/type codegen) | Same resolution for **both** PHP and Node. Includes the sqlite bootstrap + explicit wayfinder step, and a `migrations` job that round-trips the schema (migrate → rollback → seed → migrate, schema compared) on the test engine **and** the deploy engine (a MariaDB/MySQL service container at production's version). Its data-dependent checks need `ROUNDTRIP_SEEDER` — see *Migration round-trip* below. |
|
|
107
|
-
| `ci-node-heavy.yml`, `ci-python-heavy.yml`, `ci-laravel-heavy.yml` | `.github/workflows/ci-heavy.yml` | The heavy lane beside the matching `ci-<stack>.yml` (above) | `schedule` + `workflow_dispatch` + `workflow_call`, never a push. Toolchain and install steps are copied from the light lane's `build` job, so the two resolve the same versions — keep them in step. `TRUNK` is an edit point. |
|
|
107
|
+
| `ci-node-heavy.yml`, `ci-python-heavy.yml`, `ci-laravel-heavy.yml` | `.github/workflows/ci-heavy.yml` | The heavy lane beside the matching `ci-<stack>.yml` (above) | `schedule` + `workflow_dispatch` + `workflow_call`, never a push. Toolchain and install steps are copied from the light lane's `build` job, so the two resolve the same versions — keep them in step; the `secrets` job's gitleaks install and `GITLEAKS_VERSION` likewise follow the light lane's `gitleaks` job. `TRUNK` is an edit point. |
|
|
108
108
|
| `ci-python.yml` | `.github/workflows/ci.yml` | Python: FastAPI/Flask services, CLIs, libraries | Resolves Python from `project.yml` → `.python-version`/`requires-python` → **fails**. `requirements.txt` does not count. **Hybrid** Python+Node repo: copy this, then paste `ci-node.yml`'s `build:` job alongside — see the template header. Opt-in `Migration round-trip` job, as in `ci-node.yml`. |
|
|
109
109
|
| `release-tag.yml` | `.github/workflows/release.yml` | Any repo cutting `v*.*.*` releases | Triggers on tag push. Publishes a grouped GitHub Release — a `-rc.N` tag as a pre-release, every summary since the previous *final* tag. No toolchain, no deploy; its header shows the `!v*.*.*-*` exclusion a deploy trigger needs. |
|
|
110
110
|
| `release-auto.yml` | `.github/workflows/release-auto.yml` | A repo whose release tags land on `main` (`trunk: main`, `deploy: tag` or `none`) and whose §6 release route cuts candidates | Runs `colab release cut --auto` after CI goes green on main — at the sha that CI run tested (`--sha`, #622), even once main has moved on — and `colab release finalize --auto` daily — plus an hourly run that re-tries only a refused cut, once the fetched CLI has moved within `RETRY_WINDOW_MINUTES` or a run at main's head was cancelled (#547) — from a handbook ref (`HANDBOOK_REF` — the `stable` channel by default, `next` for the fast channel, or an exact tag to freeze; #445) — the handbook's own copy runs its checkout's CLI instead, since it ships the CLI it releases with (#428). **Publishes the Release in the same run** — a tag pushed with `GITHUB_TOKEN` triggers no other workflow, so `release-tag.yml` would never fire for it. A refusal is a warning, not a red run. Reads and creates tags, and fast-forwards the release channels `next` / `stable` (branches naming the newest candidate / final, #445); never pushes a commit. The audit recognises any workflow running `release cut/finalize --auto` and warns when the route leaves candidates off, when it tags without publishing while a tag-push workflow waits, or when it commits. |
|
|
@@ -47,6 +47,8 @@ on:
|
|
|
47
47
|
permissions: {}
|
|
48
48
|
|
|
49
49
|
env:
|
|
50
|
+
# EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
|
|
51
|
+
GITLEAKS_VERSION: "8.30.1"
|
|
50
52
|
# EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
|
|
51
53
|
# default branch; the nightly tests THIS branch's tip.
|
|
52
54
|
TRUNK: main
|
|
@@ -328,13 +330,141 @@ jobs:
|
|
|
328
330
|
# EDIT: integration, browser/e2e (Dusk) or load jobs go here, or as jobs of their own with
|
|
329
331
|
# `needs: plan` and the same `if:` and checkout ref as this one.
|
|
330
332
|
|
|
333
|
+
# The full-history secret scan (#716). The light lane's scan reads only the commits its run
|
|
334
|
+
# brings; this one reads every commit trunk's tip can reach — a secret committed and removed
|
|
335
|
+
# before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
|
|
336
|
+
# no merge: a red nightly files the trunk-red issue below, and a red official build holds that
|
|
337
|
+
# release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
|
|
338
|
+
# branch's own run's to report.
|
|
339
|
+
secrets:
|
|
340
|
+
name: Secret scan, full history (gitleaks)
|
|
341
|
+
needs: plan
|
|
342
|
+
if: ${{ needs.plan.outputs.run == 'true' }}
|
|
343
|
+
runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
|
|
344
|
+
timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
|
|
345
|
+
permissions:
|
|
346
|
+
contents: read
|
|
347
|
+
steps:
|
|
348
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
349
|
+
with:
|
|
350
|
+
ref: ${{ needs.plan.outputs.sha }}
|
|
351
|
+
fetch-depth: 0 # every commit the tip reaches
|
|
352
|
+
persist-credentials: false
|
|
353
|
+
|
|
354
|
+
# The light lane's restore and install steps, verbatim: keep them in step when you edit either.
|
|
355
|
+
# Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
|
|
356
|
+
# binary comes back from here and the install step below downloads nothing.
|
|
357
|
+
# - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
|
|
358
|
+
# remote cache on top was measured as a loss, not a saving (#512).
|
|
359
|
+
# - Keyed on the pin and the platform: a version bump is a miss. The post step saves
|
|
360
|
+
# only when the job passes, so a failed download is never what gets cached.
|
|
361
|
+
# A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
|
|
362
|
+
# its name: handbook-sync recognises a copy of this job by it.
|
|
363
|
+
# EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
|
|
364
|
+
- name: Restore gitleaks cache
|
|
365
|
+
if: runner.environment == 'github-hosted'
|
|
366
|
+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
367
|
+
with:
|
|
368
|
+
path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
|
|
369
|
+
key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
|
|
370
|
+
|
|
371
|
+
- name: Install gitleaks (pinned release binary)
|
|
372
|
+
env:
|
|
373
|
+
# Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
|
|
374
|
+
# measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
|
|
375
|
+
# while the API asset path below answered 200 with the same file. The job's own token
|
|
376
|
+
# is enough — reading a public release needs an identity, not a permission.
|
|
377
|
+
GH_TOKEN: ${{ github.token }}
|
|
378
|
+
run: |
|
|
379
|
+
# NOT gitleaks/gitleaks-action@v2 — it demands a paid GITLEAKS_LICENSE for
|
|
380
|
+
# organization-owned repositories. The plain release binary is MIT.
|
|
381
|
+
# Resolve OS/arch from uname — a hardcoded linux_x64 binary on a macOS
|
|
382
|
+
# self-hosted runner cannot exec, and fails as a bare exit 126.
|
|
383
|
+
case "$(uname -s)" in
|
|
384
|
+
Darwin) os=darwin ;;
|
|
385
|
+
Linux) os=linux ;;
|
|
386
|
+
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
|
|
387
|
+
esac
|
|
388
|
+
case "$(uname -m)" in
|
|
389
|
+
arm64|aarch64) arch=arm64 ;;
|
|
390
|
+
x86_64|amd64) arch=x64 ;;
|
|
391
|
+
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
|
|
392
|
+
esac
|
|
393
|
+
# Kept between jobs (#696): a pinned version is downloaded once per runner, not on
|
|
394
|
+
# every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
|
|
395
|
+
# which outlasts any retry, so the binary has to be on the runner already.
|
|
396
|
+
# Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
|
|
397
|
+
# per-runner directory setup-node and setup-python install into. A persistent
|
|
398
|
+
# self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
|
|
399
|
+
# "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
|
|
400
|
+
# miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
|
|
401
|
+
# downloads on every run.
|
|
402
|
+
dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
|
|
403
|
+
if [ -x "$dir/gitleaks" ]; then
|
|
404
|
+
echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
|
|
405
|
+
else
|
|
406
|
+
mkdir -p "$dir"
|
|
407
|
+
tmp="$(mktemp -d "$dir.XXXXXX")"
|
|
408
|
+
trap 'rm -rf "$tmp"' EXIT
|
|
409
|
+
asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
|
|
410
|
+
sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
|
|
411
|
+
# fetch <accept> <url> <file> — every request of this step, one curl line.
|
|
412
|
+
# --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
|
|
413
|
+
# then fails with "not in gzip format", naming neither the download nor the host.
|
|
414
|
+
# --retry: a re-run during the same upstream error spends the one re-run branch CI
|
|
415
|
+
# allows, so a brief error is retried here first. A longer outage still fails, at
|
|
416
|
+
# curl, naming the status (#690).
|
|
417
|
+
# The token reaches api.github.com only: curl drops an -H Authorization header when
|
|
418
|
+
# the asset's 302 leads to another host, so the download CDN never sees it.
|
|
419
|
+
fetch() {
|
|
420
|
+
curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
|
|
421
|
+
-H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
|
|
422
|
+
}
|
|
423
|
+
# The API serves an asset by id, not by name, so look the ids up in the release. No jq:
|
|
424
|
+
# a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
|
|
425
|
+
# whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
|
|
426
|
+
# that order ever to change, the wrong file would arrive and the sha256 check below
|
|
427
|
+
# would refuse it — a loud failure, never a wrong binary.
|
|
428
|
+
fetch application/vnd.github+json \
|
|
429
|
+
"https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
|
|
430
|
+
for f in "$asset" "$sums"; do
|
|
431
|
+
url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
|
|
432
|
+
$2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
|
|
433
|
+
$2 == "name" && $4 == want && !found { print u; found = 1 }')"
|
|
434
|
+
if [ -z "$url" ]; then
|
|
435
|
+
echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
|
|
436
|
+
fi
|
|
437
|
+
fetch application/octet-stream "$url" "$tmp/$f"
|
|
438
|
+
done
|
|
439
|
+
# Verified before tar (#698): an error page, a truncated file or a wrong archive stops
|
|
440
|
+
# here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
|
|
441
|
+
want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
|
|
442
|
+
if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
|
|
443
|
+
else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
|
|
444
|
+
if [ -z "$want" ]; then
|
|
445
|
+
echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
|
|
446
|
+
fi
|
|
447
|
+
if [ "$got" != "$want" ]; then
|
|
448
|
+
echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
|
|
449
|
+
fi
|
|
450
|
+
tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
|
|
451
|
+
chmod +x "$tmp/gitleaks"
|
|
452
|
+
# One rename on one filesystem: a job sharing this tool cache sees no binary or a
|
|
453
|
+
# whole one, never half of one.
|
|
454
|
+
mv -f "$tmp/gitleaks" "$dir/gitleaks"
|
|
455
|
+
fi
|
|
456
|
+
echo "$dir" >> "$GITHUB_PATH"
|
|
457
|
+
|
|
458
|
+
- name: gitleaks detect (full history)
|
|
459
|
+
run: gitleaks detect --source . --no-banner --log-opts=HEAD
|
|
460
|
+
|
|
331
461
|
# A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
|
|
332
462
|
# with the same title suppresses a new one; an issue list that cannot be read files nothing and
|
|
333
463
|
# fails this job, so a silent miss is never mistaken for "already reported".
|
|
334
464
|
report:
|
|
335
465
|
name: Report a red nightly
|
|
336
|
-
needs: [plan, heavy]
|
|
337
|
-
if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
|
|
466
|
+
needs: [plan, heavy, secrets]
|
|
467
|
+
if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
|
|
338
468
|
runs-on: ubuntu-latest # EDIT: needs `gh`.
|
|
339
469
|
timeout-minutes: 5
|
|
340
470
|
permissions:
|
package/templates/ci-laravel.yml
CHANGED
|
@@ -256,7 +256,9 @@ jobs:
|
|
|
256
256
|
steps:
|
|
257
257
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
258
258
|
with:
|
|
259
|
-
# Full history: scan
|
|
259
|
+
# Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
|
|
260
|
+
# the two ends of that range have to be in the clone. The range itself is narrow — see
|
|
261
|
+
# "gitleaks detect" below (#716).
|
|
260
262
|
fetch-depth: 0
|
|
261
263
|
persist-credentials: false
|
|
262
264
|
|
|
@@ -363,8 +365,40 @@ jobs:
|
|
|
363
365
|
fi
|
|
364
366
|
echo "$dir" >> "$GITHUB_PATH"
|
|
365
367
|
|
|
368
|
+
# The scan range (#716): only the commits this run brings, never `git log --all`. A
|
|
369
|
+
# fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
|
|
370
|
+
# them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
|
|
371
|
+
# included, until that branch rewrote its history.
|
|
372
|
+
# - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
|
|
373
|
+
# - A trunk push: the pushed range (before..HEAD).
|
|
374
|
+
# - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
|
|
375
|
+
# own history — still never another branch's.
|
|
376
|
+
# Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
|
|
377
|
+
# commit trunk's tip can reach is scanned by the heavy lane instead (templates/
|
|
378
|
+
# ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
|
|
366
379
|
- name: gitleaks detect
|
|
367
|
-
|
|
380
|
+
env:
|
|
381
|
+
EVENT: ${{ github.event_name }}
|
|
382
|
+
REF_NAME: ${{ github.ref_name }}
|
|
383
|
+
BEFORE: ${{ github.event.before }}
|
|
384
|
+
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
|
385
|
+
run: |
|
|
386
|
+
trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
|
|
387
|
+
trunk="${trunk:-$DEFAULT_BRANCH}"
|
|
388
|
+
commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
|
|
389
|
+
if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
|
|
390
|
+
if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
|
|
391
|
+
range="$BEFORE..HEAD"
|
|
392
|
+
else
|
|
393
|
+
range=HEAD
|
|
394
|
+
fi
|
|
395
|
+
elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
|
|
396
|
+
range="refs/remotes/origin/$trunk..HEAD"
|
|
397
|
+
else
|
|
398
|
+
range=HEAD
|
|
399
|
+
fi
|
|
400
|
+
echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
|
|
401
|
+
gitleaks detect --source . --no-banner --log-opts="$range"
|
|
368
402
|
|
|
369
403
|
build:
|
|
370
404
|
name: Build (Laravel + Vite)
|
|
@@ -47,6 +47,8 @@ on:
|
|
|
47
47
|
permissions: {}
|
|
48
48
|
|
|
49
49
|
env:
|
|
50
|
+
# EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
|
|
51
|
+
GITLEAKS_VERSION: "8.30.1"
|
|
50
52
|
# EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
|
|
51
53
|
# default branch; the nightly tests THIS branch's tip.
|
|
52
54
|
TRUNK: main
|
|
@@ -251,13 +253,145 @@ jobs:
|
|
|
251
253
|
# EDIT: integration, browser/e2e or load jobs go here, or as jobs of their own with
|
|
252
254
|
# `needs: plan` and the same `if:` and checkout ref as this one.
|
|
253
255
|
|
|
256
|
+
# The full-history secret scan (#716). The light lane's scan reads only the commits its run
|
|
257
|
+
# brings; this one reads every commit trunk's tip can reach — a secret committed and removed
|
|
258
|
+
# before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
|
|
259
|
+
# no merge: a red nightly files the trunk-red issue below, and a red official build holds that
|
|
260
|
+
# release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
|
|
261
|
+
# branch's own run's to report.
|
|
262
|
+
secrets:
|
|
263
|
+
name: Secret scan, full history (gitleaks)
|
|
264
|
+
needs: plan
|
|
265
|
+
if: ${{ needs.plan.outputs.run == 'true' }}
|
|
266
|
+
runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
|
|
267
|
+
timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
|
|
268
|
+
permissions:
|
|
269
|
+
contents: read
|
|
270
|
+
steps:
|
|
271
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
272
|
+
with:
|
|
273
|
+
ref: ${{ needs.plan.outputs.sha }}
|
|
274
|
+
fetch-depth: 0 # every commit the tip reaches
|
|
275
|
+
persist-credentials: false
|
|
276
|
+
|
|
277
|
+
# The light lane's restore and install steps, verbatim: keep them in step when you edit either.
|
|
278
|
+
# Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
|
|
279
|
+
# binary comes back from here and the install step below downloads nothing.
|
|
280
|
+
# - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
|
|
281
|
+
# remote cache on top was measured as a loss, not a saving (#512).
|
|
282
|
+
# - Keyed on the pin and the platform: a version bump is a miss. The post step saves
|
|
283
|
+
# only when the job passes, so a failed download is never what gets cached.
|
|
284
|
+
# A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
|
|
285
|
+
# its name: handbook-sync recognises a copy of this job by it.
|
|
286
|
+
# EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
|
|
287
|
+
- name: Restore gitleaks cache
|
|
288
|
+
if: runner.environment == 'github-hosted'
|
|
289
|
+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
290
|
+
with:
|
|
291
|
+
path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
|
|
292
|
+
key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
|
|
293
|
+
|
|
294
|
+
- name: Install gitleaks (pinned release binary)
|
|
295
|
+
env:
|
|
296
|
+
# Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
|
|
297
|
+
# measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
|
|
298
|
+
# while the API asset path below answered 200 with the same file. The job's own token
|
|
299
|
+
# is enough — reading a public release needs an identity, not a permission.
|
|
300
|
+
GH_TOKEN: ${{ github.token }}
|
|
301
|
+
run: |
|
|
302
|
+
# Deliberately NOT gitleaks/gitleaks-action@v2: that action requires a paid
|
|
303
|
+
# GITLEAKS_LICENSE for repositories owned by an organization (free only for
|
|
304
|
+
# personal repos). The release binary itself is MIT and has no such gate.
|
|
305
|
+
# Repos also carry a local pre-commit gitleaks hook; this is the second layer.
|
|
306
|
+
# The asset is OS/arch-specific, so resolve both from uname instead of
|
|
307
|
+
# assuming the GitHub-hosted default: a hardcoded linux_x64 binary on a
|
|
308
|
+
# macOS self-hosted runner is an ELF the kernel cannot exec, and the step
|
|
309
|
+
# fails with exit 126 rather than anything mentioning the platform.
|
|
310
|
+
case "$(uname -s)" in
|
|
311
|
+
Darwin) os=darwin ;;
|
|
312
|
+
Linux) os=linux ;;
|
|
313
|
+
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
|
|
314
|
+
esac
|
|
315
|
+
case "$(uname -m)" in
|
|
316
|
+
arm64|aarch64) arch=arm64 ;;
|
|
317
|
+
x86_64|amd64) arch=x64 ;;
|
|
318
|
+
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
|
|
319
|
+
esac
|
|
320
|
+
# Kept between jobs (#696): a pinned version is downloaded once per runner, not on
|
|
321
|
+
# every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
|
|
322
|
+
# which outlasts any retry, so the binary has to be on the runner already.
|
|
323
|
+
# Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
|
|
324
|
+
# per-runner directory setup-node and setup-python install into. A persistent
|
|
325
|
+
# self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
|
|
326
|
+
# "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
|
|
327
|
+
# miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
|
|
328
|
+
# downloads on every run.
|
|
329
|
+
dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
|
|
330
|
+
if [ -x "$dir/gitleaks" ]; then
|
|
331
|
+
echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
|
|
332
|
+
else
|
|
333
|
+
mkdir -p "$dir"
|
|
334
|
+
tmp="$(mktemp -d "$dir.XXXXXX")"
|
|
335
|
+
trap 'rm -rf "$tmp"' EXIT
|
|
336
|
+
asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
|
|
337
|
+
sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
|
|
338
|
+
# fetch <accept> <url> <file> — every request of this step, one curl line.
|
|
339
|
+
# --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
|
|
340
|
+
# then fails with "not in gzip format", naming neither the download nor the host.
|
|
341
|
+
# --retry: a re-run during the same upstream error spends the one re-run branch CI
|
|
342
|
+
# allows, so a brief error is retried here first. A longer outage still fails, at
|
|
343
|
+
# curl, naming the status (#690).
|
|
344
|
+
# The token reaches api.github.com only: curl drops an -H Authorization header when
|
|
345
|
+
# the asset's 302 leads to another host, so the download CDN never sees it.
|
|
346
|
+
fetch() {
|
|
347
|
+
curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
|
|
348
|
+
-H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
|
|
349
|
+
}
|
|
350
|
+
# The API serves an asset by id, not by name, so look the ids up in the release. No jq:
|
|
351
|
+
# a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
|
|
352
|
+
# whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
|
|
353
|
+
# that order ever to change, the wrong file would arrive and the sha256 check below
|
|
354
|
+
# would refuse it — a loud failure, never a wrong binary.
|
|
355
|
+
fetch application/vnd.github+json \
|
|
356
|
+
"https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
|
|
357
|
+
for f in "$asset" "$sums"; do
|
|
358
|
+
url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
|
|
359
|
+
$2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
|
|
360
|
+
$2 == "name" && $4 == want && !found { print u; found = 1 }')"
|
|
361
|
+
if [ -z "$url" ]; then
|
|
362
|
+
echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
|
|
363
|
+
fi
|
|
364
|
+
fetch application/octet-stream "$url" "$tmp/$f"
|
|
365
|
+
done
|
|
366
|
+
# Verified before tar (#698): an error page, a truncated file or a wrong archive stops
|
|
367
|
+
# here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
|
|
368
|
+
want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
|
|
369
|
+
if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
|
|
370
|
+
else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
|
|
371
|
+
if [ -z "$want" ]; then
|
|
372
|
+
echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
|
|
373
|
+
fi
|
|
374
|
+
if [ "$got" != "$want" ]; then
|
|
375
|
+
echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
|
|
376
|
+
fi
|
|
377
|
+
tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
|
|
378
|
+
chmod +x "$tmp/gitleaks"
|
|
379
|
+
# One rename on one filesystem: a job sharing this tool cache sees no binary or a
|
|
380
|
+
# whole one, never half of one.
|
|
381
|
+
mv -f "$tmp/gitleaks" "$dir/gitleaks"
|
|
382
|
+
fi
|
|
383
|
+
echo "$dir" >> "$GITHUB_PATH"
|
|
384
|
+
|
|
385
|
+
- name: gitleaks detect (full history)
|
|
386
|
+
run: gitleaks detect --source . --no-banner --log-opts=HEAD
|
|
387
|
+
|
|
254
388
|
# A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
|
|
255
389
|
# with the same title suppresses a new one; an issue list that cannot be read files nothing and
|
|
256
390
|
# fails this job, so a silent miss is never mistaken for "already reported".
|
|
257
391
|
report:
|
|
258
392
|
name: Report a red nightly
|
|
259
|
-
needs: [plan, heavy]
|
|
260
|
-
if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
|
|
393
|
+
needs: [plan, heavy, secrets]
|
|
394
|
+
if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
|
|
261
395
|
runs-on: ubuntu-latest # EDIT: needs `gh`.
|
|
262
396
|
timeout-minutes: 5
|
|
263
397
|
permissions:
|
package/templates/ci-node.yml
CHANGED
|
@@ -241,8 +241,9 @@ jobs:
|
|
|
241
241
|
steps:
|
|
242
242
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
243
243
|
with:
|
|
244
|
-
# Full history:
|
|
245
|
-
#
|
|
244
|
+
# Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
|
|
245
|
+
# the two ends of that range have to be in the clone. The range itself is narrow — see
|
|
246
|
+
# "gitleaks detect" below (#716).
|
|
246
247
|
fetch-depth: 0
|
|
247
248
|
persist-credentials: false
|
|
248
249
|
|
|
@@ -353,8 +354,40 @@ jobs:
|
|
|
353
354
|
fi
|
|
354
355
|
echo "$dir" >> "$GITHUB_PATH"
|
|
355
356
|
|
|
357
|
+
# The scan range (#716): only the commits this run brings, never `git log --all`. A
|
|
358
|
+
# fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
|
|
359
|
+
# them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
|
|
360
|
+
# included, until that branch rewrote its history.
|
|
361
|
+
# - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
|
|
362
|
+
# - A trunk push: the pushed range (before..HEAD).
|
|
363
|
+
# - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
|
|
364
|
+
# own history — still never another branch's.
|
|
365
|
+
# Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
|
|
366
|
+
# commit trunk's tip can reach is scanned by the heavy lane instead (templates/
|
|
367
|
+
# ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
|
|
356
368
|
- name: gitleaks detect
|
|
357
|
-
|
|
369
|
+
env:
|
|
370
|
+
EVENT: ${{ github.event_name }}
|
|
371
|
+
REF_NAME: ${{ github.ref_name }}
|
|
372
|
+
BEFORE: ${{ github.event.before }}
|
|
373
|
+
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
|
374
|
+
run: |
|
|
375
|
+
trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
|
|
376
|
+
trunk="${trunk:-$DEFAULT_BRANCH}"
|
|
377
|
+
commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
|
|
378
|
+
if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
|
|
379
|
+
if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
|
|
380
|
+
range="$BEFORE..HEAD"
|
|
381
|
+
else
|
|
382
|
+
range=HEAD
|
|
383
|
+
fi
|
|
384
|
+
elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
|
|
385
|
+
range="refs/remotes/origin/$trunk..HEAD"
|
|
386
|
+
else
|
|
387
|
+
range=HEAD
|
|
388
|
+
fi
|
|
389
|
+
echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
|
|
390
|
+
gitleaks detect --source . --no-banner --log-opts="$range"
|
|
358
391
|
|
|
359
392
|
build:
|
|
360
393
|
name: Build
|
|
@@ -47,6 +47,8 @@ on:
|
|
|
47
47
|
permissions: {}
|
|
48
48
|
|
|
49
49
|
env:
|
|
50
|
+
# EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
|
|
51
|
+
GITLEAKS_VERSION: "8.30.1"
|
|
50
52
|
# EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
|
|
51
53
|
# default branch; the nightly tests THIS branch's tip.
|
|
52
54
|
TRUNK: main
|
|
@@ -260,13 +262,145 @@ jobs:
|
|
|
260
262
|
# EDIT: integration, browser/e2e or load jobs go here, or as jobs of their own with
|
|
261
263
|
# `needs: plan` and the same `if:` and checkout ref as this one.
|
|
262
264
|
|
|
265
|
+
# The full-history secret scan (#716). The light lane's scan reads only the commits its run
|
|
266
|
+
# brings; this one reads every commit trunk's tip can reach — a secret committed and removed
|
|
267
|
+
# before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
|
|
268
|
+
# no merge: a red nightly files the trunk-red issue below, and a red official build holds that
|
|
269
|
+
# release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
|
|
270
|
+
# branch's own run's to report.
|
|
271
|
+
secrets:
|
|
272
|
+
name: Secret scan, full history (gitleaks)
|
|
273
|
+
needs: plan
|
|
274
|
+
if: ${{ needs.plan.outputs.run == 'true' }}
|
|
275
|
+
runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
|
|
276
|
+
timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
|
|
277
|
+
permissions:
|
|
278
|
+
contents: read
|
|
279
|
+
steps:
|
|
280
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
281
|
+
with:
|
|
282
|
+
ref: ${{ needs.plan.outputs.sha }}
|
|
283
|
+
fetch-depth: 0 # every commit the tip reaches
|
|
284
|
+
persist-credentials: false
|
|
285
|
+
|
|
286
|
+
# The light lane's restore and install steps, verbatim: keep them in step when you edit either.
|
|
287
|
+
# Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
|
|
288
|
+
# binary comes back from here and the install step below downloads nothing.
|
|
289
|
+
# - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
|
|
290
|
+
# remote cache on top was measured as a loss, not a saving (#512).
|
|
291
|
+
# - Keyed on the pin and the platform: a version bump is a miss. The post step saves
|
|
292
|
+
# only when the job passes, so a failed download is never what gets cached.
|
|
293
|
+
# A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
|
|
294
|
+
# its name: handbook-sync recognises a copy of this job by it.
|
|
295
|
+
# EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
|
|
296
|
+
- name: Restore gitleaks cache
|
|
297
|
+
if: runner.environment == 'github-hosted'
|
|
298
|
+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
299
|
+
with:
|
|
300
|
+
path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
|
|
301
|
+
key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
|
|
302
|
+
|
|
303
|
+
- name: Install gitleaks (pinned release binary)
|
|
304
|
+
env:
|
|
305
|
+
# Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
|
|
306
|
+
# measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
|
|
307
|
+
# while the API asset path below answered 200 with the same file. The job's own token
|
|
308
|
+
# is enough — reading a public release needs an identity, not a permission.
|
|
309
|
+
GH_TOKEN: ${{ github.token }}
|
|
310
|
+
run: |
|
|
311
|
+
# Deliberately NOT gitleaks/gitleaks-action@v2: that action requires a paid
|
|
312
|
+
# GITLEAKS_LICENSE for repositories owned by an organization (free only for
|
|
313
|
+
# personal repos). The release binary itself is MIT and has no such gate.
|
|
314
|
+
# Repos also carry a local pre-commit gitleaks hook; this is the second layer.
|
|
315
|
+
# The asset is OS/arch-specific, so resolve both from uname instead of
|
|
316
|
+
# assuming the GitHub-hosted default: a hardcoded linux_x64 binary on a
|
|
317
|
+
# macOS self-hosted runner is an ELF the kernel cannot exec, and the step
|
|
318
|
+
# fails with exit 126 rather than anything mentioning the platform.
|
|
319
|
+
case "$(uname -s)" in
|
|
320
|
+
Darwin) os=darwin ;;
|
|
321
|
+
Linux) os=linux ;;
|
|
322
|
+
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
|
|
323
|
+
esac
|
|
324
|
+
case "$(uname -m)" in
|
|
325
|
+
arm64|aarch64) arch=arm64 ;;
|
|
326
|
+
x86_64|amd64) arch=x64 ;;
|
|
327
|
+
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
|
|
328
|
+
esac
|
|
329
|
+
# Kept between jobs (#696): a pinned version is downloaded once per runner, not on
|
|
330
|
+
# every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
|
|
331
|
+
# which outlasts any retry, so the binary has to be on the runner already.
|
|
332
|
+
# Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
|
|
333
|
+
# per-runner directory setup-node and setup-python install into. A persistent
|
|
334
|
+
# self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
|
|
335
|
+
# "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
|
|
336
|
+
# miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
|
|
337
|
+
# downloads on every run.
|
|
338
|
+
dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
|
|
339
|
+
if [ -x "$dir/gitleaks" ]; then
|
|
340
|
+
echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
|
|
341
|
+
else
|
|
342
|
+
mkdir -p "$dir"
|
|
343
|
+
tmp="$(mktemp -d "$dir.XXXXXX")"
|
|
344
|
+
trap 'rm -rf "$tmp"' EXIT
|
|
345
|
+
asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
|
|
346
|
+
sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
|
|
347
|
+
# fetch <accept> <url> <file> — every request of this step, one curl line.
|
|
348
|
+
# --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
|
|
349
|
+
# then fails with "not in gzip format", naming neither the download nor the host.
|
|
350
|
+
# --retry: a re-run during the same upstream error spends the one re-run branch CI
|
|
351
|
+
# allows, so a brief error is retried here first. A longer outage still fails, at
|
|
352
|
+
# curl, naming the status (#690).
|
|
353
|
+
# The token reaches api.github.com only: curl drops an -H Authorization header when
|
|
354
|
+
# the asset's 302 leads to another host, so the download CDN never sees it.
|
|
355
|
+
fetch() {
|
|
356
|
+
curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
|
|
357
|
+
-H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
|
|
358
|
+
}
|
|
359
|
+
# The API serves an asset by id, not by name, so look the ids up in the release. No jq:
|
|
360
|
+
# a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
|
|
361
|
+
# whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
|
|
362
|
+
# that order ever to change, the wrong file would arrive and the sha256 check below
|
|
363
|
+
# would refuse it — a loud failure, never a wrong binary.
|
|
364
|
+
fetch application/vnd.github+json \
|
|
365
|
+
"https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
|
|
366
|
+
for f in "$asset" "$sums"; do
|
|
367
|
+
url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
|
|
368
|
+
$2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
|
|
369
|
+
$2 == "name" && $4 == want && !found { print u; found = 1 }')"
|
|
370
|
+
if [ -z "$url" ]; then
|
|
371
|
+
echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
|
|
372
|
+
fi
|
|
373
|
+
fetch application/octet-stream "$url" "$tmp/$f"
|
|
374
|
+
done
|
|
375
|
+
# Verified before tar (#698): an error page, a truncated file or a wrong archive stops
|
|
376
|
+
# here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
|
|
377
|
+
want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
|
|
378
|
+
if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
|
|
379
|
+
else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
|
|
380
|
+
if [ -z "$want" ]; then
|
|
381
|
+
echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
|
|
382
|
+
fi
|
|
383
|
+
if [ "$got" != "$want" ]; then
|
|
384
|
+
echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
|
|
385
|
+
fi
|
|
386
|
+
tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
|
|
387
|
+
chmod +x "$tmp/gitleaks"
|
|
388
|
+
# One rename on one filesystem: a job sharing this tool cache sees no binary or a
|
|
389
|
+
# whole one, never half of one.
|
|
390
|
+
mv -f "$tmp/gitleaks" "$dir/gitleaks"
|
|
391
|
+
fi
|
|
392
|
+
echo "$dir" >> "$GITHUB_PATH"
|
|
393
|
+
|
|
394
|
+
- name: gitleaks detect (full history)
|
|
395
|
+
run: gitleaks detect --source . --no-banner --log-opts=HEAD
|
|
396
|
+
|
|
263
397
|
# A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
|
|
264
398
|
# with the same title suppresses a new one; an issue list that cannot be read files nothing and
|
|
265
399
|
# fails this job, so a silent miss is never mistaken for "already reported".
|
|
266
400
|
report:
|
|
267
401
|
name: Report a red nightly
|
|
268
|
-
needs: [plan, heavy]
|
|
269
|
-
if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
|
|
402
|
+
needs: [plan, heavy, secrets]
|
|
403
|
+
if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
|
|
270
404
|
runs-on: ubuntu-latest # EDIT: needs `gh`.
|
|
271
405
|
timeout-minutes: 5
|
|
272
406
|
permissions:
|
package/templates/ci-python.yml
CHANGED
|
@@ -252,8 +252,9 @@ jobs:
|
|
|
252
252
|
steps:
|
|
253
253
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
254
254
|
with:
|
|
255
|
-
# Full history:
|
|
256
|
-
#
|
|
255
|
+
# Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
|
|
256
|
+
# the two ends of that range have to be in the clone. The range itself is narrow — see
|
|
257
|
+
# "gitleaks detect" below (#716).
|
|
257
258
|
fetch-depth: 0
|
|
258
259
|
persist-credentials: false
|
|
259
260
|
|
|
@@ -364,8 +365,40 @@ jobs:
|
|
|
364
365
|
fi
|
|
365
366
|
echo "$dir" >> "$GITHUB_PATH"
|
|
366
367
|
|
|
368
|
+
# The scan range (#716): only the commits this run brings, never `git log --all`. A
|
|
369
|
+
# fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
|
|
370
|
+
# them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
|
|
371
|
+
# included, until that branch rewrote its history.
|
|
372
|
+
# - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
|
|
373
|
+
# - A trunk push: the pushed range (before..HEAD).
|
|
374
|
+
# - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
|
|
375
|
+
# own history — still never another branch's.
|
|
376
|
+
# Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
|
|
377
|
+
# commit trunk's tip can reach is scanned by the heavy lane instead (templates/
|
|
378
|
+
# ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
|
|
367
379
|
- name: gitleaks detect
|
|
368
|
-
|
|
380
|
+
env:
|
|
381
|
+
EVENT: ${{ github.event_name }}
|
|
382
|
+
REF_NAME: ${{ github.ref_name }}
|
|
383
|
+
BEFORE: ${{ github.event.before }}
|
|
384
|
+
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
|
385
|
+
run: |
|
|
386
|
+
trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
|
|
387
|
+
trunk="${trunk:-$DEFAULT_BRANCH}"
|
|
388
|
+
commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
|
|
389
|
+
if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
|
|
390
|
+
if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
|
|
391
|
+
range="$BEFORE..HEAD"
|
|
392
|
+
else
|
|
393
|
+
range=HEAD
|
|
394
|
+
fi
|
|
395
|
+
elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
|
|
396
|
+
range="refs/remotes/origin/$trunk..HEAD"
|
|
397
|
+
else
|
|
398
|
+
range=HEAD
|
|
399
|
+
fi
|
|
400
|
+
echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
|
|
401
|
+
gitleaks detect --source . --no-banner --log-opts="$range"
|
|
369
402
|
|
|
370
403
|
build:
|
|
371
404
|
name: Build (Python)
|
|
@@ -32,6 +32,9 @@
|
|
|
32
32
|
// DOWNLOADS_KEY_FILE the private key file the workflow wrote from DOWNLOADS_SSH_KEY
|
|
33
33
|
// DOWNLOADS_KNOWN_HOSTS_FILE the known-hosts file the workflow wrote from DOWNLOADS_SSH_KNOWN_HOSTS
|
|
34
34
|
// DOWNLOADS_RESERVED names the site keeps for itself, refused as an app slug (space/comma list)
|
|
35
|
+
// DOWNLOADS_READ_HEADERS optional (#714): a site that requires sign-in for every file. One `Name: value`
|
|
36
|
+
// HTTP header per line, sent on every read of the site (the requires check, the
|
|
37
|
+
// read-back); never printed. Unset, the site is read with no credential
|
|
35
38
|
//
|
|
36
39
|
// THE APPS IT REQUIRES (#653) come from manifest.json's optional `requires`: [{"slug": "<app>", "min": "X.Y.Z"}],
|
|
37
40
|
// other apps on the same site that its installers install first. Before anything is sent, every slug must have a final
|
|
@@ -570,6 +573,29 @@ function runRsync({ rsync, sshArgs, args, log }) {
|
|
|
570
573
|
if (r.status !== 0) throw new Error(`rsync failed (${r.status ?? r.signal}): ${(r.stderr || r.stdout || '').trim()}`)
|
|
571
574
|
}
|
|
572
575
|
|
|
576
|
+
/**
|
|
577
|
+
* DOWNLOADS_READ_HEADERS (#714) as a headers object: one `Name: value` per line, blank lines ignored. A line that is not
|
|
578
|
+
* a header throws naming its line number only — the value is a credential and is never echoed.
|
|
579
|
+
*/
|
|
580
|
+
export function readHeaders(text) {
|
|
581
|
+
const out = {}
|
|
582
|
+
const lines = String(text ?? '').split(/\r?\n/)
|
|
583
|
+
for (let i = 0; i < lines.length; i++) {
|
|
584
|
+
const line = lines[i].trim()
|
|
585
|
+
if (!line) continue
|
|
586
|
+
const m = /^([A-Za-z0-9!#$%&'*+.^_`|~-]+):[ \t]*(\S.*)$/.exec(line)
|
|
587
|
+
if (!m) throw new Error(`DOWNLOADS_READ_HEADERS: line ${i + 1} is not a "Name: value" header (the value is not shown); nothing was published`)
|
|
588
|
+
out[m[1].toLowerCase()] = m[2]
|
|
589
|
+
}
|
|
590
|
+
return out
|
|
591
|
+
}
|
|
592
|
+
|
|
593
|
+
/** `fetchImpl` with `headers` added to every request — how every read of the site carries DOWNLOADS_READ_HEADERS. */
|
|
594
|
+
export function withHeaders(fetchImpl, headers) {
|
|
595
|
+
if (!Object.keys(headers).length) return fetchImpl
|
|
596
|
+
return (url, init = {}) => fetchImpl(url, { ...init, headers: { ...headers, ...(init.headers ?? {}) } })
|
|
597
|
+
}
|
|
598
|
+
|
|
573
599
|
async function get(fetchImpl, url, attempts = 3) {
|
|
574
600
|
let last
|
|
575
601
|
for (let i = 0; i < attempts; i++) {
|
|
@@ -589,6 +615,8 @@ async function get(fetchImpl, url, attempts = 3) {
|
|
|
589
615
|
async function readBytes(fetchImpl, url) {
|
|
590
616
|
const res = await get(fetchImpl, url)
|
|
591
617
|
if (res.status === 404) return null
|
|
618
|
+
if (res.status === 401 || res.status === 403)
|
|
619
|
+
throw new Error(`${url} answered ${res.status} — the site requires sign-in: set or check the DOWNLOADS_READ_HEADERS secret (docs/downloads.md)`)
|
|
592
620
|
if (!res.ok) throw new Error(`${url} answered ${res.status}`)
|
|
593
621
|
return Buffer.from(await res.arrayBuffer())
|
|
594
622
|
}
|
|
@@ -826,12 +854,13 @@ async function main(argv, env) {
|
|
|
826
854
|
const dryRun = rest.includes('--dry-run')
|
|
827
855
|
const assetsDir = resolve(rest.filter((a) => !a.startsWith('--'))[0] ?? '')
|
|
828
856
|
const baseUrl = env.DOWNLOADS_URL
|
|
857
|
+
const fetchImpl = withHeaders(fetch, readHeaders(env.DOWNLOADS_READ_HEADERS))
|
|
829
858
|
const outDir = mkdtempSync(join(tmpdir(), 'publish-downloads-'))
|
|
830
859
|
const staged = await stage({ tag, assetsDir, outDir, baseUrl, root, config, manifest })
|
|
831
860
|
console.log(`Staged ${config.app} ${staged.version}: ${[...Object.keys(staged.files), ...staged.images].join(', ')}`)
|
|
832
861
|
await assertResolves(baseUrl)
|
|
833
862
|
const target = dryRun ? { dest: 'DRY-RUN:', sshArgs: undefined } : sshCommand(env)
|
|
834
|
-
const result = await publish({ outDir, app: config.app, ...target, baseUrl, dryRun })
|
|
863
|
+
const result = await publish({ outDir, app: config.app, ...target, baseUrl, dryRun, fetchImpl })
|
|
835
864
|
if (dryRun) console.log(`Dry run of ${result.version} done — nothing was uploaded`)
|
|
836
865
|
else
|
|
837
866
|
console.log(
|
|
@@ -58,6 +58,10 @@
|
|
|
58
58
|
# repo set up before the rename keeps publishing; rename the secret when convenient.
|
|
59
59
|
# variable DOWNLOADS_URL — the site's root address (https). The script appends /<app>; no host is ever
|
|
60
60
|
# written in the repository.
|
|
61
|
+
# secret DOWNLOADS_READ_HEADERS — optional (#714), only for a site that requires sign-in for every file: one
|
|
62
|
+
# `Name: value` HTTP header per line (e.g. `Authorization: Bearer …`, or an access proxy's two
|
|
63
|
+
# service-token headers), sent on every read of the site. The same secret release-auto.yml's downloads
|
|
64
|
+
# sweep reads latest.json with. Unset, the site is read with no credential.
|
|
61
65
|
#
|
|
62
66
|
# THIS FILE DOES NOT CHANGE YOUR REPO'S EXPOSURE (OR LEGACY TIER): it publishes files a final already made.
|
|
63
67
|
name: Publish downloads
|
|
@@ -174,6 +178,7 @@ jobs:
|
|
|
174
178
|
- name: Publish
|
|
175
179
|
env:
|
|
176
180
|
ASSETS: ${{ steps.assets.outputs.dir }}
|
|
181
|
+
DOWNLOADS_READ_HEADERS: ${{ secrets.DOWNLOADS_READ_HEADERS }}
|
|
177
182
|
DOWNLOADS_KEY_FILE: ${{ runner.temp }}/downloads-ssh/key
|
|
178
183
|
DOWNLOADS_KNOWN_HOSTS_FILE: ${{ runner.temp }}/downloads-ssh/known_hosts
|
|
179
184
|
run: |
|
|
@@ -145,7 +145,11 @@
|
|
|
145
145
|
# one (a complete version sends nothing), and when the site cannot be read from this job's runner —
|
|
146
146
|
# a warning then, never a red run; the next daily run retries. The site is read at
|
|
147
147
|
# vars.DOWNLOADS_URL/<downloads.app>/latest.json FROM THE RELEASE JOB'S RUNNER: a site only a private
|
|
148
|
-
# network reaches needs the release job on a runner inside it (RUNNERS, below).
|
|
148
|
+
# network reaches needs the release job on a runner inside it (RUNNERS, below). A site that requires
|
|
149
|
+
# sign-in (#714) is read with the optional DOWNLOADS_READ_HEADERS secret — one `Name: value` header
|
|
150
|
+
# per line, the same secret publish-downloads.yml reads with; a 401/403 is not the transient warning
|
|
151
|
+
# but an `::error title=Downloads not shipped::` plus a failing `downloads/<app>` commit status on
|
|
152
|
+
# the final, naming it (still exit 0, for the reason "A REFUSAL IS NOT A FAILURE" gives). The wiring:
|
|
149
153
|
#
|
|
150
154
|
# build-macos: # EDIT: one job per platform you build
|
|
151
155
|
# needs: release
|
|
@@ -362,6 +366,7 @@ jobs:
|
|
|
362
366
|
contents: write # push the tags (and, with promotion: main-loop, the promotion), publish the GitHub Release
|
|
363
367
|
issues: write # finalize opens / comments on / closes the "release: vX.Y.Z" tracking issue
|
|
364
368
|
actions: write # read the runs at a commit (ci-green, full-suite, trunk-green); dispatch CI after a promotion
|
|
369
|
+
statuses: write # the downloads sweep marks a final its site refused to read (401/403) failed (#714)
|
|
365
370
|
# What the npm job below needs (#433) — empty / publish=false unless this run tagged something (or
|
|
366
371
|
# the daily sweep found a final npm lacks, #612) AND .github/project.yml opts in with release.npm.
|
|
367
372
|
outputs:
|
|
@@ -823,7 +828,8 @@ jobs:
|
|
|
823
828
|
# when the site's latest.json does not name it yet (absent → never published, older → behind) — a
|
|
824
829
|
# final a person tagged (early, #627, or `final: human`) is never tagged by a run, so without this
|
|
825
830
|
# sweep it never reached the site. The site at that version or newer, no `downloads:` block, no
|
|
826
|
-
# DOWNLOADS_URL, or an unreadable site: nothing is handed on, and the step stays green
|
|
831
|
+
# DOWNLOADS_URL, or an unreadable site: nothing is handed on, and the step stays green — a 401/403
|
|
832
|
+
# (#714) too, but as an error annotation and a failing commit status on the final, not a warning.
|
|
827
833
|
- name: Read the downloads target [publish]
|
|
828
834
|
id: downloads-target
|
|
829
835
|
if: env.DRY_RUN != 'true' && (steps.cut.outputs.tag != '' || steps.finalize.outputs.tag != '' || (github.event_name == 'schedule' && env.RETRY_RUN != 'true') || github.event_name == 'workflow_dispatch')
|
|
@@ -831,6 +837,8 @@ jobs:
|
|
|
831
837
|
TAGS: ${{ steps.cut.outputs.tag }} ${{ steps.finalize.outputs.tag }}
|
|
832
838
|
SWEEP: ${{ (github.event_name == 'schedule' && env.RETRY_RUN != 'true') || github.event_name == 'workflow_dispatch' }}
|
|
833
839
|
DOWNLOADS_URL: ${{ vars.DOWNLOADS_URL }}
|
|
840
|
+
# #714: a site that requires sign-in for every file. Optional; unset, the read sends no credential.
|
|
841
|
+
DOWNLOADS_READ_HEADERS: ${{ secrets.DOWNLOADS_READ_HEADERS }}
|
|
834
842
|
run: |
|
|
835
843
|
set -uo pipefail
|
|
836
844
|
# The `downloads:` block's `app:` — the same key publish-downloads.mjs reads. No block, no target.
|
|
@@ -871,7 +879,20 @@ jobs:
|
|
|
871
879
|
{ echo "tag=$FINAL"; echo "swept=$FINAL"; } >> "$GITHUB_OUTPUT"
|
|
872
880
|
echo "downloads sweep: $1 — handing $FINAL to the build and downloads jobs (#669)" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
873
881
|
}
|
|
874
|
-
|
|
882
|
+
# #714: DOWNLOADS_READ_HEADERS — one `Name: value` header per line (docs/downloads.md, "A site
|
|
883
|
+
# that requires sign-in") — goes to curl from a file, never on its command line, and is never printed.
|
|
884
|
+
AUTH=()
|
|
885
|
+
if [ -n "${DOWNLOADS_READ_HEADERS:-}" ]; then
|
|
886
|
+
HDRS="$RUNNER_TEMP/downloads-read-headers"
|
|
887
|
+
trap 'rm -f "$HDRS"' EXIT
|
|
888
|
+
( umask 077; printf '%s\n' "$DOWNLOADS_READ_HEADERS" | tr -d '\r' | sed '/^[[:space:]]*$/d' > "$HDRS" )
|
|
889
|
+
if grep -qvE '^[A-Za-z0-9!#$%&*+.^_`|~-]+:[[:space:]]*[^[:space:]]' "$HDRS"; then
|
|
890
|
+
echo "::error title=Downloads not shipped::the DOWNLOADS_READ_HEADERS secret has a line that is not a 'Name: value' header (the value is not shown) — $FINAL is not handed on"
|
|
891
|
+
exit 0 # a standing refusal, not a red run (the 401/403 branch below says why)
|
|
892
|
+
fi
|
|
893
|
+
AUTH=(-H "@$HDRS")
|
|
894
|
+
fi
|
|
895
|
+
if BODY="$(retry_transient curl -fsSL ${AUTH[@]+"${AUTH[@]}"} "$URL" 2> "$ERR")"; then
|
|
875
896
|
SITE="$(jq -r '.version // empty' <<< "$BODY" 2> /dev/null || true)"
|
|
876
897
|
if [ -z "$SITE" ]; then
|
|
877
898
|
echo "::warning::downloads sweep: $URL names no version — $FINAL is not handed on in this run"
|
|
@@ -884,6 +905,29 @@ jobs:
|
|
|
884
905
|
fi
|
|
885
906
|
elif grep -q 'error: 404' "$ERR"; then
|
|
886
907
|
hand_on "$FINAL is final on main and the site has no latest.json for $APP"
|
|
908
|
+
elif CODE="$(grep -oE 'error: 40[13]' "$ERR" | head -n 1 | grep -oE '40[13]')"; then
|
|
909
|
+
# #714: 401/403 is a refusal no later run clears by itself, so it is not the warning below: a
|
|
910
|
+
# final whose files would never ship. Visible, but exit 0 — the #467 rule (header, "A REFUSAL
|
|
911
|
+
# IS NOT A FAILURE"): a red run would make main's head not-green and refuse the very ship
|
|
912
|
+
# that fixes it. So: an error annotation, a summary banner, and a failing commit status
|
|
913
|
+
# `downloads/<app>` on the final's own commit. The next daily run re-reads and re-posts.
|
|
914
|
+
if [ -n "${DOWNLOADS_READ_HEADERS:-}" ]; then
|
|
915
|
+
WHY="the site refused the DOWNLOADS_READ_HEADERS credential — check or replace that secret"
|
|
916
|
+
else
|
|
917
|
+
WHY="the site requires sign-in and no DOWNLOADS_READ_HEADERS secret is set on this repo"
|
|
918
|
+
fi
|
|
919
|
+
MSG="$URL answered $CODE, so $FINAL is final on main but its files never ship: $WHY (docs/downloads.md, \"A site that requires sign-in\")"
|
|
920
|
+
echo "::error title=Downloads not shipped::downloads sweep: $MSG"
|
|
921
|
+
{ echo "#### ⚠ Downloads not shipped — $FINAL"; echo; echo "$MSG"; } >> "$GITHUB_STEP_SUMMARY"
|
|
922
|
+
SHA="$(git rev-list -n 1 "$FINAL")"
|
|
923
|
+
# github.com only: a forge reads CI from commit statuses (#681), where a failing one would be the
|
|
924
|
+
# red main this branch exists to avoid. There the annotation stands alone.
|
|
925
|
+
if [ "${GITHUB_SERVER_URL:-https://github.com}" != "https://github.com" ]; then
|
|
926
|
+
:
|
|
927
|
+
elif ! retry_transient gh api --silent -X POST "repos/$GITHUB_REPOSITORY/statuses/$SHA" -f state=failure \
|
|
928
|
+
-f context="downloads/$APP" -f description="$FINAL not on the site: latest.json answered $CODE" > /dev/null; then
|
|
929
|
+
echo "::warning::downloads sweep: could not post the failing downloads/$APP status on $FINAL — the annotation above stands"
|
|
930
|
+
fi
|
|
887
931
|
else
|
|
888
932
|
echo "::warning::downloads sweep: could not read $URL from this runner ($(tail -n 1 "$ERR" 2> /dev/null)) — $FINAL is not handed on in this run; the next daily run retries"
|
|
889
933
|
fi
|
package/tools/lib/stamp.js
CHANGED
|
@@ -515,7 +515,7 @@ const WORKFLOW_FINGERPRINTS = [
|
|
|
515
515
|
{ marker: 'Run exactly this tag, verify (deploy-container-run --deploy-only)', kind: 'step', template: 'deploy-container' },
|
|
516
516
|
// branch-name (#348): its one step.
|
|
517
517
|
{ marker: 'Check branch name (CONVENTIONS §4)', kind: 'step', template: 'branch-name' },
|
|
518
|
-
// Shared by
|
|
518
|
+
// Shared by the ci-* templates, light and heavy (#716): proves derivation, cannot say from which. Note the exact
|
|
519
519
|
// parenthetical — hand-written installers on this fleet say "(pinned binary)", which is why the
|
|
520
520
|
// step NAME discriminates where the URL inside the step does not.
|
|
521
521
|
{ marker: 'Install gitleaks (pinned release binary)', kind: 'step', template: null },
|