@futurelastic/colab-handbook 1.20.4-rc.5 → 1.20.4-rc.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -33,5 +33,5 @@
33
33
  "engines": {
34
34
  "node": ">=18"
35
35
  },
36
- "version": "1.20.4-rc.5"
36
+ "version": "1.20.4-rc.6"
37
37
  }
@@ -27,8 +27,8 @@ or warn. The handbook enforces only what `colab ship` reads from your CI
27
27
 
28
28
  | Lane | Template → copy | Runs | What |
29
29
  |---|---|---|---|
30
- | **light** | `ci-<stack>.yml` → `ci.yml` | every push (branch and trunk); the lane `colab ship` reads | secret scan, build, typecheck/lint, the fast tests — aim for under 5 minutes at p50 |
31
- | **heavy** | `ci-<stack>-heavy.yml` → `ci-heavy.yml` | nightly on trunk's tip (skipped when trunk has not moved since the last green nightly), at every release candidate and tag, and by hand — **never per merge** | the whole suite including heavy-marked tests, integration, browser/e2e, load |
30
+ | **light** | `ci-<stack>.yml` → `ci.yml` | every push (branch and trunk); the lane `colab ship` reads | secret scan of the commits the run brings (#716), build, typecheck/lint, the fast tests — aim for under 5 minutes at p50 |
31
+ | **heavy** | `ci-<stack>-heavy.yml` → `ci-heavy.yml` | nightly on trunk's tip (skipped when trunk has not moved since the last green nightly), at every release candidate and tag, and by hand — **never per merge** | the whole suite including heavy-marked tests, integration, browser/e2e, load; the secret scan of trunk's whole history (job `secrets`, #716) |
32
32
 
33
33
  - **The adopter marks which tests are heavy; an unmarked test is light**, so nothing stops
34
34
  running per merge until someone decides it should.
@@ -104,7 +104,7 @@ check that reads one of these files is that tool's behaviour, documented with th
104
104
  |---|---|---|---|
105
105
  | `ci-node.yml` | `.github/workflows/ci.yml` | Pure Node repos: Vite SPA, node libs, Astro static, **Capacitor apps** | Resolves Node version from `project.yml` → `.nvmrc`/`engines` → **fails**. Never a default. Opt-in `Migration round-trip` job — `--roundtrip` writes it live, see *Migration round-trip* below. |
106
106
  | `ci-laravel.yml` | `.github/workflows/ci.yml` | Laravel + Inertia + Vite fullstack (with route/type codegen) | Same resolution for **both** PHP and Node. Includes the sqlite bootstrap + explicit wayfinder step, and a `migrations` job that round-trips the schema (migrate → rollback → seed → migrate, schema compared) on the test engine **and** the deploy engine (a MariaDB/MySQL service container at production's version). Its data-dependent checks need `ROUNDTRIP_SEEDER` — see *Migration round-trip* below. |
107
- | `ci-node-heavy.yml`, `ci-python-heavy.yml`, `ci-laravel-heavy.yml` | `.github/workflows/ci-heavy.yml` | The heavy lane beside the matching `ci-<stack>.yml` (above) | `schedule` + `workflow_dispatch` + `workflow_call`, never a push. Toolchain and install steps are copied from the light lane's `build` job, so the two resolve the same versions — keep them in step. `TRUNK` is an edit point. |
107
+ | `ci-node-heavy.yml`, `ci-python-heavy.yml`, `ci-laravel-heavy.yml` | `.github/workflows/ci-heavy.yml` | The heavy lane beside the matching `ci-<stack>.yml` (above) | `schedule` + `workflow_dispatch` + `workflow_call`, never a push. Toolchain and install steps are copied from the light lane's `build` job, so the two resolve the same versions — keep them in step; the `secrets` job's gitleaks install and `GITLEAKS_VERSION` likewise follow the light lane's `gitleaks` job. `TRUNK` is an edit point. |
108
108
  | `ci-python.yml` | `.github/workflows/ci.yml` | Python: FastAPI/Flask services, CLIs, libraries | Resolves Python from `project.yml` → `.python-version`/`requires-python` → **fails**. `requirements.txt` does not count. **Hybrid** Python+Node repo: copy this, then paste `ci-node.yml`'s `build:` job alongside — see the template header. Opt-in `Migration round-trip` job, as in `ci-node.yml`. |
109
109
  | `release-tag.yml` | `.github/workflows/release.yml` | Any repo cutting `v*.*.*` releases | Triggers on tag push. Publishes a grouped GitHub Release — a `-rc.N` tag as a pre-release, every summary since the previous *final* tag. No toolchain, no deploy; its header shows the `!v*.*.*-*` exclusion a deploy trigger needs. |
110
110
  | `release-auto.yml` | `.github/workflows/release-auto.yml` | A repo whose release tags land on `main` (`trunk: main`, `deploy: tag` or `none`) and whose §6 release route cuts candidates | Runs `colab release cut --auto` after CI goes green on main — at the sha that CI run tested (`--sha`, #622), even once main has moved on — and `colab release finalize --auto` daily — plus an hourly run that re-tries only a refused cut, once the fetched CLI has moved within `RETRY_WINDOW_MINUTES` or a run at main's head was cancelled (#547) — from a handbook ref (`HANDBOOK_REF` — the `stable` channel by default, `next` for the fast channel, or an exact tag to freeze; #445) — the handbook's own copy runs its checkout's CLI instead, since it ships the CLI it releases with (#428). **Publishes the Release in the same run** — a tag pushed with `GITHUB_TOKEN` triggers no other workflow, so `release-tag.yml` would never fire for it. A refusal is a warning, not a red run. Reads and creates tags, and fast-forwards the release channels `next` / `stable` (branches naming the newest candidate / final, #445); never pushes a commit. The audit recognises any workflow running `release cut/finalize --auto` and warns when the route leaves candidates off, when it tags without publishing while a tag-push workflow waits, or when it commits. |
@@ -47,6 +47,8 @@ on:
47
47
  permissions: {}
48
48
 
49
49
  env:
50
+ # EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
51
+ GITLEAKS_VERSION: "8.30.1"
50
52
  # EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
51
53
  # default branch; the nightly tests THIS branch's tip.
52
54
  TRUNK: main
@@ -328,13 +330,141 @@ jobs:
328
330
  # EDIT: integration, browser/e2e (Dusk) or load jobs go here, or as jobs of their own with
329
331
  # `needs: plan` and the same `if:` and checkout ref as this one.
330
332
 
333
+ # The full-history secret scan (#716). The light lane's scan reads only the commits its run
334
+ # brings; this one reads every commit trunk's tip can reach — a secret committed and removed
335
+ # before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
336
+ # no merge: a red nightly files the trunk-red issue below, and a red official build holds that
337
+ # release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
338
+ # branch's own run's to report.
339
+ secrets:
340
+ name: Secret scan, full history (gitleaks)
341
+ needs: plan
342
+ if: ${{ needs.plan.outputs.run == 'true' }}
343
+ runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
344
+ timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
345
+ permissions:
346
+ contents: read
347
+ steps:
348
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
349
+ with:
350
+ ref: ${{ needs.plan.outputs.sha }}
351
+ fetch-depth: 0 # every commit the tip reaches
352
+ persist-credentials: false
353
+
354
+ # The light lane's restore and install steps, verbatim: keep them in step when you edit either.
355
+ # Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
356
+ # binary comes back from here and the install step below downloads nothing.
357
+ # - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
358
+ # remote cache on top was measured as a loss, not a saving (#512).
359
+ # - Keyed on the pin and the platform: a version bump is a miss. The post step saves
360
+ # only when the job passes, so a failed download is never what gets cached.
361
+ # A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
362
+ # its name: handbook-sync recognises a copy of this job by it.
363
+ # EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
364
+ - name: Restore gitleaks cache
365
+ if: runner.environment == 'github-hosted'
366
+ uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
367
+ with:
368
+ path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
369
+ key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
370
+
371
+ - name: Install gitleaks (pinned release binary)
372
+ env:
373
+ # Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
374
+ # measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
375
+ # while the API asset path below answered 200 with the same file. The job's own token
376
+ # is enough — reading a public release needs an identity, not a permission.
377
+ GH_TOKEN: ${{ github.token }}
378
+ run: |
379
+ # NOT gitleaks/gitleaks-action@v2 — it demands a paid GITLEAKS_LICENSE for
380
+ # organization-owned repositories. The plain release binary is MIT.
381
+ # Resolve OS/arch from uname — a hardcoded linux_x64 binary on a macOS
382
+ # self-hosted runner cannot exec, and fails as a bare exit 126.
383
+ case "$(uname -s)" in
384
+ Darwin) os=darwin ;;
385
+ Linux) os=linux ;;
386
+ *) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
387
+ esac
388
+ case "$(uname -m)" in
389
+ arm64|aarch64) arch=arm64 ;;
390
+ x86_64|amd64) arch=x64 ;;
391
+ *) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
392
+ esac
393
+ # Kept between jobs (#696): a pinned version is downloaded once per runner, not on
394
+ # every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
395
+ # which outlasts any retry, so the binary has to be on the runner already.
396
+ # Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
397
+ # per-runner directory setup-node and setup-python install into. A persistent
398
+ # self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
399
+ # "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
400
+ # miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
401
+ # downloads on every run.
402
+ dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
403
+ if [ -x "$dir/gitleaks" ]; then
404
+ echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
405
+ else
406
+ mkdir -p "$dir"
407
+ tmp="$(mktemp -d "$dir.XXXXXX")"
408
+ trap 'rm -rf "$tmp"' EXIT
409
+ asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
410
+ sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
411
+ # fetch <accept> <url> <file> — every request of this step, one curl line.
412
+ # --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
413
+ # then fails with "not in gzip format", naming neither the download nor the host.
414
+ # --retry: a re-run during the same upstream error spends the one re-run branch CI
415
+ # allows, so a brief error is retried here first. A longer outage still fails, at
416
+ # curl, naming the status (#690).
417
+ # The token reaches api.github.com only: curl drops an -H Authorization header when
418
+ # the asset's 302 leads to another host, so the download CDN never sees it.
419
+ fetch() {
420
+ curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
421
+ -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
422
+ }
423
+ # The API serves an asset by id, not by name, so look the ids up in the release. No jq:
424
+ # a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
425
+ # whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
426
+ # that order ever to change, the wrong file would arrive and the sha256 check below
427
+ # would refuse it — a loud failure, never a wrong binary.
428
+ fetch application/vnd.github+json \
429
+ "https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
430
+ for f in "$asset" "$sums"; do
431
+ url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
432
+ $2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
433
+ $2 == "name" && $4 == want && !found { print u; found = 1 }')"
434
+ if [ -z "$url" ]; then
435
+ echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
436
+ fi
437
+ fetch application/octet-stream "$url" "$tmp/$f"
438
+ done
439
+ # Verified before tar (#698): an error page, a truncated file or a wrong archive stops
440
+ # here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
441
+ want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
442
+ if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
443
+ else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
444
+ if [ -z "$want" ]; then
445
+ echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
446
+ fi
447
+ if [ "$got" != "$want" ]; then
448
+ echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
449
+ fi
450
+ tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
451
+ chmod +x "$tmp/gitleaks"
452
+ # One rename on one filesystem: a job sharing this tool cache sees no binary or a
453
+ # whole one, never half of one.
454
+ mv -f "$tmp/gitleaks" "$dir/gitleaks"
455
+ fi
456
+ echo "$dir" >> "$GITHUB_PATH"
457
+
458
+ - name: gitleaks detect (full history)
459
+ run: gitleaks detect --source . --no-banner --log-opts=HEAD
460
+
331
461
  # A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
332
462
  # with the same title suppresses a new one; an issue list that cannot be read files nothing and
333
463
  # fails this job, so a silent miss is never mistaken for "already reported".
334
464
  report:
335
465
  name: Report a red nightly
336
- needs: [plan, heavy]
337
- if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
466
+ needs: [plan, heavy, secrets]
467
+ if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
338
468
  runs-on: ubuntu-latest # EDIT: needs `gh`.
339
469
  timeout-minutes: 5
340
470
  permissions:
@@ -256,7 +256,9 @@ jobs:
256
256
  steps:
257
257
  - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
258
258
  with:
259
- # Full history: scan the commit log, not only the current tree.
259
+ # Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
260
+ # the two ends of that range have to be in the clone. The range itself is narrow — see
261
+ # "gitleaks detect" below (#716).
260
262
  fetch-depth: 0
261
263
  persist-credentials: false
262
264
 
@@ -363,8 +365,40 @@ jobs:
363
365
  fi
364
366
  echo "$dir" >> "$GITHUB_PATH"
365
367
 
368
+ # The scan range (#716): only the commits this run brings, never `git log --all`. A
369
+ # fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
370
+ # them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
371
+ # included, until that branch rewrote its history.
372
+ # - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
373
+ # - A trunk push: the pushed range (before..HEAD).
374
+ # - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
375
+ # own history — still never another branch's.
376
+ # Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
377
+ # commit trunk's tip can reach is scanned by the heavy lane instead (templates/
378
+ # ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
366
379
  - name: gitleaks detect
367
- run: gitleaks detect --source . --no-banner
380
+ env:
381
+ EVENT: ${{ github.event_name }}
382
+ REF_NAME: ${{ github.ref_name }}
383
+ BEFORE: ${{ github.event.before }}
384
+ DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
385
+ run: |
386
+ trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
387
+ trunk="${trunk:-$DEFAULT_BRANCH}"
388
+ commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
389
+ if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
390
+ if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
391
+ range="$BEFORE..HEAD"
392
+ else
393
+ range=HEAD
394
+ fi
395
+ elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
396
+ range="refs/remotes/origin/$trunk..HEAD"
397
+ else
398
+ range=HEAD
399
+ fi
400
+ echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
401
+ gitleaks detect --source . --no-banner --log-opts="$range"
368
402
 
369
403
  build:
370
404
  name: Build (Laravel + Vite)
@@ -47,6 +47,8 @@ on:
47
47
  permissions: {}
48
48
 
49
49
  env:
50
+ # EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
51
+ GITLEAKS_VERSION: "8.30.1"
50
52
  # EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
51
53
  # default branch; the nightly tests THIS branch's tip.
52
54
  TRUNK: main
@@ -251,13 +253,145 @@ jobs:
251
253
  # EDIT: integration, browser/e2e or load jobs go here, or as jobs of their own with
252
254
  # `needs: plan` and the same `if:` and checkout ref as this one.
253
255
 
256
+ # The full-history secret scan (#716). The light lane's scan reads only the commits its run
257
+ # brings; this one reads every commit trunk's tip can reach — a secret committed and removed
258
+ # before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
259
+ # no merge: a red nightly files the trunk-red issue below, and a red official build holds that
260
+ # release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
261
+ # branch's own run's to report.
262
+ secrets:
263
+ name: Secret scan, full history (gitleaks)
264
+ needs: plan
265
+ if: ${{ needs.plan.outputs.run == 'true' }}
266
+ runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
267
+ timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
268
+ permissions:
269
+ contents: read
270
+ steps:
271
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
272
+ with:
273
+ ref: ${{ needs.plan.outputs.sha }}
274
+ fetch-depth: 0 # every commit the tip reaches
275
+ persist-credentials: false
276
+
277
+ # The light lane's restore and install steps, verbatim: keep them in step when you edit either.
278
+ # Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
279
+ # binary comes back from here and the install step below downloads nothing.
280
+ # - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
281
+ # remote cache on top was measured as a loss, not a saving (#512).
282
+ # - Keyed on the pin and the platform: a version bump is a miss. The post step saves
283
+ # only when the job passes, so a failed download is never what gets cached.
284
+ # A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
285
+ # its name: handbook-sync recognises a copy of this job by it.
286
+ # EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
287
+ - name: Restore gitleaks cache
288
+ if: runner.environment == 'github-hosted'
289
+ uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
290
+ with:
291
+ path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
292
+ key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
293
+
294
+ - name: Install gitleaks (pinned release binary)
295
+ env:
296
+ # Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
297
+ # measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
298
+ # while the API asset path below answered 200 with the same file. The job's own token
299
+ # is enough — reading a public release needs an identity, not a permission.
300
+ GH_TOKEN: ${{ github.token }}
301
+ run: |
302
+ # Deliberately NOT gitleaks/gitleaks-action@v2: that action requires a paid
303
+ # GITLEAKS_LICENSE for repositories owned by an organization (free only for
304
+ # personal repos). The release binary itself is MIT and has no such gate.
305
+ # Repos also carry a local pre-commit gitleaks hook; this is the second layer.
306
+ # The asset is OS/arch-specific, so resolve both from uname instead of
307
+ # assuming the GitHub-hosted default: a hardcoded linux_x64 binary on a
308
+ # macOS self-hosted runner is an ELF the kernel cannot exec, and the step
309
+ # fails with exit 126 rather than anything mentioning the platform.
310
+ case "$(uname -s)" in
311
+ Darwin) os=darwin ;;
312
+ Linux) os=linux ;;
313
+ *) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
314
+ esac
315
+ case "$(uname -m)" in
316
+ arm64|aarch64) arch=arm64 ;;
317
+ x86_64|amd64) arch=x64 ;;
318
+ *) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
319
+ esac
320
+ # Kept between jobs (#696): a pinned version is downloaded once per runner, not on
321
+ # every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
322
+ # which outlasts any retry, so the binary has to be on the runner already.
323
+ # Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
324
+ # per-runner directory setup-node and setup-python install into. A persistent
325
+ # self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
326
+ # "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
327
+ # miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
328
+ # downloads on every run.
329
+ dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
330
+ if [ -x "$dir/gitleaks" ]; then
331
+ echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
332
+ else
333
+ mkdir -p "$dir"
334
+ tmp="$(mktemp -d "$dir.XXXXXX")"
335
+ trap 'rm -rf "$tmp"' EXIT
336
+ asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
337
+ sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
338
+ # fetch <accept> <url> <file> — every request of this step, one curl line.
339
+ # --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
340
+ # then fails with "not in gzip format", naming neither the download nor the host.
341
+ # --retry: a re-run during the same upstream error spends the one re-run branch CI
342
+ # allows, so a brief error is retried here first. A longer outage still fails, at
343
+ # curl, naming the status (#690).
344
+ # The token reaches api.github.com only: curl drops an -H Authorization header when
345
+ # the asset's 302 leads to another host, so the download CDN never sees it.
346
+ fetch() {
347
+ curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
348
+ -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
349
+ }
350
+ # The API serves an asset by id, not by name, so look the ids up in the release. No jq:
351
+ # a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
352
+ # whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
353
+ # that order ever to change, the wrong file would arrive and the sha256 check below
354
+ # would refuse it — a loud failure, never a wrong binary.
355
+ fetch application/vnd.github+json \
356
+ "https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
357
+ for f in "$asset" "$sums"; do
358
+ url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
359
+ $2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
360
+ $2 == "name" && $4 == want && !found { print u; found = 1 }')"
361
+ if [ -z "$url" ]; then
362
+ echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
363
+ fi
364
+ fetch application/octet-stream "$url" "$tmp/$f"
365
+ done
366
+ # Verified before tar (#698): an error page, a truncated file or a wrong archive stops
367
+ # here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
368
+ want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
369
+ if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
370
+ else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
371
+ if [ -z "$want" ]; then
372
+ echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
373
+ fi
374
+ if [ "$got" != "$want" ]; then
375
+ echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
376
+ fi
377
+ tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
378
+ chmod +x "$tmp/gitleaks"
379
+ # One rename on one filesystem: a job sharing this tool cache sees no binary or a
380
+ # whole one, never half of one.
381
+ mv -f "$tmp/gitleaks" "$dir/gitleaks"
382
+ fi
383
+ echo "$dir" >> "$GITHUB_PATH"
384
+
385
+ - name: gitleaks detect (full history)
386
+ run: gitleaks detect --source . --no-banner --log-opts=HEAD
387
+
254
388
  # A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
255
389
  # with the same title suppresses a new one; an issue list that cannot be read files nothing and
256
390
  # fails this job, so a silent miss is never mistaken for "already reported".
257
391
  report:
258
392
  name: Report a red nightly
259
- needs: [plan, heavy]
260
- if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
393
+ needs: [plan, heavy, secrets]
394
+ if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
261
395
  runs-on: ubuntu-latest # EDIT: needs `gh`.
262
396
  timeout-minutes: 5
263
397
  permissions:
@@ -241,8 +241,9 @@ jobs:
241
241
  steps:
242
242
  - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
243
243
  with:
244
- # Full history: gitleaks scans commits, not just the working tree, so a
245
- # secret that was committed and later removed is still caught.
244
+ # Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
245
+ # the two ends of that range have to be in the clone. The range itself is narrow — see
246
+ # "gitleaks detect" below (#716).
246
247
  fetch-depth: 0
247
248
  persist-credentials: false
248
249
 
@@ -353,8 +354,40 @@ jobs:
353
354
  fi
354
355
  echo "$dir" >> "$GITHUB_PATH"
355
356
 
357
+ # The scan range (#716): only the commits this run brings, never `git log --all`. A
358
+ # fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
359
+ # them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
360
+ # included, until that branch rewrote its history.
361
+ # - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
362
+ # - A trunk push: the pushed range (before..HEAD).
363
+ # - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
364
+ # own history — still never another branch's.
365
+ # Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
366
+ # commit trunk's tip can reach is scanned by the heavy lane instead (templates/
367
+ # ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
356
368
  - name: gitleaks detect
357
- run: gitleaks detect --source . --no-banner
369
+ env:
370
+ EVENT: ${{ github.event_name }}
371
+ REF_NAME: ${{ github.ref_name }}
372
+ BEFORE: ${{ github.event.before }}
373
+ DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
374
+ run: |
375
+ trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
376
+ trunk="${trunk:-$DEFAULT_BRANCH}"
377
+ commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
378
+ if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
379
+ if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
380
+ range="$BEFORE..HEAD"
381
+ else
382
+ range=HEAD
383
+ fi
384
+ elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
385
+ range="refs/remotes/origin/$trunk..HEAD"
386
+ else
387
+ range=HEAD
388
+ fi
389
+ echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
390
+ gitleaks detect --source . --no-banner --log-opts="$range"
358
391
 
359
392
  build:
360
393
  name: Build
@@ -47,6 +47,8 @@ on:
47
47
  permissions: {}
48
48
 
49
49
  env:
50
+ # EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
51
+ GITLEAKS_VERSION: "8.30.1"
50
52
  # EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
51
53
  # default branch; the nightly tests THIS branch's tip.
52
54
  TRUNK: main
@@ -260,13 +262,145 @@ jobs:
260
262
  # EDIT: integration, browser/e2e or load jobs go here, or as jobs of their own with
261
263
  # `needs: plan` and the same `if:` and checkout ref as this one.
262
264
 
265
+ # The full-history secret scan (#716). The light lane's scan reads only the commits its run
266
+ # brings; this one reads every commit trunk's tip can reach — a secret committed and removed
267
+ # before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
268
+ # no merge: a red nightly files the trunk-red issue below, and a red official build holds that
269
+ # release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
270
+ # branch's own run's to report.
271
+ secrets:
272
+ name: Secret scan, full history (gitleaks)
273
+ needs: plan
274
+ if: ${{ needs.plan.outputs.run == 'true' }}
275
+ runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
276
+ timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
277
+ permissions:
278
+ contents: read
279
+ steps:
280
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
281
+ with:
282
+ ref: ${{ needs.plan.outputs.sha }}
283
+ fetch-depth: 0 # every commit the tip reaches
284
+ persist-credentials: false
285
+
286
+ # The light lane's restore and install steps, verbatim: keep them in step when you edit either.
287
+ # Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
288
+ # binary comes back from here and the install step below downloads nothing.
289
+ # - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
290
+ # remote cache on top was measured as a loss, not a saving (#512).
291
+ # - Keyed on the pin and the platform: a version bump is a miss. The post step saves
292
+ # only when the job passes, so a failed download is never what gets cached.
293
+ # A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
294
+ # its name: handbook-sync recognises a copy of this job by it.
295
+ # EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
296
+ - name: Restore gitleaks cache
297
+ if: runner.environment == 'github-hosted'
298
+ uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
299
+ with:
300
+ path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
301
+ key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
302
+
303
+ - name: Install gitleaks (pinned release binary)
304
+ env:
305
+ # Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
306
+ # measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
307
+ # while the API asset path below answered 200 with the same file. The job's own token
308
+ # is enough — reading a public release needs an identity, not a permission.
309
+ GH_TOKEN: ${{ github.token }}
310
+ run: |
311
+ # Deliberately NOT gitleaks/gitleaks-action@v2: that action requires a paid
312
+ # GITLEAKS_LICENSE for repositories owned by an organization (free only for
313
+ # personal repos). The release binary itself is MIT and has no such gate.
314
+ # Repos also carry a local pre-commit gitleaks hook; this is the second layer.
315
+ # The asset is OS/arch-specific, so resolve both from uname instead of
316
+ # assuming the GitHub-hosted default: a hardcoded linux_x64 binary on a
317
+ # macOS self-hosted runner is an ELF the kernel cannot exec, and the step
318
+ # fails with exit 126 rather than anything mentioning the platform.
319
+ case "$(uname -s)" in
320
+ Darwin) os=darwin ;;
321
+ Linux) os=linux ;;
322
+ *) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
323
+ esac
324
+ case "$(uname -m)" in
325
+ arm64|aarch64) arch=arm64 ;;
326
+ x86_64|amd64) arch=x64 ;;
327
+ *) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
328
+ esac
329
+ # Kept between jobs (#696): a pinned version is downloaded once per runner, not on
330
+ # every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
331
+ # which outlasts any retry, so the binary has to be on the runner already.
332
+ # Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
333
+ # per-runner directory setup-node and setup-python install into. A persistent
334
+ # self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
335
+ # "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
336
+ # miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
337
+ # downloads on every run.
338
+ dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
339
+ if [ -x "$dir/gitleaks" ]; then
340
+ echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
341
+ else
342
+ mkdir -p "$dir"
343
+ tmp="$(mktemp -d "$dir.XXXXXX")"
344
+ trap 'rm -rf "$tmp"' EXIT
345
+ asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
346
+ sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
347
+ # fetch <accept> <url> <file> — every request of this step, one curl line.
348
+ # --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
349
+ # then fails with "not in gzip format", naming neither the download nor the host.
350
+ # --retry: a re-run during the same upstream error spends the one re-run branch CI
351
+ # allows, so a brief error is retried here first. A longer outage still fails, at
352
+ # curl, naming the status (#690).
353
+ # The token reaches api.github.com only: curl drops an -H Authorization header when
354
+ # the asset's 302 leads to another host, so the download CDN never sees it.
355
+ fetch() {
356
+ curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
357
+ -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
358
+ }
359
+ # The API serves an asset by id, not by name, so look the ids up in the release. No jq:
360
+ # a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
361
+ # whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
362
+ # that order ever to change, the wrong file would arrive and the sha256 check below
363
+ # would refuse it — a loud failure, never a wrong binary.
364
+ fetch application/vnd.github+json \
365
+ "https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
366
+ for f in "$asset" "$sums"; do
367
+ url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
368
+ $2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
369
+ $2 == "name" && $4 == want && !found { print u; found = 1 }')"
370
+ if [ -z "$url" ]; then
371
+ echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
372
+ fi
373
+ fetch application/octet-stream "$url" "$tmp/$f"
374
+ done
375
+ # Verified before tar (#698): an error page, a truncated file or a wrong archive stops
376
+ # here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
377
+ want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
378
+ if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
379
+ else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
380
+ if [ -z "$want" ]; then
381
+ echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
382
+ fi
383
+ if [ "$got" != "$want" ]; then
384
+ echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
385
+ fi
386
+ tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
387
+ chmod +x "$tmp/gitleaks"
388
+ # One rename on one filesystem: a job sharing this tool cache sees no binary or a
389
+ # whole one, never half of one.
390
+ mv -f "$tmp/gitleaks" "$dir/gitleaks"
391
+ fi
392
+ echo "$dir" >> "$GITHUB_PATH"
393
+
394
+ - name: gitleaks detect (full history)
395
+ run: gitleaks detect --source . --no-banner --log-opts=HEAD
396
+
263
397
  # A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
264
398
  # with the same title suppresses a new one; an issue list that cannot be read files nothing and
265
399
  # fails this job, so a silent miss is never mistaken for "already reported".
266
400
  report:
267
401
  name: Report a red nightly
268
- needs: [plan, heavy]
269
- if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
402
+ needs: [plan, heavy, secrets]
403
+ if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
270
404
  runs-on: ubuntu-latest # EDIT: needs `gh`.
271
405
  timeout-minutes: 5
272
406
  permissions:
@@ -252,8 +252,9 @@ jobs:
252
252
  steps:
253
253
  - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
254
254
  with:
255
- # Full history: gitleaks scans commits, not just the working tree, so a
256
- # secret that was committed and later removed is still caught.
255
+ # Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
256
+ # the two ends of that range have to be in the clone. The range itself is narrow — see
257
+ # "gitleaks detect" below (#716).
257
258
  fetch-depth: 0
258
259
  persist-credentials: false
259
260
 
@@ -364,8 +365,40 @@ jobs:
364
365
  fi
365
366
  echo "$dir" >> "$GITHUB_PATH"
366
367
 
368
+ # The scan range (#716): only the commits this run brings, never `git log --all`. A
369
+ # fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
370
+ # them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
371
+ # included, until that branch rewrote its history.
372
+ # - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
373
+ # - A trunk push: the pushed range (before..HEAD).
374
+ # - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
375
+ # own history — still never another branch's.
376
+ # Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
377
+ # commit trunk's tip can reach is scanned by the heavy lane instead (templates/
378
+ # ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
367
379
  - name: gitleaks detect
368
- run: gitleaks detect --source . --no-banner
380
+ env:
381
+ EVENT: ${{ github.event_name }}
382
+ REF_NAME: ${{ github.ref_name }}
383
+ BEFORE: ${{ github.event.before }}
384
+ DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
385
+ run: |
386
+ trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
387
+ trunk="${trunk:-$DEFAULT_BRANCH}"
388
+ commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
389
+ if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
390
+ if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
391
+ range="$BEFORE..HEAD"
392
+ else
393
+ range=HEAD
394
+ fi
395
+ elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
396
+ range="refs/remotes/origin/$trunk..HEAD"
397
+ else
398
+ range=HEAD
399
+ fi
400
+ echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
401
+ gitleaks detect --source . --no-banner --log-opts="$range"
369
402
 
370
403
  build:
371
404
  name: Build (Python)
@@ -515,7 +515,7 @@ const WORKFLOW_FINGERPRINTS = [
515
515
  { marker: 'Run exactly this tag, verify (deploy-container-run --deploy-only)', kind: 'step', template: 'deploy-container' },
516
516
  // branch-name (#348): its one step.
517
517
  { marker: 'Check branch name (CONVENTIONS §4)', kind: 'step', template: 'branch-name' },
518
- // Shared by all three ci-* templates: proves derivation, cannot say from which. Note the exact
518
+ // Shared by the ci-* templates, light and heavy (#716): proves derivation, cannot say from which. Note the exact
519
519
  // parenthetical — hand-written installers on this fleet say "(pinned binary)", which is why the
520
520
  // step NAME discriminates where the URL inside the step does not.
521
521
  { marker: 'Install gitleaks (pinned release binary)', kind: 'step', template: null },