@futurelastic/colab-handbook 1.20.4-rc.5 → 1.20.4-rc.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/templates/README.md +11 -4
- package/templates/ci-laravel-heavy.yml +150 -3
- package/templates/ci-laravel.yml +61 -12
- package/templates/ci-node-heavy.yml +154 -3
- package/templates/ci-node.yml +61 -13
- package/templates/ci-python-heavy.yml +154 -3
- package/templates/ci-python.yml +61 -13
- package/tools/lib/stamp.js +1 -1
package/package.json
CHANGED
package/templates/README.md
CHANGED
|
@@ -27,8 +27,8 @@ or warn. The handbook enforces only what `colab ship` reads from your CI
|
|
|
27
27
|
|
|
28
28
|
| Lane | Template → copy | Runs | What |
|
|
29
29
|
|---|---|---|---|
|
|
30
|
-
| **light** | `ci-<stack>.yml` → `ci.yml` | every push (branch and trunk); the lane `colab ship` reads | secret scan, build, typecheck/lint, the fast tests — aim for under 5 minutes at p50 |
|
|
31
|
-
| **heavy** | `ci-<stack>-heavy.yml` → `ci-heavy.yml` | nightly on trunk's tip (skipped when trunk has not moved since the last green nightly), at every release candidate and tag, and by hand — **never per merge** | the whole suite including heavy-marked tests, integration, browser/e2e, load |
|
|
30
|
+
| **light** | `ci-<stack>.yml` → `ci.yml` | every push (branch and trunk); the lane `colab ship` reads | secret scan of the commits the run brings (#716), build, typecheck/lint, the fast tests — aim for under 5 minutes at p50 |
|
|
31
|
+
| **heavy** | `ci-<stack>-heavy.yml` → `ci-heavy.yml` | nightly on trunk's tip (skipped when trunk has not moved since the last green nightly), at every release candidate and tag, and by hand — **never per merge** | the whole suite including heavy-marked tests, integration, browser/e2e, load; the secret scan of trunk's whole history (job `secrets`, #716) |
|
|
32
32
|
|
|
33
33
|
- **The adopter marks which tests are heavy; an unmarked test is light**, so nothing stops
|
|
34
34
|
running per merge until someone decides it should.
|
|
@@ -104,7 +104,7 @@ check that reads one of these files is that tool's behaviour, documented with th
|
|
|
104
104
|
|---|---|---|---|
|
|
105
105
|
| `ci-node.yml` | `.github/workflows/ci.yml` | Pure Node repos: Vite SPA, node libs, Astro static, **Capacitor apps** | Resolves Node version from `project.yml` → `.nvmrc`/`engines` → **fails**. Never a default. Opt-in `Migration round-trip` job — `--roundtrip` writes it live, see *Migration round-trip* below. |
|
|
106
106
|
| `ci-laravel.yml` | `.github/workflows/ci.yml` | Laravel + Inertia + Vite fullstack (with route/type codegen) | Same resolution for **both** PHP and Node. Includes the sqlite bootstrap + explicit wayfinder step, and a `migrations` job that round-trips the schema (migrate → rollback → seed → migrate, schema compared) on the test engine **and** the deploy engine (a MariaDB/MySQL service container at production's version). Its data-dependent checks need `ROUNDTRIP_SEEDER` — see *Migration round-trip* below. |
|
|
107
|
-
| `ci-node-heavy.yml`, `ci-python-heavy.yml`, `ci-laravel-heavy.yml` | `.github/workflows/ci-heavy.yml` | The heavy lane beside the matching `ci-<stack>.yml` (above) | `schedule` + `workflow_dispatch` + `workflow_call`, never a push. Toolchain and install steps are copied from the light lane's `build` job, so the two resolve the same versions — keep them in step. `TRUNK` is an edit point. |
|
|
107
|
+
| `ci-node-heavy.yml`, `ci-python-heavy.yml`, `ci-laravel-heavy.yml` | `.github/workflows/ci-heavy.yml` | The heavy lane beside the matching `ci-<stack>.yml` (above) | `schedule` + `workflow_dispatch` + `workflow_call`, never a push. Toolchain and install steps are copied from the light lane's `build` job, so the two resolve the same versions — keep them in step; the `secrets` job's gitleaks install and `GITLEAKS_VERSION` likewise follow the light lane's `gitleaks` job. `TRUNK` is an edit point. |
|
|
108
108
|
| `ci-python.yml` | `.github/workflows/ci.yml` | Python: FastAPI/Flask services, CLIs, libraries | Resolves Python from `project.yml` → `.python-version`/`requires-python` → **fails**. `requirements.txt` does not count. **Hybrid** Python+Node repo: copy this, then paste `ci-node.yml`'s `build:` job alongside — see the template header. Opt-in `Migration round-trip` job, as in `ci-node.yml`. |
|
|
109
109
|
| `release-tag.yml` | `.github/workflows/release.yml` | Any repo cutting `v*.*.*` releases | Triggers on tag push. Publishes a grouped GitHub Release — a `-rc.N` tag as a pre-release, every summary since the previous *final* tag. No toolchain, no deploy; its header shows the `!v*.*.*-*` exclusion a deploy trigger needs. |
|
|
110
110
|
| `release-auto.yml` | `.github/workflows/release-auto.yml` | A repo whose release tags land on `main` (`trunk: main`, `deploy: tag` or `none`) and whose §6 release route cuts candidates | Runs `colab release cut --auto` after CI goes green on main — at the sha that CI run tested (`--sha`, #622), even once main has moved on — and `colab release finalize --auto` daily — plus an hourly run that re-tries only a refused cut, once the fetched CLI has moved within `RETRY_WINDOW_MINUTES` or a run at main's head was cancelled (#547) — from a handbook ref (`HANDBOOK_REF` — the `stable` channel by default, `next` for the fast channel, or an exact tag to freeze; #445) — the handbook's own copy runs its checkout's CLI instead, since it ships the CLI it releases with (#428). **Publishes the Release in the same run** — a tag pushed with `GITHUB_TOKEN` triggers no other workflow, so `release-tag.yml` would never fire for it. A refusal is a warning, not a red run. Reads and creates tags, and fast-forwards the release channels `next` / `stable` (branches naming the newest candidate / final, #445); never pushes a commit. The audit recognises any workflow running `release cut/finalize --auto` and warns when the route leaves candidates off, when it tags without publishing while a tag-push workflow waits, or when it commits. |
|
|
@@ -164,7 +164,14 @@ notice back and print the run they relied on. Three things to keep in a copy:
|
|
|
164
164
|
name, edit all three. Until you do, the guard simply never fires on trunk — nothing breaks;
|
|
165
165
|
but a trunk left out of `group:` and `cancel-in-progress:` has its runs cancelled.
|
|
166
166
|
- **Never put a trunk-only job behind it.** Publish, deploy and release jobs must run whatever
|
|
167
|
-
the guard says. Give them their own `if:` that does not read `needs.dedupe
|
|
167
|
+
the guard says. Give them their own `if:` that does not read `needs.dedupe` — and **name them
|
|
168
|
+
in the guard's `TRUNK_ONLY_JOBS`** (#717), or move them to a workflow of their own. A branch run
|
|
169
|
+
skips such a job, and an unlisted skip makes that run unusable on trunk (#511), so a workflow
|
|
170
|
+
that mixes trunk-only jobs with its suite otherwise never reuses anything.
|
|
171
|
+
- **A batch-landing ref gets the same tree mode** (#717): the first push of a `ship-batch/**`
|
|
172
|
+
ref skips the suite when a member branch already passed its exact tree. Nothing to edit.
|
|
173
|
+
|
|
174
|
+
The rule all of this keeps: [one verdict per tree](../CONVENTIONS.md#one-verdict-per-tree--quick-on-branches-full-once-719).
|
|
168
175
|
|
|
169
176
|
Opt out with `tree-reuse: off` (`project.schema.md`) when trunk's run proves something a branch
|
|
170
177
|
run cannot: secrets only trunk gets, `github.ref`-conditional steps, unpinned runner images.
|
|
@@ -7,7 +7,8 @@
|
|
|
7
7
|
# HEAVY LANE (#607): the whole suite, including tests marked heavy, plus anything too slow for
|
|
8
8
|
# every merge (integration, browser/e2e, load). The light lane (ci.yml) runs on every push and
|
|
9
9
|
# is the only lane `colab ship` reads. This one runs:
|
|
10
|
-
# - NIGHTLY, on trunk's tip. Skipped when trunk has not moved since the last green nightly
|
|
10
|
+
# - NIGHTLY, on trunk's tip. Skipped when trunk has not moved since the last green nightly, or
|
|
11
|
+
# when a green dispatched run of this workflow already tested that exact sha (#719).
|
|
11
12
|
# A red nightly opens one `trunk-red` issue naming the commit range that could have caused
|
|
12
13
|
# it. It never blocks a merge.
|
|
13
14
|
# - On every OFFICIAL BUILD: release-tag.yml calls this file and publishes only when it is
|
|
@@ -22,6 +23,8 @@
|
|
|
22
23
|
# the Pest/PHPUnit group `heavy` (`->group('heavy')`, `#[Group('heavy')]`). The light
|
|
23
24
|
# lane excludes it; this lane runs the whole suite.
|
|
24
25
|
#
|
|
26
|
+
# One verdict per sha: never test a commit this workflow already passed (CONVENTIONS.md §7,
|
|
27
|
+
# "One verdict per tree").
|
|
25
28
|
# Deliberately NOT triggered by push or pull_request: a run of this workflow must never count as
|
|
26
29
|
# a merge verdict (tools/lib/verify-runs.js sets schedule and dispatch runs aside).
|
|
27
30
|
# No concurrency block: a heavy run is never cancelled — a release may be waiting on it.
|
|
@@ -47,6 +50,8 @@ on:
|
|
|
47
50
|
permissions: {}
|
|
48
51
|
|
|
49
52
|
env:
|
|
53
|
+
# EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
|
|
54
|
+
GITLEAKS_VERSION: "8.30.1"
|
|
50
55
|
# EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
|
|
51
56
|
# default branch; the nightly tests THIS branch's tip.
|
|
52
57
|
TRUNK: main
|
|
@@ -99,6 +104,20 @@ jobs:
|
|
|
99
104
|
if [ "$since" = "$sha" ]; then
|
|
100
105
|
run=false
|
|
101
106
|
echo "::notice title=heavy-skipped::$TRUNK has not moved since the last green nightly ($sha)"
|
|
107
|
+
else
|
|
108
|
+
# One verdict per sha (#719): a green DISPATCHED run of this workflow already tested this
|
|
109
|
+
# exact commit (a gated landing, a release candidate) — the nightly does not test it again.
|
|
110
|
+
# `since` stays the last green NIGHTLY: the trunk-red range is read from nightlies only.
|
|
111
|
+
for id in $(gh api "repos/$REPO/actions/workflows/$wf/runs?event=workflow_dispatch&status=success&per_page=10" \
|
|
112
|
+
--jq '.workflow_runs[].id' 2>/dev/null); do
|
|
113
|
+
done_at="$(gh api "repos/$REPO/actions/runs/$id/jobs" \
|
|
114
|
+
--jq '[.jobs[] | select(.conclusion == "success" and (.name | startswith("Full suite @ "))) | .name | ltrimstr("Full suite @ ")][0] // ""' 2>/dev/null || true)"
|
|
115
|
+
if [ "$done_at" = "$sha" ]; then
|
|
116
|
+
run=false
|
|
117
|
+
echo "::notice title=heavy-skipped::$sha already passed this workflow in dispatched run $id"
|
|
118
|
+
break
|
|
119
|
+
fi
|
|
120
|
+
done
|
|
102
121
|
fi
|
|
103
122
|
fi
|
|
104
123
|
echo "run=$run" >> "$GITHUB_OUTPUT"
|
|
@@ -328,13 +347,141 @@ jobs:
|
|
|
328
347
|
# EDIT: integration, browser/e2e (Dusk) or load jobs go here, or as jobs of their own with
|
|
329
348
|
# `needs: plan` and the same `if:` and checkout ref as this one.
|
|
330
349
|
|
|
350
|
+
# The full-history secret scan (#716). The light lane's scan reads only the commits its run
|
|
351
|
+
# brings; this one reads every commit trunk's tip can reach — a secret committed and removed
|
|
352
|
+
# before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
|
|
353
|
+
# no merge: a red nightly files the trunk-red issue below, and a red official build holds that
|
|
354
|
+
# release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
|
|
355
|
+
# branch's own run's to report.
|
|
356
|
+
secrets:
|
|
357
|
+
name: Secret scan, full history (gitleaks)
|
|
358
|
+
needs: plan
|
|
359
|
+
if: ${{ needs.plan.outputs.run == 'true' }}
|
|
360
|
+
runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
|
|
361
|
+
timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
|
|
362
|
+
permissions:
|
|
363
|
+
contents: read
|
|
364
|
+
steps:
|
|
365
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
366
|
+
with:
|
|
367
|
+
ref: ${{ needs.plan.outputs.sha }}
|
|
368
|
+
fetch-depth: 0 # every commit the tip reaches
|
|
369
|
+
persist-credentials: false
|
|
370
|
+
|
|
371
|
+
# The light lane's restore and install steps, verbatim: keep them in step when you edit either.
|
|
372
|
+
# Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
|
|
373
|
+
# binary comes back from here and the install step below downloads nothing.
|
|
374
|
+
# - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
|
|
375
|
+
# remote cache on top was measured as a loss, not a saving (#512).
|
|
376
|
+
# - Keyed on the pin and the platform: a version bump is a miss. The post step saves
|
|
377
|
+
# only when the job passes, so a failed download is never what gets cached.
|
|
378
|
+
# A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
|
|
379
|
+
# its name: handbook-sync recognises a copy of this job by it.
|
|
380
|
+
# EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
|
|
381
|
+
- name: Restore gitleaks cache
|
|
382
|
+
if: runner.environment == 'github-hosted'
|
|
383
|
+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
384
|
+
with:
|
|
385
|
+
path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
|
|
386
|
+
key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
|
|
387
|
+
|
|
388
|
+
- name: Install gitleaks (pinned release binary)
|
|
389
|
+
env:
|
|
390
|
+
# Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
|
|
391
|
+
# measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
|
|
392
|
+
# while the API asset path below answered 200 with the same file. The job's own token
|
|
393
|
+
# is enough — reading a public release needs an identity, not a permission.
|
|
394
|
+
GH_TOKEN: ${{ github.token }}
|
|
395
|
+
run: |
|
|
396
|
+
# NOT gitleaks/gitleaks-action@v2 — it demands a paid GITLEAKS_LICENSE for
|
|
397
|
+
# organization-owned repositories. The plain release binary is MIT.
|
|
398
|
+
# Resolve OS/arch from uname — a hardcoded linux_x64 binary on a macOS
|
|
399
|
+
# self-hosted runner cannot exec, and fails as a bare exit 126.
|
|
400
|
+
case "$(uname -s)" in
|
|
401
|
+
Darwin) os=darwin ;;
|
|
402
|
+
Linux) os=linux ;;
|
|
403
|
+
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
|
|
404
|
+
esac
|
|
405
|
+
case "$(uname -m)" in
|
|
406
|
+
arm64|aarch64) arch=arm64 ;;
|
|
407
|
+
x86_64|amd64) arch=x64 ;;
|
|
408
|
+
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
|
|
409
|
+
esac
|
|
410
|
+
# Kept between jobs (#696): a pinned version is downloaded once per runner, not on
|
|
411
|
+
# every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
|
|
412
|
+
# which outlasts any retry, so the binary has to be on the runner already.
|
|
413
|
+
# Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
|
|
414
|
+
# per-runner directory setup-node and setup-python install into. A persistent
|
|
415
|
+
# self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
|
|
416
|
+
# "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
|
|
417
|
+
# miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
|
|
418
|
+
# downloads on every run.
|
|
419
|
+
dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
|
|
420
|
+
if [ -x "$dir/gitleaks" ]; then
|
|
421
|
+
echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
|
|
422
|
+
else
|
|
423
|
+
mkdir -p "$dir"
|
|
424
|
+
tmp="$(mktemp -d "$dir.XXXXXX")"
|
|
425
|
+
trap 'rm -rf "$tmp"' EXIT
|
|
426
|
+
asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
|
|
427
|
+
sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
|
|
428
|
+
# fetch <accept> <url> <file> — every request of this step, one curl line.
|
|
429
|
+
# --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
|
|
430
|
+
# then fails with "not in gzip format", naming neither the download nor the host.
|
|
431
|
+
# --retry: a re-run during the same upstream error spends the one re-run branch CI
|
|
432
|
+
# allows, so a brief error is retried here first. A longer outage still fails, at
|
|
433
|
+
# curl, naming the status (#690).
|
|
434
|
+
# The token reaches api.github.com only: curl drops an -H Authorization header when
|
|
435
|
+
# the asset's 302 leads to another host, so the download CDN never sees it.
|
|
436
|
+
fetch() {
|
|
437
|
+
curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
|
|
438
|
+
-H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
|
|
439
|
+
}
|
|
440
|
+
# The API serves an asset by id, not by name, so look the ids up in the release. No jq:
|
|
441
|
+
# a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
|
|
442
|
+
# whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
|
|
443
|
+
# that order ever to change, the wrong file would arrive and the sha256 check below
|
|
444
|
+
# would refuse it — a loud failure, never a wrong binary.
|
|
445
|
+
fetch application/vnd.github+json \
|
|
446
|
+
"https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
|
|
447
|
+
for f in "$asset" "$sums"; do
|
|
448
|
+
url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
|
|
449
|
+
$2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
|
|
450
|
+
$2 == "name" && $4 == want && !found { print u; found = 1 }')"
|
|
451
|
+
if [ -z "$url" ]; then
|
|
452
|
+
echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
|
|
453
|
+
fi
|
|
454
|
+
fetch application/octet-stream "$url" "$tmp/$f"
|
|
455
|
+
done
|
|
456
|
+
# Verified before tar (#698): an error page, a truncated file or a wrong archive stops
|
|
457
|
+
# here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
|
|
458
|
+
want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
|
|
459
|
+
if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
|
|
460
|
+
else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
|
|
461
|
+
if [ -z "$want" ]; then
|
|
462
|
+
echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
|
|
463
|
+
fi
|
|
464
|
+
if [ "$got" != "$want" ]; then
|
|
465
|
+
echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
|
|
466
|
+
fi
|
|
467
|
+
tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
|
|
468
|
+
chmod +x "$tmp/gitleaks"
|
|
469
|
+
# One rename on one filesystem: a job sharing this tool cache sees no binary or a
|
|
470
|
+
# whole one, never half of one.
|
|
471
|
+
mv -f "$tmp/gitleaks" "$dir/gitleaks"
|
|
472
|
+
fi
|
|
473
|
+
echo "$dir" >> "$GITHUB_PATH"
|
|
474
|
+
|
|
475
|
+
- name: gitleaks detect (full history)
|
|
476
|
+
run: gitleaks detect --source . --no-banner --log-opts=HEAD
|
|
477
|
+
|
|
331
478
|
# A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
|
|
332
479
|
# with the same title suppresses a new one; an issue list that cannot be read files nothing and
|
|
333
480
|
# fails this job, so a silent miss is never mistaken for "already reported".
|
|
334
481
|
report:
|
|
335
482
|
name: Report a red nightly
|
|
336
|
-
needs: [plan, heavy]
|
|
337
|
-
if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
|
|
483
|
+
needs: [plan, heavy, secrets]
|
|
484
|
+
if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
|
|
338
485
|
runs-on: ubuntu-latest # EDIT: needs `gh`.
|
|
339
486
|
timeout-minutes: 5
|
|
340
487
|
permissions:
|
package/templates/ci-laravel.yml
CHANGED
|
@@ -127,11 +127,15 @@ jobs:
|
|
|
127
127
|
# run whose jobs ALL skip concludes `skipped`, which `colab ship` reads as not-green.
|
|
128
128
|
# - At this SHA (#418): on a ref's first push (a claim push), a green non-PR run of this
|
|
129
129
|
# workflow at the same sha means the suite already ran.
|
|
130
|
-
# - At this TREE (#493): on a push to trunk,
|
|
131
|
-
#
|
|
132
|
-
#
|
|
133
|
-
#
|
|
134
|
-
#
|
|
130
|
+
# - At this TREE (#493): on a push to trunk, and on the first push of a batch-landing ref
|
|
131
|
+
# (`ship-batch/**`, #717), a green push run elsewhere whose commit has the byte-identical tree
|
|
132
|
+
# (a squash of a branch that already contained its base) is reused and cited in a
|
|
133
|
+
# `tree-already-green` notice. `tree-reuse: off` in .github/project.yml opts out.
|
|
134
|
+
# A run that skipped a job (a trunk-only job skips on branches) is never reused (#511) — except
|
|
135
|
+
# a job named in TRUNK_ONLY_JOBS below, which sits outside this gate and runs on trunk anyway.
|
|
136
|
+
# So a trunk-only job (publish, deploy, release) never sits behind this gate: put it in a
|
|
137
|
+
# workflow of its own, or keep it here ungated and list it (#717).
|
|
138
|
+
# The rule this keeps: one CI verdict per git tree (CONVENTIONS.md §7, "One verdict per tree").
|
|
135
139
|
# Any doubt — no run, red, an API error — runs the full suite; this job never fails the run.
|
|
136
140
|
dedupe:
|
|
137
141
|
name: Already tested at this sha or tree?
|
|
@@ -156,15 +160,23 @@ jobs:
|
|
|
156
160
|
CREATED: ${{ github.event.created }}
|
|
157
161
|
REF_NAME: ${{ github.ref_name }}
|
|
158
162
|
GUARD_NAME: Already tested at this sha or tree? # EDIT: this job's `name:`, if you rename it (#511).
|
|
163
|
+
# EDIT (#717): the `name:` of every job here that is trunk-only (publish, deploy, release) and
|
|
164
|
+
# sits OUTSIDE this gate — its `if:` never reads `needs.dedupe.outputs.tested`, so it runs on
|
|
165
|
+
# trunk whatever this guard decides. Comma-separated, exact names as a run lists them. A cited
|
|
166
|
+
# branch run may have skipped these and is still reused. Never list a suite job or any job
|
|
167
|
+
# behind the gate: it would then be skipped on trunk twice. Empty = #511 unchanged. Simpler
|
|
168
|
+
# still: keep trunk-only jobs in a workflow of their own (CONVENTIONS.md §7, "One verdict per tree").
|
|
169
|
+
TRUNK_ONLY_JOBS: ""
|
|
159
170
|
run: |
|
|
160
171
|
tested=false
|
|
161
172
|
cited=
|
|
162
173
|
# #511: a cited run is reusable only if it skipped no job but this guard. A job gated on
|
|
163
174
|
# the trunk ref is `skipped` on every branch push and a skipped job does not fail the
|
|
164
|
-
# run, so citing that run would skip the trunk-only job on trunk a second time
|
|
175
|
+
# run, so citing that run would skip the trunk-only job on trunk a second time — unless
|
|
176
|
+
# the job is named in TRUNK_ONLY_JOBS: it sits outside the gate and runs here anyway (#717).
|
|
165
177
|
reusable() {
|
|
166
178
|
if ! SK="$(gh api "repos/$REPO/actions/runs/$1/jobs?per_page=100" \
|
|
167
|
-
--jq 'if .total_count > (.jobs | length) then "(more than 100 jobs)" else [.jobs[] | select(.conclusion == "skipped" and .name != env.GUARD_NAME) | .name] | join(", ") end' 2>/dev/null)"; then
|
|
179
|
+
--jq 'if .total_count > (.jobs | length) then "(more than 100 jobs)" else [.jobs[] | select(.conclusion == "skipped" and .name != env.GUARD_NAME) | .name | select(. as $n | any((env.TRUNK_ONLY_JOBS // "") | split(",")[] | gsub("^\\s+|\\s+$"; ""); . == $n) | not)] | join(", ") end' 2>/dev/null)"; then
|
|
168
180
|
echo "::notice::Could not list the jobs of run $1 — running the full suite (#511)."
|
|
169
181
|
return 1
|
|
170
182
|
elif [ -n "$SK" ]; then
|
|
@@ -211,13 +223,16 @@ jobs:
|
|
|
211
223
|
echo "::notice::gh is not installed on this runner — running the full suite (#418)."
|
|
212
224
|
elif ! WFID="$(gh api "repos/$REPO/actions/runs/$RUN_ID" --jq .workflow_id 2>/dev/null)" || [ -z "$WFID" ]; then
|
|
213
225
|
echo "::notice::Could not read this run's workflow id — running the full suite (#418)."
|
|
214
|
-
elif [ "${CREATED:-}" = "false" ]; then
|
|
215
|
-
#
|
|
226
|
+
elif [ "${CREATED:-}" = "false" ] || [[ "$REF_NAME" == ship-batch/* ]]; then
|
|
227
|
+
# Tree mode (#493): a non-creating push to trunk, or the creating push of a batch-landing
|
|
228
|
+
# ref (#717), whose new sha often carries a tree a member branch already passed. Same
|
|
229
|
+
# checks either way; only the trunk check is the ref's own. Every branch below but one
|
|
230
|
+
# runs the suite.
|
|
216
231
|
if ! DESC="$(gh api "repos/$REPO/contents/.github/project.yml?ref=$SHA" -H 'Accept: application/vnd.github.raw' 2>/dev/null)" || [ -z "$DESC" ]; then
|
|
217
232
|
echo "::notice::Could not read .github/project.yml at $SHA — running the full suite (#493)."
|
|
218
233
|
elif grep -Eq '^tree-reuse[[:space:]]*:' <<< "$DESC"; then
|
|
219
234
|
echo "::notice::tree-reuse is declared in .github/project.yml — running the full suite (#493)."
|
|
220
|
-
elif [ "$(printf '%s\n' "$DESC" | sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" | head -n 1)" != "$REF_NAME" ]; then
|
|
235
|
+
elif [ "${CREATED:-}" = "false" ] && [ "$(printf '%s\n' "$DESC" | sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" | head -n 1)" != "$REF_NAME" ]; then
|
|
221
236
|
echo "::notice::$REF_NAME is not this repo's trunk — running the full suite (#493)."
|
|
222
237
|
elif ! OURS="$(gh api "repos/$REPO/git/commits/$SHA" --jq .tree.sha 2>/dev/null)" || [ -z "$OURS" ]; then
|
|
223
238
|
echo "::notice::Could not read the tree of $SHA — running the full suite (#493)."
|
|
@@ -256,7 +271,9 @@ jobs:
|
|
|
256
271
|
steps:
|
|
257
272
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
258
273
|
with:
|
|
259
|
-
# Full history: scan
|
|
274
|
+
# Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
|
|
275
|
+
# the two ends of that range have to be in the clone. The range itself is narrow — see
|
|
276
|
+
# "gitleaks detect" below (#716).
|
|
260
277
|
fetch-depth: 0
|
|
261
278
|
persist-credentials: false
|
|
262
279
|
|
|
@@ -363,8 +380,40 @@ jobs:
|
|
|
363
380
|
fi
|
|
364
381
|
echo "$dir" >> "$GITHUB_PATH"
|
|
365
382
|
|
|
383
|
+
# The scan range (#716): only the commits this run brings, never `git log --all`. A
|
|
384
|
+
# fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
|
|
385
|
+
# them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
|
|
386
|
+
# included, until that branch rewrote its history.
|
|
387
|
+
# - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
|
|
388
|
+
# - A trunk push: the pushed range (before..HEAD).
|
|
389
|
+
# - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
|
|
390
|
+
# own history — still never another branch's.
|
|
391
|
+
# Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
|
|
392
|
+
# commit trunk's tip can reach is scanned by the heavy lane instead (templates/
|
|
393
|
+
# ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
|
|
366
394
|
- name: gitleaks detect
|
|
367
|
-
|
|
395
|
+
env:
|
|
396
|
+
EVENT: ${{ github.event_name }}
|
|
397
|
+
REF_NAME: ${{ github.ref_name }}
|
|
398
|
+
BEFORE: ${{ github.event.before }}
|
|
399
|
+
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
|
400
|
+
run: |
|
|
401
|
+
trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
|
|
402
|
+
trunk="${trunk:-$DEFAULT_BRANCH}"
|
|
403
|
+
commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
|
|
404
|
+
if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
|
|
405
|
+
if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
|
|
406
|
+
range="$BEFORE..HEAD"
|
|
407
|
+
else
|
|
408
|
+
range=HEAD
|
|
409
|
+
fi
|
|
410
|
+
elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
|
|
411
|
+
range="refs/remotes/origin/$trunk..HEAD"
|
|
412
|
+
else
|
|
413
|
+
range=HEAD
|
|
414
|
+
fi
|
|
415
|
+
echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
|
|
416
|
+
gitleaks detect --source . --no-banner --log-opts="$range"
|
|
368
417
|
|
|
369
418
|
build:
|
|
370
419
|
name: Build (Laravel + Vite)
|
|
@@ -7,7 +7,8 @@
|
|
|
7
7
|
# HEAVY LANE (#607): the whole suite, including tests marked heavy, plus anything too slow for
|
|
8
8
|
# every merge (integration, browser/e2e, load). The light lane (ci.yml) runs on every push and
|
|
9
9
|
# is the only lane `colab ship` reads. This one runs:
|
|
10
|
-
# - NIGHTLY, on trunk's tip. Skipped when trunk has not moved since the last green nightly
|
|
10
|
+
# - NIGHTLY, on trunk's tip. Skipped when trunk has not moved since the last green nightly, or
|
|
11
|
+
# when a green dispatched run of this workflow already tested that exact sha (#719).
|
|
11
12
|
# A red nightly opens one `trunk-red` issue naming the commit range that could have caused
|
|
12
13
|
# it. It never blocks a merge.
|
|
13
14
|
# - On every OFFICIAL BUILD: release-tag.yml calls this file and publishes only when it is
|
|
@@ -22,6 +23,8 @@
|
|
|
22
23
|
# a `test:heavy` script in package.json (e.g. over `*.heavy.test.*` files). `npm test`
|
|
23
24
|
# stays light; this lane runs both.
|
|
24
25
|
#
|
|
26
|
+
# One verdict per sha: never test a commit this workflow already passed (CONVENTIONS.md §7,
|
|
27
|
+
# "One verdict per tree").
|
|
25
28
|
# Deliberately NOT triggered by push or pull_request: a run of this workflow must never count as
|
|
26
29
|
# a merge verdict (tools/lib/verify-runs.js sets schedule and dispatch runs aside).
|
|
27
30
|
# No concurrency block: a heavy run is never cancelled — a release may be waiting on it.
|
|
@@ -47,6 +50,8 @@ on:
|
|
|
47
50
|
permissions: {}
|
|
48
51
|
|
|
49
52
|
env:
|
|
53
|
+
# EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
|
|
54
|
+
GITLEAKS_VERSION: "8.30.1"
|
|
50
55
|
# EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
|
|
51
56
|
# default branch; the nightly tests THIS branch's tip.
|
|
52
57
|
TRUNK: main
|
|
@@ -96,6 +101,20 @@ jobs:
|
|
|
96
101
|
if [ "$since" = "$sha" ]; then
|
|
97
102
|
run=false
|
|
98
103
|
echo "::notice title=heavy-skipped::$TRUNK has not moved since the last green nightly ($sha)"
|
|
104
|
+
else
|
|
105
|
+
# One verdict per sha (#719): a green DISPATCHED run of this workflow already tested this
|
|
106
|
+
# exact commit (a gated landing, a release candidate) — the nightly does not test it again.
|
|
107
|
+
# `since` stays the last green NIGHTLY: the trunk-red range is read from nightlies only.
|
|
108
|
+
for id in $(gh api "repos/$REPO/actions/workflows/$wf/runs?event=workflow_dispatch&status=success&per_page=10" \
|
|
109
|
+
--jq '.workflow_runs[].id' 2>/dev/null); do
|
|
110
|
+
done_at="$(gh api "repos/$REPO/actions/runs/$id/jobs" \
|
|
111
|
+
--jq '[.jobs[] | select(.conclusion == "success" and (.name | startswith("Full suite @ "))) | .name | ltrimstr("Full suite @ ")][0] // ""' 2>/dev/null || true)"
|
|
112
|
+
if [ "$done_at" = "$sha" ]; then
|
|
113
|
+
run=false
|
|
114
|
+
echo "::notice title=heavy-skipped::$sha already passed this workflow in dispatched run $id"
|
|
115
|
+
break
|
|
116
|
+
fi
|
|
117
|
+
done
|
|
99
118
|
fi
|
|
100
119
|
fi
|
|
101
120
|
echo "run=$run" >> "$GITHUB_OUTPUT"
|
|
@@ -251,13 +270,145 @@ jobs:
|
|
|
251
270
|
# EDIT: integration, browser/e2e or load jobs go here, or as jobs of their own with
|
|
252
271
|
# `needs: plan` and the same `if:` and checkout ref as this one.
|
|
253
272
|
|
|
273
|
+
# The full-history secret scan (#716). The light lane's scan reads only the commits its run
|
|
274
|
+
# brings; this one reads every commit trunk's tip can reach — a secret committed and removed
|
|
275
|
+
# before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
|
|
276
|
+
# no merge: a red nightly files the trunk-red issue below, and a red official build holds that
|
|
277
|
+
# release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
|
|
278
|
+
# branch's own run's to report.
|
|
279
|
+
secrets:
|
|
280
|
+
name: Secret scan, full history (gitleaks)
|
|
281
|
+
needs: plan
|
|
282
|
+
if: ${{ needs.plan.outputs.run == 'true' }}
|
|
283
|
+
runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
|
|
284
|
+
timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
|
|
285
|
+
permissions:
|
|
286
|
+
contents: read
|
|
287
|
+
steps:
|
|
288
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
289
|
+
with:
|
|
290
|
+
ref: ${{ needs.plan.outputs.sha }}
|
|
291
|
+
fetch-depth: 0 # every commit the tip reaches
|
|
292
|
+
persist-credentials: false
|
|
293
|
+
|
|
294
|
+
# The light lane's restore and install steps, verbatim: keep them in step when you edit either.
|
|
295
|
+
# Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
|
|
296
|
+
# binary comes back from here and the install step below downloads nothing.
|
|
297
|
+
# - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
|
|
298
|
+
# remote cache on top was measured as a loss, not a saving (#512).
|
|
299
|
+
# - Keyed on the pin and the platform: a version bump is a miss. The post step saves
|
|
300
|
+
# only when the job passes, so a failed download is never what gets cached.
|
|
301
|
+
# A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
|
|
302
|
+
# its name: handbook-sync recognises a copy of this job by it.
|
|
303
|
+
# EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
|
|
304
|
+
- name: Restore gitleaks cache
|
|
305
|
+
if: runner.environment == 'github-hosted'
|
|
306
|
+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
307
|
+
with:
|
|
308
|
+
path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
|
|
309
|
+
key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
|
|
310
|
+
|
|
311
|
+
- name: Install gitleaks (pinned release binary)
|
|
312
|
+
env:
|
|
313
|
+
# Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
|
|
314
|
+
# measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
|
|
315
|
+
# while the API asset path below answered 200 with the same file. The job's own token
|
|
316
|
+
# is enough — reading a public release needs an identity, not a permission.
|
|
317
|
+
GH_TOKEN: ${{ github.token }}
|
|
318
|
+
run: |
|
|
319
|
+
# Deliberately NOT gitleaks/gitleaks-action@v2: that action requires a paid
|
|
320
|
+
# GITLEAKS_LICENSE for repositories owned by an organization (free only for
|
|
321
|
+
# personal repos). The release binary itself is MIT and has no such gate.
|
|
322
|
+
# Repos also carry a local pre-commit gitleaks hook; this is the second layer.
|
|
323
|
+
# The asset is OS/arch-specific, so resolve both from uname instead of
|
|
324
|
+
# assuming the GitHub-hosted default: a hardcoded linux_x64 binary on a
|
|
325
|
+
# macOS self-hosted runner is an ELF the kernel cannot exec, and the step
|
|
326
|
+
# fails with exit 126 rather than anything mentioning the platform.
|
|
327
|
+
case "$(uname -s)" in
|
|
328
|
+
Darwin) os=darwin ;;
|
|
329
|
+
Linux) os=linux ;;
|
|
330
|
+
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
|
|
331
|
+
esac
|
|
332
|
+
case "$(uname -m)" in
|
|
333
|
+
arm64|aarch64) arch=arm64 ;;
|
|
334
|
+
x86_64|amd64) arch=x64 ;;
|
|
335
|
+
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
|
|
336
|
+
esac
|
|
337
|
+
# Kept between jobs (#696): a pinned version is downloaded once per runner, not on
|
|
338
|
+
# every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
|
|
339
|
+
# which outlasts any retry, so the binary has to be on the runner already.
|
|
340
|
+
# Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
|
|
341
|
+
# per-runner directory setup-node and setup-python install into. A persistent
|
|
342
|
+
# self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
|
|
343
|
+
# "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
|
|
344
|
+
# miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
|
|
345
|
+
# downloads on every run.
|
|
346
|
+
dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
|
|
347
|
+
if [ -x "$dir/gitleaks" ]; then
|
|
348
|
+
echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
|
|
349
|
+
else
|
|
350
|
+
mkdir -p "$dir"
|
|
351
|
+
tmp="$(mktemp -d "$dir.XXXXXX")"
|
|
352
|
+
trap 'rm -rf "$tmp"' EXIT
|
|
353
|
+
asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
|
|
354
|
+
sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
|
|
355
|
+
# fetch <accept> <url> <file> — every request of this step, one curl line.
|
|
356
|
+
# --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
|
|
357
|
+
# then fails with "not in gzip format", naming neither the download nor the host.
|
|
358
|
+
# --retry: a re-run during the same upstream error spends the one re-run branch CI
|
|
359
|
+
# allows, so a brief error is retried here first. A longer outage still fails, at
|
|
360
|
+
# curl, naming the status (#690).
|
|
361
|
+
# The token reaches api.github.com only: curl drops an -H Authorization header when
|
|
362
|
+
# the asset's 302 leads to another host, so the download CDN never sees it.
|
|
363
|
+
fetch() {
|
|
364
|
+
curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
|
|
365
|
+
-H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
|
|
366
|
+
}
|
|
367
|
+
# The API serves an asset by id, not by name, so look the ids up in the release. No jq:
|
|
368
|
+
# a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
|
|
369
|
+
# whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
|
|
370
|
+
# that order ever to change, the wrong file would arrive and the sha256 check below
|
|
371
|
+
# would refuse it — a loud failure, never a wrong binary.
|
|
372
|
+
fetch application/vnd.github+json \
|
|
373
|
+
"https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
|
|
374
|
+
for f in "$asset" "$sums"; do
|
|
375
|
+
url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
|
|
376
|
+
$2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
|
|
377
|
+
$2 == "name" && $4 == want && !found { print u; found = 1 }')"
|
|
378
|
+
if [ -z "$url" ]; then
|
|
379
|
+
echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
|
|
380
|
+
fi
|
|
381
|
+
fetch application/octet-stream "$url" "$tmp/$f"
|
|
382
|
+
done
|
|
383
|
+
# Verified before tar (#698): an error page, a truncated file or a wrong archive stops
|
|
384
|
+
# here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
|
|
385
|
+
want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
|
|
386
|
+
if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
|
|
387
|
+
else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
|
|
388
|
+
if [ -z "$want" ]; then
|
|
389
|
+
echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
|
|
390
|
+
fi
|
|
391
|
+
if [ "$got" != "$want" ]; then
|
|
392
|
+
echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
|
|
393
|
+
fi
|
|
394
|
+
tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
|
|
395
|
+
chmod +x "$tmp/gitleaks"
|
|
396
|
+
# One rename on one filesystem: a job sharing this tool cache sees no binary or a
|
|
397
|
+
# whole one, never half of one.
|
|
398
|
+
mv -f "$tmp/gitleaks" "$dir/gitleaks"
|
|
399
|
+
fi
|
|
400
|
+
echo "$dir" >> "$GITHUB_PATH"
|
|
401
|
+
|
|
402
|
+
- name: gitleaks detect (full history)
|
|
403
|
+
run: gitleaks detect --source . --no-banner --log-opts=HEAD
|
|
404
|
+
|
|
254
405
|
# A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
|
|
255
406
|
# with the same title suppresses a new one; an issue list that cannot be read files nothing and
|
|
256
407
|
# fails this job, so a silent miss is never mistaken for "already reported".
|
|
257
408
|
report:
|
|
258
409
|
name: Report a red nightly
|
|
259
|
-
needs: [plan, heavy]
|
|
260
|
-
if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
|
|
410
|
+
needs: [plan, heavy, secrets]
|
|
411
|
+
if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
|
|
261
412
|
runs-on: ubuntu-latest # EDIT: needs `gh`.
|
|
262
413
|
timeout-minutes: 5
|
|
263
414
|
permissions:
|
package/templates/ci-node.yml
CHANGED
|
@@ -112,11 +112,15 @@ jobs:
|
|
|
112
112
|
# run whose jobs ALL skip concludes `skipped`, which `colab ship` reads as not-green.
|
|
113
113
|
# - At this SHA (#418): on a ref's first push (a claim push), a green non-PR run of this
|
|
114
114
|
# workflow at the same sha means the suite already ran.
|
|
115
|
-
# - At this TREE (#493): on a push to trunk,
|
|
116
|
-
#
|
|
117
|
-
#
|
|
118
|
-
#
|
|
119
|
-
#
|
|
115
|
+
# - At this TREE (#493): on a push to trunk, and on the first push of a batch-landing ref
|
|
116
|
+
# (`ship-batch/**`, #717), a green push run elsewhere whose commit has the byte-identical tree
|
|
117
|
+
# (a squash of a branch that already contained its base) is reused and cited in a
|
|
118
|
+
# `tree-already-green` notice. `tree-reuse: off` in .github/project.yml opts out.
|
|
119
|
+
# A run that skipped a job (a trunk-only job skips on branches) is never reused (#511) — except
|
|
120
|
+
# a job named in TRUNK_ONLY_JOBS below, which sits outside this gate and runs on trunk anyway.
|
|
121
|
+
# So a trunk-only job (publish, deploy, release) never sits behind this gate: put it in a
|
|
122
|
+
# workflow of its own, or keep it here ungated and list it (#717).
|
|
123
|
+
# The rule this keeps: one CI verdict per git tree (CONVENTIONS.md §7, "One verdict per tree").
|
|
120
124
|
# Any doubt — no run, red, an API error — runs the full suite; this job never fails the run.
|
|
121
125
|
dedupe:
|
|
122
126
|
name: Already tested at this sha or tree?
|
|
@@ -141,15 +145,23 @@ jobs:
|
|
|
141
145
|
CREATED: ${{ github.event.created }}
|
|
142
146
|
REF_NAME: ${{ github.ref_name }}
|
|
143
147
|
GUARD_NAME: Already tested at this sha or tree? # EDIT: this job's `name:`, if you rename it (#511).
|
|
148
|
+
# EDIT (#717): the `name:` of every job here that is trunk-only (publish, deploy, release) and
|
|
149
|
+
# sits OUTSIDE this gate — its `if:` never reads `needs.dedupe.outputs.tested`, so it runs on
|
|
150
|
+
# trunk whatever this guard decides. Comma-separated, exact names as a run lists them. A cited
|
|
151
|
+
# branch run may have skipped these and is still reused. Never list a suite job or any job
|
|
152
|
+
# behind the gate: it would then be skipped on trunk twice. Empty = #511 unchanged. Simpler
|
|
153
|
+
# still: keep trunk-only jobs in a workflow of their own (CONVENTIONS.md §7, "One verdict per tree").
|
|
154
|
+
TRUNK_ONLY_JOBS: ""
|
|
144
155
|
run: |
|
|
145
156
|
tested=false
|
|
146
157
|
cited=
|
|
147
158
|
# #511: a cited run is reusable only if it skipped no job but this guard. A job gated on
|
|
148
159
|
# the trunk ref is `skipped` on every branch push and a skipped job does not fail the
|
|
149
|
-
# run, so citing that run would skip the trunk-only job on trunk a second time
|
|
160
|
+
# run, so citing that run would skip the trunk-only job on trunk a second time — unless
|
|
161
|
+
# the job is named in TRUNK_ONLY_JOBS: it sits outside the gate and runs here anyway (#717).
|
|
150
162
|
reusable() {
|
|
151
163
|
if ! SK="$(gh api "repos/$REPO/actions/runs/$1/jobs?per_page=100" \
|
|
152
|
-
--jq 'if .total_count > (.jobs | length) then "(more than 100 jobs)" else [.jobs[] | select(.conclusion == "skipped" and .name != env.GUARD_NAME) | .name] | join(", ") end' 2>/dev/null)"; then
|
|
164
|
+
--jq 'if .total_count > (.jobs | length) then "(more than 100 jobs)" else [.jobs[] | select(.conclusion == "skipped" and .name != env.GUARD_NAME) | .name | select(. as $n | any((env.TRUNK_ONLY_JOBS // "") | split(",")[] | gsub("^\\s+|\\s+$"; ""); . == $n) | not)] | join(", ") end' 2>/dev/null)"; then
|
|
153
165
|
echo "::notice::Could not list the jobs of run $1 — running the full suite (#511)."
|
|
154
166
|
return 1
|
|
155
167
|
elif [ -n "$SK" ]; then
|
|
@@ -196,13 +208,16 @@ jobs:
|
|
|
196
208
|
echo "::notice::gh is not installed on this runner — running the full suite (#418)."
|
|
197
209
|
elif ! WFID="$(gh api "repos/$REPO/actions/runs/$RUN_ID" --jq .workflow_id 2>/dev/null)" || [ -z "$WFID" ]; then
|
|
198
210
|
echo "::notice::Could not read this run's workflow id — running the full suite (#418)."
|
|
199
|
-
elif [ "${CREATED:-}" = "false" ]; then
|
|
200
|
-
#
|
|
211
|
+
elif [ "${CREATED:-}" = "false" ] || [[ "$REF_NAME" == ship-batch/* ]]; then
|
|
212
|
+
# Tree mode (#493): a non-creating push to trunk, or the creating push of a batch-landing
|
|
213
|
+
# ref (#717), whose new sha often carries a tree a member branch already passed. Same
|
|
214
|
+
# checks either way; only the trunk check is the ref's own. Every branch below but one
|
|
215
|
+
# runs the suite.
|
|
201
216
|
if ! DESC="$(gh api "repos/$REPO/contents/.github/project.yml?ref=$SHA" -H 'Accept: application/vnd.github.raw' 2>/dev/null)" || [ -z "$DESC" ]; then
|
|
202
217
|
echo "::notice::Could not read .github/project.yml at $SHA — running the full suite (#493)."
|
|
203
218
|
elif grep -Eq '^tree-reuse[[:space:]]*:' <<< "$DESC"; then
|
|
204
219
|
echo "::notice::tree-reuse is declared in .github/project.yml — running the full suite (#493)."
|
|
205
|
-
elif [ "$(printf '%s\n' "$DESC" | sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" | head -n 1)" != "$REF_NAME" ]; then
|
|
220
|
+
elif [ "${CREATED:-}" = "false" ] && [ "$(printf '%s\n' "$DESC" | sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" | head -n 1)" != "$REF_NAME" ]; then
|
|
206
221
|
echo "::notice::$REF_NAME is not this repo's trunk — running the full suite (#493)."
|
|
207
222
|
elif ! OURS="$(gh api "repos/$REPO/git/commits/$SHA" --jq .tree.sha 2>/dev/null)" || [ -z "$OURS" ]; then
|
|
208
223
|
echo "::notice::Could not read the tree of $SHA — running the full suite (#493)."
|
|
@@ -241,8 +256,9 @@ jobs:
|
|
|
241
256
|
steps:
|
|
242
257
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
243
258
|
with:
|
|
244
|
-
# Full history:
|
|
245
|
-
#
|
|
259
|
+
# Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
|
|
260
|
+
# the two ends of that range have to be in the clone. The range itself is narrow — see
|
|
261
|
+
# "gitleaks detect" below (#716).
|
|
246
262
|
fetch-depth: 0
|
|
247
263
|
persist-credentials: false
|
|
248
264
|
|
|
@@ -353,8 +369,40 @@ jobs:
|
|
|
353
369
|
fi
|
|
354
370
|
echo "$dir" >> "$GITHUB_PATH"
|
|
355
371
|
|
|
372
|
+
# The scan range (#716): only the commits this run brings, never `git log --all`. A
|
|
373
|
+
# fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
|
|
374
|
+
# them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
|
|
375
|
+
# included, until that branch rewrote its history.
|
|
376
|
+
# - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
|
|
377
|
+
# - A trunk push: the pushed range (before..HEAD).
|
|
378
|
+
# - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
|
|
379
|
+
# own history — still never another branch's.
|
|
380
|
+
# Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
|
|
381
|
+
# commit trunk's tip can reach is scanned by the heavy lane instead (templates/
|
|
382
|
+
# ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
|
|
356
383
|
- name: gitleaks detect
|
|
357
|
-
|
|
384
|
+
env:
|
|
385
|
+
EVENT: ${{ github.event_name }}
|
|
386
|
+
REF_NAME: ${{ github.ref_name }}
|
|
387
|
+
BEFORE: ${{ github.event.before }}
|
|
388
|
+
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
|
389
|
+
run: |
|
|
390
|
+
trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
|
|
391
|
+
trunk="${trunk:-$DEFAULT_BRANCH}"
|
|
392
|
+
commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
|
|
393
|
+
if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
|
|
394
|
+
if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
|
|
395
|
+
range="$BEFORE..HEAD"
|
|
396
|
+
else
|
|
397
|
+
range=HEAD
|
|
398
|
+
fi
|
|
399
|
+
elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
|
|
400
|
+
range="refs/remotes/origin/$trunk..HEAD"
|
|
401
|
+
else
|
|
402
|
+
range=HEAD
|
|
403
|
+
fi
|
|
404
|
+
echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
|
|
405
|
+
gitleaks detect --source . --no-banner --log-opts="$range"
|
|
358
406
|
|
|
359
407
|
build:
|
|
360
408
|
name: Build
|
|
@@ -7,7 +7,8 @@
|
|
|
7
7
|
# HEAVY LANE (#607): the whole suite, including tests marked heavy, plus anything too slow for
|
|
8
8
|
# every merge (integration, browser/e2e, load). The light lane (ci.yml) runs on every push and
|
|
9
9
|
# is the only lane `colab ship` reads. This one runs:
|
|
10
|
-
# - NIGHTLY, on trunk's tip. Skipped when trunk has not moved since the last green nightly
|
|
10
|
+
# - NIGHTLY, on trunk's tip. Skipped when trunk has not moved since the last green nightly, or
|
|
11
|
+
# when a green dispatched run of this workflow already tested that exact sha (#719).
|
|
11
12
|
# A red nightly opens one `trunk-red` issue naming the commit range that could have caused
|
|
12
13
|
# it. It never blocks a merge.
|
|
13
14
|
# - On every OFFICIAL BUILD: release-tag.yml calls this file and publishes only when it is
|
|
@@ -22,6 +23,8 @@
|
|
|
22
23
|
# the pytest marker `heavy` (`@pytest.mark.heavy`, registered in pytest.ini or
|
|
23
24
|
# pyproject.toml). The light lane deselects it; this lane runs everything.
|
|
24
25
|
#
|
|
26
|
+
# One verdict per sha: never test a commit this workflow already passed (CONVENTIONS.md §7,
|
|
27
|
+
# "One verdict per tree").
|
|
25
28
|
# Deliberately NOT triggered by push or pull_request: a run of this workflow must never count as
|
|
26
29
|
# a merge verdict (tools/lib/verify-runs.js sets schedule and dispatch runs aside).
|
|
27
30
|
# No concurrency block: a heavy run is never cancelled — a release may be waiting on it.
|
|
@@ -47,6 +50,8 @@ on:
|
|
|
47
50
|
permissions: {}
|
|
48
51
|
|
|
49
52
|
env:
|
|
53
|
+
# EDIT: the same pin as ci.yml's GITLEAKS_VERSION — bump both together (job `secrets`, #716).
|
|
54
|
+
GITLEAKS_VERSION: "8.30.1"
|
|
50
55
|
# EDIT: this repo's trunk (`trunk:` in .github/project.yml). A schedule always starts on the
|
|
51
56
|
# default branch; the nightly tests THIS branch's tip.
|
|
52
57
|
TRUNK: main
|
|
@@ -98,6 +103,20 @@ jobs:
|
|
|
98
103
|
if [ "$since" = "$sha" ]; then
|
|
99
104
|
run=false
|
|
100
105
|
echo "::notice title=heavy-skipped::$TRUNK has not moved since the last green nightly ($sha)"
|
|
106
|
+
else
|
|
107
|
+
# One verdict per sha (#719): a green DISPATCHED run of this workflow already tested this
|
|
108
|
+
# exact commit (a gated landing, a release candidate) — the nightly does not test it again.
|
|
109
|
+
# `since` stays the last green NIGHTLY: the trunk-red range is read from nightlies only.
|
|
110
|
+
for id in $(gh api "repos/$REPO/actions/workflows/$wf/runs?event=workflow_dispatch&status=success&per_page=10" \
|
|
111
|
+
--jq '.workflow_runs[].id' 2>/dev/null); do
|
|
112
|
+
done_at="$(gh api "repos/$REPO/actions/runs/$id/jobs" \
|
|
113
|
+
--jq '[.jobs[] | select(.conclusion == "success" and (.name | startswith("Full suite @ "))) | .name | ltrimstr("Full suite @ ")][0] // ""' 2>/dev/null || true)"
|
|
114
|
+
if [ "$done_at" = "$sha" ]; then
|
|
115
|
+
run=false
|
|
116
|
+
echo "::notice title=heavy-skipped::$sha already passed this workflow in dispatched run $id"
|
|
117
|
+
break
|
|
118
|
+
fi
|
|
119
|
+
done
|
|
101
120
|
fi
|
|
102
121
|
fi
|
|
103
122
|
echo "run=$run" >> "$GITHUB_OUTPUT"
|
|
@@ -260,13 +279,145 @@ jobs:
|
|
|
260
279
|
# EDIT: integration, browser/e2e or load jobs go here, or as jobs of their own with
|
|
261
280
|
# `needs: plan` and the same `if:` and checkout ref as this one.
|
|
262
281
|
|
|
282
|
+
# The full-history secret scan (#716). The light lane's scan reads only the commits its run
|
|
283
|
+
# brings; this one reads every commit trunk's tip can reach — a secret committed and removed
|
|
284
|
+
# before this repo scanned in CI, or one a newer GITLEAKS_VERSION's rules now catch. It blocks
|
|
285
|
+
# no merge: a red nightly files the trunk-red issue below, and a red official build holds that
|
|
286
|
+
# release, as the suite does. `HEAD` only, never `git log --all`: a finding on a branch is that
|
|
287
|
+
# branch's own run's to report.
|
|
288
|
+
secrets:
|
|
289
|
+
name: Secret scan, full history (gitleaks)
|
|
290
|
+
needs: plan
|
|
291
|
+
if: ${{ needs.plan.outputs.run == 'true' }}
|
|
292
|
+
runs-on: ubuntu-latest # EDIT: the same label as ci.yml's gitleaks job.
|
|
293
|
+
timeout-minutes: 30 # the whole history, not a range. EDIT: the default is 360.
|
|
294
|
+
permissions:
|
|
295
|
+
contents: read
|
|
296
|
+
steps:
|
|
297
|
+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
298
|
+
with:
|
|
299
|
+
ref: ${{ needs.plan.outputs.sha }}
|
|
300
|
+
fetch-depth: 0 # every commit the tip reaches
|
|
301
|
+
persist-credentials: false
|
|
302
|
+
|
|
303
|
+
# The light lane's restore and install steps, verbatim: keep them in step when you edit either.
|
|
304
|
+
# Kept between jobs (#696) — GitHub-hosted only: a fresh VM has nothing on disk, so the
|
|
305
|
+
# binary comes back from here and the install step below downloads nothing.
|
|
306
|
+
# - Self-hosted → off. A persistent runner keeps its tool cache on disk already, and a
|
|
307
|
+
# remote cache on top was measured as a loss, not a saving (#512).
|
|
308
|
+
# - Keyed on the pin and the platform: a version bump is a miss. The post step saves
|
|
309
|
+
# only when the job passes, so a failed download is never what gets cached.
|
|
310
|
+
# A NEW step, skipped on self-hosted. "Install gitleaks (pinned release binary)" keeps
|
|
311
|
+
# its name: handbook-sync recognises a copy of this job by it.
|
|
312
|
+
# EDIT: self-hosted runners are already off; to turn it off everywhere, delete this step.
|
|
313
|
+
- name: Restore gitleaks cache
|
|
314
|
+
if: runner.environment == 'github-hosted'
|
|
315
|
+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
316
|
+
with:
|
|
317
|
+
path: ${{ runner.tool_cache }}/gitleaks/${{ env.GITLEAKS_VERSION }}
|
|
318
|
+
key: gitleaks-${{ env.GITLEAKS_VERSION }}-${{ runner.os }}-${{ runner.arch }}
|
|
319
|
+
|
|
320
|
+
- name: Install gitleaks (pinned release binary)
|
|
321
|
+
env:
|
|
322
|
+
# Authenticated (#698). Under load GitHub sheds ANONYMOUS release downloads first:
|
|
323
|
+
# measured 2026-11-09, ~90 min of instant 504s from github.com/…/releases/download,
|
|
324
|
+
# while the API asset path below answered 200 with the same file. The job's own token
|
|
325
|
+
# is enough — reading a public release needs an identity, not a permission.
|
|
326
|
+
GH_TOKEN: ${{ github.token }}
|
|
327
|
+
run: |
|
|
328
|
+
# Deliberately NOT gitleaks/gitleaks-action@v2: that action requires a paid
|
|
329
|
+
# GITLEAKS_LICENSE for repositories owned by an organization (free only for
|
|
330
|
+
# personal repos). The release binary itself is MIT and has no such gate.
|
|
331
|
+
# Repos also carry a local pre-commit gitleaks hook; this is the second layer.
|
|
332
|
+
# The asset is OS/arch-specific, so resolve both from uname instead of
|
|
333
|
+
# assuming the GitHub-hosted default: a hardcoded linux_x64 binary on a
|
|
334
|
+
# macOS self-hosted runner is an ELF the kernel cannot exec, and the step
|
|
335
|
+
# fails with exit 126 rather than anything mentioning the platform.
|
|
336
|
+
case "$(uname -s)" in
|
|
337
|
+
Darwin) os=darwin ;;
|
|
338
|
+
Linux) os=linux ;;
|
|
339
|
+
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
|
|
340
|
+
esac
|
|
341
|
+
case "$(uname -m)" in
|
|
342
|
+
arm64|aarch64) arch=arm64 ;;
|
|
343
|
+
x86_64|amd64) arch=x64 ;;
|
|
344
|
+
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
|
|
345
|
+
esac
|
|
346
|
+
# Kept between jobs (#696): a pinned version is downloaded once per runner, not on
|
|
347
|
+
# every run. A release download can answer 504 for hours (measured: 20 of 20 tries),
|
|
348
|
+
# which outlasts any retry, so the binary has to be on the runner already.
|
|
349
|
+
# Into the runner's tool cache, not /usr/local/bin: no sudo, no global write — the
|
|
350
|
+
# per-runner directory setup-node and setup-python install into. A persistent
|
|
351
|
+
# self-hosted runner keeps it on disk; a GitHub-hosted VM gets it back from the
|
|
352
|
+
# "Restore gitleaks cache" step above. The version is in the path, so a pin bump is a
|
|
353
|
+
# miss, never a stale binary. EDIT: an ephemeral self-hosted runner keeps neither and
|
|
354
|
+
# downloads on every run.
|
|
355
|
+
dir="${RUNNER_TOOL_CACHE:-$RUNNER_TEMP}/gitleaks/${GITLEAKS_VERSION}/${os}-${arch}"
|
|
356
|
+
if [ -x "$dir/gitleaks" ]; then
|
|
357
|
+
echo "gitleaks ${GITLEAKS_VERSION} already on this runner: $dir"
|
|
358
|
+
else
|
|
359
|
+
mkdir -p "$dir"
|
|
360
|
+
tmp="$(mktemp -d "$dir.XXXXXX")"
|
|
361
|
+
trap 'rm -rf "$tmp"' EXIT
|
|
362
|
+
asset="gitleaks_${GITLEAKS_VERSION}_${os}_${arch}.tar.gz"
|
|
363
|
+
sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
|
|
364
|
+
# fetch <accept> <url> <file> — every request of this step, one curl line.
|
|
365
|
+
# --fail: on an HTTP error curl otherwise saves the error page and exits 0, and tar
|
|
366
|
+
# then fails with "not in gzip format", naming neither the download nor the host.
|
|
367
|
+
# --retry: a re-run during the same upstream error spends the one re-run branch CI
|
|
368
|
+
# allows, so a brief error is retried here first. A longer outage still fails, at
|
|
369
|
+
# curl, naming the status (#690).
|
|
370
|
+
# The token reaches api.github.com only: curl drops an -H Authorization header when
|
|
371
|
+
# the asset's 302 leads to another host, so the download CDN never sees it.
|
|
372
|
+
fetch() {
|
|
373
|
+
curl -fsSL --retry 5 --retry-delay 10 --retry-all-errors --connect-timeout 10 --max-time 120 \
|
|
374
|
+
-H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: $1" -o "$3" "$2"
|
|
375
|
+
}
|
|
376
|
+
# The API serves an asset by id, not by name, so look the ids up in the release. No jq:
|
|
377
|
+
# a self-hosted runner need not have it. Splitting on , { [ leaves one key per line
|
|
378
|
+
# whether the JSON comes pretty or compact; within an asset, "url" precedes "name". Were
|
|
379
|
+
# that order ever to change, the wrong file would arrive and the sha256 check below
|
|
380
|
+
# would refuse it — a loud failure, never a wrong binary.
|
|
381
|
+
fetch application/vnd.github+json \
|
|
382
|
+
"https://api.github.com/repos/gitleaks/gitleaks/releases/tags/v${GITLEAKS_VERSION}" "$tmp/release.json"
|
|
383
|
+
for f in "$asset" "$sums"; do
|
|
384
|
+
url="$(tr ',{[' '\n\n\n' < "$tmp/release.json" | awk -F'"' -v want="$f" '
|
|
385
|
+
$2 == "url" && $4 ~ /\/releases\/assets\/[0-9]+$/ { u = $4 }
|
|
386
|
+
$2 == "name" && $4 == want && !found { print u; found = 1 }')"
|
|
387
|
+
if [ -z "$url" ]; then
|
|
388
|
+
echo "::error::gitleaks release v${GITLEAKS_VERSION} has no asset $f"; exit 1
|
|
389
|
+
fi
|
|
390
|
+
fetch application/octet-stream "$url" "$tmp/$f"
|
|
391
|
+
done
|
|
392
|
+
# Verified before tar (#698): an error page, a truncated file or a wrong archive stops
|
|
393
|
+
# here, naming the file, instead of reaching tar — or, for a valid archive, the PATH.
|
|
394
|
+
want="$(awk -v f="$asset" '$2 == f { print $1 }' "$tmp/$sums")"
|
|
395
|
+
if command -v sha256sum >/dev/null; then got="$(sha256sum "$tmp/$asset" | awk '{ print $1 }')"
|
|
396
|
+
else got="$(shasum -a 256 "$tmp/$asset" | awk '{ print $1 }')"; fi # macOS self-hosted
|
|
397
|
+
if [ -z "$want" ]; then
|
|
398
|
+
echo "::error::$sums does not list $asset, so it cannot be verified"; exit 1
|
|
399
|
+
fi
|
|
400
|
+
if [ "$got" != "$want" ]; then
|
|
401
|
+
echo "::error::sha256 mismatch for $asset: got $got, $sums lists $want"; exit 1
|
|
402
|
+
fi
|
|
403
|
+
tar -xzf "$tmp/$asset" -C "$tmp" gitleaks
|
|
404
|
+
chmod +x "$tmp/gitleaks"
|
|
405
|
+
# One rename on one filesystem: a job sharing this tool cache sees no binary or a
|
|
406
|
+
# whole one, never half of one.
|
|
407
|
+
mv -f "$tmp/gitleaks" "$dir/gitleaks"
|
|
408
|
+
fi
|
|
409
|
+
echo "$dir" >> "$GITHUB_PATH"
|
|
410
|
+
|
|
411
|
+
- name: gitleaks detect (full history)
|
|
412
|
+
run: gitleaks detect --source . --no-banner --log-opts=HEAD
|
|
413
|
+
|
|
263
414
|
# A red NIGHTLY is a trunk finding, never a merge block: one open issue per lane. An open issue
|
|
264
415
|
# with the same title suppresses a new one; an issue list that cannot be read files nothing and
|
|
265
416
|
# fails this job, so a silent miss is never mistaken for "already reported".
|
|
266
417
|
report:
|
|
267
418
|
name: Report a red nightly
|
|
268
|
-
needs: [plan, heavy]
|
|
269
|
-
if: ${{ always() && github.event_name == 'schedule' && needs.heavy.result == 'failure' }}
|
|
419
|
+
needs: [plan, heavy, secrets]
|
|
420
|
+
if: ${{ always() && github.event_name == 'schedule' && (needs.heavy.result == 'failure' || needs.secrets.result == 'failure') }}
|
|
270
421
|
runs-on: ubuntu-latest # EDIT: needs `gh`.
|
|
271
422
|
timeout-minutes: 5
|
|
272
423
|
permissions:
|
package/templates/ci-python.yml
CHANGED
|
@@ -123,11 +123,15 @@ jobs:
|
|
|
123
123
|
# run whose jobs ALL skip concludes `skipped`, which `colab ship` reads as not-green.
|
|
124
124
|
# - At this SHA (#418): on a ref's first push (a claim push), a green non-PR run of this
|
|
125
125
|
# workflow at the same sha means the suite already ran.
|
|
126
|
-
# - At this TREE (#493): on a push to trunk,
|
|
127
|
-
#
|
|
128
|
-
#
|
|
129
|
-
#
|
|
130
|
-
#
|
|
126
|
+
# - At this TREE (#493): on a push to trunk, and on the first push of a batch-landing ref
|
|
127
|
+
# (`ship-batch/**`, #717), a green push run elsewhere whose commit has the byte-identical tree
|
|
128
|
+
# (a squash of a branch that already contained its base) is reused and cited in a
|
|
129
|
+
# `tree-already-green` notice. `tree-reuse: off` in .github/project.yml opts out.
|
|
130
|
+
# A run that skipped a job (a trunk-only job skips on branches) is never reused (#511) — except
|
|
131
|
+
# a job named in TRUNK_ONLY_JOBS below, which sits outside this gate and runs on trunk anyway.
|
|
132
|
+
# So a trunk-only job (publish, deploy, release) never sits behind this gate: put it in a
|
|
133
|
+
# workflow of its own, or keep it here ungated and list it (#717).
|
|
134
|
+
# The rule this keeps: one CI verdict per git tree (CONVENTIONS.md §7, "One verdict per tree").
|
|
131
135
|
# Any doubt — no run, red, an API error — runs the full suite; this job never fails the run.
|
|
132
136
|
dedupe:
|
|
133
137
|
name: Already tested at this sha or tree?
|
|
@@ -152,15 +156,23 @@ jobs:
|
|
|
152
156
|
CREATED: ${{ github.event.created }}
|
|
153
157
|
REF_NAME: ${{ github.ref_name }}
|
|
154
158
|
GUARD_NAME: Already tested at this sha or tree? # EDIT: this job's `name:`, if you rename it (#511).
|
|
159
|
+
# EDIT (#717): the `name:` of every job here that is trunk-only (publish, deploy, release) and
|
|
160
|
+
# sits OUTSIDE this gate — its `if:` never reads `needs.dedupe.outputs.tested`, so it runs on
|
|
161
|
+
# trunk whatever this guard decides. Comma-separated, exact names as a run lists them. A cited
|
|
162
|
+
# branch run may have skipped these and is still reused. Never list a suite job or any job
|
|
163
|
+
# behind the gate: it would then be skipped on trunk twice. Empty = #511 unchanged. Simpler
|
|
164
|
+
# still: keep trunk-only jobs in a workflow of their own (CONVENTIONS.md §7, "One verdict per tree").
|
|
165
|
+
TRUNK_ONLY_JOBS: ""
|
|
155
166
|
run: |
|
|
156
167
|
tested=false
|
|
157
168
|
cited=
|
|
158
169
|
# #511: a cited run is reusable only if it skipped no job but this guard. A job gated on
|
|
159
170
|
# the trunk ref is `skipped` on every branch push and a skipped job does not fail the
|
|
160
|
-
# run, so citing that run would skip the trunk-only job on trunk a second time
|
|
171
|
+
# run, so citing that run would skip the trunk-only job on trunk a second time — unless
|
|
172
|
+
# the job is named in TRUNK_ONLY_JOBS: it sits outside the gate and runs here anyway (#717).
|
|
161
173
|
reusable() {
|
|
162
174
|
if ! SK="$(gh api "repos/$REPO/actions/runs/$1/jobs?per_page=100" \
|
|
163
|
-
--jq 'if .total_count > (.jobs | length) then "(more than 100 jobs)" else [.jobs[] | select(.conclusion == "skipped" and .name != env.GUARD_NAME) | .name] | join(", ") end' 2>/dev/null)"; then
|
|
175
|
+
--jq 'if .total_count > (.jobs | length) then "(more than 100 jobs)" else [.jobs[] | select(.conclusion == "skipped" and .name != env.GUARD_NAME) | .name | select(. as $n | any((env.TRUNK_ONLY_JOBS // "") | split(",")[] | gsub("^\\s+|\\s+$"; ""); . == $n) | not)] | join(", ") end' 2>/dev/null)"; then
|
|
164
176
|
echo "::notice::Could not list the jobs of run $1 — running the full suite (#511)."
|
|
165
177
|
return 1
|
|
166
178
|
elif [ -n "$SK" ]; then
|
|
@@ -207,13 +219,16 @@ jobs:
|
|
|
207
219
|
echo "::notice::gh is not installed on this runner — running the full suite (#418)."
|
|
208
220
|
elif ! WFID="$(gh api "repos/$REPO/actions/runs/$RUN_ID" --jq .workflow_id 2>/dev/null)" || [ -z "$WFID" ]; then
|
|
209
221
|
echo "::notice::Could not read this run's workflow id — running the full suite (#418)."
|
|
210
|
-
elif [ "${CREATED:-}" = "false" ]; then
|
|
211
|
-
#
|
|
222
|
+
elif [ "${CREATED:-}" = "false" ] || [[ "$REF_NAME" == ship-batch/* ]]; then
|
|
223
|
+
# Tree mode (#493): a non-creating push to trunk, or the creating push of a batch-landing
|
|
224
|
+
# ref (#717), whose new sha often carries a tree a member branch already passed. Same
|
|
225
|
+
# checks either way; only the trunk check is the ref's own. Every branch below but one
|
|
226
|
+
# runs the suite.
|
|
212
227
|
if ! DESC="$(gh api "repos/$REPO/contents/.github/project.yml?ref=$SHA" -H 'Accept: application/vnd.github.raw' 2>/dev/null)" || [ -z "$DESC" ]; then
|
|
213
228
|
echo "::notice::Could not read .github/project.yml at $SHA — running the full suite (#493)."
|
|
214
229
|
elif grep -Eq '^tree-reuse[[:space:]]*:' <<< "$DESC"; then
|
|
215
230
|
echo "::notice::tree-reuse is declared in .github/project.yml — running the full suite (#493)."
|
|
216
|
-
elif [ "$(printf '%s\n' "$DESC" | sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" | head -n 1)" != "$REF_NAME" ]; then
|
|
231
|
+
elif [ "${CREATED:-}" = "false" ] && [ "$(printf '%s\n' "$DESC" | sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" | head -n 1)" != "$REF_NAME" ]; then
|
|
217
232
|
echo "::notice::$REF_NAME is not this repo's trunk — running the full suite (#493)."
|
|
218
233
|
elif ! OURS="$(gh api "repos/$REPO/git/commits/$SHA" --jq .tree.sha 2>/dev/null)" || [ -z "$OURS" ]; then
|
|
219
234
|
echo "::notice::Could not read the tree of $SHA — running the full suite (#493)."
|
|
@@ -252,8 +267,9 @@ jobs:
|
|
|
252
267
|
steps:
|
|
253
268
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
|
254
269
|
with:
|
|
255
|
-
# Full history:
|
|
256
|
-
#
|
|
270
|
+
# Full history: the scan's range is computed against trunk (origin/<trunk>..HEAD), and
|
|
271
|
+
# the two ends of that range have to be in the clone. The range itself is narrow — see
|
|
272
|
+
# "gitleaks detect" below (#716).
|
|
257
273
|
fetch-depth: 0
|
|
258
274
|
persist-credentials: false
|
|
259
275
|
|
|
@@ -364,8 +380,40 @@ jobs:
|
|
|
364
380
|
fi
|
|
365
381
|
echo "$dir" >> "$GITHUB_PATH"
|
|
366
382
|
|
|
383
|
+
# The scan range (#716): only the commits this run brings, never `git log --all`. A
|
|
384
|
+
# fetch-depth-0 checkout holds every branch and tag, and gitleaks' default range is all of
|
|
385
|
+
# them — so one unshipped branch's finding turned EVERY branch's run red, a trunk cure
|
|
386
|
+
# included, until that branch rewrote its history.
|
|
387
|
+
# - A branch push: the commits not yet on trunk (origin/<trunk>..HEAD).
|
|
388
|
+
# - A trunk push: the pushed range (before..HEAD).
|
|
389
|
+
# - No range to read (a dispatch, a first or forced trunk push, trunk not fetched): HEAD's
|
|
390
|
+
# own history — still never another branch's.
|
|
391
|
+
# Trunk is `trunk:` in .github/project.yml, else the repository's default branch. Every
|
|
392
|
+
# commit trunk's tip can reach is scanned by the heavy lane instead (templates/
|
|
393
|
+
# ci-<stack>-heavy.yml, job `secrets`), which blocks no merge. The step keeps its name.
|
|
367
394
|
- name: gitleaks detect
|
|
368
|
-
|
|
395
|
+
env:
|
|
396
|
+
EVENT: ${{ github.event_name }}
|
|
397
|
+
REF_NAME: ${{ github.ref_name }}
|
|
398
|
+
BEFORE: ${{ github.event.before }}
|
|
399
|
+
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
|
400
|
+
run: |
|
|
401
|
+
trunk="$(sed -nE "s/^trunk:[[:space:]]*[\"']?([A-Za-z0-9._/-]+).*/\\1/p" .github/project.yml 2>/dev/null | head -n 1 || true)"
|
|
402
|
+
trunk="${trunk:-$DEFAULT_BRANCH}"
|
|
403
|
+
commit() { [ -n "$1" ] && git rev-parse --verify -q "$1^{commit}" >/dev/null; }
|
|
404
|
+
if [ -n "$trunk" ] && [ "$REF_NAME" = "$trunk" ]; then
|
|
405
|
+
if [ "$EVENT" = push ] && commit "$BEFORE" && git merge-base --is-ancestor "$BEFORE" HEAD; then
|
|
406
|
+
range="$BEFORE..HEAD"
|
|
407
|
+
else
|
|
408
|
+
range=HEAD
|
|
409
|
+
fi
|
|
410
|
+
elif [ -n "$trunk" ] && commit "refs/remotes/origin/$trunk"; then
|
|
411
|
+
range="refs/remotes/origin/$trunk..HEAD"
|
|
412
|
+
else
|
|
413
|
+
range=HEAD
|
|
414
|
+
fi
|
|
415
|
+
echo "Secret scan range: $range ($(git rev-list --count "$range") commit(s))"
|
|
416
|
+
gitleaks detect --source . --no-banner --log-opts="$range"
|
|
369
417
|
|
|
370
418
|
build:
|
|
371
419
|
name: Build (Python)
|
package/tools/lib/stamp.js
CHANGED
|
@@ -515,7 +515,7 @@ const WORKFLOW_FINGERPRINTS = [
|
|
|
515
515
|
{ marker: 'Run exactly this tag, verify (deploy-container-run --deploy-only)', kind: 'step', template: 'deploy-container' },
|
|
516
516
|
// branch-name (#348): its one step.
|
|
517
517
|
{ marker: 'Check branch name (CONVENTIONS §4)', kind: 'step', template: 'branch-name' },
|
|
518
|
-
// Shared by
|
|
518
|
+
// Shared by the ci-* templates, light and heavy (#716): proves derivation, cannot say from which. Note the exact
|
|
519
519
|
// parenthetical — hand-written installers on this fleet say "(pinned binary)", which is why the
|
|
520
520
|
// step NAME discriminates where the URL inside the step does not.
|
|
521
521
|
{ marker: 'Install gitleaks (pinned release binary)', kind: 'step', template: null },
|