@avi2dg/checks 0.63.1 → 0.65.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/docs/bins.md +3 -4
- package/docs/design.md +3 -2
- package/docs/gates/checks-advisories.md +8 -6
- package/docs/gates/checks-ci-wiring.md +45 -7
- package/docs/gates/checks-flake.md +6 -7
- package/docs/gates/checks-lint.md +1 -1
- package/docs/gates/checks-mutation-scope.md +1 -1
- package/docs/gates/checks-mutation.md +1 -0
- package/docs/gates/checks-night-run.md +32 -26
- package/docs/gates/checks-release-notes.md +5 -0
- package/docs/gates/checks-release-pr.md +2 -1
- package/docs/gates/checks-release-push.md +4 -3
- package/docs/gates/checks-setup.md +2 -0
- package/docs/gates/checks-test.md +28 -1
- package/docs/how-to/set-up-a-repository.md +4 -0
- package/package.json +5 -1
- package/src/delivery/ci-wiring.ts +20 -7
- package/src/delivery/job-timeout.ts +30 -0
- package/src/delivery/release-push.ts +1 -1
- package/src/delivery/release-schedule.ts +53 -18
- package/src/delivery/schedule-policy.ts +64 -0
- package/src/delivery/setup-workflow.ts +43 -0
- package/src/delivery/setup.ts +2 -39
- package/src/testing/affected-tests.ts +261 -0
- package/src/testing/mutation-scope-graph.ts +13 -0
- package/src/testing/mutation-scope.ts +2 -9
- package/src/testing/night-run-findings.ts +7 -4
- package/src/testing/night-run.ts +43 -16
- package/src/testing/test-skips.ts +3 -3
- package/src/testing/test.ts +57 -10
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,22 @@
|
|
|
2
2
|
|
|
3
3
|
Every release of `@avi2dg/checks`, newest first, written by the release from its conventional commits.
|
|
4
4
|
|
|
5
|
+
## 0.65.0
|
|
6
|
+
|
|
7
|
+
Released 2026-11-03.
|
|
8
|
+
|
|
9
|
+
### Features
|
|
10
|
+
|
|
11
|
+
- **ci-wiring:** one CI run per pull request, refuse schedules and long jobs [#41](https://github.com/avi2d/checks/pull/41)
|
|
12
|
+
|
|
13
|
+
## 0.64.0
|
|
14
|
+
|
|
15
|
+
Released 2026-11-03.
|
|
16
|
+
|
|
17
|
+
### Features
|
|
18
|
+
|
|
19
|
+
- **testing:** shard the night mutation run and file each missing report [#27](https://github.com/avi2d/checks/pull/27)
|
|
20
|
+
|
|
5
21
|
## 0.63.1
|
|
6
22
|
|
|
7
23
|
Released 2026-11-03.
|
package/docs/bins.md
CHANGED
|
@@ -44,15 +44,14 @@ The night-run workflow runs them with `checks-lint --night`, which hands each ra
|
|
|
44
44
|
These bins run on their own:
|
|
45
45
|
|
|
46
46
|
- [`checks-test`](gates/checks-test.md) runs the suite as `scripts.test` and refuses a skip without a reason at its test site.
|
|
47
|
-
- [`checks-flake`](gates/checks-flake.md) runs the suite
|
|
47
|
+
- [`checks-flake`](gates/checks-flake.md) runs the suite when started by hand and records the seeds a flaky test fails with.
|
|
48
48
|
- [`checks-mutation`](gates/checks-mutation.md) runs Stryker for scoped checks and refuses a full run outside CI.
|
|
49
49
|
- [`checks-mutation-compare`](gates/checks-mutation-compare.md) compares the mutants of two reports and lists each one that regresses.
|
|
50
50
|
- [`checks-mutation-baseline`](gates/checks-mutation-baseline.md) restores the newest mutation baseline from `main`, saves it on a self-hosted runner's disk in place of an artifact, and deletes superseded baselines after a `main` run.
|
|
51
51
|
- [`checks-mutation-scope`](gates/checks-mutation-scope.md) turns a branch's diff into the files its mutation runs mutate.
|
|
52
52
|
- [`checks-subsumed-tests`](gates/checks-subsumed-tests.md) lists each test another test subsumes in a mutation run.
|
|
53
|
-
- [`checks-night-run`](gates/checks-night-run.md) opens or comments on one GitHub issue per finding of a night run on `main`, each file whose mutation score fell, each failing test,
|
|
54
|
-
It closes the issue of a test
|
|
55
|
-
It comments on a mutation issue a later night no longer reports, and leaves it open.
|
|
53
|
+
- [`checks-night-run`](gates/checks-night-run.md) opens or comments on one GitHub issue per finding of a night run on `main`, each file whose mutation score fell, each failing test, each failed nightly gate, and each missing mutation report.
|
|
54
|
+
It closes the issue of a test, a gate or a missing mutation report that a later night no longer reports, and it comments on a mutation issue it no longer reports without closing it, but only when a previous night exists and every requested mutation report is present.
|
|
56
55
|
- [`checks-changelog`](gates/checks-changelog.md) writes the pending release into `CHANGELOG.md` from the conventional commits since the last release.
|
|
57
56
|
- [`checks-release-notes`](gates/checks-release-notes.md) writes one `CHANGELOG.md` section to a file for a GitHub release.
|
|
58
57
|
- [`checks-release-report`](gates/checks-release-report.md) tells whether the history holds unreleased features or fixes since the last tag.
|
package/docs/design.md
CHANGED
|
@@ -204,7 +204,8 @@ A daily release pull request would leave each merged change unpublished until a
|
|
|
204
204
|
Only then does it move `main` to it without a force, so `main` only ever gains a release commit whose checks passed.
|
|
205
205
|
It tags the commit it landed and nothing else.
|
|
206
206
|
A repository that is no package keeps the release pull request, and it opens only when someone runs the workflow.
|
|
207
|
-
|
|
207
|
+
Only a public package that publishes to npm may release on a schedule, at most once a day.
|
|
208
|
+
A schedule elsewhere cuts releases no one asked for.
|
|
208
209
|
`checks-release-pr` writes only to the `release/<branch>` branch it owns, and rebuilds that branch on `main` rather than merging `main` into it, so the changelog it carries is the one the build writes.
|
|
209
210
|
`checks-release-tag` writes only the `v*` tag of a release commit that already landed.
|
|
210
211
|
The `github-release` job's `contents: write` creates or updates the GitHub release from the tag's `CHANGELOG.md` section.
|
|
@@ -281,7 +282,7 @@ A group or other write bit fails the run, because another user could have edited
|
|
|
281
282
|
|
|
282
283
|
`checks-advisories` compares the advisories at both ends of a range instead of failing on every advisory at the head.
|
|
283
284
|
An advisory published against a package that landed a month earlier would otherwise fail every open pull request, including one that touches only docs.
|
|
284
|
-
The
|
|
285
|
+
The weekly `--all` run finds those advisories, and `advisory-acks.json` carries the ones a repository accepts for a while.
|
|
285
286
|
|
|
286
287
|
The gate runs OSV-Scanner rather than Trivy, Grype or `bun audit`.
|
|
287
288
|
Trivy 0.74.0 reads a nested `bun.lock` entry such as `mkdirp/minimist` under a name no advisory carries, so it misses every package bun nests.
|
|
@@ -14,7 +14,7 @@ It ignores aliases, because two live advisories can list each other as aliases.
|
|
|
14
14
|
An advisory published against a package the base already held shows at both ends, so it fails no range, and `--all` reports it instead.
|
|
15
15
|
Each failure names the package, its version, the advisory id, its severity and its summary.
|
|
16
16
|
A range that leaves `bun.lock` unchanged runs no scan.
|
|
17
|
-
A run `checks-lint` starts defers the scan to the
|
|
17
|
+
A run `checks-lint` starts defers the scan to the weekly `--all` run instead, so the desk and the pull request neither fetch the database nor scan.
|
|
18
18
|
It also holds `advisory-acks.json` to the rules in [The acknowledgement file](#the-acknowledgement-file) on every run.
|
|
19
19
|
|
|
20
20
|
## What it reads
|
|
@@ -63,7 +63,7 @@ An entry whose `until` falls more than 30 days after that moment fails the run a
|
|
|
63
63
|
Once that moment passes `until`, the entry fails every run until it is deleted or renewed with a later `until`, whether or not the range touches `bun.lock`.
|
|
64
64
|
Delete it once the package is upgraded.
|
|
65
65
|
When a scan runs, an entry that matches no advisory at the head fails, so the file holds only live entries.
|
|
66
|
-
A deferred run still holds each entry to its day, while an entry that matches nothing waits for the
|
|
66
|
+
A deferred run still holds each entry to its day, while an entry that matches nothing waits for the weekly run.
|
|
67
67
|
A file that does not decode as that list exits 2.
|
|
68
68
|
|
|
69
69
|
## Arguments
|
|
@@ -99,18 +99,20 @@ advisories: 1 acknowledgement(s) in advisory-acks.json do not hold:
|
|
|
99
99
|
## When it runs
|
|
100
100
|
|
|
101
101
|
`checks-lint` runs it over each pull request's range in a repository that tracks `bun.lock`, as [checks-lint](checks-lint.md) says.
|
|
102
|
-
That run defers the scan to the
|
|
102
|
+
That run defers the scan to the weekly run, so it fetches no database and reports the deferral instead of a verdict.
|
|
103
103
|
A range that leaves `bun.lock` unchanged runs in under a second, and a direct run over a range that changes `bun.lock` scans for about 10 seconds on a warm cache.
|
|
104
104
|
|
|
105
105
|
## Running it on a schedule
|
|
106
106
|
|
|
107
|
-
A range never fails on an advisory published after its package landed, so a
|
|
108
|
-
|
|
107
|
+
A range never fails on an advisory published after its package landed, so a weekly `--all` run finds those.
|
|
108
|
+
Only a public repository that publishes its package to npm runs it on a schedule, as [checks-ci-wiring](checks-ci-wiring.md#schedules) requires.
|
|
109
|
+
Any other repository starts the same workflow by hand with `gh workflow run`, with no `schedule` trigger.
|
|
110
|
+
A package scans on Monday between 05:00 and 06:00 UTC, at a minute chosen so repositories do not start together:
|
|
109
111
|
|
|
110
112
|
```yaml
|
|
111
113
|
on:
|
|
112
114
|
schedule:
|
|
113
|
-
- cron: "15 5 * *
|
|
115
|
+
- cron: "15 5 * * 1"
|
|
114
116
|
workflow_dispatch:
|
|
115
117
|
jobs:
|
|
116
118
|
advisories:
|
|
@@ -86,7 +86,8 @@ Each refusal names the workflow file and line, the version found and where it wa
|
|
|
86
86
|
|
|
87
87
|
### Release schedule
|
|
88
88
|
|
|
89
|
-
It refuses a workflow that triggers on `schedule` and releases,
|
|
89
|
+
It refuses a workflow that triggers on `schedule` and releases, unless the repository is a public package that publishes to npm.
|
|
90
|
+
That release runs at most once a day.
|
|
90
91
|
A workflow releases when a run step calls `checks-release-pr`, `checks-release-push`, `checks-release-tag` or `checks-release-publish`.
|
|
91
92
|
A call counts directly, through `bunx`, or through `bun` or `bun run` with the bin's path or its source file.
|
|
92
93
|
A call through a `package.json` script counts, as `bun run <script>` where the named script releases.
|
|
@@ -94,12 +95,34 @@ A call through a local reusable workflow counts, as a job that uses a file under
|
|
|
94
95
|
A call through a repository shell script counts, as a path inside the repository whose file releases.
|
|
95
96
|
A workflow also releases when a run step publishes to npm, as [checks-release-preflight](checks-release-preflight.md) reads it, directly or through one of those paths.
|
|
96
97
|
Text that only names a release bin, such as an `echo`, never counts.
|
|
97
|
-
A
|
|
98
|
-
|
|
98
|
+
A repository publishes to npm when one of its workflows runs `npm publish` or `checks-release-publish`, the same ways a release call counts, and its `package.json` does not set `private` to `true`.
|
|
99
|
+
A run outside CI cannot read whether the repository is private, so there it refuses only for the `package.json` field or the missing publish step.
|
|
100
|
+
A daily release has one `cron` entry with one minute and one hour.
|
|
101
|
+
Any other release runs when someone starts it, as [checks-release-pr](checks-release-pr.md#when-it-runs) says.
|
|
102
|
+
Each refusal names the workflow file, the line of its `schedule` trigger and what to change.
|
|
103
|
+
|
|
104
|
+
### Schedules
|
|
105
|
+
|
|
106
|
+
It refuses every other workflow that triggers on `schedule`, since a test, mutation, flake, browser or night run starts only by hand, with `gh workflow run`.
|
|
107
|
+
The one schedule it keeps is a weekly advisory scan in a public repository whose `package.json` does not set `private` to `true`.
|
|
108
|
+
A workflow scans advisories when a run step calls `checks-advisories`, the same ways a release call counts, and every other run step is a `bun install`.
|
|
109
|
+
A weekly scan has one `cron` entry with one minute, one hour and one weekday, and `*` for the day of the month and the month.
|
|
110
|
+
It refuses an advisory scan that runs more often, and any scheduled advisory scan in a private repository or a package that sets `private`.
|
|
111
|
+
A run outside CI cannot read whether the repository is private, so it refuses a scan there only for the `package.json` field or the cron.
|
|
112
|
+
A workflow that releases on a schedule is judged only by the release rule above.
|
|
113
|
+
Each refusal names the workflow file, the line of its `schedule` trigger and what to change.
|
|
114
|
+
|
|
115
|
+
### Job timeouts
|
|
116
|
+
|
|
117
|
+
Every job sets `timeout-minutes` to a number of at most 60, so no job runs for hours.
|
|
118
|
+
A job without one would hold its runner for GitHub's 360-minute default when a step hangs.
|
|
119
|
+
It refuses a job that sets no `timeout-minutes`, sets more than 60, or sets an expression it cannot read before the run.
|
|
120
|
+
A job that calls a reusable workflow with `uses` sets no timeout, and the called workflow's jobs are judged in their own file.
|
|
121
|
+
Each refusal names the workflow file, the job and the limit.
|
|
99
122
|
|
|
100
123
|
## What it reads
|
|
101
124
|
|
|
102
|
-
The bin reads `.github/workflows/*.yml`, `*.yaml`, the `scripts` and `checks` runners in `package.json` and the shell scripts those steps and scripts run from the working tree.
|
|
125
|
+
The bin reads `.github/workflows/*.yml`, `*.yaml`, the `scripts`, `private` and `checks` runners in `package.json` and the shell scripts those steps and scripts run from the working tree.
|
|
103
126
|
It reads each file a setup-bun step names for its version.
|
|
104
127
|
It reads the fleet pin from the kit's own `.bun-version`.
|
|
105
128
|
It reads the target branch from `GITHUB_BASE_REF`, then from `refs/remotes/origin/HEAD` and then from the event file `GITHUB_EVENT_PATH` names.
|
|
@@ -115,7 +138,7 @@ It takes no arguments.
|
|
|
115
138
|
| Code | Result |
|
|
116
139
|
| --- | --- |
|
|
117
140
|
| 0 | Every required command has a reachable step. |
|
|
118
|
-
| 1 | A required command is missing or blocked, a job runs on the wrong runner, a workflow names a missing workflow or job, a setup-bun step drifts from the fleet bun version,
|
|
141
|
+
| 1 | A required command is missing or blocked, a job runs on the wrong runner, a workflow names a missing workflow or job, a setup-bun step drifts from the fleet bun version, a workflow that releases runs on a schedule outside a public npm package or more than once a day, a workflow other than a weekly public advisory scan runs on a schedule, or a job can run for more than 60 minutes. |
|
|
119
142
|
| 2 | A workflow or `package.json` cannot be decoded, or the kit's own `.bun-version` is missing, unreadable or blank. |
|
|
120
143
|
|
|
121
144
|
## Sample output
|
|
@@ -142,11 +165,26 @@ ci-wiring: 1 reference(s) name a missing workflow or job:
|
|
|
142
165
|
.github/workflows/daily-release.yml line 31: names missing workflow commitlint.yml
|
|
143
166
|
```
|
|
144
167
|
|
|
145
|
-
A workflow that releases on a schedule produces a report like this:
|
|
168
|
+
A workflow that releases on a schedule in a repository that publishes nothing to npm produces a report like this:
|
|
146
169
|
|
|
147
170
|
```
|
|
148
171
|
ci-wiring: 1 workflow(s) release on a schedule:
|
|
149
|
-
.github/workflows/daily-release.yml line 3: drop the schedule, so a release runs only when someone starts it
|
|
172
|
+
.github/workflows/daily-release.yml line 3: only a public package that publishes to npm releases on a schedule; drop the schedule, so a release runs only when someone starts it
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
A scheduled test run and a daily advisory scan produce a report like this:
|
|
176
|
+
|
|
177
|
+
```
|
|
178
|
+
ci-wiring: 2 workflow(s) run on a schedule:
|
|
179
|
+
.github/workflows/advisories.yml line 3: the advisory scan runs more than once a week; keep one cron at a minute, an hour and a weekday, such as "15 5 * * 1"
|
|
180
|
+
.github/workflows/night-run.yml line 4: drop the schedule, so it runs only when someone starts it with gh workflow run
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
A job that sets no timeout produces a report like this:
|
|
184
|
+
|
|
185
|
+
```
|
|
186
|
+
ci-wiring: 1 job(s) can run for hours:
|
|
187
|
+
.github/workflows/ci.yml job checks: the job sets no timeout-minutes, so a hung run holds its runner for GitHub's 360 minute default; set timeout-minutes to a number of at most 60
|
|
150
188
|
```
|
|
151
189
|
|
|
152
190
|
## When it runs
|
|
@@ -4,7 +4,7 @@ audience: consumers
|
|
|
4
4
|
---
|
|
5
5
|
# checks-flake
|
|
6
6
|
|
|
7
|
-
`checks-flake` is the
|
|
7
|
+
`checks-flake` is the run someone starts by hand that finds flaky tests and records the seeds each one fails with.
|
|
8
8
|
|
|
9
9
|
## What it checks
|
|
10
10
|
|
|
@@ -100,21 +100,20 @@ Rerunning a seed with bun test --randomize --seed=<seed> --parallel=4 reproduces
|
|
|
100
100
|
|
|
101
101
|
## When it runs
|
|
102
102
|
|
|
103
|
-
Only a
|
|
103
|
+
Only a workflow the repository starts by hand runs it.
|
|
104
104
|
|
|
105
|
-
## Running it
|
|
105
|
+
## Running it by hand
|
|
106
106
|
|
|
107
|
-
A repository runs it
|
|
108
|
-
|
|
107
|
+
A repository runs it from a workflow someone starts with `gh workflow run`, and keeps the record of a failed run as an artifact.
|
|
108
|
+
The workflow holds no schedule, since [checks-ci-wiring](checks-ci-wiring.md#schedules) refuses one:
|
|
109
109
|
|
|
110
110
|
```yaml
|
|
111
111
|
on:
|
|
112
|
-
schedule:
|
|
113
|
-
- cron: "27 22 * * *"
|
|
114
112
|
workflow_dispatch:
|
|
115
113
|
jobs:
|
|
116
114
|
flake:
|
|
117
115
|
runs-on: ubuntu-latest
|
|
116
|
+
timeout-minutes: 60
|
|
118
117
|
steps:
|
|
119
118
|
- uses: actions/checkout@v5
|
|
120
119
|
- uses: oven-sh/setup-bun@v2
|
|
@@ -18,7 +18,7 @@ A test file classifies as TypeScript, so its change runs every applicable gate t
|
|
|
18
18
|
Gates requiring tracked TypeScript files begin running when the repository tracks TypeScript.
|
|
19
19
|
`checks-lint-coverage` and `checks-graphwalk` also begin running when the repository tracks an `.astro` file.
|
|
20
20
|
`checks-advisories` begins running when the repository tracks `bun.lock`.
|
|
21
|
-
`checks-lint` defers its scan to the
|
|
21
|
+
`checks-lint` defers its scan to the weekly run, so the desk and the pull request fetch no database.
|
|
22
22
|
`checks-frontend-syntax` begins running when the repository tracks `frontend-syntax.json` at its root.
|
|
23
23
|
`checks-copy` begins running when the repository tracks `copy.json` at its root.
|
|
24
24
|
`checks-css-size` begins running when the repository tracks a `*.css` file.
|
|
@@ -38,7 +38,7 @@ Stryker writes no source without mutants into the report, so a mutated source th
|
|
|
38
38
|
When a unit test changed, each such source that exists at `HEAD` enters the scope.
|
|
39
39
|
A change under `tests/e2e/` changes no unit test, so it needs no baseline report.
|
|
40
40
|
The scope trusts the coverage edges of the latest full baseline.
|
|
41
|
-
So a helper, fixture or preload changed on `main` since that baseline can hide a lost kill until the
|
|
41
|
+
So a helper, fixture or preload changed on `main` since that baseline can hide a lost kill until the next full baseline refreshes.
|
|
42
42
|
|
|
43
43
|
## What it reads
|
|
44
44
|
|
|
@@ -4,13 +4,13 @@ audience: consumers
|
|
|
4
4
|
---
|
|
5
5
|
# checks-night-run
|
|
6
6
|
|
|
7
|
-
`checks-night-run` turns a night run on `main` into GitHub
|
|
8
|
-
|
|
9
|
-
It
|
|
7
|
+
`checks-night-run` turns each finding of a night run on `main` into a GitHub issue.
|
|
8
|
+
A finding is a file whose mutation score fell, a failing test, a failed nightly gate or a missing mutation report.
|
|
9
|
+
It closes the issue of a test, gate or missing mutation report that a later night no longer reports, and it comments on a mutation issue it no longer reports without closing it, but only when a previous night exists and every requested mutation report is present.
|
|
10
10
|
|
|
11
11
|
## What it checks
|
|
12
12
|
|
|
13
|
-
It scores each file of tonight's Stryker
|
|
13
|
+
It scores each file of tonight's Stryker reports as Stryker does, the `Killed` and `Timeout` mutants over every mutant but the `Ignored`, `CompileError`, `RuntimeError` and `Pending` ones.
|
|
14
14
|
It compares each file's score with the same file's score from the previous night.
|
|
15
15
|
A file whose score is lower than the previous night's is a finding.
|
|
16
16
|
A file only one night holds is no finding, since it lacks one of the two scores.
|
|
@@ -22,7 +22,10 @@ A log where every named gate is undecided files `checks-lint` itself, with no pe
|
|
|
22
22
|
A log with no passing run and no named gate files `checks-lint` itself, so a gate run that never finished still reaches an issue.
|
|
23
23
|
A file whose test worker crashed is one finding, since Bun reports none of its tests.
|
|
24
24
|
With no previous night it compares no score and still files each failing test.
|
|
25
|
+
Each mutation report `--mutation` names that does not exist is a finding, since the mutation job that writes it failed, timed out or was cancelled.
|
|
25
26
|
With no mutation report it compares and records no score and still files each failing test.
|
|
27
|
+
With some mutation reports missing it compares the files the others hold and records no score.
|
|
28
|
+
The next night then compares every file with the last night that scored them all.
|
|
26
29
|
|
|
27
30
|
Each finding has a key that stays the same from night to night:
|
|
28
31
|
|
|
@@ -32,6 +35,7 @@ Each finding has a key that stays the same from night to night:
|
|
|
32
35
|
| a failing test | `test:<file> > <describe> > <name>` |
|
|
33
36
|
| a file whose test worker crashed | `test:<file>` |
|
|
34
37
|
| a nightly gate that failed | `gates:<gate>` |
|
|
38
|
+
| a missing mutation report | `mutation-report:<path>` |
|
|
35
39
|
|
|
36
40
|
## How it files a finding
|
|
37
41
|
|
|
@@ -42,15 +46,14 @@ Any other finding opens one issue labelled `night-run`.
|
|
|
42
46
|
The issue and the comment name tonight's run, the file and the test, the score of both nights or the failed gate.
|
|
43
47
|
The body holds the file, the full test name or the failed gate, and the finding key.
|
|
44
48
|
A title longer than GitHub's 256 characters is cut to at most 256 and ends in `…`.
|
|
45
|
-
A night with no finding creates no label and
|
|
46
|
-
It still reads the open issues, to close or comment on each one whose finding is gone.
|
|
49
|
+
A night with no finding creates no label and writes no issue, and only reads the open issues to close any test, gate or mutation-report issue whose finding is gone and, when a previous night exists and every requested mutation report is present, to comment on any mutation issue whose finding is gone.
|
|
47
50
|
Closing an issue ends it, and the same finding on a later night opens a new one.
|
|
48
51
|
|
|
49
52
|
## How it closes an issue
|
|
50
53
|
|
|
51
|
-
An open test or
|
|
54
|
+
An open test, gate or mutation-report issue whose key tonight's findings lack gets one comment naming tonight's run and the key.
|
|
52
55
|
It then closes as completed.
|
|
53
|
-
|
|
56
|
+
When a previous night exists and every requested mutation report is present, an open mutation issue whose key tonight's findings lack gets one comment that names tonight's run and the key, and stays open.
|
|
54
57
|
A closed issue is no longer read, so a later night without the finding comments on it no more.
|
|
55
58
|
A comment that reports the finding carries a reported marker, and a note that a mutation finding is gone carries a noted marker.
|
|
56
59
|
A mutation issue whose latest marked comment already notes the finding gone gets no further note.
|
|
@@ -63,13 +66,14 @@ It judges a key only from a report it read tonight:
|
|
|
63
66
|
| `test:<file> > <describe> > <name>` | tonight read a JUnit report, the test is not failing, and `<file>`'s worker did not crash |
|
|
64
67
|
| `test:<file>` | tonight read a JUnit report and `<file>`'s worker did not crash |
|
|
65
68
|
| `gates:<gate>` | tonight read a lint log that passes or names its failed gates, and `<gate>` is not among them |
|
|
69
|
+
| `mutation-report:<path>` | tonight read the mutation report at `<path>` |
|
|
66
70
|
| `mutation:<file>` | never |
|
|
67
71
|
|
|
68
72
|
A crashed worker loses every result of its file, so the issue of a test in that file stays open.
|
|
69
73
|
A lint log that is undecided or never finished says nothing of any one gate, so every gate issue stays open.
|
|
70
74
|
A mutation finding is a fall from one night to the next.
|
|
71
75
|
A later night that does not fall again still holds the lower score.
|
|
72
|
-
|
|
76
|
+
Only a night that compared scores with every requested mutation report present comments on the mutation issue without closing it, since a score can recover and drop again.
|
|
73
77
|
So a person closes a mutation issue once the score is back.
|
|
74
78
|
An issue with no key in its body is never read, so a person's issue under the `night-run` label is never commented on or closed.
|
|
75
79
|
A run off `main` comments on and closes no issue.
|
|
@@ -78,24 +82,24 @@ A run on any ref but `refs/heads/main` files no issue and records no score, and
|
|
|
78
82
|
|
|
79
83
|
## What it reads
|
|
80
84
|
|
|
81
|
-
It reads tonight's Stryker `mutation.json`
|
|
85
|
+
It reads tonight's Stryker `mutation.json` reports and any number of JUnit reports, such as the ones `checks-test --junit=<file>` keeps, as [checks-test](checks-test.md#arguments) says.
|
|
82
86
|
It reads the lint log `--gates` names for the `gate(s) failed:` line `checks-lint --night` prints.
|
|
83
87
|
It scans the newest 30 successful runs of the same workflow on `main` from newest to oldest.
|
|
84
88
|
It takes the first run whose `night-run` artifact still exists as the previous night.
|
|
85
89
|
That artifact holds `night-run.json`, the scores the run recorded.
|
|
86
90
|
When no scanned run holds the artifact it compares no score.
|
|
87
91
|
A run on another branch never becomes the previous night.
|
|
88
|
-
On `main` with
|
|
92
|
+
On `main` with every mutation report present it writes tonight's scores to `reports/night-run.json`, and the workflow uploads that file as the `night-run` artifact.
|
|
89
93
|
It reads the run from `GITHUB_SERVER_URL`, `GITHUB_REPOSITORY`, `GITHUB_RUN_ID`, `GITHUB_WORKFLOW_REF` and `GITHUB_REF`, so it runs only inside a GitHub Actions job.
|
|
90
94
|
It calls `gh` with the job's `GH_TOKEN`, which needs `actions: read` for the previous night and `issues: write` for the issues.
|
|
91
95
|
|
|
92
96
|
## Arguments
|
|
93
97
|
|
|
94
98
|
```sh
|
|
95
|
-
checks-night-run [--mutation <mutation-report>] [--tests <junit-report>]... [--gates <lint-log>]
|
|
99
|
+
checks-night-run [--mutation <mutation-report>]... [--tests <junit-report>]... [--gates <lint-log>]
|
|
96
100
|
```
|
|
97
101
|
|
|
98
|
-
`--mutation` names tonight's Stryker
|
|
102
|
+
`--mutation` names one of tonight's Stryker reports and is given once per report the workflow's mutation jobs write, whether or not each job wrote it.
|
|
99
103
|
`--tests` names one JUnit report and is given once per report.
|
|
100
104
|
`--gates` names tonight's `checks-lint --night` log and is given at most once.
|
|
101
105
|
At least one report is given.
|
|
@@ -123,15 +127,14 @@ checks-night-run: recorded 42 file score(s) in reports/night-run.json
|
|
|
123
127
|
|
|
124
128
|
Only a night workflow the repository writes runs it, after that workflow's full mutation run, its long test tiers and its nightly lint gates.
|
|
125
129
|
|
|
126
|
-
## Running it
|
|
130
|
+
## Running it by hand
|
|
127
131
|
|
|
128
|
-
A repository runs it from a workflow named `night-run
|
|
132
|
+
A repository runs it from a workflow named `night-run`, which someone starts with `gh workflow run night-run`.
|
|
133
|
+
The workflow holds no schedule, since [checks-ci-wiring](checks-ci-wiring.md#schedules) refuses one:
|
|
129
134
|
|
|
130
135
|
```yaml
|
|
131
136
|
name: night-run
|
|
132
137
|
on:
|
|
133
|
-
schedule:
|
|
134
|
-
- cron: "7 22 * * *"
|
|
135
138
|
workflow_dispatch:
|
|
136
139
|
permissions:
|
|
137
140
|
contents: read
|
|
@@ -156,6 +159,7 @@ jobs:
|
|
|
156
159
|
retention-days: 1
|
|
157
160
|
tests:
|
|
158
161
|
runs-on: ubuntu-latest
|
|
162
|
+
timeout-minutes: 20
|
|
159
163
|
steps:
|
|
160
164
|
- uses: actions/checkout@v5
|
|
161
165
|
- uses: oven-sh/setup-bun@v2
|
|
@@ -172,6 +176,7 @@ jobs:
|
|
|
172
176
|
retention-days: 1
|
|
173
177
|
gates:
|
|
174
178
|
runs-on: ubuntu-latest
|
|
179
|
+
timeout-minutes: 20
|
|
175
180
|
steps:
|
|
176
181
|
- uses: actions/checkout@v5
|
|
177
182
|
with:
|
|
@@ -190,6 +195,7 @@ jobs:
|
|
|
190
195
|
needs: [mutation, gates, tests]
|
|
191
196
|
if: ${{ !cancelled() && needs.tests.result == 'success' }}
|
|
192
197
|
runs-on: ubuntu-latest
|
|
198
|
+
timeout-minutes: 15
|
|
193
199
|
permissions:
|
|
194
200
|
contents: read
|
|
195
201
|
actions: read
|
|
@@ -200,10 +206,9 @@ jobs:
|
|
|
200
206
|
with:
|
|
201
207
|
bun-version-file: .bun-version
|
|
202
208
|
- run: bun install --frozen-lockfile
|
|
203
|
-
-
|
|
204
|
-
uses: actions/download-artifact@v4
|
|
209
|
+
- uses: actions/download-artifact@v4
|
|
205
210
|
with:
|
|
206
|
-
|
|
211
|
+
pattern: night-run-mutation
|
|
207
212
|
path: reports/mutation
|
|
208
213
|
- if: needs.gates.result == 'success'
|
|
209
214
|
uses: actions/download-artifact@v4
|
|
@@ -216,7 +221,7 @@ jobs:
|
|
|
216
221
|
path: reports/night-run-tests
|
|
217
222
|
- env:
|
|
218
223
|
GH_TOKEN: ${{ github.token }}
|
|
219
|
-
run: bunx checks-night-run
|
|
224
|
+
run: bunx checks-night-run --mutation reports/mutation/night-run-mutation/mutation.json --tests reports/night-run-tests/live.xml --tests reports/night-run-tests/pixel.xml ${{ needs.gates.result == 'success' && '--gates reports/night-run-gates/gates.log' || '' }}
|
|
220
225
|
- uses: actions/upload-artifact@v4
|
|
221
226
|
with:
|
|
222
227
|
name: night-run
|
|
@@ -225,22 +230,23 @@ jobs:
|
|
|
225
230
|
retention-days: 90
|
|
226
231
|
```
|
|
227
232
|
|
|
228
|
-
The cron starts the run at 22:07 UTC, inside the kit's own night window from 22:00 to 22:30 UTC.
|
|
229
|
-
A repository takes minute 7 of its own tier's 30 minute night window, so repositories do not start together.
|
|
230
233
|
The tests job runs each long tier the repository holds, `live` and `pixel` as [checks-test](checks-test.md#test-tiers) names them, and keeps each JUnit report.
|
|
231
234
|
Each `|| true` lets the job go on to the upload after a failing test, since that test becomes an issue.
|
|
232
235
|
A tier that writes no report fails the report job, because `checks-night-run` cannot read it.
|
|
233
236
|
The gates job runs the nightly lint gates and keeps the log, since a failing gate becomes an issue.
|
|
234
237
|
The report job runs whenever the tests job succeeds, so a failing test that also stops Stryker still becomes an issue.
|
|
235
|
-
It passes `--mutation
|
|
238
|
+
It always passes `--mutation`, so a mutation job that failed or timed out leaves a missing report that becomes an issue.
|
|
239
|
+
The download names the artifact by `pattern`, which downloads nothing rather than failing when the artifact is missing.
|
|
240
|
+
A full run too long for one job splits into a matrix of shards.
|
|
241
|
+
Each shard uploads its own artifact under the one pattern, and each report gets its own `--mutation`.
|
|
236
242
|
It passes `--gates` only when the gates job succeeded, and the failed gates job still fails the run.
|
|
237
|
-
The concurrency group queues a run
|
|
243
|
+
The concurrency group queues a run behind one already going, so two runs never file the same finding twice.
|
|
238
244
|
A dispatch on a branch runs both jobs and lists its findings in the report job's log, and `if-no-files-found: ignore` lets that run upload no record.
|
|
239
245
|
`issues: write` on the report job is the one permission beyond reading the repository and its runs.
|
|
240
246
|
A public repository keeps `runs-on: ubuntu-latest` on every job.
|
|
241
247
|
A private repository sets `runs-on` on the mutation job to the labels declared under `checks.mutationRunner` in `package.json`, and [checks-ci-wiring](checks-ci-wiring.md#runners) refuses a mutation job that names `CI_RUNS_ON`.
|
|
242
248
|
A private repository sets `runs-on: ${{ vars.CI_RUNS_ON || 'ubuntu-latest' }}` on the tests, gates, and report jobs.
|
|
243
|
-
A successful run uploads `night-run`, so the next night compares with it, and a failed
|
|
249
|
+
A successful run with every mutation report present uploads `night-run`, so the next night compares with it, and a run that failed or missed a report uploads nothing a later night reads.
|
|
244
250
|
The artifact keeps 90 days, so a gap of nights still leaves a previous night to compare with.
|
|
245
251
|
|
|
246
252
|
## Related topics
|
|
@@ -62,6 +62,7 @@ permissions:
|
|
|
62
62
|
jobs:
|
|
63
63
|
preflight:
|
|
64
64
|
runs-on: ubuntu-latest
|
|
65
|
+
timeout-minutes: 10
|
|
65
66
|
permissions:
|
|
66
67
|
contents: read
|
|
67
68
|
steps:
|
|
@@ -77,6 +78,7 @@ jobs:
|
|
|
77
78
|
publish:
|
|
78
79
|
needs: preflight
|
|
79
80
|
runs-on: ubuntu-latest
|
|
81
|
+
timeout-minutes: 15
|
|
80
82
|
steps:
|
|
81
83
|
- uses: actions/checkout@v5
|
|
82
84
|
with:
|
|
@@ -99,6 +101,7 @@ jobs:
|
|
|
99
101
|
github-release:
|
|
100
102
|
needs: publish
|
|
101
103
|
runs-on: ubuntu-latest
|
|
104
|
+
timeout-minutes: 10
|
|
102
105
|
permissions:
|
|
103
106
|
contents: write
|
|
104
107
|
steps:
|
|
@@ -130,6 +133,7 @@ permissions:
|
|
|
130
133
|
jobs:
|
|
131
134
|
preflight:
|
|
132
135
|
runs-on: ubuntu-latest
|
|
136
|
+
timeout-minutes: 10
|
|
133
137
|
permissions:
|
|
134
138
|
contents: read
|
|
135
139
|
steps:
|
|
@@ -145,6 +149,7 @@ jobs:
|
|
|
145
149
|
github-release:
|
|
146
150
|
needs: preflight
|
|
147
151
|
runs-on: ubuntu-latest
|
|
152
|
+
timeout-minutes: 10
|
|
148
153
|
permissions:
|
|
149
154
|
contents: write
|
|
150
155
|
steps:
|
|
@@ -78,8 +78,9 @@ release-pr: https://github.com/acme/widget/pull/12 releases 0.4.0 from 3f2a9c81d
|
|
|
78
78
|
|
|
79
79
|
The `release-pr` workflow below runs it on the default branch when someone starts it with `gh workflow run release-pr`.
|
|
80
80
|
The workflow has no schedule, so a release is cut only when someone asks for one.
|
|
81
|
+
A public package that publishes to npm may add one daily `cron` to it instead, such as `"7 3 * * *"`.
|
|
81
82
|
A package published to npm releases each push to `main` through [checks-release-push](checks-release-push.md) instead, with no pull request.
|
|
82
|
-
[checks-ci-wiring](checks-ci-wiring.md#release-schedule) refuses a schedule on any workflow that releases.
|
|
83
|
+
[checks-ci-wiring](checks-ci-wiring.md#release-schedule) refuses a schedule on any other workflow that releases, and one that runs more than once a day.
|
|
83
84
|
|
|
84
85
|
## Running it in CI
|
|
85
86
|
|
|
@@ -15,7 +15,7 @@ It refuses when `v<version>` of the next version already tags a commit, since a
|
|
|
15
15
|
It writes the next version into `package.json` and runs `bun run build`, which writes `CHANGELOG.md`.
|
|
16
16
|
It commits both as `chore: release <version>` on top of `HEAD` through the GitHub API, as `checks-release-pr` does.
|
|
17
17
|
It points the branch `release/<branch>` at that commit and dispatches every check workflow its arguments name there.
|
|
18
|
-
It waits up to
|
|
18
|
+
It waits up to 22 minutes in all for those workflows to finish on the release commit.
|
|
19
19
|
It reads the newest run of each, so a rerun stands in for the run it repeats.
|
|
20
20
|
A failed read of the runs list does not end the wait.
|
|
21
21
|
The wait retries the read at the same interval.
|
|
@@ -26,7 +26,8 @@ When all of them pass, it moves `<branch>` to the release commit without a force
|
|
|
26
26
|
GitHub refuses that update when `<branch>` moved while the checks ran.
|
|
27
27
|
The run then ends cleanly, since the run of the push that moved it releases the new head.
|
|
28
28
|
When the update fails but `<branch>` already holds the release commit, the update landed and the run goes on.
|
|
29
|
-
It then tags the release commit `v<version>`, pushes the tag and dispatches the release workflow on it
|
|
29
|
+
It then tags the release commit `v<version>`, pushes the tag and dispatches the release workflow on it.
|
|
30
|
+
It waits up to 22 minutes for that workflow's run on the release commit to finish.
|
|
30
31
|
When that run concludes short of success, or does not finish in time, the bin fails, though `<branch>` already holds the release and the tag exists.
|
|
31
32
|
When `HEAD` is itself a release commit, it tags it as [checks-release-tag](checks-release-tag.md) does, and waits for the release workflow it dispatches the same way.
|
|
32
33
|
So a rerun after a failed tag push finishes the release.
|
|
@@ -104,7 +105,7 @@ jobs:
|
|
|
104
105
|
release:
|
|
105
106
|
if: github.ref == 'refs/heads/main'
|
|
106
107
|
runs-on: ubuntu-latest
|
|
107
|
-
timeout-minutes:
|
|
108
|
+
timeout-minutes: 55
|
|
108
109
|
concurrency:
|
|
109
110
|
group: release-push
|
|
110
111
|
cancel-in-progress: false
|
|
@@ -25,6 +25,8 @@ The workflow always runs `bun run lint`, `bun run typecheck` and `bun run test`.
|
|
|
25
25
|
A `build` script there also brings a `git diff --exit-code` step.
|
|
26
26
|
Other scripts a `package.json` it finds already holds stay, but the workflow never schedules them.
|
|
27
27
|
The workflow always ends with a `bunx checks-commit-title` step for the pull request title.
|
|
28
|
+
It runs on pull requests alone, so a merge to the default branch does not run the suite a second time.
|
|
29
|
+
A newer push to the same pull request cancels the run in progress.
|
|
28
30
|
A `--private` run pins each job to `${{ vars.CI_RUNS_ON || 'ubuntu-latest' }}`.
|
|
29
31
|
Setting the `CI_RUNS_ON` variable to a self-hosted runner label then lands the jobs there.
|
|
30
32
|
A public run pins each job to `ubuntu-latest`.
|
|
@@ -45,13 +45,14 @@ A skipped test or a todo without `skipReason` fails in every environment.
|
|
|
45
45
|
## What it reads
|
|
46
46
|
|
|
47
47
|
`checks-test` reads test source files and the JUnit report written by its own run.
|
|
48
|
+
A pull request run also reads the event file, the git history between the base and the head, the tracked sources, `package.json` and `bunfig.toml`.
|
|
48
49
|
It reads no skip list from `package.json`, only the smoke list.
|
|
49
50
|
It counts a run as `ci` when `CI` is true and as `local` otherwise.
|
|
50
51
|
A local run covers `./tests/unit` plus the smoke list the repository declares.
|
|
51
52
|
The smoke list lives in `package.json` under `checks.smoke` as an array of test file paths.
|
|
52
53
|
A repository with no smoke list runs the unit tier alone outside CI.
|
|
53
54
|
A local run with no unit test and no smoke list runs nothing and passes.
|
|
54
|
-
A `ci` run covers the full default suite.
|
|
55
|
+
A `ci` run covers the full default suite, unless a pull request narrows it, as [Pull request runs](#pull-request-runs) says.
|
|
55
56
|
A smoke entry that names no test file fails the run.
|
|
56
57
|
A smoke entry under `tests/quarantine/`, `tests/live/` or `tests/pixel/` fails the run, since Bun's ignored paths would drop it.
|
|
57
58
|
|
|
@@ -71,6 +72,32 @@ A unit or e2e run keeps Bun's ignored paths, runs only `./tests/unit` or `./test
|
|
|
71
72
|
|
|
72
73
|
Files under `tests/quarantine/` are not run or judged, as [checks-test-layout](checks-test-layout.md) says.
|
|
73
74
|
|
|
75
|
+
## Pull request runs
|
|
76
|
+
|
|
77
|
+
A `ci` run on a `pull_request` event runs only the test files the pull request reaches.
|
|
78
|
+
It reads the base and head from the event file `GITHUB_EVENT_PATH` names and the changed paths since their merge base from git.
|
|
79
|
+
A test reaches a changed file when it imports it, names its path, runs its `package.json` bin or imports the package by name.
|
|
80
|
+
A test also reaches a file through a test helper or a source that reaches it, at any depth.
|
|
81
|
+
A source reaches only what it imports, so a registry that lists every bin's path does not make each of its importers reach every bin.
|
|
82
|
+
A test that names a fixture's directory reaches each fixture under it.
|
|
83
|
+
A changed test always reaches itself.
|
|
84
|
+
|
|
85
|
+
A shared file runs the whole suite instead:
|
|
86
|
+
|
|
87
|
+
- a lockfile, `package.json`, a `*.config.*` or `tsconfig` file, `bunfig.toml`, or a tool pin such as `.bun-version`
|
|
88
|
+
- a file Bun preloads into every test, or a file a preload imports
|
|
89
|
+
- a test helper, which is a file under `tests/` that is neither a test nor a fixture
|
|
90
|
+
- a stylesheet, a font, or a layout under `layouts/` or named `layout.*`
|
|
91
|
+
|
|
92
|
+
A dependency bump changes `package.json` or the lockfile, so it runs the whole suite in every repository.
|
|
93
|
+
A workflow under `.github/` is not a shared file, since the suite runs under it rather than reading it.
|
|
94
|
+
A pull request that changes no file a test reaches runs no test and passes.
|
|
95
|
+
The run names how many test files it reached, or which shared file made it run the whole suite.
|
|
96
|
+
A change whose paths cannot be read runs the whole suite.
|
|
97
|
+
A run on any other event, such as `workflow_dispatch` for a release, runs the whole suite.
|
|
98
|
+
`--full` and the live and pixel tiers always run whole.
|
|
99
|
+
A narrowed run judges only the skips the files it ran declare.
|
|
100
|
+
|
|
74
101
|
## Test tiers
|
|
75
102
|
|
|
76
103
|
Tests that need a live machine go in `tests/live/`, and tests that need a screen go in `tests/pixel/`.
|
|
@@ -83,9 +83,13 @@ To run the gates in CI:
|
|
|
83
83
|
```yaml
|
|
84
84
|
on:
|
|
85
85
|
pull_request:
|
|
86
|
+
concurrency:
|
|
87
|
+
group: ci-${{ github.event.pull_request.number }}
|
|
88
|
+
cancel-in-progress: true
|
|
86
89
|
jobs:
|
|
87
90
|
checks:
|
|
88
91
|
runs-on: ubuntu-latest
|
|
92
|
+
timeout-minutes: 30
|
|
89
93
|
steps:
|
|
90
94
|
- uses: actions/checkout@v5
|
|
91
95
|
with:
|