@gfdelarue/testkit 0.5.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (215) hide show
  1. package/README.md +590 -165
  2. package/bin/testkit-windows-owner-arm64.exe +0 -0
  3. package/bin/testkit-windows-owner-x64.exe +0 -0
  4. package/dist/app/bugs/batch.js +1 -0
  5. package/dist/app/bugs/ci-report.js +2 -2
  6. package/dist/app/bugs/ci-summary.js +3 -3
  7. package/dist/app/bugs/draft.js +2 -2
  8. package/dist/app/bugs/file.js +1 -1
  9. package/dist/app/bugs/guide.js +1 -1
  10. package/dist/app/bugs/record.js +2 -2
  11. package/dist/app/bugs/run-reading.js +1 -1
  12. package/dist/app/bugs/store.js +1 -1
  13. package/dist/app/campaigns/index.js +2 -2
  14. package/dist/app/campaigns/summary.js +1 -1
  15. package/dist/app/cleanup/index.js +1 -1
  16. package/dist/app/db-schema/prepare.js +1 -1
  17. package/dist/app/db-schema/refresh.js +1 -1
  18. package/dist/app/db-schema/verify.js +1 -1
  19. package/dist/app/destroy/index.js +1 -1
  20. package/dist/app/discover/index.js +2 -2
  21. package/dist/app/environments/config-driven.js +1 -1
  22. package/dist/app/environments/file-setup.js +1 -1
  23. package/dist/app/environments/routing.js +1 -1
  24. package/dist/app/environments/static-values.js +1 -1
  25. package/dist/app/environments/supplied.js +1 -1
  26. package/dist/app/inspect/index.js +1 -1
  27. package/dist/app/lint/native.js +1 -1
  28. package/dist/app/lint/options.js +1 -1
  29. package/dist/app/lint/shared.js +1 -1
  30. package/dist/app/run/engine-orchestration.js +1 -1
  31. package/dist/app/run/environment.js +1 -1
  32. package/dist/app/run/live-run.js +1 -1
  33. package/dist/app/run/orchestrator.js +1 -1
  34. package/dist/app/run/planning.js +1 -1
  35. package/dist/app/run/request.js +1 -1
  36. package/dist/app/run/results.js +1 -1
  37. package/dist/app/run/run-finalization.js +1 -1
  38. package/dist/app/run/run-finisher.js +1 -0
  39. package/dist/app/run/setup/runtime-node-setup-protocol.js +1 -1
  40. package/dist/app/run/setup/runtime-node-setup-worker.js +1 -1
  41. package/dist/app/run/setup/runtime-node-setup.js +2 -2
  42. package/dist/app/run/task-batches.js +1 -1
  43. package/dist/app/run/worker-loop.js +1 -1
  44. package/dist/app/status/model.js +1 -1
  45. package/dist/app/status/runs.js +1 -1
  46. package/dist/app/stop/index.js +1 -1
  47. package/dist/app/tracker/index.js +1 -1
  48. package/dist/app/tracker/token.js +2 -2
  49. package/dist/app/typecheck/index.js +4 -4
  50. package/dist/cli/command-catalog.js +1 -1
  51. package/dist/cli/command-flags.js +1 -1
  52. package/dist/cli/commands/artifacts.js +2 -2
  53. package/dist/cli/commands/bug/file.js +3 -2
  54. package/dist/cli/commands/discover.js +1 -1
  55. package/dist/cli/commands/guard/unsafe-entrypoint.js +1 -1
  56. package/dist/cli/commands/run.js +2 -2
  57. package/dist/cli/commands/typecheck.js +1 -1
  58. package/dist/cli/commands/workspace.js +1 -1
  59. package/dist/cli/framework/help.js +5 -5
  60. package/dist/cli/framework/parser.js +1 -1
  61. package/dist/cli/json-errors.js +1 -1
  62. package/dist/cli/renderers/bug/text.js +1 -1
  63. package/dist/cli/renderers/run/formatting.js +1 -1
  64. package/dist/config/authoring/database-fixture-discovery.js +1 -0
  65. package/dist/config/authoring/index.js +1 -1
  66. package/dist/config/loader/config-loader.js +1 -1
  67. package/dist/config/loader/decode-discovery.js +1 -1
  68. package/dist/config/loader/discover-tests.js +1 -1
  69. package/dist/config/loader/discovery-result.js +1 -0
  70. package/dist/config/loader/discovery.js +2 -2
  71. package/dist/config/loader/file-metadata.js +1 -1
  72. package/dist/config/loader/groups.js +2 -1
  73. package/dist/config/loader/index.js +1 -1
  74. package/dist/config/loader/load.js +1 -1
  75. package/dist/config/loader/managed-configs.js +1 -1
  76. package/dist/config/loader/owned-discovery.js +1 -1
  77. package/dist/config/loader/runtime-isolation.js +1 -1
  78. package/dist/config/loader/skip-config.js +1 -1
  79. package/dist/config/loader/test-selection.js +1 -1
  80. package/dist/database/admin.js +5 -2
  81. package/dist/database/call-caps.js +1 -0
  82. package/dist/database/cleanup.js +1 -1
  83. package/dist/database/configured-steps.js +1 -1
  84. package/dist/database/constants.js +1 -1
  85. package/dist/database/docker-endpoint.js +1 -0
  86. package/dist/database/docker-metadata.js +2 -2
  87. package/dist/database/docker.js +5 -5
  88. package/dist/database/index.js +2 -1
  89. package/dist/database/lifecycle-cleanup.js +1 -1
  90. package/dist/database/local-postgres.js +4 -1
  91. package/dist/database/naming.js +1 -1
  92. package/dist/database/operation-deadline.js +1 -0
  93. package/dist/database/postgres/connection-check.js +1 -1
  94. package/dist/database/postgres/index.js +2 -2
  95. package/dist/database/postgres/owned-exec.js +27 -0
  96. package/dist/database/postgres/protocol.js +2 -2
  97. package/dist/database/provider.js +1 -0
  98. package/dist/database/roles.js +2 -2
  99. package/dist/database/schema-operations.js +5 -1
  100. package/dist/database/schema-source.js +3 -3
  101. package/dist/database/steps.js +1 -1
  102. package/dist/database/template-steps.js +4 -4
  103. package/dist/domain/identifiers.js +1 -1
  104. package/dist/domain/retired-test-types.js +1 -1
  105. package/dist/domain/suite-selection.js +1 -1
  106. package/dist/domain/test-types.js +1 -1
  107. package/dist/engines/adapters/evidence-files.js +1 -1
  108. package/dist/engines/adapters/execution.js +1 -1
  109. package/dist/engines/catalog.js +1 -1
  110. package/dist/engines/definitions.js +1 -1
  111. package/dist/engines/environment/index.js +1 -1
  112. package/dist/engines/installations/node-vitest.js +1 -1
  113. package/dist/engines/installations/playwright.js +1 -1
  114. package/dist/engines/managed-module-resolution.js +6 -2
  115. package/dist/engines/node-vitest/adapter/adapter.js +1 -1
  116. package/dist/engines/node-vitest/adapter/driver.js +2 -2
  117. package/dist/engines/node-vitest/adapter/evidence.js +2 -2
  118. package/dist/engines/node-vitest/adapter/workspace.js +8 -24
  119. package/dist/engines/node-vitest/inventory/index.js +2 -2
  120. package/dist/engines/node-vitest/inventory/list-files.js +3 -3
  121. package/dist/engines/node-vitest/runtime/data/index.js +1 -1
  122. package/dist/engines/node-vitest/runtime/data/report.js +1 -1
  123. package/dist/engines/node-vitest/runtime/data/scenario.js +2 -2
  124. package/dist/engines/node-vitest/runtime/runtime-input.js +1 -1
  125. package/dist/engines/playwright/adapter/adapter.js +1 -1
  126. package/dist/engines/playwright/adapter/browser-network-evidence.js +4 -0
  127. package/dist/engines/playwright/adapter/config.js +16 -6
  128. package/dist/engines/playwright/adapter/evidence.js +2 -1
  129. package/dist/engines/playwright/adapter/failure-details.js +2 -2
  130. package/dist/engines/playwright/adapter/reporter.js +1 -1
  131. package/dist/engines/playwright/runtime/browser-network.js +1 -0
  132. package/dist/engines/playwright/runtime/browser-traffic.js +1 -0
  133. package/dist/engines/playwright/runtime/chromium-environment.js +1 -0
  134. package/dist/engines/playwright/runtime/emulation.js +1 -0
  135. package/dist/engines/playwright/runtime/index.js +1 -1
  136. package/dist/engines/playwright/runtime/managed-test.js +1 -0
  137. package/dist/engines/playwright/runtime/runtime-input.js +2 -2
  138. package/dist/environment/lifecycle-controller.js +1 -1
  139. package/dist/environment/lifecycle.js +1 -1
  140. package/dist/environment/maintenance.js +1 -1
  141. package/dist/environment/run-environment.js +1 -1
  142. package/dist/environment/run-stopping.js +1 -1
  143. package/dist/environment/runtime-manager.js +1 -1
  144. package/dist/evidence/artifact-config.js +1 -1
  145. package/dist/evidence/artifacts.js +5 -2
  146. package/dist/evidence/bundles.js +2 -1
  147. package/dist/evidence/compressed-artifacts.js +1 -0
  148. package/dist/evidence/environment-record.js +1 -1
  149. package/dist/evidence/index.js +1 -1
  150. package/dist/evidence/live-run-summary.js +1 -0
  151. package/dist/evidence/local-paths.js +1 -0
  152. package/dist/evidence/maintenance.js +1 -0
  153. package/dist/evidence/operation-timings.js +1 -1
  154. package/dist/evidence/read-run.js +1 -1
  155. package/dist/evidence/redaction.js +4 -4
  156. package/dist/evidence/run-evidence.js +1 -0
  157. package/dist/evidence/setup-progress.js +1 -1
  158. package/dist/evidence/storage-inventory.js +1 -1
  159. package/dist/evidence/storage.js +2 -2
  160. package/dist/kernel/known-keys.js +1 -0
  161. package/dist/kernel/redaction-targets.js +1 -1
  162. package/dist/kernel/redaction.js +7 -4
  163. package/dist/process/engine-environment-keys.js +1 -0
  164. package/dist/process/environment-overlay.js +1 -1
  165. package/dist/process/environment.js +1 -1
  166. package/dist/process/execa-bundle.json +3 -3
  167. package/dist/process/files/evidence-writes.js +1 -0
  168. package/dist/process/files/product-dir.js +1 -1
  169. package/dist/process/host-process.js +1 -1
  170. package/dist/process/index.js +1 -1
  171. package/dist/process/lifecycle-fence.js +1 -1
  172. package/dist/process/lock-progress.js +2 -0
  173. package/dist/process/locks.js +4 -4
  174. package/dist/process/process-identity.js +1 -1
  175. package/dist/process/process.js +35 -35
  176. package/dist/process/recovery.js +1 -1
  177. package/dist/process/runs/manifest.js +1 -1
  178. package/dist/process/runs/ownership.js +1 -1
  179. package/dist/process/runs/stop.js +1 -1
  180. package/dist/process/shutdown.js +1 -1
  181. package/dist/process/windows-native.js +1 -1
  182. package/dist/repo/fingerprint-policy.js +1 -1
  183. package/dist/repo/history.js +7 -4
  184. package/dist/repo/managed-git.js +1 -1
  185. package/dist/repo/state.js +4 -4
  186. package/dist/steps/configured-step-testkit-modules.js +1 -1
  187. package/dist/steps/environment.js +1 -1
  188. package/dist/toolchains/index.js +2 -2
  189. package/dist/toolchains/node-provisioning.js +2 -2
  190. package/dist/toolchains/vitest/index.js +1 -1
  191. package/dist/tracker/gitea.js +1 -1
  192. package/dist/tracker/github.js +1 -1
  193. package/dist/tracker/http.js +1 -1
  194. package/dist/tracker/ownership.js +1 -0
  195. package/dist/tracker/redaction.js +1 -1
  196. package/dist/tracker/responses.js +1 -1
  197. package/dist/types/config.d.ts +7 -86
  198. package/dist/types/environment.d.ts +6 -0
  199. package/dist/types/node-vitest-data.d.ts +2 -218
  200. package/dist/types/playwright.d.ts +21 -0
  201. package/dist/types/workspace.d.ts +30 -14
  202. package/docs/bug-records.md +19 -18
  203. package/docs/engines.md +115 -21
  204. package/docs/release-notes.md +2846 -0
  205. package/docs/test-types.md +19 -69
  206. package/package.json +15 -6
  207. package/testkit-config-schema.md +101 -170
  208. package/dist/app/lint/rules/data.js +0 -1
  209. package/dist/config/authoring/runtime.js +0 -1
  210. package/dist/config/loader/task-policy.js +0 -1
  211. package/dist/engines/node-vitest/runtime/contract/index.js +0 -1
  212. package/dist/engines/node-vitest/runtime/data/http.js +0 -1
  213. package/dist/engines/node-vitest/runtime/data/runner.js +0 -1
  214. package/dist/environment/lifecycle-artifacts.js +0 -1
  215. package/dist/types/node-vitest-contract.d.ts +0 -75
@@ -0,0 +1,2846 @@
1
+ # 0.6.1
2
+
3
+ ## Each `ui` test's browser traffic is in the run evidence
4
+
5
+ Testkit now records the requests every `ui` test's pages send
6
+ ([gfdelarue/testkit#496](https://github.com/gfdelarue/testkit/issues/496)).
7
+ For each `ui` file whose tests sent requests, the run evidence has
8
+ `browser-network.json` and `browser-network-report.md`, from each test's final
9
+ attempt. Rows group page loads and API calls by service, method, and path, with
10
+ IDs replaced by `:id`. Other requests count toward totals by type, and other
11
+ origins appear by host only. The record holds no query strings, headers, or
12
+ bodies.
13
+
14
+ The report lists leads: a write sent more than once with the same body, and a
15
+ page load or API call that failed in a test that passed. Leads never fail a
16
+ test. Recording is always on. On a page that sent 400 requests it added no time
17
+ we could measure.
18
+
19
+ `browserRequests()` from `@gfdelarue/testkit/playwright` returns the test's
20
+ requests so a test can assert on them:
21
+
22
+ ```ts
23
+ const payments = (await browserRequests()).filter(
24
+ (request) => request.service === "api" && request.method === "POST" && request.path === "/payments",
25
+ );
26
+ expect(payments).toHaveLength(1);
27
+ ```
28
+
29
+ [Check a ui test's browser traffic](../README.md#check-a-ui-tests-browser-traffic)
30
+ describes the record, its limits, and the entry fields.
31
+
32
+ ## A `ui` test can slow its browser's network and CPU
33
+
34
+ `@gfdelarue/testkit/playwright` exports `emulateNetwork(page, conditions)` and
35
+ `emulateCpu(page, slowdown)`
36
+ ([gfdelarue/testkit#495](https://github.com/gfdelarue/testkit/issues/495)).
37
+ They set Chromium's own throttling through the DevTools protocol, so a test can
38
+ fail when a change makes a page unusable on a slow connection or a slow device:
39
+
40
+ ```ts
41
+ test("the dashboard is usable on a slow connection", async ({ page }) => {
42
+ await emulateNetwork(page, "slow-4g");
43
+ await page.goto("/dashboard");
44
+ await expect(page.getByRole("heading", { name: "Orders" })).toBeVisible({ timeout: 10_000 });
45
+ });
46
+ ```
47
+
48
+ `emulateNetwork` takes `"fast-4g"`, `"slow-4g"`, or `"3g"`, with the values of
49
+ DevTools' presets of the same name, or custom
50
+ `{ latencyMs, downloadKbps, uploadKbps }`. It converts kilobits per second to
51
+ the protocol's bytes per second. `emulateCpu` takes a fixed multiplier from 1
52
+ to 100, which slows the page relative to the machine that runs it. The network
53
+ conditions apply to the page you pass in, and the CPU slowdown to the browser
54
+ process that runs it.
55
+ [Slow the browser's network and CPU](../README.md#slow-the-browsers-network-and-cpu)
56
+ lists the profile values, what each setting reaches, and how to set time limits
57
+ that don't fail on a slow CI runner.
58
+
59
+ Testkit's own Postgres client calls ran with the 30-minute timeout meant for
60
+ long user SQL, and the source schema's replay into a template with the
61
+ 60-minute step timeout. A schema-only `pg_dump` that couldn't finish used up
62
+ all of `db schema refresh`'s or `verify`'s time limit, and the error said only
63
+ that a Postgres client was cancelled
64
+ ([gfdelarue/testkit#468](https://github.com/gfdelarue/testkit/issues/468)).
65
+
66
+ Each call now has a cap:
67
+
68
+ - the server version probe before a source schema dump: 60 seconds;
69
+ - a schema-only `pg_dump`: 5 minutes;
70
+ - the source schema's replay into a template: 10 minutes.
71
+
72
+ When the probe or a dump reaches its cap, the error names the call and its
73
+ cap, such as `Dockerized Postgres client for the schema-only pg_dump (300 s
74
+ cap) exceeded its managed deadline`.
75
+
76
+ In `db schema verify` and `refresh`, a `--timeout-seconds` above 600 lengthens
77
+ these caps in proportion, along with the admin statements' `lock_timeout` and
78
+ `statement_timeout` and the role setup's 60-second timeout: a limit of 1200
79
+ seconds doubles them. Elsewhere, including `testkit run`, the caps keep their
80
+ defaults. Configured template steps keep their 60-minute timeout.
81
+
82
+ ## A schema operation that stops waiting for a lock says who holds it
83
+
84
+ When `db schema verify` or `refresh` ran out of its time limit while another
85
+ operation on the same product held a lock it needed, the error said only which
86
+ service it was preparing
87
+ ([gfdelarue/testkit#468](https://github.com/gfdelarue/testkit/issues/468)). It
88
+ now names the lock, how long it waited, the holder's process and host, and the
89
+ step the holder is on and for how long. A holder from an older Testkit, or one
90
+ that recorded no step, is reported with its progress unknown.
91
+
92
+ A lock whose release failed in Testkit's own process used to block that
93
+ process's later waits until it exited. Testkit now reports it at once.
94
+
95
+ A configured template step's toolchain installation now counts against the
96
+ operation's limit. Only the step's own run doesn't.
97
+
98
+ ## `db schema verify` and `refresh` have a time limit
99
+
100
+ `db schema verify` and `db schema refresh` had no overall bound. A Testkit
101
+ statement blocked on another session's lock waited until its own
102
+ `lock_timeout`, and any wait without a limit of its own could hang the
103
+ command. Their cleanup ran with no bound and dropped its errors
104
+ ([gfdelarue/testkit#468](https://github.com/gfdelarue/testkit/issues/468)).
105
+
106
+ Each operation now has a time limit, 600 seconds unless `--timeout-seconds`
107
+ sets it. Your configured template steps don't spend it: they keep their own
108
+ step timeout. When the limit runs out, the error lists the setup steps with
109
+ their times, the service being prepared and the wait it stopped in.
110
+
111
+ Cleanup gets its own 3-minute allowance. If it can't finish, the error says
112
+ what it left, and the command fails even when the operation's work
113
+ succeeded. `testkit destroy` removes what's left.
114
+
115
+ A process Testkit stops is awaited for at most 15 seconds after its
116
+ termination begins, or 15 seconds after a local service's shutdown grace.
117
+ After that, Testkit reports it as unsettled instead of waiting.
118
+
119
+ ## A retained local database that won't start is rebuilt once
120
+
121
+ Testkit keeps each product's local Postgres cluster in a Docker volume between
122
+ runs. If that cluster was damaged, for example by a host crash while its files
123
+ were half written, its server exited while starting, and every later run failed
124
+ the same way until `testkit destroy`.
125
+
126
+ Testkit now notices when the server exits while it waits for the database to
127
+ become ready. If the cluster was on a volume an earlier run left, and the volume
128
+ carries this product's Testkit labels, Testkit removes the container and the
129
+ volume, says so, and provisions again on a fresh volume:
130
+
131
+ ```text
132
+ [testkit] Testkit is replacing database volume <volume>, because Local database container <container> exited while starting with exit code 1: <the server's last FATAL line, redacted>. The new volume starts with an empty cluster, so its template is rebuilt.
133
+ ```
134
+
135
+ It does this at most once per provisioning. A server that exits on a volume
136
+ created in the same provisioning fails it, with its exit code and its last FATAL
137
+ or PANIC line. A server that is slow to start, or that refuses connections while
138
+ it starts, is left alone, and readiness still times out after 60 seconds.
139
+
140
+ Testkit also creates new clusters with `initdb --no-sync`, through
141
+ `POSTGRES_INITDB_ARGS`. On Docker Desktop that saves about a second of a cold
142
+ start. A cluster that a host crash catches right after `initdb` is now one of
143
+ the clusters the rebuild replaces.
144
+
145
+ ## Admin statements on local Postgres time out
146
+
147
+ A statement Testkit runs on its local Postgres container, such as
148
+ `CREATE DATABASE` or `DROP DATABASE`, waited indefinitely when another session
149
+ held a lock it needed. A `psql` client that didn't return after its statement
150
+ kept the command waiting too. Neither printed anything
151
+ ([gfdelarue/testkit#468](https://github.com/gfdelarue/testkit/issues/468)).
152
+
153
+ Each admin statement now runs with a 60-second `lock_timeout` and a 10-minute
154
+ `statement_timeout`. Testkit stops its `docker exec` 30 seconds after the
155
+ statement timeout, then ends the client and its session inside the container,
156
+ as it already did after a cancellation. The error names the statement and its
157
+ limits. When the statement ran long enough to be checked, the error also says
158
+ what it was last seen doing, such as the lock it was waiting on and the session
159
+ that held it.
160
+
161
+ ## The local database keeps its data in Testkit's volume, whatever the image's tag
162
+
163
+ Testkit chose where to mount the local PostgreSQL container's named volume from
164
+ a major version it read out of the image's tag
165
+ ([gfdelarue/testkit#481](https://github.com/gfdelarue/testkit/issues/481)). A tag
166
+ that only looked like a version, such as `team-db:33ab` on a PostgreSQL 16
167
+ image, got PostgreSQL 18's mount, `/var/lib/postgresql`. The cluster then lived
168
+ in an anonymous volume that Docker mounted at the image's
169
+ `/var/lib/postgresql/data`, so a recreated container started empty, and
170
+ removing the container deleted the data. A PostgreSQL 18 image whose tag read
171
+ below 18 got the older mount, and its container never became ready.
172
+
173
+ Testkit now reads the image's own configuration with `docker image inspect`:
174
+ its `PGDATA` and the volumes it declares. It pulls the image first when it
175
+ isn't local. The named volume takes the place of the deepest declared volume
176
+ that holds `PGDATA`, or mounts at `PGDATA` when no declared volume holds it.
177
+ Testkit stops with an error rather than guess when the image sets no `PGDATA`,
178
+ sets a relative one, or declares a volume inside it:
179
+
180
+ ```text
181
+ Testkit cannot tell where image team-db:33ab keeps its PostgreSQL data: its configuration sets no PGDATA. Use an image built from the official postgres image, or set PGDATA in it.
182
+ ```
183
+
184
+ A container that an earlier version mounted in the wrong place is recreated on
185
+ a fresh named volume, so its database starts empty once.
186
+
187
+ ## A crashed run no longer blocks the next one after Windows reuses its PID
188
+
189
+ On Windows, once a crashed run's processes had ended and Windows had given one
190
+ of their process IDs to another process, the next run, `testkit cleanup` and
191
+ `testkit run stop` failed:
192
+
193
+ ```text
194
+ Application service process creation identity no longer matches pid 6944; refusing unsafe cleanup
195
+ ```
196
+
197
+ The crashed run's record stayed until the unrelated process exited
198
+ ([gfdelarue/testkit#485](https://github.com/gfdelarue/testkit/issues/485)).
199
+ Linux and macOS could hit the same error when the new process led its own
200
+ process group.
201
+
202
+ A recorded process whose PID now belongs to a different process has ended.
203
+ Testkit leaves the new process alone and clears the crashed run's record. It
204
+ still refuses to signal a live process whose creation identity it can't read.
205
+
206
+ ## The local database is ready when it accepts a connection from the host
207
+
208
+ Testkit waited for its local Postgres container by running `pg_isready` inside
209
+ it through `docker exec`, up to about once a second. On Docker Desktop each
210
+ exec costs about 0.17 s of the WSL VM's CPU, and a cold start ran seven or
211
+ eight of them.
212
+
213
+ Testkit now waits by connecting to the container's published port on
214
+ `127.0.0.1`, the address the services use, with the startup, authentication
215
+ and `SELECT 1` of its database checks. It tries every 10 to 100 ms, gives each
216
+ attempt 5 seconds, and still gives up after 60 seconds. A port that accepts the
217
+ connection but doesn't answer as PostgreSQL, as Docker Desktop's port proxy
218
+ does before Postgres listens, and a server that rejects the startup, such as
219
+ one still starting, don't count as ready. When readiness times out, the error
220
+ gives the last check's reason. A server that refuses the configured user or
221
+ password, or has no `postgres` database, fails at once with that reason instead
222
+ of waiting out the 60 seconds.
223
+
224
+ Testkit waits on the published port only when it reaches Docker through a
225
+ local socket or named pipe, whether `DOCKER_HOST`, `DOCKER_CONTEXT` or the
226
+ configuration's current context names it. A daemon behind a `tcp://` or `ssh://`
227
+ endpoint, at any address, can publish ports in another network namespace, as
228
+ Docker-in-Docker does, so readiness there still runs `pg_isready` in the
229
+ container. Through a local socket, Testkit runs that probe as well, at most once
230
+ a second, when nothing answers on the port: the connection is refused, an
231
+ attempt gets no reply, or what answers isn't PostgreSQL. After 10 seconds it
232
+ probes on any failed attempt. When the probe finds the server ready, readiness
233
+ succeeds. A socket that forwards to a daemon in another namespace works this
234
+ way. As before, the database URL Testkit gives tests names `127.0.0.1` and the
235
+ published port, which such a daemon doesn't serve on this host.
236
+
237
+ ## Managed Chromium runs with its sandbox on
238
+
239
+ Playwright starts Chromium with `--no-sandbox` unless `chromiumSandbox` is
240
+ `true`, and Testkit didn't set it. Every UI file ran without Chromium's
241
+ sandbox, so a compromised renderer ran with the user's access to the checkout,
242
+ SSH keys and env files
243
+ ([gfdelarue/testkit#190](https://github.com/gfdelarue/testkit/issues/190)).
244
+
245
+ Testkit now sets `chromiumSandbox: true` in the Playwright config it generates
246
+ and in the managed test's launch options. It holds whether a file imports
247
+ `test` from `@playwright/test`, `playwright/test`, or
248
+ `@gfdelarue/testkit/playwright`, when a project or a file sets other
249
+ `launchOptions`, and for browsers that `globalSetup` and `globalTeardown`
250
+ launch. An explicit `chromiumSandbox: false` in `playwright.config.ts` or in a
251
+ file's `test.use` still turns it off.
252
+
253
+ On a host that can't start the sandbox, such as root in a container, Chromium
254
+ now fails to launch. Testkit reports that as an infrastructure failure and puts
255
+ the opt-out first:
256
+
257
+ ```text
258
+ Testkit starts managed Chromium with its sandbox on, and this host can't start it. To run without the sandbox here, set use.launchOptions.chromiumSandbox to false in playwright.config.ts, or in one file with test.use.
259
+ ```
260
+
261
+ [Browser launch options](engines.md#browser-launch-options) shows how to turn
262
+ it off for one host.
263
+
264
+ ## Managed Chromium no longer crashes at launch under older glibc
265
+
266
+ Chromium sometimes crashed with `SIGSEGV` in `libdbus` during
267
+ `browserType.launch` on Ubuntu 24.04, and the UI file failed
268
+ ([gfdelarue/testkit#453](https://github.com/gfdelarue/testkit/issues/453)).
269
+ At startup, Chromium sets `FC_FONTATIONS` on a worker thread while its D-Bus
270
+ thread reads the environment. glibc before 2.41 can move the environment and
271
+ free the old copy under that reader, unless the distribution backported the
272
+ fix. Ubuntu 24.04's glibc 2.39 hasn't.
273
+
274
+ Playwright files now run with `DBUS_SESSION_BUS_ADDRESS=disabled:` and
275
+ `FC_FONTATIONS=1`, the values Chromium sets itself. Chromium finds both already
276
+ set and doesn't grow the environment at startup.
277
+
278
+ - Tests, and any process or browser they start, see both variables.
279
+ - A `launchOptions.env` in `playwright.config.ts` or in a file's `test.use`
280
+ gets both added, under its own values. A browser that a test or global setup
281
+ launches directly with its own `env` gets only what that sets.
282
+ - The Playwright engine reserves both keys, so a group's environment can't set
283
+ them.
284
+
285
+ ## A capture records only requests sent to its url
286
+
287
+ An HTTP capture recorded a request to any path outside `/__testkit__/` from any
288
+ caller that found its port, whatever its `Host` header
289
+ ([gfdelarue/testkit#193](https://github.com/gfdelarue/testkit/issues/193)).
290
+ Another local user, or a web page through DNS rebinding, could add captures and
291
+ push real ones out of the list.
292
+
293
+ A capture's `url` now holds an unguessable segment, as its `requestsUrl`
294
+ already did, such as `http://127.0.0.1:41000/__testkit__/<segment>/webhook`.
295
+ The capture records a request only when its path starts with that segment, and
296
+ answers any other path with 404. A captured request's `path` leaves the segment
297
+ out, so a test that checks `path` still sees `/webhook`. The capture answers
298
+ 403 to a request whose `Host` header isn't `127.0.0.1:<port>`. Testkit keeps
299
+ the new segment out of the run's records. A capture's `path` may no longer
300
+ contain `.` or `..` segments, because a client would resolve them against the
301
+ `url` and drop the segment.
302
+
303
+ If your product builds a URL from the capture's origin, give it the whole `url`
304
+ and append to that instead. [Handles](../README.md#handles) describes the
305
+ capture.
306
+
307
+ ## Tokens, temporary files, results and Git calls are harder to reach
308
+
309
+ These changes keep another bug, or another user on a shared host, from reaching
310
+ what Testkit handles
311
+ ([gfdelarue/testkit#196](https://github.com/gfdelarue/testkit/issues/196)).
312
+
313
+ - On Linux and macOS, `.testkit` is now private to its owner. Loading the
314
+ config and starting a run create it with mode `0700`, and any command that
315
+ resolves the product removes group and other access from an existing one the
316
+ user owns. Sealed objects are `0400` instead of `0444`. Another user, such as
317
+ a host user reading results that a container wrote as root, now needs the
318
+ owner to share them. On Windows, `.testkit` keeps the access its parent
319
+ folder grants.
320
+ - Services and setup steps no longer inherit `GITHUB_TOKEN`, `GH_TOKEN` or
321
+ `GITEA_TOKEN` from Testkit's environment. Testkit reads them itself for
322
+ known-failure checks and bug filing. A service that needs one declares it in
323
+ its `env`.
324
+ - Lint's `git ls-files` and bug filing's `git diff` no longer run the command
325
+ that the repository's `core.fsmonitor` names. Run fingerprinting already
326
+ turned it off.
327
+ - The `testkit-engine-*` directories that Testkit creates in the system's
328
+ temporary directory, which hold each browser profile, now record their
329
+ owner. The first engine invocation of a Testkit process removes those whose
330
+ owner has exited, such as after a crash. On Linux it only removes those from
331
+ its own PID namespace, so a container that shares the temporary directory
332
+ with its host leaves the host's alone, and the reverse. Directories from
333
+ earlier versions have no record and stay.
334
+ - On Linux, a recorded process identity now includes the boot ID, so after a
335
+ reboot a run or lock record no longer matches an unrelated process that got
336
+ the same PID and start time. A record from an earlier Testkit still matches on
337
+ start time alone, as before.
338
+
339
+ ## A Playwright companion decides a Next app's support files
340
+
341
+ In a Next application, `testkit typecheck` added `next-env.d.ts`,
342
+ `tests/playwright-fixtures`, and the generated types under `.next` and
343
+ `.next-testkit` to the Playwright program even when a `tsconfig.playwright.json`
344
+ left them out or excluded them
345
+ ([gfdelarue/testkit#108](https://github.com/gfdelarue/testkit/issues/108)). A
346
+ fixture the companion excluded, or another service's stale build output, could
347
+ then fail the check. With a companion, the program now holds the discovered
348
+ suites, the Playwright config, the files the companion selects, and what they
349
+ import. List the Next files the suites need in the companion. Without a
350
+ companion, the Next defaults stay.
351
+
352
+ ## `typecheck` checks test files outside the tsconfig's `rootDir`
353
+
354
+ The generated programs took `rootDir` and `outDir` from the tsconfig they
355
+ extend. A test file outside that `rootDir`, such as `tests/sum.test.ts` beside a
356
+ tsconfig whose `rootDir` is `src`, failed `typecheck` with `TS6059` before any of
357
+ its types were checked. TypeScript 6 also defaults `rootDir` to the directory of
358
+ the tsconfig it compiles, which is Testkit's generated one, so under TypeScript 6
359
+ every test file failed when the tsconfig set `outDir` and no `rootDir`
360
+ ([gfdelarue/testkit#407](https://github.com/gfdelarue/testkit/issues/407)).
361
+
362
+ The generated configs now clear the inherited output paths, `outDir`,
363
+ `outFile`, `declarationDir`, `sourceRoot`, and `mapRoot`, and the test programs'
364
+ configs clear `rootDir` too. TypeScript holds every file to an explicit
365
+ `rootDir`, and when an output path is set without one, TypeScript 6 uses the
366
+ tsconfig's own directory. With both cleared, it holds test files to neither. When the tsconfig sets `incremental` or `composite`,
367
+ Testkit now writes its build info beside the generated config in
368
+ `.testkit/_typecheck`, instead of under the repository's `outDir` or into the
369
+ repository's own build info file.
370
+
371
+ ## A run stops its retained database when it releases it
372
+
373
+ A run that provisioned the local Postgres container now stops it as part of releasing the run's resources, after it drops the run's task databases. It stops the container by the name in `.testkit/_infra/local-postgres` and lists no Docker resources to do it. A container that is already stopped counts as released.
374
+
375
+ If that container no longer exists, the release fails, and so does the run, with `Could not stop the retained database container <name>: it no longer exists`. Before, a successful run stopped the container later, during finish-time maintenance.
376
+
377
+ ## A run leaves Docker resources to `testkit cleanup`
378
+
379
+ A run's start and its post-run maintenance no longer list Docker resources, remove unreferenced containers and volumes, or stop the retained database. Provisioning and release still create, replace, and remove what the run needs, and release stops the retained database by name. `testkit cleanup --resources` removes the product's unreferenced containers and volumes, and stops an idle retained database, as before.
380
+
381
+ A run's start still removes stale manifests of runs whose processes have ended, and post-run maintenance still removes other runs' stale manifests.
382
+
383
+ `.testkit/results/maintenance.json` no longer has the `removed`, `skipped` and `stopped` lists, since maintenance no longer removes or stops Docker resources.
384
+
385
+ ## `run.json`'s storage inventory covers only `.testkit/results`
386
+
387
+ `run.json`'s `storage.testkit` inventory now covers only `.testkit/results`, and its `root` is `.testkit/results`. A run no longer walks the database and graph caches to build it. When Testkit can't read part of `.testkit/results`, `run.json` records `"storage": { "testkit": { "status": "failed", "error": "…" } }` and the run keeps its result.
388
+
389
+ ## Testkit's own clients of the owned database run inside its container
390
+
391
+ Testkit used to start a client container, from `postgres:16` or
392
+ `TESTKIT_POSTGRES_CLIENT_IMAGE`, to declare roles and take schema snapshots in
393
+ the database it owns. Before a snapshot, it also asked the server for its
394
+ version to choose a `pg_dump` image. Both now run inside the database container
395
+ with `docker exec`, using the database image's own `psql` and `pg_dump`, as
396
+ Testkit's admin queries already did. They no longer start a container
397
+ ([gfdelarue/testkit#139](https://github.com/gfdelarue/testkit/issues/139)).
398
+
399
+ This changes two things:
400
+
401
+ - `TESTKIT_POSTGRES_CLIENT_IMAGE` no longer applies to those operations. Your
402
+ SQL, in `sql-file` steps and `database-steps` helpers, still runs in a client
403
+ container from that image, and so do clients of external databases.
404
+ - The `image` that `database.postgres()` uses must include `psql` and
405
+ `pg_dump`, and its `pg_dump` must be at least as new as its server.
406
+ Otherwise the operation fails with a message that names the image:
407
+
408
+ ```text
409
+ pg_dump in the database image <image> is older than its server: pg_dump: error: aborting because of server version mismatch
410
+ ```
411
+
412
+ ## Redaction recognises more secret names, API key headers, and bearer tokens
413
+
414
+ Sealing, filed bugs, and error messages kept a secret under a key such as
415
+ `STRIPE_KEY`, `GITHUB_PAT`, `SMTP_PASS`, or `SENTRY_DSN`, and the value of an
416
+ `X-Api-Key` header or an `apiKey` member. They kept a bearer token outside an
417
+ `Authorization` header, and a header that a trace stores as a
418
+ `{"name": …, "value": …}` pair
419
+ ([gfdelarue/testkit#188](https://github.com/gfdelarue/testkit/issues/188)).
420
+
421
+ - A key whose last word is `PASS`, `PASSWD`, `PAT`, `DSN`, `CREDENTIAL`, or
422
+ `CREDENTIALS` now names a secret, as does a key whose last two words are
423
+ `API` and `KEY`. An upper-case key whose last word is `KEY` names a secret
424
+ unless the word before it is `PUBLIC` or `PUBLISHABLE`. A run's environment
425
+ values under these keys are redacted as other secrets are, so a value under
426
+ a key such as `CACHE_KEY` is now redacted too. `PATH`, `BYPASS`, and
427
+ `cacheKey` still name no secret.
428
+ - `Api-Key` and `X-Api-Key` header values are redacted like `Authorization`.
429
+ - The token after `Bearer` is redacted wherever it appears when it is at least
430
+ eight characters long and holds a digit, so prose such as
431
+ `Bearer authentication` stays.
432
+ - In a `name` and `value` pair, the value is redacted when the name names a
433
+ secret, such as `Authorization` or `Cookie`. When `testkit run` stores a
434
+ result, every value in a `cookies` array is redacted too.
435
+
436
+ `docs/bug-records.md` lists the rules.
437
+
438
+ ## Redaction recognises more secret forms and fewer ordinary words
439
+
440
+ Sealing, filed bugs, and error messages kept these secrets
441
+ ([gfdelarue/testkit#215](https://github.com/gfdelarue/testkit/issues/215)):
442
+
443
+ - A `privateKey`, `signingKey`, or `encryptionKey` value.
444
+ - A single-quoted member under any secret name other than `password`, as in
445
+ `{'token': '…'}`.
446
+ - The strings in a JSON array or object under a secret name, as in
447
+ `{"tokens":["…"]}`. `session` and `auth` settings stay, and an object with a
448
+ `name` and a `value` member loses only its `value` and members under a secret
449
+ name.
450
+ - A value after `password:`. A quoted value is redacted anywhere, and an
451
+ unquoted one when it is the one word of a line that it starts. A single word
452
+ of letters stays, so `password: 'Required'` and `password: required!` stay.
453
+ - A URL password that holds a `/`, a URL password in a directory name, such as
454
+ `postgres:/app:…@db`, and a token of at least 20 characters with a digit that
455
+ stands alone as the user of an `http` or `https` URL, as in
456
+ `https://ghp_…@github.com`. A resolved URL's password or token is redacted
457
+ wherever it appears, too.
458
+ - A secret right after another one with no space between them, as in
459
+ `PASSWORD=…"Cookie: sid=…`.
460
+ - A URL password that output cut short. The first line of a task's output, as
461
+ an artifact summary or a failure message, is redacted before Testkit cuts it.
462
+
463
+ They redacted these values that are not secrets:
464
+
465
+ - A value under `author`, `authority`, or `GIT_AUTHOR_NAME`, and under
466
+ `sessionCount`, `SESSION_TIMEOUT`, and other keys where `SESSION` comes before
467
+ a count or a time. A run's environment no longer redacts every occurrence of
468
+ `GIT_AUTHOR_NAME`'s value either.
469
+ - A path assigned to a key that ends in `_URL`, such as
470
+ `AUTH_URL=/oauth/authorize`. A `token=` parameter in the path is still
471
+ redacted. A JSON member such as `{"AUTH_URL":"/oauth/authorize"}` is still
472
+ redacted whole.
473
+ - The text after a quoted header value that ends in an escaped backslash.
474
+
475
+ `testkit bug record` now rejects a record whose reproduce command holds text
476
+ that Testkit redacts, as `testkit bug file` already did. Before, a Vitest
477
+ project name such as `TOKEN=…` passed `bug record` and stopped `bug file`.
478
+
479
+ ## Sealing redacts inside traces and other compressed files
480
+
481
+ Sealing didn't look inside `.zip` and `.gz` files. A Playwright `trace.zip`
482
+ recorded with `use.trace` kept form values, `Authorization` and `Cookie`
483
+ headers, and session cookies in the sealed run, compressed, while the run's
484
+ own copies of the same output were redacted
485
+ ([gfdelarue/testkit#187](https://github.com/gfdelarue/testkit/issues/187)).
486
+
487
+ Sealing now reads `.zip` and `.gz` files, and files that start like one. It
488
+ redacts each text entry with the same rules as a text file and writes the
489
+ archive again, so the trace is kept and opens in the trace viewer with
490
+ `<redacted>` in place of the secrets. A text entry is valid UTF-8 without binary
491
+ control characters, or UTF-16 with a byte order mark. Entries it doesn't change
492
+ keep their bytes. A ZIP with nothing to redact and no comment or extra fields
493
+ stays byte for byte, and a gzip file is always written again without the name
494
+ and comment in its header.
495
+
496
+ - The bundle's redaction counts include the redacted archives: each counts as
497
+ a redacted file, its text entries count as scanned files, and its
498
+ replacements count too.
499
+ - Sealing refuses an archive it can't redact: one it can't read, such as an
500
+ encrypted, ZIP64, damaged, or larger than 512 MiB archive, one whose binary
501
+ entry holds a value the run resolved, in UTF-8 or UTF-16, or one that holds
502
+ another archive.
503
+ - Sealing removes each file it refuses, an archive or a binary file that holds
504
+ a resolved value, from the run's results. It fails the run with the reasons,
505
+ then seals the rest and records them. `testkit stop`, `testkit cleanup` and
506
+ the next run do the same when they seal a run that a killed process left.
507
+ Before, a refused run left its results in place, and those commands sealed
508
+ them without the values the run had resolved, the refused file included.
509
+ - Screenshots and video, in a trace or on their own, aren't redacted.
510
+
511
+ ## Sealed bundles leave out the product and home directories
512
+
513
+ Sealing removed the product directory and host from `run.json`, but the rest of
514
+ the bundle kept them
515
+ ([gfdelarue/testkit#189](https://github.com/gfdelarue/testkit/issues/189)). A
516
+ failing Playwright file's stack in `run.json`, the native report's `sourceFile`
517
+ and attachment paths, and a `trace.zip` held the absolute product path. A trace
518
+ also held the home directory, in the path of the Chromium that Testkit manages.
519
+ On a developer machine those paths name the user.
520
+
521
+ Sealing now rewrites local paths in the run record, in every text file of the
522
+ bundle, and in every text entry of a compressed file such as `trace.zip`. A
523
+ path under the product directory becomes relative to it, and a path under the
524
+ home directory starts with `~`. When sealing changes a file, by this rewrite or
525
+ by redaction, the file's size and SHA-256 in `run.json` now describe the sealed
526
+ file.
527
+
528
+ ## An exit code set after a plain file's tests fails the run
529
+
530
+ When a reporter or global teardown set Vitest's exit code to 1 after a file's
531
+ tests failed, Testkit read the run as those failed tests alone
532
+ ([gfdelarue/testkit#215](https://github.com/gfdelarue/testkit/issues/215)). A
533
+ run whose failed tests all had issues passed the known-failure gate under
534
+ `"allow"`, though something else had also failed it.
535
+
536
+ Testkit now notes Vitest's own verdict when the tests end, and fails the run
537
+ with a run error when a reporter or global teardown sets the exit code
538
+ afterwards:
539
+
540
+ ```text
541
+ FAIL run error
542
+ Vitest failed the run after its tests finished: exit code 1 while Vitest closed
543
+ ```
544
+
545
+ When the tests passed, the run error says "after its tests passed", as it does
546
+ for a coverage threshold, and each file keeps its own outcome. Before, such an
547
+ exit failed every file as an unexplained infrastructure failure. A coverage
548
+ threshold that falls short beside a failed test still counts as part of that
549
+ failure, because a failed test leaves code unrun.
550
+
551
+ ## The known-failure gate stops missing three failures
552
+
553
+ Three failures could pass the known-failure gate under `"allow"`, or never be
554
+ known at all
555
+ ([gfdelarue/testkit#215](https://github.com/gfdelarue/testkit/issues/215)):
556
+
557
+ - A plain Vitest file whose data-flow evidence was malformed lost that error
558
+ when one of its tests also failed. The file now fails with the evidence
559
+ error beside its failed tests, which an issue for the tests doesn't cover.
560
+ - A Playwright spec that failed in another file, such as a dependency or
561
+ teardown project, now always counts as a failure that names no test, even
562
+ when the run reports nothing else about it.
563
+ - On macOS and Linux, a test file whose name holds a backslash could not be
564
+ filed, and its failures were always new. A bug record now keeps the
565
+ backslash, and treats it as a separator only on Windows.
566
+
567
+ Testkit reads the file a Playwright spec was loaded from out of its file
568
+ suite's title, resolved against `rootDir`. It now checks, as the run begins,
569
+ that each title names a file. If a Playwright release changes the title so that
570
+ it names no file, the file fails with an error that says so, instead of
571
+ reporting the spec under a path where no file exists.
572
+
573
+ ## Four Playwright failures stop passing as known
574
+
575
+ Four Playwright failures could pass the known-failure gate under `"allow"`, or
576
+ never be known at all
577
+ ([gfdelarue/testkit#215](https://github.com/gfdelarue/testkit/issues/215)):
578
+
579
+ - A reporter in the Playwright config that ended the process with exit code 1
580
+ read as Playwright's own exit for failed specs, so an issue for those specs
581
+ covered the run. Testkit's reporter runs last and marks the run once every
582
+ reporter's hooks have returned. Without that mark, exit code 1 is a failure
583
+ that names no test. A `PLAYWRIGHT_SPECS_FAILED` failure carries
584
+ `reportersFinished` in its details. A run stored by an earlier release has no
585
+ such mark, so `testkit ci report` reads its exit code 1 the same way. A
586
+ Playwright config that sets `PW_TEST_REPORTER` now fails, because Playwright
587
+ runs that reporter after Testkit's.
588
+ - A failed spec in a serial group, or a failed `beforeAll` hook, could stop
589
+ later specs of the selected file. Playwright reports those specs as skipped,
590
+ so an issue for the failed spec let the run pass. Each such spec is now a
591
+ detail of kind `playwright-not-run`, which names no test. A spec that the
592
+ file skips with `test.skip` or `test.fixme` is not one. A spec that a run
593
+ stops at `maxFailures` gets the same detail, though that run already failed
594
+ the gate on the error Playwright reports when it stops.
595
+ - Under `failOnFlakyTests`, a spec that passed only on a retry failed the run
596
+ as a failure that names no test, so no issue could cover it. It is now a
597
+ `playwright-spec` detail with its title, and an issue for that title covers
598
+ it.
599
+ - A file whose every spec came from one helper with another extension, such
600
+ as a `.ts` file and a `.mjs` helper, ran no specs and failed with
601
+ `PLAYWRIGHT_NO_TESTS`. Playwright moves the file suite's location to that
602
+ helper, and Testkit selected the file by location. Testkit now selects it
603
+ with `--grep` on its file suite's title, the file's path from `rootDir`, so
604
+ its specs run.
605
+
606
+ ## Tracker tokens go only where they belong
607
+
608
+ Five gaps in how Testkit talks to GitHub and Gitea are closed
609
+ ([gfdelarue/testkit#215](https://github.com/gfdelarue/testkit/issues/215)):
610
+
611
+ - `GITEA_TOKEN` names no host, so Testkit sent it to whichever Gitea host the
612
+ checkout's remote named. Outside a Gitea Actions job, Testkit now sends it
613
+ only to the host that `GITEA_SERVER_URL` names. Set
614
+ `GITEA_SERVER_URL=https://gitea.example.com` beside `GITEA_TOKEN`, or keep the
615
+ token in a Git credential helper, which is bound to its host already.
616
+ - Testkit sends a token only over HTTPS. A plain `http://` remote or repository
617
+ URL now fails with a message that says so, unless it names a loopback host.
618
+ - `testkit ci report` updated the first pull request comment that started with
619
+ its marker, whoever wrote it. It now updates only a comment written by its
620
+ token's user. It takes that user from the commit status it has just set, or
621
+ asks the tracker when the status names no one. When neither names the user,
622
+ it posts a new comment each time rather than edit one.
623
+ - When the tracker created an issue without its labels, `testkit bug file`
624
+ failed and left the issue open. It now closes that issue first, and says
625
+ whether it could.
626
+ - `testkit ci report` now leaves out of the summary and the commit status any CI
627
+ run or issue link that holds credentials or a secret.
628
+
629
+ ## `bug file` asks the remote for the commit, and closes a duplicate it filed
630
+
631
+ Two gaps in `testkit bug file` are closed, along with a Gitea problem that the
632
+ second turned up ([gfdelarue/testkit#215](https://github.com/gfdelarue/testkit/issues/215)):
633
+
634
+ - It took the commit as pushed when a local remote-tracking branch held it,
635
+ even after the remote rewound or deleted that branch. It now lists the
636
+ remote's branches with `git ls-remote` and checks their tips, fetching a tip
637
+ the checkout lacks. A renamed branch still counts. When a remote doesn't
638
+ answer, or a shallow clone's history stops before Git can tell, Testkit
639
+ stops and says why, unless another remote has the commit.
640
+ - Two checkouts that filed the same test at once could each create an issue,
641
+ because neither saw the other's in the open issues it read first. After it
642
+ creates the issue, Testkit now records it in the draft and reads the open
643
+ issues again. GitHub can list a new issue late, so it waits up to 7.5 seconds
644
+ for its own to appear, and on GitHub it then reads again two seconds later.
645
+ It says so when its issue never appears. When an older open issue covers the
646
+ same test, Testkit closes its own, comments with the older one's number, and
647
+ stops as it would have if it had seen that issue first.
648
+ - On Gitea, two checkouts creating the same label at once made two labels of
649
+ that name, and Gitea's label filter then returned no issue. Testkit now
650
+ deletes a label it created when an older one has the same name, and lists
651
+ the open bugs without the filter when the repository already has two labels
652
+ of one name.
653
+
654
+ ## A failed database drop no longer hides the error before it
655
+
656
+ When a template step failed, Testkit dropped the half-built template database
657
+ before it reported the step's error. If the drop failed too, the run reported
658
+ only the drop's error. A run interrupted during a template `sql-file` step,
659
+ after its Docker connection had closed, reported only `Cannot connect to the
660
+ Docker daemon`, and nothing about the step or the Postgres client container it
661
+ left running
662
+ ([gfdelarue/testkit#400](https://github.com/gfdelarue/testkit/issues/400)). A
663
+ `resetBefore` file whose database clone failed had the same problem.
664
+
665
+ The run now reports both errors, and names the database Testkit couldn't drop:
666
+
667
+ ```text
668
+ Dockerized Postgres client was cancelled
669
+ Postgres client container cleanup failed: Cannot connect to the Docker daemon at tcp://127.0.0.1:36893. Is the docker daemon running?
670
+ Dropping the template database tk_tpl_api_f14201a8f06684a4 also failed: Cannot connect to the Docker daemon at tcp://127.0.0.1:36893. Is the docker daemon running?
671
+ ```
672
+
673
+ ## A failure block prints each message once, without color codes
674
+
675
+ A failed Playwright assertion printed its message twice in the run's failure
676
+ block, once for the test and once for Playwright's overall failure. Both copies
677
+ kept Playwright's ANSI color codes, even with `NO_COLOR` set, and both were cut
678
+ off mid-word
679
+ ([gfdelarue/testkit#220](https://github.com/gfdelarue/testkit/issues/220)).
680
+
681
+ The block now strips the engine's color codes, prints each distinct message
682
+ once, and says how much of a long message it left out. It cuts at a space when
683
+ one falls within 40 characters of the limit, and otherwise at the limit, without
684
+ splitting a character:
685
+
686
+ ```text
687
+ [1/1] FAIL app ui __testkit__/failure.ui.testkit.ts 3s
688
+ Error: expect(locator).toHaveText(expected) failed Locator: locator('#status') Expected:
689
+ "expected marker" Received: "actual" Timeout: 1000ms Call log: - Expect "toHaveText" with
690
+ timeout 1000ms - waiting for locator('#status') 5 × locator … (69 more characters)
691
+ ```
692
+
693
+ The run's failure details keep the full message. The block's second and later
694
+ messages are now redacted like the first.
695
+
696
+ ## Long command output no longer warns about listeners
697
+
698
+ Commands write their text output one `log` call at a time, usually a line each,
699
+ and each call added an `error` listener to stdout until its write finished. More
700
+ than ten calls in a row, as in a longer `discover`, `status`, `artifacts`, or
701
+ `workspace` listing, printed `MaxListenersExceededWarning` on stderr
702
+ ([gfdelarue/testkit#221](https://github.com/gfdelarue/testkit/issues/221)).
703
+ Testkit now adds one listener.
704
+
705
+ ## Testkit's runtime keys take the `@gfdelarue/testkit` prefix
706
+
707
+ Three keys that Testkit's modules use to share state inside an engine process
708
+ had kept the old `elench.testkit.*` names
709
+ ([gfdelarue/testkit#99](https://github.com/gfdelarue/testkit/issues/99)). They
710
+ are now `@gfdelarue/testkit/node-vitest-runtime-input/v1`,
711
+ `@gfdelarue/testkit/node-vitest-runtime-task/v1` and
712
+ `@gfdelarue/testkit/engine-process-environment-overlay/v1`.
713
+
714
+ There is nothing to do when you upgrade. Testkit writes the keys afresh for
715
+ every invocation, stored runs don't hold them, and the CLI refuses to run when
716
+ the product's installed Testkit is a different version, so both sides always
717
+ come from one release. Code that read these undocumented globals directly needs
718
+ the new names. The `com.elench.testkit.*` Docker labels are unchanged.
719
+
720
+ ## `bug file --batch` files several bugs in one call
721
+
722
+ `testkit bug file --batch` files the drafts it is given, or every unfiled draft
723
+ when it is given none
724
+ ([gfdelarue/testkit#254](https://github.com/gfdelarue/testkit/issues/254)). It
725
+ checks the checkout once, runs the drafts' tests together, and reports filed,
726
+ already filed, rejected, or not filed for each draft. It exits non-zero when
727
+ any draft is rejected or not filed, and a rerun files only what is left. A
728
+ campaign's pull request summary now suggests it for the bugs that are recorded
729
+ and not filed.
730
+
731
+ # 0.6.0
732
+
733
+ 0.6.0 removes the `contract` and `data` test types and legacy service
734
+ ownership, and refuses config keys that Testkit doesn't read. To upgrade from
735
+ 0.5.0, work through these steps in order:
736
+
737
+ 1. If `testkit.config.ts` has `services` and no `groups`, add the groups that
738
+ the load error lists before you rename any typed file. The error lists a
739
+ group only for a service that still owns a typed file. Then, with or without
740
+ that step, move each service's `skip`, `requirements`, and
741
+ `runtimeIsolation` to its group. Move its `discovery.roots` and
742
+ `discovery.exclude` to the group's `roots` and `exclude`, as paths from the
743
+ product directory, and delete its `discovery`. See
744
+ [Remove legacy service ownership](#remove-legacy-service-ownership).
745
+ 2. Replace `--service` with `--group` on `run`, `discover`, `artifacts`, and
746
+ `workspace`, even if the config already has `groups`. See the same section.
747
+ 3. Turn each `*.contract.testkit.ts` file into a plain Vitest file. See
748
+ [Remove the contract test type](#remove-the-contract-test-type).
749
+ 4. Turn each `*.data.testkit.ts` file into a plain Vitest file that calls
750
+ `dataFlow(context)`, and delete `profiles` from `testkit.config.ts`. See
751
+ [Remove the data test type](#remove-the-data-test-type).
752
+ 5. Delete each config key that the load error names, or move its value to the
753
+ key that applies. See
754
+ [A config with an unknown key fails to load](#a-config-with-an-unknown-key-fails-to-load).
755
+ 6. Remove leftover `int`, `e2e`, `dal`, and `load` selectors. Delete or migrate
756
+ leftover files of those types, and files that declare `engine: "k6"`. 0.5.0
757
+ stopped running them, and 0.6.0 no longer flags them. See
758
+ [Testkit no longer recognises the types and engine that 0.5.0 removed](#testkit-no-longer-recognises-the-types-and-engine-that-050-removed).
759
+
760
+ A config that sets `profiles` or an unknown top-level key fails to load before
761
+ Testkit checks its groups. If that happens at step 1, delete those keys as
762
+ steps 4 and 5 say, and load the config again.
763
+
764
+ Upgrading from 0.4.x? Work through [0.4.5](#045) and then [0.5.0](#050) first.
765
+ 0.4.5 makes `local.port` a config error. 0.5.0 adds `groups`, which step 1
766
+ needs, and removes the `int`, `e2e`, `dal`, and `load` types, whose files 0.6.0
767
+ ignores without a warning.
768
+
769
+ ## Breaking changes
770
+
771
+ ### Remove legacy service ownership
772
+
773
+ Only `groups` assign test files to services
774
+ ([gfdelarue/testkit#241](https://github.com/gfdelarue/testkit/issues/241)).
775
+ Before 0.6.0, a config with `services` and no `groups` key gave each service the
776
+ typed files under its `local.cwd` and `discovery.roots`, and printed a notice.
777
+ That legacy ownership is gone. A config that relied on it fails to load and
778
+ lists the groups to add:
779
+
780
+ ```text
781
+ testkit.config.ts has services and no groups. Testkit 0.6.0 removed legacy service ownership, so services no longer own the test files under their directories. Add these groups to keep each typed test file with the service that owned it:
782
+
783
+ groups: {
784
+ api: { roots: ["services/api"], uses: ["api"] },
785
+ web: { roots: ["apps/web"], uses: ["web"] },
786
+ },
787
+
788
+ A group also owns the plain Vitest files under its roots.
789
+ Migrate with "Remove legacy service ownership" in node_modules/@gfdelarue/testkit/docs/release-notes.md.
790
+ ```
791
+
792
+ - The error lists each service that owned a typed file, or excluded one under
793
+ its directories, with that service's `roots`, `uses`, and `exclude` as 0.5.0
794
+ resolved them.
795
+ - `run`, `typecheck`, and `discover --strict` stop with it. `discover` without
796
+ `--strict` reports it as a `config_invalid` diagnostic. `status`,
797
+ `run status`, `destroy`, `cleanup`, and `db schema` still run, so you can
798
+ stop 0.5.0 services before you migrate.
799
+ - A config whose services held no typed file, such as one with only a database
800
+ service, loads, and every file belongs to `root`.
801
+
802
+ Migrate a two-service config:
803
+
804
+ ```ts
805
+ // Before
806
+ export default defineConfig({
807
+ services: {
808
+ api: app.node({ cwd: "services/api", start: "node dist/server.js", runtimeIsolation: "resetBefore" }),
809
+ web: app.next({ cwd: "apps/web", dependsOn: ["api"] }),
810
+ },
811
+ });
812
+ ```
813
+
814
+ ```ts
815
+ // After
816
+ export default defineConfig({
817
+ services: {
818
+ api: app.node({ cwd: "services/api", start: "node dist/server.js" }),
819
+ web: app.next({ cwd: "apps/web", dependsOn: ["api"] }),
820
+ },
821
+ groups: {
822
+ api: { roots: ["services/api"], uses: ["api"], runtimeIsolation: "resetBefore" },
823
+ web: { roots: ["apps/web"], uses: ["web"] },
824
+ },
825
+ });
826
+ ```
827
+
828
+ - Move a service's `skip`, `requirements`, and `runtimeIsolation` to its group.
829
+ `ServiceConfig` no longer declares them, and beside `groups` Testkit rejects
830
+ them on a service with a message that names the group field.
831
+ - Move a service's `discovery.roots` and `discovery.exclude` to its group's
832
+ `roots` and `exclude`, and delete the service's `discovery`. A service's
833
+ `discovery` paths are relative to its `local.cwd`, and a group's are relative
834
+ to the product directory. The root `__testkit__` of a service whose
835
+ `local.cwd` is `services/api` becomes `services/api/__testkit__`. The error
836
+ above prints them that way. Beside `groups`, Testkit rejects a service's
837
+ `discovery` with a message that names the group's fields:
838
+
839
+ ```text
840
+ services.api.discovery applies to tests, and services own no tests when groups is set; move its roots and exclude to groups.api.roots and groups.api.exclude, as paths from the product directory
841
+ ```
842
+
843
+ The `discovery` that `database.fixture()` adds by default is still accepted.
844
+
845
+ - A group's roots also claim plain Vitest files, which a service never did, and
846
+ a run starts the group's services for them. A service with no
847
+ `discovery.roots` and `local.cwd: "."` gets the root `.`, so its group owns
848
+ every plain file. To keep plain files in `root`, narrow the roots to the
849
+ directories that hold typed files.
850
+
851
+ `run`, `discover`, `artifacts`, and `workspace` no longer take `--service` as an
852
+ alias for `--group`:
853
+
854
+ ```text
855
+ Testkit 0.6.0 removed --service from run, discover, artifacts, and workspace; use --group api
856
+ ```
857
+
858
+ `status`, `destroy`, `cleanup`, and `db schema` keep `--service`, which names a
859
+ service there.
860
+
861
+ A discovered group's `source` is `groups` or `implicit`, and `TestGroupSource`
862
+ no longer includes `services`. `inspectWorkspace()` from
863
+ `@gfdelarue/testkit/workspace` no longer takes a `notice` callback.
864
+
865
+ ### Remove the contract test type
866
+
867
+ This release removes the `contract` test type. A contract file becomes a plain
868
+ Vitest file, which Testkit runs as it is. These are removed:
869
+
870
+ - the `*.contract.testkit.ts` suffix and the `node-vitest-contract` profile
871
+ - the `@gfdelarue/testkit/node-vitest/contract` export, with
872
+ `contractScenario`, `readContractContext`, `createHttpClient`, and their
873
+ types
874
+
875
+ Wherever you can name `contract`, Testkit fails with a message that names the
876
+ release and this section:
877
+
878
+ ```text
879
+ __testkit__/health.contract.testkit.ts: Testkit 0.6.0 removed the contract test type. Migrate with "Remove the contract test type" in node_modules/@gfdelarue/testkit/docs/release-notes.md.
880
+ ```
881
+
882
+ - `discover`, `run`, and `typecheck` stop on a leftover
883
+ `*.contract.testkit.ts` file unless a group excludes it, and `lint` and
884
+ `doctor` stop on any leftover file. A run stops even when the repository's
885
+ plain files would pass.
886
+ - `testkit run contract`, `--type contract`, `--suite contract:<name>`, and
887
+ `contract:` selectors in `skip`, `requirements`, and `runtimeIsolation` stop
888
+ with the same message.
889
+ - `bug file` refuses an unfiled draft for a contract file. For a draft it
890
+ already filed, it still returns the issue.
891
+ - `status`, `run status`, `destroy`, `cleanup`, and `db schema` still run while
892
+ leftover files remain.
893
+ - Stored runs whose suites are `contract` on `node-vitest-contract` stay
894
+ readable through `testkit artifacts`, `ci report`, and the workspace API.
895
+
896
+ Migrate each contract file:
897
+
898
+ | In a contract file | In the plain Vitest file |
899
+ | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------- |
900
+ | `import { contractScenario, expect } from "@gfdelarue/testkit/node-vitest/contract"` | `import { expect, test } from "vitest"` |
901
+ | `contractScenario(name, run)` | `test(name, run)` |
902
+ | `contractScenario(name, { timeoutMs }, run)` | `test(name, run, timeoutMs)` |
903
+ | `export const testkit = defineFile({ engine: "node-vitest" })` | Delete it and its import |
904
+ | `name.contract.testkit.ts` | A name your Vitest config includes, such as `name.test.ts` |
905
+ | `name.contract.testkit.setup.mjs` | `name.test.setup.mjs` |
906
+ | `context.baseUrl` and `context.http()` | `endpoint(name)` from `@gfdelarue/testkit/environment`, and `fetch` |
907
+ | `context.env` | `process.env` |
908
+ | `context.setupData` | `setupData()` from `@gfdelarue/testkit/environment` |
909
+ | `context.fixture("a.json")` | `readFileSync(new URL("./__fixtures__/a.json", import.meta.url))` |
910
+ | `context.serviceName`, `task`, `engineId`, and `profileId` | Nothing. Vitest owns test identity. |
911
+
912
+ A plain file has no `defineFile` metadata. Move its `skip`, `locks`, and
913
+ `runtimeIsolation` to the group's `skip.files`, `requirements.files`, and
914
+ `runtimeIsolation.files` in `testkit.config.ts`. A plain file gets services only
915
+ from a declared group. See
916
+ [Remove legacy service ownership](#remove-legacy-service-ownership).
917
+
918
+ A typed file's tests and hooks could run until the file's timeout. Under the
919
+ repository's Vitest, a plain file gets Vitest's own limits: 5 seconds for a test
920
+ and 10 seconds for a hook, unless the Vitest config sets `testTimeout` and
921
+ `hookTimeout`. Give a slow test its timeout, as in `test(name, run, timeoutMs)`,
922
+ or set the limits in the config. Under `--vitest-runner managed`, Testkit sets
923
+ them to the remaining time of the Vitest invocation, which starts with the sum
924
+ of its files' timeouts.
925
+
926
+ Each typed file was its own task, so with the default `taskConcurrency` of 1,
927
+ typed files ran one at a time. Plain files that share a Vitest invocation run
928
+ with Vitest's file parallelism, which `taskConcurrency` doesn't limit. If
929
+ migrated tests can't run side by side, give each its own rows, or set
930
+ `fileParallelism: false` or `maxWorkers` in the Vitest config. Under
931
+ `--vitest-runner managed`, they run one at a time.
932
+
933
+ Renaming a test file changes the path that bug issues name. A failure is known
934
+ only when an open issue's `test.file` equals the failing file's path. After you
935
+ rename a file, edit `test.file` in the facts block of each open issue that
936
+ names it, so that `bugs.knownFailures: "allow"` still treats its failures as
937
+ known. Issues and bug drafts that record `testType: "contract"` stay readable.
938
+
939
+ ### Remove the data test type
940
+
941
+ This release removes the `data` test type. Its recording and budgets are now a
942
+ library that a plain Vitest test calls: `dataFlow(context)` from
943
+ `@gfdelarue/testkit/node-vitest/data`
944
+ ([gfdelarue/testkit#240](https://github.com/gfdelarue/testkit/issues/240)).
945
+ These are removed:
946
+
947
+ - the `*.data.testkit.ts` suffix, the `node-vitest-data` profile, and the data
948
+ runner
949
+ - `dataScenario`, `createHttpClient`, `readDataContext`, and the `expect`
950
+ re-export from `@gfdelarue/testkit/node-vitest/data`, with the types of the
951
+ scenario context, the HTTP client, its actors, and fixtures
952
+ - the `dataSuiteContract` lint rule
953
+ - HTTP profiles: `profiles.http` in `testkit.config.ts`, and
954
+ `registerRepoConfig`, `getRepoConfig`, `clearRepoConfig`, `RepoConfig`, the
955
+ `HttpProfile*` types, `HttpRequestParams`, `RuntimeCookie`, `RuntimeEnv`, and
956
+ `RuntimeHeaders` from `@gfdelarue/testkit/config`
957
+ - `*.data.testkit.ts` from the `testkitExcludes()` globs of
958
+ `@gfdelarue/testkit/tooling/vitest`
959
+ - the `description` field of a scenario in `data-flow.json`, which only
960
+ `dataScenario` options could set
961
+
962
+ Wherever you can name `data`, Testkit fails with a message that names the
963
+ release and this section:
964
+
965
+ ```text
966
+ __testkit__/flow.data.testkit.ts: Testkit 0.6.0 removed the data test type. Migrate with "Remove the data test type" in node_modules/@gfdelarue/testkit/docs/release-notes.md.
967
+ ```
968
+
969
+ Testkit stops in the same places as for `contract`, above. A config that sets
970
+ `profiles` fails to load, and `testkit lint` rejects a config that sets
971
+ `lint.rules.dataSuiteContract`, with messages that name this section. Stored
972
+ runs whose suites are `data` on `node-vitest-data`, and issues whose facts
973
+ block says `"testType": "data"`, stay readable.
974
+
975
+ Migrate each data file:
976
+
977
+ | In a data file | In the plain Vitest file |
978
+ | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
979
+ | `import { dataScenario, expect } from "@gfdelarue/testkit/node-vitest/data"` | `import { dataFlow } from "@gfdelarue/testkit/node-vitest/data"` and `import { expect, test } from "vitest"` |
980
+ | `dataScenario(name, (flow) => { ... })` | `test(name, (context) => { const flow = dataFlow(context); ... })` |
981
+ | `dataScenario(name, { timeoutMs }, run)` | `test(name, run, timeoutMs)` |
982
+ | A helper that takes `DataScenarioContext` | A helper that takes `DataFlow` from `@gfdelarue/testkit/node-vitest/data` |
983
+ | `export const testkit = defineFile({ engine: "node-vitest" })` | Delete it and its import |
984
+ | `name.data.testkit.ts` | A name your Vitest config includes, such as `name.test.ts` |
985
+ | `flow.http()`, `flow.http({ profile })`, `createHttpClient`, and HTTP actor sessions | `fetch`, or your own client, against `endpoint(name)` from `@gfdelarue/testkit/environment`, then `flow.recordHttpRequest(...)` |
986
+ | `profiles.http` in `testkit.config.ts` | Log in from test code, and delete `profiles` |
987
+ | `flow.fixture(name)` | `readFileSync(...)`, and `flow.count(...)` and `flow.bytes(...)` if the test should record the file |
988
+ | `flow.baseUrl`, `flow.env`, and `readDataContext()` | `endpoint(name)` and `database(name)` from `@gfdelarue/testkit/environment`, and `process.env` |
989
+ | `flow.setupData` | `setupData()` from `@gfdelarue/testkit/environment` |
990
+ | `flow.serviceName` and `flow.task` | Nothing. Vitest owns test identity. |
991
+ | `lint.rules.dataSuiteContract` | Delete it |
992
+ | `--type data` or `--suite data:<name>` | `--type plain`, with `--group` or `--file` to narrow it |
993
+
994
+ A Shell test, before and after, abridged to one budget. Both versions leave out
995
+ the `record`, `bytes`, and `count` calls before it and the `message` option of
996
+ `equals`:
997
+
998
+ ```ts
999
+ // Before: src/browser/__testkit__/browser-profile-history-work.data.testkit.ts
1000
+ import { defineFile } from "@gfdelarue/testkit";
1001
+ import { dataScenario } from "@gfdelarue/testkit/node-vitest/data";
1002
+
1003
+ export const testkit = defineFile({ engine: "node-vitest" });
1004
+
1005
+ dataScenario("shell BrowserTab profile and history path stays model-owned", async (flow) => {
1006
+ const measurement = await measureBrowserProfileHistoryWork();
1007
+ flow.expectMetric("shell.browser-profile-history.profileModelFiles").equals(2);
1008
+ });
1009
+ ```
1010
+
1011
+ ```ts
1012
+ // After: src/browser/__testkit__/browser-profile-history-work.test.ts
1013
+ import { dataFlow } from "@gfdelarue/testkit/node-vitest/data";
1014
+ import { test } from "vitest";
1015
+
1016
+ test("shell BrowserTab profile and history path stays model-owned", async (context) => {
1017
+ const flow = dataFlow(context);
1018
+ const measurement = await measureBrowserProfileHistoryWork();
1019
+ flow.expectMetric("shell.browser-profile-history.profileModelFiles").equals(2);
1020
+ });
1021
+ ```
1022
+
1023
+ The migrated file keeps its metrics, budgets, `data-flow.json`,
1024
+ `data-flow-report.md`, and `data-flow-budget` failure details. As for a contract
1025
+ file, move its `defineFile` metadata to the group, give slow tests a timeout,
1026
+ check that its tests can run side by side, and after renaming it, edit
1027
+ `test.file` in the open issues that name it. See
1028
+ [Remove the contract test type](#remove-the-contract-test-type).
1029
+
1030
+ Testkit no longer imports `testkit.config.ts` into any test process. A data test
1031
+ that relied on a side effect of that import, such as a value the config set in
1032
+ `process.env`, has to do that work itself.
1033
+
1034
+ ### A config with an unknown key fails to load
1035
+
1036
+ `testkit.config.ts` used to load with keys that Testkit doesn't read, so a
1037
+ misspelled key such as `executoin` had no effect and no warning
1038
+ ([gfdelarue/testkit#218](https://github.com/gfdelarue/testkit/issues/218)).
1039
+ Testkit now refuses such a config when it loads it, whether the file calls
1040
+ `defineConfig` or exports a plain object. The error names each unknown key, and
1041
+ the closest known key when one is close:
1042
+
1043
+ ```text
1044
+ testkit.config.ts has unknown keys "executoin" (mispelled "execution"?), "madeUpKey"; it accepts areas, artifacts, bugs, discovery, execution, fingerprints, groups, lint, redact, services, toolchains, ui
1045
+ ```
1046
+
1047
+ The same check now runs inside these blocks:
1048
+
1049
+ - `discovery`, at the top level
1050
+ - each `groups` entry
1051
+ - `skip`, `requirements`, and `runtimeIsolation`, and the rules in their
1052
+ `files` and `suites` lists, in a group
1053
+ - `artifacts`, `artifacts.retention`, and `artifacts.publication`
1054
+ - `fingerprints`
1055
+ - each `toolchains` entry, and a service's inline `runtime.toolchain`
1056
+
1057
+ `execution`, `lint`, `bugs`, and each `areas` entry already rejected unknown
1058
+ keys. A config that sets `profiles` still fails with the message in
1059
+ [Remove the data test type](#remove-the-data-test-type).
1060
+
1061
+ A key that Testkit never read never took effect. Delete it, or move its value
1062
+ to the key that applies. For example, `global: { fileTimeoutSeconds: 120 }`
1063
+ never set a timeout; `execution: { fileTimeoutSeconds: 120 }` does.
1064
+
1065
+ ### Testkit no longer recognises the types and engine that 0.5.0 removed
1066
+
1067
+ 0.5.0 named the release and its migration when it found an `int`, `e2e`, `dal`,
1068
+ or `load` file or selector, or a file that declared `engine: "k6"`. One minor
1069
+ release later, those checks are gone:
1070
+
1071
+ - Discovery ignores a leftover `*.int.testkit.ts`, `*.e2e.testkit.ts`,
1072
+ `*.dal.testkit.ts`, or `*.load.testkit.ts` file, and `discover`, `run`,
1073
+ `lint`, `typecheck`, and `doctor` no longer reject it. If the repository's
1074
+ Vitest config selects such a file, Testkit runs it as a plain Vitest file,
1075
+ and `bug record` records a bug for it without a test type.
1076
+ - `--type int`, `--suite int:<name>`, and `int:` selectors in `skip`,
1077
+ `requirements`, and `runtimeIsolation` fail as unknown types, such as
1078
+ `Unknown type "int"`. Unless the Vitest config selects such a file,
1079
+ `run --file` with it fails because it matches no suite.
1080
+ - Stored runs and bug drafts that name these types stay readable, and
1081
+ `bug file` still returns the issue of a filed draft.
1082
+ - A typed file that declares `engine: "k6"` fails as any unknown engine does:
1083
+ `unsupported engine "k6"; expected one of: node-vitest, playwright`.
1084
+
1085
+ ## New behaviour
1086
+
1087
+ ### A plain Vitest file with per-file needs runs in its own invocation
1088
+
1089
+ In 0.5.0, a plain Vitest file with a setup module, locks that differ from the
1090
+ other plain files', or isolation other than `shared` stopped the run when other
1091
+ plain files ran with it. Such a file now runs in its own Vitest invocation, with
1092
+ its own environment, beside the invocation that runs the rest. Its setup module
1093
+ runs before its invocation starts, its locks hold for that invocation, and
1094
+ `resetBefore` gives it its own database clone. Root global setup, teardown, and
1095
+ the repository's reporters run once for each invocation.
1096
+
1097
+ ### A bare `testkit run` runs plain files from groups that use different services
1098
+
1099
+ In 0.5.0, a run stopped before it started anything when its plain Vitest files
1100
+ belonged to groups that gave them different services or values
1101
+ ([gfdelarue/testkit#378](https://github.com/gfdelarue/testkit/issues/378)). A
1102
+ repository with unit tests in `root` and plain files in a service-backed group
1103
+ couldn't pass the one-command CI gate the README recommends. Now plain files
1104
+ run in one Vitest invocation per environment their groups give them, each
1105
+ against its own group's services. Groups that give their files the same values
1106
+ share one, such as `root` and the other groups that use no services. A group
1107
+ that uses a service Testkit starts gets an invocation of its own. Root global setup, teardown, and the
1108
+ repository's reporters run once for each invocation, so a run calls them once
1109
+ more for each extra invocation. Invocations run one after another unless
1110
+ `--task-concurrency` or `execution.taskConcurrency` is above 1.
1111
+
1112
+ ### `--type plain` selects only plain Vitest files
1113
+
1114
+ With `ui` the only test type, a run could select UI files with `--type ui` but
1115
+ had no way to select plain Vitest test files without them
1116
+ ([gfdelarue/testkit#380](https://github.com/gfdelarue/testkit/issues/380)).
1117
+ `run` and `discover` now accept `--type plain`, which selects plain files only.
1118
+ It narrows `--group`, `--suite`, and `--file` like any other type, so
1119
+ `--group api --type plain` runs the `api` group's plain files and none of its UI
1120
+ files. `testkit run plain` does the same as `testkit run --type plain`.
1121
+
1122
+ `--type ui,plain` selects the same files as `--type all`. A run whose `--type`
1123
+ names `plain` reports it in `run.types`, as it does `ui`.
1124
+
1125
+ ### `typecheck` reads Vitest setup files and takes `--group`
1126
+
1127
+ `testkit typecheck` built the Node/Vitest program from the plain test files and
1128
+ what they import. A file in the Vitest config's `setupFiles` is named only as a
1129
+ string, so the program never reached it. Matchers that a setup file declares,
1130
+ such as `toBeInTheDocument` from `@testing-library/jest-dom/vitest`, failed with
1131
+ `TS2339` although `tsc` and `vitest run` accepted them
1132
+ ([gfdelarue/testkit#379](https://github.com/gfdelarue/testkit/issues/379)).
1133
+
1134
+ Each plain file's program now includes the TypeScript setup files of its Vitest
1135
+ project, and `typecheck` checks them like the test files. A setup file with a
1136
+ type error now fails `typecheck`. Files whose projects load different setup
1137
+ files go to separate programs, and the label of a program with setup files names
1138
+ them, as in `node-vitest with tests/setup.ts`. A JavaScript setup file, or a package named
1139
+ directly in `setupFiles`, still adds no types, so import the package from a
1140
+ TypeScript setup file.
1141
+
1142
+ `typecheck --group <name>` checks the config and the files that group owns. A
1143
+ group's setup files are those of the Vitest projects its files belong to.
1144
+
1145
+ ### `setupData()` reads a file's setup data
1146
+
1147
+ `setupData()` from `@gfdelarue/testkit/environment` returns the data the file's
1148
+ setup module returned, or the `setupData` of a supplied environment file. It
1149
+ works in plain Vitest files and Playwright files. A file
1150
+ without setup data gets `{}`. Outside a Testkit run, `setupData()` throws.
1151
+
1152
+ ### A run's result no longer depends on post-run maintenance
1153
+
1154
+ After a run seals its evidence, Testkit tidies the product's local state:
1155
+
1156
+ - it removes other runs' stale manifests;
1157
+ - it removes the product's unreferenced database containers and volumes;
1158
+ - it stops the idle retained database.
1159
+
1160
+ That maintenance now runs after the seal and has its own result:
1161
+
1162
+ - A failure no longer changes the run's status, exit code, or sealed evidence. `testkit run` prints `Notice: Post-run maintenance failed: <reason>. The run's result is unchanged.` on stderr.
1163
+ - `.testkit/results/maintenance.json` records the latest run's maintenance result. `testkit status` shows a warning while that result is failed.
1164
+ - When Testkit can't read part of the `.testkit` tree to inventory its storage, `run.json` records `"storage": { "testkit": { "status": "failed", "error": "…" } }` and the run keeps its result.
1165
+
1166
+ A run that fails to release its own resources, such as dropping its task databases, still fails. A group whose tests passed now reports that failure as an infrastructure failure, `INFRA`, instead of a test failure, `FAIL`. So does a failure while sealing the run.
1167
+
1168
+ ### Retired-type errors name the installed release notes
1169
+
1170
+ The package now ships `docs/release-notes.md`. Each error that points at a
1171
+ migration names the section and the installed file, such as
1172
+ `Migrate with "Remove the contract test type" in node_modules/@gfdelarue/testkit/docs/release-notes.md`.
1173
+ It used to name `docs/release-notes.md#…`, which the package didn't ship.
1174
+
1175
+ ## Fixes
1176
+
1177
+ ### `campaign finish` lists plain Vitest tests
1178
+
1179
+ The summary's "Testkit tests added or changed" list held only `*.testkit.ts`
1180
+ files, so it left out the plain Vitest test files that Testkit has run since
1181
+ 0.5.0. It now also lists changed files named like a Vitest test, such as
1182
+ `refund.test.ts` or `refund.spec.js`.
1183
+
1184
+ ### `typecheck` uses the repository's `lib` for plain files
1185
+
1186
+ The Node/Vitest program set `lib` to ES2022 in place of the `lib` of the
1187
+ repository's tsconfig. A plain file that runs in a jsdom or happy-dom
1188
+ environment failed `typecheck` with `Cannot find name 'document'`, or, when a
1189
+ package brought in the DOM library alone, with `TS2488` on a `NodeList` spread,
1190
+ although the repository's own `tsc` accepted it
1191
+ ([gfdelarue/testkit#398](https://github.com/gfdelarue/testkit/issues/398)).
1192
+
1193
+ A Node/Vitest program now uses the `lib` of the tsconfig it extends, and
1194
+ ES2022 only when there is no tsconfig. A test that uses a library its tsconfig
1195
+ leaves out now fails `typecheck` as it fails `tsc`.
1196
+
1197
+ ### A UI file's launch options keep managed Chromium
1198
+
1199
+ A UI file that set `launchOptions` with `test.use` lost the `executablePath`
1200
+ of the Chromium that Testkit manages, because the file's value replaced the
1201
+ config's. Playwright then started its headless shell where one was installed,
1202
+ and failed to launch where none was. The headless shell never restores a page
1203
+ from the back/forward cache, so a file couldn't test that
1204
+ ([gfdelarue/testkit#373](https://github.com/gfdelarue/testkit/issues/373)).
1205
+
1206
+ Testkit now sets the managed `executablePath` after the file's launch options,
1207
+ whether the file imports `test` from `@playwright/test`, `playwright/test`, or
1208
+ `@gfdelarue/testkit/playwright`. The file's other options still apply.
1209
+ [Browser launch options](engines.md#browser-launch-options) shows how a file
1210
+ turns on the back/forward cache.
1211
+
1212
+ ### A group's `NODE_ENV` reaches plain files under the managed runner
1213
+
1214
+ Under `--vitest-runner managed`, a service that set a key Vitest or Vite writes
1215
+ itself failed the Node/Vitest files in the groups that use it. Such keys
1216
+ include `NODE_ENV`, `TEST`, `VITEST_MODE`, `SSR`, and a `VITE_` value that a
1217
+ `.env` file also sets. With `NODE_ENV` or `VITEST_MODE`, each file took 60
1218
+ seconds to fail, the time Vitest waits for a worker that never starts
1219
+ ([gfdelarue/testkit#384](https://github.com/gfdelarue/testkit/issues/384)).
1220
+ These keys now behave as under the repository's Vitest: tests see the group's
1221
+ `NODE_ENV` and `VITE_` values, and Vitest's own `TEST`, `VITEST`,
1222
+ `VITEST_MODE`, `FORCE_TTY`, `MODE`, `DEV`, `PROD`, and `SSR`.
1223
+
1224
+ When the Node/Vitest driver or a Vitest worker fails while it installs the
1225
+ run's input, its files now fail at once with `NODE_VITEST_START_FAILED` and the
1226
+ cause, instead of after Vitest's 60-second wait:
1227
+
1228
+ ```text
1229
+ Vitest could not start: Node/Vitest runtime input size changed after preparation
1230
+ ```
1231
+
1232
+ ### A UI file can import from a `.tsx` module that contains JSX
1233
+
1234
+ A UI file that imported anything from a `.tsx` module of the product that
1235
+ contains JSX failed before its tests ran:
1236
+
1237
+ ```text
1238
+ Error: Playwright consumers may not import managed package file outside exact public exports: playwright (resolved /…/node_modules/playwright/jsx-runtime from file:///…/src/view.tsx)
1239
+ ```
1240
+
1241
+ Playwright compiles the module's JSX to import its runtime by an absolute path,
1242
+ `<playwright>/jsx-runtime`, instead of by the specifier `playwright/jsx-runtime`
1243
+ ([gfdelarue/testkit#390](https://github.com/gfdelarue/testkit/issues/390)).
1244
+ Testkit now reads a path in a managed Playwright package that names one of the
1245
+ package's exact exports as that export, so the runtime resolves to Testkit's
1246
+ pinned Playwright.
1247
+
1248
+ ### A test's data-flow evidence comes from its final repeat
1249
+
1250
+ When a repository's `aroundEach` hook failed a later repeat of a test before it
1251
+ called `runTest`, the test's `data-flow.json` entry and `data-flow-budget`
1252
+ details came from the earlier repeat
1253
+ ([gfdelarue/testkit#359](https://github.com/gfdelarue/testkit/issues/359)). Only
1254
+ Vitest 4.1's `aroundEach` could do this. `dataFlow` now records the test's
1255
+ configured repeats, and Testkit keeps a test's evidence only from its final
1256
+ repeat.
1257
+
1258
+ ### A cancelled `sql-file` step stops
1259
+
1260
+ A `sql-file` database step ignored its caller's cancellation and its step
1261
+ deadline, so its Postgres client ran until the SQL finished or the client's own
1262
+ 30-minute timeout passed, even after the run was cancelled
1263
+ ([gfdelarue/testkit#381](https://github.com/gfdelarue/testkit/issues/381)). It
1264
+ now stops as `command` and `module` steps do, when its caller cancels it or its
1265
+ step deadline passes, and its server query ends with it. The step deadline
1266
+ defaults to 60 minutes, so a `sql-file` step that 0.5.0 stopped at 30 minutes
1267
+ can now run for up to 60.
1268
+
1269
+ ### A cancelled admin query says when its server query may still run
1270
+
1271
+ When Testkit cancelled a database admin query that runs through `docker exec`,
1272
+ or found it stalled, and then failed to end the query's server backend, the
1273
+ error named only the cancellation while the server query kept running
1274
+ ([gfdelarue/testkit#382](https://github.com/gfdelarue/testkit/issues/382)). The
1275
+ error now also names the failed termination and its reason, and Testkit doesn't
1276
+ retry that attempt.
1277
+
1278
+ # 0.5.0
1279
+
1280
+ ## Remove k6
1281
+
1282
+ This release removes the k6 engine and the `int`, `e2e`, `dal`, and `load` test
1283
+ types. Testkit runs plain Vitest files, typed Vitest `contract` and `data`
1284
+ files, and Playwright `ui` files.
1285
+
1286
+ Wherever you can name a retired type, Testkit fails with a message that names
1287
+ the release and this section:
1288
+
1289
+ ```text
1290
+ __testkit__/health.int.testkit.ts: Testkit 0.5.0 removed the int test type. Migrate with docs/release-notes.md#remove-k6.
1291
+ ```
1292
+
1293
+ - Discovery rejects a file of a retired type unless a group excludes it. A
1294
+ file that still declares `engine: "k6"` fails the same way, naming the k6
1295
+ engine.
1296
+ - `--type`, `--suite`, and the selectors in `skip`, `requirements`, and
1297
+ `runtimeIsolation` reject a retired type.
1298
+ - `lint`, `typecheck`, and `doctor` stop on a retired file with the same
1299
+ message.
1300
+ - `bug file` refuses an unfiled draft for a retired type. For a draft it
1301
+ already filed, it still returns the issue.
1302
+ - `status`, `run status`, `destroy`, `cleanup`, and `db schema` still run while
1303
+ retired files remain, so you can inspect and clean up local state before you
1304
+ migrate.
1305
+ - Stored runs and bug drafts that name a retired type stay readable.
1306
+
1307
+ These are removed with the engine:
1308
+
1309
+ - the `@gfdelarue/testkit/k6` export and the k6 binary packages
1310
+ - the k6 HTTP and DAL helpers and the k6 execution options
1311
+ - the synchronous response types `RuntimeHttpClient`, `RuntimeResponse*`,
1312
+ `RuntimeHttpTrace*`, and `RuntimeHeaderValue`, from the
1313
+ `@gfdelarue/testkit/config` and `@gfdelarue/testkit/node-vitest/data`
1314
+ declarations
1315
+ - the `dalParallelSafety` lint rule. Delete it from `lint.rules`, because
1316
+ Testkit rejects an unknown rule.
1317
+
1318
+ To migrate, move HTTP, database, and service-flow checks to plain Vitest files.
1319
+ Read named endpoints and databases from `@gfdelarue/testkit/environment`, and
1320
+ use `fetch` or your application's database client. `contract` and `data` files
1321
+ keep their APIs, HTTP profiles still configure the `data` HTTP clients, and
1322
+ `ui` files still run on Playwright. Testkit has no replacement for load tests.
1323
+
1324
+ ## Plain Vitest test files run
1325
+
1326
+ Testkit runs a repository's ordinary Vitest test files, such as
1327
+ `src/sum.test.ts` and `src/sum.spec.ts`, as they are
1328
+ ([gfdelarue/testkit#237](https://github.com/gfdelarue/testkit/issues/237)):
1329
+
1330
+ - A plain file needs no Testkit suffix, `__testkit__` directory, `defineFile`,
1331
+ or wrapper, and keeps its native `vitest` imports.
1332
+ - Testkit finds plain files the way the repository's Vitest config selects
1333
+ them, as the next section describes.
1334
+ - `discovery.exclude` and a declared group's `exclude` narrow the list.
1335
+ Testkit's built-in exclusions, `discovery.roots`, and a legacy service's
1336
+ rules apply only to typed files, so a legacy service's `exclude` doesn't drop
1337
+ a plain file.
1338
+ - A plain file belongs to the implicit `root` group unless a declared group's
1339
+ roots claim it, and it gets the values of the services its group uses.
1340
+ Testkit starts no service for a plain file in `root`.
1341
+ - Plain files have no test type. They run under the general `node-vitest`
1342
+ profile, and a bug filed from one records no `testType`. A `--type` that
1343
+ names test types and typed suite selectors leave them out; `--type all`
1344
+ includes them.
1345
+ - The plain files in one directory form a suite named after that directory,
1346
+ relative to the group's root, and a file directly in a group's root names
1347
+ its own suite. In `discover --json` plain files have no `testType`, and their
1348
+ IDs have no type segment: `root|node-vitest|node-vitest|src/sum.test.ts`.
1349
+ `summary.byTestType` counts typed files only. `testkit discover` lists them
1350
+ under `Vitest`.
1351
+ - `typecheck` checks plain TypeScript test files and skips plain JavaScript
1352
+ ones.
1353
+ - `unitTestBoundaries` stays off by default and only reports. Its messages no
1354
+ longer tell a unit test to move to a Testkit suite.
1355
+
1356
+ A bare `testkit run` in a repository with ordinary Vitest files now runs them
1357
+ too. To run typed files only, name their types:
1358
+ `--type data,contract,ui`.
1359
+
1360
+ ## Plain Vitest test files run with the repository's Vitest
1361
+
1362
+ Plain files run with the repository's own Vitest installation and config
1363
+ ([gfdelarue/testkit#238](https://github.com/gfdelarue/testkit/issues/238)):
1364
+
1365
+ - Testkit loads the root `vitest.config.*` or `vite.config.*` as `vitest run`
1366
+ does from the product directory, and lists the files its projects select
1367
+ without running global setup. Nested configs and package scripts have no
1368
+ effect. The notice that Testkit ignored a root Vitest config is gone.
1369
+ - `discover`, `run`, `typecheck`, `bug record`, and `bug file` take
1370
+ `--vitest-config <path>`, `--vitest-project <name>`, and
1371
+ `--vitest-runner repo|managed`. `managed` keeps the previous behaviour:
1372
+ Testkit's own Vitest with its default rules and no repository config.
1373
+ - Testkit supports Vitest 4.x. Another major, a broken installation, a root
1374
+ Vitest config that fails to load, or a Vitest that the product declares but
1375
+ has not installed fails before any test runs, with a message that names the
1376
+ path. Most of these messages also suggest `--vitest-runner managed`. A run
1377
+ that selects only typed files, with `--type` or with typed `--file` paths,
1378
+ `bug record` or `bug file` for a typed test, and the commands that select no
1379
+ tests (`status`, `run status`, `doctor`, `destroy`, `cleanup`, and
1380
+ `db schema`) don't load the repository's Vitest, so they still work. A
1381
+ product that neither installs nor declares Vitest has no plain files.
1382
+ - All plain files in a run go to one Vitest invocation. Root global setup,
1383
+ teardown, and the repository's reporters run as often as they do under
1384
+ `vitest run`, and Testkit's reporter runs beside them. The invocation's
1385
+ deadline is the sum of its files' timeouts, and Vitest's own test and hook
1386
+ timeouts apply inside it. A file that two projects select runs in both, and
1387
+ its evidence names each run's project. Files from different groups run
1388
+ together when their groups give them the same services, values, and locks.
1389
+ Different services or locks, isolation other than `shared`, or a setup
1390
+ module stop the run before Testkit provisions anything; different values
1391
+ stop it before Vitest starts. When `testkit stop` cuts the invocation short,
1392
+ every file that hasn't finished, started or not, is cancelled.
1393
+ - Testkit sets only `allowOnly: false`, `includeTaskLocation: true`,
1394
+ `passWithNoTests: false`, and `update: "none"` over the repository's config,
1395
+ so a plain file's failure `location` has a line and column. A project that
1396
+ sets `includeTaskLocation: false` itself keeps that setting.
1397
+ - Plain files see the host environment Testkit started with, from config
1398
+ loading to the test, including `FORCE_COLOR`, `NO_COLOR`, and the shell's
1399
+ `NODE_OPTIONS`, to which Testkit adds its own `--import`. Unlike
1400
+ `vitest run`, ambient `BASE_URL` and `DATABASE_URL` never reach them; those
1401
+ come only from the group's environment. Testkit loads no env files for them.
1402
+ - Node/Vitest processes no longer receive `TESTKIT_INTERNAL_FILE_DEADLINE_MS`
1403
+ or `TESTKIT_INTERNAL_FILE_TIMEOUT_SECONDS`, which nothing in them read.
1404
+ - Testkit's own output honours `NO_COLOR` in a terminal and treats
1405
+ `FORCE_COLOR=0` or `FORCE_COLOR=false` as no colour, as Node does.
1406
+ - Node/Vitest reserves the exact keys it sets itself instead of the whole
1407
+ `TESTKIT_` and `VITEST_` prefixes, so a policy key such as
1408
+ `TESTKIT_REQUIRE_COMPLETE_SUITE` reaches the run from either environment
1409
+ source.
1410
+ - `bug record` saves the Vitest selectors it was given in the local draft, and
1411
+ asks for `--vitest-project` when two projects run the test's file.
1412
+ `bug file` reruns with them, and the issue's reproduce command carries them.
1413
+ With `TESTKIT_REQUIRE_COMPLETE_SUITE=1`, `bug file` reruns every file the
1414
+ selection lists before it reads the recorded failure.
1415
+ - Typed node-vitest files keep Testkit's own Vitest and its authoring entry
1416
+ points. Testkit no longer checks which packages their imports resolve to.
1417
+ - Typed node-vitest files now resolve `paths` from their nearest
1418
+ `tsconfig.json`, through Vite, instead of from the product root's
1419
+ `tsconfig.json` or `tsconfig.build.json`. A repository whose `paths` are only
1420
+ in `tsconfig.build.json` must move them into a `tsconfig.json`.
1421
+ - Engine evidence names only the Vitest version in `directPackages`, and its
1422
+ `installationKind` is `repository-package` when the repository's Vitest ran.
1423
+
1424
+ ## A run that stops on an error names it
1425
+
1426
+ `testkit run` names an error that stops the whole run rather than one test
1427
+ file, such as the planning refusal for plain Vitest files that cannot share
1428
+ one invocation (#352). Before, the default output showed only
1429
+ `Runtime errors: N`, and the message needed `--output-mode debug` or `--json`.
1430
+
1431
+ - The text output prints `FAIL run error` and the message when the error
1432
+ happens.
1433
+ - `--output-mode events` writes a `run.error` event with `message`, once for
1434
+ each error, not once for each group.
1435
+ - The debug summary labels each group's error `error:` instead of
1436
+ `worker error:`.
1437
+
1438
+ ## A failed coverage threshold fails the run, not each file
1439
+
1440
+ When the repository's Vitest config enables coverage with thresholds and the
1441
+ coverage falls short, Vitest exits 1 after its tests pass. Testkit used to
1442
+ mark every file in the run as an unexplained infrastructure failure. Each file
1443
+ now keeps its own outcome, and the run fails with a run error that carries
1444
+ Vitest's message:
1445
+
1446
+ ```text
1447
+ FAIL run error
1448
+ Vitest failed the run after its tests passed: ERROR: Coverage for lines (50%) does not meet global threshold (100%)
1449
+ ```
1450
+
1451
+ The thresholds apply to the files Testkit runs, as they do to
1452
+ `vitest run <file>`, so `testkit run --file` can fail on coverage while the
1453
+ whole suite passes.
1454
+
1455
+ ## Record data-flow evidence in a plain Vitest test
1456
+
1457
+ `dataFlow(context)` from `@gfdelarue/testkit/node-vitest/data` gives a plain
1458
+ Vitest test the recorder that `dataScenario()` passes to its callback:
1459
+
1460
+ ```ts
1461
+ import { test } from "vitest";
1462
+ import { dataFlow } from "@gfdelarue/testkit/node-vitest/data";
1463
+
1464
+ test("lists one page of orders", async (context) => {
1465
+ const flow = dataFlow(context);
1466
+ flow.count("orders.queries", 1);
1467
+ flow.budget("orders.queries").atMost(1);
1468
+ });
1469
+ ```
1470
+
1471
+ - Every call in one attempt returns the same recorder. Without the context of
1472
+ a running test, `dataFlow` throws and says how to call it.
1473
+ - A failed budget fails the test that declared it when the test finishes.
1474
+ - The file's `data-flow.json` lists one scenario for each test that recorded
1475
+ data, named by the test's full title and taken from its final attempt. A
1476
+ skipped test, or one that records nothing, adds no scenario.
1477
+ - `dataScenario()` now runs on `dataFlow()`. A scenario inside a `describe`
1478
+ block is named by its full title, the `describe` names and its own name
1479
+ joined by spaces. It used to have its own name only.
1480
+ - Typed `data` files still register every case with `dataScenario()`.
1481
+
1482
+ ## Readiness leaves no connection open
1483
+
1484
+ Testkit no longer holds a connection to a service after its readiness check.
1485
+ The check used `fetch`, and when it gave up on a slow attempt and tried again,
1486
+ the connection pool could open a connection that never carried a request. A
1487
+ Node service that calls `server.close()` on SIGTERM waits for that connection,
1488
+ so it ran past its `local.shutdownTimeoutMs` grace, one second by default, and
1489
+ Testkit killed it with SIGKILL. Its shutdown work, such as flushing logs, did
1490
+ not finish.
1491
+
1492
+ The readiness check now opens its own connections without keep-alive and
1493
+ closes all of them when it returns, whether the service became ready, timed
1494
+ out, exited, or the run was cancelled. It sends the headers `fetch` sent, with
1495
+ `Connection: close`. It still follows up to 20 redirects and still needs a 2xx
1496
+ response. A redirect fails the attempt when its `Location` holds a username or
1497
+ password, as with `fetch`. A redirect with more than one `Location` also fails
1498
+ the attempt. `fetch` joined the values into one URL instead.
1499
+
1500
+ Other changes:
1501
+
1502
+ - A `readyUrl` that holds a username and password now sends them as Basic
1503
+ authentication, including after a relative redirect. `fetch` refused such a
1504
+ URL, so readiness used to time out.
1505
+ - A `readyUrl` on a port that `fetch` blocks, such as 6666, now works.
1506
+ - Readiness connects to the service directly. With `NODE_USE_ENV_PROXY` or
1507
+ `--use-env-proxy`, it no longer goes through `HTTP_PROXY` or `HTTPS_PROXY`.
1508
+
1509
+ ## Every service URL answers HTTP before tests start
1510
+
1511
+ A local service's `baseUrl` now has to answer HTTP before any test receives it.
1512
+ Any response counts, including a 4xx or 5xx status, because only the test knows
1513
+ which status is wrong. A port that accepts connections but does not answer HTTP
1514
+ does not count. Testkit retries until `readyTimeoutMs` runs out and then fails
1515
+ the service's startup, so its tasks end `not_run` with an infrastructure
1516
+ failure.
1517
+
1518
+ - A service whose `readyUrl` has the same origin as its `baseUrl` is checked
1519
+ only at its `readyUrl`, which still needs a 2xx response. `app.node` and
1520
+ `app.next` default to such a `readyUrl`.
1521
+ - `readyUrl: false` used to mean the service was ready once it had started. It
1522
+ now removes only the 2xx check, and `baseUrl` must still answer HTTP.
1523
+ - `local.baseUrl` is optional. A process without an HTTP endpoint, such as a
1524
+ worker, declares none: `worker: { local: { start: "node worker.mjs" } }`. It is
1525
+ ready once it has started, and its groups get no URL for it.
1526
+ `{baseUrl:worker}` fails to resolve, and `run.json` records its `baseUrl` as
1527
+ `null`.
1528
+ - A `readyUrl` that is not an HTTP URL fails startup at once instead of at
1529
+ `readyTimeoutMs`.
1530
+
1531
+ This breaks a config whose local service never answers HTTP at its `baseUrl`,
1532
+ including one with `readyUrl: false`. Delete the service's `baseUrl`. `app.node`
1533
+ and `app.next` always set one, so declare such a service as a plain `local`
1534
+ block instead.
1535
+
1536
+ Playwright tasks no longer check their URL themselves. That check opened a TCP
1537
+ connection, only for a URL outside the local machine, and ran inside the file
1538
+ timeout. The `TESTKIT_HTTP_ENDPOINT_PREFLIGHT` variables and the
1539
+ `http_endpoint_readiness` failure detail are gone.
1540
+
1541
+ ## A port that another process answers no longer counts as ready
1542
+
1543
+ Testkit releases a service's port just before it starts the service. If another
1544
+ process took the port in between and answered HTTP, readiness passed. The
1545
+ service then failed to bind and exited, and the run stopped
1546
+ ([gfdelarue/testkit#339](https://github.com/gfdelarue/testkit/issues/339)).
1547
+ After readiness passes, Testkit now checks that every listener on the port
1548
+ belongs to the service. If one doesn't, Testkit stops the service and starts it
1549
+ again on a fresh port, within the same three attempts, and records the attempt
1550
+ as `another process held the port`. The check runs on Linux and Windows. On
1551
+ macOS, and for a listener Testkit can't attribute, readiness counts as before.
1552
+
1553
+ ## A slow database probe no longer ends readiness
1554
+
1555
+ Testkit gives a local Postgres container 60 seconds to accept connections and
1556
+ probes it with `docker exec pg_isready`, at most 10 seconds per probe. A probe
1557
+ that ran past its 10 seconds used to fail the database at once with `Docker
1558
+ local Postgres readiness probe ... exceeded its managed deadline`, even with
1559
+ most of the 60 seconds left. A busy Docker daemon can make one probe that slow.
1560
+ That probe now counts as one failed attempt, and Testkit keeps probing until the
1561
+ database answers or the 60 seconds run out. Cancelling the run still stops
1562
+ readiness at once.
1563
+
1564
+ ## A slow database query is no longer cut off at 15 seconds
1565
+
1566
+ Testkit gave each `docker exec psql` admin query 15 seconds, then failed its
1567
+ operation with `Docker Postgres admin query ... exceeded its managed deadline`.
1568
+ A `CREATE DATABASE` that waited for a lock on its template, or ran slowly on a
1569
+ busy Docker host, failed this way while Postgres was still doing the work
1570
+ ([gfdelarue/testkit#334](https://github.com/gfdelarue/testkit/issues/334)).
1571
+ A query now runs for as long as its Postgres backend is active. Every 15
1572
+ seconds Testkit looks for the query's session in `pg_stat_activity`. When two
1573
+ checks in a row miss it, or a check fails, Testkit stops the query, terminates
1574
+ its backend, and reports the same error.
1575
+
1576
+ ## Docker reads go to the CLI after a direct read times out
1577
+
1578
+ Each Docker metadata operation reads the engine directly for up to 5 seconds,
1579
+ then runs the docker CLI for the rest of its deadline. Once a direct read times
1580
+ out, later operations in the process go straight to the CLI, instead of losing
1581
+ another 5 seconds each. A deadline error now says how long each step took:
1582
+
1583
+ ```text
1584
+ Docker container inventory exceeded its managed deadline: the direct read took 5.0 s, then the docker CLI ran 25.0 s
1585
+ ```
1586
+
1587
+ ## Node/Vitest failures keep their errors
1588
+
1589
+ These changes apply to every node-vitest file
1590
+ ([gfdelarue/testkit#237](https://github.com/gfdelarue/testkit/issues/237)):
1591
+
1592
+ - A file whose import throws now fails with that error as a `vitest-error`,
1593
+ instead of `NODE_VITEST_NO_TESTS`. A file that defines no tests still fails
1594
+ with `NODE_VITEST_NO_TESTS`.
1595
+ - A `vitest-error` from an import or a suite-level hook such as `beforeAll`
1596
+ keeps the error's stack in `stack`, and its `message` now starts with the
1597
+ error's name, such as `TypeError:`.
1598
+ - A `vitest-test` failure's `message` is the error's name and its whole
1599
+ message, as Vitest prints them, such as
1600
+ `AssertionError: expected 2 to be 3 // Object.is equality`. The stack moves
1601
+ to `stack`. The message used to be the error's stack text. For a test that
1602
+ hits its own timeout, that text didn't contain "Test timed out".
1603
+ - After retries, a `vitest-test` failure describes the final attempt. The
1604
+ earlier attempts' errors stay in the native report.
1605
+ - A `vitest-test` failure's `location` gives the line and column where the
1606
+ test is declared, beside its path. It used to give the path alone.
1607
+ - A missing or invalid native report is an infrastructure failure,
1608
+ `NODE_VITEST_REPORT_INVALID`, whatever Vitest's exit code. A nonzero exit
1609
+ used to make it a product failure.
1610
+ - The `testkit-vitest-report` artifact lists every case with its state and the
1611
+ number of attempts Vitest made, so a retried test's attempts are in the run's
1612
+ evidence.
1613
+
1614
+ ## `bug record` and `bug file` check the test a bug names
1615
+
1616
+ - `bug record` records a bug only for a test that Testkit discovers. It used
1617
+ to accept any existing file whose name gave a type, so a draft could name a
1618
+ file that `bug file` could not run. A misplaced typed file now stops
1619
+ `bug record` with discovery's own error, as it stops `testkit run`.
1620
+ - A draft records `testType` only when the test file's name gives one.
1621
+ - `bug file` refuses a node-vitest test when another test in the same file has
1622
+ the same full title, whether that test passed or failed. An issue names its
1623
+ test by file and title, so it could not tell the two apart. Give each test
1624
+ its own title, then record the bug again
1625
+ ([gfdelarue/testkit#237](https://github.com/gfdelarue/testkit/issues/237)).
1626
+
1627
+ ## `cleanup --dir` cleans the product it names
1628
+
1629
+ `testkit cleanup --dir` cleans the product that `--dir` names when that product
1630
+ has no config. It used to clean the directory it was run from. A run with
1631
+ nothing to run, from a product with no config, wrote its results there too
1632
+ ([gfdelarue/testkit#322](https://github.com/gfdelarue/testkit/issues/322)).
1633
+
1634
+ ## Env files are read only for the services a run uses
1635
+
1636
+ Loading `testkit.config.ts` no longer reads any env file. Testkit reads a
1637
+ service's `envFile`, `envFiles`, or default `.env.testkit` and `.env` files only
1638
+ when a selected group uses the service or a used service depends on it. A
1639
+ missing or unreadable file of an unused service no longer stops a run. File
1640
+ order and precedence are unchanged. A later file overrides an earlier one, `env`
1641
+ overrides every file, `local.env` overrides `env`, and a group takes each name
1642
+ from the first service in its `uses` that sets it.
1643
+
1644
+ - A service that declares only `env` or env files starts nothing, and neither
1645
+ does a `dependsOn` edge between such services. Their groups get the values
1646
+ with no runtime slot, `testkit.runtime.ensure`, graph directory, or teardown.
1647
+ Such a service used to take a runtime slot.
1648
+ - A group that uses a service now gets the config's `ui` settings even when its
1649
+ services declare nothing else. It used to get them only when a service
1650
+ declared env values or a dependency.
1651
+ - Because nothing starts for it, such a service's `env` and `ui` can use only
1652
+ `{service}`, `{stateDir}`, `{productDir}`, `{lease}`, `{leaseDir}`, `{bin}`,
1653
+ and `{pathList}`. `{runtime}` and `{prepareDir}` used to resolve against the
1654
+ runtime it no longer takes. They and every port, URL, and database token now
1655
+ fail the group's tasks with an infrastructure failure that names the token.
1656
+ - `defineTestkitConfig` from `@gfdelarue/testkit/playwright` no longer loads
1657
+ dotenv files. It used to load `.env.local` outside a Testkit run, and also
1658
+ inside one with `onlyWhenAllowed: false`. Its `cwd`, `dotenvFiles`,
1659
+ `onlyWhenAllowed`, and `override` options are removed, and passing one throws.
1660
+ In a Testkit run, the services a group uses have already provided their
1661
+ values when the config loads. To load a file when you run Playwright on its
1662
+ own, do it in `playwright.config.ts` with a dotenv library of your own.
1663
+ `@gfdelarue/testkit/env` does not resolve inside a managed Playwright run, so
1664
+ importing it there fails the run.
1665
+ - `archive.declaredEnvironmentKeys` in `run.json` and the bundle provenance lists
1666
+ the names a used service declares in `env`. It no longer includes names that
1667
+ only an env file sets.
1668
+
1669
+ ## Test groups
1670
+
1671
+ `testkit.config.ts` takes a `groups` key. A group owns the test files under its
1672
+ `roots` and names the services its tests use:
1673
+
1674
+ ```ts
1675
+ export default defineConfig({
1676
+ services: {
1677
+ api: app.node({ cwd: "services/api", start: "node dist/server.js" }),
1678
+ web: app.next({ cwd: "apps/web", dependsOn: ["api"] }),
1679
+ },
1680
+ groups: {
1681
+ api: { roots: ["services/api"], uses: ["api"] },
1682
+ web: { roots: ["apps/web"], uses: ["web", "api"] },
1683
+ },
1684
+ });
1685
+ ```
1686
+
1687
+ - A file belongs to the group with the deepest root that contains it. Files
1688
+ that no group claims belong to the implicit group `root`, which uses no
1689
+ service. With no config, every file is in `root`. In a repository without
1690
+ services, discovery, file IDs, and run results now name `root` where they
1691
+ named `app`.
1692
+ - A test gets values only from the services its group uses. When two of them
1693
+ set the same name, such as `BASE_URL`, `DATABASE_URL`, or an `env` entry, the
1694
+ first in `uses` wins.
1695
+ - `skip`, `requirements`, and `runtimeIsolation` belong to the group. Beside
1696
+ `groups`, Testkit rejects them on a service with a message naming the group
1697
+ field to use, and it ignores a service's `discovery`. `groups: {}` gives
1698
+ every file to `root`.
1699
+
1700
+ A config with `services` and no `groups` key keeps legacy ownership: each
1701
+ service owns the typed files under its `local.cwd` and `discovery.roots`. A
1702
+ typed file outside every service root now belongs to `root`; discovery used to
1703
+ fail it with `unowned_test`. When legacy ownership gives a file to a service,
1704
+ `run`, `discover`, `workspace`, and `typecheck` print a notice once on stderr.
1705
+ Legacy ownership and the `--service` alias on test-selecting commands will be
1706
+ removed
1707
+ ([gfdelarue/testkit#241](https://github.com/gfdelarue/testkit/issues/241)).
1708
+
1709
+ `run`, `discover`, `artifacts`, and `workspace` take `--group <name>`. On those
1710
+ four commands `--service` still works as an alias for `--group` and prints a
1711
+ notice. Passing both with different names is a usage error. Other commands,
1712
+ such as `status`, `destroy`, `cleanup`, and `db schema`, keep `--service` for
1713
+ services. `workspace --group` filters discovery and targets but not service
1714
+ status, which `workspace --service` used to filter.
1715
+
1716
+ `discover --json` is schema 10:
1717
+
1718
+ - `services[]` is `groups[]`. Each group has `name`, `roots`, `uses`, `source`
1719
+ (`groups`, `services`, or `implicit`), `suiteCount`, `fileCount`,
1720
+ `activeFileCount`, and `skippedFileCount`. `localCwd`, `dependsOn`, and
1721
+ `discovered` are gone. `root` is always listed.
1722
+ - On files and suites, `service` is `group` and `groupLabel` is `suiteLabel`.
1723
+ `dependsOn` and `contract.executionHost` are gone.
1724
+ - `filters.service` is `filters.group`, `summary.services` is
1725
+ `summary.groups`, and `summary.byService` is `summary.byGroup`.
1726
+ `ambiguous_test` diagnostics list `groupNames`. `unowned_test` no longer
1727
+ occurs.
1728
+ - File and suite IDs keep their format, with the group name first.
1729
+
1730
+ `workspace --json` is schema 3. It embeds the schema 10 summary and
1731
+ diagnostics, and its targets carry `group` in place of `service`.
1732
+ `@gfdelarue/testkit/workspace` exports `DiscoveryGroup` in place of
1733
+ `DiscoveryService`, and `discoverTests` and `inspectWorkspace` take a `group`
1734
+ option in place of `service`.
1735
+
1736
+ ## Test ownership
1737
+
1738
+ `testkit discover` and the loader now assign test files to owners through one
1739
+ resolver, which `groups` builds on (#233). Under legacy ownership, owner names
1740
+ and file IDs stay the same, with these changes:
1741
+
1742
+ - `run` and `typecheck` find a Playwright config by walking up from the common
1743
+ parent of the owning service's discovery roots to the product directory. They
1744
+ used to look only in the service's rendered `local.cwd`. A service with
1745
+ `discovery: { roots: ["apps/web/e2e"] }` now uses
1746
+ `apps/web/playwright.config.ts`.
1747
+ - A service named `root` is rejected and must be renamed, because `root` is
1748
+ the implicit group that owns files no other group claims.
1749
+ - A `discovery.exclude` entry that names a single file now drops that file in
1750
+ every mode. Before, it was ignored when the config declared no services, and
1751
+ with services a file outside every service root was reported as
1752
+ `unowned_test`.
1753
+ - A file that two discovery roots of the same service both claim is no longer
1754
+ ambiguous. With `local.cwd: "api"` and `discovery.roots: [".", "../api"]`, a
1755
+ file under `api` used to be reported as `ambiguous_test` naming `api, api`.
1756
+ It now belongs to `api`.
1757
+ - A file that overlapping repository `discovery.roots` both reach, such as
1758
+ `[".", "src"]`, is listed once. Its suite used to list it twice.
1759
+ - Two suites in a group and type no longer get the same name. When short names
1760
+ collided, the fallback could give a suite a numbered name that another
1761
+ directory already had, such as `a-b-2`, and merge the two suites.
1762
+
1763
+ A run builds its tasks from test groups and starts only the services those
1764
+ groups use, with their `dependsOn`. Under legacy ownership each service owns
1765
+ its tests as a group that uses that service, so a run starts the same services
1766
+ as before. Two things change:
1767
+
1768
+ - A task gets values only from the services its group uses. Its endpoints no
1769
+ longer include a `dependsOn` service, so a Playwright test's
1770
+ `readPlaywrightContext().endpoints` lists only its own service. A used
1771
+ service can still pass a dependency's URL on through `env`, for example with
1772
+ `endpoint.serviceContextUrl("api")`.
1773
+ - A service whose selected files are all skipped no longer reserves a runtime
1774
+ instance or database capacity. `run.json` leaves it out of
1775
+ `environment.capacity.instanceCount` and
1776
+ `environment.capacity.resourceBudgets`.
1777
+
1778
+ ## Locks left by a dead process
1779
+
1780
+ Testkit runs, database work, and Node toolchain installs take locks that other
1781
+ Testkit processes on the host respect. A process that dies while it holds or
1782
+ waits for one of these locks leaves a claim behind, and the next process to find
1783
+ the claim removes it. When several processes found the same dead claim, each of
1784
+ them deleted it. On Windows, a process could get `EPERM` while another was still
1785
+ deleting the claim, and its command failed
1786
+ ([gfdelarue/testkit#255](https://github.com/gfdelarue/testkit/issues/255)).
1787
+
1788
+ A process now renames a dead claim out of the lock directory before it deletes
1789
+ it, and only the process whose rename succeeds deletes the claim. The others
1790
+ treat the claim as gone. Two ordinary renames of one directory can both succeed
1791
+ on Windows, so there Testkit's Windows helper makes the rename through a handle
1792
+ that keeps every other process from renaming or deleting the claim until the
1793
+ move is done. A permission error still fails the command. On Windows, another
1794
+ process reading the claim can block the rename for a moment, so Testkit retries
1795
+ it for up to about ten seconds first. A process that dies
1796
+ between the rename and the deletion leaves a `<lock>.dir.reclaiming-<id>`
1797
+ directory next to the lock. It no longer affects the lock, and you can delete
1798
+ it.
1799
+
1800
+ ## Windows locks wait out a briefly held file
1801
+
1802
+ On Windows, an antivirus scanner, an indexer, or another Testkit process
1803
+ checking a lock can hold a lock file open for a moment. Releasing a lock, and
1804
+ creating or removing a lock's directory, failed on the first such `EPERM`,
1805
+ `EBUSY` or `EACCES` error
1806
+ ([gfdelarue/testkit#270](https://github.com/gfdelarue/testkit/issues/270)).
1807
+ A failed release left the lock held until its process exited, so in a
1808
+ long-running process every later wait for that lock timed out. A new
1809
+ acquisition also failed while the previous lock directory was still being
1810
+ deleted.
1811
+
1812
+ These steps now retry for about one and a half seconds on those errors and
1813
+ throw if the error persists. Release still refuses to remove a lock whose owner
1814
+ or ticket has changed, and on Windows it checks that again before every
1815
+ attempt. It then moves the claim out of the lock directory before deleting it,
1816
+ so a process that dies in between leaves a `<lock>.dir.releasing-<id>`
1817
+ directory that no longer affects the lock and that you can delete. Linux and
1818
+ macOS behave as before.
1819
+
1820
+ ## Database bindings stay in memory
1821
+
1822
+ Testkit used to write each runtime database's URL to a `database_url` file in
1823
+ the runtime's state directory, then read the file back for `runtime.prepare`
1824
+ steps, service processes, `{db*}` placeholders, and test values. Each runtime
1825
+ instance now keeps the binding that provisioning returns and passes it to those
1826
+ consumers directly
1827
+ ([gfdelarue/testkit#140](https://github.com/gfdelarue/testkit/issues/140)).
1828
+
1829
+ - `databaseFrom` now supplies a database. A service with
1830
+ `databaseFrom: "api"` gets `api`'s database from the same runtime instance as
1831
+ `DATABASE_URL`, in `{dbUrl}` and the other database placeholders, and in its
1832
+ `runtime.prepare` steps and fingerprint. It used to get no database at all.
1833
+ The named service must provision its database in the same runtime, so list it
1834
+ in `dependsOn`. Otherwise the run fails with an error that says so.
1835
+ - Testkit no longer writes `database_url` or `local_template_database_name` to
1836
+ runtime state directories under `.testkit/_graphs`, and ignores copies that
1837
+ an earlier version left there. A script that read `database_url` should read
1838
+ `DATABASE_URL` or a `{dbUrl}` env value instead.
1839
+ - A database template counts as cached only after its rebuild finishes.
1840
+ Testkit removes the template's cached fingerprint before a rebuild and writes
1841
+ it again at the end. If a process was killed during a rebuild, the old
1842
+ fingerprint used to stay next to a partly built template, and the next run
1843
+ cloned that template.
1844
+
1845
+ ## A file with setup runs in its own acquisition
1846
+
1847
+ Testkit now runs a file's `.setup.mjs`, `.setup.js`, or `__setup__` module as
1848
+ part of preparing the file's environment, after it acquires the services the
1849
+ group uses. Setup still runs once per file, shares the file's deadline, and
1850
+ keeps its sidecars until the test finishes. Cleanup failures keep their
1851
+ classification: a consumer cleanup error fails the test, and a Testkit cleanup
1852
+ error is an infrastructure failure.
1853
+
1854
+ A Playwright file with setup no longer shares a batch with other Playwright
1855
+ files of its group. It gets an acquisition of its own, as a file with locks
1856
+ does. Files without setup still batch.
1857
+ In `run.json`, such a file has no `runtimeBatch`.
1858
+
1859
+ ## Handles: an HTTP capture by name
1860
+
1861
+ Tests can now use an HTTP capture that their file's setup module started,
1862
+ through `handle(name)` from `@gfdelarue/testkit/environment`. A capture's
1863
+ handle is named after its `id` and has `url`, `requests()`, and `reset()`, and
1864
+ both methods return Promises. A run on supplied values gets the same handle
1865
+ from an `http-capture` entry under `handles` in its environment file. See
1866
+ [Handles](../README.md#handles).
1867
+
1868
+ - A capture's `requestsUrl` and `resetUrl` now hold an unguessable segment,
1869
+ `/__testkit__/<capability>/requests`, which Testkit keeps out of the run's
1870
+ results, artifacts, and events. Setups that hand those URLs to their tests
1871
+ keep working. A request to any other `/__testkit__/` path gets `404`, and a
1872
+ method other than `GET` or `DELETE` on the requests URL gets `405`. The old
1873
+ fixed `/__testkit__/requests` path no longer answers.
1874
+ - A capture `path` may not start with `/__testkit__/`.
1875
+ - A missing handle ends the test as an infrastructure failure, like a missing
1876
+ endpoint or database. A name that several captures share throws
1877
+ `EnvironmentHandleAmbiguousError`. A refused, unavailable, invalid, or
1878
+ unsupported handle throws `EnvironmentHandleError` with that `reason`.
1879
+
1880
+ ## Named endpoints and databases
1881
+
1882
+ Tests can read each used service's URL and database by name from the new
1883
+ `@gfdelarue/testkit/environment` module:
1884
+
1885
+ ```ts
1886
+ import { database, endpoint } from "@gfdelarue/testkit/environment";
1887
+
1888
+ const api = endpoint("api");
1889
+ const reports = database("reports");
1890
+ ```
1891
+
1892
+ - `endpoint(name)` returns the URL of the service `name` that the test's group
1893
+ uses, and `database(name)` returns that service's database URL. A service's
1894
+ database is the one Testkit provisions for it, or the `DATABASE_URL` its
1895
+ `env` declares.
1896
+ - The functions work in node-vitest and Playwright files and in the modules
1897
+ they import. Testkit installs the values before the test file and the
1898
+ product code it imports load, so module-level code and native hooks can read
1899
+ them, such as a Playwright config and its `globalSetup` or a Vitest
1900
+ `beforeAll`.
1901
+ - A name with no value throws `EnvironmentValueMissingError`. Its `kind` is
1902
+ `"endpoint"` or `"database"`, `valueName` is the name asked for, or `null`
1903
+ when the missing value is the run's default URL, and `available` lists the
1904
+ names the run provides. Outside a Testkit run every call throws it.
1905
+ - A test that fails on `EnvironmentValueMissingError`, or a file whose imports
1906
+ throw it, now ends as an infrastructure failure, not a test failure, in
1907
+ node-vitest and Playwright. Its failure key is
1908
+ `TESTKIT_ENVIRONMENT_VALUE_MISSING`. The run fails, and `testkit bug file`
1909
+ refuses to file it, as it does for a server that never became ready. A test
1910
+ that catches the error and carries on is unaffected, and a failed query, a
1911
+ failed assertion against a reachable service, or a refused connection stays
1912
+ a test failure.
1913
+ - When the run gives a test no base URL, a Playwright `page.goto()` to a
1914
+ relative URL ends the same way.
1915
+ - `BASE_URL` and `DATABASE_URL` stay. They hold the endpoint and database of the
1916
+ first service in `uses` that has one. Playwright's `BASE_URL` and `baseURL`
1917
+ keep their canonical form, so an origin ends with `/`. `endpoint()` returns
1918
+ the URL unchanged in every engine.
1919
+ - `readPlaywrightContext()` gains `databases`.
1920
+ - `testkit.config.ts` and lifecycle step modules cannot import the module.
1921
+
1922
+ ## Source schema credentials stay out of tests
1923
+
1924
+ Testkit kept `TESTKIT_SCHEMA_SOURCE_DATABASE_URL` and the service's own
1925
+ `TESTKIT_<SERVICE>_SCHEMA_SOURCE_DATABASE_URL` out of the values it gives a
1926
+ group's tests. A name chosen with `database.schema.fromEnv()`, or another
1927
+ service's `TESTKIT_<SERVICE>_SCHEMA_SOURCE_DATABASE_URL`, still reached tests
1928
+ when a used service's `env` set it, so tests could receive a source database
1929
+ URL, often a production database. Tests no longer receive a value under any
1930
+ name that the source schema of a service in their runtime reads, whichever
1931
+ service sets it
1932
+ ([gfdelarue/testkit#279](https://github.com/gfdelarue/testkit/issues/279)).
1933
+
1934
+ ## A suite selector's type is the text before its colon
1935
+
1936
+ The text before the first colon of a suite selector is now always the test
1937
+ type ([gfdelarue/testkit#283](https://github.com/gfdelarue/testkit/issues/283)).
1938
+ A selector such as `Int:auth` or `v2-api:auth` used to name a plain suite with
1939
+ that whole name. It now fails with `Unknown suite selector type`.
1940
+
1941
+ ## A shared database keeps its rows when another runtime instance starts
1942
+
1943
+ With `binding: "shared"`, every runtime instance of a service uses one
1944
+ database. Testkit reset that database each time it set up an instance, and a
1945
+ run sets up an instance the first time a test needs it. An instance set up
1946
+ partway through a run dropped the shared database while tests on other
1947
+ instances were using it
1948
+ ([gfdelarue/testkit#284](https://github.com/gfdelarue/testkit/issues/284)).
1949
+ Postgres terminated their connections, and their rows disappeared.
1950
+
1951
+ Testkit now resets a shared database once per run, when the first instance sets
1952
+ it up. Later instances use it as it is, and so does an instance that Testkit
1953
+ sets up again after a failure. If the database no longer exists, the instance
1954
+ clones it again from the template. `reset: false` and the default `per-runtime`
1955
+ binding behave as before.
1956
+
1957
+ ## `resetBefore` gives a file its own database
1958
+
1959
+ `runtimeIsolation` accepted `shared`, `resetBefore`, `resetAfter`, and
1960
+ `exclusive`, but only batching read it. `resetBefore` and `resetAfter` reset
1961
+ nothing, and `exclusive` didn't keep other files off the runtime
1962
+ ([gfdelarue/testkit#281](https://github.com/gfdelarue/testkit/issues/281)).
1963
+
1964
+ `resetBefore` now works. Before the file runs, Testkit clones each database it
1965
+ provisions that the file's environment reaches, from the template that database
1966
+ came from. That includes a database a used service names in its env, such as
1967
+ `{dbUrl:audit}`. The file's
1968
+ `.setup.*` module, `database()`, `DATABASE_URL`, and `{dbUrl}`-style values in
1969
+ the group's service env point at the clone, and Testkit drops it when the file
1970
+ finishes. A clone took 0.1–0.15 s from an empty template and about 0.35 s from
1971
+ a 117 MB one, on Linux and on the Windows CI runner. Use it for a test that
1972
+ changes a whole table, such as an unfiltered `DELETE` or a migration test.
1973
+
1974
+ Running services keep the runtime's database, so `resetBefore` isolates only
1975
+ the test's own database access. A request the test sends to an API server
1976
+ still reads and writes the shared database. When the server's writes must be
1977
+ isolated too, give the test its own group whose `uses` names a second service
1978
+ entry for the same server. Groups that use the same services share their
1979
+ runtimes.
1980
+
1981
+ A `resetBefore` file on supplied values, or whose environment reaches no
1982
+ database Testkit provisions, ends as an environment failure.
1983
+
1984
+ `resetAfter` and `exclusive` are removed. A config or `defineFile` that sets
1985
+ either fails to load with a message that names the replacement: `resetBefore`
1986
+ on the file that needs a clean database, or `shared` with tests that create
1987
+ and filter by their own rows. This is a breaking removal.
1988
+
1989
+ ## Run tests against supplied values
1990
+
1991
+ `testkit run` takes `--base-url`, `--endpoint NAME=URL`, `--database NAME=URL`,
1992
+ `--env NAME=VALUE`, `--env-from NAME=HOST_KEY`, and `--environment-file PATH`
1993
+ ([gfdelarue/testkit#235](https://github.com/gfdelarue/testkit/issues/235)).
1994
+ With any of them, every selected group runs on the supplied values. Testkit
1995
+ starts none of its services, creates no database, and runs no `.setup.*`
1996
+ module, and the group keeps its name, selection, and policy. Tests read named
1997
+ endpoints and databases through `@gfdelarue/testkit/environment`. The environment
1998
+ file is versioned JSON holding values, the setup data tests receive, and handle
1999
+ descriptors. Conflicting values, a missing `--env-from` host variable, and an
2000
+ invalid or unknown-version file fail before the run starts.
2001
+
2002
+ Before a test receives a supplied endpoint, Testkit checks that it answers
2003
+ HTTP, for up to 30 seconds. Any response counts, including a 404 or 500. An
2004
+ endpoint that refuses connections or never answers HTTP ends the run as an
2005
+ infrastructure failure.
2006
+
2007
+ Before a test receives a supplied database, Testkit connects to it, signs in,
2008
+ and runs `SELECT 1`. A database that is unreachable, fails TLS, rejects the
2009
+ credentials, does not exist, or is not PostgreSQL ends the run as an
2010
+ infrastructure failure, so `testkit bug file` refuses it. Without the check,
2011
+ the test ran and failed on the connection error, which read as a product bug.
2012
+ The check speaks the PostgreSQL protocol itself, with trust, password, md5, and
2013
+ SCRAM-SHA-256 authentication, TLS from `sslmode`, and Unix sockets through
2014
+ `host=/path`.
2015
+ [Run against a supplied environment](../README.md#run-against-a-supplied-environment)
2016
+ describes the flags and the file.
2017
+
2018
+ `testkit bug file` takes the same flags and reruns the recorded test on them.
2019
+ The issue's reproduce command carries `--base-url`, `--endpoint`, `--env-from`,
2020
+ and `--environment-file` as given. `testkit bug file` refuses `--database`,
2021
+ `--env`, and URLs with credentials, a query, or a fragment, so the command never
2022
+ holds a raw secret.
2023
+
2024
+ ## A file reads the same values from every source
2025
+
2026
+ A file now gets the same names and setup data whether the run starts its
2027
+ services or takes supplied values
2028
+ ([gfdelarue/testkit#326](https://github.com/gfdelarue/testkit/issues/326)):
2029
+
2030
+ - A group's default endpoint and database are available by the group's name in
2031
+ a configured run too. For a group `checks` that uses `api`,
2032
+ `endpoint("checks")` used to throw `EnvironmentValueMissingError` unless the
2033
+ values were supplied. Supplied values now also name a default taken from
2034
+ `uses` after the group, not only an explicit one such as `--base-url`.
2035
+ - A run refuses a group whose default differs from a used service's endpoint or
2036
+ database with the group's name, such as a group `api` with
2037
+ `uses: ["web", "api"]`. From either source, that group's files end as an
2038
+ infrastructure failure that names the group, and other groups still run,
2039
+ including groups that use the same services. A supplied run used to fail
2040
+ before it started. Rename the group, or list the service of that name first
2041
+ in `uses`.
2042
+ - Each acquisition in `run.json` records `setupData`: the supplied setup data or
2043
+ what the file's setup module returned, and `{}` without either, which is what
2044
+ tests already received. A configured acquisition used to leave it out.
2045
+ - A configured acquisition no longer records a service's `BASE_URL` env value
2046
+ under `env`. Tests never received it there, because every engine sets
2047
+ `BASE_URL` from the group's default endpoint.
2048
+
2049
+ So an acquisition's recorded `values`, with `"version": 1` added, work as an
2050
+ `--environment-file` that gives a second run the same values.
2051
+
2052
+ ## Sealed runs redact the secrets a run's environment resolved
2053
+
2054
+ Sealing took the secrets it redacts from the environment Testkit runs in. A
2055
+ value that reached a service or a test from `.env.testkit`, another env file,
2056
+ config `env`, or `local.env` stayed in the sealed run whenever a server log, a
2057
+ test's output, or an assertion message printed it
2058
+ ([gfdelarue/testkit#186](https://github.com/gfdelarue/testkit/issues/186)).
2059
+ Values from `--env`, `--env-from`, and `--environment-file` had the same gap.
2060
+
2061
+ A run now keeps, in memory, the values its environment resolves for each
2062
+ service, dependency services included, and for supplied input. Sealing redacts
2063
+ each value at least eight characters long under a key that holds `PASSWORD`,
2064
+ `TOKEN`, `SECRET`, `PRIVATE_KEY`, `AUTH`, `COOKIE`, `SESSION`, or `API_KEY`,
2065
+ and the password of each resolved URL. A configured run and a supplied run that
2066
+ print the same secrets seal the same redactions. Testkit takes a runtime's
2067
+ values before it prepares the runtime's database, and adds the values that
2068
+ depend on the database once it is ready, so a failed database seed, a run
2069
+ stopped during startup and a server that never becomes ready are redacted too.
2070
+
2071
+ - Sealing also redacts a value's JSON-escaped forms, each line of a value that
2072
+ spans lines, keys in the free-form data and metadata that tests attach, and
2073
+ the part of a secret that output shows before a `…` or `...` cut.
2074
+ - `testkit run --json` and the run's progress output apply the same redaction.
2075
+ They used to show unredacted test results.
2076
+ - When a run's process is killed before it seals the run, the process that
2077
+ seals it later does not have these values. It redacts patterns and its own
2078
+ environment only, and the bundle's `archive.redaction.status` reads
2079
+ `source-targets-unavailable` instead of `applied`.
2080
+ - A secret under another key name, such as `STRIPE_SK`, counts when `redact`
2081
+ names its key. Sealing still does not look inside compressed files such as
2082
+ Playwright traces
2083
+ ([gfdelarue/testkit#187](https://github.com/gfdelarue/testkit/issues/187)).
2084
+
2085
+ ## Stored run results redact whole secrets
2086
+
2087
+ Sealing redacted a `KEY=value` secret only up to its first space, and left
2088
+ header and JSON string secrets whole. A test title such as
2089
+ `rejects PASSWORD="first-part example-secret" with 401` was stored as
2090
+ `rejects PASSWORD=<redacted> example-secret" with 401`, in `run.json`,
2091
+ `latest.json`, and the run's JSON artifacts
2092
+ ([gfdelarue/testkit#213](https://github.com/gfdelarue/testkit/issues/213)).
2093
+
2094
+ Sealing, the run's returned result and progress output, and error messages now
2095
+ use the rules `testkit bug file` uses, which `docs/bug-records.md` describes. A
2096
+ quoted value ends at its closing quote and is written as `"<redacted>"`, so
2097
+ redacting the text again changes nothing. Filed issues and CI reports show it
2098
+ the same way. A value that starts with an auth scheme such as `Bearer` ends
2099
+ with its line, and `Authorization`, `Proxy-Authorization`, `Cookie`, and
2100
+ `Set-Cookie` header values and JSON string values under a secret's key are
2101
+ redacted, also when the key is spelled with escapes such as `"\u0074oken"`. A
2102
+ single-quoted `'password': '…'` value, which only error messages redacted
2103
+ before, is redacted by the same rules, and it ends at the next apostrophe that
2104
+ no backslash precedes. A value that starts with `\"` or `\'`, as a quoted
2105
+ value does inside a JSON string, ends at the matching escaped quote. Sealing
2106
+ reads JSON value by value: a JSON artifact, a log line from its first `{` or
2107
+ `[`, and a string that holds JSON. It redacts each decoded string, redacts a
2108
+ string member under a secret's key, and rewrites only the strings it changes,
2109
+ so a secret in one value leaves the record's other values, and numbers, spacing
2110
+ and line endings keep their exact bytes.
2111
+
2112
+ ## Name more secret keys with `redact`
2113
+
2114
+ A secret under a key that holds none of `PASSWORD`, `TOKEN`, `SECRET`,
2115
+ `PRIVATE_KEY`, `AUTH`, `COOKIE`, `SESSION`, or `API_KEY`, such as `STRIPE_SK`,
2116
+ stayed in a sealed run unless it was a URL's password
2117
+ ([gfdelarue/testkit#186](https://github.com/gfdelarue/testkit/issues/186)).
2118
+
2119
+ `redact` in `testkit.config.ts` names more keys:
2120
+
2121
+ ```ts
2122
+ export default defineConfig({
2123
+ redact: ["STRIPE_SK", "PGPASS"],
2124
+ });
2125
+ ```
2126
+
2127
+ Testkit matches the names in any letter case. A value under one counts the way
2128
+ a value under a built-in key does, from every source a run's environment
2129
+ resolves and from the environment Testkit runs in, in a configured run and a
2130
+ supplied run alike. It counts when it is at least eight characters long. A
2131
+ resolved value that is a URL counts only for its password; in the environment
2132
+ Testkit runs in, it counts whole and for its password. The bundle's
2133
+ `sensitiveEnvironmentKeys` lists such a key when the environment Testkit runs
2134
+ in holds it. `redact` doesn't add text forms: `STRIPE_SK=…` in a log is redacted
2135
+ when its value is one Testkit resolved. A config whose `redact` isn't a list of
2136
+ non-empty strings fails to load, and the message names the entry.
2137
+
2138
+ ## A PostgreSQL URL's query password is redacted
2139
+
2140
+ libpq also reads a password from the URL's query string, as in
2141
+ `postgres://db.internal/app?password=…`. Sealing treated only a password before
2142
+ the `@` as a resolved secret, so a test that printed that password on its own
2143
+ left it in the sealed run. A `postgres://` or `postgresql://` URL's `password`
2144
+ query parameter now counts as the URL's password, both raw and percent-decoded.
2145
+
2146
+ ## Retention keeps the objects that every stored run lists
2147
+
2148
+ Retention, and the cleanup after a failed seal, free the stored objects under
2149
+ `.testkit/results/objects` that no run bundle lists. They used to count only the
2150
+ bundles whose `run.json` this Testkit could read, so an object that only an
2151
+ unreadable bundle listed was freed. The bundle's own files stayed readable, but
2152
+ the object store no longer held what its manifest named.
2153
+
2154
+ - An object is now kept while any bundle's `manifest.json` lists it, whether or
2155
+ not Testkit can read that bundle's `run.json`.
2156
+ - While a bundle has no readable manifest, retention frees no object. Its
2157
+ decision records `garbageCollection.status: "not_started"` and names the
2158
+ bundle, and the decision's status is `failed`.
2159
+
2160
+ ## Run documents are schema 20
2161
+
2162
+ `run.json`, `latest.json` and `live.json` move from schema 19 to schema 20.
2163
+ Each field that named the service owning a test now names its group, and the
2164
+ facts that depend on how the environment was provided move into one
2165
+ `environment` record. Testkit still reads schema-19 runs stored by 0.4.x through
2166
+ `testkit artifacts`, `ci report`, known-failure checks and the workspace API,
2167
+ and finalizes a 0.4.x `live.json` as schema 19.
2168
+
2169
+ Renamed fields:
2170
+
2171
+ | Schema 19 | Schema 20 |
2172
+ | ------------------------------- | ----------------------------- |
2173
+ | `services` | `groups` |
2174
+ | `summary.services` | `summary.groups` |
2175
+ | `summary.diagnostics[].service` | `summary.diagnostics[].group` |
2176
+ | `run.serviceFilter` | `run.groupFilter` |
2177
+ | `planning.tasks[].service` | `planning.tasks[].group` |
2178
+
2179
+ Moved into `environment`:
2180
+
2181
+ | Schema 19 | Schema 20 |
2182
+ | -------------------------- | -------------------------------------- |
2183
+ | `run.runtimeInstanceCount` | `environment.capacity.instanceCount` |
2184
+ | `run.runtimeStats` | `environment.capacity.runtimeStats` |
2185
+ | `planning.resourceBudgets` | `environment.capacity.resourceBudgets` |
2186
+ | `run.dbBackend` | `environment.dbBackend` |
2187
+ | `services[].dbBackend` | `environment.groups[].dbBackend` |
2188
+ | `environment.services` | `environment.services` (unchanged) |
2189
+ | `logs` | `environment.logs` |
2190
+ | `setup.operations` | `environment.setupOperations` |
2191
+
2192
+ `environment` also gains these fields:
2193
+
2194
+ - `acquisitions`: one entry for each environment the run acquired. It records:
2195
+ - the source that prepared it: `configured`, `static`, `supplied` or `none`;
2196
+ - the tasks it served, its runtime and services;
2197
+ - each file's setup;
2198
+ - its release outcome;
2199
+ - its timings, as ISO times;
2200
+ - the values its tests received, in the `--environment-file` shape with
2201
+ secrets and capture capabilities redacted.
2202
+ - `cleanup`: the run's cleanup outcome.
2203
+ - `missingValues`: the named values tests asked for that their environment
2204
+ lacked.
2205
+
2206
+ The same rename reaches the run's other outputs:
2207
+
2208
+ - `--output-mode events`:
2209
+ - `run.service_plans` is now `run.group_plans`, with `groups[].group`.
2210
+ - `run.service_skipped` is now `run.group_skipped`, with `group`.
2211
+ - `run.task_planned_skip`, `run.task_started`, `run.task_finished` and
2212
+ `run.runtime_error` carry `group` instead of `service`.
2213
+ - `run.summary` results no longer carry `dbBackend`.
2214
+ - Events about a real service keep `service`: `run.local_service_starting`,
2215
+ `run.toolchain_resolved` and `run.setup_finished`.
2216
+ - The run summary reads `groups N/M passed` and `N/M groups failed`.
2217
+ - `testkit run --json` prints the schema-20 `runArtifact`. Its `results` no
2218
+ longer carry `dbBackend`.
2219
+ - `testkit artifacts --json` names each artifact's `group` instead of its
2220
+ `service`.
2221
+ - In `@gfdelarue/testkit/workspace`:
2222
+ - `loadRunBundleArtifact`, `latestRunArtifact` and `liveRunArtifact` return a
2223
+ schema-19 or schema-20 document.
2224
+ - A run summary counts `groups` instead of `services`.
2225
+
2226
+ A run bundle's `provenance.json` keeps its `scenario.serviceFilter` field, which
2227
+ now holds the group filter. The `--service` flag still selects a group.
2228
+
2229
+ ## One declaration file per entry point
2230
+
2231
+ Each public entry point now has one bundled declaration file under
2232
+ `dist/types/`, and the package ships 14 of its own `.d.ts` files instead of
2233
+ the 483 that 0.4.5 shipped. The root `types` field and each export's `types`
2234
+ condition point at the bundles. A bundle exports the entry point's own names
2235
+ and every interface and type alias its public signatures use, so types that
2236
+ consumers received but could not import, such as `DataEvidence` or
2237
+ `RunBundleManifest`, now have importable names. Public signatures no longer
2238
+ name internal types:
2239
+
2240
+ - `@gfdelarue/testkit/playwright`: `EngineJsonObject`, `EngineJsonValue` and
2241
+ `EngineJsonPrimitive` are now `JsonObject`, `JsonValue` and `JsonPrimitive`.
2242
+ - `@gfdelarue/testkit/config/database-steps`: `DatabaseStepContext` no longer
2243
+ has `ownedContainer`, which Testkit supplies itself.
2244
+ - `@gfdelarue/testkit/workspace`: a run bundle's `archive.retention` is typed
2245
+ as `RunBundleRetentionSummary`, with its status, time, policy and error. The
2246
+ stored block still holds the garbage collection and removal details.
2247
+ `WorkspaceServiceStatus` and `WorkspaceRuntimeGraphStatus` type `product`,
2248
+ `caches` and `desired` directly instead of through the internal status model.
2249
+ - `__testkitRegisterNodeVitestRepositoryConfiguration` is no longer declared.
2250
+ It remains a runtime export for Testkit's generated setup modules.
2251
+ - `UnknownRecord` is gone; the signatures that used it say
2252
+ `Record<string, unknown>`.
2253
+
2254
+ A project that referenced a declaration file by its path under `dist/` must
2255
+ import the package subpath instead. See
2256
+ [gfdelarue/testkit#214](https://github.com/gfdelarue/testkit/issues/214).
2257
+
2258
+ ## Engine documentation in the package
2259
+
2260
+ The package ships `docs/engines.md` in place of `docs/engine-architecture.md`.
2261
+ `docs/engines.md` covers engine authoring, typechecking, the environment a test
2262
+ receives, Playwright attachments, installation, and the supported platforms.
2263
+ The README and the configuration reference no longer describe Testkit's
2264
+ internals, and every relative link in the shipped docs resolves inside the
2265
+ package ([gfdelarue/testkit#214](https://github.com/gfdelarue/testkit/issues/214)).
2266
+
2267
+ # 0.4.5
2268
+
2269
+ ## Service ports come from the OS
2270
+
2271
+ Testkit no longer computes service ports. It used to add a fixed stride to
2272
+ `local.port`, or to the port in `local.baseUrl`, for each runtime instance and
2273
+ graph. Two runs of the same repository on one host computed the same ports and
2274
+ collided, whether they were two CI jobs on one runner or two agent worktrees.
2275
+
2276
+ - Every service in every runtime instance now gets a port from the OS. Testkit
2277
+ binds `127.0.0.1:0` and the same port on `::1`, holds both until just before
2278
+ it spawns the service, and passes the port as `PORT` and `{port}`.
2279
+ - A service that exits before it becomes ready gets a fresh port and starts
2280
+ again, up to three attempts in total. A service that already started and
2281
+ reads the moved service's URL or port restarts with the new value. A retry
2282
+ never gets a port the service already tried. The final error is an
2283
+ infrastructure failure that lists each attempt's port and outcome and the last
2284
+ attempt's output.
2285
+ - `run.json` gains `environment.services`: each service's `runtimeLabel`,
2286
+ `serviceName`, `baseUrl`, `readyUrl`, `port`, and every start attempt, recorded
2287
+ as it happens, including those of a startup that fails. Recorded URLs drop
2288
+ credentials, query values, and fragments.
2289
+ - The prepare fingerprint now covers the environment that `runtime.prepare`
2290
+ steps see, with its templates resolved, instead of the raw env templates and
2291
+ the service's `start`, `baseUrl`, and `readyUrl`. A prepare step that embeds a
2292
+ service URL or port rebuilds on every run.
2293
+
2294
+ This breaks existing configs:
2295
+
2296
+ - `local.port` and the `port` option of `app.node` and `app.next` are config
2297
+ errors. Delete them and use `{port}` in `start`, `baseUrl`, and `readyUrl`.
2298
+ - A literal port in `local.baseUrl` or `local.readyUrl` is a config error, and
2299
+ `local.baseUrl` must contain `{port}`. `app.node` and `app.next` already
2300
+ default to `http://127.0.0.1:{port}`.
2301
+ - A literal URL in another service's env no longer follows that service's port.
2302
+ Use `{baseUrl:<service>}` or `{port:<service>}`.
2303
+ - There is no setting that pins a service to a fixed port.
2304
+
2305
+ ## Stopped Windows commands
2306
+
2307
+ On Windows, a cancelled or timed-out command now waits up to four seconds for
2308
+ its process tree to exit before it returns. It used to return as soon as its
2309
+ root process exited. The Playwright adapter then removed Chromium's profile
2310
+ while terminated browser processes still held it, and the run failed with
2311
+ `EBUSY` ([gfdelarue/testkit#205](https://github.com/gfdelarue/testkit/issues/205)).
2312
+ A terminated process on a loaded host can take longer than four seconds to
2313
+ exit. The command then returns a termination error that names the process, and
2314
+ the run can still end in an infrastructure failure. A process in the tree that
2315
+ Testkit cannot open or terminate is reported the same way, and so is a process
2316
+ whose parent exited before Testkit could confirm that it belongs to the tree.
2317
+ Testkit leaves both running. Once termination fails, the command returns
2318
+ without waiting for output that a surviving process holds open. It then
2319
+ terminates only the root, through the process handle Node opened when it
2320
+ started it. It no longer starts `taskkill` against the root's process id,
2321
+ which another process may have reused.
2322
+
2323
+ `testkit stop` now gives a Windows run its five-second finalization window
2324
+ before it forces the run, as it does on POSIX. It used to force a Windows run
2325
+ as soon as the two-second grace ended, and the run exited with code 1.
2326
+
2327
+ ## Baseline-relative data budgets removed
2328
+
2329
+ `relativeToBaseline()` is removed from the metric budgets that `budget(name)`
2330
+ and `expectMetric(name)` return in `@gfdelarue/testkit/node-vitest/data`. The
2331
+ `DataBaselineBudgetOptions` type is removed with it. Nothing compared the
2332
+ metric with the named run, so these budgets always reported passed. A scenario
2333
+ that still calls `relativeToBaseline()` fails with a `TypeError`, and
2334
+ TypeScript reports the missing method. Use `atMost`, `atLeast`, `lessThan`,
2335
+ `greaterThan`, or `equals` with a fixed value instead.
2336
+
2337
+ Budget records in `data-flow.json` no longer have the `baseline_relative`
2338
+ operator or the `policy` and `thresholdVersion` fields. This is a breaking
2339
+ removal. See [gfdelarue/testkit#224](https://github.com/gfdelarue/testkit/issues/224).
2340
+
2341
+ ## `createDataTrace()` removed
2342
+
2343
+ `createDataTrace()` is removed from `@gfdelarue/testkit/node-vitest/data`. The
2344
+ context it returned never reached the run's report, so its metrics, evidence,
2345
+ and findings were lost, and its budgets were never checked. A failing budget on
2346
+ a trace passed. Record data inside `dataScenario()` and use the context it
2347
+ passes to the callback. This is a breaking removal. See
2348
+ [gfdelarue/testkit#242](https://github.com/gfdelarue/testkit/issues/242).
2349
+
2350
+ ## Bug records: optional `testType`
2351
+
2352
+ For tools that read the facts block in Testkit's issues:
2353
+
2354
+ - `testType` may be absent. Testkit writes it when the test file's name gives a
2355
+ type, and a later release files bugs for test files whose names give none.
2356
+ Read a version 1 block without `testType` as valid. The format version stays
2357
+ `1`.
2358
+ - Every type that a Testkit release has written stays readable: `ui`, `e2e`,
2359
+ `int`, `contract`, `data`, `dal`, and `load`. Issues keep the type they were
2360
+ filed with, so a reader keeps a type after Testkit stops writing it.
2361
+ - `"testType": null` is invalid, as for every other optional field.
2362
+
2363
+ This release still writes `testType` on every issue it files.
2364
+
2365
+ Four category texts no longer assume the product is a server. They are the
2366
+ boundary rules of the `missing-validation` cause and the `workflow-breakdown`
2367
+ impact, and the definition and boundary rule of the
2368
+ `availability-or-performance` impact. Every id and every list's order is
2369
+ unchanged.
2370
+
2371
+ ## Tests that need nothing run with zero setup
2372
+
2373
+ Testkit no longer starts a server that `testkit.config.ts` does not declare, so
2374
+ a config that relied on an inferred server breaks. Before, a service without
2375
+ `local` got `{bin:next} dev -p {port}` when its directory had a `next.config.*`
2376
+ file, `go run ./cmd/server` when it had `cmd/server/`, and
2377
+ `{bin:tsx} watch src/index.ts` when it had `package.json` and `src/`. A repo
2378
+ with no config got the same server for its implicit `app` owner. Loading a
2379
+ config also failed when a suite's type expected a server that Testkit could not
2380
+ resolve. Both are gone. To keep a server that Testkit used to infer, declare
2381
+ it, for example `services: { app: app.node({ entry: "src/index.ts" }) }`.
2382
+
2383
+ - An owner that declares no `local`, `database`, `databaseFrom`, `dependsOn`,
2384
+ `runtime.build`, `runtime.prepare` step, `runtime.toolchain`, or env values
2385
+ gets no setup. Its tasks reserve no port, claim no runtime slot, run no
2386
+ `testkit.runtime.ensure`, and write nothing under `.testkit/_graphs`. A run
2387
+ made only of such tasks records no `environment.services` and does no
2388
+ Postgres or Docker maintenance.
2389
+ - With no config, the implicit `app` owner loads no `.env` or `.env.testkit`
2390
+ file. A service in `testkit.config.ts` still loads them.
2391
+ - Only `data` tests import `testkit.config.ts` in the test process, for their
2392
+ HTTP profiles. In other Node/Vitest tests, `getRepoConfig()` returns `null`.
2393
+ - `testkit status`, `doctor`, `typecheck`, and `destroy` work in a repo whose
2394
+ owners declare nothing. `destroy` lists Docker containers only when a service
2395
+ declares a database or local database state exists.
2396
+
2397
+ ## A test's type no longer decides its setup or its values
2398
+
2399
+ A task now gets everything its service declares, whatever its test type. A
2400
+ `dal` task used to get its service's database, `runtime.build` and
2401
+ `runtime.prepare` steps, and toolchain, but not the server in `local`. That
2402
+ server now starts for `dal` tasks too, and Testkit waits for its `readyUrl`
2403
+ before the test runs. If the server fails to start, the `dal` task ends with an
2404
+ infrastructure failure. A service that declares only a database still starts no
2405
+ server.
2406
+
2407
+ Every task gets `BASE_URL` when its service has a URL and `DATABASE_URL` when
2408
+ its service has a database, on every engine. Tasks other than k6 `dal` used to
2409
+ get `DATABASE_URL` only from declared `env`, and k6 `dal` tasks never got
2410
+ `BASE_URL`. The provisioned `DATABASE_URL` replaces one that the service's
2411
+ `env` declares.
2412
+
2413
+ k6 and Playwright files run without a URL or a database. A file that checks
2414
+ constants, or a Playwright test that builds its page with `page.setContent()`,
2415
+ passes. A file that reads a value its service does not provide fails where it
2416
+ reads it, for example with `BASE_URL env var required` from k6 or
2417
+ `Cannot navigate to invalid URL` from `page.goto("/")`. Testkit records these as
2418
+ product failures.
2419
+
2420
+ Without a URL, a Playwright test gets no `BASE_URL`, and
2421
+ `readPlaywrightContext().baseUrl` is `null`. `PlaywrightTestContext.baseUrl` is
2422
+ now typed `string | null`. Testkit also leaves `use.baseURL` from your
2423
+ `playwright.config.*` in place. With a URL, Testkit's `use.baseURL` still
2424
+ replaces yours.
2425
+
2426
+ `dal` files of one service now run at the same time. Every `dal` task used to
2427
+ take a `database:<service>` lock. The lock kept two `dal` files of one service
2428
+ from overlapping, but not a `dal` file and another test on the same database. If
2429
+ two files share rows that one of them changes, give both the same `locks` entry
2430
+ in `defineFile`.
2431
+
2432
+ ## Test runs
2433
+
2434
+ - `--file-timeout-seconds` now sets each file's timeout. Testkit used to record
2435
+ the flag's value in the results and run each file under the timeout from
2436
+ `testkit.config.ts` (#212).
2437
+ - A task's lease directory, `TESTKIT_LEASE_DIR`, is now under
2438
+ `.testkit/_execution/leases/`. Engine workspaces and the compilation cache
2439
+ are under `.testkit/_execution/node_modules/testkit-engine/`. Testkit removes
2440
+ them when the task or the run ends.
2441
+ - Test processes no longer get `TESTKIT_RUNTIME_ID`. The lease ID still
2442
+ identifies each task. Setup steps and servers that Testkit starts still get
2443
+ `TESTKIT_RUNTIME_ID`.
2444
+
2445
+ # 0.4.1
2446
+
2447
+ ## Bug filing, campaigns, and CI reports
2448
+
2449
+ Testkit now files confirmed bugs as issues on GitHub or Gitea and reports runs
2450
+ there. None of it needs an account or a server.
2451
+
2452
+ - `testkit bug guide` prints the bug record format. `testkit bug record` checks
2453
+ a record and stores a draft. `testkit bug file` runs the recorded test, and
2454
+ files the issue when the test fails.
2455
+ - `testkit campaign start`, `status`, and `finish` give a coding agent a target,
2456
+ list what it recorded, and write the summary for the pull request that adds
2457
+ the tests.
2458
+ - `testkit ci report` posts the last run as one commit status, the job's step
2459
+ summary, and one pull request comment that it updates in place.
2460
+ - `areas` in `testkit.config.ts` maps source paths to the product's areas. Each
2461
+ filed issue gets an `area:<id>` label.
2462
+ - `bugs.knownFailures` decides what a filed bug does to a run. The default,
2463
+ `"fail"`, keeps the gate as it was. With `"allow"`, a run passes when an open
2464
+ issue with the `testkit` label covers every failure.
2465
+
2466
+ The package now ships `docs/bug-records.md`, the reference for the issue
2467
+ format, the facts block, the known-failure rule, the commit status, and the
2468
+ pull request comment.
2469
+
2470
+ Run results under `.testkit/results/` gain failure details.
2471
+
2472
+ - Vitest. An error outside a test is a failure detail of kind `vitest-error`.
2473
+ A file that fails for a reason other than its failed tests keeps its engine
2474
+ failure beside the details of those tests. That reason is a timeout, too much
2475
+ output, a signal, or an exit code other than 1.
2476
+ - Playwright. A `playwright-spec` detail's key now holds the names of the
2477
+ `describe` blocks before the spec's title. A spec that fails in a file other
2478
+ than the selected one is a detail of kind `playwright-other-file-spec`. A
2479
+ `PLAYWRIGHT_SPECS_FAILED` failure carries `runStatus`, `failOnFlakyTests`,
2480
+ `reportErrors`, `attachmentFailures`, `evidenceErrors`, and `exitCode` or
2481
+ `exitSignal` in its details.
2482
+ - k6. A failed assertion's detail carries `labelled`, which says whether the
2483
+ script gave the assertion a label. Two assertions with one key and two titles
2484
+ are two details. A `K6_THRESHOLDS_FAILED` failure carries `failedChecks` and
2485
+ `exitCode` or `exitSignal`. A `K6_RUNTIME_EXCEPTION` failure carries
2486
+ `summaryRead`, `exitCode` or `exitSignal`, and `phase` when the suite threw
2487
+ during `setup` or `exec`. A script that does not load carries no phase.
2488
+
2489
+ A Playwright file whose specs a helper module declares now runs when the helper
2490
+ has the file's own extension. Testkit matched a spec to its file by the module
2491
+ that called `test()`, so such a file failed with
2492
+ `Selected Playwright task did not execute any tests`. A file whose every spec
2493
+ comes from one helper with another extension, such as a `.ts` file and a `.js`
2494
+ helper, still fails that way. Playwright takes the helper for the file's source
2495
+ and leaves the file out of the run.
2496
+
2497
+ # 0.4.0
2498
+
2499
+ ## Testkit Cloud commands removed
2500
+
2501
+ This release removes the `testkit cloud` commands, the `cloud` config key, and
2502
+ the `docs/cloud-control.md` and `docs/testkit-cloud-boundary.md` references.
2503
+ Testkit no longer publishes runs, findings, or campaign reports to Testkit
2504
+ Cloud. `testkit cloud` with any arguments now exits with status 2 and an error
2505
+ that names this release. The package no longer depends on `undici` or `yazl`.
2506
+
2507
+ Testkit ignores a `cloud` key that remains in `testkit.config.ts`.
2508
+ `defineConfig` still accepts the key when the config sets other fields, so
2509
+ `testkit typecheck` passes. A config object annotated as `TestkitConfig` gets a
2510
+ TypeScript unknown-property error. Delete the key.
2511
+
2512
+ Run bundles under `.testkit/results/` are unchanged. `testkit run` seals them
2513
+ and `testkit artifacts` reads them as before, and `artifacts.publication` still
2514
+ copies sealed bundles to a local directory. This is a breaking removal.
2515
+
2516
+ # 0.2.7
2517
+
2518
+ Playwright body attachments above 768 KiB now retain their bytes and test
2519
+ results. Testkit stores attachment bodies in private files and applies the
2520
+ existing 64 MiB individual and 256 MiB aggregate limits to body and path
2521
+ attachments together. Rejected attachments retain named errors alongside spec
2522
+ outcomes, assertion details, and accepted neighboring attachments. Temporary
2523
+ attachment files are removed after collection, including failed and cancelled
2524
+ runs. Text-secret redaction and binary-secret rejection still apply when sealing
2525
+ run evidence.
2526
+
2527
+ Fixed in [gfdelarue/testkit#154](https://github.com/gfdelarue/testkit/pull/154).
2528
+
2529
+ # 0.2.6
2530
+
2531
+ `testkit run --json` now emits one valid JSON document when services or test
2532
+ files are skipped. Human progress messages no longer appear before the result.
2533
+ Run IDs, exit codes, and structured skip information remain available to CI
2534
+ consumers. Compact and debug output retain their existing messages.
2535
+
2536
+ Fixed in [gfdelarue/testkit#147](https://github.com/gfdelarue/testkit/pull/147).
2537
+
2538
+ # 0.2.5
2539
+
2540
+ This release includes the changes merged in
2541
+ [gfdelarue/testkit#134](https://github.com/gfdelarue/testkit/pull/134),
2542
+ [gfdelarue/testkit#135](https://github.com/gfdelarue/testkit/pull/135), and
2543
+ [gfdelarue/testkit#136](https://github.com/gfdelarue/testkit/pull/136).
2544
+ Database schema preparation and Docker commands share their existing operations.
2545
+ Archive handling removes the unused Tar extraction path and shares accounting
2546
+ across the remaining formats.
2547
+
2548
+ ## Embedded OIDC provider removed
2549
+
2550
+ The `auth` config export, including `auth.providers.oidc()`, and the
2551
+ `@gfdelarue/testkit/config/oidc-provider` and
2552
+ `@gfdelarue/testkit/config/oidc-provider-runtime` subpaths are removed.
2553
+ Projects that need a test identity provider must supply one through ordinary
2554
+ managed services and configure their applications to use it. Testkit no longer
2555
+ implements an identity-provider server.
2556
+
2557
+ Consumer-owned API and browser authentication helpers remain supported.
2558
+ This is a breaking removal; external package consumers have not been inventoried.
2559
+
2560
+ # 0.2.4
2561
+
2562
+ The k6 engine now installs from public npm packages. The root manifest pins the
2563
+ five `@gfdelarue/testkit-engine-k6-<target>` packages at 1.0.0, and npm selects
2564
+ the one matching the host together with the shared
2565
+ `@gfdelarue/testkit-engine-k6-source` package. The packages are built and
2566
+ published with provenance from
2567
+ [gfdelarue/testkit-k6](https://github.com/gfdelarue/testkit-k6) on their own
2568
+ version line; the executable remains k6 1.7.1 with xk6-sql. Testkit no longer
2569
+ vendors binaries or builds k6, and installing it no longer needs registry
2570
+ credentials. The resolver checks the installed package's manifest and probes
2571
+ the executable once; a version other than the pinned one fails with
2572
+ `TESTKIT_K6_ENGINE_VERSION_MISMATCH`. Installing with `--omit=optional` is
2573
+ unsupported. The 0.2.x engine packages are deprecated.
2574
+
2575
+ `database.postgres` and `database.fixture` accept `roles`, an array of
2576
+ `{ name, login: false }` declarations. Testkit creates these local roles before
2577
+ source-schema replay, so imported policies can reference roles absent from a
2578
+ fresh cluster. Existing roles must match the ordinary NOLOGIN role contract,
2579
+ including on template cache hits. Conflicts stop provisioning without altering
2580
+ existing roles. Roles persist until the owned PostgreSQL volume is destroyed.
2581
+ This fixes [gfdelarue/testkit#129](https://github.com/gfdelarue/testkit/issues/129).
2582
+
2583
+ ## `auth.fixture` removed
2584
+
2585
+ `@gfdelarue/testkit/config` no longer exports `auth.fixture`, `auth.contracts`,
2586
+ `auth.topologies`, or `auth.profile`. `auth.providers.oidc` remains. The
2587
+ generated-actor profile descriptors, the `auth-fixture` k6 wire kind, and the
2588
+ `__testkitAuthProfile` suite metadata are removed with them. k6 repository
2589
+ profiles are JSON-only; a profile that carries a function fails validation.
2590
+
2591
+ Move signup, login, and header projection into the k6 suite module or a shared
2592
+ `__testkit__/helpers/` module. `defineHttpSuite({ auth: { setup, headers },
2593
+ rawHeaders })` keeps the session bundle, `actor`, `actors`, and `req.as(...)`
2594
+ runtime. Node/Vitest data profiles keep in-process `auth.setup`,
2595
+ `auth.setupAsync`, `headers`, and `rawHeaders` functions on
2596
+ `profiles.http.<name>`. Testkit 0.6.0 removed HTTP profiles; see
2597
+ [Remove the data test type](#remove-the-data-test-type).
2598
+
2599
+ This is a breaking removal. The migrated consumers are the two in-repository
2600
+ system fixtures. External consumers of the retired API have not been inventoried.
2601
+
2602
+ Local service commands on Windows keep their quoted executable paths and
2603
+ arguments, and Testkit creates its temporary directories on Windows even when
2604
+ the path exceeds `MAX_PATH`. Sensitive-text redaction runs in linear time on
2605
+ long input.
2606
+
2607
+ The undocumented `TESTKIT_PSQL_IMAGE` and `TESTKIT_PG_DUMP_IMAGE` aliases and
2608
+ the `TESTKIT_OIDC_CONFIG_JSON` override are removed. Use the documented
2609
+ `TESTKIT_POSTGRES_CLIENT_IMAGE`; the OIDC provider is configured only through
2610
+ the repository configuration. `TESTKIT_CLOUD_URL`, `TESTKIT_SETUP_NODE_OPTIONS`,
2611
+ and the Playwright `TESTKIT_FRONTEND_BASE_URL` and `TESTKIT_BACKEND_BASE_URL`
2612
+ fallbacks are now documented in the README.
2613
+
2614
+ # 0.2.3
2615
+
2616
+ Configured commands on Windows preserve quoted paths and arguments when
2617
+ invoking `cmd.exe`. TypeScript builds can now write prepared output directories
2618
+ without receiving literal quote characters in `--outDir`.
2619
+
2620
+ `testkit typecheck` supports consumer TypeScript 5.9.3 and 6.0.2. Generated
2621
+ configurations clear the deprecated `baseUrl` option while preserving inherited
2622
+ aliases and Testkit's type mappings. This fixes the TS5101 failure reported in
2623
+ [gfdelarue/testkit#117](https://github.com/gfdelarue/testkit/issues/117), including
2624
+ consumers with a separate frontend compiler.
2625
+
2626
+ Database state scans now enumerate only the directories Testkit owns under
2627
+ `.testkit` (`_infra/local-postgres`, `_dbcache`, graph runtime services, and
2628
+ top-level service directories carrying a `database_backend` marker). Unrelated
2629
+ content beneath `.testkit`, such as a consumer virtualenv or the managed Node
2630
+ toolchain's own `npm` links, no longer fails run start or resource cleanup with
2631
+ `Refusing unsafe Testkit database state symlink`. Container and volume
2632
+ references are read only from owned state, so stray files elsewhere no longer
2633
+ keep stale resources alive. This addresses
2634
+ [gfdelarue/testkit#118](https://github.com/gfdelarue/testkit/issues/118).
2635
+
2636
+ Local services accept `local.shutdownTimeoutMs`, a positive finite integer in
2637
+ milliseconds that defaults to 1000. `app.node()` and `app.next()` forward the
2638
+ same option. On POSIX, shutdown sends SIGTERM immediately and SIGKILL only when
2639
+ the allowance expires. Cancellation, repeated stop requests, descendant cleanup,
2640
+ and maintenance recovery share the first deadline. Windows retains forceful
2641
+ `taskkill /T /F`. This fixes
2642
+ [gfdelarue/testkit#116](https://github.com/gfdelarue/testkit/issues/116).
2643
+
2644
+ `stop`, `run stop`, active `cleanup`, and `destroy` let the controller finish
2645
+ service shutdown before escalating. After SIGINT, they wait at least two
2646
+ seconds and until outstanding service deadlines plus three seconds for forced
2647
+ termination have elapsed. They then send the controller SIGTERM, allow five
2648
+ seconds for finalization, and use SIGKILL if it remains active. A controller
2649
+ with no registered services retains the two-second grace and five-second
2650
+ finalization windows.
2651
+
2652
+ The `.scenario.testkit.ts` category, `defineScenarioSuite()` and `--seed` are
2653
+ removed. Use `.e2e.testkit.ts` for product journeys or `.int.testkit.ts` for
2654
+ service checks, and replace the scenario choice/resource/step helpers with
2655
+ ordinary test code. `defineHttpSuite()` remains available. Native k6
2656
+ `options.scenarios`, executors, groups and lifecycle hooks are unchanged.
2657
+
2658
+ Discovery ignores the retired suffix, so migrate those files before upgrading;
2659
+ a mixed run can still pass using its remaining supported files. Saved scenario
2660
+ results remain readable. New run documents omit `scenarioSeed`, and provenance
2661
+ omits `scenario.seed` while retaining its generic run-selection fields.
2662
+
2663
+ # 0.2.2
2664
+
2665
+ PostgreSQL clients for Testkit-owned databases now join the database container's
2666
+ network namespace. Template rebuilds, schema checks and snapshots, and database
2667
+ steps no longer depend on Docker host networking reaching a published port.
2668
+ This addresses the failure reported with Docker Desktop and WSL2 in
2669
+ [gfdelarue/testkit#111](https://github.com/gfdelarue/testkit/issues/111).
2670
+ External `sourceSchema` database URLs retain their existing routing.
2671
+
2672
+ # 0.2.1
2673
+
2674
+ This release adds explicit Cloud campaign reporting and optional local
2675
+ verification policy. It also includes the configuration changes below.
2676
+ Consumers pinned to 0.2.0 keep their existing behavior until they upgrade.
2677
+
2678
+ The pinned Vitest version is 4.1.11, which fixes the `@vitest/mocker` path
2679
+ traversal advisory GHSA-82fw-gwwq-j7x9.
2680
+
2681
+ ## Next app Clerk preset
2682
+
2683
+ `app.next` no longer accepts an `auth` option. The only value it took,
2684
+ `"disabled-clerk"`, set six product-private environment variables and was never
2685
+ documented. Put the variables in `env.values` instead:
2686
+
2687
+ ```ts
2688
+ app.next({
2689
+ port: 3000,
2690
+ env: {
2691
+ values: {
2692
+ DISABLE_CLERK_AUTH: "1",
2693
+ NEXT_PUBLIC_DISABLE_CLERK_AUTH: "1",
2694
+ NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: "",
2695
+ CLERK_PUBLISHABLE_KEY: "",
2696
+ CLERK_SECRET_KEY: "",
2697
+ CLERK_WEBHOOK_SECRET: "",
2698
+ },
2699
+ },
2700
+ });
2701
+ ```
2702
+
2703
+ ## Fly.io request routing header
2704
+
2705
+ The k6 and Node/Vitest HTTP runtimes no longer read `MACHINE_ID` from the
2706
+ environment or add a `fly-force-instance-id` header to requests. The Fly.io
2707
+ infrastructure was removed earlier and nothing documented the header. The
2708
+ `routeParams` field on `RuntimeEnv`, the `routeHeaders` option on the k6
2709
+ `createHttpClient`, and the matching positional argument on `makeReq` and
2710
+ `makeRawReq` are removed with it. A test that needs a routing header sets it
2711
+ through `headers` or `rawHeaders` on its HTTP profile.
2712
+
2713
+ ## Lint rules are booleans
2714
+
2715
+ `testkit lint` no longer has severity levels or ratchet mode. Each rule is
2716
+ enabled or disabled, enabled-rule findings fail the command, and results depend
2717
+ only on current source. `lint.level` and the rule levels `off`, `warn`,
2718
+ `ratchet`, and `strict` are rejected with migration guidance. Replace them with
2719
+ `true`, `false`, or `{ enabled: boolean }` alongside the existing `include`,
2720
+ `exclude`, `allowed`, and `trackedOnly` options:
2721
+
2722
+ ```ts
2723
+ lint: {
2724
+ rules: {
2725
+ dalParallelSafety: false,
2726
+ unitTestBoundaries: { include: ["src/**/*.test.ts"] },
2727
+ },
2728
+ }
2729
+ ```
2730
+
2731
+ Results and `.testkit/lint/latest.json` drop `level`, `severity`,
2732
+ `ratchetStatus`, `fingerprint`, `notes`, and the warning and debt counts.
2733
+ `base.json` and `ratchet.json` are no longer written. The `TestkitLintLevel`
2734
+ type is removed. Unit-test boundary findings now report every source location
2735
+ instead of one per file and operation.
2736
+
2737
+ ## Environment overrides
2738
+
2739
+ Three undocumented environment variables are no longer read:
2740
+
2741
+ - `TESTKIT_PSQL_IMAGE` and `TESTKIT_PG_DUMP_IMAGE` were aliases for the
2742
+ documented `TESTKIT_POSTGRES_CLIENT_IMAGE`. Rename them.
2743
+ - `TESTKIT_OIDC_CONFIG_JSON` let the environment replace the OIDC provider
2744
+ config, including client secrets and users, at launch. Configure the provider
2745
+ explicitly instead. The embedded provider is removed in 0.2.5;
2746
+ see [Embedded OIDC provider removed](#embedded-oidc-provider-removed).
2747
+
2748
+ `TESTKIT_CLOUD_URL`, `TESTKIT_SETUP_NODE_OPTIONS`, and the Playwright
2749
+ `TESTKIT_FRONTEND_BASE_URL` and `TESTKIT_BACKEND_BASE_URL` fallbacks keep their
2750
+ behavior and are now documented in the README and config schema.
2751
+
2752
+ ## Playwright typecheck support roots
2753
+
2754
+ `testkit typecheck` no longer evaluates Playwright configuration or traverses a
2755
+ separate dependency/provenance graph. It runs native TypeScript over engine-profiled
2756
+ projects. Add `tsconfig.playwright.json` with explicit `files`/`include` entries for
2757
+ string-named setup, teardown, reporter, fixture, and support-project modules.
2758
+ Discovered UI suites and the selected Playwright config remain implicit roots.
2759
+ See the [support-file contract](engines.md#playwright-support-files).
2760
+ Managed runtime enforcement remains authoritative; direct import lint feedback
2761
+ moves to ESLint under #22.
2762
+
2763
+ ## Data-profile Vitest imports
2764
+
2765
+ Data suites now receive restricted Vitest runtime exports. Native case
2766
+ registration is rejected through helpers and aliases as well as direct imports.
2767
+ Data imports omit runner classes, Vitest runtime subpaths, and Testkit contract
2768
+ registration. A native collection check also rejects cases obtained through
2769
+ `import.meta.vitest` or the native global. Assertions, mocks, hooks, and root type-only imports
2770
+ remain supported.
2771
+
2772
+ `testkit typecheck` reports data suites under `node-vitest-data` and checks them
2773
+ separately from contract suites sharing the same tsconfig. Direct-import lint
2774
+ feedback remains; managed imports and native collection checks replace
2775
+ whole-program provenance analysis.
2776
+
2777
+ ## UI authentication fixtures
2778
+
2779
+ The product-specific UI authentication API is deprecated and removed in the next
2780
+ release. Remove `ui.auth(...)` and its authentication configuration from
2781
+ `testkit.config.ts`. The provisioning, login, session-storage, authenticated-shell,
2782
+ and provisioned-sandbox exports from `@gfdelarue/testkit/playwright` are removed.
2783
+ Their identity, session, profile, and sandbox types are removed too.
2784
+
2785
+ Move signup, login, SQL seeding, session extraction, local storage, and
2786
+ application readiness into your own Playwright fixtures. Use native
2787
+ `test.extend`, request contexts, page locators, and browser storage APIs.
2788
+ Testkit retains `createUiSandbox`, runtime context, target safety, and settling
2789
+ helpers. Put generic `frontendBaseUrl`, `backendBaseUrl`, and `allowRemote`
2790
+ settings directly under `ui`, or pass them as sandbox options.
2791
+ The separate `auth.fixture` API is removed too; see above.
2792
+
2793
+ ## Configuration metadata
2794
+
2795
+ Auth and Node toolchain builders keep their existing names and options. Their
2796
+ returned objects no longer carry the redundant `json-session`, `auth-topology`,
2797
+ `testkit-oidc-provider`, or `node` kind fields. Remove reads of those metadata
2798
+ fields and omit `kind: "node"` from inline toolchain configuration. The
2799
+ `auth.topologies` builders are removed with `auth.fixture` (see above).
2800
+
2801
+ Existing database state directories and verified Node toolchain caches remain
2802
+ usable. The database state marker keeps its filename and no longer stores a
2803
+ backend value. Node toolchain environment variables and cache fingerprints
2804
+ retain their existing values.
2805
+
2806
+ ## Local status and history removal
2807
+
2808
+ Testkit now keeps immutable run bundles locally and publishes selected bundles
2809
+ explicitly to Testkit Cloud. These flags and outputs are removed:
2810
+
2811
+ - `testkit run --write-status` and `--allow-partial-status`. Both are unknown flags.
2812
+ - The product-root `testkit.status.json` snapshot and `run` JSON's `statusArtifact` field.
2813
+ - `.testkit/history/tests.json`, its per-test counts, durations, timestamps, and last status.
2814
+ - Discovery JSON's `history`, `files[].history`, and `summary.latestResults` fields.
2815
+ Compact and verbose discovery output no longer render history or pass rates.
2816
+ - Workspace JSON's `history`, `discovery.history`, and
2817
+ `discovery.summary.latestResults` fields. Targets no longer have source `history`;
2818
+ discovery file targets no longer receive a last status from history.
2819
+ - Run planning's `tasks[].historyKey` and the `history` estimate source.
2820
+
2821
+ Remove the retired flags from scripts. Consumers of a single run should verify
2822
+ its bundle at `.testkit/results/runs/<run-id>/`. Consumers of cross-run counts,
2823
+ last outcomes, durations, or trends should use Testkit Cloud. The `status`
2824
+ command still reports current runtime and process state.
2825
+
2826
+ `testkit run` prints the sealed run ID and returns it as `runId` with `--json`,
2827
+ including failed runs. Use `testkit cloud publish --run <run-id>` to publish
2828
+ that exact bundle. Publication has its own exit status. See the
2829
+ [README CI recipe](../README.md) for preserving the test outcome while
2830
+ publishing evidence. The newest-bundle default remains available interactively.
2831
+
2832
+ Old `.testkit/history/` files are ignored and left untouched. Testkit does not
2833
+ migrate their contents. `.testkit/timings.json` and its schema 2 are unchanged.
2834
+ Without usable timings, the scheduler uses a default estimate until execution
2835
+ records a timing. Existing status files are no longer excluded from repository
2836
+ fingerprints; remove them from the product repository if they are obsolete.
2837
+
2838
+ Run schema 19, bundle schema 2, provenance schema 3, and retention schema 1 are
2839
+ unchanged. Retention still prunes bundles and unreferenced objects but no longer
2840
+ appends the write-only `decisions.jsonl` ledger. Legacy ledger metadata in a
2841
+ supplied bundle remains readable and validated when present.
2842
+
2843
+ Cloud receipts follow the current publication API: `decisionReasons` contains
2844
+ `{ ruleId, message }` objects, and `policyId` plus `policyRevision` identify the
2845
+ evaluated policy. They replace the client's stale string reasons and
2846
+ `policyVersion` fields. Text publication output prints each reason's message.