@gfdelarue/testkit 0.5.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (215) hide show
  1. package/README.md +590 -165
  2. package/bin/testkit-windows-owner-arm64.exe +0 -0
  3. package/bin/testkit-windows-owner-x64.exe +0 -0
  4. package/dist/app/bugs/batch.js +1 -0
  5. package/dist/app/bugs/ci-report.js +2 -2
  6. package/dist/app/bugs/ci-summary.js +3 -3
  7. package/dist/app/bugs/draft.js +2 -2
  8. package/dist/app/bugs/file.js +1 -1
  9. package/dist/app/bugs/guide.js +1 -1
  10. package/dist/app/bugs/record.js +2 -2
  11. package/dist/app/bugs/run-reading.js +1 -1
  12. package/dist/app/bugs/store.js +1 -1
  13. package/dist/app/campaigns/index.js +2 -2
  14. package/dist/app/campaigns/summary.js +1 -1
  15. package/dist/app/cleanup/index.js +1 -1
  16. package/dist/app/db-schema/prepare.js +1 -1
  17. package/dist/app/db-schema/refresh.js +1 -1
  18. package/dist/app/db-schema/verify.js +1 -1
  19. package/dist/app/destroy/index.js +1 -1
  20. package/dist/app/discover/index.js +2 -2
  21. package/dist/app/environments/config-driven.js +1 -1
  22. package/dist/app/environments/file-setup.js +1 -1
  23. package/dist/app/environments/routing.js +1 -1
  24. package/dist/app/environments/static-values.js +1 -1
  25. package/dist/app/environments/supplied.js +1 -1
  26. package/dist/app/inspect/index.js +1 -1
  27. package/dist/app/lint/native.js +1 -1
  28. package/dist/app/lint/options.js +1 -1
  29. package/dist/app/lint/shared.js +1 -1
  30. package/dist/app/run/engine-orchestration.js +1 -1
  31. package/dist/app/run/environment.js +1 -1
  32. package/dist/app/run/live-run.js +1 -1
  33. package/dist/app/run/orchestrator.js +1 -1
  34. package/dist/app/run/planning.js +1 -1
  35. package/dist/app/run/request.js +1 -1
  36. package/dist/app/run/results.js +1 -1
  37. package/dist/app/run/run-finalization.js +1 -1
  38. package/dist/app/run/run-finisher.js +1 -0
  39. package/dist/app/run/setup/runtime-node-setup-protocol.js +1 -1
  40. package/dist/app/run/setup/runtime-node-setup-worker.js +1 -1
  41. package/dist/app/run/setup/runtime-node-setup.js +2 -2
  42. package/dist/app/run/task-batches.js +1 -1
  43. package/dist/app/run/worker-loop.js +1 -1
  44. package/dist/app/status/model.js +1 -1
  45. package/dist/app/status/runs.js +1 -1
  46. package/dist/app/stop/index.js +1 -1
  47. package/dist/app/tracker/index.js +1 -1
  48. package/dist/app/tracker/token.js +2 -2
  49. package/dist/app/typecheck/index.js +4 -4
  50. package/dist/cli/command-catalog.js +1 -1
  51. package/dist/cli/command-flags.js +1 -1
  52. package/dist/cli/commands/artifacts.js +2 -2
  53. package/dist/cli/commands/bug/file.js +3 -2
  54. package/dist/cli/commands/discover.js +1 -1
  55. package/dist/cli/commands/guard/unsafe-entrypoint.js +1 -1
  56. package/dist/cli/commands/run.js +2 -2
  57. package/dist/cli/commands/typecheck.js +1 -1
  58. package/dist/cli/commands/workspace.js +1 -1
  59. package/dist/cli/framework/help.js +5 -5
  60. package/dist/cli/framework/parser.js +1 -1
  61. package/dist/cli/json-errors.js +1 -1
  62. package/dist/cli/renderers/bug/text.js +1 -1
  63. package/dist/cli/renderers/run/formatting.js +1 -1
  64. package/dist/config/authoring/database-fixture-discovery.js +1 -0
  65. package/dist/config/authoring/index.js +1 -1
  66. package/dist/config/loader/config-loader.js +1 -1
  67. package/dist/config/loader/decode-discovery.js +1 -1
  68. package/dist/config/loader/discover-tests.js +1 -1
  69. package/dist/config/loader/discovery-result.js +1 -0
  70. package/dist/config/loader/discovery.js +2 -2
  71. package/dist/config/loader/file-metadata.js +1 -1
  72. package/dist/config/loader/groups.js +2 -1
  73. package/dist/config/loader/index.js +1 -1
  74. package/dist/config/loader/load.js +1 -1
  75. package/dist/config/loader/managed-configs.js +1 -1
  76. package/dist/config/loader/owned-discovery.js +1 -1
  77. package/dist/config/loader/runtime-isolation.js +1 -1
  78. package/dist/config/loader/skip-config.js +1 -1
  79. package/dist/config/loader/test-selection.js +1 -1
  80. package/dist/database/admin.js +5 -2
  81. package/dist/database/call-caps.js +1 -0
  82. package/dist/database/cleanup.js +1 -1
  83. package/dist/database/configured-steps.js +1 -1
  84. package/dist/database/constants.js +1 -1
  85. package/dist/database/docker-endpoint.js +1 -0
  86. package/dist/database/docker-metadata.js +2 -2
  87. package/dist/database/docker.js +5 -5
  88. package/dist/database/index.js +2 -1
  89. package/dist/database/lifecycle-cleanup.js +1 -1
  90. package/dist/database/local-postgres.js +4 -1
  91. package/dist/database/naming.js +1 -1
  92. package/dist/database/operation-deadline.js +1 -0
  93. package/dist/database/postgres/connection-check.js +1 -1
  94. package/dist/database/postgres/index.js +2 -2
  95. package/dist/database/postgres/owned-exec.js +27 -0
  96. package/dist/database/postgres/protocol.js +2 -2
  97. package/dist/database/provider.js +1 -0
  98. package/dist/database/roles.js +2 -2
  99. package/dist/database/schema-operations.js +5 -1
  100. package/dist/database/schema-source.js +3 -3
  101. package/dist/database/steps.js +1 -1
  102. package/dist/database/template-steps.js +4 -4
  103. package/dist/domain/identifiers.js +1 -1
  104. package/dist/domain/retired-test-types.js +1 -1
  105. package/dist/domain/suite-selection.js +1 -1
  106. package/dist/domain/test-types.js +1 -1
  107. package/dist/engines/adapters/evidence-files.js +1 -1
  108. package/dist/engines/adapters/execution.js +1 -1
  109. package/dist/engines/catalog.js +1 -1
  110. package/dist/engines/definitions.js +1 -1
  111. package/dist/engines/environment/index.js +1 -1
  112. package/dist/engines/installations/node-vitest.js +1 -1
  113. package/dist/engines/installations/playwright.js +1 -1
  114. package/dist/engines/managed-module-resolution.js +6 -2
  115. package/dist/engines/node-vitest/adapter/adapter.js +1 -1
  116. package/dist/engines/node-vitest/adapter/driver.js +2 -2
  117. package/dist/engines/node-vitest/adapter/evidence.js +2 -2
  118. package/dist/engines/node-vitest/adapter/workspace.js +8 -24
  119. package/dist/engines/node-vitest/inventory/index.js +2 -2
  120. package/dist/engines/node-vitest/inventory/list-files.js +3 -3
  121. package/dist/engines/node-vitest/runtime/data/index.js +1 -1
  122. package/dist/engines/node-vitest/runtime/data/report.js +1 -1
  123. package/dist/engines/node-vitest/runtime/data/scenario.js +2 -2
  124. package/dist/engines/node-vitest/runtime/runtime-input.js +1 -1
  125. package/dist/engines/playwright/adapter/adapter.js +1 -1
  126. package/dist/engines/playwright/adapter/browser-network-evidence.js +4 -0
  127. package/dist/engines/playwright/adapter/config.js +16 -6
  128. package/dist/engines/playwright/adapter/evidence.js +2 -1
  129. package/dist/engines/playwright/adapter/failure-details.js +2 -2
  130. package/dist/engines/playwright/adapter/reporter.js +1 -1
  131. package/dist/engines/playwright/runtime/browser-network.js +1 -0
  132. package/dist/engines/playwright/runtime/browser-traffic.js +1 -0
  133. package/dist/engines/playwright/runtime/chromium-environment.js +1 -0
  134. package/dist/engines/playwright/runtime/emulation.js +1 -0
  135. package/dist/engines/playwright/runtime/index.js +1 -1
  136. package/dist/engines/playwright/runtime/managed-test.js +1 -0
  137. package/dist/engines/playwright/runtime/runtime-input.js +2 -2
  138. package/dist/environment/lifecycle-controller.js +1 -1
  139. package/dist/environment/lifecycle.js +1 -1
  140. package/dist/environment/maintenance.js +1 -1
  141. package/dist/environment/run-environment.js +1 -1
  142. package/dist/environment/run-stopping.js +1 -1
  143. package/dist/environment/runtime-manager.js +1 -1
  144. package/dist/evidence/artifact-config.js +1 -1
  145. package/dist/evidence/artifacts.js +5 -2
  146. package/dist/evidence/bundles.js +2 -1
  147. package/dist/evidence/compressed-artifacts.js +1 -0
  148. package/dist/evidence/environment-record.js +1 -1
  149. package/dist/evidence/index.js +1 -1
  150. package/dist/evidence/live-run-summary.js +1 -0
  151. package/dist/evidence/local-paths.js +1 -0
  152. package/dist/evidence/maintenance.js +1 -0
  153. package/dist/evidence/operation-timings.js +1 -1
  154. package/dist/evidence/read-run.js +1 -1
  155. package/dist/evidence/redaction.js +4 -4
  156. package/dist/evidence/run-evidence.js +1 -0
  157. package/dist/evidence/setup-progress.js +1 -1
  158. package/dist/evidence/storage-inventory.js +1 -1
  159. package/dist/evidence/storage.js +2 -2
  160. package/dist/kernel/known-keys.js +1 -0
  161. package/dist/kernel/redaction-targets.js +1 -1
  162. package/dist/kernel/redaction.js +7 -4
  163. package/dist/process/engine-environment-keys.js +1 -0
  164. package/dist/process/environment-overlay.js +1 -1
  165. package/dist/process/environment.js +1 -1
  166. package/dist/process/execa-bundle.json +3 -3
  167. package/dist/process/files/evidence-writes.js +1 -0
  168. package/dist/process/files/product-dir.js +1 -1
  169. package/dist/process/host-process.js +1 -1
  170. package/dist/process/index.js +1 -1
  171. package/dist/process/lifecycle-fence.js +1 -1
  172. package/dist/process/lock-progress.js +2 -0
  173. package/dist/process/locks.js +4 -4
  174. package/dist/process/process-identity.js +1 -1
  175. package/dist/process/process.js +35 -35
  176. package/dist/process/recovery.js +1 -1
  177. package/dist/process/runs/manifest.js +1 -1
  178. package/dist/process/runs/ownership.js +1 -1
  179. package/dist/process/runs/stop.js +1 -1
  180. package/dist/process/shutdown.js +1 -1
  181. package/dist/process/windows-native.js +1 -1
  182. package/dist/repo/fingerprint-policy.js +1 -1
  183. package/dist/repo/history.js +7 -4
  184. package/dist/repo/managed-git.js +1 -1
  185. package/dist/repo/state.js +4 -4
  186. package/dist/steps/configured-step-testkit-modules.js +1 -1
  187. package/dist/steps/environment.js +1 -1
  188. package/dist/toolchains/index.js +2 -2
  189. package/dist/toolchains/node-provisioning.js +2 -2
  190. package/dist/toolchains/vitest/index.js +1 -1
  191. package/dist/tracker/gitea.js +1 -1
  192. package/dist/tracker/github.js +1 -1
  193. package/dist/tracker/http.js +1 -1
  194. package/dist/tracker/ownership.js +1 -0
  195. package/dist/tracker/redaction.js +1 -1
  196. package/dist/tracker/responses.js +1 -1
  197. package/dist/types/config.d.ts +7 -86
  198. package/dist/types/environment.d.ts +6 -0
  199. package/dist/types/node-vitest-data.d.ts +2 -218
  200. package/dist/types/playwright.d.ts +21 -0
  201. package/dist/types/workspace.d.ts +30 -14
  202. package/docs/bug-records.md +19 -18
  203. package/docs/engines.md +115 -21
  204. package/docs/release-notes.md +2846 -0
  205. package/docs/test-types.md +19 -69
  206. package/package.json +15 -6
  207. package/testkit-config-schema.md +101 -170
  208. package/dist/app/lint/rules/data.js +0 -1
  209. package/dist/config/authoring/runtime.js +0 -1
  210. package/dist/config/loader/task-policy.js +0 -1
  211. package/dist/engines/node-vitest/runtime/contract/index.js +0 -1
  212. package/dist/engines/node-vitest/runtime/data/http.js +0 -1
  213. package/dist/engines/node-vitest/runtime/data/runner.js +0 -1
  214. package/dist/environment/lifecycle-artifacts.js +0 -1
  215. package/dist/types/node-vitest-contract.d.ts +0 -75
package/README.md CHANGED
@@ -43,19 +43,25 @@ npm run testkit
43
43
 
44
44
  The command exits zero only when at least one test file completed and every
45
45
  required file passed. Test failures, infrastructure failures, interrupted or
46
- not-run files, an all-skipped selection, and an empty selection all fail the
47
- gate. `npm run testkit -- --allow-empty` is an explicit escape hatch for a job
46
+ not-run files, a skipped file, and an empty selection all fail the gate. A file
47
+ that a `skip` rule or its own `skip` metadata skips fails the gate even when
48
+ every other file passes, and `--allow-empty` doesn't change that. `npm run testkit -- --allow-empty` is an explicit escape hatch for a job
48
49
  whose empty selection is intentional; it should not be the default CI command.
49
50
 
50
51
  Testkit writes the immutable, verifiable run bundle under
51
52
  `.testkit/results/runs/<run-id>/`. Upload `.testkit/results` with `if: always()`
52
- when CI evidence should survive a failed job.
53
+ when CI evidence should survive a failed job. If the job can kill Testkit before
54
+ it seals the run, run `testkit cleanup` before the upload, so the run is sealed
55
+ and redacted first. See
56
+ [Read run evidence without the CLI](#read-run-evidence-without-the-cli).
53
57
 
54
58
  ## Report a run to GitHub or Gitea
55
59
 
56
60
  `testkit ci report` reads the last run in the checkout and posts it in three
57
61
  places: one commit status, the job's step summary, and one pull request comment
58
- that it updates in place.
62
+ that it updates in place. It updates only a comment that its token's user
63
+ wrote, and posts a new comment when the tracker won't say who that user is. It
64
+ leaves out a CI run or issue link that holds credentials or a secret.
59
65
 
60
66
  ```yaml
61
67
  permissions:
@@ -79,10 +85,17 @@ steps:
79
85
  it, as the `env` lines above do. Gitea Actions accepts the same lines.
80
86
  - Testkit reads the token from `GITHUB_TOKEN` or `GH_TOKEN` for GitHub, and
81
87
  from `GITEA_TOKEN` for Gitea. In a Gitea Actions job it also reads
82
- `GITHUB_TOKEN`. With no token set, it asks `git credential fill`. The token
83
- that `actions/checkout` keeps in the checkout does not answer that request.
84
- - Testkit reaches GitHub Enterprise Server only from a GitHub Actions job on
85
- that server, so `testkit bug file` works there only in a job.
88
+ `GITHUB_TOKEN`. Outside such a job, it sends `GITEA_TOKEN` only to the host
89
+ that `GITEA_SERVER_URL` names, such as `https://gitea.example.com`. With no
90
+ token set, it asks `git credential fill`. The token that `actions/checkout`
91
+ keeps in the checkout does not answer that request.
92
+ - Testkit sends a token only over HTTPS. A plain `http://` remote or repository
93
+ URL works only for a loopback host such as `127.0.0.1`.
94
+ - A job's token reaches only the job's own repository. When `bugs.repository`
95
+ names another one, give the step a token that may write to it.
96
+ - Testkit reaches GitHub Enterprise Server, and GitHub Enterprise Cloud with
97
+ data residency (`*.ghe.com`), only from a GitHub Actions job on that host, so
98
+ `testkit bug file` works there only in a job.
86
99
  - GitHub gives a pull request from a fork a read-only token. `ci report` then
87
100
  writes the step summary and fails when it posts the status. Add
88
101
  `continue-on-error: true` to the step when the repository takes pull requests
@@ -98,7 +111,7 @@ steps:
98
111
  Testkit files a confirmed bug as an issue in the repository's GitHub or Gitea.
99
112
  The issue names the regression test that fails because of the bug, the commit
100
113
  that introduced the faulty lines, and the command that reproduces the failure.
101
- [Bug records](docs/bug-records.md) gives the issue format.
114
+ [`docs/bug-records.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/bug-records.md) gives the issue format.
102
115
 
103
116
  1. Write a Testkit test that fails because of the bug.
104
117
  2. Write the bug record as a JSON file. `testkit bug guide` prints the fields
@@ -110,8 +123,15 @@ that introduced the faulty lines, and the command that reproduces the failure.
110
123
 
111
124
  `testkit bug file` runs the test at the checked-out commit and files the issue
112
125
  only when the test fails. It stops when tracked files have uncommitted changes,
113
- when no remote branch contains the commit, or when an open issue already covers
114
- the test. The token needs permission to create issues and labels.
126
+ when no branch on the remote contains the commit, or when an open issue already
127
+ covers the test. It asks the remote which branches hold the commit, with the
128
+ checkout's Git credentials, so a branch that was rewound or deleted there
129
+ doesn't count. When it can't tell, because a remote doesn't answer or a shallow
130
+ clone's history stops short, it stops and says why. After it files, it reads the open issues again. If another checkout
131
+ filed the same test at the same time and its issue is older, Testkit closes its
132
+ own issue and stops, naming the older one. GitHub can list a new issue a few
133
+ seconds late, so Testkit waits up to 7.5 seconds for its own and says so when
134
+ it doesn't appear. The token needs permission to create issues and labels.
115
135
 
116
136
  `testkit bug record` saves the `--vitest-config`, `--vitest-project`, and
117
137
  `--vitest-runner` it was given in the draft, and asks for `--vitest-project`
@@ -130,6 +150,38 @@ fragment. It refuses `--database` and `--env`; pass those values through
130
150
  root, so a relative `--environment-file` needs `testkit bug file` run from
131
151
  there too.
132
152
 
153
+ `testkit bug file --batch` files several drafts in one call: the ids it is
154
+ given, or every unfiled draft under `.testkit/bugs/` when it is given none.
155
+
156
+ - It checks the checkout once and reads the open issues before it runs the
157
+ tests. It stops before it files anything when tracked files have uncommitted
158
+ changes, when no remote branch contains the commit, or when it cannot read
159
+ the open issues.
160
+ - It runs the drafts' tests in one `testkit run`, or one run for each
161
+ `--vitest-config`, `--vitest-project`, and `--vitest-runner` the drafts were
162
+ recorded with. Each draft is judged by its own file's result, so a file that
163
+ is skipped or fails outside its tests rules out only its own draft.
164
+ - It reads the open issues again before it files, in case another checkout
165
+ filed one of the tests meanwhile.
166
+ - It reports one line per draft: filed, already filed, rejected with the
167
+ reason, or not filed.
168
+ - A draft is rejected when its own checks fail, for example when its test
169
+ passes or an open issue already covers the test, and the batch carries on.
170
+ - When the tracker refuses a request, the batch stops filing and reports the
171
+ drafts it has not reached as not filed.
172
+ - It exits non-zero when any draft is rejected or not filed. `--json` returns
173
+ the same list.
174
+
175
+ Each draft records its issue as soon as the tracker returns it, so running the
176
+ batch again files only what is left. An open issue whose facts match the ones a
177
+ draft would file counts as that draft's issue, so a batch stopped between the
178
+ two steps does not file the bug twice. When the tracker creates an issue
179
+ without its labels, the batch stops, the draft stays unfiled, and Testkit
180
+ closes that issue. If it can't, the message says so, and you close the issue.
181
+ Then run the batch again with a token that may label issues. The batch creates
182
+ at most one issue a second. When the tracker limits the rate, it stops and
183
+ says when to run it again.
184
+
133
185
  ### Name the product's areas
134
186
 
135
187
  Add `areas` to `testkit.config.ts` so that each issue names the part of the
@@ -146,7 +198,7 @@ export default defineConfig({
146
198
 
147
199
  `testkit bug file` puts an `area:<id>` label on the issue. If no area claims
148
200
  the fault's file, it stops and names the file. Without `areas`, Testkit files
149
- bugs with no area. See [`areas`](testkit-config-schema.md#areas).
201
+ bugs with no area. See the `areas` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
150
202
 
151
203
  ### Pass a run whose failures are all filed
152
204
 
@@ -162,7 +214,8 @@ export default defineConfig({
162
214
  A run then passes when an open issue with the `testkit` label covers each
163
215
  failure. A new failure still fails the run, and closing an issue makes its
164
216
  test count again. When Testkit cannot read the issues, every failure is new.
165
- [Known failures](docs/bug-records.md#known-failures) gives the exact rule.
217
+ The Known failures section of [`docs/bug-records.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/bug-records.md) gives the
218
+ exact rule.
166
219
 
167
220
  With `"allow"`, the run reads the open issues. In CI, pass the job's token to
168
221
  the `testkit run` step as well:
@@ -174,7 +227,9 @@ steps:
174
227
  GITHUB_TOKEN: ${{ github.token }}
175
228
  ```
176
229
 
177
- The job also needs the `issues: read` permission.
230
+ The job also needs the `issues: read` permission. Testkit reads
231
+ `GITHUB_TOKEN`, `GH_TOKEN` and `GITEA_TOKEN` itself and doesn't pass them to
232
+ services or setup steps. A service that needs one declares it in its `env`.
178
233
 
179
234
  ### Run a campaign
180
235
 
@@ -264,8 +319,7 @@ the report to the chosen platform.
264
319
 
265
320
  Testkit has two built-in engines:
266
321
 
267
- - `node-vitest` runs plain Vitest test files and purpose-built `data` and
268
- `contract` files with Node and Vitest.
322
+ - `node-vitest` runs plain Vitest test files with Node and Vitest.
269
323
  - `playwright` runs `ui` files with Playwright and managed Chromium.
270
324
 
271
325
  A plain Vitest test file needs no Testkit suffix, directory, metadata, or
@@ -293,17 +347,16 @@ npx @gfdelarue/testkit setup playwright --with-deps # CI and fresh hosts
293
347
  ```
294
348
 
295
349
  Normal runs never install or download a browser.
296
- [Installation](docs/engines.md#installation) covers what an install contains,
297
- the supported platforms, and how to recover a broken one.
298
- [Typechecking](docs/engines.md#typechecking) covers the supported TypeScript
299
- versions and `tsconfig.playwright.json`.
350
+ The Installation section of [`docs/engines.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/engines.md) covers
351
+ what an install contains, the supported platforms, and how to recover a broken
352
+ one. Its Typechecking section covers the supported TypeScript versions and
353
+ `tsconfig.playwright.json`.
300
354
 
301
355
  Testkit bounds the output it captures from each command it runs, and asks the
302
356
  command and its descendants to stop on cancellation or timeout.
303
357
 
304
- Managed task environment is engine input. Node/Vitest data and contract callbacks
305
- read immutable, enumerable `context.env`; and Playwright suites can read the
306
- same contract from `readPlaywrightContext()?.env`. Managed Node processes also
358
+ Managed task environment is engine input. Playwright suites read it from
359
+ `readPlaywrightContext()?.env`. Managed Node processes also
307
360
  expose each value as an immutable, non-enumerable `process.env.KEY` property so
308
361
  libraries that read `process.env` directly see it, but non-Node child processes
309
362
  do not receive consumer values unless the suite explicitly passes them.
@@ -315,17 +368,20 @@ reserves `APPDATA`, `ComSpec`, `HOME`, `LANG`, `LC_ALL`, `LOCALAPPDATA`, `PATH`,
315
368
 
316
369
  - Node/Vitest reserves `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR`,
317
370
  `UPDATE_SNAPSHOT`, `VITEST_POOL_ID`, `VITEST_WORKER_ID`, and the `TESTKIT_`
318
- keys it sets itself, listed in
319
- [Environment](docs/engines.md#environment);
371
+ keys it sets itself, listed in the Environment section of
372
+ [`docs/engines.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/engines.md);
320
373
  other `TESTKIT_` and `VITEST_` keys, such as
321
- `TESTKIT_REQUIRE_COMPLETE_SUITE`, pass through;
374
+ `TESTKIT_REQUIRE_COMPLETE_SUITE`, pass through, though Vitest's own `TEST`,
375
+ `VITEST`, `VITEST_MODE`, `FORCE_TTY`, `MODE`, `DEV`, `PROD`, and `SSR`
376
+ replace a group's values;
322
377
  - Playwright reserves the `TESTKIT_`, `PLAYWRIGHT_`, `PWTEST_`, and `PW_TEST_`
323
- prefixes plus `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, and `NO_COLOR`.
378
+ prefixes plus `BASE_URL`, `DBUS_SESSION_BUS_ADDRESS`, `FC_FONTATIONS`,
379
+ `FORCE_COLOR`, `NODE_OPTIONS`, and `NO_COLOR`.
324
380
 
325
381
  Case-variant duplicates and invalid names or NUL-containing values fail instead
326
382
  of being normalized. Node/Vitest accepts at most 512 entries and
327
383
  4 MiB of UTF-8 key/value data. Playwright accepts at most 512 entries and 2 MiB
328
- of UTF-8 key/value data. [Environment](docs/engines.md#environment) lists the
384
+ of UTF-8 key/value data. The Environment section of [`docs/engines.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/engines.md) lists the
329
385
  complete limits, and [Named endpoints and databases](#named-endpoints-and-databases)
330
386
  covers the environment API.
331
387
 
@@ -344,12 +400,10 @@ npx @gfdelarue/testkit discover --json > .testkit/discovery.json
344
400
  npx @gfdelarue/testkit discover --output .testkit/discovery.json
345
401
 
346
402
  # Filter by type
347
- npx @gfdelarue/testkit --type contract
348
- npx @gfdelarue/testkit --type data
349
- npx @gfdelarue/testkit --type contract,data
350
403
  npx @gfdelarue/testkit --type ui
404
+ npx @gfdelarue/testkit --type plain
351
405
 
352
- # Parallel file execution
406
+ # Run up to 8 test tasks at once
353
407
  npx @gfdelarue/testkit --task-concurrency 8
354
408
 
355
409
  # One file-level wall clock budget for every suite file
@@ -357,15 +411,16 @@ npx @gfdelarue/testkit --file-timeout-seconds 60
357
411
 
358
412
  # Specific group / suite
359
413
  npx @gfdelarue/testkit --group frontend --type ui -s navigation
360
- npx @gfdelarue/testkit --group api --type contract -s health
361
- npx @gfdelarue/testkit --type contract,data -s contract:queries
414
+ npx @gfdelarue/testkit --group api -s health
415
+ npx @gfdelarue/testkit --group api --type plain
416
+ npx @gfdelarue/testkit --type ui -s ui:checkout
362
417
 
363
418
  # Exact file
364
- npx @gfdelarue/testkit --type contract --file __testkit__/health.contract.testkit.ts
419
+ npx @gfdelarue/testkit --type ui --file __testkit__/checkout.ui.testkit.ts
365
420
  npx @gfdelarue/testkit run --file src/sum.test.ts
366
421
 
367
422
  # Temporarily ignore repo-declared skip rules
368
- npx @gfdelarue/testkit --ignore-skip-rules --file __testkit__/billing.contract.testkit.ts
423
+ npx @gfdelarue/testkit --ignore-skip-rules --file src/billing.test.ts
369
424
 
370
425
  # Lifecycle
371
426
  npx @gfdelarue/testkit status
@@ -380,15 +435,15 @@ npx @gfdelarue/testkit db schema refresh --service api
380
435
  npx @gfdelarue/testkit db schema verify --service api
381
436
  ```
382
437
 
383
- A `--type` that names test types, such as `--type contract,ui`, and typed suite
384
- selectors such as `contract:health` select typed files only. A run without them, or
385
- with `--type all`, also runs plain Vitest test files. To run every typed file
386
- and no plain one, name all three types: `--type ui,contract,data`.
438
+ `--type ui` and typed suite selectors such as `ui:checkout` select typed files
439
+ only. `--type plain` selects plain Vitest test files only. A run without
440
+ `--type`, or with `--type all`, runs both. `--type` narrows the other filters,
441
+ so `--group api --type plain` runs the `api` group's plain files and none of its
442
+ UI files.
387
443
 
388
- `run`, `discover`, `artifacts`, and `workspace` select test groups with
389
- `--group`. On those commands `--service` is a deprecated alias for `--group`
390
- and prints a notice on stderr. `status`, `destroy`, `cleanup`, and `db schema`
391
- take `--service` to name a service.
444
+ `run`, `discover`, `artifacts`, `workspace`, and `typecheck` select test groups
445
+ with `--group`. `status`, `destroy`, `cleanup`, and `db schema` take `--service`
446
+ to name a service.
392
447
 
393
448
  Database cleanup is scoped to the selected product. `cleanup --resources --global`
394
449
  is retired; use `--dir <product> cleanup --resources` for each product. After
@@ -396,10 +451,13 @@ deleting a product directory, remove its orphaned containers and volumes
396
451
  explicitly through Docker.
397
452
 
398
453
  Use `execution.taskConcurrency` in config and `--task-concurrency` on the CLI.
399
- `execution.workers`, `presets.nodeNext({ workers })` and `--workers` remain
400
- deprecated aliases through 0.2.x and will be removed in 0.3.0. When both names are
454
+ A task is a typed file, a batch of typed files, or one Vitest invocation of
455
+ plain files. Plain files in one invocation run with Vitest's file parallelism,
456
+ which `taskConcurrency` doesn't limit.
457
+ `execution.workers`, `presets.nodeNext({ workers })` and `--workers` are
458
+ deprecated aliases. When both names are
401
459
  provided in config or on the CLI, their values must match. The CLI value overrides
402
- config after those checks. See the [execution settings](testkit-config-schema.md#execution).
460
+ config after those checks. See the `execution` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
403
461
 
404
462
  Batch `run` output stays intentionally short: one line per completed file,
405
463
  concise failure blocks, and a final summary. Service logs, captured runtime
@@ -495,7 +553,7 @@ The environment file is JSON:
495
553
  "handles": {
496
554
  "webhook": {
497
555
  "kind": "http-capture",
498
- "url": "http://127.0.0.1:41000/webhook",
556
+ "url": "http://127.0.0.1:41000/__testkit__/<segment>/webhook",
499
557
  "requestsUrl": "http://127.0.0.1:41000/__testkit__/<capability>/requests",
500
558
  "resetUrl": "http://127.0.0.1:41000/__testkit__/<capability>/requests"
501
559
  }
@@ -586,7 +644,7 @@ assertLocalDatabaseUrl(process.env, "seed.ts");
586
644
 
587
645
  ## Setup
588
646
 
589
- See the [configuration schema](testkit-config-schema.md) for the complete,
647
+ See [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md) for the complete,
590
648
  strict `testkit.config.ts` contract.
591
649
 
592
650
  Create `testkit.config.ts` at repo root:
@@ -672,7 +730,7 @@ neither:
672
730
  import { defineFile } from "@gfdelarue/testkit";
673
731
 
674
732
  export const testkit = defineFile({
675
- engine: "node-vitest",
733
+ engine: "playwright",
676
734
  skip: "Billing is currently unavailable locally",
677
735
  });
678
736
  ```
@@ -682,15 +740,10 @@ A file can also name `locks`.
682
740
  one.
683
741
 
684
742
  In a typed file, the engine value is a required literal. The supported public
685
- authoring paths
686
- are `@gfdelarue/testkit/node-vitest/data`,
687
- `@gfdelarue/testkit/node-vitest/contract`, and `@gfdelarue/testkit/playwright`.
688
- Node/Vitest and Playwright suites may also import exact public APIs directly
689
- from Testkit's managed `vitest`, `@playwright/test`, `playwright`, and
690
- `playwright-core` packages.
691
- There are no legacy aliases or implicit-runner compatibility paths. A typed
692
- Node/Vitest file resolves `paths` from its nearest `tsconfig.json`, through
693
- Vite's `resolve.tsconfigPaths`.
743
+ authoring path is `@gfdelarue/testkit/playwright`. Playwright suites may also
744
+ import exact public APIs directly from Testkit's managed `@playwright/test`,
745
+ `playwright`, and `playwright-core` packages.
746
+ There are no legacy aliases or implicit-runner compatibility paths.
694
747
 
695
748
  `testkit.config.ts` is optional. Without it, Testkit runs every test file with
696
749
  zero setup: it starts no server, provisions no database, and loads no `.env`
@@ -727,7 +780,8 @@ setup processes, so set it deliberately.
727
780
  Testkit-owned Docker resources are labelled at creation time so they can be
728
781
  found again even if product-local `.testkit` state is deleted. `testkit destroy`
729
782
  removes labelled resources for the current product after service state is
730
- destroyed. `testkit cleanup --resources` cleans stale resources for the current
783
+ destroyed. It leaves run results under `.testkit/results` and other files
784
+ under `.testkit`, such as the compiled config and timings, in place. `testkit cleanup --resources` cleans stale resources for the current
731
785
  product. Both commands remove only the current product's resources and leave
732
786
  other products' alone, including their idle caches. Host-local Postgres data
733
787
  lives in a Testkit-labelled named Docker volume, so cleanup can find and remove
@@ -735,6 +789,10 @@ it. Database changes for one product run one at a time; other products don't
735
789
  wait for them. Use `cleanup --resources --dry-run` to preview the selected
736
790
  product's cleanup.
737
791
 
792
+ A run's start and its post-run maintenance don't list or reclaim Docker
793
+ resources. A run stops its retained database by name when it releases the run's
794
+ resources.
795
+
738
796
  `database.template` is the database-side equivalent for reusable template DB
739
797
  state. When `database.sourceSchema` is configured, Testkit treats the configured
740
798
  source database as the schema source of truth. A normal `testkit run` resolves a
@@ -845,6 +903,8 @@ services: {
845
903
  }
846
904
  ```
847
905
 
906
+ ## Run evidence
907
+
848
908
  Every terminal run is sealed under `.testkit/results/runs/<run-id>/` with a
849
909
  canonical `run.json`, `provenance.json`, `manifest.json`, and the run's exact
850
910
  artifact/log/setup files. Payloads are SHA-256 addressed under
@@ -870,25 +930,56 @@ workspace API.
870
930
  Sealing replaces secrets with `<redacted>` in the run record and in every text
871
931
  file of the bundle. A key names a secret when it holds `PASSWORD`, `TOKEN`,
872
932
  `SECRET`, `PRIVATE_KEY`, `AUTH`, `COOKIE`, `SESSION`, or `API_KEY` in any letter
873
- case. Sealing redacts:
933
+ case, or when its last word is `PASS`, `PASSWD`, `PAT`, `DSN`, `CREDENTIAL`, or
934
+ `CREDENTIALS`. `AUTH` doesn't count in `AUTHOR` or `AUTHORITY`, so
935
+ `GIT_AUTHOR_NAME` names no secret, though `AUTHORIZATION` still does. `SESSION`
936
+ doesn't count before `COUNT`, `DURATION`, `EXPIR`, `LENGTH`, `LIMIT`, `MAX`,
937
+ `TIMEOUT`, or `TTL`, as in `sessionCount` and `SESSION_TIMEOUT`. A key whose
938
+ last two words are `API`, `PRIVATE`, `SIGNING`, or `ENCRYPTION` and then `KEY`,
939
+ as in `apiKey`, `x-api-key`, and `privateKey`, names a secret too, and so does
940
+ an upper-case key whose last word is `KEY`, such as `STRIPE_KEY`, unless the
941
+ word before it is `PUBLIC` or `PUBLISHABLE`. Words break at any character other
942
+ than a letter or a digit, and before an upper-case letter that follows a
943
+ lower-case letter or a digit. Sealing redacts:
874
944
 
875
945
  - A `KEY=value` assignment under such a key, an `Authorization`,
876
- `Proxy-Authorization`, `Cookie`, or `Set-Cookie` header value, a JSON string
877
- value under such a key, even when the key is spelled with escapes, a
878
- single-quoted `'password': '…'` value, and a URL's password, wherever they
879
- appear. A quoted value ends at its closing quote and keeps its quotes, and a
880
- value that starts with an auth scheme such as `Bearer` ends with its line.
881
- A secret in one JSON value leaves the record's other values as they are.
946
+ `Proxy-Authorization`, `Cookie`, `Set-Cookie`, `Api-Key`, or `X-Api-Key`
947
+ header value, a JSON string value under such a key, even when the key is
948
+ spelled with escapes, a single-quoted string value under such a key, a token
949
+ of at least eight characters with a digit after `Bearer`, and a URL's
950
+ password, wherever they appear. A
951
+ quoted value ends at its closing quote and keeps its quotes, and a value that
952
+ starts with an auth scheme such as `Bearer` ends with its line. A secret in
953
+ one JSON value leaves the record's other values as they are.
954
+ - Every string in a JSON array or object under such a key, unless the key
955
+ names a secret only through `AUTH`, `COOKIE`, or `SESSION`, so `tokens`,
956
+ `credentials`, and `privateKey` count and `session` and `auth` settings stay.
957
+ In an object with a `name` string and a `value` string, only the `value` and
958
+ members whose own key names a secret count. Numbers and booleans stay.
959
+ - The value after `password:` or `passwd:`, or after a key whose last word is
960
+ one of those: a quoted value anywhere, as in `{ password: 'hunter2' }`, and
961
+ an unquoted value at the start of a line, as in YAML, when it is the line's
962
+ one word. A value that is one word of letters, perhaps followed by
963
+ punctuation such as `!` or `.`, stays, and so does an unquoted value after
964
+ other text, so `password: 'Required'`, `password: required!`, and
965
+ `rejects a password: too short` stay.
966
+ - A token of at least 20 characters with a digit that stands alone as the user
967
+ of an `http` or `https` URL, as in `https://ghp_…@github.com`. A URL's
968
+ password may hold `/` and `@`, and sealing also finds it in a directory name,
969
+ which keeps one slash of the URL, as in `postgres:/app:…@db`.
970
+ - A header or cookie stored as a `{"name": …, "value": …}` pair, as Playwright
971
+ traces and HAR files store them: the value when the name names a secret, and
972
+ every value in a `cookies` array.
882
973
  - The value of each such key, and of each key that `redact` names, in the
883
- environment Testkit runs in. When the value is a URL, its password counts
884
- too.
974
+ environment Testkit runs in. When the value is a URL, its password, or a
975
+ token that stands alone as its user, counts too.
885
976
  - The values the run's environment resolved: each value under such a key or a
886
- key that `redact` names, and the password of each URL. A PostgreSQL URL's
887
- password may also be a `password` query parameter, as libpq reads it. This
888
- covers config `env`, env files, `local.env`, dependency services, base,
889
- readiness, UI, and database URLs, `--env`, `--env-from`, and an
890
- `--environment-file`. A value under such a key that is a URL counts only for
891
- its password.
977
+ key that `redact` names, and the password or token user of each URL. A
978
+ PostgreSQL URL's password may also be a `password` query parameter, as libpq
979
+ reads it. This covers config `env`, env files, `local.env`, dependency
980
+ services, base, readiness, UI, and database URLs, `--env`, `--env-from`, and
981
+ an `--environment-file`. A value under such a key that is a URL counts only for
982
+ its password or token user.
892
983
  - The unguessable segment in each [HTTP capture's](#handles) operation URLs,
893
984
  from a setup module's capture or an `--environment-file` handle.
894
985
 
@@ -910,12 +1001,41 @@ and keys in the free-form data and metadata that tests attach. When output cuts
910
1001
  the cut with `…` or `...`, sealing redacts the part that shows. `testkit run
911
1002
  --json` and the run's progress output apply the same redaction.
912
1003
 
1004
+ Sealing also redacts inside `.zip` and `.gz` files, and files that start like
1005
+ one, such as a Playwright `trace.zip`. It applies the rules above to each text
1006
+ entry, such as a trace's actions, network log, and page and API responses, and
1007
+ writes the archive again with the redacted entries. A text entry is valid UTF-8
1008
+ without binary control characters, or UTF-16 with a byte order mark. Other
1009
+ entries keep their bytes. A ZIP with a comment or extra fields is written again
1010
+ without them, and a gzip file is always written again without the name and
1011
+ comment in its header. A trace stays readable in the Playwright trace viewer,
1012
+ with `<redacted>` where a typed password or a token was.
1013
+
1014
+ Sealing refuses an archive it can't redact:
1015
+
1016
+ - one it can't read: encrypted, needing ZIP64 sizes, compressed with a method
1017
+ other than deflate, larger than 512 MiB once expanded, or damaged, including
1018
+ a ZIP with bytes that none of its entries account for;
1019
+ - one with a binary entry, such as a screenshot, that holds one of the values
1020
+ above, as UTF-8 or UTF-16;
1021
+ - one that holds another archive.
1022
+
1023
+ Sealing removes each file it refuses from the run's results and fails the run
1024
+ with the reason, for example:
1025
+
1026
+ ```text
1027
+ Refusing to archive compressed file whose entry inner/trace.zip is itself an archive, and removed it: …/.testkit/results/artifacts/frontend/task-1-01-bundle.zip
1028
+ ```
1029
+
1030
+ The run, or the `testkit stop`, `testkit cleanup` or next run that seals it
1031
+ later, then seals the rest of its results and records the reason.
1032
+
913
1033
  Some secrets stay visible:
914
1034
 
915
1035
  - What a setup module prints goes to the terminal as written.
916
- - Sealing does not look inside compressed or binary files, such as a Playwright
917
- `trace.zip`. It refuses to seal a binary file that holds one of the values
918
- above in plain bytes.
1036
+ - Sealing does not redact images or video, including the screenshots in a
1037
+ trace. Text typed into a page shows in them. Sealing refuses and removes a
1038
+ binary file that holds one of the values above as UTF-8 or UTF-16.
919
1039
  - When a run's process is killed before it seals the run, `testkit stop`,
920
1040
  `testkit cleanup`, or the next run seals it from another process. That
921
1041
  process does not have the values the run's environment resolved or the
@@ -923,6 +1043,15 @@ Some secrets stay visible:
923
1043
  values under the words above. The bundle's `archive.redaction.status` is then
924
1044
  `source-targets-unavailable` instead of `applied`.
925
1045
 
1046
+ Sealing also takes out local paths. In the run record, in every text file of
1047
+ the bundle, and in every text entry of a compressed file, a path under the
1048
+ product directory becomes relative to it, the product directory itself becomes
1049
+ `.`, and a path under your home directory starts with `~`. This covers engine
1050
+ reports, failure stacks, logs, and Playwright traces, and happens after
1051
+ redaction. Binary files and binary entries keep their paths. When sealing
1052
+ changes a file, the file's size and SHA-256 in `run.json` describe the sealed
1053
+ file.
1054
+
926
1055
  Use the artifact archive config to opt into pruning or atomic publication to
927
1056
  an external directory. Local retention is indefinite by default:
928
1057
 
@@ -964,6 +1093,73 @@ Programmatic consumers can import `listRunBundles`, `loadRunBundleArtifact`,
964
1093
  `loadRunBundleManifest`, and `verifyRunBundle` from
965
1094
  `@gfdelarue/testkit/workspace`.
966
1095
 
1096
+ ### Read run evidence without the CLI
1097
+
1098
+ Run evidence is plain files, so you can investigate a run with `ls`, `grep`,
1099
+ `jq`, and `sha256sum` when `testkit` can't start, for example because the
1100
+ config no longer loads. A command that fails to load the config stops before it
1101
+ writes to `.testkit/results`, and `testkit artifacts` doesn't need the config.
1102
+
1103
+ | Path under `.testkit/results/` | What it holds |
1104
+ | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
1105
+ | `runs/<run-id>/run.json` | A sealed run's record: `run.status`, `run.interruptReason`, and each file's `status`, `failureDetails`, and `artifacts` |
1106
+ | `runs/<run-id>/manifest.json` | Every other file in the bundle, with its `sha256` and `bytes` |
1107
+ | `runs/<run-id>/provenance.json` | The run's CI, source, runtime, and redaction facts |
1108
+ | `runs/<run-id>/files/` | The run's `logs/`, `artifacts/`, and `setup/` files. Each `artifacts[].path` in `run.json` points here, relative to the product directory |
1109
+ | `latest.json` | The record of the last run that Testkit finished writing up, with its publication and retention results. A run killed after it was sealed can be in `runs/` but not here, so `runs/<run-id>/run.json` is the authority |
1110
+ | `live.json` | The record of a run that isn't sealed, with `run.status` `running` |
1111
+ | `logs/`, `artifacts/`, `setup/` | The files that unsealed run has written so far |
1112
+
1113
+ Anything else under `.testkit/` can change between releases, including
1114
+ `objects/`, which stores the bytes behind each bundle's files.
1115
+
1116
+ A run is sealed once `runs/<run-id>/` exists. Testkit writes the bundle under a
1117
+ hidden name in `runs/`, writes `manifest.json` last, and then renames the
1118
+ directory, so `runs/<run-id>/` is never partly written. Ignore names in `runs/`
1119
+ that start with `.`. A copy of a bundle, such as a CI upload, is whole when
1120
+ every file that its `manifest.json` lists has the listed `sha256`. A sealed
1121
+ `run.json` has `archive.status` `complete` and `run.status` `passed`, `failed`,
1122
+ or `interrupted`.
1123
+
1124
+ `live.json` means a run is in progress, or its process died before Testkit
1125
+ finished with it. If `runs/<run-id>/` exists for the `run.id` in `live.json`,
1126
+ the run was sealed and the bundle is its record: `live.json` and the active
1127
+ directories are leftovers, which the next `testkit run`, `cleanup`, or `destroy`
1128
+ removes. Testkit replaces `live.json` whole on each update, so it always parses.
1129
+ Its `generatedAt` is the time of the last update, and each file's `status` is
1130
+ what it was then, so a file that was still running shows `not_run`. The files
1131
+ in `logs/`, `artifacts/`, and `setup/` aren't redacted until the run is sealed.
1132
+
1133
+ How an unfinished run gets sealed:
1134
+
1135
+ - When Ctrl-C, `SIGTERM`, or `testkit stop` interrupts the run, or its process
1136
+ exits early, Testkit seals it straight away. `run.status` is `interrupted`,
1137
+ or `failed` when nothing asked the run to stop, and `run.interruptReason`
1138
+ says why.
1139
+ - When the process is killed with `SIGKILL`, by the out-of-memory killer, or
1140
+ with its host, the run stays unsealed until the next `testkit run`,
1141
+ `cleanup`, or `destroy` seals it, before doing anything else, with
1142
+ `run.status` `interrupted` and `run.interruptReason`
1143
+ `stale run process not active`. `testkit stop` seals it too, with its
1144
+ `--reason` or `stop requested`. The sealed `run.finishedAt` is when the
1145
+ command sealed it, not when the process died.
1146
+
1147
+ Testkit keeps what reached the disk. It can't keep output that a process hadn't
1148
+ written yet, anything written after the disk filled up or became read-only, or
1149
+ work that a killed process would have done on its way out. A log can end
1150
+ mid-line.
1151
+
1152
+ For example, to see what failed in the run that `latest.json` names and check
1153
+ its files:
1154
+
1155
+ ```sh
1156
+ cd .testkit/results
1157
+ run=runs/$(jq -r .run.id latest.json)
1158
+ jq '.run.status, .run.interruptReason' "$run/run.json"
1159
+ jq '.groups[].suites[].files[] | select(.status != "passed") | {path, status, failureDetails, artifacts: [.artifacts[].path]}' "$run/run.json"
1160
+ jq -r '.files[] | "\(.sha256) \(.path)"' "$run/manifest.json" | (cd "$run" && sha256sum --check --quiet)
1161
+ ```
1162
+
967
1163
  ## Authoring
968
1164
 
969
1165
  ### Named endpoints and databases
@@ -1016,14 +1212,14 @@ something its environment prepared. Today that is an HTTP capture, which a
1016
1212
  The handle is named after the capture's `id`:
1017
1213
 
1018
1214
  ```js
1019
- // tests/webhook.contract.testkit.setup.mjs
1215
+ // tests/webhook.test.setup.mjs
1020
1216
  export async function setup({ sidecars }) {
1021
1217
  await sidecars.httpCapture({ id: "webhook", path: "/webhook" });
1022
1218
  }
1023
1219
  ```
1024
1220
 
1025
1221
  ```ts
1026
- // tests/webhook.contract.testkit.ts
1222
+ // tests/webhook.test.ts
1027
1223
  import { handle } from "@gfdelarue/testkit/environment";
1028
1224
 
1029
1225
  const webhook = handle("webhook");
@@ -1045,6 +1241,14 @@ run's results, artifacts, and events, and the capture refuses any other
1045
1241
  operation. When Testkit releases the file's environment, the capture closes and
1046
1242
  the handle expires. Calling a handle never starts a sidecar or a service.
1047
1243
 
1244
+ The capture's `url` holds a second unguessable segment, which Testkit also
1245
+ keeps out of the run's records. The capture records a request only when its
1246
+ path starts with that segment, and answers any other path with 404. A captured
1247
+ request's `path` leaves the segment out: a request to `url` records `/webhook`,
1248
+ and one to `url` + `/42` records `/webhook/42`. The capture answers 403 to a
1249
+ request whose `Host` header isn't `127.0.0.1:<port>`, which keeps a web page
1250
+ from reaching it through DNS rebinding.
1251
+
1048
1252
  - A name with no handle throws `EnvironmentValueMissingError` with `kind`
1049
1253
  `"handle"`, and the test ends as an infrastructure failure, as for a missing
1050
1254
  endpoint.
@@ -1069,7 +1273,8 @@ test, in any order.
1069
1273
  so read them and leave them unchanged.
1070
1274
  - When the test ends, leave its rows in place. Testkit resets the databases it
1071
1275
  provisions. By default, every run starts from a fresh clone of the template
1072
- database. See [`database.reset`](testkit-config-schema.md#database).
1276
+ database. See `database.reset` in the `database` section of
1277
+ [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
1073
1278
  - Clean up only what Testkit doesn't own, such as an account in a third-party
1074
1279
  sandbox. Register the cleanup as soon as the test creates the account. Give
1075
1280
  the account a unique name, so that a leftover from a cleanup that never ran
@@ -1077,19 +1282,20 @@ test, in any order.
1077
1282
 
1078
1283
  ```ts
1079
1284
  import { randomUUID } from "node:crypto";
1080
- import { defineFile } from "@gfdelarue/testkit";
1081
- import { contractScenario } from "@gfdelarue/testkit/node-vitest/contract";
1082
- import { expect, onTestFinished } from "vitest";
1285
+ import { endpoint } from "@gfdelarue/testkit/environment";
1286
+ import { expect, onTestFinished, test } from "vitest";
1083
1287
 
1084
1288
  import { sandbox } from "../payments-sandbox.js";
1085
1289
 
1086
- export const testkit = defineFile({ engine: "node-vitest" });
1087
-
1088
- contractScenario("a refund reaches the sandbox customer", async ({ http }) => {
1290
+ test("a refund reaches the sandbox customer", async () => {
1089
1291
  const customer = await sandbox.createCustomer({ email: `refund-${randomUUID()}@example.test` });
1090
1292
  onTestFinished(() => sandbox.deleteCustomer(customer.id));
1091
1293
 
1092
- const response = await http().post("/api/refunds", { customerId: customer.id, amountCents: 500 });
1294
+ const response = await fetch(new URL("/api/refunds", endpoint("api")), {
1295
+ method: "POST",
1296
+ headers: { "content-type": "application/json" },
1297
+ body: JSON.stringify({ customerId: customer.id, amountCents: 500 }),
1298
+ });
1093
1299
  expect(response.status).toBe(201);
1094
1300
  });
1095
1301
  ```
@@ -1125,18 +1331,34 @@ Files that name the same lock never run at the same time within one run.
1125
1331
  Testkit doesn't share locks between runs, such as two CI jobs. Two tests that
1126
1332
  need a lock because they share rows need their own rows instead.
1127
1333
 
1334
+ A plain Vitest file names its locks in its group's `requirements`:
1335
+
1128
1336
  ```ts
1129
- export const testkit = defineFile({ engine: "node-vitest", locks: ["payments-sandbox-webhook"] });
1337
+ groups: {
1338
+ payments: {
1339
+ roots: ["src/payments"],
1340
+ uses: ["api"],
1341
+ requirements: {
1342
+ files: [{ path: "src/payments/webhook.test.ts", locks: ["payments-sandbox-webhook"] }],
1343
+ },
1344
+ },
1345
+ },
1346
+ ```
1347
+
1348
+ A browser file can name them in `defineFile`:
1349
+
1350
+ ```ts
1351
+ export const testkit = defineFile({ engine: "playwright", locks: ["payments-sandbox-webhook"] });
1130
1352
  ```
1131
1353
 
1132
1354
  ### Node setup beside a test
1133
1355
 
1134
- A test such as `tests/webhook.contract.testkit.ts` can have a sibling
1135
- `tests/webhook.contract.testkit.setup.mjs`. Testkit runs its named `setup` export,
1136
- or its default export, before starting the test engine:
1356
+ A test such as `tests/webhook.test.ts` can have a sibling
1357
+ `tests/webhook.test.setup.mjs`. Testkit runs its named `setup` export, or its
1358
+ default export, before starting the test engine:
1137
1359
 
1138
1360
  ```js
1139
- // tests/webhook.contract.testkit.setup.mjs
1361
+ // tests/webhook.test.setup.mjs
1140
1362
  export async function setup({ sidecars }) {
1141
1363
  const capture = await sidecars.httpCapture({ path: "/webhook" });
1142
1364
  return {
@@ -1148,13 +1370,15 @@ export async function setup({ sidecars }) {
1148
1370
  Setup is part of preparing the file's environment. Testkit runs it once, after
1149
1371
  it acquires the services the file's group uses and before the test engine
1150
1372
  starts. Testkit never batches a Playwright file that has setup with other
1151
- Playwright files. Files without setup still batch.
1373
+ Playwright files, and a plain Vitest file that has setup runs in its own Vitest
1374
+ invocation. Files without setup still batch.
1152
1375
 
1153
1376
  Setup runs with its own JavaScript state. It can use Node APIs, prepare
1154
1377
  external resources, and start HTTP capture listeners. Tests
1155
- reach a capture by its `id` through [`handle()`](#handles). A capture's
1156
- `requestsUrl` and `resetUrl` hold an unguessable segment that Testkit keeps out
1157
- of the run's records, and its capture `path` may not start with `/__testkit__/`.
1378
+ reach a capture by its `id` through [`handle()`](#handles). A capture's `url`,
1379
+ `requestsUrl` and `resetUrl` hold unguessable segments that Testkit keeps out
1380
+ of the run's records. Its capture `path` may not start with `/__testkit__/` or
1381
+ contain `.` or `..` segments.
1158
1382
  The context contains `deadlineMs`, `signal`, `env`, `leaseDir`, `productDir`,
1159
1383
  `serviceName`, `testFile`, and `sidecars`. `serviceName` holds the name of the
1160
1384
  test's group. Setup has the same permissions as the Testkit process. It is not
@@ -1165,7 +1389,9 @@ setup modules. Its tests receive the environment file's `setupData` instead.
1165
1389
 
1166
1390
  Return plain JSON, or `{ data, cleanup }` with an optional cleanup function.
1167
1391
  Testkit passes the data to the engine as `setupData` and records it with the
1168
- setup module path and sidecar handles. Bare `null` or `undefined`, and a
1392
+ setup module path and sidecar handles. Tests read it with `setupData()` from
1393
+ `@gfdelarue/testkit/environment`, which returns `{}` when the file has no setup
1394
+ module and throws outside a Testkit run. Bare `null` or `undefined`, and a
1169
1395
  cleanup-only result, give empty data. `{ data: null }` preserves `null`.
1170
1396
  Data must encode to at most 1 MiB of JSON. Invalid data reports its JSON path,
1171
1397
  such as `$.items[1]`.
@@ -1189,15 +1415,226 @@ most 256 connections.
1189
1415
 
1190
1416
  Discovery tries sibling `.setup.mjs`, sibling `.setup.js`, then the matching
1191
1417
  `.mjs` and `.js` files in `__setup__`, in that order. For the example above,
1192
- the fallback is `tests/__setup__/webhook.contract.testkit.mjs`.
1418
+ the fallback is `tests/__setup__/webhook.test.mjs`.
1193
1419
 
1194
1420
  ### HTTP and persistence checks
1195
1421
 
1196
1422
  Use plain Vitest files for ordinary HTTP, service, and database checks. Read
1197
1423
  configured URLs with `endpoint(name)` and `database(name)` from
1198
1424
  `@gfdelarue/testkit/environment`, then use `fetch` or your application's
1199
- database client. Use `contractScenario` when a typed contract file needs the
1200
- managed contract context. Use `dataScenario` for structured data-flow evidence.
1425
+ database client.
1426
+
1427
+ ### Record data-flow evidence
1428
+
1429
+ `dataFlow(context)` from `@gfdelarue/testkit/node-vitest/data` records what a
1430
+ plain Vitest test sees data do: metrics, counts, bytes, durations, money,
1431
+ ratios, sequences, records, actors, systems, transitions, events, findings,
1432
+ external calls, and cache, LLM, and RPC usage. Budgets check the recorded
1433
+ values, for example that a metric stays under a limit or that a sequence was
1434
+ recorded.
1435
+
1436
+ ```ts
1437
+ import { endpoint } from "@gfdelarue/testkit/environment";
1438
+ import { dataFlow } from "@gfdelarue/testkit/node-vitest/data";
1439
+ import { test } from "vitest";
1440
+
1441
+ test("lists one page of orders", async (context) => {
1442
+ const flow = dataFlow(context);
1443
+ const response = await fetch(`${endpoint("api")}/orders?page=1`);
1444
+ const body = await response.text();
1445
+ flow.recordHttpRequest({
1446
+ method: "GET",
1447
+ path: "/orders",
1448
+ status: response.status,
1449
+ responseBytes: Buffer.byteLength(body),
1450
+ });
1451
+ flow.expectMetric("http.response.bytes").atMost(64_000);
1452
+ });
1453
+ ```
1454
+
1455
+ - `dataFlow` needs the context of a running test. Every call in one attempt
1456
+ returns the same recorder, and a call without that context throws.
1457
+ - The library reads no URL, environment value, fixture file, or Testkit config.
1458
+ Read named URLs from `@gfdelarue/testkit/environment`, log in from test code,
1459
+ and read files with `fs`.
1460
+ - A failed budget fails its test when the test finishes. The failure detail has
1461
+ the kind `data-flow-budget` and a key made of the test's full title and the
1462
+ budget.
1463
+ - Testkit writes `data-flow.json` and `data-flow-report.md` for a file whose tests
1464
+ recorded something. Each such test is one scenario, from its final attempt. A
1465
+ test that records nothing adds nothing, and a file whose tests record
1466
+ nothing gets no data-flow artifacts.
1467
+
1468
+ ### Slow the browser's network and CPU
1469
+
1470
+ `emulateNetwork(page, conditions)` and `emulateCpu(page, slowdown)` from
1471
+ `@gfdelarue/testkit/playwright` slow a page the way Chrome DevTools'
1472
+ throttling does. A `ui` test can use them to fail when a change makes a page
1473
+ unusable on a slow connection or a slow device:
1474
+
1475
+ ```ts
1476
+ import { defineFile } from "@gfdelarue/testkit";
1477
+ import { emulateCpu, emulateNetwork, expect, test } from "@gfdelarue/testkit/playwright";
1478
+
1479
+ export const testkit = defineFile({ engine: "playwright" });
1480
+
1481
+ test("the dashboard is usable on a slow connection", async ({ page }) => {
1482
+ await emulateNetwork(page, "slow-4g");
1483
+ await page.goto("/dashboard");
1484
+ await expect(page.getByRole("heading", { name: "Orders" })).toBeVisible({ timeout: 10_000 });
1485
+ });
1486
+
1487
+ test("the dashboard is usable on a slow phone", async ({ page }) => {
1488
+ await emulateCpu(page, 4);
1489
+ await page.goto("/dashboard");
1490
+ await expect(page.getByRole("heading", { name: "Orders" })).toBeVisible({ timeout: 10_000 });
1491
+ });
1492
+ ```
1493
+
1494
+ The assertion's timeout is the time limit. The helpers add no timing or
1495
+ metrics API.
1496
+
1497
+ `emulateNetwork` takes a profile name or custom values. Each profile has the
1498
+ values of the DevTools preset of the same name in the Chromium that Testkit
1499
+ runs:
1500
+
1501
+ | Profile | DevTools preset | Download | Upload | Latency |
1502
+ | ----------- | --------------- | ---------- | ---------- | -------- |
1503
+ | `"fast-4g"` | Fast 4G | 8,100 kbps | 1,350 kbps | 165 ms |
1504
+ | `"slow-4g"` | Slow 4G | 1,440 kbps | 675 kbps | 562.5 ms |
1505
+ | `"3g"` | 3G | 400 kbps | 400 kbps | 2,000 ms |
1506
+
1507
+ DevTools renamed two presets in 2024: Slow 4G was Fast 3G, and 3G was Slow 3G.
1508
+ Custom values give speeds in kilobits per second and latency in milliseconds:
1509
+
1510
+ ```ts
1511
+ await emulateNetwork(page, { latencyMs: 300, downloadKbps: 400, uploadKbps: 200 });
1512
+ ```
1513
+
1514
+ `emulateCpu` makes the page's main thread, which runs its JavaScript and
1515
+ layout, `slowdown` times slower. DevTools' fixed CPU options are 4, 6, and 20.
1516
+ Testkit doesn't calibrate the slowdown to the machine, as DevTools' "Mid-tier
1517
+ mobile" and "Low-tier mobile" options do. `emulateCpu(page, 1)` restores full
1518
+ speed.
1519
+
1520
+ - Call the helpers before `page.goto()`. Both settings last across navigations
1521
+ until the page closes.
1522
+ - Neither setting reaches a cross-site iframe. Chromium runs one in a process
1523
+ of its own, at full speed.
1524
+ - The network conditions apply to that page only. A popup it opens, or another
1525
+ page, loads at full speed. Playwright's `browserContext.setOffline()` clears
1526
+ them.
1527
+ - The CPU slowdown applies to the browser process that runs the page. A
1528
+ same-site popup that keeps its `window.opener` shares that process and slows
1529
+ down too, and closing it ends the slowdown. Call `emulateCpu` again after
1530
+ closing one.
1531
+ - Like DevTools, `emulateNetwork` also sets `navigator.connection` until the
1532
+ page navigates to another site.
1533
+ - An unknown profile name or field, a negative or non-finite value, a speed of
1534
+ 0, or a CPU slowdown outside 1 to 100 throws an error that names the value.
1535
+
1536
+ Three things to keep in mind:
1537
+
1538
+ - Set generous time limits. A CI runner has less CPU than a laptop, and a
1539
+ [campaign](#run-a-campaign) files a failing test as a bug, so a tight limit
1540
+ produces false bug reports.
1541
+ - A CPU slowdown multiplies the speed of the machine that runs the test, so 4
1542
+ on a slow CI runner is much slower than 4 on a laptop. A time limit under a
1543
+ CPU slowdown depends on the runner far more than one under network
1544
+ conditions, which are the same on every machine.
1545
+ - Chromium holds each response for at least the latency and shares the
1546
+ bandwidth among the page's requests, but it doesn't model TCP. The result
1547
+ approximates a slow connection rather than reproducing one.
1548
+
1549
+ ### Check a ui test's browser traffic
1550
+
1551
+ Testkit records the requests every `ui` test's pages send, with no change to the
1552
+ test. The record catches bugs that live in how a frontend uses its API, such as
1553
+ a checkout that sends its payment twice or a list page that fetches each row
1554
+ separately, even when no assertion checks for them.
1555
+
1556
+ For each `ui` file whose tests sent requests, Testkit writes
1557
+ `browser-network.json` and `browser-network-report.md` from each test's final
1558
+ attempt. A test's section of the report looks like this:
1559
+
1560
+ ```md
1561
+ ## checkout › pays for the cart (passed)
1562
+
1563
+ 21 requests, 452 KB received: script 312 KB, font 48 KB, document 41 KB, fetch 14 KB
1564
+
1565
+ | Service | Request | Count | Sent | Received | Status |
1566
+ | ------- | ----------------- | ----- | ------ | -------- | ------ |
1567
+ | web | GET /cart | 1 | 0 B | 41 KB | 200 |
1568
+ | api | GET /products/:id | 8 | 0 B | 11 KB | 200 |
1569
+ | api | POST /payments | 2 | 1.2 KB | 400 B | 201 |
1570
+
1571
+ Other origins: fonts.gstatic.com, 2 requests, 48 KB
1572
+
1573
+ Leads:
1574
+
1575
+ - `POST /payments` to api went out 2 times with the same body.
1576
+ ```
1577
+
1578
+ - Rows cover page loads and API calls, which are Playwright's `document`,
1579
+ `fetch`, and `xhr` requests, to the services the test's group uses. A row
1580
+ names the service rather than its URL, because each run gives services
1581
+ different ports. Scripts, styles, images, and fonts count only toward the
1582
+ totals by type. Requests to other origins appear by host name only.
1583
+ - A path's numbers, UUIDs, and hex strings of 16 or more characters become
1584
+ `:id`, so eight product fetches make one row with a count of 8.
1585
+ - The record never holds query strings, headers, or bodies. Paths pass through
1586
+ Testkit's redaction, and sealing redacts them again with the run's secrets.
1587
+ To spot a repeated write, Testkit hashes each body in memory and keeps only
1588
+ the count.
1589
+ - Sizes are body bytes as they crossed the network, so a compressed response
1590
+ counts its compressed size and one that a `page.route()` handler fulfills
1591
+ counts 0. Headers don't count.
1592
+ - Each test keeps 200 requests in detail, 100 rows, and 20 other origins. The
1593
+ record counts the rest.
1594
+ - The leads point at two patterns: a POST, PUT, PATCH, or DELETE sent more than
1595
+ once to the same URL with the same body, and a page load or API call that
1596
+ failed or got a 4xx or 5xx status in a test that passed. A lead never fails a
1597
+ test.
1598
+ - The record covers every context a test opens, including one from
1599
+ `browser.newContext()`. It doesn't cover a context that a `beforeAll` hook
1600
+ opens.
1601
+
1602
+ To assert on the traffic, call `browserRequests()` from
1603
+ `@gfdelarue/testkit/playwright`. It returns the requests the test's pages have
1604
+ sent so far, oldest first:
1605
+
1606
+ ```ts
1607
+ import { browserRequests, expect, test } from "@gfdelarue/testkit/playwright";
1608
+
1609
+ test("checkout charges the card once", async ({ page }) => {
1610
+ await page.goto("/cart");
1611
+ await page.getByRole("button", { name: "Pay" }).click();
1612
+ await expect(page.getByText("Payment received")).toBeVisible();
1613
+
1614
+ const payments = (await browserRequests()).filter(
1615
+ (request) => request.service === "api" && request.method === "POST" && request.path === "/payments",
1616
+ );
1617
+ expect(payments).toHaveLength(1);
1618
+ });
1619
+ ```
1620
+
1621
+ Each entry has `service`, the name of the service it went to or `null`,
1622
+ `method`, `url`, `path`, `resourceType`, `status`, `failure`, `requestBytes`,
1623
+ `responseBytes`, and `durationMs`. `path` is the URL's path as sent, without
1624
+ `:id`. Add up entries with plain array code, for example to check a page's
1625
+ weight:
1626
+
1627
+ ```ts
1628
+ const received = (await browserRequests()).reduce((total, request) => total + request.responseBytes, 0);
1629
+ expect(received).toBeLessThan(500_000);
1630
+ ```
1631
+
1632
+ `browserRequests()` waits up to 5 seconds for requests already in flight, so
1633
+ their entries have a status and sizes. It doesn't wait for EventSource streams.
1634
+ A request still unfinished after that has its status if the response's headers
1635
+ arrived, 0 response bytes, and a `durationMs` of `null`. A WebSocket connection
1636
+ appears with the `resourceType` `websocket` and no status, sizes, or duration.
1637
+ The record counts connections, not their messages.
1201
1638
 
1202
1639
  ## Discovery
1203
1640
 
@@ -1228,24 +1665,26 @@ Testkit supports Vitest 4.x. Another major, a broken installation, a root
1228
1665
  Vitest config that fails to load, or a Vitest that `package.json` declares but
1229
1666
  that is not installed fails before any test runs, naming the version and path.
1230
1667
  `--vitest-runner managed` avoids the repository's Vitest. A run that selects
1231
- only typed files, with `--type` or with typed `--file` paths, `bug record` or
1668
+ only typed files, with `--type ui` or with typed `--file` paths, `bug record` or
1232
1669
  `bug file` for a typed test, and the commands that select no tests (`status`,
1233
1670
  `run status`, `doctor`, `destroy`, `cleanup`, and `db schema`) don't load the
1234
1671
  repository's Vitest, so they still work. A product that neither installs nor
1235
1672
  declares Vitest has no plain test files.
1236
1673
 
1237
- All plain files in a run go to one Vitest invocation, so root global setup,
1238
- teardown, and the repository's reporters run as often as they do under
1239
- `vitest run`, and Testkit's reporter runs beside them. The invocation's deadline
1240
- is the sum of its files' timeouts; Vitest's own test and hook timeouts apply
1241
- inside it. A file that two projects select runs in both, its result combines
1242
- both runs, and its evidence names each run's project. Files from different
1243
- groups run together when their groups give them the same services, values, and
1244
- locks; none may have a setup module or isolation other than `shared`.
1245
- Otherwise the run stops before any file runs and names the groups and what
1246
- differs. Services, locks, isolation, or a setup module stop it before Testkit
1247
- provisions anything; values stop it before Vitest starts. Select one group with
1248
- `--group`.
1674
+ Plain files run in one Vitest invocation per environment their groups give them.
1675
+ Groups that give their files the same values share one, such as `root` and the
1676
+ other groups that use no services. A group that uses a service Testkit starts
1677
+ gets an invocation of its own.
1678
+ Inside each invocation, root global setup, teardown, and the repository's
1679
+ reporters run as they do under `vitest run`, and Testkit's reporter runs beside
1680
+ them, so a run calls them once for each invocation. An invocation's deadline is
1681
+ the sum of its files' timeouts; Vitest's own test and hook timeouts apply inside
1682
+ it. A file that two projects select runs in both, its result combines both
1683
+ runs, and its evidence names each run's project. A file with a setup module,
1684
+ locks, or isolation other than `shared` runs in its own Vitest invocation with
1685
+ its own environment. Invocations run one after another unless
1686
+ `--task-concurrency` or `execution.taskConcurrency` is above 1. To run one group's
1687
+ files alone, select it with `--group`.
1249
1688
 
1250
1689
  Plain files see the host environment Testkit started with: `HOME`, `PATH`,
1251
1690
  temporary and XDG directories, locale, `CI`, `FORCE_COLOR`, `NO_COLOR`, and any
@@ -1259,14 +1698,21 @@ shell's environment and the group's values; as under `vitest run`, Vitest sets
1259
1698
  `TEST`, `VITEST`, and `NODE_ENV` after them. `--vitest-runner managed` leaves the
1260
1699
  shell's `NODE_OPTIONS` out.
1261
1700
 
1701
+ When Vitest detects a coding agent from the environment, such as `CLAUDECODE`,
1702
+ `AI_AGENT`, or `CURSOR_AGENT`, and the repository's Vitest config names no
1703
+ reporter, Vitest uses its agent reporter. That reporter prints no console output
1704
+ from a passing test, so the output is missing from the file's
1705
+ `node-vitest-stdout` artifact. A failing test's output still appears. Name a
1706
+ reporter in the Vitest config, such as `reporters: ["default"]`, to keep it.
1707
+
1262
1708
  Two settings narrow the files Vitest selects: `discovery.exclude` in `testkit.config.ts`, and
1263
1709
  a declared group's `exclude`. Testkit's built-in exclusions for typed files,
1264
- such as `dist`, `build`, and `coverage`, `discovery.roots`, and a legacy
1265
- service's exclusions do not apply to plain files.
1710
+ such as `dist`, `build`, and `coverage`, and `discovery.roots` do not apply to
1711
+ plain files.
1266
1712
 
1267
1713
  A plain file belongs to the group whose roots contain it, or to the implicit
1268
- `root` group, which uses no service. Legacy service ownership never claims a
1269
- plain file. Testkit starts no service for a plain file in `root`.
1714
+ `root` group, which uses no service. Testkit starts no service for a plain
1715
+ file in `root`.
1270
1716
 
1271
1717
  The plain files in one directory form a suite named after that directory,
1272
1718
  relative to the group's root. In `root`, `--suite src` selects
@@ -1275,7 +1721,10 @@ its own suite, so `sum.test.ts` in a group rooted at `src` is in suite `sum`.
1275
1721
  When two suites in a group would share a name, Testkit lengthens the later
1276
1722
  ones with parent directories, such as `api-src`, and numbers them if that is
1277
1723
  not enough. `testkit typecheck` checks plain TypeScript test files and skips
1278
- plain JavaScript ones, as `tsc` does without `allowJs`.
1724
+ plain JavaScript ones, as `tsc` does without `allowJs`. It adds the TypeScript
1725
+ files in each Vitest project's `setupFiles` to the program, so a test can use
1726
+ the matchers a setup file declares, such as those from
1727
+ `@testing-library/jest-dom/vitest`.
1279
1728
 
1280
1729
  ### Typed suites
1281
1730
 
@@ -1308,16 +1757,9 @@ production owner when a narrower owner exists.
1308
1757
 
1309
1758
  Concrete layouts:
1310
1759
 
1311
- - `src/api/routes/__testkit__/auth-me.contract.testkit.ts`
1312
- - `src/db/__testkit__/sessions-count-type.contract.testkit.ts`
1313
1760
  - `frontend/src/components/layout/__testkit__/navigation.ui.testkit.ts`
1314
- - `src/internal/handler/repos/__testkit__/crud.contract.testkit.ts`
1315
-
1316
- `testkit` uses these suffixes automatically:
1317
1761
 
1318
- - `*.contract.testkit.ts`
1319
- - `*.data.testkit.ts`
1320
- - `*.ui.testkit.ts`
1762
+ `testkit` uses the `*.ui.testkit.ts` suffix automatically.
1321
1763
 
1322
1764
  The suffix never chooses an engine. Each discovered typed file must also
1323
1765
  contain one root-package `defineFile(...)` declaration with a supported literal
@@ -1325,30 +1767,7 @@ engine.
1325
1767
  Unsupported pairs fail discovery; for example, a `*.ui.testkit.ts` file cannot
1326
1768
  declare `engine: "node-vitest"`.
1327
1769
 
1328
- See [docs/test-types.md](docs/test-types.md) for the canonical type model.
1329
- `*.contract.testkit.ts` files are Node-backed service/runtime contract suites.
1330
- They use `contractScenario(...)` from
1331
- `@gfdelarue/testkit/node-vitest/contract`, declare `engine: "node-vitest"`, and run
1332
- through the `node-vitest-contract` profile for checks that need Node APIs but
1333
- are not data-flow evidence suites. They are native Vitest suites:
1334
- `contractScenario(...)` is the managed-context convenience, while direct public
1335
- Vitest runner APIs remain valid for behavior such as `test.skip(...)` and
1336
- `test.todo(...)`.
1337
-
1338
- `*.data.testkit.ts` files are Testkit data-flow/data-contract suites, not raw
1339
- Vitest files with a Testkit suffix. They must use `dataScenario(...)` from
1340
- `@gfdelarue/testkit/node-vitest/data`, declare `engine: "node-vitest"`, and record
1341
- structured data-flow evidence through the data context. Supporting
1342
- `count(...)`/`event(...)` calls are not enough by themselves; model actors,
1343
- systems, records, transitions, external calls, payloads, budgets, cache/LLM/RPC
1344
- usage, or reconciliation evidence. Data suites run through the
1345
- `node-vitest-data` profile. Use a plain Vitest test file for plain Node
1346
- assertions and HTTP or database integration, `contract` for typed service or
1347
- protocol contracts, and `ui` when a browser is part of the assertion.
1348
- Data suites use Vitest directly for assertions, hooks, and mocking.
1349
- `dataScenario(...)` owns case registration: `testkit lint` reports native
1350
- registration imports in data suites, and the data runner rejects any case that
1351
- was not registered through the DSL.
1770
+ See [`docs/test-types.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/test-types.md) for the canonical type model.
1352
1771
 
1353
1772
  Test groups own test files by directory. Declare them under `groups` in
1354
1773
  `testkit.config.ts`. Each group lists its repo-relative `roots` and the services
@@ -1378,18 +1797,19 @@ export default defineConfig({
1378
1797
  are also available by name; see
1379
1798
  [Named endpoints and databases](#named-endpoints-and-databases).
1380
1799
  - `skip`, `requirements`, and `runtimeIsolation` are group fields. See the
1381
- [config reference](testkit-config-schema.md#groups).
1800
+ `groups` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
1382
1801
 
1383
- A config with `services` and no `groups` key uses legacy ownership. Each
1384
- service owns the typed test files under its `local.cwd` and
1385
- `discovery.roots`, as a group of the same name that uses that service. Testkit
1386
- prints a notice on stderr when legacy ownership gives a file to a service.
1387
- Legacy ownership will be removed; declare `groups` instead.
1802
+ Services own no test files. A config with `services` and no `groups` key
1803
+ fails to load when a service's `local.cwd` or `discovery.roots` holds a typed
1804
+ test file, which that service owned before 0.6.0. The error lists the `groups`
1805
+ entries that keep those files with their services. `status`, `destroy`,
1806
+ `cleanup`, and `db schema` still run on it. Otherwise every file belongs to
1807
+ `root`.
1388
1808
 
1389
1809
  Suite names are inferred from the colocated path:
1390
1810
 
1391
- - `auth/__testkit__/*.contract.testkit.ts` => `auth`
1392
- - `routes/__testkit__/auth-me.contract.testkit.ts` => `routes`
1811
+ - `auth/__testkit__/*.ui.testkit.ts` => `auth`
1812
+ - `routes/__testkit__/checkout.ui.testkit.ts` => `routes`
1393
1813
 
1394
1814
  Discovery is also a first-class CLI/API surface:
1395
1815
 
@@ -1446,11 +1866,16 @@ Testkit ignores an old `.testkit/history/` directory and leaves it in place.
1446
1866
  `@gfdelarue/testkit` provisions Docker-managed local Postgres automatically for
1447
1867
  services that define `database: database.postgres(...)`.
1448
1868
 
1449
- Docker is also the required Postgres client runtime for Testkit-owned database
1450
- operations. Testkit runs `psql` and `pg_dump` inside the configured Postgres
1451
- client image instead of using host-installed binaries, which keeps local runs
1452
- consistent across Linux, macOS, and native Windows. Override the client image
1453
- with `TESTKIT_POSTGRES_CLIENT_IMAGE` when needed.
1869
+ Docker is also the required Postgres client runtime. Testkit doesn't use
1870
+ host-installed `psql` or `pg_dump`, which keeps local runs consistent across
1871
+ Linux, macOS, and native Windows. Testkit's own operations on the database it
1872
+ owns, such as declaring roles and taking schema snapshots, run inside the
1873
+ database container. So the `image` that `database.postgres()` uses must include
1874
+ `psql` and `pg_dump`, with a `pg_dump` at least as new as its server. Your SQL,
1875
+ in `sql-file` steps and `database-steps` helpers, runs `psql` in a separate
1876
+ client container from `postgres:16`. A `pg_dump` of an external database runs
1877
+ in `postgres:<its major version>`. `TESTKIT_POSTGRES_CLIENT_IMAGE` overrides
1878
+ the image of those client containers.
1454
1879
 
1455
1880
  Template steps, schema snapshots, and `database.steps` helpers work the same
1456
1881
  under Docker Desktop, rootless Docker, and Docker-in-Docker. Clients for
@@ -1476,7 +1901,7 @@ one with `database(name)` from `@gfdelarue/testkit/environment`. Declared `env`
1476
1901
  `{dbPort}`, `{dbName}`, `{dbUser}`, `{dbPassword}`, and `{dbSsl}` tokens, with
1477
1902
  `:serviceName` to read the database of a service listed in `dependsOn`. Testkit
1478
1903
  resolves them from the database provisioned for the current run. They are
1479
- env-only; see [`env`](testkit-config-schema.md#env).
1904
+ env-only; see the `env` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
1480
1905
 
1481
1906
  `db schema refresh` forces a source database dump into the `.testkit` source
1482
1907
  schema cache. `db schema verify` prepares local templates and verifies local