@gfdelarue/testkit 0.5.0 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +590 -165
- package/bin/testkit-windows-owner-arm64.exe +0 -0
- package/bin/testkit-windows-owner-x64.exe +0 -0
- package/dist/app/bugs/batch.js +1 -0
- package/dist/app/bugs/ci-report.js +2 -2
- package/dist/app/bugs/ci-summary.js +3 -3
- package/dist/app/bugs/draft.js +2 -2
- package/dist/app/bugs/file.js +1 -1
- package/dist/app/bugs/guide.js +1 -1
- package/dist/app/bugs/record.js +2 -2
- package/dist/app/bugs/run-reading.js +1 -1
- package/dist/app/bugs/store.js +1 -1
- package/dist/app/campaigns/index.js +2 -2
- package/dist/app/campaigns/summary.js +1 -1
- package/dist/app/cleanup/index.js +1 -1
- package/dist/app/db-schema/prepare.js +1 -1
- package/dist/app/db-schema/refresh.js +1 -1
- package/dist/app/db-schema/verify.js +1 -1
- package/dist/app/destroy/index.js +1 -1
- package/dist/app/discover/index.js +2 -2
- package/dist/app/environments/config-driven.js +1 -1
- package/dist/app/environments/file-setup.js +1 -1
- package/dist/app/environments/routing.js +1 -1
- package/dist/app/environments/static-values.js +1 -1
- package/dist/app/environments/supplied.js +1 -1
- package/dist/app/inspect/index.js +1 -1
- package/dist/app/lint/native.js +1 -1
- package/dist/app/lint/options.js +1 -1
- package/dist/app/lint/shared.js +1 -1
- package/dist/app/run/engine-orchestration.js +1 -1
- package/dist/app/run/environment.js +1 -1
- package/dist/app/run/live-run.js +1 -1
- package/dist/app/run/orchestrator.js +1 -1
- package/dist/app/run/planning.js +1 -1
- package/dist/app/run/request.js +1 -1
- package/dist/app/run/results.js +1 -1
- package/dist/app/run/run-finalization.js +1 -1
- package/dist/app/run/run-finisher.js +1 -0
- package/dist/app/run/setup/runtime-node-setup-protocol.js +1 -1
- package/dist/app/run/setup/runtime-node-setup-worker.js +1 -1
- package/dist/app/run/setup/runtime-node-setup.js +2 -2
- package/dist/app/run/task-batches.js +1 -1
- package/dist/app/run/worker-loop.js +1 -1
- package/dist/app/status/model.js +1 -1
- package/dist/app/status/runs.js +1 -1
- package/dist/app/stop/index.js +1 -1
- package/dist/app/tracker/index.js +1 -1
- package/dist/app/tracker/token.js +2 -2
- package/dist/app/typecheck/index.js +4 -4
- package/dist/cli/command-catalog.js +1 -1
- package/dist/cli/command-flags.js +1 -1
- package/dist/cli/commands/artifacts.js +2 -2
- package/dist/cli/commands/bug/file.js +3 -2
- package/dist/cli/commands/discover.js +1 -1
- package/dist/cli/commands/guard/unsafe-entrypoint.js +1 -1
- package/dist/cli/commands/run.js +2 -2
- package/dist/cli/commands/typecheck.js +1 -1
- package/dist/cli/commands/workspace.js +1 -1
- package/dist/cli/framework/help.js +5 -5
- package/dist/cli/framework/parser.js +1 -1
- package/dist/cli/json-errors.js +1 -1
- package/dist/cli/renderers/bug/text.js +1 -1
- package/dist/cli/renderers/run/formatting.js +1 -1
- package/dist/config/authoring/database-fixture-discovery.js +1 -0
- package/dist/config/authoring/index.js +1 -1
- package/dist/config/loader/config-loader.js +1 -1
- package/dist/config/loader/decode-discovery.js +1 -1
- package/dist/config/loader/discover-tests.js +1 -1
- package/dist/config/loader/discovery-result.js +1 -0
- package/dist/config/loader/discovery.js +2 -2
- package/dist/config/loader/file-metadata.js +1 -1
- package/dist/config/loader/groups.js +2 -1
- package/dist/config/loader/index.js +1 -1
- package/dist/config/loader/load.js +1 -1
- package/dist/config/loader/managed-configs.js +1 -1
- package/dist/config/loader/owned-discovery.js +1 -1
- package/dist/config/loader/runtime-isolation.js +1 -1
- package/dist/config/loader/skip-config.js +1 -1
- package/dist/config/loader/test-selection.js +1 -1
- package/dist/database/admin.js +5 -2
- package/dist/database/call-caps.js +1 -0
- package/dist/database/cleanup.js +1 -1
- package/dist/database/configured-steps.js +1 -1
- package/dist/database/constants.js +1 -1
- package/dist/database/docker-endpoint.js +1 -0
- package/dist/database/docker-metadata.js +2 -2
- package/dist/database/docker.js +5 -5
- package/dist/database/index.js +2 -1
- package/dist/database/lifecycle-cleanup.js +1 -1
- package/dist/database/local-postgres.js +4 -1
- package/dist/database/naming.js +1 -1
- package/dist/database/operation-deadline.js +1 -0
- package/dist/database/postgres/connection-check.js +1 -1
- package/dist/database/postgres/index.js +2 -2
- package/dist/database/postgres/owned-exec.js +27 -0
- package/dist/database/postgres/protocol.js +2 -2
- package/dist/database/provider.js +1 -0
- package/dist/database/roles.js +2 -2
- package/dist/database/schema-operations.js +5 -1
- package/dist/database/schema-source.js +3 -3
- package/dist/database/steps.js +1 -1
- package/dist/database/template-steps.js +4 -4
- package/dist/domain/identifiers.js +1 -1
- package/dist/domain/retired-test-types.js +1 -1
- package/dist/domain/suite-selection.js +1 -1
- package/dist/domain/test-types.js +1 -1
- package/dist/engines/adapters/evidence-files.js +1 -1
- package/dist/engines/adapters/execution.js +1 -1
- package/dist/engines/catalog.js +1 -1
- package/dist/engines/definitions.js +1 -1
- package/dist/engines/environment/index.js +1 -1
- package/dist/engines/installations/node-vitest.js +1 -1
- package/dist/engines/installations/playwright.js +1 -1
- package/dist/engines/managed-module-resolution.js +6 -2
- package/dist/engines/node-vitest/adapter/adapter.js +1 -1
- package/dist/engines/node-vitest/adapter/driver.js +2 -2
- package/dist/engines/node-vitest/adapter/evidence.js +2 -2
- package/dist/engines/node-vitest/adapter/workspace.js +8 -24
- package/dist/engines/node-vitest/inventory/index.js +2 -2
- package/dist/engines/node-vitest/inventory/list-files.js +3 -3
- package/dist/engines/node-vitest/runtime/data/index.js +1 -1
- package/dist/engines/node-vitest/runtime/data/report.js +1 -1
- package/dist/engines/node-vitest/runtime/data/scenario.js +2 -2
- package/dist/engines/node-vitest/runtime/runtime-input.js +1 -1
- package/dist/engines/playwright/adapter/adapter.js +1 -1
- package/dist/engines/playwright/adapter/browser-network-evidence.js +4 -0
- package/dist/engines/playwright/adapter/config.js +16 -6
- package/dist/engines/playwright/adapter/evidence.js +2 -1
- package/dist/engines/playwright/adapter/failure-details.js +2 -2
- package/dist/engines/playwright/adapter/reporter.js +1 -1
- package/dist/engines/playwright/runtime/browser-network.js +1 -0
- package/dist/engines/playwright/runtime/browser-traffic.js +1 -0
- package/dist/engines/playwright/runtime/chromium-environment.js +1 -0
- package/dist/engines/playwright/runtime/emulation.js +1 -0
- package/dist/engines/playwright/runtime/index.js +1 -1
- package/dist/engines/playwright/runtime/managed-test.js +1 -0
- package/dist/engines/playwright/runtime/runtime-input.js +2 -2
- package/dist/environment/lifecycle-controller.js +1 -1
- package/dist/environment/lifecycle.js +1 -1
- package/dist/environment/maintenance.js +1 -1
- package/dist/environment/run-environment.js +1 -1
- package/dist/environment/run-stopping.js +1 -1
- package/dist/environment/runtime-manager.js +1 -1
- package/dist/evidence/artifact-config.js +1 -1
- package/dist/evidence/artifacts.js +5 -2
- package/dist/evidence/bundles.js +2 -1
- package/dist/evidence/compressed-artifacts.js +1 -0
- package/dist/evidence/environment-record.js +1 -1
- package/dist/evidence/index.js +1 -1
- package/dist/evidence/live-run-summary.js +1 -0
- package/dist/evidence/local-paths.js +1 -0
- package/dist/evidence/maintenance.js +1 -0
- package/dist/evidence/operation-timings.js +1 -1
- package/dist/evidence/read-run.js +1 -1
- package/dist/evidence/redaction.js +4 -4
- package/dist/evidence/run-evidence.js +1 -0
- package/dist/evidence/setup-progress.js +1 -1
- package/dist/evidence/storage-inventory.js +1 -1
- package/dist/evidence/storage.js +2 -2
- package/dist/kernel/known-keys.js +1 -0
- package/dist/kernel/redaction-targets.js +1 -1
- package/dist/kernel/redaction.js +7 -4
- package/dist/process/engine-environment-keys.js +1 -0
- package/dist/process/environment-overlay.js +1 -1
- package/dist/process/environment.js +1 -1
- package/dist/process/execa-bundle.json +3 -3
- package/dist/process/files/evidence-writes.js +1 -0
- package/dist/process/files/product-dir.js +1 -1
- package/dist/process/host-process.js +1 -1
- package/dist/process/index.js +1 -1
- package/dist/process/lifecycle-fence.js +1 -1
- package/dist/process/lock-progress.js +2 -0
- package/dist/process/locks.js +4 -4
- package/dist/process/process-identity.js +1 -1
- package/dist/process/process.js +35 -35
- package/dist/process/recovery.js +1 -1
- package/dist/process/runs/manifest.js +1 -1
- package/dist/process/runs/ownership.js +1 -1
- package/dist/process/runs/stop.js +1 -1
- package/dist/process/shutdown.js +1 -1
- package/dist/process/windows-native.js +1 -1
- package/dist/repo/fingerprint-policy.js +1 -1
- package/dist/repo/history.js +7 -4
- package/dist/repo/managed-git.js +1 -1
- package/dist/repo/state.js +4 -4
- package/dist/steps/configured-step-testkit-modules.js +1 -1
- package/dist/steps/environment.js +1 -1
- package/dist/toolchains/index.js +2 -2
- package/dist/toolchains/node-provisioning.js +2 -2
- package/dist/toolchains/vitest/index.js +1 -1
- package/dist/tracker/gitea.js +1 -1
- package/dist/tracker/github.js +1 -1
- package/dist/tracker/http.js +1 -1
- package/dist/tracker/ownership.js +1 -0
- package/dist/tracker/redaction.js +1 -1
- package/dist/tracker/responses.js +1 -1
- package/dist/types/config.d.ts +7 -86
- package/dist/types/environment.d.ts +6 -0
- package/dist/types/node-vitest-data.d.ts +2 -218
- package/dist/types/playwright.d.ts +21 -0
- package/dist/types/workspace.d.ts +30 -14
- package/docs/bug-records.md +19 -18
- package/docs/engines.md +115 -21
- package/docs/release-notes.md +2846 -0
- package/docs/test-types.md +19 -69
- package/package.json +15 -6
- package/testkit-config-schema.md +101 -170
- package/dist/app/lint/rules/data.js +0 -1
- package/dist/config/authoring/runtime.js +0 -1
- package/dist/config/loader/task-policy.js +0 -1
- package/dist/engines/node-vitest/runtime/contract/index.js +0 -1
- package/dist/engines/node-vitest/runtime/data/http.js +0 -1
- package/dist/engines/node-vitest/runtime/data/runner.js +0 -1
- package/dist/environment/lifecycle-artifacts.js +0 -1
- package/dist/types/node-vitest-contract.d.ts +0 -75
package/README.md
CHANGED
|
@@ -43,19 +43,25 @@ npm run testkit
|
|
|
43
43
|
|
|
44
44
|
The command exits zero only when at least one test file completed and every
|
|
45
45
|
required file passed. Test failures, infrastructure failures, interrupted or
|
|
46
|
-
not-run files,
|
|
47
|
-
|
|
46
|
+
not-run files, a skipped file, and an empty selection all fail the gate. A file
|
|
47
|
+
that a `skip` rule or its own `skip` metadata skips fails the gate even when
|
|
48
|
+
every other file passes, and `--allow-empty` doesn't change that. `npm run testkit -- --allow-empty` is an explicit escape hatch for a job
|
|
48
49
|
whose empty selection is intentional; it should not be the default CI command.
|
|
49
50
|
|
|
50
51
|
Testkit writes the immutable, verifiable run bundle under
|
|
51
52
|
`.testkit/results/runs/<run-id>/`. Upload `.testkit/results` with `if: always()`
|
|
52
|
-
when CI evidence should survive a failed job.
|
|
53
|
+
when CI evidence should survive a failed job. If the job can kill Testkit before
|
|
54
|
+
it seals the run, run `testkit cleanup` before the upload, so the run is sealed
|
|
55
|
+
and redacted first. See
|
|
56
|
+
[Read run evidence without the CLI](#read-run-evidence-without-the-cli).
|
|
53
57
|
|
|
54
58
|
## Report a run to GitHub or Gitea
|
|
55
59
|
|
|
56
60
|
`testkit ci report` reads the last run in the checkout and posts it in three
|
|
57
61
|
places: one commit status, the job's step summary, and one pull request comment
|
|
58
|
-
that it updates in place.
|
|
62
|
+
that it updates in place. It updates only a comment that its token's user
|
|
63
|
+
wrote, and posts a new comment when the tracker won't say who that user is. It
|
|
64
|
+
leaves out a CI run or issue link that holds credentials or a secret.
|
|
59
65
|
|
|
60
66
|
```yaml
|
|
61
67
|
permissions:
|
|
@@ -79,10 +85,17 @@ steps:
|
|
|
79
85
|
it, as the `env` lines above do. Gitea Actions accepts the same lines.
|
|
80
86
|
- Testkit reads the token from `GITHUB_TOKEN` or `GH_TOKEN` for GitHub, and
|
|
81
87
|
from `GITEA_TOKEN` for Gitea. In a Gitea Actions job it also reads
|
|
82
|
-
`GITHUB_TOKEN`.
|
|
83
|
-
that `
|
|
84
|
-
|
|
85
|
-
|
|
88
|
+
`GITHUB_TOKEN`. Outside such a job, it sends `GITEA_TOKEN` only to the host
|
|
89
|
+
that `GITEA_SERVER_URL` names, such as `https://gitea.example.com`. With no
|
|
90
|
+
token set, it asks `git credential fill`. The token that `actions/checkout`
|
|
91
|
+
keeps in the checkout does not answer that request.
|
|
92
|
+
- Testkit sends a token only over HTTPS. A plain `http://` remote or repository
|
|
93
|
+
URL works only for a loopback host such as `127.0.0.1`.
|
|
94
|
+
- A job's token reaches only the job's own repository. When `bugs.repository`
|
|
95
|
+
names another one, give the step a token that may write to it.
|
|
96
|
+
- Testkit reaches GitHub Enterprise Server, and GitHub Enterprise Cloud with
|
|
97
|
+
data residency (`*.ghe.com`), only from a GitHub Actions job on that host, so
|
|
98
|
+
`testkit bug file` works there only in a job.
|
|
86
99
|
- GitHub gives a pull request from a fork a read-only token. `ci report` then
|
|
87
100
|
writes the step summary and fails when it posts the status. Add
|
|
88
101
|
`continue-on-error: true` to the step when the repository takes pull requests
|
|
@@ -98,7 +111,7 @@ steps:
|
|
|
98
111
|
Testkit files a confirmed bug as an issue in the repository's GitHub or Gitea.
|
|
99
112
|
The issue names the regression test that fails because of the bug, the commit
|
|
100
113
|
that introduced the faulty lines, and the command that reproduces the failure.
|
|
101
|
-
[
|
|
114
|
+
[`docs/bug-records.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/bug-records.md) gives the issue format.
|
|
102
115
|
|
|
103
116
|
1. Write a Testkit test that fails because of the bug.
|
|
104
117
|
2. Write the bug record as a JSON file. `testkit bug guide` prints the fields
|
|
@@ -110,8 +123,15 @@ that introduced the faulty lines, and the command that reproduces the failure.
|
|
|
110
123
|
|
|
111
124
|
`testkit bug file` runs the test at the checked-out commit and files the issue
|
|
112
125
|
only when the test fails. It stops when tracked files have uncommitted changes,
|
|
113
|
-
when no remote
|
|
114
|
-
the test.
|
|
126
|
+
when no branch on the remote contains the commit, or when an open issue already
|
|
127
|
+
covers the test. It asks the remote which branches hold the commit, with the
|
|
128
|
+
checkout's Git credentials, so a branch that was rewound or deleted there
|
|
129
|
+
doesn't count. When it can't tell, because a remote doesn't answer or a shallow
|
|
130
|
+
clone's history stops short, it stops and says why. After it files, it reads the open issues again. If another checkout
|
|
131
|
+
filed the same test at the same time and its issue is older, Testkit closes its
|
|
132
|
+
own issue and stops, naming the older one. GitHub can list a new issue a few
|
|
133
|
+
seconds late, so Testkit waits up to 7.5 seconds for its own and says so when
|
|
134
|
+
it doesn't appear. The token needs permission to create issues and labels.
|
|
115
135
|
|
|
116
136
|
`testkit bug record` saves the `--vitest-config`, `--vitest-project`, and
|
|
117
137
|
`--vitest-runner` it was given in the draft, and asks for `--vitest-project`
|
|
@@ -130,6 +150,38 @@ fragment. It refuses `--database` and `--env`; pass those values through
|
|
|
130
150
|
root, so a relative `--environment-file` needs `testkit bug file` run from
|
|
131
151
|
there too.
|
|
132
152
|
|
|
153
|
+
`testkit bug file --batch` files several drafts in one call: the ids it is
|
|
154
|
+
given, or every unfiled draft under `.testkit/bugs/` when it is given none.
|
|
155
|
+
|
|
156
|
+
- It checks the checkout once and reads the open issues before it runs the
|
|
157
|
+
tests. It stops before it files anything when tracked files have uncommitted
|
|
158
|
+
changes, when no remote branch contains the commit, or when it cannot read
|
|
159
|
+
the open issues.
|
|
160
|
+
- It runs the drafts' tests in one `testkit run`, or one run for each
|
|
161
|
+
`--vitest-config`, `--vitest-project`, and `--vitest-runner` the drafts were
|
|
162
|
+
recorded with. Each draft is judged by its own file's result, so a file that
|
|
163
|
+
is skipped or fails outside its tests rules out only its own draft.
|
|
164
|
+
- It reads the open issues again before it files, in case another checkout
|
|
165
|
+
filed one of the tests meanwhile.
|
|
166
|
+
- It reports one line per draft: filed, already filed, rejected with the
|
|
167
|
+
reason, or not filed.
|
|
168
|
+
- A draft is rejected when its own checks fail, for example when its test
|
|
169
|
+
passes or an open issue already covers the test, and the batch carries on.
|
|
170
|
+
- When the tracker refuses a request, the batch stops filing and reports the
|
|
171
|
+
drafts it has not reached as not filed.
|
|
172
|
+
- It exits non-zero when any draft is rejected or not filed. `--json` returns
|
|
173
|
+
the same list.
|
|
174
|
+
|
|
175
|
+
Each draft records its issue as soon as the tracker returns it, so running the
|
|
176
|
+
batch again files only what is left. An open issue whose facts match the ones a
|
|
177
|
+
draft would file counts as that draft's issue, so a batch stopped between the
|
|
178
|
+
two steps does not file the bug twice. When the tracker creates an issue
|
|
179
|
+
without its labels, the batch stops, the draft stays unfiled, and Testkit
|
|
180
|
+
closes that issue. If it can't, the message says so, and you close the issue.
|
|
181
|
+
Then run the batch again with a token that may label issues. The batch creates
|
|
182
|
+
at most one issue a second. When the tracker limits the rate, it stops and
|
|
183
|
+
says when to run it again.
|
|
184
|
+
|
|
133
185
|
### Name the product's areas
|
|
134
186
|
|
|
135
187
|
Add `areas` to `testkit.config.ts` so that each issue names the part of the
|
|
@@ -146,7 +198,7 @@ export default defineConfig({
|
|
|
146
198
|
|
|
147
199
|
`testkit bug file` puts an `area:<id>` label on the issue. If no area claims
|
|
148
200
|
the fault's file, it stops and names the file. Without `areas`, Testkit files
|
|
149
|
-
bugs with no area. See
|
|
201
|
+
bugs with no area. See the `areas` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
|
|
150
202
|
|
|
151
203
|
### Pass a run whose failures are all filed
|
|
152
204
|
|
|
@@ -162,7 +214,8 @@ export default defineConfig({
|
|
|
162
214
|
A run then passes when an open issue with the `testkit` label covers each
|
|
163
215
|
failure. A new failure still fails the run, and closing an issue makes its
|
|
164
216
|
test count again. When Testkit cannot read the issues, every failure is new.
|
|
165
|
-
|
|
217
|
+
The Known failures section of [`docs/bug-records.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/bug-records.md) gives the
|
|
218
|
+
exact rule.
|
|
166
219
|
|
|
167
220
|
With `"allow"`, the run reads the open issues. In CI, pass the job's token to
|
|
168
221
|
the `testkit run` step as well:
|
|
@@ -174,7 +227,9 @@ steps:
|
|
|
174
227
|
GITHUB_TOKEN: ${{ github.token }}
|
|
175
228
|
```
|
|
176
229
|
|
|
177
|
-
The job also needs the `issues: read` permission.
|
|
230
|
+
The job also needs the `issues: read` permission. Testkit reads
|
|
231
|
+
`GITHUB_TOKEN`, `GH_TOKEN` and `GITEA_TOKEN` itself and doesn't pass them to
|
|
232
|
+
services or setup steps. A service that needs one declares it in its `env`.
|
|
178
233
|
|
|
179
234
|
### Run a campaign
|
|
180
235
|
|
|
@@ -264,8 +319,7 @@ the report to the chosen platform.
|
|
|
264
319
|
|
|
265
320
|
Testkit has two built-in engines:
|
|
266
321
|
|
|
267
|
-
- `node-vitest` runs plain Vitest test files
|
|
268
|
-
`contract` files with Node and Vitest.
|
|
322
|
+
- `node-vitest` runs plain Vitest test files with Node and Vitest.
|
|
269
323
|
- `playwright` runs `ui` files with Playwright and managed Chromium.
|
|
270
324
|
|
|
271
325
|
A plain Vitest test file needs no Testkit suffix, directory, metadata, or
|
|
@@ -293,17 +347,16 @@ npx @gfdelarue/testkit setup playwright --with-deps # CI and fresh hosts
|
|
|
293
347
|
```
|
|
294
348
|
|
|
295
349
|
Normal runs never install or download a browser.
|
|
296
|
-
[
|
|
297
|
-
the supported platforms, and how to recover a broken
|
|
298
|
-
|
|
299
|
-
|
|
350
|
+
The Installation section of [`docs/engines.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/engines.md) covers
|
|
351
|
+
what an install contains, the supported platforms, and how to recover a broken
|
|
352
|
+
one. Its Typechecking section covers the supported TypeScript versions and
|
|
353
|
+
`tsconfig.playwright.json`.
|
|
300
354
|
|
|
301
355
|
Testkit bounds the output it captures from each command it runs, and asks the
|
|
302
356
|
command and its descendants to stop on cancellation or timeout.
|
|
303
357
|
|
|
304
|
-
Managed task environment is engine input.
|
|
305
|
-
|
|
306
|
-
same contract from `readPlaywrightContext()?.env`. Managed Node processes also
|
|
358
|
+
Managed task environment is engine input. Playwright suites read it from
|
|
359
|
+
`readPlaywrightContext()?.env`. Managed Node processes also
|
|
307
360
|
expose each value as an immutable, non-enumerable `process.env.KEY` property so
|
|
308
361
|
libraries that read `process.env` directly see it, but non-Node child processes
|
|
309
362
|
do not receive consumer values unless the suite explicitly passes them.
|
|
@@ -315,17 +368,20 @@ reserves `APPDATA`, `ComSpec`, `HOME`, `LANG`, `LC_ALL`, `LOCALAPPDATA`, `PATH`,
|
|
|
315
368
|
|
|
316
369
|
- Node/Vitest reserves `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR`,
|
|
317
370
|
`UPDATE_SNAPSHOT`, `VITEST_POOL_ID`, `VITEST_WORKER_ID`, and the `TESTKIT_`
|
|
318
|
-
keys it sets itself, listed in
|
|
319
|
-
[
|
|
371
|
+
keys it sets itself, listed in the Environment section of
|
|
372
|
+
[`docs/engines.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/engines.md);
|
|
320
373
|
other `TESTKIT_` and `VITEST_` keys, such as
|
|
321
|
-
`TESTKIT_REQUIRE_COMPLETE_SUITE`, pass through
|
|
374
|
+
`TESTKIT_REQUIRE_COMPLETE_SUITE`, pass through, though Vitest's own `TEST`,
|
|
375
|
+
`VITEST`, `VITEST_MODE`, `FORCE_TTY`, `MODE`, `DEV`, `PROD`, and `SSR`
|
|
376
|
+
replace a group's values;
|
|
322
377
|
- Playwright reserves the `TESTKIT_`, `PLAYWRIGHT_`, `PWTEST_`, and `PW_TEST_`
|
|
323
|
-
prefixes plus `BASE_URL`, `
|
|
378
|
+
prefixes plus `BASE_URL`, `DBUS_SESSION_BUS_ADDRESS`, `FC_FONTATIONS`,
|
|
379
|
+
`FORCE_COLOR`, `NODE_OPTIONS`, and `NO_COLOR`.
|
|
324
380
|
|
|
325
381
|
Case-variant duplicates and invalid names or NUL-containing values fail instead
|
|
326
382
|
of being normalized. Node/Vitest accepts at most 512 entries and
|
|
327
383
|
4 MiB of UTF-8 key/value data. Playwright accepts at most 512 entries and 2 MiB
|
|
328
|
-
of UTF-8 key/value data. [
|
|
384
|
+
of UTF-8 key/value data. The Environment section of [`docs/engines.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/engines.md) lists the
|
|
329
385
|
complete limits, and [Named endpoints and databases](#named-endpoints-and-databases)
|
|
330
386
|
covers the environment API.
|
|
331
387
|
|
|
@@ -344,12 +400,10 @@ npx @gfdelarue/testkit discover --json > .testkit/discovery.json
|
|
|
344
400
|
npx @gfdelarue/testkit discover --output .testkit/discovery.json
|
|
345
401
|
|
|
346
402
|
# Filter by type
|
|
347
|
-
npx @gfdelarue/testkit --type contract
|
|
348
|
-
npx @gfdelarue/testkit --type data
|
|
349
|
-
npx @gfdelarue/testkit --type contract,data
|
|
350
403
|
npx @gfdelarue/testkit --type ui
|
|
404
|
+
npx @gfdelarue/testkit --type plain
|
|
351
405
|
|
|
352
|
-
#
|
|
406
|
+
# Run up to 8 test tasks at once
|
|
353
407
|
npx @gfdelarue/testkit --task-concurrency 8
|
|
354
408
|
|
|
355
409
|
# One file-level wall clock budget for every suite file
|
|
@@ -357,15 +411,16 @@ npx @gfdelarue/testkit --file-timeout-seconds 60
|
|
|
357
411
|
|
|
358
412
|
# Specific group / suite
|
|
359
413
|
npx @gfdelarue/testkit --group frontend --type ui -s navigation
|
|
360
|
-
npx @gfdelarue/testkit --group api
|
|
361
|
-
npx @gfdelarue/testkit --
|
|
414
|
+
npx @gfdelarue/testkit --group api -s health
|
|
415
|
+
npx @gfdelarue/testkit --group api --type plain
|
|
416
|
+
npx @gfdelarue/testkit --type ui -s ui:checkout
|
|
362
417
|
|
|
363
418
|
# Exact file
|
|
364
|
-
npx @gfdelarue/testkit --type
|
|
419
|
+
npx @gfdelarue/testkit --type ui --file __testkit__/checkout.ui.testkit.ts
|
|
365
420
|
npx @gfdelarue/testkit run --file src/sum.test.ts
|
|
366
421
|
|
|
367
422
|
# Temporarily ignore repo-declared skip rules
|
|
368
|
-
npx @gfdelarue/testkit --ignore-skip-rules --file
|
|
423
|
+
npx @gfdelarue/testkit --ignore-skip-rules --file src/billing.test.ts
|
|
369
424
|
|
|
370
425
|
# Lifecycle
|
|
371
426
|
npx @gfdelarue/testkit status
|
|
@@ -380,15 +435,15 @@ npx @gfdelarue/testkit db schema refresh --service api
|
|
|
380
435
|
npx @gfdelarue/testkit db schema verify --service api
|
|
381
436
|
```
|
|
382
437
|
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
with `--type all`,
|
|
386
|
-
|
|
438
|
+
`--type ui` and typed suite selectors such as `ui:checkout` select typed files
|
|
439
|
+
only. `--type plain` selects plain Vitest test files only. A run without
|
|
440
|
+
`--type`, or with `--type all`, runs both. `--type` narrows the other filters,
|
|
441
|
+
so `--group api --type plain` runs the `api` group's plain files and none of its
|
|
442
|
+
UI files.
|
|
387
443
|
|
|
388
|
-
`run`, `discover`, `artifacts`, and `
|
|
389
|
-
`--group`.
|
|
390
|
-
|
|
391
|
-
take `--service` to name a service.
|
|
444
|
+
`run`, `discover`, `artifacts`, `workspace`, and `typecheck` select test groups
|
|
445
|
+
with `--group`. `status`, `destroy`, `cleanup`, and `db schema` take `--service`
|
|
446
|
+
to name a service.
|
|
392
447
|
|
|
393
448
|
Database cleanup is scoped to the selected product. `cleanup --resources --global`
|
|
394
449
|
is retired; use `--dir <product> cleanup --resources` for each product. After
|
|
@@ -396,10 +451,13 @@ deleting a product directory, remove its orphaned containers and volumes
|
|
|
396
451
|
explicitly through Docker.
|
|
397
452
|
|
|
398
453
|
Use `execution.taskConcurrency` in config and `--task-concurrency` on the CLI.
|
|
399
|
-
|
|
400
|
-
|
|
454
|
+
A task is a typed file, a batch of typed files, or one Vitest invocation of
|
|
455
|
+
plain files. Plain files in one invocation run with Vitest's file parallelism,
|
|
456
|
+
which `taskConcurrency` doesn't limit.
|
|
457
|
+
`execution.workers`, `presets.nodeNext({ workers })` and `--workers` are
|
|
458
|
+
deprecated aliases. When both names are
|
|
401
459
|
provided in config or on the CLI, their values must match. The CLI value overrides
|
|
402
|
-
config after those checks. See the
|
|
460
|
+
config after those checks. See the `execution` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
|
|
403
461
|
|
|
404
462
|
Batch `run` output stays intentionally short: one line per completed file,
|
|
405
463
|
concise failure blocks, and a final summary. Service logs, captured runtime
|
|
@@ -495,7 +553,7 @@ The environment file is JSON:
|
|
|
495
553
|
"handles": {
|
|
496
554
|
"webhook": {
|
|
497
555
|
"kind": "http-capture",
|
|
498
|
-
"url": "http://127.0.0.1:41000/webhook",
|
|
556
|
+
"url": "http://127.0.0.1:41000/__testkit__/<segment>/webhook",
|
|
499
557
|
"requestsUrl": "http://127.0.0.1:41000/__testkit__/<capability>/requests",
|
|
500
558
|
"resetUrl": "http://127.0.0.1:41000/__testkit__/<capability>/requests"
|
|
501
559
|
}
|
|
@@ -586,7 +644,7 @@ assertLocalDatabaseUrl(process.env, "seed.ts");
|
|
|
586
644
|
|
|
587
645
|
## Setup
|
|
588
646
|
|
|
589
|
-
See
|
|
647
|
+
See [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md) for the complete,
|
|
590
648
|
strict `testkit.config.ts` contract.
|
|
591
649
|
|
|
592
650
|
Create `testkit.config.ts` at repo root:
|
|
@@ -672,7 +730,7 @@ neither:
|
|
|
672
730
|
import { defineFile } from "@gfdelarue/testkit";
|
|
673
731
|
|
|
674
732
|
export const testkit = defineFile({
|
|
675
|
-
engine: "
|
|
733
|
+
engine: "playwright",
|
|
676
734
|
skip: "Billing is currently unavailable locally",
|
|
677
735
|
});
|
|
678
736
|
```
|
|
@@ -682,15 +740,10 @@ A file can also name `locks`.
|
|
|
682
740
|
one.
|
|
683
741
|
|
|
684
742
|
In a typed file, the engine value is a required literal. The supported public
|
|
685
|
-
authoring
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
from Testkit's managed `vitest`, `@playwright/test`, `playwright`, and
|
|
690
|
-
`playwright-core` packages.
|
|
691
|
-
There are no legacy aliases or implicit-runner compatibility paths. A typed
|
|
692
|
-
Node/Vitest file resolves `paths` from its nearest `tsconfig.json`, through
|
|
693
|
-
Vite's `resolve.tsconfigPaths`.
|
|
743
|
+
authoring path is `@gfdelarue/testkit/playwright`. Playwright suites may also
|
|
744
|
+
import exact public APIs directly from Testkit's managed `@playwright/test`,
|
|
745
|
+
`playwright`, and `playwright-core` packages.
|
|
746
|
+
There are no legacy aliases or implicit-runner compatibility paths.
|
|
694
747
|
|
|
695
748
|
`testkit.config.ts` is optional. Without it, Testkit runs every test file with
|
|
696
749
|
zero setup: it starts no server, provisions no database, and loads no `.env`
|
|
@@ -727,7 +780,8 @@ setup processes, so set it deliberately.
|
|
|
727
780
|
Testkit-owned Docker resources are labelled at creation time so they can be
|
|
728
781
|
found again even if product-local `.testkit` state is deleted. `testkit destroy`
|
|
729
782
|
removes labelled resources for the current product after service state is
|
|
730
|
-
destroyed.
|
|
783
|
+
destroyed. It leaves run results under `.testkit/results` and other files
|
|
784
|
+
under `.testkit`, such as the compiled config and timings, in place. `testkit cleanup --resources` cleans stale resources for the current
|
|
731
785
|
product. Both commands remove only the current product's resources and leave
|
|
732
786
|
other products' alone, including their idle caches. Host-local Postgres data
|
|
733
787
|
lives in a Testkit-labelled named Docker volume, so cleanup can find and remove
|
|
@@ -735,6 +789,10 @@ it. Database changes for one product run one at a time; other products don't
|
|
|
735
789
|
wait for them. Use `cleanup --resources --dry-run` to preview the selected
|
|
736
790
|
product's cleanup.
|
|
737
791
|
|
|
792
|
+
A run's start and its post-run maintenance don't list or reclaim Docker
|
|
793
|
+
resources. A run stops its retained database by name when it releases the run's
|
|
794
|
+
resources.
|
|
795
|
+
|
|
738
796
|
`database.template` is the database-side equivalent for reusable template DB
|
|
739
797
|
state. When `database.sourceSchema` is configured, Testkit treats the configured
|
|
740
798
|
source database as the schema source of truth. A normal `testkit run` resolves a
|
|
@@ -845,6 +903,8 @@ services: {
|
|
|
845
903
|
}
|
|
846
904
|
```
|
|
847
905
|
|
|
906
|
+
## Run evidence
|
|
907
|
+
|
|
848
908
|
Every terminal run is sealed under `.testkit/results/runs/<run-id>/` with a
|
|
849
909
|
canonical `run.json`, `provenance.json`, `manifest.json`, and the run's exact
|
|
850
910
|
artifact/log/setup files. Payloads are SHA-256 addressed under
|
|
@@ -870,25 +930,56 @@ workspace API.
|
|
|
870
930
|
Sealing replaces secrets with `<redacted>` in the run record and in every text
|
|
871
931
|
file of the bundle. A key names a secret when it holds `PASSWORD`, `TOKEN`,
|
|
872
932
|
`SECRET`, `PRIVATE_KEY`, `AUTH`, `COOKIE`, `SESSION`, or `API_KEY` in any letter
|
|
873
|
-
case
|
|
933
|
+
case, or when its last word is `PASS`, `PASSWD`, `PAT`, `DSN`, `CREDENTIAL`, or
|
|
934
|
+
`CREDENTIALS`. `AUTH` doesn't count in `AUTHOR` or `AUTHORITY`, so
|
|
935
|
+
`GIT_AUTHOR_NAME` names no secret, though `AUTHORIZATION` still does. `SESSION`
|
|
936
|
+
doesn't count before `COUNT`, `DURATION`, `EXPIR`, `LENGTH`, `LIMIT`, `MAX`,
|
|
937
|
+
`TIMEOUT`, or `TTL`, as in `sessionCount` and `SESSION_TIMEOUT`. A key whose
|
|
938
|
+
last two words are `API`, `PRIVATE`, `SIGNING`, or `ENCRYPTION` and then `KEY`,
|
|
939
|
+
as in `apiKey`, `x-api-key`, and `privateKey`, names a secret too, and so does
|
|
940
|
+
an upper-case key whose last word is `KEY`, such as `STRIPE_KEY`, unless the
|
|
941
|
+
word before it is `PUBLIC` or `PUBLISHABLE`. Words break at any character other
|
|
942
|
+
than a letter or a digit, and before an upper-case letter that follows a
|
|
943
|
+
lower-case letter or a digit. Sealing redacts:
|
|
874
944
|
|
|
875
945
|
- A `KEY=value` assignment under such a key, an `Authorization`,
|
|
876
|
-
`Proxy-Authorization`, `Cookie`,
|
|
877
|
-
value under such a key, even when the key is
|
|
878
|
-
single-quoted
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
946
|
+
`Proxy-Authorization`, `Cookie`, `Set-Cookie`, `Api-Key`, or `X-Api-Key`
|
|
947
|
+
header value, a JSON string value under such a key, even when the key is
|
|
948
|
+
spelled with escapes, a single-quoted string value under such a key, a token
|
|
949
|
+
of at least eight characters with a digit after `Bearer`, and a URL's
|
|
950
|
+
password, wherever they appear. A
|
|
951
|
+
quoted value ends at its closing quote and keeps its quotes, and a value that
|
|
952
|
+
starts with an auth scheme such as `Bearer` ends with its line. A secret in
|
|
953
|
+
one JSON value leaves the record's other values as they are.
|
|
954
|
+
- Every string in a JSON array or object under such a key, unless the key
|
|
955
|
+
names a secret only through `AUTH`, `COOKIE`, or `SESSION`, so `tokens`,
|
|
956
|
+
`credentials`, and `privateKey` count and `session` and `auth` settings stay.
|
|
957
|
+
In an object with a `name` string and a `value` string, only the `value` and
|
|
958
|
+
members whose own key names a secret count. Numbers and booleans stay.
|
|
959
|
+
- The value after `password:` or `passwd:`, or after a key whose last word is
|
|
960
|
+
one of those: a quoted value anywhere, as in `{ password: 'hunter2' }`, and
|
|
961
|
+
an unquoted value at the start of a line, as in YAML, when it is the line's
|
|
962
|
+
one word. A value that is one word of letters, perhaps followed by
|
|
963
|
+
punctuation such as `!` or `.`, stays, and so does an unquoted value after
|
|
964
|
+
other text, so `password: 'Required'`, `password: required!`, and
|
|
965
|
+
`rejects a password: too short` stay.
|
|
966
|
+
- A token of at least 20 characters with a digit that stands alone as the user
|
|
967
|
+
of an `http` or `https` URL, as in `https://ghp_…@github.com`. A URL's
|
|
968
|
+
password may hold `/` and `@`, and sealing also finds it in a directory name,
|
|
969
|
+
which keeps one slash of the URL, as in `postgres:/app:…@db`.
|
|
970
|
+
- A header or cookie stored as a `{"name": …, "value": …}` pair, as Playwright
|
|
971
|
+
traces and HAR files store them: the value when the name names a secret, and
|
|
972
|
+
every value in a `cookies` array.
|
|
882
973
|
- The value of each such key, and of each key that `redact` names, in the
|
|
883
|
-
environment Testkit runs in. When the value is a URL, its password
|
|
884
|
-
too.
|
|
974
|
+
environment Testkit runs in. When the value is a URL, its password, or a
|
|
975
|
+
token that stands alone as its user, counts too.
|
|
885
976
|
- The values the run's environment resolved: each value under such a key or a
|
|
886
|
-
key that `redact` names, and the password of each URL. A
|
|
887
|
-
password may also be a `password` query parameter, as libpq
|
|
888
|
-
covers config `env`, env files, `local.env`, dependency
|
|
889
|
-
readiness, UI, and database URLs, `--env`, `--env-from`, and
|
|
890
|
-
`--environment-file`. A value under such a key that is a URL counts only for
|
|
891
|
-
its password.
|
|
977
|
+
key that `redact` names, and the password or token user of each URL. A
|
|
978
|
+
PostgreSQL URL's password may also be a `password` query parameter, as libpq
|
|
979
|
+
reads it. This covers config `env`, env files, `local.env`, dependency
|
|
980
|
+
services, base, readiness, UI, and database URLs, `--env`, `--env-from`, and
|
|
981
|
+
an `--environment-file`. A value under such a key that is a URL counts only for
|
|
982
|
+
its password or token user.
|
|
892
983
|
- The unguessable segment in each [HTTP capture's](#handles) operation URLs,
|
|
893
984
|
from a setup module's capture or an `--environment-file` handle.
|
|
894
985
|
|
|
@@ -910,12 +1001,41 @@ and keys in the free-form data and metadata that tests attach. When output cuts
|
|
|
910
1001
|
the cut with `…` or `...`, sealing redacts the part that shows. `testkit run
|
|
911
1002
|
--json` and the run's progress output apply the same redaction.
|
|
912
1003
|
|
|
1004
|
+
Sealing also redacts inside `.zip` and `.gz` files, and files that start like
|
|
1005
|
+
one, such as a Playwright `trace.zip`. It applies the rules above to each text
|
|
1006
|
+
entry, such as a trace's actions, network log, and page and API responses, and
|
|
1007
|
+
writes the archive again with the redacted entries. A text entry is valid UTF-8
|
|
1008
|
+
without binary control characters, or UTF-16 with a byte order mark. Other
|
|
1009
|
+
entries keep their bytes. A ZIP with a comment or extra fields is written again
|
|
1010
|
+
without them, and a gzip file is always written again without the name and
|
|
1011
|
+
comment in its header. A trace stays readable in the Playwright trace viewer,
|
|
1012
|
+
with `<redacted>` where a typed password or a token was.
|
|
1013
|
+
|
|
1014
|
+
Sealing refuses an archive it can't redact:
|
|
1015
|
+
|
|
1016
|
+
- one it can't read: encrypted, needing ZIP64 sizes, compressed with a method
|
|
1017
|
+
other than deflate, larger than 512 MiB once expanded, or damaged, including
|
|
1018
|
+
a ZIP with bytes that none of its entries account for;
|
|
1019
|
+
- one with a binary entry, such as a screenshot, that holds one of the values
|
|
1020
|
+
above, as UTF-8 or UTF-16;
|
|
1021
|
+
- one that holds another archive.
|
|
1022
|
+
|
|
1023
|
+
Sealing removes each file it refuses from the run's results and fails the run
|
|
1024
|
+
with the reason, for example:
|
|
1025
|
+
|
|
1026
|
+
```text
|
|
1027
|
+
Refusing to archive compressed file whose entry inner/trace.zip is itself an archive, and removed it: …/.testkit/results/artifacts/frontend/task-1-01-bundle.zip
|
|
1028
|
+
```
|
|
1029
|
+
|
|
1030
|
+
The run, or the `testkit stop`, `testkit cleanup` or next run that seals it
|
|
1031
|
+
later, then seals the rest of its results and records the reason.
|
|
1032
|
+
|
|
913
1033
|
Some secrets stay visible:
|
|
914
1034
|
|
|
915
1035
|
- What a setup module prints goes to the terminal as written.
|
|
916
|
-
- Sealing does not
|
|
917
|
-
|
|
918
|
-
above
|
|
1036
|
+
- Sealing does not redact images or video, including the screenshots in a
|
|
1037
|
+
trace. Text typed into a page shows in them. Sealing refuses and removes a
|
|
1038
|
+
binary file that holds one of the values above as UTF-8 or UTF-16.
|
|
919
1039
|
- When a run's process is killed before it seals the run, `testkit stop`,
|
|
920
1040
|
`testkit cleanup`, or the next run seals it from another process. That
|
|
921
1041
|
process does not have the values the run's environment resolved or the
|
|
@@ -923,6 +1043,15 @@ Some secrets stay visible:
|
|
|
923
1043
|
values under the words above. The bundle's `archive.redaction.status` is then
|
|
924
1044
|
`source-targets-unavailable` instead of `applied`.
|
|
925
1045
|
|
|
1046
|
+
Sealing also takes out local paths. In the run record, in every text file of
|
|
1047
|
+
the bundle, and in every text entry of a compressed file, a path under the
|
|
1048
|
+
product directory becomes relative to it, the product directory itself becomes
|
|
1049
|
+
`.`, and a path under your home directory starts with `~`. This covers engine
|
|
1050
|
+
reports, failure stacks, logs, and Playwright traces, and happens after
|
|
1051
|
+
redaction. Binary files and binary entries keep their paths. When sealing
|
|
1052
|
+
changes a file, the file's size and SHA-256 in `run.json` describe the sealed
|
|
1053
|
+
file.
|
|
1054
|
+
|
|
926
1055
|
Use the artifact archive config to opt into pruning or atomic publication to
|
|
927
1056
|
an external directory. Local retention is indefinite by default:
|
|
928
1057
|
|
|
@@ -964,6 +1093,73 @@ Programmatic consumers can import `listRunBundles`, `loadRunBundleArtifact`,
|
|
|
964
1093
|
`loadRunBundleManifest`, and `verifyRunBundle` from
|
|
965
1094
|
`@gfdelarue/testkit/workspace`.
|
|
966
1095
|
|
|
1096
|
+
### Read run evidence without the CLI
|
|
1097
|
+
|
|
1098
|
+
Run evidence is plain files, so you can investigate a run with `ls`, `grep`,
|
|
1099
|
+
`jq`, and `sha256sum` when `testkit` can't start, for example because the
|
|
1100
|
+
config no longer loads. A command that fails to load the config stops before it
|
|
1101
|
+
writes to `.testkit/results`, and `testkit artifacts` doesn't need the config.
|
|
1102
|
+
|
|
1103
|
+
| Path under `.testkit/results/` | What it holds |
|
|
1104
|
+
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
1105
|
+
| `runs/<run-id>/run.json` | A sealed run's record: `run.status`, `run.interruptReason`, and each file's `status`, `failureDetails`, and `artifacts` |
|
|
1106
|
+
| `runs/<run-id>/manifest.json` | Every other file in the bundle, with its `sha256` and `bytes` |
|
|
1107
|
+
| `runs/<run-id>/provenance.json` | The run's CI, source, runtime, and redaction facts |
|
|
1108
|
+
| `runs/<run-id>/files/` | The run's `logs/`, `artifacts/`, and `setup/` files. Each `artifacts[].path` in `run.json` points here, relative to the product directory |
|
|
1109
|
+
| `latest.json` | The record of the last run that Testkit finished writing up, with its publication and retention results. A run killed after it was sealed can be in `runs/` but not here, so `runs/<run-id>/run.json` is the authority |
|
|
1110
|
+
| `live.json` | The record of a run that isn't sealed, with `run.status` `running` |
|
|
1111
|
+
| `logs/`, `artifacts/`, `setup/` | The files that unsealed run has written so far |
|
|
1112
|
+
|
|
1113
|
+
Anything else under `.testkit/` can change between releases, including
|
|
1114
|
+
`objects/`, which stores the bytes behind each bundle's files.
|
|
1115
|
+
|
|
1116
|
+
A run is sealed once `runs/<run-id>/` exists. Testkit writes the bundle under a
|
|
1117
|
+
hidden name in `runs/`, writes `manifest.json` last, and then renames the
|
|
1118
|
+
directory, so `runs/<run-id>/` is never partly written. Ignore names in `runs/`
|
|
1119
|
+
that start with `.`. A copy of a bundle, such as a CI upload, is whole when
|
|
1120
|
+
every file that its `manifest.json` lists has the listed `sha256`. A sealed
|
|
1121
|
+
`run.json` has `archive.status` `complete` and `run.status` `passed`, `failed`,
|
|
1122
|
+
or `interrupted`.
|
|
1123
|
+
|
|
1124
|
+
`live.json` means a run is in progress, or its process died before Testkit
|
|
1125
|
+
finished with it. If `runs/<run-id>/` exists for the `run.id` in `live.json`,
|
|
1126
|
+
the run was sealed and the bundle is its record: `live.json` and the active
|
|
1127
|
+
directories are leftovers, which the next `testkit run`, `cleanup`, or `destroy`
|
|
1128
|
+
removes. Testkit replaces `live.json` whole on each update, so it always parses.
|
|
1129
|
+
Its `generatedAt` is the time of the last update, and each file's `status` is
|
|
1130
|
+
what it was then, so a file that was still running shows `not_run`. The files
|
|
1131
|
+
in `logs/`, `artifacts/`, and `setup/` aren't redacted until the run is sealed.
|
|
1132
|
+
|
|
1133
|
+
How an unfinished run gets sealed:
|
|
1134
|
+
|
|
1135
|
+
- When Ctrl-C, `SIGTERM`, or `testkit stop` interrupts the run, or its process
|
|
1136
|
+
exits early, Testkit seals it straight away. `run.status` is `interrupted`,
|
|
1137
|
+
or `failed` when nothing asked the run to stop, and `run.interruptReason`
|
|
1138
|
+
says why.
|
|
1139
|
+
- When the process is killed with `SIGKILL`, by the out-of-memory killer, or
|
|
1140
|
+
with its host, the run stays unsealed until the next `testkit run`,
|
|
1141
|
+
`cleanup`, or `destroy` seals it, before doing anything else, with
|
|
1142
|
+
`run.status` `interrupted` and `run.interruptReason`
|
|
1143
|
+
`stale run process not active`. `testkit stop` seals it too, with its
|
|
1144
|
+
`--reason` or `stop requested`. The sealed `run.finishedAt` is when the
|
|
1145
|
+
command sealed it, not when the process died.
|
|
1146
|
+
|
|
1147
|
+
Testkit keeps what reached the disk. It can't keep output that a process hadn't
|
|
1148
|
+
written yet, anything written after the disk filled up or became read-only, or
|
|
1149
|
+
work that a killed process would have done on its way out. A log can end
|
|
1150
|
+
mid-line.
|
|
1151
|
+
|
|
1152
|
+
For example, to see what failed in the run that `latest.json` names and check
|
|
1153
|
+
its files:
|
|
1154
|
+
|
|
1155
|
+
```sh
|
|
1156
|
+
cd .testkit/results
|
|
1157
|
+
run=runs/$(jq -r .run.id latest.json)
|
|
1158
|
+
jq '.run.status, .run.interruptReason' "$run/run.json"
|
|
1159
|
+
jq '.groups[].suites[].files[] | select(.status != "passed") | {path, status, failureDetails, artifacts: [.artifacts[].path]}' "$run/run.json"
|
|
1160
|
+
jq -r '.files[] | "\(.sha256) \(.path)"' "$run/manifest.json" | (cd "$run" && sha256sum --check --quiet)
|
|
1161
|
+
```
|
|
1162
|
+
|
|
967
1163
|
## Authoring
|
|
968
1164
|
|
|
969
1165
|
### Named endpoints and databases
|
|
@@ -1016,14 +1212,14 @@ something its environment prepared. Today that is an HTTP capture, which a
|
|
|
1016
1212
|
The handle is named after the capture's `id`:
|
|
1017
1213
|
|
|
1018
1214
|
```js
|
|
1019
|
-
// tests/webhook.
|
|
1215
|
+
// tests/webhook.test.setup.mjs
|
|
1020
1216
|
export async function setup({ sidecars }) {
|
|
1021
1217
|
await sidecars.httpCapture({ id: "webhook", path: "/webhook" });
|
|
1022
1218
|
}
|
|
1023
1219
|
```
|
|
1024
1220
|
|
|
1025
1221
|
```ts
|
|
1026
|
-
// tests/webhook.
|
|
1222
|
+
// tests/webhook.test.ts
|
|
1027
1223
|
import { handle } from "@gfdelarue/testkit/environment";
|
|
1028
1224
|
|
|
1029
1225
|
const webhook = handle("webhook");
|
|
@@ -1045,6 +1241,14 @@ run's results, artifacts, and events, and the capture refuses any other
|
|
|
1045
1241
|
operation. When Testkit releases the file's environment, the capture closes and
|
|
1046
1242
|
the handle expires. Calling a handle never starts a sidecar or a service.
|
|
1047
1243
|
|
|
1244
|
+
The capture's `url` holds a second unguessable segment, which Testkit also
|
|
1245
|
+
keeps out of the run's records. The capture records a request only when its
|
|
1246
|
+
path starts with that segment, and answers any other path with 404. A captured
|
|
1247
|
+
request's `path` leaves the segment out: a request to `url` records `/webhook`,
|
|
1248
|
+
and one to `url` + `/42` records `/webhook/42`. The capture answers 403 to a
|
|
1249
|
+
request whose `Host` header isn't `127.0.0.1:<port>`, which keeps a web page
|
|
1250
|
+
from reaching it through DNS rebinding.
|
|
1251
|
+
|
|
1048
1252
|
- A name with no handle throws `EnvironmentValueMissingError` with `kind`
|
|
1049
1253
|
`"handle"`, and the test ends as an infrastructure failure, as for a missing
|
|
1050
1254
|
endpoint.
|
|
@@ -1069,7 +1273,8 @@ test, in any order.
|
|
|
1069
1273
|
so read them and leave them unchanged.
|
|
1070
1274
|
- When the test ends, leave its rows in place. Testkit resets the databases it
|
|
1071
1275
|
provisions. By default, every run starts from a fresh clone of the template
|
|
1072
|
-
database. See
|
|
1276
|
+
database. See `database.reset` in the `database` section of
|
|
1277
|
+
[`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
|
|
1073
1278
|
- Clean up only what Testkit doesn't own, such as an account in a third-party
|
|
1074
1279
|
sandbox. Register the cleanup as soon as the test creates the account. Give
|
|
1075
1280
|
the account a unique name, so that a leftover from a cleanup that never ran
|
|
@@ -1077,19 +1282,20 @@ test, in any order.
|
|
|
1077
1282
|
|
|
1078
1283
|
```ts
|
|
1079
1284
|
import { randomUUID } from "node:crypto";
|
|
1080
|
-
import {
|
|
1081
|
-
import {
|
|
1082
|
-
import { expect, onTestFinished } from "vitest";
|
|
1285
|
+
import { endpoint } from "@gfdelarue/testkit/environment";
|
|
1286
|
+
import { expect, onTestFinished, test } from "vitest";
|
|
1083
1287
|
|
|
1084
1288
|
import { sandbox } from "../payments-sandbox.js";
|
|
1085
1289
|
|
|
1086
|
-
|
|
1087
|
-
|
|
1088
|
-
contractScenario("a refund reaches the sandbox customer", async ({ http }) => {
|
|
1290
|
+
test("a refund reaches the sandbox customer", async () => {
|
|
1089
1291
|
const customer = await sandbox.createCustomer({ email: `refund-${randomUUID()}@example.test` });
|
|
1090
1292
|
onTestFinished(() => sandbox.deleteCustomer(customer.id));
|
|
1091
1293
|
|
|
1092
|
-
const response = await
|
|
1294
|
+
const response = await fetch(new URL("/api/refunds", endpoint("api")), {
|
|
1295
|
+
method: "POST",
|
|
1296
|
+
headers: { "content-type": "application/json" },
|
|
1297
|
+
body: JSON.stringify({ customerId: customer.id, amountCents: 500 }),
|
|
1298
|
+
});
|
|
1093
1299
|
expect(response.status).toBe(201);
|
|
1094
1300
|
});
|
|
1095
1301
|
```
|
|
@@ -1125,18 +1331,34 @@ Files that name the same lock never run at the same time within one run.
|
|
|
1125
1331
|
Testkit doesn't share locks between runs, such as two CI jobs. Two tests that
|
|
1126
1332
|
need a lock because they share rows need their own rows instead.
|
|
1127
1333
|
|
|
1334
|
+
A plain Vitest file names its locks in its group's `requirements`:
|
|
1335
|
+
|
|
1128
1336
|
```ts
|
|
1129
|
-
|
|
1337
|
+
groups: {
|
|
1338
|
+
payments: {
|
|
1339
|
+
roots: ["src/payments"],
|
|
1340
|
+
uses: ["api"],
|
|
1341
|
+
requirements: {
|
|
1342
|
+
files: [{ path: "src/payments/webhook.test.ts", locks: ["payments-sandbox-webhook"] }],
|
|
1343
|
+
},
|
|
1344
|
+
},
|
|
1345
|
+
},
|
|
1346
|
+
```
|
|
1347
|
+
|
|
1348
|
+
A browser file can name them in `defineFile`:
|
|
1349
|
+
|
|
1350
|
+
```ts
|
|
1351
|
+
export const testkit = defineFile({ engine: "playwright", locks: ["payments-sandbox-webhook"] });
|
|
1130
1352
|
```
|
|
1131
1353
|
|
|
1132
1354
|
### Node setup beside a test
|
|
1133
1355
|
|
|
1134
|
-
A test such as `tests/webhook.
|
|
1135
|
-
`tests/webhook.
|
|
1136
|
-
|
|
1356
|
+
A test such as `tests/webhook.test.ts` can have a sibling
|
|
1357
|
+
`tests/webhook.test.setup.mjs`. Testkit runs its named `setup` export, or its
|
|
1358
|
+
default export, before starting the test engine:
|
|
1137
1359
|
|
|
1138
1360
|
```js
|
|
1139
|
-
// tests/webhook.
|
|
1361
|
+
// tests/webhook.test.setup.mjs
|
|
1140
1362
|
export async function setup({ sidecars }) {
|
|
1141
1363
|
const capture = await sidecars.httpCapture({ path: "/webhook" });
|
|
1142
1364
|
return {
|
|
@@ -1148,13 +1370,15 @@ export async function setup({ sidecars }) {
|
|
|
1148
1370
|
Setup is part of preparing the file's environment. Testkit runs it once, after
|
|
1149
1371
|
it acquires the services the file's group uses and before the test engine
|
|
1150
1372
|
starts. Testkit never batches a Playwright file that has setup with other
|
|
1151
|
-
Playwright files
|
|
1373
|
+
Playwright files, and a plain Vitest file that has setup runs in its own Vitest
|
|
1374
|
+
invocation. Files without setup still batch.
|
|
1152
1375
|
|
|
1153
1376
|
Setup runs with its own JavaScript state. It can use Node APIs, prepare
|
|
1154
1377
|
external resources, and start HTTP capture listeners. Tests
|
|
1155
|
-
reach a capture by its `id` through [`handle()`](#handles). A capture's
|
|
1156
|
-
`requestsUrl` and `resetUrl` hold
|
|
1157
|
-
of the run's records
|
|
1378
|
+
reach a capture by its `id` through [`handle()`](#handles). A capture's `url`,
|
|
1379
|
+
`requestsUrl` and `resetUrl` hold unguessable segments that Testkit keeps out
|
|
1380
|
+
of the run's records. Its capture `path` may not start with `/__testkit__/` or
|
|
1381
|
+
contain `.` or `..` segments.
|
|
1158
1382
|
The context contains `deadlineMs`, `signal`, `env`, `leaseDir`, `productDir`,
|
|
1159
1383
|
`serviceName`, `testFile`, and `sidecars`. `serviceName` holds the name of the
|
|
1160
1384
|
test's group. Setup has the same permissions as the Testkit process. It is not
|
|
@@ -1165,7 +1389,9 @@ setup modules. Its tests receive the environment file's `setupData` instead.
|
|
|
1165
1389
|
|
|
1166
1390
|
Return plain JSON, or `{ data, cleanup }` with an optional cleanup function.
|
|
1167
1391
|
Testkit passes the data to the engine as `setupData` and records it with the
|
|
1168
|
-
setup module path and sidecar handles.
|
|
1392
|
+
setup module path and sidecar handles. Tests read it with `setupData()` from
|
|
1393
|
+
`@gfdelarue/testkit/environment`, which returns `{}` when the file has no setup
|
|
1394
|
+
module and throws outside a Testkit run. Bare `null` or `undefined`, and a
|
|
1169
1395
|
cleanup-only result, give empty data. `{ data: null }` preserves `null`.
|
|
1170
1396
|
Data must encode to at most 1 MiB of JSON. Invalid data reports its JSON path,
|
|
1171
1397
|
such as `$.items[1]`.
|
|
@@ -1189,15 +1415,226 @@ most 256 connections.
|
|
|
1189
1415
|
|
|
1190
1416
|
Discovery tries sibling `.setup.mjs`, sibling `.setup.js`, then the matching
|
|
1191
1417
|
`.mjs` and `.js` files in `__setup__`, in that order. For the example above,
|
|
1192
|
-
the fallback is `tests/__setup__/webhook.
|
|
1418
|
+
the fallback is `tests/__setup__/webhook.test.mjs`.
|
|
1193
1419
|
|
|
1194
1420
|
### HTTP and persistence checks
|
|
1195
1421
|
|
|
1196
1422
|
Use plain Vitest files for ordinary HTTP, service, and database checks. Read
|
|
1197
1423
|
configured URLs with `endpoint(name)` and `database(name)` from
|
|
1198
1424
|
`@gfdelarue/testkit/environment`, then use `fetch` or your application's
|
|
1199
|
-
database client.
|
|
1200
|
-
|
|
1425
|
+
database client.
|
|
1426
|
+
|
|
1427
|
+
### Record data-flow evidence
|
|
1428
|
+
|
|
1429
|
+
`dataFlow(context)` from `@gfdelarue/testkit/node-vitest/data` records what a
|
|
1430
|
+
plain Vitest test sees data do: metrics, counts, bytes, durations, money,
|
|
1431
|
+
ratios, sequences, records, actors, systems, transitions, events, findings,
|
|
1432
|
+
external calls, and cache, LLM, and RPC usage. Budgets check the recorded
|
|
1433
|
+
values, for example that a metric stays under a limit or that a sequence was
|
|
1434
|
+
recorded.
|
|
1435
|
+
|
|
1436
|
+
```ts
|
|
1437
|
+
import { endpoint } from "@gfdelarue/testkit/environment";
|
|
1438
|
+
import { dataFlow } from "@gfdelarue/testkit/node-vitest/data";
|
|
1439
|
+
import { test } from "vitest";
|
|
1440
|
+
|
|
1441
|
+
test("lists one page of orders", async (context) => {
|
|
1442
|
+
const flow = dataFlow(context);
|
|
1443
|
+
const response = await fetch(`${endpoint("api")}/orders?page=1`);
|
|
1444
|
+
const body = await response.text();
|
|
1445
|
+
flow.recordHttpRequest({
|
|
1446
|
+
method: "GET",
|
|
1447
|
+
path: "/orders",
|
|
1448
|
+
status: response.status,
|
|
1449
|
+
responseBytes: Buffer.byteLength(body),
|
|
1450
|
+
});
|
|
1451
|
+
flow.expectMetric("http.response.bytes").atMost(64_000);
|
|
1452
|
+
});
|
|
1453
|
+
```
|
|
1454
|
+
|
|
1455
|
+
- `dataFlow` needs the context of a running test. Every call in one attempt
|
|
1456
|
+
returns the same recorder, and a call without that context throws.
|
|
1457
|
+
- The library reads no URL, environment value, fixture file, or Testkit config.
|
|
1458
|
+
Read named URLs from `@gfdelarue/testkit/environment`, log in from test code,
|
|
1459
|
+
and read files with `fs`.
|
|
1460
|
+
- A failed budget fails its test when the test finishes. The failure detail has
|
|
1461
|
+
the kind `data-flow-budget` and a key made of the test's full title and the
|
|
1462
|
+
budget.
|
|
1463
|
+
- Testkit writes `data-flow.json` and `data-flow-report.md` for a file whose tests
|
|
1464
|
+
recorded something. Each such test is one scenario, from its final attempt. A
|
|
1465
|
+
test that records nothing adds nothing, and a file whose tests record
|
|
1466
|
+
nothing gets no data-flow artifacts.
|
|
1467
|
+
|
|
1468
|
+
### Slow the browser's network and CPU
|
|
1469
|
+
|
|
1470
|
+
`emulateNetwork(page, conditions)` and `emulateCpu(page, slowdown)` from
|
|
1471
|
+
`@gfdelarue/testkit/playwright` slow a page the way Chrome DevTools'
|
|
1472
|
+
throttling does. A `ui` test can use them to fail when a change makes a page
|
|
1473
|
+
unusable on a slow connection or a slow device:
|
|
1474
|
+
|
|
1475
|
+
```ts
|
|
1476
|
+
import { defineFile } from "@gfdelarue/testkit";
|
|
1477
|
+
import { emulateCpu, emulateNetwork, expect, test } from "@gfdelarue/testkit/playwright";
|
|
1478
|
+
|
|
1479
|
+
export const testkit = defineFile({ engine: "playwright" });
|
|
1480
|
+
|
|
1481
|
+
test("the dashboard is usable on a slow connection", async ({ page }) => {
|
|
1482
|
+
await emulateNetwork(page, "slow-4g");
|
|
1483
|
+
await page.goto("/dashboard");
|
|
1484
|
+
await expect(page.getByRole("heading", { name: "Orders" })).toBeVisible({ timeout: 10_000 });
|
|
1485
|
+
});
|
|
1486
|
+
|
|
1487
|
+
test("the dashboard is usable on a slow phone", async ({ page }) => {
|
|
1488
|
+
await emulateCpu(page, 4);
|
|
1489
|
+
await page.goto("/dashboard");
|
|
1490
|
+
await expect(page.getByRole("heading", { name: "Orders" })).toBeVisible({ timeout: 10_000 });
|
|
1491
|
+
});
|
|
1492
|
+
```
|
|
1493
|
+
|
|
1494
|
+
The assertion's timeout is the time limit. The helpers add no timing or
|
|
1495
|
+
metrics API.
|
|
1496
|
+
|
|
1497
|
+
`emulateNetwork` takes a profile name or custom values. Each profile has the
|
|
1498
|
+
values of the DevTools preset of the same name in the Chromium that Testkit
|
|
1499
|
+
runs:
|
|
1500
|
+
|
|
1501
|
+
| Profile | DevTools preset | Download | Upload | Latency |
|
|
1502
|
+
| ----------- | --------------- | ---------- | ---------- | -------- |
|
|
1503
|
+
| `"fast-4g"` | Fast 4G | 8,100 kbps | 1,350 kbps | 165 ms |
|
|
1504
|
+
| `"slow-4g"` | Slow 4G | 1,440 kbps | 675 kbps | 562.5 ms |
|
|
1505
|
+
| `"3g"` | 3G | 400 kbps | 400 kbps | 2,000 ms |
|
|
1506
|
+
|
|
1507
|
+
DevTools renamed two presets in 2024: Slow 4G was Fast 3G, and 3G was Slow 3G.
|
|
1508
|
+
Custom values give speeds in kilobits per second and latency in milliseconds:
|
|
1509
|
+
|
|
1510
|
+
```ts
|
|
1511
|
+
await emulateNetwork(page, { latencyMs: 300, downloadKbps: 400, uploadKbps: 200 });
|
|
1512
|
+
```
|
|
1513
|
+
|
|
1514
|
+
`emulateCpu` makes the page's main thread, which runs its JavaScript and
|
|
1515
|
+
layout, `slowdown` times slower. DevTools' fixed CPU options are 4, 6, and 20.
|
|
1516
|
+
Testkit doesn't calibrate the slowdown to the machine, as DevTools' "Mid-tier
|
|
1517
|
+
mobile" and "Low-tier mobile" options do. `emulateCpu(page, 1)` restores full
|
|
1518
|
+
speed.
|
|
1519
|
+
|
|
1520
|
+
- Call the helpers before `page.goto()`. Both settings last across navigations
|
|
1521
|
+
until the page closes.
|
|
1522
|
+
- Neither setting reaches a cross-site iframe. Chromium runs one in a process
|
|
1523
|
+
of its own, at full speed.
|
|
1524
|
+
- The network conditions apply to that page only. A popup it opens, or another
|
|
1525
|
+
page, loads at full speed. Playwright's `browserContext.setOffline()` clears
|
|
1526
|
+
them.
|
|
1527
|
+
- The CPU slowdown applies to the browser process that runs the page. A
|
|
1528
|
+
same-site popup that keeps its `window.opener` shares that process and slows
|
|
1529
|
+
down too, and closing it ends the slowdown. Call `emulateCpu` again after
|
|
1530
|
+
closing one.
|
|
1531
|
+
- Like DevTools, `emulateNetwork` also sets `navigator.connection` until the
|
|
1532
|
+
page navigates to another site.
|
|
1533
|
+
- An unknown profile name or field, a negative or non-finite value, a speed of
|
|
1534
|
+
0, or a CPU slowdown outside 1 to 100 throws an error that names the value.
|
|
1535
|
+
|
|
1536
|
+
Three things to keep in mind:
|
|
1537
|
+
|
|
1538
|
+
- Set generous time limits. A CI runner has less CPU than a laptop, and a
|
|
1539
|
+
[campaign](#run-a-campaign) files a failing test as a bug, so a tight limit
|
|
1540
|
+
produces false bug reports.
|
|
1541
|
+
- A CPU slowdown multiplies the speed of the machine that runs the test, so 4
|
|
1542
|
+
on a slow CI runner is much slower than 4 on a laptop. A time limit under a
|
|
1543
|
+
CPU slowdown depends on the runner far more than one under network
|
|
1544
|
+
conditions, which are the same on every machine.
|
|
1545
|
+
- Chromium holds each response for at least the latency and shares the
|
|
1546
|
+
bandwidth among the page's requests, but it doesn't model TCP. The result
|
|
1547
|
+
approximates a slow connection rather than reproducing one.
|
|
1548
|
+
|
|
1549
|
+
### Check a ui test's browser traffic
|
|
1550
|
+
|
|
1551
|
+
Testkit records the requests every `ui` test's pages send, with no change to the
|
|
1552
|
+
test. The record catches bugs that live in how a frontend uses its API, such as
|
|
1553
|
+
a checkout that sends its payment twice or a list page that fetches each row
|
|
1554
|
+
separately, even when no assertion checks for them.
|
|
1555
|
+
|
|
1556
|
+
For each `ui` file whose tests sent requests, Testkit writes
|
|
1557
|
+
`browser-network.json` and `browser-network-report.md` from each test's final
|
|
1558
|
+
attempt. A test's section of the report looks like this:
|
|
1559
|
+
|
|
1560
|
+
```md
|
|
1561
|
+
## checkout › pays for the cart (passed)
|
|
1562
|
+
|
|
1563
|
+
21 requests, 452 KB received: script 312 KB, font 48 KB, document 41 KB, fetch 14 KB
|
|
1564
|
+
|
|
1565
|
+
| Service | Request | Count | Sent | Received | Status |
|
|
1566
|
+
| ------- | ----------------- | ----- | ------ | -------- | ------ |
|
|
1567
|
+
| web | GET /cart | 1 | 0 B | 41 KB | 200 |
|
|
1568
|
+
| api | GET /products/:id | 8 | 0 B | 11 KB | 200 |
|
|
1569
|
+
| api | POST /payments | 2 | 1.2 KB | 400 B | 201 |
|
|
1570
|
+
|
|
1571
|
+
Other origins: fonts.gstatic.com, 2 requests, 48 KB
|
|
1572
|
+
|
|
1573
|
+
Leads:
|
|
1574
|
+
|
|
1575
|
+
- `POST /payments` to api went out 2 times with the same body.
|
|
1576
|
+
```
|
|
1577
|
+
|
|
1578
|
+
- Rows cover page loads and API calls, which are Playwright's `document`,
|
|
1579
|
+
`fetch`, and `xhr` requests, to the services the test's group uses. A row
|
|
1580
|
+
names the service rather than its URL, because each run gives services
|
|
1581
|
+
different ports. Scripts, styles, images, and fonts count only toward the
|
|
1582
|
+
totals by type. Requests to other origins appear by host name only.
|
|
1583
|
+
- A path's numbers, UUIDs, and hex strings of 16 or more characters become
|
|
1584
|
+
`:id`, so eight product fetches make one row with a count of 8.
|
|
1585
|
+
- The record never holds query strings, headers, or bodies. Paths pass through
|
|
1586
|
+
Testkit's redaction, and sealing redacts them again with the run's secrets.
|
|
1587
|
+
To spot a repeated write, Testkit hashes each body in memory and keeps only
|
|
1588
|
+
the count.
|
|
1589
|
+
- Sizes are body bytes as they crossed the network, so a compressed response
|
|
1590
|
+
counts its compressed size and one that a `page.route()` handler fulfills
|
|
1591
|
+
counts 0. Headers don't count.
|
|
1592
|
+
- Each test keeps 200 requests in detail, 100 rows, and 20 other origins. The
|
|
1593
|
+
record counts the rest.
|
|
1594
|
+
- The leads point at two patterns: a POST, PUT, PATCH, or DELETE sent more than
|
|
1595
|
+
once to the same URL with the same body, and a page load or API call that
|
|
1596
|
+
failed or got a 4xx or 5xx status in a test that passed. A lead never fails a
|
|
1597
|
+
test.
|
|
1598
|
+
- The record covers every context a test opens, including one from
|
|
1599
|
+
`browser.newContext()`. It doesn't cover a context that a `beforeAll` hook
|
|
1600
|
+
opens.
|
|
1601
|
+
|
|
1602
|
+
To assert on the traffic, call `browserRequests()` from
|
|
1603
|
+
`@gfdelarue/testkit/playwright`. It returns the requests the test's pages have
|
|
1604
|
+
sent so far, oldest first:
|
|
1605
|
+
|
|
1606
|
+
```ts
|
|
1607
|
+
import { browserRequests, expect, test } from "@gfdelarue/testkit/playwright";
|
|
1608
|
+
|
|
1609
|
+
test("checkout charges the card once", async ({ page }) => {
|
|
1610
|
+
await page.goto("/cart");
|
|
1611
|
+
await page.getByRole("button", { name: "Pay" }).click();
|
|
1612
|
+
await expect(page.getByText("Payment received")).toBeVisible();
|
|
1613
|
+
|
|
1614
|
+
const payments = (await browserRequests()).filter(
|
|
1615
|
+
(request) => request.service === "api" && request.method === "POST" && request.path === "/payments",
|
|
1616
|
+
);
|
|
1617
|
+
expect(payments).toHaveLength(1);
|
|
1618
|
+
});
|
|
1619
|
+
```
|
|
1620
|
+
|
|
1621
|
+
Each entry has `service`, the name of the service it went to or `null`,
|
|
1622
|
+
`method`, `url`, `path`, `resourceType`, `status`, `failure`, `requestBytes`,
|
|
1623
|
+
`responseBytes`, and `durationMs`. `path` is the URL's path as sent, without
|
|
1624
|
+
`:id`. Add up entries with plain array code, for example to check a page's
|
|
1625
|
+
weight:
|
|
1626
|
+
|
|
1627
|
+
```ts
|
|
1628
|
+
const received = (await browserRequests()).reduce((total, request) => total + request.responseBytes, 0);
|
|
1629
|
+
expect(received).toBeLessThan(500_000);
|
|
1630
|
+
```
|
|
1631
|
+
|
|
1632
|
+
`browserRequests()` waits up to 5 seconds for requests already in flight, so
|
|
1633
|
+
their entries have a status and sizes. It doesn't wait for EventSource streams.
|
|
1634
|
+
A request still unfinished after that has its status if the response's headers
|
|
1635
|
+
arrived, 0 response bytes, and a `durationMs` of `null`. A WebSocket connection
|
|
1636
|
+
appears with the `resourceType` `websocket` and no status, sizes, or duration.
|
|
1637
|
+
The record counts connections, not their messages.
|
|
1201
1638
|
|
|
1202
1639
|
## Discovery
|
|
1203
1640
|
|
|
@@ -1228,24 +1665,26 @@ Testkit supports Vitest 4.x. Another major, a broken installation, a root
|
|
|
1228
1665
|
Vitest config that fails to load, or a Vitest that `package.json` declares but
|
|
1229
1666
|
that is not installed fails before any test runs, naming the version and path.
|
|
1230
1667
|
`--vitest-runner managed` avoids the repository's Vitest. A run that selects
|
|
1231
|
-
only typed files, with `--type` or with typed `--file` paths, `bug record` or
|
|
1668
|
+
only typed files, with `--type ui` or with typed `--file` paths, `bug record` or
|
|
1232
1669
|
`bug file` for a typed test, and the commands that select no tests (`status`,
|
|
1233
1670
|
`run status`, `doctor`, `destroy`, `cleanup`, and `db schema`) don't load the
|
|
1234
1671
|
repository's Vitest, so they still work. A product that neither installs nor
|
|
1235
1672
|
declares Vitest has no plain test files.
|
|
1236
1673
|
|
|
1237
|
-
|
|
1238
|
-
|
|
1239
|
-
|
|
1240
|
-
|
|
1241
|
-
|
|
1242
|
-
|
|
1243
|
-
|
|
1244
|
-
|
|
1245
|
-
|
|
1246
|
-
|
|
1247
|
-
|
|
1248
|
-
|
|
1674
|
+
Plain files run in one Vitest invocation per environment their groups give them.
|
|
1675
|
+
Groups that give their files the same values share one, such as `root` and the
|
|
1676
|
+
other groups that use no services. A group that uses a service Testkit starts
|
|
1677
|
+
gets an invocation of its own.
|
|
1678
|
+
Inside each invocation, root global setup, teardown, and the repository's
|
|
1679
|
+
reporters run as they do under `vitest run`, and Testkit's reporter runs beside
|
|
1680
|
+
them, so a run calls them once for each invocation. An invocation's deadline is
|
|
1681
|
+
the sum of its files' timeouts; Vitest's own test and hook timeouts apply inside
|
|
1682
|
+
it. A file that two projects select runs in both, its result combines both
|
|
1683
|
+
runs, and its evidence names each run's project. A file with a setup module,
|
|
1684
|
+
locks, or isolation other than `shared` runs in its own Vitest invocation with
|
|
1685
|
+
its own environment. Invocations run one after another unless
|
|
1686
|
+
`--task-concurrency` or `execution.taskConcurrency` is above 1. To run one group's
|
|
1687
|
+
files alone, select it with `--group`.
|
|
1249
1688
|
|
|
1250
1689
|
Plain files see the host environment Testkit started with: `HOME`, `PATH`,
|
|
1251
1690
|
temporary and XDG directories, locale, `CI`, `FORCE_COLOR`, `NO_COLOR`, and any
|
|
@@ -1259,14 +1698,21 @@ shell's environment and the group's values; as under `vitest run`, Vitest sets
|
|
|
1259
1698
|
`TEST`, `VITEST`, and `NODE_ENV` after them. `--vitest-runner managed` leaves the
|
|
1260
1699
|
shell's `NODE_OPTIONS` out.
|
|
1261
1700
|
|
|
1701
|
+
When Vitest detects a coding agent from the environment, such as `CLAUDECODE`,
|
|
1702
|
+
`AI_AGENT`, or `CURSOR_AGENT`, and the repository's Vitest config names no
|
|
1703
|
+
reporter, Vitest uses its agent reporter. That reporter prints no console output
|
|
1704
|
+
from a passing test, so the output is missing from the file's
|
|
1705
|
+
`node-vitest-stdout` artifact. A failing test's output still appears. Name a
|
|
1706
|
+
reporter in the Vitest config, such as `reporters: ["default"]`, to keep it.
|
|
1707
|
+
|
|
1262
1708
|
Two settings narrow the files Vitest selects: `discovery.exclude` in `testkit.config.ts`, and
|
|
1263
1709
|
a declared group's `exclude`. Testkit's built-in exclusions for typed files,
|
|
1264
|
-
such as `dist`, `build`, and `coverage`, `discovery.roots
|
|
1265
|
-
|
|
1710
|
+
such as `dist`, `build`, and `coverage`, and `discovery.roots` do not apply to
|
|
1711
|
+
plain files.
|
|
1266
1712
|
|
|
1267
1713
|
A plain file belongs to the group whose roots contain it, or to the implicit
|
|
1268
|
-
`root` group, which uses no service.
|
|
1269
|
-
|
|
1714
|
+
`root` group, which uses no service. Testkit starts no service for a plain
|
|
1715
|
+
file in `root`.
|
|
1270
1716
|
|
|
1271
1717
|
The plain files in one directory form a suite named after that directory,
|
|
1272
1718
|
relative to the group's root. In `root`, `--suite src` selects
|
|
@@ -1275,7 +1721,10 @@ its own suite, so `sum.test.ts` in a group rooted at `src` is in suite `sum`.
|
|
|
1275
1721
|
When two suites in a group would share a name, Testkit lengthens the later
|
|
1276
1722
|
ones with parent directories, such as `api-src`, and numbers them if that is
|
|
1277
1723
|
not enough. `testkit typecheck` checks plain TypeScript test files and skips
|
|
1278
|
-
plain JavaScript ones, as `tsc` does without `allowJs`.
|
|
1724
|
+
plain JavaScript ones, as `tsc` does without `allowJs`. It adds the TypeScript
|
|
1725
|
+
files in each Vitest project's `setupFiles` to the program, so a test can use
|
|
1726
|
+
the matchers a setup file declares, such as those from
|
|
1727
|
+
`@testing-library/jest-dom/vitest`.
|
|
1279
1728
|
|
|
1280
1729
|
### Typed suites
|
|
1281
1730
|
|
|
@@ -1308,16 +1757,9 @@ production owner when a narrower owner exists.
|
|
|
1308
1757
|
|
|
1309
1758
|
Concrete layouts:
|
|
1310
1759
|
|
|
1311
|
-
- `src/api/routes/__testkit__/auth-me.contract.testkit.ts`
|
|
1312
|
-
- `src/db/__testkit__/sessions-count-type.contract.testkit.ts`
|
|
1313
1760
|
- `frontend/src/components/layout/__testkit__/navigation.ui.testkit.ts`
|
|
1314
|
-
- `src/internal/handler/repos/__testkit__/crud.contract.testkit.ts`
|
|
1315
|
-
|
|
1316
|
-
`testkit` uses these suffixes automatically:
|
|
1317
1761
|
|
|
1318
|
-
|
|
1319
|
-
- `*.data.testkit.ts`
|
|
1320
|
-
- `*.ui.testkit.ts`
|
|
1762
|
+
`testkit` uses the `*.ui.testkit.ts` suffix automatically.
|
|
1321
1763
|
|
|
1322
1764
|
The suffix never chooses an engine. Each discovered typed file must also
|
|
1323
1765
|
contain one root-package `defineFile(...)` declaration with a supported literal
|
|
@@ -1325,30 +1767,7 @@ engine.
|
|
|
1325
1767
|
Unsupported pairs fail discovery; for example, a `*.ui.testkit.ts` file cannot
|
|
1326
1768
|
declare `engine: "node-vitest"`.
|
|
1327
1769
|
|
|
1328
|
-
See [docs/test-types.md](docs/test-types.md) for the canonical type model.
|
|
1329
|
-
`*.contract.testkit.ts` files are Node-backed service/runtime contract suites.
|
|
1330
|
-
They use `contractScenario(...)` from
|
|
1331
|
-
`@gfdelarue/testkit/node-vitest/contract`, declare `engine: "node-vitest"`, and run
|
|
1332
|
-
through the `node-vitest-contract` profile for checks that need Node APIs but
|
|
1333
|
-
are not data-flow evidence suites. They are native Vitest suites:
|
|
1334
|
-
`contractScenario(...)` is the managed-context convenience, while direct public
|
|
1335
|
-
Vitest runner APIs remain valid for behavior such as `test.skip(...)` and
|
|
1336
|
-
`test.todo(...)`.
|
|
1337
|
-
|
|
1338
|
-
`*.data.testkit.ts` files are Testkit data-flow/data-contract suites, not raw
|
|
1339
|
-
Vitest files with a Testkit suffix. They must use `dataScenario(...)` from
|
|
1340
|
-
`@gfdelarue/testkit/node-vitest/data`, declare `engine: "node-vitest"`, and record
|
|
1341
|
-
structured data-flow evidence through the data context. Supporting
|
|
1342
|
-
`count(...)`/`event(...)` calls are not enough by themselves; model actors,
|
|
1343
|
-
systems, records, transitions, external calls, payloads, budgets, cache/LLM/RPC
|
|
1344
|
-
usage, or reconciliation evidence. Data suites run through the
|
|
1345
|
-
`node-vitest-data` profile. Use a plain Vitest test file for plain Node
|
|
1346
|
-
assertions and HTTP or database integration, `contract` for typed service or
|
|
1347
|
-
protocol contracts, and `ui` when a browser is part of the assertion.
|
|
1348
|
-
Data suites use Vitest directly for assertions, hooks, and mocking.
|
|
1349
|
-
`dataScenario(...)` owns case registration: `testkit lint` reports native
|
|
1350
|
-
registration imports in data suites, and the data runner rejects any case that
|
|
1351
|
-
was not registered through the DSL.
|
|
1770
|
+
See [`docs/test-types.md`](https://app.unpkg.com/@gfdelarue/testkit/files/docs/test-types.md) for the canonical type model.
|
|
1352
1771
|
|
|
1353
1772
|
Test groups own test files by directory. Declare them under `groups` in
|
|
1354
1773
|
`testkit.config.ts`. Each group lists its repo-relative `roots` and the services
|
|
@@ -1378,18 +1797,19 @@ export default defineConfig({
|
|
|
1378
1797
|
are also available by name; see
|
|
1379
1798
|
[Named endpoints and databases](#named-endpoints-and-databases).
|
|
1380
1799
|
- `skip`, `requirements`, and `runtimeIsolation` are group fields. See the
|
|
1381
|
-
[config
|
|
1800
|
+
`groups` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
|
|
1382
1801
|
|
|
1383
|
-
A config with `services` and no `groups` key
|
|
1384
|
-
|
|
1385
|
-
|
|
1386
|
-
|
|
1387
|
-
|
|
1802
|
+
Services own no test files. A config with `services` and no `groups` key
|
|
1803
|
+
fails to load when a service's `local.cwd` or `discovery.roots` holds a typed
|
|
1804
|
+
test file, which that service owned before 0.6.0. The error lists the `groups`
|
|
1805
|
+
entries that keep those files with their services. `status`, `destroy`,
|
|
1806
|
+
`cleanup`, and `db schema` still run on it. Otherwise every file belongs to
|
|
1807
|
+
`root`.
|
|
1388
1808
|
|
|
1389
1809
|
Suite names are inferred from the colocated path:
|
|
1390
1810
|
|
|
1391
|
-
- `auth/__testkit__/*.
|
|
1392
|
-
- `routes/__testkit__/
|
|
1811
|
+
- `auth/__testkit__/*.ui.testkit.ts` => `auth`
|
|
1812
|
+
- `routes/__testkit__/checkout.ui.testkit.ts` => `routes`
|
|
1393
1813
|
|
|
1394
1814
|
Discovery is also a first-class CLI/API surface:
|
|
1395
1815
|
|
|
@@ -1446,11 +1866,16 @@ Testkit ignores an old `.testkit/history/` directory and leaves it in place.
|
|
|
1446
1866
|
`@gfdelarue/testkit` provisions Docker-managed local Postgres automatically for
|
|
1447
1867
|
services that define `database: database.postgres(...)`.
|
|
1448
1868
|
|
|
1449
|
-
Docker is also the required Postgres client runtime
|
|
1450
|
-
|
|
1451
|
-
|
|
1452
|
-
|
|
1453
|
-
|
|
1869
|
+
Docker is also the required Postgres client runtime. Testkit doesn't use
|
|
1870
|
+
host-installed `psql` or `pg_dump`, which keeps local runs consistent across
|
|
1871
|
+
Linux, macOS, and native Windows. Testkit's own operations on the database it
|
|
1872
|
+
owns, such as declaring roles and taking schema snapshots, run inside the
|
|
1873
|
+
database container. So the `image` that `database.postgres()` uses must include
|
|
1874
|
+
`psql` and `pg_dump`, with a `pg_dump` at least as new as its server. Your SQL,
|
|
1875
|
+
in `sql-file` steps and `database-steps` helpers, runs `psql` in a separate
|
|
1876
|
+
client container from `postgres:16`. A `pg_dump` of an external database runs
|
|
1877
|
+
in `postgres:<its major version>`. `TESTKIT_POSTGRES_CLIENT_IMAGE` overrides
|
|
1878
|
+
the image of those client containers.
|
|
1454
1879
|
|
|
1455
1880
|
Template steps, schema snapshots, and `database.steps` helpers work the same
|
|
1456
1881
|
under Docker Desktop, rootless Docker, and Docker-in-Docker. Clients for
|
|
@@ -1476,7 +1901,7 @@ one with `database(name)` from `@gfdelarue/testkit/environment`. Declared `env`
|
|
|
1476
1901
|
`{dbPort}`, `{dbName}`, `{dbUser}`, `{dbPassword}`, and `{dbSsl}` tokens, with
|
|
1477
1902
|
`:serviceName` to read the database of a service listed in `dependsOn`. Testkit
|
|
1478
1903
|
resolves them from the database provisioned for the current run. They are
|
|
1479
|
-
env-only; see
|
|
1904
|
+
env-only; see the `env` section of [`testkit-config-schema.md`](https://app.unpkg.com/@gfdelarue/testkit/files/testkit-config-schema.md).
|
|
1480
1905
|
|
|
1481
1906
|
`db schema refresh` forces a source database dump into the `.testkit` source
|
|
1482
1907
|
schema cache. `db schema verify` prepares local templates and verifies local
|