@gfdelarue/testkit 0.3.5 → 0.3.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -73
- package/dist/process/execa-bundle.json +1 -1
- package/docs/cloud-control.md +3 -0
- package/docs/testkit-cloud-boundary.md +12 -24
- package/package.json +1 -1
- package/testkit-config-schema.md +2 -0
package/README.md
CHANGED
|
@@ -7,7 +7,9 @@
|
|
|
7
7
|
`@gfdelarue/testkit` is the orchestrator and managed distribution for product test
|
|
8
8
|
suites. It discovers `*.testkit.ts` files, infers suite ownership from the
|
|
9
9
|
filesystem, starts local services, provisions Docker-managed local Postgres
|
|
10
|
-
databases, schedules suites, and collects results and artifacts.
|
|
10
|
+
databases, schedules suites, and collects results and artifacts. Coding agents
|
|
11
|
+
use it to write tests and hunt bugs, and CI uses it to run those tests. It needs
|
|
12
|
+
no account or server.
|
|
11
13
|
|
|
12
14
|
Testkit exists to replace false confidence with inspectable execution evidence.
|
|
13
15
|
It cannot prove that an entire application works. It can make narrower claims
|
|
@@ -47,33 +49,6 @@ Testkit writes the immutable, verifiable run bundle under
|
|
|
47
49
|
`.testkit/results/runs/<run-id>/`. Upload `.testkit/results` with `if: always()`
|
|
48
50
|
when CI evidence should survive a failed job.
|
|
49
51
|
|
|
50
|
-
Publishing that evidence to Testkit Cloud is an additional explicit step. Set
|
|
51
|
-
`TESTKIT_CLOUD_PROJECT_ID` and the secret `TESTKIT_CLOUD_TOKEN`, then run:
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
npx testkit cloud publish
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
The command verifies the immutable bundle before upload. Ordinary runs never
|
|
58
|
-
contact Cloud, and the presence of `CI` does not authorize publication.
|
|
59
|
-
|
|
60
|
-
## Connect to Cloud
|
|
61
|
-
|
|
62
|
-
For interactive use, approve your CLI through the browser once:
|
|
63
|
-
|
|
64
|
-
```sh
|
|
65
|
-
npx testkit cloud login
|
|
66
|
-
npx testkit cloud project create
|
|
67
|
-
npx testkit cloud publish --project PROJECT_ID --run RUN_ID
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
Login works outside Git. Project creation infers the repository from Git origin
|
|
71
|
-
or accepts `--repository owner/repo`. It resumes after any required provider
|
|
72
|
-
consent. `cloud list`, `cloud show`, and `cloud apply --file` provide discovery,
|
|
73
|
-
settings and product control. See the [Cloud control boundary](docs/testkit-cloud-boundary.md#complete-cloud-control)
|
|
74
|
-
and [complete operation reference](https://github.com/gfdelarue/testkit-cloud/blob/master/docs/cloud-control.md).
|
|
75
|
-
Explicit `TESTKIT_CLOUD_TOKEN` still takes precedence over the saved account.
|
|
76
|
-
|
|
77
52
|
## Execution engines
|
|
78
53
|
|
|
79
54
|
Testkit has three explicit built-in engines rather than one implicit runtime:
|
|
@@ -219,40 +194,9 @@ Batch `run` output stays intentionally short: one line per completed file,
|
|
|
219
194
|
concise failure blocks, and a final summary. Service logs, captured runtime
|
|
220
195
|
output, and emitted artifacts are persisted under `.testkit/results/`.
|
|
221
196
|
|
|
222
|
-
Testkit
|
|
223
|
-
|
|
224
|
-
output
|
|
225
|
-
Publish that exact evidence with `testkit cloud publish --run <id>`. Omitting
|
|
226
|
-
`--run` selects the newest bundle for interactive use. Historical summaries
|
|
227
|
-
and trends belong to Testkit Cloud.
|
|
228
|
-
|
|
229
|
-
For CI, preserve the run exit status and publish the ID from that invocation:
|
|
230
|
-
|
|
231
|
-
```sh
|
|
232
|
-
run_exit=0
|
|
233
|
-
testkit run --json > testkit-run.json || run_exit=$?
|
|
234
|
-
run_id=$(node -e 'const r = JSON.parse(require("node:fs").readFileSync("testkit-run.json", "utf8")); if (typeof r.runId !== "string" || !r.runId) process.exit(1); process.stdout.write(r.runId)') || {
|
|
235
|
-
if [ "$run_exit" -ne 0 ]; then exit "$run_exit"; fi
|
|
236
|
-
exit 1
|
|
237
|
-
}
|
|
238
|
-
publish_exit=0
|
|
239
|
-
testkit cloud publish --run "$run_id" || publish_exit=$?
|
|
240
|
-
if [ "$publish_exit" -ne 0 ]; then
|
|
241
|
-
echo "Cloud publication failed with exit $publish_exit" >&2
|
|
242
|
-
fi
|
|
243
|
-
exit "$run_exit"
|
|
244
|
-
```
|
|
245
|
-
|
|
246
|
-
Set `TESTKIT_CLOUD_PROJECT_ID` and `TESTKIT_CLOUD_TOKEN` in the CI environment.
|
|
247
|
-
`TESTKIT_CLOUD_URL`, or the `--url` flag on `cloud publish`, `cloud apply`, and
|
|
248
|
-
`cloud snapshot`, changes the Testkit Cloud origin. The bearer token and the
|
|
249
|
-
run bundle are sent only to that origin, so protect the variable like the
|
|
250
|
-
token itself. Leave it unset to use `https://testkit.cloud`.
|
|
251
|
-
A failed run can still publish a valid sealed bundle. Publication reports its
|
|
252
|
-
own exit status and does not change the test outcome. This recipe reports a
|
|
253
|
-
publication failure and retains the run status as the job status. See the
|
|
254
|
-
[Cloud boundary](docs/testkit-cloud-boundary.md) and
|
|
255
|
-
[breaking-change migration](https://github.com/gfdelarue/testkit/blob/master/docs/release-notes.md).
|
|
197
|
+
Testkit seals an immutable local run bundle. `testkit run` prints
|
|
198
|
+
`Run ID: <id>` after sealing in text output and returns `runId` in `--json`
|
|
199
|
+
output, for passed and failed runs.
|
|
256
200
|
|
|
257
201
|
Test execution also maintains a scheduler cache at `.testkit/timings.json`.
|
|
258
202
|
Completed file-level task durations are used to rank future runs with a
|
|
@@ -653,16 +597,6 @@ Programmatic consumers can import `listRunBundles`, `loadRunBundleArtifact`,
|
|
|
653
597
|
`loadRunBundleManifest`, and `verifyRunBundle` from
|
|
654
598
|
`@gfdelarue/testkit/workspace`.
|
|
655
599
|
|
|
656
|
-
The run bundle is also the boundary for Testkit Cloud. The local runner never
|
|
657
|
-
reads regression state or mutates GitHub. See
|
|
658
|
-
[`docs/testkit-cloud-boundary.md`](docs/testkit-cloud-boundary.md) for ownership,
|
|
659
|
-
ingestion, and migration rules.
|
|
660
|
-
|
|
661
|
-
Use `testkit cloud snapshot` to read versioned Cloud regression state. Apply
|
|
662
|
-
reviewed mutations from a file with `testkit cloud apply --file commands.json`;
|
|
663
|
-
the file must contain a schema version, idempotency key, expected snapshot
|
|
664
|
-
version, and commands.
|
|
665
|
-
|
|
666
600
|
## Authoring
|
|
667
601
|
|
|
668
602
|
### Node setup beside a test
|
|
@@ -1039,7 +973,7 @@ identifier plus canonical metadata such as:
|
|
|
1039
973
|
- `dependsOn`
|
|
1040
974
|
- `displayName`
|
|
1041
975
|
|
|
1042
|
-
|
|
976
|
+
Discovery output has no cross-run counts, last outcomes, durations, or trends.
|
|
1043
977
|
See the [status and history migration](https://github.com/gfdelarue/testkit/blob/master/docs/release-notes.md#local-status-and-history-removal)
|
|
1044
978
|
for the removed discovery fields and consumer migration.
|
|
1045
979
|
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
"entrypoint": "dist/process/process.js",
|
|
4
4
|
"sourceSha256": "a54f01864b603cee568b2945390dff35d578e07f6d26b5add06014169c3d17b0",
|
|
5
5
|
"bundleSha256": "62f76cf9fd019f6c552bfde2787f058917caf279670a62e7a6bfa12be34cb4b1",
|
|
6
|
-
"packageLockSha256": "
|
|
6
|
+
"packageLockSha256": "c0f28403d05b2838f4d0d104862548469ad7cc74e1aeb9fb9a33c7df856d7f5b",
|
|
7
7
|
"esbuildVersion": "0.28.1",
|
|
8
8
|
"options": {
|
|
9
9
|
"format": "esm",
|
package/docs/cloud-control.md
CHANGED
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Cloud command reference
|
|
2
2
|
|
|
3
|
+
The `testkit cloud` commands are deprecated. See the
|
|
4
|
+
[Cloud boundary](testkit-cloud-boundary.md) for the direction.
|
|
5
|
+
|
|
3
6
|
`testkit cloud apply --help` prints the account-command envelope, every supported
|
|
4
7
|
operation, its scope IDs, and its input fields. This reference ships with Testkit.
|
|
5
8
|
|
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
# Testkit Cloud boundary
|
|
2
2
|
|
|
3
|
-
The
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
The `testkit cloud` commands are deprecated. Testkit works without Cloud, and
|
|
4
|
+
bugs and evidence stay in your repository and in GitHub or Gitea. Testkit Cloud
|
|
5
|
+
is becoming an optional analytics dashboard that reads facts from GitHub or
|
|
6
|
+
Gitea instead of receiving run bundles. This document describes the commands
|
|
7
|
+
and the publication contract that still ship.
|
|
6
8
|
|
|
7
9
|
```text
|
|
8
10
|
discover -> prepare -> execute -> seal and verify run bundle -> clean up
|
|
@@ -13,13 +15,11 @@ discover -> prepare -> execute -> seal and verify run bundle -> clean up
|
|
|
13
15
|
Testkit Cloud ingest
|
|
14
16
|
```
|
|
15
17
|
|
|
16
|
-
##
|
|
18
|
+
## Cloud commands
|
|
17
19
|
|
|
18
|
-
The CLI is a full control client for Cloud. New or changed product operations
|
|
19
|
-
ship their API, installed command or file input, and acceptance coverage together.
|
|
20
20
|
The local [Cloud command reference](cloud-control.md) lists supported operations
|
|
21
21
|
and file fields. `testkit cloud apply --help` includes the account-command format.
|
|
22
|
-
Cloud
|
|
22
|
+
Cloud checks identity, permissions, and domain rules on every command.
|
|
23
23
|
|
|
24
24
|
```sh
|
|
25
25
|
testkit cloud login
|
|
@@ -176,13 +176,13 @@ The token is an environment secret. It is not stored in `testkit.config.ts` or
|
|
|
176
176
|
accepted as a command flag. `CI` is execution context only; it does not enable
|
|
177
177
|
publication or supply authority.
|
|
178
178
|
|
|
179
|
-
## Cloud
|
|
179
|
+
## Cloud ingestion
|
|
180
180
|
|
|
181
|
-
Testkit Cloud
|
|
181
|
+
Testkit Cloud ingests a verified bundle and derives regression identities,
|
|
182
182
|
baselines, comparisons, recommendations, GitHub checks, and issue lifecycle
|
|
183
183
|
actions. Cloud state is separate from the product repository and may be shared
|
|
184
|
-
across repositories. Cloud
|
|
185
|
-
|
|
184
|
+
across repositories. Cloud keeps the source bundle identity for every derived
|
|
185
|
+
decision.
|
|
186
186
|
|
|
187
187
|
Cloud ingestion uses an explicit versioned contract. It rejects an invalid
|
|
188
188
|
bundle, preserves the underlying run outcome, and reports GitHub projection
|
|
@@ -197,21 +197,9 @@ The old `./regression-workflow` package export, runner integration,
|
|
|
197
197
|
Git retains their implementation history, but that does not migrate a
|
|
198
198
|
consumer's live state.
|
|
199
199
|
|
|
200
|
-
Before a Testkit package containing this removal is published:
|
|
201
|
-
|
|
202
|
-
1. Inventory known uses of the removed export and state file.
|
|
203
|
-
2. Assign each consumer an owner and decide whether to discard or import its
|
|
204
|
-
stored state.
|
|
205
|
-
3. Rehearse required consumers against the exact packed candidate.
|
|
206
|
-
4. Block publication when a required consumer cannot move atomically.
|
|
207
|
-
|
|
208
|
-
The removed implementation is useful source material for Testkit Cloud, but it
|
|
209
|
-
must not be copied back into the runner. Cloud should consume verified bundles
|
|
210
|
-
through its own adapter and own its persistence and GitHub credentials.
|
|
211
|
-
|
|
212
200
|
## Report an externally executed campaign
|
|
213
201
|
|
|
214
|
-
Cloud
|
|
202
|
+
Cloud keeps campaign state, finding verification, and counts. A user or agent
|
|
215
203
|
starts and runs the investigation independently. These commands never start
|
|
216
204
|
execution or infer a finding from a failed assertion.
|
|
217
205
|
|
package/package.json
CHANGED
package/testkit-config-schema.md
CHANGED
|
@@ -758,6 +758,8 @@ cloud?: {
|
|
|
758
758
|
};
|
|
759
759
|
```
|
|
760
760
|
|
|
761
|
+
This setting is deprecated with the `testkit cloud` commands.
|
|
762
|
+
|
|
761
763
|
`cloud.verification.minimumConfirmingRuns` is an integer from 1 through 64,
|
|
762
764
|
defaulting to 2. It controls whether Testkit submits a new `finding.verify`
|
|
763
765
|
campaign report. It counts distinct full run IDs from available cited accepted
|