@gfdelarue/testkit 0.4.4 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +752 -462
- package/bin/testkit-windows-owner-arm64.exe +0 -0
- package/bin/testkit-windows-owner-x64.exe +0 -0
- package/dist/app/bugs/categories.js +1 -1
- package/dist/app/bugs/ci-report.js +2 -2
- package/dist/app/bugs/ci-summary.js +3 -3
- package/dist/app/bugs/draft.js +4 -4
- package/dist/app/bugs/facts.js +2 -2
- package/dist/app/bugs/file.js +1 -1
- package/dist/app/bugs/gate.js +1 -1
- package/dist/app/bugs/guide.js +3 -3
- package/dist/app/bugs/issue.js +4 -4
- package/dist/app/bugs/known-failures.js +1 -1
- package/dist/app/bugs/record.js +2 -2
- package/dist/app/bugs/run-reading.js +1 -1
- package/dist/app/bugs/supplied-reproduce.js +1 -0
- package/dist/app/campaigns/campaign.js +1 -1
- package/dist/app/cleanup/index.js +1 -1
- package/dist/app/db-schema/prepare.js +1 -1
- package/dist/app/destroy/index.js +1 -1
- package/dist/app/discover/index.js +2 -2
- package/dist/app/doctor/index.js +1 -1
- package/dist/app/environments/config-driven.js +1 -0
- package/dist/app/environments/environment-file.js +1 -0
- package/dist/app/environments/file-setup.js +1 -0
- package/dist/app/environments/routing.js +1 -0
- package/dist/app/environments/static-values.js +1 -0
- package/dist/app/environments/supplied-input.js +1 -0
- package/dist/app/environments/supplied.js +1 -0
- package/dist/app/environments/task-databases.js +2 -0
- package/dist/app/file-selection.js +1 -1
- package/dist/app/inspect/artifacts.js +1 -1
- package/dist/app/inspect/index.js +1 -1
- package/dist/app/lint/index.js +1 -1
- package/dist/app/lint/native.js +1 -1
- package/dist/app/lint/options.js +1 -1
- package/dist/app/lint/rules/data.js +1 -1
- package/dist/app/lint/rules/engine-imports.js +1 -1
- package/dist/app/lint/rules/syntax.js +1 -1
- package/dist/app/lint/rules/unit.js +1 -1
- package/dist/app/lint/shared.js +1 -1
- package/dist/app/package-inspection.js +1 -1
- package/dist/app/reports/files.js +1 -0
- package/dist/app/reports/prepare.js +2 -0
- package/dist/app/reports/publish.js +1 -0
- package/dist/app/run/caller-redaction.js +1 -0
- package/dist/app/run/engine-evidence-persistence.js +1 -1
- package/dist/app/run/engine-execution-failures.js +1 -1
- package/dist/app/run/engine-execution-reporting.js +1 -1
- package/dist/app/run/engine-execution.js +1 -1
- package/dist/app/run/engine-orchestration.js +1 -1
- package/dist/app/run/environment.js +1 -0
- package/dist/app/run/failure-details.js +1 -1
- package/dist/app/run/live-run.js +1 -1
- package/dist/app/run/orchestrator.js +1 -1
- package/dist/app/run/planning.js +1 -1
- package/dist/app/run/request.js +1 -1
- package/dist/app/run/results.js +1 -1
- package/dist/app/run/run-events.js +1 -1
- package/dist/app/run/run-finalization.js +1 -1
- package/dist/app/run/runtime-fixtures.js +1 -1
- package/dist/app/run/scheduler/estimates.js +1 -1
- package/dist/app/run/scheduler/identity.js +1 -1
- package/dist/app/run/scheduler/index.js +1 -1
- package/dist/app/run/scheduler/locks.js +1 -0
- package/dist/app/run/selection.js +1 -1
- package/dist/app/run/setup/runtime-node-setup-protocol.js +1 -1
- package/dist/app/run/setup/runtime-node-setup-worker.js +1 -1
- package/dist/app/run/setup/runtime-node-setup.js +2 -2
- package/dist/app/run/task-batches.js +1 -1
- package/dist/app/run/task-leases.js +1 -0
- package/dist/app/run/worker-loop.js +1 -1
- package/dist/app/status/runs.js +1 -1
- package/dist/app/tracker/index.js +1 -1
- package/dist/app/typecheck/index.js +4 -4
- package/dist/cli/command-catalog.js +1 -1
- package/dist/cli/command-flags.js +1 -1
- package/dist/cli/command-registry.js +1 -1
- package/dist/cli/commands/artifacts.js +2 -2
- package/dist/cli/commands/bug/file.js +2 -2
- package/dist/cli/commands/bug/record.js +2 -2
- package/dist/cli/commands/discover.js +1 -1
- package/dist/cli/commands/report/prepare.js +5 -0
- package/dist/cli/commands/report/publish.js +2 -0
- package/dist/cli/commands/run.js +2 -2
- package/dist/cli/commands/typecheck.js +1 -1
- package/dist/cli/commands/workspace.js +1 -1
- package/dist/cli/framework/errors.js +1 -1
- package/dist/cli/framework/execute.js +7 -6
- package/dist/cli/framework/parser.js +1 -1
- package/dist/cli/json-errors.js +1 -1
- package/dist/cli/operations/run/operation.js +2 -2
- package/dist/cli/renderers/bug/text.js +1 -1
- package/dist/cli/renderers/discover/report.js +1 -1
- package/dist/cli/renderers/doctor/text.js +1 -1
- package/dist/cli/renderers/run/events.js +2 -2
- package/dist/cli/renderers/run/formatting.js +1 -1
- package/dist/cli/renderers/run/status.js +1 -1
- package/dist/cli/renderers/run/text-reporter.js +24 -22
- package/dist/cli/terminal/colors.js +1 -1
- package/dist/cli/terminal/layout.js +2 -2
- package/dist/config/authoring/http-methods.js +1 -1
- package/dist/config/authoring/index.js +1 -1
- package/dist/config/authoring/preset-env.js +1 -1
- package/dist/config/loader/config-loader.js +1 -1
- package/dist/config/loader/database.js +1 -1
- package/dist/config/loader/decode-execution.js +1 -1
- package/dist/config/loader/dependencies.js +1 -1
- package/dist/config/loader/discover-tests.js +1 -1
- package/dist/config/loader/discovery-summary.js +1 -1
- package/dist/config/loader/discovery.js +2 -2
- package/dist/config/loader/env.js +1 -1
- package/dist/config/loader/execution-model.js +1 -1
- package/dist/config/loader/file-metadata.js +1 -1
- package/dist/config/loader/groups.js +1 -0
- package/dist/config/loader/index.js +1 -1
- package/dist/config/loader/load.js +1 -1
- package/dist/config/loader/managed-configs.js +1 -1
- package/dist/config/loader/owned-discovery.js +1 -0
- package/dist/config/loader/path-policy.js +1 -1
- package/dist/config/loader/runtime-isolation.js +1 -1
- package/dist/config/loader/runtime.js +1 -1
- package/dist/config/loader/skip-config.js +1 -1
- package/dist/config/loader/task-policy.js +1 -0
- package/dist/config/loader/test-selection.js +1 -1
- package/dist/config/loader/validation.js +1 -1
- package/dist/database/activity-gate.js +1 -1
- package/dist/database/admin.js +2 -2
- package/dist/database/bindings.js +1 -0
- package/dist/database/cleanup.js +1 -1
- package/dist/database/docker-metadata.js +3 -3
- package/dist/database/docker.js +3 -3
- package/dist/database/index.js +1 -1
- package/dist/database/lifecycle-cleanup.js +1 -1
- package/dist/database/local-postgres.js +1 -1
- package/dist/database/naming.js +1 -1
- package/dist/database/postgres/connection-check.js +1 -0
- package/dist/database/postgres/protocol.js +2 -0
- package/dist/database/schema-operations.js +1 -1
- package/dist/database/schema-source.js +3 -3
- package/dist/database/source-schema-env.js +1 -0
- package/dist/database/source-url.js +2 -2
- package/dist/database/state-files.js +1 -1
- package/dist/database/state.js +1 -1
- package/dist/database/steps.js +3 -3
- package/dist/domain/file-timeout.js +1 -1
- package/dist/domain/identifiers.js +1 -1
- package/dist/domain/operation-display.js +1 -1
- package/dist/domain/retired-test-types.js +1 -0
- package/dist/domain/runtime-isolation-contract.js +1 -1
- package/dist/domain/suite-selection.js +1 -1
- package/dist/domain/test-outcomes.js +1 -1
- package/dist/domain/test-types.js +1 -1
- package/dist/engines/adapters/evidence-files.js +1 -1
- package/dist/engines/adapters/evidence-identity.js +1 -1
- package/dist/engines/adapters/execution.js +1 -1
- package/dist/engines/adapters/json-bounds.js +2 -0
- package/dist/engines/adapters/native-report.js +1 -1
- package/dist/engines/api/execution-scope.js +1 -1
- package/dist/engines/api/execution.js +1 -1
- package/dist/engines/api/index.js +1 -1
- package/dist/engines/api/runtime-artifact.js +1 -1
- package/dist/engines/bundle-cache.js +1 -1
- package/dist/engines/catalog.js +1 -1
- package/dist/engines/contracts/environment-values.js +1 -0
- package/dist/engines/contracts/fixture-manifest.js +1 -1
- package/dist/engines/definitions.js +1 -1
- package/dist/engines/environment/handles.js +1 -0
- package/dist/engines/environment/index.js +1 -0
- package/dist/engines/factory.js +1 -1
- package/dist/engines/installations/node-vitest.js +1 -1
- package/dist/engines/installations/package-resolution.js +1 -1
- package/dist/engines/installations/playwright-resolution-child.js +1 -1
- package/dist/engines/installations/playwright.js +1 -1
- package/dist/engines/managed-module-resolution-policy-input.js +1 -1
- package/dist/engines/managed-module-resolution-policy.js +1 -1
- package/dist/engines/managed-module-resolution.js +2 -2
- package/dist/engines/manager.js +1 -1
- package/dist/engines/node-vitest/adapter/adapter.js +1 -1
- package/dist/engines/node-vitest/adapter/driver.js +2 -0
- package/dist/engines/node-vitest/adapter/evidence.js +2 -2
- package/dist/engines/node-vitest/adapter/failure-details.js +4 -2
- package/dist/engines/node-vitest/adapter/reporter.js +1 -1
- package/dist/engines/node-vitest/adapter/workspace.js +31 -192
- package/dist/engines/node-vitest/inventory/index.js +2 -0
- package/dist/engines/node-vitest/inventory/list-files.js +3 -0
- package/dist/engines/node-vitest/runtime/contract/index.js +1 -1
- package/dist/engines/node-vitest/runtime/data/http.js +1 -1
- package/dist/engines/node-vitest/runtime/data/index.js +1 -1
- package/dist/engines/node-vitest/runtime/data/report.js +5 -6
- package/dist/engines/node-vitest/runtime/data/scenario.js +2 -2
- package/dist/engines/node-vitest/runtime/runtime-input.js +1 -1
- package/dist/engines/node-vitest/selection.js +1 -0
- package/dist/engines/playwright/adapter/adapter.js +1 -1
- package/dist/engines/playwright/adapter/config.js +4 -4
- package/dist/engines/playwright/adapter/evidence-validation.js +1 -1
- package/dist/engines/playwright/adapter/evidence.js +1 -1
- package/dist/engines/playwright/adapter/reporter.js +1 -1
- package/dist/engines/playwright/runtime/config-file.js +1 -1
- package/dist/engines/playwright/runtime/config.js +1 -1
- package/dist/engines/playwright/runtime/runtime-input.js +43 -20
- package/dist/engines/runtime-input-file.js +1 -1
- package/dist/environment/lifecycle-artifacts.js +1 -1
- package/dist/environment/lifecycle-controller.js +1 -1
- package/dist/environment/lifecycle.js +1 -1
- package/dist/environment/ports.js +1 -0
- package/dist/environment/ready-environment.js +1 -1
- package/dist/environment/run-environment.js +1 -1
- package/dist/environment/run-stopping.js +1 -1
- package/dist/environment/runtime-contexts.js +1 -1
- package/dist/environment/runtime-manager.js +1 -1
- package/dist/environment/runtime-planning.js +1 -1
- package/dist/environment/runtime-preparation.js +1 -1
- package/dist/environment/services.js +3 -1
- package/dist/environment/state.js +1 -1
- package/dist/environment/template.js +1 -1
- package/dist/evidence/artifact-config.js +1 -1
- package/dist/evidence/artifact-validation.js +1 -1
- package/dist/evidence/artifacts.js +2 -2
- package/dist/evidence/bundles.js +1 -1
- package/dist/evidence/environment-record.js +1 -0
- package/dist/evidence/index.js +1 -1
- package/dist/evidence/logs.js +2 -2
- package/dist/evidence/operation-timings.js +1 -1
- package/dist/evidence/read-fields.js +1 -1
- package/dist/evidence/read-run.js +1 -1
- package/dist/evidence/reading.js +1 -1
- package/dist/evidence/redaction.js +4 -0
- package/dist/evidence/reporting.js +1 -1
- package/dist/evidence/run-model.js +1 -0
- package/dist/evidence/storage.js +2 -2
- package/dist/evidence/timing.js +1 -1
- package/dist/kernel/error-serialization.js +1 -1
- package/dist/kernel/redaction-targets.js +1 -0
- package/dist/kernel/redaction.js +4 -1
- package/dist/kernel/unknown-values.js +1 -1
- package/dist/process/environment.js +1 -1
- package/dist/process/execa-bundle.json +3 -3
- package/dist/process/files/json-file.js +1 -1
- package/dist/process/files/paths.js +1 -1
- package/dist/process/host/metadata.js +1 -1
- package/dist/process/host/platform.js +1 -1
- package/dist/process/host-process.js +1 -1
- package/dist/process/index.js +1 -1
- package/dist/process/lifecycle-fence.js +1 -1
- package/dist/process/locks.js +4 -4
- package/dist/process/port-listener.js +2 -0
- package/dist/process/process-group.js +1 -1
- package/dist/process/process-identity.js +1 -1
- package/dist/process/process.js +33 -33
- package/dist/process/readiness.js +1 -1
- package/dist/process/ready-deadline.js +1 -0
- package/dist/process/recovery.js +1 -1
- package/dist/process/runs/manifest.js +1 -1
- package/dist/process/runs/ownership.js +1 -1
- package/dist/process/runs/product-state.js +1 -1
- package/dist/process/runs/stop.js +1 -1
- package/dist/process/service-processes.js +1 -1
- package/dist/process/stoppable-settlement.js +1 -0
- package/dist/process/url-redaction.js +1 -0
- package/dist/process/windows-cli-owner.js +3 -3
- package/dist/process/windows-native.js +1 -1
- package/dist/repo/ci-environment.js +1 -1
- package/dist/repo/fingerprint-policy.js +1 -1
- package/dist/repo/managed-git.js +1 -1
- package/dist/repo/physical-tree.js +1 -1
- package/dist/repo/publish-directory.js +3 -0
- package/dist/repo/state.js +4 -4
- package/dist/steps/build-config.js +1 -1
- package/dist/steps/configured-step-testkit-modules.js +1 -1
- package/dist/steps/configured-steps.js +1 -1
- package/dist/steps/environment.js +1 -1
- package/dist/steps/index.js +1 -1
- package/dist/steps/run.js +2 -2
- package/dist/toolchains/archive.js +1 -1
- package/dist/toolchains/index.js +3 -3
- package/dist/toolchains/node-provisioning.js +2 -2
- package/dist/toolchains/semver.js +1 -1
- package/dist/toolchains/vitest/index.js +1 -1
- package/dist/tracker/redaction.js +1 -2
- package/dist/{config/runtime/build-assets.d.ts → types/config-build-assets.d.ts} +5 -3
- package/dist/{database/steps.d.ts → types/config-database-steps.d.ts} +7 -7
- package/dist/{config/runtime/next-runtime-tsconfig.d.ts → types/config-next-runtime-tsconfig.d.ts} +5 -3
- package/dist/types/config.d.ts +501 -0
- package/dist/types/drizzle.d.ts +13 -0
- package/dist/{config/runtime/env/index.d.ts → types/env.d.ts} +22 -20
- package/dist/types/environment.d.ts +40 -0
- package/dist/types/index.d.ts +21 -0
- package/dist/types/node-vitest-contract.d.ts +75 -0
- package/dist/types/node-vitest-data.d.ts +407 -0
- package/dist/types/playwright.d.ts +130 -0
- package/dist/types/tooling-vitest.d.ts +13 -0
- package/dist/types/workspace.d.ts +617 -0
- package/docs/bug-records.md +83 -81
- package/docs/engines.md +213 -0
- package/docs/test-types.md +35 -33
- package/package.json +22 -29
- package/testkit-config-schema.md +403 -171
- package/dist/app/bugs/areas.d.ts +0 -38
- package/dist/app/bugs/assessment.d.ts +0 -9
- package/dist/app/bugs/categories.d.ts +0 -41
- package/dist/app/bugs/ci-report.d.ts +0 -21
- package/dist/app/bugs/ci-summary.d.ts +0 -13
- package/dist/app/bugs/draft.d.ts +0 -70
- package/dist/app/bugs/facts.d.ts +0 -48
- package/dist/app/bugs/file.d.ts +0 -45
- package/dist/app/bugs/gate.d.ts +0 -22
- package/dist/app/bugs/guide.d.ts +0 -20
- package/dist/app/bugs/issue.d.ts +0 -18
- package/dist/app/bugs/known-failures.d.ts +0 -49
- package/dist/app/bugs/record.d.ts +0 -14
- package/dist/app/bugs/run-reading.d.ts +0 -51
- package/dist/app/bugs/store.d.ts +0 -34
- package/dist/app/campaigns/brief.d.ts +0 -7
- package/dist/app/campaigns/campaign.d.ts +0 -14
- package/dist/app/campaigns/index.d.ts +0 -38
- package/dist/app/campaigns/store.d.ts +0 -5
- package/dist/app/campaigns/summary.d.ts +0 -13
- package/dist/app/cleanup/index.d.ts +0 -52
- package/dist/app/db-schema/prepare.d.ts +0 -11
- package/dist/app/db-schema/refresh.d.ts +0 -17
- package/dist/app/db-schema/verify.d.ts +0 -11
- package/dist/app/destroy/index.d.ts +0 -16
- package/dist/app/discover/index.d.ts +0 -8
- package/dist/app/doctor/index.d.ts +0 -17
- package/dist/app/file-selection.d.ts +0 -1
- package/dist/app/inspect/artifacts.d.ts +0 -30
- package/dist/app/inspect/index.d.ts +0 -112
- package/dist/app/lint/index.d.ts +0 -28
- package/dist/app/lint/native.d.ts +0 -10
- package/dist/app/lint/options.d.ts +0 -67
- package/dist/app/lint/policy.d.ts +0 -21
- package/dist/app/lint/rules/data.d.ts +0 -3
- package/dist/app/lint/rules/engine-imports.d.ts +0 -3
- package/dist/app/lint/rules/syntax.d.ts +0 -4
- package/dist/app/lint/rules/unit.d.ts +0 -3
- package/dist/app/lint/shared.d.ts +0 -36
- package/dist/app/package-inspection.d.ts +0 -20
- package/dist/app/run/endpoint-readiness.d.ts +0 -43
- package/dist/app/run/endpoint-readiness.js +0 -1
- package/dist/app/run/engine-evidence-persistence.d.ts +0 -39
- package/dist/app/run/engine-execution-cleanup.d.ts +0 -8
- package/dist/app/run/engine-execution-cleanup.js +0 -1
- package/dist/app/run/engine-execution-failures.d.ts +0 -41
- package/dist/app/run/engine-execution-observer.d.ts +0 -21
- package/dist/app/run/engine-execution-reporting.d.ts +0 -17
- package/dist/app/run/engine-execution.d.ts +0 -63
- package/dist/app/run/engine-failure-stack.d.ts +0 -1
- package/dist/app/run/engine-orchestration.d.ts +0 -49
- package/dist/app/run/failure-details.d.ts +0 -37
- package/dist/app/run/live-run.d.ts +0 -27
- package/dist/app/run/operation-progress.d.ts +0 -13
- package/dist/app/run/orchestrator.d.ts +0 -68
- package/dist/app/run/planning-types.d.ts +0 -68
- package/dist/app/run/planning.d.ts +0 -15
- package/dist/app/run/playwright-configuration.d.ts +0 -2
- package/dist/app/run/playwright-configuration.js +0 -1
- package/dist/app/run/request.d.ts +0 -30
- package/dist/app/run/results.d.ts +0 -161
- package/dist/app/run/run-events.d.ts +0 -144
- package/dist/app/run/run-finalization.d.ts +0 -52
- package/dist/app/run/run-guards.d.ts +0 -3
- package/dist/app/run/runtime-failures.d.ts +0 -18
- package/dist/app/run/runtime-fixtures.d.ts +0 -2
- package/dist/app/run/runtime-task-types.d.ts +0 -20
- package/dist/app/run/scheduler/estimates.d.ts +0 -32
- package/dist/app/run/scheduler/identity.d.ts +0 -9
- package/dist/app/run/scheduler/index.d.ts +0 -124
- package/dist/app/run/scheduler/observations.d.ts +0 -20
- package/dist/app/run/selection.d.ts +0 -10
- package/dist/app/run/setup/runtime-node-setup-errors.d.ts +0 -13
- package/dist/app/run/setup/runtime-node-setup-protocol.d.ts +0 -85
- package/dist/app/run/setup/runtime-node-setup-worker.d.ts +0 -1
- package/dist/app/run/setup/runtime-node-setup.d.ts +0 -48
- package/dist/app/run/task-batches.d.ts +0 -23
- package/dist/app/run/worker-loop.d.ts +0 -57
- package/dist/app/setup/playwright.d.ts +0 -22
- package/dist/app/status/index.d.ts +0 -9
- package/dist/app/status/model.d.ts +0 -29
- package/dist/app/status/runs.d.ts +0 -99
- package/dist/app/stop/index.d.ts +0 -14
- package/dist/app/tracker/index.d.ts +0 -8
- package/dist/app/tracker/remote.d.ts +0 -13
- package/dist/app/tracker/token.d.ts +0 -7
- package/dist/app/typecheck/index.d.ts +0 -63
- package/dist/cli/command-catalog.d.ts +0 -137
- package/dist/cli/command-flags.d.ts +0 -19
- package/dist/cli/command-registry.d.ts +0 -3
- package/dist/cli/commands/artifacts.d.ts +0 -48
- package/dist/cli/commands/bug/file.d.ts +0 -16
- package/dist/cli/commands/bug/guide.d.ts +0 -5
- package/dist/cli/commands/bug/record.d.ts +0 -9
- package/dist/cli/commands/campaign/finish.d.ts +0 -10
- package/dist/cli/commands/campaign/start.d.ts +0 -7
- package/dist/cli/commands/campaign/status.d.ts +0 -13
- package/dist/cli/commands/ci/report.d.ts +0 -15
- package/dist/cli/commands/cleanup.d.ts +0 -13
- package/dist/cli/commands/db/schema/refresh.d.ts +0 -2
- package/dist/cli/commands/db/schema/verify.d.ts +0 -2
- package/dist/cli/commands/destroy.d.ts +0 -9
- package/dist/cli/commands/discover.d.ts +0 -2
- package/dist/cli/commands/doctor.d.ts +0 -2
- package/dist/cli/commands/guard/unsafe-entrypoint.d.ts +0 -2
- package/dist/cli/commands/lint.d.ts +0 -2
- package/dist/cli/commands/run/status.d.ts +0 -5
- package/dist/cli/commands/run/stop.d.ts +0 -7
- package/dist/cli/commands/run.d.ts +0 -9
- package/dist/cli/commands/setup/playwright.d.ts +0 -2
- package/dist/cli/commands/status.d.ts +0 -5
- package/dist/cli/commands/stop.d.ts +0 -12
- package/dist/cli/commands/typecheck.d.ts +0 -2
- package/dist/cli/commands/workspace.d.ts +0 -2
- package/dist/cli/entrypoint.d.ts +0 -1
- package/dist/cli/framework/command.d.ts +0 -21
- package/dist/cli/framework/definitions.d.ts +0 -100
- package/dist/cli/framework/errors.d.ts +0 -6
- package/dist/cli/framework/execute.d.ts +0 -14
- package/dist/cli/framework/help.d.ts +0 -11
- package/dist/cli/framework/index.d.ts +0 -3
- package/dist/cli/framework/parser.d.ts +0 -6
- package/dist/cli/index.d.ts +0 -4
- package/dist/cli/json-errors.d.ts +0 -3
- package/dist/cli/operations/db/schema/refresh/operation.d.ts +0 -7
- package/dist/cli/operations/db/schema/verify/operation.d.ts +0 -7
- package/dist/cli/operations/run/operation.d.ts +0 -24
- package/dist/cli/process-signal.d.ts +0 -1
- package/dist/cli/renderers/bug/text.d.ts +0 -8
- package/dist/cli/renderers/campaign/text.d.ts +0 -2
- package/dist/cli/renderers/ci/text.d.ts +0 -2
- package/dist/cli/renderers/cleanup/format.d.ts +0 -14
- package/dist/cli/renderers/db-schema/text.d.ts +0 -4
- package/dist/cli/renderers/destroy/text.d.ts +0 -4
- package/dist/cli/renderers/discover/report.d.ts +0 -7
- package/dist/cli/renderers/discover/text.d.ts +0 -3
- package/dist/cli/renderers/doctor/text.d.ts +0 -2
- package/dist/cli/renderers/lint/text.d.ts +0 -2
- package/dist/cli/renderers/run/events.d.ts +0 -8
- package/dist/cli/renderers/run/failure.d.ts +0 -8
- package/dist/cli/renderers/run/formatting.d.ts +0 -79
- package/dist/cli/renderers/run/status.d.ts +0 -6
- package/dist/cli/renderers/run/text-reporter.d.ts +0 -19
- package/dist/cli/renderers/status/text.d.ts +0 -2
- package/dist/cli/renderers/stop/text.d.ts +0 -7
- package/dist/cli/renderers/typecheck/text.d.ts +0 -2
- package/dist/cli/terminal/colors.d.ts +0 -10
- package/dist/cli/terminal/layout.d.ts +0 -11
- package/dist/cli/version-drift.d.ts +0 -25
- package/dist/config/authoring/build-config-contract.d.ts +0 -27
- package/dist/config/authoring/configured-step-contract.d.ts +0 -18
- package/dist/config/authoring/http-execution-contract.d.ts +0 -70
- package/dist/config/authoring/http-methods.d.ts +0 -18
- package/dist/config/authoring/index.d.ts +0 -381
- package/dist/config/authoring/preset-env.d.ts +0 -19
- package/dist/config/authoring/runtime.d.ts +0 -64
- package/dist/config/loader/bugs.d.ts +0 -15
- package/dist/config/loader/config-loader.d.ts +0 -9
- package/dist/config/loader/database-materialization.d.ts +0 -19
- package/dist/config/loader/database-roles.d.ts +0 -2
- package/dist/config/loader/database.d.ts +0 -55
- package/dist/config/loader/decode-discovery.d.ts +0 -5
- package/dist/config/loader/decode-execution.d.ts +0 -9
- package/dist/config/loader/dependencies.d.ts +0 -7
- package/dist/config/loader/discover-tests.d.ts +0 -111
- package/dist/config/loader/discovery-summary.d.ts +0 -29
- package/dist/config/loader/discovery.d.ts +0 -54
- package/dist/config/loader/env.d.ts +0 -4
- package/dist/config/loader/execution-model.d.ts +0 -16
- package/dist/config/loader/file-metadata.d.ts +0 -19
- package/dist/config/loader/index.d.ts +0 -10
- package/dist/config/loader/load.d.ts +0 -91
- package/dist/config/loader/managed-configs.d.ts +0 -23
- package/dist/config/loader/path-policy.d.ts +0 -9
- package/dist/config/loader/paths.d.ts +0 -5
- package/dist/config/loader/runtime-isolation.d.ts +0 -44
- package/dist/config/loader/runtime.d.ts +0 -25
- package/dist/config/loader/skip-config.d.ts +0 -44
- package/dist/config/loader/test-selection.d.ts +0 -15
- package/dist/config/loader/validation.d.ts +0 -16
- package/dist/config/loader/values.d.ts +0 -4
- package/dist/config/loader/values.js +0 -1
- package/dist/database/activity-gate.d.ts +0 -22
- package/dist/database/admin.d.ts +0 -31
- package/dist/database/cleanup.d.ts +0 -36
- package/dist/database/configured-steps.d.ts +0 -27
- package/dist/database/constants.d.ts +0 -11
- package/dist/database/docker-metadata.d.ts +0 -3
- package/dist/database/docker.d.ts +0 -65
- package/dist/database/drizzle/index.d.ts +0 -13
- package/dist/database/fingerprint.d.ts +0 -12
- package/dist/database/index.d.ts +0 -39
- package/dist/database/lifecycle-cleanup.d.ts +0 -7
- package/dist/database/local-postgres.d.ts +0 -71
- package/dist/database/naming.d.ts +0 -17
- package/dist/database/postgres/index.d.ts +0 -71
- package/dist/database/progress.d.ts +0 -6
- package/dist/database/roles.d.ts +0 -19
- package/dist/database/runtime-config.d.ts +0 -16
- package/dist/database/runtime-reservations.d.ts +0 -19
- package/dist/database/schema-operations.d.ts +0 -18
- package/dist/database/schema-source.d.ts +0 -96
- package/dist/database/source-refresh-lock.d.ts +0 -8
- package/dist/database/source-url.d.ts +0 -22
- package/dist/database/state-files.d.ts +0 -13
- package/dist/database/state.d.ts +0 -26
- package/dist/database/template-steps.d.ts +0 -39
- package/dist/domain/file-timeout.d.ts +0 -28
- package/dist/domain/identifiers.d.ts +0 -9
- package/dist/domain/operation-display.d.ts +0 -10
- package/dist/domain/operation-reason.d.ts +0 -5
- package/dist/domain/postgres-route.d.ts +0 -4
- package/dist/domain/runtime-identity.d.ts +0 -1
- package/dist/domain/runtime-isolation-contract.d.ts +0 -5
- package/dist/domain/service-name.d.ts +0 -2
- package/dist/domain/suite-selection.d.ts +0 -20
- package/dist/domain/test-outcomes.d.ts +0 -3
- package/dist/domain/test-types.d.ts +0 -16
- package/dist/engines/adapters/evidence-files.d.ts +0 -32
- package/dist/engines/adapters/evidence-identity.d.ts +0 -12
- package/dist/engines/adapters/execution.d.ts +0 -42
- package/dist/engines/adapters/native-report.d.ts +0 -25
- package/dist/engines/adapters/node-startup-cache.d.ts +0 -8
- package/dist/engines/adapters/workspace.d.ts +0 -4
- package/dist/engines/api/capabilities.d.ts +0 -31
- package/dist/engines/api/capabilities.js +0 -1
- package/dist/engines/api/evidence.d.ts +0 -38
- package/dist/engines/api/execution-observer.d.ts +0 -8
- package/dist/engines/api/execution-scope.d.ts +0 -17
- package/dist/engines/api/execution.d.ts +0 -101
- package/dist/engines/api/index.d.ts +0 -5
- package/dist/engines/api/k6-runtime-input.d.ts +0 -26
- package/dist/engines/api/k6-runtime-input.js +0 -1
- package/dist/engines/api/runtime-artifact.d.ts +0 -19
- package/dist/engines/bundle-cache.d.ts +0 -32
- package/dist/engines/catalog.d.ts +0 -27
- package/dist/engines/contracts/fixture-manifest.d.ts +0 -34
- package/dist/engines/contracts/json-bounds.d.ts +0 -36
- package/dist/engines/contracts/json-bounds.js +0 -2
- package/dist/engines/contracts/k6-repository-configuration.d.ts +0 -12
- package/dist/engines/contracts/k6-repository-configuration.js +0 -1
- package/dist/engines/contracts/test-context.d.ts +0 -15
- package/dist/engines/contracts/test-context.js +0 -1
- package/dist/engines/definitions.d.ts +0 -22
- package/dist/engines/factory.d.ts +0 -6
- package/dist/engines/generated-performance.d.ts +0 -1
- package/dist/engines/installation-catalog.d.ts +0 -27
- package/dist/engines/installation-catalog.js +0 -1
- package/dist/engines/installations/k6.d.ts +0 -98
- package/dist/engines/installations/k6.js +0 -1
- package/dist/engines/installations/node-vitest.d.ts +0 -33
- package/dist/engines/installations/package-requirements.d.ts +0 -28
- package/dist/engines/installations/package-resolution.d.ts +0 -51
- package/dist/engines/installations/playwright-resolution-child.d.ts +0 -1
- package/dist/engines/installations/playwright.d.ts +0 -75
- package/dist/engines/installations/types.d.ts +0 -10
- package/dist/engines/k6/adapter/bundle.d.ts +0 -25
- package/dist/engines/k6/adapter/bundle.js +0 -112
- package/dist/engines/k6/adapter/evidence.d.ts +0 -15
- package/dist/engines/k6/adapter/evidence.js +0 -4
- package/dist/engines/k6/adapter/executor.d.ts +0 -13
- package/dist/engines/k6/adapter/executor.js +0 -1
- package/dist/engines/k6/adapter/index.d.ts +0 -1
- package/dist/engines/k6/adapter/index.js +0 -1
- package/dist/engines/k6/adapter/input.d.ts +0 -21
- package/dist/engines/k6/adapter/input.js +0 -1
- package/dist/engines/k6/adapter/native-summary.d.ts +0 -11
- package/dist/engines/k6/adapter/native-summary.js +0 -1
- package/dist/engines/k6/adapter/repository-configuration.d.ts +0 -2
- package/dist/engines/k6/adapter/repository-configuration.js +0 -1
- package/dist/engines/k6/authoring-type-paths.d.ts +0 -3
- package/dist/engines/k6/authoring-type-paths.js +0 -1
- package/dist/engines/k6/runtime/ambient-types/bundle-manifest.json +0 -146
- package/dist/engines/k6/runtime/ambient-types/k6/LICENSE +0 -21
- package/dist/engines/k6/runtime/ambient-types/k6/browser/index.d.ts +0 -6332
- package/dist/engines/k6/runtime/ambient-types/k6/crypto/index.d.ts +0 -218
- package/dist/engines/k6/runtime/ambient-types/k6/data/index.d.ts +0 -16
- package/dist/engines/k6/runtime/ambient-types/k6/encoding/index.d.ts +0 -46
- package/dist/engines/k6/runtime/ambient-types/k6/execution/index.d.ts +0 -127
- package/dist/engines/k6/runtime/ambient-types/k6/experimental/csv/index.d.ts +0 -96
- package/dist/engines/k6/runtime/ambient-types/k6/experimental/fs/index.d.ts +0 -104
- package/dist/engines/k6/runtime/ambient-types/k6/experimental/redis/index.d.ts +0 -821
- package/dist/engines/k6/runtime/ambient-types/k6/experimental/streams/index.d.ts +0 -198
- package/dist/engines/k6/runtime/ambient-types/k6/global.d.ts +0 -83
- package/dist/engines/k6/runtime/ambient-types/k6/html/index.d.ts +0 -1325
- package/dist/engines/k6/runtime/ambient-types/k6/http/index.d.ts +0 -806
- package/dist/engines/k6/runtime/ambient-types/k6/index.d.ts +0 -126
- package/dist/engines/k6/runtime/ambient-types/k6/metrics/index.d.ts +0 -66
- package/dist/engines/k6/runtime/ambient-types/k6/net/grpc/index.d.ts +0 -247
- package/dist/engines/k6/runtime/ambient-types/k6/options/index.d.ts +0 -459
- package/dist/engines/k6/runtime/ambient-types/k6/package.json +0 -79
- package/dist/engines/k6/runtime/ambient-types/k6/secrets/index.d.ts +0 -36
- package/dist/engines/k6/runtime/ambient-types/k6/timers/index.d.ts +0 -47
- package/dist/engines/k6/runtime/ambient-types/k6/websockets/index.d.ts +0 -324
- package/dist/engines/k6/runtime/ambient-types/k6/ws/index.d.ts +0 -283
- package/dist/engines/k6/runtime/ambient-types/xk6-sql/LICENSE +0 -201
- package/dist/engines/k6/runtime/ambient-types/xk6-sql/index.d.ts +0 -397
- package/dist/engines/k6/runtime/ambient-types/xk6-sql/package.json +0 -15
- package/dist/engines/k6/runtime/artifacts.d.ts +0 -2
- package/dist/engines/k6/runtime/artifacts.js +0 -1
- package/dist/engines/k6/runtime/checks.d.ts +0 -17
- package/dist/engines/k6/runtime/checks.js +0 -1
- package/dist/engines/k6/runtime/dal-fixtures.d.ts +0 -24
- package/dist/engines/k6/runtime/dal-fixtures.js +0 -1
- package/dist/engines/k6/runtime/dal-suite.d.ts +0 -24
- package/dist/engines/k6/runtime/dal-suite.js +0 -1
- package/dist/engines/k6/runtime/dal.d.ts +0 -10
- package/dist/engines/k6/runtime/dal.js +0 -1
- package/dist/engines/k6/runtime/fixtures.d.ts +0 -3
- package/dist/engines/k6/runtime/fixtures.js +0 -1
- package/dist/engines/k6/runtime/http-assertions.d.ts +0 -19
- package/dist/engines/k6/runtime/http-assertions.js +0 -1
- package/dist/engines/k6/runtime/http-checks.d.ts +0 -19
- package/dist/engines/k6/runtime/http-checks.js +0 -1
- package/dist/engines/k6/runtime/http-suite-runtime.d.ts +0 -103
- package/dist/engines/k6/runtime/http-suite-runtime.js +0 -1
- package/dist/engines/k6/runtime/http.d.ts +0 -15
- package/dist/engines/k6/runtime/http.js +0 -1
- package/dist/engines/k6/runtime/index.d.ts +0 -11
- package/dist/engines/k6/runtime/index.js +0 -1
- package/dist/engines/k6/runtime/input.d.ts +0 -3
- package/dist/engines/k6/runtime/input.js +0 -1
- package/dist/engines/k6/runtime/k6-repository-configuration.d.ts +0 -9
- package/dist/engines/k6/runtime/k6-repository-configuration.js +0 -1
- package/dist/engines/k6/runtime/node-setup.d.ts +0 -13
- package/dist/engines/k6/runtime/node-setup.js +0 -1
- package/dist/engines/k6/runtime/runtime-globals.d.ts +0 -10
- package/dist/engines/k6/runtime/runtime.d.ts +0 -54
- package/dist/engines/k6/runtime/runtime.js +0 -1
- package/dist/engines/k6/runtime/shared/assertion-details.d.ts +0 -19
- package/dist/engines/k6/runtime/shared/assertion-details.js +0 -2
- package/dist/engines/k6/runtime/shared/db-normalization.d.ts +0 -19
- package/dist/engines/k6/runtime/shared/db-normalization.js +0 -1
- package/dist/engines/k6/runtime/shared/error-body.d.ts +0 -3
- package/dist/engines/k6/runtime/shared/error-body.js +0 -1
- package/dist/engines/k6/runtime/shared/file-data.d.ts +0 -1
- package/dist/engines/k6/runtime/shared/file-data.js +0 -1
- package/dist/engines/k6/runtime/shared/fixture-engine.d.ts +0 -8
- package/dist/engines/k6/runtime/shared/fixture-engine.js +0 -2
- package/dist/engines/k6/runtime/shared/http-check-plan.d.ts +0 -29
- package/dist/engines/k6/runtime/shared/http-check-plan.js +0 -1
- package/dist/engines/k6/runtime/shared/http-parsing.d.ts +0 -10
- package/dist/engines/k6/runtime/shared/http-parsing.js +0 -3
- package/dist/engines/k6/runtime/shared/ordering.d.ts +0 -2
- package/dist/engines/k6/runtime/shared/ordering.js +0 -1
- package/dist/engines/k6/runtime/shared/properties.d.ts +0 -1
- package/dist/engines/k6/runtime/shared/properties.js +0 -1
- package/dist/engines/k6/runtime/suite.d.ts +0 -4
- package/dist/engines/k6/runtime/suite.js +0 -1
- package/dist/engines/k6/runtime/types.d.ts +0 -244
- package/dist/engines/k6/runtime/types.js +0 -1
- package/dist/engines/k6/runtime/xk6-sql-driver-postgres.d.ts +0 -7
- package/dist/engines/k6/runtime/xk6-sql-driver-postgres.js +0 -1
- package/dist/engines/managed-module-resolution-policy-input.d.ts +0 -9
- package/dist/engines/managed-module-resolution-policy.d.ts +0 -25
- package/dist/engines/managed-module-resolution.d.ts +0 -13
- package/dist/engines/manager.d.ts +0 -25
- package/dist/engines/node-vitest/adapter/adapter.d.ts +0 -16
- package/dist/engines/node-vitest/adapter/consumer-typescript-paths.d.ts +0 -17
- package/dist/engines/node-vitest/adapter/consumer-typescript-paths.js +0 -3
- package/dist/engines/node-vitest/adapter/evidence.d.ts +0 -33
- package/dist/engines/node-vitest/adapter/failure-details.d.ts +0 -3
- package/dist/engines/node-vitest/adapter/index.d.ts +0 -1
- package/dist/engines/node-vitest/adapter/reporter.d.ts +0 -7
- package/dist/engines/node-vitest/adapter/resolution.d.ts +0 -12
- package/dist/engines/node-vitest/adapter/resolution.js +0 -1
- package/dist/engines/node-vitest/adapter/workspace.d.ts +0 -19
- package/dist/engines/node-vitest/runtime/contract/index.d.ts +0 -64
- package/dist/engines/node-vitest/runtime/data/http.d.ts +0 -4
- package/dist/engines/node-vitest/runtime/data/index.d.ts +0 -8
- package/dist/engines/node-vitest/runtime/data/primitives.d.ts +0 -1
- package/dist/engines/node-vitest/runtime/data/report.d.ts +0 -3
- package/dist/engines/node-vitest/runtime/data/runner.d.ts +0 -4
- package/dist/engines/node-vitest/runtime/data/scenario.d.ts +0 -12
- package/dist/engines/node-vitest/runtime/data/types.d.ts +0 -412
- package/dist/engines/node-vitest/runtime/runtime-input.d.ts +0 -64
- package/dist/engines/playwright/adapter/adapter.d.ts +0 -18
- package/dist/engines/playwright/adapter/attachments.d.ts +0 -26
- package/dist/engines/playwright/adapter/config.d.ts +0 -45
- package/dist/engines/playwright/adapter/evidence-validation.d.ts +0 -12
- package/dist/engines/playwright/adapter/evidence.d.ts +0 -32
- package/dist/engines/playwright/adapter/failure-details.d.ts +0 -2
- package/dist/engines/playwright/adapter/index.d.ts +0 -3
- package/dist/engines/playwright/adapter/reporter.d.ts +0 -12
- package/dist/engines/playwright/runtime/config-file.d.ts +0 -2
- package/dist/engines/playwright/runtime/config.d.ts +0 -24
- package/dist/engines/playwright/runtime/index.d.ts +0 -4
- package/dist/engines/playwright/runtime/runtime-input.d.ts +0 -36
- package/dist/engines/playwright/runtime/sandbox.d.ts +0 -84
- package/dist/engines/runtime-input-file.d.ts +0 -27
- package/dist/environment/lifecycle-artifacts.d.ts +0 -7
- package/dist/environment/lifecycle-controller.d.ts +0 -70
- package/dist/environment/lifecycle.d.ts +0 -29
- package/dist/environment/maintenance.d.ts +0 -68
- package/dist/environment/planning-types.d.ts +0 -35
- package/dist/environment/readiness.d.ts +0 -12
- package/dist/environment/readiness.js +0 -1
- package/dist/environment/ready-environment.d.ts +0 -19
- package/dist/environment/run-environment.d.ts +0 -38
- package/dist/environment/run-stopping.d.ts +0 -13
- package/dist/environment/runtime-contexts.d.ts +0 -33
- package/dist/environment/runtime-manager.d.ts +0 -95
- package/dist/environment/runtime-planning.d.ts +0 -16
- package/dist/environment/runtime-preparation.d.ts +0 -30
- package/dist/environment/runtime-service.d.ts +0 -18
- package/dist/environment/services.d.ts +0 -48
- package/dist/environment/state.d.ts +0 -18
- package/dist/environment/status-runtime.d.ts +0 -53
- package/dist/environment/template.d.ts +0 -180
- package/dist/evidence/artifact-config.d.ts +0 -14
- package/dist/evidence/artifact-validation.d.ts +0 -5
- package/dist/evidence/artifacts.d.ts +0 -75
- package/dist/evidence/bundles.d.ts +0 -27
- package/dist/evidence/ci-provenance.d.ts +0 -9
- package/dist/evidence/contracts.d.ts +0 -154
- package/dist/evidence/file-evidence.d.ts +0 -23
- package/dist/evidence/finalization.d.ts +0 -9
- package/dist/evidence/index.d.ts +0 -8
- package/dist/evidence/logs.d.ts +0 -31
- package/dist/evidence/operation-timings.d.ts +0 -60
- package/dist/evidence/read-bundle.d.ts +0 -4
- package/dist/evidence/read-fields.d.ts +0 -22
- package/dist/evidence/read-provenance.d.ts +0 -2
- package/dist/evidence/read-redaction.d.ts +0 -2
- package/dist/evidence/read-retention.d.ts +0 -2
- package/dist/evidence/read-run.d.ts +0 -2
- package/dist/evidence/reading.d.ts +0 -28
- package/dist/evidence/reporting-types.d.ts +0 -330
- package/dist/evidence/reporting.d.ts +0 -3
- package/dist/evidence/secure-file-copy.d.ts +0 -24
- package/dist/evidence/setup-progress.d.ts +0 -63
- package/dist/evidence/step-output.d.ts +0 -16
- package/dist/evidence/storage-inventory.d.ts +0 -27
- package/dist/evidence/storage.d.ts +0 -27
- package/dist/evidence/timing.d.ts +0 -23
- package/dist/file.d.ts +0 -11
- package/dist/index.d.ts +0 -4
- package/dist/kernel/error-serialization.d.ts +0 -38
- package/dist/kernel/redaction.d.ts +0 -2
- package/dist/kernel/unknown-values.d.ts +0 -6
- package/dist/kernel/work-metrics.d.ts +0 -6
- package/dist/process/capture-output.d.ts +0 -7
- package/dist/process/command-shell.d.ts +0 -1
- package/dist/process/environment-overlay.d.ts +0 -17
- package/dist/process/environment.d.ts +0 -33
- package/dist/process/files/contained-path.d.ts +0 -8
- package/dist/process/files/directory-status.d.ts +0 -14
- package/dist/process/files/dotenv.d.ts +0 -2
- package/dist/process/files/file-snapshot.d.ts +0 -34
- package/dist/process/files/json-file.d.ts +0 -12
- package/dist/process/files/paths.d.ts +0 -1
- package/dist/process/files/product-dir.d.ts +0 -1
- package/dist/process/files/temp-dir.d.ts +0 -11
- package/dist/process/host/metadata.d.ts +0 -6
- package/dist/process/host/platform.d.ts +0 -57
- package/dist/process/host-command.d.ts +0 -4
- package/dist/process/host-inspection-command.d.ts +0 -13
- package/dist/process/host-process.d.ts +0 -19
- package/dist/process/index.d.ts +0 -9
- package/dist/process/lifecycle-fence.d.ts +0 -11
- package/dist/process/locks.d.ts +0 -28
- package/dist/process/process-contracts.d.ts +0 -78
- package/dist/process/process-group.d.ts +0 -12
- package/dist/process/process-identity.d.ts +0 -27
- package/dist/process/process-limits.d.ts +0 -6
- package/dist/process/process.d.ts +0 -12
- package/dist/process/readiness.d.ts +0 -37
- package/dist/process/recovery.d.ts +0 -3
- package/dist/process/runs/inspection.d.ts +0 -5
- package/dist/process/runs/manifest.d.ts +0 -86
- package/dist/process/runs/ownership.d.ts +0 -11
- package/dist/process/runs/product-state.d.ts +0 -2
- package/dist/process/runs/schema.d.ts +0 -3
- package/dist/process/runs/stop.d.ts +0 -19
- package/dist/process/runtime-socket.d.ts +0 -4
- package/dist/process/service-processes.d.ts +0 -13
- package/dist/process/shutdown.d.ts +0 -10
- package/dist/process/windows-cli-owner.d.ts +0 -2
- package/dist/process/windows-native.d.ts +0 -6
- package/dist/process/windows-process-identity.d.ts +0 -4
- package/dist/repo/ci-environment.d.ts +0 -7
- package/dist/repo/fingerprint-policy.d.ts +0 -15
- package/dist/repo/git-path.d.ts +0 -2
- package/dist/repo/history.d.ts +0 -17
- package/dist/repo/managed-git.d.ts +0 -10
- package/dist/repo/metadata.d.ts +0 -3
- package/dist/repo/physical-tree.d.ts +0 -37
- package/dist/repo/remote.d.ts +0 -6
- package/dist/repo/state.d.ts +0 -26
- package/dist/repo/tree-listing.d.ts +0 -10
- package/dist/steps/build-config.d.ts +0 -35
- package/dist/steps/configured-step-testkit-modules.d.ts +0 -19
- package/dist/steps/configured-steps.d.ts +0 -51
- package/dist/steps/environment.d.ts +0 -84
- package/dist/steps/index.d.ts +0 -2
- package/dist/steps/module-runner.d.ts +0 -1
- package/dist/steps/run.d.ts +0 -36
- package/dist/toolchains/archive.d.ts +0 -53
- package/dist/toolchains/esbuild.d.ts +0 -9
- package/dist/toolchains/host-node-support.d.ts +0 -3
- package/dist/toolchains/index.d.ts +0 -114
- package/dist/toolchains/managed-node-releases.d.ts +0 -28
- package/dist/toolchains/node-cache.d.ts +0 -20
- package/dist/toolchains/node-integrity.d.ts +0 -18
- package/dist/toolchains/node-provisioning.d.ts +0 -29
- package/dist/toolchains/semver.d.ts +0 -13
- package/dist/toolchains/shared-operation.d.ts +0 -11
- package/dist/toolchains/timeouts.d.ts +0 -1
- package/dist/toolchains/vitest/index.d.ts +0 -13
- package/dist/tracker/gitea.d.ts +0 -14
- package/dist/tracker/github.d.ts +0 -14
- package/dist/tracker/http.d.ts +0 -33
- package/dist/tracker/index.d.ts +0 -4
- package/dist/tracker/redaction.d.ts +0 -4
- package/dist/tracker/requests.d.ts +0 -17
- package/dist/tracker/responses.d.ts +0 -24
- package/dist/tracker/types.d.ts +0 -51
- package/docs/engine-architecture.md +0 -654
- /package/dist/engines/{k6/runtime/runtime-globals.js → api/profiles.js} +0 -0
package/README.md
CHANGED
|
@@ -4,17 +4,19 @@
|
|
|
4
4
|
|
|
5
5
|
# @gfdelarue/testkit
|
|
6
6
|
|
|
7
|
-
`@gfdelarue/testkit` is
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
7
|
+
`@gfdelarue/testkit` is a test runner and CLI for coding agents and CI. It runs
|
|
8
|
+
a repository's ordinary Vitest test files, such as `src/sum.test.ts`, as they
|
|
9
|
+
are, and typed `*.testkit.ts` suites for Playwright and Node. It assigns
|
|
10
|
+
tests to groups by directory, starts only the local services a group uses,
|
|
11
|
+
provisions Docker-managed local Postgres databases when a group needs one,
|
|
12
|
+
schedules the files, and collects results and artifacts. It starts no service
|
|
13
|
+
for a test whose group uses none. Coding agents use Testkit to write tests
|
|
14
|
+
and hunt bugs, and CI uses it to run those tests. It needs no account or
|
|
15
|
+
server.
|
|
13
16
|
|
|
14
17
|
Testkit exists to replace false confidence with inspectable execution evidence.
|
|
15
|
-
It
|
|
16
|
-
|
|
17
|
-
fixtures, and inputs supplied to the run; this is what passed, failed, was
|
|
18
|
+
It makes claims that deserve trust: these tests genuinely executed; these were
|
|
19
|
+
the runtimes, fixtures, and inputs supplied to the run; this is what passed, failed, was
|
|
18
20
|
skipped, or did not run; and this is the evidence behind the result. Testkit
|
|
19
21
|
retains provenance, logs, and verifiable artifacts so those claims can be
|
|
20
22
|
audited rather than accepted on the strength of a green check mark.
|
|
@@ -111,6 +113,23 @@ only when the test fails. It stops when tracked files have uncommitted changes,
|
|
|
111
113
|
when no remote branch contains the commit, or when an open issue already covers
|
|
112
114
|
the test. The token needs permission to create issues and labels.
|
|
113
115
|
|
|
116
|
+
`testkit bug record` saves the `--vitest-config`, `--vitest-project`, and
|
|
117
|
+
`--vitest-runner` it was given in the draft, and asks for `--vitest-project`
|
|
118
|
+
when two Vitest projects run the test's file. `testkit bug file` reruns with
|
|
119
|
+
them, and the issue's reproduce command carries them. With
|
|
120
|
+
`TESTKIT_REQUIRE_COMPLETE_SUITE=1`, it reruns every file the selection lists,
|
|
121
|
+
then reads the recorded failure.
|
|
122
|
+
|
|
123
|
+
`testkit bug file` takes the same [supplied values](#run-against-a-supplied-environment)
|
|
124
|
+
as `testkit run` and reruns the test on them. The issue's reproduce command
|
|
125
|
+
carries those flags, so `testkit bug file` refuses inputs that would put a raw
|
|
126
|
+
secret in it. It takes `--env-from` and `--environment-file`, and takes
|
|
127
|
+
`--base-url` and `--endpoint` only for URLs without credentials, a query, or a
|
|
128
|
+
fragment. It refuses `--database` and `--env`; pass those values through
|
|
129
|
+
`--env-from` or the environment file. The command runs from the repository
|
|
130
|
+
root, so a relative `--environment-file` needs `testkit bug file` run from
|
|
131
|
+
there too.
|
|
132
|
+
|
|
114
133
|
### Name the product's areas
|
|
115
134
|
|
|
116
135
|
Add `areas` to `testkit.config.ts` so that each issue names the part of the
|
|
@@ -174,30 +193,97 @@ claims, how to work, and the bug record format. Give that output to the agent.
|
|
|
174
193
|
summary for the pull request that adds the tests to
|
|
175
194
|
`.testkit/campaigns/<id>/summary.md` and prints it.
|
|
176
195
|
|
|
196
|
+
## Prepare and publish a report bundle
|
|
197
|
+
|
|
198
|
+
Prepare a new directory from your own template, recorded bug IDs, and selected
|
|
199
|
+
run artifacts. All inputs are optional. Repeat `--bug` for related findings and
|
|
200
|
+
`--artifact` for supporting files.
|
|
201
|
+
|
|
202
|
+
```bash
|
|
203
|
+
npx testkit artifacts --run <run-id> --json
|
|
204
|
+
npx testkit report prepare ./draft-report \
|
|
205
|
+
--template ./report-template.md \
|
|
206
|
+
--instructions ./reporting-instructions.md \
|
|
207
|
+
--bug <bug-id> --run <run-id> --artifact <ref.path>
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Preparation copies the template unchanged to `report.md`, or creates an empty
|
|
211
|
+
file. It puts recorded facts in `context.json` and selected, verified run files
|
|
212
|
+
under `evidence/`. Drafting instructions are printed for the author and are not
|
|
213
|
+
copied into the bundle. Write and review the narrative yourself. Templates have
|
|
214
|
+
no special syntax, and preparation never overwrites an existing directory.
|
|
215
|
+
|
|
216
|
+
Publish that directory, or any directory you prepared yourself, to an explicit
|
|
217
|
+
repository, branch, and bundle path:
|
|
218
|
+
|
|
219
|
+
```bash
|
|
220
|
+
npx testkit report publish ./draft-report \
|
|
221
|
+
--repository https://github.com/your-team/private-reports \
|
|
222
|
+
--branch main --path reports/refund-race --json
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
Use a full Gitea repository URL to publish there. `owner/name` uses the host of
|
|
226
|
+
the checkout's origin. Authentication uses the same environment variables and
|
|
227
|
+
Git credential helper as `testkit bug file`; the token needs repository write
|
|
228
|
+
access. Both commands support `--dir` to select the Testkit checkout and
|
|
229
|
+
`--json` for structured output. Local input paths are relative to your current
|
|
230
|
+
directory. Publication needs Git installed and works without Testkit config.
|
|
231
|
+
|
|
232
|
+
Publication replaces the chosen bundle directory, including removing files
|
|
233
|
+
that are no longer in your local bundle. It preserves other repository paths
|
|
234
|
+
and leaves your checkout, index, and branch untouched. The destination branch
|
|
235
|
+
must exist unless the repository is empty. A concurrent branch update rejects
|
|
236
|
+
the push; review the new branch state before trying again. Fetching requests
|
|
237
|
+
a blob-free snapshot and monitors temporary Git storage against a 256 MiB
|
|
238
|
+
budget, with possible brief overshoot between checks. Use a smaller reports
|
|
239
|
+
repository if it exceeds that budget. An unchanged bundle
|
|
240
|
+
returns its existing revision without creating a commit.
|
|
241
|
+
|
|
242
|
+
The result includes the exact `revision` and a provider `url` pinned to it.
|
|
243
|
+
`--message` sets the commit message. Commits use the standard `GIT_AUTHOR_*` and
|
|
244
|
+
`GIT_COMMITTER_*` identity variables when set, otherwise `Testkit
|
|
245
|
+
<testkit@localhost>`. Keep the revision used for a submission. A failed push can
|
|
246
|
+
have an uncertain result after a lost connection, so check the reported commit
|
|
247
|
+
at the destination before retrying.
|
|
248
|
+
|
|
249
|
+
Nested text and binary files keep their bytes and names, without line-ending
|
|
250
|
+
conversion or Git LFS filters. Publication accepts ordinary files up to 16 MiB
|
|
251
|
+
each, 64 MiB total, and 1000 files. Directory depth is limited to 32, with at
|
|
252
|
+
most 128 directories including the repository root and destination parents.
|
|
253
|
+
Symlinks, Git metadata, traversal paths, and
|
|
254
|
+
filenames containing backslashes or line breaks are rejected. Filenames must
|
|
255
|
+
be UTF-8. Empty directories are not stored by Git. Review everything in the
|
|
256
|
+
selected directory before publishing it, including hidden files.
|
|
257
|
+
|
|
258
|
+
Neither command needs Turnstone, a paid feature, or a fixed report format.
|
|
259
|
+
To view the result in Turnstone, connect the destination repository and select
|
|
260
|
+
the same branch and the parent folder, such as `reports`. The author submits
|
|
261
|
+
the report to the chosen platform.
|
|
262
|
+
|
|
177
263
|
## Execution engines
|
|
178
264
|
|
|
179
|
-
Testkit has
|
|
180
|
-
|
|
181
|
-
- `
|
|
182
|
-
and
|
|
183
|
-
- `
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
265
|
+
Testkit has two built-in engines:
|
|
266
|
+
|
|
267
|
+
- `node-vitest` runs plain Vitest test files and purpose-built `data` and
|
|
268
|
+
`contract` files with Node and Vitest.
|
|
269
|
+
- `playwright` runs `ui` files with Playwright and managed Chromium.
|
|
270
|
+
|
|
271
|
+
A plain Vitest test file needs no Testkit suffix, directory, metadata, or
|
|
272
|
+
wrapper. It keeps its native `vitest` imports and its repository paths, has no
|
|
273
|
+
test type, and runs under the engine's general `node-vitest` profile. Testkit
|
|
274
|
+
does not evaluate `testkit.config.ts` in the test's process, and it installs the
|
|
275
|
+
values of the services the test's group uses before the test's imports
|
|
276
|
+
evaluate. See [Plain Vitest test files](#plain-vitest-test-files) for which
|
|
277
|
+
files Testkit runs.
|
|
278
|
+
|
|
279
|
+
Every typed `*.testkit.ts` file must export `defineFile({ engine: "..." })`. The
|
|
280
|
+
filename suffix supplies the public test type; Testkit validates that explicit
|
|
281
|
+
pair and derives a runner profile. It never infers a default engine for a typed
|
|
282
|
+
file or falls back to a different engine.
|
|
283
|
+
|
|
284
|
+
Consumers declare only `@gfdelarue/testkit`, which installs the exact versions
|
|
285
|
+
of the tools its engines run. A product's own copies of Vitest, Playwright, or
|
|
286
|
+
their tools don't change what Testkit runs.
|
|
201
287
|
|
|
202
288
|
Playwright browser installation is explicit:
|
|
203
289
|
|
|
@@ -206,26 +292,16 @@ npx @gfdelarue/testkit setup playwright
|
|
|
206
292
|
npx @gfdelarue/testkit setup playwright --with-deps # CI and fresh hosts
|
|
207
293
|
```
|
|
208
294
|
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
output
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
The five k6 target packages share one exact-version source package that carries
|
|
220
|
-
the AGPL corresponding source archive and licenses. Runtime evidence records
|
|
221
|
-
the package name and version and the k6 version observed by executing the
|
|
222
|
-
installed binary. The
|
|
223
|
-
[testkit-k6 README](https://github.com/gfdelarue/testkit-k6#readme) documents
|
|
224
|
-
the reproducible build, the expected executable digests, and the license
|
|
225
|
-
notes.
|
|
226
|
-
|
|
227
|
-
Managed task environment is engine input rather than ambient host state. k6
|
|
228
|
-
suites read it through native `__ENV`; Node/Vitest data and contract callbacks
|
|
295
|
+
Normal runs never install or download a browser.
|
|
296
|
+
[Installation](docs/engines.md#installation) covers what an install contains,
|
|
297
|
+
the supported platforms, and how to recover a broken one.
|
|
298
|
+
[Typechecking](docs/engines.md#typechecking) covers the supported TypeScript
|
|
299
|
+
versions and `tsconfig.playwright.json`.
|
|
300
|
+
|
|
301
|
+
Testkit bounds the output it captures from each command it runs, and asks the
|
|
302
|
+
command and its descendants to stop on cancellation or timeout.
|
|
303
|
+
|
|
304
|
+
Managed task environment is engine input. Node/Vitest data and contract callbacks
|
|
229
305
|
read immutable, enumerable `context.env`; and Playwright suites can read the
|
|
230
306
|
same contract from `readPlaywrightContext()?.env`. Managed Node processes also
|
|
231
307
|
expose each value as an immutable, non-enumerable `process.env.KEY` property so
|
|
@@ -237,21 +313,21 @@ reserves `APPDATA`, `ComSpec`, `HOME`, `LANG`, `LC_ALL`, `LOCALAPPDATA`, `PATH`,
|
|
|
237
313
|
`PATHEXT`, `SystemRoot`, `TEMP`, `TMP`, `TMPDIR`, `USERPROFILE`,
|
|
238
314
|
`XDG_CACHE_HOME`, `XDG_CONFIG_HOME`, and `XDG_DATA_HOME`. In addition:
|
|
239
315
|
|
|
240
|
-
-
|
|
241
|
-
|
|
242
|
-
|
|
316
|
+
- Node/Vitest reserves `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR`,
|
|
317
|
+
`UPDATE_SNAPSHOT`, `VITEST_POOL_ID`, `VITEST_WORKER_ID`, and the `TESTKIT_`
|
|
318
|
+
keys it sets itself, listed in
|
|
319
|
+
[Environment](docs/engines.md#environment);
|
|
320
|
+
other `TESTKIT_` and `VITEST_` keys, such as
|
|
321
|
+
`TESTKIT_REQUIRE_COMPLETE_SUITE`, pass through;
|
|
243
322
|
- Playwright reserves the `TESTKIT_`, `PLAYWRIGHT_`, `PWTEST_`, and `PW_TEST_`
|
|
244
323
|
prefixes plus `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, and `NO_COLOR`.
|
|
245
324
|
|
|
246
325
|
Case-variant duplicates and invalid names or NUL-containing values fail instead
|
|
247
|
-
of being normalized.
|
|
248
|
-
environment-block character units. Node/Vitest accepts at most 512 entries and
|
|
326
|
+
of being normalized. Node/Vitest accepts at most 512 entries and
|
|
249
327
|
4 MiB of UTF-8 key/value data. Playwright accepts at most 512 entries and 2 MiB
|
|
250
|
-
of UTF-8 key/value data.
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
[Engine Architecture](docs/engine-architecture.md#managed-environment) for the
|
|
254
|
-
complete transport and aggregate-input limits.
|
|
328
|
+
of UTF-8 key/value data. [Environment](docs/engines.md#environment) lists the
|
|
329
|
+
complete limits, and [Named endpoints and databases](#named-endpoints-and-databases)
|
|
330
|
+
covers the environment API.
|
|
255
331
|
|
|
256
332
|
## Usage
|
|
257
333
|
|
|
@@ -268,10 +344,9 @@ npx @gfdelarue/testkit discover --json > .testkit/discovery.json
|
|
|
268
344
|
npx @gfdelarue/testkit discover --output .testkit/discovery.json
|
|
269
345
|
|
|
270
346
|
# Filter by type
|
|
271
|
-
npx @gfdelarue/testkit --type
|
|
272
|
-
npx @gfdelarue/testkit --type
|
|
273
|
-
npx @gfdelarue/testkit --type
|
|
274
|
-
npx @gfdelarue/testkit --type int,e2e,dal
|
|
347
|
+
npx @gfdelarue/testkit --type contract
|
|
348
|
+
npx @gfdelarue/testkit --type data
|
|
349
|
+
npx @gfdelarue/testkit --type contract,data
|
|
275
350
|
npx @gfdelarue/testkit --type ui
|
|
276
351
|
|
|
277
352
|
# Parallel file execution
|
|
@@ -280,16 +355,17 @@ npx @gfdelarue/testkit --task-concurrency 8
|
|
|
280
355
|
# One file-level wall clock budget for every suite file
|
|
281
356
|
npx @gfdelarue/testkit --file-timeout-seconds 60
|
|
282
357
|
|
|
283
|
-
# Specific
|
|
284
|
-
npx @gfdelarue/testkit --
|
|
285
|
-
npx @gfdelarue/testkit --
|
|
286
|
-
npx @gfdelarue/testkit --type
|
|
358
|
+
# Specific group / suite
|
|
359
|
+
npx @gfdelarue/testkit --group frontend --type ui -s navigation
|
|
360
|
+
npx @gfdelarue/testkit --group api --type contract -s health
|
|
361
|
+
npx @gfdelarue/testkit --type contract,data -s contract:queries
|
|
287
362
|
|
|
288
363
|
# Exact file
|
|
289
|
-
npx @gfdelarue/testkit --type
|
|
364
|
+
npx @gfdelarue/testkit --type contract --file __testkit__/health.contract.testkit.ts
|
|
365
|
+
npx @gfdelarue/testkit run --file src/sum.test.ts
|
|
290
366
|
|
|
291
367
|
# Temporarily ignore repo-declared skip rules
|
|
292
|
-
npx @gfdelarue/testkit --ignore-skip-rules --file __testkit__/billing.
|
|
368
|
+
npx @gfdelarue/testkit --ignore-skip-rules --file __testkit__/billing.contract.testkit.ts
|
|
293
369
|
|
|
294
370
|
# Lifecycle
|
|
295
371
|
npx @gfdelarue/testkit status
|
|
@@ -304,6 +380,16 @@ npx @gfdelarue/testkit db schema refresh --service api
|
|
|
304
380
|
npx @gfdelarue/testkit db schema verify --service api
|
|
305
381
|
```
|
|
306
382
|
|
|
383
|
+
A `--type` that names test types, such as `--type contract,ui`, and typed suite
|
|
384
|
+
selectors such as `contract:health` select typed files only. A run without them, or
|
|
385
|
+
with `--type all`, also runs plain Vitest test files. To run every typed file
|
|
386
|
+
and no plain one, name all three types: `--type ui,contract,data`.
|
|
387
|
+
|
|
388
|
+
`run`, `discover`, `artifacts`, and `workspace` select test groups with
|
|
389
|
+
`--group`. On those commands `--service` is a deprecated alias for `--group`
|
|
390
|
+
and prints a notice on stderr. `status`, `destroy`, `cleanup`, and `db schema`
|
|
391
|
+
take `--service` to name a service.
|
|
392
|
+
|
|
307
393
|
Database cleanup is scoped to the selected product. `cleanup --resources --global`
|
|
308
394
|
is retired; use `--dir <product> cleanup --resources` for each product. After
|
|
309
395
|
deleting a product directory, remove its orphaned containers and volumes
|
|
@@ -319,15 +405,125 @@ Batch `run` output stays intentionally short: one line per completed file,
|
|
|
319
405
|
concise failure blocks, and a final summary. Service logs, captured runtime
|
|
320
406
|
output, and emitted artifacts are persisted under `.testkit/results/`.
|
|
321
407
|
|
|
408
|
+
Text output is coloured in a terminal and plain otherwise. As in Node,
|
|
409
|
+
`NO_COLOR` turns colour off and `FORCE_COLOR` overrides it: `0` or `false`
|
|
410
|
+
turns colour off and any other value turns it on.
|
|
411
|
+
|
|
322
412
|
Testkit seals an immutable local run bundle. `testkit run` prints
|
|
323
413
|
`Run ID: <id>` after sealing in text output and returns `runId` in `--json`
|
|
324
414
|
output, for passed and failed runs.
|
|
325
415
|
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
416
|
+
Testkit starts the files that took longest in earlier runs first, so slow
|
|
417
|
+
files start while workers are free. The run document records the order under
|
|
418
|
+
`planning`.
|
|
419
|
+
|
|
420
|
+
### Run against a supplied environment
|
|
421
|
+
|
|
422
|
+
`testkit run` can take the values its tests need instead of provisioning them.
|
|
423
|
+
Every selected group then runs on those values alone. Testkit starts none of
|
|
424
|
+
the group's services, creates no database, and runs no `.setup.*` module beside
|
|
425
|
+
a test. It also ignores the env values and env files those services declare, so
|
|
426
|
+
supply every value the tests read. Groups, selection, skip rules, and other
|
|
427
|
+
policy work as before.
|
|
428
|
+
|
|
429
|
+
```bash
|
|
430
|
+
npx @gfdelarue/testkit run --group checks \
|
|
431
|
+
--base-url http://127.0.0.1:3000 \
|
|
432
|
+
--endpoint api=http://127.0.0.1:3001 \
|
|
433
|
+
--database store=postgres://app:password@127.0.0.1:5432/store \
|
|
434
|
+
--env FEATURE_FLAG=on \
|
|
435
|
+
--env-from API_TOKEN=CI_API_TOKEN
|
|
436
|
+
|
|
437
|
+
npx @gfdelarue/testkit run --environment-file supplied.json
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
- `--base-url URL` supplies the default endpoint, which tests receive as
|
|
441
|
+
`BASE_URL`.
|
|
442
|
+
- `--endpoint NAME=URL` supplies a named endpoint, usually named after the
|
|
443
|
+
service it stands in for.
|
|
444
|
+
- `--database NAME=URL` supplies a named PostgreSQL database.
|
|
445
|
+
- `--env NAME=VALUE` supplies an environment value.
|
|
446
|
+
- `--env-from NAME=HOST_KEY` supplies the value of host variable `HOST_KEY`
|
|
447
|
+
under `NAME`.
|
|
448
|
+
- `--environment-file PATH` reads values, prepared setup data, and handle
|
|
449
|
+
descriptors from a file.
|
|
450
|
+
|
|
451
|
+
Every flag except `--base-url` and `--environment-file` can be repeated. Tests
|
|
452
|
+
read named values with `endpoint(name)` and `database(name)` from
|
|
453
|
+
`@gfdelarue/testkit/environment`. An `--env`, `--env-from`, or file `env` entry
|
|
454
|
+
named `BASE_URL` or `DATABASE_URL` sets the default endpoint or database, and an
|
|
455
|
+
empty database URL means no database. Without an explicit default, a group's
|
|
456
|
+
`BASE_URL` and `DATABASE_URL` are those of the first service in its `uses` that
|
|
457
|
+
has a supplied endpoint or database. Either way the default is also available
|
|
458
|
+
by the group's name, as in a configured run, so it may not differ from a named
|
|
459
|
+
endpoint or database with that name.
|
|
460
|
+
|
|
461
|
+
Testkit reads host variables only through `--env-from`. A `BASE_URL` or
|
|
462
|
+
`DATABASE_URL` already set in the shell does not switch a run to supplied
|
|
463
|
+
values. Two inputs may define the same value only if they agree. Any other
|
|
464
|
+
repeat, such as `--base-url` and `--env BASE_URL` with different URLs, or a
|
|
465
|
+
missing `--env-from` host variable, fails the run before it starts. Before a
|
|
466
|
+
test starts, each supplied endpoint must give a valid HTTP response. A 4xx or
|
|
467
|
+
5xx response counts as valid.
|
|
468
|
+
|
|
469
|
+
Each supplied database must also accept a connection before a test starts.
|
|
470
|
+
Testkit connects, signs in with the URL's credentials, runs `SELECT 1`, and
|
|
471
|
+
disconnects. It handles trust, password, md5, and SCRAM-SHA-256 authentication.
|
|
472
|
+
TLS follows the URL's `sslmode` as in libpq. The default, `prefer`, tries TLS
|
|
473
|
+
first and then a plain connection, and `allow` tries them the other way round.
|
|
474
|
+
`verify-ca` and `verify-full` need `sslrootcert`, set to a CA file or to
|
|
475
|
+
`system`, and `sslrootcert=system` alone means `verify-full`. `host=/path`
|
|
476
|
+
connects through the Unix socket in that directory and ignores `sslmode`. The
|
|
477
|
+
check reads only the URL. It ignores `PG*` variables, `~/.pgpass`, and
|
|
478
|
+
`~/.postgresql/root.crt`, and it presents no client certificate. A database
|
|
479
|
+
that is unreachable, fails TLS, rejects the credentials, does not exist, or is
|
|
480
|
+
not PostgreSQL fails the run before any test starts. The message names the
|
|
481
|
+
database, never its URL.
|
|
482
|
+
|
|
483
|
+
The environment file is JSON:
|
|
484
|
+
|
|
485
|
+
```json
|
|
486
|
+
{
|
|
487
|
+
"version": 1,
|
|
488
|
+
"baseUrl": "http://127.0.0.1:3000",
|
|
489
|
+
"databaseUrl": "postgres://app:password@127.0.0.1:5432/app",
|
|
490
|
+
"endpoints": { "api": "http://127.0.0.1:3001" },
|
|
491
|
+
"databases": { "store": "postgres://app:password@127.0.0.1:5432/store" },
|
|
492
|
+
"env": { "FEATURE_FLAG": "on" },
|
|
493
|
+
"ui": {},
|
|
494
|
+
"setupData": { "actor": "Grace" },
|
|
495
|
+
"handles": {
|
|
496
|
+
"webhook": {
|
|
497
|
+
"kind": "http-capture",
|
|
498
|
+
"url": "http://127.0.0.1:41000/webhook",
|
|
499
|
+
"requestsUrl": "http://127.0.0.1:41000/__testkit__/<capability>/requests",
|
|
500
|
+
"resetUrl": "http://127.0.0.1:41000/__testkit__/<capability>/requests"
|
|
501
|
+
}
|
|
502
|
+
}
|
|
503
|
+
}
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
Only `version` is required, and this Testkit reads version 1. `baseUrl`,
|
|
507
|
+
`databaseUrl`, and `ui` may be `null`, as an environment rendered by Testkit
|
|
508
|
+
writes them, and `null` there counts as absent. Tests receive
|
|
509
|
+
`setupData` as the data a setup module would have returned: `{}` when the file
|
|
510
|
+
leaves it out, and `null` when the file sets it to `null`. `handles` entries must each name a `kind`, and tests reach them with
|
|
511
|
+
[`handle(name)`](#handles). An `http-capture` entry holds the `url`,
|
|
512
|
+
`requestsUrl`, and `resetUrl` that a capture sidecar reports, and gives the
|
|
513
|
+
test the same handle a setup module's capture would. Other kinds are handed on,
|
|
514
|
+
and `handle()` throws `EnvironmentHandleError` with `"unsupported"` for them. Any other field, such as
|
|
515
|
+
service or database configuration, is rejected. Errors name the file and the
|
|
516
|
+
field and do not repeat the file's contents.
|
|
517
|
+
|
|
518
|
+
A run records the values its tests received in this shape. Each acquisition in
|
|
519
|
+
`run.json`'s `environment` section has them under `values`, with secrets and
|
|
520
|
+
capture capabilities redacted. Its `setupData` is the supplied setup data or
|
|
521
|
+
what the file's setup module returned, and `{}` without either. Sealing redacts
|
|
522
|
+
setup data under a key that names a secret or that `redact` lists, and records
|
|
523
|
+
the rest as returned, so list any other sensitive key in `redact`. With
|
|
524
|
+
`"version": 1` added, an
|
|
525
|
+
acquisition's `values` are an environment file that gives a second run the
|
|
526
|
+
same values, unless redaction changed one of them.
|
|
331
527
|
|
|
332
528
|
## Tooling Adapters
|
|
333
529
|
|
|
@@ -363,23 +559,17 @@ export default defineConfig({
|
|
|
363
559
|
// playwright.config.ts
|
|
364
560
|
import { defineTestkitConfig, devices } from "@gfdelarue/testkit/playwright";
|
|
365
561
|
|
|
366
|
-
export default defineTestkitConfig(
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
},
|
|
371
|
-
{
|
|
372
|
-
dotenvFiles: [".env.local"],
|
|
373
|
-
},
|
|
374
|
-
);
|
|
562
|
+
export default defineTestkitConfig({
|
|
563
|
+
testDir: "./__testkit__",
|
|
564
|
+
projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
|
|
565
|
+
});
|
|
375
566
|
```
|
|
376
567
|
|
|
377
568
|
Managed UI suites use `@gfdelarue/testkit/playwright` for Testkit's Playwright
|
|
378
569
|
helpers and re-exported authoring API. They may also import exact public exports
|
|
379
570
|
from the managed `@playwright/test`, `playwright`, and `playwright-core`
|
|
380
|
-
packages.
|
|
381
|
-
|
|
382
|
-
different compile-time API.
|
|
571
|
+
packages. Those imports resolve to Testkit's pinned versions at run time and
|
|
572
|
+
when typechecking, whatever Playwright version the product installs.
|
|
383
573
|
|
|
384
574
|
For scripts and app-runtime code, `@gfdelarue/testkit/env` provides shared helpers
|
|
385
575
|
for managed runtime detection, dotenv loading, and local-database safety:
|
|
@@ -428,7 +618,6 @@ export default defineConfig({
|
|
|
428
618
|
api: app.node({
|
|
429
619
|
cwd: ".",
|
|
430
620
|
entry: "src/index.ts",
|
|
431
|
-
port: 3004,
|
|
432
621
|
envFiles: [".env.testkit"],
|
|
433
622
|
database: database.postgres({
|
|
434
623
|
runtimeConnections: 20,
|
|
@@ -448,7 +637,6 @@ export default defineConfig({
|
|
|
448
637
|
frontend: app.next({
|
|
449
638
|
cwd: "frontend",
|
|
450
639
|
mode: "start",
|
|
451
|
-
port: 3000,
|
|
452
640
|
dependsOn: ["api"],
|
|
453
641
|
envFiles: ["frontend/.env.testkit"],
|
|
454
642
|
env: {
|
|
@@ -466,28 +654,48 @@ export default defineConfig({
|
|
|
466
654
|
});
|
|
467
655
|
```
|
|
468
656
|
|
|
469
|
-
Every
|
|
470
|
-
|
|
657
|
+
Every service Testkit starts listens on a port the OS assigns for that run.
|
|
658
|
+
Testkit passes it to the service as `PORT` and as `{port}` in `start`, `env`,
|
|
659
|
+
`baseUrl`, and `readyUrl`, so two runs of the same repository on one host, such
|
|
660
|
+
as two CI jobs on a runner or two agent worktrees, never compete for a port. A
|
|
661
|
+
service that exits before it becomes ready gets a fresh port and up to three
|
|
662
|
+
attempts in total. Each run's `run.json` lists every service's actual URL, port,
|
|
663
|
+
and attempts under `environment.services`. A config that sets `local.port`, the
|
|
664
|
+
`port` option of `app.node` or `app.next`, or a literal port in `local.baseUrl`
|
|
665
|
+
or `local.readyUrl` fails to load; delete it and use `{port}`.
|
|
666
|
+
|
|
667
|
+
Every typed test file declares its execution engine and any file-local
|
|
668
|
+
scheduling metadata next to the test. A plain Vitest test file declares
|
|
669
|
+
neither:
|
|
471
670
|
|
|
472
671
|
```ts
|
|
473
672
|
import { defineFile } from "@gfdelarue/testkit";
|
|
474
673
|
|
|
475
674
|
export const testkit = defineFile({
|
|
476
|
-
engine: "
|
|
675
|
+
engine: "node-vitest",
|
|
477
676
|
skip: "Billing is currently unavailable locally",
|
|
478
|
-
locks: ["global-worker-loop"],
|
|
479
677
|
});
|
|
480
678
|
```
|
|
481
679
|
|
|
482
|
-
|
|
483
|
-
|
|
680
|
+
A file can also name `locks`.
|
|
681
|
+
[Give each test its own data](#give-each-test-its-own-data) says when to use
|
|
682
|
+
one.
|
|
683
|
+
|
|
684
|
+
In a typed file, the engine value is a required literal. The supported public
|
|
685
|
+
authoring paths
|
|
686
|
+
are `@gfdelarue/testkit/node-vitest/data`,
|
|
484
687
|
`@gfdelarue/testkit/node-vitest/contract`, and `@gfdelarue/testkit/playwright`.
|
|
485
688
|
Node/Vitest and Playwright suites may also import exact public APIs directly
|
|
486
689
|
from Testkit's managed `vitest`, `@playwright/test`, `playwright`, and
|
|
487
690
|
`playwright-core` packages.
|
|
488
|
-
There are no legacy aliases or implicit-runner compatibility paths.
|
|
489
|
-
|
|
490
|
-
`
|
|
691
|
+
There are no legacy aliases or implicit-runner compatibility paths. A typed
|
|
692
|
+
Node/Vitest file resolves `paths` from its nearest `tsconfig.json`, through
|
|
693
|
+
Vite's `resolve.tsconfigPaths`.
|
|
694
|
+
|
|
695
|
+
`testkit.config.ts` is optional. Without it, Testkit runs every test file with
|
|
696
|
+
zero setup: it starts no server, provisions no database, and loads no `.env`
|
|
697
|
+
file. A test that needs a server or a database gets one from a service that
|
|
698
|
+
`testkit.config.ts` declares. The config is also the primary escape hatch
|
|
491
699
|
for:
|
|
492
700
|
|
|
493
701
|
- worker count and per-file runtime budget
|
|
@@ -506,10 +714,9 @@ for:
|
|
|
506
714
|
- repo-declared suite/file skip policies with explicit reasons
|
|
507
715
|
|
|
508
716
|
`runtime.prepare` is the generic build-once hook for shared runtimes. It runs
|
|
509
|
-
once per runtime generation before local services start,
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
stable build-and-start flows.
|
|
717
|
+
once per runtime generation before local services start, and runs again when
|
|
718
|
+
its declared inputs change. This is the right way to move expensive browser
|
|
719
|
+
targets from `next dev` / watch mode to stable build-and-start flows.
|
|
513
720
|
|
|
514
721
|
Setup steps (`runtime.prepare`, database `template`, and other Testkit-run
|
|
515
722
|
Node commands) inherit `NODE_OPTIONS`. `TESTKIT_SETUP_NODE_OPTIONS` appends
|
|
@@ -521,18 +728,12 @@ Testkit-owned Docker resources are labelled at creation time so they can be
|
|
|
521
728
|
found again even if product-local `.testkit` state is deleted. `testkit destroy`
|
|
522
729
|
removes labelled resources for the current product after service state is
|
|
523
730
|
destroyed. `testkit cleanup --resources` cleans stale resources for the current
|
|
524
|
-
product. Both commands
|
|
525
|
-
|
|
526
|
-
lives in a Testkit-labelled named Docker
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
generation immediately before Docker's name-based removal. Docker has no
|
|
531
|
-
immutable volume ID or conditional-delete API, so direct Docker-daemon
|
|
532
|
-
principals are part of the host trust boundary and must not replace that
|
|
533
|
-
unguessable name during the final removal call. Testkit serializes database
|
|
534
|
-
mutations for the same product; independent products can proceed concurrently.
|
|
535
|
-
Use `cleanup --resources --dry-run` to preview the selected product's cleanup.
|
|
731
|
+
product. Both commands remove only the current product's resources and leave
|
|
732
|
+
other products' alone, including their idle caches. Host-local Postgres data
|
|
733
|
+
lives in a Testkit-labelled named Docker volume, so cleanup can find and remove
|
|
734
|
+
it. Database changes for one product run one at a time; other products don't
|
|
735
|
+
wait for them. Use `cleanup --resources --dry-run` to preview the selected
|
|
736
|
+
product's cleanup.
|
|
536
737
|
|
|
537
738
|
`database.template` is the database-side equivalent for reusable template DB
|
|
538
739
|
state. When `database.sourceSchema` is configured, Testkit treats the configured
|
|
@@ -544,34 +745,16 @@ schema still matches the source dump. If local replay differs, Testkit refreshes
|
|
|
544
745
|
from the source once for the current cache key and retries. If it still differs,
|
|
545
746
|
the run fails with schema diagnostics under `.testkit/results/schema`.
|
|
546
747
|
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
the
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
`.gitignore` files are respected; host-global excludes, `core.excludesFile`, and
|
|
558
|
-
`.git/info/exclude` do not alter Testkit's cache identity.
|
|
559
|
-
|
|
560
|
-
Fingerprint Git commands disable replacement objects and optional locks and do
|
|
561
|
-
not read system or user Git configuration. Git path records are decoded from
|
|
562
|
-
their byte-preserving quoted form and must be valid UTF-8. Uninitialized
|
|
563
|
-
gitlinks and special filesystem nodes fail explicitly instead of being assigned
|
|
564
|
-
an incomplete or ambiguous identity. Testkit reads the complete HEAD, index,
|
|
565
|
-
untracked, submodule, and physical snapshot twice and fails if the observations
|
|
566
|
-
differ.
|
|
567
|
-
|
|
568
|
-
Physical reads open regular files by descriptor and verify every pathname
|
|
569
|
-
component before and after use; on POSIX, ancestor directory descriptors remain
|
|
570
|
-
open during the read. Node does not expose a portable `openat`/Windows reparse-
|
|
571
|
-
handle traversal API, so this detects ordinary replacement races but is not a
|
|
572
|
-
security boundary against a same-user adversary able to perform a perfectly
|
|
573
|
-
timed replace-and-restore ABA attack. Run Testkit against source trees that are
|
|
574
|
-
not concurrently controlled by an untrusted local process.
|
|
748
|
+
Testkit keys the source schema cache on the repository's state. A clean
|
|
749
|
+
worktree shares the entry for its commit. A dirty worktree or a directory
|
|
750
|
+
outside Git gets its own entry, keyed on the content of its files.
|
|
751
|
+
|
|
752
|
+
The key changes when a file's bytes, type, or executable mode changes, including
|
|
753
|
+
changes Git's own status can hide, such as sparse checkout or line-ending
|
|
754
|
+
normalization, and when a submodule moves. Untracked files are included unless
|
|
755
|
+
the repository's `.gitignore` files exclude them; global or per-clone Git
|
|
756
|
+
excludes don't count. A path Testkit can't read, or a tree that changes while
|
|
757
|
+
Testkit reads it, fails the run instead of producing a key.
|
|
575
758
|
|
|
576
759
|
Testkit-owned outputs are always excluded from fingerprints: `.testkit/`,
|
|
577
760
|
and `.next-testkit/`. `testkit.config.ts` is normal repo
|
|
@@ -596,22 +779,16 @@ Template setup executes in three explicit phases:
|
|
|
596
779
|
|
|
597
780
|
Schema drift is checked after each successful `migrate` and `seed` step, and
|
|
598
781
|
`verify` only runs once local replay matches the source schema. Source refreshes
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
PgBouncer/pooler URLs fail closed; configure a direct source URL for those
|
|
610
|
-
providers. Concurrent refreshes for the same service and cache key are
|
|
611
|
-
serialized with a cache-local lock so multiple Testkit processes do not
|
|
612
|
-
stampede the source database. Testkit also maintains
|
|
613
|
-
`.testkit/db/<service>/source-schemas/index.json` and prunes old inactive cache
|
|
614
|
-
entries automatically.
|
|
782
|
+
dump the schema only, in Docker, so seed and reference data never enter the
|
|
783
|
+
baseline and no host `pg_dump` is needed. Keep schema-changing setup in its own
|
|
784
|
+
step where possible; a single command that changes schema and then fails before
|
|
785
|
+
exiting cannot be refreshed at the midpoint.
|
|
786
|
+
|
|
787
|
+
A source schema refresh uses one connection. A Neon pooled source URL is
|
|
788
|
+
rewritten to the matching direct Neon endpoint. Other PgBouncer or pooler URLs
|
|
789
|
+
fail; configure a direct source URL for those providers. Refreshes of the same
|
|
790
|
+
cache entry run one at a time, even from separate Testkit processes, and
|
|
791
|
+
Testkit prunes old inactive cache entries automatically.
|
|
615
792
|
|
|
616
793
|
For most repos, prefer declarative step objects directly inside
|
|
617
794
|
`database.postgres({ template: ... })` and `runtime.prepare.steps`.
|
|
@@ -621,10 +798,9 @@ The supported shapes are:
|
|
|
621
798
|
- `{ kind: "sql-file", path: "..." }`
|
|
622
799
|
- `{ kind: "module", target: "file.ts#exportName" }`
|
|
623
800
|
|
|
624
|
-
Local files and ordinary external packages remain valid module targets.
|
|
625
|
-
|
|
626
|
-
|
|
627
|
-
private Testkit subpath fails before preparation or database setup starts.
|
|
801
|
+
Local files and ordinary external packages remain valid module targets. An
|
|
802
|
+
unsupported or private `@gfdelarue/testkit` target fails when the config loads,
|
|
803
|
+
before preparation or database setup starts.
|
|
628
804
|
|
|
629
805
|
`runtime.toolchain` is the first-class way to make those prepare/start commands
|
|
630
806
|
run under the correct Node toolchain instead of whatever `node`/`npm` happened
|
|
@@ -642,16 +818,12 @@ to launch `testkit`. Node toolchains support:
|
|
|
642
818
|
- `package.json#packageManager`
|
|
643
819
|
- `package.json#engines.npm`
|
|
644
820
|
|
|
645
|
-
|
|
646
|
-
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
rejected. `runtime.toolchain.cwd` is always a normalized product-relative path
|
|
652
|
-
and cannot contain a parent-directory segment. The trusted official Linux Node
|
|
653
|
-
archives are glibc builds; `install: "download"` fails before fetch or cache
|
|
654
|
-
mutation on musl instead of pretending those binaries are compatible.
|
|
821
|
+
`install: "download"` supports the Node versions and platforms Testkit lists,
|
|
822
|
+
and fails before downloading for any other. Downloads are cached under
|
|
823
|
+
`.testkit/_toolchains/node`, and a modified cache entry fails instead of
|
|
824
|
+
running. `runtime.toolchain.cwd` is a product-relative path and cannot contain
|
|
825
|
+
a parent-directory segment. The official Linux Node archives are glibc builds,
|
|
826
|
+
so `install: "download"` fails on musl before it downloads anything.
|
|
655
827
|
|
|
656
828
|
Example:
|
|
657
829
|
|
|
@@ -666,7 +838,6 @@ toolchains: {
|
|
|
666
838
|
services: {
|
|
667
839
|
frontend: app.next({
|
|
668
840
|
cwd: "frontend",
|
|
669
|
-
port: 3000,
|
|
670
841
|
runtime: {
|
|
671
842
|
toolchain: "frontendNode",
|
|
672
843
|
},
|
|
@@ -681,6 +852,77 @@ artifact/log/setup files. Payloads are SHA-256 addressed under
|
|
|
681
852
|
removing their logical per-run paths. `.testkit/results/latest.json` remains a
|
|
682
853
|
convenience projection; it is not the historical authority.
|
|
683
854
|
|
|
855
|
+
`run.json` is schema 20. `groups` holds each group's suites and files.
|
|
856
|
+
`environment` is the run's environment record. For each acquisition, it holds:
|
|
857
|
+
|
|
858
|
+
- the source that prepared it: `configured`, `static`, `supplied`, or `none`;
|
|
859
|
+
- the tasks it served and the services it started;
|
|
860
|
+
- each file's setup;
|
|
861
|
+
- how it was released;
|
|
862
|
+
- the values its tests received.
|
|
863
|
+
|
|
864
|
+
It also holds each service's start attempts, the service and setup logs, the
|
|
865
|
+
run's capacity and database backend, and the values tests asked for that the
|
|
866
|
+
environment lacked. Runs that Testkit 0.4.x stored as schema 19 stay readable
|
|
867
|
+
through `testkit artifacts`, `ci report`, known-failure checks, and the
|
|
868
|
+
workspace API.
|
|
869
|
+
|
|
870
|
+
Sealing replaces secrets with `<redacted>` in the run record and in every text
|
|
871
|
+
file of the bundle. A key names a secret when it holds `PASSWORD`, `TOKEN`,
|
|
872
|
+
`SECRET`, `PRIVATE_KEY`, `AUTH`, `COOKIE`, `SESSION`, or `API_KEY` in any letter
|
|
873
|
+
case. Sealing redacts:
|
|
874
|
+
|
|
875
|
+
- A `KEY=value` assignment under such a key, an `Authorization`,
|
|
876
|
+
`Proxy-Authorization`, `Cookie`, or `Set-Cookie` header value, a JSON string
|
|
877
|
+
value under such a key, even when the key is spelled with escapes, a
|
|
878
|
+
single-quoted `'password': '…'` value, and a URL's password, wherever they
|
|
879
|
+
appear. A quoted value ends at its closing quote and keeps its quotes, and a
|
|
880
|
+
value that starts with an auth scheme such as `Bearer` ends with its line.
|
|
881
|
+
A secret in one JSON value leaves the record's other values as they are.
|
|
882
|
+
- The value of each such key, and of each key that `redact` names, in the
|
|
883
|
+
environment Testkit runs in. When the value is a URL, its password counts
|
|
884
|
+
too.
|
|
885
|
+
- The values the run's environment resolved: each value under such a key or a
|
|
886
|
+
key that `redact` names, and the password of each URL. A PostgreSQL URL's
|
|
887
|
+
password may also be a `password` query parameter, as libpq reads it. This
|
|
888
|
+
covers config `env`, env files, `local.env`, dependency services, base,
|
|
889
|
+
readiness, UI, and database URLs, `--env`, `--env-from`, and an
|
|
890
|
+
`--environment-file`. A value under such a key that is a URL counts only for
|
|
891
|
+
its password.
|
|
892
|
+
- The unguessable segment in each [HTTP capture's](#handles) operation URLs,
|
|
893
|
+
from a setup module's capture or an `--environment-file` handle.
|
|
894
|
+
|
|
895
|
+
Name a secret's key in `redact` when it holds none of the words above:
|
|
896
|
+
|
|
897
|
+
```ts
|
|
898
|
+
export default defineConfig({
|
|
899
|
+
redact: ["STRIPE_SK", "PGPASS"],
|
|
900
|
+
});
|
|
901
|
+
```
|
|
902
|
+
|
|
903
|
+
Testkit matches the names in any letter case, in a configured run and a
|
|
904
|
+
supplied run alike. `redact` adds keys whose values sealing redacts; the forms
|
|
905
|
+
in the first item still use the words above.
|
|
906
|
+
|
|
907
|
+
A value shorter than eight characters is not redacted on its own. Sealing also
|
|
908
|
+
redacts a value's JSON-escaped forms, each line of a value that spans lines,
|
|
909
|
+
and keys in the free-form data and metadata that tests attach. When output cuts a secret short and marks
|
|
910
|
+
the cut with `…` or `...`, sealing redacts the part that shows. `testkit run
|
|
911
|
+
--json` and the run's progress output apply the same redaction.
|
|
912
|
+
|
|
913
|
+
Some secrets stay visible:
|
|
914
|
+
|
|
915
|
+
- What a setup module prints goes to the terminal as written.
|
|
916
|
+
- Sealing does not look inside compressed or binary files, such as a Playwright
|
|
917
|
+
`trace.zip`. It refuses to seal a binary file that holds one of the values
|
|
918
|
+
above in plain bytes.
|
|
919
|
+
- When a run's process is killed before it seals the run, `testkit stop`,
|
|
920
|
+
`testkit cleanup`, or the next run seals it from another process. That
|
|
921
|
+
process does not have the values the run's environment resolved or the
|
|
922
|
+
`redact` list, so it redacts only the first item and its own environment's
|
|
923
|
+
values under the words above. The bundle's `archive.redaction.status` is then
|
|
924
|
+
`source-targets-unavailable` instead of `applied`.
|
|
925
|
+
|
|
684
926
|
Use the artifact archive config to opt into pruning or atomic publication to
|
|
685
927
|
an external directory. Local retention is indefinite by default:
|
|
686
928
|
|
|
@@ -724,253 +966,320 @@ Programmatic consumers can import `listRunBundles`, `loadRunBundleArtifact`,
|
|
|
724
966
|
|
|
725
967
|
## Authoring
|
|
726
968
|
|
|
727
|
-
###
|
|
969
|
+
### Named endpoints and databases
|
|
728
970
|
|
|
729
|
-
|
|
730
|
-
|
|
731
|
-
|
|
971
|
+
`@gfdelarue/testkit/environment` gives a test the URL and database of each
|
|
972
|
+
service its group uses, by the service's name:
|
|
973
|
+
|
|
974
|
+
```ts
|
|
975
|
+
import { database, endpoint } from "@gfdelarue/testkit/environment";
|
|
976
|
+
|
|
977
|
+
const response = await fetch(new URL("/health", endpoint("api")));
|
|
978
|
+
const reportsDatabase = database("reports");
|
|
979
|
+
```
|
|
980
|
+
|
|
981
|
+
Both functions return strings. A service's database is the one Testkit
|
|
982
|
+
provisions for it, or the `DATABASE_URL` its `env` declares. The module works
|
|
983
|
+
in Vitest and Playwright files and in the modules they import.
|
|
984
|
+
Testkit installs the values before any of that code loads, so module-level code
|
|
985
|
+
and native hooks can read them, such as a Playwright config and its
|
|
986
|
+
`globalSetup` or a Vitest `beforeAll`.
|
|
987
|
+
|
|
988
|
+
A name with no value throws `EnvironmentValueMissingError`, with `kind`,
|
|
989
|
+
`valueName`, and the `available` names. `valueName` is `null` when the missing
|
|
990
|
+
value is the run's default URL. Outside a Testkit run every call throws it.
|
|
991
|
+
|
|
992
|
+
A test that fails on this error, or a file whose imports throw it, ends as an
|
|
993
|
+
infrastructure failure in Vitest and Playwright. The run fails, and
|
|
994
|
+
`testkit bug file` refuses to file it. A test that catches the error and
|
|
995
|
+
carries on is unaffected. When the run gives a test no base URL, a Playwright
|
|
996
|
+
`page.goto()` to a relative URL ends the same way.
|
|
997
|
+
|
|
998
|
+
`BASE_URL` and `DATABASE_URL` hold the values of the first service
|
|
999
|
+
in `uses` that has each one. Playwright gives `BASE_URL` and `baseURL` in
|
|
1000
|
+
canonical form, so an origin ends with `/`. `endpoint()` returns the URL
|
|
1001
|
+
unchanged in every engine.
|
|
1002
|
+
|
|
1003
|
+
The group's default endpoint and database are also available by the group's
|
|
1004
|
+
name, whether the run starts its services or takes supplied values. For a group
|
|
1005
|
+
`checks` that uses `api`, `endpoint("checks")` and `endpoint("api")` return the
|
|
1006
|
+
same URL. A group whose default differs from a used service's value with the
|
|
1007
|
+
group's name, such as a group `api` with `uses: ["web", "api"]`, gets no
|
|
1008
|
+
values: its files end as an infrastructure failure that names the group, and
|
|
1009
|
+
other groups still run.
|
|
1010
|
+
|
|
1011
|
+
### Handles
|
|
1012
|
+
|
|
1013
|
+
`handle(name)` from `@gfdelarue/testkit/environment` gives a test an object for
|
|
1014
|
+
something its environment prepared. Today that is an HTTP capture, which a
|
|
1015
|
+
[setup module](#node-setup-beside-a-test) starts with `sidecars.httpCapture()`.
|
|
1016
|
+
The handle is named after the capture's `id`:
|
|
732
1017
|
|
|
733
1018
|
```js
|
|
734
|
-
// tests/webhook.
|
|
1019
|
+
// tests/webhook.contract.testkit.setup.mjs
|
|
735
1020
|
export async function setup({ sidecars }) {
|
|
736
|
-
|
|
737
|
-
return {
|
|
738
|
-
data: { webhookUrl: capture.url, requestsUrl: capture.requestsUrl },
|
|
739
|
-
};
|
|
1021
|
+
await sidecars.httpCapture({ id: "webhook", path: "/webhook" });
|
|
740
1022
|
}
|
|
741
1023
|
```
|
|
742
1024
|
|
|
743
|
-
Setup runs in a Node worker thread with its own JavaScript state. It can use
|
|
744
|
-
Node APIs, prepare external resources, and start HTTP capture listeners.
|
|
745
|
-
The context contains `deadlineMs`, `signal`, `env`, `leaseDir`, `productDir`,
|
|
746
|
-
`serviceName`, `testFile`, and `sidecars`. The worker shares the host process's
|
|
747
|
-
permissions. It is not a sandbox for untrusted code.
|
|
748
|
-
|
|
749
|
-
Return plain JSON, or `{ data, cleanup }` with an optional cleanup function.
|
|
750
|
-
Testkit passes the data to the engine as `setupData` and records it with the
|
|
751
|
-
setup module path and sidecar handles. Bare `null` or `undefined`, and a
|
|
752
|
-
cleanup-only result, give empty data. `{ data: null }` preserves `null`.
|
|
753
|
-
Data must encode to at most 1 MiB of JSON. Invalid data reports its JSON path,
|
|
754
|
-
such as `$.items[1]`, without invoking property getters.
|
|
755
|
-
|
|
756
|
-
Setup shares the task's absolute deadline. On failure or cancellation, Testkit
|
|
757
|
-
runs registered cleanup and closes its sidecars. Once setup succeeds, those
|
|
758
|
-
resources remain available until task cleanup. Cleanup callbacks run in reverse
|
|
759
|
-
registration order. The runner uses the task timeout for cleanup, capped at
|
|
760
|
-
60 seconds. A hung worker is terminated so it cannot keep its listeners open.
|
|
761
|
-
|
|
762
|
-
Thrown setup errors, deadline expiry, and invalid data fail the task with the
|
|
763
|
-
setup module path. Consumer cleanup errors fail the task; Testkit cleanup or
|
|
764
|
-
worker failures count as infrastructure failures. Cleanup remains idempotent.
|
|
765
|
-
HTTP capture retains at most 100 requests by default, configurable up to 1,000.
|
|
766
|
-
Each body defaults to 1 MiB, configurable up to 8 MiB, and retained bodies total
|
|
767
|
-
at most 8 MiB per listener. A setup can create up to 16 listeners, each with at
|
|
768
|
-
most 256 connections.
|
|
769
|
-
|
|
770
|
-
Discovery tries sibling `.setup.mjs`, sibling `.setup.js`, then the matching
|
|
771
|
-
`.mjs` and `.js` files in `__setup__`, in that order. For the example above,
|
|
772
|
-
the fallback is `tests/__setup__/webhook.int.testkit.mjs`.
|
|
773
|
-
|
|
774
|
-
### HTTP suites
|
|
775
|
-
|
|
776
1025
|
```ts
|
|
777
|
-
|
|
778
|
-
import {
|
|
779
|
-
|
|
780
|
-
export const testkit = defineFile({ engine: "k6" });
|
|
781
|
-
|
|
782
|
-
const suite = defineHttpSuite(({ rawReq }) => {
|
|
783
|
-
const response = rawReq.get("/health");
|
|
784
|
-
expect.status(response, 200, "health returns 200");
|
|
785
|
-
});
|
|
1026
|
+
// tests/webhook.contract.testkit.ts
|
|
1027
|
+
import { handle } from "@gfdelarue/testkit/environment";
|
|
786
1028
|
|
|
787
|
-
|
|
788
|
-
|
|
789
|
-
|
|
1029
|
+
const webhook = handle("webhook");
|
|
1030
|
+
await placeOrder({ callbackUrl: webhook.url });
|
|
1031
|
+
const [delivery] = await webhook.requests();
|
|
1032
|
+
await webhook.reset();
|
|
790
1033
|
```
|
|
791
1034
|
|
|
792
|
-
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
1035
|
+
A capture handle has `kind` (`"http-capture"`), `url` (where the product sends
|
|
1036
|
+
its requests), `requests()`, which returns the captured requests oldest first,
|
|
1037
|
+
and `reset()`, which forgets them. Both methods return a Promise in
|
|
1038
|
+
Vitest and Playwright. Each call fails after 10 seconds if the capture hasn't
|
|
1039
|
+
answered in full.
|
|
1040
|
+
|
|
1041
|
+
A handle is a reference, not a connection. It can list or reset its own capture
|
|
1042
|
+
and nothing else. The capture's operation URLs hold an unguessable segment, so
|
|
1043
|
+
holding the handle is the authority. Testkit keeps that segment out of the
|
|
1044
|
+
run's results, artifacts, and events, and the capture refuses any other
|
|
1045
|
+
operation. When Testkit releases the file's environment, the capture closes and
|
|
1046
|
+
the handle expires. Calling a handle never starts a sidecar or a service.
|
|
1047
|
+
|
|
1048
|
+
- A name with no handle throws `EnvironmentValueMissingError` with `kind`
|
|
1049
|
+
`"handle"`, and the test ends as an infrastructure failure, as for a missing
|
|
1050
|
+
endpoint.
|
|
1051
|
+
- Every capture created without an `id` gets the id `http-capture`. If two do,
|
|
1052
|
+
`handle("http-capture")` throws `EnvironmentHandleAmbiguousError` with the
|
|
1053
|
+
`count`. Give each capture a distinct `id`.
|
|
1054
|
+
- A call that the capture refuses, such as one from a stale or forged
|
|
1055
|
+
reference, throws `EnvironmentHandleError` with `reason` `"refused"`. A call
|
|
1056
|
+
whose capture no longer answers throws it with `"unavailable"`, and a
|
|
1057
|
+
descriptor this Testkit cannot use throws it with `"invalid"` or
|
|
1058
|
+
`"unsupported"`. These end the test as ordinary failures.
|
|
1059
|
+
|
|
1060
|
+
### Give each test its own data
|
|
1061
|
+
|
|
1062
|
+
Testkit runs test files at the same time. Files on one runtime share its
|
|
1063
|
+
services and its database. Write each test so that it passes beside any other
|
|
1064
|
+
test, in any order.
|
|
1065
|
+
|
|
1066
|
+
- Create the rows the test needs, and filter every query and update to them.
|
|
1067
|
+
Don't depend on rows that another test creates, and don't count, update, or
|
|
1068
|
+
delete across a whole table. Every test sees the rows from `template.seed`,
|
|
1069
|
+
so read them and leave them unchanged.
|
|
1070
|
+
- When the test ends, leave its rows in place. Testkit resets the databases it
|
|
1071
|
+
provisions. By default, every run starts from a fresh clone of the template
|
|
1072
|
+
database. See [`database.reset`](testkit-config-schema.md#database).
|
|
1073
|
+
- Clean up only what Testkit doesn't own, such as an account in a third-party
|
|
1074
|
+
sandbox. Register the cleanup as soon as the test creates the account. Give
|
|
1075
|
+
the account a unique name, so that a leftover from a cleanup that never ran
|
|
1076
|
+
can't collide with the next run.
|
|
799
1077
|
|
|
800
1078
|
```ts
|
|
1079
|
+
import { randomUUID } from "node:crypto";
|
|
801
1080
|
import { defineFile } from "@gfdelarue/testkit";
|
|
802
|
-
import {
|
|
803
|
-
|
|
804
|
-
import { organizationHeaders, sessionAuth } from "../helpers/session.js";
|
|
1081
|
+
import { contractScenario } from "@gfdelarue/testkit/node-vitest/contract";
|
|
1082
|
+
import { expect, onTestFinished } from "vitest";
|
|
805
1083
|
|
|
806
|
-
|
|
807
|
-
|
|
808
|
-
const suite = defineHttpSuite(
|
|
809
|
-
{
|
|
810
|
-
auth: sessionAuth(["reviewer", "outsider"]),
|
|
811
|
-
rawHeaders: organizationHeaders,
|
|
812
|
-
},
|
|
813
|
-
({ actor, actors, req }) => {
|
|
814
|
-
req.get("/api/auth/session");
|
|
815
|
-
req("GET", "/api/auth/session");
|
|
816
|
-
actor?.req.get("/api/auth/session");
|
|
817
|
-
req.as("outsider").get("/api/auth/session");
|
|
818
|
-
actors.get("reviewer").rawReq.get("/api/auth/session");
|
|
819
|
-
},
|
|
820
|
-
);
|
|
1084
|
+
import { sandbox } from "../payments-sandbox.js";
|
|
821
1085
|
|
|
822
|
-
export const
|
|
823
|
-
export const setup = suite.setup;
|
|
824
|
-
export default suite.exec;
|
|
825
|
-
```
|
|
1086
|
+
export const testkit = defineFile({ engine: "node-vitest" });
|
|
826
1087
|
|
|
827
|
-
|
|
828
|
-
|
|
829
|
-
|
|
1088
|
+
contractScenario("a refund reaches the sandbox customer", async ({ http }) => {
|
|
1089
|
+
const customer = await sandbox.createCustomer({ email: `refund-${randomUUID()}@example.test` });
|
|
1090
|
+
onTestFinished(() => sandbox.deleteCustomer(customer.id));
|
|
830
1091
|
|
|
831
|
-
|
|
832
|
-
|
|
833
|
-
|
|
834
|
-
const actors = Object.fromEntries(
|
|
835
|
-
actorNames.map((actorName, actorIndex) => [
|
|
836
|
-
actorName,
|
|
837
|
-
{ actorIndex, actorName, session: login(http, env.BASE, actorName) },
|
|
838
|
-
]),
|
|
839
|
-
);
|
|
840
|
-
return { actors, primaryActor: actorNames[0] };
|
|
841
|
-
},
|
|
842
|
-
headers(bundle, { actor }) {
|
|
843
|
-
const session = bundle?.actors[actor ?? bundle.primaryActor ?? ""]?.session;
|
|
844
|
-
return session ? { Authorization: `Bearer ${session.jwt}` } : {};
|
|
845
|
-
},
|
|
846
|
-
};
|
|
847
|
-
}
|
|
1092
|
+
const response = await http().post("/api/refunds", { customerId: customer.id, amountCents: 500 });
|
|
1093
|
+
expect(response.status).toBe(201);
|
|
1094
|
+
});
|
|
848
1095
|
```
|
|
849
1096
|
|
|
850
|
-
|
|
851
|
-
|
|
852
|
-
|
|
853
|
-
directly. See the [config reference](testkit-config-schema.md#profileshttp).
|
|
1097
|
+
A [Node setup module](#node-setup-beside-a-test) returns its cleanup with its
|
|
1098
|
+
data. Testkit runs only the cleanup that `setup` returns. If a step in `setup`
|
|
1099
|
+
can throw after `setup` creates something, undo the creation before rethrowing.
|
|
854
1100
|
|
|
855
|
-
|
|
1101
|
+
A test that has to count or change a whole table needs a database that other
|
|
1102
|
+
tests don't write to. Give the test a group of its own, and have the group use
|
|
1103
|
+
a service with its own database. Below, the `ledger` group owns the test files
|
|
1104
|
+
under `src/ledger`, and its tests get the database of `ledger-db`, built from
|
|
1105
|
+
the same schema:
|
|
856
1106
|
|
|
857
1107
|
```ts
|
|
858
|
-
|
|
859
|
-
|
|
860
|
-
|
|
861
|
-
|
|
862
|
-
|
|
863
|
-
|
|
864
|
-
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
`);
|
|
871
|
-
return { widgetId };
|
|
872
|
-
});
|
|
873
|
-
},
|
|
874
|
-
}));
|
|
875
|
-
|
|
876
|
-
const suite = defineDalSuite({ fixtures }, ({ db, fixtureScope, fixtures }) => {
|
|
877
|
-
const widget = fixtures.widget();
|
|
878
|
-
db.query(`select id from widgets where id = '${widget.widgetId}'`);
|
|
879
|
-
fixtureScope.records();
|
|
880
|
-
});
|
|
881
|
-
|
|
882
|
-
export const options = suite.options;
|
|
883
|
-
export const setup = suite.setup;
|
|
884
|
-
export default suite.exec;
|
|
1108
|
+
services: {
|
|
1109
|
+
api: app.node({
|
|
1110
|
+
// ...
|
|
1111
|
+
}),
|
|
1112
|
+
"ledger-db": database.fixture({
|
|
1113
|
+
template: { migrate: [{ kind: "sql-file", path: "db/schema.sql" }] },
|
|
1114
|
+
}),
|
|
1115
|
+
},
|
|
1116
|
+
groups: {
|
|
1117
|
+
api: { roots: ["src"], uses: ["api"] },
|
|
1118
|
+
ledger: { roots: ["src/ledger"], uses: ["ledger-db"] },
|
|
1119
|
+
},
|
|
885
1120
|
```
|
|
886
1121
|
|
|
887
|
-
|
|
888
|
-
|
|
889
|
-
|
|
890
|
-
|
|
891
|
-
|
|
892
|
-
reservedConnections })` for the backing Postgres capacity.
|
|
1122
|
+
`locks` is an escape hatch for a resource that can't be duplicated, such as a
|
|
1123
|
+
payment provider's sandbox account with one webhook URL that some tests change.
|
|
1124
|
+
Files that name the same lock never run at the same time within one run.
|
|
1125
|
+
Testkit doesn't share locks between runs, such as two CI jobs. Two tests that
|
|
1126
|
+
need a lock because they share rows need their own rows instead.
|
|
893
1127
|
|
|
894
|
-
|
|
895
|
-
|
|
1128
|
+
```ts
|
|
1129
|
+
export const testkit = defineFile({ engine: "node-vitest", locks: ["payments-sandbox-webhook"] });
|
|
1130
|
+
```
|
|
896
1131
|
|
|
897
|
-
|
|
898
|
-
- `fixtureScope.slug(label)`
|
|
899
|
-
- `fixtureScope.email(label)`
|
|
900
|
-
- `fixtureScope.string(label, options)`
|
|
901
|
-
- `fixtureScope.token(label)`
|
|
902
|
-
- `fixtureScope.seed(kind, key, signature, create)`
|
|
903
|
-
- `fixtureScope.records()`
|
|
1132
|
+
### Node setup beside a test
|
|
904
1133
|
|
|
905
|
-
`testkit`
|
|
1134
|
+
A test such as `tests/webhook.contract.testkit.ts` can have a sibling
|
|
1135
|
+
`tests/webhook.contract.testkit.setup.mjs`. Testkit runs its named `setup` export,
|
|
1136
|
+
or its default export, before starting the test engine:
|
|
906
1137
|
|
|
907
|
-
|
|
908
|
-
|
|
909
|
-
|
|
910
|
-
|
|
911
|
-
|
|
1138
|
+
```js
|
|
1139
|
+
// tests/webhook.contract.testkit.setup.mjs
|
|
1140
|
+
export async function setup({ sidecars }) {
|
|
1141
|
+
const capture = await sidecars.httpCapture({ path: "/webhook" });
|
|
1142
|
+
return {
|
|
1143
|
+
data: { webhookUrl: capture.url, requestsUrl: capture.requestsUrl },
|
|
1144
|
+
};
|
|
1145
|
+
}
|
|
1146
|
+
```
|
|
912
1147
|
|
|
913
|
-
|
|
1148
|
+
Setup is part of preparing the file's environment. Testkit runs it once, after
|
|
1149
|
+
it acquires the services the file's group uses and before the test engine
|
|
1150
|
+
starts. Testkit never batches a Playwright file that has setup with other
|
|
1151
|
+
Playwright files. Files without setup still batch.
|
|
914
1152
|
|
|
915
|
-
|
|
916
|
-
|
|
917
|
-
|
|
918
|
-
|
|
919
|
-
|
|
920
|
-
|
|
921
|
-
|
|
922
|
-
|
|
923
|
-
|
|
924
|
-
get: ["/api/v1/sessions"],
|
|
925
|
-
});
|
|
926
|
-
|
|
927
|
-
actor?.rawHeaders();
|
|
928
|
-
|
|
929
|
-
const upload = req.multipart.post("/api/v1/uploads", {
|
|
930
|
-
fields: { name: "fixture" },
|
|
931
|
-
files: [
|
|
932
|
-
{
|
|
933
|
-
field: "file",
|
|
934
|
-
data: "hello",
|
|
935
|
-
filename: "fixture.txt",
|
|
936
|
-
contentType: "text/plain",
|
|
937
|
-
},
|
|
938
|
-
],
|
|
939
|
-
});
|
|
1153
|
+
Setup runs with its own JavaScript state. It can use Node APIs, prepare
|
|
1154
|
+
external resources, and start HTTP capture listeners. Tests
|
|
1155
|
+
reach a capture by its `id` through [`handle()`](#handles). A capture's
|
|
1156
|
+
`requestsUrl` and `resetUrl` hold an unguessable segment that Testkit keeps out
|
|
1157
|
+
of the run's records, and its capture `path` may not start with `/__testkit__/`.
|
|
1158
|
+
The context contains `deadlineMs`, `signal`, `env`, `leaseDir`, `productDir`,
|
|
1159
|
+
`serviceName`, `testFile`, and `sidecars`. `serviceName` holds the name of the
|
|
1160
|
+
test's group. Setup has the same permissions as the Testkit process. It is not
|
|
1161
|
+
a sandbox for untrusted code.
|
|
940
1162
|
|
|
941
|
-
|
|
942
|
-
|
|
943
|
-
});
|
|
944
|
-
```
|
|
1163
|
+
A run on [supplied values](#run-against-a-supplied-environment) does not run
|
|
1164
|
+
setup modules. Its tests receive the environment file's `setupData` instead.
|
|
945
1165
|
|
|
946
|
-
|
|
1166
|
+
Return plain JSON, or `{ data, cleanup }` with an optional cleanup function.
|
|
1167
|
+
Testkit passes the data to the engine as `setupData` and records it with the
|
|
1168
|
+
setup module path and sidecar handles. Bare `null` or `undefined`, and a
|
|
1169
|
+
cleanup-only result, give empty data. `{ data: null }` preserves `null`.
|
|
1170
|
+
Data must encode to at most 1 MiB of JSON. Invalid data reports its JSON path,
|
|
1171
|
+
such as `$.items[1]`.
|
|
947
1172
|
|
|
948
|
-
|
|
949
|
-
|
|
950
|
-
|
|
951
|
-
|
|
952
|
-
|
|
1173
|
+
Setup shares the task's absolute deadline. On failure or cancellation, Testkit
|
|
1174
|
+
runs registered cleanup and closes its sidecars. Once setup succeeds, those
|
|
1175
|
+
resources remain available until the test and its native hooks finish and
|
|
1176
|
+
Testkit releases the file's environment. Cleanup callbacks run in reverse
|
|
1177
|
+
registration order. The runner uses the task timeout for cleanup, capped at
|
|
1178
|
+
60 seconds. Testkit stops a setup that hangs past that limit, which closes its
|
|
1179
|
+
listeners.
|
|
953
1180
|
|
|
954
|
-
|
|
955
|
-
|
|
956
|
-
|
|
1181
|
+
Thrown setup errors, deadline expiry, and invalid data fail the task with the
|
|
1182
|
+
setup module path. Consumer cleanup errors fail the task; a failure in
|
|
1183
|
+
Testkit's own cleanup, or a setup that stops unexpectedly, counts as an
|
|
1184
|
+
infrastructure failure. Cleanup remains idempotent.
|
|
1185
|
+
HTTP capture retains at most 100 requests by default, configurable up to 1,000.
|
|
1186
|
+
Each body defaults to 1 MiB, configurable up to 8 MiB, and retained bodies total
|
|
1187
|
+
at most 8 MiB per listener. A setup can create up to 16 listeners, each with at
|
|
1188
|
+
most 256 connections.
|
|
957
1189
|
|
|
958
|
-
|
|
959
|
-
|
|
1190
|
+
Discovery tries sibling `.setup.mjs`, sibling `.setup.js`, then the matching
|
|
1191
|
+
`.mjs` and `.js` files in `__setup__`, in that order. For the example above,
|
|
1192
|
+
the fallback is `tests/__setup__/webhook.contract.testkit.mjs`.
|
|
960
1193
|
|
|
961
|
-
|
|
962
|
-
import { parse, waitFor } from "@gfdelarue/testkit/k6";
|
|
1194
|
+
### HTTP and persistence checks
|
|
963
1195
|
|
|
964
|
-
|
|
965
|
-
|
|
966
|
-
|
|
967
|
-
|
|
968
|
-
|
|
969
|
-
```
|
|
1196
|
+
Use plain Vitest files for ordinary HTTP, service, and database checks. Read
|
|
1197
|
+
configured URLs with `endpoint(name)` and `database(name)` from
|
|
1198
|
+
`@gfdelarue/testkit/environment`, then use `fetch` or your application's
|
|
1199
|
+
database client. Use `contractScenario` when a typed contract file needs the
|
|
1200
|
+
managed contract context. Use `dataScenario` for structured data-flow evidence.
|
|
970
1201
|
|
|
971
1202
|
## Discovery
|
|
972
1203
|
|
|
973
|
-
|
|
1204
|
+
Testkit discovers plain Vitest test files and typed suites.
|
|
1205
|
+
|
|
1206
|
+
### Plain Vitest test files
|
|
1207
|
+
|
|
1208
|
+
Testkit runs plain Vitest test files with the repository's own Vitest and its
|
|
1209
|
+
config, as `vitest run` does from the product directory. It loads the root
|
|
1210
|
+
`vitest.config.*` or `vite.config.*` the way Vitest does and lists the files
|
|
1211
|
+
that config's projects select, without running global setup. Nested configs and
|
|
1212
|
+
package scripts have no effect. Without a config, Vitest's defaults select files
|
|
1213
|
+
that match `**/*.{test,spec}.?(c|m)[jt]s?(x)`, outside `node_modules` and
|
|
1214
|
+
`.git`.
|
|
1215
|
+
|
|
1216
|
+
`discover`, `run`, `typecheck`, `bug record`, and `bug file` take three flags
|
|
1217
|
+
that change the selection:
|
|
1218
|
+
|
|
1219
|
+
- `--vitest-config <path>` uses that config, relative to the product
|
|
1220
|
+
directory. A missing file, or one outside the product directory, fails before
|
|
1221
|
+
any test runs.
|
|
1222
|
+
- `--vitest-project <name>` runs only that project. An unknown name fails and
|
|
1223
|
+
lists the projects.
|
|
1224
|
+
- `--vitest-runner managed` runs plain files with Testkit's own Vitest and its
|
|
1225
|
+
default rules, without the repository's config.
|
|
1226
|
+
|
|
1227
|
+
Testkit supports Vitest 4.x. Another major, a broken installation, a root
|
|
1228
|
+
Vitest config that fails to load, or a Vitest that `package.json` declares but
|
|
1229
|
+
that is not installed fails before any test runs, naming the version and path.
|
|
1230
|
+
`--vitest-runner managed` avoids the repository's Vitest. A run that selects
|
|
1231
|
+
only typed files, with `--type` or with typed `--file` paths, `bug record` or
|
|
1232
|
+
`bug file` for a typed test, and the commands that select no tests (`status`,
|
|
1233
|
+
`run status`, `doctor`, `destroy`, `cleanup`, and `db schema`) don't load the
|
|
1234
|
+
repository's Vitest, so they still work. A product that neither installs nor
|
|
1235
|
+
declares Vitest has no plain test files.
|
|
1236
|
+
|
|
1237
|
+
All plain files in a run go to one Vitest invocation, so root global setup,
|
|
1238
|
+
teardown, and the repository's reporters run as often as they do under
|
|
1239
|
+
`vitest run`, and Testkit's reporter runs beside them. The invocation's deadline
|
|
1240
|
+
is the sum of its files' timeouts; Vitest's own test and hook timeouts apply
|
|
1241
|
+
inside it. A file that two projects select runs in both, its result combines
|
|
1242
|
+
both runs, and its evidence names each run's project. Files from different
|
|
1243
|
+
groups run together when their groups give them the same services, values, and
|
|
1244
|
+
locks; none may have a setup module or isolation other than `shared`.
|
|
1245
|
+
Otherwise the run stops before any file runs and names the groups and what
|
|
1246
|
+
differs. Services, locks, isolation, or a setup module stop it before Testkit
|
|
1247
|
+
provisions anything; values stop it before Vitest starts. Select one group with
|
|
1248
|
+
`--group`.
|
|
1249
|
+
|
|
1250
|
+
Plain files see the host environment Testkit started with: `HOME`, `PATH`,
|
|
1251
|
+
temporary and XDG directories, locale, `CI`, `FORCE_COLOR`, `NO_COLOR`, and any
|
|
1252
|
+
other variable reach config loading, setup files, and the tests as they do
|
|
1253
|
+
under `vitest run`. Unlike `vitest run`, ambient `BASE_URL` and `DATABASE_URL`
|
|
1254
|
+
never reach them; those come only from the group's environment, from
|
|
1255
|
+
provisioned services or supplied values. Testkit loads no env files for plain
|
|
1256
|
+
files, and Vitest's own env handling and `test.env` work as usual. Testkit puts
|
|
1257
|
+
its own `--import` in front of the shell's `NODE_OPTIONS`. Preloads in it see the
|
|
1258
|
+
shell's environment and the group's values; as under `vitest run`, Vitest sets
|
|
1259
|
+
`TEST`, `VITEST`, and `NODE_ENV` after them. `--vitest-runner managed` leaves the
|
|
1260
|
+
shell's `NODE_OPTIONS` out.
|
|
1261
|
+
|
|
1262
|
+
Two settings narrow the files Vitest selects: `discovery.exclude` in `testkit.config.ts`, and
|
|
1263
|
+
a declared group's `exclude`. Testkit's built-in exclusions for typed files,
|
|
1264
|
+
such as `dist`, `build`, and `coverage`, `discovery.roots`, and a legacy
|
|
1265
|
+
service's exclusions do not apply to plain files.
|
|
1266
|
+
|
|
1267
|
+
A plain file belongs to the group whose roots contain it, or to the implicit
|
|
1268
|
+
`root` group, which uses no service. Legacy service ownership never claims a
|
|
1269
|
+
plain file. Testkit starts no service for a plain file in `root`.
|
|
1270
|
+
|
|
1271
|
+
The plain files in one directory form a suite named after that directory,
|
|
1272
|
+
relative to the group's root. In `root`, `--suite src` selects
|
|
1273
|
+
`src/sum.test.ts` and `src/sum.spec.ts`. A file directly in a group's root names
|
|
1274
|
+
its own suite, so `sum.test.ts` in a group rooted at `src` is in suite `sum`.
|
|
1275
|
+
When two suites in a group would share a name, Testkit lengthens the later
|
|
1276
|
+
ones with parent directories, such as `api-src`, and numbers them if that is
|
|
1277
|
+
not enough. `testkit typecheck` checks plain TypeScript test files and skips
|
|
1278
|
+
plain JavaScript ones, as `tsc` does without `allowJs`.
|
|
1279
|
+
|
|
1280
|
+
### Typed suites
|
|
1281
|
+
|
|
1282
|
+
`testkit` discovers typed suites from `__testkit__/` directories.
|
|
974
1283
|
|
|
975
1284
|
Place each `__testkit__/` directory as a peer of the thing being tested. The
|
|
976
1285
|
target is the smallest owned directory or package surface that honestly contains
|
|
@@ -999,25 +1308,22 @@ production owner when a narrower owner exists.
|
|
|
999
1308
|
|
|
1000
1309
|
Concrete layouts:
|
|
1001
1310
|
|
|
1002
|
-
- `src/api/routes/__testkit__/auth-me.
|
|
1003
|
-
- `src/db/__testkit__/sessions-count-type.
|
|
1311
|
+
- `src/api/routes/__testkit__/auth-me.contract.testkit.ts`
|
|
1312
|
+
- `src/db/__testkit__/sessions-count-type.contract.testkit.ts`
|
|
1004
1313
|
- `frontend/src/components/layout/__testkit__/navigation.ui.testkit.ts`
|
|
1005
|
-
- `src/internal/handler/repos/__testkit__/crud.
|
|
1314
|
+
- `src/internal/handler/repos/__testkit__/crud.contract.testkit.ts`
|
|
1006
1315
|
|
|
1007
1316
|
`testkit` uses these suffixes automatically:
|
|
1008
1317
|
|
|
1009
|
-
- `*.int.testkit.ts`
|
|
1010
|
-
- `*.e2e.testkit.ts`
|
|
1011
1318
|
- `*.contract.testkit.ts`
|
|
1012
1319
|
- `*.data.testkit.ts`
|
|
1013
|
-
- `*.dal.testkit.ts`
|
|
1014
|
-
- `*.load.testkit.ts`
|
|
1015
1320
|
- `*.ui.testkit.ts`
|
|
1016
1321
|
|
|
1017
|
-
The suffix never chooses an engine. Each discovered file must also
|
|
1018
|
-
root-package `defineFile(...)` declaration with a supported literal
|
|
1322
|
+
The suffix never chooses an engine. Each discovered typed file must also
|
|
1323
|
+
contain one root-package `defineFile(...)` declaration with a supported literal
|
|
1324
|
+
engine.
|
|
1019
1325
|
Unsupported pairs fail discovery; for example, a `*.ui.testkit.ts` file cannot
|
|
1020
|
-
declare `engine: "
|
|
1326
|
+
declare `engine: "node-vitest"`.
|
|
1021
1327
|
|
|
1022
1328
|
See [docs/test-types.md](docs/test-types.md) for the canonical type model.
|
|
1023
1329
|
`*.contract.testkit.ts` files are Node-backed service/runtime contract suites.
|
|
@@ -1036,30 +1342,61 @@ structured data-flow evidence through the data context. Supporting
|
|
|
1036
1342
|
`count(...)`/`event(...)` calls are not enough by themselves; model actors,
|
|
1037
1343
|
systems, records, transitions, external calls, payloads, budgets, cache/LLM/RPC
|
|
1038
1344
|
usage, or reconciliation evidence. Data suites run through the
|
|
1039
|
-
`node-vitest-data` profile. Use
|
|
1040
|
-
|
|
1041
|
-
|
|
1345
|
+
`node-vitest-data` profile. Use a plain Vitest test file for plain Node
|
|
1346
|
+
assertions and HTTP or database integration, `contract` for typed service or
|
|
1347
|
+
protocol contracts, and `ui` when a browser is part of the assertion.
|
|
1042
1348
|
Data suites use Vitest directly for assertions, hooks, and mocking.
|
|
1043
1349
|
`dataScenario(...)` owns case registration: `testkit lint` reports native
|
|
1044
1350
|
registration imports in data suites, and the data runner rejects any case that
|
|
1045
1351
|
was not registered through the DSL.
|
|
1046
1352
|
|
|
1047
|
-
|
|
1353
|
+
Test groups own test files by directory. Declare them under `groups` in
|
|
1354
|
+
`testkit.config.ts`. Each group lists its repo-relative `roots` and the services
|
|
1355
|
+
its tests `use`:
|
|
1048
1356
|
|
|
1049
|
-
|
|
1050
|
-
|
|
1357
|
+
```ts
|
|
1358
|
+
export default defineConfig({
|
|
1359
|
+
services: {
|
|
1360
|
+
api: app.node({ cwd: "services/api", start: "node dist/server.js" }),
|
|
1361
|
+
web: app.next({ cwd: "apps/web", dependsOn: ["api"] }),
|
|
1362
|
+
},
|
|
1363
|
+
groups: {
|
|
1364
|
+
api: { roots: ["services/api"], uses: ["api"] },
|
|
1365
|
+
web: { roots: ["apps/web"], uses: ["web", "api"] },
|
|
1366
|
+
},
|
|
1367
|
+
});
|
|
1368
|
+
```
|
|
1369
|
+
|
|
1370
|
+
- A file belongs to the group with the deepest root that contains it. When two
|
|
1371
|
+
groups have roots at the same depth, discovery reports `ambiguous_test`.
|
|
1372
|
+
- Files that no group's roots claim belong to the implicit group `root`, which
|
|
1373
|
+
uses no service. With no `testkit.config.ts`, every file belongs to `root`.
|
|
1374
|
+
- A run starts the services that the selected files' groups use, with their
|
|
1375
|
+
`dependsOn`, and nothing else. A test gets `BASE_URL`, `DATABASE_URL`, and
|
|
1376
|
+
`env` values only from the services its group uses. When two of them set the
|
|
1377
|
+
same name, the first in `uses` wins. Each used service's URL and database
|
|
1378
|
+
are also available by name; see
|
|
1379
|
+
[Named endpoints and databases](#named-endpoints-and-databases).
|
|
1380
|
+
- `skip`, `requirements`, and `runtimeIsolation` are group fields. See the
|
|
1381
|
+
[config reference](testkit-config-schema.md#groups).
|
|
1382
|
+
|
|
1383
|
+
A config with `services` and no `groups` key uses legacy ownership. Each
|
|
1384
|
+
service owns the typed test files under its `local.cwd` and
|
|
1385
|
+
`discovery.roots`, as a group of the same name that uses that service. Testkit
|
|
1386
|
+
prints a notice on stderr when legacy ownership gives a file to a service.
|
|
1387
|
+
Legacy ownership will be removed; declare `groups` instead.
|
|
1051
1388
|
|
|
1052
1389
|
Suite names are inferred from the colocated path:
|
|
1053
1390
|
|
|
1054
|
-
- `auth/__testkit__/*.
|
|
1055
|
-
- `routes/__testkit__/auth-me.
|
|
1391
|
+
- `auth/__testkit__/*.contract.testkit.ts` => `auth`
|
|
1392
|
+
- `routes/__testkit__/auth-me.contract.testkit.ts` => `routes`
|
|
1056
1393
|
|
|
1057
1394
|
Discovery is also a first-class CLI/API surface:
|
|
1058
1395
|
|
|
1059
1396
|
- `testkit discover`
|
|
1060
|
-
- human-first compact output with
|
|
1397
|
+
- human-first compact output with group -> type -> suite -> file hierarchy
|
|
1061
1398
|
- `testkit discover --output-mode verbose`
|
|
1062
|
-
- explicit paths, IDs, locks,
|
|
1399
|
+
- group roots and uses, explicit paths, IDs, locks, and skip reasons
|
|
1063
1400
|
- `testkit discover --json`
|
|
1064
1401
|
- machine-readable output with stable enums, canonical paths, and summary data
|
|
1065
1402
|
- `testkit discover --output .testkit/discovery.json`
|
|
@@ -1088,7 +1425,7 @@ identifier plus canonical metadata such as:
|
|
|
1088
1425
|
- `path`
|
|
1089
1426
|
- `service`
|
|
1090
1427
|
- `suiteName`
|
|
1091
|
-
- `testType
|
|
1428
|
+
- `testType`, absent for a plain Vitest test file
|
|
1092
1429
|
- `engineId`
|
|
1093
1430
|
- `profileId`
|
|
1094
1431
|
- `contract`
|
|
@@ -1099,8 +1436,10 @@ identifier plus canonical metadata such as:
|
|
|
1099
1436
|
- `displayName`
|
|
1100
1437
|
|
|
1101
1438
|
Discovery output has no cross-run counts, last outcomes, durations, or trends.
|
|
1102
|
-
|
|
1103
|
-
|
|
1439
|
+
Testkit 0.2.1 removed `testkit.status.json`, `.testkit/history/`, and
|
|
1440
|
+
discovery's `history`, `files[].history`, and `summary.latestResults` fields.
|
|
1441
|
+
To check one run, read its bundle under `.testkit/results/runs/<run-id>/`.
|
|
1442
|
+
Testkit ignores an old `.testkit/history/` directory and leaves it in place.
|
|
1104
1443
|
|
|
1105
1444
|
## Local Databases
|
|
1106
1445
|
|
|
@@ -1113,13 +1452,10 @@ client image instead of using host-installed binaries, which keeps local runs
|
|
|
1113
1452
|
consistent across Linux, macOS, and native Windows. Override the client image
|
|
1114
1453
|
with `TESTKIT_POSTGRES_CLIENT_IMAGE` when needed.
|
|
1115
1454
|
|
|
1116
|
-
|
|
1117
|
-
|
|
1118
|
-
|
|
1119
|
-
|
|
1120
|
-
the same under Docker Desktop, rootless Docker, and Docker-in-Docker. Clients
|
|
1121
|
-
for external databases, such as `sourceSchema` URLs, still connect from the
|
|
1122
|
-
platform's host address.
|
|
1455
|
+
Template steps, schema snapshots, and `database.steps` helpers work the same
|
|
1456
|
+
under Docker Desktop, rootless Docker, and Docker-in-Docker. Clients for
|
|
1457
|
+
external databases, such as `sourceSchema` URLs, connect from the platform's
|
|
1458
|
+
host address, so those databases must be reachable from there.
|
|
1123
1459
|
|
|
1124
1460
|
- normal test runs always use transient host-managed Postgres
|
|
1125
1461
|
- template databases are cached
|
|
@@ -1131,8 +1467,12 @@ platform's host address.
|
|
|
1131
1467
|
- template fingerprints are derived automatically from env files, source schema
|
|
1132
1468
|
cache, migrate/seed config, and repo contents
|
|
1133
1469
|
|
|
1134
|
-
The service process receives its provisioned database as `DATABASE_URL`.
|
|
1135
|
-
|
|
1470
|
+
The service process receives its provisioned database as `DATABASE_URL`. Every
|
|
1471
|
+
task gets the same `DATABASE_URL` when its service declares a database, and its
|
|
1472
|
+
service's URL as `BASE_URL` when the service declares a server, whatever its
|
|
1473
|
+
engine. The provisioned `DATABASE_URL` replaces one that the service's `env`
|
|
1474
|
+
declares. A test whose group uses several services with databases reads each
|
|
1475
|
+
one with `database(name)` from `@gfdelarue/testkit/environment`. Declared `env` values can also use the `{dbUrl}`, `{dbHost}`,
|
|
1136
1476
|
`{dbPort}`, `{dbName}`, `{dbUser}`, `{dbPassword}`, and `{dbSsl}` tokens, with
|
|
1137
1477
|
`:serviceName` to read the database of a service listed in `dependsOn`. Testkit
|
|
1138
1478
|
resolves them from the database provisioned for the current run. They are
|
|
@@ -1143,53 +1483,3 @@ schema cache. `db schema verify` prepares local templates and verifies local
|
|
|
1143
1483
|
replay against the cached/refreshed source schema. `--skip-schema-source-verify`
|
|
1144
1484
|
is available as a narrow escape hatch when users need to run tests while schema
|
|
1145
1485
|
verification is temporarily blocked.
|
|
1146
|
-
|
|
1147
|
-
## Development Tests
|
|
1148
|
-
|
|
1149
|
-
```bash
|
|
1150
|
-
npm test
|
|
1151
|
-
npm run test:unit
|
|
1152
|
-
npm run test:integration
|
|
1153
|
-
npm run test:system
|
|
1154
|
-
npm run test:live
|
|
1155
|
-
npm run test:integration -- fixture-package
|
|
1156
|
-
```
|
|
1157
|
-
|
|
1158
|
-
The four lanes use Vitest directly. Unit tests (source and pure tooling) run in
|
|
1159
|
-
parallel; integration and system files run serially; live tests are explicitly
|
|
1160
|
-
opt-in and serial. Pass ordinary Vitest filters after npm's `--` separator.
|
|
1161
|
-
`npm test` runs the first three lanes and never collects live cases.
|
|
1162
|
-
|
|
1163
|
-
Live tests cover credentials, external services, network-acquired runtimes,
|
|
1164
|
-
alternate toolchains, and Docker/platform matrices. An explicitly selected live
|
|
1165
|
-
test fails clearly when its prerequisites are unavailable; it does not silently
|
|
1166
|
-
skip.
|
|
1167
|
-
|
|
1168
|
-
Repository commands use npm's native timing support, enabled project-wide by
|
|
1169
|
-
the tracked `.npmrc`. Build, lint, and release modes support `help`; release
|
|
1170
|
-
provides `bump` and `check` modes.
|
|
1171
|
-
Every merge to `master` publishes the next patch version. The existing
|
|
1172
|
-
pre-merge CI checks provide verification. The
|
|
1173
|
-
[release workflow](https://github.com/gfdelarue/testkit/blob/master/docs/release-gates.md) tags the merge and publishes through npm
|
|
1174
|
-
trusted publishing; merge a `package.json` bump to start a new minor or major.
|
|
1175
|
-
|
|
1176
|
-
The static audit retains package/lock/runtime policy, engine dependency/import
|
|
1177
|
-
intent, retired engine/product references, minimal repository traversal, and
|
|
1178
|
-
the dependency direction between top-level `src/` modules. The edge table in
|
|
1179
|
-
`scripts/lint/audits/import-direction.ts` mirrors the import graph exactly: an
|
|
1180
|
-
import outside the table fails lint, and so does an entry that no import
|
|
1181
|
-
exercises. Module direction is the one structural rule lint enforces; other
|
|
1182
|
-
implementation-shape audits stay out of lint.
|
|
1183
|
-
|
|
1184
|
-
JavaScript modules under `test/fixtures/**` describe fixture behavior and keep
|
|
1185
|
-
their fixture-owned paths. Outside fixtures, only `bin/testkit.mjs` and the
|
|
1186
|
-
documented raw Node process probe remain `.mjs`; the probe is a program, not a
|
|
1187
|
-
Vitest suite.
|
|
1188
|
-
|
|
1189
|
-
GitHub Actions run a narrow installed Node/Vitest consumer smoke on dedicated
|
|
1190
|
-
Orthrus Ubuntu x64 and Windows x64 self-hosted runners for pull requests from
|
|
1191
|
-
this repository and manual dispatches. Fork pull requests skip this workflow's
|
|
1192
|
-
jobs. Local `npm test`,
|
|
1193
|
-
`npm run test:unit`, `npm run test:integration`, and `npm run test:system` remain
|
|
1194
|
-
the full developer entrypoints. PostgreSQL and Docker compatibility are explicit
|
|
1195
|
-
opt-in dispatcher cases. The smoke workflow is not five-platform release proof.
|