@cbartens/bkit 6.1.1 → 6.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.bkit/README.md +14 -12
- package/.bkit/core/.agents/docs.md +1 -11
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-gateguard.cjs +30 -9
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-guardrails.cjs +25 -27
- package/.bkit/core/.agents/rules.md +22 -9
- package/.bkit/core/.bkit/README.md +13 -11
- package/.bkit/core/.bkit/authoring.md +58 -13
- package/.bkit/core/.bkit/cli.md +42 -32
- package/.bkit/core/.bkit/cloud.md +93 -30
- package/.bkit/core/.bkit/install.md +55 -16
- package/.bkit/core/.bkit/jsonl.md +24 -7
- package/.bkit/core/.bkit/pipeline.md +8 -8
- package/.bkit/core/.bkit/prompts.md +160 -43
- package/.bkit/core/.bkit/providers.md +97 -36
- package/.bkit/core/.bkit/templates.md +162 -34
- package/.bkit/core/.bkit/trackers.md +39 -6
- package/.bkit/core/.bkit/workflows.md +344 -136
- package/.bkit/core/.claude/hooks/bkit-guardrails.cjs +50 -27
- package/.bkit/core/.claude/hooks/bkit-preflight.sh +11 -11
- package/.bkit/core/.codex/hooks/bkit-gateguard.cjs +13 -8
- package/.bkit/core/.codex/hooks/bkit-guardrails.cjs +14 -25
- package/.bkit/core/.cursor/hooks/bkit-guardrails.cjs +20 -9
- package/.bkit/core/.cursor/hooks/bkit-preflight.sh +8 -8
- package/.bkit/core/.opencode/plugin/bkit-gateguard.js +14 -4
- package/.bkit/core/.opencode/plugin/bkit-guardrails.js +114 -0
- package/.bkit/core/docs/templates/bkit-audit-accessibility.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-adoption.md +30 -6
- package/.bkit/core/docs/templates/bkit-audit-agents.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-aio-seo.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-analytics.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-annotations.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-app-performance.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-app-store.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-architecture.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-ci-performance.md +2 -2
- package/.bkit/core/docs/templates/bkit-audit-config.md +3 -2
- package/.bkit/core/docs/templates/bkit-audit-cost.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-coverage.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-database.md +3 -3
- package/.bkit/core/docs/templates/bkit-audit-dead-code.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-deferrals.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-dependencies.md +3 -3
- package/.bkit/core/docs/templates/bkit-audit-documentation.md +32 -2
- package/.bkit/core/docs/templates/bkit-audit-duplication.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-failures.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-flakes.md +2 -2
- package/.bkit/core/docs/templates/bkit-audit-gates.md +5 -3
- package/.bkit/core/docs/templates/bkit-audit-issues.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-llm.md +2 -2
- package/.bkit/core/docs/templates/bkit-audit-logs.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-models.md +2 -2
- package/.bkit/core/docs/templates/bkit-audit-privacy.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-prompts.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-red-team.md +2 -2
- package/.bkit/core/docs/templates/bkit-audit-reliability.md +3 -2
- package/.bkit/core/docs/templates/bkit-audit-repo.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-security.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-slop.md +1 -1
- package/.bkit/core/docs/templates/bkit-audit-testing.md +3 -2
- package/.bkit/core/docs/templates/bkit-audit-usability.md +1 -1
- package/.bkit/core/docs/templates/bkit-plan.md +38 -8
- package/.bkit/core/engine/checks/after/auth.ts +118 -47
- package/.bkit/core/engine/checks/after/github.ts +89 -37
- package/.bkit/core/engine/checks/after/index.ts +15 -3
- package/.bkit/core/engine/checks/after/tools.ts +245 -1
- package/.bkit/core/engine/checks/before/files.ts +59 -1
- package/.bkit/core/engine/checks/before/index.ts +8 -0
- package/.bkit/core/engine/checks/status-holes.ts +20 -26
- package/.bkit/core/engine/checks/tips/catalogue.ts +107 -10
- package/.bkit/core/engine/checks/tips/detect.ts +36 -3
- package/.bkit/core/engine/checks/tips/index.ts +1 -1
- package/.bkit/core/engine/cli.ts +13 -13
- package/.bkit/core/engine/commands/ci/ci.ts +64 -11
- package/.bkit/core/engine/commands/ci/push-gate.ts +101 -43
- package/.bkit/core/engine/commands/install/install.ts +28 -25
- package/.bkit/core/engine/commands/install/uninstall.ts +42 -30
- package/.bkit/core/engine/commands/install/upgrade.ts +14 -2
- package/.bkit/core/engine/commands/land.ts +2 -3
- package/.bkit/core/engine/commands/lib.ts +394 -39
- package/.bkit/core/engine/commands/registry.ts +1 -1
- package/.bkit/core/engine/commands/workflow/branches.ts +351 -78
- package/.bkit/core/engine/commands/workflow/challenge.ts +63 -14
- package/.bkit/core/engine/commands/workflow/context.ts +71 -10
- package/.bkit/core/engine/commands/workflow/counters.ts +298 -23
- package/.bkit/core/engine/commands/workflow/decisions.ts +260 -125
- package/.bkit/core/engine/commands/workflow/disk.ts +2 -1
- package/.bkit/core/engine/commands/workflow/feature-base.ts +66 -0
- package/.bkit/core/engine/commands/workflow/issues.ts +312 -17
- package/.bkit/core/engine/commands/workflow/junit.ts +204 -0
- package/.bkit/core/engine/commands/workflow/ledgers.ts +904 -99
- package/.bkit/core/engine/commands/workflow/locks.ts +186 -29
- package/.bkit/core/engine/commands/workflow/loops.ts +87 -0
- package/.bkit/core/engine/commands/workflow/open-decision.ts +251 -0
- package/.bkit/core/engine/commands/workflow/plan-blocks.ts +8 -0
- package/.bkit/core/engine/commands/workflow/plans.ts +1 -1
- package/.bkit/core/engine/commands/workflow/prs.ts +311 -16
- package/.bkit/core/engine/commands/workflow/repo.ts +1 -1
- package/.bkit/core/engine/commands/workflow/run-facts.ts +6 -0
- package/.bkit/core/engine/commands/workflow/runs.ts +877 -174
- package/.bkit/core/engine/commands/workflow/runtime.ts +12 -4
- package/.bkit/core/engine/commands/workflow/status.ts +324 -11
- package/.bkit/core/engine/commands/workflow/step-table.ts +41 -15
- package/.bkit/core/engine/commands/workflow/steps.ts +604 -29
- package/.bkit/core/engine/commands/workflow/summary-findings.ts +1 -1
- package/.bkit/core/engine/commands/workflow/summary.ts +314 -8
- package/.bkit/core/engine/commands/workflow/tasks.ts +400 -27
- package/.bkit/core/engine/commands/workflow/tests-base.ts +319 -0
- package/.bkit/core/engine/commands/workflow/tests-changed.ts +138 -74
- package/.bkit/core/engine/commands/workflow/tests-jump.ts +93 -0
- package/.bkit/core/engine/commands/workflow/tests.ts +815 -182
- package/.bkit/core/engine/commands/workflow/tracker.ts +1134 -137
- package/.bkit/core/engine/commands/workflow/workflows.ts +100 -72
- package/.bkit/core/engine/commands/workflow/worktrees.ts +421 -57
- package/.bkit/core/engine/install/cloud-env.ts +38 -10
- package/.bkit/core/engine/install/managed-block.ts +105 -0
- package/.bkit/core/engine/install/manifest.ts +7 -1
- package/.bkit/core/engine/install/modules/claude.ts +6 -2
- package/.bkit/core/engine/install/modules/cloud.ts +84 -95
- package/.bkit/core/engine/install/modules/codex.ts +10 -2
- package/.bkit/core/engine/install/modules/core.ts +148 -14
- package/.bkit/core/engine/install/modules/opencode.ts +18 -23
- package/.bkit/core/engine/install/network-allowlist.ts +8 -3
- package/.bkit/core/engine/install/project-docs.ts +650 -0
- package/.bkit/core/engine/install/template-copies.ts +58 -5
- package/.bkit/core/engine/install/write.ts +128 -7
- package/.bkit/core/engine/ledgers/journal.ts +18 -10
- package/.bkit/core/engine/ledgers/migrate.ts +20 -8
- package/.bkit/core/engine/ledgers/status.ts +604 -8
- package/.bkit/core/engine/ledgers/telemetry.ts +115 -13
- package/.bkit/core/engine/render/trackers.ts +3 -1
- package/.bkit/core/engine/shared/package-manager.ts +7 -3
- package/.bkit/core/engine/shared/settings.ts +12 -1
- package/.bkit/core/engine/shared/text.ts +14 -0
- package/.bkit/core/engine/tools/instructions-sources.ts +1 -0
- package/.bkit/core/engine/tools/instructions.ts +116 -20
- package/.bkit/core/hooks/antigravity-hook.js +20 -4
- package/.bkit/core/hooks/gateguard-edit-payload.js +8 -5
- package/.bkit/core/hooks/gateguard-engine.js +249 -23
- package/.bkit/core/hooks/guardrails-deny.js +167 -0
- package/.bkit/core/hooks/guardrails-read.js +3 -3
- package/.bkit/core/hooks/guardrails-shell.js +1122 -167
- package/.bkit/core/hooks/guardrails-tiers.js +247 -176
- package/.bkit/core/hooks/preflight-tracker.sh +219 -0
- package/.bkit/core/prompts/bkit-audit-generate.md +23 -6
- package/.bkit/core/prompts/bkit-build-challenge-1.md +5 -4
- package/.bkit/core/prompts/bkit-build-challenge-2.md +6 -5
- package/.bkit/core/prompts/bkit-build-install-skills.md +2 -2
- package/.bkit/core/prompts/bkit-build-plan-convert.md +21 -7
- package/.bkit/core/prompts/bkit-build-plan-current.md +5 -4
- package/.bkit/core/prompts/bkit-build-plan-gaps-remediate.md +22 -17
- package/.bkit/core/prompts/bkit-build-plan-tasks.md +16 -12
- package/.bkit/core/prompts/bkit-build-plan-tests.md +14 -10
- package/.bkit/core/prompts/bkit-build-plan-verify.md +21 -8
- package/.bkit/core/prompts/bkit-debug-abort-handover.md +10 -5
- package/.bkit/core/prompts/bkit-debug-failure-diagnose.md +22 -6
- package/.bkit/core/prompts/bkit-debug-failure-remediate.md +19 -3
- package/.bkit/core/prompts/bkit-debug-failure-reproduce.md +22 -6
- package/.bkit/core/prompts/bkit-debug-failure-soak.md +10 -7
- package/.bkit/core/prompts/bkit-design-brainstorm-ideate-1.md +3 -3
- package/.bkit/core/prompts/bkit-design-brainstorm-ideate-2.md +4 -4
- package/.bkit/core/prompts/bkit-design-brainstorm-ideate-3.md +3 -3
- package/.bkit/core/prompts/bkit-design-generate.md +6 -4
- package/.bkit/core/prompts/bkit-design-organise.md +4 -3
- package/.bkit/core/prompts/bkit-design-review-1.md +5 -4
- package/.bkit/core/prompts/bkit-design-review-2.md +6 -5
- package/.bkit/core/prompts/bkit-design-spike.md +4 -3
- package/.bkit/core/prompts/bkit-design-update.md +6 -4
- package/.bkit/core/prompts/bkit-plan-generate.md +35 -14
- package/.bkit/core/prompts/bkit-plan-review-1.md +9 -7
- package/.bkit/core/prompts/bkit-plan-review-2.md +9 -7
- package/.bkit/core/prompts/bkit-plan-scope-check.md +8 -6
- package/.bkit/core/prompts/bkit-plan-update.md +22 -8
- package/.bkit/core/prompts/bkit-ship-conflict-resolve.md +11 -8
- package/.bkit/core/prompts/bkit-ship-deferred-sweep.md +9 -5
- package/.bkit/core/prompts/bkit-ship-update-docs.md +26 -4
- package/.bkit/core/prompts/bkit-utility-compress-markdown.md +3 -3
- package/.bkit/core/prompts/bkit-utility-update-decisions.md +10 -7
- package/.bkit/core/prompts/bkit-utility-update-deferrals.md +8 -5
- package/.bkit/core/prompts/bkit-utility-update-documentation.md +31 -7
- package/.bkit/core/prompts/bkit-utility-update-failures.md +8 -5
- package/.bkit/core/prompts/bkit-utility-update-knowledge.md +10 -7
- package/.bkit/core/prompts/bkit-utility-update-roadmap.md +4 -4
- package/.bkit/core/prompts/bkit-utility-update-tracker.md +4 -4
- package/.bkit/core/workflows/bkit-auto.md +23 -11
- package/.bkit/core/workflows/bkit-build.md +331 -100
- package/.bkit/core/workflows/bkit-debug.md +135 -64
- package/.bkit/core/workflows/bkit-design.md +58 -32
- package/.bkit/core/workflows/bkit-plan.md +73 -45
- package/.bkit/core/workflows/bkit-ship.md +233 -167
- package/.bkit/shared/.agents/plugins/bkit/hooks.json +1 -1
- package/.bkit/shared/.bkit/.env.example +5 -0
- package/.bkit/shared/.claude/bkit-cloud.md +7 -7
- package/.bkit/shared/.claude/settings.json +5 -1
- package/.bkit/shared/.cursor/bkit-cloud.md +1 -1
- package/.bkit/shared/.cursor/hooks.json +1 -1
- package/.bkit/shared/.opencode/opencode.json +11 -2
- package/.bkit/templates/.claude/hooks/bkit-cloud-bootstrap.sh +27 -0
- package/.bkit/templates/.cursor/Dockerfile +7 -7
- package/.bkit/templates/.github/ISSUE_TEMPLATE/bkit-idea.yml +1 -1
- package/.bkit/templates/.github/dependabot.yml +5 -0
- package/.bkit/templates/.github/renovate.json5 +5 -3
- package/.bkit/templates/.github/workflows/bkit-ci.yml +11 -11
- package/.bkit/templates/.github/workflows/bkit-cloud.yml +525 -161
- package/.bkit/templates/.github/workflows/bkit-renovate.yml +2 -2
- package/.bkit/templates/.husky/pre-push +123 -6
- package/.bkit/templates/context/bkit-serena.md +1 -1
- package/.bkit/templates/docs/architecture.md +98 -0
- package/.bkit/templates/docs/capabilities.md +33 -37
- package/.bkit/templates/docs/components.md +142 -0
- package/.bkit/templates/docs/data-schema.md +87 -0
- package/.bkit/{shared → templates}/docs/deployments.md +128 -142
- package/.bkit/templates/docs/design.md +117 -0
- package/.bkit/templates/docs/noise.md +54 -0
- package/.bkit/templates/docs/observability.md +99 -0
- package/.bkit/templates/docs/security.md +137 -0
- package/.bkit/templates/docs/setup.md +93 -0
- package/.bkit/templates/docs/technology.md +70 -0
- package/.bkit/{shared → templates}/docs/testing.md +200 -87
- package/.bkit/templates/scripts/ci/bkit-machine-lock.sh +511 -103
- package/.bkit/templates/scripts/ci/bkit-test-awake.mjs +228 -35
- package/.bkit/templates/scripts/ci/bkit-test-awake.test.mjs +270 -2
- package/.bkit/templates/scripts/graphify/bkit-graphify-graphml.py +1 -1
- package/.bkit/templates/scripts/graphify/bkit-graphify-rollup.py +92 -63
- package/.bkit/templates/scripts/graphify/bkit-graphify.sh +59 -11
- package/.bkit/templates/slop-scan.config.json +2 -0
- package/CHANGELOG.md +64 -0
- package/DETAILS.md +24 -19
- package/README.md +1 -1
- package/dist/checks/after/auth.js +114 -46
- package/dist/checks/after/github.js +77 -35
- package/dist/checks/after/index.js +12 -3
- package/dist/checks/after/tools.js +224 -1
- package/dist/checks/before/files.js +59 -1
- package/dist/checks/before/index.js +7 -1
- package/dist/checks/status-holes.js +20 -21
- package/dist/checks/tips/catalogue.js +99 -10
- package/dist/checks/tips/detect.js +35 -3
- package/dist/checks/tips/index.js +1 -1
- package/dist/cli.js +13 -13
- package/dist/commands/ci/ci.js +56 -12
- package/dist/commands/ci/push-gate.js +79 -44
- package/dist/commands/install/install.js +23 -22
- package/dist/commands/install/uninstall.js +36 -30
- package/dist/commands/install/upgrade.js +10 -2
- package/dist/commands/land.js +2 -3
- package/dist/commands/lib.js +323 -25
- package/dist/commands/workflow/branches.js +312 -72
- package/dist/commands/workflow/challenge.js +66 -13
- package/dist/commands/workflow/context.js +46 -10
- package/dist/commands/workflow/counters.js +231 -23
- package/dist/commands/workflow/decisions.js +221 -106
- package/dist/commands/workflow/disk.js +2 -1
- package/dist/commands/workflow/feature-base.js +60 -0
- package/dist/commands/workflow/issues.js +243 -15
- package/dist/commands/workflow/junit.js +137 -0
- package/dist/commands/workflow/ledgers.js +726 -112
- package/dist/commands/workflow/locks.js +150 -29
- package/dist/commands/workflow/loops.js +67 -0
- package/dist/commands/workflow/open-decision.js +199 -0
- package/dist/commands/workflow/plan-blocks.js +6 -0
- package/dist/commands/workflow/plans.js +1 -1
- package/dist/commands/workflow/prs.js +272 -16
- package/dist/commands/workflow/repo.js +1 -1
- package/dist/commands/workflow/run-facts.js +6 -0
- package/dist/commands/workflow/runs.js +747 -155
- package/dist/commands/workflow/runtime.js +8 -4
- package/dist/commands/workflow/status.js +259 -13
- package/dist/commands/workflow/step-table.js +41 -15
- package/dist/commands/workflow/steps.js +530 -26
- package/dist/commands/workflow/summary-findings.js +1 -1
- package/dist/commands/workflow/summary.js +289 -8
- package/dist/commands/workflow/tasks.js +294 -27
- package/dist/commands/workflow/tests-base.js +266 -0
- package/dist/commands/workflow/tests-changed.js +130 -56
- package/dist/commands/workflow/tests-jump.js +77 -0
- package/dist/commands/workflow/tests.js +608 -131
- package/dist/commands/workflow/tracker.js +920 -112
- package/dist/commands/workflow/workflows.js +80 -59
- package/dist/commands/workflow/worktrees.js +387 -58
- package/dist/install/cloud-env.js +32 -10
- package/dist/install/managed-block.js +101 -0
- package/dist/install/manifest.js +7 -1
- package/dist/install/modules/claude.js +6 -2
- package/dist/install/modules/cloud.js +73 -86
- package/dist/install/modules/codex.js +9 -2
- package/dist/install/modules/core.js +133 -14
- package/dist/install/modules/opencode.js +18 -23
- package/dist/install/network-allowlist.js +8 -3
- package/dist/install/project-docs.js +551 -0
- package/dist/install/template-copies.js +55 -5
- package/dist/install/write.js +117 -5
- package/dist/ledgers/journal.js +10 -8
- package/dist/ledgers/migrate.js +20 -9
- package/dist/ledgers/status.js +461 -8
- package/dist/ledgers/telemetry.js +71 -13
- package/dist/render/trackers.js +3 -1
- package/dist/shared/package-manager.js +7 -2
- package/dist/shared/settings.js +8 -0
- package/dist/shared/text.js +13 -0
- package/dist/tools/instructions-sources.js +1 -0
- package/dist/tools/instructions.js +97 -24
- package/package.json +22 -15
- package/.bkit/core/hooks/preflight-linear.sh +0 -73
- package/.bkit/shared/docs/architecture.md +0 -16
- package/.bkit/shared/docs/data-schema.md +0 -16
- package/.bkit/shared/docs/design.md +0 -38
- package/.bkit/shared/docs/noise.md +0 -20
- package/.bkit/shared/docs/observability.md +0 -22
- package/.bkit/shared/docs/security.md +0 -62
- package/.bkit/shared/docs/setup.md +0 -52
- package/.bkit/shared/docs/technology.md +0 -15
- /package/.bkit/templates/{.lintstagedrc.json → lintstagedrc.example.json} +0 -0
- /package/.bkit/templates/{.prettierignore → prettierignore.example} +0 -0
|
@@ -1,15 +1,44 @@
|
|
|
1
|
-
|
|
2
|
-
|
|
3
|
-
|
|
1
|
+
> Starter document. `bkit install` does not write this file and does not overwrite a copy. Copy it to `docs/testing.md`, fill the project slots, keep the bkit practice sections, then delete this introduction.
|
|
2
|
+
>
|
|
3
|
+
> **Fill these first:** `docs/architecture.md`.
|
|
4
|
+
>
|
|
5
|
+
> **Finish against:** `docs/data-schema.md` for `## Test data`, which waits for the fixtures, the seed and the fresh store each tier needs.
|
|
6
|
+
>
|
|
7
|
+
> **Where the facts come from:** the package manifest scripts, the test runner configuration, the folders that hold test files, the CI configuration and the shared test helpers.
|
|
8
|
+
>
|
|
9
|
+
> **Fill in:**
|
|
10
|
+
>
|
|
11
|
+
> 1. Copy this starter to `docs/testing.md`.
|
|
12
|
+
> 2. Fill `## Stack` with the runner and framework each tier uses.
|
|
13
|
+
> 3. Fill `## Tiers this project has` with one row per tier. Write `Not applicable` in a tier the project does not have.
|
|
14
|
+
> 4. Fill `## Where tests live` with the folder, file naming and shared helpers of each tier. Record every new test file pattern here.
|
|
15
|
+
> 5. Fill `## Test data` with the shared login setup that UI tests reuse, the typed factories that build test data and the fresh store each tier starts from.
|
|
16
|
+
> 6. Fill `## CI lanes` with the lanes the project runs, what each needs and when it runs.
|
|
17
|
+
> 7. Fill `## Test reports` with where each tier writes its report, and the `bkit:test:file` command.
|
|
18
|
+
> 8. Fill `## Running locally` with the commands that run each tier.
|
|
19
|
+
> 9. Keep every section marked `bkit practice`. Edit its text only to correct it for the stack, and never change its heading.
|
|
20
|
+
> 10. Record unresolved facts under `## Open questions`.
|
|
21
|
+
> 11. Add this row to the `## Project Documents` table in `.agents/docs.md`: | `docs/testing.md` | ○ | ● | ● | ● | — | Testing strategy: tier boundaries, framework choices, naming conventions and the CI lane graph. |
|
|
22
|
+
> 12. Check the document against each completion check, then delete this introduction.
|
|
23
|
+
>
|
|
24
|
+
> **If it does not apply:** write `Not applicable: {reason}` under a project slot that does not apply, such as `## Test data` for a project with no stored data.
|
|
25
|
+
>
|
|
26
|
+
> **Completion checks:**
|
|
27
|
+
>
|
|
28
|
+
> - Every tier named in `## Which tier does a behaviour need?` has a row in `## Tiers this project has`.
|
|
29
|
+
> - Every lane in the CI configuration appears in `## CI lanes`.
|
|
30
|
+
> - Every tier with a runner has a command in `## Running locally`.
|
|
31
|
+
> - Every section is marked as a project slot or as bkit practice.
|
|
32
|
+
> - No example row remains.
|
|
4
33
|
|
|
5
34
|
# Testing
|
|
6
35
|
|
|
7
36
|
> [!CAUTION]
|
|
8
|
-
> **Purpose:**
|
|
37
|
+
> **Purpose:** Testing strategy: tier boundaries, framework choices, naming conventions and the CI lane graph.
|
|
9
38
|
>
|
|
10
39
|
> **Rules:**
|
|
11
40
|
>
|
|
12
|
-
> - Follow `.bkit/authoring.md` §
|
|
41
|
+
> - Follow `.bkit/authoring.md` § Docs when you edit this file.
|
|
13
42
|
> - Keep the tiers strictly separate: Static · Unit · Integration · E2E · @live · Smoke.
|
|
14
43
|
> - Pre-push runs `bkit check`, then selects one gate script in this order:
|
|
15
44
|
> - **If** `bkit ci min-push` accepts every outgoing ref → pre-push runs `pnpm bkit:gate:min`.
|
|
@@ -17,56 +46,146 @@
|
|
|
17
46
|
> - **If** `pnpm bkit:gate:static-unit-integration` is declared → pre-push runs it.
|
|
18
47
|
> - **Else** → pre-push runs `pnpm bkit:gate:static-unit`.
|
|
19
48
|
> - Gate scripts are consumer-owned. bkit's default is typecheck + lint + format:check + bkit:lint:prose + bkit:slop:delta + `pnpm test` (unit tier only).
|
|
49
|
+
> - A gate runs its tiers one after another, never two at once. A push gate stops at the first red tier. The E2E gate runs every tier after a red one, reports each, and runs the whole E2E suite. Keep parallel work inside a tier. A cached task replay reports the earlier run's result, not a fresh run.
|
|
20
50
|
> - Integration runs at pre-push once its gate is declared. Local services, such as a Docker database, are fine there.
|
|
21
51
|
> - E2E, @live and tests against a deployed service are excluded from pre-push. They run in CI (see below).
|
|
22
|
-
>
|
|
52
|
+
>
|
|
53
|
+
> **bkit practice:** every section marked `bkit practice` is bkit's own advice. Sections marked **bkit practice** keep their heading text and stay separate sections. Do not rename, merge or compress them, because bkit's CI template, advisories and audits cite them by name. Every section marked `Project slot` holds the project's own facts.
|
|
54
|
+
>
|
|
55
|
+
> **Not here:** Live alerts and log signals go in `docs/observability.md`. The runner, gate, lock and JUnit requirements bkit adds go in `.bkit/workflows.md` § Test and release mechanics. Deploy promotion and branch protection go in `docs/deployments.md`.
|
|
56
|
+
>
|
|
57
|
+
> **Update when:** A tier, runner, test folder, shared helper, CI lane or test command is added, changed or removed.
|
|
23
58
|
|
|
24
|
-
##
|
|
59
|
+
## Stack
|
|
25
60
|
|
|
26
|
-
|
|
61
|
+
> **Project slot:** the runner and framework each tier uses.
|
|
27
62
|
|
|
28
|
-
|
|
63
|
+
<!-- prettier-ignore -->
|
|
64
|
+
| Tier | Runner | Framework or tool |
|
|
65
|
+
| --- | --- | --- |
|
|
66
|
+
| [Tier] | [Command that runs it.] | [Framework or tool.] |
|
|
67
|
+
| [e.g., Unit] | The package's test script. | A test runner and its assertion library. |
|
|
29
68
|
|
|
30
|
-
|
|
69
|
+
## Tiers this project has
|
|
31
70
|
|
|
32
|
-
|
|
71
|
+
> **Project slot:** one row per tier. Names match § Which tier does a behaviour need?, because a plan's `Derived Test Matrix` copies the tier name from here.
|
|
33
72
|
|
|
34
|
-
|
|
73
|
+
<!-- prettier-ignore -->
|
|
74
|
+
| Tier | Has it | Covers |
|
|
75
|
+
| --- | --- | --- |
|
|
76
|
+
| Static | [Yes or no.] | [What it checks.] |
|
|
77
|
+
| Unit | [Yes or no.] | [What it checks.] |
|
|
78
|
+
| Integration | [Yes or no.] | [What it checks.] |
|
|
79
|
+
| @live (golden) | [Yes or no.] | [What it checks.] |
|
|
80
|
+
| E2E | [Yes or no.] | [What it checks.] |
|
|
81
|
+
| fresh-DB replay | [Yes or no.] | [What it checks.] |
|
|
82
|
+
| Smoke | [Yes or no.] | [What it checks.] |
|
|
83
|
+
| [e.g., Unit] | Yes | Pure logic in the report service. |
|
|
84
|
+
|
|
85
|
+
## Where tests live
|
|
86
|
+
|
|
87
|
+
> **Project slot:** where each tier's files sit. Record a new test file or pattern here.
|
|
88
|
+
|
|
89
|
+
<!-- prettier-ignore -->
|
|
90
|
+
| Tier | Folder | File naming | Shared helpers |
|
|
91
|
+
| --- | --- | --- | --- |
|
|
92
|
+
| [Tier] | [Folder.] | [Name pattern.] | [Helper modules.] |
|
|
93
|
+
| [e.g., Unit] | Next to the file it tests. | The source file's name with a test suffix. | The shared test utilities folder. |
|
|
94
|
+
|
|
95
|
+
## Test data
|
|
96
|
+
|
|
97
|
+
> **Project slot:** the login setup, the factories and the fresh store.
|
|
98
|
+
|
|
99
|
+
[The shared login setup whose saved auth state UI tests reuse, the typed factories that build test data, and the fresh store each tier starts from.]
|
|
35
100
|
|
|
36
|
-
|
|
101
|
+
## CI lanes
|
|
37
102
|
|
|
103
|
+
> **Project slot:** the lanes this project runs, as a chart and a table. § CI test tiers (the bkit scaffold) holds bkit's full lane graph.
|
|
104
|
+
|
|
105
|
+
```mermaid
|
|
106
|
+
flowchart LR
|
|
107
|
+
STATIC["static"] --> CHECK["check"]
|
|
108
|
+
UNIT["unit"] --> CHECK
|
|
38
109
|
```
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
110
|
+
|
|
111
|
+
<!-- prettier-ignore -->
|
|
112
|
+
| Lane | Needs | Runs when |
|
|
113
|
+
| --- | --- | --- |
|
|
114
|
+
| [Lane] | [Lanes it waits for.] | [Event or condition.] |
|
|
115
|
+
| [e.g., unit] | scope | Every push and pull request. |
|
|
116
|
+
|
|
117
|
+
## Test reports
|
|
118
|
+
|
|
119
|
+
> **Project slot:** where each tier writes its report.
|
|
120
|
+
|
|
121
|
+
<!-- prettier-ignore -->
|
|
122
|
+
| Tier | Report format | Path |
|
|
123
|
+
| --- | --- | --- |
|
|
124
|
+
| [Tier] | [Format.] | [Where it is written.] |
|
|
125
|
+
| [e.g., Unit] | JUnit XML | The path in `$BKIT_TEST_REPORT`. |
|
|
126
|
+
|
|
127
|
+
## Running locally
|
|
128
|
+
|
|
129
|
+
> **Project slot:** the commands a developer runs for each tier.
|
|
130
|
+
|
|
131
|
+
<!-- prettier-ignore -->
|
|
132
|
+
| Tier | Command |
|
|
133
|
+
| --- | --- |
|
|
134
|
+
| [Tier] | [Command.] |
|
|
135
|
+
| [e.g., Unit] | `pnpm test` |
|
|
136
|
+
|
|
137
|
+
## CI test tiers (the bkit scaffold)
|
|
138
|
+
|
|
139
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
140
|
+
|
|
141
|
+
CI runs the tiers through the bkit scaffold as one parallel lane (job) per tier, not as steps in one job. The step-by-step shape is in `.github/workflows/bkit-ci.yml` § "full CI scaffold reference", and the run-level shape is in `deployments.md` § Run-level decision matrix.
|
|
142
|
+
|
|
143
|
+
The graph is flat and wide for static / unit / build / integration: scope fans out, and the capstone joins. **One exception:** when `bkit-tests-migration` owns the shared-remote migration apply, e2e / live `needs:` that lane for ordering. Never gate live behind static / unit / build / e2e.
|
|
144
|
+
|
|
145
|
+
```mermaid
|
|
146
|
+
flowchart LR
|
|
147
|
+
SCOPE["bkit-tests-scope"] --> STATIC["bkit-tests-static"]
|
|
148
|
+
SCOPE --> UNIT["bkit-tests-unit"]
|
|
149
|
+
SCOPE --> BWEB["bkit-tests-build-web"]
|
|
150
|
+
SCOPE --> INTEG["bkit-tests-integration"]
|
|
151
|
+
SCOPE --> BIOS["bkit-tests-build-ios"]
|
|
152
|
+
SCOPE --> BAND["bkit-tests-build-android"]
|
|
153
|
+
SCOPE --> MIG["bkit-tests-migration"]
|
|
154
|
+
MIG --> DEPLOY["bkit-tests-deploy"]
|
|
155
|
+
BWEB --> DEPLOY
|
|
156
|
+
MIG --> RESET["bkit-tests-reset"]
|
|
157
|
+
DEPLOY --> E2EW["bkit-tests-e2e-web"]
|
|
158
|
+
DEPLOY --> LIVE["bkit-tests-live-web"]
|
|
159
|
+
DEPLOY --> E2EI["bkit-tests-e2e-ios"]
|
|
160
|
+
DEPLOY --> E2EA["bkit-tests-e2e-android"]
|
|
161
|
+
RESET --> E2EW
|
|
162
|
+
RESET --> LIVE
|
|
163
|
+
RESET --> E2EI
|
|
164
|
+
RESET --> E2EA
|
|
165
|
+
E2EW --> CLEAN["bkit-tests-cleanup"]
|
|
166
|
+
LIVE --> CLEAN
|
|
167
|
+
E2EI --> CLEAN
|
|
168
|
+
E2EA --> CLEAN
|
|
169
|
+
CLEAN --> CHECK["bkit-tests-check"]
|
|
170
|
+
STATIC --> CHECK
|
|
171
|
+
UNIT --> CHECK
|
|
172
|
+
INTEG --> CHECK
|
|
68
173
|
```
|
|
69
174
|
|
|
175
|
+
Each lane's `needs:` and `if:` are in this table.
|
|
176
|
+
|
|
177
|
+
<!-- prettier-ignore -->
|
|
178
|
+
| Lane | `needs:` | `if:` |
|
|
179
|
+
| --- | --- | --- |
|
|
180
|
+
| `bkit-tests-scope` | none | always on; never skips (~15s) |
|
|
181
|
+
| `bkit-tests-static` / `-unit` / `-integration` / `-build-*` | `bkit-tests-scope` | `run_level != 'none'` |
|
|
182
|
+
| `bkit-tests-migration` | `bkit-tests-scope` | no job-level `if:` (always on) |
|
|
183
|
+
| `bkit-tests-deploy` | `bkit-tests-migration`, `bkit-tests-build-web` | default-branch push only |
|
|
184
|
+
| `bkit-tests-reset` | scope, migration | `tenant_target != ''` |
|
|
185
|
+
| `-e2e-web` / `-live-web` / `-e2e-ios` / `-e2e-android` | scope, migration, deploy, reset, the lane's build | `!cancelled() && scope == success && <own gate> && migration != 'failure' && deploy != 'failure' && reset != 'failure'` |
|
|
186
|
+
| `bkit-tests-cleanup` | the lanes that use the lease | `always()`; releases the lease |
|
|
187
|
+
| `bkit-tests-check` | scope and every lane you have, deploy included | always |
|
|
188
|
+
|
|
70
189
|
Adopt only lanes your repo has. Add a lane the moment its tier exists, because a tier with no lane is a tier nothing gates. The jobs above the reference in `bkit-ci.yml` are the starter: `bkit-tests-static` and `bkit-tests-unit`, joined by the `bkit-tests-check` capstone `/bkit-ship` merges on. Every lane you add joins the capstone's `needs:` and its assert. The reference is the target. A starter stub that still sits inside `bkit-tests-unit` is the pre-split shape.
|
|
71
190
|
|
|
72
191
|
Adopt in this order, and stop at the first tier this repo does not have. You write the step bodies. These files name the job, the `needs`, the `if`, and the rule.
|
|
@@ -133,7 +252,7 @@ CI branches on the event in a few places. Each one is a deliberate, recorded div
|
|
|
133
252
|
| `human-confirmation` | A laptop reset of a remote tenant needs an explicit yes. The CI run is the approval. |
|
|
134
253
|
| `revert-recovery` | A revert push skips the pre-push unit and integration tiers. CI still runs every tier. |
|
|
135
254
|
| `output-channel` | CI prints annotations and job summaries. A terminal prints lines. |
|
|
136
|
-
| `ci-strict-time-limits` |
|
|
255
|
+
| `ci-strict-time-limits` | Time limits stay strict under `CI`; locally they are a fixed multiple of the CI limit that only catches a hang, with no load reading. |
|
|
137
256
|
| `playwright-outcomes` | Same retries, traces, and flake handling on every machine. A pass on retry is a failure. The target comes from `TEST_URL` alone. No spec is filtered by tag. |
|
|
138
257
|
|
|
139
258
|
### Which resources may each lane touch?
|
|
@@ -251,9 +370,9 @@ The same expression-context trap has a neighbour outside composites. The `env` c
|
|
|
251
370
|
|
|
252
371
|
### E2E on PRs: run the full suite, watch which DB it hits
|
|
253
372
|
|
|
254
|
-
|
|
373
|
+
PR-time E2E has two concerns. Both are gated on real preconditions (event, head ref), never on a kill-switch flag:
|
|
255
374
|
|
|
256
|
-
- **Parity — PR and push run the full suite.** `pull_request` runs the same E2E spec set as the `dev`/`
|
|
375
|
+
- **Parity — PR and push run the full suite.** `pull_request` runs the same E2E spec set as the `dev`/`main` push. Scale a slow suite by sharding (parallel lanes, more workers). PR and push runs differ only in their DB target (below) and in push-only tiers (for example, `@live`).
|
|
257
376
|
- **Correctness — migration-coupled PRs.** When preview deploys share one `dev` database, that database does not yet carry a PR's unmerged migration, because the migration applies only on the post-merge push. A PR that adds or renames a column and also uses it therefore fails with a 500 every time before the merge. A force-all-E2E marker does not help, because it hits the same migration-less DB.
|
|
258
377
|
- On feature→`dev` PRs, run E2E against a local DB that has applied this PR's migrations: reset local → prod build → run against `localhost`.
|
|
259
378
|
- Keep preview-backed E2E for `dev`/`main` pushes, for deploy fidelity.
|
|
@@ -281,6 +400,8 @@ Two further axes share the same root cause:
|
|
|
281
400
|
|
|
282
401
|
## Which tier does a behaviour need?
|
|
283
402
|
|
|
403
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
404
|
+
|
|
284
405
|
Assign every behaviour its **lowest sufficient tier(s)**: the lowest tier that actually exercises it. A behaviour may need more than one. Never substitute a lower or mocked tier for a boundary that a higher tier validates. **Under-tiering** is a defect, not a shortcut. It means using a lower or mocked tier where the rubric demands a higher or `@live` one, for example a user-visible flow with no E2E row, an external-API behaviour tested only against a local fixture, or a migration with no fresh-DB replay.
|
|
285
406
|
|
|
286
407
|
<!-- prettier-ignore -->
|
|
@@ -298,45 +419,10 @@ Assign every behaviour its **lowest sufficient tier(s)**: the lowest tier that a
|
|
|
298
419
|
|
|
299
420
|
A plan's `Derived Test Matrix` (`docs/templates/bkit-plan.md`) applies this rubric to one feature: one row per behaviour, each mapped to a build + verify task. Boundary rows carry the rigor above: reach the boundary, assert a signal only it can produce, and use no `as unknown as` at the call-site.
|
|
300
421
|
|
|
301
|
-
## Test first
|
|
302
|
-
|
|
303
|
-
`/bkit-build` writes each group's tests before its code and proves them: one subagent writes the planned tests and may touch nothing else, the engine records each one failing (red), and only then does another subagent write the code, which may touch no planned test, until each passes (green). This covers the unit and integration tiers only. E2E and `@live` rows run in CI and stay planned in the `Derived Test Matrix`. Each plan task names its tests on a `**TEST FIRST:**` field, one sub-bullet per test, or `n/a — {reason}`.
|
|
304
|
-
|
|
305
|
-
- **Declare `bkit:test:file`:** it runs the one test file bkit appends as its argument (a repo-relative path), and writes a JUnit XML report to the path in `$BKIT_TEST_REPORT`. Vitest: `vitest run --reporter=default --reporter=junit --outputFile.junit="$BKIT_TEST_REPORT"`. pytest: `pytest --junitxml="$BKIT_TEST_REPORT"`. jest needs the `jest-junit` reporter.
|
|
306
|
-
- **If** it is undeclared → the build reports test-first as not enforced.
|
|
307
|
-
- **If** the test files belong to more than one runner (for example, `.bats` and `node:test`) → point it at a script that picks the runner by file type.
|
|
308
|
-
- **If** `$BKIT_TEST_REPORT` is unset → pass no report flag. An empty `--outputFile.junit=` takes the next argument, the test file, as the report path, and overwrites it.
|
|
309
|
-
- **Declare `bkit:lint:file`:** it lints the one test file bkit appends (a repo-relative path) with the static gate's linter config, and exits 0 when clean. ESLint: `eslint --max-warnings=0`. Ruff: `ruff check`.
|
|
310
|
-
- **If** the linter does not cover the test file (for example, `.bats`) → exit 0 (Oxlint: `--no-error-on-unmatched-pattern`).
|
|
311
|
-
- The build lints each planned test file before running it for its red. **If** `bkit:lint:file` is undeclared → the red is recorded unlinted, and a lint rewrite after it changes a frozen test file, which the code step's finish refuses.
|
|
312
|
-
- **Call the runner directly:** a task runner that caches results, or runs dependent packages' suites first, breaks the one-file contract. A monorepo routes the path to the package and config that own it. An integration file gets the same environment setup that the integration suite gets.
|
|
313
|
-
- **Judged by the report, never the exit code:** a misspelled path, a syntax error and an import of unwritten code all exit non-zero, just like a real failure. A red counts when the named test ran and failed, or when its file failed to load with no case of its own in the report (the tests step writes no source, so an import of unwritten code is the expected shape of a red); a green only when it ran and passed.
|
|
314
|
-
- **Test name:** use the name as the runner reports it, or its last segment (`clamp > keeps a value inside the range` → `keeps a value inside the range`).
|
|
315
|
-
- **The red test stays:** after the red, the test may gain tests or formatting. A weakened, removed or rewritten assertion fails the build's verify.
|
|
316
|
-
|
|
317
|
-
## Tests lock
|
|
318
|
-
|
|
319
|
-
Wrap each test-running `bkit:gate:*` script with the adopted machine lock (`.bkit/templates.md` § Machine lock). The wrapped script holds the `tests` lock for the whole gate, across repositories. An unwired repository takes no lock.
|
|
320
|
-
|
|
321
|
-
- `bkit tests run` and `bkit tests changed` pass `--lock-wait-s` as `LOCAL_STACK_LOCK_WAIT_S` and their run name as `LOCAL_STACK_LOCK_WHO`; they run the declared gate without executing the lock script itself.
|
|
322
|
-
- `bkit branches push` passes the same wait budget and its run name to Git. When its pre-push hook's gate gives up on the tests lock, it exits 81 with `lockHolder` and reports nothing pushed; the workflow ends the turn and re-runs the block.
|
|
323
|
-
- A tests-lock give-up of this run becomes exit 81 with `lockHolder`, and a tests-lock refusal becomes exit 4. A lock wait writes no `test-run` status row. A gate's own failure or a nested lock's give-up remains a test failure. A raw gate caller sees exit 75 and the holder line.
|
|
324
|
-
- Light runs (`bkit tests red`, `green` and `bkit:test:file`) take no tests lock. Nested wraps reuse the live parent's tests lock. `CI=true` / `CI=1` bypasses locking.
|
|
325
|
-
- `BKIT_TEST_SLOTS` sets the number of tests slots (default 1). The lock exports `BKIT_TEST_WORKERS` for the consumer to pass to its runner.
|
|
326
|
-
- The CLI wait defaults to `BKIT_LOCK_WAIT_S`, else 60 seconds; `--lock-wait-s=0` tries once. Claude Code ships `BKIT_LOCK_WAIT_S=1200`; other hosts use the shorter wait and re-run on exit 81 (`.bkit/providers.md` § Long-running shell commands).
|
|
327
|
-
- The script defaults to 60 seconds without a TTY when no wait is set. Waiting notes name the holders and their checkouts' earlier run times.
|
|
328
|
-
|
|
329
|
-
## JUnit reports for bkit
|
|
330
|
-
|
|
331
|
-
`bkit tests run` counts a gate's failures from JUnit XML, not from its log. It empties a report folder before each run and passes its absolute path to the gate as `$BKIT_TEST_REPORT_DIR`.
|
|
332
|
-
|
|
333
|
-
- **Every runner in a gate writes JUnit there** when the variable is set: one file per runner or per run, `*.xml`, with any name. Two runs in one gate never write the same file name.
|
|
334
|
-
- **Counted:** failing plus erroring test cases across every file. Skipped cases do not count.
|
|
335
|
-
- **No reports:** `failures` is `null` (count unknown), and a hint says to turn the reporters on. The verdict still comes from the gate's exit code.
|
|
336
|
-
- **Unset variable:** the runner behaves as it always does, so a gate run by hand writes nothing extra.
|
|
337
|
-
|
|
338
422
|
## What each tier must do
|
|
339
423
|
|
|
424
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
425
|
+
|
|
340
426
|
Each tier lists the rule to follow, then a **Not:** line naming the failure the rule prevents. Stack tools (Vitest / Playwright / a Supabase-style local stack / a Vercel-style preview) appear only as worked examples, because the rule is the principle. Implement it for your stack.
|
|
341
427
|
|
|
342
428
|
### Static
|
|
@@ -379,7 +465,7 @@ Each tier lists the rule to follow, then a **Not:** line naming the failure the
|
|
|
379
465
|
|
|
380
466
|
### @live (golden)
|
|
381
467
|
|
|
382
|
-
- Cross the real external or deployed boundary, and assert a ground-truth signal that only it can produce (§ Validated means executed).
|
|
468
|
+
- Cross the real external or deployed boundary, and assert a ground-truth signal that only it can produce (§ Validated means executed). Two shapes exist: **in-process** (the test calls the real SDK / model / API directly) and **deployed-endpoint** (the test fetches the running app). Use whichever one observes the boundary you are validating.
|
|
383
469
|
- It runs on a push to `dev` or the default branch only, when `web_affected` (`run_live`). A planned failure reds the capstone. It does not run on a pull request or a dispatch. A newer run on the same ref cancels the older one (`deployments.md` § One run per branch or pull request).
|
|
384
470
|
- **Honest skip vs hollow:**
|
|
385
471
|
- **If** there is genuinely nothing deployed to validate → skip with a recorded reason. The capstone then asserts nothing about the count. This is different from a tier that ran 0 tests.
|
|
@@ -402,8 +488,23 @@ Each tier lists the rule to follow, then a **Not:** line naming the failure the
|
|
|
402
488
|
- It is an observability gate, not a merge gate. It has its own workflow and is never in branch protection. When `bkit-tests-deploy` promotes inside CI, this probe is the health workflow, not a second file (`deployments.md` § Smoke trigger plumbing, § Promote before the lanes).
|
|
403
489
|
- **Not:** a smoke suite wired into branch protection, or one so heavy that it cannot run on every deploy.
|
|
404
490
|
|
|
491
|
+
## Test first
|
|
492
|
+
|
|
493
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
494
|
+
|
|
495
|
+
`/bkit-build` runs each plan task's `**TEST FIRST:**` tests before their code and records each one failing (red) before the code step. This covers the unit and integration tiers only. The runners below are examples; declare the one the project uses.
|
|
496
|
+
|
|
497
|
+
- **Declare `bkit:test:file`:** it runs the one test file bkit appends as its argument (a repo-relative path), and writes a JUnit XML report to the path in `$BKIT_TEST_REPORT`. Vitest: `vitest run --reporter=default --reporter=junit --outputFile.junit="$BKIT_TEST_REPORT"`. pytest: `pytest --junitxml="$BKIT_TEST_REPORT"`. jest: `jest --runTestsByPath` with the `jest-junit` reporter and `JEST_JUNIT_OUTPUT_FILE="$BKIT_TEST_REPORT"`. Node: `node --test --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination="$BKIT_TEST_REPORT"`; write Node tests with `node:test` (`test` / `describe` + `it`).
|
|
498
|
+
- **Node case names:** a `node:test` file reports one case per `test()` call, named by the leaf `test` / `it` name only (the junit reporter drops the enclosing `describe` names, so the plan names the leaf, never `suite > test`). A file that does not use `node:test` (a plain script) reports one case named after the file path as passed to the command, so the plan names that test by the same path.
|
|
499
|
+
- **If** jest gets the path without `--runTestsByPath` → it reads the path as a regex, so a folder such as `(tabs)` matches no file and no report is written.
|
|
500
|
+
- **If** it is undeclared → the build reports test-first as not enforced.
|
|
501
|
+
- **If** the test files belong to more than one runner (for example, `.bats` and `node:test`) → point it at a script that picks the runner by file type.
|
|
502
|
+
- **If** `$BKIT_TEST_REPORT` is unset → pass no report flag. An empty `--outputFile.junit=` takes the next argument, the test file, as the report path, and overwrites it.
|
|
503
|
+
|
|
405
504
|
## Design assertions are a dimension, not a lane
|
|
406
505
|
|
|
506
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
507
|
+
|
|
407
508
|
**Never add a design lane.** The lane axis is **resource profile** (a millisecond file read vs a booted app + browser), not subject matter, because design assertions span both. A `bkit-tests-design` job (or a design-only browser project) splits one concern across two profiles and buys nothing that the existing lanes do not already provide. Fold each assertion into the tier that already fits it. The CI topology, the capstone's `needs:` and the browser-project partition all stay unchanged. A real consumer's full design-system rollout landed with `bkit-ci.yml` byte-unchanged.
|
|
408
509
|
|
|
409
510
|
<!-- prettier-ignore -->
|
|
@@ -419,7 +520,7 @@ Each tier lists the rule to follow, then a **Not:** line naming the failure the
|
|
|
419
520
|
| Visual regression (screenshot baselines) | **opt-in, not baseline** (§ Optional hardening) — highest maintenance cost of the set |
|
|
420
521
|
|
|
421
522
|
- **One reference, asserted independently.** The reference doc owns the token contract and its parsing rules (`design.md` § Machine-checked reference). What the lanes care about is that each implementation is its own test, so one red names its culprit.
|
|
422
|
-
- **Merge-gated from day one, never an advisory window.** The assertions live inside lanes that the capstone already gates, so a design red blocks the merge on the first PR that adds one.
|
|
523
|
+
- **Merge-gated from day one, never an advisory window.** The assertions live inside lanes that the capstone already gates, so a design red blocks the merge on the first PR that adds one. No promotion step needs scheduling, so no deferral row can be forgotten.
|
|
423
524
|
- **A group tag carries zero routing power.**
|
|
424
525
|
- Tag design tests in the test title, never by a filename convention, and add a `test:design` script for local iteration only.
|
|
425
526
|
- The tag must never appear in an execution gate or an inverse-grep filter, because a test without it still runs in its normal lane.
|
|
@@ -433,6 +534,8 @@ Each tier lists the rule to follow, then a **Not:** line naming the failure the
|
|
|
433
534
|
|
|
434
535
|
## Schema-invariant DB checks (e.g. RLS coverage)
|
|
435
536
|
|
|
537
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
538
|
+
|
|
436
539
|
Access-policy coverage is stack-specific. bkit ships the pattern (the two routes below), not a drop-in test. Supabase / PostgREST is the worked example. Implement it for your backing store by swapping the catalog source and the anon probe for your stack's equivalent.
|
|
437
540
|
|
|
438
541
|
Assert DB-level invariants (for example, "every public table has RLS enabled", with no hardcoded list) in one of two co-equal ways. Pick by deploy topology, not by whether you can run a local DB-test runner (for example, pgTAP):
|
|
@@ -445,6 +548,8 @@ An auto-discovering access-policy suite belongs in `bkit-tests-live-web`. For RL
|
|
|
445
548
|
|
|
446
549
|
## Provision backing services fresh
|
|
447
550
|
|
|
551
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
552
|
+
|
|
448
553
|
A tier that needs a stateful backing service (database, cache, queue, search index) must run against a freshly-provisioned instance, not a long-lived shared one. Green against a shared environment carrying accumulated state is not evidence of reproducibility. A migration that never re-applies cleanly, a seed that aborts on an empty store, or a row another run left behind all stay invisible until the service is rebuilt from scratch. Prove that a fresh provision + seed succeeds in CI: for example, an ephemeral per-PR database started fresh each run (Supabase: `supabase db reset` + seed). Reserve the shared remote / dev instance for the `@live` tier and push-side E2E, because they validate the real deployed schema. It is never the integration tier's source of truth.
|
|
449
554
|
|
|
450
555
|
bkit's `bkit-tests-migration` lane (`bkit-stub:bkit-tests-migration`) is the ready-made home.
|
|
@@ -473,6 +578,8 @@ bkit's `bkit-tests-migration` lane (`bkit-stub:bkit-tests-migration`) is the rea
|
|
|
473
578
|
|
|
474
579
|
## Optional hardening (adopt when the cost is justified)
|
|
475
580
|
|
|
581
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
582
|
+
|
|
476
583
|
Beyond the gates above, three practices raise confidence further. They are opt-in, not baseline.
|
|
477
584
|
|
|
478
585
|
- **Mutation testing** (for example, StrykerJS over a Vitest suite) automates the "executed != exercised" intent. It injects bugs into production code and fails when no test catches them.
|
|
@@ -484,13 +591,15 @@ Beyond the gates above, three practices raise confidence further. They are opt-i
|
|
|
484
591
|
|
|
485
592
|
## Coverage on PRs
|
|
486
593
|
|
|
594
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
595
|
+
|
|
487
596
|
Shipped stubs target Vitest (`bkit-stub:coverage`) and Playwright (`bkit-stub:e2e-coverage`). On another runner, keep the rules (per-tier, report-only, ratchet from current) and swap the tooling.
|
|
488
597
|
|
|
489
598
|
Report coverage on each PR without a paid product or a third-party service. For example, the `bkit-stub:coverage` block in `.github/workflows/bkit-ci.yml` runs `davelosert/vitest-coverage-report-action` entirely inside Actions with `GITHUB_TOKEN`, and posts sticky per-PR comments. That is safe for private repos, because no coverage data leaves the runner. It replaces GitHub Code Quality's coverage-on-PR value (`deployments.md` § Disable GitHub Code Quality billing). It does not replace CodeQL security scanning, which is a separate product.
|
|
490
599
|
|
|
491
600
|
**Per-tier, not merged.** Unit and integration are measured and commented separately: two runs into `coverage/unit` and `coverage/integration`, and two comments keyed by a distinct `name:`. A single merged number is the weakest option. Coverage is a union, so a line covered only by an integration test reads as covered and hides a missing unit test. A merged gate therefore lets integration mask a hollow unit suite. The two numbers also have different denominators, so they cannot be honestly combined into one.
|
|
492
601
|
|
|
493
|
-
- **Report-only by default.**
|
|
602
|
+
- **Report-only by default.** The block sets no `thresholds`, so coverage never reds the lane, and enabling the block on an existing repo cannot break it.
|
|
494
603
|
- **Reporters are load-bearing.** The action reads `coverage-summary.json` (from the `json-summary` reporter, which is mandatory) and `coverage-final.json` (from `json`, which `file-coverage-mode: changes` requires). Enable both in `coverage.reporter`, or the step does nothing.
|
|
495
604
|
- **Two runs, separate dirs.** Run `vitest run --coverage --coverage.reportsDirectory=coverage/unit --exclude '**/*.integration.test.ts'` for unit, and a second run into `coverage/integration` for the integration tier. Separate dirs also avoid the last-writer-wins clobber that a second `--coverage` run into the same dir would cause.
|
|
496
605
|
- **Thresholds are opt-in and per tier:**
|
|
@@ -530,7 +639,9 @@ The E2E tier (`bkit-stub:e2e-coverage`) reports Playwright coverage in the same
|
|
|
530
639
|
|
|
531
640
|
## CI verbosity (`[verbose ci]`)
|
|
532
641
|
|
|
533
|
-
|
|
642
|
+
> **bkit practice:** keep this section; bkit's CI template, advisories and audits cite its heading by name, so leave the heading text unchanged.
|
|
643
|
+
|
|
644
|
+
See `.bkit/workflows.md` § Test and release mechanics (Commit-message force tags) for the marker mechanics. That covers where to put it: the PR title for feature-branch pushes, and the commit message for a direct push to `dev` / `main`. This section covers the testing-specific knobs.
|
|
534
645
|
|
|
535
646
|
**Knobs flipped under the marker.** The shipped stub flips turbo / Node / Vercel / Supabase / Playwright knobs. Add your stack's equivalents.
|
|
536
647
|
|
|
@@ -548,4 +659,6 @@ See `deployments.md` § Commit-message force tags for the marker mechanics. That
|
|
|
548
659
|
|
|
549
660
|
**Artifact paths** (already in `.gitignore` via the bkit template): `test-results/`, `playwright-report/`, `blob-report/`, `playwright/.auth/`. Never commit these, because traces alone run 1-50MB each.
|
|
550
661
|
|
|
551
|
-
|
|
662
|
+
## Open questions
|
|
663
|
+
|
|
664
|
+
[Each unresolved fact, who can answer it and which section it affects.]
|