@cbartens/bkit 5.5.1 → 6.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.bkit/README.md +167 -154
- package/.bkit/core/.agents/context.md +95 -29
- package/.bkit/core/.agents/docs.md +18 -15
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-gateguard.cjs +1 -0
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-guardrails.cjs +6 -5
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-readguard.cjs +1 -0
- package/.bkit/core/.agents/plugins/bkit/hooks/bkit-rules.cjs +65 -0
- package/.bkit/core/.agents/rules.md +112 -102
- package/.bkit/core/.agents/skills.md +63 -15
- package/.bkit/core/.bkit/README.md +167 -154
- package/.bkit/core/.bkit/authoring.md +747 -0
- package/.bkit/core/.bkit/cli.md +407 -0
- package/.bkit/core/.bkit/cloud.md +283 -195
- package/.bkit/core/.bkit/comparison.md +45 -43
- package/.bkit/core/.bkit/install.md +166 -95
- package/.bkit/core/.bkit/jsonl.md +339 -196
- package/.bkit/core/.bkit/pipeline.md +142 -145
- package/.bkit/core/.bkit/prompts.md +295 -264
- package/.bkit/core/.bkit/providers.md +460 -277
- package/.bkit/core/.bkit/router.md +69 -0
- package/.bkit/core/.bkit/templates.md +530 -256
- package/.bkit/core/.bkit/trackers.md +116 -65
- package/.bkit/core/.bkit/workflows.md +742 -618
- package/.bkit/core/.claude/hooks/bkit-gateguard.cjs +2 -2
- package/.bkit/core/.claude/hooks/bkit-guardrails.cjs +3 -3
- package/.bkit/core/.claude/hooks/bkit-preflight.sh +45 -14
- package/.bkit/core/.codex/hooks/bkit-guardrails.cjs +3 -3
- package/.bkit/core/.codex/hooks/bkit-rules.cjs +57 -0
- package/.bkit/core/.cursor/hooks/bkit-guardrails.cjs +7 -7
- package/.bkit/core/.cursor/hooks/bkit-preflight.sh +47 -12
- package/.bkit/core/.opencode/plugin/bkit-rules.js +89 -0
- package/.bkit/core/AGENTS.md +6 -80
- package/.bkit/core/docs/templates/bkit-audit-accessibility.md +128 -130
- package/.bkit/core/docs/templates/bkit-audit-adoption.md +104 -94
- package/.bkit/core/docs/templates/bkit-audit-agents.md +96 -98
- package/.bkit/core/docs/templates/bkit-audit-aio-seo.md +100 -103
- package/.bkit/core/docs/templates/bkit-audit-analytics.md +141 -107
- package/.bkit/core/docs/templates/bkit-audit-annotations.md +99 -97
- package/.bkit/core/docs/templates/bkit-audit-app-performance.md +122 -121
- package/.bkit/core/docs/templates/bkit-audit-app-store.md +158 -101
- package/.bkit/core/docs/templates/bkit-audit-architecture.md +114 -96
- package/.bkit/core/docs/templates/bkit-audit-ci-performance.md +76 -97
- package/.bkit/core/docs/templates/bkit-audit-config.md +123 -101
- package/.bkit/core/docs/templates/bkit-audit-cost.md +86 -99
- package/.bkit/core/docs/templates/bkit-audit-coverage.md +76 -93
- package/.bkit/core/docs/templates/bkit-audit-database.md +102 -95
- package/.bkit/core/docs/templates/bkit-audit-dead-code.md +165 -108
- package/.bkit/core/docs/templates/bkit-audit-deferrals.md +115 -109
- package/.bkit/core/docs/templates/bkit-audit-dependencies.md +265 -113
- package/.bkit/core/docs/templates/bkit-audit-documentation.md +119 -118
- package/.bkit/core/docs/templates/bkit-audit-duplication.md +140 -111
- package/.bkit/core/docs/templates/bkit-audit-failures.md +111 -106
- package/.bkit/core/docs/templates/bkit-audit-flakes.md +100 -99
- package/.bkit/core/docs/templates/bkit-audit-gates.md +107 -103
- package/.bkit/core/docs/templates/bkit-audit-issues.md +105 -105
- package/.bkit/core/docs/templates/bkit-audit-llm.md +134 -85
- package/.bkit/core/docs/templates/bkit-audit-logs.md +124 -125
- package/.bkit/core/docs/templates/bkit-audit-models.md +99 -78
- package/.bkit/core/docs/templates/bkit-audit-privacy.md +56 -76
- package/.bkit/core/docs/templates/bkit-audit-prompts.md +63 -78
- package/.bkit/core/docs/templates/bkit-audit-red-team.md +125 -93
- package/.bkit/core/docs/templates/bkit-audit-reliability.md +97 -96
- package/.bkit/core/docs/templates/bkit-audit-repo.md +170 -98
- package/.bkit/core/docs/templates/bkit-audit-security.md +207 -121
- package/.bkit/core/docs/templates/bkit-audit-slop.md +78 -95
- package/.bkit/core/docs/templates/bkit-audit-testing.md +163 -139
- package/.bkit/core/docs/templates/bkit-audit-usability.md +114 -96
- package/.bkit/core/docs/templates/bkit-plan.md +158 -281
- package/.bkit/core/engine/checks/after/auth.ts +40 -37
- package/.bkit/core/engine/checks/after/ci-yml.ts +104 -31
- package/.bkit/core/engine/checks/after/github.ts +95 -17
- package/.bkit/core/engine/checks/after/index.ts +21 -8
- package/.bkit/core/engine/checks/after/suppression.ts +8 -5
- package/.bkit/core/engine/checks/after/tools.ts +125 -123
- package/.bkit/core/engine/checks/before/context-fetch.ts +227 -33
- package/.bkit/core/engine/checks/before/env.ts +3 -3
- package/.bkit/core/engine/checks/before/files.ts +5 -4
- package/.bkit/core/engine/checks/before/index.ts +25 -1
- package/.bkit/core/engine/checks/before/repo.ts +208 -2
- package/.bkit/core/engine/checks/status-holes.ts +916 -123
- package/.bkit/core/engine/checks/tips/catalogue.ts +82 -20
- package/.bkit/core/engine/checks/tips/detect.ts +66 -0
- package/.bkit/core/engine/checks/tips/index.ts +52 -11
- package/.bkit/core/engine/cli.ts +251 -130
- package/.bkit/core/engine/commands/analysis.ts +282 -199
- package/.bkit/core/engine/commands/ci/ci.ts +363 -29
- package/.bkit/core/engine/commands/ci/pins.ts +172 -0
- package/.bkit/core/engine/commands/ci/push-gate.ts +223 -0
- package/.bkit/core/engine/commands/comply.ts +77 -103
- package/.bkit/core/engine/commands/install/doctor.ts +2 -2
- package/.bkit/core/engine/commands/install/install.ts +57 -18
- package/.bkit/core/engine/commands/install/uninstall.ts +86 -55
- package/.bkit/core/engine/commands/install/upgrade.ts +32 -15
- package/.bkit/core/engine/commands/land.ts +139 -18
- package/.bkit/core/engine/commands/lib.ts +381 -22
- package/.bkit/core/engine/commands/registry.ts +10 -2
- package/.bkit/core/engine/commands/setup/setup.ts +8 -7
- package/.bkit/core/engine/commands/stats.ts +260 -0
- package/.bkit/core/engine/commands/tips.ts +10 -5
- package/.bkit/core/engine/commands/workflow/branches.ts +2215 -176
- package/.bkit/core/engine/commands/workflow/challenge.ts +748 -0
- package/.bkit/core/engine/commands/workflow/completeness.ts +109 -0
- package/.bkit/core/engine/commands/workflow/context.ts +1249 -47
- package/.bkit/core/engine/commands/workflow/counters.ts +483 -95
- package/.bkit/core/engine/commands/workflow/decisions.ts +585 -0
- package/.bkit/core/engine/commands/workflow/disk.ts +238 -56
- package/.bkit/core/engine/commands/workflow/fids.ts +811 -33
- package/.bkit/core/engine/commands/workflow/issues.ts +413 -64
- package/.bkit/core/engine/commands/workflow/ledgers.ts +1333 -36
- package/.bkit/core/engine/commands/workflow/locks.ts +28 -2
- package/.bkit/core/engine/commands/workflow/plan-blocks.ts +691 -0
- package/.bkit/core/engine/commands/workflow/plan-versions.ts +21 -0
- package/.bkit/core/engine/commands/workflow/plans.ts +257 -0
- package/.bkit/core/engine/commands/workflow/prs.ts +178 -33
- package/.bkit/core/engine/commands/workflow/repo.ts +66 -8
- package/.bkit/core/engine/commands/workflow/roadmap.ts +72 -22
- package/.bkit/core/engine/commands/workflow/run-facts.ts +492 -0
- package/.bkit/core/engine/commands/workflow/runs.ts +827 -79
- package/.bkit/core/engine/commands/workflow/runtime.ts +25 -1
- package/.bkit/core/engine/commands/workflow/skills.ts +1 -0
- package/.bkit/core/engine/commands/workflow/status.ts +329 -33
- package/.bkit/core/engine/commands/workflow/step-table.ts +381 -0
- package/.bkit/core/engine/commands/workflow/steps.ts +841 -105
- package/.bkit/core/engine/commands/workflow/summary-findings.ts +177 -0
- package/.bkit/core/engine/commands/workflow/summary.ts +544 -0
- package/.bkit/core/engine/commands/workflow/tasks.ts +409 -0
- package/.bkit/core/engine/commands/workflow/test-blocks.ts +254 -0
- package/.bkit/core/engine/commands/workflow/tests-changed.ts +201 -0
- package/.bkit/core/engine/commands/workflow/tests.ts +2124 -330
- package/.bkit/core/engine/commands/workflow/tracker.ts +1195 -0
- package/.bkit/core/engine/commands/workflow/workflows.ts +1197 -235
- package/.bkit/core/engine/commands/workflow/worktrees.ts +543 -24
- package/.bkit/core/engine/install/cloud-env.ts +251 -121
- package/.bkit/core/engine/install/context-ci-workflow.ts +102 -0
- package/.bkit/core/engine/install/manifest.ts +67 -17
- package/.bkit/core/engine/install/modules/antigravity.ts +3 -2
- package/.bkit/core/engine/install/modules/cloud.ts +24 -16
- package/.bkit/core/engine/install/modules/codex.ts +15 -2
- package/.bkit/core/engine/install/modules/core.ts +1196 -45
- package/.bkit/core/engine/install/modules/cursor.ts +6 -6
- package/.bkit/core/engine/install/modules/opencode.ts +18 -6
- package/.bkit/core/engine/install/template-notes.ts +130 -0
- package/.bkit/core/engine/install/version-file.ts +9 -4
- package/.bkit/core/engine/install/write.ts +5 -0
- package/.bkit/core/engine/ledgers/compact.ts +124 -0
- package/.bkit/core/engine/ledgers/journal.ts +291 -0
- package/.bkit/core/engine/ledgers/migrate.ts +92 -0
- package/.bkit/core/engine/ledgers/status.ts +249 -21
- package/.bkit/core/engine/ledgers/telemetry.ts +1397 -253
- package/.bkit/core/engine/render/hosts.ts +118 -14
- package/.bkit/core/engine/render/models.ts +12 -0
- package/.bkit/core/engine/shared/bkit-latest.ts +26 -4
- package/.bkit/core/engine/shared/branches.ts +4 -1
- package/.bkit/core/engine/shared/dep-updater.ts +60 -0
- package/.bkit/core/engine/shared/egress.ts +8 -1
- package/.bkit/core/engine/shared/json.ts +25 -7
- package/.bkit/core/engine/shared/package-manager.ts +9 -0
- package/.bkit/core/engine/shared/settings.ts +243 -31
- package/.bkit/core/engine/shared/text.ts +361 -0
- package/.bkit/core/engine/shared/timeouts.ts +2 -0
- package/.bkit/core/engine/tools/cli-doc.ts +111 -0
- package/.bkit/core/engine/tools/grammar-density.ts +215 -0
- package/.bkit/core/engine/tools/instructions-sources.ts +832 -0
- package/.bkit/core/engine/tools/instructions.ts +1802 -0
- package/.bkit/core/hooks/antigravity-hook.js +25 -10
- package/.bkit/core/hooks/gateguard-engine.js +54 -29
- package/.bkit/core/hooks/guardrails-shell.js +4 -5
- package/.bkit/core/hooks/guardrails-tiers.js +10 -4
- package/.bkit/core/hooks/preflight-linear.sh +9 -9
- package/.bkit/core/hooks/preflight-rules.sh +8 -0
- package/.bkit/core/hooks/readguard-engine.js +29 -10
- package/.bkit/core/hooks/session-rules-advisory.js +61 -0
- package/.bkit/core/prompts/bkit-audit-generate.md +202 -94
- package/.bkit/core/prompts/bkit-build-challenge-1.md +76 -34
- package/.bkit/core/prompts/bkit-build-challenge-2.md +85 -38
- package/.bkit/core/prompts/bkit-build-install-skills.md +50 -28
- package/.bkit/core/prompts/bkit-build-plan-convert.md +85 -0
- package/.bkit/core/prompts/bkit-build-plan-current.md +59 -36
- package/.bkit/core/prompts/bkit-build-plan-gaps-remediate.md +101 -36
- package/.bkit/core/prompts/bkit-build-plan-tasks.md +149 -0
- package/.bkit/core/prompts/bkit-build-plan-tests.md +120 -0
- package/.bkit/core/prompts/bkit-build-plan-verify.md +104 -43
- package/.bkit/core/prompts/bkit-debug-abort-handover.md +47 -32
- package/.bkit/core/prompts/bkit-debug-failure-diagnose.md +109 -57
- package/.bkit/core/prompts/bkit-debug-failure-remediate.md +65 -39
- package/.bkit/core/prompts/bkit-debug-failure-reproduce.md +53 -31
- package/.bkit/core/prompts/bkit-debug-failure-soak.md +65 -34
- package/.bkit/core/prompts/bkit-design-brainstorm-ideate-1.md +46 -31
- package/.bkit/core/prompts/bkit-design-brainstorm-ideate-2.md +45 -31
- package/.bkit/core/prompts/bkit-design-brainstorm-ideate-3.md +43 -30
- package/.bkit/core/prompts/bkit-design-generate.md +80 -46
- package/.bkit/core/prompts/bkit-design-organise.md +83 -32
- package/.bkit/core/prompts/bkit-design-review-1.md +70 -50
- package/.bkit/core/prompts/bkit-design-review-2.md +65 -46
- package/.bkit/core/prompts/bkit-design-spike.md +64 -40
- package/.bkit/core/prompts/bkit-design-update.md +53 -28
- package/.bkit/core/prompts/bkit-plan-generate.md +101 -29
- package/.bkit/core/prompts/bkit-plan-review-1.md +78 -41
- package/.bkit/core/prompts/bkit-plan-review-2.md +78 -35
- package/.bkit/core/prompts/bkit-plan-scope-check.md +116 -0
- package/.bkit/core/prompts/bkit-plan-update.md +86 -31
- package/.bkit/core/prompts/bkit-ship-conflict-resolve.md +94 -50
- package/.bkit/core/prompts/bkit-ship-deferred-sweep.md +91 -41
- package/.bkit/core/prompts/bkit-ship-update-docs.md +45 -24
- package/.bkit/core/prompts/bkit-utility-compress-markdown.md +65 -49
- package/.bkit/core/prompts/bkit-utility-update-decisions.md +87 -42
- package/.bkit/core/prompts/bkit-utility-update-deferrals.md +81 -37
- package/.bkit/core/prompts/bkit-utility-update-documentation.md +43 -25
- package/.bkit/core/prompts/bkit-utility-update-failures.md +84 -37
- package/.bkit/core/prompts/bkit-utility-update-knowledge.md +88 -42
- package/.bkit/core/prompts/bkit-utility-update-roadmap.md +92 -39
- package/.bkit/core/prompts/bkit-utility-update-tracker.md +159 -60
- package/.bkit/core/workflows/bkit-auto.md +89 -58
- package/.bkit/core/workflows/bkit-build.md +967 -517
- package/.bkit/core/workflows/bkit-debug.md +555 -387
- package/.bkit/core/workflows/bkit-design.md +319 -283
- package/.bkit/core/workflows/bkit-plan.md +319 -342
- package/.bkit/core/workflows/bkit-ship.md +525 -328
- package/.bkit/shared/.agents/plugins/bkit/hooks.json +10 -0
- package/.bkit/shared/.bkit/.env.example +2 -2
- package/.bkit/shared/.claude/bkit-cloud.md +12 -10
- package/.bkit/shared/.claude/settings.json +14 -0
- package/.bkit/shared/.codex/bkit-cloud.md +98 -0
- package/.bkit/shared/.codex/hooks.json +20 -0
- package/.bkit/shared/.cursor/bkit-cloud.md +3 -3
- package/.bkit/shared/.cursor/mcp.json +1 -1
- package/.bkit/shared/.cursor/rules/bkit-rules.mdc +1 -1
- package/.bkit/shared/.gitattributes +1 -1
- package/.bkit/shared/.mcp.json +1 -1
- package/.bkit/shared/.opencode/opencode.json +12 -1
- package/.bkit/shared/CLAUDE.md +6 -2
- package/.bkit/shared/GEMINI.md +5 -2
- package/.bkit/shared/docs/architecture.md +4 -7
- package/.bkit/shared/docs/data-schema.md +4 -7
- package/.bkit/shared/docs/deployments.md +241 -168
- package/.bkit/shared/docs/design.md +11 -12
- package/.bkit/shared/docs/noise.md +8 -7
- package/.bkit/shared/docs/observability.md +5 -8
- package/.bkit/shared/docs/security.md +35 -17
- package/.bkit/shared/docs/setup.md +29 -18
- package/.bkit/shared/docs/technology.md +3 -6
- package/.bkit/shared/docs/testing.md +289 -174
- package/.bkit/shared/gitignore +6 -2
- package/.bkit/templates/.claude/hooks/bkit-cloud-bootstrap.sh +15 -16
- package/.bkit/templates/.cursor/Dockerfile +12 -8
- package/.bkit/templates/.cursor/environment.json +1 -1
- package/.bkit/templates/.github/ISSUE_TEMPLATE/bkit-bug.yml +10 -8
- package/.bkit/templates/.github/ISSUE_TEMPLATE/bkit-change.yml +45 -12
- package/.bkit/templates/.github/ISSUE_TEMPLATE/bkit-idea.yml +3 -3
- package/.bkit/templates/.github/renovate.json5 +34 -6
- package/.bkit/templates/.github/workflows/bkit-ci.yml +54 -51
- package/.bkit/templates/.github/workflows/bkit-cloud.yml +34 -33
- package/.bkit/templates/.github/workflows/bkit-health.yml +97 -0
- package/.bkit/templates/.github/workflows/bkit-renovate.yml +5 -4
- package/.bkit/templates/.github/workflows/bkit-trivy.yml +5 -4
- package/.bkit/templates/.husky/pre-commit +2 -2
- package/.bkit/templates/.husky/pre-push +96 -36
- package/.bkit/templates/.prettierignore +6 -0
- package/.bkit/templates/.vale.ini +27 -27
- package/.bkit/templates/context/bkit-cloudrun.md +23 -15
- package/.bkit/templates/context/bkit-expo.md +70 -52
- package/.bkit/templates/context/bkit-github.md +93 -56
- package/.bkit/templates/context/bkit-inngest.md +21 -15
- package/.bkit/templates/context/bkit-maestro.md +22 -16
- package/.bkit/templates/context/bkit-playwright.md +23 -17
- package/.bkit/templates/context/bkit-resend.md +12 -10
- package/.bkit/templates/context/bkit-sentry.md +30 -20
- package/.bkit/templates/context/bkit-serena.md +8 -7
- package/.bkit/templates/context/bkit-stripe.md +14 -13
- package/.bkit/templates/context/bkit-supabase.md +16 -13
- package/.bkit/templates/context/bkit-tavily.md +14 -11
- package/.bkit/templates/context/bkit-vercel.md +27 -18
- package/.bkit/templates/docs/capabilities.md +15 -14
- package/.bkit/templates/scripts/ci/{check-accepted-divergences.mjs → bkit-check-accepted-divergences.mjs} +6 -3
- package/.bkit/templates/scripts/ci/{check-accepted-divergences.test.mjs → bkit-check-accepted-divergences.test.mjs} +3 -3
- package/.bkit/templates/scripts/ci/{check-capstone.mjs → bkit-check-capstone.mjs} +2 -2
- package/.bkit/templates/scripts/ci/{check-capstone.test.mjs → bkit-check-capstone.test.mjs} +4 -4
- package/.bkit/templates/scripts/ci/{check-dependency-audit.mjs → bkit-check-dependency-audit.mjs} +103 -18
- package/.bkit/templates/scripts/ci/{check-dependency-audit.test.mjs → bkit-check-dependency-audit.test.mjs} +31 -3
- package/.bkit/templates/scripts/ci/bkit-machine-lock.sh +387 -0
- package/.bkit/templates/scripts/ci/bkit-test-awake.mjs +75 -0
- package/.bkit/templates/scripts/ci/bkit-test-awake.test.mjs +160 -0
- package/.bkit/templates/scripts/ci/lanes/{resolve-push-base.mjs → bkit-resolve-push-base.mjs} +3 -3
- package/.bkit/templates/scripts/ci/lanes/{resolve-push-base.test.mjs → bkit-resolve-push-base.test.mjs} +4 -4
- package/.bkit/templates/scripts/cleanup/bkit-disk-prune.mjs +10 -2
- package/.bkit/templates/scripts/graphify/bkit-graph-drift.py +1 -1
- package/.bkit/templates/scripts/graphify/bkit-graph-eager.py +3 -3
- package/.bkit/templates/scripts/graphify/bkit-graph-embed.py +63 -0
- package/.bkit/templates/scripts/graphify/bkit-graph-imports.py +2 -2
- package/.bkit/templates/scripts/graphify/bkit-graph-pre-push.sh +69 -18
- package/.bkit/templates/scripts/graphify/bkit-graph-repair.py +3 -3
- package/.bkit/templates/scripts/graphify/bkit-graph-surface.py +1 -1
- package/.bkit/templates/scripts/graphify/bkit-graphify-graphml.py +1 -1
- package/.bkit/templates/scripts/graphify/bkit-graphify-rollup.py +197 -126
- package/.bkit/templates/scripts/graphify/bkit-graphify.sh +64 -41
- package/.bkit/templates/scripts/slop/bkit-slop-type-suppression.mjs +23 -7
- package/.bkit/templates/slop-scan.config.json +9 -2
- package/CHANGELOG.md +231 -0
- package/README.md +278 -89
- package/dist/checks/after/auth.js +36 -33
- package/dist/checks/after/ci-yml.js +91 -30
- package/dist/checks/after/github.js +91 -17
- package/dist/checks/after/index.js +22 -11
- package/dist/checks/after/suppression.js +8 -5
- package/dist/checks/after/tools.js +111 -114
- package/dist/checks/before/context-fetch.js +218 -33
- package/dist/checks/before/env.js +3 -3
- package/dist/checks/before/files.js +5 -3
- package/dist/checks/before/index.js +20 -3
- package/dist/checks/before/repo.js +197 -2
- package/dist/checks/status-holes.js +739 -114
- package/dist/checks/tips/catalogue.js +70 -19
- package/dist/checks/tips/detect.js +62 -0
- package/dist/checks/tips/index.js +51 -9
- package/dist/cli.js +234 -124
- package/dist/commands/analysis.js +174 -141
- package/dist/commands/ci/ci.js +302 -25
- package/dist/commands/ci/pins.js +142 -0
- package/dist/commands/ci/push-gate.js +194 -0
- package/dist/commands/comply.js +54 -91
- package/dist/commands/install/doctor.js +2 -2
- package/dist/commands/install/install.js +50 -19
- package/dist/commands/install/uninstall.js +82 -55
- package/dist/commands/install/upgrade.js +21 -11
- package/dist/commands/land.js +137 -17
- package/dist/commands/lib.js +295 -21
- package/dist/commands/registry.js +10 -2
- package/dist/commands/setup/setup.js +8 -7
- package/dist/commands/stats.js +175 -0
- package/dist/commands/tips.js +7 -3
- package/dist/commands/workflow/branches.js +1873 -152
- package/dist/commands/workflow/challenge.js +609 -0
- package/dist/commands/workflow/completeness.js +97 -0
- package/dist/commands/workflow/context.js +1045 -46
- package/dist/commands/workflow/counters.js +385 -86
- package/dist/commands/workflow/decisions.js +484 -0
- package/dist/commands/workflow/disk.js +214 -56
- package/dist/commands/workflow/fids.js +706 -33
- package/dist/commands/workflow/issues.js +349 -59
- package/dist/commands/workflow/ledgers.js +1123 -37
- package/dist/commands/workflow/locks.js +26 -2
- package/dist/commands/workflow/plan-blocks.js +501 -0
- package/dist/commands/workflow/plan-versions.js +22 -0
- package/dist/commands/workflow/plans.js +205 -0
- package/dist/commands/workflow/prs.js +154 -35
- package/dist/commands/workflow/repo.js +58 -8
- package/dist/commands/workflow/roadmap.js +58 -23
- package/dist/commands/workflow/run-facts.js +421 -0
- package/dist/commands/workflow/runs.js +670 -72
- package/dist/commands/workflow/runtime.js +15 -1
- package/dist/commands/workflow/skills.js +1 -0
- package/dist/commands/workflow/status.js +282 -34
- package/dist/commands/workflow/step-table.js +308 -0
- package/dist/commands/workflow/steps.js +703 -96
- package/dist/commands/workflow/summary-findings.js +117 -0
- package/dist/commands/workflow/summary.js +388 -0
- package/dist/commands/workflow/tasks.js +378 -0
- package/dist/commands/workflow/test-blocks.js +231 -0
- package/dist/commands/workflow/tests-changed.js +172 -0
- package/dist/commands/workflow/tests.js +1695 -305
- package/dist/commands/workflow/tracker.js +983 -0
- package/dist/commands/workflow/workflows.js +1014 -222
- package/dist/commands/workflow/worktrees.js +432 -26
- package/dist/install/cloud-env.js +233 -99
- package/dist/install/context-ci-workflow.js +85 -0
- package/dist/install/manifest.js +64 -17
- package/dist/install/modules/antigravity.js +3 -2
- package/dist/install/modules/cloud.js +25 -16
- package/dist/install/modules/codex.js +12 -2
- package/dist/install/modules/core.js +1090 -48
- package/dist/install/modules/cursor.js +6 -6
- package/dist/install/modules/opencode.js +13 -7
- package/dist/install/template-notes.js +116 -0
- package/dist/install/version-file.js +9 -4
- package/dist/ledgers/compact.js +87 -0
- package/dist/ledgers/journal.js +233 -0
- package/dist/ledgers/migrate.js +81 -0
- package/dist/ledgers/status.js +176 -13
- package/dist/ledgers/telemetry.js +1010 -125
- package/dist/render/hosts.js +104 -14
- package/dist/render/models.js +12 -0
- package/dist/shared/bkit-latest.js +26 -4
- package/dist/shared/branches.js +3 -1
- package/dist/shared/dep-updater.js +49 -0
- package/dist/shared/egress.js +8 -1
- package/dist/shared/json.js +20 -7
- package/dist/shared/package-manager.js +6 -0
- package/dist/shared/settings.js +181 -24
- package/dist/shared/text.js +333 -0
- package/dist/shared/timeouts.js +2 -0
- package/dist/tools/cli-doc.js +97 -0
- package/dist/tools/grammar-density.js +183 -0
- package/dist/tools/instructions-sources.js +720 -0
- package/dist/tools/instructions.js +1623 -0
- package/package.json +156 -51
- package/.bkit/core/.bkit/lite.md +0 -73
- package/.bkit/core/engine/commands/harvest.ts +0 -459
- package/.bkit/core/engine/commands/workflow/workstreams.ts +0 -166
- package/.bkit/core/engine/tools/wording.ts +0 -748
- package/.bkit/core/prompts/bkit-build-challenge-reconcile.md +0 -69
- package/.bkit/core/prompts/bkit-build-challenge-remediate.md +0 -71
- package/.bkit/core/prompts/bkit-build-challenge-tests.md +0 -74
- package/.bkit/core/prompts/bkit-build-plan-execute.md +0 -76
- package/.bkit/core/prompts/bkit-build-summary.md +0 -94
- package/.bkit/core/prompts/bkit-debug-summary.md +0 -96
- package/.bkit/core/prompts/bkit-design-summary.md +0 -107
- package/.bkit/core/prompts/bkit-plan-complexity-check.md +0 -67
- package/.bkit/core/prompts/bkit-plan-complexity-split.md +0 -88
- package/.bkit/core/prompts/bkit-plan-review-3.md +0 -93
- package/.bkit/core/prompts/bkit-plan-scope-check-1.md +0 -85
- package/.bkit/core/prompts/bkit-plan-scope-check-2.md +0 -85
- package/.bkit/core/prompts/bkit-plan-summary.md +0 -107
- package/.bkit/core/prompts/bkit-ship-summary.md +0 -78
- package/.bkit/templates/scripts/ci/bkit-local-stack-lock.sh +0 -194
- package/dist/commands/harvest.js +0 -353
- package/dist/commands/workflow/workstreams.js +0 -143
- package/dist/tools/wording.js +0 -680
- /package/.bkit/templates/.vale/styles/{House → Bkit}/Filler.yml +0 -0
- /package/.bkit/templates/.vale/styles/{House → Bkit}/Hedging.yml +0 -0
- /package/.bkit/templates/.vale/styles/{House → Bkit}/LLMSlop.yml +0 -0
- /package/.bkit/templates/.vale/styles/{House → Bkit}/Verbose.yml +0 -0
- /package/.bkit/templates/scripts/ci/{accepted-advisories.json → bkit-accepted-advisories.json} +0 -0
- /package/.bkit/templates/scripts/ci/{accepted-divergences.json → bkit-accepted-divergences.json} +0 -0
- /package/.bkit/templates/scripts/launchd/{launchd.env.example → bkit-launchd.env.example} +0 -0
- /package/.bkit/templates/scripts/launchd/{ai.bkit.launchd.plist → bkit-launchd.plist} +0 -0
|
@@ -5,22 +5,35 @@
|
|
|
5
5
|
# Testing
|
|
6
6
|
|
|
7
7
|
> [!CAUTION]
|
|
8
|
-
> **Purpose:**
|
|
8
|
+
> **Purpose:** The testing strategy: tier boundaries, framework choices, naming conventions and CI execution.
|
|
9
9
|
>
|
|
10
10
|
> **Rules:**
|
|
11
11
|
>
|
|
12
|
-
> -
|
|
13
|
-
> -
|
|
14
|
-
> -
|
|
15
|
-
>
|
|
16
|
-
>
|
|
17
|
-
>
|
|
12
|
+
> - Follow `.bkit/authoring.md` § All Agent-Read Files and § Docs when you edit this file.
|
|
13
|
+
> - Keep the tiers strictly separate: Static · Unit · Integration · E2E · @live · Smoke.
|
|
14
|
+
> - Pre-push runs `bkit check`, then selects one gate script in this order:
|
|
15
|
+
> - **If** `bkit ci min-push` accepts every outgoing ref → pre-push runs `pnpm bkit:gate:min`.
|
|
16
|
+
> - **If** the branch or commit is a revert → pre-push runs `pnpm bkit:gate:static`.
|
|
17
|
+
> - **If** `pnpm bkit:gate:static-unit-integration` is declared → pre-push runs it.
|
|
18
|
+
> - **Else** → pre-push runs `pnpm bkit:gate:static-unit`.
|
|
19
|
+
> - Gate scripts are consumer-owned. bkit's default is typecheck + lint + format:check + bkit:lint:prose + bkit:slop:delta + `pnpm test` (unit tier only).
|
|
20
|
+
> - Integration runs at pre-push once its gate is declared. Local services, such as a Docker database, are fine there.
|
|
21
|
+
> - E2E, @live and tests against a deployed service are excluded from pre-push. They run in CI (see below).
|
|
22
|
+
> - Never bypass the hooks with `--no-verify`.
|
|
23
|
+
|
|
24
|
+
## Local test sleep prevention
|
|
25
|
+
|
|
26
|
+
`bkit tests run`, `bkit tests changed` and the individual-file runs in `bkit tests red` / `bkit tests green` hold a macOS sleep assertion while the command runs. They use `caffeinate -is`: idle sleep is prevented on battery and AC power, and system-sleep prevention applies on AC power. The assertion ends when the command finishes or is cancelled. Other platforms run the original command.
|
|
27
|
+
|
|
28
|
+
For direct test commands and package scripts, adopt `scripts/ci/bkit-test-awake.mjs` as described in `.bkit/templates.md` § Local test sleep prevention. Wrap every test entry point and the complete gate chain. CI calls those same scripts. A raw runner command outside these entry points needs the wrapper too.
|
|
29
|
+
|
|
30
|
+
Keep the existing test timeouts. Keep the Mac's lid open; the assertion does not guarantee protection against lid closure or an explicit sleep request. The display may sleep normally.
|
|
18
31
|
|
|
19
32
|
## CI test tiers (the bkit scaffold)
|
|
20
33
|
|
|
21
|
-
CI runs tiers through bkit scaffold
|
|
34
|
+
CI runs the tiers through the bkit scaffold as one parallel lane (job) per tier, not as steps in one job. The step-by-step shape is in `.github/workflows/bkit-ci.yml` § "full CI scaffold reference", and the run-level shape is in `deployments.md` § Run-level decision matrix.
|
|
22
35
|
|
|
23
|
-
|
|
36
|
+
The graph is flat and wide for static / unit / build / integration: scope fans out, and the capstone joins. **One exception:** when `bkit-tests-migration` owns the shared-remote migration apply, e2e / live `needs:` that lane for ordering. Never gate live behind static / unit / build / e2e:
|
|
24
37
|
|
|
25
38
|
```
|
|
26
39
|
bkit-tests-scope ─┬─ bkit-tests-static
|
|
@@ -56,40 +69,52 @@ Graph **flat and wide** for static / unit / build / integration: scope fans out;
|
|
|
56
69
|
|
|
57
70
|
Adopt only lanes your repo has. Add a lane the moment its tier exists, because a tier with no lane is a tier nothing gates. The jobs above the reference in `bkit-ci.yml` are the starter: `bkit-tests-static` and `bkit-tests-unit`, joined by the `bkit-tests-check` capstone `/bkit-ship` merges on. Every lane you add joins the capstone's `needs:` and its assert. The reference is the target. A starter stub that still sits inside `bkit-tests-unit` is the pre-split shape.
|
|
58
71
|
|
|
59
|
-
Adopt in this order
|
|
72
|
+
Adopt in this order, and stop at the first tier this repo does not have. You write the step bodies. These files name the job, the `needs`, the `if`, and the rule.
|
|
60
73
|
|
|
61
74
|
1. Keep `bkit-tests-static` and `bkit-tests-unit`.
|
|
62
|
-
2. Add `bkit-tests-scope`. Every later `if:` reads `plan
|
|
75
|
+
2. Add `bkit-tests-scope`. Every later `if:` reads `plan`, and the markers are parsed once. The push base is the last green run (`deployments.md` § One run).
|
|
63
76
|
3. Move integration out of unit into `bkit-tests-integration`.
|
|
64
|
-
4. Add `bkit-tests-build-web`.
|
|
65
|
-
5. Add `bkit-tests-migration` when you have migrations.
|
|
77
|
+
4. Add `bkit-tests-build-web`. It compiles, or it creates this SHA's deployment and publishes `deployment_url` + `deployment_id`, leaving the deployment un-promoted on the default branch. Do not also enable `bkit-stub:vercel-await`.
|
|
78
|
+
5. Add `bkit-tests-migration` when you have migrations. The remote apply of new migrations (for example, `supabase db push --linked --include-all`) lives here, after the from-empty replay. On the default branch it waits for static / unit / integration / `build-web` before touching production's database. Immutability and slop-delta use the planner base on every event.
|
|
66
79
|
6. Add `bkit-tests-reset` / `-cleanup` only when two or more lanes share one long-lived resource. Otherwise provision fresh and drop both.
|
|
67
|
-
7. Add `bkit-tests-e2e-web`.
|
|
68
|
-
8. Add device build and device E2E only for a platform you ship. Same rule on a PR, a push, and a dispatch. `bkit-tests-e2e-ios` needs `build-ios` and `build-web
|
|
69
|
-
9. Add `bkit-tests-live-web` for a push to `dev` or the default branch when `web_affected`.
|
|
70
|
-
10. `bkit-tests-check` `needs:` every lane you kept.
|
|
71
|
-
11. `bkit-tests-deploy` on a default-branch push,
|
|
72
|
-
12. Health is `health.yml`, `workflow_run` after
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
- **`bkit-tests-scope`** (always-on) computes `run_level` (
|
|
77
|
-
- **`bkit-tests-static` / `-unit` / `-integration` / `-e2e-web
|
|
78
|
-
-
|
|
79
|
-
-
|
|
80
|
-
-
|
|
81
|
-
-
|
|
82
|
-
-
|
|
83
|
-
-
|
|
84
|
-
-
|
|
85
|
-
-
|
|
86
|
-
|
|
80
|
+
7. Add `bkit-tests-e2e-web`. It runs the full suite on a PR and on a push. A feature → `dev` PR hits a local stack. The target comes from `TEST_URL` only. A pass on retry is a failure.
|
|
81
|
+
8. Add device build and device E2E only for a platform you ship. Same rule on a PR, a push, and a dispatch. `bkit-tests-e2e-ios` needs `build-ios` and `build-web`. Android is the same, with its own build lane in place of `build-ios`.
|
|
82
|
+
9. Add `bkit-tests-live-web` for a push to `dev` or the default branch when `web_affected`. It never runs on a PR or on a dispatch. Do not `needs:` static, unit, integration, or e2e.
|
|
83
|
+
10. `bkit-tests-check` `needs:` every lane you kept. It is the sole required check, and its assert is a script (§ Validated means executed).
|
|
84
|
+
11. Run `bkit-tests-deploy` on a default-branch push, before the E2E / `@live` lanes. It promotes that exact `deployment_id`, and each tenant lane `needs:` it and refuses unless production serves this run's build. Not a required check (`deployments.md` § Promote before the lanes).
|
|
85
|
+
12. Health is `bkit-health.yml`, which runs as `workflow_run` after every finished CI run on the default branch. A logging probe is a step inside each runtime lane, not a job.
|
|
86
|
+
|
|
87
|
+
These files do not ship some scripts, so the plan names them and you write them: the marker parser, and lease acquire and release. The last-green push base and the capstone assert ship as templates (`.bkit/templates/scripts/ci/lanes/bkit-resolve-push-base.mjs`, copied to `scripts/ci/lanes/`; `.bkit/templates/scripts/ci/bkit-check-capstone.mjs`, copied to `scripts/ci/`).
|
|
88
|
+
|
|
89
|
+
- **`bkit-tests-scope`** (always-on) computes `run_level` (plus the optional `run_migration`) and emits the `bkit-test-plan` contract, which says what should run.
|
|
90
|
+
- **`bkit-tests-static` / `-unit` / `-integration` / `-e2e-web`:** one lane per tier, as separate parallel jobs.
|
|
91
|
+
- A platform suffix (`-web` / `-ios` / `-android`) is required when that tier exists for more than one platform. `static` / `unit` / `integration` / `migration` stay bare.
|
|
92
|
+
- Integration and web E2E gate on `web_affected` / `run_e2e`.
|
|
93
|
+
- Each lane emits its executed count for the capstone, and each coverage tier posts its own coverage comment.
|
|
94
|
+
- Start with `-static` + `-unit`, then add one lane for each tier (`deployments.md` § Parallel test lanes).
|
|
95
|
+
- Do not put the remote migration apply in E2E, because it lives in `-migration`.
|
|
96
|
+
- Web E2E on a feature → `dev` PR hits a local stack with this PR's migrations. Every other event hits the deployment.
|
|
97
|
+
- The outcome rules (retries, traces, flake handling) are the same everywhere. A pass on retry is a failure. The target comes from `TEST_URL` alone. No spec is filtered by tag.
|
|
98
|
+
- **`bkit-tests-migration`** (optional; DB repos) is an always-on ordering and provisioning lane. It does two things:
|
|
99
|
+
1. A fresh-DB replay and a seed assert, each step gated on the plan or the changed paths.
|
|
100
|
+
2. On every push to `dev`/`main`, the remote apply of new migrations (for example, `supabase db push --linked --include-all`). This step is not gated by `run_migration`, so it self-heals a remote left behind.
|
|
101
|
+
|
|
102
|
+
The lane has no job-level `if:`, because e2e / live `needs:` it, and a skip would cascade. The capstone asserts `success` whenever scope succeeded. See § Provision backing services fresh.
|
|
103
|
+
|
|
104
|
+
- **`bkit-tests-build-web`** compiles the web app, or acts as the host deployment gate (for example, on Vercel: create this SHA's deployment, publish `deployment_url` + `deployment_id`, and leave it un-promoted on the default branch). It emits no `*_ran`.
|
|
105
|
+
- **`bkit-tests-build-ios` / `-e2e-ios` / `bkit-tests-build-android` / `-e2e-android`** (optional) run the device build and the device flows. Same rule on a PR, a push, and a dispatch: these lanes are not a post-merge canary. `bkit-tests-e2e-ios` `needs:` scope, migration, reset, `bkit-tests-build-ios`, and `bkit-tests-build-web`. Android matches, with `bkit-tests-build-android`. A planned red reds the capstone. Drop any platform you do not ship. One example setup is a hosted build service plus a device-flow runner.
|
|
106
|
+
- **`bkit-tests-live-web`** runs `@live` against the deployed URL (`run_live`). Drop the job when there is no deploy. It runs on a push to `dev` or the default branch only, when `web_affected`, and never on a pull request or a dispatch. It `needs:` the planner, migration, reset and `bkit-tests-build-web`. Do not gate it behind static / unit / integration / e2e. A planned failure reds the capstone. `/bkit-ship` still filters a job named `bkit-tests-live`. This lane's name does not match it, so the capstone is the gate.
|
|
107
|
+
- **`bkit-tests-reset` / `bkit-tests-cleanup`** (optional; one shared long-lived resource): reset takes the lease and puts the resource in its known state before any lane uses it; cleanup runs last, `if: always()`, and releases the lease. Planner names at most one target. Empty target skips both. Capstone reds if a planned reset skipped or either job failed. Drop both when the resource is provisioned fresh per run (§ Provision backing services fresh — the preferred answer).
|
|
108
|
+
- **Logging probe** (an optional step, not a lane): inside each runtime lane, emit one marked event and ask the error tracker (for example, Sentry) whether it arrived. Skip when that lane did not run the probe.
|
|
109
|
+
- **`bkit-tests-deploy`** (optional) runs on a push to the default branch only, before the E2E / `@live` lanes. It promotes the exact `deployment_id` that `bkit-tests-build-web` published. Each tenant lane `needs:` it, and its first step proves (by deployment id) that production serves this run's build. The job carries its own `concurrency:` group (`cancel-in-progress: false`) and is not a required check. A red job means the previous deployment stays live.
|
|
110
|
+
- **Health** (optional, separate workflow) runs as `workflow_run` after every finished CI run on the default branch, green or red, because the deploy promotes before the tenant lanes. It probes only when production serves that run's commit. Not a required check. Copy `.bkit/templates/.github/workflows/bkit-health.yml` to `.github/workflows/bkit-health.yml` (`.bkit/templates.md` § Health check).
|
|
111
|
+
- **`bkit-tests-check`** (the capstone) `needs:` all lanes. It fails when any expected tier skipped or executed zero tests, so a green required check means the run was validated, not merely that nothing failed. It is the sole branch-protection required check.
|
|
87
112
|
|
|
88
113
|
### Accepted divergences
|
|
89
114
|
|
|
90
|
-
CI branches on the event in a few places. Each is a deliberate, recorded divergence from "one script runs everywhere"
|
|
115
|
+
CI branches on the event in a few places. Each one is a deliberate, recorded divergence from "one script runs everywhere", not an accident. Every such site is registered, so a new one is caught.
|
|
91
116
|
|
|
92
|
-
**The guard.** A workflow step, or the workflow-level `concurrency:` block (keyed `file#(workflow)#concurrency#field`), that reads an event field (`github.event_name`, `github.ref`, `github.ref_name`, `github.base_ref`, `github.head_ref`, `github.event.head_commit.message`, `github.event.pull_request.title`) must be listed in the registry (`scripts/ci/accepted-divergences.json`) with its decision and reason. The guard (`scripts/ci/check-accepted-divergences.mjs`) reds on an unregistered site, on a registered site that no longer exists, and on a decision missing from the table below. A new comparison inside an already-registered step is a new site. The guard scans `.github/workflows/*.yml` only; scripts, hooks and tests that read `CI` / `GITHUB_*` are outside it.
|
|
117
|
+
**The guard.** A workflow step, or the workflow-level `concurrency:` block (keyed `file#(workflow)#concurrency#field`), that reads an event field (`github.event_name`, `github.ref`, `github.ref_name`, `github.base_ref`, `github.head_ref`, `github.event.head_commit.message`, `github.event.pull_request.title`) must be listed in the registry (`scripts/ci/bkit-accepted-divergences.json`) with its decision and reason. The guard (`scripts/ci/bkit-check-accepted-divergences.mjs`) reds on an unregistered site, on a registered site that no longer exists, and on a decision missing from the table below. A new comparison inside an already-registered step is a new site. The guard scans `.github/workflows/*.yml` only; scripts, hooks and tests that read `CI` / `GITHUB_*` are outside it.
|
|
93
118
|
|
|
94
119
|
<!-- prettier-ignore -->
|
|
95
120
|
| Decision | Rule |
|
|
@@ -113,7 +138,7 @@ CI branches on the event in a few places. Each is a deliberate, recorded diverge
|
|
|
113
138
|
|
|
114
139
|
### Which resources may each lane touch?
|
|
115
140
|
|
|
116
|
-
Every lane must
|
|
141
|
+
Every lane must declare what it may reach, and the lane's env enforces the declaration, not a comment. A lane that can reach a shared remote instance eventually will, and the result is cross-run interference, not a failure.
|
|
117
142
|
|
|
118
143
|
<!-- prettier-ignore -->
|
|
119
144
|
| Lane | Real external APIs | Stateful backing service (DB / cache / queue) | Deployed app |
|
|
@@ -123,44 +148,64 @@ Every lane must **declare** what it may reach; declaration enforced by lane env
|
|
|
123
148
|
| `bkit-tests-build-web` | none | none | creates this SHA's deployment; does not promote it |
|
|
124
149
|
| `bkit-tests-build-ios` / `-android` | the build host (e.g. a device-build service) | none | none |
|
|
125
150
|
| `bkit-tests-integration` | none — **mocked** | **ephemeral, CI-local, fresh each run** | none |
|
|
126
|
-
| `bkit-tests-migration` | none | **ephemeral, replayed from
|
|
151
|
+
| `bkit-tests-migration` | none | **ephemeral, replayed from empty** (plan- / path-gated) + **shared remote push** on every `dev`/`main` push | none |
|
|
127
152
|
| `bkit-tests-reset` / `-cleanup` | none | the **one** shared long-lived resource this run named | none |
|
|
128
153
|
| `bkit-tests-e2e-web` | none — **mocked** | ephemeral **local** (feature → `dev` PR) / the leased shared one (other events) | this SHA's deployment |
|
|
129
154
|
| `bkit-tests-e2e-ios` / `-android` | the device host | the leased shared resource | the deployed backend |
|
|
130
155
|
| `bkit-tests-live-web` | **real** | the deployed environment's **real** instance | **real** deployed URL |
|
|
131
156
|
| `bkit-tests-deploy` | the host API | none | promotes the deployment the tenant lanes then test |
|
|
132
157
|
|
|
133
|
-
Read table as rule
|
|
158
|
+
Read the table as a rule. `bkit-tests-live-web` and the device E2E lanes are the only lanes that may spend real external quota. Only those lanes, plus push-side `-e2e-web`, may touch the shared long-lived resource, and only between reset and cleanup. Every other lane provisions its own (§ Provision backing services fresh).
|
|
134
159
|
|
|
135
160
|
> [!IMPORTANT]
|
|
136
|
-
> **"Real external API" and "real remote backing service" are two
|
|
137
|
-
>
|
|
161
|
+
> **"Real external API" and "real remote backing service" are two independent axes — never one flag.**
|
|
162
|
+
> This is the most expensive mistake in this doc. A single overloaded gate (`LIVE_API`, `TEST_REAL`, …) that means both "hit real third-party APIs" and "hit the real remote database" collapses unrelated needs. Suites that need only the real DB (no external API) then sit in the serial, push-only `@live` lane, where they are invisible on PRs and inflate the gating lane. Moving them to `-integration` (ephemeral local DB) shrinks `@live` and gives them PR coverage. Gate each axis on what it needs: an external-boundary test on an API credential, and a DB test on a DB URL. Prefer an ephemeral DB, which needs no gate.
|
|
138
163
|
|
|
139
|
-
**Moving suite between lanes
|
|
164
|
+
**Moving a suite between lanes takes three edits, and the third is the one that bites.** Retag the file, move it to the new lane, and delete the old gate. A suite with `skipIf(!LIVE_API)` in a lane that does not set `LIVE_API` silently skips forever: the lane is green, the count looks plausible, and the coverage is gone (a hollow green through the back door, § Validated means executed). After a lane move, assert that the destination's executed count went up by the moved test count.
|
|
140
165
|
|
|
141
|
-
**
|
|
166
|
+
**A missing gate variable must give an honest, detectable skip, never a silent pass.** A tier that needs a URL or credential the run lacks must report itself skipped, not exit 0 having run nothing. **If** that tier was planned → it must also fail the capstone. Where it matters, add a per-file assert, for example (Vitest JSON) `numPassedTests >= 1 && numPendingTests === 0`. It catches both a ghost-green no-op and a quiet skip.
|
|
142
167
|
|
|
143
|
-
**
|
|
168
|
+
**A missing tool that a test needs (binary, CLI, interpreter) must fail loudly, never `return` early.** Resolve the tool through one shared test helper that throws when it is absent (for example, `requireTsx()`), with no per-file check. An early `return` is a pass that asserted nothing. A self-skip on an absent tool (`skipIf(!hasTool)`) must not skip on CI. Install the tool on that lane, or fail when `CI` is set. Otherwise the lane skips the same tests on every run and stays green.
|
|
144
169
|
|
|
145
170
|
### Validated means executed, not "didn't fail"
|
|
146
171
|
|
|
147
|
-
|
|
172
|
+
A green capstone means the planned tiers ran tests and passed, not merely that nothing visibly failed. Each tier is in one of three states, and each state is gated independently:
|
|
173
|
+
|
|
174
|
+
- **Absent:** no declared entrypoint (for example, a `test:integration` / `test:live` script, or an enabled E2E step). The capstone asserts nothing. A repo with no integration / E2E / `@live` tier is fully valid.
|
|
175
|
+
- **Declared + planned to run:** assert that the tier ran and executed > 0 tests. This closes the green-with-zero hole, where everything exits 0 and looks green: `--passWithNoTests` matching nothing, a `@live`-tagged suite whose owning package declares no live entrypoint (for example, no `test:live` script, so `turbo run test:live` runs nothing), or every test gated off by `skipIf`.
|
|
176
|
+
- **Declared + correctly skipped** (a `partial` run, or not `web_affected`): assert that it was skipped, not vacuously passed.
|
|
148
177
|
|
|
149
|
-
|
|
150
|
-
- **Declared + planned to run** — assert the tier ran **and** executed **> 0** tests. This closes the green-with-zero hole: `--passWithNoTests` matching nothing, a `@live`-tagged suite whose owning package declares no live entrypoint (e.g. no `test:live` script, so `turbo run test:live` runs nothing), or every test `skipIf`-gated off, so all exit 0 and look green.
|
|
151
|
-
- **Declared + correctly skipped** (light run / not `web_affected`) — assert it was skipped, not vacuously passed.
|
|
178
|
+
A declared tier therefore emits an **executed count**, taken from the runner's machine summary (for example, vitest `numPassedTests + numFailedTests`). The capstone asserts `count > 0` for every declared-and-planned tier. At install time, the same rule reads "every tagged suite has a CI home": each `@live`-tagged test must live in a package whose declared entrypoint (for example, a `test:live` script) the tier actually invokes. Otherwise the tag is dead code.
|
|
152
179
|
|
|
153
|
-
|
|
180
|
+
**The assert is a script, not a heredoc in the workflow.** It is the only required check, and a shell heredoc inside YAML cannot be unit-tested. Move the decision into a script that the normal test command already runs, with the environment as an argument. Four cases must red:
|
|
154
181
|
|
|
155
|
-
|
|
182
|
+
- a lane that failed
|
|
183
|
+
- a gated lane that ran when its gate said skip
|
|
184
|
+
- a tier that reports success with zero tests
|
|
185
|
+
- a plan that will not parse
|
|
186
|
+
|
|
187
|
+
The checkout that fetches that script is ungated and not `continue-on-error`. A checkout gated on `run_level == 'full'` leaves every other run with no script, and the required check then exits on "cannot find module" without reading the plan. bkit ships the script as `.bkit/templates/scripts/ci/bkit-check-capstone.mjs` with its test `bkit-check-capstone.test.mjs`. The script takes the environment in, and puts `::error::` / `::warning::` lines and an exit code out. A consumer copies both files with `bkit-ci.yml`. When the script is missing, the capstone step reds with a copy-the-script message, and never skips.
|
|
156
188
|
|
|
157
189
|
**Namespace.** `bkit-tests-scope` is the planner. A loop over every `bkit-tests-*` job skips it, and a scope failure then leaves the capstone green. Exempt `bkit-tests-scope`, `bkit-tests-reset`, and `bkit-tests-cleanup` by name from any guard that treats every `bkit-tests-*` job as a counting lane. Point branch protection at `bkit-tests-check`.
|
|
158
190
|
|
|
159
|
-
**A per-tier count
|
|
191
|
+
**A per-tier count will not catch a file that stops running everywhere.** `count > 0` proves each tier ran tests, but it does not prove every test file ran in some lane. A file renamed out of its glob, moved to a lane whose filter excludes it, or left carrying a stale `skipIf` (the lane-move trap above) can quietly stop executing in every lane, while each lane stays green and each count still looks plausible. The capstone is the only job that `needs:` every lane, so it is the only place to answer this, on the union: reconcile the on-disk test-file inventory against the set of files that executed across all lanes. A file skipped in one lane but run in another is fine. Only a file that is dead in every lane is the bug.
|
|
192
|
+
|
|
193
|
+
- Its checkout stays `continue-on-error`, and must not be the script checkout above.
|
|
194
|
+
- Keep it a non-blocking diagnostic. It must never red the required check.
|
|
195
|
+
- Run it on full runs only. A PR's unit lane is affected-scoped and `bkit-tests-live-web` is absent, so the PR delta is expected and large, and a guard that cries wolf gets turned off.
|
|
196
|
+
- Start it report-only. Flip it to an assert once the allowlist of files that legitimately never run is seeded from a real inventory.
|
|
197
|
+
|
|
198
|
+
**Executed is not the same as exercised.** `count > 0` proves a test ran, but it does not prove the test reached the behaviour it claims to cover. A **hollow boundary test** is non-mock, owns a CI home, executes and passes, and can still cover nothing when it short-circuits before the boundary. A worked example: a golden test for an external OCR lane called the extractor with the wrong argument shape. The input field was `undefined`, the extractor returned `{ text: "", extractor: "none" }` in ~2ms, no API call was ever made, and the assertion (on a field that did not exist) was never reached. The test was green in CI and covered nothing.
|
|
199
|
+
|
|
200
|
+
A boundary test must therefore assert a ground-truth signal that only the real boundary can produce: a specific returned value, shape, identity or latency floor (for example, `extractor === "mistral-ocr"` and `text.length > 50`). Never assert "did not throw", or a value that a degenerate or short-circuit path could also return. Two corollaries:
|
|
160
201
|
|
|
161
|
-
|
|
202
|
+
1. When a deploy / `@live` tier exists, route external and deployed-boundary behaviours (3rd-party API, model, payment, deployed schema, RLS / tenancy) to `@live`, because a passing local fixture is not evidence that prod is correct (§ Schema-invariant DB checks).
|
|
203
|
+
2. Type-system escapes at a boundary call-site are forbidden. Examples are TS `as unknown as` / `as any` casts, the `@ts-ignore` / `@ts-nocheck` / `@ts-expect-error` directives, and Python `# type: ignore`. They suppress the contract drift (a renamed SDK field, a changed signature) that the boundary test exists to catch, so the bug ships while the test passes against the wrong shape.
|
|
162
204
|
|
|
163
|
-
**Never manufacture a vacuous green.** A pass-with-no-tests option (
|
|
205
|
+
**Never manufacture a vacuous green.** A pass-with-no-tests option (for example, `passWithNoTests: true` / `--passWithNoTests`) makes a runner report PASS with zero tests run. That is the exact hole the executed-count gate exists to close, so it is a **banned override** in any bkit-managed test config. Two adjacent guards:
|
|
206
|
+
|
|
207
|
+
- Emit the executed count as a build output, not a log line, so a build-cache replay (for example, from a task runner like turbo) restores the real number. Let a cache short-circuit report a `cached` sentinel (validated on cache), not a hollow 0.
|
|
208
|
+
- Assert unit / integration counts only on non-PR full runs, because a PR run is affected-scoped and can legitimately resolve to 0 affected tests.
|
|
164
209
|
|
|
165
210
|
### Step names say whether a red is real
|
|
166
211
|
|
|
@@ -174,7 +219,7 @@ One verb scheme across every step in `bkit-ci.yml`, so a red run can be triaged
|
|
|
174
219
|
| `Run` | invokes a named tool or suite |
|
|
175
220
|
| `Report` / `Upload` / `Count` / `Merge` / `Fetch` | move or publish results |
|
|
176
221
|
|
|
177
|
-
`Guard`, `Validate`, `Verify` and `Audit` are
|
|
222
|
+
`Guard`, `Validate`, `Verify` and `Audit` are not in the scheme: each means "fail the job if this isn't true", which is `Assert`. `Check` is the load-bearing half: it is reserved for a step that genuinely cannot red the lane, either because it carries `continue-on-error: true` or because its body only ever warns. A step named `Check` that can fail the build is the one naming error this scheme exists to prevent, because it trains you to skim past a real failure.
|
|
178
223
|
|
|
179
224
|
Two smaller rules that follow from the same idea:
|
|
180
225
|
|
|
@@ -183,56 +228,60 @@ Two smaller rules that follow from the same idea:
|
|
|
183
228
|
|
|
184
229
|
### A guard asserts the rule, not the sentence that explains it
|
|
185
230
|
|
|
186
|
-
A shape guard over a config file reads that file as text, so it is easy to anchor it on whatever string happens to be nearby. Anchor on the
|
|
231
|
+
A shape guard over a config file reads that file as text, so it is easy to anchor it on whatever string happens to be nearby. Anchor it on the rule. A guard pinned to the prose around the rule fails the moment someone improves the comment, and passes for years after the rule itself is deleted.
|
|
187
232
|
|
|
188
|
-
|
|
233
|
+
For example, a guard asserting `expect(content).toMatch(/bootstrap paradox/)` to protect a real invariant (a lane checks the repo out before it calls a local composite) goes red when someone tidies the phrase out of the comment, while the invariant still holds. Assert the thing that must hold (`must already be checked out`, a step id, a flag, an `if:` shape), not the words chosen to describe it.
|
|
189
234
|
|
|
190
235
|
Three practical rules:
|
|
191
236
|
|
|
192
237
|
- **Pin the mechanism, not the wording.** `expect(step).not.toMatch(/exit [1-9]/)` proves a step is non-blocking; matching a comment that says "this only warns" proves nothing about the code.
|
|
193
|
-
- **Derive the list, never hand-write it.** A guard that enumerates lanes goes stale the day a lane is added. Derive the set from the file under test
|
|
238
|
+
- **Derive the list, never hand-write it.** A guard that enumerates lanes goes stale the day a lane is added. Derive the set from the file under test (count the `VERBOSE_CI` declarations, or read the `needs:` array), so a new lane is covered without editing the guard.
|
|
194
239
|
- **Prove it can fail.** Put the bug back and watch the guard red before you trust it. A guard that has never failed is indistinguishable from a guard that cannot.
|
|
195
240
|
|
|
196
|
-
**The exception, stated so it is not mistaken for the rule.** When the shipped artefact is prose
|
|
241
|
+
**The exception, stated so it is not mistaken for the rule.** When the shipped artefact is prose (a rules file an agent reads, a CLI help string, an advisory message), the wording is the behaviour, and asserting on it is correct. The distinction is whether the text is the deliverable or a description of one.
|
|
197
242
|
|
|
198
243
|
### Composite actions: two rules that fail silently
|
|
199
244
|
|
|
200
245
|
A composite action (`uses: ./.github/actions/setup`) is the right way to stop copy-pasting a setup block across lanes. Two of its rules are not obvious and neither fails loudly:
|
|
201
246
|
|
|
202
|
-
- **A composite cannot own the checkout that fetches it.** A local `uses: ./…` path is resolved from the runner's own filesystem, so the repo must already be checked out before the runner can read the composite's `action.yml` at all. Putting `actions/checkout` inside the composite and making the composite each lane's first step is a bootstrap paradox, so the lane reds with "can't find action.yml". Every adopting lane runs `actions/checkout` as its
|
|
203
|
-
- **Job-level `env:` reaches a composite step's process, but not its `env` expression context.** The `env` context inside a composite reflects only what the composite itself declares
|
|
247
|
+
- **A composite cannot own the checkout that fetches it.** A local `uses: ./…` path is resolved from the runner's own filesystem, so the repo must already be checked out before the runner can read the composite's `action.yml` at all. Putting `actions/checkout` inside the composite and making the composite each lane's first step is a bootstrap paradox, so the lane reds with "can't find action.yml". Every adopting lane runs `actions/checkout` as its own first step, with its own explicit `fetch-depth`, before it calls the composite.
|
|
248
|
+
- **Job-level `env:` reaches a composite step's process, but not its `env` expression context.** The `env` context inside a composite reflects only what the composite itself declares. `if: env.VERBOSE_CI == '1'` on a composite step is therefore always false: the step silently never runs, and nothing reds. Gate in the shell instead (`if [ "${VERBOSE_CI:-}" != "1" ]; then exit 0; fi`), and say so in a comment, or the next person tidies it back into an `if:`.
|
|
204
249
|
|
|
205
|
-
The same expression-context trap has a neighbour outside composites
|
|
250
|
+
The same expression-context trap has a neighbour outside composites. The `env` context is not available to a job-level `env:` value either, so `NODE_OPTIONS: ${{ env.VERBOSE_CI && '…' || '' }}` resolves empty on every run. Spell the marker test out (see § CI verbosity).
|
|
206
251
|
|
|
207
252
|
### E2E on PRs: run the full suite, watch which DB it hits
|
|
208
253
|
|
|
209
|
-
|
|
254
|
+
There are two PR-time E2E concerns. Both are gated on real preconditions (event, head ref), never on a kill-switch flag:
|
|
210
255
|
|
|
211
|
-
- **Parity — PR and push run the full suite.** `pull_request` runs the
|
|
212
|
-
- **Correctness — migration-coupled PRs.**
|
|
256
|
+
- **Parity — PR and push run the full suite.** `pull_request` runs the same E2E spec set as the `dev`/`{{DEFAULT_BRANCH}}` push. Scale a slow suite by sharding (parallel lanes, more workers). PR and push runs differ only in their DB target (below) and in push-only tiers (for example, `@live`).
|
|
257
|
+
- **Correctness — migration-coupled PRs.** When preview deploys share one `dev` database, that database does not yet carry a PR's unmerged migration, because the migration applies only on the post-merge push. A PR that adds or renames a column and also uses it therefore fails with a 500 every time before the merge. A force-all-E2E marker does not help, because it hits the same migration-less DB.
|
|
258
|
+
- On feature→`dev` PRs, run E2E against a local DB that has applied this PR's migrations: reset local → prod build → run against `localhost`.
|
|
259
|
+
- Keep preview-backed E2E for `dev`/`main` pushes, for deploy fidelity.
|
|
260
|
+
- The `dev`→`main` release PR (head ref `dev`) already targets the migrated `dev` DB, so it stays on the remote.
|
|
261
|
+
- A repo with more than one migration system must apply all of them locally.
|
|
213
262
|
|
|
214
263
|
### Concurrent invocations collide on disk, not just in the database
|
|
215
264
|
|
|
216
|
-
Per-shard data isolation (§ Optional hardening) is necessary,
|
|
265
|
+
Per-shard data isolation (§ Optional hardening) is necessary, but not sufficient. When one job launches several runner processes at once, anything keyed on the runner's own worker index collides, because that index restarts at 0 in every process.
|
|
217
266
|
|
|
218
|
-
|
|
267
|
+
For example, when one E2E step fans out over several concurrent Playwright invocations and each writes its minted session into one shared auth dir as `user-w<N>.json` (`<N>` = `workerInfo.parallelIndex`), every invocation has a worker 0, so several of them write `user-w0.json`. The later write wins, and browsers load sessions their own worker never built, which surfaces as `Cannot read properties of undefined` errors and sign-in redirects. DB rows with random owners do not collide, so the collision is purely on disk. A cross-tenant bleed tripwire cannot see it, because a clobbered file still holds a valid identity.
|
|
219
268
|
|
|
220
|
-
- **Make the index global.**
|
|
221
|
-
- **Throw on a bad value, never clamp.** An index silently
|
|
222
|
-
- **Assert distinctness, not presence.** A guard
|
|
223
|
-
- **
|
|
224
|
-
- **Test the loop by running it.**
|
|
269
|
+
- **Make the index global.** Give each invocation a distinct lane index, and use `laneIndex * workersPerLane + parallelIndex`. When the lane index is unset, the offset is 0, so local runs stay byte-identical.
|
|
270
|
+
- **Throw on a bad value, never clamp.** An index that silently reads as 0 is the defect being prevented.
|
|
271
|
+
- **Assert distinctness, not presence.** A guard that checks each invocation has an index still passes when two lanes share 0, which restores the bug.
|
|
272
|
+
- **Keep bands unique, never dense.** A `--workers=1` lane leaves most of its band unused, and that waste is the point. Tightening the spacing to real worker counts is how bands overlap again.
|
|
273
|
+
- **Test the loop by running it.** A static scan cannot see loop-generated invocations. Execute the real `run:` body in the guard: a control run, plus mutations (a pinned lane index, a stripped cache-dir suffix) that must red.
|
|
225
274
|
|
|
226
|
-
|
|
275
|
+
The same rule applies to every per-worker artefact keyed that way: transform caches, blob report dirs, JSON output names and artifact dirs. The fix also removes the reason to serialise the lanes at all.
|
|
227
276
|
|
|
228
|
-
Two further axes
|
|
277
|
+
Two further axes share the same root cause:
|
|
229
278
|
|
|
230
|
-
- **Shared identity vs per-principal rate limits.**
|
|
231
|
-
- **Shared service load.** A per-navigation auth call
|
|
279
|
+
- **Shared identity vs per-principal rate limits.** Several lanes driving one fixture owner create more records in a minute than the product allows one user, so the lanes past the per-user limit get a 429. The limiter is working, and the fixture is pretending several lanes are one person. Give each lane its own fixture identity under a suffix, make teardown delete by id, and guard the fixture build separately from the lanes.
|
|
280
|
+
- **Shared service load.** A per-navigation auth call under a wide fan-out turns into thousands of `GET /auth/v1/user` requests in minutes, which stalls the provider and reddens lanes on other machines that share it. Before fanning out, audit per-request calls to shared external services. Do the local check first (JWKS). Call the service only for the cases that check cannot decide. This is a different lever from the phase-split one (`deployments.md` § Parallel test lanes): it cuts per-request cost, not peak concurrency.
|
|
232
281
|
|
|
233
282
|
## Which tier does a behaviour need?
|
|
234
283
|
|
|
235
|
-
Assign every behaviour its **lowest sufficient tier(s)
|
|
284
|
+
Assign every behaviour its **lowest sufficient tier(s)**: the lowest tier that actually exercises it. A behaviour may need more than one. Never substitute a lower or mocked tier for a boundary that a higher tier validates. **Under-tiering** is a defect, not a shortcut. It means using a lower or mocked tier where the rubric demands a higher or `@live` one, for example a user-visible flow with no E2E row, an external-API behaviour tested only against a local fixture, or a migration with no fresh-DB replay.
|
|
236
285
|
|
|
237
286
|
<!-- prettier-ignore -->
|
|
238
287
|
| Behaviour | Required tier |
|
|
@@ -240,93 +289,122 @@ Assign every behaviour its **lowest sufficient tier(s)** — the lowest tier tha
|
|
|
240
289
|
| Any change (every commit) | **Static** — type / lint / format / prose / drift; pre-push + CI fail-fast |
|
|
241
290
|
| Pure function / single module, no I/O | **Unit** |
|
|
242
291
|
| Crosses a LOCAL stateful boundary (own DB / own API, no 3rd party) | **Integration** vs a freshly-provisioned instance, no mocks at the boundary |
|
|
243
|
-
| Crosses an
|
|
292
|
+
| Crosses an external / deployed boundary (3rd-party API, model / LLM, payment, deployed schema, RLS / tenancy) | **@live (golden)** — reaches the real boundary, asserts a boundary-only signal |
|
|
244
293
|
| User-visible UI flow / DOM state | **E2E** |
|
|
245
294
|
| Design token value / reference-doc parity | **Unit** — reference doc vs each implementation, asserted independently (§ Design assertions) |
|
|
246
295
|
| Rendered geometry, contrast, touch-target size | **E2E** — computed style + a11y scan in the existing browser project (§ Design assertions) |
|
|
247
296
|
| New / changed migration or seed | **fresh-DB replay** (`bkit-tests-migration`) — replays from empty + the seed loads |
|
|
248
297
|
| Post-deploy health of a deployed surface | **Smoke** (post-CI or post-deploy — see `deployments.md`) |
|
|
249
298
|
|
|
250
|
-
A plan's `Derived Test Matrix` (`docs/templates/bkit-plan.md`)
|
|
299
|
+
A plan's `Derived Test Matrix` (`docs/templates/bkit-plan.md`) applies this rubric to one feature: one row per behaviour, each mapped to a build + verify task. Boundary rows carry the rigor above: reach the boundary, assert a signal only it can produce, and use no `as unknown as` at the call-site.
|
|
251
300
|
|
|
252
301
|
## Test first
|
|
253
302
|
|
|
254
|
-
`/bkit-build` writes each
|
|
303
|
+
`/bkit-build` writes each group's tests before its code and proves them: one subagent writes the planned tests and may touch nothing else, the engine records each one failing (red), and only then does another subagent write the code, which may touch no planned test, until each passes (green). This covers the unit and integration tiers only. E2E and `@live` rows run in CI and stay planned in the `Derived Test Matrix`. Each plan task names its tests on a `**TEST FIRST:**` field, one sub-bullet per test, or `n/a — {reason}`.
|
|
304
|
+
|
|
305
|
+
- **Declare `bkit:test:file`:** it runs the one test file bkit appends as its argument (a repo-relative path), and writes a JUnit XML report to the path in `$BKIT_TEST_REPORT`. Vitest: `vitest run --reporter=default --reporter=junit --outputFile.junit="$BKIT_TEST_REPORT"`. pytest: `pytest --junitxml="$BKIT_TEST_REPORT"`. jest needs the `jest-junit` reporter.
|
|
306
|
+
- **If** it is undeclared → the build reports test-first as not enforced.
|
|
307
|
+
- **If** the test files belong to more than one runner (for example, `.bats` and `node:test`) → point it at a script that picks the runner by file type.
|
|
308
|
+
- **If** `$BKIT_TEST_REPORT` is unset → pass no report flag. An empty `--outputFile.junit=` takes the next argument, the test file, as the report path, and overwrites it.
|
|
309
|
+
- **Declare `bkit:lint:file`:** it lints the one test file bkit appends (a repo-relative path) with the static gate's linter config, and exits 0 when clean. ESLint: `eslint --max-warnings=0`. Ruff: `ruff check`.
|
|
310
|
+
- **If** the linter does not cover the test file (for example, `.bats`) → exit 0 (Oxlint: `--no-error-on-unmatched-pattern`).
|
|
311
|
+
- The build lints each planned test file before running it for its red. **If** `bkit:lint:file` is undeclared → the red is recorded unlinted, and a lint rewrite after it changes a frozen test file, which the code step's finish refuses.
|
|
312
|
+
- **Call the runner directly:** a task runner that caches results, or runs dependent packages' suites first, breaks the one-file contract. A monorepo routes the path to the package and config that own it. An integration file gets the same environment setup that the integration suite gets.
|
|
313
|
+
- **Judged by the report, never the exit code:** a misspelled path, a syntax error and an import of unwritten code all exit non-zero, just like a real failure. A red counts when the named test ran and failed, or when its file failed to load with no case of its own in the report (the tests step writes no source, so an import of unwritten code is the expected shape of a red); a green only when it ran and passed.
|
|
314
|
+
- **Test name:** use the name as the runner reports it, or its last segment (`clamp > keeps a value inside the range` → `keeps a value inside the range`).
|
|
315
|
+
- **The red test stays:** after the red, the test may gain tests or formatting. A weakened, removed or rewritten assertion fails the build's verify.
|
|
316
|
+
|
|
317
|
+
## Tests lock
|
|
318
|
+
|
|
319
|
+
Wrap each test-running `bkit:gate:*` script with the adopted machine lock (`.bkit/templates.md` § Machine lock). The wrapped script holds the `tests` lock for the whole gate, across repositories. An unwired repository takes no lock.
|
|
320
|
+
|
|
321
|
+
- `bkit tests run` and `bkit tests changed` pass `--lock-wait-s` as `LOCAL_STACK_LOCK_WAIT_S` and their run name as `LOCAL_STACK_LOCK_WHO`; they run the declared gate without executing the lock script itself.
|
|
322
|
+
- `bkit branches push` passes the same wait budget and its run name to Git. When its pre-push hook's gate gives up on the tests lock, it exits 81 with `lockHolder` and reports nothing pushed; the workflow ends the turn and re-runs the block.
|
|
323
|
+
- A tests-lock give-up of this run becomes exit 81 with `lockHolder`, and a tests-lock refusal becomes exit 4. A lock wait writes no `test-run` status row. A gate's own failure or a nested lock's give-up remains a test failure. A raw gate caller sees exit 75 and the holder line.
|
|
324
|
+
- Light runs (`bkit tests red`, `green` and `bkit:test:file`) take no tests lock. Nested wraps reuse the live parent's tests lock. `CI=true` / `CI=1` bypasses locking.
|
|
325
|
+
- `BKIT_TEST_SLOTS` sets the number of tests slots (default 1). The lock exports `BKIT_TEST_WORKERS` for the consumer to pass to its runner.
|
|
326
|
+
- The CLI wait defaults to `BKIT_LOCK_WAIT_S`, else 60 seconds; `--lock-wait-s=0` tries once. Claude Code ships `BKIT_LOCK_WAIT_S=1200`; other hosts use the shorter wait and re-run on exit 81 (`.bkit/providers.md` § Long-running shell commands).
|
|
327
|
+
- The script defaults to 60 seconds without a TTY when no wait is set. Waiting notes name the holders and their checkouts' earlier run times.
|
|
255
328
|
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
- **
|
|
261
|
-
- **
|
|
329
|
+
## JUnit reports for bkit
|
|
330
|
+
|
|
331
|
+
`bkit tests run` counts a gate's failures from JUnit XML, not from its log. It empties a report folder before each run and passes its absolute path to the gate as `$BKIT_TEST_REPORT_DIR`.
|
|
332
|
+
|
|
333
|
+
- **Every runner in a gate writes JUnit there** when the variable is set: one file per runner or per run, `*.xml`, with any name. Two runs in one gate never write the same file name.
|
|
334
|
+
- **Counted:** failing plus erroring test cases across every file. Skipped cases do not count.
|
|
335
|
+
- **No reports:** `failures` is `null` (count unknown), and a hint says to turn the reporters on. The verdict still comes from the gate's exit code.
|
|
336
|
+
- **Unset variable:** the runner behaves as it always does, so a gate run by hand writes nothing extra.
|
|
262
337
|
|
|
263
338
|
## What each tier must do
|
|
264
339
|
|
|
265
|
-
|
|
340
|
+
Each tier lists the rule to follow, then a **Not:** line naming the failure the rule prevents. Stack tools (Vitest / Playwright / a Supabase-style local stack / a Vercel-style preview) appear only as worked examples, because the rule is the principle. Implement it for your stack.
|
|
266
341
|
|
|
267
342
|
### Static
|
|
268
343
|
|
|
269
|
-
- Run the whole non-test gate set: typecheck · format check · lint · prose lint · code-slop scan (full
|
|
270
|
-
- Lint covers
|
|
271
|
-
- Scan a
|
|
272
|
-
- On PRs scope to the
|
|
273
|
-
- **Not:** a green CI that never ran the slop / drift / prose gates because they sat behind a test step that exited early.
|
|
344
|
+
- Run the whole non-test gate set: typecheck · format check · lint · prose lint · code-slop scan (full, plus a PR delta gate that fails only on net-new findings) · drift check · workflow linter. Locally this set is one script, `bkit:gate:static`, and `bkit:gate:static-unit` chains the unit tier on top. Pre-push, the build loop and the remediate agents all run these same scripts, so a green push and a green agent gate mean the same thing. CI runs the rest fail-fast, before any env pull or heavy tier.
|
|
345
|
+
- Lint covers test files too, with rules that catch tests that cannot fail: no assertion, a conditional inside a test, a conditional `expect`, an unawaited promise. The linter is consumer-owned (for example, Vitest `expect-expect`, `no-conditional-in-test`, `no-conditional-expect`, `valid-expect`, and type-aware `no-floating-promises`). An optional JS / TS starting point is `.bkit/templates/.oxlintrc.json`.
|
|
346
|
+
- Scan a clean tree. Run static before the test tiers write coverage or result artifacts, or the scan flags the artifacts instead of your code.
|
|
347
|
+
- On PRs, scope to the affected graph (for example, a monorepo task runner's `--filter=...[base]`) over a non-shallow checkout (full history). Never use a shallow clone, because its affected-selection fails open to "everything".
|
|
348
|
+
- **Not:** a green CI that never ran the slop / drift / prose gates, because they sat behind a test step that exited early.
|
|
274
349
|
|
|
275
350
|
### Unit
|
|
276
351
|
|
|
277
|
-
-
|
|
278
|
-
- **Not:** a "unit" test that opens a DB connection or hits the network
|
|
352
|
+
- Test pure functions and single-module logic, in-process, with no I/O. Mock or inject the slow or side-effecting collaborators.
|
|
353
|
+
- **Not:** a "unit" test that opens a DB connection or hits the network. That is an integration or @live behaviour wearing a unit suffix, and it will flake.
|
|
279
354
|
|
|
280
355
|
### Integration
|
|
281
356
|
|
|
282
|
-
- Cross a
|
|
283
|
-
-
|
|
284
|
-
- Design for **unique-ID independence**: each case creates uniquely-keyed data and never assumes a clean shared store, so the suite parallelises. Transaction-rollback isolation is a pgTAP-style fast
|
|
285
|
-
- The DB boundary stays hermetic
|
|
286
|
-
-
|
|
287
|
-
- **Not:** an "integration" test green against a long-lived shared dev DB carrying another run's rows
|
|
357
|
+
- Cross a local stateful boundary (your own DB / API) against a freshly-provisioned instance, with no mocks at the boundary (§ Provision backing services fresh).
|
|
358
|
+
- Self-seed each test's own fixtures (for example, in `beforeAll`). Do not assert on rows the seed file plants. Coupling to the shared seed makes the tier brittle to seed edits and lets seed regressions hide, because the seed has its own gate (§ Migration-replay + seed).
|
|
359
|
+
- Design for **unique-ID independence**: each case creates uniquely-keyed data and never assumes a clean shared store, so the suite parallelises. Transaction-rollback isolation is a pgTAP-style fast path that application-level tests cannot use, so pin file-level serialism where files must share one local instance.
|
|
360
|
+
- The DB boundary stays hermetic and local, but a read-only external API may legitimately be exercised here behind an opt-in flag. Anything with spend, side effects, a deployed schema, or tenancy / RLS belongs in @live, not here.
|
|
361
|
+
- When the runner does not auto-load `.env`, source real credentials explicitly. Then override the backing-service vars to point at the local instance, so a stray remote URL cannot redirect the tier at production data.
|
|
362
|
+
- **Not:** an "integration" test that is green against a long-lived shared dev DB carrying another run's rows. Its reproducibility is unproven.
|
|
288
363
|
|
|
289
364
|
### Deploy-reachability (between integration and E2E)
|
|
290
365
|
|
|
291
|
-
- Before any tier runs against a freshly-deployed URL, assert it is
|
|
292
|
-
- Make a missing target an
|
|
366
|
+
- Before any tier runs against a freshly-deployed URL, assert that it is actually serving, because **ready != serving**. A platform can report a deployment ready while it returns 404 / 500, and a bare "did the deploy finish" check passes anyway.
|
|
367
|
+
- Make a missing target an honest failure, not a silent skip. Assert that >=1 reachability test actually ran (per-file executed count), so an unreachable target reds the build instead of passing as a green no-op.
|
|
293
368
|
- **Not:** an E2E suite that "passes" because every spec silently skipped against a target that was never serving.
|
|
294
369
|
|
|
295
370
|
### E2E
|
|
296
371
|
|
|
297
|
-
- Drive the real UI against the
|
|
298
|
-
- Locally, run against a
|
|
299
|
-
- Use
|
|
300
|
-
-
|
|
301
|
-
-
|
|
302
|
-
-
|
|
372
|
+
- Drive the real UI against the specific commit's preview, after confirming it is ready and serving. Resolve the deploy by commit SHA, never by a static branch alias that may point at a stale build.
|
|
373
|
+
- Locally, run against a production build, not a dev server, because a dev file-watcher remounts components mid-test and breaks stateful UI (dialogs, wizards).
|
|
374
|
+
- Use stable selectors (test-ids) and condition-based waits. Never use a fixed-time or network-idle wait.
|
|
375
|
+
- Do not gate a response-wait on status. Match URL + method, then assert success explicitly, or a non-2xx response silently spins until the timeout.
|
|
376
|
+
- Clean up what you create (`afterAll`), and warn on accumulated `test_` / `e2e_` rows, so leaked fixtures cannot slowly corrupt a shared environment.
|
|
377
|
+
- On a pull request and on a push, run the same spec set (§ "E2E on PRs"). Scale a slow suite by sharding, not by dropping specs.
|
|
378
|
+
- **Not:** clicking by visible text and waiting on `networkidle`, which are the two classic causes of flaky E2E.
|
|
303
379
|
|
|
304
380
|
### @live (golden)
|
|
305
381
|
|
|
306
|
-
- Cross the
|
|
307
|
-
-
|
|
308
|
-
- **Honest skip vs hollow:**
|
|
309
|
-
-
|
|
382
|
+
- Cross the real external or deployed boundary, and assert a ground-truth signal that only it can produce (§ Validated means executed). There are two shapes: **in-process** (the test calls the real SDK / model / API directly) and **deployed-endpoint** (the test fetches the running app). Use whichever one observes the boundary you are validating.
|
|
383
|
+
- It runs on a push to `dev` or the default branch only, when `web_affected` (`run_live`). A planned failure reds the capstone. It does not run on a pull request or a dispatch. A newer run on the same ref cancels the older one (`deployments.md` § One run per branch or pull request).
|
|
384
|
+
- **Honest skip vs hollow:**
|
|
385
|
+
- **If** there is genuinely nothing deployed to validate → skip with a recorded reason. The capstone then asserts nothing about the count. This is different from a tier that ran 0 tests.
|
|
386
|
+
- **If** a credential is missing while the tier is enabled → fail loudly at connect, never skip, because a vacuous skip hides infra rot.
|
|
387
|
+
- Never blast a production system with test traffic. Use a synthetic-identity namespace, plus a guarded teardown that refuses to delete anything outside it.
|
|
310
388
|
- **Not:** a non-mock @live test that short-circuits before the boundary (the hollow case), or one that asserts only "did not throw".
|
|
311
389
|
|
|
312
390
|
### Migration-replay + seed (the fresh-DB gate)
|
|
313
391
|
|
|
314
|
-
- Prove every migration
|
|
315
|
-
-
|
|
316
|
-
- Generate the seed from a
|
|
317
|
-
- **Replay-from-empty, not reused-volume.** A bare "start the local stack" can reuse a stale data volume and silently skip newer migrations, so it is
|
|
318
|
-
- Trigger the gate on
|
|
319
|
-
- **Not:** treating a reset
|
|
392
|
+
- Prove that every migration replays from an empty database on a fresh ephemeral instance, and that the seed then loads. This is the always-on sibling gate `bkit-tests-migration` (§ Provision backing services fresh). It is different from the deploy-time step that applies only new migrations onto the already-migrated remote (`deployments.md` § Applying migrations at deploy time). That remote push lives in the same always-on lane too. It comes after the replay, which keeps a from-empty failure away from the shared remote.
|
|
393
|
+
- Assert that the seed actually loaded by checking that a known row exists. A reset or replay can exit 0 while a conditional insert silently does nothing or the seed aborts mid-file. A green replay is not proof that the fixture is present (this is the seed version of the hollow-boundary rule).
|
|
394
|
+
- Generate the seed from a single deterministic source, and regenerate the checked-in seed file from it. The source is a pure data definition with no wall-clock `now()`, no random values, and explicit IDs / enums so triggers do not randomise it. Two hand-maintained copies drift.
|
|
395
|
+
- **Replay-from-empty, not reused-volume.** A bare "start the local stack" can reuse a stale data volume and silently skip newer migrations, so it is not a freshness proof. Only a reset or replay from empty is.
|
|
396
|
+
- Trigger the gate on every migration and seed source, including the seed generator and its data definition, not just the emitted seed file. Replay every migration chain, because a second ORM / CMS chain that this gate never replays is an unverified hole.
|
|
397
|
+
- **Not:** treating a reset or replay command's exit 0 as "seed loaded", or proving freshness with a start command that reuses the volume.
|
|
320
398
|
|
|
321
399
|
### Smoke
|
|
322
400
|
|
|
323
|
-
- After deploy, run a
|
|
324
|
-
- It is an
|
|
325
|
-
- **Not:** a smoke suite wired into branch protection, or one so heavy it
|
|
401
|
+
- After the deploy, run a fast, dependency-light probe of the live surface (a handful of health / route / contract checks, or a tiny tagged browser subset). It checks breadth ("is the app fundamentally up?"), not depth.
|
|
402
|
+
- It is an observability gate, not a merge gate. It has its own workflow and is never in branch protection. When `bkit-tests-deploy` promotes inside CI, this probe is the health workflow, not a second file (`deployments.md` § Smoke trigger plumbing, § Promote before the lanes).
|
|
403
|
+
- **Not:** a smoke suite wired into branch protection, or one so heavy that it cannot run on every deploy.
|
|
326
404
|
|
|
327
405
|
## Design assertions are a dimension, not a lane
|
|
328
406
|
|
|
329
|
-
**
|
|
407
|
+
**Never add a design lane.** The lane axis is **resource profile** (a millisecond file read vs a booted app + browser), not subject matter, because design assertions span both. A `bkit-tests-design` job (or a design-only browser project) splits one concern across two profiles and buys nothing that the existing lanes do not already provide. Fold each assertion into the tier that already fits it. The CI topology, the capstone's `needs:` and the browser-project partition all stay unchanged. A real consumer's full design-system rollout landed with `bkit-ci.yml` byte-unchanged.
|
|
330
408
|
|
|
331
409
|
<!-- prettier-ignore -->
|
|
332
410
|
| Design assertion | Lane |
|
|
@@ -340,97 +418,134 @@ Tier-by-tier rules: **do** the rule, avoid the **Not** (the failure it prevents)
|
|
|
340
418
|
| Design-source parity (a rendered style guide vs the reference doc) | `bkit-tests-unit` |
|
|
341
419
|
| Visual regression (screenshot baselines) | **opt-in, not baseline** (§ Optional hardening) — highest maintenance cost of the set |
|
|
342
420
|
|
|
343
|
-
- **One reference, asserted independently.** The reference doc owns the token contract and its parsing rules (`design.md` § Machine-checked reference). What lanes care about
|
|
344
|
-
- **Merge-gated from day one
|
|
345
|
-
- **A group tag carries zero routing power.**
|
|
346
|
-
-
|
|
347
|
-
-
|
|
421
|
+
- **One reference, asserted independently.** The reference doc owns the token contract and its parsing rules (`design.md` § Machine-checked reference). What the lanes care about is that each implementation is its own test, so one red names its culprit.
|
|
422
|
+
- **Merge-gated from day one, never an advisory window.** The assertions live inside lanes that the capstone already gates, so a design red blocks the merge on the first PR that adds one. There is no promotion step to schedule, so there is no deferral row to forget.
|
|
423
|
+
- **A group tag carries zero routing power.**
|
|
424
|
+
- Tag design tests in the test title, never by a filename convention, and add a `test:design` script for local iteration only.
|
|
425
|
+
- The tag must never appear in an execution gate or an inverse-grep filter, because a test without it still runs in its normal lane.
|
|
426
|
+
- Keep the standard filename, in a location that the package's include glob already matches. A bespoke `*.design.test.ts` or an unreached dir runs in zero lanes, so it is green everywhere and covers nothing (§ Validated means executed).
|
|
427
|
+
- Pass-with-no-tests is mandatory on every `test:design` fan-out, and only there. The lane configs keep the ban (§ Validated means executed). Each runner spells that flag differently, and a browser runner hard-errors on a path or project name that matches nothing.
|
|
428
|
+
- **Cache footgun: design tests read files outside their own package.** A token-parity test reads the reference doc, and a raw-hex contract scans the source tree. Under a per-package-hash runner, editing the reference doc changes no package hash, so a warm cache replays a PASS against a reference that the code now contradicts. Put the reference doc (and any rendered style guide) in the runner's global dependencies, and mark the design-selection task uncached. A design run that overwrites the machine-readable results file the executed-count gate consumes is the same kind of trap, so re-run the full suite before trusting either.
|
|
429
|
+
- **Keep the first honest reds.** A first real geometry or a11y sweep surfaces genuine defects (undersized controls, failing contrast).
|
|
430
|
+
- File them as issues, allowlist them behind a **named constant**, and ship the allowlist's **stale-entry check** in the same change, because an allowlist that never rejects a fixed entry decays into a silent exemption.
|
|
431
|
+
- Enumerate the sweep scope by role and tag, never by test-id presence, because library-generated markup escapes that gate by construction.
|
|
432
|
+
- Measure every theme before asserting any, or a light-theme red hides the dark-theme offenders.
|
|
348
433
|
|
|
349
434
|
## Schema-invariant DB checks (e.g. RLS coverage)
|
|
350
435
|
|
|
351
|
-
Access-policy coverage is stack-specific
|
|
436
|
+
Access-policy coverage is stack-specific. bkit ships the pattern (the two routes below), not a drop-in test. Supabase / PostgREST is the worked example. Implement it for your backing store by swapping the catalog source and the anon probe for your stack's equivalent.
|
|
352
437
|
|
|
353
|
-
Assert DB-level invariants (
|
|
438
|
+
Assert DB-level invariants (for example, "every public table has RLS enabled", with no hardcoded list) in one of two co-equal ways. Pick by deploy topology, not by whether you can run a local DB-test runner (for example, pgTAP):
|
|
354
439
|
|
|
355
|
-
- **pgTAP** (hermetic-fixture route)
|
|
356
|
-
- **@live catalog-RPC
|
|
357
|
-
- **PostgREST gotcha
|
|
440
|
+
- **pgTAP** (the hermetic-fixture route): for example, `supabase/tests/*.sql` run by `supabase db test` in the `bkit-stub:test-db` step, against a CI-local fixture built from `supabase/migrations`. It is fast, hermetic and offline, and it keeps the invariant out of app code. Caveat: the fixture can differ from prod, so a migration that does not reproduce the live schema leaves this green while the prod access policies are actually broken. It is best when there is no deploy / `@live` tier, or for pure-logic DB unit tests.
|
|
441
|
+
- **@live catalog-RPC:** a privileged, server-only catalog query called from the `@live` tier (for example, Supabase: a `SECURITY DEFINER`, service_role-only RPC `public.rls_coverage_gaps()` over REST). It validates the real deployed schema in `bkit-tests-live-web`, and it needs a deploy and the RPC. It is the recommended default once you have a deploy / `@live` tier, because for access-policy and tenancy invariants especially, a passing local-fixture check is not evidence that prod is locked down. For example, on Supabase a direct port-5432 connection from CI runners is IPv4-flaky, so use the REST path that the tier already has.
|
|
442
|
+
- **PostgREST gotcha:** `GET /rest/v1/` (OpenAPI) is service_role-only by default. Fetch the exposed surface with the service-role key, then probe each table with the ANON key. Only the leak assertion must use anon.
|
|
358
443
|
|
|
359
|
-
An auto-discovering access-policy suite
|
|
444
|
+
An auto-discovering access-policy suite belongs in `bkit-tests-live-web`. For RLS it checks RLS-enabled, anon-reads-`[]` and cross-tenant-denial with no hardcoded table list, so it catches the next table too.
|
|
360
445
|
|
|
361
446
|
## Provision backing services fresh
|
|
362
447
|
|
|
363
|
-
A tier that needs a stateful backing service (database, cache, queue, search index) must run against a
|
|
448
|
+
A tier that needs a stateful backing service (database, cache, queue, search index) must run against a freshly-provisioned instance, not a long-lived shared one. Green against a shared environment carrying accumulated state is not evidence of reproducibility. A migration that never re-applies cleanly, a seed that aborts on an empty store, or a row another run left behind all stay invisible until the service is rebuilt from scratch. Prove that a fresh provision + seed succeeds in CI: for example, an ephemeral per-PR database started fresh each run (Supabase: `supabase db reset` + seed). Reserve the shared remote / dev instance for the `@live` tier and push-side E2E, because they validate the real deployed schema. It is never the integration tier's source of truth.
|
|
449
|
+
|
|
450
|
+
bkit's `bkit-tests-migration` lane (`bkit-stub:bkit-tests-migration`) is the ready-made home.
|
|
451
|
+
|
|
452
|
+
- It is always-on. It has no job-level `if:`, because the e2e and live lanes `needs:` it for remote-push ordering, and a skip would cascade.
|
|
453
|
+
- The replay and the seed assert stay plan-gated and path-gated per step.
|
|
454
|
+
- On every push to `dev`/`main`, the same lane owns the remote apply of new migrations (for example, `supabase db push --linked --include-all`). That apply has no `run_migration` gate, so the lane self-heals. Because the apply comes after the replay, a from-empty failure never gets to the remote.
|
|
455
|
+
- The capstone asserts `result == success` whenever the scope succeeded. Fold the lane into `bkit-tests-check`'s `needs:`.
|
|
364
456
|
|
|
365
|
-
|
|
457
|
+
**Local integration freshness.** A bare start of a local stack that reuses a warm data volume (for example, Docker + `supabase start`) can leave migrations unapplied (the stale-volume footgun).
|
|
366
458
|
|
|
367
|
-
|
|
459
|
+
- Prefer a reset-by-default runner over a bare integration run (for example, `turbo run test:integration`). Mirror local E2E: reset unless `--no-reset`.
|
|
460
|
+
- Shape: `pnpm test:integration:local` → a script that starts the local stack, exports its URL + keys, resets from empty (for example, `supabase db reset --local`), then runs integration. In CI it passes straight through, because CI boots its own hermetic stack.
|
|
368
461
|
|
|
369
|
-
**Local runners never touch env files** (
|
|
462
|
+
**Local runners never touch env files** (for example, `.env.local` or `.env.production`, which may be the only copy of live credentials):
|
|
370
463
|
|
|
371
|
-
- Target local stack by exporting its values before config loads
|
|
372
|
-
- Blank a hosted key with `export KEY=""`, never `unset
|
|
373
|
-
-
|
|
374
|
-
-
|
|
375
|
-
-
|
|
464
|
+
- Target the local stack by exporting its values before the config loads. Loaders (Next, dotenv `override: false`) never override a key that is already set.
|
|
465
|
+
- Blank a hosted key with `export KEY=""`, never with `unset`, because the loader refills unset keys from the file.
|
|
466
|
+
- NEVER delete, move or hide-and-restore the file, because a killed run loses it.
|
|
467
|
+
- NEVER `vercel env pull` onto it, because that replaces it with the host's keys only. Pull to a scratch file, then copy the keys across.
|
|
468
|
+
- A guard test fails any tracked script that deletes, moves or overwrites an env file.
|
|
376
469
|
|
|
377
|
-
**"Fresh" means replay-from-empty, not a reused volume.** A bare "start the local stack" command often
|
|
470
|
+
**"Fresh" means replay-from-empty, not a reused volume.** A bare "start the local stack" command often reuses an existing data volume and applies only what is missing, so a "fresh" start can silently run a months-old schema while reporting success. Only a reset or replay from an empty store is a freshness proof. A green replay is also not a green seed. Assert that the seed actually loaded (a known row exists), because the reset can exit 0 while a conditional insert does nothing or the seed aborts mid-file. § Migration-replay + seed carries the full seed-determinism rules.
|
|
378
471
|
|
|
379
|
-
**Multiple migration systems:** a repo
|
|
472
|
+
**Multiple migration systems:** a repo that runs more than one migration chain (for example, a SQL chain for the public schema and an ORM / CMS chain such as Payload or Prisma) must fresh-apply every chain here, and trigger on every migration source dir. A chain that this gate never replays is an unverified hole, so a broken migration in it lands green.
|
|
380
473
|
|
|
381
474
|
## Optional hardening (adopt when the cost is justified)
|
|
382
475
|
|
|
383
|
-
Beyond the gates above, three practices raise confidence further
|
|
476
|
+
Beyond the gates above, three practices raise confidence further. They are opt-in, not baseline.
|
|
384
477
|
|
|
385
|
-
- **Mutation testing** (
|
|
386
|
-
-
|
|
387
|
-
-
|
|
478
|
+
- **Mutation testing** (for example, StrykerJS over a Vitest suite) automates the "executed != exercised" intent. It injects bugs into production code and fails when no test catches them.
|
|
479
|
+
- Run it locally, on demand. It is not a CI lane and not a nightly workflow.
|
|
480
|
+
- Gate a local run on a survival-score threshold, never 100%, because equivalent mutants make perfection unreachable.
|
|
481
|
+
- The capstone asks "did tests run?". Mutation asks "do they catch injected bugs?".
|
|
482
|
+
- **E2E sharding** (a `--shard` matrix + report merge) cuts wall-clock time on a large browser suite, but it is gated on per-shard data isolation (the unique-ID independence above). A single shared seed pins `workers: 1` and blocks sharding until the data is isolated. Isolated data is still not enough once several invocations share one job (§ Concurrent invocations collide on disk, not just in the database).
|
|
483
|
+
- **Bounded flaky-retry with tracking.** A capped retry on the flake-prone E2E tier is low-risk only when it is paired with explicit flake tracking. An auto-retry that silently passes hides a real regression, which is the inverse of "validated != didn't fail".
|
|
388
484
|
|
|
389
485
|
## Coverage on PRs
|
|
390
486
|
|
|
391
|
-
Shipped stubs target Vitest (`bkit-stub:coverage`) and Playwright (`bkit-stub:e2e-coverage`)
|
|
487
|
+
Shipped stubs target Vitest (`bkit-stub:coverage`) and Playwright (`bkit-stub:e2e-coverage`). On another runner, keep the rules (per-tier, report-only, ratchet from current) and swap the tooling.
|
|
392
488
|
|
|
393
|
-
Report coverage on each PR
|
|
489
|
+
Report coverage on each PR without a paid product or a third-party service. For example, the `bkit-stub:coverage` block in `.github/workflows/bkit-ci.yml` runs `davelosert/vitest-coverage-report-action` entirely inside Actions with `GITHUB_TOKEN`, and posts sticky per-PR comments. That is safe for private repos, because no coverage data leaves the runner. It replaces GitHub Code Quality's coverage-on-PR value (`deployments.md` § Disable GitHub Code Quality billing). It does not replace CodeQL security scanning, which is a separate product.
|
|
394
490
|
|
|
395
|
-
**Per-tier, not merged.** Unit and integration are measured and commented
|
|
491
|
+
**Per-tier, not merged.** Unit and integration are measured and commented separately: two runs into `coverage/unit` and `coverage/integration`, and two comments keyed by a distinct `name:`. A single merged number is the weakest option. Coverage is a union, so a line covered only by an integration test reads as covered and hides a missing unit test. A merged gate therefore lets integration mask a hollow unit suite. The two numbers also have different denominators, so they cannot be honestly combined into one.
|
|
396
492
|
|
|
397
|
-
- **Report-only by default.**
|
|
398
|
-
- **Reporters are load-bearing.** The action reads `coverage-summary.json` (from the `json-summary` reporter, mandatory) and `coverage-final.json` (from `json`,
|
|
399
|
-
- **Two runs, separate dirs.** `vitest run --coverage --coverage.reportsDirectory=coverage/unit --exclude '**/*.integration.test.ts'` for unit
|
|
493
|
+
- **Report-only by default.** There are no `thresholds`, so coverage never reds the lane, and enabling the block on an existing repo cannot break it.
|
|
494
|
+
- **Reporters are load-bearing.** The action reads `coverage-summary.json` (from the `json-summary` reporter, which is mandatory) and `coverage-final.json` (from `json`, which `file-coverage-mode: changes` requires). Enable both in `coverage.reporter`, or the step does nothing.
|
|
495
|
+
- **Two runs, separate dirs.** Run `vitest run --coverage --coverage.reportsDirectory=coverage/unit --exclude '**/*.integration.test.ts'` for unit, and a second run into `coverage/integration` for the integration tier. Separate dirs also avoid the last-writer-wins clobber that a second `--coverage` run into the same dir would cause.
|
|
400
496
|
- **Thresholds are opt-in and per tier:**
|
|
401
|
-
- **Unit = the gate.**
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
-
|
|
405
|
-
-
|
|
406
|
-
- **
|
|
407
|
-
- **
|
|
497
|
+
- **Unit = the gate.**
|
|
498
|
+
- Put `coverage.thresholds.branches: <current>` in a `vitest.unit.config.ts` (`mergeConfig` over `vitest.config`) that only the unit run loads via `-c`. Branches is the meaningful floor, because line / statement % overstates confidence. Set it to your current coverage and ratchet up.
|
|
499
|
+
- vitest coverage is per run (root config), not per project. `projects` does not scope it, and a `thresholds` block in `vitest.config` would red both runs. The `-c` file is what keeps the gate off integration.
|
|
500
|
+
- A `--coverage.thresholds.branches=<n>` CLI flag also gates, but the report step cannot read it.
|
|
501
|
+
- 80% is convention, not best practice, because a fetishised high bar breeds assertion-free line-touching tests. The executed-!=-exercised rule (§ Validated means executed) matters far more than the number.
|
|
502
|
+
- **Floor beside the figure.** The unit report step runs on every event, a red run too (via `!cancelled()` plus a `hashFiles` guard). It passes `vite-config-path: vitest.unit.config.ts`, and comments only on same-repo PRs (`comment-on: pr`, else `none`), so every job summary, pushes included, shows the measured % against its 🎯 floor. The action regex-matches `<metric>: <number>` in that file's raw text. Keep the floor on one unquoted line, with no other metric name + colon + number in the file, because a broken match drops the target silently while the gate keeps passing.
|
|
503
|
+
- **Integration = report-only.** Gate integration on execution (the capstone's `integration_ran > 0`), not on a %. A whole-repo integration % is denominator noise. Integration covers boundaries and flows, not pure logic, and rarely drives every branch, so a hard threshold on it punishes the correct pyramid shape.
|
|
504
|
+
- **If** you want a real integration % → add `--coverage.include='src/server/**'` on the integration run (the same CLI-flag idiom as `--exclude`) to scope it to the integration surface. Then set a floor, or ratchet from current. Never use a number picked from thin air.
|
|
505
|
+
- **Share the unit gate with the build workflow.**
|
|
506
|
+
- Put the unit run (`vitest run -c vitest.unit.config.ts --coverage …`) in a `test:coverage:unit` script, and chain it into `bkit:gate:static-unit` in place of `pnpm test`. Never put it beside `pnpm test`, because both run every unit test, so each test runs twice.
|
|
507
|
+
- The CI unit lane calls the script, and `/bkit-build` Run Tests calls the gate, so the floor has one source, and a regression fails in the worktree, not only on the PR.
|
|
508
|
+
- Never duplicate the command and sync it by comment, because that drifts.
|
|
509
|
+
- The build runs only the deepest declared `bkit:gate:*`, never the script by name. Left out of the chain, the script never gates the build. bkit's own repo does exactly this.
|
|
510
|
+
- **Permissions.**
|
|
511
|
+
- The report steps need `pull-requests: write`. Declare it at job level on the lane that posts.
|
|
512
|
+
- A job-level `permissions:` block replaces the workflow-level block rather than adding to it, so re-include `contents: read` too, or the checkout breaks.
|
|
513
|
+
- **Separate lanes:** with a lane per tier (recommended, see `deployments.md` § Parallel test lanes), each coverage comment posts from its own lane (unit-cov → `bkit-tests-unit`, integration-cov → `bkit-tests-integration`). Put the block on each lane that reports. With the combined starter, both live on `bkit-tests-unit`.
|
|
514
|
+
- **Fork PRs.** `GITHUB_TOKEN` is read-only for PRs from a fork, so commenting fails.
|
|
515
|
+
- The shipped unit step sets `comment-on: none` there, and its job summary still lands.
|
|
516
|
+
- The integration step's `head.repo == repo` guard skips it.
|
|
517
|
+
- **If** you must comment on fork PRs → adopt the action's `workflow_run` two-workflow pattern.
|
|
518
|
+
- The integration step is also guarded on `hashFiles(...)`, so a repo with no integration tier skips it cleanly.
|
|
519
|
+
- **Base-branch delta** (▲▼ vs the default branch) is not shipped by default, because it needs the base's `coverage-summary.json` (from a second run or a stored artifact). **If** you want it → wire the action's `json-summary-compare-path`.
|
|
408
520
|
|
|
409
521
|
### E2E coverage on PRs
|
|
410
522
|
|
|
411
|
-
The
|
|
523
|
+
The E2E tier (`bkit-stub:e2e-coverage`) reports Playwright coverage in the same free, in-Actions, private-repo-safe way. It posts a third sticky comment keyed `name: e2e`, and it is never merged into the unit / integration numbers. Unlike vitest, there is no `--coverage` flag. Playwright is black-box and does not instrument your source, so this is a 3-part pipeline.
|
|
412
524
|
|
|
413
|
-
- **Collect during the run.**
|
|
414
|
-
- **Report with Monocart.** `monocart-coverage-reports` (wired via the `monocart-reporter` Playwright reporter) ingests
|
|
415
|
-
- **Comment by reuse.** Point `davelosert/vitest-coverage-report-action@v2` at `coverage/e2e` with `name: e2e`. If Monocart's `coverage-final.json`
|
|
416
|
-
- **Report-only
|
|
417
|
-
- **Permissions / forks.** The E2E coverage report runs in the
|
|
525
|
+
- **Collect during the run.** Use either an **Istanbul instrumented build** (`vite-plugin-istanbul` exposes `window.__coverage__`, which a Playwright fixture scrapes; it works on every browser) or the **V8 API** (`page.coverage`, which is Chromium-only and needs source maps). Istanbul is the cross-browser default.
|
|
526
|
+
- **Report with Monocart.** `monocart-coverage-reports` (wired via the `monocart-reporter` Playwright reporter) ingests either V8 or Istanbul input. It emits Istanbul-format `coverage/e2e/coverage-summary.json` + `coverage-final.json`, the same two files the vitest action reads (`reports: ['json-summary', 'json', 'lcov']`), so no new commenter is needed.
|
|
527
|
+
- **Comment by reuse.** Point `davelosert/vitest-coverage-report-action@v2` at `coverage/e2e` with `name: e2e`. **If** `file-coverage-mode: changes` does not accept Monocart's `coverage-final.json` → drop to `file-coverage-mode: all`, or comment via a cobertura action (`irongut/CodeCoverageSummary`) on Monocart's `cobertura` report.
|
|
528
|
+
- **Report-only, no threshold by default.** A whole-app E2E % is denominator noise, because E2E exercises user flows, not every branch, so gate the tier on the E2E step succeeding, not on a number. For a real E2E %, scope Monocart's `entryFilter`/`sourceFilter` to the surface E2E is meant to cover, then ratchet from current. Never use a picked number.
|
|
529
|
+
- **Permissions / forks.** The E2E coverage report runs in the `bkit-tests-e2e-web` lane (or, with the combined starter, in `bkit-tests-unit`). It needs that lane's job-level `pull-requests: write` + `contents: read` block, or the comment cannot post. The `head.repo == repo` guard skips fork PRs. A `hashFiles('coverage/e2e/coverage-summary.json')` guard skips repos with no E2E coverage.
|
|
418
530
|
|
|
419
531
|
## CI verbosity (`[verbose ci]`)
|
|
420
532
|
|
|
421
|
-
See `deployments.md` § Commit-message force tags for the marker mechanics
|
|
533
|
+
See `deployments.md` § Commit-message force tags for the marker mechanics. That covers where to put it: the PR title for feature-branch pushes, and the commit message for a direct push to `dev` / `{{DEFAULT_BRANCH}}`. This section covers the testing-specific knobs.
|
|
422
534
|
|
|
423
|
-
**Knobs flipped under the marker
|
|
535
|
+
**Knobs flipped under the marker.** The shipped stub flips turbo / Node / Vercel / Supabase / Playwright knobs. Add your stack's equivalents.
|
|
424
536
|
|
|
425
|
-
- `TURBO_LOG_LEVEL=full`
|
|
426
|
-
- `NODE_OPTIONS=--trace-warnings --trace-deprecation`
|
|
427
|
-
- `
|
|
428
|
-
-
|
|
537
|
+
- `TURBO_LOG_LEVEL=full` surfaces Vitest / build output across all turbo steps. Without it, that output is compressed to errors only.
|
|
538
|
+
- `NODE_OPTIONS=--trace-warnings --trace-deprecation` prints stacks on deprecations and transitive-dep regressions.
|
|
539
|
+
- Set it as a job-level `env:` key in static YAML, never by writing it to `$GITHUB_ENV`. The runner blocklists `NODE_OPTIONS` for that command: it rejects the write with an `::error::` and still concludes the step `success`. The flag then reaches nothing while the run goes green.
|
|
540
|
+
- Spell the `contains()` marker test out rather than reusing `env.VERBOSE_CI`, because the `env` context is not available to a job-level `env:` value.
|
|
541
|
+
- A step with its own `env:` block replaces the job-level value rather than merging with it, so that step must restate the flags.
|
|
542
|
+
- `DEBUG=vercel:*,supabase:*` shows the Vercel deploy and Supabase migration internal chain. It leaves out `pw:*`, because that duplicates the Playwright trace.
|
|
543
|
+
- Playwright (when `bkit-stub:e2e` is enabled) appends `--trace=on` only. `--video` and `--screenshot` are `playwright.config.ts use:` keys, not CLI flags, and Playwright errors on unknown CLI options. The trace.zip captures snapshots / network / console / source, which covers the main diagnose use case. Never pass `--reporter=line`. Let the `playwright.config.ts` chain run.
|
|
429
544
|
|
|
430
|
-
**Use when:** debug returns thin evidence (no `file:line`, only "tests failed", e2e
|
|
545
|
+
**Use when:** debug returns thin evidence (no `file:line`, only "tests failed", or an e2e failure without a trace). `bkit-debug` / `failure-diagnose` recommends it automatically. Never use it on a first attempt, and never on `bkit-ship`, because verbose output on a green run is pure cost in artifact storage and run time.
|
|
431
546
|
|
|
432
|
-
**Hostile gap
|
|
547
|
+
**Hostile gap: Playwright `--reporter=line` override.** **If** your `playwright test` invocation passes `--reporter=line` → drop it. With the marker on, traces are written but no reporter surfaces them, so the artifact upload is wasted. `bkit install` flags this as an advisory.
|
|
433
548
|
|
|
434
|
-
**Artifact paths** (already in `.gitignore` via bkit template): `test-results/`, `playwright-report/`, `blob-report/`, `playwright/.auth/`. Never commit these, because traces alone run 1-50MB each.
|
|
549
|
+
**Artifact paths** (already in `.gitignore` via the bkit template): `test-results/`, `playwright-report/`, `blob-report/`, `playwright/.auth/`. Never commit these, because traces alone run 1-50MB each.
|
|
435
550
|
|
|
436
551
|
<!-- bkit:end -->
|