@boopkit/cap-testing 0.5.9 → 0.5.10

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@boopkit/cap-testing",
-  "version": "0.5.9",
+  "version": "0.5.10",
   "description": "Testing skills for AI agents: the testing principles and the TDD-Lite workflow",
   "files": [
     "skills/",

package/skills/boop-testing-testing/SKILL.md

@@ -46,6 +46,27 @@ expect(item.state).toBe('submitted');   // the effect
 
 ---
 
+## Test a promise the design makes, not what the code emits
+
+**A test earns its keep by checking a promise the design makes, and the difference lies in where the expected value comes from.**
+Derive it from the design — a rule in the spec, an invariant, a grammar, a stated guarantee — and the test fails the moment the code breaks the promise.
+Copy it from what the code produces, and the test only confirms that the code agrees with itself: it stays green through a bug, and certifies the bug as correct.
+
+This is the layer beneath "Assert the effect, not the envelope".
+Asserting a real state change is necessary, but if the expected state was read off the running implementation, the test still mirrors the code.
+
+Ask of every assertion whether it would pass on a plausible wrong implementation.
+A test you cannot picture failing validates nothing.
+An expectation drifts onto the implementation in two common ways:
+
+- **The mirror.** The expected value is whatever the code printed, pasted into the assertion, so the test locks in current behaviour, bugs included. Compute the expected value independently, from the inputs and the design's rule, rather than reading it back from the unit under test.
+- **The single producer.** A promise that must hold everywhere ("every record carries a valid owner", "each key maps to exactly one handler") is asserted only on the path you happened to be working on, so another producer of the same promise, overlooked or added later, breaks it silently. Make the promise the subject of the test, and cover every producer of it.
+
+Asserting the effect, round-trip fidelity, data-driven config coverage and schema contract walks are all ways to pin a design's promise to real behaviour.
+Pointed at the implementation's output instead, each one becomes a mirror that passes forever and proves nothing.
+
+---
+
 ## Quarantine convention — a skipped test must cite a tracking ticket
 
 **"Known failing" must be a bounded, tracked, VISIBLE state — never ambient noise.** A suite that runs chronically red carries zero signal: a genuine new failure hides among the tolerated ones. Quarantine-with-a-ticket converts "N flaky failures" into "N tracked exceptions", so any *unexpected* red is a real signal again.
@@ -65,7 +86,7 @@ A skip carrying neither a ticket reference nor a `SKIP-OK:` reason is a conventi
 
 ## Which tests to run, and when
 
-**The rule.** A change runs the focused tests for what it touched, plus the project's whole-program static check. The full suite runs once per branch, on the work that completes it. The `boop-project-mgmt-beads-planning` skill carries the canonical wording to put on issues; this section is the reasoning behind it, and the two have to keep agreeing.
+**The rule.** A change runs the focused tests for what it touched, plus the project's whole-program static check. A change that reaches further than its own tests widens that run deliberately, as described below. The full suite runs once per branch, on the work that completes it. The `boop-project-mgmt-beads-planning` skill carries the canonical wording to put on issues; this section is the reasoning behind it, and the two have to keep agreeing.
 
 **The commands are `npm test` for the suite and `npm run typecheck` for the static check.** The project's stack capability supplies both as npm scripts — `typescript` in a TypeScript project — so every project has the same two names, and the static check has a name rather than an invocation each agent has to derive. A focused run names the test files or folders it runs; the stack's testing skill gives its exact form (`boop-typescript-testing` in a TypeScript project). Write that form on an issue, rather than leaving the claiming agent to derive it.
 
@@ -76,6 +97,20 @@ If a project has no `typecheck` script, add one before following the rule, and c
 Second, it is slower than the focused run, though see the caution below about how much weight that carries.
 
 Third, a suite already red for an unrelated reason gives you nothing to compare against, so mid-change runs on a red tree get retried and re-read rather than acted on.
+A red suite is also slower than a green one.
+A test that fails by waiting for something that never happens runs until its timeout rather than returning, and a runner configured to retry failures runs it again, so the full suite takes longest at the moment you most want its answer.
+Learn what your own suite costs in both states.
+
+**Widen a run deliberately, and say so.**
+The static check sees every caller that reaches a change through a type.
+It does not see a change that reaches further by another route: the data layer (a schema, a migration, a hand-written query), the test infrastructure itself (the runner's config, a shared helper, a fixture or setup file), or a file other code reads at run time (a config table, a template).
+For a change like that, whoever writes the issue widens its run scope to the tests that exercise what the change reaches — a folder, a subsystem's tests, or the whole suite when the test infrastructure changed — and says on the issue what to run and why.
+Those are examples; the outcome to aim for is a run that would go red if the change broke something it reaches.
+
+The claiming agent runs the scope its issue names, no narrower and no wider, and its closing comment names what it ran.
+If the work turns out to reach further than that scope, the agent says so in its closing comment, naming the tests it expects would catch the break, rather than widening the run on its own judgement.
+The branch's full-suite run still catches what the narrow run missed, and the note tells the reviewer where to look.
+A reflexive widening and a considered one look the same in a log; the stated reason is what tells them apart.
 
 **Do not rest the rule on the suite being slow.** A suite's wall-clock time is a property of how the runner is invoked as much as of the tests themselves, and it can fall by an order of magnitude on a single configuration change — running the files in a recyclable worker pool rather than in one long-lived process is the common case. A rule argued from "the suite takes tens of minutes" quietly expires the day somebody fixes the runner, and then it reads as superstition. Argue it from instrument fit, which does not expire.
 
@@ -89,6 +124,13 @@ Those settings are not decoration, and the one that matters is not always the on
 
 Keep those settings in the runner's config file rather than in the `test` script, so that the focused runs this section asks for inherit them too. The stack capability scaffolds them that way.
 
+**Before trusting a narrowed run that went through an npm script, check that the script is a single runner command.**
+npm appends everything after `--` to the end of the script's command string.
+While the script is one runner invocation, `npm test -- <path>` and a name filter narrow the run just as they would on the runner's own command line.
+If the script chains commands, as in `a && b`, the arguments reach only the last command, which is often not the runner at all.
+The runner then runs everything, and a pass looks like the narrowed run you asked for.
+Read the script in `package.json` first; if it chains, use the stack's focused-run form, which calls the runner directly and still reads its config file.
+
 **A worker pool makes test isolation a requirement, not a nicety.** More than one worker means test *files* run at the same time, in separate processes, against the same backing services. Anything two files share and do not namespace — a table one of them truncates in setup, a fixture directory, a well-known key or account id — is now a race, and it surfaces as a suite that passes alone and fails in a full run. That is the same symptom the shared-dev-database section below describes, reached by a different route, and it invites the same wrong fix of weakening the assertion that caught it. Namespace what each test creates. Where a suite is not isolation-safe yet, run one file at a time until it is, keeping whatever setting bounds each worker's memory.
 
 ## Integration tests never touch the shared dev database