@boopkit/cap-testing 0.5.0 → 0.5.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@boopkit/cap-testing",
-  "version": "0.5.0",
+  "version": "0.5.1",
   "description": "Jest testing setup and testing skills for AI agents",
   "files": [
     "scaffold/",

package/skills/boop-testing-testing/SKILL.md

@@ -12,12 +12,52 @@ metadata:
 **Functional testing, not ceremony.** Tests prove code works — not that mocks were wired correctly. Every test should exercise real logic. Type-checking or compilation alone is never sufficient. Be pessimistic about your assumptions: if you think "this should never fail," write a test that verifies it doesn't.
 
 - **Observe the failure first.** A plausible explanation for repeated failures is still a hypothesis until you see success. Always verify the expected *positive* outcome actually occurs (e.g., `cache_hit: true`, not just "no errors").
-- **Mocks are a last resort**, acceptable only for genuinely external dependencies (third-party HTTP, email providers) or to isolate complex decision logic (error classification, retry semantics). If it doesn't hit a real store, call it a unit test — not an "integration test with mocks."
+- **Mocks are a last resort**, acceptable only for genuinely external dependencies (third-party HTTP, email providers), to pin a nondeterministic input the test is not asserting on (a model/LLM call in a plumbing test — see "Stub the model, not the machinery"), or to isolate complex decision logic (error classification, retry semantics). If it doesn't hit a real store, call it a unit test — not an "integration test with mocks."
 - **Data-driven coverage for config tables.** Code backed by configuration (pattern tables, route maps, enum mappings) must have tests that exercise *every entry*, not a hand-picked subset. Include a count sanity check (`expect(records.length).toBeGreaterThan(N)`) so an empty response doesn't silently pass.
 - **Undocumented assumptions are spec gaps, not test targets.** When the spec is silent on a behavior (e.g., can a user have multiple sessions? does logout delete or expire state?), that's a design decision to resolve — not something to test around. **Resolve the ambiguity first** (update the spec or get a decision), **then** write the test encoding the decided behavior. If you're writing tests before code (TDD), a spec gap blocks you — don't guess the intent and test your guess. If you discover the gap while reviewing existing code, file a spec issue and name the test to surface the question (e.g., `'login creates additional session token — multi-device support'` not `'login works'`).
 
 ---
 
+## Assert the effect, not the envelope
+
+**A test must assert that the state actually changed — not just the shape of the response.** The response envelope (status code, redirect target, `{ ok: true }` body) is what the handler *says* happened; the effect (a row transitioned, a record was created, a balance moved) is what *actually* happened. Assert the effect.
+
+This matters most when **the success path and a failure path can produce the same envelope.** A handler that redirects to the same place on success and on silently-rejected input will pass any test that checks only the redirect — on both paths. The test proves nothing about which path ran.
+
+**Neutral example.** A handler that submits a form redirects to `/items` both when the item is accepted *and* when validation silently drops it (no error surfaced). A test asserting `302 -> /items` is green on both paths, so a regression that makes every submission silently bounce still passes. The test that also reads the item back and asserts it reached the `submitted` state is the one that catches the regression.
+
+```
+// Weak — green on success AND on a silent bounce
+expect(res.status).toBe(302);
+expect(res.headers.location).toBe('/items');
+
+// Strong — asserts the state change the request was supposed to cause
+expect(res.status).toBe(302);
+const item = await store.get(id);
+expect(item.state).toBe('submitted');   // the effect
+```
+
+**Rule:** whenever success and failure share a response shape, assert the observable state change. The envelope alone is not evidence the operation happened.
+
+---
+
+## Quarantine convention — a skipped test must cite a tracking ticket
+
+**"Known failing" must be a bounded, tracked, VISIBLE state — never ambient noise.** A suite that runs chronically red carries zero signal: a genuine new failure hides among the tolerated ones. Quarantine-with-a-ticket converts "N flaky failures" into "N tracked exceptions", so any *unexpected* red is a real signal again.
+
+**Rule.** Every statically-skipped test (`describe.skip`, `it.skip`, `test.skip`, `xit`, `xdescribe`, `xtest`) must carry one of these markers, on the same line or within the two comment lines directly above it:
+
+```
+// QUARANTINE: <issue-id> <reason>   — known-failing, tracked by an issue.
+// SKIP-OK: <reason>                 — intentional non-failure skip (opt-in / env-gated / platform-specific).
+```
+
+A skip carrying neither a ticket reference nor a `SKIP-OK:` reason is a convention violation.
+
+**Enforcement.** A guard test scans every test file and fails when a skip lacks a marker, so it runs as part of the normal suite with no extra tooling. When you must quarantine, file (or reuse) a tracking issue first, then add the marker — don't skip silently.
+
+---
+
 ## Generic Testing Approach
 
 These principles apply across all subsystems.
@@ -60,6 +100,18 @@ Mocks are helpful for isolating orchestration logic, but their limitations must
 | Third-party HTTP (email, OAuth) | Adjacent services you own (use real `getServices()`) |
 | `Date.now()` for deterministic assertions | Database queries (use real Postgres/CH) |
 | Complex branching inputs | The service under test's own methods |
+| A model/LLM call when the test verifies your *system's handling* of its output (see "Stub the model, not the machinery") | A model/LLM call when the test's *subject* is the model's behavior — evals, agent decision quality |
+
+### Stub the model, not the machinery
+
+This looks like it contradicts "mocks are a last resort," but it is the same principle applied correctly. The reason not to mock a boundary is that mocking hides the boundary the test exists to exercise — the store, the queue, the serialization, the contract between components — where the real bugs live. A model/LLM call in a test that is *about the plumbing* is not that boundary. It is a nondeterministic **input source**.
+
+So when you test code that drives a model, first decide what the test actually verifies:
+
+- **Verifying your system's handling of a model's output** — the routing, the persistence, the downstream effects, the contracts between components. Here the model is an input. Give it a **fixed, representative output** so the test is deterministic, and keep every real boundary it touches (database, message queue, serialization, the producer/consumer handoff) **real**. This is not a dumb mock: you are pinning a nondeterministic input so the boundaries that carry bugs get exercised for real. A live model call here buys nothing and actively harms the test — it now goes red when the model merely phrases things differently today, which is noise, not signal, and it makes an integration suite flaky and slow.
+- **Verifying the model's behavior itself** — an eval, a prompt-quality check, "does the agent choose the right tool." Here the model *is* the boundary under test, so it must be real. Stubbing it would defeat the test.
+
+The question to ask: **"If this test goes red, do I want that to mean the model said something different today?"** If no, stub the model and keep the real stores. If yes, use the real model.
 
 ### Data-driven config coverage
 When code is backed by a config table (patterns, routes, flags, enums):