@boopkit/cap-testing 0.5.0 → 0.5.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@boopkit/cap-testing",
-  "version": "0.5.0",
+  "version": "0.5.2",
   "description": "Jest testing setup and testing skills for AI agents",
   "files": [
     "scaffold/",

package/skills/boop-testing-testing/SKILL.md

@@ -12,12 +12,52 @@ metadata:
 **Functional testing, not ceremony.** Tests prove code works — not that mocks were wired correctly. Every test should exercise real logic. Type-checking or compilation alone is never sufficient. Be pessimistic about your assumptions: if you think "this should never fail," write a test that verifies it doesn't.
 
 - **Observe the failure first.** A plausible explanation for repeated failures is still a hypothesis until you see success. Always verify the expected *positive* outcome actually occurs (e.g., `cache_hit: true`, not just "no errors").
-- **Mocks are a last resort**, acceptable only for genuinely external dependencies (third-party HTTP, email providers) or to isolate complex decision logic (error classification, retry semantics). If it doesn't hit a real store, call it a unit test — not an "integration test with mocks."
+- **Mocks are a last resort**, acceptable only for genuinely external dependencies (third-party HTTP, email providers), to pin a nondeterministic input the test is not asserting on (a model/LLM call in a plumbing test — see "Stub the model, not the machinery"), or to isolate complex decision logic (error classification, retry semantics). If it doesn't hit a real store, call it a unit test — not an "integration test with mocks."
 - **Data-driven coverage for config tables.** Code backed by configuration (pattern tables, route maps, enum mappings) must have tests that exercise *every entry*, not a hand-picked subset. Include a count sanity check (`expect(records.length).toBeGreaterThan(N)`) so an empty response doesn't silently pass.
 - **Undocumented assumptions are spec gaps, not test targets.** When the spec is silent on a behavior (e.g., can a user have multiple sessions? does logout delete or expire state?), that's a design decision to resolve — not something to test around. **Resolve the ambiguity first** (update the spec or get a decision), **then** write the test encoding the decided behavior. If you're writing tests before code (TDD), a spec gap blocks you — don't guess the intent and test your guess. If you discover the gap while reviewing existing code, file a spec issue and name the test to surface the question (e.g., `'login creates additional session token — multi-device support'` not `'login works'`).
 
 ---
 
+## Assert the effect, not the envelope
+
+**A test must assert that the state actually changed — not just the shape of the response.** The response envelope (status code, redirect target, `{ ok: true }` body) is what the handler *says* happened; the effect (a row transitioned, a record was created, a balance moved) is what *actually* happened. Assert the effect.
+
+This matters most when **the success path and a failure path can produce the same envelope.** A handler that redirects to the same place on success and on silently-rejected input will pass any test that checks only the redirect — on both paths. The test proves nothing about which path ran.
+
+**Neutral example.** A handler that submits a form redirects to `/items` both when the item is accepted *and* when validation silently drops it (no error surfaced). A test asserting `302 -> /items` is green on both paths, so a regression that makes every submission silently bounce still passes. The test that also reads the item back and asserts it reached the `submitted` state is the one that catches the regression.
+
+```
+// Weak — green on success AND on a silent bounce
+expect(res.status).toBe(302);
+expect(res.headers.location).toBe('/items');
+
+// Strong — asserts the state change the request was supposed to cause
+expect(res.status).toBe(302);
+const item = await store.get(id);
+expect(item.state).toBe('submitted');   // the effect
+```
+
+**Rule:** whenever success and failure share a response shape, assert the observable state change. The envelope alone is not evidence the operation happened.
+
+---
+
+## Quarantine convention — a skipped test must cite a tracking ticket
+
+**"Known failing" must be a bounded, tracked, VISIBLE state — never ambient noise.** A suite that runs chronically red carries zero signal: a genuine new failure hides among the tolerated ones. Quarantine-with-a-ticket converts "N flaky failures" into "N tracked exceptions", so any *unexpected* red is a real signal again.
+
+**Rule.** Every statically-skipped test (`describe.skip`, `it.skip`, `test.skip`, `xit`, `xdescribe`, `xtest`) must carry one of these markers, on the same line or within the two comment lines directly above it:
+
+```
+// QUARANTINE: <issue-id> <reason>   — known-failing, tracked by an issue.
+// SKIP-OK: <reason>                 — intentional non-failure skip (opt-in / env-gated / platform-specific).
+```
+
+A skip carrying neither a ticket reference nor a `SKIP-OK:` reason is a convention violation.
+
+**Enforcement.** A guard test scans every test file and fails when a skip lacks a marker, so it runs as part of the normal suite with no extra tooling. When you must quarantine, file (or reuse) a tracking issue first, then add the marker — don't skip silently.
+
+---
+
 ## Generic Testing Approach
 
 These principles apply across all subsystems.
@@ -60,6 +100,18 @@ Mocks are helpful for isolating orchestration logic, but their limitations must
 | Third-party HTTP (email, OAuth) | Adjacent services you own (use real `getServices()`) |
 | `Date.now()` for deterministic assertions | Database queries (use real Postgres/CH) |
 | Complex branching inputs | The service under test's own methods |
+| A model/LLM call when the test verifies your *system's handling* of its output (see "Stub the model, not the machinery") | A model/LLM call when the test's *subject* is the model's behavior — evals, agent decision quality |
+
+### Stub the model, not the machinery
+
+This looks like it contradicts "mocks are a last resort," but it is the same principle applied correctly. The reason not to mock a boundary is that mocking hides the boundary the test exists to exercise — the store, the queue, the serialization, the contract between components — where the real bugs live. A model/LLM call in a test that is *about the plumbing* is not that boundary. It is a nondeterministic **input source**.
+
+So when you test code that drives a model, first decide what the test actually verifies:
+
+- **Verifying your system's handling of a model's output** — the routing, the persistence, the downstream effects, the contracts between components. Here the model is an input. Give it a **fixed, representative output** so the test is deterministic, and keep every real boundary it touches (database, message queue, serialization, the producer/consumer handoff) **real**. This is not a dumb mock: you are pinning a nondeterministic input so the boundaries that carry bugs get exercised for real. A live model call here buys nothing and actively harms the test — it now goes red when the model merely phrases things differently today, which is noise, not signal, and it makes an integration suite flaky and slow.
+- **Verifying the model's behavior itself** — an eval, a prompt-quality check, "does the agent choose the right tool." Here the model *is* the boundary under test, so it must be real. Stubbing it would defeat the test.
+
+The question to ask: **"If this test goes red, do I want that to mean the model said something different today?"** If no, stub the model and keep the real stores. If yes, use the real model.
 
 ### Data-driven config coverage
 When code is backed by a config table (patterns, routes, flags, enums):

package/skills/boop-testing-testing-express-web/SKILL.md

@@ -7,7 +7,7 @@ metadata:
 # Testing: Express App (Web & APIs)
 
 
-Parent skill: [testing.skill.md](testing.skill.md) (philosophy, generic approach)
+Parent skill: [`boop-testing-testing`](../boop-testing-testing/SKILL.md) (philosophy, generic approach)
 
 ---
 
@@ -20,63 +20,68 @@ npx jest tests/services/foo --no-coverage                   # Pattern match
 npx jest tests/services/foo.test.ts -t "handles replay"     # By test name
 ```
 
-**Prerequisites**: local Postgres, ClickHouse, and AWS credentials must be available. Integration tests use real services via `getServices()`.
+**Prerequisites**: integration tests run against the project's real backing services rather than mocks, so whatever the app talks to at runtime — the database first of all — has to be reachable from the machine running the tests. Which services those are depends on the capabilities the project has installed; its README and environment template are the authority.
 
 ## Test File Layout
 
 ```
 tests/
-  helpers.ts                        # getServices(), TestEntityManager, factories
+  helpers/                          # shared setup, factories, harnesses (see below)
   services/
     foo.test.ts                     # Unit tests (mocked deps)
     foo.integration.test.ts         # Integration tests (real services)
   controllers/
-    walletAPI.test.ts               # HTTP route tests
+    itemsAPI.test.ts                # HTTP route tests
   middleware/
   utils/
 ```
 
-- `.test.ts` for Jest auto-discovery; `.integration.test.ts` when `jest.mock` conflicts with `getServices()` in the same file
+- `.test.ts` for Jest auto-discovery. Keep the mocked tests and the real-service tests for the same subject in **separate files**: `jest.mock` is hoisted above the imports and applies to every test in the module, so one file cannot both stub a dependency and exercise the real one. `.integration.test.ts` is the conventional name for the half that uses the real services.
 - Group by domain: `services/`, `controllers/`, `middleware/`
 
-## Test Helpers — [tests/helpers.ts](../tests/helpers.ts)
-- `getServices()` — real services for integration tests
-- `TestEntityManager` — tracks entities, cleans up in `afterEach`
-- Factory helpers (`createTestAccount`, `createTestDomain`, etc.) — use instead of raw Prisma
+## Test Helpers
+
+Shared test setup belongs in one place under `tests/`, not re-derived in each spec file: booting the app, creating the entities a test needs and tearing them down again, and harnesses that make an awkward dependency — a broker, a queue, a rendered template — drivable from a test.
+
+**Read what the project already has before you write one.** These helpers are not scaffolded and cannot be, because they follow whichever capabilities and domain the project has. Every project grows its own set, and the shape varies from one shared module to a directory of purpose-built ones. Reuse what fits and extend it, rather than starting a private copy inside your test file. A helper used by exactly one test is not a helper — leave it in the test.
+
+When you do add one, keep two properties. A factory returns a real, persisted entity rather than a hand-built object literal, so the test exercises the same shape production does. And whatever creates data owns its teardown, so one failed run does not poison the next.
 
 ## Integration Test Data Isolation
-- Unique prefixes (`_test/${Date.now()}`) so parallel runs don't collide
-- Track created resources for cleanup in `afterAll`; wrap each cleanup in `try/catch`
-- For ClickHouse: use test-specific `account_id` / `hostname` (row deletes are expensive with RMT)
+
+Integration tests share their backing services with every other integration test, and with whatever the last run left behind. Three habits keep them from interfering with each other:
+
+- **Namespace whatever you create.** A per-run prefix (`_test/${Date.now()}`, a UUID) keeps parallel runs and repeat runs from colliding on unique constraints.
+- **Track created records and remove them in `afterAll`**, each cleanup wrapped in its own `try/catch` so one failure doesn't strand the rest.
+- **Scope instead of deleting where deletes are expensive.** Some stores — append-only logs, analytics columnstores, anything with asynchronous merge semantics — make row deletion slow or eventually-consistent. Isolate those by writing under a test-specific tenant or key and leave the rows in place.
 
 ## HTTP Route Tests
-Test Express routes via **real HTTP requests** (`fetch()`) against a running local server. This exercises the full middleware chain (auth → feature flags → controller → service → DB → response).
+Test Express routes via **real HTTP requests** (`fetch()`) against a running local server. This exercises the full middleware chain (auth → controller → service → DB → response).
 
 Why real HTTP over supertest: supertest skips TCP/listen and can mask middleware ordering or config bugs. Auth middleware is a critical path worth testing through, not around.
 
 **Setup:**
 ```typescript
 const BASE_URL = process.env.TEST_BASE_URL || 'http://localhost:3000';
-const TEST_API_KEY = process.env.TEST_API_KEY; // npm run admin -- key-generate
+const TEST_API_KEY = process.env.TEST_API_KEY; // however this project issues API keys
 const authHeaders = { 'Authorization': `Bearer ${TEST_API_KEY}`, 'Content-Type': 'application/json' };
 ```
 
-**What to cover:** auth rejection (401/403), feature-flag gates (404), request validation (400), happy-path response shape, cross-account access checks.
+**What to cover:** auth rejection (401/403), request validation (400), the happy-path response shape, and the authorization boundary — that one account cannot reach another's data. If the project gates routes on anything further (feature flags, roles, plan tiers), cover the closed gate too, and assert the status code it actually returns rather than the one that seems right.
+
+**Prereqs:** a local server running against a reachable database, a durable test account, and whatever credential the auth middleware expects (`TEST_API_KEY` by convention).
 
-**Prereqs:** local server running, durable test account, valid API key (`TEST_API_KEY` env var), relevant feature flags enabled.
+## Ad-Hoc Testing Tools
 
-## CLI Tools for Ad-Hoc Testing
+Jest is what runs in CI and before a commit. Alongside it, projects grow small CLIs for the things a test suite is the wrong shape for: generating fixture data, poking a live local server, checking what actually landed in the database.
 
-| Tool | Purpose |
-|---|---|
-| `npm run tester -- <cmd>` | Exploratory smoke tests, fixture data generation (`crawl`, `access-logs-generate`, `s3-raw-smoke`, `user-agent`, etc.) |
-| `npm run admin -- <cmd>` | Entity/data operations (accounts, agents, credentials, CH export/replicate) |
+Boop scaffolds one — `npm run db:query` (database capability) — for reading the local database without opening Prisma Studio. Anything beyond that is project-grown; `package.json` scripts and `src/scripts/` are where to look for what yours has, and `--help` on each is faster than reading its source.
 
-Run `--help` on either for full command list. These are not automated or part of pre-commit — use Jest for that.
+These are exploratory aids. They are not part of the automated suite, and none of them substitutes for a Jest test that can go red on its own.
 
 ## References
 
-- [tests/helpers.ts](../tests/helpers.ts) — shared test utilities and entity factories
-- [src/tester.ts](../src/tester.ts) — ad-hoc CLI (`npm run tester -- help`)
-- [src/admin.ts](../src/admin.ts) — entity management CLI (`npm run admin -- help`)
-- [jest.config.js](../jest.config.js) — Jest configuration
+- [`boop-testing-testing`](../boop-testing-testing/SKILL.md) — testing philosophy and the generic approach
+- [`boop-testing-tdd-light`](../boop-testing-tdd-light/SKILL.md) — the red/green loop for a single change
+- The project's own shared test setup under `tests/` — read it before adding another helper
+- `jest.config.js` at the project root — test discovery, transform, and match config