pi-gauntlet 7.6.1 → 7.7.0

package/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## v7.7.0 - 2026-10-07
+
+- **Test economy in TDD, implementer, and code review.** The TDD skill's RED step carries six declinable defaults (extend before add, one scenario per test, parametrize equivalent cases, minimal synthetic input, shared expensive setup, cheapest honest tier); `agents/implementer.md` and the SDD self-review carry the imperative view, `agents/code-reviewer.md` and the review template the check view (economy findings are Minor `shrink:`/`delete:`, lost coverage on consolidation is Critical); anti-pattern 4 is scoped to external API responses; writing-plans prefers the existing test file as `Test:`. Two new eval targets, `eval/implementer` and `eval/code-reviewer`, measure the change on five synthetic ledger-import samples (Python and RSpec).
+- **Evals: one driver, paired intent-aware judge, `eval/_template`.** `node eval/run.mjs <target>` replaces the eight per-target runners; two frozen replay models produce before/after outputs, a frozen judge reports per-fact outcomes, and the driver labels each against the target's committed `intent.md` (`held`, `intended change`, `regression`, `unexplained change`). Only the newest `baseline.json`/`candidate.json` per sample is kept; 54 baselines are committed. `forge-skill` runs as `kind: edit`. `AGENTS.md` scopes the mandatory eval to behavior-changing edits; `AGENTS.core.md` v9 bans private data and secrets in tickets, specs, and samples. (#50)
+
 ## v7.6.1 - 2026-10-07
 
 ### Changed

package/README.md

@@ -155,11 +155,7 @@ pi install -l ~/repos/pi-gauntlet
 
 ## Evals
 
-Evals compare prompt wording on fixed inputs against human-approved must-hold facts. A non-trivial edit under `skills/`, `agents/`, or a prompt file creates or extends `eval/<target>/`; design its samples and facts during brainstorming, and run the baseline on current wording before the edit. Existing targets gain a sample exercising the changed behavior. Typo, formatting, dependency bump, and release-commit edits are exempt; deterministic code under `extensions/`, `src/bins/`, and `scripts/` stays on `npm test`.
-
-Run an arm with externally supplied models: `node eval/spec-summarizer/run.mjs run --arm candidate --persona agents/spec-summarizer.md --candidate-model <candidate-id> --reviewers <reviewer-a>,<reviewer-b>`. The [eval convention](eval/README.md) owns the layout, baseline/candidate process, and publication rules; the [spec-summarizer eval](eval/spec-summarizer/README.md) is the worked example with eight samples and committed results. Model calls run manually, never in CI; CI runs deterministic eval tests and hygiene checks. `eval/` is checkout-only and excluded from the npm tarball.
-
-`eval/forge-skill/` measures an edit to `skills/forge-skill/SKILL.md`: `node eval/forge-skill/run.mjs --skill-dir <path> --candidate-model <id> --reviewers <id>,<id>` runs a worker over ten samples (four skill fixtures, six persona fixtures - four from this repo's `agents/` history, two constructed to exercise the persona rules), has two reviewer models vote per must-hold fact, and writes records under `$TMPDIR`; `run.mjs compare <baseline> <candidate>` refuses mismatched inputs and fails on any fact that moved from kept to lost. Models are arguments, never committed. Its samples are `case.md` + `expected.md` and its raw results stay under `$TMPDIR` - the two places it deviates from the convention. Process and sample grammar: [`eval/forge-skill/README.md`](eval/forge-skill/README.md).
+Run `node eval/run.mjs <target>` to check a wording change on fixed inputs: two frozen replay models produce before/after outputs, and one frozen judge checks must-hold facts against a committed `intent.md`. The [eval process](eval/README.md) explains the single driver, target and sample layout, newest-only records, and anonymization rule. Model calls run manually, never in CI; CI runs deterministic library tests, structural lint, and hygiene checks. `eval/` is checkout-only and excluded from the npm tarball.
 
 ## Use from Claude Code
 

package/agents/code-reviewer.md

@@ -14,7 +14,15 @@ You are a code reviewer. You find issues before they ship. You **do not edit cod
 ## Review priorities, in order
 
 1. **Correctness** — Does the change do what it claims? Are edge cases handled? Off-by-one, null/undefined, empty input, concurrency.
-2. **Tests** — Is the new behavior covered? Are tests meaningful, or do they only assert type shape? Are negative cases tested?
+2. **Tests** — Is the new behavior covered? Are tests meaningful, or do they only assert type shape? Name any missing behavior worth a test concretely; never a reflexive "add edge cases". Then check added **and changed** tests for economy - each of these is a `[Minor]` finding the implementer may decline with a reason, except the last:
+   - several new tests repeat one setup and operation with the assertions split across them -> `shrink:` to one test with the assertions together.
+   - a new test repeats an existing test's assertions -> `delete:`.
+   - equivalent cases written as separate tests -> `shrink:` to one table with named rows.
+   - input or setup larger than the tested behavior and its contract need -> `shrink:` naming the fields or records the path reads. A mock of an external API response keeps that response's shape; a field the contract requires is never oversized.
+   - an expensive artifact (parsed file, built document, loaded corpus) rebuilt per test -> `shrink:` naming the shared scope.
+   - a slow pure-logic test labeled integration -> name the repo's tier for it.
+   - a test at a higher layer than the one that proves the behavior (a browser test asserting a model rule, a request test asserting a pure function) -> `shrink:` naming the lower layer that owns it.
+   - a consolidation whose combined assertions no longer cover a behavior the removed tests covered -> `[Critical]`, lost coverage; judge by behavior covered, not by assertion count.
 3. **Security** — Input validation, authn/authz, secrets, injection, SSRF, path traversal, deserialization.
 4. **Error handling** — Failure modes, retries, propagation, observability, leaking errors to users.
 5. **Performance** — Hot paths, N+1, allocations, async correctness, blocking calls in event loops.

package/agents/implementer.md

@@ -24,6 +24,18 @@ You are an implementation specialist. You execute an approved plan using strict
 - **Modifying tested code** → run the existing tests first, modify, re-run; add new tests only for new behavior.
 - **Trivial change** (typo, comment, formatting) → use judgment; if a dispatch-supplied test command touches the surface, run it.
 
+## Test economy
+
+Tests are supporting constructs for the code. Follow these defaults; decline one with a one-sentence reason in your report.
+
+- Read the existing test file for the behavior before writing RED. Add a parameter row or a stronger assertion to an existing test when one covers the behavior; add a new test function for a distinct scenario, not for every function or statement.
+- Treat several assertions on one operation's result as one scenario. Split on different setup or a different operation, not on a name containing "and".
+- Write near-identical cases with the same setup and assertion shape as one table with named rows; rows that need different setup or a different assertion are separate tests. Keep the few rows that cover the distinct outcomes, not the full product of inputs; parametrization removes duplication, not run time.
+- Build test input by hand: the smallest value the tested behavior and its contract need. Copy no production sample and load no fixture the test does not read. A mock standing in for an external API response keeps that response's shape.
+- Share an expensive immutable artifact (a parsed file, a built document, a loaded corpus) at the narrowest scope the harness offers instead of rebuilding it per test; a test that mutates it keeps its own copy. Rebuild for readability or isolation only with the reason stated.
+- Test at the cheapest layer that proves the behavior; a higher-layer test asserts what only that layer owns (rendering, navigation, a real boundary). Put unavoidably expensive work in the repo's slow tier and a real external boundary in its integration tier, by the convention `AGENTS.md`, the overrides file, or your task names; never relabel a slow pure-logic test as integration. No convention named or discoverable: place the test beside its siblings and say so in your report.
+- A test that must stay heavy because the behavior needs the full input, or a rebuild kept for isolation, goes in your report under deviations with the reason.
+
 ## Hard rules
 
 - Run ONLY the test commands your dispatch hands you (`SCOPED_TEST_COMMANDS`). Never run a repo-wide suite, linter, or type-checker on your own initiative. Dispatch carries no test commands: say so in your report; run nothing.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-gauntlet",
-  "version": "7.6.1",
+  "version": "7.7.0",
   "description": "Opinionated, gated workflow skills, subagent personas, and runtime extensions for the pi coding agent.",
   "author": "Jacek Juraszek",
   "type": "module",

package/skills/requesting-code-review/code-reviewer.md

@@ -62,8 +62,7 @@ git diff {BASE_SHA}..{HEAD_SHA}
 
 **Testing:**
 - Tests actually test logic (not mocks)?
-- Edge cases covered?
-- Integration tests where needed?
+- Each added or changed test earns its run: no repeat of an existing test's setup and operation, input no larger than the behavior and its contract need (external-API mocks keep their shape), expensive setup shared, right tier?
 - Scoped test commands passing (quote actual output; if `none` supplied, note the orchestrator gate owns execution)?
 
 **Requirements:**

package/skills/subagent-driven-development/implementer-prompt.md

@@ -104,7 +104,7 @@ Dispatch a subagent with this prompt:
     **Testing:**
     - Do tests actually verify behavior (not just mock behavior)?
     - Did I follow TDD (step 2)?
-    - Are tests comprehensive?
+    - Does each added test earn its run - extends an existing test where one covers the behavior, minimal synthetic input, shared expensive setup, right tier?
 
     If you find issues during self-review, fix them now before reporting.
 
@@ -121,7 +121,7 @@ Dispatch a subagent with this prompt:
     - Self-review findings (if any)
     - Any issues or concerns
 
-    Use `DONE_WITH_CONCERNS` if you completed the work but have doubts about correctness.
+    Use `DONE_WITH_CONCERNS` if you completed the work but have doubts about correctness, or when you kept a test heavy or rebuilt its setup on purpose (a declined test-economy default) - name it under issues or concerns with the reason.
     Use `BLOCKED` if you cannot complete the task. Use `NEEDS_CONTEXT` if you need
     information that wasn't provided. Never silently produce work you're unsure about.
 ```

package/skills/test-driven-development/SKILL.md

@@ -60,11 +60,18 @@ Implement fresh from tests. Period.
 
 Write one minimal test showing what should happen.
 
-Before writing setup, look for an existing helper, fixture, or factory. Extend a suitable existing one before creating another. Write near-identical cases for the same behavior as one parametrized or table-driven example.
+Test economy - defaults a competent engineer follows and may decline with a stated reason:
+
+1. **Extend before add.** Read the existing test file for the behavior first. Prefer a new parameter row or a stronger assertion in an existing test; add a new test function for a distinct scenario, not for every function or statement.
+2. **One scenario per test.** Several assertions on one operation's result are one scenario. A name containing "and" is not by itself a split signal; different setup or a different operation is.
+3. **Parametrize equivalent cases.** Near-identical cases with the same setup and assertion shape are one table with named rows; rows that need different setup or a different assertion are separate tests. Prefer the few rows that cover the distinct outcomes over the full product of inputs. Parametrization removes duplication, not run time.
+4. **Minimal synthetic input.** Test input is the smallest hand-built value the tested behavior and its contract need - never a copied production sample, never a fixture the test does not read. A mock standing in for an external API response mirrors that response's shape (anti-pattern 4); a field the contract requires is never oversized.
+5. **Setup is measured in work, not lines.** Look for an existing helper, fixture, or factory and extend it before creating another. Share an expensive immutable artifact (a parsed file, a built document, a loaded corpus) at the narrowest scope the harness offers instead of rebuilding it per example; a test that mutates the artifact keeps its own copy. Readability or isolation may justify a rebuild - say so.
+6. **Right tier, honestly.** Test at the cheapest layer that proves the behavior: a behavior a unit or request-level test proves does not get a browser or end-to-end test, and a higher-layer test asserts what only that layer owns (rendering, navigation, a real boundary). Cost and dependency are separate axes: unavoidably expensive work goes in the repo's slow tier, a real external boundary in its integration tier, each by the convention `AGENTS.md`, the overrides file, or the task text names. Never relabel a slow pure-logic test as integration to hide its cost. When no convention is named or discoverable, place the test beside its siblings and report that.
 
 **Requirements:**
-- One behavior per test
-- Clear name describing behavior (if the name contains "and", split it)
+- One scenario per test
+- Clear name describing the scenario
 - Real code (no mocks unless unavoidable)
 - Shows desired API — demonstrates how code should be called
 
@@ -174,14 +181,14 @@ If you catch yourself doing any of these, stop immediately:
 
 Before marking work complete:
 
-- [ ] Every new function/method has a test
+- [ ] Every new behavior is covered - by an extended existing test or a new one
 - [ ] Watched each test fail before implementing
 - [ ] Each test failed for expected reason (feature missing, not typo)
 - [ ] Wrote minimal code to pass each test
 - [ ] The task's `Tests:` commands pass (full suite belongs to the verify phase)
 - [ ] Output pristine (no errors, warnings)
 - [ ] Tests use real code (mocks only if unavoidable)
-- [ ] Edge cases and errors covered
+- [ ] Edge cases and errors covered - as rows or assertions where they share setup
 
 Can't check all boxes? You skipped TDD. Start over.
 

package/skills/test-driven-development/reference/when-stuck.md

@@ -13,14 +13,14 @@
 
 Before marking work complete:
 
-- [ ] Every new function/method has a test
+- [ ] Every new behavior is covered - by an extended existing test or a new one
 - [ ] Watched each test fail before implementing
 - [ ] Each test failed for expected reason (feature missing, not typo)
 - [ ] Wrote minimal code to pass each test
 - [ ] All tests pass
 - [ ] Output pristine (no errors, warnings)
 - [ ] Tests use real code (mocks only if unavoidable)
-- [ ] Edge cases and errors covered
+- [ ] Edge cases and errors covered - as rows or assertions where they share setup
 
 Can't check all boxes? You skipped TDD. Start over.
 

package/skills/test-driven-development/testing-anti-patterns.md

@@ -176,6 +176,8 @@ BEFORE mocking any method:
 
 ## Anti-Pattern 4: Incomplete Mocks
 
+Scope: a mock standing in for an external API response. Internal inputs (records, config, fixtures your own code consumes) follow the minimal-synthetic default in `SKILL.md` RED step rule 4 - the smallest hand-built value the tested behavior and its contract need.
+
 **The violation:**
 ```typescript
 // ❌ BAD: Partial mock - only fields you think you need
@@ -194,7 +196,7 @@ const mockResponse = {
 - **Tests pass but integration fails** - Mock incomplete, real API complete
 - **False confidence** - Test proves nothing about real behavior
 
-**The Iron Rule:** Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses.
+**The Iron Rule:** Mock the COMPLETE external response as it exists in reality, not just fields your immediate test uses.
 
 **The fix:**
 ```typescript
@@ -210,7 +212,7 @@ const mockResponse = {
 ### Gate Function
 
 ```
-BEFORE creating mock responses:
+BEFORE creating a mock of an external API response:
   Check: "What fields does the real API response contain?"
 
   Actions:
@@ -222,7 +224,7 @@ BEFORE creating mock responses:
     If you're creating a mock, you must understand the ENTIRE structure
     Partial mocks fail silently when code depends on omitted fields
 
-  If uncertain: Include all documented fields
+  If uncertain: Include all documented fields of that response
 ```
 
 ## Anti-Pattern 5: Integration Tests as Afterthought

package/skills/writing-plans/SKILL.md

@@ -105,7 +105,7 @@ Group tasks into **waves** so the executor can parallelize independent work (see
 
 **File-ownership contract.** See [reference/plan-contract.md § Files](reference/plan-contract.md).
 
-**Test contract.** Every task that creates or modifies code declares a `Test:` path and an executable `**Tests:**` command anchored to it (grammar: [reference/plan-contract.md § Tests](reference/plan-contract.md)); `- none: <category>` only when no tests apply. Anchoring is presence, not coverage. When the anchored spec names the thing under test, the task carries `- via:` naming it; a fixture path the spec names goes under `Create:`.
+**Test contract.** Every task that creates or modifies code declares a `Test:` path and an executable `**Tests:**` command anchored to it (grammar: [reference/plan-contract.md § Tests](reference/plan-contract.md)); `- none: <category>` only when no tests apply. Anchoring is presence, not coverage. When the anchored spec names the thing under test, the task carries `- via:` naming it; a fixture path the spec names goes under `Create:`. Prefer the existing test file for the behavior as `Test:`; a new test file is declared as both `Create:` and `Test:`.
 
 **Runtime-resource disjointness.** File-disjoint is necessary but not sufficient: two tasks with disjoint files that both mutate the same DB, bind the same port, or share a fixture are **not** parallel-safe and must land in different waves. The executor auto-selects parallel for *every* multi-task wave, so this grouping is the sole parallel-safety guarantee — there is no selection-time judgment downstream. No new mandatory per-task syntax; when a shared runtime resource is the reason two file-disjoint tasks sit in different waves, record it in an inline note on the later wave.
 
@@ -190,9 +190,11 @@ Each task uses `- [ ]` checkbox steps so execution tools (and humans) can track
 - [ ] **Step 1: Write the failing test**
 
   ```python
-  def test_specific_behavior():
-      result = function(input)
-      assert result == expected
+  # add a row to the existing table when one covers the behavior
+  CASES = [
+      ("existing", input, expected),
+      ("new_behavior", new_input, new_expected),
+  ]
   ```
 
 - [ ] **Step 2: Run test, confirm failure**
@@ -262,7 +264,7 @@ After drafting the plan and before announcing it complete, run the deterministic
 - **Deterministic checker.** Run `plan_check({ planPath: "<abs plan path>" })` on the saved plan. Assess and fix every finding yourself (no human involvement), then re-run until it passes — a pass writes the execution stamp that implement-start verifies mechanically. If the same finding survives 3 fix rounds, convert it to an explicit Open Question and stop (the pre-existing Open-Questions halt, resolved by the human in-session — not a new gate). Findings are defined in [reference/plan-contract.md](reference/plan-contract.md).
 - **Code-vs-anchor sanity.** For each task-owned requirement row, re-read the anchored spec lines and confirm the owner tasks' bodies do what they say - mechanism present, not just the quoted literal. For each `Verification` row, confirm the header command exercises the anchored requirement. Fix the task, don't annotate.
 - **Type / API consistency.** Function signatures and field names that appear in multiple tasks must match exactly. The plan is its own contract — internal contradictions surface as bugs during execution.
-- **Test contract.** Every code task's `Tests:` commands are anchored to its `Test:` path(s); `none:` only where no tests apply; a spec-named seam appears as `via:`, a spec-named fixture path as `Create:`.
+- **Test contract.** Every code task's `Tests:` commands are anchored to its `Test:` path(s); `none:` only where no tests apply; a spec-named seam appears as `via:`, a spec-named fixture path as `Create:`. Prefer the existing test file for the behavior as `Test:`; a new test file is declared as both `Create:` and `Test:`.
 - **Runtime-resource disjointness.** For every multi-task wave, confirm no two tasks contend on a shared mutable runtime resource (DB/schema, port, fixture, external service, shared temp path) — `Files:` overlap is checked mechanically, resource contention is not. Contention = mis-grouped wave; split or re-order before handoff.
 - **Solo-reason validity.** Every single-task wave's `Solo:` line (presence is checked mechanically) must name its specific blocker — the blocking task/wave, the contended resource, or `lone remaining task`. Category-only justifications are under-justified; merge or justify before handoff.
 - **Waiver authorization.** `waived: <reason>` is only for requirements the spec marks out of scope **and** that exclude work from the change. A requirement whose text carries an inline code span (`` `literal` ``) names concrete behaviour and is never waivable - it maps to a task or `Verification`. A waiver on an in-scope normative requirement is a Self-Review failure — there is no human plan-review gate to catch it downstream.