pi-gauntlet 7.6.1 → 7.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/README.md +4 -8
- package/agents/code-reviewer.md +9 -1
- package/agents/implementer.md +12 -0
- package/package.json +1 -1
- package/skills/requesting-code-review/code-reviewer.md +1 -2
- package/skills/shape-ticket/SKILL.md +3 -2
- package/skills/subagent-driven-development/implementer-prompt.md +2 -2
- package/skills/test-driven-development/SKILL.md +12 -5
- package/skills/test-driven-development/reference/when-stuck.md +2 -2
- package/skills/test-driven-development/testing-anti-patterns.md +5 -3
- package/skills/writing-plans/SKILL.md +7 -5
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## v7.7.1 - 2026-10-07
|
|
4
|
+
|
|
5
|
+
### Changed
|
|
6
|
+
|
|
7
|
+
- **shape-ticket: model invocation allowed; tracker writes still require the explicit confirmation gate.** `disable-model-invocation: true` is gone from `skills/shape-ticket/SKILL.md`, so another skill or a downstream command can route "file a ticket" to it (Claude Code's Skill tool refused the call before, and `skillOverrides` cannot loosen a plugin skill). The description now triggers on file/create/shape a ticket, open a tracker issue, repair/clean up `ABC-123`. Who starts the pipeline changed; who approves the write did not.
|
|
8
|
+
|
|
9
|
+
## v7.7.0 - 2026-10-07
|
|
10
|
+
|
|
11
|
+
- **Test economy in TDD, implementer, and code review.** The TDD skill's RED step carries six declinable defaults (extend before add, one scenario per test, parametrize equivalent cases, minimal synthetic input, shared expensive setup, cheapest honest tier); `agents/implementer.md` and the SDD self-review carry the imperative view, `agents/code-reviewer.md` and the review template the check view (economy findings are Minor `shrink:`/`delete:`, lost coverage on consolidation is Critical); anti-pattern 4 is scoped to external API responses; writing-plans prefers the existing test file as `Test:`. Two new eval targets, `eval/implementer` and `eval/code-reviewer`, measure the change on five synthetic ledger-import samples (Python and RSpec).
|
|
12
|
+
- **Evals: one driver, paired intent-aware judge, `eval/_template`.** `node eval/run.mjs <target>` replaces the eight per-target runners; two frozen replay models produce before/after outputs, a frozen judge reports per-fact outcomes, and the driver labels each against the target's committed `intent.md` (`held`, `intended change`, `regression`, `unexplained change`). Only the newest `baseline.json`/`candidate.json` per sample is kept; 54 baselines are committed. `forge-skill` runs as `kind: edit`. `AGENTS.md` scopes the mandatory eval to behavior-changing edits; `AGENTS.core.md` v9 bans private data and secrets in tickets, specs, and samples. (#50)
|
|
13
|
+
|
|
3
14
|
## v7.6.1 - 2026-10-07
|
|
4
15
|
|
|
5
16
|
### Changed
|
package/README.md
CHANGED
|
@@ -35,7 +35,7 @@ pi-gauntlet's only hard dependency is pi-cohort - every gate that dispatches a r
|
|
|
35
35
|
|
|
36
36
|
Concretely, one change through the gauntlet:
|
|
37
37
|
|
|
38
|
-
0. *(Optional)* Before there's even a spec, `/skill:shape-ticket` can create or repair a single tracker issue - shaping a raw ask into a Context/Problem/Idea/Acceptance Criteria ticket, gated by an AC integrity check, a cheap council roast, and one human-confirmed write that may include one optional gated Reporter-note comment. A failed roast is retried once, then surfaced inline at the gate if it fails again. It's a tool, not a phase: no worktree, no plan/phase tracker, runs from any repo state.
|
|
38
|
+
0. *(Optional)* Before there's even a spec, `/skill:shape-ticket` can create or repair a single tracker issue - shaping a raw ask into a Context/Problem/Idea/Acceptance Criteria ticket, gated by an AC integrity check, a cheap council roast, and one human-confirmed write that may include one optional gated Reporter-note comment. A failed roast is retried once, then surfaced inline at the gate if it fails again. It's a tool, not a phase: no worktree, no plan/phase tracker, runs from any repo state. The agent may start it on its own when the ask is "file a ticket" or another skill routes there; the write still waits for your yes on the presented diff. Similarly, `/skill:chase-bug` triages a raw bug report into an evidenced verdict - and can hand off to shape-ticket, brainstorming, or a bounded hotfix - before any spec exists.
|
|
39
39
|
1. Opt in with `/skill:brainstorming` and describe the change. The command is the reliable discovery-and-entry path; explicit prose requesting brainstorming/the gauntlet also supplies consent when the skill is already loaded or an available instruction or route identifies it. An ordinary implementation request or an agent recommendation is not consent. **`brainstorming`** sets up an isolated worktree, explores the codebase, and turns your description into a written spec - when its recon finds a cited case that the ask is the wrong change (misdiagnosed, wrong-sized, already solved, a band-aid, or cutting across a sound pattern - including a repo convention that is itself debt) and can name the alternative, question one is a framing question (`as framed`, `pivot: <alternative>`, or `smaller or none`); otherwise it opens with one sentence naming what it checked, and every approach it proposes ends with a `Pattern:` line naming what it follows or cuts across; when the run starts from a ticket, the spec carries the ticket's acceptance criteria verbatim, each with a disposition (`in-scope`, `deviates:`, `deferred:`, `venue:`), the conformance gate checks the in-scope ones, and `/skill:check-delivery` verifies `venue:` rows after deploy. A multi-model critique runs on it automatically. If the spec replaces a known prior spec, brainstorming marks the predecessor with a `> **Superseded by:**` banner under its title (default format, syntax overridable via `.pi/gauntlet-overrides.md`; candidates come from brainstorming's scout recon, never a mechanical sweep). **You read and approve the spec - human gate 1.** No implementation code exists yet.
|
|
40
40
|
2. **`writing-plans`** decomposes the approved spec into atomic, independently-verifiable tasks, grouped into parallel waves where they don't touch the same files.
|
|
41
41
|
3. **`subagent-driven-development`** executes the plan one task at a time, each in a fresh subagent, behind spec-compliance review then code-quality review. TDD-locked: red, green, refactor. Independent implementation, review, and council batches remain parallel, but their dispatches are foreground: the orchestrator waits for terminal results before accepting work or advancing a phase.
|
|
@@ -47,7 +47,7 @@ Only the machine-owned `plan -> implement` and `verify -> ship` handoffs receive
|
|
|
47
47
|
|
|
48
48
|
```mermaid
|
|
49
49
|
flowchart LR
|
|
50
|
-
T["shape-ticket<br/>(optional
|
|
50
|
+
T["shape-ticket<br/>(optional)"]
|
|
51
51
|
R([explicit opt-in]) --> B[brainstorm<br/>+ spec]
|
|
52
52
|
T -.-> R
|
|
53
53
|
B --> G1{{human gate 1:<br/>approve spec}}
|
|
@@ -69,7 +69,7 @@ Everything between gate 1 and gate 2 - task breakdown, implementation, both revi
|
|
|
69
69
|
|
|
70
70
|
pi-gauntlet ships three kinds of pieces, layered on top of pi-cohort's dispatch:
|
|
71
71
|
|
|
72
|
-
- **20 skills** - the workflow logic.
|
|
72
|
+
- **20 skills** - the workflow logic. Thirteen activate automatically when pi sees the matching kind of task, and the workflow skills chain through their gates: `shape-ticket` (creates or repairs one tracker issue per run against a Context/Problem/Idea/Acceptance-Criteria template, gated by an AC integrity check, a cheap council roast, and a single human-confirmed write; model-invocable so other skills and commands can route "file a ticket" to it, the gate still needs your yes), `writing-plans`, `roasting-the-spec`, `test-driven-development`, `subagent-driven-development`, `dispatching-parallel-agents`, `verification-before-completion`, `requesting-code-review`, `receiving-code-review`, `using-git-worktrees`, `finishing-a-development-branch`, `forge-skill`, `linear` (reads/searches/comments on/manages Linear tickets via the `linearis` CLI; owns all linearis mechanics and the `## Issue tracker` overrides schema; tracker-facing skills route to it). Seven more require explicit entry (`disable-model-invocation: true`): `brainstorming` starts the gated workflow on explicit consent - use `/skill:brainstorming` for reliable discovery, or opt in through reachable explicit prose or a human-selected handoff; after spec approval, downstream planning and implementation still chain automatically. `gatekeep-pr` is consent-gated pre-merge verification of a PR against the scope contract at its head - the approved spec's acceptance-criteria dispositions and Design when the PR carries one, else the PR's stated intent; the ticket is cross-checked, never judged here (`check-delivery` owns it) - read-only gathering, verification evidence resolved CI-first (green checks on the exact assessed head count as evidence; a pending commit status that GitHub does not report as blocking the gating actor (`mergeStateStatus` not `BLOCKED`) is not waited for - external-CI commit statuses included - while a running Actions check, an unknown merge state, and every other merge precondition still withhold; the project's verification command runs only as fallback), a rubric-based review, a compact bottom-up report (what the PR delivers, including rows deferred or deviating per spec, PR comments, nits, blockers, then a `mergeable` / `fixable` verdict) and a verb menu whose rows derive from the actor's permissions on the PR; nothing mutates (fixes, pushes, reviews, merges, tracker comments) until you pick a row, except the opt-in `--rebase [base]` sync that rebases, reviews, and lease-pushes the head before verification, or holds the rebased head locally when a conflict stop's scoped tests fail, so the normal `fix` row resolves it before the lease push; after your own merge two opt-in rows post the `AC coverage` block to the ticket or the PR - run it with `/skill:gatekeep-pr <pr> [--rebase [base]]`. `check-delivery` is a post-merge detective control: proves an issue actually shipped (default-branch landing, delivery target, per-AC evidence) before its tracker status advances; it never writes a terminal status - run it with `/skill:check-delivery <ref>`. `chase-bug` is human-only bug triage: read-only root-cause discovery to an evidenced verdict menu (real bug -> ticket/brainstorm/hotfix/respond; five negative verdicts), then a gated response to the reporter for addressable origins (GitHub issue / tracker ticket) and a rendered verdict summary otherwise - it never fixes during triage; the hotfix row hands off to `skills/chase-bug/hotfix.md` after the menu - run it with `/skill:chase-bug`. `gauntlet-performance` reads the committed run telemetry (current repo by default, `--dir <path>` adds others) through the parse-only `gauntlet-performance` CLI and answers with one example-led recommendation, 3-5 cornerstone numbers, a council assessment when available, and a menu of at most three actions (render a report file, open the recommendation as a ticket via `shape-ticket`, drill into one run); it writes nothing unless you pick render - run it with `/skill:gauntlet-performance [--dir <path>]... [--since <version>]`. `gauntlet-handoff` ends a session whose flow a fresh session will continue: it invokes pi-cohort's `handoff` skill (`--out <path>` or `--key <name>`, default key = the run worktree's branch with `/` flattened to `-`, default file `<tmpdir>/pi-handoff/<key>.md`), then appends the gauntlet process-state section (phase/plan tracker status) per the shared contract `skills/gauntlet-resume/reference/brief-contract.md` - run it with `/skill:gauntlet-handoff [--out <path> | --key <name>]`. `gauntlet-resume` is the only way back into an interrupted flow from a fresh session: it takes a `gauntlet-handoff` brief (a file path, pasted text, or - with no arguments - a pick from the briefs under `<tmpdir>/pi-handoff/`), a bare worktree that already holds a spec, or a spec tracked on main under a configured spec dir (a spec seed: it creates or reuses the worktree through `using-git-worktrees` and pins the spec), restores phase/plan tracker state through the legal arming sequence (`start brainstorm`, `skip` with `resume:` reasons, `plan_check` before implement-or-later), and never infers approval from artifacts - run it with `/skill:gauntlet-resume [<brief-file> | <spec>.md | <worktree-name-or-path>]`.
|
|
73
73
|
- **7 subagent personas** - the specialized child agents the skills dispatch via pi-cohort: `implementer`, `code-reviewer`, `spec-reviewer`, `conformance-reviewer`, `spec-summarizer`, `spec-council-member`, `spec-council-synthesizer`. See [doc/personas.md](./doc/personas.md) for what each one does and why its permissions are scoped the way they are.
|
|
74
74
|
- **4 runtime extensions** - the enforcement layer. `plan-tracker` and `phase-tracker` are tools skills call to track progress (with a TUI widget); `verify-before-ship` is a hook that warns if you push or open a PR without a passing test run since your last edit; a phase-tracker flow guard reminds on implement-phase commits missing spec/code review. In a brainstorming-entered flow, phase-tracker rejects `implement` or `verify` completion while tracker tasks remain pending or in progress; see [its configuration reference](./doc/configuration.md#phase-tracker). phase-tracker also registers `plan_check`, which verifies a plan against its spec and against the grammar in [skills/writing-plans/reference/plan-contract.md](./skills/writing-plans/reference/plan-contract.md), including that each task's `Tests:` commands are selective and never the full suite; a pass stamps the plan for implementation. `telemetry` records one YAML record per gauntlet run (phase timing, models, personas, gate/fix rounds, diff at ship) that ships in the squash beside the spec - it is written only inside a brainstorming-entered flow, kept out of history until `finishing-a-development-branch` seals it once with `gauntlet-telemetry-seal` (a `telemetry:` commit under git; the working-copy snapshot in a plain jj workspace) - and blocks a brainstorm `write` into an already-shipped spec; see [its configuration reference](./doc/configuration.md#telemetry). See [doc/configuration.md](./doc/configuration.md) for the settings each one reads.
|
|
75
75
|
|
|
@@ -155,11 +155,7 @@ pi install -l ~/repos/pi-gauntlet
|
|
|
155
155
|
|
|
156
156
|
## Evals
|
|
157
157
|
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
Run an arm with externally supplied models: `node eval/spec-summarizer/run.mjs run --arm candidate --persona agents/spec-summarizer.md --candidate-model <candidate-id> --reviewers <reviewer-a>,<reviewer-b>`. The [eval convention](eval/README.md) owns the layout, baseline/candidate process, and publication rules; the [spec-summarizer eval](eval/spec-summarizer/README.md) is the worked example with eight samples and committed results. Model calls run manually, never in CI; CI runs deterministic eval tests and hygiene checks. `eval/` is checkout-only and excluded from the npm tarball.
|
|
161
|
-
|
|
162
|
-
`eval/forge-skill/` measures an edit to `skills/forge-skill/SKILL.md`: `node eval/forge-skill/run.mjs --skill-dir <path> --candidate-model <id> --reviewers <id>,<id>` runs a worker over ten samples (four skill fixtures, six persona fixtures - four from this repo's `agents/` history, two constructed to exercise the persona rules), has two reviewer models vote per must-hold fact, and writes records under `$TMPDIR`; `run.mjs compare <baseline> <candidate>` refuses mismatched inputs and fails on any fact that moved from kept to lost. Models are arguments, never committed. Its samples are `case.md` + `expected.md` and its raw results stay under `$TMPDIR` - the two places it deviates from the convention. Process and sample grammar: [`eval/forge-skill/README.md`](eval/forge-skill/README.md).
|
|
158
|
+
Run `node eval/run.mjs <target>` to check a wording change on fixed inputs: two frozen replay models produce before/after outputs, and one frozen judge checks must-hold facts against a committed `intent.md`. The [eval process](eval/README.md) explains the single driver, target and sample layout, newest-only records, and anonymization rule. Model calls run manually, never in CI; CI runs deterministic library tests, structural lint, and hygiene checks. `eval/` is checkout-only and excluded from the npm tarball.
|
|
163
159
|
|
|
164
160
|
## Use from Claude Code
|
|
165
161
|
|
package/agents/code-reviewer.md
CHANGED
|
@@ -14,7 +14,15 @@ You are a code reviewer. You find issues before they ship. You **do not edit cod
|
|
|
14
14
|
## Review priorities, in order
|
|
15
15
|
|
|
16
16
|
1. **Correctness** — Does the change do what it claims? Are edge cases handled? Off-by-one, null/undefined, empty input, concurrency.
|
|
17
|
-
2. **Tests** — Is the new behavior covered? Are tests meaningful, or do they only assert type shape?
|
|
17
|
+
2. **Tests** — Is the new behavior covered? Are tests meaningful, or do they only assert type shape? Name any missing behavior worth a test concretely; never a reflexive "add edge cases". Then check added **and changed** tests for economy - each of these is a `[Minor]` finding the implementer may decline with a reason, except the last:
|
|
18
|
+
- several new tests repeat one setup and operation with the assertions split across them -> `shrink:` to one test with the assertions together.
|
|
19
|
+
- a new test repeats an existing test's assertions -> `delete:`.
|
|
20
|
+
- equivalent cases written as separate tests -> `shrink:` to one table with named rows.
|
|
21
|
+
- input or setup larger than the tested behavior and its contract need -> `shrink:` naming the fields or records the path reads. A mock of an external API response keeps that response's shape; a field the contract requires is never oversized.
|
|
22
|
+
- an expensive artifact (parsed file, built document, loaded corpus) rebuilt per test -> `shrink:` naming the shared scope.
|
|
23
|
+
- a slow pure-logic test labeled integration -> name the repo's tier for it.
|
|
24
|
+
- a test at a higher layer than the one that proves the behavior (a browser test asserting a model rule, a request test asserting a pure function) -> `shrink:` naming the lower layer that owns it.
|
|
25
|
+
- a consolidation whose combined assertions no longer cover a behavior the removed tests covered -> `[Critical]`, lost coverage; judge by behavior covered, not by assertion count.
|
|
18
26
|
3. **Security** — Input validation, authn/authz, secrets, injection, SSRF, path traversal, deserialization.
|
|
19
27
|
4. **Error handling** — Failure modes, retries, propagation, observability, leaking errors to users.
|
|
20
28
|
5. **Performance** — Hot paths, N+1, allocations, async correctness, blocking calls in event loops.
|
package/agents/implementer.md
CHANGED
|
@@ -24,6 +24,18 @@ You are an implementation specialist. You execute an approved plan using strict
|
|
|
24
24
|
- **Modifying tested code** → run the existing tests first, modify, re-run; add new tests only for new behavior.
|
|
25
25
|
- **Trivial change** (typo, comment, formatting) → use judgment; if a dispatch-supplied test command touches the surface, run it.
|
|
26
26
|
|
|
27
|
+
## Test economy
|
|
28
|
+
|
|
29
|
+
Tests are supporting constructs for the code. Follow these defaults; decline one with a one-sentence reason in your report.
|
|
30
|
+
|
|
31
|
+
- Read the existing test file for the behavior before writing RED. Add a parameter row or a stronger assertion to an existing test when one covers the behavior; add a new test function for a distinct scenario, not for every function or statement.
|
|
32
|
+
- Treat several assertions on one operation's result as one scenario. Split on different setup or a different operation, not on a name containing "and".
|
|
33
|
+
- Write near-identical cases with the same setup and assertion shape as one table with named rows; rows that need different setup or a different assertion are separate tests. Keep the few rows that cover the distinct outcomes, not the full product of inputs; parametrization removes duplication, not run time.
|
|
34
|
+
- Build test input by hand: the smallest value the tested behavior and its contract need. Copy no production sample and load no fixture the test does not read. A mock standing in for an external API response keeps that response's shape.
|
|
35
|
+
- Share an expensive immutable artifact (a parsed file, a built document, a loaded corpus) at the narrowest scope the harness offers instead of rebuilding it per test; a test that mutates it keeps its own copy. Rebuild for readability or isolation only with the reason stated.
|
|
36
|
+
- Test at the cheapest layer that proves the behavior; a higher-layer test asserts what only that layer owns (rendering, navigation, a real boundary). Put unavoidably expensive work in the repo's slow tier and a real external boundary in its integration tier, by the convention `AGENTS.md`, the overrides file, or your task names; never relabel a slow pure-logic test as integration. No convention named or discoverable: place the test beside its siblings and say so in your report.
|
|
37
|
+
- A test that must stay heavy because the behavior needs the full input, or a rebuild kept for isolation, goes in your report under deviations with the reason.
|
|
38
|
+
|
|
27
39
|
## Hard rules
|
|
28
40
|
|
|
29
41
|
- Run ONLY the test commands your dispatch hands you (`SCOPED_TEST_COMMANDS`). Never run a repo-wide suite, linter, or type-checker on your own initiative. Dispatch carries no test commands: say so in your report; run nothing.
|
package/package.json
CHANGED
|
@@ -62,8 +62,7 @@ git diff {BASE_SHA}..{HEAD_SHA}
|
|
|
62
62
|
|
|
63
63
|
**Testing:**
|
|
64
64
|
- Tests actually test logic (not mocks)?
|
|
65
|
-
-
|
|
66
|
-
- Integration tests where needed?
|
|
65
|
+
- Each added or changed test earns its run: no repeat of an existing test's setup and operation, input no larger than the behavior and its contract need (external-API mocks keep their shape), expensive setup shared, right tier?
|
|
67
66
|
- Scoped test commands passing (quote actual output; if `none` supplied, note the orchestrator gate owns execution)?
|
|
68
67
|
|
|
69
68
|
**Requirements:**
|
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: shape-ticket
|
|
3
|
-
description: Use when
|
|
4
|
-
disable-model-invocation: true
|
|
3
|
+
description: Use when asked to file, create, or shape a ticket, open a tracker issue, or repair or clean up an existing one (ticket IDs like ABC-123) - including when another skill or command routes ticket authoring here.
|
|
5
4
|
---
|
|
6
5
|
|
|
7
6
|
# Shape Ticket
|
|
@@ -21,6 +20,8 @@ disable-model-invocation: true
|
|
|
21
20
|
|
|
22
21
|
Every path that writes ends at the **same single confirmation gate** - no write happens without an explicit yes on the exact presented diff. The no-op path stops before any gate opens; the hard-stop path aborts and never opens one either.
|
|
23
22
|
|
|
23
|
+
Model-initiated invocation is allowed: another skill, a command, or an ask matching the description starts this pipeline without a human typing the command. That changes who starts the pipeline, never who approves the write - the human answers the gate.
|
|
24
|
+
|
|
24
25
|
## Overview
|
|
25
26
|
|
|
26
27
|
One process, two entry points: **create** a new tracker issue from a prompt, or **repair** an existing one by re-fetching it and proposing a full replacement. Repair is never a silent patch - it is always shown old->new, links and tracker fields preserved. Running this skill on any ticket trues it up; repeated use is a self-healing backlog pass.
|
|
@@ -104,7 +104,7 @@ Dispatch a subagent with this prompt:
|
|
|
104
104
|
**Testing:**
|
|
105
105
|
- Do tests actually verify behavior (not just mock behavior)?
|
|
106
106
|
- Did I follow TDD (step 2)?
|
|
107
|
-
-
|
|
107
|
+
- Does each added test earn its run - extends an existing test where one covers the behavior, minimal synthetic input, shared expensive setup, right tier?
|
|
108
108
|
|
|
109
109
|
If you find issues during self-review, fix them now before reporting.
|
|
110
110
|
|
|
@@ -121,7 +121,7 @@ Dispatch a subagent with this prompt:
|
|
|
121
121
|
- Self-review findings (if any)
|
|
122
122
|
- Any issues or concerns
|
|
123
123
|
|
|
124
|
-
Use `DONE_WITH_CONCERNS` if you completed the work but have doubts about correctness.
|
|
124
|
+
Use `DONE_WITH_CONCERNS` if you completed the work but have doubts about correctness, or when you kept a test heavy or rebuilt its setup on purpose (a declined test-economy default) - name it under issues or concerns with the reason.
|
|
125
125
|
Use `BLOCKED` if you cannot complete the task. Use `NEEDS_CONTEXT` if you need
|
|
126
126
|
information that wasn't provided. Never silently produce work you're unsure about.
|
|
127
127
|
```
|
|
@@ -60,11 +60,18 @@ Implement fresh from tests. Period.
|
|
|
60
60
|
|
|
61
61
|
Write one minimal test showing what should happen.
|
|
62
62
|
|
|
63
|
-
|
|
63
|
+
Test economy - defaults a competent engineer follows and may decline with a stated reason:
|
|
64
|
+
|
|
65
|
+
1. **Extend before add.** Read the existing test file for the behavior first. Prefer a new parameter row or a stronger assertion in an existing test; add a new test function for a distinct scenario, not for every function or statement.
|
|
66
|
+
2. **One scenario per test.** Several assertions on one operation's result are one scenario. A name containing "and" is not by itself a split signal; different setup or a different operation is.
|
|
67
|
+
3. **Parametrize equivalent cases.** Near-identical cases with the same setup and assertion shape are one table with named rows; rows that need different setup or a different assertion are separate tests. Prefer the few rows that cover the distinct outcomes over the full product of inputs. Parametrization removes duplication, not run time.
|
|
68
|
+
4. **Minimal synthetic input.** Test input is the smallest hand-built value the tested behavior and its contract need - never a copied production sample, never a fixture the test does not read. A mock standing in for an external API response mirrors that response's shape (anti-pattern 4); a field the contract requires is never oversized.
|
|
69
|
+
5. **Setup is measured in work, not lines.** Look for an existing helper, fixture, or factory and extend it before creating another. Share an expensive immutable artifact (a parsed file, a built document, a loaded corpus) at the narrowest scope the harness offers instead of rebuilding it per example; a test that mutates the artifact keeps its own copy. Readability or isolation may justify a rebuild - say so.
|
|
70
|
+
6. **Right tier, honestly.** Test at the cheapest layer that proves the behavior: a behavior a unit or request-level test proves does not get a browser or end-to-end test, and a higher-layer test asserts what only that layer owns (rendering, navigation, a real boundary). Cost and dependency are separate axes: unavoidably expensive work goes in the repo's slow tier, a real external boundary in its integration tier, each by the convention `AGENTS.md`, the overrides file, or the task text names. Never relabel a slow pure-logic test as integration to hide its cost. When no convention is named or discoverable, place the test beside its siblings and report that.
|
|
64
71
|
|
|
65
72
|
**Requirements:**
|
|
66
|
-
- One
|
|
67
|
-
- Clear name describing
|
|
73
|
+
- One scenario per test
|
|
74
|
+
- Clear name describing the scenario
|
|
68
75
|
- Real code (no mocks unless unavoidable)
|
|
69
76
|
- Shows desired API — demonstrates how code should be called
|
|
70
77
|
|
|
@@ -174,14 +181,14 @@ If you catch yourself doing any of these, stop immediately:
|
|
|
174
181
|
|
|
175
182
|
Before marking work complete:
|
|
176
183
|
|
|
177
|
-
- [ ] Every new
|
|
184
|
+
- [ ] Every new behavior is covered - by an extended existing test or a new one
|
|
178
185
|
- [ ] Watched each test fail before implementing
|
|
179
186
|
- [ ] Each test failed for expected reason (feature missing, not typo)
|
|
180
187
|
- [ ] Wrote minimal code to pass each test
|
|
181
188
|
- [ ] The task's `Tests:` commands pass (full suite belongs to the verify phase)
|
|
182
189
|
- [ ] Output pristine (no errors, warnings)
|
|
183
190
|
- [ ] Tests use real code (mocks only if unavoidable)
|
|
184
|
-
- [ ] Edge cases and errors covered
|
|
191
|
+
- [ ] Edge cases and errors covered - as rows or assertions where they share setup
|
|
185
192
|
|
|
186
193
|
Can't check all boxes? You skipped TDD. Start over.
|
|
187
194
|
|
|
@@ -13,14 +13,14 @@
|
|
|
13
13
|
|
|
14
14
|
Before marking work complete:
|
|
15
15
|
|
|
16
|
-
- [ ] Every new
|
|
16
|
+
- [ ] Every new behavior is covered - by an extended existing test or a new one
|
|
17
17
|
- [ ] Watched each test fail before implementing
|
|
18
18
|
- [ ] Each test failed for expected reason (feature missing, not typo)
|
|
19
19
|
- [ ] Wrote minimal code to pass each test
|
|
20
20
|
- [ ] All tests pass
|
|
21
21
|
- [ ] Output pristine (no errors, warnings)
|
|
22
22
|
- [ ] Tests use real code (mocks only if unavoidable)
|
|
23
|
-
- [ ] Edge cases and errors covered
|
|
23
|
+
- [ ] Edge cases and errors covered - as rows or assertions where they share setup
|
|
24
24
|
|
|
25
25
|
Can't check all boxes? You skipped TDD. Start over.
|
|
26
26
|
|
|
@@ -176,6 +176,8 @@ BEFORE mocking any method:
|
|
|
176
176
|
|
|
177
177
|
## Anti-Pattern 4: Incomplete Mocks
|
|
178
178
|
|
|
179
|
+
Scope: a mock standing in for an external API response. Internal inputs (records, config, fixtures your own code consumes) follow the minimal-synthetic default in `SKILL.md` RED step rule 4 - the smallest hand-built value the tested behavior and its contract need.
|
|
180
|
+
|
|
179
181
|
**The violation:**
|
|
180
182
|
```typescript
|
|
181
183
|
// ❌ BAD: Partial mock - only fields you think you need
|
|
@@ -194,7 +196,7 @@ const mockResponse = {
|
|
|
194
196
|
- **Tests pass but integration fails** - Mock incomplete, real API complete
|
|
195
197
|
- **False confidence** - Test proves nothing about real behavior
|
|
196
198
|
|
|
197
|
-
**The Iron Rule:** Mock the COMPLETE
|
|
199
|
+
**The Iron Rule:** Mock the COMPLETE external response as it exists in reality, not just fields your immediate test uses.
|
|
198
200
|
|
|
199
201
|
**The fix:**
|
|
200
202
|
```typescript
|
|
@@ -210,7 +212,7 @@ const mockResponse = {
|
|
|
210
212
|
### Gate Function
|
|
211
213
|
|
|
212
214
|
```
|
|
213
|
-
BEFORE creating mock
|
|
215
|
+
BEFORE creating a mock of an external API response:
|
|
214
216
|
Check: "What fields does the real API response contain?"
|
|
215
217
|
|
|
216
218
|
Actions:
|
|
@@ -222,7 +224,7 @@ BEFORE creating mock responses:
|
|
|
222
224
|
If you're creating a mock, you must understand the ENTIRE structure
|
|
223
225
|
Partial mocks fail silently when code depends on omitted fields
|
|
224
226
|
|
|
225
|
-
If uncertain: Include all documented fields
|
|
227
|
+
If uncertain: Include all documented fields of that response
|
|
226
228
|
```
|
|
227
229
|
|
|
228
230
|
## Anti-Pattern 5: Integration Tests as Afterthought
|
|
@@ -105,7 +105,7 @@ Group tasks into **waves** so the executor can parallelize independent work (see
|
|
|
105
105
|
|
|
106
106
|
**File-ownership contract.** See [reference/plan-contract.md § Files](reference/plan-contract.md).
|
|
107
107
|
|
|
108
|
-
**Test contract.** Every task that creates or modifies code declares a `Test:` path and an executable `**Tests:**` command anchored to it (grammar: [reference/plan-contract.md § Tests](reference/plan-contract.md)); `- none: <category>` only when no tests apply. Anchoring is presence, not coverage. When the anchored spec names the thing under test, the task carries `- via:` naming it; a fixture path the spec names goes under `Create:`.
|
|
108
|
+
**Test contract.** Every task that creates or modifies code declares a `Test:` path and an executable `**Tests:**` command anchored to it (grammar: [reference/plan-contract.md § Tests](reference/plan-contract.md)); `- none: <category>` only when no tests apply. Anchoring is presence, not coverage. When the anchored spec names the thing under test, the task carries `- via:` naming it; a fixture path the spec names goes under `Create:`. Prefer the existing test file for the behavior as `Test:`; a new test file is declared as both `Create:` and `Test:`.
|
|
109
109
|
|
|
110
110
|
**Runtime-resource disjointness.** File-disjoint is necessary but not sufficient: two tasks with disjoint files that both mutate the same DB, bind the same port, or share a fixture are **not** parallel-safe and must land in different waves. The executor auto-selects parallel for *every* multi-task wave, so this grouping is the sole parallel-safety guarantee — there is no selection-time judgment downstream. No new mandatory per-task syntax; when a shared runtime resource is the reason two file-disjoint tasks sit in different waves, record it in an inline note on the later wave.
|
|
111
111
|
|
|
@@ -190,9 +190,11 @@ Each task uses `- [ ]` checkbox steps so execution tools (and humans) can track
|
|
|
190
190
|
- [ ] **Step 1: Write the failing test**
|
|
191
191
|
|
|
192
192
|
```python
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
193
|
+
# add a row to the existing table when one covers the behavior
|
|
194
|
+
CASES = [
|
|
195
|
+
("existing", input, expected),
|
|
196
|
+
("new_behavior", new_input, new_expected),
|
|
197
|
+
]
|
|
196
198
|
```
|
|
197
199
|
|
|
198
200
|
- [ ] **Step 2: Run test, confirm failure**
|
|
@@ -262,7 +264,7 @@ After drafting the plan and before announcing it complete, run the deterministic
|
|
|
262
264
|
- **Deterministic checker.** Run `plan_check({ planPath: "<abs plan path>" })` on the saved plan. Assess and fix every finding yourself (no human involvement), then re-run until it passes — a pass writes the execution stamp that implement-start verifies mechanically. If the same finding survives 3 fix rounds, convert it to an explicit Open Question and stop (the pre-existing Open-Questions halt, resolved by the human in-session — not a new gate). Findings are defined in [reference/plan-contract.md](reference/plan-contract.md).
|
|
263
265
|
- **Code-vs-anchor sanity.** For each task-owned requirement row, re-read the anchored spec lines and confirm the owner tasks' bodies do what they say - mechanism present, not just the quoted literal. For each `Verification` row, confirm the header command exercises the anchored requirement. Fix the task, don't annotate.
|
|
264
266
|
- **Type / API consistency.** Function signatures and field names that appear in multiple tasks must match exactly. The plan is its own contract — internal contradictions surface as bugs during execution.
|
|
265
|
-
- **Test contract.** Every code task's `Tests:` commands are anchored to its `Test:` path(s); `none:` only where no tests apply; a spec-named seam appears as `via:`, a spec-named fixture path as `Create:`.
|
|
267
|
+
- **Test contract.** Every code task's `Tests:` commands are anchored to its `Test:` path(s); `none:` only where no tests apply; a spec-named seam appears as `via:`, a spec-named fixture path as `Create:`. Prefer the existing test file for the behavior as `Test:`; a new test file is declared as both `Create:` and `Test:`.
|
|
266
268
|
- **Runtime-resource disjointness.** For every multi-task wave, confirm no two tasks contend on a shared mutable runtime resource (DB/schema, port, fixture, external service, shared temp path) — `Files:` overlap is checked mechanically, resource contention is not. Contention = mis-grouped wave; split or re-order before handoff.
|
|
267
269
|
- **Solo-reason validity.** Every single-task wave's `Solo:` line (presence is checked mechanically) must name its specific blocker — the blocking task/wave, the contended resource, or `lone remaining task`. Category-only justifications are under-justified; merge or justify before handoff.
|
|
268
270
|
- **Waiver authorization.** `waived: <reason>` is only for requirements the spec marks out of scope **and** that exclude work from the change. A requirement whose text carries an inline code span (`` `literal` ``) names concrete behaviour and is never waivable - it maps to a task or `Verification`. A waiver on an in-scope normative requirement is a Self-Review failure — there is no human plan-review gate to catch it downstream.
|