pi-gauntlet 7.1.2 → 7.2.0

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## v7.2.0 - 2026-10-04
+
+### Changed
+
+- **spec-summarizer: briefing instead of decision log.** The persona emits a plain-language briefing of at most 300 words, written to about 220: entry paragraph, What changes, Approval risks, Done when, and Missing from the spec when external context is missing. It uses behavior and outcomes instead of code identifiers. The brainstorming gate drops the `under ~2%` degrade check; file-only transport, Read-last rendering, and the absolute size checks stay.
+- **Non-trivial skill, persona, and prompt edits create or extend an eval.** `AGENTS.md` requires samples and must-hold facts designed during brainstorming and a baseline run before the wording changes; existing targets gain a sample exercising the change. Trivial-edit exemptions stay unchanged, and deterministic code stays on `npm test`. `README.md` routes the procedure to `eval/README.md`.
+
+### Added
+
+- **Committed eval convention and spec-summarizer replay set.** `eval/README.md` defines fixed inputs, human-approved facts, independent reviewers with models supplied externally, and recorded baseline/candidate comparisons. `eval/spec-summarizer/` includes eight samples (two anonymized product specs) and first-run results against the persona pinned at `2166283`: 9,497 -> 2,235 words, 327 -> 0 backticked lines, and quality from `mixed` on 15/16 votes to `readable` or `briefing` on 12/16. Two of 47 facts were lost and reviewed; the comparison permits at most two losses across a run and lists each for human review. `eval/` stays outside the npm tarball; model calls are manual.
+- **Deterministic eval checks in CI.** `scripts/ci.mjs` runs `eval/spec-summarizer/run.test.mjs` and an `eval/` hygiene scan without model calls.
+
 ## v7.1.2 - 2026-10-04
 
 ### Changed

package/README.md

@@ -153,6 +153,12 @@ cd ~/path/to/your/repo
 pi install -l ~/repos/pi-gauntlet
 ```
 
+## Evals
+
+Evals compare prompt wording on fixed inputs against human-approved must-hold facts. A non-trivial edit under `skills/`, `agents/`, or a prompt file creates or extends `eval/<target>/`; design its samples and facts during brainstorming, and run the baseline on current wording before the edit. Existing targets gain a sample exercising the changed behavior. Typo, formatting, dependency bump, and release-commit edits are exempt; deterministic code under `extensions/`, `src/bins/`, and `scripts/` stays on `npm test`.
+
+Run an arm with externally supplied models: `node eval/spec-summarizer/run.mjs run --arm candidate --persona agents/spec-summarizer.md --candidate-model <candidate-id> --reviewers <reviewer-a>,<reviewer-b>`. The [eval convention](eval/README.md) owns the layout, baseline/candidate process, and publication rules; the [spec-summarizer eval](eval/spec-summarizer/README.md) is the worked example with eight samples and committed results. Model calls run manually, never in CI; CI runs deterministic eval tests and hygiene checks. `eval/` is checkout-only and excluded from the npm tarball.
+
 ## Use from Claude Code
 
 Six skills are exposed to Claude Code via the plugin marketplace at

package/agents/spec-summarizer.md

@@ -1,6 +1,6 @@
 ---
 name: spec-summarizer
-description: Produces a tight, human-readable summary of a single spec for the brainstorming user review gate. Fresh context, read-only, reads only the spec file it is given. Dispatched only by the brainstorming skill's gate step; not for direct dispatch.
+description: Produces a plain-language briefing (at most 300 words) of a single spec for the brainstorming user review gate. Fresh context, read-only, reads only the spec file it is given. Dispatched only by the brainstorming skill's gate step; not for direct dispatch.
 tools: read
 defaultContext: fresh
 inheritProjectContext: false
@@ -11,33 +11,23 @@ systemPromptMode: replace
 
 You are a cold reader producing a tight, human-readable summary of one spec, so a supervising human can green-light plan + execution without reading the whole document. The summary is a decision aid, not a rewrite.
 
-You receive the absolute path to a spec file. Read **only that file**. Do not read any other file, do not grep, find, ls, or explore the codebase, and do not infer anything beyond what the spec states. If the spec references external context it does not contain (a ticket, an acceptance criterion, a commit SHA, another doc), do not invent it - list it in the gap footer.
+You receive the absolute path to a spec file. Read **only that file**. Do not read any other file, do not grep, find, ls, or explore the codebase, and do not infer anything beyond what the spec states. If the spec references external context it does not contain (a ticket, an acceptance criterion, a commit SHA, another doc), do not invent it - list it under Missing from the spec, by its plain-language title (the ticket's subject, the document's name), never by SHA or path.
 
-Your output is judged on whether a busy supervisor can decide from it alone. A thin or confused summary is a faithful signal that the spec itself is thin - do not paper over gaps to look complete.
+Your output is judged on whether a busy supervisor can decide from it alone. A confused briefing, or one whose Missing from the spec block is long, is a faithful signal that the spec itself is thin - do not paper over gaps to look complete.
 
 ## What to emit
 
-Optimize for "what does a supervisor need to approve **this** spec." The list below is a recommended checklist, not a rigid template:
+Emit a briefing of at most 300 words in total - write to about 220, since measured counts run a fifth over an estimate and the cap is checked on the measured count - in this order, each part within its own budget:
 
-- **Scale length to the spec.** The summary is proportional to the spec's size and complexity - a short or simple spec gets a short summary. Do not expand every recommended section to full depth to look thorough; the summary should be a fraction of the spec, not a near-copy of it. This is proportionality, **not** aggressive compression - never drop a decision-relevant point, rejected-alternative, risk, or gap-footer entry to hit a length target. When in doubt, keep the point and cut the words around it.
-- **Omit any section that is empty.** A bugfix has no new endpoint; a refactor has no algorithm. Write nothing for an empty section - never "N/A" or filler.
-- **Order decision-layer-first** (problem -> decisions -> scope -> risk), then the descriptive layer, so the reader can stop early once confident.
-- **Add a section the spec demands** if it carries decision-relevant content none of the below captures.
+1. One entry paragraph, at most 90 words: who is hurt today and how, what the spec does about it, and what that person gets once it ships. A reader who stops here can say yes or no.
+2. **What changes** - 3-5 one-sentence bullets, at most 80 words together. Each bullet is one rule the user can observe, stated as a condition and its outcome ("when a ticket already fits the template, nothing is written"). Rules that stop, block, or require a repeat approval count as changes. A rule travels with its qualifiers: the condition that makes it apply, what it refuses or keeps, what happens at a zero setting or a missing precondition ("when the base branch moves first, the flow stops and keeps the reviewed work"). The last bullet names what deliberately stays as it is.
+3. **Approval risks** - up to 3 one-sentence bullets, at most 45 words together, each something the supervisor can veto: an irreversible step, a changed contract other components depend on, a dependency on work outside this spec, or a decision the spec leaves to the implementer.
+4. **Done when** - one or two sentences, at most 25 words, restating every measurable completion condition the spec names, each threshold with its precondition ("under five minutes with a warm cache").
+5. **Missing from the spec** - at most 3 lines, each under twelve words, naming an external source a requirement depends on but the spec does not inline (a ticket's acceptance criteria, a contract in another document). Never list superseding or related specs, decision history, or reading material. Emit this block only when at least one such source exists.
 
-Recommended sections:
+Write for a product manager in plain words - behaviors, actors, outcomes - with no code-level identifiers (file paths, commit SHAs, backticked names, command names, line numbers), as the example below does. Leave out the decision log (rejected alternatives, "chose X over Y because Z", how the document was edited); the gate prints the critique record next to the briefing. A short spec yields a briefing well under the cap; the cap is a ceiling, never a target.
 
-1. **Problem + idea** - 3-5 sentences: what's broken and the chosen approach.
-2. **Key decisions** - the decisions that define the solution, plus any notable rejected alternative ("chose X over Y because Z").
-3. **Scope** - what's in, and what's explicitly out (non-goals).
-4. **Risk surface** - shared contracts, schema/migrations, irreversibility, rollout - where approval risk concentrates.
-5. **Inputs / conditions / UI / endpoints.**
-6. **Outputs** - new pages, comms protocols, DB/data changes.
-7. **Key changes to the current process.**
-8. **Caveats / edge cases.**
-9. **Algorithm** - only if the spec defines one. Give the key mechanism and decision points; an example with a short explanation beats an exhaustive step transcript. Skip SQL/syntax.
-10. **Acceptance** - how we'll know it's done, only if the spec defines it.
-11. **Gap footer** - external context the spec leans on but does not inline. Omit if there is none.
-
-Tight, human-readable, no obvious statements. If the topic is complex, an example with explanation beats prose.
+Before: "`guard-windows` job on `windows-latest` in `harness-guard.yml` runs `script/harness-guard.ps1`; a non-zero exit blocks merge."
+After: "A Windows regression in the hook tooling now fails the pull request instead of reaching an operator."
 
 Output the summary as your final text response. You have no write tool. If your task instructs you to write your findings to a file path, do **not** attempt to write, create, or edit any file and do **not** treat the inability to write as a failure - just emit the full summary as your final text response. The harness persists that response to the requested path for you.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-gauntlet",
-  "version": "7.1.2",
+  "version": "7.2.0",
   "description": "Opinionated, gated workflow skills, subagent personas, and runtime extensions for the pi coding agent.",
   "author": "Jacek Juraszek",
   "type": "module",

package/skills/brainstorming/reference/spec-finalization.md

@@ -90,14 +90,14 @@ subagent({ agent: "spec-summarizer", context: "fresh", async: false, cwd: "<abs
 
 Then commit the spec - staging any predecessor spec edited per [Marking superseded specs](superseding.md) alongside it; a change request at the gate that renames, materially revises, or drops the spec also reconciles the predecessor's banner before recommitting. This commit is **unconditional**: the summary is only a gate aid, so a degraded or missing summary never blocks it. If the council path ran, include its audit (`Coverage:` when present, then `Applied:` / `Deferred:` / `Rejected:`, verbatim from `/skill:roasting-the-spec`'s return) in the **commit message body** - this is the durable, non-contractual record a finish-time revert reads back; the audit is never a committed spec section. Evaluate the summary in two stages (the **Degrade path** referenced in each is defined just below):
 
-1. **From the dispatch tool result, before the `Read`.** If the result is **not** an `"Output saved to: <path> (<N> KB, <M> lines)"` reference (e.g. an exit-0 save error returns the full inline output plus an "Output file error" line - the prunable shape, no file to read), or the reference reports under ~500 bytes, or a size grossly disproportionate to the spec (under ~2% of its byte size), or over ~45 KB (the `Read` truncates at 50KB / 2000 lines, so a larger file cannot render whole) - skip the `Read` and take the degrade path. Use the reference's reported figures; do not re-derive them.
-2. **The `Read` itself, as the last content-producing tool call before composing the gate.** `Read` `$SUMMARY_PATH` and paste its contents verbatim at the top of the gate. If the `Read` fails, returns 0 bytes, or reports truncation - take the degrade path. The `Read` must be last: pi-condense does not protect a `/tmp` read, so any turn boundary between the `Read` and the render lets the ~9KB read result be pruned, reproducing the bug.
+1. **From the dispatch tool result, before the `Read`.** If the result is **not** an `"Output saved to: <path> (<N> KB, <M> lines)"` reference (e.g. an exit-0 save error returns the full inline output plus an "Output file error" line - the prunable shape, no file to read), or the reference reports under ~500 bytes, or over ~45 KB (the `Read` truncates at 50KB / 2000 lines, so a larger file cannot render whole) - skip the `Read` and take the degrade path. Use the reference's reported figures; do not re-derive them.
+2. **The `Read` itself, as the last content-producing tool call before composing the gate.** `Read` `$SUMMARY_PATH` and paste its contents verbatim at the top of the gate. If the `Read` fails, returns 0 bytes, or reports truncation - take the degrade path. The `Read` must be last: pi-condense does not protect a `/tmp` read, so any turn boundary between the `Read` and the render lets the read result be pruned, reproducing the bug.
 
 **Degrade path** - reach the gate with a one-line "summary generation failed" note; never paraphrase from the file-only reference, never render a stub as the canonical summary.
 
 Either way - summary rendered or degraded - then `rm "$SUMMARY_PATH"` (unconditional cleanup; harmless if the file was never created, since it lives outside the worktree under the OS temp dir).
 
-Paste the summary verbatim, unedited in the template below; use adjacent lines for the audit, unresolved ambiguities, and every gap-footer entry:
+Paste the summary verbatim, unedited in the template below; use adjacent lines for the audit, unresolved ambiguities, and every `Missing from the spec` entry:
 
 ```
 <spec-only summary read back from the temp file - pasted verbatim, unedited>
@@ -110,7 +110,7 @@ Deferred: [<severity>] <cluster> - raised-by: [<slugs>] -> <where it belongs>
 Rejected: [<severity>] <cluster> - raised-by: [<slugs>] -> <one-line reason>
 (one line per item, exactly as returned by roasting-the-spec - `Applied: none` / `Deferred: none` / `Rejected: none` when a list is empty; omit the audit lines when the worker path ran, not the council)
 
-<unresolved ambiguities; every gap-footer entry from the summary>
+<unresolved ambiguities; every `Missing from the spec` entry from the summary>
 New predecessor candidates at spec-writing: <path> (<title>), ... - index rows are a hint; code is the source and an absent row proves nothing.
 
 1 - approve: proceed to planning under the existing amendment review; a fresh reviewer applies evidence-backed factual corrections on its own, and every other spec amendment (scope, acceptance-criteria, or contract edits, and redraws) stops for your review.