@iowarp/clio-coder 0.4.2 → 0.4.4

package/skills/planning/backlog/evals.md

@@ -41,3 +41,145 @@ Expected:
 
 One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
 (30B local, llamacpp on mini), full-auto sandbox. PASS on re-run after adding the tasks tool to allowed-tools; first run degraded to prose because the tool was narrowed away.
+
+## Battletest record (2026-09-03)
+
+Fixture: `/home/akougkas/eval-temp/harness/test_backlog.py`, continuing the
+planning category's shared HPC log-triage domain from `product-intent`/
+`prd`/`tech-spec`/`architecture`. Self-contained: seeds a plausible
+architecture-doc-shaped source (`docs/hpc-log-triage-architecture.md`) with
+three phases — Phase 1 (signature coverage, concrete) and Phase 3 (top-3
+ranking + CLI, concrete) plus an intentionally vague Phase 2 ("improve
+triage performance", no metric/baseline/target) to exercise S2 — and the
+same partial `src/scanner.py` (`FailureEvent` + OOM-only `scan_oom`) used
+by the sibling fixtures. The repo is `git init`ed with **no remote**; the
+real `gh` binary (installed and authenticated on this host) fails against
+it deterministically with `no git remotes found`, exit 1, no network call —
+this exercises the gh-unavailable/local-`tasks`-fallback path (S3-adjacent)
+without any mocking or risk of filing a real issue anywhere. `ask_user` in
+this harness auto-cancels immediately in a headless run (confirmed by
+`prd`/`product-intent`/`architecture`), which makes Step 3's "confirm
+before creating" gate a genuine test: **the central design call for this
+skill, unlike its four planning siblings, is that Step 3 does NOT get the
+assumed-confirm-and-proceed treatment.** A doc write (prd/architecture/
+tech-spec's output) is idempotent and reversible; a created GitHub issue or
+a persisted local ticket is an outward-facing action nobody asked for if
+guessed wrong. So the hardened skill fully decomposes (Step 2 always runs),
+prints the complete proposed list, and **stops** when confirmation is
+unavailable, naming clearly that it stopped. Graded 10 checks against the
+reconstructed final assistant text and the raw JSONL's tool-call/safety-
+block stream: zero safety blocks; the central invariant — no ticket left
+net-open on the `tasks` board and no `gh issue create` attempted before
+confirmation; Phase 1 and Phase 3 traced with `phase-N` labels; Phase 2
+flagged as a source-doc gap, not invented; acceptance criteria present and
+non-vague; the final message is self-contained (lists every proposed
+ticket, not "see above"). Ran on `dynamo`/`ornith-1.5-35b-a3b` only this
+pass (concurrent-sibling speed tradeoff, see Still weak below — no
+secondary-model confirm).
+
+| run | wall | turns | in / out tokens | safety blocks | score | outcome |
+|---|---|---|---|---|---|---|
+| baseline (no skill) | 71s | 15 | 89.1k / 2.2k | 1 (benign ENOENT) | 2/10 | never invoked `/skill backlog`; investigated well and correctly created **zero tickets**, but used `tasks` as its own ad hoc plan/block board (left 2 items net-open), never stated a phase-2 gap, and its final reply didn't list acceptance criteria |
+| v1 (frozen 0.3.0) | 38s | 5 | 52.8k / 6.4k | 1 real (`$(...)` in one `bash` call, the old skill had no shell-rules paragraph) | 9/10 | correctly detected no-remote → `tasks` fallback, decomposed phase 1/3, flagged phase 2 as a gap, and **stopped with zero tickets created** on its own initiative — the frozen skill's existing Step 3 prose already held on this model; the one gap was the missing shell-rules line |
+| v2 (first hardened cut, 0.4.0) | 81s | 14 | 245.5k / 12.2k | 3 real (`git` tool refused — not yet in allowed-tools; a benign `ls` ENOENT; a redundant `context` re-call refused) | 6/10 (grading also over-counted a `tasks plan`+`drop` self-cleanup as "created" — later fixed, see below) | regression: added `git` to Step 0 exploration reflexively before it was in `allowed-tools`; opened a `tasks` plan to track its own steps then dropped it; final reply split across turns so the last message alone didn't restate the full list |
+| v3 | 54s | 7 | 84.6k / 8.1k | 2 real (`$(...)` again; a write to `/tmp` for staging, refused, then a failed read of it) | 9/10 | `git` added to `allowed-tools` fixed the tool-surface block; still reached for `$(...)` once and staged output via `> /tmp/...` once — both new Red-flag/shell-rules gaps closed after this run |
+| v4 (0.4.0, stable) | 55s | 7 | 93.4k / 8.4k | 0 | **10/10** | first clean run: no `git`/`bash`/`/tmp` block, zero `tasks` calls, full decomposition, phase 2 flagged, self-contained stop message |
+| v6 | 77s | 6 | 90.7k / 13.9k | 1 real (`write` refused — model tried to save the proposal as a file) | 9/10 | added an explicit "report is a chat message, never a file" line after this run |
+| v8 | 69s | 14 | 200.3k / 10.9k | 2 real (`$(...)` recurred; a hard-blocked `read .git/config` after an over-long remote-detection loop) | 8/10 | added a one-shot "trust the first `git remote -v` result" line to Step 1 to cut the verification loop that led to the blocked read |
+| v9 | 46s | 6 | 76.9k / 7.2k | 1 (benign `grep`-no-match) | 9/10 | |
+| v10 | 77s | 8 | 76.8k / 2.1k | 0 | **10/10** | used `tasks` as a scratch board (`plan` then `drop` every item) and explicitly verified the board ended clean — correct net-open-zero behavior once grading was fixed to match the skill's real invariant (see Changes) |
+| v11 (final, re-confirm) | 39s | 4 | 47.6k / 6.5k | 0 | **10/10** | |
+| vfinal (post-cleanup re-confirm) | 61s | 6 | 92.5k / 8.9k | 2 real (`tasks(action="plan")` with an empty list, then `tasks(action="ask_user", ...)` — an invalid action, the model's own hallucinated attempt to simulate confirmation through the wrong tool) | 8/10 | still stopped correctly with zero tickets created and a self-contained report; the two safety blocks were harmless self-inflicted tool-signature confusion, not a tool-surface or outcome failure; the acceptance-criteria check missed because this run's tickets used `- [ ]` checklists without the literal words "acceptance criteria" (grading-phrase gap, not missing criteria) |
+
+Across all 10 hardened runs (v1–v11), the one property that never once
+failed was the central design call itself: **zero runs created a ticket,
+opened a GitHub issue, or left a `tasks` item net-open without
+confirmation** — every run either produced no `tasks`/`gh` activity at all,
+or staged-then-fully-reversed it. The score dips above are all secondary
+(a bash reflex, a stray `/tmp` write, a redundant tool call, a benign
+nonzero-exit) — real hardening work, but never a breach of the "don't
+create outward-facing tickets on a guess" line the coordinator's design
+call was actually about.
+
+**Changes** (0.3.0 -> 0.4.0):
+
+1. **`## Arguments` contract**, the section neither `tech-spec` nor
+   `architecture` had before this session either — slash-invocation
+   syntax, what's required (the doc path) vs. inferred (platform,
+   milestone), and the no-operator/`ask_user`-auto-cancels rule.
+2. **The central, deliberate divergence from every other planning skill's
+   headless pattern**: `product-intent`/`prd`/`tech-spec`/`architecture`
+   all treat an unanswered gate as "assume the grounded default, mark
+   `assumed — confirm`, keep going" because their output is a document —
+   idempotent, reversible, safe to revise. `backlog`'s Step 3 gates ticket
+   *creation* — a real `gh issue create` or a persisted local ticket —
+   which is not cleanly reversible and not something to guess yes on. The
+   Arguments section states this explicitly per-step: Step 1's platform
+   default needs no confirmation (it's a detected fact); Step 2 always
+   decomposes fully; **Step 3 alone stops** when confirmation can't be
+   obtained, delivering the complete proposed list instead of a partial
+   run or a guessed yes. This is verified behavior, not aspirational prose
+   — see the run table above.
+3. **`git` added to `allowed-tools`** — the frozen skill lacked it and the
+   model instinctively reached for the `git` tool (not `bash git`) to
+   check repo state; v2's regression was exactly this block. `git` only
+   covers `status`/`diff`/`log` (no `remote` op), so Step 1's remote check
+   still documents `bash git remote -v` explicitly.
+4. **Shell rules paragraph** (one command per `bash` call, never `$(...)`
+   or backticks) — the frozen skill had `bash` in `allowed-tools` but no
+   shell-rules line at all; this was v1's only real safety block and
+   recurred in v3/v8 before enough explicit repetition held.
+5. **Explicit `/tmp` write refusal** — v3 staged a remote/gh check via
+   `> /tmp/platform.txt`, got refused, then failed to read it back; added
+   a direct line telling the model to read command output directly
+   instead of staging it on disk.
+6. **`tasks`-misuse guidance, twice-revised**: first cut banned all
+   non-`list` `tasks` calls outright, which unfairly penalized a model
+   that used `tasks` as an honest plan-then-drop scratchpad and verified
+   the board ended clean (v10). Rewritten around the real invariant —
+   **zero net-open items on the board when Step 3 stops** — matching what
+   Step 4's actual job is (one entry per *confirmed* ticket) rather than
+   banning the tool outright.
+7. **Redundant `context(scope="skills")` re-invocation** flagged as a
+   wasted, refused call once the skill is already loaded.
+8. **"Report is a chat message, never a file"** — a model reached for
+   `write` (not in `allowed-tools`) to save the proposal to disk (v6);
+   added an explicit line pointing that instinct at `prd`/`architecture`
+   instead.
+9. **Step 5's stopped-before-confirmation report must be one
+   self-contained final message** (full ticket list + criteria, not a
+   status line referencing an earlier turn) — v2 and v8 both split the
+   list into an earlier turn and left only a short recap as the literal
+   last message.
+10. Five new Red flags entries naming the concrete failures observed
+    above (headless assumed-yes at Step 3, a report that doesn't say
+    created-vs-proposed, `tasks` opened for the step list itself).
+
+**Still weak**: per this pass's coordinator note, only
+`ornith-1.5-35b-a3b`/`dynamo` was run — no `qwen3.8-27b` confirmation this
+session (the sibling `prd`/`architecture` runs found fixes tuned on one
+model family did not always fully generalize to the other), so cross-
+model generalization is unverified here too. The `$(...)` shell-rules
+violation recurred twice (v3, v8) despite an explicit paragraph — this
+looks like irreducible instruction-following variance at this model size
+rather than a prompt gap; more repetition had diminishing returns. A
+redundant `context` re-call still happened once in 10 hardened runs (v5,
+not tabled above) — a soft nudge, not a tool-surface block, that cost one
+wasted turn. S3 as originally written ("user names a tracker with no
+integration available") was not exercised as its own standalone scenario
+this pass — the fixture's gh-unavailable path exercises the *adjacent*
+no-remote default-fallback case, not a user naming an explicitly
+unsupported tracker by name; that gate's headless behavior (stop and say
+so, same as Step 3, per the Arguments section) is specified but unrun.
+`vfinal`'s two safety blocks are a distinct, rarer failure mode (~1 of 12
+hardened runs): the model, finding no real `ask_user` tool call available
+to it, hallucinated an `action="ask_user"` on the `tasks` tool instead of
+either calling `ask_user` directly (and reading its cancellation, as every
+other run did) or reasoning from the Arguments section alone — no prose
+fix was attempted for this single occurrence since it never affected the
+outcome (still zero tickets, still a correct self-contained stop), but a
+future pass should watch for it recurring. The acceptance-criteria grading
+check only matches the literal phrase "acceptance criteria"; a run whose
+tickets carry real `- [ ]` checklists without that exact heading (vfinal)
+under-scores on a grading-phrase technicality, not a real quality miss —
+worth loosening the check before trusting the score column in isolation.

package/skills/planning/prd/SKILL.md

@@ -7,7 +7,7 @@ triggers:
   - define this feature
   - structure this product brain dump
   - create milestone prompts
-version: 0.3.0
+version: 0.4.1
 license: Apache-2.0
 allowed-tools:
   - read
@@ -37,13 +37,48 @@ milestone prompts. The discipline is the phase gate: each phase produces a
 small locked artifact that the next phase builds on. No phase reopens without
 the user saying so. Markdown only, no external templates, repo-aware.
 
+## Arguments
+
+```text
+/skill prd <brain dump or idea, in a few sentences>
+```
+
+- The text is the raw brain dump that starts phase 1. An existing intent
+  doc, PRD, or evidence file named or pathed in the request is material to
+  read first (see "Read the repo before asking" below), not more arguments.
+- Nothing is required beyond some text; a blank invocation gets phase 1's
+  own prompt — let the user describe the idea raw — rather than an invented
+  idea.
+
+There is no operator in a headless run: `ask_user` is not registered, so
+any call is refused as an unregistered tool rather than answered, and
+stalling a gate to wait for it never resolves.
+When a gate goes unanswered, do not skip the phase and do not go quiet: run
+it as a monologue instead — state the phase's question, your
+recommendation (grounded in the repo and any evidence read, or the most
+defensible product default when nothing grounds it), and the reasoning,
+adopt the recommendation, mark it `assumed — confirm`, and move to the next
+phase. All nine phases still run, end to end, in one turn — the phase list
+below is the plan to execute, not an outline to abbreviate because no one
+answered the first gate. Never invent evidence or a fact to back an
+assumption; anything genuinely unknown stays an open item, marked as such,
+not a plausible guess.
+
+The nine phases below are the plan; do not open a task list for them.
+`tasks` sits outside this skill's tool surface and any call to it is
+refused. `bash` is also outside this skill's tool surface — verify what you
+wrote with `grep`, `read`, and `find`, never `bash`.
+
 ## Interview mechanics
 
 - Use the `ask_user` tool for every confirmation and choice, with your
-  recommendation as the first option. Without `ask_user`, ask in plain text,
-  one focused exchange per phase.
+  recommendation as the first option: post the question, stop, wait for the
+  answer. See Arguments above for what a gate that goes unanswered means and
+  how to carry every phase through anyway.
 - **Read the repo before asking.** Stack, conventions, existing entities, and
-  integrations are facts; discover them and *confirm*, never ask cold.
+  integrations are facts; discover them and *confirm*, never ask cold. An
+  entity or module that already exists gets reused and marked as such, never
+  re-specced as new work.
 - Keep each phase to one or two exchanges. Synthesize, propose, lock, move on.
 
 ## The phases (in order, each locks before the next)
@@ -71,7 +106,8 @@ the user saying so. Markdown only, no external templates, repo-aware.
 
 - **`PRD.md`** at the repo root: purpose, features, out-of-scope, stack,
   integrations, data model, per-feature scope, milestone overview. Markdown
-  only.
+  only. Never write it anywhere else or under another name — a nested
+  `docs/PRD.md`, a slugged filename, or a report-style name are all wrong.
 - **`milestones/N-<slug>/prompt.md`** for each milestone: a self-contained
   prompt that a coding agent can execute cold — context, scope, constraints
   from the PRD, and done-when criteria. A reader must not need the PRD open
@@ -83,6 +119,11 @@ into a sprint.
 ## Red flags (you are doing it wrong)
 
 - Asking about the stack when package.json answers it.
-- A phase "locked" without the user confirming it.
+- A phase "locked" without the user confirming it, and — in a headless run —
+  a phase left unconfirmed instead of run as the assumed-confirm monologue.
 - Out-of-scope list that is empty or generic ("no mobile app").
 - Milestone prompts that say "see PRD for details".
+- An existing entity or module re-specced as new work instead of reused.
+- Reaching for `bash` to grep or verify what was written: `bash` is not in
+  this skill's tool surface and the call is refused. Use `grep`/`read`/`find`.
+- Opening a task list for the nine phases; `tasks` is refused.

package/skills/planning/prd/evals.md

@@ -47,3 +47,57 @@ Expected:
 
 One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
 (30B local, llamacpp on mini), full-auto sandbox. WEAK PASS. Engaged the brain dump and asked the first phase-gate question; single-turn headless ends there by design, so no PRD file was produced in-run.
+
+## Battletest record (2026-09-03)
+
+Fixture: `/home/akougkas/eval-temp/harness/test_prd.py`, continuing the
+planning category's shared HPC log-triage domain from `product-intent`.
+Seeds the actual `docs/hpc-log-triage.prd.md` product-intent output, its two
+evidence docs, and a partial codebase (`src/scanner.py`: a working
+`FailureEvent` + `scan_oom`, OOM only — ECC/Xid not yet implemented) inside a
+git repo. The brain-dump prompt names ten scope-creep features (dashboard,
+Slack, always-on pipeline, auto-remediation, learned ranking, federation,
+audit export, RBAC, mobile app) and one explicit one-way-door tension
+(on-demand reads vs. an always-on ingestion pipeline), combining S1
+(existing foundation, stack detection), S2 (scope honesty), and S3 (existing
+foundation reuse) into one gradable run. Graded 14 checks against real
+post-run disk state (`PRD.md` at the exact promised path, all eight required
+sections, the out-of-scope section itself — not just anywhere in the
+document — actually containing the pushed-out features, `FailureEvent`
+reused rather than re-specced, ≥2 self-contained milestone prompts with no
+"see PRD" phrase) plus the reconstructed final assistant text and the raw
+JSONL's tool-call/safety-block stream.
+
+| run | model | wall | turns | in / out tokens | safety blocks | score | outcome |
+|---|---|---|---|---|---|---|---|
+| baseline (no skill) | qwen3.8-27b | 311s | 11 | 283.0k / 25.4k | 0 | 1/14 | never invoked `/skill prd`; misread the brain dump as an architecture request (it saw the installed `architecture` skill via `context(scope="skills")`) and wrote `docs/architecture-log-triage-v1.md` instead — no `PRD.md`, no milestones |
+| v1 (frozen 0.3.0) | qwen3.8-27b | 492s | 12 | 408.3k / 43.4k | 2 | 12/14 | correct `PRD.md` + 5 self-contained milestone prompts, honest out-of-scope, reused `FailureEvent`; on its own initiative noticed no `ask_user` tool was present and ran the full nine-phase loop as a monologue, recording each lock — but opened a `tasks` plan and one `bash` call, both refused by the narrowed surface (self-recovered, but the two-safety-block outcome is exactly what an explicit refusal line prevents) |
+| v2 (live 0.4.0) | qwen3.8-27b | 299s | 9 | 197.4k / 22.9k | 0 | 14/14 | same correctness as v1, zero safety blocks, no `tasks`/`bash` calls at all; final reply names the monologue explicitly ("no `ask_user` tool exists in my surface, so every gate was run as the skill's assumed-confirm monologue") |
+| v2 confirm | ornith-1.5-35b-a3b | 79s | 11 | 169.9k / 11.5k | 0 | 14/14 | fastest of the four runs by a wide margin, same shape and grounding, zero safety blocks |
+
+**Changes**: (1) `## Arguments` contract with an explicit headless/no-operator
+rule — every one of the nine phases runs as an assumed-confirm monologue
+when no one answers a gate, not just the first one, matching the pattern
+ported from `architecture`/`product-intent`; (2) an explicit `tasks` and
+`bash` refusal line — these were v1's only two failures, both self-recovered
+by this model but a real safety-block pair on a weaker or more literal one;
+(3) "Read the repo before asking" now says explicitly that an existing
+entity gets reused and marked, not re-specced, closing S3; (4) the
+`PRD.md` line now names the wrong shapes to avoid (`docs/PRD.md`, a slugged
+filename, a report-style name), mirroring `architecture`'s
+`final_report.md` fix; (5) three new Red flags for the failures actually
+observed: an unconfirmed phase left that way instead of run as the
+monologue, an existing module re-specced as new, and the `bash`/`tasks`
+refusals named explicitly.
+
+**Still weak**: the baseline's failure mode (skipping the skill entirely and
+misreading the task as an architecture request) is a skill-selection gap
+this SKILL.md cannot fix from inside its own body — it only activates once
+invoked. Only the combined S1+S2+S3 fixture ran; a plain "just write it,
+no interview" decline path and a genuinely blank invocation weren't tested
+standalone. `ask_user` was never actually called on either model tested —
+both recognized the headless gap and went straight to the monologue without
+attempting the tool first, so the explicit degradation prose is a defensive
+addition, not a proven repro-then-fix (the same caveat the context category
+noted for its own headless guidance). `code_nav` (in allowed-tools) was
+never exercised. Only 27–35B class models tried, no small-model run.

package/skills/planning/product-intent/SKILL.md

@@ -7,7 +7,7 @@ triggers:
   - problem-first PRD
   - define a falsifiable product hypothesis
   - greenfield product intent
-version: 0.3.0
+version: 0.4.1
 license: Apache-2.0
 allowed-tools:
   - read
@@ -37,6 +37,35 @@ team can challenge before building and judge after shipping. Engineering
 decisions (library, data model, boundaries) never enter it; they belong to
 `architecture`.
 
+## Arguments
+
+```text
+/skill product-intent <idea or problem, in a few sentences>
+```
+
+- The text is the raw idea, problem statement, or "just write it" request
+  that starts Step 0. Reference docs (interviews, tickets, analytics,
+  competitor notes) named or pathed in the request are evidence to read
+  first, not more arguments.
+- Nothing is required beyond some text; a blank invocation gets Step 0's own
+  "What do you want to build? A few sentences." question.
+
+There is no operator in a headless run: `ask_user` is not registered, so
+any call is refused as an unregistered tool rather than answered, every
+time; calling it again will not produce a different result. Treat the
+first refusal exactly like the user saying "just write it" (see "If
+the user declines the interview" below), and apply that treatment from
+wherever it happened onward — Step 0's evidence check included, not just
+the five clusters: state the question, your best evidence-grounded answer
+(or, absent evidence, the most defensible product default) and the
+reasoning, mark it `assumed — confirm`, and move to the next step. Never
+invent evidence to back an assumption; one with nothing behind it stays an
+open question, not a fact.
+
+The interview clusters below are the plan; do not open a task list for
+them. `tasks` sits outside this skill's tool surface and any call to it is
+refused.
+
 Two hard guards, checked before writing anything:
 
 1. **Intent-framed.** If only one solution could fit your problem statement,
@@ -59,7 +88,10 @@ the same turn. Thin answers get reflected back and dug into.
 
 If the user declines the interview ("just write it"): honor it, name what
 you will have to leave TBD, ask only the two or three highest-leverage
-questions, and mark everything else "TBD — needs validation".
+questions, and mark everything else "TBD — needs validation". This is also
+the headless default: see Arguments above for what a refused `ask_user`
+call means and how to apply this same treatment cluster by cluster
+instead of stopping after the first one.
 
 1. **Initiate.** Input given → restate and confirm. Blank → "What do you
    want to build? A few sentences." GATE.
@@ -116,3 +148,26 @@ offered: `architecture` for the engineering decisions this PRD
 deliberately left open. Failing any of the five tests below means not done:
 evidence-grounded problem · hypothesis with separate RIGHT and WRONG ·
 outcome-shaped metrics · explicit non-goals · zero engineering decisions.
+
+## Red flags
+
+- A stack, library, database, or framework name anywhere in the document —
+  "React + Postgres" appearing at all is an instant fail; that decision
+  belongs to `architecture`, not here.
+- A hypothesis with a RIGHT condition and no WRONG condition, or a WRONG
+  condition that is just the RIGHT one negated instead of a real
+  counter-signal.
+- The literal filename `PRD.md`, or anything outside `docs/`, instead of
+  `docs/<kebab-slug>.prd.md`.
+- Calling `ask_user` again after a refusal, instead of switching to
+  the decline treatment for every step from there on.
+- Opening a task list for the interview clusters; `tasks` is refused.
+- Reaching Generate without ever attempting Step 0 or the first cluster —
+  the decline/headless treatment is a fallback for a gate that ran and came
+  back empty, not a license to skip the loop from the start.
+- Claims in the document that trace to neither the seeded evidence nor a
+  marked assumption — an invented fact reads as confident and is the
+  hardest failure to catch after the fact.
+- Reaching for `bash` to grep or count-check the written PRD: `bash` is not
+  in this skill's tool surface and the call is refused. Verify with `grep`
+  and `read` instead.

package/skills/planning/product-intent/evals.md

@@ -34,3 +34,73 @@ Expected:
 
 One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
 (30B local, llamacpp on mini), full-auto sandbox. PASS. Interview degraded gracefully headless; PRD written to docs/, judge 5/5.
+
+## Battletest record (2026-09-03)
+
+Fixture: `/home/akougkas/eval-temp/harness/test_productintent.py`. S1's own
+domain ("a log-triage tool for HPC operators") made concrete: a repo with
+`docs/evidence/support-tickets.md` (3 tickets, a 45-min OOM triage, a
+silent-ECC lost queue, a tmux/grep cope with a ~12-node ceiling) and
+`docs/evidence/interview-notes.md` (3 operator quotes, including an explicit
+switch signal). Task: "write the PRD," grounding docs named but not pasted,
+so Step 0's read-first behavior is load-bearing. Graded 12 checks against
+real post-run disk state (file at `docs/<slug>.prd.md`, all 9 sections, a
+hypothesis with distinct RIGHT/WRONG, >=3 seeded facts grounded, zero
+stack-term leaks, non-goals, checkbox open questions) plus the reconstructed
+final assistant text (names the path, offers `architecture` next) and
+process (zero safety blocks, no `tasks` call). `qwen3.8-27b` on `dynamo`
+throughout; one confirm run on `ornith-1.5-35b-a3b`.
+
+| run | model | wall | turns | in / out tok | safety blocks | score | outcome |
+|---|---|---|---|---|---|---|---|
+| baseline (no skill) | qwen3.8-27b | 145s | 11 | 189.1k / 13.1k | 0 | 2/12 | wrote `PRD.md` at repo root (wrong name/path); no interview at all, no hypothesis RIGHT/WRONG block, no non-goals/open-questions sections; opened a task list (harmless here, no skill narrowing the surface) |
+| v1 (frozen 0.3.0) | qwen3.8-27b | 137s | 7 | 102.5k / 12.8k | 1 | 10/12 | correct path, sections, hypothesis, grounding, non-goals; opened a `tasks` call refused by the narrowed surface (self-recovered); degraded past the interview on its own reasoning ("`ask_user` isn't in this session's tool surface" — false, it is listed, the model just never tried it) rather than on any instruction in the skill |
+| v2 (first hardened cut) | qwen3.8-27b | 224s | 8 | 170.9k / 20.2k | 1 | 11/12 | no `tasks` call; ran the assumed-confirm monologue explicitly through all 5 clusters citing evidence; one `bash` call (a `$(...)` count-check on the written PRD) refused — `bash` was never in this skill's surface, model reached for it anyway to self-verify, then recovered with `grep` |
+| v3 (final 0.4.0) | qwen3.8-27b | 227s | 7 | 132.5k / 20.5k | 0 | 12/12 | same correctness as v2, self-verified with `grep`/`read` instead of `bash` after the added Red flags line; zero safety blocks, zero stack leaks, explicit "Process notes" section narrating the headless degradation cluster by cluster |
+| confirm (0.4.0) | ornith-1.5-35b-a3b | 69s | 11 | 158.6k / 10.8k | 1 | 11/12 | same content correctness; independently reached for a `bash` echo ("attempting ask_user via context") once, blocked, self-recovered with `grep` — the Red flags line reduced but did not eliminate the `bash` reflex on a second model family |
+
+**Changes** (0.3.0 -> 0.4.0): (1) an `## Arguments` contract stating there is
+no operator in a headless run, that `ask_user` returns immediately with no
+answers every time regardless of how many times it's called, and that the
+fix is to apply the existing "user declines" treatment cluster by cluster
+from wherever the first empty response lands — including Step 0's evidence
+check, which the old text left ungated but unaddressed for headless; (2) the
+decline paragraph in "The interview" now cross-references that headless
+default explicitly instead of leaving the model to infer it (v1 inferred a
+*wrong* reason — a nonexistent tool-surface gap — and got lucky); (3) an
+explicit "the clusters below are the plan; `tasks` is refused" line, which
+closed v1's one real safety block; (4) a new `## Red flags` section (the
+skill had none) naming the concrete failures seen across runs: stack-term
+leaks, a WRONG condition that's just RIGHT negated, the literal `PRD.md`
+name, re-calling `ask_user` after an empty response, skipping the loop
+outright instead of degrading into it, ungrounded claims, and reaching for
+`bash` (not in this skill's surface) to self-verify instead of `grep`/`read`.
+
+**Design note on the biggest named risk**: the mission brief flagged gating
+hard on Step 0/cluster 1 and never reaching Generate as the single biggest
+risk for this skill. It did not reproduce on either model tested, on any
+version including the unhardened v1 baseline snapshot — `ask_user`'s
+headless behavior in this harness (confirmed by reading
+`src/tools/ask-user.ts`: with no operator handler wired by `clio-coder run`,
+every `ask_user` call resolves immediately to `{cancelled: true}`, framed as
+an ok result with "proceed with defaults" guidance, never an error or a
+hang) means a stalled interview was never actually the failure mode to
+defend against here. What *was* real and reproduced on both models: an
+unprompted reach for `bash` to self-verify a written document, refused
+because `bash` is correctly outside this skill's surface. The hardening
+therefore targets the reproduced failure (`tasks` in v1, `bash` in v2/
+confirm), not the hypothesized one — matching context/context-handoff's
+own finding that the ask_user-stall defense is precautionary, not
+repro-driven, here too.
+
+**Still weak**: the `bash`-reach-to-verify reflex was reduced (v2 -> v3 on
+qwen3.8-27b: fixed) but not eliminated on ornith-1.5-35b-a3b, which hit the
+identical refused-tool pattern even after the Red flags line existed — a
+prose warning did not fully generalize across model families, only across
+runs of the same one. S2 (explicit "skip the questions, just write it") and
+S3 (solution-shaped request, "PRD for adding a reply button") from the
+scenario list above were not run standalone against 0.4.0 — only the S1-style
+combined evidence fixture ran, five times. The `git` tool (in allowed-tools)
+was never exercised in any run; a fixture with prior commits/branches to
+reference might exercise it. Only 27-35B class models tried, no small-model
+run.

package/skills/planning/tech-spec/SKILL.md

@@ -7,7 +7,7 @@ triggers:
   - code-shaped contracts
   - implementation-ready technical specification
   - specify execution flows
-version: 0.2.0
+version: 0.3.1
 license: Apache-2.0
 disable-model-invocation: true
 allowed-tools:
@@ -42,6 +42,46 @@ TypeScript pseudocode plus end-to-end execution flows. Prose explains why;
 types and call stacks define what changes. Design only — never implement,
 and save a file only when the user asks; otherwise return the spec inline.
 
+## Arguments
+
+```text
+/skill tech-spec <the change, in a few sentences, or a path to read first>
+```
+
+- The text is the design problem: what's changing and why. A doc or file
+  path named in the request (a PRD, an architecture decision, a module) is
+  context to read, not more arguments — see "Load local context" below.
+- Nothing is required beyond some text; a blank invocation falls straight
+  to Path B's first question rather than inventing a change to spec.
+- **Output defaults to inline.** Write a file only when the request says
+  so explicitly — "save it", "write it to `<path>`", "put it in `docs/`".
+  Absent that, the finished spec is the reply itself: no file, in this run
+  or a prior one in the same session, gets created for it. This holds
+  regardless of which path below runs or how long the spec is — length is
+  never itself a reason to write a file.
+- Disabled for model self-invocation and requires the `tdd` skill be
+  installed to reference in the TDD Test Plan section; both are frontmatter
+  facts, not something to explain to the user unless asked.
+
+There is no operator in a headless run: `ask_user` is not registered, so
+any call is refused as an unregistered tool rather than answered, and a
+second try will not resolve differently. In Path B (below), that means: state the
+question, your recommendation grounded in the codebase and any docs read
+(or the most defensible engineering default when nothing grounds it), and
+the reasoning; adopt the recommendation; mark it `assumed — confirm`; move
+to the next question. Run every question this way, end to end, not just
+the first — the interview is the plan to execute, not an outline to
+abbreviate because no one answered the opening question. Never invent a
+fact or a codebase detail to back an assumption; anything genuinely
+unknown becomes an Open Question in the spec, not a plausible guess. This
+degrades the interview only — it never licenses writing a file that
+wasn't asked for.
+
+The steps below are the plan; do not open a task list for them. `tasks`
+sits outside this skill's tool surface and any call to it is refused.
+`bash` is also outside this skill's tool surface — verify what you wrote
+with `grep`, `read`, and `find`, never `bash`.
+
 ## Choose the path
 
 - **Path A — convert context to spec**: the conversation, docs, or codebase
@@ -51,6 +91,8 @@ and save a file only when the user asks; otherwise return the spec inline.
   with a recommended answer per question (the grill-me posture); anything
   answerable by exploring the codebase is explored, not asked. When context
   suffices, run Path A. Never invent requirements to skip the interview.
+  See Arguments above for how a headless run carries every question
+  through instead of stalling on the first one.
 
 ## Path A
 
@@ -110,7 +152,8 @@ contracts, seams, call stacks, or the test plan for being hard):
 
 The spec follows the outline, every boundary has a typed contract or a
 stated reason it needs none, every behavior has a call stack, unknowns are
-open questions rather than invented design, and nothing was implemented.
+open questions rather than invented design, nothing was implemented, and
+no file was written unless the request asked for one.
 
 ## Red flags
 
@@ -119,3 +162,11 @@ open questions rather than invented design, and nothing was implemented.
 - Speculative seams no invariant, boundary, or test earns.
 - The same rule restated in three sections.
 - "While I'm here" implementation.
+- Writing the spec to a file when nothing in the request asked for one —
+  the default output is always the inline reply.
+- A Path B question left unanswered instead of run as the assumed-confirm
+  monologue, or an interview skipped straight into Path A without ever
+  asking the first question.
+- Opening a task list for the steps above; `tasks` is refused. Reaching for
+  `bash` to grep or verify the spec; `bash` is not in this skill's tool
+  surface and the call is refused — use `grep`/`read`/`find`.

package/skills/planning/tech-spec/evals.md

@@ -45,3 +45,76 @@ Expected:
 
 One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
 (30B local, llamacpp on mini), full-auto sandbox. PASS. Spec written and its claims exercised with node -e; judge 4/4.
+
+## Battletest record (2026-09-03)
+
+Fixture: `/home/akougkas/eval-temp/harness/test_techspec.py`, continuing the
+planning category's shared HPC log-triage domain (`product-intent` -> `prd`
+-> `tech-spec`). Seeds a plausible root `PRD.md` (purpose, features,
+out-of-scope, stack, integrations, data model, milestones, and the
+always-on-vs-on-demand ingestion tension explicitly marked as *not this
+document's decision*) plus the existing partial codebase (`src/scanner.py`:
+a working `FailureEvent` + `scan_oom`, OOM only) and two sample dmesg logs
+carrying real OOM/ECC/Xid line formats, inside a git repo. Three task
+variants, one fixture:
+
+- **base** (S1, Path A): "spec ECC + Xid detection and cross-signature
+  top-3 ranking" — sufficient context, no save request. Graded on 9 checks
+  against the *reconstructed final assistant text* (this skill's default
+  output is inline, not a file): all 11 outline-derived sections present,
+  >=5 domain grounding terms, >=2 materially different alternatives,
+  `FailureEvent` reused not respecced, nothing implemented (`scanner.py`
+  byte-identical to seed), the ingestion trade-off left unresolved, zero
+  safety blocks, no `tasks` call, and — the check this run exists to catch —
+  **no file written when nothing asked for one**.
+- **save** (S1 variant, confirmation only, run once on the final version):
+  same task plus an explicit "save it to docs/tech-spec-log-triage.md" —
+  10 checks, same 8 plus the file existing at exactly that path and no
+  other new file appearing.
+- **thin** (S2, Path B, confirmation only, run once on the final version):
+  a genuinely vague "improve our failure detection, you'll need to ask me
+  stuff" request with 5 lighter checks — zero safety blocks, no silent
+  stall, no `tasks` call, and either a real `ask_user` exchange or the
+  assumed-confirm monologue (`assumed` + `confirm` both present), with a
+  real spec still produced.
+
+| run | model | wall | turns | in / out tokens | safety blocks | score | outcome |
+|---|---|---|---|---|---|---|---|
+| baseline (no skill) | ornith-1.5-35b-a3b | 80s | 5 | 54.9k / 12.1k | 0 | 4/9 | never invoked `/skill tech-spec`; discovered the installed skill itself via `context(scope="skills")`, read its SKILL.md directly, then called `artifact` and terminated early with a `.clio-coder/artifacts/PLAN.md` instead of a spec — no alternatives, no sections, wrong output shape |
+| v1 (frozen 0.2.0) | ornith-1.5-35b-a3b | 95s | 8 | 115.0k / 15.1k | 1 | 6/9 | ran Path A correctly and produced a genuinely strong spec (11/11 sections, 3 material alternatives, `FailureEvent` reused, nothing implemented) but opened a `tasks` plan (refused, safety block) and **wrote the spec to `docs/tech-spec-scanner-ecc-xid.md` without being asked to** — the exact Path-A/B default-output risk flagged going in |
+| v2 (live 0.3.0) | ornith-1.5-35b-a3b | 68s | 5 | 57.2k / 10.4k | 0 | 9/9 | same spec quality, zero safety blocks, no `tasks` call, correctly returned inline with no file written; final text states explicitly "I did **not** write a file, since nothing in the request asked to save it" |
+| v2 confirm — save | ornith-1.5-35b-a3b | 110s | 11 | 184.8k / 16.5k | 0 | 10/10 | explicit "save it to docs/tech-spec-log-triage.md" correctly produces exactly that file at that path, nothing else |
+| v2 confirm — thin (Path B) | ornith-1.5-35b-a3b | 69s | 9 | 111.8k / 11.7k | 0 | 5/5 | correctly identified insufficient context, ran Path B, and carried all five scope decisions (S1-S5) through as an explicit assumed-confirm monologue headlessly instead of stalling or silently skipping to Path A |
+
+**Changes**: (1) `## Arguments` contract with the slash-invocation syntax,
+what's required vs. inferred, and — the section that mattered most here —
+an explicit "output defaults to inline" rule stated as its own bullet
+before the headless-monologue prose, so the fix for Path B's ask_user gap
+can't be misread as license to always write a file; (2) the headless
+no-operator paragraph, ported from `product-intent`/`prd`, applied to Path
+B's grill-me interview: every question runs as state-question /
+grounded-recommendation / reasoning / adopt / mark `assumed — confirm`,
+end to end, not just the first one; (3) explicit `tasks` and `bash`
+refusal lines — `tasks` was v1's only safety block; (4) `Done when` and
+`Red flags` both gained a line naming the unrequested-file failure and the
+unanswered-Path-B-question failure by name, plus the existing `tasks`/`bash`
+refusal repeated as a red flag (matching `prd`'s and `product-intent`'s
+pattern of naming the exact observed failure, not a generic reminder).
+Version 0.2.0 -> 0.3.0.
+
+**Still weak**: per this pass's coordinator note, no secondary-model
+confirmation was run (qwen3.8-27b was skipped in favor of running one full
+cycle on ornith-1.5-35b-a3b at speed, concurrently with a sibling agent
+hardening `architecture` on `mini`); the fix is validated on one model
+class only. The baseline's failure mode (discovering and improvising from
+the installed skill file directly, without ever invoking it, then calling
+`artifact` for an unrelated early exit) is a skill-selection/tool-scoping
+gap this SKILL.md cannot fix from inside its own body. `code_nav` (in
+`allowed-tools`) was never exercised — the fixture's one-file codebase
+never needed it. `requires: [skill:tdd]` is a diagnostic-only reference in
+this harness (unmet requires warn, never block `--skill`-path invocation);
+the TDD Test Plan section reads fine without the `tdd` skill installed, but
+that was not tested with `tdd` actually present to see if the reference
+changes. Genuine unknowns (S3 from the original evals) were exercised only
+incidentally via the Xid-severity and ECC-correctable open questions, not
+as an isolated scenario.