@iowarp/clio-coder 0.4.1 → 0.4.3

package/skills/workflow/cut-it/SKILL.md

@@ -1,13 +1,13 @@
 ---
 name: cut-it
-description: Use when a plan, PRD, or milestone must become an executable sprint — dependency-ordered vertical slices sized for one focused agent run each, with done-when verification per slice. Never fabricates a plan; if none exists or it is too vague to slice, says so and recommends an interview first. Triggers on "cut it", "slice this plan", "make this executable", "turn this into a sprint".
+description: Slices an existing plan, PRD, or milestone into an executable sprint of dependency-ordered vertical slices sized for one agent run each, with done-when verification per slice; never fabricates a plan. Not for deciding the approach; use architecture.
 triggers:
   - cut it
   - slice this plan
   - make this plan executable
   - turn this milestone into a sprint
   - write dependency-ordered vertical slices
-version: 0.2.3
+version: 0.4.0
 license: Apache-2.0
 allowed-tools:
   - read
@@ -18,7 +18,6 @@ allowed-tools:
   - context
   - code_nav
   - write
-  - artifact
   - ask_user
 clio-coder:
   registry-id: iowarp/clio-coder
@@ -39,6 +38,43 @@ can run one at a time, leaving the build green after every slice. The output
 is a `SPRINT.md` another agent can execute cold — no conversation context
 required.
 
+## Arguments
+
+```text
+cut it [<path to plan>]
+```
+
+There is no flag syntax; the trigger is conversational — "cut it", "slice
+this plan", "turn this milestone into a sprint". A path the user names in
+the same request (a specific `PLAN.md`, `PRD.md`, or `milestones/*/prompt.md`)
+is the plan to slice; a path with no plan words near it, or a bare
+destination like "write it to docs/SPRINT.md", is Step 3's output location,
+not the input. When neither is named, Step 1 finds the plan and Step 3
+writes to the repo-root default.
+
+This run has no back-and-forth. `ask_user` still executes — it is registered
+and the call succeeds — but nothing answers it in a headless run: every round
+returns `{cancelled: true}` immediately, as an ordinary result, not an error.
+The happy path here rarely needs a question at all — Step 1's "stop and say
+so" for a missing or vague plan is already headless-safe, and Step 3's output
+path defaults without asking. If several plans or milestones are plausible
+candidates and the choice matters, do not stop on an open question: pick the
+most recently modified or most specifically named one, state that choice and
+the alternative you set aside, mark it `assumed — confirm`, and keep going in
+the same turn. Do not end a turn on an unanswered question in `ask_user` or
+in plain text.
+
+The steps below are the plan; do not open a task list for them. This skill's
+tool surface is exactly `read`, `grep`, `ls`, `find`, `git`, `context`,
+`code_nav`, `write`, and `ask_user` (`context` and `ask_user` are always
+available regardless). `tasks` and `bash` both sit outside it and any call to
+either is refused — track progress by walking the steps below, not a task
+board; check for a build/test/lint setup (a `package.json`, a Makefile, a
+node/toolchain version) with `find`, `ls`, and `read`, not `bash node
+--version` or `bash find`. The read-only `git` tool (`status`, `log`) is
+useful in Step 1 when locating the plan benefits from recent history or
+uncommitted changes.
+
 ## Step 1 — Locate the plan
 
 In priority order: a file the user names, a plan in the conversation,
@@ -60,10 +96,14 @@ slicing of a vague plan hides gaps; flagging them is the deliverable.
 - **Self-contained.** Real file paths, real commands, concrete steps. A reader
   with zero conversation context can execute it.
 
-## Step 3 — Write the artifact
+## Step 3 — Write SPRINT.md
 
-Default output is `SPRINT.md` at the repo root (honor a caller-supplied path).
-Format:
+Use the `write` tool. `SPRINT.md` is a plain file in the working tree, not a
+generated report — do not call a tool literally named `artifact` for this;
+that tool is not on this skill's surface and, in this harness, is a
+terminal call that ends the run the instant it is invoked, before you can
+report back. Default output path is `SPRINT.md` at the repo root; honor a
+caller-supplied path from Arguments instead. Format:
 
 ```markdown
 # Sprint: <name>
@@ -86,9 +126,29 @@ Format:
 "Done when" is the contract, not decoration. If you cannot write a testable
 done-when for a slice, the slice is not ready to cut — go back to the plan.
 
+## Step 4 — Report back
+
+End the turn with a short final reply, not silence after the write: the path
+you wrote (`SPRINT.md` or the caller-supplied path), the number of slices,
+and a one-line summary of the battle order. This is the only confirmation
+the caller gets that the write actually happened.
+
 ## Red flags (you are doing it wrong)
 
 - A slice whose steps say "and related changes" or "etc."
 - Done-when criteria that restate the goal instead of naming a check.
 - A slice that only compiles when a later slice lands.
 - Slicing a plan you had to invent on the spot.
+- Calling a tool literally named `artifact` because Step 3 talks about "the
+  artifact" — that word here means "the deliverable document," not the
+  `artifact` tool. That tool is off this skill's surface and, in this
+  harness, terminates the run on the spot, writes to
+  `.clio-coder/artifacts/` instead of `SPRINT.md`, and skips Step 4 entirely.
+  Use `write`.
+- Ending the run right after the write with no final reply — Step 4 is not
+  optional.
+- Opening a `tasks` list for the steps above; `tasks` is refused.
+- Reaching for `bash` (`node --version`, `find`, `ls -la`, ...) to check the
+  repo's toolchain or structure; `bash` is refused, use `find`/`ls`/`read`.
+- Stopping to wait for an `ask_user` reply that headless runs never send;
+  see Arguments.

package/skills/workflow/cut-it/evals.md

@@ -40,3 +40,104 @@ Expected:
 
 One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
 (30B local, llamacpp on mini), full-auto sandbox. PASS. Sliced the seeded PLAN.md; judge 4/4.
+
+## Battletest record (2026-09-03)
+
+Fixture: `/home/akougkas/eval-temp/harness/test_cutit.py`. S1 reuses this
+evals.md's own fixture text verbatim (`src/todos.js` stub + a concrete
+three-feature `PLAN.md`); S2 is an empty repo with only a `README.md`; S3 is
+a `PLAN.md` that says only "improve performance and clean up the code."
+S1 is the primary grading fixture, scored on 12 checks against the raw
+JSONL's tool-call/safety-block stream, the actual `SPRINT.md` written to
+disk, and the reconstructed final assistant text: zero safety blocks, zero
+real `artifact` tool calls, zero `tasks` calls, `SPRINT.md` exists with a
+`## Battle order` and 2+ numbered slices, every slice carries all six
+required fields (Goal/Depends on/Files/Steps/Done when/Out of scope), no
+horizontal-layering red-flag language, slices trace to the plan's concrete
+features, done-when blocks are command-shaped and testable, and the final
+reply names the path and slice count. S2/S3 are graded on 5 checks each
+(zero safety blocks, zero `artifact` calls, no `SPRINT.md` fabricated,
+correctly flags the gap, recommends a next step). Primary model
+`qwen3.8-27b` on `dynamo` (LM Studio); cross-model confirm on
+`ornith1.5-35b-moe` on `mini` (llama.cpp), run against S1 and S2 both.
+
+**The bug found reading the source, confirmed empirically first**: the
+frozen skill (0.3.0) had `artifact` in `allowed-tools` and titled Step 3
+"Write the artifact." `src/tools/artifact.ts` sets `terminate: true` on
+every successful call — the run ends the instant the tool executes, with
+no further LLM turn to confirm what happened. v1's run called `artifact`
+with `kind: "plan"` and, by luck, an explicit `path: "SPRINT.md"` (the
+model inferred this from "honor a caller-supplied path" even though no
+caller supplied one) — so the file landed in the right place this time, but
+the run still ended mid-sentence ("Writing the sprint:") with no
+confirmation reply, and a separate `tasks` call (also off-surface) drew a
+real safety block. Score 8/12: missing `reply_mentions_sprint_path`, one
+real safety block. Had the model not guessed an explicit path, the default
+would have been `.clio-coder/artifacts/PLAN.md` (kind defaults to `plan`,
+see `core/artifact-paths.ts`) — wrong file, wrong location, same silent
+termination. This is exactly the failure mode the planning category's
+tech-spec baseline hit ("called `artifact` for an early exit... instead of
+a spec").
+
+| run | model | wall | turns | in / out tokens | safety blocks | score | outcome |
+|---|---|---|---|---|---|---|---|
+| baseline (no skill) | qwen3.8-27b | 76s | 7 | 73.8k / 7.1k | 5 (repeated `ls ".cl"` truncated-path retries, benign) | 5/12 | never invoked `/skill cut-it`; read the plan and module correctly, reasoned to genuinely good vertical slices with real done-when checks in its head, then hit a tool-call loop guard and delivered the entire sprint as **prose in the reply, never wrote `SPRINT.md`** — the exact gap this skill exists to close |
+| v1 (frozen 0.3.0) | qwen3.8-27b | 115s | 6 | 68.6k / 11.3k | 1 real (`tasks` refused) | 8/12 | called the real `artifact` tool for Step 3 as titled; terminated the turn immediately after writing, mid-sentence, with no confirmation reply — the artifact-tool bug, confirmed |
+| v2 (first hardened cut, 0.4.0) | qwen3.8-27b | 70s | 6 | 72.4k / 6.5k | 0 | 11/12 (12/12 after a grading-regex fix, see below) | used `write` correctly, used the `git` tool in Step 1, reported the path and slice count in the final reply; the one score miss was a test-harness regex that didn't handle a `**Done when** (fresh state...):` label followed by bulleted checks on the next lines — the actual done-when content was already command-shaped and testable, fixed in the harness, not the skill |
+| v3 (bash-reflex found) | qwen3.8-27b | — | — | — | 2 (1 benign ENOENT, 1 real: `bash` refused) | 11/12 | reached for `bash` (`node --version`, `find`, `ls -la` chained with `&&`) to survey the toolchain even though `bash` was never in `allowed-tools` and nothing in the body named it explicitly — added the same explicit `bash`-refusal line the `tasks` refusal already had, plus a Red flags entry |
+| v4 (final, stable) | qwen3.8-27b | 60s | 6 | 71.4k / 5.6k | 0 | **12/12** | clean run: `context` → `read`/`ls` → `git status` → `write`, self-contained final reply naming path, slice count, and battle order |
+| final-s2 (no plan) | qwen3.8-27b | 26s | 4 | 43.2k / 2.0k | 0 | **5/5** | checked all three plan locations plus `git log`/`status`, correctly stopped with no `SPRINT.md` written, cited the skill's own red-flag language, recommended `grill-me` |
+| final-s3 (vague plan) | qwen3.8-27b | 28s | 5 | 56.2k / 2.5k | 0 | **5/5** | read the one-line `PLAN.md`, explicitly invoked "the skill's own test" (a testable done-when), listed three concrete missing pieces (object/baseline/target for "performance", a definition of "clean", the absent codebase), stopped without writing `SPRINT.md` |
+| final-mini (cross-model, S1) | ornith1.5-35b-moe (mini) | 42s | 6 | 10.8k / 3.2k | 0 | **12/12** | same clean shape on the second model family: `context` → `ls`/`read` → `write`, self-contained final reply |
+| final-mini-s2 (cross-model, S2) | ornith1.5-35b-moe (mini) | 19s | 6 | 4.0k / 1.1k | 0 | **5/5** | correctly found nothing to slice, recommended `grill-me`, offered to slice immediately if pointed at a plan |
+
+**Changes** (0.3.0 -> 0.4.0):
+
+1. **`artifact` removed from `allowed-tools`.** cut-it never needs the real
+   `artifact` tool — `SPRINT.md` is a plain file, always written with
+   `write`. This is the fix for the bug above.
+2. **Step 3 retitled** "Write SPRINT.md" (was "Write the artifact") and its
+   body now says explicitly "use the `write` tool" and names the
+   `artifact`-tool confusion directly, including what it actually does
+   wrong (terminal call, wrong default path under `.clio-coder/artifacts/`,
+   skips the report-back step).
+3. **New Step 4 — Report back**, an explicit final-reply requirement (path
+   written, slice count, one-line battle-order summary). Nothing in the
+   frozen skill told the model to confirm after writing; every hardened run
+   now does.
+4. **`## Arguments` contract**, the section the frozen skill never had:
+   conversational trigger syntax, how a named path splits between "the plan
+   to slice" and "where to write `SPRINT.md`", the no-operator/`ask_user`-
+   auto-cancels rule (adapted from `grill-me` 0.5.0's "this run has no
+   back-and-forth" framing — stated as a fact about the run, not gated on a
+   cancellation response), and — since cut-it's happy path rarely needs a
+   question at all — explicit guidance for the one place it plausibly might
+   (choosing among several candidate plans/milestones): pick the best one,
+   state the alternative, mark `assumed — confirm`, keep going.
+5. **`tasks` and `bash` explicitly named as refused**, in Arguments and Red
+   Flags, both found empirically: v1's `tasks` call (tracking its own
+   steps) and v3's `bash` call (`node --version`, `find`, `ls -la` chained
+   with `&&`, to survey the toolchain) were both real safety blocks despite
+   neither tool ever having been in `allowed-tools`.
+6. Three new Red Flags entries for the concrete failures observed: the
+   `artifact`-tool confusion, ending the run with no final reply, and the
+   `bash` reflex (`tasks` already had informal coverage, now explicit too).
+
+**Still weak**: `code_nav` (in `allowed-tools`) was never exercised — this
+fixture's grounding fit entirely in one small stub file, so `read`/`grep`
+sufficed; a plan referencing a larger call graph might exercise it, none
+was built here. The `ask_user`-unavailable path and the "several candidate
+plans" disambiguation guidance in Arguments are reasoned prose, not
+empirically run — no fixture here seeds multiple plausible plan files or a
+scenario where the model actually reaches for `ask_user`; every hardened
+run reasoned straight to the assumed-confirm default or never needed a
+question at all. S3's "flag as vague" grading is phrase-matching against a
+fixed term list (`vague`, `underspecified`, `insufficient`, ...) — a run
+that flags the same gap in different words would under-score on a
+technicality, though neither observed run did. The v2 grading-regex miss
+(`done_when_testable`, fixed in the harness before v4) is a reminder that
+this fixture's automated score can undercount a genuinely correct skill
+output; the raw `SPRINT.md` files are worth spot-reading, not just the
+score column. No timing was captured for v3 (an incremental re-run to
+confirm the `bash` finding, superseded immediately by v4) — not a gap in
+the skill's coverage, just an artifact of the iteration order.

package/skills/workflow/design-council/SKILL.md

@@ -1,13 +1,13 @@
 ---
 name: design-council
-description: Use when a design decision has real tradeoffs and needs several expert perspectives that challenge each other before code is written, such as architecture choices, API shapes, storage formats, parallelization strategies, or dependency decisions. Quick mode runs a single round for a fast perspective check. Triggers on "council", "debate this", "multiple perspectives", "weigh the options", "what would experts say". Not for a one-question-at-a-time interrogation of a plan; use grill-me. Not for splitting implementation work across workers; use dispatch directly.
+description: Convenes several expert perspectives that challenge each other on a design decision with real trade-offs before code is written; quick mode runs a single round. Not for a one-question-at-a-time interrogation of a plan; use grill-me. Not for splitting implementation across workers; use dispatch directly.
 triggers:
   - convene a design council
   - debate this design
   - get multiple expert perspectives
   - weigh the architecture options
   - what would experts say
-version: 0.3.2
+version: 0.5.0
 license: Apache-2.0
 allowed-tools:
   - dispatch
@@ -17,12 +17,13 @@ allowed-tools:
   - ls
   - context
   - code_nav
+  - ask_user
 clio-coder:
   registry-id: iowarp/clio-coder
   source-url: https://github.com/iowarp/clio-coder/tree/main/skills/workflow/design-council
   audit: pass
   provenance: designed
-  eval-status: scenarios-recorded
+  eval-status: smoke-checked
   model-size: large
   agents:
     - scout
@@ -36,22 +37,95 @@ Run a bounded multi-perspective debate on a real design decision. The council
 surfaces the crux of a disagreement before code commits to one side. It is not
 a ritual: if experts would agree, do not convene it.
 
+## Arguments
+
+```text
+convene a design council on <decision>
+```
+
+There is no flag syntax; the trigger is conversational — "convene a design
+council on X", "debate this design", "get multiple expert perspectives".
+Whatever the user names is the decision. A referenced file, doc, or repo path
+in the same request is Step 0/1's grounding to read first, not a separate
+argument.
+
+**Headless is the enforced default, not a suggestion.** A council is several
+worker runs; nobody is waiting between rounds in a headless run, and every
+round that actually runs costs real wall-clock time on top of the
+orchestrator's own turns. `ask_user` still executes here even though it is
+not in the list below — it is always available regardless — but nothing
+answers it: the single question Step 1 asks comes back `{cancelled: true}`
+immediately, as an ordinary result, not an error. Treat that cancellation (or
+skip the call and reason from this paragraph directly — one is not more valid
+than the other) as the fixed answer **quick mode: exactly three perspectives,
+exactly one round (Positions) plus synthesis.** Do not compose four or five
+perspectives headlessly and do not run a Responses or Convergence round
+headlessly, no matter how contested the topic looks — four/five perspectives
+and multi-round debate are for a live session with an operator who actually
+asked for the deeper pass. This is the one rule the skill's own prior smoke
+history says a model will not reliably self-infer from prose alone, so treat
+the number 3 and the number 1 as hard, not as defaults to raise if the topic
+seems to deserve more.
+
+**Dispatch call shape.** Compose the round's perspectives, then make exactly
+one `dispatch` call with all of them in `tasks` and `mode="parallel"`. If
+that call comes back admission-denied for endpoint or target capacity (a
+single local model instance commonly allows only one concurrent worker, so a
+3-task parallel wave can be denied outright rather than queued), retry the
+identical `tasks` batch in one dispatch call with `mode="sequential"` instead
+— the tool runs them one after another itself. Never split a round into
+several separate one-task `dispatch` calls made one at a time waiting on each
+result before deciding the next; that is the serial-perspectives pattern that
+produced the round-trip cost the skill's own timeout history is about, and it
+does not fix the capacity problem the parallel call already reported. Do not
+call `dispatch(list:true)` to probe capacity first — it answers nothing about
+concurrency and only spends a call.
+
+Declare `intent: {read_roots: [...], relevant_paths: [...]}` with paths
+relative to the repo root on every dispatch call instead of pasting an
+absolute path into `task`/`briefing` prose (a config value, a mount point).
+An absolute path token in briefing/task text with no declared `intent` is
+rejected as `legacy_scope_path_absolute`. Keep a persona's argument in prose,
+never literal shell syntax — a phrase like "you can `rm -rf` the directory"
+inside a dispatch call's text can trip the same damage-control pattern that
+blocks a real destructive shell command, even though nothing executes; say
+"delete the directory" instead.
+
+A dispatch call's own synchronous result already carries every worker's
+output — do not follow it with a `bash`/`read` pass over the receipt file on
+disk to re-read what you already have. The steps below are the plan; do not
+open a `tasks` list for them. This skill's tool surface is exactly
+`dispatch`, `read`, `grep`, `find`, `ls`, `context`, `code_nav`, and
+`ask_user` (`context` and `ask_user` are always available regardless).
+`tasks` and `bash` both sit outside it and any call to either is refused —
+locate files with `find`/`ls`, not `bash find`/`bash ls`; inspect a receipt
+with `read`, not `bash cat`. This skill never writes: `write` and `artifact`
+are not on its surface, so the synthesis in Step 4 is chat output, never a
+file.
+
 ## Step 0 — Check the question is contested
 
 Before composing anyone, ask: would credible experts actually disagree on the
 answer? If every perspective you can imagine picks the same option and differs
-only in caveats, stop here. Say the council is not needed, give the consensus
-answer with the caveats attached, and end.
+only in caveats, stop here — do not dispatch anything. Say the council is not
+needed, give the consensus answer with the caveats attached, and end. Do not
+dispatch a round "just to confirm" a consensus call you already reached —
+that is the ritual this step exists to skip, and it still costs the same
+wall-clock time and worker slots as a real debate. Trust this self-check the
+same way you trust the rest of your own reasoning; a council you convened to
+double-check yourself is not more rigorous than the judgment behind it.
 
 ## Step 1 — Compose perspectives
 
 Derive perspectives from the topic itself, never from a generic role menu.
-Three is the default and the right number for almost every decision. Go to
-four or five only when the decision genuinely has that many independent
-stances, and never headless: each perspective is a worker run, and a model
-that dispatches the round serially instead of in parallel turns five
-perspectives into five sequential runs. If you are running without a user to
-wait on you, use three perspectives and one round.
+On a genuinely contested topic, call `ask_user` once with `mode:
+"single_question"` asking whether this should be a quick pass (three
+perspectives, one round) or the full debate (up to five perspectives, up to
+three rounds). See Arguments for what a headless run does with that call.
+In a live session where the user answers, honor the requested depth. Compose
+three perspectives by default; go to four or five only in that live full-
+debate case, and only when the decision genuinely has that many independent
+stances.
 
 Each perspective gets:
 
@@ -74,11 +148,13 @@ perspective from the read-only recipes in the live catalog:
 - `researcher`: stance leaning on external docs, standards, or papers.
 - `provenance`: stance arguing from runtime evidence and receipts.
 
-Run one round's perspectives in parallel: one `dispatch` call with the round's
-task prompts in `tasks` and `mode="parallel"`. Rounds are sequential. Each
-task prompt carries the persona block, the decision context, and the full
-transcript so far. Workers never edit files; the debate is analysis only.
-Dispatch receipts link every statement to a worker run.
+See Arguments for the exact call shape (one batched `tasks` call, the
+`mode="sequential"` capacity fallback, `intent` for scope, no literal shell
+syntax). Rounds are sequential; a round's own perspectives are the one
+dispatch call. Each task prompt carries the persona block, the decision
+context, and the full transcript so far. Workers never edit files; the
+debate is analysis only. Dispatch receipts link every statement to a worker
+run.
 
 ## Step 3 — Run the rounds
 
@@ -90,14 +166,15 @@ Dispatch receipts link every statement to a worker run.
    still disagrees and why that crux is the crux, and its final
    recommendation.
 
-**Quick mode** (user asked for a light pass): round 1 plus synthesis. No
-responses round.
+**Quick mode** (headless default, or a user asking for a light pass): round 1
+plus synthesis. No responses round, no convergence round.
 
-**Early termination.** After round 1, judge disagreement on the decision
-question itself, not on side conditions. If every position picks the same
-option and differs only in caveats, toggles, or requests to measure later,
-that is consensus: skip rounds 2 and 3, report that the council was not
-needed, and return the consensus with caveats. Never manufacture friction.
+**Early termination** (full-debate mode only). After round 1, judge
+disagreement on the decision question itself, not on side conditions. If
+every position picks the same option and differs only in caveats, toggles,
+or requests to measure later, that is consensus: skip rounds 2 and 3, report
+that the council was not needed, and return the consensus with caveats.
+Never manufacture friction.
 
 ## Step 4 — Synthesize
 
@@ -124,10 +201,11 @@ synthesis line has a citation and the recommendation names its crux.
 
 ## Degraded mode
 
-If dispatch is unavailable or admission-denied, run the same rounds inline:
-write each perspective's contribution yourself, sequentially, same round
-structure and synthesis format. Label the output as degraded (single-model
-debate, no receipts).
+If dispatch is unavailable or admission-denied for a reason other than the
+capacity retry in Arguments (the tool itself is missing from the surface, or
+every retry is denied), run the same rounds inline: write each perspective's
+contribution yourself, sequentially, same round structure and synthesis
+format. Label the output as degraded (single-model debate, no receipts).
 
 ## Boundaries
 
@@ -141,7 +219,19 @@ this skill when the debate needs the full round structure and receipts.
 ## Red Flags
 
 - Perspectives named "optimist" and "pessimist" (role menu, not topic).
-- More than five perspectives, or debate rounds beyond three.
+- Composing four or five perspectives, or running a Responses/Convergence
+  round, in a headless run — the enforced default is exactly three and
+  exactly one, not a ceiling to raise because the topic looks deep.
+- Splitting a round into several single-task `dispatch` calls issued one at a
+  time instead of one batched `tasks` call (retried as `mode="sequential"`
+  on a capacity denial).
+- Calling `dispatch(list:true)` before dispatching the round.
+- Pasting an absolute path into a dispatch `task`/`briefing` instead of a
+  declared `intent`, or quoting literal shell syntax inside a persona's
+  argument.
+- Reaching for `bash`/`read` on a receipt file the dispatch call's own result
+  already contains.
+- Opening a `tasks` list for the round structure; `tasks` is refused.
 - A synthesis that averages positions instead of naming the crux.
 - Manufactured disagreement on a settled question.
 - A worker asked to edit files as part of the debate.

package/skills/workflow/design-council/evals.md

@@ -95,3 +95,164 @@ default and tells a headless run to use three and one round, which is the
 blocker the timeout exposed. Not re-run, so `eval-status` stays
 `scenarios-recorded`; the next campaign has to confirm the shortened council
 fits the ceiling.
+
+## Battletest record (2026-09-03) — 0.4.0 -> 0.5.0
+
+Fixture: `/home/akougkas/eval-temp/harness/test_designcouncil.py`, a
+self-contained repo (`src/checkpoint.py` writing one raw `.npy` per rank per
+step to a shared parallel filesystem, `src/config.py`/`config.yaml` using
+`pyyaml`) adapted from this file's own worked examples. S1 = HDF5-vs-Zarr
+(genuinely contested), S2 = hand-roll-YAML-vs-keep-pyyaml (consensus), S4 =
+"poke holes in my plan" anti-trigger. `runner.py`'s `timeout=` was raised to
+1800s for this skill specifically — the 900s default is a `dispatch` fan-out
+ceiling, not a model-quality one, and the mission called for confirming the
+v0.3.0 "three perspectives, one round" headless fix on its own terms, not
+routing around it. Primary: `qwen3.8-27b`/`dynamo`. Cross-model confirm:
+`ornith1.5-35b-moe`/`mini`. Both runs shared the `dynamo` LM Studio endpoint
+with concurrently active sibling battletest sessions (`grill-me`,
+`workflow-distiller`) launched via `herdr` during this pass — real,
+externally-caused contention, not a fixture artifact; see below.
+
+| run | scenario | model | wall | turns | dispatch calls | safety blocks | score | outcome |
+|---|---|---|---|---|---|---|---|---|
+| baseline (no skill) | S1 | qwen3.8-27b | 121s | 5 | 0 | 0 | 3/11 | no skill invoked (ran under `--no-skills`); the model noticed the installed skill anyway and narrated "four positions, real cross-examination" as one inline monologue — a solid single-model analysis but no dispatch, no receipts, no named-recipe workers |
+| v1 (frozen 0.4.0) | S1 | qwen3.8-27b | 1740s, **killed at the 1800s ceiling** (exit -9) | 22 | 12 | 7 | 4/11 | composed **four** perspectives and ran into **round 2**, both violating the stated headless "three perspectives, one round" default; opened a `tasks` plan (refused); two `dispatch` calls rejected for an absolute path token in `briefing` (`legacy_scope_path_absolute`); one denied for endpoint capacity; one **hard-blocked as `rm-recursive-or-force`** because a perspective's own round-1 argument prose said "...you can `ls`, checksum, `rm -rf`, and publish..." — the admission layer's damage-control scan matched the quoted shell syntax inside the debate text itself, not an executed command; reached for `bash` to inspect a receipt file (refused). This is the same failure class the 2026-08-13 smoke record flagged, confirmed still present, worse: the v0.3.0 fix was never actually followed |
+| v2 (first hardened cut) | S1 | qwen3.8-27b | 416s | 11 | 3 | 4 | 8/11 | one batched `tasks`-array `dispatch` call, all 3 capacity/timeout-denied under real endpoint contention; correctly fell back to Degraded mode, cited the exact denial reasons, labeled the output degraded, produced a complete synthesis with agreements/crux/dissent — no `tasks` misuse, no shell-block trigger, no one-at-a-time call splitting |
+| v2 (post Step 0 strengthen) | S1 | qwen3.8-27b | 304s | 7 | 3 | 4 | 8/11 | same shape: parallel wave denied, sequential retry denied/timed out, correct Degraded fallback, well-formed synthesis citing the transcript |
+| v2 | S2 | qwen3.8-27b | 52.5s / 60.6s (before/after Step 0 edit) | 5 / 6 | **0** | 0 | **5/5** both | Step 0 stopped before any dispatch, stated the council was not needed, gave the consensus (keep `pyyaml`, `safe_load` only) with caveats, and named the repo's actually-contested question (the checkpoint format) as a pointer — no manufactured friction either run |
+| v2 | S4 | qwen3.8-27b | 69.4s | — | 0 | 0 | 3/3 | did not convene a council; ran a grill-me-shaped one-question-at-a-time interrogation instead and named `grill-me` |
+| v2 (cross-model) | S1 | ornith1.5-35b-moe/mini | 384s | 10 | 4 | 5 | 7/11 | 3 of 4 `dispatch` calls denied/timed out on the **same `dynamo` endpoint** (workers defaulted there even though the orchestrator itself ran on `mini`); one stray `monitor` call refused (outside the skill's surface); correctly diagnosed the capacity pattern in its own words ("times out at ~57s... despite reporting 0/4 slots in use... exhausted the capacity fallback") and ran Degraded mode with a clearly labeled warning banner |
+| v2 (cross-model) | S2 | ornith1.5-35b-moe/mini | 239.5s / 188.1s (before/after Step 0 edit) | 6 | 3 / 2 | 3 / 2 | 2/5 both | did **not** skip Step 0's dispatch the way qwen did — ran one capacity-retried round anyway, but stopped there (no round 2/3), reached the same correct consensus ("keep pyyaml", supply-chain caveat preserved as a contingent dissent, not manufactured), and stated the council-not-needed conclusion explicitly; the Step 0 wording strengthen did not change this model's behavior |
+
+**S3 (dispatch unavailable) — no clean forced trigger found, confirmed by
+reading the CLI, not assumed**: `clio-coder run --help`'s `--tool-profile`
+narrows a *dispatched sub-agent's own* tool set and requires `--agent`
+(`clio-coder run: fleet dispatch flags require --agent <recipe-id>`); there
+is no flag that strips `dispatch` from a top-level `--skill` orchestrator
+run's own surface in this harness. Editing the skill's own `allowed-tools`
+to omit `dispatch` would test a different skill, not this one. This is a
+real, documented harness gap, not faked around. In its place, S3's expected
+behavior was exercised **organically, repeatedly, for real**: every S1 run
+on both models hit genuine `dispatch` admission denial or timeout from
+endpoint capacity, and Degraded mode fired correctly every single time —
+labeled degraded, same round structure, synthesis format intact, no
+receipts claimed that did not exist.
+
+**The dispatch-viability and recipe-existence questions, resolved
+empirically:**
+
+- `scout`, `researcher`, and `provenance` all exist as builtin shadow-agent
+  recipes (`src/domains/agents/builtins/*.md`, `capabilityClass: read-only`)
+  and are dispatchable — every hardened run that got a `dispatch` call
+  admitted used one of them by name and got real worker output back.
+- Each recipe's own system prompt declares a rigid JSON-only result
+  contract (`{"findings":[...]}`, `{"source":...}`, `{"confirmedFacts":...}`)
+  that has nothing to do with a debate position. In practice this was
+  **not** a blocker: V1's successfully-admitted round 1 came back as full
+  argumentative prose (named claims, attacks, citations, mind-change
+  conditions) from `researcher`/`scout` workers, not the declared narrow
+  JSON shape — the persona in the task prompt won out over the recipe's
+  own stated output contract for the caller-facing transcript. Worth
+  knowing, not worth re-architecting around.
+- The actual, dominant blocker is **`dispatch` admission capacity on a
+  single-instance local LM Studio target**. `~/.local/state/clio-coder/
+  endpoint-slots.json` records the `dynamo` endpoint (`100.104.197.69:1234`)
+  at `"slots": 1`. A 3-task parallel wave is denied outright
+  (`capacity exceeded (3/1 slots)`), and the `mode="sequential"` retry this
+  pass added to the skill is followed correctly by both models but still
+  gets denied or times out (`admission timed out after ~57s... 0/4 worker
+  slots in use` — a scheduling state that never resolves within the wait
+  window), evidently because the orchestrator's own foreground session
+  already holds the endpoint's only slot. This reproduced independently on
+  `mini` too: workers dispatched from an orchestrator running on `mini`
+  still routed to `dynamo`'s endpoint by default, so the same 1-slot
+  contention applied there as well — confirming the brief's suspicion that
+  worker fan-out may silently default to an unexpected node/target. Some
+  of this pass's contention was real concurrent load, not just self-
+  contention: `ps` showed sibling `grill-me`/`workflow-distiller`
+  battletest sessions actively running via `herdr` against the same
+  endpoint during these runs.
+- The full-auto/`authorityBasis` auto-grant fact supplied going in
+  (`deps.getAutonomy?.() === "full-auto" ? "full-auto-policy" :
+  "operator-plan-approval"`) turned out to be **moot for this skill**:
+  that gate only applies to `agent:"auto"` dispatch requests
+  (`src/tools/dispatch-arguments.ts`, `agentSelection` is only populated
+  when `requestedAgent === "auto"`). Design-council always pins an explicit
+  recipe id (`scout`/`researcher`/`provenance`), so `agentSelection` stays
+  `undefined` and the operator-approval/full-auto-policy distinction never
+  engages — admission for this skill's calls is governed purely by the
+  capacity/reservation machinery above, independent of `--autonomy`.
+- `legacy_scope_path_absolute`: an absolute path token (e.g. a value read
+  from `config.yaml`) pasted into `task`/`briefing` prose without a
+  declared `intent` is rejected. Fixed by telling the skill to declare
+  `intent.read_roots`/`relevant_paths` (relative) on every call instead.
+
+**Changes (0.4.0 -> 0.5.0):**
+
+1. **`## Arguments` contract**, ported from `grill-me`/`cut-it`'s shape.
+   States headless is the *enforced* default (exactly three perspectives,
+   exactly one round), not a self-assessed suggestion — the v0.3.0 fix's
+   prose alone did not hold on either model (V1 composed four perspectives
+   and ran round 2 headlessly).
+2. **Dispatch call shape spelled out**: one batched `tasks`-array call per
+   round; on capacity denial, retry the *same* batch with
+   `mode="sequential"` in one call rather than splitting into several
+   single-task calls issued one at a time (V1's actual failure — 12
+   separate `dispatch` calls, a `list:true` probe, and manual receipt
+   reads via `bash`, all of which V2 stopped doing).
+3. **`intent.read_roots`/`relevant_paths` guidance** to avoid
+   `legacy_scope_path_absolute` rejections from paths quoted in prose.
+4. **No literal shell syntax in a persona's argument text** — added after
+   V1's `rm-recursive-or-force` hard block fired on debate prose, not a
+   real command.
+5. **`ask_user` added to `allowed-tools`** (already always-exempt, now
+   documented) with one call at Step 1 on a contested topic, asking
+   quick-vs-full-debate depth; headlessly it cancels, which *is* the
+   three-perspective/one-round confirmation, mirroring the auto-cancel
+   pattern the planning category established rather than relying on
+   unaided self-assessment.
+6. **Explicit `tasks`/`bash`/receipt-re-read refusal lines**, matching the
+   sibling skills' pattern (`tasks` opened a plan in V1; `bash` was reached
+   for twice across the runs above to re-inspect a receipt the dispatch
+   call's own result already contained).
+7. **Step 0 strengthened** against dispatching "just to confirm" a
+   consensus call already reached — tested on `ornith1.5-35b-moe`/`mini`
+   (still dispatches once) and `qwen3.8-27b`/`dynamo` (unaffected, already
+   correct); see Still weak.
+8. Five new Red flags entries naming the concrete failures observed above.
+
+**Still weak:**
+
+- **Step 0's short-circuit is model-family-dependent.** `qwen3.8-27b`
+  trusts its own contested-or-not judgment and skips `dispatch` entirely
+  for S2 (5/5, 0 dispatch calls, both before and after the Step 0 edit).
+  `ornith1.5-35b-moe` does not: it dispatches one (capacity-retried) round
+  "to confirm" even on the same consensus topic, both before and after the
+  strengthened wording — 2/5 unchanged. It still stops after that one
+  round, still reaches the correct consensus, still preserves the dissent
+  as a contingent caveat rather than manufacturing friction, so the
+  user-facing outcome is fine; the wasted dispatch cost is the actual gap,
+  and more prose did not move it, consistent with the planning category's
+  own finding that some model-family behaviors don't fully generalize no
+  matter how much repetition is added.
+- **S1 never completed as a genuine 3-perspective/1-round council with
+  real receipts** on this pass — every attempt on both models hit real
+  endpoint capacity contention and degraded. The Degraded path is now
+  proven solid, but the "happy path" (dispatch succeeds, synthesis cites
+  real worker receipts) is unverified under this specific pass's
+  conditions; V1's transcript shows it *can* succeed (four real worker
+  runs completed with citable output before the round-2/shell-block
+  failure), so this reads as an availability problem this pass's timing
+  ran into, not a structural block — but it means the exact scoring
+  bullets that depend on `dispatch_receipts_ok` (perspective count,
+  receipts-ok) were not exercised clean this round.
+  `perspectives_dispatched`/`used_named_recipes` in the grading script
+  count attempted, not completed, dispatch tasks for this reason.
+- **No purpose-built S3 fixture** — see above; a future pass with a
+  dedicated low-capacity or offline dispatch target would let this be
+  tested directly instead of relying on organic contention.
+- `code_nav` and `grep` were barely exercised (fixture is small enough
+  that `read`/`ls`/`find` covered grounding in most runs).
+- Only one fixture domain (HDF5-vs-Zarr / YAML-parser) ran this pass; the
+  richer four-perspective MPI-checkpoint example from "Observed live-smoke
+  results" above was not re-run against 0.5.0.