@iowarp/clio-coder 0.3.1 → 0.3.3

package/skills/research/scientific-modernization/evals.md

@@ -0,0 +1,84 @@
+# Evals - scientific-modernization
+
+Baseline scenarios (run a subagent WITHOUT the skill to capture the gap, then
+WITH the skill to confirm it closes). Rubric is pass/fail per bullet.
+
+## S1 - rewrite an established genomics tool in Rust
+
+Setup: a maintained genomics parser has a large user base, a reference CLI,
+and representative public datasets. The new repository has no validation
+contract. Prompt: "Rewrite this parser in Rust and make it the new default."
+
+Expected:
+
+- Checks the upstream contribution path, maintainers, license, and downstream
+  compatibility obligations before choosing a fork or successor.
+- Writes an acceptance contract before implementation, using the released
+  parser and fixed datasets as an independent parity oracle.
+- Specifies file, API, CLI, error, ordering, missing-value, and numerical
+  compatibility rather than comparing only happy-path output.
+- Splits the migration into independently validated stages with retained raw
+  comparison artifacts and rollback points.
+- Calls the result a prototype unless ownership, release, migration, and
+  maintenance are settled.
+
+## S2 - port a numerical solver to GPU
+
+Setup: a CPU solver is trusted, but floating-point reduction order will change
+on the GPU. Prompt: "Port the solver to CUDA and prove it is correct and
+faster."
+
+Expected:
+
+- Uses the CPU implementation, analytical cases, conserved quantities, or
+  fixed simulated truth as an oracle independent of the CUDA code.
+- Defines units, shapes, absolute and relative tolerances, nondeterminism, and
+  platform variation before the first GPU result.
+- Separates correctness parity from a pre-registered performance experiment
+  and invokes experiment-protocol for the latter.
+- Revalidates the full oracle matrix after each optimization rather than
+  inheriting correctness from an earlier version.
+- Includes large inputs, fresh environments, failures, and numerical edge
+  cases in the last-mile plan.
+
+## S3 - modernize packaging in place
+
+Setup: a scientific Python package has a legacy build, multiple supported
+platforms, and an active upstream. Prompt: "Replace the packaging system and
+release it without changing scientific behavior."
+
+Expected:
+
+- Prefers an upstreamable bounded change over creating a replacement package.
+- Captures fresh-install, upgrade, uninstall, import, CLI, and documented
+  workflow behavior from released artifacts before editing the build.
+- Uses existing scientific outputs as a no-regression oracle even though the
+  requested change appears packaging-only.
+- Delivers the migration in reversible stages and records platform-specific
+  evidence.
+- Names release ownership, deprecation impact, and rollback instructions.
+
+## S4 - anti-triggers
+
+Setup: one request asks why a solver emits NaNs; another asks to benchmark two
+MPI collectives without migrating software.
+
+Expected:
+
+- Routes the NaN diagnosis to scientific-debugging.
+- Routes the isolated performance comparison to experiment-protocol.
+- Does not manufacture a modernization or stewardship project for either.
+
+## Baseline failure modes to watch for (RED)
+
+- Starts a big-bang rewrite immediately and treats self-authored tests as proof.
+- Declares parity from a few hand-picked examples with no tolerance semantics.
+- Mixes correctness and performance so a speedup excuses changed answers.
+- Discovers edge cases only in one final comparison after the rewrite.
+- Replaces an active project without early maintainer coordination.
+- Calls an unowned fork production-ready because installation succeeds.
+
+## Smoke record (2026-08-13)
+
+One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
+(30B local, llamacpp on mini), full-auto sandbox. NOT COMPLETED: loop guard at 75 tool calls. The run also surfaced two harness containment findings (write-tool workspace escape; cross-arm workspace visibility) — harness issues, not skill issues.

package/skills/workflow/design-council/SKILL.md

@@ -0,0 +1,139 @@
+---
+name: design-council
+description: Use when a design decision has real tradeoffs and needs several expert perspectives that challenge each other before code is written, such as architecture choices, API shapes, storage formats, parallelization strategies, or dependency decisions. Quick mode runs a single round for a fast perspective check. Triggers on "council", "debate this", "multiple perspectives", "weigh the options", "what would experts say". Not for a one-question-at-a-time interrogation of a plan; use grill-me. Not for splitting implementation work across workers; use dispatch directly.
+version: 0.3.0
+license: Apache-2.0
+allowed-tools:
+  - dispatch
+  - read
+  - grep
+  - find
+  - ls
+  - context
+  - code_nav
+clio:
+  registry-id: iowarp/clio-coder
+  source-url: https://github.com/iowarp/clio-coder/tree/main/skills/workflow/design-council
+  audit: pass
+  provenance: designed
+  eval-status: scenarios-recorded
+  model-size: large
+  agents:
+    - scout
+    - researcher
+    - provenance
+---
+
+# Design Council
+
+Run a bounded multi-perspective debate on a real design decision. The council
+surfaces the crux of a disagreement before code commits to one side. It is not
+a ritual: if experts would agree, do not convene it.
+
+## Step 0 — Check the question is contested
+
+Before composing anyone, ask: would credible experts actually disagree on the
+answer? If every perspective you can imagine picks the same option and differs
+only in caveats, stop here. Say the council is not needed, give the consensus
+answer with the caveats attached, and end.
+
+## Step 1 — Compose perspectives
+
+Derive perspectives from the topic itself, never from a generic role menu.
+Three is the default and the right number for almost every decision. Go to
+four or five only when the decision genuinely has that many independent
+stances, and never headless: each perspective is a worker run, and a model
+that dispatches the round serially instead of in parallel turns five
+perspectives into five sequential runs. If you are running without a user to
+wait on you, use three perspectives and one round.
+
+Each perspective gets:
+
+- a name and a stance (what it argues for);
+- the expertise it argues from;
+- the specific thing it must attack in the other positions.
+
+Example, "HDF5 vs Zarr for checkpoints": an HPC I/O veteran defending
+single-file HDF5 on parallel filesystems; a cloud-native engineer arguing
+object-store-first Zarr; an operator worried about tooling and recovery; a
+numerics lead demanding bit-exact round-trips. Never "optimist, pessimist,
+pragmatist".
+
+## Step 2 — Dispatch each perspective as a read-only worker
+
+The recipe fixes capability; your task prompt supplies the persona. Pick per
+perspective from the read-only recipes in the live catalog:
+
+- `scout`: stance grounded in this repository's code.
+- `researcher`: stance leaning on external docs, standards, or papers.
+- `provenance`: stance arguing from runtime evidence and receipts.
+
+Run one round's perspectives in parallel: one `dispatch` call with the round's
+task prompts in `tasks` and `mode="parallel"`. Rounds are sequential. Each
+task prompt carries the persona block, the decision context, and the full
+transcript so far. Workers never edit files; the debate is analysis only.
+Dispatch receipts link every statement to a worker run.
+
+## Step 3 — Run the rounds
+
+1. **Positions.** Each perspective states its position, its strongest
+   argument, and what evidence would change its mind.
+2. **Responses.** Each perspective receives the round 1 transcript and must
+   respond to named points from the others: concede, rebut, or sharpen.
+3. **Convergence.** Each perspective states what it now agrees with, where it
+   still disagrees and why that crux is the crux, and its final
+   recommendation.
+
+**Quick mode** (user asked for a light pass): round 1 plus synthesis. No
+responses round.
+
+**Early termination.** After round 1, judge disagreement on the decision
+question itself, not on side conditions. If every position picks the same
+option and differs only in caveats, toggles, or requests to measure later,
+that is consensus: skip rounds 2 and 3, report that the council was not
+needed, and return the consensus with caveats. Never manufacture friction.
+
+## Step 4 — Synthesize
+
+After the last round, you (the orchestrator) write:
+
+```markdown
+## Council Synthesis - <decision>
+
+Agreements:
+- <point> (all perspectives, round <n>)
+
+Live disagreements:
+- <point> - crux: <the fact or value judgment that would settle it>
+
+Recommendation:
+- <choice> because <reasoning grounded in the transcript>
+
+Dissent preserved:
+- <perspective>: <the objection that survives the recommendation>
+```
+
+Cite the transcript (perspective and round) for every claim. Done when every
+synthesis line has a citation and the recommendation names its crux.
+
+## Degraded mode
+
+If dispatch is unavailable or admission-denied, run the same rounds inline:
+write each perspective's contribution yourself, sequentially, same round
+structure and synthesis format. Label the output as degraded (single-model
+debate, no receipts).
+
+## Boundaries
+
+Stress-testing a plan by questioning its author one question at a time is
+`grill-me`, not a council. Splitting implementation work across workers is
+plain dispatch, not a council. Council workers analyze; they never build.
+
+## Red Flags
+
+- Perspectives named "optimist" and "pessimist" (role menu, not topic).
+- More than five perspectives, or debate rounds beyond three.
+- A synthesis that averages positions instead of naming the crux.
+- Manufactured disagreement on a settled question.
+- A worker asked to edit files as part of the debate.
+- Synthesis statements that no transcript line supports.

package/skills/workflow/design-council/evals.md

@@ -0,0 +1,97 @@
+# Evals - design-council
+
+Baseline scenarios (run a subagent WITHOUT the skill to capture the gap, then
+WITH the skill to confirm it closes). Rubric is pass/fail per bullet.
+
+## S1 - storage-format decision with real tradeoffs
+
+Setup: a project checkpointing large arrays; prompt: "council: should we use
+HDF5 or Zarr for our checkpoint format?"
+
+Expected:
+
+- Composes 3 to 5 perspectives from the topic (each with a name, stance,
+  expertise, and a target to attack), not from a generic role menu.
+- Dispatches perspectives as read-only workers (scout, researcher, or
+  provenance recipes), parallel within a round.
+- Round 2 responses address named points from the round 1 transcript.
+- Disagreement is surfaced with its crux stated, not averaged away.
+- Synthesis lists agreements, live disagreements with the crux,
+  a recommendation, and preserved dissent, citing the transcript.
+- Dispatch receipts exist linking statements to worker runs.
+
+## S2 - consensus topic
+
+Setup: prompt asks the council to debate a question with an obvious answer for
+this repo (for example "should we hand-roll our own YAML parser or keep the
+existing dependency?").
+
+Expected:
+
+- Round 1 comes back in agreement; the council stops there.
+- Says explicitly that the council was not needed and returns the consensus.
+- Does not run responses or convergence rounds, and does not manufacture
+  friction.
+
+## S3 - dispatch unavailable
+
+Setup: dispatch is admission-denied or absent in the environment.
+
+Expected:
+
+- Falls back to inline sequential perspectives with the same round structure
+  and synthesis format.
+- Labels the result as degraded (single-model debate, no receipts).
+
+## S4 - anti-trigger: plan stress-test
+
+Setup: user says "poke holes in my plan" or wants their decisions interrogated
+one at a time.
+
+Expected:
+
+- Refers to grill-me instead of convening a council.
+
+## Baseline failure modes to watch for (RED)
+
+- Generic personas (optimist, pessimist, devil's advocate) with no expertise
+  or attack target.
+- One blended essay of pros and cons instead of independent positions that
+  respond to each other.
+- No worker dispatch at all, or workers asked to edit files.
+- A "balanced" synthesis that hides the crux of disagreement.
+- Fabricated debate on a consensus topic to justify the ceremony.
+
+## Observed live-smoke results
+
+Run 2026-07-01/02, headless `clio-coder run --skill` against a scratch fixture (an
+MPI checkpointing module writing one raw npy per rank per step).
+
+- Full council (atomic rename vs direct write): three bounded rounds, four
+  topic-composed perspectives each dispatched as read-only `scout` workers
+  (parallel within rounds, sequential between), receipts on every statement
+  (exit 0, agentId scout), synthesis with agreements, a crux, and the
+  Performance Engineer's dissent preserved and answered. Note the topic
+  proved genuinely debatable, so full rounds were correct.
+- Degraded fallback (HDF5 vs Zarr, run while the fleet target had model
+  residency failures): the skill attempted a parallel `dispatch` first,
+  observed worker failures in receipts, then ran the same rounds inline and
+  labeled the result degraded exactly as the clause requires.
+- Early termination (S2, zero-padded vs unpadded checkpoint filenames): the
+  first draft of the skill ran full rounds on side-quibbles for two
+  debatable-in-hindsight "consensus" topics; the composition and termination
+  clauses were tightened to judge consensus on the decision question itself.
+  With the shipped wording the council declared itself not needed before any
+  dispatch, cited the rule, and returned the consensus with caveats. No
+  manufactured friction.
+
+## Smoke record (2026-08-13)
+
+One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
+(30B local, llamacpp on mini), full-auto sandbox. NOT COMPLETED: treatment timed out at the 900s ceiling mid-council (serial perspectives). Needs a longer timeout or fewer personas for headless eval; skill was visibly working when cut.
+
+Follow-up (2026-08-13, v0.3.0): Step 1 now makes three perspectives the
+default and tells a headless run to use three and one round, which is the
+blocker the timeout exposed. Not re-run, so `eval-status` stays
+`scenarios-recorded`; the next campaign has to confirm the shortened council
+fits the ceiling.

package/skills/workflow/grill-me/SKILL.md

@@ -0,0 +1,186 @@
+---
+name: grill-me
+description: Use when the user wants a plan, design, or idea stress-tested through a phased one-question-at-a-time interview before any code is written, or when intent is too ambiguous to plan from. Scans available context first, reviews known facts, fills missing decisions, respects stop signals, and ends with a compact decision log. Triggers on "grill me", "interview me", "stress-test this plan", "poke holes in this".
+version: 0.3.2
+license: Apache-2.0
+allowed-tools:
+  - read
+  - grep
+  - ls
+  - find
+  - git
+  - context
+  - code_nav
+  - ask_user
+clio:
+  registry-id: iowarp/clio-coder
+  source-url: https://github.com/iowarp/clio-coder/tree/main/skills/workflow/grill-me
+  audit: pass
+  provenance: designed
+  eval-status: smoke-checked
+  model-size: large
+---
+
+# Grill Me
+
+Run a rigorous, repo-aware interview that turns a vague plan into explicit
+decisions. The point is not to interrogate for sport; it is to surface hidden
+branches before anyone writes code.
+
+## Operating Contract
+
+- Use `ask_user` for the interview whenever it is active.
+- For every interview round, call `ask_user` with `mode: "single_question"` and
+  exactly one question.
+- On the first ask for a normal grill-me run, set `max_rounds` to a bounded
+  value, usually `12` and at most `16` unless the user explicitly asked for a
+  very deep interview.
+- Put your recommended answer first when options are natural. Include 2-3 real
+  alternatives with short tradeoff descriptions.
+- The user answers in natural language. You translate answers into compact
+  decision keys and rationale when you call `ask_user` with `action: "complete"`.
+- If `ask_user` is unavailable, ask in plain text, still one question at a
+  time, and keep an internal decision log.
+
+## Phase Map
+
+Walk phases in order unless the user names a narrower target. Do not skip the
+scan. A phase can be "review" when context already contains a plausible answer
+or "fill" when the decision is genuinely missing.
+
+| Phase | Scope | Default mode |
+|---|---|---|
+| 0 | Scan available context: user prompt, named files, repo structure, git state, existing plans/specs | No questions unless the subject is missing |
+| 1 | Frame: user, problem, outcome, non-goals, success criteria | Fill or review |
+| 2 | Current state: existing code, constraints, conventions, integration points, prior attempts | Review |
+| 3 | Shape: data model, API/UX surface, ownership boundaries, naming, compatibility | Fill |
+| 4 | Risk: failure modes, migrations, rollout, test strategy, observability, reversibility | Fill |
+| 5 | Delivery: first slice, done-when checks, deferrals, handoff target (`prd`, `cut-it`, or direct implementation) | Review then complete |
+
+## Workflow
+
+### Step 1 - Scan
+
+Read what the user already gave you. If the task references files, plans, code,
+tests, or project conventions, inspect them before asking. Prefer
+`context(scope="workspace")`, `grep`, `read`, and codewiki tools over guessing.
+
+Privately build a phase map:
+
+- known facts
+- assumptions worth challenging
+- missing decisions
+- dependencies between decisions
+- likely deferrals and why they might be safe
+
+Never spend the user's attention on facts that the repo answers. If you found
+the answer in code, summarize it briefly and ask only whether it should remain
+true.
+
+### Step 2 - Choose Review Or Fill
+
+For each phase:
+
+- **Review mode**: context already has an answer. Present the finding in one or
+  two sentences, then ask a targeted question such as "Is this still accurate?"
+  or "Should we keep this constraint?"
+- **Fill mode**: context is sparse or ambiguous. Ask the highest-leverage
+  missing decision first.
+
+Always resolve root decisions before leaves, in the Question Priority order
+below.
+
+### Step 3 - Ask One Question
+
+Each `ask_user` round contains one question only:
+
+- one stable `header`
+- one concrete question
+- recommended option first when options fit
+- no multi-part wording hidden inside the question
+
+Bad: "Who is this for, what should v1 include, and how should we test it?"
+
+Good: "Which user should v1 optimize for first?"
+
+If an answer is vague, ask a follow-up on the same branch. Do not jump to a new
+branch while the current one is still unresolved.
+
+### Step 4 - Respect Stop Signals
+
+Stop immediately when the user says "stop", "enough", "later", "done", "next
+time", or cancels the modal. Do not ask another question to confirm stopping.
+
+If you have enough decisions to be useful, call:
+
+```json
+{
+  "action": "complete",
+  "summary": "Short interview closeout.",
+  "decisions": [
+    {
+      "key": "primary_outcome",
+      "value": "Smallest useful slice",
+      "rationale": "The user prioritized fast validation over broad architecture.",
+      "confidence": "high",
+      "source_question": "What should this plan optimize for first?"
+    }
+  ]
+}
+```
+
+Then provide the decision log. If the stop happened before enough context,
+state the partial decisions and the next unresolved root question.
+
+### Step 5 - Complete
+
+Before final prose, call `ask_user` with `action: "complete"` and a compact
+`decisions` array. Then write the final decision log:
+
+```markdown
+## Decision Log - <topic>
+1. <decision> - chosen over <alternative> because <reason>
+2. ...
+
+Deferred:
+- <item> - safe because <reason>
+
+Open risks:
+- <risk or unresolved branch>
+
+Recommended next step:
+- <prd | cut-it | direct implementation> - <why>
+```
+
+## Question Priority
+
+Use this ordering when several questions are possible:
+
+1. User and problem being solved.
+2. Primary success measure.
+3. Explicit non-goals.
+4. Existing constraints from repo or environment.
+5. Data/API/UX boundary.
+6. Failure modes and recovery.
+7. Tests and done-when checks.
+8. First implementation slice.
+9. Naming and polish.
+
+## Decision Rules
+
+- If the user says "whatever you think", record your recommendation as the
+  decision and say so.
+- If two choices are both viable, choose the one that reduces irreversible
+  work unless the user explicitly values speed or breadth more.
+- If a decision is safe to defer, record why and what later signal will force
+  it.
+- If the plan is too vague to slice or implement, say that clearly and continue
+  interviewing instead of fabricating certainty.
+
+## Red Flags
+
+- Asking multiple questions in one `ask_user` round.
+- Asking about facts discoverable from the repo.
+- Letting `ask_user` hit the round limit without completing the interview.
+- Ending with a summary paragraph instead of the decision log.
+- Treating cancellation as permission to keep asking.

package/skills/workflow/grill-me/evals.md

@@ -0,0 +1,78 @@
+# Evals - grill-me
+
+Baseline scenarios (run a subagent WITHOUT the skill to capture the gap, then
+WITH the skill to confirm it closes). Rubric is pass/fail per bullet.
+
+## S1 - vague feature request
+
+Setup: repo with an existing config domain. Prompt: "grill me on adding plugin
+support."
+
+Expected:
+
+- Scans the existing config/extension code before asking anything the repo can
+  answer.
+- Starts with a root decision such as outcome, target user, or scope boundary.
+- Uses `ask_user` with `mode: "single_question"` and exactly one question in
+  the round.
+- Puts its recommended answer first when options are supplied.
+- Ends by calling `ask_user` with `action: "complete"` and then writes a
+  decision log, not a summary paragraph.
+
+## S2 - answerable-from-repo question
+
+Setup: plan mentions "the test runner". The repo's package.json defines it.
+
+Expected:
+
+- Does NOT ask the user which test runner is used; reads package.json instead.
+- States what it found and asks only whether that constraint should remain
+  true.
+- Treats the area as review mode, not fill mode.
+
+## S3 - user defers
+
+Setup: mid-interview, user answers "whatever you think is best."
+
+Expected:
+
+- Records its own recommendation as the decision and says so explicitly.
+- Does not silently skip the branch.
+- Continues only if another root decision remains.
+
+## S4 - long phased interview
+
+Setup: prompt asks for a deep stress test of a large design.
+
+Expected:
+
+- First `ask_user` call sets a bounded `max_rounds` value such as 12 or 16.
+- Still asks one question per round.
+- Completes before the limit when decisions are sufficient.
+- If the limit is near, closes with current decisions and open risks instead
+  of continuing to ask.
+
+## S5 - stop signal
+
+Setup: mid-interview, user says "stop", "enough", "later", or cancels the
+modal.
+
+Expected:
+
+- Stops immediately and does not ask a confirmation question.
+- Calls `ask_user` complete when possible with partial decisions.
+- Final response includes partial decisions and the next unresolved root
+  question.
+
+## Baseline failure modes to watch for (RED)
+
+- Question batching: more than one question in an `ask_user` round.
+- Interview starts cold without reading repo facts that are clearly relevant.
+- Interview ends when the user gets tired, with no decision log.
+- Asks about facts discoverable via grep/read.
+- Hits the round limit without a useful closeout.
+
+## Smoke record (2026-08-13)
+
+One representative scenario via `clio-coder skills eval` against Nemo-3.5-Lightning
+(30B local, llamacpp on mini), full-auto sandbox. PASS (smoke). Skill loaded and interviewed; judge emitted nothing (truncation).