@soleur/opencode 0.0.0-stage → 1.0.0

package/skills/eval-harness/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: eval-harness
+description: This skill provides a promptfoo eval harness that measures whether a Soleur skill or agent edit actually improves behavior, comparing a skill arm against a baseline control arm.
+---
+
+<!-- soleur-cloud-mode:start -->
+**Cloud Mode (Devin):** before pipeline work run `bash "${CLAUDE_PLUGIN_ROOT}/scripts/cloud-detect.sh"`. If `CLAUDE_PLUGIN_ROOT` is unset (cloud exec shells do not export it), resolve the root by IDENTITY, never by script basename: for `d` in `"$HOME/.local/share/devin/cli/plugins/cache"` and `/opt/.devin/plugins`, skip unless `[ -d "$d" ]`, then `MANIFEST="$(find "$d" -path '*/.claude-plugin/plugin.json' -exec grep -l '"name"[[:space:]]*:[[:space:]]*"soleur"' {} + 2>/dev/null | head -1)"`, `ROOT="${MANIFEST%/.claude-plugin/plugin.json}"` — one resolution, two consumers: `$ROOT/scripts/cloud-detect.sh` and `$ROOT/scripts/precommit-guard.sh`. (Shape check, not authentication: a planted `{"name":"soleur"}` dir passes — ADR-179 A11.) `local` or `not-local:no-devin-env` proceeds normally; any other `not-local:<reason>` applies `<plugin-root>/devin/INSTRUCTIONS.md` §Cloud Mode: emit the `--banner`, fan out sequentially with `Reviewed-Coverage: sequential-fallback` disclosure (never claim an independent review ran), `message_user` ack before any secrets read or production mutation, and run `precommit-guard.sh` before any `git commit` — hooks do not fire in cloud.
+<!-- soleur-cloud-mode:end -->
+
+<!-- grok-harness-invoke:start -->
+**Grok Build (`plugins/soleur/lib/harness.ts` `invokeSkill()`):** Read this SKILL.md in this process and run it to completion. A one-segment `soleur:<name>` in this document names a SKILL — on Grok Build, Read `plugins/soleur/skills/<name>/SKILL.md` in this process; it is not a nested tool_use. A multi-segment id such as `soleur:<domain>:<name>` names an AGENT: spawn it, never Read it, and on Grok Build spawn_subagent takes the id with its colons replaced by hyphens (`agentIdToGrokSubagentType`). **Claude Code:** Skill tool for a skill (`soleur:<name>`), Task tool with `subagent_type` for an agent. Forbidden is executing a subset, not the Read.
+<!-- grok-harness-invoke:end -->
+
+# Eval Harness — empirical prompt/agent regression checking
+
+A reproducible [promptfoo](https://www.promptfoo.dev) harness that answers the question
+Soleur could not answer before: *does this prompt/agent edit actually change behavior, or
+does it just carry new text?* It adapts the benchmark methodology of the
+[ponytail](https://github.com/DietrichGebert/ponytail) Claude Code plugin (MIT) to Soleur's
+own surfaces.
+
+**v1 targets two high-traffic classifiers** (the design makes adding a third cheap):
+
+1. **`soleur:go` routing accuracy** — does the routing table produce the correct route token?
+2. **soleur:support:ticket-triage P-level accuracy** — does the priority rubric produce the correct P1/P2/P3?
+
+**Grok Build arm (Phase C #6323):** The harness now covers Grok slash-command + `spawn_subagent` semantics for `soleur:go` routes (in addition to Claude Skill/Task). Golden assertions and regression tests exercise the adapter contract from `lib/harness.ts` (detect via GROK_* markers or argv). The go-routing target in `gated-skills.json` + eval-gate block in `go.md` are the source; projections feed the skill arm for both harnesses. See plan 2026-07-11-feat-grok-phase-c-go-md-eval-harness-plan.md and self-ref in `go.md`.
+
+<decision_gate>
+**API budget.** Each `npx promptfoo eval --repeat 3` run calls the Anthropic API against the key in
+your session — 2 arms × 3 models × the target's golden tasks × 3 repeats. At the current task counts
+that is ≈ **144 API calls** for go-routing (8 tasks) and ≈ **108** for soleur:support:ticket-triage (6 tasks), so
+≈ **230 to run both**. Cost scales with the model mix (one arm runs Opus), the task count, and the
+`--repeat` value (outputs are single tokens, so per-call cost is small — the first full run was well
+under $1). This harness is **opt-in and manual** — it is deliberately NOT wired into per-PR CI (that
+cost decision is separate and later). Soleur does not bill or proxy these calls — Anthropic does,
+against your key. The Soleur LICENSE (BSL 1.1) disclaims warranty for runtime cost; you operate this
+harness against your own budget. To inspect the config without spending, use
+`npx promptfoo validate config` (no API calls).
+</decision_gate>
+
+**Pipeline tally (#9403).** `bash "${CLAUDE_PLUGIN_ROOT}/scripts/pipeline-tally.sh" init` first. Before each `npx promptfoo eval` arm: `bash "${CLAUDE_PLUGIN_ROOT}/scripts/pipeline-tally.sh" gate agent_rounds` — `STOP` → write `knowledge-base/project/specs/<feature>/session-state.md` (`status: budget-capped` + `budget-capped: agent_rounds=<count>/<cap>` + resume) and stop cleanly; `WARN`/`UNKNOWN` → continue. `bash "${CLAUDE_PLUGIN_ROOT}/scripts/pipeline-tally.sh" incr agent_rounds` once per eval arm dispatched.
+
+## How it works (the four ponytail patterns)
+
+1. **promptfoo-driven grid** — arms × models × tasks, N runs. One config file per target:
+   [promptfooconfig-go-routing.yaml](./promptfooconfig-go-routing.yaml) and
+   [promptfooconfig-ticket-triage.yaml](./promptfooconfig-ticket-triage.yaml).
+2. **MEASUREMENT assert** (always passes, records a number) —
+   [measure-classification.cjs](./scripts/measure-classification.cjs) records the
+   classification-correct rate (1.0 if the emitted label matches the golden label, else 0.0). The
+   ponytail `loc.js` analog.
+3. **GATE assert** (fails on wrong output) —
+   [gate-classification.cjs](./scripts/gate-classification.cjs) fails when the emitted label is not
+   a member of the target's closed enum ([go-routes.json](./enums/go-routes.json),
+   [triage-levels.json](./enums/triage-levels.json)). The ponytail `correctness.js` analog.
+4. **BASELINE / CONTROL arm** — the baseline prompt
+   ([go-baseline.txt](./prompts/go-baseline.txt),
+   [triage-baseline.txt](./prompts/triage-baseline.txt)) knows only the label set, not the
+   classifier rules; the skill prompt ([go-skill.txt](./prompts/go-skill.txt),
+   [triage-skill.txt](./prompts/triage-skill.txt)) embeds the production classifier prose. The
+   delta between the two arms is the evidence the rules *produce* the behavior — "that delta is the
+   point."
+
+Both asserts share one parser, [parse-label.cjs](./scripts/parse-label.cjs). Golden tasks are
+synthesized fixtures only (no real user data): [go-routing.jsonl](./tasks/go-routing.jsonl),
+[tasks/ticket-triage.jsonl](./tasks/ticket-triage.jsonl).
+
+**Skill-arm prompts are generated projections (no hand-copy).** The skill-arm prompts
+(`prompts/go-skill.txt`, `prompts/triage-skill.txt`) are a **mechanical projection** of the
+production classifier block, not a hand-distilled paraphrase. Each gated source wraps its rules in
+HTML-comment sentinels — the `soleur:go` routing table in `plugins/soleur/commands/go.md`
+(`<!-- eval-gate:block:go-routing:start -->` … `:end`) and the soleur:support:ticket-triage priority rubric in
+`plugins/soleur/agents/support/ticket-triage.md` (`eval-gate:block:ticket-triage`). The block is the
+single source of truth; [scripts/extract-block.cjs](./scripts/extract-block.cjs) projects it and
+[scripts/gen-skill-prompt.cjs](./scripts/gen-skill-prompt.cjs) wraps it into the skill-arm prompt.
+**Regenerate on any source-block edit** (`node scripts/gen-skill-prompt.cjs --all`); the AC4
+round-trip test ([test/extract-block.test.sh](./test/extract-block.test.sh)) asserts the committed
+projection equals the freshly generated one byte-for-byte, so a stale projection fails CI rather than
+silently measuring an out-of-date classifier.
+
+## Gate mode — validation-gated classifier-skill edits
+
+Beyond measuring a delta, the harness gates *edits* to a classifier block: an edit must not regress
+the corpus AND must make its targeted case pass before it is applied
+([scripts/eval-gate.cjs](./scripts/eval-gate.cjs), proposer-agnostic — heal-skill, compound, or a
+manual edit). The registry [gated-skills.json](./gated-skills.json) lists each gated block:
+`{ source_file, block_id, block_start_marker, block_end_marker, target, projected_prompt_path }`.
+
+- **`--check <file>`** — print `{gated, target, block_id}` for whether a file is a gated source
+  (lookup-only, no API). A proposer runs this first; a non-gated edit proceeds unchanged.
+- **`--dry-run --target <id>`** — print the **skill-arm-only** API-call estimate
+  (`2 (current+candidate) × models × (corpus+1) × repeat`; the baseline control arm is skipped) and
+  exit without spending.
+- **Real run** (`--target <id> --candidate-file <edited-source> --target-task <json|path>
+  [--repeat N] [--append-on-accept]`) — extract the current block (source on disk) and candidate
+  block (edited file); if they are identical it prints `{accept:true, reason:"no gated-block
+  change"}` (the ungateable no-op). Otherwise it projects both into skill-arm prompts, runs promptfoo
+  skill-arm-only for each with `--output`, normalizes the JSON, and calls the **pure**
+  [computeVerdict()](./scripts/verdict.cjs). On accept with `--append-on-accept`, the synthesized
+  target task is appended to `tasks/<target>.jsonl` (real-data-shaped input is rejected per
+  `cq-test-fixtures-synthesized-only`).
+
+The verdict math is pure and unit-tested with zero API
+([test/verdict.test.sh](./test/verdict.test.sh)): `corpus_regressed = candidate_rate < current_rate
+− epsilon` (ε = one-task-equivalent; boundary equality is NOT a regression), `target_task_passes =
+pooled candidate target rate ≥ 0.5`, `accept = !corpus_regressed && target_task_passes`. **Fail-closed:**
+any gate error (missing key, promptfoo non-zero, malformed task) exits non-zero and defaults to NOT
+accept. The gate is only honest while the skill-arm prompt is a mechanical projection of the block
+(see ADR-069) — if the projection link is broken the gate silently no-ops on out-of-block edits.
+
+## Run it
+
+See [README.md](./README.md) for the reproduce commands, how to read the baseline-vs-skill delta,
+and the additive recipe for adding a new target. In short:
+
+```bash
+cd plugins/soleur/skills/eval-harness
+bash scripts/gen-models.sh                                            # refresh model IDs from the registry
+npx promptfoo eval -c promptfooconfig-go-routing.yaml --repeat 3      # ~144 API calls (8 tasks)
+npx promptfoo eval -c promptfooconfig-ticket-triage.yaml --repeat 3   # ~108 API calls (6 tasks)
+npx promptfoo eval -c promptfooconfig-tool-selection.yaml --repeat 5  # ~450 API calls (15 tasks) — manual only
+```
+
+**`tool-selection`** is a **manual measurement-only** target (#5768 AC(c)): it
+measures whether the L3 phase-scoped surface (the hint
+`.claude/hooks/phase-surface-hint.sh` injects) lets the model pick the correct
+next skill more often than the full-surface baseline. The mean of the MEASUREMENT
+score across the two arms IS the before/after uplift. Unlike the **target keys** behind
+[promptfooconfig-go-routing.yaml](./promptfooconfig-go-routing.yaml) and
+[promptfooconfig-ticket-triage.yaml](./promptfooconfig-ticket-triage.yaml) it is **not** in
+[gated-skills.json](./gated-skills.json) — there
+is no prose block to project (the surface lives in `phase-surface-map.json`, not a
+SKILL.md `eval-gate` block), so it never runs as a per-PR projection round-trip;
+run it by hand when you want the AC(c) number.
+
+`--repeat 3` runs each cell 3× so the rate can be a median over runs — a config-level `repeat:` key
+is NOT honored by promptfoo, so the flag is required.
+
+Model IDs are single-sourced via [gen-models.sh](./scripts/gen-models.sh), which reads the three
+current IDs from the TypeScript registry into `models.generated.json` — no model literal is
+hardcoded in any config-class file.
+
+## Tests
+
+Deterministic, no live LLM / no API (stubbed model outputs, recorded result fixtures):
+[gen-models.test.sh](./test/gen-models.test.sh),
+[measure-classification.test.sh](./test/measure-classification.test.sh),
+[gate-classification.test.sh](./test/gate-classification.test.sh),
+[extract-block.test.sh](./test/extract-block.test.sh) (block extraction + AC4 round-trip,
+registry-driven target loop),
+[verdict.test.sh](./test/verdict.test.sh) (pure `computeVerdict` — accept / corpus-regress /
+target-fail / ε-boundary), [eval-gate.test.sh](./test/eval-gate.test.sh) (`--check`, `--dry-run`,
+no-op — no API), and
+[registry-completeness.test.sh](./test/registry-completeness.test.sh) (bidirectional parity
+between `eval-gate:block` source markers and `gated-skills.json` `block_id`s — DEDUP +
+set-equality + charset guard). They run under the standard `bash scripts/test-all.sh` discovery.

package/skills/eval-harness/enums/go-routes.json

@@ -0,0 +1,13 @@
+[
+  "fix",
+  "drain",
+  "drain-prs",
+  "clo-attestation",
+  "review",
+  "legal-threshold",
+  "incident",
+  "implement",
+  "questionnaire",
+  "glossary",
+  "default"
+]

package/skills/eval-harness/enums/incident-threshold.json

@@ -0,0 +1,5 @@
+[
+  "none",
+  "single-user incident",
+  "aggregate pattern"
+]

package/skills/eval-harness/enums/lane.json

@@ -0,0 +1,5 @@
+[
+  "procedural",
+  "single-domain",
+  "cross-domain"
+]

package/skills/eval-harness/enums/tool-selection.json

@@ -0,0 +1,20 @@
+[
+  "brainstorm",
+  "brainstorm-techniques",
+  "plan",
+  "deepen-plan",
+  "plan-review",
+  "spec-templates",
+  "work",
+  "atdd-developer",
+  "test-fix-loop",
+  "resolve-todo-parallel",
+  "qa",
+  "review",
+  "resolve-pr-parallel",
+  "ship",
+  "preflight",
+  "merge-pr",
+  "postmerge",
+  "changelog"
+]

package/skills/eval-harness/enums/triage-levels.json

@@ -0,0 +1,5 @@
+[
+  "P1",
+  "P2",
+  "P3"
+]

package/skills/eval-harness/gated-skills.json

@@ -0,0 +1,34 @@
+[
+  {
+    "source_file": "plugins/soleur/commands/go.md",
+    "block_id": "go-routing",
+    "block_start_marker": "<!-- eval-gate:block:go-routing:start -->",
+    "block_end_marker": "<!-- eval-gate:block:go-routing:end -->",
+    "target": "go-routing",
+    "projected_prompt_path": "plugins/soleur/skills/eval-harness/prompts/go-skill.txt"
+  },
+  {
+    "source_file": "plugins/soleur/agents/support/ticket-triage.md",
+    "block_id": "ticket-triage",
+    "block_start_marker": "<!-- eval-gate:block:ticket-triage:start -->",
+    "block_end_marker": "<!-- eval-gate:block:ticket-triage:end -->",
+    "target": "ticket-triage",
+    "projected_prompt_path": "plugins/soleur/skills/eval-harness/prompts/triage-skill.txt"
+  },
+  {
+    "source_file": "plugins/soleur/skills/brainstorm/references/brainstorm-domain-config.md",
+    "block_id": "lane-inference",
+    "block_start_marker": "<!-- eval-gate:block:lane-inference:start -->",
+    "block_end_marker": "<!-- eval-gate:block:lane-inference:end -->",
+    "target": "lane-inference",
+    "projected_prompt_path": "plugins/soleur/skills/eval-harness/prompts/lane-inference-skill.txt"
+  },
+  {
+    "source_file": "plugins/soleur/skills/incident/SKILL.md",
+    "block_id": "incident-threshold",
+    "block_start_marker": "<!-- eval-gate:block:incident-threshold:start -->",
+    "block_end_marker": "<!-- eval-gate:block:incident-threshold:end -->",
+    "target": "incident-threshold",
+    "projected_prompt_path": "plugins/soleur/skills/eval-harness/prompts/incident-threshold-skill.txt"
+  }
+]

package/skills/eval-harness/models.generated.json

@@ -0,0 +1,5 @@
+[
+  "anthropic:messages:claude-opus-5-5",
+  "anthropic:messages:claude-sonnet-5-5",
+  "anthropic:messages:claude-haiku-5-5"
+]

package/skills/eval-harness/promptfooconfig-go-routing.yaml

@@ -0,0 +1,35 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: "soleur:go routing accuracy — skill arm (routing table) vs baseline (no table). Grok arm (Phase C #6323): asserts /go slash + spawn_subagent paths via harness adapter (see SKILL.md + go.md self-ref)."
+
+# Model IDs are single-sourced. Run `bash scripts/gen-models.sh` to (re)generate
+# models.generated.json from the TS registry; no model literal is hardcoded here,
+# which keeps this config off the model-launch-review auto-fixer and stale-proof.
+providers: file://models.generated.json
+
+# Two arms become two grid columns. The skill arm embeds the /go routing table;
+# the baseline arm knows only the route token set (the control — "that delta is the point").
+prompts:
+  - file://prompts/go-skill.txt
+  - file://prompts/go-baseline.txt
+
+defaultTest:
+  vars:
+    # Closed 7-route enum (single source). promptfoo passes this file:// ref to the
+    # custom asserts VERBATIM (it does not resolve defaultTest.vars file:// refs), so
+    # the asserts read the file themselves via loadEnum() in scripts/parse-label.cjs.
+    enum: file://enums/go-routes.json
+  assert:
+    # MEASUREMENT: always passes, records routing-correct rate (1.0 / 0.0).
+    - type: javascript
+      value: file://scripts/measure-classification.cjs
+    # GATE: fails if the emitted label is not a member of the enum.
+    - type: javascript
+      value: file://scripts/gate-classification.cjs
+
+# N runs per cell come from the `--repeat N` CLI flag (a config-level `repeat:` key
+# is NOT honored by promptfoo). The classification-correct rate is the mean of the
+# measurement assert's per-run scores; promptfoo has no native median, so read the
+# aggregate per-cell score or compute the median across the --repeat runs. See README.
+#
+# Golden tasks: { vars: { input, golden_label } } per line.
+tests: file://tasks/go-routing.jsonl

package/skills/eval-harness/promptfooconfig-incident-threshold.yaml

@@ -0,0 +1,32 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: "incident brand_survival_threshold accuracy — skill arm (Phase 1 criteria) vs baseline (label set only)"
+
+# Model IDs are single-sourced. Run `bash scripts/gen-models.sh` to (re)generate
+# models.generated.json from the TS registry; no model literal is hardcoded here.
+providers: file://models.generated.json
+
+# Two arms become two grid columns. The skill arm embeds the incident §Phase 1 criteria;
+# the baseline arm knows only the threshold token set (the control — "that delta is the point").
+prompts:
+  - file://prompts/incident-threshold-skill.txt
+  - file://prompts/incident-threshold-baseline.txt
+
+defaultTest:
+  vars:
+    # Closed 3-value enum (single source). promptfoo passes this file:// ref to the
+    # custom asserts VERBATIM; the asserts read the file themselves via loadEnum().
+    enum: file://enums/incident-threshold.json
+  assert:
+    # MEASUREMENT: always passes, records threshold-correct rate (1.0 / 0.0).
+    - type: javascript
+      value: file://scripts/measure-classification.cjs
+    # GATE: fails if the emitted label is not a member of the enum.
+    - type: javascript
+      value: file://scripts/gate-classification.cjs
+
+# N runs per cell come from the `--repeat N` CLI flag. Unlike lane-inference, the incident
+# threshold is a genuine LLM-applied prose rubric, so a meaningful skill-vs-baseline delta
+# is expected.
+#
+# Golden tasks: { vars: { input, golden_label } } per line.
+tests: file://tasks/incident-threshold.jsonl

package/skills/eval-harness/promptfooconfig-lane-inference.yaml

@@ -0,0 +1,33 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: "brainstorm lane-inference accuracy — skill arm (lane rule) vs baseline (label set only)"
+
+# Model IDs are single-sourced. Run `bash scripts/gen-models.sh` to (re)generate
+# models.generated.json from the TS registry; no model literal is hardcoded here.
+providers: file://models.generated.json
+
+# Two arms become two grid columns. The skill arm embeds the §Lane Inference rule;
+# the baseline arm knows only the lane token set (the control — "that delta is the point").
+prompts:
+  - file://prompts/lane-inference-skill.txt
+  - file://prompts/lane-inference-baseline.txt
+
+defaultTest:
+  vars:
+    # Closed 3-lane enum (single source). promptfoo passes this file:// ref to the
+    # custom asserts VERBATIM; the asserts read the file themselves via loadEnum().
+    enum: file://enums/lane.json
+  assert:
+    # MEASUREMENT: always passes, records lane-correct rate (1.0 / 0.0).
+    - type: javascript
+      value: file://scripts/measure-classification.cjs
+    # GATE: fails if the emitted label is not a member of the enum.
+    - type: javascript
+      value: file://scripts/gate-classification.cjs
+
+# N runs per cell come from the `--repeat N` CLI flag. lane-inference is a deterministic
+# keyword scan, so the skill-vs-baseline delta is expected to be MUTED — the harness value
+# here is regression detection (an edit to the lane table must not drop a passing task), not
+# arm uplift. See README.
+#
+# Golden tasks: { vars: { input, golden_label } } per line.
+tests: file://tasks/lane-inference.jsonl

package/skills/eval-harness/promptfooconfig-ticket-triage.yaml

@@ -0,0 +1,34 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: "ticket-triage P-level accuracy — skill arm (priority rubric) vs baseline (no rubric)"
+
+# Model IDs are single-sourced — shared with the go-routing config. Run
+# `bash scripts/gen-models.sh` to (re)generate models.generated.json from the TS registry.
+providers: file://models.generated.json
+
+# Two arms become two grid columns. The skill arm embeds the P-level priority rubric;
+# the baseline arm knows only the P1/P2/P3 token set (the control).
+prompts:
+  - file://prompts/triage-skill.txt
+  - file://prompts/triage-baseline.txt
+
+defaultTest:
+  vars:
+    # Closed P1/P2/P3 enum (single source). promptfoo passes this file:// ref to the
+    # custom asserts VERBATIM (it does not resolve defaultTest.vars file:// refs), so
+    # the asserts read the file themselves via loadEnum() in scripts/parse-label.cjs.
+    enum: file://enums/triage-levels.json
+  assert:
+    # MEASUREMENT: always passes, records P-level-correct rate (1.0 / 0.0).
+    - type: javascript
+      value: file://scripts/measure-classification.cjs
+    # GATE: fails if the emitted label is not a member of the enum.
+    - type: javascript
+      value: file://scripts/gate-classification.cjs
+
+# N runs per cell come from the `--repeat N` CLI flag (a config-level `repeat:` key
+# is NOT honored by promptfoo). The classification-correct rate is the mean of the
+# measurement assert's per-run scores; promptfoo has no native median, so read the
+# aggregate per-cell score or compute the median across the --repeat runs. See README.
+#
+# Golden tasks: { vars: { input, golden_label } } per line.
+tests: file://tasks/ticket-triage.jsonl

package/skills/eval-harness/promptfooconfig-tool-selection.yaml

@@ -0,0 +1,33 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: "L3 phase-scoped tool/skill selection accuracy — phase-scoped arm (hint surface) vs full-surface baseline (#5768 AC(c))"
+
+# Model IDs are single-sourced. Run `bash scripts/gen-models.sh` to (re)generate
+# models.generated.json from the TS registry; no model literal is hardcoded here.
+providers: file://models.generated.json
+
+# Two arms become two grid columns. The skill arm foregrounds the current phase's
+# surface (the same hint phase-surface-hint.sh injects); the baseline arm shows the
+# full flat skill catalog with no phase scoping — that delta is the point.
+prompts:
+  - file://prompts/tool-selection-skill.txt
+  - file://prompts/tool-selection-baseline.txt
+
+defaultTest:
+  vars:
+    # Closed skill-label enum (single source). promptfoo passes this file:// ref to
+    # the custom asserts VERBATIM; the asserts read the file themselves via loadEnum().
+    enum: file://enums/tool-selection.json
+  assert:
+    # MEASUREMENT: always passes, records next-skill-correct rate (1.0 / 0.0).
+    # Aggregated across --repeat N and across arms, the mean IS the
+    # phase-scoped-vs-full-surface uplift (the AC(c) before/after number).
+    - type: javascript
+      value: file://scripts/measure-classification.cjs
+    # GATE: fails if the emitted label is not a member of the enum.
+    - type: javascript
+      value: file://scripts/gate-classification.cjs
+
+# Opt-in/manual only (API spend; NOT CI-wired — see SKILL.md API-budget gate).
+# Run: promptfoo eval -c promptfooconfig-tool-selection.yaml --repeat 5
+# Golden tasks: { vars: { phase, input, golden_label } } per line.
+tests: file://tasks/tool-selection.jsonl

package/skills/eval-harness/prompts/go-baseline.txt

@@ -0,0 +1,6 @@
+You are a request router. Classify the user's request into EXACTLY ONE route token from this set: fix, drain, clo-attestation, review, legal-threshold, incident, default.
+
+Respond with ONLY the single route token. No explanation, no punctuation, no extra words.
+
+User request: {{input}}
+Route:

package/skills/eval-harness/prompts/go-skill.txt

@@ -0,0 +1,21 @@
+You are the Soleur /go router. Classify the user's request into EXACTLY ONE route token using this routing table.
+
+Routing table:
+| Intent | Trigger Signals | Routes To |
+|--------|----------------|-----------|
+| fix | The user describes broken behavior, errors, regressions, or something that needs fixing | `soleur:one-shot` |
+| drain | "fix all issues labeled X", "drain the Y backlog", "close all label:Z", "clean up the X backlog" | `soleur:drain-labeled-backlog` |
+| drain-prs | "drain the open PRs", "review and merge all open PRs", "merge all the green PRs", "clear the PR queue/backlog", "triage open pull requests" — draining open PULL REQUESTS (not labeled issues, which is the `drain` row above; not a single named PR, which is the `review` row below) | `soleur:drain-prs` |
+| clo-attestation | The user input is `#N` (or a bare number) AND `gh issue view N` returns a body containing `clo_routable: true` OR matches ALL of: `type:\s*manual`, `manual_because:\s*subjective-design-call`, AND at least one external legal-source signal (`eur-lex\.europa\.eu`, `leginfo\.legislature\.ca\.gov`, `congress\.gov`, `federalregister\.gov`, `legislation\.gov\.uk`, `laws-lois\.justice\.gc\.ca`, OR a `Art\.\s*[0-9]+` / `§\s*[0-9]+` statute citation in the body). Must fire BEFORE the `review` row. | `soleur:legal:clo` agent (Task spawn; the agent's prompt receives the full issue body + the verification question(s) extracted from the body) |
+| review | "review PR", "check this code", PR number reference (when `#N` resolves via `gh pr view N` — confirm PR-vs-issue type before routing) | `soleur:review` |
+| legal-threshold | The user input mentions an inbound vendor MSA, DSAR (data subject access request) / right-to-be-forgotten / data deletion request / account deletion request / data export request / "what data do you have on me", AI vendor terms / vendor AI review, OSS license question (GPL/AGPL/SSPL/copyleft), OR a personal-data exposure / unauthorized access / PII leak that crosses a statutory clock (GDPR Art. 33 72h) — events that exceed founder-grade compliance helping and warrant a downstream specialist | `soleur:legal:clo` agent (Task spawn; the Assess phase emits the threshold catalog from `knowledge-base/legal/recommended-tools.md`) |
+| incident | The user describes a live or recent production incident (outage, customer-impact, Sentry alert) needing classification + PIR. NOTE: pure data breaches without an operational outage route to `legal-threshold` above (statutory clock takes precedence); use `incident` for ops-postmortem scope (uptime, latency, error-rate). | `soleur:incident` |
+| implement | User asks to **implement**, **build**, or **ship** a scoped feature, issue, or phase (e.g. "implement Phase F", "#6325 implement", "ship this feature") — concrete deliverables, not open-ended exploration | `soleur:one-shot` |
+| questionnaire | The user needs an ANSWER THEY DO NOT HOLD from a NAMED THIRD PARTY outside the company — an accountant, bookkeeper, lawyer, notary, auditor, insurer, bank, regulator, landlord, or a vendor — and the deliverable is a document to send them ("ask my accountant", "draft questions for the lawyer", "what do I need from the insurer", "I need to email our bookkeeper about"). The tell is a recipient plus a decision the founder is blocked on; a question the founder can answer, or one about this codebase, is NOT this row — those are `default` and `fix` respectively. Also reached without routing when `.claude/hooks/pre-ask-technical-fork-gate.sh` denies an external-expert `AskUserQuestion` | `soleur:questionnaire-generate` |
+| glossary | The user asks what a term MEANS inside this repository, reports that two agents or documents use one word for different things, or asks to record or sharpen the definition of a project term ("what do we mean by lane", "shard and shard mean two different things here", "write down what a rung is"). Scoped to the repository's own vocabulary — a question about a third-party product's terminology is `default` | `soleur:kb-glossary` |
+| default | Everything else — features, exploration, questions, generation, vague scope without implement/build/ship intent | `soleur:brainstorm` |
+
+Respond with ONLY the single route token (one of: fix, drain, drain-prs, clo-attestation, review, legal-threshold, incident, implement, questionnaire, glossary, default). No explanation, no punctuation, no extra words.
+
+User request: {{input}}
+Route:

package/skills/eval-harness/prompts/incident-threshold-baseline.txt

@@ -0,0 +1,6 @@
+You are an incident brand-survival classifier. Classify the incident description into EXACTLY ONE threshold from this set: none, single-user incident, aggregate pattern.
+
+Respond with ONLY the single threshold value. No explanation, no extra words.
+
+Incident: {{input}}
+Threshold:

package/skills/eval-harness/prompts/incident-threshold-skill.txt

@@ -0,0 +1,12 @@
+You are the Soleur incident brand-survival classifier. Assign a threshold to the incident description using this rubric.
+
+Classification rubric:
+- **none** — no user-facing artifact, no credential surface, no billing path; internal tooling / docs / CI.
+- **single-user incident** — at least one real user impacted (data loss, trust breach, credential exposure, billing surprise) OR any sensitive-data surface is at risk.
+- **aggregate pattern** — repeated or systemic impact across multiple users or tenants; brand-survival-level severity.
+
+Respond with ONLY the single threshold value (exactly one of: none, single-user incident, aggregate pattern). No explanation, no extra words.
+
+Incident:
+{{input}}
+Threshold:

package/skills/eval-harness/prompts/lane-inference-baseline.txt

@@ -0,0 +1,6 @@
+You are a brainstorm lane classifier. Classify the feature description into EXACTLY ONE lane token from this set: procedural, single-domain, cross-domain.
+
+Respond with ONLY the single lane token. No explanation, no punctuation, no extra words.
+
+Feature description: {{input}}
+Lane:

package/skills/eval-harness/prompts/lane-inference-skill.txt

@@ -0,0 +1,15 @@
+You are the Soleur brainstorm lane classifier. Classify the feature description into EXACTLY ONE lane using this rule.
+
+Lane inference rule:
+| Lane | Phase 0.5 effect | Triggers (case-insensitive token scan) |
+|---|---|---|
+| `single-domain` | Spawn one leader (highest-relevance per Assessment Questions). On tie, fall back to **config declaration order** in this file's domain table (first match wins). | No `cross-domain` trigger AND no `procedural` trigger. |
+| `cross-domain` | Spawn ≥2 leaders in parallel. If fewer than 2 match Assessment Questions, **fail-closed expand**: add the next-highest-relevance domain not yet in the set; tie-break by config declaration order. | `audit`, `review`, `security`, `compliance`, `migration`, `data`, `infra`, `regulated`, `payment`, `auth`, `cross-tenant`, `billing`, `gdpr`, `privacy`. |
+| `procedural` | Spawn zero leaders. | `scaffold`, `lockfile`, `format-only`, `rename-only`, `dep-bump`, `version-bump`, `lint-fix` AND no `cross-domain` trigger. |
+
+**Fail-closed default** when keyword inference returns no signal AND operator selects nothing resolvable: `cross-domain`. Cost asymmetry — false-positive fan-out is recoverable; missed user-impact gate is shipped breach.
+
+Respond with ONLY the single lane token (one of: procedural, single-domain, cross-domain). No explanation, no punctuation, no extra words.
+
+Feature description: {{input}}
+Lane:

package/skills/eval-harness/prompts/tool-selection-baseline.txt

@@ -0,0 +1,19 @@
+You are the Soleur workflow agent, mid-task. The full skill catalog is available
+(no phase scoping):
+
+brainstorm, brainstorm-techniques, plan, deepen-plan, plan-review, spec-templates,
+work, atdd-developer, test-fix-loop, resolve-todo-parallel, resolve-pr-parallel,
+review, qa, compound, ship, preflight, merge-pr, postmerge, changelog,
+release-announce, deploy, deploy-docs, archive-kb, kb-search, ship, incident,
+reproduce-bug, harvest-debt, resolve-debt, growth, seo-aeo, content-writer,
+competitive-analysis, frontend-design, ux-audit, gdpr-gate, legal-audit,
+legal-generate, schedule.
+
+Given the situation below, which ONE skill should fire next? Respond with ONLY
+the single skill token (one of: brainstorm, brainstorm-techniques, plan,
+deepen-plan, plan-review, spec-templates, work, atdd-developer, test-fix-loop,
+resolve-todo-parallel, qa, review, resolve-pr-parallel, ship, preflight,
+merge-pr, postmerge, changelog). No explanation, no punctuation, no extra words.
+
+Situation: {{input}}
+Next skill:

package/skills/eval-harness/prompts/tool-selection-skill.txt

@@ -0,0 +1,22 @@
+You are the Soleur workflow agent, mid-task. You are in the {{phase}} phase.
+
+This is the L3 phase-scoped surface for your CURRENT phase (the same hint
+.claude/hooks/phase-surface-hint.sh injects from .claude/phase-surface-map.json).
+Foreground these; later-phase skills are not yet relevant.
+
+Phase-relevant skills by phase:
+- brainstorm: brainstorm, brainstorm-techniques  → next step is usually `plan`
+- plan: plan, deepen-plan, plan-review, spec-templates  → next step is usually `work`
+- work: work, atdd-developer, test-fix-loop, resolve-todo-parallel, qa  → next step is usually `review`
+- review: review, qa, resolve-pr-parallel  → next step is usually `ship`
+- ship: ship, preflight, merge-pr, postmerge, changelog  (terminal)
+
+Given your current phase and the situation below, which ONE skill should fire
+next? Respond with ONLY the single skill token (one of: brainstorm,
+brainstorm-techniques, plan, deepen-plan, plan-review, spec-templates, work,
+atdd-developer, test-fix-loop, resolve-todo-parallel, qa, review,
+resolve-pr-parallel, ship, preflight, merge-pr, postmerge, changelog). No
+explanation, no punctuation, no extra words.
+
+Situation: {{input}}
+Next skill:

package/skills/eval-harness/prompts/triage-baseline.txt

@@ -0,0 +1,7 @@
+You are an issue triager. Assign a priority level of P1, P2, or P3 to the GitHub issue.
+
+Respond with ONLY the single priority token (one of: P1, P2, P3). No explanation, no extra words.
+
+Issue:
+{{input}}
+Priority:

package/skills/eval-harness/prompts/triage-skill.txt

@@ -0,0 +1,10 @@
+You are the Soleur ticket-triage classifier. Assign a priority level to the GitHub issue using this rubric.
+
+Priority rubric:
+- **Severity assignment:** Assign P1 (critical -- blocking, data loss, security), P2 (important -- degraded functionality, workaround exists), P3 (nice-to-have -- cosmetic, enhancement).
+
+Respond with ONLY the single priority token (one of: P1, P2, P3). No explanation, no extra words.
+
+Issue:
+{{input}}
+Priority: