selftune 0.2.9 → 0.2.12

package/skill/references/interactive-config.md

@@ -9,21 +9,21 @@ execution mode, model selection, and key parameters.
 Each mutating workflow has a **Pre-Flight Configuration** step. Follow this pattern:
 
 1. Present a brief summary of what the command will do
-2. Use the `AskUserQuestion` tool to present structured options (max 4 questions per call — split into multiple calls if needed). Mark recommended defaults in option text with `(recommended)`.
+2. Use the `AskUserQuestion` tool to present structured options, one question per tool call. Mark recommended defaults in option text with `(recommended)`.
 3. Parse the user's selections from the tool response
 4. Show a confirmation summary of selected options before executing
 
-**IMPORTANT:** Always use `AskUserQuestion` for pre-flight — never present options as inline numbered text. The tool provides a structured UI that is easier for users to interact with. If `AskUserQuestion` is not available, fall back to inline numbered options.
+**IMPORTANT:** Prefer `AskUserQuestion` for pre-flight, but never batch multiple questions into one payload. Ask one question at a time. If `AskUserQuestion` is not available or Claude Code does not invoke it, fall back to inline numbered options. Do not invent tool responses.
 
 ## Model Tier Reference
 
 When presenting model choices, use this table:
 
-| Tier | Model | Speed | Cost | Quality | Best for |
-|------|-------|-------|------|---------|----------|
-| Fast | `haiku` | ~2s/call | $ | Good | Iteration loops, bulk validation |
-| Balanced | `sonnet` | ~5s/call | $$ | Great | Single-pass proposals, gate checks |
-| Best | `opus` | ~10s/call | $$$ | Excellent | High-stakes final validation |
+| Tier     | Model    | Speed     | Cost | Quality   | Best for                           |
+| -------- | -------- | --------- | ---- | --------- | ---------------------------------- |
+| Fast     | `haiku`  | ~2s/call  | $    | Good      | Iteration loops, bulk validation   |
+| Balanced | `sonnet` | ~5s/call  | $$   | Great     | Single-pass proposals, gate checks |
+| Best     | `opus`   | ~10s/call | $$$  | Excellent | High-stakes final validation       |
 
 ## Quick Path
 

package/skill/references/invocation-taxonomy.md

@@ -67,10 +67,10 @@ Separate from eval types, selftune classifies each **live** skill invocation by
 the user triggered it. This is shown as the `invocation_mode` field in canonical
 telemetry and the "Mode" column in the dashboard.
 
-| Mode | Definition | Example |
-|------|-----------|---------|
-| `explicit` | User typed a slash command (`/skillname`) | `/selftune grade` |
-| `implicit` | User mentioned the skill by name in their prompt | `evolve the selftune skill` |
+| Mode       | Definition                                               | Example                                         |
+| ---------- | -------------------------------------------------------- | ----------------------------------------------- |
+| `explicit` | User typed a slash command (`/skillname`)                | `/selftune grade`                               |
+| `implicit` | User mentioned the skill by name in their prompt         | `evolve the selftune skill`                     |
 | `inferred` | Agent chose the skill autonomously — user never named it | `show me the dashboard` → agent invokes Browser |
 
 ### How classification works
@@ -84,10 +84,10 @@ and mapped to canonical modes in `cli/selftune/normalization.ts` (`deriveInvocat
 
 ### Eval types vs runtime modes
 
-| Concept | Purpose | Values |
-|---------|---------|--------|
-| **Eval invocation type** | Classifying test cases | explicit, implicit, contextual, negative |
-| **Runtime invocation mode** | Classifying live usage | explicit, implicit, inferred |
+| Concept                     | Purpose                | Values                                   |
+| --------------------------- | ---------------------- | ---------------------------------------- |
+| **Eval invocation type**    | Classifying test cases | explicit, implicit, contextual, negative |
+| **Runtime invocation mode** | Classifying live usage | explicit, implicit, inferred             |
 
 `contextual` (eval) and `inferred` (runtime) are related but different: contextual means
 the user's intent is buried in domain context, while inferred means the agent chose the
@@ -99,12 +99,12 @@ skill without any user mention at all.
 
 A healthy skill catches all three positive invocation types:
 
-| Type | Healthy | Unhealthy |
-|------|---------|-----------|
-| Explicit | Catches all | Misses some (broken) |
-| Implicit | Catches most | Only catches explicit (too rigid) |
+| Type       | Healthy      | Unhealthy                                               |
+| ---------- | ------------ | ------------------------------------------------------- |
+| Explicit   | Catches all  | Misses some (broken)                                    |
+| Implicit   | Catches most | Only catches explicit (too rigid)                       |
 | Contextual | Catches many | Only catches explicit + some implicit (needs evolution) |
-| Negative | Rejects all | False positives on keyword overlap |
+| Negative   | Rejects all  | False positives on keyword overlap                      |
 
 ### The Coverage Spectrum
 
@@ -128,6 +128,7 @@ The invocation taxonomy directly drives the evolution feedback loop:
 
 When `selftune eval generate` shows implicit queries that don't trigger the skill, the
 description is too narrow. The `evolve` command will:
+
 1. Extract the missed implicit patterns
 2. Propose description changes that cover them
 3. Validate that existing triggers still work
@@ -146,6 +147,7 @@ Evolution should tighten the scope or add "Don't Use When" clauses.
 ### The Evolution Priority
 
 Fix in this order:
+
 1. **Missed explicit** -- broken, fix immediately
 2. **Missed implicit** -- undertriggering, evolve next
 3. **Missed contextual** -- under-evolved, evolve when implicit is clean
@@ -168,11 +170,11 @@ Each entry in a generated eval set looks like:
 }
 ```
 
-| Field | Description |
-|-------|-------------|
-| `id` | Sequential identifier |
-| `query` | The user's original query text |
-| `expected` | `true` = should trigger, `false` = should not |
+| Field             | Description                                              |
+| ----------------- | -------------------------------------------------------- |
+| `id`              | Sequential identifier                                    |
+| `query`           | The user's original query text                           |
+| `expected`        | `true` = should trigger, `false` = should not            |
 | `invocation_type` | One of: `explicit`, `implicit`, `contextual`, `negative` |
-| `skill_name` | The skill this eval targets |
-| `source_session` | Session ID the query came from (if positive) |
+| `skill_name`      | The skill this eval targets                              |
+| `source_session`  | Session ID the query came from (if positive)             |

package/skill/references/logs.md

@@ -43,6 +43,7 @@ One JSON record per line. Each record is one completed agent session.
 ```
 
 **source values:**
+
 - `claude_code` — written by session-stop.ts (Stop hook)
 - `codex` — written by ingestors/codex-wrapper.ts
 - `codex_rollout` — written by ingestors/codex-rollout.ts
@@ -78,6 +79,7 @@ One record per skill trigger event. Populated by skill-eval.ts (PostToolUse hook
 ```
 
 Optional provenance fields:
+
 - `skill_scope`: `project | global | admin | system | unknown`
 - `skill_project_root`: resolved repo/worktree root when the skill came from a project-local registry
 - `skill_registry_dir`: the registry directory where the resolved `SKILL.md` came from
@@ -88,6 +90,7 @@ record shape, but is rebuilt from source-truth transcripts/rollouts rather than
 hooks alone.
 
 Notes:
+
 - `launcher_base_dir` means selftune recovered Claude's `Base directory for this skill:` launcher metadata. If that recovered `SKILL.md` path points into a stable registry like `~/.agents/skills`, `~/.claude/skills`, `/etc/codex/skills`, or a real project checkout, selftune now reclassifies the invocation into the matching `skill_scope`. Ephemeral temp launcher directories still remain `unknown`.
 - `fallback` means selftune confirmed a skill invocation but could not yet resolve a concrete `SKILL.md` path.
 
@@ -243,6 +246,7 @@ Consumed signal example:
 ```
 
 **signal_type values:**
+
 - `correction` — User pointing out a missed skill ("why didn't you use X?", "you should have used X", "next time use X")
 - `explicit_request` — User asking to use a skill ("please use the X skill", "use the commit skill")
 - `manual_invocation` — Direct `/skill` invocation detected
@@ -265,12 +269,13 @@ One record per evolution action. Written by the evolution and rollback modules.
     "total": 50,
     "passed": 35,
     "failed": 15,
-    "pass_rate": 0.70
+    "pass_rate": 0.7
   }
 }
 ```
 
 **action values:**
+
 - `created` — New evolution proposal generated. `details` starts with `original_description:` prefix preserving the pre-evolution SKILL.md content.
 - `validated` — Proposal tested against eval set. `eval_snapshot` contains before/after pass rates.
 - `deployed` — Updated SKILL.md written to disk. `eval_snapshot` contains final pass rates.
@@ -289,6 +294,7 @@ One record per evolution action. Written by the evolution and rollback modules.
 One JSON object per line. Two observed variants:
 
 **Variant A (nested, current):**
+
 ```json
 {"type": "user", "message": {"role": "user", "content": [{"type": "text", "text": "..."}]}}
 {"type": "assistant", "message": {"role": "assistant", "content": [
@@ -298,6 +304,7 @@ One JSON object per line. Two observed variants:
 ```
 
 **Variant B (flat, older):**
+
 ```json
 {"role": "user", "content": "..."}
 {"role": "assistant", "content": [{"type": "tool_use", "name": "Bash", "input": {"command": "..."}}]}
@@ -309,7 +316,7 @@ Skill reads appear as `Read` tool calls where `input.file_path` ends in `SKILL.m
 
 ---
 
-## Codex Rollout Format ($CODEX_HOME/sessions/YYYY/MM/DD/rollout-*.jsonl)
+## Codex Rollout Format ($CODEX_HOME/sessions/YYYY/MM/DD/rollout-\*.jsonl)
 
 ```json
 {"type": "thread.started", "thread_id": "..."}
@@ -332,13 +339,14 @@ Content is a JSON string containing an array of blocks. Anthropic format:
 
 ```json
 [
-  {"type": "text", "text": "I'll create the presentation."},
-  {"type": "tool_use", "name": "Bash", "input": {"command": "pip install python-pptx"}},
-  {"type": "tool_use", "name": "Read", "input": {"file_path": "/skills/pptx/SKILL.md"}}
+  { "type": "text", "text": "I'll create the presentation." },
+  { "type": "tool_use", "name": "Bash", "input": { "command": "pip install python-pptx" } },
+  { "type": "tool_use", "name": "Read", "input": { "file_path": "/skills/pptx/SKILL.md" } }
 ]
 ```
 
 Tool results appear in subsequent user messages:
+
 ```json
-[{"type": "tool_result", "tool_use_id": "...", "content": "OK", "is_error": false}]
+[{ "type": "tool_result", "tool_use_id": "...", "content": "OK", "is_error": false }]
 ```

package/skill/references/setup-patterns.md

@@ -61,5 +61,7 @@ combined.
 ## Optional Repository Extensions
 
 selftune bundles specialized agent instruction files in `skill/agents/` for
-diagnosis, evolution review, pattern analysis, and setup help. These ship with
-the skill package and are read directly when needed — no installation step required.
+diagnosis, evolution review, pattern analysis, and setup help. These bundled
+files are the canonical source. On Claude Code, `selftune init` syncs
+compatibility copies into `~/.claude/agents/` so native subagent calls keep
+matching the bundled definitions.

package/.claude/agents/diagnosis-analyst.md

@@ -1,156 +0,0 @@
----
-name: diagnosis-analyst
-description: Deep-dive analysis of underperforming skills with root cause identification and actionable recommendations.
----
-
-# Diagnosis Analyst
-
-## Role
-
-Investigate why a specific skill is underperforming. Analyze telemetry logs,
-grading results, and session transcripts to identify root causes and recommend
-targeted fixes.
-
-**Activation policy:** This is a subagent-only role, spawned by the main agent.
-If a user asks for diagnosis directly, the main agent should route to this subagent.
-
-## Connection to Workflows
-
-This agent is spawned by the main agent as a subagent when deeper analysis is
-needed — it is not called directly by the user.
-
-**Connected workflows:**
-- **Doctor** — when `selftune doctor` reveals persistent issues with a specific skill, spawn this agent for root cause analysis
-- **Grade** — when grades are consistently low for a skill, spawn this agent to investigate why
-- **Status** — when `selftune status` shows CRITICAL or WARNING flags on a skill, spawn this agent for a deep dive
-
-The main agent decides when to escalate to this subagent based on severity
-and persistence of the issue. One-off failures are handled inline; recurring
-or unexplained failures warrant spawning this agent.
-
-## Context
-
-You need access to:
-- `~/.claude/session_telemetry_log.jsonl` — session-level metrics
-- `~/.claude/skill_usage_log.jsonl` — skill trigger events
-- `~/.claude/all_queries_log.jsonl` — all user queries (triggered and missed)
-- `~/.claude/evolution_audit_log.jsonl` — evolution history
-- The target skill's `SKILL.md` file
-- Session transcripts referenced in telemetry entries
-
-## Workflow
-
-### Step 1: Identify the target skill
-
-Ask the user which skill to diagnose, or infer from context. Confirm the
-skill name before proceeding.
-
-### Step 2: Gather current health snapshot
-
-```bash
-selftune status
-selftune last
-```
-
-Parse JSON output. Note the skill's current pass rate, session count, and
-any warnings or regression flags.
-
-### Step 3: Pull telemetry stats
-
-```bash
-selftune eval generate --skill <name> --stats
-```
-
-Review aggregate metrics:
-- **Error rate** — high error rate suggests process failures, not trigger issues
-- **Tool call breakdown** — unusual patterns (e.g., excessive Bash retries) indicate thrashing
-- **Average turns** — abnormally high turn count suggests the agent is struggling
-
-### Step 4: Analyze trigger coverage
-
-```bash
-selftune eval generate --skill <name> --max 50
-```
-
-Review the generated eval set. Count entries by invocation type:
-- **Explicit missed** = description is fundamentally broken (critical)
-- **Implicit missed** = description too narrow (common, fixable via evolve)
-- **Contextual missed** = lacks domain vocabulary (fixable via evolve)
-- **False-positive negatives** = overtriggering (description too broad)
-
-Reference `skill/references/invocation-taxonomy.md` for the full taxonomy.
-
-### Step 5: Review grading evidence
-
-Read the skill's `SKILL.md` and check recent grading results. For each
-failed expectation, look at:
-- **Trigger tier** — did the skill fire at all?
-- **Process tier** — did the agent follow the right steps?
-- **Quality tier** — was the output actually good?
-
-Reference `skill/references/grading-methodology.md` for the 3-tier model.
-
-### Step 6: Check evolution history
-
-Read `~/.claude/evolution_audit_log.jsonl` for entries matching the skill.
-Look for:
-- Recent evolutions that may have introduced regressions
-- Rollbacks that suggest instability
-- Plateau patterns (repeated evolutions with no improvement)
-
-### Step 7: Inspect session transcripts
-
-For the worst-performing sessions, read the transcript JSONL files. Look for:
-- SKILL.md not being read (trigger failure)
-- Steps executed out of order (process failure)
-- Repeated errors or thrashing (quality failure)
-- Missing tool calls that should have occurred
-
-### Step 8: Synthesize diagnosis
-
-Compile findings into a structured report.
-
-## Commands
-
-| Command | Purpose |
-|---------|---------|
-| `selftune status` | Overall health snapshot |
-| `selftune last` | Most recent session details |
-| `selftune eval generate --skill <name> --stats` | Aggregate telemetry |
-| `selftune eval generate --skill <name> --max 50` | Generate eval set for coverage analysis |
-| `selftune doctor` | Check infrastructure health |
-
-## Output
-
-Produce a structured diagnosis report:
-
-```markdown
-## Diagnosis Report: <skill-name>
-
-### Summary
-[One-paragraph overview of the problem]
-
-### Health Metrics
-- Pass rate: X%
-- Sessions analyzed: N
-- Error rate: X%
-- Trigger coverage: explicit X% / implicit X% / contextual X%
-
-### Root Cause
-[Primary reason for underperformance, categorized as:]
-- TRIGGER: Skill not firing when it should
-- PROCESS: Skill fires but agent follows wrong steps
-- QUALITY: Steps are correct but output is poor
-- INFRASTRUCTURE: Hooks, logs, or config issues
-
-### Evidence
-[Specific log entries, transcript lines, or metrics supporting the diagnosis]
-
-### Recommendations
-1. [Highest priority fix]
-2. [Secondary fix]
-3. [Optional improvement]
-
-### Suggested Commands
-[Exact selftune commands to execute the recommended fixes]
-```

package/.claude/agents/evolution-reviewer.md

@@ -1,180 +0,0 @@
----
-name: evolution-reviewer
-description: Safety gate that reviews pending evolution proposals before deployment, checking for regressions and quality.
----
-
-# Evolution Reviewer
-
-## Role
-
-Review pending evolution proposals before they are deployed. Act as a safety
-gate that checks for regressions, validates eval set coverage, compares old
-vs. new descriptions, and provides an approve/reject verdict with reasoning.
-
-**Activate when the user says:**
-- "review evolution proposal"
-- "check before deploying evolution"
-- "is this evolution safe"
-- "review pending changes"
-- "should I deploy this evolution"
-
-## Connection to Workflows
-
-This agent is spawned by the main agent as a subagent to provide a safety
-review before deploying an evolution.
-
-**Connected workflows:**
-- **Evolve** — in the review-before-deploy step, spawn this agent to evaluate the proposal for regressions, scope creep, and eval set quality
-- **EvolveBody** — same role for full-body and routing-table evolutions
-
-**Mode behavior:**
-- **Interactive mode** — spawn this agent before deploying an evolution to get a human-readable safety review with an approve/reject verdict
-- **Autonomous mode** — the orchestrator handles validation internally using regression thresholds and auto-rollback; this agent is for interactive safety reviews only
-
-## Context
-
-You need access to:
-- `~/.claude/evolution_audit_log.jsonl` — proposal entries with before/after data
-- The target skill's `SKILL.md` file (current version)
-- The skill's `SKILL.md.bak` file (pre-evolution backup, if it exists)
-- The eval set used for validation (path from evolve output or `evals-<skill>.json`)
-- `skill/references/invocation-taxonomy.md` — invocation type definitions
-- `skill/references/grading-methodology.md` — grading standards
-
-## Workflow
-
-### Step 1: Identify the proposal
-
-Ask the user for the proposal ID, or find the latest pending proposal:
-
-```bash
-# Read the evolution audit log and find the most recent 'validated' entry
-# that has not yet been 'deployed'
-```
-
-Parse `~/.claude/evolution_audit_log.jsonl` for entries matching the skill.
-The latest `validated` entry without a subsequent `deployed` entry is the
-pending proposal.
-
-### Step 2: Run a dry-run if no proposal exists
-
-If no pending proposal is found, generate one:
-
-```bash
-selftune evolve --skill <name> --skill-path <path> --dry-run
-```
-
-Parse the JSON output for the proposal details.
-
-### Step 3: Compare descriptions
-
-Extract the original description from the audit log `created` entry
-(the `details` field starts with `original_description:`). Compare against
-the proposed new description.
-
-**Fallback:** If `created.details` does not contain the `original_description:`
-prefix, read the skill's `SKILL.md.bak` file (created by the evolve workflow
-as a pre-evolution backup) to obtain the original description.
-
-Check for:
-- **Preserved triggers** — all existing trigger phrases still present
-- **Added triggers** — new phrases covering missed queries
-- **Removed content** — anything removed that should not have been
-- **Tone consistency** — new text matches the style of the original
-- **Scope creep** — new description doesn't expand beyond the skill's purpose
-
-### Step 4: Validate eval set quality
-
-Read the eval set used for validation. Check:
-- **Size** — at least 20 entries for meaningful coverage
-- **Type balance** — mix of explicit, implicit, contextual, and negative
-- **Negative coverage** — enough negatives to catch overtriggering
-- **Representativeness** — queries reflect real usage, not synthetic edge cases
-
-Reference `skill/references/invocation-taxonomy.md` for healthy distribution.
-
-### Step 5: Check regression metrics
-
-From the proposal output or audit log `validated` entry, verify:
-- **Pass rate improved** — proposed rate > original rate
-- **No excessive regressions** — regression count < 5% of total evals
-- **Confidence above threshold** — proposal confidence >= 0.7
-- **No explicit regressions** — zero previously-passing explicit queries now failing
-
-### Step 6: Review evolution history
-
-Check for patterns that suggest instability:
-- Multiple evolutions in a short time (churn)
-- Previous rollbacks for this skill (fragility)
-- Plateau pattern (evolution not producing meaningful gains)
-
-### Step 7: Cross-check with watch baseline
-
-If the skill has been monitored with `selftune watch`, check:
-
-```bash
-selftune watch --skill <name> --skill-path <path>
-```
-
-Ensure the current baseline is healthy before introducing changes.
-
-### Step 8: Render verdict
-
-Issue an approve or reject decision with full reasoning.
-
-## Commands
-
-| Command | Purpose |
-|---------|---------|
-| `selftune evolve --skill <name> --skill-path <path> --dry-run` | Generate proposal without deploying |
-| Read eval file from evolve output or audit log | Inspect the exact eval set used for validation |
-| `selftune watch --skill <name> --skill-path <path>` | Check current performance baseline |
-| `selftune status` | Overall skill health context |
-
-## Output
-
-Produce a structured review verdict:
-
-```
-## Evolution Review: <skill-name>
-
-### Proposal ID
-<proposal-id>
-
-### Verdict: APPROVE / REJECT
-
-### Description Diff
-- Added: [new trigger phrases or content]
-- Removed: [anything removed]
-- Changed: [modified sections]
-
-### Metrics
-| Metric | Before | After | Delta |
-|--------|--------|-------|-------|
-| Pass rate | X% | Y% | +Z% |
-| Regression count | - | N | - |
-| Confidence | - | 0.XX | - |
-
-### Eval Set Assessment
-- Total entries: N
-- Type distribution: explicit X / implicit Y / contextual Z / negative W
-- Quality: [adequate / insufficient — with reason]
-
-### Risk Assessment
-- Regression risk: LOW / MEDIUM / HIGH
-- Overtriggering risk: LOW / MEDIUM / HIGH
-- Stability history: [stable / unstable — based on evolution history]
-
-### Reasoning
-[Detailed explanation of the verdict, citing specific evidence]
-
-### Conditions (if APPROVE)
-[Any conditions that should be met post-deploy:]
-- Run `selftune watch` for N sessions after deployment
-- Re-evaluate if pass rate drops below X%
-
-### Required Changes (if REJECT)
-[Specific changes needed before re-review:]
-1. [First required change]
-2. [Second required change]
-```