selftune 0.1.4 → 0.2.1

package/skill/Workflows/Doctor.md

@@ -82,8 +82,37 @@ Doctor validates these areas:
 
 | Check | What it validates |
 |-------|-------------------|
-| Hooks installed | `UserPromptSubmit`, `PostToolUse`, and `Stop` hooks are configured in `~/.claude/settings.json` |
+| Hooks installed | `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, and `Stop` hooks are configured in `~/.claude/settings.json` |
 | Hook scripts exist | The script files referenced by hooks exist on disk |
+| Auto-activate hook | `hooks/auto-activate.ts` is registered in `UserPromptSubmit` and the file is executable |
+| Evolution guard hook | `hooks/evolution-guard.ts` is registered in `PreToolUse` and the file exists |
+
+### Memory Checks
+
+| Check | What it validates |
+|-------|-------------------|
+| Memory directory exists | `~/.selftune/memory/` directory is present |
+| Memory files valid | `context.md`, `decisions.md`, `plan.md` exist and are non-empty (if previously written) |
+
+### Activation Rules Checks
+
+| Check | What it validates |
+|-------|-------------------|
+| Rules file exists | `~/.selftune/activation-rules.json` is present |
+| Rules file valid | The file contains valid JSON conforming to the activation rules schema |
+
+### Agent Checks
+
+| Check | What it validates |
+|-------|-------------------|
+| Optional agent directory exists | If `.claude/agents/` is present, it is readable |
+| Optional agent files present | If the repo bundles helper agents, the expected files are present |
+
+### Dashboard Checks (optional)
+
+| Check | What it validates |
+|-------|-------------------|
+| Dashboard server accessible | `dashboard-server.ts` exists in the CLI directory |
 
 ### Evolution Audit Checks
 
@@ -114,25 +143,45 @@ For each failed check, take the appropriate action:
 | Logs not parseable | Inspect the corrupted log file. Remove or fix invalid lines. |
 | Hooks not installed | Merge `skill/settings_snippet.json` into `~/.claude/settings.json`. Update paths. |
 | Hook scripts missing | Verify the selftune repo path. Re-run `init` if the repo was moved. |
+| Auto-activate missing | Add `hooks/auto-activate.ts` to `UserPromptSubmit` in settings. |
+| Evolution guard missing | Add `hooks/evolution-guard.ts` to `PreToolUse` in settings. |
+| Memory directory missing | Run `mkdir -p ~/.selftune/memory`. |
+| Memory files invalid | Delete and let the memory writer recreate them on next evolve/watch. |
+| Activation rules missing | Copy `assets/activation-rules-default.json` to `~/.selftune/activation-rules.json`. |
+| Activation rules invalid | Validate JSON syntax. Re-copy from template if corrupted. |
+| Agent files missing | If your repo uses optional helper agents, restore them in `.claude/agents/`. Otherwise ignore this advisory. |
 | Audit log invalid | Remove corrupted entries. Future operations will append clean entries. |
 
 ### 4. Re-run Doctor
 
 After fixes, run doctor again to verify all checks pass.
 
-## Common Patterns
-
-**"Something seems broken"**
-> Run doctor first. Report any failing checks with their detail messages.
+## Subagent Escalation
 
-**"Are my hooks working?"**
-> Doctor checks hook installation. If hooks pass but no data appears,
-> verify the hook script paths point to actual files.
+If doctor reveals persistent issues with a specific skill — especially
+recurring failures that basic fixes do not resolve — spawn the
+`diagnosis-analyst` agent as a subagent for root cause analysis.
 
-**"No telemetry available"**
-> Doctor will report missing log files. Install hooks using the
-> `settings_snippet.json` in the skill directory, then run a session.
+## Common Patterns
 
-**"Check selftune health"**
-> Run doctor and report the summary. A clean bill of health means
-> all checks pass and selftune is ready to grade/evolve/watch.
+**User reports something seems broken**
+> Run `selftune doctor`. Parse the JSON output for failed checks. Report
+> each failure's `name` and `detail` to the user with the recommended fix.
+
+**User asks if hooks are working**
+> Run `selftune doctor`. Parse `.checks[]` for hook-related entries. If
+> hooks pass but no data appears, verify hook script paths in
+> `~/.claude/settings.json` point to actual files.
+
+**No telemetry data available**
+> Run `selftune doctor`. Route fixes by platform:
+> - **Claude Code** — route to the Initialize workflow to install hooks
+> - **Codex** — run `selftune ingest codex` or `selftune ingest wrap-codex`
+> - **OpenCode** — run `selftune ingest opencode`
+> - **OpenClaw** — run `selftune ingest openclaw`
+> At least one session must complete after setup to generate telemetry.
+
+**User asks to check selftune health**
+> Run `selftune doctor`. Parse `.healthy` and `.summary`. If `healthy: true`,
+> report that selftune is fully operational. If false, report failed checks
+> and recommended fixes.

package/skill/Workflows/Evals.md

@@ -4,10 +4,19 @@ Generate eval sets from hook logs. Detects false negatives (queries that
 should have triggered a skill but did not) and annotates each entry with
 its invocation type.
 
+## When to Invoke
+
+Invoke this workflow when the user requests any of the following:
+- Generating eval sets or test data for a skill
+- Checking which skills are undertriggering
+- Viewing skill telemetry or usage stats
+- Preparing data before running the Evolve workflow
+- Any request containing "evals", "eval set", "test queries", or "skill stats"
+
 ## Default Command
 
 ```bash
-selftune evals --skill <name> [options]
+selftune eval generate --skill <name> [options]
 ```
 
 ## Options
@@ -20,6 +29,9 @@ selftune evals --skill <name> [options]
 | `--max <n>` | Maximum eval entries to generate | 50 |
 | `--seed <n>` | Random seed for negative sampling | Random |
 | `--out <path>` | Output file path | `evals-<skill>.json` |
+| `--synthetic` | Generate evals from SKILL.md via LLM (no logs needed) | Off |
+| `--skill-path <path>` | Path to SKILL.md (required with `--synthetic`) | — |
+| `--model <model>` | LLM model to use for synthetic generation | Agent default |
 
 ## Output Format
 
@@ -98,19 +110,41 @@ selftune evals --skill <name> [options]
 Discover which skills have telemetry data and how many queries each has.
 
 ```bash
-selftune evals --list-skills
+selftune eval generate --list-skills
+```
+
+Run this first to identify which skills have enough data for eval generation.
+
+### Generate Synthetic Evals (Cold Start)
+
+When a skill has no telemetry data yet, use `--synthetic` to generate eval
+queries directly from the SKILL.md content via an LLM.
+
+```bash
+selftune eval generate --skill pptx --synthetic --skill-path /path/to/skills/pptx/SKILL.md
 ```
 
-Use this first to identify which skills have enough data for eval generation.
+The command:
+1. Reads the SKILL.md file content
+2. Sends it to an LLM with a prompt requesting realistic test queries
+3. Parses the response into eval entries with invocation type annotations
+4. Classifies each positive query using the deterministic `classifyInvocation()` heuristic
+5. Writes the eval set to the output file
+
+Use `--model` to override the default LLM model:
+
+```bash
+selftune eval generate --skill pptx --synthetic --skill-path ./skills/pptx/SKILL.md --model claude-sonnet-4-5-20250514
+```
 
-### Generate Evals
+### Generate Evals (Log-Based)
 
 Cross-reference `skill_usage_log.jsonl` (positive triggers) against
 `all_queries_log.jsonl` (all queries, including non-triggers) to produce
 an eval set annotated with invocation types.
 
 ```bash
-selftune evals --skill pptx --max 50 --out evals-pptx.json
+selftune eval generate --skill pptx --max 50 --out evals-pptx.json
 ```
 
 The command:
@@ -127,20 +161,74 @@ View aggregate telemetry for a skill: average turns, tool call breakdown,
 error rates, and common bash command patterns.
 
 ```bash
-selftune evals --skill pptx --stats
+selftune eval generate --skill pptx --stats
 ```
 
 ## Steps
 
+### 0. Pre-Flight Configuration
+
+Before generating evals, present numbered configuration options to the user inline in your response, then wait for the user's answer before proceeding.
+
+If the user responds with "use defaults", "just do it", or similar shorthand, skip to step 1 using the recommended defaults.
+
+For `--list-skills` or `--stats` requests, skip pre-flight entirely — these are read-only operations.
+
+Present the following options inline in your response:
+
+1. **Generation Mode**
+   - a) Log-based — build evals from real usage logs (recommended if logs exist)
+   - b) Synthetic — generate evals from SKILL.md via LLM (for new skills with no data)
+
+2. **Skill Path** (synthetic mode only)
+   - Provide absolute or relative path to the target SKILL.md
+   - Example: `./skills/pptx/SKILL.md`
+
+3. **Max Entries:** 50 (default — how many eval entries to generate)
+
+4. **Model** (synthetic mode only)
+   - a) Fast (haiku) — quick generation
+   - b) Balanced (sonnet) — better query diversity (recommended)
+   - c) Best (opus) — highest quality synthetic queries
+
+5. **Output Path:** `evals-<skill>.json` (default)
+
+Ask: "Reply with your choices or 'use defaults' for recommended settings."
+
+After the user responds, parse their selections and map each choice to the corresponding CLI flags:
+
+| Selection | CLI Flag |
+|-----------|----------|
+| 1a (log-based) | _(no flag, default)_ |
+| 1b (synthetic) | `--synthetic --skill-path <path>` |
+| Custom max entries | `--max <value>` |
+| 4a (haiku) | `--model haiku` (resolved internally by selftune) |
+| 4b (sonnet) | `--model sonnet` |
+| 4c (opus) | `--model opus` |
+| Custom output path | `--out <path>` |
+
+Show a confirmation summary to the user:
+
+```text
+Configuration Summary:
+  Mode:          log-based
+  Max entries:   50
+  Output:        evals-pptx.json
+
+Proceeding...
+```
+
+Build the CLI command string with all selected flags and continue to step 1.
+
 ### 1. List Available Skills
 
-Run `selftune evals --list-skills` to see what skills have telemetry data. If the target
+Run `selftune eval generate --list-skills` to see what skills have telemetry data. If the target
 skill has zero or very few queries, more sessions are needed before
 eval generation is useful.
 
 ### 2. Generate the Eval Set
 
-Run with `--skill <name>`. Review the output file for:
+Run with `--skill <name>`. Parse the JSON output and review for:
 - Balance between positive and negative entries
 - Coverage of all three positive invocation types (explicit, implicit, contextual)
 - Reasonable negative examples (keyword overlap but wrong intent)
@@ -169,16 +257,20 @@ beyond trigger coverage.
 
 ## Common Patterns
 
-**"What skills are undertriggering?"**
-> Run `selftune evals --list-skills`, then for each skill with significant query counts,
-> generate evals and check for missed implicit/contextual queries.
+**User asks which skills are undertriggering:**
+Run `selftune eval generate --list-skills`, then for each skill with significant query counts,
+generate evals and check for missed implicit/contextual queries.
+
+**User asks to generate evals for a specific skill:**
+Run `selftune eval generate --skill <name>`. Parse the JSON output and review the invocation type distribution.
+Feed the output to the Evolve workflow if coverage gaps exist.
 
-**"Generate evals for pptx"**
-> Run `selftune evals --skill pptx`. Review the invocation type distribution.
-> Feed the output to `evolve` if coverage gaps exist.
+**User asks for skill telemetry or stats:**
+Run `selftune eval generate --skill <name> --stats` for aggregate telemetry.
 
-**"Show me skill stats"**
-> Run `selftune evals --skill <name> --stats` for aggregate telemetry.
+**User has a new skill with no usage data:**
+Use `selftune eval generate --skill <name> --synthetic --skill-path /path/to/SKILL.md`.
+This generates eval queries from the skill description without needing session logs.
 
-**"I want reproducible evals"**
-> Use `--seed <n>` to fix the random sampling of negative examples.
+**User wants reproducible evals:**
+Add `--seed <n>` to fix the random sampling of negative examples.

package/skill/Workflows/EvolutionMemory.md

@@ -0,0 +1,154 @@
+# selftune Evolution Memory
+
+This reference documents the evolution memory system. The agent reads these files automatically during evolve, watch, and rollback workflows for session continuity.
+
+Human-readable session context that survives context window resets. Provides
+continuity across evolve, watch, and rollback workflows by recording outcomes,
+decisions, and known issues in plain markdown files.
+
+## When to Use
+
+- **Reading evolution context for continuity** -- Step 0 in Evolve, Watch, and
+  Rollback workflows reads memory before starting.
+- **Diagnosing what happened in previous sessions** -- the decision log provides
+  a chronological record of every evolution action and its outcome.
+
+## Location
+
+```text
+~/.selftune/memory/
+```
+
+All memory files live in this directory. The directory is created automatically
+on the first write.
+
+## The Three Files
+
+### 1. context.md -- Active Evolutions
+
+Tracks the current state of every skill that has been evolved, watched, or
+rolled back.
+
+**Format:** Markdown with `##` sections.
+
+```markdown
+# Selftune Context
+
+## Active Evolutions
+- pptx: deployed -- Added implicit triggers for slide deck queries
+- csv-parser: regression -- pass_rate=0.65, baseline=0.88
+
+## Known Issues
+- Regression detected for csv-parser: pass_rate=0.65 below baseline=0.88
+
+## Last Updated
+2026-03-01T14:00:00.000Z
+```
+
+**Status values:**
+
+| Status | Meaning |
+|--------|---------|
+| `deployed` | Evolution was deployed successfully |
+| `failed` | Evolution attempted but did not deploy |
+| `regression` | Watch detected a regression in pass rate |
+| `healthy` | Watch confirmed pass rate is within threshold |
+| `rolled-back` | Rollback completed successfully |
+| `rollback-failed` | Rollback was attempted but failed |
+
+### 2. plan.md -- Current Priorities
+
+Records evolution priorities and strategy.
+
+**Format:** Markdown with `##` sections.
+
+```markdown
+# Evolution Plan
+
+## Current Priorities
+1. Improve csv-parser implicit trigger coverage
+2. Re-evolve pptx after eval set expansion
+
+## Strategy
+Focus on skills with highest session volume first.
+
+## Last Updated
+2026-03-01T14:00:00.000Z
+```
+
+### 3. decisions.md -- Append-Only Decision Log
+
+Chronological record of every evolution action. Entries are never removed,
+only appended.
+
+**Entry format:**
+
+```markdown
+## 2026-03-01T14:00:00.000Z -- evolve
+- **Skill:** pptx
+- **Action:** evolved
+- **Rationale:** Missed implicit triggers for slide deck queries
+- **Result:** Deployed with pass_rate improvement 0.70 -> 0.92
+
+---
+```
+
+Each entry contains:
+
+| Field | Description |
+|-------|-------------|
+| Timestamp | ISO 8601 timestamp in the `##` heading |
+| Action type | `evolve`, `rollback`, or `watch` in the heading |
+| Skill | The skill name |
+| Action | Past-tense result: `evolved`, `rolled-back`, or `watched` |
+| Rationale | Why the action was taken |
+| Result | What happened |
+
+Entries are separated by `---` markers.
+
+## Auto-Update Triggers
+
+Memory is updated automatically by the memory writer (`cli/selftune/memory/writer.ts`).
+No manual editing is required during normal operation.
+
+| Trigger | Function | Updates |
+|---------|----------|---------|
+| After evolve completes | `updateContextAfterEvolve` | context.md + decisions.md |
+| After rollback completes | `updateContextAfterRollback` | context.md + decisions.md |
+| After watch completes | `updateContextAfterWatch` | context.md + decisions.md, adds known issues on regression |
+
+## Reading Memory
+
+Step 0 in the Evolve, Watch, and Rollback workflows reads `~/.selftune/memory/context.md`
+before starting any operation. This provides:
+
+- Active evolutions and their current status
+- Known issues from previous runs
+- Last update timestamp
+
+If the file does not exist, the workflow proceeds normally. Memory files are
+created automatically after the first evolve, watch, or rollback operation.
+
+## Resetting Memory
+
+Delete the files in `~/.selftune/memory/` to start fresh:
+
+```bash
+rm -rf ~/.selftune/memory/
+```
+
+They will be recreated automatically on the next evolve, watch, or rollback run.
+
+## Common Patterns
+
+**"What happened in the last evolution?"**
+> Read `~/.selftune/memory/decisions.md`. The most recent entry at the bottom
+> of the file contains the last action, skill, rationale, and result.
+
+**"What's the current state?"**
+> Read `~/.selftune/memory/context.md`. The Active Evolutions section lists
+> every tracked skill and its current status.
+
+**"Memory seems stale"**
+> Delete the files in `~/.selftune/memory/` and run `selftune evolve` or
+> `selftune watch` to recreate them with fresh data.