selftune 0.1.4 → 0.2.0

package/skill/Workflows/EvolutionMemory.md

@@ -0,0 +1,152 @@
+# selftune Evolution Memory
+
+Human-readable session context that survives context window resets. Provides
+continuity across evolve, watch, and rollback workflows by recording outcomes,
+decisions, and known issues in plain markdown files.
+
+## When to Use
+
+- **Reading evolution context for continuity** -- Step 0 in Evolve, Watch, and
+  Rollback workflows reads memory before starting.
+- **Diagnosing what happened in previous sessions** -- the decision log provides
+  a chronological record of every evolution action and its outcome.
+
+## Location
+
+```text
+~/.selftune/memory/
+```
+
+All memory files live in this directory. The directory is created automatically
+on the first write.
+
+## The Three Files
+
+### 1. context.md -- Active Evolutions
+
+Tracks the current state of every skill that has been evolved, watched, or
+rolled back.
+
+**Format:** Markdown with `##` sections.
+
+```markdown
+# Selftune Context
+
+## Active Evolutions
+- pptx: deployed -- Added implicit triggers for slide deck queries
+- csv-parser: regression -- pass_rate=0.65, baseline=0.88
+
+## Known Issues
+- Regression detected for csv-parser: pass_rate=0.65 below baseline=0.88
+
+## Last Updated
+2026-03-01T14:00:00.000Z
+```
+
+**Status values:**
+
+| Status | Meaning |
+|--------|---------|
+| `deployed` | Evolution was deployed successfully |
+| `failed` | Evolution attempted but did not deploy |
+| `regression` | Watch detected a regression in pass rate |
+| `healthy` | Watch confirmed pass rate is within threshold |
+| `rolled-back` | Rollback completed successfully |
+| `rollback-failed` | Rollback was attempted but failed |
+
+### 2. plan.md -- Current Priorities
+
+Records evolution priorities and strategy.
+
+**Format:** Markdown with `##` sections.
+
+```markdown
+# Evolution Plan
+
+## Current Priorities
+1. Improve csv-parser implicit trigger coverage
+2. Re-evolve pptx after eval set expansion
+
+## Strategy
+Focus on skills with highest session volume first.
+
+## Last Updated
+2026-03-01T14:00:00.000Z
+```
+
+### 3. decisions.md -- Append-Only Decision Log
+
+Chronological record of every evolution action. Entries are never removed,
+only appended.
+
+**Entry format:**
+
+```markdown
+## 2026-03-01T14:00:00.000Z -- evolve
+- **Skill:** pptx
+- **Action:** evolved
+- **Rationale:** Missed implicit triggers for slide deck queries
+- **Result:** Deployed with pass_rate improvement 0.70 -> 0.92
+
+---
+```
+
+Each entry contains:
+
+| Field | Description |
+|-------|-------------|
+| Timestamp | ISO 8601 timestamp in the `##` heading |
+| Action type | `evolve`, `rollback`, or `watch` in the heading |
+| Skill | The skill name |
+| Action | Past-tense result: `evolved`, `rolled-back`, or `watched` |
+| Rationale | Why the action was taken |
+| Result | What happened |
+
+Entries are separated by `---` markers.
+
+## Auto-Update Triggers
+
+Memory is updated automatically by the memory writer (`cli/selftune/memory/writer.ts`).
+No manual editing is required during normal operation.
+
+| Trigger | Function | Updates |
+|---------|----------|---------|
+| After evolve completes | `updateContextAfterEvolve` | context.md + decisions.md |
+| After rollback completes | `updateContextAfterRollback` | context.md + decisions.md |
+| After watch completes | `updateContextAfterWatch` | context.md + decisions.md, adds known issues on regression |
+
+## Reading Memory
+
+Step 0 in the Evolve, Watch, and Rollback workflows reads `~/.selftune/memory/context.md`
+before starting any operation. This provides:
+
+- Active evolutions and their current status
+- Known issues from previous runs
+- Last update timestamp
+
+If the file does not exist, the workflow proceeds normally. Memory files are
+created automatically after the first evolve, watch, or rollback operation.
+
+## Resetting Memory
+
+Delete the files in `~/.selftune/memory/` to start fresh:
+
+```bash
+rm -rf ~/.selftune/memory/
+```
+
+They will be recreated automatically on the next evolve, watch, or rollback run.
+
+## Common Patterns
+
+**"What happened in the last evolution?"**
+> Read `~/.selftune/memory/decisions.md`. The most recent entry at the bottom
+> of the file contains the last action, skill, rationale, and result.
+
+**"What's the current state?"**
+> Read `~/.selftune/memory/context.md`. The Active Evolutions section lists
+> every tracked skill and its current status.
+
+**"Memory seems stale"**
+> Delete the files in `~/.selftune/memory/` and run `selftune evolve` or
+> `selftune watch` to recreate them with fresh data.

package/skill/Workflows/Evolve.md

@@ -19,8 +19,12 @@ selftune evolve --skill <name> --skill-path <path> [options]
 | `--eval-set <path>` | Pre-built eval set JSON | Auto-generated from logs |
 | `--agent <name>` | Agent CLI to use (claude, codex, opencode) | Auto-detected |
 | `--dry-run` | Propose and validate without deploying | Off |
-| `--confidence <n>` | Minimum confidence threshold (0-1) | 0.7 |
+| `--confidence <n>` | Minimum confidence threshold (0-1) | 0.6 |
 | `--max-iterations <n>` | Maximum retry iterations | 3 |
+| `--validation-model <model>` | Model for trigger-check validation LLM calls | `haiku` |
+| `--cheap-loop` | Use cheap models for loop, expensive for final gate | Off |
+| `--gate-model <model>` | Model for final gate validation | `sonnet` (when `--cheap-loop`) |
+| `--proposal-model <model>` | Model for proposal generation LLM calls | None |
 
 ## Output Format
 
@@ -73,7 +77,73 @@ The evolution process writes multiple audit entries:
 
 ## Steps
 
-### 1. Load or Generate Eval Set
+### 0. Pre-Flight Configuration
+
+Before running the evolve command, present configuration options to the user.
+If the user says "use defaults", "just run it", or similar, skip to step 1
+with the recommended defaults marked below.
+
+Present these options (use AskUserQuestion or inline prompt):
+
+```
+selftune evolve — Pre-Flight Configuration
+
+1. Execution Mode
+   a) Dry run — preview proposal without deploying (recommended for first run)
+   b) Live — validate and deploy if improved
+
+2. Model Tier (see SKILL.md Model Tier Reference)
+   a) Fast (haiku) — cheapest, ~2s/call (recommended with cheap-loop)
+   b) Balanced (sonnet) — good quality, ~5s/call
+   c) Best (opus) — highest quality, ~10s/call
+
+3. Cost Optimization
+   a) Cheap loop — haiku for iteration, sonnet for final gate (recommended)
+   b) Single model — use one model throughout
+
+4. Confidence Threshold: [0.6] (default, higher = stricter)
+
+5. Max Iterations: [3] (default, more = longer but better results)
+
+6. Multi-Candidate Selection
+   a) Single candidate — one proposal per iteration (recommended)
+   b) Pareto mode — generate multiple candidates, pick best on frontier
+
+→ Reply with your choices (e.g., "1a, 2a, 3a, defaults for rest")
+  or "use defaults" for recommended settings.
+```
+
+After the user responds, show a confirmation summary:
+
+```
+Configuration Summary:
+  Mode:        dry-run
+  Model:       haiku (cheap-loop: sonnet gate)
+  Confidence:  0.6
+  Iterations:  3
+  Pareto:      off
+
+Proceeding...
+```
+
+Then build the CLI command with the selected flags and continue to step 1.
+
+### 1. Read Evolution Context
+
+Before running, read `~/.selftune/memory/context.md` for session context:
+- Active evolutions and their current status
+- Known issues from previous runs
+- Last update timestamp
+
+This provides continuity across context resets. If the file doesn't exist,
+proceed normally -- it will be created after the first evolution.
+
+The evolution-guard hook (`hooks/evolution-guard.ts`) blocks direct SKILL.md
+edits on monitored skills during active evolution. This prevents conflicting
+changes while the evolve process is running. The guard is automatically
+engaged when evolve starts and released when it completes.
+
+### 2. Load or Generate Eval Set
 
 If `--eval-set` is provided, use it directly. Otherwise, the command
 generates one from logs (equivalent to running `evals --skill <name>`).
@@ -81,7 +151,7 @@ generates one from logs (equivalent to running `evals --skill <name>`).
 An eval set is required for validation. Without enough telemetry data,
 evolution cannot reliably measure improvement.
 
-### 2. Extract Failure Patterns
+### 3. Extract Failure Patterns
 
 The command groups missed queries by invocation type:
 - Missed explicit: description is broken (rare, high priority)
@@ -90,7 +160,7 @@ The command groups missed queries by invocation type:
 
 See `references/invocation-taxonomy.md` for the taxonomy.
 
-### 3. Propose Description Changes
+### 4. Propose Description Changes
 
 An LLM generates a candidate description that would catch the missed
 queries. The candidate:
@@ -98,7 +168,7 @@ queries. The candidate:
 - Adds new phrases covering missed patterns
 - Maintains the description's structure and tone
 
-### 4. Validate Against Eval Set
+### 5. Validate Against Eval Set
 
 The candidate is tested against the full eval set:
 - Must improve overall pass rate
@@ -108,7 +178,7 @@ The candidate is tested against the full eval set:
 If validation fails, the command retries up to `--max-iterations` times
 with adjusted proposals.
 
-### 5. Deploy (or Preview)
+### 6. Deploy (or Preview)
 
 If `--dry-run`, the proposal is printed but not deployed. The audit log
 still records `created` and `validated` entries for review.
@@ -118,6 +188,15 @@ If deploying:
 2. The updated description is written to SKILL.md
 3. A `deployed` entry is logged to the evolution audit
 
+### 7. Update Memory
+
+After evolution completes (deploy or dry-run), the memory writer updates:
+- `~/.selftune/memory/context.md` -- records the evolution outcome and current state
+- `~/.selftune/memory/decisions.md` -- logs the decision rationale and proposal details
+
+This ensures the next evolve, watch, or rollback workflow has full context
+even after a context window reset.
+
 ### Stopping Criteria
 
 The evolution loop stops when any of these conditions is met (priority order):
@@ -130,6 +209,32 @@ The evolution loop stops when any of these conditions is met (priority order):
 | 4 | **Plateau** | Pass rate unchanged across 3 consecutive iterations |
 | 5 | **Continue** | None of the above -- keep iterating |
 
+## Cheap Loop Mode
+
+The `--cheap-loop` flag runs the entire evolution loop with cheap models (haiku)
+and only uses an expensive model (sonnet) for a final gate validation before
+deploying. This reduces cost while maintaining deployment quality.
+
+When `--cheap-loop` is set:
+- `--proposal-model` defaults to `haiku`
+- `--validation-model` defaults to `haiku`
+- `--gate-model` defaults to `sonnet`
+
+The gate validation is a new step between validation and deploy. It re-runs
+`validateProposal` using the gate model. If the gate fails, the proposal is
+not deployed.
+
+```bash
+# Cheap loop with default models
+selftune evolve --skill X --skill-path Y --cheap-loop
+
+# Cheap loop with opus gate
+selftune evolve --skill X --skill-path Y --cheap-loop --gate-model opus
+
+# Manual model control without cheap-loop
+selftune evolve --skill X --skill-path Y --proposal-model haiku --validation-model sonnet
+```
+
 ## Common Patterns
 
 **"Evolve the pptx skill"**

package/skill/Workflows/EvolveBody.md

@@ -0,0 +1,159 @@
+# selftune Evolve Body Workflow
+
+Evolve a skill's full body content or routing table, not just the description.
+Uses a teacher-student model: a stronger LLM generates proposals, a cheaper
+LLM validates them through a 3-gate pipeline.
+
+## Default Command
+
+```bash
+selftune evolve-body --skill <name> --skill-path <path> --target <target> [options]
+```
+
+## Options
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--skill <name>` | Skill name | Required |
+| `--skill-path <path>` | Path to the skill's SKILL.md | Required |
+| `--target <type>` | Evolution target: `routing_table` or `full_body` | Required |
+| `--teacher-agent <name>` | Agent CLI for proposal generation | Auto-detected |
+| `--student-agent <name>` | Agent CLI for validation | Same as teacher |
+| `--teacher-model <flag>` | Model flag for teacher (e.g. `opus`) | Agent default |
+| `--student-model <flag>` | Model flag for student (e.g. `haiku`) | Agent default |
+| `--eval-set <path>` | Pre-built eval set JSON | Auto-generated from logs |
+| `--dry-run` | Propose and validate without deploying | Off |
+| `--max-iterations <n>` | Maximum refinement iterations | 3 |
+| `--task-description <text>` | Context for the evolution goal | None |
+| `--validation-model <model>` | Model for trigger-check validation calls (overrides `--student-model` for validation) | None |
+| `--few-shot <paths>` | Comma-separated paths to example SKILL.md files | None |
+
+## Evolution Targets
+
+### `routing_table`
+
+Optimizes the `## Workflow Routing` markdown table in SKILL.md. The teacher
+LLM analyzes missed triggers and proposes new routing entries that map
+trigger keywords to the correct workflow files.
+
+### `full_body`
+
+Rewrites the entire SKILL.md body below the frontmatter. This includes
+the description, routing table, examples, and all other sections. The
+teacher generates a complete replacement, validated through 3 gates.
+
+## 3-Gate Validation Pipeline
+
+Every proposal passes through three sequential gates:
+
+| Gate | Type | What it checks | Cost |
+|------|------|---------------|------|
+| **Gate 1: Structural** | Pure code | YAML frontmatter present, `# Title` exists, `## Workflow Routing` preserved if original had one | Free |
+| **Gate 2: Trigger Accuracy** | Student LLM | YES/NO trigger check per eval entry on the extracted description | Cheap |
+| **Gate 3: Quality** | Student LLM | Body clarity and completeness score (0.0-1.0) | Cheap |
+
+If any gate fails, the teacher receives structured feedback and generates
+a refined proposal. This repeats up to `--max-iterations` times.
+
+## Steps
+
+### 0. Pre-Flight Configuration
+
+Before running evolve-body, present configuration options to the user.
+If the user says "use defaults" or similar, skip to step 1 with recommended defaults.
+
+Present these options:
+
+```
+selftune evolve-body — Pre-Flight Configuration
+
+1. Evolution Target
+   a) Routing table — optimize the workflow routing table only
+   b) Full body — rewrite entire SKILL.md body (more aggressive)
+
+2. Execution Mode
+   a) Dry run — preview proposal without deploying (recommended)
+   b) Live — validate and deploy if improved
+
+3. Teacher Model (generates proposals)
+   a) Balanced (sonnet) — good quality proposals (recommended)
+   b) Best (opus) — highest quality, slower and more expensive
+
+4. Student Model (validates proposals)
+   a) Fast (haiku) — cheap validation (recommended)
+   b) Balanced (sonnet) — higher quality validation
+
+5. Max Iterations: [3] (default)
+
+6. Few-Shot Examples: [none] (paths to example SKILL.md files for guidance)
+
+→ Reply with your choices or "use defaults" for recommended settings.
+```
+
+After the user responds, show a confirmation summary:
+
+```
+Configuration Summary:
+  Target:        routing_table
+  Mode:          dry-run
+  Teacher model: sonnet
+  Student model: haiku
+  Iterations:    3
+  Few-shot:      none
+
+Proceeding...
+```
+
+### 1. Parse Current Skill
+
+The command reads SKILL.md and splits it into sections using `parseSkillSections()`:
+- Frontmatter (YAML between `---` markers)
+- Title (first `# Heading`)
+- Description (text between title and first `## Section`)
+- Workflow Routing (the `## Workflow Routing` section, if present)
+- Remaining Body (everything else)
+
+### 2. Build Eval Set
+
+If `--eval-set` is provided, use it directly. Otherwise, generate from logs
+(same as `selftune evals --skill <name>`).
+
+### 3. Extract Failure Patterns
+
+Groups missed queries by invocation type, same as the description evolution
+pipeline. See `references/invocation-taxonomy.md`.
+
+### 4. Generate Proposal (Teacher)
+
+The teacher LLM generates a proposal based on the target:
+- **routing_table**: Optimized `## Workflow Routing` markdown table
+- **full_body**: Complete SKILL.md body replacement
+
+Few-shot examples from `--few-shot` paths provide structural guidance.
+
+### 5. Validate (3 Gates)
+
+Each gate runs in sequence. If a gate fails, the teacher receives the
+failure details and generates a refined proposal.
+
+### 6. Deploy or Preview
+
+If `--dry-run`, prints the proposal without deploying. Otherwise:
+1. Creates a timestamped backup of the current SKILL.md
+2. Applies the change: `replaceSection()` for routing, `replaceBody()` for full_body
+3. Records audit entries
+4. Updates evolution memory
+
+## Common Patterns
+
+**"Evolve the routing table for the Research skill"**
+> `selftune evolve-body --skill Research --skill-path ~/.claude/skills/Research/SKILL.md --target routing_table`
+
+**"Rewrite the entire skill body"**
+> `selftune evolve-body --skill Research --skill-path ~/.claude/skills/Research/SKILL.md --target full_body --dry-run`
+
+**"Use a stronger model for generation"**
+> `selftune evolve-body --skill pptx --skill-path /path/SKILL.md --target full_body --teacher-model opus --student-model haiku`
+
+**"Preview what would change"**
+> Always start with `--dry-run` to review the proposal before deploying.

package/skill/Workflows/ImportSkillsBench.md

@@ -0,0 +1,111 @@
+# selftune Import SkillsBench Workflow
+
+Import evaluation tasks from the SkillsBench corpus (87 real-world agent
+benchmarks) and convert them to selftune eval entries. This enriches
+your skill's eval set with externally validated test cases.
+
+## Default Command
+
+```bash
+selftune import-skillsbench --dir <path> --skill <name> --output <path> [options]
+```
+
+## Options
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--dir <path>` | Path to SkillsBench tasks directory | Required |
+| `--skill <name>` | Target skill to match tasks against | Required |
+| `--output <path>` | Output eval set JSON file | Required |
+| `--match-strategy <type>` | Matching strategy: `exact` or `fuzzy` | `exact` |
+
+## Match Strategies
+
+### `exact`
+
+Matches tasks where `expected_skill` in `task.toml` exactly matches the
+target skill name. Precise but may miss relevant tasks.
+
+### `fuzzy`
+
+Uses keyword overlap between the task's category/tags and the skill name.
+Casts a wider net but may include marginally relevant tasks. Review the
+output and remove false matches.
+
+## SkillsBench Directory Structure
+
+The importer expects this layout:
+
+```
+tasks/
+├── task-001/
+│   ├── instruction.md     # Task description (used as query)
+│   └── task.toml          # Metadata (difficulty, category, tags, expected_skill)
+├── task-002/
+│   ├── instruction.md
+│   └── task.toml
+└── ...
+```
+
+### `task.toml` Format
+
+```toml
+difficulty = "medium"
+category = "research"
+tags = ["web-search", "analysis", "summarization"]
+expected_skill = "Research"
+expected_tools = ["WebSearch", "Read"]
+```
+
+All fields are optional. Tasks without `task.toml` use default values.
+
+## Output Format
+
+Standard selftune eval entries:
+
+```json
+[
+  {
+    "id": 1,
+    "query": "Find and summarize the latest papers on transformer architectures",
+    "expected": true,
+    "invocation_type": "implicit",
+    "skill_name": "Research",
+    "source_session": null,
+    "source": "skillsbench"
+  }
+]
+```
+
+## Steps
+
+### 1. Obtain SkillsBench Corpus
+
+Clone or download the SkillsBench repository containing the task directory.
+
+### 2. Import Tasks
+
+```bash
+selftune import-skillsbench --dir /path/to/skillsbench/tasks --skill Research --output evals-bench.json
+```
+
+### 3. Review Output
+
+Inspect the generated eval entries. Remove any that don't match your skill's
+intended scope. Adjust match strategy if needed.
+
+### 4. Merge with Existing Evals
+
+Combine imported entries with your existing eval set for a richer validation
+corpus. Use the merged set with `selftune evolve --eval-set merged-evals.json`.
+
+## Common Patterns
+
+**"Import SkillsBench tasks for Research"**
+> `selftune import-skillsbench --dir /path/tasks --skill Research --output bench-evals.json`
+
+**"Use fuzzy matching for broader coverage"**
+> `selftune import-skillsbench --dir /path/tasks --skill pptx --output bench-evals.json --match-strategy fuzzy`
+
+**"Enrich my eval set with external benchmarks"**
+> Import with `import-skillsbench`, then pass the output to `evolve --eval-set`.