selftune 0.1.0

package/skill/SKILL.md

@@ -0,0 +1,120 @@
+---
+name: selftune
+description: >
+  Skill observability and continuous improvement. Use when the user wants to:
+  grade a session, generate evals, check undertriggering, evolve a skill
+  description, rollback an evolution, monitor post-deploy performance, run
+  health checks, or ingest sessions from Codex/OpenCode.
+---
+
+# selftune
+
+Observe real agent sessions, detect missed triggers, grade execution quality,
+and evolve skill descriptions toward the language real users actually use.
+
+## Bootstrap
+
+If `~/.selftune/config.json` does not exist, read `Workflows/Initialize.md`
+first. Do not proceed with other commands until initialization is complete.
+
+## Command Execution Policy
+
+Build every CLI invocation from the config:
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH <command> [options]
+```
+
+Fallback (if config is missing or stale):
+```bash
+bun run <repo-path>/cli/selftune/index.ts <command> [options]
+```
+
+All commands output deterministic JSON. Always parse JSON output -- never
+text-match against output strings.
+
+## Quick Reference
+
+```bash
+selftune grade    --skill <name> [--expectations "..."] [--use-agent]
+selftune evals    --skill <name> [--list-skills] [--stats] [--max N]
+selftune evolve   --skill <name> --skill-path <path> [--dry-run]
+selftune rollback --skill <name> --skill-path <path> [--proposal-id <id>]
+selftune watch    --skill <name> --skill-path <path> [--auto-rollback]
+selftune doctor
+selftune ingest-codex
+selftune ingest-opencode
+selftune wrap-codex -- <codex args>
+```
+
+## Workflow Routing
+
+| Trigger keywords | Workflow | File |
+|------------------|----------|------|
+| grade, score, evaluate, assess session | Grade | Workflows/Grade.md |
+| evals, eval set, undertriggering, skill stats | Evals | Workflows/Evals.md |
+| evolve, improve, triggers, catch more queries | Evolve | Workflows/Evolve.md |
+| rollback, undo, restore, revert evolution | Rollback | Workflows/Rollback.md |
+| watch, monitor, regression, post-deploy, performing | Watch | Workflows/Watch.md |
+| doctor, health, hooks, broken, diagnose | Doctor | Workflows/Doctor.md |
+| ingest, import, codex logs, opencode, wrap codex | Ingest | Workflows/Ingest.md |
+| init, setup, bootstrap, first time | Initialize | Workflows/Initialize.md |
+
+## The Feedback Loop
+
+```
+Observe --> Detect --> Diagnose --> Propose --> Validate --> Deploy --> Watch
+   |                                                                    |
+   +--------------------------------------------------------------------+
+```
+
+1. **Observe** -- Hooks capture every session (queries, triggers, metrics)
+2. **Detect** -- `evals` finds missed triggers across invocation types
+3. **Diagnose** -- `grade` evaluates session quality with evidence
+4. **Propose** -- `evolve` generates description improvements
+5. **Validate** -- Evolution is tested against the eval set
+6. **Deploy** -- Updated description replaces the original (with backup)
+7. **Watch** -- `watch` monitors for regressions post-deploy
+
+## Resource Index
+
+| Resource | Purpose |
+|----------|---------|
+| `SKILL.md` | This file -- routing, triggers, quick reference |
+| `references/logs.md` | Log file formats (telemetry, usage, queries, audit) |
+| `references/grading-methodology.md` | 3-tier grading model, evidence standards, grading.json schema |
+| `references/invocation-taxonomy.md` | 4 invocation types, coverage analysis, evolution connection |
+| `settings_snippet.json` | Claude Code hook configuration template |
+| `Workflows/Initialize.md` | First-time setup and config bootstrap |
+| `Workflows/Grade.md` | Grade a session with expectations and evidence |
+| `Workflows/Evals.md` | Generate eval sets, list skills, show stats |
+| `Workflows/Evolve.md` | Evolve a skill description from failure patterns |
+| `Workflows/Rollback.md` | Undo an evolution, restore previous description |
+| `Workflows/Watch.md` | Post-deploy regression monitoring |
+| `Workflows/Doctor.md` | Health checks on logs, hooks, schema |
+| `Workflows/Ingest.md` | Import sessions from Codex and OpenCode |
+
+## Examples
+
+- "Grade my last pptx session"
+- "What skills are undertriggering?"
+- "Generate evals for the pptx skill"
+- "Evolve the pptx skill to catch more queries"
+- "Rollback the last evolution"
+- "Is the skill performing well after the change?"
+- "Check selftune health"
+- "Ingest my codex logs"
+- "Show me skill stats"
+
+## Negative Examples
+
+These should NOT trigger selftune:
+
+- "Fix this React hydration bug"
+- "Create a PowerPoint about Q3 results" (this is pptx, not selftune)
+- "Run my unit tests"
+- "What does this error mean?"
+
+Route to other skills or general workflows unless the user explicitly
+asks about grading, evals, evolution, monitoring, or skill observability.

package/skill/Workflows/Doctor.md

@@ -0,0 +1,145 @@
+# selftune Doctor Workflow
+
+Run health checks on selftune logs, hooks, and schema integrity.
+Reports pass/fail status for each check with actionable guidance.
+
+## Default Command
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH doctor
+```
+
+Fallback:
+```bash
+bun run <repo-path>/cli/selftune/index.ts doctor
+```
+
+## Options
+
+None. Doctor runs all checks unconditionally.
+
+## Output Format
+
+```json
+{
+  "healthy": true,
+  "checks": [
+    {
+      "name": "session_telemetry_log exists",
+      "status": "pass",
+      "detail": "Found 142 entries"
+    },
+    {
+      "name": "skill_usage_log parseable",
+      "status": "pass",
+      "detail": "All 89 entries valid JSON"
+    },
+    {
+      "name": "hooks installed",
+      "status": "fail",
+      "detail": "PostToolUse hook not found in ~/.claude/settings.json"
+    }
+  ],
+  "summary": {
+    "passed": 5,
+    "failed": 1,
+    "total": 6
+  }
+}
+```
+
+The process exits with code 0 if `healthy: true`, code 1 otherwise.
+
+## Parsing Instructions
+
+### Check Overall Health
+
+```bash
+# Parse: .healthy (boolean)
+# Quick check: exit code 0 = healthy, 1 = unhealthy
+```
+
+### Find Failed Checks
+
+```bash
+# Parse: .checks[] | select(.status == "fail") | { name, detail }
+```
+
+### Get Summary Counts
+
+```bash
+# Parse: .summary.passed, .summary.failed, .summary.total
+```
+
+## Health Checks
+
+Doctor validates these areas:
+
+### Log File Checks
+
+| Check | What it validates |
+|-------|-------------------|
+| Log files exist | `session_telemetry_log.jsonl`, `skill_usage_log.jsonl`, `all_queries_log.jsonl` exist in `~/.claude/` |
+| Logs are parseable | Every line in each log file is valid JSON |
+| Schema conformance | Required fields present per log type (see `references/logs.md`) |
+
+### Hook Checks
+
+| Check | What it validates |
+|-------|-------------------|
+| Hooks installed | `UserPromptSubmit`, `PostToolUse`, and `Stop` hooks are configured in `~/.claude/settings.json` |
+| Hook scripts exist | The script files referenced by hooks exist on disk |
+
+### Evolution Audit Checks
+
+| Check | What it validates |
+|-------|-------------------|
+| Audit log integrity | `evolution_audit_log.jsonl` entries have required fields (`timestamp`, `proposal_id`, `action`) |
+| Valid action values | All entries use known action types: `created`, `validated`, `deployed`, `rolled_back` |
+
+## Steps
+
+### 1. Run Doctor
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH doctor
+```
+
+### 2. Check Results
+
+Parse the JSON output. If `healthy: true`, selftune is fully operational.
+
+### 3. Fix Any Issues
+
+For each failed check, take the appropriate action:
+
+| Failed check | Fix |
+|-------------|-----|
+| Log files missing | Run a session to generate initial log entries. Check hook installation. |
+| Logs not parseable | Inspect the corrupted log file. Remove or fix invalid lines. |
+| Hooks not installed | Merge `skill/settings_snippet.json` into `~/.claude/settings.json`. Update paths. |
+| Hook scripts missing | Verify the selftune repo path. Re-run `init` if the repo was moved. |
+| Audit log invalid | Remove corrupted entries. Future operations will append clean entries. |
+
+### 4. Re-run Doctor
+
+After fixes, run doctor again to verify all checks pass.
+
+## Common Patterns
+
+**"Something seems broken"**
+> Run doctor first. Report any failing checks with their detail messages.
+
+**"Are my hooks working?"**
+> Doctor checks hook installation. If hooks pass but no data appears,
+> verify the hook script paths point to actual files.
+
+**"No telemetry available"**
+> Doctor will report missing log files. Install hooks using the
+> `settings_snippet.json` in the skill directory, then run a session.
+
+**"Check selftune health"**
+> Run doctor and report the summary. A clean bill of health means
+> all checks pass and selftune is ready to grade/evolve/watch.

package/skill/Workflows/Evals.md

@@ -0,0 +1,193 @@
+# selftune Evals Workflow
+
+Generate eval sets from hook logs. Detects false negatives (queries that
+should have triggered a skill but did not) and annotates each entry with
+its invocation type.
+
+## Default Command
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH evals --skill <name> [options]
+```
+
+Fallback:
+```bash
+bun run <repo-path>/cli/selftune/index.ts evals --skill <name> [options]
+```
+
+## Options
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--skill <name>` | Skill to generate evals for | Required (unless `--list-skills`) |
+| `--list-skills` | List all logged skills with query counts | Off |
+| `--stats` | Show aggregate telemetry stats for the skill | Off |
+| `--max <n>` | Maximum eval entries to generate | 50 |
+| `--seed <n>` | Random seed for negative sampling | Random |
+| `--out <path>` | Output file path | `evals-<skill>.json` |
+
+## Output Format
+
+### Eval Set (default)
+
+```json
+[
+  {
+    "id": 1,
+    "query": "Make me a slide deck for the Q3 board meeting",
+    "expected": true,
+    "invocation_type": "contextual",
+    "skill_name": "pptx",
+    "source_session": "abc123"
+  },
+  {
+    "id": 2,
+    "query": "What format should I use for a presentation?",
+    "expected": false,
+    "invocation_type": "negative",
+    "skill_name": "pptx",
+    "source_session": null
+  }
+]
+```
+
+### List Skills
+
+```json
+{
+  "skills": [
+    { "name": "pptx", "query_count": 42, "session_count": 15 },
+    { "name": "selftune", "query_count": 28, "session_count": 10 }
+  ]
+}
+```
+
+### Stats
+
+```json
+{
+  "skill_name": "pptx",
+  "sessions": 15,
+  "avg_turns": 4.2,
+  "tool_call_breakdown": { "Read": 30, "Write": 15, "Bash": 45 },
+  "error_rate": 0.13,
+  "bash_patterns": ["pip install python-pptx", "python3 /tmp/create_pptx.py"]
+}
+```
+
+## Parsing Instructions
+
+### Count by Invocation Type
+
+```bash
+# Parse: group_by(.invocation_type) | map({ type: .[0].invocation_type, count: length })
+```
+
+### Find Missed Queries (False Negatives)
+
+```bash
+# Parse: .[] | select(.expected == true and .invocation_type != "explicit")
+# These are queries that should trigger but might be missed
+```
+
+### Get Negative Examples
+
+```bash
+# Parse: .[] | select(.expected == false)
+```
+
+## Sub-Workflows
+
+### List Skills
+
+Discover which skills have telemetry data and how many queries each has.
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH evals --list-skills
+```
+
+Use this first to identify which skills have enough data for eval generation.
+
+### Generate Evals
+
+Cross-reference `skill_usage_log.jsonl` (positive triggers) against
+`all_queries_log.jsonl` (all queries, including non-triggers) to produce
+an eval set annotated with invocation types.
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH evals --skill pptx --max 50 --out evals-pptx.json
+```
+
+The command:
+1. Reads positive triggers from `skill_usage_log.jsonl`
+2. Reads all queries from `all_queries_log.jsonl`
+3. Identifies queries that should have triggered but did not
+4. Samples negative examples (unrelated queries)
+5. Annotates each entry with invocation type
+6. Writes the eval set to the output file
+
+### Show Stats
+
+View aggregate telemetry for a skill: average turns, tool call breakdown,
+error rates, and common bash command patterns.
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH evals --skill pptx --stats
+```
+
+## Steps
+
+### 1. List Available Skills
+
+Run `--list-skills` to see what skills have telemetry data. If the target
+skill has zero or very few queries, more sessions are needed before
+eval generation is useful.
+
+### 2. Generate the Eval Set
+
+Run with `--skill <name>`. Review the output file for:
+- Balance between positive and negative entries
+- Coverage of all three positive invocation types (explicit, implicit, contextual)
+- Reasonable negative examples (keyword overlap but wrong intent)
+
+### 3. Review Invocation Type Distribution
+
+A healthy eval set has:
+- Some explicit queries (easy baseline)
+- Many implicit queries (natural usage)
+- Several contextual queries (real-world usage)
+- Enough negatives to prevent false positives
+
+See `references/invocation-taxonomy.md` for what each type means and
+what healthy distribution looks like.
+
+### 4. Identify Coverage Gaps
+
+If the eval set is missing implicit or contextual queries, the skill may be
+undertriggering. This is the signal for `evolve` to improve the description.
+
+### 5. Optional: Check Stats
+
+Use `--stats` to understand session patterns before evolution. High error
+rates or unusual tool call distributions may indicate process issues
+beyond trigger coverage.
+
+## Common Patterns
+
+**"What skills are undertriggering?"**
+> Run `--list-skills`, then for each skill with significant query counts,
+> generate evals and check for missed implicit/contextual queries.
+
+**"Generate evals for pptx"**
+> Run `evals --skill pptx`. Review the invocation type distribution.
+> Feed the output to `evolve` if coverage gaps exist.
+
+**"Show me skill stats"**
+> Run `evals --skill <name> --stats` for aggregate telemetry.
+
+**"I want reproducible evals"**
+> Use `--seed <n>` to fix the random sampling of negative examples.

package/skill/Workflows/Evolve.md

@@ -0,0 +1,159 @@
+# selftune Evolve Workflow
+
+Improve a skill's description based on real usage signal. Analyzes failure
+patterns from eval sets and proposes description changes that catch more
+natural-language queries without breaking existing triggers.
+
+## Default Command
+
+```bash
+CLI_PATH=$(cat ~/.selftune/config.json | jq -r .cli_path)
+bun run $CLI_PATH evolve --skill <name> --skill-path <path> [options]
+```
+
+Fallback:
+```bash
+bun run <repo-path>/cli/selftune/index.ts evolve --skill <name> --skill-path <path> [options]
+```
+
+## Options
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--skill <name>` | Skill name | Required |
+| `--skill-path <path>` | Path to the skill's SKILL.md | Required |
+| `--eval-set <path>` | Pre-built eval set JSON | Auto-generated from logs |
+| `--mode api\|agent` | LLM mode for proposal generation | `agent` |
+| `--agent <name>` | Agent CLI binary to use | Auto-detected |
+| `--dry-run` | Propose and validate without deploying | Off |
+| `--confidence <n>` | Minimum confidence threshold (0-1) | 0.7 |
+| `--max-iterations <n>` | Maximum retry iterations | 3 |
+
+## Output Format
+
+Each evolution action is logged to `~/.claude/evolution_audit_log.jsonl`.
+See `references/logs.md` for the audit log schema.
+
+### Proposal Output (dry-run or pre-deploy)
+
+```json
+{
+  "proposal_id": "evolve-pptx-1709125200000",
+  "skill_name": "pptx",
+  "iteration": 1,
+  "original_pass_rate": 0.70,
+  "proposed_pass_rate": 0.92,
+  "regression_count": 0,
+  "confidence": 0.85,
+  "status": "validated",
+  "changes_summary": "Added implicit triggers: 'slide deck', 'presentation', 'board meeting slides'"
+}
+```
+
+### Audit Log Entries
+
+The evolution process writes multiple audit entries:
+
+| Action | When | Key details |
+|--------|------|-------------|
+| `created` | Proposal generated | `details` contains `original_description:` prefix |
+| `validated` | Proposal tested against eval set | `eval_snapshot` with before/after pass rates |
+| `deployed` | Updated SKILL.md written to disk | `eval_snapshot` with final rates |
+
+## Parsing Instructions
+
+### Track Evolution Progress
+
+```bash
+# Read audit log for the proposal
+# Parse: entries where proposal_id matches
+# Check: action sequence should be created -> validated -> deployed
+```
+
+### Check for Regression
+
+```bash
+# Parse: .eval_snapshot in validated entry
+# Verify: proposed pass_rate > original pass_rate
+# Verify: regression_count < 5% of total evals
+```
+
+## Steps
+
+### 1. Load or Generate Eval Set
+
+If `--eval-set` is provided, use it directly. Otherwise, the command
+generates one from logs (equivalent to running `evals --skill <name>`).
+
+An eval set is required for validation. Without enough telemetry data,
+evolution cannot reliably measure improvement.
+
+### 2. Extract Failure Patterns
+
+The command groups missed queries by invocation type:
+- Missed explicit: description is broken (rare, high priority)
+- Missed implicit: description is too narrow (common, evolve target)
+- Missed contextual: description lacks domain vocabulary (evolve target)
+
+See `references/invocation-taxonomy.md` for the taxonomy.
+
+### 3. Propose Description Changes
+
+An LLM generates a candidate description that would catch the missed
+queries. The candidate:
+- Preserves existing trigger phrases that work
+- Adds new phrases covering missed patterns
+- Maintains the description's structure and tone
+
+### 4. Validate Against Eval Set
+
+The candidate is tested against the full eval set:
+- Must improve overall pass rate
+- Must not regress more than 5% on previously-passing entries
+- Must exceed the `--confidence` threshold
+
+If validation fails, the command retries up to `--max-iterations` times
+with adjusted proposals.
+
+### 5. Deploy (or Preview)
+
+If `--dry-run`, the proposal is printed but not deployed. The audit log
+still records `created` and `validated` entries for review.
+
+If deploying:
+1. The current SKILL.md is backed up to `SKILL.md.bak`
+2. The updated description is written to SKILL.md
+3. A `deployed` entry is logged to the evolution audit
+
+### Stopping Criteria
+
+The evolution loop stops when any of these conditions is met (priority order):
+
+| # | Condition | Meaning |
+|---|-----------|---------|
+| 1 | **Converged** | Pass rate >= 0.95 |
+| 2 | **Max iterations** | Reached `--max-iterations` limit |
+| 3 | **Low confidence** | Proposal confidence below `--confidence` threshold |
+| 4 | **Plateau** | Pass rate unchanged across 3 consecutive iterations |
+| 5 | **Continue** | None of the above -- keep iterating |
+
+## Common Patterns
+
+**"Evolve the pptx skill"**
+> Need `--skill pptx` and `--skill-path /path/to/pptx/SKILL.md`.
+> If the user hasn't specified the path, search for the SKILL.md file
+> in the workspace or ask.
+
+**"Just show me what would change"**
+> Use `--dry-run` to preview proposals without deploying.
+
+**"The evolution didn't help enough"**
+> Check the eval set quality. Missing contextual examples will limit
+> what evolution can learn. Generate a richer eval set first.
+
+**"Evolution keeps failing validation"**
+> Lower `--confidence` slightly or increase `--max-iterations`.
+> Also check if the eval set has contradictory expectations.
+
+**"I want to use the API directly"**
+> Pass `--mode api`. Requires `ANTHROPIC_API_KEY` in the environment.