selftune 0.1.2 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/diagnosis-analyst.md +146 -0
- package/.claude/agents/evolution-reviewer.md +167 -0
- package/.claude/agents/integration-guide.md +200 -0
- package/.claude/agents/pattern-analyst.md +147 -0
- package/CHANGELOG.md +38 -1
- package/README.md +96 -256
- package/assets/BeforeAfter.gif +0 -0
- package/assets/FeedbackLoop.gif +0 -0
- package/assets/logo.svg +9 -0
- package/assets/skill-health-badge.svg +20 -0
- package/cli/selftune/activation-rules.ts +171 -0
- package/cli/selftune/badge/badge-data.ts +108 -0
- package/cli/selftune/badge/badge-svg.ts +212 -0
- package/cli/selftune/badge/badge.ts +103 -0
- package/cli/selftune/constants.ts +75 -1
- package/cli/selftune/contribute/bundle.ts +314 -0
- package/cli/selftune/contribute/contribute.ts +214 -0
- package/cli/selftune/contribute/sanitize.ts +162 -0
- package/cli/selftune/cron/setup.ts +266 -0
- package/cli/selftune/dashboard-server.ts +582 -0
- package/cli/selftune/dashboard.ts +31 -12
- package/cli/selftune/eval/baseline.ts +247 -0
- package/cli/selftune/eval/composability.ts +117 -0
- package/cli/selftune/eval/generate-unit-tests.ts +143 -0
- package/cli/selftune/eval/hooks-to-evals.ts +68 -2
- package/cli/selftune/eval/import-skillsbench.ts +221 -0
- package/cli/selftune/eval/synthetic-evals.ts +172 -0
- package/cli/selftune/eval/unit-test-cli.ts +152 -0
- package/cli/selftune/eval/unit-test.ts +196 -0
- package/cli/selftune/evolution/deploy-proposal.ts +142 -1
- package/cli/selftune/evolution/evolve-body.ts +492 -0
- package/cli/selftune/evolution/evolve.ts +479 -104
- package/cli/selftune/evolution/extract-patterns.ts +32 -1
- package/cli/selftune/evolution/pareto.ts +314 -0
- package/cli/selftune/evolution/propose-body.ts +171 -0
- package/cli/selftune/evolution/propose-description.ts +100 -2
- package/cli/selftune/evolution/propose-routing.ts +166 -0
- package/cli/selftune/evolution/refine-body.ts +141 -0
- package/cli/selftune/evolution/rollback.ts +20 -3
- package/cli/selftune/evolution/validate-body.ts +254 -0
- package/cli/selftune/evolution/validate-proposal.ts +257 -35
- package/cli/selftune/evolution/validate-routing.ts +177 -0
- package/cli/selftune/grading/grade-session.ts +145 -19
- package/cli/selftune/grading/pre-gates.ts +104 -0
- package/cli/selftune/hooks/auto-activate.ts +185 -0
- package/cli/selftune/hooks/evolution-guard.ts +165 -0
- package/cli/selftune/hooks/skill-change-guard.ts +112 -0
- package/cli/selftune/index.ts +88 -0
- package/cli/selftune/ingestors/claude-replay.ts +351 -0
- package/cli/selftune/ingestors/codex-rollout.ts +1 -1
- package/cli/selftune/ingestors/openclaw-ingest.ts +440 -0
- package/cli/selftune/ingestors/opencode-ingest.ts +2 -2
- package/cli/selftune/init.ts +168 -5
- package/cli/selftune/last.ts +2 -2
- package/cli/selftune/memory/writer.ts +447 -0
- package/cli/selftune/monitoring/watch.ts +25 -2
- package/cli/selftune/status.ts +18 -15
- package/cli/selftune/types.ts +377 -5
- package/cli/selftune/utils/frontmatter.ts +217 -0
- package/cli/selftune/utils/llm-call.ts +29 -3
- package/cli/selftune/utils/transcript.ts +35 -0
- package/cli/selftune/utils/trigger-check.ts +89 -0
- package/cli/selftune/utils/tui.ts +156 -0
- package/dashboard/index.html +585 -19
- package/package.json +17 -6
- package/skill/SKILL.md +127 -10
- package/skill/Workflows/AutoActivation.md +144 -0
- package/skill/Workflows/Badge.md +118 -0
- package/skill/Workflows/Baseline.md +121 -0
- package/skill/Workflows/Composability.md +100 -0
- package/skill/Workflows/Contribute.md +91 -0
- package/skill/Workflows/Cron.md +155 -0
- package/skill/Workflows/Dashboard.md +203 -0
- package/skill/Workflows/Doctor.md +37 -1
- package/skill/Workflows/Evals.md +73 -5
- package/skill/Workflows/EvolutionMemory.md +152 -0
- package/skill/Workflows/Evolve.md +111 -6
- package/skill/Workflows/EvolveBody.md +159 -0
- package/skill/Workflows/ImportSkillsBench.md +111 -0
- package/skill/Workflows/Ingest.md +129 -15
- package/skill/Workflows/Initialize.md +58 -3
- package/skill/Workflows/Replay.md +70 -0
- package/skill/Workflows/Rollback.md +20 -1
- package/skill/Workflows/UnitTest.md +138 -0
- package/skill/Workflows/Watch.md +22 -0
- package/skill/settings_snippet.json +23 -0
- package/templates/activation-rules-default.json +27 -0
- package/templates/multi-skill-settings.json +64 -0
- package/templates/single-skill-settings.json +58 -0
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# selftune Composability Workflow
|
|
2
|
+
|
|
3
|
+
Analyze how skills interact when triggered together in the same session.
|
|
4
|
+
Detects conflict candidates — skill pairs that produce more errors when
|
|
5
|
+
co-occurring than when used alone.
|
|
6
|
+
|
|
7
|
+
## Default Command
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
selftune composability --skill <name> [options]
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## Options
|
|
14
|
+
|
|
15
|
+
| Flag | Description | Default |
|
|
16
|
+
|------|-------------|---------|
|
|
17
|
+
| `--skill <name>` | Skill to analyze | Required |
|
|
18
|
+
| `--window <n>` | Only analyze sessions from last N days | All sessions |
|
|
19
|
+
| `--telemetry-log <path>` | Path to telemetry log | `~/.claude/session_telemetry_log.jsonl` |
|
|
20
|
+
|
|
21
|
+
## Output Format
|
|
22
|
+
|
|
23
|
+
```json
|
|
24
|
+
{
|
|
25
|
+
"skill_name": "Research",
|
|
26
|
+
"analyzed_sessions": 150,
|
|
27
|
+
"co_occurring_skills": [
|
|
28
|
+
{
|
|
29
|
+
"skill_a": "Research",
|
|
30
|
+
"skill_b": "Browser",
|
|
31
|
+
"co_occurrence_count": 42,
|
|
32
|
+
"conflict_score": 0.12,
|
|
33
|
+
"avg_errors_together": 1.5,
|
|
34
|
+
"avg_errors_alone": 1.3
|
|
35
|
+
}
|
|
36
|
+
],
|
|
37
|
+
"conflict_candidates": [
|
|
38
|
+
{
|
|
39
|
+
"skill_a": "Research",
|
|
40
|
+
"skill_b": "Content",
|
|
41
|
+
"co_occurrence_count": 15,
|
|
42
|
+
"conflict_score": 0.45,
|
|
43
|
+
"avg_errors_together": 3.2,
|
|
44
|
+
"avg_errors_alone": 1.1
|
|
45
|
+
}
|
|
46
|
+
],
|
|
47
|
+
"generated_at": "2026-03-04T12:00:00.000Z"
|
|
48
|
+
}
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
## How It Works
|
|
52
|
+
|
|
53
|
+
The analyzer is a pure function that computes conflict scores from telemetry:
|
|
54
|
+
|
|
55
|
+
1. Filters sessions where `skills_triggered` includes the target skill
|
|
56
|
+
2. For each co-occurring skill, computes:
|
|
57
|
+
- Average errors when both skills are triggered together
|
|
58
|
+
- Average errors when each skill is triggered alone
|
|
59
|
+
- `conflict_score = clamp((errors_together - errors_alone) / (errors_alone + 1), 0, 1)`
|
|
60
|
+
3. Pairs with `conflict_score > 0.3` are flagged as conflict candidates
|
|
61
|
+
4. Results sorted by co-occurrence count (most common first)
|
|
62
|
+
|
|
63
|
+
## Steps
|
|
64
|
+
|
|
65
|
+
### 1. Run Analysis
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
selftune composability --skill Research
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
### 2. Interpret Results
|
|
72
|
+
|
|
73
|
+
| Conflict Score | Interpretation |
|
|
74
|
+
|---------------|---------------|
|
|
75
|
+
| 0.0–0.1 | No conflict — skills work well together |
|
|
76
|
+
| 0.1–0.3 | Minor friction — monitor but no action needed |
|
|
77
|
+
| 0.3–0.6 | Moderate conflict — investigate trigger overlap |
|
|
78
|
+
| 0.6–1.0 | Severe conflict — skills likely interfere with each other |
|
|
79
|
+
|
|
80
|
+
### 3. Address Conflicts
|
|
81
|
+
|
|
82
|
+
For conflict candidates:
|
|
83
|
+
- Check for trigger keyword overlap between the skills
|
|
84
|
+
- Check if one skill's workflow interferes with the other's
|
|
85
|
+
- Consider evolving descriptions to reduce false triggers
|
|
86
|
+
- Use the `pattern-analyst` agent for deeper cross-skill analysis
|
|
87
|
+
|
|
88
|
+
## Common Patterns
|
|
89
|
+
|
|
90
|
+
**"Are there conflicts between my skills?"**
|
|
91
|
+
> `selftune composability --skill Research`
|
|
92
|
+
|
|
93
|
+
**"Check composability for recent sessions only"**
|
|
94
|
+
> `selftune composability --skill pptx --window 7`
|
|
95
|
+
|
|
96
|
+
**"Which skills conflict with Research?"**
|
|
97
|
+
> Run composability and check the `conflict_candidates` array.
|
|
98
|
+
|
|
99
|
+
**"Why are sessions with multiple skills failing?"**
|
|
100
|
+
> Run composability for each skill involved, look for high conflict scores.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# selftune Contribute Workflow
|
|
2
|
+
|
|
3
|
+
Export anonymized skill observability data as a JSON bundle for community
|
|
4
|
+
contribution. Helps improve selftune's skill routing without exposing
|
|
5
|
+
private data.
|
|
6
|
+
|
|
7
|
+
## When to Use
|
|
8
|
+
|
|
9
|
+
- Want to help improve selftune's skill routing
|
|
10
|
+
- Sharing anonymized usage patterns with the community
|
|
11
|
+
- Contributing eval data for skill evolution
|
|
12
|
+
|
|
13
|
+
## Default Command
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
selftune contribute --skill selftune
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## Options
|
|
20
|
+
|
|
21
|
+
| Flag | Description |
|
|
22
|
+
|------|-------------|
|
|
23
|
+
| `--skill <name>` | Skill to contribute data for (default: "selftune") |
|
|
24
|
+
| `--output <path>` | Output file path (default: auto-generated in ~/.selftune/contributions/) |
|
|
25
|
+
| `--preview` | Show what would be shared without writing |
|
|
26
|
+
| `--sanitize <level>` | `conservative` (default) or `aggressive` |
|
|
27
|
+
| `--since <date>` | Only include data from this date onward |
|
|
28
|
+
| `--submit` | Auto-create GitHub Issue via `gh` CLI |
|
|
29
|
+
|
|
30
|
+
## Sanitization Levels
|
|
31
|
+
|
|
32
|
+
### Conservative (default)
|
|
33
|
+
|
|
34
|
+
| Pattern | Replacement |
|
|
35
|
+
|---------|-------------|
|
|
36
|
+
| File paths | `[PATH]` |
|
|
37
|
+
| Email addresses | `[EMAIL]` |
|
|
38
|
+
| API keys, tokens, JWTs | `[SECRET]` |
|
|
39
|
+
| IP addresses | `[IP]` |
|
|
40
|
+
| Project name from cwd | `[PROJECT]` |
|
|
41
|
+
| Session IDs | `[SESSION]` |
|
|
42
|
+
|
|
43
|
+
### Aggressive
|
|
44
|
+
|
|
45
|
+
Extends conservative with:
|
|
46
|
+
|
|
47
|
+
| Pattern | Replacement |
|
|
48
|
+
|---------|-------------|
|
|
49
|
+
| camelCase/PascalCase identifiers > 8 chars | `[IDENTIFIER]` |
|
|
50
|
+
| Quoted strings | `[STRING]` |
|
|
51
|
+
| Import/require module paths | `[MODULE]` |
|
|
52
|
+
| Queries > 200 chars | Truncated |
|
|
53
|
+
|
|
54
|
+
## Bundle Contents
|
|
55
|
+
|
|
56
|
+
The contribution bundle includes:
|
|
57
|
+
- **Positive queries** -- queries that triggered the skill (sanitized)
|
|
58
|
+
- **Eval entries** -- trigger eval set for the skill
|
|
59
|
+
- **Grading summary** -- aggregate pass rates (no raw transcripts)
|
|
60
|
+
- **Evolution summary** -- proposal counts and outcomes
|
|
61
|
+
- **Session metrics** -- average turns, tool usage, error rates
|
|
62
|
+
|
|
63
|
+
No raw transcripts, file contents, or identifiable information is included.
|
|
64
|
+
|
|
65
|
+
## Submission
|
|
66
|
+
|
|
67
|
+
- Default: writes JSON file to `~/.selftune/contributions/`
|
|
68
|
+
- `--submit`: creates a GitHub Issue with the bundle
|
|
69
|
+
- Small bundles (< 50KB): inlined in issue body
|
|
70
|
+
- Large bundles (>= 50KB): uploaded as a gist
|
|
71
|
+
|
|
72
|
+
## Steps
|
|
73
|
+
|
|
74
|
+
1. Run `selftune contribute --preview --skill selftune` to see what would be shared
|
|
75
|
+
2. Review the sanitized output
|
|
76
|
+
3. Run `selftune contribute --skill selftune` to write the bundle
|
|
77
|
+
4. Optionally: `selftune contribute --skill selftune --submit` to create GitHub issue
|
|
78
|
+
|
|
79
|
+
## Common Patterns
|
|
80
|
+
|
|
81
|
+
**"Preview what I'd share"**
|
|
82
|
+
> `selftune contribute --preview`
|
|
83
|
+
|
|
84
|
+
**"Use aggressive sanitization"**
|
|
85
|
+
> `selftune contribute --sanitize aggressive`
|
|
86
|
+
|
|
87
|
+
**"Submit directly to GitHub"**
|
|
88
|
+
> `selftune contribute --submit`
|
|
89
|
+
|
|
90
|
+
**"Only contribute recent data"**
|
|
91
|
+
> `selftune contribute --since 2026-02-01`
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
# selftune Cron Workflow
|
|
2
|
+
|
|
3
|
+
Manage OpenClaw cron jobs that run the selftune pipeline on a schedule.
|
|
4
|
+
Enables fully autonomous skill evolution — skills improve while you sleep.
|
|
5
|
+
|
|
6
|
+
## When to Use
|
|
7
|
+
|
|
8
|
+
- Setting up selftune automation for the first time on OpenClaw
|
|
9
|
+
- Checking which cron jobs are registered
|
|
10
|
+
- Removing selftune cron jobs (cleanup or reconfiguration)
|
|
11
|
+
- Enabling the autonomous observe-grade-evolve-deploy loop
|
|
12
|
+
|
|
13
|
+
## Prerequisites
|
|
14
|
+
|
|
15
|
+
OpenClaw must be installed and in your PATH. The setup command will check
|
|
16
|
+
for this and exit with instructions if OpenClaw is not found.
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
which openclaw # Must resolve
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Default Command
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
selftune cron setup
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Subcommands
|
|
29
|
+
|
|
30
|
+
### `selftune cron setup`
|
|
31
|
+
|
|
32
|
+
Register the default selftune cron jobs with OpenClaw.
|
|
33
|
+
|
|
34
|
+
| Flag | Description | Default |
|
|
35
|
+
|------|-------------|---------|
|
|
36
|
+
| `--dry-run` | Preview commands without registering jobs | Off |
|
|
37
|
+
| `--tz <timezone>` | IANA timezone for job schedules | Flag > `TZ` env > system timezone |
|
|
38
|
+
|
|
39
|
+
### `selftune cron list`
|
|
40
|
+
|
|
41
|
+
Show all registered selftune cron jobs. Reads from
|
|
42
|
+
`~/.openclaw/cron/jobs.json` and filters for `selftune-*` entries.
|
|
43
|
+
No flags.
|
|
44
|
+
|
|
45
|
+
### `selftune cron remove`
|
|
46
|
+
|
|
47
|
+
Remove all selftune cron jobs from OpenClaw.
|
|
48
|
+
|
|
49
|
+
| Flag | Description | Default |
|
|
50
|
+
|------|-------------|---------|
|
|
51
|
+
| `--dry-run` | Preview which jobs would be removed without deleting | Off |
|
|
52
|
+
|
|
53
|
+
## Default Job Schedule
|
|
54
|
+
|
|
55
|
+
Setup registers these four jobs:
|
|
56
|
+
|
|
57
|
+
| Name | Cron Expression | Schedule | Description |
|
|
58
|
+
|------|----------------|----------|-------------|
|
|
59
|
+
| `selftune-ingest` | `*/30 * * * *` | Every 30 minutes | Ingest new sessions from OpenClaw transcripts |
|
|
60
|
+
| `selftune-status` | `0 8 * * *` | Daily at 8am | Health check — report skills with pass rate below 80% |
|
|
61
|
+
| `selftune-evolve` | `0 3 * * 0` | Weekly at 3am Sunday | Full evolution pipeline for undertriggering skills |
|
|
62
|
+
| `selftune-watch` | `0 */6 * * *` | Every 6 hours | Monitor recently evolved skills for regressions |
|
|
63
|
+
|
|
64
|
+
All jobs run in **isolated session** mode — each execution gets a clean
|
|
65
|
+
session with no context accumulation from previous runs.
|
|
66
|
+
|
|
67
|
+
## Output
|
|
68
|
+
|
|
69
|
+
- **setup:** Registers jobs via `openclaw cron add` and confirms each registration
|
|
70
|
+
- **list:** Prints a formatted table of registered selftune cron jobs (name, schedule, description)
|
|
71
|
+
- **remove:** Deletes each selftune cron job via `openclaw cron remove` and confirms
|
|
72
|
+
|
|
73
|
+
Jobs persist at `~/.openclaw/cron/jobs.json` and survive OpenClaw restarts.
|
|
74
|
+
|
|
75
|
+
## Steps
|
|
76
|
+
|
|
77
|
+
1. Run `selftune cron setup --dry-run` to preview what would be registered
|
|
78
|
+
2. Run `selftune cron setup` to register the default jobs
|
|
79
|
+
3. Run `selftune cron list` to verify jobs are registered
|
|
80
|
+
4. Wait for the first cron cycle to fire (ingest runs every 30 minutes)
|
|
81
|
+
5. Check results with `selftune status` after the first daily health check
|
|
82
|
+
|
|
83
|
+
## The Autonomous Evolution Loop
|
|
84
|
+
|
|
85
|
+
When cron jobs are active, selftune operates as a self-correcting system:
|
|
86
|
+
|
|
87
|
+
```
|
|
88
|
+
Cron fires (isolated session)
|
|
89
|
+
|
|
|
90
|
+
v
|
|
91
|
+
Agent runs selftune pipeline (ingest -> status -> evolve -> watch)
|
|
92
|
+
|
|
|
93
|
+
v
|
|
94
|
+
Improved SKILL.md written to disk
|
|
95
|
+
|
|
|
96
|
+
v
|
|
97
|
+
OpenClaw file watcher detects change (250ms debounce)
|
|
98
|
+
|
|
|
99
|
+
v
|
|
100
|
+
Skill snapshot version bumped — next agent turn uses updated description
|
|
101
|
+
|
|
|
102
|
+
v
|
|
103
|
+
Better triggering in real-time, no restart needed
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
The four jobs form a continuous loop:
|
|
107
|
+
- **ingest** captures raw session data every 30 minutes
|
|
108
|
+
- **status** identifies undertriggering skills daily
|
|
109
|
+
- **evolve** proposes and deploys improvements weekly
|
|
110
|
+
- **watch** monitors for regressions every 6 hours and auto-rolls back if needed
|
|
111
|
+
|
|
112
|
+
Skills improve and take effect within seconds of the cron job completing.
|
|
113
|
+
No deployment step, no restart, no manual intervention.
|
|
114
|
+
|
|
115
|
+
## Safety Controls
|
|
116
|
+
|
|
117
|
+
| Control | How It Works |
|
|
118
|
+
|---------|-------------|
|
|
119
|
+
| Dry-run first | `selftune cron setup --dry-run` previews commands before registering |
|
|
120
|
+
| Regression threshold | Evolution only deploys if improvement exceeds 5% on existing triggers |
|
|
121
|
+
| Auto-rollback | `selftune watch` automatically rolls back if pass rate drops below baseline minus threshold |
|
|
122
|
+
| Audit trail | Every evolution recorded in `evolution_audit_log.jsonl` with full history |
|
|
123
|
+
| SKILL.md backup | `.bak` file created before every deploy — primary rollback path exists via .bak; fallback depends on audit metadata integrity |
|
|
124
|
+
| Isolated sessions | Each cron run gets a clean session (no context pollution between runs) |
|
|
125
|
+
| Human override | `selftune rollback --skill <name> --skill-path <path>` available anytime to manually revert |
|
|
126
|
+
| Pin descriptions | Config flag to freeze specific skills and prevent evolution on sensitive skills |
|
|
127
|
+
|
|
128
|
+
## Common Patterns
|
|
129
|
+
|
|
130
|
+
**"Set up autonomous skill evolution"**
|
|
131
|
+
> Run `selftune cron setup`. The four default jobs handle ingestion,
|
|
132
|
+
> health checks, evolution, and regression monitoring.
|
|
133
|
+
|
|
134
|
+
**"Preview before registering"**
|
|
135
|
+
> Run `selftune cron setup --dry-run` to see exactly what commands
|
|
136
|
+
> would be executed without registering anything.
|
|
137
|
+
|
|
138
|
+
**"Use a specific timezone"**
|
|
139
|
+
> Run `selftune cron setup --tz America/New_York`. Without the flag,
|
|
140
|
+
> timezone resolution is: `--tz` flag > `TZ` environment variable > system timezone.
|
|
141
|
+
|
|
142
|
+
**"What jobs are registered?"**
|
|
143
|
+
> Run `selftune cron list`. Shows a table of all selftune cron jobs
|
|
144
|
+
> with their schedules and descriptions.
|
|
145
|
+
|
|
146
|
+
**"Remove all cron automation"**
|
|
147
|
+
> Run `selftune cron remove`. Preview first with `selftune cron remove --dry-run`.
|
|
148
|
+
|
|
149
|
+
**"A skill regressed after cron evolution"**
|
|
150
|
+
> The watch job should catch this automatically. If not, run
|
|
151
|
+
> `selftune rollback --skill <name>` manually. See `Workflows/Rollback.md`.
|
|
152
|
+
|
|
153
|
+
**"How do I know the cron loop is working?"**
|
|
154
|
+
> Run `selftune status` after the first daily health check fires (8am).
|
|
155
|
+
> Check `evolution_audit_log.jsonl` for entries with recent timestamps.
|
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# selftune Dashboard Workflow
|
|
2
|
+
|
|
3
|
+
Visual dashboard for selftune telemetry, skill performance, evolution
|
|
4
|
+
audit, and monitoring data. Supports static HTML export, file output,
|
|
5
|
+
and a live server with SSE auto-refresh and action buttons.
|
|
6
|
+
|
|
7
|
+
## Default Command
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
selftune dashboard
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Opens a standalone HTML dashboard in the default browser with embedded
|
|
14
|
+
data from all selftune log files.
|
|
15
|
+
|
|
16
|
+
## Options
|
|
17
|
+
|
|
18
|
+
| Flag | Description | Default |
|
|
19
|
+
|------|-------------|---------|
|
|
20
|
+
| `--export` | Export data-embedded HTML to stdout | Off |
|
|
21
|
+
| `--out FILE` | Write data-embedded HTML to FILE | None |
|
|
22
|
+
| `--serve` | Start live dashboard server | Off |
|
|
23
|
+
| `--port <port>` | Custom port for live server (requires `--serve`) | 3141 |
|
|
24
|
+
|
|
25
|
+
## Modes
|
|
26
|
+
|
|
27
|
+
### Static (Default)
|
|
28
|
+
|
|
29
|
+
Builds an HTML file with all telemetry data embedded as JSON, saves it
|
|
30
|
+
to `~/.selftune/dashboard.html`, and opens it in the default browser.
|
|
31
|
+
The data is a point-in-time snapshot -- refresh by re-running the command.
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
selftune dashboard
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
### Export
|
|
38
|
+
|
|
39
|
+
Writes the same data-embedded HTML to stdout. Useful for piping to other
|
|
40
|
+
tools or capturing output programmatically.
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
selftune dashboard --export > dashboard.html
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
### File
|
|
47
|
+
|
|
48
|
+
Writes the data-embedded HTML to a specific file path.
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
selftune dashboard --out /tmp/report.html
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
### Live Server
|
|
55
|
+
|
|
56
|
+
Starts a Bun HTTP server with real-time data updates via Server-Sent
|
|
57
|
+
Events (SSE). The dashboard auto-refreshes every 5 seconds and provides
|
|
58
|
+
action buttons to trigger selftune commands.
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
selftune dashboard --serve
|
|
62
|
+
selftune dashboard --serve --port 8080
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Live Server
|
|
66
|
+
|
|
67
|
+
### Default Port
|
|
68
|
+
|
|
69
|
+
The live server binds to `localhost:3141` by default. Use `--port` to
|
|
70
|
+
override.
|
|
71
|
+
|
|
72
|
+
### Endpoints
|
|
73
|
+
|
|
74
|
+
| Method | Path | Description |
|
|
75
|
+
|--------|------|-------------|
|
|
76
|
+
| `GET` | `/` | Serve dashboard HTML with embedded data and live mode flag |
|
|
77
|
+
| `GET` | `/api/data` | JSON endpoint returning current telemetry data |
|
|
78
|
+
| `GET` | `/api/events` | SSE stream sending data updates every 5 seconds |
|
|
79
|
+
| `POST` | `/api/actions/watch` | Trigger `selftune watch` for a skill |
|
|
80
|
+
| `POST` | `/api/actions/evolve` | Trigger `selftune evolve` for a skill |
|
|
81
|
+
| `POST` | `/api/actions/rollback` | Trigger `selftune rollback` for a skill |
|
|
82
|
+
|
|
83
|
+
### SSE Auto-Refresh
|
|
84
|
+
|
|
85
|
+
The `/api/events` endpoint opens an SSE connection that pushes fresh
|
|
86
|
+
data every 5 seconds. The dashboard client listens for `data` events
|
|
87
|
+
and re-renders automatically. When `window.__SELFTUNE_LIVE__` is set
|
|
88
|
+
(injected by the live server), the dashboard enables SSE polling.
|
|
89
|
+
|
|
90
|
+
### Action Endpoints
|
|
91
|
+
|
|
92
|
+
Action buttons in the dashboard trigger selftune commands via POST
|
|
93
|
+
requests. Each endpoint spawns a `bun run` subprocess.
|
|
94
|
+
|
|
95
|
+
**Watch and Evolve** request body:
|
|
96
|
+
|
|
97
|
+
```json
|
|
98
|
+
{
|
|
99
|
+
"skill": "skill-name",
|
|
100
|
+
"skillPath": "/path/to/SKILL.md"
|
|
101
|
+
}
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**Rollback** request body:
|
|
105
|
+
|
|
106
|
+
```json
|
|
107
|
+
{
|
|
108
|
+
"skill": "skill-name",
|
|
109
|
+
"skillPath": "/path/to/SKILL.md",
|
|
110
|
+
"proposalId": "proposal-uuid"
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
All action endpoints return:
|
|
115
|
+
|
|
116
|
+
```json
|
|
117
|
+
{
|
|
118
|
+
"success": true,
|
|
119
|
+
"output": "command stdout",
|
|
120
|
+
"error": null
|
|
121
|
+
}
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
On failure, `success` is `false` and `error` contains the error message.
|
|
125
|
+
|
|
126
|
+
### Browser and Shutdown
|
|
127
|
+
|
|
128
|
+
The live server auto-opens the dashboard URL in the default browser on
|
|
129
|
+
macOS (`open`) and Linux (`xdg-open`).
|
|
130
|
+
|
|
131
|
+
Graceful shutdown on `SIGINT` (Ctrl+C) and `SIGTERM`: closes all SSE
|
|
132
|
+
client connections and stops the server.
|
|
133
|
+
|
|
134
|
+
## Data Contents
|
|
135
|
+
|
|
136
|
+
The dashboard displays data from these sources:
|
|
137
|
+
|
|
138
|
+
| Data | Source | Description |
|
|
139
|
+
|------|--------|-------------|
|
|
140
|
+
| Telemetry | `session_telemetry_log.jsonl` | Session-level telemetry records |
|
|
141
|
+
| Skills | `skill_usage_log.jsonl` | Skill activation and usage events |
|
|
142
|
+
| Queries | `all_queries_log.jsonl` | All user queries across sessions |
|
|
143
|
+
| Evolution | `evolution_audit_log.jsonl` | Evolution audit trail (create, deploy, rollback) |
|
|
144
|
+
| Decisions | `~/.selftune/memory/` | Evolution decision records |
|
|
145
|
+
| Snapshots | Computed | Per-skill monitoring snapshots (pass rate, regression status) |
|
|
146
|
+
| Unmatched | Computed | Queries that did not trigger any skill |
|
|
147
|
+
| Pending | Computed | Evolution proposals not yet deployed, rejected, or rolled back |
|
|
148
|
+
|
|
149
|
+
If no log data is found, the static modes exit with an error message
|
|
150
|
+
listing the checked file paths.
|
|
151
|
+
|
|
152
|
+
## Steps
|
|
153
|
+
|
|
154
|
+
### 1. Choose Mode
|
|
155
|
+
|
|
156
|
+
| Goal | Command |
|
|
157
|
+
|------|---------|
|
|
158
|
+
| Quick visual check | `selftune dashboard` |
|
|
159
|
+
| Save report to file | `selftune dashboard --out report.html` |
|
|
160
|
+
| Pipe to another tool | `selftune dashboard --export` |
|
|
161
|
+
| Live monitoring | `selftune dashboard --serve` |
|
|
162
|
+
|
|
163
|
+
### 2. Run Command
|
|
164
|
+
|
|
165
|
+
```bash
|
|
166
|
+
# Static (opens browser)
|
|
167
|
+
selftune dashboard
|
|
168
|
+
|
|
169
|
+
# Live server
|
|
170
|
+
selftune dashboard --serve
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
### 3. Interact with Dashboard
|
|
174
|
+
|
|
175
|
+
- **Static mode**: View the snapshot. Re-run to refresh.
|
|
176
|
+
- **Live mode**: Data refreshes automatically every 5 seconds. Use
|
|
177
|
+
action buttons to trigger watch, evolve, or rollback directly from
|
|
178
|
+
the dashboard.
|
|
179
|
+
|
|
180
|
+
## Common Patterns
|
|
181
|
+
|
|
182
|
+
**"Show me the dashboard"**
|
|
183
|
+
> Run `selftune dashboard`. Opens a browser with current data.
|
|
184
|
+
|
|
185
|
+
**"I want live updates"**
|
|
186
|
+
> Run `selftune dashboard --serve`. The SSE stream refreshes every 5
|
|
187
|
+
> seconds without manual intervention.
|
|
188
|
+
|
|
189
|
+
**"Export a report"**
|
|
190
|
+
> Use `selftune dashboard --out report.html` to save a self-contained
|
|
191
|
+
> HTML file. Share it -- no server needed, all data is embedded.
|
|
192
|
+
|
|
193
|
+
**"The dashboard shows no data"**
|
|
194
|
+
> No log files found. Run some sessions first so hooks generate
|
|
195
|
+
> telemetry. Check `selftune doctor` to verify hooks are installed.
|
|
196
|
+
|
|
197
|
+
**"Use a different port"**
|
|
198
|
+
> `selftune dashboard --serve --port 8080`. Port must be 1-65535.
|
|
199
|
+
|
|
200
|
+
**"Trigger actions from the dashboard"**
|
|
201
|
+
> In live server mode, the dashboard provides buttons to trigger watch,
|
|
202
|
+
> evolve, and rollback for each skill. These call the action endpoints
|
|
203
|
+
> which spawn selftune subprocesses.
|
|
@@ -82,8 +82,37 @@ Doctor validates these areas:
|
|
|
82
82
|
|
|
83
83
|
| Check | What it validates |
|
|
84
84
|
|-------|-------------------|
|
|
85
|
-
| Hooks installed | `UserPromptSubmit`, `PostToolUse`, and `Stop` hooks are configured in `~/.claude/settings.json` |
|
|
85
|
+
| Hooks installed | `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, and `Stop` hooks are configured in `~/.claude/settings.json` |
|
|
86
86
|
| Hook scripts exist | The script files referenced by hooks exist on disk |
|
|
87
|
+
| Auto-activate hook | `hooks/auto-activate.ts` is registered in `UserPromptSubmit` and the file is executable |
|
|
88
|
+
| Evolution guard hook | `hooks/evolution-guard.ts` is registered in `PreToolUse` and the file exists |
|
|
89
|
+
|
|
90
|
+
### Memory Checks
|
|
91
|
+
|
|
92
|
+
| Check | What it validates |
|
|
93
|
+
|-------|-------------------|
|
|
94
|
+
| Memory directory exists | `~/.selftune/memory/` directory is present |
|
|
95
|
+
| Memory files valid | `context.md`, `decisions.md`, `plan.md` exist and are non-empty (if previously written) |
|
|
96
|
+
|
|
97
|
+
### Activation Rules Checks
|
|
98
|
+
|
|
99
|
+
| Check | What it validates |
|
|
100
|
+
|-------|-------------------|
|
|
101
|
+
| Rules file exists | `~/.selftune/activation-rules.json` is present |
|
|
102
|
+
| Rules file valid | The file contains valid JSON conforming to the activation rules schema |
|
|
103
|
+
|
|
104
|
+
### Agent Checks
|
|
105
|
+
|
|
106
|
+
| Check | What it validates |
|
|
107
|
+
|-------|-------------------|
|
|
108
|
+
| Agent directory exists | `.claude/agents/` directory is present |
|
|
109
|
+
| Agent files present | Expected agent files exist: `diagnosis-analyst.md`, `pattern-analyst.md`, `evolution-reviewer.md`, `integration-guide.md` |
|
|
110
|
+
|
|
111
|
+
### Dashboard Checks (optional)
|
|
112
|
+
|
|
113
|
+
| Check | What it validates |
|
|
114
|
+
|-------|-------------------|
|
|
115
|
+
| Dashboard server accessible | `dashboard-server.ts` exists in the CLI directory |
|
|
87
116
|
|
|
88
117
|
### Evolution Audit Checks
|
|
89
118
|
|
|
@@ -114,6 +143,13 @@ For each failed check, take the appropriate action:
|
|
|
114
143
|
| Logs not parseable | Inspect the corrupted log file. Remove or fix invalid lines. |
|
|
115
144
|
| Hooks not installed | Merge `skill/settings_snippet.json` into `~/.claude/settings.json`. Update paths. |
|
|
116
145
|
| Hook scripts missing | Verify the selftune repo path. Re-run `init` if the repo was moved. |
|
|
146
|
+
| Auto-activate missing | Add `hooks/auto-activate.ts` to `UserPromptSubmit` in settings. |
|
|
147
|
+
| Evolution guard missing | Add `hooks/evolution-guard.ts` to `PreToolUse` in settings. |
|
|
148
|
+
| Memory directory missing | Run `mkdir -p ~/.selftune/memory`. |
|
|
149
|
+
| Memory files invalid | Delete and let the memory writer recreate them on next evolve/watch. |
|
|
150
|
+
| Activation rules missing | Copy `templates/activation-rules-default.json` to `~/.selftune/activation-rules.json`. |
|
|
151
|
+
| Activation rules invalid | Validate JSON syntax. Re-copy from template if corrupted. |
|
|
152
|
+
| Agent files missing | Copy agents from the selftune repo `.claude/agents/` directory. |
|
|
117
153
|
| Audit log invalid | Remove corrupted entries. Future operations will append clean entries. |
|
|
118
154
|
|
|
119
155
|
### 4. Re-run Doctor
|