selftune 0.2.16 → 0.2.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +32 -22
- package/apps/local-dashboard/dist/assets/index-DnhnXQm6.js +60 -0
- package/apps/local-dashboard/dist/assets/index-_EcLywDg.css +1 -0
- package/apps/local-dashboard/dist/assets/vendor-table-BIiI3YhS.js +1 -0
- package/apps/local-dashboard/dist/assets/vendor-ui-CGEmUayx.js +12 -0
- package/apps/local-dashboard/dist/index.html +5 -5
- package/cli/selftune/alpha-upload/build-payloads.ts +14 -1
- package/cli/selftune/alpha-upload/client.ts +51 -1
- package/cli/selftune/alpha-upload/flush.ts +46 -5
- package/cli/selftune/alpha-upload/stage-canonical.ts +32 -10
- package/cli/selftune/alpha-upload-contract.ts +9 -0
- package/cli/selftune/constants.ts +92 -5
- package/cli/selftune/contribute/contribute.ts +30 -2
- package/cli/selftune/contribute/sanitize.ts +52 -5
- package/cli/selftune/contribution-config.ts +249 -0
- package/cli/selftune/contribution-relay.ts +177 -0
- package/cli/selftune/contribution-signals.ts +219 -0
- package/cli/selftune/contribution-staging.ts +147 -0
- package/cli/selftune/contributions.ts +532 -0
- package/cli/selftune/creator-contributions.ts +333 -0
- package/cli/selftune/dashboard-contract.ts +305 -1
- package/cli/selftune/dashboard-server.ts +47 -13
- package/cli/selftune/eval/family-overlap.ts +395 -0
- package/cli/selftune/eval/hooks-to-evals.ts +182 -28
- package/cli/selftune/eval/synthetic-evals.ts +298 -11
- package/cli/selftune/evolution/description-quality.ts +12 -11
- package/cli/selftune/evolution/evolve.ts +214 -51
- package/cli/selftune/evolution/validate-proposal.ts +9 -6
- package/cli/selftune/export.ts +2 -2
- package/cli/selftune/grading/grade-session.ts +20 -0
- package/cli/selftune/hooks/commit-track.ts +188 -0
- package/cli/selftune/hooks/prompt-log.ts +10 -1
- package/cli/selftune/hooks/session-stop.ts +2 -2
- package/cli/selftune/hooks/skill-eval.ts +15 -1
- package/cli/selftune/hooks/stdin-preview.ts +32 -0
- package/cli/selftune/index.ts +41 -5
- package/cli/selftune/ingestors/codex-rollout.ts +31 -35
- package/cli/selftune/ingestors/codex-wrapper.ts +32 -24
- package/cli/selftune/localdb/db.ts +2 -2
- package/cli/selftune/localdb/direct-write.ts +69 -6
- package/cli/selftune/localdb/queries.ts +1253 -37
- package/cli/selftune/localdb/schema.ts +66 -0
- package/cli/selftune/orchestrate.ts +32 -4
- package/cli/selftune/recover.ts +153 -0
- package/cli/selftune/repair/skill-usage.ts +363 -4
- package/cli/selftune/routes/actions.ts +35 -1
- package/cli/selftune/routes/analytics.ts +14 -0
- package/cli/selftune/routes/index.ts +1 -0
- package/cli/selftune/routes/overview.ts +150 -4
- package/cli/selftune/routes/skill-report.ts +648 -18
- package/cli/selftune/status.ts +81 -2
- package/cli/selftune/sync.ts +56 -2
- package/cli/selftune/trust-model.ts +66 -0
- package/cli/selftune/types.ts +80 -0
- package/cli/selftune/utils/skill-detection.ts +43 -0
- package/cli/selftune/utils/transcript.ts +210 -1
- package/cli/selftune/watchlist.ts +65 -0
- package/node_modules/@selftune/telemetry-contract/src/types.ts +11 -0
- package/package.json +1 -1
- package/packages/telemetry-contract/src/types.ts +11 -0
- package/packages/ui/src/components/ActivityTimeline.tsx +165 -150
- package/packages/ui/src/components/EvidenceViewer.tsx +335 -144
- package/packages/ui/src/components/EvolutionTimeline.tsx +58 -28
- package/packages/ui/src/components/OrchestrateRunsPanel.tsx +33 -16
- package/packages/ui/src/components/RecentActivityFeed.tsx +72 -41
- package/packages/ui/src/components/section-cards.tsx +12 -9
- package/packages/ui/src/primitives/card.tsx +1 -1
- package/skill/SKILL.md +40 -2
- package/skill/Workflows/AlphaUpload.md +4 -0
- package/skill/Workflows/Composability.md +64 -0
- package/skill/Workflows/Contribute.md +6 -3
- package/skill/Workflows/Contributions.md +97 -0
- package/skill/Workflows/CreatorContributions.md +74 -0
- package/skill/Workflows/Dashboard.md +31 -0
- package/skill/Workflows/Evals.md +57 -8
- package/skill/Workflows/Evolve.md +31 -13
- package/skill/Workflows/ExportCanonical.md +121 -0
- package/skill/Workflows/Hook.md +131 -0
- package/skill/Workflows/Ingest.md +7 -0
- package/skill/Workflows/Initialize.md +29 -9
- package/skill/Workflows/Orchestrate.md +27 -5
- package/skill/Workflows/Quickstart.md +94 -0
- package/skill/Workflows/Recover.md +84 -0
- package/skill/Workflows/RepairSkillUsage.md +95 -0
- package/skill/Workflows/Sync.md +18 -12
- package/skill/Workflows/Uninstall.md +82 -0
- package/skill/settings_snippet.json +11 -0
- package/apps/local-dashboard/dist/assets/index-BMIS6uUh.css +0 -2
- package/apps/local-dashboard/dist/assets/index-DOu3iLD9.js +0 -16
- package/apps/local-dashboard/dist/assets/vendor-table-pHbDxq36.js +0 -8
- package/apps/local-dashboard/dist/assets/vendor-ui-DIwlrGlb.js +0 -12
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# selftune Hook Workflow
|
|
2
|
+
|
|
3
|
+
Manually invoke individual Claude Code hooks for debugging and testing.
|
|
4
|
+
Each hook reads its payload from stdin and behaves exactly as it would when
|
|
5
|
+
triggered by the Claude Code host agent.
|
|
6
|
+
|
|
7
|
+
## When to Use
|
|
8
|
+
|
|
9
|
+
- Debugging a specific hook's behavior with a known payload
|
|
10
|
+
- The user says "hook", "run hook", "invoke hook", "manual hook", or "debug hook"
|
|
11
|
+
- Testing hook installation by simulating a hook event
|
|
12
|
+
- Verifying hook output before or after configuration changes
|
|
13
|
+
|
|
14
|
+
## Default Command
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
echo '{"payload":"..."}' | selftune hook <name>
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Where `<name>` is one of the 6 available hooks.
|
|
21
|
+
|
|
22
|
+
## Available Hooks
|
|
23
|
+
|
|
24
|
+
| Hook Name | Claude Code Event | Purpose |
|
|
25
|
+
| ---------------------- | ---------------------- | ----------------------------------------------------------------------- |
|
|
26
|
+
| `prompt-log` | UserPromptSubmit | Logs every user query to SQLite for false-negative eval detection |
|
|
27
|
+
| `session-stop` | Stop | Extracts session-level telemetry from transcript when a session ends |
|
|
28
|
+
| `skill-eval` | PostToolUse | Records skill usage when a SKILL.md is read or a Skill tool is invoked |
|
|
29
|
+
| `auto-activate` | UserPromptSubmit | Evaluates activation rules and suggests selftune actions via stderr |
|
|
30
|
+
| `skill-change-guard` | PreToolUse | Warns (advisory) when an agent is about to write to a SKILL.md file |
|
|
31
|
+
| `evolution-guard` | PreToolUse | Blocks writes to monitored SKILL.md files until `selftune watch` runs |
|
|
32
|
+
|
|
33
|
+
## Hook Details
|
|
34
|
+
|
|
35
|
+
### prompt-log
|
|
36
|
+
|
|
37
|
+
Fires on every user message before Claude processes it. Writes the query to
|
|
38
|
+
SQLite so that `hooks-to-evals` can identify prompts that did NOT trigger a
|
|
39
|
+
skill — the raw material for false-negative eval entries. Also writes a
|
|
40
|
+
canonical prompt record.
|
|
41
|
+
|
|
42
|
+
### session-stop
|
|
43
|
+
|
|
44
|
+
Fires when a Claude Code session ends. Reads the session transcript JSONL and
|
|
45
|
+
extracts process-level telemetry (tool calls, errors, skills triggered, token
|
|
46
|
+
counts). Writes one record per session to SQLite with a JSONL backup. May
|
|
47
|
+
trigger a reactive `selftune orchestrate` spawn if conditions are met.
|
|
48
|
+
|
|
49
|
+
### skill-eval
|
|
50
|
+
|
|
51
|
+
Fires after Read or Skill tool calls. If the target is a SKILL.md file or a
|
|
52
|
+
Skill invocation, finds the triggering user query from the transcript and
|
|
53
|
+
writes a usage record. Builds the real-usage eval dataset over time.
|
|
54
|
+
|
|
55
|
+
### auto-activate
|
|
56
|
+
|
|
57
|
+
Fires on every user message. Evaluates activation rules against the session
|
|
58
|
+
context and outputs suggestions to stderr (shown to Claude as system messages).
|
|
59
|
+
Suggestions are advisory only — exit code is always 0. Tracks session state to
|
|
60
|
+
avoid repeated suggestions.
|
|
61
|
+
|
|
62
|
+
### skill-change-guard
|
|
63
|
+
|
|
64
|
+
Fires before Write/Edit tool calls. If the target is a SKILL.md file, outputs
|
|
65
|
+
a suggestion to run `selftune watch --skill <name>` to monitor impact. Advisory
|
|
66
|
+
only — exit code is always 0, never blocking. Uses session state to avoid
|
|
67
|
+
repeating suggestions for the same skill.
|
|
68
|
+
|
|
69
|
+
### evolution-guard
|
|
70
|
+
|
|
71
|
+
Fires before Write/Edit tool calls. If the target is a SKILL.md file that has
|
|
72
|
+
a deployed evolution under active monitoring, and no recent `selftune watch`
|
|
73
|
+
snapshot exists, this hook BLOCKS the write with exit code 2. This prevents
|
|
74
|
+
unmonitored changes to skills that are being tracked.
|
|
75
|
+
|
|
76
|
+
Exit codes:
|
|
77
|
+
|
|
78
|
+
- `0` — Allow (not a SKILL.md, not monitored, or watch is recent)
|
|
79
|
+
- `2` — Block with message (Claude Code convention for PreToolUse hooks)
|
|
80
|
+
|
|
81
|
+
Fail-open: any internal error results in exit 0 (never blocks accidentally).
|
|
82
|
+
|
|
83
|
+
## Output Format
|
|
84
|
+
|
|
85
|
+
Hook output varies by hook type:
|
|
86
|
+
|
|
87
|
+
- **prompt-log, session-stop, skill-eval**: Write to SQLite and JSONL logs silently. Exit 0 on success.
|
|
88
|
+
- **auto-activate**: Writes suggestions to stderr. Exit 0 always.
|
|
89
|
+
- **skill-change-guard**: Writes advisory message to stderr. Exit 0 always.
|
|
90
|
+
- **evolution-guard**: Writes block message to stderr on exit 2. Exit 0 when allowing.
|
|
91
|
+
|
|
92
|
+
## Common Patterns
|
|
93
|
+
|
|
94
|
+
**Debug a prompt-log hook**
|
|
95
|
+
|
|
96
|
+
> Pipe a UserPromptSubmit payload to test prompt logging:
|
|
97
|
+
>
|
|
98
|
+
> ```bash
|
|
99
|
+
> echo '{"session_id":"test","query":"improve my skills"}' | selftune hook prompt-log
|
|
100
|
+
> ```
|
|
101
|
+
|
|
102
|
+
**Test skill-eval with a PostToolUse payload**
|
|
103
|
+
|
|
104
|
+
> ```bash
|
|
105
|
+
> echo '{"tool_name":"Read","file_path":"/path/to/SKILL.md","session_id":"test"}' | selftune hook skill-eval
|
|
106
|
+
> ```
|
|
107
|
+
|
|
108
|
+
**Verify evolution-guard blocks correctly**
|
|
109
|
+
|
|
110
|
+
> ```bash
|
|
111
|
+
> echo '{"tool_name":"Write","file_path":"/path/to/monitored/SKILL.md"}' | selftune hook evolution-guard
|
|
112
|
+
> echo $? # Should be 2 if skill is monitored without recent watch
|
|
113
|
+
> ```
|
|
114
|
+
|
|
115
|
+
## Error Handling
|
|
116
|
+
|
|
117
|
+
If no hook name is provided or the name is unrecognized, the command exits with
|
|
118
|
+
a `UNKNOWN_COMMAND` error listing available hooks:
|
|
119
|
+
|
|
120
|
+
```
|
|
121
|
+
Unknown hook: (none). Available: prompt-log, session-stop, skill-eval, auto-activate, skill-change-guard, evolution-guard
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
## Troubleshooting
|
|
125
|
+
|
|
126
|
+
| Symptom | Cause | Fix |
|
|
127
|
+
| --- | --- | --- |
|
|
128
|
+
| "Unknown hook" error | Typo in hook name | Use one of: `prompt-log`, `session-stop`, `skill-eval`, `auto-activate`, `skill-change-guard`, `evolution-guard` |
|
|
129
|
+
| Hook exits 0 but no data written | Payload missing required fields | Check the hook's expected payload schema in `cli/selftune/types.ts` |
|
|
130
|
+
| evolution-guard always exits 0 | No deployed evolution for the target skill | Run `selftune evolve` first to deploy an evolution, then test the guard |
|
|
131
|
+
| auto-activate produces no suggestions | Activation rules not configured or already suggested in session | Check `~/.selftune/` for activation rules and session state files |
|
|
@@ -93,6 +93,13 @@ Writes to:
|
|
|
93
93
|
- `~/.claude/all_queries_log.jsonl` -- extracted user queries
|
|
94
94
|
- `~/.claude/session_telemetry_log.jsonl` -- per-session metrics with `source: "codex_rollout"`
|
|
95
95
|
|
|
96
|
+
### Notes
|
|
97
|
+
|
|
98
|
+
- Conservative skill attribution: Codex rollout ingest only attributes a skill when it has
|
|
99
|
+
explicit evidence, such as a skill file/path read or an explicit user mention that invokes
|
|
100
|
+
the skill. Incidental mentions inside assistant reasoning, optimizer prompts, or eval text do
|
|
101
|
+
not count as triggers.
|
|
102
|
+
|
|
96
103
|
### Steps
|
|
97
104
|
|
|
98
105
|
1. Verify `$CODEX_HOME/sessions/` directory exists and contains session files
|
|
@@ -126,14 +126,15 @@ Code subagent calls stay up to date.
|
|
|
126
126
|
|
|
127
127
|
**Hook reference** (for troubleshooting):
|
|
128
128
|
|
|
129
|
-
| Hook | Script | Purpose | Notes
|
|
130
|
-
| -------------------------- | ----------------------------- | ----------------------------------------------- |
|
|
131
|
-
| `UserPromptSubmit` | `hooks/prompt-log.ts` | Log every user query | Accepts both `prompt` and legacy `user_prompt`
|
|
132
|
-
| `UserPromptSubmit` | `hooks/auto-activate.ts` | Suggest skills before prompt processing | Uses `additionalContext` JSON for suggestions
|
|
133
|
-
| `PreToolUse` (Write/Edit) | `hooks/skill-change-guard.ts` | Detect uncontrolled skill edits | `if` filter: only fires on `*SKILL.md` paths
|
|
134
|
-
| `PreToolUse` (Write/Edit) | `hooks/evolution-guard.ts` | Block SKILL.md edits on monitored skills | `if` filter: only fires on `*SKILL.md` paths
|
|
135
|
-
| `PostToolUse` (Read/Skill) | `hooks/skill-eval.ts` | Track skill triggers and Skill tool invocations |
|
|
136
|
-
| `
|
|
129
|
+
| Hook | Script | Purpose | Notes |
|
|
130
|
+
| -------------------------- | ----------------------------- | ----------------------------------------------- | ----------------------------------------------- |
|
|
131
|
+
| `UserPromptSubmit` | `hooks/prompt-log.ts` | Log every user query | Accepts both `prompt` and legacy `user_prompt` |
|
|
132
|
+
| `UserPromptSubmit` | `hooks/auto-activate.ts` | Suggest skills before prompt processing | Uses `additionalContext` JSON for suggestions |
|
|
133
|
+
| `PreToolUse` (Write/Edit) | `hooks/skill-change-guard.ts` | Detect uncontrolled skill edits | `if` filter: only fires on `*SKILL.md` paths |
|
|
134
|
+
| `PreToolUse` (Write/Edit) | `hooks/evolution-guard.ts` | Block SKILL.md edits on monitored skills | `if` filter: only fires on `*SKILL.md` paths |
|
|
135
|
+
| `PostToolUse` (Read/Skill) | `hooks/skill-eval.ts` | Track skill triggers and Skill tool invocations | Fast-path: skips non-PostToolUse/non-Read/Skill |
|
|
136
|
+
| `PostToolUse` (Bash) | `hooks/commit-track.ts` | Track git commits for session traceability | Fast-path: skips non-git Bash commands |
|
|
137
|
+
| `Stop` | `hooks/session-stop.ts` | Capture session telemetry | Runs async (non-blocking), 60s timeout |
|
|
137
138
|
|
|
138
139
|
**Codex agents:**
|
|
139
140
|
|
|
@@ -191,6 +192,25 @@ and evolution pipeline have data to work with immediately.
|
|
|
191
192
|
The sync step is fail-open — if it encounters errors, init continues.
|
|
192
193
|
Skip with `--no-sync` if you only want hooks for forward-looking data.
|
|
193
194
|
|
|
195
|
+
If the user is migrating from a much older pre-SQLite install and wants to
|
|
196
|
+
recover legacy selftune JSONL history itself, use `selftune recover` as a
|
|
197
|
+
separate recovery step. That is not part of normal first-time setup.
|
|
198
|
+
|
|
199
|
+
Recovery quick reference:
|
|
200
|
+
|
|
201
|
+
| Flag | Description |
|
|
202
|
+
| --- | --- |
|
|
203
|
+
| `--full` | Rebuild SQLite from the available JSONL/export sources |
|
|
204
|
+
| `--force` | Skip the SQLite-only preflight guard during a full rebuild |
|
|
205
|
+
| `--since <date>` | Recover only rows on or after the given date |
|
|
206
|
+
| `--json` | Output JSON summary instead of human-readable text |
|
|
207
|
+
|
|
208
|
+
Example:
|
|
209
|
+
|
|
210
|
+
```bash
|
|
211
|
+
selftune recover --full --force
|
|
212
|
+
```
|
|
213
|
+
|
|
194
214
|
### 9. Autonomy Scheduling
|
|
195
215
|
|
|
196
216
|
Init automatically installs OS-level scheduling (launchd on macOS, cron/systemd
|
|
@@ -270,7 +290,7 @@ Enrollment uses a device-code flow — one command, one browser approval, fully
|
|
|
270
290
|
|
|
271
291
|
### Setup Sequence
|
|
272
292
|
|
|
273
|
-
1. **Check local config**: Run `selftune status` — look for the "Alpha Upload" section
|
|
293
|
+
1. **Check local config**: Run `selftune status` — use the first summary line and compact `Highlights` section to explain current skill health, then look for the "Alpha Upload" section
|
|
274
294
|
2. **If not linked**: First use `AskUserQuestion` for the opt-in decision. Only if the user says yes, collect their email and run:
|
|
275
295
|
|
|
276
296
|
```bash
|
|
@@ -20,6 +20,22 @@ recent changes with auto-rollback enabled.
|
|
|
20
20
|
selftune orchestrate
|
|
21
21
|
```
|
|
22
22
|
|
|
23
|
+
Autonomous evolve settings used by orchestrate:
|
|
24
|
+
|
|
25
|
+
```text
|
|
26
|
+
confidenceThreshold = 0.6
|
|
27
|
+
maxIterations = 3
|
|
28
|
+
paretoEnabled = true
|
|
29
|
+
candidateCount = 3
|
|
30
|
+
tokenEfficiencyEnabled = false
|
|
31
|
+
withBaseline = false
|
|
32
|
+
validationModel = haiku
|
|
33
|
+
cheapLoop = true
|
|
34
|
+
gateModel = sonnet
|
|
35
|
+
adaptiveGate = true
|
|
36
|
+
proposalModel = haiku
|
|
37
|
+
```
|
|
38
|
+
|
|
23
39
|
## Flags
|
|
24
40
|
|
|
25
41
|
| Flag | Description | Default |
|
|
@@ -109,10 +125,11 @@ This is the recommended runtime for recurring autonomous scheduling.
|
|
|
109
125
|
| **Automated (loop)** | `selftune orchestrate --loop` | No agent session; LLM cost only if evolution triggers | Configurable interval |
|
|
110
126
|
|
|
111
127
|
In automated mode, the OS calls the CLI binary directly. No agent session
|
|
112
|
-
is created.
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
128
|
+
is created. Outside of the regular sync/status/candidate-selection logic,
|
|
129
|
+
LLM calls can come from auto-grading ungraded skills and from the evolution
|
|
130
|
+
step itself. By default, orchestrate runs proposal generation and validation
|
|
131
|
+
on `haiku`, then re-runs the final gate on `sonnet` before deploy. Risky
|
|
132
|
+
candidates are escalated to `opus` with `high` effort for the gate only.
|
|
116
133
|
|
|
117
134
|
**Cron mode:** Install OS-level scheduling with `selftune cron setup`.
|
|
118
135
|
Runs as separate invocations on a schedule (default: every 6 hours).
|
|
@@ -144,10 +161,15 @@ In autonomous mode, orchestrate calls sub-workflows in this fixed order:
|
|
|
144
161
|
1. **Sync** — refresh source-truth telemetry across all supported agents (`selftune sync`)
|
|
145
162
|
2. **Status** — compute skill health using existing grade results (reads `grading.json` outputs from previous sessions)
|
|
146
163
|
3. **Auto-grade** — grade up to `--max-auto-grade` (default 5) ungraded skills that have session data but no grades yet. Skipped during `--dry-run` (grading makes LLM calls). After grading, status is recomputed so candidate selection sees updated grades. Fail-open: individual grading errors are logged but never block the loop.
|
|
147
|
-
4. **Evolve** — run evolution on selected candidates (pre-flight is skipped
|
|
164
|
+
4. **Evolve** — run evolution on selected candidates (pre-flight is skipped; Pareto mode uses 3 candidates; cheap-loop uses `haiku` for proposal + validation and `sonnet` for the final gate; adaptive gate escalation promotes risky proposals to `opus` + `high` effort; baseline and token-efficiency stay off)
|
|
148
165
|
5. **Watch** — monitor recently evolved skills (auto-rollback enabled by default, `--recent-window` hours lookback)
|
|
149
166
|
6. **Alpha Upload** — if enrolled in the alpha program (`config.alpha.enrolled === true`) and an API key is configured, stage new canonical records (sessions, invocations, evolution evidence, orchestrate runs) into `canonical_upload_staging`, build V2 push payloads, and flush to the cloud API (`POST /api/v1/push`) with Bearer auth. Fail-open: upload errors never block the orchestrate loop. Respects `--dry-run`.
|
|
150
167
|
|
|
168
|
+
When orchestrate invokes evolve for a selected candidate, it always passes
|
|
169
|
+
`confidenceThreshold: 0.6` and `maxIterations: 3`, plus the autonomous evolve
|
|
170
|
+
defaults listed above. Those defaults are the recurring-run policy for the
|
|
171
|
+
autonomy-first loop; there are no orchestrate flags to override them per run.
|
|
172
|
+
|
|
151
173
|
Between candidate selection and evolution, orchestrate checks for
|
|
152
174
|
**cross-skill eval set overlap**. When two or more evolution candidates
|
|
153
175
|
share >30% of their positive eval queries, a warning is logged to stderr.
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# selftune Quickstart Workflow
|
|
2
|
+
|
|
3
|
+
Guided onboarding that runs init, ingest, and status in a single command.
|
|
4
|
+
Designed for first-time users who want to get selftune working immediately.
|
|
5
|
+
|
|
6
|
+
## When to Use
|
|
7
|
+
|
|
8
|
+
- The user is setting up selftune for the first time
|
|
9
|
+
- The user says "getting started", "quickstart", "onboard", or "first time"
|
|
10
|
+
- The agent needs to bootstrap selftune in one step without running init, ingest, and status separately
|
|
11
|
+
|
|
12
|
+
## Default Command
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
selftune quickstart
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Help:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
selftune quickstart --help
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Options
|
|
25
|
+
|
|
26
|
+
| Flag | Description |
|
|
27
|
+
| -------- | ---------------------- |
|
|
28
|
+
| `--help` | Show usage information |
|
|
29
|
+
|
|
30
|
+
## Steps Performed
|
|
31
|
+
|
|
32
|
+
Quickstart runs three steps automatically:
|
|
33
|
+
|
|
34
|
+
1. **Init** — Creates `~/.selftune/config.json` if it does not exist. Skips if config is already present.
|
|
35
|
+
2. **Ingest** — Runs Claude Code transcript replay if the ingest marker file does not exist. Discovers transcripts from `~/.claude/projects/` and writes session telemetry to SQLite.
|
|
36
|
+
3. **Status** — Displays current skill health using `computeStatus`. Shows pass rates, trends, and health indicators for all detected skills; when you need per-skill check volume, look at `snapshot.skill_checks` rather than a "session count" field.
|
|
37
|
+
|
|
38
|
+
After status, quickstart suggests the top 3 skills that would benefit from evolution, prioritized by:
|
|
39
|
+
|
|
40
|
+
- **UNGRADED/UNKNOWN** skills (highest priority) — suggests running `selftune grade`
|
|
41
|
+
- **CRITICAL** skills (pass rate below threshold) — suggests evolution
|
|
42
|
+
- **WARNING** skills — suggests improvement
|
|
43
|
+
|
|
44
|
+
## Output Format
|
|
45
|
+
|
|
46
|
+
```text
|
|
47
|
+
selftune quickstart
|
|
48
|
+
====================
|
|
49
|
+
|
|
50
|
+
[1/3] Config exists, skipping init.
|
|
51
|
+
[2/3] Running ingest claude...
|
|
52
|
+
Ingested 12 sessions.
|
|
53
|
+
[3/3] Current status:
|
|
54
|
+
|
|
55
|
+
Skill Health Summary
|
|
56
|
+
...
|
|
57
|
+
|
|
58
|
+
Suggested next steps:
|
|
59
|
+
- my-skill: pass rate 45% — needs evolution
|
|
60
|
+
- other-skill: needs grading — run `selftune grade --skill other-skill`
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
If all skills are healthy, the output ends with:
|
|
64
|
+
|
|
65
|
+
```text
|
|
66
|
+
All skills are healthy. No immediate actions needed.
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
## Common Patterns
|
|
70
|
+
|
|
71
|
+
**First-time setup**
|
|
72
|
+
|
|
73
|
+
> Run `selftune quickstart`. It handles init, ingest, and status automatically.
|
|
74
|
+
|
|
75
|
+
**Already initialized**
|
|
76
|
+
|
|
77
|
+
> Quickstart skips steps that are already complete (config exists, ingest marker exists). It is safe to run multiple times.
|
|
78
|
+
|
|
79
|
+
**No transcripts found**
|
|
80
|
+
|
|
81
|
+
> If no Claude Code transcripts exist in `~/.claude/projects/`, quickstart reports "No Claude Code transcripts found" and continues to the status step. The user should run some agent sessions first, then re-run quickstart.
|
|
82
|
+
|
|
83
|
+
**Status or ingest fails**
|
|
84
|
+
|
|
85
|
+
> Quickstart catches errors in each step and suggests the manual command for troubleshooting (e.g., `selftune init`, `selftune ingest claude`, or `selftune status`).
|
|
86
|
+
|
|
87
|
+
## Troubleshooting
|
|
88
|
+
|
|
89
|
+
| Symptom | Cause | Fix |
|
|
90
|
+
| --- | --- | --- |
|
|
91
|
+
| "Init failed" at step 1 | Config directory permissions or corrupted config | Run `selftune init --force` manually |
|
|
92
|
+
| "Ingest failed" at step 2 | Transcript directory missing or unreadable | Verify `~/.claude/projects/` exists and contains session directories |
|
|
93
|
+
| "No sessions found" after ingest | No actionable transcripts or no skill usage detected | Run agent sessions that use skills, then re-run quickstart |
|
|
94
|
+
| "Status failed" at step 3 | SQLite database issue | Run `selftune doctor` to diagnose |
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# selftune Recover Workflow
|
|
2
|
+
|
|
3
|
+
Recover or backfill the local SQLite database from legacy JSONL files or an
|
|
4
|
+
explicit `selftune export` snapshot.
|
|
5
|
+
|
|
6
|
+
This is a recovery-only workflow. Normal operation should use `selftune sync`,
|
|
7
|
+
which replays native source data into SQLite and also triggers alpha upload
|
|
8
|
+
when enrolled.
|
|
9
|
+
|
|
10
|
+
## When to Use
|
|
11
|
+
|
|
12
|
+
- The user is migrating from a pre-SQLite selftune install and still has
|
|
13
|
+
legacy JSONL history that is not in SQLite yet
|
|
14
|
+
- The user exported SQLite to JSONL and now needs to rebuild a fresh DB from
|
|
15
|
+
that snapshot
|
|
16
|
+
- The user explicitly asks to recover, rebuild, or backfill SQLite from JSONL
|
|
17
|
+
|
|
18
|
+
## Default Command
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
selftune recover
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Options
|
|
25
|
+
|
|
26
|
+
| Flag | Description |
|
|
27
|
+
| ------------------------------ | ------------------------------------------------------------- |
|
|
28
|
+
| `--full` | Rebuild SQLite tables from scratch |
|
|
29
|
+
| `--force` | Skip the preflight guard for SQLite-only rows during rebuild |
|
|
30
|
+
| `--since <date>` | Incrementally materialize records on/after this date |
|
|
31
|
+
| `--canonical-log <path>` | Canonical JSONL path override |
|
|
32
|
+
| `--telemetry-log <path>` | Session telemetry JSONL path override |
|
|
33
|
+
| `--evolution-audit-log <path>` | Evolution audit JSONL path override |
|
|
34
|
+
| `--evolution-evidence-log <path>` | Evolution evidence JSONL path override |
|
|
35
|
+
| `--orchestrate-run-log <path>` | Orchestrate runs JSONL path override |
|
|
36
|
+
| `--json` | Output a JSON summary |
|
|
37
|
+
|
|
38
|
+
## Output
|
|
39
|
+
|
|
40
|
+
The command prints a summary of what was materialized into SQLite:
|
|
41
|
+
|
|
42
|
+
- sessions
|
|
43
|
+
- prompts
|
|
44
|
+
- skill invocations
|
|
45
|
+
- execution facts
|
|
46
|
+
- session telemetry
|
|
47
|
+
- legacy skill usage
|
|
48
|
+
- evolution audit
|
|
49
|
+
- evolution evidence
|
|
50
|
+
- orchestrate runs
|
|
51
|
+
|
|
52
|
+
With `--json`, the result includes `mode`, `source`, `since`, `force`, and the
|
|
53
|
+
full count breakdown.
|
|
54
|
+
|
|
55
|
+
## Common Patterns
|
|
56
|
+
|
|
57
|
+
**Backfill legacy JSONL into an existing SQLite DB**
|
|
58
|
+
|
|
59
|
+
> Run `selftune recover`.
|
|
60
|
+
|
|
61
|
+
**Rebuild a deleted DB from an explicit export snapshot**
|
|
62
|
+
|
|
63
|
+
> Run `selftune export --output ./recovery-snapshot`, then recover from the exported JSONL files explicitly:
|
|
64
|
+
>
|
|
65
|
+
> `selftune recover --full --force --telemetry-log ./recovery-snapshot/session_telemetry_log.jsonl --evolution-audit-log ./recovery-snapshot/evolution_audit_log.jsonl --evolution-evidence-log ./recovery-snapshot/evolution_evidence_log.jsonl --orchestrate-run-log ./recovery-snapshot/orchestrate_run_log.jsonl`
|
|
66
|
+
|
|
67
|
+
**Recover only recent JSONL rows**
|
|
68
|
+
|
|
69
|
+
> Run `selftune recover --since 2026-01-01`.
|
|
70
|
+
|
|
71
|
+
## Important Notes
|
|
72
|
+
|
|
73
|
+
- Do **not** use this as a normal freshness command. Use `selftune sync` for day-to-day operation.
|
|
74
|
+
- Alpha upload remains SQLite-first. Recovery only repopulates SQLite so the normal upload pipeline can stage and send data afterward.
|
|
75
|
+
- If you are recovering from post-cutover data, prefer a SQLite backup or `selftune export` snapshot. Passive legacy JSONL files do not contain all post-cutover records.
|
|
76
|
+
|
|
77
|
+
## Example Flags Used Above
|
|
78
|
+
|
|
79
|
+
| Flag | Description |
|
|
80
|
+
| --- | --- |
|
|
81
|
+
| `-o, --output <dir>` | Export SQLite into a portable snapshot directory |
|
|
82
|
+
| `--full` | Rebuild SQLite tables from scratch |
|
|
83
|
+
| `--force` | Skip the SQLite-only preflight guard during full rebuild |
|
|
84
|
+
| `--telemetry-log <path>` | Point recover at the exported telemetry JSONL file |
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# selftune Repair Skill Usage Workflow
|
|
2
|
+
|
|
3
|
+
Rebuild trustworthy skill-usage records by replaying Claude Code transcripts
|
|
4
|
+
and Codex rollouts. Repairs the canonical SQLite `skill_invocations` table for
|
|
5
|
+
historical legacy rows, and also writes a compatibility/export overlay
|
|
6
|
+
(`skill_usage_repaired.jsonl`) that corrects missing or inaccurate skill paths,
|
|
7
|
+
scopes, and query associations from the raw hook-captured data. The overlay is
|
|
8
|
+
for compatibility/export only; SQLite is the operational source of truth.
|
|
9
|
+
|
|
10
|
+
## When to Use
|
|
11
|
+
|
|
12
|
+
- Skill usage data appears incomplete or has missing `skill_path` values
|
|
13
|
+
- The user says "repair", "rebuild usage", "fix skill usage", or "trustworthy usage"
|
|
14
|
+
- Before evolution or grading when hook data may be unreliable
|
|
15
|
+
- After changing skill directory layouts or reinstalling skills
|
|
16
|
+
|
|
17
|
+
## Default Command
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
selftune repair-skill-usage
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## Options
|
|
24
|
+
|
|
25
|
+
| Flag | Description |
|
|
26
|
+
| --------------------------- | -------------------------------------------------------- |
|
|
27
|
+
| `--projects-dir <dir>` | Claude transcript directory (default: `~/.claude/projects`) |
|
|
28
|
+
| `--codex-home <dir>` | Codex home directory (default: `~/.codex`) |
|
|
29
|
+
| `--since <date>` | Only repair sessions modified on/after this date |
|
|
30
|
+
| `--out <path>` | Repaired overlay log output path |
|
|
31
|
+
| `--sessions-marker <path>` | Repaired session-id marker file path |
|
|
32
|
+
| `--skill-log <path>` | Raw skill usage log path (for path lookup bootstrap) |
|
|
33
|
+
| `--dry-run` | Show summary counts without writing files |
|
|
34
|
+
| `--help` | Show usage information |
|
|
35
|
+
|
|
36
|
+
## How It Works
|
|
37
|
+
|
|
38
|
+
1. **Discover transcripts** — Finds all Claude Code transcript JSONL files under `--projects-dir` and Codex rollout files under `--codex-home`.
|
|
39
|
+
2. **Bootstrap path lookup** — Reads existing skill usage records from SQLite (or `--skill-log` JSONL) to build a skill-name-to-path lookup table.
|
|
40
|
+
3. **Replay transcripts** — For each transcript, parses the conversation to find Skill tool invocations, associates them with the preceding user query, and resolves skill paths via installed-scope discovery or the lookup table.
|
|
41
|
+
4. **Replay Codex rollouts** — Same process for Codex rollout files, using Codex-specific path resolution.
|
|
42
|
+
5. **Repair SQLite + write overlay** — Replaces legacy triggered skill rows in SQLite when safe, inserts repaired canonical rows for legacy-only session/skill pairs, reconstructs contextual missed triggers from `Read .../SKILL.md` evidence, then writes deduplicated records to the repaired log and updates the session marker.
|
|
43
|
+
|
|
44
|
+
## Output Format
|
|
45
|
+
|
|
46
|
+
JSON summary printed to stdout:
|
|
47
|
+
|
|
48
|
+
```json
|
|
49
|
+
{
|
|
50
|
+
"transcripts_scanned": 42,
|
|
51
|
+
"codex_rollouts_scanned": 8,
|
|
52
|
+
"repaired_sessions": 35,
|
|
53
|
+
"repaired_records": 127,
|
|
54
|
+
"codex_repaired_records": 12,
|
|
55
|
+
"unique_matched_queries": 98,
|
|
56
|
+
"reins_queries_seen": 5,
|
|
57
|
+
"reins_skill_matches": 3,
|
|
58
|
+
"output": "~/.claude/skill_usage_repaired.jsonl"
|
|
59
|
+
}
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
With `--dry-run`, the same summary is printed but no files are written.
|
|
63
|
+
|
|
64
|
+
On non-dry runs, the JSON summary also includes a `sqlite` object with counts
|
|
65
|
+
for deleted legacy rows, inserted repair rows, and skipped pairs that already
|
|
66
|
+
had canonical data. Repaired rows can be either triggered invocations or
|
|
67
|
+
contextual misses reconstructed from transcript reads of `SKILL.md`.
|
|
68
|
+
|
|
69
|
+
## Common Patterns
|
|
70
|
+
|
|
71
|
+
**Preview repair scope**
|
|
72
|
+
|
|
73
|
+
> Run `selftune repair-skill-usage --dry-run` to see how many sessions and records would be repaired.
|
|
74
|
+
|
|
75
|
+
**Repair only recent sessions**
|
|
76
|
+
|
|
77
|
+
> Run `selftune repair-skill-usage --since 2025-01-01` to limit the repair to sessions from this year.
|
|
78
|
+
|
|
79
|
+
**Full rebuild from scratch**
|
|
80
|
+
|
|
81
|
+
> Run `selftune repair-skill-usage` without `--since` to rebuild the entire overlay from all available transcripts.
|
|
82
|
+
> This also repairs SQLite-backed skill invocations for legacy-only historical rows.
|
|
83
|
+
|
|
84
|
+
**Agent chaining into evolution**
|
|
85
|
+
|
|
86
|
+
> Run repair before evolution to ensure skill usage data is accurate: `selftune repair-skill-usage && selftune evolve --skill my-skill`.
|
|
87
|
+
|
|
88
|
+
## Troubleshooting
|
|
89
|
+
|
|
90
|
+
| Symptom | Cause | Fix |
|
|
91
|
+
| --- | --- | --- |
|
|
92
|
+
| `repaired_records: 0` | No Skill tool invocations found in transcripts | Verify skills are being invoked in sessions; check `selftune status` |
|
|
93
|
+
| Many `(repaired:name)` paths | Skill not installed or path not discoverable | Install the skill or provide a `--skill-log` with known paths |
|
|
94
|
+
| `transcripts_scanned: 0` | No transcripts in projects directory | Verify `~/.claude/projects/` contains session directories |
|
|
95
|
+
| Invalid `--since` date error | Date format not parseable | Use ISO format: `--since 2025-01-15` |
|
package/skill/Workflows/Sync.md
CHANGED
|
@@ -1,15 +1,18 @@
|
|
|
1
1
|
# selftune Sync Workflow
|
|
2
2
|
|
|
3
|
-
Refresh source-truth telemetry across supported agent CLIs, then
|
|
4
|
-
repaired skill-usage
|
|
5
|
-
from real transcripts/rollouts instead of stale hook data.
|
|
3
|
+
Refresh source-truth telemetry across supported agent CLIs into SQLite, then
|
|
4
|
+
rebuild the repaired skill-usage layer so status, dashboard, grading, and
|
|
5
|
+
evolution work from real transcripts/rollouts instead of stale hook data. The
|
|
6
|
+
repair phase updates canonical SQLite skill invocations for legacy historical
|
|
7
|
+
rows, reconstructs contextual misses from transcript `SKILL.md` reads, and
|
|
8
|
+
also writes the compatibility repaired overlay JSONL.
|
|
6
9
|
|
|
7
10
|
## When to Use
|
|
8
11
|
|
|
9
12
|
- Before running `status`, `dashboard`, `watch`, or `evolve` when data may be stale
|
|
10
13
|
- The user has run many Claude Code, Codex, OpenCode, or OpenClaw sessions since last sync
|
|
11
14
|
- The agent detects host logs may be polluted and needs the repaired/source-first view
|
|
12
|
-
- Before
|
|
15
|
+
- Before inspecting alpha-upload readiness or pushing fresh cloud data
|
|
13
16
|
|
|
14
17
|
## Default Command
|
|
15
18
|
|
|
@@ -22,24 +25,25 @@ selftune sync
|
|
|
22
25
|
| Flag | Description |
|
|
23
26
|
| ---------------- | ----------------------------------------------- |
|
|
24
27
|
| `--since <date>` | Only sync sessions modified on/after this date |
|
|
25
|
-
| `--dry-run` | Show summary without writing files
|
|
28
|
+
| `--dry-run` | Show summary without writing to SQLite or files |
|
|
26
29
|
| `--force` | Ignore per-source markers and rescan everything |
|
|
27
30
|
| `--no-claude` | Skip Claude transcript replay |
|
|
28
31
|
| `--no-codex` | Skip Codex rollout ingest |
|
|
29
32
|
| `--no-opencode` | Skip OpenCode ingest |
|
|
30
33
|
| `--no-openclaw` | Skip OpenClaw ingest |
|
|
31
|
-
| `--no-repair` | Skip rebuilding
|
|
34
|
+
| `--no-repair` | Skip rebuilding repaired skill-usage data |
|
|
32
35
|
| `--json` | Output results as JSON |
|
|
33
36
|
|
|
34
37
|
## Output
|
|
35
38
|
|
|
36
39
|
Writes/refreshed data:
|
|
37
40
|
|
|
38
|
-
- `~/.
|
|
39
|
-
-
|
|
40
|
-
- `~/.claude/
|
|
41
|
-
-
|
|
41
|
+
- SQLite operational tables in `~/.selftune/selftune.db`
|
|
42
|
+
- canonical SQLite `skill_invocations` repair rows / legacy-row cleanup
|
|
43
|
+
- `~/.claude/skill_usage_repaired.jsonl` compatibility/export overlay
|
|
44
|
+
- local creator-directed contribution staging rows for approved skills
|
|
42
45
|
- per-source marker files
|
|
46
|
+
- alpha upload queue/staging activity when alpha is enrolled
|
|
43
47
|
|
|
44
48
|
## Steps
|
|
45
49
|
|
|
@@ -53,8 +57,10 @@ counts. Report the preview summary to the user.
|
|
|
53
57
|
Run `selftune sync`. The output includes:
|
|
54
58
|
|
|
55
59
|
- Per-source `scanned`, `synced`, and `skipped` counts
|
|
56
|
-
- Repaired
|
|
60
|
+
- Repaired skill-usage totals
|
|
61
|
+
- Creator-directed contribution staging totals when approved skills are installed
|
|
57
62
|
- Any errors or warnings
|
|
63
|
+
- Alpha upload summary when cloud upload is enabled
|
|
58
64
|
|
|
59
65
|
### 3. Verify Results
|
|
60
66
|
|
|
@@ -67,7 +73,7 @@ sync reports source/hook failures or expected active sources are missing.
|
|
|
67
73
|
|
|
68
74
|
After sync completes, proceed with the user's intended workflow:
|
|
69
75
|
`selftune status`, `selftune dashboard`, `selftune watch --sync-first`,
|
|
70
|
-
|
|
76
|
+
`selftune evolve --sync-first`, or `selftune alpha upload` for a manual upload.
|
|
71
77
|
|
|
72
78
|
## `--json` Usage
|
|
73
79
|
|