xtrm-tools 0.7.12 → 0.7.14

package/.xtrm/skills/default/update-specialists/SKILL.md

@@ -1,256 +1,154 @@
 ---
 name: update-specialists
 description: >
-  Reconcile a project with current canonical specialists install state.
-  Use this skill when a user says "update specialists", "specialists is broken",
-  "sp is out of date", "hooks not firing", "skills not loading after update",
-  or when drift is detected in installed specialists config, hooks, jobs, DB,
-  extensions, or worktree cleanup.
-version: 1.0
-synced_at: 00000000
+  Reconcile all xtrm-managed asset drift across repos.
+  Use this skill when user says "update specialists", "xtrm drift", "assets out of date",
+  or when operator needs guided refresh across one repo or many.
+version: 2.0
+synced_at: 2026-05-05
 ---
 
 # update-specialists
 
-Bring specialists install back to canonical state. Detect drift, apply targeted
-fixes, then verify with `sp doctor`.
+Interactive wrapper over `xt update` for xtrm-managed asset drift.
 
-## Canonical State
+Canonical-live model:
+- **Category A**: specialist runtime / loader-live surfaces. No refresh needed; verify only.
+- **Category B**: xtrm-managed snapshots under repos (`.xtrm/skills/default/`, `.xtrm/hooks/default/`, and related managed assets). These can drift and need operator-confirmed refresh.
 
-Check each item explicitly. This is what a healthy specialists-initialized project
-looks like.
+Skill goal:
+1. find projects root,
+2. inspect drift,
+3. summarize per-repo state,
+4. ask operator which repos to refresh,
+5. run `xt update --apply`,
+6. re-check,
+7. report final state.
 
-### Specialists configs
+No automatic execution. Always operator-confirmed.
 
-| Check | Expected value |
-|-------|----------------|
-| `.specialists/default/*.specialist.json` | JSON-first specialist configs present |
-| `metadata.name` | Matches filename stem |
-| `metadata.version` | Valid semver string |
-| `metadata.description` | Present |
-| `metadata.category` | Present |
-| `execution.model` | Present and pingable |
-| `execution.fallback_model` | Present, different provider from primary |
-| `execution.permission_required` | Valid enum |
-| `execution.extensions.serena` | Present when skill needs opt-out or default true |
-| `execution.extensions.gitnexus` | Present when skill needs opt-out or default true |
-| `execution.interactive` | Matches intended keep-alive behavior |
+## Operator Flow
 
-### Hooks wiring
+### 1) Discover projects root
 
-| Check | Expected value |
-|-------|----------------|
-| `.claude/settings.json` | Has hook entries for active events |
-| Hook events | At minimum: `SessionStart`, `PreToolUse`, `PostToolUse`, `Stop` |
-| Hook paths | Point at specialists runtime hook scripts, not stale xtrm-only paths |
-| Hook format | Matches project's installed settings format and loads cleanly |
+Ask for root if user did not name one.
 
-### CLI reachability
+Default order:
+1. explicit user root,
+2. `~/dev`,
+3. git-discovered repo root / workspace root,
+4. current directory as last fallback.
 
-| Check | Expected value |
-|-------|----------------|
-| `sp` command | On PATH and runs |
-| `specialists` command | On PATH and runs |
-| Version compatibility | `sp doctor` reports matching runtime / install state |
-| Command surface | `sp doctor`, `sp init`, `sp clean`, `sp status` available |
+If multiple candidate roots exist, ask which one to use.
 
-### Jobs and runtime dirs
+### 2) Run doctor
 
-| Check | Expected value |
-|-------|----------------|
-| `.specialists/jobs/` | Exists |
-| `.specialists/ready/` | Exists if used by runtime |
-| `.specialists/default/` | Canonical install copy present |
-| Orphaned worktrees | None under `.worktrees/` |
-| Worktree ownership | No stale entries for deleted jobs |
-
-### SQLite / observability
-
-| Check | Expected value |
-|-------|----------------|
-| specialists DB | Opens cleanly |
-| Schema version | Matches runtime expectation |
-| WAL / busy timeout settings | Present when runtime uses SQLite |
-| Corruption / lock errors | None in `sp doctor` |
-
-### Pi extensions
-
-| Check | Expected value |
-|-------|----------------|
-| `quality-gates` | Registered if project uses quality gates |
-| `pi-gitnexus` | Registered when GitNexus integration is expected |
-| `pi-serena-tools` | Registered when Serena integration is expected |
-| Extension paths | Resolve from installed project, not stale workspace copies |
-
-## Detection
-
-Run these in order. Report which checks pass and which drift.
+Use:
 
 ```bash
-# 1. Primary health check
-sp doctor
-
-# 2. Runtime status
-sp status
-
-# 3. Config shape
-find .specialists/default -maxdepth 1 -name '*.specialist.json' -print
-
-# 4. Validate specialist JSON files
-node -e "const fs=require('fs'); const path=require('path'); const dir='.specialists/default'; for (const file of fs.readdirSync(dir)) { if (!file.endsWith('.specialist.json')) continue; const data=JSON.parse(fs.readFileSync(path.join(dir,file),'utf8')); const s=data.specialist||data; const m=s.metadata||{}; const e=s.execution||{}; const missing=[]; for (const key of ['name','version','description','category']) if (!m[key]) missing.push('metadata.'+key); for (const key of ['model','fallback_model','permission_required']) if (!e[key]) missing.push('execution.'+key); if (missing.length) console.log(file+': MISSING '+missing.join(', ')); if (m.name && m.name !== file.replace(/\.specialist\.json$/, '')) console.log(file+': NAME MISMATCH '+m.name); }"
-
-# 5. Hooks wiring
-node -e "const fs=require('fs'); const p='.claude/settings.json'; if (fs.existsSync(p)) { const s=JSON.parse(fs.readFileSync(p,'utf8')); console.log(JSON.stringify(s.hooks ?? s, null, 2)); } else { console.log('MISSING .claude/settings.json'); }"
-
-# 6. Command availability
-command -v sp
-command -v specialists
-sp doctor --json 2>/dev/null || true
-
-# 7. Jobs and worktrees
-ls -1 .specialists/jobs 2>/dev/null || true
-find .worktrees -maxdepth 2 -mindepth 1 -type d 2>/dev/null || true
-
-# 8. Extension registration
-node -e "const fs=require('fs'); const p='.pi/settings.json'; if (fs.existsSync(p)) console.log(JSON.stringify(JSON.parse(fs.readFileSync(p,'utf8')).skills ?? JSON.parse(fs.readFileSync(p,'utf8')).extensions ?? {}, null, 2)); else console.log('MISSING .pi/settings.json')"
+xt doctor --cwd <root> --json
 ```
 
-## Drift -> Fix Mapping
+If `xt` is unavailable, stop and switch to fallback guidance below.
 
-Use targeted fixes first. Escalate to full sync only if needed.
+### 3) Summarize drift
 
-| Drift | Fix |
-|-------|-----|
-| Specialist JSON missing required fields | `sp edit <name> ...` or regenerate via `sp init --sync-skills` |
-| Specialist JSON schema mismatch | `sp init --sync-skills` |
-| Hooks missing or stale | `sp init --sync-hooks` if available, otherwise `sp init --sync-skills` or `sp init -y` |
-| `sp` / `specialists` missing from PATH | Reinstall / re-bootstrap specialists runtime |
-| Job dir missing | `sp init -y` |
-| Orphaned `.worktrees/` entries | `specialists clean` |
-| SQLite schema/version mismatch | `sp doctor` first, then `sp init --sync-skills` or runtime migration command |
-| Pi extensions missing | `sp init --sync-skills` or reinstall extension registration |
-| Hook config format stale | `sp init -y` |
-| Unknown manual drift | Stop, inspect, then apply user-approved fix |
+Render clean table grouped by repo:
+- repo path
+- status
+- drift count
+- missing / extra / mismatched assets
+- suggested action
 
-## Remediation
+Keep focus on operator action, not internal diagnostics.
 
-### Fix: Specialist configs drifted
+### 4) Ask for confirm
 
-If `sp doctor` or JSON validation shows missing fields, wrong names, or schema
-mismatch:
+Offer three paths:
+- refresh all repos,
+- refresh specific repos,
+- dry-run only.
 
-```bash
-sp init --sync-skills
-```
+If user names one repo, keep flow narrow and confirm only that repo.
 
-If one specialist needs a small repair and `sp edit` supports it, prefer that over
-full sync.
+### 5) Apply refresh
 
-### Fix: Hooks not firing
-
-If hooks are missing, wrong events, or stale script paths:
+Use:
 
 ```bash
-sp init -y
+xt update --apply --root <root>
 ```
 
-If runtime exposes a narrower hook sync command, prefer it. Use full init only
-when hook-only sync is not enough.
-
-### Fix: CLI not reachable
-
-If `sp` or `specialists` is missing or incompatible:
+Or for one repo:
 
 ```bash
-sp doctor
+xt update --apply --repo <repo>
 ```
 
-If doctor confirms install drift, reinstall or re-bootstrap specialists runtime.
-Do not guess at file edits when command surface itself is broken.
+For dry-run, omit `--apply`.
 
-### Fix: Job dirs or worktree GC drift
+### 6) Re-run doctor
 
-If jobs exist without owners, worktrees are orphaned, or cleanup state is stale:
+Run same doctor command again after update and confirm clean state.
 
-```bash
-specialists clean
-```
+### 7) Final report
 
-Then re-run `sp doctor`.
+State:
+- what drift existed,
+- what refreshed,
+- what stayed untouched,
+- any residual manual fixes.
 
-### Fix: SQLite schema drift
+## Fallback When xt Missing
 
-If doctor reports DB version mismatch or recovery issue:
+If `xt` / `xtrm` not installed or doctor/update help unavailable:
+- do not block user,
+- switch to per-repo guidance,
+- tell user to run repo-local checks manually,
+- do not invent bulk repair commands.
 
-1. Run `sp doctor` and capture exact schema error.
-2. Apply runtime migration command if available.
-3. If no automated migration exists, flag manual intervention.
+Fallback response shape:
+- identify likely drifted repos,
+- point user at repo-local `sp doctor` / package-specific checks already available in that repo,
+- say bulk refresh needs `xt` installed.
 
-### Fix: Pi extensions not registered
+## Drift Review Rules
 
-If `quality-gates`, `pi-gitnexus`, or `pi-serena-tools` are missing:
+- Treat repo-custom overlays as intentional unless doctor marks them mismatched against managed snapshot.
+- Do not overwrite user-owned layers.
+- Prefer dry-run first when drift touches multiple repos.
+- If only one repo needs refresh, keep output narrow and use single-repo update path.
+- If doctor shows mixed drift across 3 repos, summarize each repo separately and ask which to refresh.
 
-```bash
-sp init --sync-skills
-```
+## Output Shape
 
-If project uses different extension packaging, re-run install step that writes
-`.pi/settings.json`.
-
-## Verification
+Use this order:
+1. root chosen
+2. doctor summary
+3. drift table
+4. confirm prompt
+5. update action
+6. post-update doctor result
+7. final status
 
-After fixes, confirm canonical state restored.
+## Example Operator Loop
 
-```bash
-sp doctor
-sp status
+```text
+Root: ~/dev
+Doctor: 3 repos checked
 
-command -v sp
-command -v specialists
+repo                      status      drift
+repo-a                    drifted     4 assets
+repo-b                    in-sync     0 assets
+repo-c                    drifted     1 asset
 
-node -e "const fs=require('fs'); const p='.claude/settings.json'; const s=JSON.parse(fs.readFileSync(p,'utf8')); console.log(Boolean(s.hooks || Object.keys(s).length))"
+Refresh all / specific repos / dry-run?
 ```
 
-Expected outcome:
-- `sp doctor` clean
-- `sp status` no drift / no repair hints
-- `sp` and `specialists` reachable
-- specialist JSON files valid
-- hooks present on required events
-- no orphaned worktrees
-- SQLite state healthy
-
-## Manual Intervention
-
-Flag these when automatic fix is unsafe or impossible:
-
-- `sp doctor` reports corrupt DB / unreadable SQLite file
-- command surface missing because install itself is broken
-- hook scripts absent from repo and cannot be regenerated
-- schema mismatch with no available migration path
-- worktree cleanup would remove user changes
-- extensions required by project are not installed at package level
-
-When manual intervention needed, report:
-1. exact drift
-2. exact command tried
-3. why auto-fix stopped
-4. next safe operator action
-
-## User Summary Format
-
-After detection + remediation, answer with compact status:
-
-```text
-## specialists update complete
-
-✓ sp doctor clean
-✓ specialist configs valid
-✓ hooks wired
-✓ CLI reachable
-✓ jobs/worktrees clean
-✓ SQLite healthy
-✓ extensions registered
+## Verification
 
-[manual items, if any]
-```
+After refresh:
+- `xt doctor --cwd <root> --json` clean or reduced to intentional custom drift,
+- repo-specific follow-up actions called out only when needed,
+- single-repo case stays single-repo,
+- missing `xt` path falls back cleanly.

package/.xtrm/skills/default/update-xt/SKILL.md

@@ -74,6 +74,31 @@ node -e "const s=require('./.pi/settings.json'); console.log(s.skills)" 2>/dev/n
 for f in .xtrm/skills/active/*; do [ -L "$f" ] || echo "NOT A SYMLINK: $f"; done
 ```
 
+## Implementation Self-Check
+
+Do not trust the surface commands alone. Before claiming that `xt init` handles
+drift correctly, verify the underlying implementation behavior in the CLI source.
+
+Required checks:
+
+| File | What to verify |
+|------|----------------|
+| `cli/src/core/drift.ts` | Drift is classified by comparing installed user file hashes against registry hashes from the package payload |
+| `cli/src/core/registry-scaffold.ts` | Drifted files are reported and skipped by default unless `force` is enabled |
+| `cli/src/commands/init.ts` | `xt init` calls the registry install step with `force: false` |
+
+What you must confirm from code before reporting success:
+
+- `xt init` does check for local drift between the user's `.xtrm` files and the
+  package payload that bootstrapped them.
+- That check is hash-based for registry-managed `.xtrm` files, not just a loose
+  status heuristic.
+- `xt init -y` is non-destructive for drifted `.xtrm` files by default. It
+  preserves local edits unless a separate force path is used.
+
+If the implementation no longer matches those rules, stop and report the mismatch
+instead of repeating this skill's older assumptions.
+
 ## Remediation
 
 Two commands cover almost all drift. Know which fixes what:
@@ -86,6 +111,8 @@ Two commands cover almost all drift. Know which fixes what:
 ### Fix: Skills symlink stale or active/ view wrong
 
 `xt claude install` does NOT rebuild skills. Only `xt init` does (Phase 6b).
+`xt init -y` will repair missing/outdated registry-managed files, but it will
+preserve locally drifted `.xtrm` files by default.
 
 ```bash
 xt init -y
@@ -159,6 +186,13 @@ node -e "const s=require('./.pi/settings.json'); console.log(s.skills.includes('
 # Must output: true
 ```
 
+Also restate the implementation-level conclusion in your report:
+
+- `xt init` verified drift against package registry hashes
+- local drifted `.xtrm` files were preserved by default
+- no forced overwrite path was used unless explicitly requested
+
+
 If `xt status` still shows drift after targeted fixes, run the full sync:
 ```bash
 xt init

package/.xtrm/skills/default/using-kpi/SKILL.md

@@ -0,0 +1,150 @@
+---
+name: using-kpi
+description: >-
+  Analyze specialist KPI data in observability SQLite. Use for runtime, payload,
+  waiting, tool-call, and outlier analysis. Token estimates use cl100k_base-style
+  approximation with ~±5% accuracy.
+gemini-command: using-kpi
+version: 3.1.0
+---
+
+# using-kpi
+
+KPI analysis skill for `sp db stats` / `sp db extract` data.
+
+## Quick rule
+
+`active_runtime_ms` = real paid runtime. Rank by that first. `elapsed_ms` is total wall time. `waiting_ms` catches forgotten keep-alives.
+
+Token counts are approximate, cl100k_base-style, about ±5%. Bytes are exact UTF-8 size.
+
+## Recipe 1 — specialist × model leaderboard by active cost
+
+```bash
+sp db stats --format json \
+  | jq -r '
+      .rows
+      | group_by([.specialist, .model])
+      | map({
+          specialist: .[0].specialist,
+          model: .[0].model,
+          jobs: length,
+          active_ms: (map((.active_runtime_ms // 0)) | add),
+          total_ms: (map((.total_runtime_ms // .elapsed_ms // 0)) | add),
+          turns: (map((.total_turns // 0)) | add),
+          tools: (map((.total_tools // 0)) | add),
+          payload_kb: (map((.payload_kb // 0)) | add)
+        })
+      | sort_by(-.active_ms, -.jobs)
+      | .[]
+      | [ .specialist, .model, .jobs, .active_ms, .total_ms, .turns, .tools, .payload_kb ]
+      | @tsv'
+```
+
+## Recipe 2 — outliers above p95
+
+```bash
+sp db stats --format json \
+  | jq '
+      .rows as $rows
+      | {
+          active: ($rows | map(.active_runtime_ms // 0) | sort),
+          tools: ($rows | map(.total_tools // 0) | sort),
+          turns: ($rows | map(.total_turns // 0) | sort),
+          payload: ($rows | map(.payload_kb // 0) | sort)
+        } as $s
+      | {
+          active_p95: $s.active[(($s.active|length)*95/100|floor)],
+          tools_p95: $s.tools[(($s.tools|length)*95/100|floor)],
+          turns_p95: $s.turns[(($s.turns|length)*95/100|floor)],
+          payload_p95: $s.payload[(($s.payload|length)*95/100|floor)]
+        } as $p
+      | $rows
+      | map(select(
+          ((.active_runtime_ms // 0) >= $p.active_p95) or
+          ((.total_tools // 0) >= $p.tools_p95) or
+          ((.total_turns // 0) >= $p.turns_p95) or
+          ((.payload_kb // 0) >= $p.payload_p95)
+        ))
+      | .[]
+      | [ .job_id, .specialist, .model, .active_runtime_ms, .total_tools, .total_turns, .payload_kb ]
+      | @tsv'
+```
+
+## Recipe 3 — payload bloat ranking
+
+```bash
+sp db stats --with-payload --format json \
+  | jq -r '
+      .rows
+      | group_by(.specialist)
+      | map({
+          specialist: .[0].specialist,
+          jobs: length,
+          avg_payload_kb: ((map((.payload_kb // 0)) | add) / length),
+          max_payload_kb: (map((.payload_kb // 0)) | max)
+        })
+      | sort_by(-.avg_payload_kb)
+      | .[:10]
+      | .[]
+      | [ .specialist, .jobs, (.avg_payload_kb|tostring), (.max_payload_kb|tostring) ]
+      | @tsv'
+```
+
+## Recipe 4 — waiting-state hygiene
+
+```bash
+sp db stats --format json \
+  | jq -r '
+      .rows
+      | map(select((.waiting_s? // 0) != 0))
+      | map(. + {waiting_ratio: ((.waiting_ms // 0) / ((.total_runtime_ms // .elapsed_ms // 1) + 0.0))})
+      | sort_by(-.waiting_ratio, -.waiting_ms)
+      | .[]
+      | [ .job_id, .specialist, .model, (.waiting_ms|tostring), (.total_runtime_ms // .elapsed_ms|tostring), (.waiting_ratio|tostring) ]
+      | @tsv'
+```
+
+## Recipe 5 — tool-call distribution per specialist
+
+```bash
+sp db stats --format json \
+  | jq -r '
+      .rows
+      | group_by(.specialist)
+      | map({
+          specialist: .[0].specialist,
+          counts: (map(.tool_call_counts_json? // "{}")
+            | map(fromjson)
+            | add)
+        })
+      | .[]
+      | .counts
+      | to_entries
+      | sort_by(-.value)
+      | .[]
+      | [ .key, .value ]
+      | @tsv'
+```
+
+## Recipe 6 — payload vs active runtime correlation
+
+```bash
+sp db stats --with-payload --format json \
+  | jq -r '
+      .rows
+      | map(select((.payload_kb? // 0) > 0 and ((.active_runtime_ms? // 0) > 0)))
+      | map([(.payload_kb|tonumber), (.active_runtime_ms|tonumber)])
+      | if length < 2 then empty else
+          (map(.[0]) | add / length) as $mx |
+          (map(.[1]) | add / length) as $my |
+          (map((.[0]-$mx)*(.[1]-$my)) | add) /
+          ((map((.[0]-$mx)^2) | add) * (map((.[1]-$my)^2) | add)) ^ 0.5
+        end'
+```
+
+## References
+
+- `docs/observability-metrics.md`
+- `src/cli/db.ts`
+- `src/specialist/observability-sqlite.ts`