@jaggerxtrm/specialists 3.7.1 → 3.10.0

package/README.md

@@ -47,7 +47,9 @@ specialists merge <bead-id>           # single chain or epic (topological)
 specialists merge <bead-id> --rebuild # rebuild after merge
 ```
 
-`specialists run` prints `[job started: <id>]` early and also writes the id to `.specialists/jobs/latest`.
+`specialists run` prints `[job started: <id>]` early. Normal runtime is DB-backed; `.specialists/jobs/latest` is legacy/operator-only.
+
+Runtime state lives in `observability.db`; `.specialists/jobs/latest` is legacy convenience pointer only.
 
 Ad-hoc work:
 
@@ -77,6 +79,9 @@ specialists doctor
 | Need | Doc |
 |---|---|
 | Install and bootstrap a project | [docs/bootstrap.md](docs/bootstrap.md) |
+| Run a script-class specialist over HTTP (`sp serve`) — overview & contract | [docs/specialists-service.md](docs/specialists-service.md) |
+| Install `sp serve` in another project (sidecar Docker / Podman) | [docs/specialists-service-install.md](docs/specialists-service-install.md) |
+| Build & publish the specialists-service image | [docs/release-image.md](docs/release-image.md) |
 | Bead-first workflow and semantics | [docs/workflow.md](docs/workflow.md) |
 | CLI commands and flags | [docs/cli-reference.md](docs/cli-reference.md) |
 | Background jobs, feed, result, stop | [docs/background-jobs.md](docs/background-jobs.md) |
@@ -90,25 +95,34 @@ specialists doctor
 | Pi subprocess isolation and extensions | [docs/pi-session.md](docs/pi-session.md) |
 | NodeSupervisor architecture, node lifecycle, and `sp node` CLI | [docs/nodes.md](docs/nodes.md) |
 
+## Ownership model
+
+Specialists uses layered ownership with deterministic loader precedence: user layer overrides default layer, and default layer falls back to package source (`.specialists/user/*` > `.specialists/default/*` > `config/*`). Operationally: `config/*` is upstream source shipped by package, `.specialists/default/*` is managed mirror refreshed by `specialists init --sync-defaults` (scope: specialists + mandatory-rules + nodes), `.specialists/user/*` is repo customization layer, and `.specialists/{jobs,ready,db}` is runtime/generated state; `.specialists/jobs/` is legacy mirror/debug surface, not normal-runtime source of truth. Use `sp edit --fork-from <base>` to promote non-user specialist into user layer before editing.
+
 ## Project structure
 
 ```text
 config/
-├── specialists/    canonical specialist definitions (.specialist.yaml)
-├── hooks/          bundled hook scripts
-├── skills/         repo-local skills used by specialists
-└── extensions/     pi extensions (future)
+├── specialists/       canonical specialist definitions (.specialist.json)
+├── mandatory-rules/   canonical rule sets injected into specialist prompts (+ README)
+├── nodes/             canonical node configs
+├── hooks/             bundled hook scripts
+├── skills/            repo-local skills used by specialists
+└── extensions/        pi extensions (future)
 .specialists/
-├── default/        canonical assets (from init)
+├── default/           managed mirror of canonical (from sp init --sync-defaults)
 │   ├── specialists/
+│   ├── mandatory-rules/
+│   ├── nodes/
 │   ├── hooks/
 │   └── skills/
-├── user/           custom additions
+├── user/              repo-owned customizations (overrides default + canonical)
 │   ├── specialists/
 │   ├── hooks/
 │   └── skills/
-├── jobs/           runtime — gitignored
-└── ready/          runtime — gitignored
+├── mandatory-rules/   repo-specific rule overlay (wins on set-id conflict)
+├── jobs/              runtime — gitignored
+└── ready/             runtime — gitignored
 src/                CLI, server, loader, runner, tools
 ```
 

package/config/mandatory-rules/README.md

@@ -0,0 +1,113 @@
+# MANDATORY_RULES
+
+Rule sets injected at the end of every specialist prompt at spawn time. Enforces
+behaviors the model must follow regardless of the specific task.
+
+> **Quick introspection:** `sp list-rules` prints the rule × specialist matrix.
+> Filter with `sp list-rules --rule <id>` or `sp list-rules --specialist <name>`,
+> machine output via `--json`.
+
+## Layout (three tiers)
+
+The loader reads and unions indexes from three paths, in this precedence:
+
+| Tier | Path | Writer | Role |
+|------|------|--------|------|
+| 1. Source | `config/mandatory-rules/` | specialists repo commits | Canonical source of truth. Ships with the tool. |
+| 2. Canonical copy | `.specialists/default/mandatory-rules/` | `sp init --sync-defaults` | Mirror of canonical, placed in every downstream project. |
+| 3. Overlay | `.specialists/mandatory-rules/` | you (per-repo) | Repo-specific additions and overrides. Wins on set-id conflict. |
+
+A rule set defined in tier 3 overrides a same-id rule set from tier 2 or 1,
+letting a repo tailor or replace canonical rules without editing the source.
+
+## What gets injected
+
+At specialist spawn, the runner resolves sets from:
+
+1. `required_template_sets` — always loaded
+2. `default_template_sets` — loaded unless `mandatory_rules.disable_default_globals: true` on the specialist
+3. `specialist.mandatory_rules.template_sets` — per-specialist additions
+4. `specialist.mandatory_rules.inline_rules` — per-specialist inline rules (no file)
+
+The resulting block is appended to the rendered task prompt as
+`## MANDATORY_RULES` with one `### <set-id>` section per set.
+
+## Authoring a rule set
+
+Create `<your-set>.md` in one of the three tiers. File format:
+
+```markdown
+---
+name: <your-set>
+kind: mandatory-rule
+rules:
+  - id: my-rule-1
+    level: required
+    text: "Single-line rule text. Use quotes if it contains colons."
+  - id: my-rule-2
+    level: warn
+    text: "Another rule."
+    when: "optional guard, e.g. 'node > 20'"
+---
+```
+
+- **`id`** — stable identifier, shown in the rendered block. Auto-filled from `<set-id>-<n>` if omitted.
+- **`level`** — `required`, `error`, `warn`, `info`. Display only; the model treats them as priority hints.
+- **`text`** — one-line rule (multi-line supported via YAML `|` block).
+- **`when`** — optional conditional context.
+
+**Shorthand**: a file with only a body (no `rules:` frontmatter) is loaded as
+a single rule with `level: required` and the body as `text`.
+
+## Wiring a set
+
+### Option 1 — via index.json (applies to all specialists)
+
+Add the set id to the index in the tier you're writing to. Tier-3 example:
+
+```json
+// .specialists/mandatory-rules/index.json
+{
+  "required_template_sets": [],
+  "default_template_sets": ["my-repo-rule"]
+}
+```
+
+Index files are union-merged across tiers; dedup by set id.
+
+### Option 2 — per specialist
+
+Reference the set id in the specialist's JSON:
+
+```json
+// config/specialists/<name>.specialist.json
+{
+  "specialist": {
+    "mandatory_rules": {
+      "template_sets": ["my-set"],
+      "inline_rules": [
+        { "id": "xtra-1", "level": "required", "text": "One-off rule inline." }
+      ],
+      "disable_default_globals": false
+    }
+  }
+}
+```
+
+## Keep repo-specific rules out of canonical
+
+Canonical rules (`config/` tier) ship to every downstream project. If a rule
+only applies to this repo (e.g. "use bunx here"), put it in
+`.specialists/mandatory-rules/` (tier 3). Other repos won't see it.
+
+## Budget
+
+The full injection block is capped at ~2000 tokens (`src/specialist/runner.ts`).
+Over-budget: the block is skipped and a warning is logged.
+
+## Debugging
+
+- Loader warnings appear on stderr prefixed `[specialist runner]`.
+- The supervisor emits a `mandatory_rules_injection` meta event on every run
+  with `sets_loaded`, `rules_count`, `inline_rules_count`, `token_estimate`.
+- Inspect any run's injection: `sp ps <job-id>` shows the metadata.

package/config/mandatory-rules/gitnexus-required.md

@@ -2,4 +2,15 @@
 name: gitnexus-required
 kind: mandatory-rule
 ---
-Use GitNexus for symbol changes. Run impact before edit and detect_changes before finish.
+Use GitNexus before editing any function/class/method.
+
+Tools (prefer MCP; fall back to CLI if MCP unavailable):
+- Blast radius before edit: `gitnexus_impact({target, direction:"upstream"})` or `npx gitnexus impact <target>`. STOP and warn if HIGH/CRITICAL.
+- Symbol callers/callees: `gitnexus_context({name})` or `npx gitnexus context <name>`.
+- Concept search: `gitnexus_query({query})` or `npx gitnexus query "<text>"`.
+- Pre-commit scope check: `gitnexus_detect_changes()` (MCP only — fallback: `git diff --stat`).
+
+Rules:
+- Run impact for every symbol you modify; never edit without it.
+- Never rename via find-replace — use `gitnexus_rename({symbol_name, new_name, dry_run:true})` first.
+- If index is stale, ask the user to run `npx gitnexus analyze`.

package/config/mandatory-rules/serena-cheatsheet.md

@@ -0,0 +1,41 @@
+---
+name: serena-cheatsheet
+kind: mandatory-rule
+rules:
+  - id: prefer-serena
+    level: required
+    text: "Prefer Serena tools over read/grep/find/ls for source code (.ts .py .go .rs .js etc.). Native tools fine for .md .json .yaml configs."
+  - id: get_symbols_overview
+    level: required
+    text: "get_symbols_overview <path> — symbol skeleton of a file or dir (~300 tokens vs reading the whole file). Use first to pick what you actually need."
+  - id: find_symbol
+    level: required
+    text: "find_symbol <name_path> — fetch one symbol's body (or just signature). Drop-in replacement for read-then-scan-for-function workflows."
+  - id: find_referencing_symbols
+    level: required
+    text: "find_referencing_symbols <name_path> — who calls or uses this symbol across the codebase."
+  - id: search_for_pattern
+    level: required
+    text: "search_for_pattern <regex> — replaces grep. Returns matches with file:line context."
+  - id: find_file
+    level: required
+    text: "find_file <glob> and list_dir <path> — replace native find and ls."
+  - id: read_file
+    level: required
+    text: "read_file <path> — full or sliced read. Use only when navigation tools are not enough; check get_symbols_overview first."
+  - id: replace_symbol_body
+    level: required
+    text: "replace_symbol_body <name_path> — swap a function or class body in place. Use instead of Edit string-match for symbol-scoped changes (MEDIUM+ permission)."
+  - id: insert_around_symbol
+    level: required
+    text: "insert_before_symbol / insert_after_symbol <name_path> — add adjacent code such as imports or helpers (MEDIUM+ permission)."
+  - id: rename_symbol
+    level: required
+    text: "rename_symbol <name_path> <new_name> — refactor-safe across all references. Use instead of find-and-replace (MEDIUM+ permission)."
+  - id: replace_content
+    level: required
+    text: "replace_content <path> <pattern> <replacement> — line-range or regex edits when no symbol target fits (MEDIUM+ permission)."
+  - id: cost-rule
+    level: info
+    text: "Rule of thumb: read of a 500-line source file ~5000 tokens; find_symbol on one function ~200 tokens (~25x cheaper). Use get_symbols_overview before deciding."
+---

package/config/skills/specialists-creator/SKILL.md

@@ -169,7 +169,7 @@ specialists models  # confirm assignments look balanced
 node config/skills/specialists-creator/scripts/scaffold-specialist.ts config/specialists/my-specialist.specialist.json
 
 # 2. Apply a preset for common model/thinking defaults (optional but preferred)
-sp edit my-specialist --preset standard
+sp edit my-specialist --preset medium
 
 # 3. Set individual fields via dot.path (primary mutation workflow)
 sp edit my-specialist specialist.metadata.name my-specialist
@@ -188,7 +188,7 @@ sp edit my-specialist specialist.prompt.task_template --file .tmp/task-template.
 sp view my-specialist
 
 # 6. Validate schema
-bun skills/specialist-author/scripts/validate-specialist.ts config/specialists/my-specialist.specialist.json
+bun config/skills/specialists-creator/scripts/validate-specialist.ts config/specialists/my-specialist.specialist.json
 ```
 
 ---
@@ -383,8 +383,6 @@ planner — epic result:
 
 `run` accepts either a **file path** (`./scripts/foo.sh`, `~/scripts/foo.sh`) or a **shell command** (`bd ready`, `git status`). Pre-run validation checks that file paths exist and shell commands are on `PATH`. Shebang typos (e.g. `pytho` instead of `python`) are caught and reported as errors before the session starts.
 
-`path` is accepted as a deprecated alias for `run`.
-
 ### `specialist.capabilities` (optional)
 
 Informational declarations used by pre-run validation and future tooling (e.g. `specialists doctor`).
@@ -410,27 +408,6 @@ Informational declarations used by pre-run validation and future tooling (e.g. `
 
 Writes the final session output to this file path after the session completes. Relative to the working directory.
 
-### `specialist.communication` (optional)
-
-```json
-{
-  "communication": {
-    "next_specialists": "planner"
-  }
-}
-```
-
-Or as an array:
-```json
-{
-  "communication": {
-    "next_specialists": ["planner", "test-runner"]
-  }
-}
-```
-
-`next_specialists` declares which specialist(s) should receive this specialist's output as `$previous_result`. Chaining is executed by the caller (e.g. `run_parallel` pipeline) — this field is declarative metadata.
-
 ### `specialist.validation` (optional)
 
 Drives the staleness detection shown in `specialists status` and `specialists list`.
@@ -507,7 +484,7 @@ Files listed under `skills.paths` are read and appended to the system prompt at
 {
   "skills": {
     "paths": [
-      "skills/specialist-author/SKILL.md",
+      ".xtrm/skills/active/specialists-creator/SKILL.md",
       ".claude/agents.md"
     ]
   }
@@ -603,9 +580,6 @@ Scripts run **locally** (not inside the agent session):
       "required_tools": ["bash", "read"],
       "external_commands": ["git"]
     },
-    "communication": {
-      "next_specialists": ["sync-docs"]
-    },
     "output_file": ".specialists/review.md",
     "beads_integration": "auto"
   }
@@ -708,7 +682,7 @@ pi --model <provider>/<fallback-model-id> --print "ping"   # must return "pong"
 node config/skills/specialists-creator/scripts/scaffold-specialist.ts config/specialists/my-specialist.specialist.json
 
 # 3. Mutate with sp edit (dot.path + presets)
-sp edit my-specialist --preset standard
+sp edit my-specialist --preset medium
 sp edit my-specialist specialist.execution.model <provider>/<primary-model-id>
 sp edit my-specialist specialist.execution.fallback_model <provider>/<fallback-model-id>
 
@@ -720,7 +694,7 @@ sp edit my-specialist specialist.prompt.task_template --file .tmp/task-template.
 sp view my-specialist
 
 # 6. Validate schema with the bundled helper
-bun skills/specialist-author/scripts/validate-specialist.ts config/specialists/my-specialist.specialist.json
+bun config/skills/specialists-creator/scripts/validate-specialist.ts config/specialists/my-specialist.specialist.json
 
 # 7. List to confirm discovery
 specialists list
@@ -729,4 +703,4 @@ specialists list
 specialists run my-specialist --prompt "ping" --no-beads
 ```
 
-If you need the underlying implementation, read `skills/specialist-author/scripts/validate-specialist.ts`. It is a thin Bun/TypeScript wrapper over `parseSpecialist()` from `src/specialist/schema.ts`, which keeps the helper cross-platform for Windows, macOS, and Linux.
+If you need the underlying implementation, read `config/skills/specialists-creator/scripts/validate-specialist.ts`. It is a thin Bun/TypeScript wrapper over `parseSpecialist()` from `src/specialist/schema.ts`, which keeps the helper cross-platform for Windows, macOS, and Linux.

package/config/skills/specialists-creator/scripts/audit-spec-uniformity.mjs

@@ -0,0 +1,86 @@
+// Audit every .specialist.json under config/specialists/ and .specialists/default/
+// for: (1) schema parse failures, (2) unknown keys that survive .passthrough() silently.
+//
+// Usage (from repo root): bun config/skills/specialists-creator/scripts/audit-spec-uniformity.mjs
+//
+// Keep KNOWN sets in sync with src/specialist/schema.ts. If a sub-schema gains
+// or drops a field, update this file in the same commit.
+
+import { readFileSync, readdirSync } from 'node:fs';
+import { join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const here = fileURLToPath(import.meta.url);
+const repoRoot = resolve(here, '../../../../..');
+const { validateSpecialist } = await import(resolve(repoRoot, 'src/specialist/schema.ts'));
+
+// Walk known schema keys to detect "unknown" passthrough survivors
+const KNOWN = {
+  root: new Set(['specialist']),
+  specialist: new Set(['metadata','execution','prompt','skills','capabilities','communication','validation','beads_integration','beads_write_notes','stall_detection','heartbeat','mandatory_rules','output_file']),
+  metadata: new Set(['name','version','description','category','author','created','updated','tags']),
+  execution: new Set(['mode','model','fallback_model','timeout_ms','stall_timeout_ms','max_retries','interactive','response_format','output_type','permission_required','requires_worktree','thinking_level','auto_commit','extensions','preferred_profile','approval_mode']),
+  'execution.extensions': new Set(['serena','gitnexus']),
+  prompt: new Set(['system','task_template','normalize_template','output_schema','examples','skill_inherit']),
+  skills: new Set(['paths','scripts']),
+  'skills.scripts.item': new Set(['run','path','phase','inject_output']),
+  capabilities: new Set(['required_tools','external_commands','diagnostic_scripts']),
+  communication: new Set(['next_specialists','publishes']),
+  validation: new Set(['files_to_watch','stale_threshold_days']),
+  stall_detection: new Set(['running_idle_warn_ms','running_idle_kill_ms','waiting_stale_ms','tool_duration_warn_ms']),
+};
+
+function unknownKeys(obj, knownSet, path) {
+  const out = [];
+  for (const k of Object.keys(obj || {})) if (!knownSet.has(k)) out.push(`${path}.${k}`);
+  return out;
+}
+
+function audit(file) {
+  const raw = JSON.parse(readFileSync(file,'utf8'));
+  const findings = [];
+  // raw key check (before parse strips/preserves)
+  findings.push(...unknownKeys(raw, KNOWN.root, ''));
+  const s = raw.specialist ?? {};
+  findings.push(...unknownKeys(s, KNOWN.specialist, 'specialist'));
+  if (s.metadata) findings.push(...unknownKeys(s.metadata, KNOWN.metadata, 'specialist.metadata'));
+  if (s.execution) findings.push(...unknownKeys(s.execution, KNOWN.execution, 'specialist.execution'));
+  if (s.execution?.extensions) findings.push(...unknownKeys(s.execution.extensions, KNOWN['execution.extensions'], 'specialist.execution.extensions'));
+  if (s.prompt) findings.push(...unknownKeys(s.prompt, KNOWN.prompt, 'specialist.prompt'));
+  if (s.skills) findings.push(...unknownKeys(s.skills, KNOWN.skills, 'specialist.skills'));
+  if (Array.isArray(s.skills?.scripts)) for (const [i, sc] of s.skills.scripts.entries()) findings.push(...unknownKeys(sc, KNOWN['skills.scripts.item'], `specialist.skills.scripts[${i}]`));
+  if (s.capabilities) findings.push(...unknownKeys(s.capabilities, KNOWN.capabilities, 'specialist.capabilities'));
+  if (s.communication) findings.push(...unknownKeys(s.communication, KNOWN.communication, 'specialist.communication'));
+  if (s.validation) findings.push(...unknownKeys(s.validation, KNOWN.validation, 'specialist.validation'));
+  if (s.stall_detection) findings.push(...unknownKeys(s.stall_detection, KNOWN.stall_detection, 'specialist.stall_detection'));
+  return findings;
+}
+
+const files = [
+  ...readdirSync(resolve(repoRoot,'config/specialists')).filter(f=>f.endsWith('.specialist.json')).map(f=>join(repoRoot,'config/specialists',f)),
+  ...readdirSync(resolve(repoRoot,'.specialists/default')).filter(f=>f.endsWith('.specialist.json')).map(f=>join(repoRoot,'.specialists/default',f)),
+];
+
+let totalUnknown = 0;
+let parseErrors = 0;
+for (const file of files) {
+  try {
+    const v = await validateSpecialist(readFileSync(file,'utf8'));
+    if (!v.valid) {
+      parseErrors++;
+      console.log(`\n✗ PARSE FAIL ${file}`);
+      for (const e of v.errors) console.log(`    ${e.path}: ${e.message}`);
+      continue;
+    }
+    const unk = audit(file);
+    if (unk.length) {
+      totalUnknown += unk.length;
+      console.log(`\n⚠ ${file}`);
+      for (const k of unk) console.log(`    unknown key: ${k}`);
+    }
+  } catch (e) {
+    parseErrors++;
+    console.log(`\n✗ ERROR ${file}: ${e.message}`);
+  }
+}
+console.log(`\n=== ${files.length} specs · ${parseErrors} parse errors · ${totalUnknown} unknown keys ===`);

package/config/skills/specialists-creator/scripts/scaffold-specialist.ts

@@ -3,12 +3,7 @@ import { join } from "node:path";
 import * as z from "zod";
 import { SpecialistSchema } from "../../../../src/specialist/schema.ts";
 
-const DEAD_FIELDS = new Set([
-  "preferred_profile",
-  "approval_mode",
-  "normalize_template",
-  "heartbeat",
-]);
+const DEAD_FIELDS = new Set<string>([]);
 
 interface AddedField {
   path: string;

package/config/skills/update-specialists/SKILL.md

@@ -6,8 +6,8 @@ description: >
   "sp is out of date", "hooks not firing", "skills not loading after update",
   or when drift is detected in installed specialists config, hooks, jobs, DB,
   extensions, or worktree cleanup.
-version: 1.1
-synced_at: 00000000
+version: 1.3
+synced_at: 2026-04-25
 ---
 
 # update-specialists
@@ -15,9 +15,17 @@ synced_at: 00000000
 Bring specialists install back to canonical state. Detect drift, apply targeted
 fixes, then verify with `sp doctor`. Treat canonical state as both:
 1. healthy repo wiring and runtime behavior, and
-2. parity with the currently installed `@jaggerxtrm/specialists` package version
+2. parity with currently installed `@jaggerxtrm/specialists` package version
    when package-level comparison is available.
 
+Ownership contract during repair:
+- upstream source: package `config/*` (read-only for repo operators)
+- managed mirror: `.specialists/default/*` (refresh via `sp init --sync-defaults`; sync scope = specialists + mandatory-rules + nodes; no hand edits)
+- repo custom layer: `.specialists/user/*` + `config/nodes/*` + `.specialists/mandatory-rules/*` (rule overlay, wins on set-id conflict; NOT drift — do not overwrite or flag)
+- runtime/generated: `.specialists/{jobs,ready,db}`
+
+Isolation rule: backlog-clean surfaces out of scope for this skill.
+
 ## Canonical State
 
 Check each item explicitly. This is what a healthy specialists-initialized project
@@ -82,10 +90,14 @@ looks like.
 
 | Check | Expected value |
 |-------|----------------|
-| specialists DB | Opens cleanly |
-| Schema version | Matches runtime expectation |
+| specialists DB | Opens cleanly (`.specialists/db/observability.db`) |
+| Schema version | Matches runtime expectation (current: v11) |
+| `specialist_job_metrics` table | Present at v11+ — holds aggregated per-job metrics |
 | WAL / busy timeout settings | Present when runtime uses SQLite |
 | Corruption / lock errors | None in `sp doctor` |
+| Pre-prune extract | `sp db prune --apply` extracts metrics to `specialist_job_metrics` before deleting events |
+| Extract backfill | `sp db extract --all-missing` populates metrics for jobs whose events still exist |
+| Historical stats query | `sp db stats [--spec <name>] [--model <glob>] [--since <dur>]` reads the aggregated table |
 
 ### Skills + extensions parity
 
@@ -99,6 +111,21 @@ looks like.
 | `pi-serena-tools` | Registered when Serena integration is expected |
 | Extension paths | Resolve from installed project, not stale workspace copies |
 
+### Mandatory-rules template parity (three-tier)
+
+Loader unions indexes from three paths and probes set files in reverse precedence
+(overlay wins on set-id conflict). Full authoring guide:
+`config/mandatory-rules/README.md`.
+
+| Check | Expected value |
+|-------|----------------|
+| `.specialists/default/mandatory-rules/*` | Mirrors canonical package templates after `sp init --sync-defaults` (managed mirror, no hand edits) |
+| `.specialists/mandatory-rules/*` | Repo-specific overlay (user-maintained). Present when repo ships its own rules. NOT drift. |
+| Template frontmatter | YAML frontmatter present and parseable |
+| `specialist.mandatory_rules.template_sets` references | Resolve in order: `.specialists/mandatory-rules/` → `.specialists/default/mandatory-rules/` → `config/mandatory-rules/` |
+| Index files (`index.json`) | Any of the three tiers may define `required_template_sets` / `default_template_sets`; loader unions + dedups |
+| Prompt injection behavior | Runner appends resolved `MANDATORY_RULES` block at end of prompt; supervisor emits `mandatory_rules_injection` meta event |
+
 ## Detection
 
 Run these in order. Report which checks pass and which drift.
@@ -145,6 +172,7 @@ node -e "const fs=require('fs'); const p='.claude/settings.json'; if (fs.existsS
 command -v sp
 command -v specialists
 specialists init --help | sed -n '1,120p'
+specialists edit --help | sed -n '1,120p' | grep -E -- '--fork-from|fork-from' || true
 sp doctor --json 2>/dev/null || true
 
 # 11. Jobs and worktrees
@@ -154,7 +182,15 @@ find .worktrees -maxdepth 2 -mindepth 1 -type d 2>/dev/null || true
 # 12. Extension registration
 node -e "const fs=require('fs'); const p='.pi/settings.json'; if (fs.existsSync(p)) console.log(JSON.stringify(JSON.parse(fs.readFileSync(p,'utf8')).skills ?? JSON.parse(fs.readFileSync(p,'utf8')).extensions ?? {}, null, 2)); else console.log('MISSING .pi/settings.json')"
 
-# 13. Shipped skill frontmatter parity
+# 13a. Observability schema + metrics coverage
+node -e "const {Database} = require('bun:sqlite'); const p='.specialists/db/observability.db'; const fs=require('fs'); if (!fs.existsSync(p)) { console.log('NO_DB'); process.exit(0); } const db=new Database(p,{readonly:true}); const v=db.query(\"SELECT value FROM schema_meta WHERE key='version'\").get(); const has=db.query(\"SELECT name FROM sqlite_master WHERE type='table' AND name='specialist_job_metrics'\").get(); const jobs=db.query('SELECT COUNT(*) c FROM specialist_jobs').get(); const metrics=has ? db.query('SELECT COUNT(*) c FROM specialist_job_metrics').get() : null; console.log(JSON.stringify({schema_version: v?.value, has_metrics_table: !!has, jobs: jobs.c, metrics_rows: metrics?.c ?? 0, metrics_coverage: metrics ? (metrics.c/jobs.c).toFixed(2) : null}, null, 2));" 2>/dev/null || echo "REQUIRES_BUN_RUNTIME"
+
+# 13. Mandatory-rules template tiers + reference checks (three-tier resolution)
+find .specialists/default/mandatory-rules -maxdepth 1 -type f 2>/dev/null || true
+find .specialists/mandatory-rules -maxdepth 1 -type f 2>/dev/null || true
+node -e "const fs=require('fs'); const path=require('path'); const roots=['.specialists/default/specialists','.specialists/user/specialists']; const missing=[]; for (const root of roots) { if (!fs.existsSync(root)) continue; for (const file of fs.readdirSync(root)) { if (!file.endsWith('.specialist.json')) continue; const spec=JSON.parse(fs.readFileSync(path.join(root,file),'utf8')); const sets=spec.specialist?.mandatory_rules?.template_sets ?? []; for (const set of sets) { const candidates=[path.join('.specialists/mandatory-rules',set+'.md'), path.join('.specialists/default/mandatory-rules',set+'.md'), path.join('config/mandatory-rules',set+'.md')]; if (!candidates.some((p)=>fs.existsSync(p))) missing.push(file+': missing template set '+set); } } } if (missing.length) console.log(missing.join('\n'));"
+
+# 14. Shipped skill frontmatter parity
 node -e "const fs=require('fs'); const path=require('path'); const dir='.xtrm/skills/default'; if (!fs.existsSync(dir)) process.exit(0); for (const name of fs.readdirSync(dir)) { const p=path.join(dir,name,'SKILL.md'); if (!fs.existsSync(p)) continue; const head=fs.readFileSync(p,'utf8').split('---')[1] || ''; const version=(head.match(/version:\s*([^\n]+)/)||[])[1]; const synced=(head.match(/synced_at:\s*([^\n]+)/)||[])[1]; console.log(name+': version='+(version||'missing')+' synced_at='+(synced||'missing')); }"
 ```
 
@@ -167,7 +203,8 @@ Use targeted fixes first. Escalate to full sync only if needed.
 | Installed package version mismatch | reinstall / upgrade `@jaggerxtrm/specialists`, then re-run checks |
 | CLI version mismatch vs package | reinstall runtime so `sp` / `specialists` align with installed package |
 | Specialist JSON missing required fields | `sp edit <name> ...` or regenerate via `specialists init --sync-defaults` |
-| Specialist JSON schema mismatch | `specialists init --sync-defaults` |
+| Need user-layer override from default/package specialist | `sp edit <name> --fork-from <base>` to materialize editable copy in `.specialists/user/` |
+| Specialist JSON schema mismatch | `specialists init --sync-defaults` (refreshes specialists + mandatory-rules + nodes) |
 | Installed specialist default differs from canonical package copy | `specialists init --sync-defaults` unless local customization is intentional |
 | Hooks missing or stale | `specialists init` |
 | Installed hook file differs from canonical package copy | `specialists init` unless local customization is intentional |
@@ -175,11 +212,18 @@ Use targeted fixes first. Escalate to full sync only if needed.
 | Job dir missing | `specialists init` |
 | Orphaned `.worktrees/` entries | `specialists clean` |
 | SQLite schema/version mismatch | `sp doctor` first, then `specialists init --sync-defaults` or runtime migration command |
+| Schema below v11 (no `specialist_job_metrics`) | Reinstall / upgrade runtime; table is created by initSchema / migrateToV11. No data loss — raw events untouched. |
+| Events about to be pruned but never aggregated | `sp db extract --all-missing` BEFORE `sp db prune --apply`. Prune refuses when extract fails (safe by design). |
+| Emergency: need to prune but extract is wedged | `sp db prune --apply --skip-extract` — raw events deleted without aggregation. Use only when data loss is acceptable. |
+| Historical per-job stats needed | `sp db stats` reads `specialist_job_metrics`. Replaces ad-hoc `status.json` scans. Supports `--format json\|table`. |
 | Pi extensions missing | `specialists init --sync-skills` or reinstall extension registration |
 | Hook config format stale | `specialists init` |
 | Skill symlink / active-skill drift | `specialists init --sync-skills` |
 | Installed default skill differs from canonical package copy | `specialists init --sync-skills` unless local customization is intentional |
 | Skill frontmatter version / synced_at drift | `specialists init --sync-skills` or refresh packaged skills |
+| Mandatory-rules mirror drift (`.specialists/default/mandatory-rules`) | `specialists init --sync-defaults` |
+| `.specialists/mandatory-rules/` overlay present | Leave alone — this is repo overlay, NOT drift |
+| Missing/invalid `template_sets` references | Check all three tiers first; `sp edit <name> --fork-from <base>` then fix references, or sync defaults if mirror missing, or add set to overlay if intended |
 | Unknown manual drift | Stop, inspect, then apply user-approved fix |
 
 ## Remediation
@@ -206,8 +250,14 @@ shows missing fields, wrong names, or schema mismatch:
 specialists init --sync-defaults
 ```
 
+`--sync-defaults` refreshes specialists + mandatory-rules + nodes mirrors.
+
 If one specialist needs a small repair and `sp edit` supports it, prefer that over
-full sync.
+full sync. If target specialist lives in default/package layer, fork first:
+
+```bash
+sp edit <name> --fork-from <base>
+```
 
 ### Fix: Hooks not firing
 
@@ -250,6 +300,31 @@ If doctor reports DB version mismatch or recovery issue:
 2. Apply runtime migration command if available.
 3. If no automated migration exists, flag manual intervention.
 
+### Fix: metrics aggregation missing or stale
+
+Schema v11 introduced `specialist_job_metrics` (aggregated per-job stats). If you see low `metrics_coverage` in the detection output, or want historical stats before running `sp db prune`:
+
+```bash
+# Backfill metrics for any job whose events still exist but lack a metrics row.
+sp db extract --all-missing
+
+# Inspect specific job metrics.
+sp db extract --job <job-id>
+
+# Query aggregates.
+sp db stats
+sp db stats --spec executor --since 7d --format json
+sp db stats --model 'openai-codex/*' --since 30d
+```
+
+`sp db prune --apply` automatically extracts for every job whose events will be deleted (unless `--skip-extract`). If extract throws, prune aborts — investigate the failing job instead of bypassing.
+
+Safe order before a retention cleanup:
+1. `sp db extract --all-missing` — verify no extract errors.
+2. `sp db prune --before 30d --dry-run` — confirm scope.
+3. `sp db prune --before 30d --apply` — prune with pre-extract built in.
+4. `sp db vacuum` — compact file size.
+
 ### Fix: Skills/defaults differ from shipped package copy
 
 If diff against the installed package shows `.specialists/default/`,

package/config/skills/using-specialists/SKILL.md

@@ -62,6 +62,17 @@ Specialists are autonomous AI agents that run independently — fresh context, d
 8. **No destructive operations by specialists.** No `rm -rf`, no force pushes, no database drops, no credential rotation, no mass deletes, no history rewrites. Surface destructive requirements to the user.
 9. **Executor does not run tests.** Executor runs lint + tsc only. Tests are the reviewer's and test-runner's responsibility in the chained pipeline.
 10. **Keep specialists alive through the review cycle.** Never `sp stop` an executor or debugger before the reviewer delivers its verdict. The specialist stays in `waiting` so you can `resume` it — to commit changes, apply fixes from reviewer feedback, or continue work. Only stop after final reviewer PASS and confirmed commit.
+11. **Respect ownership layers and loader precedence.** Loader resolution order is `.specialists/user/*` > `.specialists/default/*` > package fallback `config/*`. Upstream source = package `config/*` (read-only for repo operators); managed mirror = `.specialists/default/*` (no hand edits); repo custom layer = `.specialists/user/*`; runtime/generated = `.specialists/{jobs,ready,db}`.
+12. **Keep backlog-clean isolated.** Do not mix backlog-clean changes into specialist ownership/migration tasks.
+
+## Mandatory-rules template sets
+
+Use template-driven mandatory rules for repeatable policy bundles.
+
+- Specialist config field: `specialist.mandatory_rules.template_sets`
+- Template source: `config/mandatory-rules/*.md`
+- Template format: YAML frontmatter + body content
+- Runtime behavior: runner resolves templates and injects rendered rules at end of prompt
 
 ---
 
@@ -127,11 +138,13 @@ specialists stop <job-id> --force             # 5s SIGTERM timeout, then pgroup
 
 # Management
 specialists edit <name>                       # edit specialist config (dot-path, --preset)
+specialists edit <name> --fork-from <base>   # fork non-user specialist into .specialists/user/ then edit
 specialists clean                             # purge old job dirs + worktree GC
 specialists clean --processes                 # kill all running/starting specialist jobs
 specialists db vacuum                         # compact SQLite storage (refuses if jobs running)
 specialists db prune --before <iso|duration> --dry-run|--apply  # prune old events/results/terminal jobs
 specialists doctor orphans                    # integrity scan: orphan, stale-pointer, integrity-violation
+specialists init --sync-defaults              # refresh specialists + mandatory-rules + nodes from canonical defaults
 specialists init --sync-skills                # re-sync skills only (no full init)
 specialists init --no-xtrm-check              # skip xtrm prerequisite check (CI/testing)
 ```