@jaggerxtrm/specialists 3.17.0 → 3.18.0

package/docs/specialists-service.md

@@ -0,0 +1,363 @@
+---
+title: Specialists Service
+scope: specialists-service
+category: architecture
+version: 3.1.2
+updated: 2026-06-24
+synced_at: bf6baf7a
+description: Human SSOT for sp serve, sp script, script-class authoring, trust/readiness/hot-reload, and image release.
+source_of_truth_for:
+  - src/cli/script.ts
+  - src/cli/serve.ts
+  - src/specialist/script-runner.ts
+  - Dockerfile
+  - .github/workflows/pi-compat.yml
+domain:
+  - specialists
+  - service
+  - release
+---
+
+<!-- INDEX: auto-generated by validate_doc.py — do not edit manually -->
+| Section | Summary |
+|---|---|
+| [Shape](#shape) | - `sp serve` = HTTP sidecar for script-class specialists |
+| [HTTP contract](#http-contract) | `POST /v1/generate` |
+| [`sp serve`](#sp-serve) | Defaults: |
+| [`sp script`](#sp-script) | One-shot CLI on same runtime path as `sp serve` |
+| [Script-class authoring](#script-class-authoring) | Script-class specialists are JSON specialists with one-shot runtime constraints |
+| [Observability](#observability) | Every call writes one row to ` |
+| [Deployment](#deployment) | Sidecar-per-consumer |
+| [Image release runbook](#image-release-runbook) | Current state: |
+| [Merged legacy content](#merged-legacy-content) | - old architecture intent absorbed here as transport/runtime split |
+| [Dropped on purpose](#dropped-on-purpose) | - Python/qwen architecture |
+| [Operator summary](#operator-summary) | If you need one page for current service behavior, use this doc |
+<!-- END INDEX -->
+
+# Specialists Service
+
+Single SSOT for `sp serve` and `sp script`. This doc covers service contract, script-class specialist authoring, trust and readiness gates, hot reload, and image release flow.
+
+## Shape
+
+- `sp serve` = HTTP sidecar for script-class specialists.
+- `sp script` = one-shot CLI for same runtime path.
+- Both reuse `script-runner.ts` and `SpecialistLoader`.
+- Both write observability rows into project `.specialists/db/observability.db` unless trace disabled.
+- Both reject interactive / worktree / keep-alive / orchestration use cases. Use `sp run` for those.
+
+## HTTP contract
+
+### Endpoint
+
+`POST /v1/generate`
+
+### Request
+
+```json
+{
+  "specialist": "kebab-case-name",
+  "variables": {"key": "value"},
+  "template": "task_template",
+  "template_field": "task_template",
+  "model_override": "anthropic/...",
+  "timeout_ms": 60000,
+  "trace": true
+}
+```
+
+### Response
+
+```json
+{
+  "success": true,
+  "output": "final assistant text",
+  "parsed_json": {},
+  "meta": {
+    "specialist": "name",
+    "model": "resolved-model",
+    "duration_ms": 1234,
+    "trace_id": "uuid"
+  }
+}
+```
+
+Failure response stays `200` with `success: false`. Closed `error_type` taxonomy:
+
+- `specialist_not_found`
+- `specialist_load_error`
+- `template_variable_missing`
+- `template_field_misuse` — `input.template` was the literal name of a key on `spec.prompt` (e.g. `"task_template"`, `"normalize_template"`). Pass template body inline, or use `template_field` for named prompt field; do not pass key name.
+- `auth`
+- `quota`
+- `timeout`
+- `network`
+- `invalid_json`
+- `prompt_too_large`
+- `output_too_large`
+- `internal`
+
+`400` only for malformed request JSON, missing `specialist`, or unknown template name.
+
+### Reference clients
+
+A stdlib-only Python client lives at [`clients/python/`](../clients/python/). It mirrors the closed `error_type` taxonomy 1:1 and adds a single caller-side `transport` value for HTTP/socket failures. Import as `from specialists_client import SpecialistsClient, SpecialistErrorType`.
+
+## `sp serve`
+
+### CLI
+
+```bash
+sp serve \
+  --port <n> \
+  --user-dir <path> \
+  --concurrency <n> \
+  --request-timeout-ms <n> \
+  --audit-failure-threshold <n> \
+  --allow-skills \
+  --allow-skills-roots <p1>:<p2> \
+  --reload-poll-ms <n> \
+  --readiness-canary off|warn|require \
+  --readiness-required-pi-flag <flag> \
+  --readiness-canary-specialist <name>
+```
+
+Defaults:
+
+- port `8000`
+- user dir `./.specialists/user`
+- concurrency `4`
+- request timeout `120000`
+
+### Runtime
+
+- One Node process, one HTTP listener, one `pi` subprocess per in-flight request.
+- `pi` is spawned in `options.projectDir ?? process.cwd()` and gets current one-shot runner contract: `--mode json --no-session --no-extensions --no-tools --offline --no-context-files --no-prompt-templates --no-themes --model <resolved> [--thinking <level>] [--system-prompt <text>] [--skill <path> ... | --no-skills]`; rendered prompt is written to stdin, not argv.
+- Credentials stay inside `pi`; service never handles API keys.
+- Excess load waits behind semaphore, then returns `429`.
+
+### `/healthz`
+
+- `200 { ok: true }`
+- Liveness only.
+
+### `/readyz`
+
+Readiness gate with explicit reason strings. Failure order:
+
+1. `draining`
+2. `degraded:audit`
+3. `pi_config_unreadable`
+4. `db_not_writable`
+5. optional Pi canary: `pi_binary_missing`, `pi_flag_missing`, `pi_smoke_failed`
+6. `empty_user_dir`
+7. `invalid_spec_in_user_dir`
+
+Notes:
+
+- `/readyz` uses same state as serve loop.
+- audit failure window is sliding 60s.
+- `SIGTERM` flips service into draining.
+- `evaluateReadiness()` is pure async and testable.
+- Pi canary defaults to `off`; use `--readiness-canary warn` in Docker/Podman rollout to surface `warning` without failing readiness, then `--readiness-canary require` when deployments should block on Pi compatibility.
+- `--readiness-required-pi-flag <flag>` may be repeated; without explicit flags the canary checks the script-runner isolation flags.
+- `--readiness-canary-specialist <name>` adds a minimal smoke generation through a script-class specialist; keep its prompt non-secret and deterministic.
+
+### Trust flags
+
+Default-reject. Nothing from user dir runs unless explicitly allowed.
+
+Flags:
+
+- `--allow-skills`
+  - allows `skills.paths` and `prompt.skill_inherit`
+- `--allow-skills-roots <p1>:<p2>`
+  - prefix allowlist for `skills.paths` and `prompt.skill_inherit`
+  - only active when `--allow-skills` on
+- `--allow-local-scripts` is not supported. `skills.scripts` are always rejected in service/script mode until a separate sandboxed script-execution design exists.
+
+Audit behavior:
+
+- when `--allow-skills` on, each resolved skill source gets `{ path, sha256 }` in status audit data
+- unreadable file records `sha256: "unreadable"`
+- compat guard still blocks interactive, worktree, and all `skills.scripts` cases
+
+### Hot reload
+
+- default watcher: native `fs.watch`
+- fallback: `--reload-poll-ms <n>` polling mode for platforms where native watch is weak
+- debounce: `300ms`
+- change action: `loader.invalidateCache(name)` per touched specialist name
+- reload path stays request-safe: request captures current spec at entry
+
+## `sp script`
+
+One-shot CLI on same runtime path as `sp serve`.
+
+### Shape
+
+```bash
+sp script <name> \
+  [--vars k=v ...] \
+  [--template <text>] \
+  [--template-field <name>] \
+  [--model <override>] \
+  [--thinking <level>] \
+  [--project-dir <path>] \
+  [--db-path <path>] \
+  [--timeout-ms <n>] \
+  [--json] \
+  [--single-instance <lockpath>] \
+  [--no-trace] \
+  [--allow-local-scripts] \
+  [--allow-write-capable]
+```
+
+Notes:
+
+- `--project-dir` resolves specs from `<project-dir>/.specialists/user/` and also becomes `pi` child cwd
+- `--user-dir` still accepted as deprecated alias
+- `--db-path` is an exact SQLite file path for script observability; default remains `<git-root>/.specialists/db/observability.db`
+- rendered prompt is streamed to `pi` stdin; no positional prompt arg anymore
+- default output = assistant text on stdout
+- `--json` returns full response payload
+- `--single-instance` uses `flock`; contention exits `75`
+- trusted local mode only on `sp script`: `--allow-local-scripts` enables `skills.scripts`; `--allow-write-capable` permits non-`READ_ONLY` specialists
+
+### Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | success |
+| `1` | internal / unknown |
+| `2` | `specialist_not_found` or `specialist_load_error` |
+| `3` | `template_variable_missing` |
+| `4` | `auth` or `quota` |
+| `5` | `timeout` or `network` |
+| `6` | `invalid_json` |
+| `7` | `output_too_large` |
+| `75` | `--single-instance` lock contention |
+
+## Script-class authoring
+
+Script-class specialists are JSON specialists with one-shot runtime constraints.
+
+Reference schema and examples: [`docs/authoring.md`](authoring.md)
+
+### Required shape
+
+- `execution.interactive` must be `false`
+- `execution.requires_worktree` must be `false`
+- `execution.permission_required` must be `READ_ONLY`
+- `skills.scripts` must be empty
+- `prompt.task_template` must exist
+- template variables must all resolve at run time
+
+### Behavior
+
+- same model resolution as `sp run`
+- same fallback-model behavior
+- same thinking-level forwarding
+- `prompt.system` becomes `--system-prompt` on `pi` (full override, not append)
+- trusted `skills.paths` and `prompt.skill_inherit` are forwarded as `--skill`; if none are allowed, child gets `--no-skills`
+- same output-schema validation behavior
+- stdout cap defaults to 32MB; override with `SPECIALISTS_SCRIPT_STDOUT_LIMIT_BYTES` or `execution.stdout_limit_bytes`
+- `template_variable_missing` if template references missing `$var`
+
+### What this doc does not cover
+
+- full authoring schema details
+- older YAML shape
+- legacy Python/qwen runtime
+
+Use [`docs/authoring.md`](authoring.md) for full field reference.
+
+## Observability
+
+Every call writes one row to `.specialists/db/observability.db` unless `trace: false` or `--no-trace`.
+
+Same observability table, same trace shape, same writer as `sp run`.
+
+## Deployment
+
+Sidecar-per-consumer.
+
+```yaml
+services:
+  specialists:
+    image: specialists-service:local
+    command: [sp, serve, --port=8000]
+    user: "${UID:-1000}:${GID:-1000}"
+    environment:
+      HOME: /pi-home
+    volumes:
+      - ./.specialists:/work/.specialists
+      - ${HOME}/.pi:/pi-home/.pi:ro
+    working_dir: /work
+    networks: [app]
+    restart: unless-stopped
+```
+
+Rules:
+
+- mount `.specialists/` directory, not only DB file
+- keep local filesystem, not NFS/SMB
+- align container UID/GID with host files
+- keep `pi` auth inside mounted home
+
+## Image release runbook
+
+Current state:
+
+- build from repo
+- image build keeps `config/` and `scripts/generate-asset-contract.mjs` in context for asset-contract generation
+- publish to registry deferred
+- `Dockerfile` keeps `ARG PI_VERSION=latest`
+- CI canary lives in [`.github/workflows/pi-compat.yml`](../.github/workflows/pi-compat.yml)
+
+### Build
+
+```bash
+docker build -t specialists-service:local .
+```
+
+### Release intent
+
+- root `Dockerfile` is source of runtime image truth
+- pin `@earendil-works/pi-coding-agent` only in release pipeline if needed
+- keep canary green before bumping runtime dependency
+- registry publishing stays future work until release path lands
+
+### Compatibility canary
+
+`pi-compat.yml` checks:
+
+- required `pi` spawn flags still exist
+- stdin prompt handling still works
+- image boots with latest `pi`
+
+## Merged legacy content
+
+### From `docs/script-specialists.md`
+
+- old architecture intent absorbed here as transport/runtime split
+- service-side authoring constraints copied into script-class authoring section
+- old Python/qwen implementation details dropped
+- old YAML-first wording dropped
+
+### From retired image-release notes
+
+- build / push intent folded into image release runbook
+- current publish state recorded as deferred
+- CI canary and `PI_VERSION=latest` preserved
+
+## Dropped on purpose
+
+- Python/qwen architecture
+- legacy `specialists/**/*.specialist.yaml` framing
+- service claims about being tied to old gateway runtime
+- obsolete image push instructions as active maintainer path
+
+## Operator summary
+
+If you need one page for current service behavior, use this doc. If you need field-level authoring schema, use [`docs/authoring.md`](authoring.md).

package/docs/surface-ownership.md

@@ -0,0 +1,138 @@
+# Surface Ownership and Precedence
+
+> Canonical ownership map for specialist assets after layered refactor.
+
+## Four-layer model
+
+1. **Upstream source (package, read-only at runtime)**
+   - `config/specialists/*.specialist.json`
+   - `config/mandatory-rules/*`
+   - `config/nodes/*.node.json`
+2. **Optional managed/pinned mirror (repo, compatibility/pin layer)**
+   - `.specialists/default/*.specialist.json`
+   - `.specialists/default/mandatory-rules/*`
+   - `.specialists/default/nodes/*.node.json`
+   - Empty by default in fresh package-canonical installs; populated only for intentional pins or compatibility snapshots
+3. **Repo authoring / custom layer (repo-owned)**
+   - `.specialists/user/*.specialist.json`
+   - `config/nodes/*.node.json` (repo node overrides)
+4. **Runtime / generated state (never hand-author)**
+   - `.specialists/jobs/`
+   - `.specialists/ready/`
+   - `.specialists/db/`
+
+## Ownership boundaries
+
+- `config/*` in package = canonical upstream source.
+- `.specialists/default/*` = optional pin / compatibility snapshot. Prefer package-canonical fallback; prune stale defaults with `sp prune-stale-defaults`.
+- `.specialists/user/*` = repo custom specialists. Safe place for overrides/forks.
+- `.specialists/{jobs,ready,db}` = runtime state. Do not treat as config surface.
+
+## Precedence ladders
+
+### Specialists
+
+Runtime loader order (first match wins):
+
+1. `.specialists/user/`
+2. `.specialists/default/`
+3. `config/specialists/` (package fallback)
+4. legacy paths (migration compatibility only)
+
+Meaning:
+- Same filename in `.specialists/user/` overrides mirror/package.
+- New filename in `.specialists/user/` extends catalog.
+- `.specialists/default/` can override package fallback for same name.
+
+### Mandatory rules
+
+Four-tier index resolution. Loader reads and union-merges all that exist (dedup by set id):
+
+1. `.specialists/user/mandatory-rules/index.json` — repo-owned user overlay, highest priority
+2. `config/mandatory-rules/index.json` — upstream source, ships with package
+3. `.specialists/default/mandatory-rules/index.json` — optional pin / compatibility snapshot, pruned by `sp prune-stale-defaults` unless intentionally retained
+4. `.specialists/mandatory-rules/index.json` — repo overlay, wins on set-id conflict
+
+Set-file lookup (`<set-id>.md`) probes the same four paths in reverse precedence:
+`.specialists/user/mandatory-rules/` first, then `.specialists/mandatory-rules/`, then `.specialists/default/mandatory-rules/`, then `config/mandatory-rules/`.
+
+Injection merge order (within the resolved index):
+
+1. required sets (union of all tiers)
+2. default sets (union of all tiers)
+3. specialist `mandatory_rules.template_sets`
+4. specialist `mandatory_rules.inline_rules`
+
+Global workflow block injected unless `mandatory_rules.disable_default_globals=true`.
+
+Full authoring guide: [`config/mandatory-rules/README.md`](../config/mandatory-rules/README.md).
+
+### Nodes
+
+Node resolution order:
+
+1. explicit path (`specialists node run ./path/to/file.node.json`)
+2. `config/nodes/` (repo-owned override layer)
+3. `.specialists/default/nodes/` (managed mirror fallback)
+
+`specialists node list` discovery follows same repo-first then mirror order.
+
+## Repo fork flow (`specialists edit`)
+
+### Same-name override
+
+Use when customizing existing specialist without new catalog name.
+
+- Create/edit `.specialists/user/<name>.specialist.json`
+- Name unchanged
+- Loader picks user file first
+
+### New-name fork
+
+Use when creating variant specialist.
+
+- `specialists edit <new-name> --fork-from <base-name>`
+- Writes `.specialists/user/<new-name>.specialist.json`
+- New specialist appears alongside base
+
+## Init, prune, and sync behavior
+
+`specialists init` always:
+- ensures `.specialists/user`, `.specialists/jobs`, `.specialists/ready`, `.specialists/db`
+- wires hooks, MCP, skill symlinks
+
+Package-canonical runtime assets are read from the installed package when no repo override exists. Fresh repos should normally leave `.specialists/default/` empty.
+
+`specialists init --sync-defaults` still exists for compatibility, but it is deprecated because it creates default snapshots that drift. Use it only when an operator deliberately wants a repo-local pin/compatibility mirror.
+
+Preferred cleanup:
+- `sp doctor --check-drift` — report stale Category A defaults
+- `sp prune-stale-defaults --dry-run` — preview removals
+- `sp prune-stale-defaults` — remove stale defaults, including diverged defaults by default
+- `sp prune-stale-defaults --keep-diverged` — preserve intentionally diverged defaults
+
+## Migration notes
+
+Legacy layouts still recognized for compatibility:
+- `.specialists/user/specialists/*.specialist.json`
+- `.specialists/default/specialists/*.specialist.json`
+
+Migration target:
+- `.specialists/user/*.specialist.json`
+- `.specialists/default/*.specialist.json`
+
+Operator rule:
+- do not hand-edit `.specialists/default/*`
+- edit in `.specialists/user/*` for repo custom behavior
+- run `sp doctor --check-drift` and `sp prune-stale-defaults` to reconcile default mirror drift
+
+## Explicit non-goal: backlog-clean isolation
+
+This ownership/precedence migration does **not** change backlog-clean surfaces.
+
+Untouched by this contract:
+- backlog-clean command behavior
+- backlog-clean data model
+- backlog-clean runtime hooks/automation
+
+Keep backlog-clean work isolated in dedicated issues/tasks.

package/docs/upgrade-notes/kan-90-global-user-config.md

@@ -0,0 +1,114 @@
+# KAN-90 / unitAI-1gtou — Upgrading to the global user-config layer
+
+This release moves model selection out of the shipped specialist files and into a
+single per-user config file. Before this change every specialist in
+`config/specialists/*.specialist.json` shipped with a hard-coded
+`execution.model` and `execution.fallback_model`. That was a user-environment
+detail (each user has their own provider keys in `pi`), not something the
+package should ship.
+
+## What changed
+
+- Every `config/specialists/*.specialist.json` ships with
+  `execution.model = null` and `execution.fallback_model = null`.
+- The `SpecialistLoader` resolves each specialist via a 3-layer field-merge:
+
+  ```
+  package canonical            (base)
+    ⇡ ~/.config/specialists/user.json    (your model + per-spec tuning)
+    ⇡ .specialists/user/<name>           (repo authoring layer)
+  ```
+
+  The repo-managed `.specialists/default/<name>` mirror was retired by commit
+  `31a6421c` and is no longer walked by the loader. Stale entries left on disk
+  are detected by `drift-detector` and removed by `sp prune-stale-defaults`.
+
+- If `execution.model` is still `null` after the full merge, the loader throws
+  a `SpecialistMissingModelError` pointing you at `sp edit --global`.
+- `sp init --global` generates the global user config; `sp edit --global` reads
+  / writes individual fields; `sp doctor --specialists` reports coverage and
+  flags blocked-field attempts.
+
+## Upgrade procedure
+
+Run this **once** after upgrading:
+
+```bash
+# 1. Generate ~/.config/specialists/user.json (or $XDG_CONFIG_HOME/...).
+#    Idempotent: it is safe to re-run later.
+sp init --global
+
+# 2. Pick a default model for any specialist you actually use:
+sp edit --global executor.execution.model         anthropic/claude-sonnet-4-6
+sp edit --global executor.execution.fallback_model openai-codex/gpt-5.4-mini
+# ... repeat for any specialist you dispatch
+
+# 3. Verify coverage:
+sp doctor --specialists
+```
+
+The `sp doctor` output will list every specialist that still has no model and
+suggest the exact `sp edit --global` command to fix it. Specialists you never
+dispatch can be left alone — the missing-model error only fires at dispatch
+time, never during `specialists list` or `sp doctor`.
+
+## What you can override (per-specialist) in the global config
+
+Allowed in `~/.config/specialists/user.json`:
+
+- `execution.model`
+- `execution.fallback_model`
+- `execution.timeout_ms`
+- `execution.stall_timeout_ms`
+- `execution.thinking_level`
+- `execution.max_retries`
+- `beads_write_notes`
+- `skills.paths` (append + dedup against the base)
+
+Anything else in `user.json` is a blocked field. The loader strips it during
+merge and `sp doctor --specialists` surfaces a `STRIPPED` warning:
+
+- `execution.permission_required` — identity field; the package decides what
+  tools each role can call.
+- `execution.auto_commit`
+- `prompt.system`
+- `prompt.output_schema`
+- `skills.scripts` — pre/post hooks are part of the role contract.
+- `mandatory_rules`
+- `capabilities`
+
+Repo-level overrides (`.specialists/user/<name>`) keep the existing whole-file
+replacement behaviour: blocked fields are applied but `sp doctor --specialists`
+flags them as a `warn`-severity finding so the divergence is visible.
+
+## Path resolution
+
+The loader checks in this order:
+
+1. `$XDG_CONFIG_HOME/specialists/user.json` (if `XDG_CONFIG_HOME` is set)
+2. `~/.config/specialists/user.json`
+3. `~/.specialists/user.json` (legacy, read-only fallback)
+
+`sp init --global` always writes to slot 1 or 2 (the XDG-compliant target).
+
+## Why a single file
+
+Earlier sketches considered `~/.specialists/defaults.json` plus per-specialist
+sparse files. The single-file shape (`unitAI-o328h`) was picked because:
+
+- One file is the canonical place to look at and edit.
+- No drift between defaults and per-specialist overrides.
+- `sp init --global` only has to manage one file.
+- Removing a specialist from a future release still leaves its block in the
+  user's file (with a `removed` notice in `sp init --global` output) so the
+  user keeps the history.
+
+## Cross-references
+
+- Jira: [KAN-90](https://xtrmxt.atlassian.net/browse/KAN-90)
+- Design decision: bd `unitAI-o328h`
+- Implementation epic: bd `unitAI-1gtou`
+  - C1 (loader): `unitAI-1gtou.12` (commit `6604c144`)
+  - C2 (CLI): `unitAI-1gtou.13` (commit `5f8d725e`)
+  - C3 (doctor): `unitAI-1gtou.14` (commit `6b69a6fe`)
+  - C4 (config strip + this doc): `unitAI-1gtou.15`