@jaggerxtrm/specialists 3.16.0 → 3.18.0

package/docs/worktrees.md

@@ -0,0 +1,309 @@
+---
+title: Worktree Isolation
+scope: worktrees
+category: reference
+version: 1.4.1
+updated: 2026-05-15
+synced_at: bf6baf7a
+description: Technical reference for worktree-per-executor isolation — CLI flags, job registry, GC, and chained bead patterns.
+source_of_truth_for:
+  - "src/specialist/job-root.ts"
+  - "src/specialist/worktree.ts"
+  - "src/specialist/worktree-gc.ts"
+  - "src/cli/run.ts"
+  - "src/specialist/supervisor.ts"
+domain:
+  - worktrees
+  - jobs
+  - isolation
+---
+
+# Worktree Isolation
+
+Each edit-permission specialist runs in an isolated git worktree (branch). This prevents concurrent file corruption when multiple executors modify overlapping paths, and produces a clean per-task branch that the orchestrator merges in dependency order.
+
+> Design decisions: overthinker bead `abb9`. Implementation: `hgpu.1–hgpu.5`.
+
+---
+
+## CLI flags
+
+```
+specialists run <name> [--worktree] [--job <id>] [--force-stale-base]
+```
+
+| Flag | Semantics | Creates worktree? |
+|------|-----------|:-:|
+| `--worktree` | Explicitly provision a new isolated workspace; requires `--bead` | Yes |
+| `--job <id>` | Reuse the workspace of an existing job | No |
+| `--force-stale-base` | Bypass stale-base guard when provisioning worktree | Yes (if `--worktree`) |
+
+`--worktree` and `--job` are **mutually exclusive**. Specifying both exits with an error.
+
+`--force-stale-base` is only meaningful with `--worktree` — it forces provisioning even when sibling epic chains have unmerged substantive commits.
+
+---
+
+## Isolation guard for edit-capable specialists
+
+Specialists with `permission_required = MEDIUM` or `HIGH` can modify files. Launching them in the main checkout creates last-writer-wins races when multiple specialists run concurrently. The **worktree guard** (`unitAI-fdvt`) blocks these runs unless an isolation option is supplied.
+
+### Trigger condition
+
+Automatic worktree provisioning triggers when **all** of the following are true:
+
+1. `specialist.execution.permission_required` is `MEDIUM` or `HIGH`.
+2. `specialist.execution.requires_worktree` is not set to `false`.
+3. `--job <id>` was not passed.
+4. `--bead <id>` is available (required for deterministic branch naming).
+
+### Error message
+
+If automatic provisioning is required but no `--bead` was supplied:
+
+```
+Error: specialist '<name>' has permission_required=<MEDIUM|HIGH> and requires worktree isolation.
+Provide --bead <id> for automatic worktree provisioning, or use --job <id> to reuse an existing worktree.
+```
+
+The process exits with code `1`.
+
+`READ_ONLY` specialists are **never** gated — the requirement only applies to edit-capable specialists with `requires_worktree=true`.
+
+### Stale-base guard
+
+When `--worktree` provisions a new worktree, the stale-base guard checks whether sibling chains in the same epic have unmerged substantive commits. If detected, dispatch is blocked:
+
+```
+Error: Epic 'unitAI-3f7b' has sibling chains with unmerged changes.
+  - impl-a: 2 substantive commits on 'feature/unitAI-impl-a-executor'
+  - impl-b: 3 substantive commits on 'feature/unitAI-impl-b-executor'
+Merge sibling chains first via 'sp epic merge unitAI-3f7b', or use --force-stale-base to bypass.
+```
+
+**Why this matters**: When parallel chains branch from the same base, Wave B's worktree lacks Wave A's merged changes. If Wave A merges first, Wave B's diff shows reversions. Rebase at merge-time resolves this, but starting from a stale base increases conflict risk.
+
+Use `--force-stale-base` to bypass when you knowingly accept merge complexity later.
+
+### `--worktree`
+
+Optional explicit flag. MEDIUM/HIGH specialists that require isolation now auto-provision without this flag when `--bead` is present.
+
+Requires `--bead <id>` — the bead id drives the deterministic branch name.
+
+```bash
+specialists run executor --worktree --bead hgpu.3
+# stderr: [worktree created: /repo/.worktrees/hgpu.3/hgpu.3-executor  branch: feature/hgpu.3-executor]
+```
+
+If a worktree for that branch already exists (e.g. from a prior interrupted run) it is reused:
+
+```bash
+# stderr: [worktree reused: /repo/.worktrees/hgpu.3/hgpu.3-executor  branch: feature/hgpu.3-executor]
+```
+
+### `--job <id>`
+
+Resolves the target job's `worktree_path` from the DB-first runtime state, falling back to legacy `status.json` mirrors when enabled, and uses that directory as `cwd`. The **caller's** `--bead` remains authoritative — only the workspace is borrowed.
+
+```bash
+specialists run reviewer --job 49adda --bead hgpu.3-review
+# stderr: [workspace reused from job 49adda: /repo/.worktrees/hgpu.3/hgpu.3-executor]
+```
+
+Hard fail conditions (both exit 1):
+- `status.json` missing or unreadable for the given job id.
+- `worktree_path` absent — the target job was not started with `--worktree`.
+
+**Concurrency guard:** READ_ONLY and LOW specialists may run against an active worktree; MEDIUM/HIGH specialists are blocked until the owning job reaches a terminal state. Use `--force-job` to bypass.
+
+---
+
+## How it works
+
+### Branch naming
+
+`provisionWorktree()` in `worktree.ts` derives deterministic names:
+
+| Artifact | Convention | Example |
+|----------|-----------|---------|
+| Git branch | `feature/<beadId>-<slug>` | `feature/hgpu.3-executor` |
+| Worktree dir | `<beadId>-<slug>` | `hgpu.3-executor` |
+| Parent dir | `<git-common-root>/.worktrees/<beadId>/` | `.worktrees/hgpu.3/` |
+
+`<slug>` is the specialist name lowercased with non-alphanumeric runs collapsed to `-`.
+
+### Worktree creation
+
+`bd worktree create <path> --branch <branch>` is the **only** creation path. There is no silent `git worktree add` fallback — failure throws immediately with the bd stderr included in the message.
+
+Reuse detection runs first via `git worktree list --porcelain`; creation is skipped if the branch is already checked out.
+
+### Central job registry
+
+`resolveJobsDir()` in `job-root.ts` anchors `.specialists/jobs/` to the git **common root** using `git rev-parse --git-common-dir`. From any worktree, `dirname(resolve(cwd, gitCommonDir))` resolves to the main checkout root — all worktrees converge on the same jobs directory.
+
+```
+/repo/.git/                     ← common git dir
+/repo/.specialists/jobs/        ← shared job registry (all worktrees read/write here)
+/repo/.worktrees/hgpu.3/hgpu.3-executor/   ← isolated cwd for that run
+```
+
+### Persisted metadata
+
+`Supervisor` persists `worktree_path` and `branch` in canonical runtime state and, when legacy file mirrors are enabled, also writes them to `status.json` immediately on job start:
+
+```json
+{
+  "id": "49adda",
+  "specialist": "executor",
+  "status": "running",
+  "worktree_path": "/repo/.worktrees/hgpu.3/hgpu.3-executor",
+  "branch": "feature/hgpu.3-executor"
+}
+```
+
+`--job` resolution uses persisted job metadata — no git scanning required.
+
+### Status / steer / resume
+
+`status`, `steer`, and `resume` commands all call `resolveJobsDir()` with their local `cwd`, which returns the common-root path regardless of whether they are invoked from a worktree or the main checkout. The job record is always found.
+
+### Pi bootstrap
+
+Pi extensions are global (`~/.pi/`). No per-worktree bootstrap step is required.
+
+---
+
+## Worktree GC
+
+```bash
+specialists clean            # prunes job dirs AND terminal worktrees
+specialists clean --dry-run  # preview removals without deleting
+```
+
+GC runs automatically as part of `specialists clean`. Candidates must satisfy **all** conditions:
+
+1. Job status is `done` or `error` (terminal).
+2. `worktree_path` is recorded in `status.json`.
+3. The directory still exists on disk.
+4. Job status is **not** `starting`, `running`, or `waiting` (active guard runs first, unconditionally).
+
+Removal uses `git worktree remove --force` so both the directory and the git registry entry are cleaned atomically. Failures are skipped silently — missing cleanup is preferred over data loss.
+
+---
+
+## Session close: `sp end`
+
+`specialists end` closes a worktree session with epic-aware publication.
+
+### Synopsis
+
+```bash
+sp end [--bead <id>|--epic <id>] [--pr] [--rebuild]
+```
+
+### Epic-aware behavior
+
+If the current chain belongs to an unresolved epic (`open`, `resolving`, `merge_ready`):
+
+1. `sp end` detects epic membership via `checkEpicUnresolvedGuard()`
+2. Prints redirect message
+3. Delegates to `sp epic merge <epic-id>`
+
+Example:
+```
+Chain unitAI-impl belongs to unresolved epic unitAI-3f7b (status: resolving).
+Redirecting session close publication to epic merge (direct mode).
+Epic unitAI-3f7b: resolving -> merge_ready
+Epic unitAI-3f7b: merge_ready -> merged
+Publication successful.
+```
+
+### Direct chain publication
+
+For standalone chains NOT belonging to an epic:
+
+```bash
+sp end --bead unitAI-55d
+# → merges branch feature/unitAI-55d-executor
+```
+
+---
+
+---
+
+## Chained bead review/fix loop
+
+A common orchestration pattern with worktree isolation:
+
+```bash
+# 1. Executor claims bead, provisions worktree, does implementation
+specialists run executor --worktree --bead hgpu.3
+# → executor closes bead as COMPLETE/PARTIAL, job id: 49adda
+
+# 2. Reviewer enters the same worktree (read bead notes from the executor's run)
+specialists run reviewer --job 49adda --bead hgpu.3-review
+
+# 3. If reviewer returns PARTIAL, fix-it agent re-enters same workspace
+specialists run executor --job 49adda --bead hgpu.3-fix
+```
+
+Key invariants:
+- Reviewer sees exactly the state the executor left — same branch, same files.
+- Caller's `--bead` controls which bead is opened/closed; `--job` only selects the workspace.
+- The executor's bead is never re-opened by the reviewer — lifecycle stays with the original claimer.
+
+For orchestration patterns that compose this loop, see `SKILL.md` and `workflow.md`.
+
+---
+
+## Auto-checkpoint behavior
+
+Specialists with `auto_commit: checkpoint_on_waiting` or `checkpoint_on_terminal` automatically commit substantive worktree changes at designated lifecycle points. This prevents lost work if a job crashes or is stopped mid-task.
+
+| Policy | When commits happen | Typical use |
+|--------|---------------------|-------------|
+| `never` | Never | Default — no auto-commit |
+| `checkpoint_on_waiting` | Each keep-alive turn entering `waiting` | Executors, debuggers — preserve partial work before review |
+| `checkpoint_on_terminal` | Terminal completion (`done`/`error`) | One-shot specialists — commit only at end |
+
+Default specialists with auto-checkpoint enabled:
+- **executor**: `checkpoint_on_waiting`
+- **debugger**: `checkpoint_on_waiting`
+
+Commit messages follow the pattern:
+```
+checkpoint(executor): unitAI-55d turn 1
+```
+
+Status metadata tracks checkpoints:
+```json
+{
+  "auto_commit_count": 3,
+  "last_auto_commit_sha": "a1b2c3d",
+  "last_auto_commit_at_ms": 1714020000000
+}
+```
+
+Auto-checkpoint runs silently — no user action required. The guard persists result + bead notes even on initial waiting turn (unitAI-8b812), ensuring no work is lost before the first checkpoint.
+
+---
+
+## Key files
+
+| File | Responsibility |
+|------|---------------|
+| `src/specialist/job-root.ts` | `resolveJobsDir()` — common-root job registry anchor |
+| `src/specialist/worktree.ts` | `provisionWorktree()`, branch/path derivation, `listWorktrees()` |
+| `src/specialist/worktree-gc.ts` | `collectWorktreeGcCandidates()`, `pruneWorktrees()` |
+| `src/cli/run.ts` | `resolveWorkingDirectory()` — `--worktree`/`--job` dispatch |
+| `src/specialist/supervisor.ts` | Persists `worktree_path` + `branch` to DB-first runtime state and legacy mirrors when enabled |
+
+---
+
+## See also
+
+- [background-jobs.md](background-jobs.md) — job lifecycle, status polling, keep-alive
+- [workflow.md](workflow.md) — orchestration patterns and specialist chaining
+- [worktree.md](worktree.md) — xtrm `xt pi` / `xt end` integration (separate topic)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jaggerxtrm/specialists",
-  "version": "3.16.0",
+  "version": "3.18.0",
   "description": "OmniSpecialist — 7-tool MCP orchestration layer powered by the Specialist System. Discover and execute .specialist.yaml files across project/user/system scopes via pi.",
   "main": "dist/index.js",
   "types": "dist/types/lib.d.ts",
@@ -27,6 +27,7 @@
     "config/nodes/",
     "config/hooks/",
     "config/presets.json",
+    "docs/",
     "LICENSE",
     "!config/benchmarks/",
     "!config/skills/**/evals/",
@@ -73,28 +74,32 @@
     }
   },
   "dependencies": {
-    "@earendil-works/pi-tui": "^0.75.4",
-    "@modelcontextprotocol/sdk": "^1.29.0",
+    "@earendil-works/pi-tui": "0.75.5",
+    "@modelcontextprotocol/sdk": "1.29.0",
+    "hono": "4.12.26",
+    "qs": "6.15.2",
     "yaml": "2.9.0",
-    "zod": "^3.25.76",
-    "zod-to-json-schema": "^3.24.6"
+    "zod": "3.25.76",
+    "zod-to-json-schema": "3.25.2"
   },
   "devDependencies": {
     "@types/bun": "1.3.14",
-    "@types/node": "^25.8.0",
-    "@vitest/coverage-v8": "^4.1.6",
-    "tsx": "^4.20.6",
-    "typescript": "^5.0.0",
-    "vitest": "^4.1.6"
+    "@types/node": "25.9.1",
+    "@vitest/coverage-v8": "4.1.7",
+    "tsx": "4.22.3",
+    "typescript": "5.9.3",
+    "vitest": "4.1.7",
+    "vite": "8.0.16"
   },
   "engines": {
     "bun": ">=1.0.0"
   },
   "overrides": {
     "@hono/node-server": "^1.19.13",
+    "esbuild": "^0.28.1",
     "fast-uri": "^3.1.2",
-    "hono": "^4.12.18",
     "ip-address": "^10.2.0",
-    "path-to-regexp": "^8.4.0"
+    "path-to-regexp": "^8.4.0",
+    "qs": "6.15.2"
   }
 }

package/config/specialists/code-sanity.specialist.json

@@ -1,108 +0,0 @@
-{
-  "specialist": {
-    "metadata": {
-      "name": "code-sanity",
-      "version": "1.0.0",
-      "description": "READ_ONLY smell pass after executor and before reviewer. Checks simplicity, type safety, dead code, async/error handling, maintainability.",
-      "category": "quality",
-      "tags": [
-        "code-quality",
-        "clean-code",
-        "type-safety",
-        "simplicity",
-        "review",
-        "read-only"
-      ],
-      "updated": "2026-05-04"
-    },
-    "execution": {
-      "mode": "tool",
-      "model": "openai-codex/gpt-5.4-mini",
-      "fallback_model": "zai/glm-5-turbo",
-      "timeout_ms": 0,
-      "stall_timeout_ms": 120000,
-      "response_format": "markdown",
-      "output_type": "review",
-      "permission_required": "READ_ONLY",
-      "interactive": true,
-      "thinking_level": "low",
-      "max_retries": 0,
-      "extensions": {
-        "serena": false
-      }
-    },
-    "mandatory_rules": {
-      "template_sets": [
-        "explorer-readonly",
-        "gitnexus-required",
-        "code-quality-defaults",
-        "per-turn-handoff-schema",
-        "bead-id-verbatim"
-      ]
-    },
-    "permissions": {
-      "READ_ONLY": {
-        "denied_natives_when_extension": [
-          "grep",
-          "find",
-          "ls"
-        ],
-        "denied_natives_mode": "hard"
-      }
-    },
-    "prompt": {
-      "system": "You are a READ_ONLY code-sanity specialist. Your job is a cheap, focused implementation-quality smell pass after executor and before final reviewer. You do not decide whether requirements are met; the reviewer owns that merge gate. You do not edit files.\n\n## Scope\n\nInspect an executor job, diff, or explicitly scoped files for implementation sanity only:\n- unnecessary complexity\n- over-abstraction or premature generalization\n- type unsafety (`any`, unsafe casts, non-null assertions, untyped boundaries, ignored unknowns)\n- brittle async/error handling\n- duplicated logic that should be trivially consolidated\n- dead code, unreachable branches, unused exports introduced by the change\n- unclear names or control flow that make the implementation harder than necessary\n- violations of the clean-code guidance that are concrete enough to fix\n\n## Non-goals\n\nDo not validate product requirements, release readiness, security posture, broad architecture, or test coverage. Do not request preference-only rewrites. Do not broaden into a full code review. Do not run broad test suites.\n\n## Tooling approach\n\nUse GitNexus and Serena for precise inspection when available. Prefer symbol/diff-aware navigation over raw file reading. If inspecting changed symbols, use impact/context evidence when useful, but keep the pass bounded. Stop after at most 8 tool calls unless the prompt explicitly asks for deeper analysis.\n\n## Output discipline\n\nDefault to OK when the implementation is simple enough. If findings exist, return at most 5. Each finding must be concrete, actionable, and suitable to paste into `specialists resume <executor-job>`. Cite file/symbol/line evidence where possible. Avoid taste-only comments.\n\n## Required output format\n\n## Code Sanity Verdict\n- Verdict: OK | FINDINGS | BLOCKED\n- Scope reviewed:\n- Confidence: high | medium | low\n\n## Findings\nAt most 5. For each:\n- Location:\n- Issue:\n- Why it matters:\n- Suggested executor instruction:\n\n## Non-findings\nBriefly note any tempting but intentionally ignored concerns because they belong to reviewer/test-runner/security-auditor.\n\n## Handoff\nOne short paragraph for the orchestrator: proceed to reviewer, resume executor with findings, or rerun with clearer scope.\n\n## Machine-readable block\nReturn exactly one JSON object in a single fenced ```json block matching the output_schema.\n\nAfter delivering the report, enter waiting state for follow-up or re-check.",
-      "task_template": "Run a code-sanity pass for the following implementation context:\n\n$prompt\n\nWorking directory: $cwd\n\nIf an executor job id is provided, inspect that job's output/diff if available. If only files are provided, inspect only those files. Stay READ_ONLY and return the required bounded report.",
-      "output_schema": {
-        "type": "object",
-        "properties": {
-          "verdict": {
-            "enum": [
-              "OK",
-              "FINDINGS",
-              "BLOCKED"
-            ]
-          },
-          "findings_count": {
-            "type": "number"
-          },
-          "locations": {
-            "type": "array",
-            "items": {
-              "type": "string"
-            }
-          },
-          "executor_instructions": {
-            "type": "array",
-            "items": {
-              "type": "string"
-            }
-          },
-          "handoff": {
-            "type": "string"
-          }
-        }
-      }
-    },
-    "skills": {
-      "paths": [],
-      "scripts": []
-    },
-    "validation": {
-      "files_to_watch": [
-        "src/specialist/schema.ts",
-        "src/specialist/runner.ts",
-        ".xtrm/skills/active/clean-code/SKILL.md",
-        ".xtrm/skills/active/gitnexus-impact-analysis/SKILL.md"
-      ],
-      "stale_threshold_days": 30
-    },
-    "capabilities": {
-      "required_tools": [],
-      "external_commands": []
-    },
-    "stall_detection": {},
-    "beads_integration": "auto",
-    "beads_write_notes": true
-  }
-}