@jaggerxtrm/specialists 3.17.0 → 3.18.1

package/docs/archive/issuetracking.md

@@ -0,0 +1,312 @@
+# Agent-Native Issue Tracking Contract Board
+
+Status: deferred design. Do not implement now.
+
+This document captures a future direction for the xt/sp ecosystem after the current specialists and xtrm backlogs settle, conversations layer lands, and preflight/long-run stress testing has exposed more orchestration friction.
+
+The idea is likely a third package in the xt/sp ecosystem, not an immediate replacement of bd.
+
+## Problem
+
+Specialist job quality depends heavily on issue quality.
+
+A vague issue produces vague prompts, weak specialist behavior, missing validation, and reviewer ambiguity. A precise issue behaves like an executable contract: it gives the orchestrator a clear chain, gives specialists tight scope, gives reviewers evidence targets, and makes failures diagnosable.
+
+Current bd usage gives us useful IDs, status, dependency links, JSONL export, and CLI ergonomics, but bd is still primarily an issue tracker. The emerging need is a contract runtime for agent work.
+
+## Core Thesis
+
+Job quality is roughly:
+
+```text
+issue contract quality × orchestration discipline × specialist prompt invariants
+```
+
+The issue/prompt determines the job heavily. Therefore issue creation and update should not be a passive write. It should trigger contract validation before any specialist dispatch.
+
+## Future System Shape
+
+The future system should treat issues as executable job contracts.
+
+```text
+issue contract → validator → specialist chain → evidence graph → review → merge/close
+```
+
+Key properties:
+
+- Contract-first: every dispatchable issue has enough information for an AFK specialist chain.
+- Validator-gated: a small/simple model validates readiness and blocks dispatch if required fields are missing or vague.
+- Issue-local prompt rules: issue-specific mandatory rules append into spawned specialist prompts at job start.
+- State-aware: readiness, work state, review state, dependency state, and evidence state are distinct.
+- Auditable: every validator verdict, override, dispatch, result, review, and close reason is persisted.
+- Conversation-aware: later conversations/agent-mail layer can coordinate clarification, handoff, and cross-agent review around the same contract.
+
+## Contract Readiness States
+
+These are conceptual states, not a final schema.
+
+Work state:
+
+```text
+draft
+ready
+claimed
+running
+waiting
+reviewing
+blocked
+failed
+done
+archived
+```
+
+Contract readiness:
+
+```text
+invalid
+partial
+ready
+waived
+```
+
+Review state:
+
+```text
+unreviewed
+partial
+pass
+fail
+```
+
+Execution readiness:
+
+```text
+blocked_by_dependencies
+blocked_by_contract
+blocked_by_workspace
+ready_to_dispatch
+```
+
+The important split: `ready` should not mean only “not blocked by deps”. It should mean the issue contract is specific enough to launch the right specialist chain.
+
+## Dispatchable Issue Contract
+
+A dispatchable issue should include, either explicitly or derivable from structured fields:
+
+```text
+PROBLEM
+Current bad/missing behavior. Why work exists.
+
+DESIRED OUTCOME
+Observable behavior after completion.
+
+SCOPE
+Allowed files/modules/surfaces. Use likely scope when discovery is expected.
+
+NON-GOALS
+What must not be changed.
+
+ACCEPTANCE CRITERIA
+Concrete behavior checks, preferably user-visible or externally observable.
+
+VALIDATION
+Commands, tests, smoke path, or manual verification instructions.
+
+CONTEXT
+Relevant previous jobs, commits, issues, reports, decisions, known failures.
+
+DEPENDENCIES
+Blocking issues and why they block.
+
+RISK
+Auth/data/destructive/perf/migration/UI/security/config/etc.
+
+SPECIALIST ROUTING
+Suggested chain: explorer/debugger/executor/test-writer/test-runner/code-sanity/security-auditor/reviewer/etc.
+
+ISSUE-LOCAL MANDATORY RULES
+Extra prompt invariants appended to every job spawned from this issue.
+```
+
+## Contract Validator
+
+On issue create/update, a small model should validate the contract and write a structured verdict.
+
+Example output:
+
+```json
+{
+  "contract_status": "not_ready",
+  "dispatch_allowed": false,
+  "blocking_gaps": [
+    "Acceptance criteria are not observable",
+    "Validation command missing",
+    "Scope mixes two unrelated runtime surfaces"
+  ],
+  "suggested_rewrite": {
+    "problem": "...",
+    "scope": "...",
+    "non_goals": "...",
+    "validation": "..."
+  },
+  "recommended_chain": ["explorer", "executor", "test-writer", "test-runner", "reviewer"]
+}
+```
+
+Dispatch should be refused when `dispatch_allowed=false` unless the orchestrator explicitly records an override reason.
+
+```bash
+sp run executor --issue X
+# refuses: contract not ready
+
+sp run executor --issue X --allow-unready --reason "emergency prod hotfix; validation is manual"
+# allowed, but override is persisted and review confidence is reduced
+```
+
+## Issue-Local Mandatory Rules
+
+Global and role mandatory rules help specialists, but issue-local rules can make each job much tighter.
+
+Example:
+
+```text
+ISSUE_MANDATORY_RULES:
+- Do not touch source files outside src/specialist/runner.ts and tests/unit/specialist.
+- Preserve DB-first behavior; file fallback remains legacy-only.
+- Reviewer must verify staged diff and branch-vs-base diff.
+```
+
+At specialist spawn time, these rules should append alongside package mandatory rules, after global and role rules, so the specialist starts with exact constraints.
+
+## Orchestrator Guardrails
+
+The orchestrator should behave like a workflow engine, not a heroic manual driver.
+
+Expected loop:
+
+1. Read issue contract.
+2. Validate or refresh readiness verdict.
+3. Improve contract before dispatch if validator blocks.
+4. Decide specialist chain.
+5. Launch first specialist.
+6. Monitor job state and consume structured handoff.
+7. Decide next edge in the chain.
+8. Route missing tests to test-writer/test-runner.
+9. Route risky diffs to code-sanity/security-auditor.
+10. Route final diff to reviewer.
+11. File friction bugs discovered during long runs.
+12. Close only with evidence mapped to acceptance criteria.
+
+Principles for a future `using-specialists-v4`:
+
+```text
+1. No specialist dispatch from vague contract.
+2. Issue contract is source of truth; job prompt is generated from it.
+3. Readiness is validated before dispatch.
+4. Orchestrator must repair contract before launching.
+5. Issue-local mandatory rules append into every job spawned from issue.
+6. Every specialist output maps back to acceptance criteria.
+7. Reviewer checks contract compliance, not vibes.
+8. Long runs are expected; orchestration must tolerate context, process, and state friction.
+9. Friction discovered during runs becomes first-class follow-up issues.
+10. Merge/close requires evidence, not completion claims.
+```
+
+## Long Runs as Stress Harness
+
+Long specialist/test runs are valuable. They are not just slow validation; they are chaos engineering for orchestration.
+
+They expose:
+
+- context rot
+- missing progress semantics
+- malformed handoff JSON
+- weak issue contracts
+- reviewer evidence gaps
+- process leaks
+- stuck waiting states
+- bad retry/resume semantics
+- tool/runtime drift
+- model-specific failure modes
+
+Future preflight may include soak mode:
+
+```bash
+sp preflight --soak
+```
+
+Possible checks:
+
+- no orphan processes
+- no unconsumed jobs
+- no missing timeline events
+- no invalid handoff JSON
+- no unresolved issue states
+- reviewer can reconstruct evidence
+- contract validator blocks intentionally bad issues
+
+## Relationship to bd
+
+Do not replace bd now.
+
+Near-term, bd remains the tracker and dependency store. The contract-board idea should influence orchestration docs, hooks, and specialist prompts first.
+
+Long-term options:
+
+1. Build a new package that wraps bd initially, adding validator state and contract metadata.
+2. Migrate to a separate issue store only once contract runtime semantics are proven.
+3. Keep export/import compatibility so current repositories can transition gradually.
+
+## Near-Term Work Before the New Package
+
+These ideas should land incrementally in current surfaces:
+
+### `using-specialists` / `using-specialists-v4`
+
+- Make issue contract quality a first-class orchestration step.
+- Refuse vague/title-only beads before dispatch.
+- Define required bead contract fields.
+- Teach orchestrator to improve a bead before launching specialists.
+- Teach chain selection from issue shape.
+- Treat long-run friction as source of follow-up issues.
+
+### `CLAUDE.md` / `AGENTS.md`
+
+- Add issue-contract expectations for any agent creating bd issues.
+- Tell agents not to dispatch specialists from vague issues.
+- Require acceptance criteria, validation, non-goals, and scope for implementation beads.
+
+### Hooks
+
+Potential Claude orchestrator hook set:
+
+- On `bd create`: inspect created issue and nudge if contract is weak.
+- On `bd update`: re-check when description/scope/acceptance criteria changed.
+- On `specialists run` / `sp run`: block or warn if bead is not contract-ready.
+- On session stop: warn about in-progress issues without contract/readiness notes.
+
+Initial hooks should be nudges, not hard blockers, until false positives are understood.
+
+### Mandatory Rules
+
+- Add compact issue-contract mandatory rule for orchestrators and planning specialists.
+- Allow issue-local mandatory rules to flow into specialist prompt at job start.
+
+## Open Questions
+
+- Should readiness validation be deterministic schema checks first, then small-model review?
+- Where should validator verdicts live while bd remains the tracker?
+- How are issue-local mandatory rules represented in bd descriptions without becoming fragile prose?
+- What override levels are acceptable, and who can waive readiness?
+- How does conversations/agent-mail attach clarification threads to a contract?
+- Should evidence be stored as timeline events, issue comments, or a separate evidence graph?
+- How much of this belongs in specialists vs a third package?
+
+## Non-Goals For Now
+
+- Do not build the new issue system now.
+- Do not replace bd now.
+- Do not make all hooks blocking immediately.
+- Do not require every small chore/doc edit to pass heavyweight validation.
+
+The current goal is to preserve the design, then backport the most valuable constraints into existing specialist orchestration docs and prompts.

package/docs/archive/qa-v3.0.2.md

@@ -0,0 +1,220 @@
+---
+title: QA Report — `@jaggerxtrm/specialists` v3.0.2
+scope: qa-v3.0.2
+category: reference
+version: 1.0.0
+updated: 2026-03-19
+domain: []
+---
+
+<!-- INDEX: auto-generated by validate_doc.py — do not edit manually -->
+| Section | Summary |
+|---|---|
+| [Summary](#summary) | | Result | Count | |
+| [Full Results](#full-results) | | ID | Description | Result | Notes | |
+| [Bugs](#bugs) | **Tests:** T16, T17 |
+| [Backend Note](#backend-note) | All original failures on `google-gemini-cli/gemini-3-flash-preview` (T07, T12, T13, T18 in initial runs) are **environme |
+| [Hook Implementation Reference](#hook-implementation-reference) | `specialists-complete |
+| [Bead Policy Reference](#bead-policy-reference) | | Specialist | Permission | Bead auto-created | |
+<!-- END INDEX -->
+
+# QA Report — `@jaggerxtrm/specialists` v3.0.2
+
+**Date:** 2026-03-11
+**Tester:** Claude Sonnet 4.6 (automated QA agent)
+**Environment:** Linux 6.18.13, Node v25.2.1, Bun runtime, project `gitboard`
+**Scope:** Full CLI + MCP surface, background job lifecycle, hook integration, bead integration
+
+---
+
+## Summary
+
+| Result | Count |
+|--------|-------|
+| **PASS** | 23 |
+| **FAIL** | 6 |
+| **SKIP** | 1 |
+| **Total** | 30 |
+
+---
+
+## Full Results
+
+### Phase 0 — Preflight
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| T01 | `specialists version` | **PASS** | Prints `@jaggerxtrm/specialists v3.0.2` |
+| T02 | `specialists install` | **FAIL** | Completely silent — no output under any condition (piped, TTY, `--force`). Cannot verify hook status or count. |
+| T03 | `specialists status` includes Active Jobs | **FAIL** | pi, beads, MCP sections present; Active Jobs section only renders when ≥1 job exists — absent when queue is empty. |
+| T04 | `specialists list` | **PASS** | Lists 9 specialists with model and description. |
+| T05 | `specialists models` | **PASS** | Lists 42 models across 5 providers (anthropic, google-gemini-cli, qwen, qwen-cli, zai). |
+
+### Phase 1 — Foreground Run
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| T06 | `specialists init` | **FAIL** | `.specialists/` pre-existed; `jobs/` and `ready/` subdirs not created by init (lazy-created on first background run). `.gitignore` entry confirmed. |
+| T07 | Foreground run | **PASS** | `report-generator` (haiku-4-5) returned full response in 16.9 s. *gemini-3-flash-preview times out at 180 s in this environment — backend-specific, not a CLI bug.* |
+| T08 | `specialists status` after T07 shows Active Jobs | **FAIL** | Active Jobs section absent when no background jobs in queue (same as T03). |
+
+### Phase 2 — Background Job Lifecycle
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| T09 | `--background` prints `Job started: <id>` | **PASS** | `Job started: 43221b` captured. Parent CLI exits immediately; pi subprocess continues in background. |
+| T10 | `specialists status` shows active JOB_ID | **PASS** | Active Jobs table with id, specialist name, status, elapsed time. |
+| T11 | `specialists feed <JOB_ID>` | **PASS** | Prints `meta`, `text`, `tool_execution_end`, `done` event lines; no crash; exit 0. |
+| T12 | `specialists feed <JOB_ID> --follow` streams to "Job complete." | **PASS** | 30+ live events streamed on `init-session` job; printed `Job complete. Run: specialists result 2d7516` on completion. |
+| T13 | `specialists result <JOB_ID>` prints non-empty output | **PASS** | 8,963-byte markdown report printed, exit 0. *gemini `done` jobs produce 0-byte `result.txt` — backend-specific.* |
+| T14 | `specialists status --job <JOB_ID>` single-job detail | **SKIP** | `--job` flag not implemented; shows full table instead. |
+
+### Phase 3 — Stop / Error Path
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| T15 | `specialists stop <JOB_ID>` | **PASS** | Prints `✓ Sent SIGTERM to PID … (job …)`, exit 0. |
+| T16 | `specialists result` after stop exits 1 with error message | **FAIL** | Exits 1 (correct) but prints "still running" or "still starting" — never "failed" or "error". Status file not updated post-SIGTERM. |
+| T17 | `specialists feed` after stop shows error event | **FAIL** | SIGTERM kills pi process (events freeze, confirmed by EPIPE crash); `status.json` stays `"running"` forever — no error event written. |
+
+### Phase 4 — MCP Tool Surface
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| T18 | MCP `use_specialist` | **PASS** | `report-generator` returned non-empty result in 6.5 s. *gemini times out — backend-specific.* |
+| T19 | MCP `specialist_status` | **PASS** | Returns JSON with `background_jobs` array. |
+| T20 | MCP `list_specialists` | **PASS** | Returns array of 9 entries each with `name`, `model`, `description`. |
+| T21 | MCP `start_specialist` description is deprecated | **PASS** | Description starts with `[DEPRECATED v3] Start a specialist asynchronously…` |
+
+### Phase 5 — Completion Hook
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| T22 | `ready/` empty after completed job | **FAIL** | At check time an unrelated concurrent completed job had an unconsumed marker. Hook is functional (see T23) — timing-dependent. |
+| T23 | Marker created, hook injects on next message | **PASS** | Marker created on job completion; hook consumed it at next `UserPromptSubmit` and emitted correct `{"type":"inject","content":"[Specialist '...' completed …]"}`. |
+
+### Phase 6 — Edge Cases
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| T24 | `specialists result nonexistent` | **PASS** | Prints `No job found: nonexistent`, exit 1. |
+| T25 | `specialists stop nonexistent` | **PASS** | Prints `No job found: nonexistent`, exit 1. |
+| T26 | `specialists result <still-running>` | **PASS** | Prints `Job … is still starting. Use 'specialists feed --job …' to follow.`, exit 1. |
+| T27 | `specialists run nonexistent-specialist` | **PASS** | Prints `Specialist not found: nonexistent-specialist`, exit 1. No stack trace. |
+
+### Bead Integration (supplemental)
+
+| ID | Description | Result | Notes |
+|----|-------------|--------|-------|
+| B1 | MCP `use_specialist` (LOW permission) creates bead | **PASS** | `test-runner` created `forge-dz4`; auto-closed with `COMPLETE, 19373ms, anthropic/claude-haiku-4-5`. `beadId` returned in JSON response. |
+| B2 | CLI `specialists run` (LOW permission) creates bead | **PASS** | `test-runner` created `forge-3hg`; `[bead: forge-3hg]` printed inline during run; `✓ bead forge-3hg  18.9s` footer on completion. |
+| B3 | READ_ONLY specialists skip bead creation | **PASS** | `codebase-explorer`, `report-generator`, `init-session` produced no beads — correct per `auto` policy. |
+
+---
+
+## Bugs
+
+### Bug 1 — SIGTERM doesn't update job status `(HIGH)`
+
+**Tests:** T16, T17
+
+**Symptom:** After `specialists stop <id>`, the job's `status.json` stays `"running"` indefinitely. `specialists result` reports "still running" forever. No error event is written to `events.jsonl`.
+
+**Root cause (confirmed via EPIPE crash log):**
+`specialists run --background` forks the pi subprocess and immediately exits after printing `Job started:`. There is no persistent watcher process. When `specialists stop` sends SIGTERM to the pi PID, pi attempts to write its final event to a now-dead pipe and crashes with an unhandled EPIPE:
+
+```
+Error: write EPIPE
+  at afterWriteDispatched (node:internal/stream_base_commons:159:15)
+  at rpc-mode.js:22  ← pi trying to emit final event
+  at AgentSession._emit (agent-session.js:140:13)
+```
+
+Since no parent process is watching the pi subprocess's exit, `status.json` is never updated to `"error"` or `"cancelled"`.
+
+**Suggested fix:** Spawn a lightweight watcher/daemon per background job that traps the pi process exit (via `SIGCHLD` or `process.on('exit')`) and writes a final `status.json` with `status: "error"` plus an error event to `events.jsonl`.
+
+---
+
+### Bug 2 — `specialists install` is completely silent `(MEDIUM)`
+
+**Test:** T02
+
+**Symptom:** `specialists install` produces zero output under all conditions — interactive TTY, piped, `--force` flag. Cannot verify which hooks are registered or whether anything is up to date.
+
+**Expected:** Per-hook status lines (e.g. `○ specialists-complete.mjs — up to date`) and a final count of registered hooks.
+
+---
+
+### Bug 3 — Active Jobs section absent when queue is empty `(MEDIUM)`
+
+**Tests:** T03, T08
+
+**Symptom:** `specialists status` omits the Active Jobs section entirely when there are zero background jobs.
+
+**Expected:** Active Jobs section always present, rendered empty (e.g. `── Active Jobs ─── (none)`), so users know the feature is available without needing to start a job first.
+
+---
+
+### Bug 4 — `specialists init` doesn't create `jobs/` and `ready/` subdirs `(MEDIUM)`
+
+**Test:** T06
+
+**Symptom:** `specialists init` does not create `.specialists/jobs/` or `.specialists/ready/`. These directories are created lazily on the first background job run.
+
+**Expected:** `init` should create the full directory structure upfront so users can inspect the layout before running any jobs.
+
+---
+
+### Bug 5 — `specialists status --job` not implemented `(LOW)`
+
+**Test:** T14 (SKIP)
+
+**Symptom:** `specialists status --job <id>` shows the full status table instead of a single-job detail view with model, elapsed, bead_id fields.
+
+---
+
+### Bug 6 — `ready/` marker timing `(LOW)`
+
+**Test:** T22
+
+**Symptom:** Markers accumulate in `.specialists/ready/` between user messages. Checking immediately after a job completes may show stale markers from other concurrently completed jobs. The `specialists-complete.mjs` hook only fires at `UserPromptSubmit` boundaries.
+
+**Impact:** Low — markers are always consumed correctly at the next message boundary; no data loss.
+
+---
+
+## Backend Note
+
+All original failures on `google-gemini-cli/gemini-3-flash-preview` (T07, T12, T13, T18 in initial runs) are **environment-specific** — the gemini backend times out at 180 s on every invocation in this environment. All tests pass cleanly with `anthropic/claude-haiku-4-5`. The empty `result.txt` observed on gemini `done` jobs is also backend-specific (pi exits before writing output).
+
+---
+
+## Hook Implementation Reference
+
+`specialists-complete.mjs` installed at `~/.claude/hooks/specialists-complete.mjs`:
+
+- Fires on every `UserPromptSubmit`
+- Reads `.specialists/ready/` for completed-job marker files
+- For each marker: reads `status.json`, emits injection JSON to stdout
+- Injection format: `{"type":"inject","content":"[Specialist '<name>' completed (job <id>, <n>s). Run: specialists result <id>]"}`
+- Deletes marker after injection (fires once per job)
+- Only `status: done` jobs create markers; errored jobs do not
+
+---
+
+## Bead Policy Reference
+
+| Specialist | Permission | Bead auto-created |
+|------------|-----------|-------------------|
+| `auto-remediation` | HIGH | Yes |
+| `bug-hunt` | LOW | Yes |
+| `test-runner` | LOW | Yes (tested) |
+| `codebase-explorer` | READ_ONLY | No |
+| `feature-design` | READ_ONLY | No |
+| `init-session` | READ_ONLY | No |
+| `overthinker` | READ_ONLY | No |
+| `parallel-review` | READ_ONLY | No |
+| `report-generator` | READ_ONLY | No |
+
+`beads_integration` is not set in any specialist YAML — all use the `auto` default (create for LOW/MEDIUM/HIGH, skip for READ_ONLY).