@jaggerxtrm/specialists 3.18.1 → 3.18.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (29) hide show
  1. package/config/mandatory-rules/code-quality-defaults.md +58 -1
  2. package/config/skills/setup-specialists/SKILL.md +1 -1
  3. package/config/skills/using-script-specialists/SKILL.md +5 -5
  4. package/config/skills/using-specialists/SKILL.md +1135 -879
  5. package/config/skills/using-specialists-auto/SKILL.md +21 -21
  6. package/config/specialists/chain-coordinator.specialist.json +63 -0
  7. package/config/specialists/reviewer.specialist.json +1 -1
  8. package/config/specialists/seconder.specialist.json +1 -1
  9. package/dist/asset-contract.json +8 -13
  10. package/dist/index.js +151 -28
  11. package/dist/lib.js +8 -0
  12. package/dist/types/cli/clean.d.ts.map +1 -1
  13. package/dist/types/cli/init.d.ts.map +1 -1
  14. package/dist/types/cli/view.d.ts.map +1 -1
  15. package/dist/types/specialist/control.d.ts.map +1 -1
  16. package/dist/types/specialist/loader.d.ts +12 -0
  17. package/dist/types/specialist/loader.d.ts.map +1 -1
  18. package/dist/types/specialist/process-health.d.ts +16 -1
  19. package/dist/types/specialist/process-health.d.ts.map +1 -1
  20. package/dist/types/specialist/supervisor.d.ts.map +1 -1
  21. package/docs/design/roadmap/chain-templates/README.md +1 -1
  22. package/docs/design/roadmap/chains-prompt-evals.md +3054 -0
  23. package/docs/design/roadmap/enhanced-prd.md +4282 -0
  24. package/docs/design/roadmap/specialists-roadmap.md +7 -7
  25. package/docs/design/xt-pi-role-pi-flag-passthrough.md +174 -0
  26. package/docs/skills.md +4 -13
  27. package/package.json +1 -1
  28. package/config/skills/using-specialists-v2/SKILL.md +0 -766
  29. package/config/skills/using-specialists-v3/SKILL.md +0 -1327
@@ -0,0 +1,3054 @@
1
+ # Specialists Prompt, Chain Context, Observability and Evaluation Modernization
2
+
3
+ ## Product Requirements Document
4
+
5
+ **Status:** Implementation-ready input for Beads decomposition, chain planning and multiplexed execution
6
+ **Date:** 12 July 2026
7
+ **Primary repository:** `xtrm-dev/specialists`
8
+ **Related repositories:** `Jaggerxtrm/console` / Omniforge, `xtrm-dev/xtrm`, `Jaggerxtrm/xtmux`
9
+ **Audience:** Specialists runtime maintainers, agent-orchestration engineers, Console maintainers, evaluation engineers, platform/observability engineers
10
+ **Scope:** Prompt and rule modernization for reviewer, executor, overthinker, seconder, test-engineer and researcher; chain-member identity and context reconstruction; memory retrieval policy; forensic and metrics hardening; historical and continuous evaluation; controlled model and prompt A/B testing; Console integration.
11
+
12
+ **Canonical companions:** `docs/design/roadmap/specialists-roadmap.md` for the bridge-runtime roadmap; `xtrm/docs/channels/channels.md` for future channel semantics; a new dated xtrm reconciliation document for current cross-repository sequencing.
13
+
14
+ ---
15
+
16
+ ## Document purpose
17
+
18
+ This PRD consolidates the complete set of findings and recommendations produced during a detailed audit of the Specialists system. It begins with the original request to reduce the static context payload of six heavy specialist system prompts and expands the scope where the investigation showed that prompt text was only one layer of a larger runtime-composition problem.
19
+
20
+ The document is intentionally comprehensive. It records not only the final recommendations, but also the corrections and refinements made during the investigation. In particular, the first prompt audit correctly identified substantial duplication and role overlap, but initially did not account for every mandatory-rule layer, runtime-injected block, local override, eager skill body or generated output contract. Subsequent repository inspection established the full package-canonical prompt composition and revealed additional sources of context bloat, policy conflict and observability ambiguity. This PRD incorporates that corrected understanding.
21
+
22
+ ## Document authority and execution contract
23
+
24
+ This PRD is the implementation and decomposition source of truth for the prompt, chain-context, memory, telemetry and evaluation modernization program. It is intentionally more operational than `specialists-roadmap.md`: the roadmap remains authoritative for the bridge-runtime architecture and its Opportunities, while this PRD defines the concrete work packages, evaluation gates, parallelization boundaries and promotion criteria needed to implement the overlapping modernization safely.
25
+
26
+ The following precedence applies when local agents encounter apparently conflicting prose:
27
+
28
+ 1. **Current code and executable schemas** define what exists today. Agents must verify before modifying.
29
+ 2. **`specialists-roadmap.md`** defines the canonical bridge direction, retained Opportunities, Substrate read-forward and Beads/molecule mental model.
30
+ 3. **This PRD** defines implementation scope, sequencing, grader requirements, work-package boundaries and readiness gates for the modernization program.
31
+ 4. **`xtrm/docs/channels/channels.md`** defines the future semantic messaging contract. No bridge implementation may invent an incompatible message protocol.
32
+ 5. **Dated `_meta` reconciliation documents** are historical snapshots. They inform drift analysis but do not override newer code, roadmap status or this PRD.
33
+ 6. **Archived Iron/friction documents** are design archaeology only unless the canonical roadmap explicitly retains a requirement.
34
+
35
+ This document is ready to be supplied to a local planning agent for Pass-1/Pass-2 decomposition. The planning agent must create Beads work contracts and dependency edges; it must not silently redesign the architecture, duplicate normative text into every Bead, or start implementation during decomposition. The decomposition output must preserve the work-package identifiers and dependency constraints defined in Sections 14 and 15.
36
+
37
+ The implementation program is split into three coordinated tracks:
38
+
39
+ * **Track A — Specialists bridge runtime:** the critical chain-first spine and remaining roadmap work on the live runtime.
40
+ * **Track B — xtrm Stage 0, Substrate and Channels:** daemon/state-store and semantic messaging work owned by xtrm, not reimplemented ad hoc in Specialists.
41
+ * **Track C — Prompt, context, telemetry and eval modernization:** the program defined by this PRD.
42
+
43
+ Track C starts immediately with telemetry integrity and historical baselining. Its chain-aware portions consume Track A foundations. Autonomous lateral remediation consumes Channels 0.1 in Track B when that layer exists.
44
+
45
+ The required result is not merely shorter prompts. The required result is a measurable, versioned and chain-aware agent runtime in which:
46
+
47
+ 1. Each specialist has a small, stable role identity and a clear decision boundary.
48
+ 2. Shared policies have one source of truth and are injected only when applicable.
49
+ 3. Chain membership, upstream evidence and downstream obligations are explicit.
50
+ 4. Historical memories are retrieved deliberately rather than pushed indiscriminately.
51
+ 5. Runtime telemetry accurately represents thinking, tools, turns, payload, evidence and chain state.
52
+ 6. Existing and future runs can be evaluated using deterministic, model-based and human graders.
53
+ 7. Prompt and model changes are promoted through controlled paired experiments rather than intuition.
54
+ 8. Console provides historical, live and experiment-specific evaluation surfaces without becoming a second telemetry producer.
55
+
56
+ # 1. Executive summary
57
+
58
+ The Specialists architecture already has strong primitives: role-specific agents, Beads contracts, worktree isolation, mandatory gates, structured output schemas, `xtrm.forensic.v1`, per-repository observability databases, Prometheus projection and a Console materializer. The principal problem is that these primitives evolved in parallel and currently overlap in ways that increase context size and weaken semantic clarity.
59
+
60
+ The six audited specialists carry large system prompts, in some cases more than 10,000 tokens once all layers are counted. The largest source is not always the role prompt itself. A non-`bare` specialist can receive all of the following:
61
+
62
+ * its `prompt.system`;
63
+ * a task template and Bead context;
64
+ * completed dependency descriptions and notes;
65
+ * required and default mandatory rules;
66
+ * specialist-specific mandatory rules and inline rules;
67
+ * eager skill bodies passed through Pi;
68
+ * a runtime Specialist Run Context block;
69
+ * a global output-style directive;
70
+ * a hardcoded GitNexus workflow mandate;
71
+ * static Beads workflow and close instructions;
72
+ * filtered memory injection;
73
+ * an optional GitNexus pre-query snapshot;
74
+ * a generated output contract based on response format, output type and output schema;
75
+ * reviewer-specific diff and lineage instructions;
76
+ * tool definitions and harness framing.
77
+
78
+ This composition creates four classes of defect.
79
+
80
+ **Redundancy:** The same policy is expressed in a system prompt, mandatory rule, task template and runner injection. Executor receives multiple GitNexus mandates, multiple scope rules and multiple handoff requirements. Reviewer repeats seconder work and restates an output format already generated by the runner. Test-engineer repeats its source boundary in inline rules, system prompt and task template.
81
+
82
+ **Contradiction:** The generated output contract, handoff-schema rule and role-specific output prose use incompatible status and verdict vocabularies. Global worktree and close rules are injected into read-only or external-research roles where they are irrelevant or conflicting. A global “stop if local evidence is missing” rule conflicts with researcher’s external-evidence mandate.
83
+
84
+ **Missing chain identity:** The architecture defines specialists as members of a chain, but the role prompts mostly treat them as isolated agents. A reviewer may need executor, seconder, test-engineer, test-runner, obligations and security evidence, yet dependency preloading is bounded and does not guarantee the full upstream chain. The agent is not deterministically told its chain, position, root contract, upstream members, completed gates or downstream handoff.
85
+
86
+ **Insufficient evaluation discipline:** Observability captures substantial raw data, but the system lacks a first-class evaluation control plane. It cannot yet reliably answer whether one prompt or model is better for a role, whether chain-awareness improved outcomes, whether memory retrieval was necessary, or whether a shorter prompt changed false-PASS rates. Some current metrics are also unsuitable for A/B decisions until corrected, notably thinking duration and tool-call counting.
87
+
88
+ The modernization is therefore organized into five tightly related programs:
89
+
90
+ 1. **Prompt and policy consolidation.** Reduce the six core role prompts by approximately 45–90%, depending on role, while preserving role-specific capability. Move deterministic policies to runtime, shared behavior to mandatory rules and deep procedures to on-demand skills.
91
+ 2. **Chain participant context.** Inject a compact, deterministic chain identity and pointer block. Teach specialists how to reconstruct only the necessary upstream context using Beads and Specialists commands. Extend handoffs with decisions, evidence, assumptions and downstream attention.
92
+ 3. **Pull-based memory.** Remove eager semantic memory dumps. Provide a short retrieval rule and targeted `bd memories`/`bd recall` workflow, with provenance and observability.
93
+ 4. **Telemetry hardening.** Correct tool accounting, model thinking spans, forensic-to-Prometheus wiring and activity semantics. Add metrics required for role, chain and evaluation analysis.
94
+ 5. **Evaluation platform.** Add eval suites, cases, experiments, trials, graders, scores and pairwise comparisons. Support retrospective scoring of existing per-repo databases, automatic post-run and post-chain deterministic evaluation, controlled replay benchmarks and Console visualization.
95
+
96
+ The recommended implementation order is telemetry integrity first, then evaluation core, then prompt/chain/memory experiments. This prevents decisions from being made with metrics that are known to be ambiguous.
97
+
98
+ # 2. Background and verified current-state architecture
99
+
100
+ ## 2.1 Specialist roles in scope
101
+
102
+ The first modernization wave covers six specialists:
103
+
104
+ | Role | Intended responsibility | Current strategic position |
105
+ | --------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------ |
106
+ | `executor` | Implement the Bead contract in the assigned worktree | Primary production writer |
107
+ | `seconder` | Cheap bounded pre-QA gate for scope compliance and implementation sanity | Immediately after writer |
108
+ | `test-engineer` | Author tests, fixtures, smoke/E2E assets and telemetry assertions from the actual diff | Between seconder and test-runner |
109
+ | `reviewer` | Final evidence-based release verdict and Release Checklist | Final gate |
110
+ | `researcher` | Gather current external evidence and synthesize findings | Advisor or standalone investigation |
111
+ | `overthinker` | Stress-test uncertain or high-impact decisions | Advisor, premortem or standalone decision review |
112
+
113
+ The role separation is fundamentally sound and should not be collapsed. The modernization should make these boundaries clearer, not replace them with a general-purpose super-agent.
114
+
115
+ ## 2.2 Canonical production-diff chain
116
+
117
+ The current design canon defines the production pipeline as:
118
+
119
+ ```text
120
+ writer (executor or debugger)
121
+ -> seconder
122
+ -> test-engineer
123
+ -> test-runner
124
+ -> security-auditor when required
125
+ -> obligations-scanner
126
+ -> reviewer
127
+ -> release decision
128
+ ```
129
+
130
+ The formula catalog represents a chain as a Beads molecule. The molecule is the chain identity. It contains a root child carrying the overall change contract and step children carrying role-specific mandates. `needs` creates blocking relationships; labels can carry richer semantic edges such as `validates`, `informs`, `implements`, `discovered-from` and `parent-child`.
131
+
132
+ The `code-standard` formula explicitly states that reviewer inputs include the executor diff, seconder verdict, test evidence, obligations output and root contract. The `code-with-advisors` formula adds explorer, researcher and overthinker outputs before executor and expects the final reviewer to consume advisor findings as well.
133
+
134
+ ## 2.3 Prompt composition at runtime
135
+
136
+ The effective specialist prompt is not equivalent to `prompt.system`. It is composed from multiple sources.
137
+
138
+ ### System-side content
139
+
140
+ The Pi session receives:
141
+
142
+ 1. Rendered `prompt.system`.
143
+ 2. A Specialist Run Context block that distinguishes the specialist from a human developer and controls Bead lifecycle behavior.
144
+ 3. A global output-style directive.
145
+ 4. A GitNexus workflow mandate when the repository is indexed.
146
+ 5. Static Beads workflow rules and a close checklist.
147
+ 6. Filtered Beads memories when a Bead is present.
148
+ 7. An optional GitNexus pre-query snapshot.
149
+ 8. A generated output contract derived from `response_format`, `output_type` and `prompt.output_schema`.
150
+ 9. Pi-native skill files from `prompt.skill_inherit` and `skills.paths`.
151
+ 10. Tool and extension definitions resolved from the manifest catalog and permission tier.
152
+
153
+ ### Task-side content
154
+
155
+ The task prompt receives:
156
+
157
+ 1. The rendered role task template.
158
+ 2. The root or step Bead contract.
159
+ 3. Notes attached to the current Bead.
160
+ 4. Completed blocking dependencies, recursively up to the configured context depth.
161
+ 5. Runtime worktree boundary rules.
162
+ 6. Required, default, specialist-specific and inline mandatory rules.
163
+ 7. Reviewer-specific injected diff context where applicable.
164
+ 8. Pre-script output and invocation variables.
165
+
166
+ This distinction matters for optimization. A small `prompt.system` does not guarantee a small first-turn input. The API-billed first-turn input must remain the source of truth, while payload component events are used to identify which layer contributes the bloat.
167
+
168
+ ## 2.4 Mandatory-rule resolution
169
+
170
+ The canonical rule loader combines:
171
+
172
+ 1. required template sets;
173
+ 2. default template sets unless default globals are disabled;
174
+ 3. specialist-declared template sets;
175
+ 4. specialist inline rules.
176
+
177
+ The package-level index currently requires `core-session-boundary` and defaults to `git-workflow-safe`. Specialist-specific rules then add executor delivery, code-quality defaults, GitNexus and Serena instructions, handoff schema, Bead ID discipline, research routing and other role-specific behavior.
178
+
179
+ The package-canonical rules can be audited from the repository. The effective installation may still differ because user or repository overlays and global user configuration can override or append content. Therefore all implementation and experiment tooling must fingerprint and capture the **resolved** configuration, not assume package defaults.
180
+
181
+ The authoritative inspection path is conceptually:
182
+
183
+ ```bash
184
+ sp config show <specialist> --resolved
185
+ sp list-rules --specialist <specialist> --json
186
+ ```
187
+
188
+ ## 2.5 Skills
189
+
190
+ Among the six package-canonical specialist files inspected, `test-engineer` declares `test-planning`; the other five do not declare explicit skill paths. The corresponding package-canonical skill body was not present at the expected repository path during the audit. It may resolve from an installation/user layer or be stale. This is a concrete reason that eval fingerprints must capture the activated skill path and body hash at runtime.
191
+
192
+ Skill bodies are eagerly injected by Pi and billed as input tokens, while the current payload breakdown records only skill labels or paths. Their real contribution must be derived from first-turn token use or measured directly by hashing and tokenizing the activated files.
193
+
194
+ ## 2.6 Bead dependency context
195
+
196
+ `buildBeadContext` includes the current Bead title, description, parent and notes. It also includes descriptions and notes of completed blocking dependencies. Dependency traversal is recursive and bounded by a configured depth, currently defaulting to three in the runner path.
197
+
198
+ This is useful but insufficient as a chain context contract:
199
+
200
+ * it follows only completed `blocks` dependencies, not all semantic edges;
201
+ * a final reviewer in a long chain may not receive executor or seconder evidence within depth three;
202
+ * security gates inserted outside the simple blocking path may be omitted;
203
+ * the current parent label can describe a chain molecule as a “Parent epic”, conflating hierarchy levels;
204
+ * the agent is not told whether the preloaded window is complete.
205
+
206
+ ## 2.7 Observability storage
207
+
208
+ Each repository has an observability database. Core tables include:
209
+
210
+ * `specialist_jobs` for durable run state and startup payload;
211
+ * `specialist_events` for canonical timeline events;
212
+ * `specialist_results` for final or latest completed output;
213
+ * `specialist_job_metrics` for aggregated run metrics;
214
+ * `specialist_forensic_events` for `xtrm.forensic.v1` envelopes;
215
+ * chain and epic linkage tables;
216
+ * node run, member, event and memory tables;
217
+ * memory cache and FTS tables.
218
+
219
+ `specialist_job_metrics` persists identity, timestamps, active and waiting time, turns, tools, token trajectory, context trajectory, stall gaps, run-complete data and startup payload.
220
+
221
+ Historical job directories can be imported. Aggregate metrics can be recomputed from persisted timeline events. This makes retrospective evaluation feasible for a large subset of dimensions.
222
+
223
+ ## 2.8 Forensic event model
224
+
225
+ `xtrm.forensic.v1` is already suitable as the universal event envelope. It supports:
226
+
227
+ * resource identity, including participant role and model;
228
+ * high-cardinality correlation, including job, Bead, chain, turn, tool call, trace, commit and eval IDs;
229
+ * structured event body;
230
+ * redaction state and rules;
231
+ * optional trace, OTel, links and diagnostics.
232
+
233
+ The schema deliberately forbids high-cardinality and sensitive values from Prometheus labels. Detailed evidence stays in forensic storage and evidence references, while Prometheus remains a bounded aggregate projection.
234
+
235
+ ## 2.9 Current Prometheus projection
236
+
237
+ The current projection emits operational metrics for job state, queue depth, processes, worktrees, terminal job count, job duration, active runtime, chain count and duration, waiting time, turns, context utilization, tool calls and token direction.
238
+
239
+ The renderer also contains support for forensic-derived metrics such as gate verdicts, evidence references, MCP operations, identity, policy and eval runs/scores. However, the normal `sp metrics --prometheus` path currently reads statuses and job metrics but does not supply forensic events to the renderer. Those metric families are therefore not reliably present in the live scrape despite being implemented in the rendering layer.
240
+
241
+ ## 2.10 Console architecture
242
+
243
+ Console already has the correct ownership boundary for this project:
244
+
245
+ * Specialists owns runtime events, forensic envelopes, projections, evaluation semantics and evidence production.
246
+ * The materializer reads per-repo Specialists databases and projects job rows, token metrics, forensic events and evidence references into its read model.
247
+ * Console owns queries, dashboards, drilldowns, experiment UX and operator decisions.
248
+ * Prometheus is used for aggregate operational symptoms; forensic/evidence storage is used for exact run and trial detail.
249
+
250
+ The Console observability design already includes an `eval` signal kind, eval lookup queries, eval result summaries and an AgentOps Governance dashboard pack. A new standalone application is therefore not required.
251
+
252
+ # 3. Problem statement
253
+
254
+ ## 3.1 Prompt payload and instruction competition
255
+
256
+ The six specialist prompts are costly not merely because they are verbose, but because they compete with other instructions. Long prompts contain universal programming opinions, CLI manuals, detailed output examples, role procedures, shared policy and personality cues. The runner and mandatory-rule layers then inject additional versions of the same concepts.
257
+
258
+ The result is an instruction graph with multiple authorities. The model must infer which version of scope discipline, output schema, Git workflow or evidence ordering is current. This creates hidden failure modes:
259
+
260
+ * a stale system-prompt section can override a newer shared rule;
261
+ * examples become treated as requirements;
262
+ * a role over-applies generic guidance that conflicts with repository conventions;
263
+ * a generated schema and a prose schema disagree;
264
+ * a global rule unintentionally constrains a specialist for which it was not designed;
265
+ * critical role instructions are buried beneath tool manuals and style advice.
266
+
267
+ ## 3.2 Role overlap
268
+
269
+ Reviewer is nominally phase-two and should consume seconder results, but its prompt reconstructs atomic requirements and repeats broad compliance review. Executor carries a complete TypeScript and clean-code manual despite being a general implementation specialist. Overthinker repeats the four-phase rule already injected separately. Researcher embeds command manuals already represented in routing rules. Test-engineer repeats the same source-boundary rule three times.
270
+
271
+ This overlap increases both token use and behavioral ambiguity. The modernization must enforce the principle that each role owns a narrow decision and trusts upstream work conditionally rather than starting from zero.
272
+
273
+ ## 3.3 Missing participant identity
274
+
275
+ Specialists are chain participants, but the current role identity is mostly local: “you are reviewer”, “you are executor”. The system does not consistently say:
276
+
277
+ * which chain the specialist belongs to;
278
+ * which formula/template was resolved;
279
+ * which root contract governs the overall work;
280
+ * which step contract governs the current mandate;
281
+ * what upstream roles have completed;
282
+ * what evidence each upstream role produced;
283
+ * which gates remain pending;
284
+ * which downstream role will consume this handoff;
285
+ * which evidence is preloaded and which is available on demand.
286
+
287
+ Without this identity, agents either duplicate upstream work or fail to reconstruct necessary context.
288
+
289
+ ## 3.4 Eager memory context
290
+
291
+ The runtime currently retrieves a bounded set of memories based on Bead keywords. Although smaller than earlier full-memory injection designs, the retrieval is still eager and lexical. It can inject irrelevant or stale knowledge into fully specified tasks.
292
+
293
+ The desired model is capability-based memory: the agent knows that historical knowledge exists, recognizes when prior decisions or incidents matter, and retrieves targeted memory with provenance. This should be measured rather than assumed.
294
+
295
+ ## 3.5 Telemetry ambiguity
296
+
297
+ Several current metrics cannot yet support rigorous comparisons:
298
+
299
+ * thinking events lack start/end phases and duration;
300
+ * thinking character counts are cumulative rather than per delta or final segment;
301
+ * reasoning tokens are provider-dependent and not equivalent to reasoning time;
302
+ * tool metrics may count lifecycle events rather than unique calls;
303
+ * the Pi callback path can report both tool-call construction and tool execution;
304
+ * a query checks `think` while the canonical event type is `thinking`;
305
+ * forensic-derived Prometheus metrics are not wired into the default scrape path;
306
+ * historical rows have heterogeneous schema completeness.
307
+
308
+ ## 3.6 Evaluation gap
309
+
310
+ Existing observability can describe cost and activity, but it does not yet determine quality. Reviewer PASS rate, for example, is not equivalent to reviewer accuracy. A model may PASS more often because it misses defects. Historical cohort comparisons can be confounded by task difficulty. A shorter prompt can save tokens while increasing false PASS rates. An LLM judge can prefer verbosity or the first answer shown.
311
+
312
+ A first-class evaluation platform is required to establish repeatable evidence for prompt, rule, runtime and model changes.
313
+
314
+ # 4. Goals, non-goals and design principles
315
+
316
+ ## 4.1 Goals
317
+
318
+ The project shall:
319
+
320
+ 1. Reduce static and eager context for the six target roles without reducing task success or role compliance.
321
+ 2. Establish one source of truth for every shared policy and output contract.
322
+ 3. Make chain membership and context reconstruction explicit and deterministic.
323
+ 4. Replace indiscriminate memory push with measured, targeted recall.
324
+ 5. Correct telemetry semantics before using them for promotion decisions.
325
+ 6. Support retrospective evaluation of existing runs with explicit completeness labels.
326
+ 7. Support automatic deterministic evaluation after every run and every chain.
327
+ 8. Support controlled paired A/B testing across models, prompts, rules and runtime versions.
328
+ 9. Support external software-engineering benchmarks and internal SWE-bench-style fixtures.
329
+ 10. Integrate results into Console with historical, live and experiment-specific views.
330
+ 11. Preserve low-cardinality Prometheus discipline and forensic drilldown.
331
+ 12. Make every experiment reproducible through candidate and environment fingerprints.
332
+
333
+ ## 4.2 Non-goals
334
+
335
+ The project shall not:
336
+
337
+ * merge the six roles into a single generic agent;
338
+ * expose private chain-of-thought or require models to reveal it;
339
+ * treat model-written success claims as environment truth;
340
+ * make LLM graders authoritative blocking gates before calibration;
341
+ * add high-cardinality IDs to Prometheus labels;
342
+ * create a second telemetry or Console application unnecessarily;
343
+ * use historical cohort averages as a substitute for controlled replay;
344
+ * force every role to retrieve memories on every run;
345
+ * inject entire chain histories into every participant;
346
+ * replace repository conventions with a universal clean-code doctrine;
347
+ * couple Console materialization to source-of-truth mutation.
348
+
349
+ ## 4.3 Design principles
350
+
351
+ ### Intrinsic role identity in the system prompt
352
+
353
+ A role prompt should contain only the mandate, boundaries, evidence hierarchy and decision rules intrinsic to that role.
354
+
355
+ ### Shared policy in one reusable layer
356
+
357
+ Cross-role policies belong in mandatory rules or runtime enforcement, not copied prose.
358
+
359
+ ### Deterministic decisions in code
360
+
361
+ Scrutiny floors, required gates, path matching, worktree boundaries, output-schema validation and chain identity should be computed by the runtime.
362
+
363
+ ### Deep procedure on demand
364
+
365
+ CLI manuals and lengthy playbooks belong in focused skills or tool descriptions, loaded only when needed.
366
+
367
+ ### Identity pushed, content pulled
368
+
369
+ The runtime should inject compact chain identity, member indexes and evidence pointers. The specialist should retrieve substantive upstream content only when its role requires it.
370
+
371
+ ### Outcome over self-report
372
+
373
+ Tests, environment state, diff and evidence artifacts outrank the model’s final statement.
374
+
375
+ ### Hard gates before weighted scores
376
+
377
+ A role-boundary violation or critical false PASS cannot be compensated by token efficiency.
378
+
379
+ ### Observe before enforce
380
+
381
+ New graders and policies begin in shadow mode. Enforcement requires measured precision and low false-positive rates.
382
+
383
+ ### Version every evaluator
384
+
385
+ Prompt, rules, skills, runtime, grader, dataset and environment versions must be recorded.
386
+
387
+ # 5. Workstream A — Prompt and policy modernization
388
+
389
+ ## 5.1 Target prompt architecture
390
+
391
+ Each target specialist should resolve into five conceptual layers:
392
+
393
+ ```text
394
+ role core system prompt
395
+ + applicable shared mandatory rules
396
+ + compact runtime context
397
+ + role-specific task contract
398
+ + on-demand procedural skills/tools
399
+ ```
400
+
401
+ The role core should generally remain within approximately 180–600 tokens. This is a design target, not a hard limit. Content is retained when it materially changes decisions.
402
+
403
+ The target reductions from the initial audit are:
404
+
405
+ | Specialist | Estimated system-prompt reduction | Target core size |
406
+ | ------------- | --------------------------------: | ---------------: |
407
+ | Reviewer | 70–80% | 350–600 tokens |
408
+ | Executor | 80–90% | 250–450 tokens |
409
+ | Overthinker | 60–75% | 180–300 tokens |
410
+ | Seconder | 45–60% | 220–350 tokens |
411
+ | Test-engineer | 55–70% | 250–400 tokens |
412
+ | Researcher | 70–85% | 220–350 tokens |
413
+
414
+ The real KPI is first-turn input and outcome, not the length of the JSON field.
415
+
416
+ ## 5.2 Output contract consolidation
417
+
418
+ The runtime already generates a base output contract and merges output-type extensions and specialist output schemas. This must become the only normative output definition.
419
+
420
+ Required changes:
421
+
422
+ 1. Normalize base statuses. Choose one canonical vocabulary, recommended:
423
+
424
+ ```text
425
+ success | partial | failed | waiting
426
+ ```
427
+
428
+ 2. Normalize reviewer verdicts, recommended uppercase at presentation boundaries and a canonical normalized enum in storage.
429
+ 3. Convert example-shaped output schemas into valid JSON Schema.
430
+ 4. Remove prose instructions that restate field lists already represented in schema.
431
+ 5. Remove or radically shrink `per-turn-handoff-schema`; it may explain semantics but must not define a competing schema.
432
+ 6. Emit schema-validation results as deterministic evaluation events.
433
+ 7. Version the merged schema and store its hash on each run.
434
+
435
+ ## 5.3 Reviewer modernization
436
+
437
+ ### Current defects
438
+
439
+ * Repeats requirement normalization and compliance work assigned to seconder.
440
+ * Uses an adversarial persona that may increase cosmetic findings and severity inflation.
441
+ * Contains a full CLI and evidence-reconstruction manual.
442
+ * Computes scrutiny and gate requirements in prompt text rather than consuming resolved policy.
443
+ * Repeats output and Release Checklist formats.
444
+ * Duplicates the `seconder ran` checklist item.
445
+ * Contains obligations and simplification procedures that should be separate gate/policy inputs.
446
+
447
+ ### Target responsibility
448
+
449
+ Reviewer is the final evidence-based release gate. It should:
450
+
451
+ * consume the root contract and resolved chain state;
452
+ * consume final upstream gate evidence;
453
+ * inspect unresolved findings, critical requirement coverage and high-risk changed surfaces;
454
+ * detect contradictions or missing required evidence;
455
+ * produce PASS, PARTIAL or FAIL with evidence-linked findings;
456
+ * use delta mode for re-review after PARTIAL;
457
+ * never edit files;
458
+ * avoid preference-only findings.
459
+
460
+ ### Runtime-provided inputs
461
+
462
+ ```text
463
+ reviewed_job_id
464
+ chain_id
465
+ root_bead_id
466
+ current_step_bead_id
467
+ resolved_scrutiny
468
+ required_gates
469
+ gate_results
470
+ upstream_job_ids
471
+ injected diff/evidence refs
472
+ previous reviewer findings for re-review
473
+ ```
474
+
475
+ ### Procedures moved out of core prompt
476
+
477
+ * job and feed lookup command details;
478
+ * scrutiny path tables;
479
+ * obligation marker vocabulary;
480
+ * Release Checklist rendering;
481
+ * output JSON examples;
482
+ * tool-specific GitNexus syntax;
483
+ * full Ddiff procedure, except a concise semantic statement.
484
+
485
+ ## 5.4 Executor modernization
486
+
487
+ ### Current defects
488
+
489
+ The executor prompt contains an extensive universal style guide: naming rules, function-length limits, nesting limits, TypeScript and Zod requirements, error doctrine, concurrency doctrine, performance doctrine, security rules, testing advice and anti-pattern tables.
490
+
491
+ Several rules conflict with one another or with repository-specific practice. Examples include “extract at the second duplication” versus “wait until the third use”, “fix adjacent smells” versus strict scope, mandatory Zod for every external input, hard limits on function length and arguments, and “never return null”.
492
+
493
+ ### Target responsibility
494
+
495
+ Executor should:
496
+
497
+ * resolve and obey the Bead contract;
498
+ * remain within the assigned worktree and scope;
499
+ * inspect relevant code and dependency impact;
500
+ * make the smallest correct change;
501
+ * follow local language/framework/repository conventions;
502
+ * reuse existing code and dependencies;
503
+ * avoid speculative abstraction and unrelated cleanup;
504
+ * perform declared validation and focused checks;
505
+ * return a complete evidence-based handoff.
506
+
507
+ Language-specific standards should be loaded from repository or on-demand skills, not hardcoded globally.
508
+
509
+ ### Validation policy
510
+
511
+ Executor may run focused validation required by the Bead. Broad authoritative suite execution remains owned by test-runner unless explicitly required. The current blanket “do not run tests” language should be replaced with ownership-aware wording.
512
+
513
+ ## 5.5 Overthinker modernization
514
+
515
+ ### Current defects
516
+
517
+ * Repeats the four-phase rule already injected separately.
518
+ * Forces a multi-phase structure even for simple reversible decisions.
519
+ * Uses “multi-persona chain-of-thought” framing.
520
+ * Contains hardcoded command permissions that are already enforced by tool tiers.
521
+
522
+ ### Target responsibility
523
+
524
+ Overthinker is a read-only decision-review specialist. It should scale depth to consequence and uncertainty:
525
+
526
+ * simple/reversible decision: direct recommendation;
527
+ * uncertain/costly decision: compare viable alternatives and trade-offs;
528
+ * high-impact/irreversible decision: premortem, disconfirming evidence and confidence.
529
+
530
+ It reports conclusions, assumptions, alternatives, risks and validation steps, not private reasoning traces.
531
+
532
+ ## 5.6 Seconder modernization
533
+
534
+ ### Current strengths
535
+
536
+ Seconder is already the best-bounded prompt in the group. Its split between scope verdict and quality verdict is useful. `UNCLEAR` is a valuable state when evidence is truly insufficient.
537
+
538
+ ### Current defects
539
+
540
+ * smell lists repeat shared code-quality rules;
541
+ * output schema appears both in prose and JSON;
542
+ * tool-call budget belongs in runtime configuration;
543
+ * evidence hierarchy can be shared with other gates;
544
+ * eager context expansion could make it a second reviewer.
545
+
546
+ ### Target responsibility
547
+
548
+ Seconder answers one question: whether the writer output is sufficiently compliant and implementation-sane to justify expensive QA. It should not become a release, security or broad architecture review.
549
+
550
+ ## 5.7 Test-engineer modernization
551
+
552
+ ### Current defects
553
+
554
+ * source boundary repeated in inline rules, system prompt and task template;
555
+ * mode inferred from free text rather than explicit dispatch metadata;
556
+ * output schema is example-shaped rather than formal JSON Schema;
557
+ * report-section requirements duplicate generated output contract;
558
+ * possible stale or installation-local `test-planning` skill reference.
559
+
560
+ ### Target responsibility
561
+
562
+ Test-engineer receives an explicit mode:
563
+
564
+ ```text
565
+ test_only | post_implementation
566
+ ```
567
+
568
+ It authors tests and testing assets from actual behavior and diff evidence. Production edits are forbidden unless an explicitly named helper/export change is authorized. When source behavior is wrong, it returns `source_bug_suspected` with evidence.
569
+
570
+ ## 5.8 Researcher modernization
571
+
572
+ ### Current defects
573
+
574
+ * embeds detailed tool and CLI manuals;
575
+ * duplicates research routing rules;
576
+ * absolute “never answer from memory” language is too broad;
577
+ * does not sufficiently separate internal project memory from external evidence.
578
+
579
+ ### Target responsibility
580
+
581
+ Researcher applies an epistemic policy:
582
+
583
+ * use current authoritative sources for unstable claims;
584
+ * prefer primary documentation and source repositories;
585
+ * distinguish facts, inference and unknowns;
586
+ * record dates and versions;
587
+ * verify consequential claims where practical;
588
+ * stop when success criteria are met;
589
+ * report tool failure rather than invent evidence.
590
+
591
+ Tool syntax stays in tool descriptions or focused skills.
592
+
593
+ ## 5.9 Runner-injected content review
594
+
595
+ The prompt modernization must include runner blocks, not only specialist JSON.
596
+
597
+ ### Specialist Run Context
598
+
599
+ Retain, but split into applicable fragments. Bead claim/close instructions should not be injected into every read-only or chain-step role when the orchestrator owns lifecycle.
600
+
601
+ ### Output-style directive
602
+
603
+ The current global “smart caveman” rule may help agent-to-agent brevity but conflicts with evidence uncertainty and readable research/review output. Replace it with a bounded, role-aware style contract:
604
+
605
+ ```text
606
+ Be concise and technical. Preserve uncertainty, evidence qualifiers and exact identifiers. Do not omit required rationale or schema fields.
607
+ ```
608
+
609
+ Style effects should be evaluated by role.
610
+
611
+ ### GitNexus mandate
612
+
613
+ Remove duplication between runner hardcoding, mandatory rules and task templates. Prefer a single runtime-generated tool policy that is aware of repository index health and role. Read-only roles should not receive “before editing” language.
614
+
615
+ ### Static Beads close checklist
616
+
617
+ Do not inject `git add`, `commit`, `push` into read-only roles. Make it writer-specific and consistent with auto-commit ownership.
618
+
619
+ ### Core session boundary
620
+
621
+ Split local worktree boundary from evidence-retrieval policy. Researcher must be permitted to retrieve external evidence. Missing local artifacts should not globally prohibit external or chain-context lookup.
622
+
623
+ ## 5.10 Prompt fingerprinting
624
+
625
+ Every run must record:
626
+
627
+ ```text
628
+ system_prompt_hash
629
+ task_template_hash
630
+ merged_output_schema_hash
631
+ mandatory_rules_hash
632
+ mandatory_rule_ids and versions
633
+ skill_paths and skill_body_hashes
634
+ runner_injection_version
635
+ tool_catalog_version
636
+ resolved_tool_set_hash
637
+ specialist_version
638
+ ```
639
+
640
+ The system should optionally store redacted prompt component metadata and token counts, not raw sensitive prompt text in Prometheus.
641
+
642
+ # 6. Workstream B — Chain participant identity and context reconstruction
643
+
644
+ ## 6.1 Product requirement
645
+
646
+ A specialist dispatched as a chain step must be told that it is a chain participant. The runtime shall provide enough identity to orient the agent without injecting the complete chain history.
647
+
648
+ ## 6.2 Chain context envelope
649
+
650
+ Recommended task/system injection:
651
+
652
+ ```xml
653
+ <chain-context version="1">
654
+ <chain-id>...</chain-id>
655
+ <chain-template>code-standard</chain-template>
656
+ <root-bead-id>...</root-bead-id>
657
+ <current-step-bead-id>...</current-step-bead-id>
658
+ <current-role>reviewer</current-role>
659
+ <current-class>gate</current-class>
660
+ <position>final</position>
661
+ <scrutiny stated="medium" effective="high" />
662
+ <upstream complete="false">
663
+ <step role="executor" bead-id="..." job-id="..." status="completed" />
664
+ <step role="seconder" bead-id="..." job-id="..." status="completed" />
665
+ </upstream>
666
+ <pending>
667
+ ...
668
+ </pending>
669
+ <preloaded-context depth="3" complete="false" />
670
+ <worktree-path>...</worktree-path>
671
+ </chain-context>
672
+ ```
673
+
674
+ The envelope must be generated from persisted chain shape and Beads relationships, not inferred by the model.
675
+
676
+ ## 6.3 Shared chain-participant rule
677
+
678
+ A shared mandatory rule should teach the operational semantics:
679
+
680
+ * the root change contract governs overall outcome;
681
+ * the current step contract governs mandate and boundaries;
682
+ * upstream handoffs are evidence, not overriding authority;
683
+ * consume preloaded dependency context first;
684
+ * reconstruct only missing necessary context;
685
+ * prefer `sp result` for final handoff and `sp feed --json` only for claims that require trace verification;
686
+ * use `bd show`, `bd dep list` and `bd dep tree` to understand issue relationships;
687
+ * do not repeat upstream work unless the current role validates it, evidence is missing or artifacts contradict it;
688
+ * report contradictions explicitly;
689
+ * produce a downstream-usable handoff.
690
+
691
+ The rule should apply only when a real chain context exists or the Bead is marked `kind:step`.
692
+
693
+ ## 6.4 Context retrieval hierarchy
694
+
695
+ Recommended hierarchy:
696
+
697
+ 1. Injected chain context and authoritative runtime metadata.
698
+ 2. Current step Bead and root change contract.
699
+ 3. Preloaded completed dependency handoffs.
700
+ 4. `sp result` for specific upstream jobs.
701
+ 5. Current diff, repository state and evidence artifacts.
702
+ 6. `sp feed --json` for verifying claimed tool use, exact chronology or contradictory evidence.
703
+ 7. Semantic Beads edges for related, discovered-from or validates context.
704
+ 8. Historical memories only if prior decisions/incidents are materially relevant.
705
+
706
+ ## 6.5 Handoff schema extension
707
+
708
+ The common output schema should support optional chain collaboration fields:
709
+
710
+ ```json
711
+ {
712
+ "inputs_consumed": [
713
+ {"role": "seconder", "bead_id": "...", "job_id": "..."}
714
+ ],
715
+ "decisions": [
716
+ {
717
+ "decision": "...",
718
+ "rationale": "...",
719
+ "evidence": ["..."]
720
+ }
721
+ ],
722
+ "assumptions": [],
723
+ "unresolved_questions": [],
724
+ "downstream_attention": []
725
+ }
726
+ ```
727
+
728
+ These fields capture decision records, not hidden chain-of-thought.
729
+
730
+ ## 6.6 Role-specific chain behavior
731
+
732
+ ### Executor
733
+
734
+ Consumes advisor findings and root contract. It should cite material advisor decisions it accepted or rejected.
735
+
736
+ ### Seconder
737
+
738
+ Consumes writer diff and root contract. It should remain bounded and avoid loading the full chain unless a required input is absent.
739
+
740
+ ### Test-engineer
741
+
742
+ Consumes writer diff, root validation contract, seconder verdict and prior test-runner output on retries.
743
+
744
+ ### Reviewer
745
+
746
+ Consumes final evidence from all required gates. It does not independently recreate every gate unless evidence is missing or contradictory.
747
+
748
+ ### Researcher and overthinker
749
+
750
+ When used as advisors, they must identify the downstream executor as consumer and produce action-oriented evidence rather than standalone essays.
751
+
752
+ ## 6.7 Chain-context evaluation
753
+
754
+ The `chain-participation-v1` suite shall measure:
755
+
756
+ ```text
757
+ chain identity recognition
758
+ root and step contract resolution
759
+ upstream handoff discovery
760
+ selective reconstruction
761
+ semantic edge navigation
762
+ contradiction detection
763
+ non-duplication of upstream work
764
+ downstream handoff quality
765
+ ```
766
+
767
+ Adversarial cases shall include missing handoffs, conflicting gates, stale memory, incomplete preloaded depth, unrelated semantic edges and cross-chain contamination.
768
+
769
+ # 7. Workstream C — Pull-based memory
770
+
771
+ ## 7.1 Current behavior
772
+
773
+ The current runtime extracts keywords from Bead title and description, queries the memory cache and injects a bounded set of ranked memories. Ranking uses lexical BM25, recency and access frequency. The injection budget is bounded, but the model pays the cost even when historical context is unnecessary.
774
+
775
+ ## 7.2 Target behavior
776
+
777
+ Memory becomes an explicit capability. The runtime injects a short policy, not memory content:
778
+
779
+ ```text
780
+ Historical project memories are available through bd memories and bd recall.
781
+ Query them only when the task depends on prior decisions, conventions, incidents,
782
+ accepted exceptions or unresolved ambiguity. Treat memories as context, not
783
+ current authority; verify against the root contract, current code and documentation.
784
+ ```
785
+
786
+ ## 7.3 Retrieval triggers
787
+
788
+ A specialist should consider memory when:
789
+
790
+ * the Bead references an existing convention or prior implementation;
791
+ * multiple conflicting repository patterns exist;
792
+ * a known incident or regression history may change validation;
793
+ * an architecture or API choice depends on a previous decision;
794
+ * a reviewer needs to understand an accepted exception;
795
+ * an overthinker is evaluating a high-impact repeated failure mode.
796
+
797
+ A specialist should normally avoid memory when:
798
+
799
+ * the task is fully specified and local;
800
+ * the information is directly available in current code or documentation;
801
+ * the role is a cheap bounded gate and history is not required;
802
+ * the claim concerns current external APIs, versions or standards that require external sources.
803
+
804
+ ## 7.4 Retrieval interface
805
+
806
+ Preferred capabilities:
807
+
808
+ ```bash
809
+ bd memories search --query "..." --limit 5 --json
810
+ bd memories relevant --bead <id> --limit 5 --max-tokens 500 --json
811
+ bd recall <memory-key> --json
812
+ ```
813
+
814
+ Memory results should expose provenance:
815
+
816
+ ```text
817
+ memory key
818
+ summary/value
819
+ created and updated timestamps
820
+ source Bead/job/commit
821
+ confidence
822
+ status: current | superseded | unknown
823
+ ```
824
+
825
+ If the current CLI cannot provide this shape, the project should add a Specialists-side wrapper over the existing FTS/ranking cache.
826
+
827
+ ## 7.5 Memory telemetry
828
+
829
+ Record:
830
+
831
+ ```text
832
+ memory query count
833
+ query kind and normalized keyword count
834
+ result count
835
+ tokens/bytes returned
836
+ recall count
837
+ memory identifiers consumed
838
+ whether memory was cited in handoff
839
+ query latency
840
+ ```
841
+
842
+ Do not put raw query text or memory content in Prometheus labels.
843
+
844
+ ## 7.6 Memory evaluation
845
+
846
+ Required dimensions:
847
+
848
+ * necessary-memory recall rate;
849
+ * unnecessary-query rate;
850
+ * task success with and without memory;
851
+ * stale-memory contradiction handling;
852
+ * memory token cost;
853
+ * retrieval latency;
854
+ * provenance use;
855
+ * model and role differences in deciding when to query.
856
+
857
+ A/B rollout should compare eager injection, pull guidance and no-memory control on curated cases.
858
+
859
+ # 8. Workstream D — Telemetry and forensic hardening
860
+
861
+ ## 8.1 Requirement
862
+
863
+ No model, prompt or chain promotion decision may depend on a metric with unresolved counting semantics. A telemetry integrity suite must be introduced and pass before the eval platform is used for promotion.
864
+
865
+ ## 8.2 Thinking events
866
+
867
+ ### Current behavior
868
+
869
+ Pi exposes `thinking_start`, `thinking_delta` and `thinking_end`. Specialists currently emits generic `thinking` events for start and deltas, with only a character count. `thinking_end` is not persisted. The supervisor accumulates characters, so persisted values are cumulative snapshots.
870
+
871
+ ### Required event model
872
+
873
+ ```json
874
+ {
875
+ "type": "thinking",
876
+ "phase": "start|delta|end",
877
+ "segment_id": "...",
878
+ "turn_index": 2,
879
+ "delta_char_count": 128,
880
+ "total_char_count": 1840,
881
+ "duration_ms": 7320
882
+ }
883
+ ```
884
+
885
+ The end event should carry final segment duration and total characters. Raw thinking content should not be stored by default.
886
+
887
+ ### Derived measures
888
+
889
+ ```text
890
+ thinking_stream_duration_ms
891
+ thinking_segments
892
+ thinking_chars
893
+ reasoning_tokens
894
+ thinking_to_active_runtime_ratio
895
+ time_to_first_thinking
896
+ time_to_first_tool
897
+ time_to_first_text
898
+ inter_tool_deliberation_ms
899
+ ```
900
+
901
+ The metric must be named as a stream duration, not provider compute time.
902
+
903
+ ## 8.3 Tool-call accounting
904
+
905
+ ### Current risk
906
+
907
+ Timeline aggregation increments tool counts for every `tool` event, which includes start, update and end. Pi may also call tool callbacks during both LLM tool-call construction and execution.
908
+
909
+ ### Required correction
910
+
911
+ * Count a tool call only on canonical `tool phase=start` execution events.
912
+ * Deduplicate by `tool_call_id`.
913
+ * Track uncorrelated calls separately.
914
+ * Distinguish construction events from execution events if retained.
915
+ * Compute duration by matching start/end on `tool_call_id`.
916
+ * Support parallel active tool calls without a single global `toolStartMs`.
917
+
918
+ ### Metrics
919
+
920
+ ```text
921
+ xtrm_tool_calls_total
922
+ xtrm_tool_errors_total
923
+ xtrm_tool_call_duration_seconds
924
+ xtrm_uncorrelated_tool_events_total
925
+ xtrm_parallel_tool_concurrency
926
+ ```
927
+
928
+ ## 8.4 Turn and message timing
929
+
930
+ Add stable start/end pairing for:
931
+
932
+ ```text
933
+ turn duration
934
+ assistant message duration
935
+ tool-result message duration
936
+ time to first output token
937
+ time from last tool end to final response
938
+ ```
939
+
940
+ ## 8.5 Activity and stall semantics
941
+
942
+ Correct queries and state logic to use the canonical `thinking` event name rather than `think`. Stalls should distinguish:
943
+
944
+ * no protocol activity;
945
+ * active thinking stream;
946
+ * long-running tool;
947
+ * provider/network silence;
948
+ * waiting state.
949
+
950
+ ## 8.6 Forensic-to-Prometheus wiring
951
+
952
+ The default metrics collector must read forensic events within the requested time range and pass them to the projection renderer. Required families include:
953
+
954
+ ```text
955
+ xtrm_gate_verdicts_total
956
+ xtrm_evidence_refs_total
957
+ xtrm_mcp_operations_total
958
+ xtrm_identity_operations_total
959
+ xtrm_policy_decisions_total
960
+ xtrm_policy_mismatches_total
961
+ xtrm_eval_runs_total
962
+ xtrm_eval_score
963
+ ```
964
+
965
+ The collector must use bounded event-family queries and not load unbounded event history.
966
+
967
+ ## 8.7 New operational metrics
968
+
969
+ Recommended additions:
970
+
971
+ ```text
972
+ xtrm_thinking_duration_seconds
973
+ xtrm_thinking_segments_total
974
+ xtrm_turn_duration_seconds
975
+ xtrm_message_duration_seconds
976
+ xtrm_time_to_first_action_seconds
977
+ xtrm_schema_validation_total
978
+ xtrm_handoff_validation_total
979
+ xtrm_chain_fix_loops_total
980
+ xtrm_chain_time_to_pass_seconds
981
+ xtrm_chain_tokens_to_pass_total
982
+ xtrm_memory_queries_total
983
+ xtrm_memory_results_total
984
+ xtrm_grader_duration_seconds
985
+ xtrm_eval_assertions_total
986
+ xtrm_eval_regressions_total
987
+ xtrm_eval_judge_disagreements_total
988
+ ```
989
+
990
+ ## 8.8 Historical completeness
991
+
992
+ Every derived historical metric shall record availability:
993
+
994
+ ```text
995
+ complete
996
+ partial
997
+ unavailable
998
+ estimated
999
+ ```
1000
+
1001
+ Missing historical fields are unknown, not zero.
1002
+
1003
+ ## 8.9 Telemetry integrity eval suite
1004
+
1005
+ `telemetry-integrity-v1` shall verify:
1006
+
1007
+ * exactly one canonical run-complete per run turn;
1008
+ * monotonic sequence;
1009
+ * tool starts equal unique calls;
1010
+ * paired tool durations where IDs exist;
1011
+ * no negative durations;
1012
+ * waiting plus active time is consistent with elapsed time within tolerance;
1013
+ * token trajectory is monotonic/semantically consistent;
1014
+ * context ratio remains bounded;
1015
+ * thinking spans are paired;
1016
+ * forensic conversion preserves identity and timestamps;
1017
+ * Prometheus projection rejects forbidden labels;
1018
+ * forensic-derived metrics appear in live scrape when events exist.
1019
+
1020
+ # 9. Workstream E — Evaluation platform
1021
+
1022
+ ## 9.1 Evaluation subject
1023
+
1024
+ A candidate is not merely a model. It is the resolved combination:
1025
+
1026
+ ```text
1027
+ model
1028
+ x specialist version
1029
+ x system prompt
1030
+ x mandatory rules
1031
+ x skills
1032
+ x runner injections
1033
+ x tool catalog
1034
+ x permission tier
1035
+ x thinking level
1036
+ x chain position
1037
+ x task/environment
1038
+ ```
1039
+
1040
+ Every candidate must have a deterministic fingerprint.
1041
+
1042
+ ## 9.2 Evaluation levels
1043
+
1044
+ ### Outcome quality
1045
+
1046
+ Did the final environment satisfy the task? Examples: hidden tests, state assertions, schema validation, smoke/E2E, telemetry assertions, diff correctness and absence of regressions.
1047
+
1048
+ ### Role behavior
1049
+
1050
+ Did the specialist obey its mandate and boundaries? Examples: no reviewer edits, no unauthorized test-engineer source changes, bounded seconder behavior, researcher source discipline.
1051
+
1052
+ ### Chain contribution
1053
+
1054
+ Did the specialist consume the right upstream evidence, avoid duplication, detect contradictions and produce a useful downstream handoff?
1055
+
1056
+ ### Efficiency and reliability
1057
+
1058
+ What were the token, time, tool, retry, compaction, stall and variance characteristics?
1059
+
1060
+ Efficiency is reported separately and cannot compensate for failed hard gates.
1061
+
1062
+ ## 9.3 Evaluation entities
1063
+
1064
+ ### Eval suite
1065
+
1066
+ A versioned collection of cases for a capability or regression domain.
1067
+
1068
+ ### Eval case
1069
+
1070
+ A pinned task, environment, expected outcomes and grader configuration.
1071
+
1072
+ ### Eval experiment
1073
+
1074
+ A controlled comparison of one or more candidate configurations.
1075
+
1076
+ ### Eval trial
1077
+
1078
+ One candidate executing one case for one attempt.
1079
+
1080
+ ### Eval artifact
1081
+
1082
+ A reference to output, trace, diff, commit, tests, logs, environment state or external source bundle.
1083
+
1084
+ ### Eval score
1085
+
1086
+ One grader’s result for one trial and dimension.
1087
+
1088
+ ### Pairwise comparison
1089
+
1090
+ A blinded judge preference between two trial outputs/evidence bundles.
1091
+
1092
+ ### Human annotation
1093
+
1094
+ A gold label, adjudication or override with provenance.
1095
+
1096
+ ## 9.4 Proposed SQLite schema
1097
+
1098
+ ```sql
1099
+ CREATE TABLE eval_suites (...);
1100
+ CREATE TABLE eval_cases (...);
1101
+ CREATE TABLE eval_experiments (...);
1102
+ CREATE TABLE eval_trials (...);
1103
+ CREATE TABLE eval_scores (...);
1104
+ CREATE TABLE eval_pairwise_results (...);
1105
+ CREATE TABLE eval_human_annotations (...);
1106
+ ```
1107
+
1108
+ Minimum fields are defined below.
1109
+
1110
+ ### `eval_suites`
1111
+
1112
+ ```text
1113
+ suite_id
1114
+ name
1115
+ participant_role
1116
+ suite_kind: regression | capability | telemetry | chain | external
1117
+ version
1118
+ dataset_hash
1119
+ rubric_hash
1120
+ created_at_ms
1121
+ ```
1122
+
1123
+ ### `eval_cases`
1124
+
1125
+ ```text
1126
+ case_id
1127
+ suite_id
1128
+ case_version
1129
+ title
1130
+ fixture_ref
1131
+ task_contract_json
1132
+ environment_spec_json
1133
+ expected_outcome_json
1134
+ grader_config_json
1135
+ difficulty
1136
+ tags_json
1137
+ source_kind
1138
+ source_ref
1139
+ is_held_out
1140
+ ```
1141
+
1142
+ ### `eval_experiments`
1143
+
1144
+ ```text
1145
+ experiment_id
1146
+ suite_id
1147
+ experiment_kind
1148
+ candidate_configs_json
1149
+ randomization_seed
1150
+ trials_per_case
1151
+ status
1152
+ started_at_ms
1153
+ completed_at_ms
1154
+ ```
1155
+
1156
+ ### `eval_trials`
1157
+
1158
+ ```text
1159
+ trial_id
1160
+ experiment_id
1161
+ case_id
1162
+ candidate_id
1163
+ job_id
1164
+ chain_id
1165
+ model
1166
+ prompt_hash
1167
+ rules_hash
1168
+ skills_hash
1169
+ runtime_version
1170
+ tool_catalog_version
1171
+ environment_hash
1172
+ attempt_index
1173
+ status
1174
+ data_completeness
1175
+ started_at_ms
1176
+ completed_at_ms
1177
+ ```
1178
+
1179
+ ### `eval_scores`
1180
+
1181
+ ```text
1182
+ score_id
1183
+ trial_id
1184
+ grader_id
1185
+ grader_type
1186
+ dimension
1187
+ score
1188
+ passed
1189
+ confidence
1190
+ evidence_json
1191
+ grader_version
1192
+ created_at_ms
1193
+ ```
1194
+
1195
+ ### `eval_pairwise_results`
1196
+
1197
+ ```text
1198
+ comparison_id
1199
+ experiment_id
1200
+ case_id
1201
+ trial_a_id
1202
+ trial_b_id
1203
+ judge_id
1204
+ presentation_order
1205
+ winner: A | B | tie | insufficient_evidence
1206
+ confidence
1207
+ rationale_json
1208
+ ```
1209
+
1210
+ ## 9.5 Forensic eval event catalog
1211
+
1212
+ Add events:
1213
+
1214
+ ```text
1215
+ eval.experiment.started
1216
+ eval.experiment.completed
1217
+ eval.experiment.failed
1218
+ eval.trial.started
1219
+ eval.trial.completed
1220
+ eval.trial.failed
1221
+ eval.grader.started
1222
+ eval.grader.completed
1223
+ eval.grader.failed
1224
+ eval.score.recorded
1225
+ eval.assertion.passed
1226
+ eval.assertion.failed
1227
+ eval.comparison.recorded
1228
+ eval.judge.disagreement
1229
+ eval.human.annotation.recorded
1230
+ eval.human.override.recorded
1231
+ eval.regression.detected
1232
+ eval.promotion.recommended
1233
+ eval.promotion.blocked
1234
+ ```
1235
+
1236
+ Use `eval_id` and existing job/chain/commit correlation fields.
1237
+
1238
+ ## 9.6 Grader hierarchy
1239
+
1240
+ Use the cheapest reliable grader first:
1241
+
1242
+ ```text
1243
+ environment/state grader
1244
+ -> deterministic trace grader
1245
+ -> deterministic artifact grader
1246
+ -> model grader
1247
+ -> human adjudication
1248
+ ```
1249
+
1250
+ ### Environment/state graders
1251
+
1252
+ Examples:
1253
+
1254
+ * hidden tests;
1255
+ * expected database/file state;
1256
+ * smoke/E2E result;
1257
+ * emitted telemetry;
1258
+ * no leaked secret;
1259
+ * worktree and commit state.
1260
+
1261
+ ### Trace graders
1262
+
1263
+ Examples:
1264
+
1265
+ * required tool called;
1266
+ * forbidden tool absent;
1267
+ * no edit/write by read-only role;
1268
+ * `sp result` used before `sp feed` where required;
1269
+ * memory recall performed only when triggered;
1270
+ * chain context consumed;
1271
+ * output schema valid;
1272
+ * tool-call budget respected.
1273
+
1274
+ ### Artifact graders
1275
+
1276
+ Examples:
1277
+
1278
+ * changed paths match scope;
1279
+ * required tests added;
1280
+ * diff minimality heuristics;
1281
+ * evidence references resolve;
1282
+ * handoff fields complete.
1283
+
1284
+ ### Model graders
1285
+
1286
+ Use for nuanced qualities:
1287
+
1288
+ * reviewer finding materiality and evidence quality;
1289
+ * researcher citation support and synthesis;
1290
+ * overthinker risk quality and depth calibration;
1291
+ * executor unnecessary complexity;
1292
+ * handoff actionability.
1293
+
1294
+ Model graders must be calibrated against human labels and versioned.
1295
+
1296
+ ## 9.7 Hard gates and score vector
1297
+
1298
+ A trial first passes hard gates:
1299
+
1300
+ ```text
1301
+ outcome correct
1302
+ no forbidden action
1303
+ schema valid
1304
+ no critical requirement missed
1305
+ no critical security/scope violation
1306
+ ```
1307
+
1308
+ Then a quality vector is reported:
1309
+
1310
+ ```text
1311
+ contract compliance
1312
+ role adherence
1313
+ evidence quality
1314
+ chain-context use
1315
+ handoff quality
1316
+ operational reliability
1317
+ ```
1318
+
1319
+ Cost is reported separately:
1320
+
1321
+ ```text
1322
+ input/output/reasoning/tool tokens
1323
+ active/waiting/elapsed time
1324
+ tool calls
1325
+ provider cost when authoritative
1326
+ ```
1327
+
1328
+ A composite quality score may be displayed but must not replace dimension-level promotion rules.
1329
+
1330
+ ## 9.8 Role-specific scorecards
1331
+
1332
+ ### Reviewer
1333
+
1334
+ ```text
1335
+ verdict accuracy
1336
+ blocking finding precision
1337
+ blocking finding recall
1338
+ false PASS rate
1339
+ false block rate
1340
+ upstream evidence coverage
1341
+ contradiction handling
1342
+ delta-review discipline
1343
+ evidence traceability
1344
+ next-action usefulness
1345
+ ```
1346
+
1347
+ False PASS is the primary safety metric.
1348
+
1349
+ ### Executor
1350
+
1351
+ ```text
1352
+ acceptance pass rate
1353
+ scope precision and recall
1354
+ regression rate
1355
+ minimality
1356
+ repository convention adherence
1357
+ validation execution
1358
+ handoff completeness
1359
+ fix-loop count
1360
+ ```
1361
+
1362
+ ### Seconder
1363
+
1364
+ ```text
1365
+ gate accuracy
1366
+ finding precision and recall
1367
+ boundedness
1368
+ context expansion discipline
1369
+ UNCLEAR calibration
1370
+ agreement with final outcome
1371
+ ```
1372
+
1373
+ ### Test-engineer
1374
+
1375
+ ```text
1376
+ fail-to-pass effectiveness
1377
+ pass-to-pass preservation
1378
+ critical-path coverage
1379
+ mutation sensitivity
1380
+ command executability
1381
+ source-boundary violations
1382
+ failure-owner accuracy
1383
+ telemetry assertion quality
1384
+ harness reuse
1385
+ ```
1386
+
1387
+ ### Researcher
1388
+
1389
+ ```text
1390
+ claim correctness
1391
+ citation precision and coverage
1392
+ source authority
1393
+ freshness
1394
+ fact/inference separation
1395
+ contradiction handling
1396
+ query efficiency
1397
+ ```
1398
+
1399
+ ### Overthinker
1400
+
1401
+ ```text
1402
+ decision quality
1403
+ assumption discovery
1404
+ risk precision and recall
1405
+ alternative quality
1406
+ disconfirming evidence
1407
+ actionability
1408
+ depth calibration
1409
+ redundancy
1410
+ ```
1411
+
1412
+ ## 9.9 Regression and capability suites
1413
+
1414
+ Regression suites protect reliable behavior and should approach 100% pass rate. Capability suites are intentionally difficult and are used to improve performance.
1415
+
1416
+ Initial suites:
1417
+
1418
+ ```text
1419
+ reviewer-regression-v1
1420
+ executor-regression-v1
1421
+ seconder-regression-v1
1422
+ test-engineer-regression-v1
1423
+ researcher-regression-v1
1424
+ overthinker-regression-v1
1425
+ chain-participation-v1
1426
+ telemetry-integrity-v1
1427
+ ```
1428
+
1429
+ ## 9.10 Dataset sources
1430
+
1431
+ Use real system failures first:
1432
+
1433
+ * reviewer PARTIAL followed by successful fix;
1434
+ * false PASS discovered later;
1435
+ * scope leak;
1436
+ * wrong diff or wrong worktree;
1437
+ * invalid handoff schema;
1438
+ * unnecessary or missing memory retrieval;
1439
+ * skipped mandatory gate;
1440
+ * contradictory gate outputs;
1441
+ * context overflow/compaction;
1442
+ * tool or extension failure;
1443
+ * testing ownership misclassification.
1444
+
1445
+ Add generated mutation cases:
1446
+
1447
+ * remove an upstream handoff;
1448
+ * make result contradict diff;
1449
+ * inject stale memory;
1450
+ * alter root scope;
1451
+ * add irrelevant relationship edges;
1452
+ * remove chain identity;
1453
+ * falsify a tool-use claim;
1454
+ * insert seeded functional or quality defects.
1455
+
1456
+ ## 9.11 Retrospective evaluation
1457
+
1458
+ ### Supported
1459
+
1460
+ Existing runs can be evaluated for dimensions whose evidence remains in the database:
1461
+
1462
+ * runtime and token efficiency;
1463
+ * tool use;
1464
+ * output schema;
1465
+ * role-boundary compliance;
1466
+ * handoff quality;
1467
+ * chain sequencing;
1468
+ * gate verdict distributions;
1469
+ * evidence references;
1470
+ * context and compaction behavior;
1471
+ * some reviewer and executor benchmark rows.
1472
+
1473
+ ### Partial
1474
+
1475
+ Older runs may lack exact thinking spans, resolved prompt hashes, skill body hashes, complete forensic conversion or correlated tool IDs.
1476
+
1477
+ ### Requires replay
1478
+
1479
+ Causal model comparisons and hidden outcome testing require a pinned repository snapshot and rerun. Historical cohort analysis must not be presented as a controlled A/B result.
1480
+
1481
+ ### Backfill command
1482
+
1483
+ Recommended:
1484
+
1485
+ ```bash
1486
+ sp eval backfill --repo .
1487
+ sp eval backfill --all-repos
1488
+ ```
1489
+
1490
+ The command should calculate compatible deterministic graders, emit eval events and mark data completeness.
1491
+
1492
+ ## 9.12 Continuous evaluation
1493
+
1494
+ ### After every run
1495
+
1496
+ Run deterministic graders asynchronously after run-complete persistence:
1497
+
1498
+ ```text
1499
+ schema validation
1500
+ role boundary
1501
+ forbidden path/tool
1502
+ handoff completeness
1503
+ startup context completeness
1504
+ telemetry integrity
1505
+ required evidence refs
1506
+ ```
1507
+
1508
+ ### After every chain
1509
+
1510
+ Run:
1511
+
1512
+ ```text
1513
+ canonical gate sequence
1514
+ mandatory gate completion
1515
+ final verdict consistency
1516
+ unresolved failure state
1517
+ fix loops
1518
+ time and tokens to PASS
1519
+ chain outcome
1520
+ ```
1521
+
1522
+ ### Sampled/high-risk model graders
1523
+
1524
+ Run model-based quality graders on:
1525
+
1526
+ * all high/critical chains;
1527
+ * a configured percentage of medium chains;
1528
+ * detected anomalies/regressions;
1529
+ * explicit experiments.
1530
+
1531
+ Begin in shadow mode and do not block the main run.
1532
+
1533
+ ## 9.13 Controlled A/B experiments
1534
+
1535
+ Use paired design:
1536
+
1537
+ ```text
1538
+ same case
1539
+ same base SHA
1540
+ same task contract
1541
+ same upstream artifacts
1542
+ same tool catalog
1543
+ same budgets
1544
+ isolated worktrees
1545
+ ```
1546
+
1547
+ Use multiple trials for stochastic agents.
1548
+
1549
+ Prompt and model changes should use factorial design where possible:
1550
+
1551
+ | | Current prompt | New prompt |
1552
+ | ------- | -------------: | ---------: |
1553
+ | Model A | A-current | A-new |
1554
+ | Model B | B-current | B-new |
1555
+
1556
+ This isolates model effect, prompt effect and interaction.
1557
+
1558
+ ## 9.14 Pairwise judges
1559
+
1560
+ For subjective dimensions:
1561
+
1562
+ * hide model/provider/cost identity;
1563
+ * randomize A/B order;
1564
+ * repeat with swapped order;
1565
+ * allow tie and insufficient evidence;
1566
+ * use multiple judges for high-value cases;
1567
+ * route disagreement to human adjudication;
1568
+ * monitor position and verbosity bias.
1569
+
1570
+ ## 9.15 Statistical reporting
1571
+
1572
+ Report:
1573
+
1574
+ ```text
1575
+ baseline and candidate means
1576
+ paired delta
1577
+ bootstrap confidence interval
1578
+ win/tie/loss
1579
+ variance across trials
1580
+ failure category delta
1581
+ ```
1582
+
1583
+ Use paired binary methods for pass/fail outcomes and paired bootstrap for continuous scores. Promotion requires practical as well as statistical significance.
1584
+
1585
+ ## 9.16 Promotion policy
1586
+
1587
+ Example:
1588
+
1589
+ ```yaml
1590
+ hard_requirements:
1591
+ critical_false_pass_increase: 0
1592
+ new_role_boundary_violations: 0
1593
+ regression_suite_lower_bound: 0.97
1594
+ output_schema_validity: 0.995
1595
+ quality:
1596
+ paired_quality_delta_lower_ci: -0.01
1597
+ chain_contribution_delta_lower_ci: 0
1598
+ pairwise_win_rate: 0.55
1599
+ efficiency:
1600
+ max_active_runtime_p50_regression: 0.20
1601
+ ```
1602
+
1603
+ A cost increase requires a material quality improvement. A quality tie with meaningful cost reduction may justify promotion.
1604
+
1605
+ ## 9.17 External and internal software-engineering benchmarks
1606
+
1607
+ External suites such as SWE-bench Pro public and Multi-SWE-bench can measure general model+harness coding capability. They should not replace internal role benchmarks.
1608
+
1609
+ The internal executor benchmark should use SWE-bench methodology:
1610
+
1611
+ ```text
1612
+ real Bead/issue
1613
+ pinned repository and base SHA
1614
+ isolated environment
1615
+ hidden fail-to-pass tests
1616
+ pass-to-pass regression tests
1617
+ deterministic artifact checks
1618
+ full forensic trajectory
1619
+ ```
1620
+
1621
+ Reviewer and seconder require seeded-patch benchmarks with gold defects and verdicts. Test-engineer requires buggy and fixed states to verify test sensitivity. Researcher requires source-backed claim sets. Chain eval requires mutated handoffs, missing gates and contradictions.
1622
+
1623
+ # 10. Console product requirements
1624
+
1625
+ ## 10.1 Decision
1626
+
1627
+ Use the existing Console/Omniforge application. Do not create a separate eval application in the first implementation phase.
1628
+
1629
+ ## 10.2 Data ownership
1630
+
1631
+ * Per-repo Specialists databases remain run-level sources of truth.
1632
+ * Specialists owns eval definitions, trial execution, grading events and detailed evidence.
1633
+ * The Console materializer projects summaries and forensic references into its read model.
1634
+ * Prometheus provides low-cardinality time-series aggregates.
1635
+ * Console provides operator UX and does not mutate source telemetry during normal reads.
1636
+
1637
+ ## 10.3 Eval routes and surfaces
1638
+
1639
+ Add `/console/evals` with:
1640
+
1641
+ ### Eval overview
1642
+
1643
+ Suite health, recent regressions, score by role/model, data freshness and missing signals.
1644
+
1645
+ ### Experiment detail
1646
+
1647
+ Candidate fingerprints, case coverage, win/tie/loss, paired deltas, confidence intervals and promotion status.
1648
+
1649
+ ### Model/prompt matrix
1650
+
1651
+ Role × model × prompt/rule/runtime version.
1652
+
1653
+ ### Case explorer
1654
+
1655
+ Case source, difficulty, tags, expected outcome, fixture and grader definitions.
1656
+
1657
+ ### Trial detail
1658
+
1659
+ Timeline of thinking, tools, turns and output; artifacts; diff; evidence; grader results; data completeness.
1660
+
1661
+ ### Chain evaluation
1662
+
1663
+ Step timeline, queue/runtime, gates, contradictions, fix loops, time/tokens to PASS and final outcome.
1664
+
1665
+ ### Historical mining
1666
+
1667
+ Search existing runs, inspect completeness and promote representative failures into eval cases.
1668
+
1669
+ ### Live evaluation
1670
+
1671
+ Display post-run deterministic graders and experiment progress without blocking the originating job.
1672
+
1673
+ ### Regression drift
1674
+
1675
+ Trend score, false-PASS rate, schema validity, tool/error metrics and cost over runtime/prompt/model versions.
1676
+
1677
+ ## 10.4 Drilldown
1678
+
1679
+ Aggregate panels must drill down through evidence references to:
1680
+
1681
+ ```text
1682
+ specialist job
1683
+ forensic event
1684
+ Bead/root/step
1685
+ chain
1686
+ commit/diff/PR
1687
+ test artifact
1688
+ eval trial and grader
1689
+ ```
1690
+
1691
+ ## 10.5 Realtime
1692
+
1693
+ The materializer and WebSocket hint path should publish after committed eval writes. Live views show status transitions, not raw unbounded forensic streams.
1694
+
1695
+ ## 10.6 Prometheus versus forensic UI
1696
+
1697
+ Prometheus charts answer “is a class of runs getting worse?” Forensic/trial detail answers “what happened in this exact run?”. High-cardinality IDs stay in correlation and evidence refs.
1698
+
1699
+ # 11. CLI and API requirements
1700
+
1701
+ ## 11.1 CLI
1702
+
1703
+ Recommended commands:
1704
+
1705
+ ```bash
1706
+ sp eval suite list
1707
+ sp eval suite show <suite>
1708
+ sp eval case show <case>
1709
+ sp eval run <suite> --candidate <config> [--candidate <config>] --trials N
1710
+ sp eval compare <experiment-id>
1711
+ sp eval inspect <trial-id>
1712
+ sp eval failures <experiment-id>
1713
+ sp eval disagreements <experiment-id>
1714
+ sp eval backfill --repo .
1715
+ sp eval backfill --all-repos
1716
+ sp eval calibrate <rubric> --human-labels <file>
1717
+ sp eval promote <experiment-id> --policy <file>
1718
+ ```
1719
+
1720
+ ## 11.2 Existing DB commands
1721
+
1722
+ Retain and integrate:
1723
+
1724
+ ```text
1725
+ sp db extract
1726
+ sp db stats
1727
+ sp db benchmark-export
1728
+ sp metrics --prometheus
1729
+ ```
1730
+
1731
+ `benchmark-export` should either be versioned as a legacy exporter or rebuilt as an adapter over eval tables.
1732
+
1733
+ ## 11.3 Evaluator interface
1734
+
1735
+ Conceptual TypeScript contract:
1736
+
1737
+ ```ts
1738
+ interface EvalGrader {
1739
+ id: string;
1740
+ version: string;
1741
+ type: "environment" | "trace" | "artifact" | "model" | "human";
1742
+ supportedRoles?: string[];
1743
+ requiredEvidence: string[];
1744
+ grade(input: EvalTrialBundle): Promise<EvalScoreResult>;
1745
+ }
1746
+ ```
1747
+
1748
+ ## 11.4 Post-run evaluator
1749
+
1750
+ The supervisor or an event consumer emits run-complete, then enqueues evaluation. It must not recursively grade evaluator jobs. It must tolerate unavailable graders and record partial results.
1751
+
1752
+ ## 11.5 Cross-repo experiments
1753
+
1754
+ Use globally unique experiment/eval IDs. Each repository retains local trial data; Console aggregates summaries by eval ID. A future state daemon may become the registry, but the initial bridge must not block on it.
1755
+
1756
+ # 12. Security, privacy and data governance
1757
+
1758
+ ## 12.1 Prompt and output storage
1759
+
1760
+ Raw prompts and model outputs may contain repository-sensitive data. Store hashes and bounded metadata by default. Detailed raw artifacts remain in existing protected job/evidence storage and are not exported to Prometheus.
1761
+
1762
+ ## 12.2 Forensic redaction
1763
+
1764
+ Continue using `xtrm.forensic.v1` redaction rules. Add grader output and model-judge prompts to the sensitive-field policy. Judge evidence bundles must be minimized and redacted.
1765
+
1766
+ ## 12.3 Prometheus labels
1767
+
1768
+ Never label by:
1769
+
1770
+ ```text
1771
+ job_id
1772
+ bead_id
1773
+ chain_id
1774
+ participant_id
1775
+ trace/span/tool-call/session/eval IDs
1776
+ raw path/command/error/diff/URL
1777
+ prompt or output content
1778
+ credentials or user identifiers
1779
+ ```
1780
+
1781
+ ## 12.4 Held-out cases
1782
+
1783
+ Held-out benchmark fixtures and hidden tests must not be exposed to the candidate agent. The runner must separate candidate worktree/tool access from grader assets.
1784
+
1785
+ ## 12.5 Human annotation
1786
+
1787
+ Record annotator identity internally, but expose only the minimum needed in shared dashboards. Gold labels require provenance and versioning.
1788
+
1789
+ # 13. Integration with the Specialists roadmap, xtrm Stage 0 and Channels
1790
+
1791
+ ## 13.1 Required direct documentation changes
1792
+
1793
+ Direct documentation changes are necessary, but they must preserve the distinct role of each source.
1794
+
1795
+ ### `specialists-roadmap.md`
1796
+
1797
+ The canonical roadmap must be updated directly because it is the bridge-runtime source of truth and currently does not represent this modernization as an integrated, evaluation-gated program. The update should be concise and architectural rather than copying this PRD in full.
1798
+
1799
+ Required roadmap changes:
1800
+
1801
+ 1. Add a prominent companion-program reference to this PRD and state that detailed work-package decomposition, graders and model/prompt promotion criteria live here.
1802
+
1803
+ 2. Add **Track C — Prompt, Chain Context, Telemetry and Evaluation Modernization** to the roadmap sequencing model.
1804
+
1805
+ 3. Record the ordering invariant:
1806
+
1807
+ ```text
1808
+ telemetry integrity and historical baseline
1809
+ -> critical chain-first roadmap spine
1810
+ -> prompt/rule and memory changes under paired evaluation
1811
+ -> Channels replacement of bridge communication
1812
+ ```
1813
+
1814
+ 4. Mark the following as prerequisites for the complete chain-context implementation in this PRD:
1815
+
1816
+ * Opportunity 2: READ_ONLY path binding;
1817
+ * Opportunity 3: persisted resolved chain shape;
1818
+ * Opportunity 4: composition gate;
1819
+ * Opportunity 5: authoritative step identity;
1820
+ * Opportunity 10: chain-driven dispatch.
1821
+
1822
+ 5. Retain Opportunity 8 only as a deliberately minimal bridge. Its payload must be compatible with future `work.turn`, `work.finding` and `work.verdict` messages, and it must not grow into a second scheduler.
1823
+
1824
+ 6. Add evaluation gates to roadmap changes that alter prompts, rules, memory injection, chain context or operator workflow. “Implemented” is not sufficient; the change must pass its regression and capability suites.
1825
+
1826
+ 7. Update status annotations against current code. Shipped QA pipeline, seconder fusion and forensic identity work must not be listed as unstarted. Conversely, persisted chain shape, composition gate, `--chain`, memory pull, XML contracts and Skills v4 must not be represented as complete unless verified.
1827
+
1828
+ 8. Reconcile Opportunities 1, 4 and 8 against existing `bd merge-slot`, `bd gate` and `bd swarm` before creating implementation Beads. The result must be a reuse decision record, not an assumption.
1829
+
1830
+ 9. Add bridge retirement criteria: every bridge must name the future owner, replacement trigger, migration/export path and deletion condition.
1831
+
1832
+ 10. Preserve the roadmap as architecture-level SSOT. Do not paste the scorecards, SQL schemas or complete work-package tables from this PRD into it.
1833
+
1834
+ ### `xtrm/docs/_meta/2026-06-21-design-reconciliation.md`
1835
+
1836
+ The 21 June document is explicitly a dated snapshot and should not be rewritten as though it were a live status page. Several claims are now historical, including Console state and portions of Specialists implementation status.
1837
+
1838
+ Required treatment:
1839
+
1840
+ 1. Add a short banner at the top stating that it is a historical snapshot and linking to a newer reconciliation document.
1841
+
1842
+ 2. Do not rewrite its original body or silently alter conclusions that were correct on 21 June.
1843
+
1844
+ 3. Create a new dated document, recommended path:
1845
+
1846
+ ```text
1847
+ xtrm/docs/_meta/2026-07-12-specialists-roadmap-evals-integration.md
1848
+ ```
1849
+
1850
+ 4. The new reconciliation must record:
1851
+
1852
+ * current verified roadmap status;
1853
+ * Track A/B/C ownership;
1854
+ * the dependency matrix in this section;
1855
+ * Stage 0 ownership and Channels storage decision;
1856
+ * the fact that Console now exists and already materializes Specialists observability;
1857
+ * the role of xtmux as a notification/transport adapter, not canonical Channels storage;
1858
+ * the current bridge-retirement plan;
1859
+ * links to this PRD and the canonical roadmap.
1860
+
1861
+ 5. If the organization prefers append-only meta history, the same content may be an addendum in a new file; it must not replace the June snapshot.
1862
+
1863
+ ## 13.2 Dependency matrix
1864
+
1865
+ | PRD capability | Start now | Requires critical Specialists roadmap spine | Requires Stage 0 | Requires Channels 0.1 | Requires Channels 0.2 |
1866
+ | --------------------------------------------------------- | ------------------: | ------------------------------------------: | ---------------: | --------------------: | --------------------: |
1867
+ | Telemetry correctness fixes | yes | no | no | no | no |
1868
+ | Historical run backfill and cohort analysis | yes | no | no | no | no |
1869
+ | Candidate/prompt/rule/skill fingerprinting | yes | no | no | no | no |
1870
+ | Deterministic post-run graders | yes | no | no | no | no |
1871
+ | Prompt composition inventory | yes | no | no | no | no |
1872
+ | Output-schema unification | yes | no | no | no | no |
1873
+ | Paired model A/B on controlled standalone cases | yes | no | no | no | no |
1874
+ | Minimal chain-participant guidance using current pointers | yes | partial | no | no | no |
1875
+ | Complete deterministic `<chain-context>` envelope | no | yes | no | no | no |
1876
+ | Reliable post-chain evaluator | partial | yes | no | no | no |
1877
+ | Pull-based memory experiment | yes | Opportunity 11 coordination | no | no | no |
1878
+ | Automatic reviewer-to-executor remediation | no | useful but insufficient | yes | yes | no |
1879
+ | Typed message/ack/delivery evaluation | no | no | yes | yes | no |
1880
+ | Freeform node collaboration evaluation | no | no | yes | yes | yes |
1881
+ | Console historical eval UI | yes after eval APIs | no | no | no | no |
1882
+ | Console live channel visualization | no | no | yes | yes | no |
1883
+
1884
+ ## 13.3 Critical chain-first spine
1885
+
1886
+ The following is the minimum Track A foundation that must exist before this PRD can claim complete chain awareness:
1887
+
1888
+ ```text
1889
+ bd primitive reuse decision
1890
+ -> READ_ONLY path binding
1891
+ -> persisted resolved chain shape
1892
+ -> composition review/approve/insert
1893
+ -> authoritative kind:step and semantic-edge lookup
1894
+ -> chain-driven dispatch and workspace resolution
1895
+ ```
1896
+
1897
+ This spine provides stable chain identity and pointers. It does not require Channels. It enables the chain-context envelope, selective upstream retrieval, post-chain grading and reliable model comparisons within a fixed chain topology.
1898
+
1899
+ The complete roadmap is not a prerequisite for prompt modernization. `sp epic` rewrite, final branch naming, per-repo bootstrap cleanup, Skills v4 and all dispatch guardrails can follow after the critical spine. Prompt changes that do not depend on chain topology—especially researcher and overthinker—may enter experiments earlier once telemetry and eval baselines exist.
1900
+
1901
+ ## 13.4 Channels boundary
1902
+
1903
+ Channels is the future semantic communication layer. This PRD must not create a parallel protocol in Specialists.
1904
+
1905
+ Until Channels 0.1 ships:
1906
+
1907
+ * `specialist_results` and Bead notes remain durable verbose handoff stores;
1908
+ * `sp result`, `sp feed`, Beads edges and the resolved chain shape provide pull-based context;
1909
+ * `step_completed` may provide a next-step recommendation and evaluation event;
1910
+ * the orchestrator remains responsible for dispatch/resume;
1911
+ * xtmux may notify operators or agents, but it is not the semantic source of truth.
1912
+
1913
+ After Channels 0.1:
1914
+
1915
+ * reviewer findings and verdicts become typed `work.finding`/`work.verdict` messages;
1916
+ * executor remediation is triggered through scheduler intent, not direct pane injection;
1917
+ * message delivery, acknowledgement, authority, stop-condition and convergence metrics join the eval framework;
1918
+ * bridge `step_completed` scheduling logic retires;
1919
+ * `sp steer` and `sp resume` can be implemented over the ChannelClient while retaining CLI compatibility.
1920
+
1921
+ The canonical Channels invariants must be preserved: participant L3 identity distinct from job L4 activation, typed message bodies, pointer-based payloads, read/ack separation, at-least-once delivery, scheduler indirection, authority derived from state rather than body text, forensic/evidence dual-write and explicit stop conditions.
1922
+
1923
+ ## 13.5 xtmux integration boundary
1924
+
1925
+ xtmux already provides practical inter-agent messaging, acknowledgements, unread indicators, safe handoffs and a local event log. It should be used as an operator-facing bridge and as an experimental source of usability data.
1926
+
1927
+ Allowed integration:
1928
+
1929
+ * show a short notification when a canonical Channel message or Specialists handoff becomes available;
1930
+ * increment unread badges for a participant/session;
1931
+ * link to Bead, job, chain, eval or result identifiers;
1932
+ * measure acknowledgement latency and operator intervention;
1933
+ * provide safe pointer-based handoff during the bridge period;
1934
+ * expose multiplexing dashboard, collision audit and local transport health.
1935
+
1936
+ Forbidden integration:
1937
+
1938
+ * treating the rotating xtmux JSONL log as canonical chain state;
1939
+ * encoding release authority or gate satisfaction in pane metadata;
1940
+ * bypassing Channel scheduler/authority checks through direct `send-keys`;
1941
+ * using session names as canonical participant identity;
1942
+ * inventing message kinds incompatible with Channels.
1943
+
1944
+ The target relationship is:
1945
+
1946
+ ```text
1947
+ Channels or Specialists durable state
1948
+ -> forensic/evidence event
1949
+ -> optional xtmux notification adapter
1950
+ -> human/agent attention
1951
+ ```
1952
+
1953
+ # 14. Agent-ready decomposition contract
1954
+
1955
+ ## 14.1 Required planning output
1956
+
1957
+ A local planning agent receiving this PRD must produce a Beads hierarchy, not implementation patches. Its output must contain:
1958
+
1959
+ 1. One program epic for this PRD, or a linked set of repository-owned epics when cross-repo ownership requires separation.
1960
+ 2. Root change Beads corresponding to the work packages in Section 15.
1961
+ 3. Explicit dependency edges between root Beads.
1962
+ 4. `recommended_template` for every root Bead, validated against the installed formula catalog where possible.
1963
+ 5. Priority, scrutiny, owning repository, intended specialist role and parallel lane.
1964
+ 6. File/surface ownership and collision risk.
1965
+ 7. Deterministic validation and eval gates.
1966
+ 8. Non-goals and bridge-retirement requirements.
1967
+ 9. Integration Beads for work that crosses repository boundaries.
1968
+ 10. A final plan-validation report proving that every PRD acceptance criterion maps to at least one Bead.
1969
+
1970
+ The planner must not create vague tasks such as “implement evals” or “improve reviewer”. Each root Bead must have a falsifiable externally observable outcome.
1971
+
1972
+ ## 14.2 Hierarchy and identity
1973
+
1974
+ Use this hierarchy:
1975
+
1976
+ ```text
1977
+ program epic
1978
+ -> repository/workstream epic when needed
1979
+ -> root change Bead
1980
+ -> chain molecule and step Beads at composition time
1981
+ ```
1982
+
1983
+ A root Bead is the durable work contract. A chain molecule is an execution instance around that contract. Step Beads are role-specific participation contracts. Do not conflate the program epic, chain molecule, root issue and specialist step.
1984
+
1985
+ Before the composition gate exists, manual bridge execution may create explicit step/tracking Beads. Such Beads must be labeled `bridge:manual-composition`, linked to the root and treated as temporary orchestration scaffolding. They must not become the permanent data model.
1986
+
1987
+ ## 14.3 Root change-contract template
1988
+
1989
+ Every root Bead produced from this PRD must contain equivalent fields. XML may be stored directly in the Bead description even before a deterministic parser ships because Beads stores text and the future schema is forward-compatible.
1990
+
1991
+ ```xml
1992
+ <change-contract work-package="WP-XX" type="feature|bug|refactor|research|docs" scrutiny="low|medium|high|critical">
1993
+ <problem>Concrete current defect or missing capability.</problem>
1994
+ <scope>
1995
+ <repo>xtrm-dev/specialists</repo>
1996
+ <path>explicit/file/or/directory</path>
1997
+ </scope>
1998
+ <non-goals>
1999
+ <item>Explicitly excluded adjacent work.</item>
2000
+ </non-goals>
2001
+ <dependencies>
2002
+ <work-package>WP-YY</work-package>
2003
+ </dependencies>
2004
+ <deliverables>
2005
+ <item>Code, schema, docs, fixture or report.</item>
2006
+ </deliverables>
2007
+ <validation>
2008
+ <criterion>Exact deterministic command or state assertion.</criterion>
2009
+ <criterion>Named eval suite and threshold.</criterion>
2010
+ </validation>
2011
+ <acceptance>
2012
+ <criterion>Externally observable completion condition.</criterion>
2013
+ </acceptance>
2014
+ <rollback>How to disable or revert safely.</rollback>
2015
+ <retirement-trigger>Required for bridge-only work.</retirement-trigger>
2016
+ </change-contract>
2017
+ ```
2018
+
2019
+ ## 14.4 Step-contract template
2020
+
2021
+ When a chain is composed, each specialist step must receive:
2022
+
2023
+ ```xml
2024
+ <step-contract role="executor|seconder|test-engineer|test-runner|reviewer|researcher|overthinker|...">
2025
+ <mandate>One bounded responsibility.</mandate>
2026
+ <inputs>
2027
+ <item>Root contract and named upstream evidence.</item>
2028
+ </inputs>
2029
+ <outputs>
2030
+ <item>Typed artifact or verdict.</item>
2031
+ </outputs>
2032
+ <scope>
2033
+ <path>Allowed paths or READ_ONLY evidence surface.</path>
2034
+ </scope>
2035
+ <non-goals>
2036
+ <item>Actions owned by other chain members.</item>
2037
+ </non-goals>
2038
+ <validation>
2039
+ <criterion>Evidence the next member can verify.</criterion>
2040
+ </validation>
2041
+ </step-contract>
2042
+ ```
2043
+
2044
+ The step contract must not restate the entire PRD. It references the root Bead and only adds role-specific obligations.
2045
+
2046
+ ## 14.5 Required Bead metadata
2047
+
2048
+ Each root Bead should expose, through labels, fields or a structured note:
2049
+
2050
+ ```json
2051
+ {
2052
+ "work_package": "WP-T02",
2053
+ "program": "specialists-modernization",
2054
+ "repository": "xtrm-dev/specialists",
2055
+ "lane": "telemetry-thinking",
2056
+ "priority": "P1",
2057
+ "scrutiny": "high",
2058
+ "recommended_template": "code-standard",
2059
+ "primary_role": "executor",
2060
+ "integration_owner": "telemetry-integrator",
2061
+ "parallel_safe": true,
2062
+ "file_ownership": ["src/pi/session.ts", "src/specialist/timeline-events.ts"],
2063
+ "eval_gates": ["telemetry-integrity-v1"],
2064
+ "bridge": false
2065
+ }
2066
+ ```
2067
+
2068
+ Where Beads does not support a native field, store the value in labels or the structured description rather than dropping it.
2069
+
2070
+ ## 14.6 Decomposition quality gates
2071
+
2072
+ Before implementation starts, a planning/reviewer pass must verify:
2073
+
2074
+ * every work package is represented exactly once;
2075
+ * no acceptance criterion is orphaned;
2076
+ * dependency cycles do not exist;
2077
+ * no two parallel Beads claim exclusive ownership of the same high-conflict file;
2078
+ * all cross-repo work has an explicit integration owner;
2079
+ * bridge work names its retirement trigger;
2080
+ * prompt/model changes have baseline and promotion evals;
2081
+ * DB migrations have compatibility and rollback tests;
2082
+ * Console work consumes upstream data rather than synthesizing source truth;
2083
+ * Channels work is not smuggled into Specialists bridge Beads.
2084
+
2085
+ ## 14.7 Implementation handoff requirements
2086
+
2087
+ Every implementation agent must leave a structured handoff with:
2088
+
2089
+ ```json
2090
+ {
2091
+ "work_package": "WP-XX",
2092
+ "status": "success|partial|failed|waiting",
2093
+ "files_changed": [],
2094
+ "contracts_or_schemas_changed": [],
2095
+ "decisions": [],
2096
+ "assumptions": [],
2097
+ "validation_commands": [],
2098
+ "validation_results": [],
2099
+ "eval_results": [],
2100
+ "evidence_refs": [],
2101
+ "known_deferred_paths": [],
2102
+ "downstream_attention": [],
2103
+ "bridge_retirement_notes": []
2104
+ }
2105
+ ```
2106
+
2107
+ The handoff contains decision rationale and evidence, not private chain-of-thought.
2108
+
2109
+ # 15. Work packages and multiplexed execution plan
2110
+
2111
+ ## 15.1 Work-package conventions
2112
+
2113
+ The identifiers below are stable planning anchors. The local planner may create repository-specific Bead IDs, but each Bead title or metadata must retain its `WP-*` identifier so reports, evals and integration work can map back to this PRD.
2114
+
2115
+ Scrutiny defaults:
2116
+
2117
+ * **critical:** migrations, lifecycle/identity, authority, worktree safety, evaluator enforcement;
2118
+ * **high:** telemetry semantics, output contracts, chain state, prompt/rule changes for reviewer/executor;
2119
+ * **medium:** isolated CLI, fixtures, role prompts with bounded blast radius, Console read surfaces;
2120
+ * **low:** non-normative documentation or display-only changes.
2121
+
2122
+ ## 15.2 Gate and documentation packages
2123
+
2124
+ | WP | Deliverable | Repo | Dependencies | Parallel lane | Suggested chain | Scrutiny |
2125
+ | -------- | ---------------------------------------------------------------------------------------------------- | ------------------ | ------------ | ----------------- | ------------------------- | -------- |
2126
+ | `WP-G00` | Patch canonical roadmap with Track C, dependency rules, status corrections and PRD reference | specialists | none | docs-specialists | doc-sync or code-quick | medium |
2127
+ | `WP-G01` | Add historical-snapshot banner to 2026-06-21 reconciliation and create new dated reconciliation | xtrm | none | docs-xtrm | doc-sync | medium |
2128
+ | `WP-G02` | Re-evaluate Opp 1/4/8 against `bd merge-slot`, `bd gate`, `bd swarm`; produce binding reuse decision | specialists | none | architecture-gate | research-only + premortem | high |
2129
+ | `WP-G03` | Register this PRD in roadmap/MOC indexes and map all acceptance criteria to WPs | specialists + xtrm | G00,G01 | docs-integration | doc-sync | medium |
2130
+
2131
+ `WP-G02` is a hard planning gate for chain-foundation implementation, not for telemetry baselining.
2132
+
2133
+ ## 15.3 Telemetry-integrity packages
2134
+
2135
+ | WP | Deliverable | Primary files/surfaces | Dependencies | Parallel safety | Eval gate |
2136
+ | -------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------- | --------------------------------------------------------- | --------------------------- |
2137
+ | `WP-T01` | Correct unique tool-call accounting and lifecycle dedupe | timeline events, supervisor callbacks, job metric aggregation | none | high-conflict owner | tool lifecycle fixtures |
2138
+ | `WP-T02` | Add thinking start/end spans, segment identity and durations | Pi session, timeline event schema, supervisor mapping | none | separate from T01 only with explicit file split | thinking-span fixtures |
2139
+ | `WP-T03` | Add turn/message/tool timing and time-to-first-action metrics | event constructors and aggregation | T01,T02 interfaces stable | medium | timing reconciliation |
2140
+ | `WP-T04` | Correct activity/stall semantics including `think`/`thinking` | observability queries and watchdog fixtures | none | parallel-safe if query-only | activity fixtures |
2141
+ | `WP-T05` | Wire forensic events into live Prometheus projection | forensic reader, projection collector, metrics tests | none | isolated projection owner | scrape contract |
2142
+ | `WP-T06` | Persist candidate fingerprint: model, prompt, rule, skill, runtime, tool catalog hashes | loader/runner/startup payload/schema | none | high-conflict with prompt inventory; same owner preferred | fingerprint reproducibility |
2143
+ | `WP-T07` | Ship `telemetry-integrity-v1` suite and compatibility/version markers | tests/fixtures/docs | T01–T06 | integrator-only | full suite |
2144
+
2145
+ `WP-T01`, `WP-T02` and `WP-T06` touch central runtime surfaces and must not be assigned to independent agents without a written file-ownership partition. The safer pattern is one runtime telemetry owner plus separate test/fixture agents.
2146
+
2147
+ ## 15.4 Evaluation-core packages
2148
+
2149
+ | WP | Deliverable | Dependencies | Parallel lane | Suggested role/template | Scrutiny |
2150
+ | -------- | -------------------------------------------------------------------- | --------------------------- | ------------------- | --------------------------- | -------- |
2151
+ | `WP-E01` | Eval tables, migrations, DAO and versioned types | T06 schema conventions | eval-storage | executor + seconder + tests | critical |
2152
+ | `WP-E02` | Eval forensic event catalog and shared writer | E01 type IDs may be stubbed | eval-events | executor | high |
2153
+ | `WP-E03` | Grader interface, registry and deterministic assertion result schema | none | grader-core | executor | high |
2154
+ | `WP-E04` | Data-completeness model and historical backfill | E01,E03,T07 | eval-backfill | executor/test-engineer | high |
2155
+ | `WP-E05` | Post-run evaluator in observe-only mode | E01–E03,T07 | eval-runtime | executor | high |
2156
+ | `WP-E06` | Post-chain evaluator over resolved shape | E01–E03,C03,C06 | eval-chain | executor | high |
2157
+ | `WP-E07` | Paired experiment runner, randomization and candidate manifest | E01,E03,T06 | experiment-runner | executor | high |
2158
+ | `WP-E08` | Statistics, pairwise judge records and promotion policy engine | E07 | experiment-analysis | executor/researcher | high |
2159
+ | `WP-E09` | `sp eval` CLI: suite/case/run/compare/inspect/backfill | E01,E03,E04,E07 | eval-cli | executor | medium |
2160
+ | `WP-E10` | Continuous eval scheduling and recursion exclusion | E05,E06 | eval-ops | executor/test-runner | critical |
2161
+
2162
+ `WP-E01` owns eval migrations. No other lane may independently edit the same migration block; dependent work must consume exported types or coordinate through the integration owner.
2163
+
2164
+ ## 15.5 Prompt and policy packages
2165
+
2166
+ | WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
2167
+ | -------- | --------------------------------------------------------------------------------------- | ------------------------- | ---------------- | ----------- |
2168
+ | `WP-P01` | Resolved prompt manifest with component hashes/tokens and duplicate/conflict detection | T06 | prompt-inventory | high |
2169
+ | `WP-P02` | Canonical base status/verdict vocabulary and formal merged JSON Schema | E03 optional | output-contract | critical |
2170
+ | `WP-P03` | Replace duplicate manual handoff/output rules with generated contract as SSOT | P02 | output-contract | high |
2171
+ | `WP-P04` | Role-aware runner injections: style, GitNexus, Beads close, read-only/research behavior | P01,P02,T07 | runner-policy | critical |
2172
+ | `WP-P05` | Overthinker and researcher slim prompts plus procedures moved to skills | P01–P04,E05 | prompt-advisors | medium/high |
2173
+ | `WP-P06` | Seconder and test-engineer slim prompts; formal test-engineer schema and mode | P01–P04,E05 | prompt-gates | high |
2174
+ | `WP-P07` | Executor slim prompt and language/procedure skill separation | P01–P04,E05 | prompt-executor | high |
2175
+ | `WP-P08` | Reviewer slim prompt, upstream-evidence contract and false-PASS safeguards | P01–P04,E05,C06 desirable | prompt-reviewer | critical |
2176
+ | `WP-P09` | Factorial prompt×model experiments and staged promotion for all six roles | P05–P08,E07,E08 | prompt-eval | high |
2177
+
2178
+ Role prompt files are parallel-safe only after `WP-P02` and `WP-P04` freeze shared contracts. Each role owns its own specialist JSON and role-specific tests; one integration owner handles shared mandatory-rule/index edits.
2179
+
2180
+ ## 15.6 Critical chain-foundation packages
2181
+
2182
+ | WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
2183
+ | -------- | ---------------------------------------------------------------------------------------- | ------------ | ----------------- | ----------- |
2184
+ | `WP-C01` | READ_ONLY path binding independent of owner liveness | G02 | chain-workspace | critical |
2185
+ | `WP-C02` | Persisted resolved chain shape with ordered steps/gates/status/job pointers | G02 | chain-state | critical |
2186
+ | `WP-C03` | `sp chain review/approve/insert` using resolved shape and reused bd primitives | C02,G02 | chain-composition | critical |
2187
+ | `WP-C04` | Authoritative `kind:step`, root/step resolution and semantic-edge query helpers | G02 | chain-graph | high |
2188
+ | `WP-C05` | Chain-driven dispatch/workspace resolution and deprecation bridge for job/worktree flags | C01–C04 | chain-dispatch | critical |
2189
+ | `WP-C06` | Deterministic `<chain-context>` envelope with completeness flags and upstream pointers | C02,C04,T06 | chain-context | high |
2190
+ | `WP-C07` | Shared chain-participant rule and role-specific identity clauses | C06,P04 | chain-policy | high |
2191
+ | `WP-C08` | Structured handoff extension for decisions, inputs consumed and downstream attention | P02,C06 | handoff-contract | high |
2192
+ | `WP-C09` | Minimal `step_completed` visibility/recommendation bridge compatible with Channels | C02,C03,E02 | chain-events | medium/high |
2193
+ | `WP-C10` | `chain-participation-v1` eval suite | C06–C09,E03 | chain-eval | high |
2194
+
2195
+ `WP-C09` must not auto-resume peers or implement a scheduler. Its retirement trigger is Channels 0.1 acceptance.
2196
+
2197
+ ## 15.7 Pull-based memory packages
2198
+
2199
+ | WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
2200
+ | -------- | --------------------------------------------------------------------------- | ------------ | ----------------- | -------- |
2201
+ | `WP-M01` | Audit actual `bd memories`/`bd recall` behavior, provenance and filtering | none | memory-research | medium |
2202
+ | `WP-M02` | Targeted memory search/relevant interface or wrapper with bounded output | M01 | memory-cli | high |
2203
+ | `WP-M03` | Disable eager injection behind experiment flag and preserve rollback | M01,T06 | memory-runtime | high |
2204
+ | `WP-M04` | Memory-recall mandatory rule, role opt-outs and authority/provenance policy | M01,P04 | memory-policy | high |
2205
+ | `WP-M05` | Memory query/recall telemetry and deterministic graders | M02–M04,E03 | memory-eval | high |
2206
+ | `WP-M06` | Role-by-role A/B rollout and promotion | M05,E07,E08 | memory-experiment | high |
2207
+
2208
+ ## 15.8 Role-suite packages
2209
+
2210
+ These packages are fixture/dataset heavy and can run in parallel after the grader interface stabilizes.
2211
+
2212
+ | WP | Suite | Core outcome |
2213
+ | -------- | ----------------------------- | -------------------------------------------------------------------------- |
2214
+ | `WP-S01` | `reviewer-regression-v1` | verdict accuracy, false PASS, finding precision/recall |
2215
+ | `WP-S02` | `executor-swe-style-v1` | hidden tests, scope precision, regression and minimality |
2216
+ | `WP-S03` | `seconder-regression-v1` | bounded gate accuracy and calibration |
2217
+ | `WP-S04` | `test-engineer-regression-v1` | fail-to-pass, pass-to-pass and source-boundary discipline |
2218
+ | `WP-S05` | `researcher-regression-v1` | claim correctness, citation entailment, freshness and inference separation |
2219
+ | `WP-S06` | `overthinker-capability-v1` | risk/assumption recall, precision and depth calibration |
2220
+ | `WP-S07` | `chain-participation-v1` | chain identity, selective retrieval, contradiction handling and handoff |
2221
+ | `WP-S08` | `telemetry-integrity-v1` | metric/event correctness and reconciliation |
2222
+
2223
+ All suite cases require versioned provenance. Capability cases that become reliably solved should be promoted into regression suites.
2224
+
2225
+ ## 15.9 Console packages
2226
+
2227
+ | WP | Deliverable | Repo | Dependencies | Scrutiny |
2228
+ | -------- | ---------------------------------------------------------- | ------- | ------------ | ----------- |
2229
+ | `WP-U01` | Materialize eval summaries, completeness and evidence refs | console | E01,E02 | high |
2230
+ | `WP-U02` | Eval query APIs and DTOs | console | U01 | high |
2231
+ | `WP-U03` | Eval overview, experiment, case and trial routes | console | U02 | medium |
2232
+ | `WP-U04` | Trial timeline with thinking/tool/text/evidence drilldown | console | U02,T07 | high |
2233
+ | `WP-U05` | Historical mining and case-promotion workflow | console | U02,E04 | medium/high |
2234
+ | `WP-U06` | Live eval updates, adjudication and promotion UX | console | U02,E08,E10 | high |
2235
+
2236
+ Console does not own evaluator execution or canonical scores. It reads/materializes Specialists output and links to forensic evidence.
2237
+
2238
+ ## 15.10 Deferred xtrm/Channels packages
2239
+
2240
+ These are linked future epics, not immediate child Beads of the Specialists implementation program unless the operator explicitly starts Track B.
2241
+
2242
+ | WP | Deliverable | Dependency |
2243
+ | -------- | ---------------------------------------------------------------------------- | -------------------------------- |
2244
+ | `WP-X01` | Stage 0 detail spec: daemon, state.db, socket, schema registry, registration | architecture gate |
2245
+ | `WP-X02` | Resolve canonical Channels storage/ownership in `packages/channels` | X01 |
2246
+ | `WP-X03` | Channels 0.1 autonomous chain path | X01,X02,critical chain semantics |
2247
+ | `WP-X04` | xtmux notification adapter over canonical message/handoff events | X03 |
2248
+ | `WP-X05` | Channels 0.2 freeform/node activation and collaboration evals | X03 plus runtime tuning |
2249
+
2250
+ ## 15.11 Initial parallel wave
2251
+
2252
+ After documentation gate creation, the first multiplexed wave may safely start these lanes:
2253
+
2254
+ ```text
2255
+ Lane A: WP-G00 + WP-G01
2256
+ Lane B: WP-G02 architecture/reuse audit
2257
+ Lane C: WP-T01/T02 design and fixtures under one runtime telemetry owner
2258
+ Lane D: WP-T04 activity query fix
2259
+ Lane E: WP-T05 forensic Prometheus wiring
2260
+ Lane F: WP-T06 fingerprint design
2261
+ Lane G: WP-E03 grader interface design
2262
+ Lane H: WP-M01 memory CLI/provenance audit
2263
+ Lane I: Console eval datasource/API design spike, no source writes
2264
+ ```
2265
+
2266
+ Implementation of `WP-E01`, `WP-P02`, `WP-C01–C05` begins only after their named gates are closed.
2267
+
2268
+ ## 15.12 File-ownership and collision policy
2269
+
2270
+ High-conflict surfaces require a single owner per wave:
2271
+
2272
+ | Surface | Exclusive owner rule |
2273
+ | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
2274
+ | `src/specialist/runner.ts` | one prompt/runtime-policy owner |
2275
+ | `src/pi/session.ts` + `src/specialist/timeline-events.ts` + central supervisor callback path | one telemetry-runtime owner per wave |
2276
+ | `src/specialist/observability-sqlite.ts` migrations | one schema/migration owner; other agents consume interfaces |
2277
+ | `src/specialist/prometheus-projection.ts` | one metrics-projection owner |
2278
+ | mandatory-rules index/global sets | one policy integration owner |
2279
+ | base output schema/contract builder | one output-contract owner |
2280
+ | each `config/specialists/<role>.specialist.json` | one role owner after shared contracts freeze |
2281
+ | Console materializer schema | one Console data owner |
2282
+
2283
+ Agents may work concurrently on tests, fixtures and role-specific files when their production surface does not overlap. `xtmux worktree-collisions` and repository diff checks must be run before integration.
2284
+
2285
+ ## 15.13 Multiplexing operating protocol
2286
+
2287
+ During bridge-era execution:
2288
+
2289
+ 1. Every agent receives one root Bead and one worktree/branch.
2290
+ 2. Handoffs use Bead notes/results as durable content and xtmux pointer messages as notification.
2291
+ 3. `message-send` is for short state or pointer messages; long requirements remain in the Bead or a versioned file.
2292
+ 4. Recipients acknowledge messages. Lack of acknowledgement is visible but does not mutate release authority.
2293
+ 5. `safe-send-pointer` may steer an idle/needs-input target; it must not inject into a running/thinking target.
2294
+ 6. The integration owner monitors `dashboard`, `audit` and `worktree-collisions`.
2295
+ 7. Agents do not merge or push outside their contract unless the chain explicitly assigns publication ownership.
2296
+ 8. Cross-lane decisions are recorded in the relevant Bead and echoed to dependent lane owners.
2297
+ 9. Findings that require new scope become `discovered-from` follow-up Beads rather than silent expansion.
2298
+ 10. Final integration runs the mapped regression suites and records evidence before closing the parent work package.
2299
+
2300
+ ## 15.14 Recommended agent roster and assignment
2301
+
2302
+ The local orchestrator should instantiate bounded agents by surface, not one agent per arbitrary bullet. A practical initial roster is:
2303
+
2304
+ | Agent identity | Primary responsibility | Assigned WPs | Must not own concurrently |
2305
+ | ---------------------------- | ----------------------------------------------------- | ----------------------- | -------------------------------------------------- |
2306
+ | `docs-reconciler` | Canonical roadmap/meta patches and cross-links | G00,G01,G03 | runtime code |
2307
+ | `architecture-reuse-auditor` | `bd` primitive reuse and bridge-retirement decision | G02 | implementation before decision approval |
2308
+ | `telemetry-runtime-owner` | Pi/timeline/supervisor tool and thinking semantics | T01,T02,T03 | eval DB migrations |
2309
+ | `telemetry-query-owner` | activity/stall query and historical compatibility | T04 | central callback path without coordination |
2310
+ | `metrics-projection-owner` | forensic-to-Prometheus wiring | T05 | Console panel implementation |
2311
+ | `fingerprint-owner` | resolved candidate manifest and startup hashes | T06,P01 | role prompt edits before schema approval |
2312
+ | `telemetry-integrator` | fixtures, compatibility and telemetry suite | T07,S08 | unrelated prompt work |
2313
+ | `eval-schema-owner` | eval migrations, DAO and storage types | E01 | other observability migrations in same wave |
2314
+ | `grader-core-owner` | grader interface and assertion schema | E03 | role-specific judge calibration |
2315
+ | `eval-runtime-owner` | post-run/backfill/eval CLI | E04,E05,E09 | post-chain logic before chain shape exists |
2316
+ | `experiment-owner` | paired trials, statistics and promotion | E07,E08 | candidate prompt implementation |
2317
+ | `chain-architecture-owner` | C01–C05 critical chain spine | C01,C02,C03,C04,C05 | prompt refactors |
2318
+ | `chain-context-owner` | context envelope, handoff and chain eval | C06,C07,C08,C09,C10,E06 | Channels implementation |
2319
+ | `prompt-contract-owner` | output schema and runner policy | P02,P03,P04 | role prompt files in parallel without freeze point |
2320
+ | `advisor-prompt-owner` | researcher/overthinker candidates | P05,S05,S06 | shared runner or global rule index |
2321
+ | `gate-prompt-owner` | seconder/test-engineer candidates | P06,S03,S04 | reviewer/executor files |
2322
+ | `executor-prompt-owner` | executor candidate and suite | P07,S02 | reviewer prompt |
2323
+ | `reviewer-prompt-owner` | reviewer candidate and suite | P08,S01 | executor prompt |
2324
+ | `memory-owner` | memory audit, runtime, rule and eval | M01–M06 | chain messaging protocol |
2325
+ | `console-eval-owner` | materializer/API/eval views | U01–U06 | upstream evaluator semantics |
2326
+ | `integration-reviewer` | cross-lane review, dependency and acceptance coverage | all integration gates | production implementation in reviewed lane |
2327
+
2328
+ The roster is a role map, not a requirement to run every agent simultaneously. Concurrency should be limited by integration bandwidth and file overlap. A recommended first wave is six to nine active agents, with one integration reviewer and one orchestrator tracking the whole program.
2329
+
2330
+ Assignment rules:
2331
+
2332
+ 1. An agent owns one high-conflict production surface at a time.
2333
+ 2. Test/fixture agents may support a production owner but cannot independently redefine event or schema semantics.
2334
+ 3. Role prompt owners start only after the prompt-contract owner publishes the frozen base output and injection contract.
2335
+ 4. The chain-context owner consumes the chain-architecture owner’s published resolved-shape API rather than editing its internals opportunistically.
2336
+ 5. The Console owner receives versioned upstream DTO/event contracts; it does not infer them from private database internals.
2337
+ 6. The integration reviewer verifies the Bead contract, diff, tests, eval evidence and downstream effects, but does not become the default fixer.
2338
+
2339
+ ## 15.15 Recommended integration order
2340
+
2341
+ Merge and promotion order:
2342
+
2343
+ ```text
2344
+ documentation/reuse decisions
2345
+ -> telemetry schema and integrity
2346
+ -> eval schema/grader core
2347
+ -> chain state foundations
2348
+ -> output contract and runner policy
2349
+ -> role prompt candidates
2350
+ -> chain context and handoff
2351
+ -> memory treatment
2352
+ -> model/prompt experiments
2353
+ -> Console product surfaces
2354
+ -> enforcement
2355
+ -> Channels replacement of bridge communication
2356
+ ```
2357
+
2358
+ A lane may develop ahead on a feature branch, but it may not be promoted across an unmet integration gate.
2359
+
2360
+ # 16. Rollout plan
2361
+
2362
+ ## Phase 0 — Documentation reconciliation and immutable baseline
2363
+
2364
+ 1. Complete `WP-G00`–`WP-G03`.
2365
+ 2. Capture resolved configs for all six roles across representative repos.
2366
+ 3. Hash prompt, rules, skills, runner and tool catalog.
2367
+ 4. Export first-turn input, active runtime, tools, tokens, verdicts and fix loops.
2368
+ 5. Freeze historical reviewer/executor/chain cohorts and document completeness.
2369
+ 6. Record current roadmap and code status so later reports do not compare against a moving baseline.
2370
+
2371
+ **Exit criteria:** canonical cross-links exist; the new dated reconciliation is approved; every implementation WP is traceable; a repeatable baseline report exists.
2372
+
2373
+ ## Phase 1 — Telemetry integrity
2374
+
2375
+ 1. Complete `WP-T01`–`WP-T07`.
2376
+ 2. Version changed metric semantics rather than silently rewriting historical dashboards.
2377
+ 3. Recompute a sample of historical metrics and compare against raw event traces.
2378
+
2379
+ **Exit criteria:** no known counting ambiguity in any metric used for A/B promotion; `telemetry-integrity-v1` passes.
2380
+
2381
+ ## Phase 2 — Eval core and historical backfill
2382
+
2383
+ 1. Complete `WP-E01`–`WP-E05` and `WP-E09` in observe-only mode.
2384
+ 2. Backfill selected per-repo databases without mutating raw events.
2385
+ 3. Create data-completeness summaries and initial role cohorts.
2386
+ 4. Start `WP-S01`–`WP-S06` fixture creation in parallel.
2387
+
2388
+ **Exit criteria:** historical trials are inspectable; new runs automatically receive deterministic post-run scores; no evaluator can block execution yet.
2389
+
2390
+ ## Phase 3 — Critical chain-first roadmap spine
2391
+
2392
+ 1. Close `WP-G02` reuse decisions.
2393
+ 2. Implement `WP-C01`–`WP-C05`.
2394
+ 3. Persist chain shapes and expose composition/read surfaces.
2395
+ 4. Keep any `step_completed` behavior visibility-only.
2396
+
2397
+ **Exit criteria:** a chain has durable identity independent of a live executor; ordered step/gate shape and workspace resolution are queryable and tested.
2398
+
2399
+ ## Phase 4 — Output contract and runner policy consolidation
2400
+
2401
+ 1. Complete `WP-P01`–`WP-P04`.
2402
+ 2. Normalize status/verdict schemas.
2403
+ 3. Remove duplicated output, GitNexus, style and Beads workflow instructions.
2404
+ 4. Introduce compatibility shims and output-validation events.
2405
+
2406
+ **Exit criteria:** the generated output contract is the single machine-readable SSOT; role-aware injections pass regression fixtures.
2407
+
2408
+ ## Phase 5 — Role prompt experiments
2409
+
2410
+ 1. Run `WP-P05` first for overthinker/researcher.
2411
+ 2. Run `WP-P06` for seconder/test-engineer.
2412
+ 3. Run `WP-P07` and `WP-P08` last because executor/reviewer have the highest operational blast radius.
2413
+ 4. Use paired or factorial experiments, never unpaired historical PASS rates as the promotion criterion.
2414
+
2415
+ **Exit criteria:** prompt reductions meet role-specific thresholds without deterioration of hard gates or confidence-interval promotion rules.
2416
+
2417
+ ## Phase 6 — Complete chain context and handoff
2418
+
2419
+ 1. Complete `WP-C06`–`WP-C10` and `WP-E06`.
2420
+ 2. Inject deterministic chain identity and pointers.
2421
+ 3. Add selective retrieval, handoff decision records and chain participation graders.
2422
+
2423
+ **Exit criteria:** reviewers and downstream gates can identify all required upstream inputs; context completeness is explicit; chain-participation suite meets threshold.
2424
+
2425
+ ## Phase 7 — Pull-based memory
2426
+
2427
+ 1. Complete `WP-M01`–`WP-M05`.
2428
+ 2. Run treatment/control with eager injection still available as rollback.
2429
+ 3. Complete `WP-M06` only after necessary-memory and unnecessary-query graders are calibrated.
2430
+
2431
+ **Exit criteria:** first-turn memory payload materially decreases with no statistically material increase in historical-context misses.
2432
+
2433
+ ## Phase 8 — Model/prompt benchmark matrix
2434
+
2435
+ 1. Complete `WP-E07`, `WP-E08` and remaining role suites.
2436
+ 2. Run reviewer prompt×model factorial experiments first.
2437
+ 3. Expand to executor, test-engineer and researcher.
2438
+ 4. Add internal SWE-bench-style and selected external public benchmarks.
2439
+ 5. Establish scheduled regression and capability runs.
2440
+
2441
+ ## Phase 9 — Console Eval Lab
2442
+
2443
+ 1. Complete `WP-U01`–`WP-U06`.
2444
+ 2. Preserve aggregate-to-forensic drilldown and freshness/completeness metadata.
2445
+ 3. Support live experiment visibility without moving evaluator ownership into Console.
2446
+
2447
+ ## Phase 10 — Enforcement
2448
+
2449
+ 1. Promote high-precision deterministic graders from observe to warn.
2450
+ 2. Measure false positives and operator overrides.
2451
+ 3. Promote only approved rules to enforce.
2452
+ 4. Keep LLM judges advisory unless separately authorized.
2453
+
2454
+ ## Phase 11 — Channels integration
2455
+
2456
+ After xtrm Stage 0 and Channels 0.1 acceptance:
2457
+
2458
+ 1. replace manual remediation dispatch with typed scheduler intents;
2459
+ 2. dual-write semantic messages to forensic/evidence;
2460
+ 3. add message delivery, acknowledgement, authority and convergence evals;
2461
+ 4. retire bridge scheduling behavior from `step_completed`;
2462
+ 5. add optional xtmux notification adapter;
2463
+ 6. defer freeform/node collaboration enforcement until Channels 0.2 is measured.
2464
+
2465
+ # 17. Detailed implementation backlog
2466
+
2467
+ The Epics below are capability groupings. For actual Beads decomposition and parallel assignment, Section 15 work packages are normative. A planner may group multiple WPs under one Epic, but must not merge their contracts or erase their dependencies.
2468
+
2469
+ ## Epic A — Prompt composition inventory
2470
+
2471
+ * Add a resolved prompt manifest command.
2472
+ * Report component bytes/tokens and hashes.
2473
+ * Capture eager skill body costs.
2474
+ * Compare API first-turn truth with component estimate.
2475
+ * Detect duplicate rule IDs and repeated normalized text.
2476
+ * Detect conflicting enums/schema fields.
2477
+
2478
+ ## Epic B — Output contract unification
2479
+
2480
+ * Select canonical status/verdict vocabulary.
2481
+ * Convert all role output schemas to formal JSON Schema.
2482
+ * Version merged output schema.
2483
+ * Remove handoff-rule duplication.
2484
+ * Emit output validation forensic events.
2485
+ * Add regression fixtures for markdown and JSON modes.
2486
+
2487
+ ## Epic C — Six prompt refactors
2488
+
2489
+ * Reviewer core prompt and policy extraction.
2490
+ * Executor core prompt and language-skill separation.
2491
+ * Overthinker adaptive-depth prompt.
2492
+ * Seconder bounded gate prompt.
2493
+ * Test-engineer explicit mode and source boundary.
2494
+ * Researcher epistemic core and routing extraction.
2495
+
2496
+ ## Epic D — Runner injection cleanup
2497
+
2498
+ * Role-aware style directive.
2499
+ * Single GitNexus policy source.
2500
+ * Writer-only commit/close instructions.
2501
+ * Split worktree boundary from evidence retrieval.
2502
+ * Remove global contradictions for researcher/read-only roles.
2503
+
2504
+ ## Epic E — Chain context
2505
+
2506
+ * Persist/resolve chain template and ordered steps.
2507
+ * Chain context envelope builder.
2508
+ * Beads semantic edge query helpers.
2509
+ * Context completeness flags.
2510
+ * Shared chain participant rule.
2511
+ * Handoff extension.
2512
+
2513
+ ## Epic F — Memory capability
2514
+
2515
+ * Search/relevant/recall command design.
2516
+ * Provenance and supersession fields.
2517
+ * Query telemetry.
2518
+ * Pull policy rule.
2519
+ * Eager injection experiment flag.
2520
+
2521
+ ## Epic G — Telemetry integrity
2522
+
2523
+ * Thinking span model.
2524
+ * Tool-call deduplication and parallel duration.
2525
+ * Turn/message timing.
2526
+ * Activity query fixes.
2527
+ * Forensic Prometheus collector wiring.
2528
+ * Candidate fingerprint fields.
2529
+ * Telemetry integrity suite.
2530
+
2531
+ ## Epic H — Eval storage and runtime
2532
+
2533
+ * Eval tables and migrations.
2534
+ * Eval forensic events.
2535
+ * Grader interface and registry.
2536
+ * Post-run evaluator.
2537
+ * Post-chain evaluator.
2538
+ * Historical backfill.
2539
+ * Trial artifact bundle.
2540
+
2541
+ ## Epic I — Role suites
2542
+
2543
+ * Reviewer seeded-patch suite.
2544
+ * Executor hidden-test suite.
2545
+ * Seconder smell/scope suite.
2546
+ * Test-engineer fail-to-pass suite.
2547
+ * Research claim/citation suite.
2548
+ * Overthinker decision-risk suite.
2549
+ * Chain participation suite.
2550
+
2551
+ ## Epic J — Experimentation and statistics
2552
+
2553
+ * Candidate configuration format.
2554
+ * Paired runner.
2555
+ * Trial randomization.
2556
+ * Pairwise judge and order swap.
2557
+ * Bootstrap and binary paired analysis.
2558
+ * Promotion policy engine.
2559
+
2560
+ ## Epic K — Console integration
2561
+
2562
+ * Eval datasource/materializer rows.
2563
+ * Eval APIs.
2564
+ * Overview and experiment pages.
2565
+ * Trial timeline and evidence drawer.
2566
+ * Historical mining and case promotion.
2567
+ * Realtime updates.
2568
+ * Human adjudication and promotion UX.
2569
+
2570
+ # 18. Acceptance criteria
2571
+
2572
+ ## Prompt modernization
2573
+
2574
+ * First-turn input tokens decrease materially for each refactored role.
2575
+ * No role exceeds baseline critical failure rate.
2576
+ * Output schema validity is at least 99.5%.
2577
+ * Reviewer false-PASS rate does not increase.
2578
+ * Executor scope violations do not increase.
2579
+ * Seconder remains within configured time/tool budget.
2580
+ * Test-engineer source-boundary violations remain zero.
2581
+ * Researcher citation correctness does not regress.
2582
+
2583
+ ## Chain context
2584
+
2585
+ * Every chain step receives valid chain identity and root/step pointers.
2586
+ * Context completeness is explicitly marked.
2587
+ * Reviewer can identify all required upstream gates without filesystem hunting.
2588
+ * Upstream-result retrieval is selective and evidence-linked.
2589
+ * Handoff decision records are parseable.
2590
+
2591
+ ## Memory
2592
+
2593
+ * Eager memory tokens are eliminated or substantially reduced for treatment candidates.
2594
+ * Necessary-memory recall remains above an approved threshold.
2595
+ * Unnecessary memory-query rate is measured and bounded.
2596
+ * Memory results include provenance.
2597
+
2598
+ ## Telemetry
2599
+
2600
+ * Unique tool-call count is correct in fixture traces.
2601
+ * Thinking spans are paired and durations non-negative.
2602
+ * Active plus waiting time reconciles with elapsed time within tolerance.
2603
+ * Forensic-derived metric families appear in the live projection.
2604
+ * No forbidden labels are emitted.
2605
+
2606
+ ## Eval platform
2607
+
2608
+ * Existing runs can be backfilled without modifying raw events.
2609
+ * New runs receive deterministic post-run scores.
2610
+ * Chains receive post-chain scores.
2611
+ * Candidate fingerprints allow exact comparison.
2612
+ * Paired experiments produce confidence intervals and win/tie/loss.
2613
+ * Promotion policy can block regressions and recommend qualifying candidates.
2614
+
2615
+ ## Console
2616
+
2617
+ * Eval overview and trial detail are accessible from existing Console navigation.
2618
+ * Aggregate views drill down to forensic evidence.
2619
+ * Live and historical data display freshness/completeness.
2620
+ * No direct frontend access to per-repo SQLite is introduced.
2621
+
2622
+ # 19. Risks and mitigations
2623
+
2624
+ | Risk | Consequence | Mitigation |
2625
+ | ---------------------------------------------- | ------------------------------- | -------------------------------------------------------------------------------- |
2626
+ | Prompt slimming removes useful tacit behavior | Quality regression | Paired regression suite, staged roles, rollback by version |
2627
+ | Runtime and prompt change simultaneously | Attribution failure | Factorial design and candidate fingerprints |
2628
+ | Historical data is incomplete | Misleading retrospective scores | Data completeness flags; separate estimated from complete |
2629
+ | LLM judge bias | Incorrect promotion | Human calibration, order swap, multiple judges, deterministic hard gates |
2630
+ | Continuous eval adds latency | Slower chains | Asynchronous post-run evaluation; only hard deterministic gates on critical path |
2631
+ | Eval recursion | Evaluator jobs grade themselves | Explicit evaluator participant kind and exclusion policy |
2632
+ | High-cardinality metric explosion | Prometheus instability | Existing allowlist/forbidden labels; forensic drilldown |
2633
+ | Memory retrieval becomes another ritual | Token/tool waste | Trigger-based rule and unnecessary-query grader |
2634
+ | Chain context becomes a full history dump | Context bloat | Pointer envelope plus on-demand retrieval |
2635
+ | Tool count changes break historical dashboards | Apparent discontinuity | Version metric semantics and retain legacy field during transition |
2636
+ | External benchmark overfits coding only | Misleading role comparison | Internal role and chain suites remain primary |
2637
+ | Console materializer duplicates source truth | Architectural drift | Summary/evidence projection only; no evaluator ownership in UI |
2638
+ | Local rule/skill overrides escape experiments | Non-reproducible trial | Resolve and hash all effective components at launch |
2639
+
2640
+ # 20. Open decisions
2641
+
2642
+ The following decisions should be closed before implementation reaches enforcement:
2643
+
2644
+ 1. Canonical base status and reviewer verdict casing.
2645
+ 2. Whether eval tables live in every per-repo observability DB or in a separate namespace within the same file; this PRD recommends the same per-repo DB initially.
2646
+ 3. Global registry location for cross-repo experiment definitions before `state.db` exists.
2647
+ 4. Exact memory retrieval CLI surface and provenance availability in Beads.
2648
+ 5. Which deterministic graders are allowed to block at first enforcement.
2649
+ 6. Authoritative provider cost source; token usage should remain primary until cost provenance is versioned.
2650
+ 7. Minimum trial count and confidence thresholds by role.
2651
+ 8. Retention policy for eval artifacts and hidden benchmark environments.
2652
+ 9. Whether the global style directive is removed entirely or replaced by role-specific concise-output settings.
2653
+ 10. How package and installation-local skill references are resolved and fingerprinted when they are not repository files.
2654
+ 11. Canonical storage ownership for Channels after Stage 0; recommendation is `packages/channels` over `state.db`, with any observability-DB implementation treated as a removable adapter.
2655
+ 12. Whether the new dated reconciliation is maintained as a one-time snapshot or becomes an append-only sequence of dated status documents.
2656
+ 13. Exact boundary between `step_completed` visibility and future Channel scheduler intent.
2657
+ 14. Which work packages the operator wants started concurrently given available local agents and integration bandwidth.
2658
+
2659
+ # Appendix A — Proposed core system prompts
2660
+
2661
+ ## A.1 Reviewer
2662
+
2663
+ ```text
2664
+ You are the final evidence-based release reviewer for a completed specialist chain.
2665
+
2666
+ The seconder owns the initial contract and implementation-sanity gate. Consume
2667
+ upstream QA, security and obligations evidence before reviewing. Do not repeat an
2668
+ upstream gate unless its result is missing, contradicted by stronger evidence or
2669
+ invalidated by a later diff.
2670
+
2671
+ Review unresolved findings, critical requirement coverage, changed high-risk
2672
+ surfaces and required release evidence. Use the root contract and current artifacts
2673
+ as authority. Treat upstream outputs as evidence-backed claims, not as authority over
2674
+ observed code or tests.
2675
+
2676
+ Never edit files. Do not emit preference-only findings. Every blocking finding must
2677
+ identify the unmet requirement or gate and cite concrete evidence.
2678
+
2679
+ PASS requires all critical requirements and required gates to be evidenced with no
2680
+ blocking finding. PARTIAL means a fixable gap or insufficient evidence remains. FAIL
2681
+ means a core requirement is unmet, authoritative evidence is contradictory, or
2682
+ required review inputs are unusable.
2683
+
2684
+ On re-review after PARTIAL, inspect the delta and prior findings. Carry forward
2685
+ approvals for unchanged areas. Return only the configured output contract.
2686
+ ```
2687
+
2688
+ ## A.2 Executor
2689
+
2690
+ ```text
2691
+ You are a scoped implementation specialist. Implement the supplied Bead contract in
2692
+ the assigned worktree.
2693
+
2694
+ Resolve scope, inspect the relevant implementation and dependency impact, and make
2695
+ the smallest correct change. Follow repository, language and framework conventions.
2696
+ Prefer existing code and dependencies over new abstractions or packages.
2697
+
2698
+ Do not add adjacent cleanup, speculative flexibility, broad refactors, tests,
2699
+ documentation or features unless they are in scope. If a necessary change lies
2700
+ outside authorized scope, return a partial handoff describing the required expansion.
2701
+
2702
+ Preserve security, data integrity, compatibility and external-boundary validation.
2703
+ Run the validation declared by the task plus focused checks needed to establish
2704
+ implementation correctness. Broad authoritative test execution belongs to the
2705
+ configured QA gate unless the task explicitly requires it.
2706
+
2707
+ Before completion, verify changed paths against scope. Return only the configured
2708
+ output contract with changes, verification, assumptions, risks and follow-ups.
2709
+ ```
2710
+
2711
+ ## A.3 Overthinker
2712
+
2713
+ ```text
2714
+ You are a read-only decision-review specialist for uncertain or high-impact choices.
2715
+
2716
+ Analyze only as deeply as the decision warrants. Report conclusions, evidence,
2717
+ assumptions, alternatives and risks; do not expose private chain-of-thought.
2718
+
2719
+ For costly, risky or difficult-to-reverse decisions, stress-test the leading option
2720
+ with a counterargument or premortem and identify evidence that would change the
2721
+ recommendation. For simple reversible decisions, answer directly.
2722
+
2723
+ State confidence, material uncertainty and the next validation step. Never edit
2724
+ files. Return only the configured output contract.
2725
+ ```
2726
+
2727
+ ## A.4 Seconder
2728
+
2729
+ ```text
2730
+ You are the bounded read-only pre-QA gate for a completed writer diff.
2731
+
2732
+ Determine whether the output satisfies the root contract and appears implementation-
2733
+ sane enough to justify expensive QA. Evaluate scope/compliance and implementation
2734
+ quality as separate dimensions. If evidence for one dimension is missing, mark only
2735
+ that dimension UNCLEAR rather than guessing.
2736
+
2737
+ Do not perform release review, security audit, broad architecture critique, test-
2738
+ coverage review or style review. Do not emit preference-only findings.
2739
+
2740
+ Overall PASS requires both dimensions to pass; FAIL applies when either fails;
2741
+ otherwise return PARTIAL. Keep inspection within the runtime budget and return only
2742
+ the configured output contract.
2743
+ ```
2744
+
2745
+ ## A.5 Test-engineer
2746
+
2747
+ ```text
2748
+ You are a test-authoring specialist. The dispatch context explicitly defines
2749
+ `test_only` or `post_implementation` mode.
2750
+
2751
+ Build tests from the root requirements and actual implementation evidence. Follow
2752
+ existing test, fixture, smoke and harness conventions before introducing new patterns.
2753
+
2754
+ Your default edit boundary is tests, fixtures, recorded fixtures, smoke/E2E assets
2755
+ and test-harness support. Production source changes are forbidden unless dispatch
2756
+ explicitly authorizes a named helper or export. When source behavior or
2757
+ instrumentation is wrong, return `source_bug_suspected` with evidence.
2758
+
2759
+ Cover critical behavior rather than optimizing line coverage. Add operational or
2760
+ telemetry assertions when required by the changed surface. Run focused checks needed
2761
+ to validate authored assets; test-runner owns authoritative broad execution.
2762
+
2763
+ Return exact commands, setup/cleanup, coverage mapping, deferred paths and failure
2764
+ ownership through the configured output contract.
2765
+ ```
2766
+
2767
+ ## A.6 Researcher
2768
+
2769
+ ```text
2770
+ You are an external-evidence researcher.
2771
+
2772
+ Use current authoritative sources for unstable claims involving libraries, APIs,
2773
+ versions, releases, ecosystem behavior or external products. Do not perform external
2774
+ research when current repository evidence or stable reasoning is sufficient.
2775
+
2776
+ Prefer primary documentation, standards, research papers and source repositories.
2777
+ Record relevant versions and dates. Distinguish sourced facts from inference and
2778
+ report unresolved gaps. Verify consequential claims across independent sources when
2779
+ practical and stop when the requested success criteria are met.
2780
+
2781
+ Report tool failures instead of inventing evidence. Never expose secrets in queries
2782
+ and never edit project files. Lead with the answer and return findings, citations,
2783
+ confidence and gaps through the configured output contract.
2784
+ ```
2785
+
2786
+ # Appendix B — Proposed `chain-participant-context` rule
2787
+
2788
+ ```text
2789
+ You are a participant in a multi-step specialist chain, not an isolated agent.
2790
+
2791
+ The chain root change-contract defines the overall objective and acceptance criteria.
2792
+ Your current step-contract defines your mandate and edit boundary. Upstream specialist
2793
+ outputs are evidence and handoffs; they do not override the root contract, current
2794
+ repository state or directly observed artifacts.
2795
+
2796
+ At startup, read the injected chain context and identify the chain, root Bead, current
2797
+ step, completed upstream steps and required downstream handoff. Consume preloaded
2798
+ dependency context before additional commands.
2799
+
2800
+ When required inputs are absent or ambiguous, reconstruct only the necessary context
2801
+ using the current step Bead, Beads dependency/relationship commands, `sp result` for
2802
+ upstream final handoffs and `sp feed --json` only when a claimed action or chronology
2803
+ must be verified.
2804
+
2805
+ Do not repeat work already completed upstream unless your role explicitly validates
2806
+ it, evidence is missing or current artifacts contradict it. Record contradictions.
2807
+ Do not request private chain-of-thought; consume decisions, rationale, assumptions,
2808
+ evidence, verification results and unresolved questions.
2809
+
2810
+ Your final handoff must allow the next chain member to continue without reconstructing
2811
+ your entire session.
2812
+ ```
2813
+
2814
+ # Appendix C — Proposed memory recall rule
2815
+
2816
+ ```text
2817
+ Historical project memories are available through targeted Beads memory queries.
2818
+
2819
+ Query memory only when the task depends on prior decisions, conventions, known
2820
+ incidents, accepted exceptions or unresolved ambiguity. Use specific subsystem,
2821
+ operation or file keywords. Refine broad result sets instead of scrolling an entire
2822
+ memory corpus. Fetch a full memory only after a relevant summary is identified.
2823
+
2824
+ Treat memories as contextual evidence, not current authority. Verify them against the
2825
+ root contract, current code, documentation and external sources where applicable.
2826
+ Report stale or contradictory memories explicitly.
2827
+
2828
+ Do not query memory for fully specified local work or as a ritual. Do not use internal
2829
+ memory as a source for current external API, release or standards claims.
2830
+ ```
2831
+
2832
+ # Appendix D — Initial deterministic grader catalog
2833
+
2834
+ | Grader ID | Role/scope | Evidence | Result |
2835
+ | ---------------------------------- | ---------------- | ---------------------------- | -------------------- |
2836
+ | `output-schema-v1` | all | final output + merged schema | valid/invalid |
2837
+ | `role-write-boundary-v1` | read-only roles | tool events + diff | pass/fail |
2838
+ | `scope-paths-v1` | writers | root/step scope + diff | precision/violations |
2839
+ | `required-tools-v1` | configured roles | trace | pass/fail |
2840
+ | `tool-accounting-v1` | telemetry | tool spans | pass/fail |
2841
+ | `thinking-span-integrity-v1` | telemetry | thinking spans | pass/fail |
2842
+ | `chain-context-present-v1` | chain steps | startup context | pass/fail |
2843
+ | `upstream-result-consumption-v1` | gates/advisors | tools + handoff | score |
2844
+ | `handoff-completeness-v1` | all | final schema | score |
2845
+ | `review-verdict-consistency-v1` | reviewer | gates + verdict | pass/fail |
2846
+ | `test-engineer-source-boundary-v1` | test-engineer | diff | pass/fail |
2847
+ | `research-citation-shape-v1` | researcher | output | pass/fail |
2848
+ | `memory-query-necessity-v1` | selected cases | trace + case label | pass/fail |
2849
+ | `canonical-chain-sequence-v1` | chain | member timeline | pass/fail |
2850
+ | `time-to-pass-v1` | chain | statuses/verdicts | value |
2851
+
2852
+ # Appendix E — Metrics catalog after modernization
2853
+
2854
+ ## Job and model
2855
+
2856
+ ```text
2857
+ xtrm_jobs_total
2858
+ xtrm_job_duration_seconds
2859
+ xtrm_job_active_runtime_seconds
2860
+ xtrm_job_wait_seconds
2861
+ xtrm_turns_total
2862
+ xtrm_turn_duration_seconds
2863
+ xtrm_message_duration_seconds
2864
+ xtrm_context_usage_ratio
2865
+ xtrm_llm_tokens_total
2866
+ xtrm_thinking_duration_seconds
2867
+ xtrm_thinking_segments_total
2868
+ xtrm_time_to_first_action_seconds
2869
+ ```
2870
+
2871
+ ## Tools and dependencies
2872
+
2873
+ ```text
2874
+ xtrm_tool_calls_total
2875
+ xtrm_tool_errors_total
2876
+ xtrm_tool_call_duration_seconds
2877
+ xtrm_uncorrelated_tool_events_total
2878
+ xtrm_mcp_operations_total
2879
+ ```
2880
+
2881
+ ## Chain and gates
2882
+
2883
+ ```text
2884
+ xtrm_chains_total
2885
+ xtrm_chain_duration_seconds
2886
+ xtrm_gate_verdicts_total
2887
+ xtrm_chain_fix_loops_total
2888
+ xtrm_chain_time_to_pass_seconds
2889
+ xtrm_chain_tokens_to_pass_total
2890
+ xtrm_evidence_refs_total
2891
+ ```
2892
+
2893
+ ## Memory
2894
+
2895
+ ```text
2896
+ xtrm_memory_queries_total
2897
+ xtrm_memory_results_total
2898
+ xtrm_memory_recall_total
2899
+ xtrm_memory_query_duration_seconds
2900
+ ```
2901
+
2902
+ ## Evaluation
2903
+
2904
+ ```text
2905
+ xtrm_eval_runs_total
2906
+ xtrm_eval_score
2907
+ xtrm_eval_assertions_total
2908
+ xtrm_eval_regressions_total
2909
+ xtrm_eval_judge_disagreements_total
2910
+ xtrm_grader_duration_seconds
2911
+ ```
2912
+
2913
+ # Appendix F — Source inventory used for this PRD
2914
+
2915
+ ## Specialists repository
2916
+
2917
+ The current-state findings in this PRD were verified against the following repository areas during the audit:
2918
+
2919
+ ```text
2920
+ config/specialists/*.specialist.json
2921
+ config/mandatory-rules/
2922
+ config/skills/
2923
+ docs/design/roadmap/chain-templates/
2924
+ docs/archive/iron-review-hardening-qa-chain-substrate.md
2925
+ docs/design/roadmap/specialists-roadmap.md
2926
+ config/skills/using-specialists-v3/SKILL.md
2927
+ config/skills/using-kpi/SKILL.md
2928
+ src/specialist/runner.ts
2929
+ src/specialist/supervisor.ts
2930
+ src/specialist/beads.ts
2931
+ src/specialist/timeline-events.ts
2932
+ src/specialist/forensic-events.ts
2933
+ src/specialist/observability-sqlite.ts
2934
+ src/specialist/prometheus-projection.ts
2935
+ src/pi/session.ts
2936
+ src/cli/db.ts
2937
+ src/cli/metrics.ts
2938
+ ```
2939
+
2940
+ ## xtrm and xtmux repositories
2941
+
2942
+ * `xtrm-dev/xtrm/docs/_meta/2026-06-21-design-reconciliation.md`
2943
+ * `xtrm-dev/xtrm/docs/channels/channels.md`
2944
+ * `xtrm-dev/xtrm/docs/substrate/substrate_design_it.md`
2945
+ * `Jaggerxtrm/xtmux/README.md` and its event/message channel contract
2946
+
2947
+ ## Console repository
2948
+
2949
+ ```text
2950
+ README.md
2951
+ docs/architecture/console-observability-spec.md
2952
+ docs/architecture/console-architecture.md
2953
+ ```
2954
+
2955
+ ## External methodological references
2956
+
2957
+ The evaluation design is informed by current public guidance and research on agent evaluations, deterministic and model-based graders, trace versus outcome evaluation, pairwise judge bias, and software-engineering agent benchmarks. Representative sources include OpenAI Agent Evals and Graders guidance, Anthropic publications on building effective agents and agent evaluations, UK AISI Inspect AI, Lost-in-the-Middle research, SWE-bench Pro and Multi-SWE-bench.
2958
+
2959
+ These external references inform methodology only. The Specialists runtime, chain, telemetry and Console facts in this document derive from direct repository inspection.
2960
+
2961
+ # Appendix G — Copy-paste contract for the local decomposition agent
2962
+
2963
+ Use the following instruction with a planning-capable local agent. Replace repository paths only when the local checkout differs.
2964
+
2965
+ ```text
2966
+ You are the planning and decomposition agent for the Specialists modernization program.
2967
+
2968
+ Authoritative inputs, in order:
2969
+ 1. Current code and executable schemas in the connected repositories.
2970
+ 2. xtrm-dev/specialists/docs/design/roadmap/specialists-roadmap.md.
2971
+ 3. specialists_prompt_chain_evals_prd.md, especially Sections 13–18.
2972
+ 4. xtrm/docs/channels/channels.md for future channel semantics.
2973
+ 5. The newest dated xtrm _meta reconciliation; the 2026-06-21 file is historical.
2974
+
2975
+ Planning-only mandate:
2976
+ - Do not edit production code.
2977
+ - Do not begin implementation.
2978
+ - Inspect the live Beads formula catalog and current open/closed Beads before creating duplicates.
2979
+ - Reconcile existing roadmap Beads to PRD work packages rather than blindly creating a second backlog.
2980
+ - Preserve every WP-* identifier from Section 15.
2981
+ - Create or update program/workstream epics and root change Beads.
2982
+ - Add explicit dependencies and typed relationships.
2983
+ - Assign repository, priority, scrutiny, recommended_template, primary role, lane, file ownership, eval gates, non-goals, rollback and bridge-retirement trigger.
2984
+ - Use the root change-contract template in Section 14.3.
2985
+ - Do not materialize specialist step Beads during Pass 1 unless the current runtime requires a bridge tracking Bead; label such work bridge:manual-composition.
2986
+ - Flag file collisions and propose a safe multiplexing wave.
2987
+ - Map every acceptance criterion in Section 18 to one or more Beads.
2988
+ - Mark existing Beads as reused, superseded, extended or duplicate candidates; never silently discard them.
2989
+
2990
+ Required outputs:
2991
+ 1. Program hierarchy and Bead IDs.
2992
+ 2. WP-to-Bead mapping table.
2993
+ 3. Dependency DAG and critical path.
2994
+ 4. Initial parallel wave with assigned agent identities from Section 15.14.
2995
+ 5. File-ownership/collision matrix.
2996
+ 6. Existing-Bead reconciliation report.
2997
+ 7. Acceptance-criterion coverage report.
2998
+ 8. Open decisions that genuinely block dispatch.
2999
+ 9. Exact next dispatch commands or xtmux handoff pointers, but do not execute them.
3000
+
3001
+ Stop and report rather than inventing a design when canonical documents conflict materially.
3002
+ ```
3003
+
3004
+ # Appendix H — Bridge-era multiplexing dispatcher checklist
3005
+
3006
+ Before dispatch:
3007
+
3008
+ ```text
3009
+ [ ] Decomposition and reuse audit approved
3010
+ [ ] Root Bead contains WP identifier and falsifiable contract
3011
+ [ ] Recommended chain template verified against live catalog
3012
+ [ ] Dependencies satisfied or explicitly represented
3013
+ [ ] Worktree/branch created and unique
3014
+ [ ] File ownership has no collision with active lanes
3015
+ [ ] Candidate fingerprint/baseline requirement known
3016
+ [ ] Eval and validation commands named
3017
+ [ ] Bridge retirement trigger present where applicable
3018
+ ```
3019
+
3020
+ Dispatch and coordination:
3021
+
3022
+ ```bash
3023
+ # Inspect current agent/worktree state
3024
+ tmux-session-picker dashboard expanded
3025
+ tmux-session-picker worktree-collisions
3026
+ tmux-session-picker audit
3027
+
3028
+ # Produce a dry-run handoff pointer for an idle target
3029
+ tmux-session-picker handoff --target <pane> --bead <bead-id> --note 'WP-XX; read root contract; no scope expansion'
3030
+
3031
+ # Send after reviewing the dry-run
3032
+ tmux-session-picker handoff --yes --target <pane> --bead <bead-id> --note 'WP-XX; read root contract; no scope expansion'
3033
+
3034
+ # Short pointer/status messages only
3035
+ tmux-session-picker message-send --from <orchestrator> --to <agent> --bead <bead-id> --text 'dependency WP-YY merged; rebase/check before continuing'
3036
+
3037
+ # Recipient acknowledgement
3038
+ tmux-session-picker message-list --for <agent> --unacked
3039
+ tmux-session-picker message-ack <message-id> --by <agent>
3040
+ ```
3041
+
3042
+ Before integration:
3043
+
3044
+ ```text
3045
+ [ ] Structured implementation handoff present
3046
+ [ ] Changed files match declared ownership
3047
+ [ ] Deterministic validation evidence present
3048
+ [ ] Named eval suite executed or explicitly deferred by gate
3049
+ [ ] No hidden cross-WP scope expansion
3050
+ [ ] New findings have discovered-from follow-up Beads
3051
+ [ ] Compatibility and migration tests pass
3052
+ [ ] Integration reviewer verdict recorded
3053
+ [ ] Dependent lane owners notified through durable Bead note + pointer message
3054
+ ```