@jaggerxtrm/specialists 3.18.0 → 3.18.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/config/mandatory-rules/code-quality-defaults.md +58 -1
- package/config/mandatory-rules/executor-delivery.md +4 -0
- package/config/mandatory-rules/test-runner-execution-scope.md +4 -0
- package/config/skills/setup-specialists/SKILL.md +1 -1
- package/config/skills/using-script-specialists/SKILL.md +5 -5
- package/config/skills/using-specialists/SKILL.md +1135 -879
- package/config/skills/using-specialists-auto/SKILL.md +21 -21
- package/config/specialists/chain-coordinator.specialist.json +63 -0
- package/config/specialists/reviewer.specialist.json +1 -1
- package/config/specialists/seconder.specialist.json +1 -1
- package/dist/asset-contract.json +8 -13
- package/dist/index.js +1070 -81
- package/dist/lib.js +185 -9
- package/dist/types/cli/clean.d.ts.map +1 -1
- package/dist/types/cli/doctor.d.ts.map +1 -1
- package/dist/types/cli/feed.d.ts.map +1 -1
- package/dist/types/cli/format-helpers.d.ts.map +1 -1
- package/dist/types/cli/init.d.ts.map +1 -1
- package/dist/types/cli/merge.d.ts.map +1 -1
- package/dist/types/cli/ps.d.ts.map +1 -1
- package/dist/types/cli/run.d.ts +19 -0
- package/dist/types/cli/run.d.ts.map +1 -1
- package/dist/types/cli/status.d.ts.map +1 -1
- package/dist/types/cli/view.d.ts.map +1 -1
- package/dist/types/pi/session.d.ts +1 -0
- package/dist/types/pi/session.d.ts.map +1 -1
- package/dist/types/specialist/control.d.ts.map +1 -1
- package/dist/types/specialist/dead-job-audit.d.ts +20 -0
- package/dist/types/specialist/dead-job-audit.d.ts.map +1 -0
- package/dist/types/specialist/launch.d.ts +8 -0
- package/dist/types/specialist/launch.d.ts.map +1 -1
- package/dist/types/specialist/loader.d.ts +12 -0
- package/dist/types/specialist/loader.d.ts.map +1 -1
- package/dist/types/specialist/observability-db.d.ts +1 -1
- package/dist/types/specialist/observability-sqlite.d.ts +76 -0
- package/dist/types/specialist/observability-sqlite.d.ts.map +1 -1
- package/dist/types/specialist/pr-drift-refresh.d.ts +17 -0
- package/dist/types/specialist/pr-drift-refresh.d.ts.map +1 -0
- package/dist/types/specialist/process-health.d.ts +16 -1
- package/dist/types/specialist/process-health.d.ts.map +1 -1
- package/dist/types/specialist/runner.d.ts +3 -0
- package/dist/types/specialist/runner.d.ts.map +1 -1
- package/dist/types/specialist/status-load.d.ts.map +1 -1
- package/dist/types/specialist/supervisor.d.ts +10 -0
- package/dist/types/specialist/supervisor.d.ts.map +1 -1
- package/dist/types/specialist/timeline-events.d.ts +2 -0
- package/dist/types/specialist/timeline-events.d.ts.map +1 -1
- package/docs/design/roadmap/chain-templates/README.md +1 -1
- package/docs/design/roadmap/chains-prompt-evals.md +3054 -0
- package/docs/design/roadmap/specialists-roadmap.md +75 -7
- package/docs/design/xt-pi-role-pi-flag-passthrough.md +174 -0
- package/docs/skills.md +4 -13
- package/package.json +2 -2
- package/config/skills/using-specialists-v2/SKILL.md +0 -766
- package/config/skills/using-specialists-v3/SKILL.md +0 -1267
|
@@ -0,0 +1,3054 @@
|
|
|
1
|
+
# Specialists Prompt, Chain Context, Observability and Evaluation Modernization
|
|
2
|
+
|
|
3
|
+
## Product Requirements Document
|
|
4
|
+
|
|
5
|
+
**Status:** Implementation-ready input for Beads decomposition, chain planning and multiplexed execution
|
|
6
|
+
**Date:** 12 July 2026
|
|
7
|
+
**Primary repository:** `xtrm-dev/specialists`
|
|
8
|
+
**Related repositories:** `Jaggerxtrm/console` / Omniforge, `xtrm-dev/xtrm`, `Jaggerxtrm/xtmux`
|
|
9
|
+
**Audience:** Specialists runtime maintainers, agent-orchestration engineers, Console maintainers, evaluation engineers, platform/observability engineers
|
|
10
|
+
**Scope:** Prompt and rule modernization for reviewer, executor, overthinker, seconder, test-engineer and researcher; chain-member identity and context reconstruction; memory retrieval policy; forensic and metrics hardening; historical and continuous evaluation; controlled model and prompt A/B testing; Console integration.
|
|
11
|
+
|
|
12
|
+
**Canonical companions:** `docs/design/roadmap/specialists-roadmap.md` for the bridge-runtime roadmap; `xtrm/docs/channels/channels.md` for future channel semantics; a new dated xtrm reconciliation document for current cross-repository sequencing.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## Document purpose
|
|
17
|
+
|
|
18
|
+
This PRD consolidates the complete set of findings and recommendations produced during a detailed audit of the Specialists system. It begins with the original request to reduce the static context payload of six heavy specialist system prompts and expands the scope where the investigation showed that prompt text was only one layer of a larger runtime-composition problem.
|
|
19
|
+
|
|
20
|
+
The document is intentionally comprehensive. It records not only the final recommendations, but also the corrections and refinements made during the investigation. In particular, the first prompt audit correctly identified substantial duplication and role overlap, but initially did not account for every mandatory-rule layer, runtime-injected block, local override, eager skill body or generated output contract. Subsequent repository inspection established the full package-canonical prompt composition and revealed additional sources of context bloat, policy conflict and observability ambiguity. This PRD incorporates that corrected understanding.
|
|
21
|
+
|
|
22
|
+
## Document authority and execution contract
|
|
23
|
+
|
|
24
|
+
This PRD is the implementation and decomposition source of truth for the prompt, chain-context, memory, telemetry and evaluation modernization program. It is intentionally more operational than `specialists-roadmap.md`: the roadmap remains authoritative for the bridge-runtime architecture and its Opportunities, while this PRD defines the concrete work packages, evaluation gates, parallelization boundaries and promotion criteria needed to implement the overlapping modernization safely.
|
|
25
|
+
|
|
26
|
+
The following precedence applies when local agents encounter apparently conflicting prose:
|
|
27
|
+
|
|
28
|
+
1. **Current code and executable schemas** define what exists today. Agents must verify before modifying.
|
|
29
|
+
2. **`specialists-roadmap.md`** defines the canonical bridge direction, retained Opportunities, Substrate read-forward and Beads/molecule mental model.
|
|
30
|
+
3. **This PRD** defines implementation scope, sequencing, grader requirements, work-package boundaries and readiness gates for the modernization program.
|
|
31
|
+
4. **`xtrm/docs/channels/channels.md`** defines the future semantic messaging contract. No bridge implementation may invent an incompatible message protocol.
|
|
32
|
+
5. **Dated `_meta` reconciliation documents** are historical snapshots. They inform drift analysis but do not override newer code, roadmap status or this PRD.
|
|
33
|
+
6. **Archived Iron/friction documents** are design archaeology only unless the canonical roadmap explicitly retains a requirement.
|
|
34
|
+
|
|
35
|
+
This document is ready to be supplied to a local planning agent for Pass-1/Pass-2 decomposition. The planning agent must create Beads work contracts and dependency edges; it must not silently redesign the architecture, duplicate normative text into every Bead, or start implementation during decomposition. The decomposition output must preserve the work-package identifiers and dependency constraints defined in Sections 14 and 15.
|
|
36
|
+
|
|
37
|
+
The implementation program is split into three coordinated tracks:
|
|
38
|
+
|
|
39
|
+
* **Track A — Specialists bridge runtime:** the critical chain-first spine and remaining roadmap work on the live runtime.
|
|
40
|
+
* **Track B — xtrm Stage 0, Substrate and Channels:** daemon/state-store and semantic messaging work owned by xtrm, not reimplemented ad hoc in Specialists.
|
|
41
|
+
* **Track C — Prompt, context, telemetry and eval modernization:** the program defined by this PRD.
|
|
42
|
+
|
|
43
|
+
Track C starts immediately with telemetry integrity and historical baselining. Its chain-aware portions consume Track A foundations. Autonomous lateral remediation consumes Channels 0.1 in Track B when that layer exists.
|
|
44
|
+
|
|
45
|
+
The required result is not merely shorter prompts. The required result is a measurable, versioned and chain-aware agent runtime in which:
|
|
46
|
+
|
|
47
|
+
1. Each specialist has a small, stable role identity and a clear decision boundary.
|
|
48
|
+
2. Shared policies have one source of truth and are injected only when applicable.
|
|
49
|
+
3. Chain membership, upstream evidence and downstream obligations are explicit.
|
|
50
|
+
4. Historical memories are retrieved deliberately rather than pushed indiscriminately.
|
|
51
|
+
5. Runtime telemetry accurately represents thinking, tools, turns, payload, evidence and chain state.
|
|
52
|
+
6. Existing and future runs can be evaluated using deterministic, model-based and human graders.
|
|
53
|
+
7. Prompt and model changes are promoted through controlled paired experiments rather than intuition.
|
|
54
|
+
8. Console provides historical, live and experiment-specific evaluation surfaces without becoming a second telemetry producer.
|
|
55
|
+
|
|
56
|
+
# 1. Executive summary
|
|
57
|
+
|
|
58
|
+
The Specialists architecture already has strong primitives: role-specific agents, Beads contracts, worktree isolation, mandatory gates, structured output schemas, `xtrm.forensic.v1`, per-repository observability databases, Prometheus projection and a Console materializer. The principal problem is that these primitives evolved in parallel and currently overlap in ways that increase context size and weaken semantic clarity.
|
|
59
|
+
|
|
60
|
+
The six audited specialists carry large system prompts, in some cases more than 10,000 tokens once all layers are counted. The largest source is not always the role prompt itself. A non-`bare` specialist can receive all of the following:
|
|
61
|
+
|
|
62
|
+
* its `prompt.system`;
|
|
63
|
+
* a task template and Bead context;
|
|
64
|
+
* completed dependency descriptions and notes;
|
|
65
|
+
* required and default mandatory rules;
|
|
66
|
+
* specialist-specific mandatory rules and inline rules;
|
|
67
|
+
* eager skill bodies passed through Pi;
|
|
68
|
+
* a runtime Specialist Run Context block;
|
|
69
|
+
* a global output-style directive;
|
|
70
|
+
* a hardcoded GitNexus workflow mandate;
|
|
71
|
+
* static Beads workflow and close instructions;
|
|
72
|
+
* filtered memory injection;
|
|
73
|
+
* an optional GitNexus pre-query snapshot;
|
|
74
|
+
* a generated output contract based on response format, output type and output schema;
|
|
75
|
+
* reviewer-specific diff and lineage instructions;
|
|
76
|
+
* tool definitions and harness framing.
|
|
77
|
+
|
|
78
|
+
This composition creates four classes of defect.
|
|
79
|
+
|
|
80
|
+
**Redundancy:** The same policy is expressed in a system prompt, mandatory rule, task template and runner injection. Executor receives multiple GitNexus mandates, multiple scope rules and multiple handoff requirements. Reviewer repeats seconder work and restates an output format already generated by the runner. Test-engineer repeats its source boundary in inline rules, system prompt and task template.
|
|
81
|
+
|
|
82
|
+
**Contradiction:** The generated output contract, handoff-schema rule and role-specific output prose use incompatible status and verdict vocabularies. Global worktree and close rules are injected into read-only or external-research roles where they are irrelevant or conflicting. A global “stop if local evidence is missing” rule conflicts with researcher’s external-evidence mandate.
|
|
83
|
+
|
|
84
|
+
**Missing chain identity:** The architecture defines specialists as members of a chain, but the role prompts mostly treat them as isolated agents. A reviewer may need executor, seconder, test-engineer, test-runner, obligations and security evidence, yet dependency preloading is bounded and does not guarantee the full upstream chain. The agent is not deterministically told its chain, position, root contract, upstream members, completed gates or downstream handoff.
|
|
85
|
+
|
|
86
|
+
**Insufficient evaluation discipline:** Observability captures substantial raw data, but the system lacks a first-class evaluation control plane. It cannot yet reliably answer whether one prompt or model is better for a role, whether chain-awareness improved outcomes, whether memory retrieval was necessary, or whether a shorter prompt changed false-PASS rates. Some current metrics are also unsuitable for A/B decisions until corrected, notably thinking duration and tool-call counting.
|
|
87
|
+
|
|
88
|
+
The modernization is therefore organized into five tightly related programs:
|
|
89
|
+
|
|
90
|
+
1. **Prompt and policy consolidation.** Reduce the six core role prompts by approximately 45–90%, depending on role, while preserving role-specific capability. Move deterministic policies to runtime, shared behavior to mandatory rules and deep procedures to on-demand skills.
|
|
91
|
+
2. **Chain participant context.** Inject a compact, deterministic chain identity and pointer block. Teach specialists how to reconstruct only the necessary upstream context using Beads and Specialists commands. Extend handoffs with decisions, evidence, assumptions and downstream attention.
|
|
92
|
+
3. **Pull-based memory.** Remove eager semantic memory dumps. Provide a short retrieval rule and targeted `bd memories`/`bd recall` workflow, with provenance and observability.
|
|
93
|
+
4. **Telemetry hardening.** Correct tool accounting, model thinking spans, forensic-to-Prometheus wiring and activity semantics. Add metrics required for role, chain and evaluation analysis.
|
|
94
|
+
5. **Evaluation platform.** Add eval suites, cases, experiments, trials, graders, scores and pairwise comparisons. Support retrospective scoring of existing per-repo databases, automatic post-run and post-chain deterministic evaluation, controlled replay benchmarks and Console visualization.
|
|
95
|
+
|
|
96
|
+
The recommended implementation order is telemetry integrity first, then evaluation core, then prompt/chain/memory experiments. This prevents decisions from being made with metrics that are known to be ambiguous.
|
|
97
|
+
|
|
98
|
+
# 2. Background and verified current-state architecture
|
|
99
|
+
|
|
100
|
+
## 2.1 Specialist roles in scope
|
|
101
|
+
|
|
102
|
+
The first modernization wave covers six specialists:
|
|
103
|
+
|
|
104
|
+
| Role | Intended responsibility | Current strategic position |
|
|
105
|
+
| --------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------ |
|
|
106
|
+
| `executor` | Implement the Bead contract in the assigned worktree | Primary production writer |
|
|
107
|
+
| `seconder` | Cheap bounded pre-QA gate for scope compliance and implementation sanity | Immediately after writer |
|
|
108
|
+
| `test-engineer` | Author tests, fixtures, smoke/E2E assets and telemetry assertions from the actual diff | Between seconder and test-runner |
|
|
109
|
+
| `reviewer` | Final evidence-based release verdict and Release Checklist | Final gate |
|
|
110
|
+
| `researcher` | Gather current external evidence and synthesize findings | Advisor or standalone investigation |
|
|
111
|
+
| `overthinker` | Stress-test uncertain or high-impact decisions | Advisor, premortem or standalone decision review |
|
|
112
|
+
|
|
113
|
+
The role separation is fundamentally sound and should not be collapsed. The modernization should make these boundaries clearer, not replace them with a general-purpose super-agent.
|
|
114
|
+
|
|
115
|
+
## 2.2 Canonical production-diff chain
|
|
116
|
+
|
|
117
|
+
The current design canon defines the production pipeline as:
|
|
118
|
+
|
|
119
|
+
```text
|
|
120
|
+
writer (executor or debugger)
|
|
121
|
+
-> seconder
|
|
122
|
+
-> test-engineer
|
|
123
|
+
-> test-runner
|
|
124
|
+
-> security-auditor when required
|
|
125
|
+
-> obligations-scanner
|
|
126
|
+
-> reviewer
|
|
127
|
+
-> release decision
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
The formula catalog represents a chain as a Beads molecule. The molecule is the chain identity. It contains a root child carrying the overall change contract and step children carrying role-specific mandates. `needs` creates blocking relationships; labels can carry richer semantic edges such as `validates`, `informs`, `implements`, `discovered-from` and `parent-child`.
|
|
131
|
+
|
|
132
|
+
The `code-standard` formula explicitly states that reviewer inputs include the executor diff, seconder verdict, test evidence, obligations output and root contract. The `code-with-advisors` formula adds explorer, researcher and overthinker outputs before executor and expects the final reviewer to consume advisor findings as well.
|
|
133
|
+
|
|
134
|
+
## 2.3 Prompt composition at runtime
|
|
135
|
+
|
|
136
|
+
The effective specialist prompt is not equivalent to `prompt.system`. It is composed from multiple sources.
|
|
137
|
+
|
|
138
|
+
### System-side content
|
|
139
|
+
|
|
140
|
+
The Pi session receives:
|
|
141
|
+
|
|
142
|
+
1. Rendered `prompt.system`.
|
|
143
|
+
2. A Specialist Run Context block that distinguishes the specialist from a human developer and controls Bead lifecycle behavior.
|
|
144
|
+
3. A global output-style directive.
|
|
145
|
+
4. A GitNexus workflow mandate when the repository is indexed.
|
|
146
|
+
5. Static Beads workflow rules and a close checklist.
|
|
147
|
+
6. Filtered Beads memories when a Bead is present.
|
|
148
|
+
7. An optional GitNexus pre-query snapshot.
|
|
149
|
+
8. A generated output contract derived from `response_format`, `output_type` and `prompt.output_schema`.
|
|
150
|
+
9. Pi-native skill files from `prompt.skill_inherit` and `skills.paths`.
|
|
151
|
+
10. Tool and extension definitions resolved from the manifest catalog and permission tier.
|
|
152
|
+
|
|
153
|
+
### Task-side content
|
|
154
|
+
|
|
155
|
+
The task prompt receives:
|
|
156
|
+
|
|
157
|
+
1. The rendered role task template.
|
|
158
|
+
2. The root or step Bead contract.
|
|
159
|
+
3. Notes attached to the current Bead.
|
|
160
|
+
4. Completed blocking dependencies, recursively up to the configured context depth.
|
|
161
|
+
5. Runtime worktree boundary rules.
|
|
162
|
+
6. Required, default, specialist-specific and inline mandatory rules.
|
|
163
|
+
7. Reviewer-specific injected diff context where applicable.
|
|
164
|
+
8. Pre-script output and invocation variables.
|
|
165
|
+
|
|
166
|
+
This distinction matters for optimization. A small `prompt.system` does not guarantee a small first-turn input. The API-billed first-turn input must remain the source of truth, while payload component events are used to identify which layer contributes the bloat.
|
|
167
|
+
|
|
168
|
+
## 2.4 Mandatory-rule resolution
|
|
169
|
+
|
|
170
|
+
The canonical rule loader combines:
|
|
171
|
+
|
|
172
|
+
1. required template sets;
|
|
173
|
+
2. default template sets unless default globals are disabled;
|
|
174
|
+
3. specialist-declared template sets;
|
|
175
|
+
4. specialist inline rules.
|
|
176
|
+
|
|
177
|
+
The package-level index currently requires `core-session-boundary` and defaults to `git-workflow-safe`. Specialist-specific rules then add executor delivery, code-quality defaults, GitNexus and Serena instructions, handoff schema, Bead ID discipline, research routing and other role-specific behavior.
|
|
178
|
+
|
|
179
|
+
The package-canonical rules can be audited from the repository. The effective installation may still differ because user or repository overlays and global user configuration can override or append content. Therefore all implementation and experiment tooling must fingerprint and capture the **resolved** configuration, not assume package defaults.
|
|
180
|
+
|
|
181
|
+
The authoritative inspection path is conceptually:
|
|
182
|
+
|
|
183
|
+
```bash
|
|
184
|
+
sp config show <specialist> --resolved
|
|
185
|
+
sp list-rules --specialist <specialist> --json
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
## 2.5 Skills
|
|
189
|
+
|
|
190
|
+
Among the six package-canonical specialist files inspected, `test-engineer` declares `test-planning`; the other five do not declare explicit skill paths. The corresponding package-canonical skill body was not present at the expected repository path during the audit. It may resolve from an installation/user layer or be stale. This is a concrete reason that eval fingerprints must capture the activated skill path and body hash at runtime.
|
|
191
|
+
|
|
192
|
+
Skill bodies are eagerly injected by Pi and billed as input tokens, while the current payload breakdown records only skill labels or paths. Their real contribution must be derived from first-turn token use or measured directly by hashing and tokenizing the activated files.
|
|
193
|
+
|
|
194
|
+
## 2.6 Bead dependency context
|
|
195
|
+
|
|
196
|
+
`buildBeadContext` includes the current Bead title, description, parent and notes. It also includes descriptions and notes of completed blocking dependencies. Dependency traversal is recursive and bounded by a configured depth, currently defaulting to three in the runner path.
|
|
197
|
+
|
|
198
|
+
This is useful but insufficient as a chain context contract:
|
|
199
|
+
|
|
200
|
+
* it follows only completed `blocks` dependencies, not all semantic edges;
|
|
201
|
+
* a final reviewer in a long chain may not receive executor or seconder evidence within depth three;
|
|
202
|
+
* security gates inserted outside the simple blocking path may be omitted;
|
|
203
|
+
* the current parent label can describe a chain molecule as a “Parent epic”, conflating hierarchy levels;
|
|
204
|
+
* the agent is not told whether the preloaded window is complete.
|
|
205
|
+
|
|
206
|
+
## 2.7 Observability storage
|
|
207
|
+
|
|
208
|
+
Each repository has an observability database. Core tables include:
|
|
209
|
+
|
|
210
|
+
* `specialist_jobs` for durable run state and startup payload;
|
|
211
|
+
* `specialist_events` for canonical timeline events;
|
|
212
|
+
* `specialist_results` for final or latest completed output;
|
|
213
|
+
* `specialist_job_metrics` for aggregated run metrics;
|
|
214
|
+
* `specialist_forensic_events` for `xtrm.forensic.v1` envelopes;
|
|
215
|
+
* chain and epic linkage tables;
|
|
216
|
+
* node run, member, event and memory tables;
|
|
217
|
+
* memory cache and FTS tables.
|
|
218
|
+
|
|
219
|
+
`specialist_job_metrics` persists identity, timestamps, active and waiting time, turns, tools, token trajectory, context trajectory, stall gaps, run-complete data and startup payload.
|
|
220
|
+
|
|
221
|
+
Historical job directories can be imported. Aggregate metrics can be recomputed from persisted timeline events. This makes retrospective evaluation feasible for a large subset of dimensions.
|
|
222
|
+
|
|
223
|
+
## 2.8 Forensic event model
|
|
224
|
+
|
|
225
|
+
`xtrm.forensic.v1` is already suitable as the universal event envelope. It supports:
|
|
226
|
+
|
|
227
|
+
* resource identity, including participant role and model;
|
|
228
|
+
* high-cardinality correlation, including job, Bead, chain, turn, tool call, trace, commit and eval IDs;
|
|
229
|
+
* structured event body;
|
|
230
|
+
* redaction state and rules;
|
|
231
|
+
* optional trace, OTel, links and diagnostics.
|
|
232
|
+
|
|
233
|
+
The schema deliberately forbids high-cardinality and sensitive values from Prometheus labels. Detailed evidence stays in forensic storage and evidence references, while Prometheus remains a bounded aggregate projection.
|
|
234
|
+
|
|
235
|
+
## 2.9 Current Prometheus projection
|
|
236
|
+
|
|
237
|
+
The current projection emits operational metrics for job state, queue depth, processes, worktrees, terminal job count, job duration, active runtime, chain count and duration, waiting time, turns, context utilization, tool calls and token direction.
|
|
238
|
+
|
|
239
|
+
The renderer also contains support for forensic-derived metrics such as gate verdicts, evidence references, MCP operations, identity, policy and eval runs/scores. However, the normal `sp metrics --prometheus` path currently reads statuses and job metrics but does not supply forensic events to the renderer. Those metric families are therefore not reliably present in the live scrape despite being implemented in the rendering layer.
|
|
240
|
+
|
|
241
|
+
## 2.10 Console architecture
|
|
242
|
+
|
|
243
|
+
Console already has the correct ownership boundary for this project:
|
|
244
|
+
|
|
245
|
+
* Specialists owns runtime events, forensic envelopes, projections, evaluation semantics and evidence production.
|
|
246
|
+
* The materializer reads per-repo Specialists databases and projects job rows, token metrics, forensic events and evidence references into its read model.
|
|
247
|
+
* Console owns queries, dashboards, drilldowns, experiment UX and operator decisions.
|
|
248
|
+
* Prometheus is used for aggregate operational symptoms; forensic/evidence storage is used for exact run and trial detail.
|
|
249
|
+
|
|
250
|
+
The Console observability design already includes an `eval` signal kind, eval lookup queries, eval result summaries and an AgentOps Governance dashboard pack. A new standalone application is therefore not required.
|
|
251
|
+
|
|
252
|
+
# 3. Problem statement
|
|
253
|
+
|
|
254
|
+
## 3.1 Prompt payload and instruction competition
|
|
255
|
+
|
|
256
|
+
The six specialist prompts are costly not merely because they are verbose, but because they compete with other instructions. Long prompts contain universal programming opinions, CLI manuals, detailed output examples, role procedures, shared policy and personality cues. The runner and mandatory-rule layers then inject additional versions of the same concepts.
|
|
257
|
+
|
|
258
|
+
The result is an instruction graph with multiple authorities. The model must infer which version of scope discipline, output schema, Git workflow or evidence ordering is current. This creates hidden failure modes:
|
|
259
|
+
|
|
260
|
+
* a stale system-prompt section can override a newer shared rule;
|
|
261
|
+
* examples become treated as requirements;
|
|
262
|
+
* a role over-applies generic guidance that conflicts with repository conventions;
|
|
263
|
+
* a generated schema and a prose schema disagree;
|
|
264
|
+
* a global rule unintentionally constrains a specialist for which it was not designed;
|
|
265
|
+
* critical role instructions are buried beneath tool manuals and style advice.
|
|
266
|
+
|
|
267
|
+
## 3.2 Role overlap
|
|
268
|
+
|
|
269
|
+
Reviewer is nominally phase-two and should consume seconder results, but its prompt reconstructs atomic requirements and repeats broad compliance review. Executor carries a complete TypeScript and clean-code manual despite being a general implementation specialist. Overthinker repeats the four-phase rule already injected separately. Researcher embeds command manuals already represented in routing rules. Test-engineer repeats the same source-boundary rule three times.
|
|
270
|
+
|
|
271
|
+
This overlap increases both token use and behavioral ambiguity. The modernization must enforce the principle that each role owns a narrow decision and trusts upstream work conditionally rather than starting from zero.
|
|
272
|
+
|
|
273
|
+
## 3.3 Missing participant identity
|
|
274
|
+
|
|
275
|
+
Specialists are chain participants, but the current role identity is mostly local: “you are reviewer”, “you are executor”. The system does not consistently say:
|
|
276
|
+
|
|
277
|
+
* which chain the specialist belongs to;
|
|
278
|
+
* which formula/template was resolved;
|
|
279
|
+
* which root contract governs the overall work;
|
|
280
|
+
* which step contract governs the current mandate;
|
|
281
|
+
* what upstream roles have completed;
|
|
282
|
+
* what evidence each upstream role produced;
|
|
283
|
+
* which gates remain pending;
|
|
284
|
+
* which downstream role will consume this handoff;
|
|
285
|
+
* which evidence is preloaded and which is available on demand.
|
|
286
|
+
|
|
287
|
+
Without this identity, agents either duplicate upstream work or fail to reconstruct necessary context.
|
|
288
|
+
|
|
289
|
+
## 3.4 Eager memory context
|
|
290
|
+
|
|
291
|
+
The runtime currently retrieves a bounded set of memories based on Bead keywords. Although smaller than earlier full-memory injection designs, the retrieval is still eager and lexical. It can inject irrelevant or stale knowledge into fully specified tasks.
|
|
292
|
+
|
|
293
|
+
The desired model is capability-based memory: the agent knows that historical knowledge exists, recognizes when prior decisions or incidents matter, and retrieves targeted memory with provenance. This should be measured rather than assumed.
|
|
294
|
+
|
|
295
|
+
## 3.5 Telemetry ambiguity
|
|
296
|
+
|
|
297
|
+
Several current metrics cannot yet support rigorous comparisons:
|
|
298
|
+
|
|
299
|
+
* thinking events lack start/end phases and duration;
|
|
300
|
+
* thinking character counts are cumulative rather than per delta or final segment;
|
|
301
|
+
* reasoning tokens are provider-dependent and not equivalent to reasoning time;
|
|
302
|
+
* tool metrics may count lifecycle events rather than unique calls;
|
|
303
|
+
* the Pi callback path can report both tool-call construction and tool execution;
|
|
304
|
+
* a query checks `think` while the canonical event type is `thinking`;
|
|
305
|
+
* forensic-derived Prometheus metrics are not wired into the default scrape path;
|
|
306
|
+
* historical rows have heterogeneous schema completeness.
|
|
307
|
+
|
|
308
|
+
## 3.6 Evaluation gap
|
|
309
|
+
|
|
310
|
+
Existing observability can describe cost and activity, but it does not yet determine quality. Reviewer PASS rate, for example, is not equivalent to reviewer accuracy. A model may PASS more often because it misses defects. Historical cohort comparisons can be confounded by task difficulty. A shorter prompt can save tokens while increasing false PASS rates. An LLM judge can prefer verbosity or the first answer shown.
|
|
311
|
+
|
|
312
|
+
A first-class evaluation platform is required to establish repeatable evidence for prompt, rule, runtime and model changes.
|
|
313
|
+
|
|
314
|
+
# 4. Goals, non-goals and design principles
|
|
315
|
+
|
|
316
|
+
## 4.1 Goals
|
|
317
|
+
|
|
318
|
+
The project shall:
|
|
319
|
+
|
|
320
|
+
1. Reduce static and eager context for the six target roles without reducing task success or role compliance.
|
|
321
|
+
2. Establish one source of truth for every shared policy and output contract.
|
|
322
|
+
3. Make chain membership and context reconstruction explicit and deterministic.
|
|
323
|
+
4. Replace indiscriminate memory push with measured, targeted recall.
|
|
324
|
+
5. Correct telemetry semantics before using them for promotion decisions.
|
|
325
|
+
6. Support retrospective evaluation of existing runs with explicit completeness labels.
|
|
326
|
+
7. Support automatic deterministic evaluation after every run and every chain.
|
|
327
|
+
8. Support controlled paired A/B testing across models, prompts, rules and runtime versions.
|
|
328
|
+
9. Support external software-engineering benchmarks and internal SWE-bench-style fixtures.
|
|
329
|
+
10. Integrate results into Console with historical, live and experiment-specific views.
|
|
330
|
+
11. Preserve low-cardinality Prometheus discipline and forensic drilldown.
|
|
331
|
+
12. Make every experiment reproducible through candidate and environment fingerprints.
|
|
332
|
+
|
|
333
|
+
## 4.2 Non-goals
|
|
334
|
+
|
|
335
|
+
The project shall not:
|
|
336
|
+
|
|
337
|
+
* merge the six roles into a single generic agent;
|
|
338
|
+
* expose private chain-of-thought or require models to reveal it;
|
|
339
|
+
* treat model-written success claims as environment truth;
|
|
340
|
+
* make LLM graders authoritative blocking gates before calibration;
|
|
341
|
+
* add high-cardinality IDs to Prometheus labels;
|
|
342
|
+
* create a second telemetry or Console application unnecessarily;
|
|
343
|
+
* use historical cohort averages as a substitute for controlled replay;
|
|
344
|
+
* force every role to retrieve memories on every run;
|
|
345
|
+
* inject entire chain histories into every participant;
|
|
346
|
+
* replace repository conventions with a universal clean-code doctrine;
|
|
347
|
+
* couple Console materialization to source-of-truth mutation.
|
|
348
|
+
|
|
349
|
+
## 4.3 Design principles
|
|
350
|
+
|
|
351
|
+
### Intrinsic role identity in the system prompt
|
|
352
|
+
|
|
353
|
+
A role prompt should contain only the mandate, boundaries, evidence hierarchy and decision rules intrinsic to that role.
|
|
354
|
+
|
|
355
|
+
### Shared policy in one reusable layer
|
|
356
|
+
|
|
357
|
+
Cross-role policies belong in mandatory rules or runtime enforcement, not copied prose.
|
|
358
|
+
|
|
359
|
+
### Deterministic decisions in code
|
|
360
|
+
|
|
361
|
+
Scrutiny floors, required gates, path matching, worktree boundaries, output-schema validation and chain identity should be computed by the runtime.
|
|
362
|
+
|
|
363
|
+
### Deep procedure on demand
|
|
364
|
+
|
|
365
|
+
CLI manuals and lengthy playbooks belong in focused skills or tool descriptions, loaded only when needed.
|
|
366
|
+
|
|
367
|
+
### Identity pushed, content pulled
|
|
368
|
+
|
|
369
|
+
The runtime should inject compact chain identity, member indexes and evidence pointers. The specialist should retrieve substantive upstream content only when its role requires it.
|
|
370
|
+
|
|
371
|
+
### Outcome over self-report
|
|
372
|
+
|
|
373
|
+
Tests, environment state, diff and evidence artifacts outrank the model’s final statement.
|
|
374
|
+
|
|
375
|
+
### Hard gates before weighted scores
|
|
376
|
+
|
|
377
|
+
A role-boundary violation or critical false PASS cannot be compensated by token efficiency.
|
|
378
|
+
|
|
379
|
+
### Observe before enforce
|
|
380
|
+
|
|
381
|
+
New graders and policies begin in shadow mode. Enforcement requires measured precision and low false-positive rates.
|
|
382
|
+
|
|
383
|
+
### Version every evaluator
|
|
384
|
+
|
|
385
|
+
Prompt, rules, skills, runtime, grader, dataset and environment versions must be recorded.
|
|
386
|
+
|
|
387
|
+
# 5. Workstream A — Prompt and policy modernization
|
|
388
|
+
|
|
389
|
+
## 5.1 Target prompt architecture
|
|
390
|
+
|
|
391
|
+
Each target specialist should resolve into five conceptual layers:
|
|
392
|
+
|
|
393
|
+
```text
|
|
394
|
+
role core system prompt
|
|
395
|
+
+ applicable shared mandatory rules
|
|
396
|
+
+ compact runtime context
|
|
397
|
+
+ role-specific task contract
|
|
398
|
+
+ on-demand procedural skills/tools
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
The role core should generally remain within approximately 180–600 tokens. This is a design target, not a hard limit. Content is retained when it materially changes decisions.
|
|
402
|
+
|
|
403
|
+
The target reductions from the initial audit are:
|
|
404
|
+
|
|
405
|
+
| Specialist | Estimated system-prompt reduction | Target core size |
|
|
406
|
+
| ------------- | --------------------------------: | ---------------: |
|
|
407
|
+
| Reviewer | 70–80% | 350–600 tokens |
|
|
408
|
+
| Executor | 80–90% | 250–450 tokens |
|
|
409
|
+
| Overthinker | 60–75% | 180–300 tokens |
|
|
410
|
+
| Seconder | 45–60% | 220–350 tokens |
|
|
411
|
+
| Test-engineer | 55–70% | 250–400 tokens |
|
|
412
|
+
| Researcher | 70–85% | 220–350 tokens |
|
|
413
|
+
|
|
414
|
+
The real KPI is first-turn input and outcome, not the length of the JSON field.
|
|
415
|
+
|
|
416
|
+
## 5.2 Output contract consolidation
|
|
417
|
+
|
|
418
|
+
The runtime already generates a base output contract and merges output-type extensions and specialist output schemas. This must become the only normative output definition.
|
|
419
|
+
|
|
420
|
+
Required changes:
|
|
421
|
+
|
|
422
|
+
1. Normalize base statuses. Choose one canonical vocabulary, recommended:
|
|
423
|
+
|
|
424
|
+
```text
|
|
425
|
+
success | partial | failed | waiting
|
|
426
|
+
```
|
|
427
|
+
|
|
428
|
+
2. Normalize reviewer verdicts, recommended uppercase at presentation boundaries and a canonical normalized enum in storage.
|
|
429
|
+
3. Convert example-shaped output schemas into valid JSON Schema.
|
|
430
|
+
4. Remove prose instructions that restate field lists already represented in schema.
|
|
431
|
+
5. Remove or radically shrink `per-turn-handoff-schema`; it may explain semantics but must not define a competing schema.
|
|
432
|
+
6. Emit schema-validation results as deterministic evaluation events.
|
|
433
|
+
7. Version the merged schema and store its hash on each run.
|
|
434
|
+
|
|
435
|
+
## 5.3 Reviewer modernization
|
|
436
|
+
|
|
437
|
+
### Current defects
|
|
438
|
+
|
|
439
|
+
* Repeats requirement normalization and compliance work assigned to seconder.
|
|
440
|
+
* Uses an adversarial persona that may increase cosmetic findings and severity inflation.
|
|
441
|
+
* Contains a full CLI and evidence-reconstruction manual.
|
|
442
|
+
* Computes scrutiny and gate requirements in prompt text rather than consuming resolved policy.
|
|
443
|
+
* Repeats output and Release Checklist formats.
|
|
444
|
+
* Duplicates the `seconder ran` checklist item.
|
|
445
|
+
* Contains obligations and simplification procedures that should be separate gate/policy inputs.
|
|
446
|
+
|
|
447
|
+
### Target responsibility
|
|
448
|
+
|
|
449
|
+
Reviewer is the final evidence-based release gate. It should:
|
|
450
|
+
|
|
451
|
+
* consume the root contract and resolved chain state;
|
|
452
|
+
* consume final upstream gate evidence;
|
|
453
|
+
* inspect unresolved findings, critical requirement coverage and high-risk changed surfaces;
|
|
454
|
+
* detect contradictions or missing required evidence;
|
|
455
|
+
* produce PASS, PARTIAL or FAIL with evidence-linked findings;
|
|
456
|
+
* use delta mode for re-review after PARTIAL;
|
|
457
|
+
* never edit files;
|
|
458
|
+
* avoid preference-only findings.
|
|
459
|
+
|
|
460
|
+
### Runtime-provided inputs
|
|
461
|
+
|
|
462
|
+
```text
|
|
463
|
+
reviewed_job_id
|
|
464
|
+
chain_id
|
|
465
|
+
root_bead_id
|
|
466
|
+
current_step_bead_id
|
|
467
|
+
resolved_scrutiny
|
|
468
|
+
required_gates
|
|
469
|
+
gate_results
|
|
470
|
+
upstream_job_ids
|
|
471
|
+
injected diff/evidence refs
|
|
472
|
+
previous reviewer findings for re-review
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
### Procedures moved out of core prompt
|
|
476
|
+
|
|
477
|
+
* job and feed lookup command details;
|
|
478
|
+
* scrutiny path tables;
|
|
479
|
+
* obligation marker vocabulary;
|
|
480
|
+
* Release Checklist rendering;
|
|
481
|
+
* output JSON examples;
|
|
482
|
+
* tool-specific GitNexus syntax;
|
|
483
|
+
* full Ddiff procedure, except a concise semantic statement.
|
|
484
|
+
|
|
485
|
+
## 5.4 Executor modernization
|
|
486
|
+
|
|
487
|
+
### Current defects
|
|
488
|
+
|
|
489
|
+
The executor prompt contains an extensive universal style guide: naming rules, function-length limits, nesting limits, TypeScript and Zod requirements, error doctrine, concurrency doctrine, performance doctrine, security rules, testing advice and anti-pattern tables.
|
|
490
|
+
|
|
491
|
+
Several rules conflict with one another or with repository-specific practice. Examples include “extract at the second duplication” versus “wait until the third use”, “fix adjacent smells” versus strict scope, mandatory Zod for every external input, hard limits on function length and arguments, and “never return null”.
|
|
492
|
+
|
|
493
|
+
### Target responsibility
|
|
494
|
+
|
|
495
|
+
Executor should:
|
|
496
|
+
|
|
497
|
+
* resolve and obey the Bead contract;
|
|
498
|
+
* remain within the assigned worktree and scope;
|
|
499
|
+
* inspect relevant code and dependency impact;
|
|
500
|
+
* make the smallest correct change;
|
|
501
|
+
* follow local language/framework/repository conventions;
|
|
502
|
+
* reuse existing code and dependencies;
|
|
503
|
+
* avoid speculative abstraction and unrelated cleanup;
|
|
504
|
+
* perform declared validation and focused checks;
|
|
505
|
+
* return a complete evidence-based handoff.
|
|
506
|
+
|
|
507
|
+
Language-specific standards should be loaded from repository or on-demand skills, not hardcoded globally.
|
|
508
|
+
|
|
509
|
+
### Validation policy
|
|
510
|
+
|
|
511
|
+
Executor may run focused validation required by the Bead. Broad authoritative suite execution remains owned by test-runner unless explicitly required. The current blanket “do not run tests” language should be replaced with ownership-aware wording.
|
|
512
|
+
|
|
513
|
+
## 5.5 Overthinker modernization
|
|
514
|
+
|
|
515
|
+
### Current defects
|
|
516
|
+
|
|
517
|
+
* Repeats the four-phase rule already injected separately.
|
|
518
|
+
* Forces a multi-phase structure even for simple reversible decisions.
|
|
519
|
+
* Uses “multi-persona chain-of-thought” framing.
|
|
520
|
+
* Contains hardcoded command permissions that are already enforced by tool tiers.
|
|
521
|
+
|
|
522
|
+
### Target responsibility
|
|
523
|
+
|
|
524
|
+
Overthinker is a read-only decision-review specialist. It should scale depth to consequence and uncertainty:
|
|
525
|
+
|
|
526
|
+
* simple/reversible decision: direct recommendation;
|
|
527
|
+
* uncertain/costly decision: compare viable alternatives and trade-offs;
|
|
528
|
+
* high-impact/irreversible decision: premortem, disconfirming evidence and confidence.
|
|
529
|
+
|
|
530
|
+
It reports conclusions, assumptions, alternatives, risks and validation steps, not private reasoning traces.
|
|
531
|
+
|
|
532
|
+
## 5.6 Seconder modernization
|
|
533
|
+
|
|
534
|
+
### Current strengths
|
|
535
|
+
|
|
536
|
+
Seconder is already the best-bounded prompt in the group. Its split between scope verdict and quality verdict is useful. `UNCLEAR` is a valuable state when evidence is truly insufficient.
|
|
537
|
+
|
|
538
|
+
### Current defects
|
|
539
|
+
|
|
540
|
+
* smell lists repeat shared code-quality rules;
|
|
541
|
+
* output schema appears both in prose and JSON;
|
|
542
|
+
* tool-call budget belongs in runtime configuration;
|
|
543
|
+
* evidence hierarchy can be shared with other gates;
|
|
544
|
+
* eager context expansion could make it a second reviewer.
|
|
545
|
+
|
|
546
|
+
### Target responsibility
|
|
547
|
+
|
|
548
|
+
Seconder answers one question: whether the writer output is sufficiently compliant and implementation-sane to justify expensive QA. It should not become a release, security or broad architecture review.
|
|
549
|
+
|
|
550
|
+
## 5.7 Test-engineer modernization
|
|
551
|
+
|
|
552
|
+
### Current defects
|
|
553
|
+
|
|
554
|
+
* source boundary repeated in inline rules, system prompt and task template;
|
|
555
|
+
* mode inferred from free text rather than explicit dispatch metadata;
|
|
556
|
+
* output schema is example-shaped rather than formal JSON Schema;
|
|
557
|
+
* report-section requirements duplicate generated output contract;
|
|
558
|
+
* possible stale or installation-local `test-planning` skill reference.
|
|
559
|
+
|
|
560
|
+
### Target responsibility
|
|
561
|
+
|
|
562
|
+
Test-engineer receives an explicit mode:
|
|
563
|
+
|
|
564
|
+
```text
|
|
565
|
+
test_only | post_implementation
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
It authors tests and testing assets from actual behavior and diff evidence. Production edits are forbidden unless an explicitly named helper/export change is authorized. When source behavior is wrong, it returns `source_bug_suspected` with evidence.
|
|
569
|
+
|
|
570
|
+
## 5.8 Researcher modernization
|
|
571
|
+
|
|
572
|
+
### Current defects
|
|
573
|
+
|
|
574
|
+
* embeds detailed tool and CLI manuals;
|
|
575
|
+
* duplicates research routing rules;
|
|
576
|
+
* absolute “never answer from memory” language is too broad;
|
|
577
|
+
* does not sufficiently separate internal project memory from external evidence.
|
|
578
|
+
|
|
579
|
+
### Target responsibility
|
|
580
|
+
|
|
581
|
+
Researcher applies an epistemic policy:
|
|
582
|
+
|
|
583
|
+
* use current authoritative sources for unstable claims;
|
|
584
|
+
* prefer primary documentation and source repositories;
|
|
585
|
+
* distinguish facts, inference and unknowns;
|
|
586
|
+
* record dates and versions;
|
|
587
|
+
* verify consequential claims where practical;
|
|
588
|
+
* stop when success criteria are met;
|
|
589
|
+
* report tool failure rather than invent evidence.
|
|
590
|
+
|
|
591
|
+
Tool syntax stays in tool descriptions or focused skills.
|
|
592
|
+
|
|
593
|
+
## 5.9 Runner-injected content review
|
|
594
|
+
|
|
595
|
+
The prompt modernization must include runner blocks, not only specialist JSON.
|
|
596
|
+
|
|
597
|
+
### Specialist Run Context
|
|
598
|
+
|
|
599
|
+
Retain, but split into applicable fragments. Bead claim/close instructions should not be injected into every read-only or chain-step role when the orchestrator owns lifecycle.
|
|
600
|
+
|
|
601
|
+
### Output-style directive
|
|
602
|
+
|
|
603
|
+
The current global “smart caveman” rule may help agent-to-agent brevity but conflicts with evidence uncertainty and readable research/review output. Replace it with a bounded, role-aware style contract:
|
|
604
|
+
|
|
605
|
+
```text
|
|
606
|
+
Be concise and technical. Preserve uncertainty, evidence qualifiers and exact identifiers. Do not omit required rationale or schema fields.
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
Style effects should be evaluated by role.
|
|
610
|
+
|
|
611
|
+
### GitNexus mandate
|
|
612
|
+
|
|
613
|
+
Remove duplication between runner hardcoding, mandatory rules and task templates. Prefer a single runtime-generated tool policy that is aware of repository index health and role. Read-only roles should not receive “before editing” language.
|
|
614
|
+
|
|
615
|
+
### Static Beads close checklist
|
|
616
|
+
|
|
617
|
+
Do not inject `git add`, `commit`, `push` into read-only roles. Make it writer-specific and consistent with auto-commit ownership.
|
|
618
|
+
|
|
619
|
+
### Core session boundary
|
|
620
|
+
|
|
621
|
+
Split local worktree boundary from evidence-retrieval policy. Researcher must be permitted to retrieve external evidence. Missing local artifacts should not globally prohibit external or chain-context lookup.
|
|
622
|
+
|
|
623
|
+
## 5.10 Prompt fingerprinting
|
|
624
|
+
|
|
625
|
+
Every run must record:
|
|
626
|
+
|
|
627
|
+
```text
|
|
628
|
+
system_prompt_hash
|
|
629
|
+
task_template_hash
|
|
630
|
+
merged_output_schema_hash
|
|
631
|
+
mandatory_rules_hash
|
|
632
|
+
mandatory_rule_ids and versions
|
|
633
|
+
skill_paths and skill_body_hashes
|
|
634
|
+
runner_injection_version
|
|
635
|
+
tool_catalog_version
|
|
636
|
+
resolved_tool_set_hash
|
|
637
|
+
specialist_version
|
|
638
|
+
```
|
|
639
|
+
|
|
640
|
+
The system should optionally store redacted prompt component metadata and token counts, not raw sensitive prompt text in Prometheus.
|
|
641
|
+
|
|
642
|
+
# 6. Workstream B — Chain participant identity and context reconstruction
|
|
643
|
+
|
|
644
|
+
## 6.1 Product requirement
|
|
645
|
+
|
|
646
|
+
A specialist dispatched as a chain step must be told that it is a chain participant. The runtime shall provide enough identity to orient the agent without injecting the complete chain history.
|
|
647
|
+
|
|
648
|
+
## 6.2 Chain context envelope
|
|
649
|
+
|
|
650
|
+
Recommended task/system injection:
|
|
651
|
+
|
|
652
|
+
```xml
|
|
653
|
+
<chain-context version="1">
|
|
654
|
+
<chain-id>...</chain-id>
|
|
655
|
+
<chain-template>code-standard</chain-template>
|
|
656
|
+
<root-bead-id>...</root-bead-id>
|
|
657
|
+
<current-step-bead-id>...</current-step-bead-id>
|
|
658
|
+
<current-role>reviewer</current-role>
|
|
659
|
+
<current-class>gate</current-class>
|
|
660
|
+
<position>final</position>
|
|
661
|
+
<scrutiny stated="medium" effective="high" />
|
|
662
|
+
<upstream complete="false">
|
|
663
|
+
<step role="executor" bead-id="..." job-id="..." status="completed" />
|
|
664
|
+
<step role="seconder" bead-id="..." job-id="..." status="completed" />
|
|
665
|
+
</upstream>
|
|
666
|
+
<pending>
|
|
667
|
+
...
|
|
668
|
+
</pending>
|
|
669
|
+
<preloaded-context depth="3" complete="false" />
|
|
670
|
+
<worktree-path>...</worktree-path>
|
|
671
|
+
</chain-context>
|
|
672
|
+
```
|
|
673
|
+
|
|
674
|
+
The envelope must be generated from persisted chain shape and Beads relationships, not inferred by the model.
|
|
675
|
+
|
|
676
|
+
## 6.3 Shared chain-participant rule
|
|
677
|
+
|
|
678
|
+
A shared mandatory rule should teach the operational semantics:
|
|
679
|
+
|
|
680
|
+
* the root change contract governs overall outcome;
|
|
681
|
+
* the current step contract governs mandate and boundaries;
|
|
682
|
+
* upstream handoffs are evidence, not overriding authority;
|
|
683
|
+
* consume preloaded dependency context first;
|
|
684
|
+
* reconstruct only missing necessary context;
|
|
685
|
+
* prefer `sp result` for final handoff and `sp feed --json` only for claims that require trace verification;
|
|
686
|
+
* use `bd show`, `bd dep list` and `bd dep tree` to understand issue relationships;
|
|
687
|
+
* do not repeat upstream work unless the current role validates it, evidence is missing or artifacts contradict it;
|
|
688
|
+
* report contradictions explicitly;
|
|
689
|
+
* produce a downstream-usable handoff.
|
|
690
|
+
|
|
691
|
+
The rule should apply only when a real chain context exists or the Bead is marked `kind:step`.
|
|
692
|
+
|
|
693
|
+
## 6.4 Context retrieval hierarchy
|
|
694
|
+
|
|
695
|
+
Recommended hierarchy:
|
|
696
|
+
|
|
697
|
+
1. Injected chain context and authoritative runtime metadata.
|
|
698
|
+
2. Current step Bead and root change contract.
|
|
699
|
+
3. Preloaded completed dependency handoffs.
|
|
700
|
+
4. `sp result` for specific upstream jobs.
|
|
701
|
+
5. Current diff, repository state and evidence artifacts.
|
|
702
|
+
6. `sp feed --json` for verifying claimed tool use, exact chronology or contradictory evidence.
|
|
703
|
+
7. Semantic Beads edges for related, discovered-from or validates context.
|
|
704
|
+
8. Historical memories only if prior decisions/incidents are materially relevant.
|
|
705
|
+
|
|
706
|
+
## 6.5 Handoff schema extension
|
|
707
|
+
|
|
708
|
+
The common output schema should support optional chain collaboration fields:
|
|
709
|
+
|
|
710
|
+
```json
|
|
711
|
+
{
|
|
712
|
+
"inputs_consumed": [
|
|
713
|
+
{"role": "seconder", "bead_id": "...", "job_id": "..."}
|
|
714
|
+
],
|
|
715
|
+
"decisions": [
|
|
716
|
+
{
|
|
717
|
+
"decision": "...",
|
|
718
|
+
"rationale": "...",
|
|
719
|
+
"evidence": ["..."]
|
|
720
|
+
}
|
|
721
|
+
],
|
|
722
|
+
"assumptions": [],
|
|
723
|
+
"unresolved_questions": [],
|
|
724
|
+
"downstream_attention": []
|
|
725
|
+
}
|
|
726
|
+
```
|
|
727
|
+
|
|
728
|
+
These fields capture decision records, not hidden chain-of-thought.
|
|
729
|
+
|
|
730
|
+
## 6.6 Role-specific chain behavior
|
|
731
|
+
|
|
732
|
+
### Executor
|
|
733
|
+
|
|
734
|
+
Consumes advisor findings and root contract. It should cite material advisor decisions it accepted or rejected.
|
|
735
|
+
|
|
736
|
+
### Seconder
|
|
737
|
+
|
|
738
|
+
Consumes writer diff and root contract. It should remain bounded and avoid loading the full chain unless a required input is absent.
|
|
739
|
+
|
|
740
|
+
### Test-engineer
|
|
741
|
+
|
|
742
|
+
Consumes writer diff, root validation contract, seconder verdict and prior test-runner output on retries.
|
|
743
|
+
|
|
744
|
+
### Reviewer
|
|
745
|
+
|
|
746
|
+
Consumes final evidence from all required gates. It does not independently recreate every gate unless evidence is missing or contradictory.
|
|
747
|
+
|
|
748
|
+
### Researcher and overthinker
|
|
749
|
+
|
|
750
|
+
When used as advisors, they must identify the downstream executor as consumer and produce action-oriented evidence rather than standalone essays.
|
|
751
|
+
|
|
752
|
+
## 6.7 Chain-context evaluation
|
|
753
|
+
|
|
754
|
+
The `chain-participation-v1` suite shall measure:
|
|
755
|
+
|
|
756
|
+
```text
|
|
757
|
+
chain identity recognition
|
|
758
|
+
root and step contract resolution
|
|
759
|
+
upstream handoff discovery
|
|
760
|
+
selective reconstruction
|
|
761
|
+
semantic edge navigation
|
|
762
|
+
contradiction detection
|
|
763
|
+
non-duplication of upstream work
|
|
764
|
+
downstream handoff quality
|
|
765
|
+
```
|
|
766
|
+
|
|
767
|
+
Adversarial cases shall include missing handoffs, conflicting gates, stale memory, incomplete preloaded depth, unrelated semantic edges and cross-chain contamination.
|
|
768
|
+
|
|
769
|
+
# 7. Workstream C — Pull-based memory
|
|
770
|
+
|
|
771
|
+
## 7.1 Current behavior
|
|
772
|
+
|
|
773
|
+
The current runtime extracts keywords from Bead title and description, queries the memory cache and injects a bounded set of ranked memories. Ranking uses lexical BM25, recency and access frequency. The injection budget is bounded, but the model pays the cost even when historical context is unnecessary.
|
|
774
|
+
|
|
775
|
+
## 7.2 Target behavior
|
|
776
|
+
|
|
777
|
+
Memory becomes an explicit capability. The runtime injects a short policy, not memory content:
|
|
778
|
+
|
|
779
|
+
```text
|
|
780
|
+
Historical project memories are available through bd memories and bd recall.
|
|
781
|
+
Query them only when the task depends on prior decisions, conventions, incidents,
|
|
782
|
+
accepted exceptions or unresolved ambiguity. Treat memories as context, not
|
|
783
|
+
current authority; verify against the root contract, current code and documentation.
|
|
784
|
+
```
|
|
785
|
+
|
|
786
|
+
## 7.3 Retrieval triggers
|
|
787
|
+
|
|
788
|
+
A specialist should consider memory when:
|
|
789
|
+
|
|
790
|
+
* the Bead references an existing convention or prior implementation;
|
|
791
|
+
* multiple conflicting repository patterns exist;
|
|
792
|
+
* a known incident or regression history may change validation;
|
|
793
|
+
* an architecture or API choice depends on a previous decision;
|
|
794
|
+
* a reviewer needs to understand an accepted exception;
|
|
795
|
+
* an overthinker is evaluating a high-impact repeated failure mode.
|
|
796
|
+
|
|
797
|
+
A specialist should normally avoid memory when:
|
|
798
|
+
|
|
799
|
+
* the task is fully specified and local;
|
|
800
|
+
* the information is directly available in current code or documentation;
|
|
801
|
+
* the role is a cheap bounded gate and history is not required;
|
|
802
|
+
* the claim concerns current external APIs, versions or standards that require external sources.
|
|
803
|
+
|
|
804
|
+
## 7.4 Retrieval interface
|
|
805
|
+
|
|
806
|
+
Preferred capabilities:
|
|
807
|
+
|
|
808
|
+
```bash
|
|
809
|
+
bd memories search --query "..." --limit 5 --json
|
|
810
|
+
bd memories relevant --bead <id> --limit 5 --max-tokens 500 --json
|
|
811
|
+
bd recall <memory-key> --json
|
|
812
|
+
```
|
|
813
|
+
|
|
814
|
+
Memory results should expose provenance:
|
|
815
|
+
|
|
816
|
+
```text
|
|
817
|
+
memory key
|
|
818
|
+
summary/value
|
|
819
|
+
created and updated timestamps
|
|
820
|
+
source Bead/job/commit
|
|
821
|
+
confidence
|
|
822
|
+
status: current | superseded | unknown
|
|
823
|
+
```
|
|
824
|
+
|
|
825
|
+
If the current CLI cannot provide this shape, the project should add a Specialists-side wrapper over the existing FTS/ranking cache.
|
|
826
|
+
|
|
827
|
+
## 7.5 Memory telemetry
|
|
828
|
+
|
|
829
|
+
Record:
|
|
830
|
+
|
|
831
|
+
```text
|
|
832
|
+
memory query count
|
|
833
|
+
query kind and normalized keyword count
|
|
834
|
+
result count
|
|
835
|
+
tokens/bytes returned
|
|
836
|
+
recall count
|
|
837
|
+
memory identifiers consumed
|
|
838
|
+
whether memory was cited in handoff
|
|
839
|
+
query latency
|
|
840
|
+
```
|
|
841
|
+
|
|
842
|
+
Do not put raw query text or memory content in Prometheus labels.
|
|
843
|
+
|
|
844
|
+
## 7.6 Memory evaluation
|
|
845
|
+
|
|
846
|
+
Required dimensions:
|
|
847
|
+
|
|
848
|
+
* necessary-memory recall rate;
|
|
849
|
+
* unnecessary-query rate;
|
|
850
|
+
* task success with and without memory;
|
|
851
|
+
* stale-memory contradiction handling;
|
|
852
|
+
* memory token cost;
|
|
853
|
+
* retrieval latency;
|
|
854
|
+
* provenance use;
|
|
855
|
+
* model and role differences in deciding when to query.
|
|
856
|
+
|
|
857
|
+
A/B rollout should compare eager injection, pull guidance and no-memory control on curated cases.
|
|
858
|
+
|
|
859
|
+
# 8. Workstream D — Telemetry and forensic hardening
|
|
860
|
+
|
|
861
|
+
## 8.1 Requirement
|
|
862
|
+
|
|
863
|
+
No model, prompt or chain promotion decision may depend on a metric with unresolved counting semantics. A telemetry integrity suite must be introduced and pass before the eval platform is used for promotion.
|
|
864
|
+
|
|
865
|
+
## 8.2 Thinking events
|
|
866
|
+
|
|
867
|
+
### Current behavior
|
|
868
|
+
|
|
869
|
+
Pi exposes `thinking_start`, `thinking_delta` and `thinking_end`. Specialists currently emits generic `thinking` events for start and deltas, with only a character count. `thinking_end` is not persisted. The supervisor accumulates characters, so persisted values are cumulative snapshots.
|
|
870
|
+
|
|
871
|
+
### Required event model
|
|
872
|
+
|
|
873
|
+
```json
|
|
874
|
+
{
|
|
875
|
+
"type": "thinking",
|
|
876
|
+
"phase": "start|delta|end",
|
|
877
|
+
"segment_id": "...",
|
|
878
|
+
"turn_index": 2,
|
|
879
|
+
"delta_char_count": 128,
|
|
880
|
+
"total_char_count": 1840,
|
|
881
|
+
"duration_ms": 7320
|
|
882
|
+
}
|
|
883
|
+
```
|
|
884
|
+
|
|
885
|
+
The end event should carry final segment duration and total characters. Raw thinking content should not be stored by default.
|
|
886
|
+
|
|
887
|
+
### Derived measures
|
|
888
|
+
|
|
889
|
+
```text
|
|
890
|
+
thinking_stream_duration_ms
|
|
891
|
+
thinking_segments
|
|
892
|
+
thinking_chars
|
|
893
|
+
reasoning_tokens
|
|
894
|
+
thinking_to_active_runtime_ratio
|
|
895
|
+
time_to_first_thinking
|
|
896
|
+
time_to_first_tool
|
|
897
|
+
time_to_first_text
|
|
898
|
+
inter_tool_deliberation_ms
|
|
899
|
+
```
|
|
900
|
+
|
|
901
|
+
The metric must be named as a stream duration, not provider compute time.
|
|
902
|
+
|
|
903
|
+
## 8.3 Tool-call accounting
|
|
904
|
+
|
|
905
|
+
### Current risk
|
|
906
|
+
|
|
907
|
+
Timeline aggregation increments tool counts for every `tool` event, which includes start, update and end. Pi may also call tool callbacks during both LLM tool-call construction and execution.
|
|
908
|
+
|
|
909
|
+
### Required correction
|
|
910
|
+
|
|
911
|
+
* Count a tool call only on canonical `tool phase=start` execution events.
|
|
912
|
+
* Deduplicate by `tool_call_id`.
|
|
913
|
+
* Track uncorrelated calls separately.
|
|
914
|
+
* Distinguish construction events from execution events if retained.
|
|
915
|
+
* Compute duration by matching start/end on `tool_call_id`.
|
|
916
|
+
* Support parallel active tool calls without a single global `toolStartMs`.
|
|
917
|
+
|
|
918
|
+
### Metrics
|
|
919
|
+
|
|
920
|
+
```text
|
|
921
|
+
xtrm_tool_calls_total
|
|
922
|
+
xtrm_tool_errors_total
|
|
923
|
+
xtrm_tool_call_duration_seconds
|
|
924
|
+
xtrm_uncorrelated_tool_events_total
|
|
925
|
+
xtrm_parallel_tool_concurrency
|
|
926
|
+
```
|
|
927
|
+
|
|
928
|
+
## 8.4 Turn and message timing
|
|
929
|
+
|
|
930
|
+
Add stable start/end pairing for:
|
|
931
|
+
|
|
932
|
+
```text
|
|
933
|
+
turn duration
|
|
934
|
+
assistant message duration
|
|
935
|
+
tool-result message duration
|
|
936
|
+
time to first output token
|
|
937
|
+
time from last tool end to final response
|
|
938
|
+
```
|
|
939
|
+
|
|
940
|
+
## 8.5 Activity and stall semantics
|
|
941
|
+
|
|
942
|
+
Correct queries and state logic to use the canonical `thinking` event name rather than `think`. Stalls should distinguish:
|
|
943
|
+
|
|
944
|
+
* no protocol activity;
|
|
945
|
+
* active thinking stream;
|
|
946
|
+
* long-running tool;
|
|
947
|
+
* provider/network silence;
|
|
948
|
+
* waiting state.
|
|
949
|
+
|
|
950
|
+
## 8.6 Forensic-to-Prometheus wiring
|
|
951
|
+
|
|
952
|
+
The default metrics collector must read forensic events within the requested time range and pass them to the projection renderer. Required families include:
|
|
953
|
+
|
|
954
|
+
```text
|
|
955
|
+
xtrm_gate_verdicts_total
|
|
956
|
+
xtrm_evidence_refs_total
|
|
957
|
+
xtrm_mcp_operations_total
|
|
958
|
+
xtrm_identity_operations_total
|
|
959
|
+
xtrm_policy_decisions_total
|
|
960
|
+
xtrm_policy_mismatches_total
|
|
961
|
+
xtrm_eval_runs_total
|
|
962
|
+
xtrm_eval_score
|
|
963
|
+
```
|
|
964
|
+
|
|
965
|
+
The collector must use bounded event-family queries and not load unbounded event history.
|
|
966
|
+
|
|
967
|
+
## 8.7 New operational metrics
|
|
968
|
+
|
|
969
|
+
Recommended additions:
|
|
970
|
+
|
|
971
|
+
```text
|
|
972
|
+
xtrm_thinking_duration_seconds
|
|
973
|
+
xtrm_thinking_segments_total
|
|
974
|
+
xtrm_turn_duration_seconds
|
|
975
|
+
xtrm_message_duration_seconds
|
|
976
|
+
xtrm_time_to_first_action_seconds
|
|
977
|
+
xtrm_schema_validation_total
|
|
978
|
+
xtrm_handoff_validation_total
|
|
979
|
+
xtrm_chain_fix_loops_total
|
|
980
|
+
xtrm_chain_time_to_pass_seconds
|
|
981
|
+
xtrm_chain_tokens_to_pass_total
|
|
982
|
+
xtrm_memory_queries_total
|
|
983
|
+
xtrm_memory_results_total
|
|
984
|
+
xtrm_grader_duration_seconds
|
|
985
|
+
xtrm_eval_assertions_total
|
|
986
|
+
xtrm_eval_regressions_total
|
|
987
|
+
xtrm_eval_judge_disagreements_total
|
|
988
|
+
```
|
|
989
|
+
|
|
990
|
+
## 8.8 Historical completeness
|
|
991
|
+
|
|
992
|
+
Every derived historical metric shall record availability:
|
|
993
|
+
|
|
994
|
+
```text
|
|
995
|
+
complete
|
|
996
|
+
partial
|
|
997
|
+
unavailable
|
|
998
|
+
estimated
|
|
999
|
+
```
|
|
1000
|
+
|
|
1001
|
+
Missing historical fields are unknown, not zero.
|
|
1002
|
+
|
|
1003
|
+
## 8.9 Telemetry integrity eval suite
|
|
1004
|
+
|
|
1005
|
+
`telemetry-integrity-v1` shall verify:
|
|
1006
|
+
|
|
1007
|
+
* exactly one canonical run-complete per run turn;
|
|
1008
|
+
* monotonic sequence;
|
|
1009
|
+
* tool starts equal unique calls;
|
|
1010
|
+
* paired tool durations where IDs exist;
|
|
1011
|
+
* no negative durations;
|
|
1012
|
+
* waiting plus active time is consistent with elapsed time within tolerance;
|
|
1013
|
+
* token trajectory is monotonic/semantically consistent;
|
|
1014
|
+
* context ratio remains bounded;
|
|
1015
|
+
* thinking spans are paired;
|
|
1016
|
+
* forensic conversion preserves identity and timestamps;
|
|
1017
|
+
* Prometheus projection rejects forbidden labels;
|
|
1018
|
+
* forensic-derived metrics appear in live scrape when events exist.
|
|
1019
|
+
|
|
1020
|
+
# 9. Workstream E — Evaluation platform
|
|
1021
|
+
|
|
1022
|
+
## 9.1 Evaluation subject
|
|
1023
|
+
|
|
1024
|
+
A candidate is not merely a model. It is the resolved combination:
|
|
1025
|
+
|
|
1026
|
+
```text
|
|
1027
|
+
model
|
|
1028
|
+
x specialist version
|
|
1029
|
+
x system prompt
|
|
1030
|
+
x mandatory rules
|
|
1031
|
+
x skills
|
|
1032
|
+
x runner injections
|
|
1033
|
+
x tool catalog
|
|
1034
|
+
x permission tier
|
|
1035
|
+
x thinking level
|
|
1036
|
+
x chain position
|
|
1037
|
+
x task/environment
|
|
1038
|
+
```
|
|
1039
|
+
|
|
1040
|
+
Every candidate must have a deterministic fingerprint.
|
|
1041
|
+
|
|
1042
|
+
## 9.2 Evaluation levels
|
|
1043
|
+
|
|
1044
|
+
### Outcome quality
|
|
1045
|
+
|
|
1046
|
+
Did the final environment satisfy the task? Examples: hidden tests, state assertions, schema validation, smoke/E2E, telemetry assertions, diff correctness and absence of regressions.
|
|
1047
|
+
|
|
1048
|
+
### Role behavior
|
|
1049
|
+
|
|
1050
|
+
Did the specialist obey its mandate and boundaries? Examples: no reviewer edits, no unauthorized test-engineer source changes, bounded seconder behavior, researcher source discipline.
|
|
1051
|
+
|
|
1052
|
+
### Chain contribution
|
|
1053
|
+
|
|
1054
|
+
Did the specialist consume the right upstream evidence, avoid duplication, detect contradictions and produce a useful downstream handoff?
|
|
1055
|
+
|
|
1056
|
+
### Efficiency and reliability
|
|
1057
|
+
|
|
1058
|
+
What were the token, time, tool, retry, compaction, stall and variance characteristics?
|
|
1059
|
+
|
|
1060
|
+
Efficiency is reported separately and cannot compensate for failed hard gates.
|
|
1061
|
+
|
|
1062
|
+
## 9.3 Evaluation entities
|
|
1063
|
+
|
|
1064
|
+
### Eval suite
|
|
1065
|
+
|
|
1066
|
+
A versioned collection of cases for a capability or regression domain.
|
|
1067
|
+
|
|
1068
|
+
### Eval case
|
|
1069
|
+
|
|
1070
|
+
A pinned task, environment, expected outcomes and grader configuration.
|
|
1071
|
+
|
|
1072
|
+
### Eval experiment
|
|
1073
|
+
|
|
1074
|
+
A controlled comparison of one or more candidate configurations.
|
|
1075
|
+
|
|
1076
|
+
### Eval trial
|
|
1077
|
+
|
|
1078
|
+
One candidate executing one case for one attempt.
|
|
1079
|
+
|
|
1080
|
+
### Eval artifact
|
|
1081
|
+
|
|
1082
|
+
A reference to output, trace, diff, commit, tests, logs, environment state or external source bundle.
|
|
1083
|
+
|
|
1084
|
+
### Eval score
|
|
1085
|
+
|
|
1086
|
+
One grader’s result for one trial and dimension.
|
|
1087
|
+
|
|
1088
|
+
### Pairwise comparison
|
|
1089
|
+
|
|
1090
|
+
A blinded judge preference between two trial outputs/evidence bundles.
|
|
1091
|
+
|
|
1092
|
+
### Human annotation
|
|
1093
|
+
|
|
1094
|
+
A gold label, adjudication or override with provenance.
|
|
1095
|
+
|
|
1096
|
+
## 9.4 Proposed SQLite schema
|
|
1097
|
+
|
|
1098
|
+
```sql
|
|
1099
|
+
CREATE TABLE eval_suites (...);
|
|
1100
|
+
CREATE TABLE eval_cases (...);
|
|
1101
|
+
CREATE TABLE eval_experiments (...);
|
|
1102
|
+
CREATE TABLE eval_trials (...);
|
|
1103
|
+
CREATE TABLE eval_scores (...);
|
|
1104
|
+
CREATE TABLE eval_pairwise_results (...);
|
|
1105
|
+
CREATE TABLE eval_human_annotations (...);
|
|
1106
|
+
```
|
|
1107
|
+
|
|
1108
|
+
Minimum fields are defined below.
|
|
1109
|
+
|
|
1110
|
+
### `eval_suites`
|
|
1111
|
+
|
|
1112
|
+
```text
|
|
1113
|
+
suite_id
|
|
1114
|
+
name
|
|
1115
|
+
participant_role
|
|
1116
|
+
suite_kind: regression | capability | telemetry | chain | external
|
|
1117
|
+
version
|
|
1118
|
+
dataset_hash
|
|
1119
|
+
rubric_hash
|
|
1120
|
+
created_at_ms
|
|
1121
|
+
```
|
|
1122
|
+
|
|
1123
|
+
### `eval_cases`
|
|
1124
|
+
|
|
1125
|
+
```text
|
|
1126
|
+
case_id
|
|
1127
|
+
suite_id
|
|
1128
|
+
case_version
|
|
1129
|
+
title
|
|
1130
|
+
fixture_ref
|
|
1131
|
+
task_contract_json
|
|
1132
|
+
environment_spec_json
|
|
1133
|
+
expected_outcome_json
|
|
1134
|
+
grader_config_json
|
|
1135
|
+
difficulty
|
|
1136
|
+
tags_json
|
|
1137
|
+
source_kind
|
|
1138
|
+
source_ref
|
|
1139
|
+
is_held_out
|
|
1140
|
+
```
|
|
1141
|
+
|
|
1142
|
+
### `eval_experiments`
|
|
1143
|
+
|
|
1144
|
+
```text
|
|
1145
|
+
experiment_id
|
|
1146
|
+
suite_id
|
|
1147
|
+
experiment_kind
|
|
1148
|
+
candidate_configs_json
|
|
1149
|
+
randomization_seed
|
|
1150
|
+
trials_per_case
|
|
1151
|
+
status
|
|
1152
|
+
started_at_ms
|
|
1153
|
+
completed_at_ms
|
|
1154
|
+
```
|
|
1155
|
+
|
|
1156
|
+
### `eval_trials`
|
|
1157
|
+
|
|
1158
|
+
```text
|
|
1159
|
+
trial_id
|
|
1160
|
+
experiment_id
|
|
1161
|
+
case_id
|
|
1162
|
+
candidate_id
|
|
1163
|
+
job_id
|
|
1164
|
+
chain_id
|
|
1165
|
+
model
|
|
1166
|
+
prompt_hash
|
|
1167
|
+
rules_hash
|
|
1168
|
+
skills_hash
|
|
1169
|
+
runtime_version
|
|
1170
|
+
tool_catalog_version
|
|
1171
|
+
environment_hash
|
|
1172
|
+
attempt_index
|
|
1173
|
+
status
|
|
1174
|
+
data_completeness
|
|
1175
|
+
started_at_ms
|
|
1176
|
+
completed_at_ms
|
|
1177
|
+
```
|
|
1178
|
+
|
|
1179
|
+
### `eval_scores`
|
|
1180
|
+
|
|
1181
|
+
```text
|
|
1182
|
+
score_id
|
|
1183
|
+
trial_id
|
|
1184
|
+
grader_id
|
|
1185
|
+
grader_type
|
|
1186
|
+
dimension
|
|
1187
|
+
score
|
|
1188
|
+
passed
|
|
1189
|
+
confidence
|
|
1190
|
+
evidence_json
|
|
1191
|
+
grader_version
|
|
1192
|
+
created_at_ms
|
|
1193
|
+
```
|
|
1194
|
+
|
|
1195
|
+
### `eval_pairwise_results`
|
|
1196
|
+
|
|
1197
|
+
```text
|
|
1198
|
+
comparison_id
|
|
1199
|
+
experiment_id
|
|
1200
|
+
case_id
|
|
1201
|
+
trial_a_id
|
|
1202
|
+
trial_b_id
|
|
1203
|
+
judge_id
|
|
1204
|
+
presentation_order
|
|
1205
|
+
winner: A | B | tie | insufficient_evidence
|
|
1206
|
+
confidence
|
|
1207
|
+
rationale_json
|
|
1208
|
+
```
|
|
1209
|
+
|
|
1210
|
+
## 9.5 Forensic eval event catalog
|
|
1211
|
+
|
|
1212
|
+
Add events:
|
|
1213
|
+
|
|
1214
|
+
```text
|
|
1215
|
+
eval.experiment.started
|
|
1216
|
+
eval.experiment.completed
|
|
1217
|
+
eval.experiment.failed
|
|
1218
|
+
eval.trial.started
|
|
1219
|
+
eval.trial.completed
|
|
1220
|
+
eval.trial.failed
|
|
1221
|
+
eval.grader.started
|
|
1222
|
+
eval.grader.completed
|
|
1223
|
+
eval.grader.failed
|
|
1224
|
+
eval.score.recorded
|
|
1225
|
+
eval.assertion.passed
|
|
1226
|
+
eval.assertion.failed
|
|
1227
|
+
eval.comparison.recorded
|
|
1228
|
+
eval.judge.disagreement
|
|
1229
|
+
eval.human.annotation.recorded
|
|
1230
|
+
eval.human.override.recorded
|
|
1231
|
+
eval.regression.detected
|
|
1232
|
+
eval.promotion.recommended
|
|
1233
|
+
eval.promotion.blocked
|
|
1234
|
+
```
|
|
1235
|
+
|
|
1236
|
+
Use `eval_id` and existing job/chain/commit correlation fields.
|
|
1237
|
+
|
|
1238
|
+
## 9.6 Grader hierarchy
|
|
1239
|
+
|
|
1240
|
+
Use the cheapest reliable grader first:
|
|
1241
|
+
|
|
1242
|
+
```text
|
|
1243
|
+
environment/state grader
|
|
1244
|
+
-> deterministic trace grader
|
|
1245
|
+
-> deterministic artifact grader
|
|
1246
|
+
-> model grader
|
|
1247
|
+
-> human adjudication
|
|
1248
|
+
```
|
|
1249
|
+
|
|
1250
|
+
### Environment/state graders
|
|
1251
|
+
|
|
1252
|
+
Examples:
|
|
1253
|
+
|
|
1254
|
+
* hidden tests;
|
|
1255
|
+
* expected database/file state;
|
|
1256
|
+
* smoke/E2E result;
|
|
1257
|
+
* emitted telemetry;
|
|
1258
|
+
* no leaked secret;
|
|
1259
|
+
* worktree and commit state.
|
|
1260
|
+
|
|
1261
|
+
### Trace graders
|
|
1262
|
+
|
|
1263
|
+
Examples:
|
|
1264
|
+
|
|
1265
|
+
* required tool called;
|
|
1266
|
+
* forbidden tool absent;
|
|
1267
|
+
* no edit/write by read-only role;
|
|
1268
|
+
* `sp result` used before `sp feed` where required;
|
|
1269
|
+
* memory recall performed only when triggered;
|
|
1270
|
+
* chain context consumed;
|
|
1271
|
+
* output schema valid;
|
|
1272
|
+
* tool-call budget respected.
|
|
1273
|
+
|
|
1274
|
+
### Artifact graders
|
|
1275
|
+
|
|
1276
|
+
Examples:
|
|
1277
|
+
|
|
1278
|
+
* changed paths match scope;
|
|
1279
|
+
* required tests added;
|
|
1280
|
+
* diff minimality heuristics;
|
|
1281
|
+
* evidence references resolve;
|
|
1282
|
+
* handoff fields complete.
|
|
1283
|
+
|
|
1284
|
+
### Model graders
|
|
1285
|
+
|
|
1286
|
+
Use for nuanced qualities:
|
|
1287
|
+
|
|
1288
|
+
* reviewer finding materiality and evidence quality;
|
|
1289
|
+
* researcher citation support and synthesis;
|
|
1290
|
+
* overthinker risk quality and depth calibration;
|
|
1291
|
+
* executor unnecessary complexity;
|
|
1292
|
+
* handoff actionability.
|
|
1293
|
+
|
|
1294
|
+
Model graders must be calibrated against human labels and versioned.
|
|
1295
|
+
|
|
1296
|
+
## 9.7 Hard gates and score vector
|
|
1297
|
+
|
|
1298
|
+
A trial first passes hard gates:
|
|
1299
|
+
|
|
1300
|
+
```text
|
|
1301
|
+
outcome correct
|
|
1302
|
+
no forbidden action
|
|
1303
|
+
schema valid
|
|
1304
|
+
no critical requirement missed
|
|
1305
|
+
no critical security/scope violation
|
|
1306
|
+
```
|
|
1307
|
+
|
|
1308
|
+
Then a quality vector is reported:
|
|
1309
|
+
|
|
1310
|
+
```text
|
|
1311
|
+
contract compliance
|
|
1312
|
+
role adherence
|
|
1313
|
+
evidence quality
|
|
1314
|
+
chain-context use
|
|
1315
|
+
handoff quality
|
|
1316
|
+
operational reliability
|
|
1317
|
+
```
|
|
1318
|
+
|
|
1319
|
+
Cost is reported separately:
|
|
1320
|
+
|
|
1321
|
+
```text
|
|
1322
|
+
input/output/reasoning/tool tokens
|
|
1323
|
+
active/waiting/elapsed time
|
|
1324
|
+
tool calls
|
|
1325
|
+
provider cost when authoritative
|
|
1326
|
+
```
|
|
1327
|
+
|
|
1328
|
+
A composite quality score may be displayed but must not replace dimension-level promotion rules.
|
|
1329
|
+
|
|
1330
|
+
## 9.8 Role-specific scorecards
|
|
1331
|
+
|
|
1332
|
+
### Reviewer
|
|
1333
|
+
|
|
1334
|
+
```text
|
|
1335
|
+
verdict accuracy
|
|
1336
|
+
blocking finding precision
|
|
1337
|
+
blocking finding recall
|
|
1338
|
+
false PASS rate
|
|
1339
|
+
false block rate
|
|
1340
|
+
upstream evidence coverage
|
|
1341
|
+
contradiction handling
|
|
1342
|
+
delta-review discipline
|
|
1343
|
+
evidence traceability
|
|
1344
|
+
next-action usefulness
|
|
1345
|
+
```
|
|
1346
|
+
|
|
1347
|
+
False PASS is the primary safety metric.
|
|
1348
|
+
|
|
1349
|
+
### Executor
|
|
1350
|
+
|
|
1351
|
+
```text
|
|
1352
|
+
acceptance pass rate
|
|
1353
|
+
scope precision and recall
|
|
1354
|
+
regression rate
|
|
1355
|
+
minimality
|
|
1356
|
+
repository convention adherence
|
|
1357
|
+
validation execution
|
|
1358
|
+
handoff completeness
|
|
1359
|
+
fix-loop count
|
|
1360
|
+
```
|
|
1361
|
+
|
|
1362
|
+
### Seconder
|
|
1363
|
+
|
|
1364
|
+
```text
|
|
1365
|
+
gate accuracy
|
|
1366
|
+
finding precision and recall
|
|
1367
|
+
boundedness
|
|
1368
|
+
context expansion discipline
|
|
1369
|
+
UNCLEAR calibration
|
|
1370
|
+
agreement with final outcome
|
|
1371
|
+
```
|
|
1372
|
+
|
|
1373
|
+
### Test-engineer
|
|
1374
|
+
|
|
1375
|
+
```text
|
|
1376
|
+
fail-to-pass effectiveness
|
|
1377
|
+
pass-to-pass preservation
|
|
1378
|
+
critical-path coverage
|
|
1379
|
+
mutation sensitivity
|
|
1380
|
+
command executability
|
|
1381
|
+
source-boundary violations
|
|
1382
|
+
failure-owner accuracy
|
|
1383
|
+
telemetry assertion quality
|
|
1384
|
+
harness reuse
|
|
1385
|
+
```
|
|
1386
|
+
|
|
1387
|
+
### Researcher
|
|
1388
|
+
|
|
1389
|
+
```text
|
|
1390
|
+
claim correctness
|
|
1391
|
+
citation precision and coverage
|
|
1392
|
+
source authority
|
|
1393
|
+
freshness
|
|
1394
|
+
fact/inference separation
|
|
1395
|
+
contradiction handling
|
|
1396
|
+
query efficiency
|
|
1397
|
+
```
|
|
1398
|
+
|
|
1399
|
+
### Overthinker
|
|
1400
|
+
|
|
1401
|
+
```text
|
|
1402
|
+
decision quality
|
|
1403
|
+
assumption discovery
|
|
1404
|
+
risk precision and recall
|
|
1405
|
+
alternative quality
|
|
1406
|
+
disconfirming evidence
|
|
1407
|
+
actionability
|
|
1408
|
+
depth calibration
|
|
1409
|
+
redundancy
|
|
1410
|
+
```
|
|
1411
|
+
|
|
1412
|
+
## 9.9 Regression and capability suites
|
|
1413
|
+
|
|
1414
|
+
Regression suites protect reliable behavior and should approach 100% pass rate. Capability suites are intentionally difficult and are used to improve performance.
|
|
1415
|
+
|
|
1416
|
+
Initial suites:
|
|
1417
|
+
|
|
1418
|
+
```text
|
|
1419
|
+
reviewer-regression-v1
|
|
1420
|
+
executor-regression-v1
|
|
1421
|
+
seconder-regression-v1
|
|
1422
|
+
test-engineer-regression-v1
|
|
1423
|
+
researcher-regression-v1
|
|
1424
|
+
overthinker-regression-v1
|
|
1425
|
+
chain-participation-v1
|
|
1426
|
+
telemetry-integrity-v1
|
|
1427
|
+
```
|
|
1428
|
+
|
|
1429
|
+
## 9.10 Dataset sources
|
|
1430
|
+
|
|
1431
|
+
Use real system failures first:
|
|
1432
|
+
|
|
1433
|
+
* reviewer PARTIAL followed by successful fix;
|
|
1434
|
+
* false PASS discovered later;
|
|
1435
|
+
* scope leak;
|
|
1436
|
+
* wrong diff or wrong worktree;
|
|
1437
|
+
* invalid handoff schema;
|
|
1438
|
+
* unnecessary or missing memory retrieval;
|
|
1439
|
+
* skipped mandatory gate;
|
|
1440
|
+
* contradictory gate outputs;
|
|
1441
|
+
* context overflow/compaction;
|
|
1442
|
+
* tool or extension failure;
|
|
1443
|
+
* testing ownership misclassification.
|
|
1444
|
+
|
|
1445
|
+
Add generated mutation cases:
|
|
1446
|
+
|
|
1447
|
+
* remove an upstream handoff;
|
|
1448
|
+
* make result contradict diff;
|
|
1449
|
+
* inject stale memory;
|
|
1450
|
+
* alter root scope;
|
|
1451
|
+
* add irrelevant relationship edges;
|
|
1452
|
+
* remove chain identity;
|
|
1453
|
+
* falsify a tool-use claim;
|
|
1454
|
+
* insert seeded functional or quality defects.
|
|
1455
|
+
|
|
1456
|
+
## 9.11 Retrospective evaluation
|
|
1457
|
+
|
|
1458
|
+
### Supported
|
|
1459
|
+
|
|
1460
|
+
Existing runs can be evaluated for dimensions whose evidence remains in the database:
|
|
1461
|
+
|
|
1462
|
+
* runtime and token efficiency;
|
|
1463
|
+
* tool use;
|
|
1464
|
+
* output schema;
|
|
1465
|
+
* role-boundary compliance;
|
|
1466
|
+
* handoff quality;
|
|
1467
|
+
* chain sequencing;
|
|
1468
|
+
* gate verdict distributions;
|
|
1469
|
+
* evidence references;
|
|
1470
|
+
* context and compaction behavior;
|
|
1471
|
+
* some reviewer and executor benchmark rows.
|
|
1472
|
+
|
|
1473
|
+
### Partial
|
|
1474
|
+
|
|
1475
|
+
Older runs may lack exact thinking spans, resolved prompt hashes, skill body hashes, complete forensic conversion or correlated tool IDs.
|
|
1476
|
+
|
|
1477
|
+
### Requires replay
|
|
1478
|
+
|
|
1479
|
+
Causal model comparisons and hidden outcome testing require a pinned repository snapshot and rerun. Historical cohort analysis must not be presented as a controlled A/B result.
|
|
1480
|
+
|
|
1481
|
+
### Backfill command
|
|
1482
|
+
|
|
1483
|
+
Recommended:
|
|
1484
|
+
|
|
1485
|
+
```bash
|
|
1486
|
+
sp eval backfill --repo .
|
|
1487
|
+
sp eval backfill --all-repos
|
|
1488
|
+
```
|
|
1489
|
+
|
|
1490
|
+
The command should calculate compatible deterministic graders, emit eval events and mark data completeness.
|
|
1491
|
+
|
|
1492
|
+
## 9.12 Continuous evaluation
|
|
1493
|
+
|
|
1494
|
+
### After every run
|
|
1495
|
+
|
|
1496
|
+
Run deterministic graders asynchronously after run-complete persistence:
|
|
1497
|
+
|
|
1498
|
+
```text
|
|
1499
|
+
schema validation
|
|
1500
|
+
role boundary
|
|
1501
|
+
forbidden path/tool
|
|
1502
|
+
handoff completeness
|
|
1503
|
+
startup context completeness
|
|
1504
|
+
telemetry integrity
|
|
1505
|
+
required evidence refs
|
|
1506
|
+
```
|
|
1507
|
+
|
|
1508
|
+
### After every chain
|
|
1509
|
+
|
|
1510
|
+
Run:
|
|
1511
|
+
|
|
1512
|
+
```text
|
|
1513
|
+
canonical gate sequence
|
|
1514
|
+
mandatory gate completion
|
|
1515
|
+
final verdict consistency
|
|
1516
|
+
unresolved failure state
|
|
1517
|
+
fix loops
|
|
1518
|
+
time and tokens to PASS
|
|
1519
|
+
chain outcome
|
|
1520
|
+
```
|
|
1521
|
+
|
|
1522
|
+
### Sampled/high-risk model graders
|
|
1523
|
+
|
|
1524
|
+
Run model-based quality graders on:
|
|
1525
|
+
|
|
1526
|
+
* all high/critical chains;
|
|
1527
|
+
* a configured percentage of medium chains;
|
|
1528
|
+
* detected anomalies/regressions;
|
|
1529
|
+
* explicit experiments.
|
|
1530
|
+
|
|
1531
|
+
Begin in shadow mode and do not block the main run.
|
|
1532
|
+
|
|
1533
|
+
## 9.13 Controlled A/B experiments
|
|
1534
|
+
|
|
1535
|
+
Use paired design:
|
|
1536
|
+
|
|
1537
|
+
```text
|
|
1538
|
+
same case
|
|
1539
|
+
same base SHA
|
|
1540
|
+
same task contract
|
|
1541
|
+
same upstream artifacts
|
|
1542
|
+
same tool catalog
|
|
1543
|
+
same budgets
|
|
1544
|
+
isolated worktrees
|
|
1545
|
+
```
|
|
1546
|
+
|
|
1547
|
+
Use multiple trials for stochastic agents.
|
|
1548
|
+
|
|
1549
|
+
Prompt and model changes should use factorial design where possible:
|
|
1550
|
+
|
|
1551
|
+
| | Current prompt | New prompt |
|
|
1552
|
+
| ------- | -------------: | ---------: |
|
|
1553
|
+
| Model A | A-current | A-new |
|
|
1554
|
+
| Model B | B-current | B-new |
|
|
1555
|
+
|
|
1556
|
+
This isolates model effect, prompt effect and interaction.
|
|
1557
|
+
|
|
1558
|
+
## 9.14 Pairwise judges
|
|
1559
|
+
|
|
1560
|
+
For subjective dimensions:
|
|
1561
|
+
|
|
1562
|
+
* hide model/provider/cost identity;
|
|
1563
|
+
* randomize A/B order;
|
|
1564
|
+
* repeat with swapped order;
|
|
1565
|
+
* allow tie and insufficient evidence;
|
|
1566
|
+
* use multiple judges for high-value cases;
|
|
1567
|
+
* route disagreement to human adjudication;
|
|
1568
|
+
* monitor position and verbosity bias.
|
|
1569
|
+
|
|
1570
|
+
## 9.15 Statistical reporting
|
|
1571
|
+
|
|
1572
|
+
Report:
|
|
1573
|
+
|
|
1574
|
+
```text
|
|
1575
|
+
baseline and candidate means
|
|
1576
|
+
paired delta
|
|
1577
|
+
bootstrap confidence interval
|
|
1578
|
+
win/tie/loss
|
|
1579
|
+
variance across trials
|
|
1580
|
+
failure category delta
|
|
1581
|
+
```
|
|
1582
|
+
|
|
1583
|
+
Use paired binary methods for pass/fail outcomes and paired bootstrap for continuous scores. Promotion requires practical as well as statistical significance.
|
|
1584
|
+
|
|
1585
|
+
## 9.16 Promotion policy
|
|
1586
|
+
|
|
1587
|
+
Example:
|
|
1588
|
+
|
|
1589
|
+
```yaml
|
|
1590
|
+
hard_requirements:
|
|
1591
|
+
critical_false_pass_increase: 0
|
|
1592
|
+
new_role_boundary_violations: 0
|
|
1593
|
+
regression_suite_lower_bound: 0.97
|
|
1594
|
+
output_schema_validity: 0.995
|
|
1595
|
+
quality:
|
|
1596
|
+
paired_quality_delta_lower_ci: -0.01
|
|
1597
|
+
chain_contribution_delta_lower_ci: 0
|
|
1598
|
+
pairwise_win_rate: 0.55
|
|
1599
|
+
efficiency:
|
|
1600
|
+
max_active_runtime_p50_regression: 0.20
|
|
1601
|
+
```
|
|
1602
|
+
|
|
1603
|
+
A cost increase requires a material quality improvement. A quality tie with meaningful cost reduction may justify promotion.
|
|
1604
|
+
|
|
1605
|
+
## 9.17 External and internal software-engineering benchmarks
|
|
1606
|
+
|
|
1607
|
+
External suites such as SWE-bench Pro public and Multi-SWE-bench can measure general model+harness coding capability. They should not replace internal role benchmarks.
|
|
1608
|
+
|
|
1609
|
+
The internal executor benchmark should use SWE-bench methodology:
|
|
1610
|
+
|
|
1611
|
+
```text
|
|
1612
|
+
real Bead/issue
|
|
1613
|
+
pinned repository and base SHA
|
|
1614
|
+
isolated environment
|
|
1615
|
+
hidden fail-to-pass tests
|
|
1616
|
+
pass-to-pass regression tests
|
|
1617
|
+
deterministic artifact checks
|
|
1618
|
+
full forensic trajectory
|
|
1619
|
+
```
|
|
1620
|
+
|
|
1621
|
+
Reviewer and seconder require seeded-patch benchmarks with gold defects and verdicts. Test-engineer requires buggy and fixed states to verify test sensitivity. Researcher requires source-backed claim sets. Chain eval requires mutated handoffs, missing gates and contradictions.
|
|
1622
|
+
|
|
1623
|
+
# 10. Console product requirements
|
|
1624
|
+
|
|
1625
|
+
## 10.1 Decision
|
|
1626
|
+
|
|
1627
|
+
Use the existing Console/Omniforge application. Do not create a separate eval application in the first implementation phase.
|
|
1628
|
+
|
|
1629
|
+
## 10.2 Data ownership
|
|
1630
|
+
|
|
1631
|
+
* Per-repo Specialists databases remain run-level sources of truth.
|
|
1632
|
+
* Specialists owns eval definitions, trial execution, grading events and detailed evidence.
|
|
1633
|
+
* The Console materializer projects summaries and forensic references into its read model.
|
|
1634
|
+
* Prometheus provides low-cardinality time-series aggregates.
|
|
1635
|
+
* Console provides operator UX and does not mutate source telemetry during normal reads.
|
|
1636
|
+
|
|
1637
|
+
## 10.3 Eval routes and surfaces
|
|
1638
|
+
|
|
1639
|
+
Add `/console/evals` with:
|
|
1640
|
+
|
|
1641
|
+
### Eval overview
|
|
1642
|
+
|
|
1643
|
+
Suite health, recent regressions, score by role/model, data freshness and missing signals.
|
|
1644
|
+
|
|
1645
|
+
### Experiment detail
|
|
1646
|
+
|
|
1647
|
+
Candidate fingerprints, case coverage, win/tie/loss, paired deltas, confidence intervals and promotion status.
|
|
1648
|
+
|
|
1649
|
+
### Model/prompt matrix
|
|
1650
|
+
|
|
1651
|
+
Role × model × prompt/rule/runtime version.
|
|
1652
|
+
|
|
1653
|
+
### Case explorer
|
|
1654
|
+
|
|
1655
|
+
Case source, difficulty, tags, expected outcome, fixture and grader definitions.
|
|
1656
|
+
|
|
1657
|
+
### Trial detail
|
|
1658
|
+
|
|
1659
|
+
Timeline of thinking, tools, turns and output; artifacts; diff; evidence; grader results; data completeness.
|
|
1660
|
+
|
|
1661
|
+
### Chain evaluation
|
|
1662
|
+
|
|
1663
|
+
Step timeline, queue/runtime, gates, contradictions, fix loops, time/tokens to PASS and final outcome.
|
|
1664
|
+
|
|
1665
|
+
### Historical mining
|
|
1666
|
+
|
|
1667
|
+
Search existing runs, inspect completeness and promote representative failures into eval cases.
|
|
1668
|
+
|
|
1669
|
+
### Live evaluation
|
|
1670
|
+
|
|
1671
|
+
Display post-run deterministic graders and experiment progress without blocking the originating job.
|
|
1672
|
+
|
|
1673
|
+
### Regression drift
|
|
1674
|
+
|
|
1675
|
+
Trend score, false-PASS rate, schema validity, tool/error metrics and cost over runtime/prompt/model versions.
|
|
1676
|
+
|
|
1677
|
+
## 10.4 Drilldown
|
|
1678
|
+
|
|
1679
|
+
Aggregate panels must drill down through evidence references to:
|
|
1680
|
+
|
|
1681
|
+
```text
|
|
1682
|
+
specialist job
|
|
1683
|
+
forensic event
|
|
1684
|
+
Bead/root/step
|
|
1685
|
+
chain
|
|
1686
|
+
commit/diff/PR
|
|
1687
|
+
test artifact
|
|
1688
|
+
eval trial and grader
|
|
1689
|
+
```
|
|
1690
|
+
|
|
1691
|
+
## 10.5 Realtime
|
|
1692
|
+
|
|
1693
|
+
The materializer and WebSocket hint path should publish after committed eval writes. Live views show status transitions, not raw unbounded forensic streams.
|
|
1694
|
+
|
|
1695
|
+
## 10.6 Prometheus versus forensic UI
|
|
1696
|
+
|
|
1697
|
+
Prometheus charts answer “is a class of runs getting worse?” Forensic/trial detail answers “what happened in this exact run?”. High-cardinality IDs stay in correlation and evidence refs.
|
|
1698
|
+
|
|
1699
|
+
# 11. CLI and API requirements
|
|
1700
|
+
|
|
1701
|
+
## 11.1 CLI
|
|
1702
|
+
|
|
1703
|
+
Recommended commands:
|
|
1704
|
+
|
|
1705
|
+
```bash
|
|
1706
|
+
sp eval suite list
|
|
1707
|
+
sp eval suite show <suite>
|
|
1708
|
+
sp eval case show <case>
|
|
1709
|
+
sp eval run <suite> --candidate <config> [--candidate <config>] --trials N
|
|
1710
|
+
sp eval compare <experiment-id>
|
|
1711
|
+
sp eval inspect <trial-id>
|
|
1712
|
+
sp eval failures <experiment-id>
|
|
1713
|
+
sp eval disagreements <experiment-id>
|
|
1714
|
+
sp eval backfill --repo .
|
|
1715
|
+
sp eval backfill --all-repos
|
|
1716
|
+
sp eval calibrate <rubric> --human-labels <file>
|
|
1717
|
+
sp eval promote <experiment-id> --policy <file>
|
|
1718
|
+
```
|
|
1719
|
+
|
|
1720
|
+
## 11.2 Existing DB commands
|
|
1721
|
+
|
|
1722
|
+
Retain and integrate:
|
|
1723
|
+
|
|
1724
|
+
```text
|
|
1725
|
+
sp db extract
|
|
1726
|
+
sp db stats
|
|
1727
|
+
sp db benchmark-export
|
|
1728
|
+
sp metrics --prometheus
|
|
1729
|
+
```
|
|
1730
|
+
|
|
1731
|
+
`benchmark-export` should either be versioned as a legacy exporter or rebuilt as an adapter over eval tables.
|
|
1732
|
+
|
|
1733
|
+
## 11.3 Evaluator interface
|
|
1734
|
+
|
|
1735
|
+
Conceptual TypeScript contract:
|
|
1736
|
+
|
|
1737
|
+
```ts
|
|
1738
|
+
interface EvalGrader {
|
|
1739
|
+
id: string;
|
|
1740
|
+
version: string;
|
|
1741
|
+
type: "environment" | "trace" | "artifact" | "model" | "human";
|
|
1742
|
+
supportedRoles?: string[];
|
|
1743
|
+
requiredEvidence: string[];
|
|
1744
|
+
grade(input: EvalTrialBundle): Promise<EvalScoreResult>;
|
|
1745
|
+
}
|
|
1746
|
+
```
|
|
1747
|
+
|
|
1748
|
+
## 11.4 Post-run evaluator
|
|
1749
|
+
|
|
1750
|
+
The supervisor or an event consumer emits run-complete, then enqueues evaluation. It must not recursively grade evaluator jobs. It must tolerate unavailable graders and record partial results.
|
|
1751
|
+
|
|
1752
|
+
## 11.5 Cross-repo experiments
|
|
1753
|
+
|
|
1754
|
+
Use globally unique experiment/eval IDs. Each repository retains local trial data; Console aggregates summaries by eval ID. A future state daemon may become the registry, but the initial bridge must not block on it.
|
|
1755
|
+
|
|
1756
|
+
# 12. Security, privacy and data governance
|
|
1757
|
+
|
|
1758
|
+
## 12.1 Prompt and output storage
|
|
1759
|
+
|
|
1760
|
+
Raw prompts and model outputs may contain repository-sensitive data. Store hashes and bounded metadata by default. Detailed raw artifacts remain in existing protected job/evidence storage and are not exported to Prometheus.
|
|
1761
|
+
|
|
1762
|
+
## 12.2 Forensic redaction
|
|
1763
|
+
|
|
1764
|
+
Continue using `xtrm.forensic.v1` redaction rules. Add grader output and model-judge prompts to the sensitive-field policy. Judge evidence bundles must be minimized and redacted.
|
|
1765
|
+
|
|
1766
|
+
## 12.3 Prometheus labels
|
|
1767
|
+
|
|
1768
|
+
Never label by:
|
|
1769
|
+
|
|
1770
|
+
```text
|
|
1771
|
+
job_id
|
|
1772
|
+
bead_id
|
|
1773
|
+
chain_id
|
|
1774
|
+
participant_id
|
|
1775
|
+
trace/span/tool-call/session/eval IDs
|
|
1776
|
+
raw path/command/error/diff/URL
|
|
1777
|
+
prompt or output content
|
|
1778
|
+
credentials or user identifiers
|
|
1779
|
+
```
|
|
1780
|
+
|
|
1781
|
+
## 12.4 Held-out cases
|
|
1782
|
+
|
|
1783
|
+
Held-out benchmark fixtures and hidden tests must not be exposed to the candidate agent. The runner must separate candidate worktree/tool access from grader assets.
|
|
1784
|
+
|
|
1785
|
+
## 12.5 Human annotation
|
|
1786
|
+
|
|
1787
|
+
Record annotator identity internally, but expose only the minimum needed in shared dashboards. Gold labels require provenance and versioning.
|
|
1788
|
+
|
|
1789
|
+
# 13. Integration with the Specialists roadmap, xtrm Stage 0 and Channels
|
|
1790
|
+
|
|
1791
|
+
## 13.1 Required direct documentation changes
|
|
1792
|
+
|
|
1793
|
+
Direct documentation changes are necessary, but they must preserve the distinct role of each source.
|
|
1794
|
+
|
|
1795
|
+
### `specialists-roadmap.md`
|
|
1796
|
+
|
|
1797
|
+
The canonical roadmap must be updated directly because it is the bridge-runtime source of truth and currently does not represent this modernization as an integrated, evaluation-gated program. The update should be concise and architectural rather than copying this PRD in full.
|
|
1798
|
+
|
|
1799
|
+
Required roadmap changes:
|
|
1800
|
+
|
|
1801
|
+
1. Add a prominent companion-program reference to this PRD and state that detailed work-package decomposition, graders and model/prompt promotion criteria live here.
|
|
1802
|
+
|
|
1803
|
+
2. Add **Track C — Prompt, Chain Context, Telemetry and Evaluation Modernization** to the roadmap sequencing model.
|
|
1804
|
+
|
|
1805
|
+
3. Record the ordering invariant:
|
|
1806
|
+
|
|
1807
|
+
```text
|
|
1808
|
+
telemetry integrity and historical baseline
|
|
1809
|
+
-> critical chain-first roadmap spine
|
|
1810
|
+
-> prompt/rule and memory changes under paired evaluation
|
|
1811
|
+
-> Channels replacement of bridge communication
|
|
1812
|
+
```
|
|
1813
|
+
|
|
1814
|
+
4. Mark the following as prerequisites for the complete chain-context implementation in this PRD:
|
|
1815
|
+
|
|
1816
|
+
* Opportunity 2: READ_ONLY path binding;
|
|
1817
|
+
* Opportunity 3: persisted resolved chain shape;
|
|
1818
|
+
* Opportunity 4: composition gate;
|
|
1819
|
+
* Opportunity 5: authoritative step identity;
|
|
1820
|
+
* Opportunity 10: chain-driven dispatch.
|
|
1821
|
+
|
|
1822
|
+
5. Retain Opportunity 8 only as a deliberately minimal bridge. Its payload must be compatible with future `work.turn`, `work.finding` and `work.verdict` messages, and it must not grow into a second scheduler.
|
|
1823
|
+
|
|
1824
|
+
6. Add evaluation gates to roadmap changes that alter prompts, rules, memory injection, chain context or operator workflow. “Implemented” is not sufficient; the change must pass its regression and capability suites.
|
|
1825
|
+
|
|
1826
|
+
7. Update status annotations against current code. Shipped QA pipeline, seconder fusion and forensic identity work must not be listed as unstarted. Conversely, persisted chain shape, composition gate, `--chain`, memory pull, XML contracts and Skills v4 must not be represented as complete unless verified.
|
|
1827
|
+
|
|
1828
|
+
8. Reconcile Opportunities 1, 4 and 8 against existing `bd merge-slot`, `bd gate` and `bd swarm` before creating implementation Beads. The result must be a reuse decision record, not an assumption.
|
|
1829
|
+
|
|
1830
|
+
9. Add bridge retirement criteria: every bridge must name the future owner, replacement trigger, migration/export path and deletion condition.
|
|
1831
|
+
|
|
1832
|
+
10. Preserve the roadmap as architecture-level SSOT. Do not paste the scorecards, SQL schemas or complete work-package tables from this PRD into it.
|
|
1833
|
+
|
|
1834
|
+
### `xtrm/docs/_meta/2026-06-21-design-reconciliation.md`
|
|
1835
|
+
|
|
1836
|
+
The 21 June document is explicitly a dated snapshot and should not be rewritten as though it were a live status page. Several claims are now historical, including Console state and portions of Specialists implementation status.
|
|
1837
|
+
|
|
1838
|
+
Required treatment:
|
|
1839
|
+
|
|
1840
|
+
1. Add a short banner at the top stating that it is a historical snapshot and linking to a newer reconciliation document.
|
|
1841
|
+
|
|
1842
|
+
2. Do not rewrite its original body or silently alter conclusions that were correct on 21 June.
|
|
1843
|
+
|
|
1844
|
+
3. Create a new dated document, recommended path:
|
|
1845
|
+
|
|
1846
|
+
```text
|
|
1847
|
+
xtrm/docs/_meta/2026-07-12-specialists-roadmap-evals-integration.md
|
|
1848
|
+
```
|
|
1849
|
+
|
|
1850
|
+
4. The new reconciliation must record:
|
|
1851
|
+
|
|
1852
|
+
* current verified roadmap status;
|
|
1853
|
+
* Track A/B/C ownership;
|
|
1854
|
+
* the dependency matrix in this section;
|
|
1855
|
+
* Stage 0 ownership and Channels storage decision;
|
|
1856
|
+
* the fact that Console now exists and already materializes Specialists observability;
|
|
1857
|
+
* the role of xtmux as a notification/transport adapter, not canonical Channels storage;
|
|
1858
|
+
* the current bridge-retirement plan;
|
|
1859
|
+
* links to this PRD and the canonical roadmap.
|
|
1860
|
+
|
|
1861
|
+
5. If the organization prefers append-only meta history, the same content may be an addendum in a new file; it must not replace the June snapshot.
|
|
1862
|
+
|
|
1863
|
+
## 13.2 Dependency matrix
|
|
1864
|
+
|
|
1865
|
+
| PRD capability | Start now | Requires critical Specialists roadmap spine | Requires Stage 0 | Requires Channels 0.1 | Requires Channels 0.2 |
|
|
1866
|
+
| --------------------------------------------------------- | ------------------: | ------------------------------------------: | ---------------: | --------------------: | --------------------: |
|
|
1867
|
+
| Telemetry correctness fixes | yes | no | no | no | no |
|
|
1868
|
+
| Historical run backfill and cohort analysis | yes | no | no | no | no |
|
|
1869
|
+
| Candidate/prompt/rule/skill fingerprinting | yes | no | no | no | no |
|
|
1870
|
+
| Deterministic post-run graders | yes | no | no | no | no |
|
|
1871
|
+
| Prompt composition inventory | yes | no | no | no | no |
|
|
1872
|
+
| Output-schema unification | yes | no | no | no | no |
|
|
1873
|
+
| Paired model A/B on controlled standalone cases | yes | no | no | no | no |
|
|
1874
|
+
| Minimal chain-participant guidance using current pointers | yes | partial | no | no | no |
|
|
1875
|
+
| Complete deterministic `<chain-context>` envelope | no | yes | no | no | no |
|
|
1876
|
+
| Reliable post-chain evaluator | partial | yes | no | no | no |
|
|
1877
|
+
| Pull-based memory experiment | yes | Opportunity 11 coordination | no | no | no |
|
|
1878
|
+
| Automatic reviewer-to-executor remediation | no | useful but insufficient | yes | yes | no |
|
|
1879
|
+
| Typed message/ack/delivery evaluation | no | no | yes | yes | no |
|
|
1880
|
+
| Freeform node collaboration evaluation | no | no | yes | yes | yes |
|
|
1881
|
+
| Console historical eval UI | yes after eval APIs | no | no | no | no |
|
|
1882
|
+
| Console live channel visualization | no | no | yes | yes | no |
|
|
1883
|
+
|
|
1884
|
+
## 13.3 Critical chain-first spine
|
|
1885
|
+
|
|
1886
|
+
The following is the minimum Track A foundation that must exist before this PRD can claim complete chain awareness:
|
|
1887
|
+
|
|
1888
|
+
```text
|
|
1889
|
+
bd primitive reuse decision
|
|
1890
|
+
-> READ_ONLY path binding
|
|
1891
|
+
-> persisted resolved chain shape
|
|
1892
|
+
-> composition review/approve/insert
|
|
1893
|
+
-> authoritative kind:step and semantic-edge lookup
|
|
1894
|
+
-> chain-driven dispatch and workspace resolution
|
|
1895
|
+
```
|
|
1896
|
+
|
|
1897
|
+
This spine provides stable chain identity and pointers. It does not require Channels. It enables the chain-context envelope, selective upstream retrieval, post-chain grading and reliable model comparisons within a fixed chain topology.
|
|
1898
|
+
|
|
1899
|
+
The complete roadmap is not a prerequisite for prompt modernization. `sp epic` rewrite, final branch naming, per-repo bootstrap cleanup, Skills v4 and all dispatch guardrails can follow after the critical spine. Prompt changes that do not depend on chain topology—especially researcher and overthinker—may enter experiments earlier once telemetry and eval baselines exist.
|
|
1900
|
+
|
|
1901
|
+
## 13.4 Channels boundary
|
|
1902
|
+
|
|
1903
|
+
Channels is the future semantic communication layer. This PRD must not create a parallel protocol in Specialists.
|
|
1904
|
+
|
|
1905
|
+
Until Channels 0.1 ships:
|
|
1906
|
+
|
|
1907
|
+
* `specialist_results` and Bead notes remain durable verbose handoff stores;
|
|
1908
|
+
* `sp result`, `sp feed`, Beads edges and the resolved chain shape provide pull-based context;
|
|
1909
|
+
* `step_completed` may provide a next-step recommendation and evaluation event;
|
|
1910
|
+
* the orchestrator remains responsible for dispatch/resume;
|
|
1911
|
+
* xtmux may notify operators or agents, but it is not the semantic source of truth.
|
|
1912
|
+
|
|
1913
|
+
After Channels 0.1:
|
|
1914
|
+
|
|
1915
|
+
* reviewer findings and verdicts become typed `work.finding`/`work.verdict` messages;
|
|
1916
|
+
* executor remediation is triggered through scheduler intent, not direct pane injection;
|
|
1917
|
+
* message delivery, acknowledgement, authority, stop-condition and convergence metrics join the eval framework;
|
|
1918
|
+
* bridge `step_completed` scheduling logic retires;
|
|
1919
|
+
* `sp steer` and `sp resume` can be implemented over the ChannelClient while retaining CLI compatibility.
|
|
1920
|
+
|
|
1921
|
+
The canonical Channels invariants must be preserved: participant L3 identity distinct from job L4 activation, typed message bodies, pointer-based payloads, read/ack separation, at-least-once delivery, scheduler indirection, authority derived from state rather than body text, forensic/evidence dual-write and explicit stop conditions.
|
|
1922
|
+
|
|
1923
|
+
## 13.5 xtmux integration boundary
|
|
1924
|
+
|
|
1925
|
+
xtmux already provides practical inter-agent messaging, acknowledgements, unread indicators, safe handoffs and a local event log. It should be used as an operator-facing bridge and as an experimental source of usability data.
|
|
1926
|
+
|
|
1927
|
+
Allowed integration:
|
|
1928
|
+
|
|
1929
|
+
* show a short notification when a canonical Channel message or Specialists handoff becomes available;
|
|
1930
|
+
* increment unread badges for a participant/session;
|
|
1931
|
+
* link to Bead, job, chain, eval or result identifiers;
|
|
1932
|
+
* measure acknowledgement latency and operator intervention;
|
|
1933
|
+
* provide safe pointer-based handoff during the bridge period;
|
|
1934
|
+
* expose multiplexing dashboard, collision audit and local transport health.
|
|
1935
|
+
|
|
1936
|
+
Forbidden integration:
|
|
1937
|
+
|
|
1938
|
+
* treating the rotating xtmux JSONL log as canonical chain state;
|
|
1939
|
+
* encoding release authority or gate satisfaction in pane metadata;
|
|
1940
|
+
* bypassing Channel scheduler/authority checks through direct `send-keys`;
|
|
1941
|
+
* using session names as canonical participant identity;
|
|
1942
|
+
* inventing message kinds incompatible with Channels.
|
|
1943
|
+
|
|
1944
|
+
The target relationship is:
|
|
1945
|
+
|
|
1946
|
+
```text
|
|
1947
|
+
Channels or Specialists durable state
|
|
1948
|
+
-> forensic/evidence event
|
|
1949
|
+
-> optional xtmux notification adapter
|
|
1950
|
+
-> human/agent attention
|
|
1951
|
+
```
|
|
1952
|
+
|
|
1953
|
+
# 14. Agent-ready decomposition contract
|
|
1954
|
+
|
|
1955
|
+
## 14.1 Required planning output
|
|
1956
|
+
|
|
1957
|
+
A local planning agent receiving this PRD must produce a Beads hierarchy, not implementation patches. Its output must contain:
|
|
1958
|
+
|
|
1959
|
+
1. One program epic for this PRD, or a linked set of repository-owned epics when cross-repo ownership requires separation.
|
|
1960
|
+
2. Root change Beads corresponding to the work packages in Section 15.
|
|
1961
|
+
3. Explicit dependency edges between root Beads.
|
|
1962
|
+
4. `recommended_template` for every root Bead, validated against the installed formula catalog where possible.
|
|
1963
|
+
5. Priority, scrutiny, owning repository, intended specialist role and parallel lane.
|
|
1964
|
+
6. File/surface ownership and collision risk.
|
|
1965
|
+
7. Deterministic validation and eval gates.
|
|
1966
|
+
8. Non-goals and bridge-retirement requirements.
|
|
1967
|
+
9. Integration Beads for work that crosses repository boundaries.
|
|
1968
|
+
10. A final plan-validation report proving that every PRD acceptance criterion maps to at least one Bead.
|
|
1969
|
+
|
|
1970
|
+
The planner must not create vague tasks such as “implement evals” or “improve reviewer”. Each root Bead must have a falsifiable externally observable outcome.
|
|
1971
|
+
|
|
1972
|
+
## 14.2 Hierarchy and identity
|
|
1973
|
+
|
|
1974
|
+
Use this hierarchy:
|
|
1975
|
+
|
|
1976
|
+
```text
|
|
1977
|
+
program epic
|
|
1978
|
+
-> repository/workstream epic when needed
|
|
1979
|
+
-> root change Bead
|
|
1980
|
+
-> chain molecule and step Beads at composition time
|
|
1981
|
+
```
|
|
1982
|
+
|
|
1983
|
+
A root Bead is the durable work contract. A chain molecule is an execution instance around that contract. Step Beads are role-specific participation contracts. Do not conflate the program epic, chain molecule, root issue and specialist step.
|
|
1984
|
+
|
|
1985
|
+
Before the composition gate exists, manual bridge execution may create explicit step/tracking Beads. Such Beads must be labeled `bridge:manual-composition`, linked to the root and treated as temporary orchestration scaffolding. They must not become the permanent data model.
|
|
1986
|
+
|
|
1987
|
+
## 14.3 Root change-contract template
|
|
1988
|
+
|
|
1989
|
+
Every root Bead produced from this PRD must contain equivalent fields. XML may be stored directly in the Bead description even before a deterministic parser ships because Beads stores text and the future schema is forward-compatible.
|
|
1990
|
+
|
|
1991
|
+
```xml
|
|
1992
|
+
<change-contract work-package="WP-XX" type="feature|bug|refactor|research|docs" scrutiny="low|medium|high|critical">
|
|
1993
|
+
<problem>Concrete current defect or missing capability.</problem>
|
|
1994
|
+
<scope>
|
|
1995
|
+
<repo>xtrm-dev/specialists</repo>
|
|
1996
|
+
<path>explicit/file/or/directory</path>
|
|
1997
|
+
</scope>
|
|
1998
|
+
<non-goals>
|
|
1999
|
+
<item>Explicitly excluded adjacent work.</item>
|
|
2000
|
+
</non-goals>
|
|
2001
|
+
<dependencies>
|
|
2002
|
+
<work-package>WP-YY</work-package>
|
|
2003
|
+
</dependencies>
|
|
2004
|
+
<deliverables>
|
|
2005
|
+
<item>Code, schema, docs, fixture or report.</item>
|
|
2006
|
+
</deliverables>
|
|
2007
|
+
<validation>
|
|
2008
|
+
<criterion>Exact deterministic command or state assertion.</criterion>
|
|
2009
|
+
<criterion>Named eval suite and threshold.</criterion>
|
|
2010
|
+
</validation>
|
|
2011
|
+
<acceptance>
|
|
2012
|
+
<criterion>Externally observable completion condition.</criterion>
|
|
2013
|
+
</acceptance>
|
|
2014
|
+
<rollback>How to disable or revert safely.</rollback>
|
|
2015
|
+
<retirement-trigger>Required for bridge-only work.</retirement-trigger>
|
|
2016
|
+
</change-contract>
|
|
2017
|
+
```
|
|
2018
|
+
|
|
2019
|
+
## 14.4 Step-contract template
|
|
2020
|
+
|
|
2021
|
+
When a chain is composed, each specialist step must receive:
|
|
2022
|
+
|
|
2023
|
+
```xml
|
|
2024
|
+
<step-contract role="executor|seconder|test-engineer|test-runner|reviewer|researcher|overthinker|...">
|
|
2025
|
+
<mandate>One bounded responsibility.</mandate>
|
|
2026
|
+
<inputs>
|
|
2027
|
+
<item>Root contract and named upstream evidence.</item>
|
|
2028
|
+
</inputs>
|
|
2029
|
+
<outputs>
|
|
2030
|
+
<item>Typed artifact or verdict.</item>
|
|
2031
|
+
</outputs>
|
|
2032
|
+
<scope>
|
|
2033
|
+
<path>Allowed paths or READ_ONLY evidence surface.</path>
|
|
2034
|
+
</scope>
|
|
2035
|
+
<non-goals>
|
|
2036
|
+
<item>Actions owned by other chain members.</item>
|
|
2037
|
+
</non-goals>
|
|
2038
|
+
<validation>
|
|
2039
|
+
<criterion>Evidence the next member can verify.</criterion>
|
|
2040
|
+
</validation>
|
|
2041
|
+
</step-contract>
|
|
2042
|
+
```
|
|
2043
|
+
|
|
2044
|
+
The step contract must not restate the entire PRD. It references the root Bead and only adds role-specific obligations.
|
|
2045
|
+
|
|
2046
|
+
## 14.5 Required Bead metadata
|
|
2047
|
+
|
|
2048
|
+
Each root Bead should expose, through labels, fields or a structured note:
|
|
2049
|
+
|
|
2050
|
+
```json
|
|
2051
|
+
{
|
|
2052
|
+
"work_package": "WP-T02",
|
|
2053
|
+
"program": "specialists-modernization",
|
|
2054
|
+
"repository": "xtrm-dev/specialists",
|
|
2055
|
+
"lane": "telemetry-thinking",
|
|
2056
|
+
"priority": "P1",
|
|
2057
|
+
"scrutiny": "high",
|
|
2058
|
+
"recommended_template": "code-standard",
|
|
2059
|
+
"primary_role": "executor",
|
|
2060
|
+
"integration_owner": "telemetry-integrator",
|
|
2061
|
+
"parallel_safe": true,
|
|
2062
|
+
"file_ownership": ["src/pi/session.ts", "src/specialist/timeline-events.ts"],
|
|
2063
|
+
"eval_gates": ["telemetry-integrity-v1"],
|
|
2064
|
+
"bridge": false
|
|
2065
|
+
}
|
|
2066
|
+
```
|
|
2067
|
+
|
|
2068
|
+
Where Beads does not support a native field, store the value in labels or the structured description rather than dropping it.
|
|
2069
|
+
|
|
2070
|
+
## 14.6 Decomposition quality gates
|
|
2071
|
+
|
|
2072
|
+
Before implementation starts, a planning/reviewer pass must verify:
|
|
2073
|
+
|
|
2074
|
+
* every work package is represented exactly once;
|
|
2075
|
+
* no acceptance criterion is orphaned;
|
|
2076
|
+
* dependency cycles do not exist;
|
|
2077
|
+
* no two parallel Beads claim exclusive ownership of the same high-conflict file;
|
|
2078
|
+
* all cross-repo work has an explicit integration owner;
|
|
2079
|
+
* bridge work names its retirement trigger;
|
|
2080
|
+
* prompt/model changes have baseline and promotion evals;
|
|
2081
|
+
* DB migrations have compatibility and rollback tests;
|
|
2082
|
+
* Console work consumes upstream data rather than synthesizing source truth;
|
|
2083
|
+
* Channels work is not smuggled into Specialists bridge Beads.
|
|
2084
|
+
|
|
2085
|
+
## 14.7 Implementation handoff requirements
|
|
2086
|
+
|
|
2087
|
+
Every implementation agent must leave a structured handoff with:
|
|
2088
|
+
|
|
2089
|
+
```json
|
|
2090
|
+
{
|
|
2091
|
+
"work_package": "WP-XX",
|
|
2092
|
+
"status": "success|partial|failed|waiting",
|
|
2093
|
+
"files_changed": [],
|
|
2094
|
+
"contracts_or_schemas_changed": [],
|
|
2095
|
+
"decisions": [],
|
|
2096
|
+
"assumptions": [],
|
|
2097
|
+
"validation_commands": [],
|
|
2098
|
+
"validation_results": [],
|
|
2099
|
+
"eval_results": [],
|
|
2100
|
+
"evidence_refs": [],
|
|
2101
|
+
"known_deferred_paths": [],
|
|
2102
|
+
"downstream_attention": [],
|
|
2103
|
+
"bridge_retirement_notes": []
|
|
2104
|
+
}
|
|
2105
|
+
```
|
|
2106
|
+
|
|
2107
|
+
The handoff contains decision rationale and evidence, not private chain-of-thought.
|
|
2108
|
+
|
|
2109
|
+
# 15. Work packages and multiplexed execution plan
|
|
2110
|
+
|
|
2111
|
+
## 15.1 Work-package conventions
|
|
2112
|
+
|
|
2113
|
+
The identifiers below are stable planning anchors. The local planner may create repository-specific Bead IDs, but each Bead title or metadata must retain its `WP-*` identifier so reports, evals and integration work can map back to this PRD.
|
|
2114
|
+
|
|
2115
|
+
Scrutiny defaults:
|
|
2116
|
+
|
|
2117
|
+
* **critical:** migrations, lifecycle/identity, authority, worktree safety, evaluator enforcement;
|
|
2118
|
+
* **high:** telemetry semantics, output contracts, chain state, prompt/rule changes for reviewer/executor;
|
|
2119
|
+
* **medium:** isolated CLI, fixtures, role prompts with bounded blast radius, Console read surfaces;
|
|
2120
|
+
* **low:** non-normative documentation or display-only changes.
|
|
2121
|
+
|
|
2122
|
+
## 15.2 Gate and documentation packages
|
|
2123
|
+
|
|
2124
|
+
| WP | Deliverable | Repo | Dependencies | Parallel lane | Suggested chain | Scrutiny |
|
|
2125
|
+
| -------- | ---------------------------------------------------------------------------------------------------- | ------------------ | ------------ | ----------------- | ------------------------- | -------- |
|
|
2126
|
+
| `WP-G00` | Patch canonical roadmap with Track C, dependency rules, status corrections and PRD reference | specialists | none | docs-specialists | doc-sync or code-quick | medium |
|
|
2127
|
+
| `WP-G01` | Add historical-snapshot banner to 2026-06-21 reconciliation and create new dated reconciliation | xtrm | none | docs-xtrm | doc-sync | medium |
|
|
2128
|
+
| `WP-G02` | Re-evaluate Opp 1/4/8 against `bd merge-slot`, `bd gate`, `bd swarm`; produce binding reuse decision | specialists | none | architecture-gate | research-only + premortem | high |
|
|
2129
|
+
| `WP-G03` | Register this PRD in roadmap/MOC indexes and map all acceptance criteria to WPs | specialists + xtrm | G00,G01 | docs-integration | doc-sync | medium |
|
|
2130
|
+
|
|
2131
|
+
`WP-G02` is a hard planning gate for chain-foundation implementation, not for telemetry baselining.
|
|
2132
|
+
|
|
2133
|
+
## 15.3 Telemetry-integrity packages
|
|
2134
|
+
|
|
2135
|
+
| WP | Deliverable | Primary files/surfaces | Dependencies | Parallel safety | Eval gate |
|
|
2136
|
+
| -------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------- | --------------------------------------------------------- | --------------------------- |
|
|
2137
|
+
| `WP-T01` | Correct unique tool-call accounting and lifecycle dedupe | timeline events, supervisor callbacks, job metric aggregation | none | high-conflict owner | tool lifecycle fixtures |
|
|
2138
|
+
| `WP-T02` | Add thinking start/end spans, segment identity and durations | Pi session, timeline event schema, supervisor mapping | none | separate from T01 only with explicit file split | thinking-span fixtures |
|
|
2139
|
+
| `WP-T03` | Add turn/message/tool timing and time-to-first-action metrics | event constructors and aggregation | T01,T02 interfaces stable | medium | timing reconciliation |
|
|
2140
|
+
| `WP-T04` | Correct activity/stall semantics including `think`/`thinking` | observability queries and watchdog fixtures | none | parallel-safe if query-only | activity fixtures |
|
|
2141
|
+
| `WP-T05` | Wire forensic events into live Prometheus projection | forensic reader, projection collector, metrics tests | none | isolated projection owner | scrape contract |
|
|
2142
|
+
| `WP-T06` | Persist candidate fingerprint: model, prompt, rule, skill, runtime, tool catalog hashes | loader/runner/startup payload/schema | none | high-conflict with prompt inventory; same owner preferred | fingerprint reproducibility |
|
|
2143
|
+
| `WP-T07` | Ship `telemetry-integrity-v1` suite and compatibility/version markers | tests/fixtures/docs | T01–T06 | integrator-only | full suite |
|
|
2144
|
+
|
|
2145
|
+
`WP-T01`, `WP-T02` and `WP-T06` touch central runtime surfaces and must not be assigned to independent agents without a written file-ownership partition. The safer pattern is one runtime telemetry owner plus separate test/fixture agents.
|
|
2146
|
+
|
|
2147
|
+
## 15.4 Evaluation-core packages
|
|
2148
|
+
|
|
2149
|
+
| WP | Deliverable | Dependencies | Parallel lane | Suggested role/template | Scrutiny |
|
|
2150
|
+
| -------- | -------------------------------------------------------------------- | --------------------------- | ------------------- | --------------------------- | -------- |
|
|
2151
|
+
| `WP-E01` | Eval tables, migrations, DAO and versioned types | T06 schema conventions | eval-storage | executor + seconder + tests | critical |
|
|
2152
|
+
| `WP-E02` | Eval forensic event catalog and shared writer | E01 type IDs may be stubbed | eval-events | executor | high |
|
|
2153
|
+
| `WP-E03` | Grader interface, registry and deterministic assertion result schema | none | grader-core | executor | high |
|
|
2154
|
+
| `WP-E04` | Data-completeness model and historical backfill | E01,E03,T07 | eval-backfill | executor/test-engineer | high |
|
|
2155
|
+
| `WP-E05` | Post-run evaluator in observe-only mode | E01–E03,T07 | eval-runtime | executor | high |
|
|
2156
|
+
| `WP-E06` | Post-chain evaluator over resolved shape | E01–E03,C03,C06 | eval-chain | executor | high |
|
|
2157
|
+
| `WP-E07` | Paired experiment runner, randomization and candidate manifest | E01,E03,T06 | experiment-runner | executor | high |
|
|
2158
|
+
| `WP-E08` | Statistics, pairwise judge records and promotion policy engine | E07 | experiment-analysis | executor/researcher | high |
|
|
2159
|
+
| `WP-E09` | `sp eval` CLI: suite/case/run/compare/inspect/backfill | E01,E03,E04,E07 | eval-cli | executor | medium |
|
|
2160
|
+
| `WP-E10` | Continuous eval scheduling and recursion exclusion | E05,E06 | eval-ops | executor/test-runner | critical |
|
|
2161
|
+
|
|
2162
|
+
`WP-E01` owns eval migrations. No other lane may independently edit the same migration block; dependent work must consume exported types or coordinate through the integration owner.
|
|
2163
|
+
|
|
2164
|
+
## 15.5 Prompt and policy packages
|
|
2165
|
+
|
|
2166
|
+
| WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
|
|
2167
|
+
| -------- | --------------------------------------------------------------------------------------- | ------------------------- | ---------------- | ----------- |
|
|
2168
|
+
| `WP-P01` | Resolved prompt manifest with component hashes/tokens and duplicate/conflict detection | T06 | prompt-inventory | high |
|
|
2169
|
+
| `WP-P02` | Canonical base status/verdict vocabulary and formal merged JSON Schema | E03 optional | output-contract | critical |
|
|
2170
|
+
| `WP-P03` | Replace duplicate manual handoff/output rules with generated contract as SSOT | P02 | output-contract | high |
|
|
2171
|
+
| `WP-P04` | Role-aware runner injections: style, GitNexus, Beads close, read-only/research behavior | P01,P02,T07 | runner-policy | critical |
|
|
2172
|
+
| `WP-P05` | Overthinker and researcher slim prompts plus procedures moved to skills | P01–P04,E05 | prompt-advisors | medium/high |
|
|
2173
|
+
| `WP-P06` | Seconder and test-engineer slim prompts; formal test-engineer schema and mode | P01–P04,E05 | prompt-gates | high |
|
|
2174
|
+
| `WP-P07` | Executor slim prompt and language/procedure skill separation | P01–P04,E05 | prompt-executor | high |
|
|
2175
|
+
| `WP-P08` | Reviewer slim prompt, upstream-evidence contract and false-PASS safeguards | P01–P04,E05,C06 desirable | prompt-reviewer | critical |
|
|
2176
|
+
| `WP-P09` | Factorial prompt×model experiments and staged promotion for all six roles | P05–P08,E07,E08 | prompt-eval | high |
|
|
2177
|
+
|
|
2178
|
+
Role prompt files are parallel-safe only after `WP-P02` and `WP-P04` freeze shared contracts. Each role owns its own specialist JSON and role-specific tests; one integration owner handles shared mandatory-rule/index edits.
|
|
2179
|
+
|
|
2180
|
+
## 15.6 Critical chain-foundation packages
|
|
2181
|
+
|
|
2182
|
+
| WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
|
|
2183
|
+
| -------- | ---------------------------------------------------------------------------------------- | ------------ | ----------------- | ----------- |
|
|
2184
|
+
| `WP-C01` | READ_ONLY path binding independent of owner liveness | G02 | chain-workspace | critical |
|
|
2185
|
+
| `WP-C02` | Persisted resolved chain shape with ordered steps/gates/status/job pointers | G02 | chain-state | critical |
|
|
2186
|
+
| `WP-C03` | `sp chain review/approve/insert` using resolved shape and reused bd primitives | C02,G02 | chain-composition | critical |
|
|
2187
|
+
| `WP-C04` | Authoritative `kind:step`, root/step resolution and semantic-edge query helpers | G02 | chain-graph | high |
|
|
2188
|
+
| `WP-C05` | Chain-driven dispatch/workspace resolution and deprecation bridge for job/worktree flags | C01–C04 | chain-dispatch | critical |
|
|
2189
|
+
| `WP-C06` | Deterministic `<chain-context>` envelope with completeness flags and upstream pointers | C02,C04,T06 | chain-context | high |
|
|
2190
|
+
| `WP-C07` | Shared chain-participant rule and role-specific identity clauses | C06,P04 | chain-policy | high |
|
|
2191
|
+
| `WP-C08` | Structured handoff extension for decisions, inputs consumed and downstream attention | P02,C06 | handoff-contract | high |
|
|
2192
|
+
| `WP-C09` | Minimal `step_completed` visibility/recommendation bridge compatible with Channels | C02,C03,E02 | chain-events | medium/high |
|
|
2193
|
+
| `WP-C10` | `chain-participation-v1` eval suite | C06–C09,E03 | chain-eval | high |
|
|
2194
|
+
|
|
2195
|
+
`WP-C09` must not auto-resume peers or implement a scheduler. Its retirement trigger is Channels 0.1 acceptance.
|
|
2196
|
+
|
|
2197
|
+
## 15.7 Pull-based memory packages
|
|
2198
|
+
|
|
2199
|
+
| WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
|
|
2200
|
+
| -------- | --------------------------------------------------------------------------- | ------------ | ----------------- | -------- |
|
|
2201
|
+
| `WP-M01` | Audit actual `bd memories`/`bd recall` behavior, provenance and filtering | none | memory-research | medium |
|
|
2202
|
+
| `WP-M02` | Targeted memory search/relevant interface or wrapper with bounded output | M01 | memory-cli | high |
|
|
2203
|
+
| `WP-M03` | Disable eager injection behind experiment flag and preserve rollback | M01,T06 | memory-runtime | high |
|
|
2204
|
+
| `WP-M04` | Memory-recall mandatory rule, role opt-outs and authority/provenance policy | M01,P04 | memory-policy | high |
|
|
2205
|
+
| `WP-M05` | Memory query/recall telemetry and deterministic graders | M02–M04,E03 | memory-eval | high |
|
|
2206
|
+
| `WP-M06` | Role-by-role A/B rollout and promotion | M05,E07,E08 | memory-experiment | high |
|
|
2207
|
+
|
|
2208
|
+
## 15.8 Role-suite packages
|
|
2209
|
+
|
|
2210
|
+
These packages are fixture/dataset heavy and can run in parallel after the grader interface stabilizes.
|
|
2211
|
+
|
|
2212
|
+
| WP | Suite | Core outcome |
|
|
2213
|
+
| -------- | ----------------------------- | -------------------------------------------------------------------------- |
|
|
2214
|
+
| `WP-S01` | `reviewer-regression-v1` | verdict accuracy, false PASS, finding precision/recall |
|
|
2215
|
+
| `WP-S02` | `executor-swe-style-v1` | hidden tests, scope precision, regression and minimality |
|
|
2216
|
+
| `WP-S03` | `seconder-regression-v1` | bounded gate accuracy and calibration |
|
|
2217
|
+
| `WP-S04` | `test-engineer-regression-v1` | fail-to-pass, pass-to-pass and source-boundary discipline |
|
|
2218
|
+
| `WP-S05` | `researcher-regression-v1` | claim correctness, citation entailment, freshness and inference separation |
|
|
2219
|
+
| `WP-S06` | `overthinker-capability-v1` | risk/assumption recall, precision and depth calibration |
|
|
2220
|
+
| `WP-S07` | `chain-participation-v1` | chain identity, selective retrieval, contradiction handling and handoff |
|
|
2221
|
+
| `WP-S08` | `telemetry-integrity-v1` | metric/event correctness and reconciliation |
|
|
2222
|
+
|
|
2223
|
+
All suite cases require versioned provenance. Capability cases that become reliably solved should be promoted into regression suites.
|
|
2224
|
+
|
|
2225
|
+
## 15.9 Console packages
|
|
2226
|
+
|
|
2227
|
+
| WP | Deliverable | Repo | Dependencies | Scrutiny |
|
|
2228
|
+
| -------- | ---------------------------------------------------------- | ------- | ------------ | ----------- |
|
|
2229
|
+
| `WP-U01` | Materialize eval summaries, completeness and evidence refs | console | E01,E02 | high |
|
|
2230
|
+
| `WP-U02` | Eval query APIs and DTOs | console | U01 | high |
|
|
2231
|
+
| `WP-U03` | Eval overview, experiment, case and trial routes | console | U02 | medium |
|
|
2232
|
+
| `WP-U04` | Trial timeline with thinking/tool/text/evidence drilldown | console | U02,T07 | high |
|
|
2233
|
+
| `WP-U05` | Historical mining and case-promotion workflow | console | U02,E04 | medium/high |
|
|
2234
|
+
| `WP-U06` | Live eval updates, adjudication and promotion UX | console | U02,E08,E10 | high |
|
|
2235
|
+
|
|
2236
|
+
Console does not own evaluator execution or canonical scores. It reads/materializes Specialists output and links to forensic evidence.
|
|
2237
|
+
|
|
2238
|
+
## 15.10 Deferred xtrm/Channels packages
|
|
2239
|
+
|
|
2240
|
+
These are linked future epics, not immediate child Beads of the Specialists implementation program unless the operator explicitly starts Track B.
|
|
2241
|
+
|
|
2242
|
+
| WP | Deliverable | Dependency |
|
|
2243
|
+
| -------- | ---------------------------------------------------------------------------- | -------------------------------- |
|
|
2244
|
+
| `WP-X01` | Stage 0 detail spec: daemon, state.db, socket, schema registry, registration | architecture gate |
|
|
2245
|
+
| `WP-X02` | Resolve canonical Channels storage/ownership in `packages/channels` | X01 |
|
|
2246
|
+
| `WP-X03` | Channels 0.1 autonomous chain path | X01,X02,critical chain semantics |
|
|
2247
|
+
| `WP-X04` | xtmux notification adapter over canonical message/handoff events | X03 |
|
|
2248
|
+
| `WP-X05` | Channels 0.2 freeform/node activation and collaboration evals | X03 plus runtime tuning |
|
|
2249
|
+
|
|
2250
|
+
## 15.11 Initial parallel wave
|
|
2251
|
+
|
|
2252
|
+
After documentation gate creation, the first multiplexed wave may safely start these lanes:
|
|
2253
|
+
|
|
2254
|
+
```text
|
|
2255
|
+
Lane A: WP-G00 + WP-G01
|
|
2256
|
+
Lane B: WP-G02 architecture/reuse audit
|
|
2257
|
+
Lane C: WP-T01/T02 design and fixtures under one runtime telemetry owner
|
|
2258
|
+
Lane D: WP-T04 activity query fix
|
|
2259
|
+
Lane E: WP-T05 forensic Prometheus wiring
|
|
2260
|
+
Lane F: WP-T06 fingerprint design
|
|
2261
|
+
Lane G: WP-E03 grader interface design
|
|
2262
|
+
Lane H: WP-M01 memory CLI/provenance audit
|
|
2263
|
+
Lane I: Console eval datasource/API design spike, no source writes
|
|
2264
|
+
```
|
|
2265
|
+
|
|
2266
|
+
Implementation of `WP-E01`, `WP-P02`, `WP-C01–C05` begins only after their named gates are closed.
|
|
2267
|
+
|
|
2268
|
+
## 15.12 File-ownership and collision policy
|
|
2269
|
+
|
|
2270
|
+
High-conflict surfaces require a single owner per wave:
|
|
2271
|
+
|
|
2272
|
+
| Surface | Exclusive owner rule |
|
|
2273
|
+
| -------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
|
|
2274
|
+
| `src/specialist/runner.ts` | one prompt/runtime-policy owner |
|
|
2275
|
+
| `src/pi/session.ts` + `src/specialist/timeline-events.ts` + central supervisor callback path | one telemetry-runtime owner per wave |
|
|
2276
|
+
| `src/specialist/observability-sqlite.ts` migrations | one schema/migration owner; other agents consume interfaces |
|
|
2277
|
+
| `src/specialist/prometheus-projection.ts` | one metrics-projection owner |
|
|
2278
|
+
| mandatory-rules index/global sets | one policy integration owner |
|
|
2279
|
+
| base output schema/contract builder | one output-contract owner |
|
|
2280
|
+
| each `config/specialists/<role>.specialist.json` | one role owner after shared contracts freeze |
|
|
2281
|
+
| Console materializer schema | one Console data owner |
|
|
2282
|
+
|
|
2283
|
+
Agents may work concurrently on tests, fixtures and role-specific files when their production surface does not overlap. `xtmux worktree-collisions` and repository diff checks must be run before integration.
|
|
2284
|
+
|
|
2285
|
+
## 15.13 Multiplexing operating protocol
|
|
2286
|
+
|
|
2287
|
+
During bridge-era execution:
|
|
2288
|
+
|
|
2289
|
+
1. Every agent receives one root Bead and one worktree/branch.
|
|
2290
|
+
2. Handoffs use Bead notes/results as durable content and xtmux pointer messages as notification.
|
|
2291
|
+
3. `message-send` is for short state or pointer messages; long requirements remain in the Bead or a versioned file.
|
|
2292
|
+
4. Recipients acknowledge messages. Lack of acknowledgement is visible but does not mutate release authority.
|
|
2293
|
+
5. `safe-send-pointer` may steer an idle/needs-input target; it must not inject into a running/thinking target.
|
|
2294
|
+
6. The integration owner monitors `dashboard`, `audit` and `worktree-collisions`.
|
|
2295
|
+
7. Agents do not merge or push outside their contract unless the chain explicitly assigns publication ownership.
|
|
2296
|
+
8. Cross-lane decisions are recorded in the relevant Bead and echoed to dependent lane owners.
|
|
2297
|
+
9. Findings that require new scope become `discovered-from` follow-up Beads rather than silent expansion.
|
|
2298
|
+
10. Final integration runs the mapped regression suites and records evidence before closing the parent work package.
|
|
2299
|
+
|
|
2300
|
+
## 15.14 Recommended agent roster and assignment
|
|
2301
|
+
|
|
2302
|
+
The local orchestrator should instantiate bounded agents by surface, not one agent per arbitrary bullet. A practical initial roster is:
|
|
2303
|
+
|
|
2304
|
+
| Agent identity | Primary responsibility | Assigned WPs | Must not own concurrently |
|
|
2305
|
+
| ---------------------------- | ----------------------------------------------------- | ----------------------- | -------------------------------------------------- |
|
|
2306
|
+
| `docs-reconciler` | Canonical roadmap/meta patches and cross-links | G00,G01,G03 | runtime code |
|
|
2307
|
+
| `architecture-reuse-auditor` | `bd` primitive reuse and bridge-retirement decision | G02 | implementation before decision approval |
|
|
2308
|
+
| `telemetry-runtime-owner` | Pi/timeline/supervisor tool and thinking semantics | T01,T02,T03 | eval DB migrations |
|
|
2309
|
+
| `telemetry-query-owner` | activity/stall query and historical compatibility | T04 | central callback path without coordination |
|
|
2310
|
+
| `metrics-projection-owner` | forensic-to-Prometheus wiring | T05 | Console panel implementation |
|
|
2311
|
+
| `fingerprint-owner` | resolved candidate manifest and startup hashes | T06,P01 | role prompt edits before schema approval |
|
|
2312
|
+
| `telemetry-integrator` | fixtures, compatibility and telemetry suite | T07,S08 | unrelated prompt work |
|
|
2313
|
+
| `eval-schema-owner` | eval migrations, DAO and storage types | E01 | other observability migrations in same wave |
|
|
2314
|
+
| `grader-core-owner` | grader interface and assertion schema | E03 | role-specific judge calibration |
|
|
2315
|
+
| `eval-runtime-owner` | post-run/backfill/eval CLI | E04,E05,E09 | post-chain logic before chain shape exists |
|
|
2316
|
+
| `experiment-owner` | paired trials, statistics and promotion | E07,E08 | candidate prompt implementation |
|
|
2317
|
+
| `chain-architecture-owner` | C01–C05 critical chain spine | C01,C02,C03,C04,C05 | prompt refactors |
|
|
2318
|
+
| `chain-context-owner` | context envelope, handoff and chain eval | C06,C07,C08,C09,C10,E06 | Channels implementation |
|
|
2319
|
+
| `prompt-contract-owner` | output schema and runner policy | P02,P03,P04 | role prompt files in parallel without freeze point |
|
|
2320
|
+
| `advisor-prompt-owner` | researcher/overthinker candidates | P05,S05,S06 | shared runner or global rule index |
|
|
2321
|
+
| `gate-prompt-owner` | seconder/test-engineer candidates | P06,S03,S04 | reviewer/executor files |
|
|
2322
|
+
| `executor-prompt-owner` | executor candidate and suite | P07,S02 | reviewer prompt |
|
|
2323
|
+
| `reviewer-prompt-owner` | reviewer candidate and suite | P08,S01 | executor prompt |
|
|
2324
|
+
| `memory-owner` | memory audit, runtime, rule and eval | M01–M06 | chain messaging protocol |
|
|
2325
|
+
| `console-eval-owner` | materializer/API/eval views | U01–U06 | upstream evaluator semantics |
|
|
2326
|
+
| `integration-reviewer` | cross-lane review, dependency and acceptance coverage | all integration gates | production implementation in reviewed lane |
|
|
2327
|
+
|
|
2328
|
+
The roster is a role map, not a requirement to run every agent simultaneously. Concurrency should be limited by integration bandwidth and file overlap. A recommended first wave is six to nine active agents, with one integration reviewer and one orchestrator tracking the whole program.
|
|
2329
|
+
|
|
2330
|
+
Assignment rules:
|
|
2331
|
+
|
|
2332
|
+
1. An agent owns one high-conflict production surface at a time.
|
|
2333
|
+
2. Test/fixture agents may support a production owner but cannot independently redefine event or schema semantics.
|
|
2334
|
+
3. Role prompt owners start only after the prompt-contract owner publishes the frozen base output and injection contract.
|
|
2335
|
+
4. The chain-context owner consumes the chain-architecture owner’s published resolved-shape API rather than editing its internals opportunistically.
|
|
2336
|
+
5. The Console owner receives versioned upstream DTO/event contracts; it does not infer them from private database internals.
|
|
2337
|
+
6. The integration reviewer verifies the Bead contract, diff, tests, eval evidence and downstream effects, but does not become the default fixer.
|
|
2338
|
+
|
|
2339
|
+
## 15.15 Recommended integration order
|
|
2340
|
+
|
|
2341
|
+
Merge and promotion order:
|
|
2342
|
+
|
|
2343
|
+
```text
|
|
2344
|
+
documentation/reuse decisions
|
|
2345
|
+
-> telemetry schema and integrity
|
|
2346
|
+
-> eval schema/grader core
|
|
2347
|
+
-> chain state foundations
|
|
2348
|
+
-> output contract and runner policy
|
|
2349
|
+
-> role prompt candidates
|
|
2350
|
+
-> chain context and handoff
|
|
2351
|
+
-> memory treatment
|
|
2352
|
+
-> model/prompt experiments
|
|
2353
|
+
-> Console product surfaces
|
|
2354
|
+
-> enforcement
|
|
2355
|
+
-> Channels replacement of bridge communication
|
|
2356
|
+
```
|
|
2357
|
+
|
|
2358
|
+
A lane may develop ahead on a feature branch, but it may not be promoted across an unmet integration gate.
|
|
2359
|
+
|
|
2360
|
+
# 16. Rollout plan
|
|
2361
|
+
|
|
2362
|
+
## Phase 0 — Documentation reconciliation and immutable baseline
|
|
2363
|
+
|
|
2364
|
+
1. Complete `WP-G00`–`WP-G03`.
|
|
2365
|
+
2. Capture resolved configs for all six roles across representative repos.
|
|
2366
|
+
3. Hash prompt, rules, skills, runner and tool catalog.
|
|
2367
|
+
4. Export first-turn input, active runtime, tools, tokens, verdicts and fix loops.
|
|
2368
|
+
5. Freeze historical reviewer/executor/chain cohorts and document completeness.
|
|
2369
|
+
6. Record current roadmap and code status so later reports do not compare against a moving baseline.
|
|
2370
|
+
|
|
2371
|
+
**Exit criteria:** canonical cross-links exist; the new dated reconciliation is approved; every implementation WP is traceable; a repeatable baseline report exists.
|
|
2372
|
+
|
|
2373
|
+
## Phase 1 — Telemetry integrity
|
|
2374
|
+
|
|
2375
|
+
1. Complete `WP-T01`–`WP-T07`.
|
|
2376
|
+
2. Version changed metric semantics rather than silently rewriting historical dashboards.
|
|
2377
|
+
3. Recompute a sample of historical metrics and compare against raw event traces.
|
|
2378
|
+
|
|
2379
|
+
**Exit criteria:** no known counting ambiguity in any metric used for A/B promotion; `telemetry-integrity-v1` passes.
|
|
2380
|
+
|
|
2381
|
+
## Phase 2 — Eval core and historical backfill
|
|
2382
|
+
|
|
2383
|
+
1. Complete `WP-E01`–`WP-E05` and `WP-E09` in observe-only mode.
|
|
2384
|
+
2. Backfill selected per-repo databases without mutating raw events.
|
|
2385
|
+
3. Create data-completeness summaries and initial role cohorts.
|
|
2386
|
+
4. Start `WP-S01`–`WP-S06` fixture creation in parallel.
|
|
2387
|
+
|
|
2388
|
+
**Exit criteria:** historical trials are inspectable; new runs automatically receive deterministic post-run scores; no evaluator can block execution yet.
|
|
2389
|
+
|
|
2390
|
+
## Phase 3 — Critical chain-first roadmap spine
|
|
2391
|
+
|
|
2392
|
+
1. Close `WP-G02` reuse decisions.
|
|
2393
|
+
2. Implement `WP-C01`–`WP-C05`.
|
|
2394
|
+
3. Persist chain shapes and expose composition/read surfaces.
|
|
2395
|
+
4. Keep any `step_completed` behavior visibility-only.
|
|
2396
|
+
|
|
2397
|
+
**Exit criteria:** a chain has durable identity independent of a live executor; ordered step/gate shape and workspace resolution are queryable and tested.
|
|
2398
|
+
|
|
2399
|
+
## Phase 4 — Output contract and runner policy consolidation
|
|
2400
|
+
|
|
2401
|
+
1. Complete `WP-P01`–`WP-P04`.
|
|
2402
|
+
2. Normalize status/verdict schemas.
|
|
2403
|
+
3. Remove duplicated output, GitNexus, style and Beads workflow instructions.
|
|
2404
|
+
4. Introduce compatibility shims and output-validation events.
|
|
2405
|
+
|
|
2406
|
+
**Exit criteria:** the generated output contract is the single machine-readable SSOT; role-aware injections pass regression fixtures.
|
|
2407
|
+
|
|
2408
|
+
## Phase 5 — Role prompt experiments
|
|
2409
|
+
|
|
2410
|
+
1. Run `WP-P05` first for overthinker/researcher.
|
|
2411
|
+
2. Run `WP-P06` for seconder/test-engineer.
|
|
2412
|
+
3. Run `WP-P07` and `WP-P08` last because executor/reviewer have the highest operational blast radius.
|
|
2413
|
+
4. Use paired or factorial experiments, never unpaired historical PASS rates as the promotion criterion.
|
|
2414
|
+
|
|
2415
|
+
**Exit criteria:** prompt reductions meet role-specific thresholds without deterioration of hard gates or confidence-interval promotion rules.
|
|
2416
|
+
|
|
2417
|
+
## Phase 6 — Complete chain context and handoff
|
|
2418
|
+
|
|
2419
|
+
1. Complete `WP-C06`–`WP-C10` and `WP-E06`.
|
|
2420
|
+
2. Inject deterministic chain identity and pointers.
|
|
2421
|
+
3. Add selective retrieval, handoff decision records and chain participation graders.
|
|
2422
|
+
|
|
2423
|
+
**Exit criteria:** reviewers and downstream gates can identify all required upstream inputs; context completeness is explicit; chain-participation suite meets threshold.
|
|
2424
|
+
|
|
2425
|
+
## Phase 7 — Pull-based memory
|
|
2426
|
+
|
|
2427
|
+
1. Complete `WP-M01`–`WP-M05`.
|
|
2428
|
+
2. Run treatment/control with eager injection still available as rollback.
|
|
2429
|
+
3. Complete `WP-M06` only after necessary-memory and unnecessary-query graders are calibrated.
|
|
2430
|
+
|
|
2431
|
+
**Exit criteria:** first-turn memory payload materially decreases with no statistically material increase in historical-context misses.
|
|
2432
|
+
|
|
2433
|
+
## Phase 8 — Model/prompt benchmark matrix
|
|
2434
|
+
|
|
2435
|
+
1. Complete `WP-E07`, `WP-E08` and remaining role suites.
|
|
2436
|
+
2. Run reviewer prompt×model factorial experiments first.
|
|
2437
|
+
3. Expand to executor, test-engineer and researcher.
|
|
2438
|
+
4. Add internal SWE-bench-style and selected external public benchmarks.
|
|
2439
|
+
5. Establish scheduled regression and capability runs.
|
|
2440
|
+
|
|
2441
|
+
## Phase 9 — Console Eval Lab
|
|
2442
|
+
|
|
2443
|
+
1. Complete `WP-U01`–`WP-U06`.
|
|
2444
|
+
2. Preserve aggregate-to-forensic drilldown and freshness/completeness metadata.
|
|
2445
|
+
3. Support live experiment visibility without moving evaluator ownership into Console.
|
|
2446
|
+
|
|
2447
|
+
## Phase 10 — Enforcement
|
|
2448
|
+
|
|
2449
|
+
1. Promote high-precision deterministic graders from observe to warn.
|
|
2450
|
+
2. Measure false positives and operator overrides.
|
|
2451
|
+
3. Promote only approved rules to enforce.
|
|
2452
|
+
4. Keep LLM judges advisory unless separately authorized.
|
|
2453
|
+
|
|
2454
|
+
## Phase 11 — Channels integration
|
|
2455
|
+
|
|
2456
|
+
After xtrm Stage 0 and Channels 0.1 acceptance:
|
|
2457
|
+
|
|
2458
|
+
1. replace manual remediation dispatch with typed scheduler intents;
|
|
2459
|
+
2. dual-write semantic messages to forensic/evidence;
|
|
2460
|
+
3. add message delivery, acknowledgement, authority and convergence evals;
|
|
2461
|
+
4. retire bridge scheduling behavior from `step_completed`;
|
|
2462
|
+
5. add optional xtmux notification adapter;
|
|
2463
|
+
6. defer freeform/node collaboration enforcement until Channels 0.2 is measured.
|
|
2464
|
+
|
|
2465
|
+
# 17. Detailed implementation backlog
|
|
2466
|
+
|
|
2467
|
+
The Epics below are capability groupings. For actual Beads decomposition and parallel assignment, Section 15 work packages are normative. A planner may group multiple WPs under one Epic, but must not merge their contracts or erase their dependencies.
|
|
2468
|
+
|
|
2469
|
+
## Epic A — Prompt composition inventory
|
|
2470
|
+
|
|
2471
|
+
* Add a resolved prompt manifest command.
|
|
2472
|
+
* Report component bytes/tokens and hashes.
|
|
2473
|
+
* Capture eager skill body costs.
|
|
2474
|
+
* Compare API first-turn truth with component estimate.
|
|
2475
|
+
* Detect duplicate rule IDs and repeated normalized text.
|
|
2476
|
+
* Detect conflicting enums/schema fields.
|
|
2477
|
+
|
|
2478
|
+
## Epic B — Output contract unification
|
|
2479
|
+
|
|
2480
|
+
* Select canonical status/verdict vocabulary.
|
|
2481
|
+
* Convert all role output schemas to formal JSON Schema.
|
|
2482
|
+
* Version merged output schema.
|
|
2483
|
+
* Remove handoff-rule duplication.
|
|
2484
|
+
* Emit output validation forensic events.
|
|
2485
|
+
* Add regression fixtures for markdown and JSON modes.
|
|
2486
|
+
|
|
2487
|
+
## Epic C — Six prompt refactors
|
|
2488
|
+
|
|
2489
|
+
* Reviewer core prompt and policy extraction.
|
|
2490
|
+
* Executor core prompt and language-skill separation.
|
|
2491
|
+
* Overthinker adaptive-depth prompt.
|
|
2492
|
+
* Seconder bounded gate prompt.
|
|
2493
|
+
* Test-engineer explicit mode and source boundary.
|
|
2494
|
+
* Researcher epistemic core and routing extraction.
|
|
2495
|
+
|
|
2496
|
+
## Epic D — Runner injection cleanup
|
|
2497
|
+
|
|
2498
|
+
* Role-aware style directive.
|
|
2499
|
+
* Single GitNexus policy source.
|
|
2500
|
+
* Writer-only commit/close instructions.
|
|
2501
|
+
* Split worktree boundary from evidence retrieval.
|
|
2502
|
+
* Remove global contradictions for researcher/read-only roles.
|
|
2503
|
+
|
|
2504
|
+
## Epic E — Chain context
|
|
2505
|
+
|
|
2506
|
+
* Persist/resolve chain template and ordered steps.
|
|
2507
|
+
* Chain context envelope builder.
|
|
2508
|
+
* Beads semantic edge query helpers.
|
|
2509
|
+
* Context completeness flags.
|
|
2510
|
+
* Shared chain participant rule.
|
|
2511
|
+
* Handoff extension.
|
|
2512
|
+
|
|
2513
|
+
## Epic F — Memory capability
|
|
2514
|
+
|
|
2515
|
+
* Search/relevant/recall command design.
|
|
2516
|
+
* Provenance and supersession fields.
|
|
2517
|
+
* Query telemetry.
|
|
2518
|
+
* Pull policy rule.
|
|
2519
|
+
* Eager injection experiment flag.
|
|
2520
|
+
|
|
2521
|
+
## Epic G — Telemetry integrity
|
|
2522
|
+
|
|
2523
|
+
* Thinking span model.
|
|
2524
|
+
* Tool-call deduplication and parallel duration.
|
|
2525
|
+
* Turn/message timing.
|
|
2526
|
+
* Activity query fixes.
|
|
2527
|
+
* Forensic Prometheus collector wiring.
|
|
2528
|
+
* Candidate fingerprint fields.
|
|
2529
|
+
* Telemetry integrity suite.
|
|
2530
|
+
|
|
2531
|
+
## Epic H — Eval storage and runtime
|
|
2532
|
+
|
|
2533
|
+
* Eval tables and migrations.
|
|
2534
|
+
* Eval forensic events.
|
|
2535
|
+
* Grader interface and registry.
|
|
2536
|
+
* Post-run evaluator.
|
|
2537
|
+
* Post-chain evaluator.
|
|
2538
|
+
* Historical backfill.
|
|
2539
|
+
* Trial artifact bundle.
|
|
2540
|
+
|
|
2541
|
+
## Epic I — Role suites
|
|
2542
|
+
|
|
2543
|
+
* Reviewer seeded-patch suite.
|
|
2544
|
+
* Executor hidden-test suite.
|
|
2545
|
+
* Seconder smell/scope suite.
|
|
2546
|
+
* Test-engineer fail-to-pass suite.
|
|
2547
|
+
* Research claim/citation suite.
|
|
2548
|
+
* Overthinker decision-risk suite.
|
|
2549
|
+
* Chain participation suite.
|
|
2550
|
+
|
|
2551
|
+
## Epic J — Experimentation and statistics
|
|
2552
|
+
|
|
2553
|
+
* Candidate configuration format.
|
|
2554
|
+
* Paired runner.
|
|
2555
|
+
* Trial randomization.
|
|
2556
|
+
* Pairwise judge and order swap.
|
|
2557
|
+
* Bootstrap and binary paired analysis.
|
|
2558
|
+
* Promotion policy engine.
|
|
2559
|
+
|
|
2560
|
+
## Epic K — Console integration
|
|
2561
|
+
|
|
2562
|
+
* Eval datasource/materializer rows.
|
|
2563
|
+
* Eval APIs.
|
|
2564
|
+
* Overview and experiment pages.
|
|
2565
|
+
* Trial timeline and evidence drawer.
|
|
2566
|
+
* Historical mining and case promotion.
|
|
2567
|
+
* Realtime updates.
|
|
2568
|
+
* Human adjudication and promotion UX.
|
|
2569
|
+
|
|
2570
|
+
# 18. Acceptance criteria
|
|
2571
|
+
|
|
2572
|
+
## Prompt modernization
|
|
2573
|
+
|
|
2574
|
+
* First-turn input tokens decrease materially for each refactored role.
|
|
2575
|
+
* No role exceeds baseline critical failure rate.
|
|
2576
|
+
* Output schema validity is at least 99.5%.
|
|
2577
|
+
* Reviewer false-PASS rate does not increase.
|
|
2578
|
+
* Executor scope violations do not increase.
|
|
2579
|
+
* Seconder remains within configured time/tool budget.
|
|
2580
|
+
* Test-engineer source-boundary violations remain zero.
|
|
2581
|
+
* Researcher citation correctness does not regress.
|
|
2582
|
+
|
|
2583
|
+
## Chain context
|
|
2584
|
+
|
|
2585
|
+
* Every chain step receives valid chain identity and root/step pointers.
|
|
2586
|
+
* Context completeness is explicitly marked.
|
|
2587
|
+
* Reviewer can identify all required upstream gates without filesystem hunting.
|
|
2588
|
+
* Upstream-result retrieval is selective and evidence-linked.
|
|
2589
|
+
* Handoff decision records are parseable.
|
|
2590
|
+
|
|
2591
|
+
## Memory
|
|
2592
|
+
|
|
2593
|
+
* Eager memory tokens are eliminated or substantially reduced for treatment candidates.
|
|
2594
|
+
* Necessary-memory recall remains above an approved threshold.
|
|
2595
|
+
* Unnecessary memory-query rate is measured and bounded.
|
|
2596
|
+
* Memory results include provenance.
|
|
2597
|
+
|
|
2598
|
+
## Telemetry
|
|
2599
|
+
|
|
2600
|
+
* Unique tool-call count is correct in fixture traces.
|
|
2601
|
+
* Thinking spans are paired and durations non-negative.
|
|
2602
|
+
* Active plus waiting time reconciles with elapsed time within tolerance.
|
|
2603
|
+
* Forensic-derived metric families appear in the live projection.
|
|
2604
|
+
* No forbidden labels are emitted.
|
|
2605
|
+
|
|
2606
|
+
## Eval platform
|
|
2607
|
+
|
|
2608
|
+
* Existing runs can be backfilled without modifying raw events.
|
|
2609
|
+
* New runs receive deterministic post-run scores.
|
|
2610
|
+
* Chains receive post-chain scores.
|
|
2611
|
+
* Candidate fingerprints allow exact comparison.
|
|
2612
|
+
* Paired experiments produce confidence intervals and win/tie/loss.
|
|
2613
|
+
* Promotion policy can block regressions and recommend qualifying candidates.
|
|
2614
|
+
|
|
2615
|
+
## Console
|
|
2616
|
+
|
|
2617
|
+
* Eval overview and trial detail are accessible from existing Console navigation.
|
|
2618
|
+
* Aggregate views drill down to forensic evidence.
|
|
2619
|
+
* Live and historical data display freshness/completeness.
|
|
2620
|
+
* No direct frontend access to per-repo SQLite is introduced.
|
|
2621
|
+
|
|
2622
|
+
# 19. Risks and mitigations
|
|
2623
|
+
|
|
2624
|
+
| Risk | Consequence | Mitigation |
|
|
2625
|
+
| ---------------------------------------------- | ------------------------------- | -------------------------------------------------------------------------------- |
|
|
2626
|
+
| Prompt slimming removes useful tacit behavior | Quality regression | Paired regression suite, staged roles, rollback by version |
|
|
2627
|
+
| Runtime and prompt change simultaneously | Attribution failure | Factorial design and candidate fingerprints |
|
|
2628
|
+
| Historical data is incomplete | Misleading retrospective scores | Data completeness flags; separate estimated from complete |
|
|
2629
|
+
| LLM judge bias | Incorrect promotion | Human calibration, order swap, multiple judges, deterministic hard gates |
|
|
2630
|
+
| Continuous eval adds latency | Slower chains | Asynchronous post-run evaluation; only hard deterministic gates on critical path |
|
|
2631
|
+
| Eval recursion | Evaluator jobs grade themselves | Explicit evaluator participant kind and exclusion policy |
|
|
2632
|
+
| High-cardinality metric explosion | Prometheus instability | Existing allowlist/forbidden labels; forensic drilldown |
|
|
2633
|
+
| Memory retrieval becomes another ritual | Token/tool waste | Trigger-based rule and unnecessary-query grader |
|
|
2634
|
+
| Chain context becomes a full history dump | Context bloat | Pointer envelope plus on-demand retrieval |
|
|
2635
|
+
| Tool count changes break historical dashboards | Apparent discontinuity | Version metric semantics and retain legacy field during transition |
|
|
2636
|
+
| External benchmark overfits coding only | Misleading role comparison | Internal role and chain suites remain primary |
|
|
2637
|
+
| Console materializer duplicates source truth | Architectural drift | Summary/evidence projection only; no evaluator ownership in UI |
|
|
2638
|
+
| Local rule/skill overrides escape experiments | Non-reproducible trial | Resolve and hash all effective components at launch |
|
|
2639
|
+
|
|
2640
|
+
# 20. Open decisions
|
|
2641
|
+
|
|
2642
|
+
The following decisions should be closed before implementation reaches enforcement:
|
|
2643
|
+
|
|
2644
|
+
1. Canonical base status and reviewer verdict casing.
|
|
2645
|
+
2. Whether eval tables live in every per-repo observability DB or in a separate namespace within the same file; this PRD recommends the same per-repo DB initially.
|
|
2646
|
+
3. Global registry location for cross-repo experiment definitions before `state.db` exists.
|
|
2647
|
+
4. Exact memory retrieval CLI surface and provenance availability in Beads.
|
|
2648
|
+
5. Which deterministic graders are allowed to block at first enforcement.
|
|
2649
|
+
6. Authoritative provider cost source; token usage should remain primary until cost provenance is versioned.
|
|
2650
|
+
7. Minimum trial count and confidence thresholds by role.
|
|
2651
|
+
8. Retention policy for eval artifacts and hidden benchmark environments.
|
|
2652
|
+
9. Whether the global style directive is removed entirely or replaced by role-specific concise-output settings.
|
|
2653
|
+
10. How package and installation-local skill references are resolved and fingerprinted when they are not repository files.
|
|
2654
|
+
11. Canonical storage ownership for Channels after Stage 0; recommendation is `packages/channels` over `state.db`, with any observability-DB implementation treated as a removable adapter.
|
|
2655
|
+
12. Whether the new dated reconciliation is maintained as a one-time snapshot or becomes an append-only sequence of dated status documents.
|
|
2656
|
+
13. Exact boundary between `step_completed` visibility and future Channel scheduler intent.
|
|
2657
|
+
14. Which work packages the operator wants started concurrently given available local agents and integration bandwidth.
|
|
2658
|
+
|
|
2659
|
+
# Appendix A — Proposed core system prompts
|
|
2660
|
+
|
|
2661
|
+
## A.1 Reviewer
|
|
2662
|
+
|
|
2663
|
+
```text
|
|
2664
|
+
You are the final evidence-based release reviewer for a completed specialist chain.
|
|
2665
|
+
|
|
2666
|
+
The seconder owns the initial contract and implementation-sanity gate. Consume
|
|
2667
|
+
upstream QA, security and obligations evidence before reviewing. Do not repeat an
|
|
2668
|
+
upstream gate unless its result is missing, contradicted by stronger evidence or
|
|
2669
|
+
invalidated by a later diff.
|
|
2670
|
+
|
|
2671
|
+
Review unresolved findings, critical requirement coverage, changed high-risk
|
|
2672
|
+
surfaces and required release evidence. Use the root contract and current artifacts
|
|
2673
|
+
as authority. Treat upstream outputs as evidence-backed claims, not as authority over
|
|
2674
|
+
observed code or tests.
|
|
2675
|
+
|
|
2676
|
+
Never edit files. Do not emit preference-only findings. Every blocking finding must
|
|
2677
|
+
identify the unmet requirement or gate and cite concrete evidence.
|
|
2678
|
+
|
|
2679
|
+
PASS requires all critical requirements and required gates to be evidenced with no
|
|
2680
|
+
blocking finding. PARTIAL means a fixable gap or insufficient evidence remains. FAIL
|
|
2681
|
+
means a core requirement is unmet, authoritative evidence is contradictory, or
|
|
2682
|
+
required review inputs are unusable.
|
|
2683
|
+
|
|
2684
|
+
On re-review after PARTIAL, inspect the delta and prior findings. Carry forward
|
|
2685
|
+
approvals for unchanged areas. Return only the configured output contract.
|
|
2686
|
+
```
|
|
2687
|
+
|
|
2688
|
+
## A.2 Executor
|
|
2689
|
+
|
|
2690
|
+
```text
|
|
2691
|
+
You are a scoped implementation specialist. Implement the supplied Bead contract in
|
|
2692
|
+
the assigned worktree.
|
|
2693
|
+
|
|
2694
|
+
Resolve scope, inspect the relevant implementation and dependency impact, and make
|
|
2695
|
+
the smallest correct change. Follow repository, language and framework conventions.
|
|
2696
|
+
Prefer existing code and dependencies over new abstractions or packages.
|
|
2697
|
+
|
|
2698
|
+
Do not add adjacent cleanup, speculative flexibility, broad refactors, tests,
|
|
2699
|
+
documentation or features unless they are in scope. If a necessary change lies
|
|
2700
|
+
outside authorized scope, return a partial handoff describing the required expansion.
|
|
2701
|
+
|
|
2702
|
+
Preserve security, data integrity, compatibility and external-boundary validation.
|
|
2703
|
+
Run the validation declared by the task plus focused checks needed to establish
|
|
2704
|
+
implementation correctness. Broad authoritative test execution belongs to the
|
|
2705
|
+
configured QA gate unless the task explicitly requires it.
|
|
2706
|
+
|
|
2707
|
+
Before completion, verify changed paths against scope. Return only the configured
|
|
2708
|
+
output contract with changes, verification, assumptions, risks and follow-ups.
|
|
2709
|
+
```
|
|
2710
|
+
|
|
2711
|
+
## A.3 Overthinker
|
|
2712
|
+
|
|
2713
|
+
```text
|
|
2714
|
+
You are a read-only decision-review specialist for uncertain or high-impact choices.
|
|
2715
|
+
|
|
2716
|
+
Analyze only as deeply as the decision warrants. Report conclusions, evidence,
|
|
2717
|
+
assumptions, alternatives and risks; do not expose private chain-of-thought.
|
|
2718
|
+
|
|
2719
|
+
For costly, risky or difficult-to-reverse decisions, stress-test the leading option
|
|
2720
|
+
with a counterargument or premortem and identify evidence that would change the
|
|
2721
|
+
recommendation. For simple reversible decisions, answer directly.
|
|
2722
|
+
|
|
2723
|
+
State confidence, material uncertainty and the next validation step. Never edit
|
|
2724
|
+
files. Return only the configured output contract.
|
|
2725
|
+
```
|
|
2726
|
+
|
|
2727
|
+
## A.4 Seconder
|
|
2728
|
+
|
|
2729
|
+
```text
|
|
2730
|
+
You are the bounded read-only pre-QA gate for a completed writer diff.
|
|
2731
|
+
|
|
2732
|
+
Determine whether the output satisfies the root contract and appears implementation-
|
|
2733
|
+
sane enough to justify expensive QA. Evaluate scope/compliance and implementation
|
|
2734
|
+
quality as separate dimensions. If evidence for one dimension is missing, mark only
|
|
2735
|
+
that dimension UNCLEAR rather than guessing.
|
|
2736
|
+
|
|
2737
|
+
Do not perform release review, security audit, broad architecture critique, test-
|
|
2738
|
+
coverage review or style review. Do not emit preference-only findings.
|
|
2739
|
+
|
|
2740
|
+
Overall PASS requires both dimensions to pass; FAIL applies when either fails;
|
|
2741
|
+
otherwise return PARTIAL. Keep inspection within the runtime budget and return only
|
|
2742
|
+
the configured output contract.
|
|
2743
|
+
```
|
|
2744
|
+
|
|
2745
|
+
## A.5 Test-engineer
|
|
2746
|
+
|
|
2747
|
+
```text
|
|
2748
|
+
You are a test-authoring specialist. The dispatch context explicitly defines
|
|
2749
|
+
`test_only` or `post_implementation` mode.
|
|
2750
|
+
|
|
2751
|
+
Build tests from the root requirements and actual implementation evidence. Follow
|
|
2752
|
+
existing test, fixture, smoke and harness conventions before introducing new patterns.
|
|
2753
|
+
|
|
2754
|
+
Your default edit boundary is tests, fixtures, recorded fixtures, smoke/E2E assets
|
|
2755
|
+
and test-harness support. Production source changes are forbidden unless dispatch
|
|
2756
|
+
explicitly authorizes a named helper or export. When source behavior or
|
|
2757
|
+
instrumentation is wrong, return `source_bug_suspected` with evidence.
|
|
2758
|
+
|
|
2759
|
+
Cover critical behavior rather than optimizing line coverage. Add operational or
|
|
2760
|
+
telemetry assertions when required by the changed surface. Run focused checks needed
|
|
2761
|
+
to validate authored assets; test-runner owns authoritative broad execution.
|
|
2762
|
+
|
|
2763
|
+
Return exact commands, setup/cleanup, coverage mapping, deferred paths and failure
|
|
2764
|
+
ownership through the configured output contract.
|
|
2765
|
+
```
|
|
2766
|
+
|
|
2767
|
+
## A.6 Researcher
|
|
2768
|
+
|
|
2769
|
+
```text
|
|
2770
|
+
You are an external-evidence researcher.
|
|
2771
|
+
|
|
2772
|
+
Use current authoritative sources for unstable claims involving libraries, APIs,
|
|
2773
|
+
versions, releases, ecosystem behavior or external products. Do not perform external
|
|
2774
|
+
research when current repository evidence or stable reasoning is sufficient.
|
|
2775
|
+
|
|
2776
|
+
Prefer primary documentation, standards, research papers and source repositories.
|
|
2777
|
+
Record relevant versions and dates. Distinguish sourced facts from inference and
|
|
2778
|
+
report unresolved gaps. Verify consequential claims across independent sources when
|
|
2779
|
+
practical and stop when the requested success criteria are met.
|
|
2780
|
+
|
|
2781
|
+
Report tool failures instead of inventing evidence. Never expose secrets in queries
|
|
2782
|
+
and never edit project files. Lead with the answer and return findings, citations,
|
|
2783
|
+
confidence and gaps through the configured output contract.
|
|
2784
|
+
```
|
|
2785
|
+
|
|
2786
|
+
# Appendix B — Proposed `chain-participant-context` rule
|
|
2787
|
+
|
|
2788
|
+
```text
|
|
2789
|
+
You are a participant in a multi-step specialist chain, not an isolated agent.
|
|
2790
|
+
|
|
2791
|
+
The chain root change-contract defines the overall objective and acceptance criteria.
|
|
2792
|
+
Your current step-contract defines your mandate and edit boundary. Upstream specialist
|
|
2793
|
+
outputs are evidence and handoffs; they do not override the root contract, current
|
|
2794
|
+
repository state or directly observed artifacts.
|
|
2795
|
+
|
|
2796
|
+
At startup, read the injected chain context and identify the chain, root Bead, current
|
|
2797
|
+
step, completed upstream steps and required downstream handoff. Consume preloaded
|
|
2798
|
+
dependency context before additional commands.
|
|
2799
|
+
|
|
2800
|
+
When required inputs are absent or ambiguous, reconstruct only the necessary context
|
|
2801
|
+
using the current step Bead, Beads dependency/relationship commands, `sp result` for
|
|
2802
|
+
upstream final handoffs and `sp feed --json` only when a claimed action or chronology
|
|
2803
|
+
must be verified.
|
|
2804
|
+
|
|
2805
|
+
Do not repeat work already completed upstream unless your role explicitly validates
|
|
2806
|
+
it, evidence is missing or current artifacts contradict it. Record contradictions.
|
|
2807
|
+
Do not request private chain-of-thought; consume decisions, rationale, assumptions,
|
|
2808
|
+
evidence, verification results and unresolved questions.
|
|
2809
|
+
|
|
2810
|
+
Your final handoff must allow the next chain member to continue without reconstructing
|
|
2811
|
+
your entire session.
|
|
2812
|
+
```
|
|
2813
|
+
|
|
2814
|
+
# Appendix C — Proposed memory recall rule
|
|
2815
|
+
|
|
2816
|
+
```text
|
|
2817
|
+
Historical project memories are available through targeted Beads memory queries.
|
|
2818
|
+
|
|
2819
|
+
Query memory only when the task depends on prior decisions, conventions, known
|
|
2820
|
+
incidents, accepted exceptions or unresolved ambiguity. Use specific subsystem,
|
|
2821
|
+
operation or file keywords. Refine broad result sets instead of scrolling an entire
|
|
2822
|
+
memory corpus. Fetch a full memory only after a relevant summary is identified.
|
|
2823
|
+
|
|
2824
|
+
Treat memories as contextual evidence, not current authority. Verify them against the
|
|
2825
|
+
root contract, current code, documentation and external sources where applicable.
|
|
2826
|
+
Report stale or contradictory memories explicitly.
|
|
2827
|
+
|
|
2828
|
+
Do not query memory for fully specified local work or as a ritual. Do not use internal
|
|
2829
|
+
memory as a source for current external API, release or standards claims.
|
|
2830
|
+
```
|
|
2831
|
+
|
|
2832
|
+
# Appendix D — Initial deterministic grader catalog
|
|
2833
|
+
|
|
2834
|
+
| Grader ID | Role/scope | Evidence | Result |
|
|
2835
|
+
| ---------------------------------- | ---------------- | ---------------------------- | -------------------- |
|
|
2836
|
+
| `output-schema-v1` | all | final output + merged schema | valid/invalid |
|
|
2837
|
+
| `role-write-boundary-v1` | read-only roles | tool events + diff | pass/fail |
|
|
2838
|
+
| `scope-paths-v1` | writers | root/step scope + diff | precision/violations |
|
|
2839
|
+
| `required-tools-v1` | configured roles | trace | pass/fail |
|
|
2840
|
+
| `tool-accounting-v1` | telemetry | tool spans | pass/fail |
|
|
2841
|
+
| `thinking-span-integrity-v1` | telemetry | thinking spans | pass/fail |
|
|
2842
|
+
| `chain-context-present-v1` | chain steps | startup context | pass/fail |
|
|
2843
|
+
| `upstream-result-consumption-v1` | gates/advisors | tools + handoff | score |
|
|
2844
|
+
| `handoff-completeness-v1` | all | final schema | score |
|
|
2845
|
+
| `review-verdict-consistency-v1` | reviewer | gates + verdict | pass/fail |
|
|
2846
|
+
| `test-engineer-source-boundary-v1` | test-engineer | diff | pass/fail |
|
|
2847
|
+
| `research-citation-shape-v1` | researcher | output | pass/fail |
|
|
2848
|
+
| `memory-query-necessity-v1` | selected cases | trace + case label | pass/fail |
|
|
2849
|
+
| `canonical-chain-sequence-v1` | chain | member timeline | pass/fail |
|
|
2850
|
+
| `time-to-pass-v1` | chain | statuses/verdicts | value |
|
|
2851
|
+
|
|
2852
|
+
# Appendix E — Metrics catalog after modernization
|
|
2853
|
+
|
|
2854
|
+
## Job and model
|
|
2855
|
+
|
|
2856
|
+
```text
|
|
2857
|
+
xtrm_jobs_total
|
|
2858
|
+
xtrm_job_duration_seconds
|
|
2859
|
+
xtrm_job_active_runtime_seconds
|
|
2860
|
+
xtrm_job_wait_seconds
|
|
2861
|
+
xtrm_turns_total
|
|
2862
|
+
xtrm_turn_duration_seconds
|
|
2863
|
+
xtrm_message_duration_seconds
|
|
2864
|
+
xtrm_context_usage_ratio
|
|
2865
|
+
xtrm_llm_tokens_total
|
|
2866
|
+
xtrm_thinking_duration_seconds
|
|
2867
|
+
xtrm_thinking_segments_total
|
|
2868
|
+
xtrm_time_to_first_action_seconds
|
|
2869
|
+
```
|
|
2870
|
+
|
|
2871
|
+
## Tools and dependencies
|
|
2872
|
+
|
|
2873
|
+
```text
|
|
2874
|
+
xtrm_tool_calls_total
|
|
2875
|
+
xtrm_tool_errors_total
|
|
2876
|
+
xtrm_tool_call_duration_seconds
|
|
2877
|
+
xtrm_uncorrelated_tool_events_total
|
|
2878
|
+
xtrm_mcp_operations_total
|
|
2879
|
+
```
|
|
2880
|
+
|
|
2881
|
+
## Chain and gates
|
|
2882
|
+
|
|
2883
|
+
```text
|
|
2884
|
+
xtrm_chains_total
|
|
2885
|
+
xtrm_chain_duration_seconds
|
|
2886
|
+
xtrm_gate_verdicts_total
|
|
2887
|
+
xtrm_chain_fix_loops_total
|
|
2888
|
+
xtrm_chain_time_to_pass_seconds
|
|
2889
|
+
xtrm_chain_tokens_to_pass_total
|
|
2890
|
+
xtrm_evidence_refs_total
|
|
2891
|
+
```
|
|
2892
|
+
|
|
2893
|
+
## Memory
|
|
2894
|
+
|
|
2895
|
+
```text
|
|
2896
|
+
xtrm_memory_queries_total
|
|
2897
|
+
xtrm_memory_results_total
|
|
2898
|
+
xtrm_memory_recall_total
|
|
2899
|
+
xtrm_memory_query_duration_seconds
|
|
2900
|
+
```
|
|
2901
|
+
|
|
2902
|
+
## Evaluation
|
|
2903
|
+
|
|
2904
|
+
```text
|
|
2905
|
+
xtrm_eval_runs_total
|
|
2906
|
+
xtrm_eval_score
|
|
2907
|
+
xtrm_eval_assertions_total
|
|
2908
|
+
xtrm_eval_regressions_total
|
|
2909
|
+
xtrm_eval_judge_disagreements_total
|
|
2910
|
+
xtrm_grader_duration_seconds
|
|
2911
|
+
```
|
|
2912
|
+
|
|
2913
|
+
# Appendix F — Source inventory used for this PRD
|
|
2914
|
+
|
|
2915
|
+
## Specialists repository
|
|
2916
|
+
|
|
2917
|
+
The current-state findings in this PRD were verified against the following repository areas during the audit:
|
|
2918
|
+
|
|
2919
|
+
```text
|
|
2920
|
+
config/specialists/*.specialist.json
|
|
2921
|
+
config/mandatory-rules/
|
|
2922
|
+
config/skills/
|
|
2923
|
+
docs/design/roadmap/chain-templates/
|
|
2924
|
+
docs/archive/iron-review-hardening-qa-chain-substrate.md
|
|
2925
|
+
docs/design/roadmap/specialists-roadmap.md
|
|
2926
|
+
config/skills/using-specialists-v3/SKILL.md
|
|
2927
|
+
config/skills/using-kpi/SKILL.md
|
|
2928
|
+
src/specialist/runner.ts
|
|
2929
|
+
src/specialist/supervisor.ts
|
|
2930
|
+
src/specialist/beads.ts
|
|
2931
|
+
src/specialist/timeline-events.ts
|
|
2932
|
+
src/specialist/forensic-events.ts
|
|
2933
|
+
src/specialist/observability-sqlite.ts
|
|
2934
|
+
src/specialist/prometheus-projection.ts
|
|
2935
|
+
src/pi/session.ts
|
|
2936
|
+
src/cli/db.ts
|
|
2937
|
+
src/cli/metrics.ts
|
|
2938
|
+
```
|
|
2939
|
+
|
|
2940
|
+
## xtrm and xtmux repositories
|
|
2941
|
+
|
|
2942
|
+
* `xtrm-dev/xtrm/docs/_meta/2026-06-21-design-reconciliation.md`
|
|
2943
|
+
* `xtrm-dev/xtrm/docs/channels/channels.md`
|
|
2944
|
+
* `xtrm-dev/xtrm/docs/substrate/substrate_design_it.md`
|
|
2945
|
+
* `Jaggerxtrm/xtmux/README.md` and its event/message channel contract
|
|
2946
|
+
|
|
2947
|
+
## Console repository
|
|
2948
|
+
|
|
2949
|
+
```text
|
|
2950
|
+
README.md
|
|
2951
|
+
docs/architecture/console-observability-spec.md
|
|
2952
|
+
docs/architecture/console-architecture.md
|
|
2953
|
+
```
|
|
2954
|
+
|
|
2955
|
+
## External methodological references
|
|
2956
|
+
|
|
2957
|
+
The evaluation design is informed by current public guidance and research on agent evaluations, deterministic and model-based graders, trace versus outcome evaluation, pairwise judge bias, and software-engineering agent benchmarks. Representative sources include OpenAI Agent Evals and Graders guidance, Anthropic publications on building effective agents and agent evaluations, UK AISI Inspect AI, Lost-in-the-Middle research, SWE-bench Pro and Multi-SWE-bench.
|
|
2958
|
+
|
|
2959
|
+
These external references inform methodology only. The Specialists runtime, chain, telemetry and Console facts in this document derive from direct repository inspection.
|
|
2960
|
+
|
|
2961
|
+
# Appendix G — Copy-paste contract for the local decomposition agent
|
|
2962
|
+
|
|
2963
|
+
Use the following instruction with a planning-capable local agent. Replace repository paths only when the local checkout differs.
|
|
2964
|
+
|
|
2965
|
+
```text
|
|
2966
|
+
You are the planning and decomposition agent for the Specialists modernization program.
|
|
2967
|
+
|
|
2968
|
+
Authoritative inputs, in order:
|
|
2969
|
+
1. Current code and executable schemas in the connected repositories.
|
|
2970
|
+
2. xtrm-dev/specialists/docs/design/roadmap/specialists-roadmap.md.
|
|
2971
|
+
3. specialists_prompt_chain_evals_prd.md, especially Sections 13–18.
|
|
2972
|
+
4. xtrm/docs/channels/channels.md for future channel semantics.
|
|
2973
|
+
5. The newest dated xtrm _meta reconciliation; the 2026-06-21 file is historical.
|
|
2974
|
+
|
|
2975
|
+
Planning-only mandate:
|
|
2976
|
+
- Do not edit production code.
|
|
2977
|
+
- Do not begin implementation.
|
|
2978
|
+
- Inspect the live Beads formula catalog and current open/closed Beads before creating duplicates.
|
|
2979
|
+
- Reconcile existing roadmap Beads to PRD work packages rather than blindly creating a second backlog.
|
|
2980
|
+
- Preserve every WP-* identifier from Section 15.
|
|
2981
|
+
- Create or update program/workstream epics and root change Beads.
|
|
2982
|
+
- Add explicit dependencies and typed relationships.
|
|
2983
|
+
- Assign repository, priority, scrutiny, recommended_template, primary role, lane, file ownership, eval gates, non-goals, rollback and bridge-retirement trigger.
|
|
2984
|
+
- Use the root change-contract template in Section 14.3.
|
|
2985
|
+
- Do not materialize specialist step Beads during Pass 1 unless the current runtime requires a bridge tracking Bead; label such work bridge:manual-composition.
|
|
2986
|
+
- Flag file collisions and propose a safe multiplexing wave.
|
|
2987
|
+
- Map every acceptance criterion in Section 18 to one or more Beads.
|
|
2988
|
+
- Mark existing Beads as reused, superseded, extended or duplicate candidates; never silently discard them.
|
|
2989
|
+
|
|
2990
|
+
Required outputs:
|
|
2991
|
+
1. Program hierarchy and Bead IDs.
|
|
2992
|
+
2. WP-to-Bead mapping table.
|
|
2993
|
+
3. Dependency DAG and critical path.
|
|
2994
|
+
4. Initial parallel wave with assigned agent identities from Section 15.14.
|
|
2995
|
+
5. File-ownership/collision matrix.
|
|
2996
|
+
6. Existing-Bead reconciliation report.
|
|
2997
|
+
7. Acceptance-criterion coverage report.
|
|
2998
|
+
8. Open decisions that genuinely block dispatch.
|
|
2999
|
+
9. Exact next dispatch commands or xtmux handoff pointers, but do not execute them.
|
|
3000
|
+
|
|
3001
|
+
Stop and report rather than inventing a design when canonical documents conflict materially.
|
|
3002
|
+
```
|
|
3003
|
+
|
|
3004
|
+
# Appendix H — Bridge-era multiplexing dispatcher checklist
|
|
3005
|
+
|
|
3006
|
+
Before dispatch:
|
|
3007
|
+
|
|
3008
|
+
```text
|
|
3009
|
+
[ ] Decomposition and reuse audit approved
|
|
3010
|
+
[ ] Root Bead contains WP identifier and falsifiable contract
|
|
3011
|
+
[ ] Recommended chain template verified against live catalog
|
|
3012
|
+
[ ] Dependencies satisfied or explicitly represented
|
|
3013
|
+
[ ] Worktree/branch created and unique
|
|
3014
|
+
[ ] File ownership has no collision with active lanes
|
|
3015
|
+
[ ] Candidate fingerprint/baseline requirement known
|
|
3016
|
+
[ ] Eval and validation commands named
|
|
3017
|
+
[ ] Bridge retirement trigger present where applicable
|
|
3018
|
+
```
|
|
3019
|
+
|
|
3020
|
+
Dispatch and coordination:
|
|
3021
|
+
|
|
3022
|
+
```bash
|
|
3023
|
+
# Inspect current agent/worktree state
|
|
3024
|
+
tmux-session-picker dashboard expanded
|
|
3025
|
+
tmux-session-picker worktree-collisions
|
|
3026
|
+
tmux-session-picker audit
|
|
3027
|
+
|
|
3028
|
+
# Produce a dry-run handoff pointer for an idle target
|
|
3029
|
+
tmux-session-picker handoff --target <pane> --bead <bead-id> --note 'WP-XX; read root contract; no scope expansion'
|
|
3030
|
+
|
|
3031
|
+
# Send after reviewing the dry-run
|
|
3032
|
+
tmux-session-picker handoff --yes --target <pane> --bead <bead-id> --note 'WP-XX; read root contract; no scope expansion'
|
|
3033
|
+
|
|
3034
|
+
# Short pointer/status messages only
|
|
3035
|
+
tmux-session-picker message-send --from <orchestrator> --to <agent> --bead <bead-id> --text 'dependency WP-YY merged; rebase/check before continuing'
|
|
3036
|
+
|
|
3037
|
+
# Recipient acknowledgement
|
|
3038
|
+
tmux-session-picker message-list --for <agent> --unacked
|
|
3039
|
+
tmux-session-picker message-ack <message-id> --by <agent>
|
|
3040
|
+
```
|
|
3041
|
+
|
|
3042
|
+
Before integration:
|
|
3043
|
+
|
|
3044
|
+
```text
|
|
3045
|
+
[ ] Structured implementation handoff present
|
|
3046
|
+
[ ] Changed files match declared ownership
|
|
3047
|
+
[ ] Deterministic validation evidence present
|
|
3048
|
+
[ ] Named eval suite executed or explicitly deferred by gate
|
|
3049
|
+
[ ] No hidden cross-WP scope expansion
|
|
3050
|
+
[ ] New findings have discovered-from follow-up Beads
|
|
3051
|
+
[ ] Compatibility and migration tests pass
|
|
3052
|
+
[ ] Integration reviewer verdict recorded
|
|
3053
|
+
[ ] Dependent lane owners notified through durable Bead note + pointer message
|
|
3054
|
+
```
|