@jaggerxtrm/specialists 3.18.1 → 3.18.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/config/mandatory-rules/code-quality-defaults.md +58 -1
- package/config/skills/setup-specialists/SKILL.md +1 -1
- package/config/skills/using-script-specialists/SKILL.md +5 -5
- package/config/skills/using-specialists/SKILL.md +1135 -879
- package/config/skills/using-specialists-auto/SKILL.md +21 -21
- package/config/specialists/chain-coordinator.specialist.json +63 -0
- package/config/specialists/reviewer.specialist.json +1 -1
- package/config/specialists/seconder.specialist.json +1 -1
- package/dist/asset-contract.json +8 -13
- package/dist/index.js +151 -28
- package/dist/lib.js +8 -0
- package/dist/types/cli/clean.d.ts.map +1 -1
- package/dist/types/cli/init.d.ts.map +1 -1
- package/dist/types/cli/view.d.ts.map +1 -1
- package/dist/types/specialist/control.d.ts.map +1 -1
- package/dist/types/specialist/loader.d.ts +12 -0
- package/dist/types/specialist/loader.d.ts.map +1 -1
- package/dist/types/specialist/process-health.d.ts +16 -1
- package/dist/types/specialist/process-health.d.ts.map +1 -1
- package/dist/types/specialist/supervisor.d.ts.map +1 -1
- package/docs/design/roadmap/chain-templates/README.md +1 -1
- package/docs/design/roadmap/chains-prompt-evals.md +3054 -0
- package/docs/design/roadmap/enhanced-prd.md +4282 -0
- package/docs/design/roadmap/specialists-roadmap.md +7 -7
- package/docs/design/xt-pi-role-pi-flag-passthrough.md +174 -0
- package/docs/skills.md +4 -13
- package/package.json +1 -1
- package/config/skills/using-specialists-v2/SKILL.md +0 -766
- package/config/skills/using-specialists-v3/SKILL.md +0 -1327
|
@@ -0,0 +1,4282 @@
|
|
|
1
|
+
# Specialists Modernization: Prompt, Chain Context, Interactive Coordination, Observability and Evaluation
|
|
2
|
+
|
|
3
|
+
## Product Requirements Document
|
|
4
|
+
|
|
5
|
+
**Status:** Final consolidated implementation and decomposition PRD; includes the 13 July 2026 Beads v1.1/main capability reconciliation
|
|
6
|
+
**Date:** 13 July 2026
|
|
7
|
+
**Primary repository:** `xtrm-dev/specialists`
|
|
8
|
+
**Related repositories:** `xtrm-dev/core`, `Jaggerxtrm/xtmux`, `Jaggerxtrm/console` / Omniforge, `xtrm-dev/xtrm`
|
|
9
|
+
**Audience:** Specialists runtime maintainers, Core/launcher maintainers, xtmux maintainers, agent-orchestration engineers, Console maintainers, evaluation engineers and platform/observability engineers
|
|
10
|
+
**Scope:** Prompt and rule modernization for reviewer, executor, overthinker, seconder, test-engineer and researcher; progressive-disclosure redesign of `using-specialists`; interactive `chain-coordinator` bootstrap and assignment; chain-member identity and context reconstruction; memory retrieval policy; forensic and metrics hardening; historical and continuous evaluation; controlled model and prompt A/B testing; Console integration.
|
|
11
|
+
|
|
12
|
+
**Canonical companions:** `docs/design/roadmap/specialists-roadmap.md` for the bridge-runtime roadmap; `docs/design/roadmap/chain-templates/` for executable chain shapes; `xtrm-dev/core/docs/xt-pi-role.md` for the interactive launcher; `Jaggerxtrm/xtmux/docs/observability-redesign.md` for the current local attention/messaging runtime; `xtrm/docs/channels/channels.md` for future channel semantics; and the newest dated xtrm reconciliation for cross-repository sequencing.
|
|
13
|
+
|
|
14
|
+
## Consolidation record
|
|
15
|
+
|
|
16
|
+
This final document merges two prior artifacts:
|
|
17
|
+
|
|
18
|
+
1. the 12 July prompt, chain-context, telemetry and evaluation PRD;
|
|
19
|
+
2. the 13 July branch consolidation that added current interactive-role, coordinator, xtmux V2, policy-hook and filed-Beads evidence.
|
|
20
|
+
|
|
21
|
+
A mechanical comparison found that the branch consolidation retained approximately **98.6% of the earlier PRD line-for-line** and added approximately **1,170 lines**. The additions are therefore treated as extensions of one program rather than a competing design.
|
|
22
|
+
|
|
23
|
+
Accepted net-new scope:
|
|
24
|
+
|
|
25
|
+
- the shipped `xt pi --role` / `xt claude --role` launch family;
|
|
26
|
+
- the bridge interactive `chain-coordinator`;
|
|
27
|
+
- progressive disclosure for `using-specialists`;
|
|
28
|
+
- a pure interactive assignment renderer owned by Specialists;
|
|
29
|
+
- runtime-readiness and secure prompt-file delivery across Core and xtmux;
|
|
30
|
+
- the xtmux SQLite V2 attention, message, receipt, handoff and monitor primitives;
|
|
31
|
+
- a shared cross-harness Bead/chain policy kernel with durable findings and shadow-first rollout;
|
|
32
|
+
- 18 additional work packages: `WP-IC01`–`WP-IC13` and `WP-H01`–`WP-H05`;
|
|
33
|
+
- reconciliation instructions for the Beads already filed by the parallel planning branch;
|
|
34
|
+
- the Beads v1.1 and post-v1.1 capability audit: molecule-aware readiness, atomic graph creation, fan-in gates, contract metadata, migration safety, memory reuse and unreleased worker-lease/formula/history capabilities.
|
|
35
|
+
|
|
36
|
+
No previously accepted prompt, chain, memory, telemetry, evaluation, Console or future Channels requirement is removed. Where this document repeats a requirement in an executive section and a detailed workstream, the detailed numbered workstream and Section 15 work-package contract are normative.
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
# 0. Verified baseline and closed runway decisions
|
|
43
|
+
|
|
44
|
+
This revision incorporates three classes of new evidence that were not represented in the original prompt/eval PRD:
|
|
45
|
+
|
|
46
|
+
1. A filed cross-repository Beads packet covering `using-specialists` progressive disclosure and interactive role task-context parity.
|
|
47
|
+
2. The xtmux Bun/TypeScript + SQLite overhaul and its cross-harness hook/extension pattern, which were evaluated before the later monorepo and interactive-coordinator discussions. This includes typed messages, receipts, handoffs, delivery attempts, agent lifecycle, monitors, durable audit findings, machine-readable inbox queries, Pi reply reminders and automatic monitor arming.
|
|
48
|
+
3. The subsequently shipped `xt pi --role` / `xt claude --role` launcher family and the interactive `chain-coordinator` specialist.
|
|
49
|
+
|
|
50
|
+
The filed Beads are **not authoritative**. They are reconciled in Appendices I and J against current code, this conversation's closed decisions, the canonical Specialists roadmap and the future Substrate design. A local planning agent must update, extend, split or supersede the existing Beads according to that mapping rather than executing them verbatim.
|
|
51
|
+
|
|
52
|
+
## 0.1 Verified implementation baseline
|
|
53
|
+
|
|
54
|
+
| Repository | Verified baseline used by this PRD | Consequence |
|
|
55
|
+
|---|---|---|
|
|
56
|
+
| `xtrm-dev/specialists` | package version `3.18.2`; `chain-coordinator` shipped; `using-specialists` is the canonical skill name; `sp view --raw` returns the merged effective spec | Do not plan creation of these features as net-new. Plan refinement, rendering parity, coordinator doctrine and evaluation. |
|
|
57
|
+
| `xtrm-dev/core` | package version `0.10.2`; global skill migration and direct runtime links shipped; Pi/Claude `--role` parity shipped | Runtime paths must follow the global installed skill model. Do not depend on retired `.xtrm/skills/active/*` project paths. |
|
|
58
|
+
| `Jaggerxtrm/xtmux` | SQLite V2 is default-on; 39-commit overhaul after the first `xt/hnjk` design deliverable; rollback is explicit `XTMUX_OBS_V2=0` | Treat typed messages, receipts, handoffs, monitors and agent lifecycle as current bridge primitives, not speculative future work. |
|
|
59
|
+
| Console/Omniforge | existing materializer consumes per-repository Specialists observability state | Continue the materializer architecture until Substrate and `state.db` actually land. |
|
|
60
|
+
|
|
61
|
+
## 0.2 Runway assumptions fixed by the operator
|
|
62
|
+
|
|
63
|
+
- Substrate will arrive materially later than the roadmap bridge work.
|
|
64
|
+
- Specialists continues to use each repository's `.specialists/db/observability.db` as its authoritative runtime and eval store during the bridge runway.
|
|
65
|
+
- Console remains a separate repository and continues to use its own materializer. It is expanded only after Specialists event, metric and eval contracts stabilize.
|
|
66
|
+
- `state.db`, the Substrate daemon and Channels remain future migration targets and are not on the critical path of this PRD.
|
|
67
|
+
- No monorepo migration or package graft is required to begin or complete the bridge program.
|
|
68
|
+
- Interactive-role permission enforcement is explicitly deferred. Worktree isolation and model discipline remain the bridge safety model; this PRD does not add a new permission subsystem.
|
|
69
|
+
|
|
70
|
+
## 0.3 Foundational xtmux-derived hook and policy architecture
|
|
71
|
+
|
|
72
|
+
This decision track predates both the monorepo discussion and the interactive `chain-coordinator` work. It arose from a direct audit of the xtmux overhaul and its latest consolidated commits. It must therefore be represented as an independent architectural input to `specialists-roadmap.md`, not as a coordinator implementation detail.
|
|
73
|
+
|
|
74
|
+
The verified xtmux implementation is Bun/TypeScript plus SQLite, with Node-compatible Claude hooks and in-process Pi extensions. The important reusable result is not any single hook file. It is the operational pattern:
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
pure semantic decision logic
|
|
78
|
+
↓
|
|
79
|
+
typed durable state and correlated evidence
|
|
80
|
+
↓
|
|
81
|
+
thin harness-specific adapters
|
|
82
|
+
↓
|
|
83
|
+
agent/operator-visible feedback and wake-up
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
xtmux proved this pattern across messages, receipts, delivery attempts, monitors, handoffs, command telemetry, audit findings and agent lifecycle. In particular:
|
|
87
|
+
|
|
88
|
+
- a durable message is distinct from a best-effort pane delivery;
|
|
89
|
+
- `message-list` is a pure read and `message-ack` is the explicit receipt mutation;
|
|
90
|
+
- monitor state is durable, lease-backed and reconciled after crashes;
|
|
91
|
+
- repeated audit findings use stable fingerprints and update `last_seen` rather than duplicating rows;
|
|
92
|
+
- command runs share correlation IDs across start/finish envelopes;
|
|
93
|
+
- hooks and extensions make a repeatedly forgotten workflow invariant structural rather than relying on prompt memory;
|
|
94
|
+
- the Claude no-match path uses a cheap shell prefilter so an irrelevant Bash call does not pay a runtime cold-start cost;
|
|
95
|
+
- the Pi equivalent runs in-process and can append a concise confirmation to the tool result;
|
|
96
|
+
- rollout can proceed through compatibility/shadow modes before a new path becomes authoritative.
|
|
97
|
+
|
|
98
|
+
### 0.3.1 Consequence for the Specialists roadmap hooks
|
|
99
|
+
|
|
100
|
+
The roadmap's original Claude-only `PostToolUse` script for `bd create` is too narrow. It must be redesigned as a cross-harness policy layer:
|
|
101
|
+
|
|
102
|
+
```text
|
|
103
|
+
shared Bead/chain policy kernel
|
|
104
|
+
├── Claude Code cheap prefilter
|
|
105
|
+
│ ├── PreToolUse adapter where an operation can be rejected before mutation
|
|
106
|
+
│ └── PostToolUse adapter for created-object inspection, annotation and findings
|
|
107
|
+
├── Pi in-process extension adapter
|
|
108
|
+
└── authoritative Specialists runtime consumers
|
|
109
|
+
├── sp chain review
|
|
110
|
+
├── sp chain approve
|
|
111
|
+
└── sp run / dispatch preflight
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
The shared kernel owns semantic decisions such as:
|
|
115
|
+
|
|
116
|
+
- contract readiness and `contract:draft` refusal;
|
|
117
|
+
- SCRUTINY presence and minimum level;
|
|
118
|
+
- bead type/shape mismatch;
|
|
119
|
+
- template recommendation;
|
|
120
|
+
- scope and sensitive-surface classification;
|
|
121
|
+
- whether a finding is advisory, blocking or requires escalation.
|
|
122
|
+
|
|
123
|
+
It accepts structured input and returns structured decisions. It does not call tmux, Claude, Pi or SQLite directly. Harness adapters translate tool events into kernel inputs and render the result. The authoritative `sp` commands call the same kernel directly with richer runtime state.
|
|
124
|
+
|
|
125
|
+
### 0.3.2 Lifecycle split
|
|
126
|
+
|
|
127
|
+
A single `PostToolUse` hook cannot honestly "prevent" a Bead that has already been created. The lifecycle is therefore split:
|
|
128
|
+
|
|
129
|
+
| Boundary | Responsibility |
|
|
130
|
+
|---|---|
|
|
131
|
+
| Claude/Pi pre-operation | reject only what is safely decidable from the proposed command/input |
|
|
132
|
+
| post-operation adapter | resolve the created Bead, inspect actual state, persist findings, suggest or apply bounded metadata repair |
|
|
133
|
+
| `sp chain review/approve` | authoritative composition and contract validation |
|
|
134
|
+
| `sp run` dispatch boundary | final refusal when an unsafe or incomplete contract would execute |
|
|
135
|
+
|
|
136
|
+
Early feedback may be helpful, but optional harness hooks are never the final authority.
|
|
137
|
+
|
|
138
|
+
### 0.3.3 Durable findings instead of transient warnings
|
|
139
|
+
|
|
140
|
+
Policy findings must not be printed and lost. They use stable identity and persistence analogous to xtmux audit findings:
|
|
141
|
+
|
|
142
|
+
```json
|
|
143
|
+
{
|
|
144
|
+
"policy_version": "...",
|
|
145
|
+
"fingerprint": "sha256(policy-version + kind + stable-identity)",
|
|
146
|
+
"kind": "reviewer_missing_seconder",
|
|
147
|
+
"bead_id": "...",
|
|
148
|
+
"chain_id": "...",
|
|
149
|
+
"first_seen_ms": 0,
|
|
150
|
+
"last_seen_ms": 0,
|
|
151
|
+
"resolved_at_ms": null
|
|
152
|
+
}
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Volatile details such as current dirty-file count are evidence fields, not fingerprint identity. Re-evaluation updates the existing finding; remediation resolves it. The same finding becomes available to forensic queries, Console materialization and eval backfill.
|
|
156
|
+
|
|
157
|
+
### 0.3.4 Rollout and evaluation
|
|
158
|
+
|
|
159
|
+
Every new policy surface supports:
|
|
160
|
+
|
|
161
|
+
```text
|
|
162
|
+
off → shadow → warn → enforce
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
Shadow mode evaluates and persists the decision without blocking or mutating the authoritative flow. Promotion requires measured precision, false-positive rate, latency and parity across Claude, Pi and runtime invocation.
|
|
166
|
+
|
|
167
|
+
Required measurements include:
|
|
168
|
+
|
|
169
|
+
- no-match hook latency;
|
|
170
|
+
- matched hook latency;
|
|
171
|
+
- adapter/kernel parity;
|
|
172
|
+
- false positive and override rates;
|
|
173
|
+
- finding recurrence and resolution rate;
|
|
174
|
+
- divergence between hook advice and authoritative runtime decision;
|
|
175
|
+
- missed unsafe dispatches;
|
|
176
|
+
- duplicate-mutation/idempotency behavior after retries or crashes.
|
|
177
|
+
|
|
178
|
+
### 0.3.5 Correlation and identity
|
|
179
|
+
|
|
180
|
+
Every decision should be correlatable to the available execution identities:
|
|
181
|
+
|
|
182
|
+
```text
|
|
183
|
+
policy_decision_id
|
|
184
|
+
command_run_id / tool_call_id
|
|
185
|
+
bead_id
|
|
186
|
+
chain_id
|
|
187
|
+
job_id or interactive activation/session identity
|
|
188
|
+
repository/worktree
|
|
189
|
+
candidate/config fingerprint
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
The kernel must not infer durable participant identity from free-form message bodies or recycled tmux display names.
|
|
193
|
+
|
|
194
|
+
### 0.3.6 Directly reusable xtmux behavior
|
|
195
|
+
|
|
196
|
+
The following current behavior should be reused rather than recreated:
|
|
197
|
+
|
|
198
|
+
- auto-monitor after `message-send`, `safe-send-pointer` or equivalent handoff;
|
|
199
|
+
- agent-visible confirmation that monitoring was armed;
|
|
200
|
+
- message/receipt/status queries for short coordination obligations;
|
|
201
|
+
- delivery-attempt evidence for pointer injection;
|
|
202
|
+
- idempotent monitor registration and crash reconciliation;
|
|
203
|
+
- safe no-match filtering and runtime-specific adapters;
|
|
204
|
+
- shadow/differential validation techniques.
|
|
205
|
+
|
|
206
|
+
### 0.3.7 Boundaries that remain separate
|
|
207
|
+
|
|
208
|
+
The Specialists roadmap must not copy the wrong ownership semantics from xtmux:
|
|
209
|
+
|
|
210
|
+
- Specialists `observability.db` remains authoritative for managed jobs, chain shape, gate evidence, `step_completed` and evals.
|
|
211
|
+
- xtmux `observability.db` remains authoritative for local interactive session state, short messages, receipts, delivery attempts, monitors and handoffs.
|
|
212
|
+
- `agent.turn.done` is not `step_completed` and must never advance a chain.
|
|
213
|
+
- an xtmux acknowledgement is not a gate verdict or merge authorization.
|
|
214
|
+
- tmux session/pane identity is not a substitute for Specialists participant/job/event identity.
|
|
215
|
+
- xtmux remains a local attention and transport runtime, not a second chain scheduler and not an early implementation of Channels authority.
|
|
216
|
+
|
|
217
|
+
### 0.3.8 Immediate roadmap effect
|
|
218
|
+
|
|
219
|
+
This foundation changes the first roadmap patch itself:
|
|
220
|
+
|
|
221
|
+
1. `WP-G00` must replace the Claude-only hook description with the shared-kernel/multi-adapter architecture.
|
|
222
|
+
2. `WP-G02` must audit existing bd, Core, xtmux hook and Specialists validator primitives before creating new mechanisms.
|
|
223
|
+
3. `WP-H01`–`WP-H05` are a standalone workstream, not children of the interactive coordinator work.
|
|
224
|
+
4. Interactive coordinator delivery later consumes the same xtmux monitor/handoff primitives, but does not own or motivate this hook architecture.
|
|
225
|
+
5. The hook work can begin without monorepo migration, Substrate, Channels or the interactive coordinator.
|
|
226
|
+
|
|
227
|
+
## 0.4 Repository, persistence and sequencing decisions
|
|
228
|
+
|
|
229
|
+
The following repository, persistence and sequencing decisions were reached after the foundational xtmux hook audit and before the interactive `chain-coordinator` work entered scope. They are normative for decomposition and must not be reopened implicitly by the coordinator implementation.
|
|
230
|
+
|
|
231
|
+
### 0.4.1 Repository and program ownership
|
|
232
|
+
|
|
233
|
+
1. **Start decomposition in `xtrm-dev/specialists`.** The live runtime, mandatory rules, chain templates, per-repository observability store and canonical roadmap are there. The program-level PRD may be indexed from `xtrm`, but implementation Beads belong to the repository that owns the affected files.
|
|
234
|
+
2. **Use cross-repository workstreams rather than one fictitious cross-repo Beads tree.** The PRD owns the global hierarchy and dependency map. Specialists, Core, xtmux and Console own their implementation Beads and integration contracts.
|
|
235
|
+
3. **The first assigned work remains the documentation/reuse gate:** `WP-G00` through `WP-G03`. This reconciles the roadmap, marks the June reconciliation as historical, audits reusable Beads primitives and registers acceptance-criterion coverage before high-conflict runtime changes.
|
|
236
|
+
4. **Do not wait for Substrate.** The Specialists roadmap is a long-lived bridge and must be implemented as a reliable runtime, not as disposable scaffolding expected to survive only a few weeks.
|
|
237
|
+
|
|
238
|
+
### 0.4.2 Monorepo decision for this runway
|
|
239
|
+
|
|
240
|
+
The monorepo migration is deferred. No graft is a prerequisite for this PRD. The working repository shape remains:
|
|
241
|
+
|
|
242
|
+
```text
|
|
243
|
+
specialists autonomous runtime repository
|
|
244
|
+
core autonomous launcher/install repository
|
|
245
|
+
xtmux autonomous local attention and transport repository
|
|
246
|
+
Console autonomous Omniforge/materializer repository
|
|
247
|
+
xtrm umbrella architecture and reconciliation repository
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
A future runtime monorepo containing `core + channels + substrate + specialists` may still be justified because those packages will share protocol, schema and integration tests. Absorbing the full Console/Omniforge workspace is not currently justified; Console should remain an external consumer until a later architecture decision.
|
|
251
|
+
|
|
252
|
+
The reason for deferral is not that Git worktrees fail to isolate uncommitted changes. Each worktree has its own working tree, index and branch, so dirty state in one worktree is not visible in another. The unresolved risks are instead:
|
|
253
|
+
|
|
254
|
+
- shared root files and lockfiles;
|
|
255
|
+
- package-level write-boundary enforcement;
|
|
256
|
+
- merge contention across branches that represent the whole product;
|
|
257
|
+
- a shared daemon/socket/database when branches change incompatible schemas or protocols;
|
|
258
|
+
- generated mirrors and registry files;
|
|
259
|
+
- CI and release coupling;
|
|
260
|
+
- the fact that Console is already a separate monorepo with its own deployment and materializer lifecycle.
|
|
261
|
+
|
|
262
|
+
Before any future graft, a dedicated monorepo readiness decision must prove the following operating model:
|
|
263
|
+
|
|
264
|
+
| Invariant | Required behavior |
|
|
265
|
+
|---|---|
|
|
266
|
+
| Worktree ownership | one chain/root Bead owns one worktree |
|
|
267
|
+
| Writer lease | at most one write-capable agent operates in a worktree at a time; read-only participants may share it |
|
|
268
|
+
| Package write scope | every Bead declares allowed write paths; dependency closure is readable but not automatically writable |
|
|
269
|
+
| Root mutation | lockfile, workspace config, CI, schema registry and generated-root changes use a serialized merge slot |
|
|
270
|
+
| Staging | path-explicit staging is the default; agents do not use unbounded `git add -A` |
|
|
271
|
+
| Cross-package changes | represented as one atomic chain with explicit multi-package scope, not accidental scope expansion |
|
|
272
|
+
| Runtime isolation | schema/protocol work can use a worktree-specific daemon, socket and database namespace |
|
|
273
|
+
| Cache isolation | non-content-addressed caches are namespaced by branch/worktree |
|
|
274
|
+
| CI | affected packages plus reverse dependants are tested |
|
|
275
|
+
| Merge | root mutations use rebase/merge queue discipline and a single integration owner |
|
|
276
|
+
|
|
277
|
+
Sparse checkout may reduce disk use for leaf packages, but it is not a security or ownership boundary. In particular, Specialists' dependency closure would include most of the runtime and therefore yield little isolation.
|
|
278
|
+
|
|
279
|
+
### 0.4.3 Current persistence and Console boundary
|
|
280
|
+
|
|
281
|
+
During the bridge runway:
|
|
282
|
+
|
|
283
|
+
```text
|
|
284
|
+
Specialists managed jobs/chains/evals
|
|
285
|
+
-> per-repository .specialists/db/observability.db
|
|
286
|
+
|
|
287
|
+
Interactive tmux sessions/messages/receipts/monitors
|
|
288
|
+
-> xtmux observability.db
|
|
289
|
+
|
|
290
|
+
Console
|
|
291
|
+
-> its existing materializer and read model
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
Console is expanded only after Specialists telemetry, forensic and eval contracts stabilize. It must not become the source of truth for evaluation or read directly from arbitrary runtime files. Prometheus remains the aggregate projection; the materialized read model carries high-cardinality drilldown and evidence references.
|
|
295
|
+
|
|
296
|
+
### 0.4.4 Sequencing fixed before coordinator scope
|
|
297
|
+
|
|
298
|
+
The pre-coordinator critical path was and remains:
|
|
299
|
+
|
|
300
|
+
```text
|
|
301
|
+
roadmap/reconciliation/reuse gates
|
|
302
|
+
-> telemetry correctness and immutable baseline
|
|
303
|
+
-> eval core and historical backfill
|
|
304
|
+
-> critical chain spine
|
|
305
|
+
-> output/runner consolidation
|
|
306
|
+
-> role prompt experiments
|
|
307
|
+
-> complete chain context and handoff
|
|
308
|
+
-> pull-based memory
|
|
309
|
+
-> Console materializer/API/UI expansion
|
|
310
|
+
-> later Substrate/Channels replacement
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
The interactive coordinator bridge is inserted into this program; it does not replace or reorder the telemetry baseline, eval foundation or chain-spine prerequisites.
|
|
314
|
+
|
|
315
|
+
|
|
316
|
+
## 0.5 Beads v1.1 and post-v1.1 capability integration
|
|
317
|
+
|
|
318
|
+
A repository audit performed on 13 July 2026 found that Beads now owns more of the graph, ready-queue, claim and workflow mechanics than the earlier Specialists roadmap assumed. The bridge program must therefore prefer thin Specialists orchestration over duplicate graph/scheduler logic.
|
|
319
|
+
|
|
320
|
+
### 0.5.1 Adoption classes
|
|
321
|
+
|
|
322
|
+
| Capability | Availability at audit snapshot | PRD decision |
|
|
323
|
+
|---|---|---|
|
|
324
|
+
| `bd ready --mol`, `--gated`, `--claim`, `--explain`, parent/label/metadata filters | stable v1.1 CLI | adopt as the blocker-aware readiness source used by `sp chain` and coordinators |
|
|
325
|
+
| `bd create --graph <plan.json> --dry-run` with symbolic nodes, typed edges, metadata, cycle validation and atomic creation | stable v1.1 CLI | make the preferred Pass-1/Pass-2 backlog materialization path |
|
|
326
|
+
| `--waits-for` with `all-children` / `any-children` gates | stable CLI | reuse for deterministic fan-in; do not confuse with semantic judge/Channels consensus |
|
|
327
|
+
| `--spec-id`, `--skills`, `--validate`, `--metadata`, `--external-ref`, `--mol-type` | stable CLI | map PRD/WP metadata into native Bead fields where possible |
|
|
328
|
+
| `bd init --init-if-missing` and idempotent init/doctor behavior | available in current code line | use in bootstrap after minimum-version verification |
|
|
329
|
+
| migration content hashes, designated migrator, smart remote migration/adoption and recovery guidance | stable v1.1 line | add one explicit multi-clone upgrade runbook; agents do not migrate independently |
|
|
330
|
+
| `bd memories [query]`, `bd recall`, guarded `bd remember` ergonomics | stable/current | reuse before adding a Specialists memory wrapper |
|
|
331
|
+
| claim TTL, `bd heartbeat`, `bd reclaim`, owner-safe unclaim | post-v1.1 `main` / unreleased | design integration now; activate only after released minimum version or pinned compatibility validation |
|
|
332
|
+
| `bd formula schema --json`, discoverable verified primitives and smoke fixtures | post-v1.1 `main` / unreleased | use for formula authoring/validation after release or pin; never rely on declared-but-unwired primitives |
|
|
333
|
+
| `bd history <id> --events` and issue-scoped `bd gate list <id>` | post-v1.1 `main` / unreleased | add as future evidence/query surfaces behind capability checks |
|
|
334
|
+
| PostgreSQL, MySQL and SQLite storage backends behind a conformance seam | post-v1.1 `main` / unreleased | record as architecture watch item; retain Dolt for this bridge runway |
|
|
335
|
+
| differential conformance/oracle catalog | post-v1.1 test infrastructure | reuse the methodology for Specialists telemetry, CLI and runtime refactors |
|
|
336
|
+
|
|
337
|
+
### 0.5.2 Normative consequences
|
|
338
|
+
|
|
339
|
+
1. **Beads is authoritative for dependency readiness.** Specialists may persist resolved chain shape, job linkage, evidence and UI projections, but it must not maintain a competing blocker-resolution engine. `sp chain` combines its chain/job state with `bd ready --mol`, `bd ready --gated` and `bd gate` results.
|
|
340
|
+
2. **Atomic graph creation becomes the default decomposition handoff.** The planning agent emits a versioned `bead-graph-plan.json`; the operator runs `bd create --graph ... --dry-run`; composition/reviewer validates the preview; only then is the graph applied atomically.
|
|
341
|
+
3. **Structural fan-in uses native gates.** `all-children` and `any-children` may coordinate deterministic completion. They do not provide semantic review, release authority, quorum judgment or Channels behavior.
|
|
342
|
+
4. **Claim ownership and worktree ownership remain distinct.** Native claim leases describe who owns a Bead contract. `merge-slot`/worktree lease describes who may write or integrate in a checkout. Neither replaces the other.
|
|
343
|
+
5. **`step_completed` shrinks further.** It records visibility, evidence and recommendation only. It neither computes a parallel readiness DAG nor auto-resumes peers.
|
|
344
|
+
6. **Memory work starts with a gap analysis.** A Specialists wrapper is justified only for missing provenance, supersession, ranking, token budgets, memory types or telemetry—not for basic search/recall already provided by Beads.
|
|
345
|
+
7. **Dolt remains the bridge backend.** Alternative backends are not adopted during this program because history, time travel, merge and remote sync remain useful. A later ADR may reconsider this separately.
|
|
346
|
+
8. **Minimum-version and capability checks are explicit.** Stable features may be required by version; unreleased features require a pinned binary/commit and compatibility suite. No agent assumes `main` behavior merely because the design mentions it.
|
|
347
|
+
9. **Beads upgrades are serialized.** For remote-backed/multi-clone databases, one designated migrator performs schema migration after backup and synchronization; other clones adopt the migrated state.
|
|
348
|
+
10. **Differential testing is preferred for contract-preserving refactors.** Capture current canonical behavior, execute the same scenario corpus against the candidate, canonicalize volatile fields and review only intentional deltas.
|
|
349
|
+
|
|
350
|
+
### 0.5.3 Capability snapshot required before decomposition
|
|
351
|
+
|
|
352
|
+
`WP-G02` must capture, for the locally installed binary:
|
|
353
|
+
|
|
354
|
+
```text
|
|
355
|
+
bd version
|
|
356
|
+
bd ready --help
|
|
357
|
+
bd create --help
|
|
358
|
+
bd formula schema --json # when supported
|
|
359
|
+
bd gate list --help
|
|
360
|
+
bd history --help
|
|
361
|
+
bd memories --help
|
|
362
|
+
bd migrate --help
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
The resulting capability matrix records `supported | unsupported | behavior-needs-verification`, minimum version/commit and fallback behavior. It becomes an input to every Beads-dependent root Bead.
|
|
366
|
+
|
|
367
|
+
## Document purpose
|
|
368
|
+
|
|
369
|
+
This PRD consolidates the complete set of findings and recommendations produced during a detailed audit of the Specialists system. It begins with the original request to reduce the static context payload of six heavy specialist system prompts and expands the scope where the investigation showed that prompt text was only one layer of a larger runtime-composition problem.
|
|
370
|
+
|
|
371
|
+
The document is intentionally comprehensive. It records not only the final recommendations, but also the corrections and refinements made during the investigation. In particular, the first prompt audit correctly identified substantial duplication and role overlap, but initially did not account for every mandatory-rule layer, runtime-injected block, local override, eager skill body or generated output contract. Subsequent repository inspection established the full package-canonical prompt composition and revealed additional sources of context bloat, policy conflict and observability ambiguity. This PRD incorporates that corrected understanding.
|
|
372
|
+
|
|
373
|
+
## Document authority and execution contract
|
|
374
|
+
|
|
375
|
+
This PRD is the implementation and decomposition source of truth for the prompt, chain-context, memory, telemetry and evaluation modernization program. It is intentionally more operational than `specialists-roadmap.md`: the roadmap remains authoritative for the bridge-runtime architecture and its Opportunities, while this PRD defines the concrete work packages, evaluation gates, parallelization boundaries and promotion criteria needed to implement the overlapping modernization safely.
|
|
376
|
+
|
|
377
|
+
The following precedence applies when local agents encounter apparently conflicting prose:
|
|
378
|
+
|
|
379
|
+
1. **Current code and executable schemas** define what exists today. Agents must verify before modifying.
|
|
380
|
+
2. **The installed Beads executable and its machine-readable capability surfaces** define available graph, gate, claim and formula primitives. The decomposition must record its version/capability snapshot.
|
|
381
|
+
3. **`specialists-roadmap.md`** defines the canonical bridge direction, retained Opportunities, Substrate read-forward and Beads/molecule mental model.
|
|
382
|
+
4. **This PRD** defines implementation scope, sequencing, grader requirements, work-package boundaries and readiness gates for the modernization program.
|
|
383
|
+
5. **Current Core and xtmux public contracts** define the shipped interactive-role launch, tmux identity, message, receipt, handoff, monitor and delivery behavior used during the bridge runway.
|
|
384
|
+
6. **`xtrm/docs/channels/channels.md`** defines the future semantic messaging contract. No bridge implementation may grow xtmux into a competing Channels runtime.
|
|
385
|
+
7. **Dated `_meta` reconciliation documents** are historical snapshots. They inform drift analysis but do not override newer code, roadmap status or this PRD.
|
|
386
|
+
8. **Archived Iron/friction documents** are design archaeology only unless the canonical roadmap explicitly retains a requirement.
|
|
387
|
+
|
|
388
|
+
This document is ready to be supplied to a local planning agent for Pass-1/Pass-2 decomposition. The planning agent must create Beads work contracts and dependency edges; it must not silently redesign the architecture, duplicate normative text into every Bead, or start implementation during decomposition. The decomposition output must preserve the work-package identifiers and dependency constraints defined in Sections 14 and 15.
|
|
389
|
+
|
|
390
|
+
The implementation program is split into five coordinated execution tracks:
|
|
391
|
+
|
|
392
|
+
- **Track A — Specialists bridge runtime:** the critical chain-first spine and remaining roadmap work on the live runtime.
|
|
393
|
+
- **Track B — xtrm Stage 0, Substrate and Channels:** daemon/state-store and semantic messaging work owned by xtrm, not reimplemented ad hoc in Specialists.
|
|
394
|
+
- **Track C — Prompt, context, telemetry and eval modernization:** the original modernization program defined by this PRD.
|
|
395
|
+
- **Track D — Cross-harness policy hooks and xtmux attention architecture:** shared policy kernel, Claude/Pi adapters, durable findings, shadow-first rollout, monitor/handoff reuse and authoritative Specialists validation.
|
|
396
|
+
- **Track E — Interactive coordinator bridge:** progressive-disclosure skills, first-user-prompt composition, preheated/direct assignment modes, coordinator doctrine, cross-coordinator hygiene and current attention/messaging integration.
|
|
397
|
+
|
|
398
|
+
Tracks C, D and E start immediately with telemetry integrity, historical baselining, content audit, the policy-kernel reuse audit and the interactive-role parity contract. Numbered Workstreams A–E later in this document describe capability domains and are not the same labels as these execution tracks. Their fully chain-aware portions consume Track A foundations. Track B is intentionally delayed and replaces bridge mechanisms only when Stage 0, Substrate and Channels are real.
|
|
399
|
+
|
|
400
|
+
The required result is not merely shorter prompts. The required result is a measurable, versioned and chain-aware agent runtime in which:
|
|
401
|
+
|
|
402
|
+
1. Each specialist has a small, stable role identity and a clear decision boundary.
|
|
403
|
+
2. Shared policies have one source of truth and are injected only when applicable.
|
|
404
|
+
3. Chain membership, upstream evidence and downstream obligations are explicit.
|
|
405
|
+
4. Historical memories are retrieved deliberately rather than pushed indiscriminately.
|
|
406
|
+
5. Runtime telemetry accurately represents thinking, tools, turns, payload, evidence and chain state.
|
|
407
|
+
6. Existing and future runs can be evaluated using deterministic, model-based and human graders.
|
|
408
|
+
7. Prompt and model changes are promoted through controlled paired experiments rather than intuition.
|
|
409
|
+
8. Console provides historical, live and experiment-specific evaluation surfaces without becoming a second telemetry producer.
|
|
410
|
+
9. Interactive role sessions receive current task context and mandatory rules as a first user assignment, never buried inside the long-lived system prompt.
|
|
411
|
+
10. The interactive coordinator applies future chain-coordinator judgment duties through current Specialists, Beads and xtmux primitives without pretending that xtmux is Substrate or Channels.
|
|
412
|
+
|
|
413
|
+
# 1. Executive summary
|
|
414
|
+
|
|
415
|
+
The Specialists architecture already has strong primitives: role-specific agents, Beads contracts, worktree isolation, mandatory gates, structured output schemas, `xtrm.forensic.v1`, per-repository observability databases, Prometheus projection and a Console materializer. The principal problem is that these primitives evolved in parallel and currently overlap in ways that increase context size and weaken semantic clarity.
|
|
416
|
+
|
|
417
|
+
The six original audited specialists carry large system prompts, in some cases more than 10,000 tokens once all layers are counted. The largest source is not always the role prompt itself. A non-`bare` specialist can receive all of the following:
|
|
418
|
+
|
|
419
|
+
- its `prompt.system`;
|
|
420
|
+
- a task template and Bead context;
|
|
421
|
+
- completed dependency descriptions and notes;
|
|
422
|
+
- required and default mandatory rules;
|
|
423
|
+
- specialist-specific mandatory rules and inline rules;
|
|
424
|
+
- eager skill bodies passed through Pi;
|
|
425
|
+
- a runtime Specialist Run Context block;
|
|
426
|
+
- a global output-style directive;
|
|
427
|
+
- a hardcoded GitNexus workflow mandate;
|
|
428
|
+
- static Beads workflow and close instructions;
|
|
429
|
+
- filtered memory injection;
|
|
430
|
+
- an optional GitNexus pre-query snapshot;
|
|
431
|
+
- a generated output contract based on response format, output type and output schema;
|
|
432
|
+
- reviewer-specific diff and lineage instructions;
|
|
433
|
+
- tool definitions and harness framing.
|
|
434
|
+
|
|
435
|
+
This composition creates four classes of defect.
|
|
436
|
+
|
|
437
|
+
**Redundancy:** The same policy is expressed in a system prompt, mandatory rule, task template and runner injection. Executor receives multiple GitNexus mandates, multiple scope rules and multiple handoff requirements. Reviewer repeats seconder work and restates an output format already generated by the runner. Test-engineer repeats its source boundary in inline rules, system prompt and task template.
|
|
438
|
+
|
|
439
|
+
**Contradiction:** The generated output contract, handoff-schema rule and role-specific output prose use incompatible status and verdict vocabularies. Global worktree and close rules are injected into read-only or external-research roles where they are irrelevant or conflicting. A global “stop if local evidence is missing” rule conflicts with researcher’s external-evidence mandate.
|
|
440
|
+
|
|
441
|
+
**Missing chain identity:** The architecture defines specialists as members of a chain, but the role prompts mostly treat them as isolated agents. A reviewer may need executor, seconder, test-engineer, test-runner, obligations and security evidence, yet dependency preloading is bounded and does not guarantee the full upstream chain. The agent is not deterministically told its chain, position, root contract, upstream members, completed gates or downstream handoff.
|
|
442
|
+
|
|
443
|
+
**Insufficient evaluation discipline:** Observability captures substantial raw data, but the system lacks a first-class evaluation control plane. It cannot yet reliably answer whether one prompt or model is better for a role, whether chain-awareness improved outcomes, whether memory retrieval was necessary, or whether a shorter prompt changed false-PASS rates. Some current metrics are also unsuitable for A/B decisions until corrected, notably thinking duration and tool-call counting.
|
|
444
|
+
|
|
445
|
+
The modernization is therefore organized into five tightly related programs:
|
|
446
|
+
|
|
447
|
+
1. **Prompt and policy consolidation.** Reduce the six core role prompts by approximately 45–90%, depending on role, while preserving role-specific capability. Move deterministic policies to runtime, shared behavior to mandatory rules and deep procedures to on-demand skills.
|
|
448
|
+
2. **Chain participant context.** Inject a compact, deterministic chain identity and pointer block. Teach specialists how to reconstruct only the necessary upstream context using Beads and Specialists commands. Extend handoffs with decisions, evidence, assumptions and downstream attention.
|
|
449
|
+
3. **Pull-based memory.** Remove eager semantic memory dumps. Provide a short retrieval rule and targeted `bd memories`/`bd recall` workflow, with provenance and observability.
|
|
450
|
+
4. **Telemetry hardening.** Correct tool accounting, model thinking spans, forensic-to-Prometheus wiring and activity semantics. Add metrics required for role, chain and evaluation analysis.
|
|
451
|
+
5. **Evaluation platform.** Add eval suites, cases, experiments, trials, graders, scores and pairwise comparisons. Support retrospective scoring of existing per-repo databases, automatic post-run and post-chain deterministic evaluation, controlled replay benchmarks and Console visualization.
|
|
452
|
+
6. **Interactive coordinator bridge.** Refactor `using-specialists` for progressive disclosure; inject current mandatory rules and tracked Bead context as a first user assignment; support preheated standby and direct `--bead` launch; teach the coordinator canonical chain templates, scope review, collision management, cross-coordinator mechanics, close-time evidence and escalation.
|
|
453
|
+
|
|
454
|
+
The recommended implementation order is telemetry integrity and immutable baselines first; in parallel, audit and build the interactive assignment seam; then land the critical chain spine; then run prompt, memory and coordinator experiments. This prevents decisions from being made with ambiguous metrics while still removing immediate bridge friction.
|
|
455
|
+
|
|
456
|
+
# 2. Background and verified current-state architecture
|
|
457
|
+
|
|
458
|
+
## 2.1 Specialist roles in scope
|
|
459
|
+
|
|
460
|
+
The original prompt-modernization wave covers six managed specialists:
|
|
461
|
+
|
|
462
|
+
| Role | Intended responsibility | Current strategic position |
|
|
463
|
+
|---|---|---|
|
|
464
|
+
| `executor` | Implement the Bead contract in the assigned worktree | Primary production writer |
|
|
465
|
+
| `seconder` | Cheap bounded pre-QA gate for scope compliance and implementation sanity | Immediately after writer |
|
|
466
|
+
| `test-engineer` | Author tests, fixtures, smoke/E2E assets and telemetry assertions from the actual diff | Between seconder and test-runner |
|
|
467
|
+
| `reviewer` | Final evidence-based release verdict and Release Checklist | Final gate |
|
|
468
|
+
| `researcher` | Gather current external evidence and synthesize findings | Advisor or standalone investigation |
|
|
469
|
+
| `overthinker` | Stress-test uncertain or high-impact decisions | Advisor, premortem or standalone decision review |
|
|
470
|
+
|
|
471
|
+
The role separation is fundamentally sound and should not be collapsed. The modernization should make these boundaries clearer, not replace them with a general-purpose super-agent.
|
|
472
|
+
|
|
473
|
+
## 2.2 Canonical production-diff chain
|
|
474
|
+
|
|
475
|
+
The current design canon defines the production pipeline as:
|
|
476
|
+
|
|
477
|
+
```text
|
|
478
|
+
writer (executor or debugger)
|
|
479
|
+
-> seconder
|
|
480
|
+
-> test-engineer
|
|
481
|
+
-> test-runner
|
|
482
|
+
-> security-auditor when required
|
|
483
|
+
-> obligations-scanner
|
|
484
|
+
-> reviewer
|
|
485
|
+
-> release decision
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
The formula catalog represents a chain as a Beads molecule. The molecule is the chain identity. It contains a root child carrying the overall change contract and step children carrying role-specific mandates. `needs` creates blocking relationships; labels can carry richer semantic edges such as `validates`, `informs`, `implements`, `discovered-from` and `parent-child`.
|
|
489
|
+
|
|
490
|
+
The `code-standard` formula explicitly states that reviewer inputs include the executor diff, seconder verdict, test evidence, obligations output and root contract. The `code-with-advisors` formula adds explorer, researcher and overthinker outputs before executor and expects the final reviewer to consume advisor findings as well.
|
|
491
|
+
|
|
492
|
+
## 2.3 Prompt composition at runtime
|
|
493
|
+
|
|
494
|
+
The effective specialist prompt is not equivalent to `prompt.system`. It is composed from multiple sources.
|
|
495
|
+
|
|
496
|
+
### System-side content
|
|
497
|
+
|
|
498
|
+
The Pi session receives:
|
|
499
|
+
|
|
500
|
+
1. Rendered `prompt.system`.
|
|
501
|
+
2. A Specialist Run Context block that distinguishes the specialist from a human developer and controls Bead lifecycle behavior.
|
|
502
|
+
3. A global output-style directive.
|
|
503
|
+
4. A GitNexus workflow mandate when the repository is indexed.
|
|
504
|
+
5. Static Beads workflow rules and a close checklist.
|
|
505
|
+
6. Filtered Beads memories when a Bead is present.
|
|
506
|
+
7. An optional GitNexus pre-query snapshot.
|
|
507
|
+
8. A generated output contract derived from `response_format`, `output_type` and `prompt.output_schema`.
|
|
508
|
+
9. Pi-native skill files from `prompt.skill_inherit` and `skills.paths`.
|
|
509
|
+
10. Tool and extension definitions resolved from the manifest catalog and permission tier.
|
|
510
|
+
|
|
511
|
+
### Task-side content
|
|
512
|
+
|
|
513
|
+
The task prompt receives:
|
|
514
|
+
|
|
515
|
+
1. The rendered role task template.
|
|
516
|
+
2. The root or step Bead contract.
|
|
517
|
+
3. Notes attached to the current Bead.
|
|
518
|
+
4. Completed blocking dependencies, recursively up to the configured context depth.
|
|
519
|
+
5. Runtime worktree boundary rules.
|
|
520
|
+
6. Required, default, specialist-specific and inline mandatory rules.
|
|
521
|
+
7. Reviewer-specific injected diff context where applicable.
|
|
522
|
+
8. Pre-script output and invocation variables.
|
|
523
|
+
|
|
524
|
+
This distinction matters for optimization. A small `prompt.system` does not guarantee a small first-turn input. The API-billed first-turn input must remain the source of truth, while payload component events are used to identify which layer contributes the bloat.
|
|
525
|
+
|
|
526
|
+
## 2.4 Mandatory-rule resolution
|
|
527
|
+
|
|
528
|
+
The canonical rule loader combines:
|
|
529
|
+
|
|
530
|
+
1. required template sets;
|
|
531
|
+
2. default template sets unless default globals are disabled;
|
|
532
|
+
3. specialist-declared template sets;
|
|
533
|
+
4. specialist inline rules.
|
|
534
|
+
|
|
535
|
+
The package-level index currently requires `core-session-boundary` and defaults to `git-workflow-safe`. Specialist-specific rules then add executor delivery, code-quality defaults, GitNexus and Serena instructions, handoff schema, Bead ID discipline, research routing and other role-specific behavior.
|
|
536
|
+
|
|
537
|
+
The package-canonical rules can be audited from the repository. The effective installation may still differ because user or repository overlays and global user configuration can override or append content. Therefore all implementation and experiment tooling must fingerprint and capture the **resolved** configuration, not assume package defaults.
|
|
538
|
+
|
|
539
|
+
The authoritative inspection path is conceptually:
|
|
540
|
+
|
|
541
|
+
```bash
|
|
542
|
+
sp config show <specialist> --resolved
|
|
543
|
+
sp list-rules --specialist <specialist> --json
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
## 2.5 Skills
|
|
547
|
+
|
|
548
|
+
Among the six package-canonical specialist files inspected, `test-engineer` declares `test-planning`; the other five do not declare explicit skill paths. The corresponding package-canonical skill body was not present at the expected repository path during the audit. It may resolve from an installation/user layer or be stale. This is a concrete reason that eval fingerprints must capture the activated skill path and body hash at runtime.
|
|
549
|
+
|
|
550
|
+
Skill bodies are eagerly injected by Pi and billed as input tokens, while the current payload breakdown records only skill labels or paths. Their real contribution must be derived from first-turn token use or measured directly by hashing and tokenizing the activated files.
|
|
551
|
+
|
|
552
|
+
## 2.6 Bead dependency context
|
|
553
|
+
|
|
554
|
+
`buildBeadContext` includes the current Bead title, description, parent and notes. It also includes descriptions and notes of completed blocking dependencies. Dependency traversal is recursive and bounded by a configured depth, currently defaulting to three in the runner path.
|
|
555
|
+
|
|
556
|
+
This is useful but insufficient as a chain context contract:
|
|
557
|
+
|
|
558
|
+
- it follows only completed `blocks` dependencies, not all semantic edges;
|
|
559
|
+
- a final reviewer in a long chain may not receive executor or seconder evidence within depth three;
|
|
560
|
+
- security gates inserted outside the simple blocking path may be omitted;
|
|
561
|
+
- the current parent label can describe a chain molecule as a “Parent epic”, conflating hierarchy levels;
|
|
562
|
+
- the agent is not told whether the preloaded window is complete.
|
|
563
|
+
|
|
564
|
+
## 2.7 Observability storage
|
|
565
|
+
|
|
566
|
+
Each repository has an observability database. Core tables include:
|
|
567
|
+
|
|
568
|
+
- `specialist_jobs` for durable run state and startup payload;
|
|
569
|
+
- `specialist_events` for canonical timeline events;
|
|
570
|
+
- `specialist_results` for final or latest completed output;
|
|
571
|
+
- `specialist_job_metrics` for aggregated run metrics;
|
|
572
|
+
- `specialist_forensic_events` for `xtrm.forensic.v1` envelopes;
|
|
573
|
+
- chain and epic linkage tables;
|
|
574
|
+
- node run, member, event and memory tables;
|
|
575
|
+
- memory cache and FTS tables.
|
|
576
|
+
|
|
577
|
+
`specialist_job_metrics` persists identity, timestamps, active and waiting time, turns, tools, token trajectory, context trajectory, stall gaps, run-complete data and startup payload.
|
|
578
|
+
|
|
579
|
+
Historical job directories can be imported. Aggregate metrics can be recomputed from persisted timeline events. This makes retrospective evaluation feasible for a large subset of dimensions.
|
|
580
|
+
|
|
581
|
+
## 2.8 Forensic event model
|
|
582
|
+
|
|
583
|
+
`xtrm.forensic.v1` is already suitable as the universal event envelope. It supports:
|
|
584
|
+
|
|
585
|
+
- resource identity, including participant role and model;
|
|
586
|
+
- high-cardinality correlation, including job, Bead, chain, turn, tool call, trace, commit and eval IDs;
|
|
587
|
+
- structured event body;
|
|
588
|
+
- redaction state and rules;
|
|
589
|
+
- optional trace, OTel, links and diagnostics.
|
|
590
|
+
|
|
591
|
+
The schema deliberately forbids high-cardinality and sensitive values from Prometheus labels. Detailed evidence stays in forensic storage and evidence references, while Prometheus remains a bounded aggregate projection.
|
|
592
|
+
|
|
593
|
+
## 2.9 Current Prometheus projection
|
|
594
|
+
|
|
595
|
+
The current projection emits operational metrics for job state, queue depth, processes, worktrees, terminal job count, job duration, active runtime, chain count and duration, waiting time, turns, context utilization, tool calls and token direction.
|
|
596
|
+
|
|
597
|
+
The renderer also contains support for forensic-derived metrics such as gate verdicts, evidence references, MCP operations, identity, policy and eval runs/scores. However, the normal `sp metrics --prometheus` path currently reads statuses and job metrics but does not supply forensic events to the renderer. Those metric families are therefore not reliably present in the live scrape despite being implemented in the rendering layer.
|
|
598
|
+
|
|
599
|
+
## 2.10 Console architecture
|
|
600
|
+
|
|
601
|
+
Console already has the correct ownership boundary for this project:
|
|
602
|
+
|
|
603
|
+
- Specialists owns runtime events, forensic envelopes, projections, evaluation semantics and evidence production.
|
|
604
|
+
- The materializer reads per-repo Specialists databases and projects job rows, token metrics, forensic events and evidence references into its read model.
|
|
605
|
+
- Console owns queries, dashboards, drilldowns, experiment UX and operator decisions.
|
|
606
|
+
- Prometheus is used for aggregate operational symptoms; forensic/evidence storage is used for exact run and trial detail.
|
|
607
|
+
|
|
608
|
+
The Console observability design already includes an `eval` signal kind, eval lookup queries, eval result summaries and an AgentOps Governance dashboard pack. A new standalone application is therefore not required.
|
|
609
|
+
|
|
610
|
+
|
|
611
|
+
## 2.11 Shipped interactive role launcher
|
|
612
|
+
|
|
613
|
+
`xt pi --role <name>` and `xt claude --role <name>` are current production surfaces. They:
|
|
614
|
+
|
|
615
|
+
- resolve the effective specialist through `sp view <name> --raw`;
|
|
616
|
+
- create a dedicated worktree and branch;
|
|
617
|
+
- launch Pi or Claude in the current pane or a new tmux session;
|
|
618
|
+
- preserve role, parent, prompt-file and optional Bead metadata through pane options and `XTMUX_AGENT_*` environment variables;
|
|
619
|
+
- emit an `agent.role.launched` event;
|
|
620
|
+
- support detached orchestration, runtime-specific names, collision suffixing/reuse, model overrides and Pi thinking overrides.
|
|
621
|
+
|
|
622
|
+
The launcher currently resolves stable role identity and skills but does not reproduce the managed `sp run` task-side assembly. In particular, passing `--bead` currently supplies identity metadata but does not itself guarantee a first user turn containing the rendered Bead/dependency context and mandatory-rule block.
|
|
623
|
+
|
|
624
|
+
## 2.12 Interactive `chain-coordinator`
|
|
625
|
+
|
|
626
|
+
The shipped `chain-coordinator` is an interactive, persistent specialist persona scoped today to one epic or task group. It dispatches and monitors child `sp` chains, persists decisions in Beads and escalates judgment calls to a parent orchestrator through xtmux.
|
|
627
|
+
|
|
628
|
+
This role is retained under its current name during the bridge runway. The future Substrate design uses the same name for a transient coordinator scoped to one chain. The PRD therefore distinguishes:
|
|
629
|
+
|
|
630
|
+
```text
|
|
631
|
+
bridge interactive chain-coordinator
|
|
632
|
+
one epic/task-group
|
|
633
|
+
persistent Pi or Claude session
|
|
634
|
+
applies coordinator duties across child chains
|
|
635
|
+
|
|
636
|
+
future Substrate chain coordinator
|
|
637
|
+
one chain container
|
|
638
|
+
participant of that chain
|
|
639
|
+
spawned after composition and dies with the chain
|
|
640
|
+
```
|
|
641
|
+
|
|
642
|
+
The semantic overlap is deliberate: the bridge role should inherit the future coordinator's judgment duties where current primitives can support them. The lifecycle and authority model are not claimed to be identical.
|
|
643
|
+
|
|
644
|
+
## 2.13 xtmux SQLite attention and messaging runtime
|
|
645
|
+
|
|
646
|
+
xtmux has moved from flat JSONL as the primary runtime into a Bun/TypeScript SQLite implementation. SQLite V2 is default-on and owns typed operational domains for:
|
|
647
|
+
|
|
648
|
+
```text
|
|
649
|
+
messages and receipts
|
|
650
|
+
delivery attempts
|
|
651
|
+
agent instances, state transitions and completed turns
|
|
652
|
+
monitors, leases and terminal outcomes
|
|
653
|
+
handoff lifecycle
|
|
654
|
+
command telemetry
|
|
655
|
+
audit runs and durable findings
|
|
656
|
+
migration and retention
|
|
657
|
+
```
|
|
658
|
+
|
|
659
|
+
The generic journal remains a forensic and compatibility surface. Live tmux options remain reconstructible projections.
|
|
660
|
+
|
|
661
|
+
Current primitives relevant to this PRD include:
|
|
662
|
+
|
|
663
|
+
- `message-send`, `message-list`, `message-ack`;
|
|
664
|
+
- `message-status <message-key>` and `unread-count --for <recipient>`;
|
|
665
|
+
- `handoff`, `safe-send-pointer`, `wait-agent`, `monitor-agent`;
|
|
666
|
+
- automatic monitor arming after send/pointer operations in both Claude and Pi;
|
|
667
|
+
- Pi inbox widgets and deferred-reply reminders;
|
|
668
|
+
- durable handoff records and delivery-attempt separation;
|
|
669
|
+
- agent-state and turn-completion events.
|
|
670
|
+
|
|
671
|
+
These mechanisms are the bridge attention plane. They do not own Specialist chain truth, gate authority, resolved chain shape or final release decisions.
|
|
672
|
+
|
|
673
|
+
## 2.14 Bridge persistence split
|
|
674
|
+
|
|
675
|
+
During this PRD's implementation horizon:
|
|
676
|
+
|
|
677
|
+
| State | Authoritative store |
|
|
678
|
+
|---|---|
|
|
679
|
+
| Managed Specialist jobs, results, chain links, forensic events, metrics and evals | per-repository Specialists `observability.db` |
|
|
680
|
+
| Interactive tmux role lifecycle, short messages, receipts, monitors, handoffs and delivery attempts | xtmux `observability.db` |
|
|
681
|
+
| Durable task contracts, relationships, notes and memory | Beads |
|
|
682
|
+
| Console read model | Console materializer output |
|
|
683
|
+
|
|
684
|
+
Correlation uses Bead, chain, job, participant/activation and message/handoff identifiers. The stores are not merged prematurely. Console may materialize both sources later, but Specialists remains the evaluation and chain-semantics owner.
|
|
685
|
+
|
|
686
|
+
# 3. Problem statement
|
|
687
|
+
|
|
688
|
+
## 3.1 Prompt payload and instruction competition
|
|
689
|
+
|
|
690
|
+
The six specialist prompts are costly not merely because they are verbose, but because they compete with other instructions. Long prompts contain universal programming opinions, CLI manuals, detailed output examples, role procedures, shared policy and personality cues. The runner and mandatory-rule layers then inject additional versions of the same concepts.
|
|
691
|
+
|
|
692
|
+
The result is an instruction graph with multiple authorities. The model must infer which version of scope discipline, output schema, Git workflow or evidence ordering is current. This creates hidden failure modes:
|
|
693
|
+
|
|
694
|
+
- a stale system-prompt section can override a newer shared rule;
|
|
695
|
+
- examples become treated as requirements;
|
|
696
|
+
- a role over-applies generic guidance that conflicts with repository conventions;
|
|
697
|
+
- a generated schema and a prose schema disagree;
|
|
698
|
+
- a global rule unintentionally constrains a specialist for which it was not designed;
|
|
699
|
+
- critical role instructions are buried beneath tool manuals and style advice.
|
|
700
|
+
|
|
701
|
+
## 3.2 Role overlap
|
|
702
|
+
|
|
703
|
+
Reviewer is nominally phase-two and should consume seconder results, but its prompt reconstructs atomic requirements and repeats broad compliance review. Executor carries a complete TypeScript and clean-code manual despite being a general implementation specialist. Overthinker repeats the four-phase rule already injected separately. Researcher embeds command manuals already represented in routing rules. Test-engineer repeats the same source-boundary rule three times.
|
|
704
|
+
|
|
705
|
+
This overlap increases both token use and behavioral ambiguity. The modernization must enforce the principle that each role owns a narrow decision and trusts upstream work conditionally rather than starting from zero.
|
|
706
|
+
|
|
707
|
+
## 3.3 Missing participant identity
|
|
708
|
+
|
|
709
|
+
Specialists are chain participants, but the current role identity is mostly local: “you are reviewer”, “you are executor”. The system does not consistently say:
|
|
710
|
+
|
|
711
|
+
- which chain the specialist belongs to;
|
|
712
|
+
- which formula/template was resolved;
|
|
713
|
+
- which root contract governs the overall work;
|
|
714
|
+
- which step contract governs the current mandate;
|
|
715
|
+
- what upstream roles have completed;
|
|
716
|
+
- what evidence each upstream role produced;
|
|
717
|
+
- which gates remain pending;
|
|
718
|
+
- which downstream role will consume this handoff;
|
|
719
|
+
- which evidence is preloaded and which is available on demand.
|
|
720
|
+
|
|
721
|
+
Without this identity, agents either duplicate upstream work or fail to reconstruct necessary context.
|
|
722
|
+
|
|
723
|
+
## 3.4 Eager memory context
|
|
724
|
+
|
|
725
|
+
The runtime currently retrieves a bounded set of memories based on Bead keywords. Although smaller than earlier full-memory injection designs, the retrieval is still eager and lexical. It can inject irrelevant or stale knowledge into fully specified tasks.
|
|
726
|
+
|
|
727
|
+
The desired model is capability-based memory: the agent knows that historical knowledge exists, recognizes when prior decisions or incidents matter, and retrieves targeted memory with provenance. This should be measured rather than assumed.
|
|
728
|
+
|
|
729
|
+
## 3.5 Telemetry ambiguity
|
|
730
|
+
|
|
731
|
+
Several current metrics cannot yet support rigorous comparisons:
|
|
732
|
+
|
|
733
|
+
- thinking events lack start/end phases and duration;
|
|
734
|
+
- thinking character counts are cumulative rather than per delta or final segment;
|
|
735
|
+
- reasoning tokens are provider-dependent and not equivalent to reasoning time;
|
|
736
|
+
- tool metrics may count lifecycle events rather than unique calls;
|
|
737
|
+
- the Pi callback path can report both tool-call construction and tool execution;
|
|
738
|
+
- a query checks `think` while the canonical event type is `thinking`;
|
|
739
|
+
- forensic-derived Prometheus metrics are not wired into the default scrape path;
|
|
740
|
+
- historical rows have heterogeneous schema completeness.
|
|
741
|
+
|
|
742
|
+
## 3.6 Evaluation gap
|
|
743
|
+
|
|
744
|
+
Existing observability can describe cost and activity, but it does not yet determine quality. Reviewer PASS rate, for example, is not equivalent to reviewer accuracy. A model may PASS more often because it misses defects. Historical cohort comparisons can be confounded by task difficulty. A shorter prompt can save tokens while increasing false PASS rates. An LLM judge can prefer verbosity or the first answer shown.
|
|
745
|
+
|
|
746
|
+
A first-class evaluation platform is required to establish repeatable evidence for prompt, rule, runtime and model changes.
|
|
747
|
+
|
|
748
|
+
# 4. Goals, non-goals and design principles
|
|
749
|
+
|
|
750
|
+
## 4.1 Goals
|
|
751
|
+
|
|
752
|
+
The project shall:
|
|
753
|
+
|
|
754
|
+
1. Reduce static and eager context for the six target roles without reducing task success or role compliance.
|
|
755
|
+
2. Establish one source of truth for every shared policy and output contract.
|
|
756
|
+
3. Make chain membership and context reconstruction explicit and deterministic.
|
|
757
|
+
4. Replace indiscriminate memory push with measured, targeted recall.
|
|
758
|
+
5. Correct telemetry semantics before using them for promotion decisions.
|
|
759
|
+
6. Support retrospective evaluation of existing runs with explicit completeness labels.
|
|
760
|
+
7. Support automatic deterministic evaluation after every run and every chain.
|
|
761
|
+
8. Support controlled paired A/B testing across models, prompts, rules and runtime versions.
|
|
762
|
+
9. Support external software-engineering benchmarks and internal SWE-bench-style fixtures.
|
|
763
|
+
10. Integrate results into Console with historical, live and experiment-specific views.
|
|
764
|
+
11. Preserve low-cardinality Prometheus discipline and forensic drilldown.
|
|
765
|
+
12. Make every experiment reproducible through candidate and environment fingerprints.
|
|
766
|
+
|
|
767
|
+
## 4.2 Non-goals
|
|
768
|
+
|
|
769
|
+
The project shall not:
|
|
770
|
+
|
|
771
|
+
- merge the six roles into a single generic agent;
|
|
772
|
+
- expose private chain-of-thought or require models to reveal it;
|
|
773
|
+
- treat model-written success claims as environment truth;
|
|
774
|
+
- make LLM graders authoritative blocking gates before calibration;
|
|
775
|
+
- add high-cardinality IDs to Prometheus labels;
|
|
776
|
+
- create a second telemetry or Console application unnecessarily;
|
|
777
|
+
- use historical cohort averages as a substitute for controlled replay;
|
|
778
|
+
- force every role to retrieve memories on every run;
|
|
779
|
+
- inject entire chain histories into every participant;
|
|
780
|
+
- replace repository conventions with a universal clean-code doctrine;
|
|
781
|
+
- couple Console materialization to source-of-truth mutation.
|
|
782
|
+
|
|
783
|
+
## 4.3 Design principles
|
|
784
|
+
|
|
785
|
+
### Intrinsic role identity in the system prompt
|
|
786
|
+
|
|
787
|
+
A role prompt should contain only the mandate, boundaries, evidence hierarchy and decision rules intrinsic to that role.
|
|
788
|
+
|
|
789
|
+
### Shared policy in one reusable layer
|
|
790
|
+
|
|
791
|
+
Cross-role policies belong in mandatory rules or runtime enforcement, not copied prose.
|
|
792
|
+
|
|
793
|
+
### Deterministic decisions in code
|
|
794
|
+
|
|
795
|
+
Scrutiny floors, required gates, path matching, worktree boundaries, output-schema validation and chain identity should be computed by the runtime.
|
|
796
|
+
|
|
797
|
+
### Deep procedure on demand
|
|
798
|
+
|
|
799
|
+
CLI manuals and lengthy playbooks belong in focused skills or tool descriptions, loaded only when needed.
|
|
800
|
+
|
|
801
|
+
### Identity pushed, content pulled
|
|
802
|
+
|
|
803
|
+
The runtime should inject compact chain identity, member indexes and evidence pointers. The specialist should retrieve substantive upstream content only when its role requires it.
|
|
804
|
+
|
|
805
|
+
### Outcome over self-report
|
|
806
|
+
|
|
807
|
+
Tests, environment state, diff and evidence artifacts outrank the model’s final statement.
|
|
808
|
+
|
|
809
|
+
### Hard gates before weighted scores
|
|
810
|
+
|
|
811
|
+
A role-boundary violation or critical false PASS cannot be compensated by token efficiency.
|
|
812
|
+
|
|
813
|
+
### Observe before enforce
|
|
814
|
+
|
|
815
|
+
New graders and policies begin in shadow mode. Enforcement requires measured precision and low false-positive rates.
|
|
816
|
+
|
|
817
|
+
### Version every evaluator
|
|
818
|
+
|
|
819
|
+
Prompt, rules, skills, runtime, grader, dataset and environment versions must be recorded.
|
|
820
|
+
|
|
821
|
+
# 5. Workstream A — Prompt and policy modernization
|
|
822
|
+
|
|
823
|
+
## 5.1 Target prompt architecture
|
|
824
|
+
|
|
825
|
+
Each target specialist should resolve into five conceptual layers:
|
|
826
|
+
|
|
827
|
+
```text
|
|
828
|
+
role core system prompt
|
|
829
|
+
+ applicable shared mandatory rules
|
|
830
|
+
+ compact runtime context
|
|
831
|
+
+ role-specific task contract
|
|
832
|
+
+ on-demand procedural skills/tools
|
|
833
|
+
```
|
|
834
|
+
|
|
835
|
+
The role core should generally remain within approximately 180–600 tokens. This is a design target, not a hard limit. Content is retained when it materially changes decisions.
|
|
836
|
+
|
|
837
|
+
The target reductions from the initial audit are:
|
|
838
|
+
|
|
839
|
+
| Specialist | Estimated system-prompt reduction | Target core size |
|
|
840
|
+
|---|---:|---:|
|
|
841
|
+
| Reviewer | 70–80% | 350–600 tokens |
|
|
842
|
+
| Executor | 80–90% | 250–450 tokens |
|
|
843
|
+
| Overthinker | 60–75% | 180–300 tokens |
|
|
844
|
+
| Seconder | 45–60% | 220–350 tokens |
|
|
845
|
+
| Test-engineer | 55–70% | 250–400 tokens |
|
|
846
|
+
| Researcher | 70–85% | 220–350 tokens |
|
|
847
|
+
|
|
848
|
+
The real KPI is first-turn input and outcome, not the length of the JSON field.
|
|
849
|
+
|
|
850
|
+
## 5.2 Output contract consolidation
|
|
851
|
+
|
|
852
|
+
The runtime already generates a base output contract and merges output-type extensions and specialist output schemas. This must become the only normative output definition.
|
|
853
|
+
|
|
854
|
+
Required changes:
|
|
855
|
+
|
|
856
|
+
1. Normalize base statuses. Choose one canonical vocabulary, recommended:
|
|
857
|
+
|
|
858
|
+
```text
|
|
859
|
+
success | partial | failed | waiting
|
|
860
|
+
```
|
|
861
|
+
|
|
862
|
+
2. Normalize reviewer verdicts, recommended uppercase at presentation boundaries and a canonical normalized enum in storage.
|
|
863
|
+
3. Convert example-shaped output schemas into valid JSON Schema.
|
|
864
|
+
4. Remove prose instructions that restate field lists already represented in schema.
|
|
865
|
+
5. Remove or radically shrink `per-turn-handoff-schema`; it may explain semantics but must not define a competing schema.
|
|
866
|
+
6. Emit schema-validation results as deterministic evaluation events.
|
|
867
|
+
7. Version the merged schema and store its hash on each run.
|
|
868
|
+
|
|
869
|
+
## 5.3 Reviewer modernization
|
|
870
|
+
|
|
871
|
+
### Current defects
|
|
872
|
+
|
|
873
|
+
- Repeats requirement normalization and compliance work assigned to seconder.
|
|
874
|
+
- Uses an adversarial persona that may increase cosmetic findings and severity inflation.
|
|
875
|
+
- Contains a full CLI and evidence-reconstruction manual.
|
|
876
|
+
- Computes scrutiny and gate requirements in prompt text rather than consuming resolved policy.
|
|
877
|
+
- Repeats output and Release Checklist formats.
|
|
878
|
+
- Duplicates the `seconder ran` checklist item.
|
|
879
|
+
- Contains obligations and simplification procedures that should be separate gate/policy inputs.
|
|
880
|
+
|
|
881
|
+
### Target responsibility
|
|
882
|
+
|
|
883
|
+
Reviewer is the final evidence-based release gate. It should:
|
|
884
|
+
|
|
885
|
+
- consume the root contract and resolved chain state;
|
|
886
|
+
- consume final upstream gate evidence;
|
|
887
|
+
- inspect unresolved findings, critical requirement coverage and high-risk changed surfaces;
|
|
888
|
+
- detect contradictions or missing required evidence;
|
|
889
|
+
- produce PASS, PARTIAL or FAIL with evidence-linked findings;
|
|
890
|
+
- use delta mode for re-review after PARTIAL;
|
|
891
|
+
- never edit files;
|
|
892
|
+
- avoid preference-only findings.
|
|
893
|
+
|
|
894
|
+
### Runtime-provided inputs
|
|
895
|
+
|
|
896
|
+
```text
|
|
897
|
+
reviewed_job_id
|
|
898
|
+
chain_id
|
|
899
|
+
root_bead_id
|
|
900
|
+
current_step_bead_id
|
|
901
|
+
resolved_scrutiny
|
|
902
|
+
required_gates
|
|
903
|
+
gate_results
|
|
904
|
+
upstream_job_ids
|
|
905
|
+
injected diff/evidence refs
|
|
906
|
+
previous reviewer findings for re-review
|
|
907
|
+
```
|
|
908
|
+
|
|
909
|
+
### Procedures moved out of core prompt
|
|
910
|
+
|
|
911
|
+
- job and feed lookup command details;
|
|
912
|
+
- scrutiny path tables;
|
|
913
|
+
- obligation marker vocabulary;
|
|
914
|
+
- Release Checklist rendering;
|
|
915
|
+
- output JSON examples;
|
|
916
|
+
- tool-specific GitNexus syntax;
|
|
917
|
+
- full Ddiff procedure, except a concise semantic statement.
|
|
918
|
+
|
|
919
|
+
## 5.4 Executor modernization
|
|
920
|
+
|
|
921
|
+
### Current defects
|
|
922
|
+
|
|
923
|
+
The executor prompt contains an extensive universal style guide: naming rules, function-length limits, nesting limits, TypeScript and Zod requirements, error doctrine, concurrency doctrine, performance doctrine, security rules, testing advice and anti-pattern tables.
|
|
924
|
+
|
|
925
|
+
Several rules conflict with one another or with repository-specific practice. Examples include “extract at the second duplication” versus “wait until the third use”, “fix adjacent smells” versus strict scope, mandatory Zod for every external input, hard limits on function length and arguments, and “never return null”.
|
|
926
|
+
|
|
927
|
+
### Target responsibility
|
|
928
|
+
|
|
929
|
+
Executor should:
|
|
930
|
+
|
|
931
|
+
- resolve and obey the Bead contract;
|
|
932
|
+
- remain within the assigned worktree and scope;
|
|
933
|
+
- inspect relevant code and dependency impact;
|
|
934
|
+
- make the smallest correct change;
|
|
935
|
+
- follow local language/framework/repository conventions;
|
|
936
|
+
- reuse existing code and dependencies;
|
|
937
|
+
- avoid speculative abstraction and unrelated cleanup;
|
|
938
|
+
- perform declared validation and focused checks;
|
|
939
|
+
- return a complete evidence-based handoff.
|
|
940
|
+
|
|
941
|
+
Language-specific standards should be loaded from repository or on-demand skills, not hardcoded globally.
|
|
942
|
+
|
|
943
|
+
### Validation policy
|
|
944
|
+
|
|
945
|
+
Executor may run focused validation required by the Bead. Broad authoritative suite execution remains owned by test-runner unless explicitly required. The current blanket “do not run tests” language should be replaced with ownership-aware wording.
|
|
946
|
+
|
|
947
|
+
## 5.5 Overthinker modernization
|
|
948
|
+
|
|
949
|
+
### Current defects
|
|
950
|
+
|
|
951
|
+
- Repeats the four-phase rule already injected separately.
|
|
952
|
+
- Forces a multi-phase structure even for simple reversible decisions.
|
|
953
|
+
- Uses “multi-persona chain-of-thought” framing.
|
|
954
|
+
- Contains hardcoded command permissions that are already enforced by tool tiers.
|
|
955
|
+
|
|
956
|
+
### Target responsibility
|
|
957
|
+
|
|
958
|
+
Overthinker is a read-only decision-review specialist. It should scale depth to consequence and uncertainty:
|
|
959
|
+
|
|
960
|
+
- simple/reversible decision: direct recommendation;
|
|
961
|
+
- uncertain/costly decision: compare viable alternatives and trade-offs;
|
|
962
|
+
- high-impact/irreversible decision: premortem, disconfirming evidence and confidence.
|
|
963
|
+
|
|
964
|
+
It reports conclusions, assumptions, alternatives, risks and validation steps, not private reasoning traces.
|
|
965
|
+
|
|
966
|
+
## 5.6 Seconder modernization
|
|
967
|
+
|
|
968
|
+
### Current strengths
|
|
969
|
+
|
|
970
|
+
Seconder is already the best-bounded prompt in the group. Its split between scope verdict and quality verdict is useful. `UNCLEAR` is a valuable state when evidence is truly insufficient.
|
|
971
|
+
|
|
972
|
+
### Current defects
|
|
973
|
+
|
|
974
|
+
- smell lists repeat shared code-quality rules;
|
|
975
|
+
- output schema appears both in prose and JSON;
|
|
976
|
+
- tool-call budget belongs in runtime configuration;
|
|
977
|
+
- evidence hierarchy can be shared with other gates;
|
|
978
|
+
- eager context expansion could make it a second reviewer.
|
|
979
|
+
|
|
980
|
+
### Target responsibility
|
|
981
|
+
|
|
982
|
+
Seconder answers one question: whether the writer output is sufficiently compliant and implementation-sane to justify expensive QA. It should not become a release, security or broad architecture review.
|
|
983
|
+
|
|
984
|
+
## 5.7 Test-engineer modernization
|
|
985
|
+
|
|
986
|
+
### Current defects
|
|
987
|
+
|
|
988
|
+
- source boundary repeated in inline rules, system prompt and task template;
|
|
989
|
+
- mode inferred from free text rather than explicit dispatch metadata;
|
|
990
|
+
- output schema is example-shaped rather than formal JSON Schema;
|
|
991
|
+
- report-section requirements duplicate generated output contract;
|
|
992
|
+
- possible stale or installation-local `test-planning` skill reference.
|
|
993
|
+
|
|
994
|
+
### Target responsibility
|
|
995
|
+
|
|
996
|
+
Test-engineer receives an explicit mode:
|
|
997
|
+
|
|
998
|
+
```text
|
|
999
|
+
test_only | post_implementation
|
|
1000
|
+
```
|
|
1001
|
+
|
|
1002
|
+
It authors tests and testing assets from actual behavior and diff evidence. Production edits are forbidden unless an explicitly named helper/export change is authorized. When source behavior is wrong, it returns `source_bug_suspected` with evidence.
|
|
1003
|
+
|
|
1004
|
+
## 5.8 Researcher modernization
|
|
1005
|
+
|
|
1006
|
+
### Current defects
|
|
1007
|
+
|
|
1008
|
+
- embeds detailed tool and CLI manuals;
|
|
1009
|
+
- duplicates research routing rules;
|
|
1010
|
+
- absolute “never answer from memory” language is too broad;
|
|
1011
|
+
- does not sufficiently separate internal project memory from external evidence.
|
|
1012
|
+
|
|
1013
|
+
### Target responsibility
|
|
1014
|
+
|
|
1015
|
+
Researcher applies an epistemic policy:
|
|
1016
|
+
|
|
1017
|
+
- use current authoritative sources for unstable claims;
|
|
1018
|
+
- prefer primary documentation and source repositories;
|
|
1019
|
+
- distinguish facts, inference and unknowns;
|
|
1020
|
+
- record dates and versions;
|
|
1021
|
+
- verify consequential claims where practical;
|
|
1022
|
+
- stop when success criteria are met;
|
|
1023
|
+
- report tool failure rather than invent evidence.
|
|
1024
|
+
|
|
1025
|
+
Tool syntax stays in tool descriptions or focused skills.
|
|
1026
|
+
|
|
1027
|
+
## 5.9 Runner-injected content review
|
|
1028
|
+
|
|
1029
|
+
The prompt modernization must include runner blocks, not only specialist JSON.
|
|
1030
|
+
|
|
1031
|
+
### Specialist Run Context
|
|
1032
|
+
|
|
1033
|
+
Retain, but split into applicable fragments. Bead claim/close instructions should not be injected into every read-only or chain-step role when the orchestrator owns lifecycle.
|
|
1034
|
+
|
|
1035
|
+
### Output-style directive
|
|
1036
|
+
|
|
1037
|
+
The current global “smart caveman” rule may help agent-to-agent brevity but conflicts with evidence uncertainty and readable research/review output. Replace it with a bounded, role-aware style contract:
|
|
1038
|
+
|
|
1039
|
+
```text
|
|
1040
|
+
Be concise and technical. Preserve uncertainty, evidence qualifiers and exact identifiers. Do not omit required rationale or schema fields.
|
|
1041
|
+
```
|
|
1042
|
+
|
|
1043
|
+
Style effects should be evaluated by role.
|
|
1044
|
+
|
|
1045
|
+
### GitNexus mandate
|
|
1046
|
+
|
|
1047
|
+
Remove duplication between runner hardcoding, mandatory rules and task templates. Prefer a single runtime-generated tool policy that is aware of repository index health and role. Read-only roles should not receive “before editing” language.
|
|
1048
|
+
|
|
1049
|
+
### Static Beads close checklist
|
|
1050
|
+
|
|
1051
|
+
Do not inject `git add`, `commit`, `push` into read-only roles. Make it writer-specific and consistent with auto-commit ownership.
|
|
1052
|
+
|
|
1053
|
+
### Core session boundary
|
|
1054
|
+
|
|
1055
|
+
Split local worktree boundary from evidence-retrieval policy. Researcher must be permitted to retrieve external evidence. Missing local artifacts should not globally prohibit external or chain-context lookup.
|
|
1056
|
+
|
|
1057
|
+
## 5.10 Prompt fingerprinting
|
|
1058
|
+
|
|
1059
|
+
Every run must record:
|
|
1060
|
+
|
|
1061
|
+
```text
|
|
1062
|
+
system_prompt_hash
|
|
1063
|
+
task_template_hash
|
|
1064
|
+
merged_output_schema_hash
|
|
1065
|
+
mandatory_rules_hash
|
|
1066
|
+
mandatory_rule_ids and versions
|
|
1067
|
+
skill_paths and skill_body_hashes
|
|
1068
|
+
runner_injection_version
|
|
1069
|
+
tool_catalog_version
|
|
1070
|
+
resolved_tool_set_hash
|
|
1071
|
+
specialist_version
|
|
1072
|
+
```
|
|
1073
|
+
|
|
1074
|
+
The system should optionally store redacted prompt component metadata and token counts, not raw sensitive prompt text in Prometheus.
|
|
1075
|
+
|
|
1076
|
+
|
|
1077
|
+
## 5.11 `using-specialists` progressive disclosure
|
|
1078
|
+
|
|
1079
|
+
The canonical `using-specialists` skill has grown into a broad orchestration manual. It currently mixes stable doctrine, volatile CLI facts, long chain recipes, recovery procedures, interactive-coordinator guidance and operational reference material. Eagerly loading the complete body into persistent coordinator sessions conflicts with the objective of reducing monitoring context rot.
|
|
1080
|
+
|
|
1081
|
+
Target structure:
|
|
1082
|
+
|
|
1083
|
+
```text
|
|
1084
|
+
config/skills/using-specialists/
|
|
1085
|
+
├── SKILL.md compact router and invariant discipline
|
|
1086
|
+
├── references/
|
|
1087
|
+
│ ├── contracts-and-beads.md
|
|
1088
|
+
│ ├── chain-composition.md
|
|
1089
|
+
│ ├── qa-and-release-gates.md
|
|
1090
|
+
│ ├── monitoring-and-recovery.md
|
|
1091
|
+
│ ├── worktrees-merge-restitch.md
|
|
1092
|
+
│ ├── observability-and-evidence.md
|
|
1093
|
+
│ └── interactive-coordination.md
|
|
1094
|
+
├── playbooks/
|
|
1095
|
+
│ ├── single-chain.md
|
|
1096
|
+
│ ├── diagnosis-debug.md
|
|
1097
|
+
│ ├── epic-multiplexing.md
|
|
1098
|
+
│ ├── security-sensitive.md
|
|
1099
|
+
│ └── release-integration.md
|
|
1100
|
+
└── templates/
|
|
1101
|
+
└── pointers or packaged copies required for current chain-template discovery
|
|
1102
|
+
```
|
|
1103
|
+
|
|
1104
|
+
Requirements:
|
|
1105
|
+
|
|
1106
|
+
- Root `SKILL.md` target: 250–350 lines; hard ceiling 500.
|
|
1107
|
+
- Every old heading maps to the compact core, an on-demand resource, another canonical document or an explicit deduplication/removal decision.
|
|
1108
|
+
- No information is copied blindly. Volatile CLI guidance is reconciled against current `sp help`, Core `--role` behavior and xtmux V2.
|
|
1109
|
+
- Installed global layout is recursively complete and link-valid under the current Core global-skills model.
|
|
1110
|
+
- Coordinator sessions load only the root skill and coordinator-relevant references initially; phase-specific playbooks are read on demand.
|
|
1111
|
+
- The old skill remains the baseline artifact for parity and behavior evaluation.
|
|
1112
|
+
|
|
1113
|
+
## 5.12 Interactive coordinator role doctrine
|
|
1114
|
+
|
|
1115
|
+
The coordinator's stable system prompt defines role identity and long-lived boundaries. It must not contain the current Bead payload or a flattened copy of mandatory rules.
|
|
1116
|
+
|
|
1117
|
+
The coordinator receives:
|
|
1118
|
+
|
|
1119
|
+
```text
|
|
1120
|
+
system prompt
|
|
1121
|
+
stable identity, one-epic scope, parent relationship, escalation boundary
|
|
1122
|
+
|
|
1123
|
+
using-specialists root skill
|
|
1124
|
+
current execution mechanics and routing
|
|
1125
|
+
|
|
1126
|
+
coordinating-chains skill/reference
|
|
1127
|
+
chain-template awareness and coordinator-specific procedures
|
|
1128
|
+
|
|
1129
|
+
first user assignment
|
|
1130
|
+
current Bead/dependency context
|
|
1131
|
+
current mandatory rules
|
|
1132
|
+
current parent/peer pointers
|
|
1133
|
+
current chain/collision snapshot
|
|
1134
|
+
```
|
|
1135
|
+
|
|
1136
|
+
The current hardcoded role sequence in the coordinator prompt is removed. The coordinator must resolve chain shape from the canonical formula catalog and, once available, from `sp chain review/approve/insert` and the persisted resolved chain shape.
|
|
1137
|
+
|
|
1138
|
+
The coordinator-specific guidance must cover:
|
|
1139
|
+
|
|
1140
|
+
- correct hierarchy: epic/task group → root Bead → molecule/chain → step Beads;
|
|
1141
|
+
- chain-template catalog discovery and selection;
|
|
1142
|
+
- scope/non-goal/scrutiny validation;
|
|
1143
|
+
- mandatory-gate completeness;
|
|
1144
|
+
- collision matrix construction and serialization/unification/restitch decisions;
|
|
1145
|
+
- routine fix-loop routing;
|
|
1146
|
+
- cross-coordinator mechanical communication;
|
|
1147
|
+
- parent escalation;
|
|
1148
|
+
- close-time evidence, clean-git verification, follow-up Beads and memory consideration.
|
|
1149
|
+
|
|
1150
|
+
Permission/tool enforcement for interactive roles is not part of this runway. Role-boundary violations are measured in evals and remain prompt/skill discipline until Substrate provides the proper capability model.
|
|
1151
|
+
|
|
1152
|
+
# 6. Workstream B — Chain participant identity and context reconstruction
|
|
1153
|
+
|
|
1154
|
+
## 6.1 Product requirement
|
|
1155
|
+
|
|
1156
|
+
A specialist dispatched as a chain step must be told that it is a chain participant. The runtime shall provide enough identity to orient the agent without injecting the complete chain history.
|
|
1157
|
+
|
|
1158
|
+
## 6.2 Chain context envelope
|
|
1159
|
+
|
|
1160
|
+
Recommended task/system injection:
|
|
1161
|
+
|
|
1162
|
+
```xml
|
|
1163
|
+
<chain-context version="1">
|
|
1164
|
+
<chain-id>...</chain-id>
|
|
1165
|
+
<chain-template>code-standard</chain-template>
|
|
1166
|
+
<root-bead-id>...</root-bead-id>
|
|
1167
|
+
<current-step-bead-id>...</current-step-bead-id>
|
|
1168
|
+
<current-role>reviewer</current-role>
|
|
1169
|
+
<current-class>gate</current-class>
|
|
1170
|
+
<position>final</position>
|
|
1171
|
+
<scrutiny stated="medium" effective="high" />
|
|
1172
|
+
<upstream complete="false">
|
|
1173
|
+
<step role="executor" bead-id="..." job-id="..." status="completed" />
|
|
1174
|
+
<step role="seconder" bead-id="..." job-id="..." status="completed" />
|
|
1175
|
+
</upstream>
|
|
1176
|
+
<pending>
|
|
1177
|
+
...
|
|
1178
|
+
</pending>
|
|
1179
|
+
<preloaded-context depth="3" complete="false" />
|
|
1180
|
+
<worktree-path>...</worktree-path>
|
|
1181
|
+
</chain-context>
|
|
1182
|
+
```
|
|
1183
|
+
|
|
1184
|
+
The envelope must be generated from persisted chain shape and Beads relationships, not inferred by the model.
|
|
1185
|
+
|
|
1186
|
+
## 6.3 Shared chain-participant rule
|
|
1187
|
+
|
|
1188
|
+
A shared mandatory rule should teach the operational semantics:
|
|
1189
|
+
|
|
1190
|
+
- the root change contract governs overall outcome;
|
|
1191
|
+
- the current step contract governs mandate and boundaries;
|
|
1192
|
+
- upstream handoffs are evidence, not overriding authority;
|
|
1193
|
+
- consume preloaded dependency context first;
|
|
1194
|
+
- reconstruct only missing necessary context;
|
|
1195
|
+
- prefer `sp result` for final handoff and `sp feed --json` only for claims that require trace verification;
|
|
1196
|
+
- use `bd show`, `bd dep list` and `bd dep tree` to understand issue relationships;
|
|
1197
|
+
- do not repeat upstream work unless the current role validates it, evidence is missing or artifacts contradict it;
|
|
1198
|
+
- report contradictions explicitly;
|
|
1199
|
+
- produce a downstream-usable handoff.
|
|
1200
|
+
|
|
1201
|
+
The rule should apply only when a real chain context exists or the Bead is marked `kind:step`.
|
|
1202
|
+
|
|
1203
|
+
## 6.4 Context retrieval hierarchy
|
|
1204
|
+
|
|
1205
|
+
Recommended hierarchy:
|
|
1206
|
+
|
|
1207
|
+
1. Injected chain context and authoritative runtime metadata.
|
|
1208
|
+
2. Current step Bead and root change contract.
|
|
1209
|
+
3. Preloaded completed dependency handoffs.
|
|
1210
|
+
4. `sp result` for specific upstream jobs.
|
|
1211
|
+
5. Current diff, repository state and evidence artifacts.
|
|
1212
|
+
6. `sp feed --json` for verifying claimed tool use, exact chronology or contradictory evidence.
|
|
1213
|
+
7. Semantic Beads edges for related, discovered-from or validates context.
|
|
1214
|
+
8. Historical memories only if prior decisions/incidents are materially relevant.
|
|
1215
|
+
|
|
1216
|
+
## 6.5 Handoff schema extension
|
|
1217
|
+
|
|
1218
|
+
The common output schema should support optional chain collaboration fields:
|
|
1219
|
+
|
|
1220
|
+
```json
|
|
1221
|
+
{
|
|
1222
|
+
"inputs_consumed": [
|
|
1223
|
+
{"role": "seconder", "bead_id": "...", "job_id": "..."}
|
|
1224
|
+
],
|
|
1225
|
+
"decisions": [
|
|
1226
|
+
{
|
|
1227
|
+
"decision": "...",
|
|
1228
|
+
"rationale": "...",
|
|
1229
|
+
"evidence": ["..."]
|
|
1230
|
+
}
|
|
1231
|
+
],
|
|
1232
|
+
"assumptions": [],
|
|
1233
|
+
"unresolved_questions": [],
|
|
1234
|
+
"downstream_attention": []
|
|
1235
|
+
}
|
|
1236
|
+
```
|
|
1237
|
+
|
|
1238
|
+
These fields capture decision records, not hidden chain-of-thought.
|
|
1239
|
+
|
|
1240
|
+
## 6.6 Role-specific chain behavior
|
|
1241
|
+
|
|
1242
|
+
### Executor
|
|
1243
|
+
|
|
1244
|
+
Consumes advisor findings and root contract. It should cite material advisor decisions it accepted or rejected.
|
|
1245
|
+
|
|
1246
|
+
### Seconder
|
|
1247
|
+
|
|
1248
|
+
Consumes writer diff and root contract. It should remain bounded and avoid loading the full chain unless a required input is absent.
|
|
1249
|
+
|
|
1250
|
+
### Test-engineer
|
|
1251
|
+
|
|
1252
|
+
Consumes writer diff, root validation contract, seconder verdict and prior test-runner output on retries.
|
|
1253
|
+
|
|
1254
|
+
### Reviewer
|
|
1255
|
+
|
|
1256
|
+
Consumes final evidence from all required gates. It does not independently recreate every gate unless evidence is missing or contradictory.
|
|
1257
|
+
|
|
1258
|
+
### Researcher and overthinker
|
|
1259
|
+
|
|
1260
|
+
When used as advisors, they must identify the downstream executor as consumer and produce action-oriented evidence rather than standalone essays.
|
|
1261
|
+
|
|
1262
|
+
## 6.7 Chain-context evaluation
|
|
1263
|
+
|
|
1264
|
+
The `chain-participation-v1` suite shall measure:
|
|
1265
|
+
|
|
1266
|
+
```text
|
|
1267
|
+
chain identity recognition
|
|
1268
|
+
root and step contract resolution
|
|
1269
|
+
upstream handoff discovery
|
|
1270
|
+
selective reconstruction
|
|
1271
|
+
semantic edge navigation
|
|
1272
|
+
contradiction detection
|
|
1273
|
+
non-duplication of upstream work
|
|
1274
|
+
downstream handoff quality
|
|
1275
|
+
```
|
|
1276
|
+
|
|
1277
|
+
Adversarial cases shall include missing handoffs, conflicting gates, stale memory, incomplete preloaded depth, unrelated semantic edges and cross-chain contamination.
|
|
1278
|
+
|
|
1279
|
+
|
|
1280
|
+
## 6.8 Interactive assignment contract
|
|
1281
|
+
|
|
1282
|
+
Interactive roles require a task-side composition path that preserves the managed Specialists design: current task context and mandatory rules are prominent in the user turn and remain separate from the system prompt.
|
|
1283
|
+
|
|
1284
|
+
Specialists owns a pure, non-executing renderer. Conceptual contract:
|
|
1285
|
+
|
|
1286
|
+
```bash
|
|
1287
|
+
sp role compose chain-coordinator \
|
|
1288
|
+
--surface interactive \
|
|
1289
|
+
--mode tracked|standby \
|
|
1290
|
+
[--bead <id>] \
|
|
1291
|
+
--project-cwd <path> \
|
|
1292
|
+
--parent-session <session-id> \
|
|
1293
|
+
--output <secure-prompt-file> \
|
|
1294
|
+
--json
|
|
1295
|
+
```
|
|
1296
|
+
|
|
1297
|
+
For `mode=tracked`, the renderer reuses the same pure seams as `sp run` for:
|
|
1298
|
+
|
|
1299
|
+
1. effective specialist resolution;
|
|
1300
|
+
2. bounded Bead and dependency context;
|
|
1301
|
+
3. `task_template` rendering;
|
|
1302
|
+
4. required/default/role/inline mandatory-rule resolution;
|
|
1303
|
+
5. token limits and fatal-resolution behavior;
|
|
1304
|
+
6. component metadata and fingerprinting.
|
|
1305
|
+
|
|
1306
|
+
The renderer does not create a job, worktree, session, Bead mutation or runtime event. It never returns the role system prompt. It emits a secure prompt file plus machine-readable metadata containing hashes, component IDs and byte/token counts.
|
|
1307
|
+
|
|
1308
|
+
For `mode=standby`, the renderer emits only the minimal bootstrap required for a preheated role to acknowledge its parent and wait for one epic assignment. It does not inject a fabricated Bead context or the full tracked mandatory-rule set.
|
|
1309
|
+
|
|
1310
|
+
The coordinator `task_template` must be audited because `$prompt` and `$bead_context` may currently resolve from the same tracked Bead payload. The final contract contains one complete Bead context exactly once.
|
|
1311
|
+
|
|
1312
|
+
## 6.9 Two launch modes
|
|
1313
|
+
|
|
1314
|
+
### Mode A — preheated standby
|
|
1315
|
+
|
|
1316
|
+
```bash
|
|
1317
|
+
xt pi --role chain-coordinator --new-session --no-attach
|
|
1318
|
+
xt claude --role chain-coordinator --new-session --no-attach
|
|
1319
|
+
```
|
|
1320
|
+
|
|
1321
|
+
Flow:
|
|
1322
|
+
|
|
1323
|
+
```text
|
|
1324
|
+
launch role and worktree
|
|
1325
|
+
→ runtime readiness event
|
|
1326
|
+
→ deliver standby bootstrap as first user turn
|
|
1327
|
+
→ coordinator sends role.ready to parent
|
|
1328
|
+
→ coordinator waits
|
|
1329
|
+
→ parent later assigns exactly one epic
|
|
1330
|
+
→ Specialists renders tracked assignment
|
|
1331
|
+
→ xtmux delivers pointer
|
|
1332
|
+
→ coordinator optionally sends assignment.accepted
|
|
1333
|
+
```
|
|
1334
|
+
|
|
1335
|
+
A preheated coordinator is not a general reusable daemon. It accepts one epic/task group and is retired after that scope completes.
|
|
1336
|
+
|
|
1337
|
+
### Mode B — direct tracked launch
|
|
1338
|
+
|
|
1339
|
+
```bash
|
|
1340
|
+
xt pi --role chain-coordinator --bead <epic> --new-session --no-attach
|
|
1341
|
+
xt claude --role chain-coordinator --bead <epic> --new-session --no-attach
|
|
1342
|
+
```
|
|
1343
|
+
|
|
1344
|
+
Flow:
|
|
1345
|
+
|
|
1346
|
+
```text
|
|
1347
|
+
resolve effective role
|
|
1348
|
+
→ render tracked assignment in original project cwd
|
|
1349
|
+
→ fail before provisioning if rendering fails
|
|
1350
|
+
→ create worktree/session
|
|
1351
|
+
→ wait for runtime readiness
|
|
1352
|
+
→ deliver prompt-file pointer as first user turn
|
|
1353
|
+
→ arm monitor
|
|
1354
|
+
→ retain @agent_bead and XTMUX_AGENT_BEAD as live identity pointers
|
|
1355
|
+
```
|
|
1356
|
+
|
|
1357
|
+
Passing `--bead` implies automatic tracked assignment unless an explicit debugging escape hatch such as `--no-auto-assign` is provided.
|
|
1358
|
+
|
|
1359
|
+
## 6.10 Runtime readiness and secure delivery
|
|
1360
|
+
|
|
1361
|
+
The current launcher writes `@agent_state=idle` before the runtime startup hook has proved that Pi or Claude can safely accept input. This creates a startup race.
|
|
1362
|
+
|
|
1363
|
+
Required bridge state:
|
|
1364
|
+
|
|
1365
|
+
```text
|
|
1366
|
+
starting | running | needs-input | done | idle | off
|
|
1367
|
+
```
|
|
1368
|
+
|
|
1369
|
+
Core sets `starting` at spawn. The Pi extension or Claude SessionStart hook transitions to `idle` and emits `agent.ready`. `starting` is treated as unsafe by `safe-send-pointer` and readiness waits.
|
|
1370
|
+
|
|
1371
|
+
Canonical delivery:
|
|
1372
|
+
|
|
1373
|
+
```text
|
|
1374
|
+
precomposed secure prompt file
|
|
1375
|
+
→ wait-agent/readiness gate
|
|
1376
|
+
→ handoff existing prompt file
|
|
1377
|
+
→ safe-send-pointer one-line pointer
|
|
1378
|
+
→ monitor-agent or existing automatic monitor hook
|
|
1379
|
+
```
|
|
1380
|
+
|
|
1381
|
+
Full prompt bodies must not appear in process argv, shell-quoted command strings, pane options, messages, launch events or telemetry. A positional initial prompt is rejected as the default bridge transport because process-list exposure and multiline/runtime divergence are avoidable. Pi and Claude receive semantically identical assignment content through their interactive prompt surface after readiness.
|
|
1382
|
+
|
|
1383
|
+
xtmux `handoff` should accept a precomposed prompt file without rewriting it and support an atomic high-level mode equivalent to:
|
|
1384
|
+
|
|
1385
|
+
```bash
|
|
1386
|
+
tmux-session-picker handoff \
|
|
1387
|
+
--target <pane> \
|
|
1388
|
+
--bead <id> \
|
|
1389
|
+
--existing-prompt-file <file> \
|
|
1390
|
+
--parent <session-id> \
|
|
1391
|
+
--wait-ready \
|
|
1392
|
+
--monitor \
|
|
1393
|
+
--yes
|
|
1394
|
+
```
|
|
1395
|
+
|
|
1396
|
+
`message-send` is used for durable short acknowledgements, status and escalations. It does not replace prompt injection.
|
|
1397
|
+
|
|
1398
|
+
## 6.11 Coordinator bridge duties
|
|
1399
|
+
|
|
1400
|
+
For each child chain, the interactive coordinator applies four future-facing duties through current primitives.
|
|
1401
|
+
|
|
1402
|
+
### Entry gate
|
|
1403
|
+
|
|
1404
|
+
Before dispatching step 1:
|
|
1405
|
+
|
|
1406
|
+
- verify root contract, scope, non-goals and acceptance criteria;
|
|
1407
|
+
- verify `contract:ready` state and SCRUTINY;
|
|
1408
|
+
- inspect the recommended/selected formula;
|
|
1409
|
+
- verify mandatory pipeline overlays and sensitive-surface gates;
|
|
1410
|
+
- inspect active collision state and dependency ordering;
|
|
1411
|
+
- insert or propose missing advisors/gates within the current roadmap's composition rules;
|
|
1412
|
+
- escalate when scope, architecture or budget is ambiguous.
|
|
1413
|
+
|
|
1414
|
+
### Borderline judge
|
|
1415
|
+
|
|
1416
|
+
During execution:
|
|
1417
|
+
|
|
1418
|
+
- route mechanically clear remediation to the correct writer/gate;
|
|
1419
|
+
- interpret minor findings against explicit non-goals;
|
|
1420
|
+
- detect contradictory or stale evidence;
|
|
1421
|
+
- escalate reviewer FAIL, sensitive findings, architectural disagreement and ambiguous completion.
|
|
1422
|
+
|
|
1423
|
+
### Hygiene coordinator
|
|
1424
|
+
|
|
1425
|
+
Across parallel child chains:
|
|
1426
|
+
|
|
1427
|
+
- maintain an explicit conflict matrix;
|
|
1428
|
+
- advertise touched paths, gate availability and wait dependencies;
|
|
1429
|
+
- communicate mechanical hygiene to peer coordinators through xtmux;
|
|
1430
|
+
- never use peer messages to decide product vision or final merge policy.
|
|
1431
|
+
|
|
1432
|
+
### Close-time judge
|
|
1433
|
+
|
|
1434
|
+
Before reporting readiness to the parent:
|
|
1435
|
+
|
|
1436
|
+
- verify all mandatory gates ran against the latest diff;
|
|
1437
|
+
- verify no unresolved PARTIAL/FAIL remains;
|
|
1438
|
+
- verify every intended change is committed and no stray artifacts remain;
|
|
1439
|
+
- verify evidence references and test results;
|
|
1440
|
+
- file `discovered-from` follow-up Beads for out-of-scope findings;
|
|
1441
|
+
- consider `bd remember` for durable lessons;
|
|
1442
|
+
- report `READY_FOR_PARENT_DECISION` with evidence.
|
|
1443
|
+
|
|
1444
|
+
The parent orchestrator retains final vision, merge, abandonment and beyond-policy authority during the bridge runway.
|
|
1445
|
+
|
|
1446
|
+
## 6.12 Coordinator communication vocabulary
|
|
1447
|
+
|
|
1448
|
+
Current xtmux messages should use a small pointer-oriented vocabulary that reads forward to future Channels/pulses:
|
|
1449
|
+
|
|
1450
|
+
```text
|
|
1451
|
+
role.ready
|
|
1452
|
+
assignment.accepted
|
|
1453
|
+
chain.status
|
|
1454
|
+
collision.notice
|
|
1455
|
+
gate.available
|
|
1456
|
+
wait.request
|
|
1457
|
+
handoff.pointer
|
|
1458
|
+
escalation.scope
|
|
1459
|
+
escalation.security
|
|
1460
|
+
escalation.verdict
|
|
1461
|
+
chain.ready
|
|
1462
|
+
```
|
|
1463
|
+
|
|
1464
|
+
Messages contain bounded summaries and durable references. Beads and Specialists state remain authoritative. Acknowledgement proves receipt/processing responsibility, not release authority.
|
|
1465
|
+
|
|
1466
|
+
Coordinator-to-coordinator communication is limited to mechanics. Direction changes, abandonment, merge ordering, sensitive judgment and architecture go to the parent orchestrator.
|
|
1467
|
+
|
|
1468
|
+
## 6.13 Interactive coordinator evaluation
|
|
1469
|
+
|
|
1470
|
+
Add an `interactive-coordinator-v1` suite with at least:
|
|
1471
|
+
|
|
1472
|
+
```text
|
|
1473
|
+
template-selection correctness
|
|
1474
|
+
scope/non-goal validation accuracy
|
|
1475
|
+
mandatory-gate completeness
|
|
1476
|
+
collision detection recall and false serialization
|
|
1477
|
+
routine fix-loop routing accuracy
|
|
1478
|
+
escalation precision and recall
|
|
1479
|
+
child-completion consumption latency
|
|
1480
|
+
role.ready and assignment.accepted behavior
|
|
1481
|
+
reply-obligation handling
|
|
1482
|
+
close-readiness accuracy
|
|
1483
|
+
follow-up Bead quality
|
|
1484
|
+
memory retrieval precision
|
|
1485
|
+
parent orchestrator context/token reduction
|
|
1486
|
+
coordinator compaction/context growth
|
|
1487
|
+
Pi vs Claude behavior deltas
|
|
1488
|
+
```
|
|
1489
|
+
|
|
1490
|
+
Primary experiment:
|
|
1491
|
+
|
|
1492
|
+
```text
|
|
1493
|
+
parent orchestrator coordinates directly
|
|
1494
|
+
vs Pi interactive chain-coordinator
|
|
1495
|
+
vs Claude interactive chain-coordinator
|
|
1496
|
+
```
|
|
1497
|
+
|
|
1498
|
+
# 7. Workstream C — Pull-based memory
|
|
1499
|
+
|
|
1500
|
+
## 7.1 Current behavior
|
|
1501
|
+
|
|
1502
|
+
The current runtime extracts keywords from Bead title and description, queries the memory cache and injects a bounded set of ranked memories. Ranking uses lexical BM25, recency and access frequency. The injection budget is bounded, but the model pays the cost even when historical context is unnecessary.
|
|
1503
|
+
|
|
1504
|
+
## 7.2 Target behavior
|
|
1505
|
+
|
|
1506
|
+
Memory becomes an explicit capability. The runtime injects a short policy, not memory content:
|
|
1507
|
+
|
|
1508
|
+
```text
|
|
1509
|
+
Historical project memories are available through bd memories and bd recall.
|
|
1510
|
+
Query them only when the task depends on prior decisions, conventions, incidents,
|
|
1511
|
+
accepted exceptions or unresolved ambiguity. Treat memories as context, not
|
|
1512
|
+
current authority; verify against the root contract, current code and documentation.
|
|
1513
|
+
```
|
|
1514
|
+
|
|
1515
|
+
## 7.3 Retrieval triggers
|
|
1516
|
+
|
|
1517
|
+
A specialist should consider memory when:
|
|
1518
|
+
|
|
1519
|
+
- the Bead references an existing convention or prior implementation;
|
|
1520
|
+
- multiple conflicting repository patterns exist;
|
|
1521
|
+
- a known incident or regression history may change validation;
|
|
1522
|
+
- an architecture or API choice depends on a previous decision;
|
|
1523
|
+
- a reviewer needs to understand an accepted exception;
|
|
1524
|
+
- an overthinker is evaluating a high-impact repeated failure mode.
|
|
1525
|
+
|
|
1526
|
+
A specialist should normally avoid memory when:
|
|
1527
|
+
|
|
1528
|
+
- the task is fully specified and local;
|
|
1529
|
+
- the information is directly available in current code or documentation;
|
|
1530
|
+
- the role is a cheap bounded gate and history is not required;
|
|
1531
|
+
- the claim concerns current external APIs, versions or standards that require external sources.
|
|
1532
|
+
|
|
1533
|
+
## 7.4 Retrieval interface
|
|
1534
|
+
|
|
1535
|
+
Preferred capabilities:
|
|
1536
|
+
|
|
1537
|
+
```bash
|
|
1538
|
+
bd memories search --query "..." --limit 5 --json
|
|
1539
|
+
bd memories relevant --bead <id> --limit 5 --max-tokens 500 --json
|
|
1540
|
+
bd recall <memory-key> --json
|
|
1541
|
+
```
|
|
1542
|
+
|
|
1543
|
+
Memory results should expose provenance:
|
|
1544
|
+
|
|
1545
|
+
```text
|
|
1546
|
+
memory key
|
|
1547
|
+
summary/value
|
|
1548
|
+
created and updated timestamps
|
|
1549
|
+
source Bead/job/commit
|
|
1550
|
+
confidence
|
|
1551
|
+
status: current | superseded | unknown
|
|
1552
|
+
```
|
|
1553
|
+
|
|
1554
|
+
If the current CLI cannot provide this shape, the project should add a Specialists-side wrapper over the existing FTS/ranking cache.
|
|
1555
|
+
|
|
1556
|
+
## 7.5 Memory telemetry
|
|
1557
|
+
|
|
1558
|
+
Record:
|
|
1559
|
+
|
|
1560
|
+
```text
|
|
1561
|
+
memory query count
|
|
1562
|
+
query kind and normalized keyword count
|
|
1563
|
+
result count
|
|
1564
|
+
tokens/bytes returned
|
|
1565
|
+
recall count
|
|
1566
|
+
memory identifiers consumed
|
|
1567
|
+
whether memory was cited in handoff
|
|
1568
|
+
query latency
|
|
1569
|
+
```
|
|
1570
|
+
|
|
1571
|
+
Do not put raw query text or memory content in Prometheus labels.
|
|
1572
|
+
|
|
1573
|
+
## 7.6 Memory evaluation
|
|
1574
|
+
|
|
1575
|
+
Required dimensions:
|
|
1576
|
+
|
|
1577
|
+
- necessary-memory recall rate;
|
|
1578
|
+
- unnecessary-query rate;
|
|
1579
|
+
- task success with and without memory;
|
|
1580
|
+
- stale-memory contradiction handling;
|
|
1581
|
+
- memory token cost;
|
|
1582
|
+
- retrieval latency;
|
|
1583
|
+
- provenance use;
|
|
1584
|
+
- model and role differences in deciding when to query.
|
|
1585
|
+
|
|
1586
|
+
A/B rollout should compare eager injection, pull guidance and no-memory control on curated cases.
|
|
1587
|
+
|
|
1588
|
+
# 8. Workstream D — Telemetry and forensic hardening
|
|
1589
|
+
|
|
1590
|
+
## 8.1 Requirement
|
|
1591
|
+
|
|
1592
|
+
No model, prompt or chain promotion decision may depend on a metric with unresolved counting semantics. A telemetry integrity suite must be introduced and pass before the eval platform is used for promotion.
|
|
1593
|
+
|
|
1594
|
+
## 8.2 Thinking events
|
|
1595
|
+
|
|
1596
|
+
### Current behavior
|
|
1597
|
+
|
|
1598
|
+
Pi exposes `thinking_start`, `thinking_delta` and `thinking_end`. Specialists currently emits generic `thinking` events for start and deltas, with only a character count. `thinking_end` is not persisted. The supervisor accumulates characters, so persisted values are cumulative snapshots.
|
|
1599
|
+
|
|
1600
|
+
### Required event model
|
|
1601
|
+
|
|
1602
|
+
```json
|
|
1603
|
+
{
|
|
1604
|
+
"type": "thinking",
|
|
1605
|
+
"phase": "start|delta|end",
|
|
1606
|
+
"segment_id": "...",
|
|
1607
|
+
"turn_index": 2,
|
|
1608
|
+
"delta_char_count": 128,
|
|
1609
|
+
"total_char_count": 1840,
|
|
1610
|
+
"duration_ms": 7320
|
|
1611
|
+
}
|
|
1612
|
+
```
|
|
1613
|
+
|
|
1614
|
+
The end event should carry final segment duration and total characters. Raw thinking content should not be stored by default.
|
|
1615
|
+
|
|
1616
|
+
### Derived measures
|
|
1617
|
+
|
|
1618
|
+
```text
|
|
1619
|
+
thinking_stream_duration_ms
|
|
1620
|
+
thinking_segments
|
|
1621
|
+
thinking_chars
|
|
1622
|
+
reasoning_tokens
|
|
1623
|
+
thinking_to_active_runtime_ratio
|
|
1624
|
+
time_to_first_thinking
|
|
1625
|
+
time_to_first_tool
|
|
1626
|
+
time_to_first_text
|
|
1627
|
+
inter_tool_deliberation_ms
|
|
1628
|
+
```
|
|
1629
|
+
|
|
1630
|
+
The metric must be named as a stream duration, not provider compute time.
|
|
1631
|
+
|
|
1632
|
+
## 8.3 Tool-call accounting
|
|
1633
|
+
|
|
1634
|
+
### Current risk
|
|
1635
|
+
|
|
1636
|
+
Timeline aggregation increments tool counts for every `tool` event, which includes start, update and end. Pi may also call tool callbacks during both LLM tool-call construction and execution.
|
|
1637
|
+
|
|
1638
|
+
### Required correction
|
|
1639
|
+
|
|
1640
|
+
- Count a tool call only on canonical `tool phase=start` execution events.
|
|
1641
|
+
- Deduplicate by `tool_call_id`.
|
|
1642
|
+
- Track uncorrelated calls separately.
|
|
1643
|
+
- Distinguish construction events from execution events if retained.
|
|
1644
|
+
- Compute duration by matching start/end on `tool_call_id`.
|
|
1645
|
+
- Support parallel active tool calls without a single global `toolStartMs`.
|
|
1646
|
+
|
|
1647
|
+
### Metrics
|
|
1648
|
+
|
|
1649
|
+
```text
|
|
1650
|
+
xtrm_tool_calls_total
|
|
1651
|
+
xtrm_tool_errors_total
|
|
1652
|
+
xtrm_tool_call_duration_seconds
|
|
1653
|
+
xtrm_uncorrelated_tool_events_total
|
|
1654
|
+
xtrm_parallel_tool_concurrency
|
|
1655
|
+
```
|
|
1656
|
+
|
|
1657
|
+
## 8.4 Turn and message timing
|
|
1658
|
+
|
|
1659
|
+
Add stable start/end pairing for:
|
|
1660
|
+
|
|
1661
|
+
```text
|
|
1662
|
+
turn duration
|
|
1663
|
+
assistant message duration
|
|
1664
|
+
tool-result message duration
|
|
1665
|
+
time to first output token
|
|
1666
|
+
time from last tool end to final response
|
|
1667
|
+
```
|
|
1668
|
+
|
|
1669
|
+
## 8.5 Activity and stall semantics
|
|
1670
|
+
|
|
1671
|
+
Correct queries and state logic to use the canonical `thinking` event name rather than `think`. Stalls should distinguish:
|
|
1672
|
+
|
|
1673
|
+
- no protocol activity;
|
|
1674
|
+
- active thinking stream;
|
|
1675
|
+
- long-running tool;
|
|
1676
|
+
- provider/network silence;
|
|
1677
|
+
- waiting state.
|
|
1678
|
+
|
|
1679
|
+
## 8.6 Forensic-to-Prometheus wiring
|
|
1680
|
+
|
|
1681
|
+
The default metrics collector must read forensic events within the requested time range and pass them to the projection renderer. Required families include:
|
|
1682
|
+
|
|
1683
|
+
```text
|
|
1684
|
+
xtrm_gate_verdicts_total
|
|
1685
|
+
xtrm_evidence_refs_total
|
|
1686
|
+
xtrm_mcp_operations_total
|
|
1687
|
+
xtrm_identity_operations_total
|
|
1688
|
+
xtrm_policy_decisions_total
|
|
1689
|
+
xtrm_policy_mismatches_total
|
|
1690
|
+
xtrm_eval_runs_total
|
|
1691
|
+
xtrm_eval_score
|
|
1692
|
+
```
|
|
1693
|
+
|
|
1694
|
+
The collector must use bounded event-family queries and not load unbounded event history.
|
|
1695
|
+
|
|
1696
|
+
## 8.7 New operational metrics
|
|
1697
|
+
|
|
1698
|
+
Recommended additions:
|
|
1699
|
+
|
|
1700
|
+
```text
|
|
1701
|
+
xtrm_thinking_duration_seconds
|
|
1702
|
+
xtrm_thinking_segments_total
|
|
1703
|
+
xtrm_turn_duration_seconds
|
|
1704
|
+
xtrm_message_duration_seconds
|
|
1705
|
+
xtrm_time_to_first_action_seconds
|
|
1706
|
+
xtrm_schema_validation_total
|
|
1707
|
+
xtrm_handoff_validation_total
|
|
1708
|
+
xtrm_chain_fix_loops_total
|
|
1709
|
+
xtrm_chain_time_to_pass_seconds
|
|
1710
|
+
xtrm_chain_tokens_to_pass_total
|
|
1711
|
+
xtrm_memory_queries_total
|
|
1712
|
+
xtrm_memory_results_total
|
|
1713
|
+
xtrm_grader_duration_seconds
|
|
1714
|
+
xtrm_eval_assertions_total
|
|
1715
|
+
xtrm_eval_regressions_total
|
|
1716
|
+
xtrm_eval_judge_disagreements_total
|
|
1717
|
+
```
|
|
1718
|
+
|
|
1719
|
+
## 8.8 Historical completeness
|
|
1720
|
+
|
|
1721
|
+
Every derived historical metric shall record availability:
|
|
1722
|
+
|
|
1723
|
+
```text
|
|
1724
|
+
complete
|
|
1725
|
+
partial
|
|
1726
|
+
unavailable
|
|
1727
|
+
estimated
|
|
1728
|
+
```
|
|
1729
|
+
|
|
1730
|
+
Missing historical fields are unknown, not zero.
|
|
1731
|
+
|
|
1732
|
+
## 8.9 Telemetry integrity eval suite
|
|
1733
|
+
|
|
1734
|
+
`telemetry-integrity-v1` shall verify:
|
|
1735
|
+
|
|
1736
|
+
- exactly one canonical run-complete per run turn;
|
|
1737
|
+
- monotonic sequence;
|
|
1738
|
+
- tool starts equal unique calls;
|
|
1739
|
+
- paired tool durations where IDs exist;
|
|
1740
|
+
- no negative durations;
|
|
1741
|
+
- waiting plus active time is consistent with elapsed time within tolerance;
|
|
1742
|
+
- token trajectory is monotonic/semantically consistent;
|
|
1743
|
+
- context ratio remains bounded;
|
|
1744
|
+
- thinking spans are paired;
|
|
1745
|
+
- forensic conversion preserves identity and timestamps;
|
|
1746
|
+
- Prometheus projection rejects forbidden labels;
|
|
1747
|
+
- forensic-derived metrics appear in live scrape when events exist.
|
|
1748
|
+
|
|
1749
|
+
# 9. Workstream E — Evaluation platform
|
|
1750
|
+
|
|
1751
|
+
## 9.1 Evaluation subject
|
|
1752
|
+
|
|
1753
|
+
A candidate is not merely a model. It is the resolved combination:
|
|
1754
|
+
|
|
1755
|
+
```text
|
|
1756
|
+
model
|
|
1757
|
+
x specialist version
|
|
1758
|
+
x system prompt
|
|
1759
|
+
x mandatory rules
|
|
1760
|
+
x skills
|
|
1761
|
+
x runner injections
|
|
1762
|
+
x tool catalog
|
|
1763
|
+
x permission tier
|
|
1764
|
+
x thinking level
|
|
1765
|
+
x chain position
|
|
1766
|
+
x task/environment
|
|
1767
|
+
```
|
|
1768
|
+
|
|
1769
|
+
Every candidate must have a deterministic fingerprint.
|
|
1770
|
+
|
|
1771
|
+
## 9.2 Evaluation levels
|
|
1772
|
+
|
|
1773
|
+
### Outcome quality
|
|
1774
|
+
|
|
1775
|
+
Did the final environment satisfy the task? Examples: hidden tests, state assertions, schema validation, smoke/E2E, telemetry assertions, diff correctness and absence of regressions.
|
|
1776
|
+
|
|
1777
|
+
### Role behavior
|
|
1778
|
+
|
|
1779
|
+
Did the specialist obey its mandate and boundaries? Examples: no reviewer edits, no unauthorized test-engineer source changes, bounded seconder behavior, researcher source discipline.
|
|
1780
|
+
|
|
1781
|
+
### Chain contribution
|
|
1782
|
+
|
|
1783
|
+
Did the specialist consume the right upstream evidence, avoid duplication, detect contradictions and produce a useful downstream handoff?
|
|
1784
|
+
|
|
1785
|
+
### Efficiency and reliability
|
|
1786
|
+
|
|
1787
|
+
What were the token, time, tool, retry, compaction, stall and variance characteristics?
|
|
1788
|
+
|
|
1789
|
+
Efficiency is reported separately and cannot compensate for failed hard gates.
|
|
1790
|
+
|
|
1791
|
+
## 9.3 Evaluation entities
|
|
1792
|
+
|
|
1793
|
+
### Eval suite
|
|
1794
|
+
|
|
1795
|
+
A versioned collection of cases for a capability or regression domain.
|
|
1796
|
+
|
|
1797
|
+
### Eval case
|
|
1798
|
+
|
|
1799
|
+
A pinned task, environment, expected outcomes and grader configuration.
|
|
1800
|
+
|
|
1801
|
+
### Eval experiment
|
|
1802
|
+
|
|
1803
|
+
A controlled comparison of one or more candidate configurations.
|
|
1804
|
+
|
|
1805
|
+
### Eval trial
|
|
1806
|
+
|
|
1807
|
+
One candidate executing one case for one attempt.
|
|
1808
|
+
|
|
1809
|
+
### Eval artifact
|
|
1810
|
+
|
|
1811
|
+
A reference to output, trace, diff, commit, tests, logs, environment state or external source bundle.
|
|
1812
|
+
|
|
1813
|
+
### Eval score
|
|
1814
|
+
|
|
1815
|
+
One grader’s result for one trial and dimension.
|
|
1816
|
+
|
|
1817
|
+
### Pairwise comparison
|
|
1818
|
+
|
|
1819
|
+
A blinded judge preference between two trial outputs/evidence bundles.
|
|
1820
|
+
|
|
1821
|
+
### Human annotation
|
|
1822
|
+
|
|
1823
|
+
A gold label, adjudication or override with provenance.
|
|
1824
|
+
|
|
1825
|
+
## 9.4 Proposed SQLite schema
|
|
1826
|
+
|
|
1827
|
+
```sql
|
|
1828
|
+
CREATE TABLE eval_suites (...);
|
|
1829
|
+
CREATE TABLE eval_cases (...);
|
|
1830
|
+
CREATE TABLE eval_experiments (...);
|
|
1831
|
+
CREATE TABLE eval_trials (...);
|
|
1832
|
+
CREATE TABLE eval_scores (...);
|
|
1833
|
+
CREATE TABLE eval_pairwise_results (...);
|
|
1834
|
+
CREATE TABLE eval_human_annotations (...);
|
|
1835
|
+
```
|
|
1836
|
+
|
|
1837
|
+
Minimum fields are defined below.
|
|
1838
|
+
|
|
1839
|
+
### `eval_suites`
|
|
1840
|
+
|
|
1841
|
+
```text
|
|
1842
|
+
suite_id
|
|
1843
|
+
name
|
|
1844
|
+
participant_role
|
|
1845
|
+
suite_kind: regression | capability | telemetry | chain | external
|
|
1846
|
+
version
|
|
1847
|
+
dataset_hash
|
|
1848
|
+
rubric_hash
|
|
1849
|
+
created_at_ms
|
|
1850
|
+
```
|
|
1851
|
+
|
|
1852
|
+
### `eval_cases`
|
|
1853
|
+
|
|
1854
|
+
```text
|
|
1855
|
+
case_id
|
|
1856
|
+
suite_id
|
|
1857
|
+
case_version
|
|
1858
|
+
title
|
|
1859
|
+
fixture_ref
|
|
1860
|
+
task_contract_json
|
|
1861
|
+
environment_spec_json
|
|
1862
|
+
expected_outcome_json
|
|
1863
|
+
grader_config_json
|
|
1864
|
+
difficulty
|
|
1865
|
+
tags_json
|
|
1866
|
+
source_kind
|
|
1867
|
+
source_ref
|
|
1868
|
+
is_held_out
|
|
1869
|
+
```
|
|
1870
|
+
|
|
1871
|
+
### `eval_experiments`
|
|
1872
|
+
|
|
1873
|
+
```text
|
|
1874
|
+
experiment_id
|
|
1875
|
+
suite_id
|
|
1876
|
+
experiment_kind
|
|
1877
|
+
candidate_configs_json
|
|
1878
|
+
randomization_seed
|
|
1879
|
+
trials_per_case
|
|
1880
|
+
status
|
|
1881
|
+
started_at_ms
|
|
1882
|
+
completed_at_ms
|
|
1883
|
+
```
|
|
1884
|
+
|
|
1885
|
+
### `eval_trials`
|
|
1886
|
+
|
|
1887
|
+
```text
|
|
1888
|
+
trial_id
|
|
1889
|
+
experiment_id
|
|
1890
|
+
case_id
|
|
1891
|
+
candidate_id
|
|
1892
|
+
job_id
|
|
1893
|
+
chain_id
|
|
1894
|
+
model
|
|
1895
|
+
prompt_hash
|
|
1896
|
+
rules_hash
|
|
1897
|
+
skills_hash
|
|
1898
|
+
runtime_version
|
|
1899
|
+
tool_catalog_version
|
|
1900
|
+
environment_hash
|
|
1901
|
+
attempt_index
|
|
1902
|
+
status
|
|
1903
|
+
data_completeness
|
|
1904
|
+
started_at_ms
|
|
1905
|
+
completed_at_ms
|
|
1906
|
+
```
|
|
1907
|
+
|
|
1908
|
+
### `eval_scores`
|
|
1909
|
+
|
|
1910
|
+
```text
|
|
1911
|
+
score_id
|
|
1912
|
+
trial_id
|
|
1913
|
+
grader_id
|
|
1914
|
+
grader_type
|
|
1915
|
+
dimension
|
|
1916
|
+
score
|
|
1917
|
+
passed
|
|
1918
|
+
confidence
|
|
1919
|
+
evidence_json
|
|
1920
|
+
grader_version
|
|
1921
|
+
created_at_ms
|
|
1922
|
+
```
|
|
1923
|
+
|
|
1924
|
+
### `eval_pairwise_results`
|
|
1925
|
+
|
|
1926
|
+
```text
|
|
1927
|
+
comparison_id
|
|
1928
|
+
experiment_id
|
|
1929
|
+
case_id
|
|
1930
|
+
trial_a_id
|
|
1931
|
+
trial_b_id
|
|
1932
|
+
judge_id
|
|
1933
|
+
presentation_order
|
|
1934
|
+
winner: A | B | tie | insufficient_evidence
|
|
1935
|
+
confidence
|
|
1936
|
+
rationale_json
|
|
1937
|
+
```
|
|
1938
|
+
|
|
1939
|
+
## 9.5 Forensic eval event catalog
|
|
1940
|
+
|
|
1941
|
+
Add events:
|
|
1942
|
+
|
|
1943
|
+
```text
|
|
1944
|
+
eval.experiment.started
|
|
1945
|
+
eval.experiment.completed
|
|
1946
|
+
eval.experiment.failed
|
|
1947
|
+
eval.trial.started
|
|
1948
|
+
eval.trial.completed
|
|
1949
|
+
eval.trial.failed
|
|
1950
|
+
eval.grader.started
|
|
1951
|
+
eval.grader.completed
|
|
1952
|
+
eval.grader.failed
|
|
1953
|
+
eval.score.recorded
|
|
1954
|
+
eval.assertion.passed
|
|
1955
|
+
eval.assertion.failed
|
|
1956
|
+
eval.comparison.recorded
|
|
1957
|
+
eval.judge.disagreement
|
|
1958
|
+
eval.human.annotation.recorded
|
|
1959
|
+
eval.human.override.recorded
|
|
1960
|
+
eval.regression.detected
|
|
1961
|
+
eval.promotion.recommended
|
|
1962
|
+
eval.promotion.blocked
|
|
1963
|
+
```
|
|
1964
|
+
|
|
1965
|
+
Use `eval_id` and existing job/chain/commit correlation fields.
|
|
1966
|
+
|
|
1967
|
+
## 9.6 Grader hierarchy
|
|
1968
|
+
|
|
1969
|
+
Use the cheapest reliable grader first:
|
|
1970
|
+
|
|
1971
|
+
```text
|
|
1972
|
+
environment/state grader
|
|
1973
|
+
-> deterministic trace grader
|
|
1974
|
+
-> deterministic artifact grader
|
|
1975
|
+
-> model grader
|
|
1976
|
+
-> human adjudication
|
|
1977
|
+
```
|
|
1978
|
+
|
|
1979
|
+
### Environment/state graders
|
|
1980
|
+
|
|
1981
|
+
Examples:
|
|
1982
|
+
|
|
1983
|
+
- hidden tests;
|
|
1984
|
+
- expected database/file state;
|
|
1985
|
+
- smoke/E2E result;
|
|
1986
|
+
- emitted telemetry;
|
|
1987
|
+
- no leaked secret;
|
|
1988
|
+
- worktree and commit state.
|
|
1989
|
+
|
|
1990
|
+
### Trace graders
|
|
1991
|
+
|
|
1992
|
+
Examples:
|
|
1993
|
+
|
|
1994
|
+
- required tool called;
|
|
1995
|
+
- forbidden tool absent;
|
|
1996
|
+
- no edit/write by read-only role;
|
|
1997
|
+
- `sp result` used before `sp feed` where required;
|
|
1998
|
+
- memory recall performed only when triggered;
|
|
1999
|
+
- chain context consumed;
|
|
2000
|
+
- output schema valid;
|
|
2001
|
+
- tool-call budget respected.
|
|
2002
|
+
|
|
2003
|
+
### Artifact graders
|
|
2004
|
+
|
|
2005
|
+
Examples:
|
|
2006
|
+
|
|
2007
|
+
- changed paths match scope;
|
|
2008
|
+
- required tests added;
|
|
2009
|
+
- diff minimality heuristics;
|
|
2010
|
+
- evidence references resolve;
|
|
2011
|
+
- handoff fields complete.
|
|
2012
|
+
|
|
2013
|
+
### Model graders
|
|
2014
|
+
|
|
2015
|
+
Use for nuanced qualities:
|
|
2016
|
+
|
|
2017
|
+
- reviewer finding materiality and evidence quality;
|
|
2018
|
+
- researcher citation support and synthesis;
|
|
2019
|
+
- overthinker risk quality and depth calibration;
|
|
2020
|
+
- executor unnecessary complexity;
|
|
2021
|
+
- handoff actionability.
|
|
2022
|
+
|
|
2023
|
+
Model graders must be calibrated against human labels and versioned.
|
|
2024
|
+
|
|
2025
|
+
## 9.7 Hard gates and score vector
|
|
2026
|
+
|
|
2027
|
+
A trial first passes hard gates:
|
|
2028
|
+
|
|
2029
|
+
```text
|
|
2030
|
+
outcome correct
|
|
2031
|
+
no forbidden action
|
|
2032
|
+
schema valid
|
|
2033
|
+
no critical requirement missed
|
|
2034
|
+
no critical security/scope violation
|
|
2035
|
+
```
|
|
2036
|
+
|
|
2037
|
+
Then a quality vector is reported:
|
|
2038
|
+
|
|
2039
|
+
```text
|
|
2040
|
+
contract compliance
|
|
2041
|
+
role adherence
|
|
2042
|
+
evidence quality
|
|
2043
|
+
chain-context use
|
|
2044
|
+
handoff quality
|
|
2045
|
+
operational reliability
|
|
2046
|
+
```
|
|
2047
|
+
|
|
2048
|
+
Cost is reported separately:
|
|
2049
|
+
|
|
2050
|
+
```text
|
|
2051
|
+
input/output/reasoning/tool tokens
|
|
2052
|
+
active/waiting/elapsed time
|
|
2053
|
+
tool calls
|
|
2054
|
+
provider cost when authoritative
|
|
2055
|
+
```
|
|
2056
|
+
|
|
2057
|
+
A composite quality score may be displayed but must not replace dimension-level promotion rules.
|
|
2058
|
+
|
|
2059
|
+
## 9.8 Role-specific scorecards
|
|
2060
|
+
|
|
2061
|
+
### Reviewer
|
|
2062
|
+
|
|
2063
|
+
```text
|
|
2064
|
+
verdict accuracy
|
|
2065
|
+
blocking finding precision
|
|
2066
|
+
blocking finding recall
|
|
2067
|
+
false PASS rate
|
|
2068
|
+
false block rate
|
|
2069
|
+
upstream evidence coverage
|
|
2070
|
+
contradiction handling
|
|
2071
|
+
delta-review discipline
|
|
2072
|
+
evidence traceability
|
|
2073
|
+
next-action usefulness
|
|
2074
|
+
```
|
|
2075
|
+
|
|
2076
|
+
False PASS is the primary safety metric.
|
|
2077
|
+
|
|
2078
|
+
### Executor
|
|
2079
|
+
|
|
2080
|
+
```text
|
|
2081
|
+
acceptance pass rate
|
|
2082
|
+
scope precision and recall
|
|
2083
|
+
regression rate
|
|
2084
|
+
minimality
|
|
2085
|
+
repository convention adherence
|
|
2086
|
+
validation execution
|
|
2087
|
+
handoff completeness
|
|
2088
|
+
fix-loop count
|
|
2089
|
+
```
|
|
2090
|
+
|
|
2091
|
+
### Seconder
|
|
2092
|
+
|
|
2093
|
+
```text
|
|
2094
|
+
gate accuracy
|
|
2095
|
+
finding precision and recall
|
|
2096
|
+
boundedness
|
|
2097
|
+
context expansion discipline
|
|
2098
|
+
UNCLEAR calibration
|
|
2099
|
+
agreement with final outcome
|
|
2100
|
+
```
|
|
2101
|
+
|
|
2102
|
+
### Test-engineer
|
|
2103
|
+
|
|
2104
|
+
```text
|
|
2105
|
+
fail-to-pass effectiveness
|
|
2106
|
+
pass-to-pass preservation
|
|
2107
|
+
critical-path coverage
|
|
2108
|
+
mutation sensitivity
|
|
2109
|
+
command executability
|
|
2110
|
+
source-boundary violations
|
|
2111
|
+
failure-owner accuracy
|
|
2112
|
+
telemetry assertion quality
|
|
2113
|
+
harness reuse
|
|
2114
|
+
```
|
|
2115
|
+
|
|
2116
|
+
### Researcher
|
|
2117
|
+
|
|
2118
|
+
```text
|
|
2119
|
+
claim correctness
|
|
2120
|
+
citation precision and coverage
|
|
2121
|
+
source authority
|
|
2122
|
+
freshness
|
|
2123
|
+
fact/inference separation
|
|
2124
|
+
contradiction handling
|
|
2125
|
+
query efficiency
|
|
2126
|
+
```
|
|
2127
|
+
|
|
2128
|
+
### Overthinker
|
|
2129
|
+
|
|
2130
|
+
```text
|
|
2131
|
+
decision quality
|
|
2132
|
+
assumption discovery
|
|
2133
|
+
risk precision and recall
|
|
2134
|
+
alternative quality
|
|
2135
|
+
disconfirming evidence
|
|
2136
|
+
actionability
|
|
2137
|
+
depth calibration
|
|
2138
|
+
redundancy
|
|
2139
|
+
```
|
|
2140
|
+
|
|
2141
|
+
## 9.9 Regression and capability suites
|
|
2142
|
+
|
|
2143
|
+
Regression suites protect reliable behavior and should approach 100% pass rate. Capability suites are intentionally difficult and are used to improve performance.
|
|
2144
|
+
|
|
2145
|
+
Initial suites:
|
|
2146
|
+
|
|
2147
|
+
```text
|
|
2148
|
+
reviewer-regression-v1
|
|
2149
|
+
executor-regression-v1
|
|
2150
|
+
seconder-regression-v1
|
|
2151
|
+
test-engineer-regression-v1
|
|
2152
|
+
researcher-regression-v1
|
|
2153
|
+
overthinker-regression-v1
|
|
2154
|
+
chain-participation-v1
|
|
2155
|
+
telemetry-integrity-v1
|
|
2156
|
+
```
|
|
2157
|
+
|
|
2158
|
+
## 9.10 Dataset sources
|
|
2159
|
+
|
|
2160
|
+
Use real system failures first:
|
|
2161
|
+
|
|
2162
|
+
- reviewer PARTIAL followed by successful fix;
|
|
2163
|
+
- false PASS discovered later;
|
|
2164
|
+
- scope leak;
|
|
2165
|
+
- wrong diff or wrong worktree;
|
|
2166
|
+
- invalid handoff schema;
|
|
2167
|
+
- unnecessary or missing memory retrieval;
|
|
2168
|
+
- skipped mandatory gate;
|
|
2169
|
+
- contradictory gate outputs;
|
|
2170
|
+
- context overflow/compaction;
|
|
2171
|
+
- tool or extension failure;
|
|
2172
|
+
- testing ownership misclassification.
|
|
2173
|
+
|
|
2174
|
+
Add generated mutation cases:
|
|
2175
|
+
|
|
2176
|
+
- remove an upstream handoff;
|
|
2177
|
+
- make result contradict diff;
|
|
2178
|
+
- inject stale memory;
|
|
2179
|
+
- alter root scope;
|
|
2180
|
+
- add irrelevant relationship edges;
|
|
2181
|
+
- remove chain identity;
|
|
2182
|
+
- falsify a tool-use claim;
|
|
2183
|
+
- insert seeded functional or quality defects.
|
|
2184
|
+
|
|
2185
|
+
## 9.11 Retrospective evaluation
|
|
2186
|
+
|
|
2187
|
+
### Supported
|
|
2188
|
+
|
|
2189
|
+
Existing runs can be evaluated for dimensions whose evidence remains in the database:
|
|
2190
|
+
|
|
2191
|
+
- runtime and token efficiency;
|
|
2192
|
+
- tool use;
|
|
2193
|
+
- output schema;
|
|
2194
|
+
- role-boundary compliance;
|
|
2195
|
+
- handoff quality;
|
|
2196
|
+
- chain sequencing;
|
|
2197
|
+
- gate verdict distributions;
|
|
2198
|
+
- evidence references;
|
|
2199
|
+
- context and compaction behavior;
|
|
2200
|
+
- some reviewer and executor benchmark rows.
|
|
2201
|
+
|
|
2202
|
+
### Partial
|
|
2203
|
+
|
|
2204
|
+
Older runs may lack exact thinking spans, resolved prompt hashes, skill body hashes, complete forensic conversion or correlated tool IDs.
|
|
2205
|
+
|
|
2206
|
+
### Requires replay
|
|
2207
|
+
|
|
2208
|
+
Causal model comparisons and hidden outcome testing require a pinned repository snapshot and rerun. Historical cohort analysis must not be presented as a controlled A/B result.
|
|
2209
|
+
|
|
2210
|
+
### Backfill command
|
|
2211
|
+
|
|
2212
|
+
Recommended:
|
|
2213
|
+
|
|
2214
|
+
```bash
|
|
2215
|
+
sp eval backfill --repo .
|
|
2216
|
+
sp eval backfill --all-repos
|
|
2217
|
+
```
|
|
2218
|
+
|
|
2219
|
+
The command should calculate compatible deterministic graders, emit eval events and mark data completeness.
|
|
2220
|
+
|
|
2221
|
+
## 9.12 Continuous evaluation
|
|
2222
|
+
|
|
2223
|
+
### After every run
|
|
2224
|
+
|
|
2225
|
+
Run deterministic graders asynchronously after run-complete persistence:
|
|
2226
|
+
|
|
2227
|
+
```text
|
|
2228
|
+
schema validation
|
|
2229
|
+
role boundary
|
|
2230
|
+
forbidden path/tool
|
|
2231
|
+
handoff completeness
|
|
2232
|
+
startup context completeness
|
|
2233
|
+
telemetry integrity
|
|
2234
|
+
required evidence refs
|
|
2235
|
+
```
|
|
2236
|
+
|
|
2237
|
+
### After every chain
|
|
2238
|
+
|
|
2239
|
+
Run:
|
|
2240
|
+
|
|
2241
|
+
```text
|
|
2242
|
+
canonical gate sequence
|
|
2243
|
+
mandatory gate completion
|
|
2244
|
+
final verdict consistency
|
|
2245
|
+
unresolved failure state
|
|
2246
|
+
fix loops
|
|
2247
|
+
time and tokens to PASS
|
|
2248
|
+
chain outcome
|
|
2249
|
+
```
|
|
2250
|
+
|
|
2251
|
+
### Sampled/high-risk model graders
|
|
2252
|
+
|
|
2253
|
+
Run model-based quality graders on:
|
|
2254
|
+
|
|
2255
|
+
- all high/critical chains;
|
|
2256
|
+
- a configured percentage of medium chains;
|
|
2257
|
+
- detected anomalies/regressions;
|
|
2258
|
+
- explicit experiments.
|
|
2259
|
+
|
|
2260
|
+
Begin in shadow mode and do not block the main run.
|
|
2261
|
+
|
|
2262
|
+
## 9.13 Controlled A/B experiments
|
|
2263
|
+
|
|
2264
|
+
Use paired design:
|
|
2265
|
+
|
|
2266
|
+
```text
|
|
2267
|
+
same case
|
|
2268
|
+
same base SHA
|
|
2269
|
+
same task contract
|
|
2270
|
+
same upstream artifacts
|
|
2271
|
+
same tool catalog
|
|
2272
|
+
same budgets
|
|
2273
|
+
isolated worktrees
|
|
2274
|
+
```
|
|
2275
|
+
|
|
2276
|
+
Use multiple trials for stochastic agents.
|
|
2277
|
+
|
|
2278
|
+
Prompt and model changes should use factorial design where possible:
|
|
2279
|
+
|
|
2280
|
+
| | Current prompt | New prompt |
|
|
2281
|
+
|---|---:|---:|
|
|
2282
|
+
| Model A | A-current | A-new |
|
|
2283
|
+
| Model B | B-current | B-new |
|
|
2284
|
+
|
|
2285
|
+
This isolates model effect, prompt effect and interaction.
|
|
2286
|
+
|
|
2287
|
+
## 9.14 Pairwise judges
|
|
2288
|
+
|
|
2289
|
+
For subjective dimensions:
|
|
2290
|
+
|
|
2291
|
+
- hide model/provider/cost identity;
|
|
2292
|
+
- randomize A/B order;
|
|
2293
|
+
- repeat with swapped order;
|
|
2294
|
+
- allow tie and insufficient evidence;
|
|
2295
|
+
- use multiple judges for high-value cases;
|
|
2296
|
+
- route disagreement to human adjudication;
|
|
2297
|
+
- monitor position and verbosity bias.
|
|
2298
|
+
|
|
2299
|
+
## 9.15 Statistical reporting
|
|
2300
|
+
|
|
2301
|
+
Report:
|
|
2302
|
+
|
|
2303
|
+
```text
|
|
2304
|
+
baseline and candidate means
|
|
2305
|
+
paired delta
|
|
2306
|
+
bootstrap confidence interval
|
|
2307
|
+
win/tie/loss
|
|
2308
|
+
variance across trials
|
|
2309
|
+
failure category delta
|
|
2310
|
+
```
|
|
2311
|
+
|
|
2312
|
+
Use paired binary methods for pass/fail outcomes and paired bootstrap for continuous scores. Promotion requires practical as well as statistical significance.
|
|
2313
|
+
|
|
2314
|
+
## 9.16 Promotion policy
|
|
2315
|
+
|
|
2316
|
+
Example:
|
|
2317
|
+
|
|
2318
|
+
```yaml
|
|
2319
|
+
hard_requirements:
|
|
2320
|
+
critical_false_pass_increase: 0
|
|
2321
|
+
new_role_boundary_violations: 0
|
|
2322
|
+
regression_suite_lower_bound: 0.97
|
|
2323
|
+
output_schema_validity: 0.995
|
|
2324
|
+
quality:
|
|
2325
|
+
paired_quality_delta_lower_ci: -0.01
|
|
2326
|
+
chain_contribution_delta_lower_ci: 0
|
|
2327
|
+
pairwise_win_rate: 0.55
|
|
2328
|
+
efficiency:
|
|
2329
|
+
max_active_runtime_p50_regression: 0.20
|
|
2330
|
+
```
|
|
2331
|
+
|
|
2332
|
+
A cost increase requires a material quality improvement. A quality tie with meaningful cost reduction may justify promotion.
|
|
2333
|
+
|
|
2334
|
+
## 9.17 External and internal software-engineering benchmarks
|
|
2335
|
+
|
|
2336
|
+
External suites such as SWE-bench Pro public and Multi-SWE-bench can measure general model+harness coding capability. They should not replace internal role benchmarks.
|
|
2337
|
+
|
|
2338
|
+
The internal executor benchmark should use SWE-bench methodology:
|
|
2339
|
+
|
|
2340
|
+
```text
|
|
2341
|
+
real Bead/issue
|
|
2342
|
+
pinned repository and base SHA
|
|
2343
|
+
isolated environment
|
|
2344
|
+
hidden fail-to-pass tests
|
|
2345
|
+
pass-to-pass regression tests
|
|
2346
|
+
deterministic artifact checks
|
|
2347
|
+
full forensic trajectory
|
|
2348
|
+
```
|
|
2349
|
+
|
|
2350
|
+
Reviewer and seconder require seeded-patch benchmarks with gold defects and verdicts. Test-engineer requires buggy and fixed states to verify test sensitivity. Researcher requires source-backed claim sets. Chain eval requires mutated handoffs, missing gates and contradictions.
|
|
2351
|
+
|
|
2352
|
+
# 10. Console product requirements
|
|
2353
|
+
|
|
2354
|
+
## 10.1 Decision
|
|
2355
|
+
|
|
2356
|
+
Use the existing Console/Omniforge application. Do not create a separate eval application in the first implementation phase.
|
|
2357
|
+
|
|
2358
|
+
## 10.2 Data ownership
|
|
2359
|
+
|
|
2360
|
+
- Per-repo Specialists databases remain run-level sources of truth.
|
|
2361
|
+
- Specialists owns eval definitions, trial execution, grading events and detailed evidence.
|
|
2362
|
+
- The Console materializer projects summaries and forensic references into its read model.
|
|
2363
|
+
- Prometheus provides low-cardinality time-series aggregates.
|
|
2364
|
+
- Console provides operator UX and does not mutate source telemetry during normal reads.
|
|
2365
|
+
|
|
2366
|
+
## 10.3 Eval routes and surfaces
|
|
2367
|
+
|
|
2368
|
+
Add `/console/evals` with:
|
|
2369
|
+
|
|
2370
|
+
### Eval overview
|
|
2371
|
+
|
|
2372
|
+
Suite health, recent regressions, score by role/model, data freshness and missing signals.
|
|
2373
|
+
|
|
2374
|
+
### Experiment detail
|
|
2375
|
+
|
|
2376
|
+
Candidate fingerprints, case coverage, win/tie/loss, paired deltas, confidence intervals and promotion status.
|
|
2377
|
+
|
|
2378
|
+
### Model/prompt matrix
|
|
2379
|
+
|
|
2380
|
+
Role × model × prompt/rule/runtime version.
|
|
2381
|
+
|
|
2382
|
+
### Case explorer
|
|
2383
|
+
|
|
2384
|
+
Case source, difficulty, tags, expected outcome, fixture and grader definitions.
|
|
2385
|
+
|
|
2386
|
+
### Trial detail
|
|
2387
|
+
|
|
2388
|
+
Timeline of thinking, tools, turns and output; artifacts; diff; evidence; grader results; data completeness.
|
|
2389
|
+
|
|
2390
|
+
### Chain evaluation
|
|
2391
|
+
|
|
2392
|
+
Step timeline, queue/runtime, gates, contradictions, fix loops, time/tokens to PASS and final outcome.
|
|
2393
|
+
|
|
2394
|
+
### Historical mining
|
|
2395
|
+
|
|
2396
|
+
Search existing runs, inspect completeness and promote representative failures into eval cases.
|
|
2397
|
+
|
|
2398
|
+
### Live evaluation
|
|
2399
|
+
|
|
2400
|
+
Display post-run deterministic graders and experiment progress without blocking the originating job.
|
|
2401
|
+
|
|
2402
|
+
### Regression drift
|
|
2403
|
+
|
|
2404
|
+
Trend score, false-PASS rate, schema validity, tool/error metrics and cost over runtime/prompt/model versions.
|
|
2405
|
+
|
|
2406
|
+
## 10.4 Drilldown
|
|
2407
|
+
|
|
2408
|
+
Aggregate panels must drill down through evidence references to:
|
|
2409
|
+
|
|
2410
|
+
```text
|
|
2411
|
+
specialist job
|
|
2412
|
+
forensic event
|
|
2413
|
+
Bead/root/step
|
|
2414
|
+
chain
|
|
2415
|
+
commit/diff/PR
|
|
2416
|
+
test artifact
|
|
2417
|
+
eval trial and grader
|
|
2418
|
+
```
|
|
2419
|
+
|
|
2420
|
+
## 10.5 Realtime
|
|
2421
|
+
|
|
2422
|
+
The materializer and WebSocket hint path should publish after committed eval writes. Live views show status transitions, not raw unbounded forensic streams.
|
|
2423
|
+
|
|
2424
|
+
## 10.6 Prometheus versus forensic UI
|
|
2425
|
+
|
|
2426
|
+
Prometheus charts answer “is a class of runs getting worse?” Forensic/trial detail answers “what happened in this exact run?”. High-cardinality IDs stay in correlation and evidence refs.
|
|
2427
|
+
|
|
2428
|
+
# 11. CLI and API requirements
|
|
2429
|
+
|
|
2430
|
+
## 11.1 CLI
|
|
2431
|
+
|
|
2432
|
+
Recommended commands:
|
|
2433
|
+
|
|
2434
|
+
```bash
|
|
2435
|
+
sp eval suite list
|
|
2436
|
+
sp eval suite show <suite>
|
|
2437
|
+
sp eval case show <case>
|
|
2438
|
+
sp eval run <suite> --candidate <config> [--candidate <config>] --trials N
|
|
2439
|
+
sp eval compare <experiment-id>
|
|
2440
|
+
sp eval inspect <trial-id>
|
|
2441
|
+
sp eval failures <experiment-id>
|
|
2442
|
+
sp eval disagreements <experiment-id>
|
|
2443
|
+
sp eval backfill --repo .
|
|
2444
|
+
sp eval backfill --all-repos
|
|
2445
|
+
sp eval calibrate <rubric> --human-labels <file>
|
|
2446
|
+
sp eval promote <experiment-id> --policy <file>
|
|
2447
|
+
```
|
|
2448
|
+
|
|
2449
|
+
## 11.2 Existing DB commands
|
|
2450
|
+
|
|
2451
|
+
Retain and integrate:
|
|
2452
|
+
|
|
2453
|
+
```text
|
|
2454
|
+
sp db extract
|
|
2455
|
+
sp db stats
|
|
2456
|
+
sp db benchmark-export
|
|
2457
|
+
sp metrics --prometheus
|
|
2458
|
+
```
|
|
2459
|
+
|
|
2460
|
+
`benchmark-export` should either be versioned as a legacy exporter or rebuilt as an adapter over eval tables.
|
|
2461
|
+
|
|
2462
|
+
## 11.3 Evaluator interface
|
|
2463
|
+
|
|
2464
|
+
Conceptual TypeScript contract:
|
|
2465
|
+
|
|
2466
|
+
```ts
|
|
2467
|
+
interface EvalGrader {
|
|
2468
|
+
id: string;
|
|
2469
|
+
version: string;
|
|
2470
|
+
type: "environment" | "trace" | "artifact" | "model" | "human";
|
|
2471
|
+
supportedRoles?: string[];
|
|
2472
|
+
requiredEvidence: string[];
|
|
2473
|
+
grade(input: EvalTrialBundle): Promise<EvalScoreResult>;
|
|
2474
|
+
}
|
|
2475
|
+
```
|
|
2476
|
+
|
|
2477
|
+
## 11.4 Post-run evaluator
|
|
2478
|
+
|
|
2479
|
+
The supervisor or an event consumer emits run-complete, then enqueues evaluation. It must not recursively grade evaluator jobs. It must tolerate unavailable graders and record partial results.
|
|
2480
|
+
|
|
2481
|
+
## 11.5 Cross-repo experiments
|
|
2482
|
+
|
|
2483
|
+
Use globally unique experiment/eval IDs. Each repository retains local trial data; Console aggregates summaries by eval ID. A future state daemon may become the registry, but the initial bridge must not block on it.
|
|
2484
|
+
|
|
2485
|
+
|
|
2486
|
+
## 11.6 Interactive role composition and assignment
|
|
2487
|
+
|
|
2488
|
+
Recommended public surfaces:
|
|
2489
|
+
|
|
2490
|
+
```bash
|
|
2491
|
+
# Specialists-owned pure rendering
|
|
2492
|
+
sp role compose <role> --surface interactive --mode tracked --bead <id> --output <file> --json
|
|
2493
|
+
sp role compose <role> --surface interactive --mode standby --output <file> --json
|
|
2494
|
+
|
|
2495
|
+
# Core-owned launch/assignment UX
|
|
2496
|
+
xt pi --role <role> [--bead <id>] [--preheat] [--no-auto-assign]
|
|
2497
|
+
xt claude --role <role> [--bead <id>] [--preheat] [--no-auto-assign]
|
|
2498
|
+
xt role assign --target <pane-or-session> --role <role> --bead <id>
|
|
2499
|
+
```
|
|
2500
|
+
|
|
2501
|
+
The exact command name may change after CLI review, but ownership and semantics are fixed:
|
|
2502
|
+
|
|
2503
|
+
- Specialists composes content.
|
|
2504
|
+
- Core orchestrates launch and calls the renderer from the original repository cwd.
|
|
2505
|
+
- xtmux performs readiness-gated delivery and monitoring.
|
|
2506
|
+
- The assignment is one user turn, not a system-prompt append.
|
|
2507
|
+
|
|
2508
|
+
## 11.7 xtmux bridge query surfaces
|
|
2509
|
+
|
|
2510
|
+
The bridge may rely on current xtmux V2 machine queries:
|
|
2511
|
+
|
|
2512
|
+
```bash
|
|
2513
|
+
tmux-session-picker message-status <message-key>
|
|
2514
|
+
tmux-session-picker unread-count --for <session-id> [--pane <pane-id>]
|
|
2515
|
+
tmux-session-picker wait-agent <target> --timeout <duration>
|
|
2516
|
+
tmux-session-picker monitor-agent <target> --wait-for-transition --timeout <duration>
|
|
2517
|
+
```
|
|
2518
|
+
|
|
2519
|
+
Do not scrape formatted inbox output when a typed query exists.
|
|
2520
|
+
|
|
2521
|
+
# 12. Security, privacy and data governance
|
|
2522
|
+
|
|
2523
|
+
## 12.1 Prompt and output storage
|
|
2524
|
+
|
|
2525
|
+
Raw prompts and model outputs may contain repository-sensitive data. Store hashes and bounded metadata by default. Detailed raw artifacts remain in existing protected job/evidence storage and are not exported to Prometheus.
|
|
2526
|
+
|
|
2527
|
+
## 12.2 Forensic redaction
|
|
2528
|
+
|
|
2529
|
+
Continue using `xtrm.forensic.v1` redaction rules. Add grader output and model-judge prompts to the sensitive-field policy. Judge evidence bundles must be minimized and redacted.
|
|
2530
|
+
|
|
2531
|
+
## 12.3 Prometheus labels
|
|
2532
|
+
|
|
2533
|
+
Never label by:
|
|
2534
|
+
|
|
2535
|
+
```text
|
|
2536
|
+
job_id
|
|
2537
|
+
bead_id
|
|
2538
|
+
chain_id
|
|
2539
|
+
participant_id
|
|
2540
|
+
trace/span/tool-call/session/eval IDs
|
|
2541
|
+
raw path/command/error/diff/URL
|
|
2542
|
+
prompt or output content
|
|
2543
|
+
credentials or user identifiers
|
|
2544
|
+
```
|
|
2545
|
+
|
|
2546
|
+
## 12.4 Held-out cases
|
|
2547
|
+
|
|
2548
|
+
Held-out benchmark fixtures and hidden tests must not be exposed to the candidate agent. The runner must separate candidate worktree/tool access from grader assets.
|
|
2549
|
+
|
|
2550
|
+
## 12.5 Human annotation
|
|
2551
|
+
|
|
2552
|
+
Record annotator identity internally, but expose only the minimum needed in shared dashboards. Gold labels require provenance and versioning.
|
|
2553
|
+
|
|
2554
|
+
|
|
2555
|
+
## 12.6 Interactive assignment transport
|
|
2556
|
+
|
|
2557
|
+
Rendered Bead and mandatory-rule content is sensitive task material.
|
|
2558
|
+
|
|
2559
|
+
- Store it in a mode-restricted temporary/runtime file.
|
|
2560
|
+
- Deliver only a bounded one-line pointer into the interactive prompt field.
|
|
2561
|
+
- Hash the file at composition and verify before delivery where practical.
|
|
2562
|
+
- Delete or expire the transport artifact after acceptance according to a documented policy.
|
|
2563
|
+
- Record component IDs, hashes, sizes, readiness timestamps, delivery result and acknowledgement state; never record the full body.
|
|
2564
|
+
- A renderer error, missing Bead, malformed task template or fatal mandatory-rule resolution must fail before worktree/session provisioning in direct tracked mode.
|
|
2565
|
+
- Preheated standby mode may provision without a Bead but must not fabricate tracked context.
|
|
2566
|
+
|
|
2567
|
+
# 13. Integration with the Specialists roadmap, xtrm Stage 0 and Channels
|
|
2568
|
+
|
|
2569
|
+
## 13.1 Required direct documentation changes
|
|
2570
|
+
|
|
2571
|
+
Direct documentation changes are necessary, but they must preserve the distinct role of each source.
|
|
2572
|
+
|
|
2573
|
+
### `specialists-roadmap.md`
|
|
2574
|
+
|
|
2575
|
+
The canonical roadmap must be updated directly because it is the bridge-runtime source of truth and currently does not represent this modernization as an integrated, evaluation-gated program. The update should be concise and architectural rather than copying this PRD in full.
|
|
2576
|
+
|
|
2577
|
+
Required roadmap changes:
|
|
2578
|
+
|
|
2579
|
+
1. Add a prominent companion-program reference to this PRD and state that detailed work-package decomposition, graders and model/prompt promotion criteria live here.
|
|
2580
|
+
2. Add **Track C — Prompt, Chain Context, Telemetry and Evaluation Modernization** to the roadmap sequencing model.
|
|
2581
|
+
3. Record the ordering invariant:
|
|
2582
|
+
|
|
2583
|
+
```text
|
|
2584
|
+
telemetry integrity and historical baseline
|
|
2585
|
+
-> critical chain-first roadmap spine
|
|
2586
|
+
-> prompt/rule and memory changes under paired evaluation
|
|
2587
|
+
-> Channels replacement of bridge communication
|
|
2588
|
+
```
|
|
2589
|
+
|
|
2590
|
+
4. Mark the following as prerequisites for the complete chain-context implementation in this PRD:
|
|
2591
|
+
- Opportunity 2: READ_ONLY path binding;
|
|
2592
|
+
- Opportunity 3: persisted resolved chain shape;
|
|
2593
|
+
- Opportunity 4: composition gate;
|
|
2594
|
+
- Opportunity 5: authoritative step identity;
|
|
2595
|
+
- Opportunity 10: chain-driven dispatch.
|
|
2596
|
+
5. Retain Opportunity 8 only as a deliberately minimal bridge. Its payload must be compatible with future `work.turn`, `work.finding` and `work.verdict` messages, and it must not grow into a second scheduler.
|
|
2597
|
+
6. Add evaluation gates to roadmap changes that alter prompts, rules, memory injection, chain context or operator workflow. “Implemented” is not sufficient; the change must pass its regression and capability suites.
|
|
2598
|
+
7. Update status annotations against current code. Shipped QA pipeline, seconder fusion and forensic identity work must not be listed as unstarted. Conversely, persisted chain shape, composition gate, `--chain`, memory pull, XML contracts and Skills v4 must not be represented as complete unless verified.
|
|
2599
|
+
8. Reconcile Opportunities 1, 4 and 8 against existing `bd merge-slot`, `bd gate` and `bd swarm` before creating implementation Beads. The result must be a reuse decision record, not an assumption.
|
|
2600
|
+
9. Add bridge retirement criteria: every bridge must name the future owner, replacement trigger, migration/export path and deletion condition.
|
|
2601
|
+
10. Preserve the roadmap as architecture-level SSOT. Do not paste the scorecards, SQL schemas or complete work-package tables from this PRD into it.
|
|
2602
|
+
|
|
2603
|
+
### `xtrm/docs/_meta/2026-06-21-design-reconciliation.md`
|
|
2604
|
+
|
|
2605
|
+
The 21 June document is explicitly a dated snapshot and should not be rewritten as though it were a live status page. Several claims are now historical, including Console state and portions of Specialists implementation status.
|
|
2606
|
+
|
|
2607
|
+
Required treatment:
|
|
2608
|
+
|
|
2609
|
+
1. Add a short banner at the top stating that it is a historical snapshot and linking to a newer reconciliation document.
|
|
2610
|
+
2. Do not rewrite its original body or silently alter conclusions that were correct on 21 June.
|
|
2611
|
+
3. Create a new dated document, recommended path:
|
|
2612
|
+
|
|
2613
|
+
```text
|
|
2614
|
+
xtrm/docs/_meta/2026-07-13-specialists-roadmap-evals-integration.md
|
|
2615
|
+
```
|
|
2616
|
+
|
|
2617
|
+
4. The new reconciliation must record:
|
|
2618
|
+
- current verified roadmap status;
|
|
2619
|
+
- Track A/B/C ownership;
|
|
2620
|
+
- the dependency matrix in this section;
|
|
2621
|
+
- Stage 0 ownership and Channels storage decision;
|
|
2622
|
+
- the fact that Console now exists and already materializes Specialists observability;
|
|
2623
|
+
- the role of xtmux as a notification/transport adapter, not canonical Channels storage;
|
|
2624
|
+
- the current bridge-retirement plan;
|
|
2625
|
+
- links to this PRD and the canonical roadmap.
|
|
2626
|
+
5. If the organization prefers append-only meta history, the same content may be an addendum in a new file; it must not replace the June snapshot.
|
|
2627
|
+
|
|
2628
|
+
|
|
2629
|
+
## 13.1A Additional roadmap amendments required by the shipped interactive bridge
|
|
2630
|
+
|
|
2631
|
+
`specialists-roadmap.md` must also be amended to recognize that the system now has two bridge execution forms: managed `sp run` jobs and persistent interactive `xt --role` sessions.
|
|
2632
|
+
|
|
2633
|
+
Required additions:
|
|
2634
|
+
|
|
2635
|
+
1. Add a cross-cutting **Interactive Chain Coordinator Bridge** section.
|
|
2636
|
+
2. State that the current coordinator is epic/task-group scoped and applies future chain-coordinator duties across child chains; it is not yet the Substrate one-chain participant.
|
|
2637
|
+
3. Add the two launch modes: preheated standby and direct tracked `--bead` launch.
|
|
2638
|
+
4. Require task-side Bead context and mandatory rules to enter as a first user assignment, not system prompt content.
|
|
2639
|
+
5. Make the coordinator a primary consumer of Opportunities 2, 3, 4, 5, 8, 9, 10, 11, 12 and 16.
|
|
2640
|
+
6. Remove hardcoded chain sequences from coordinator guidance; the chain-template catalog and resolved shape are authoritative.
|
|
2641
|
+
7. Add scope validation, conflict matrix, peer hygiene communication, close-time evidence and parent escalation to coordinator responsibilities.
|
|
2642
|
+
8. Record xtmux as the current attention/transport layer and `observability.db` as the current Specialists chain/eval authority.
|
|
2643
|
+
9. Explicitly defer interactive permission enforcement and Substrate/Channels authority semantics.
|
|
2644
|
+
10. Add retirement mapping: assignment pointer delivery, xtmux mechanical messages and `step_completed` remain bridge mechanisms until Substrate/Channels provide participant activation, typed semantic messaging and scheduler intent.
|
|
2645
|
+
|
|
2646
|
+
## 13.1B Foundational hook architecture amendment — established before coordinator scope
|
|
2647
|
+
|
|
2648
|
+
The roadmap's Claude-only `bd create` hook proposal must be replaced using the foundational pattern established by the xtmux audit described near the beginning of this PRD. This is an independent roadmap correction and does not depend on the interactive coordinator:
|
|
2649
|
+
|
|
2650
|
+
```text
|
|
2651
|
+
shared pure policy kernel
|
|
2652
|
+
├── Claude Code cheap prefilter + Pre/PostToolUse adapters
|
|
2653
|
+
├── Pi in-process extension adapter
|
|
2654
|
+
└── authoritative sp chain review/run validator
|
|
2655
|
+
```
|
|
2656
|
+
|
|
2657
|
+
Requirements:
|
|
2658
|
+
|
|
2659
|
+
- One matcher/policy implementation for SCRUTINY, contract readiness, type/shape mismatch and template recommendation.
|
|
2660
|
+
- Cheap no-match path for Claude hooks; no unconditional Node cold start on every Bash call.
|
|
2661
|
+
- Pi uses in-process extensions where possible.
|
|
2662
|
+
- Early hook feedback is advisory or shadow/warn; final composition/dispatch validation remains in Specialists.
|
|
2663
|
+
- Durable policy findings use stable fingerprints and are persisted to forensic/eval state rather than printed and lost.
|
|
2664
|
+
- Rollout modes: `off → shadow → warn → enforce`.
|
|
2665
|
+
- Hooks do not decide final release authority or implement a second chain scheduler.
|
|
2666
|
+
|
|
2667
|
+
## 13.2 Dependency matrix
|
|
2668
|
+
|
|
2669
|
+
| PRD capability | Start now | Requires critical Specialists roadmap spine | Requires Stage 0 | Requires Channels 0.1 | Requires Channels 0.2 |
|
|
2670
|
+
|---|---:|---:|---:|---:|---:|
|
|
2671
|
+
| Telemetry correctness fixes | yes | no | no | no | no |
|
|
2672
|
+
| Historical run backfill and cohort analysis | yes | no | no | no | no |
|
|
2673
|
+
| Candidate/prompt/rule/skill fingerprinting | yes | no | no | no | no |
|
|
2674
|
+
| Deterministic post-run graders | yes | no | no | no | no |
|
|
2675
|
+
| Prompt composition inventory | yes | no | no | no | no |
|
|
2676
|
+
| Output-schema unification | yes | no | no | no | no |
|
|
2677
|
+
| Paired model A/B on controlled standalone cases | yes | no | no | no | no |
|
|
2678
|
+
| Minimal chain-participant guidance using current pointers | yes | partial | no | no | no |
|
|
2679
|
+
| Complete deterministic `<chain-context>` envelope | no | yes | no | no | no |
|
|
2680
|
+
| Reliable post-chain evaluator | partial | yes | no | no | no |
|
|
2681
|
+
| Pull-based memory experiment | yes | Opportunity 11 coordination | no | no | no |
|
|
2682
|
+
| Automatic reviewer-to-executor remediation | no | useful but insufficient | yes | yes | no |
|
|
2683
|
+
| Typed message/ack/delivery evaluation | no | no | yes | yes | no |
|
|
2684
|
+
| Freeform node collaboration evaluation | no | no | yes | yes | yes |
|
|
2685
|
+
| Console historical eval UI | yes after eval APIs | no | no | no | no |
|
|
2686
|
+
| Console live channel visualization | no | no | yes | yes | no |
|
|
2687
|
+
|
|
2688
|
+
## 13.3 Critical chain-first spine
|
|
2689
|
+
|
|
2690
|
+
The following is the minimum Track A foundation that must exist before this PRD can claim complete chain awareness:
|
|
2691
|
+
|
|
2692
|
+
```text
|
|
2693
|
+
bd primitive reuse decision
|
|
2694
|
+
-> READ_ONLY path binding
|
|
2695
|
+
-> persisted resolved chain shape
|
|
2696
|
+
-> composition review/approve/insert
|
|
2697
|
+
-> authoritative kind:step and semantic-edge lookup
|
|
2698
|
+
-> chain-driven dispatch and workspace resolution
|
|
2699
|
+
```
|
|
2700
|
+
|
|
2701
|
+
This spine provides stable chain identity and pointers. It does not require Channels. It enables the chain-context envelope, selective upstream retrieval, post-chain grading and reliable model comparisons within a fixed chain topology.
|
|
2702
|
+
|
|
2703
|
+
The complete roadmap is not a prerequisite for prompt modernization. `sp epic` rewrite, final branch naming, per-repo bootstrap cleanup, Skills v4 and all dispatch guardrails can follow after the critical spine. Prompt changes that do not depend on chain topology—especially researcher and overthinker—may enter experiments earlier once telemetry and eval baselines exist.
|
|
2704
|
+
|
|
2705
|
+
## 13.4 Channels boundary
|
|
2706
|
+
|
|
2707
|
+
Channels is the future semantic communication layer. This PRD must not create a parallel protocol in Specialists.
|
|
2708
|
+
|
|
2709
|
+
Until Channels 0.1 ships:
|
|
2710
|
+
|
|
2711
|
+
- `specialist_results` and Bead notes remain durable verbose handoff stores;
|
|
2712
|
+
- `sp result`, `sp feed`, Beads edges and the resolved chain shape provide pull-based context;
|
|
2713
|
+
- `step_completed` may provide a next-step recommendation and evaluation event;
|
|
2714
|
+
- the orchestrator remains responsible for dispatch/resume;
|
|
2715
|
+
- xtmux may notify operators or agents, but it is not the semantic source of truth.
|
|
2716
|
+
|
|
2717
|
+
After Channels 0.1:
|
|
2718
|
+
|
|
2719
|
+
- reviewer findings and verdicts become typed `work.finding`/`work.verdict` messages;
|
|
2720
|
+
- executor remediation is triggered through scheduler intent, not direct pane injection;
|
|
2721
|
+
- message delivery, acknowledgement, authority, stop-condition and convergence metrics join the eval framework;
|
|
2722
|
+
- bridge `step_completed` scheduling logic retires;
|
|
2723
|
+
- `sp steer` and `sp resume` can be implemented over the ChannelClient while retaining CLI compatibility.
|
|
2724
|
+
|
|
2725
|
+
The canonical Channels invariants must be preserved: participant L3 identity distinct from job L4 activation, typed message bodies, pointer-based payloads, read/ack separation, at-least-once delivery, scheduler indirection, authority derived from state rather than body text, forensic/evidence dual-write and explicit stop conditions.
|
|
2726
|
+
|
|
2727
|
+
## 13.5 xtmux integration boundary
|
|
2728
|
+
|
|
2729
|
+
xtmux already provides practical inter-agent messaging, acknowledgements, unread indicators, safe handoffs and a local event log. It should be used as an operator-facing bridge and as an experimental source of usability data.
|
|
2730
|
+
|
|
2731
|
+
Allowed integration:
|
|
2732
|
+
|
|
2733
|
+
- show a short notification when a canonical Channel message or Specialists handoff becomes available;
|
|
2734
|
+
- increment unread badges for a participant/session;
|
|
2735
|
+
- link to Bead, job, chain, eval or result identifiers;
|
|
2736
|
+
- measure acknowledgement latency and operator intervention;
|
|
2737
|
+
- provide safe pointer-based handoff during the bridge period;
|
|
2738
|
+
- expose multiplexing dashboard, collision audit and local transport health.
|
|
2739
|
+
|
|
2740
|
+
Forbidden integration:
|
|
2741
|
+
|
|
2742
|
+
- treating the rotating xtmux JSONL log as canonical chain state;
|
|
2743
|
+
- encoding release authority or gate satisfaction in pane metadata;
|
|
2744
|
+
- bypassing Channel scheduler/authority checks through direct `send-keys`;
|
|
2745
|
+
- using session names as canonical participant identity;
|
|
2746
|
+
- inventing message kinds incompatible with Channels.
|
|
2747
|
+
|
|
2748
|
+
The target relationship is:
|
|
2749
|
+
|
|
2750
|
+
```text
|
|
2751
|
+
Channels or Specialists durable state
|
|
2752
|
+
-> forensic/evidence event
|
|
2753
|
+
-> optional xtmux notification adapter
|
|
2754
|
+
-> human/agent attention
|
|
2755
|
+
```
|
|
2756
|
+
|
|
2757
|
+
# 14. Agent-ready decomposition contract
|
|
2758
|
+
|
|
2759
|
+
## 14.1 Required planning output
|
|
2760
|
+
|
|
2761
|
+
A local planning agent receiving this PRD must produce a Beads hierarchy, not implementation patches. Its output must contain:
|
|
2762
|
+
|
|
2763
|
+
1. One program epic for this PRD, or a linked set of repository-owned epics when cross-repo ownership requires separation.
|
|
2764
|
+
2. Root change Beads corresponding to the work packages in Section 15.
|
|
2765
|
+
3. Explicit dependency edges between root Beads.
|
|
2766
|
+
4. `recommended_template` for every root Bead, validated against the installed formula catalog where possible.
|
|
2767
|
+
5. Priority, scrutiny, owning repository, intended specialist role and parallel lane.
|
|
2768
|
+
6. File/surface ownership and collision risk.
|
|
2769
|
+
7. Deterministic validation and eval gates.
|
|
2770
|
+
8. Non-goals and bridge-retirement requirements.
|
|
2771
|
+
9. Integration Beads for work that crosses repository boundaries.
|
|
2772
|
+
10. A final plan-validation report proving that every PRD acceptance criterion maps to at least one Bead.
|
|
2773
|
+
11. A versioned `bead-graph-plan.json` and the captured `bd create --graph --dry-run --json` preview for the proposed hierarchy.
|
|
2774
|
+
12. A Beads capability matrix recording the installed version, supported primitives, minimum-version constraints and fallbacks.
|
|
2775
|
+
|
|
2776
|
+
The planner must not create vague tasks such as “implement evals” or “improve reviewer”. Each root Bead must have a falsifiable externally observable outcome.
|
|
2777
|
+
|
|
2778
|
+
## 14.2 Hierarchy and identity
|
|
2779
|
+
|
|
2780
|
+
Use this hierarchy:
|
|
2781
|
+
|
|
2782
|
+
```text
|
|
2783
|
+
program epic
|
|
2784
|
+
-> repository/workstream epic when needed
|
|
2785
|
+
-> root change Bead
|
|
2786
|
+
-> chain molecule and step Beads at composition time
|
|
2787
|
+
```
|
|
2788
|
+
|
|
2789
|
+
A root Bead is the durable work contract. A chain molecule is an execution instance around that contract. Step Beads are role-specific participation contracts. Do not conflate the program epic, chain molecule, root issue and specialist step.
|
|
2790
|
+
|
|
2791
|
+
Before the composition gate exists, manual bridge execution may create explicit step/tracking Beads. Such Beads must be labeled `bridge:manual-composition`, linked to the root and treated as temporary orchestration scaffolding. They must not become the permanent data model.
|
|
2792
|
+
|
|
2793
|
+
## 14.3 Root change-contract template
|
|
2794
|
+
|
|
2795
|
+
Every root Bead produced from this PRD must contain equivalent fields. XML may be stored directly in the Bead description even before a deterministic parser ships because Beads stores text and the future schema is forward-compatible.
|
|
2796
|
+
|
|
2797
|
+
```xml
|
|
2798
|
+
<change-contract work-package="WP-XX" type="feature|bug|refactor|research|docs" scrutiny="low|medium|high|critical">
|
|
2799
|
+
<problem>Concrete current defect or missing capability.</problem>
|
|
2800
|
+
<scope>
|
|
2801
|
+
<repo>xtrm-dev/specialists</repo>
|
|
2802
|
+
<path>explicit/file/or/directory</path>
|
|
2803
|
+
</scope>
|
|
2804
|
+
<non-goals>
|
|
2805
|
+
<item>Explicitly excluded adjacent work.</item>
|
|
2806
|
+
</non-goals>
|
|
2807
|
+
<dependencies>
|
|
2808
|
+
<work-package>WP-YY</work-package>
|
|
2809
|
+
</dependencies>
|
|
2810
|
+
<deliverables>
|
|
2811
|
+
<item>Code, schema, docs, fixture or report.</item>
|
|
2812
|
+
</deliverables>
|
|
2813
|
+
<validation>
|
|
2814
|
+
<criterion>Exact deterministic command or state assertion.</criterion>
|
|
2815
|
+
<criterion>Named eval suite and threshold.</criterion>
|
|
2816
|
+
</validation>
|
|
2817
|
+
<acceptance>
|
|
2818
|
+
<criterion>Externally observable completion condition.</criterion>
|
|
2819
|
+
</acceptance>
|
|
2820
|
+
<rollback>How to disable or revert safely.</rollback>
|
|
2821
|
+
<retirement-trigger>Required for bridge-only work.</retirement-trigger>
|
|
2822
|
+
</change-contract>
|
|
2823
|
+
```
|
|
2824
|
+
|
|
2825
|
+
## 14.4 Step-contract template
|
|
2826
|
+
|
|
2827
|
+
When a chain is composed, each specialist step must receive:
|
|
2828
|
+
|
|
2829
|
+
```xml
|
|
2830
|
+
<step-contract role="executor|seconder|test-engineer|test-runner|reviewer|researcher|overthinker|...">
|
|
2831
|
+
<mandate>One bounded responsibility.</mandate>
|
|
2832
|
+
<inputs>
|
|
2833
|
+
<item>Root contract and named upstream evidence.</item>
|
|
2834
|
+
</inputs>
|
|
2835
|
+
<outputs>
|
|
2836
|
+
<item>Typed artifact or verdict.</item>
|
|
2837
|
+
</outputs>
|
|
2838
|
+
<scope>
|
|
2839
|
+
<path>Allowed paths or READ_ONLY evidence surface.</path>
|
|
2840
|
+
</scope>
|
|
2841
|
+
<non-goals>
|
|
2842
|
+
<item>Actions owned by other chain members.</item>
|
|
2843
|
+
</non-goals>
|
|
2844
|
+
<validation>
|
|
2845
|
+
<criterion>Evidence the next member can verify.</criterion>
|
|
2846
|
+
</validation>
|
|
2847
|
+
</step-contract>
|
|
2848
|
+
```
|
|
2849
|
+
|
|
2850
|
+
The step contract must not restate the entire PRD. It references the root Bead and only adds role-specific obligations.
|
|
2851
|
+
|
|
2852
|
+
## 14.5 Required Bead metadata
|
|
2853
|
+
|
|
2854
|
+
Each root Bead should expose, through labels, fields or a structured note:
|
|
2855
|
+
|
|
2856
|
+
```json
|
|
2857
|
+
{
|
|
2858
|
+
"work_package": "WP-T02",
|
|
2859
|
+
"program": "specialists-modernization",
|
|
2860
|
+
"repository": "xtrm-dev/specialists",
|
|
2861
|
+
"lane": "telemetry-thinking",
|
|
2862
|
+
"priority": "P1",
|
|
2863
|
+
"scrutiny": "high",
|
|
2864
|
+
"recommended_template": "code-standard",
|
|
2865
|
+
"primary_role": "executor",
|
|
2866
|
+
"integration_owner": "telemetry-integrator",
|
|
2867
|
+
"parallel_safe": true,
|
|
2868
|
+
"file_ownership": ["src/pi/session.ts", "src/specialist/timeline-events.ts"],
|
|
2869
|
+
"eval_gates": ["telemetry-integrity-v1"],
|
|
2870
|
+
"bridge": false
|
|
2871
|
+
}
|
|
2872
|
+
```
|
|
2873
|
+
|
|
2874
|
+
Where Beads does not support a native field, store the value in labels or the structured description rather than dropping it.
|
|
2875
|
+
|
|
2876
|
+
## 14.6 Decomposition quality gates
|
|
2877
|
+
|
|
2878
|
+
Before implementation starts, a planning/reviewer pass must verify:
|
|
2879
|
+
|
|
2880
|
+
- every work package is represented exactly once;
|
|
2881
|
+
- the installed Beads version and required capabilities are recorded;
|
|
2882
|
+
- the proposed graph passes `bd create --graph --dry-run` structural validation before live application;
|
|
2883
|
+
- no acceptance criterion is orphaned;
|
|
2884
|
+
- dependency cycles do not exist;
|
|
2885
|
+
- no two parallel Beads claim exclusive ownership of the same high-conflict file;
|
|
2886
|
+
- all cross-repo work has an explicit integration owner;
|
|
2887
|
+
- bridge work names its retirement trigger;
|
|
2888
|
+
- prompt/model changes have baseline and promotion evals;
|
|
2889
|
+
- DB migrations have compatibility and rollback tests;
|
|
2890
|
+
- Console work consumes upstream data rather than synthesizing source truth;
|
|
2891
|
+
- Channels work is not smuggled into Specialists bridge Beads.
|
|
2892
|
+
|
|
2893
|
+
## 14.7 Implementation handoff requirements
|
|
2894
|
+
|
|
2895
|
+
Every implementation agent must leave a structured handoff with:
|
|
2896
|
+
|
|
2897
|
+
```json
|
|
2898
|
+
{
|
|
2899
|
+
"work_package": "WP-XX",
|
|
2900
|
+
"status": "success|partial|failed|waiting",
|
|
2901
|
+
"files_changed": [],
|
|
2902
|
+
"contracts_or_schemas_changed": [],
|
|
2903
|
+
"decisions": [],
|
|
2904
|
+
"assumptions": [],
|
|
2905
|
+
"validation_commands": [],
|
|
2906
|
+
"validation_results": [],
|
|
2907
|
+
"eval_results": [],
|
|
2908
|
+
"evidence_refs": [],
|
|
2909
|
+
"known_deferred_paths": [],
|
|
2910
|
+
"downstream_attention": [],
|
|
2911
|
+
"bridge_retirement_notes": []
|
|
2912
|
+
}
|
|
2913
|
+
```
|
|
2914
|
+
|
|
2915
|
+
The handoff contains decision rationale and evidence, not private chain-of-thought.
|
|
2916
|
+
|
|
2917
|
+
# 15. Work packages and multiplexed execution plan
|
|
2918
|
+
|
|
2919
|
+
## 15.1 Work-package conventions
|
|
2920
|
+
|
|
2921
|
+
The identifiers below are stable planning anchors. The local planner may create repository-specific Bead IDs, but each Bead title or metadata must retain its `WP-*` identifier so reports, evals and integration work can map back to this PRD.
|
|
2922
|
+
|
|
2923
|
+
Scrutiny defaults:
|
|
2924
|
+
|
|
2925
|
+
- **critical:** migrations, lifecycle/identity, authority, worktree safety, evaluator enforcement;
|
|
2926
|
+
- **high:** telemetry semantics, output contracts, chain state, prompt/rule changes for reviewer/executor;
|
|
2927
|
+
- **medium:** isolated CLI, fixtures, role prompts with bounded blast radius, Console read surfaces;
|
|
2928
|
+
- **low:** non-normative documentation or display-only changes.
|
|
2929
|
+
|
|
2930
|
+
## 15.2 Gate and documentation packages
|
|
2931
|
+
|
|
2932
|
+
| WP | Deliverable | Repo | Dependencies | Parallel lane | Suggested chain | Scrutiny |
|
|
2933
|
+
|---|---|---|---|---|---|---|
|
|
2934
|
+
| `WP-G00` | Patch canonical roadmap with Track C, dependency rules, status corrections and PRD reference | specialists | none | docs-specialists | doc-sync or code-quick | medium |
|
|
2935
|
+
| `WP-G01` | Add historical-snapshot banner to 2026-06-21 reconciliation and create new dated reconciliation | xtrm | none | docs-xtrm | doc-sync | medium |
|
|
2936
|
+
| `WP-G02` | Produce binding Beads capability/reuse decision covering merge-slot, gate, swarm/molecule, ready `--mol/--gated/--claim/--explain`, graph create/dry-run, fan-in gates, claim leases, formula schema, history events, memory and backend capabilities; record minimum version/pin and fallbacks | specialists | none | architecture-gate | research-only + premortem | high |
|
|
2937
|
+
| `WP-G03` | Register this PRD in roadmap/MOC indexes and map all acceptance criteria to WPs | specialists + xtrm | G00,G01 | docs-integration | doc-sync | medium |
|
|
2938
|
+
|
|
2939
|
+
`WP-G02` is a hard planning gate for chain-foundation implementation, not for telemetry baselining. Its output includes the local Beads capability snapshot, stable-versus-pinned adoption table and the decision on which Specialists code paths are deleted or reduced through native reuse.
|
|
2940
|
+
|
|
2941
|
+
## 15.3 Telemetry-integrity packages
|
|
2942
|
+
|
|
2943
|
+
| WP | Deliverable | Primary files/surfaces | Dependencies | Parallel safety | Eval gate |
|
|
2944
|
+
|---|---|---|---|---|---|
|
|
2945
|
+
| `WP-T01` | Correct unique tool-call accounting and lifecycle dedupe | timeline events, supervisor callbacks, job metric aggregation | none | high-conflict owner | tool lifecycle fixtures |
|
|
2946
|
+
| `WP-T02` | Add thinking start/end spans, segment identity and durations | Pi session, timeline event schema, supervisor mapping | none | separate from T01 only with explicit file split | thinking-span fixtures |
|
|
2947
|
+
| `WP-T03` | Add turn/message/tool timing and time-to-first-action metrics | event constructors and aggregation | T01,T02 interfaces stable | medium | timing reconciliation |
|
|
2948
|
+
| `WP-T04` | Correct activity/stall semantics including `think`/`thinking` | observability queries and watchdog fixtures | none | parallel-safe if query-only | activity fixtures |
|
|
2949
|
+
| `WP-T05` | Wire forensic events into live Prometheus projection | forensic reader, projection collector, metrics tests | none | isolated projection owner | scrape contract |
|
|
2950
|
+
| `WP-T06` | Persist candidate fingerprint: model, prompt, rule, skill, runtime, tool catalog hashes | loader/runner/startup payload/schema | none | high-conflict with prompt inventory; same owner preferred | fingerprint reproducibility |
|
|
2951
|
+
| `WP-T07` | Ship `telemetry-integrity-v1` suite and compatibility/version markers | tests/fixtures/docs | T01–T06 | integrator-only | full suite |
|
|
2952
|
+
|
|
2953
|
+
`WP-T01`, `WP-T02` and `WP-T06` touch central runtime surfaces and must not be assigned to independent agents without a written file-ownership partition. The safer pattern is one runtime telemetry owner plus separate test/fixture agents.
|
|
2954
|
+
|
|
2955
|
+
## 15.4 Evaluation-core packages
|
|
2956
|
+
|
|
2957
|
+
| WP | Deliverable | Dependencies | Parallel lane | Suggested role/template | Scrutiny |
|
|
2958
|
+
|---|---|---|---|---|---|
|
|
2959
|
+
| `WP-E01` | Eval tables, migrations, DAO and versioned types | T06 schema conventions | eval-storage | executor + seconder + tests | critical |
|
|
2960
|
+
| `WP-E02` | Eval forensic event catalog and shared writer | E01 type IDs may be stubbed | eval-events | executor | high |
|
|
2961
|
+
| `WP-E03` | Grader interface, registry and deterministic assertion result schema | none | grader-core | executor | high |
|
|
2962
|
+
| `WP-E04` | Data-completeness model and historical backfill | E01,E03,T07 | eval-backfill | executor/test-engineer | high |
|
|
2963
|
+
| `WP-E05` | Post-run evaluator in observe-only mode | E01–E03,T07 | eval-runtime | executor | high |
|
|
2964
|
+
| `WP-E06` | Post-chain evaluator over resolved shape | E01–E03,C03,C06 | eval-chain | executor | high |
|
|
2965
|
+
| `WP-E07` | Paired experiment runner, randomization and candidate manifest | E01,E03,T06 | experiment-runner | executor | high |
|
|
2966
|
+
| `WP-E08` | Statistics, pairwise judge records and promotion policy engine | E07 | experiment-analysis | executor/researcher | high |
|
|
2967
|
+
| `WP-E09` | `sp eval` CLI: suite/case/run/compare/inspect/backfill | E01,E03,E04,E07 | eval-cli | executor | medium |
|
|
2968
|
+
| `WP-E10` | Continuous eval scheduling and recursion exclusion | E05,E06 | eval-ops | executor/test-runner | critical |
|
|
2969
|
+
|
|
2970
|
+
`WP-E01` owns eval migrations. No other lane may independently edit the same migration block; dependent work must consume exported types or coordinate through the integration owner.
|
|
2971
|
+
|
|
2972
|
+
## 15.5 Prompt and policy packages
|
|
2973
|
+
|
|
2974
|
+
| WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
|
|
2975
|
+
|---|---|---|---|---|
|
|
2976
|
+
| `WP-P01` | Resolved prompt manifest with component hashes/tokens and duplicate/conflict detection | T06 | prompt-inventory | high |
|
|
2977
|
+
| `WP-P02` | Canonical base status/verdict vocabulary and formal merged JSON Schema | E03 optional | output-contract | critical |
|
|
2978
|
+
| `WP-P03` | Replace duplicate manual handoff/output rules with generated contract as SSOT | P02 | output-contract | high |
|
|
2979
|
+
| `WP-P04` | Role-aware runner injections: style, GitNexus, Beads close, read-only/research behavior | P01,P02,T07 | runner-policy | critical |
|
|
2980
|
+
| `WP-P05` | Overthinker and researcher slim prompts plus procedures moved to skills | P01–P04,E05 | prompt-advisors | medium/high |
|
|
2981
|
+
| `WP-P06` | Seconder and test-engineer slim prompts; formal test-engineer schema and mode | P01–P04,E05 | prompt-gates | high |
|
|
2982
|
+
| `WP-P07` | Executor slim prompt and language/procedure skill separation | P01–P04,E05 | prompt-executor | high |
|
|
2983
|
+
| `WP-P08` | Reviewer slim prompt, upstream-evidence contract and false-PASS safeguards | P01–P04,E05,C06 desirable | prompt-reviewer | critical |
|
|
2984
|
+
| `WP-P09` | Factorial prompt×model experiments and staged promotion for all six roles | P05–P08,E07,E08 | prompt-eval | high |
|
|
2985
|
+
|
|
2986
|
+
Role prompt files are parallel-safe only after `WP-P02` and `WP-P04` freeze shared contracts. Each role owns its own specialist JSON and role-specific tests; one integration owner handles shared mandatory-rule/index edits.
|
|
2987
|
+
|
|
2988
|
+
## 15.6 Critical chain-foundation packages
|
|
2989
|
+
|
|
2990
|
+
| WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
|
|
2991
|
+
|---|---|---|---|---|
|
|
2992
|
+
| `WP-C01` | READ_ONLY path binding independent of owner liveness | G02 | chain-workspace | critical |
|
|
2993
|
+
| `WP-C02` | Persisted resolved chain shape with ordered steps/gates/status/job pointers | G02 | chain-state | critical |
|
|
2994
|
+
| `WP-C03` | `sp chain review/approve/insert` using resolved shape plus native Beads graph/gate/readiness primitives; no duplicate blocker engine | C02,G02 | chain-composition | critical |
|
|
2995
|
+
| `WP-C04` | Authoritative `kind:step`, root/step resolution and semantic-edge query helpers | G02 | chain-graph | high |
|
|
2996
|
+
| `WP-C05` | Chain-driven dispatch/workspace resolution using `bd ready --mol/--gated/--claim` as readiness/claim authority, with deprecation bridge for job/worktree flags | C01–C04 | chain-dispatch | critical |
|
|
2997
|
+
| `WP-C06` | Deterministic `<chain-context>` envelope with completeness flags and upstream pointers | C02,C04,T06 | chain-context | high |
|
|
2998
|
+
| `WP-C07` | Shared chain-participant rule and role-specific identity clauses | C06,P04 | chain-policy | high |
|
|
2999
|
+
| `WP-C08` | Structured handoff extension for decisions, inputs consumed and downstream attention | P02,C06 | handoff-contract | high |
|
|
3000
|
+
| `WP-C09` | Minimal `step_completed` visibility/evidence/recommendation bridge; no scheduler or duplicate readiness computation; compatible with Channels | C02,C03,E02 | chain-events | medium/high |
|
|
3001
|
+
| `WP-C10` | `chain-participation-v1` eval suite | C06–C09,E03 | chain-eval | high |
|
|
3002
|
+
|
|
3003
|
+
`WP-C09` must not auto-resume peers or implement a scheduler. Its retirement trigger is Channels 0.1 acceptance.
|
|
3004
|
+
|
|
3005
|
+
## 15.7 Pull-based memory packages
|
|
3006
|
+
|
|
3007
|
+
| WP | Deliverable | Dependencies | Parallel lane | Scrutiny |
|
|
3008
|
+
|---|---|---|---|---|
|
|
3009
|
+
| `WP-M01` | Audit actual `bd memories`/`bd recall`/guarded `bd remember` behavior, provenance gaps, filtering, ranking and backend/version constraints before proposing a wrapper | none | memory-research | medium |
|
|
3010
|
+
| `WP-M02` | Targeted memory search/relevant interface or wrapper with bounded output | M01 | memory-cli | high |
|
|
3011
|
+
| `WP-M03` | Disable eager injection behind experiment flag and preserve rollback | M01,T06 | memory-runtime | high |
|
|
3012
|
+
| `WP-M04` | Memory-recall mandatory rule, role opt-outs and authority/provenance policy | M01,P04 | memory-policy | high |
|
|
3013
|
+
| `WP-M05` | Memory query/recall telemetry and deterministic graders | M02–M04,E03 | memory-eval | high |
|
|
3014
|
+
| `WP-M06` | Role-by-role A/B rollout and promotion | M05,E07,E08 | memory-experiment | high |
|
|
3015
|
+
|
|
3016
|
+
## 15.8 Role-suite packages
|
|
3017
|
+
|
|
3018
|
+
These packages are fixture/dataset heavy and can run in parallel after the grader interface stabilizes.
|
|
3019
|
+
|
|
3020
|
+
| WP | Suite | Core outcome |
|
|
3021
|
+
|---|---|---|
|
|
3022
|
+
| `WP-S01` | `reviewer-regression-v1` | verdict accuracy, false PASS, finding precision/recall |
|
|
3023
|
+
| `WP-S02` | `executor-swe-style-v1` | hidden tests, scope precision, regression and minimality |
|
|
3024
|
+
| `WP-S03` | `seconder-regression-v1` | bounded gate accuracy and calibration |
|
|
3025
|
+
| `WP-S04` | `test-engineer-regression-v1` | fail-to-pass, pass-to-pass and source-boundary discipline |
|
|
3026
|
+
| `WP-S05` | `researcher-regression-v1` | claim correctness, citation entailment, freshness and inference separation |
|
|
3027
|
+
| `WP-S06` | `overthinker-capability-v1` | risk/assumption recall, precision and depth calibration |
|
|
3028
|
+
| `WP-S07` | `chain-participation-v1` | chain identity, selective retrieval, contradiction handling and handoff |
|
|
3029
|
+
| `WP-S08` | `telemetry-integrity-v1` | metric/event correctness and reconciliation |
|
|
3030
|
+
|
|
3031
|
+
All suite cases require versioned provenance. Capability cases that become reliably solved should be promoted into regression suites.
|
|
3032
|
+
|
|
3033
|
+
## 15.9 Console packages
|
|
3034
|
+
|
|
3035
|
+
| WP | Deliverable | Repo | Dependencies | Scrutiny |
|
|
3036
|
+
|---|---|---|---|---|
|
|
3037
|
+
| `WP-U01` | Materialize eval summaries, completeness and evidence refs | console | E01,E02 | high |
|
|
3038
|
+
| `WP-U02` | Eval query APIs and DTOs | console | U01 | high |
|
|
3039
|
+
| `WP-U03` | Eval overview, experiment, case and trial routes | console | U02 | medium |
|
|
3040
|
+
| `WP-U04` | Trial timeline with thinking/tool/text/evidence drilldown | console | U02,T07 | high |
|
|
3041
|
+
| `WP-U05` | Historical mining and case-promotion workflow | console | U02,E04 | medium/high |
|
|
3042
|
+
| `WP-U06` | Live eval updates, adjudication and promotion UX | console | U02,E08,E10 | high |
|
|
3043
|
+
|
|
3044
|
+
Console does not own evaluator execution or canonical scores. It reads/materializes Specialists output and links to forensic evidence.
|
|
3045
|
+
|
|
3046
|
+
|
|
3047
|
+
## 15.10 Interactive coordinator and skill packages
|
|
3048
|
+
|
|
3049
|
+
| WP | Deliverable | Repo | Dependencies | Parallel lane | Scrutiny |
|
|
3050
|
+
|---|---|---|---|---|---|
|
|
3051
|
+
| `WP-IC01` | Audit every `using-specialists` heading; produce content migration and prompt-envelope parity matrices | specialists + core | G00 baseline | coordinator-audit | high |
|
|
3052
|
+
| `WP-IC02` | Implement compact progressive-disclosure `using-specialists` tree and global recursive installation | specialists | IC01 | skill-layout | high |
|
|
3053
|
+
| `WP-IC03` | Rewrite `chain-coordinator` stable prompt; remove hardcoded chain sequence | specialists | IC01 | coordinator-prompt | high |
|
|
3054
|
+
| `WP-IC04` | Add coordinator-specific chain-template/scope/conflict/close guidance using native `bd ready --mol --explain`, `--gated`, gate/history queries when supported | specialists | IC01,IC03,G02 | coordinator-skill | high |
|
|
3055
|
+
| `WP-IC05` | Expose pure interactive task/standby renderer reusing `sp run` task-side assembly | specialists | IC01,T06 | role-renderer | critical |
|
|
3056
|
+
| `WP-IC06` | Audit/deduplicate coordinator `task_template` Bead rendering | specialists | IC05 | role-renderer | high |
|
|
3057
|
+
| `WP-IC07` | Add startup `starting → agent.ready → idle` handshake for Pi and Claude | core + xtmux | none | readiness | critical |
|
|
3058
|
+
| `WP-IC08` | Extend xtmux handoff for an existing prompt file, readiness wait and monitor integration | xtmux | IC07 | assignment-delivery | high |
|
|
3059
|
+
| `WP-IC09` | Integrate Core direct `--bead` auto-assignment through Specialists renderer + xtmux secure delivery | core | IC05,IC07,IC08 | role-launcher | critical |
|
|
3060
|
+
| `WP-IC10` | Implement preheated standby bootstrap, `role.ready`, later `xt role assign` and one-epic lifecycle | specialists + core + xtmux | IC05,IC07,IC08 | role-preheat | high |
|
|
3061
|
+
| `WP-IC11` | Add coordinator message vocabulary, conflict-matrix artifact and peer/orchestrator protocol | specialists + xtmux docs/tests | IC04 | coordinator-protocol | high |
|
|
3062
|
+
| `WP-IC12` | Add deterministic/live coordinator evals, Pi/Claude/direct comparison and no-leak tests | specialists + core + xtmux | IC02–IC11,E03 | coordinator-eval | high |
|
|
3063
|
+
| `WP-IC13` | Integrate coordinator with resolved chain shape/composition APIs and Beads-native readiness/gate evidence rather than manual DAG reconstruction | specialists | C02,C03,C04,IC04 | coordinator-chain-api | critical |
|
|
3064
|
+
|
|
3065
|
+
`WP-IC05` owns semantic prompt composition. Core and xtmux must not implement their own Bead traversal or mandatory-rule resolver.
|
|
3066
|
+
|
|
3067
|
+
`WP-IC07` and `WP-IC08` should be separate from `WP-IC09`: readiness/delivery are reusable xtmux/Core capabilities; direct tracked assignment is one consumer.
|
|
3068
|
+
|
|
3069
|
+
## 15.11 Policy-hook packages
|
|
3070
|
+
|
|
3071
|
+
These packages implement the pre-monorepo, pre-coordinator xtmux-derived architecture decision. They are independently valuable and must not be nested under `WP-IC*`.
|
|
3072
|
+
|
|
3073
|
+
| WP | Deliverable | Repo | Dependencies | Scrutiny |
|
|
3074
|
+
|---|---|---|---|---|
|
|
3075
|
+
| `WP-H01` | Shared pure Bead/chain policy kernel for readiness, SCRUTINY and template recommendation | specialists | G02 | high |
|
|
3076
|
+
| `WP-H02` | Claude cheap prefilter plus Pre/PostToolUse adapters | specialists/core distribution | H01 | high |
|
|
3077
|
+
| `WP-H03` | Pi in-process adapter using the same kernel | specialists/core Pi extensions | H01 | high |
|
|
3078
|
+
| `WP-H04` | Durable policy findings, stable fingerprints and reconciliation | specialists | T06,E02,H01 | high |
|
|
3079
|
+
| `WP-H05` | Shadow/warn/enforce parity, latency and false-positive evaluation | specialists | H02–H04,E03 | high |
|
|
3080
|
+
|
|
3081
|
+
## 15.12 Deferred xtrm/Channels packages
|
|
3082
|
+
|
|
3083
|
+
These are linked future epics, not immediate child Beads of the Specialists implementation program unless the operator explicitly starts Track B.
|
|
3084
|
+
|
|
3085
|
+
| WP | Deliverable | Dependency |
|
|
3086
|
+
|---|---|---|
|
|
3087
|
+
| `WP-X01` | Stage 0 detail spec: daemon, state.db, socket, schema registry, registration | architecture gate |
|
|
3088
|
+
| `WP-X02` | Resolve canonical Channels storage/ownership in `packages/channels` | X01 |
|
|
3089
|
+
| `WP-X03` | Channels 0.1 autonomous chain path | X01,X02,critical chain semantics |
|
|
3090
|
+
| `WP-X04` | xtmux notification adapter over canonical message/handoff events | X03 |
|
|
3091
|
+
| `WP-X05` | Channels 0.2 freeform/node activation and collaboration evals | X03 plus runtime tuning |
|
|
3092
|
+
|
|
3093
|
+
## 15.13 Initial parallel wave
|
|
3094
|
+
|
|
3095
|
+
After documentation gate creation, the first multiplexed wave may safely start these lanes:
|
|
3096
|
+
|
|
3097
|
+
```text
|
|
3098
|
+
Lane A: WP-G00 + WP-G01
|
|
3099
|
+
Lane B: WP-G02 architecture/reuse audit
|
|
3100
|
+
Lane C: WP-T01/T02 design and fixtures under one runtime telemetry owner
|
|
3101
|
+
Lane D: WP-T04 activity query fix
|
|
3102
|
+
Lane E: WP-T05 forensic Prometheus wiring
|
|
3103
|
+
Lane F: WP-T06 fingerprint design
|
|
3104
|
+
Lane G: WP-E03 grader interface design
|
|
3105
|
+
Lane H: WP-M01 memory CLI/provenance audit
|
|
3106
|
+
Lane I: Console eval datasource/API design spike, no source writes
|
|
3107
|
+
Lane J: WP-IC01 content/parity audit
|
|
3108
|
+
Lane K: WP-IC07 readiness contract design and fixtures
|
|
3109
|
+
Lane L: WP-H01 policy-kernel design in shadow-only scope
|
|
3110
|
+
```
|
|
3111
|
+
|
|
3112
|
+
Implementation of `WP-E01`, `WP-P02`, `WP-C01–C05`, `WP-IC02`, `WP-IC05` and `WP-H02/H03` begins only after their named audit/interface gates are closed.
|
|
3113
|
+
|
|
3114
|
+
## 15.14 File-ownership and collision policy
|
|
3115
|
+
|
|
3116
|
+
High-conflict surfaces require a single owner per wave:
|
|
3117
|
+
|
|
3118
|
+
| Surface | Exclusive owner rule |
|
|
3119
|
+
|---|---|
|
|
3120
|
+
| `src/specialist/runner.ts` | one prompt/runtime-policy owner |
|
|
3121
|
+
| `src/pi/session.ts` + `src/specialist/timeline-events.ts` + central supervisor callback path | one telemetry-runtime owner per wave |
|
|
3122
|
+
| `src/specialist/observability-sqlite.ts` migrations | one schema/migration owner; other agents consume interfaces |
|
|
3123
|
+
| `src/specialist/prometheus-projection.ts` | one metrics-projection owner |
|
|
3124
|
+
| mandatory-rules index/global sets | one policy integration owner |
|
|
3125
|
+
| base output schema/contract builder | one output-contract owner |
|
|
3126
|
+
| each `config/specialists/<role>.specialist.json` | one role owner after shared contracts freeze |
|
|
3127
|
+
| Console materializer schema | one Console data owner |
|
|
3128
|
+
| `config/skills/using-specialists/**` | one progressive-disclosure skill owner; eval fixtures may be separate |
|
|
3129
|
+
| `config/specialists/chain-coordinator.specialist.json` | one coordinator prompt/template owner |
|
|
3130
|
+
| Core `worktree-session.ts` role-launch path | one launcher integration owner |
|
|
3131
|
+
| xtmux `handoff` / `safe-send-pointer` / readiness state | one xtmux delivery owner |
|
|
3132
|
+
|
|
3133
|
+
Agents may work concurrently on tests, fixtures and role-specific files when their production surface does not overlap. `xtmux worktree-collisions` and repository diff checks must be run before integration.
|
|
3134
|
+
|
|
3135
|
+
## 15.15 Multiplexing operating protocol
|
|
3136
|
+
|
|
3137
|
+
During bridge-era execution:
|
|
3138
|
+
|
|
3139
|
+
1. Every agent receives one root Bead and one worktree/branch.
|
|
3140
|
+
2. Handoffs use Bead notes/results as durable content and xtmux pointer messages as notification.
|
|
3141
|
+
3. `message-send` is for short state or pointer messages; long requirements remain in the Bead or a versioned file.
|
|
3142
|
+
4. Recipients acknowledge messages. Lack of acknowledgement is visible but does not mutate release authority.
|
|
3143
|
+
5. `safe-send-pointer` may steer an idle/needs-input target; it must not inject into a running/thinking target.
|
|
3144
|
+
6. The integration owner monitors `dashboard`, `audit` and `worktree-collisions`.
|
|
3145
|
+
7. Agents do not merge or push outside their contract unless the chain explicitly assigns publication ownership.
|
|
3146
|
+
8. Cross-lane decisions are recorded in the relevant Bead and echoed to dependent lane owners.
|
|
3147
|
+
9. Findings that require new scope become `discovered-from` follow-up Beads rather than silent expansion.
|
|
3148
|
+
10. Final integration runs the mapped regression suites and records evidence before closing the parent work package.
|
|
3149
|
+
|
|
3150
|
+
## 15.16 Recommended agent roster and assignment
|
|
3151
|
+
|
|
3152
|
+
The local orchestrator should instantiate bounded agents by surface, not one agent per arbitrary bullet. A practical initial roster is:
|
|
3153
|
+
|
|
3154
|
+
| Agent identity | Primary responsibility | Assigned WPs | Must not own concurrently |
|
|
3155
|
+
|---|---|---|---|
|
|
3156
|
+
| `docs-reconciler` | Canonical roadmap/meta patches and cross-links | G00,G01,G03 | runtime code |
|
|
3157
|
+
| `architecture-reuse-auditor` | `bd` primitive reuse and bridge-retirement decision | G02 | implementation before decision approval |
|
|
3158
|
+
| `telemetry-runtime-owner` | Pi/timeline/supervisor tool and thinking semantics | T01,T02,T03 | eval DB migrations |
|
|
3159
|
+
| `telemetry-query-owner` | activity/stall query and historical compatibility | T04 | central callback path without coordination |
|
|
3160
|
+
| `metrics-projection-owner` | forensic-to-Prometheus wiring | T05 | Console panel implementation |
|
|
3161
|
+
| `fingerprint-owner` | resolved candidate manifest and startup hashes | T06,P01 | role prompt edits before schema approval |
|
|
3162
|
+
| `telemetry-integrator` | fixtures, compatibility and telemetry suite | T07,S08 | unrelated prompt work |
|
|
3163
|
+
| `eval-schema-owner` | eval migrations, DAO and storage types | E01 | other observability migrations in same wave |
|
|
3164
|
+
| `grader-core-owner` | grader interface and assertion schema | E03 | role-specific judge calibration |
|
|
3165
|
+
| `eval-runtime-owner` | post-run/backfill/eval CLI | E04,E05,E09 | post-chain logic before chain shape exists |
|
|
3166
|
+
| `experiment-owner` | paired trials, statistics and promotion | E07,E08 | candidate prompt implementation |
|
|
3167
|
+
| `chain-architecture-owner` | C01–C05 critical chain spine | C01,C02,C03,C04,C05 | prompt refactors |
|
|
3168
|
+
| `chain-context-owner` | context envelope, handoff and chain eval | C06,C07,C08,C09,C10,E06 | Channels implementation |
|
|
3169
|
+
| `prompt-contract-owner` | output schema and runner policy | P02,P03,P04 | role prompt files in parallel without freeze point |
|
|
3170
|
+
| `advisor-prompt-owner` | researcher/overthinker candidates | P05,S05,S06 | shared runner or global rule index |
|
|
3171
|
+
| `gate-prompt-owner` | seconder/test-engineer candidates | P06,S03,S04 | reviewer/executor files |
|
|
3172
|
+
| `executor-prompt-owner` | executor candidate and suite | P07,S02 | reviewer prompt |
|
|
3173
|
+
| `reviewer-prompt-owner` | reviewer candidate and suite | P08,S01 | executor prompt |
|
|
3174
|
+
| `memory-owner` | memory audit, runtime, rule and eval | M01–M06 | chain messaging protocol |
|
|
3175
|
+
| `console-eval-owner` | materializer/API/eval views | U01–U06 | upstream evaluator semantics |
|
|
3176
|
+
| `interactive-context-auditor` | using-specialists migration map and parity contract | IC01 | implementation before audit decision |
|
|
3177
|
+
| `coordinator-skill-owner` | progressive skill tree and coordinator doctrine | IC02,IC03,IC04,IC11 | role launcher code |
|
|
3178
|
+
| `interactive-renderer-owner` | Specialists pure assignment renderer | IC05,IC06 | Core launch mechanics |
|
|
3179
|
+
| `role-readiness-owner` | Pi/Claude startup readiness contract | IC07 | Specialists prompt composition |
|
|
3180
|
+
| `xtmux-delivery-owner` | existing-file handoff, readiness wait and monitoring | IC08 | Core launcher planning |
|
|
3181
|
+
| `role-launcher-owner` | Core tracked/preheated role integration | IC09,IC10 | xtmux internal storage semantics |
|
|
3182
|
+
| `coordinator-eval-owner` | cross-runtime behavior/no-leak/context tests | IC12 | production implementation under evaluation |
|
|
3183
|
+
| `policy-kernel-owner` | shared policy decisions and durable findings | H01,H04 | harness-specific adapters initially |
|
|
3184
|
+
| `integration-reviewer` | cross-lane review, dependency and acceptance coverage | all integration gates | production implementation in reviewed lane |
|
|
3185
|
+
|
|
3186
|
+
The roster is a role map, not a requirement to run every agent simultaneously. Concurrency should be limited by integration bandwidth and file overlap. A recommended first wave is six to nine active agents, with one integration reviewer and one orchestrator tracking the whole program.
|
|
3187
|
+
|
|
3188
|
+
Assignment rules:
|
|
3189
|
+
|
|
3190
|
+
1. An agent owns one high-conflict production surface at a time.
|
|
3191
|
+
2. Test/fixture agents may support a production owner but cannot independently redefine event or schema semantics.
|
|
3192
|
+
3. Role prompt owners start only after the prompt-contract owner publishes the frozen base output and injection contract.
|
|
3193
|
+
4. The chain-context owner consumes the chain-architecture owner’s published resolved-shape API rather than editing its internals opportunistically.
|
|
3194
|
+
5. The Console owner receives versioned upstream DTO/event contracts; it does not infer them from private database internals.
|
|
3195
|
+
6. The integration reviewer verifies the Bead contract, diff, tests, eval evidence and downstream effects, but does not become the default fixer.
|
|
3196
|
+
|
|
3197
|
+
## 15.17 Recommended integration order
|
|
3198
|
+
|
|
3199
|
+
Merge and promotion order:
|
|
3200
|
+
|
|
3201
|
+
```text
|
|
3202
|
+
documentation/reuse decisions
|
|
3203
|
+
-> telemetry schema and integrity
|
|
3204
|
+
-> eval schema/grader core
|
|
3205
|
+
-> interactive content/parity audit
|
|
3206
|
+
-> readiness and secure delivery primitives
|
|
3207
|
+
-> progressive skill + assignment renderer
|
|
3208
|
+
-> chain state foundations
|
|
3209
|
+
-> output contract and runner policy
|
|
3210
|
+
-> role prompt candidates
|
|
3211
|
+
-> chain context and handoff
|
|
3212
|
+
-> memory treatment
|
|
3213
|
+
-> model/prompt experiments
|
|
3214
|
+
-> Console product surfaces
|
|
3215
|
+
-> enforcement
|
|
3216
|
+
-> Channels replacement of bridge communication
|
|
3217
|
+
```
|
|
3218
|
+
|
|
3219
|
+
A lane may develop ahead on a feature branch, but it may not be promoted across an unmet integration gate.
|
|
3220
|
+
|
|
3221
|
+
# 16. Rollout plan
|
|
3222
|
+
|
|
3223
|
+
## Phase 0 — Documentation reconciliation and immutable baseline
|
|
3224
|
+
|
|
3225
|
+
1. Complete `WP-G00`–`WP-G03`.
|
|
3226
|
+
2. Capture resolved configs for all six roles across representative repos.
|
|
3227
|
+
3. Hash prompt, rules, skills, runner and tool catalog.
|
|
3228
|
+
4. Export first-turn input, active runtime, tools, tokens, verdicts and fix loops.
|
|
3229
|
+
5. Freeze historical reviewer/executor/chain cohorts and document completeness.
|
|
3230
|
+
6. Record current roadmap and code status so later reports do not compare against a moving baseline.
|
|
3231
|
+
7. Capture the installed Beads version/capability matrix and generate a dry-run graph plan for the modernization backlog.
|
|
3232
|
+
8. Approve the designated-migrator and backup/adoption runbook for future Beads upgrades.
|
|
3233
|
+
|
|
3234
|
+
**Exit criteria:** canonical cross-links exist; the new dated reconciliation is approved; every implementation WP is traceable; a repeatable baseline report exists; the Beads graph dry-run succeeds; stable and pinned capabilities are explicitly separated.
|
|
3235
|
+
|
|
3236
|
+
## Phase 1 — Telemetry integrity
|
|
3237
|
+
|
|
3238
|
+
1. Complete `WP-T01`–`WP-T07`.
|
|
3239
|
+
2. Version changed metric semantics rather than silently rewriting historical dashboards.
|
|
3240
|
+
3. Recompute a sample of historical metrics and compare against raw event traces.
|
|
3241
|
+
|
|
3242
|
+
**Exit criteria:** no known counting ambiguity in any metric used for A/B promotion; `telemetry-integrity-v1` passes.
|
|
3243
|
+
|
|
3244
|
+
## Phase 2 — Eval core and historical backfill
|
|
3245
|
+
|
|
3246
|
+
1. Complete `WP-E01`–`WP-E05` and `WP-E09` in observe-only mode.
|
|
3247
|
+
2. Backfill selected per-repo databases without mutating raw events.
|
|
3248
|
+
3. Create data-completeness summaries and initial role cohorts.
|
|
3249
|
+
4. Start `WP-S01`–`WP-S06` fixture creation in parallel.
|
|
3250
|
+
|
|
3251
|
+
**Exit criteria:** historical trials are inspectable; new runs automatically receive deterministic post-run scores; no evaluator can block execution yet.
|
|
3252
|
+
|
|
3253
|
+
|
|
3254
|
+
## Phase 2A — Interactive coordinator bridge foundations
|
|
3255
|
+
|
|
3256
|
+
1. Complete `WP-IC01` and approve the content migration/parity matrix.
|
|
3257
|
+
2. Complete `WP-IC07` and define an observable runtime-ready transition.
|
|
3258
|
+
3. Complete `WP-IC02`–`WP-IC06` without waiting for the full chain spine.
|
|
3259
|
+
4. Complete `WP-IC08`–`WP-IC10` using secure prompt-file pointer delivery.
|
|
3260
|
+
5. Keep `--bead` identity metadata after initial assignment.
|
|
3261
|
+
6. Validate preheated standby and direct tracked launch under both Pi and Claude.
|
|
3262
|
+
7. Run `WP-IC12` baseline comparison before enabling the coordinator for broad autonomous epic work.
|
|
3263
|
+
|
|
3264
|
+
**Exit criteria:** the coordinator receives current Bead context and mandatory rules exactly once as a user assignment; preheated mode sends `role.ready`; direct tracked mode auto-assigns only after runtime readiness; prompt bodies do not leak; both runtimes remain interactive.
|
|
3265
|
+
|
|
3266
|
+
## Phase 3 — Critical chain-first roadmap spine
|
|
3267
|
+
|
|
3268
|
+
1. Close `WP-G02` reuse decisions.
|
|
3269
|
+
2. Implement `WP-C01`–`WP-C05`.
|
|
3270
|
+
3. Persist chain shapes and expose composition/read surfaces.
|
|
3271
|
+
4. Use Beads-native molecule readiness, gated-resume and atomic claim semantics; Specialists adds job/evidence/workspace state rather than reimplementing dependency readiness.
|
|
3272
|
+
5. Integrate `WP-IC13` so the interactive coordinator consumes resolved shape and composition APIs instead of reconstructing them manually.
|
|
3273
|
+
6. Keep any `step_completed` behavior visibility-only.
|
|
3274
|
+
|
|
3275
|
+
**Exit criteria:** a chain has durable identity independent of a live executor; ordered step/gate shape and workspace resolution are queryable and tested; native Beads readiness agrees with `sp chain` projection on fixture graphs.
|
|
3276
|
+
|
|
3277
|
+
## Phase 4 — Output contract and runner policy consolidation
|
|
3278
|
+
|
|
3279
|
+
1. Complete `WP-P01`–`WP-P04`.
|
|
3280
|
+
2. Normalize status/verdict schemas.
|
|
3281
|
+
3. Remove duplicated output, GitNexus, style and Beads workflow instructions.
|
|
3282
|
+
4. Introduce compatibility shims and output-validation events.
|
|
3283
|
+
|
|
3284
|
+
**Exit criteria:** the generated output contract is the single machine-readable SSOT; role-aware injections pass regression fixtures.
|
|
3285
|
+
|
|
3286
|
+
## Phase 5 — Role prompt experiments
|
|
3287
|
+
|
|
3288
|
+
1. Run `WP-P05` first for overthinker/researcher.
|
|
3289
|
+
2. Run `WP-P06` for seconder/test-engineer.
|
|
3290
|
+
3. Run `WP-P07` and `WP-P08` last because executor/reviewer have the highest operational blast radius.
|
|
3291
|
+
4. Use paired or factorial experiments, never unpaired historical PASS rates as the promotion criterion.
|
|
3292
|
+
|
|
3293
|
+
**Exit criteria:** prompt reductions meet role-specific thresholds without deterioration of hard gates or confidence-interval promotion rules.
|
|
3294
|
+
|
|
3295
|
+
## Phase 6 — Complete chain context and handoff
|
|
3296
|
+
|
|
3297
|
+
1. Complete `WP-C06`–`WP-C10` and `WP-E06`.
|
|
3298
|
+
2. Inject deterministic chain identity and pointers.
|
|
3299
|
+
3. Add selective retrieval, handoff decision records and chain participation graders.
|
|
3300
|
+
|
|
3301
|
+
**Exit criteria:** reviewers and downstream gates can identify all required upstream inputs; context completeness is explicit; chain-participation suite meets threshold.
|
|
3302
|
+
|
|
3303
|
+
## Phase 7 — Pull-based memory
|
|
3304
|
+
|
|
3305
|
+
1. Complete `WP-M01`–`WP-M05`, beginning with proof of which requirements native Beads memory already satisfies.
|
|
3306
|
+
2. Run treatment/control with eager injection still available as rollback.
|
|
3307
|
+
3. Complete `WP-M06` only after necessary-memory and unnecessary-query graders are calibrated.
|
|
3308
|
+
|
|
3309
|
+
**Exit criteria:** first-turn memory payload materially decreases with no statistically material increase in historical-context misses.
|
|
3310
|
+
|
|
3311
|
+
## Phase 8 — Model/prompt benchmark matrix
|
|
3312
|
+
|
|
3313
|
+
1. Complete `WP-E07`, `WP-E08` and remaining role suites.
|
|
3314
|
+
2. Run reviewer prompt×model factorial experiments first.
|
|
3315
|
+
3. Expand to executor, test-engineer and researcher.
|
|
3316
|
+
4. Add internal SWE-bench-style and selected external public benchmarks.
|
|
3317
|
+
5. Establish scheduled regression and capability runs.
|
|
3318
|
+
|
|
3319
|
+
## Phase 9 — Console Eval Lab
|
|
3320
|
+
|
|
3321
|
+
1. Complete `WP-U01`–`WP-U06`.
|
|
3322
|
+
2. Preserve aggregate-to-forensic drilldown and freshness/completeness metadata.
|
|
3323
|
+
3. Support live experiment visibility without moving evaluator ownership into Console.
|
|
3324
|
+
|
|
3325
|
+
## Phase 10 — Enforcement
|
|
3326
|
+
|
|
3327
|
+
1. Promote high-precision deterministic graders from observe to warn.
|
|
3328
|
+
2. Measure false positives and operator overrides.
|
|
3329
|
+
3. Promote only approved rules to enforce.
|
|
3330
|
+
4. Keep LLM judges advisory unless separately authorized.
|
|
3331
|
+
|
|
3332
|
+
## Phase 11 — Channels integration
|
|
3333
|
+
|
|
3334
|
+
After xtrm Stage 0 and Channels 0.1 acceptance:
|
|
3335
|
+
|
|
3336
|
+
1. replace manual remediation dispatch with typed scheduler intents;
|
|
3337
|
+
2. dual-write semantic messages to forensic/evidence;
|
|
3338
|
+
3. add message delivery, acknowledgement, authority and convergence evals;
|
|
3339
|
+
4. retire bridge scheduling behavior from `step_completed`;
|
|
3340
|
+
5. add optional xtmux notification adapter;
|
|
3341
|
+
6. defer freeform/node collaboration enforcement until Channels 0.2 is measured.
|
|
3342
|
+
|
|
3343
|
+
# 17. Detailed implementation backlog
|
|
3344
|
+
|
|
3345
|
+
The Epics below are capability groupings. For actual Beads decomposition and parallel assignment, Section 15 work packages are normative. A planner may group multiple WPs under one Epic, but must not merge their contracts or erase their dependencies.
|
|
3346
|
+
|
|
3347
|
+
|
|
3348
|
+
## Epic A — Prompt composition inventory
|
|
3349
|
+
|
|
3350
|
+
- Add a resolved prompt manifest command.
|
|
3351
|
+
- Report component bytes/tokens and hashes.
|
|
3352
|
+
- Capture eager skill body costs.
|
|
3353
|
+
- Compare API first-turn truth with component estimate.
|
|
3354
|
+
- Detect duplicate rule IDs and repeated normalized text.
|
|
3355
|
+
- Detect conflicting enums/schema fields.
|
|
3356
|
+
|
|
3357
|
+
## Epic B — Output contract unification
|
|
3358
|
+
|
|
3359
|
+
- Select canonical status/verdict vocabulary.
|
|
3360
|
+
- Convert all role output schemas to formal JSON Schema.
|
|
3361
|
+
- Version merged output schema.
|
|
3362
|
+
- Remove handoff-rule duplication.
|
|
3363
|
+
- Emit output validation forensic events.
|
|
3364
|
+
- Add regression fixtures for markdown and JSON modes.
|
|
3365
|
+
|
|
3366
|
+
## Epic C — Six prompt refactors
|
|
3367
|
+
|
|
3368
|
+
- Reviewer core prompt and policy extraction.
|
|
3369
|
+
- Executor core prompt and language-skill separation.
|
|
3370
|
+
- Overthinker adaptive-depth prompt.
|
|
3371
|
+
- Seconder bounded gate prompt.
|
|
3372
|
+
- Test-engineer explicit mode and source boundary.
|
|
3373
|
+
- Researcher epistemic core and routing extraction.
|
|
3374
|
+
|
|
3375
|
+
## Epic D — Runner injection cleanup
|
|
3376
|
+
|
|
3377
|
+
- Role-aware style directive.
|
|
3378
|
+
- Single GitNexus policy source.
|
|
3379
|
+
- Writer-only commit/close instructions.
|
|
3380
|
+
- Split worktree boundary from evidence retrieval.
|
|
3381
|
+
- Remove global contradictions for researcher/read-only roles.
|
|
3382
|
+
|
|
3383
|
+
## Epic E — Chain context
|
|
3384
|
+
|
|
3385
|
+
- Persist/resolve chain template and ordered steps.
|
|
3386
|
+
- Chain context envelope builder.
|
|
3387
|
+
- Beads semantic edge query helpers.
|
|
3388
|
+
- Context completeness flags.
|
|
3389
|
+
- Shared chain participant rule.
|
|
3390
|
+
- Handoff extension.
|
|
3391
|
+
|
|
3392
|
+
## Epic F — Memory capability
|
|
3393
|
+
|
|
3394
|
+
- Search/relevant/recall command design.
|
|
3395
|
+
- Provenance and supersession fields.
|
|
3396
|
+
- Query telemetry.
|
|
3397
|
+
- Pull policy rule.
|
|
3398
|
+
- Eager injection experiment flag.
|
|
3399
|
+
|
|
3400
|
+
## Epic G — Telemetry integrity
|
|
3401
|
+
|
|
3402
|
+
- Thinking span model.
|
|
3403
|
+
- Tool-call deduplication and parallel duration.
|
|
3404
|
+
- Turn/message timing.
|
|
3405
|
+
- Activity query fixes.
|
|
3406
|
+
- Forensic Prometheus collector wiring.
|
|
3407
|
+
- Candidate fingerprint fields.
|
|
3408
|
+
- Telemetry integrity suite.
|
|
3409
|
+
|
|
3410
|
+
## Epic H — Eval storage and runtime
|
|
3411
|
+
|
|
3412
|
+
- Eval tables and migrations.
|
|
3413
|
+
- Eval forensic events.
|
|
3414
|
+
- Grader interface and registry.
|
|
3415
|
+
- Post-run evaluator.
|
|
3416
|
+
- Post-chain evaluator.
|
|
3417
|
+
- Historical backfill.
|
|
3418
|
+
- Trial artifact bundle.
|
|
3419
|
+
|
|
3420
|
+
## Epic I — Role suites
|
|
3421
|
+
|
|
3422
|
+
- Reviewer seeded-patch suite.
|
|
3423
|
+
- Executor hidden-test suite.
|
|
3424
|
+
- Seconder smell/scope suite.
|
|
3425
|
+
- Test-engineer fail-to-pass suite.
|
|
3426
|
+
- Research claim/citation suite.
|
|
3427
|
+
- Overthinker decision-risk suite.
|
|
3428
|
+
- Chain participation suite.
|
|
3429
|
+
|
|
3430
|
+
## Epic J — Experimentation and statistics
|
|
3431
|
+
|
|
3432
|
+
- Candidate configuration format.
|
|
3433
|
+
- Paired runner.
|
|
3434
|
+
- Trial randomization.
|
|
3435
|
+
- Pairwise judge and order swap.
|
|
3436
|
+
- Bootstrap and binary paired analysis.
|
|
3437
|
+
- Promotion policy engine.
|
|
3438
|
+
|
|
3439
|
+
## Epic K — Console integration
|
|
3440
|
+
|
|
3441
|
+
- Eval datasource/materializer rows.
|
|
3442
|
+
- Eval APIs.
|
|
3443
|
+
- Overview and experiment pages.
|
|
3444
|
+
- Trial timeline and evidence drawer.
|
|
3445
|
+
- Historical mining and case promotion.
|
|
3446
|
+
- Realtime updates.
|
|
3447
|
+
- Human adjudication and promotion UX.
|
|
3448
|
+
|
|
3449
|
+
# 18. Acceptance criteria
|
|
3450
|
+
|
|
3451
|
+
|
|
3452
|
+
## Beads capability reuse
|
|
3453
|
+
|
|
3454
|
+
- The decomposition graph is materialized from a reviewed `bead-graph-plan.json` after a successful dry-run.
|
|
3455
|
+
- Every Beads-dependent runtime feature declares a minimum released version or pinned commit and a fallback path.
|
|
3456
|
+
- `sp chain` readiness agrees with `bd ready --mol/--gated` on the canonical fixture set.
|
|
3457
|
+
- Atomic claims use native `bd ready --claim`; Specialists does not implement a competing issue-claim protocol.
|
|
3458
|
+
- Structural fan-in uses native Beads waits/gates where sufficient; semantic review remains in Specialists.
|
|
3459
|
+
- Bead claim lease and worktree/merge-slot lease are modeled as separate ownership dimensions.
|
|
3460
|
+
- No unreleased Beads capability becomes mandatory without compatibility tests and operator approval.
|
|
3461
|
+
- Dolt remains the selected backend during the bridge runway.
|
|
3462
|
+
- Remote-backed upgrades follow a backup, designated-migrator and adoption runbook.
|
|
3463
|
+
|
|
3464
|
+
## Prompt modernization
|
|
3465
|
+
|
|
3466
|
+
- First-turn input tokens decrease materially for each refactored role.
|
|
3467
|
+
- No role exceeds baseline critical failure rate.
|
|
3468
|
+
- Output schema validity is at least 99.5%.
|
|
3469
|
+
- Reviewer false-PASS rate does not increase.
|
|
3470
|
+
- Executor scope violations do not increase.
|
|
3471
|
+
- Seconder remains within configured time/tool budget.
|
|
3472
|
+
- Test-engineer source-boundary violations remain zero.
|
|
3473
|
+
- Researcher citation correctness does not regress.
|
|
3474
|
+
|
|
3475
|
+
## Chain context
|
|
3476
|
+
|
|
3477
|
+
- Every chain step receives valid chain identity and root/step pointers.
|
|
3478
|
+
- Context completeness is explicitly marked.
|
|
3479
|
+
- Reviewer can identify all required upstream gates without filesystem hunting.
|
|
3480
|
+
- Upstream-result retrieval is selective and evidence-linked.
|
|
3481
|
+
- Handoff decision records are parseable.
|
|
3482
|
+
|
|
3483
|
+
## Memory
|
|
3484
|
+
|
|
3485
|
+
- Eager memory tokens are eliminated or substantially reduced for treatment candidates.
|
|
3486
|
+
- Necessary-memory recall remains above an approved threshold.
|
|
3487
|
+
- Unnecessary memory-query rate is measured and bounded.
|
|
3488
|
+
- Memory results include provenance.
|
|
3489
|
+
|
|
3490
|
+
## Telemetry
|
|
3491
|
+
|
|
3492
|
+
- Unique tool-call count is correct in fixture traces.
|
|
3493
|
+
- Thinking spans are paired and durations non-negative.
|
|
3494
|
+
- Active plus waiting time reconciles with elapsed time within tolerance.
|
|
3495
|
+
- Forensic-derived metric families appear in the live projection.
|
|
3496
|
+
- No forbidden labels are emitted.
|
|
3497
|
+
|
|
3498
|
+
## Eval platform
|
|
3499
|
+
|
|
3500
|
+
- Existing runs can be backfilled without modifying raw events.
|
|
3501
|
+
- New runs receive deterministic post-run scores.
|
|
3502
|
+
- Chains receive post-chain scores.
|
|
3503
|
+
- Candidate fingerprints allow exact comparison.
|
|
3504
|
+
- Paired experiments produce confidence intervals and win/tie/loss.
|
|
3505
|
+
- Promotion policy can block regressions and recommend qualifying candidates.
|
|
3506
|
+
|
|
3507
|
+
## Console
|
|
3508
|
+
|
|
3509
|
+
- Eval overview and trial detail are accessible from existing Console navigation.
|
|
3510
|
+
- Aggregate views drill down to forensic evidence.
|
|
3511
|
+
- Live and historical data display freshness/completeness.
|
|
3512
|
+
- No direct frontend access to per-repo SQLite is introduced.
|
|
3513
|
+
|
|
3514
|
+
|
|
3515
|
+
## 18.7 Interactive coordinator and role assignment
|
|
3516
|
+
|
|
3517
|
+
- `using-specialists/SKILL.md` is within the line budget and all displaced content has a tested destination or explicit removal rationale.
|
|
3518
|
+
- Source and global installed skill trees are recursively complete and link-valid.
|
|
3519
|
+
- The coordinator stable system prompt contains no tracked Bead payload or flattened mandatory-rule block.
|
|
3520
|
+
- A tracked interactive assignment contains task-template/Bead/dependency context followed by current mandatory rules, each exactly once.
|
|
3521
|
+
- `sp role compose` is read-only and creates no job, worktree, session or Bead mutation.
|
|
3522
|
+
- Direct `--bead` launch fails before provisioning on renderer failure.
|
|
3523
|
+
- Runtime readiness is explicit; a newly spawned but unready pane cannot receive the assignment.
|
|
3524
|
+
- Preheated launch produces `role.ready` and waits for exactly one epic assignment.
|
|
3525
|
+
- Core/xtmux transport exposes no full prompt body through argv, pane metadata, logs or telemetry.
|
|
3526
|
+
- Pi and Claude remain interactive after the first assigned turn.
|
|
3527
|
+
- The coordinator selects/validates chains through the canonical formula catalog and resolved shape, not a hardcoded role sequence.
|
|
3528
|
+
- Conflict matrix, peer hygiene messaging, close-readiness evidence and escalation behavior pass `interactive-coordinator-v1`.
|
|
3529
|
+
|
|
3530
|
+
## 18.8 Existing Beads reconciliation
|
|
3531
|
+
|
|
3532
|
+
- Every filed Bead in the 13 July integration packet is mapped to KEEP, REFINE, MERGE, RESEQUENCE, REPLACE or ADD.
|
|
3533
|
+
- Existing Bead descriptions are updated before dispatch when this PRD changes their transport, scope, dependency or validation contract.
|
|
3534
|
+
- New xtmux-owned work is not hidden inside a Core Bead.
|
|
3535
|
+
- Superseded constraints remain visible in Bead notes with a pointer to this PRD; they are not silently deleted.
|
|
3536
|
+
|
|
3537
|
+
# 19. Risks and mitigations
|
|
3538
|
+
|
|
3539
|
+
| Risk | Consequence | Mitigation |
|
|
3540
|
+
|---|---|---|
|
|
3541
|
+
| Prompt slimming removes useful tacit behavior | Quality regression | Paired regression suite, staged roles, rollback by version |
|
|
3542
|
+
| Runtime and prompt change simultaneously | Attribution failure | Factorial design and candidate fingerprints |
|
|
3543
|
+
| Historical data is incomplete | Misleading retrospective scores | Data completeness flags; separate estimated from complete |
|
|
3544
|
+
| LLM judge bias | Incorrect promotion | Human calibration, order swap, multiple judges, deterministic hard gates |
|
|
3545
|
+
| Continuous eval adds latency | Slower chains | Asynchronous post-run evaluation; only hard deterministic gates on critical path |
|
|
3546
|
+
| Eval recursion | Evaluator jobs grade themselves | Explicit evaluator participant kind and exclusion policy |
|
|
3547
|
+
| High-cardinality metric explosion | Prometheus instability | Existing allowlist/forbidden labels; forensic drilldown |
|
|
3548
|
+
| Memory retrieval becomes another ritual | Token/tool waste | Trigger-based rule and unnecessary-query grader |
|
|
3549
|
+
| Chain context becomes a full history dump | Context bloat | Pointer envelope plus on-demand retrieval |
|
|
3550
|
+
| Tool count changes break historical dashboards | Apparent discontinuity | Version metric semantics and retain legacy field during transition |
|
|
3551
|
+
| External benchmark overfits coding only | Misleading role comparison | Internal role and chain suites remain primary |
|
|
3552
|
+
| Console materializer duplicates source truth | Architectural drift | Summary/evidence projection only; no evaluator ownership in UI |
|
|
3553
|
+
| Local rule/skill overrides escape experiments | Non-reproducible trial | Resolve and hash all effective components at launch |
|
|
3554
|
+
| Specialists duplicates Beads readiness/claim logic | Divergent scheduler state and maintenance burden | Make Beads capability reuse a hard architecture gate and differential-test projections |
|
|
3555
|
+
| Unreleased Beads feature assumed available | Runtime failure across repos | Minimum-version/pin checks, fallbacks and compatibility fixtures |
|
|
3556
|
+
| Independent Beads migrations across clones | Unmergeable schema forks or downtime | Backup, one designated migrator, push/adopt runbook |
|
|
3557
|
+
|
|
3558
|
+
|
|
3559
|
+
## 19.1 Startup readiness false-positive
|
|
3560
|
+
|
|
3561
|
+
**Risk:** the launcher marks a pane idle before the runtime can accept prompt input, causing lost or shell-interpreted assignments.
|
|
3562
|
+
|
|
3563
|
+
**Mitigation:** explicit `starting` state, runtime startup event, readiness-gated handoff, bounded timeout and deterministic cleanup.
|
|
3564
|
+
|
|
3565
|
+
## 19.2 Prompt transport leakage
|
|
3566
|
+
|
|
3567
|
+
**Risk:** passing the rendered assignment as positional argv exposes Bead/rule content in process listings or logs.
|
|
3568
|
+
|
|
3569
|
+
**Mitigation:** mode-restricted prompt file, one-line pointer delivery, hash/size-only telemetry and no body in pane options/messages.
|
|
3570
|
+
|
|
3571
|
+
## 19.3 Coordinator authority creep
|
|
3572
|
+
|
|
3573
|
+
**Risk:** the bridge coordinator starts making vision, merge or architectural decisions intended for the parent orchestrator.
|
|
3574
|
+
|
|
3575
|
+
**Mitigation:** explicit mechanical-vs-vision boundary, bounded message vocabulary, escalation evals and `READY_FOR_PARENT_DECISION` rather than autonomous merge.
|
|
3576
|
+
|
|
3577
|
+
## 19.4 xtmux becomes accidental Channels
|
|
3578
|
+
|
|
3579
|
+
**Risk:** richer peer messaging and receipts evolve into a second semantic scheduler before Substrate.
|
|
3580
|
+
|
|
3581
|
+
**Mitigation:** xtmux remains attention/delivery only; authoritative state and gate outcomes stay in Specialists/Beads; bridge retirement triggers are mandatory.
|
|
3582
|
+
|
|
3583
|
+
# 20. Decisions and remaining implementation choices
|
|
3584
|
+
|
|
3585
|
+
## 20.1 Decisions closed in this revision
|
|
3586
|
+
|
|
3587
|
+
- Keep the current `chain-coordinator` name during the bridge runway; document lifecycle/scope distinction from future Substrate semantics.
|
|
3588
|
+
- Do not add interactive permission enforcement now.
|
|
3589
|
+
- Mandatory rules and tracked Bead context are user-assignment content, not system-prompt content.
|
|
3590
|
+
- Specialists owns composition; Core owns launch; xtmux owns readiness-gated delivery and attention.
|
|
3591
|
+
- Use secure prompt-file pointer delivery as the default bridge transport, not full positional argv.
|
|
3592
|
+
- Support both preheated standby and direct tracked `--bead` launch.
|
|
3593
|
+
- Keep Specialists and xtmux observability databases separate until Substrate migration.
|
|
3594
|
+
- Keep Console's current materializer architecture and expand it after upstream contracts stabilize.
|
|
3595
|
+
- Keep monorepo migration, Stage 0, Substrate and Channels outside the critical path.
|
|
3596
|
+
- Keep xtmux as a bridge attention/delivery runtime, not a semantic scheduler or authority layer.
|
|
3597
|
+
- Use Beads-native molecule readiness, atomic graph creation, fan-in gates and claims before adding Specialists equivalents.
|
|
3598
|
+
- Keep Dolt as the Beads backend during the bridge runway; alternative backends remain a later ADR/watch item.
|
|
3599
|
+
- Treat post-v1.1 claim leases, formula schema and history-event features as release/pin-gated capabilities, not assumptions.
|
|
3600
|
+
|
|
3601
|
+
## 20.2 Decisions required before the affected package is dispatched
|
|
3602
|
+
|
|
3603
|
+
1. Canonical base status and reviewer verdict casing.
|
|
3604
|
+
2. Exact CLI naming and versioned JSON schema for `sp role compose` and `xt role assign`.
|
|
3605
|
+
3. Transport-file location, permissions, expiry and deletion policy.
|
|
3606
|
+
4. Whether `assignment.accepted` is mandatory or advisory after direct tracked launch.
|
|
3607
|
+
5. Exact coordinator conflict-matrix serialization and durable location before resolved chain shape lands.
|
|
3608
|
+
6. Which coordinator-general rules belong in standby bootstrap versus tracked mandatory-rule resolution.
|
|
3609
|
+
7. How Claude emits the same `agent.ready` fact as Pi without relying on UI scraping.
|
|
3610
|
+
8. Whether xtmux extends `handoff` compatibly or introduces a narrowly named assignment-delivery subcommand.
|
|
3611
|
+
9. Exact mapping of roadmap Opportunity 8 events into coordinator notification before Channels.
|
|
3612
|
+
10. Which coordinator model defaults are selected per formula during the bridge period.
|
|
3613
|
+
11. Exact memory retrieval CLI surface and provenance availability in Beads.
|
|
3614
|
+
12. Which deterministic graders may block in the first enforcement wave.
|
|
3615
|
+
|
|
3616
|
+
## 20.3 Decisions that do not block the current bridge program
|
|
3617
|
+
|
|
3618
|
+
- Global registry location for cross-repository experiment definitions before `state.db`; per-repository eval state remains sufficient initially.
|
|
3619
|
+
- Authoritative provider cost source; token use remains primary until cost provenance is versioned.
|
|
3620
|
+
- Minimum trial counts and confidence thresholds by role.
|
|
3621
|
+
- Retention policy for long-lived eval artifacts and hidden benchmark environments.
|
|
3622
|
+
- Whether the global style directive is removed entirely or replaced by role-specific concise-output settings.
|
|
3623
|
+
- Final Channels storage ownership and Stage 0 implementation details.
|
|
3624
|
+
- Whether dated `_meta` reconciliations remain one-time snapshots or become an append-only series.
|
|
3625
|
+
- Final retirement boundary between `step_completed`, xtmux bridge notifications and Channel scheduler intent.
|
|
3626
|
+
|
|
3627
|
+
# Appendix A — Proposed core system prompts
|
|
3628
|
+
|
|
3629
|
+
## A.1 Reviewer
|
|
3630
|
+
|
|
3631
|
+
```text
|
|
3632
|
+
You are the final evidence-based release reviewer for a completed specialist chain.
|
|
3633
|
+
|
|
3634
|
+
The seconder owns the initial contract and implementation-sanity gate. Consume
|
|
3635
|
+
upstream QA, security and obligations evidence before reviewing. Do not repeat an
|
|
3636
|
+
upstream gate unless its result is missing, contradicted by stronger evidence or
|
|
3637
|
+
invalidated by a later diff.
|
|
3638
|
+
|
|
3639
|
+
Review unresolved findings, critical requirement coverage, changed high-risk
|
|
3640
|
+
surfaces and required release evidence. Use the root contract and current artifacts
|
|
3641
|
+
as authority. Treat upstream outputs as evidence-backed claims, not as authority over
|
|
3642
|
+
observed code or tests.
|
|
3643
|
+
|
|
3644
|
+
Never edit files. Do not emit preference-only findings. Every blocking finding must
|
|
3645
|
+
identify the unmet requirement or gate and cite concrete evidence.
|
|
3646
|
+
|
|
3647
|
+
PASS requires all critical requirements and required gates to be evidenced with no
|
|
3648
|
+
blocking finding. PARTIAL means a fixable gap or insufficient evidence remains. FAIL
|
|
3649
|
+
means a core requirement is unmet, authoritative evidence is contradictory, or
|
|
3650
|
+
required review inputs are unusable.
|
|
3651
|
+
|
|
3652
|
+
On re-review after PARTIAL, inspect the delta and prior findings. Carry forward
|
|
3653
|
+
approvals for unchanged areas. Return only the configured output contract.
|
|
3654
|
+
```
|
|
3655
|
+
|
|
3656
|
+
## A.2 Executor
|
|
3657
|
+
|
|
3658
|
+
```text
|
|
3659
|
+
You are a scoped implementation specialist. Implement the supplied Bead contract in
|
|
3660
|
+
the assigned worktree.
|
|
3661
|
+
|
|
3662
|
+
Resolve scope, inspect the relevant implementation and dependency impact, and make
|
|
3663
|
+
the smallest correct change. Follow repository, language and framework conventions.
|
|
3664
|
+
Prefer existing code and dependencies over new abstractions or packages.
|
|
3665
|
+
|
|
3666
|
+
Do not add adjacent cleanup, speculative flexibility, broad refactors, tests,
|
|
3667
|
+
documentation or features unless they are in scope. If a necessary change lies
|
|
3668
|
+
outside authorized scope, return a partial handoff describing the required expansion.
|
|
3669
|
+
|
|
3670
|
+
Preserve security, data integrity, compatibility and external-boundary validation.
|
|
3671
|
+
Run the validation declared by the task plus focused checks needed to establish
|
|
3672
|
+
implementation correctness. Broad authoritative test execution belongs to the
|
|
3673
|
+
configured QA gate unless the task explicitly requires it.
|
|
3674
|
+
|
|
3675
|
+
Before completion, verify changed paths against scope. Return only the configured
|
|
3676
|
+
output contract with changes, verification, assumptions, risks and follow-ups.
|
|
3677
|
+
```
|
|
3678
|
+
|
|
3679
|
+
## A.3 Overthinker
|
|
3680
|
+
|
|
3681
|
+
```text
|
|
3682
|
+
You are a read-only decision-review specialist for uncertain or high-impact choices.
|
|
3683
|
+
|
|
3684
|
+
Analyze only as deeply as the decision warrants. Report conclusions, evidence,
|
|
3685
|
+
assumptions, alternatives and risks; do not expose private chain-of-thought.
|
|
3686
|
+
|
|
3687
|
+
For costly, risky or difficult-to-reverse decisions, stress-test the leading option
|
|
3688
|
+
with a counterargument or premortem and identify evidence that would change the
|
|
3689
|
+
recommendation. For simple reversible decisions, answer directly.
|
|
3690
|
+
|
|
3691
|
+
State confidence, material uncertainty and the next validation step. Never edit
|
|
3692
|
+
files. Return only the configured output contract.
|
|
3693
|
+
```
|
|
3694
|
+
|
|
3695
|
+
## A.4 Seconder
|
|
3696
|
+
|
|
3697
|
+
```text
|
|
3698
|
+
You are the bounded read-only pre-QA gate for a completed writer diff.
|
|
3699
|
+
|
|
3700
|
+
Determine whether the output satisfies the root contract and appears implementation-
|
|
3701
|
+
sane enough to justify expensive QA. Evaluate scope/compliance and implementation
|
|
3702
|
+
quality as separate dimensions. If evidence for one dimension is missing, mark only
|
|
3703
|
+
that dimension UNCLEAR rather than guessing.
|
|
3704
|
+
|
|
3705
|
+
Do not perform release review, security audit, broad architecture critique, test-
|
|
3706
|
+
coverage review or style review. Do not emit preference-only findings.
|
|
3707
|
+
|
|
3708
|
+
Overall PASS requires both dimensions to pass; FAIL applies when either fails;
|
|
3709
|
+
otherwise return PARTIAL. Keep inspection within the runtime budget and return only
|
|
3710
|
+
the configured output contract.
|
|
3711
|
+
```
|
|
3712
|
+
|
|
3713
|
+
## A.5 Test-engineer
|
|
3714
|
+
|
|
3715
|
+
```text
|
|
3716
|
+
You are a test-authoring specialist. The dispatch context explicitly defines
|
|
3717
|
+
`test_only` or `post_implementation` mode.
|
|
3718
|
+
|
|
3719
|
+
Build tests from the root requirements and actual implementation evidence. Follow
|
|
3720
|
+
existing test, fixture, smoke and harness conventions before introducing new patterns.
|
|
3721
|
+
|
|
3722
|
+
Your default edit boundary is tests, fixtures, recorded fixtures, smoke/E2E assets
|
|
3723
|
+
and test-harness support. Production source changes are forbidden unless dispatch
|
|
3724
|
+
explicitly authorizes a named helper or export. When source behavior or
|
|
3725
|
+
instrumentation is wrong, return `source_bug_suspected` with evidence.
|
|
3726
|
+
|
|
3727
|
+
Cover critical behavior rather than optimizing line coverage. Add operational or
|
|
3728
|
+
telemetry assertions when required by the changed surface. Run focused checks needed
|
|
3729
|
+
to validate authored assets; test-runner owns authoritative broad execution.
|
|
3730
|
+
|
|
3731
|
+
Return exact commands, setup/cleanup, coverage mapping, deferred paths and failure
|
|
3732
|
+
ownership through the configured output contract.
|
|
3733
|
+
```
|
|
3734
|
+
|
|
3735
|
+
## A.6 Researcher
|
|
3736
|
+
|
|
3737
|
+
```text
|
|
3738
|
+
You are an external-evidence researcher.
|
|
3739
|
+
|
|
3740
|
+
Use current authoritative sources for unstable claims involving libraries, APIs,
|
|
3741
|
+
versions, releases, ecosystem behavior or external products. Do not perform external
|
|
3742
|
+
research when current repository evidence or stable reasoning is sufficient.
|
|
3743
|
+
|
|
3744
|
+
Prefer primary documentation, standards, research papers and source repositories.
|
|
3745
|
+
Record relevant versions and dates. Distinguish sourced facts from inference and
|
|
3746
|
+
report unresolved gaps. Verify consequential claims across independent sources when
|
|
3747
|
+
practical and stop when the requested success criteria are met.
|
|
3748
|
+
|
|
3749
|
+
Report tool failures instead of inventing evidence. Never expose secrets in queries
|
|
3750
|
+
and never edit project files. Lead with the answer and return findings, citations,
|
|
3751
|
+
confidence and gaps through the configured output contract.
|
|
3752
|
+
```
|
|
3753
|
+
|
|
3754
|
+
# Appendix B — Proposed `chain-participant-context` rule
|
|
3755
|
+
|
|
3756
|
+
```text
|
|
3757
|
+
You are a participant in a multi-step specialist chain, not an isolated agent.
|
|
3758
|
+
|
|
3759
|
+
The chain root change-contract defines the overall objective and acceptance criteria.
|
|
3760
|
+
Your current step-contract defines your mandate and edit boundary. Upstream specialist
|
|
3761
|
+
outputs are evidence and handoffs; they do not override the root contract, current
|
|
3762
|
+
repository state or directly observed artifacts.
|
|
3763
|
+
|
|
3764
|
+
At startup, read the injected chain context and identify the chain, root Bead, current
|
|
3765
|
+
step, completed upstream steps and required downstream handoff. Consume preloaded
|
|
3766
|
+
dependency context before additional commands.
|
|
3767
|
+
|
|
3768
|
+
When required inputs are absent or ambiguous, reconstruct only the necessary context
|
|
3769
|
+
using the current step Bead, Beads dependency/relationship commands, `sp result` for
|
|
3770
|
+
upstream final handoffs and `sp feed --json` only when a claimed action or chronology
|
|
3771
|
+
must be verified.
|
|
3772
|
+
|
|
3773
|
+
Do not repeat work already completed upstream unless your role explicitly validates
|
|
3774
|
+
it, evidence is missing or current artifacts contradict it. Record contradictions.
|
|
3775
|
+
Do not request private chain-of-thought; consume decisions, rationale, assumptions,
|
|
3776
|
+
evidence, verification results and unresolved questions.
|
|
3777
|
+
|
|
3778
|
+
Your final handoff must allow the next chain member to continue without reconstructing
|
|
3779
|
+
your entire session.
|
|
3780
|
+
```
|
|
3781
|
+
|
|
3782
|
+
# Appendix C — Proposed memory recall rule
|
|
3783
|
+
|
|
3784
|
+
```text
|
|
3785
|
+
Historical project memories are available through targeted Beads memory queries.
|
|
3786
|
+
|
|
3787
|
+
Query memory only when the task depends on prior decisions, conventions, known
|
|
3788
|
+
incidents, accepted exceptions or unresolved ambiguity. Use specific subsystem,
|
|
3789
|
+
operation or file keywords. Refine broad result sets instead of scrolling an entire
|
|
3790
|
+
memory corpus. Fetch a full memory only after a relevant summary is identified.
|
|
3791
|
+
|
|
3792
|
+
Treat memories as contextual evidence, not current authority. Verify them against the
|
|
3793
|
+
root contract, current code, documentation and external sources where applicable.
|
|
3794
|
+
Report stale or contradictory memories explicitly.
|
|
3795
|
+
|
|
3796
|
+
Do not query memory for fully specified local work or as a ritual. Do not use internal
|
|
3797
|
+
memory as a source for current external API, release or standards claims.
|
|
3798
|
+
```
|
|
3799
|
+
|
|
3800
|
+
# Appendix D — Initial deterministic grader catalog
|
|
3801
|
+
|
|
3802
|
+
| Grader ID | Role/scope | Evidence | Result |
|
|
3803
|
+
|---|---|---|---|
|
|
3804
|
+
| `output-schema-v1` | all | final output + merged schema | valid/invalid |
|
|
3805
|
+
| `role-write-boundary-v1` | read-only roles | tool events + diff | pass/fail |
|
|
3806
|
+
| `scope-paths-v1` | writers | root/step scope + diff | precision/violations |
|
|
3807
|
+
| `required-tools-v1` | configured roles | trace | pass/fail |
|
|
3808
|
+
| `tool-accounting-v1` | telemetry | tool spans | pass/fail |
|
|
3809
|
+
| `thinking-span-integrity-v1` | telemetry | thinking spans | pass/fail |
|
|
3810
|
+
| `chain-context-present-v1` | chain steps | startup context | pass/fail |
|
|
3811
|
+
| `upstream-result-consumption-v1` | gates/advisors | tools + handoff | score |
|
|
3812
|
+
| `handoff-completeness-v1` | all | final schema | score |
|
|
3813
|
+
| `review-verdict-consistency-v1` | reviewer | gates + verdict | pass/fail |
|
|
3814
|
+
| `test-engineer-source-boundary-v1` | test-engineer | diff | pass/fail |
|
|
3815
|
+
| `research-citation-shape-v1` | researcher | output | pass/fail |
|
|
3816
|
+
| `memory-query-necessity-v1` | selected cases | trace + case label | pass/fail |
|
|
3817
|
+
| `canonical-chain-sequence-v1` | chain | member timeline | pass/fail |
|
|
3818
|
+
| `time-to-pass-v1` | chain | statuses/verdicts | value |
|
|
3819
|
+
|
|
3820
|
+
# Appendix E — Metrics catalog after modernization
|
|
3821
|
+
|
|
3822
|
+
## Job and model
|
|
3823
|
+
|
|
3824
|
+
```text
|
|
3825
|
+
xtrm_jobs_total
|
|
3826
|
+
xtrm_job_duration_seconds
|
|
3827
|
+
xtrm_job_active_runtime_seconds
|
|
3828
|
+
xtrm_job_wait_seconds
|
|
3829
|
+
xtrm_turns_total
|
|
3830
|
+
xtrm_turn_duration_seconds
|
|
3831
|
+
xtrm_message_duration_seconds
|
|
3832
|
+
xtrm_context_usage_ratio
|
|
3833
|
+
xtrm_llm_tokens_total
|
|
3834
|
+
xtrm_thinking_duration_seconds
|
|
3835
|
+
xtrm_thinking_segments_total
|
|
3836
|
+
xtrm_time_to_first_action_seconds
|
|
3837
|
+
```
|
|
3838
|
+
|
|
3839
|
+
## Tools and dependencies
|
|
3840
|
+
|
|
3841
|
+
```text
|
|
3842
|
+
xtrm_tool_calls_total
|
|
3843
|
+
xtrm_tool_errors_total
|
|
3844
|
+
xtrm_tool_call_duration_seconds
|
|
3845
|
+
xtrm_uncorrelated_tool_events_total
|
|
3846
|
+
xtrm_mcp_operations_total
|
|
3847
|
+
```
|
|
3848
|
+
|
|
3849
|
+
## Chain and gates
|
|
3850
|
+
|
|
3851
|
+
```text
|
|
3852
|
+
xtrm_chains_total
|
|
3853
|
+
xtrm_chain_duration_seconds
|
|
3854
|
+
xtrm_gate_verdicts_total
|
|
3855
|
+
xtrm_chain_fix_loops_total
|
|
3856
|
+
xtrm_chain_time_to_pass_seconds
|
|
3857
|
+
xtrm_chain_tokens_to_pass_total
|
|
3858
|
+
xtrm_evidence_refs_total
|
|
3859
|
+
```
|
|
3860
|
+
|
|
3861
|
+
## Memory
|
|
3862
|
+
|
|
3863
|
+
```text
|
|
3864
|
+
xtrm_memory_queries_total
|
|
3865
|
+
xtrm_memory_results_total
|
|
3866
|
+
xtrm_memory_recall_total
|
|
3867
|
+
xtrm_memory_query_duration_seconds
|
|
3868
|
+
```
|
|
3869
|
+
|
|
3870
|
+
## Evaluation
|
|
3871
|
+
|
|
3872
|
+
```text
|
|
3873
|
+
xtrm_eval_runs_total
|
|
3874
|
+
xtrm_eval_score
|
|
3875
|
+
xtrm_eval_assertions_total
|
|
3876
|
+
xtrm_eval_regressions_total
|
|
3877
|
+
xtrm_eval_judge_disagreements_total
|
|
3878
|
+
xtrm_grader_duration_seconds
|
|
3879
|
+
```
|
|
3880
|
+
|
|
3881
|
+
# Appendix F — Source inventory used for this PRD
|
|
3882
|
+
|
|
3883
|
+
## Specialists repository
|
|
3884
|
+
|
|
3885
|
+
The current-state findings in this PRD were verified against the following repository areas during the audit:
|
|
3886
|
+
|
|
3887
|
+
```text
|
|
3888
|
+
config/specialists/*.specialist.json
|
|
3889
|
+
config/mandatory-rules/
|
|
3890
|
+
config/skills/
|
|
3891
|
+
docs/design/roadmap/chain-templates/
|
|
3892
|
+
docs/archive/iron-review-hardening-qa-chain-substrate.md
|
|
3893
|
+
docs/design/roadmap/specialists-roadmap.md
|
|
3894
|
+
config/skills/using-specialists/SKILL.md
|
|
3895
|
+
config/skills/using-kpi/SKILL.md
|
|
3896
|
+
src/specialist/runner.ts
|
|
3897
|
+
src/specialist/supervisor.ts
|
|
3898
|
+
src/specialist/beads.ts
|
|
3899
|
+
src/specialist/timeline-events.ts
|
|
3900
|
+
src/specialist/forensic-events.ts
|
|
3901
|
+
src/specialist/observability-sqlite.ts
|
|
3902
|
+
src/specialist/prometheus-projection.ts
|
|
3903
|
+
src/pi/session.ts
|
|
3904
|
+
src/cli/db.ts
|
|
3905
|
+
src/cli/metrics.ts
|
|
3906
|
+
```
|
|
3907
|
+
|
|
3908
|
+
## xtrm and xtmux repositories
|
|
3909
|
+
|
|
3910
|
+
- `xtrm-dev/xtrm/docs/_meta/2026-06-21-design-reconciliation.md`
|
|
3911
|
+
- `xtrm-dev/xtrm/docs/channels/channels.md`
|
|
3912
|
+
- `xtrm-dev/xtrm/docs/substrate/substrate_design_it.md`
|
|
3913
|
+
- `xtrm-dev/core/docs/xt-pi-role.md` and `cli/src/utils/worktree-session.ts`
|
|
3914
|
+
- `Jaggerxtrm/xtmux/docs/observability-redesign.md`, `bin/tmux-session-picker`, Claude hooks and Pi extensions
|
|
3915
|
+
|
|
3916
|
+
## Console repository
|
|
3917
|
+
|
|
3918
|
+
```text
|
|
3919
|
+
README.md
|
|
3920
|
+
docs/architecture/console-observability-spec.md
|
|
3921
|
+
docs/architecture/console-architecture.md
|
|
3922
|
+
```
|
|
3923
|
+
|
|
3924
|
+
## Beads repository
|
|
3925
|
+
|
|
3926
|
+
The 13 July capability reconciliation inspected `gastownhall/beads` v1.1 and current `main`, including `CHANGELOG.md`, `bd ready`, graph creation, formula primitives/schema, leases/heartbeat/reclaim, history events, memory commands, migration safety, storage backends and differential conformance fixtures. The local planning gate must repeat the capability snapshot against the installed binary because upstream `main` is not a release contract.
|
|
3927
|
+
|
|
3928
|
+
## External methodological references
|
|
3929
|
+
|
|
3930
|
+
The evaluation design is informed by current public guidance and research on agent evaluations, deterministic and model-based graders, trace versus outcome evaluation, pairwise judge bias, and software-engineering agent benchmarks. Representative sources include OpenAI Agent Evals and Graders guidance, Anthropic publications on building effective agents and agent evaluations, UK AISI Inspect AI, Lost-in-the-Middle research, SWE-bench Pro and Multi-SWE-bench.
|
|
3931
|
+
|
|
3932
|
+
These external references inform methodology only. The Specialists runtime, chain, telemetry and Console facts in this document derive from direct repository inspection.
|
|
3933
|
+
|
|
3934
|
+
# Appendix G — Copy-paste contract for the local decomposition agent
|
|
3935
|
+
|
|
3936
|
+
Use the following instruction with a planning-capable local agent. Replace repository paths only when the local checkout differs.
|
|
3937
|
+
|
|
3938
|
+
```text
|
|
3939
|
+
You are the planning and decomposition agent for the Specialists modernization program.
|
|
3940
|
+
|
|
3941
|
+
Authoritative inputs, in order:
|
|
3942
|
+
1. Current code and executable schemas in the connected repositories.
|
|
3943
|
+
2. xtrm-dev/specialists/docs/design/roadmap/specialists-roadmap.md.
|
|
3944
|
+
3. specialists_prompt_chain_evals_prd.md, especially Sections 13–18.
|
|
3945
|
+
4. xtrm/docs/channels/channels.md for future channel semantics.
|
|
3946
|
+
5. The newest dated xtrm _meta reconciliation; the 2026-06-21 file is historical.
|
|
3947
|
+
|
|
3948
|
+
Planning-only mandate:
|
|
3949
|
+
- Do not edit production code.
|
|
3950
|
+
- Do not begin implementation.
|
|
3951
|
+
- Inspect the installed Beads version, `bd ready/create/formula/gate/history/memories/migrate` capabilities, the live formula catalog and current open/closed Beads before creating duplicates.
|
|
3952
|
+
- Reconcile existing roadmap Beads to PRD work packages rather than blindly creating a second backlog.
|
|
3953
|
+
- Preserve every WP-* identifier from Section 15.
|
|
3954
|
+
- Produce `bead-graph-plan.json`, run `bd create --graph --dry-run`, then create or update program/workstream epics and root change Beads only after review.
|
|
3955
|
+
- Add explicit dependencies and typed relationships.
|
|
3956
|
+
- Assign repository, priority, scrutiny, recommended_template, primary role, lane, file ownership, eval gates, non-goals, rollback and bridge-retirement trigger.
|
|
3957
|
+
- Use the root change-contract template in Section 14.3.
|
|
3958
|
+
- Do not materialize specialist step Beads during Pass 1 unless the current runtime requires a bridge tracking Bead; label such work bridge:manual-composition.
|
|
3959
|
+
- Flag file collisions and propose a safe multiplexing wave.
|
|
3960
|
+
- Map every acceptance criterion in Section 18 to one or more Beads.
|
|
3961
|
+
- Mark existing Beads as reused, superseded, extended or duplicate candidates; never silently discard them.
|
|
3962
|
+
|
|
3963
|
+
Required outputs:
|
|
3964
|
+
1. Program hierarchy and Bead IDs.
|
|
3965
|
+
2. WP-to-Bead mapping table.
|
|
3966
|
+
3. Dependency DAG and critical path.
|
|
3967
|
+
4. Initial parallel wave with assigned agent identities from Section 15.14.
|
|
3968
|
+
5. File-ownership/collision matrix.
|
|
3969
|
+
6. Existing-Bead reconciliation report.
|
|
3970
|
+
7. Acceptance-criterion coverage report.
|
|
3971
|
+
8. Open decisions that genuinely block dispatch.
|
|
3972
|
+
9. Beads capability/version report and graph dry-run result.
|
|
3973
|
+
10. Exact next dispatch commands or xtmux handoff pointers, but do not execute them.
|
|
3974
|
+
|
|
3975
|
+
Stop and report rather than inventing a design when canonical documents conflict materially.
|
|
3976
|
+
```
|
|
3977
|
+
|
|
3978
|
+
# Appendix H — Bridge-era multiplexing dispatcher checklist
|
|
3979
|
+
|
|
3980
|
+
Before dispatch:
|
|
3981
|
+
|
|
3982
|
+
```text
|
|
3983
|
+
[ ] Decomposition and reuse audit approved
|
|
3984
|
+
[ ] Installed Beads version/capability matrix recorded
|
|
3985
|
+
[ ] `bead-graph-plan.json` passed `bd create --graph --dry-run` review
|
|
3986
|
+
[ ] Root Bead contains WP identifier and falsifiable contract
|
|
3987
|
+
[ ] Recommended chain template verified against live catalog
|
|
3988
|
+
[ ] Dependencies satisfied or explicitly represented
|
|
3989
|
+
[ ] Worktree/branch created and unique
|
|
3990
|
+
[ ] File ownership has no collision with active lanes
|
|
3991
|
+
[ ] Candidate fingerprint/baseline requirement known
|
|
3992
|
+
[ ] Eval and validation commands named
|
|
3993
|
+
[ ] Bridge retirement trigger present where applicable
|
|
3994
|
+
```
|
|
3995
|
+
|
|
3996
|
+
Dispatch and coordination:
|
|
3997
|
+
|
|
3998
|
+
```bash
|
|
3999
|
+
# Inspect current agent/worktree state
|
|
4000
|
+
tmux-session-picker dashboard expanded
|
|
4001
|
+
tmux-session-picker worktree-collisions
|
|
4002
|
+
tmux-session-picker audit
|
|
4003
|
+
|
|
4004
|
+
# Produce a dry-run handoff pointer for an idle target
|
|
4005
|
+
tmux-session-picker handoff --target <pane> --bead <bead-id> --note 'WP-XX; read root contract; no scope expansion'
|
|
4006
|
+
|
|
4007
|
+
# Send after reviewing the dry-run
|
|
4008
|
+
tmux-session-picker handoff --yes --target <pane> --bead <bead-id> --note 'WP-XX; read root contract; no scope expansion'
|
|
4009
|
+
|
|
4010
|
+
# Short pointer/status messages only
|
|
4011
|
+
tmux-session-picker message-send --from <orchestrator> --to <agent> --bead <bead-id> --text 'dependency WP-YY merged; rebase/check before continuing'
|
|
4012
|
+
|
|
4013
|
+
# Recipient acknowledgement
|
|
4014
|
+
tmux-session-picker message-list --for <agent> --unacked
|
|
4015
|
+
tmux-session-picker message-ack <message-id> --by <agent>
|
|
4016
|
+
```
|
|
4017
|
+
|
|
4018
|
+
Before integration:
|
|
4019
|
+
|
|
4020
|
+
```text
|
|
4021
|
+
[ ] Structured implementation handoff present
|
|
4022
|
+
[ ] Changed files match declared ownership
|
|
4023
|
+
[ ] Deterministic validation evidence present
|
|
4024
|
+
[ ] Named eval suite executed or explicitly deferred by gate
|
|
4025
|
+
[ ] No hidden cross-WP scope expansion
|
|
4026
|
+
[ ] New findings have discovered-from follow-up Beads
|
|
4027
|
+
[ ] Compatibility and migration tests pass
|
|
4028
|
+
[ ] Native Beads readiness/claim/gate evidence agrees with Specialists projection
|
|
4029
|
+
[ ] Integration reviewer verdict recorded
|
|
4030
|
+
[ ] Dependent lane owners notified through durable Bead note + pointer message
|
|
4031
|
+
```
|
|
4032
|
+
|
|
4033
|
+
|
|
4034
|
+
|
|
4035
|
+
# Appendix I — Filed Beads reconciliation matrix
|
|
4036
|
+
|
|
4037
|
+
The following mapping is normative for updating the Beads packet generated on 13 July 2026. Status describes the contract, not whether the Bead remains open.
|
|
4038
|
+
|
|
4039
|
+
| Existing Bead | Disposition | Final WP mapping | Required action |
|
|
4040
|
+
|---|---|---|---|
|
|
4041
|
+
| `unitAI-6639v` | **REFINE** | IC01–IC06, IC11–IC13 | Retain epic, expand objective beyond skill split/parity to full interactive coordinator bridge and roadmap integration. Remove stale assumptions about project-local skill paths. |
|
|
4042
|
+
| `unitAI-6639v.1` | **REFINE** | IC01 | Add current release baseline, two launch modes, xtmux V2 primitives, future coordinator four-duty mapping, roadmap Opportunity mapping and explicit secure-delivery decision. |
|
|
4043
|
+
| `unitAI-6639v.2` | **REFINE/SPLIT** | IC02,IC03,IC04 | Keep skill restructuring but separate coordinator prompt and coordinator-specific doctrine. Remove any Core implementation scope. |
|
|
4044
|
+
| `unitAI-6639v.3` | **REFINE** | IC12 | Expand scenarios to preheat/direct, readiness race, no-leak, role/assignment ack, chain-template selection, conflict matrix, cross-coordinator mechanics and parent-context reduction. |
|
|
4045
|
+
| `unitAI-6639v.4` | **REFINE** | IC05,IC06 | Support `tracked` and `standby` render modes; return secure file + metadata; reuse task-side assembly; no system prompt; audit duplicate Bead context. |
|
|
4046
|
+
| `xtrm-k2ufi` | **REFINE** | IC07–IC10 | Retain Core epic but change transport from presumed positional prompt to readiness-gated secure pointer delivery; include preheated mode. |
|
|
4047
|
+
| `xtrm-k2ufi.1` | **REPLACE CONTRACT** | IC07,IC09,IC10 | Remove “final positional argument” as default. Add `starting/agent.ready`, renderer-before-provision direct mode, prompt file, xtmux handoff, preheat and atomic cleanup. |
|
|
4048
|
+
| `xtrm-k2ufi.2` | **REFINE** | IC12 | Add readiness, preheat, direct launch, second-turn interactivity, prompt-leak inspection, message/ack behavior and cleanup across Pi/Claude. |
|
|
4049
|
+
| No filed xtmux epic | **ADD** | IC08 plus xtmux share of IC07/IC10/IC11 | Create explicit xtmux work rather than hiding handoff/readiness changes inside Core. |
|
|
4050
|
+
| No filed roadmap-hook work | **ADD** | H01–H05 | Create only after G02/reuse audit; preserve shared kernel and shadow-first rollout. |
|
|
4051
|
+
|
|
4052
|
+
# Appendix J — Exact required edits to filed Beads
|
|
4053
|
+
|
|
4054
|
+
## J.1 `unitAI-6639v` epic
|
|
4055
|
+
|
|
4056
|
+
Add to SUCCESS:
|
|
4057
|
+
|
|
4058
|
+
```text
|
|
4059
|
+
The interactive chain-coordinator becomes the bridge implementation of future
|
|
4060
|
+
chain-coordination judgment: it validates template/scope/gates, maintains child-chain
|
|
4061
|
+
collision state, communicates mechanical hygiene to peer coordinators, escalates vision
|
|
4062
|
+
and ambiguous judgment to the parent, and performs close-time evidence/follow-up checks.
|
|
4063
|
+
```
|
|
4064
|
+
|
|
4065
|
+
Add to SCOPE:
|
|
4066
|
+
|
|
4067
|
+
```text
|
|
4068
|
+
- coordinating-chains coordinator-specific skill/reference
|
|
4069
|
+
- interactive tracked + standby assignment contract
|
|
4070
|
+
- roadmap amendment and Bead/WP reconciliation
|
|
4071
|
+
- coordinator behavior/eval fixtures
|
|
4072
|
+
```
|
|
4073
|
+
|
|
4074
|
+
Remove or supersede:
|
|
4075
|
+
|
|
4076
|
+
```text
|
|
4077
|
+
- any implication that parity is only “same initial prompt as sp run”
|
|
4078
|
+
- any dependency on project-local .xtrm/skills/active paths
|
|
4079
|
+
```
|
|
4080
|
+
|
|
4081
|
+
## J.2 `unitAI-6639v.1`
|
|
4082
|
+
|
|
4083
|
+
The parity matrix must classify:
|
|
4084
|
+
|
|
4085
|
+
```text
|
|
4086
|
+
system prompt deliberately separate
|
|
4087
|
+
root using-specialists skill align by effective installed resource
|
|
4088
|
+
phase-specific references on-demand
|
|
4089
|
+
tracked task template align
|
|
4090
|
+
Bead/dependency context align
|
|
4091
|
+
mandatory rules align, task-side and current
|
|
4092
|
+
specialist run context interactive variant
|
|
4093
|
+
output contract omit or adapt; persistent session is not a terminal job
|
|
4094
|
+
memories follow approved pull policy
|
|
4095
|
+
GitNexus snapshot classify by coordinator phase
|
|
4096
|
+
reviewer diff context not applicable unless coordinator is acting as gate
|
|
4097
|
+
pre-script output classify
|
|
4098
|
+
permissions deliberately deferred
|
|
4099
|
+
extensions/hooks runtime-specific but fingerprinted
|
|
4100
|
+
lifecycle telemetry xtmux interactive semantics, not sp job semantics
|
|
4101
|
+
```
|
|
4102
|
+
|
|
4103
|
+
Add evidence from Specialists `3.18.2`, Core `0.10.2` and xtmux V2 default-on.
|
|
4104
|
+
|
|
4105
|
+
## J.3 `unitAI-6639v.2`
|
|
4106
|
+
|
|
4107
|
+
Split implementation ownership:
|
|
4108
|
+
|
|
4109
|
+
```text
|
|
4110
|
+
IC02 progressive skill tree/global install
|
|
4111
|
+
IC03 coordinator stable prompt
|
|
4112
|
+
IC04 coordinator-specific doctrine/templates/conflicts/close behavior
|
|
4113
|
+
```
|
|
4114
|
+
|
|
4115
|
+
Do not include Core launcher edits. Add explicit validation that the coordinator no longer hardcodes the production chain sequence.
|
|
4116
|
+
|
|
4117
|
+
## J.4 `unitAI-6639v.3`
|
|
4118
|
+
|
|
4119
|
+
Add validation cases:
|
|
4120
|
+
|
|
4121
|
+
```text
|
|
4122
|
+
preheated role sends role.ready and does no work before assignment
|
|
4123
|
+
direct --bead waits for runtime ready before prompt delivery
|
|
4124
|
+
tracked assignment contains Bead + mandatory rules exactly once
|
|
4125
|
+
prompt not present in system prompt, argv, pane metadata, message body or telemetry
|
|
4126
|
+
Pi and Claude remain interactive for a second turn
|
|
4127
|
+
coordinator selects code-standard/security-deep/etc. from catalog correctly
|
|
4128
|
+
coordinator detects seeded path conflict and chooses a justified strategy
|
|
4129
|
+
coordinator sends mechanics to peer and vision/escalation to parent
|
|
4130
|
+
close-readiness false-positive corpus
|
|
4131
|
+
```
|
|
4132
|
+
|
|
4133
|
+
## J.5 `unitAI-6639v.4`
|
|
4134
|
+
|
|
4135
|
+
Replace the output contract with:
|
|
4136
|
+
|
|
4137
|
+
```json
|
|
4138
|
+
{
|
|
4139
|
+
"schema": "specialists.interactive-assignment.v1",
|
|
4140
|
+
"role": "chain-coordinator",
|
|
4141
|
+
"mode": "tracked|standby",
|
|
4142
|
+
"bead_id": "nullable",
|
|
4143
|
+
"prompt_file": "/secure/path",
|
|
4144
|
+
"prompt_sha256": "...",
|
|
4145
|
+
"component_ids": [],
|
|
4146
|
+
"mandatory_rule_ids": [],
|
|
4147
|
+
"bytes": 0,
|
|
4148
|
+
"estimated_tokens": 0
|
|
4149
|
+
}
|
|
4150
|
+
```
|
|
4151
|
+
|
|
4152
|
+
For tracked mode, content order is task-template/Bead/dependencies then mandatory rules. For standby mode, content is a minimal parent-ack/wait bootstrap.
|
|
4153
|
+
|
|
4154
|
+
## J.6 `xtrm-k2ufi` and `.1`
|
|
4155
|
+
|
|
4156
|
+
Replace the positional-prompt transport assumption with:
|
|
4157
|
+
|
|
4158
|
+
```text
|
|
4159
|
+
call Specialists renderer from original cwd
|
|
4160
|
+
fail before provisioning on tracked render failure
|
|
4161
|
+
create worktree/session with @agent_state=starting
|
|
4162
|
+
wait for runtime agent.ready/idle
|
|
4163
|
+
call xtmux handoff with existing prompt file
|
|
4164
|
+
safe-send-pointer sends one-line pointer into prompt input
|
|
4165
|
+
automatic monitor is armed or verified
|
|
4166
|
+
retain Bead metadata
|
|
4167
|
+
```
|
|
4168
|
+
|
|
4169
|
+
Add preheat behavior when no Bead is provided. Core does not own prompt-file contents or xtmux SQLite internals.
|
|
4170
|
+
|
|
4171
|
+
## J.7 `xtrm-k2ufi.2`
|
|
4172
|
+
|
|
4173
|
+
Add no-leak and readiness inspection across:
|
|
4174
|
+
|
|
4175
|
+
```text
|
|
4176
|
+
ps/process argv
|
|
4177
|
+
tmux pane options
|
|
4178
|
+
agent.role.launched event
|
|
4179
|
+
delivery_attempts and handoff rows
|
|
4180
|
+
message summaries
|
|
4181
|
+
Core stdout/stderr
|
|
4182
|
+
transport artifact permissions and cleanup
|
|
4183
|
+
```
|
|
4184
|
+
|
|
4185
|
+
## J.8 New xtmux epic
|
|
4186
|
+
|
|
4187
|
+
Recommended title:
|
|
4188
|
+
|
|
4189
|
+
```text
|
|
4190
|
+
Readiness-gated interactive role assignment and preheated coordinator handoff
|
|
4191
|
+
```
|
|
4192
|
+
|
|
4193
|
+
Children:
|
|
4194
|
+
|
|
4195
|
+
```text
|
|
4196
|
+
1. startup readiness contract for Claude/Pi integration
|
|
4197
|
+
2. existing-prompt-file handoff + wait-ready
|
|
4198
|
+
3. preheated role/assignment acknowledgement protocol
|
|
4199
|
+
4. durable delivery/ack/no-leak tests
|
|
4200
|
+
5. coordinator mechanical-message conventions
|
|
4201
|
+
```
|
|
4202
|
+
|
|
4203
|
+
# Appendix K — Required direct documentation changes
|
|
4204
|
+
|
|
4205
|
+
## K.1 `specialists-roadmap.md`
|
|
4206
|
+
|
|
4207
|
+
The roadmap patch must include all items from Sections 13.1 and 13.1A–B, update shipped status to Specialists `3.18.2`, add Track D (policy hooks/xtmux architecture) and Track E (interactive coordinator), and link this PRD. It must not copy all WP tables into the roadmap.
|
|
4208
|
+
|
|
4209
|
+
The roadmap should show this bridge sequence:
|
|
4210
|
+
|
|
4211
|
+
```text
|
|
4212
|
+
telemetry/eval baseline
|
|
4213
|
+
+ interactive content/parity audit
|
|
4214
|
+
→ readiness + secure assignment bridge
|
|
4215
|
+
→ critical chain-first spine
|
|
4216
|
+
→ coordinator consumes resolved shape
|
|
4217
|
+
→ evaluated prompt/rule/memory modernization
|
|
4218
|
+
→ Console materializer expansion
|
|
4219
|
+
→ eventual Substrate/Channels replacement
|
|
4220
|
+
```
|
|
4221
|
+
|
|
4222
|
+
## K.2 xtrm `_meta`
|
|
4223
|
+
|
|
4224
|
+
The 21 June reconciliation remains historical. Add a banner and create a new dated reconciliation recording:
|
|
4225
|
+
|
|
4226
|
+
- Specialists `3.18.2`, Core `0.10.2`, xtmux SQLite V2 default-on; the xtmux hook/extension audit is treated as an earlier independent design input, before monorepo and coordinator work;
|
|
4227
|
+
- Track A/C/D immediate work and delayed Track B;
|
|
4228
|
+
- per-repository Specialists `observability.db` and Console materializer runway;
|
|
4229
|
+
- two-mode interactive assignment;
|
|
4230
|
+
- current/future coordinator lifecycle distinction;
|
|
4231
|
+
- bridge retirement triggers.
|
|
4232
|
+
|
|
4233
|
+
## K.3 Core documentation
|
|
4234
|
+
|
|
4235
|
+
Update `docs/xt-pi-role.md` with:
|
|
4236
|
+
|
|
4237
|
+
- current global skill resolution;
|
|
4238
|
+
- preheated and direct tracked modes;
|
|
4239
|
+
- readiness and assignment flow;
|
|
4240
|
+
- Bead metadata versus startup snapshot distinction;
|
|
4241
|
+
- secure pointer transport and no-leak contract;
|
|
4242
|
+
- role/message address spaces and acknowledgement semantics.
|
|
4243
|
+
|
|
4244
|
+
## K.4 xtmux documentation
|
|
4245
|
+
|
|
4246
|
+
Update the observability/coordination docs with:
|
|
4247
|
+
|
|
4248
|
+
- `agent.ready`/starting semantics;
|
|
4249
|
+
- handoff of a precomposed Specialists assignment;
|
|
4250
|
+
- prompt delivery as a `delivery_attempt`, not a durable semantic message;
|
|
4251
|
+
- coordinator message vocabulary;
|
|
4252
|
+
- exact boundary from Specialists and future Channels.
|
|
4253
|
+
|
|
4254
|
+
# Appendix L — Decomposition handoff for the originating PRD agent
|
|
4255
|
+
|
|
4256
|
+
The agent receiving this document must not regenerate a fresh backlog without reconciling live Beads. Required sequence:
|
|
4257
|
+
|
|
4258
|
+
```text
|
|
4259
|
+
1. Read current code and release baselines.
|
|
4260
|
+
2. Read specialists-roadmap.md and chain-template catalog.
|
|
4261
|
+
3. Read this consolidated PRD.
|
|
4262
|
+
4. Inspect the filed Beads packet and live Beads state.
|
|
4263
|
+
5. Apply Appendix I dispositions and Appendix J contract changes.
|
|
4264
|
+
6. Create missing xtmux and policy-hook Beads.
|
|
4265
|
+
7. Produce a WP-to-Bead mapping with no orphan acceptance criteria.
|
|
4266
|
+
8. Recompute dependency DAG and initial parallel wave.
|
|
4267
|
+
9. Run composition review before implementation dispatch.
|
|
4268
|
+
```
|
|
4269
|
+
|
|
4270
|
+
Required output report:
|
|
4271
|
+
|
|
4272
|
+
```text
|
|
4273
|
+
updated/reused/superseded Bead IDs
|
|
4274
|
+
WP mapping
|
|
4275
|
+
changed descriptions and dependency edges
|
|
4276
|
+
new missing Beads
|
|
4277
|
+
critical path
|
|
4278
|
+
parallel-safe wave
|
|
4279
|
+
file collision matrix
|
|
4280
|
+
eval/validation coverage
|
|
4281
|
+
roadmap/meta documentation patch plan
|
|
4282
|
+
```
|