@luminascale/shepherd 0.0.0-stage → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (203) hide show
  1. package/.agents/agents/README.md +12 -0
  2. package/.agents/agents/planner.md +38 -0
  3. package/.agents/agents/reviewer.md +38 -0
  4. package/.agents/agents/scout.md +43 -0
  5. package/.agents/agents/worker.md +33 -0
  6. package/.agents/skills/smoke-test/SKILL.md +155 -0
  7. package/.agents/skills/smoke-test/references/forensics.md +46 -0
  8. package/.agents/skills/smoke-test/scripts/live-smoke-poll.mjs +130 -0
  9. package/.github/workflows/publish.yml +77 -0
  10. package/.pi/prompts/release.md +21 -0
  11. package/.prettierrc +10 -0
  12. package/AGENTS.md +88 -0
  13. package/CONTRIBUTING.md +177 -0
  14. package/LICENSE +201 -0
  15. package/README.md +424 -2
  16. package/docs/README.md +25 -0
  17. package/docs/guides/agent-definitions.md +42 -0
  18. package/docs/guides/diagnostics.md +199 -0
  19. package/docs/guides/tool-reference.md +55 -0
  20. package/docs/manual-tests/ping-pong-peer-messaging.md +147 -0
  21. package/docs/plans/agent-labels.md +257 -0
  22. package/docs/plans/architecture.md +110 -0
  23. package/docs/plans/settings-scope.md +163 -0
  24. package/docs/plans/tool-surface-split.md +278 -0
  25. package/docs/reference/dictionary +26 -0
  26. package/docs/releasing.md +47 -0
  27. package/docs/specs/005-agent-messaging-and-task-lifecycle/baseline.md +115 -0
  28. package/docs/specs/005-agent-messaging-and-task-lifecycle/handoff.md +564 -0
  29. package/docs/specs/005-agent-messaging-and-task-lifecycle/phase10-notes.md +109 -0
  30. package/docs/specs/005-agent-messaging-and-task-lifecycle/phase6-notes.md +69 -0
  31. package/docs/specs/005-agent-messaging-and-task-lifecycle/phase7-notes.md +100 -0
  32. package/docs/specs/005-agent-messaging-and-task-lifecycle/phase8-notes.md +127 -0
  33. package/docs/specs/005-agent-messaging-and-task-lifecycle/phase9-notes.md +104 -0
  34. package/docs/specs/005-agent-messaging-and-task-lifecycle/plan.md +1093 -0
  35. package/docs/specs/005-agent-messaging-and-task-lifecycle/spec.md +583 -0
  36. package/docs/specs/005-agent-messaging-and-task-lifecycle/tasks.md +838 -0
  37. package/docs/specs/006-shepherd-tool-rendering/plan.md +175 -0
  38. package/docs/specs/006-shepherd-tool-rendering/spec.md +469 -0
  39. package/index.ts +599 -0
  40. package/package.json +112 -4
  41. package/scripts/capture-system-prompt.mjs +45 -0
  42. package/scripts/extract-pi-system-prompt.mjs +277 -0
  43. package/scripts/host-impact.mjs +74 -0
  44. package/scripts/resolve-agent.mjs +43 -0
  45. package/scripts/show-shepherd-orchestrator.mjs +58 -0
  46. package/scripts/show-shepherd-prompt.mjs +96 -0
  47. package/scripts/verify-suites.mjs +67 -0
  48. package/specs/001-artifacts/spec.md +436 -0
  49. package/specs/002-simplify-agent-orchestration/plan.md +287 -0
  50. package/specs/002-simplify-agent-orchestration/spec.md +323 -0
  51. package/specs/002-simplify-agent-orchestration/tasks.md +212 -0
  52. package/specs/003-orchestrator-session-artifacts/plan.md +396 -0
  53. package/specs/003-orchestrator-session-artifacts/spec.md +453 -0
  54. package/specs/003-orchestrator-session-artifacts/tasks.md +141 -0
  55. package/specs/004-asynchronous-prompt-watchers/plan.md +139 -0
  56. package/specs/004-asynchronous-prompt-watchers/spec.md +202 -0
  57. package/specs/004-asynchronous-prompt-watchers/tasks.md +126 -0
  58. package/specs/005-per-workspace-settings/plan.md +230 -0
  59. package/specs/005-per-workspace-settings/spec.md +368 -0
  60. package/specs/005-per-workspace-settings/tasks.md +178 -0
  61. package/specs/006-dual-runtime-herdr-durable/intent.md +86 -0
  62. package/specs/006-dual-runtime-herdr-durable/journal.md +80 -0
  63. package/specs/006-dual-runtime-herdr-durable/plan.md +242 -0
  64. package/specs/006-dual-runtime-herdr-durable/spec.md +389 -0
  65. package/specs/006-dual-runtime-herdr-durable/tasks.md +199 -0
  66. package/specs/007-unwatched-completion-notice/journal.md +145 -0
  67. package/specs/007-unwatched-completion-notice/plan.md +186 -0
  68. package/specs/007-unwatched-completion-notice/spec.md +284 -0
  69. package/specs/007-unwatched-completion-notice/tasks.md +147 -0
  70. package/specs/008-remote-machine-sheep/intent.md +174 -0
  71. package/specs/008-remote-machine-sheep/journal.md +57 -0
  72. package/specs/008-remote-machine-sheep/prerequisites.md +219 -0
  73. package/specs/009-hc-runtime/intent.md +44 -0
  74. package/specs/009-hc-runtime/plan.md +49 -0
  75. package/specs/009-hc-runtime/spec.md +118 -0
  76. package/specs/009-hc-runtime/tasks.md +62 -0
  77. package/specs/010-unseen-watch-parity/intent.md +57 -0
  78. package/specs/010-unseen-watch-parity/journal.md +172 -0
  79. package/specs/010-unseen-watch-parity/plan.md +74 -0
  80. package/specs/010-unseen-watch-parity/spec.md +191 -0
  81. package/specs/010-unseen-watch-parity/tasks.md +119 -0
  82. package/specs/011-hc-background-poll/intent.md +57 -0
  83. package/specs/011-hc-background-poll/journal.md +95 -0
  84. package/specs/011-hc-background-poll/plan.md +87 -0
  85. package/specs/011-hc-background-poll/spec.md +141 -0
  86. package/specs/011-hc-background-poll/tasks.md +83 -0
  87. package/specs/012-steer-default/intent.md +12 -0
  88. package/specs/012-steer-default/journal.md +40 -0
  89. package/specs/012-steer-default/plan.md +7 -0
  90. package/specs/012-steer-default/spec.md +42 -0
  91. package/specs/012-steer-default/tasks.md +6 -0
  92. package/specs/013-opencode-shepherd/handoff-t011.md +94 -0
  93. package/specs/013-opencode-shepherd/intent.md +59 -0
  94. package/specs/013-opencode-shepherd/plan.md +204 -0
  95. package/specs/013-opencode-shepherd/spec.md +258 -0
  96. package/specs/013-opencode-shepherd/tasks.md +259 -0
  97. package/src/core/artifact-sessions.ts +835 -0
  98. package/src/core/config.ts +329 -0
  99. package/src/core/discovery.ts +342 -0
  100. package/src/core/herdr.ts +1 -0
  101. package/src/core/host.ts +77 -0
  102. package/src/core/ids.ts +78 -0
  103. package/src/core/lifecycle.ts +1 -0
  104. package/src/core/messaging.ts +1 -0
  105. package/src/core/migration.ts +55 -0
  106. package/src/core/orchestration.ts +1 -0
  107. package/src/core/types.ts +143 -0
  108. package/src/core/unseen-completions.ts +114 -0
  109. package/src/extension/cli.ts +417 -0
  110. package/src/extension/config.ts +1 -0
  111. package/src/extension/settings-ui.ts +296 -0
  112. package/src/extension/shepherd-done.ts +768 -0
  113. package/src/extension/shepherd.ts +2548 -0
  114. package/src/extension/system-prompt.ts +38 -0
  115. package/src/extension/unseen-completions.ts +237 -0
  116. package/src/opencode/actions.ts +794 -0
  117. package/src/opencode/bridge.ts +777 -0
  118. package/src/opencode/display.ts +36 -0
  119. package/src/opencode/fieldnotes.ts +60 -0
  120. package/src/opencode/host.ts +62 -0
  121. package/src/opencode/orphans.ts +162 -0
  122. package/src/opencode/server.ts +458 -0
  123. package/src/runtimes/backend.ts +119 -0
  124. package/src/runtimes/detect.ts +102 -0
  125. package/src/runtimes/hc/hc.ts +527 -0
  126. package/src/runtimes/hc/runtime.ts +1898 -0
  127. package/src/runtimes/herdr/herdr.ts +912 -0
  128. package/src/runtimes/herdr/lifecycle.ts +1712 -0
  129. package/src/runtimes/herdr/messaging.ts +703 -0
  130. package/src/runtimes/herdr/orchestration.ts +1583 -0
  131. package/src/runtimes/herdr/runtime.ts +224 -0
  132. package/src/runtimes/runtime.ts +133 -0
  133. package/src/runtimes/settle-observations.ts +114 -0
  134. package/test/fixtures/context-project/AGENTS.md +2 -0
  135. package/test/fixtures/home/.agents/agents/same.md +6 -0
  136. package/test/fixtures/home/.agents/agents/shared-only.md +6 -0
  137. package/test/fixtures/home/.pi/agent/agents/invalid-thinking.md +6 -0
  138. package/test/fixtures/home/.pi/agent/agents/model-empty.md +7 -0
  139. package/test/fixtures/home/.pi/agent/agents/model-explicit.md +7 -0
  140. package/test/fixtures/home/.pi/agent/agents/omit-context-absent.md +5 -0
  141. package/test/fixtures/home/.pi/agent/agents/omit-context-false.md +6 -0
  142. package/test/fixtures/home/.pi/agent/agents/omit-context-invalid.md +6 -0
  143. package/test/fixtures/home/.pi/agent/agents/omit-context-true.md +6 -0
  144. package/test/fixtures/home/.pi/agent/agents/omit-false.md +6 -0
  145. package/test/fixtures/home/.pi/agent/agents/omit-invalid.md +6 -0
  146. package/test/fixtures/home/.pi/agent/agents/omit-pi-docs.md +7 -0
  147. package/test/fixtures/home/.pi/agent/agents/omit-true.md +6 -0
  148. package/test/fixtures/home/.pi/agent/agents/only-user.md +9 -0
  149. package/test/fixtures/home/.pi/agent/agents/same.md +6 -0
  150. package/test/fixtures/proj/.agents/agents/proj-shared.md +6 -0
  151. package/test/fixtures/proj/.pi/agents/project-pi.md +6 -0
  152. package/test/fixtures/proj/.pi/agents/same.md +6 -0
  153. package/test/helpers/fake-clock.mjs +39 -0
  154. package/test/helpers/feature-contracts.mjs +44 -0
  155. package/test/helpers/test-utils.mjs +44 -0
  156. package/test/verify-artifact-sessions.mjs +258 -0
  157. package/test/verify-backend.mjs +297 -0
  158. package/test/verify-child-mislabel.mjs +135 -0
  159. package/test/verify-child-surface.mjs +321 -0
  160. package/test/verify-cli.mjs +197 -0
  161. package/test/verify-command-ux.mjs +100 -0
  162. package/test/verify-compat.mjs +32 -0
  163. package/test/verify-cross-host-notes.mjs +128 -0
  164. package/test/verify-delegate.mjs +161 -0
  165. package/test/verify-detect.mjs +151 -0
  166. package/test/verify-discovery.mjs +239 -0
  167. package/test/verify-done-nudge.mjs +238 -0
  168. package/test/verify-effective-prompt.mjs +98 -0
  169. package/test/verify-hc-matrix.mjs +246 -0
  170. package/test/verify-hc-poll.mjs +260 -0
  171. package/test/verify-host-docs.mjs +35 -0
  172. package/test/verify-host-impact.mjs +49 -0
  173. package/test/verify-ids.mjs +141 -0
  174. package/test/verify-launch.mjs +183 -0
  175. package/test/verify-message-routing.mjs +560 -0
  176. package/test/verify-messaging.mjs +303 -0
  177. package/test/verify-migration-notice.mjs +69 -0
  178. package/test/verify-multi-wait.mjs +42 -0
  179. package/test/verify-opencode-display.mjs +47 -0
  180. package/test/verify-opencode-fieldnotes.mjs +121 -0
  181. package/test/verify-opencode-notice.mjs +77 -0
  182. package/test/verify-opencode-nudge.mjs +116 -0
  183. package/test/verify-opencode-orphans.mjs +160 -0
  184. package/test/verify-opencode-surface.mjs +107 -0
  185. package/test/verify-packaging.mjs +82 -0
  186. package/test/verify-parent-surface.mjs +118 -0
  187. package/test/verify-phase0.mjs +95 -0
  188. package/test/verify-registry.mjs +124 -0
  189. package/test/verify-rename-migration.mjs +111 -0
  190. package/test/verify-rendering.mjs +379 -0
  191. package/test/verify-session-boundary.mjs +146 -0
  192. package/test/verify-settings.mjs +260 -0
  193. package/test/verify-stale-wait.mjs +339 -0
  194. package/test/verify-status-widget.mjs +377 -0
  195. package/test/verify-status.mjs +172 -0
  196. package/test/verify-task-completion.mjs +156 -0
  197. package/test/verify-task-failures.mjs +201 -0
  198. package/test/verify-task-mislabel.mjs +143 -0
  199. package/test/verify-task-registry.mjs +194 -0
  200. package/test/verify-task-watchers.mjs +217 -0
  201. package/test/verify-tool-schemas.mjs +120 -0
  202. package/test/verify-unseen-completions.mjs +1211 -0
  203. package/test/verify-watchers.mjs +181 -0
@@ -0,0 +1,389 @@
1
+ # Dual-runtime Shepherd: Herdr + Durable
2
+
3
+ ## Status
4
+
5
+ Proposed specification. No behavior change yet.
6
+
7
+ ## Summary
8
+
9
+ Add a `durable` execution backend to pi-shepherd alongside the existing
10
+ `herdr` backend, behind one stable Shepherd surface.
11
+
12
+ Today pi-shepherd is Herdr-native: `shepherd_spawn` creates a real Herdr
13
+ pane with a full `pi` process, tracked in `created-panes.json`, with shared
14
+ context in `.shepherd/sessions/` fieldnotes. That stays the default and the
15
+ hero.
16
+
17
+ The new backend runs the same orchestrator pattern — orchestrator launches a
18
+ specialist per task, crash-reviewer follows test-driver on failure — as
19
+ durable owned conversations + tasks in `@earendil-works/pi-durable`
20
+ (`Harness`, `Conversation`, `Submission`, SQLite/Memory storage) instead of
21
+ OS processes. Same tool names, same opaque IDs, same agent Markdown
22
+ definitions, same agent-facing `shepherd.md` shared memory.
23
+
24
+ Motivation is a concrete existing workflow plus a concrete new requirement:
25
+
26
+ 1. Existing: orchestrator launches one agent per task to drive the program
27
+ under test; if the app crashes, a different-context agent reviews the
28
+ crash. This must keep working unchanged in Herdr.
29
+ 2. New: company AI test agent, multi-user, per-user models (free local vs.
30
+ company Copilot subscription), test plans run overnight, report in the
31
+ morning. Herdr panes alone cannot do crash-resume, single-owner
32
+ persistence, usage accounting, or headless operation. Durable can.
33
+
34
+ ## Concept guardrail: no washout
35
+
36
+ Shepherd's name comes from herding in Herdr, and that concept must not
37
+ dilute. To prevent it:
38
+
39
+ - Shepherd = orchestration pattern (`spawn -> delegate -> watch -> done`,
40
+ visible sheep, shared fieldnotes), not an execution substrate. Herdr is
41
+ today's pasture, durable is the night pen. Same flock.
42
+ - Herdr remains default and documented first. `backend: "herdr"` default.
43
+ `/shepherd herd/spawn/read`, 🐑 widget, pane visibility unchanged.
44
+ - Durable is opt-in (`backend: "durable" | "auto"`), experimental.
45
+ - No durable vocabulary leaks: users see `agent ID / task ID / message ID`,
46
+ never `conversationId / submissionId / pi.live`. `shepherd_read` tails a
47
+ pane *or* a view; caller does not care.
48
+ - Files stay agent-facing: `.shepherd/sessions/.../shepherd.md` + per-agent
49
+ notes are the shepherd/sheep shared working memory. They survive
50
+ compaction, waiting, close, timeout, and restart, and are the agents'
51
+ source of truth — never the human handoff. Durable docs mirror them
52
+ transactionally. The morning human report is a separate templated HTML
53
+ file owned by the calling app (e.g. test-report renderer), built from job
54
+ state — it has nothing to do with shepherd internals.
55
+ - If the existing test-driver -> crash-reviewer prompt runs unchanged on
56
+ both backends, this spec succeeded. If users must rewrite prompts per
57
+ backend, it failed.
58
+
59
+ ## Goals
60
+
61
+ - Keep 100% of current Herdr behavior unchanged when `backend: herdr`.
62
+ - Run the same orchestrator/specialist workflows headless on durable with
63
+ crash `resume()`, `requestId` dedup, queued steers/follow-ups, and
64
+ usage accounting.
65
+ - Reuse agent Markdown definitions (`name/description/tools/model/thinking`)
66
+ on both backends. `model: null/default` inherits Shepherd model on both.
67
+ - Map per-user models: Shepherd session per user carries that user's
68
+ default model; children inherit unless pinned (e.g. cheap local triage
69
+ vs. Copilot deep review).
70
+ - Bridge fieldnotes to durable docs: single commit writes entry + doc,
71
+ file mirror for inspection.
72
+ - Support foreground (return answer) and background (persistent worker,
73
+ survives parent abort) subagents on both backends.
74
+ - Keep safety invariants: only close panes shepherd created; raw pane IDs
75
+ never bypass ownership; project agents stay trust-gated with
76
+ `confirmProjectAgents` user-owned.
77
+ - Keep `timeout` + `staleWaitThreshold` semantics on both backends
78
+ (durable has no native timeout; shepherd implements via abort).
79
+
80
+ ## Non-goals
81
+
82
+ - No replacement of Herdr execution. No removal of panes, widgets, or
83
+ `created-panes.json`.
84
+ - No requirement to run SQLite to use shepherd. Herdr-only installs keep
85
+ working with no new dependencies.
86
+ - No second tool surface. No `durable_spawn` / `herdr_spawn` split in the
87
+ model-facing API. Backend selects inside `shepherd_*`.
88
+ - No automatic migration of in-flight Herdr tasks to durable or vice versa.
89
+ - No multi-writer SQLite. One process owns durable storage at a time;
90
+ federated multi-host is out of scope (document headless-single-owner
91
+ + optional Herdr tail viewer instead).
92
+ - No change to `confirmProjectAgents` security boundary.
93
+
94
+ ## Architecture
95
+
96
+ ### Layout
97
+
98
+ `src/core/` holds shared domain only. Runtime implementations live under
99
+ `src/runtimes/`; the pi-coding-agent surface stays in `src/extension/`:
100
+
101
+ ```text
102
+ src/core/ discovery, artifact-sessions, types, ids, config (settings store)
103
+ src/runtimes/
104
+ runtime.ts ShepherdRuntime interface (opaque types only)
105
+ herdr/ current herdr.ts, orchestration.ts, lifecycle.ts (+ messaging.ts support)
106
+ durable/ new runtime.ts + extension.ts
107
+ src/extension/ pi-coding-agent tools/commands/settings UI; calls runtime.ts
108
+ ```
109
+
110
+ ### Runtime interface
111
+
112
+ Extract what `src/core/{herdr,orchestration,lifecycle}.ts` do today behind
113
+ a pure interface; `messaging.ts` moves along as Herdr child I/O support.
114
+ Existing Herdr code moves, behavior does not.
115
+
116
+ ```ts
117
+ // src/runtimes/runtime.ts (new, no herdr or pi-durable import)
118
+ interface ShepherdRuntime {
119
+ spawn(def: AgentDef, opts: { cwd?: string; label?: string; placement?: "pane_right"|"pane_down"|"tab"|"workspace" }): Promise<AgentHandle>
120
+ delegate(agentId: string, task: string, opts: { requestId?: string; timeoutMin?: number }): Promise<TaskId>
121
+ message(target: AgentId | "parent", msg: string, opts: { taskId?: string; expectsReply?: boolean; replyTo?: string; delivery?: "followUp"|"steer" }): Promise<MessageId>
122
+ watch(taskIds: string[]): Promise<Completion[]>
123
+ status(agentId: string): Promise<{ proc: "idle"|"working"|"unknown"; task: "running"|"waiting"|"done"|"blocked"|"failed"; waitingOn?: string; stale?: boolean }>
124
+ close(agentId: string): Promise<void>
125
+ read(agentId: string, opts?: { lines?: number }): Promise<string>
126
+ }
127
+ ```
128
+
129
+ - `src/core/ids.ts` (new): opaque ID mint/parse — formats and validation
130
+ errors extracted from `LifecycleRegistry` so both runtimes mint and
131
+ validate through one codec.
132
+ - `src/runtimes/herdr/` (new): `herdr.ts`, `orchestration.ts`, and
133
+ `lifecycle.ts` move here rename-only and implement the interface;
134
+ `messaging.ts` (child filesystem broker) moves alongside as a support
135
+ module, not an interface implementer.
136
+ - `src/runtimes/durable/runtime.ts` (new): implements the same interface
137
+ over pi-durable.
138
+ - `src/core/config.ts`: the two-layer settings store moves here from
139
+ `src/extension/config.ts` so runtimes never import the extension layer;
140
+ `/shepherd settings` UI stays in `src/extension/`.
141
+ - `src/extension/shepherd.ts`: unchanged tool schemas. Lifecycle calls
142
+ that map 1:1 to the lifecycle-only interface (today: `delegate` and
143
+ `close`) go through it; calls whose results or callbacks the interface
144
+ does not carry (spawn discovery gate, prompt, message result shape,
145
+ watch callbacks, status detail, read source selection) stay on the
146
+ concrete runtime until a later phase needs them. The per-action routing
147
+ record lives in `tasks.md` 0.3.
148
+
149
+ The interface is lifecycle-only. `/shepherd herd|agents|prune` and the 🐑
150
+ widget read runtime introspection (owned-pane/registry summaries,
151
+ ownership checks) and keep those accessors on the runtime module in
152
+ Phase 0; durable defines its own in Phase 2 (owned conversations/tasks
153
+ only). Introspection must not expose native IDs either.
154
+
155
+ Phase 0 leaves temporary re-export shims at the old `src/core/*` and
156
+ `src/extension/config.ts` paths so the existing `verify-*.mjs` suite
157
+ passes untouched. Shims and import paths are cleaned up together in
158
+ Phase 1.
159
+
160
+ IDs stay opaque session-scoped strings as today (`shepherd_spawn -> agent
161
+ ID`, `shepherd_delegate -> task ID`, `shepherd_message -> message ID`,
162
+ reply via `replyTo`). Internally map to `paneId` OR
163
+ `{ conversationId, submissionId, taskId }`. Never expose Herdr pane IDs or
164
+ durable conversation IDs. Pane IDs remain diagnostic-only for
165
+ `shepherd_read`.
166
+
167
+ ### Durable mapping
168
+
169
+ One sheep = one owned conversation (mirrors pi-durable examples
170
+ `22-subagent-foreground`, `23-subagent-background`):
171
+
172
+ - `spawn` -> `tx.createConversation({ ownership: { kind: "task", taskId: parentTask } })`
173
+ + `configure({ model, thinkingLevel, tools, cwd, instructions })` from
174
+ agent frontmatter. Fresh `pi.provider` identity per sheep (no cache leak
175
+ across users). `placement` ignored; `cwd` honored.
176
+ - `delegate` -> `conversation.submit({ type: "input", content: task, requestId })`.
177
+ `requestId` defaults to shepherd task ID. Retried overnight submits dedup
178
+ instead of double-running tests.
179
+ - child `shepherd_done` -> completes owner task. Parent `shepherd_watch` ->
180
+ `submission.wait() / harness.waitForTask()`.
181
+ - `message` without model impact -> `submit({ type: "write", entry: { kind, data } })`,
182
+ appended immediately even mid-run. With model impact -> steer/follow-up
183
+ (`whenBusy: "steer" | "follow-up"`), queued in `docs["pi.inbox"]`.
184
+ - `expectsReply: true` -> durable `waiting` on another task. Reuse
185
+ `staleWaitThreshold` reminder (informational only) + `timeout` as
186
+ authoritative abort -> `blocked`, same as Herdr today.
187
+ - `close/abort` -> `conversation.abort()` bottom-up (owned work first),
188
+ queued inputs withdrawn, queued writes stay. `background: true` anchor
189
+ tasks implement today's `stayOpen` persistent workers: survive parent
190
+ abort, reporter task posts answer back as follow-up.
191
+ - `status` -> `viewState()` (`entries`, `docs[pi.live, pi.inbox]`) +
192
+ `taskGraph()`. `proc: working` iff live generation/tools running;
193
+ `task: waiting` iff parked on reply.
194
+ - `read` -> tail of committed entries + `pi.live` partials/output
195
+ (or `watchEvents()` deltas), not pane scrollback.
196
+
197
+ Replay policy: test-driving tools that launch apps are NOT `replay: "safe"`.
198
+ Crash-interrupted calls yield `interrupted` error results with partial output,
199
+ never silent rerun. Only idempotent helpers (read-only git/log readers)
200
+ may opt into `replay: "safe"` with keyed `executeTool` reattach.
201
+
202
+ ### Extension duality
203
+
204
+ pi-shepherd today is `ExtensionAPI from '@earendil-works/pi-coding-agent'`.
205
+ pi-durable has its own `defineExtension/defineTool` with
206
+ `(args, api, chordContext)` signatures. Keep one shared core, two adapters:
207
+
208
+ - Shared (no runtime import): `src/core/discovery.ts` (Markdown
209
+ frontmatter: `name/description/tools/model/thinking/omit-*`),
210
+ `src/core/artifact-sessions.ts` (fieldnotes paths/session persistence),
211
+ `src/core/ids.ts` (opaque ID codec), `src/core/types.ts` (tool-schema
212
+ types), and `src/core/config.ts` (settings store + fieldnotes snapshot).
213
+ - Adapter-owned: prompt/instructions assembly (depends on the adapter's
214
+ tool list) and runtime introspection.
215
+ - `src/extension/*`: the pi-coding-agent surface, unchanged behavior; it
216
+ resolves the active runtime through `src/runtimes/runtime.ts`.
217
+ - `src/runtimes/durable/extension.ts` (new): `defineExtension({ name: "shepherd",
218
+ tools: [shepherd, shepherd_spawn, shepherd_delegate, shepherd_message,
219
+ shepherd_prompt, shepherd_watch, shepherd_status, shepherd_close,
220
+ shepherd_read] })` plus the child surface (`shepherd_message` /
221
+ `shepherd_done`) and docs (`ShepherdHerdDoc, ShepherdTasksDoc`). Same
222
+ names and schemas as the pi-coding-agent adapter, including the
223
+ deprecated `shepherd_prompt` compatibility path.
224
+
225
+ `@earendil-works/pi-durable` + `@earendil-works/chord` become optional
226
+ peerDeps. Herdr-only installs must not grow install size or require new
227
+ storage.
228
+
229
+ ### Agent definitions
230
+
231
+ No frontmatter change. Mapping:
232
+
233
+ | Shepherd md | Durable |
234
+ | --- | --- |
235
+ | `tools: read, bash` | `configure({ tools })` + `modelTools` narrowing |
236
+ | `model: null/default` | inherit Shepherd session model (per-user default) |
237
+ | `model: provider/id` | pin (e.g. crash triage on local, deep review on Copilot) |
238
+ | `thinking: high` | `thinkingLevel`, clamped per model |
239
+ | `omit-system-prompt/docs/context-files` | section resolution at request prep |
240
+
241
+ Bundled `scout/planner/worker/reviewer` keep working on both. Project
242
+ workflow adds e.g. `.agents/agents/test-driver.md` (full tools, captures
243
+ logs, never analyzes) and `.agents/agents/crash-reviewer.md` (read-only
244
+ bash, never launches app, `thinking: high`, `omit-context-files: true`).
245
+ Orchestrator prompt is backend-agnostic:
246
+
247
+ > Ask test-driver to run plan A. Watch it. On non-zero exit, give its
248
+ > fieldnote + log tail to crash-reviewer. Return both.
249
+
250
+ ### Fieldnotes bridge (agent memory, not human report)
251
+
252
+ Today: `.shepherd/sessions/NNN-orchestrator-<id>/shepherd.md` index +
253
+ `<agent>-NN.md` notes. Audience is shepherd + sheep, not humans: shared
254
+ working memory so delegated work survives compaction, waiting, close,
255
+ timeout, and restart. That agent-truth role is what must be preserved on
256
+ durable.
257
+
258
+ Durable: introduce typed doc mirroring the same content:
259
+
260
+ ```ts
261
+ ShepherdNotes = defineDoc<{ index: string; notes: Record<string,string> }>({
262
+ kind: "shepherd.notes", scope: "conversation", history: "rewindable", fork: "asOf", initial: ...
263
+ })
264
+ ```
265
+
266
+ Write note + entry in one `commit()` (atomic). Mirror to files via `env`
267
+ for agent continuity and debugging. Doc is truth for `resume()`; files
268
+ are the agents' view. `fieldnotes` setting stays session-scoped (new session after
269
+ toggle). Keep `.gitignore` pattern:
270
+
271
+ ```gitignore
272
+ .shepherd/*
273
+ !.shepherd/config.json
274
+ ```
275
+
276
+ ### Settings
277
+
278
+ Extend `ShepherdSettings` with two fields, Herdr defaults preserved:
279
+
280
+ ```jsonc
281
+ // ~/.pi/agent/pi-shepherd/config.json
282
+ {
283
+ "agentScope": "user", "includeBundledAgents": true,
284
+ "keepOpen": true, "stayOpen": false, "fieldnotes": true,
285
+ "timeout": 20, "staleWaitThreshold": 5,
286
+ "backend": "herdr" // | "durable" | "auto"
287
+ }
288
+ // .shepherd/config.json adds when active:
289
+ // { ..., "durableStorage": "sqlite:./.shepherd/durable.sqlite", "projectScope": true }
290
+ ```
291
+
292
+ - `herdr` (default): current behavior, no durable code loaded.
293
+ - `durable`: all `shepherd_*` route to durable runtime backed by
294
+ `MemoryStorage` (tests) or `openNodeSqliteStorage(path)` /
295
+ `openNodeJsonlStorage(dir)` (service). WAL `synchronous=NORMAL` noted:
296
+ survives process crash, newest commit may be lost on power failure.
297
+ - `auto`: execute on durable, and if Herdr reachable, open read-only tail
298
+ panes fed by `watchEvents()` for visibility. Execution stays durable;
299
+ panes never write.
300
+
301
+ `/shepherd settings` gains one `backend` row + `durableStorage` row.
302
+ `confirmProjectAgents` handling unchanged (user-owned).
303
+
304
+ `done-nudge` (one reminder turn when child ends turn with open task,
305
+ `PI_SHEPHERD_DONE_NUDGES`) maps to durable `onYield` hook on
306
+ `GenerationTask`, same one-turn budget, then stall reported as normal
307
+ message, never invented status.
308
+
309
+ ## Overnight test-agent profile
310
+
311
+ Concrete settings for the motivating use case:
312
+
313
+ - One service process owns `./.shepherd/durable.sqlite`. `harness.resume()`
314
+ on boot. `harness.closed.then(reopen if failed)`.
315
+ - One conversation per overnight job, `requestId = jobId`,
316
+ `configure({ model: modelFor(user), cwd: /workspaces/<user>/<job> })`.
317
+ Free-local vs. Copilot resolved at configure time; tokens never land in
318
+ shared docs. Spend via `harness.usage()` per `provider/model` + tool.
319
+ - `settings.compaction { reserveTokens, keepRecentTokens, backgroundTokens }`
320
+ tuned for 8h logs; `progress { partialIntervalMs: 100 }` default, raise
321
+ to 500 for remote storage.
322
+ - Morning human report is out of shepherd scope: a separate templated HTML
323
+ file owned by the app, rendered by a reporter from the job's report doc
324
+ (e.g. `app.test-report`) + logs via `watchEvents()` snapshot + deltas.
325
+ It must never be `shepherd.md`, a doc snapshot dump, or `/export`.
326
+ Shepherd only guarantees the agent memory it was built from survived.
327
+ - Outer job file `jobId -> plan` only as boot re-submitter; `requestId`
328
+ dedup makes it safe.
329
+
330
+ ## Security and safety
331
+
332
+ - Keep: user discovery default, project agents trust-gated,
333
+ `close` only closes shepherd-created panes (durable: only aborts owned
334
+ conversations/tasks), raw pane IDs rejected, host-user tool permissions
335
+ documented.
336
+ - Durable additions: per-user `cwd`/`env` isolation (`env.id` per user or
337
+ container), `modelTools` least-privilege per sheep, `beforeTool` guard
338
+ keeping `crash-reviewer` bash read-only, secrets only in service process
339
+ (never in entries/docs/fieldnotes files committed to git).
340
+ - Storage: single-owner SQLite, no NFS multi-writer. Document
341
+ `prefix: "pi_"` namespacing if sharing a DB file with host tables.
342
+
343
+ ## Verification requirements
344
+
345
+ Implementation is complete only when:
346
+
347
+ - existing `verify-*.mjs` suite passes on `backend: herdr` (assertions
348
+ unchanged; Phase 1 only rewrites import specifiers);
349
+ - new `verify-durable-matrix.mjs` runs `delegate/task-completion/
350
+ task-failures/message-routing/multi-wait/watchers` against
351
+ `MemoryStorage + fauxProvider` on both backends with identical
352
+ orchestrator prompts (test-driver -> crash-reviewer fixture);
353
+ - opaque ID contract holds: Herdr pane IDs and durable conversation IDs
354
+ never accepted as `shepherd_*` IDs; `prune/status/read` never leak them;
355
+ - crash-resume test: kill mid `bash` on durable, `reopen + resume()`
356
+ yields `interrupted` (unsafe) or reattached result (safe-keyed), no
357
+ double test-run for same `requestId`;
358
+ - `expectsReply` waiting + `staleWaitThreshold` reminder + `timeout`
359
+ `blocked` verified on both backends, including cwd-differing agents;
360
+ - fieldnotes <-> doc mirror test: note written in same commit as entry,
361
+ file tail matches doc snapshot after restart; `fork: asOf` preserves
362
+ handoff;
363
+ - `done-nudge` budget (1 turn, `0` disables) verified via `onYield` on
364
+ durable;
365
+ - `npm test` passes; Herdr-only install pulls no durable dependency;
366
+ - live checks: `/shepherd herd/spawn/read/settings` on herdr;
367
+ overnight dry-run (submit evening, kill service, resume morning, report
368
+ renders) on durable.
369
+
370
+ ## Migration
371
+
372
+ No migration. Default `backend: herdr` is current behavior. Opt-in per
373
+ workspace via `/shepherd settings` or `.shepherd/config.json`
374
+ (`projectScope: true` for teams). Compatibility wrappers from spec 002
375
+ (`delegate = start+prompt+wait+close`) remain valid; they compose
376
+ primitives regardless of backend.
377
+
378
+ ## Open questions
379
+
380
+ 1. `placement` in durable: ignore vs. map to Herdr tail-viewer panes in
381
+ `auto` mode? Proposal: ignore for execution, honor for viewer.
382
+ 2. `read --lines` pagination over durable ops frames vs. full view?
383
+ Proposal: tail rendered entries, cap 100 undelivered frames like
384
+ `watch()` does.
385
+ 3. Compaction of `shepherd.notes` doc on very long overnight runs:
386
+ `rewindable` bounded history vs. summary entry? Needs bench
387
+ (`bench:storage`) before fixing a limit.
388
+ 4. Should `auto` tail panes be `stayOpen: false` viewers that never accept
389
+ input? Likely yes to avoid dual-write.
@@ -0,0 +1,199 @@
1
+ # Tasks: Dual-runtime Shepherd (Herdr + Durable)
2
+
3
+ Work in order. Phase 0 must stay green before Phase 1 starts. A task is done
4
+ only when its implementation + verification are complete. This spec is
5
+ `006-dual-runtime-herdr-durable` (`spec.md` contract, `intent.md` vision,
6
+ `plan.md` design).
7
+
8
+ ## Phase 0 — Extract 2-runtimes-1-core boundary (no behavior change)
9
+
10
+ - [x] **0.1 Add `src/runtimes/runtime.ts` interface**
11
+ - Define opaque `AgentHandle / TaskId / MessageId / Completion /
12
+ RuntimeStatus` shapes; no Herdr pane IDs, no durable conversation IDs
13
+ in the public contract.
14
+ - Document internal mapping (Herdr `paneId` OR durable
15
+ `{conversationId, submissionId, taskId}`) and diagnostic-only pane-ID
16
+ rule for `shepherd_read`.
17
+ - Interface file imports neither `herdr` CLI nor `pi-durable`; `src/core/`
18
+ keeps only shared domain modules (`discovery.ts`, `artifact-sessions.ts`,
19
+ `types.ts`, `ids.ts`, `config.ts`).
20
+
21
+ - [x] **0.2 Extract `src/core/ids.ts`**
22
+ - Move ID minting (`shepherd-<kind>-<session>-<uuid>`) and handle
23
+ parse/validation (`handleId`, `taskHandleId`) out of
24
+ `LifecycleRegistry` into a pure codec both runtimes call.
25
+ - `orchestration.ts` keeps handles and registries only; formats and
26
+ `invalid_handle` messages stay byte-identical.
27
+ - Unit test: id formats mint/parse/validate the same before and after.
28
+
29
+ - [x] **0.3 Move Herdr implementation behind it**
30
+ - Move current `herdr.ts + orchestration.ts + lifecycle.ts +
31
+ messaging.ts` rename-only into `src/runtimes/herdr/`; `lifecycle`/
32
+ `orchestration`/`herdr` implement the interface, `messaging.ts` is
33
+ support. Move the settings store `src/extension/config.ts` →
34
+ `src/core/config.ts` rename-only; `/shepherd settings` UI stays put.
35
+ - Route the lifecycle calls that map 1:1 to the lifecycle-only
36
+ interface (today: `delegate`, `close`) through it; leave call sites
37
+ whose results/callbacks the interface does not carry (spawn discovery
38
+ gate, prompt, message result shape, watch callbacks, status detail,
39
+ read source selection) on the concrete runtime with a comment, with
40
+ zero tool-schema, prompt, settings, widget, or fieldnotes-path change.
41
+ Discovery stays shared; herd/prune/widget keep runtime introspection
42
+ accessors.
43
+ - Leave temporary re-export shims at the old `src/core/*` and
44
+ `src/extension/config.ts` paths so the Phase 0 suite passes untouched;
45
+ 1.5 removes them.
46
+ - Preserve: `created-panes.json` ownership, `--no-focus`, temp-file
47
+ lifetime, session registries, `shepherd_done` 1-nudge
48
+ (`PI_SHEPHERD_DONE_NUDGES`), `confirmProjectAgents` user-owned.
49
+
50
+ - [x] **0.4 Verify boundary**
51
+ - Full existing `verify-*.mjs` suite passes untouched.
52
+ - After 0.3, `src/` and `index.ts` import runtime paths, not shims; old
53
+ paths survive only as shims for the test suite until 1.5.
54
+ - Live: idle spawn, non-blocking prompt, single + multi-wait, status,
55
+ close, ownership refusal, project-agent gate.
56
+ - Herdr-only install unchanged (no new dep evaluated).
57
+
58
+ ## Phase 1 — Durable spawn/delegate/watch/done
59
+
60
+ - [ ] **1.1 Scaffold `src/runtimes/durable/` adapter**
61
+ - Add `@earendil-works/pi-durable` + `@earendil-works/chord` as optional
62
+ peerDeps; Herdr-only path never imports them.
63
+ - Add `src/runtimes/durable/runtime.ts` implementing `runtime.ts` over
64
+ `Harness`.
65
+ - Add `src/runtimes/durable/extension.ts`:
66
+ `defineExtension({name:"shepherd"})` with shepherd tools + docs
67
+ (`ShepherdHerdDoc, ShepherdTasksDoc`).
68
+
69
+ - [ ] **1.2 Implement sheep-as-conversation**
70
+ - `spawn` -> `tx.createConversation({ownership:{kind:"task",taskId}})` +
71
+ `configure({model, thinkingLevel, tools, modelTools, cwd,
72
+ instructions})` from agent frontmatter; fresh `pi.provider` identity.
73
+ - `model:null/default` inherits Shepherd session model (per-user default);
74
+ pinned model + clamped `thinking` honored; `placement` ignored for
75
+ execution; `cwd` honored with per-user `env.id` isolation.
76
+ - `delegate` -> `submit({type:"input", content, requestId})`, defaulting
77
+ `requestId` to shepherd task ID; retries dedup, never double-run.
78
+ - `done` (child) completes owner task; `watch` (parent) =
79
+ `submission.wait() / waitForTask()` with steered delivery (mid-turn
80
+ inject, idle new turn).
81
+ - Wire `Harness.open(MemoryStorage | sqlite | jsonl)`, `resume()` on boot,
82
+ `closed.then(reopen if failed)`; pass through
83
+ `compaction/progress/retry/toolExecution`.
84
+
85
+ - [ ] **1.3 Replay policy**
86
+ - App-driving tools default unsafe: crash -> `interrupted` + partials,
87
+ never silent rerun.
88
+ - Only idempotent readers may use `replay:"safe"` with keyed
89
+ `executeTool` reattach; add fixture proving no double side effect for
90
+ same `requestId`.
91
+
92
+ - [ ] **1.4 Verify foreground loop**
93
+ - New `test/verify-durable-matrix.mjs` runs test-driver ->
94
+ crash-reviewer fixture on herdr AND durable (`MemoryStorage +
95
+ fauxProvider`) with identical orchestrator prompts.
96
+ - Asserts same handoff, opaque-ID rejection (pane + conversation IDs
97
+ refused), `MemoryStorage` resume after restart.
98
+
99
+ - [ ] **1.5 Update import/path references + drop compat shims**
100
+ - Rewrite imports in `test/verify-*.mjs` and root `index.ts` from
101
+ `src/core/{herdr,orchestration,lifecycle,messaging}.ts` and
102
+ `src/extension/config.ts` to `src/runtimes/herdr/*` and
103
+ `src/core/config.ts`; assertions stay unchanged.
104
+ - Update non-import path strings (e.g. `test/verify-status-widget.mjs`
105
+ module-path assertions) and the `index.ts` header comment describing
106
+ `src/core/` as the engine.
107
+ - Delete the Phase 0 re-export shims; no old-path survivors remain.
108
+ - Refresh layout and semantic-center references in `AGENTS.md` and
109
+ `docs/plans/architecture.md` (stale `src/core` tree) to the new tree.
110
+ - `npm test` green with shims gone and Herdr-only install still clean.
111
+
112
+ ## Phase 2 — Message/status/read/close parity
113
+
114
+ - [ ] **2.1 Implement durable messaging**
115
+ - Passive -> `submit({type:"write"})` (immediate even mid-run, except
116
+ behind queued reset); active -> steer/follow-up into `pi.inbox`.
117
+ - `expectsReply:true` -> durable `waiting`; `staleWaitThreshold` single
118
+ info reminder; `timeout` abort -> `blocked`; reply/`resume`/complete
119
+ clears episode. Cover cwd-differing agents.
120
+
121
+ - [ ] **2.2 Implement durable status/read/close**
122
+ - `status` from `viewState() + taskGraph()` with
123
+ `proc {idle|working|unknown} x task {running|waiting|done|blocked|
124
+ failed}` + `waitingOn + stale`; keep idle-but-waiting visible.
125
+ - `read` tails committed entries + `pi.live` partials/output (100-frame
126
+ collapse rule); never leaks native IDs.
127
+ - `close/abort` bottom-up, withdraw queued inputs (keep queued writes);
128
+ `background:true` anchors implement `stayOpen` persistent workers +
129
+ reporter follow-up pattern.
130
+
131
+ - [ ] **2.3 Map done-nudge**
132
+ - Implement 1-turn reminder as durable `onYield` on `GenerationTask`
133
+ (`0` disables); exhausted budget escalates as ordinary message, never
134
+ invents terminal status. Verify on both backends.
135
+
136
+ - [ ] **2.4 Verify parity**
137
+ - Extend matrix: `message-routing / task-failures / multi-wait /
138
+ watchers / stale-wait` green on both; kill-mid-bash test yields
139
+ `interrupted` (unsafe) or reattached (safe-keyed).
140
+
141
+ ## Phase 3 — Agent memory bridge + settings + viewer
142
+
143
+ - [ ] **3.1 Fieldnotes <-> doc mirror (agent memory)**
144
+ - Add `ShepherdNotes = defineDoc({kind:"shepherd.notes",
145
+ scope:"conversation", history:"rewindable", fork:"asOf"})`.
146
+ - Note + entry committed atomically; file mirror via `env` for agent
147
+ continuity/debugging; `fieldnotes` stays session-scoped.
148
+ - Document explicitly: `shepherd.md` is shepherd/sheep working memory
149
+ surviving compaction — never the human morning report.
150
+ - Test: file tail == doc snapshot after restart; `fork:asOf` handoff.
151
+
152
+ - [ ] **3.2 Settings + backend switch**
153
+ - Add `backend: "herdr"|"durable"|"auto"` (default `herdr`) +
154
+ `durableStorage` (`sqlite:path | jsonl:dir | memory`) to user +
155
+ active-project (`.shepherd/config.json`, `projectScope:true`) layers.
156
+ - `/shepherd settings` two new rows; project file cannot disable
157
+ `confirmProjectAgents` nor force backend for others.
158
+ - Keep `.gitignore` (` .shepherd/* + !config.json`) — sessions ignored,
159
+ config committable.
160
+
161
+ - [ ] **3.3 Auto read-only viewer (optional but spec'd)**
162
+ - In `auto`, execution stays durable; if Herdr reachable, open
163
+ input-refusing tail panes fed by `watchEvents()` snapshot + deltas.
164
+ - `placement` honored for viewer only; viewers are `stayOpen:false`.
165
+ - Test no dual-write: viewer input rejected, kill-viewer never aborts job.
166
+
167
+ - [ ] **3.4 Reporter contract (docs only)**
168
+ - Document app-owned HTML path: reporter reads job report doc
169
+ (e.g. `app.test-report`) + logs, renders template. Shepherd implements
170
+ no HTML, dumps no doc snapshot as report, claims no `/export` as
171
+ report. Guarantee is only that agent memory survived.
172
+
173
+ ## Phase 4 — Docs, tests, live proof
174
+
175
+ - [ ] **4.1 Update docs**
176
+ - README: Herdr hero first, one backend section (durable experimental,
177
+ `requestId`, single-owner SQLite, no multi-writer).
178
+ - Agent-definition guide: frontmatter dual-mapping table.
179
+ - Diagnostics: `extract:system-prompt` for both adapters.
180
+
181
+ - [ ] **4.2 Full test gate**
182
+ - `npm test` green; Herdr-only install pulls no durable dep.
183
+ - Matrix + mirror + nudge + timeout/stale tests green on memory.
184
+ - SQLite crash-resume + `prefix:"pi_"` sharing note covered.
185
+
186
+ - [ ] **4.3 Live verification**
187
+ - Herdr: `/shepherd herd/spawn/read/settings`, widget, ownership refusal.
188
+ - Durable dry-run: submit evening -> kill service mid-run -> reopen +
189
+ `resume()` morning -> agent memory intact -> app HTML renders from job
190
+ doc (proving shepherd/report separation).
191
+
192
+ ## Completion checklist
193
+
194
+ - [ ] Phases 0-3 implemented with 1 core + 2 runtimes, no surface fork.
195
+ - [ ] Herdr default behavior bit-identical; durable opt-in experimental.
196
+ - [ ] `shepherd.md` role preserved as agent memory; human HTML separate.
197
+ - [ ] Per-user models, `requestId` dedup, usage accounting work overnight.
198
+ - [ ] No pane-ID / conversation-ID leaks; ownership + trust intact.
199
+ - [ ] Docs, settings UI, tests, live proofs all green.