pi-worker-graph 0.1.0-dev.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +455 -0
- package/SECURITY.md +54 -0
- package/dist/config.d.ts +25 -0
- package/dist/config.d.ts.map +1 -0
- package/dist/config.js +181 -0
- package/dist/config.js.map +1 -0
- package/dist/context.d.ts +22 -0
- package/dist/context.d.ts.map +1 -0
- package/dist/context.js +81 -0
- package/dist/context.js.map +1 -0
- package/dist/coordination.d.ts +19 -0
- package/dist/coordination.d.ts.map +1 -0
- package/dist/coordination.js +267 -0
- package/dist/coordination.js.map +1 -0
- package/dist/execution-failure.d.ts +42 -0
- package/dist/execution-failure.d.ts.map +1 -0
- package/dist/execution-failure.js +90 -0
- package/dist/execution-failure.js.map +1 -0
- package/dist/extension.d.ts +8 -0
- package/dist/extension.d.ts.map +1 -0
- package/dist/extension.js +641 -0
- package/dist/extension.js.map +1 -0
- package/dist/graph.d.ts +44 -0
- package/dist/graph.d.ts.map +1 -0
- package/dist/graph.js +292 -0
- package/dist/graph.js.map +1 -0
- package/dist/index.d.ts +16 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +8 -0
- package/dist/index.js.map +1 -0
- package/dist/json.d.ts +9 -0
- package/dist/json.d.ts.map +1 -0
- package/dist/json.js +66 -0
- package/dist/json.js.map +1 -0
- package/dist/orchestrator.d.ts +15 -0
- package/dist/orchestrator.d.ts.map +1 -0
- package/dist/orchestrator.js +473 -0
- package/dist/orchestrator.js.map +1 -0
- package/dist/output.d.ts +61 -0
- package/dist/output.d.ts.map +1 -0
- package/dist/output.js +248 -0
- package/dist/output.js.map +1 -0
- package/dist/pi-subprocess.d.ts +92 -0
- package/dist/pi-subprocess.d.ts.map +1 -0
- package/dist/pi-subprocess.js +897 -0
- package/dist/pi-subprocess.js.map +1 -0
- package/dist/run.d.ts +89 -0
- package/dist/run.d.ts.map +1 -0
- package/dist/run.js +562 -0
- package/dist/run.js.map +1 -0
- package/dist/store.d.ts +331 -0
- package/dist/store.d.ts.map +1 -0
- package/dist/store.js +1993 -0
- package/dist/store.js.map +1 -0
- package/dist/usage.d.ts +35 -0
- package/dist/usage.d.ts.map +1 -0
- package/dist/usage.js +88 -0
- package/dist/usage.js.map +1 -0
- package/docs/DECISIONS.md +221 -0
- package/docs/DESIGN.md +392 -0
- package/docs/NEXT.md +229 -0
- package/docs/PLAN.md +203 -0
- package/docs/worker-graph.example.json +12 -0
- package/extensions/index.ts +1 -0
- package/extensions/tsconfig.json +11 -0
- package/package.json +66 -0
package/docs/DESIGN.md
ADDED
|
@@ -0,0 +1,392 @@
|
|
|
1
|
+
# Design
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
`pi-worker-graph` lets one Pi orchestrator execute a dependency graph of writable
|
|
6
|
+
workers against one shared checkout. It minimizes context sharing: immutable edge
|
|
7
|
+
outputs carry planned dependencies, while a small journal carries unexpected
|
|
8
|
+
coordination facts.
|
|
9
|
+
|
|
10
|
+
The graph domain is implemented independently of Pi. Process execution,
|
|
11
|
+
persistence, tools, and user interface behavior are later integration layers.
|
|
12
|
+
|
|
13
|
+
## Runtime roles
|
|
14
|
+
|
|
15
|
+
### Orchestrator
|
|
16
|
+
|
|
17
|
+
The parent Pi session:
|
|
18
|
+
|
|
19
|
+
- understands the task and repository constraints;
|
|
20
|
+
- defines nodes and dependency edges;
|
|
21
|
+
- decides which ready nodes may safely overlap;
|
|
22
|
+
- receives graph progress and structured results;
|
|
23
|
+
- reviews the integrated checkout with read-only tools;
|
|
24
|
+
- commissions final validation as dependent tasks and weighs what they report,
|
|
25
|
+
because the mode suppresses the parent's own command execution;
|
|
26
|
+
- delegates repairs rather than editing while worker-graph mode is active.
|
|
27
|
+
|
|
28
|
+
### Worker
|
|
29
|
+
|
|
30
|
+
A child Pi session:
|
|
31
|
+
|
|
32
|
+
- receives one bounded assignment;
|
|
33
|
+
- operates directly in the shared checkout;
|
|
34
|
+
- receives outputs only from its direct prerequisites;
|
|
35
|
+
- may publish run-scoped coordination events or messages;
|
|
36
|
+
- reports a structured result;
|
|
37
|
+
- cannot spawn another worker.
|
|
38
|
+
|
|
39
|
+
## Graph domain
|
|
40
|
+
|
|
41
|
+
The public graph module accepts arbitrary payloads:
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
interface GraphTask<TPayload = unknown> {
|
|
45
|
+
id: string;
|
|
46
|
+
needs?: readonly string[];
|
|
47
|
+
payload?: TPayload;
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
interface GraphRequest<TPayload = unknown> {
|
|
51
|
+
tasks: readonly GraphTask<TPayload>[];
|
|
52
|
+
concurrency?: number;
|
|
53
|
+
}
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Normalization trims IDs, deduplicates and sorts dependency lists, and preserves
|
|
57
|
+
task order and payload identity. Validation completes before execution:
|
|
58
|
+
|
|
59
|
+
- IDs are non-empty and unique after normalization.
|
|
60
|
+
- Every dependency names a node in the request.
|
|
61
|
+
- A node cannot depend on itself.
|
|
62
|
+
- The graph is acyclic.
|
|
63
|
+
- Concurrency, when provided, is a positive integer.
|
|
64
|
+
|
|
65
|
+
Executor-specific schemas add assignment text, working directory policy, expected
|
|
66
|
+
paths, acceptance criteria, profiles, and hard resource limits. The runtime must
|
|
67
|
+
apply graph-size and dependency-count limits before invoking the graph domain.
|
|
68
|
+
|
|
69
|
+
## Node states and scheduling
|
|
70
|
+
|
|
71
|
+
Node states are:
|
|
72
|
+
|
|
73
|
+
```text
|
|
74
|
+
pending -> running -> succeeded
|
|
75
|
+
| -> failed
|
|
76
|
+
| -> aborted
|
|
77
|
+
-> failed
|
|
78
|
+
-> aborted
|
|
79
|
+
-> blocked
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Terminal states do not transition. A task can fail or abort before reaching
|
|
83
|
+
`running` when startup fails or cancellation arrives before execution.
|
|
84
|
+
|
|
85
|
+
Scheduling uses deterministic topological frontiers:
|
|
86
|
+
|
|
87
|
+
1. Find pending nodes whose direct prerequisites all succeeded.
|
|
88
|
+
2. Start eligible nodes up to the configured concurrency limit.
|
|
89
|
+
3. Persist each terminal result.
|
|
90
|
+
4. Mark pending nodes blocked when any prerequisite failed, aborted, or was
|
|
91
|
+
blocked.
|
|
92
|
+
5. Recompute the ready frontier.
|
|
93
|
+
6. Finish when every node is succeeded, failed, aborted, or blocked.
|
|
94
|
+
|
|
95
|
+
A graph with no edges is an ordinary parallel batch. A linear graph is a chain.
|
|
96
|
+
The pure graph module returns every eligible node in a deterministic order; the
|
|
97
|
+
executor applies the configured concurrency limit when selecting work to start.
|
|
98
|
+
The graph module does not spawn workers.
|
|
99
|
+
|
|
100
|
+
## Edge context
|
|
101
|
+
|
|
102
|
+
Each worker reports a compact result:
|
|
103
|
+
|
|
104
|
+
```ts
|
|
105
|
+
interface NodeOutput {
|
|
106
|
+
schemaVersion: 1;
|
|
107
|
+
summary: string;
|
|
108
|
+
changedFiles: Array<{ path: string; description: string }>;
|
|
109
|
+
interfaces: string[];
|
|
110
|
+
decisions: string[];
|
|
111
|
+
validation: Array<{ command: string; result: string }>;
|
|
112
|
+
blockers: string[];
|
|
113
|
+
}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
The report schema is versioned independently of the enclosing filesystem record.
|
|
117
|
+
Reports require exactly these fields, use non-empty bounded strings, limit each
|
|
118
|
+
section's item count, and have a hard aggregate UTF-8 JSON size limit.
|
|
119
|
+
Changed-file paths use normalized repository-relative form; they are report data,
|
|
120
|
+
not authority to access a path. Untyped executor values are validated and copied
|
|
121
|
+
into immutable snapshots before publication. Malformed or oversized reports fail
|
|
122
|
+
the node. A valid report with one or more blockers also fails the node, but is
|
|
123
|
+
retained for parent review; its dependents do not run.
|
|
124
|
+
|
|
125
|
+
For a node, outputs from each task named directly in `needs` are serialized in
|
|
126
|
+
task-ID order into canonical named JSON blocks for the execution adapter to
|
|
127
|
+
prepend to its assignment. Worker-authored text is explicitly marked as
|
|
128
|
+
untrusted report data within those blocks. Full transcripts and undeclared report
|
|
129
|
+
fields are not propagated. The complete serialization, including block labels
|
|
130
|
+
and the warning, is measured in UTF-8 bytes against a hard limit. Overflow fails
|
|
131
|
+
the downstream node without truncation, and stays that way: truncating it would
|
|
132
|
+
run a dependent worker against an incomplete prerequisite contract, and a
|
|
133
|
+
retained artifact cannot repair that because artifacts never cross a dependency
|
|
134
|
+
edge (D19).
|
|
135
|
+
|
|
136
|
+
Persisted output records use their own envelope schema version, independently of
|
|
137
|
+
the worker-report schema version. Current envelopes include run and task identity,
|
|
138
|
+
attempt, completion time, status, bounded diagnostics, a reference to a
|
|
139
|
+
retained text artifact when there is one, and what the attempt spent.
|
|
140
|
+
|
|
141
|
+
Usage is recorded for every terminal status, aborted included: an attempt the
|
|
142
|
+
run stopped had still spent what it spent by then. A timeout and an abort are
|
|
143
|
+
decided by the runner, which discards whatever the executor eventually settles
|
|
144
|
+
with, so usage the executor reported is carried onto the runner's own outcome
|
|
145
|
+
rather than lost with it. Failures carry usage too, on
|
|
146
|
+
`TaskExecutionFailure`. An executor that does not account for its own spend
|
|
147
|
+
reports nothing rather than zeros, which would read as work that cost nothing.
|
|
148
|
+
|
|
149
|
+
Zero spend and unknown spend are different answers. An executor whose
|
|
150
|
+
telemetry was absent, or carried a field that was not a usable number, records
|
|
151
|
+
no usage rather than zeros: a complete-looking zero would read as work that
|
|
152
|
+
cost nothing. An absent optional field is still zero, because a provider that
|
|
153
|
+
bills no cache write reports none.
|
|
154
|
+
|
|
155
|
+
A run's total is derived from the outputs it published rather than stored
|
|
156
|
+
alongside them: derived is correct for a run that was interrupted, and needs no
|
|
157
|
+
summary kept in step. Reading it applies the same output-to-node-state
|
|
158
|
+
agreement every other reader requires, so a total is never assembled from
|
|
159
|
+
records the rest of the store refuses; only a task that published no output at
|
|
160
|
+
all is an expected absence. Such a task is reported as unaccounted when it ran,
|
|
161
|
+
and as not started when it did not — the first is a gap in the accounting, the
|
|
162
|
+
second is not.
|
|
163
|
+
|
|
164
|
+
Token counts come from the provider's telemetry and cost from the runtime's
|
|
165
|
+
pricing of those tokens, so cost is an estimate and tokens are the sturdier
|
|
166
|
+
number.
|
|
167
|
+
|
|
168
|
+
A retained text artifact is bounded long-form text that does not belong in the
|
|
169
|
+
structured report — a log, investigation notes, detailed review findings. A
|
|
170
|
+
worker publishes one deliberately, through an optional `artifact` field on the
|
|
171
|
+
final-report tool; it is never runtime spillover from a report that did not
|
|
172
|
+
fit. The runtime never parses it and never propagates it over a dependency
|
|
173
|
+
edge, so a report that leans on its artifact is an incomplete report. It is published under the same mutation lock as the output
|
|
174
|
+
that references it, and before that output, so an interruption can only strand
|
|
175
|
+
an artifact no output claims. An artifact is therefore readable only through
|
|
176
|
+
its reference, which names the byte length the artifact must still have. An
|
|
177
|
+
aborted task produced nothing to retain and may not publish one. Artifacts are
|
|
178
|
+
records with the same identity envelope as every other stored record, rather
|
|
179
|
+
than bare text files, so an artifact cannot be read as belonging to a run, task,
|
|
180
|
+
or attempt other than the one that wrote it. They are removed with their run.
|
|
181
|
+
|
|
182
|
+
## Coordination journal
|
|
183
|
+
|
|
184
|
+
Journal events are for information discovered after scheduling:
|
|
185
|
+
|
|
186
|
+
```ts
|
|
187
|
+
type EventKind =
|
|
188
|
+
| "decision"
|
|
189
|
+
| "interface"
|
|
190
|
+
| "risk"
|
|
191
|
+
| "conflict"
|
|
192
|
+
| "handoff"
|
|
193
|
+
| "progress";
|
|
194
|
+
|
|
195
|
+
interface RunEvent {
|
|
196
|
+
schemaVersion: 1;
|
|
197
|
+
kind: "run-event";
|
|
198
|
+
eventId: string;
|
|
199
|
+
runId: string;
|
|
200
|
+
taskId: string;
|
|
201
|
+
timestamp: string;
|
|
202
|
+
eventKind: EventKind;
|
|
203
|
+
message: string;
|
|
204
|
+
paths?: string[];
|
|
205
|
+
symbols?: string[];
|
|
206
|
+
recipients?: string[];
|
|
207
|
+
}
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Directed messages carry the same envelope with a sender and one recipient task.
|
|
211
|
+
|
|
212
|
+
Workers explicitly query relevant entries by recipient, path, symbol, or cursor.
|
|
213
|
+
The runtime does not inject the entire journal into every turn. Live steering is
|
|
214
|
+
outside the MVP.
|
|
215
|
+
|
|
216
|
+
Every record in a run shares one monotonic sequence, so an identifier is also a
|
|
217
|
+
position: a cursor names a point that no later record can precede, and a worker
|
|
218
|
+
polling with one cannot silently skip a record published between two reads. A
|
|
219
|
+
publisher holds the run mutation lock while checking the active owner and
|
|
220
|
+
linking its record, so ownership release cannot land between validation and
|
|
221
|
+
publication. Within the critical section, the record itself is exclusively
|
|
222
|
+
created under the first free identifier; an interrupted publisher leaves no
|
|
223
|
+
identifier behind.
|
|
224
|
+
|
|
225
|
+
The orchestrator and every worker of a run contend for that one lock, so the
|
|
226
|
+
parent has to tell a lock a live worker holds from one a worker was killed
|
|
227
|
+
while holding. It is not a judgement about elapsed time: the lock is claimed by
|
|
228
|
+
linking a record that is already complete on disk, so it never exists without
|
|
229
|
+
naming its holder, and contention is reported apart from an ownership conflict.
|
|
230
|
+
A parent mutation that meets a lock waits it out, and recovers it only when the
|
|
231
|
+
lock names a task of its own graph whose promise has already settled. Its own
|
|
232
|
+
task the graph runner knows to be finished; a sibling still running is waited
|
|
233
|
+
for rather than interrupted, so ordinary contention no longer costs a graph.
|
|
234
|
+
The runner recovers whatever remains before releasing ownership, once every
|
|
235
|
+
task has settled.
|
|
236
|
+
|
|
237
|
+
Coordination is bounded on both sides. A run retains a fixed maximum number of
|
|
238
|
+
records, and publishing past it fails explicitly instead of silently degrading
|
|
239
|
+
reads. One record is bounded as a whole when it is published, at half a page, so
|
|
240
|
+
field bounds cannot combine into a record larger than a page. A read is bounded
|
|
241
|
+
by a requested record count and by the serialized JSON array size, including its
|
|
242
|
+
brackets and separators, and reports a cursor for the remainder, so no record
|
|
243
|
+
can enlarge a worker's context beyond the page bound.
|
|
244
|
+
|
|
245
|
+
One journal holds both kinds, so a read passes over records it will never
|
|
246
|
+
return: the other kind, and another task's mail. Its cursor runs past those as
|
|
247
|
+
well as past what it delivered, so a polling worker pays for each record once
|
|
248
|
+
rather than re-reading the journal on every call. Only a record examined and
|
|
249
|
+
held back — the one that overflowed the page — stays ahead of the cursor. A
|
|
250
|
+
cursor consequently belongs to the query that produced it; reused under a
|
|
251
|
+
different filter or recipient it starts past records that filter would have
|
|
252
|
+
matched.
|
|
253
|
+
|
|
254
|
+
Publishing requires the run to have an active owner, but not the owner's
|
|
255
|
+
capability. Workers are the authors of coordination records and never hold the
|
|
256
|
+
capability that advances node state, so an unowned or finished run stays
|
|
257
|
+
immutable while worker writes remain unprivileged. Records are attributed to a
|
|
258
|
+
task rather than authenticated: every worker of a run shares this state root, as
|
|
259
|
+
they already share the checkout.
|
|
260
|
+
|
|
261
|
+
## Persistence
|
|
262
|
+
|
|
263
|
+
The state root is resolved from runtime configuration and defaults beneath Pi's
|
|
264
|
+
agent data directory. It never defaults inside the target checkout. A bounded
|
|
265
|
+
run count is enforced through atomically created run reservations. Reaching the
|
|
266
|
+
limit fails before a worker starts and never deletes prior state implicitly.
|
|
267
|
+
|
|
268
|
+
Per-run layout:
|
|
269
|
+
|
|
270
|
+
```text
|
|
271
|
+
<state-root>/runs/<run-id>/
|
|
272
|
+
run.json immutable graph and configuration
|
|
273
|
+
owner.json exclusive active lifecycle owner
|
|
274
|
+
nodes/<task-key>.json parent-owned current node state
|
|
275
|
+
outputs/<task-key>.json terminal structured output
|
|
276
|
+
artifacts/<task-key>.json bounded retained text for one task attempt
|
|
277
|
+
coordination/<id>.json immutable events and directed messages, in one
|
|
278
|
+
run-global sequence
|
|
279
|
+
sessions/<task-key>/... optional child session state
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
Task IDs are validated or encoded before use as path components. Immutable
|
|
283
|
+
records are published by writing a same-directory temporary file with restrictive
|
|
284
|
+
permissions and atomically renaming it. Mutable parent-owned state is replaced
|
|
285
|
+
atomically. One extension instance permits only one graph lifecycle at a time.
|
|
286
|
+
Active graph execution claims an exclusive owner record before mutating node
|
|
287
|
+
state or outputs. Each mutation also holds a per-run filesystem lock through its
|
|
288
|
+
full asynchronous read/validate/commit sequence, so ownership cannot be
|
|
289
|
+
released and reassigned mid-mutation. Resumable or externally addressable runs
|
|
290
|
+
still require an API that acquires and verifies that ownership before advancing
|
|
291
|
+
an existing run.
|
|
292
|
+
|
|
293
|
+
## Child process contract
|
|
294
|
+
|
|
295
|
+
Every worker receives explicit run and task identity plus the state directory,
|
|
296
|
+
and no other runtime state; in particular it never receives the run's ownership
|
|
297
|
+
capability. The child detects worker mode and registers coordination and
|
|
298
|
+
reporting tools, but not graph-spawning tools or orchestrator commands. Pi's
|
|
299
|
+
tool allowlist is strict over built-in and extension tools, so the adapter
|
|
300
|
+
allowlists the coordination tools on exactly the condition under which the child
|
|
301
|
+
registers them, and advertises them in the assignment only then. Secrets are not
|
|
302
|
+
added to child arguments, prompts, environment metadata persisted by the
|
|
303
|
+
runtime, or graph state.
|
|
304
|
+
|
|
305
|
+
The MVP transport is a one-shot Pi JSON-mode subprocess behind the execution
|
|
306
|
+
adapter. It runs without session persistence or discovered extensions, skills,
|
|
307
|
+
and prompt templates; only an explicit built-in tool allowlist and the child
|
|
308
|
+
report extension are active. Repository context files remain enabled as trusted
|
|
309
|
+
worker instructions. Task content is sent over stdin rather than argv.
|
|
310
|
+
The adapter owns process-group cancellation, forced cleanup, event-stream
|
|
311
|
+
framing bounds, and bounded exit classification. It projects capped progress
|
|
312
|
+
snapshots containing only task identity, phase, allowlisted tool names, and
|
|
313
|
+
numeric usage. Worker text and tool payloads are not forwarded to the parent.
|
|
314
|
+
|
|
315
|
+
Classification favours completed work over incidental process noise. A captured,
|
|
316
|
+
validated report outranks a provider error or a nonzero exit reported after it,
|
|
317
|
+
because the report is the task contract and the child is already finished by
|
|
318
|
+
then. Cancellation and a distrusted event stream still outrank a report, since
|
|
319
|
+
neither leaves it trustworthy. Bounds that protect the parent are framing bounds:
|
|
320
|
+
an unparseable line is skipped, never fatal, so a worker that legitimately emits
|
|
321
|
+
a large transcript is not failed after its edits have landed.
|
|
322
|
+
|
|
323
|
+
A report only counts when it was genuinely the worker's last action. Pi batches
|
|
324
|
+
the tool calls of one assistant message and honours a terminating result only
|
|
325
|
+
when the whole batch terminates, so the adapter reconstructs the batch the
|
|
326
|
+
report belonged to from the assistant `message_end` event and watches for tool
|
|
327
|
+
executions after it. A report that shared its batch, or that work outlived, is
|
|
328
|
+
rejected rather than reported as success.
|
|
329
|
+
|
|
330
|
+
The worker executable is identified positively rather than inferred. The adapter
|
|
331
|
+
resolves Pi's CLI entry point through this package's dependency on Pi and runs it
|
|
332
|
+
with the current JavaScript runtime, so there is no `PATH` search and no command
|
|
333
|
+
interpreter on any platform. `PI_CODING_AGENT` is not evidence of identity — Pi
|
|
334
|
+
exports it into every process it starts, including programs run by its own `bash`
|
|
335
|
+
tool — so it only corroborates the single-file-build case, where the running
|
|
336
|
+
executable is neither `node` nor `bun`. When neither route identifies Pi, the
|
|
337
|
+
adapter requires an explicit command instead of guessing through a shell.
|
|
338
|
+
|
|
339
|
+
## Mode lifecycle
|
|
340
|
+
|
|
341
|
+
The extension provides:
|
|
342
|
+
|
|
343
|
+
```text
|
|
344
|
+
/swarm on
|
|
345
|
+
/swarm status
|
|
346
|
+
/swarm off
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
and an opt-in startup flag. The parent `worker_graph` tool is registered once but
|
|
350
|
+
removed from the active tool set until the mode is enabled. Worker children take
|
|
351
|
+
a mutually exclusive extension path and receive only their final-report tool.
|
|
352
|
+
|
|
353
|
+
Entering the mode snapshots the complete active-tool set, removes the built-in
|
|
354
|
+
`bash`, `edit`, and `write` tools, and activates `worker_graph` in their place.
|
|
355
|
+
The snapshot and enabled state are stored in a bounded custom Pi session entry,
|
|
356
|
+
restored on resume and tree navigation, and restored exactly when the mode ends.
|
|
357
|
+
Worker children never register this lifecycle.
|
|
358
|
+
|
|
359
|
+
## Failure semantics
|
|
360
|
+
|
|
361
|
+
- Invalid graph: reject before spawning.
|
|
362
|
+
- Worker startup, provider, or tool failure: mark the node failed with bounded
|
|
363
|
+
diagnostics.
|
|
364
|
+
- Failed or aborted prerequisite: block dependents without running them.
|
|
365
|
+
- Abort: terminate running children and settle remaining nodes as aborted or
|
|
366
|
+
blocked.
|
|
367
|
+
- Journal failure: report it; never silently claim coordination succeeded.
|
|
368
|
+
- Worker-report overflow: reject the report so the worker can correct and
|
|
369
|
+
resubmit it; compact parent review projections mark every truncated field or
|
|
370
|
+
omitted report explicitly. Truncating a report is the runtime choosing which
|
|
371
|
+
structured fields to cut, so it does not do that (D19).
|
|
372
|
+
- Process crash: persisted terminal nodes remain terminal; running nodes become
|
|
373
|
+
interrupted and require an explicit retry or recovery decision.
|
|
374
|
+
|
|
375
|
+
`interrupted` is recovery metadata rather than a normal graph-domain terminal
|
|
376
|
+
state until retry semantics are defined.
|
|
377
|
+
|
|
378
|
+
## Same-tree concurrency
|
|
379
|
+
|
|
380
|
+
The runtime does not claim to make simultaneous writes safe. It reduces risk
|
|
381
|
+
through:
|
|
382
|
+
|
|
383
|
+
- careful graph decomposition;
|
|
384
|
+
- optional advisory path and symbol claims;
|
|
385
|
+
- instructions to re-read files before editing;
|
|
386
|
+
- preference for exact edits over whole-file rewrites;
|
|
387
|
+
- avoiding broad formatter or generator operations during a parallel frontier;
|
|
388
|
+
- orchestrator review of the integrated checkout;
|
|
389
|
+
- deterministic repository checks before acceptance.
|
|
390
|
+
|
|
391
|
+
Cross-process file locking is not part of the initial design. It would introduce
|
|
392
|
+
hidden serialization without protecting semantic coupling across different files.
|
package/docs/NEXT.md
ADDED
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
# Current development status
|
|
2
|
+
|
|
3
|
+
## Implemented
|
|
4
|
+
|
|
5
|
+
The repository now has a small transport-independent core plus an initial Pi
|
|
6
|
+
adapter. Automated tests remain provider-free:
|
|
7
|
+
|
|
8
|
+
- normalized, fully validated DAGs with frozen graph structure;
|
|
9
|
+
- deterministic frontiers and guarded node transitions;
|
|
10
|
+
- failed and aborted dependency blocking;
|
|
11
|
+
- immutable versioned run manifests with opaque task storage keys;
|
|
12
|
+
- one bounded retained text artifact per task attempt, published deliberately
|
|
13
|
+
through an optional `worker_graph_report` field, revalidated at every
|
|
14
|
+
boundary it crosses, published before and vouched for by the output that
|
|
15
|
+
references it, kept off dependency edges, named by byte length in the
|
|
16
|
+
parent-facing result, and removed with its run;
|
|
17
|
+
- report and prerequisite-context overflow that stay fail-closed rather than
|
|
18
|
+
truncating into an artifact;
|
|
19
|
+
- per-attempt token and cost accounting persisted with the attempt, kept for a
|
|
20
|
+
failed, timed-out, or aborted attempt as well as a succeeded one, summed per
|
|
21
|
+
run through `/swarm usage` and `readRunUsage()`, and attributed per task in
|
|
22
|
+
the parent-facing result;
|
|
23
|
+
- accounting that keeps zero and unknown apart: absent or unusable telemetry
|
|
24
|
+
leaves an attempt unaccounted rather than free, a run total is only summed
|
|
25
|
+
from outputs that agree with node state, and a task that never ran is
|
|
26
|
+
reported apart from one that ran and recorded nothing;
|
|
27
|
+
- parent-owned node state and immutable terminal outputs;
|
|
28
|
+
- restrictive permissions and atomic filesystem publication;
|
|
29
|
+
- bounded UTF-8 JSON records with explicit read and identity errors;
|
|
30
|
+
- an injected asynchronous task-executor interface;
|
|
31
|
+
- a strict versioned structured worker-report contract;
|
|
32
|
+
- defensive report validation and immutable JSON snapshots;
|
|
33
|
+
- normalized repository-relative changed-file paths and bounded diagnostics;
|
|
34
|
+
- blocker-bearing report failure with retained parent-visible output;
|
|
35
|
+
- bounded concurrent DAG execution with deterministic frontier selection;
|
|
36
|
+
- durable terminal output before dependent activation;
|
|
37
|
+
- direct-prerequisite-only output propagation;
|
|
38
|
+
- deterministic task-ordered prerequisite report serialization;
|
|
39
|
+
- named JSON report blocks with explicit untrusted-data labeling;
|
|
40
|
+
- exact UTF-8 context accounting and fail-without-truncation overflow behavior;
|
|
41
|
+
- explicit provider/model/thinking/tool worker profiles with no model defaults;
|
|
42
|
+
- one-shot Pi JSON-mode subprocess workers in the target checkout;
|
|
43
|
+
- task delivery over stdin rather than child-process arguments;
|
|
44
|
+
- child isolation from discovered extensions, skills, prompt templates, and sessions;
|
|
45
|
+
- a child-only terminating structured final-report tool, with recoverable
|
|
46
|
+
rejection so a worker can correct and resubmit a report;
|
|
47
|
+
- enforcement that the report really was the worker's final action, covering
|
|
48
|
+
Pi's parallel tool batches;
|
|
49
|
+
- positive identification of Pi's executable, with no shell on any platform;
|
|
50
|
+
- event-stream framing bounds that skip unparseable lines instead of failing a
|
|
51
|
+
worker whose transcript is legitimately large;
|
|
52
|
+
- allowlisted adapter failure diagnostics, surfaced through one pre-run error
|
|
53
|
+
type and one execution failure type;
|
|
54
|
+
- process-group cancellation with forced termination fallback and post-exit
|
|
55
|
+
collection of orphaned grandchildren;
|
|
56
|
+
- sanitized executor failures, cancellation, and task timeouts;
|
|
57
|
+
- task, dependency, concurrency, payload, output, and context limits;
|
|
58
|
+
- bounded, redacted worker progress projection and aggregate usage accounting;
|
|
59
|
+
- a strict global `worker-graph.json` profile configuration;
|
|
60
|
+
- a default state root beneath Pi's agent directory, with checkout-local roots
|
|
61
|
+
rejected by the extension;
|
|
62
|
+
- one static, fully bounded `worker_graph` parent tool;
|
|
63
|
+
- explicit `/swarm on`, `/swarm status`, `/swarm off`, `/swarm usage`, and
|
|
64
|
+
`--swarm` activation;
|
|
65
|
+
- an optional configured orchestrator session model and thinking level, applied
|
|
66
|
+
on activation and restored on exit, refused unless the mode knows the
|
|
67
|
+
identifiers that restore it, the model is one Pi can find, and its provider
|
|
68
|
+
is authenticated, with a model the configuration stopped naming still put
|
|
69
|
+
back and a restore that cannot be performed reported rather than swallowed;
|
|
70
|
+
- session-persisted swarm mode with exact active-tool restoration and built-in
|
|
71
|
+
parent mutation tools suppressed while active, with branch-recorded state
|
|
72
|
+
governing session-tree navigation and activation refused for any tool set the
|
|
73
|
+
extension could not restore;
|
|
74
|
+
- compact, explicitly bounded worker-report projection for parent review and
|
|
75
|
+
focused follow-up graphs, delivered in a labeled block worker text cannot
|
|
76
|
+
close;
|
|
77
|
+
- a configurable retained-run cap enforced by atomically claimed capacity slots,
|
|
78
|
+
with deterministic arbitration between concurrent creators, no implicit
|
|
79
|
+
deletion, stranded slots reported for explicit removal, and inconsistent
|
|
80
|
+
run/slot state failing closed;
|
|
81
|
+
- explicit retention cleanup through `/swarm runs` and `/swarm delete`: a
|
|
82
|
+
listing of everything holding capacity, and deletion of a named run with its
|
|
83
|
+
slot, refused while an orchestrator holds the run and ordered so an
|
|
84
|
+
interruption can only strand a slot;
|
|
85
|
+
- immutable bounded run-scoped coordination events and directed inbox messages,
|
|
86
|
+
in one run-global journal, with cursor-based queries and child-only Pi tools;
|
|
87
|
+
- a run mutation lock that names its holder, so contention between the parent
|
|
88
|
+
and a publishing worker is reported apart from an ownership conflict, waited
|
|
89
|
+
out rather than resolved by force, and recovered only when the lock names a
|
|
90
|
+
task the graph runner knows has finished;
|
|
91
|
+
- coordination cursors that run past records a reader is never given, so
|
|
92
|
+
polling costs only the records published since the previous call;
|
|
93
|
+
- a verified npm artifact containing the compiled runtime, loaded from a clean
|
|
94
|
+
temporary install in offline Pi RPC mode;
|
|
95
|
+
- behavioral coverage using `node:test`, fakes, and temporary directories.
|
|
96
|
+
|
|
97
|
+
Normal parent sessions register the `/swarm` control command, but the
|
|
98
|
+
`worker_graph` tool is inactive until explicitly enabled. The report tool is
|
|
99
|
+
registered only inside explicitly marked worker children. Active graph
|
|
100
|
+
lifecycles claim an exclusive owner before mutating run state; resumable or
|
|
101
|
+
externally addressable runs are not implemented yet.
|
|
102
|
+
|
|
103
|
+
## Verify the baseline
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
npm install
|
|
107
|
+
npm run check
|
|
108
|
+
npm run build
|
|
109
|
+
cp docs/worker-graph.example.json ~/.pi/agent/worker-graph.json
|
|
110
|
+
pi -e .
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
`npm run check` currently runs the adapter, configuration, context, extension,
|
|
114
|
+
graph, orchestrator, report, store, and runner suites. `npm run build` must run
|
|
115
|
+
before `pi -e .`, because `extensions/index.ts` re-exports the compiled entry
|
|
116
|
+
point from `dist/`. The configuration copy is required rather than optional:
|
|
117
|
+
the extension has no provider or model defaults, and `/swarm runs` and
|
|
118
|
+
`/swarm delete` resolve the state root through the same file, so a missing
|
|
119
|
+
`worker-graph.json` answers both subcommands with a configuration error.
|
|
120
|
+
Loading the package in Pi adds the `/swarm` control command, whose `runs` and
|
|
121
|
+
`delete` subcommands work whether or not the mode is enabled, but leaves the
|
|
122
|
+
parent tool set unchanged. The entry point registers
|
|
123
|
+
the worker report tool only when `PI_WORKER_GRAPH_ROLE=worker`, which the
|
|
124
|
+
parent sets on worker subprocesses and never on its own session. Active graph
|
|
125
|
+
lifecycles claim an exclusive owner record before mutating run state; resumable
|
|
126
|
+
or externally addressable runs are not implemented yet.
|
|
127
|
+
|
|
128
|
+
## Next implementation slice
|
|
129
|
+
|
|
130
|
+
Prepare the vertical slice for a prerelease:
|
|
131
|
+
|
|
132
|
+
1. Run an optional provider-backed smoke test with two independent workers and a
|
|
133
|
+
dependent validation node when provider-backed testing is desired; this is
|
|
134
|
+
intentionally skipped for the current local pass.
|
|
135
|
+
2. Review, tag, and publish the npm prerelease through the maintainer-owned Git
|
|
136
|
+
and registry workflow.
|
|
137
|
+
3. Type `/swarm runs` and `/swarm delete <run-id>` once in a live Pi session.
|
|
138
|
+
Their state-root resolution has now been exercised against a real agent
|
|
139
|
+
directory: a real `worker-graph.json` resolved the default
|
|
140
|
+
`~/.pi/agent/worker-graph` root, and a created run was listed with its slot
|
|
141
|
+
and then deleted, leaving `runs/slots` empty at mode `0700`. What remains
|
|
142
|
+
unverified is the command surface itself — argument parsing, the notified
|
|
143
|
+
text, and `ctx.cwd` as the working directory — because those need the
|
|
144
|
+
interactive session rather than a direct store call.
|
|
145
|
+
4. Keep every automated path provider-free behind the existing fake subprocess
|
|
146
|
+
and injected orchestrator boundaries.
|
|
147
|
+
5. Exercise the configured orchestrator model once in a live session. Its
|
|
148
|
+
activation, refusal, and restore paths are covered by fakes; what no test
|
|
149
|
+
can cover is Pi's own `setModel` against a real provider catalogue and real
|
|
150
|
+
authentication. Confirm specifically whether the shutdown restore lands:
|
|
151
|
+
`session_shutdown` awaits an asynchronous model change, and Pi does wait for
|
|
152
|
+
that handler on both quit paths: `interactive-mode.js` awaits
|
|
153
|
+
`runtimeHost.dispose()`, which awaits `emitSessionShutdownEvent`, which
|
|
154
|
+
awaits each handler in turn. It is not awaited on `emergencyTerminalExit`
|
|
155
|
+
or `uncaughtCrash`, where the terminal is already gone.
|
|
156
|
+
|
|
157
|
+
## Deferred run-store work
|
|
158
|
+
|
|
159
|
+
Before a resumable or externally addressable run API is added, retain the
|
|
160
|
+
fail-closed ownership contract so two orchestrators cannot advance one run, and
|
|
161
|
+
the rule that a mutation lock is recovered only by a holder that can be shown
|
|
162
|
+
to have finished.
|
|
163
|
+
|
|
164
|
+
Retained text artifacts are complete end to end and their policy is settled in
|
|
165
|
+
D19: publication is deliberate, and neither overflow becomes truncation.
|
|
166
|
+
|
|
167
|
+
One question is left open deliberately. The orchestrator learns that an
|
|
168
|
+
artifact exists, and how large it is, but has no way to read it: the review
|
|
169
|
+
names `artifactBytes` and not a path, and no tool returns the text. A library
|
|
170
|
+
caller uses `readNodeArtifact()`. Giving the orchestrator the artifact's path
|
|
171
|
+
would make its own `read` tool sufficient, but that widens what the parent may
|
|
172
|
+
reach outside the checkout, so it is a decision rather than an addition.
|
|
173
|
+
|
|
174
|
+
## Deferred worker instruction work
|
|
175
|
+
|
|
176
|
+
The worker concurrency contract is now sent in the worker prompt
|
|
177
|
+
(`src/pi-subprocess.ts`) and asserted by the adapter suite: stay inside the
|
|
178
|
+
assignment, prefer small exact edits, reconcile rather than restore a file to
|
|
179
|
+
the version first read, never run git restore/reset/checkout/stash/clean and
|
|
180
|
+
never commit, push, or branch, no repository-wide formatters or generators or
|
|
181
|
+
dependency updates without explicit ownership, re-read changed files before
|
|
182
|
+
reporting, and report an unclear semantic conflict as a blocker instead of
|
|
183
|
+
guessing. The orchestrator guidance in `src/orchestrator.ts` carries the
|
|
184
|
+
matching parent-side decomposition, overlap, serialization, and acceptance
|
|
185
|
+
rules.
|
|
186
|
+
|
|
187
|
+
What remains undecided is one report field. `NodeOutput` carries `summary`,
|
|
188
|
+
`changedFiles`, `interfaces`, `decisions`, `validation`, and `blockers`
|
|
189
|
+
(`src/output.ts`), so an observation about another worker's edits can only
|
|
190
|
+
reach the parent's terminal report as prose in `summary` or as a blocker.
|
|
191
|
+
Adding a field is a schema version change.
|
|
192
|
+
|
|
193
|
+
It is less pressing than it was. A worker that notices a concurrent change
|
|
194
|
+
already has a structured channel while it runs — a `conflict` coordination
|
|
195
|
+
event (`src/store.ts`), which the parent and other workers can read — and a
|
|
196
|
+
retained artifact for the long form of what it saw. Neither is the terminal
|
|
197
|
+
report, so the question stands; it is no longer a dead end.
|
|
198
|
+
|
|
199
|
+
## Deferred budget work
|
|
200
|
+
|
|
201
|
+
Usage is now recorded but nothing acts on it. The runtime bounds tasks,
|
|
202
|
+
concurrency, payload, output, context, and per-task runtime; it has no token or
|
|
203
|
+
cost ceiling, so a graph can spend without limit as long as each worker stays
|
|
204
|
+
inside its timeout.
|
|
205
|
+
|
|
206
|
+
The next slice is a configured budget in `worker-graph.json`, checked in the
|
|
207
|
+
runner between frontiers and enforced through the existing abort path, which
|
|
208
|
+
already settles remaining nodes and returns a result with usage intact. Two
|
|
209
|
+
things to settle first: whether the budget counts tokens or cost — cost is the
|
|
210
|
+
provider's estimate, tokens are the sturdy number — and whether crossing it
|
|
211
|
+
aborts the graph or only refuses to open the next frontier.
|
|
212
|
+
|
|
213
|
+
## Constraints to preserve
|
|
214
|
+
|
|
215
|
+
- Keep the implementation and exported API as small as possible.
|
|
216
|
+
- Validate the complete graph and runtime bounds before starting work.
|
|
217
|
+
- Do not add provider or model defaults.
|
|
218
|
+
- Do not add automatic Git operations or worktrees.
|
|
219
|
+
- Do not store runtime state in the target checkout by default.
|
|
220
|
+
- Do not propagate undeclared or unbounded context between tasks.
|
|
221
|
+
- Do not add recursive worker delegation.
|
|
222
|
+
- Use fake executors for automated integration tests.
|
|
223
|
+
|
|
224
|
+
## Decisions still needed
|
|
225
|
+
|
|
226
|
+
Review barriers, retention cleanup, and worker-session resumption semantics
|
|
227
|
+
remain deferred until their runtime layers are implemented. Broader Pi
|
|
228
|
+
compatibility can be claimed only after testing versions beyond the current
|
|
229
|
+
0.85.1 development pin.
|