@osolmaz/pi-workflows 0.16.4 → 0.16.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -5
- package/dist/builtins/autoimplement.workflow.d.ts +5 -1
- package/dist/builtins/autoimplement.workflow.js +29 -10
- package/dist/builtins/autoimplement.workflow.js.map +1 -1
- package/dist/client/client.js +3 -1
- package/dist/client/client.js.map +1 -1
- package/dist/client/protocol.d.ts +1 -1
- package/dist/client/protocol.js +18 -1
- package/dist/client/protocol.js.map +1 -1
- package/dist/client/view.d.ts +7 -5
- package/dist/extension/index.d.ts +2 -1
- package/dist/extension/index.js +115 -165
- package/dist/extension/index.js.map +1 -1
- package/dist/extension/recorder.d.ts +1 -1
- package/dist/extension/recorder.js +2 -2
- package/dist/extension/recorder.js.map +1 -1
- package/dist/extension/step-message.d.ts +1 -1
- package/dist/extension/step-message.js +4 -4
- package/dist/extension/step-message.js.map +1 -1
- package/dist/extension/workflow-message-coordinator.d.ts +12 -1
- package/dist/extension/workflow-message-coordinator.js +65 -54
- package/dist/extension/workflow-message-coordinator.js.map +1 -1
- package/dist/resource-managers/index.d.ts +1 -2
- package/dist/resource-managers/index.js +1 -2
- package/dist/resource-managers/index.js.map +1 -1
- package/dist/resource-managers/sqlite.d.ts +2 -295
- package/dist/resource-managers/sqlite.js +71 -1528
- package/dist/resource-managers/sqlite.js.map +1 -1
- package/dist/resource-managers/types.d.ts +2 -7
- package/dist/server/resource-runner-entry.js +2 -6
- package/dist/server/resource-runner-entry.js.map +1 -1
- package/dist/server/rpc-bridge.js +1 -1
- package/dist/server/rpc-bridge.js.map +1 -1
- package/dist/server/rpc-executor.js +2 -4
- package/dist/server/rpc-executor.js.map +1 -1
- package/dist/server/server-entry.js +3 -0
- package/dist/server/server-entry.js.map +1 -1
- package/dist/server/server.d.ts +22 -8
- package/dist/server/server.js +498 -543
- package/dist/server/server.js.map +1 -1
- package/dist/server/state.d.ts +16 -29
- package/dist/server/state.js +105 -319
- package/dist/server/state.js.map +1 -1
- package/dist/server/view.d.ts +4 -5
- package/dist/server/view.js +51 -51
- package/dist/server/view.js.map +1 -1
- package/dist/server/workflow-runner-entry.d.ts +1 -1
- package/dist/server/workflow-runner-entry.js +1 -49
- package/dist/server/workflow-runner-entry.js.map +1 -1
- package/dist/server/workflow-runner-protocol.d.ts +2 -7
- package/dist/server/workflow-runner-protocol.js +5 -10
- package/dist/server/workflow-runner-protocol.js.map +1 -1
- package/dist/server/workflow-runner-store.d.ts +5 -6
- package/dist/server/workflow-runner-store.js +7 -14
- package/dist/server/workflow-runner-store.js.map +1 -1
- package/dist/state/attempt-time.d.ts +21 -0
- package/dist/state/attempt-time.js +95 -0
- package/dist/state/attempt-time.js.map +1 -0
- package/dist/state/json.d.ts +2 -1
- package/dist/state/json.js.map +1 -1
- package/dist/state/project-store.d.ts +39 -0
- package/dist/state/project-store.js +154 -0
- package/dist/state/project-store.js.map +1 -0
- package/dist/state/prune.d.ts +28 -0
- package/dist/state/prune.js +278 -83
- package/dist/state/prune.js.map +1 -1
- package/dist/state/schema.js +23 -23
- package/dist/state/workflow-messages.d.ts +1 -1
- package/dist/state/workflow-messages.js +3 -1
- package/dist/state/workflow-messages.js.map +1 -1
- package/dist/viewer/backup.js +5 -0
- package/dist/viewer/backup.js.map +1 -1
- package/dist/workflows/diagnostics.d.ts +8 -0
- package/dist/workflows/diagnostics.js +61 -0
- package/dist/workflows/diagnostics.js.map +1 -0
- package/dist/workflows/engine.d.ts +2 -15
- package/dist/workflows/engine.js +125 -245
- package/dist/workflows/engine.js.map +1 -1
- package/dist/workflows/human-decision.d.ts +3 -7
- package/dist/workflows/human-decision.js +2 -56
- package/dist/workflows/human-decision.js.map +1 -1
- package/dist/workflows/index.d.ts +2 -1
- package/dist/workflows/index.js +1 -0
- package/dist/workflows/index.js.map +1 -1
- package/dist/workflows/queue.d.ts +267 -0
- package/dist/workflows/queue.js +1248 -0
- package/dist/workflows/queue.js.map +1 -0
- package/dist/workflows/requests.d.ts +47 -0
- package/dist/workflows/requests.js +158 -0
- package/dist/workflows/requests.js.map +1 -0
- package/dist/workflows/schema.js +2 -4
- package/dist/workflows/schema.js.map +1 -1
- package/dist/workflows/store.d.ts +11 -16
- package/dist/workflows/store.js +174 -169
- package/dist/workflows/store.js.map +1 -1
- package/dist/workflows/tool-input.d.ts +6 -9
- package/dist/workflows/tool-input.js +15 -12
- package/dist/workflows/tool-input.js.map +1 -1
- package/dist/workflows/transitions.d.ts +38 -0
- package/dist/workflows/transitions.js +276 -0
- package/dist/workflows/transitions.js.map +1 -0
- package/dist/workflows/types.d.ts +9 -42
- package/dist/workflows/workflow-message-content.d.ts +7 -1
- package/dist/workflows/workflow-message-content.js +22 -4
- package/dist/workflows/workflow-message-content.js.map +1 -1
- package/docs/2026-08-25-workflow-follow-ups.md +5 -5
- package/docs/2026-09-02-unify-workflow-messages-plan.md +2 -0
- package/docs/2026-09-04-workflow-run-state-plan.md +2 -0
- package/docs/2026-09-06-durable-execution-plan.md +176 -0
- package/docs/DEFERRED_TURNS.md +84 -105
- package/docs/HUMAN_DECISIONS.md +17 -19
- package/docs/HUMAN_DECISION_PRESENTATIONS.md +1 -1
- package/docs/SQLITE_STATE.md +87 -30
- package/docs/WORKFLOW_SERVER.md +50 -11
- package/docs/WORKFLOW_STEP_MESSAGES.md +72 -54
- package/docs/plans/2026-08-04-controller-runtime-plan.md +2 -0
- package/docs/plans/2026-08-21-deferred-turn-intents-plan.md +2 -0
- package/docs/plans/2026-08-27-workflow-terminal-restart-plan.md +2 -0
- package/docs/plans/2026-09-04-automatic-state-retention-plan.md +283 -0
- package/docs/workflows.md +109 -44
- package/examples/workflows/autoresearch.workflow.ts +0 -2
- package/examples/workflows/branch.workflow.ts +0 -4
- package/examples/workflows/echo.workflow.ts +0 -2
- package/examples/workflows/live-settings.workflow.ts +0 -2
- package/examples/workflows/two-turn.workflow.ts +0 -2
- package/herdr-plugin.toml +1 -1
- package/package.json +1 -1
- package/protocol/fixtures/client-v1.json +1 -1
- package/skills/pi-workflows/SKILL.md +8 -7
- package/src/builtins/autoimplement.workflow.ts +31 -10
- package/src/client/client.ts +3 -1
- package/src/client/protocol.ts +21 -1
- package/src/client/view.ts +9 -5
- package/src/extension/index.ts +117 -189
- package/src/extension/recorder.ts +2 -2
- package/src/extension/step-message.ts +4 -4
- package/src/extension/workflow-message-coordinator.ts +78 -50
- package/src/resource-managers/index.ts +1 -9
- package/src/resource-managers/sqlite.ts +330 -2655
- package/src/resource-managers/types.ts +2 -3
- package/src/server/resource-runner-entry.ts +2 -7
- package/src/server/rpc-bridge.ts +1 -1
- package/src/server/rpc-executor.ts +4 -9
- package/src/server/server-entry.ts +3 -0
- package/src/server/server.ts +568 -644
- package/src/server/state.ts +122 -466
- package/src/server/view.ts +56 -60
- package/src/server/workflow-runner-entry.ts +2 -55
- package/src/server/workflow-runner-protocol.ts +5 -15
- package/src/server/workflow-runner-store.ts +12 -27
- package/src/state/attempt-time.ts +122 -0
- package/src/state/json.ts +2 -1
- package/src/state/project-store.ts +218 -0
- package/src/state/prune.ts +408 -95
- package/src/state/schema.ts +23 -23
- package/src/state/workflow-messages.ts +4 -2
- package/src/viewer/backup.ts +7 -0
- package/src/workflows/diagnostics.ts +78 -0
- package/src/workflows/engine.ts +133 -291
- package/src/workflows/human-decision.ts +3 -81
- package/src/workflows/index.ts +8 -2
- package/src/workflows/queue.ts +2091 -0
- package/src/workflows/requests.ts +283 -0
- package/src/workflows/schema.ts +2 -6
- package/src/workflows/store.ts +223 -231
- package/src/workflows/tool-input.ts +20 -12
- package/src/workflows/transitions.ts +330 -0
- package/src/workflows/types.ts +9 -46
- package/src/workflows/workflow-message-content.ts +28 -4
- package/dist/resource-managers/runtime.d.ts +0 -59
- package/dist/resource-managers/runtime.js +0 -423
- package/dist/resource-managers/runtime.js.map +0 -1
- package/src/resource-managers/runtime.ts +0 -576
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Workflow messages in Pi
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
This is the current workflow-message contract. The [durable execution plan](2026-09-06-durable-execution-plan.md) supersedes the earlier automatic terminal-turn and reminder behavior.
|
|
4
4
|
|
|
5
5
|
## Goal
|
|
6
6
|
|
|
7
|
-
Pi Workflows must add several kinds of content to an origin Pi conversation. These include interactive step prompts, protected human decisions, passive notifications, terminal results, and follow-up prompts. Initial
|
|
7
|
+
Pi Workflows must add several kinds of content to an origin Pi conversation. These include interactive step prompts, protected human decisions, passive notifications, terminal results, and follow-up prompts. Initial and resumed prompts are one step-message kind with different display reasons.
|
|
8
8
|
|
|
9
9
|
The server saves all of them as workflow messages. One extension component sends them through documented Pi APIs. Feature records continue to own workflow results, answers, settings, and timeouts.
|
|
10
10
|
|
|
@@ -16,13 +16,13 @@ A workflow message is content that Pi Workflows requires Pi to add to one conver
|
|
|
16
16
|
|
|
17
17
|
The message kinds are:
|
|
18
18
|
|
|
19
|
-
| Kind | Pi behavior
|
|
20
|
-
| -------------- |
|
|
21
|
-
| `step` | Custom message that starts a model turn
|
|
22
|
-
| `decision` | Custom message that does not start a model turn
|
|
23
|
-
| `notification` | Custom message that does not start a model turn
|
|
24
|
-
| `terminal` |
|
|
25
|
-
| `followUp` | Custom message that starts normal work
|
|
19
|
+
| Kind | Pi behavior | Purpose |
|
|
20
|
+
| -------------- | ------------------------------------------------ | ------------------------------------------ |
|
|
21
|
+
| `step` | Custom message that starts a model turn | Initial or resumed interactive prompt |
|
|
22
|
+
| `decision` | Custom message that does not start a model turn | Protected choice for a person |
|
|
23
|
+
| `notification` | Custom message that does not start a model turn | Passive workflow notice |
|
|
24
|
+
| `terminal` | Visible message that does not start a model turn | Recorded terminal result |
|
|
25
|
+
| `followUp` | Custom message that starts normal work | Work saved for after successful completion |
|
|
26
26
|
|
|
27
27
|
The server stores one `WorkflowMessage` record before Pi can send it:
|
|
28
28
|
|
|
@@ -61,7 +61,7 @@ Every step message uses the custom type `pi-workflows-step`:
|
|
|
61
61
|
export type WorkflowAgentStepMessageDetails = {
|
|
62
62
|
schema: "pi-workflows.agent-step-message.v1";
|
|
63
63
|
workflowMessageId: string;
|
|
64
|
-
reason: "initial" | "
|
|
64
|
+
reason: "initial" | "resumed";
|
|
65
65
|
contract: AgentStepContract;
|
|
66
66
|
presentation?: {
|
|
67
67
|
runTitle?: string;
|
|
@@ -84,7 +84,7 @@ pi.sendMessage(
|
|
|
84
84
|
|
|
85
85
|
The renderer reads `details` and does not parse the prompt. It shows a compact summary by default and the complete prompt when expanded. If the renderer is unavailable, Pi still retains the custom message and its content.
|
|
86
86
|
|
|
87
|
-
Submitted agent steps call
|
|
87
|
+
Submitted agent steps call `workflow submit` or `workflow update` with the exact `requestId`. Assistant-message steps reply normally and are accepted only through the separate assistant-response path at `agent_settled`. Ordinary checkpoints accept `answer`; protected decisions accept only the verified human path or their declared timeout policy. Both forms keep the existing `agent` node and use `expectedOutput` to select the completion form.
|
|
88
88
|
|
|
89
89
|
## Engine boundary
|
|
90
90
|
|
|
@@ -103,10 +103,10 @@ A collapsed step card shows only the workflow identity and current work. For exa
|
|
|
103
103
|
Checking the monitored target
|
|
104
104
|
```
|
|
105
105
|
|
|
106
|
-
A
|
|
106
|
+
A resumed prompt adds a short label:
|
|
107
107
|
|
|
108
108
|
```text
|
|
109
|
-
↻ monitor › check ·
|
|
109
|
+
↻ monitor › check · resumed
|
|
110
110
|
Checking the monitored target
|
|
111
111
|
```
|
|
112
112
|
|
|
@@ -141,58 +141,75 @@ After Pi, the extension, or the server restarts, branch reporting runs before an
|
|
|
141
141
|
|
|
142
142
|
## Model-turn status
|
|
143
143
|
|
|
144
|
-
`agent_start` has no message payload.
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
144
|
+
`agent_start` has no message payload. A locally delivered prompt binds its start
|
|
145
|
+
through the coordinator's saved message identity. Late binding and reconnect
|
|
146
|
+
also require the exact message in the active branch. A session view alone does
|
|
147
|
+
not prove that a message caused the current turn.
|
|
148
|
+
|
|
149
|
+
Only pending, unpaused agent requests and explicit follow-ups can open model
|
|
150
|
+
turns. Decisions, notifications, and terminal notices cannot. A stale start can
|
|
151
|
+
adopt an already closed result but cannot create new activity. An identity
|
|
152
|
+
mismatch is rejected before a turn changes.
|
|
153
|
+
|
|
154
|
+
`agent_end` records low-level stop information. The coordinator retains the
|
|
155
|
+
workflow turn through automatic Pi retries and waits for `agent_settled` before
|
|
156
|
+
submitting visible assistant text, recording the turn end, or delivering more
|
|
157
|
+
work. End reports carry `completed`, `aborted`, `error`, or `lost` and the exact
|
|
158
|
+
response-entry evidence when available. A delayed acknowledgment keeps the settled
|
|
159
|
+
response ID pending. Once message and turn ownership are confirmed, the coordinator
|
|
160
|
+
submits that exact response before it reports the end. Later events cannot replace
|
|
161
|
+
the pending turn or its response. Repeated reports adopt the saved result.
|
|
162
|
+
|
|
163
|
+
An aborted pending step pauses its run. Resume retains the request and attempt,
|
|
164
|
+
advances the request revision, and creates one resumed step message when needed.
|
|
165
|
+
A protected decision keeps its answer revision and decision message. A missing
|
|
166
|
+
submission stays pending: the host adds no reminder turn or hidden retry limit.
|
|
167
|
+
|
|
168
|
+
Execution status and Pi activity remain separate. A completed run remains
|
|
169
|
+
completed during reporting or follow-up work. A waiting run can have an active
|
|
170
|
+
origin-session turn without changing its execution status to running. Host
|
|
171
|
+
recovery closes active-time intervals at their last durable samples, not the Pi
|
|
172
|
+
turn itself. Only an idle-session branch report can prove an unended turn lost.
|
|
159
173
|
|
|
160
174
|
## Feature ownership
|
|
161
175
|
|
|
162
176
|
The workflow message stores only Pi send facts. Other records remain authoritative:
|
|
163
177
|
|
|
164
|
-
- interactive requests own
|
|
165
|
-
-
|
|
178
|
+
- interactive requests own exact response identity, kind, validation, and accepted submissions;
|
|
179
|
+
- node attempts and active intervals own execution history and the active-time budget;
|
|
180
|
+
- human decisions own choices, verified answers, and absolute expiry;
|
|
166
181
|
- terminal runs own outcomes, reasons, restart lineage, and results;
|
|
167
182
|
- notification nodes own their node results;
|
|
168
183
|
- follow-up records own prompt source and authority;
|
|
169
184
|
- settings records own current values and accepted changes.
|
|
170
185
|
|
|
171
|
-
Submitted
|
|
172
|
-
|
|
173
|
-
|
|
186
|
+
Submitted and assistant-message steps keep their exact attempt while parked.
|
|
187
|
+
Recovery adopts accepted receipts and matching branch evidence; it does not use
|
|
188
|
+
the oldest pending request or accept text from another request or branch.
|
|
174
189
|
|
|
175
|
-
|
|
190
|
+
Requests and their required messages commit together. Terminal reporting can be
|
|
191
|
+
retried from recorded facts after execution settles; a reporting failure cannot
|
|
192
|
+
reverse execution or cancellation.
|
|
176
193
|
|
|
177
|
-
|
|
194
|
+
## Terminal results and follow-ups
|
|
178
195
|
|
|
179
|
-
|
|
180
|
-
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
- protected decisions in Pi and approved external channels;
|
|
185
|
-
- passive notifications that do not start model turns;
|
|
186
|
-
- terminal result retention while its message is pending or its first turn is open, and then for 60 seconds after that turn ends, in the widget and `piw`;
|
|
187
|
-
- conversation recording linked to workflow attempts.
|
|
196
|
+
A terminal result is visible but does not start a model turn. It stays in the
|
|
197
|
+
origin-session view while pending and for 60 seconds after confirmed delivery.
|
|
198
|
+
An explicit restart targets the terminal run and revision and creates fresh
|
|
199
|
+
work without copying old steps, changed settings, approvals, or effects. It has
|
|
200
|
+
no terminal-turn prerequisite or hard-coded count limit.
|
|
188
201
|
|
|
189
|
-
|
|
202
|
+
Explicit follow-ups wait for successful completion, terminal notice delivery,
|
|
203
|
+
prior follow-up settlement, and release of the session reservation. They remain
|
|
204
|
+
normal conversation work. Slash-looking text cannot dispatch an extension
|
|
205
|
+
command. External effects still require saved receipts or explicit recovery of
|
|
206
|
+
an ambiguous outcome. See [terminal workflow messages](DEFERRED_TURNS.md).
|
|
190
207
|
|
|
191
208
|
## Session recording
|
|
192
209
|
|
|
193
210
|
The extension records workflow-related Pi events through a batched server client operation. It uses documented Pi events and does not read or edit Pi session files.
|
|
194
211
|
|
|
195
|
-
The server deduplicates settled entries by Pi entry ID. It links attempts to their prompt, response, first, and last entries. A recording failure does not fail workflow execution.
|
|
212
|
+
The server deduplicates settled entries by Pi entry ID. It links attempts to their prompt, response, first, and last entries. The coordinator finalizes capture after the terminal notice has a confirmed entry on the current branch, without starting a model turn. Explicit follow-up turns have their own capture segment. A recording failure does not fail workflow execution or block the next delivery.
|
|
196
213
|
|
|
197
214
|
## Public API boundary
|
|
198
215
|
|
|
@@ -205,26 +222,27 @@ This design does not change Pi core, use private Pi APIs, or change Pi session s
|
|
|
205
222
|
Tests must prove:
|
|
206
223
|
|
|
207
224
|
- every message kind uses the one coordinator;
|
|
208
|
-
- one workflow message ID
|
|
225
|
+
- one workflow message ID adopts its confirmed Pi entry before another send;
|
|
226
|
+
- an automatic Pi retry cannot submit a partial response before the settled boundary;
|
|
209
227
|
- a later manual turn uses a new workflow turn ID without creating another Pi entry;
|
|
210
228
|
- two Pi processes that open one session cannot both send because one process-local coordinator epoch is active;
|
|
211
229
|
- restart recovery reports the branch and adopts an existing entry before it sends;
|
|
212
230
|
- branch absence is usable only when Pi is idle and has no pending input;
|
|
213
231
|
- a crash after send leaves the message pending and cannot cause a resend before branch reporting;
|
|
214
232
|
- messages remain in saved order without message-pointer deadlocks;
|
|
215
|
-
- an early `agent_start` and `
|
|
216
|
-
-
|
|
217
|
-
- normal runner
|
|
233
|
+
- an early `agent_start` and `agent_settled` wait for the message receipt and session view, then apply in order;
|
|
234
|
+
- model activity remains separate from durable execution status;
|
|
235
|
+
- normal runner progress leaves the active session capture open until the matching turn ends;
|
|
218
236
|
- stale turn-end reports and starts against closed messages are rejected;
|
|
219
237
|
- a manual turn cancels pending step messages that it supersedes;
|
|
220
238
|
- a turn cannot bind to an interaction whose run is paused;
|
|
221
|
-
- aborted turns pause without
|
|
239
|
+
- aborted pending turns pause without adding reminder work;
|
|
222
240
|
- resuming an aborted step creates one new resumed message and one fresh model turn;
|
|
223
241
|
- resuming a protected decision keeps its answer revision and does not create a duplicate decision message;
|
|
224
242
|
- server restart does not close a live Pi turn, while an idle-session branch report can close an unended turn as lost;
|
|
225
|
-
-
|
|
226
|
-
- initial
|
|
227
|
-
- terminal and follow-
|
|
243
|
+
- repeated missing submissions remain pending without automatic extra model turns;
|
|
244
|
+
- initial and resumed prompts use the same step kind and differ only by reason;
|
|
245
|
+
- terminal messages never start turns, and follow-ups start only at their declared boundary;
|
|
228
246
|
- a branch switch creates one resumed-reason step only when that branch has no entry for the interaction;
|
|
229
247
|
- a missing protected decision creates another decision message, not a step;
|
|
230
248
|
- branch evidence changes a cancelled message to sent;
|
|
@@ -8,6 +8,8 @@ status: implemented
|
|
|
8
8
|
|
|
9
9
|
# ResourceManager runtime plan
|
|
10
10
|
|
|
11
|
+
> Historical plan. The [durable execution plan](../2026-09-06-durable-execution-plan.md) supersedes its continuation, scheduling, and automatic model-turn choices. Use the current reference pages for the implemented contract.
|
|
12
|
+
|
|
11
13
|
pi-workflows needs a controller mode for automation that spans repeated events, external state changes, and process restarts. The design in [RESOURCE_MANAGERS.md](../RESOURCE_MANAGERS.md) follows the Kubernetes controller pattern. Durable resources hold desired and observed state, events enqueue resource keys, and each reconciliation reads current facts before acting.
|
|
12
14
|
|
|
13
15
|
The implementation keeps the graph engine focused on finite jobs. Controllers start and observe workflows through a child-run interface. Workflow graphs keep their finite execution model.
|
|
@@ -6,6 +6,8 @@ date: 2026-08-21
|
|
|
6
6
|
|
|
7
7
|
# Guarantee one successor turn after workflow interruption
|
|
8
8
|
|
|
9
|
+
> Historical plan. The [durable execution plan](../2026-09-06-durable-execution-plan.md) supersedes its continuation, scheduling, and automatic model-turn choices. Use the current reference pages for the implemented contract.
|
|
10
|
+
|
|
9
11
|
Pi Workflows can stop the active agent turn before the agent receives a tool result or can take its next action. This happens when an active workflow step is cancelled, times out, loses its queue claim, or is interrupted by a controller. The workflow can also return a successful start result and then fail asynchronously before it sends its first prompt. In that case, the user can see a UI error while the model still believes that the workflow started successfully.
|
|
10
12
|
|
|
11
13
|
Add one general rule: an eligible workflow event creates one durable obligation for one later agent turn. The next normal workflow prompt or result presentation satisfies the obligation when one exists. Otherwise, Pi Workflows sends one factual fallback turn after the old turn settles.
|
|
@@ -6,6 +6,8 @@ date: 2026-08-27
|
|
|
6
6
|
|
|
7
7
|
# Full plan
|
|
8
8
|
|
|
9
|
+
> Historical plan. The [durable execution plan](../2026-09-06-durable-execution-plan.md) supersedes its continuation, scheduling, and automatic model-turn choices. Use the current reference pages for the implemented contract.
|
|
10
|
+
|
|
9
11
|
## Goal
|
|
10
12
|
|
|
11
13
|
After every top-level workflow run ends, give the model one normal successor turn.
|
|
@@ -0,0 +1,283 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Add automatic workflow state retention
|
|
3
|
+
author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
|
|
4
|
+
date: 2026-09-04
|
|
5
|
+
status: approved
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Add automatic workflow state retention
|
|
9
|
+
|
|
10
|
+
## Goal
|
|
11
|
+
|
|
12
|
+
Pi Workflows will remove old completed workflow state automatically. It will keep active, resumable, pending, unsettled, and undelivered work.
|
|
13
|
+
|
|
14
|
+
The server will retain terminal run trees for 30 days. It will then remove safe expired trees and blobs that no retained record or active runner uses. The manual prune command will use the same safety rules.
|
|
15
|
+
|
|
16
|
+
A regression test will also protect the earlier runner fix. Repeated runner replies must not copy growing Pi session history and make the database grow by several gigabytes.
|
|
17
|
+
|
|
18
|
+
## Observed problem
|
|
19
|
+
|
|
20
|
+
One old state database reached about 6.1 GB. The `blobs` table held about 6.06 GB across 101,355 blobs, which were almost all JSON. `worker_messages.result_hash` was the main reference path.
|
|
21
|
+
|
|
22
|
+
Older runner replies copied growing session and workflow history into new immutable result blobs. Small changes produced another large blob, so content hashing could not deduplicate the replies. [Keep large workflow history out of runner resume replies](2026-09-04-workflow-runner-resume-state-plan.md) fixed that interface by returning only the execution state a runner needs.
|
|
23
|
+
|
|
24
|
+
That fix stops the main amplification path. Completed workflow history still remains until a person runs `pi-workflows state prune`, so the database can still grow without a retention rule.
|
|
25
|
+
|
|
26
|
+
## Selected design
|
|
27
|
+
|
|
28
|
+
Use the existing run-tree prune logic as the one cleanup engine. Automatic cleanup and manual prune will share selection, deletion, blob collection, and compaction rules.
|
|
29
|
+
|
|
30
|
+
A terminal root run and all its restart or continuation descendants remain for 30 days after `finished_at`. Every descendant must be terminal and older than the cutoff before the server can remove the tree. Protected state or a reference from outside the tree blocks removal.
|
|
31
|
+
|
|
32
|
+
The Workflow Server checks for cleanup after startup recovery and after workflow runners exit. It also schedules a daily check after each completed sweep. It starts cleanup only while normal server work is idle. One server process completes no more than one sweep in 24 hours. An interrupted sweep remains due. The next idle lifecycle trigger or a five-minute idle retry continues it.
|
|
33
|
+
|
|
34
|
+
Automatic cleanup creates no backup. Creating a new backup for every sweep would cause another unbounded store. The explicit manual prune command keeps its current dry-run and backup-first apply forms.
|
|
35
|
+
|
|
36
|
+
This design does not add size-based deletion. A clear time limit gives users predictable history. Recent or protected data can still be large, so this design does not promise a hard disk-size cap.
|
|
37
|
+
|
|
38
|
+
## Retention contract
|
|
39
|
+
|
|
40
|
+
### Eligible run trees
|
|
41
|
+
|
|
42
|
+
A run tree is eligible only when all of these facts are true:
|
|
43
|
+
|
|
44
|
+
- The root run and every descendant have status `completed`, `failed`, `timed_out`, or `cancelled`.
|
|
45
|
+
- Every run in the tree has `finished_at` earlier than the 30-day cutoff.
|
|
46
|
+
- No row outside the tree depends on a row inside it.
|
|
47
|
+
- No protected work belongs to the tree.
|
|
48
|
+
|
|
49
|
+
The cleanup transaction rechecks the exact root tree before deletion. A changed tree is skipped.
|
|
50
|
+
|
|
51
|
+
### Protected work
|
|
52
|
+
|
|
53
|
+
Automatic cleanup must keep a tree when it contains or owns any of this work:
|
|
54
|
+
|
|
55
|
+
- a waiting or parked run
|
|
56
|
+
- a queued, starting, running, or parked queue row
|
|
57
|
+
- a pending workflow message
|
|
58
|
+
- an open workflow turn
|
|
59
|
+
- a pending interaction or human decision
|
|
60
|
+
- a recording session segment
|
|
61
|
+
- a queued follow-up
|
|
62
|
+
- an active lease
|
|
63
|
+
- a pending, applying, or ambiguous effect
|
|
64
|
+
- controller ownership or a managed resource reference
|
|
65
|
+
- a continuation or step reference from outside the tree
|
|
66
|
+
- an active runner content hash
|
|
67
|
+
- a resumable checkpoint
|
|
68
|
+
- an undelivered terminal result
|
|
69
|
+
|
|
70
|
+
Unknown or conflicting ownership blocks deletion. Cleanup must fail closed.
|
|
71
|
+
|
|
72
|
+
### Automatic scheduling
|
|
73
|
+
|
|
74
|
+
The server requests one automatic sweep after recovery and after a workflow runner exits. Overlapping requests join the same in-process task.
|
|
75
|
+
|
|
76
|
+
A sweep starts only when there is no active or pending workflow runner, resource-manager runner, state-maintenance command, or server shutdown. It deletes one complete root tree in one transaction, yields, and checks for new work before it selects another tree.
|
|
77
|
+
|
|
78
|
+
A completed sweep starts a 24-hour in-process interval and schedules the next daily check. A sweep that stops because new work appeared remains due. The next idle lifecycle trigger or a five-minute idle retry continues it. A new request during the completed-sweep interval waits until that interval ends.
|
|
79
|
+
|
|
80
|
+
The cleanup scheduler is part of the existing server process. It does not add a service, scheduler process, or second writer.
|
|
81
|
+
|
|
82
|
+
### Blob cleanup
|
|
83
|
+
|
|
84
|
+
After run deletion, the state layer removes only blobs with no database foreign-key reference and no active runner reference. The existing schema scan remains the source for blob references, so a future blob foreign key is protected automatically.
|
|
85
|
+
|
|
86
|
+
A repeated sweep is safe. A second sweep finds no extra rows from work that was already removed.
|
|
87
|
+
|
|
88
|
+
### Space reuse and file compaction
|
|
89
|
+
|
|
90
|
+
After logical deletion, the server truncates the WAL while idle and reads SQLite `page_count`, `freelist_count`, and `page_size`. SQLite can reuse free pages even when the main file does not shrink.
|
|
91
|
+
|
|
92
|
+
Automatic cleanup runs `VACUUM` only when all of these facts are true:
|
|
93
|
+
|
|
94
|
+
- normal server work is still idle
|
|
95
|
+
- at least 64 MiB is reclaimable
|
|
96
|
+
- at least 20 percent of database pages are free
|
|
97
|
+
|
|
98
|
+
A skipped or failed `VACUUM` does not undo committed logical deletion. The server reports the outcome and leaves the free pages available for reuse. It does not claim that the file shrank unless measurement proves it.
|
|
99
|
+
|
|
100
|
+
Manual prune keeps its current backup-first full compaction and integrity checks.
|
|
101
|
+
|
|
102
|
+
### Failure handling
|
|
103
|
+
|
|
104
|
+
A cleanup error does not fail a workflow or stop the server. The server records one clear diagnostic and waits five minutes before another safe attempt. It does not retry in a tight loop.
|
|
105
|
+
|
|
106
|
+
Cleanup is atomic for one complete root tree. An interruption can leave later trees for another sweep, but it cannot leave half of one lineage tree deleted.
|
|
107
|
+
|
|
108
|
+
## Public behavior
|
|
109
|
+
|
|
110
|
+
Retained runs keep their current resume, viewer, content-reference, and terminal-result behavior. The viewer continues to read bounded pages. Pi Workflows does not edit Pi session history.
|
|
111
|
+
|
|
112
|
+
After an expired tree is removed, it no longer appears in run lists. A direct run view returns not found.
|
|
113
|
+
|
|
114
|
+
The manual commands remain:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
pi-workflows state prune --before <timestamp> --dry-run
|
|
118
|
+
pi-workflows state prune --before <timestamp> --backup <absolute-path> --apply
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Automatic cleanup adds no client protocol operation. The SQLite schema name and version remain `pi-workflows-state` version 1. This change adds no migration, compatibility reader, second state path, fallback, feature flag, archive, or external resource.
|
|
122
|
+
|
|
123
|
+
## Implementation
|
|
124
|
+
|
|
125
|
+
### Share the cleanup engine
|
|
126
|
+
|
|
127
|
+
**Where**
|
|
128
|
+
|
|
129
|
+
- `src/state/prune.ts`
|
|
130
|
+
- version-1 invariants in `src/state/schema.ts`
|
|
131
|
+
|
|
132
|
+
**Change**
|
|
133
|
+
|
|
134
|
+
Split the current prune work into shared run-tree selection, exact-tree deletion, unreferenced-blob collection, page measurement, and compaction operations. Add every protected-state check from this plan.
|
|
135
|
+
|
|
136
|
+
Automatic deletion must recheck and delete one root tree in the same transaction. Manual prune keeps its backup and complete-selection recheck.
|
|
137
|
+
|
|
138
|
+
**Check**
|
|
139
|
+
|
|
140
|
+
State tests prove that eligible trees are removed as one unit. Each protected state keeps its whole tree. Foreign-key and integrity checks pass after deletion.
|
|
141
|
+
|
|
142
|
+
### Add automatic cleanup
|
|
143
|
+
|
|
144
|
+
**Where**
|
|
145
|
+
|
|
146
|
+
- `src/state/prune.ts`
|
|
147
|
+
- `src/server/server.ts`
|
|
148
|
+
|
|
149
|
+
**Change**
|
|
150
|
+
|
|
151
|
+
Add an automatic entry point with a cutoff of the current time minus 30 days. It uses the shared cleanup engine without creating a backup.
|
|
152
|
+
|
|
153
|
+
Process one complete root tree per transaction. Yield between trees. Stop when normal server work appears and leave the sweep due.
|
|
154
|
+
|
|
155
|
+
**Check**
|
|
156
|
+
|
|
157
|
+
Automatic and manual cleanup select the same eligible trees. Automatic cleanup creates no backup. A stopped sweep continues later without duplicate deletion.
|
|
158
|
+
|
|
159
|
+
### Schedule cleanup from the server lifecycle
|
|
160
|
+
|
|
161
|
+
**Where**
|
|
162
|
+
|
|
163
|
+
- lifecycle fields in `src/server/server.ts`
|
|
164
|
+
- `WorkflowServer.start()`
|
|
165
|
+
- server shutdown
|
|
166
|
+
- the active-run exit path after the server removes the run from `activeRuns`
|
|
167
|
+
|
|
168
|
+
**Change**
|
|
169
|
+
|
|
170
|
+
Request cleanup after startup recovery and after runner exit. Schedule a daily check after each completed sweep. Coalesce overlapping requests. Enforce the idle check, five-minute idle retry, and 24-hour completed-sweep interval.
|
|
171
|
+
|
|
172
|
+
A failed sweep logs one bounded error and waits for a later trigger. Shutdown starts no new sweep and settles any current scheduling task safely.
|
|
173
|
+
|
|
174
|
+
**Check**
|
|
175
|
+
|
|
176
|
+
Server tests use controlled scheduler timestamps and temporary databases to cover startup, runner exit, coalescing, the 24-hour interval, interruption, clean shutdown, and one injected failure.
|
|
177
|
+
|
|
178
|
+
### Reuse pages and compact when worthwhile
|
|
179
|
+
|
|
180
|
+
**Where**
|
|
181
|
+
|
|
182
|
+
- database-size helpers in `src/state/prune.ts`
|
|
183
|
+
- `test/prune.test.ts`
|
|
184
|
+
|
|
185
|
+
**Change**
|
|
186
|
+
|
|
187
|
+
Measure free SQLite pages after deletion. Truncate the WAL while idle. Run automatic `VACUUM` only above the 64 MiB and 20 percent thresholds.
|
|
188
|
+
|
|
189
|
+
Keep committed deletion when compaction is skipped or fails. Report logical deletion and physical file size separately.
|
|
190
|
+
|
|
191
|
+
**Check**
|
|
192
|
+
|
|
193
|
+
Tests prove page reuse below the thresholds, no early automatic `VACUUM`, physical reduction after successful compaction, and intact retained data after a simulated compaction failure.
|
|
194
|
+
|
|
195
|
+
### Add the large-state regression
|
|
196
|
+
|
|
197
|
+
**Where**
|
|
198
|
+
|
|
199
|
+
- `test/server.test.ts`
|
|
200
|
+
- `test/workflow-runner-content.test.ts`
|
|
201
|
+
- the existing large-resume test fixture
|
|
202
|
+
|
|
203
|
+
**Change**
|
|
204
|
+
|
|
205
|
+
Create more than 2 MiB of distinct recorded session history. Run several resume, wait, continue, and terminal runner exchanges. Add a small unique item between exchanges.
|
|
206
|
+
|
|
207
|
+
Measure distinct blobs referenced by `worker_messages.result_hash`, every runner frame, total blob bytes, and SQLite pages. The test must prove that runner control replies contain only required execution state or content references. They must not copy complete session history into every reply.
|
|
208
|
+
|
|
209
|
+
**Check**
|
|
210
|
+
|
|
211
|
+
The regression fails when each reply includes the growing history. It passes when result growth follows new workflow state, and every runner frame stays inside its protocol limit.
|
|
212
|
+
|
|
213
|
+
### Cover retention behavior
|
|
214
|
+
|
|
215
|
+
**Where**
|
|
216
|
+
|
|
217
|
+
- `test/prune.test.ts`
|
|
218
|
+
- `test/server.test.ts`
|
|
219
|
+
- `test/server-view.test.ts`
|
|
220
|
+
- the existing state-prune client tests
|
|
221
|
+
|
|
222
|
+
**Change**
|
|
223
|
+
|
|
224
|
+
Test recent and expired trees, restart and continuation families, pending and sent terminal messages, open turns, waiting and parked work, every blocker, concurrent manual prune, automatic blob cleanup, viewer access, and resume.
|
|
225
|
+
|
|
226
|
+
Use temporary databases and deterministic executors. Automated tests must not call a model or touch the user's live state.
|
|
227
|
+
|
|
228
|
+
**Check**
|
|
229
|
+
|
|
230
|
+
Retained runs still resume and render. Expired delivered terminal trees disappear as one unit. Protected trees remain. Repeated cleanup has no additional effect.
|
|
231
|
+
|
|
232
|
+
### Update documentation
|
|
233
|
+
|
|
234
|
+
**Where**
|
|
235
|
+
|
|
236
|
+
- `docs/SQLITE_STATE.md`
|
|
237
|
+
- `docs/WORKFLOW_SERVER.md`
|
|
238
|
+
- `docs/workflows.md`
|
|
239
|
+
- CLI help where it describes state retention
|
|
240
|
+
|
|
241
|
+
**Change**
|
|
242
|
+
|
|
243
|
+
Document the contract in this plan. Remove the old statement that Pi Workflows never prunes at startup.
|
|
244
|
+
|
|
245
|
+
**Check**
|
|
246
|
+
|
|
247
|
+
Documentation matches the code constants and tests. It makes no hard size or file-shrink promise.
|
|
248
|
+
|
|
249
|
+
## Tests and checks
|
|
250
|
+
|
|
251
|
+
Run:
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
npm run check
|
|
255
|
+
npm run test:e2e
|
|
256
|
+
npx slophammer-ts@latest dry .
|
|
257
|
+
npx slophammer-ts@latest check . --only ts.dependency-boundaries-required
|
|
258
|
+
npx -y @simpledoc/simpledoc check
|
|
259
|
+
git diff --check
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
Automated tests use temporary directories and deterministic executors. They do not call a real model.
|
|
263
|
+
|
|
264
|
+
After these checks pass, run the separate installed-package E2E with the exact authenticated low-cost model `openai/gpt-5.6-luna`. Use temporary workflow state. Do not prune or seed the user's live database.
|
|
265
|
+
|
|
266
|
+
## Acceptance criteria
|
|
267
|
+
|
|
268
|
+
The work is complete when all of these statements are true:
|
|
269
|
+
|
|
270
|
+
- Terminal root-run trees remain for 30 days.
|
|
271
|
+
- Cleanup removes only complete eligible trees.
|
|
272
|
+
- Every active, pending, open, resumable, recording, queued, leased, unsettled, controller-owned, cross-linked, undelivered, or active-content case remains protected.
|
|
273
|
+
- Automatic cleanup runs without a second service or manual command.
|
|
274
|
+
- Manual prune remains backup-first.
|
|
275
|
+
- Repeated cleanup is safe and does not partly delete a lineage tree.
|
|
276
|
+
- Freed pages can be reused.
|
|
277
|
+
- Automatic `VACUUM` follows the 64 MiB, 20 percent, and idle rules.
|
|
278
|
+
- Retained runs still resume and render through bounded reads.
|
|
279
|
+
- Deleted runs return not found.
|
|
280
|
+
- The large-state regression proves that repeated runner replies do not copy growing session history.
|
|
281
|
+
- The SQLite schema stays at version 1 with no migration or compatibility path.
|
|
282
|
+
- Pi core, Pi session history, other repositories, external services, and the user's live workflow state remain unchanged.
|
|
283
|
+
- All repository checks and the separate real-model E2E pass.
|