@osolmaz/pi-workflows 0.16.4 → 0.16.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (173) hide show
  1. package/README.md +5 -5
  2. package/dist/builtins/autoimplement.workflow.d.ts +5 -1
  3. package/dist/builtins/autoimplement.workflow.js +29 -10
  4. package/dist/builtins/autoimplement.workflow.js.map +1 -1
  5. package/dist/client/client.js +3 -1
  6. package/dist/client/client.js.map +1 -1
  7. package/dist/client/protocol.d.ts +1 -1
  8. package/dist/client/protocol.js +18 -1
  9. package/dist/client/protocol.js.map +1 -1
  10. package/dist/client/view.d.ts +7 -5
  11. package/dist/extension/index.d.ts +2 -1
  12. package/dist/extension/index.js +115 -165
  13. package/dist/extension/index.js.map +1 -1
  14. package/dist/extension/recorder.d.ts +1 -1
  15. package/dist/extension/recorder.js +2 -2
  16. package/dist/extension/recorder.js.map +1 -1
  17. package/dist/extension/step-message.d.ts +1 -1
  18. package/dist/extension/step-message.js +4 -4
  19. package/dist/extension/step-message.js.map +1 -1
  20. package/dist/extension/workflow-message-coordinator.d.ts +12 -1
  21. package/dist/extension/workflow-message-coordinator.js +65 -54
  22. package/dist/extension/workflow-message-coordinator.js.map +1 -1
  23. package/dist/resource-managers/index.d.ts +1 -2
  24. package/dist/resource-managers/index.js +1 -2
  25. package/dist/resource-managers/index.js.map +1 -1
  26. package/dist/resource-managers/sqlite.d.ts +2 -295
  27. package/dist/resource-managers/sqlite.js +71 -1528
  28. package/dist/resource-managers/sqlite.js.map +1 -1
  29. package/dist/resource-managers/types.d.ts +2 -7
  30. package/dist/server/resource-runner-entry.js +2 -6
  31. package/dist/server/resource-runner-entry.js.map +1 -1
  32. package/dist/server/rpc-bridge.js +1 -1
  33. package/dist/server/rpc-bridge.js.map +1 -1
  34. package/dist/server/rpc-executor.js +2 -4
  35. package/dist/server/rpc-executor.js.map +1 -1
  36. package/dist/server/server-entry.js +3 -0
  37. package/dist/server/server-entry.js.map +1 -1
  38. package/dist/server/server.d.ts +22 -8
  39. package/dist/server/server.js +498 -543
  40. package/dist/server/server.js.map +1 -1
  41. package/dist/server/state.d.ts +16 -29
  42. package/dist/server/state.js +105 -319
  43. package/dist/server/state.js.map +1 -1
  44. package/dist/server/view.d.ts +4 -5
  45. package/dist/server/view.js +51 -51
  46. package/dist/server/view.js.map +1 -1
  47. package/dist/server/workflow-runner-entry.d.ts +1 -1
  48. package/dist/server/workflow-runner-entry.js +1 -49
  49. package/dist/server/workflow-runner-entry.js.map +1 -1
  50. package/dist/server/workflow-runner-protocol.d.ts +2 -7
  51. package/dist/server/workflow-runner-protocol.js +5 -10
  52. package/dist/server/workflow-runner-protocol.js.map +1 -1
  53. package/dist/server/workflow-runner-store.d.ts +5 -6
  54. package/dist/server/workflow-runner-store.js +7 -14
  55. package/dist/server/workflow-runner-store.js.map +1 -1
  56. package/dist/state/attempt-time.d.ts +21 -0
  57. package/dist/state/attempt-time.js +95 -0
  58. package/dist/state/attempt-time.js.map +1 -0
  59. package/dist/state/json.d.ts +2 -1
  60. package/dist/state/json.js.map +1 -1
  61. package/dist/state/project-store.d.ts +39 -0
  62. package/dist/state/project-store.js +154 -0
  63. package/dist/state/project-store.js.map +1 -0
  64. package/dist/state/prune.d.ts +28 -0
  65. package/dist/state/prune.js +278 -83
  66. package/dist/state/prune.js.map +1 -1
  67. package/dist/state/schema.js +23 -23
  68. package/dist/state/workflow-messages.d.ts +1 -1
  69. package/dist/state/workflow-messages.js +3 -1
  70. package/dist/state/workflow-messages.js.map +1 -1
  71. package/dist/viewer/backup.js +5 -0
  72. package/dist/viewer/backup.js.map +1 -1
  73. package/dist/workflows/diagnostics.d.ts +8 -0
  74. package/dist/workflows/diagnostics.js +61 -0
  75. package/dist/workflows/diagnostics.js.map +1 -0
  76. package/dist/workflows/engine.d.ts +2 -15
  77. package/dist/workflows/engine.js +125 -245
  78. package/dist/workflows/engine.js.map +1 -1
  79. package/dist/workflows/human-decision.d.ts +3 -7
  80. package/dist/workflows/human-decision.js +2 -56
  81. package/dist/workflows/human-decision.js.map +1 -1
  82. package/dist/workflows/index.d.ts +2 -1
  83. package/dist/workflows/index.js +1 -0
  84. package/dist/workflows/index.js.map +1 -1
  85. package/dist/workflows/queue.d.ts +267 -0
  86. package/dist/workflows/queue.js +1248 -0
  87. package/dist/workflows/queue.js.map +1 -0
  88. package/dist/workflows/requests.d.ts +47 -0
  89. package/dist/workflows/requests.js +158 -0
  90. package/dist/workflows/requests.js.map +1 -0
  91. package/dist/workflows/schema.js +2 -4
  92. package/dist/workflows/schema.js.map +1 -1
  93. package/dist/workflows/store.d.ts +11 -16
  94. package/dist/workflows/store.js +174 -169
  95. package/dist/workflows/store.js.map +1 -1
  96. package/dist/workflows/tool-input.d.ts +6 -9
  97. package/dist/workflows/tool-input.js +15 -12
  98. package/dist/workflows/tool-input.js.map +1 -1
  99. package/dist/workflows/transitions.d.ts +38 -0
  100. package/dist/workflows/transitions.js +276 -0
  101. package/dist/workflows/transitions.js.map +1 -0
  102. package/dist/workflows/types.d.ts +9 -42
  103. package/dist/workflows/workflow-message-content.d.ts +7 -1
  104. package/dist/workflows/workflow-message-content.js +22 -4
  105. package/dist/workflows/workflow-message-content.js.map +1 -1
  106. package/docs/2026-08-25-workflow-follow-ups.md +5 -5
  107. package/docs/2026-09-02-unify-workflow-messages-plan.md +2 -0
  108. package/docs/2026-09-04-workflow-run-state-plan.md +2 -0
  109. package/docs/2026-09-06-durable-execution-plan.md +176 -0
  110. package/docs/DEFERRED_TURNS.md +84 -105
  111. package/docs/HUMAN_DECISIONS.md +17 -19
  112. package/docs/HUMAN_DECISION_PRESENTATIONS.md +1 -1
  113. package/docs/SQLITE_STATE.md +87 -30
  114. package/docs/WORKFLOW_SERVER.md +50 -11
  115. package/docs/WORKFLOW_STEP_MESSAGES.md +72 -54
  116. package/docs/plans/2026-08-04-controller-runtime-plan.md +2 -0
  117. package/docs/plans/2026-08-21-deferred-turn-intents-plan.md +2 -0
  118. package/docs/plans/2026-08-27-workflow-terminal-restart-plan.md +2 -0
  119. package/docs/plans/2026-09-04-automatic-state-retention-plan.md +283 -0
  120. package/docs/workflows.md +109 -44
  121. package/examples/workflows/autoresearch.workflow.ts +0 -2
  122. package/examples/workflows/branch.workflow.ts +0 -4
  123. package/examples/workflows/echo.workflow.ts +0 -2
  124. package/examples/workflows/live-settings.workflow.ts +0 -2
  125. package/examples/workflows/two-turn.workflow.ts +0 -2
  126. package/herdr-plugin.toml +1 -1
  127. package/package.json +1 -1
  128. package/protocol/fixtures/client-v1.json +1 -1
  129. package/skills/pi-workflows/SKILL.md +8 -7
  130. package/src/builtins/autoimplement.workflow.ts +31 -10
  131. package/src/client/client.ts +3 -1
  132. package/src/client/protocol.ts +21 -1
  133. package/src/client/view.ts +9 -5
  134. package/src/extension/index.ts +117 -189
  135. package/src/extension/recorder.ts +2 -2
  136. package/src/extension/step-message.ts +4 -4
  137. package/src/extension/workflow-message-coordinator.ts +78 -50
  138. package/src/resource-managers/index.ts +1 -9
  139. package/src/resource-managers/sqlite.ts +330 -2655
  140. package/src/resource-managers/types.ts +2 -3
  141. package/src/server/resource-runner-entry.ts +2 -7
  142. package/src/server/rpc-bridge.ts +1 -1
  143. package/src/server/rpc-executor.ts +4 -9
  144. package/src/server/server-entry.ts +3 -0
  145. package/src/server/server.ts +568 -644
  146. package/src/server/state.ts +122 -466
  147. package/src/server/view.ts +56 -60
  148. package/src/server/workflow-runner-entry.ts +2 -55
  149. package/src/server/workflow-runner-protocol.ts +5 -15
  150. package/src/server/workflow-runner-store.ts +12 -27
  151. package/src/state/attempt-time.ts +122 -0
  152. package/src/state/json.ts +2 -1
  153. package/src/state/project-store.ts +218 -0
  154. package/src/state/prune.ts +408 -95
  155. package/src/state/schema.ts +23 -23
  156. package/src/state/workflow-messages.ts +4 -2
  157. package/src/viewer/backup.ts +7 -0
  158. package/src/workflows/diagnostics.ts +78 -0
  159. package/src/workflows/engine.ts +133 -291
  160. package/src/workflows/human-decision.ts +3 -81
  161. package/src/workflows/index.ts +8 -2
  162. package/src/workflows/queue.ts +2091 -0
  163. package/src/workflows/requests.ts +283 -0
  164. package/src/workflows/schema.ts +2 -6
  165. package/src/workflows/store.ts +223 -231
  166. package/src/workflows/tool-input.ts +20 -12
  167. package/src/workflows/transitions.ts +330 -0
  168. package/src/workflows/types.ts +9 -46
  169. package/src/workflows/workflow-message-content.ts +28 -4
  170. package/dist/resource-managers/runtime.d.ts +0 -59
  171. package/dist/resource-managers/runtime.js +0 -423
  172. package/dist/resource-managers/runtime.js.map +0 -1
  173. package/src/resource-managers/runtime.ts +0 -576
@@ -1,10 +1,10 @@
1
1
  # Workflow messages in Pi
2
2
 
3
- Status: this is the implemented workflow-message contract. [Unify workflow messages and restore hosted behavior](2026-09-02-unify-workflow-messages-plan.md) records the approved design and implementation plan.
3
+ This is the current workflow-message contract. The [durable execution plan](2026-09-06-durable-execution-plan.md) supersedes the earlier automatic terminal-turn and reminder behavior.
4
4
 
5
5
  ## Goal
6
6
 
7
- Pi Workflows must add several kinds of content to an origin Pi conversation. These include interactive step prompts, protected human decisions, passive notifications, terminal results, and follow-up prompts. Initial, reminder, and resumed prompts are one step-message kind with different display reasons.
7
+ Pi Workflows must add several kinds of content to an origin Pi conversation. These include interactive step prompts, protected human decisions, passive notifications, terminal results, and follow-up prompts. Initial and resumed prompts are one step-message kind with different display reasons.
8
8
 
9
9
  The server saves all of them as workflow messages. One extension component sends them through documented Pi APIs. Feature records continue to own workflow results, answers, settings, and timeouts.
10
10
 
@@ -16,13 +16,13 @@ A workflow message is content that Pi Workflows requires Pi to add to one conver
16
16
 
17
17
  The message kinds are:
18
18
 
19
- | Kind | Pi behavior | Purpose |
20
- | -------------- | ----------------------------------------------- | ------------------------------------------------ |
21
- | `step` | Custom message that starts a model turn | Initial, reminder, or resumed interactive prompt |
22
- | `decision` | Custom message that does not start a model turn | Protected choice for a person |
23
- | `notification` | Custom message that does not start a model turn | Passive workflow notice |
24
- | `terminal` | Custom message that starts a model turn | Final result and safe recovery choice |
25
- | `followUp` | Custom message that starts normal work | Work saved for after successful completion |
19
+ | Kind | Pi behavior | Purpose |
20
+ | -------------- | ------------------------------------------------ | ------------------------------------------ |
21
+ | `step` | Custom message that starts a model turn | Initial or resumed interactive prompt |
22
+ | `decision` | Custom message that does not start a model turn | Protected choice for a person |
23
+ | `notification` | Custom message that does not start a model turn | Passive workflow notice |
24
+ | `terminal` | Visible message that does not start a model turn | Recorded terminal result |
25
+ | `followUp` | Custom message that starts normal work | Work saved for after successful completion |
26
26
 
27
27
  The server stores one `WorkflowMessage` record before Pi can send it:
28
28
 
@@ -61,7 +61,7 @@ Every step message uses the custom type `pi-workflows-step`:
61
61
  export type WorkflowAgentStepMessageDetails = {
62
62
  schema: "pi-workflows.agent-step-message.v1";
63
63
  workflowMessageId: string;
64
- reason: "initial" | "reminder" | "resumed";
64
+ reason: "initial" | "resumed";
65
65
  contract: AgentStepContract;
66
66
  presentation?: {
67
67
  runTitle?: string;
@@ -84,7 +84,7 @@ pi.sendMessage(
84
84
 
85
85
  The renderer reads `details` and does not parse the prompt. It shows a compact summary by default and the complete prompt when expanded. If the renderer is unavailable, Pi still retains the custom message and its content.
86
86
 
87
- Submitted agent steps call the `workflow` tool. Assistant-message steps reply normally. Both forms keep the existing `agent` node and use `expectedOutput` to select the completion form.
87
+ Submitted agent steps call `workflow submit` or `workflow update` with the exact `requestId`. Assistant-message steps reply normally and are accepted only through the separate assistant-response path at `agent_settled`. Ordinary checkpoints accept `answer`; protected decisions accept only the verified human path or their declared timeout policy. Both forms keep the existing `agent` node and use `expectedOutput` to select the completion form.
88
88
 
89
89
  ## Engine boundary
90
90
 
@@ -103,10 +103,10 @@ A collapsed step card shows only the workflow identity and current work. For exa
103
103
  Checking the monitored target
104
104
  ```
105
105
 
106
- A reminder or resumed prompt adds a short label:
106
+ A resumed prompt adds a short label:
107
107
 
108
108
  ```text
109
- ↻ monitor › check · reminder
109
+ ↻ monitor › check · resumed
110
110
  Checking the monitored target
111
111
  ```
112
112
 
@@ -141,58 +141,75 @@ After Pi, the extension, or the server restarts, branch reporting runs before an
141
141
 
142
142
  ## Model-turn status
143
143
 
144
- `agent_start` has no message payload. The extension binds it through the current origin-session view. Turn binding ignores branch membership; only branch reporting and re-presentation inspect the active branch:
145
-
146
- - the latest sent step is open while its interaction remains pending and its run is not paused;
147
- - a terminal or follow-up message is open only until its first turn ends;
148
- - decisions and notifications never open a turn.
149
-
150
- A start against a closed message is rejected. Follow-up turns are reported for ordering but do not show the completed workflow as `running`.
151
-
152
- The extension creates one workflow turn ID at `agent_start` and keeps it through the matching `agent_end` and server reconnect. It buffers starts and ends until the session view and message receipt are ready.
153
-
154
- At `agent_end`, it derives `completed`, `aborted`, or `error` from the documented assistant messages. It reads response-entry evidence from `ctx.sessionManager.getBranch()`; the entry ID can be null. The server saves one immutable end result. A repeated report adopts it, and a stale turn ID cannot clear newer activity. A supervised runner can continue after accepted tool output while that Pi turn is still open. This normal continuation leaves the session capture recording until `agent_end`; it does not mark the capture as interrupted.
155
-
156
- The server applies the end and its workflow consequence in one transaction. An aborted turn sets the run pause and cancels the request's pending step messages; the interaction derives its paused state from the run. Resuming that submitted-output step atomically clears the pause, increments the interaction revision, and creates one new step message with reason `resumed`. That message starts a fresh Pi model turn. A protected decision does not start a model turn, so its revision and decision message do not change when its run resumes. A completed, recoverably failed, or proved-lost turn increments `unproductiveTurnEnds` only when the submitted-output step is pending, not paused, and has no accepted or validating submission. Values one and two create one step message with reason `reminder`; a value above two fails the attempt. A partial unique index enforces at most one pending step message for the request, regardless of reason. Acceptance, pause, cancel, timeout, and branch re-presentation cancel all pending step messages. A cancelled message did not start a turn and does not increment the counter.
157
-
158
- There is no activity heartbeat, refresh lease, or sequence counter. The server shows `running` from the matching start until the matching end. Server startup never marks a Pi turn lost. On `session_start`, only an idle-session branch report can close an open sent message with synthetic stop reason `lost`. Polling and time alone cannot create a reminder.
144
+ `agent_start` has no message payload. A locally delivered prompt binds its start
145
+ through the coordinator's saved message identity. Late binding and reconnect
146
+ also require the exact message in the active branch. A session view alone does
147
+ not prove that a message caused the current turn.
148
+
149
+ Only pending, unpaused agent requests and explicit follow-ups can open model
150
+ turns. Decisions, notifications, and terminal notices cannot. A stale start can
151
+ adopt an already closed result but cannot create new activity. An identity
152
+ mismatch is rejected before a turn changes.
153
+
154
+ `agent_end` records low-level stop information. The coordinator retains the
155
+ workflow turn through automatic Pi retries and waits for `agent_settled` before
156
+ submitting visible assistant text, recording the turn end, or delivering more
157
+ work. End reports carry `completed`, `aborted`, `error`, or `lost` and the exact
158
+ response-entry evidence when available. A delayed acknowledgment keeps the settled
159
+ response ID pending. Once message and turn ownership are confirmed, the coordinator
160
+ submits that exact response before it reports the end. Later events cannot replace
161
+ the pending turn or its response. Repeated reports adopt the saved result.
162
+
163
+ An aborted pending step pauses its run. Resume retains the request and attempt,
164
+ advances the request revision, and creates one resumed step message when needed.
165
+ A protected decision keeps its answer revision and decision message. A missing
166
+ submission stays pending: the host adds no reminder turn or hidden retry limit.
167
+
168
+ Execution status and Pi activity remain separate. A completed run remains
169
+ completed during reporting or follow-up work. A waiting run can have an active
170
+ origin-session turn without changing its execution status to running. Host
171
+ recovery closes active-time intervals at their last durable samples, not the Pi
172
+ turn itself. Only an idle-session branch report can prove an unended turn lost.
159
173
 
160
174
  ## Feature ownership
161
175
 
162
176
  The workflow message stores only Pi send facts. Other records remain authoritative:
163
177
 
164
- - interactive requests own step contracts, attempts, deadlines, validation, model submissions, and `unproductiveTurnEnds`;
165
- - human decisions own choices, verified answers, expiry, and continuation;
178
+ - interactive requests own exact response identity, kind, validation, and accepted submissions;
179
+ - node attempts and active intervals own execution history and the active-time budget;
180
+ - human decisions own choices, verified answers, and absolute expiry;
166
181
  - terminal runs own outcomes, reasons, restart lineage, and results;
167
182
  - notification nodes own their node results;
168
183
  - follow-up records own prompt source and authority;
169
184
  - settings records own current values and accepted changes.
170
185
 
171
- Submitted-output steps can use reminders. Assistant-message steps do not send a reminder after a visible response. An interrupted assistant-message step keeps its attempt ID and adopts a matching completed response from the active branch. A stale attempt or another branch remains invalid.
172
-
173
- Each source event creates its workflow message in the same SQLite transaction. The message cannot exist without its source fact, and a source fact cannot require a Pi message without the matching record.
186
+ Submitted and assistant-message steps keep their exact attempt while parked.
187
+ Recovery adopts accepted receipts and matching branch evidence; it does not use
188
+ the oldest pending request or accept text from another request or branch.
174
189
 
175
- ## Restored behavior
190
+ Requests and their required messages commit together. Terminal reporting can be
191
+ retried from recorded facts after execution settles; a reporting failure cannot
192
+ reverse execution or cancellation.
176
193
 
177
- The shared contract supports these features without separate send paths:
194
+ ## Terminal results and follow-ups
178
195
 
179
- - at most two reminder-reason step messages after model turns end without a valid submission;
180
- - one resumed-reason step message after a presented paused step resumes;
181
- - one terminal result and recovery turn for the final outcome of each interactive continuation chain;
182
- - safe restart with lineage, a limit of three, and repeated-failure protection;
183
- - ordered follow-up prompts after successful completion, terminal turn end, and release of later workflow reservations;
184
- - protected decisions in Pi and approved external channels;
185
- - passive notifications that do not start model turns;
186
- - terminal result retention while its message is pending or its first turn is open, and then for 60 seconds after that turn ends, in the widget and `piw`;
187
- - conversation recording linked to workflow attempts.
196
+ A terminal result is visible but does not start a model turn. It stays in the
197
+ origin-session view while pending and for 60 seconds after confirmed delivery.
198
+ An explicit restart targets the terminal run and revision and creates fresh
199
+ work without copying old steps, changed settings, approvals, or effects. It has
200
+ no terminal-turn prerequisite or hard-coded count limit.
188
201
 
189
- A terminal or follow-up message does not reopen the completed workflow. A slash-looking follow-up remains plain model input because `pi.sendMessage()` does not dispatch extension commands or expand prompt templates. External effects remain idempotent or explicitly ambiguous.
202
+ Explicit follow-ups wait for successful completion, terminal notice delivery,
203
+ prior follow-up settlement, and release of the session reservation. They remain
204
+ normal conversation work. Slash-looking text cannot dispatch an extension
205
+ command. External effects still require saved receipts or explicit recovery of
206
+ an ambiguous outcome. See [terminal workflow messages](DEFERRED_TURNS.md).
190
207
 
191
208
  ## Session recording
192
209
 
193
210
  The extension records workflow-related Pi events through a batched server client operation. It uses documented Pi events and does not read or edit Pi session files.
194
211
 
195
- The server deduplicates settled entries by Pi entry ID. It links attempts to their prompt, response, first, and last entries. A recording failure does not fail workflow execution.
212
+ The server deduplicates settled entries by Pi entry ID. It links attempts to their prompt, response, first, and last entries. The coordinator finalizes capture after the terminal notice has a confirmed entry on the current branch, without starting a model turn. Explicit follow-up turns have their own capture segment. A recording failure does not fail workflow execution or block the next delivery.
196
213
 
197
214
  ## Public API boundary
198
215
 
@@ -205,26 +222,27 @@ This design does not change Pi core, use private Pi APIs, or change Pi session s
205
222
  Tests must prove:
206
223
 
207
224
  - every message kind uses the one coordinator;
208
- - one workflow message ID creates at most one confirmed Pi entry and one automatic model turn;
225
+ - one workflow message ID adopts its confirmed Pi entry before another send;
226
+ - an automatic Pi retry cannot submit a partial response before the settled boundary;
209
227
  - a later manual turn uses a new workflow turn ID without creating another Pi entry;
210
228
  - two Pi processes that open one session cannot both send because one process-local coordinator epoch is active;
211
229
  - restart recovery reports the branch and adopts an existing entry before it sends;
212
230
  - branch absence is usable only when Pi is idle and has no pending input;
213
231
  - a crash after send leaves the message pending and cannot cause a resend before branch reporting;
214
232
  - messages remain in saved order without message-pointer deadlocks;
215
- - an early `agent_start` and `agent_end` wait for the message receipt and session view, then apply in order;
216
- - every matching model turn shows `running` for its full duration;
217
- - normal runner continuation leaves the active session capture open until the matching turn ends;
233
+ - an early `agent_start` and `agent_settled` wait for the message receipt and session view, then apply in order;
234
+ - model activity remains separate from durable execution status;
235
+ - normal runner progress leaves the active session capture open until the matching turn ends;
218
236
  - stale turn-end reports and starts against closed messages are rejected;
219
237
  - a manual turn cancels pending step messages that it supersedes;
220
238
  - a turn cannot bind to an interaction whose run is paused;
221
- - aborted turns pause without incrementing the unproductive-turn counter;
239
+ - aborted pending turns pause without adding reminder work;
222
240
  - resuming an aborted step creates one new resumed message and one fresh model turn;
223
241
  - resuming a protected decision keeps its answer revision and does not create a duplicate decision message;
224
242
  - server restart does not close a live Pi turn, while an idle-session branch report can close an unended turn as lost;
225
- - two reminder-reason steps are sent at most once and the next unproductive turn fails;
226
- - initial, reminder, and resumed prompts use the same step kind and differ only by reason;
227
- - terminal and follow-up messages each start only at their durable boundary;
243
+ - repeated missing submissions remain pending without automatic extra model turns;
244
+ - initial and resumed prompts use the same step kind and differ only by reason;
245
+ - terminal messages never start turns, and follow-ups start only at their declared boundary;
228
246
  - a branch switch creates one resumed-reason step only when that branch has no entry for the interaction;
229
247
  - a missing protected decision creates another decision message, not a step;
230
248
  - branch evidence changes a cancelled message to sent;
@@ -8,6 +8,8 @@ status: implemented
8
8
 
9
9
  # ResourceManager runtime plan
10
10
 
11
+ > Historical plan. The [durable execution plan](../2026-09-06-durable-execution-plan.md) supersedes its continuation, scheduling, and automatic model-turn choices. Use the current reference pages for the implemented contract.
12
+
11
13
  pi-workflows needs a controller mode for automation that spans repeated events, external state changes, and process restarts. The design in [RESOURCE_MANAGERS.md](../RESOURCE_MANAGERS.md) follows the Kubernetes controller pattern. Durable resources hold desired and observed state, events enqueue resource keys, and each reconciliation reads current facts before acting.
12
14
 
13
15
  The implementation keeps the graph engine focused on finite jobs. Controllers start and observe workflows through a child-run interface. Workflow graphs keep their finite execution model.
@@ -6,6 +6,8 @@ date: 2026-08-21
6
6
 
7
7
  # Guarantee one successor turn after workflow interruption
8
8
 
9
+ > Historical plan. The [durable execution plan](../2026-09-06-durable-execution-plan.md) supersedes its continuation, scheduling, and automatic model-turn choices. Use the current reference pages for the implemented contract.
10
+
9
11
  Pi Workflows can stop the active agent turn before the agent receives a tool result or can take its next action. This happens when an active workflow step is cancelled, times out, loses its queue claim, or is interrupted by a controller. The workflow can also return a successful start result and then fail asynchronously before it sends its first prompt. In that case, the user can see a UI error while the model still believes that the workflow started successfully.
10
12
 
11
13
  Add one general rule: an eligible workflow event creates one durable obligation for one later agent turn. The next normal workflow prompt or result presentation satisfies the obligation when one exists. Otherwise, Pi Workflows sends one factual fallback turn after the old turn settles.
@@ -6,6 +6,8 @@ date: 2026-08-27
6
6
 
7
7
  # Full plan
8
8
 
9
+ > Historical plan. The [durable execution plan](../2026-09-06-durable-execution-plan.md) supersedes its continuation, scheduling, and automatic model-turn choices. Use the current reference pages for the implemented contract.
10
+
9
11
  ## Goal
10
12
 
11
13
  After every top-level workflow run ends, give the model one normal successor turn.
@@ -0,0 +1,283 @@
1
+ ---
2
+ title: Add automatic workflow state retention
3
+ author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
4
+ date: 2026-09-04
5
+ status: approved
6
+ ---
7
+
8
+ # Add automatic workflow state retention
9
+
10
+ ## Goal
11
+
12
+ Pi Workflows will remove old completed workflow state automatically. It will keep active, resumable, pending, unsettled, and undelivered work.
13
+
14
+ The server will retain terminal run trees for 30 days. It will then remove safe expired trees and blobs that no retained record or active runner uses. The manual prune command will use the same safety rules.
15
+
16
+ A regression test will also protect the earlier runner fix. Repeated runner replies must not copy growing Pi session history and make the database grow by several gigabytes.
17
+
18
+ ## Observed problem
19
+
20
+ One old state database reached about 6.1 GB. The `blobs` table held about 6.06 GB across 101,355 blobs, which were almost all JSON. `worker_messages.result_hash` was the main reference path.
21
+
22
+ Older runner replies copied growing session and workflow history into new immutable result blobs. Small changes produced another large blob, so content hashing could not deduplicate the replies. [Keep large workflow history out of runner resume replies](2026-09-04-workflow-runner-resume-state-plan.md) fixed that interface by returning only the execution state a runner needs.
23
+
24
+ That fix stops the main amplification path. Completed workflow history still remains until a person runs `pi-workflows state prune`, so the database can still grow without a retention rule.
25
+
26
+ ## Selected design
27
+
28
+ Use the existing run-tree prune logic as the one cleanup engine. Automatic cleanup and manual prune will share selection, deletion, blob collection, and compaction rules.
29
+
30
+ A terminal root run and all its restart or continuation descendants remain for 30 days after `finished_at`. Every descendant must be terminal and older than the cutoff before the server can remove the tree. Protected state or a reference from outside the tree blocks removal.
31
+
32
+ The Workflow Server checks for cleanup after startup recovery and after workflow runners exit. It also schedules a daily check after each completed sweep. It starts cleanup only while normal server work is idle. One server process completes no more than one sweep in 24 hours. An interrupted sweep remains due. The next idle lifecycle trigger or a five-minute idle retry continues it.
33
+
34
+ Automatic cleanup creates no backup. Creating a new backup for every sweep would cause another unbounded store. The explicit manual prune command keeps its current dry-run and backup-first apply forms.
35
+
36
+ This design does not add size-based deletion. A clear time limit gives users predictable history. Recent or protected data can still be large, so this design does not promise a hard disk-size cap.
37
+
38
+ ## Retention contract
39
+
40
+ ### Eligible run trees
41
+
42
+ A run tree is eligible only when all of these facts are true:
43
+
44
+ - The root run and every descendant have status `completed`, `failed`, `timed_out`, or `cancelled`.
45
+ - Every run in the tree has `finished_at` earlier than the 30-day cutoff.
46
+ - No row outside the tree depends on a row inside it.
47
+ - No protected work belongs to the tree.
48
+
49
+ The cleanup transaction rechecks the exact root tree before deletion. A changed tree is skipped.
50
+
51
+ ### Protected work
52
+
53
+ Automatic cleanup must keep a tree when it contains or owns any of this work:
54
+
55
+ - a waiting or parked run
56
+ - a queued, starting, running, or parked queue row
57
+ - a pending workflow message
58
+ - an open workflow turn
59
+ - a pending interaction or human decision
60
+ - a recording session segment
61
+ - a queued follow-up
62
+ - an active lease
63
+ - a pending, applying, or ambiguous effect
64
+ - controller ownership or a managed resource reference
65
+ - a continuation or step reference from outside the tree
66
+ - an active runner content hash
67
+ - a resumable checkpoint
68
+ - an undelivered terminal result
69
+
70
+ Unknown or conflicting ownership blocks deletion. Cleanup must fail closed.
71
+
72
+ ### Automatic scheduling
73
+
74
+ The server requests one automatic sweep after recovery and after a workflow runner exits. Overlapping requests join the same in-process task.
75
+
76
+ A sweep starts only when there is no active or pending workflow runner, resource-manager runner, state-maintenance command, or server shutdown. It deletes one complete root tree in one transaction, yields, and checks for new work before it selects another tree.
77
+
78
+ A completed sweep starts a 24-hour in-process interval and schedules the next daily check. A sweep that stops because new work appeared remains due. The next idle lifecycle trigger or a five-minute idle retry continues it. A new request during the completed-sweep interval waits until that interval ends.
79
+
80
+ The cleanup scheduler is part of the existing server process. It does not add a service, scheduler process, or second writer.
81
+
82
+ ### Blob cleanup
83
+
84
+ After run deletion, the state layer removes only blobs with no database foreign-key reference and no active runner reference. The existing schema scan remains the source for blob references, so a future blob foreign key is protected automatically.
85
+
86
+ A repeated sweep is safe. A second sweep finds no extra rows from work that was already removed.
87
+
88
+ ### Space reuse and file compaction
89
+
90
+ After logical deletion, the server truncates the WAL while idle and reads SQLite `page_count`, `freelist_count`, and `page_size`. SQLite can reuse free pages even when the main file does not shrink.
91
+
92
+ Automatic cleanup runs `VACUUM` only when all of these facts are true:
93
+
94
+ - normal server work is still idle
95
+ - at least 64 MiB is reclaimable
96
+ - at least 20 percent of database pages are free
97
+
98
+ A skipped or failed `VACUUM` does not undo committed logical deletion. The server reports the outcome and leaves the free pages available for reuse. It does not claim that the file shrank unless measurement proves it.
99
+
100
+ Manual prune keeps its current backup-first full compaction and integrity checks.
101
+
102
+ ### Failure handling
103
+
104
+ A cleanup error does not fail a workflow or stop the server. The server records one clear diagnostic and waits five minutes before another safe attempt. It does not retry in a tight loop.
105
+
106
+ Cleanup is atomic for one complete root tree. An interruption can leave later trees for another sweep, but it cannot leave half of one lineage tree deleted.
107
+
108
+ ## Public behavior
109
+
110
+ Retained runs keep their current resume, viewer, content-reference, and terminal-result behavior. The viewer continues to read bounded pages. Pi Workflows does not edit Pi session history.
111
+
112
+ After an expired tree is removed, it no longer appears in run lists. A direct run view returns not found.
113
+
114
+ The manual commands remain:
115
+
116
+ ```bash
117
+ pi-workflows state prune --before <timestamp> --dry-run
118
+ pi-workflows state prune --before <timestamp> --backup <absolute-path> --apply
119
+ ```
120
+
121
+ Automatic cleanup adds no client protocol operation. The SQLite schema name and version remain `pi-workflows-state` version 1. This change adds no migration, compatibility reader, second state path, fallback, feature flag, archive, or external resource.
122
+
123
+ ## Implementation
124
+
125
+ ### Share the cleanup engine
126
+
127
+ **Where**
128
+
129
+ - `src/state/prune.ts`
130
+ - version-1 invariants in `src/state/schema.ts`
131
+
132
+ **Change**
133
+
134
+ Split the current prune work into shared run-tree selection, exact-tree deletion, unreferenced-blob collection, page measurement, and compaction operations. Add every protected-state check from this plan.
135
+
136
+ Automatic deletion must recheck and delete one root tree in the same transaction. Manual prune keeps its backup and complete-selection recheck.
137
+
138
+ **Check**
139
+
140
+ State tests prove that eligible trees are removed as one unit. Each protected state keeps its whole tree. Foreign-key and integrity checks pass after deletion.
141
+
142
+ ### Add automatic cleanup
143
+
144
+ **Where**
145
+
146
+ - `src/state/prune.ts`
147
+ - `src/server/server.ts`
148
+
149
+ **Change**
150
+
151
+ Add an automatic entry point with a cutoff of the current time minus 30 days. It uses the shared cleanup engine without creating a backup.
152
+
153
+ Process one complete root tree per transaction. Yield between trees. Stop when normal server work appears and leave the sweep due.
154
+
155
+ **Check**
156
+
157
+ Automatic and manual cleanup select the same eligible trees. Automatic cleanup creates no backup. A stopped sweep continues later without duplicate deletion.
158
+
159
+ ### Schedule cleanup from the server lifecycle
160
+
161
+ **Where**
162
+
163
+ - lifecycle fields in `src/server/server.ts`
164
+ - `WorkflowServer.start()`
165
+ - server shutdown
166
+ - the active-run exit path after the server removes the run from `activeRuns`
167
+
168
+ **Change**
169
+
170
+ Request cleanup after startup recovery and after runner exit. Schedule a daily check after each completed sweep. Coalesce overlapping requests. Enforce the idle check, five-minute idle retry, and 24-hour completed-sweep interval.
171
+
172
+ A failed sweep logs one bounded error and waits for a later trigger. Shutdown starts no new sweep and settles any current scheduling task safely.
173
+
174
+ **Check**
175
+
176
+ Server tests use controlled scheduler timestamps and temporary databases to cover startup, runner exit, coalescing, the 24-hour interval, interruption, clean shutdown, and one injected failure.
177
+
178
+ ### Reuse pages and compact when worthwhile
179
+
180
+ **Where**
181
+
182
+ - database-size helpers in `src/state/prune.ts`
183
+ - `test/prune.test.ts`
184
+
185
+ **Change**
186
+
187
+ Measure free SQLite pages after deletion. Truncate the WAL while idle. Run automatic `VACUUM` only above the 64 MiB and 20 percent thresholds.
188
+
189
+ Keep committed deletion when compaction is skipped or fails. Report logical deletion and physical file size separately.
190
+
191
+ **Check**
192
+
193
+ Tests prove page reuse below the thresholds, no early automatic `VACUUM`, physical reduction after successful compaction, and intact retained data after a simulated compaction failure.
194
+
195
+ ### Add the large-state regression
196
+
197
+ **Where**
198
+
199
+ - `test/server.test.ts`
200
+ - `test/workflow-runner-content.test.ts`
201
+ - the existing large-resume test fixture
202
+
203
+ **Change**
204
+
205
+ Create more than 2 MiB of distinct recorded session history. Run several resume, wait, continue, and terminal runner exchanges. Add a small unique item between exchanges.
206
+
207
+ Measure distinct blobs referenced by `worker_messages.result_hash`, every runner frame, total blob bytes, and SQLite pages. The test must prove that runner control replies contain only required execution state or content references. They must not copy complete session history into every reply.
208
+
209
+ **Check**
210
+
211
+ The regression fails when each reply includes the growing history. It passes when result growth follows new workflow state, and every runner frame stays inside its protocol limit.
212
+
213
+ ### Cover retention behavior
214
+
215
+ **Where**
216
+
217
+ - `test/prune.test.ts`
218
+ - `test/server.test.ts`
219
+ - `test/server-view.test.ts`
220
+ - the existing state-prune client tests
221
+
222
+ **Change**
223
+
224
+ Test recent and expired trees, restart and continuation families, pending and sent terminal messages, open turns, waiting and parked work, every blocker, concurrent manual prune, automatic blob cleanup, viewer access, and resume.
225
+
226
+ Use temporary databases and deterministic executors. Automated tests must not call a model or touch the user's live state.
227
+
228
+ **Check**
229
+
230
+ Retained runs still resume and render. Expired delivered terminal trees disappear as one unit. Protected trees remain. Repeated cleanup has no additional effect.
231
+
232
+ ### Update documentation
233
+
234
+ **Where**
235
+
236
+ - `docs/SQLITE_STATE.md`
237
+ - `docs/WORKFLOW_SERVER.md`
238
+ - `docs/workflows.md`
239
+ - CLI help where it describes state retention
240
+
241
+ **Change**
242
+
243
+ Document the contract in this plan. Remove the old statement that Pi Workflows never prunes at startup.
244
+
245
+ **Check**
246
+
247
+ Documentation matches the code constants and tests. It makes no hard size or file-shrink promise.
248
+
249
+ ## Tests and checks
250
+
251
+ Run:
252
+
253
+ ```bash
254
+ npm run check
255
+ npm run test:e2e
256
+ npx slophammer-ts@latest dry .
257
+ npx slophammer-ts@latest check . --only ts.dependency-boundaries-required
258
+ npx -y @simpledoc/simpledoc check
259
+ git diff --check
260
+ ```
261
+
262
+ Automated tests use temporary directories and deterministic executors. They do not call a real model.
263
+
264
+ After these checks pass, run the separate installed-package E2E with the exact authenticated low-cost model `openai/gpt-5.6-luna`. Use temporary workflow state. Do not prune or seed the user's live database.
265
+
266
+ ## Acceptance criteria
267
+
268
+ The work is complete when all of these statements are true:
269
+
270
+ - Terminal root-run trees remain for 30 days.
271
+ - Cleanup removes only complete eligible trees.
272
+ - Every active, pending, open, resumable, recording, queued, leased, unsettled, controller-owned, cross-linked, undelivered, or active-content case remains protected.
273
+ - Automatic cleanup runs without a second service or manual command.
274
+ - Manual prune remains backup-first.
275
+ - Repeated cleanup is safe and does not partly delete a lineage tree.
276
+ - Freed pages can be reused.
277
+ - Automatic `VACUUM` follows the 64 MiB, 20 percent, and idle rules.
278
+ - Retained runs still resume and render through bounded reads.
279
+ - Deleted runs return not found.
280
+ - The large-state regression proves that repeated runner replies do not copy growing session history.
281
+ - The SQLite schema stays at version 1 with no migration or compatibility path.
282
+ - Pi core, Pi session history, other repositories, external services, and the user's live workflow state remain unchanged.
283
+ - All repository checks and the separate real-model E2E pass.