zidane 7.0.14 → 7.0.16

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (124) hide show
  1. package/README.md +18 -9
  2. package/dist/{acp-CZpX91YE.js → acp-Bw2b_kka.js} +3 -3
  3. package/dist/{acp-CZpX91YE.js.map → acp-Bw2b_kka.js.map} +1 -1
  4. package/dist/acp-cli.js +4 -4
  5. package/dist/acp.d.ts +1 -1
  6. package/dist/acp.js +1 -1
  7. package/dist/{agent-B1jsFaDq.js → agent-B_uQo4f1.js} +649 -48
  8. package/dist/agent-B_uQo4f1.js.map +1 -0
  9. package/dist/agent.d.ts +1 -1
  10. package/dist/agent.js +1 -1
  11. package/dist/{anthropic-Da39cE-d.js → anthropic-QbHqqTkV.js} +2 -2
  12. package/dist/{anthropic-Da39cE-d.js.map → anthropic-QbHqqTkV.js.map} +1 -1
  13. package/dist/{approvals-S100xf1d.js → approvals-B9aIQJRQ.js} +287 -45
  14. package/dist/approvals-B9aIQJRQ.js.map +1 -0
  15. package/dist/artifacts-CV2hUKnL.js +112 -0
  16. package/dist/artifacts-CV2hUKnL.js.map +1 -0
  17. package/dist/{auth-DfEWnkC7.js → auth-plw6npda.js} +19 -7
  18. package/dist/auth-plw6npda.js.map +1 -0
  19. package/dist/chat/pure.d.ts +3 -3
  20. package/dist/chat/pure.js +1 -1
  21. package/dist/chat.d.ts +6 -6
  22. package/dist/chat.js +5 -5
  23. package/dist/contexts/daytona.d.ts +1 -1
  24. package/dist/contexts/e2b.d.ts +1 -1
  25. package/dist/daemon-cli.js +1 -1
  26. package/dist/eval.d.ts +1 -1
  27. package/dist/eval.js +1 -1
  28. package/dist/extensions.d.ts +2 -2
  29. package/dist/extensions.js +2 -2
  30. package/dist/headless.d.ts +1 -1
  31. package/dist/headless.js +2 -2
  32. package/dist/{index-DFGECPfm.d.ts → index-Dc_zYpEu.d.ts} +351 -2
  33. package/dist/index-Dc_zYpEu.d.ts.map +1 -0
  34. package/dist/{index-DP7xYXr9.d.ts → index-Df2CH6bn.d.ts} +92 -7
  35. package/dist/index-Df2CH6bn.d.ts.map +1 -0
  36. package/dist/index.d.ts +2 -2
  37. package/dist/index.js +7 -6
  38. package/dist/{ipc-BLLscoIl.js → ipc-T_pPnHkp.js} +2 -2
  39. package/dist/{ipc-BLLscoIl.js.map → ipc-T_pPnHkp.js.map} +1 -1
  40. package/dist/ipc.d.ts +54 -6
  41. package/dist/ipc.d.ts.map +1 -1
  42. package/dist/ipc.js +3 -3
  43. package/dist/mcp.d.ts +1 -1
  44. package/dist/{oauth-C4GvY_p1.js → oauth-EIBUuSDK.js} +2 -2
  45. package/dist/{oauth-C4GvY_p1.js.map → oauth-EIBUuSDK.js.map} +1 -1
  46. package/dist/output/stream-json.d.ts +1 -1
  47. package/dist/output/terminal.d.ts +1 -1
  48. package/dist/plugins.d.ts +1 -1
  49. package/dist/presets.d.ts +2 -2
  50. package/dist/presets.js +2 -2
  51. package/dist/providers/anthropic.d.ts +1 -1
  52. package/dist/providers/anthropic.js +1 -1
  53. package/dist/providers/arcee.d.ts +1 -1
  54. package/dist/providers/baseten.d.ts +1 -1
  55. package/dist/providers/cerebras.d.ts +1 -1
  56. package/dist/providers/local.d.ts +1 -1
  57. package/dist/providers/openai-compat.d.ts +1 -1
  58. package/dist/providers/openai.d.ts +1 -1
  59. package/dist/providers/openai.js +1 -1
  60. package/dist/providers/openrouter.d.ts +1 -1
  61. package/dist/providers/xai.d.ts +1 -1
  62. package/dist/providers/xai.js +2 -2
  63. package/dist/{providers-BYgyPb15.js → providers-CoK1rZ1p.js} +2 -2
  64. package/dist/{providers-BYgyPb15.js.map → providers-CoK1rZ1p.js.map} +1 -1
  65. package/dist/providers.d.ts +1 -1
  66. package/dist/providers.js +3 -3
  67. package/dist/restate.d.ts +1 -1
  68. package/dist/restate.js +1 -1
  69. package/dist/{server-C8gULJ0d.js → server-DXM8ZbGi.js} +64 -6
  70. package/dist/server-DXM8ZbGi.js.map +1 -0
  71. package/dist/session/sqlite.d.ts +1 -1
  72. package/dist/session/sqlite.d.ts.map +1 -1
  73. package/dist/session/sqlite.js +117 -1
  74. package/dist/session/sqlite.js.map +1 -1
  75. package/dist/{session-K3icbPH9.js → session-DPZnncaQ.js} +60 -1
  76. package/dist/session-DPZnncaQ.js.map +1 -0
  77. package/dist/session.d.ts +2 -2
  78. package/dist/session.js +3 -2
  79. package/dist/skills.d.ts +1 -1
  80. package/dist/{tool-formatters-C-s_-u0v.d.ts → tool-formatters-BgdOYAy9.d.ts} +2 -2
  81. package/dist/{tool-formatters-C-s_-u0v.d.ts.map → tool-formatters-BgdOYAy9.d.ts.map} +1 -1
  82. package/dist/tools/fetch-url.d.ts +1 -1
  83. package/dist/tools/web-search.d.ts +1 -1
  84. package/dist/tools.d.ts +2 -2
  85. package/dist/tools.js +2 -2
  86. package/dist/{transcript-anchors-D1ToF-EW.d.ts → transcript-anchors-CtnwuQJM.d.ts} +3 -3
  87. package/dist/transcript-anchors-CtnwuQJM.d.ts.map +1 -0
  88. package/dist/tui.d.ts +3 -3
  89. package/dist/tui.d.ts.map +1 -1
  90. package/dist/tui.js +576 -102
  91. package/dist/tui.js.map +1 -1
  92. package/dist/{turn-operations-CZLsNgw-.js → turn-operations-DkJ1R0p8.js} +4 -3
  93. package/dist/turn-operations-DkJ1R0p8.js.map +1 -0
  94. package/dist/{turn-operations-Bp5-BAki.d.ts → turn-operations-khAvr7KJ.d.ts} +5 -3
  95. package/dist/turn-operations-khAvr7KJ.d.ts.map +1 -0
  96. package/dist/types.d.ts +1 -1
  97. package/dist/{xai-DILdFGzN.js → xai-CazjStOK.js} +3 -3
  98. package/dist/{xai-DILdFGzN.js.map → xai-CazjStOK.js.map} +1 -1
  99. package/docs/ACP.md +6 -0
  100. package/docs/ARCHITECTURE.md +179 -356
  101. package/docs/ARTIFACTS.md +130 -0
  102. package/docs/CHAT.md +426 -1312
  103. package/docs/EXECUTION_CONTEXT.md +46 -19
  104. package/docs/EXTENSIONS.md +234 -584
  105. package/docs/HEADLESS.md +5 -0
  106. package/docs/INTERACTIONS.md +21 -22
  107. package/docs/IPC.md +318 -335
  108. package/docs/PLUGINS.md +2 -0
  109. package/docs/README.md +57 -0
  110. package/docs/RESTATE.md +199 -491
  111. package/docs/RUN_IN_BACKGROUND.md +107 -558
  112. package/docs/SKILL.md +171 -238
  113. package/docs/TUI.md +213 -591
  114. package/package.json +1 -1
  115. package/dist/agent-B1jsFaDq.js.map +0 -1
  116. package/dist/approvals-S100xf1d.js.map +0 -1
  117. package/dist/auth-DfEWnkC7.js.map +0 -1
  118. package/dist/index-DFGECPfm.d.ts.map +0 -1
  119. package/dist/index-DP7xYXr9.d.ts.map +0 -1
  120. package/dist/server-C8gULJ0d.js.map +0 -1
  121. package/dist/session-K3icbPH9.js.map +0 -1
  122. package/dist/transcript-anchors-D1ToF-EW.d.ts.map +0 -1
  123. package/dist/turn-operations-Bp5-BAki.d.ts.map +0 -1
  124. package/dist/turn-operations-CZLsNgw-.js.map +0 -1
package/docs/RESTATE.md CHANGED
@@ -1,44 +1,37 @@
1
1
  # Zidane × Restate — durable agents
2
2
 
3
- `zidane/restate` is an **optional** adapter that runs the agent loop inside a [Restate](https://docs.restate.dev/) handler so every LLM call and completed tool execution becomes a journaled side effect. Crash mid-run → resume from the journal without re-billing tokens or re-running persisted tool calls.
3
+ `zidane/restate` is an **optional** adapter that runs the agent loop inside a [Restate](https://docs.restate.dev/) handler so every LLM call and completed tool execution becomes a journaled side effect: crash mid-run and resume from the journal without re-billing tokens or re-running persisted tools. It's a thin layer over the existing agent surface — you opt in by wiring wrappers, nothing forks.
4
4
 
5
- It's a thin layer over the existing agent surface — no fork, no separate runtime. The core harness has zero dependency on Restate; you opt in by wiring four wrappers.
5
+ Entry points: wire the wrappers directly into your own Restate handler (see `examples/restate/agent.ts`), or use `bun start --restate` as a local demo/validation runner around the same adapter.
6
6
 
7
- ```
8
- zidane (core)
9
- └── zidane/restate ← this adapter (Bun + Node, structurally typed against @restatedev/restate-sdk)
10
- ```
11
-
12
- A production-style runnable example lives at `examples/restate/agent.ts`. The `bun start --restate` CLI path is a local/demo runner around the same adapter; it is useful for exercising the flow against a local Restate runtime, but production services should wire `zidane/restate` inside their own Restate handlers.
7
+ Contents: [Wrappers](#wrappers) · [Local CLI runner](#local-cli-runner) · [How replay works](#how-replay-works) · [Performance](#performance-minimizing-suspensions) · [Wrapper details](#wrapper-details) · [Subagents](#subagents) · [Parallel subagents](#parallel-subagents) · [Durable tool state](#durable-tool-state) · [Recommended AgentBehavior](#recommended-agentbehavior-for-restate-deployments) · [MCP tools](#mcp-tools) · [Background jobs](#long-running-background-jobs-durable-contexts) · [Edge deployments](#edge--long-lived-deployments) · [Failure modes](#failure-modes) · [Tests](#tests)
13
8
 
14
- ## When to use it (and when not to)
9
+ ## When to use it
15
10
 
16
- **Use Restate when** you need a server-side, long-running agent that must survive process restarts mid-run: an unattended job, a webhook-driven worker, a multi-hour pipeline that calls expensive providers and writes to external systems. The journal turns crash recovery from "replay from a session snapshot" into "replay from the last completed side effect, mid-turn".
11
+ **Use it** for a server-side, long-running agent that must survive process restarts mid-run: unattended jobs, webhook-driven workers, multi-hour pipelines against expensive providers.
17
12
 
18
- **Skip Restate when** you're running:
19
- - A normal interactive TUI / CLI session (the user owns process lifetime; sessions are the recovery story).
20
- - A request/response API where the agent finishes inside the request lifetime.
21
- - A worker where crash → start-over is acceptable (sessions still persist conversation; you just lose the in-flight turn).
22
-
23
- Restate adds a deployment surface (runtime + SDK + journal storage). Don't pay for it unless mid-run crashes are an actual concern.
13
+ **Skip it** for an interactive TUI/CLI session (the user owns process lifetime), a request/response API where the agent finishes inside the request, or a worker where crash-and-restart is acceptable. Restate adds a deployment surface (runtime, SDK, journal storage); don't pay for it unless mid-run crashes are an actual concern.
24
14
 
25
15
  ## Wrappers
26
16
 
27
- Adapter helpers — pick the ones you need. Pass-through for everything else (`Provider.formatTools`, `ToolDef.spec`, etc.) so the rest of Zidane is unchanged.
17
+ Pick what you need; everything else on `Provider` / `ToolDef` passes through unchanged.
28
18
 
29
- | Helper | Wraps / configures | What it does |
19
+ | Helper | Wraps | Does |
30
20
  |---|---|---|
31
- | `restateBehavior()` | `AgentBehavior` | Strict Restate defaults: sequential tools, background tasks disabled for non-durable contexts, cache on, Restate control errors re-thrown |
32
- | `restateClock(ctx)` | `AgentClock` | `now()`, `randomUUID()` — timestamps, run IDs, and fallback turn IDs stable across replays |
33
- | `restateProvider(provider, ctx)` | `Provider.stream` | Each LLM turn's `TurnResult` — replay returns cached value, no re-billing |
34
- | `restateTool(tool, ctx)` / `wrapAgentTools(tools, ctx)` | `ToolDef.execute` | Each completed tool invocation's result — replay skips re-running persisted tools |
35
- | `restateMcpToolWrap(ctx)` | `mcpToolWrap` hook (DISCOVERED MCP tools) | The MCP analogue of `wrapAgentTools`: journals every discovered MCP tool's result (soft-error-preserving). MCP tools are connected internally, past the public tool-wrapping seam, so the agent applies this on your behalf |
36
- | `restateToolBatchExecutor({ promises })` | Tool scheduler | Opt-in Restate-native parallel join for concurrency-safe tool fleets |
37
- | `restateSessionStore(ctx)` | `SessionStore` | Session header + turns + runs as virtual-object K/V state, header timestamps journal-sourced |
38
- | `restateToolStateStore(ctx)` | `AgentOptions.toolState` | Backs `ToolContext.state` — durable key/value state tools keep across resumes and invocations (see [Durable tool state](#durable-tool-state)) |
39
- | `restateAwakeableTool(options)` | `ToolDef` | Tool that parks on a Restate awakeable until an external resumer resolves it (human-in-the-loop waits) |
40
- | `restateWaitBackground(options)` | `ExecutionContext.waitBackground` | Packages the race-safe durable wait for a background task's exit (mint → register-before-probe → already-exited fast-path → park → settle); the host supplies only persistence callbacks |
41
- | `awakeableWithTimeout(ctx, opts)` | awakeable + `ctx.sleep` | Awakeable raced against a durable timer — resolves the resumer payload, or `null` on deadline. The low-level building block `restateWaitBackground` is built on |
21
+ | `restateBehavior(overrides?)` | `AgentBehavior` | Strict defaults: sequential tools, background tasks off for non-durable contexts, cache on, Restate control errors re-thrown |
22
+ | `restateClock(ctx, opts?)` | `AgentClock` | `now()` / `randomUUID()` stable across replays |
23
+ | `restateProvider(provider, ctx, opts?)` | `Provider.stream` | Journals each turn's `TurnResult`; replay skips the model call |
24
+ | `restateTool(tool, ctx, opts?)` / `wrapAgentTools(tools, ctx, opts?)` | `ToolDef.execute` | Journals each tool result; replay skips re-execution |
25
+ | `restateMcpToolWrap(ctx, opts?)` | `mcpToolWrap` hook | Same journaling for discovered MCP tools (wired automatically via `createAgent({ mcpToolWrap })`) |
26
+ | `restateToolBatchExecutor({ promises })` | `AgentBehavior.toolBatchExecutor` | Opt-in durable parallel join for concurrency-safe tool fleets |
27
+ | `restateSessionStore(ctx, opts?)` | `SessionStore` | Session header/turns/runs as virtual-object K/V state |
28
+ | `restateToolStateStore(ctx, opts?)` | `AgentOptions.toolState` | Durable `ToolContext.state` — see [Durable tool state](#durable-tool-state) |
29
+ | `restateAwakeableTool(opts)` | `ToolDef` | Tool that parks on an awakeable until an external resumer resolves it (human-in-the-loop) |
30
+ | `restateSpawnTool(opts)` | `ToolDef` | Dispatches subagent children to their own invocations — see [Parallel subagents](#parallel-subagents) |
31
+ | `restateWaitBackground(opts)` | `ExecutionContext.waitBackground` | Race-safe durable wait for a background task's exit |
32
+ | `awakeableWithTimeout(ctx, opts)` | awakeable + `ctx.sleep` | Low-level building block `restateWaitBackground` is built on |
33
+ | `createReplayTracker(opts?)` / `replayAwareAbort(tracker, abort)` | — | Replay-liveness probe + abort deferral — see [replay-safe abort](#replay-safe-abort) |
34
+ | `shouldRethrowRestateControlError(err)` / `deterministicUuid(seed)` | — | Structural error classifier / pure UUID-from-seed helper used internally, exported for custom wrapping |
42
35
 
43
36
  ### Minimal wiring
44
37
 
@@ -54,21 +47,15 @@ const zidaneAgent = restate.object({
54
47
  name: 'zidane-agent',
55
48
  handlers: {
56
49
  async run(ctx, { prompt }: { prompt: string }) {
57
- const session = await createSession({
58
- id: ctx.key,
59
- store: restateSessionStore(ctx),
60
- })
61
-
50
+ const session = await createSession({ id: ctx.key, store: restateSessionStore(ctx) })
62
51
  const agent = createAgent({
63
52
  provider: restateProvider(anthropic({ defaultModel: 'claude-sonnet-4-5' }), ctx),
64
53
  tools: wrapAgentTools({ shell }, ctx),
65
54
  clock: restateClock(ctx),
66
55
  session,
67
- // Optional — only if your tools use `ctx.state`.
68
- toolState: restateToolStateStore(ctx),
56
+ toolState: restateToolStateStore(ctx), // optional — only if tools use `ctx.state`
69
57
  behavior: restateBehavior(),
70
58
  })
71
-
72
59
  return await agent.run({ prompt })
73
60
  },
74
61
  },
@@ -77,230 +64,133 @@ const zidaneAgent = restate.object({
77
64
  restate.serve({ services: [zidaneAgent] })
78
65
  ```
79
66
 
80
- ### Tuned for fewest suspensions
81
-
82
- On Lambda / sandbox-reattach hosts, share one coalescing clock across the agent and the store, raise the safe-tool concurrency, and exclude deterministic MCP reads from journaling. See [Performance: minimizing suspensions](#performance-minimizing-suspensions) for the full rationale.
83
-
84
- ```ts
85
- const clock = restateClock(ctx, { coalesce: true })
86
-
87
- const session = await createSession({
88
- id: ctx.key,
89
- store: restateSessionStore(ctx, { clock }), // store timestamps ride the same anchor
90
- })
91
-
92
- const agent = createAgent({
93
- provider: restateProvider(anthropic({ defaultModel: 'claude-sonnet-4-5' }), ctx),
94
- tools: wrapAgentTools({ shell }, ctx),
95
- clock,
96
- session,
97
- behavior: restateBehavior({ maxConcurrentTools: 8 }), // batch concurrency-safe tools
98
- mcpToolWrap: restateMcpToolWrap(ctx, { exclude: ['mcp_docs_search'] }),
99
- })
100
- ```
101
-
102
- ### Local CLI runner
67
+ ## Local CLI runner
103
68
 
104
- `bun start --restate` starts a local Restate endpoint in the current process, waits for the Admin API, probes the local service port, registers the deployment, and invokes the `zidane-start-agent` virtual object through ingress. It is intentionally a convenience runner for local validation and demos.
69
+ `bun start --restate` boots a local Restate endpoint in-process, waits for the Admin API, probes the local service port, registers the deployment, and invokes the `zidane-start-agent` virtual object through ingress. It's a convenience runner for local validation and demos, not a production deployment shape.
105
70
 
106
71
  ```bash
107
72
  bun run restate:fixture:up
108
- bun start \
109
- --restate \
110
- --prompt "exercise the durable runner" \
111
- --session-id demo-1
73
+ bun start --restate --prompt "exercise the durable runner" --session-id demo-1
112
74
  ```
113
75
 
114
- Useful flags:
115
-
116
- - `--restate-admin-url` / `--restate-ingress-url` — Restate control-plane and ingress URLs. Defaults match the Docker fixture.
117
- - `--restate-service-port` — local Bun endpoint port. Defaults to `19081`. The Docker smoke fixture's standalone service uses `19080`; the start CLI uses `19081` to avoid colliding with it.
118
- - `--restate-deployment-uri` — URI Restate uses to reach the local endpoint. Defaults to `http://host.docker.internal:<service-port>` for the Docker fixture.
119
- - `--restate-keep-alive` — keep the local endpoint alive after the first invocation so Restate can retry or you can inspect/reinvoke it.
120
- - `--restate-invocation-timeout-ms` — optional client-side timeout for the CLI's ingress `fetch`. This is not a durable handler execution timeout. Omit it for long-running local agent runs.
121
- - `--restate-provider-retry-attempts` / `--restate-tool-retry-attempts` — advanced overrides for the `ctx.run` retry attempts used by the CLI runner's provider and tool wrappers.
122
- - `--restate-random-failure` — local chaos switch that injects retryable provider failures for exercising replay/retry behavior.
76
+ | Flag | Default | Notes |
77
+ |---|---|---|
78
+ | `--restate-admin-url` | `http://localhost:19070` | Matches the Docker fixture |
79
+ | `--restate-ingress-url` | `http://localhost:18080` | Matches the Docker fixture |
80
+ | `--restate-service-port` | `19081` | Local Bun endpoint port; the Docker fixture's own service uses `19080` |
81
+ | `--restate-deployment-uri` | `http://host.docker.internal:<service-port>` | URI Restate uses to reach the local endpoint |
82
+ | `--restate-keep-alive` | off | Keep the endpoint alive after the first invocation for retries/manual invokes |
83
+ | `--restate-invocation-timeout-ms` | none | Client-side ingress `fetch` timeout only — not a handler execution timeout |
84
+ | `--restate-provider-retry-attempts` / `--restate-tool-retry-attempts` | `3` / `1` | Override the CLI's `ctx.run` retry attempts for the provider/tool wrappers |
85
+ | `--restate-random-failure` | off | Chaos switch: injects retryable provider failures to exercise replay/retry |
123
86
 
124
- `--session-db` is rejected in Restate mode because the virtual-object state owns the session projection.
87
+ `--session-db` is rejected in Restate mode — virtual-object state owns the session projection.
125
88
 
126
- The CLI service exposes:
89
+ The service exposes three handlers:
127
90
 
128
- - `run` — drive an agent run for the object key.
129
- - `status` — read the last run status: `idle`, `running`, `completed`, or `failed`.
130
- - `history` — read the `restateSessionStore` projection for the object key.
91
+ | Handler | Returns |
92
+ |---|---|
93
+ | `run(input)` | Drives an agent run for the object key. Same input + `completed` status replays the persisted result; different input on a non-idle key throws a 409-style `TerminalError` |
94
+ | `status()` | `{ status: 'idle' \| 'running' \| 'completed' \| 'failed', ... }` plus a completion summary |
95
+ | `history()` | The `restateSessionStore` projection for the object key |
131
96
 
132
- When a run completes, the service stores the input fingerprint plus a small completion summary. A later `run` with the same object key and same input returns the persisted session result; a completed object key with different input is rejected with a 409-style terminal error. Use a new `--session-id` for a distinct local run.
97
+ Use a new `--session-id` for a distinct local run.
133
98
 
134
99
  ## How replay works
135
100
 
136
- Restate journals each `ctx.run(name, fn)` call. The original execution runs `fn` and stores its return value (or error). On any subsequent invocation of the same handler attempt — typically a crash recovery — `ctx.run(name, fn)` looks the entry up by `name` and returns the cached value **without invoking `fn`**.
101
+ Restate journals each `ctx.run(name, fn)` call. The first execution runs `fn` and stores its return value. On a later invocation of the same handler attempt (typically crash recovery), `ctx.run(name, fn)` looks the entry up by `name` and returns the cached value **without invoking `fn`**.
137
102
 
138
103
  ```
139
- ┌──────────────── live ─────────────────┐
140
- │ │
141
- turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => anthropic.stream(...))
142
- │
104
+ turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => provider.stream(...))
143
105
  ├─ journal entry: TurnResult
144
- │
145
106
  ┌───┴───┐
146
107
  │ crash │
147
108
  └───┬───┘
148
- │
149
- turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => anthropic.stream(...))
150
- │
109
+ turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => provider.stream(...))
151
110
  └─ returns journaled TurnResult (no model call)
152
111
  ```
153
112
 
154
- Journal-entry names are scoped to the handler invocation:
155
-
156
- - **LLM calls**: `llm-call-1`, `llm-call-2`, … (monotonic per wrapper instance).
157
- - **Tools**: `tool-<name>-1`, `tool-<name>-2`, … (per wrapper, per tool name).
158
-
159
- Both names are configurable via the `entryName` option if multiple agents share one service and you need disambiguation.
113
+ Journal-entry names are per-invocation: `llm-call-1`, `llm-call-2`, … for LLM calls (monotonic per provider wrapper instance); `tool-<name>-1`, `tool-<name>-2`, … for tools (per wrapper, per tool name). Both are configurable via `entryName` when multiple agents share one service.
160
114
 
161
- ### What re-fires on replay (and why that's fine)
115
+ **What re-fires on replay.** Restate only replays `ctx.run`; everything else re-executes, including hooks:
162
116
 
163
- Restate replays only what's inside `ctx.run`. Everything else re-executes — including Zidane's hooks:
117
+ - `tool:gate` / `tool:transform` / `tool:after` re-fire; built-ins (truncation, image-stripping) are pure over journaled inputs so the result is replay-stable.
118
+ - `turn:before` / `turn:after` / `usage` re-fire, reading the journaled `TurnResult`.
119
+ - `stream:text` / `stream:thinking` do **not** re-fire — they're part of the live stream. Reconstruct transcripts from `session.turns` (`eventsFromTurns` in `zidane/chat`) instead.
120
+ - Consumer hooks touching external services (logging, metrics) should wrap those calls in `ctx.run` themselves, or they fire once per attempt.
164
121
 
165
- - `tool:gate`, `tool:transform`, `tool:after` re-fire. Built-in handlers (truncation, image-stripping, `<edit-outcomes>` merge) are pure on journaled inputs, so the substitution is replay-stable.
166
- - `turn:before`, `turn:after`, `usage` re-fire. They read the journaled `TurnResult`, so emitted values match the original.
167
- - `stream:text`, `stream:thinking` do **not** re-fire — those callbacks are part of the live stream, which doesn't replay. UIs should reconstruct transcripts from `session.turns` (`eventsFromTurns` in `zidane/chat`) rather than relying on hook re-emission. The chat layer already does this.
168
- - Consumer hooks that hit external services (logging APIs, metrics endpoints) should wrap those side effects in `ctx.run` themselves — otherwise they'll fire twice across the original + replay.
169
-
170
- ### Per-key serialization
171
-
172
- Restate gives one concurrent invocation per virtual-object key. With sessions bound 1:1 to `ctx.key`, concurrent users hit different keys and never race; the runtime serializes invocations on the same key. No application-level locks, no `BEGIN IMMEDIATE` dance.
122
+ **Per-key serialization.** Restate allows one concurrent invocation per virtual-object key. Bind sessions 1:1 to `ctx.key` and concurrent users never race — no application-level locks needed.
173
123
 
174
124
  ## Performance: minimizing suspensions
175
125
 
176
- On platforms without bidirectional streaming (AWS Lambda, sandbox-reattach runners), each awaited durable completion can suspend the invocation, and every resume replays the journal and (under eager state) re-ships the full K/V snapshot. So execution time scales with **(A) the number of awaited durable completions per run** and **(B) the journal + state size**. Levers, highest-impact first:
126
+ On platforms without bidirectional streaming (Lambda, sandbox-reattach runners), every awaited durable completion can suspend the invocation, and each resume replays the journal — execution time scales with the number of awaited completions and the journal/state size. `ctx.run` for LLM/tool calls is an irreducible floor; these levers remove overhead around it, highest-impact first:
177
127
 
178
- 1. **`maxConcurrentTools`** (biggest for tool-heavy turns). The default is `1`, which serializes tools — N tool calls become N separate `ctx.run` awaits = N suspend/replay cycles. Raise it (`restateBehavior({ maxConcurrentTools: 8 })`) so concurrency-safe calls batch through the durable `all()` combinator: one park per batch *and* parallel wall-clock execution. Safety is judged per call (`isConcurrencySafe` can inspect the input — a read-only `shell` batches, a mutating one doesn't); unsafe calls (writes/edits, mutating commands) stay barriers, so correctness is unchanged — you're only trading resource concurrency. Set it deliberately; the right ceiling is workload-specific.
179
- 2. **Coalescing clock**: `restateClock(ctx, { coalesce: true })` collapses the ~per-turn `ctx.date.now()` round-trips into a journaled anchor + monotonic counter. See [`restateClock`](#restateclock) for the precision tradeoff.
180
- 3. **Share that clock with the store**: `restateSessionStore(ctx, { clock })` routes the store's finalize-time `save` / `updateStatus` timestamps through the same coalesced anchor instead of spending their own `ctx.date.now()`.
181
- 4. **MCP `exclude`**: list deterministic, side-effect-free MCP read tools in `restateMcpToolWrap(ctx, { exclude: [...] })` so they skip journaling entirely.
182
- 5. **Stay on eager state.** Lazy state (`enableLazyState`) makes each `ctx.get` a suspend+replay on no-streaming platforms and gets worse as the journal grows — the opposite of what you want here. The session store reads each slot at most once and (under default eager state) those reads are local/free anyway.
183
- 6. **Context discipline** (already on in `restateBehavior`): `compactStrategy: 'tail'`, `elideStaleReads: 'stub'`, `toolOutputBudget`, `autoCompact` shrink the per-turn provider payload — fewer tokens and lower LLM latency every turn. Note the deliberate divergence from the chat profiles' `'notice'`: restate optimizes for payload size (journal weight + latency), so it accepts the prompt-cache re-bill that wire-level stubbing costs; see `AgentBehavior.elideStaleReads` for the trade.
184
-
185
- `ctx.run` for LLM and tool calls is the irreducible floor — it's the real work. Levers 1–4 remove the overhead *around* it.
128
+ | Lever | Effect |
129
+ |---|---|
130
+ | Raise `maxConcurrentTools` (`restateBehavior({ maxConcurrentTools: 8 })`) + `restateToolBatchExecutor` | N sequential tool suspend/replay cycles collapse into one durable `all()` batch for concurrency-safe calls |
131
+ | `restateClock(ctx, { coalesce: true })` | Collapses `ctx.date.now()` round-trips into a journaled anchor + monotonic counter (precision tradeoff in [`restateClock`](#restateclock)) |
132
+ | `restateSessionStore(ctx, { clock })` | Shares that same anchor for the store's own timestamp writes |
133
+ | `restateMcpToolWrap(ctx, { exclude: [...] })` | Skips journaling for deterministic, side-effect-free MCP read tools |
134
+ | Stay on eager state | Lazy state (`enableLazyState`) turns each `ctx.get` into a suspend+replay; the session store already reads each slot once |
135
+ | Context discipline (on by default in `restateBehavior`) | `compactStrategy: 'tail'`, `elideStaleReads: 'stub'`, `toolOutputBudget`, `autoCompact` shrink the per-turn payload |
186
136
 
187
137
  ## Wrapper details
188
138
 
189
139
  ### `restateClock`
190
140
 
191
- `AgentClock` backed by `ctx.date.now()` + `ctx.rand.uuidv4()`. Pass it via `AgentRunOptions.clock` (or `createAgent({ clock })`) so journaled metadata — timestamps, run IDs, and fallback turn IDs — stays byte-identical across replays.
141
+ Backs `AgentClock.now()` with `ctx.date.now()` (journaled, async) so `SessionTurn.createdAt`, run start/end, and fallback turn ids stay identical across replays.
192
142
 
193
- `ctx.date.now()` is async (the SDK routes it through `ctx.run` internally; replay returns the journaled timestamp). `ctx.rand.uuidv4()` is sync (the SDK seeds an in-memory RNG from the invocation id, itself replay-stable).
143
+ `randomUUID()` is **not** `ctx.rand.uuidv4()` per call: `ctx.rand` is a single invocation-seeded stream, and the agent can't guarantee every consumer draws the same count on every replay (this swapped turn ids across a suspension in practice). Instead each id is `deterministicUuid(idSeed:uuid:N)` — a sha256-derived UUID from a clock-private counter, pure and sync. `idSeed` defaults to one `ctx.rand.uuidv4()` draw at construction; pass `{ idSeed: ctx.key }` for a draw-free construction.
194
144
 
195
- **Suspension cost & coalescing.** Each `ctx.date.now()` is a journaled durable op — a round-trip the invocation can suspend on. On sandbox-reattach hosts a suspension tears down and re-attaches the worker, so these add up: the loop samples `now()` a few times per turn (the assistant turn's `createdAt`, the tool-result turn's `createdAt`, the per-turn wall check when `maxWallMs` is set, plus run start/end). `randomUUID()` is sync and never suspends.
145
+ **Coalescing.** `restateClock(ctx, { coalesce: true })` journals a real anchor once per `reanchorEvery` calls (default `8`) and derives intervening timestamps as `anchor + n` — replay-deterministic and monotonic, but clustered near the anchor rather than wall-clock-accurate. Live-only measurements (TTFT, `maxWallMs`) read `Date.now()` directly and are unaffected. Default (off) journals every `now()` call. Under coalescing, a subagent's reported elapsed time becomes a draw count — read `ChildRunStats.stats.elapsed` (always live) for the real duration.
196
146
 
197
- Pass `restateClock(ctx, { coalesce: true })` to collapse most of those round-trips: the clock journals a real anchor once, derives a monotonic logical sequence from it (`anchor`, `anchor+1`, …) without re-journaling, and re-anchors against the real clock only every `reanchorEvery` calls (default 8; tune via `{ coalesce: { reanchorEvery: n } }`). It stays fully replay-deterministic (anchors replay verbatim; the counter is a pure function of call order) and strictly monotonic.
198
-
199
- The **tradeoff**: timestamps within a re-anchor window are logical — ordered and stable, but clustered near the window's anchor rather than wall-clock-accurate. Run `elapsed` (`endedAt - startedAt`) and per-turn `createdAt` lose precision down to the window granularity; live-only measurements (TTFT, `maxWallMs` budget) are unaffected because the loop reads those from `Date.now()` directly.
200
-
201
- Two subagent-specific consequences, both from differencing two coalesced draws: the elapsed figure in the spawn result string the **model reads** (`[sub-agent child-1] Completed in 2 turns (3ms)`) becomes a count of intervening clock draws rather than milliseconds, so a child that ran 40 seconds can report single-digit ms; and `createSpawnTool({ timeoutMs })`'s turn-boundary deadline measures the same logical scale, so it effectively counts draws too and won't fire on wall-clock time. Leave coalescing off if the model needs true durations or the child budget must track real time — hosts can always read the real duration off `ChildRunStats.stats.elapsed` (`spawn:complete` / `AgentStats.children`), which stays live `Date.now()`. Lower `reanchorEvery` ⇒ more accurate timestamps + more suspensions; higher ⇒ fewer suspensions + more clustering. The default (coalescing off) journals every `now()` and keeps wall-accurate timestamps, so opt in per deployment. Provider/tool `ctx.run` calls remain irreducible suspension points — coalescing only addresses the clock.
202
-
203
- **Gotcha**: when you pair a `restateSessionStore` with a session, the store's own `generateTurnId` runs ahead of `clock.randomUUID()` (the loop short-circuits via `session?.generateTurnId() ?? clock.randomUUID()`). The store derives deterministic IDs from the persisted turn count (`turn_1`, `turn_2`, …) so turn IDs stay stable across replay and across later invocations on the same object key. If you wire a custom store, keep turn ID generation deterministic from persisted state or let the Restate-backed clock provide the fallback UUIDs.
147
+ **Gotcha**: with `restateSessionStore`, the store's deterministic `generateTurnId` (`turn_1`, `turn_2`, …) takes priority over `clock.randomUUID()` — the loop only falls back to the clock with no store present. A custom store's turn-id generation must stay deterministic from persisted state.
204
148
 
205
149
  ### `restateProvider`
206
150
 
207
- Wraps `Provider.stream` in `ctx.run`. One journal entry per assistant turn carrying the full `TurnResult`. All other `Provider` methods (`formatTools`, `userMessage`, `toolResultsMessage`, `classifyError`) pass through unchanged — they're pure over already-deterministic inputs.
208
-
209
- Default `runOptions`: `{ maxRetryAttempts: 3 }`. Provider transients (rate limits, 5xx, connection resets) get retried inside the journal entry. Once the cap is reached the failure becomes terminal and surfaces through Zidane's normal `AgentProviderError` path.
151
+ Wraps `Provider.stream` in `ctx.run`; one journal entry per assistant turn. All other `Provider` methods pass through unwrapped (pure over already-deterministic inputs).
210
152
 
211
- `streamCallbacks: 'suppress'` disables live `onText` / `onThinking` / server-tool callbacks inside the durable step. Use this for strict replay-safe deployments where UIs reconstruct from persisted `session.turns`; the default remains `'live'` for compatibility.
153
+ Default `runOptions` depends on `streamCallbacks`:
154
+ - `'live'` (default) → `{ maxRetryAttempts: 1 }`. A live stream that fails after emitting deltas would, on retry, re-fire `onText` into the loop's accumulator and duplicate partial output — the loop's own retry (clean-slate re-issue) is the correct layer here.
155
+ - `'suppress'` → `{ maxRetryAttempts: 3 }`. No live callbacks fire, so an in-journal retry is invisible; use this for strict replay-safe UIs that reconstruct from `session.turns`.
212
156
 
213
- ### Content refs
214
-
215
- Zidane can persist image/document blocks as opaque refs (`{ type: 'image' | 'document', ref, mediaType }`) and resolve them only when assembling provider wire messages. `restateSessionStore(ctx)` needs no special handling: `session.turns` still contains refs, so `ctx.set` journals the same ref bytes it stores.
216
-
217
- Wrap the host download in `ctx.run` from a `content-ref:resolve` hook, so replay returns the journaled base64 instead of re-downloading:
218
-
219
- ```ts
220
- import { hashContent } from 'zidane/tools'
221
-
222
- const agent = createAgent({
223
- // ...
224
- hooks: {
225
- 'content-ref:resolve': async resolveCtx => {
226
- const { data } = await ctx.run(
227
- `resolve-content-${hashContent(resolveCtx.block.ref)}`,
228
- async () => ({ data: await downloadMediaAsBase64(resolveCtx.block.ref) }),
229
- )
230
- resolveCtx.data = data
231
- },
232
- },
233
- })
234
- ```
157
+ Beyond the retry cap the failure surfaces through the normal `AgentProviderError` path.
235
158
 
236
- That keeps virtual-object state lean (refs only) while the Restate journal records the resolved bytes only for the provider attempt that actually needed them.
159
+ **Content refs.** Zidane can persist image/document blocks as opaque `{ type, ref, mediaType }` refs, resolved only when assembling wire messages; `restateSessionStore` journals the ref bytes as-is, no special handling needed. Wrap the host download itself in `ctx.run` inside the `content-ref:resolve` hook (`resolveCtx.data = await ctx.run(name, () => downloadMediaAsBase64(resolveCtx.block.ref))`) so replay returns the journaled base64 instead of re-downloading.
237
160
 
238
161
  ### `restateTool` / `wrapAgentTools`
239
162
 
240
- Wraps `ToolDef.execute` in `ctx.run`. Once Restate persists the result, replay returns the journaled value without re-executing the tool. If the process dies before that journal entry is durable, the latest in-flight tool can run again; tools that hit the network, write to disk, or poke external systems still need application-level idempotency keys.
241
-
242
- Default `runOptions`: `{ maxRetryAttempts: 1 }`. Tool errors are typically deterministic (validation failures, missing files); blind retries waste budget. Override per-tool for network-heavy custom tools that benefit from SDK-level retries.
243
-
244
- **Soft tool errors (`softErrors`, default `true`).** A recoverable tool throw (a failed shell command, a `RemoteExecHttpError`, any `execute` rejection) is captured inside the journal as a sentinel value and re-thrown as a plain (non-control) error *outside* it — so the loop produces a model-visible `tool_result { isError }` and the run continues, identical to the non-Restate soft-error model. Without it the throw would exhaust `ctx.run`'s retries, surface as a Restate `TerminalError`, and `restateBehavior`'s `shouldRethrowToolError` would abort the whole run. Real Restate control / abort errors are never softened. Replay reproduces the journaled error without re-running the tool. This is the same mechanism `restateMcpToolWrap` applies to MCP tools — native and MCP tools now behave symmetrically. Set `softErrors: false` to restore the legacy fatal propagation.
245
-
246
- **Result-block coercion (`coerceResultBlocks`, default `true`).** Malformed structured result blocks (a `{ type: 'image' }` with no payload, a foreign block shape) collapse to a text placeholder before the result is journaled, so a downstream provider wire serializer never has to project an unprojectable block (which, under Restate, would throw inside the journaled provider call and become fatal). Well-formed results pass through untouched.
247
-
248
- Both options apply to the single-phase journaling path (the default wrapping and the MCP path); the two-phase `durableStart` mode owns its own start/wait result shape and is unaffected.
249
-
250
- `isConcurrencySafe` is preserved. `restateBehavior()` defaults `maxConcurrentTools` to `1` so strict Restate mode is deterministic first. If you want parallel tools, opt into the Restate scheduler below instead of relying on native promise fan-out.
163
+ Wraps `ToolDef.execute` in `ctx.run`, default `{ maxRetryAttempts: 1 }` (tool errors are typically deterministic, so a blind retry wastes budget; override per-tool for network-heavy custom tools). If the process dies before the journal entry lands, the in-flight tool can run again — non-idempotent tools need an application-level idempotency key.
251
164
 
252
- `wrapAgentTools(tools, ctx, { exclude })` is a batch helper. Pass-through for `exclude`-listed names (fully-deterministic local helpers where journaling is overhead). By default it hides and rejects the built-in `shell` tool's `run_in_background` mode because host-local background processes are not recoverable by Restate. Two opt-outs:
165
+ - **`softErrors` (default `true`).** A recoverable throw is captured inside the journal as a sentinel and re-thrown as a plain (non-control) error outside it, so the loop still produces `tool_result { isError }` and continues, matching non-Restate behavior. Without it, the throw exhausts `ctx.run`'s retries into a Restate `TerminalError` and `shouldRethrowToolError` aborts the whole run. Control/abort errors are never softened. `restateMcpToolWrap` applies the same mechanism.
166
+ - **`coerceResultBlocks` (default `true`).** Malformed structured result blocks (e.g. an `{ type: 'image' }` with no payload) collapse to a text placeholder before journaling, so the wire serializer never chokes on one inside a journaled call.
167
+ - `isConcurrencySafe` is preserved from the wrapped `ToolDef` (see SKILL.md § Per-tool concurrency); `restateBehavior()` defaults `maxConcurrentTools: 1` regardless.
168
+ - `wrapAgentTools(tools, ctx, { exclude })` batch-wraps, passing exclude-listed tools through untouched, and disables the built-in `shell` tool's `run_in_background` mode by default (host-local background processes aren't Restate-recoverable) unless `{ detachedTasks: 'durable' }` or `{ disableShellBackground: false }` says otherwise.
253
169
 
254
- - `{ detachedTasks: 'durable' }` — declare the execution context's `ContextCapabilities.detachedTasks` tier. A `'durable'` context (remote runner whose tasks survive worker crashes) keeps background mode automatically.
255
- - `{ disableShellBackground: false }` — explicit boolean override; always wins over the capability.
170
+ **`durableStart` — crash-safe long tools.** Default wrapping only journals the *completed* result, so a crash mid-execute re-runs the tool — unacceptable for builds, migrations, deploys. It splits execution into a journaled `start` (kicks off the work, returns a re-attach token) and either:
256
171
 
257
- #### `durableStart` — crash-safe long tools (re-attach instead of re-run)
258
-
259
- The default wrapping journals only the *completed* result, so a crash mid-execute re-runs the tool on replay — unacceptable for builds, migrations, deploys. `restateTool`'s `durableStart` option splits the execution into two journal entries:
260
-
261
- ```ts
262
- restateTool(deployTool, ctx, {
263
- durableStart: {
264
- // Journaled BEFORE the body — kick off the work, return a re-attach token.
265
- start: async (input, toolCtx) => startDeploy(input), // → { jobId }
266
- // On replay, receives the SAME journaled token — re-attach, don't re-run.
267
- attach: async (token, input, toolCtx) => awaitDeploy(token.jobId),
268
- },
269
- })
270
- ```
271
-
272
- A crash after `start` journals replays into the wait phase with the recorded token. The wrapped tool deliberately does not advertise `durableExecute` (the two-phase composition is native-async); the batch scheduler runs it sequentially, which is the right shape for long re-attachable work.
273
-
274
- **`attach` holds the wait inside one `ctx.run` entry** — fine for waits bounded well below the server's inactivity timeout, but a long busy-poll in there starves the journal: the inactivity timeout fires, the abort timeout kills the attempt, and each retry re-runs `attach` with a fresh deadline. The invocation can never complete. Keep `attach` only as the degraded mode for hosts without a push seam.
275
-
276
- **`wait` is the awakeable-based alternative for long-running work.** Hosts that can push completion (e.g. a runner firing `task:exit` → resumer → `ctx.resolveAwakeable`) configure it instead of `attach` (it wins when both are set):
172
+ - **`attach`** — replay re-attaches with the same token inside one `ctx.run` entry. Fine for waits well below the server's inactivity timeout; a long busy-poll here starves the journal (inactivity timeout → abort → re-run on every retry, never completing). Use only as a degraded mode.
173
+ - **`wait`** (wins if both set) — awakeable-based: `register` (journaled) hands the host an awakeable id as exit resumer, the park happens **outside** `ctx.run` (raced against `timeoutMs` via `awakeableWithTimeout` if set), and `result` (journaled) shapes the final value from the resume payload.
277
174
 
278
175
  ```ts
279
176
  restateTool(shellTool, ctx, {
280
177
  durableStart: {
281
- start: async input => startBackgroundTask(input), // → { taskId }
178
+ start: async input => startBackgroundTask(input), // → { taskId }
282
179
  wait: {
283
- // Journaled — hand the awakeable id to the host as the exit resumer.
284
180
  register: (token, awakeableId) => registerExitResumer(token.taskId, awakeableId),
285
- // The model's own timeout becomes the durable park deadline.
286
181
  timeoutMs: input => input.timeout as number,
287
- promises: RestatePromise, // durable race combinator
288
- // Journaled — synthesize the foreground-shaped result. `null` = timeout:
289
- // kill the job and shape the timeout result here.
290
- result: async (token, resumed) =>
291
- resumed ? formatExit(token.taskId, resumed) : killAndFormatTimeout(token.taskId),
182
+ promises: RestatePromise,
183
+ result: async (token, resumed) => resumed ? formatExit(token.taskId, resumed) : killAndFormatTimeout(token.taskId),
292
184
  },
293
185
  },
294
186
  })
295
187
  ```
296
188
 
297
- The orchestration follows `restateAwakeableTool`'s journal discipline, keyed by the same entry naming: `register` runs in a journaled entry (`…-register`, exactly once across replays, marker invariant included), the park happens **outside** any `ctx.run` so the invocation suspends cleanly (no held worker, immune to inactivity/abort timeouts), and `result` runs in the final journaled result entry. With `timeoutMs` the park is an `awakeableWithTimeout` race against a durable `ctx.sleep` — replay-stable, unlike a `Date.now()` deadline that resets per attempt.
298
-
299
- The optional `onAbort(token, input, toolCtx)` is the abort seam: when the per-call signal fires before the wait settles (run abort, `cancelTool`, `destroy()` mid-run), it runs once, best-effort, so the started job doesn't keep running with nobody attached. In `wait` mode the abort also unblocks the park (the call rejects instead of waiting on a resumer nobody will consume). It is NOT called on process death — that's not termination; the journaled token re-attaches on replay. It runs outside the journal (an aborting invocation can't reliably open new `ctx.run` entries); journal host-side cleanup yourself if it must be durable.
189
+ The wrapped tool doesn't advertise `durableExecute`, so the batch scheduler always runs it sequentially — correct for long re-attachable work. Optional `onAbort(token, input, toolCtx)` fires once, best-effort, when the per-call signal fires before the wait settles (run abort, `cancelTool`, `destroy()`); it is NOT called on process death (the journaled token re-attaches on replay instead) and runs outside the journal.
300
190
 
301
191
  ### `restateAwakeableTool`
302
192
 
303
- Builds a `ToolDef` that parks on a Restate awakeable until an external resumer calls `ctx.resolveAwakeable(id, value)` — the canonical shape for `ask_user` / `request_review` tools inside durable handlers.
193
+ Builds a `ToolDef` that parks on a Restate awakeable until an external resumer calls `ctx.resolveAwakeable(id, value)` — the shape for `ask_user` / `request_review` tools:
304
194
 
305
195
  ```ts
306
196
  const requestReview = restateAwakeableTool<{ approved: boolean }>({
@@ -313,24 +203,15 @@ const requestReview = restateAwakeableTool<{ approved: boolean }>({
313
203
  })
314
204
  ```
315
205
 
316
- Replay contract: the `notify` broadcast runs inside `ctx.run`, so it fires exactly once across replays; the journaled marker records the awakeable id and the tool refuses to park if a replay mints a different id (journal drift). Do not additionally wrap the returned tool in `restateTool` — the suspension point must not nest inside a journaled closure.
206
+ `notify` runs inside `ctx.run` so it fires exactly once across replays; a journaled marker records the awakeable id and the tool refuses to park if a replay mints a different one (journal drift). Do not wrap the returned tool in `restateTool` — a suspension point must not nest inside another journaled closure.
317
207
 
318
208
  ### `restateSessionStore`
319
209
 
320
- A `SessionStore` backed by the bound virtual object's K/V state. Three slots per session — `session-data` (header), `session-turns` (history), `session-runs` (run records).
321
-
322
- **State I/O is coalesced per invocation.** Each slot is read from Restate at most once and then served from an in-memory read-through cache; writes update the cache and `ctx.set` eagerly. Without this, `appendTurns` (which fires on every turn boundary) re-read and re-sanitized the entire growing `session-turns` array on each call — O(N²) sequential K/V round-trips over an N-turn run. Writes stay eager rather than deferred to a single flush, because external read visibility is the whole reason to pick this store: a concurrent reader must see each turn as it lands. The cache is safe because Restate processes one invocation at a time per object key, so no competing writer can invalidate it mid-invocation.
323
-
324
- **Optional shared clock.** `restateSessionStore(ctx, { clock })` sources the header `createdAt` / `updatedAt` timestamps from the injected `AgentClock` instead of a direct `ctx.date.now()`. Pass the same `restateClock(ctx, { coalesce: true })` you give the agent so the store's finalize-time `save` / `updateStatus` timestamps ride the coalesced anchor (≈2 fewer `ctx.date.now()` round-trips per run on suspend-per-round-trip hosts). Whatever you pass must be replay-stable — the value lands in persisted state via `ctx.set`. Default (no clock) is unchanged: direct `ctx.date.now()`.
325
-
326
- **When to use it.** Pick this store when **external** consumers (a TUI subscribed to the runtime, a web GUI, support tooling) need to read a session without re-entering the agent handler. Virtual-object state is exposed via the SDK's read APIs and the Restate UI — same data, different access path.
210
+ A `SessionStore` backed by the bound virtual object's K/V state across three slots: `session-data` (header), `session-turns` (history), `session-runs` (run records). Each slot is read at most once per invocation and cached in memory after that (writes update the cache and call `ctx.set` eagerly — safe since Restate serializes one invocation per key), avoiding an O(N²) re-read/re-sanitize of the whole `session-turns` array on every turn. `{ clock }` sources header/finalize timestamps from the injected `AgentClock` instead of a direct `ctx.date.now()`.
327
211
 
328
- **When to skip it.** If you only care about durable execution within one invocation, an in-memory store is enough: the journaled `ctx.run` entries already replay the loop to byte-identical state after a crash. Turns rematerialize from the journal on every attempt.
212
+ **Use it** when external consumers (a TUI, a web GUI) need to read a session without re-entering the handler. **Skip it** for pure in-invocation durability — journaled `ctx.run` entries already replay the loop to identical state.
329
213
 
330
- Limitations baked into the contract:
331
- - `list()` returns `[]` — virtual objects don't expose cross-key enumeration. Maintain a separate index handler if discovery matters.
332
- - `delete()` only clears the current object's keys (no cross-key fan-out).
333
- - Single-tenant per virtual-object key: every method ignores the `sessionId` argument because the object is already keyed. Calling `appendTurns('other-id', …)` writes to the CURRENT object. Bind one virtual-object key per session and the contract holds.
214
+ Contract limits: `list()` returns `[]` (virtual objects don't enumerate cross-key); `delete()` only clears the current object's keys; every method ignores its `sessionId` argument because the object is already keyed — bind one virtual-object key per session.
334
215
 
335
216
  ### Restate-native parallel tools
336
217
 
@@ -347,75 +228,46 @@ createAgent({
347
228
  })
348
229
  ```
349
230
 
350
- The scheduler starts concurrency-safe, Restate-wrapped tools in submission order, joins each chunk through `RestatePromise.all()`, and emits tool results back to the model in original call order. Unsafe tools remain barriers. If a safe tool is not wrapped with `restateTool()` / `wrapAgentTools()`, Restate mode runs that chunk sequentially instead of falling back to native promise parallelism. This mode is opt-in so the TUI/headless harness keeps its native sliding-window scheduler.
231
+ Starts concurrency-safe, Restate-wrapped tools in submission order, joins each chunk through `RestatePromise.all()`, and emits results in original call order. Unsafe tools remain barriers. A safe tool not wrapped with `restateTool()`/`wrapAgentTools()` makes its whole chunk run sequentially instead of falling back to native promise parallelism. Opt-in, so the TUI/headless harness keeps its native sliding-window scheduler.
351
232
 
352
233
  ## Subagents
353
234
 
354
- `spawn` works under Restate, with one rule: **the spawn tool is never journaled — its child's steps are.**
235
+ `spawn` works under Restate with one rule: **the spawn tool is never journaled — its child's steps are**, because `wrapAgentTools` skips any tool declaring `ToolDef.runsNestedAgent` (the built-in `spawn` sets it).
355
236
 
356
237
  ```ts
357
- const spawnTool = createSpawnTool({
358
- persist: true, // child turns + runs land in the parent's session
359
- shareHandle: true, // no unjournaled per-child sandbox mint
360
- timeoutMs: 600_000,
361
- })
238
+ const spawnTool = createSpawnTool({ persist: true, shareHandle: true, timeoutMs: 600_000 })
362
239
 
363
240
  createAgent({
364
241
  provider: restateProvider(myProvider, ctx, { streamCallbacks: 'suppress' }),
365
- // Pass `spawn` in like any other tool — `wrapAgentTools` leaves it unwrapped.
366
- tools: wrapAgentTools({ ...tools, spawn: spawnTool }, ctx),
242
+ tools: wrapAgentTools({ ...tools, spawn: spawnTool }, ctx), // spawn passes through unwrapped
367
243
  clock: restateClock(ctx),
368
244
  session,
369
245
  behavior: restateBehavior({ maxTurns: 8 }),
370
246
  })
371
247
  ```
372
248
 
373
- ### Why the spawn tool is excluded from wrapping
374
-
375
- A child agent drives the *same* journaled provider (`ctx.provider`) and the *same* wrapped tools (`ctx.tools`) as its parent. If the spawn tool itself were wrapped, the child's first provider call would open a `ctx.run` **inside** the spawn tool's own `ctx.run`. That fails on the retry with `RestateError (570) Found a mismatch between the code paths taken during the previous execution and the paths taken during this execution` — observed against a real server, which is why `test/fixtures/restate-docker` now carries a subagent scenario.
249
+ If the spawn tool itself were wrapped, the child's first provider call would open a `ctx.run` **inside** the spawn tool's own, which fails on retry with `RestateError 570` (journal mismatch). Instead the child's provider/tool calls become journal entries *of the parent's invocation*: a crash mid-subagent replays completed child steps, and only the unfinished step re-executes.
376
250
 
377
- `wrapAgentTools` therefore skips any tool declaring `ToolDef.runsNestedAgent` (the built-in `spawn` sets it; host orchestrator tools that call `agent.run` should too). The skip is keyed on that marker, not the tool name, so a host tool that merely happens to be called `spawn` still gets wrapped.
251
+ **Escape hatch**: wrap a spawn tool explicitly (`restateTool(spawnTool, ctx, …)`) only if the child is fully non-durable (raw provider, unwrapped tools) — it collapses into one journal entry, cheap to replay but a crash re-runs the whole child and re-bills every LLM call.
378
252
 
379
- What you get instead is better: the child's provider calls and tool calls are journal entries **of the parent's invocation**. A crash mid-subagent replays the child's completed steps from the journal without re-running their bodies; only the unfinished step re-executes. Suspensions inside a child behave the same way.
253
+ **Inherited automatically**: `AgentClock`, `AgentBehavior` (incl. `toolBatchExecutor`), wrapped tools/provider. **Opt in**: session (`persist: true` — child turn ids come from the store's deterministic mint) and execution handle (`shareHandle: true` — otherwise every child's unjournaled `execution.spawn()` orphans a sandbox on replay). **Never inherited**: durable tool state — a child's `ctx.state` is always an isolated in-memory map.
380
254
 
381
- **Escape hatch.** Wrapping a spawn tool explicitly (`restateTool(spawnTool, ctx, …)`) is legal *only* if that child is fully non-durable — a raw provider from a model catalog, unwrapped tools. Then the whole child collapses into ONE journal entry: cheap to replay, but a crash inside the child re-runs the entire child (re-billing every LLM call) and none of its side effects are individually journaled.
255
+ **Per-child timeout.** `createSpawnTool({ timeoutMs })` normally races a `setTimeout` — a wall-clock branch that can abort a live provider stream mid-journal-entry. When the clock reports `journaled: true` (as `restateClock` does), it instead samples the clock at turn boundaries, so a child stuck inside one long tool call isn't interrupted (use the runtime's invocation timeout for a hard cut) and the measured elapsed time spans any crash-to-retry gap. Don't use `behavior.maxWallMs` for this: it clamps the provider watchdogs too, aborting an in-flight stream inside its own journal entry.
382
256
 
383
- ### What the child inherits
384
-
385
- | Inherited | How |
386
- |---|---|
387
- | `AgentClock` | Automatic. The child stamps `SessionTurn.createdAt`, run start/end, and fallback turn ids from the parent's journaled clock, so a shared session gets byte-identical metadata across attempts. A child on the default clock mints fresh wall-clock values on every replay. |
388
- | `AgentBehavior` | Automatic, including `toolBatchExecutor` — a child's own concurrency-safe fleets join through `RestatePromise.all()` like the parent's. |
389
- | Wrapped tools + provider | Automatic (`ctx.tools` / `ctx.provider`) — this is what makes the child's steps journaled at all. |
390
- | Session | `persist: true`. Child turn ids then come from `restateSessionStore`'s deterministic `(object key, seq)` mint. |
391
- | Execution handle | `shareHandle: true`. Otherwise every child calls `execution.spawn()` — an unjournaled allocation, so a replayed attempt mints a fresh sandbox and orphans the first. |
392
- | Durable tool state | **Never.** A child's `ctx.state` is an isolated in-memory map — see the note under Durable tool state below. |
393
-
394
- ### Per-child timeout
395
-
396
- `createSpawnTool({ timeoutMs })` normally races the child against a `setTimeout`. That is a wall-clock branch: the same child can time out live and not on the replay, and the abort can land mid-stream, tearing a provider call down from inside its own journal entry. When the clock reports `journaled` (as `restateClock` does), the spawn tool leaves the timer off and instead samples the journaled clock at the child's **turn boundaries** — one sample before the run, one per completed turn — aborting the child at a boundary where nothing is in flight. Both mechanisms report `status: 'timeout'`.
397
-
398
- Deliberately **not** implemented by borrowing `behavior.maxWallMs`, which looks like the natural fit: the loop derives the provider stream watchdogs from the wall budget (`providerWatchdogTimeouts` clamps start / total / idle by it), so a small `timeoutMs` would also shrink the child's TTFT ceiling and abort a live stream inside its `ctx.run`. Against a real server that fails the attempt with `awaitNext already pending` and then `RestateError 570` on the retry. The fixture's deadline scenario exists because that bug shipped for exactly one iteration.
399
-
400
- Two consequences of the turn-boundary check: a child stuck inside ONE long tool call is not interrupted (use the runtime's own invocation timeout for a hard cut), and the measured elapsed time **spans the crash-to-retry gap**, because the run-start timestamp replays from before the crash. Budget accordingly.
401
-
402
- ### Limits
403
-
404
- - **Sequential spawns only** with `createSpawnTool` — for parallel children use `restateSpawnTool` (next section), which sidesteps both reasons below by giving each child its own journal. The weaker one: parallel children race on shared session state — the `child-N` label counter and the run-id re-sync both read `session.runs` at mint time — so label / run-id assignment can differ between attempts. The stronger one: children share the parent's *clock instance*, and `restateClock` carries mutable draw state (the `randomUUID` sequence counter, plus the anchor / call counter in `coalesce` mode). Concurrent children interleave those draws, so the same logical turn can draw a different id or timestamp ordinal on a replay — the `JournalMismatchError` class the clock module exists to prevent. `restateBehavior`'s default `maxConcurrentTools: 1` already gives you the safe posture; don't raise it for a turn that can spawn more than once.
405
- - **Anything that reaches the transcript must come from `ctx.clock`.** The spawn result string is persisted in the parent's turns, and a state write's payload IS compared across attempts — so the elapsed figure in `[sub-agent child-1] Completed in 2 turns (35ms)` is sampled from the clock, not from `stats.elapsed` (which stays live `Date.now()` by design, for measurement accuracy). Getting this wrong fails the invocation at `set state` with `RestateError 570`, and only on an attempt that fails AFTER a child completed — a crash mid-child never produces the string, so the failure hides until a retry happens later in the run. The fixture's retry scenario exists for exactly that reason. The same rule is why the child inherits the clock at all.
406
- - **The parent's spawn tool call is not itself journaled**, so it re-executes on replay: agent construction, MCP re-use, hook bubbling, and result-string formatting all run again. Keep host `onChildStart` / `spawn:*` handlers idempotent.
407
- - **Don't set `behavior.maxWallMs` on a Restate-backed agent** (parent or child) unless you mean it: the provider stream watchdogs are clamped by it, so a tight budget aborts an in-flight stream from inside its journal entry rather than stopping cleanly at a turn boundary. Reach for the runtime's own invocation timeout, or `createSpawnTool({ timeoutMs })` for a child.
257
+ **Limits**:
258
+ - Sequential spawns only with `createSpawnTool` — parallel children would race on shared session state (label counter, run-id re-sync) and the parent clock's mutable draw state. Use `restateSpawnTool` for parallel children; don't raise `maxConcurrentTools` for a turn that can spawn more than once.
259
+ - Anything reaching the transcript must come from `ctx.clock`, never `stats.elapsed` — a live timestamp in a journaled state write fails the invocation on a later retry.
260
+ - The parent's spawn tool call re-executes on replay (agent construction, MCP re-use, hook bubbling, result formatting) since it isn't journaled — keep `onChildStart`/`spawn:*` handlers idempotent.
261
+ - Don't set `behavior.maxWallMs` on a Restate-backed agent, parent or child — same watchdog-clamp hazard. Use the runtime's invocation timeout or `createSpawnTool({ timeoutMs })`.
408
262
 
409
263
  ## Parallel subagents
410
264
 
411
- `createSpawnTool` runs children in the parent's journal, which is what makes them sequential. `restateSpawnTool` dispatches each child to **its own invocation**, so children fan out through the durable fleet — and a crash in one child retries only that child.
265
+ `createSpawnTool` runs children in the parent's journal (sequential by construction). `restateSpawnTool` dispatches each child to **its own invocation** so children fan out through the durable fleet, and a crash in one child retries only that child.
412
266
 
413
267
  ```ts
414
268
  const spawn = restateSpawnTool({
415
- // Return the runtime's promise UNTOUCHED — no await, no .then/.catch.
416
- dispatch: req => ctx.serviceClient(childService).runChild(req),
417
- // Stop in-flight children when the parent's own loop aborts.
418
- onAbort: () => ctx.invocation(ctx.request().id).cancel(),
269
+ dispatch: req => ctx.serviceClient(childService).runChild(req), // return the promise UNTOUCHED — no await/.then
270
+ onAbort: () => ctx.invocation(ctx.request().id).cancel(), // stop in-flight children on parent loop abort
419
271
  })
420
272
 
421
273
  createAgent({
@@ -423,14 +275,12 @@ createAgent({
423
275
  clock: restateClock(ctx),
424
276
  session,
425
277
  behavior: restateBehavior({
426
- maxConcurrentTools: 4, // defaults to 1 — without this there is no fan-out
278
+ maxConcurrentTools: 4, // default 1 — without this there is no fan-out
427
279
  toolBatchExecutor: restateToolBatchExecutor({ promises: RestatePromise }),
428
280
  }),
429
281
  })
430
282
  ```
431
283
 
432
- ### Choosing between the two
433
-
434
284
  | | `createSpawnTool` | `restateSpawnTool` |
435
285
  |---|---|---|
436
286
  | Children per turn | one at a time | parallel (fleet-capped) |
@@ -439,337 +289,195 @@ createAgent({
439
289
  | `child:*` events | live (in-process bubbling) | replayed onto the parent bus when the child merges |
440
290
  | Host code needed | none | a child handler |
441
291
 
442
- ### Host obligations
443
-
444
- - **The child handler must be a `service`**, or an object on a *different* key. A virtual object serializes per key, so children sharing the parent's key queue behind the parent that still holds it — no parallelism, and a deadlock risk.
445
- - **Raise `maxConcurrentTools` and install the durable batch executor.** The fleet gate is all-or-nothing (`fleet.every(canExecuteDurably)`), so with the default cap of 1 the children serialize silently.
446
- - **The child runs sessionless** and returns a `ChildRunResult` (`status`, `text`, `turns`, `stats`). The parent stays the single writer: it merges each child's turns under a `SessionRun` labeled in submission order, so the persisted tree matches the in-process tool's.
447
- - **Enforce depth in the handler** (`req.depth`). The tool always dispatches, because short-circuiting would mean returning a non-durable promise and the fleet rejects those.
448
-
449
- ### What you give up
450
-
451
- - **`child:*` events fire, but not live** — see the section below.
452
- - **`spawn:before` is fired without awaiting** — `durableExecute` must return the durable promise synchronously, so listeners on it must not perform journaled I/O.
453
- - **Merge ordering**: children are merged on `tool-results:after` (with `turn:after` as a backstop for turns that pack no results), so their turns land just *after* the parent's tool-result turn rather than just before it. `createdAt` still orders them correctly for consumers that sort chronologically. If the invocation is torn down before the turn ends at all — a cancellation, say — a child that already settled is never merged and its `spawn:complete` never fires; its work still happened, it just isn't recorded.
454
- - **The child should report its own `startedAt` / `endedAt`** (measured on its clock, e.g. `restateClock(ctx)`). Those land in the merged `SessionRun`, which is persisted through a journaled state write, so they have to be replay-stable — the parent never samples a clock for them. Omit them and it falls back to the child's first/last turn timestamps, then to the parent's run start.
455
- - **Per-child timeouts are the callee's job.** Racing the call promise is the forbidden native-combinator pattern; give the child handler its own budget, or use the runtime's `.orTimeout()`.
456
- - **Cancelling the parent INVOCATION stops the children** — Restate cancels the call tree, so each child halts at its next journal interaction, at any depth. Verified: with two children mid-fan-out, no further work steps started after a `PATCH /invocations/{id}/cancel`. A step already inside a `ctx.run` finishes, since run bodies are not interruptible.
457
- - **An in-agent `agent.abort()` needs `onAbort`.** On its own, `abort()` unwinds the parent's loop while its invocation keeps running, so nothing tells the children to stop. `onAbort` fires once with the in-flight set; cancel the parent's own invocation from it and the runtime cancels the tree. The cost is that the parent invocation ends too — an in-agent abort becomes "stop everything", so persist any summary BEFORE aborting.
458
- - **Cancelling individual children is not possible in this tool shape.** The `invocationId` on the runtime's call handle is itself a journal-backed promise, so reading it while the call is in flight registers a second consumer on the SDK's single-consumer progress channel: `awaitNext already pending`, a failed attempt, and a replay on *every* fan-out (measured — 13 spurious attempts across the fixture suite before this was reverted). Per-child cancellation needs a two-phase `send → id → attach` tool, where the id is consumed before the result promise exists.
292
+ **Host obligations**:
293
+ - The child handler must be a `service`, or an object on a *different* key — a virtual object serializes per key, so children sharing the parent's key would queue behind it.
294
+ - Raise `maxConcurrentTools` and install `restateToolBatchExecutor` — the fleet gate is all-or-nothing (`fleet.every(canExecuteDurably)`), so the default cap of `1` serializes children silently.
295
+ - The child runs sessionless and returns a `ChildRunResult` (`status`, `text`, `turns`, `stats`); the parent is the single writer, merging each child's turns under a labeled `SessionRun`.
296
+ - Enforce depth in the handler (`req.depth`) — the tool always dispatches; short-circuiting would return a non-durable promise, which the fleet rejects.
297
+
298
+ **What you give up**:
299
+ - `child:*` events fire, but not live — see [Child events](#child-events).
300
+ - `spawn:before` fires without awaiting — `durableExecute` must return the durable promise synchronously.
301
+ - Merge happens on `tool-results:after` (with `turn:after` as backstop), so merged turns land just after the parent's tool-result turn, not before (`createdAt` still orders them correctly). A child that settles after the invocation tears down (cancellation) is never merged.
302
+ - The child should report its own `startedAt`/`endedAt` off its own clock — they land in a journaled state write and must be replay-stable. Omitted, it falls back to first/last turn timestamps, then the parent's run start.
303
+ - Per-child timeouts are the callee's job (racing the call promise is the forbidden native-combinator pattern) — give the child handler its own budget or use `.orTimeout()`.
304
+ - Cancelling the parent invocation cancels the whole call tree; a step already inside `ctx.run` finishes first.
305
+ - An in-agent `agent.abort()` needs `onAbort` wired — alone, `abort()` unwinds the parent's loop but leaves children running. `onAbort` fires once with the in-flight set; cancel the parent's own invocation from it (this ends the parent invocation too, so persist any summary first).
306
+ - Cancelling *individual* children isn't possible in this tool shape: reading a call handle's `invocationId` mid-flight registers a second consumer on the SDK's single-consumer progress channel (`awaitNext already pending`, replay on every fan-out). Needs a two-phase send → id → attach tool instead.
459
307
 
460
308
  ### Child events
461
309
 
462
- Hook bubbling can't cross an invocation boundary, so a parallel child's events can't reach the parent's bus *as they happen*. Instead, each child's transcript is **replayed onto the parent's bus when it merges** — so `src/output/terminal.ts`, the GUI engine, and any `child:tool:*` listener render parallel children with no changes. Turn it off with `emitChildEvents: false`.
463
-
464
- | Fires | From |
465
- |---|---|
466
- | `child:stream:text` | assistant text, one chunk per turn |
467
- | `child:tool:before` | each `tool_call` block, with a run-cumulative `runToolCounts` tally |
468
- | `child:tool:after` / `child:tool:error` | the matching `tool_result`, split on `isError` |
469
- | `child:turn:after` | each assistant turn, `usage` straight off the turn |
470
- | `child:stream:end` | end of each turn |
471
-
472
- Every event carries `childId` (the reserved label), `depth`, and — on the tool events, which extend `ToolHookContext` — `runId`, `parentRunId`, and `parentCallId` (the spawn call that owns the child). Per-child blocks stay contiguous and follow **submission** order, not settlement order.
473
-
474
- **Not synthesized**: `tool:gate`, `tool:transform`, `tool:dispatched`, `tool:cancelled`, `background:*`. Those describe decisions taken inside another process; inventing them would be fiction.
475
-
476
- **They re-fire on a parent replay**, exactly as the in-process tool's `child:tool:*` does — the drain runs live on every attempt. Dedupe by `callId` / `turnId`, or gate on `createReplayTracker`. The fixture pins this rather than asserting it in prose: a post-merge replay emits 24 events for 12 events' worth of work, measured through a file sink that outlives the attempt.
477
-
478
- #### Nested fan-out
479
-
480
- A child dispatched to a service runs **sessionless**, so it has nowhere to persist its own children. Left alone, a depth-2 fan-out loses everything below depth 1 — the parent sees the child's `spawn` tool calls and none of the work they did. Hand the subtree up instead:
310
+ Hook bubbling can't cross an invocation boundary, so each child's transcript is **replayed onto the parent's bus when it merges** (`child:stream:text`, `child:tool:before/after/error`, `child:turn:after`, `child:stream:end`) — existing `child:tool:*` listeners need no changes. Every event carries `childId`, `depth`, and (tool events) `runId`/`parentRunId`/`parentCallId`; blocks stay contiguous per child in submission order. Not synthesized: `tool:gate`, `tool:transform`, `tool:dispatched`, `tool:cancelled`, `background:*` (decisions taken inside another process).
481
311
 
482
- ```ts
483
- const merged: MergedChild[] = []
484
- const spawn = restateSpawnTool({
485
- onChildMerged: child => merged.push(child),
486
- dispatch: req => ctx.serviceClient(childService).runChild(req),
487
- })
488
- // … and return it from the handler
489
- return { status: 'completed', text, turns: agent.turns, stats, children: merged }
490
- ```
491
-
492
- The merging parent walks that subtree depth-first, **re-labeling** each descendant from its own session counter (every level mints `child-1` locally, so the arriving labels collide), and emits their `child:*` events at the right depth. Verified end to end: parent → one child → two grandchildren lands four runs at depths 0/1/2/2 with labels `child-1` / `child-2` / `child-3`, and the grandchildren's tool events arrive at `depth: 2`.
312
+ They **re-fire on a parent replay** — dedupe by `callId`/`turnId`, or gate on `createReplayTracker`. Disable with `emitChildEvents: false`.
493
313
 
494
- #### If you need it live
314
+ **Nested fan-out**: a child dispatched to a service is sessionless, so a depth-2 fan-out loses everything below depth 1 unless the subtree is handed up via `onChildMerged: child => merged.push(child)`, returned as `children: merged` from the handler. The merging parent walks it depth-first, re-labels each descendant from its own session counter (every level mints `child-1` locally), and emits `child:*` at the right depth.
495
315
 
496
- Publish from the child's own bus inside your handler; the child is an ordinary agent there and its hooks fire normally. `ChildRunRequest` carries `parentRunId`, `parentCallId` and `sessionId` so the child can tag what it publishes and your UI can group it with the parent's own events. Two caveats:
497
-
498
- - `restateProvider(..., { streamCallbacks: 'suppress' })` no-ops `onText`, so text deltas never fire even inside the child. Drop `suppress` and you get `maxRetryAttempts: 1` instead — otherwise an entry retry re-emits deltas and duplicates text.
499
- - A child's events re-fire on the *child's* own replay, so the sink must be idempotent.
500
-
501
- What's deliberately not offered is journaling each child event so the parent can consume it as a journal entry. That is the only live option that stays inside the determinism envelope, and it costs a journal entry per event plus a parent that must stay awake to read them — which defeats the suspension that makes long children cheap. Defensible at coarse milestones, not per token.
502
-
503
- Every property here is pinned by the fixture against a real server: two children overlapping by ~394ms of their 400ms of work; four children at a cap of two running as two clean waves (max concurrency exactly 2, wall time 1031ms versus 2090ms for the same fan-out at cap 1); a crash inside one child leaving its sibling with exactly one execution; and both children's transcripts arriving on the parent's bus as `child:*` events, in contiguous per-child blocks.
316
+ **For live events instead**, publish from the child's own bus inside your handler (it's an ordinary agent there); `ChildRunRequest` carries `parentRunId`/`parentCallId`/`sessionId` for tagging. `restateProvider(..., { streamCallbacks: 'suppress' })` no-ops `onText` even inside the child — drop `suppress` if you need live text. A child's events re-fire on the child's own replay, so the sink must be idempotent.
504
317
 
505
318
  ## Durable tool state
506
319
 
507
- The problem it solves: **a tool cannot keep state in a closure.** A journaled result skips the tool body on replay, so a `consumed = true` written inside `execute` never happens the second time around — while the closure itself was rebuilt from the original input. The tool then re-serves a payload it already delivered.
320
+ **The problem**: a tool cannot keep state in a closure. A journaled result skips the tool body on replay, so a `consumed = true` written inside `execute` never happens the second time — the tool re-serves a payload it already delivered.
508
321
 
509
322
  `ToolContext.state` is a key/value map that survives replays, suspensions, and whole invocations:
510
323
 
511
324
  ```ts
512
- const readErrors: ToolDef = {
513
- spec: { name: 'read_errors', description: 'Report build errors once', inputSchema },
514
- async execute(_input, ctx) {
515
- if (ctx.state?.get<boolean>('delivered'))
516
- return 'already delivered'
517
- ctx.state?.set('delivered', true)
518
- return problems
519
- },
325
+ async execute(_input, ctx) {
326
+ if (ctx.state?.get<boolean>('delivered')) return 'already delivered'
327
+ ctx.state?.set('delivered', true)
328
+ return problems
520
329
  }
521
-
522
- createAgent({
523
- tools: wrapAgentTools({ read_errors: readErrors }, ctx),
524
- toolState: restateToolStateStore(ctx),
525
- })
330
+ // createAgent({ tools: wrapAgentTools({ read_errors: readErrors }, ctx), toolState: restateToolStateStore(ctx) })
526
331
  ```
527
332
 
528
- ### The three phases
529
-
530
- Reads are synchronous by design. A wrapped tool body runs inside `ctx.run`, and Restate forbids context use in there (no `ctx.get`, no `ctx.sleep`, no nested `ctx.run`) — so the durable I/O happens on either side of the body instead:
333
+ **Three phases**, because Restate forbids context use (`ctx.get`/`ctx.sleep`/nested `ctx.run`) inside a `ctx.run` body:
531
334
 
532
335
  | Phase | When | Where |
533
336
  |---|---|---|
534
- | **hydrate** | Once per **agent instance**, before the first tool dispatch | Agent, outside every journaled action — one `ctx.get` |
535
- | **stage** | Inside the tool body | In memory; `set` / `delete` record ops, they are not written through |
536
- | **commit** | When the call settles | Loop, outside the journaled action — one `ctx.set` |
537
-
538
- The load is memoized on the controller, and the controller is built once in `createAgent` — so a long-lived agent doing many runs loads exactly once. Under Restate that's one agent per invocation and the distinction doesn't bite, but elsewhere it means a resident agent never picks up state another process wrote to the same store out of band. Build a fresh agent (or a fresh controller) per unit of work if you need that.
337
+ | hydrate | Once per agent instance, before the first tool dispatch | One `ctx.get`, outside every journaled action |
338
+ | stage | Inside the tool body | In memory — `set`/`delete` record ops, not written through |
339
+ | commit | When the call settles | One `ctx.set`, outside the journaled action |
539
340
 
540
- The invariant that makes it replay-safe: **a staged mutation is committed only if it came back out of a journal entry.** `restateTool` reads the staged ops inside its journaled closure and ships them out as part of the journaled value (`{ __zidaneToolState: 1, ops, output }`), then re-seeds them outside it. On replay the body never runs, the ops come from the journal, and the commit re-issues an identical `ctx.set`. Attempts that produced no journal entry (control error, abort, dead process) have their staged ops dropped — the replay re-runs the body and stages them again.
341
+ The load is memoized on the controller (built once per `createAgent`), so one invocation loads exactly once. Replay safety: **a staged mutation commits only if it came back out of a journal entry** — `restateTool` ships staged ops as part of the journaled value (`{ __zidaneToolState: 1, ops, output }`) and re-seeds them outside it. Attempts with no journal entry (control error, abort, dead process) drop their staged ops; replay re-runs the body and re-stages.
541
342
 
542
- ### Semantics
343
+ **Semantics**:
344
+ - Configuring `toolState` at all changes the journal shape (adds a run-start `ctx.get`) — drain in-flight invocations before rolling it out. A call that writes no state journals unchanged.
345
+ - Commit order is call-index order, never completion order — a durable fleet joined via `RestatePromise.all()` commits only once fully settled.
346
+ - A write is visible once its call commits, not when staged (read-through to the live snapshot). Soft tool failures still commit — sentinel and delta land in the same journal entry.
347
+ - A store **failure drops the write, never fails the call** (the tool body already ran). Fires `tool-state:commit-failed` with the dropped ops (set `handled = true` to own reporting), else a stderr fallback. Make `commit` idempotent if the write matters.
348
+ - `ToolStateView.set` caps values at 64 KiB of JSON (`restateToolStateStore(ctx, { maxValueBytes })` to override) and throws inside the tool body — an oversized write that reached commit would instead fail the whole invocation with `RT0003`. Non-serializable values are rejected the same way.
349
+ - A failing `load` fails the run — asymmetric with `commit` on purpose, since running on an empty snapshot re-arms the bug this exists to prevent. `set(key, undefined)` deletes.
350
+ - Two-phase tools (`durableStart`) journal their delta on the final entry (`wait.result`/`attach`) — a write staged in `start` is lost if the process dies before the wait completes.
351
+ - Keep values small and JSON-serializable (flags, ids, counters, cursors, not transcripts). Not inherited by subagents — `createSpawnTool` has no `shareToolState`; a spawned child's tools get an isolated in-memory map unless you construct its agent explicitly with the same store.
352
+ - The store is bound to one invocation's `ctx` — build it inside the handler, like `restateSessionStore`/`restateProvider`. It's one JSON object with no built-in versioning; use `restateToolStateStore(ctx, { key })` for a clean namespace on migration.
543
353
 
544
- - **Envelope is conditional.** A call that writes no state journals exactly what it journaled before this feature existed, so in-flight invocations replay unchanged across the deploy. What *does* change the journal shape is configuring `toolState` at all (it adds the run-start `ctx.get`) — drain in-flight invocations before rolling it out.
545
- - **Commit order is call-index order**, never completion order — including durable fleets joined through `RestatePromise.all()`. Two parallel tools writing the same key would otherwise produce a nondeterministic write sequence.
546
- - **A write becomes visible when its call commits**, not when it is staged. Views read through to the live snapshot, so a call that runs *after* a sibling committed does see that write (the sequential path `restateBehavior` defaults to reads back the previous call's write). A durable fleet commits only once the whole batch has settled, so calls racing inside one batch never see each other.
547
- - **Errors still commit.** A soft tool failure journals its sentinel and its delta in the same entry, so the write is honored — live and replay agree.
548
- - **A store failure drops the write, it doesn't fail the call.** The tool body already ran, so failing the run over a bookkeeping write would be worse than losing it — but losing it silently is worse still, since a dropped one-shot flag re-arms the exact bug this state prevents. A rejecting `commit` fires the `tool-state:commit-failed` hook with the ops that did NOT land (replaying them by hand is a legitimate recovery), and falls back to an unconditional stderr line when nothing is listening; set `handled = true` in your handler to own the reporting. A throwing handler can't take the dispatch path down with it. Make `commit` idempotent and retry inside it if the write matters more than latency.
549
- - **Oversized values are rejected where they're written.** `ToolStateView.set` enforces a per-value budget (`DEFAULT_MAX_VALUE_BYTES`, 64 KiB of JSON; override with `restateToolStateStore(ctx, { maxValueBytes })`) and throws inside the tool body rather than letting the write fail at commit time. That matters more under Restate than elsewhere: an oversized state value doesn't fail politely, it fails the invocation with `RT0003` (MessageSizeLimit) and retries it under the configured policy — a fat write is a poison pill, not a dropped write. Non-serializable values (cycles, functions) are rejected the same way.
550
- - **A failing `load` fails the run**, deliberately asymmetric with `commit`: running on an empty snapshot would re-arm the one-shot bugs this state exists to prevent. It throws from `agent.run()` on the normal error path (`errorRun` + `agent:done` still fire), and the memo is cleared so the next run retries.
551
- - **`set(key, undefined)` is a deletion.** It is what a read would report anyway, and an undefined value is not JSON-stable across the journal.
552
- - **Two-phase tools** (`durableStart`) journal their delta on the final entry (`wait.result` / `attach`). A write staged in `start` is lost if the process dies before the wait completes; write state in the final phase when it must be crash-durable.
553
- - **Keep values small and JSON-serializable.** They cross a journal and land in virtual-object state, which Restate size-caps: flags, ids, counters, cursors — not transcripts.
554
- - **Not inherited by subagents.** A child you construct yourself can take the same store; `createSpawnTool` has **no** `shareToolState` equivalent to `shareReadState`, so tools running inside a *spawned* subagent get an isolated in-memory map and their writes never reach the parent's store. This is the one thing a subagent does NOT get under Restate — everything else (journaled clock, wrapped tools/provider, session, handle) is covered in § Subagents. A child's one-shot delivery is therefore not crash-safe; keep such tools in the parent, or hand the child an explicitly constructed agent with the same `toolState` store.
555
- - **The store is bound to one invocation's `ctx`.** Build it inside the handler, like `restateSessionStore` and `restateProvider`. An agent constructed once at module scope and reused across invocations would commit through a dead context — and never re-hydrate. `toolState` reads like agent-lifetime configuration, which makes this easier to get wrong than the other ctx-bound wrappers.
556
- - **The stored map is a schema you own.** It is one JSON object under one state key, with no versioning or migration built in. Renaming a key or changing a value's shape has to tolerate the old value still being there from a previous deploy; `restateToolStateStore(ctx, { key })` gives you a clean namespace when a migration is easier than a tolerant read.
557
-
558
- ### Without Restate
559
-
560
- `ToolStateStore` is two methods (`load`, `commit`) and nothing in it is Restate-specific — implement it against Redis, Postgres, or a file for other hosts. With no store configured `ctx.state` is still present and functional, backed by an in-memory map scoped to the agent instance, so a tool written against it behaves identically on the local CLI path and only loses durability.
354
+ **Without Restate**: `ToolStateStore` is just `{ load, commit }` — implement it against Redis/Postgres/a file for other hosts. With no store configured, `ctx.state` still works, backed by an in-memory map scoped to the agent instance (functional, not durable).
561
355
 
562
356
  ## Recommended `AgentBehavior` for Restate deployments
563
357
 
564
358
  ```ts
565
359
  createAgent({
566
- // ... wrappers as above
567
- behavior: restateBehavior({
568
- maxCostUsd: 5,
569
- maxTotalTokens: 2_000_000,
570
- }),
360
+ behavior: restateBehavior({ maxCostUsd: 5, maxTotalTokens: 2_000_000 }),
571
361
  })
572
362
  ```
573
363
 
574
- - **`maxConcurrentTools: 1`** — strict Restate default. Enable parallelism explicitly with `restateToolBatchExecutor`.
575
- - **`disableBackgroundTasks: 'non-durable'`** — capability-aware: background tasks (`shell` with `run_in_background: true`) stay enabled only when the execution context declares `capabilities.detachedTasks: 'durable'` (tasks live on a remote runner and survive worker crashes). For host-local `ProcessContext` this is equivalent to the old blanket `true` — those processes aren't journaled and don't survive process death.
576
- - **`cache: true`** (default) — prompt-cache breakpoints reward stable system/tool prefixes. Cuts cold-start cost on retries when the cache is still warm.
577
- - **`shouldRethrowToolError`** — Restate terminal/suspension control errors escape the tool-result path.
578
- - **`maxCostUsd` / `maxTotalTokens`** — server-side budget circuit breaker. Throws `AgentBudgetExceededError` and finalizes the session run as `aborted`. Especially useful here because durable replay + retry means a runaway loop is a runaway *bill*; a soft cap stops it cleanly. See ARCHITECTURE.md → run-level budget circuit breaker.
364
+ | Default | Why |
365
+ |---|---|
366
+ | `maxConcurrentTools: 1` | Strict-first. Enable parallelism explicitly via `restateToolBatchExecutor`. |
367
+ | `disableBackgroundTasks: 'non-durable'` | Background tasks (`shell` with `run_in_background`) stay enabled only when the context declares `capabilities.detachedTasks: 'durable'` — host-local tasks aren't journaled. |
368
+ | `cache: true` | Stable system/tool prefixes reward prompt-cache breakpoints, cutting cold-start cost on retries. |
369
+ | `shouldRethrowToolError: shouldRethrowRestateControlError` | Restate terminal/suspension control errors escape the tool-result path instead of becoming a fake tool error. |
370
+ | `compactStrategy: 'tail'`, `compactThreshold: 128 KiB`, `compactKeepTurns: 8`, `elideStaleReads: 'stub'`, `mediaKeepTurns: 8`, `toolOutputBudget: 64 KiB`, `autoCompact: { threshold: 0.85 }` | Shrinks the per-turn provider payload — journal weight and latency, not just tokens. |
371
+
372
+ Add `maxCostUsd`/`maxTotalTokens` as a server-side budget breaker: throws `AgentBudgetExceededError` and finalizes the run as `aborted` — durable replay+retry otherwise turns a runaway loop into a runaway bill. See `docs/ARCHITECTURE.md` § run-level budget circuit breaker.
579
373
 
580
- ### MCP tools
374
+ ## MCP tools
581
375
 
582
- MCP servers re-bootstrap on every handler invocation (`warmup()` is lazy + idempotent). Discovered MCP tools aren't reachable through the public tool-wrapping seam (`wrapAgentTools` only sees the tools you pass), so under Restate their raw `callTool` never rides `ctx.run` and **every journal replay re-runs the real MCP call** — a side-effecting tool fires once per replay. Close the gap with the **`mcpToolWrap`** hook:
376
+ MCP servers re-bootstrap on every handler invocation (`warmup()` is lazy + idempotent). Discovered MCP tools aren't reachable through the public tool-wrapping seam (`wrapAgentTools` only sees tools you pass), so their raw `callTool` never rides `ctx.run` unless you wire `mcpToolWrap`:
583
377
 
584
378
  ```ts
585
- const agent = createAgent({
379
+ createAgent({
586
380
  provider: restateProvider(anthropic({ defaultModel: 'claude-sonnet-4-5' }), ctx),
587
- tools: wrapAgentTools(nativeTools, ctx), // native tools: journaled
381
+ tools: wrapAgentTools(nativeTools, ctx),
588
382
  mcpServers,
589
- mcpToolWrap: restateMcpToolWrap(ctx), // discovered MCP tools: journaled too
383
+ mcpToolWrap: restateMcpToolWrap(ctx), // discovered MCP tools: journaled too
590
384
  clock: restateClock(ctx),
591
385
  session,
592
386
  behavior: restateBehavior(),
593
387
  })
594
388
  ```
595
389
 
596
- The agent owns discovery (`mcp_<server>_<tool>` namespacing, per-server `instructions`, lazy connect, hooks, OAuth) and applies the wrap to each discovered tool — eager or lazily-connected, and regardless of which `mcpConnector` produced them. The wrap only re-shapes `execute`.
390
+ The agent applies the wrap to every discovered tool regardless of `mcpConnector` or lazy-connect timing; the wrap only re-shapes `execute`.
597
391
 
598
- - **Default is plain result-journaling** — correct for the short request/response shape of most MCP calls. A genuinely long-running MCP tool (a remote build, a migration) can opt into two-phase `durableStart` per-tool via the `override` option; `exclude` skips journaling for deterministic reads.
599
- - **Error semantics are preserved.** MCP tools signal failure by throwing; left naked, that throw exhausts `ctx.run`'s retries → Restate `TerminalError` → `restateBehavior` re-throws it → a recoverable MCP error becomes a *fatal* run. `restateMcpToolWrap` journals with `restateTool`'s `softErrors` on (the same soft-error mechanism native tools get): the failure is captured inside the journaled closure and re-thrown as a plain (non-control) error outside it, so the loop still produces a `tool_result { isError }` and continues — and replay reproduces the same error without re-calling. Real Restate control / abort errors pass through untouched.
600
- - **Custom transport / auth.** Journaling is orthogonal to connection — keep your own `mcpConnector` (e.g. an OAuth-aware `connectMcpServers(configs, undefined, hooks, { buildAuthProvider })`) and add `mcpToolWrap: restateMcpToolWrap(ctx)` alongside it. The hook wraps whatever the connector discovers.
392
+ - Default is plain result-journaling. A genuinely long-running MCP tool (a remote build, a migration) can opt into two-phase `durableStart` per-tool via `override`; `exclude` skips journaling for deterministic reads.
393
+ - Error semantics match native tools: `restateMcpToolWrap` applies the same `softErrors` mechanism, so a thrown MCP failure becomes `tool_result { isError }` instead of a fatal `TerminalError`.
394
+ - Journaling is orthogonal to connection — keep any custom `mcpConnector` (OAuth, custom transport) and add `mcpToolWrap: restateMcpToolWrap(ctx)` alongside it.
601
395
 
602
396
  Hosts that journal bootstrap outcomes should hook the deterministic `mcp:bootstrap:settled` aggregate (one name-sorted event per batch) instead of the per-server `mcp:bootstrap:end` stream, which fires in network-completion order.
603
397
 
604
- Host-only configuration rides in `behavior.extensions` — a namespaced bag the framework never reads but threads through behavior resolution onto `ToolContext.behavior.extensions`. Keep custom keys there instead of patching `AgentBehavior` with unknown top-level fields.
398
+ Host-only configuration rides in `behavior.extensions` (a namespaced bag threaded onto `ToolContext.behavior.extensions`) rather than unknown top-level `AgentBehavior` fields.
605
399
 
606
400
  ## Long-running background jobs (durable contexts)
607
401
 
608
- A remote execution context whose `capabilities.detachedTasks` is `'durable'` can run `shell({ run_in_background: true })` jobs that outlive worker crashes. Three seams make this journal-safe:
609
-
610
- 1. **Pull-based exit reconcile.** The context does NOT need to call `execBackground`'s `onExit` (it's optional — a remote context has no legal way to push from a timer into a journal). At the start of every `agent.run()` the loop sweeps `listBackground` for terminated tasks the model hasn't seen and merges them with any push-fed exits. `TaskEntry.endedAt` makes `durationMs` sweep-time-independent.
611
- 2. **Deterministic reconcile (`journalReconcile`).** Pass `journalReconcile` on `agent.run()` to record the `listBackground` sweep as a single journaled side effect — replay returns the recorded snapshot instead of re-polling the registry live, so the (conditionally-fired) `background:reconcile` hook lands at a stable journal position across attempts. Everything downstream of the snapshot is pure over journaled/replayed inputs, so this is enough to make the whole reconcile replay-deterministic *without* journaling the entire `ExecutionContext`:
612
-
613
- ```ts
614
- await agent.run({
615
- prompt,
616
- clock: restateClock(ctx),
617
- journalReconcile: sweep => ctx.run('background-reconcile', sweep),
618
- })
619
- ```
620
-
621
- The `background:reconcile` hook still fires on the merged exit list inside the `run()` await chain — mutate `hookCtx.exits` to filter / reorder what the model sees. (Before `journalReconcile`, hosts journaled inside that hook via `hookCtx.exits = await ctx.run('background-reconcile', () => hookCtx.exits)`; that still works, but `journalReconcile` is the deterministic seam because it also journals the underlying `listBackground` poll, not just the post-sweep list.)
622
-
623
- 3. **`waitBackground` / `wait_task`.** When the model genuinely needs a task's outcome before continuing, the `wait_task` tool delegates to `ExecutionContext.waitBackground` — an injectable seam. In-process contexts park on their own `onExit` machinery; durable hosts implement it as an awakeable park (runner fires task-exit over their bridge → `resolveAwakeable`), so a crashed worker re-attaches to the same wait on replay. `wrapAgentTools({ detachedTasks: 'durable' })` automatically leaves `wait_task` unwrapped — a Restate suspension point must not nest inside a journaled `ctx.run` closure (same rule as `restateAwakeableTool`). Hosts whose wait is journal-safe (e.g. a bridge RPC poll) can re-wrap it explicitly via `restateTool`.
624
-
625
- Because `wait_task` is excluded from journaling its body re-runs on every replay, so on a `'durable'` context it must not branch on an un-journaled read. It therefore skips its "does this task exist?" `listBackground` pre-check and lets `waitBackground`'s own journaled probe decide — an unknown task collapses into the wait's `null` (the timeout result), replay-stable even if the runner's registry changed across attempts (e.g. a restart mid-park). Non-durable contexts keep the precise "no such task" message (no journal to diverge).
402
+ General background-task mechanics (`execBackground`, `listBackground`, the `waitBackground` seam, output caps, stall watchdog, `detachedTasks` capability) are owned by `docs/RUN_IN_BACKGROUND.md`. This section covers only the Restate-specific wiring.
626
403
 
627
- `restateWaitBackground` packages the full race-safe choreography (mint the awakeable → register the resumer **before** probing `listBackground` → fast-path if the task already exited → park outside the journal, raced against the durable deadline → settle). The host supplies only persistence callbacks:
404
+ 1. **`restateWaitBackground`** packages the race-safe choreography for `ExecutionContext.waitBackground`: mint an awakeable → register the resumer id **before** probing `listBackground` (probing first risks a lost wakeup) → fast-path if already exited/unknown → park outside the journal, raced against the durable deadline → settle. The host supplies only persistence callbacks (`register`, `probe`, `settle`):
628
405
 
629
406
  ```ts
630
- import { restateWaitBackground } from 'zidane/restate'
631
-
632
407
  execution.waitBackground = restateWaitBackground({
633
408
  ctx,
634
- // Persist the awakeable id as the task's exit resumer BEFORE the probe.
635
409
  register: (awakeableId, taskId) => persistExitResumer(taskId, awakeableId), // runner → ctx.resolveAwakeable(awakeableId, exitInfo)
636
- // Snapshot the registry (the `listBackground` shape) to detect already-exited / unknown tasks.
637
410
  probe: handle => execution.listBackground!(handle),
638
- // Finalize the resumer row (exit info, or null on timeout / abort / unknown).
639
411
  settle: (awakeableId, taskId) => clearExitResumer(taskId),
640
412
  })
641
413
  ```
642
414
 
643
- `awakeableWithTimeout` (which `restateWaitBackground` builds on) remains available if you need a bespoke wait — `RestateContextLike` carries `sleep(millis)` and it does the race:
644
-
645
- ```ts
646
- import { RestatePromise } from '@restatedev/restate-sdk'
647
- import { awakeableWithTimeout } from 'zidane/restate'
648
-
649
- const { id, promise } = awakeableWithTimeout<TaskExitInfo>(ctx, { timeoutMs: opts?.timeoutMs, promises: RestatePromise })
650
- await registerExitResumer(taskId, id) // runner → ctx.resolveAwakeable(id, exitInfo)
651
- return promise // TaskExitInfo, or null on timeout
652
- ```
653
- 4. **Cross-invocation dedupe is history-derived.** The agent's in-memory notified-task latch dies with the handler invocation, but the persisted session turns carry every acknowledgment (`<task-notification>` blocks, `shell_kill` calls, successful `wait_task` results, post-exit `read_file`s of the output path). The reconcile sweep re-derives the latch from the **full** `session.turns` history — subagent-run acks included — so a fresh agent on a resumed session (the normal Restate shape) never re-notifies an exit a prior invocation already delivered. This survives the built-in compaction: `compact-summary` markers elide turns at the wire level only, persisted turns are append-only by contract (the same assumption lazy-disclosure replay and read-state hydration make). Read-acks cross two clock domains (host `AgentClock` vs runner wall clock), so they apply only when the read landed ≥ 5 s after the exit — within that window the agent fails open (one redundant notification, never false suppression). Only a host that rewrites `session.turns` destructively forfeits the dedupe; remote registries can prune terminated entries after delivery as a belt-and-braces measure there.
654
-
655
- Guardrails for unattended jobs (any context): `behavior.backgroundOutputCap` caps each task's log file without killing the process (a `<output-truncated bytes-dropped="N"/>` marker is appended at settle), and `behavior.backgroundStallWatchdogMs` queues a one-shot `<task-stall>` notification when a task goes quiet — typically a process stuck at an interactive `(y/n)` prompt that the model should `shell_kill` and re-run non-interactively.
415
+ `wrapAgentTools({ detachedTasks: 'durable' })` leaves `wait_task` unwrapped automatically (same no-suspension-inside-`ctx.run` rule as `restateAwakeableTool`) and it skips its own `listBackground` pre-check there, deferring to the journaled probe inside `restateWaitBackground` instead.
656
416
 
657
- ### Skills in worker/runner deployments
417
+ 2. **`journalReconcile`** records the per-run `listBackground` sweep as one journaled side effect, keeping the `background:reconcile` hook at a stable journal position across attempts: `agent.run({ prompt, clock: restateClock(ctx), journalReconcile: sweep => ctx.run('background-reconcile', sweep) })`.
658
418
 
659
- If the agent loop runs on a Restate worker but the skill files live on a separate runner (the `skills_read` / `skills_run_script` tools route file access through `ctx.execution`), pre-resolve the catalog on the worker and pass it via `SkillsConfig.catalog` instead of `scan` / `write`. The default disk discovery is bypassed — `parseSkillFile` is never called — so the worker never tries to `readFileSync` SKILL.md files that don't live on its disk.
660
-
661
- ```ts
662
- createAgent({
663
- // … wrappers as above
664
- skills: {
665
- skipDefaultPaths: true, // worker has no skill dirs of its own
666
- catalog: hostProvidedCatalog, // pre-resolved by the worker bootstrap
667
- },
668
- })
669
- ```
670
-
671
- `baseDir` / `location` on each catalog entry must point to paths resolvable on the runner (where `skills_read` / `skills_run_script` ultimately execute). Catalog entries seed resolution first and shadow disk-discovered skills on name collision — useful if a project still wants to layer `.zidane/skills/*` on top of a host-provided catalog.
419
+ 3. **Cross-invocation notification dedupe is history-derived.** The in-memory notified-task latch dies with the handler invocation; the reconcile sweep re-derives it from the full `session.turns` history (`<task-notification>` blocks, `shell_kill` calls, `wait_task` results, post-exit `read_file`s), so a fresh agent on a resumed session never re-notifies an exit already delivered. Read-acks apply only ≥5s after the exit (fails open otherwise). Requires `session.turns` to stay append-only.
672
420
 
673
421
  ## Edge & long-lived deployments
674
422
 
675
- The journal makes the *loop* durable; these seams make the rest of the runtime durable on a long-lived / edge host (a Restate worker, Cloudflare Workers, etc.).
423
+ The journal makes the *loop* durable; these seams cover the rest of the runtime on a long-lived/edge host.
676
424
 
677
- ### Re-attach to the execution context across invocations
678
-
679
- The agent spawns its execution context lazily on the first run (`execution.spawn()`), which is fine for a single invocation — but that spawn is NOT a journaled side effect, so on a crash mid-run the replay would mint a *fresh* sandbox/container and orphan the first. For a context the host owns across invocations, drive the spawn yourself (so it's journaled) and hand the agent the handle:
425
+ **Re-attach the execution context across invocations.** `execution.spawn()` isn't journaled, so a crash mid-run would replay into a *fresh* sandbox on retry. Drive the spawn yourself and hand the agent the handle:
680
426
 
681
427
  ```ts
682
- import { createSandboxContext } from 'zidane/contexts/sandbox'
683
- import { createDaytonaProvider } from 'zidane/contexts/daytona'
684
-
685
- // Journal the spawn → replay returns the same sandbox id, no re-mint.
686
428
  const handle = await ctx.run('spawn-sandbox', () => execution.spawn())
687
-
688
- const agent = createAgent({
689
- provider: restateProvider(inner, ctx),
690
- execution,
691
- handle, // used verbatim — agent does NOT spawn, and does NOT destroy it
692
- clock: restateClock(ctx),
693
- behavior: restateBehavior(),
694
- })
429
+ createAgent({ provider: restateProvider(inner, ctx), execution, handle, clock: restateClock(ctx), behavior: restateBehavior() })
695
430
  ```
696
431
 
697
- - **`AgentOptions.handle`** — when set, the agent skips its internal `spawn()` and never `destroy()`s the handle on teardown (the host owns that lifecycle).
698
- - **`execution:ready` hook** — if you'd rather let the agent spawn and just *observe* the minted id (to journal it for the next invocation's re-attach), this fires once with `{ handle, spawned }`.
699
- - **`createSandboxContext`** re-attaches by handle id: its handle id IS the sandbox id, so a handle from a prior invocation routes operations to the same sandbox even though this instance never ran `spawn()`. Pair with the provider's reconnect option (`createDaytonaProvider({ sandboxId })`).
700
- - **`createDaytonaProvider({ ownAttached: true })`** — by default an attached (`sandboxId`) sandbox is treated as externally owned and left running on `destroy()`. Set `ownAttached` when the *host* owns it across invocations so `destroy()` applies `onDestroy` (stop / archive / delete). Spawn-time failures still never delete a pre-existing attached sandbox.
701
- - **`zidane/contexts/sandbox` subpath** — import `createSandboxContext` from here (not the `zidane/contexts` barrel) on workerd/edge: this entry's graph is node-free, so it doesn't drag `createProcessContext` → `node:child_process` into the bundle.
702
- - **`@daytona/sdk` needs a root `tslib` override.** It is a peer dependency, so *you* install it — and it declares no `tslib` dependency of its own while reaching AWS SDK v3 through `@aws-sdk/client-s3`. A stale `@aws-crypto` v3/v4 floor (`tslib: ^1.11.1`) can win the hoist, and the AWS SDK's initializer helpers then resolve against tslib 1.x, which lacks them — `import('@daytona/sdk')` fails with `Export named '__runInitializers' not found`. npm/bun overrides are **root-only** and are not inherited from zidane, so add this to your own `package.json`:
703
-
704
- ```json
705
- { "overrides": { "tslib": "^2.8.1" } }
706
- ```
432
+ `AgentOptions.handle` skips the agent's internal `spawn()` and its `destroy()` on teardown (the host owns that lifecycle) — see `docs/SKILL.md` § `execution:ready` for observing a freshly-minted handle to journal for next time. `createSandboxContext`'s handle id IS the sandbox id, so a prior handle re-attaches by id; pair with a reconnect option like `createDaytonaProvider({ sandboxId, ownAttached: true })`. Import `createSandboxContext` from the `zidane/contexts/sandbox` subpath (not the barrel) on workerd/edge to keep `node:child_process` out of the bundle.
707
433
 
708
- Safe because tslib 2 is a superset of the v1 helper set; the `^1.11.1` requirements are stale floors from packages AWS has since moved off. `createDaytonaProvider` reports the underlying load error (and names tslib) rather than claiming the package is missing.
709
-
710
- ### Streaming on fresh vs resumed invocations
711
-
712
- `createReplayTracker().isReplaying()` is conservatively `true` until the first journaled closure executes — correct for deferred abort, but on a *fresh* run it would briefly suppress opening output for a streamer that gates on it. Pass `resuming` (the host knows whether this invocation is a crash-recovery replay — e.g. the bound object already has persisted turns) and gate live streaming on `isResuming()` instead: `false` from the start of a fresh run, `true` only while replaying a resumed journal.
434
+ **Streaming on fresh vs. resumed invocations.** `createReplayTracker().isReplaying()` is conservatively `true` until the first journaled closure executes, which would briefly suppress opening output on a *fresh* run. Pass `resuming` (the host knows whether this invocation is a crash-recovery replay) and gate live streaming on `isResuming()` instead:
713
435
 
714
436
  ```ts
715
437
  const tracker = createReplayTracker({ hooks: agent.hooks, resuming: hasPriorTurns })
716
438
  // stream:text consumer: emit when `!tracker.isResuming()`
717
439
  ```
718
440
 
719
- ### Auth on long / suspended runs
441
+ ### Replay-safe abort
720
442
 
721
- A durable run can outlive a short-lived provider token (a gateway minting signed, expiring work tokens). `anthropic({ extraHeaders })` accepts a **function** resolved per request, so each turn (and each replay-time live call) can carry a freshly-minted token without rebuilding the provider:
443
+ `agent.abort()` during a journal replay is undefined behavior — it throws `AgentAbortedError` at whatever point the replay cursor is, and the host's catch arm typically journals something mid-replay, permanently mismatching the journal on every subsequent retry. `replayAwareAbort(tracker, () => agent.abort())` defers an external abort until the first live (non-journaled) execution — the earliest point where the resulting throw appends to the journal instead of forking it:
722
444
 
723
445
  ```ts
724
- anthropic({
725
- baseURL: gatewayUrl,
726
- extraHeaders: async () => ({ authorization: `Bearer ${await mintWorkToken()}` }),
727
- })
446
+ const tracker = createReplayTracker({ hooks: agent.hooks })
447
+ const provider = restateProvider(inner, ctx, { replayTracker: tracker })
448
+ const tools = wrapAgentTools(agentTools, ctx, { replayTracker: tracker })
449
+ const abort = replayAwareAbort(tracker, () => agent.abort()) // external cancel signal → abort(), not agent.abort()
728
450
  ```
729
451
 
730
- ### Large tool outputs & media
731
-
732
- Large tool-output elision is NOT local-disk-bound: with `behavior.persistDir` set, the loop routes the blob through `ctx.execution.writeFile` (and the model reads it back via `read_file` → `ctx.execution.readFile`), so on a remote/sandbox context the blob lands on that context's filesystem — workerd-safe, no host disk required. Point `persistDir` at a path valid inside the execution context. For media kept *out* of the journal, persist `{ type: 'image' | 'document', ref }` blocks and resolve them at wire time via the **real** `content-ref:resolve` hook (see [Content refs](#content-refs)). There is no separate R2/KV blob backend distinct from the execution filesystem — if you need one, resolve refs through it in the hook.
452
+ **Auth on long/suspended runs.** A durable run can outlive a short-lived provider token — pass `extraHeaders` as a function (e.g. `anthropic({ extraHeaders: async () => ({ authorization: \`Bearer ${await mintWorkToken()}\` }) })`) so each turn, including replay-time live calls, carries a freshly-minted token.
733
453
 
734
- ### Unbounded runs on CPU-capped runtimes
454
+ **Large tool outputs & media.** With `behavior.persistDir` set, large tool-output elision routes through `ctx.execution.writeFile`/`readFile` (not local disk), so it works on a remote/sandbox context. For media kept out of the journal, persist `{ type, ref }` blocks and resolve them at wire time via `content-ref:resolve` (see [Content refs](#restateprovider)).
735
455
 
736
- A runtime with a hard per-invocation CPU/wall cap (Cloudflare's ~5-minute limit) can't host an arbitrarily long *continuous* run — replay CPU accumulates with no built-in tail-call. Use the budget breakers to checkpoint at a turn boundary and re-invoke (Restate's delayed-tail-call / continue-as-new pattern): cap the invocation with `behavior.maxTurns` / `maxWallMs`, persist via `restateSessionStore`, and when a run returns without finishing, schedule a fresh invocation of the same object key that resumes from the persisted turns. The session store is the checkpoint; no dedicated zidane helper is required (and a generic one would have to own the host's re-invocation contract).
456
+ **Unbounded runs on CPU-capped runtimes.** A hard per-invocation CPU/wall cap (e.g. Cloudflare's ~5-minute limit) can't host an arbitrarily long continuous run: cap the invocation with `behavior.maxTurns`/`maxWallMs`, persist via `restateSessionStore`, and when a run returns unfinished, schedule a fresh invocation of the same object key to resume from the persisted turns (Restate's delayed-tail-call pattern) — the session store is the checkpoint.
737
457
 
738
458
  ## Failure modes
739
459
 
740
460
  | Scenario | What happens |
741
461
  |---|---|
742
462
  | Crash between turn N and N+1 | Re-enter the handler. Journal replays turns 1..N from cache. Turn N+1 runs live. |
743
- | Crash mid-tool-call | Re-enter. Journal replays prior turns + already-completed tools. The in-flight tool re-executes (Restate didn't see it complete). Tools that should not be re-run on retry need application-level idempotency (e.g. a `requestId` in the API call). |
744
- | Provider rate limit / 5xx | `restateProvider` wraps `Provider.stream` in `ctx.run` and applies its `runOptions` (default `{ maxRetryAttempts: 3 }`) to that durable side effect. The agent loop's own provider retry policy can still retry failed turns before the error escapes the loop; each retry calls the wrapped provider again and therefore gets its own durable side-effect boundary. Beyond the configured retries, the failure surfaces as `AgentProviderError`. |
745
- | Tool throws | `restateTool`'s default `maxRetryAttempts: 1` — no blind retry. The error is journaled; replay returns the same error. |
746
- | Restate runtime down | Handler invocations queue / fail at the SDK transport layer; Zidane never sees them. Recovery is operational, not in-app. |
463
+ | Crash mid-tool-call | Re-enter. Prior turns + completed tools replay from cache. The in-flight tool re-executes (Restate never saw it complete) — needs application-level idempotency for non-idempotent side effects. |
464
+ | Provider rate limit / 5xx | `restateProvider`'s `runOptions` retries the durable call; the loop's own retry can retry beyond that, each with its own durable boundary. Past all retries: `AgentProviderError`. |
465
+ | Tool throws | `restateTool`'s default `maxRetryAttempts: 1` — no blind retry. The error journals; replay returns the same error. |
466
+ | Restate runtime down | Handler invocations queue/fail at the SDK transport layer; Zidane never sees them — recovery is operational, not in-app. |
747
467
 
748
468
  ## Tests
749
469
 
750
- `test/restate.test.ts` covers each wrapper individually + end-to-end agent.run() with the clock + provider in place. A fake `RestateContextLike` records `ctx.run` calls and can be pre-seeded with a journal to verify replay semantics (inner fn not invoked, cached value returned verbatim). Its `durable tool state` block pairs a recording ctx with a replaying one to assert the live→replay contract end to end: the delta is journaled with the output, replay restores it without entering the body, and the commit is identical.
751
-
752
- `test/tool-state.test.ts` covers the runtime-agnostic half — snapshot hydration, staging, commit-on-settle, the size guard, the commit-failure hook, the in-memory default, and the loop round-trip.
753
-
754
- `bun run restate:fixture:test` is the end-to-end suite against a **real** Restate server in Docker, and is deliberately **not** part of CI — run it locally when you touch the adapter. It boots the container, supervises the fixture service, and drives nineteen scenarios through the real ingress.
755
-
756
- Five cover the adapter itself: baseline agent run; crash → replay (the service process is killed mid-invocation and respawned, so the journal, not a mock, decides what re-runs); per-object-key state isolation; commit-after-soft-failure; and an awakeable suspend/resume.
757
-
758
- Five cover **subagents**, each pinned on an on-disk ledger that has to outlive the process: a crash inside the child; an awakeable park inside the child (suspend → external resume, with the child's `register` phase required to fire exactly once, or the resumed awakeable id would be one nobody resolves); a depth-2 grandchild crash, two un-journaled spawn frames deep; a journaled-deadline trip; and an attempt that fails **without** a process death, so the journal replays against warm module state — that one asserts an unchanged boot count to prove it took the retry path and not the crash path.
759
-
760
- Eight cover **parallel** subagents (`restateSpawnTool`). Two children as separate invocations, asserted to have *overlapping* work intervals — counting executions can't tell parallel from sequential. A crash inside one of them, where the surviving sibling must show exactly one execution: proof that the children really are journal-isolated. Then the scheduling pair: four children in one turn against a fleet cap of two, where the interval ledger must show a maximum of exactly **two** open at once (3+ means the cap is ignored, 1 means the fleet silently fell back to sequential), and the same fan-out at cap 1 as a control, where nothing may overlap. Measured: 1031ms wall at cap 2 versus 2090ms at cap 1 for identical work.
761
-
762
- The remaining three cover the failure modes that decide whether this is production-safe. **A parent crash mid-fan-out**: the parent dies while both children are still working, and on replay must re-attach to the running invocations rather than dispatch new ones — exactly one execution per child, even though the parent frame ran twice. **Depth-2 fan-out**: a child that itself spawns two grandchildren, whose intervals must overlap, so parallelism survives nesting. **Cancellation**, twice: once from outside (the parent invocation is cancelled mid-flight and no child may start another work step) and once from inside (`agent.abort()` with `onAbort` wired, where children finish the step in flight and never begin the next). **A post-merge replay**: the fan-out completes and merges, then the handler fails once, so the parent replays *after* the merge — the only scenario that re-executes the merge path, and the one that caught wall-clock timestamps leaking into the merged run records.
763
-
764
- Both subagent bugs found in review — the `maxWallMs` watchdog clamp and the live `elapsed` inside a persisted string — were caught here and nowhere else. The suite fails if the crash never forced a respawn, so it cannot pass without actually exercising replay. This is the only coverage where replay, suspension, and virtual-object state semantics come from the SDK rather than from our reading of it.
765
-
766
- `test/start.test.ts` covers the local `bun start --restate` runner helpers: option parsing, deployment registration, ingress invocation timeout wiring, readiness/service probing, runtime-state cleanup, cached completion, changed-input rejection, and `status` / `history` handlers.
470
+ - `test/restate.test.ts` — every wrapper in isolation plus end-to-end `agent.run()`, against a fake `RestateContextLike` that records `ctx.run` calls and can be pre-seeded with a journal to assert replay semantics (inner fn not invoked, cached value returned).
471
+ - `test/tool-state.test.ts` — the runtime-agnostic half of durable tool state: hydration, staging, commit-on-settle, the size guard, the commit-failure hook, the in-memory default.
472
+ - `test/start.test.ts` — the local `bun start --restate` runner: option parsing, deployment registration, timeout wiring, readiness probing, runtime-state cleanup, cached completion, changed-input rejection, `status`/`history`.
473
+ - `bun run restate:fixture:test` — end-to-end suite against a **real** Restate server in Docker (`test/fixtures/restate-docker`). Deliberately not part of CI; run it locally when touching the adapter. Covers the adapter itself (crash → replay, per-key isolation, soft-failure commit, awakeable suspend/resume), sequential and parallel subagents (nested fan-out, crash isolation, fleet-cap concurrency), and cancellation (external invocation cancel, in-agent `abort()` with `onAbort`).
767
474
 
768
- The fake structural types live in `src/restate/types.ts` — the adapter typechecks **without** `@restatedev/restate-sdk` installed, useful in monorepos that vend `zidane` upstream of Restate. Consumers wiring the real SDK pass `restate.Context` / `restate.ObjectContext` directly; they're structurally assignable to the in-house shapes.
475
+ The structural types the adapter typechecks against live in `src/restate/types.ts` — `zidane/restate` compiles **without** `@restatedev/restate-sdk` installed; the real SDK's `Context`/`ObjectContext` are structurally assignable to them.
769
476
 
770
477
  ## Related
771
478
 
772
479
  - `examples/restate/agent.ts` — runnable end-to-end virtual object.
773
480
  - `docs/ARCHITECTURE.md` — agent lifecycle, hook ordering, run-level budget breaker.
774
- - `docs/SKILL.md` — `createAgent`, `Provider`, `ToolDef`, `SessionStore` interfaces this adapter wraps.
481
+ - `docs/SKILL.md` — `createAgent`, `Provider`, `ToolDef`, `SessionStore`, hook table (`execution:ready`, etc.).
482
+ - `docs/RUN_IN_BACKGROUND.md` — background-task execution model this adapter journals.
775
483
  - [Restate docs](https://docs.restate.dev/) — runtime, SDK, deployment model.