zidane 7.0.14 → 7.0.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +18 -9
- package/dist/{acp-CZpX91YE.js → acp-Bw2b_kka.js} +3 -3
- package/dist/{acp-CZpX91YE.js.map → acp-Bw2b_kka.js.map} +1 -1
- package/dist/acp-cli.js +4 -4
- package/dist/acp.d.ts +1 -1
- package/dist/acp.js +1 -1
- package/dist/{agent-B1jsFaDq.js → agent-B_uQo4f1.js} +649 -48
- package/dist/agent-B_uQo4f1.js.map +1 -0
- package/dist/agent.d.ts +1 -1
- package/dist/agent.js +1 -1
- package/dist/{anthropic-Da39cE-d.js → anthropic-QbHqqTkV.js} +2 -2
- package/dist/{anthropic-Da39cE-d.js.map → anthropic-QbHqqTkV.js.map} +1 -1
- package/dist/{approvals-S100xf1d.js → approvals-B9aIQJRQ.js} +287 -45
- package/dist/approvals-B9aIQJRQ.js.map +1 -0
- package/dist/artifacts-CV2hUKnL.js +112 -0
- package/dist/artifacts-CV2hUKnL.js.map +1 -0
- package/dist/{auth-DfEWnkC7.js → auth-plw6npda.js} +19 -7
- package/dist/auth-plw6npda.js.map +1 -0
- package/dist/chat/pure.d.ts +3 -3
- package/dist/chat/pure.js +1 -1
- package/dist/chat.d.ts +6 -6
- package/dist/chat.js +5 -5
- package/dist/contexts/daytona.d.ts +1 -1
- package/dist/contexts/e2b.d.ts +1 -1
- package/dist/daemon-cli.js +1 -1
- package/dist/eval.d.ts +1 -1
- package/dist/eval.js +1 -1
- package/dist/extensions.d.ts +2 -2
- package/dist/extensions.js +2 -2
- package/dist/headless.d.ts +1 -1
- package/dist/headless.js +2 -2
- package/dist/{index-DFGECPfm.d.ts → index-Dc_zYpEu.d.ts} +351 -2
- package/dist/index-Dc_zYpEu.d.ts.map +1 -0
- package/dist/{index-DP7xYXr9.d.ts → index-Df2CH6bn.d.ts} +92 -7
- package/dist/index-Df2CH6bn.d.ts.map +1 -0
- package/dist/index.d.ts +2 -2
- package/dist/index.js +7 -6
- package/dist/{ipc-BLLscoIl.js → ipc-T_pPnHkp.js} +2 -2
- package/dist/{ipc-BLLscoIl.js.map → ipc-T_pPnHkp.js.map} +1 -1
- package/dist/ipc.d.ts +54 -6
- package/dist/ipc.d.ts.map +1 -1
- package/dist/ipc.js +3 -3
- package/dist/mcp.d.ts +1 -1
- package/dist/{oauth-C4GvY_p1.js → oauth-EIBUuSDK.js} +2 -2
- package/dist/{oauth-C4GvY_p1.js.map → oauth-EIBUuSDK.js.map} +1 -1
- package/dist/output/stream-json.d.ts +1 -1
- package/dist/output/terminal.d.ts +1 -1
- package/dist/plugins.d.ts +1 -1
- package/dist/presets.d.ts +2 -2
- package/dist/presets.js +2 -2
- package/dist/providers/anthropic.d.ts +1 -1
- package/dist/providers/anthropic.js +1 -1
- package/dist/providers/arcee.d.ts +1 -1
- package/dist/providers/baseten.d.ts +1 -1
- package/dist/providers/cerebras.d.ts +1 -1
- package/dist/providers/local.d.ts +1 -1
- package/dist/providers/openai-compat.d.ts +1 -1
- package/dist/providers/openai.d.ts +1 -1
- package/dist/providers/openai.js +1 -1
- package/dist/providers/openrouter.d.ts +1 -1
- package/dist/providers/xai.d.ts +1 -1
- package/dist/providers/xai.js +2 -2
- package/dist/{providers-BYgyPb15.js → providers-CoK1rZ1p.js} +2 -2
- package/dist/{providers-BYgyPb15.js.map → providers-CoK1rZ1p.js.map} +1 -1
- package/dist/providers.d.ts +1 -1
- package/dist/providers.js +3 -3
- package/dist/restate.d.ts +1 -1
- package/dist/restate.js +1 -1
- package/dist/{server-C8gULJ0d.js → server-DXM8ZbGi.js} +64 -6
- package/dist/server-DXM8ZbGi.js.map +1 -0
- package/dist/session/sqlite.d.ts +1 -1
- package/dist/session/sqlite.d.ts.map +1 -1
- package/dist/session/sqlite.js +117 -1
- package/dist/session/sqlite.js.map +1 -1
- package/dist/{session-K3icbPH9.js → session-DPZnncaQ.js} +60 -1
- package/dist/session-DPZnncaQ.js.map +1 -0
- package/dist/session.d.ts +2 -2
- package/dist/session.js +3 -2
- package/dist/skills.d.ts +1 -1
- package/dist/{tool-formatters-C-s_-u0v.d.ts → tool-formatters-BgdOYAy9.d.ts} +2 -2
- package/dist/{tool-formatters-C-s_-u0v.d.ts.map → tool-formatters-BgdOYAy9.d.ts.map} +1 -1
- package/dist/tools/fetch-url.d.ts +1 -1
- package/dist/tools/web-search.d.ts +1 -1
- package/dist/tools.d.ts +2 -2
- package/dist/tools.js +2 -2
- package/dist/{transcript-anchors-D1ToF-EW.d.ts → transcript-anchors-CtnwuQJM.d.ts} +3 -3
- package/dist/transcript-anchors-CtnwuQJM.d.ts.map +1 -0
- package/dist/tui.d.ts +3 -3
- package/dist/tui.d.ts.map +1 -1
- package/dist/tui.js +576 -102
- package/dist/tui.js.map +1 -1
- package/dist/{turn-operations-CZLsNgw-.js → turn-operations-DkJ1R0p8.js} +4 -3
- package/dist/turn-operations-DkJ1R0p8.js.map +1 -0
- package/dist/{turn-operations-Bp5-BAki.d.ts → turn-operations-khAvr7KJ.d.ts} +5 -3
- package/dist/turn-operations-khAvr7KJ.d.ts.map +1 -0
- package/dist/types.d.ts +1 -1
- package/dist/{xai-DILdFGzN.js → xai-CazjStOK.js} +3 -3
- package/dist/{xai-DILdFGzN.js.map → xai-CazjStOK.js.map} +1 -1
- package/docs/ACP.md +6 -0
- package/docs/ARCHITECTURE.md +179 -356
- package/docs/ARTIFACTS.md +130 -0
- package/docs/CHAT.md +426 -1312
- package/docs/EXECUTION_CONTEXT.md +46 -19
- package/docs/EXTENSIONS.md +234 -584
- package/docs/HEADLESS.md +5 -0
- package/docs/INTERACTIONS.md +21 -22
- package/docs/IPC.md +318 -335
- package/docs/PLUGINS.md +2 -0
- package/docs/README.md +57 -0
- package/docs/RESTATE.md +199 -491
- package/docs/RUN_IN_BACKGROUND.md +107 -558
- package/docs/SKILL.md +171 -238
- package/docs/TUI.md +213 -591
- package/package.json +1 -1
- package/dist/agent-B1jsFaDq.js.map +0 -1
- package/dist/approvals-S100xf1d.js.map +0 -1
- package/dist/auth-DfEWnkC7.js.map +0 -1
- package/dist/index-DFGECPfm.d.ts.map +0 -1
- package/dist/index-DP7xYXr9.d.ts.map +0 -1
- package/dist/server-C8gULJ0d.js.map +0 -1
- package/dist/session-K3icbPH9.js.map +0 -1
- package/dist/transcript-anchors-D1ToF-EW.d.ts.map +0 -1
- package/dist/turn-operations-Bp5-BAki.d.ts.map +0 -1
- package/dist/turn-operations-CZLsNgw-.js.map +0 -1
package/docs/RESTATE.md
CHANGED
|
@@ -1,44 +1,37 @@
|
|
|
1
1
|
# Zidane × Restate — durable agents
|
|
2
2
|
|
|
3
|
-
`zidane/restate` is an **optional** adapter that runs the agent loop inside a [Restate](https://docs.restate.dev/) handler so every LLM call and completed tool execution becomes a journaled side effect
|
|
3
|
+
`zidane/restate` is an **optional** adapter that runs the agent loop inside a [Restate](https://docs.restate.dev/) handler so every LLM call and completed tool execution becomes a journaled side effect: crash mid-run and resume from the journal without re-billing tokens or re-running persisted tools. It's a thin layer over the existing agent surface — you opt in by wiring wrappers, nothing forks.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Entry points: wire the wrappers directly into your own Restate handler (see `examples/restate/agent.ts`), or use `bun start --restate` as a local demo/validation runner around the same adapter.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
zidane (core)
|
|
9
|
-
└── zidane/restate ← this adapter (Bun + Node, structurally typed against @restatedev/restate-sdk)
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
A production-style runnable example lives at `examples/restate/agent.ts`. The `bun start --restate` CLI path is a local/demo runner around the same adapter; it is useful for exercising the flow against a local Restate runtime, but production services should wire `zidane/restate` inside their own Restate handlers.
|
|
7
|
+
Contents: [Wrappers](#wrappers) · [Local CLI runner](#local-cli-runner) · [How replay works](#how-replay-works) · [Performance](#performance-minimizing-suspensions) · [Wrapper details](#wrapper-details) · [Subagents](#subagents) · [Parallel subagents](#parallel-subagents) · [Durable tool state](#durable-tool-state) · [Recommended AgentBehavior](#recommended-agentbehavior-for-restate-deployments) · [MCP tools](#mcp-tools) · [Background jobs](#long-running-background-jobs-durable-contexts) · [Edge deployments](#edge--long-lived-deployments) · [Failure modes](#failure-modes) · [Tests](#tests)
|
|
13
8
|
|
|
14
|
-
## When to use it
|
|
9
|
+
## When to use it
|
|
15
10
|
|
|
16
|
-
**Use
|
|
11
|
+
**Use it** for a server-side, long-running agent that must survive process restarts mid-run: unattended jobs, webhook-driven workers, multi-hour pipelines against expensive providers.
|
|
17
12
|
|
|
18
|
-
**Skip Restate
|
|
19
|
-
- A normal interactive TUI / CLI session (the user owns process lifetime; sessions are the recovery story).
|
|
20
|
-
- A request/response API where the agent finishes inside the request lifetime.
|
|
21
|
-
- A worker where crash → start-over is acceptable (sessions still persist conversation; you just lose the in-flight turn).
|
|
22
|
-
|
|
23
|
-
Restate adds a deployment surface (runtime + SDK + journal storage). Don't pay for it unless mid-run crashes are an actual concern.
|
|
13
|
+
**Skip it** for an interactive TUI/CLI session (the user owns process lifetime), a request/response API where the agent finishes inside the request, or a worker where crash-and-restart is acceptable. Restate adds a deployment surface (runtime, SDK, journal storage); don't pay for it unless mid-run crashes are an actual concern.
|
|
24
14
|
|
|
25
15
|
## Wrappers
|
|
26
16
|
|
|
27
|
-
|
|
17
|
+
Pick what you need; everything else on `Provider` / `ToolDef` passes through unchanged.
|
|
28
18
|
|
|
29
|
-
| Helper | Wraps
|
|
19
|
+
| Helper | Wraps | Does |
|
|
30
20
|
|---|---|---|
|
|
31
|
-
| `restateBehavior()` | `AgentBehavior` | Strict
|
|
32
|
-
| `restateClock(ctx)` | `AgentClock` | `now()
|
|
33
|
-
| `restateProvider(provider, ctx)` | `Provider.stream` |
|
|
34
|
-
| `restateTool(tool, ctx)` / `wrapAgentTools(tools, ctx)` | `ToolDef.execute` |
|
|
35
|
-
| `restateMcpToolWrap(ctx)` | `mcpToolWrap` hook
|
|
36
|
-
| `restateToolBatchExecutor({ promises })` |
|
|
37
|
-
| `restateSessionStore(ctx)` | `SessionStore` | Session header
|
|
38
|
-
| `restateToolStateStore(ctx)` | `AgentOptions.toolState` |
|
|
39
|
-
| `restateAwakeableTool(
|
|
40
|
-
| `
|
|
41
|
-
| `
|
|
21
|
+
| `restateBehavior(overrides?)` | `AgentBehavior` | Strict defaults: sequential tools, background tasks off for non-durable contexts, cache on, Restate control errors re-thrown |
|
|
22
|
+
| `restateClock(ctx, opts?)` | `AgentClock` | `now()` / `randomUUID()` stable across replays |
|
|
23
|
+
| `restateProvider(provider, ctx, opts?)` | `Provider.stream` | Journals each turn's `TurnResult`; replay skips the model call |
|
|
24
|
+
| `restateTool(tool, ctx, opts?)` / `wrapAgentTools(tools, ctx, opts?)` | `ToolDef.execute` | Journals each tool result; replay skips re-execution |
|
|
25
|
+
| `restateMcpToolWrap(ctx, opts?)` | `mcpToolWrap` hook | Same journaling for discovered MCP tools (wired automatically via `createAgent({ mcpToolWrap })`) |
|
|
26
|
+
| `restateToolBatchExecutor({ promises })` | `AgentBehavior.toolBatchExecutor` | Opt-in durable parallel join for concurrency-safe tool fleets |
|
|
27
|
+
| `restateSessionStore(ctx, opts?)` | `SessionStore` | Session header/turns/runs as virtual-object K/V state |
|
|
28
|
+
| `restateToolStateStore(ctx, opts?)` | `AgentOptions.toolState` | Durable `ToolContext.state` — see [Durable tool state](#durable-tool-state) |
|
|
29
|
+
| `restateAwakeableTool(opts)` | `ToolDef` | Tool that parks on an awakeable until an external resumer resolves it (human-in-the-loop) |
|
|
30
|
+
| `restateSpawnTool(opts)` | `ToolDef` | Dispatches subagent children to their own invocations — see [Parallel subagents](#parallel-subagents) |
|
|
31
|
+
| `restateWaitBackground(opts)` | `ExecutionContext.waitBackground` | Race-safe durable wait for a background task's exit |
|
|
32
|
+
| `awakeableWithTimeout(ctx, opts)` | awakeable + `ctx.sleep` | Low-level building block `restateWaitBackground` is built on |
|
|
33
|
+
| `createReplayTracker(opts?)` / `replayAwareAbort(tracker, abort)` | — | Replay-liveness probe + abort deferral — see [replay-safe abort](#replay-safe-abort) |
|
|
34
|
+
| `shouldRethrowRestateControlError(err)` / `deterministicUuid(seed)` | — | Structural error classifier / pure UUID-from-seed helper used internally, exported for custom wrapping |
|
|
42
35
|
|
|
43
36
|
### Minimal wiring
|
|
44
37
|
|
|
@@ -54,21 +47,15 @@ const zidaneAgent = restate.object({
|
|
|
54
47
|
name: 'zidane-agent',
|
|
55
48
|
handlers: {
|
|
56
49
|
async run(ctx, { prompt }: { prompt: string }) {
|
|
57
|
-
const session = await createSession({
|
|
58
|
-
id: ctx.key,
|
|
59
|
-
store: restateSessionStore(ctx),
|
|
60
|
-
})
|
|
61
|
-
|
|
50
|
+
const session = await createSession({ id: ctx.key, store: restateSessionStore(ctx) })
|
|
62
51
|
const agent = createAgent({
|
|
63
52
|
provider: restateProvider(anthropic({ defaultModel: 'claude-sonnet-4-5' }), ctx),
|
|
64
53
|
tools: wrapAgentTools({ shell }, ctx),
|
|
65
54
|
clock: restateClock(ctx),
|
|
66
55
|
session,
|
|
67
|
-
//
|
|
68
|
-
toolState: restateToolStateStore(ctx),
|
|
56
|
+
toolState: restateToolStateStore(ctx), // optional — only if tools use `ctx.state`
|
|
69
57
|
behavior: restateBehavior(),
|
|
70
58
|
})
|
|
71
|
-
|
|
72
59
|
return await agent.run({ prompt })
|
|
73
60
|
},
|
|
74
61
|
},
|
|
@@ -77,230 +64,133 @@ const zidaneAgent = restate.object({
|
|
|
77
64
|
restate.serve({ services: [zidaneAgent] })
|
|
78
65
|
```
|
|
79
66
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
On Lambda / sandbox-reattach hosts, share one coalescing clock across the agent and the store, raise the safe-tool concurrency, and exclude deterministic MCP reads from journaling. See [Performance: minimizing suspensions](#performance-minimizing-suspensions) for the full rationale.
|
|
83
|
-
|
|
84
|
-
```ts
|
|
85
|
-
const clock = restateClock(ctx, { coalesce: true })
|
|
86
|
-
|
|
87
|
-
const session = await createSession({
|
|
88
|
-
id: ctx.key,
|
|
89
|
-
store: restateSessionStore(ctx, { clock }), // store timestamps ride the same anchor
|
|
90
|
-
})
|
|
91
|
-
|
|
92
|
-
const agent = createAgent({
|
|
93
|
-
provider: restateProvider(anthropic({ defaultModel: 'claude-sonnet-4-5' }), ctx),
|
|
94
|
-
tools: wrapAgentTools({ shell }, ctx),
|
|
95
|
-
clock,
|
|
96
|
-
session,
|
|
97
|
-
behavior: restateBehavior({ maxConcurrentTools: 8 }), // batch concurrency-safe tools
|
|
98
|
-
mcpToolWrap: restateMcpToolWrap(ctx, { exclude: ['mcp_docs_search'] }),
|
|
99
|
-
})
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
### Local CLI runner
|
|
67
|
+
## Local CLI runner
|
|
103
68
|
|
|
104
|
-
`bun start --restate`
|
|
69
|
+
`bun start --restate` boots a local Restate endpoint in-process, waits for the Admin API, probes the local service port, registers the deployment, and invokes the `zidane-start-agent` virtual object through ingress. It's a convenience runner for local validation and demos, not a production deployment shape.
|
|
105
70
|
|
|
106
71
|
```bash
|
|
107
72
|
bun run restate:fixture:up
|
|
108
|
-
bun start
|
|
109
|
-
--restate \
|
|
110
|
-
--prompt "exercise the durable runner" \
|
|
111
|
-
--session-id demo-1
|
|
73
|
+
bun start --restate --prompt "exercise the durable runner" --session-id demo-1
|
|
112
74
|
```
|
|
113
75
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
- `--restate-
|
|
76
|
+
| Flag | Default | Notes |
|
|
77
|
+
|---|---|---|
|
|
78
|
+
| `--restate-admin-url` | `http://localhost:19070` | Matches the Docker fixture |
|
|
79
|
+
| `--restate-ingress-url` | `http://localhost:18080` | Matches the Docker fixture |
|
|
80
|
+
| `--restate-service-port` | `19081` | Local Bun endpoint port; the Docker fixture's own service uses `19080` |
|
|
81
|
+
| `--restate-deployment-uri` | `http://host.docker.internal:<service-port>` | URI Restate uses to reach the local endpoint |
|
|
82
|
+
| `--restate-keep-alive` | off | Keep the endpoint alive after the first invocation for retries/manual invokes |
|
|
83
|
+
| `--restate-invocation-timeout-ms` | none | Client-side ingress `fetch` timeout only — not a handler execution timeout |
|
|
84
|
+
| `--restate-provider-retry-attempts` / `--restate-tool-retry-attempts` | `3` / `1` | Override the CLI's `ctx.run` retry attempts for the provider/tool wrappers |
|
|
85
|
+
| `--restate-random-failure` | off | Chaos switch: injects retryable provider failures to exercise replay/retry |
|
|
123
86
|
|
|
124
|
-
`--session-db` is rejected in Restate mode
|
|
87
|
+
`--session-db` is rejected in Restate mode — virtual-object state owns the session projection.
|
|
125
88
|
|
|
126
|
-
The
|
|
89
|
+
The service exposes three handlers:
|
|
127
90
|
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
91
|
+
| Handler | Returns |
|
|
92
|
+
|---|---|
|
|
93
|
+
| `run(input)` | Drives an agent run for the object key. Same input + `completed` status replays the persisted result; different input on a non-idle key throws a 409-style `TerminalError` |
|
|
94
|
+
| `status()` | `{ status: 'idle' \| 'running' \| 'completed' \| 'failed', ... }` plus a completion summary |
|
|
95
|
+
| `history()` | The `restateSessionStore` projection for the object key |
|
|
131
96
|
|
|
132
|
-
|
|
97
|
+
Use a new `--session-id` for a distinct local run.
|
|
133
98
|
|
|
134
99
|
## How replay works
|
|
135
100
|
|
|
136
|
-
Restate journals each `ctx.run(name, fn)` call. The
|
|
101
|
+
Restate journals each `ctx.run(name, fn)` call. The first execution runs `fn` and stores its return value. On a later invocation of the same handler attempt (typically crash recovery), `ctx.run(name, fn)` looks the entry up by `name` and returns the cached value **without invoking `fn`**.
|
|
137
102
|
|
|
138
103
|
```
|
|
139
|
-
|
|
140
|
-
│ │
|
|
141
|
-
turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => anthropic.stream(...))
|
|
142
|
-
│
|
|
104
|
+
turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => provider.stream(...))
|
|
143
105
|
├─ journal entry: TurnResult
|
|
144
|
-
│
|
|
145
106
|
┌───┴───┐
|
|
146
107
|
│ crash │
|
|
147
108
|
└───┬───┘
|
|
148
|
-
|
|
149
|
-
turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => anthropic.stream(...))
|
|
150
|
-
│
|
|
109
|
+
turn N: loop ──▶ restateProvider.stream() ──▶ ctx.run("llm-call-N", () => provider.stream(...))
|
|
151
110
|
└─ returns journaled TurnResult (no model call)
|
|
152
111
|
```
|
|
153
112
|
|
|
154
|
-
Journal-entry names are
|
|
155
|
-
|
|
156
|
-
- **LLM calls**: `llm-call-1`, `llm-call-2`, … (monotonic per wrapper instance).
|
|
157
|
-
- **Tools**: `tool-<name>-1`, `tool-<name>-2`, … (per wrapper, per tool name).
|
|
158
|
-
|
|
159
|
-
Both names are configurable via the `entryName` option if multiple agents share one service and you need disambiguation.
|
|
113
|
+
Journal-entry names are per-invocation: `llm-call-1`, `llm-call-2`, … for LLM calls (monotonic per provider wrapper instance); `tool-<name>-1`, `tool-<name>-2`, … for tools (per wrapper, per tool name). Both are configurable via `entryName` when multiple agents share one service.
|
|
160
114
|
|
|
161
|
-
|
|
115
|
+
**What re-fires on replay.** Restate only replays `ctx.run`; everything else re-executes, including hooks:
|
|
162
116
|
|
|
163
|
-
|
|
117
|
+
- `tool:gate` / `tool:transform` / `tool:after` re-fire; built-ins (truncation, image-stripping) are pure over journaled inputs so the result is replay-stable.
|
|
118
|
+
- `turn:before` / `turn:after` / `usage` re-fire, reading the journaled `TurnResult`.
|
|
119
|
+
- `stream:text` / `stream:thinking` do **not** re-fire — they're part of the live stream. Reconstruct transcripts from `session.turns` (`eventsFromTurns` in `zidane/chat`) instead.
|
|
120
|
+
- Consumer hooks touching external services (logging, metrics) should wrap those calls in `ctx.run` themselves, or they fire once per attempt.
|
|
164
121
|
|
|
165
|
-
-
|
|
166
|
-
- `turn:before`, `turn:after`, `usage` re-fire. They read the journaled `TurnResult`, so emitted values match the original.
|
|
167
|
-
- `stream:text`, `stream:thinking` do **not** re-fire — those callbacks are part of the live stream, which doesn't replay. UIs should reconstruct transcripts from `session.turns` (`eventsFromTurns` in `zidane/chat`) rather than relying on hook re-emission. The chat layer already does this.
|
|
168
|
-
- Consumer hooks that hit external services (logging APIs, metrics endpoints) should wrap those side effects in `ctx.run` themselves — otherwise they'll fire twice across the original + replay.
|
|
169
|
-
|
|
170
|
-
### Per-key serialization
|
|
171
|
-
|
|
172
|
-
Restate gives one concurrent invocation per virtual-object key. With sessions bound 1:1 to `ctx.key`, concurrent users hit different keys and never race; the runtime serializes invocations on the same key. No application-level locks, no `BEGIN IMMEDIATE` dance.
|
|
122
|
+
**Per-key serialization.** Restate allows one concurrent invocation per virtual-object key. Bind sessions 1:1 to `ctx.key` and concurrent users never race — no application-level locks needed.
|
|
173
123
|
|
|
174
124
|
## Performance: minimizing suspensions
|
|
175
125
|
|
|
176
|
-
On platforms without bidirectional streaming (
|
|
126
|
+
On platforms without bidirectional streaming (Lambda, sandbox-reattach runners), every awaited durable completion can suspend the invocation, and each resume replays the journal — execution time scales with the number of awaited completions and the journal/state size. `ctx.run` for LLM/tool calls is an irreducible floor; these levers remove overhead around it, highest-impact first:
|
|
177
127
|
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
128
|
+
| Lever | Effect |
|
|
129
|
+
|---|---|
|
|
130
|
+
| Raise `maxConcurrentTools` (`restateBehavior({ maxConcurrentTools: 8 })`) + `restateToolBatchExecutor` | N sequential tool suspend/replay cycles collapse into one durable `all()` batch for concurrency-safe calls |
|
|
131
|
+
| `restateClock(ctx, { coalesce: true })` | Collapses `ctx.date.now()` round-trips into a journaled anchor + monotonic counter (precision tradeoff in [`restateClock`](#restateclock)) |
|
|
132
|
+
| `restateSessionStore(ctx, { clock })` | Shares that same anchor for the store's own timestamp writes |
|
|
133
|
+
| `restateMcpToolWrap(ctx, { exclude: [...] })` | Skips journaling for deterministic, side-effect-free MCP read tools |
|
|
134
|
+
| Stay on eager state | Lazy state (`enableLazyState`) turns each `ctx.get` into a suspend+replay; the session store already reads each slot once |
|
|
135
|
+
| Context discipline (on by default in `restateBehavior`) | `compactStrategy: 'tail'`, `elideStaleReads: 'stub'`, `toolOutputBudget`, `autoCompact` shrink the per-turn payload |
|
|
186
136
|
|
|
187
137
|
## Wrapper details
|
|
188
138
|
|
|
189
139
|
### `restateClock`
|
|
190
140
|
|
|
191
|
-
`AgentClock
|
|
141
|
+
Backs `AgentClock.now()` with `ctx.date.now()` (journaled, async) so `SessionTurn.createdAt`, run start/end, and fallback turn ids stay identical across replays.
|
|
192
142
|
|
|
193
|
-
`ctx.
|
|
143
|
+
`randomUUID()` is **not** `ctx.rand.uuidv4()` per call: `ctx.rand` is a single invocation-seeded stream, and the agent can't guarantee every consumer draws the same count on every replay (this swapped turn ids across a suspension in practice). Instead each id is `deterministicUuid(idSeed:uuid:N)` — a sha256-derived UUID from a clock-private counter, pure and sync. `idSeed` defaults to one `ctx.rand.uuidv4()` draw at construction; pass `{ idSeed: ctx.key }` for a draw-free construction.
|
|
194
144
|
|
|
195
|
-
**
|
|
145
|
+
**Coalescing.** `restateClock(ctx, { coalesce: true })` journals a real anchor once per `reanchorEvery` calls (default `8`) and derives intervening timestamps as `anchor + n` — replay-deterministic and monotonic, but clustered near the anchor rather than wall-clock-accurate. Live-only measurements (TTFT, `maxWallMs`) read `Date.now()` directly and are unaffected. Default (off) journals every `now()` call. Under coalescing, a subagent's reported elapsed time becomes a draw count — read `ChildRunStats.stats.elapsed` (always live) for the real duration.
|
|
196
146
|
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
The **tradeoff**: timestamps within a re-anchor window are logical — ordered and stable, but clustered near the window's anchor rather than wall-clock-accurate. Run `elapsed` (`endedAt - startedAt`) and per-turn `createdAt` lose precision down to the window granularity; live-only measurements (TTFT, `maxWallMs` budget) are unaffected because the loop reads those from `Date.now()` directly.
|
|
200
|
-
|
|
201
|
-
Two subagent-specific consequences, both from differencing two coalesced draws: the elapsed figure in the spawn result string the **model reads** (`[sub-agent child-1] Completed in 2 turns (3ms)`) becomes a count of intervening clock draws rather than milliseconds, so a child that ran 40 seconds can report single-digit ms; and `createSpawnTool({ timeoutMs })`'s turn-boundary deadline measures the same logical scale, so it effectively counts draws too and won't fire on wall-clock time. Leave coalescing off if the model needs true durations or the child budget must track real time — hosts can always read the real duration off `ChildRunStats.stats.elapsed` (`spawn:complete` / `AgentStats.children`), which stays live `Date.now()`. Lower `reanchorEvery` ⇒ more accurate timestamps + more suspensions; higher ⇒ fewer suspensions + more clustering. The default (coalescing off) journals every `now()` and keeps wall-accurate timestamps, so opt in per deployment. Provider/tool `ctx.run` calls remain irreducible suspension points — coalescing only addresses the clock.
|
|
202
|
-
|
|
203
|
-
**Gotcha**: when you pair a `restateSessionStore` with a session, the store's own `generateTurnId` runs ahead of `clock.randomUUID()` (the loop short-circuits via `session?.generateTurnId() ?? clock.randomUUID()`). The store derives deterministic IDs from the persisted turn count (`turn_1`, `turn_2`, …) so turn IDs stay stable across replay and across later invocations on the same object key. If you wire a custom store, keep turn ID generation deterministic from persisted state or let the Restate-backed clock provide the fallback UUIDs.
|
|
147
|
+
**Gotcha**: with `restateSessionStore`, the store's deterministic `generateTurnId` (`turn_1`, `turn_2`, …) takes priority over `clock.randomUUID()` — the loop only falls back to the clock with no store present. A custom store's turn-id generation must stay deterministic from persisted state.
|
|
204
148
|
|
|
205
149
|
### `restateProvider`
|
|
206
150
|
|
|
207
|
-
Wraps `Provider.stream` in `ctx.run
|
|
208
|
-
|
|
209
|
-
Default `runOptions`: `{ maxRetryAttempts: 3 }`. Provider transients (rate limits, 5xx, connection resets) get retried inside the journal entry. Once the cap is reached the failure becomes terminal and surfaces through Zidane's normal `AgentProviderError` path.
|
|
151
|
+
Wraps `Provider.stream` in `ctx.run`; one journal entry per assistant turn. All other `Provider` methods pass through unwrapped (pure over already-deterministic inputs).
|
|
210
152
|
|
|
211
|
-
|
|
153
|
+
Default `runOptions` depends on `streamCallbacks`:
|
|
154
|
+
- `'live'` (default) → `{ maxRetryAttempts: 1 }`. A live stream that fails after emitting deltas would, on retry, re-fire `onText` into the loop's accumulator and duplicate partial output — the loop's own retry (clean-slate re-issue) is the correct layer here.
|
|
155
|
+
- `'suppress'` → `{ maxRetryAttempts: 3 }`. No live callbacks fire, so an in-journal retry is invisible; use this for strict replay-safe UIs that reconstruct from `session.turns`.
|
|
212
156
|
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
Zidane can persist image/document blocks as opaque refs (`{ type: 'image' | 'document', ref, mediaType }`) and resolve them only when assembling provider wire messages. `restateSessionStore(ctx)` needs no special handling: `session.turns` still contains refs, so `ctx.set` journals the same ref bytes it stores.
|
|
216
|
-
|
|
217
|
-
Wrap the host download in `ctx.run` from a `content-ref:resolve` hook, so replay returns the journaled base64 instead of re-downloading:
|
|
218
|
-
|
|
219
|
-
```ts
|
|
220
|
-
import { hashContent } from 'zidane/tools'
|
|
221
|
-
|
|
222
|
-
const agent = createAgent({
|
|
223
|
-
// ...
|
|
224
|
-
hooks: {
|
|
225
|
-
'content-ref:resolve': async resolveCtx => {
|
|
226
|
-
const { data } = await ctx.run(
|
|
227
|
-
`resolve-content-${hashContent(resolveCtx.block.ref)}`,
|
|
228
|
-
async () => ({ data: await downloadMediaAsBase64(resolveCtx.block.ref) }),
|
|
229
|
-
)
|
|
230
|
-
resolveCtx.data = data
|
|
231
|
-
},
|
|
232
|
-
},
|
|
233
|
-
})
|
|
234
|
-
```
|
|
157
|
+
Beyond the retry cap the failure surfaces through the normal `AgentProviderError` path.
|
|
235
158
|
|
|
236
|
-
|
|
159
|
+
**Content refs.** Zidane can persist image/document blocks as opaque `{ type, ref, mediaType }` refs, resolved only when assembling wire messages; `restateSessionStore` journals the ref bytes as-is, no special handling needed. Wrap the host download itself in `ctx.run` inside the `content-ref:resolve` hook (`resolveCtx.data = await ctx.run(name, () => downloadMediaAsBase64(resolveCtx.block.ref))`) so replay returns the journaled base64 instead of re-downloading.
|
|
237
160
|
|
|
238
161
|
### `restateTool` / `wrapAgentTools`
|
|
239
162
|
|
|
240
|
-
Wraps `ToolDef.execute` in `ctx.run
|
|
241
|
-
|
|
242
|
-
Default `runOptions`: `{ maxRetryAttempts: 1 }`. Tool errors are typically deterministic (validation failures, missing files); blind retries waste budget. Override per-tool for network-heavy custom tools that benefit from SDK-level retries.
|
|
243
|
-
|
|
244
|
-
**Soft tool errors (`softErrors`, default `true`).** A recoverable tool throw (a failed shell command, a `RemoteExecHttpError`, any `execute` rejection) is captured inside the journal as a sentinel value and re-thrown as a plain (non-control) error *outside* it — so the loop produces a model-visible `tool_result { isError }` and the run continues, identical to the non-Restate soft-error model. Without it the throw would exhaust `ctx.run`'s retries, surface as a Restate `TerminalError`, and `restateBehavior`'s `shouldRethrowToolError` would abort the whole run. Real Restate control / abort errors are never softened. Replay reproduces the journaled error without re-running the tool. This is the same mechanism `restateMcpToolWrap` applies to MCP tools — native and MCP tools now behave symmetrically. Set `softErrors: false` to restore the legacy fatal propagation.
|
|
245
|
-
|
|
246
|
-
**Result-block coercion (`coerceResultBlocks`, default `true`).** Malformed structured result blocks (a `{ type: 'image' }` with no payload, a foreign block shape) collapse to a text placeholder before the result is journaled, so a downstream provider wire serializer never has to project an unprojectable block (which, under Restate, would throw inside the journaled provider call and become fatal). Well-formed results pass through untouched.
|
|
247
|
-
|
|
248
|
-
Both options apply to the single-phase journaling path (the default wrapping and the MCP path); the two-phase `durableStart` mode owns its own start/wait result shape and is unaffected.
|
|
249
|
-
|
|
250
|
-
`isConcurrencySafe` is preserved. `restateBehavior()` defaults `maxConcurrentTools` to `1` so strict Restate mode is deterministic first. If you want parallel tools, opt into the Restate scheduler below instead of relying on native promise fan-out.
|
|
163
|
+
Wraps `ToolDef.execute` in `ctx.run`, default `{ maxRetryAttempts: 1 }` (tool errors are typically deterministic, so a blind retry wastes budget; override per-tool for network-heavy custom tools). If the process dies before the journal entry lands, the in-flight tool can run again — non-idempotent tools need an application-level idempotency key.
|
|
251
164
|
|
|
252
|
-
`
|
|
165
|
+
- **`softErrors` (default `true`).** A recoverable throw is captured inside the journal as a sentinel and re-thrown as a plain (non-control) error outside it, so the loop still produces `tool_result { isError }` and continues, matching non-Restate behavior. Without it, the throw exhausts `ctx.run`'s retries into a Restate `TerminalError` and `shouldRethrowToolError` aborts the whole run. Control/abort errors are never softened. `restateMcpToolWrap` applies the same mechanism.
|
|
166
|
+
- **`coerceResultBlocks` (default `true`).** Malformed structured result blocks (e.g. an `{ type: 'image' }` with no payload) collapse to a text placeholder before journaling, so the wire serializer never chokes on one inside a journaled call.
|
|
167
|
+
- `isConcurrencySafe` is preserved from the wrapped `ToolDef` (see SKILL.md § Per-tool concurrency); `restateBehavior()` defaults `maxConcurrentTools: 1` regardless.
|
|
168
|
+
- `wrapAgentTools(tools, ctx, { exclude })` batch-wraps, passing exclude-listed tools through untouched, and disables the built-in `shell` tool's `run_in_background` mode by default (host-local background processes aren't Restate-recoverable) unless `{ detachedTasks: 'durable' }` or `{ disableShellBackground: false }` says otherwise.
|
|
253
169
|
|
|
254
|
-
-
|
|
255
|
-
- `{ disableShellBackground: false }` — explicit boolean override; always wins over the capability.
|
|
170
|
+
**`durableStart` — crash-safe long tools.** Default wrapping only journals the *completed* result, so a crash mid-execute re-runs the tool — unacceptable for builds, migrations, deploys. It splits execution into a journaled `start` (kicks off the work, returns a re-attach token) and either:
|
|
256
171
|
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
The default wrapping journals only the *completed* result, so a crash mid-execute re-runs the tool on replay — unacceptable for builds, migrations, deploys. `restateTool`'s `durableStart` option splits the execution into two journal entries:
|
|
260
|
-
|
|
261
|
-
```ts
|
|
262
|
-
restateTool(deployTool, ctx, {
|
|
263
|
-
durableStart: {
|
|
264
|
-
// Journaled BEFORE the body — kick off the work, return a re-attach token.
|
|
265
|
-
start: async (input, toolCtx) => startDeploy(input), // → { jobId }
|
|
266
|
-
// On replay, receives the SAME journaled token — re-attach, don't re-run.
|
|
267
|
-
attach: async (token, input, toolCtx) => awaitDeploy(token.jobId),
|
|
268
|
-
},
|
|
269
|
-
})
|
|
270
|
-
```
|
|
271
|
-
|
|
272
|
-
A crash after `start` journals replays into the wait phase with the recorded token. The wrapped tool deliberately does not advertise `durableExecute` (the two-phase composition is native-async); the batch scheduler runs it sequentially, which is the right shape for long re-attachable work.
|
|
273
|
-
|
|
274
|
-
**`attach` holds the wait inside one `ctx.run` entry** — fine for waits bounded well below the server's inactivity timeout, but a long busy-poll in there starves the journal: the inactivity timeout fires, the abort timeout kills the attempt, and each retry re-runs `attach` with a fresh deadline. The invocation can never complete. Keep `attach` only as the degraded mode for hosts without a push seam.
|
|
275
|
-
|
|
276
|
-
**`wait` is the awakeable-based alternative for long-running work.** Hosts that can push completion (e.g. a runner firing `task:exit` → resumer → `ctx.resolveAwakeable`) configure it instead of `attach` (it wins when both are set):
|
|
172
|
+
- **`attach`** — replay re-attaches with the same token inside one `ctx.run` entry. Fine for waits well below the server's inactivity timeout; a long busy-poll here starves the journal (inactivity timeout → abort → re-run on every retry, never completing). Use only as a degraded mode.
|
|
173
|
+
- **`wait`** (wins if both set) — awakeable-based: `register` (journaled) hands the host an awakeable id as exit resumer, the park happens **outside** `ctx.run` (raced against `timeoutMs` via `awakeableWithTimeout` if set), and `result` (journaled) shapes the final value from the resume payload.
|
|
277
174
|
|
|
278
175
|
```ts
|
|
279
176
|
restateTool(shellTool, ctx, {
|
|
280
177
|
durableStart: {
|
|
281
|
-
start: async input => startBackgroundTask(input),
|
|
178
|
+
start: async input => startBackgroundTask(input), // → { taskId }
|
|
282
179
|
wait: {
|
|
283
|
-
// Journaled — hand the awakeable id to the host as the exit resumer.
|
|
284
180
|
register: (token, awakeableId) => registerExitResumer(token.taskId, awakeableId),
|
|
285
|
-
// The model's own timeout becomes the durable park deadline.
|
|
286
181
|
timeoutMs: input => input.timeout as number,
|
|
287
|
-
promises: RestatePromise,
|
|
288
|
-
|
|
289
|
-
// kill the job and shape the timeout result here.
|
|
290
|
-
result: async (token, resumed) =>
|
|
291
|
-
resumed ? formatExit(token.taskId, resumed) : killAndFormatTimeout(token.taskId),
|
|
182
|
+
promises: RestatePromise,
|
|
183
|
+
result: async (token, resumed) => resumed ? formatExit(token.taskId, resumed) : killAndFormatTimeout(token.taskId),
|
|
292
184
|
},
|
|
293
185
|
},
|
|
294
186
|
})
|
|
295
187
|
```
|
|
296
188
|
|
|
297
|
-
The
|
|
298
|
-
|
|
299
|
-
The optional `onAbort(token, input, toolCtx)` is the abort seam: when the per-call signal fires before the wait settles (run abort, `cancelTool`, `destroy()` mid-run), it runs once, best-effort, so the started job doesn't keep running with nobody attached. In `wait` mode the abort also unblocks the park (the call rejects instead of waiting on a resumer nobody will consume). It is NOT called on process death — that's not termination; the journaled token re-attaches on replay. It runs outside the journal (an aborting invocation can't reliably open new `ctx.run` entries); journal host-side cleanup yourself if it must be durable.
|
|
189
|
+
The wrapped tool doesn't advertise `durableExecute`, so the batch scheduler always runs it sequentially — correct for long re-attachable work. Optional `onAbort(token, input, toolCtx)` fires once, best-effort, when the per-call signal fires before the wait settles (run abort, `cancelTool`, `destroy()`); it is NOT called on process death (the journaled token re-attaches on replay instead) and runs outside the journal.
|
|
300
190
|
|
|
301
191
|
### `restateAwakeableTool`
|
|
302
192
|
|
|
303
|
-
Builds a `ToolDef` that parks on a Restate awakeable until an external resumer calls `ctx.resolveAwakeable(id, value)` — the
|
|
193
|
+
Builds a `ToolDef` that parks on a Restate awakeable until an external resumer calls `ctx.resolveAwakeable(id, value)` — the shape for `ask_user` / `request_review` tools:
|
|
304
194
|
|
|
305
195
|
```ts
|
|
306
196
|
const requestReview = restateAwakeableTool<{ approved: boolean }>({
|
|
@@ -313,24 +203,15 @@ const requestReview = restateAwakeableTool<{ approved: boolean }>({
|
|
|
313
203
|
})
|
|
314
204
|
```
|
|
315
205
|
|
|
316
|
-
|
|
206
|
+
`notify` runs inside `ctx.run` so it fires exactly once across replays; a journaled marker records the awakeable id and the tool refuses to park if a replay mints a different one (journal drift). Do not wrap the returned tool in `restateTool` — a suspension point must not nest inside another journaled closure.
|
|
317
207
|
|
|
318
208
|
### `restateSessionStore`
|
|
319
209
|
|
|
320
|
-
A `SessionStore` backed by the bound virtual object's K/V state
|
|
321
|
-
|
|
322
|
-
**State I/O is coalesced per invocation.** Each slot is read from Restate at most once and then served from an in-memory read-through cache; writes update the cache and `ctx.set` eagerly. Without this, `appendTurns` (which fires on every turn boundary) re-read and re-sanitized the entire growing `session-turns` array on each call — O(N²) sequential K/V round-trips over an N-turn run. Writes stay eager rather than deferred to a single flush, because external read visibility is the whole reason to pick this store: a concurrent reader must see each turn as it lands. The cache is safe because Restate processes one invocation at a time per object key, so no competing writer can invalidate it mid-invocation.
|
|
323
|
-
|
|
324
|
-
**Optional shared clock.** `restateSessionStore(ctx, { clock })` sources the header `createdAt` / `updatedAt` timestamps from the injected `AgentClock` instead of a direct `ctx.date.now()`. Pass the same `restateClock(ctx, { coalesce: true })` you give the agent so the store's finalize-time `save` / `updateStatus` timestamps ride the coalesced anchor (≈2 fewer `ctx.date.now()` round-trips per run on suspend-per-round-trip hosts). Whatever you pass must be replay-stable — the value lands in persisted state via `ctx.set`. Default (no clock) is unchanged: direct `ctx.date.now()`.
|
|
325
|
-
|
|
326
|
-
**When to use it.** Pick this store when **external** consumers (a TUI subscribed to the runtime, a web GUI, support tooling) need to read a session without re-entering the agent handler. Virtual-object state is exposed via the SDK's read APIs and the Restate UI — same data, different access path.
|
|
210
|
+
A `SessionStore` backed by the bound virtual object's K/V state across three slots: `session-data` (header), `session-turns` (history), `session-runs` (run records). Each slot is read at most once per invocation and cached in memory after that (writes update the cache and call `ctx.set` eagerly — safe since Restate serializes one invocation per key), avoiding an O(N²) re-read/re-sanitize of the whole `session-turns` array on every turn. `{ clock }` sources header/finalize timestamps from the injected `AgentClock` instead of a direct `ctx.date.now()`.
|
|
327
211
|
|
|
328
|
-
**
|
|
212
|
+
**Use it** when external consumers (a TUI, a web GUI) need to read a session without re-entering the handler. **Skip it** for pure in-invocation durability — journaled `ctx.run` entries already replay the loop to identical state.
|
|
329
213
|
|
|
330
|
-
|
|
331
|
-
- `list()` returns `[]` — virtual objects don't expose cross-key enumeration. Maintain a separate index handler if discovery matters.
|
|
332
|
-
- `delete()` only clears the current object's keys (no cross-key fan-out).
|
|
333
|
-
- Single-tenant per virtual-object key: every method ignores the `sessionId` argument because the object is already keyed. Calling `appendTurns('other-id', …)` writes to the CURRENT object. Bind one virtual-object key per session and the contract holds.
|
|
214
|
+
Contract limits: `list()` returns `[]` (virtual objects don't enumerate cross-key); `delete()` only clears the current object's keys; every method ignores its `sessionId` argument because the object is already keyed — bind one virtual-object key per session.
|
|
334
215
|
|
|
335
216
|
### Restate-native parallel tools
|
|
336
217
|
|
|
@@ -347,75 +228,46 @@ createAgent({
|
|
|
347
228
|
})
|
|
348
229
|
```
|
|
349
230
|
|
|
350
|
-
|
|
231
|
+
Starts concurrency-safe, Restate-wrapped tools in submission order, joins each chunk through `RestatePromise.all()`, and emits results in original call order. Unsafe tools remain barriers. A safe tool not wrapped with `restateTool()`/`wrapAgentTools()` makes its whole chunk run sequentially instead of falling back to native promise parallelism. Opt-in, so the TUI/headless harness keeps its native sliding-window scheduler.
|
|
351
232
|
|
|
352
233
|
## Subagents
|
|
353
234
|
|
|
354
|
-
`spawn` works under Restate
|
|
235
|
+
`spawn` works under Restate with one rule: **the spawn tool is never journaled — its child's steps are**, because `wrapAgentTools` skips any tool declaring `ToolDef.runsNestedAgent` (the built-in `spawn` sets it).
|
|
355
236
|
|
|
356
237
|
```ts
|
|
357
|
-
const spawnTool = createSpawnTool({
|
|
358
|
-
persist: true, // child turns + runs land in the parent's session
|
|
359
|
-
shareHandle: true, // no unjournaled per-child sandbox mint
|
|
360
|
-
timeoutMs: 600_000,
|
|
361
|
-
})
|
|
238
|
+
const spawnTool = createSpawnTool({ persist: true, shareHandle: true, timeoutMs: 600_000 })
|
|
362
239
|
|
|
363
240
|
createAgent({
|
|
364
241
|
provider: restateProvider(myProvider, ctx, { streamCallbacks: 'suppress' }),
|
|
365
|
-
|
|
366
|
-
tools: wrapAgentTools({ ...tools, spawn: spawnTool }, ctx),
|
|
242
|
+
tools: wrapAgentTools({ ...tools, spawn: spawnTool }, ctx), // spawn passes through unwrapped
|
|
367
243
|
clock: restateClock(ctx),
|
|
368
244
|
session,
|
|
369
245
|
behavior: restateBehavior({ maxTurns: 8 }),
|
|
370
246
|
})
|
|
371
247
|
```
|
|
372
248
|
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
A child agent drives the *same* journaled provider (`ctx.provider`) and the *same* wrapped tools (`ctx.tools`) as its parent. If the spawn tool itself were wrapped, the child's first provider call would open a `ctx.run` **inside** the spawn tool's own `ctx.run`. That fails on the retry with `RestateError (570) Found a mismatch between the code paths taken during the previous execution and the paths taken during this execution` — observed against a real server, which is why `test/fixtures/restate-docker` now carries a subagent scenario.
|
|
249
|
+
If the spawn tool itself were wrapped, the child's first provider call would open a `ctx.run` **inside** the spawn tool's own, which fails on retry with `RestateError 570` (journal mismatch). Instead the child's provider/tool calls become journal entries *of the parent's invocation*: a crash mid-subagent replays completed child steps, and only the unfinished step re-executes.
|
|
376
250
|
|
|
377
|
-
|
|
251
|
+
**Escape hatch**: wrap a spawn tool explicitly (`restateTool(spawnTool, ctx, …)`) only if the child is fully non-durable (raw provider, unwrapped tools) — it collapses into one journal entry, cheap to replay but a crash re-runs the whole child and re-bills every LLM call.
|
|
378
252
|
|
|
379
|
-
|
|
253
|
+
**Inherited automatically**: `AgentClock`, `AgentBehavior` (incl. `toolBatchExecutor`), wrapped tools/provider. **Opt in**: session (`persist: true` — child turn ids come from the store's deterministic mint) and execution handle (`shareHandle: true` — otherwise every child's unjournaled `execution.spawn()` orphans a sandbox on replay). **Never inherited**: durable tool state — a child's `ctx.state` is always an isolated in-memory map.
|
|
380
254
|
|
|
381
|
-
**
|
|
255
|
+
**Per-child timeout.** `createSpawnTool({ timeoutMs })` normally races a `setTimeout` — a wall-clock branch that can abort a live provider stream mid-journal-entry. When the clock reports `journaled: true` (as `restateClock` does), it instead samples the clock at turn boundaries, so a child stuck inside one long tool call isn't interrupted (use the runtime's invocation timeout for a hard cut) and the measured elapsed time spans any crash-to-retry gap. Don't use `behavior.maxWallMs` for this: it clamps the provider watchdogs too, aborting an in-flight stream inside its own journal entry.
|
|
382
256
|
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
| `AgentBehavior` | Automatic, including `toolBatchExecutor` — a child's own concurrency-safe fleets join through `RestatePromise.all()` like the parent's. |
|
|
389
|
-
| Wrapped tools + provider | Automatic (`ctx.tools` / `ctx.provider`) — this is what makes the child's steps journaled at all. |
|
|
390
|
-
| Session | `persist: true`. Child turn ids then come from `restateSessionStore`'s deterministic `(object key, seq)` mint. |
|
|
391
|
-
| Execution handle | `shareHandle: true`. Otherwise every child calls `execution.spawn()` — an unjournaled allocation, so a replayed attempt mints a fresh sandbox and orphans the first. |
|
|
392
|
-
| Durable tool state | **Never.** A child's `ctx.state` is an isolated in-memory map — see the note under Durable tool state below. |
|
|
393
|
-
|
|
394
|
-
### Per-child timeout
|
|
395
|
-
|
|
396
|
-
`createSpawnTool({ timeoutMs })` normally races the child against a `setTimeout`. That is a wall-clock branch: the same child can time out live and not on the replay, and the abort can land mid-stream, tearing a provider call down from inside its own journal entry. When the clock reports `journaled` (as `restateClock` does), the spawn tool leaves the timer off and instead samples the journaled clock at the child's **turn boundaries** — one sample before the run, one per completed turn — aborting the child at a boundary where nothing is in flight. Both mechanisms report `status: 'timeout'`.
|
|
397
|
-
|
|
398
|
-
Deliberately **not** implemented by borrowing `behavior.maxWallMs`, which looks like the natural fit: the loop derives the provider stream watchdogs from the wall budget (`providerWatchdogTimeouts` clamps start / total / idle by it), so a small `timeoutMs` would also shrink the child's TTFT ceiling and abort a live stream inside its `ctx.run`. Against a real server that fails the attempt with `awaitNext already pending` and then `RestateError 570` on the retry. The fixture's deadline scenario exists because that bug shipped for exactly one iteration.
|
|
399
|
-
|
|
400
|
-
Two consequences of the turn-boundary check: a child stuck inside ONE long tool call is not interrupted (use the runtime's own invocation timeout for a hard cut), and the measured elapsed time **spans the crash-to-retry gap**, because the run-start timestamp replays from before the crash. Budget accordingly.
|
|
401
|
-
|
|
402
|
-
### Limits
|
|
403
|
-
|
|
404
|
-
- **Sequential spawns only** with `createSpawnTool` — for parallel children use `restateSpawnTool` (next section), which sidesteps both reasons below by giving each child its own journal. The weaker one: parallel children race on shared session state — the `child-N` label counter and the run-id re-sync both read `session.runs` at mint time — so label / run-id assignment can differ between attempts. The stronger one: children share the parent's *clock instance*, and `restateClock` carries mutable draw state (the `randomUUID` sequence counter, plus the anchor / call counter in `coalesce` mode). Concurrent children interleave those draws, so the same logical turn can draw a different id or timestamp ordinal on a replay — the `JournalMismatchError` class the clock module exists to prevent. `restateBehavior`'s default `maxConcurrentTools: 1` already gives you the safe posture; don't raise it for a turn that can spawn more than once.
|
|
405
|
-
- **Anything that reaches the transcript must come from `ctx.clock`.** The spawn result string is persisted in the parent's turns, and a state write's payload IS compared across attempts — so the elapsed figure in `[sub-agent child-1] Completed in 2 turns (35ms)` is sampled from the clock, not from `stats.elapsed` (which stays live `Date.now()` by design, for measurement accuracy). Getting this wrong fails the invocation at `set state` with `RestateError 570`, and only on an attempt that fails AFTER a child completed — a crash mid-child never produces the string, so the failure hides until a retry happens later in the run. The fixture's retry scenario exists for exactly that reason. The same rule is why the child inherits the clock at all.
|
|
406
|
-
- **The parent's spawn tool call is not itself journaled**, so it re-executes on replay: agent construction, MCP re-use, hook bubbling, and result-string formatting all run again. Keep host `onChildStart` / `spawn:*` handlers idempotent.
|
|
407
|
-
- **Don't set `behavior.maxWallMs` on a Restate-backed agent** (parent or child) unless you mean it: the provider stream watchdogs are clamped by it, so a tight budget aborts an in-flight stream from inside its journal entry rather than stopping cleanly at a turn boundary. Reach for the runtime's own invocation timeout, or `createSpawnTool({ timeoutMs })` for a child.
|
|
257
|
+
**Limits**:
|
|
258
|
+
- Sequential spawns only with `createSpawnTool` — parallel children would race on shared session state (label counter, run-id re-sync) and the parent clock's mutable draw state. Use `restateSpawnTool` for parallel children; don't raise `maxConcurrentTools` for a turn that can spawn more than once.
|
|
259
|
+
- Anything reaching the transcript must come from `ctx.clock`, never `stats.elapsed` — a live timestamp in a journaled state write fails the invocation on a later retry.
|
|
260
|
+
- The parent's spawn tool call re-executes on replay (agent construction, MCP re-use, hook bubbling, result formatting) since it isn't journaled — keep `onChildStart`/`spawn:*` handlers idempotent.
|
|
261
|
+
- Don't set `behavior.maxWallMs` on a Restate-backed agent, parent or child — same watchdog-clamp hazard. Use the runtime's invocation timeout or `createSpawnTool({ timeoutMs })`.
|
|
408
262
|
|
|
409
263
|
## Parallel subagents
|
|
410
264
|
|
|
411
|
-
`createSpawnTool` runs children in the parent's journal
|
|
265
|
+
`createSpawnTool` runs children in the parent's journal (sequential by construction). `restateSpawnTool` dispatches each child to **its own invocation** so children fan out through the durable fleet, and a crash in one child retries only that child.
|
|
412
266
|
|
|
413
267
|
```ts
|
|
414
268
|
const spawn = restateSpawnTool({
|
|
415
|
-
//
|
|
416
|
-
|
|
417
|
-
// Stop in-flight children when the parent's own loop aborts.
|
|
418
|
-
onAbort: () => ctx.invocation(ctx.request().id).cancel(),
|
|
269
|
+
dispatch: req => ctx.serviceClient(childService).runChild(req), // return the promise UNTOUCHED — no await/.then
|
|
270
|
+
onAbort: () => ctx.invocation(ctx.request().id).cancel(), // stop in-flight children on parent loop abort
|
|
419
271
|
})
|
|
420
272
|
|
|
421
273
|
createAgent({
|
|
@@ -423,14 +275,12 @@ createAgent({
|
|
|
423
275
|
clock: restateClock(ctx),
|
|
424
276
|
session,
|
|
425
277
|
behavior: restateBehavior({
|
|
426
|
-
maxConcurrentTools: 4, //
|
|
278
|
+
maxConcurrentTools: 4, // default 1 — without this there is no fan-out
|
|
427
279
|
toolBatchExecutor: restateToolBatchExecutor({ promises: RestatePromise }),
|
|
428
280
|
}),
|
|
429
281
|
})
|
|
430
282
|
```
|
|
431
283
|
|
|
432
|
-
### Choosing between the two
|
|
433
|
-
|
|
434
284
|
| | `createSpawnTool` | `restateSpawnTool` |
|
|
435
285
|
|---|---|---|
|
|
436
286
|
| Children per turn | one at a time | parallel (fleet-capped) |
|
|
@@ -439,337 +289,195 @@ createAgent({
|
|
|
439
289
|
| `child:*` events | live (in-process bubbling) | replayed onto the parent bus when the child merges |
|
|
440
290
|
| Host code needed | none | a child handler |
|
|
441
291
|
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
-
|
|
445
|
-
-
|
|
446
|
-
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
-
|
|
452
|
-
-
|
|
453
|
-
-
|
|
454
|
-
-
|
|
455
|
-
-
|
|
456
|
-
-
|
|
457
|
-
- **An in-agent `agent.abort()` needs `onAbort`.** On its own, `abort()` unwinds the parent's loop while its invocation keeps running, so nothing tells the children to stop. `onAbort` fires once with the in-flight set; cancel the parent's own invocation from it and the runtime cancels the tree. The cost is that the parent invocation ends too — an in-agent abort becomes "stop everything", so persist any summary BEFORE aborting.
|
|
458
|
-
- **Cancelling individual children is not possible in this tool shape.** The `invocationId` on the runtime's call handle is itself a journal-backed promise, so reading it while the call is in flight registers a second consumer on the SDK's single-consumer progress channel: `awaitNext already pending`, a failed attempt, and a replay on *every* fan-out (measured — 13 spurious attempts across the fixture suite before this was reverted). Per-child cancellation needs a two-phase `send → id → attach` tool, where the id is consumed before the result promise exists.
|
|
292
|
+
**Host obligations**:
|
|
293
|
+
- The child handler must be a `service`, or an object on a *different* key — a virtual object serializes per key, so children sharing the parent's key would queue behind it.
|
|
294
|
+
- Raise `maxConcurrentTools` and install `restateToolBatchExecutor` — the fleet gate is all-or-nothing (`fleet.every(canExecuteDurably)`), so the default cap of `1` serializes children silently.
|
|
295
|
+
- The child runs sessionless and returns a `ChildRunResult` (`status`, `text`, `turns`, `stats`); the parent is the single writer, merging each child's turns under a labeled `SessionRun`.
|
|
296
|
+
- Enforce depth in the handler (`req.depth`) — the tool always dispatches; short-circuiting would return a non-durable promise, which the fleet rejects.
|
|
297
|
+
|
|
298
|
+
**What you give up**:
|
|
299
|
+
- `child:*` events fire, but not live — see [Child events](#child-events).
|
|
300
|
+
- `spawn:before` fires without awaiting — `durableExecute` must return the durable promise synchronously.
|
|
301
|
+
- Merge happens on `tool-results:after` (with `turn:after` as backstop), so merged turns land just after the parent's tool-result turn, not before (`createdAt` still orders them correctly). A child that settles after the invocation tears down (cancellation) is never merged.
|
|
302
|
+
- The child should report its own `startedAt`/`endedAt` off its own clock — they land in a journaled state write and must be replay-stable. Omitted, it falls back to first/last turn timestamps, then the parent's run start.
|
|
303
|
+
- Per-child timeouts are the callee's job (racing the call promise is the forbidden native-combinator pattern) — give the child handler its own budget or use `.orTimeout()`.
|
|
304
|
+
- Cancelling the parent invocation cancels the whole call tree; a step already inside `ctx.run` finishes first.
|
|
305
|
+
- An in-agent `agent.abort()` needs `onAbort` wired — alone, `abort()` unwinds the parent's loop but leaves children running. `onAbort` fires once with the in-flight set; cancel the parent's own invocation from it (this ends the parent invocation too, so persist any summary first).
|
|
306
|
+
- Cancelling *individual* children isn't possible in this tool shape: reading a call handle's `invocationId` mid-flight registers a second consumer on the SDK's single-consumer progress channel (`awaitNext already pending`, replay on every fan-out). Needs a two-phase send → id → attach tool instead.
|
|
459
307
|
|
|
460
308
|
### Child events
|
|
461
309
|
|
|
462
|
-
Hook bubbling can't cross an invocation boundary, so
|
|
463
|
-
|
|
464
|
-
| Fires | From |
|
|
465
|
-
|---|---|
|
|
466
|
-
| `child:stream:text` | assistant text, one chunk per turn |
|
|
467
|
-
| `child:tool:before` | each `tool_call` block, with a run-cumulative `runToolCounts` tally |
|
|
468
|
-
| `child:tool:after` / `child:tool:error` | the matching `tool_result`, split on `isError` |
|
|
469
|
-
| `child:turn:after` | each assistant turn, `usage` straight off the turn |
|
|
470
|
-
| `child:stream:end` | end of each turn |
|
|
471
|
-
|
|
472
|
-
Every event carries `childId` (the reserved label), `depth`, and — on the tool events, which extend `ToolHookContext` — `runId`, `parentRunId`, and `parentCallId` (the spawn call that owns the child). Per-child blocks stay contiguous and follow **submission** order, not settlement order.
|
|
473
|
-
|
|
474
|
-
**Not synthesized**: `tool:gate`, `tool:transform`, `tool:dispatched`, `tool:cancelled`, `background:*`. Those describe decisions taken inside another process; inventing them would be fiction.
|
|
475
|
-
|
|
476
|
-
**They re-fire on a parent replay**, exactly as the in-process tool's `child:tool:*` does — the drain runs live on every attempt. Dedupe by `callId` / `turnId`, or gate on `createReplayTracker`. The fixture pins this rather than asserting it in prose: a post-merge replay emits 24 events for 12 events' worth of work, measured through a file sink that outlives the attempt.
|
|
477
|
-
|
|
478
|
-
#### Nested fan-out
|
|
479
|
-
|
|
480
|
-
A child dispatched to a service runs **sessionless**, so it has nowhere to persist its own children. Left alone, a depth-2 fan-out loses everything below depth 1 — the parent sees the child's `spawn` tool calls and none of the work they did. Hand the subtree up instead:
|
|
310
|
+
Hook bubbling can't cross an invocation boundary, so each child's transcript is **replayed onto the parent's bus when it merges** (`child:stream:text`, `child:tool:before/after/error`, `child:turn:after`, `child:stream:end`) — existing `child:tool:*` listeners need no changes. Every event carries `childId`, `depth`, and (tool events) `runId`/`parentRunId`/`parentCallId`; blocks stay contiguous per child in submission order. Not synthesized: `tool:gate`, `tool:transform`, `tool:dispatched`, `tool:cancelled`, `background:*` (decisions taken inside another process).
|
|
481
311
|
|
|
482
|
-
|
|
483
|
-
const merged: MergedChild[] = []
|
|
484
|
-
const spawn = restateSpawnTool({
|
|
485
|
-
onChildMerged: child => merged.push(child),
|
|
486
|
-
dispatch: req => ctx.serviceClient(childService).runChild(req),
|
|
487
|
-
})
|
|
488
|
-
// … and return it from the handler
|
|
489
|
-
return { status: 'completed', text, turns: agent.turns, stats, children: merged }
|
|
490
|
-
```
|
|
491
|
-
|
|
492
|
-
The merging parent walks that subtree depth-first, **re-labeling** each descendant from its own session counter (every level mints `child-1` locally, so the arriving labels collide), and emits their `child:*` events at the right depth. Verified end to end: parent → one child → two grandchildren lands four runs at depths 0/1/2/2 with labels `child-1` / `child-2` / `child-3`, and the grandchildren's tool events arrive at `depth: 2`.
|
|
312
|
+
They **re-fire on a parent replay** — dedupe by `callId`/`turnId`, or gate on `createReplayTracker`. Disable with `emitChildEvents: false`.
|
|
493
313
|
|
|
494
|
-
|
|
314
|
+
**Nested fan-out**: a child dispatched to a service is sessionless, so a depth-2 fan-out loses everything below depth 1 unless the subtree is handed up via `onChildMerged: child => merged.push(child)`, returned as `children: merged` from the handler. The merging parent walks it depth-first, re-labels each descendant from its own session counter (every level mints `child-1` locally), and emits `child:*` at the right depth.
|
|
495
315
|
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
- `restateProvider(..., { streamCallbacks: 'suppress' })` no-ops `onText`, so text deltas never fire even inside the child. Drop `suppress` and you get `maxRetryAttempts: 1` instead — otherwise an entry retry re-emits deltas and duplicates text.
|
|
499
|
-
- A child's events re-fire on the *child's* own replay, so the sink must be idempotent.
|
|
500
|
-
|
|
501
|
-
What's deliberately not offered is journaling each child event so the parent can consume it as a journal entry. That is the only live option that stays inside the determinism envelope, and it costs a journal entry per event plus a parent that must stay awake to read them — which defeats the suspension that makes long children cheap. Defensible at coarse milestones, not per token.
|
|
502
|
-
|
|
503
|
-
Every property here is pinned by the fixture against a real server: two children overlapping by ~394ms of their 400ms of work; four children at a cap of two running as two clean waves (max concurrency exactly 2, wall time 1031ms versus 2090ms for the same fan-out at cap 1); a crash inside one child leaving its sibling with exactly one execution; and both children's transcripts arriving on the parent's bus as `child:*` events, in contiguous per-child blocks.
|
|
316
|
+
**For live events instead**, publish from the child's own bus inside your handler (it's an ordinary agent there); `ChildRunRequest` carries `parentRunId`/`parentCallId`/`sessionId` for tagging. `restateProvider(..., { streamCallbacks: 'suppress' })` no-ops `onText` even inside the child — drop `suppress` if you need live text. A child's events re-fire on the child's own replay, so the sink must be idempotent.
|
|
504
317
|
|
|
505
318
|
## Durable tool state
|
|
506
319
|
|
|
507
|
-
The problem
|
|
320
|
+
**The problem**: a tool cannot keep state in a closure. A journaled result skips the tool body on replay, so a `consumed = true` written inside `execute` never happens the second time — the tool re-serves a payload it already delivered.
|
|
508
321
|
|
|
509
322
|
`ToolContext.state` is a key/value map that survives replays, suspensions, and whole invocations:
|
|
510
323
|
|
|
511
324
|
```ts
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
return 'already delivered'
|
|
517
|
-
ctx.state?.set('delivered', true)
|
|
518
|
-
return problems
|
|
519
|
-
},
|
|
325
|
+
async execute(_input, ctx) {
|
|
326
|
+
if (ctx.state?.get<boolean>('delivered')) return 'already delivered'
|
|
327
|
+
ctx.state?.set('delivered', true)
|
|
328
|
+
return problems
|
|
520
329
|
}
|
|
521
|
-
|
|
522
|
-
createAgent({
|
|
523
|
-
tools: wrapAgentTools({ read_errors: readErrors }, ctx),
|
|
524
|
-
toolState: restateToolStateStore(ctx),
|
|
525
|
-
})
|
|
330
|
+
// createAgent({ tools: wrapAgentTools({ read_errors: readErrors }, ctx), toolState: restateToolStateStore(ctx) })
|
|
526
331
|
```
|
|
527
332
|
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
Reads are synchronous by design. A wrapped tool body runs inside `ctx.run`, and Restate forbids context use in there (no `ctx.get`, no `ctx.sleep`, no nested `ctx.run`) — so the durable I/O happens on either side of the body instead:
|
|
333
|
+
**Three phases**, because Restate forbids context use (`ctx.get`/`ctx.sleep`/nested `ctx.run`) inside a `ctx.run` body:
|
|
531
334
|
|
|
532
335
|
| Phase | When | Where |
|
|
533
336
|
|---|---|---|
|
|
534
|
-
|
|
|
535
|
-
|
|
|
536
|
-
|
|
|
537
|
-
|
|
538
|
-
The load is memoized on the controller, and the controller is built once in `createAgent` — so a long-lived agent doing many runs loads exactly once. Under Restate that's one agent per invocation and the distinction doesn't bite, but elsewhere it means a resident agent never picks up state another process wrote to the same store out of band. Build a fresh agent (or a fresh controller) per unit of work if you need that.
|
|
337
|
+
| hydrate | Once per agent instance, before the first tool dispatch | One `ctx.get`, outside every journaled action |
|
|
338
|
+
| stage | Inside the tool body | In memory — `set`/`delete` record ops, not written through |
|
|
339
|
+
| commit | When the call settles | One `ctx.set`, outside the journaled action |
|
|
539
340
|
|
|
540
|
-
The
|
|
341
|
+
The load is memoized on the controller (built once per `createAgent`), so one invocation loads exactly once. Replay safety: **a staged mutation commits only if it came back out of a journal entry** — `restateTool` ships staged ops as part of the journaled value (`{ __zidaneToolState: 1, ops, output }`) and re-seeds them outside it. Attempts with no journal entry (control error, abort, dead process) drop their staged ops; replay re-runs the body and re-stages.
|
|
541
342
|
|
|
542
|
-
|
|
343
|
+
**Semantics**:
|
|
344
|
+
- Configuring `toolState` at all changes the journal shape (adds a run-start `ctx.get`) — drain in-flight invocations before rolling it out. A call that writes no state journals unchanged.
|
|
345
|
+
- Commit order is call-index order, never completion order — a durable fleet joined via `RestatePromise.all()` commits only once fully settled.
|
|
346
|
+
- A write is visible once its call commits, not when staged (read-through to the live snapshot). Soft tool failures still commit — sentinel and delta land in the same journal entry.
|
|
347
|
+
- A store **failure drops the write, never fails the call** (the tool body already ran). Fires `tool-state:commit-failed` with the dropped ops (set `handled = true` to own reporting), else a stderr fallback. Make `commit` idempotent if the write matters.
|
|
348
|
+
- `ToolStateView.set` caps values at 64 KiB of JSON (`restateToolStateStore(ctx, { maxValueBytes })` to override) and throws inside the tool body — an oversized write that reached commit would instead fail the whole invocation with `RT0003`. Non-serializable values are rejected the same way.
|
|
349
|
+
- A failing `load` fails the run — asymmetric with `commit` on purpose, since running on an empty snapshot re-arms the bug this exists to prevent. `set(key, undefined)` deletes.
|
|
350
|
+
- Two-phase tools (`durableStart`) journal their delta on the final entry (`wait.result`/`attach`) — a write staged in `start` is lost if the process dies before the wait completes.
|
|
351
|
+
- Keep values small and JSON-serializable (flags, ids, counters, cursors, not transcripts). Not inherited by subagents — `createSpawnTool` has no `shareToolState`; a spawned child's tools get an isolated in-memory map unless you construct its agent explicitly with the same store.
|
|
352
|
+
- The store is bound to one invocation's `ctx` — build it inside the handler, like `restateSessionStore`/`restateProvider`. It's one JSON object with no built-in versioning; use `restateToolStateStore(ctx, { key })` for a clean namespace on migration.
|
|
543
353
|
|
|
544
|
-
|
|
545
|
-
- **Commit order is call-index order**, never completion order — including durable fleets joined through `RestatePromise.all()`. Two parallel tools writing the same key would otherwise produce a nondeterministic write sequence.
|
|
546
|
-
- **A write becomes visible when its call commits**, not when it is staged. Views read through to the live snapshot, so a call that runs *after* a sibling committed does see that write (the sequential path `restateBehavior` defaults to reads back the previous call's write). A durable fleet commits only once the whole batch has settled, so calls racing inside one batch never see each other.
|
|
547
|
-
- **Errors still commit.** A soft tool failure journals its sentinel and its delta in the same entry, so the write is honored — live and replay agree.
|
|
548
|
-
- **A store failure drops the write, it doesn't fail the call.** The tool body already ran, so failing the run over a bookkeeping write would be worse than losing it — but losing it silently is worse still, since a dropped one-shot flag re-arms the exact bug this state prevents. A rejecting `commit` fires the `tool-state:commit-failed` hook with the ops that did NOT land (replaying them by hand is a legitimate recovery), and falls back to an unconditional stderr line when nothing is listening; set `handled = true` in your handler to own the reporting. A throwing handler can't take the dispatch path down with it. Make `commit` idempotent and retry inside it if the write matters more than latency.
|
|
549
|
-
- **Oversized values are rejected where they're written.** `ToolStateView.set` enforces a per-value budget (`DEFAULT_MAX_VALUE_BYTES`, 64 KiB of JSON; override with `restateToolStateStore(ctx, { maxValueBytes })`) and throws inside the tool body rather than letting the write fail at commit time. That matters more under Restate than elsewhere: an oversized state value doesn't fail politely, it fails the invocation with `RT0003` (MessageSizeLimit) and retries it under the configured policy — a fat write is a poison pill, not a dropped write. Non-serializable values (cycles, functions) are rejected the same way.
|
|
550
|
-
- **A failing `load` fails the run**, deliberately asymmetric with `commit`: running on an empty snapshot would re-arm the one-shot bugs this state exists to prevent. It throws from `agent.run()` on the normal error path (`errorRun` + `agent:done` still fire), and the memo is cleared so the next run retries.
|
|
551
|
-
- **`set(key, undefined)` is a deletion.** It is what a read would report anyway, and an undefined value is not JSON-stable across the journal.
|
|
552
|
-
- **Two-phase tools** (`durableStart`) journal their delta on the final entry (`wait.result` / `attach`). A write staged in `start` is lost if the process dies before the wait completes; write state in the final phase when it must be crash-durable.
|
|
553
|
-
- **Keep values small and JSON-serializable.** They cross a journal and land in virtual-object state, which Restate size-caps: flags, ids, counters, cursors — not transcripts.
|
|
554
|
-
- **Not inherited by subagents.** A child you construct yourself can take the same store; `createSpawnTool` has **no** `shareToolState` equivalent to `shareReadState`, so tools running inside a *spawned* subagent get an isolated in-memory map and their writes never reach the parent's store. This is the one thing a subagent does NOT get under Restate — everything else (journaled clock, wrapped tools/provider, session, handle) is covered in § Subagents. A child's one-shot delivery is therefore not crash-safe; keep such tools in the parent, or hand the child an explicitly constructed agent with the same `toolState` store.
|
|
555
|
-
- **The store is bound to one invocation's `ctx`.** Build it inside the handler, like `restateSessionStore` and `restateProvider`. An agent constructed once at module scope and reused across invocations would commit through a dead context — and never re-hydrate. `toolState` reads like agent-lifetime configuration, which makes this easier to get wrong than the other ctx-bound wrappers.
|
|
556
|
-
- **The stored map is a schema you own.** It is one JSON object under one state key, with no versioning or migration built in. Renaming a key or changing a value's shape has to tolerate the old value still being there from a previous deploy; `restateToolStateStore(ctx, { key })` gives you a clean namespace when a migration is easier than a tolerant read.
|
|
557
|
-
|
|
558
|
-
### Without Restate
|
|
559
|
-
|
|
560
|
-
`ToolStateStore` is two methods (`load`, `commit`) and nothing in it is Restate-specific — implement it against Redis, Postgres, or a file for other hosts. With no store configured `ctx.state` is still present and functional, backed by an in-memory map scoped to the agent instance, so a tool written against it behaves identically on the local CLI path and only loses durability.
|
|
354
|
+
**Without Restate**: `ToolStateStore` is just `{ load, commit }` — implement it against Redis/Postgres/a file for other hosts. With no store configured, `ctx.state` still works, backed by an in-memory map scoped to the agent instance (functional, not durable).
|
|
561
355
|
|
|
562
356
|
## Recommended `AgentBehavior` for Restate deployments
|
|
563
357
|
|
|
564
358
|
```ts
|
|
565
359
|
createAgent({
|
|
566
|
-
|
|
567
|
-
behavior: restateBehavior({
|
|
568
|
-
maxCostUsd: 5,
|
|
569
|
-
maxTotalTokens: 2_000_000,
|
|
570
|
-
}),
|
|
360
|
+
behavior: restateBehavior({ maxCostUsd: 5, maxTotalTokens: 2_000_000 }),
|
|
571
361
|
})
|
|
572
362
|
```
|
|
573
363
|
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
-
|
|
578
|
-
|
|
364
|
+
| Default | Why |
|
|
365
|
+
|---|---|
|
|
366
|
+
| `maxConcurrentTools: 1` | Strict-first. Enable parallelism explicitly via `restateToolBatchExecutor`. |
|
|
367
|
+
| `disableBackgroundTasks: 'non-durable'` | Background tasks (`shell` with `run_in_background`) stay enabled only when the context declares `capabilities.detachedTasks: 'durable'` — host-local tasks aren't journaled. |
|
|
368
|
+
| `cache: true` | Stable system/tool prefixes reward prompt-cache breakpoints, cutting cold-start cost on retries. |
|
|
369
|
+
| `shouldRethrowToolError: shouldRethrowRestateControlError` | Restate terminal/suspension control errors escape the tool-result path instead of becoming a fake tool error. |
|
|
370
|
+
| `compactStrategy: 'tail'`, `compactThreshold: 128 KiB`, `compactKeepTurns: 8`, `elideStaleReads: 'stub'`, `mediaKeepTurns: 8`, `toolOutputBudget: 64 KiB`, `autoCompact: { threshold: 0.85 }` | Shrinks the per-turn provider payload — journal weight and latency, not just tokens. |
|
|
371
|
+
|
|
372
|
+
Add `maxCostUsd`/`maxTotalTokens` as a server-side budget breaker: throws `AgentBudgetExceededError` and finalizes the run as `aborted` — durable replay+retry otherwise turns a runaway loop into a runaway bill. See `docs/ARCHITECTURE.md` § run-level budget circuit breaker.
|
|
579
373
|
|
|
580
|
-
|
|
374
|
+
## MCP tools
|
|
581
375
|
|
|
582
|
-
MCP servers re-bootstrap on every handler invocation (`warmup()` is lazy + idempotent). Discovered MCP tools aren't reachable through the public tool-wrapping seam (`wrapAgentTools` only sees
|
|
376
|
+
MCP servers re-bootstrap on every handler invocation (`warmup()` is lazy + idempotent). Discovered MCP tools aren't reachable through the public tool-wrapping seam (`wrapAgentTools` only sees tools you pass), so their raw `callTool` never rides `ctx.run` unless you wire `mcpToolWrap`:
|
|
583
377
|
|
|
584
378
|
```ts
|
|
585
|
-
|
|
379
|
+
createAgent({
|
|
586
380
|
provider: restateProvider(anthropic({ defaultModel: 'claude-sonnet-4-5' }), ctx),
|
|
587
|
-
tools: wrapAgentTools(nativeTools, ctx),
|
|
381
|
+
tools: wrapAgentTools(nativeTools, ctx),
|
|
588
382
|
mcpServers,
|
|
589
|
-
mcpToolWrap: restateMcpToolWrap(ctx),
|
|
383
|
+
mcpToolWrap: restateMcpToolWrap(ctx), // discovered MCP tools: journaled too
|
|
590
384
|
clock: restateClock(ctx),
|
|
591
385
|
session,
|
|
592
386
|
behavior: restateBehavior(),
|
|
593
387
|
})
|
|
594
388
|
```
|
|
595
389
|
|
|
596
|
-
The agent
|
|
390
|
+
The agent applies the wrap to every discovered tool regardless of `mcpConnector` or lazy-connect timing; the wrap only re-shapes `execute`.
|
|
597
391
|
|
|
598
|
-
-
|
|
599
|
-
-
|
|
600
|
-
-
|
|
392
|
+
- Default is plain result-journaling. A genuinely long-running MCP tool (a remote build, a migration) can opt into two-phase `durableStart` per-tool via `override`; `exclude` skips journaling for deterministic reads.
|
|
393
|
+
- Error semantics match native tools: `restateMcpToolWrap` applies the same `softErrors` mechanism, so a thrown MCP failure becomes `tool_result { isError }` instead of a fatal `TerminalError`.
|
|
394
|
+
- Journaling is orthogonal to connection — keep any custom `mcpConnector` (OAuth, custom transport) and add `mcpToolWrap: restateMcpToolWrap(ctx)` alongside it.
|
|
601
395
|
|
|
602
396
|
Hosts that journal bootstrap outcomes should hook the deterministic `mcp:bootstrap:settled` aggregate (one name-sorted event per batch) instead of the per-server `mcp:bootstrap:end` stream, which fires in network-completion order.
|
|
603
397
|
|
|
604
|
-
Host-only configuration rides in `behavior.extensions`
|
|
398
|
+
Host-only configuration rides in `behavior.extensions` (a namespaced bag threaded onto `ToolContext.behavior.extensions`) rather than unknown top-level `AgentBehavior` fields.
|
|
605
399
|
|
|
606
400
|
## Long-running background jobs (durable contexts)
|
|
607
401
|
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
1. **Pull-based exit reconcile.** The context does NOT need to call `execBackground`'s `onExit` (it's optional — a remote context has no legal way to push from a timer into a journal). At the start of every `agent.run()` the loop sweeps `listBackground` for terminated tasks the model hasn't seen and merges them with any push-fed exits. `TaskEntry.endedAt` makes `durationMs` sweep-time-independent.
|
|
611
|
-
2. **Deterministic reconcile (`journalReconcile`).** Pass `journalReconcile` on `agent.run()` to record the `listBackground` sweep as a single journaled side effect — replay returns the recorded snapshot instead of re-polling the registry live, so the (conditionally-fired) `background:reconcile` hook lands at a stable journal position across attempts. Everything downstream of the snapshot is pure over journaled/replayed inputs, so this is enough to make the whole reconcile replay-deterministic *without* journaling the entire `ExecutionContext`:
|
|
612
|
-
|
|
613
|
-
```ts
|
|
614
|
-
await agent.run({
|
|
615
|
-
prompt,
|
|
616
|
-
clock: restateClock(ctx),
|
|
617
|
-
journalReconcile: sweep => ctx.run('background-reconcile', sweep),
|
|
618
|
-
})
|
|
619
|
-
```
|
|
620
|
-
|
|
621
|
-
The `background:reconcile` hook still fires on the merged exit list inside the `run()` await chain — mutate `hookCtx.exits` to filter / reorder what the model sees. (Before `journalReconcile`, hosts journaled inside that hook via `hookCtx.exits = await ctx.run('background-reconcile', () => hookCtx.exits)`; that still works, but `journalReconcile` is the deterministic seam because it also journals the underlying `listBackground` poll, not just the post-sweep list.)
|
|
622
|
-
|
|
623
|
-
3. **`waitBackground` / `wait_task`.** When the model genuinely needs a task's outcome before continuing, the `wait_task` tool delegates to `ExecutionContext.waitBackground` — an injectable seam. In-process contexts park on their own `onExit` machinery; durable hosts implement it as an awakeable park (runner fires task-exit over their bridge → `resolveAwakeable`), so a crashed worker re-attaches to the same wait on replay. `wrapAgentTools({ detachedTasks: 'durable' })` automatically leaves `wait_task` unwrapped — a Restate suspension point must not nest inside a journaled `ctx.run` closure (same rule as `restateAwakeableTool`). Hosts whose wait is journal-safe (e.g. a bridge RPC poll) can re-wrap it explicitly via `restateTool`.
|
|
624
|
-
|
|
625
|
-
Because `wait_task` is excluded from journaling its body re-runs on every replay, so on a `'durable'` context it must not branch on an un-journaled read. It therefore skips its "does this task exist?" `listBackground` pre-check and lets `waitBackground`'s own journaled probe decide — an unknown task collapses into the wait's `null` (the timeout result), replay-stable even if the runner's registry changed across attempts (e.g. a restart mid-park). Non-durable contexts keep the precise "no such task" message (no journal to diverge).
|
|
402
|
+
General background-task mechanics (`execBackground`, `listBackground`, the `waitBackground` seam, output caps, stall watchdog, `detachedTasks` capability) are owned by `docs/RUN_IN_BACKGROUND.md`. This section covers only the Restate-specific wiring.
|
|
626
403
|
|
|
627
|
-
|
|
404
|
+
1. **`restateWaitBackground`** packages the race-safe choreography for `ExecutionContext.waitBackground`: mint an awakeable → register the resumer id **before** probing `listBackground` (probing first risks a lost wakeup) → fast-path if already exited/unknown → park outside the journal, raced against the durable deadline → settle. The host supplies only persistence callbacks (`register`, `probe`, `settle`):
|
|
628
405
|
|
|
629
406
|
```ts
|
|
630
|
-
import { restateWaitBackground } from 'zidane/restate'
|
|
631
|
-
|
|
632
407
|
execution.waitBackground = restateWaitBackground({
|
|
633
408
|
ctx,
|
|
634
|
-
// Persist the awakeable id as the task's exit resumer BEFORE the probe.
|
|
635
409
|
register: (awakeableId, taskId) => persistExitResumer(taskId, awakeableId), // runner → ctx.resolveAwakeable(awakeableId, exitInfo)
|
|
636
|
-
// Snapshot the registry (the `listBackground` shape) to detect already-exited / unknown tasks.
|
|
637
410
|
probe: handle => execution.listBackground!(handle),
|
|
638
|
-
// Finalize the resumer row (exit info, or null on timeout / abort / unknown).
|
|
639
411
|
settle: (awakeableId, taskId) => clearExitResumer(taskId),
|
|
640
412
|
})
|
|
641
413
|
```
|
|
642
414
|
|
|
643
|
-
`
|
|
644
|
-
|
|
645
|
-
```ts
|
|
646
|
-
import { RestatePromise } from '@restatedev/restate-sdk'
|
|
647
|
-
import { awakeableWithTimeout } from 'zidane/restate'
|
|
648
|
-
|
|
649
|
-
const { id, promise } = awakeableWithTimeout<TaskExitInfo>(ctx, { timeoutMs: opts?.timeoutMs, promises: RestatePromise })
|
|
650
|
-
await registerExitResumer(taskId, id) // runner → ctx.resolveAwakeable(id, exitInfo)
|
|
651
|
-
return promise // TaskExitInfo, or null on timeout
|
|
652
|
-
```
|
|
653
|
-
4. **Cross-invocation dedupe is history-derived.** The agent's in-memory notified-task latch dies with the handler invocation, but the persisted session turns carry every acknowledgment (`<task-notification>` blocks, `shell_kill` calls, successful `wait_task` results, post-exit `read_file`s of the output path). The reconcile sweep re-derives the latch from the **full** `session.turns` history — subagent-run acks included — so a fresh agent on a resumed session (the normal Restate shape) never re-notifies an exit a prior invocation already delivered. This survives the built-in compaction: `compact-summary` markers elide turns at the wire level only, persisted turns are append-only by contract (the same assumption lazy-disclosure replay and read-state hydration make). Read-acks cross two clock domains (host `AgentClock` vs runner wall clock), so they apply only when the read landed ≥ 5 s after the exit — within that window the agent fails open (one redundant notification, never false suppression). Only a host that rewrites `session.turns` destructively forfeits the dedupe; remote registries can prune terminated entries after delivery as a belt-and-braces measure there.
|
|
654
|
-
|
|
655
|
-
Guardrails for unattended jobs (any context): `behavior.backgroundOutputCap` caps each task's log file without killing the process (a `<output-truncated bytes-dropped="N"/>` marker is appended at settle), and `behavior.backgroundStallWatchdogMs` queues a one-shot `<task-stall>` notification when a task goes quiet — typically a process stuck at an interactive `(y/n)` prompt that the model should `shell_kill` and re-run non-interactively.
|
|
415
|
+
`wrapAgentTools({ detachedTasks: 'durable' })` leaves `wait_task` unwrapped automatically (same no-suspension-inside-`ctx.run` rule as `restateAwakeableTool`) and it skips its own `listBackground` pre-check there, deferring to the journaled probe inside `restateWaitBackground` instead.
|
|
656
416
|
|
|
657
|
-
|
|
417
|
+
2. **`journalReconcile`** records the per-run `listBackground` sweep as one journaled side effect, keeping the `background:reconcile` hook at a stable journal position across attempts: `agent.run({ prompt, clock: restateClock(ctx), journalReconcile: sweep => ctx.run('background-reconcile', sweep) })`.
|
|
658
418
|
|
|
659
|
-
|
|
660
|
-
|
|
661
|
-
```ts
|
|
662
|
-
createAgent({
|
|
663
|
-
// … wrappers as above
|
|
664
|
-
skills: {
|
|
665
|
-
skipDefaultPaths: true, // worker has no skill dirs of its own
|
|
666
|
-
catalog: hostProvidedCatalog, // pre-resolved by the worker bootstrap
|
|
667
|
-
},
|
|
668
|
-
})
|
|
669
|
-
```
|
|
670
|
-
|
|
671
|
-
`baseDir` / `location` on each catalog entry must point to paths resolvable on the runner (where `skills_read` / `skills_run_script` ultimately execute). Catalog entries seed resolution first and shadow disk-discovered skills on name collision — useful if a project still wants to layer `.zidane/skills/*` on top of a host-provided catalog.
|
|
419
|
+
3. **Cross-invocation notification dedupe is history-derived.** The in-memory notified-task latch dies with the handler invocation; the reconcile sweep re-derives it from the full `session.turns` history (`<task-notification>` blocks, `shell_kill` calls, `wait_task` results, post-exit `read_file`s), so a fresh agent on a resumed session never re-notifies an exit already delivered. Read-acks apply only ≥5s after the exit (fails open otherwise). Requires `session.turns` to stay append-only.
|
|
672
420
|
|
|
673
421
|
## Edge & long-lived deployments
|
|
674
422
|
|
|
675
|
-
The journal makes the *loop* durable; these seams
|
|
423
|
+
The journal makes the *loop* durable; these seams cover the rest of the runtime on a long-lived/edge host.
|
|
676
424
|
|
|
677
|
-
|
|
678
|
-
|
|
679
|
-
The agent spawns its execution context lazily on the first run (`execution.spawn()`), which is fine for a single invocation — but that spawn is NOT a journaled side effect, so on a crash mid-run the replay would mint a *fresh* sandbox/container and orphan the first. For a context the host owns across invocations, drive the spawn yourself (so it's journaled) and hand the agent the handle:
|
|
425
|
+
**Re-attach the execution context across invocations.** `execution.spawn()` isn't journaled, so a crash mid-run would replay into a *fresh* sandbox on retry. Drive the spawn yourself and hand the agent the handle:
|
|
680
426
|
|
|
681
427
|
```ts
|
|
682
|
-
import { createSandboxContext } from 'zidane/contexts/sandbox'
|
|
683
|
-
import { createDaytonaProvider } from 'zidane/contexts/daytona'
|
|
684
|
-
|
|
685
|
-
// Journal the spawn → replay returns the same sandbox id, no re-mint.
|
|
686
428
|
const handle = await ctx.run('spawn-sandbox', () => execution.spawn())
|
|
687
|
-
|
|
688
|
-
const agent = createAgent({
|
|
689
|
-
provider: restateProvider(inner, ctx),
|
|
690
|
-
execution,
|
|
691
|
-
handle, // used verbatim — agent does NOT spawn, and does NOT destroy it
|
|
692
|
-
clock: restateClock(ctx),
|
|
693
|
-
behavior: restateBehavior(),
|
|
694
|
-
})
|
|
429
|
+
createAgent({ provider: restateProvider(inner, ctx), execution, handle, clock: restateClock(ctx), behavior: restateBehavior() })
|
|
695
430
|
```
|
|
696
431
|
|
|
697
|
-
|
|
698
|
-
- **`execution:ready` hook** — if you'd rather let the agent spawn and just *observe* the minted id (to journal it for the next invocation's re-attach), this fires once with `{ handle, spawned }`.
|
|
699
|
-
- **`createSandboxContext`** re-attaches by handle id: its handle id IS the sandbox id, so a handle from a prior invocation routes operations to the same sandbox even though this instance never ran `spawn()`. Pair with the provider's reconnect option (`createDaytonaProvider({ sandboxId })`).
|
|
700
|
-
- **`createDaytonaProvider({ ownAttached: true })`** — by default an attached (`sandboxId`) sandbox is treated as externally owned and left running on `destroy()`. Set `ownAttached` when the *host* owns it across invocations so `destroy()` applies `onDestroy` (stop / archive / delete). Spawn-time failures still never delete a pre-existing attached sandbox.
|
|
701
|
-
- **`zidane/contexts/sandbox` subpath** — import `createSandboxContext` from here (not the `zidane/contexts` barrel) on workerd/edge: this entry's graph is node-free, so it doesn't drag `createProcessContext` → `node:child_process` into the bundle.
|
|
702
|
-
- **`@daytona/sdk` needs a root `tslib` override.** It is a peer dependency, so *you* install it — and it declares no `tslib` dependency of its own while reaching AWS SDK v3 through `@aws-sdk/client-s3`. A stale `@aws-crypto` v3/v4 floor (`tslib: ^1.11.1`) can win the hoist, and the AWS SDK's initializer helpers then resolve against tslib 1.x, which lacks them — `import('@daytona/sdk')` fails with `Export named '__runInitializers' not found`. npm/bun overrides are **root-only** and are not inherited from zidane, so add this to your own `package.json`:
|
|
703
|
-
|
|
704
|
-
```json
|
|
705
|
-
{ "overrides": { "tslib": "^2.8.1" } }
|
|
706
|
-
```
|
|
432
|
+
`AgentOptions.handle` skips the agent's internal `spawn()` and its `destroy()` on teardown (the host owns that lifecycle) — see `docs/SKILL.md` § `execution:ready` for observing a freshly-minted handle to journal for next time. `createSandboxContext`'s handle id IS the sandbox id, so a prior handle re-attaches by id; pair with a reconnect option like `createDaytonaProvider({ sandboxId, ownAttached: true })`. Import `createSandboxContext` from the `zidane/contexts/sandbox` subpath (not the barrel) on workerd/edge to keep `node:child_process` out of the bundle.
|
|
707
433
|
|
|
708
|
-
|
|
709
|
-
|
|
710
|
-
### Streaming on fresh vs resumed invocations
|
|
711
|
-
|
|
712
|
-
`createReplayTracker().isReplaying()` is conservatively `true` until the first journaled closure executes — correct for deferred abort, but on a *fresh* run it would briefly suppress opening output for a streamer that gates on it. Pass `resuming` (the host knows whether this invocation is a crash-recovery replay — e.g. the bound object already has persisted turns) and gate live streaming on `isResuming()` instead: `false` from the start of a fresh run, `true` only while replaying a resumed journal.
|
|
434
|
+
**Streaming on fresh vs. resumed invocations.** `createReplayTracker().isReplaying()` is conservatively `true` until the first journaled closure executes, which would briefly suppress opening output on a *fresh* run. Pass `resuming` (the host knows whether this invocation is a crash-recovery replay) and gate live streaming on `isResuming()` instead:
|
|
713
435
|
|
|
714
436
|
```ts
|
|
715
437
|
const tracker = createReplayTracker({ hooks: agent.hooks, resuming: hasPriorTurns })
|
|
716
438
|
// stream:text consumer: emit when `!tracker.isResuming()`
|
|
717
439
|
```
|
|
718
440
|
|
|
719
|
-
###
|
|
441
|
+
### Replay-safe abort
|
|
720
442
|
|
|
721
|
-
|
|
443
|
+
`agent.abort()` during a journal replay is undefined behavior — it throws `AgentAbortedError` at whatever point the replay cursor is, and the host's catch arm typically journals something mid-replay, permanently mismatching the journal on every subsequent retry. `replayAwareAbort(tracker, () => agent.abort())` defers an external abort until the first live (non-journaled) execution — the earliest point where the resulting throw appends to the journal instead of forking it:
|
|
722
444
|
|
|
723
445
|
```ts
|
|
724
|
-
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
|
|
446
|
+
const tracker = createReplayTracker({ hooks: agent.hooks })
|
|
447
|
+
const provider = restateProvider(inner, ctx, { replayTracker: tracker })
|
|
448
|
+
const tools = wrapAgentTools(agentTools, ctx, { replayTracker: tracker })
|
|
449
|
+
const abort = replayAwareAbort(tracker, () => agent.abort()) // external cancel signal → abort(), not agent.abort()
|
|
728
450
|
```
|
|
729
451
|
|
|
730
|
-
|
|
731
|
-
|
|
732
|
-
Large tool-output elision is NOT local-disk-bound: with `behavior.persistDir` set, the loop routes the blob through `ctx.execution.writeFile` (and the model reads it back via `read_file` → `ctx.execution.readFile`), so on a remote/sandbox context the blob lands on that context's filesystem — workerd-safe, no host disk required. Point `persistDir` at a path valid inside the execution context. For media kept *out* of the journal, persist `{ type: 'image' | 'document', ref }` blocks and resolve them at wire time via the **real** `content-ref:resolve` hook (see [Content refs](#content-refs)). There is no separate R2/KV blob backend distinct from the execution filesystem — if you need one, resolve refs through it in the hook.
|
|
452
|
+
**Auth on long/suspended runs.** A durable run can outlive a short-lived provider token — pass `extraHeaders` as a function (e.g. `anthropic({ extraHeaders: async () => ({ authorization: \`Bearer ${await mintWorkToken()}\` }) })`) so each turn, including replay-time live calls, carries a freshly-minted token.
|
|
733
453
|
|
|
734
|
-
|
|
454
|
+
**Large tool outputs & media.** With `behavior.persistDir` set, large tool-output elision routes through `ctx.execution.writeFile`/`readFile` (not local disk), so it works on a remote/sandbox context. For media kept out of the journal, persist `{ type, ref }` blocks and resolve them at wire time via `content-ref:resolve` (see [Content refs](#restateprovider)).
|
|
735
455
|
|
|
736
|
-
|
|
456
|
+
**Unbounded runs on CPU-capped runtimes.** A hard per-invocation CPU/wall cap (e.g. Cloudflare's ~5-minute limit) can't host an arbitrarily long continuous run: cap the invocation with `behavior.maxTurns`/`maxWallMs`, persist via `restateSessionStore`, and when a run returns unfinished, schedule a fresh invocation of the same object key to resume from the persisted turns (Restate's delayed-tail-call pattern) — the session store is the checkpoint.
|
|
737
457
|
|
|
738
458
|
## Failure modes
|
|
739
459
|
|
|
740
460
|
| Scenario | What happens |
|
|
741
461
|
|---|---|
|
|
742
462
|
| Crash between turn N and N+1 | Re-enter the handler. Journal replays turns 1..N from cache. Turn N+1 runs live. |
|
|
743
|
-
| Crash mid-tool-call | Re-enter.
|
|
744
|
-
| Provider rate limit / 5xx | `restateProvider`
|
|
745
|
-
| Tool throws | `restateTool`'s default `maxRetryAttempts: 1` — no blind retry. The error
|
|
746
|
-
| Restate runtime down | Handler invocations queue
|
|
463
|
+
| Crash mid-tool-call | Re-enter. Prior turns + completed tools replay from cache. The in-flight tool re-executes (Restate never saw it complete) — needs application-level idempotency for non-idempotent side effects. |
|
|
464
|
+
| Provider rate limit / 5xx | `restateProvider`'s `runOptions` retries the durable call; the loop's own retry can retry beyond that, each with its own durable boundary. Past all retries: `AgentProviderError`. |
|
|
465
|
+
| Tool throws | `restateTool`'s default `maxRetryAttempts: 1` — no blind retry. The error journals; replay returns the same error. |
|
|
466
|
+
| Restate runtime down | Handler invocations queue/fail at the SDK transport layer; Zidane never sees them — recovery is operational, not in-app. |
|
|
747
467
|
|
|
748
468
|
## Tests
|
|
749
469
|
|
|
750
|
-
`test/restate.test.ts`
|
|
751
|
-
|
|
752
|
-
`test/
|
|
753
|
-
|
|
754
|
-
`bun run restate:fixture:test` is the end-to-end suite against a **real** Restate server in Docker, and is deliberately **not** part of CI — run it locally when you touch the adapter. It boots the container, supervises the fixture service, and drives nineteen scenarios through the real ingress.
|
|
755
|
-
|
|
756
|
-
Five cover the adapter itself: baseline agent run; crash → replay (the service process is killed mid-invocation and respawned, so the journal, not a mock, decides what re-runs); per-object-key state isolation; commit-after-soft-failure; and an awakeable suspend/resume.
|
|
757
|
-
|
|
758
|
-
Five cover **subagents**, each pinned on an on-disk ledger that has to outlive the process: a crash inside the child; an awakeable park inside the child (suspend → external resume, with the child's `register` phase required to fire exactly once, or the resumed awakeable id would be one nobody resolves); a depth-2 grandchild crash, two un-journaled spawn frames deep; a journaled-deadline trip; and an attempt that fails **without** a process death, so the journal replays against warm module state — that one asserts an unchanged boot count to prove it took the retry path and not the crash path.
|
|
759
|
-
|
|
760
|
-
Eight cover **parallel** subagents (`restateSpawnTool`). Two children as separate invocations, asserted to have *overlapping* work intervals — counting executions can't tell parallel from sequential. A crash inside one of them, where the surviving sibling must show exactly one execution: proof that the children really are journal-isolated. Then the scheduling pair: four children in one turn against a fleet cap of two, where the interval ledger must show a maximum of exactly **two** open at once (3+ means the cap is ignored, 1 means the fleet silently fell back to sequential), and the same fan-out at cap 1 as a control, where nothing may overlap. Measured: 1031ms wall at cap 2 versus 2090ms at cap 1 for identical work.
|
|
761
|
-
|
|
762
|
-
The remaining three cover the failure modes that decide whether this is production-safe. **A parent crash mid-fan-out**: the parent dies while both children are still working, and on replay must re-attach to the running invocations rather than dispatch new ones — exactly one execution per child, even though the parent frame ran twice. **Depth-2 fan-out**: a child that itself spawns two grandchildren, whose intervals must overlap, so parallelism survives nesting. **Cancellation**, twice: once from outside (the parent invocation is cancelled mid-flight and no child may start another work step) and once from inside (`agent.abort()` with `onAbort` wired, where children finish the step in flight and never begin the next). **A post-merge replay**: the fan-out completes and merges, then the handler fails once, so the parent replays *after* the merge — the only scenario that re-executes the merge path, and the one that caught wall-clock timestamps leaking into the merged run records.
|
|
763
|
-
|
|
764
|
-
Both subagent bugs found in review — the `maxWallMs` watchdog clamp and the live `elapsed` inside a persisted string — were caught here and nowhere else. The suite fails if the crash never forced a respawn, so it cannot pass without actually exercising replay. This is the only coverage where replay, suspension, and virtual-object state semantics come from the SDK rather than from our reading of it.
|
|
765
|
-
|
|
766
|
-
`test/start.test.ts` covers the local `bun start --restate` runner helpers: option parsing, deployment registration, ingress invocation timeout wiring, readiness/service probing, runtime-state cleanup, cached completion, changed-input rejection, and `status` / `history` handlers.
|
|
470
|
+
- `test/restate.test.ts` — every wrapper in isolation plus end-to-end `agent.run()`, against a fake `RestateContextLike` that records `ctx.run` calls and can be pre-seeded with a journal to assert replay semantics (inner fn not invoked, cached value returned).
|
|
471
|
+
- `test/tool-state.test.ts` — the runtime-agnostic half of durable tool state: hydration, staging, commit-on-settle, the size guard, the commit-failure hook, the in-memory default.
|
|
472
|
+
- `test/start.test.ts` — the local `bun start --restate` runner: option parsing, deployment registration, timeout wiring, readiness probing, runtime-state cleanup, cached completion, changed-input rejection, `status`/`history`.
|
|
473
|
+
- `bun run restate:fixture:test` — end-to-end suite against a **real** Restate server in Docker (`test/fixtures/restate-docker`). Deliberately not part of CI; run it locally when touching the adapter. Covers the adapter itself (crash → replay, per-key isolation, soft-failure commit, awakeable suspend/resume), sequential and parallel subagents (nested fan-out, crash isolation, fleet-cap concurrency), and cancellation (external invocation cancel, in-agent `abort()` with `onAbort`).
|
|
767
474
|
|
|
768
|
-
The
|
|
475
|
+
The structural types the adapter typechecks against live in `src/restate/types.ts` — `zidane/restate` compiles **without** `@restatedev/restate-sdk` installed; the real SDK's `Context`/`ObjectContext` are structurally assignable to them.
|
|
769
476
|
|
|
770
477
|
## Related
|
|
771
478
|
|
|
772
479
|
- `examples/restate/agent.ts` — runnable end-to-end virtual object.
|
|
773
480
|
- `docs/ARCHITECTURE.md` — agent lifecycle, hook ordering, run-level budget breaker.
|
|
774
|
-
- `docs/SKILL.md` — `createAgent`, `Provider`, `ToolDef`, `SessionStore
|
|
481
|
+
- `docs/SKILL.md` — `createAgent`, `Provider`, `ToolDef`, `SessionStore`, hook table (`execution:ready`, etc.).
|
|
482
|
+
- `docs/RUN_IN_BACKGROUND.md` — background-task execution model this adapter journals.
|
|
775
483
|
- [Restate docs](https://docs.restate.dev/) — runtime, SDK, deployment model.
|