eve 0.89.0 → 0.91.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (247) hide show
  1. package/CHANGELOG.md +39 -9
  2. package/dist/src/channel/adapter.d.ts +12 -3
  3. package/dist/src/channel/adapter.js +1 -1
  4. package/dist/src/channel/auth/definitions.d.ts +7 -20
  5. package/dist/src/channel/auth/definitions.js +1 -1
  6. package/dist/src/channel/auth/http-basic.js +1 -1
  7. package/dist/src/channel/auth/jwt-ecdsa.js +1 -1
  8. package/dist/src/channel/auth/jwt-hmac.js +1 -1
  9. package/dist/src/channel/auth/oidc.js +1 -1
  10. package/dist/src/channel/auth/result.d.ts +2 -3
  11. package/dist/src/channel/auth/result.js +1 -1
  12. package/dist/src/channel/channel-address.d.ts +1 -1
  13. package/dist/src/channel/channel-address.js +1 -1
  14. package/dist/src/channel/channel-operations.d.ts +1 -1
  15. package/dist/src/channel/schedule.js +1 -1
  16. package/dist/src/channel/types.d.ts +2 -0
  17. package/dist/src/chunks/{use-eve-agent-Bbl4H5aA.js → use-eve-agent-Bwloorf8.js} +77 -63
  18. package/dist/src/chunks/{use-eve-agent-CNpxgKEH.js → use-eve-agent-XCB0-tLX.js} +77 -63
  19. package/dist/src/cli/dev/environment.js +1 -1
  20. package/dist/src/cli/dev/tui/prompt-commands.d.ts +1 -1
  21. package/dist/src/cli/dev/tui/remote-connection-probe.js +1 -1
  22. package/dist/src/cli/dev/tui/remote-connection-types.d.ts +1 -0
  23. package/dist/src/cli/dev/tui/runner.js +2 -2
  24. package/dist/src/cli/dev/tui/status-line.d.ts +2 -2
  25. package/dist/src/cli/dev/tui/status-line.js +1 -1
  26. package/dist/src/client/agent-info-schema.d.ts +341 -326
  27. package/dist/src/client/agent-info-schema.js +1 -1
  28. package/dist/src/client/agent-info-types.d.ts +13 -10
  29. package/dist/src/client/client-error.d.ts +1 -1
  30. package/dist/src/client/client.d.ts +5 -3
  31. package/dist/src/client/client.js +1 -1
  32. package/dist/src/client/session.d.ts +1 -1
  33. package/dist/src/compiled/.vendor-stamp.json +1 -1
  34. package/dist/src/evals/target.js +1 -1
  35. package/dist/src/eve-channel/auth.d.ts +19 -0
  36. package/dist/src/eve-channel/auth.js +1 -0
  37. package/dist/src/eve-channel/index.d.ts +1 -0
  38. package/dist/src/eve-channel/index.js +1 -1
  39. package/dist/src/eve-channel/types.d.ts +23 -4
  40. package/dist/src/execution/catalog/connection-entry.js +1 -1
  41. package/dist/src/execution/coordination-dispatch-shared.js +1 -1
  42. package/dist/src/execution/dynamic-workflow/tool.js +1 -1
  43. package/dist/src/execution/proxied-deliver-step.js +1 -1
  44. package/dist/src/execution/sandbox/bash.js +2 -2
  45. package/dist/src/execution/session/active-turn.js +1 -1
  46. package/dist/src/execution/session/admission.d.ts +6 -0
  47. package/dist/src/execution/session/admission.js +1 -1
  48. package/dist/src/execution/session/delivery-facts.js +1 -1
  49. package/dist/src/execution/session/next-input.d.ts +2 -2
  50. package/dist/src/execution/session/next-input.js +1 -1
  51. package/dist/src/execution/session/program.js +1 -1
  52. package/dist/src/execution/session-inbox/inbox.d.ts +2 -2
  53. package/dist/src/execution/session-inbox/inbox.js +1 -1
  54. package/dist/src/execution/session-inbox/protocol.js +1 -1
  55. package/dist/src/execution/tools/agent-router-workflow.js +1 -1
  56. package/dist/src/harness/compaction/engine.js +2 -2
  57. package/dist/src/harness/current-messages.js +1 -1
  58. package/dist/src/harness/emission.d.ts +3 -0
  59. package/dist/src/harness/emission.js +1 -1
  60. package/dist/src/harness/facts.d.ts +5 -2
  61. package/dist/src/harness/facts.js +1 -1
  62. package/dist/src/harness/model-call/call.d.ts +7 -1
  63. package/dist/src/harness/model-call/call.js +3 -1
  64. package/dist/src/harness/provider-tool-history.d.ts +8 -0
  65. package/dist/src/harness/provider-tool-history.js +1 -1
  66. package/dist/src/harness/session-machine/transitions.d.ts +3 -2
  67. package/dist/src/harness/step/after-model.js +1 -1
  68. package/dist/src/harness/step/context.js +1 -1
  69. package/dist/src/harness/step/intake.js +1 -1
  70. package/dist/src/harness/step/prompt.js +1 -1
  71. package/dist/src/harness/tool-history-repair.d.ts +2 -0
  72. package/dist/src/harness/tool-history-repair.js +1 -1
  73. package/dist/src/harness/tool-loop.js +1 -1
  74. package/dist/src/internal/application/stamped-package-version.js +1 -1
  75. package/dist/src/internal/authored-module-graph.js +1 -1
  76. package/dist/src/internal/nitro/routes/agent-info/build-agent-info-response.d.ts +2 -2
  77. package/dist/src/protocol/routes.d.ts +7 -1
  78. package/dist/src/protocol/routes.js +1 -1
  79. package/dist/src/protocol/session-events/families/delivery.d.ts +5 -3
  80. package/dist/src/protocol/session-events/families/delivery.js +1 -1
  81. package/dist/src/protocol/session-projection/fold.js +1 -1
  82. package/dist/src/public/channels/discord/hitl.js +1 -1
  83. package/dist/src/public/channels/github/githubChannel.js +1 -1
  84. package/dist/src/public/channels/github/state.d.ts +7 -0
  85. package/dist/src/public/channels/github/state.js +1 -1
  86. package/dist/src/public/channels/mcp.js +1 -1
  87. package/dist/src/public/channels/slack/hitl.d.ts +2 -1
  88. package/dist/src/public/channels/slack/hitl.js +3 -3
  89. package/dist/src/public/channels/slack/interaction-cards.js +1 -1
  90. package/dist/src/public/channels/slack/request-card-render.js +1 -1
  91. package/dist/src/public/channels/teams/hitl.js +1 -1
  92. package/dist/src/public/channels/teams/verify.js +1 -1
  93. package/dist/src/public/channels/telegram/telegramChannel.js +1 -1
  94. package/dist/src/public/connections/index.d.ts +1 -1
  95. package/dist/src/public/definitions/channel.js +1 -1
  96. package/dist/src/public/definitions/connections/tool-call.d.ts +2 -2
  97. package/dist/src/public/definitions/hook.d.ts +90 -7
  98. package/dist/src/public/definitions/hook.js +1 -1
  99. package/dist/src/public/hooks/index.d.ts +3 -2
  100. package/dist/src/reactions/continuation.d.ts +19 -0
  101. package/dist/src/reactions/continuation.js +1 -0
  102. package/dist/src/reactions/frame.js +1 -1
  103. package/dist/src/reactions/intents.d.ts +10 -4
  104. package/dist/src/reactions/intents.js +1 -1
  105. package/dist/src/reactions/kinds/hook.js +1 -1
  106. package/dist/src/reactions/runner.js +1 -1
  107. package/dist/src/reactions/state.d.ts +15 -2
  108. package/dist/src/reactions/state.js +1 -1
  109. package/dist/src/runtime/channels/registry.d.ts +2 -2
  110. package/dist/src/runtime/channels/registry.js +1 -1
  111. package/dist/src/runtime/connections/approval-scope.d.ts +9 -0
  112. package/dist/src/runtime/connections/approval-scope.js +1 -0
  113. package/dist/src/runtime/connections/mcp-client.d.ts +1 -1
  114. package/dist/src/runtime/connections/mcp-client.js +1 -1
  115. package/dist/src/runtime/connections/provided-arguments.js +1 -1
  116. package/dist/src/runtime/resolve-connection.js +1 -1
  117. package/dist/src/runtime/subagents/workflow.js +1 -1
  118. package/dist/src/setup/primitives/open-url.js +1 -1
  119. package/dist/src/setup/scaffold/create/project.js +1 -1
  120. package/dist/src/shared/agent-info-access.d.ts +5 -0
  121. package/dist/src/shared/agent-info-access.js +1 -0
  122. package/dist/src/shared/connection-types.d.ts +21 -4
  123. package/dist/src/shared/value-delta.d.ts +5 -4
  124. package/dist/src/shared/value-delta.js +1 -1
  125. package/dist/src/svelte/index.js +1 -1
  126. package/dist/src/svelte/use-eve-agent.js +1 -1
  127. package/dist/src/tools/provided/ask-question-workflow.js +1 -1
  128. package/dist/src/tools/provided/sleep-workflow.js +1 -1
  129. package/dist/src/tools/provided/web-fetch.js +1 -1
  130. package/dist/src/vue/index.js +1 -1
  131. package/dist/src/vue/use-eve-agent.js +1 -1
  132. package/docs/agent-config.md +199 -222
  133. package/docs/channels/chat-sdk.mdx +60 -64
  134. package/docs/channels/custom.mdx +265 -280
  135. package/docs/channels/discord.mdx +176 -123
  136. package/docs/channels/eve.mdx +136 -204
  137. package/docs/channels/github.mdx +24 -22
  138. package/docs/channels/linear.mdx +30 -30
  139. package/docs/channels/linq.mdx +25 -13
  140. package/docs/channels/mcp.mdx +154 -116
  141. package/docs/channels/overview.mdx +31 -32
  142. package/docs/channels/photon.mdx +32 -56
  143. package/docs/channels/slack.mdx +538 -375
  144. package/docs/channels/teams.mdx +40 -26
  145. package/docs/channels/telegram.mdx +13 -13
  146. package/docs/channels/twilio.mdx +11 -11
  147. package/docs/code-extension.mdx +40 -42
  148. package/docs/concepts/built-in-tools.md +89 -57
  149. package/docs/concepts/context-control.md +20 -20
  150. package/docs/concepts/default-harness.md +20 -36
  151. package/docs/concepts/execution-model-and-durability.mdx +96 -160
  152. package/docs/concepts/project-structure.mdx +21 -14
  153. package/docs/concepts/security-model.md +27 -27
  154. package/docs/concepts/sessions-runs-and-streaming.md +184 -194
  155. package/docs/concepts/state.md +7 -15
  156. package/docs/connections/mcp.mdx +109 -93
  157. package/docs/connections/openapi.mdx +57 -59
  158. package/docs/connections/overview.mdx +205 -170
  159. package/docs/evals/assertions.mdx +36 -29
  160. package/docs/evals/cases.mdx +38 -42
  161. package/docs/evals/judge.mdx +21 -29
  162. package/docs/evals/overview.mdx +42 -38
  163. package/docs/evals/reporters.mdx +54 -27
  164. package/docs/evals/running.mdx +15 -8
  165. package/docs/evals/targets.mdx +17 -18
  166. package/docs/evals/tool-stubs.mdx +111 -97
  167. package/docs/experimental/dynamic-schedules.mdx +118 -63
  168. package/docs/extensions.md +58 -65
  169. package/docs/getting-started.mdx +4 -6
  170. package/docs/guides/auth-and-route-protection.md +100 -70
  171. package/docs/guides/client/continuations.mdx +23 -26
  172. package/docs/guides/client/messages.mdx +47 -66
  173. package/docs/guides/client/output-schema.mdx +8 -8
  174. package/docs/guides/client/overview.mdx +47 -41
  175. package/docs/guides/client/streaming.mdx +62 -71
  176. package/docs/guides/decide.md +24 -79
  177. package/docs/guides/deployment/meta.json +1 -1
  178. package/docs/guides/deployment/overview.md +27 -18
  179. package/docs/guides/deployment/self-hosted-operations.md +99 -0
  180. package/docs/guides/deployment/self-hosting.md +63 -80
  181. package/docs/guides/deployment/vercel.mdx +31 -27
  182. package/docs/guides/deployment/workflow-worlds.md +100 -0
  183. package/docs/guides/dev-tui.md +48 -41
  184. package/docs/guides/dynamic-capabilities.md +49 -50
  185. package/docs/guides/frontend/nextjs.mdx +2 -2
  186. package/docs/guides/frontend/nuxt.mdx +11 -11
  187. package/docs/guides/frontend/overview.mdx +1 -1
  188. package/docs/guides/frontend/sveltekit.mdx +11 -11
  189. package/docs/guides/frontend/tanstack.mdx +16 -16
  190. package/docs/guides/frontend/use-eve-agent-svelte.mdx +21 -21
  191. package/docs/guides/frontend/use-eve-agent-vue.mdx +22 -22
  192. package/docs/guides/frontend/use-eve-agent.mdx +25 -25
  193. package/docs/guides/hooks.md +152 -106
  194. package/docs/guides/instrumentation/agent-runs.mdx +26 -30
  195. package/docs/guides/instrumentation/instrumentation.mdx +49 -76
  196. package/docs/guides/instrumentation/meta.json +1 -1
  197. package/docs/guides/instrumentation/otel.mdx +113 -179
  198. package/docs/guides/prepare-for-1-0.md +23 -1
  199. package/docs/guides/remote-agents.md +125 -187
  200. package/docs/guides/self-modification.md +18 -18
  201. package/docs/guides/session-context.md +16 -16
  202. package/docs/install-integrations.mdx +26 -24
  203. package/docs/instructions.mdx +37 -42
  204. package/docs/memory/custom-provider.md +35 -41
  205. package/docs/memory/file.md +93 -76
  206. package/docs/memory/overview.mdx +145 -156
  207. package/docs/patterns/durable-cross-channel-notifications.md +15 -19
  208. package/docs/patterns/dynamic-scheduling.md +19 -23
  209. package/docs/patterns/multi-tenant-approvals.md +17 -22
  210. package/docs/patterns/multi-tenant-auth.md +43 -42
  211. package/docs/patterns/multi-tenant-memory.md +39 -36
  212. package/docs/patterns/voice-notes.md +3 -3
  213. package/docs/protocols/acp.md +13 -13
  214. package/docs/protocols/ucp.mdx +7 -7
  215. package/docs/reference/agent-files.md +6 -6
  216. package/docs/reference/cli.md +99 -98
  217. package/docs/reference/telemetry.md +1 -1
  218. package/docs/reference/typescript-api.md +52 -53
  219. package/docs/responsible-use.md +2 -2
  220. package/docs/sandbox/custom-providers.mdx +60 -56
  221. package/docs/sandbox/default.mdx +23 -24
  222. package/docs/sandbox/docker.mdx +16 -10
  223. package/docs/sandbox/index.mdx +93 -61
  224. package/docs/sandbox/just-bash.mdx +17 -11
  225. package/docs/sandbox/troubleshooting.mdx +29 -23
  226. package/docs/sandbox/vercel.mdx +45 -52
  227. package/docs/schedules.mdx +65 -54
  228. package/docs/skills.mdx +74 -49
  229. package/docs/subagents/index.mdx +97 -136
  230. package/docs/tools/human-in-the-loop.md +160 -90
  231. package/docs/tools/meta.json +1 -1
  232. package/docs/tools/overview.mdx +229 -237
  233. package/docs/tools/tasks.md +118 -117
  234. package/docs/tools/workflows.mdx +447 -543
  235. package/docs/tutorial/connect-a-warehouse.mdx +11 -13
  236. package/docs/tutorial/first-agent.mdx +14 -16
  237. package/docs/tutorial/guard-the-spend.mdx +6 -8
  238. package/docs/tutorial/how-it-runs.mdx +8 -8
  239. package/docs/tutorial/query-sample-data.mdx +8 -8
  240. package/docs/tutorial/remember-definitions.mdx +4 -4
  241. package/docs/tutorial/run-analysis.mdx +20 -22
  242. package/docs/tutorial/ship-it.mdx +19 -19
  243. package/docs/tutorial/team-playbooks.mdx +9 -9
  244. package/docs/use-cases.mdx +13 -11
  245. package/package.json +1 -1
  246. package/docs/guides/instrumentation/migration.md +0 -96
  247. package/docs/tools/tasks-upgrade.md +0 -218
@@ -3,9 +3,9 @@ title: "Tool Stubs"
3
3
  description: "Define tool outcomes and outcome sequences in your evals."
4
4
  ---
5
5
 
6
- Pass `stubs` when creating an eval session to replace selected tool executions with JSON responses. The same rules work against local and deployed agents, without deploying mock functions or keeping the runner connected to answer tool calls.
6
+ A tool stub makes a tool return fixed JSON, or throw an error, for one eval session. Pass `stubs` to `t.session()`. Stubs work against local and deployed agents, with no mock code to deploy.
7
7
 
8
- Stubs specify what tools return. [Assertions](./assertions) check what the agent does and tells the user. Unused stubs produce a warning without failing the eval.
8
+ Stubs set what tools return; [assertions](./assertions) check what the agent does with it. The target must [permit tool replacement](#permit-tool-replacement) for the eval's caller.
9
9
 
10
10
  ## Author an eval
11
11
 
@@ -65,21 +65,50 @@ export default defineEval({
65
65
 
66
66
  Use your agent's actual tool names and response shapes. The client accepts the same option on `client.sessions.create({ stubs })` or `client.sessions.create({ message, stubs })`.
67
67
 
68
- These responses describe a sequence, not a simulated task database. The second `list_tasks` call returns two tasks even if `complete_task` was never called. The completion and order assertions check that the agent actually requested the write; testing a real database update requires the real tool.
68
+ Responses are a fixed sequence, not a simulated database: the second `list_tasks` call returns two tasks even if `complete_task` never ran. The assertions check only that the agent asked for the write. To test a real update, use the real tool.
69
69
 
70
- ## Stub fields
70
+ ## Permit tool replacement
71
71
 
72
- | Field | Meaning |
73
- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------- |
74
- | `id` | Your session-unique rule identifier. eve uses it to track this rule's response sequence. It is neither a tool name nor a tool-call ID. |
75
- | `tool` | The existing tool's name, optionally qualified by its local subagent path. |
76
- | `match` | Optional input-property constraints, expressed as JSON Schema. Omit it or use `{}` to match any valid input. |
77
- | `outcome` | One outcome repeated on every matching call, such as `{ response: { tasks: [] } }`. |
78
- | `outcomes` | A nonempty array of outcomes returned in sequence. Use exactly one of `outcome` or `outcomes`. |
72
+ eve rejects stubs by default, including in development. With [Vercel OIDC](../guides/auth-and-route-protection#verceloidc), use `eveVercelOidc` and grant the permission to the eval runner's verified subject:
73
+
74
+ ```ts title="agent/channels/eve.ts"
75
+ import { vercelSubject } from "eve/channels/auth";
76
+ import { eveChannel, eveVercelOidc } from "eve/channels/eve";
77
+
78
+ export default eveChannel({
79
+ auth: [
80
+ eveVercelOidc({
81
+ subjects: [
82
+ vercelSubject({ teamSlug: "acme", projectName: "app" }),
83
+ {
84
+ subject: vercelSubject({ teamSlug: "acme", projectName: "eval-runner" }),
85
+ allowToolStubs: true,
86
+ },
87
+ ],
88
+ }),
89
+ ],
90
+ });
91
+ ```
92
+
93
+ Both projects can authenticate, but only the eval runner can replace tools. `vercelSubject` matches production only; pass `environment: "*"` to include all environments. Only the target's server configuration grants the permission, after token verification. A request can't grant it, and users don't inherit a project's grant.
94
+
95
+ Custom authenticators can return the same `allowToolStubs: true` after verifying the caller. See [authentication and route protection](../guides/auth-and-route-protection#tool-replacement-permission) for the grant rules and a local development example.
96
+
97
+ eve checks the permission only when it creates the session. Later requests use normal channel authentication, so anyone your channel admits to the session can use its stubs. Isolate sessions with your session-access policy, or use a separate eval deployment of the same agent code.
98
+
99
+ Checking the stubs after the eval also requires `allowToolStubs`.
100
+
101
+ ## Stub fields
79
102
 
80
- Each outcome contains exactly one of `response` or `throw`. `response` holds the JSON returned by the tool; `throw` describes an error raised immediately instead. The enclosing outcome is not part of the tool's response. Top-level `response` and `responses` are not accepted.
103
+ | Field | Meaning |
104
+ | ---------- | --------------------------------------------------------------------------------------------------------------- |
105
+ | `id` | A rule name, unique in the session, that tracks the rule's place in its sequence. Traces don't show it. |
106
+ | `tool` | The existing tool's name, optionally prefixed with a local subagent path. |
107
+ | `match` | Optional constraints on input properties, written as JSON Schema. Omit it or use `{}` to match any valid input. |
108
+ | `outcome` | One outcome returned on every matching call, such as `{ response: { tasks: [] } }`. |
109
+ | `outcomes` | A nonempty array of outcomes returned in order. Use exactly one of `outcome` or `outcomes`. |
81
110
 
82
- Exposing rule IDs and response positions in traces is a follow-up: [AX-5157](https://linear.app/vercel/issue/AX-5157).
111
+ Each outcome contains exactly one of `response`, the JSON the tool returns, or `throw`, an error [raised instead](#inject-tool-errors). The tool returns only the `response` value, not the outcome object.
83
112
 
84
113
  Stubs are plain JSON and can be shared between evals:
85
114
 
@@ -91,11 +120,13 @@ export const taskStubs = [
91
120
  ] satisfies readonly ToolStub[];
92
121
  ```
93
122
 
94
- Pass the fixture to `t.session({ stubs: taskStubs })`. Each session has its own sequence position. Both `eve/evals` and `eve/client` export `ToolStub` and `ToolStubOutcome`. These types check the JSON contract; they do not infer the response type from your tool definition.
123
+ Pass the fixture to `t.session({ stubs: taskStubs })`; each session keeps its own place in each sequence. Both `eve/evals` and `eve/client` export `ToolStub` and `ToolStubOutcome`. These types check the JSON shape but don't infer the response type from your tool definition.
124
+
125
+ Configuration is limited to 100 rules, 20,000 JSON nodes, nesting depth 32, and about one million characters. The `__proto__` property name is reserved.
95
126
 
96
127
  ## Match tool arguments
97
128
 
98
- **Rules are evaluated from top to bottom. The first matching stub wins.** Put specific rules before an unconditional fallback:
129
+ **eve checks rules from top to bottom. The first matching stub wins.** Put specific rules before an unconditional fallback:
99
130
 
100
131
  ```ts
101
132
  const stubs = [
@@ -109,9 +140,7 @@ const stubs = [
109
140
  ];
110
141
  ```
111
142
 
112
- A call for milk receives success; other valid calls receive the fallback. Reversing these rules makes the unconditional rule win every time. If no rule matches, eve executes the real tool. A broken stub or failure processing its response fails the eval, even if the agent recovers. An intentional `throw` outcome is a normal tool failure that the eval can assert on. Neither falls back to real execution.
113
-
114
- If eve cannot record an output-processing failure, it cancels the workflow serving stubs. The original tool error or fallback is preserved, but verification rejects the eval. Failed or cancelled stub sessions cannot pass verification.
143
+ A call for milk receives success; other valid calls receive the fallback. Reversed, the unconditional rule wins every time. If no rule matches, eve runs the real tool.
115
144
 
116
145
  Every property listed in `match` must exist and satisfy its constraint. Extra top-level properties are allowed:
117
146
 
@@ -132,70 +161,84 @@ Every property listed in `match` must exist and satisfy its constraint. Extra to
132
161
  }
133
162
  ```
134
163
 
135
- Nested properties are optional unless listed in `required`. Matching does not coerce values, apply defaults, or change arguments. The real tool schema validates input before replacement; normal approvals and output processing still apply.
164
+ Nested properties are optional unless listed in `required`. Matching doesn't coerce values, apply defaults, or change arguments. The real tool's input schema still validates the input first, and normal approvals and output processing still apply.
136
165
 
137
- Constraints use a reference-free subset of JSON Schema 2020-12: types, scalar `const` and `enum`, string patterns and bounds, numeric bounds, array items/contains, object properties/required/additionalProperties, and logical/conditional schemas. `pattern` uses JavaScript regular expressions. Schema validation, regex syntax checks, and validator initialization run before session creation.
166
+ Constraints use a subset of JSON Schema 2020-12 with no references:
138
167
 
139
- Unsupported constraints fail setup: `patternProperties`, references, formats, `multipleOf`, `minContains`, `maxContains`, `uniqueItems`, dependencies, and object/array values in `const` or `enum`. Nested `properties` and `required` cannot use inherited property names such as `constructor` or `toString`. Use nested properties or array items to match structured values.
168
+ - Types, and scalar `const` and `enum`
169
+ - String patterns and bounds, and numeric bounds
170
+ - Array `items` and `contains`
171
+ - Object `properties`, `required`, and `additionalProperties`
172
+ - Logical and conditional schemas
140
173
 
141
- Unknown static tool names, invalid local-agent paths, and paths into remote agents fail session creation. Dynamic names and connection operations are checked where possible; unresolved names are allowed. After the eval, unused rule IDs are reported as warnings. Use tool assertions when a call must happen.
174
+ `pattern` uses JavaScript regular expressions. eve validates schemas and patterns before it creates the session.
142
175
 
143
- Configuration is limited to 100 rules, 20,000 JSON nodes, nesting depth 32, and approximately one million characters. The `__proto__` property name is reserved.
176
+ Unsupported constraints fail setup: `patternProperties`, references, formats, `multipleOf`, `minContains`, `maxContains`, `uniqueItems`, dependencies, and object/array values in `const` or `enum`. Nested `properties` and `required` can't use inherited property names such as `constructor` or `toString`. Use nested properties or array items to match structured values.
144
177
 
145
- ## Sequences and session scope
178
+ Unknown static tool names, invalid local subagent paths, and paths into remote agents fail session creation. eve checks dynamic tool names and connection operations where it can and allows the rest. After the eval, eve warns about rules that matched no calls but doesn't fail the eval. Use tool assertions when a call must happen.
146
179
 
147
- `outcome` repeats on every matching call. `outcomes` advances on each new matching call, then repeats its final entry. For example, `outcomes: [{ response: "pending" }, { response: "completed" }]` returns `"pending"` once and `"completed"` thereafter. Only the selected rule advances; exhaustion never selects a later rule.
180
+ A broken stub, or a failure while processing its response, fails the eval even if the agent recovers. An intentional `throw` outcome is a normal tool failure the eval can assert on. Neither falls back to the real tool. If eve can't record such a failure, it cancels the session's run. A failed or cancelled stubbed session always fails the eval.
148
181
 
149
- Two calls within one agent turn consume two entries. Later turns continue the same sequence. Calls awaiting or denied approval do not consume entries. Each session supports up to 10,000 recorded calls.
182
+ ## Sequences and session scope
150
183
 
151
- Choose the rules at session creation; they stay fixed for that session. Local subagents share playback through explicit tool paths, and deployment handoffs preserve it. An independent session starts at the first response.
184
+ `outcome` repeats on every matching call. `outcomes` moves to the next entry on each matching call, then repeats its last entry. For example, `outcomes: [{ response: "pending" }, { response: "completed" }]` returns `"pending"` once and `"completed"` after that. Only the selected rule moves forward; running out of entries never selects a later rule.
152
185
 
153
- eve's transparent internal workflow replay reuses the recorded outcome without consuming another entry.
186
+ Two calls in one turn use two entries, and later turns continue the same sequence. Calls waiting for approval, or denied, don't use an entry. A replayed workflow step returns its recorded outcome without using another entry. Each session supports up to 10,000 tool calls.
154
187
 
155
- Parallel calls consume responses in arrival order, which can vary. Use separate argument matchers when particular arguments must receive particular responses.
188
+ Rules are fixed when the session is created. Local subagents use them through qualified tool paths, and sequence positions survive a handoff to a new deployment. A separate session starts each sequence over.
156
189
 
157
- ## Permit tool replacement
190
+ Parallel calls use responses in the order they arrive, which can vary. Use separate argument matchers when particular arguments must receive particular responses.
158
191
 
159
- Tool replacement is denied by default, including in development. With [Vercel OIDC](../guides/auth-and-route-protection#verceloidc), put the permission on the eval runner's verified subject:
192
+ ## Inject tool errors
160
193
 
161
- ```ts title="agent/channels/eve.ts"
162
- import { vercelOidc, vercelSubject } from "eve/channels/auth";
163
- import { eveChannel } from "eve/channels/eve";
194
+ Use `throw` to test how the agent handles a failed tool call. Mix errors and responses to test recovery:
164
195
 
165
- export default eveChannel({
166
- auth: [
167
- vercelOidc({
168
- subjects: [
169
- vercelSubject({ teamSlug: "acme", projectName: "app" }),
170
- {
171
- subject: vercelSubject({ teamSlug: "acme", projectName: "eval-runner" }),
172
- allowToolStubs: true,
173
- },
174
- ],
175
- }),
176
- ],
177
- });
196
+ ```ts
197
+ const stubs = [
198
+ {
199
+ id: "tasks",
200
+ tool: "list_tasks",
201
+ outcomes: [
202
+ { throw: { name: "TimeoutError", message: "Task service timed out" } },
203
+ { response: { tasks: [] } },
204
+ ],
205
+ },
206
+ ];
178
207
  ```
179
208
 
180
- Both projects can authenticate; only the eval runner receives replacement permission. `vercelSubject` defaults to production; pass `environment: "*"` to include all environments. The target's server configuration supplies the grant after token verification. Request JSON cannot grant permission, and user principals do not inherit project grants.
209
+ `throw.message` is required. Omit `throw.name` to use `"Error"`. eve throws a plain JavaScript `Error` with that name and message. A `TimeoutError` fails the call right away instead of waiting for a deadline.
181
210
 
182
- Custom authenticators can return the same `allowToolStubs: true` after verifying the caller. See [authentication and route protection](../guides/auth-and-route-protection#tool-replacement-permission) for the grant rules and a local development example.
211
+ Ordinary tools and connection operations show the model the message without the name; workflow task errors keep the name. Put details the model needs in `message`.
183
212
 
184
- Permission is checked at session creation before identity projection. Later messages and result reads use normal channel authentication. Anyone your channel admits to an existing session can use its stubs. Your application's session-access policy must provide the required isolation; an isolated eval deployment can use the same agent code.
213
+ Use `outcome: { throw: { message: "Service unavailable" } }` to fail every matching call. In a sequence, a throwing entry is used up like a response.
185
214
 
186
- The stub verification endpoint also requires `allowToolStubs`, checked before reading workflow status.
215
+ An injected error follows the tool's normal error handling. Ordinary and blocking workflow calls are recorded as failed; background tasks report failure when they finish. The model can report the failure or retry. Assert the failed call explicitly:
216
+
217
+ ```ts
218
+ const turn = await session.send(
219
+ "List my tasks once. If the service fails, explain the failure and wait for me to retry.",
220
+ );
221
+ turn.expectOk();
222
+ turn.calledTool("list_tasks", { status: "failed", count: 1 });
223
+ ```
224
+
225
+ `noFailedCalls()` fails on the injected error. Stub setup and response-processing failures still fail the eval.
226
+
227
+ `throw` works for ordinary tools, workflow tools, whole-subagent replacements, and qualified connection operations. An error in a persistent workflow tool ends that task, so retry with a new call instead of the failed task ID. Normal approvals apply before eve uses an outcome.
228
+
229
+ `{ response: { error: "timeout" } }` is ordinary JSON; only `throw` raises an error.
187
230
 
188
231
  ## Supported tools and subagents
189
232
 
190
- Stubs replace execution of existing ordinary, dynamic, and workflow tools. They retain the real tool's input schema, approvals, and output processing. A stub does not register a new tool or make it visible to the model.
233
+ Stubs replace the execution of existing ordinary, dynamic, and workflow tools. The real tool's input schema, approvals, and output processing still apply. A stub doesn't add a tool or show one to the model.
191
234
 
192
- Ordinary tools with configured stub rules can emit an extra `call.progress` containing the final response. This also applies when no rule matches and the real tool returns a non-streaming result. The final `call.settled` remains correct, and real streaming still delivers progress. Use `call.settled` for final-result assertions. Removing the extra event is tracked in [AX-5164](https://linear.app/vercel/issue/AX-5164).
235
+ An ordinary tool with stub rules can emit an extra `call.progress` event containing the final response, even when no rule matches. Assert final results on `call.settled`, which stays correct.
193
236
 
194
237
  ### Subagents
195
238
 
196
- Stubbing a delegation tool returns its configured result without starting the subagent. Persistent tools implemented with `serve`, including the built-in `agent`, require an unconditional stub: omit `match` or use `{}`. Mixing real and mocked calls to the same persistent tool is unsupported.
239
+ Stubbing a subagent's tool returns the configured result without starting the subagent. Persistent tools implemented with `serve`, including the built-in `agent`, require an unconditional stub: omit `match` or use `{}`. You can't mix real and stubbed calls to the same persistent tool.
197
240
 
198
- To let a local subagent run but replace its tools, use `/` to qualify the delegation path:
241
+ To let a local subagent run but replace its tools, prefix the tool name with the subagent path and `/`:
199
242
 
200
243
  ```ts
201
244
  const stubs = [
@@ -205,55 +248,26 @@ const stubs = [
205
248
  ];
206
249
  ```
207
250
 
208
- `researcher` replaces the whole delegation; `researcher/list_tasks` replaces only that subagent's list tool. The built-in `agent` uses `agent/list_tasks`. Model-visible names stay unchanged. A root rule never implicitly matches a subagent's same-named tool.
251
+ `researcher` replaces the whole subagent; `researcher/list_tasks` replaces only that subagent's list tool. The built-in `agent` uses `agent/list_tasks`. The model still sees the usual tool names. A root rule never matches a subagent's tool with the same name.
209
252
 
210
- Repeated subagent sessions at the same path share that rule's sequence. Different paths use different rules. Remote agents do not receive the configuration; replace their whole delegation tool instead.
253
+ Repeated sessions of a subagent at the same path share that rule's sequence. Remote agents don't receive stub rules, so replace the whole remote agent tool.
211
254
 
212
255
  ### Connections and provider tools
213
256
 
214
257
  Use the qualified operation name, such as `linear__list_issues` or `researcher/linear__list_issues`. Put operation arguments in `match` and its JSON result in `outcome.response`, or in each sequence entry's `response`.
215
258
 
216
- Connection discovery stays live: eve lists the connection's tools to validate the operation and its arguments before a stub replaces the call, so stubs do not provide an offline connection.
217
-
218
- A rule names the tool itself, whether the model calls it directly or through [`eve__tool`](/docs/concepts/built-in-tools#eve__search-eve__tool-and-eve__skill), as it does for deferred tools and connection operations. A call through `eve__tool` matches by the name and `input` of the tool it reaches. `eve__search`, `eve__tool`, and `eve__skill` cannot themselves be stubbed.
219
-
220
- Tools executed by the model provider, such as provider-hosted web search, have no eve executor to replace. A search tool implemented in your own agent can be stubbed.
221
-
222
- ## Inject tool errors
223
-
224
- Use `throw` to test how the agent handles a failed tool call. Mix errors and responses to test recovery:
259
+ A stubbed connection must still be reachable, because eve lists the connection's tools to validate the operation and its arguments.
225
260
 
226
- ```ts
227
- const stubs = [
228
- {
229
- id: "tasks",
230
- tool: "list_tasks",
231
- outcomes: [
232
- { throw: { name: "TimeoutError", message: "Task service timed out" } },
233
- { response: { tasks: [] } },
234
- ],
235
- },
236
- ];
237
- ```
238
-
239
- `throw.message` is required. `throw.name` is optional and defaults to `"Error"`. eve creates an ordinary JavaScript `Error` with that name and message; it does not instantiate custom error classes. `TimeoutError` reports a timeout immediately, without delaying the call or testing an actual deadline.
240
-
241
- Ordinary tools and connection operations expose the error message to the model, without the error name. Workflow task errors also retain the name. Put any details the model needs to act on in `message`.
242
-
243
- Use `outcome: { throw: { message: "Service unavailable" } }` to fail every matching call. A sequence consumes one entry per new call, whether it returns or throws, and repeats its final entry. Workflow replay reuses the same recorded outcome.
244
-
245
- An injected exception follows the tool's normal error handling. Ordinary and blocking workflow calls are recorded as failed; background tasks report failure when they settle. The model can report the failure or try again. Assert the failed call explicitly:
246
-
247
- ```ts
248
- const turn = await session.send(
249
- "List my tasks once. If the service fails, explain the failure and wait for me to retry.",
250
- );
251
- turn.expectOk();
252
- turn.calledTool("list_tasks", { status: "failed", count: 1 });
253
- ```
261
+ A rule names the tool itself, whether the model calls it directly or through [`eve__tool`](/docs/concepts/built-in-tools#eve__search-eve__tool-and-eve__skill). The model uses `eve__tool` for deferred tools and connection operations. A call through `eve__tool` matches by the name and `input` of the tool it reaches. `eve__search`, `eve__tool`, and `eve__skill` can't be stubbed.
254
262
 
255
- The eval can pass when the agent handles the failure correctly. `noFailedCalls()` still rejects the failed call, so use explicit assertions for expected failures. Stub infrastructure and response-processing failures still fail verification.
263
+ Tools the model provider runs, such as provider-hosted web search, can't be stubbed. A search tool in your own agent can.
256
264
 
257
- This works for ordinary tools, workflow tools, whole-agent replacements, and qualified connection operations. An exception in a persistent workflow tool ends that task; retry with a new call, rather than its failed task ID. Normal approvals apply before an outcome is consumed.
265
+ ## Troubleshooting
258
266
 
259
- Returning `{ response: { error: "timeout" } }` remains ordinary JSON. Only `throw` raises an exception.
267
+ | What you see | What to do |
268
+ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
269
+ | `403` with `Tool stubbing is not permitted for this caller.` | Grant `allowToolStubs: true` to the eval runner's subject; see [Permit tool replacement](#permit-tool-replacement). |
270
+ | `Tool stub "<id>" names an unknown or unsupported tool path "<tool>".` | Fix the tool name or subagent path. For a remote agent, stub the whole remote agent tool. |
271
+ | `Persistent tool "<tool>" requires an unconditional stub.` | Remove `match` from every rule for that tool, or set it to `{}`. |
272
+ | Warning `Tool stubs did not match any calls: <ids>.` | Check the rule order, the `match` constraints, and the tool name. eve ran the real tool for calls no rule matched. |
273
+ | `Tool stub session exceeded 10,000 calls. Start a new eval session.` | Split the eval into several sessions. |