eve 0.90.0 → 0.91.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (185) hide show
  1. package/CHANGELOG.md +34 -23
  2. package/dist/src/channel/auth/http-basic.js +1 -1
  3. package/dist/src/channel/channel-operations.d.ts +1 -1
  4. package/dist/src/channel/types.d.ts +2 -0
  5. package/dist/src/cli/dev/tui/prompt-commands.d.ts +1 -1
  6. package/dist/src/client/session.d.ts +1 -1
  7. package/dist/src/compiled/.vendor-stamp.json +1 -1
  8. package/dist/src/execution/dynamic-workflow/tool.js +1 -1
  9. package/dist/src/execution/proxied-deliver-step.js +1 -1
  10. package/dist/src/execution/sandbox/bash.js +2 -2
  11. package/dist/src/execution/session/active-turn.js +1 -1
  12. package/dist/src/execution/session/admission.d.ts +6 -0
  13. package/dist/src/execution/session/admission.js +1 -1
  14. package/dist/src/execution/session/delivery-facts.js +1 -1
  15. package/dist/src/execution/session/next-input.d.ts +2 -2
  16. package/dist/src/execution/session/next-input.js +1 -1
  17. package/dist/src/execution/session/program.js +1 -1
  18. package/dist/src/execution/session-inbox/inbox.d.ts +2 -2
  19. package/dist/src/execution/session-inbox/inbox.js +1 -1
  20. package/dist/src/execution/session-inbox/protocol.js +1 -1
  21. package/dist/src/execution/tools/agent-router-workflow.js +1 -1
  22. package/dist/src/harness/compaction/engine.js +2 -2
  23. package/dist/src/harness/current-messages.js +1 -1
  24. package/dist/src/harness/emission.d.ts +3 -0
  25. package/dist/src/harness/emission.js +1 -1
  26. package/dist/src/harness/facts.d.ts +5 -2
  27. package/dist/src/harness/facts.js +1 -1
  28. package/dist/src/harness/model-call/call.d.ts +7 -1
  29. package/dist/src/harness/model-call/call.js +3 -1
  30. package/dist/src/harness/provider-tool-history.d.ts +8 -0
  31. package/dist/src/harness/provider-tool-history.js +1 -1
  32. package/dist/src/harness/session-machine/transitions.d.ts +3 -2
  33. package/dist/src/harness/step/after-model.js +1 -1
  34. package/dist/src/harness/step/context.js +1 -1
  35. package/dist/src/harness/step/intake.js +1 -1
  36. package/dist/src/harness/step/prompt.js +1 -1
  37. package/dist/src/harness/tool-history-repair.d.ts +2 -0
  38. package/dist/src/harness/tool-history-repair.js +1 -1
  39. package/dist/src/harness/tool-loop.js +1 -1
  40. package/dist/src/internal/application/stamped-package-version.js +1 -1
  41. package/dist/src/protocol/session-events/families/delivery.d.ts +5 -3
  42. package/dist/src/protocol/session-events/families/delivery.js +1 -1
  43. package/dist/src/protocol/session-projection/fold.js +1 -1
  44. package/dist/src/public/channels/discord/hitl.js +1 -1
  45. package/dist/src/public/channels/slack/hitl.d.ts +2 -1
  46. package/dist/src/public/channels/slack/hitl.js +3 -3
  47. package/dist/src/public/channels/slack/interaction-cards.js +1 -1
  48. package/dist/src/public/channels/slack/request-card-render.js +1 -1
  49. package/dist/src/public/channels/teams/hitl.js +1 -1
  50. package/dist/src/public/channels/teams/verify.js +1 -1
  51. package/dist/src/public/definitions/hook.d.ts +90 -7
  52. package/dist/src/public/definitions/hook.js +1 -1
  53. package/dist/src/public/hooks/index.d.ts +3 -2
  54. package/dist/src/reactions/continuation.d.ts +19 -0
  55. package/dist/src/reactions/continuation.js +1 -0
  56. package/dist/src/reactions/intents.d.ts +10 -4
  57. package/dist/src/reactions/intents.js +1 -1
  58. package/dist/src/reactions/kinds/hook.js +1 -1
  59. package/dist/src/reactions/runner.js +1 -1
  60. package/dist/src/reactions/state.d.ts +15 -2
  61. package/dist/src/reactions/state.js +1 -1
  62. package/dist/src/runtime/resolve-connection.js +1 -1
  63. package/dist/src/runtime/subagents/workflow.js +1 -1
  64. package/dist/src/setup/primitives/open-url.js +1 -1
  65. package/dist/src/setup/scaffold/create/project.js +1 -1
  66. package/dist/src/shared/value-delta.d.ts +5 -4
  67. package/dist/src/shared/value-delta.js +1 -1
  68. package/dist/src/tools/provided/ask-question-workflow.js +1 -1
  69. package/dist/src/tools/provided/sleep-workflow.js +1 -1
  70. package/dist/src/tools/provided/web-fetch.js +1 -1
  71. package/docs/agent-config.md +199 -222
  72. package/docs/channels/chat-sdk.mdx +60 -64
  73. package/docs/channels/custom.mdx +264 -281
  74. package/docs/channels/discord.mdx +176 -123
  75. package/docs/channels/eve.mdx +136 -206
  76. package/docs/channels/github.mdx +24 -22
  77. package/docs/channels/linear.mdx +30 -30
  78. package/docs/channels/linq.mdx +25 -13
  79. package/docs/channels/mcp.mdx +154 -116
  80. package/docs/channels/overview.mdx +31 -32
  81. package/docs/channels/photon.mdx +32 -56
  82. package/docs/channels/slack.mdx +538 -375
  83. package/docs/channels/teams.mdx +40 -26
  84. package/docs/channels/telegram.mdx +13 -13
  85. package/docs/channels/twilio.mdx +11 -11
  86. package/docs/code-extension.mdx +40 -42
  87. package/docs/concepts/built-in-tools.md +89 -57
  88. package/docs/concepts/context-control.md +20 -20
  89. package/docs/concepts/default-harness.md +20 -36
  90. package/docs/concepts/execution-model-and-durability.mdx +96 -160
  91. package/docs/concepts/project-structure.mdx +21 -14
  92. package/docs/concepts/security-model.md +27 -27
  93. package/docs/concepts/sessions-runs-and-streaming.md +184 -198
  94. package/docs/concepts/state.md +7 -15
  95. package/docs/connections/mcp.mdx +109 -93
  96. package/docs/connections/openapi.mdx +57 -59
  97. package/docs/connections/overview.mdx +175 -170
  98. package/docs/evals/assertions.mdx +36 -29
  99. package/docs/evals/cases.mdx +38 -42
  100. package/docs/evals/judge.mdx +21 -29
  101. package/docs/evals/overview.mdx +42 -38
  102. package/docs/evals/reporters.mdx +54 -27
  103. package/docs/evals/running.mdx +15 -8
  104. package/docs/evals/targets.mdx +16 -19
  105. package/docs/evals/tool-stubs.mdx +111 -97
  106. package/docs/experimental/dynamic-schedules.mdx +118 -63
  107. package/docs/extensions.md +58 -65
  108. package/docs/getting-started.mdx +4 -6
  109. package/docs/guides/auth-and-route-protection.md +87 -70
  110. package/docs/guides/client/continuations.mdx +23 -26
  111. package/docs/guides/client/messages.mdx +47 -66
  112. package/docs/guides/client/output-schema.mdx +8 -8
  113. package/docs/guides/client/overview.mdx +45 -43
  114. package/docs/guides/client/streaming.mdx +62 -71
  115. package/docs/guides/decide.md +24 -79
  116. package/docs/guides/deployment/meta.json +1 -1
  117. package/docs/guides/deployment/overview.md +27 -18
  118. package/docs/guides/deployment/self-hosted-operations.md +99 -0
  119. package/docs/guides/deployment/self-hosting.md +63 -80
  120. package/docs/guides/deployment/vercel.mdx +31 -27
  121. package/docs/guides/deployment/workflow-worlds.md +100 -0
  122. package/docs/guides/dev-tui.md +48 -41
  123. package/docs/guides/dynamic-capabilities.md +49 -50
  124. package/docs/guides/frontend/nextjs.mdx +2 -2
  125. package/docs/guides/frontend/nuxt.mdx +11 -11
  126. package/docs/guides/frontend/overview.mdx +1 -1
  127. package/docs/guides/frontend/sveltekit.mdx +11 -11
  128. package/docs/guides/frontend/tanstack.mdx +16 -16
  129. package/docs/guides/frontend/use-eve-agent-svelte.mdx +21 -21
  130. package/docs/guides/frontend/use-eve-agent-vue.mdx +22 -22
  131. package/docs/guides/frontend/use-eve-agent.mdx +25 -25
  132. package/docs/guides/hooks.md +152 -106
  133. package/docs/guides/instrumentation/agent-runs.mdx +26 -30
  134. package/docs/guides/instrumentation/instrumentation.mdx +49 -76
  135. package/docs/guides/instrumentation/meta.json +1 -1
  136. package/docs/guides/instrumentation/otel.mdx +113 -179
  137. package/docs/guides/remote-agents.md +125 -187
  138. package/docs/guides/self-modification.md +18 -18
  139. package/docs/guides/session-context.md +16 -16
  140. package/docs/install-integrations.mdx +26 -24
  141. package/docs/instructions.mdx +37 -42
  142. package/docs/memory/custom-provider.md +35 -41
  143. package/docs/memory/file.md +93 -76
  144. package/docs/memory/overview.mdx +145 -156
  145. package/docs/patterns/durable-cross-channel-notifications.md +15 -19
  146. package/docs/patterns/dynamic-scheduling.md +19 -23
  147. package/docs/patterns/multi-tenant-approvals.md +17 -22
  148. package/docs/patterns/multi-tenant-auth.md +43 -42
  149. package/docs/patterns/multi-tenant-memory.md +39 -36
  150. package/docs/patterns/voice-notes.md +3 -3
  151. package/docs/protocols/acp.md +13 -13
  152. package/docs/protocols/ucp.mdx +7 -7
  153. package/docs/reference/agent-files.md +6 -6
  154. package/docs/reference/cli.md +99 -98
  155. package/docs/reference/telemetry.md +1 -1
  156. package/docs/reference/typescript-api.md +52 -53
  157. package/docs/responsible-use.md +2 -2
  158. package/docs/sandbox/custom-providers.mdx +60 -56
  159. package/docs/sandbox/default.mdx +23 -24
  160. package/docs/sandbox/docker.mdx +16 -10
  161. package/docs/sandbox/index.mdx +93 -61
  162. package/docs/sandbox/just-bash.mdx +17 -11
  163. package/docs/sandbox/troubleshooting.mdx +29 -23
  164. package/docs/sandbox/vercel.mdx +45 -52
  165. package/docs/schedules.mdx +65 -54
  166. package/docs/skills.mdx +74 -49
  167. package/docs/subagents/index.mdx +97 -136
  168. package/docs/tools/human-in-the-loop.md +160 -90
  169. package/docs/tools/meta.json +1 -1
  170. package/docs/tools/overview.mdx +229 -237
  171. package/docs/tools/tasks.md +118 -117
  172. package/docs/tools/workflows.mdx +447 -543
  173. package/docs/tutorial/connect-a-warehouse.mdx +11 -13
  174. package/docs/tutorial/first-agent.mdx +14 -16
  175. package/docs/tutorial/guard-the-spend.mdx +6 -8
  176. package/docs/tutorial/how-it-runs.mdx +8 -8
  177. package/docs/tutorial/query-sample-data.mdx +8 -8
  178. package/docs/tutorial/remember-definitions.mdx +4 -4
  179. package/docs/tutorial/run-analysis.mdx +20 -22
  180. package/docs/tutorial/ship-it.mdx +19 -19
  181. package/docs/tutorial/team-playbooks.mdx +9 -9
  182. package/docs/use-cases.mdx +13 -11
  183. package/package.json +1 -1
  184. package/docs/guides/instrumentation/migration.md +0 -96
  185. package/docs/tools/tasks-upgrade.md +0 -218
@@ -3,11 +3,11 @@ title: "Assertions"
3
3
  description: "Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity."
4
4
  ---
5
5
 
6
- Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are three deterministic surfaces: scoped methods, `t.check` for grading a specific value, and `t.score` for recording a measurement you computed yourself. For model-graded assertions, see [Judge](./judge).
6
+ An assertion grades what an eval's `test(t)` function produced. Each assertion records a result instead of throwing, so one run reports every failing assertion, not only the first. Check what the agent did with scoped methods, grade a value with `t.check`, or record your own measurement with `t.score`. For grading by a model, see [Judge](./judge).
7
7
 
8
8
  ## Scoped assertions
9
9
 
10
- Scoped assertions take no explicit value and gate by default. Assertions on `t` inspect the whole run after `test` finishes. Session assertions snapshot that session when called, and turn assertions inspect one immutable response. A scope is **parked** when it paused on unanswered human-in-the-loop (HITL) input.
10
+ Scoped assertions take no value and are gates by default. Call them on `t` to check the whole run after `test` finishes. Call them on a session to check that session so far, or on a turn to check one reply. A run is **parked** when it stopped to wait on an approval or question that nobody answered (human-in-the-loop, or HITL, input).
11
11
 
12
12
  | Assertion | Asserts |
13
13
  | ----------------------------------------------------- | ----------------------------------------------------------------------------- |
@@ -17,7 +17,7 @@ Scoped assertions take no explicit value and gate by default. Assertions on `t`
17
17
  | `turn.outputEquals(value)` / `.outputMatches(schema)` | Deep equality or Standard Schema validation of turn/session structured output |
18
18
  | `t.calledTool(name, opts?)` | A matching tool call completed (`input`, `output`, `status`, `count`) |
19
19
  | `t.calledSkill(skill, opts?)` | A load of `skill` completed (`output`, `status`, `count`) |
20
- | `t.notCalledTool(name)` | No request for `name` in any lifecycle state |
20
+ | `t.notCalledTool(name)` | No call to `name`, whatever its status |
21
21
  | `t.toolOrder([...names])` | Tool requests appear in order |
22
22
  | `t.usedNoTools()` | No tool calls at all; skill loads don't count |
23
23
  | `t.maxToolCalls(n)` | At most `n` tool calls; skill loads don't count |
@@ -27,7 +27,7 @@ Scoped assertions take no explicit value and gate by default. Assertions on `t`
27
27
  | `t.eventOrder([...matchers])` | Matching event groups occur in order |
28
28
  | `t.eventsSatisfy(label, predicate)` | Escape hatch: any predicate over the typed event stream |
29
29
 
30
- `succeeded()` accepts both a closed session and a healthy session left open for the next user message; it rejects protocol failures and unanswered HITL. `awaitingInput()` requires a clean HITL park. Structured output assertions live on turns and independent sessions, where the output is unambiguous (see the [output schema guide](../guides/client/output-schema)).
30
+ `succeeded()` passes when the session ended or is waiting for the next message. It fails when the run failed, an event reports an error, or the run is parked. Structured output assertions exist only on turns and sessions, where one output is unambiguous (see the [output schema guide](../guides/client/output-schema)).
31
31
 
32
32
  ```ts
33
33
  await t.send("What is the weather in Brooklyn?");
@@ -35,7 +35,7 @@ t.succeeded();
35
35
  t.calledTool("get_weather");
36
36
  ```
37
37
 
38
- The same vocabulary narrows naturally in multi-turn and externally-created session evals:
38
+ The same methods work on a single turn or on an attached session:
39
39
 
40
40
  ```ts
41
41
  const first = await t.send("Call get_weather for Brooklyn");
@@ -47,13 +47,15 @@ attached.succeeded();
47
47
  attached.messageIncludes("Sunny");
48
48
  ```
49
49
 
50
- `t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other, never both in the same run.
50
+ `t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other in a run.
51
51
 
52
- A call through [`eve__tool`](../concepts/built-in-tools#eve__search-eve__tool-and-eve__skill) is reported under the name and input of the entry it reaches, such as `linear__list_issues`, so assert on that entry; `eve__search` calls count as tool calls in `t.usedNoTools()` and `t.maxToolCalls(n)`, and skill loads don't. The tool-name assertions and `turn.requireToolCall` throw for `eve__tool` and `eve__skill`, whose calls are reported as their entries and as skill loads, and for a retired name such as `load_skill` unless the root agent declares a tool by that name or has a dynamic tool resolver, and say what to assert instead.
52
+ A call through [`eve__tool`](../concepts/built-in-tools#eve__search-eve__tool-and-eve__skill) is reported under the name and input of the tool it reaches, such as `linear__list_issues`. Assert on that tool. `eve__search` calls count toward `t.usedNoTools()` and `t.maxToolCalls(n)`; skill loads don't.
53
+
54
+ The tool-name assertions and `turn.requireToolCall` throw a `TypeError` for `eve__tool`, `eve__skill`, and built-in names such as `load_skill`. The error says what to assert instead. eve accepts a name like `load_skill` when the root agent declares a tool by that name or has a dynamic tool resolver.
53
55
 
54
56
  ## Value assertions with `t.check`
55
57
 
56
- `t.check(value, assertion)` grades an explicit value against a builder from `eve/evals/expect`. The value can be a turn’s `.reply`, parsed JSON, or any local you computed:
58
+ `t.check(value, assertion)` grades any value, such as a turn's `.reply` or parsed JSON, with a builder from `eve/evals/expect`:
57
59
 
58
60
  ```ts
59
61
  import { includes, equals, matches, satisfies, similarity } from "eve/evals/expect";
@@ -82,11 +84,11 @@ For example, a failed labeled equality check prints:
82
84
  ✗ equals [status] (0% < 100%): expected {"status":"active"}; received {"status":"disabled"}
83
85
  ```
84
86
 
85
- Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge).
87
+ Pick the cheapest builder that captures "correct". Use `similarity` when an exact match is too strict, and the [judge](./judge) for nuanced grading.
86
88
 
87
89
  ## Scores with `t.score`
88
90
 
89
- `t.score(evaluation)` records a score from 0 to 1. The evaluation is the number or `{ score, message?, metadata? }`, or a promise of either. Chain `.label(key)` to name it; reporters and artifacts expose the label as `key`. Without a label the score reports as `score`, and several unlabeled scores share that name.
91
+ `t.score(evaluation)` records a score from 0 to 1. Pass a number, `{ score, message?, metadata? }`, or a promise of either. Chain `.label(key)` to name it; reporters and artifacts show the label as `key`. Unlabeled scores all report as `score`.
90
92
 
91
93
  ```ts
92
94
  const reply = turn.reply ?? "";
@@ -97,16 +99,16 @@ t.score({
97
99
  t.score(gradeFaithfulness(reply, reference)).label("faithfulness"); // async
98
100
  ```
99
101
 
100
- On its own a score has no verdict and does not affect the eval's outcome. Chain a rule when it should:
102
+ A score without a rule doesn't affect the eval's outcome. Chain a rule when it should:
101
103
 
102
104
  ```ts
103
105
  t.score(gradeFaithfulness(reply, reference)).label("faithfulness").gate(0.9); // fails below 0.9
104
106
  t.score(brevity).label("reply-brevity").atLeast(0.5); // marks the eval `scored`, fatal under --strict
105
107
  ```
106
108
 
107
- The score is reported as measured. A `0.82` gated at `0.9` is reported as `score: 0.82`, `threshold: 0.9`, `passed: false`. An evaluator that throws or returns a score outside 0 to 1 records an errored outcome with the error message and no score, and fails the eval.
109
+ Reports show the measured score: a `0.82` gated at `0.9` reports `score: 0.82`, `threshold: 0.9`, `passed: false`. An evaluator that throws or returns a score outside 0 to 1 fails the eval. The result is marked errored, with the error message and no score.
108
110
 
109
- Every [reporter](./reporters#custom-reporters) receives the entries in `onEvalComplete` and can read them by `key`:
111
+ [Reporters](./reporters#custom-reporters) receive the entries in `onEvalComplete` and can read them by `key`:
110
112
 
111
113
  ```ts
112
114
  onEvalComplete(result) {
@@ -117,7 +119,7 @@ onEvalComplete(result) {
117
119
  }
118
120
  ```
119
121
 
120
- The same entries land in the [run artifacts](./running#artifacts) under `.eve/evals/`, in `summary.json` and one line per eval in `results.jsonl`:
122
+ The same entries appear in the [run artifacts](./running#artifacts) under `.eve/evals/`, in `summary.json` and in one `results.jsonl` line per eval:
121
123
 
122
124
  ```json
123
125
  {
@@ -133,11 +135,16 @@ The same entries land in the [run artifacts](./running#artifacts) under `.eve/ev
133
135
 
134
136
  ## The matcher mini-language
135
137
 
136
- `t.calledTool` and `t.calledAgent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, output, status, count }`. Calls match `status: "completed"` by default. Tool lifecycle checks also accept `"pending"`, `"failed"`, and `"rejected"`. Subagent checks accept `"pending"`, `"completed"`, `"failed"`, `"rejected"`, `"interrupted"`, and `"abandoned"`. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls.
138
+ `t.calledTool` and `t.calledAgent` take matcher objects:
139
+
140
+ | Matcher | Fields | `status` values |
141
+ | ------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
142
+ | `calledTool` | `input`, `output`, `status`, `count` | `"completed"` (default), `"pending"`, `"failed"`, `"rejected"` |
143
+ | `calledAgent` | `callId`, `childSessionId`, `output`, `status`, `count` | `"completed"` (default), `"pending"`, `"failed"`, `"rejected"`, `"interrupted"`, `"abandoned"` |
137
144
 
138
- A subagent call is a task call: it stays `pending` until its `call.settled` arrives, then takes that call's outcome and, when completed, its `output`. A completed call leaves the child session available for later calls with the same `taskId`.
145
+ A numeric `count` requires that exact number of calls matching every supplied constraint. For ranges, pass a predicate that receives the number of matching calls.
139
146
 
140
- The runner counts each call that reaches an agent task (`call.started` whose task has `kind: "agent"`) as a subagent call and takes its status from its `call.settled`. `childSessionId` comes from the `child.opened` for the call or its task, so it is absent for a call that failed before its session opened. A proxied input request alone does not change the subagent call's status; assert that request with `requireInteraction`.
147
+ `calledAgent` matches each call to a subagent. A call is `pending` until it finishes, then takes its final status and, when completed, its `output`. A completed subagent keeps its session for later calls with the same `taskId`. `childSessionId` is absent when the call failed before the subagent's session opened. A subagent's approval or question doesn't change the call's status; assert it with `requireInteraction`.
141
148
 
142
149
  Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean:
143
150
 
@@ -156,7 +163,7 @@ t.calledAgent("weather", {
156
163
  });
157
164
  ```
158
165
 
159
- `requireInteraction` uses the same matcher language for `input`, `prompt`, and `display`. Its `optionIds` matcher receives option ids in request order; a literal array must match that complete ordered list exactly:
166
+ `requireInteraction` finds an approval or question. It uses the same matchers for `input`, `prompt`, and `display`. Its `optionIds` matcher receives the option ids in order, and a literal array must match the whole list exactly:
160
167
 
161
168
  ```ts
162
169
  const request = session.requireInteraction({
@@ -167,9 +174,9 @@ const request = session.requireInteraction({
167
174
 
168
175
  ## Run state and derived facts
169
176
 
170
- Beyond the raw `session.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, lifecycle status), subagent calls, and HITL input requests. A turn that leaves the session open for a next message is the normal end state of a successful turn; parking on unanswered HITL input is tracked separately.
177
+ Assertions read what happened from `session.events`: tool calls with their name, input, output, and status, subagent calls, and approvals and questions.
171
178
 
172
- Typed event matching covers presence, absence, numeric or predicate counts, partial event data, and ordering:
179
+ The event assertions match by presence, absence, a numeric or predicate count, partial event data, and order:
173
180
 
174
181
  ```ts
175
182
  turn.notEvent("content.completed", { data: { kind: "result" } });
@@ -179,7 +186,7 @@ turn.eventOrder([
179
186
  ]);
180
187
  ```
181
188
 
182
- When a protocol invariant needs cross-event correlation, `eventsSatisfy` remains the escape hatch:
189
+ To check a rule that spans several events, use `eventsSatisfy`:
183
190
 
184
191
  ```ts
185
192
  t.eventsSatisfy("assistant reply includes the marker", (events) =>
@@ -195,7 +202,7 @@ t.eventsSatisfy("assistant reply includes the marker", (events) =>
195
202
 
196
203
  ### Assert and score reasoning
197
204
 
198
- When the provider exposes reasoning, eve captures each completed block as a `reasoning.completed` event. Assert on its text with `turn.event(...)`, or join the blocks and pass them to a judge:
205
+ When the model provider exposes reasoning, eve records each completed block as a `reasoning.completed` event. Assert on its text with `turn.event(...)`, or join the blocks and pass them to a judge:
199
206
 
200
207
  ```ts
201
208
  turn.event("reasoning.completed", {
@@ -212,11 +219,11 @@ t.judge("The reasoning checks its assumptions before concluding.", {
212
219
  }).atLeast(0.8);
213
220
  ```
214
221
 
215
- A provider that does not expose reasoning emits no `reasoning.completed` event, so the `turn.event` gate fails and the judge grades an empty string. Reasoning the provider keeps hidden cannot be graded. The default judge input and `session.transcript` exclude reasoning, which is why the example passes it with `on`.
222
+ A provider that doesn't expose reasoning emits no `reasoning.completed` event, so the `turn.event` gate fails and the judge grades an empty string. The default judge input and `session.transcript` exclude reasoning, so pass it with `on`.
216
223
 
217
224
  ## Required preconditions
218
225
 
219
- Recorded assertions never throw and are not awaitable. When later control flow depends on a value, use `await t.require(value, assertion)`. It records a gate, returns the original value when it passes, and stops the test body without adding a duplicate execution error when it fails:
226
+ Recorded assertions aren't awaitable. When later code depends on a value, use `await t.require(value, assertion)`. It records a gate and returns the value when it passes. When it fails, it stops the test without reporting a second, duplicate error:
220
227
 
221
228
  ```ts
222
229
  await t.require(
@@ -226,7 +233,7 @@ await t.require(
226
233
  await t.target.attach(sessionIds[0]!).nextTurn();
227
234
  ```
228
235
 
229
- Use the matching `require*` lookups when dependent code needs protocol data:
236
+ Use the matching `require*` lookups when later code needs a tool call or a pending request:
230
237
 
231
238
  ```ts
232
239
  const call = turn.requireToolCall("search");
@@ -235,14 +242,14 @@ const request = session.requireInteraction({ toolName: "guarded" });
235
242
 
236
243
  ## Severity
237
244
 
238
- Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync.
245
+ Every assertion returns a handle. Chain one of these to set its severity:
239
246
 
240
247
  - `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
241
- - `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold it records no verdict and never fails.
248
+ - `.soft(threshold?)` is recorded in reports. A miss below the threshold marks the eval `scored`, which fails the run only under `--strict`. With no threshold, it never fails.
242
249
  - `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
243
- - `.label(key)` adds a stable name when an eval records several assertions from the same family. Reporters and artifacts expose it as `key`. Braintrust and Datadog keep one score per name, so `eve eval` warns when soft assertions in one eval share a name with different scores.
250
+ - `.label(key)` adds a stable name when an eval records several assertions of the same kind. Braintrust and Datadog keep one score per name. `eve eval` warns when soft assertions in one eval share a name but have different scores.
244
251
 
245
- The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge(...)` assertion are soft. Annotate only when you deviate:
252
+ Scoped methods and `includes`, `equals`, `matches`, and `satisfies` are gates; `similarity`, `t.judge(...)`, and `t.score(...)` are soft. Chain a severity only to change the default:
246
253
 
247
254
  ```ts
248
255
  t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate
@@ -3,9 +3,9 @@ title: "Cases"
3
3
  description: "Author single-turn and multi-turn evals with test(t), and fan one file out over a dataset."
4
4
  ---
5
5
 
6
- Each eval file is one graded case by default, and a single file can fan out over a dataset by default-exporting an array (covered below). The runner executes each `test(t)` function against the target, captures every event, and computes a verdict from the [assertions](./assertions) you recorded. Every eval shares one shape, whether single-turn, multi-turn, human-in-the-loop (HITL), or dataset-driven: one `async test(t)` function that drives the agent and asserts inline.
6
+ A case is one `async test(t)` function that sends messages to the agent and records [assertions](./assertions) as it goes. A case can send one message or several, answer approvals, or run once per row of a [dataset](#datasets-exporting-an-array). Each eval file exports one case, or an array of cases.
7
7
 
8
- Before adding a case, create the required config at the root of `evals/`. An empty config is enough when you do not need shared judge, reporter, concurrency, or timeout settings:
8
+ Every `evals/` directory needs an `evals.config.ts` at its root. An empty config works when you don't need shared settings:
9
9
 
10
10
  ```ts title="evals/evals.config.ts"
11
11
  import { defineEvalConfig } from "eve/evals";
@@ -15,7 +15,7 @@ export default defineEvalConfig({});
15
15
 
16
16
  ## Single-turn evals
17
17
 
18
- The common case sends one turn and asserts on the reply. `t.send(input)` resolves once the turn settles, and the returned turn’s `.reply` is the assistant reply:
18
+ A single-turn eval sends one message and asserts on the reply. `t.send(input)` resolves when the agent finishes its turn, and the returned turn's `.reply` holds the reply:
19
19
 
20
20
  ```ts title="evals/weather/brooklyn-forecast.eval.ts"
21
21
  import { defineEval } from "eve/evals";
@@ -30,7 +30,7 @@ export default defineEval({
30
30
  });
31
31
  ```
32
32
 
33
- Some evals only care about behavior, not text. Assert on the run and skip the content check entirely:
33
+ To test behavior rather than text, assert on the run and skip the content check:
34
34
 
35
35
  ```ts title="evals/weather/no-tools-for-greetings.eval.ts"
36
36
  import { defineEval } from "eve/evals";
@@ -46,7 +46,7 @@ export default defineEval({
46
46
 
47
47
  ## Organizing with directories
48
48
 
49
- Identity is the file path, so directories are the grouping mechanism. `evals/weather/brooklyn-forecast.eval.ts` gets the id `weather/brooklyn-forecast`, and `eve eval weather` runs everything under `evals/weather/`. Shared constants and helpers live in sibling non-eval files (any name that doesn't end in `.eval.ts`):
49
+ The file path is the eval's id, so directories group evals. `evals/weather/brooklyn-forecast.eval.ts` gets the id `weather/brooklyn-forecast`, and `eve eval weather` runs everything under `evals/weather/`. Put shared constants and helpers in sibling files whose names don't end in `.eval.ts`:
50
50
 
51
51
  ```text
52
52
  evals/
@@ -60,7 +60,7 @@ evals/
60
60
 
61
61
  ## Multi-turn evals
62
62
 
63
- Drive several turns in sequence for branching, HITL approvals, structured output, attachments, or multiple sessions. Because assertions live in the function, an intermediate value is a local variable. Keep each turn in a local variable and continue through its `.session`.
63
+ Send several messages in order to test branching, approvals, structured output, attachments, or more than one conversation. Keep each turn in a variable and send the next message through its `.session`:
64
64
 
65
65
  ```ts title="evals/draft-then-send.eval.ts"
66
66
  import { defineEval } from "eve/evals";
@@ -78,7 +78,7 @@ export default defineEval({
78
78
  });
79
79
  ```
80
80
 
81
- Use scoped assertions for intermediate turns. When later control flow depends on a value-level check, `t.require` records a gate and stops the script if it fails:
81
+ Use scoped assertions on turns in the middle of the conversation. When later code depends on a value check, `t.require` records a gate and stops the eval if it fails:
82
82
 
83
83
  ```ts title="evals/session-continuity.eval.ts"
84
84
  import { defineEval } from "eve/evals";
@@ -103,24 +103,32 @@ export default defineEval({
103
103
 
104
104
  ## The drive API
105
105
 
106
- Every `t.session()` or `t.send()` creates a new independent conversation. Continue an existing conversation through its session handle. Events from all sessions feed the same run-level assertions.
106
+ Every `t.session()` or `t.send()` call starts a new, separate conversation (a session). Send more messages through its session handle. The run-level assertions on `t` see the events from every session.
107
107
 
108
- - `t.session(options?)` creates a session without starting a turn or consuming its stream. It resolves on `202 Accepted` with an `EveEvalSession` whose `.id` and `.state` are available. Options accept per-request `headers` and `signal`; creation uses the eval timeout signal by default.
109
- - `t.send(message, options?)` creates a new session with its first message in one request and waits for the turn to settle. The returned turn carries `.session`, `.reply`, and `.expectOk()`.
110
- - `session.send(message, options?)` sends a follow-up on that session and waits for it to settle.
111
- - `session.start(message, options?)` starts a text turn and returns as soon as the server accepts it. The live turn exposes `.session`, `.sessionId`, `.waitForEvent(...)`, `.cancel()`, and `.result()`.
112
- - `session.cancel()` returns a delivery for cooperative cancellation. A live idle session also accepts a no-op cancellation; an inactive session throws `SessionNotActiveError`.
113
- - `session.compact()` compacts that session's history between turns and waits for it to finish. It resolves to the captured events of the compaction, a context change, so assertions such as `.event("context.settled", { data: { kind: "compaction", outcome: "completed" } })` work on the result. It throws when the session is no longer active.
114
- - `session.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
115
- - `session.agent(opened).stream(options?)` follows a delegated child's stream through the parent with the eval client's credentials. Pass a `child.opened` event from that session. It also reaches children on another deployment, which `t.target.attach(...).watch()` cannot address.
116
- - `session.requireInteraction(filter?)` records a gate, requires exactly one pending request, and returns it. Filters match tool name, action input, prompt, display, and option ids.
117
- - `session.respond(responses, options?)` answers specific pending input requests and sends them as the next turn. `session.startRespond(...)` returns a live turn for the same operation.
118
- - `session.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
119
- - `session.events` contains the events captured for that session. `session.transcript` formats its observed user and assistant messages in turn order.
108
+ | Method | Does |
109
+ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
110
+ | `t.session(options?)` | Creates an empty session and resolves on `202 Accepted`, with `.id` and `.state` set |
111
+ | `t.send(message, options?)` | Creates a session with its first message and waits for the turn to finish |
112
+ | `session.send(message, options?)` | Sends a follow-up and waits for the turn to finish |
113
+ | `session.start(message, options?)` | Starts a turn and returns once the server accepts it; see [In-flight turns](#in-flight-turns) |
114
+ | `session.cancel()` | Asks the agent to stop and returns a `Delivery`; a no-op when idle |
115
+ | `session.compact()` | Compacts the conversation history between turns and resolves to a turn with the compaction's events |
116
+ | `session.sendFile(text, path, mediaType?)` | Sends a message with a local file attached as a data URL |
117
+ | `session.agent(opened).stream(options?)` | Follows a subagent's event stream, given a `child.opened` event from that session |
118
+ | `session.requireInteraction(filter?)` | Records a gate that exactly one approval or question is pending, and returns it |
119
+ | `session.respond(responses, options?)` | Answers specific pending requests as the next turn; `session.startRespond(...)` returns without waiting |
120
+ | `session.respondAll(optionId)` | Answers every pending request with the same option as the next turn |
121
+ | `session.events`, `session.transcript` | The session's recorded events, and its user and assistant messages as text |
120
122
 
121
- The transcript uses `User:` and `Assistant:` labels separated by blank lines. It excludes reasoning, tool calls, tool results, and messages from other sessions. It updates after each turn settles, so read it after `await session.send(...)`, `await session.respond(...)`, or `await live.result()`.
123
+ `t.session()` and `t.send()` options accept per-request `headers` and `signal`. Omit `signal` to use the eval's timeout signal.
122
124
 
123
- Each `send` (and `respond`/`respondAll`) resolves to a turn with `.reply`, `.data`, `.events`, `.interactions`, `.toolCalls`, `.session`, `.sessionId`, `.status`, and `.expectOk()`. Turn results retain their captured output and events; `.session` references the live conversation handle. `expectOk()` throws only when the turn ended failed; a session left open for a next message is the normal end state of a successful turn.
125
+ `session.cancel()` throws `SessionNotActiveError` when the session is no longer active, and `session.compact()` also throws. Assert on a compaction with `.event("context.settled", { data: { kind: "compaction", outcome: "completed" } })`.
126
+
127
+ `session.agent(...)` uses the eval's credentials, so unlike `t.target.attach(...).watch()`, it also reaches subagents on another deployment. `requireInteraction` filters match tool name, tool input, prompt, display, and option ids.
128
+
129
+ The transcript uses `User:` and `Assistant:` labels separated by blank lines. It leaves out reasoning, tool calls, tool results, and other sessions' messages. It updates when a turn finishes, so read it after `await session.send(...)`, `await session.respond(...)`, or `await live.result()`.
130
+
131
+ Each `send`, `respond`, and `respondAll` resolves to a turn, which doesn't change after it returns. The turn has `.reply`, `.data`, `.events`, `.interactions`, `.toolCalls`, `.session`, `.sessionId`, `.status`, and `.expectOk()`. `.session` is the conversation handle for sending more messages. `expectOk()` throws only when the turn failed.
124
132
 
125
133
  ## Prewarming a session
126
134
 
@@ -142,23 +150,11 @@ export default defineEval({
142
150
  });
143
151
  ```
144
152
 
145
- Acceptance is not an initialization barrier. The first message runs session initialization;
146
- the TypeScript client handles bounded retries if the workflow inbox is still starting.
147
- Creation failures reject `t.session()` without returning a handle. Each call creates an
148
- independent session, including concurrent calls. Accepted sessions appear in eval results
149
- and participate in timeout cleanup even when no message was sent.
150
-
151
- ## Migrating existing evals
152
-
153
- Replace `t.newSession()` and `t.prewarm()` with `const session = await t.session()`.
154
- Replace consecutive `t.send()` calls with `session.send()` calls, or continue the first
155
- returned turn through `turn.session.send()`. Read identity, cursor, events, and transcript
156
- from `session`; read replies from `turn.reply`. Session operations such as `respond()`,
157
- `requireInteraction()`, and `cancel()` also move from `t` to `session`.
153
+ The agent sets up the session when the first message arrives. If eve can't create the session, `t.session()` rejects. A created session appears in eval results and is cleaned up on timeout, even when no message was sent.
158
154
 
159
155
  ## In-flight turns
160
156
 
161
- Use `start()` when the eval must observe or affect a turn before it settles. A live turn owns one stream consumer: `waitForEvent()` reads typed events from its buffer, and `result()` waits for the boundary and records the same buffered stream as an immutable turn. Event data matchers use the same partial-deep matcher language as `t.event(...)`.
157
+ Use `start()` when the eval must watch or change a turn before it finishes. The returned live turn has `.session`, `.sessionId`, `.waitForEvent(...)`, `.cancel()`, and `.result()`. `waitForEvent()` resolves with the first matching event, including one that arrived before the call. `result()` waits for the turn to end and returns it. Event data matchers use the same partial matching as `t.event(...)`.
162
158
 
163
159
  ```ts title="evals/cancel-running-tool.eval.ts"
164
160
  import { defineEval } from "eve/evals";
@@ -179,19 +175,19 @@ export default defineEval({
179
175
  });
180
176
  ```
181
177
 
182
- `waitForEvent()` rejects if the stream fails or reaches its turn boundary before the requested event. Call `result()` after any coordination to settle the stream and make its events available to run-level assertions.
178
+ `waitForEvent()` rejects if the stream fails or the turn ends before the event arrives. Call `result()` afterward so the run-level assertions see the turn's events.
183
179
 
184
- Use `expectOk()` only when the next operation depends on that intermediate turn succeeding. A final `t.succeeded()` already records a complete-run gate.
180
+ Use `expectOk()` only when the next step depends on an earlier turn succeeding; a final `t.succeeded()` already gates the whole run.
185
181
 
186
- To intentionally omit an eval for the current target, call `t.skip(reason)` before creating sessions, sending messages, or recording assertions. Skipped evals are reported separately and do not affect the exit code.
182
+ To omit an eval for the current target, call `t.skip(reason)` before creating sessions, sending messages, or recording assertions. Skipped evals are reported separately and don't affect the exit code.
187
183
 
188
- Events from every session are captured in the result and artifacts. `t.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run. `t.abortSignal` is an `AbortSignal` that fires on timeout.
184
+ `t.log(message)` writes debug lines to the eval artifact, and `--verbose` also streams them to stdout. `t.abortSignal` fires on timeout.
189
185
 
190
186
  For driving sessions created outside the eval, by a channel webhook or a schedule, see [Targets](./targets).
191
187
 
192
188
  ## Datasets: exporting an array
193
189
 
194
- To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index in array order (`sql/0000`, `sql/0001`, and so on). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
190
+ To run one file once per dataset row, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` works. Each id is the file name plus a zero-padded index: `sql/0000`, `sql/0001`, and so on. `loadJson` and `loadYaml` from `eve/evals/loaders` parse data files relative to the app root:
195
191
 
196
192
  ```ts title="evals/sql.eval.ts"
197
193
  import { defineEval } from "eve/evals";
@@ -213,7 +209,7 @@ export default rows.map((row) =>
213
209
  );
214
210
  ```
215
211
 
216
- The loaders are meant for fixtures, not runtime agent code.
212
+ Use the loaders in evals, not in runtime agent code.
217
213
 
218
214
  ## What to read next
219
215
 
@@ -3,7 +3,7 @@ title: "Judge"
3
3
  description: "Grade evals with decision models using criteria, typed questions, or batches, and set thresholds on each assertion."
4
4
  ---
5
5
 
6
- Use `t.judge(...)` when a deterministic [assertion](./assertions) cannot capture what a good response looks like. It calls [`decide` from `eve/ai`](../guides/decide#decide-inside-a-tool) and defaults to `typesafe-ai/jev`, TypeSafe AI's [Jev decision model](https://vercel.com/i/what-is-jev). The judge is separate from the agent under test and is used only for scoring.
6
+ `t.judge(...)` asks a separate judge model to grade your agent's response. Use it when a fixed [assertion](./assertions) can't describe a good response. It calls [`decide` from `eve/ai`](../guides/decide#decide-inside-a-tool) and defaults to `typesafe-ai/jev`, TypeSafe AI's [Jev decision model](https://vercel.com/i/what-is-jev).
7
7
 
8
8
  ```ts
9
9
  import { defineEval } from "eve/evals";
@@ -17,7 +17,7 @@ export default defineEval({
17
17
  });
18
18
  ```
19
19
 
20
- No judge configuration is required. Configure credentials for the decider as described under [Configuring the judge model](#configuring-the-judge-model).
20
+ The judge needs no configuration, only credentials for the judge model; see [Configuring the judge model](#configuring-the-judge-model).
21
21
 
22
22
  ## The graders
23
23
 
@@ -29,7 +29,7 @@ No judge configuration is required. Configure credentials for the decider as des
29
29
  | `score` | The returned rubric position divided by the highest level index |
30
30
  | `choice` | 1 when the selected option equals `expected`; otherwise 0 |
31
31
 
32
- Boolean scores are model estimates, not guaranteed calibrated confidence. A rubric needs at least two levels, ordered from worst to best. The SDK can return a fractional position on that rubric; eve normalizes it to the 0–1 scale used by assertion thresholds.
32
+ Boolean scores are model estimates, not calibrated confidence. A rubric needs at least two levels, ordered from worst to best; eve normalizes a fractional rubric position to the 0–1 threshold scale.
33
33
 
34
34
  ```ts
35
35
  t.judge({
@@ -41,13 +41,13 @@ t.judge({
41
41
  .atLeast(0.75);
42
42
  ```
43
43
 
44
- Choice questions require an `expected` option key. eve uses that expectation locally for scoring; it does not send it to the decider. Instructions and rubric descriptions accept strings, JSON objects, or JSON arrays. Descriptions can also be `null`.
44
+ Choice questions require an `expected` option key. eve uses it for scoring and doesn't send it to the judge model. Instructions and rubric descriptions accept strings, JSON objects, or JSON arrays; descriptions can also be `null`.
45
45
 
46
46
  ## Choose the value to grade
47
47
 
48
- By default, the decider receives `{ input, output }`: the latest prompt and assistant message from the session whose turn most recently settled. Approval responses and stream reads preserve that prompt. For multipart messages, including `session.sendFile()`, the prompt contains the text parts.
48
+ By default, the judge model receives `{ input, output }`: the user message and reply of the most recently finished turn. Answering an approval or reading an attached session keeps that user message. For a message with a file, such as from `session.sendFile()`, `input` is its text.
49
49
 
50
- Pass `on` to replace the output. JSON values stay structured rather than being converted to strings:
50
+ Pass `on` to replace the output. JSON values stay structured:
51
51
 
52
52
  ```ts
53
53
  const draft = await t.send("Draft the welcome email.");
@@ -66,7 +66,7 @@ t.judge("The assistant remembers the user's favorite word across turns.", {
66
66
 
67
67
  The transcript contains user and assistant messages in turn order, excluding reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals).
68
68
 
69
- `turn.reply` is the turn's final reply. A turn can reply more than once, for example to confirm that work is underway before it waits on tasks. `session.transcript` includes every reply alongside the user's messages. To grade only one turn's replies, filter its messages with `transcriptOf` from `eve/client`:
69
+ `turn.reply` is the turn's last reply. A turn can reply more than once, for example to say work has started before waiting on tasks. `session.transcript` includes every reply. To grade all of one turn's replies, filter its messages with `transcriptOf` from `eve/client`:
70
70
 
71
71
  ```ts
72
72
  import { transcriptOf } from "eve/client";
@@ -81,7 +81,7 @@ t.judge("The agent reported the deploy's result.", { on: replies.join("\n\n") })
81
81
 
82
82
  ## Decide several questions together
83
83
 
84
- A batch makes one decision request against one shared state. Pass `state` to replace the default `{ input, output }` completely. Batch calls use `state` instead of `on`.
84
+ A batch asks several questions in one request about the same value. Pass that value as `state` instead of `on`; it replaces the default `{ input, output }` entirely.
85
85
 
86
86
  ```ts
87
87
  const judgments = t.judge({
@@ -106,25 +106,21 @@ judgments.accurate.atLeast(0.9);
106
106
  judgments.outcome.gate();
107
107
  ```
108
108
 
109
- Keys identify questions and their assertion handles; instructions must explain what to judge. Every question sees the same state. Use separate calls for unrelated inputs. A batch must contain at least one question, and a provider or validation failure fails every assertion in the batch.
109
+ Each key names a question and its assertion handle. Every question sees the same `state`, so use separate calls for unrelated values. A batch needs at least one question, and a provider or validation failure fails every assertion in it.
110
110
 
111
111
  ## Soft scoring and thresholds
112
112
 
113
113
  Judges are soft by default:
114
114
 
115
- - Without a threshold, the score is tracked in reports and artifacts without failing the eval.
116
- - `.atLeast(threshold)` sets a soft bar. A lower score marks the eval `scored`, fatal only under `eve eval --strict`.
117
- - `.gate(threshold)` makes a lower score fail the eval outright. An omitted gate threshold is 1.
118
-
119
115
  ```ts
120
- t.judge("The response cites a source.");
121
- t.judge("The response cites a source.").label("citation").atLeast(0.8);
122
- t.judge("The response cites a source.").gate(0.9);
116
+ t.judge("The response cites a source."); // tracked, never fails
117
+ t.judge("The response cites a source.").label("citation").atLeast(0.8); // below 0.8 marks `scored`, fatal under --strict
118
+ t.judge("The response cites a source.").gate(0.9); // below 0.9 fails; omit the threshold to require 1
123
119
  ```
124
120
 
125
- A boolean probability need not reach exactly 1 even for a good response, so choose an explicit gate threshold for boolean judgments. Choice scores are binary.
121
+ A good response's boolean probability rarely reaches exactly 1, so pass an explicit gate threshold for boolean judgments. Choice scores are binary.
126
122
 
127
- Calls capture their state and questions and start immediately. The runner waits for them during finalization; handles are not awaitable. Each standalone call makes its own request, while a batch shares one request. Eval timeouts cancel pending judgments and stop waiting even if a provider ignores cancellation.
123
+ A judge call copies its input and starts right away. Handles aren't awaitable; eve waits for pending judgments after `test` returns. Each call outside a batch makes its own request. An eval timeout cancels pending judgments, even if the provider ignores cancellation.
128
124
 
129
125
  ## Configuring the judge model
130
126
 
@@ -135,7 +131,7 @@ Model settings resolve in this order:
135
131
  3. Project default: `defineEvalConfig({ judge: { model, modelOptions } })`.
136
132
  4. The default model used by `decide`, currently `typesafe-ai/jev`.
137
133
 
138
- A per-eval `judge` replaces the project's judge configuration. Per-call fields override the resolved configuration. `judge` and its `model` are optional; deterministic evals make no decision requests.
134
+ A per-eval `judge` replaces the project's judge configuration, and per-call fields override both. Evals that never call `t.judge` make no judge requests.
139
135
 
140
136
  ```ts title="evals/evals.config.ts"
141
137
  import { defineEvalConfig } from "eve/evals";
@@ -145,7 +141,7 @@ export default defineEvalConfig({
145
141
  });
146
142
  ```
147
143
 
148
- String IDs resolve through the AI SDK default provider, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY` or Vercel OIDC credentials; eve's existing local Gateway connection is also honored by `decide`. An explicit provider decision model uses that provider's credentials:
144
+ String IDs resolve through the AI SDK default provider's decision API, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY`, Vercel OIDC credentials, or eve's local Gateway connection. Gateway accepts native decision models such as `typesafe-ai/jev`. A language model ID such as `openai/gpt-6-luna` doesn't select the OpenAI adapter. For a language model, pass a provider decision model, which uses that provider's credentials:
149
145
 
150
146
  ```ts
151
147
  import { openai } from "@ai-sdk/openai";
@@ -156,19 +152,15 @@ t.judge("The response addresses the request.", {
156
152
  });
157
153
  ```
158
154
 
159
- Use the provider's `decisionModel` factory, not a language model instance. AI SDK supplies decision adapters for supported language models. Provider support and the decision contract are experimental; see [AI SDK decision](https://ai-sdk.dev/docs/ai-sdk-core/decisions).
160
-
161
- String model IDs resolve through the default provider's decision API. Gateway accepts native decision models such as `typesafe-ai/jev`; passing a language model ID such as `openai/gpt-6-luna` does not automatically select the OpenAI adapter. Use a decision-model instance as shown above for language models.
162
-
163
- Missing credentials, invalid answers, unsupported question types, and provider errors become failed gates even for tracked-only judgments. They do not silently skip the eval or select another model. Validation, authentication, and retries use the shared `decide` implementation.
155
+ Use the provider's `decisionModel` factory, not a language model instance. Provider support and the decision contract are experimental; see [AI SDK decision](https://ai-sdk.dev/docs/ai-sdk-core/decisions).
164
156
 
165
- ## Diagnostics and migration
157
+ Missing credentials, invalid answers, unsupported question types, and provider errors fail as gates, even for soft judgments. eve never skips the eval or falls back to another model.
166
158
 
167
- Reports use `judge.boolean`, `judge.score`, and `judge.choice`; batch assertion names also include their question key. Diagnostics include the captured state, question, and raw answer. Assertion metadata retains the normalized score, model identity, token usage, warnings, and available provider and response metadata. Usage for a batch is marked as shared across its assertions, not a separate charge for each question. Decision models do not promise a prose rationale.
159
+ ## Diagnostics
168
160
 
169
- The autoevals graders have been removed. Replace `t.judge.autoevals.closedQA(criteria, options)` with `t.judge(criteria, options)`. Scores now represent a probability rather than a binary yes/no answer, so review thresholds against your examples. For factuality, summaries, or SQL equivalence, write an explicit question and include the reference in shared state. No old grader buckets are preserved.
161
+ Reports name judge assertions `judge.boolean`, `judge.score`, and `judge.choice`, plus the question key in a batch. Failure details show the graded value, the question, and the raw answer. Assertion metadata holds the normalized score, model id, token usage, warnings, and any provider and response metadata. A batch's usage is marked as shared, not charged per question. Decision models don't promise a written rationale.
170
162
 
171
- Change configured language model instances to decision model instances. `modelOptions.providerOptions`, assertion labels, and threshold methods remain available. [Braintrust reporting](./reporters) continues to accept judge scores independently of the grader implementation.
163
+ Set thresholds against your own examples. For factuality, summaries, or SQL equivalence, write an explicit question and include the reference in shared state. [Braintrust reporting](./reporters) accepts judge scores.
172
164
 
173
165
  ## What to read next
174
166