eve 0.90.0 → 0.91.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (185) hide show
  1. package/CHANGELOG.md +34 -23
  2. package/dist/src/channel/auth/http-basic.js +1 -1
  3. package/dist/src/channel/channel-operations.d.ts +1 -1
  4. package/dist/src/channel/types.d.ts +2 -0
  5. package/dist/src/cli/dev/tui/prompt-commands.d.ts +1 -1
  6. package/dist/src/client/session.d.ts +1 -1
  7. package/dist/src/compiled/.vendor-stamp.json +1 -1
  8. package/dist/src/execution/dynamic-workflow/tool.js +1 -1
  9. package/dist/src/execution/proxied-deliver-step.js +1 -1
  10. package/dist/src/execution/sandbox/bash.js +2 -2
  11. package/dist/src/execution/session/active-turn.js +1 -1
  12. package/dist/src/execution/session/admission.d.ts +6 -0
  13. package/dist/src/execution/session/admission.js +1 -1
  14. package/dist/src/execution/session/delivery-facts.js +1 -1
  15. package/dist/src/execution/session/next-input.d.ts +2 -2
  16. package/dist/src/execution/session/next-input.js +1 -1
  17. package/dist/src/execution/session/program.js +1 -1
  18. package/dist/src/execution/session-inbox/inbox.d.ts +2 -2
  19. package/dist/src/execution/session-inbox/inbox.js +1 -1
  20. package/dist/src/execution/session-inbox/protocol.js +1 -1
  21. package/dist/src/execution/tools/agent-router-workflow.js +1 -1
  22. package/dist/src/harness/compaction/engine.js +2 -2
  23. package/dist/src/harness/current-messages.js +1 -1
  24. package/dist/src/harness/emission.d.ts +3 -0
  25. package/dist/src/harness/emission.js +1 -1
  26. package/dist/src/harness/facts.d.ts +5 -2
  27. package/dist/src/harness/facts.js +1 -1
  28. package/dist/src/harness/model-call/call.d.ts +7 -1
  29. package/dist/src/harness/model-call/call.js +3 -1
  30. package/dist/src/harness/provider-tool-history.d.ts +8 -0
  31. package/dist/src/harness/provider-tool-history.js +1 -1
  32. package/dist/src/harness/session-machine/transitions.d.ts +3 -2
  33. package/dist/src/harness/step/after-model.js +1 -1
  34. package/dist/src/harness/step/context.js +1 -1
  35. package/dist/src/harness/step/intake.js +1 -1
  36. package/dist/src/harness/step/prompt.js +1 -1
  37. package/dist/src/harness/tool-history-repair.d.ts +2 -0
  38. package/dist/src/harness/tool-history-repair.js +1 -1
  39. package/dist/src/harness/tool-loop.js +1 -1
  40. package/dist/src/internal/application/stamped-package-version.js +1 -1
  41. package/dist/src/protocol/session-events/families/delivery.d.ts +5 -3
  42. package/dist/src/protocol/session-events/families/delivery.js +1 -1
  43. package/dist/src/protocol/session-projection/fold.js +1 -1
  44. package/dist/src/public/channels/discord/hitl.js +1 -1
  45. package/dist/src/public/channels/slack/hitl.d.ts +2 -1
  46. package/dist/src/public/channels/slack/hitl.js +3 -3
  47. package/dist/src/public/channels/slack/interaction-cards.js +1 -1
  48. package/dist/src/public/channels/slack/request-card-render.js +1 -1
  49. package/dist/src/public/channels/teams/hitl.js +1 -1
  50. package/dist/src/public/channels/teams/verify.js +1 -1
  51. package/dist/src/public/definitions/hook.d.ts +90 -7
  52. package/dist/src/public/definitions/hook.js +1 -1
  53. package/dist/src/public/hooks/index.d.ts +3 -2
  54. package/dist/src/reactions/continuation.d.ts +19 -0
  55. package/dist/src/reactions/continuation.js +1 -0
  56. package/dist/src/reactions/intents.d.ts +10 -4
  57. package/dist/src/reactions/intents.js +1 -1
  58. package/dist/src/reactions/kinds/hook.js +1 -1
  59. package/dist/src/reactions/runner.js +1 -1
  60. package/dist/src/reactions/state.d.ts +15 -2
  61. package/dist/src/reactions/state.js +1 -1
  62. package/dist/src/runtime/resolve-connection.js +1 -1
  63. package/dist/src/runtime/subagents/workflow.js +1 -1
  64. package/dist/src/setup/primitives/open-url.js +1 -1
  65. package/dist/src/setup/scaffold/create/project.js +1 -1
  66. package/dist/src/shared/value-delta.d.ts +5 -4
  67. package/dist/src/shared/value-delta.js +1 -1
  68. package/dist/src/tools/provided/ask-question-workflow.js +1 -1
  69. package/dist/src/tools/provided/sleep-workflow.js +1 -1
  70. package/dist/src/tools/provided/web-fetch.js +1 -1
  71. package/docs/agent-config.md +199 -222
  72. package/docs/channels/chat-sdk.mdx +60 -64
  73. package/docs/channels/custom.mdx +264 -281
  74. package/docs/channels/discord.mdx +176 -123
  75. package/docs/channels/eve.mdx +136 -206
  76. package/docs/channels/github.mdx +24 -22
  77. package/docs/channels/linear.mdx +30 -30
  78. package/docs/channels/linq.mdx +25 -13
  79. package/docs/channels/mcp.mdx +154 -116
  80. package/docs/channels/overview.mdx +31 -32
  81. package/docs/channels/photon.mdx +32 -56
  82. package/docs/channels/slack.mdx +538 -375
  83. package/docs/channels/teams.mdx +40 -26
  84. package/docs/channels/telegram.mdx +13 -13
  85. package/docs/channels/twilio.mdx +11 -11
  86. package/docs/code-extension.mdx +40 -42
  87. package/docs/concepts/built-in-tools.md +89 -57
  88. package/docs/concepts/context-control.md +20 -20
  89. package/docs/concepts/default-harness.md +20 -36
  90. package/docs/concepts/execution-model-and-durability.mdx +96 -160
  91. package/docs/concepts/project-structure.mdx +21 -14
  92. package/docs/concepts/security-model.md +27 -27
  93. package/docs/concepts/sessions-runs-and-streaming.md +184 -198
  94. package/docs/concepts/state.md +7 -15
  95. package/docs/connections/mcp.mdx +109 -93
  96. package/docs/connections/openapi.mdx +57 -59
  97. package/docs/connections/overview.mdx +175 -170
  98. package/docs/evals/assertions.mdx +36 -29
  99. package/docs/evals/cases.mdx +38 -42
  100. package/docs/evals/judge.mdx +21 -29
  101. package/docs/evals/overview.mdx +42 -38
  102. package/docs/evals/reporters.mdx +54 -27
  103. package/docs/evals/running.mdx +15 -8
  104. package/docs/evals/targets.mdx +16 -19
  105. package/docs/evals/tool-stubs.mdx +111 -97
  106. package/docs/experimental/dynamic-schedules.mdx +118 -63
  107. package/docs/extensions.md +58 -65
  108. package/docs/getting-started.mdx +4 -6
  109. package/docs/guides/auth-and-route-protection.md +87 -70
  110. package/docs/guides/client/continuations.mdx +23 -26
  111. package/docs/guides/client/messages.mdx +47 -66
  112. package/docs/guides/client/output-schema.mdx +8 -8
  113. package/docs/guides/client/overview.mdx +45 -43
  114. package/docs/guides/client/streaming.mdx +62 -71
  115. package/docs/guides/decide.md +24 -79
  116. package/docs/guides/deployment/meta.json +1 -1
  117. package/docs/guides/deployment/overview.md +27 -18
  118. package/docs/guides/deployment/self-hosted-operations.md +99 -0
  119. package/docs/guides/deployment/self-hosting.md +63 -80
  120. package/docs/guides/deployment/vercel.mdx +31 -27
  121. package/docs/guides/deployment/workflow-worlds.md +100 -0
  122. package/docs/guides/dev-tui.md +48 -41
  123. package/docs/guides/dynamic-capabilities.md +49 -50
  124. package/docs/guides/frontend/nextjs.mdx +2 -2
  125. package/docs/guides/frontend/nuxt.mdx +11 -11
  126. package/docs/guides/frontend/overview.mdx +1 -1
  127. package/docs/guides/frontend/sveltekit.mdx +11 -11
  128. package/docs/guides/frontend/tanstack.mdx +16 -16
  129. package/docs/guides/frontend/use-eve-agent-svelte.mdx +21 -21
  130. package/docs/guides/frontend/use-eve-agent-vue.mdx +22 -22
  131. package/docs/guides/frontend/use-eve-agent.mdx +25 -25
  132. package/docs/guides/hooks.md +152 -106
  133. package/docs/guides/instrumentation/agent-runs.mdx +26 -30
  134. package/docs/guides/instrumentation/instrumentation.mdx +49 -76
  135. package/docs/guides/instrumentation/meta.json +1 -1
  136. package/docs/guides/instrumentation/otel.mdx +113 -179
  137. package/docs/guides/remote-agents.md +125 -187
  138. package/docs/guides/self-modification.md +18 -18
  139. package/docs/guides/session-context.md +16 -16
  140. package/docs/install-integrations.mdx +26 -24
  141. package/docs/instructions.mdx +37 -42
  142. package/docs/memory/custom-provider.md +35 -41
  143. package/docs/memory/file.md +93 -76
  144. package/docs/memory/overview.mdx +145 -156
  145. package/docs/patterns/durable-cross-channel-notifications.md +15 -19
  146. package/docs/patterns/dynamic-scheduling.md +19 -23
  147. package/docs/patterns/multi-tenant-approvals.md +17 -22
  148. package/docs/patterns/multi-tenant-auth.md +43 -42
  149. package/docs/patterns/multi-tenant-memory.md +39 -36
  150. package/docs/patterns/voice-notes.md +3 -3
  151. package/docs/protocols/acp.md +13 -13
  152. package/docs/protocols/ucp.mdx +7 -7
  153. package/docs/reference/agent-files.md +6 -6
  154. package/docs/reference/cli.md +99 -98
  155. package/docs/reference/telemetry.md +1 -1
  156. package/docs/reference/typescript-api.md +52 -53
  157. package/docs/responsible-use.md +2 -2
  158. package/docs/sandbox/custom-providers.mdx +60 -56
  159. package/docs/sandbox/default.mdx +23 -24
  160. package/docs/sandbox/docker.mdx +16 -10
  161. package/docs/sandbox/index.mdx +93 -61
  162. package/docs/sandbox/just-bash.mdx +17 -11
  163. package/docs/sandbox/troubleshooting.mdx +29 -23
  164. package/docs/sandbox/vercel.mdx +45 -52
  165. package/docs/schedules.mdx +65 -54
  166. package/docs/skills.mdx +74 -49
  167. package/docs/subagents/index.mdx +97 -136
  168. package/docs/tools/human-in-the-loop.md +160 -90
  169. package/docs/tools/meta.json +1 -1
  170. package/docs/tools/overview.mdx +229 -237
  171. package/docs/tools/tasks.md +118 -117
  172. package/docs/tools/workflows.mdx +447 -543
  173. package/docs/tutorial/connect-a-warehouse.mdx +11 -13
  174. package/docs/tutorial/first-agent.mdx +14 -16
  175. package/docs/tutorial/guard-the-spend.mdx +6 -8
  176. package/docs/tutorial/how-it-runs.mdx +8 -8
  177. package/docs/tutorial/query-sample-data.mdx +8 -8
  178. package/docs/tutorial/remember-definitions.mdx +4 -4
  179. package/docs/tutorial/run-analysis.mdx +20 -22
  180. package/docs/tutorial/ship-it.mdx +19 -19
  181. package/docs/tutorial/team-playbooks.mdx +9 -9
  182. package/docs/use-cases.mdx +13 -11
  183. package/package.json +1 -1
  184. package/docs/guides/instrumentation/migration.md +0 -96
  185. package/docs/tools/tasks-upgrade.md +0 -218
@@ -3,15 +3,13 @@ title: "Overview"
3
3
  description: "Define repeatable scored checks for an eve agent with defineEval and run them with eve eval."
4
4
  ---
5
5
 
6
- An eval is a scored check that runs your agent against real sessions and grades the result, catching regressions when you change a prompt or a tool. Drive the agent through one or more turns, assert on what it did (the run completed, the right tool ran, the reply contains the right text), and optionally ship the results to Braintrust.
6
+ An eval is a scored check that sends messages to your agent and grades what it did. Each eval lives in a `.eval.ts` file under `evals/`, and `eve eval` runs them. Use evals to catch regressions when you change a prompt or a tool.
7
7
 
8
- Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../guides/client/overview) protocol, and grades what comes back, so a passing eval means the agent booted, accepted a request, and produced the result you asserted.
9
-
10
- Define [tool stubs](./tool-stubs) in your eval to control tool results, then assert on the agent's tool calls and responses.
8
+ Evals talk to a real agent server over the same HTTP API as your users, through the [TypeScript client](../guides/client/overview). A passing eval means the agent started, accepted a request, and produced the asserted result. To control what tools return, define [tool stubs](./tool-stubs) in the eval.
11
9
 
12
10
  ## `defineEval`
13
11
 
14
- eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is one eval by default. A file can also default-export an array to fan out over a dataset (see [Cases](./cases)). The file path is the eval's identity, so you don't author an `id` or `name`. Directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` becomes id `weather/brooklyn-forecast`).
12
+ The `evals/` directory sits at the app root. Each `.eval.ts` file exports one eval, or an array of evals that runs once per [dataset](./cases#datasets-exporting-an-array) row. The file path is the eval's id, so `evals/weather/brooklyn-forecast.eval.ts` becomes `weather/brooklyn-forecast`. Don't add an `id` or `name`.
15
13
 
16
14
  ```text
17
15
  my-agent/
@@ -25,7 +23,7 @@ my-agent/
25
23
  └── package.json
26
24
  ```
27
25
 
28
- An eval is a single `async test(t)` function. You drive the agent with `t` and assert on the run with the same `t`:
26
+ An eval is one `async test(t)` function. It sends messages and records assertions through `t`:
29
27
 
30
28
  ```ts title="evals/weather/brooklyn-forecast.eval.ts"
31
29
  import { defineEval } from "eve/evals";
@@ -42,11 +40,11 @@ export default defineEval({
42
40
  });
43
41
  ```
44
42
 
45
- `test` is the only required field. The rest are optional: `description`, `judge`, `tags`, `metadata`, `timeoutMs`, and `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
43
+ `test` is the only required field; `description`, `judge`, `tags`, `metadata`, `timeoutMs`, and `reporters` are optional. The init template adds `evals/**/*.ts` to `tsconfig.json`, so eval code type-checks with the app.
46
44
 
47
45
  ## `evals.config.ts`
48
46
 
49
- Every `evals/` directory needs exactly one `evals.config.ts` at its root. It declares the defaults every eval shares:
47
+ Every `evals/` directory needs exactly one `evals.config.ts` at its root. It sets the defaults that all evals share:
50
48
 
51
49
  ```ts title="evals/evals.config.ts"
52
50
  import { defineEvalConfig } from "eve/evals";
@@ -58,11 +56,22 @@ export default defineEvalConfig({
58
56
  });
59
57
  ```
60
58
 
61
- Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge(...)`); omit it to use the default decider, `typesafe-ai/jev`. Deterministic evals make no judge requests. `reporters`, `maxConcurrency`, `timeoutMs`, and `repeat` round out the defaults. Config `reporters` observe every eval in the run, so set one `braintrustReporter()` here instead of adding it to each eval. `repeat` runs every eval several times to measure repeatability; see [Repeat evals](./running#repeat-evals). CLI flags (`--max-concurrency`, `--timeout`, `--repeat`) and per-eval values take precedence over the config defaults.
59
+ Every field is optional:
60
+
61
+ | Field | Sets |
62
+ | ------------------- | ---------------------------------------------------------------------------------------------------- |
63
+ | `judge` | The model for [LLM-as-judge](./judge) assertions (`t.judge(...)`). Omit it to use `typesafe-ai/jev`. |
64
+ | `reporters` | Reporters that observe every eval in the run, so one `braintrustReporter()` here covers all evals. |
65
+ | `maxConcurrency` | The concurrent eval limit. |
66
+ | `timeoutMs` | The per-eval timeout. |
67
+ | `repeat` | How many times each eval runs, to measure repeatability. See [Repeat evals](./running#repeat-evals). |
68
+ | `setup`, `teardown` | Run-wide hooks before and after the run. See [Run setup and teardown](#run-setup-and-teardown). |
69
+
70
+ CLI flags (`--max-concurrency`, `--timeout`, `--repeat`) and per-eval values take precedence over config defaults.
62
71
 
63
72
  ### Run setup and teardown
64
73
 
65
- Use `setup` to start shared resources or run migrations before eve starts the local agent server. Return shared context directly from `setup`, then release resources in `teardown`:
74
+ Use `setup` to start shared resources or run migrations before eve starts the local agent server. Return shared values from `setup`, and release resources in `teardown`:
66
75
 
67
76
  ```ts title="evals/evals.config.ts"
68
77
  import { defineEvalConfig } from "eve/evals";
@@ -79,7 +88,7 @@ export default defineEvalConfig({
79
88
  });
80
89
  ```
81
90
 
82
- `context` is inferred from the setup return value. To use that type in an eval file, pass the config's type to `defineEval`:
91
+ `context` is typed from the setup return value. To use that type in an eval file, pass the config's type to `defineEval`:
83
92
 
84
93
  ```ts title="evals/database.eval.ts"
85
94
  import { defineEval } from "eve/evals";
@@ -94,18 +103,15 @@ export default defineEval<typeof config>({
94
103
  });
95
104
  ```
96
105
 
97
- Setup, eval bodies, and teardown share the same context object in the runner process. Class instances and functions keep their identity; context is neither serialized nor sent to the agent server. Concurrent evals share these resources, so isolate any database records they modify.
98
-
99
- `setup` can also return nothing.
106
+ `setup`, every eval, and `teardown` share one context object in the `eve eval` process. The agent server never sees it, so class instances and functions work as values. Evals that run at the same time share these resources, so isolate any database records they change.
100
107
 
101
- - Setup runs once per `eve eval` invocation, after discovery and config loading, before target startup.
102
- - Teardown runs after the local server and its sandbox handles shut down, including after setup, startup, or eval failures. Cleanup failures fail the command.
108
+ `setup` runs once per `eve eval` command, before the agent server starts. `teardown` runs after the local server and its sandboxes stop, even when setup, startup, or an eval fails. A `teardown` failure fails the command. The per-eval `timeoutMs` doesn't apply to either hook.
103
109
 
104
- `teardown` receives `undefined` when setup returns no context or throws before returning. In that case, setup must clean up resources that only it can access. Teardown can also be used without setup. The per-eval `timeoutMs` does not apply to setup or teardown.
110
+ `setup` can return nothing, and `teardown` works without `setup`. `teardown` receives `undefined` when `setup` returns nothing or throws, so `setup` must clean up anything only it can reach.
105
111
 
106
112
  ## Deterministic fixture models
107
113
 
108
- Use `mockModel` when an eval fixture needs to exercise eve's runtime without calling a model provider. A static fixture can be one line:
114
+ `mockModel` gives a test agent fixed replies, so it runs without calling a model provider. A fixed reply is one line:
109
115
 
110
116
  ```ts title="agent/agent.ts"
111
117
  import { defineAgent } from "eve";
@@ -116,7 +122,7 @@ export default defineAgent({
116
122
  });
117
123
  ```
118
124
 
119
- Pass a callback when the reply depends on the conversation. The callback receives an eve-owned view of the prompt, including `lastUserMessage`, `userMessages`, `userMessageCount`, available `tools`, and prior `toolResults`:
125
+ Pass a callback when the reply depends on the conversation. It receives `lastUserMessage`, `userMessages`, `userMessageCount`, the available `tools`, and prior `toolResults`:
120
126
 
121
127
  ```ts title="agent/agent.ts"
122
128
  export default defineAgent({
@@ -126,7 +132,7 @@ export default defineAgent({
126
132
  });
127
133
  ```
128
134
 
129
- The callback may return `{ text, toolCalls, usage }` for deterministic tool loops or explicit token counts. Use the options form only when a fixture also needs a custom model identity:
135
+ The callback can return `{ text, toolCalls, usage }` for deterministic tool loops or explicit token counts. Use the options form when a fixture also needs a custom model identity:
130
136
 
131
137
  ```ts title="agent/agent.ts"
132
138
  model: mockModel({
@@ -139,32 +145,32 @@ model: mockModel({
139
145
  });
140
146
  ```
141
147
 
142
- `mockModel()` uses `"Mock response"` when no response is supplied. It handles both generated and streamed responses, derives deterministic response metadata, and estimates token usage. It compiles with a 1,000,000-token context window, so compaction stays out of the way; set `modelContextWindowTokens` on the agent to exercise compaction with a smaller window. Because the model is part of the agent definition, use it for a dedicated fixture agent; it remains mocked whether that fixture runs locally or as a deployed eval target.
148
+ Omit the response to reply `"Mock response"`. `mockModel` supports streamed and non-streamed replies and estimates token usage. Its context window is 1,000,000 tokens, so eve never compacts the conversation. To test compaction, set `modelContextWindowTokens` on the agent.
149
+
150
+ The model is part of the agent definition, so use `mockModel` in a separate test agent. That agent stays mocked locally and when you deploy it as an eval target.
143
151
 
144
152
  ## The `t` context
145
153
 
146
- `t` is both the driver and the assertion surface. There are no separate `input`, `run`, `checks`, or `scores` fields. You write ordinary control flow, sending turns and asserting inline.
154
+ `t` both sends messages and records assertions, so an eval is ordinary code that sends a message, then asserts on the reply.
147
155
 
148
- - **Drive** the agent: `t.session()` creates an empty session; `t.send(...)` creates a session with its first message and returns a turn. Continue through `turn.session.send(...)`. Session handles also expose `start`, `cancel`, `respond`, `respondAll`, `sendFile`, and `requireInteraction`. Read output from `turn.reply`, and conversation state from `turn.session`. See [Cases](./cases).
149
- - **Assert** with three surfaces, covered next.
156
+ - **Send:** `t.send(...)` starts a conversation with its first message and returns the turn. `t.session()` starts an empty conversation. Send follow-ups with `turn.session.send(...)`, and read the reply from `turn.reply`. The full list of session methods is in [the drive API](./cases#the-drive-api).
157
+ - **Assert:** use the three surfaces in the next section.
150
158
 
151
159
  ## Three assertion surfaces
152
160
 
153
- Each surface matches a genuinely different kind of judgment:
161
+ | Surface | Grades |
162
+ | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
163
+ | Methods such as `t.calledTool("get_weather")` | What the agent did: on `t` for the whole run, or on one session or turn. See [Assertions](./assertions). |
164
+ | `t.check(value, assertion)` and `t.score(…)` | Any value, such as `turn.reply` or parsed JSON, or a 0–1 score you compute. See [Assertions](./assertions#scores-with-tscore). |
165
+ | `t.judge("cites a source")` | Quality, graded by a separate judge model. See [Judge](./judge). |
154
166
 
155
- - **Scoped methods** read the final whole run on `t`, snapshot one independent session when invoked there, or inspect one immutable `EveEvalTurn`. See [Assertions](./assertions).
156
- - **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(turn.reply, includes("sunny"))`. Grade `turn.reply`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
157
- - **`t.score(evaluation)`** records a score, such as a length ratio or an evaluator's result. Chain `.label(key)` to name it, and `.gate(threshold)` or `.atLeast(threshold)` to let it decide the eval. See [Assertions](./assertions#scores-with-tscore).
158
- - **`t.judge(...)`** is the LLM-as-judge surface, like `t.judge("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).
167
+ `t.judge` grades the reply of the most recently finished turn. Pass `{ on: turn.session.transcript }` to grade a whole conversation.
159
168
 
160
169
  ## Gate vs soft
161
170
 
162
- Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.
163
-
164
- - **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
165
- - **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity`, every `t.judge(...)` assertion, and every `t.score(...)` are soft by default. A soft assertion with no threshold records no verdict and never fails.
171
+ A failed **gate** marks the eval `failed`, and `eve eval` exits non-zero. A **soft** assertion appears in reports. A soft assertion below its threshold marks the eval `scored`, which fails the run only under `--strict`. A soft assertion with no threshold never fails.
166
172
 
167
- Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.
173
+ `t.judge(...)`, `t.score(...)`, and `similarity` are soft by default; the rest are gates. Chain `.gate(threshold?)`, `.soft(threshold?)`, or `.atLeast(threshold)` to change one. See [Severity](./assertions#severity).
168
174
 
169
175
  ```ts
170
176
  t.succeeded(); // gate
@@ -175,7 +181,7 @@ t.judge("The response is consistent with output.reference.", {
175
181
  }).atLeast(0.7); // soft, gated under --strict at 0.7
176
182
  ```
177
183
 
178
- Use `await t.require(value, assertion)` for a gate that must pass before the script can safely continue. Use `t.skip(reason)` as the first operation for an intentionally unsupported target capability.
184
+ Use `await t.require(value, assertion)` for a gate that must pass before the eval continues. Call `t.skip(reason)` first to skip an eval on a target that doesn't support it.
179
185
 
180
186
  ## Run evals with eve eval
181
187
 
@@ -189,12 +195,10 @@ Exit code `0` means every eval passed its gates. See [Running evals](./running)
189
195
 
190
196
  ## A good baseline
191
197
 
192
- Most apps do fine with a few small smoke evals. Assert behavior with `t.succeeded()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and reach for a judge or Braintrust only when you need fuzzy grading or shared result review. In CI, run `eve eval --strict` so soft threshold misses fail the build too.
198
+ Start with a few small smoke evals: `t.succeeded()` plus one or two content checks, with dataset files in `evals/data/`. Add a judge or Braintrust when you need fuzzy grading or a shared place to review results. In CI, run `eve eval --strict` so soft threshold misses fail the build.
193
199
 
194
200
  ## What to read next
195
201
 
196
- The rest of this section covers each piece:
197
-
198
202
  - [Cases](./cases): single-turn evals, scripted multi-turn evals, and dataset fan-out
199
203
  - [Assertions](./assertions): run-level methods and `t.check` value assertions, with matchers and severity
200
204
  - [Judge](./judge): LLM-as-judge grading and the judge model
@@ -3,11 +3,11 @@ title: "Reporters"
3
3
  description: "Ship eval results to Braintrust, Datadog, or JUnit XML. eve runs and scores everything itself."
4
4
  ---
5
5
 
6
- eve runs and grades everything itself; reporters ship the results out. The CLI prints a console summary by default (one line per eval, with failed assertions and their messages), and reporters from `eve/evals/reporters` add destinations on top.
6
+ A reporter sends the results eve graded to Braintrust, Datadog, a JUnit file, or your own destination. Import reporters from `eve/evals/reporters`. The CLI always prints a summary with one line per eval and each failed assertion's message.
7
7
 
8
8
  You are responsible for ensuring any observability or eval provider is approved for the data exported to it.
9
9
 
10
- Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run, the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
10
+ Declare reporters in `evals.config.ts` to report every eval in the run, such as to one shared Braintrust experiment. List them in an eval's `reporters` to report only the evals that share that reporter instance.
11
11
 
12
12
  ## Console
13
13
 
@@ -32,7 +32,7 @@ duplicates the CLI's output.
32
32
 
33
33
  ## Braintrust
34
34
 
35
- `braintrustReporter(...)` uploads eval results to Braintrust experiments. Put one instance in the config so it covers the whole run:
35
+ `braintrustReporter(...)` uploads eval results to Braintrust experiments. Put one instance in the config to cover the whole run:
36
36
 
37
37
  ```ts title="evals/evals.config.ts"
38
38
  import { defineEvalConfig } from "eve/evals";
@@ -44,7 +44,7 @@ export default defineEvalConfig({
44
44
  });
45
45
  ```
46
46
 
47
- Need a destination for only some evals? Attach it per eval instead:
47
+ To report only some evals, attach the reporter per eval:
48
48
 
49
49
  ```ts title="evals/brooklyn-forecast.eval.ts"
50
50
  import { defineEval } from "eve/evals";
@@ -59,15 +59,21 @@ export default defineEval({
59
59
  });
60
60
  ```
61
61
 
62
- The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. The console warns under the eval when merged soft scores differ; use `.label(name)` to group or separate them. Errored assertions log no score. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`. In a [repeated run](./running#repeat-evals), each attempt logs its own row with the id `<id>#<attempt>` and the attempt number under `eveAttempt`.
62
+ The reporter takes optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against.
63
63
 
64
- A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
64
+ - Gate assertions log as scores of 0 or 1 under a `gate:` prefix, so experiment diffs show gate regressions like soft-score regressions.
65
+ - Assertions that share a name, such as several unlabeled judges, log the lowest score. The console warns when merged soft scores differ; use `.label(name)` to group or separate them.
66
+ - Errored assertions log no score.
67
+ - Metadata holds assertion scores under `eveAssertionScores`, failed assertion details under `eveFailedAssertions`, and the eval's traces under `eveTraceIds` and `eveTraceContexts`.
68
+ - In a [repeated run](./running#repeat-evals), each attempt logs its own row with the id `<id>#<attempt>` and the attempt number under `eveAttempt`.
65
69
 
66
- Braintrust needs its SDK installed in the app and credentials in the environment: install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run the eval without shipping results, which also suppresses config reporters and is useful locally when iterating.
70
+ Evals that share one reporter instance, through the config, a `shared.ts` export, or a dataset array, land in one experiment. Listing the config reporter on an eval too doesn't report it twice.
71
+
72
+ Install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run without shipping results, including from config reporters.
67
73
 
68
74
  ## Datadog
69
75
 
70
- `datadogReporter(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config so it covers the whole run:
76
+ `datadogReporter(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config to cover the whole run:
71
77
 
72
78
  ```ts title="evals/evals.config.ts"
73
79
  import { defineEvalConfig } from "eve/evals";
@@ -78,29 +84,44 @@ export default defineEvalConfig({
78
84
  });
79
85
  ```
80
86
 
81
- The reporter creates one Datadog Experiment, submits one synthetic experiment span per completed eve eval, associates that eval's assertion scores as experiment metrics, and prints the Datadog Experiment URL after the run completes. Without input recording, the external Experiment uses a placeholder dataset and spans are submitted as evals finish. With `recordInputs` (the default), the reporter instead syncs a versioned Datadog dataset after the evals finish, starts the Experiment against that dataset version, passes each dataset record id to its corresponding experiment span, and prints both Dataset and Experiment URLs. Buffering is necessary because an imperative eval's first `t.send(...)` input is only known after that eval executes.
87
+ The reporter creates one Datadog Experiment per run and prints its URL afterward. Each completed eval becomes one experiment span, with its assertion scores as metrics.
88
+
89
+ With `recordInputs` (the default), the reporter syncs a versioned Datadog dataset after the evals finish and runs the Experiment against that version. Each span links to its dataset record, and the reporter prints the Dataset URL. With `recordInputs: false`, the Experiment uses a placeholder dataset, and spans are submitted as evals finish.
90
+
91
+ When an eve session's trace is sampled, the reporter adds `experimentRuntimeTraceLinks` metadata, so an Experiment row links to the agent trace that produced it.
92
+
93
+ Every run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records match by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description. In a [repeated run](./running#repeat-evals), every attempt links to the eval's one dataset record, and each span's `runIteration` is the attempt number.
82
94
 
83
- `dd-trace` generates dataset record, experiment trace, and experiment span identifiers. For sampled eve sessions, the reporter also adds identifier-only `experimentRuntimeTraceLinks` metadata, holding the IDs Datadog assigns when it indexes the runtime traces, so an Experiment row can link to the agent trace that produced it. Runtime spans keep their stored IDs and parentage.
95
+ Metric names come from assertion names:
84
96
 
85
- Every run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records are matched by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description. In a [repeated run](./running#repeat-evals), every attempt of an eval links to that eval's single dataset record, and each experiment span's `runIteration` is the attempt number.
97
+ - Characters other than letters, numbers, underscores, and hyphens become underscores.
98
+ - Gates get a `gate_` prefix.
99
+ - Assertions that share a name report the lowest score.
100
+ - Names that collide with built-in metrics get a numeric suffix.
86
101
 
87
- Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, assertions that share a label report one metric with the lowest score, and labels that collide with built-in metrics receive a numeric suffix. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Each assertion's name and score are kept in row metadata under `eveAssertionScores`; pass `recordAssertionDetails: false` to omit them and failed assertion messages.
102
+ For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Use `.label("stable name")` to choose a stable metric name. Assertion names can contain authored expectations, so review them before exporting sensitive evals. Row metadata keeps each assertion's name and score under `eveAssertionScores`; pass `recordAssertionDetails: false` to omit them and failed assertion messages.
88
103
 
89
- Datadog needs `dd-trace` installed in the app and Datadog credentials in the environment. The reporter is tested against `dd-trace@6.13.0`; install it with `npm install dd-trace@6.13.0` and set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate; without credentials, reporting is skipped. Like the Braintrust reporter, it records eval prompts, outputs, authored expectations, and execution error messages by default. Pass `recordInputs: false`, `recordOutputs: false`, `recordExpectedOutputs: false`, or `recordErrors: false` if your destination is not approved for that content. `recordInputs` stores the eval input in both the linked dataset record and experiment span. Input is read from the eval's first `t.send(...)` message, falling back to the eval description when no message event is available. When `recordExpectedOutputs` is enabled, expected output is read from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` and stored in both the dataset record and experiment span; when no expected output is authored, the input-only dataset record is still valid. Those expected-output keys are excluded from the general metadata payload. Target URL credentials, paths, query parameters, and fragments are never reported.
104
+ Install `dd-trace` (tested against `dd-trace@6.13.0`: `npm install dd-trace@6.13.0`). Set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate. Without credentials, reporting is skipped.
105
+
106
+ Like the Braintrust reporter, it records eval prompts, outputs, authored expectations, and execution error messages by default. Pass `recordInputs: false`, `recordOutputs: false`, `recordExpectedOutputs: false`, or `recordErrors: false` if your destination isn't approved for that content.
107
+
108
+ - Input is the eval's first `t.send(...)` message, or the eval description when there is none. It is stored in the dataset record and the experiment span.
109
+ - Expected output comes from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output`, and is stored in the dataset record and the experiment span. Those keys are excluded from the general metadata. A dataset record without expected output is still valid.
110
+ - Target URL credentials, paths, query parameters, and fragments are never reported.
90
111
 
91
112
  ## JUnit
92
113
 
93
- `junitReporter({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, usually the better fit because CI owns the output path, not the eval:
114
+ `junitReporter({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same without touching eval files. The flag usually fits better because CI owns the output path:
94
115
 
95
116
  ```bash
96
117
  eve eval --strict --junit .eve/junit.xml
97
118
  ```
98
119
 
99
- Each eval becomes one `<testcase>` named by its path-derived id (one per attempt, named `<id>#<attempt>`, in a repeated run); failed gates and execution errors become failures, while `t.skip(reason)` produces a JUnit `<skipped>` result.
120
+ Each eval becomes one `<testcase>` named by its id, or one per attempt named `<id>#<attempt>` in a repeated run. Failed gates and execution errors become failures, and `t.skip(reason)` becomes `<skipped>`.
100
121
 
101
122
  ## Custom reporters
102
123
 
103
- A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. Every callback may return a promise for async work like a remote upload:
124
+ Build a custom reporter only when no built-in reporter covers the destination. It implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same results as the built-ins. Every callback can return a promise:
104
125
 
105
126
  ```ts
106
127
  interface EvalReporter {
@@ -112,25 +133,31 @@ interface EvalReporter {
112
133
  }
113
134
  ```
114
135
 
115
- `onRunStart` fires once before any eval runs, and `onRunComplete` fires once with the aggregated summary. Within that run:
136
+ | Callback | Fires |
137
+ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
138
+ | `onRunStart` | Once, before any eval runs. |
139
+ | `onEvalStart` | When an eval is scheduled, with the eval definition, target, and start time. No trace exists yet. |
140
+ | `onSessionStart` | Once per session, when its first trace context arrives, with `sessionId`, `primary`, and `traceContext` (`traceId`, `spanId`, `traceFlags`). Never fires without tracing. |
141
+ | `onEvalComplete` | With the checks, scores, and verdict. `context` lists every distinct trace context across the eval's sessions, as does `result.result.traceContexts`. |
142
+ | `onRunComplete` | Once, with the aggregated summary. |
143
+
144
+ An eval's callbacks stay in that order even when evals run concurrently. In a [repeated run](./running#repeat-evals), `onEvalStart` fires once per attempt. Its event and each `onSessionStart` event carry the 1-based `attempt`, which matches `result.attempt`.
116
145
 
117
- - `onEvalStart` fires when an eval is scheduled. It includes the eval definition, target, and start time. In a [repeated run](./running#repeat-evals), it fires once per attempt, and the event and each `onSessionStart` event carry the 1-based `attempt`, which matches `result.attempt` on completion.
118
- - `onSessionStart` fires once for each session after eve receives its first trace context. It includes `sessionId`, `primary`, and `traceContext` with `traceId`, `spanId`, and `traceFlags`.
119
- - `onEvalComplete` fires with the checks, scores, and verdict. The runner also supplies `context`, including every distinct trace context collected across the eval's sessions. The same list is available as `result.result.traceContexts`.
146
+ `onEvalComplete` gets a list of trace contexts because an eval can create several sessions, and a long session can produce several traces. The list is empty without tracing.
120
147
 
121
- Each entry in `result.assertions` is an `AssertionResult`. `name` is the display label, `key` the identifier set by `.label(key)`, and `score` the measurement. `threshold` and `passed` describe the rule and are absent when no rule applies. When a scorer threw, `errored` is `true`, `message` carries the error, and `score` is absent. The `.eve/evals/` artifacts write the same fields.
148
+ Each entry in `result.assertions` is an `AssertionResult`. `name` is the display label, `key` the identifier set by `.label(key)`, and `score` the measurement. `threshold` and `passed` describe the rule and are absent when no rule applies. When a scorer threw, `errored` is `true`, `message` carries the error, and `score` is absent. The `.eve/evals/` artifacts write the same fields, plus the trace contexts.
122
149
 
123
- An eval can create several sessions, and a long session can produce several traces, so completion exposes a list instead of a single trace id. `onEvalStart` does not include a trace because the agent session has not started yet. If the target has no tracing configured, `onSessionStart` does not fire and the completed trace list is empty.
150
+ `result.result.derived` summarizes what happened in the eval's sessions, such as tool calls and `models`. `models` lists the distinct ids of the models the agent called, in session order. Each entry in `result.result.sessions` holds the same summary for one session, with `models` in first-use order. A reporter can use it to tell the primary session's models from other sessions'.
124
151
 
125
- `result.result.derived` holds the facts the runner derived from the eval's sessions, such as tool calls and `models`: the distinct ids of the models the steps started with, in session order. Each entry in `result.result.sessions` holds the same facts for one session, in first-use order, so a reporter can tell the primary session's models from those of the other sessions the eval created or attached. A delegated subagent runs in its own session, so its models appear only when the eval attaches and reads that session with `t.target.attach(...).nextTurn()` or `watch()`. Compaction summaries are included; provider-internal routing without an eve model lifecycle is not.
152
+ A subagent runs in its own session. Its models appear only when the eval attaches and reads that session with `t.target.attach(...).nextTurn()` or `watch()`. `models` includes the models that wrote compaction summaries, but not routing calls made inside a provider.
126
153
 
127
- `result.result.reply` is the last observed turn's text; `data` is its structured output, without a text fallback. Its `status` describes the request, while `sessionEnded` and the optional `view` describe that turn's session. The top-level `sessionId` still identifies the primary captured session; inspect `sessions` for each session's events and view. When selecting one value for a reporter, fall back to `reply` only when `data === undefined`, so structured `null` remains an output.
154
+ `result.result.reply` is the last turn's reply text, and `data` is its structured output, with no text fallback. `status` describes the request, while `sessionEnded` and the optional `view` describe that turn's session. The top-level `sessionId` identifies the primary session; inspect `sessions` for each session's events and view. To pick one output value, fall back to `reply` only when `data === undefined`, so structured `null` stays an output.
128
155
 
129
- Braintrust and Datadog metadata use `eveStatus`, `eveAwaitingInput`, and `eveSessionEnded`. Update saved filters that used `eveParked` to use `eveAwaitingInput`, and use `eveSessionEnded` to distinguish request completion from session retirement. Existing records are not rewritten.
156
+ Braintrust and Datadog metadata use `eveStatus`, `eveAwaitingInput`, and `eveSessionEnded`. Use `eveSessionEnded` to tell a finished request from an ended session.
130
157
 
131
- `derived.usage` is the token usage and cost a session recorded, summed from its `usage.recorded` facts, including what its delegated agents spent; on a turn, it is the session's total so far. The eval's `usage` counts each captured session once, using the snapshot with the latest recorded session activity rather than the last capture order, and leaves out sessions another captured session opened, because that session already counts their spend. It is absent when a counted session recorded no usage. It includes compaction summaries.
158
+ `derived.usage` is the token usage and cost a session recorded in its `usage.recorded` events. It includes subagents' spend and compaction summaries. On a turn, it is the session's total so far.
132
159
 
133
- Reporter callbacks for an eval stay ordered—eval start, traced session starts, then completion—even when several evals run concurrently. Reach for a custom reporter only when a destination isn't covered. The per-run artifacts under `.eve/evals/` retain the trace contexts alongside the result for ad-hoc inspection.
160
+ The eval's `usage` counts each session the eval read once, using its latest total. It skips a session that another session in the eval opened, because the opening session's total already includes it. It is absent when a counted session recorded no usage.
134
161
 
135
162
  ## What to read next
136
163
 
@@ -3,7 +3,7 @@ title: "Running Evals"
3
3
  description: "The eve eval CLI: flags, filters, exit codes, artifacts, and how to wire evals into CI."
4
4
  ---
5
5
 
6
- `eve eval` discovers every `.eval.ts` file under `evals/`, boots a local dev server (or targets a remote one), runs the evals concurrently, and prints a per-eval summary.
6
+ `eve eval` runs every `.eval.ts` file under `evals/` concurrently and prints a per-eval summary. It boots a local dev server, or targets a remote one with `--url`.
7
7
 
8
8
  ```bash
9
9
  eve eval # run all discovered evals locally
@@ -25,7 +25,7 @@ eve eval --keep-state # keep the local server's state and Workflow data
25
25
 
26
26
  Positional ids match exactly or by directory prefix: `eve eval weather` runs `evals/weather.eval.ts`, every eval under `evals/weather/`, and every entry of an array-exported `weather.eval.ts`.
27
27
 
28
- Tag filters compose: `--tag` keeps evals carrying at least one listed tag, then `--exclude-tag` drops any eval carrying an excluded tag. A `--tag` filter that matches nothing is a configuration error (exit `2`), but a run where `--exclude-tag` removes every match succeeds with nothing executed — exclusion expresses "this suite does not apply here."
28
+ Tag filters compose: `--tag` keeps evals carrying at least one listed tag, then `--exclude-tag` drops any eval carrying an excluded tag. A `--tag` filter that matches nothing is a configuration error (exit `2`). A run where `--exclude-tag` removes every match succeeds with nothing executed.
29
29
 
30
30
  ## Exit codes
31
31
 
@@ -37,13 +37,13 @@ Tag filters compose: `--tag` keeps evals carrying at least one listed tag, then
37
37
 
38
38
  An eval that calls `t.skip(reason)` is reported as skipped, does not count as passed or failed, and never changes the exit code.
39
39
 
40
- `eve eval --json` flushes the complete report before exiting, including when stdout is piped to another process or redirected to a file.
40
+ `eve eval --json` writes the complete report before exiting, including when stdout is piped to another process or redirected to a file.
41
41
 
42
- With `--verbose`, eve prefixes each line with its eval ID. It streams authored `t.log()` lines as they occur, then prints the workflow run ID for every captured primary and secondary session. Use those `wrun_...` IDs to find the corresponding runs in your workflow provider.
42
+ With `--verbose`, eve prefixes each line with its eval ID and prints `t.log()` lines as they happen. It then prints the `wrun_...` workflow run ID for every session the eval read, so you can find the runs in your workflow provider.
43
43
 
44
44
  ## Repeat evals
45
45
 
46
- Model-backed evals can pass on one run and fail on the next. To measure how often an eval passes, pass `--repeat <n>` or set `repeat` in `evals.config.ts`, and eve runs every selected eval `n` times in one run. The CLI flag overrides the config value. The default is `1`, and output does not change at that value.
46
+ Evals that call a real model can pass on one run and fail on the next. To measure how often an eval passes, pass `--repeat <n>` or set `repeat` in `evals.config.ts`. eve then runs every selected eval `n` times. The CLI flag overrides the config value. The default is `1`.
47
47
 
48
48
  ```bash
49
49
  eve eval weather --repeat 5
@@ -55,15 +55,22 @@ With `n` greater than `1`, each attempt is a separate result with a 1-based `att
55
55
  - `--json` results, JUnit test cases (named `<id>#<attempt>`), and [reporters](./reporters) receive one result per attempt.
56
56
  - Every attempt counts toward the run totals and the exit code, so one failed attempt fails the run.
57
57
 
58
- Attempts share the `--max-concurrency` pool with every other eval, so attempts of the same eval can run at the same time. If an eval depends on shared state or a rate-limited provider, lower `--max-concurrency` so contention does not show up as flakiness.
58
+ Attempts share the `--max-concurrency` limit with every other eval, so attempts of one eval can run at the same time. If an eval depends on shared state or a rate-limited provider, lower `--max-concurrency` so contention doesn't look like flakiness.
59
59
 
60
60
  ## Artifacts
61
61
 
62
- Each run drops artifacts under `.eve/evals/<timestamp>/`: a run `summary.json`, a `results.jsonl` index, and per-eval assertion results, verdicts, start and completion times, captured event streams with the client's send and arrival times for each session, and `t.log` lines under `evals/`. In a repeated run, each attempt's detail and event files get an `.attempt-<n>` suffix, such as `evals/weather.attempt-2.json`. The console output stays tight on purpose; when an eval fails, the artifact has the full story.
62
+ Each run writes artifacts under `.eve/evals/<timestamp>/`:
63
+
64
+ - `summary.json` summarizes the run.
65
+ - `results.jsonl` indexes the results, one line per eval.
66
+ - `evals/` holds each eval's assertion results, verdict, start and completion times, and `t.log` lines.
67
+ - `evals/` also holds each session's events, with when the client sent each request and received each event.
68
+
69
+ In a repeated run, each attempt's detail and event files get an `.attempt-<n>` suffix, such as `evals/weather.attempt-2.json`. When an eval fails, check its artifact for details the console omits.
63
70
 
64
71
  ## CI
65
72
 
66
- A solid CI invocation is strict and machine-reportable:
73
+ In CI, run evals strict and machine-reportable:
67
74
 
68
75
  ```bash
69
76
  eve eval --strict --junit .eve/junit.xml
@@ -3,9 +3,9 @@ title: "Targets"
3
3
  description: "Point evals at a local dev server or a deployment with the same eval files."
4
4
  ---
5
5
 
6
- An eval target is always an HTTP URL. `eve eval` starts a local dev server, while `eve eval --url <url>` runs against an existing server or deployment. The same eval files work for both, which is what makes evals usable as end-to-end tests in CI.
6
+ An eval target is the URL of the agent server an eval runs against. `eve eval` starts a local dev server, and `eve eval --url <url>` targets a running server or deployment. The same eval files work for both, so evals also serve as end-to-end tests in CI.
7
7
 
8
- The runner polls `/eve/v1/health`, verifies `/eve/v1/info`, and exposes the live target as `t.target` inside the `test` function.
8
+ Before running evals, `eve eval` waits for `/eve/v1/health` and checks `/eve/v1/info`. Inside `test`, `t.target` is the target.
9
9
 
10
10
  ## Target helpers
11
11
 
@@ -23,40 +23,37 @@ export default defineEval({
23
23
  });
24
24
  ```
25
25
 
26
- - `t.target.fetch(path, init)` performs an authenticated fetch against the target, useful for channel and webhook ingress. See [Authentication](#authentication) for how the runner authenticates.
27
- - `t.target.dispatchSchedule(id)` triggers a [schedule](../schedules) through the dev-only schedule route and returns the session ids it created. It works only against a target with dev routes enabled (the local `eve eval` dev server, or a deployment running in development mode), and throws otherwise.
28
- - `t.target.attach(sessionId, { startIndex? })` returns a handle without I/O. Its `id` is the exact session ID, and `startIndex` sets the initial stream cursor. An unknown or inaccessible session fails when you read it, not when you attach it.
29
- - `session.nextTurn()` reads and records the next turn boundary. Its events feed both the session's assertions and the run-level assertions. Each read advances the same session cursor.
30
- - `session.watch()` starts consuming immediately and returns a live-turn handle. Use `waitForEvent(...)` to coordinate with mid-turn work, `cancel()` to request cancellation, and `result()` to read through the boundary. Pass `{ until }` to stop at the commit containing a matching event, such as `context.settled` for a context change between turns.
26
+ - `t.target.fetch(path, init)` sends an [authenticated](#authentication) request to the target, to test channel and webhook routes.
27
+ - `t.target.dispatchSchedule(id)` triggers a [schedule](../schedules) through the dev-only schedule route and returns the session ids it created. It throws unless the target has dev routes enabled. The local `eve eval` dev server and a deployment in development mode enable them.
28
+ - `t.target.attach(sessionId, { startIndex? })` returns a handle for an existing session without making a request. `startIndex` sets where reading starts. An unknown or inaccessible session fails when you read it, not when you attach it.
29
+ - `session.nextTurn()` reads and records the session's next turn. Its events feed both the session's assertions and the run-level assertions. Each read continues where the last one stopped.
30
+ - `session.watch()` starts reading right away and returns a [live turn](./cases#in-flight-turns) with `waitForEvent(...)`, `cancel()`, and `result()`. Pass `{ until }` to stop at a matching event instead of the end of a turn, such as `context.settled` for a context change between turns.
31
31
 
32
- Sessions attached this way are full `EveEvalSession`s: you can keep driving them and assert directly on that session (`session.succeeded()`, `session.calledTool(...)`). Aggregate assertions on `t` continue to read the whole run, including every attached session.
32
+ Attached sessions are full `EveEvalSession`s. Send them messages and assert on them directly, as in `session.succeeded()` or `session.calledTool(...)`. Assertions on `t` cover the whole run, including every attached session.
33
33
 
34
- For session creation and message delivery, use the [drive API](./cases#the-drive-api).
35
- `t.session()` creates a session without a first message; call `session.send()` or `session.start()`
36
- on the returned handle to use it. `t.send()` creates a new session with its first message.
37
- These helpers handle the session protocol, retries, and event collection for the eval.
34
+ To create sessions and send messages, use the [drive API](./cases#the-drive-api).
38
35
 
39
36
  ## Authentication
40
37
 
41
- Local targets send no auth: `eve eval` owns the dev server it boots.
38
+ Requests to a local target carry no credentials, because `eve eval` started that server.
42
39
 
43
- For a remote `--url`, eve gets the expected Vercel owner and project from `VERCEL_ORG_ID` and `VERCEL_PROJECT_ID` when both values are nonblank. Otherwise it reads the local Vercel project link: `.vercel/project.json`, or the matching project in `.vercel/repo.json` for a linked monorepo.
40
+ For a remote `--url`, eve finds the expected Vercel team and project from `VERCEL_ORG_ID` and `VERCEL_PROJECT_ID` when both are nonblank. Otherwise it reads the local Vercel project link: `.vercel/project.json`, or the matching project in `.vercel/repo.json` for a linked monorepo.
44
41
 
45
- With a project reference, eve looks up the exact HTTPS origin under that owner, regardless of the Vercel CLI's current team. The deployment's project ID must match before eve sends ambient credentials. Without a project reference, eve verifies the URL under the CLI's current scope. URLs that Vercel cannot verify remain anonymous.
42
+ With a project, eve looks up the exact HTTPS origin under that team, regardless of the Vercel CLI's current team. It sends your Vercel credentials only when the deployment's project ID matches. Without a project, eve checks the URL under the Vercel CLI's current team. A URL that Vercel can't verify gets no credentials.
46
43
 
47
44
  After verification, eve sends the available Vercel credentials:
48
45
 
49
46
  - The resolved OIDC token as both the bearer and Vercel trusted-IDP header.
50
47
  - `VERCEL_AUTOMATION_BYPASS_SECRET`, when set, as the Protection Bypass for Automation header.
51
48
 
52
- `EVE_EVAL_AUTH_TOKEN` is an explicit bearer override for targets whose auth is not Vercel OIDC. Credential-bearing clients do not follow redirects, so those headers cannot be forwarded to another origin.
49
+ Set `EVE_EVAL_AUTH_TOKEN` to send that bearer token instead, for targets that don't use Vercel OIDC. Requests that carry credentials don't follow redirects, so the credentials never reach another origin.
53
50
 
54
- `/eve/v1/info` serves only operator principals, so the credential must authenticate as a service principal, for example through Vercel OIDC, `jwtHmac()`, `jwtEcdsa()`, or a custom `AuthFn` that returns `principalType: "service"`. When the target answers `403`, the run fails before any eval starts and names the credential it needs.
51
+ `/eve/v1/info` serves only operator callers, so the credential must authenticate with `principalType: "service"`. Vercel OIDC, `jwtHmac()`, `jwtEcdsa()`, or a custom `AuthFn` can provide it. When the target answers `403`, the run fails before any eval starts and names the credential it needs.
55
52
 
56
- `t.target.fetch(path, init)` carries these same credentials, so channel and webhook ingress you exercise through it authenticates the same way the session protocol does.
53
+ `t.target.fetch(path, init)` sends these same credentials.
57
54
 
58
55
  ## What to read next
59
56
 
60
57
  - [Running evals](./running): `--url` and the rest of the CLI in practice
61
58
  - [Schedules](../schedules): the surface `dispatchSchedule` drives
62
- - [Channels](../channels/overview): ingress you can exercise with `target.fetch`
59
+ - [Channels](../channels/overview): routes you can test with `target.fetch`