eve 0.90.0 → 0.91.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -23
- package/dist/src/channel/auth/http-basic.js +1 -1
- package/dist/src/channel/channel-operations.d.ts +1 -1
- package/dist/src/channel/types.d.ts +2 -0
- package/dist/src/cli/dev/tui/prompt-commands.d.ts +1 -1
- package/dist/src/client/session.d.ts +1 -1
- package/dist/src/compiled/.vendor-stamp.json +1 -1
- package/dist/src/execution/dynamic-workflow/tool.js +1 -1
- package/dist/src/execution/proxied-deliver-step.js +1 -1
- package/dist/src/execution/sandbox/bash.js +2 -2
- package/dist/src/execution/session/active-turn.js +1 -1
- package/dist/src/execution/session/admission.d.ts +6 -0
- package/dist/src/execution/session/admission.js +1 -1
- package/dist/src/execution/session/delivery-facts.js +1 -1
- package/dist/src/execution/session/next-input.d.ts +2 -2
- package/dist/src/execution/session/next-input.js +1 -1
- package/dist/src/execution/session/program.js +1 -1
- package/dist/src/execution/session-inbox/inbox.d.ts +2 -2
- package/dist/src/execution/session-inbox/inbox.js +1 -1
- package/dist/src/execution/session-inbox/protocol.js +1 -1
- package/dist/src/execution/tools/agent-router-workflow.js +1 -1
- package/dist/src/harness/compaction/engine.js +2 -2
- package/dist/src/harness/current-messages.js +1 -1
- package/dist/src/harness/emission.d.ts +3 -0
- package/dist/src/harness/emission.js +1 -1
- package/dist/src/harness/facts.d.ts +5 -2
- package/dist/src/harness/facts.js +1 -1
- package/dist/src/harness/model-call/call.d.ts +7 -1
- package/dist/src/harness/model-call/call.js +3 -1
- package/dist/src/harness/provider-tool-history.d.ts +8 -0
- package/dist/src/harness/provider-tool-history.js +1 -1
- package/dist/src/harness/session-machine/transitions.d.ts +3 -2
- package/dist/src/harness/step/after-model.js +1 -1
- package/dist/src/harness/step/context.js +1 -1
- package/dist/src/harness/step/intake.js +1 -1
- package/dist/src/harness/step/prompt.js +1 -1
- package/dist/src/harness/tool-history-repair.d.ts +2 -0
- package/dist/src/harness/tool-history-repair.js +1 -1
- package/dist/src/harness/tool-loop.js +1 -1
- package/dist/src/internal/application/stamped-package-version.js +1 -1
- package/dist/src/protocol/session-events/families/delivery.d.ts +5 -3
- package/dist/src/protocol/session-events/families/delivery.js +1 -1
- package/dist/src/protocol/session-projection/fold.js +1 -1
- package/dist/src/public/channels/discord/hitl.js +1 -1
- package/dist/src/public/channels/slack/hitl.d.ts +2 -1
- package/dist/src/public/channels/slack/hitl.js +3 -3
- package/dist/src/public/channels/slack/interaction-cards.js +1 -1
- package/dist/src/public/channels/slack/request-card-render.js +1 -1
- package/dist/src/public/channels/teams/hitl.js +1 -1
- package/dist/src/public/channels/teams/verify.js +1 -1
- package/dist/src/public/definitions/hook.d.ts +90 -7
- package/dist/src/public/definitions/hook.js +1 -1
- package/dist/src/public/hooks/index.d.ts +3 -2
- package/dist/src/reactions/continuation.d.ts +19 -0
- package/dist/src/reactions/continuation.js +1 -0
- package/dist/src/reactions/intents.d.ts +10 -4
- package/dist/src/reactions/intents.js +1 -1
- package/dist/src/reactions/kinds/hook.js +1 -1
- package/dist/src/reactions/runner.js +1 -1
- package/dist/src/reactions/state.d.ts +15 -2
- package/dist/src/reactions/state.js +1 -1
- package/dist/src/runtime/resolve-connection.js +1 -1
- package/dist/src/runtime/subagents/workflow.js +1 -1
- package/dist/src/setup/primitives/open-url.js +1 -1
- package/dist/src/setup/scaffold/create/project.js +1 -1
- package/dist/src/shared/value-delta.d.ts +5 -4
- package/dist/src/shared/value-delta.js +1 -1
- package/dist/src/tools/provided/ask-question-workflow.js +1 -1
- package/dist/src/tools/provided/sleep-workflow.js +1 -1
- package/dist/src/tools/provided/web-fetch.js +1 -1
- package/docs/agent-config.md +199 -222
- package/docs/channels/chat-sdk.mdx +60 -64
- package/docs/channels/custom.mdx +264 -281
- package/docs/channels/discord.mdx +176 -123
- package/docs/channels/eve.mdx +136 -206
- package/docs/channels/github.mdx +24 -22
- package/docs/channels/linear.mdx +30 -30
- package/docs/channels/linq.mdx +25 -13
- package/docs/channels/mcp.mdx +154 -116
- package/docs/channels/overview.mdx +31 -32
- package/docs/channels/photon.mdx +32 -56
- package/docs/channels/slack.mdx +538 -375
- package/docs/channels/teams.mdx +40 -26
- package/docs/channels/telegram.mdx +13 -13
- package/docs/channels/twilio.mdx +11 -11
- package/docs/code-extension.mdx +40 -42
- package/docs/concepts/built-in-tools.md +89 -57
- package/docs/concepts/context-control.md +20 -20
- package/docs/concepts/default-harness.md +20 -36
- package/docs/concepts/execution-model-and-durability.mdx +96 -160
- package/docs/concepts/project-structure.mdx +21 -14
- package/docs/concepts/security-model.md +27 -27
- package/docs/concepts/sessions-runs-and-streaming.md +184 -198
- package/docs/concepts/state.md +7 -15
- package/docs/connections/mcp.mdx +109 -93
- package/docs/connections/openapi.mdx +57 -59
- package/docs/connections/overview.mdx +175 -170
- package/docs/evals/assertions.mdx +36 -29
- package/docs/evals/cases.mdx +38 -42
- package/docs/evals/judge.mdx +21 -29
- package/docs/evals/overview.mdx +42 -38
- package/docs/evals/reporters.mdx +54 -27
- package/docs/evals/running.mdx +15 -8
- package/docs/evals/targets.mdx +16 -19
- package/docs/evals/tool-stubs.mdx +111 -97
- package/docs/experimental/dynamic-schedules.mdx +118 -63
- package/docs/extensions.md +58 -65
- package/docs/getting-started.mdx +4 -6
- package/docs/guides/auth-and-route-protection.md +87 -70
- package/docs/guides/client/continuations.mdx +23 -26
- package/docs/guides/client/messages.mdx +47 -66
- package/docs/guides/client/output-schema.mdx +8 -8
- package/docs/guides/client/overview.mdx +45 -43
- package/docs/guides/client/streaming.mdx +62 -71
- package/docs/guides/decide.md +24 -79
- package/docs/guides/deployment/meta.json +1 -1
- package/docs/guides/deployment/overview.md +27 -18
- package/docs/guides/deployment/self-hosted-operations.md +99 -0
- package/docs/guides/deployment/self-hosting.md +63 -80
- package/docs/guides/deployment/vercel.mdx +31 -27
- package/docs/guides/deployment/workflow-worlds.md +100 -0
- package/docs/guides/dev-tui.md +48 -41
- package/docs/guides/dynamic-capabilities.md +49 -50
- package/docs/guides/frontend/nextjs.mdx +2 -2
- package/docs/guides/frontend/nuxt.mdx +11 -11
- package/docs/guides/frontend/overview.mdx +1 -1
- package/docs/guides/frontend/sveltekit.mdx +11 -11
- package/docs/guides/frontend/tanstack.mdx +16 -16
- package/docs/guides/frontend/use-eve-agent-svelte.mdx +21 -21
- package/docs/guides/frontend/use-eve-agent-vue.mdx +22 -22
- package/docs/guides/frontend/use-eve-agent.mdx +25 -25
- package/docs/guides/hooks.md +152 -106
- package/docs/guides/instrumentation/agent-runs.mdx +26 -30
- package/docs/guides/instrumentation/instrumentation.mdx +49 -76
- package/docs/guides/instrumentation/meta.json +1 -1
- package/docs/guides/instrumentation/otel.mdx +113 -179
- package/docs/guides/remote-agents.md +125 -187
- package/docs/guides/self-modification.md +18 -18
- package/docs/guides/session-context.md +16 -16
- package/docs/install-integrations.mdx +26 -24
- package/docs/instructions.mdx +37 -42
- package/docs/memory/custom-provider.md +35 -41
- package/docs/memory/file.md +93 -76
- package/docs/memory/overview.mdx +145 -156
- package/docs/patterns/durable-cross-channel-notifications.md +15 -19
- package/docs/patterns/dynamic-scheduling.md +19 -23
- package/docs/patterns/multi-tenant-approvals.md +17 -22
- package/docs/patterns/multi-tenant-auth.md +43 -42
- package/docs/patterns/multi-tenant-memory.md +39 -36
- package/docs/patterns/voice-notes.md +3 -3
- package/docs/protocols/acp.md +13 -13
- package/docs/protocols/ucp.mdx +7 -7
- package/docs/reference/agent-files.md +6 -6
- package/docs/reference/cli.md +99 -98
- package/docs/reference/telemetry.md +1 -1
- package/docs/reference/typescript-api.md +52 -53
- package/docs/responsible-use.md +2 -2
- package/docs/sandbox/custom-providers.mdx +60 -56
- package/docs/sandbox/default.mdx +23 -24
- package/docs/sandbox/docker.mdx +16 -10
- package/docs/sandbox/index.mdx +93 -61
- package/docs/sandbox/just-bash.mdx +17 -11
- package/docs/sandbox/troubleshooting.mdx +29 -23
- package/docs/sandbox/vercel.mdx +45 -52
- package/docs/schedules.mdx +65 -54
- package/docs/skills.mdx +74 -49
- package/docs/subagents/index.mdx +97 -136
- package/docs/tools/human-in-the-loop.md +160 -90
- package/docs/tools/meta.json +1 -1
- package/docs/tools/overview.mdx +229 -237
- package/docs/tools/tasks.md +118 -117
- package/docs/tools/workflows.mdx +447 -543
- package/docs/tutorial/connect-a-warehouse.mdx +11 -13
- package/docs/tutorial/first-agent.mdx +14 -16
- package/docs/tutorial/guard-the-spend.mdx +6 -8
- package/docs/tutorial/how-it-runs.mdx +8 -8
- package/docs/tutorial/query-sample-data.mdx +8 -8
- package/docs/tutorial/remember-definitions.mdx +4 -4
- package/docs/tutorial/run-analysis.mdx +20 -22
- package/docs/tutorial/ship-it.mdx +19 -19
- package/docs/tutorial/team-playbooks.mdx +9 -9
- package/docs/use-cases.mdx +13 -11
- package/package.json +1 -1
- package/docs/guides/instrumentation/migration.md +0 -96
- package/docs/tools/tasks-upgrade.md +0 -218
|
@@ -3,11 +3,11 @@ title: "Assertions"
|
|
|
3
3
|
description: "Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
An assertion grades what an eval's `test(t)` function produced. Each assertion records a result instead of throwing, so one run reports every failing assertion, not only the first. Check what the agent did with scoped methods, grade a value with `t.check`, or record your own measurement with `t.score`. For grading by a model, see [Judge](./judge).
|
|
7
7
|
|
|
8
8
|
## Scoped assertions
|
|
9
9
|
|
|
10
|
-
Scoped assertions take no
|
|
10
|
+
Scoped assertions take no value and are gates by default. Call them on `t` to check the whole run after `test` finishes. Call them on a session to check that session so far, or on a turn to check one reply. A run is **parked** when it stopped to wait on an approval or question that nobody answered (human-in-the-loop, or HITL, input).
|
|
11
11
|
|
|
12
12
|
| Assertion | Asserts |
|
|
13
13
|
| ----------------------------------------------------- | ----------------------------------------------------------------------------- |
|
|
@@ -17,7 +17,7 @@ Scoped assertions take no explicit value and gate by default. Assertions on `t`
|
|
|
17
17
|
| `turn.outputEquals(value)` / `.outputMatches(schema)` | Deep equality or Standard Schema validation of turn/session structured output |
|
|
18
18
|
| `t.calledTool(name, opts?)` | A matching tool call completed (`input`, `output`, `status`, `count`) |
|
|
19
19
|
| `t.calledSkill(skill, opts?)` | A load of `skill` completed (`output`, `status`, `count`) |
|
|
20
|
-
| `t.notCalledTool(name)` | No
|
|
20
|
+
| `t.notCalledTool(name)` | No call to `name`, whatever its status |
|
|
21
21
|
| `t.toolOrder([...names])` | Tool requests appear in order |
|
|
22
22
|
| `t.usedNoTools()` | No tool calls at all; skill loads don't count |
|
|
23
23
|
| `t.maxToolCalls(n)` | At most `n` tool calls; skill loads don't count |
|
|
@@ -27,7 +27,7 @@ Scoped assertions take no explicit value and gate by default. Assertions on `t`
|
|
|
27
27
|
| `t.eventOrder([...matchers])` | Matching event groups occur in order |
|
|
28
28
|
| `t.eventsSatisfy(label, predicate)` | Escape hatch: any predicate over the typed event stream |
|
|
29
29
|
|
|
30
|
-
`succeeded()`
|
|
30
|
+
`succeeded()` passes when the session ended or is waiting for the next message. It fails when the run failed, an event reports an error, or the run is parked. Structured output assertions exist only on turns and sessions, where one output is unambiguous (see the [output schema guide](../guides/client/output-schema)).
|
|
31
31
|
|
|
32
32
|
```ts
|
|
33
33
|
await t.send("What is the weather in Brooklyn?");
|
|
@@ -35,7 +35,7 @@ t.succeeded();
|
|
|
35
35
|
t.calledTool("get_weather");
|
|
36
36
|
```
|
|
37
37
|
|
|
38
|
-
The same
|
|
38
|
+
The same methods work on a single turn or on an attached session:
|
|
39
39
|
|
|
40
40
|
```ts
|
|
41
41
|
const first = await t.send("Call get_weather for Brooklyn");
|
|
@@ -47,13 +47,15 @@ attached.succeeded();
|
|
|
47
47
|
attached.messageIncludes("Sunny");
|
|
48
48
|
```
|
|
49
49
|
|
|
50
|
-
`t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other
|
|
50
|
+
`t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other in a run.
|
|
51
51
|
|
|
52
|
-
A call through [`eve__tool`](../concepts/built-in-tools#eve__search-eve__tool-and-eve__skill) is reported under the name and input of the
|
|
52
|
+
A call through [`eve__tool`](../concepts/built-in-tools#eve__search-eve__tool-and-eve__skill) is reported under the name and input of the tool it reaches, such as `linear__list_issues`. Assert on that tool. `eve__search` calls count toward `t.usedNoTools()` and `t.maxToolCalls(n)`; skill loads don't.
|
|
53
|
+
|
|
54
|
+
The tool-name assertions and `turn.requireToolCall` throw a `TypeError` for `eve__tool`, `eve__skill`, and built-in names such as `load_skill`. The error says what to assert instead. eve accepts a name like `load_skill` when the root agent declares a tool by that name or has a dynamic tool resolver.
|
|
53
55
|
|
|
54
56
|
## Value assertions with `t.check`
|
|
55
57
|
|
|
56
|
-
`t.check(value, assertion)` grades
|
|
58
|
+
`t.check(value, assertion)` grades any value, such as a turn's `.reply` or parsed JSON, with a builder from `eve/evals/expect`:
|
|
57
59
|
|
|
58
60
|
```ts
|
|
59
61
|
import { includes, equals, matches, satisfies, similarity } from "eve/evals/expect";
|
|
@@ -82,11 +84,11 @@ For example, a failed labeled equality check prints:
|
|
|
82
84
|
✗ equals [status] (0% < 100%): expected {"status":"active"}; received {"status":"disabled"}
|
|
83
85
|
```
|
|
84
86
|
|
|
85
|
-
Pick the cheapest builder that captures
|
|
87
|
+
Pick the cheapest builder that captures "correct". Use `similarity` when an exact match is too strict, and the [judge](./judge) for nuanced grading.
|
|
86
88
|
|
|
87
89
|
## Scores with `t.score`
|
|
88
90
|
|
|
89
|
-
`t.score(evaluation)` records a score from 0 to 1.
|
|
91
|
+
`t.score(evaluation)` records a score from 0 to 1. Pass a number, `{ score, message?, metadata? }`, or a promise of either. Chain `.label(key)` to name it; reporters and artifacts show the label as `key`. Unlabeled scores all report as `score`.
|
|
90
92
|
|
|
91
93
|
```ts
|
|
92
94
|
const reply = turn.reply ?? "";
|
|
@@ -97,16 +99,16 @@ t.score({
|
|
|
97
99
|
t.score(gradeFaithfulness(reply, reference)).label("faithfulness"); // async
|
|
98
100
|
```
|
|
99
101
|
|
|
100
|
-
|
|
102
|
+
A score without a rule doesn't affect the eval's outcome. Chain a rule when it should:
|
|
101
103
|
|
|
102
104
|
```ts
|
|
103
105
|
t.score(gradeFaithfulness(reply, reference)).label("faithfulness").gate(0.9); // fails below 0.9
|
|
104
106
|
t.score(brevity).label("reply-brevity").atLeast(0.5); // marks the eval `scored`, fatal under --strict
|
|
105
107
|
```
|
|
106
108
|
|
|
107
|
-
|
|
109
|
+
Reports show the measured score: a `0.82` gated at `0.9` reports `score: 0.82`, `threshold: 0.9`, `passed: false`. An evaluator that throws or returns a score outside 0 to 1 fails the eval. The result is marked errored, with the error message and no score.
|
|
108
110
|
|
|
109
|
-
|
|
111
|
+
[Reporters](./reporters#custom-reporters) receive the entries in `onEvalComplete` and can read them by `key`:
|
|
110
112
|
|
|
111
113
|
```ts
|
|
112
114
|
onEvalComplete(result) {
|
|
@@ -117,7 +119,7 @@ onEvalComplete(result) {
|
|
|
117
119
|
}
|
|
118
120
|
```
|
|
119
121
|
|
|
120
|
-
The same entries
|
|
122
|
+
The same entries appear in the [run artifacts](./running#artifacts) under `.eve/evals/`, in `summary.json` and in one `results.jsonl` line per eval:
|
|
121
123
|
|
|
122
124
|
```json
|
|
123
125
|
{
|
|
@@ -133,11 +135,16 @@ The same entries land in the [run artifacts](./running#artifacts) under `.eve/ev
|
|
|
133
135
|
|
|
134
136
|
## The matcher mini-language
|
|
135
137
|
|
|
136
|
-
`t.calledTool` and `t.calledAgent` take matcher objects
|
|
138
|
+
`t.calledTool` and `t.calledAgent` take matcher objects:
|
|
139
|
+
|
|
140
|
+
| Matcher | Fields | `status` values |
|
|
141
|
+
| ------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
|
|
142
|
+
| `calledTool` | `input`, `output`, `status`, `count` | `"completed"` (default), `"pending"`, `"failed"`, `"rejected"` |
|
|
143
|
+
| `calledAgent` | `callId`, `childSessionId`, `output`, `status`, `count` | `"completed"` (default), `"pending"`, `"failed"`, `"rejected"`, `"interrupted"`, `"abandoned"` |
|
|
137
144
|
|
|
138
|
-
A
|
|
145
|
+
A numeric `count` requires that exact number of calls matching every supplied constraint. For ranges, pass a predicate that receives the number of matching calls.
|
|
139
146
|
|
|
140
|
-
|
|
147
|
+
`calledAgent` matches each call to a subagent. A call is `pending` until it finishes, then takes its final status and, when completed, its `output`. A completed subagent keeps its session for later calls with the same `taskId`. `childSessionId` is absent when the call failed before the subagent's session opened. A subagent's approval or question doesn't change the call's status; assert it with `requireInteraction`.
|
|
141
148
|
|
|
142
149
|
Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean:
|
|
143
150
|
|
|
@@ -156,7 +163,7 @@ t.calledAgent("weather", {
|
|
|
156
163
|
});
|
|
157
164
|
```
|
|
158
165
|
|
|
159
|
-
`requireInteraction` uses the same
|
|
166
|
+
`requireInteraction` finds an approval or question. It uses the same matchers for `input`, `prompt`, and `display`. Its `optionIds` matcher receives the option ids in order, and a literal array must match the whole list exactly:
|
|
160
167
|
|
|
161
168
|
```ts
|
|
162
169
|
const request = session.requireInteraction({
|
|
@@ -167,9 +174,9 @@ const request = session.requireInteraction({
|
|
|
167
174
|
|
|
168
175
|
## Run state and derived facts
|
|
169
176
|
|
|
170
|
-
|
|
177
|
+
Assertions read what happened from `session.events`: tool calls with their name, input, output, and status, subagent calls, and approvals and questions.
|
|
171
178
|
|
|
172
|
-
|
|
179
|
+
The event assertions match by presence, absence, a numeric or predicate count, partial event data, and order:
|
|
173
180
|
|
|
174
181
|
```ts
|
|
175
182
|
turn.notEvent("content.completed", { data: { kind: "result" } });
|
|
@@ -179,7 +186,7 @@ turn.eventOrder([
|
|
|
179
186
|
]);
|
|
180
187
|
```
|
|
181
188
|
|
|
182
|
-
|
|
189
|
+
To check a rule that spans several events, use `eventsSatisfy`:
|
|
183
190
|
|
|
184
191
|
```ts
|
|
185
192
|
t.eventsSatisfy("assistant reply includes the marker", (events) =>
|
|
@@ -195,7 +202,7 @@ t.eventsSatisfy("assistant reply includes the marker", (events) =>
|
|
|
195
202
|
|
|
196
203
|
### Assert and score reasoning
|
|
197
204
|
|
|
198
|
-
When the provider exposes reasoning, eve
|
|
205
|
+
When the model provider exposes reasoning, eve records each completed block as a `reasoning.completed` event. Assert on its text with `turn.event(...)`, or join the blocks and pass them to a judge:
|
|
199
206
|
|
|
200
207
|
```ts
|
|
201
208
|
turn.event("reasoning.completed", {
|
|
@@ -212,11 +219,11 @@ t.judge("The reasoning checks its assumptions before concluding.", {
|
|
|
212
219
|
}).atLeast(0.8);
|
|
213
220
|
```
|
|
214
221
|
|
|
215
|
-
A provider that
|
|
222
|
+
A provider that doesn't expose reasoning emits no `reasoning.completed` event, so the `turn.event` gate fails and the judge grades an empty string. The default judge input and `session.transcript` exclude reasoning, so pass it with `on`.
|
|
216
223
|
|
|
217
224
|
## Required preconditions
|
|
218
225
|
|
|
219
|
-
Recorded assertions
|
|
226
|
+
Recorded assertions aren't awaitable. When later code depends on a value, use `await t.require(value, assertion)`. It records a gate and returns the value when it passes. When it fails, it stops the test without reporting a second, duplicate error:
|
|
220
227
|
|
|
221
228
|
```ts
|
|
222
229
|
await t.require(
|
|
@@ -226,7 +233,7 @@ await t.require(
|
|
|
226
233
|
await t.target.attach(sessionIds[0]!).nextTurn();
|
|
227
234
|
```
|
|
228
235
|
|
|
229
|
-
Use the matching `require*` lookups when
|
|
236
|
+
Use the matching `require*` lookups when later code needs a tool call or a pending request:
|
|
230
237
|
|
|
231
238
|
```ts
|
|
232
239
|
const call = turn.requireToolCall("search");
|
|
@@ -235,14 +242,14 @@ const request = session.requireInteraction({ toolName: "guarded" });
|
|
|
235
242
|
|
|
236
243
|
## Severity
|
|
237
244
|
|
|
238
|
-
Every assertion returns a
|
|
245
|
+
Every assertion returns a handle. Chain one of these to set its severity:
|
|
239
246
|
|
|
240
247
|
- `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
|
|
241
|
-
- `.soft(threshold?)` is
|
|
248
|
+
- `.soft(threshold?)` is recorded in reports. A miss below the threshold marks the eval `scored`, which fails the run only under `--strict`. With no threshold, it never fails.
|
|
242
249
|
- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
|
|
243
|
-
- `.label(key)` adds a stable name when an eval records several assertions
|
|
250
|
+
- `.label(key)` adds a stable name when an eval records several assertions of the same kind. Braintrust and Datadog keep one score per name. `eve eval` warns when soft assertions in one eval share a name but have different scores.
|
|
244
251
|
|
|
245
|
-
|
|
252
|
+
Scoped methods and `includes`, `equals`, `matches`, and `satisfies` are gates; `similarity`, `t.judge(...)`, and `t.score(...)` are soft. Chain a severity only to change the default:
|
|
246
253
|
|
|
247
254
|
```ts
|
|
248
255
|
t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate
|
package/docs/evals/cases.mdx
CHANGED
|
@@ -3,9 +3,9 @@ title: "Cases"
|
|
|
3
3
|
description: "Author single-turn and multi-turn evals with test(t), and fan one file out over a dataset."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
A case is one `async test(t)` function that sends messages to the agent and records [assertions](./assertions) as it goes. A case can send one message or several, answer approvals, or run once per row of a [dataset](#datasets-exporting-an-array). Each eval file exports one case, or an array of cases.
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Every `evals/` directory needs an `evals.config.ts` at its root. An empty config works when you don't need shared settings:
|
|
9
9
|
|
|
10
10
|
```ts title="evals/evals.config.ts"
|
|
11
11
|
import { defineEvalConfig } from "eve/evals";
|
|
@@ -15,7 +15,7 @@ export default defineEvalConfig({});
|
|
|
15
15
|
|
|
16
16
|
## Single-turn evals
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
A single-turn eval sends one message and asserts on the reply. `t.send(input)` resolves when the agent finishes its turn, and the returned turn's `.reply` holds the reply:
|
|
19
19
|
|
|
20
20
|
```ts title="evals/weather/brooklyn-forecast.eval.ts"
|
|
21
21
|
import { defineEval } from "eve/evals";
|
|
@@ -30,7 +30,7 @@ export default defineEval({
|
|
|
30
30
|
});
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
|
|
33
|
+
To test behavior rather than text, assert on the run and skip the content check:
|
|
34
34
|
|
|
35
35
|
```ts title="evals/weather/no-tools-for-greetings.eval.ts"
|
|
36
36
|
import { defineEval } from "eve/evals";
|
|
@@ -46,7 +46,7 @@ export default defineEval({
|
|
|
46
46
|
|
|
47
47
|
## Organizing with directories
|
|
48
48
|
|
|
49
|
-
|
|
49
|
+
The file path is the eval's id, so directories group evals. `evals/weather/brooklyn-forecast.eval.ts` gets the id `weather/brooklyn-forecast`, and `eve eval weather` runs everything under `evals/weather/`. Put shared constants and helpers in sibling files whose names don't end in `.eval.ts`:
|
|
50
50
|
|
|
51
51
|
```text
|
|
52
52
|
evals/
|
|
@@ -60,7 +60,7 @@ evals/
|
|
|
60
60
|
|
|
61
61
|
## Multi-turn evals
|
|
62
62
|
|
|
63
|
-
|
|
63
|
+
Send several messages in order to test branching, approvals, structured output, attachments, or more than one conversation. Keep each turn in a variable and send the next message through its `.session`:
|
|
64
64
|
|
|
65
65
|
```ts title="evals/draft-then-send.eval.ts"
|
|
66
66
|
import { defineEval } from "eve/evals";
|
|
@@ -78,7 +78,7 @@ export default defineEval({
|
|
|
78
78
|
});
|
|
79
79
|
```
|
|
80
80
|
|
|
81
|
-
Use scoped assertions
|
|
81
|
+
Use scoped assertions on turns in the middle of the conversation. When later code depends on a value check, `t.require` records a gate and stops the eval if it fails:
|
|
82
82
|
|
|
83
83
|
```ts title="evals/session-continuity.eval.ts"
|
|
84
84
|
import { defineEval } from "eve/evals";
|
|
@@ -103,24 +103,32 @@ export default defineEval({
|
|
|
103
103
|
|
|
104
104
|
## The drive API
|
|
105
105
|
|
|
106
|
-
Every `t.session()` or `t.send()`
|
|
106
|
+
Every `t.session()` or `t.send()` call starts a new, separate conversation (a session). Send more messages through its session handle. The run-level assertions on `t` see the events from every session.
|
|
107
107
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
108
|
+
| Method | Does |
|
|
109
|
+
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
|
|
110
|
+
| `t.session(options?)` | Creates an empty session and resolves on `202 Accepted`, with `.id` and `.state` set |
|
|
111
|
+
| `t.send(message, options?)` | Creates a session with its first message and waits for the turn to finish |
|
|
112
|
+
| `session.send(message, options?)` | Sends a follow-up and waits for the turn to finish |
|
|
113
|
+
| `session.start(message, options?)` | Starts a turn and returns once the server accepts it; see [In-flight turns](#in-flight-turns) |
|
|
114
|
+
| `session.cancel()` | Asks the agent to stop and returns a `Delivery`; a no-op when idle |
|
|
115
|
+
| `session.compact()` | Compacts the conversation history between turns and resolves to a turn with the compaction's events |
|
|
116
|
+
| `session.sendFile(text, path, mediaType?)` | Sends a message with a local file attached as a data URL |
|
|
117
|
+
| `session.agent(opened).stream(options?)` | Follows a subagent's event stream, given a `child.opened` event from that session |
|
|
118
|
+
| `session.requireInteraction(filter?)` | Records a gate that exactly one approval or question is pending, and returns it |
|
|
119
|
+
| `session.respond(responses, options?)` | Answers specific pending requests as the next turn; `session.startRespond(...)` returns without waiting |
|
|
120
|
+
| `session.respondAll(optionId)` | Answers every pending request with the same option as the next turn |
|
|
121
|
+
| `session.events`, `session.transcript` | The session's recorded events, and its user and assistant messages as text |
|
|
120
122
|
|
|
121
|
-
|
|
123
|
+
`t.session()` and `t.send()` options accept per-request `headers` and `signal`. Omit `signal` to use the eval's timeout signal.
|
|
122
124
|
|
|
123
|
-
|
|
125
|
+
`session.cancel()` throws `SessionNotActiveError` when the session is no longer active, and `session.compact()` also throws. Assert on a compaction with `.event("context.settled", { data: { kind: "compaction", outcome: "completed" } })`.
|
|
126
|
+
|
|
127
|
+
`session.agent(...)` uses the eval's credentials, so unlike `t.target.attach(...).watch()`, it also reaches subagents on another deployment. `requireInteraction` filters match tool name, tool input, prompt, display, and option ids.
|
|
128
|
+
|
|
129
|
+
The transcript uses `User:` and `Assistant:` labels separated by blank lines. It leaves out reasoning, tool calls, tool results, and other sessions' messages. It updates when a turn finishes, so read it after `await session.send(...)`, `await session.respond(...)`, or `await live.result()`.
|
|
130
|
+
|
|
131
|
+
Each `send`, `respond`, and `respondAll` resolves to a turn, which doesn't change after it returns. The turn has `.reply`, `.data`, `.events`, `.interactions`, `.toolCalls`, `.session`, `.sessionId`, `.status`, and `.expectOk()`. `.session` is the conversation handle for sending more messages. `expectOk()` throws only when the turn failed.
|
|
124
132
|
|
|
125
133
|
## Prewarming a session
|
|
126
134
|
|
|
@@ -142,23 +150,11 @@ export default defineEval({
|
|
|
142
150
|
});
|
|
143
151
|
```
|
|
144
152
|
|
|
145
|
-
|
|
146
|
-
the TypeScript client handles bounded retries if the workflow inbox is still starting.
|
|
147
|
-
Creation failures reject `t.session()` without returning a handle. Each call creates an
|
|
148
|
-
independent session, including concurrent calls. Accepted sessions appear in eval results
|
|
149
|
-
and participate in timeout cleanup even when no message was sent.
|
|
150
|
-
|
|
151
|
-
## Migrating existing evals
|
|
152
|
-
|
|
153
|
-
Replace `t.newSession()` and `t.prewarm()` with `const session = await t.session()`.
|
|
154
|
-
Replace consecutive `t.send()` calls with `session.send()` calls, or continue the first
|
|
155
|
-
returned turn through `turn.session.send()`. Read identity, cursor, events, and transcript
|
|
156
|
-
from `session`; read replies from `turn.reply`. Session operations such as `respond()`,
|
|
157
|
-
`requireInteraction()`, and `cancel()` also move from `t` to `session`.
|
|
153
|
+
The agent sets up the session when the first message arrives. If eve can't create the session, `t.session()` rejects. A created session appears in eval results and is cleaned up on timeout, even when no message was sent.
|
|
158
154
|
|
|
159
155
|
## In-flight turns
|
|
160
156
|
|
|
161
|
-
Use `start()` when the eval must
|
|
157
|
+
Use `start()` when the eval must watch or change a turn before it finishes. The returned live turn has `.session`, `.sessionId`, `.waitForEvent(...)`, `.cancel()`, and `.result()`. `waitForEvent()` resolves with the first matching event, including one that arrived before the call. `result()` waits for the turn to end and returns it. Event data matchers use the same partial matching as `t.event(...)`.
|
|
162
158
|
|
|
163
159
|
```ts title="evals/cancel-running-tool.eval.ts"
|
|
164
160
|
import { defineEval } from "eve/evals";
|
|
@@ -179,19 +175,19 @@ export default defineEval({
|
|
|
179
175
|
});
|
|
180
176
|
```
|
|
181
177
|
|
|
182
|
-
`waitForEvent()` rejects if the stream fails or
|
|
178
|
+
`waitForEvent()` rejects if the stream fails or the turn ends before the event arrives. Call `result()` afterward so the run-level assertions see the turn's events.
|
|
183
179
|
|
|
184
|
-
Use `expectOk()` only when the next
|
|
180
|
+
Use `expectOk()` only when the next step depends on an earlier turn succeeding; a final `t.succeeded()` already gates the whole run.
|
|
185
181
|
|
|
186
|
-
To
|
|
182
|
+
To omit an eval for the current target, call `t.skip(reason)` before creating sessions, sending messages, or recording assertions. Skipped evals are reported separately and don't affect the exit code.
|
|
187
183
|
|
|
188
|
-
|
|
184
|
+
`t.log(message)` writes debug lines to the eval artifact, and `--verbose` also streams them to stdout. `t.abortSignal` fires on timeout.
|
|
189
185
|
|
|
190
186
|
For driving sessions created outside the eval, by a channel webhook or a schedule, see [Targets](./targets).
|
|
191
187
|
|
|
192
188
|
## Datasets: exporting an array
|
|
193
189
|
|
|
194
|
-
To
|
|
190
|
+
To run one file once per dataset row, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` works. Each id is the file name plus a zero-padded index: `sql/0000`, `sql/0001`, and so on. `loadJson` and `loadYaml` from `eve/evals/loaders` parse data files relative to the app root:
|
|
195
191
|
|
|
196
192
|
```ts title="evals/sql.eval.ts"
|
|
197
193
|
import { defineEval } from "eve/evals";
|
|
@@ -213,7 +209,7 @@ export default rows.map((row) =>
|
|
|
213
209
|
);
|
|
214
210
|
```
|
|
215
211
|
|
|
216
|
-
|
|
212
|
+
Use the loaders in evals, not in runtime agent code.
|
|
217
213
|
|
|
218
214
|
## What to read next
|
|
219
215
|
|
package/docs/evals/judge.mdx
CHANGED
|
@@ -3,7 +3,7 @@ title: "Judge"
|
|
|
3
3
|
description: "Grade evals with decision models using criteria, typed questions, or batches, and set thresholds on each assertion."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
`t.judge(...)` asks a separate judge model to grade your agent's response. Use it when a fixed [assertion](./assertions) can't describe a good response. It calls [`decide` from `eve/ai`](../guides/decide#decide-inside-a-tool) and defaults to `typesafe-ai/jev`, TypeSafe AI's [Jev decision model](https://vercel.com/i/what-is-jev).
|
|
7
7
|
|
|
8
8
|
```ts
|
|
9
9
|
import { defineEval } from "eve/evals";
|
|
@@ -17,7 +17,7 @@ export default defineEval({
|
|
|
17
17
|
});
|
|
18
18
|
```
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
The judge needs no configuration, only credentials for the judge model; see [Configuring the judge model](#configuring-the-judge-model).
|
|
21
21
|
|
|
22
22
|
## The graders
|
|
23
23
|
|
|
@@ -29,7 +29,7 @@ No judge configuration is required. Configure credentials for the decider as des
|
|
|
29
29
|
| `score` | The returned rubric position divided by the highest level index |
|
|
30
30
|
| `choice` | 1 when the selected option equals `expected`; otherwise 0 |
|
|
31
31
|
|
|
32
|
-
Boolean scores are model estimates, not
|
|
32
|
+
Boolean scores are model estimates, not calibrated confidence. A rubric needs at least two levels, ordered from worst to best; eve normalizes a fractional rubric position to the 0–1 threshold scale.
|
|
33
33
|
|
|
34
34
|
```ts
|
|
35
35
|
t.judge({
|
|
@@ -41,13 +41,13 @@ t.judge({
|
|
|
41
41
|
.atLeast(0.75);
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
Choice questions require an `expected` option key. eve uses
|
|
44
|
+
Choice questions require an `expected` option key. eve uses it for scoring and doesn't send it to the judge model. Instructions and rubric descriptions accept strings, JSON objects, or JSON arrays; descriptions can also be `null`.
|
|
45
45
|
|
|
46
46
|
## Choose the value to grade
|
|
47
47
|
|
|
48
|
-
By default, the
|
|
48
|
+
By default, the judge model receives `{ input, output }`: the user message and reply of the most recently finished turn. Answering an approval or reading an attached session keeps that user message. For a message with a file, such as from `session.sendFile()`, `input` is its text.
|
|
49
49
|
|
|
50
|
-
Pass `on` to replace the output. JSON values stay structured
|
|
50
|
+
Pass `on` to replace the output. JSON values stay structured:
|
|
51
51
|
|
|
52
52
|
```ts
|
|
53
53
|
const draft = await t.send("Draft the welcome email.");
|
|
@@ -66,7 +66,7 @@ t.judge("The assistant remembers the user's favorite word across turns.", {
|
|
|
66
66
|
|
|
67
67
|
The transcript contains user and assistant messages in turn order, excluding reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals).
|
|
68
68
|
|
|
69
|
-
`turn.reply` is the turn's
|
|
69
|
+
`turn.reply` is the turn's last reply. A turn can reply more than once, for example to say work has started before waiting on tasks. `session.transcript` includes every reply. To grade all of one turn's replies, filter its messages with `transcriptOf` from `eve/client`:
|
|
70
70
|
|
|
71
71
|
```ts
|
|
72
72
|
import { transcriptOf } from "eve/client";
|
|
@@ -81,7 +81,7 @@ t.judge("The agent reported the deploy's result.", { on: replies.join("\n\n") })
|
|
|
81
81
|
|
|
82
82
|
## Decide several questions together
|
|
83
83
|
|
|
84
|
-
A batch
|
|
84
|
+
A batch asks several questions in one request about the same value. Pass that value as `state` instead of `on`; it replaces the default `{ input, output }` entirely.
|
|
85
85
|
|
|
86
86
|
```ts
|
|
87
87
|
const judgments = t.judge({
|
|
@@ -106,25 +106,21 @@ judgments.accurate.atLeast(0.9);
|
|
|
106
106
|
judgments.outcome.gate();
|
|
107
107
|
```
|
|
108
108
|
|
|
109
|
-
|
|
109
|
+
Each key names a question and its assertion handle. Every question sees the same `state`, so use separate calls for unrelated values. A batch needs at least one question, and a provider or validation failure fails every assertion in it.
|
|
110
110
|
|
|
111
111
|
## Soft scoring and thresholds
|
|
112
112
|
|
|
113
113
|
Judges are soft by default:
|
|
114
114
|
|
|
115
|
-
- Without a threshold, the score is tracked in reports and artifacts without failing the eval.
|
|
116
|
-
- `.atLeast(threshold)` sets a soft bar. A lower score marks the eval `scored`, fatal only under `eve eval --strict`.
|
|
117
|
-
- `.gate(threshold)` makes a lower score fail the eval outright. An omitted gate threshold is 1.
|
|
118
|
-
|
|
119
115
|
```ts
|
|
120
|
-
t.judge("The response cites a source.");
|
|
121
|
-
t.judge("The response cites a source.").label("citation").atLeast(0.8);
|
|
122
|
-
t.judge("The response cites a source.").gate(0.9);
|
|
116
|
+
t.judge("The response cites a source."); // tracked, never fails
|
|
117
|
+
t.judge("The response cites a source.").label("citation").atLeast(0.8); // below 0.8 marks `scored`, fatal under --strict
|
|
118
|
+
t.judge("The response cites a source.").gate(0.9); // below 0.9 fails; omit the threshold to require 1
|
|
123
119
|
```
|
|
124
120
|
|
|
125
|
-
A boolean probability
|
|
121
|
+
A good response's boolean probability rarely reaches exactly 1, so pass an explicit gate threshold for boolean judgments. Choice scores are binary.
|
|
126
122
|
|
|
127
|
-
|
|
123
|
+
A judge call copies its input and starts right away. Handles aren't awaitable; eve waits for pending judgments after `test` returns. Each call outside a batch makes its own request. An eval timeout cancels pending judgments, even if the provider ignores cancellation.
|
|
128
124
|
|
|
129
125
|
## Configuring the judge model
|
|
130
126
|
|
|
@@ -135,7 +131,7 @@ Model settings resolve in this order:
|
|
|
135
131
|
3. Project default: `defineEvalConfig({ judge: { model, modelOptions } })`.
|
|
136
132
|
4. The default model used by `decide`, currently `typesafe-ai/jev`.
|
|
137
133
|
|
|
138
|
-
A per-eval `judge` replaces the project's judge configuration
|
|
134
|
+
A per-eval `judge` replaces the project's judge configuration, and per-call fields override both. Evals that never call `t.judge` make no judge requests.
|
|
139
135
|
|
|
140
136
|
```ts title="evals/evals.config.ts"
|
|
141
137
|
import { defineEvalConfig } from "eve/evals";
|
|
@@ -145,7 +141,7 @@ export default defineEvalConfig({
|
|
|
145
141
|
});
|
|
146
142
|
```
|
|
147
143
|
|
|
148
|
-
String IDs resolve through the AI SDK default provider, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY
|
|
144
|
+
String IDs resolve through the AI SDK default provider's decision API, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY`, Vercel OIDC credentials, or eve's local Gateway connection. Gateway accepts native decision models such as `typesafe-ai/jev`. A language model ID such as `openai/gpt-6-luna` doesn't select the OpenAI adapter. For a language model, pass a provider decision model, which uses that provider's credentials:
|
|
149
145
|
|
|
150
146
|
```ts
|
|
151
147
|
import { openai } from "@ai-sdk/openai";
|
|
@@ -156,19 +152,15 @@ t.judge("The response addresses the request.", {
|
|
|
156
152
|
});
|
|
157
153
|
```
|
|
158
154
|
|
|
159
|
-
Use the provider's `decisionModel` factory, not a language model instance.
|
|
160
|
-
|
|
161
|
-
String model IDs resolve through the default provider's decision API. Gateway accepts native decision models such as `typesafe-ai/jev`; passing a language model ID such as `openai/gpt-6-luna` does not automatically select the OpenAI adapter. Use a decision-model instance as shown above for language models.
|
|
162
|
-
|
|
163
|
-
Missing credentials, invalid answers, unsupported question types, and provider errors become failed gates even for tracked-only judgments. They do not silently skip the eval or select another model. Validation, authentication, and retries use the shared `decide` implementation.
|
|
155
|
+
Use the provider's `decisionModel` factory, not a language model instance. Provider support and the decision contract are experimental; see [AI SDK decision](https://ai-sdk.dev/docs/ai-sdk-core/decisions).
|
|
164
156
|
|
|
165
|
-
|
|
157
|
+
Missing credentials, invalid answers, unsupported question types, and provider errors fail as gates, even for soft judgments. eve never skips the eval or falls back to another model.
|
|
166
158
|
|
|
167
|
-
|
|
159
|
+
## Diagnostics
|
|
168
160
|
|
|
169
|
-
|
|
161
|
+
Reports name judge assertions `judge.boolean`, `judge.score`, and `judge.choice`, plus the question key in a batch. Failure details show the graded value, the question, and the raw answer. Assertion metadata holds the normalized score, model id, token usage, warnings, and any provider and response metadata. A batch's usage is marked as shared, not charged per question. Decision models don't promise a written rationale.
|
|
170
162
|
|
|
171
|
-
|
|
163
|
+
Set thresholds against your own examples. For factuality, summaries, or SQL equivalence, write an explicit question and include the reference in shared state. [Braintrust reporting](./reporters) accepts judge scores.
|
|
172
164
|
|
|
173
165
|
## What to read next
|
|
174
166
|
|