eve 0.90.0 → 0.91.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -23
- package/dist/src/channel/auth/http-basic.js +1 -1
- package/dist/src/channel/channel-operations.d.ts +1 -1
- package/dist/src/channel/types.d.ts +2 -0
- package/dist/src/cli/dev/tui/prompt-commands.d.ts +1 -1
- package/dist/src/client/session.d.ts +1 -1
- package/dist/src/compiled/.vendor-stamp.json +1 -1
- package/dist/src/execution/dynamic-workflow/tool.js +1 -1
- package/dist/src/execution/proxied-deliver-step.js +1 -1
- package/dist/src/execution/sandbox/bash.js +2 -2
- package/dist/src/execution/session/active-turn.js +1 -1
- package/dist/src/execution/session/admission.d.ts +6 -0
- package/dist/src/execution/session/admission.js +1 -1
- package/dist/src/execution/session/delivery-facts.js +1 -1
- package/dist/src/execution/session/next-input.d.ts +2 -2
- package/dist/src/execution/session/next-input.js +1 -1
- package/dist/src/execution/session/program.js +1 -1
- package/dist/src/execution/session-inbox/inbox.d.ts +2 -2
- package/dist/src/execution/session-inbox/inbox.js +1 -1
- package/dist/src/execution/session-inbox/protocol.js +1 -1
- package/dist/src/execution/tools/agent-router-workflow.js +1 -1
- package/dist/src/harness/compaction/engine.js +2 -2
- package/dist/src/harness/current-messages.js +1 -1
- package/dist/src/harness/emission.d.ts +3 -0
- package/dist/src/harness/emission.js +1 -1
- package/dist/src/harness/facts.d.ts +5 -2
- package/dist/src/harness/facts.js +1 -1
- package/dist/src/harness/model-call/call.d.ts +7 -1
- package/dist/src/harness/model-call/call.js +3 -1
- package/dist/src/harness/provider-tool-history.d.ts +8 -0
- package/dist/src/harness/provider-tool-history.js +1 -1
- package/dist/src/harness/session-machine/transitions.d.ts +3 -2
- package/dist/src/harness/step/after-model.js +1 -1
- package/dist/src/harness/step/context.js +1 -1
- package/dist/src/harness/step/intake.js +1 -1
- package/dist/src/harness/step/prompt.js +1 -1
- package/dist/src/harness/tool-history-repair.d.ts +2 -0
- package/dist/src/harness/tool-history-repair.js +1 -1
- package/dist/src/harness/tool-loop.js +1 -1
- package/dist/src/internal/application/stamped-package-version.js +1 -1
- package/dist/src/protocol/session-events/families/delivery.d.ts +5 -3
- package/dist/src/protocol/session-events/families/delivery.js +1 -1
- package/dist/src/protocol/session-projection/fold.js +1 -1
- package/dist/src/public/channels/discord/hitl.js +1 -1
- package/dist/src/public/channels/slack/hitl.d.ts +2 -1
- package/dist/src/public/channels/slack/hitl.js +3 -3
- package/dist/src/public/channels/slack/interaction-cards.js +1 -1
- package/dist/src/public/channels/slack/request-card-render.js +1 -1
- package/dist/src/public/channels/teams/hitl.js +1 -1
- package/dist/src/public/channels/teams/verify.js +1 -1
- package/dist/src/public/definitions/hook.d.ts +90 -7
- package/dist/src/public/definitions/hook.js +1 -1
- package/dist/src/public/hooks/index.d.ts +3 -2
- package/dist/src/reactions/continuation.d.ts +19 -0
- package/dist/src/reactions/continuation.js +1 -0
- package/dist/src/reactions/intents.d.ts +10 -4
- package/dist/src/reactions/intents.js +1 -1
- package/dist/src/reactions/kinds/hook.js +1 -1
- package/dist/src/reactions/runner.js +1 -1
- package/dist/src/reactions/state.d.ts +15 -2
- package/dist/src/reactions/state.js +1 -1
- package/dist/src/runtime/resolve-connection.js +1 -1
- package/dist/src/runtime/subagents/workflow.js +1 -1
- package/dist/src/setup/primitives/open-url.js +1 -1
- package/dist/src/setup/scaffold/create/project.js +1 -1
- package/dist/src/shared/value-delta.d.ts +5 -4
- package/dist/src/shared/value-delta.js +1 -1
- package/dist/src/tools/provided/ask-question-workflow.js +1 -1
- package/dist/src/tools/provided/sleep-workflow.js +1 -1
- package/dist/src/tools/provided/web-fetch.js +1 -1
- package/docs/agent-config.md +199 -222
- package/docs/channels/chat-sdk.mdx +60 -64
- package/docs/channels/custom.mdx +264 -281
- package/docs/channels/discord.mdx +176 -123
- package/docs/channels/eve.mdx +136 -206
- package/docs/channels/github.mdx +24 -22
- package/docs/channels/linear.mdx +30 -30
- package/docs/channels/linq.mdx +25 -13
- package/docs/channels/mcp.mdx +154 -116
- package/docs/channels/overview.mdx +31 -32
- package/docs/channels/photon.mdx +32 -56
- package/docs/channels/slack.mdx +538 -375
- package/docs/channels/teams.mdx +40 -26
- package/docs/channels/telegram.mdx +13 -13
- package/docs/channels/twilio.mdx +11 -11
- package/docs/code-extension.mdx +40 -42
- package/docs/concepts/built-in-tools.md +89 -57
- package/docs/concepts/context-control.md +20 -20
- package/docs/concepts/default-harness.md +20 -36
- package/docs/concepts/execution-model-and-durability.mdx +96 -160
- package/docs/concepts/project-structure.mdx +21 -14
- package/docs/concepts/security-model.md +27 -27
- package/docs/concepts/sessions-runs-and-streaming.md +184 -198
- package/docs/concepts/state.md +7 -15
- package/docs/connections/mcp.mdx +109 -93
- package/docs/connections/openapi.mdx +57 -59
- package/docs/connections/overview.mdx +175 -170
- package/docs/evals/assertions.mdx +36 -29
- package/docs/evals/cases.mdx +38 -42
- package/docs/evals/judge.mdx +21 -29
- package/docs/evals/overview.mdx +42 -38
- package/docs/evals/reporters.mdx +54 -27
- package/docs/evals/running.mdx +15 -8
- package/docs/evals/targets.mdx +16 -19
- package/docs/evals/tool-stubs.mdx +111 -97
- package/docs/experimental/dynamic-schedules.mdx +118 -63
- package/docs/extensions.md +58 -65
- package/docs/getting-started.mdx +4 -6
- package/docs/guides/auth-and-route-protection.md +87 -70
- package/docs/guides/client/continuations.mdx +23 -26
- package/docs/guides/client/messages.mdx +47 -66
- package/docs/guides/client/output-schema.mdx +8 -8
- package/docs/guides/client/overview.mdx +45 -43
- package/docs/guides/client/streaming.mdx +62 -71
- package/docs/guides/decide.md +24 -79
- package/docs/guides/deployment/meta.json +1 -1
- package/docs/guides/deployment/overview.md +27 -18
- package/docs/guides/deployment/self-hosted-operations.md +99 -0
- package/docs/guides/deployment/self-hosting.md +63 -80
- package/docs/guides/deployment/vercel.mdx +31 -27
- package/docs/guides/deployment/workflow-worlds.md +100 -0
- package/docs/guides/dev-tui.md +48 -41
- package/docs/guides/dynamic-capabilities.md +49 -50
- package/docs/guides/frontend/nextjs.mdx +2 -2
- package/docs/guides/frontend/nuxt.mdx +11 -11
- package/docs/guides/frontend/overview.mdx +1 -1
- package/docs/guides/frontend/sveltekit.mdx +11 -11
- package/docs/guides/frontend/tanstack.mdx +16 -16
- package/docs/guides/frontend/use-eve-agent-svelte.mdx +21 -21
- package/docs/guides/frontend/use-eve-agent-vue.mdx +22 -22
- package/docs/guides/frontend/use-eve-agent.mdx +25 -25
- package/docs/guides/hooks.md +152 -106
- package/docs/guides/instrumentation/agent-runs.mdx +26 -30
- package/docs/guides/instrumentation/instrumentation.mdx +49 -76
- package/docs/guides/instrumentation/meta.json +1 -1
- package/docs/guides/instrumentation/otel.mdx +113 -179
- package/docs/guides/remote-agents.md +125 -187
- package/docs/guides/self-modification.md +18 -18
- package/docs/guides/session-context.md +16 -16
- package/docs/install-integrations.mdx +26 -24
- package/docs/instructions.mdx +37 -42
- package/docs/memory/custom-provider.md +35 -41
- package/docs/memory/file.md +93 -76
- package/docs/memory/overview.mdx +145 -156
- package/docs/patterns/durable-cross-channel-notifications.md +15 -19
- package/docs/patterns/dynamic-scheduling.md +19 -23
- package/docs/patterns/multi-tenant-approvals.md +17 -22
- package/docs/patterns/multi-tenant-auth.md +43 -42
- package/docs/patterns/multi-tenant-memory.md +39 -36
- package/docs/patterns/voice-notes.md +3 -3
- package/docs/protocols/acp.md +13 -13
- package/docs/protocols/ucp.mdx +7 -7
- package/docs/reference/agent-files.md +6 -6
- package/docs/reference/cli.md +99 -98
- package/docs/reference/telemetry.md +1 -1
- package/docs/reference/typescript-api.md +52 -53
- package/docs/responsible-use.md +2 -2
- package/docs/sandbox/custom-providers.mdx +60 -56
- package/docs/sandbox/default.mdx +23 -24
- package/docs/sandbox/docker.mdx +16 -10
- package/docs/sandbox/index.mdx +93 -61
- package/docs/sandbox/just-bash.mdx +17 -11
- package/docs/sandbox/troubleshooting.mdx +29 -23
- package/docs/sandbox/vercel.mdx +45 -52
- package/docs/schedules.mdx +65 -54
- package/docs/skills.mdx +74 -49
- package/docs/subagents/index.mdx +97 -136
- package/docs/tools/human-in-the-loop.md +160 -90
- package/docs/tools/meta.json +1 -1
- package/docs/tools/overview.mdx +229 -237
- package/docs/tools/tasks.md +118 -117
- package/docs/tools/workflows.mdx +447 -543
- package/docs/tutorial/connect-a-warehouse.mdx +11 -13
- package/docs/tutorial/first-agent.mdx +14 -16
- package/docs/tutorial/guard-the-spend.mdx +6 -8
- package/docs/tutorial/how-it-runs.mdx +8 -8
- package/docs/tutorial/query-sample-data.mdx +8 -8
- package/docs/tutorial/remember-definitions.mdx +4 -4
- package/docs/tutorial/run-analysis.mdx +20 -22
- package/docs/tutorial/ship-it.mdx +19 -19
- package/docs/tutorial/team-playbooks.mdx +9 -9
- package/docs/use-cases.mdx +13 -11
- package/package.json +1 -1
- package/docs/guides/instrumentation/migration.md +0 -96
- package/docs/tools/tasks-upgrade.md +0 -218
package/docs/evals/overview.mdx
CHANGED
|
@@ -3,15 +3,13 @@ title: "Overview"
|
|
|
3
3
|
description: "Define repeatable scored checks for an eve agent with defineEval and run them with eve eval."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
An eval is a scored check that
|
|
6
|
+
An eval is a scored check that sends messages to your agent and grades what it did. Each eval lives in a `.eval.ts` file under `evals/`, and `eve eval` runs them. Use evals to catch regressions when you change a prompt or a tool.
|
|
7
7
|
|
|
8
|
-
Evals
|
|
9
|
-
|
|
10
|
-
Define [tool stubs](./tool-stubs) in your eval to control tool results, then assert on the agent's tool calls and responses.
|
|
8
|
+
Evals talk to a real agent server over the same HTTP API as your users, through the [TypeScript client](../guides/client/overview). A passing eval means the agent started, accepted a request, and produced the asserted result. To control what tools return, define [tool stubs](./tool-stubs) in the eval.
|
|
11
9
|
|
|
12
10
|
## `defineEval`
|
|
13
11
|
|
|
14
|
-
|
|
12
|
+
The `evals/` directory sits at the app root. Each `.eval.ts` file exports one eval, or an array of evals that runs once per [dataset](./cases#datasets-exporting-an-array) row. The file path is the eval's id, so `evals/weather/brooklyn-forecast.eval.ts` becomes `weather/brooklyn-forecast`. Don't add an `id` or `name`.
|
|
15
13
|
|
|
16
14
|
```text
|
|
17
15
|
my-agent/
|
|
@@ -25,7 +23,7 @@ my-agent/
|
|
|
25
23
|
└── package.json
|
|
26
24
|
```
|
|
27
25
|
|
|
28
|
-
An eval is
|
|
26
|
+
An eval is one `async test(t)` function. It sends messages and records assertions through `t`:
|
|
29
27
|
|
|
30
28
|
```ts title="evals/weather/brooklyn-forecast.eval.ts"
|
|
31
29
|
import { defineEval } from "eve/evals";
|
|
@@ -42,11 +40,11 @@ export default defineEval({
|
|
|
42
40
|
});
|
|
43
41
|
```
|
|
44
42
|
|
|
45
|
-
`test` is the only required field
|
|
43
|
+
`test` is the only required field; `description`, `judge`, `tags`, `metadata`, `timeoutMs`, and `reporters` are optional. The init template adds `evals/**/*.ts` to `tsconfig.json`, so eval code type-checks with the app.
|
|
46
44
|
|
|
47
45
|
## `evals.config.ts`
|
|
48
46
|
|
|
49
|
-
Every `evals/` directory needs exactly one `evals.config.ts` at its root. It
|
|
47
|
+
Every `evals/` directory needs exactly one `evals.config.ts` at its root. It sets the defaults that all evals share:
|
|
50
48
|
|
|
51
49
|
```ts title="evals/evals.config.ts"
|
|
52
50
|
import { defineEvalConfig } from "eve/evals";
|
|
@@ -58,11 +56,22 @@ export default defineEvalConfig({
|
|
|
58
56
|
});
|
|
59
57
|
```
|
|
60
58
|
|
|
61
|
-
|
|
59
|
+
Every field is optional:
|
|
60
|
+
|
|
61
|
+
| Field | Sets |
|
|
62
|
+
| ------------------- | ---------------------------------------------------------------------------------------------------- |
|
|
63
|
+
| `judge` | The model for [LLM-as-judge](./judge) assertions (`t.judge(...)`). Omit it to use `typesafe-ai/jev`. |
|
|
64
|
+
| `reporters` | Reporters that observe every eval in the run, so one `braintrustReporter()` here covers all evals. |
|
|
65
|
+
| `maxConcurrency` | The concurrent eval limit. |
|
|
66
|
+
| `timeoutMs` | The per-eval timeout. |
|
|
67
|
+
| `repeat` | How many times each eval runs, to measure repeatability. See [Repeat evals](./running#repeat-evals). |
|
|
68
|
+
| `setup`, `teardown` | Run-wide hooks before and after the run. See [Run setup and teardown](#run-setup-and-teardown). |
|
|
69
|
+
|
|
70
|
+
CLI flags (`--max-concurrency`, `--timeout`, `--repeat`) and per-eval values take precedence over config defaults.
|
|
62
71
|
|
|
63
72
|
### Run setup and teardown
|
|
64
73
|
|
|
65
|
-
Use `setup` to start shared resources or run migrations before eve starts the local agent server. Return shared
|
|
74
|
+
Use `setup` to start shared resources or run migrations before eve starts the local agent server. Return shared values from `setup`, and release resources in `teardown`:
|
|
66
75
|
|
|
67
76
|
```ts title="evals/evals.config.ts"
|
|
68
77
|
import { defineEvalConfig } from "eve/evals";
|
|
@@ -79,7 +88,7 @@ export default defineEvalConfig({
|
|
|
79
88
|
});
|
|
80
89
|
```
|
|
81
90
|
|
|
82
|
-
`context` is
|
|
91
|
+
`context` is typed from the setup return value. To use that type in an eval file, pass the config's type to `defineEval`:
|
|
83
92
|
|
|
84
93
|
```ts title="evals/database.eval.ts"
|
|
85
94
|
import { defineEval } from "eve/evals";
|
|
@@ -94,18 +103,15 @@ export default defineEval<typeof config>({
|
|
|
94
103
|
});
|
|
95
104
|
```
|
|
96
105
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
`setup` can also return nothing.
|
|
106
|
+
`setup`, every eval, and `teardown` share one context object in the `eve eval` process. The agent server never sees it, so class instances and functions work as values. Evals that run at the same time share these resources, so isolate any database records they change.
|
|
100
107
|
|
|
101
|
-
|
|
102
|
-
- Teardown runs after the local server and its sandbox handles shut down, including after setup, startup, or eval failures. Cleanup failures fail the command.
|
|
108
|
+
`setup` runs once per `eve eval` command, before the agent server starts. `teardown` runs after the local server and its sandboxes stop, even when setup, startup, or an eval fails. A `teardown` failure fails the command. The per-eval `timeoutMs` doesn't apply to either hook.
|
|
103
109
|
|
|
104
|
-
`teardown` receives `undefined` when setup returns
|
|
110
|
+
`setup` can return nothing, and `teardown` works without `setup`. `teardown` receives `undefined` when `setup` returns nothing or throws, so `setup` must clean up anything only it can reach.
|
|
105
111
|
|
|
106
112
|
## Deterministic fixture models
|
|
107
113
|
|
|
108
|
-
|
|
114
|
+
`mockModel` gives a test agent fixed replies, so it runs without calling a model provider. A fixed reply is one line:
|
|
109
115
|
|
|
110
116
|
```ts title="agent/agent.ts"
|
|
111
117
|
import { defineAgent } from "eve";
|
|
@@ -116,7 +122,7 @@ export default defineAgent({
|
|
|
116
122
|
});
|
|
117
123
|
```
|
|
118
124
|
|
|
119
|
-
Pass a callback when the reply depends on the conversation.
|
|
125
|
+
Pass a callback when the reply depends on the conversation. It receives `lastUserMessage`, `userMessages`, `userMessageCount`, the available `tools`, and prior `toolResults`:
|
|
120
126
|
|
|
121
127
|
```ts title="agent/agent.ts"
|
|
122
128
|
export default defineAgent({
|
|
@@ -126,7 +132,7 @@ export default defineAgent({
|
|
|
126
132
|
});
|
|
127
133
|
```
|
|
128
134
|
|
|
129
|
-
The callback
|
|
135
|
+
The callback can return `{ text, toolCalls, usage }` for deterministic tool loops or explicit token counts. Use the options form when a fixture also needs a custom model identity:
|
|
130
136
|
|
|
131
137
|
```ts title="agent/agent.ts"
|
|
132
138
|
model: mockModel({
|
|
@@ -139,32 +145,32 @@ model: mockModel({
|
|
|
139
145
|
});
|
|
140
146
|
```
|
|
141
147
|
|
|
142
|
-
|
|
148
|
+
Omit the response to reply `"Mock response"`. `mockModel` supports streamed and non-streamed replies and estimates token usage. Its context window is 1,000,000 tokens, so eve never compacts the conversation. To test compaction, set `modelContextWindowTokens` on the agent.
|
|
149
|
+
|
|
150
|
+
The model is part of the agent definition, so use `mockModel` in a separate test agent. That agent stays mocked locally and when you deploy it as an eval target.
|
|
143
151
|
|
|
144
152
|
## The `t` context
|
|
145
153
|
|
|
146
|
-
`t`
|
|
154
|
+
`t` both sends messages and records assertions, so an eval is ordinary code that sends a message, then asserts on the reply.
|
|
147
155
|
|
|
148
|
-
- **
|
|
149
|
-
- **Assert
|
|
156
|
+
- **Send:** `t.send(...)` starts a conversation with its first message and returns the turn. `t.session()` starts an empty conversation. Send follow-ups with `turn.session.send(...)`, and read the reply from `turn.reply`. The full list of session methods is in [the drive API](./cases#the-drive-api).
|
|
157
|
+
- **Assert:** use the three surfaces in the next section.
|
|
150
158
|
|
|
151
159
|
## Three assertion surfaces
|
|
152
160
|
|
|
153
|
-
|
|
161
|
+
| Surface | Grades |
|
|
162
|
+
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
|
163
|
+
| Methods such as `t.calledTool("get_weather")` | What the agent did: on `t` for the whole run, or on one session or turn. See [Assertions](./assertions). |
|
|
164
|
+
| `t.check(value, assertion)` and `t.score(…)` | Any value, such as `turn.reply` or parsed JSON, or a 0–1 score you compute. See [Assertions](./assertions#scores-with-tscore). |
|
|
165
|
+
| `t.judge("cites a source")` | Quality, graded by a separate judge model. See [Judge](./judge). |
|
|
154
166
|
|
|
155
|
-
|
|
156
|
-
- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(turn.reply, includes("sunny"))`. Grade `turn.reply`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
|
|
157
|
-
- **`t.score(evaluation)`** records a score, such as a length ratio or an evaluator's result. Chain `.label(key)` to name it, and `.gate(threshold)` or `.atLeast(threshold)` to let it decide the eval. See [Assertions](./assertions#scores-with-tscore).
|
|
158
|
-
- **`t.judge(...)`** is the LLM-as-judge surface, like `t.judge("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).
|
|
167
|
+
`t.judge` grades the reply of the most recently finished turn. Pass `{ on: turn.session.transcript }` to grade a whole conversation.
|
|
159
168
|
|
|
160
169
|
## Gate vs soft
|
|
161
170
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
- **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
|
|
165
|
-
- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity`, every `t.judge(...)` assertion, and every `t.score(...)` are soft by default. A soft assertion with no threshold records no verdict and never fails.
|
|
171
|
+
A failed **gate** marks the eval `failed`, and `eve eval` exits non-zero. A **soft** assertion appears in reports. A soft assertion below its threshold marks the eval `scored`, which fails the run only under `--strict`. A soft assertion with no threshold never fails.
|
|
166
172
|
|
|
167
|
-
|
|
173
|
+
`t.judge(...)`, `t.score(...)`, and `similarity` are soft by default; the rest are gates. Chain `.gate(threshold?)`, `.soft(threshold?)`, or `.atLeast(threshold)` to change one. See [Severity](./assertions#severity).
|
|
168
174
|
|
|
169
175
|
```ts
|
|
170
176
|
t.succeeded(); // gate
|
|
@@ -175,7 +181,7 @@ t.judge("The response is consistent with output.reference.", {
|
|
|
175
181
|
}).atLeast(0.7); // soft, gated under --strict at 0.7
|
|
176
182
|
```
|
|
177
183
|
|
|
178
|
-
Use `await t.require(value, assertion)` for a gate that must pass before the
|
|
184
|
+
Use `await t.require(value, assertion)` for a gate that must pass before the eval continues. Call `t.skip(reason)` first to skip an eval on a target that doesn't support it.
|
|
179
185
|
|
|
180
186
|
## Run evals with eve eval
|
|
181
187
|
|
|
@@ -189,12 +195,10 @@ Exit code `0` means every eval passed its gates. See [Running evals](./running)
|
|
|
189
195
|
|
|
190
196
|
## A good baseline
|
|
191
197
|
|
|
192
|
-
|
|
198
|
+
Start with a few small smoke evals: `t.succeeded()` plus one or two content checks, with dataset files in `evals/data/`. Add a judge or Braintrust when you need fuzzy grading or a shared place to review results. In CI, run `eve eval --strict` so soft threshold misses fail the build.
|
|
193
199
|
|
|
194
200
|
## What to read next
|
|
195
201
|
|
|
196
|
-
The rest of this section covers each piece:
|
|
197
|
-
|
|
198
202
|
- [Cases](./cases): single-turn evals, scripted multi-turn evals, and dataset fan-out
|
|
199
203
|
- [Assertions](./assertions): run-level methods and `t.check` value assertions, with matchers and severity
|
|
200
204
|
- [Judge](./judge): LLM-as-judge grading and the judge model
|
package/docs/evals/reporters.mdx
CHANGED
|
@@ -3,11 +3,11 @@ title: "Reporters"
|
|
|
3
3
|
description: "Ship eval results to Braintrust, Datadog, or JUnit XML. eve runs and scores everything itself."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
eve
|
|
6
|
+
A reporter sends the results eve graded to Braintrust, Datadog, a JUnit file, or your own destination. Import reporters from `eve/evals/reporters`. The CLI always prints a summary with one line per eval and each failed assertion's message.
|
|
7
7
|
|
|
8
8
|
You are responsible for ensuring any observability or eval provider is approved for the data exported to it.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Declare reporters in `evals.config.ts` to report every eval in the run, such as to one shared Braintrust experiment. List them in an eval's `reporters` to report only the evals that share that reporter instance.
|
|
11
11
|
|
|
12
12
|
## Console
|
|
13
13
|
|
|
@@ -32,7 +32,7 @@ duplicates the CLI's output.
|
|
|
32
32
|
|
|
33
33
|
## Braintrust
|
|
34
34
|
|
|
35
|
-
`braintrustReporter(...)` uploads eval results to Braintrust experiments. Put one instance in the config
|
|
35
|
+
`braintrustReporter(...)` uploads eval results to Braintrust experiments. Put one instance in the config to cover the whole run:
|
|
36
36
|
|
|
37
37
|
```ts title="evals/evals.config.ts"
|
|
38
38
|
import { defineEvalConfig } from "eve/evals";
|
|
@@ -44,7 +44,7 @@ export default defineEvalConfig({
|
|
|
44
44
|
});
|
|
45
45
|
```
|
|
46
46
|
|
|
47
|
-
|
|
47
|
+
To report only some evals, attach the reporter per eval:
|
|
48
48
|
|
|
49
49
|
```ts title="evals/brooklyn-forecast.eval.ts"
|
|
50
50
|
import { defineEval } from "eve/evals";
|
|
@@ -59,15 +59,21 @@ export default defineEval({
|
|
|
59
59
|
});
|
|
60
60
|
```
|
|
61
61
|
|
|
62
|
-
The reporter
|
|
62
|
+
The reporter takes optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against.
|
|
63
63
|
|
|
64
|
-
|
|
64
|
+
- Gate assertions log as scores of 0 or 1 under a `gate:` prefix, so experiment diffs show gate regressions like soft-score regressions.
|
|
65
|
+
- Assertions that share a name, such as several unlabeled judges, log the lowest score. The console warns when merged soft scores differ; use `.label(name)` to group or separate them.
|
|
66
|
+
- Errored assertions log no score.
|
|
67
|
+
- Metadata holds assertion scores under `eveAssertionScores`, failed assertion details under `eveFailedAssertions`, and the eval's traces under `eveTraceIds` and `eveTraceContexts`.
|
|
68
|
+
- In a [repeated run](./running#repeat-evals), each attempt logs its own row with the id `<id>#<attempt>` and the attempt number under `eveAttempt`.
|
|
65
69
|
|
|
66
|
-
|
|
70
|
+
Evals that share one reporter instance, through the config, a `shared.ts` export, or a dataset array, land in one experiment. Listing the config reporter on an eval too doesn't report it twice.
|
|
71
|
+
|
|
72
|
+
Install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run without shipping results, including from config reporters.
|
|
67
73
|
|
|
68
74
|
## Datadog
|
|
69
75
|
|
|
70
|
-
`datadogReporter(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config
|
|
76
|
+
`datadogReporter(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config to cover the whole run:
|
|
71
77
|
|
|
72
78
|
```ts title="evals/evals.config.ts"
|
|
73
79
|
import { defineEvalConfig } from "eve/evals";
|
|
@@ -78,29 +84,44 @@ export default defineEvalConfig({
|
|
|
78
84
|
});
|
|
79
85
|
```
|
|
80
86
|
|
|
81
|
-
The reporter creates one Datadog Experiment
|
|
87
|
+
The reporter creates one Datadog Experiment per run and prints its URL afterward. Each completed eval becomes one experiment span, with its assertion scores as metrics.
|
|
88
|
+
|
|
89
|
+
With `recordInputs` (the default), the reporter syncs a versioned Datadog dataset after the evals finish and runs the Experiment against that version. Each span links to its dataset record, and the reporter prints the Dataset URL. With `recordInputs: false`, the Experiment uses a placeholder dataset, and spans are submitted as evals finish.
|
|
90
|
+
|
|
91
|
+
When an eve session's trace is sampled, the reporter adds `experimentRuntimeTraceLinks` metadata, so an Experiment row links to the agent trace that produced it.
|
|
92
|
+
|
|
93
|
+
Every run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records match by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description. In a [repeated run](./running#repeat-evals), every attempt links to the eval's one dataset record, and each span's `runIteration` is the attempt number.
|
|
82
94
|
|
|
83
|
-
|
|
95
|
+
Metric names come from assertion names:
|
|
84
96
|
|
|
85
|
-
|
|
97
|
+
- Characters other than letters, numbers, underscores, and hyphens become underscores.
|
|
98
|
+
- Gates get a `gate_` prefix.
|
|
99
|
+
- Assertions that share a name report the lowest score.
|
|
100
|
+
- Names that collide with built-in metrics get a numeric suffix.
|
|
86
101
|
|
|
87
|
-
|
|
102
|
+
For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Use `.label("stable name")` to choose a stable metric name. Assertion names can contain authored expectations, so review them before exporting sensitive evals. Row metadata keeps each assertion's name and score under `eveAssertionScores`; pass `recordAssertionDetails: false` to omit them and failed assertion messages.
|
|
88
103
|
|
|
89
|
-
|
|
104
|
+
Install `dd-trace` (tested against `dd-trace@6.13.0`: `npm install dd-trace@6.13.0`). Set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate. Without credentials, reporting is skipped.
|
|
105
|
+
|
|
106
|
+
Like the Braintrust reporter, it records eval prompts, outputs, authored expectations, and execution error messages by default. Pass `recordInputs: false`, `recordOutputs: false`, `recordExpectedOutputs: false`, or `recordErrors: false` if your destination isn't approved for that content.
|
|
107
|
+
|
|
108
|
+
- Input is the eval's first `t.send(...)` message, or the eval description when there is none. It is stored in the dataset record and the experiment span.
|
|
109
|
+
- Expected output comes from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output`, and is stored in the dataset record and the experiment span. Those keys are excluded from the general metadata. A dataset record without expected output is still valid.
|
|
110
|
+
- Target URL credentials, paths, query parameters, and fragments are never reported.
|
|
90
111
|
|
|
91
112
|
## JUnit
|
|
92
113
|
|
|
93
|
-
`junitReporter({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same
|
|
114
|
+
`junitReporter({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same without touching eval files. The flag usually fits better because CI owns the output path:
|
|
94
115
|
|
|
95
116
|
```bash
|
|
96
117
|
eve eval --strict --junit .eve/junit.xml
|
|
97
118
|
```
|
|
98
119
|
|
|
99
|
-
Each eval becomes one `<testcase>` named by its
|
|
120
|
+
Each eval becomes one `<testcase>` named by its id, or one per attempt named `<id>#<attempt>` in a repeated run. Failed gates and execution errors become failures, and `t.skip(reason)` becomes `<skipped>`.
|
|
100
121
|
|
|
101
122
|
## Custom reporters
|
|
102
123
|
|
|
103
|
-
|
|
124
|
+
Build a custom reporter only when no built-in reporter covers the destination. It implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same results as the built-ins. Every callback can return a promise:
|
|
104
125
|
|
|
105
126
|
```ts
|
|
106
127
|
interface EvalReporter {
|
|
@@ -112,25 +133,31 @@ interface EvalReporter {
|
|
|
112
133
|
}
|
|
113
134
|
```
|
|
114
135
|
|
|
115
|
-
|
|
136
|
+
| Callback | Fires |
|
|
137
|
+
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
138
|
+
| `onRunStart` | Once, before any eval runs. |
|
|
139
|
+
| `onEvalStart` | When an eval is scheduled, with the eval definition, target, and start time. No trace exists yet. |
|
|
140
|
+
| `onSessionStart` | Once per session, when its first trace context arrives, with `sessionId`, `primary`, and `traceContext` (`traceId`, `spanId`, `traceFlags`). Never fires without tracing. |
|
|
141
|
+
| `onEvalComplete` | With the checks, scores, and verdict. `context` lists every distinct trace context across the eval's sessions, as does `result.result.traceContexts`. |
|
|
142
|
+
| `onRunComplete` | Once, with the aggregated summary. |
|
|
143
|
+
|
|
144
|
+
An eval's callbacks stay in that order even when evals run concurrently. In a [repeated run](./running#repeat-evals), `onEvalStart` fires once per attempt. Its event and each `onSessionStart` event carry the 1-based `attempt`, which matches `result.attempt`.
|
|
116
145
|
|
|
117
|
-
|
|
118
|
-
- `onSessionStart` fires once for each session after eve receives its first trace context. It includes `sessionId`, `primary`, and `traceContext` with `traceId`, `spanId`, and `traceFlags`.
|
|
119
|
-
- `onEvalComplete` fires with the checks, scores, and verdict. The runner also supplies `context`, including every distinct trace context collected across the eval's sessions. The same list is available as `result.result.traceContexts`.
|
|
146
|
+
`onEvalComplete` gets a list of trace contexts because an eval can create several sessions, and a long session can produce several traces. The list is empty without tracing.
|
|
120
147
|
|
|
121
|
-
Each entry in `result.assertions` is an `AssertionResult`. `name` is the display label, `key` the identifier set by `.label(key)`, and `score` the measurement. `threshold` and `passed` describe the rule and are absent when no rule applies. When a scorer threw, `errored` is `true`, `message` carries the error, and `score` is absent. The `.eve/evals/` artifacts write the same fields.
|
|
148
|
+
Each entry in `result.assertions` is an `AssertionResult`. `name` is the display label, `key` the identifier set by `.label(key)`, and `score` the measurement. `threshold` and `passed` describe the rule and are absent when no rule applies. When a scorer threw, `errored` is `true`, `message` carries the error, and `score` is absent. The `.eve/evals/` artifacts write the same fields, plus the trace contexts.
|
|
122
149
|
|
|
123
|
-
|
|
150
|
+
`result.result.derived` summarizes what happened in the eval's sessions, such as tool calls and `models`. `models` lists the distinct ids of the models the agent called, in session order. Each entry in `result.result.sessions` holds the same summary for one session, with `models` in first-use order. A reporter can use it to tell the primary session's models from other sessions'.
|
|
124
151
|
|
|
125
|
-
|
|
152
|
+
A subagent runs in its own session. Its models appear only when the eval attaches and reads that session with `t.target.attach(...).nextTurn()` or `watch()`. `models` includes the models that wrote compaction summaries, but not routing calls made inside a provider.
|
|
126
153
|
|
|
127
|
-
`result.result.reply` is the last
|
|
154
|
+
`result.result.reply` is the last turn's reply text, and `data` is its structured output, with no text fallback. `status` describes the request, while `sessionEnded` and the optional `view` describe that turn's session. The top-level `sessionId` identifies the primary session; inspect `sessions` for each session's events and view. To pick one output value, fall back to `reply` only when `data === undefined`, so structured `null` stays an output.
|
|
128
155
|
|
|
129
|
-
Braintrust and Datadog metadata use `eveStatus`, `eveAwaitingInput`, and `eveSessionEnded`.
|
|
156
|
+
Braintrust and Datadog metadata use `eveStatus`, `eveAwaitingInput`, and `eveSessionEnded`. Use `eveSessionEnded` to tell a finished request from an ended session.
|
|
130
157
|
|
|
131
|
-
`derived.usage` is the token usage and cost a session recorded
|
|
158
|
+
`derived.usage` is the token usage and cost a session recorded in its `usage.recorded` events. It includes subagents' spend and compaction summaries. On a turn, it is the session's total so far.
|
|
132
159
|
|
|
133
|
-
|
|
160
|
+
The eval's `usage` counts each session the eval read once, using its latest total. It skips a session that another session in the eval opened, because the opening session's total already includes it. It is absent when a counted session recorded no usage.
|
|
134
161
|
|
|
135
162
|
## What to read next
|
|
136
163
|
|
package/docs/evals/running.mdx
CHANGED
|
@@ -3,7 +3,7 @@ title: "Running Evals"
|
|
|
3
3
|
description: "The eve eval CLI: flags, filters, exit codes, artifacts, and how to wire evals into CI."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
`eve eval`
|
|
6
|
+
`eve eval` runs every `.eval.ts` file under `evals/` concurrently and prints a per-eval summary. It boots a local dev server, or targets a remote one with `--url`.
|
|
7
7
|
|
|
8
8
|
```bash
|
|
9
9
|
eve eval # run all discovered evals locally
|
|
@@ -25,7 +25,7 @@ eve eval --keep-state # keep the local server's state and Workflow data
|
|
|
25
25
|
|
|
26
26
|
Positional ids match exactly or by directory prefix: `eve eval weather` runs `evals/weather.eval.ts`, every eval under `evals/weather/`, and every entry of an array-exported `weather.eval.ts`.
|
|
27
27
|
|
|
28
|
-
Tag filters compose: `--tag` keeps evals carrying at least one listed tag, then `--exclude-tag` drops any eval carrying an excluded tag. A `--tag` filter that matches nothing is a configuration error (exit `2`)
|
|
28
|
+
Tag filters compose: `--tag` keeps evals carrying at least one listed tag, then `--exclude-tag` drops any eval carrying an excluded tag. A `--tag` filter that matches nothing is a configuration error (exit `2`). A run where `--exclude-tag` removes every match succeeds with nothing executed.
|
|
29
29
|
|
|
30
30
|
## Exit codes
|
|
31
31
|
|
|
@@ -37,13 +37,13 @@ Tag filters compose: `--tag` keeps evals carrying at least one listed tag, then
|
|
|
37
37
|
|
|
38
38
|
An eval that calls `t.skip(reason)` is reported as skipped, does not count as passed or failed, and never changes the exit code.
|
|
39
39
|
|
|
40
|
-
`eve eval --json`
|
|
40
|
+
`eve eval --json` writes the complete report before exiting, including when stdout is piped to another process or redirected to a file.
|
|
41
41
|
|
|
42
|
-
With `--verbose`, eve prefixes each line with its eval ID
|
|
42
|
+
With `--verbose`, eve prefixes each line with its eval ID and prints `t.log()` lines as they happen. It then prints the `wrun_...` workflow run ID for every session the eval read, so you can find the runs in your workflow provider.
|
|
43
43
|
|
|
44
44
|
## Repeat evals
|
|
45
45
|
|
|
46
|
-
|
|
46
|
+
Evals that call a real model can pass on one run and fail on the next. To measure how often an eval passes, pass `--repeat <n>` or set `repeat` in `evals.config.ts`. eve then runs every selected eval `n` times. The CLI flag overrides the config value. The default is `1`.
|
|
47
47
|
|
|
48
48
|
```bash
|
|
49
49
|
eve eval weather --repeat 5
|
|
@@ -55,15 +55,22 @@ With `n` greater than `1`, each attempt is a separate result with a 1-based `att
|
|
|
55
55
|
- `--json` results, JUnit test cases (named `<id>#<attempt>`), and [reporters](./reporters) receive one result per attempt.
|
|
56
56
|
- Every attempt counts toward the run totals and the exit code, so one failed attempt fails the run.
|
|
57
57
|
|
|
58
|
-
Attempts share the `--max-concurrency`
|
|
58
|
+
Attempts share the `--max-concurrency` limit with every other eval, so attempts of one eval can run at the same time. If an eval depends on shared state or a rate-limited provider, lower `--max-concurrency` so contention doesn't look like flakiness.
|
|
59
59
|
|
|
60
60
|
## Artifacts
|
|
61
61
|
|
|
62
|
-
Each run
|
|
62
|
+
Each run writes artifacts under `.eve/evals/<timestamp>/`:
|
|
63
|
+
|
|
64
|
+
- `summary.json` summarizes the run.
|
|
65
|
+
- `results.jsonl` indexes the results, one line per eval.
|
|
66
|
+
- `evals/` holds each eval's assertion results, verdict, start and completion times, and `t.log` lines.
|
|
67
|
+
- `evals/` also holds each session's events, with when the client sent each request and received each event.
|
|
68
|
+
|
|
69
|
+
In a repeated run, each attempt's detail and event files get an `.attempt-<n>` suffix, such as `evals/weather.attempt-2.json`. When an eval fails, check its artifact for details the console omits.
|
|
63
70
|
|
|
64
71
|
## CI
|
|
65
72
|
|
|
66
|
-
|
|
73
|
+
In CI, run evals strict and machine-reportable:
|
|
67
74
|
|
|
68
75
|
```bash
|
|
69
76
|
eve eval --strict --junit .eve/junit.xml
|
package/docs/evals/targets.mdx
CHANGED
|
@@ -3,9 +3,9 @@ title: "Targets"
|
|
|
3
3
|
description: "Point evals at a local dev server or a deployment with the same eval files."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
An eval target is
|
|
6
|
+
An eval target is the URL of the agent server an eval runs against. `eve eval` starts a local dev server, and `eve eval --url <url>` targets a running server or deployment. The same eval files work for both, so evals also serve as end-to-end tests in CI.
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Before running evals, `eve eval` waits for `/eve/v1/health` and checks `/eve/v1/info`. Inside `test`, `t.target` is the target.
|
|
9
9
|
|
|
10
10
|
## Target helpers
|
|
11
11
|
|
|
@@ -23,40 +23,37 @@ export default defineEval({
|
|
|
23
23
|
});
|
|
24
24
|
```
|
|
25
25
|
|
|
26
|
-
- `t.target.fetch(path, init)`
|
|
27
|
-
- `t.target.dispatchSchedule(id)` triggers a [schedule](../schedules) through the dev-only schedule route and returns the session ids it created. It
|
|
28
|
-
- `t.target.attach(sessionId, { startIndex? })` returns a handle
|
|
29
|
-
- `session.nextTurn()` reads and records the next turn
|
|
30
|
-
- `session.watch()` starts
|
|
26
|
+
- `t.target.fetch(path, init)` sends an [authenticated](#authentication) request to the target, to test channel and webhook routes.
|
|
27
|
+
- `t.target.dispatchSchedule(id)` triggers a [schedule](../schedules) through the dev-only schedule route and returns the session ids it created. It throws unless the target has dev routes enabled. The local `eve eval` dev server and a deployment in development mode enable them.
|
|
28
|
+
- `t.target.attach(sessionId, { startIndex? })` returns a handle for an existing session without making a request. `startIndex` sets where reading starts. An unknown or inaccessible session fails when you read it, not when you attach it.
|
|
29
|
+
- `session.nextTurn()` reads and records the session's next turn. Its events feed both the session's assertions and the run-level assertions. Each read continues where the last one stopped.
|
|
30
|
+
- `session.watch()` starts reading right away and returns a [live turn](./cases#in-flight-turns) with `waitForEvent(...)`, `cancel()`, and `result()`. Pass `{ until }` to stop at a matching event instead of the end of a turn, such as `context.settled` for a context change between turns.
|
|
31
31
|
|
|
32
|
-
|
|
32
|
+
Attached sessions are full `EveEvalSession`s. Send them messages and assert on them directly, as in `session.succeeded()` or `session.calledTool(...)`. Assertions on `t` cover the whole run, including every attached session.
|
|
33
33
|
|
|
34
|
-
|
|
35
|
-
`t.session()` creates a session without a first message; call `session.send()` or `session.start()`
|
|
36
|
-
on the returned handle to use it. `t.send()` creates a new session with its first message.
|
|
37
|
-
These helpers handle the session protocol, retries, and event collection for the eval.
|
|
34
|
+
To create sessions and send messages, use the [drive API](./cases#the-drive-api).
|
|
38
35
|
|
|
39
36
|
## Authentication
|
|
40
37
|
|
|
41
|
-
|
|
38
|
+
Requests to a local target carry no credentials, because `eve eval` started that server.
|
|
42
39
|
|
|
43
|
-
For a remote `--url`, eve
|
|
40
|
+
For a remote `--url`, eve finds the expected Vercel team and project from `VERCEL_ORG_ID` and `VERCEL_PROJECT_ID` when both are nonblank. Otherwise it reads the local Vercel project link: `.vercel/project.json`, or the matching project in `.vercel/repo.json` for a linked monorepo.
|
|
44
41
|
|
|
45
|
-
With a project
|
|
42
|
+
With a project, eve looks up the exact HTTPS origin under that team, regardless of the Vercel CLI's current team. It sends your Vercel credentials only when the deployment's project ID matches. Without a project, eve checks the URL under the Vercel CLI's current team. A URL that Vercel can't verify gets no credentials.
|
|
46
43
|
|
|
47
44
|
After verification, eve sends the available Vercel credentials:
|
|
48
45
|
|
|
49
46
|
- The resolved OIDC token as both the bearer and Vercel trusted-IDP header.
|
|
50
47
|
- `VERCEL_AUTOMATION_BYPASS_SECRET`, when set, as the Protection Bypass for Automation header.
|
|
51
48
|
|
|
52
|
-
`EVE_EVAL_AUTH_TOKEN`
|
|
49
|
+
Set `EVE_EVAL_AUTH_TOKEN` to send that bearer token instead, for targets that don't use Vercel OIDC. Requests that carry credentials don't follow redirects, so the credentials never reach another origin.
|
|
53
50
|
|
|
54
|
-
`/eve/v1/info` serves only operator
|
|
51
|
+
`/eve/v1/info` serves only operator callers, so the credential must authenticate with `principalType: "service"`. Vercel OIDC, `jwtHmac()`, `jwtEcdsa()`, or a custom `AuthFn` can provide it. When the target answers `403`, the run fails before any eval starts and names the credential it needs.
|
|
55
52
|
|
|
56
|
-
`t.target.fetch(path, init)`
|
|
53
|
+
`t.target.fetch(path, init)` sends these same credentials.
|
|
57
54
|
|
|
58
55
|
## What to read next
|
|
59
56
|
|
|
60
57
|
- [Running evals](./running): `--url` and the rest of the CLI in practice
|
|
61
58
|
- [Schedules](../schedules): the surface `dispatchSchedule` drives
|
|
62
|
-
- [Channels](../channels/overview):
|
|
59
|
+
- [Channels](../channels/overview): routes you can test with `target.fetch`
|