eve 0.83.0 → 0.83.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/dist/src/channel/types.d.ts +3 -2
- package/dist/src/chunks/{use-eve-agent-Ci5dr0aR.js → use-eve-agent-DnbG9c7c.js} +33 -9
- package/dist/src/chunks/{use-eve-agent-3-WRmJkd.js → use-eve-agent-ZY7NQ2VW.js} +33 -9
- package/dist/src/cli/run.js +1 -1
- package/dist/src/client/message-response.d.ts +18 -0
- package/dist/src/client/message-response.js +1 -1
- package/dist/src/client/session.js +1 -1
- package/dist/src/compiled/.vendor-stamp.json +2 -2
- package/dist/src/compiled/@ai-sdk/mcp/index.js +1 -1
- package/dist/src/compiled/@eve/catalog/index.js +1 -1
- package/dist/src/context/build-dynamic-tools.js +1 -1
- package/dist/src/context/dynamic-skill-lifecycle.d.ts +2 -2
- package/dist/src/context/dynamic-skill-lifecycle.js +1 -1
- package/dist/src/context/dynamic-tool-lifecycle.js +1 -1
- package/dist/src/context/prewarm.d.ts +27 -0
- package/dist/src/context/prewarm.js +1 -0
- package/dist/src/context/providers/sandbox.d.ts +7 -0
- package/dist/src/context/providers/sandbox.js +1 -1
- package/dist/src/context/run-step.js +1 -1
- package/dist/src/discover/lib.d.ts +4 -3
- package/dist/src/discover/lib.js +1 -1
- package/dist/src/discover/named-source-directory.d.ts +3 -2
- package/dist/src/evals/assertions/run.js +1 -1
- package/dist/src/evals/cli/eval.d.ts +2 -0
- package/dist/src/evals/cli/eval.js +2 -2
- package/dist/src/evals/define-eval-config.d.ts +5 -4
- package/dist/src/evals/define-eval-config.js +1 -1
- package/dist/src/evals/index.d.ts +1 -1
- package/dist/src/evals/runner/artifacts.js +2 -2
- package/dist/src/evals/runner/attempt-label.d.ts +2 -0
- package/dist/src/evals/runner/attempt-label.js +1 -0
- package/dist/src/evals/runner/derive-run-facts.d.ts +13 -0
- package/dist/src/evals/runner/derive-run-facts.js +1 -1
- package/dist/src/evals/runner/reporters/braintrust.js +1 -1
- package/dist/src/evals/runner/reporters/console.js +1 -1
- package/dist/src/evals/runner/reporters/datadog.js +1 -1
- package/dist/src/evals/runner/reporters/junit.js +2 -2
- package/dist/src/evals/runner/reporters/types.d.ts +2 -0
- package/dist/src/evals/runner/run-evals.d.ts +5 -0
- package/dist/src/evals/runner/run-evals.js +1 -1
- package/dist/src/evals/session-manager.js +1 -1
- package/dist/src/evals/session.d.ts +4 -1
- package/dist/src/evals/session.js +2 -2
- package/dist/src/evals/types.d.ts +37 -1
- package/dist/src/eve-channel/request.js +1 -1
- package/dist/src/execution/catalog/connection-entry.js +1 -1
- package/dist/src/execution/connections/callback-route.d.ts +3 -2
- package/dist/src/execution/connections/callback-route.js +1 -1
- package/dist/src/execution/dynamic-workflow/tool.js +1 -1
- package/dist/src/execution/node-step.d.ts +1 -0
- package/dist/src/execution/node-step.js +1 -1
- package/dist/src/execution/sandbox/bindings/vercel.js +1 -1
- package/dist/src/execution/sandbox/ensure.d.ts +8 -0
- package/dist/src/execution/sandbox/ensure.js +1 -1
- package/dist/src/execution/sandbox/multiplexed-command.d.ts +9 -3
- package/dist/src/execution/sandbox/multiplexed-command.js +1 -1
- package/dist/src/execution/sandbox/read-file.js +1 -1
- package/dist/src/execution/session/handoff-steps.d.ts +1 -0
- package/dist/src/execution/session/handoff-steps.js +1 -1
- package/dist/src/execution/session/handoff.js +1 -1
- package/dist/src/execution/session/handover-tracing.d.ts +3 -0
- package/dist/src/execution/session/handover-tracing.js +1 -0
- package/dist/src/execution/session/prewarm-step.d.ts +13 -0
- package/dist/src/execution/session/prewarm-step.js +1 -0
- package/dist/src/execution/session/program.d.ts +4 -1
- package/dist/src/execution/session/program.js +1 -1
- package/dist/src/execution/session/state-cursor.d.ts +11 -0
- package/dist/src/execution/session/state-cursor.js +1 -1
- package/dist/src/execution/session/turn-step-types.d.ts +2 -0
- package/dist/src/execution/session/turn-step.js +1 -1
- package/dist/src/execution/session/turn.d.ts +1 -0
- package/dist/src/execution/session/turn.js +1 -1
- package/dist/src/execution/session-event-stream.d.ts +6 -0
- package/dist/src/execution/session-event-stream.js +1 -1
- package/dist/src/execution/session-inbox/inbox.d.ts +8 -0
- package/dist/src/execution/session-inbox/inbox.js +1 -1
- package/dist/src/execution/tools/agent-router-workflow.js +1 -1
- package/dist/src/execution/workflow-runtime.js +1 -1
- package/dist/src/extensions/git/extension/lib/gh-tool.d.ts +11 -2
- package/dist/src/extensions/git/extension/lib/gh-tool.js +1 -1
- package/dist/src/harness/attachment-staging.d.ts +3 -0
- package/dist/src/harness/attachment-staging.js +1 -1
- package/dist/src/harness/compaction/step.d.ts +7 -1
- package/dist/src/harness/compaction/step.js +1 -1
- package/dist/src/harness/hitl/approvals.d.ts +2 -0
- package/dist/src/harness/hitl/approvals.js +1 -1
- package/dist/src/harness/hitl/delivery.d.ts +1 -0
- package/dist/src/harness/hitl/delivery.js +1 -1
- package/dist/src/harness/hitl/intake.js +1 -1
- package/dist/src/harness/model-call/call.d.ts +2 -0
- package/dist/src/harness/model-call/call.js +1 -1
- package/dist/src/harness/model-call/request.d.ts +2 -1
- package/dist/src/harness/token-estimate.js +1 -1
- package/dist/src/harness/types.d.ts +6 -0
- package/dist/src/internal/application/stamped-package-version.js +1 -1
- package/dist/src/internal/attachments/media-metadata.d.ts +0 -2
- package/dist/src/internal/attachments/media-metadata.js +1 -1
- package/dist/src/internal/attachments/read-media-metadata.d.ts +7 -0
- package/dist/src/internal/attachments/read-media-metadata.js +2 -0
- package/dist/src/internal/authored-asset-import-plugin.d.ts +7 -0
- package/dist/src/internal/authored-asset-import-plugin.js +1 -1
- package/dist/src/internal/authored-module-build-options.js +1 -1
- package/dist/src/internal/authored-module-graph.js +1 -1
- package/dist/src/internal/authored-module-loader.d.ts +0 -1
- package/dist/src/internal/authored-module-loader.js +1 -1
- package/dist/src/internal/authored-package-boundary.d.ts +3 -0
- package/dist/src/internal/authored-package-boundary.js +1 -1
- package/dist/src/internal/input-text.d.ts +2 -1
- package/dist/src/internal/nitro/host/create-application-nitro.js +1 -1
- package/dist/src/internal/workflow-bundle/builder.js +2 -2
- package/dist/src/internal/workflow-bundle/dynamic-tool-transform.js +1 -1
- package/dist/src/public/channels/slack/slackChannel.js +1 -1
- package/dist/src/runtime/connections/authorization-complete-page.d.ts +14 -2
- package/dist/src/runtime/connections/authorization-complete-page.js +5 -5
- package/dist/src/runtime/subagents/workflow.js +1 -1
- package/dist/src/sandbox/state.d.ts +17 -0
- package/dist/src/setup/scaffold/create/project.js +1 -1
- package/dist/src/svelte/index.js +1 -1
- package/dist/src/svelte/use-eve-agent.js +1 -1
- package/dist/src/tools/definition.js +1 -1
- package/dist/src/tools/durable-callbacks.d.ts +5 -1
- package/dist/src/tools/durable-callbacks.js +1 -1
- package/dist/src/tools/dynamic.d.ts +3 -3
- package/dist/src/tools/provided/ask-question-workflow.js +1 -1
- package/dist/src/tools/provided/sleep-workflow.js +1 -1
- package/dist/src/vue/index.js +1 -1
- package/dist/src/vue/use-eve-agent.js +1 -1
- package/docs/channels/custom.mdx +4 -0
- package/docs/channels/slack.mdx +2 -0
- package/docs/concepts/built-in-tools.md +1 -1
- package/docs/connections/mcp.mdx +1 -1
- package/docs/evals/overview.mdx +1 -1
- package/docs/evals/reporters.mdx +4 -4
- package/docs/evals/running.mdx +19 -1
- package/docs/guides/dynamic-capabilities.md +2 -2
- package/docs/guides/frontend/overview.mdx +7 -3
- package/docs/guides/instrumentation/otel.mdx +23 -0
- package/docs/meta.json +1 -0
- package/docs/reference/agent-files.md +1 -1
- package/docs/reference/cli.md +17 -15
- package/docs/sandbox/custom-providers.mdx +10 -10
- package/docs/sandbox/index.mdx +6 -4
- package/docs/tools/human-in-the-loop.md +3 -1
- package/docs/tools/overview.mdx +1 -1
- package/docs/use-cases.mdx +250 -0
- package/package.json +1 -1
|
@@ -1,9 +1,9 @@
|
|
|
1
|
-
function buildAuthorizationCompletePage(){
|
|
1
|
+
const PAGES={authorized:{title:`Authorization complete`,icon:`M16 8A8 8 0 1 1 0 8a8 8 0 0 1 16 0m-4.47-1.47.53-.53L11 4.94l-.53.53L6.5 9.44l-.97-.97L5 7.94 3.94 9l.53.53 1.5 1.5c.3.3.77.3 1.06 0z`},failed:{title:`Authorization not completed`,icon:`M16 8A8 8 0 1 1 0 8a8 8 0 0 1 16 0M8 6.94l2.47-2.47 1.06 1.06L9.06 8l2.47 2.47-1.06 1.06L8 9.06l-2.47 2.47-1.06-1.06L6.94 8 4.47 5.53l1.06-1.06z`}};function buildAuthorizationCompletePage(e){let t=PAGES[e],n=`<!doctype html>
|
|
2
2
|
<html lang="en">
|
|
3
3
|
<head>
|
|
4
4
|
<meta charset="utf-8" />
|
|
5
5
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
6
|
-
<title
|
|
6
|
+
<title>${t.title}</title>
|
|
7
7
|
<style>
|
|
8
8
|
:root {
|
|
9
9
|
color-scheme: light dark;
|
|
@@ -80,10 +80,10 @@ function buildAuthorizationCompletePage(){return new Response(`<!doctype html>
|
|
|
80
80
|
<main class="screen" aria-labelledby="authorization-title">
|
|
81
81
|
<div class="content">
|
|
82
82
|
<div class="header">
|
|
83
|
-
<h1 id="authorization-title"
|
|
83
|
+
<h1 id="authorization-title">${t.title}</h1>
|
|
84
84
|
<div class="icon" aria-hidden="true">
|
|
85
85
|
<svg viewBox="0 0 16 16" fill="none">
|
|
86
|
-
<path fill="currentColor" fill-rule="evenodd" d="
|
|
86
|
+
<path fill="currentColor" fill-rule="evenodd" d="${t.icon}" clip-rule="evenodd" />
|
|
87
87
|
</svg>
|
|
88
88
|
</div>
|
|
89
89
|
</div>
|
|
@@ -92,4 +92,4 @@ function buildAuthorizationCompletePage(){return new Response(`<!doctype html>
|
|
|
92
92
|
</div>
|
|
93
93
|
</main>
|
|
94
94
|
</body>
|
|
95
|
-
</html
|
|
95
|
+
</html>`;return new Response(n,{headers:{"cache-control":`no-store`,"content-type":`text/html; charset=utf-8`},status:200})}export{buildAuthorizationCompletePage};
|
|
@@ -1 +1 @@
|
|
|
1
|
-
async function agentToolServeWorkflow(e,t){"use workflow";let n=t.agent(t.toolName),r=sendToAgent(n,await e()),i=e();for(;;){let a=await nextCallOrResult(i,r);if(a.kind===`call`){r=await forwardCall(n,r,a.call),i=e();continue}if(!r.signal.aborted){if(a.result.status===`failed`)throw Error(`The agent's turn failed: ${a.result.error?.message??`no reason given`}\nThis task has ended; to retry, call ${t.toolName} without taskId.`);let o=e();if(o!==i){r=sendToAgent(n,await i),i=o;continue}t.reply(a.result.message??``)}r=sendToAgent(n,await i),i=e()}}async function agentWithModelServeWorkflow(e,t){"use workflow";let n=await e(),r=n.input.model.trim();if(r===``)throw TypeError(`agent model must not be empty.`);let i=t.agent(`agent`,{model:r}),a=sendToModelAgent(i,n,r),o=e();for(;;){let n=await nextModelCallOrResult(o,a);if(n.kind===`call`){a=await forwardModelCall(i,a,n.call,r),o=e();continue}if(!a.signal.aborted){if(n.result.status===`failed`)throw Error(`The agent's turn failed: ${n.result.error?.message??`no reason given`}\nThis task has ended; to retry, call ${t.toolName} without taskId.`);let s=e();if(s!==o){a=sendToModelAgent(i,await o,r),o=s;continue}t.reply(n.result.message??``)}a=sendToModelAgent(i,await o,r),o=e()}}function sendToAgent(e,t){return{result:e.send(t.input.message,{signal:t.abortSignal}).then(e=>e.result()),signal:t.abortSignal}}function sendToModelAgent(e,t,n){if(t.input.model!==n)throw Error(`An agent task keeps the model it started with. This task has ended; start a new task to use another model.`);return{result:e.send(t.input.message,{signal:t.abortSignal}).then(e=>e.result()),signal:t.abortSignal}}async function nextCallOrResult(e,t){return await Promise.race([t.result.then(e=>({kind:`result`,result:e})),e.then(e=>({call:e,kind:`call`}))])}async function nextModelCallOrResult(e,t){return await Promise.race([t.result.then(e=>({kind:`result`,result:e})),e.then(e=>({call:e,kind:`call`}))])}async function forwardCall(e,t,n){return t.signal.aborted&&await t.result.catch(()=>void 0),sendToAgent(e,n)}async function forwardModelCall(e,t,n,r){return t.signal.aborted&&await t.result.catch(()=>void 0),sendToModelAgent(e,n,r)}agentToolServeWorkflow.workflowId=`workflow//eve//agentToolServeWorkflow`,agentWithModelServeWorkflow.workflowId=`workflow//eve@0.83.
|
|
1
|
+
async function agentToolServeWorkflow(e,t){"use workflow";let n=t.agent(t.toolName),r=sendToAgent(n,await e()),i=e();for(;;){let a=await nextCallOrResult(i,r);if(a.kind===`call`){r=await forwardCall(n,r,a.call),i=e();continue}if(!r.signal.aborted){if(a.result.status===`failed`)throw Error(`The agent's turn failed: ${a.result.error?.message??`no reason given`}\nThis task has ended; to retry, call ${t.toolName} without taskId.`);let o=e();if(o!==i){r=sendToAgent(n,await i),i=o;continue}t.reply(a.result.message??``)}r=sendToAgent(n,await i),i=e()}}async function agentWithModelServeWorkflow(e,t){"use workflow";let n=await e(),r=n.input.model.trim();if(r===``)throw TypeError(`agent model must not be empty.`);let i=t.agent(`agent`,{model:r}),a=sendToModelAgent(i,n,r),o=e();for(;;){let n=await nextModelCallOrResult(o,a);if(n.kind===`call`){a=await forwardModelCall(i,a,n.call,r),o=e();continue}if(!a.signal.aborted){if(n.result.status===`failed`)throw Error(`The agent's turn failed: ${n.result.error?.message??`no reason given`}\nThis task has ended; to retry, call ${t.toolName} without taskId.`);let s=e();if(s!==o){a=sendToModelAgent(i,await o,r),o=s;continue}t.reply(n.result.message??``)}a=sendToModelAgent(i,await o,r),o=e()}}function sendToAgent(e,t){return{result:e.send(t.input.message,{signal:t.abortSignal}).then(e=>e.result()),signal:t.abortSignal}}function sendToModelAgent(e,t,n){if(t.input.model!==n)throw Error(`An agent task keeps the model it started with. This task has ended; start a new task to use another model.`);return{result:e.send(t.input.message,{signal:t.abortSignal}).then(e=>e.result()),signal:t.abortSignal}}async function nextCallOrResult(e,t){return await Promise.race([t.result.then(e=>({kind:`result`,result:e})),e.then(e=>({call:e,kind:`call`}))])}async function nextModelCallOrResult(e,t){return await Promise.race([t.result.then(e=>({kind:`result`,result:e})),e.then(e=>({call:e,kind:`call`}))])}async function forwardCall(e,t,n){return t.signal.aborted&&await t.result.catch(()=>void 0),sendToAgent(e,n)}async function forwardModelCall(e,t,n,r){return t.signal.aborted&&await t.result.catch(()=>void 0),sendToModelAgent(e,n,r)}agentToolServeWorkflow.workflowId=`workflow//eve//agentToolServeWorkflow`,agentWithModelServeWorkflow.workflowId=`workflow//eve@0.83.2//agentWithModelServeWorkflow`;export{agentToolServeWorkflow,agentWithModelServeWorkflow};
|
|
@@ -53,11 +53,28 @@ type SandboxSessionEndWithArtifact = (context: SandboxSessionContext, state: San
|
|
|
53
53
|
*/
|
|
54
54
|
export declare function withSessionEndArtifact<Implementation extends object>(implementation: Implementation, onSessionEnd: SandboxSessionEndWithArtifact): Implementation;
|
|
55
55
|
export declare function getSessionEndWithArtifact(implementation: object): SandboxSessionEndWithArtifact | undefined;
|
|
56
|
+
/**
|
|
57
|
+
* Background work that may need the sandbox. It calls `open` only when it
|
|
58
|
+
* does, so work that turns out to have nothing to do never opens it. Calling
|
|
59
|
+
* the access's `get()` from the work waits on the work itself.
|
|
60
|
+
*/
|
|
61
|
+
export type SandboxPreparation = (open: () => Promise<SandboxSession>) => Promise<void>;
|
|
56
62
|
export interface SandboxAccess {
|
|
57
63
|
readonly environment?: SandboxEnvironmentIdentity;
|
|
58
64
|
captureState(): Promise<SandboxState>;
|
|
59
65
|
delete?(options?: SandboxDeleteOptions): Promise<void>;
|
|
66
|
+
/** Resolves once the sandbox is open and every queued preparation has run. */
|
|
60
67
|
get(): Promise<SandboxSession | null>;
|
|
68
|
+
/**
|
|
69
|
+
* Opens the sandbox without waiting for queued preparation, for framework
|
|
70
|
+
* work that needs only the connection, such as staging attachments.
|
|
71
|
+
*/
|
|
72
|
+
open(): Promise<SandboxSession | null>;
|
|
73
|
+
/**
|
|
74
|
+
* Queues `work` to run in the background, in order; without `work`, opens
|
|
75
|
+
* the sandbox in the background. Returns at once and never throws.
|
|
76
|
+
*/
|
|
77
|
+
prepare(work?: SandboxPreparation): void;
|
|
61
78
|
stop(): Promise<void>;
|
|
62
79
|
}
|
|
63
80
|
export {};
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import{pinnedNodeEngineMajor}from"../../node-engine.js";import{parseChatGptModelSelection}from"../../../shared/chatgpt-model.js";import{SUPPORTED_AUTHORED_MODULE_FILE_EXTENSIONS}from"../update/module-files.js";import{pathExists,writeTextFile}from"../files.js";import{blockingCreateInPlaceEntries}from"../create-in-place.js";import{DEFAULT_CONNECT_PACKAGE_VERSION,resolveVersionToken}from"../version-tokens.js";import{applyPackageManagerWorkspaceConfiguration,isPackageManagerWorkspaceMember,patchWorkspaceRootPackageJson}from"../workspace-root.js";import{WEB_CHANNEL_TEMPLATES}from"./web-template.js";import{AGENT_INSTRUCTIONS_TEMPLATE}from"./instructions-template.js";import{SCAFFOLDED_AGENT_PATHS}from"./agent-paths.js";import{mkdir,readdir,stat}from"node:fs/promises";import{basename,join,resolve}from"node:path";const CURRENT_DIRECTORY_PROJECT_NAME=`.`,DEFAULT_AI_PACKAGE_VERSION=`^7.0.137`,DEFAULT_ZOD_PACKAGE_VERSION=`4.5.4`,DEFAULT_EVE_PACKAGE_CONTRACT={version:`0.83.
|
|
1
|
+
import{pinnedNodeEngineMajor}from"../../node-engine.js";import{parseChatGptModelSelection}from"../../../shared/chatgpt-model.js";import{SUPPORTED_AUTHORED_MODULE_FILE_EXTENSIONS}from"../update/module-files.js";import{pathExists,writeTextFile}from"../files.js";import{blockingCreateInPlaceEntries}from"../create-in-place.js";import{DEFAULT_CONNECT_PACKAGE_VERSION,resolveVersionToken}from"../version-tokens.js";import{applyPackageManagerWorkspaceConfiguration,isPackageManagerWorkspaceMember,patchWorkspaceRootPackageJson}from"../workspace-root.js";import{WEB_CHANNEL_TEMPLATES}from"./web-template.js";import{AGENT_INSTRUCTIONS_TEMPLATE}from"./instructions-template.js";import{SCAFFOLDED_AGENT_PATHS}from"./agent-paths.js";import{mkdir,readdir,stat}from"node:fs/promises";import{basename,join,resolve}from"node:path";const CURRENT_DIRECTORY_PROJECT_NAME=`.`,DEFAULT_AI_PACKAGE_VERSION=`^7.0.137`,DEFAULT_ZOD_PACKAGE_VERSION=`4.5.4`,DEFAULT_EVE_PACKAGE_CONTRACT={version:`0.83.2`,nodeEngine:`>=24`};function resolveEvePackageContract(e=DEFAULT_EVE_PACKAGE_CONTRACT){return{version:resolveVersionToken(`evePackage.version`,e.version),nodeEngine:resolveVersionToken(`evePackage.nodeEngine`,e.nodeEngine)}}function modelProviderSlug(e){let t=(e.split(`/`)[0]??``).replaceAll(/[^A-Za-z0-9._-]/gu,``);return t.length>0?t:`anthropic`}function byokProviderEnvVar(e){let t=modelProviderSlug(e).toUpperCase().replaceAll(/[^A-Z0-9]/gu,`_`);return`${/^[0-9]/.test(t)?`_`:``}${t}_API_KEY`}function agentTemplateFiles(e,t){return{[SCAFFOLDED_AGENT_PATHS.config]:renderAgentTemplate(e,t),[SCAFFOLDED_AGENT_PATHS.channel]:WEB_CHANNEL_TEMPLATES.default,[SCAFFOLDED_AGENT_PATHS.instructions]:AGENT_INSTRUCTIONS_TEMPLATE}}function renderAgentTemplate(e,t){let n=parseChatGptModelSelection(e);return n===void 0?BASE_AGENT_TEMPLATE.replaceAll(`__EVE_INIT_MODEL__`,e).replaceAll(`__EVE_INIT_REASONING__`,reasoningTemplateLine(t)):`import { defineAgent } from "eve";\nimport { chatgpt } from "eve/models/openai";\n\nexport default defineAgent({\n model: chatgpt(${JSON.stringify(n)}),\n${reasoningTemplateLine(t)}});\n`}function reasoningTemplateLine(e){return e===void 0||e===`provider-default`?``:` reasoning: "${e}",\n`}function renderTemplate(e,t){return e===BASE_AGENT_TEMPLATE&&parseChatGptModelSelection(t.model)!==void 0?renderAgentTemplate(t.model,t.reasoning):e.replaceAll(`__EVE_INIT_APP_NAME__`,t.appName).replaceAll(`__EVE_INIT_MODEL__`,t.model).replaceAll(`__EVE_INIT_REASONING__`,reasoningTemplateLine(t.reasoning)).replaceAll(`__EVE_INIT_BYOK_PROVIDER__`,modelProviderSlug(t.model)).replaceAll(`__EVE_INIT_BYOK_ENV_VAR__`,byokProviderEnvVar(t.model)).replaceAll(`__EVE_INIT_PACKAGE_VERSION__`,formatEveDependencySpecifier(t.eveVersion)).replaceAll(`__EVE_INIT_AI_SDK_VERSION__`,t.aiPackageVersion).replaceAll(`__EVE_INIT_CONNECT_VERSION__`,t.connectPackageVersion).replaceAll(`__EVE_INIT_ZOD_VERSION__`,t.zodPackageVersion).replaceAll(`__EVE_INIT_TYPESCRIPT_VERSION__`,t.typescriptPackageVersion).replaceAll(`__EVE_INIT_TYPES_NODE_VERSION__`,t.nodeTypesVersion).replaceAll(`__EVE_INIT_NODE_ENGINE__`,t.nodeEngine)}function formatEveDependencySpecifier(e){return/^\d+\.\d+\.\d+(?:[-+][0-9A-Za-z-.]+)?$/.test(e)?`^${e}`:e}const BASE_AGENT_TEMPLATE=`import { defineAgent } from "eve";
|
|
2
2
|
|
|
3
3
|
export default defineAgent({
|
|
4
4
|
model: "__EVE_INIT_MODEL__",
|
package/dist/src/svelte/index.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { a as conversationProjection, i as openConversationInputs, n as conversationReducer, o as callStatus, r as defaultMessageReducer, t as useEveAgent } from "../chunks/use-eve-agent-
|
|
1
|
+
import { a as conversationProjection, i as openConversationInputs, n as conversationReducer, o as callStatus, r as defaultMessageReducer, t as useEveAgent } from "../chunks/use-eve-agent-ZY7NQ2VW.js";
|
|
2
2
|
|
|
3
3
|
//#region src/client/tool-call-state.ts
|
|
4
4
|
function toolCallState(conversation, callId, options = {}) {
|
|
@@ -1 +1 @@
|
|
|
1
|
-
import{collectDurableDynamicToolCallbacks,stampDurableDynamicToolCallbacks}from"#tools/durable-callbacks.js";import{TOOL_BRAND}from"#tools/dynamic.js";import{stampDefinitionKey}from"#internal/authored-definition/source-identity.js";function defineTool(e){return stampToolDefinition(e,`defineTool`)}function stampToolDefinition(e,t){if(e.auth!==void 0)throw Error(`${t}: The "auth" field is no longer supported. Pass auth providers inline to ctx.getToken(provider) or ctx.requireAuth(provider).`);return Object.assign(e,{[TOOL_BRAND]:!0}),stampDurableDynamicToolCallbacks(e,collectDurableDynamicToolCallbacks({inputSchema:e.inputSchema,outputSchema:e.outputSchema,label:e.label,approval:e.approval,approvalKey:e.approvalKey,execute:e.execute,toModelOutput:e.toModelOutput})),stampDefinitionKey(e,`tool:${e.description}`),e}const DISABLED_TOOL_SENTINEL_KIND=`eve:disabled-tool`;function disableTool(){return{kind:DISABLED_TOOL_SENTINEL_KIND}}function isDisabledToolSentinel(e){return typeof e==`object`&&!!e&&e.kind===DISABLED_TOOL_SENTINEL_KIND}export{defineTool,disableTool,isDisabledToolSentinel,stampToolDefinition};
|
|
1
|
+
import{collectDurableDynamicToolCallbacks,stampDurableDynamicToolCallbacks}from"#tools/durable-callbacks.js";import{TOOL_BRAND}from"#tools/dynamic.js";import{stampDefinitionKey}from"#internal/authored-definition/source-identity.js";function defineTool(e){return stampToolDefinition(e,`defineTool`)}function stampToolDefinition(e,t){if(e.auth!==void 0)throw Error(`${t}: The "auth" field is no longer supported. Pass auth providers inline to ctx.getToken(provider) or ctx.requireAuth(provider).`);return Object.assign(e,{[TOOL_BRAND]:!0}),stampDurableDynamicToolCallbacks(e,collectDurableDynamicToolCallbacks({inputSchema:e.inputSchema,outputSchema:e.outputSchema,label:e.label,approval:e.approval,approvalKey:e.approvalKey,endsTurn:e.endsTurn,execute:e.execute,toModelOutput:e.toModelOutput})),stampDefinitionKey(e,`tool:${e.description}`),e}const DISABLED_TOOL_SENTINEL_KIND=`eve:disabled-tool`;function disableTool(){return{kind:DISABLED_TOOL_SENTINEL_KIND}}function isDisabledToolSentinel(e){return typeof e==`object`&&!!e&&e.kind===DISABLED_TOOL_SENTINEL_KIND}export{defineTool,disableTool,isDisabledToolSentinel,stampToolDefinition};
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import type { ToolApproval } from "#approval/definition.js";
|
|
2
2
|
import type { JsonObject, JsonPrimitive } from "#shared/json.js";
|
|
3
|
-
export type DurableDynamicCallbackPhase = "labelComplete" | "labelDelta" | "labelStart" | "approvalKey" | "approvalPrompt" | "approvalRequest" | "approvalResponse" | "execute" | "toModelOutput" | "inputSchema" | "outputSchema";
|
|
3
|
+
export type DurableDynamicCallbackPhase = "labelComplete" | "labelDelta" | "labelStart" | "approvalKey" | "approvalPrompt" | "approvalRequest" | "approvalResponse" | "endsTurn" | "execute" | "toModelOutput" | "inputSchema" | "outputSchema";
|
|
4
4
|
export type DurableDynamicCallbackFn = (closure: JsonObject, ...args: never[]) => unknown;
|
|
5
5
|
/**
|
|
6
6
|
* Persisted binding for one callback phase. Identity is
|
|
@@ -13,6 +13,7 @@ export interface DurableDynamicCallbackReference {
|
|
|
13
13
|
readonly stepId?: never;
|
|
14
14
|
}
|
|
15
15
|
export interface DurableDynamicToolCallbacks {
|
|
16
|
+
readonly endsTurn?: DurableDynamicCallbackReference;
|
|
16
17
|
readonly execute: DurableDynamicCallbackReference;
|
|
17
18
|
readonly inputSchema?: DurableDynamicCallbackReference;
|
|
18
19
|
readonly outputSchema?: DurableDynamicCallbackReference;
|
|
@@ -33,6 +34,7 @@ export interface StampedDurableDynamicCallback {
|
|
|
33
34
|
readonly closure: JsonObject;
|
|
34
35
|
}
|
|
35
36
|
type LiveDurableDynamicToolCallbacks = Partial<{
|
|
37
|
+
endsTurn: StampedDurableDynamicCallback;
|
|
36
38
|
execute: StampedDurableDynamicCallback;
|
|
37
39
|
inputSchema: StampedDurableDynamicCallback;
|
|
38
40
|
outputSchema: StampedDurableDynamicCallback;
|
|
@@ -107,6 +109,8 @@ export declare function collectDurableDynamicToolCallbacks(input: {
|
|
|
107
109
|
};
|
|
108
110
|
readonly approval?: ToolApproval<never>;
|
|
109
111
|
readonly approvalKey?: (...args: never[]) => unknown;
|
|
112
|
+
/** A boolean `endsTurn` is durable metadata already; only a function needs a descriptor. */
|
|
113
|
+
readonly endsTurn?: unknown;
|
|
110
114
|
readonly execute?: (...args: never[]) => unknown;
|
|
111
115
|
readonly toModelOutput?: (...args: never[]) => unknown;
|
|
112
116
|
}): LiveDurableDynamicToolCallbacks;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
import{readDurableSchema}from"#tools/durable-schema.js";import{resolveApprovalPolicy}from"#approval/definition.js";const STAMPED_CALLBACK=Symbol.for(`eve:durable-dynamic-callback`),DURABLE_DYNAMIC_TOOL_CALLBACKS=Symbol.for(`eve:durable-dynamic-tool-callbacks`),REGISTRY=Symbol.for(`eve:scoped-dynamic-tool-callbacks`);function getRegistry(){let e=globalThis;return e[REGISTRY]??=new Map}function getSessionBindings(e){let t=getRegistry(),n=t.get(e);return n!==void 0&&(t.delete(e),t.set(e,n)),n}function callbackKey(e,t){return JSON.stringify([e.scope,e.resolverSlug,e.entryKey,e.name,t])}function registerDurableDynamicCallback(e){let t=getRegistry(),n=getSessionBindings(e.owner.sessionId);n===void 0&&(n=new Map,t.set(e.owner.sessionId,n),t.size>1024&&t.delete(t.keys().next().value)),n.set(callbackKey(e.owner,e.phase),e.callback)}function lookupDurableDynamicCallback(e,t){return getSessionBindings(e.sessionId)?.get(callbackKey(e,t))}function clearDurableDynamicCallbacks(e,t){if(t===void 0){getRegistry().delete(e);return}let n=getSessionBindings(e);if(n===void 0)return;let r=JSON.stringify([t.scope,t.resolverSlug]).slice(0,-1)+`,`;for(let e of n.keys())e.startsWith(r)&&n.delete(e)}function callDurableDynamicCallback(e,t,...n){return e(t,...n)}function hasUnregisteredDurableDynamicCallbacks(e,t){return e.some(e=>durableCallbackPhases(e.callbacks).some(n=>lookupDurableDynamicCallback({...e,...t},n)===void 0))}function durableCallbackPhases(e){return[[`execute`,e.execute],[`inputSchema`,e.inputSchema],[`outputSchema`,e.outputSchema],[`labelComplete`,e.label?.complete],[`labelDelta`,e.label?.delta],[`labelStart`,e.label?.start],[`approvalKey`,e.approvalKey],[`approvalPrompt`,e.approvalPrompt],[`approvalRequest`,e.approvalRequest],[`approvalResponse`,e.approvalResponse],[`toModelOutput`,e.toModelOutput]].flatMap(([e,t])=>t===void 0?[]:[e])}function defineDurableCallback(e){return stampDurableDynamicCallback((...t)=>e.callback(e.closure,...t),{callback:(t,...n)=>Reflect.apply(e.callback,void 0,[t,...n]),closure:e.closure})}function stampDurableDynamicCallback(e,t){return Object.defineProperty(e,STAMPED_CALLBACK,{configurable:!0,value:t}),e}function readDurableDynamicCallback(e){if(typeof e==`function`)return Reflect.get(e,STAMPED_CALLBACK)}function stampDurableDynamicToolCallbacks(e,t){Object.defineProperty(e,DURABLE_DYNAMIC_TOOL_CALLBACKS,{configurable:!0,value:t})}function collectDurableDynamicToolCallbacks(e){let t=readDurableDynamicCallback(e.label?.complete),n=readDurableDynamicCallback(e.label?.delta),r=readDurableDynamicCallback(e.label?.start),i=e.approval===void 0?void 0:readDurableDynamicCallback(resolveApprovalPolicy(e.approval)),a=e.approval===void 0||typeof e.approval==`function`?void 0:readDurableDynamicCallback(e.approval.response),o=readDurableDynamicCallback(e.approvalKey),s=readDurableDynamicCallback(e.approval===void 0||typeof e.approval==`function`?void 0:e.approval.prompt),c=readDurableDynamicCallback(e.
|
|
1
|
+
import{readDurableSchema}from"#tools/durable-schema.js";import{resolveApprovalPolicy}from"#approval/definition.js";const STAMPED_CALLBACK=Symbol.for(`eve:durable-dynamic-callback`),DURABLE_DYNAMIC_TOOL_CALLBACKS=Symbol.for(`eve:durable-dynamic-tool-callbacks`),REGISTRY=Symbol.for(`eve:scoped-dynamic-tool-callbacks`);function getRegistry(){let e=globalThis;return e[REGISTRY]??=new Map}function getSessionBindings(e){let t=getRegistry(),n=t.get(e);return n!==void 0&&(t.delete(e),t.set(e,n)),n}function callbackKey(e,t){return JSON.stringify([e.scope,e.resolverSlug,e.entryKey,e.name,t])}function registerDurableDynamicCallback(e){let t=getRegistry(),n=getSessionBindings(e.owner.sessionId);n===void 0&&(n=new Map,t.set(e.owner.sessionId,n),t.size>1024&&t.delete(t.keys().next().value)),n.set(callbackKey(e.owner,e.phase),e.callback)}function lookupDurableDynamicCallback(e,t){return getSessionBindings(e.sessionId)?.get(callbackKey(e,t))}function clearDurableDynamicCallbacks(e,t){if(t===void 0){getRegistry().delete(e);return}let n=getSessionBindings(e);if(n===void 0)return;let r=JSON.stringify([t.scope,t.resolverSlug]).slice(0,-1)+`,`;for(let e of n.keys())e.startsWith(r)&&n.delete(e)}function callDurableDynamicCallback(e,t,...n){return e(t,...n)}function hasUnregisteredDurableDynamicCallbacks(e,t){return e.some(e=>durableCallbackPhases(e.callbacks).some(n=>lookupDurableDynamicCallback({...e,...t},n)===void 0))}function durableCallbackPhases(e){return[[`endsTurn`,e.endsTurn],[`execute`,e.execute],[`inputSchema`,e.inputSchema],[`outputSchema`,e.outputSchema],[`labelComplete`,e.label?.complete],[`labelDelta`,e.label?.delta],[`labelStart`,e.label?.start],[`approvalKey`,e.approvalKey],[`approvalPrompt`,e.approvalPrompt],[`approvalRequest`,e.approvalRequest],[`approvalResponse`,e.approvalResponse],[`toModelOutput`,e.toModelOutput]].flatMap(([e,t])=>t===void 0?[]:[e])}function defineDurableCallback(e){return stampDurableDynamicCallback((...t)=>e.callback(e.closure,...t),{callback:(t,...n)=>Reflect.apply(e.callback,void 0,[t,...n]),closure:e.closure})}function stampDurableDynamicCallback(e,t){return Object.defineProperty(e,STAMPED_CALLBACK,{configurable:!0,value:t}),e}function readDurableDynamicCallback(e){if(typeof e==`function`)return Reflect.get(e,STAMPED_CALLBACK)}function stampDurableDynamicToolCallbacks(e,t){Object.defineProperty(e,DURABLE_DYNAMIC_TOOL_CALLBACKS,{configurable:!0,value:t})}function collectDurableDynamicToolCallbacks(e){let t=readDurableDynamicCallback(e.label?.complete),n=readDurableDynamicCallback(e.label?.delta),r=readDurableDynamicCallback(e.label?.start),i=e.approval===void 0?void 0:readDurableDynamicCallback(resolveApprovalPolicy(e.approval)),a=e.approval===void 0||typeof e.approval==`function`?void 0:readDurableDynamicCallback(e.approval.response),o=readDurableDynamicCallback(e.approvalKey),s=readDurableDynamicCallback(e.approval===void 0||typeof e.approval==`function`?void 0:e.approval.prompt),c=readDurableDynamicCallback(e.endsTurn),l=readDurableDynamicCallback(e.execute),u=readDurableDynamicCallback(e.toModelOutput),d={},f=readDurableSchema(e.inputSchema),p=readDurableSchema(e.outputSchema);return f!==void 0&&(d.inputSchema=f),p!==void 0&&(d.outputSchema=p),c!==void 0&&(d.endsTurn=c),l!==void 0&&(d.execute=l),(t!==void 0||n!==void 0||r!==void 0)&&(d.label={complete:t,delta:n,start:r}),o!==void 0&&(d.approvalKey=o),s!==void 0&&(d.approvalPrompt=s),i!==void 0&&(d.approvalRequest=i),a!==void 0&&(d.approvalResponse=a),u!==void 0&&(d.toModelOutput=u),d}function readDurableDynamicToolCallbacks(e){return e[DURABLE_DYNAMIC_TOOL_CALLBACKS]}export{callDurableDynamicCallback,clearDurableDynamicCallbacks,collectDurableDynamicToolCallbacks,defineDurableCallback,hasUnregisteredDurableDynamicCallbacks,lookupDurableDynamicCallback,readDurableDynamicCallback,readDurableDynamicToolCallbacks,registerDurableDynamicCallback,stampDurableDynamicCallback,stampDurableDynamicToolCallbacks};
|
|
@@ -22,9 +22,9 @@ export interface DynamicToolEntry<TInput = Record<string, unknown>, TOutput = an
|
|
|
22
22
|
readonly label?: ToolLabelDefinition<TInput, TOutput>;
|
|
23
23
|
readonly description: string;
|
|
24
24
|
/**
|
|
25
|
-
* Ends the turn after a successful call, as `endsTurn
|
|
26
|
-
* `defineTool`.
|
|
27
|
-
*
|
|
25
|
+
* Ends the turn after a successful call, as `endsTurn` on a static
|
|
26
|
+
* `defineTool`. A function is replayed from its durable descriptor, like
|
|
27
|
+
* `execute`.
|
|
28
28
|
*/
|
|
29
29
|
readonly endsTurn?: boolean | ((output: TOutput) => boolean | Promise<boolean>);
|
|
30
30
|
readonly inputSchema: PublicToolInputSchema<TInput>;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
async function executeAskQuestionTool(e,t){"use workflow";return toAskQuestionOutput(await t.ask(toAskQuestionRequest(e)))}function toAskQuestionRequest(e){let t=e.options?.map(e=>({description:e.description,id:e.label,label:e.label}));return{allowFreeform:!0,display:t===void 0?`text`:`select`,prompt:e.question,...t!==void 0&&{options:t}}}function toAskQuestionOutput(e){switch(e.status){case`answered`:return{answer:e.optionId??e.text??``,status:`answered`};case`cancelled`:return{interrupted:!0};case`unavailable`:return{status:`unavailable`}}}executeAskQuestionTool.workflowId=`workflow//eve@0.83.
|
|
1
|
+
async function executeAskQuestionTool(e,t){"use workflow";return toAskQuestionOutput(await t.ask(toAskQuestionRequest(e)))}function toAskQuestionRequest(e){let t=e.options?.map(e=>({description:e.description,id:e.label,label:e.label}));return{allowFreeform:!0,display:t===void 0?`text`:`select`,prompt:e.question,...t!==void 0&&{options:t}}}function toAskQuestionOutput(e){switch(e.status){case`answered`:return{answer:e.optionId??e.text??``,status:`answered`};case`cancelled`:return{interrupted:!0};case`unavailable`:return{status:`unavailable`}}}executeAskQuestionTool.workflowId=`workflow//eve@0.83.2//executeAskQuestionTool`;export{executeAskQuestionTool,toAskQuestionOutput,toAskQuestionRequest};
|
|
@@ -1 +1 @@
|
|
|
1
|
-
import{sleep}from"#compiled/@workflow/core/index.js";async function executeSleepTool(e,t){"use workflow";if(t.abortSignal.aborted)return{interrupted:!0};let n=new Promise(e=>t.abortSignal.addEventListener(`abort`,()=>e(`interrupted`),{once:!0})),r=sleep(Math.ceil(e.seconds*1e3)).then(()=>`elapsed`);return await Promise.race([r,n])===`elapsed`?{waitedSeconds:e.seconds}:{interrupted:!0}}executeSleepTool.workflowId=`workflow//eve@0.83.
|
|
1
|
+
import{sleep}from"#compiled/@workflow/core/index.js";async function executeSleepTool(e,t){"use workflow";if(t.abortSignal.aborted)return{interrupted:!0};let n=new Promise(e=>t.abortSignal.addEventListener(`abort`,()=>e(`interrupted`),{once:!0})),r=sleep(Math.ceil(e.seconds*1e3)).then(()=>`elapsed`);return await Promise.race([r,n])===`elapsed`?{waitedSeconds:e.seconds}:{interrupted:!0}}executeSleepTool.workflowId=`workflow//eve@0.83.2//executeSleepTool`;export{executeSleepTool};
|
package/dist/src/vue/index.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { a as conversationProjection, i as openConversationInputs, n as conversationReducer, o as callStatus, r as defaultMessageReducer, t as useEveAgent } from "../chunks/use-eve-agent-
|
|
1
|
+
import { a as conversationProjection, i as openConversationInputs, n as conversationReducer, o as callStatus, r as defaultMessageReducer, t as useEveAgent } from "../chunks/use-eve-agent-DnbG9c7c.js";
|
|
2
2
|
|
|
3
3
|
//#region src/client/tool-call-state.ts
|
|
4
4
|
function toolCallState(conversation, callId, options = {}) {
|
package/docs/channels/custom.mdx
CHANGED
|
@@ -147,6 +147,10 @@ await session.reset({ reason: "Retire this session" });
|
|
|
147
147
|
await session.getEventStream({ startIndex: 12 });
|
|
148
148
|
```
|
|
149
149
|
|
|
150
|
+
A following `getEventStream()` rejects before it returns a stream when the session does not exist
|
|
151
|
+
or its recorded history has expired, so a route can answer with an error status before it sends any
|
|
152
|
+
bytes.
|
|
153
|
+
|
|
150
154
|
For a stranded session, `Session.send()`, `Session.respond()`, and a following
|
|
151
155
|
`Session.getEventStream()` throw `SessionStrandedError` from `eve/channels` and deliver nothing.
|
|
152
156
|
`getEventStream({ follow: false })` still reads the recorded history up to the current tail without
|
package/docs/channels/slack.mdx
CHANGED
|
@@ -189,6 +189,8 @@ Keep authorization checks inside sensitive tools too. The inbound allowlist cont
|
|
|
189
189
|
|
|
190
190
|
`onInputResponse` runs after eve decodes a built-in HITL answer but before the parked session resumes. Return `{ auth }` to accept the answer. Returning `null` or throwing rejects it, leaves the request pending, and keeps the Slack controls active. `ctx.defaultAuth` identifies the Slack user who submitted the signed interaction; it does not authorize that user by itself.
|
|
191
191
|
|
|
192
|
+
A message typed in a thread or DM never passes through `onInputResponse`, so while it or `approvalChannel` is set, a typed message doesn't answer a pending request. Answers come only through the Slack controls. The typed message reaches the agent as a new message and follows the [turn policy](#control-overlapping-turns). With the default steering policy, it withdraws an open question.
|
|
193
|
+
|
|
192
194
|
If you omit `onInputResponse`, Slack accepts HITL answers with the submitting user's auth, regardless of which message or event handlers you define. Define `onInputResponse` when those handlers enforce an invocation policy that must also apply to HITL answers. Put sensitive prompts in conversations limited to the intended approvers, and re-check `ctx.session.auth.current` inside a sensitive tool before performing its side effect.
|
|
193
195
|
|
|
194
196
|
#### Continue conversations without repeated mentions
|
|
@@ -337,7 +337,7 @@ Entries the agent declares count from the session's first step. An entry a dynam
|
|
|
337
337
|
- A query whose first word contains `__` searches one namespace: everything before its last `__`, without trailing underscores. The namespace only filters. `eve__search` keeps names under `<namespace>__` and anything named exactly the namespace, such as a connection's sign-in entry, then ranks them by the whole query, so `eve__search({ query: "linear__" })` returns only the `linear` connection's tools, or its sign-in result, and an exact name that contains `__` still ranks first. eve lists only the connections that can own names in that namespace, so other connections make no network call and don't appear in `unavailable`. A namespace that matches nothing fails with the closest connection names. Characters that can't appear in a name, such as a leading `^`, are ignored; `query` isn't a regular expression.
|
|
338
338
|
- `eve__search` never asks the user to sign in. A connection whose server won't list its tools until the user signs in appears as one result named after the connection, such as `linear`; a connection whose server lists its tools without a token is searched like any other. A connection whose tools fail to load, or don't load within 10 seconds, appears under `unavailable` with its `error`, such as `"crm" did not list its tools within 10s. Try again later.` Matches from everything else are still returned, so a slow server doesn't hold up the search. An `eve__tool` call on that connection's tools fails with the same error.
|
|
339
339
|
- `eve__tool({ name: "linear" })`, with a connection's own name, asks the user to sign in to that connection when its tools need it, and returns once they can be listed, so the next `eve__search` finds them. On a server that lists its tools without sign-in, it confirms the tools are available without asking.
|
|
340
|
-
- `eve__tool({ name, input? })` calls the entry named `name` with `input`, which defaults to `{}`. A connection tool's name is `<connection>__<tool>`, such as `linear__list_issues`. eve checks `input` against the entry's input schema. An invalid input fails with the entry's signature, an unknown name fails with the closest names, and a skill's name fails with a reminder to load it with `eve__skill`. When the server asks for the user's authorization, the call asks the user to sign in and parks until sign-in completes.
|
|
340
|
+
- `eve__tool({ name, input? })` calls the entry named `name` with `input`, which defaults to `{}`. A connection tool's name is `<connection>__<tool>`, such as `linear__list_issues`. eve checks `input` against the entry's input schema. An invalid input fails with the entry's signature, an unknown name fails with the closest names, and a skill's name fails with a reminder to load it with `eve__skill`. When the server asks for the user's authorization, the call asks the user to sign in and parks until sign-in completes. A connection tool call the server doesn't answer within 5 minutes is cancelled and fails with an error such as `"linear" did not answer list_issues within 5 minutes. It may still have run, so check before you retry.` The server may finish the call after eve stops waiting, so a tool that changes data may already have made the change.
|
|
341
341
|
- `eve__skill({ name })` loads the named skill, deferred or not, and returns its instructions. An unknown name fails with the closest skill names, a tool's name fails with a reminder to call it with `eve__tool` or directly, and a name that matches a connection says to find its tools with `eve__search({ query: "<connection>__" })`.
|
|
342
342
|
|
|
343
343
|
After `eve__tool` or `eve__skill` resolves its entry, the call runs exactly like a direct call to that entry. A skill load reports a `load-skill` action and a `load-skill-result`, both with the skill's `name`, and an `execute_tool eve:load-skill` span. Approval policies and `approvedTools`, workflow tools and tasks, agents, `endsTurn`, `toModelOutput`, hooks, and stream events all see the entry's own name and input, such as `linear__list_issues`. Only model history records the call as `eve__tool` or `eve__skill`.
|
package/docs/connections/mcp.mdx
CHANGED
|
@@ -26,7 +26,7 @@ The `url` must speak Streamable HTTP or SSE. Write the `description` for the mod
|
|
|
26
26
|
|
|
27
27
|
## Configure protocol discovery
|
|
28
28
|
|
|
29
|
-
eve discovers the server's MCP protocol by default. If a server requires the older `initialize` handshake, disable discovery for that connection:
|
|
29
|
+
eve discovers the server's MCP protocol by default. If a server rejects discovery with an unsupported protocol version error (`-32022`) and lists a handshake-based version eve supports, eve falls back to `initialize` with the newest version both sides support. If a server requires the older `initialize` handshake and does not report a supported version, disable discovery for that connection:
|
|
30
30
|
|
|
31
31
|
```ts
|
|
32
32
|
export default defineMcpClientConnection({
|
package/docs/evals/overview.mdx
CHANGED
|
@@ -58,7 +58,7 @@ export default defineEvalConfig({
|
|
|
58
58
|
});
|
|
59
59
|
```
|
|
60
60
|
|
|
61
|
-
Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge(...)`); omit it to use the default decider, `typesafe-ai/jev`. Deterministic evals make no judge requests. `reporters`, `maxConcurrency`, and `
|
|
61
|
+
Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge(...)`); omit it to use the default decider, `typesafe-ai/jev`. Deterministic evals make no judge requests. `reporters`, `maxConcurrency`, `timeoutMs`, and `repeat` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. `repeat` runs every eval several times to measure repeatability; see [Repeat evals](./running#repeat-evals). CLI flags (`--max-concurrency`, `--timeout`, `--repeat`) and per-eval values take precedence over the config defaults.
|
|
62
62
|
|
|
63
63
|
### Run setup and teardown
|
|
64
64
|
|
package/docs/evals/reporters.mdx
CHANGED
|
@@ -38,7 +38,7 @@ export default defineEval({
|
|
|
38
38
|
});
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
-
The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. The console warns under the eval when merged soft scores differ; use `.label(name)` to group or separate them. Errored assertions log no score. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.
|
|
41
|
+
The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. The console warns under the eval when merged soft scores differ; use `.label(name)` to group or separate them. Errored assertions log no score. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`. In a [repeated run](./running#repeat-evals), each attempt logs its own row with the id `<id>#<attempt>` and the attempt number under `eveAttempt`.
|
|
42
42
|
|
|
43
43
|
A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
|
|
44
44
|
|
|
@@ -61,7 +61,7 @@ The reporter creates one Datadog Experiment, submits one synthetic experiment sp
|
|
|
61
61
|
|
|
62
62
|
`dd-trace` generates dataset record, experiment trace, and experiment span identifiers. For sampled eve sessions, the reporter also adds identifier-only `experimentRuntimeTraceLinks` metadata, holding the IDs Datadog assigns when it indexes the runtime traces, so an Experiment row can link to the agent trace that produced it. Runtime spans keep their stored IDs and parentage.
|
|
63
63
|
|
|
64
|
-
Every run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records are matched by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description.
|
|
64
|
+
Every run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records are matched by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description. In a [repeated run](./running#repeat-evals), every attempt of an eval links to that eval's single dataset record, and each experiment span's `runIteration` is the attempt number.
|
|
65
65
|
|
|
66
66
|
Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, assertions that share a label report one metric with the lowest score, and labels that collide with built-in metrics receive a numeric suffix. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Each assertion's name and score are kept in row metadata under `eveAssertionScores`; pass `recordAssertionDetails: false` to omit them and failed assertion messages.
|
|
67
67
|
|
|
@@ -75,7 +75,7 @@ Datadog needs `dd-trace` installed in the app and Datadog credentials in the env
|
|
|
75
75
|
eve eval --strict --junit .eve/junit.xml
|
|
76
76
|
```
|
|
77
77
|
|
|
78
|
-
Each eval becomes one `<testcase>` named by its path-derived id; failed gates and execution errors become failures, while `t.skip(reason)` produces a JUnit `<skipped>` result.
|
|
78
|
+
Each eval becomes one `<testcase>` named by its path-derived id (one per attempt, named `<id>#<attempt>`, in a repeated run); failed gates and execution errors become failures, while `t.skip(reason)` produces a JUnit `<skipped>` result.
|
|
79
79
|
|
|
80
80
|
## Custom reporters
|
|
81
81
|
|
|
@@ -93,7 +93,7 @@ interface EvalReporter {
|
|
|
93
93
|
|
|
94
94
|
`onRunStart` fires once before any eval runs, and `onRunComplete` fires once with the aggregated summary. Within that run:
|
|
95
95
|
|
|
96
|
-
- `onEvalStart` fires when an eval is scheduled. It includes the eval definition, target, and start time.
|
|
96
|
+
- `onEvalStart` fires when an eval is scheduled. It includes the eval definition, target, and start time. In a [repeated run](./running#repeat-evals), it fires once per attempt, and the event and each `onSessionStart` event carry the 1-based `attempt`, which matches `result.attempt` on completion.
|
|
97
97
|
- `onSessionStart` fires once for each session after eve receives its first trace context. It includes `sessionId`, `primary`, and `traceContext` with `traceId`, `spanId`, and `traceFlags`.
|
|
98
98
|
- `onEvalComplete` fires with the checks, scores, and verdict. The runner also supplies `context`, including every distinct trace context collected across the eval's sessions. The same list is available as `result.result.traceContexts`.
|
|
99
99
|
|
package/docs/evals/running.mdx
CHANGED
|
@@ -14,11 +14,13 @@ eve eval --exclude-tag slow # skip evals carrying a tag
|
|
|
14
14
|
eve eval --strict # soft below-threshold assertions also fail the exit code
|
|
15
15
|
eve eval --timeout 60000 # per-eval timeout in milliseconds
|
|
16
16
|
eve eval --max-concurrency 4 # cap concurrent eval executions (default 8)
|
|
17
|
+
eve eval --repeat 5 # run every eval 5 times to measure repeatability
|
|
17
18
|
eve eval --junit .eve/junit.xml # write JUnit XML
|
|
18
19
|
eve eval --list # print discovered evals without running
|
|
19
20
|
eve eval --verbose # stream per-eval logs and workflow run IDs
|
|
20
21
|
eve eval --json # machine-readable output
|
|
21
22
|
eve eval --skip-report # skip config and eval-defined reporters (e.g. Braintrust)
|
|
23
|
+
eve eval --keep-state # keep the local server's state and Workflow data after the run
|
|
22
24
|
```
|
|
23
25
|
|
|
24
26
|
Positional ids match exactly or by directory prefix: `eve eval weather` runs `evals/weather.eval.ts`, every eval under `evals/weather/`, and every entry of an array-exported `weather.eval.ts`.
|
|
@@ -39,9 +41,25 @@ An eval that calls `t.skip(reason)` is reported as skipped, does not count as pa
|
|
|
39
41
|
|
|
40
42
|
With `--verbose`, eve prefixes each line with its eval ID. It streams authored `t.log()` lines as they occur, then prints the workflow run ID for every captured primary and secondary session. Use those `wrun_...` IDs to find the corresponding runs in your workflow provider.
|
|
41
43
|
|
|
44
|
+
## Repeat evals
|
|
45
|
+
|
|
46
|
+
Model-backed evals can pass on one run and fail on the next. To measure how often an eval passes, pass `--repeat <n>` or set `repeat` in `evals.config.ts`, and eve runs every selected eval `n` times in one run. The CLI flag overrides the config value. The default is `1`, and output does not change at that value.
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
eve eval weather --repeat 5
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
With `n` greater than `1`, each attempt is a separate result with a 1-based `attempt` field:
|
|
53
|
+
|
|
54
|
+
- The console and `--verbose` lines label each attempt `<id>#<attempt>`, and the summary adds a per-eval pass rate such as `3/5 passed`. The pass rate counts attempts that passed or failed their gates; scored and skipped attempts are left out.
|
|
55
|
+
- `--json` results, JUnit test cases (named `<id>#<attempt>`), and [reporters](./reporters) receive one result per attempt.
|
|
56
|
+
- Every attempt counts toward the run totals and the exit code, so one failed attempt fails the run.
|
|
57
|
+
|
|
58
|
+
Attempts share the `--max-concurrency` pool with every other eval, so attempts of the same eval can run at the same time. If an eval depends on shared state or a rate-limited provider, lower `--max-concurrency` so contention does not show up as flakiness.
|
|
59
|
+
|
|
42
60
|
## Artifacts
|
|
43
61
|
|
|
44
|
-
Each run drops artifacts under `.eve/evals/<timestamp>/`: a run `summary.json`, a `results.jsonl` index, and per-eval assertion results, verdicts, start and completion times, captured event streams, and `t.log` lines under `evals/`. The console output stays tight on purpose; when an eval fails, the artifact has the full story.
|
|
62
|
+
Each run drops artifacts under `.eve/evals/<timestamp>/`: a run `summary.json`, a `results.jsonl` index, and per-eval assertion results, verdicts, start and completion times, captured event streams with the client's send and arrival times for each session, and `t.log` lines under `evals/`. In a repeated run, each attempt's detail and event files get an `.attempt-<n>` suffix, such as `evals/weather.attempt-2.json`. The console output stays tight on purpose; when an eval fails, the artifact has the full story.
|
|
45
63
|
|
|
46
64
|
## CI
|
|
47
65
|
|
|
@@ -248,7 +248,7 @@ the handler.
|
|
|
248
248
|
|
|
249
249
|
## Dynamic tools
|
|
250
250
|
|
|
251
|
-
Pass `defineDynamic` an `events` object whose handlers return either a single `defineTool(...)`, a `Record<string, defineTool(...)>`, or `null` for no tools. Wrap every entry in `defineTool()`. eve records durable descriptors for `execute`, approval request and response policies, input-scoped `approvalKey` callbacks,
|
|
251
|
+
Pass `defineDynamic` an `events` object whose handlers return either a single `defineTool(...)`, a `Record<string, defineTool(...)>`, or `null` for no tools. Wrap every entry in `defineTool()`. eve records durable descriptors for `execute`, approval request and response policies, input-scoped `approvalKey` callbacks, `toModelOutput`, and an `endsTurn` function, so a parked call can reconstruct the same callbacks in a fresh process.
|
|
252
252
|
|
|
253
253
|
Dynamic tool executors receive the same `ToolContext` as static authored tools, including inline provider auth through `ctx.getToken(provider)` and `ctx.requireAuth(provider)`.
|
|
254
254
|
|
|
@@ -448,7 +448,7 @@ Skills follow the same naming rule as tools: a single `defineSkill(...)` is name
|
|
|
448
448
|
|
|
449
449
|
A dynamic skill with `deferred: true` is left out of the dynamic skill announcement and isn't named in context. The model finds it with `eve__search` and loads it with `eve__skill({ name })`, like a static deferred skill. When it overrides a static skill that isn't deferred, the system prompt still lists the static skill's description, and loading returns the dynamic body.
|
|
450
450
|
|
|
451
|
-
A dynamic skill that returns only `markdown` never starts a sandbox: eve keeps its instructions in session state and returns them when the model loads the skill. When the skill also returns `files`, eve writes the package to the sandbox skill root when the resolver first returns it, and again only when its contents change or the session gets a new sandbox. Resolvers run when the turn starts, before the sandbox opens, so when eve replaces a sandbox that was removed outside eve, it writes the files into the new sandbox at the start of the next turn. A replacement made by a subagent or workflow step that shares the session's sandbox is not detected: the session treats the files as written until their contents change. A changed package replaces the previous directory, so files omitted from the new result are removed, and a skill the resolver stops returning has its files removed. A skill that keeps its name but stops returning `files` keeps the files already written, since the sandbox may provide them another way, such as from its snapshot. Keep content meant for only some callers in `markdown`, or stop returning the skill for the others.
|
|
451
|
+
A dynamic skill that returns only `markdown` never starts a sandbox: eve keeps its instructions in session state and returns them when the model loads the skill. When the skill also returns `files`, eve writes the package to the sandbox skill root when the resolver first returns it, and again only when its contents change or the session gets a new sandbox. The write does not delay the model call: eve opens the sandbox and writes the package in the background, and a tool or hook that uses the sandbox waits until the write finishes. If the write fails twice in a row, eve logs it and writes the package again the next time the resolver returns it; a resolver that runs only at `session.started` does not run again, so its files stay missing for the rest of the session. Resolvers run when the turn starts, before the sandbox opens, so when eve replaces a sandbox that was removed outside eve, it writes the files into the new sandbox at the start of the next turn. A replacement made by a subagent or workflow step that shares the session's sandbox is not detected: the session treats the files as written until their contents change. A changed package replaces the previous directory, so files omitted from the new result are removed, and a skill the resolver stops returning has its files removed. A skill that keeps its name but stops returning `files` keeps the files already written, since the sandbox may provide them another way, such as from its snapshot. Keep content meant for only some callers in `markdown`, or stop returning the skill for the others.
|
|
452
452
|
|
|
453
453
|
## Dynamic instructions
|
|
454
454
|
|
|
@@ -137,9 +137,13 @@ Prewarming defaults to `false`: the first send creates the session. Set it to `t
|
|
|
137
137
|
your application's auth, headers, and any chat-record binding are ready. Applications that
|
|
138
138
|
prepare those in the submit handler should keep the default until then.
|
|
139
139
|
|
|
140
|
-
Prewarming starts the workflow and establishes its inbox, then parks before session hooks
|
|
141
|
-
dynamic definitions
|
|
142
|
-
|
|
140
|
+
Prewarming starts the workflow and establishes its inbox, then parks before session hooks or
|
|
141
|
+
dynamic definitions run. Meanwhile, it starts the agent's [sandbox](/docs/sandbox) in the
|
|
142
|
+
background. The first message passes through the channel's `onMessage` hook and supplies the
|
|
143
|
+
identity and context for initialization and `turn_0`. That turn does not wait for the sandbox;
|
|
144
|
+
only a tool or hook that uses the sandbox waits for the rest of its start. If the sandbox fails
|
|
145
|
+
to start during prewarming, the first turn that uses it starts it instead. eve deletes the
|
|
146
|
+
sandbox when the session ends, including when an unused prewarmed session times out.
|
|
143
147
|
|
|
144
148
|
For an existing session, `prewarm()` checks the owner against the exact deployment accepting
|
|
145
149
|
that request. On Vercel, an idle owner on another deployment can transfer in the background
|
|
@@ -7,6 +7,29 @@ url: "/observability/otel"
|
|
|
7
7
|
`eve dev` records local traces by default. Use `otel()` for process-wide
|
|
8
8
|
settings and `otelIntegration()` for third-party destinations.
|
|
9
9
|
|
|
10
|
+
## Session handover events
|
|
11
|
+
|
|
12
|
+
Existing Workflow step spans include these events when tracing is active:
|
|
13
|
+
|
|
14
|
+
- `eve.session.handover.forwarding_started`: the previous owner starts submitting its buffered messages, including an empty buffer.
|
|
15
|
+
- `eve.session.handover.part_submitted`: a buffered part was durably submitted.
|
|
16
|
+
- `eve.session.handover.final_submitted`: the final marker was durably submitted. This does not confirm that the successor has processed it.
|
|
17
|
+
- `eve.session.handover.inbox_unblocked`: an existing turn step observes that the inbox applied every buffered part and the final marker.
|
|
18
|
+
|
|
19
|
+
Join both owners with `eve.session.id` and `eve.handoff.attempt_key`.
|
|
20
|
+
`workflow.run.id` identifies the emitting owner; submission events also include
|
|
21
|
+
`eve.handoff.destination_run_id`. Counts and part indexes describe the transfer
|
|
22
|
+
without recording message content. Step IDs and `eve.handoff.step_attempt`
|
|
23
|
+
distinguish retries; these events are not exactly-once counters.
|
|
24
|
+
|
|
25
|
+
The receiver records `eve.handoff.applied_at_ms` using Workflow's replay-stable
|
|
26
|
+
clock, identified by `eve.handoff.applied_at_clock: workflow`. This is logical
|
|
27
|
+
event time, not the host time when replay applied the marker. The event's own
|
|
28
|
+
timestamp is when the next turn step exports the observation. No receiver
|
|
29
|
+
observation is exported if no subsequent turn step runs. An abandoned handover
|
|
30
|
+
does not emit a successful receipt. No additional Workflow steps or writes are
|
|
31
|
+
created for tracing.
|
|
32
|
+
|
|
10
33
|
## Configure OpenTelemetry
|
|
11
34
|
|
|
12
35
|
`otel()` and `otelIntegration()` are optional:
|
package/docs/meta.json
CHANGED
|
@@ -56,7 +56,7 @@ Paths below are relative to the agent directory. Root agents can use every path;
|
|
|
56
56
|
| `extensions/` | Mounted reusable capabilities | Yes | File or directory mounts. See [Extensions](/docs/extensions). |
|
|
57
57
|
| `hooks/` | Lifecycle and stream-event subscribers | Yes | Module-backed only; recursive directories are supported. |
|
|
58
58
|
| `skills/` | On-demand procedures and capability packs | Yes | Flat Markdown, module-backed skills, or packaged skills. |
|
|
59
|
-
| `lib/` | Shared authored helper code | Yes | Import-only; not copied into the sandbox.
|
|
59
|
+
| `lib/` | Shared authored helper code | Yes | Import-only; not copied into the sandbox. Helpers can import data files such as JSON from here. |
|
|
60
60
|
| `memory.ts` or `memory/<name>.ts` | Cross-session memory | Yes | Provider-backed slots. See [Memory](/docs/memory). |
|
|
61
61
|
| `sandbox.ts` or `sandbox/sandbox.ts` | The agent's sandbox | Yes | The framework default applies when neither is authored. When `sandbox/` exists, only `sandbox/sandbox.ts` is read. |
|
|
62
62
|
| `sandbox/workspace/**` | Files seeded into the sandbox | Yes | Copied into the prepared environment; each new session sandbox starts with them in the workspace root. |
|
package/docs/reference/cli.md
CHANGED
|
@@ -459,21 +459,23 @@ eve eval [evalId...] [--url <url>] [options]
|
|
|
459
459
|
|
|
460
460
|
Runs all discovered evals when no eval ids are given; ids match exactly or by directory prefix (`eve eval weather` runs everything under `evals/weather/`). Exits `0` when every eval passed its checks, `1` when any eval failed (a failed check, an execution error, or a `--strict` threshold miss), `2` on configuration errors.
|
|
461
461
|
|
|
462
|
-
Without `--url`, `eve eval` starts its own local server on a free port with state under `.eve/eval-servers/<run id>/`, removed when the run ends.
|
|
463
|
-
|
|
464
|
-
| Flag | Type | Default | Description
|
|
465
|
-
| ------------------------ | ------ | ------- |
|
|
466
|
-
| `--url <url>` | string | none | Remote agent URL (skip local host startup)
|
|
467
|
-
| `--tag <tag...>` | string | none | Run only evals carrying a tag
|
|
468
|
-
| `--exclude-tag <tag...>` | string | none | Skip evals carrying a tag
|
|
469
|
-
| `--strict` | flag | off | Below-threshold scores also fail the exit code
|
|
470
|
-
| `--list` | flag | off | Print evals selected by the tag filters, without running them
|
|
471
|
-
| `--timeout <ms>` | number | none | Per-eval timeout in milliseconds
|
|
472
|
-
| `--max-concurrency <n>` | number | 8 | Max concurrent eval executions
|
|
473
|
-
| `--
|
|
474
|
-
| `--
|
|
475
|
-
| `--
|
|
476
|
-
| `--
|
|
462
|
+
Without `--url`, `eve eval` starts its own local server on a free port with state under `.eve/eval-servers/<run id>/`, removed when the run ends unless you pass `--keep-state`. A kept directory, including the server's Workflow data in `.workflow-data/`, stays until a later `eve eval` in the same app cleans it up. The local server runs beside an `eve dev` session for the same agent without attaching to it or changing it.
|
|
463
|
+
|
|
464
|
+
| Flag | Type | Default | Description |
|
|
465
|
+
| ------------------------ | ------ | ------- | -------------------------------------------------------------- |
|
|
466
|
+
| `--url <url>` | string | none | Remote agent URL (skip local host startup) |
|
|
467
|
+
| `--tag <tag...>` | string | none | Run only evals carrying a tag |
|
|
468
|
+
| `--exclude-tag <tag...>` | string | none | Skip evals carrying a tag |
|
|
469
|
+
| `--strict` | flag | off | Below-threshold scores also fail the exit code |
|
|
470
|
+
| `--list` | flag | off | Print evals selected by the tag filters, without running them |
|
|
471
|
+
| `--timeout <ms>` | number | none | Per-eval timeout in milliseconds |
|
|
472
|
+
| `--max-concurrency <n>` | number | 8 | Max concurrent eval executions |
|
|
473
|
+
| `--repeat <n>` | number | 1 | Run every eval n times to measure repeatability |
|
|
474
|
+
| `--json` | flag | off | Output results as JSON |
|
|
475
|
+
| `--junit <path>` | string | none | Write JUnit XML results to a file |
|
|
476
|
+
| `--skip-report` | flag | off | Skip eval-defined reporters (e.g. Braintrust) |
|
|
477
|
+
| `--keep-state` | flag | off | Keep the local eval server's state after the run (not `--url`) |
|
|
478
|
+
| `--verbose` | flag | off | Stream per-eval logs and workflow run IDs to stdout |
|
|
477
479
|
|
|
478
480
|
See [Evals](../evals/overview) for authoring evals.
|
|
479
481
|
|
|
@@ -138,16 +138,16 @@ Type the connection with its extensions, such as `SandboxConnection<{ setNetwork
|
|
|
138
138
|
|
|
139
139
|
eve calls the lifecycle methods at these points:
|
|
140
140
|
|
|
141
|
-
| When | eve calls
|
|
142
|
-
| ----------------------------------------------------- |
|
|
143
|
-
| `eve build`, or the first sandbox access in `eve dev` | `prepare(ctx)`, and stores the artifact it returns with the deployment
|
|
144
|
-
| The first sandbox access in a session
|
|
145
|
-
| A later workflow step or process
|
|
146
|
-
| The step that opened a connection ends | `connection.close()`, for a turn's step, a `"use step"` function, or an `invokeTool` call
|
|
147
|
-
| `sandbox.stop()` | `connection.stop()`, then `connection.close()`
|
|
148
|
-
| `sandbox.delete()` | `connection.close()`, then `onSessionEnd(ctx, state, { reason: "deleted" })`
|
|
149
|
-
| The eve server shuts down | `connection.stop()`, then `connection.close()`, on every connection eve still holds: any connection mid-step, and between steps a connection without `close()`; eve logs failures and does not throw them
|
|
150
|
-
| The owning session reaches a terminal outcome | `connection.close()` on a connection the ending step still holds, then `onSessionEnd(ctx, state, { reason })` with `"completed"`, `"expired"`, or `"failed"`
|
|
141
|
+
| When | eve calls |
|
|
142
|
+
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
143
|
+
| `eve build`, or the first sandbox access in `eve dev` | `prepare(ctx)`, and stores the artifact it returns with the deployment |
|
|
144
|
+
| The first sandbox access in a session, or prewarming | `start(ctx, options, artifact)` from `environment.open(options)` in the selector. A prewarmed session calls it in the background before its first message, and its first turn can call it again concurrently |
|
|
145
|
+
| A later workflow step or process, or a turn starting | `resume(ctx, artifact, state)`. Each turn of a session that recorded a sandbox calls it in the background as the turn starts, whether or not the turn uses the sandbox |
|
|
146
|
+
| The step that opened a connection ends | `connection.close()`, for a turn's step, a `"use step"` function, or an `invokeTool` call |
|
|
147
|
+
| `sandbox.stop()` | `connection.stop()`, then `connection.close()` |
|
|
148
|
+
| `sandbox.delete()` | `connection.close()`, then `onSessionEnd(ctx, state, { reason: "deleted" })` |
|
|
149
|
+
| The eve server shuts down | `connection.stop()`, then `connection.close()`, on every connection eve still holds: any connection mid-step, and between steps a connection without `close()`; eve logs failures and does not throw them |
|
|
150
|
+
| The owning session reaches a terminal outcome | `connection.close()` on a connection the ending step still holds, then `onSessionEnd(ctx, state, { reason })` with `"completed"`, `"expired"`, or `"failed"` |
|
|
151
151
|
|
|
152
152
|
eve calls `close()` at most once per connection, retrying only after a failure. After a turn's step ends, eve keeps a connection without `close()` registered, so server shutdown still stops the session's sandbox. eve lets go of a connection with `close()` at the end of its step or after `stop()`, so server shutdown between steps does not stop that sandbox. If your provider defines `close()` and runs remote compute, bound idle compute yourself, for example with an idle timeout. `stop()` can run again after the session uses a stopped connection, so make it a no-op when the sandbox is already stopped. A failed `close()` never prevents `onSessionEnd()` from deleting the sandbox.
|
|
153
153
|
|
package/docs/sandbox/index.mdx
CHANGED
|
@@ -52,9 +52,9 @@ export const environment = DefaultSandbox.environment();
|
|
|
52
52
|
export default defineSandbox(() => environment.open());
|
|
53
53
|
```
|
|
54
54
|
|
|
55
|
-
`environment.open()` starts the persistent sandbox owned by the current eve session. The `defineSandbox()` selector runs until initialization succeeds, then eve checkpoints the selected provider and its state. Later workflow steps and process restarts call the provider's `resume()` directly without rerunning the selector or `environment.open()`.
|
|
55
|
+
`environment.open()` starts the persistent sandbox owned by the current eve session. The `defineSandbox()` selector runs until initialization succeeds, then eve checkpoints the selected provider and its state. Later workflow steps and process restarts call the provider's `resume()` directly without rerunning the selector or `environment.open()`. Once a session has recorded its sandbox, eve resumes it in the background when each turn starts, so a tool that uses the sandbox waits only for whatever remains of the resume.
|
|
56
56
|
|
|
57
|
-
Pass options for the live sandbox to `open()`, then configure the returned sandbox before returning it. `open()` applies its options only when it creates the sandbox. Initialization can run more than once against the same sandbox, for example when a subagent that shares the sandbox opens it before the parent does, when a [workflow step](/docs/tools/workflows#use-the-session-sandbox-in-a-step) opens it before the session records it, or on the next access after an initialization that failed. Make it safe to repeat:
|
|
57
|
+
Pass options for the live sandbox to `open()`, then configure the returned sandbox before returning it. `open()` applies its options only when it creates the sandbox. Initialization can run more than once against the same sandbox, for example when a subagent that shares the sandbox opens it before the parent does, when a [workflow step](/docs/tools/workflows#use-the-session-sandbox-in-a-step) opens it before the session records it, when the first turn of a prewarmed session opens it while prewarming is still starting it, or on the next access after an initialization that failed. Make it safe to repeat:
|
|
58
58
|
|
|
59
59
|
```ts
|
|
60
60
|
import { defineSandbox } from "eve/sandbox";
|
|
@@ -69,6 +69,8 @@ export default defineSandbox(async ({ session }) => {
|
|
|
69
69
|
});
|
|
70
70
|
```
|
|
71
71
|
|
|
72
|
+
When a client prewarms a session, for example with `useEveAgent({ prewarm: true })`, eve runs the selector before the first message arrives. `session.auth` then describes the request that created the session, and the first message's `onMessage` hook can return a different identity. Do not derive per-user sandbox options or `defineState()` values from `session.auth` in an agent whose clients prewarm sessions: eve keeps both from the prewarming run. See [prewarming](/docs/guides/frontend/overview#returned-state) for when eve prewarms a session.
|
|
73
|
+
|
|
72
74
|
The environment export is required because `eve build` prepares immutable environment inputs before a session exists.
|
|
73
75
|
|
|
74
76
|
### Prepare a reusable environment
|
|
@@ -138,7 +140,7 @@ eve writes byte-backed files that reach the agent into the sandbox under `.eve/a
|
|
|
138
140
|
|
|
139
141
|
Files that tools return by URL with `toolOutputPart.fileUrl` are not staged: the model provider reads the URL directly. See [Send a file by URL](/docs/tools#send-a-file-by-url).
|
|
140
142
|
|
|
141
|
-
|
|
143
|
+
Every file, whether attached to a message or returned by a tool, reaches the model on every call with a label that names its sandbox path, such as `Attached file /workspace/.eve/attachments/<hash>/report.csv (text/csv)`. The agent can open that path with `read_file` or `bash`. The file's bytes follow its label only when every major provider reads the format natively:
|
|
142
144
|
|
|
143
145
|
- A PNG, JPEG, GIF, or WebP image up to 3 MiB and at most 8000 pixels per side.
|
|
144
146
|
- A PDF up to 20 MiB.
|
|
@@ -147,7 +149,7 @@ Other files, such as CSV, HEIC, audio, and video, reach the model as the label a
|
|
|
147
149
|
|
|
148
150
|
To let the agent answer voice notes and other audio, add a tool that transcribes the staged file. See [Voice notes](/docs/patterns/voice-notes).
|
|
149
151
|
|
|
150
|
-
If a
|
|
152
|
+
eve keeps the bytes of recent images and PDFs it staged or read for a session in memory, so preparing a model call's attachments resumes the sandbox only when it needs bytes that eve doesn't already hold. If a file read back from the sandbox is missing, or its bytes no longer match what eve staged because something in the sandbox overwrote it, the model receives a `FileNotFound` note in its place.
|
|
151
153
|
|
|
152
154
|
A file that eve can't stage never enters session history. When a channel's `fetchFile` fails, or a file part's `data` is not bytes, base64, a `data:` URL, or a URL, the model receives a note that names the file instead.
|
|
153
155
|
|