niceeval 0.1.1 → 0.1.2

package/docs/vision.md

@@ -0,0 +1,88 @@
+# Vision
+
+niceeval 应该让人感觉是**一个评测工具**,而不是"一堆按被测对象形状各写一遍的脚本"。无论你评的是一个远程 HTTP agent,还是一个塞进 Docker 里跑编码任务的 Claude Code,写法和读法都应该是同一套。
+
+为了在差异巨大的被测对象之间维持这种一致体验,代码库守一条规则:
+
+> **核心(core)拥有通用的"评测控制面";"连到哪个 AI"由 `Agent`(自实现的 `Adapter`)拥有,"在哪里跑"由 `Sandbox` 拥有。**
+
+这条规则是从 crabbox(一个面向 ~27 个云 provider 的远程执行层)学来的同一种纪律:核心永远不按名字分支,它对着能力(capability)分发。
+
+## 三种"快"
+
+niceeval 这个名字是一份承诺。把它拆开,就是三条要落到设计上的约束。
+
+### 写得快
+
+写一个 eval 的成本必须趋近于"写一句断言"。为此:
+
+- **路径即身份。** `evals/weather/brooklyn.eval.ts` 的 id 自动是 `weather/brooklyn`,禁止手写 id/name。改名即改路径,不会有"id 和文件名对不上"的腐烂。
+- **一文件一 eval,数组即扇出。** 默认导出一个 eval;默认导出一个数组就扇出成 `weather/0000`、`weather/0001`…… 数据集就是 `rows.map(row => defineEval(...))`。
+- **线性书写,就地断言。** `async test(t)` 里顺着写 `t.send` / `t.check`,把中间结果赋给局部变量再断言。没有回调金字塔。
+- **gate 与 atLeast 两档。** 硬性要求用 `gate`(不过即 fail),带阈值的质量分用 `atLeast`(默认记录为软失败,只在 `--strict` 下才 fail)。作者不必为"这条算不算致命"反复纠结。
+
+### 跑得快
+
+评测天然是 I/O 密集且昂贵(每个 case 可能是一次 LLM 调用、一次沙箱启动、一次完整测试套件)。运行器必须把这份昂贵摊薄:
+
+- **有界并发。** 可配置的最大并发,池满则等任一完成再补位。
+- **指纹缓存。** 用 `(eval 代码 + 配置)` 的指纹标识一次运行;已通过且指纹未变的,默认跳过。重跑只重算改动过的。
+- **沙箱复用与预热。** 沙箱启动是大头;支持预热池、跨 case 复用,把冷启动从关键路径上挪走。
+- **早停(earlyExit)。** 一个任务跑 N 次取通过率时,一旦先过了一次,就中止该任务剩余的重试(可关)。
+
+### 看得快
+
+评测的产出不只是"过/挂",而是"为什么"。失败必须能被快速归因:
+
+- **流式控制台。** 每个 case 完成即出一行,失败的断言内联展开,不用等全部跑完。
+- **结构化工件。** 事件流、transcript、文件 diff、断言结果都落盘成机器可读格式,可回放、可二次分析。
+- **统一 trace。** 不同 agent 的原始 transcript 被归一化成同一套事件模型(工具调用、文件读写、shell 命令、思考块),"agent 到底干了什么"一眼可见;OTLP traces 则统一归一到 **OpenTelemetry GenAI 语义约定**(每 agent 一个薄 mapper),view 只认这套 canonical,跨 agent 瀑布图天然可比。详见 [Observability](observability.md)。
+
+## 两个正交概念,一条承重墙
+
+被测对象的差异被收进两个互相正交的轴。理解它们的边界,就理解了整个库。
+
+### `Agent` —— 连到哪个 AI(以及谁来定协议)
+
+`Agent` 回答"我把任务发给谁、怎么收结果"。这里有一个**关键的、区别于 eve 的前提**:
+
+> **niceeval 不定义任何 agent 协议。** 每一条连到 AI 的连接 —— 你自己的 agent、你的后端服务、Claude Code / Codex —— 都是自己实现的 **Adapter**。experiment 引用一个 agent,而不是给一个 url。
+
+eve 能用一个 url 当 target,是因为它定义了一套协议、被测 agent 恰好会说。niceeval 没有这个假设,所以**没有 `--url`、没有通用的 http target**:连你自己的服务也是你写一个 agent,URL 是它的内部配置。
+
+- **Agent** 是抽象(niceeval 眼里"一条连到 AI 的连接")。`Agent.kind` 只有两类。
+- **Adapter** 是它的实现,由用户编写;按 `kind` 分两类:远程(`defineAgent`,按你服务的协议发请求)、沙箱(`defineSandboxAgent`,在 `Sandbox` 里 spawn coding agent 的 CLI)。进程内直调你的函数不是独立的第三类——它只是远程型 `send` 内部的一种实现方式,而且不推荐:测函数绕过了用户实际走的链路,见[接入你的 Agent · 为什么不直调](../docs-site/zh/guides/connect-your-agent.mdx)。
+
+核心通过 `Agent.kind` 和 `send` 的构造证据(而不是声明式能力位)决定 `t` 暴露哪些动作:会话型暴露 `t.send` / `t.calledTool`;沙箱型额外暴露 `t.sandbox`,里面按语义分成文件 IO、命令执行、结果断言 / diff。接一个新 agent(无论是你的 agent 还是 bub)= 实现一个 Adapter,**不动核心一行**。详见 [Adapter 契约](adapters/contract.md)。
+
+### `Sandbox` —— 沙箱型 agent 在哪里跑
+
+一个 `Sandbox` 把"隔离环境"的全部特殊性关进一个盒子:跑命令、读写文件、上传/下载、用 `cwd` 指定命令目录、起停。Docker 是一个实现,Vercel Sandbox 是一个实现,其它三方各是一个实现。后端按环境自动选择(有云 token 用云,否则用 Docker),也可显式指定。它与 Agent 正交:任意沙箱型 agent × 任意 sandbox 后端自由组合。详见 [Sandbox](sandbox.md)。
+
+## 边界画在"行为"上
+
+哪些地方允许出现 agent 名字 / sandbox 名字?**路由可以,行为不行。**
+
+- **允许:** 配置 schema、CLI 标志、experiment 引用 agent / sandbox 这种路由。这是核心的活。
+- **不允许:** 一旦代码要决定"CLI 参数怎么拼""transcript 在哪""命令怎么在容器里多路复用",这个决定就属于对应的 Adapter / Sandbox,**绝不**以 `if (name === ...)` 的形式穿过运行器、评分、报告这些核心路径。
+
+健康的形状是这样:
+
+- **核心** 发现 eval、收集断言、计算判决、调度并发、做缓存、写报告与工件。它对每个被测对象一视同仁。
+- **Agent(Adapter) / Sandbox** 各自解决自己那一层的特殊性,通过接口把能力交还核心。
+
+当一个修复看起来"需要在核心里加一个 agent-specific 分支"时,先去对应适配器加或复用一个 hook。一个中性的小 hook,几乎总比把 `name == ...` 的分支线穿过核心要干净。
+
+## 与参考项目的关系
+
+- **eve evals** 给了我们 DX 形状:`defineEval`、路径即身份、gate/soft 断言、LLM-as-judge、reporter 插件、有界并发运行器。niceeval 的会话型 eval 基本是这套模型的再实现。
+- **Vercel agent-eval** 给了我们 agent 评测的工程形状:`Adapter` 接口、`Sandbox` 抽象与多后端、transcript 归一化与注入、失败分类、指纹缓存。它的 fixture(目录约定自动发现 PROMPT + EVAL 测试)没有照搬——niceeval 里沙箱要放什么文件,是 `test()` 里的手工调用,不靠目录约定隐式发现。
+- **crabbox** 给了我们这条"核心 vs 适配器"的纪律,以及"文档是用户面真相、source-map 把行为映射回代码"的文档观。
+
+niceeval 的新意不在任何单一机制,而在于**把这两种本来分裂的评测范式(会话型 / 沙箱型)收敛进同一套 `defineEval` + 评分器 + 运行器 + 报告器**,让"评我自己的函数"和"评一个塞进容器的 Claude Code"读起来是同一种东西。
+
+## 相关阅读
+
+- [Architecture](architecture.md) —— 模块分层与数据流。
+- [Concepts](concepts.md) —— 术语表。
+- [Agents 与 Adapters](adapters/README.md)、[Sandbox](sandbox.md) —— 连接与沙箱契约。

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "niceeval",
-  "version": "0.1.1",
-  "description": "Lightweight TypeScript agent eval tool — eval agents, services, functions, and coding-agent fixtures",
+  "version": "0.1.2",
+  "description": "Agent-native eval tool — eval agents, services, functions, and coding-agent fixtures",
   "type": "module",
   "license": "MIT",
   "repository": {
@@ -43,7 +43,8 @@
   },
   "files": [
     "src",
-    "bin"
+    "bin",
+    "docs"
   ],
   "dependencies": {
     "autoevals": "0.0.132",
@@ -57,6 +58,9 @@
     "e2b": "^2.31.0"
   },
   "devDependencies": {
+    "@ai-sdk/otel": "^1.0.9",
+    "@opentelemetry/exporter-trace-otlp-http": "^0.219.0",
+    "@opentelemetry/sdk-trace-node": "^2.8.0",
     "@radix-ui/react-collapsible": "^1.1.14",
     "@radix-ui/react-dialog": "^1.1.17",
     "@radix-ui/react-tabs": "^1.1.15",
@@ -70,24 +74,50 @@
     "class-variance-authority": "^0.7.1",
     "clsx": "^2.1.1",
     "lucide-react": "^1.21.0",
+    "mixpanel-browser": "^2.80.0",
+    "next": "16.2.10",
     "prism-react-renderer": "^2.4.1",
     "react": "^19.2.7",
     "react-dom": "^19.2.7",
     "react-grab": "^0.1.47",
+    "shiki": "^4.3.0",
     "tailwind-merge": "^3.6.0",
     "tailwindcss": "^4.3.1",
     "typescript": "^5.6.0",
     "vite": "^8.1.0",
     "vitest": "^4.1.9"
   },
+  "peerDependencies": {
+    "@ai-sdk/otel": ">=1.0.0",
+    "@opentelemetry/exporter-trace-otlp-http": ">=0.200.0",
+    "@opentelemetry/sdk-trace-node": ">=2.0.0",
+    "ai": ">=5.0.0"
+  },
+  "peerDependenciesMeta": {
+    "@ai-sdk/otel": {
+      "optional": true
+    },
+    "ai": {
+      "optional": true
+    },
+    "@opentelemetry/exporter-trace-otlp-http": {
+      "optional": true
+    },
+    "@opentelemetry/sdk-trace-node": {
+      "optional": true
+    }
+  },
   "scripts": {
     "typecheck": "tsc --noEmit",
     "test": "vitest run",
     "test:watch": "vitest",
     "niceeval": "node bin/niceeval.js",
+    "gen:diff-code": "tsx scripts/gen-diff-code.ts",
+    "tiers:sync": "node scripts/sync-tiers.mjs sync",
+    "tiers:check": "node scripts/sync-tiers.mjs check",
     "view:build": "vite build --config src/view/app/vite.config.ts",
-    "site:dev": "vite site --host 127.0.0.1 --port 5174",
-    "site:build": "vite build site",
+    "site:dev": "cd site && node scripts/dev.mjs",
+    "site:build": "cd site && next build",
     "docs:dev": "cd docs-site && npx --yes mint@latest dev",
     "docs:validate": "cd docs-site && npx --yes mint@latest validate",
     "docs:links": "cd docs-site && npx --yes mint@latest broken-links --check-anchors --check-redirects"

package/src/agents/ai-sdk.test.ts

@@ -0,0 +1,377 @@
+import { describe, expect, it } from "vitest";
+
+import { aiSdkAgent, fromAiSdk } from "./ai-sdk.ts";
+import type { AiSdkGenerateContext, AiSdkResultLike } from "./ai-sdk.ts";
+import type { AgentContext } from "../types.ts";
+import { createAgentSession } from "../context/session.ts";
+
+describe("fromAiSdk", () => {
+  it("v5+ content parts:保留真实顺序,tool-error 映射成 failed", () => {
+    const { events } = fromAiSdk({
+      steps: [
+        {
+          content: [
+            { type: "reasoning", text: "先查天气" },
+            { type: "tool-call", toolCallId: "call_1", toolName: "get_weather", input: { city: "Brooklyn" } },
+            { type: "tool-result", toolCallId: "call_1", output: { temp: 21 } },
+            { type: "tool-call", toolCallId: "call_2", toolName: "web_search", input: { query: "穿衣" } },
+            { type: "tool-error", toolCallId: "call_2", error: new Error("rate limited") },
+          ],
+        },
+        { content: [{ type: "text", text: "布鲁克林 21 度。" }] },
+      ],
+    });
+
+    expect(events.map((e) => e.type)).toEqual([
+      "thinking",
+      "action.called",
+      "action.result",
+      "action.called",
+      "action.result",
+      "message",
+    ]);
+    expect(events[1]).toMatchObject({ callId: "call_1", name: "get_weather", tool: "unknown" });
+    expect(events[3]).toMatchObject({ callId: "call_2", name: "web_search", tool: "web_search" });
+    expect(events[4]).toMatchObject({ status: "failed", output: { error: "rate limited" } });
+    expect(events[5]).toMatchObject({ role: "assistant", text: "布鲁克林 21 度。" });
+  });
+
+  it("v4 退路:认 args / result / promptTokens 旧命名", () => {
+    const { events, usage } = fromAiSdk({
+      steps: [
+        {
+          text: "算好了,是 42。",
+          toolCalls: [{ toolCallId: "c1", toolName: "calculate", args: { expression: "6*7" } }],
+          toolResults: [{ toolCallId: "c1", result: { value: 42 } }],
+        },
+      ],
+      usage: { promptTokens: 100, completionTokens: 20 },
+    });
+
+    expect(events).toEqual([
+      { type: "action.called", callId: "c1", name: "calculate", input: { expression: "6*7" }, tool: "unknown" },
+      { type: "action.result", callId: "c1", output: { value: 42 }, status: "completed" },
+      { type: "message", role: "assistant", text: "算好了,是 42。" },
+    ]);
+    expect(usage).toEqual({ inputTokens: 100, outputTokens: 20, requests: 1 });
+  });
+
+  it("没有 steps:退回顶层 text / toolCalls / toolResults", () => {
+    const { events } = fromAiSdk({
+      text: "你好!",
+      toolCalls: [{ toolCallId: "c1", toolName: "read_file", input: { path: "a.ts" } }],
+      toolResults: [{ toolCallId: "c1", output: "content" }],
+    });
+
+    expect(events.map((e) => e.type)).toEqual(["action.called", "action.result", "message"]);
+    expect(events[0]).toMatchObject({ tool: "file_read" });
+  });
+
+  it("usage:totalUsage 优先于 usage,cachedInputTokens 进 cacheReadTokens,requests = step 数", () => {
+    const { usage } = fromAiSdk({
+      steps: [{ text: "a" }, { text: "b" }],
+      totalUsage: { inputTokens: 300, outputTokens: 50, cachedInputTokens: 120 },
+      usage: { inputTokens: 1, outputTokens: 1 },
+    });
+
+    expect(usage).toEqual({ inputTokens: 300, outputTokens: 50, requests: 2, cacheReadTokens: 120 });
+  });
+
+  it("全零 usage 视为缺失(别让 maxTokens 拿 0 假通过时看起来像有数据)", () => {
+    const { usage } = fromAiSdk({ steps: [{ text: "hi" }], usage: {} });
+    expect(usage).toBeUndefined();
+  });
+
+  it("空文本 / 空白 step 不产 message 事件", () => {
+    const { events } = fromAiSdk({ steps: [{ text: "  " }, { content: [{ type: "text", text: "" }] }] });
+    expect(events).toEqual([]);
+  });
+
+  it("无 approval 请求时 status 是 completed", () => {
+    const { status } = fromAiSdk({ steps: [{ content: [{ type: "text", text: "你好" }] }] });
+    expect(status).toBe("completed");
+  });
+
+  it("v7 tool-approval-request:input.requested + waiting,与 tool-call part 按 callId 去重", () => {
+    const { events, status } = fromAiSdk({
+      steps: [
+        {
+          content: [
+            { type: "tool-call", toolCallId: "call_1", toolName: "send_email", input: { to: "a@b.c" } },
+            {
+              type: "tool-approval-request",
+              approvalId: "appr_1",
+              toolCall: { toolCallId: "call_1", toolName: "send_email", input: { to: "a@b.c" } },
+            },
+          ],
+        },
+      ],
+    });
+
+    expect(status).toBe("waiting");
+    expect(events.map((e) => e.type)).toEqual(["action.called", "input.requested"]);
+    expect(events[0]).toMatchObject({ callId: "call_1", name: "send_email" });
+    expect(events[1]).toMatchObject({
+      request: {
+        id: "appr_1",
+        action: "send_email",
+        input: { to: "a@b.c" },
+        options: [{ id: "approve" }, { id: "deny" }],
+      },
+    });
+  });
+
+  it("automatic 的 approval 请求不吐 input.requested(SDK 自动裁决,不等人)", () => {
+    const { events, status } = fromAiSdk({
+      steps: [
+        {
+          content: [
+            {
+              type: "tool-approval-request",
+              approvalId: "appr_1",
+              isAutomatic: true,
+              toolCall: { toolCallId: "call_1", toolName: "send_email", input: {} },
+            },
+          ],
+        },
+      ],
+    });
+
+    expect(status).toBe("completed");
+    expect(events.map((e) => e.type)).toEqual(["action.called"]);
+  });
+
+  it("approval resume:responseMessages 里的执行结果补成 action.result,排在本轮事件之前", () => {
+    const { events, status } = fromAiSdk({
+      steps: [{ content: [{ type: "text", text: "邮件已发送。" }] }],
+      responseMessages: [
+        {
+          role: "tool",
+          content: [
+            {
+              type: "tool-result",
+              toolCallId: "call_1",
+              toolName: "send_email",
+              output: { type: "json", value: { delivered: true } },
+            },
+          ],
+        },
+        { role: "assistant", content: [{ type: "text", text: "邮件已发送。" }] },
+      ],
+    });
+
+    expect(status).toBe("completed");
+    expect(events).toEqual([
+      { type: "action.result", callId: "call_1", output: { delivered: true }, status: "completed" },
+      { type: "message", role: "assistant", text: "邮件已发送。" },
+    ]);
+  });
+
+  it("approval 拒绝:execution-denied 映射成 rejected", () => {
+    const { events } = fromAiSdk({
+      steps: [{ content: [{ type: "text", text: "好的,不发了。" }] }],
+      responseMessages: [
+        {
+          role: "tool",
+          content: [
+            {
+              type: "tool-result",
+              toolCallId: "call_1",
+              toolName: "send_email",
+              output: { type: "execution-denied", reason: "用户拒绝" },
+            },
+          ],
+        },
+      ],
+    });
+
+    expect(events[0]).toEqual({
+      type: "action.result",
+      callId: "call_1",
+      output: { reason: "用户拒绝" },
+      status: "rejected",
+    });
+  });
+
+  it("steps 里已有的 tool-result 不因 responseMessages 重复(普通工具循环两边都有)", () => {
+    const { events } = fromAiSdk({
+      steps: [
+        {
+          content: [
+            { type: "tool-call", toolCallId: "c1", toolName: "get_weather", input: { city: "北京" } },
+            { type: "tool-result", toolCallId: "c1", output: { temp: 26 } },
+            { type: "text", text: "北京 26 度。" },
+          ],
+        },
+      ],
+      responseMessages: [
+        {
+          role: "tool",
+          content: [
+            { type: "tool-result", toolCallId: "c1", toolName: "get_weather", output: { type: "json", value: { temp: 26 } } },
+          ],
+        },
+      ],
+    });
+
+    expect(events.filter((e) => e.type === "action.result")).toHaveLength(1);
+  });
+
+  it("aiSdkAgent:ctx.session.id 未记录时开新会话线,同 id 续接同一份历史", async () => {
+    const seen: unknown[][] = [];
+    const agent = aiSdkAgent({
+      generate: async ({ messages }: AiSdkGenerateContext) => {
+        seen.push([...messages]);
+        const result: AiSdkResultLike = {
+          steps: [{ content: [{ type: "text", text: `回复#${seen.length}` }] }],
+          responseMessages: [{ role: "assistant", content: `回复#${seen.length}` }],
+        };
+        return result;
+      },
+    });
+
+    const ctx = fakeCtx();
+    await agent.send({ text: "第一轮" }, ctx);
+    expect(ctx.session.id).toBeDefined();
+    const turn2 = await agent.send({ text: "第二轮" }, fakeCtx({ id: ctx.session.id }));
+
+    expect(turn2.status).toBe("completed");
+    // 第二轮的历史 = 用户#1 + 助手#1 + 用户#2
+    expect(seen[1]).toEqual([
+      { role: "user", content: "第一轮" },
+      { role: "assistant", content: "回复#1" },
+      { role: "user", content: "第二轮" },
+    ]);
+
+    // 新会话线:历史必须是干净的
+    const ctx3 = fakeCtx();
+    await agent.send({ text: "新会话" }, ctx3);
+    expect(ctx3.session.id).not.toBe(ctx.session.id);
+    expect(seen[2]).toEqual([{ role: "user", content: "新会话" }]);
+  });
+
+  it("aiSdkAgent:approval 停轮 → waiting;下一轮输入翻译成 tool-approval-response", async () => {
+    const seen: unknown[][] = [];
+    let call = 0;
+    const agent = aiSdkAgent({
+      generate: async ({ messages }: AiSdkGenerateContext) => {
+        seen.push([...messages]);
+        call++;
+        if (call === 1) {
+          return {
+            steps: [
+              {
+                content: [
+                  { type: "tool-call", toolCallId: "c1", toolName: "send_email", input: { to: "a@b.c" } },
+                  { type: "tool-approval-request", approvalId: "appr_1", toolCall: { toolCallId: "c1", toolName: "send_email", input: { to: "a@b.c" } } },
+                ],
+              },
+            ],
+            responseMessages: [{ role: "assistant", content: [] }],
+          } satisfies AiSdkResultLike;
+        }
+        return {
+          steps: [{ content: [{ type: "text", text: "邮件已发送。" }] }],
+          responseMessages: [],
+        } satisfies AiSdkResultLike;
+      },
+    });
+
+    const ctx = fakeCtx();
+    const first = await agent.send({ text: "发邮件" }, ctx);
+    expect(first.status).toBe("waiting");
+    expect(first.events.some((e) => e.type === "input.requested")).toBe(true);
+
+    const second = await agent.send({ text: "approve" }, fakeCtx({ id: ctx.session.id }));
+    expect(second.status).toBe("completed");
+    // 第二轮历史的最后一条是翻译好的裁决(tool 消息),不是用户文本
+    const last = (seen[1] as { role: string; content: unknown }[]).at(-1);
+    expect(last).toEqual({
+      role: "tool",
+      content: [{ type: "tool-approval-response", approvalId: "appr_1", approved: true }],
+    });
+  });
+
+  it("aiSdkAgent:approval resume 优先按 requestId 从 input.responses 读裁决,而不是猜 input.text", async () => {
+    const seen: unknown[][] = [];
+    let call = 0;
+    const agent = aiSdkAgent({
+      generate: async ({ messages }: AiSdkGenerateContext) => {
+        seen.push([...messages]);
+        call++;
+        if (call === 1) {
+          return {
+            steps: [
+              {
+                content: [
+                  { type: "tool-call", toolCallId: "c1", toolName: "send_email", input: { to: "a@b.c" } },
+                  { type: "tool-approval-request", approvalId: "appr_1", toolCall: { toolCallId: "c1", toolName: "send_email", input: { to: "a@b.c" } } },
+                ],
+              },
+            ],
+            responseMessages: [{ role: "assistant", content: [] }],
+          } satisfies AiSdkResultLike;
+        }
+        return {
+          steps: [{ content: [{ type: "text", text: "好的,没有发送。" }] }],
+          responseMessages: [],
+        } satisfies AiSdkResultLike;
+      },
+    });
+
+    const ctx = fakeCtx();
+    await agent.send({ text: "发邮件" }, ctx);
+
+    // input.text 读起来像批准,但 responses 结构化裁决说 deny——必须以 responses 为准。
+    const second = await agent.send(
+      { text: "approve", responses: [{ requestId: "appr_1", optionId: "deny" }] },
+      fakeCtx({ id: ctx.session.id }),
+    );
+    expect(second.status).toBe("completed");
+    const last = (seen[1] as { role: string; content: unknown }[]).at(-1);
+    expect(last).toEqual({
+      role: "tool",
+      content: [{ type: "tool-approval-response", approvalId: "appr_1", approved: false }],
+    });
+  });
+
+  it("aiSdkAgent:generate 抛错 / 空结果 → failed + error 事件", async () => {
+    const boom = aiSdkAgent({ generate: async () => { throw new Error("上游超时"); } });
+    const failed = await boom.send({ text: "hi" }, fakeCtx());
+    expect(failed.status).toBe("failed");
+    expect(failed.events[0]).toMatchObject({ type: "error", message: "上游超时" });
+
+    const empty = aiSdkAgent({ generate: async () => ({ steps: [{ text: "" }] }) });
+    const emptyTurn = await empty.send({ text: "hi" }, fakeCtx());
+    expect(emptyTurn.status).toBe("failed");
+  });
+
+  it("v7 usage:cache tokens 从 inputTokenDetails 读", () => {
+    const { usage } = fromAiSdk({
+      steps: [{ text: "hi" }],
+      usage: { inputTokens: 100, outputTokens: 20, inputTokenDetails: { cacheReadTokens: 40, cacheWriteTokens: 8 } },
+    });
+
+    expect(usage).toEqual({
+      inputTokens: 100,
+      outputTokens: 20,
+      requests: 1,
+      cacheReadTokens: 40,
+      cacheWriteTokens: 8,
+    });
+  });
+});
+
+/** 省略 `id`(或不给参数)= 全新会话线;传 `id` = 这条线已经 capture 过这个 id(续接轮)。 */
+function fakeCtx(opts: { id?: string } = {}): AgentContext {
+  const session = createAgentSession();
+  if (opts.id) session.capture(opts.id);
+  return {
+    signal: new AbortController().signal,
+    model: undefined,
+    flags: {},
+    sandbox: undefined as unknown as AgentContext["sandbox"],
+    session,
+    log: () => {},
+  };
+}