niceeval 0.15.1-canary.76 → 0.16.1

package/INDEX.md

@@ -14,7 +14,7 @@
 
 - **已有项目要运行或复验。** 先读 `docs-site/zh/tutorials/agent-feedback-loop.mdx`，再按任务从索引树补读 Authoring、接入或 Reference 页面。保留用户已经给出的范围、授权和预算；只有缺少会改变被测目标、运行边界、凭证边界或费用的决定时才提问。
 - **只读诊断。** 先读 `docs-site/zh/tutorials/agent-feedback-loop.mdx`，再从索引树选择相关 Troubleshooting 或 Reference 页面。只从 `show`、`query` 或 `view` 收集公开证据；不要修改项目、重跑 Experiment、采用结果或写入 `.niceeval/`。
-- **编写或修正评估。** 先读 `docs-site/zh/tutorials/authoring.mdx` 和 `docs-site/zh/tutorials/evaluation-kinds.mdx`；要连接被测对象时再读对应接入教程。评估输入必须覆盖真实业务能力，并能区分被测系统是否完成任务。
+- **编写或修正评估。** 先读 `docs-site/zh/tutorials/authoring.mdx` 和 `docs-site/zh/tutorials/evaluation-kinds.mdx`，交付前按 `docs-site/zh/tutorials/niceeval-taste.mdx` 审阅判据；被测对象提供业务事实时再读 `docs-site/zh/tutorials/niceeval-taste-application.mdx`。要连接被测对象时再读对应接入教程。评估输入必须覆盖真实业务能力，并能区分被测系统是否完成任务。
 - **首次接入。** 读 `docs-site/zh/tutorials/agent-onboarding.mdx`，它拥有探索、必要决策、事件映射和生命周期的完整步骤。
 
 ## 帮用户跑通实验
@@ -39,73 +39,78 @@
 
 ## `docs-site/zh/tutorials/`
 
-- `docs-site/zh/tutorials/accept-results.mdx` — 审阅并采用历史 Attempt:从 --dry 取得候选 locator，核对不可变 Attempt，再用 niceeval accept 原子采用明确结果，并用新 Run ID 验证分母与来源。
-- `docs-site/zh/tutorials/agent-feedback-loop.mdx` — 让 Coding Agent 根据结果迭代:让 Coding Agent 读取 receipt 的 createdRunIds，查看结果，再区分被测失败、证据错误与评估用例误判。
+- `docs-site/zh/tutorials/accept-results.mdx` — 确认旧结果仍然成立，免去重跑:配置变了、NiceEval 不敢沿用旧结果，而你确认它不受影响时，用 niceeval accept 明确接受它，省下一次付费重跑。
+- `docs-site/zh/tutorials/agent-feedback-loop.mdx` — 让 Coding Agent 根据结果迭代:让 Coding Agent 自己跑评估、读结果、判断该改应用还是改断言，再复验，不用你把终端输出复制来复制去。
 - `docs-site/zh/tutorials/agent-onboarding.mdx` — Coding Agent 从零接入项目:给 Coding Agent 的完整接入流程：探索项目、与用户确认路径、配置 Judge、写出 Adapter、Experiment 与评估用例，并跑通第一个实验。
-- `docs-site/zh/tutorials/authoring.mdx` — 编写评估用例: 单轮、多轮和测试集模式:用 defineEval 编写评估用例，包括单轮对话、多轮对话、数据驱动测试、Sandbox Workspace 和评估的生命周期与 Fixture。
-- `docs-site/zh/tutorials/ci-integration.mdx` — 在 CI 中运行 NiceEval:让 CI 使用退出状态判定门禁，保留最后的 receipt，并向平台提供 JUnit 文件。
-- `docs-site/zh/tutorials/compare-runs.mdx` — 比较多个固定 Run:用 `runs.compare` operation 比较明确 Run 的已发布事实，并在浏览器中审阅相同的结果。
-- `docs-site/zh/tutorials/concurrency.mdx` — 调好并发：吞吐、共享状态与固定执行顺序:全局并发、实验上限与 Provider capacity 共同限制实际吞吐。这一页说明排队状态、共享状态与固定顺序。
+- `docs-site/zh/tutorials/authoring.mdx` — 编写评估用例：单轮、多轮和测试集:用 defineEval 写评估用例：发送消息、检查回复和工具调用、多轮对话、用一份测试集生成多条用例，以及给 Coding Agent 准备工作区和收尾外部资源。
+- `docs-site/zh/tutorials/ci-integration.mdx` — 在 CI 里自动跑评估，挡住回归:在每个 PR 上运行同一条 niceeval exp，评估不通过就让 CI 变红。保留这次运行的 Run ID 供后续步骤读取，并把 JUnit 文件交给 CI 平台展示。
+- `docs-site/zh/tutorials/compare-runs.mdx` — 比较两次运行：基线和候选谁更好:固定两次运行的 Run ID，在浏览器里并排审阅，或用 runs.compare 让脚本和 CI 得到逐题配对的比较结果。
+- `docs-site/zh/tutorials/concurrency.mdx` — 调好并发：吞吐、共享状态与固定执行顺序:一批评估跑得太慢时，先判断时间花在哪，再调全局并发和实验的 maxConcurrency。需要跨 Attempt 共享状态时，让它们按固定顺序串行。
 - `docs-site/zh/tutorials/configuration.mdx` — 把配置和密钥各放到该放的地方:跑几次、超时、并发、judge 这些值写进代码的哪一层，API key 和 provider token 用哪些环境变量。
-- `docs-site/zh/tutorials/connect-otel.mdx` — OTel 接入:把应用已有的 OTel span 发送给 NiceEval，在 niceeval view 中查看每轮调用瀑布图。评估用例断言仍以 Send 返回的事件和用量为准。
-- `docs-site/zh/tutorials/connect-your-agent.mdx` — 接入你的 Agent:写一个 Adapter、配置一个 Experiment，并运行第一条评估用例。再把配置和参数从 Experiment 传给 Adapter 与被测应用。
-- `docs-site/zh/tutorials/control-run-cost.mdx` — 控制一次运行的成本:用 attempts、首过即停、budget、并发和结果沿用分别控制统计样本、停止派发、花费保护与资源占用。
-- `docs-site/zh/tutorials/criteria-files.mdx` — 隐藏测试判分：让判据文件进缓存 fingerprint:用 loadText 在模块顶层读入隐藏测试、跑测脚本等判据文件，文件内容进入 fingerprint：改一字节自动重跑，不靠人工 --rerun。
-- `docs-site/zh/tutorials/custom-application.mdx` — 评估自定义应用:保留应用自己的方法与上下文，用 NiceEval 评估结构化结果和业务操作。
-- `docs-site/zh/tutorials/dataset-fanout.mdx` — 数据驱动测试（dataset fan-out）：用多份数据运行同一套评估用例:从 .eval.ts 文件导出数组或 keyed record，将一套评估逻辑展开为多个 case。用 loadYaml 或 loadJson 读取外部测试集，并获得稳定 ID。
-- `docs-site/zh/tutorials/debug-lifecycle-plan.mdx` — 运行前检查 Sandbox 计划:用 niceeval debug 查看 Experiment、评估组、评估用例与 Agent 的实际 action 顺序和数值，而不创建运行资源。
+- `docs-site/zh/tutorials/connect-otel.mdx` — 接入 OTel：在浏览器里看每轮调用瀑布图:应用已经在发 OTel trace 时，让它把 span 也发给 NiceEval 一份，就能在 niceeval view 里看到每一轮的模型调用、工具执行和耗时。断言不受影响。
+- `docs-site/zh/tutorials/connect-your-agent.mdx` — 接入你的 Agent：选接入方式，传配置:按被测对象选择内置适配器或手写 Adapter，把地址和鉴权放进 Adapter 工厂、把 model 和 flags 经 ctx 转给应用，并按需加上工具调用、多轮和 tracing。
+- `docs-site/zh/tutorials/control-run-cost.mdx` — 控制一次运行的成本:一次评估可能花掉几十美元。按你要控制的是什么，分别用 attempts、首过即停、budget、并发和结果沿用。
+- `docs-site/zh/tutorials/criteria-files.mdx` — 用隐藏测试判分，改了测试自动重跑:用 loadText 读入隐藏测试、跑测脚本等判分文件。NiceEval 会记住文件内容，测试一改，用到它的评估用例下次自动重跑。
+- `docs-site/zh/tutorials/custom-application.mdx` — 评估自定义应用：发帖、生图这类业务操作:被测应用不是对话式 Agent 时，用 defineAdapter 把它的业务方法直接暴露给评估用例，检查结构化结果和业务操作。
+- `docs-site/zh/tutorials/dataset-fanout.mdx` — 数据驱动测试：一份测试集生成多条评估用例:几十条评估逻辑相同、只有输入不同时，从一个 .eval.ts 导出数组或对象，一套逻辑生成多条评估用例。也可以用 loadYaml 或 loadJson 读外部测试集，并给每条起稳定的 ID。
+- `docs-site/zh/tutorials/debug-lifecycle-plan.mdx` — 运行前检查 Sandbox 会怎样准备:用 niceeval debug 列出一条评估在 Sandbox 里会按什么顺序准备、安装、执行和收尾，以及每一步的实际参数，不创建任何资源。
 - `docs-site/zh/tutorials/docker-in-docker.mdx` — 让 Sandbox 使用 Docker:根据任务信任边界选择 Docker socket、raw privileged DinD 或受管 rootless DinD，并为 Agent 准备 Docker CLI 与 daemon。
-- `docs-site/zh/tutorials/eval-groups.mdx` — 用评估组复用 Sandbox:用 defineEvalGroup 显式声明兼容成员：同组按稳定 ID 串行复用一台 Sandbox，不同组继续并行，并把公共准备放到正确的 Sandbox Layer。
-- `docs-site/zh/tutorials/evaluation-kinds.mdx` — 通过制与计分制评估:用 defineEval 与 defineScoreEval 编写通过制和计分制评估，调用即登记 Assertion。
-- `docs-site/zh/tutorials/experiments.mdx` — 实验矩阵：用运行矩阵比较 agents 和 models:使用 NiceEval experiments 让同一批评估用例横跨多个 agents、models 和 flags，比较 pass rate、成本和延迟。
-- `docs-site/zh/tutorials/fixtures.mdx` — Sandbox Fixture：用任务评估 coding agents:用 .eval.ts 给 coding agent 准备隔离 workspace、发送真实任务，并用 Sandbox 文件、命令、diff 和 judge 验证结果。
-- `docs-site/zh/tutorials/handle-execution-failures.mdx` — 在基础设施失败时停止无效运行:读懂自动重试，为确定性故障声明影响范围，修复后继续未完成的评估。
+- `docs-site/zh/tutorials/eval-groups.mdx` — 用评估组让相关评估共用 Sandbox:几组评估各自需要一套很重的工具链时，用 defineEvalGroup 把它们分组：同组共用一台 Sandbox 依次执行，不同组照常并行。
+- `docs-site/zh/tutorials/evaluation-kinds.mdx` — 按完成度给分：通过制与计分制:任务只有做到和没做到两种结果时用 defineEval 读通过率；有部分进展时用 defineScoreEval 按完成的步骤累计分数。
+- `docs-site/zh/tutorials/experiment-services.mdx` — 启动实验共享服务:用 Experiment 的 setup / teardown 启停整个实验共用的隧道、mock server 或租约，把运行时地址交给 Agent 和 Sandbox，又不让它作废已有结果。
+- `docs-site/zh/tutorials/fixtures.mdx` — 给 Coding Agent 准备任务：工作区、提示和验收:在评估用例里把一个真实项目放进 Sandbox，给 Coding Agent 发任务，再用文件内容、命令结果、diff 和 Judge 检查它做得对不对。
+- `docs-site/zh/tutorials/handle-execution-failures.mdx` — 凭据失效或服务挂掉时，及时止损:了解 NiceEval 会自动重试哪些失败；发现凭据失效、共享服务断开这类注定失败的故障时，停止剩下的 Attempt，修好后只补跑没完成的部分。
 - `docs-site/zh/tutorials/install-custom-sandbox-agent.mdx` — 为自定义 Sandbox Agent 安装 CLI:用 ensure 和 installer 检查、安装并复检自定义 coding-agent CLI 的精确版本。
+- `docs-site/zh/tutorials/niceeval-taste-application.mdx` — 自定义应用的评估判据：业务事实、领域 Match 与普通计算:被测应用提供客服记录、游戏状态这类业务事实时，让 Adapter 交付事实，用领域 Match 组合小条件，在正文里写普通计算，并保留未知状态。
+- `docs-site/zh/tutorials/niceeval-taste.mdx` — NiceEval Taste：写出一眼能看懂判据的评估:让同事只读评估用例正文，就能说出它测什么、什么算通过、证据为什么够。用已有事实、简单条件和普通算术写判据，少造新概念。
 - `docs-site/zh/tutorials/nixos-managed-dind.mdx` — 在 NixOS 上配置 Managed DinD:使用 NiceEval 的 NixOS module 部署 managed rootless Docker profile，并通过 doctor 验证宿主环境。
-- `docs-site/zh/tutorials/plugins.mdx` — 用 Plugin 复用完整运行条件:用 definePlugin 组合宿主生命周期与 command-only SandboxLayer，并在多个 owner 之间复用。
-- `docs-site/zh/tutorials/quickstart.mdx` — 为你的 Agent 项目设置评估:安装 NiceEval，写三个文件，10 分钟内对你自己的应用跑通第一条评估用例。
-- `docs-site/zh/tutorials/reporters.mdx` — 将当前运行接入外部系统:使用 Reporter、JUnit 和 receipt 把当前 Invocation 的信息交给外部系统，同时保持 Record 的所有权边界。
-- `docs-site/zh/tutorials/rerun-and-cache.mdx` — 重跑与沿用结果:查看每个 slot 会执行、自动采用还是由操作者接受，并理解这些关系怎样进入 Record。
-- `docs-site/zh/tutorials/sandbox-agent.mdx` — Sandbox Agent：评估 Claude Code、Codex 和 bub:使用 NiceEval 内置 agents 或自定义 adapter，在 Docker 或云端 sandbox 中运行 coding-agent CLI。
-- `docs-site/zh/tutorials/sandbox-providers.mdx` — Sandbox:官方 Sandbox Docker、Vercel 或 E2B Provider。通过预制快照加速评估
-- `docs-site/zh/tutorials/sandbox-reuse.mdx` — 在一批评估中复用 Sandbox:在 Experiment 里声明 sandboxReuse 让多条 Attempt 共用 Sandbox，让 before/after 可安全重放，并配好 lifetimeMs。无法安全保留状态时改用并发。
-- `docs-site/zh/tutorials/select-and-preview.mdx` — 选择并预览运行范围:先列出 Experiment，再用路径、评估用例前缀和 tag 收窄范围，最后用 --dry 确认计划后再花费模型与 Sandbox 成本。
-- `docs-site/zh/tutorials/verify-judge.mdx` — 配置并验证 Judge:选择具名 Judge Provider，用 rubric 和命名材料完成一次真实判分。
-- `docs-site/zh/tutorials/viewing-results.mdx` — 查看运行结果:在终端用 show 逐层查看结果；在浏览器用 View 审阅，在自动化中用 query 读取 JSON。
-- `docs-site/zh/tutorials/write-experiment.mdx` — 实验与生命周期:如何写好实验。选择 Agent、传递 model 和 flags，设置 attempts、预算、并发与 Sandbox，并用 setup / teardown 起停实验级共享服务。
+- `docs-site/zh/tutorials/plugins.mdx` — 用 Plugin 复用一整套运行条件:同一套“先在宿主机起服务、再在 Sandbox 里准备环境”的条件要在多个 Experiment、评估组或评估用例里用时，用 definePlugin 写一次，到处挂载。
+- `docs-site/zh/tutorials/prebuilt-sandbox.mdx` — 预制 Sandbox 环境：让每次评估更快开始:把系统包、Agent CLI 和大缓存做进 Docker 镜像、Vercel 快照或 E2B template，从官方基线派生；运行时缓存用 checkpoint 在 Attempt 之间延续。
+- `docs-site/zh/tutorials/quickstart.mdx` — 快速开始：跑通第一条评估用例:安装 NiceEval，写一个 Adapter、一个 Experiment 和一条评估用例，十分钟左右对你自己的应用跑通第一条评估，并在终端和浏览器里看到结果。
+- `docs-site/zh/tutorials/reporters.mdx` — 把运行结果上报给 Braintrust 等外部系统:团队已经在用 Braintrust 或自己的看板时，用 Reporter 在运行过程中把结果发过去；CI 平台用 JUnit；后续脚本用 receipt 里的 Run ID 读取结果。
+- `docs-site/zh/tutorials/rerun-and-cache.mdx` — 修改后只重跑受影响的评估:改了 prompt、评估用例或模型之后，用 --dry 看清哪些结果会沿用、哪些会重跑，再用 --rerun 和 accept 控制范围。
+- `docs-site/zh/tutorials/sandbox-agent.mdx` — Sandbox Agent：评估 Claude Code、Codex 和 bub:把 Claude Code、Codex、bub 等 Coding Agent 放进 Docker 或云端 Sandbox，给它一个真实仓库和任务，再用文件改动和项目测试判分。
+- `docs-site/zh/tutorials/sandbox-providers.mdx` — 选择并配置 Sandbox：Docker、Vercel 或 E2B:给 Coding Agent 选一个隔离运行环境：本地 Docker、Vercel 或 E2B。按顺序准备环境，并用预制镜像和缓存让每次评估更快开始。
+- `docs-site/zh/tutorials/sandbox-reuse.mdx` — 在一批评估中复用 Sandbox:每条 Attempt 都新建 Sandbox 太慢时，用 sandboxReuse 让一批 Attempt 依次共用同一个 Sandbox。说明它省了什么、留下了什么，以及准备代码要怎么写才能安全重放。
+- `docs-site/zh/tutorials/select-and-preview.mdx` — 运行前确认跑哪些、跑几次:列出有哪些 Experiment，用 ID 前缀和 tag 缩小范围，再用 --dry 看清这次会跑哪些评估用例、多少次，确认无误再花钱。
+- `docs-site/zh/tutorials/verify-judge.mdx` — 配置 Judge，并确认它真的能打分:给裁判模型 Judge 选好 Provider、模型和 API Key，再用一条最小评估跑一次真实判分，确认配置无误后再大批使用。
+- `docs-site/zh/tutorials/viewing-results.mdx` — 查看运行结果：从总览找到失败原因:跑完评估后，用 niceeval show 在终端从总览逐层找到失败的那次运行，查看对话、耗时、用量和文件改动；在浏览器用 view 翻看；脚本和 CI 用 query 读 JSON。
+- `docs-site/zh/tutorials/write-experiment.mdx` — 编写实验：比较模型、prompt 和 Agent:用 defineExperiment 固定被测 Agent、模型、flags 和运行次数。同一批评估用例写两份只差一个变量的 Experiment，就能比较两个模型或两版 prompt。
 - `docs-site/zh/tutorials/write-send.mdx` — 编写 Send:分七步编写 Adapter 的 send 函数：发送消息、续接会话、记录用量、映射工具事件、处理人工介入（HITL）、接入 OTel trace，并透传 Experiment flags。
 
 ## `docs-site/zh/explanation/`
 
 - `docs-site/zh/explanation/adapter.mdx` — Adapter：把 Agent 接入 NiceEval:Adapter 是你写的适配器。本文讲清楚 send 函数传入什么返回什么、被测系统配置怎么传、三个接入等级，以及 t 上的能力从哪来。
-- `docs-site/zh/explanation/assert.mdx` — NiceEval Assertion 与 Match:用 Match 描述值比较，调用即登记 Assertion；handle 只配置同一条 Assertion。
+- `docs-site/zh/explanation/assert.mdx` — NiceEval Assertion 与 Match:Match 描述怎样比较一个值；调用检查方法时立即登记一条 Assertion，后面接的方法只配置这一条。
 - `docs-site/zh/explanation/drive.mdx` — NiceEval 里怎么驱动 agent：send、session 与 HITL:t.send() 和它返回的 Turn、t.sendFile()、多轮对话、t.newSession()，以及 respond() 处理的人工介入(HITL)。
-- `docs-site/zh/explanation/evals.mdx` — NiceEval 中的评估用例：生命周期、verdict 与文件:评估用例是一个测试用例：描述和 test 函数，agent-neutral。了解评估用例如何被发现、调度、评分和报告。
-- `docs-site/zh/explanation/experiment.mdx` — 实验（Experiment）：评谁、怎么跑:Experiment 是可签入的运行配置：同一批评估用例对着哪个 agent、哪个模型、开哪些 flags、跑几次。评估用例与 experiment 分开的原因——晚绑定，让同一份评估用例换着对象跑。
+- `docs-site/zh/explanation/evals.mdx` — NiceEval 中的评估用例：生命周期、判定与文件:评估用例说明发什么、什么算做对，不绑定被测对象。了解评估用例如何被发现、调度、评分和报告。
+- `docs-site/zh/explanation/experiment.mdx` — 实验（Experiment）：评谁、怎么跑:Experiment 是可签入的运行配置：同一批评估用例对着哪个 Agent、哪个模型、开哪些 flags、跑几次。评估用例与 Experiment 分开，同一批评估用例才能换着对象跑。
 - `docs-site/zh/explanation/hitl.mdx` — 人工介入（HITL）：停轮、回答、继续:什么是 Human-in-the-loop，Claude Code / Codex / AI SDK 应用里它长什么样。NiceEval 如何把它变成可评的路径——waiting 停轮、input.requested 说明在等什么、t.respond 替人回答后同一轮继续。
 - `docs-site/zh/explanation/judge.mdx` — NiceEval 里的 Judge：评开放式输出:用 rubric、anchors 和显式材料评估开放式输出，需要时附上图片。
-- `docs-site/zh/explanation/overview.mdx` — NiceEval 的运行后数据流:理解 Record、固定 Inspection operation、Query、Show 和 View 的职责边界。
-- `docs-site/zh/explanation/record.mdx` — Record:了解 canonical SQLite Record 的写入、携带与读取边界。
-- `docs-site/zh/explanation/runner.mdx` — 运行、封口与读取:了解 Runner 如何发布 Run，以及读取路径为什么只看已封口事实。
+- `docs-site/zh/explanation/overview.mdx` — NiceEval 的整体模型：一次评估由哪些部分组成:评估用例、Experiment、Adapter、Sandbox、Runner 和 Record 各管什么，它们怎样连成一次运行，以及为什么同一条评估要跑多次。
+- `docs-site/zh/explanation/record.mdx` — Record:了解项目结果文件 .niceeval/record.sqlite 保存什么、怎样复制给别人，以及读取时会校验什么。
+- `docs-site/zh/explanation/results-flow.mdx` — NiceEval 的运行后数据流:理解运行结果保存在哪里，以及 query、show 和 view 三个查看入口各管什么。
+- `docs-site/zh/explanation/runner.mdx` — 运行怎样产出可读的结果:了解 Runner 怎样创建 Run、逐个发布 Attempt，以及查看命令为什么只读已发布的结果。
 - `docs-site/zh/explanation/tier.mdx` — 接入等级（Tier）：三级投入，三组能力:按「Adapter 接到哪里、额外拿到什么观测数据」接入分三级：Tier 1 只接 send，Tier 2 加 OTel，Tier 3 侵入改造暴露 experiment flags。每档买到什么、什么时候升级。
 
 ## `docs-site/zh/reference/`
 
 - `docs-site/zh/reference/builtin-agents.mdx` — 内置 Agent 能力参考:NiceEval 内置的 claude-code、codex、bub 适配器分别做到了哪些能力，对应哪些断言，以及已知限制。
 - `docs-site/zh/reference/capabilities.mdx` — 能力与证据覆盖参考:Agent 的构造方式、运行时行为与必填的六通道 evidenceCoverage 声明，怎样共同决定 NiceEval 的断言能相信什么。
-- `docs-site/zh/reference/cli.mdx` — CLI:用固定的 query、show 和 view 读取 canonical SQLite Record。
+- `docs-site/zh/reference/cli.mdx` — CLI:用 query、show 和 view 读取运行记录。
 - `docs-site/zh/reference/define-agent.mdx` — 定义评估对象：Adapter 参考:自定义 Adapter 与 Agent 接入的上下文、断言方法和资源生命周期。
 - `docs-site/zh/reference/define-config.mdx` — defineConfig：项目默认配置:defineConfig 参考：Judge Runtime、Reporters、并发、超时和 Sandbox 默认值。
 - `docs-site/zh/reference/define-eval.mdx` — defineEval：声明、配置并运行 NiceEval 评估用例:defineEval 参考：选项、test context t、Turn 返回值、Sandbox 辅助函数，以及数组和 keyed record 测试集导出。
-- `docs-site/zh/reference/events.mdx` — 标准事件流参考:StreamEvent 的十种事件：每种的字段、什么时候吐、哪些断言消费它。adapter 的核心工作就是产出这条流。
+- `docs-site/zh/reference/define-experiment.mdx` — defineExperiment：实验配置:defineExperiment 的实验字段、类型与默认行为参考。
+- `docs-site/zh/reference/events.mdx` — 标准事件流参考:StreamEvent 的十种事件：每种的字段、何时产出、哪些断言读取它。Adapter 的主要工作就是产出这条流。
 - `docs-site/zh/reference/expect.mdx` — niceeval/expect Match 参考:niceeval/expect 的纯 Match factory：组合文字、结构、条件，再交给 t.check 登记 Assertion。
 - `docs-site/zh/reference/official-adapters.mdx` — 官方适配器一览:NiceEval 内置的 Sandbox 和非 Sandbox 适配器分别是什么、怎么鉴权，Sandbox 型里怎么装 MCP server、Skill、插件，怎么使用 Agent 官方配置文件。
 
 ## `docs-site/zh/troubleshooting/`
 
 - `docs-site/zh/troubleshooting/debug-sandbox.mdx` — 保留 Sandbox 现场排查问题:用 --keep-sandbox 把失败 Attempt 的 Sandbox 保留成可随时唤醒的现场，用 niceeval sandbox enter 进去手动排查，用 sandbox list / stop 查看和清理。
-- `docs-site/zh/troubleshooting/debugging.mdx` — 排查已封口的结果:根据具名错误和闭合 Inspection result，通过 Show、Query 与 View 排查问题。
-- `docs-site/zh/troubleshooting/recover-after-kill.mdx` — 运行被强杀后恢复:进程被 kill -9、CI 超时或断电中断后，保留已发布结果、收口失去 owner 的 Run，并收回遗留 Sandbox。
+- `docs-site/zh/troubleshooting/debugging.mdx` — 排查失败的评估结果:评估失败或出错时，从具体的 Run 或 Attempt 出发，用 show 逐层查看错误、对话、耗时和改动，找到是应用、Adapter、环境还是断言的问题。
+- `docs-site/zh/troubleshooting/recover-after-kill.mdx` — 运行被强杀后恢复:进程被 kill -9、CI 超时或断电中断后，保留已经跑完的结果，结束失去进程的 Run，并收回遗留的 Sandbox。
 
 ## `docs-site/zh/examples/`
 

package/dist/adapter-execution-trace.cjs.map

@@ -1 +1 @@
-{"version":3,"file":"adapter-execution-trace.js","sourceRoot":"","sources":["adapter-execution-trace.ts"],"names":[],"mappings":"","sourcesContent":["/** Finite plain JSON accepted by the generic execution-trace boundary. */\nexport type TraceJson = null | boolean | number | string | readonly TraceJson[] | {\n    readonly [key: string]: TraceJson;\n};\nexport interface ExecutionTraceLimitation {\n    readonly code: string;\n    readonly message: string;\n}\nexport interface ExecutionTraceScope {\n    readonly scopeId: string;\n    readonly label: string;\n    readonly boundary: TraceJson;\n}\nexport interface ExecutionTraceSource {\n    readonly id: string;\n    readonly eventId?: string;\n    readonly sequence?: number;\n}\nexport interface ExecutionTraceActor {\n    readonly id: string;\n    readonly label?: string;\n}\nexport interface ExecutionTraceTime {\n    readonly clockId: string;\n    readonly value: number;\n    readonly unit: string;\n}\nexport type ExecutionTraceLink = {\n    readonly relation: string;\n    readonly targetKey: string;\n} | {\n    readonly relation: string;\n    readonly unresolved: {\n        readonly sourceEventId: string;\n        readonly reason: string;\n    };\n};\nexport interface ExecutionTraceEvidence {\n    readonly key: string;\n    readonly label: string;\n    readonly artifactId: string;\n    readonly pointer: string;\n}\nexport interface ExecutionTraceScopeMembership {\n    readonly scopeId: string;\n    readonly state: \"included\" | \"excluded\" | \"unknown\";\n}\n/**\n * One Adapter-owned domain event. `Type` and `Payload` preserve a caller's\n * discriminated union without requiring NiceEval to execute its domain parser.\n */\nexport interface ExecutionTraceEvent<out Type extends string = string, out Payload extends TraceJson = TraceJson> {\n    readonly key: string;\n    readonly type: Type;\n    readonly source: ExecutionTraceSource;\n    readonly actor?: ExecutionTraceActor;\n    readonly time?: ExecutionTraceTime;\n    readonly summary: string;\n    readonly payload?: Payload;\n    readonly links?: readonly ExecutionTraceLink[];\n    readonly evidence?: readonly ExecutionTraceEvidence[];\n    readonly scopeMemberships?: readonly ExecutionTraceScopeMembership[];\n}\nexport interface ExecutionTraceInput<out Event extends ExecutionTraceEvent = ExecutionTraceEvent> {\n    readonly traceId: string;\n    readonly schema: {\n        readonly id: string;\n    };\n    readonly collection: {\n        readonly state: \"complete\" | \"partial\";\n        readonly limitations: readonly ExecutionTraceLimitation[];\n    };\n    readonly scopes: readonly ExecutionTraceScope[];\n    readonly events: readonly Event[];\n}\nexport interface ExecutionTraceReceipt {\n    readonly state: \"accepted\";\n    readonly traceId: string;\n    readonly events: readonly {\n        readonly key: string;\n        readonly eventId: string;\n        readonly evidence: readonly {\n            readonly key: string;\n            readonly evidenceId: string;\n        }[];\n    }[];\n}\n"]}
+{"version":3,"file":"adapter-execution-trace.js","sourceRoot":"","sources":["adapter-execution-trace.ts"],"names":[],"mappings":"","sourcesContent":["/** Finite plain JSON accepted by the generic execution-trace boundary. */\nexport type TraceJson = null | boolean | number | string | readonly TraceJson[] | {\n    readonly [key: string]: TraceJson;\n};\nexport interface ExecutionTraceLimitation {\n    readonly code: string;\n    readonly message: string;\n}\nexport interface ExecutionTraceScope {\n    readonly scopeId: string;\n    readonly label: string;\n    readonly boundary: TraceJson;\n}\nexport interface ExecutionTraceSource {\n    readonly id: string;\n    readonly eventId?: string;\n    readonly sequence?: number;\n}\nexport interface ExecutionTraceActor {\n    readonly id: string;\n    readonly label?: string;\n}\nexport interface ExecutionTraceTime {\n    readonly clockId: string;\n    readonly value: number;\n    readonly unit: string;\n}\nexport type ExecutionTraceLink = {\n    readonly relation: string;\n    readonly targetKey: string;\n} | {\n    readonly relation: string;\n    readonly unresolved: {\n        readonly sourceEventId: string;\n        readonly reason: string;\n    };\n};\nexport interface ExecutionTraceEvidence {\n    readonly key: string;\n    readonly label: string;\n    readonly artifactId: string;\n    readonly pointer: string;\n}\nexport interface ExecutionTraceScopeMembership {\n    readonly scopeId: string;\n    readonly state: \"included\" | \"excluded\" | \"unknown\";\n}\n/** One closed, human-readable display block sealed with its event. */\nexport type ExecutionDisplayBlock = {\n    readonly kind: \"text\";\n    readonly text: string;\n} | {\n    readonly kind: \"message\";\n    readonly role: \"user\" | \"assistant\" | \"system\" | \"other\";\n    readonly speaker?: string;\n    readonly text: string;\n} | {\n    readonly kind: \"fields\";\n    readonly fields: readonly {\n        readonly label: string;\n        readonly value: string | number | boolean | null;\n    }[];\n} | {\n    readonly kind: \"code\";\n    readonly language?: string;\n    readonly text: string;\n} | {\n    readonly kind: \"image\";\n    readonly artifactId: string;\n    readonly alt: string;\n};\n/**\n * One Adapter-owned domain event. `Type` and `Payload` preserve a caller's\n * discriminated union without requiring NiceEval to execute its domain parser.\n */\nexport interface ExecutionTraceEvent<out Type extends string = string, out Payload extends TraceJson = TraceJson> {\n    readonly key: string;\n    readonly type: Type;\n    readonly source: ExecutionTraceSource;\n    readonly actor?: ExecutionTraceActor;\n    readonly time?: ExecutionTraceTime;\n    readonly summary: string;\n    readonly payload?: Payload;\n    readonly links?: readonly ExecutionTraceLink[];\n    readonly evidence?: readonly ExecutionTraceEvidence[];\n    readonly scopeMemberships?: readonly ExecutionTraceScopeMembership[];\n    /** Optional human-readable blocks; omitted means the event has no display. */\n    readonly display?: readonly ExecutionDisplayBlock[];\n}\nexport interface ExecutionTraceInput<out Event extends ExecutionTraceEvent = ExecutionTraceEvent> {\n    readonly traceId: string;\n    readonly schema: {\n        readonly id: string;\n    };\n    readonly collection: {\n        readonly state: \"complete\" | \"partial\";\n        readonly limitations: readonly ExecutionTraceLimitation[];\n    };\n    readonly scopes: readonly ExecutionTraceScope[];\n    readonly events: readonly Event[];\n}\nexport interface ExecutionTraceReceipt {\n    readonly state: \"accepted\";\n    readonly traceId: string;\n    readonly events: readonly {\n        readonly key: string;\n        readonly eventId: string;\n        readonly evidence: readonly {\n            readonly key: string;\n            readonly evidenceId: string;\n        }[];\n    }[];\n}\n"]}

package/dist/adapter-execution-trace.d.cts

@@ -45,6 +45,30 @@ export interface ExecutionTraceScopeMembership {
     readonly scopeId: string;
     readonly state: "included" | "excluded" | "unknown";
 }
+/** One closed, human-readable display block sealed with its event. */
+export type ExecutionDisplayBlock = {
+    readonly kind: "text";
+    readonly text: string;
+} | {
+    readonly kind: "message";
+    readonly role: "user" | "assistant" | "system" | "other";
+    readonly speaker?: string;
+    readonly text: string;
+} | {
+    readonly kind: "fields";
+    readonly fields: readonly {
+        readonly label: string;
+        readonly value: string | number | boolean | null;
+    }[];
+} | {
+    readonly kind: "code";
+    readonly language?: string;
+    readonly text: string;
+} | {
+    readonly kind: "image";
+    readonly artifactId: string;
+    readonly alt: string;
+};
 /**
  * One Adapter-owned domain event. `Type` and `Payload` preserve a caller's
  * discriminated union without requiring NiceEval to execute its domain parser.
@@ -60,6 +84,8 @@ export interface ExecutionTraceEvent<out Type extends string = string, out Paylo
     readonly links?: readonly ExecutionTraceLink[];
     readonly evidence?: readonly ExecutionTraceEvidence[];
     readonly scopeMemberships?: readonly ExecutionTraceScopeMembership[];
+    /** Optional human-readable blocks; omitted means the event has no display. */
+    readonly display?: readonly ExecutionDisplayBlock[];
 }
 export interface ExecutionTraceInput<out Event extends ExecutionTraceEvent = ExecutionTraceEvent> {
     readonly traceId: string;

package/dist/adapter-execution-trace.d.mts

@@ -45,6 +45,30 @@ export interface ExecutionTraceScopeMembership {
     readonly scopeId: string;
     readonly state: "included" | "excluded" | "unknown";
 }
+/** One closed, human-readable display block sealed with its event. */
+export type ExecutionDisplayBlock = {
+    readonly kind: "text";
+    readonly text: string;
+} | {
+    readonly kind: "message";
+    readonly role: "user" | "assistant" | "system" | "other";
+    readonly speaker?: string;
+    readonly text: string;
+} | {
+    readonly kind: "fields";
+    readonly fields: readonly {
+        readonly label: string;
+        readonly value: string | number | boolean | null;
+    }[];
+} | {
+    readonly kind: "code";
+    readonly language?: string;
+    readonly text: string;
+} | {
+    readonly kind: "image";
+    readonly artifactId: string;
+    readonly alt: string;
+};
 /**
  * One Adapter-owned domain event. `Type` and `Payload` preserve a caller's
  * discriminated union without requiring NiceEval to execute its domain parser.
@@ -60,6 +84,8 @@ export interface ExecutionTraceEvent<out Type extends string = string, out Paylo
     readonly links?: readonly ExecutionTraceLink[];
     readonly evidence?: readonly ExecutionTraceEvidence[];
     readonly scopeMemberships?: readonly ExecutionTraceScopeMembership[];
+    /** Optional human-readable blocks; omitted means the event has no display. */
+    readonly display?: readonly ExecutionDisplayBlock[];
 }
 export interface ExecutionTraceInput<out Event extends ExecutionTraceEvent = ExecutionTraceEvent> {
     readonly traceId: string;

package/dist/adapter-execution-trace.d.ts

@@ -45,6 +45,30 @@ export interface ExecutionTraceScopeMembership {
     readonly scopeId: string;
     readonly state: "included" | "excluded" | "unknown";
 }
+/** One closed, human-readable display block sealed with its event. */
+export type ExecutionDisplayBlock = {
+    readonly kind: "text";
+    readonly text: string;
+} | {
+    readonly kind: "message";
+    readonly role: "user" | "assistant" | "system" | "other";
+    readonly speaker?: string;
+    readonly text: string;
+} | {
+    readonly kind: "fields";
+    readonly fields: readonly {
+        readonly label: string;
+        readonly value: string | number | boolean | null;
+    }[];
+} | {
+    readonly kind: "code";
+    readonly language?: string;
+    readonly text: string;
+} | {
+    readonly kind: "image";
+    readonly artifactId: string;
+    readonly alt: string;
+};
 /**
  * One Adapter-owned domain event. `Type` and `Payload` preserve a caller's
  * discriminated union without requiring NiceEval to execute its domain parser.
@@ -60,6 +84,8 @@ export interface ExecutionTraceEvent<out Type extends string = string, out Paylo
     readonly links?: readonly ExecutionTraceLink[];
     readonly evidence?: readonly ExecutionTraceEvidence[];
     readonly scopeMemberships?: readonly ExecutionTraceScopeMembership[];
+    /** Optional human-readable blocks; omitted means the event has no display. */
+    readonly display?: readonly ExecutionDisplayBlock[];
 }
 export interface ExecutionTraceInput<out Event extends ExecutionTraceEvent = ExecutionTraceEvent> {
     readonly traceId: string;

package/dist/adapter-usage.cjs.map

@@ -1 +1 @@
-{"version":3,"file":"adapter-usage.js","sourceRoot":"","sources":["adapter-usage.ts"],"names":[],"mappings":"","sourcesContent":["/** Final observation of one physical external request, scoped to an Adapter Attempt. */\nexport interface AdapterUsageInput {\n    readonly callId: string;\n    readonly retryOf?: string;\n    /** Upstream-proven serving provider. Transport and protocol identities belong in route. */\n    readonly provider: string | null;\n    readonly model: string | null;\n    readonly route?: {\n        readonly transportProvider: string | null;\n        /** Stable, non-secret endpoint identity. Never include URLs, account identifiers, or credentials. */\n        readonly endpointId: string | null;\n    };\n    readonly status: \"succeeded\" | \"failed\" | \"cancelled\" | \"unknown\";\n    /** Input tokens excluding the cache read and cache write buckets. */\n    readonly inputTokens: number | null;\n    /** Independently observed input total including cache buckets; never added to those buckets. */\n    readonly inputTotalTokens?: number | null;\n    readonly outputTokens: number | null;\n    readonly cacheReadTokens?: number | null;\n    readonly cacheWriteTokens?: number | null;\n    /** One provider-reported effective cost. A reported zero remains preferred to estimates. */\n    readonly cost?: {\n        readonly amount: string;\n        readonly currency: string;\n        readonly source: {\n            readonly kind: \"reported\";\n            readonly id: string;\n        };\n    };\n}\n"]}
+{"version":3,"file":"adapter-usage.js","sourceRoot":"","sources":["adapter-usage.ts"],"names":[],"mappings":"","sourcesContent":["/** Producer declaration after all obtainable final request snapshots have been recorded. */\nexport type AdapterUsageSeal = {\n    readonly state: \"complete\";\n} | {\n    readonly state: \"partial\";\n    readonly reason: string;\n};\n/** Final observation of one physical external request, scoped to an Adapter Attempt. */\nexport interface AdapterUsageInput {\n    readonly callId: string;\n    readonly retryOf?: string;\n    /** Frozen application model purpose; omission records an unknown purpose. */\n    readonly modelSlot?: string | null;\n    /** Upstream-proven serving provider. Transport and protocol identities belong in route. */\n    readonly provider: string | null;\n    readonly model: string | null;\n    readonly route?: {\n        readonly transportProvider: string | null;\n        /** Stable, non-secret endpoint identity. Never include URLs, account identifiers, or credentials. */\n        readonly endpointId: string | null;\n    };\n    readonly status: \"succeeded\" | \"failed\" | \"cancelled\" | \"unknown\";\n    /** Input tokens excluding the cache read and cache write buckets. */\n    readonly inputTokens: number | null;\n    /** Independently observed input total including cache buckets; never added to those buckets. */\n    readonly inputTotalTokens?: number | null;\n    readonly outputTokens: number | null;\n    readonly cacheReadTokens?: number | null;\n    readonly cacheWriteTokens?: number | null;\n    /** One provider-reported effective cost. A reported zero remains preferred to estimates. */\n    readonly cost?: {\n        readonly amount: string;\n        readonly currency: string;\n        readonly source: {\n            readonly kind: \"reported\";\n            readonly id: string;\n        };\n    };\n}\n"]}

package/dist/adapter-usage.d.cts

@@ -1,7 +1,16 @@
+/** Producer declaration after all obtainable final request snapshots have been recorded. */
+export type AdapterUsageSeal = {
+    readonly state: "complete";
+} | {
+    readonly state: "partial";
+    readonly reason: string;
+};
 /** Final observation of one physical external request, scoped to an Adapter Attempt. */
 export interface AdapterUsageInput {
     readonly callId: string;
     readonly retryOf?: string;
+    /** Frozen application model purpose; omission records an unknown purpose. */
+    readonly modelSlot?: string | null;
     /** Upstream-proven serving provider. Transport and protocol identities belong in route. */
     readonly provider: string | null;
     readonly model: string | null;

package/dist/adapter-usage.d.mts

@@ -1,7 +1,16 @@
+/** Producer declaration after all obtainable final request snapshots have been recorded. */
+export type AdapterUsageSeal = {
+    readonly state: "complete";
+} | {
+    readonly state: "partial";
+    readonly reason: string;
+};
 /** Final observation of one physical external request, scoped to an Adapter Attempt. */
 export interface AdapterUsageInput {
     readonly callId: string;
     readonly retryOf?: string;
+    /** Frozen application model purpose; omission records an unknown purpose. */
+    readonly modelSlot?: string | null;
     /** Upstream-proven serving provider. Transport and protocol identities belong in route. */
     readonly provider: string | null;
     readonly model: string | null;

package/dist/adapter-usage.d.ts

@@ -1,7 +1,16 @@
+/** Producer declaration after all obtainable final request snapshots have been recorded. */
+export type AdapterUsageSeal = {
+    readonly state: "complete";
+} | {
+    readonly state: "partial";
+    readonly reason: string;
+};
 /** Final observation of one physical external request, scoped to an Adapter Attempt. */
 export interface AdapterUsageInput {
     readonly callId: string;
     readonly retryOf?: string;
+    /** Frozen application model purpose; omission records an unknown purpose. */
+    readonly modelSlot?: string | null;
     /** Upstream-proven serving provider. Transport and protocol identities belong in route. */
     readonly provider: string | null;
     readonly model: string | null;

package/dist/adapter.cjs

@@ -6,6 +6,8 @@ exports.adapterIdentity = adapterIdentity;
 exports.adapterAcceptsEval = adapterAcceptsEval;
 exports.adapterContractRequiredByEval = adapterContractRequiredByEval;
 exports.bindAdapterEvalContext = bindAdapterEvalContext;
+const adapter_cleanup_ts_1 = require("./shared/adapter-cleanup.cjs");
+const runtime_ts_1 = require("./assertions/runtime.cjs");
 const adapter_flags_ts_1 = require("./adapter-flags.cjs");
 const define_ts_1 = require("./define.cjs");
 const ADAPTER_CONTRACT_TOKEN = Symbol("niceeval.adapterContractToken");
@@ -20,6 +22,10 @@ const RESERVED_ADAPTER_CONTEXT_KEYS = [
     "instructionFollowing",
     "pairwisePreference",
     "closeQA",
+    "usage",
+    "elapsedMs",
+    "maxTokens",
+    "maxCost",
     "score",
     "group",
     "skip",
@@ -91,6 +97,7 @@ function implementAdapter(token, input, factory) {
     if (typeof input.create !== "function") {
         throw new TypeError(`${factory} requires create(context).`);
     }
+    const cleanupTimeoutMs = (0, adapter_cleanup_ts_1.resolveAdapterCleanupTimeoutMs)(input.cleanupTimeoutMs);
     const authorCreate = input.create;
     const create = (context) => {
         const created = authorCreate(context);
@@ -110,11 +117,15 @@ function implementAdapter(token, input, factory) {
         name: input.name,
         contract: token.name,
         behaviorRevision: input.behaviorRevision ?? null,
+        cleanupTimeoutMs,
         create,
         [ADAPTER_CONTRACT_TOKEN]: token,
         ...boundEvalFactories(token),
     });
 }
+// @concord-code ne-adapter-define-adapter
+// @concord-implements docs/feature/adapters/README.md
+// @concord-implements docs/feature/adapters/library/writing-an-adapter.md
 function defineAdapter(input) {
     if (input.parseFlags !== undefined)
         (0, adapter_flags_ts_1.assertAdapterFlagsParser)(input.parseFlags);
@@ -125,6 +136,9 @@ function defineAdapter(input) {
     return implementAdapter(contractToken(input.name, assertions, input.parseFlags), input, "defineAdapter");
 }
 /** Defines a reusable contract whose implementations and Evals share one runtime-only token. */
+// @concord-code ne-adapter-define-contract
+// @concord-implements docs/feature/adapters/README.md
+// @concord-implements docs/feature/adapters/architecture/agent-contract.md
 function defineAdapterContract(input) {
     assertNonEmptyString(input.name, "name", "defineAdapterContract");
     return makeAdapterContract(contractToken(input.name, undefined, undefined));
@@ -157,6 +171,7 @@ function adapterIdentity(adapter) {
             name: adapter.name,
             contract: adapter.contract,
             behaviorRevision: adapter.behaviorRevision,
+            cleanupTimeoutMs: adapter.cleanupTimeoutMs,
         })
         : Object.freeze({
             name: adapter.name,
@@ -301,10 +316,7 @@ function bindAdapterAssertions(core, context, factory, assertAuthorOpen) {
         frame.add(handle);
         return handle;
     };
-    const authored = factory({
-        app,
-        check: check,
-    });
+    const authored = factory({ app: app, check: check });
     assertPlainAdapterAssertions(authored, context);
     Object.freeze(authored);
     assembled = true;
@@ -337,6 +349,8 @@ function bindAdapterAssertions(core, context, factory, assertAuthorOpen) {
     return Object.freeze({ app, assertions: Object.freeze(assertions) });
 }
 /** @internal Builds the frozen single-t facade after create settles. */
+// @concord-code ne-adapter-bind-eval-context
+// @concord-implements docs/feature/adapters/architecture/agent-contract.md
 function bindAdapterEvalContext(core, context, adapter, assertAuthorOpen) {
     assertPlainAdapterContext(context);
     const token = adapter[ADAPTER_CONTRACT_TOKEN];
@@ -350,6 +364,7 @@ function bindAdapterEvalContext(core, context, adapter, assertAuthorOpen) {
         app = bound.app;
         assertions = bound.assertions;
     }
+    (0, runtime_ts_1.bindAssertionContext)(core.check, () => app);
     const target = Object.create(null);
     forwardProperties(target, core, false, assertAuthorOpen);
     forwardProperties(target, app, false, assertAuthorOpen);