niceeval 0.4.4 → 0.4.5

package/docs-site/concepts/judge.mdx

@@ -0,0 +1,129 @@
+---
+title: "LLM-as-judge in niceeval: scoring open-ended output"
+sidebarTitle: "Judge"
+description: "How t.judge, session.judge, and turn.judge use a separate evaluator model to score factuality, closed-ended quality, and summarization — plus model resolution and severity."
+---
+
+Judge assertions are the fifth scoring mechanism, alongside the four covered in [Assert](/concepts/assert). Use them when correctness cannot be expressed as a rule — for open-ended prose, tone, factual consistency, or summarization quality. The evaluator model is **entirely separate** from the agent under test, which prevents self-evaluation bias: the same model grading its own output would be prone to rating itself generously.
+
+```ts
+t.judge.autoevals.factuality(reference).atLeast(0.8);   // factual consistency with a reference text
+t.judge.autoevals.closedQA("Is the response appropriate for a 10-year-old?");
+t.judge.autoevals.summarizes(sourceDocument);            // faithful summarization check
+t.judge.autoevals.closedQA("Custom scoring rubric description", { on: t.reply });
+```
+
+## The three judge shapes
+
+`t.judge.autoevals` exposes exactly three fixed forms, whatever you're evaluating maps onto one of them:
+
+- **`closedQA(question, opts?)`** — a closed-ended judgment: does the output satisfy this criterion? Best for rubric-style checks ("is the tone polite and specific", "does this avoid inventing policy details").
+- **`factuality(reference, opts?)`** — factual consistency between the output and a reference text you provide.
+- **`summarizes(source, opts?)`** — whether the output is a faithful summary of a source document.
+
+The `{ on }` option specifies which value to evaluate; it defaults based on the receiver (see below). The `{ model }` option overrides the judge model for a single call.
+
+## Where judge lives: `t`, `session`, and `turn`
+
+Like scoped assertions, judge calls follow the rule that **the receiver decides the default material, not the call site**:
+
+```ts
+t.judge.autoevals.closedQA("...")        // defaults to the primary session's full conversation text
+session.judge.autoevals.closedQA("...")  // (from t.newSession()) defaults to that session's conversation text
+turn.judge.autoevals.closedQA("...")     // (from t.send()'s return value) defaults to turn.message only
+```
+
+`t.judge` and `session.judge` are session-level: they're the right choice for judging overall reply quality or cross-turn consistency across a whole conversation. `turn.judge` is turn-level: it only sees the message from that one turn, which is what you want when different turns in a multi-turn eval need different rubrics:
+
+```ts
+const turn1 = await t.send("What's in this image?");                 // turn 1: image
+const turn2 = await t.send("What color is the background?");         // turns 2–3: text follow-ups
+const turn3 = await t.send("What color is the shape in the middle?"); // testing cross-turn memory
+
+t.judge.autoevals
+  .closedQA("Does the assistant stay grounded in the image from turn 1 throughout?")
+  .atLeast(0.7);
+
+turn3.judge.autoevals.closedQA("Does this turn answer the shape's color?").gate();
+```
+
+To judge material that isn't the conversation itself — a sandbox diff, file content, or anything else — pass it explicitly with `{ on }`, regardless of which receiver you're calling from:
+
+```ts
+t.judge.autoevals.closedQA("Is the generated code idiomatic TypeScript?", {
+  on: t.sandbox.diff.get("src/handler.ts"),
+}).atLeast(0.7);
+```
+
+## Judge model resolution
+
+The judge model is resolved from most- to least-specific:
+
+```
+Per-call { model } option
+  ↓
+Per-eval judge.model
+  ↓
+Global config judge.model
+```
+
+```ts
+// niceeval.config.ts — global default
+defineConfig({ judge: { model: "anthropic/claude-haiku-4-5" } });
+
+// A specific eval that needs a more capable judge
+defineEval({
+  judge: { model: "anthropic/claude-opus-4-8" },
+  async test(t) {
+    t.judge.autoevals.factuality(reference);  // uses claude-opus-4-8 for this eval
+  },
+});
+
+// A single call override
+t.judge.autoevals.closedQA("rubric", { on: t.reply, model: "openai/gpt-4o" });
+```
+
+If all three levels are unset, NiceEval falls back to the environment variable `NICEEVAL_JUDGE_MODEL`. If that is also unset, the judge call fails as a configuration error. There is no built-in default judge model.
+
+## Judge endpoint and key: OpenAI-compatible protocol
+
+Judge calls use an **OpenAI-compatible `/chat/completions`** endpoint. OpenAI, DeepSeek, and any compatible gateway can act as the judge backend:
+
+```ts
+defineConfig({
+  judge: {
+    model: "deepseek-chat",
+    baseUrl: "https://api.deepseek.com/v1",
+    apiKeyEnv: "DEEPSEEK_API_KEY",
+  },
+});
+```
+
+`baseUrl` and the API key are both resolved from most-specific to most-generic:
+
+| Item | Resolution order |
+|---|---|
+| Endpoint | `judge.baseUrl` -> `NICEEVAL_JUDGE_BASE` -> `CODEX_BASE_URL` -> `OPENAI_BASE_URL` -> `https://api.openai.com/v1` |
+| Key | env var named by `judge.apiKeyEnv` -> `NICEEVAL_JUDGE_KEY` -> `CODEX_API_KEY` -> `OPENAI_API_KEY` |
+
+<Warning>
+  When no API key resolves, judge assertions are skipped silently: no error, no score, and the eval keeps running. After configuring keys, run one eval with `t.judge` and confirm the score appears in `niceeval view`.
+</Warning>
+
+## Severity: judge defaults to soft
+
+Judge calls are scoring functions like any other assertion, so they follow the same gate/soft mechanics described in [Assert · Gate vs soft severity](/concepts/assert#gate-vs-soft-severity) — but their **default** is different from most value matchers:
+
+```ts
+t.judge.autoevals.closedQA("Is the tone polite?");              // no threshold → soft, pure score, never fails the eval
+t.judge.autoevals.closedQA("Is the tone polite?").atLeast(0.7); // soft with a threshold → fails only under --strict
+t.judge.autoevals.closedQA("Is the tone polite?").gate();       // promoted to gate → fails immediately if below threshold
+```
+
+With no `.atLeast()` and no `.gate()`, a bare judge call is recorded purely as a quality score — it shows up on the eval's score chip but can never fail the run on its own. This is the right default for judge: it's a probabilistic model scoring another probabilistic model's output, and treating every rubric miss as a hard failure would make suites flaky. Reach for `.gate()` only when the judged property is truly load-bearing (for example, "the response must not fabricate account details").
+
+## Related pages
+
+- [Assert](/concepts/assert) — gate vs soft severity in full, plus the outcome rules judge scores feed into.
+- [Drive](/concepts/drive) — `t.send()`, `t.newSession()`, and the `Turn` / session handles that `t.judge`, `session.judge`, and `turn.judge` hang off of.
+- [Evals](/concepts/evals) — how judge scores fold into the eval lifecycle and outcome types.

package/docs-site/concepts/overview.mdx

@@ -0,0 +1,98 @@
+---
+title: "NiceEval architecture: evals, agents, and sandboxes"
+sidebarTitle: "Overview"
+description: "Understand how NiceEval, Adapters, and sandbox backends work together to evaluate any AI agent through one API."
+---
+
+NiceEval separates **what you evaluate** from **how the subject under test is connected**. NiceEval discovers evals, schedules runs, scores assertions, and writes reports. Adapters call the system under test. Sandbox backends isolate filesystems for coding-agent tasks.
+
+## Four layers
+
+```text
+Eval files / fixtures
+        |
+        v
+NiceEval
+        |
+        v
+Adapter
+        |
+        v
+Subject under test / Sandbox backend
+```
+
+## What NiceEval owns
+
+<CardGroup cols={2}>
+  <Card title="Eval discovery" icon="magnifying-glass">
+    Finds `*.eval.ts` files and fixture directories, then derives stable IDs from paths.
+  </Card>
+  <Card title="Scheduling" icon="timeline">
+    Controls concurrency, retries, attempts, runs, and early exit.
+  </Card>
+  <Card title="Assertions and scoring" icon="chart-bar">
+    Collects `t.check`, scoped assertions, judge scores, and test results.
+  </Card>
+  <Card title="Cache" icon="database">
+    Skips cases that already passed when the fingerprint is unchanged.
+  </Card>
+  <Card title="Reports" icon="file-lines">
+    Writes console, JSON, JUnit, and local viewer data.
+  </Card>
+  <Card title="Artifacts" icon="folder-open">
+    Saves summaries, event streams, transcripts, diffs, traces, and logs.
+  </Card>
+</CardGroup>
+
+## Adapter
+
+<Note>
+  `Agent` is the abstraction NiceEval calls. `Adapter` is the concrete implementation you write. NiceEval does not know your private protocol, CLI arguments, or auth rules.
+</Note>
+
+That boundary keeps the runner and scorers generic. Evaluating your own AI app, Claude Code, Codex, or a custom agent should not change core scoring behavior.
+
+## Sandbox
+
+<Tabs>
+  <Tab title="Docker">
+    Local container backend for coding-agent evals in development and CI.
+  </Tab>
+  <Tab title="Vercel Sandbox">
+    Cloud sandbox backend for stronger isolation or larger resources.
+  </Tab>
+  <Tab title="Third-party backend">
+    Any service can integrate by implementing the `Sandbox` interface.
+  </Tab>
+</Tabs>
+
+## Key terms
+
+<AccordionGroup>
+  <Accordion title="Eval">A test case: description, optional agent reference, and `test(t)`.</Accordion>
+  <Accordion title="Agent">The connection NiceEval calls by name. It returns a standard `Turn`.</Accordion>
+  <Accordion title="Adapter">The concrete implementation that knows how to call your runtime or CLI.</Accordion>
+  <Accordion title="Sandbox">The isolated runtime used by coding agents.</Accordion>
+  <Accordion title="Turn">The immutable result snapshot from one `t.send()` or `t.respond()`.</Accordion>
+  <Accordion title="Artifact">A structured result file written after a run.</Accordion>
+  <Accordion title="Experiment">A checked-in run configuration for comparing agents, models, flags, runs, budgets, and sandboxes.</Accordion>
+</AccordionGroup>
+
+## End-to-end flow
+
+<Steps>
+  <Step title="Discovery">Find eval files and fixtures.</Step>
+  <Step title="Scheduling">Plan attempts from concurrency, cache, runs, and early-exit settings.</Step>
+  <Step title="Agent send">Call the selected adapter and receive a standard `Turn`.</Step>
+  <Step title="Scoring">Run assertions, judge checks, and tests.</Step>
+  <Step title="Outcome">Fold results into `passed`, `failed`, `errored`, or `skipped`.</Step>
+  <Step title="Reporting & artifacts">Write reports and structured files.</Step>
+</Steps>
+
+## Related
+
+- [Evals](/concepts/evals) - what an eval is and how its lifecycle works.
+- [Adapter](/concepts/adapter) - how to write an adapter and use it in an experiment.
+- [Drive](/concepts/drive) - `t.send()`, sessions, and HITL from the eval side.
+- [Assert](/concepts/assert) - assertion vocabulary and outcome rules.
+- [Judge](/concepts/judge) - LLM-as-judge for open-ended quality.

package/docs-site/concepts/tier.mdx

@@ -0,0 +1,42 @@
+---
+title: "Integration Tiers: Three Levels of Investment, Three Sets of Capabilities"
+sidebarTitle: "Tier"
+description: "Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers: Tier 1 is send only, Tier 2 adds OTel, Tier 3 modifies the application to expose experiment flags. What each tier buys and when to move up."
+---
+
+Connecting to NiceEval is not an all-or-nothing one-time project. Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers — each tier adds one kind of investment on top of the previous one and buys a new set of capabilities, and every eval you have already written keeps working unchanged the whole way up.
+
+## Tier 1: send only
+
+Your application code stays untouched; the integration is a single Adapter file — `send` talks to the same interface your user-facing frontend already uses.
+
+**The full assertion set lives at this tier**: text assertions and judges, structured output validation, multi-turn and session isolation, HITL approval flows, tool assertions from event mapping, and usage assertions from the `usage` returned by `send` — every verdict is based entirely on the `Turn` returned by `send`, and the next two tiers add no further assertions. If the application interface itself exposes model selection, **model comparison** experiments also live at this tier — the experiment's `model` reaches the Adapter via `ctx.model` and is simply forwarded with the request.
+
+Most teams start here, and for many this tier is all they ever need.
+
+## Tier 2: send + OTel
+
+Same `send`, same event mapping — the only change is that your application also sends its OTel spans to NiceEval. If the app is already instrumented (AI SDK telemetry, LangGraph, OpenLLMetry / OpenInference, hand-rolled gen_ai spans), this is zero change; if not, what you add is a piece of generic OTel initialization — that is observability infrastructure, not an eval-specific modification.
+
+What this tier buys is **observability**: the call waterfall in `niceeval view` — every model call and tool execution inside the application, each with its own latency and tokens, laid out as a per-turn timeline. Assertions are unaffected: spans only feed the waterfall; they never enter the event stream or feed assertions. See [OTel Integration](/guides/connect-otel) for the walkthrough.
+
+## Tier 3: application changes + experiment flags
+
+When you want to evaluate "which prompt / tool set / feature toggle works better" — a **feature A/B test** — the point of comparison lives inside the application, out of reach for the first two tiers. This tier changes the application's internal code to expose the variants as externally selectable configuration: the experiment's `flags` pass through `ctx.flags` to the Adapter, the Adapter hands them to the application with the request (an HTTP header, a request body field, or an environment variable all work), and the application switches variants by flag.
+
+Note that what gets modified is the application (exposing variants as configuration), not the integration surface — the Adapter itself stays the same and still calls the application only through its external interface.
+
+## Non-intrusive is the baseline for the first two tiers
+
+The first two tiers are both **non-intrusive**: you start the application your own way (`pnpm start`, deployed wherever) — the eval side never spawns the application process or opens a new port. The Adapter only talks to **the same interface your user-facing frontend already uses** (an HTTP endpoint, an SSE stream), and reports a clear "start the application first" error when it cannot connect. The only difference between the two tiers is observability data: Tier 1 has only the events returned by `send`; Tier 2 adds a copy of spans sent to NiceEval, and the waterfall appears with them.
+
+## How to move up
+
+The three tiers are progressive, not mutually exclusive: start with Tier 1 to get baselines and model comparison running; move up to Tier 2 when you want the call waterfall; move up to Tier 3 when you need to compare variants inside the application. Each upgrade only adds something to the Adapter or the application — how to organize comparisons on the experiment side is covered in [Experiments](/concepts/experiment).
+
+## Related reading
+
+- [Connect Your Agent](/guides/connect-your-agent) — The integration overview: minimal integration and parameter channels.
+- [Adapter](/concepts/adapter) — The contract itself: what `send` receives and returns, and how `ctx.model` / `ctx.telemetry` / `ctx.flags` appear per tier.
+- [OTel Integration](/guides/connect-otel) — The full Tier 2 walkthrough.
+- [Experiments](/concepts/experiment) — How model / flags comparisons are declared on the experiment side.

package/docs-site/docs-ref/00-index.md

@@ -0,0 +1,23 @@
+# 给 Agent 的 docs 写作指南
+
+整理日期:2026-07-04。蒸馏自 [`../dev-docs-dx-sources/`](../dev-docs-dx-sources/00-index.md) 的来源级提取(Diátaxis、Divio、Google / Microsoft / GitHub 风格指南、Write the Docs、The Good Docs Project、MDN、I'd Rather Be Writing、GetDX / Twilio / Phodal 的 DX 资料)。
+
+这套指南回答"怎么写好一页用户文档",按写作流程组织。仓库特定的术语表、目录职责和校验命令以 [`docs-site/AGENTS.md`](../../AGENTS.md) 为准,本指南不重复维护;两者冲突时以 AGENTS.md 为准。
+
+## 使用顺序
+
+写任何 `docs-site/` 页面前,按顺序过一遍:
+
+1. [01-page-types.md](./01-page-types.md):先判定这页是 Tutorial、How-to、Reference、Explanation 还是 Troubleshooting。这是最重要的一步,页面类型错了后面全错。
+3. [03-style-rules.md](./03-style-rules.md):行文、步骤、可扫描性、标题与搜索意图。
+4. [04-code-examples.md](./04-code-examples.md):代码示例的写法和维护义务。
+5. [05-dx-failure-paths.md](./05-dx-failure-paths.md):失败路径覆盖和"文档是 DX 的一部分"视角。
+6. [06-checklists.md](./06-checklists.md):动笔前、按类型、发布前三层检查清单。
+
+## 三条最高原则
+
+如果只记三件事:
+
+1. **一页只服务一种需求。** 学习、干活、查事实、理解,四选一。页面边界模糊是文档难用的首要原因。(Diátaxis)
+2. **"刚好足够"胜过"全面覆盖"。** 更多内容会让所有内容更难找;优先写高频高价值路径,不为完整性堆低频分支。(GitHub Docs)
+3. **错误的文档比没有文档更糟。** 每个声明的命令、字段、路径、能力,写之前先对当前代码或真实运行结果核实;未实现的能力不进正式文档。(Write the Docs / MDN)

package/docs-site/docs-ref/01-page-types.md

@@ -0,0 +1,43 @@
+# 第一步:判定页面类型
+
+蒸馏自 [01-diataxis-divio.md](../dev-docs-dx-sources/01-diataxis-divio.md)、[06-good-docs-project.md](../dev-docs-dx-sources/06-good-docs-project.md)。
+
+## 核心规则
+
+动笔前先回答:**这页服务读者的哪种需求——学习、干活、查事实,还是理解?**
+
+| 类型 | 读者需求 | 一句话定义 |
+| --- | --- | --- |
+| Tutorial | 学习 | 带用户安全地完成第一次成功,建立信心和心智模型 |
+| How-to | 干活 | 帮已有基础的用户完成一个现实任务 |
+| Reference | 查事实 | 准确、完整、低解释负担地描述字段、命令、类型 |
+| Explanation (Concept) | 理解 | 解释概念边界、背景、取舍和"为什么" |
+| Troubleshooting | 修复 | 按症状定位原因并给出解决步骤 |
+
+如果一个页面同时承担两种以上主要目标,**优先拆页**,不要加小标题硬塞。
+
+## NiceEval 目录映射
+
+- `quickstart`:Tutorial。只保留一条最短成功路径。
+- `guides/*`:How-to。一个页面解决一个任务。
+- `concepts/*`:Explanation。解释核心心智模型和概念边界。
+- `reference/*`:Reference。只列当前实现支持的事实。
+- `troubleshooting/*`(未来):按症状组织排错。
+- `example` / `examples/*`:真实可运行例子,不承担完整解释。
+
+## 每种类型的典型越界(写之前自查)
+
+- Tutorial 不要变成 API 字段大全,不要展示全部选项和复杂边界。
+- How-to 不要教零基础概念,不要写成长篇设计说明。
+- Reference 不要混入 onboarding 叙事,不要写未实现字段或 roadmap。
+- Explanation 不要伪装成操作步骤,不要写成内部设计流水账。
+- Concept 页要写"它不是什么",防止用户把相邻层次混在一起(例如 Adapter 和 Sandbox)。
+
+## 检查问题
+
+- 这页是在服务学习、工作、查事实,还是理解?
+- 页面是否同时承担了两种以上主要目标?
+- 如果是 Tutorial,用户能否安全地拿到第一次成功?
+- 如果是 How-to,读完能否完成一个具体任务?
+- 如果是 Reference,信息是否完整、准确、无叙事打断?
+- 如果是 Explanation,是否真的解释了边界和原因,而不是在重复操作步骤?

package/docs-site/docs-ref/03-style-rules.md

@@ -0,0 +1,45 @@
+# 第三步:行文规则
+
+## 语气与句法
+
+- 第二人称,直接对读者说话;主动语态,动作主体明确。
+- 短句。一段一个想法,一句一个主要信息。
+- 对话式但不轻浮;直接帮用户完成任务,不要营销式铺垫。
+- 不解释开发者已知的基础常识(一般的 TypeScript、Node.js、CLI 用法),除非 NiceEval 依赖某个不明显细节。
+- 删掉"下面我们将会""值得注意的是"这类预告式和填充式表达,直接说要完成什么。
+- 命令、路径、flag、文件名、包名、代码标识一律用反引号(术语用法见 `docs-site/AGENTS.md` 术语表)。
+
+## 步骤写作
+
+- 步骤为完成任务服务,用编号列表;单步任务不要硬写成编号流程。
+- 引导句提供上下文,不要只重复标题。
+- 一步一个主要动作。
+- 先说在哪里操作,再说做什么;条件放在动作前面("如果要 X,先运行 Y",不是"运行 Y,如果你要 X")。
+- 可选步骤用 `Optional:` / "可选:" 明确标注。
+- 多种做法并存时,先写最短、最简单的默认路径;变体放独立小节或独立页面,不要在主路径里分叉。
+- 最好的步骤是用户不需要读的步骤:如果 CLI 输出或错误信息能自解释,优先改进它们,再考虑补文档。
+
+## 可扫描性
+
+多数读者在扫描,不在通读:
+
+- 重要信息放页面前 1/3。
+- 小标题要有信息量,读者只看标题也能定位内容。
+- 列表、表格、代码块、提示块合理分隔正文。
+- 强调(加粗等)只用于真正重要的信息,滥用等于没有。
+- 链接文字要有描述性,不写"点击这里"。
+
+## 标题与搜索意图
+
+- 搜索是用户进入文档的主要入口。标题、description、正文要用**用户会搜索的词**,不是内部功能名——"连接远程 Agent"比"Adapter primitives"更适合作任务页标题。
+- 每篇文章一个清晰、离散的主题。
+- 重命名或移动页面时维护 redirect,避免旧链接失效。
+
+## 内容边界与重复
+
+- "刚好足够"优于穷举:优先高影响、高价值场景,不为完整性堆低频分支。
+- ARID,不是 DRY:文档允许适度重复关键上下文,不要为消灭重复迫使读者跨三页拼信息。但**同一事实的权威来源只有一处**,其他地方链接过去——docs-site、README、示例、CLI help 之间要有清晰边界,避免同一能力多处漂移。
+- 一旦选择覆盖某类信息就覆盖完整;只做部分覆盖时要明说。
+- FAQ 不是信息架构:真实高频问题应沉淀为 guide、reference 或 troubleshooting 页。
+- 内容要可精确链接(addressable):重要段落有稳定的标题锚点,方便从错误信息、报告和其它页面直链。
+- 先讲前置概念,再讲依赖它的内容(cumulative)。

package/docs-site/docs-ref/04-code-examples.md

@@ -0,0 +1,37 @@
+# 第四步:代码示例规则
+
+## 写示例前
+
+- **先真实跑过。** 写 reference 或示例前,实际运行过这个 API / 命令,而不是从类型定义或设计稿抄字段。这是 MDN 和 I'd Rather Be Writing 共同的第一条规则。
+- 知道这个 API 最常见的使用路径,示例展示的应该是它,不是罕见变体。
+
+## 示例的结构
+
+按"目的 → 代码 → 解释 → 结果"组织:
+
+1. 代码块前一句话说明整体目的(它完成什么任务,不是逐行复述)。
+2. 代码使用真实导入路径、真实文件名;能被复制运行。
+3. 占位符(`<your-agent-url>` 之类)逐个解释清楚。
+4. 如果输出影响读者判断,展示或描述期望输出。
+
+## 注释
+
+- 注释解释关键行的作用和**为什么这样写**,不假设读者知道示例为什么长这样。
+- 注释增加复杂度,只在确有必要时用;短示例配好前置说明通常不需要行内注释。
+
+## 输入与输出成对
+
+发送/调用类 API 的示例要同时有 request / input 示例和 response / output 示例。只给输入不给输出,读者无法验证自己跑对了。
+
+## Reference 与示例互相补足
+
+- Reference 讲字段、类型、默认值;示例讲具体用法。只有其一都不完整。
+- 每个 reference 页配至少一个短示例;每个示例链接回字段细节所在的 reference。
+
+## 维护义务
+
+- 示例随 API 更新维护:改 API 时必须同步检查所有相关示例、注释、描述和链接。
+- 过期示例比没有示例更危险,发现后优先修正或删除。
+- 一页里多个相似示例考虑合并,减少读者负担。
+- 新旧写法都有效时,明确记录差异,不要让读者猜哪个是现在的推荐。
+- 文档链接示例目录时必须指向真实存在的路径(当前完整示例在 `examples/zh/`)。

package/docs-site/docs-ref/05-dx-failure-paths.md

@@ -0,0 +1,45 @@
+# 第五步:失败路径与 DX 视角
+
+## 文档是 DX 系统的一部分
+
+文档不是孤立内容资产。错误信息、CLI 输出、报告页、示例、CHANGELOG、迁移指南都在承担文档职责。写一页文档前先问:**这页降低了哪类开发者摩擦?这个摩擦是不是该由 CLI 输出或错误信息解决?**
+
+- 报错即文档:错误信息应链接到可操作说明,并给出下一步。
+- 能用更好的 CLI 输出、默认值或自动化解决的问题,优先改产品,再补文档。
+- 测试用例和可运行示例也是用户理解用法的资料。
+
+## 覆盖失败路径,不只是成功路径
+
+用户第一次失败后能否自助修复,是文档质量的核心指标。NiceEval 文档要覆盖的失败类别:
+
+- setup 失败。
+- adapter 连接失败。
+- sandbox / runtime 失败。
+- eval assertion 失败。
+- judge 失败。
+- report / view 失败。
+
+每类失败都要告诉用户五件事:
+
+1. 现象(用户看到什么)。
+2. 看哪个 artifact。
+3. 运行哪个命令。
+4. 常见原因。
+5. 下一步。
+
+## 可预测性:结构一致是一种功能
+
+Twilio 的 DX 光谱里,比"能跑通"更高的目标是"可预测":同类页面结构一致,用户熟悉一个 adapter 的文档后,能零成本迁移到另一个 adapter 的文档,知道信息在哪个位置。所以:
+
+- 同类型页面用同一骨架(见 [02-page-skeletons.md](./02-page-skeletons.md)),不要每页自创结构。
+- 概念(Agent / Adapter / Sandbox / Turn)的解释要让用户能**预测系统行为**,不只是背定义。
+
+## 衡量文档好坏的信号
+
+写作和评审时可以对照:
+
+- Time to first successful eval 是否被 quickstart 缩短。
+- 反复出现的用户问题是否有对应文档页。
+- 文档能否从错误信息、报告、CLI 输出、README 到达。
+- 文档与产品实际行为是否一致——不一致按 Twilio 的标准就是 Broken。
+- 同一问题在 Slack / issue 里反复出现、新人 onboarding 慢、PR 因缺上下文变慢,都是文档缺口的组织信号。

package/docs-site/docs-ref/06-checklists.md

@@ -0,0 +1,53 @@
+# 第六步:检查清单
+
+## 动笔前
+
+- [ ] 谁会读这页?他们要完成什么?读完应该能做什么或理解什么?
+- [ ] 页面类型判定了吗(Tutorial / How-to / Reference / Explanation / Troubleshooting)?
+- [ ] 是否已实际跑过要写的命令、API 或示例?
+- [ ] 声明的每个能力、字段、路径,是否对当前代码或 `docs/` 核实过?
+- [ ] 这个信息的权威来源是否已存在于别处?如果是,链接过去而不是复制。
+
+## 按类型
+
+**Tutorial / Quickstart**
+
+- [ ] 只有一条最短成功路径,没有分叉。
+- [ ] 每步有明确反馈,用户知道自己成功了。
+- [ ] 没有混入字段大全和概念长文。
+
+**How-to**
+
+- [ ] 一页只解决一个任务;标题匹配用户会搜索的词。
+- [ ] 步骤按"动作 → 命令 → 占位符 → 期望输出 → 失败时下一步"组织。
+- [ ] 有"验证结果"和"常见失败"小节。
+
+**Reference**
+
+- [ ] 字段、类型、默认值、必填性完整。
+- [ ] 没有未实现字段、roadmap 或叙事化教学。
+- [ ] 有短示例并链接到对应 guide。
+
+**Concept**
+
+- [ ] 写了"它不是什么",和相邻概念划清边界。
+- [ ] 没有变成操作步骤或内部设计流水账。
+
+**Troubleshooting**
+
+- [ ] 症状、原因、检查步骤、解决办法齐全。
+- [ ] 按用户可见症状组织,不按内部模块。
+
+## 发布前
+
+- [ ] 重要信息在页面前 1/3;第一屏能判断相关性。
+- [ ] 每个步骤一个动作;条件在动作前;可选步骤已标注。
+- [ ] 代码相关文本用反引号;链接文字有描述性。
+- [ ] 示例可复制运行,导入路径真实,输出已展示或说明。
+- [ ] 改动涉及的其它入口(README、示例、英文/中文对应页)已同步或明确留待同步。
+- [ ] 新增/重命名页面已更新 `docs.json`,旧链接加了 redirect。
+- [ ] 跑过 `docs:validate` 和 `docs:links`(命令见 `docs-site/AGENTS.md`)。
+
+## 只改 `docs-ref/` 内部参考时
+
+不进导航,不需要跑 Mintlify 校验,但至少检查:文件路径正确、相对链接可达、没有把内部草稿写成正式产品承诺。