niceeval 0.2.0 → 0.3.0

package/README.md

@@ -12,13 +12,17 @@
 
 </div>
 
-NiceEval is an agent-native eval tool inspired by [eve](https://eve.dev). It has an excellent DX design — anyone can get started and configured in about 10 minutes. It's also versatile: it can eval plugins, Hooks, and Skills written for Claude Code/Codex coding agents, and can directly eval your own AI agent application or framework (AI SDK, LangGraph, Pi, or any custom agent loop).
+NiceEval is an Agent-Native eval tool inspired by [eve](https://eve.dev), built for the best possible DX.
 
-After the eval completes, it generates readable reports and lets you view agent behavior details. Convenient for debugging and optimization.
+Thanks to its universal design, NiceEval can evaluate almost any agent application.
+Whether you need to evaluate coding agent plugins, Hooks, and Skills written for Claude Code / Codex, or your own AI agent application — both plug in easily.
+
+After the eval completes, it generates readable reports and lets you view agent behavior details. Convenient for debugging and understanding agent behavior.
 
 ## Why NiceEval when DeepEval, LangFuse, and BrainTrust already exist
 
-NiceEval is an AI-native eval tool. In tools built around Dataset/golden-style Input vs. Expected Output, that shape doesn't fit real agent evaluation well. NiceEval is built for evaluating agents at a finer grain — multi-turn conversations, multi-agent setups, tool calls, skill loading, and more.
+NiceEval is an Agent-Native eval tool. The Dataset / golden pattern of building an Input and an Expected Output doesn't fit real agent evaluation.
+Agents today need to be evaluated at a finer grain — multi-turn conversations, multi-agent collaboration, tool calls, skill loading — and NiceEval does this better.
 
 It also coexists with LangFuse and BrainTrust: use them for tracing, or upload eval results to both (in progress).
 
@@ -32,19 +36,19 @@ NiceEval supports two integration modes, depending on whether the agent under te
    evals/*.eval.ts
         │
         ▼
-   ┌─────────────────────┐
-   │     NiceEval        │
-   └─────────────────────┘
+   ┌────────────┐
+   │  NiceEval  │
+   └────────────┘
         │
         │ Agent adapter (official)
         ▼
-   ┌──────────────────────────────┐
-   │        Docker Sandbox         │
-   │   ┌────────────────────────┐  │
-   │   │ Codex / Claude Code /  │  │
-   │   │ apps needing isolation │  │
-   │   └────────────────────────┘  │
-   └──────────────────────────────┘
+   ┌────────────────────────────────┐
+   │         Docker Sandbox         │
+   │    ┌────────────────────────┐  │
+   │    │ Codex / Claude Code    │  │
+   │    │ apps needing isolation │  │
+   │    └────────────────────────┘  │
+   └────────────────────────────────┘
 ```
 
 **Mode 2: Direct — connect straight to your own AI Agent**
@@ -53,18 +57,16 @@ NiceEval supports two integration modes, depending on whether the agent under te
    evals/*.eval.ts
         │
         ▼
-   ┌─────────────────────┐
-   │     NiceEval        │
-   └─────────────────────┘
+   ┌────────────┐
+   │  NiceEval  │
+   └────────────┘
         │
         │ Agent adapter (official, or your own implementation)
         ▼
-   ┌──────────────────────────────┐
-   │       your own AI Agent       │
-   │   (AI SDK·LangGraph·Pi and    │
-   │    other agent frameworks —   │
-   │         no Docker needed)     │
-   └──────────────────────────────┘
+   ┌──────────────────────────┐
+   │    your own AI Agent     │
+   │   (AI SDK·LangGraph·Pi)  │
+   └──────────────────────────┘
 ```
 
 - **NiceEval core** owns discovery, scheduling, scoring, reporting, and artifacts.
@@ -74,8 +76,6 @@ NiceEval supports two integration modes, depending on whether the agent under te
 
 ## Example
 
-Running an eval takes two files: the eval itself (what to check) and an experiment (which agent to run it against). The CLI won't run a bare eval id — the experiment in `niceeval exp <experiment> <eval prefix>` is what picks the system under test. Here's a real eval against a directly-connected web agent (full project in [`examples/zh/ai-sdk/`](examples/zh/ai-sdk/)), checking that the agent calls a tool for live weather questions and answers from the tool result instead of making it up:
-
 ```ts
 // evals/eval-tool-call.eval.ts
 import { defineEval } from "niceeval";
@@ -109,16 +109,15 @@ import { webAgent } from "./adapter"; // your agent adapter, pointed at the syst
 
 export default defineExperiment({
   agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
+  model: "gpt-5.5"
 });
 ```
 
 ```sh
-npx niceeval exp local eval-tool-call  // run only eval-tool-call under the local experiment
-npx niceeval view
+pnpm exec niceeval exp local eval-tool-call  // run only eval-tool-call under the local experiment
+pnpm exec niceeval view // view eval results
 ```
 
-For coding agents that need an isolated workspace (Codex, Claude Code plugins/skills), see [`examples/zh/coding-agent-skill/`](examples/zh/coding-agent-skill/): evals there use `t.sandbox.uploadDirectory()` to seed the workspace, `t.fileChanged()` / `t.file()` to check what changed, and `t.sandbox.runCommand()` to run tests.
-
 ## Quick Start
 
 ```text
@@ -135,19 +134,20 @@ Start from the scenario that matches what you need to evaluate:
 ## Roadmap
 Official Adapters
 - [ ] Agent Software
-  - [ ] Claude Code
-  - [ ] Codex
-  - [ ] Bub
+  - [x] Claude Code
+  - [x] Codex
+  - [x] Bub
   - [ ] OpenClaw
   - [ ] Hermess Agent
   - [ ] Alma
   - [ ] ...
 
 - [ ] Agent Frameworks
-  - [ ] AI SDK
+  - [x] AI SDK
+  - [x] Claude SDK
+  - [x] Codex SDK
+  - [x] Pi Agent SDK
   - [ ] LangGraph
-  - [ ] Claude SDK
-  - [ ] Codex SDK
   - [ ] vm0
   - [ ] Cursor Agent SDK
 

package/README.zh.md

@@ -12,12 +12,16 @@
 
 </div>
 
-NiceEval 是一个受 [eve](https://eve.dev) 启发的 agent-native eval 工具，追求足够好的开发体验：大多数项目可以在 10 分钟左右接入并配置。它的设计也足够通用——既可以用来 eval 给 Claude Code / Codex 写的 coding agent 插件、Hook 和 Skill，也可以直接 eval 你自己的 AI Agent 应用或框架（无论基于 AI SDK、LangGraph、Pi 还是自研 agent loop，都能轻松接入）。
+NiceEval 是一个受 [eve](https://eve.dev) 启发的 Agent-Native eval 工具，追求极致的DX。
+
+通过通用式的设计，使 NiceEval 可以为几乎所有 Agent 应用进行评估。
+无论你是需要评估为 Claude Code / Codex 写的 coding agent 插件、Hook 和 Skill，还是可以评估你自己的 AI Agent 应用都能轻松接入。
 
 在 eval 完成之后可以生成易读的报告与查看 Agent 的行为细节。方便 Debug 与理解 Agent 行为。
 
 ## 为什么有了 DeepEval、LangFuse、BrainTrust 还需要 NiceEval
-NiceEval 是一个 AI-native 的评估工具。Dataset / golden 那一套「构建 Input 与 Expected Output」的模式，并不适合真实的 agent 评估；当 agent 需要在多轮对话、多 agent 协作、工具调用、Skill 加载等细粒度场景下被评估时，NiceEval 能做得更好。
+NiceEval 是一个 Agent-Native 的评估工具。Dataset / golden 那一套「构建 Input 与 Expected Output」的模式，并不适合真实的 Agent 评估。
+现在 Agent 需要在多轮对话、多 agent 协作、工具调用、Skill 加载等细粒度场景下进行评估，NiceEval 能做得更好。
 
 同时，NiceEval 也能与 LangFuse、BrainTrust 共存：可以用它们做 tracing，或者把评估结果上传到两者（这部分功能还在开发中）。
 
@@ -40,8 +44,8 @@ NiceEval 支持两种接入方式，取决于被测 agent 是否需要隔离的
    ┌──────────────────────────────┐
    │        Docker Sandbox        │
    │   ┌────────────────────────┐ │
-   │   │ Codex / Claude Code /  │ │
-   │   │ 需要隔离文件系统的应用   │ │
+   │   │ Codex / Claude Code    │ │
+   │   │ 需要隔离文件系统的应用     │ │
    │   └────────────────────────┘ │
    └──────────────────────────────┘
 ```
@@ -59,9 +63,8 @@ NiceEval 支持两种接入方式，取决于被测 agent 是否需要隔离的
         │ Agent 适配器(官方，或者自己实现)
         ▼
    ┌──────────────────────────────┐
-   │       你自己的 AI Agent      │
-   │   (AI SDK·LangGraph·Pi 等     │
-   │     agent 框架，无需 Docker)  │
+   │       你自己的 AI Agent        │
+   │   (AI SDK·LangGraph·Pi )     │ 
    └──────────────────────────────┘
 ```
 
@@ -72,8 +75,6 @@ NiceEval 支持两种接入方式，取决于被测 agent 是否需要隔离的
 
 ## 示例
 
-跑一条 eval 需要两个文件：eval 本身（测什么）和 experiment（跑哪个 agent）。CLI 不接受裸 eval id——`niceeval exp <experiment> <eval 前缀>` 里的 experiment 才是决定「连哪个被测对象」的地方。下面是一个直连 AI Agent 的真实场景（完整项目见 [`examples/zh/ai-sdk/`](examples/zh/ai-sdk/)），验证 agent 遇到实时天气问题时会调用工具、并基于工具结果作答，而不是凭空编造：
-
 ```ts
 // evals/eval-tool-call.eval.ts
 import { defineEval } from "niceeval";
@@ -107,6 +108,7 @@ import { webAgent } from "./adapter"; // 你自己写的 agent adapter，接被
 
 export default defineExperiment({
   agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
+  model: "gpt-5.5"
 });
 ```
 
@@ -131,19 +133,20 @@ READ https://niceeval.com/INIT.md and install niceeval for this repo.
 ## Roadmap
 官方适配器
 - [ ] Agent 软件
-  - [ ] Claude Code
-  - [ ] Codex
-  - [ ] Bub
+  - [x] Claude Code
+  - [x] Codex
+  - [x] Bub
   - [ ] OpenClaw
   - [ ] Hermess Agent
   - [ ] Alma
   - [ ] ...
 
 - [ ] Agent 框架
-  - [ ] AI SDK
+  - [x] AI SDK
+  - [x] Claude SDK
+  - [x] Codex SDK
+  - [x] Pi Agent SDK
   - [ ] LangGraph
-  - [ ] Claude SDK
-  - [ ] Codex SDK
   - [ ] vm0
   - [ ] Cursor Agent SDK
 

package/docs/architecture.md

@@ -7,26 +7,26 @@ niceeval 把一个评测过程拆成四段职责:**发现**要跑什么、**驱
 ## 系统总览
 
 ```text
-                          ┌──────────────────────────── core ────────────────────────────┐
-  evals/                  │                                                               │
-  └─ *.eval.ts ──────────►│  Discovery ──► Runner ──► Scoring ──► Outcome ──► Reporters    │
-                          │   (发现)       (调度)     (评分)      (判决)      (报告)         │
-                          │                  │ 驱动                              │          │
-                          └──────────────────┼──────────────────────────────────┼──────────┘
-                                             │                                  ▼
-                                             ▼  对接口分发,不按名字分支    .niceeval/<run>/
-                          ┌──────────── Agent(自实现 Adapter)────────────┐
-                          │        远程 adapter          沙箱 adapter      │
-                          │        (你的服务)          ┌─────┴─────┐      │
-                          │                            │ claude-code│      │
-                          │                            │ codex / bub│      │
-                          │                            └─────┬──────┘     │
-                          │                                  ▼            │
-                          │                            ┌── Sandbox ──┐    │
-                          │                            │ docker      │    │
-                          │                            │ vercel / 三方│    │
-                          │                            └─────────────┘    │
-                          └────────────────────────────────────────────────┘
+                          ┌─────────────────────────── core ───────────────────────────┐
+  evals/                  │                                                            │
+  └─ *.eval.ts ──────────►│ Discovery ──► Runner ──► Scoring ──► Outcome ──► Reporters │
+                          │  (发现)       (调度)     (评分)      (判决)       (报告)   │
+                          │              │ 驱动                             │          │
+                          └──────────────┼──────────────────────────────────┼──────────┘
+                                         │                                  ▼
+                                         ▼  对接口分发,不按名字分支    .niceeval/<run>/
+                          ┌────── Agent(自实现 Adapter) ──────┐
+                          │   远程 adapter    沙箱 adapter    │
+                          │    (你的服务)   ┌───────────────┐ │
+                          │                 │  claude-code  │ │
+                          │                 │  codex / bub  │ │
+                          │                 └───────────────┘ │
+                          │                         ▼         │
+                          │                 ┌─── Sandbox ───┐ │
+                          │                 │    docker     │ │
+                          │                 │ vercel / 三方 │ │
+                          │                 └───────────────┘ │
+                          └───────────────────────────────────┘
 ```
 
 四段职责是**单向数据流**:发现产出一批 `Eval`,运行器逐个对 Agent `send` 得到 `Turn`,评分器把 `Turn` 折叠成 `Assertion[]`,再把全部断言折叠成一个互斥的 `Outcome`;报告器消费 `Outcome` + 工件。没有反向耦合 —— 评分器不知道 agent 的 transport 是 HTTP 还是沙箱 CLI,它只看 `Turn`。

package/docs/view.md

@@ -6,10 +6,13 @@
 niceeval view                         # 起本地 web,自动打开浏览器,读 .niceeval/ 下所有历史运行
 niceeval view .niceeval/<run>/summary.json
 niceeval view --no-open               # 只打印 URL,不打开浏览器
-niceeval view --out .niceeval/report.html  # 导出静态 HTML
+niceeval view --out .niceeval/report.html  # 导出单文件静态 HTML
+niceeval view --out site                   # 目录式静态导出:index.html + artifact/
 ```
 
-架构上是**一次性烘焙进单个 HTML+JSON 的静态产物**(`src/view/index.ts` 的 `renderHtml`),不是常驻的多页面 server——`niceeval view` 起的 web 服务每次请求现读现渲染,`--out` 则直接导出成一个可以当 CI 附件传、单文件分享的 HTML。这是刻意的取舍,详见 [References](references.md#调研过判断不值得抄的及理由)。
+架构上是**一次性烘焙进单个 HTML+JSON 的静态产物**(`src/view/index.ts` 的 `renderHtml`),不是常驻的多页面 server——`niceeval view` 起的 web 服务每次请求现读现渲染,`--out` 则直接导出。这是刻意的取舍,详见 [References](references.md#调研过判断不值得抄的及理由)。
+
+`--out` 按目标形态分两种(`isSingleFileOut`):`*.html` 导出单文件,可当 CI 附件传、单文件分享,但工件视图(代码/transcript/trace)降级;目录则额外把前端会 fetch 的三类工件(`sources.json` / `events.json` / `trace.json`)复制到 `<out>/artifact/<base>/`,与本地 server 的 `/artifact/<rel>` 路径路由同一布局,同一份前端产物在两种托管下用同一个相对 URL(`src/view/app/lib/artifact-url.ts`)。`diff.json` / `o11y.json` 刻意不复制:查看器从不读取,且 diff 可达上百 MB,带上只会拖垮静态部署体积。
 
 ## 结果版本机制
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "niceeval",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Agent-native eval tool — eval agents, services, functions, and coding-agent fixtures",
   "type": "module",
   "license": "MIT",
@@ -56,13 +56,11 @@
     "tar-stream": "^3.1.7",
     "tsx": "^4.19.2"
   },
-  "optionalDependencies": {
-    "@vercel/sandbox": "^2.2.1",
-    "dockerode": "^4.0.2",
-    "e2b": "^2.31.0"
-  },
   "devDependencies": {
     "@ai-sdk/otel": "^1.0.9",
+    "@vercel/sandbox": "^2.2.1",
+    "dockerode": "^4.0.2",
+    "e2b": "^2.31.0",
     "@opentelemetry/exporter-trace-otlp-http": "^0.219.0",
     "@opentelemetry/sdk-trace-node": "^2.8.0",
     "@radix-ui/react-collapsible": "^1.1.14",
@@ -95,12 +93,18 @@
     "@ai-sdk/otel": ">=1.0.0",
     "@opentelemetry/exporter-trace-otlp-http": ">=0.200.0",
     "@opentelemetry/sdk-trace-node": ">=2.0.0",
-    "ai": ">=5.0.0"
+    "@vercel/sandbox": ">=2.0.0",
+    "ai": ">=5.0.0",
+    "dockerode": ">=4.0.0",
+    "e2b": ">=2.0.0"
   },
   "peerDependenciesMeta": {
     "@ai-sdk/otel": {
       "optional": true
     },
+    "@vercel/sandbox": {
+      "optional": true
+    },
     "ai": {
       "optional": true
     },
@@ -109,6 +113,12 @@
     },
     "@opentelemetry/sdk-trace-node": {
       "optional": true
+    },
+    "dockerode": {
+      "optional": true
+    },
+    "e2b": {
+      "optional": true
     }
   },
   "scripts": {

package/src/cli.ts

@@ -17,7 +17,7 @@ import { Console as ConsoleReporter } from "./runner/reporters/console.ts";
 import { JUnit } from "./runner/reporters/json.ts";
 import { Live as LiveReporter, type LiveRow } from "./runner/reporters/live.ts";
 import { Artifacts as ArtifactsReporter } from "./runner/reporters/artifacts.ts";
-import { buildView, startViewServer, loadMostRecentResults, IncompatibleResultsError } from "./view/index.ts";
+import { buildView, isSingleFileOut, startViewServer, loadMostRecentResults, IncompatibleResultsError } from "./view/index.ts";
 import { t } from "./i18n/index.ts";
 import { formatThrown } from "./util.ts";
 import type { Config, DiscoveredExperiment, Reporter } from "./types.ts";
@@ -248,7 +248,7 @@ async function main(): Promise<void> {
   if (command === "view") {
     if (flags.out) {
       const out = await buildView({ input: positionals[0], out: flags.out }).catch(exitOnIncompatibleResults);
-      process.stdout.write(t("cli.view.exported", { out }));
+      process.stdout.write(t(isSingleFileOut(out) ? "cli.view.exported" : "cli.view.exportedDir", { out }));
       process.exit(0);
     }
     const server = await startViewServer({ input: positionals[0], port: flags.port }).catch(exitOnIncompatibleResults);

package/src/i18n/en.ts

@@ -35,7 +35,9 @@ export const en = {
     "Usage:\n" +
     "  niceeval exp [group|experiment] [eval-id-prefix…]   run experiments\n" +
     "  niceeval list                                       list discovered evals\n" +
-    "  niceeval view [summary.json|dir] [--out d] [--port n] [--no-open]\n" +
+    "  niceeval view [summary.json|dir] [--out x.html|dir] [--port n] [--no-open]\n" +
+    "      --out *.html exports a single-file report; --out <dir> also bundles\n" +
+    "      artifacts so the directory can be served by any static host\n" +
     "  niceeval clean                                      delete .niceeval/ artifacts\n" +
     "  niceeval init                                       scaffold config + evals/\n\n" +
     "Flags:\n" +
@@ -65,6 +67,7 @@ export const en = {
   "cli.sandboxFlagRemoved": "`--sandbox` is not a CLI flag. Set `sandbox` in the experiment (or `niceeval.config.ts` as a project-wide fallback) to dockerSandbox() / vercelSandbox() / e2bSandbox() (import from \"niceeval/sandbox\").\n",
   "cli.unimplemented": "Command \"{{command}}\" is not implemented yet (MVP).\n",
   "cli.view.exported": "Exported eval report page: {{out}}\n",
+  "cli.view.exportedDir": "Exported static report site: {{out}} (serve the whole directory with any static host; opening index.html via file:// cannot fetch artifacts)\n",
   "cli.view.incompatible": "{{dir}}: written by niceeval {{producer}} (schemaVersion {{schemaVersion}}); this CLI reads schemaVersion {{supported}}.\nRun `{{command}}` to view it.\n",
   "cli.view.url": "niceeval view: {{url}}\n",
   "context.capabilityMissing":
@@ -97,8 +100,9 @@ export const en = {
   "judge.httpError": "judge HTTP {{status}}: {{body}}",
   "judge.probeFailed": "judge precheck failed ({{model}}): {{error}}",
   "judge.probeMissingKey": "judge model {{model}} is missing an API key; configure {{envHint}}",
-  "live.running": "  Running {{totalRuns}} attempts ({{evals}} evals × {{configs}} configs)       {{completed}}/{{total}} done",
+  "live.running": "  Running {{totalRuns}} attempts ({{evals}} evals × {{configs}} configs, concurrency {{concurrency}})       {{completed}}/{{total}} done",
   "live.runningUnknown": "  Running...  {{completed}}/{{total}} done",
+  "live.waiting": "waiting for a slot...",
   "report.assertionThreshold": " (got {{score}} < {{threshold}})",
   "report.error": "error",
   "report.errored": "errored",
@@ -106,8 +110,9 @@ export const en = {
   "report.gate": "gate",
   "report.passed": "passed",
   "report.result": "\nResult: {{parts}}  ({{duration}} · {{tokens}}{{cost}})\n\n",
-  "report.runStart": "\nRunning {{count}} evals{{extra}}\n\n",
+  "report.runStart": "\nRunning {{count}} evals{{extra}} (concurrency {{concurrency}})\n\n",
   "report.runStartExtra": " × {{configs}} configs = {{totalRuns}} runs",
+  "report.viewHint": "Run `pnpm exec niceeval view` to see the results in the graphical viewer.\n",
   "report.skipped": "skipped",
   "report.soft": "soft",
   "report.summary.errored": "{{count}} errored",

package/src/i18n/zh-CN.ts

@@ -33,7 +33,8 @@ export const zhCN = {
     "用法:\n" +
     "  niceeval exp [组|实验] [eval-id 前缀…]   跑实验\n" +
     "  niceeval list                            列出发现到的 eval\n" +
-    "  niceeval view [summary.json|目录] [--out 目录] [--port n] [--no-open]\n" +
+    "  niceeval view [summary.json|目录] [--out x.html|目录] [--port n] [--no-open]\n" +
+    "      --out 给 *.html 导出单文件报告;给目录则连工件一起导出,可直接静态托管\n" +
     "  niceeval clean                           删除 .niceeval/ 历史工件\n" +
     "  niceeval init                            脚手架 config + evals/\n\n" +
     "标志:\n" +
@@ -63,6 +64,7 @@ export const zhCN = {
   "cli.sandboxFlagRemoved": "`--sandbox` 不是 CLI flag。请在 experiment(或 niceeval.config.ts 做全项目兜底)里把 sandbox 设成 dockerSandbox() / vercelSandbox() / e2bSandbox()(从 \"niceeval/sandbox\" 导入)。\n",
   "cli.unimplemented": "命令 \"{{command}}\" 暂未实现(MVP)。\n",
   "cli.view.exported": "已导出实验查看页:{{out}}\n",
+  "cli.view.exportedDir": "已导出静态查看站:{{out}}(整个目录可直接静态托管;本地打开 {{out}}/index.html 需经 http 服务,file:// 下工件 fetch 不可用)\n",
   "cli.view.incompatible": "{{dir}}: 由 niceeval {{producer}} 写入(schemaVersion {{schemaVersion}}),当前 CLI 只读 schemaVersion {{supported}}。\n运行 `{{command}}` 查看这份报告。\n",
   "cli.view.url": "niceeval view: {{url}}\n",
   "context.capabilityMissing":
@@ -95,8 +97,9 @@ export const zhCN = {
   "judge.httpError": "judge HTTP {{status}}: {{body}}",
   "judge.probeFailed": "judge 预检失败({{model}}): {{error}}",
   "judge.probeMissingKey": "judge 模型 {{model}} 缺少 API key —— 请配置 {{envHint}}",
-  "live.running": "  正在运行 {{totalRuns}} 次 ({{evals}} eval × {{configs}} 配置)       {{completed}}/{{total}} 完成",
+  "live.running": "  正在运行 {{totalRuns}} 次 ({{evals}} eval × {{configs}} 配置, 并发 {{concurrency}})       {{completed}}/{{total}} 完成",
   "live.runningUnknown": "  正在运行…  {{completed}}/{{total}} 完成",
+  "live.waiting": "排队等待中…",
   "report.assertionThreshold": " (得分 {{score}} < {{threshold}})",
   "report.error": "错误",
   "report.errored": "错误",
@@ -104,8 +107,9 @@ export const zhCN = {
   "report.gate": "gate",
   "report.passed": "通过",
   "report.result": "\n结果:{{parts}}  ({{duration}} · {{tokens}}{{cost}})\n\n",
-  "report.runStart": "\n本次运行 {{count}} 个 eval{{extra}}\n\n",
+  "report.runStart": "\n本次运行 {{count}} 个 eval{{extra}}(并发 {{concurrency}})\n\n",
   "report.runStartExtra": " × {{configs}} 配置 = {{totalRuns}} 次运行",
+  "report.viewHint": "运行 `pnpm exec niceeval view` 以图形化查看结果。\n",
   "report.skipped": "跳过",
   "report.soft": "soft",
   "report.summary.errored": "{{count}} 错误",

package/src/runner/reporters/console.ts

@@ -2,7 +2,7 @@
 
 import type { EvalResult, Reporter, RunSummary } from "../../types.ts";
 import { t } from "../../i18n/index.ts";
-import { formatDuration, formatTokens, renderRunReport } from "./table.ts";
+import { formatCost, formatDuration, formatTokens, renderRunReport } from "./table.ts";
 import { outcomeSymbol } from "./shared.ts";
 
 export function Console(): Reporter {
@@ -15,14 +15,14 @@ export function Console(): Reporter {
         shape && shape.totalRuns > n
           ? t("report.runStartExtra", { configs: shape.configs, totalRuns: shape.totalRuns })
           : "";
-      process.stdout.write(t("report.runStart", { count: n, extra }));
+      process.stdout.write(t("report.runStart", { count: n, extra, concurrency: shape?.maxConcurrency ?? "?" }));
     },
     onEvalComplete(result: EvalResult) {
       const sym = outcomeSymbol(result.outcome);
       const tok = (result.usage?.inputTokens ?? 0) + (result.usage?.outputTokens ?? 0);
       // requests > 0 但 tokens = 0 → agent 跑了但不上报用量(如 bub);显示 — 而非误导性的 0
       const tokStr = tok > 0 ? `${formatTokens(tok)} tok` : (result.usage?.requests ?? 0) > 0 ? `— tok` : `0 tok`;
-      const cost = result.estimatedCostUSD !== undefined ? `  $${result.estimatedCostUSD.toFixed(3)}` : "";
+      const cost = result.estimatedCostUSD !== undefined ? `  ${formatCost(result.estimatedCostUSD)}` : "";
       const who = result.model ? `${result.agent}/${result.model}` : result.agent;
       const meta = `(${formatDuration(result.durationMs)}  ${tokStr}${cost})`;
       const label = result.outcome === "passed" ? "" : ` ${formatOutcome(result.outcome)}`;

package/src/runner/reporters/live.ts

@@ -2,10 +2,10 @@
 // spinner 每 80ms 刷新;attempt 完成后行内显示 ✓/✗/~ 符号。
 // onRunComplete 时清除状态表,打印和网页榜单同口径的表格报告。
 
-import type { Reporter, RunShape, RunSummary } from "../../types.ts";
+import type { Reporter, ReporterEvent, RunShape, RunSummary } from "../../types.ts";
 import { t } from "../../i18n/index.ts";
 import { renderRunReport } from "./table.ts";
-import { outcomeSymbol } from "./shared.ts";
+import { outcomeSymbol, WAITING_SYM } from "./shared.ts";
 
 const SPINNER = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
 
@@ -42,6 +42,8 @@ interface RowState extends LiveRow {
   completed: number;
   lastMsg: string;
   dominantOutcome: string | undefined;
+  /** true 一旦拿到并发名额、attempt effect 真正开始跑;之前是排队等待,不该转圈误导。 */
+  started: boolean;
 }
 
 export function Live(rows: LiveRow[], totalAttempts: number): LiveReporter {
@@ -51,7 +53,7 @@ export function Live(rows: LiveRow[], totalAttempts: number): LiveReporter {
   for (const r of rows) {
     const k = `${r.evalId}|${r.who}`;
     if (!stateMap.has(k)) {
-      stateMap.set(k, { ...r, completed: 0, lastMsg: "", dominantOutcome: undefined });
+      stateMap.set(k, { ...r, completed: 0, lastMsg: "", dominantOutcome: undefined, started: false });
       keyOrder.push(k);
     } else {
       // 同一 (evalId, who) 可能在多个 agentRun 里出现(不应发生,但做防御)
@@ -71,7 +73,9 @@ export function Live(rows: LiveRow[], totalAttempts: number): LiveReporter {
     const done = state.completed >= state.total;
     const sym = done
       ? outcomeSymbol(state.dominantOutcome ?? "")
-      : SPINNER[frame % SPINNER.length];
+      : state.started
+        ? SPINNER[frame % SPINNER.length]
+        : WAITING_SYM;
 
     const evalCol = state.evalId.slice(0, 24).padEnd(24);
     const whoCol = `[${state.who}]`.slice(0, 26).padEnd(26);
@@ -79,7 +83,7 @@ export function Live(rows: LiveRow[], totalAttempts: number): LiveReporter {
 
     const prefix = `  ${sym} ${evalCol} ${whoCol} ${cntCol}  `;
     const budget = Math.max(0, cols() - prefix.length - 1);
-    const msg = done ? "" : state.lastMsg.slice(0, budget);
+    const msg = done ? "" : state.started ? state.lastMsg.slice(0, budget) : t("live.waiting").slice(0, budget);
 
     return `\x1B[2K${prefix}${msg}`;
   }
@@ -90,6 +94,7 @@ export function Live(rows: LiveRow[], totalAttempts: number): LiveReporter {
           totalRuns: shape.totalRuns,
           evals: shape.evals,
           configs: shape.configs,
+          concurrency: shape.maxConcurrency,
           completed: totalCompleted,
           total: totalAttempts,
         })
@@ -132,6 +137,13 @@ export function Live(rows: LiveRow[], totalAttempts: number): LiveReporter {
       // 不在这里 draw();由 interval 驱动,避免每条日志都刷屏
     },
 
+    onEvent(event: ReporterEvent) {
+      if (event.type !== "eval:start") return;
+      const who = event.model ? `${event.agent.name}/${event.model}` : event.agent.name;
+      const state = stateMap.get(`${event.eval.id}|${who}`);
+      if (state) state.started = true;
+    },
+
     onRunStart(_evals, _agent, s) {
       shape = s;
       // 初始渲染:让用户看到行表

package/src/runner/reporters/shared.ts

@@ -13,3 +13,6 @@ export const OUTCOME_SYM: Record<ResultOutcome, string> = {
 export function outcomeSymbol(outcome: string): string {
   return OUTCOME_SYM[outcome as ResultOutcome] ?? "?";
 }
+
+/** live 表格里「还没抢到并发名额」的行:和转圈的 SPINNER、完成后的 OUTCOME_SYM 三态区分开。 */
+export const WAITING_SYM = "·";

package/src/runner/reporters/table.ts

@@ -104,6 +104,7 @@ export function renderRunReport(summary: RunSummary): string {
     tokens: tokStr,
     cost,
   }).trimEnd());
+  lines.push(t("report.viewHint").trimEnd());
 
   return `${lines.join("\n")}\n\n`;
 }

package/src/runner/run.ts

@@ -113,6 +113,7 @@ export async function runEvals(opts: RunOptions): Promise<RunSummary> {
     evals: runningEvals.length,
     configs: opts.agentRuns.length,
     totalRuns: attempts.length,
+    maxConcurrency: opts.maxConcurrency,
   };
   for (const r of opts.reporters) {
     // reporter 只是结果消费方:单个 reporter 抛错记 diagnostic,不能让整次调度崩(P2)。
@@ -259,6 +260,7 @@ export async function runEvals(opts: RunOptions): Promise<RunSummary> {
                 type: "eval:start",
                 eval: { id: a.evalDef.id },
                 agent: a.run.agent,
+                model: a.run.model,
                 attempt: a.attempt,
                 experimentId: a.run.experimentId,
               }),

package/src/runner/types.ts

@@ -97,6 +97,8 @@ export interface RunShape {
   configs: number;
   /** 总 attempt 数(evals × configs × runs);逐行输出与汇总计数都按它。 */
   totalRuns: number;
+  /** 本次运行实际生效的并发数(flag/env/experiment/config/sandbox 默认值解析后的结果)。 */
+  maxConcurrency: number;
 }
 
 export interface Reporter {
@@ -108,7 +110,7 @@ export interface Reporter {
 
 export type ReporterEvent =
   | { type: "run:start"; evals: { id: string }[]; agent: Agent; shape: RunShape }
-  | { type: "eval:start"; eval: { id: string }; agent: Agent; attempt: number; experimentId?: string }
+  | { type: "eval:start"; eval: { id: string }; agent: Agent; model?: string; attempt: number; experimentId?: string }
   | { type: "eval:complete"; result: EvalResult }
   | { type: "run:earlyExit"; evalId: string; experimentId?: string }
   | { type: "run:budgetExceeded"; budget: number; spent: number }

package/src/shared/format.ts

@@ -15,6 +15,12 @@ export function formatDuration(ms?: number): string {
 }
 
 export function formatCost(value?: number): string {
-  if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) return "$0";
-  return "$" + value.toFixed(value < 1 ? 3 : 2);
+  // 查不到价 / 没有 model 时上游传 undefined(见 o11y/cost.ts) —— 显示 "—" 而不是騙人的 $0,
+  // 否则「真实但极小的花费」和「压根没算出成本」在界面上长得一模一样。
+  if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) return "—";
+  if (value >= 1) return "$" + value.toFixed(2);
+  if (value >= 0.001) return "$" + value.toFixed(3);
+  // 便宜模型 + 小样本(如 tier1 示例)常见的真实成本,3 位小数会整个舍成 0.000,
+  // 这里退到 2 位有效数字,保留"确实花了一点点钱"的信号。
+  return "$" + value.toPrecision(2);
 }