niceeval 0.7.1 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/dist/report/components.d.ts +4 -5
  2. package/dist/report/components.js +7 -7
  3. package/dist/report/compute.d.ts +4 -4
  4. package/dist/report/compute.js +8 -12
  5. package/dist/report/index.d.ts +2 -2
  6. package/dist/report/report.d.ts +23 -1
  7. package/dist/report/report.js +82 -5
  8. package/dist/report/types.d.ts +0 -10
  9. package/dist/results/types.d.ts +0 -11
  10. package/docs-site/zh/how-to/custom-reports.mdx +3 -3
  11. package/docs-site/zh/how-to/publish-report.mdx +43 -9
  12. package/docs-site/zh/how-to/viewing-results.mdx +4 -4
  13. package/docs-site/zh/reference/cli.mdx +2 -3
  14. package/docs-site/zh/reference/report-components.mdx +2 -2
  15. package/docs-site/zh/reference/results-data.mdx +2 -4
  16. package/docs-site/zh/troubleshooting/debugging.mdx +2 -2
  17. package/package.json +7 -6
  18. package/src/cli.ts +1 -5
  19. package/src/context/context.ts +11 -5
  20. package/src/report/components.tsx +13 -15
  21. package/src/report/compute.ts +8 -20
  22. package/src/report/index.ts +1 -1
  23. package/src/report/report.test.ts +2 -20
  24. package/src/report/report.ts +128 -6
  25. package/src/report/shell-head.test.ts +102 -0
  26. package/src/report/types.ts +0 -11
  27. package/src/results/copy.ts +15 -78
  28. package/src/results/publish.ts +4 -146
  29. package/src/results/results.test.ts +8 -8
  30. package/src/results/types.ts +0 -7
  31. package/src/show/report-host.ts +13 -0
  32. package/src/view/app/components/CodeView.test.tsx +142 -0
  33. package/src/view/app/components/CodeView.tsx +15 -1
  34. package/src/view/app/components/Transcript.tsx +28 -1
  35. package/src/view/app/i18n.ts +6 -0
  36. package/src/view/app/lib/guards.test.ts +108 -0
  37. package/src/view/app/lib/guards.ts +13 -3
  38. package/src/view/app/lib/transcript-data.tsx +14 -0
  39. package/src/view/app/types.ts +17 -1
  40. package/src/view/artifact-serving.test.ts +1 -1
  41. package/src/view/client-dist/app.css +1 -1
  42. package/src/view/client-dist/app.js +20 -20
  43. package/src/view/data.ts +54 -14
  44. package/src/view/index.ts +18 -58
  45. package/src/view/server.ts +49 -144
  46. package/src/view/site-head.test.ts +177 -0
  47. package/src/view/site-parity.test.ts +117 -0
  48. package/src/view/site.ts +209 -0
  49. package/src/view/styles.css +10 -0
  50. package/src/view/view-report.test.ts +6 -6
@@ -211,7 +211,7 @@ inspect: niceeval show @<id> [--eval|--execution|--diff]
211
211
  每项固定代表一个 Attempt,显示 experiment、Eval、Attempt 序号、判定、耗时、成本、失败断言、结构化 error 的一层摘要、Judge 评语和证据链接。diagnostics、cause 和 stack 留给 locator 下钻详情,避免比较列表被基础设施日志撑开。它既能列失败证据,也能列通过样本,不把 verdict 过滤写死在组件名里。
212
212
 
213
213
  ```tsx
214
- const attempts = await AttemptList.data(selection, { redact });
214
+ const attempts = await AttemptList.data(selection);
215
215
 
216
216
  <AttemptList
217
217
  items={attempts.filter((item) => item.verdict === "failed" || item.verdict === "errored")}
@@ -234,7 +234,7 @@ inspect: niceeval show @<id> [--eval|--execution|--diff]
234
234
  (3 more not shown · showing 20 of 23)
235
235
  ```
236
236
 
237
- 要在页面显示前遮蔽 error message/cause/stack、diagnostic message/data、断言 detail 或 Judge 评语,把 `redact` 交给 `.data()`;稳定 code、lifecycle operation、experiment、Eval 和 locator 不改。它只影响这份组件数据,管不到发布目录里的 artifact 文件——发布数据集的消毒用 [`copySnapshots` 的 `redact` 选项](/zh/reference/results-data)。要展示哪些 Attempt,过滤返回的 `AttemptListItem[]`。`limit` 也由报告作者在数组上用 `.slice(0, 20)` 表达,截断时把原始数量交给组件的 `total`,组件据此显示“还有 n 项未展示”,不静默截断。
237
+ 要展示哪些 Attempt,过滤返回的 `AttemptListItem[]`。`limit` 也由报告作者在数组上用 `.slice(0, 20)` 表达,截断时把原始数量交给组件的 `total`,组件据此显示“还有 n 项未展示”,不静默截断。
238
238
 
239
239
  ## 指标表(`MetricTable`)
240
240
 
@@ -235,9 +235,7 @@ import { openResults, copySnapshots } from "niceeval/results";
235
235
  const results = await openResults(".niceeval");
236
236
  await copySnapshots(results.latest(), "site-data/run", {
237
237
  artifacts: ["sources", "events", "trace", "o11y"], // diff 不截断,缺省也不带;
238
- redact: (text) => text.replaceAll(/sk-[A-Za-z0-9]+/g, "[redacted]"),
239
- }); // redact 必填:函数消毒,或 false 显式声明原文发布;
240
- // 每个待发布文件还会经过 50 MiB 预检;
238
+ }); // 每个待发布文件还会经过 50 MiB 预检;
241
239
  // o11y 只有几 KB,报告用到 turns 这类
242
240
  // 读 o11y 的指标就把它带上,不然渲染成「—」
243
241
  ```
@@ -246,7 +244,7 @@ await copySnapshots(results.latest(), "site-data/run", {
246
244
 
247
245
  复制开始前,NiceEval 会规划全部目标文件并检查序列化后的大小。任一文件超过固定的 50 MiB,整次复制在创建目标目录前失败,错误会列出路径、实际大小和处理建议。你可以从 `artifacts` 排除那类证据;如果是旧版本留下的超大 events / trace,用当前版本重跑后再发布。这个检查既覆盖没有逐值截断的源码 / diff,也覆盖单值都正常但累计过大的 JSON,避免直到 `git push` 才撞上 Git host 的单文件限制。
248
246
 
249
- 大小预检只决定整次复制成功或失败,不会从一个超大文件中间删内容。消毒不是可选项——`copySnapshots` 要求显式传 `redact`:给一个函数就改写复制出来的所有文件里的自由文本(events、trace、源码、diff、运行摘要都在内;id、事件类型这类标识字段不动),确定这批数据可以原文公开就传 `redact: false`,两个都不传会直接报错。注意报告积木 `AttemptList.data` 的 `redact` 只影响页面上显示的数据,管不到发布目录里的 artifact 文件;发布场景一律在 `copySnapshots` 这一步消毒。唯一随行补记的是挑选时的**覆盖事实**:`partial-coverage` 警告的分母是实验的历史并集,而发布目录没有历史——所以每个复制出的快照带上 `knownEvalIds`(复制时刻该实验已知的 eval 并集),reader 端把它并进 `exp.evalIds` 的计算(取本地历史与快照携带值的并集)。发布目录上重新 `openResults().latest()`,残缺警告被同一套机制重新算出来,不靠发布者转述。复制出的目录就是标准结果目录,`niceeval view --run <目录>` 直接能看;要让报告站随 push 自动更新,workflow 见[通过 CI 发布报告](/zh/how-to/publish-report)。
247
+ 大小预检只决定整次复制成功或失败,不会从一个超大文件中间删内容。复制忠实于源:artifact 按原字节复制,不重新序列化、不改写。唯一随行补记的是挑选时的**覆盖事实**:`partial-coverage` 警告的分母是实验的历史并集,而发布目录没有历史——所以每个复制出的快照带上 `knownEvalIds`(复制时刻该实验已知的 eval 并集),reader 端把它并进 `exp.evalIds` 的计算(取本地历史与快照携带值的并集)。发布目录上重新 `openResults().latest()`,残缺警告被同一套机制重新算出来,不靠发布者转述。复制出的目录就是标准结果目录,`niceeval view --results <目录>` 直接能看;要让报告站随 push 自动更新,workflow 见[通过 CI 发布报告](/zh/how-to/publish-report)。
250
248
 
251
249
  ## 分层速览
252
250
 
@@ -190,8 +190,8 @@ niceeval view
190
190
  **打开归档或别人发来的结果。** 结果目录是自包含的——从 CI 下载的、同事拷给你的、发布到静态站前生成的目录,都能直接指过去:
191
191
 
192
192
  ```bash
193
- niceeval show --run tmp/ci-artifacts/results
194
- niceeval view --run site-data/run
193
+ niceeval show --results tmp/ci-artifacts/results
194
+ niceeval view --results site-data/run
195
195
  ```
196
196
 
197
197
  一个注意点:如果本地清理过旧快照目录,之后的运行里「沿用上次结果」的条目会找不到原始证据(显示为缺失)。要长期归档某次运行,先用 [`copySnapshots`](/zh/reference/results-data) 复制出一份再删。
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "niceeval",
3
- "version": "0.7.1",
3
+ "version": "0.8.1",
4
4
  "description": "Agent-native eval tool — eval agents, services, functions, and coding-agent fixtures",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -83,10 +83,6 @@
83
83
  },
84
84
  "devDependencies": {
85
85
  "@ai-sdk/otel": "^1.0.9",
86
- "@vercel/sandbox": "^2.2.1",
87
- "braintrust": "^3.20.0",
88
- "dockerode": "^4.0.2",
89
- "e2b": "^2.31.0",
90
86
  "@opentelemetry/exporter-trace-otlp-http": "^0.219.0",
91
87
  "@opentelemetry/sdk-trace-node": "^2.8.0",
92
88
  "@radix-ui/react-collapsible": "^1.1.14",
@@ -98,9 +94,15 @@
98
94
  "@types/react": "^19.2.17",
99
95
  "@types/react-dom": "^19.2.3",
100
96
  "@types/tar-stream": "^3.1.3",
97
+ "@typescript/native": "npm:typescript@^7.0.2",
98
+ "@vercel/sandbox": "^2.2.1",
101
99
  "@vitejs/plugin-react": "^6.0.3",
100
+ "braintrust": "^3.20.0",
102
101
  "class-variance-authority": "^0.7.1",
103
102
  "clsx": "^2.1.1",
103
+ "dockerode": "^4.0.2",
104
+ "e2b": "^2.31.0",
105
+ "jsdom": "^29.1.1",
104
106
  "lucide-react": "^1.21.0",
105
107
  "mixpanel-browser": "^2.80.0",
106
108
  "next": "16.2.10",
@@ -112,7 +114,6 @@
112
114
  "tailwind-merge": "^3.6.0",
113
115
  "tailwindcss": "^4.3.1",
114
116
  "typescript": "npm:@typescript/typescript6@^6.0.2",
115
- "@typescript/native": "npm:typescript@^7.0.2",
116
117
  "vite": "^8.1.0",
117
118
  "vitest": "^4.1.9"
118
119
  },
package/src/cli.ts CHANGED
@@ -105,7 +105,6 @@ interface Flags {
105
105
  timing?: "summary" | "full";
106
106
  keepSandbox?: "failed" | "all";
107
107
  all: boolean;
108
- allowSensitiveArtifacts: boolean;
109
108
  window?: string;
110
109
  sandboxPath?: string;
111
110
  leaveRunning: boolean;
@@ -156,8 +155,6 @@ const FLAG_OPTIONS = {
156
155
  out: { type: "string" },
157
156
  /** `view` 命令专用:指定本地服务器监听端口。 */
158
157
  port: { type: "string" },
159
- /** `view --out` 专用:对非发布根(快照没有 publish:{redaction:"applied"} 标记)导出时的显式确认——静态站会原样携带未消毒的证据文件。 */
160
- "allow-sensitive-artifacts": { type: "boolean" },
161
158
  // show 的证据切面 / 时间轴 / 报告装载(docs-site/zh/how-to/viewing-results.mdx)。
162
159
  // 证据切面只认 `@<locator>`(或收窄到单个 eval 的前缀)选出的那一个 attempt——不再有
163
160
  // 数字 `--attempt`,选哪个 attempt 由 locator 精确指名,不是「先选 eval 再挑第几次」。
@@ -307,7 +304,6 @@ function parseArgs(argv: string[]): { command: string; positionals: string[]; fl
307
304
  timing: values.timing === true ? (timingMode ?? "summary") : undefined,
308
305
  keepSandbox: values["keep-sandbox"] === true ? (keepSandboxTier ?? "failed") : undefined,
309
306
  all: values.all === true,
310
- allowSensitiveArtifacts: values["allow-sensitive-artifacts"] === true,
311
307
  window: values.window as string | undefined,
312
308
  sandboxPath: values.path as string | undefined,
313
309
  leaveRunning: values["leave-running"] === true,
@@ -591,7 +587,7 @@ async function main(): Promise<void> {
591
587
  ...(flags.page !== undefined ? { page: flags.page } : {}),
592
588
  };
593
589
  if (flags.out) {
594
- const out = await buildView({ input: viewInput.input, out: flags.out, allowSensitiveArtifacts: flags.allowSensitiveArtifacts, scan }).catch(exitOnViewUserError);
590
+ const out = await buildView({ input: viewInput.input, out: flags.out, scan }).catch(exitOnViewUserError);
595
591
  process.stdout.write(t("cli.view.exportedDir", { out }));
596
592
  process.exit(0);
597
593
  }
@@ -165,7 +165,7 @@ export function createEvalContext(deps: ContextDeps): { context: TestContext; st
165
165
  return brief(value, 4000);
166
166
  }
167
167
 
168
- /** 失败断言的 evidence:被检查值自带命令摘要(CommandResult.command)时就是「命令行本身」。 */
168
+ /** evidence 不区分 pass/fail(与 judge 同口径):被检查值自带命令摘要(CommandResult.command)时就是「命令行本身」。 */
169
169
  function checkedValueEvidence(value: unknown): string | undefined {
170
170
  const command = asCommandResult(value)?.command;
171
171
  return typeof command === "string" && command.length > 0 ? command : undefined;
@@ -378,12 +378,15 @@ export function createEvalContext(deps: ContextDeps): { context: TestContext; st
378
378
  evaluate: async (sc) => {
379
379
  const resolved = await resolveValue(value, sc);
380
380
  const score = await assertion.score(resolved);
381
- if (computePassed(spec.severity, spec.threshold, score)) return score;
381
+ const evidence = checkedValueEvidence(resolved);
382
+ if (computePassed(spec.severity, spec.threshold, score)) {
383
+ return evidence !== undefined ? { score, evidence } : score;
384
+ }
382
385
  return {
383
386
  score,
384
387
  expected: assertion.expected,
385
388
  received: previewCheckedValue(resolved),
386
- ...(checkedValueEvidence(resolved) !== undefined ? { evidence: checkedValueEvidence(resolved) } : {}),
389
+ ...(evidence !== undefined ? { evidence } : {}),
387
390
  };
388
391
  },
389
392
  };
@@ -395,18 +398,21 @@ export function createEvalContext(deps: ContextDeps): { context: TestContext; st
395
398
  const score = await assertion.score(v);
396
399
  // require 恒为硬门槛(不过即中止 eval),判定口径与 finalize 同一份 computePassed。
397
400
  const passed = computePassed("gate", assertion.threshold, score);
401
+ const evidence = checkedValueEvidence(v);
398
402
  collector.record({
399
403
  name: assertion.name,
400
404
  severity: "gate",
401
405
  threshold: assertion.threshold,
402
406
  evaluate: () =>
403
407
  passed
404
- ? score
408
+ ? evidence !== undefined
409
+ ? { score, evidence }
410
+ : score
405
411
  : {
406
412
  score,
407
413
  expected: assertion.expected,
408
414
  received: previewCheckedValue(v),
409
- ...(checkedValueEvidence(v) !== undefined ? { evidence: checkedValueEvidence(v) } : {}),
415
+ ...(evidence !== undefined ? { evidence } : {}),
410
416
  },
411
417
  });
412
418
  if (!passed) throw new EvalRequirementFailed(assertion.name);
@@ -26,7 +26,6 @@ import type { AttemptLocator } from "../results/locator.ts";
26
26
  import type {
27
27
  AttemptListItem,
28
28
  DeltaData,
29
- EntityListDataOptions,
30
29
  EvalListItem,
31
30
  ExperimentComparisonData,
32
31
  ExperimentListItem,
@@ -400,7 +399,7 @@ interface EntityListChrome extends ChromeProps {
400
399
 
401
400
  export type ExperimentListProps = DataProps<
402
401
  readonly ExperimentListItem[],
403
- EntityListDataOptions,
402
+ Record<never, never>,
404
403
  EntityListChrome & {
405
404
  /** web 面在比较表前显示实验过滤框;text 面忽略。 */
406
405
  filter?: boolean;
@@ -415,14 +414,14 @@ export type ExperimentListProps = DataProps<
415
414
  /** 实验列表:每项一个 experiment,固定八列比较表 + 展开到 Eval / Attempt。 */
416
415
  export const ExperimentList = makeDataComponent<
417
416
  readonly ExperimentListItem[],
418
- EntityListDataOptions,
417
+ Record<never, never>,
419
418
  EntityListChrome & { filter?: boolean; relativeTo?: string }
420
419
  >({
421
420
  name: "ExperimentList",
422
421
  dataFnName: "experimentListData",
423
422
  shapeName: "ExperimentListItem[]",
424
- dataFn: (input, options) => experimentListData(input, options),
425
- specKeys: ["redact"],
423
+ dataFn: (input) => experimentListData(input),
424
+ specKeys: [],
426
425
  validate: validateExperimentListData,
427
426
  web: (props, ctx) => (
428
427
  <ExperimentListWeb
@@ -437,15 +436,15 @@ export const ExperimentList = makeDataComponent<
437
436
  text: (props, ctx) => experimentListText(props.data, ctx, props.relativeTo),
438
437
  }) as unknown as ReportComponent<ExperimentListProps>;
439
438
 
440
- export type EvalListProps = DataProps<readonly EvalListItem[], EntityListDataOptions, EntityListChrome>;
439
+ export type EvalListProps = DataProps<readonly EvalListItem[], Record<never, never>, EntityListChrome>;
441
440
 
442
441
  /** Eval 列表:每项一个 experimentId + evalId,展开到这道题的 Attempt。 */
443
- export const EvalList = makeDataComponent<readonly EvalListItem[], EntityListDataOptions, EntityListChrome>({
442
+ export const EvalList = makeDataComponent<readonly EvalListItem[], Record<never, never>, EntityListChrome>({
444
443
  name: "EvalList",
445
444
  dataFnName: "evalListData",
446
445
  shapeName: "EvalListItem[]",
447
- dataFn: (input, options) => evalListData(input, options),
448
- specKeys: ["redact"],
446
+ dataFn: (input) => evalListData(input),
447
+ specKeys: [],
449
448
  validate: validateEvalListData,
450
449
  web: (props, ctx) => (
451
450
  <EvalListWeb
@@ -460,7 +459,7 @@ export const EvalList = makeDataComponent<readonly EvalListItem[], EntityListDat
460
459
 
461
460
  export type AttemptListProps = DataProps<
462
461
  readonly AttemptListItem[],
463
- EntityListDataOptions,
462
+ Record<never, never>,
464
463
  EntityListChrome & {
465
464
  /** 过滤 / 截断前的总数;省略时等于 data 长度。 */
466
465
  total?: number;
@@ -470,14 +469,14 @@ export type AttemptListProps = DataProps<
470
469
  /** Attempt 列表:实体列表的叶子层,每项一次 attempt 的判定、单行摘要与 locator。 */
471
470
  export const AttemptList = makeDataComponent<
472
471
  readonly AttemptListItem[],
473
- EntityListDataOptions,
472
+ Record<never, never>,
474
473
  EntityListChrome & { total?: number }
475
474
  >({
476
475
  name: "AttemptList",
477
476
  dataFnName: "attemptListData",
478
477
  shapeName: "AttemptListItem[]",
479
- dataFn: (input, options) => attemptListData(input, options),
480
- specKeys: ["redact"],
478
+ dataFn: (input) => attemptListData(input),
479
+ specKeys: [],
481
480
  validate: validateAttemptListData,
482
481
  web: (props, ctx) => (
483
482
  <AttemptListWeb
@@ -498,7 +497,6 @@ export interface FailureListProps {
498
497
  limit?: number;
499
498
  /** 默认宿主注入的 Scope。 */
500
499
  input?: ReportInput;
501
- redact?: (text: string) => string;
502
500
  attemptHref?: (locator: AttemptLocator) => string;
503
501
  locale?: ReportLocale;
504
502
  className?: string;
@@ -512,7 +510,7 @@ export interface FailureListProps {
512
510
  */
513
511
  export const FailureList = defineComponent<FailureListProps>(async (props, ctx) => {
514
512
  const input = props.input ?? ctx.scope;
515
- const all = await attemptListData(input, props.redact !== undefined ? { redact: props.redact } : undefined);
513
+ const all = await attemptListData(input);
516
514
  // attempt 开始时间不在列表条目里(它不是列表展示字段);从同一 input 的读取面按 locator 对回。
517
515
  const startedAtByLocator = new Map<string, string>();
518
516
  for (const item of collectItems(resolveInput(input).snapshots)) {
@@ -17,7 +17,6 @@ import type {
17
17
  DeltaData,
18
18
  DeltaPair,
19
19
  DimensionInput,
20
- EntityListDataOptions,
21
20
  EvalListItem,
22
21
  ExperimentComparisonData,
23
22
  ExperimentComparisonGroupData,
@@ -199,10 +198,8 @@ function failureSummaryOf(result: EvalResult): { summary: string | null; more: n
199
198
  return { summary: null, more: 0 };
200
199
  }
201
200
 
202
- const identityRedact = (text: string): string => text;
203
-
204
201
  /** AttemptList / ExperimentList / EvalList 共用的叶子构造:一个 Item → 一个 AttemptListItem。 */
205
- async function attemptListItemOf(item: Item, redact: (text: string) => string): Promise<AttemptListItem> {
202
+ async function attemptListItemOf(item: Item): Promise<AttemptListItem> {
206
203
  const result = item.attempt.result;
207
204
  const { summary, more } = failureSummaryOf(result);
208
205
  return {
@@ -211,7 +208,7 @@ async function attemptListItemOf(item: Item, redact: (text: string) => string):
211
208
  attempt: result.attempt,
212
209
  agent: result.agent,
213
210
  verdict: result.verdict,
214
- failureSummary: summary === null ? null : redact(summary),
211
+ failureSummary: summary,
215
212
  moreFailures: more,
216
213
  examScore: await computeCell(examScore, [item]),
217
214
  durationMs: result.durationMs,
@@ -221,20 +218,15 @@ async function attemptListItemOf(item: Item, redact: (text: string) => string):
221
218
  }
222
219
 
223
220
  /** `attemptListData(input)`:每个 Attempt 一项,顺序取自 Scope 展平顺序(不重排)。 */
224
- export async function attemptListData(
225
- input: ReportInput,
226
- options?: EntityListDataOptions,
227
- ): Promise<AttemptListItem[]> {
221
+ export async function attemptListData(input: ReportInput): Promise<AttemptListItem[]> {
228
222
  const { snapshots } = resolveInput(input);
229
- const redact = options?.redact ?? identityRedact;
230
223
  const items = collectItems(snapshots);
231
- return Promise.all(items.map((item) => attemptListItemOf(item, redact)));
224
+ return Promise.all(items.map((item) => attemptListItemOf(item)));
232
225
  }
233
226
 
234
227
  /** `evalListData(input)`:每个 `experimentId + evalId` 一项,按 evalId 再按 experimentId 升序。 */
235
- export async function evalListData(input: ReportInput, options?: EntityListDataOptions): Promise<EvalListItem[]> {
228
+ export async function evalListData(input: ReportInput): Promise<EvalListItem[]> {
236
229
  const { snapshots } = resolveInput(input);
237
- const redact = options?.redact ?? identityRedact;
238
230
  const items = collectItems(snapshots);
239
231
  const groups = new Map<string, Item[]>();
240
232
  for (const item of items) {
@@ -247,7 +239,7 @@ export async function evalListData(input: ReportInput, options?: EntityListDataO
247
239
  for (const group of groups.values()) {
248
240
  const sorted = [...group].sort((a, b) => a.attempt.result.attempt - b.attempt.result.attempt);
249
241
  const verdict = foldEvalVerdict(sorted.map((item) => item.attempt.result));
250
- const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item, redact)));
242
+ const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item)));
251
243
  out.push({
252
244
  experimentId: experimentIdOf(sorted[0]!),
253
245
  evalId: evalIdOf(sorted[0]!),
@@ -269,12 +261,8 @@ export async function evalListData(input: ReportInput, options?: EntityListDataO
269
261
  * Snapshot[] 时若同一 experiment 混入不一致的可比性配置,按完整用户反馈失败并指引——
270
262
  * 看跨配置演化用 snapshot 维度或 MetricLine,不把两套配置拼成一行冒充单一配置。
271
263
  */
272
- export async function experimentListData(
273
- input: ReportInput,
274
- options?: EntityListDataOptions,
275
- ): Promise<ExperimentListItem[]> {
264
+ export async function experimentListData(input: ReportInput): Promise<ExperimentListItem[]> {
276
265
  const { snapshots } = resolveInput(input);
277
- const redact = options?.redact ?? identityRedact;
278
266
 
279
267
  // 可比性配置单义检查:同一 experiment 的输入快照必须共享一套可比性配置。
280
268
  const configByExperiment = new Map<string, { snapshot: Snapshot; config: unknown }>();
@@ -304,7 +292,7 @@ export async function experimentListData(
304
292
  for (const [evalId, evalItems] of evalGroups) {
305
293
  const sorted = [...evalItems].sort((a, b) => a.attempt.result.attempt - b.attempt.result.attempt);
306
294
  const verdict = foldEvalVerdict(sorted.map((item) => item.attempt.result));
307
- const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item, redact)));
295
+ const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item)));
308
296
  evalRows.push({
309
297
  evalId,
310
298
  verdict,
@@ -40,6 +40,7 @@ export type {
40
40
  RenderReportTextOptions,
41
41
  RenderTreeTextOptions,
42
42
  ReportTreeHostContext,
43
+ HeadTag,
43
44
  ReportAsset,
44
45
  ReportDef,
45
46
  ReportDefinition,
@@ -167,7 +168,6 @@ export type {
167
168
  DimensionInput,
168
169
  DimensionOptions,
169
170
  DimensionRef,
170
- EntityListDataOptions,
171
171
  EvalListItem,
172
172
  ExperimentComparisonData,
173
173
  ExperimentComparisonGroupData,
@@ -3,8 +3,8 @@
3
3
  // niceeval/results 的读取契约手工构造)。覆盖登记行:两级聚合 vs 平铺、errored=0 口径、
4
4
  // skipped=null、null≠0、Scoreboard 固定分母(notRun/unscorable 分开)、权重最长前缀、
5
5
  // 身份键去重、现刻水位、自定义指标 where/aggregate、evalGroup 完整父路径、verdict 权威、
6
- // MetricCell 诚实、缺 artifact 指标、repeatedFailedCommands、实体列表 failureSummary /
7
- // redact、scopeSummaryData 两级计票、experimentComparisonData 分区、pairsByFlag、
6
+ // MetricCell 诚实、缺 artifact 指标、repeatedFailedCommands、实体列表 failureSummary
7
+ // scopeSummaryData 两级计票、experimentComparisonData 分区、pairsByFlag、
8
8
  // MetricLine 点身份、空数组反馈、metricTableData sort。
9
9
 
10
10
  import { describe, expect, it } from "vitest";
@@ -595,24 +595,6 @@ describe("实体列表 data", () => {
595
595
  expect(byEval.get("list/errored")!.costUSD).toBeNull();
596
596
  });
597
597
 
598
- it("redact 只改写 failureSummary(含嵌套 attempt 条目);身份字段、locator 与数值指标原样", async () => {
599
- const redact = (text: string) => text.replaceAll("41", "[redacted]");
600
- const attempts = await attemptListData([listSnap()], { redact });
601
- const failedItem = attempts.find((item) => item.evalId === "list/failed")!;
602
- expect(failedItem.failureSummary).toContain("[redacted]");
603
- expect(failedItem.evalId).toBe("list/failed");
604
- expect(failedItem.experimentId).toBe("exp/list");
605
- expect(failedItem.costUSD).toBe(0.1);
606
-
607
- const evals = await evalListData([listSnap()], { redact });
608
- const nested = evals.find((item) => item.evalId === "list/failed")!.attempts[0]!;
609
- expect(nested.failureSummary).toContain("[redacted]");
610
-
611
- const experiments = await experimentListData([listSnap()], { redact });
612
- const nestedInExp = experiments[0]!.evalRows.find((row) => row.evalId === "list/failed")!.attempts[0]!;
613
- expect(nestedInExp.failureSummary).toContain("[redacted]");
614
- });
615
-
616
598
  it("experimentListData:evalVerdicts / endToEndPassRate / costUSD / durationMs / tokens 齐全,默认按端到端成功率降序", async () => {
617
599
  const winner = snap({ experimentId: "exp/win", results: [res("a", "passed"), res("b", "passed")] });
618
600
  const loser = snap({ experimentId: "exp/lose", results: [res("a", "failed"), res("b", "passed")] });
@@ -1,4 +1,4 @@
1
- // defineReport:唯一可被宿主装载的产物 —— 一层外壳(标题、外链、页脚、脚本、样式)加
1
+ // defineReport:唯一可被宿主装载的产物 —— 一层外壳(标题、外链、页脚、head 标签、脚本、样式)加
2
2
  // 非空页列表;单页与多页不是两种机制,页数只是列表长度(docs/feature/reports/library/shell.md)。
3
3
  // 入参有两级缩写,各有精确展开:树入参 ≡ { content: 树 } ≡ pages: [{ id: "report",
4
4
  // title: 内置页名, content: 树 }]。`content` 与 `pages` 恰好声明一个,没有隐式默认。
@@ -40,6 +40,16 @@ export interface ReportLink {
40
40
  /** src 是相对顶层报告文件的路径;两种形态不可同时出现。 */
41
41
  export type ReportAsset = { src: string; inline?: never } | { inline: string; src?: never };
42
42
 
43
+ /**
44
+ * 结构化 head 标签。tag 是白名单闭集——head 是元数据与第三方脚本的注入口,不是 HTML 后门。
45
+ * attrs 值为 true 渲染裸布尔属性(async、defer),字符串渲染 `key="value"`(值转义后落 HTML);
46
+ * 属性语义与脚本内容同一约定——作者义务,宿主不校验。
47
+ * meta / link 无子内容由类型表达;script / style 的 children 是原样文本,不转义。
48
+ */
49
+ export type HeadTag =
50
+ | { tag: "meta" | "link"; attrs: Record<string, string | true>; children?: never }
51
+ | { tag: "script" | "style"; attrs?: Record<string, string | true>; children?: string };
52
+
43
53
  export interface ReportShell {
44
54
  /** 标题:首页 hero 与浏览器标题。页头左端是恒定的 NiceEval 品牌字标,不由 title 覆盖;回退链 def.title → 唯一快照 name → 内置文案「Eval 运行结果 / Eval Results」。 */
45
55
  title?: LocalizedText;
@@ -47,6 +57,12 @@ export interface ReportShell {
47
57
  links?: ReportLink[];
48
58
  /** 每页页脚的一段文字;省略时不渲染页脚(品牌行恒在 hero 下方,不占页脚)。 */
49
59
  footer?: LocalizedText;
60
+ /**
61
+ * 注入每页 `<head>` 的结构化标签,在官方与外壳样式之后按声明顺序渲染。
62
+ * 第三方 snippet(分析、埋点、评论)、SEO meta、favicon、字体、JSON-LD 的家:
63
+ * 声明什么标签就渲染什么标签,宿主只做结构校验,新的第三方接入不需要契约变更。
64
+ */
65
+ head?: HeadTag[];
50
66
  /** 注入每个页面的脚本,在官方增强脚本之后、按声明顺序于 </body> 前加载。 */
51
67
  scripts?: ReportAsset[];
52
68
  /** 注入每个页面的样式表,在官方样式之后按声明顺序加载。 */
@@ -76,13 +92,14 @@ const REPORT_DEFINITION: unique symbol = Symbol.for("niceeval.report.definition"
76
92
  /**
77
93
  * defineReport 的唯一产物:只作 --report 文件的默认导出,交给宿主装载。
78
94
  * 它不是 ReportNode——不能放进任何 content 或报告树,外壳因此不可嵌套。
79
- * 字段是装载规范化后的形态:pages 恒非空,links / scripts / styles 恒为数组。
95
+ * 字段是装载规范化后的形态:pages 恒非空,links / head / scripts / styles 恒为数组。
80
96
  */
81
97
  export interface ReportDefinition {
82
98
  readonly kind: "report";
83
99
  readonly title?: LocalizedText;
84
100
  readonly links: readonly ReportLink[];
85
101
  readonly footer?: LocalizedText;
102
+ readonly head: readonly HeadTag[];
86
103
  readonly scripts: readonly ReportAsset[];
87
104
  readonly styles: readonly ReportAsset[];
88
105
  readonly pages: NonEmptyArray<ReportPage>;
@@ -161,6 +178,16 @@ function assertLocalizedText(value: unknown, where: string): asserts value is Lo
161
178
 
162
179
  const PAGE_ID_PATTERN = /^[a-z0-9-]+$/;
163
180
 
181
+ /** 本地资产路径纪律(shell.md「行为约束」):相对报告文件的普通相对路径,拒绝 `..` 段、绝对路径与 `~`。 */
182
+ function assertLocalAssetPath(src: string, where: string): void {
183
+ const segments = src.split(/[\\/]+/);
184
+ if (src.startsWith("/") || /^[A-Za-z]:/.test(src) || src.startsWith("~") || segments.includes("..")) {
185
+ throw new Error(
186
+ `defineReport ${where} "${src}" is not allowed: only plain relative paths (optionally with a ./ prefix) resolve against the report file — no ".." segments, absolute paths, or "~". Move the asset next to the report file and reference it relatively.`,
187
+ );
188
+ }
189
+ }
190
+
164
191
  function assertAssets(assets: unknown, field: "scripts" | "styles"): ReportAsset[] {
165
192
  if (assets === undefined) return [];
166
193
  if (!Array.isArray(assets)) {
@@ -176,17 +203,111 @@ function assertAssets(assets: unknown, field: "scripts" | "styles"): ReportAsset
176
203
  }
177
204
  if (hasSrc) {
178
205
  const src = asset.src as string;
179
- const segments = src.split(/[\\/]+/);
180
- if (src.startsWith("/") || /^[A-Za-z]:/.test(src) || src.startsWith("~") || segments.includes("..")) {
206
+ // 外链不属于增强层资产:第三方外链标签的家是 head 通道。
207
+ if (/^https?:\/\//i.test(src) || src.startsWith("//")) {
181
208
  throw new Error(
182
- `defineReport ${field} src "${src}" is not allowed: only plain relative paths (optionally with a ./ prefix) resolve against the report file no ".." segments, absolute paths, or "~". Move the asset next to the report file and reference it relatively.`,
209
+ `defineReport ${field} src "${src}" is an external URL ${field} take local files and inline content (the host pipeline vendors them). Declare third-party external tags in "head" instead, e.g. head: [{ tag: "script", attrs: { async: true, src: "…" } }].`,
183
210
  );
184
211
  }
212
+ assertLocalAssetPath(src, `${field} src`);
185
213
  }
186
214
  }
187
215
  return assets as ReportAsset[];
188
216
  }
189
217
 
218
+ const HEAD_TAG_NAMES = new Set(["meta", "link", "script", "style"]);
219
+ const HEAD_ATTR_NAME_PATTERN = /^[a-zA-Z_][a-zA-Z0-9_.:-]*$/;
220
+
221
+ function assertHeadTags(tags: unknown): HeadTag[] {
222
+ if (tags === undefined) return [];
223
+ if (!Array.isArray(tags)) {
224
+ throw new Error(
225
+ 'defineReport head must be an array of { tag, attrs?, children? } entries (tag: "meta" | "link" | "script" | "style").',
226
+ );
227
+ }
228
+ for (const entry of tags as Array<Record<string, unknown>>) {
229
+ const tag = entry?.tag;
230
+ // 白名单闭集:head 是元数据与第三方脚本的注入口,不是 HTML 后门;标题走 title 字段回退链。
231
+ if (typeof tag !== "string" || !HEAD_TAG_NAMES.has(tag)) {
232
+ throw new Error(
233
+ `defineReport head tag ${JSON.stringify(tag)} is not allowed — head injects metadata and third-party tags, and the allowed tags are "meta", "link", "script", "style". For the document title, use the shell "title" field instead.`,
234
+ );
235
+ }
236
+ const attrs = entry.attrs;
237
+ if (attrs !== undefined && (typeof attrs !== "object" || attrs === null || Array.isArray(attrs))) {
238
+ throw new Error(
239
+ `defineReport head <${tag}> attrs must be a { name: string | true } record (true renders a bare boolean attribute like async).`,
240
+ );
241
+ }
242
+ if ((tag === "meta" || tag === "link") && attrs === undefined) {
243
+ throw new Error(
244
+ `defineReport head <${tag}> needs attrs — a bare <${tag}> renders nothing. Declare e.g. { tag: "${tag}", attrs: { ${tag === "meta" ? 'name: "…", content: "…"' : 'rel: "…", href: "…"'} } }.`,
245
+ );
246
+ }
247
+ const attrRecord = (attrs ?? {}) as Record<string, unknown>;
248
+ for (const [name, value] of Object.entries(attrRecord)) {
249
+ if (!HEAD_ATTR_NAME_PATTERN.test(name)) {
250
+ throw new Error(
251
+ `defineReport head <${tag}> attribute name ${JSON.stringify(name)} is not a valid HTML attribute name. Use letters, digits, "-", "_", ":" or ".".`,
252
+ );
253
+ }
254
+ if (value !== true && typeof value !== "string") {
255
+ throw new Error(
256
+ `defineReport head <${tag}> attribute "${name}" must be a string or true (true renders a bare boolean attribute like async); got ${typeof value}.`,
257
+ );
258
+ }
259
+ }
260
+ // 宿主自有的文档单例:charset / viewport 由宿主外壳拥有,声明它们装载报错。
261
+ if (tag === "meta" && attrRecord.charset !== undefined) {
262
+ throw new Error(
263
+ "defineReport head must not declare <meta charset> — the document charset is owned by the host shell. Remove the entry.",
264
+ );
265
+ }
266
+ if (tag === "meta" && typeof attrRecord.name === "string" && attrRecord.name.toLowerCase() === "viewport") {
267
+ throw new Error(
268
+ 'defineReport head must not declare <meta name="viewport"> — the viewport is owned by the host shell. Remove the entry.',
269
+ );
270
+ }
271
+ const children = entry.children;
272
+ if (children !== undefined) {
273
+ if (tag === "meta" || tag === "link") {
274
+ throw new Error(
275
+ `defineReport head <${tag}> does not take children — <${tag}> is a void element; put the content in attrs.`,
276
+ );
277
+ }
278
+ if (typeof children !== "string") {
279
+ throw new Error(
280
+ `defineReport head <${tag}> children must be a string of literal ${tag === "script" ? "JavaScript" : "CSS"}; got ${typeof children}.`,
281
+ );
282
+ }
283
+ // children 原样落进标签,闭合序列在该上下文无法转义,会提前截断标签。
284
+ if (children.toLowerCase().includes(`</${tag}`)) {
285
+ throw new Error(
286
+ `defineReport head <${tag}> children contain "</${tag}>" — that sequence cannot be escaped inside a <${tag}> and would close the tag early. Split the content into two entries or move it into a local file asset.`,
287
+ );
288
+ }
289
+ }
290
+ // src / href 按 scheme 分流:http(s) 外链原样透传;其余按本地路径纪律解析。
291
+ for (const name of ["src", "href"]) {
292
+ const value = attrRecord[name];
293
+ if (typeof value !== "string") continue;
294
+ if (/^https?:\/\//i.test(value)) continue;
295
+ if (value.startsWith("//")) {
296
+ throw new Error(
297
+ `defineReport head <${tag}> ${name} "${value}" is protocol-relative — declare the scheme explicitly, e.g. "https:${value}".`,
298
+ );
299
+ }
300
+ if (/^[a-z][a-z0-9+.-]*:/i.test(value)) {
301
+ throw new Error(
302
+ `defineReport head <${tag}> ${name} "${value}" uses a scheme other than http(s) — external head assets must be http(s) URLs. Anything else, ship as a local file next to the report and reference it relatively.`,
303
+ );
304
+ }
305
+ assertLocalAssetPath(value, `head <${tag}> ${name}`);
306
+ }
307
+ }
308
+ return tags as HeadTag[];
309
+ }
310
+
190
311
  export function defineReport(content: ReportNode): ReportDefinition;
191
312
  export function defineReport(def: ReportDef): ReportDefinition;
192
313
  export function defineReport(input: ReportNode | ReportDef): ReportDefinition {
@@ -196,7 +317,7 @@ export function defineReport(input: ReportNode | ReportDef): ReportDefinition {
196
317
  : (input as ReportDef);
197
318
  if (typeof def !== "object" || def === null) {
198
319
  throw new Error(
199
- "defineReport expects a report tree or a config object ({ title?, links?, footer?, scripts?, styles?, content | pages }). " +
320
+ "defineReport expects a report tree or a config object ({ title?, links?, footer?, head?, scripts?, styles?, content | pages }). " +
200
321
  CONTENT_NEXT_STEP,
201
322
  );
202
323
  }
@@ -273,6 +394,7 @@ export function defineReport(input: ReportNode | ReportDef): ReportDefinition {
273
394
  ...(def.title !== undefined ? { title: def.title } : {}),
274
395
  links: [...links],
275
396
  ...(def.footer !== undefined ? { footer: def.footer } : {}),
397
+ head: assertHeadTags(def.head),
276
398
  scripts: assertAssets(def.scripts, "scripts"),
277
399
  styles: assertAssets(def.styles, "styles"),
278
400
  pages: pages as unknown as NonEmptyArray<ReportPage>,