niceeval 0.7.1 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/report/components.d.ts +4 -5
- package/dist/report/components.js +7 -7
- package/dist/report/compute.d.ts +4 -4
- package/dist/report/compute.js +8 -12
- package/dist/report/index.d.ts +2 -2
- package/dist/report/report.d.ts +23 -1
- package/dist/report/report.js +82 -5
- package/dist/report/types.d.ts +0 -10
- package/dist/results/types.d.ts +0 -11
- package/docs-site/zh/how-to/custom-reports.mdx +3 -3
- package/docs-site/zh/how-to/publish-report.mdx +43 -9
- package/docs-site/zh/how-to/viewing-results.mdx +4 -4
- package/docs-site/zh/reference/cli.mdx +2 -3
- package/docs-site/zh/reference/report-components.mdx +2 -2
- package/docs-site/zh/reference/results-data.mdx +2 -4
- package/docs-site/zh/troubleshooting/debugging.mdx +2 -2
- package/package.json +7 -6
- package/src/cli.ts +1 -5
- package/src/context/context.ts +11 -5
- package/src/report/components.tsx +13 -15
- package/src/report/compute.ts +8 -20
- package/src/report/index.ts +1 -1
- package/src/report/report.test.ts +2 -20
- package/src/report/report.ts +128 -6
- package/src/report/shell-head.test.ts +102 -0
- package/src/report/types.ts +0 -11
- package/src/results/copy.ts +15 -78
- package/src/results/publish.ts +4 -146
- package/src/results/results.test.ts +8 -8
- package/src/results/types.ts +0 -7
- package/src/show/report-host.ts +13 -0
- package/src/view/app/components/CodeView.test.tsx +142 -0
- package/src/view/app/components/CodeView.tsx +15 -1
- package/src/view/app/components/Transcript.tsx +28 -1
- package/src/view/app/i18n.ts +6 -0
- package/src/view/app/lib/guards.test.ts +108 -0
- package/src/view/app/lib/guards.ts +13 -3
- package/src/view/app/lib/transcript-data.tsx +14 -0
- package/src/view/app/types.ts +17 -1
- package/src/view/artifact-serving.test.ts +1 -1
- package/src/view/client-dist/app.css +1 -1
- package/src/view/client-dist/app.js +20 -20
- package/src/view/data.ts +54 -14
- package/src/view/index.ts +18 -58
- package/src/view/server.ts +49 -144
- package/src/view/site-head.test.ts +177 -0
- package/src/view/site-parity.test.ts +117 -0
- package/src/view/site.ts +209 -0
- package/src/view/styles.css +10 -0
- package/src/view/view-report.test.ts +6 -6
|
@@ -211,7 +211,7 @@ inspect: niceeval show @<id> [--eval|--execution|--diff]
|
|
|
211
211
|
每项固定代表一个 Attempt,显示 experiment、Eval、Attempt 序号、判定、耗时、成本、失败断言、结构化 error 的一层摘要、Judge 评语和证据链接。diagnostics、cause 和 stack 留给 locator 下钻详情,避免比较列表被基础设施日志撑开。它既能列失败证据,也能列通过样本,不把 verdict 过滤写死在组件名里。
|
|
212
212
|
|
|
213
213
|
```tsx
|
|
214
|
-
const attempts = await AttemptList.data(selection
|
|
214
|
+
const attempts = await AttemptList.data(selection);
|
|
215
215
|
|
|
216
216
|
<AttemptList
|
|
217
217
|
items={attempts.filter((item) => item.verdict === "failed" || item.verdict === "errored")}
|
|
@@ -234,7 +234,7 @@ inspect: niceeval show @<id> [--eval|--execution|--diff]
|
|
|
234
234
|
(3 more not shown · showing 20 of 23)
|
|
235
235
|
```
|
|
236
236
|
|
|
237
|
-
|
|
237
|
+
要展示哪些 Attempt,过滤返回的 `AttemptListItem[]`。`limit` 也由报告作者在数组上用 `.slice(0, 20)` 表达,截断时把原始数量交给组件的 `total`,组件据此显示“还有 n 项未展示”,不静默截断。
|
|
238
238
|
|
|
239
239
|
## 指标表(`MetricTable`)
|
|
240
240
|
|
|
@@ -235,9 +235,7 @@ import { openResults, copySnapshots } from "niceeval/results";
|
|
|
235
235
|
const results = await openResults(".niceeval");
|
|
236
236
|
await copySnapshots(results.latest(), "site-data/run", {
|
|
237
237
|
artifacts: ["sources", "events", "trace", "o11y"], // diff 不截断,缺省也不带;
|
|
238
|
-
|
|
239
|
-
}); // redact 必填:函数消毒,或 false 显式声明原文发布;
|
|
240
|
-
// 每个待发布文件还会经过 50 MiB 预检;
|
|
238
|
+
}); // 每个待发布文件还会经过 50 MiB 预检;
|
|
241
239
|
// o11y 只有几 KB,报告用到 turns 这类
|
|
242
240
|
// 读 o11y 的指标就把它带上,不然渲染成「—」
|
|
243
241
|
```
|
|
@@ -246,7 +244,7 @@ await copySnapshots(results.latest(), "site-data/run", {
|
|
|
246
244
|
|
|
247
245
|
复制开始前,NiceEval 会规划全部目标文件并检查序列化后的大小。任一文件超过固定的 50 MiB,整次复制在创建目标目录前失败,错误会列出路径、实际大小和处理建议。你可以从 `artifacts` 排除那类证据;如果是旧版本留下的超大 events / trace,用当前版本重跑后再发布。这个检查既覆盖没有逐值截断的源码 / diff,也覆盖单值都正常但累计过大的 JSON,避免直到 `git push` 才撞上 Git host 的单文件限制。
|
|
248
246
|
|
|
249
|
-
|
|
247
|
+
大小预检只决定整次复制成功或失败,不会从一个超大文件中间删内容。复制忠实于源:artifact 按原字节复制,不重新序列化、不改写。唯一随行补记的是挑选时的**覆盖事实**:`partial-coverage` 警告的分母是实验的历史并集,而发布目录没有历史——所以每个复制出的快照带上 `knownEvalIds`(复制时刻该实验已知的 eval 并集),reader 端把它并进 `exp.evalIds` 的计算(取本地历史与快照携带值的并集)。发布目录上重新 `openResults().latest()`,残缺警告被同一套机制重新算出来,不靠发布者转述。复制出的目录就是标准结果目录,`niceeval view --results <目录>` 直接能看;要让报告站随 push 自动更新,workflow 见[通过 CI 发布报告](/zh/how-to/publish-report)。
|
|
250
248
|
|
|
251
249
|
## 分层速览
|
|
252
250
|
|
|
@@ -190,8 +190,8 @@ niceeval view
|
|
|
190
190
|
**打开归档或别人发来的结果。** 结果目录是自包含的——从 CI 下载的、同事拷给你的、发布到静态站前生成的目录,都能直接指过去:
|
|
191
191
|
|
|
192
192
|
```bash
|
|
193
|
-
niceeval show --
|
|
194
|
-
niceeval view --
|
|
193
|
+
niceeval show --results tmp/ci-artifacts/results
|
|
194
|
+
niceeval view --results site-data/run
|
|
195
195
|
```
|
|
196
196
|
|
|
197
197
|
一个注意点:如果本地清理过旧快照目录,之后的运行里「沿用上次结果」的条目会找不到原始证据(显示为缺失)。要长期归档某次运行,先用 [`copySnapshots`](/zh/reference/results-data) 复制出一份再删。
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "niceeval",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"description": "Agent-native eval tool — eval agents, services, functions, and coding-agent fixtures",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -83,10 +83,6 @@
|
|
|
83
83
|
},
|
|
84
84
|
"devDependencies": {
|
|
85
85
|
"@ai-sdk/otel": "^1.0.9",
|
|
86
|
-
"@vercel/sandbox": "^2.2.1",
|
|
87
|
-
"braintrust": "^3.20.0",
|
|
88
|
-
"dockerode": "^4.0.2",
|
|
89
|
-
"e2b": "^2.31.0",
|
|
90
86
|
"@opentelemetry/exporter-trace-otlp-http": "^0.219.0",
|
|
91
87
|
"@opentelemetry/sdk-trace-node": "^2.8.0",
|
|
92
88
|
"@radix-ui/react-collapsible": "^1.1.14",
|
|
@@ -98,9 +94,15 @@
|
|
|
98
94
|
"@types/react": "^19.2.17",
|
|
99
95
|
"@types/react-dom": "^19.2.3",
|
|
100
96
|
"@types/tar-stream": "^3.1.3",
|
|
97
|
+
"@typescript/native": "npm:typescript@^7.0.2",
|
|
98
|
+
"@vercel/sandbox": "^2.2.1",
|
|
101
99
|
"@vitejs/plugin-react": "^6.0.3",
|
|
100
|
+
"braintrust": "^3.20.0",
|
|
102
101
|
"class-variance-authority": "^0.7.1",
|
|
103
102
|
"clsx": "^2.1.1",
|
|
103
|
+
"dockerode": "^4.0.2",
|
|
104
|
+
"e2b": "^2.31.0",
|
|
105
|
+
"jsdom": "^29.1.1",
|
|
104
106
|
"lucide-react": "^1.21.0",
|
|
105
107
|
"mixpanel-browser": "^2.80.0",
|
|
106
108
|
"next": "16.2.10",
|
|
@@ -112,7 +114,6 @@
|
|
|
112
114
|
"tailwind-merge": "^3.6.0",
|
|
113
115
|
"tailwindcss": "^4.3.1",
|
|
114
116
|
"typescript": "npm:@typescript/typescript6@^6.0.2",
|
|
115
|
-
"@typescript/native": "npm:typescript@^7.0.2",
|
|
116
117
|
"vite": "^8.1.0",
|
|
117
118
|
"vitest": "^4.1.9"
|
|
118
119
|
},
|
package/src/cli.ts
CHANGED
|
@@ -105,7 +105,6 @@ interface Flags {
|
|
|
105
105
|
timing?: "summary" | "full";
|
|
106
106
|
keepSandbox?: "failed" | "all";
|
|
107
107
|
all: boolean;
|
|
108
|
-
allowSensitiveArtifacts: boolean;
|
|
109
108
|
window?: string;
|
|
110
109
|
sandboxPath?: string;
|
|
111
110
|
leaveRunning: boolean;
|
|
@@ -156,8 +155,6 @@ const FLAG_OPTIONS = {
|
|
|
156
155
|
out: { type: "string" },
|
|
157
156
|
/** `view` 命令专用:指定本地服务器监听端口。 */
|
|
158
157
|
port: { type: "string" },
|
|
159
|
-
/** `view --out` 专用:对非发布根(快照没有 publish:{redaction:"applied"} 标记)导出时的显式确认——静态站会原样携带未消毒的证据文件。 */
|
|
160
|
-
"allow-sensitive-artifacts": { type: "boolean" },
|
|
161
158
|
// show 的证据切面 / 时间轴 / 报告装载(docs-site/zh/how-to/viewing-results.mdx)。
|
|
162
159
|
// 证据切面只认 `@<locator>`(或收窄到单个 eval 的前缀)选出的那一个 attempt——不再有
|
|
163
160
|
// 数字 `--attempt`,选哪个 attempt 由 locator 精确指名,不是「先选 eval 再挑第几次」。
|
|
@@ -307,7 +304,6 @@ function parseArgs(argv: string[]): { command: string; positionals: string[]; fl
|
|
|
307
304
|
timing: values.timing === true ? (timingMode ?? "summary") : undefined,
|
|
308
305
|
keepSandbox: values["keep-sandbox"] === true ? (keepSandboxTier ?? "failed") : undefined,
|
|
309
306
|
all: values.all === true,
|
|
310
|
-
allowSensitiveArtifacts: values["allow-sensitive-artifacts"] === true,
|
|
311
307
|
window: values.window as string | undefined,
|
|
312
308
|
sandboxPath: values.path as string | undefined,
|
|
313
309
|
leaveRunning: values["leave-running"] === true,
|
|
@@ -591,7 +587,7 @@ async function main(): Promise<void> {
|
|
|
591
587
|
...(flags.page !== undefined ? { page: flags.page } : {}),
|
|
592
588
|
};
|
|
593
589
|
if (flags.out) {
|
|
594
|
-
const out = await buildView({ input: viewInput.input, out: flags.out,
|
|
590
|
+
const out = await buildView({ input: viewInput.input, out: flags.out, scan }).catch(exitOnViewUserError);
|
|
595
591
|
process.stdout.write(t("cli.view.exportedDir", { out }));
|
|
596
592
|
process.exit(0);
|
|
597
593
|
}
|
package/src/context/context.ts
CHANGED
|
@@ -165,7 +165,7 @@ export function createEvalContext(deps: ContextDeps): { context: TestContext; st
|
|
|
165
165
|
return brief(value, 4000);
|
|
166
166
|
}
|
|
167
167
|
|
|
168
|
-
/**
|
|
168
|
+
/** evidence 不区分 pass/fail(与 judge 同口径):被检查值自带命令摘要(CommandResult.command)时就是「命令行本身」。 */
|
|
169
169
|
function checkedValueEvidence(value: unknown): string | undefined {
|
|
170
170
|
const command = asCommandResult(value)?.command;
|
|
171
171
|
return typeof command === "string" && command.length > 0 ? command : undefined;
|
|
@@ -378,12 +378,15 @@ export function createEvalContext(deps: ContextDeps): { context: TestContext; st
|
|
|
378
378
|
evaluate: async (sc) => {
|
|
379
379
|
const resolved = await resolveValue(value, sc);
|
|
380
380
|
const score = await assertion.score(resolved);
|
|
381
|
-
|
|
381
|
+
const evidence = checkedValueEvidence(resolved);
|
|
382
|
+
if (computePassed(spec.severity, spec.threshold, score)) {
|
|
383
|
+
return evidence !== undefined ? { score, evidence } : score;
|
|
384
|
+
}
|
|
382
385
|
return {
|
|
383
386
|
score,
|
|
384
387
|
expected: assertion.expected,
|
|
385
388
|
received: previewCheckedValue(resolved),
|
|
386
|
-
...(
|
|
389
|
+
...(evidence !== undefined ? { evidence } : {}),
|
|
387
390
|
};
|
|
388
391
|
},
|
|
389
392
|
};
|
|
@@ -395,18 +398,21 @@ export function createEvalContext(deps: ContextDeps): { context: TestContext; st
|
|
|
395
398
|
const score = await assertion.score(v);
|
|
396
399
|
// require 恒为硬门槛(不过即中止 eval),判定口径与 finalize 同一份 computePassed。
|
|
397
400
|
const passed = computePassed("gate", assertion.threshold, score);
|
|
401
|
+
const evidence = checkedValueEvidence(v);
|
|
398
402
|
collector.record({
|
|
399
403
|
name: assertion.name,
|
|
400
404
|
severity: "gate",
|
|
401
405
|
threshold: assertion.threshold,
|
|
402
406
|
evaluate: () =>
|
|
403
407
|
passed
|
|
404
|
-
?
|
|
408
|
+
? evidence !== undefined
|
|
409
|
+
? { score, evidence }
|
|
410
|
+
: score
|
|
405
411
|
: {
|
|
406
412
|
score,
|
|
407
413
|
expected: assertion.expected,
|
|
408
414
|
received: previewCheckedValue(v),
|
|
409
|
-
...(
|
|
415
|
+
...(evidence !== undefined ? { evidence } : {}),
|
|
410
416
|
},
|
|
411
417
|
});
|
|
412
418
|
if (!passed) throw new EvalRequirementFailed(assertion.name);
|
|
@@ -26,7 +26,6 @@ import type { AttemptLocator } from "../results/locator.ts";
|
|
|
26
26
|
import type {
|
|
27
27
|
AttemptListItem,
|
|
28
28
|
DeltaData,
|
|
29
|
-
EntityListDataOptions,
|
|
30
29
|
EvalListItem,
|
|
31
30
|
ExperimentComparisonData,
|
|
32
31
|
ExperimentListItem,
|
|
@@ -400,7 +399,7 @@ interface EntityListChrome extends ChromeProps {
|
|
|
400
399
|
|
|
401
400
|
export type ExperimentListProps = DataProps<
|
|
402
401
|
readonly ExperimentListItem[],
|
|
403
|
-
|
|
402
|
+
Record<never, never>,
|
|
404
403
|
EntityListChrome & {
|
|
405
404
|
/** web 面在比较表前显示实验过滤框;text 面忽略。 */
|
|
406
405
|
filter?: boolean;
|
|
@@ -415,14 +414,14 @@ export type ExperimentListProps = DataProps<
|
|
|
415
414
|
/** 实验列表:每项一个 experiment,固定八列比较表 + 展开到 Eval / Attempt。 */
|
|
416
415
|
export const ExperimentList = makeDataComponent<
|
|
417
416
|
readonly ExperimentListItem[],
|
|
418
|
-
|
|
417
|
+
Record<never, never>,
|
|
419
418
|
EntityListChrome & { filter?: boolean; relativeTo?: string }
|
|
420
419
|
>({
|
|
421
420
|
name: "ExperimentList",
|
|
422
421
|
dataFnName: "experimentListData",
|
|
423
422
|
shapeName: "ExperimentListItem[]",
|
|
424
|
-
dataFn: (input
|
|
425
|
-
specKeys: [
|
|
423
|
+
dataFn: (input) => experimentListData(input),
|
|
424
|
+
specKeys: [],
|
|
426
425
|
validate: validateExperimentListData,
|
|
427
426
|
web: (props, ctx) => (
|
|
428
427
|
<ExperimentListWeb
|
|
@@ -437,15 +436,15 @@ export const ExperimentList = makeDataComponent<
|
|
|
437
436
|
text: (props, ctx) => experimentListText(props.data, ctx, props.relativeTo),
|
|
438
437
|
}) as unknown as ReportComponent<ExperimentListProps>;
|
|
439
438
|
|
|
440
|
-
export type EvalListProps = DataProps<readonly EvalListItem[],
|
|
439
|
+
export type EvalListProps = DataProps<readonly EvalListItem[], Record<never, never>, EntityListChrome>;
|
|
441
440
|
|
|
442
441
|
/** Eval 列表:每项一个 experimentId + evalId,展开到这道题的 Attempt。 */
|
|
443
|
-
export const EvalList = makeDataComponent<readonly EvalListItem[],
|
|
442
|
+
export const EvalList = makeDataComponent<readonly EvalListItem[], Record<never, never>, EntityListChrome>({
|
|
444
443
|
name: "EvalList",
|
|
445
444
|
dataFnName: "evalListData",
|
|
446
445
|
shapeName: "EvalListItem[]",
|
|
447
|
-
dataFn: (input
|
|
448
|
-
specKeys: [
|
|
446
|
+
dataFn: (input) => evalListData(input),
|
|
447
|
+
specKeys: [],
|
|
449
448
|
validate: validateEvalListData,
|
|
450
449
|
web: (props, ctx) => (
|
|
451
450
|
<EvalListWeb
|
|
@@ -460,7 +459,7 @@ export const EvalList = makeDataComponent<readonly EvalListItem[], EntityListDat
|
|
|
460
459
|
|
|
461
460
|
export type AttemptListProps = DataProps<
|
|
462
461
|
readonly AttemptListItem[],
|
|
463
|
-
|
|
462
|
+
Record<never, never>,
|
|
464
463
|
EntityListChrome & {
|
|
465
464
|
/** 过滤 / 截断前的总数;省略时等于 data 长度。 */
|
|
466
465
|
total?: number;
|
|
@@ -470,14 +469,14 @@ export type AttemptListProps = DataProps<
|
|
|
470
469
|
/** Attempt 列表:实体列表的叶子层,每项一次 attempt 的判定、单行摘要与 locator。 */
|
|
471
470
|
export const AttemptList = makeDataComponent<
|
|
472
471
|
readonly AttemptListItem[],
|
|
473
|
-
|
|
472
|
+
Record<never, never>,
|
|
474
473
|
EntityListChrome & { total?: number }
|
|
475
474
|
>({
|
|
476
475
|
name: "AttemptList",
|
|
477
476
|
dataFnName: "attemptListData",
|
|
478
477
|
shapeName: "AttemptListItem[]",
|
|
479
|
-
dataFn: (input
|
|
480
|
-
specKeys: [
|
|
478
|
+
dataFn: (input) => attemptListData(input),
|
|
479
|
+
specKeys: [],
|
|
481
480
|
validate: validateAttemptListData,
|
|
482
481
|
web: (props, ctx) => (
|
|
483
482
|
<AttemptListWeb
|
|
@@ -498,7 +497,6 @@ export interface FailureListProps {
|
|
|
498
497
|
limit?: number;
|
|
499
498
|
/** 默认宿主注入的 Scope。 */
|
|
500
499
|
input?: ReportInput;
|
|
501
|
-
redact?: (text: string) => string;
|
|
502
500
|
attemptHref?: (locator: AttemptLocator) => string;
|
|
503
501
|
locale?: ReportLocale;
|
|
504
502
|
className?: string;
|
|
@@ -512,7 +510,7 @@ export interface FailureListProps {
|
|
|
512
510
|
*/
|
|
513
511
|
export const FailureList = defineComponent<FailureListProps>(async (props, ctx) => {
|
|
514
512
|
const input = props.input ?? ctx.scope;
|
|
515
|
-
const all = await attemptListData(input
|
|
513
|
+
const all = await attemptListData(input);
|
|
516
514
|
// attempt 开始时间不在列表条目里(它不是列表展示字段);从同一 input 的读取面按 locator 对回。
|
|
517
515
|
const startedAtByLocator = new Map<string, string>();
|
|
518
516
|
for (const item of collectItems(resolveInput(input).snapshots)) {
|
package/src/report/compute.ts
CHANGED
|
@@ -17,7 +17,6 @@ import type {
|
|
|
17
17
|
DeltaData,
|
|
18
18
|
DeltaPair,
|
|
19
19
|
DimensionInput,
|
|
20
|
-
EntityListDataOptions,
|
|
21
20
|
EvalListItem,
|
|
22
21
|
ExperimentComparisonData,
|
|
23
22
|
ExperimentComparisonGroupData,
|
|
@@ -199,10 +198,8 @@ function failureSummaryOf(result: EvalResult): { summary: string | null; more: n
|
|
|
199
198
|
return { summary: null, more: 0 };
|
|
200
199
|
}
|
|
201
200
|
|
|
202
|
-
const identityRedact = (text: string): string => text;
|
|
203
|
-
|
|
204
201
|
/** AttemptList / ExperimentList / EvalList 共用的叶子构造:一个 Item → 一个 AttemptListItem。 */
|
|
205
|
-
async function attemptListItemOf(item: Item
|
|
202
|
+
async function attemptListItemOf(item: Item): Promise<AttemptListItem> {
|
|
206
203
|
const result = item.attempt.result;
|
|
207
204
|
const { summary, more } = failureSummaryOf(result);
|
|
208
205
|
return {
|
|
@@ -211,7 +208,7 @@ async function attemptListItemOf(item: Item, redact: (text: string) => string):
|
|
|
211
208
|
attempt: result.attempt,
|
|
212
209
|
agent: result.agent,
|
|
213
210
|
verdict: result.verdict,
|
|
214
|
-
failureSummary: summary
|
|
211
|
+
failureSummary: summary,
|
|
215
212
|
moreFailures: more,
|
|
216
213
|
examScore: await computeCell(examScore, [item]),
|
|
217
214
|
durationMs: result.durationMs,
|
|
@@ -221,20 +218,15 @@ async function attemptListItemOf(item: Item, redact: (text: string) => string):
|
|
|
221
218
|
}
|
|
222
219
|
|
|
223
220
|
/** `attemptListData(input)`:每个 Attempt 一项,顺序取自 Scope 展平顺序(不重排)。 */
|
|
224
|
-
export async function attemptListData(
|
|
225
|
-
input: ReportInput,
|
|
226
|
-
options?: EntityListDataOptions,
|
|
227
|
-
): Promise<AttemptListItem[]> {
|
|
221
|
+
export async function attemptListData(input: ReportInput): Promise<AttemptListItem[]> {
|
|
228
222
|
const { snapshots } = resolveInput(input);
|
|
229
|
-
const redact = options?.redact ?? identityRedact;
|
|
230
223
|
const items = collectItems(snapshots);
|
|
231
|
-
return Promise.all(items.map((item) => attemptListItemOf(item
|
|
224
|
+
return Promise.all(items.map((item) => attemptListItemOf(item)));
|
|
232
225
|
}
|
|
233
226
|
|
|
234
227
|
/** `evalListData(input)`:每个 `experimentId + evalId` 一项,按 evalId 再按 experimentId 升序。 */
|
|
235
|
-
export async function evalListData(input: ReportInput
|
|
228
|
+
export async function evalListData(input: ReportInput): Promise<EvalListItem[]> {
|
|
236
229
|
const { snapshots } = resolveInput(input);
|
|
237
|
-
const redact = options?.redact ?? identityRedact;
|
|
238
230
|
const items = collectItems(snapshots);
|
|
239
231
|
const groups = new Map<string, Item[]>();
|
|
240
232
|
for (const item of items) {
|
|
@@ -247,7 +239,7 @@ export async function evalListData(input: ReportInput, options?: EntityListDataO
|
|
|
247
239
|
for (const group of groups.values()) {
|
|
248
240
|
const sorted = [...group].sort((a, b) => a.attempt.result.attempt - b.attempt.result.attempt);
|
|
249
241
|
const verdict = foldEvalVerdict(sorted.map((item) => item.attempt.result));
|
|
250
|
-
const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item
|
|
242
|
+
const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item)));
|
|
251
243
|
out.push({
|
|
252
244
|
experimentId: experimentIdOf(sorted[0]!),
|
|
253
245
|
evalId: evalIdOf(sorted[0]!),
|
|
@@ -269,12 +261,8 @@ export async function evalListData(input: ReportInput, options?: EntityListDataO
|
|
|
269
261
|
* Snapshot[] 时若同一 experiment 混入不一致的可比性配置,按完整用户反馈失败并指引——
|
|
270
262
|
* 看跨配置演化用 snapshot 维度或 MetricLine,不把两套配置拼成一行冒充单一配置。
|
|
271
263
|
*/
|
|
272
|
-
export async function experimentListData(
|
|
273
|
-
input: ReportInput,
|
|
274
|
-
options?: EntityListDataOptions,
|
|
275
|
-
): Promise<ExperimentListItem[]> {
|
|
264
|
+
export async function experimentListData(input: ReportInput): Promise<ExperimentListItem[]> {
|
|
276
265
|
const { snapshots } = resolveInput(input);
|
|
277
|
-
const redact = options?.redact ?? identityRedact;
|
|
278
266
|
|
|
279
267
|
// 可比性配置单义检查:同一 experiment 的输入快照必须共享一套可比性配置。
|
|
280
268
|
const configByExperiment = new Map<string, { snapshot: Snapshot; config: unknown }>();
|
|
@@ -304,7 +292,7 @@ export async function experimentListData(
|
|
|
304
292
|
for (const [evalId, evalItems] of evalGroups) {
|
|
305
293
|
const sorted = [...evalItems].sort((a, b) => a.attempt.result.attempt - b.attempt.result.attempt);
|
|
306
294
|
const verdict = foldEvalVerdict(sorted.map((item) => item.attempt.result));
|
|
307
|
-
const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item
|
|
295
|
+
const attempts = await Promise.all(sorted.map((item) => attemptListItemOf(item)));
|
|
308
296
|
evalRows.push({
|
|
309
297
|
evalId,
|
|
310
298
|
verdict,
|
package/src/report/index.ts
CHANGED
|
@@ -40,6 +40,7 @@ export type {
|
|
|
40
40
|
RenderReportTextOptions,
|
|
41
41
|
RenderTreeTextOptions,
|
|
42
42
|
ReportTreeHostContext,
|
|
43
|
+
HeadTag,
|
|
43
44
|
ReportAsset,
|
|
44
45
|
ReportDef,
|
|
45
46
|
ReportDefinition,
|
|
@@ -167,7 +168,6 @@ export type {
|
|
|
167
168
|
DimensionInput,
|
|
168
169
|
DimensionOptions,
|
|
169
170
|
DimensionRef,
|
|
170
|
-
EntityListDataOptions,
|
|
171
171
|
EvalListItem,
|
|
172
172
|
ExperimentComparisonData,
|
|
173
173
|
ExperimentComparisonGroupData,
|
|
@@ -3,8 +3,8 @@
|
|
|
3
3
|
// niceeval/results 的读取契约手工构造)。覆盖登记行:两级聚合 vs 平铺、errored=0 口径、
|
|
4
4
|
// skipped=null、null≠0、Scoreboard 固定分母(notRun/unscorable 分开)、权重最长前缀、
|
|
5
5
|
// 身份键去重、现刻水位、自定义指标 where/aggregate、evalGroup 完整父路径、verdict 权威、
|
|
6
|
-
// MetricCell 诚实、缺 artifact 指标、repeatedFailedCommands、实体列表 failureSummary
|
|
7
|
-
//
|
|
6
|
+
// MetricCell 诚实、缺 artifact 指标、repeatedFailedCommands、实体列表 failureSummary、
|
|
7
|
+
// scopeSummaryData 两级计票、experimentComparisonData 分区、pairsByFlag、
|
|
8
8
|
// MetricLine 点身份、空数组反馈、metricTableData sort。
|
|
9
9
|
|
|
10
10
|
import { describe, expect, it } from "vitest";
|
|
@@ -595,24 +595,6 @@ describe("实体列表 data", () => {
|
|
|
595
595
|
expect(byEval.get("list/errored")!.costUSD).toBeNull();
|
|
596
596
|
});
|
|
597
597
|
|
|
598
|
-
it("redact 只改写 failureSummary(含嵌套 attempt 条目);身份字段、locator 与数值指标原样", async () => {
|
|
599
|
-
const redact = (text: string) => text.replaceAll("41", "[redacted]");
|
|
600
|
-
const attempts = await attemptListData([listSnap()], { redact });
|
|
601
|
-
const failedItem = attempts.find((item) => item.evalId === "list/failed")!;
|
|
602
|
-
expect(failedItem.failureSummary).toContain("[redacted]");
|
|
603
|
-
expect(failedItem.evalId).toBe("list/failed");
|
|
604
|
-
expect(failedItem.experimentId).toBe("exp/list");
|
|
605
|
-
expect(failedItem.costUSD).toBe(0.1);
|
|
606
|
-
|
|
607
|
-
const evals = await evalListData([listSnap()], { redact });
|
|
608
|
-
const nested = evals.find((item) => item.evalId === "list/failed")!.attempts[0]!;
|
|
609
|
-
expect(nested.failureSummary).toContain("[redacted]");
|
|
610
|
-
|
|
611
|
-
const experiments = await experimentListData([listSnap()], { redact });
|
|
612
|
-
const nestedInExp = experiments[0]!.evalRows.find((row) => row.evalId === "list/failed")!.attempts[0]!;
|
|
613
|
-
expect(nestedInExp.failureSummary).toContain("[redacted]");
|
|
614
|
-
});
|
|
615
|
-
|
|
616
598
|
it("experimentListData:evalVerdicts / endToEndPassRate / costUSD / durationMs / tokens 齐全,默认按端到端成功率降序", async () => {
|
|
617
599
|
const winner = snap({ experimentId: "exp/win", results: [res("a", "passed"), res("b", "passed")] });
|
|
618
600
|
const loser = snap({ experimentId: "exp/lose", results: [res("a", "failed"), res("b", "passed")] });
|
package/src/report/report.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// defineReport:唯一可被宿主装载的产物 —— 一层外壳(
|
|
1
|
+
// defineReport:唯一可被宿主装载的产物 —— 一层外壳(标题、外链、页脚、head 标签、脚本、样式)加
|
|
2
2
|
// 非空页列表;单页与多页不是两种机制,页数只是列表长度(docs/feature/reports/library/shell.md)。
|
|
3
3
|
// 入参有两级缩写,各有精确展开:树入参 ≡ { content: 树 } ≡ pages: [{ id: "report",
|
|
4
4
|
// title: 内置页名, content: 树 }]。`content` 与 `pages` 恰好声明一个,没有隐式默认。
|
|
@@ -40,6 +40,16 @@ export interface ReportLink {
|
|
|
40
40
|
/** src 是相对顶层报告文件的路径;两种形态不可同时出现。 */
|
|
41
41
|
export type ReportAsset = { src: string; inline?: never } | { inline: string; src?: never };
|
|
42
42
|
|
|
43
|
+
/**
|
|
44
|
+
* 结构化 head 标签。tag 是白名单闭集——head 是元数据与第三方脚本的注入口,不是 HTML 后门。
|
|
45
|
+
* attrs 值为 true 渲染裸布尔属性(async、defer),字符串渲染 `key="value"`(值转义后落 HTML);
|
|
46
|
+
* 属性语义与脚本内容同一约定——作者义务,宿主不校验。
|
|
47
|
+
* meta / link 无子内容由类型表达;script / style 的 children 是原样文本,不转义。
|
|
48
|
+
*/
|
|
49
|
+
export type HeadTag =
|
|
50
|
+
| { tag: "meta" | "link"; attrs: Record<string, string | true>; children?: never }
|
|
51
|
+
| { tag: "script" | "style"; attrs?: Record<string, string | true>; children?: string };
|
|
52
|
+
|
|
43
53
|
export interface ReportShell {
|
|
44
54
|
/** 标题:首页 hero 与浏览器标题。页头左端是恒定的 NiceEval 品牌字标,不由 title 覆盖;回退链 def.title → 唯一快照 name → 内置文案「Eval 运行结果 / Eval Results」。 */
|
|
45
55
|
title?: LocalizedText;
|
|
@@ -47,6 +57,12 @@ export interface ReportShell {
|
|
|
47
57
|
links?: ReportLink[];
|
|
48
58
|
/** 每页页脚的一段文字;省略时不渲染页脚(品牌行恒在 hero 下方,不占页脚)。 */
|
|
49
59
|
footer?: LocalizedText;
|
|
60
|
+
/**
|
|
61
|
+
* 注入每页 `<head>` 的结构化标签,在官方与外壳样式之后按声明顺序渲染。
|
|
62
|
+
* 第三方 snippet(分析、埋点、评论)、SEO meta、favicon、字体、JSON-LD 的家:
|
|
63
|
+
* 声明什么标签就渲染什么标签,宿主只做结构校验,新的第三方接入不需要契约变更。
|
|
64
|
+
*/
|
|
65
|
+
head?: HeadTag[];
|
|
50
66
|
/** 注入每个页面的脚本,在官方增强脚本之后、按声明顺序于 </body> 前加载。 */
|
|
51
67
|
scripts?: ReportAsset[];
|
|
52
68
|
/** 注入每个页面的样式表,在官方样式之后按声明顺序加载。 */
|
|
@@ -76,13 +92,14 @@ const REPORT_DEFINITION: unique symbol = Symbol.for("niceeval.report.definition"
|
|
|
76
92
|
/**
|
|
77
93
|
* defineReport 的唯一产物:只作 --report 文件的默认导出,交给宿主装载。
|
|
78
94
|
* 它不是 ReportNode——不能放进任何 content 或报告树,外壳因此不可嵌套。
|
|
79
|
-
* 字段是装载规范化后的形态:pages 恒非空,links / scripts / styles 恒为数组。
|
|
95
|
+
* 字段是装载规范化后的形态:pages 恒非空,links / head / scripts / styles 恒为数组。
|
|
80
96
|
*/
|
|
81
97
|
export interface ReportDefinition {
|
|
82
98
|
readonly kind: "report";
|
|
83
99
|
readonly title?: LocalizedText;
|
|
84
100
|
readonly links: readonly ReportLink[];
|
|
85
101
|
readonly footer?: LocalizedText;
|
|
102
|
+
readonly head: readonly HeadTag[];
|
|
86
103
|
readonly scripts: readonly ReportAsset[];
|
|
87
104
|
readonly styles: readonly ReportAsset[];
|
|
88
105
|
readonly pages: NonEmptyArray<ReportPage>;
|
|
@@ -161,6 +178,16 @@ function assertLocalizedText(value: unknown, where: string): asserts value is Lo
|
|
|
161
178
|
|
|
162
179
|
const PAGE_ID_PATTERN = /^[a-z0-9-]+$/;
|
|
163
180
|
|
|
181
|
+
/** 本地资产路径纪律(shell.md「行为约束」):相对报告文件的普通相对路径,拒绝 `..` 段、绝对路径与 `~`。 */
|
|
182
|
+
function assertLocalAssetPath(src: string, where: string): void {
|
|
183
|
+
const segments = src.split(/[\\/]+/);
|
|
184
|
+
if (src.startsWith("/") || /^[A-Za-z]:/.test(src) || src.startsWith("~") || segments.includes("..")) {
|
|
185
|
+
throw new Error(
|
|
186
|
+
`defineReport ${where} "${src}" is not allowed: only plain relative paths (optionally with a ./ prefix) resolve against the report file — no ".." segments, absolute paths, or "~". Move the asset next to the report file and reference it relatively.`,
|
|
187
|
+
);
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
|
|
164
191
|
function assertAssets(assets: unknown, field: "scripts" | "styles"): ReportAsset[] {
|
|
165
192
|
if (assets === undefined) return [];
|
|
166
193
|
if (!Array.isArray(assets)) {
|
|
@@ -176,17 +203,111 @@ function assertAssets(assets: unknown, field: "scripts" | "styles"): ReportAsset
|
|
|
176
203
|
}
|
|
177
204
|
if (hasSrc) {
|
|
178
205
|
const src = asset.src as string;
|
|
179
|
-
|
|
180
|
-
if (
|
|
206
|
+
// 外链不属于增强层资产:第三方外链标签的家是 head 通道。
|
|
207
|
+
if (/^https?:\/\//i.test(src) || src.startsWith("//")) {
|
|
181
208
|
throw new Error(
|
|
182
|
-
`defineReport ${field} src "${src}" is
|
|
209
|
+
`defineReport ${field} src "${src}" is an external URL — ${field} take local files and inline content (the host pipeline vendors them). Declare third-party external tags in "head" instead, e.g. head: [{ tag: "script", attrs: { async: true, src: "…" } }].`,
|
|
183
210
|
);
|
|
184
211
|
}
|
|
212
|
+
assertLocalAssetPath(src, `${field} src`);
|
|
185
213
|
}
|
|
186
214
|
}
|
|
187
215
|
return assets as ReportAsset[];
|
|
188
216
|
}
|
|
189
217
|
|
|
218
|
+
const HEAD_TAG_NAMES = new Set(["meta", "link", "script", "style"]);
|
|
219
|
+
const HEAD_ATTR_NAME_PATTERN = /^[a-zA-Z_][a-zA-Z0-9_.:-]*$/;
|
|
220
|
+
|
|
221
|
+
function assertHeadTags(tags: unknown): HeadTag[] {
|
|
222
|
+
if (tags === undefined) return [];
|
|
223
|
+
if (!Array.isArray(tags)) {
|
|
224
|
+
throw new Error(
|
|
225
|
+
'defineReport head must be an array of { tag, attrs?, children? } entries (tag: "meta" | "link" | "script" | "style").',
|
|
226
|
+
);
|
|
227
|
+
}
|
|
228
|
+
for (const entry of tags as Array<Record<string, unknown>>) {
|
|
229
|
+
const tag = entry?.tag;
|
|
230
|
+
// 白名单闭集:head 是元数据与第三方脚本的注入口,不是 HTML 后门;标题走 title 字段回退链。
|
|
231
|
+
if (typeof tag !== "string" || !HEAD_TAG_NAMES.has(tag)) {
|
|
232
|
+
throw new Error(
|
|
233
|
+
`defineReport head tag ${JSON.stringify(tag)} is not allowed — head injects metadata and third-party tags, and the allowed tags are "meta", "link", "script", "style". For the document title, use the shell "title" field instead.`,
|
|
234
|
+
);
|
|
235
|
+
}
|
|
236
|
+
const attrs = entry.attrs;
|
|
237
|
+
if (attrs !== undefined && (typeof attrs !== "object" || attrs === null || Array.isArray(attrs))) {
|
|
238
|
+
throw new Error(
|
|
239
|
+
`defineReport head <${tag}> attrs must be a { name: string | true } record (true renders a bare boolean attribute like async).`,
|
|
240
|
+
);
|
|
241
|
+
}
|
|
242
|
+
if ((tag === "meta" || tag === "link") && attrs === undefined) {
|
|
243
|
+
throw new Error(
|
|
244
|
+
`defineReport head <${tag}> needs attrs — a bare <${tag}> renders nothing. Declare e.g. { tag: "${tag}", attrs: { ${tag === "meta" ? 'name: "…", content: "…"' : 'rel: "…", href: "…"'} } }.`,
|
|
245
|
+
);
|
|
246
|
+
}
|
|
247
|
+
const attrRecord = (attrs ?? {}) as Record<string, unknown>;
|
|
248
|
+
for (const [name, value] of Object.entries(attrRecord)) {
|
|
249
|
+
if (!HEAD_ATTR_NAME_PATTERN.test(name)) {
|
|
250
|
+
throw new Error(
|
|
251
|
+
`defineReport head <${tag}> attribute name ${JSON.stringify(name)} is not a valid HTML attribute name. Use letters, digits, "-", "_", ":" or ".".`,
|
|
252
|
+
);
|
|
253
|
+
}
|
|
254
|
+
if (value !== true && typeof value !== "string") {
|
|
255
|
+
throw new Error(
|
|
256
|
+
`defineReport head <${tag}> attribute "${name}" must be a string or true (true renders a bare boolean attribute like async); got ${typeof value}.`,
|
|
257
|
+
);
|
|
258
|
+
}
|
|
259
|
+
}
|
|
260
|
+
// 宿主自有的文档单例:charset / viewport 由宿主外壳拥有,声明它们装载报错。
|
|
261
|
+
if (tag === "meta" && attrRecord.charset !== undefined) {
|
|
262
|
+
throw new Error(
|
|
263
|
+
"defineReport head must not declare <meta charset> — the document charset is owned by the host shell. Remove the entry.",
|
|
264
|
+
);
|
|
265
|
+
}
|
|
266
|
+
if (tag === "meta" && typeof attrRecord.name === "string" && attrRecord.name.toLowerCase() === "viewport") {
|
|
267
|
+
throw new Error(
|
|
268
|
+
'defineReport head must not declare <meta name="viewport"> — the viewport is owned by the host shell. Remove the entry.',
|
|
269
|
+
);
|
|
270
|
+
}
|
|
271
|
+
const children = entry.children;
|
|
272
|
+
if (children !== undefined) {
|
|
273
|
+
if (tag === "meta" || tag === "link") {
|
|
274
|
+
throw new Error(
|
|
275
|
+
`defineReport head <${tag}> does not take children — <${tag}> is a void element; put the content in attrs.`,
|
|
276
|
+
);
|
|
277
|
+
}
|
|
278
|
+
if (typeof children !== "string") {
|
|
279
|
+
throw new Error(
|
|
280
|
+
`defineReport head <${tag}> children must be a string of literal ${tag === "script" ? "JavaScript" : "CSS"}; got ${typeof children}.`,
|
|
281
|
+
);
|
|
282
|
+
}
|
|
283
|
+
// children 原样落进标签,闭合序列在该上下文无法转义,会提前截断标签。
|
|
284
|
+
if (children.toLowerCase().includes(`</${tag}`)) {
|
|
285
|
+
throw new Error(
|
|
286
|
+
`defineReport head <${tag}> children contain "</${tag}>" — that sequence cannot be escaped inside a <${tag}> and would close the tag early. Split the content into two entries or move it into a local file asset.`,
|
|
287
|
+
);
|
|
288
|
+
}
|
|
289
|
+
}
|
|
290
|
+
// src / href 按 scheme 分流:http(s) 外链原样透传;其余按本地路径纪律解析。
|
|
291
|
+
for (const name of ["src", "href"]) {
|
|
292
|
+
const value = attrRecord[name];
|
|
293
|
+
if (typeof value !== "string") continue;
|
|
294
|
+
if (/^https?:\/\//i.test(value)) continue;
|
|
295
|
+
if (value.startsWith("//")) {
|
|
296
|
+
throw new Error(
|
|
297
|
+
`defineReport head <${tag}> ${name} "${value}" is protocol-relative — declare the scheme explicitly, e.g. "https:${value}".`,
|
|
298
|
+
);
|
|
299
|
+
}
|
|
300
|
+
if (/^[a-z][a-z0-9+.-]*:/i.test(value)) {
|
|
301
|
+
throw new Error(
|
|
302
|
+
`defineReport head <${tag}> ${name} "${value}" uses a scheme other than http(s) — external head assets must be http(s) URLs. Anything else, ship as a local file next to the report and reference it relatively.`,
|
|
303
|
+
);
|
|
304
|
+
}
|
|
305
|
+
assertLocalAssetPath(value, `head <${tag}> ${name}`);
|
|
306
|
+
}
|
|
307
|
+
}
|
|
308
|
+
return tags as HeadTag[];
|
|
309
|
+
}
|
|
310
|
+
|
|
190
311
|
export function defineReport(content: ReportNode): ReportDefinition;
|
|
191
312
|
export function defineReport(def: ReportDef): ReportDefinition;
|
|
192
313
|
export function defineReport(input: ReportNode | ReportDef): ReportDefinition {
|
|
@@ -196,7 +317,7 @@ export function defineReport(input: ReportNode | ReportDef): ReportDefinition {
|
|
|
196
317
|
: (input as ReportDef);
|
|
197
318
|
if (typeof def !== "object" || def === null) {
|
|
198
319
|
throw new Error(
|
|
199
|
-
"defineReport expects a report tree or a config object ({ title?, links?, footer?, scripts?, styles?, content | pages }). " +
|
|
320
|
+
"defineReport expects a report tree or a config object ({ title?, links?, footer?, head?, scripts?, styles?, content | pages }). " +
|
|
200
321
|
CONTENT_NEXT_STEP,
|
|
201
322
|
);
|
|
202
323
|
}
|
|
@@ -273,6 +394,7 @@ export function defineReport(input: ReportNode | ReportDef): ReportDefinition {
|
|
|
273
394
|
...(def.title !== undefined ? { title: def.title } : {}),
|
|
274
395
|
links: [...links],
|
|
275
396
|
...(def.footer !== undefined ? { footer: def.footer } : {}),
|
|
397
|
+
head: assertHeadTags(def.head),
|
|
276
398
|
scripts: assertAssets(def.scripts, "scripts"),
|
|
277
399
|
styles: assertAssets(def.styles, "styles"),
|
|
278
400
|
pages: pages as unknown as NonEmptyArray<ReportPage>,
|