niceeval 0.11.1 → 0.11.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INDEX.md +1 -0
- package/README.md +4 -2
- package/README.zh.md +4 -2
- package/dist/agents/types.d.ts +2 -2
- package/dist/o11y/types.d.ts +3 -3
- package/dist/report/assets/colors.d.ts +1 -1
- package/dist/report/assets/colors.js +1 -1
- package/dist/report/components/attempt-detail/AttemptSource.js +1 -2
- package/dist/report/components/attempt-detail/compute.d.ts +4 -4
- package/dist/report/components/attempt-detail/compute.js +6 -6
- package/dist/report/components/attempt-detail/faces.d.ts +1 -1
- package/dist/report/components/attempt-detail/faces.js +3 -3
- package/dist/report/components/entity-lists/AttemptList.d.ts +3 -3
- package/dist/report/components/entity-lists/AttemptList.js +3 -3
- package/dist/report/components/entity-lists/ExperimentList.js +2 -2
- package/dist/report/components/entity-lists/compute.js +1 -1
- package/dist/report/components/entity-lists/faces.js +1 -1
- package/dist/report/components/entity-lists/index.d.ts +1 -1
- package/dist/report/components/entity-lists/index.js +1 -1
- package/dist/report/components/metric-views/chart-math.d.ts +2 -2
- package/dist/report/components/metric-views/chart-math.js +2 -2
- package/dist/report/components/metric-views/compute.d.ts +2 -3
- package/dist/report/components/metric-views/compute.js +3 -4
- package/dist/report/components/shared.d.ts +1 -1
- package/dist/report/components/site-components/compute.d.ts +5 -5
- package/dist/report/components/site-components/compute.js +6 -6
- package/dist/report/components/site-components/index.d.ts +7 -7
- package/dist/report/components/site-components/index.js +7 -7
- package/dist/report/components/site-components/scope-warnings.d.ts +1 -1
- package/dist/report/components/site-components/scope-warnings.js +3 -3
- package/dist/report/components/site-components/snapshot-diagnostics.js +1 -1
- package/dist/report/components/summaries/compute.d.ts +1 -1
- package/dist/report/components/summaries/compute.js +2 -2
- package/dist/report/components/summaries/faces.js +1 -1
- package/dist/report/components/summaries/index.d.ts +1 -1
- package/dist/report/components/summaries/index.js +1 -1
- package/dist/report/index.js +1 -1
- package/dist/report/model/aggregate.d.ts +2 -2
- package/dist/report/model/aggregate.js +2 -2
- package/dist/report/model/format.d.ts +2 -2
- package/dist/report/model/format.js +2 -2
- package/dist/report/model/metrics.js +3 -3
- package/dist/report/model/types.d.ts +11 -12
- package/dist/results/locator.js +1 -1
- package/dist/results/select.d.ts +4 -4
- package/dist/results/select.js +5 -5
- package/dist/results/types.d.ts +8 -8
- package/dist/runner/types.d.ts +14 -14
- package/dist/sandbox/resolve.d.ts +1 -1
- package/dist/sandbox/types.d.ts +1 -1
- package/docs-site/zh/reference/builtin-agents.mdx +21 -0
- package/docs-site/zh/reference/define-agent.mdx +1 -1
- package/docs-site/zh/reference/report-components.mdx +156 -74
- package/docs-site/zh/tutorials/custom-reports.mdx +112 -22
- package/docs-site/zh/tutorials/sandbox-agent.mdx +3 -4
- package/docs-site/zh/tutorials/sandbox-providers.mdx +15 -15
- package/docs-site/zh/tutorials/theming.mdx +169 -0
- package/package.json +2 -1
- package/src/agents/ai-sdk.test.ts +1 -1
- package/src/agents/bub-install-spec.ts +26 -8
- package/src/agents/bub.ts +40 -15
- package/src/agents/builtin.ts +11 -1
- package/src/agents/coding-cli-versions.ts +67 -0
- package/src/agents/hermes.ts +235 -0
- package/src/agents/index.ts +4 -0
- package/src/agents/langgraph.test.ts +1 -1
- package/src/agents/manifest.ts +1 -1
- package/src/agents/openai-compat.test.ts +1 -1
- package/src/agents/openclaw.ts +103 -24
- package/src/agents/opencode.ts +183 -0
- package/src/agents/sdk-streams.test.ts +1 -1
- package/src/agents/shared.ts +13 -2
- package/src/agents/types.ts +2 -2
- package/src/context/session.test.ts +1 -1
- package/src/context/session.ts +1 -1
- package/src/o11y/cost.ts +1 -1
- package/src/o11y/parsers/bub.test.ts +1 -1
- package/src/o11y/parsers/bub.ts +1 -1
- package/src/o11y/parsers/codex.test.ts +1 -1
- package/src/o11y/parsers/codex.ts +1 -1
- package/src/o11y/parsers/hermes.ts +198 -0
- package/src/o11y/parsers/openclaw.ts +25 -3
- package/src/o11y/parsers/opencode.ts +295 -0
- package/src/o11y/types.ts +3 -3
- package/src/report/assets/colors.ts +1 -1
- package/src/report/assets/styles.css +17 -15
- package/src/report/components/attempt-detail/AttemptAssertions.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptConversation.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptDiagnostics.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptDiff.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptError.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptFixPrompt.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptSource.tsx +2 -3
- package/src/report/components/attempt-detail/AttemptSummary.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptTimeline.tsx +1 -1
- package/src/report/components/attempt-detail/AttemptTrace.tsx +1 -1
- package/src/report/components/attempt-detail/UsageTable.tsx +1 -1
- package/src/report/components/attempt-detail/attempt-components.test.tsx +1 -1
- package/src/report/components/attempt-detail/compute.ts +6 -6
- package/src/report/components/attempt-detail/faces.ts +3 -3
- package/src/report/components/attempt-detail/index.tsx +1 -1
- package/src/report/components/entity-lists/AttemptList.tsx +3 -3
- package/src/report/components/entity-lists/ExperimentList.tsx +2 -2
- package/src/report/components/entity-lists/compute.ts +1 -1
- package/src/report/components/entity-lists/faces.ts +1 -1
- package/src/report/components/entity-lists/index.tsx +1 -1
- package/src/report/components/metric-views/DeltaTable.tsx +1 -2
- package/src/report/components/metric-views/StabilityMatrix.tsx +1 -1
- package/src/report/components/metric-views/chart-math.test.ts +1 -1
- package/src/report/components/metric-views/chart-math.ts +2 -2
- package/src/report/components/metric-views/compute.ts +3 -4
- package/src/report/components/shared.ts +1 -1
- package/src/report/components/site-components/CopyFixPrompt.tsx +1 -1
- package/src/report/components/site-components/HeroCard.tsx +1 -1
- package/src/report/components/site-components/PoweredBy.tsx +1 -1
- package/src/report/components/site-components/ScopeWarnings.tsx +1 -1
- package/src/report/components/site-components/SnapshotDiagnostics.tsx +1 -1
- package/src/report/components/site-components/TraceWaterfall.tsx +1 -1
- package/src/report/components/site-components/compute.ts +6 -6
- package/src/report/components/site-components/index.tsx +7 -7
- package/src/report/components/site-components/scope-warnings.ts +3 -3
- package/src/report/components/site-components/snapshot-diagnostics.ts +1 -1
- package/src/report/components/summaries/ScopeSummary.tsx +2 -2
- package/src/report/components/summaries/compute.ts +2 -2
- package/src/report/components/summaries/faces.ts +1 -1
- package/src/report/components/summaries/index.tsx +1 -1
- package/src/report/index.ts +1 -1
- package/src/report/model/aggregate.ts +2 -2
- package/src/report/model/format.ts +2 -2
- package/src/report/model/metrics.ts +3 -3
- package/src/report/model/types.ts +12 -13
- package/src/results/annotated-source.test.ts +1 -1
- package/src/results/attempt-evidence.test.ts +1 -1
- package/src/results/copy.ts +3 -3
- package/src/results/format.ts +3 -3
- package/src/results/host-equivalence.test.ts +6 -6
- package/src/results/index.ts +1 -1
- package/src/results/locator.test.ts +2 -2
- package/src/results/locator.ts +1 -1
- package/src/results/open.ts +2 -2
- package/src/results/publish.ts +1 -1
- package/src/results/results.test.ts +6 -323
- package/src/results/select.test.ts +438 -0
- package/src/results/select.ts +6 -6
- package/src/results/skipped-notice.ts +1 -1
- package/src/results/truncate.ts +2 -2
- package/src/results/types.ts +9 -9
- package/src/results/writer.ts +4 -4
- package/src/runner/attempt.test.ts +2 -2
- package/src/runner/attempt.ts +6 -6
- package/src/runner/eval-source.test.ts +1 -1
- package/src/runner/eval-source.ts +1 -1
- package/src/runner/reporters/artifacts.ts +1 -1
- package/src/runner/run.ts +2 -2
- package/src/runner/timing.ts +1 -1
- package/src/runner/types.ts +14 -14
- package/src/sandbox/docker-agent-image.ts +36 -0
- package/src/sandbox/e2b-agent-template.test.ts +6 -21
- package/src/sandbox/e2b-agent-template.ts +44 -11
- package/src/sandbox/index.ts +8 -0
- package/src/sandbox/official-baselines.test.ts +175 -0
- package/src/sandbox/resolve.ts +1 -1
- package/src/sandbox/types.ts +1 -1
- package/src/scoring/diff.ts +1 -1
- package/src/scoring/types.ts +1 -1
- package/src/shared/facts.ts +1 -1
- package/src/show/index.ts +1 -1
- package/src/show/render.ts +1 -1
- package/src/show/show.test.ts +3 -3
- package/src/view/app/App.tsx +3 -1
- package/src/view/client-dist/app.css +1 -1
- package/src/view/client-dist/app.js +1 -1
- package/src/view/data.test.ts +2 -2
- package/src/view/data.ts +1 -1
- package/src/view/site-base.test.ts +51 -0
- package/src/view/site.ts +16 -0
- package/src/view/styles.css +1 -2
- package/src/view/template.html +1 -0
- package/src/view/view-report.test.ts +1 -1
|
@@ -136,7 +136,7 @@ function EvalAttempts({
|
|
|
136
136
|
/**
|
|
137
137
|
* 覆盖缺口的占位行:状态列为 —,结果列为「当前配置下无结果」+ 可复制的补跑命令,无 attempt
|
|
138
138
|
* 子行,不参与任何指标聚合——只把分母缺口摆进读者正在看的表里
|
|
139
|
-
* (docs/feature/reports/
|
|
139
|
+
* (docs/feature/reports/components/entity-lists/experiment-list.md)。
|
|
140
140
|
*/
|
|
141
141
|
function MissingEvalRow({
|
|
142
142
|
evalId,
|
|
@@ -279,7 +279,7 @@ export function ExperimentList({
|
|
|
279
279
|
}): ReactElement {
|
|
280
280
|
const experimentLabels = shortestUniqueLabels(data.map((item) => item.experimentId));
|
|
281
281
|
// 主读数列按列表内题型构成选择;web/text 共用同一份判据,不各自重新判断
|
|
282
|
-
// (docs/feature/reports/
|
|
282
|
+
// (docs/feature/reports/components/entity-lists/experiment-list.md)。
|
|
283
283
|
const composition = experimentListScoringComposition(data);
|
|
284
284
|
const showPassRate = composition !== "points";
|
|
285
285
|
const showTotalScore = composition !== "pass";
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// 计算函数(*Data):ReportInput → 一份组件数据。实体列表族(ExperimentList / EvalList /
|
|
2
|
-
// AttemptList / FailureList)的 *Data 都住在这里(docs/feature/reports/
|
|
2
|
+
// AttemptList / FailureList)的 *Data 都住在这里(docs/feature/reports/components/entity-lists/README.md)。
|
|
3
3
|
//
|
|
4
4
|
// 共同约定(docs/feature/reports/architecture.md「指标聚合不变量」):
|
|
5
5
|
// - 第一参收 ReportInput = Scope | readonly Snapshot[];warnings 不进组件数据(宿主统一显示);
|
|
@@ -34,7 +34,7 @@ function locatorBadge(item: { locator: string; verdict: AttemptListItem["verdict
|
|
|
34
34
|
/**
|
|
35
35
|
* 时效标注(`↩` + 紧凑时距)的 text 面:历史执行(携带,或跨快照拼入)才输出,新执行为
|
|
36
36
|
* 空串;三面(ExperimentList / EvalList / AttemptList)共用
|
|
37
|
-
* (docs/feature/reports/
|
|
37
|
+
* (docs/feature/reports/components/entity-lists/README.md「时效标注」)。
|
|
38
38
|
*/
|
|
39
39
|
function historicalSuffix(item: Pick<AttemptListItem, "startedAt" | "historical">): string {
|
|
40
40
|
return item.historical ? ` ↩ ${formatHistoricalGap(item.startedAt)}` : "";
|
|
@@ -220,7 +220,7 @@ export interface FailureListProps {
|
|
|
220
220
|
|
|
221
221
|
/**
|
|
222
222
|
* 「现在有哪些失败要处理」的成品组合件:内部就是 attemptListData → 过滤 → AttemptList
|
|
223
|
-
* data 形态,与手写组合严格等价、没有私有能力(docs/feature/reports/
|
|
223
|
+
* data 形态,与手写组合严格等价、没有私有能力(docs/feature/reports/components/entity-lists/README.md)。
|
|
224
224
|
* verdict ∈ failed / errored,按 attempt 开始时间降序(同刻按 locator 字典序),
|
|
225
225
|
* 截断到 limit(默认 20),total 报告截断前总数。
|
|
226
226
|
*/
|
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
// DeltaTable:对照矩阵。每行一道 eval,每组列一个条件(首个是基准),尾部对基准的 Δ 列组;
|
|
2
2
|
// 通过制显示 verdict、计分制在同一位置显示挣分,tokens / 成本恒显示;翻转标记 ⇄ 只在该行各
|
|
3
|
-
// 条件判定不一致时出现;历史执行叠加 ↩ 标注(docs/feature/reports/
|
|
4
|
-
// 「DeltaTable」)。任一侧缺数据时对应分量各自显示缺,不硬算成 0。
|
|
3
|
+
// 条件判定不一致时出现;历史执行叠加 ↩ 标注(docs/feature/reports/components/tables/delta-table.md)。任一侧缺数据时对应分量各自显示缺,不硬算成 0。
|
|
5
4
|
|
|
6
5
|
import type { ReactElement } from "react";
|
|
7
6
|
import type { DeltaData } from "../../model/types.ts";
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
// StabilityMatrix:历史全执行的稳定性矩阵。行 = eval,列 = by 维度取值,格是该组合全部历史
|
|
2
2
|
// 执行(跨快照按身份键去重、不设可比性门槛)的判定计数——回答「这道题历史上稳不稳」,不是
|
|
3
|
-
// 现刻水位下「现在算不算过」(docs/feature/reports/
|
|
3
|
+
// 现刻水位下「现在算不算过」(docs/feature/reports/components/tables/stability-matrix.md)。
|
|
4
4
|
// 稀疏格子:没有样本的格子不出现,不编三个 0 冒充跑过。
|
|
5
5
|
|
|
6
6
|
import type { ReactElement } from "react";
|
|
@@ -93,7 +93,7 @@ describe("placePointLabels", () => {
|
|
|
93
93
|
});
|
|
94
94
|
|
|
95
95
|
// 分区「图轴值域」:对 paddedAxisDomain / axisScale 直接断言扩后的 [min, max],
|
|
96
|
-
// 不经渲染(docs/feature/reports/
|
|
96
|
+
// 不经渲染(docs/feature/reports/components/charts/README.md「值域」)。
|
|
97
97
|
|
|
98
98
|
describe("paddedAxisDomain", () => {
|
|
99
99
|
it("两端各扩数据跨度的 20%:数据极值不落在域端点上", () => {
|
|
@@ -26,7 +26,7 @@ export interface AxisBounds {
|
|
|
26
26
|
}
|
|
27
27
|
|
|
28
28
|
/**
|
|
29
|
-
* 图轴值域推定(docs/feature/reports/
|
|
29
|
+
* 图轴值域推定(docs/feature/reports/components/charts/README.md「值域」),两步:
|
|
30
30
|
*
|
|
31
31
|
* ① 呼吸边距:数据极值向两端各扩数据跨度的 20%,数据极值点因此不落在绘图框线上。数据跨度
|
|
32
32
|
* 为零(单点,或全部点同值)时,边距改取该值绝对值的 20%;值恰为 0 时取 1(否则唯一的点仍会
|
|
@@ -111,7 +111,7 @@ export interface AxisScale {
|
|
|
111
111
|
* `ticksInDomain` 在域内取整齐刻度,最后按 `[pixelLo, pixelHi]` 做线性映射。`invert` 只影响
|
|
112
112
|
* 最后这一步的映射方向(`better: "lower"` 的轴反向渲染)——值域先按呼吸边距 / bounds 推定,
|
|
113
113
|
* 再决定要不要反向,反向不改变值域本身或钳制结果。MetricScatter 与 MetricLine 的两轴共用
|
|
114
|
-
* 这一个函数(docs/feature/reports/
|
|
114
|
+
* 这一个函数(docs/feature/reports/components/charts/README.md「值域」)。
|
|
115
115
|
*/
|
|
116
116
|
export function axisScale(
|
|
117
117
|
values: readonly number[],
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// 计算函数(*Data):ReportInput → 一份组件数据。跑在 Node 侧,产物是算好的、可序列化的
|
|
2
2
|
// 普通 JSON(终值 + 渲染提示,不含公式);渲染面(web/text)只做展示。指标图形族
|
|
3
3
|
// (MetricTable / MetricMatrix / MetricBars / Scoreboard / MetricScatter / MetricLine /
|
|
4
|
-
// DeltaTable)的 *Data 与配套 Options 都住在这里(docs/feature/reports/
|
|
4
|
+
// DeltaTable)的 *Data 与配套 Options 都住在这里(docs/feature/reports/components/tables/README.md)。
|
|
5
5
|
//
|
|
6
6
|
// 共同约定(docs/feature/reports/architecture.md「指标聚合不变量」):
|
|
7
7
|
// - 第一参收 ReportInput = Scope | readonly Snapshot[];warnings 不进组件数据(宿主统一显示);
|
|
@@ -462,7 +462,7 @@ export async function metricLineData(input: ReportInput, options: MetricLineOpti
|
|
|
462
462
|
// ───────────────────────── deltaTableData 与 conditionsByFlag ─────────────────────────
|
|
463
463
|
|
|
464
464
|
/**
|
|
465
|
-
* 按 flag 机械导出全部有序条件(docs/feature/reports/
|
|
465
|
+
* 按 flag 机械导出全部有序条件(docs/feature/reports/components/tables/delta-table.md):
|
|
466
466
|
* 条件域 = input Scope 内 `by: "experiment"` 的全部取值,删除该 flag 后必须可比性配置深相等
|
|
467
467
|
* (不额外按 experiment id 的目录前缀分组——architecture.md「Scope 是默认报告的比较边界」同一条
|
|
468
468
|
* 契约);基准取 `baseline` 声明的值(缺省 = 未声明该 flag),候选是该 flag 每个其它取值各一个
|
|
@@ -839,8 +839,7 @@ export interface StabilityMatrixOptions {
|
|
|
839
839
|
|
|
840
840
|
/**
|
|
841
841
|
* 历史全执行的稳定性矩阵:行是 eval,列是 by 维度上的取值,格是该组合全部历史执行(跨快照按
|
|
842
|
-
* 身份键去重、不设可比性门槛)的判定计数(docs/feature/reports/
|
|
843
|
-
* 「StabilityMatrix」)。消费的是调用方传入的 input 本身——传 current() Scope 只看得到现刻
|
|
842
|
+
* 身份键去重、不设可比性门槛)的判定计数(docs/feature/reports/components/tables/stability-matrix.md)。消费的是调用方传入的 input 本身——传 current() Scope 只看得到现刻
|
|
844
843
|
* 水位,要看完整历史需由调用方从 ctx.results 显式选择 Snapshot[] 传入。
|
|
845
844
|
*/
|
|
846
845
|
export async function stabilityMatrixData(
|
|
@@ -28,7 +28,7 @@ export function cx(...parts: (string | undefined | false)[]): string {
|
|
|
28
28
|
type Never<T> = { [K in keyof T]?: never };
|
|
29
29
|
|
|
30
30
|
/**
|
|
31
|
-
* 官方数据组件的统一 props 组合(docs/feature/reports/
|
|
31
|
+
* 官方数据组件的统一 props 组合(docs/feature/reports/components/tables/README.md):
|
|
32
32
|
* data 形态(接收配套 *Data 的产物)或 spec 形态(Options 平铺 + 可选 input)。
|
|
33
33
|
*/
|
|
34
34
|
export type DataProps<Data, Options, Presentation> =
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// CopyFixPrompt:批量修复 prompt 的 web 面。prompt 在 resolve 阶段算好、烘进静态 HTML;
|
|
2
2
|
// 无 JS 时 prompt 全文在原生 <details> 折叠块里完整可读,「复制到剪贴板」是 enhance.js 的
|
|
3
3
|
// 增强行为(data-nre-copy),增强只加浏览行为、不改内容。failures 为 0 时零输出
|
|
4
|
-
// (docs/feature/reports/
|
|
4
|
+
// (docs/feature/reports/components/site/copy-fix-prompt.md)。
|
|
5
5
|
|
|
6
6
|
import type { ReactElement } from "react";
|
|
7
7
|
import type { CopyFixPromptData } from "../../model/types.ts";
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// HeroCard:站点标题区的 web 面——hero 标题(h1)、运行 meta(最后运行时间按渲染 locale
|
|
2
2
|
// 格式化;latestStartedAt 为 null 时内置「暂无运行」文案;snapshots > 1 时标注合成来源)
|
|
3
3
|
// 与品牌行(等同 PoweredBy,恒含、无拆除 prop)。标题输入是站点声明与 Scope 的合成物,
|
|
4
|
-
// 组件只收 data 形态(docs/feature/reports/
|
|
4
|
+
// 组件只收 data 形态(docs/feature/reports/components/site/hero-card.md)。
|
|
5
5
|
|
|
6
6
|
import type { ReactElement } from "react";
|
|
7
7
|
import type { HeroData } from "../../model/types.ts";
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// PoweredBy:唯一的品牌件的 web 面。品牌契约是「提供一个组件,不给开关」——无 props、
|
|
2
|
-
// 无关闭配置;不想要品牌 = 不用它(docs/feature/reports/
|
|
2
|
+
// 无关闭配置;不想要品牌 = 不用它(docs/feature/reports/components/site/powered-by.md)。
|
|
3
3
|
// rel 只声明 noopener 以保留 Referer(默认策略只发 origin):官网统计由此得知点击来自哪个
|
|
4
4
|
// 报告站点;静态导出在构建期不知道自己托管在哪个域名,来源不进 URL 参数。
|
|
5
5
|
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// 分类计数汇总行(恒可见);展开后每组组头 = 标题 + kind 徽标 + 去重后恰一条的
|
|
4
4
|
// 可复制命令,逐条原始 message 收进第二层 <details>(总条数 ≤ 3 默认展开)。
|
|
5
5
|
// 空警告集零输出,不渲染空容器
|
|
6
|
-
// (docs/feature/reports/
|
|
6
|
+
// (docs/feature/reports/components/site/sample-warnings.md)。
|
|
7
7
|
|
|
8
8
|
import type { ReactElement } from "react";
|
|
9
9
|
import type { ScopeWarning } from "../../model/types.ts";
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// <details>,<summary> 是恒可见的计数汇总行(涉及多少个 experiment、多少个 Snapshot、
|
|
4
4
|
// 多少条记录与最高严重度)。单个快照只有一条诊断时退化成一行,不摆只有一个孩子的空壳
|
|
5
5
|
// 层级。空诊断集零输出,不渲染空容器
|
|
6
|
-
// (docs/feature/reports/
|
|
6
|
+
// (docs/feature/reports/components/site/run-diagnostics.md)。
|
|
7
7
|
|
|
8
8
|
import type { ReactElement } from "react";
|
|
9
9
|
import type { DiagnosticRecord } from "../../../types.ts";
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// TraceWaterfall:执行时间瀑布的 web 面——每个 attempt 一行,静态渲染顶层 span 分解条
|
|
2
2
|
// (定位按 startOffsetMs / durationMs 百分比;失败 span 带失败标记),行链接 attempt 详情。
|
|
3
3
|
// trace 缺失的行照常出现并如实显示缺失;排序、缩放是渐进增强,静态 HTML 无 JS 完整可读
|
|
4
|
-
// (docs/feature/reports/
|
|
4
|
+
// (docs/feature/reports/components/site/trace-waterfall.md)。
|
|
5
5
|
|
|
6
6
|
import type { ReactElement } from "react";
|
|
7
7
|
import type { TraceWaterfallRow } from "../../model/types.ts";
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
// 计算函数(*Data):ReportInput → 一份组件数据。站点组件族(Hero / ScopeWarnings /
|
|
2
2
|
// CopyFixPrompt / TraceWaterfall)的 *Data 都住在这里
|
|
3
|
-
// (docs/feature/reports/
|
|
3
|
+
// (docs/feature/reports/components/site/README.md)。
|
|
4
4
|
//
|
|
5
5
|
// 共同约定(docs/feature/reports/architecture.md「指标聚合不变量」):
|
|
6
6
|
// - 第一参收 ReportInput = Scope | readonly Snapshot[];warnings 不进组件数据(宿主统一显示);
|
|
@@ -24,7 +24,7 @@ import { attemptListData } from "../entity-lists/compute.ts";
|
|
|
24
24
|
/**
|
|
25
25
|
* `heroData(input)`:站点标题区的运行 meta——`latestStartedAt` 取范围内最新快照的开始时间
|
|
26
26
|
* (空范围为 null,不编造当前时间),`snapshots` 计贡献当前水位的快照数
|
|
27
|
-
* (docs/feature/reports/
|
|
27
|
+
* (docs/feature/reports/components/site/hero-card.md)。
|
|
28
28
|
*/
|
|
29
29
|
export async function heroData(input: ReportInput): Promise<HeroData> {
|
|
30
30
|
const { snapshots } = resolveInput(input);
|
|
@@ -38,7 +38,7 @@ export async function heroData(input: ReportInput): Promise<HeroData> {
|
|
|
38
38
|
/**
|
|
39
39
|
* `scopeWarningsData(input)`:Scope 携带的挑选警告原样透出;`input` 是裸 `Snapshot[]` 时
|
|
40
40
|
* 没有挑选过程、没有警告,返回空数组,也如实
|
|
41
|
-
* (docs/feature/reports/
|
|
41
|
+
* (docs/feature/reports/components/site/sample-warnings.md)。
|
|
42
42
|
*/
|
|
43
43
|
export async function scopeWarningsData(input: ReportInput): Promise<readonly ScopeWarning[]> {
|
|
44
44
|
return resolveInput(input).warnings;
|
|
@@ -47,7 +47,7 @@ export async function scopeWarningsData(input: ReportInput): Promise<readonly Sc
|
|
|
47
47
|
/**
|
|
48
48
|
* `snapshotDiagnosticsData(input)`:只投影 diagnostics 非空的真实 Snapshot,不携带 `evals` 或
|
|
49
49
|
* `AttemptHandle`,不跨快照合并;按 experiment id 字典序排列,同一实验内按 startedAt 从新到旧
|
|
50
|
-
* (docs/feature/reports/
|
|
50
|
+
* (docs/feature/reports/components/site/run-diagnostics.md)。
|
|
51
51
|
*/
|
|
52
52
|
export async function snapshotDiagnosticsData(input: ReportInput): Promise<SnapshotDiagnosticsData> {
|
|
53
53
|
const { snapshots } = resolveInput(input);
|
|
@@ -61,7 +61,7 @@ export async function snapshotDiagnosticsData(input: ReportInput): Promise<Snaps
|
|
|
61
61
|
* `copyFixPromptData(input)`:把范围内全部失败(verdict 为 failed / errored 的 attempt)
|
|
62
62
|
* 整理成一段可交给 coding agent 的修复 prompt——逐失败含 eval id、主失败摘要与 attempt
|
|
63
63
|
* 下钻命令(`niceeval show @<locator>`)。prompt 面向 agent,固定英文
|
|
64
|
-
* (docs/feature/reports/
|
|
64
|
+
* (docs/feature/reports/components/site/copy-fix-prompt.md)。
|
|
65
65
|
*/
|
|
66
66
|
export async function copyFixPromptData(input: ReportInput): Promise<CopyFixPromptData> {
|
|
67
67
|
const items = await attemptListData(input);
|
|
@@ -121,7 +121,7 @@ function waterfallKindOf(kind: TraceSpan["kind"]): TraceSpanSummary["kind"] {
|
|
|
121
121
|
* trace artifact(经 AttemptHandle 懒加载的 canonical OTel span);runner 生命周期节点
|
|
122
122
|
* (`result.phases`)不进瀑布。行内只汇总顶层 span(parentSpanId 缺失或不在本 trace 内),
|
|
123
123
|
* 按 startOffsetMs 升序;trace 缺失或为空时 `durationMs` 为 null、行照常出现
|
|
124
|
-
* (docs/feature/reports/
|
|
124
|
+
* (docs/feature/reports/components/site/trace-waterfall.md)。
|
|
125
125
|
*/
|
|
126
126
|
export async function traceWaterfallData(input: ReportInput): Promise<readonly TraceWaterfallRow[]> {
|
|
127
127
|
const { snapshots, attempts } = resolveInput(input);
|
|
@@ -152,7 +152,7 @@ const assertHeroData = (data: unknown): HeroData => {
|
|
|
152
152
|
* 合成物,没有单独的 spec 等价形。web 面渲染 hero 标题(h1)、按渲染 locale 格式化的运行
|
|
153
153
|
* meta(latestStartedAt 为 null 时内置「暂无运行」文案)与品牌行(等同 PoweredBy,恒含、
|
|
154
154
|
* 无拆除 prop);text 面输出标题行与 meta 行,不含品牌行
|
|
155
|
-
* (docs/feature/reports/
|
|
155
|
+
* (docs/feature/reports/components/site/hero-card.md)。
|
|
156
156
|
*/
|
|
157
157
|
export const HeroCard = defineComponent<HeroCardProps>({
|
|
158
158
|
web: (props, ctx) => {
|
|
@@ -177,7 +177,7 @@ export interface HeroCardProps {
|
|
|
177
177
|
* `Hero`:页首的站点标题区——标题、最后运行时间、快照合成来源,恒含品牌行。官方组合组件,
|
|
178
178
|
* 与手写 `<HeroCard title={title ?? ctx.report.title} data={await heroData(ctx.scope)} />`
|
|
179
179
|
* 严格等价、没有私有能力;读 `ctx.report` 意味着输出跟随站点,要站点无关的标题区直接用
|
|
180
|
-
* `HeroCard` 显式传值(docs/feature/reports/
|
|
180
|
+
* `HeroCard` 显式传值(docs/feature/reports/components/site/hero.md)。
|
|
181
181
|
*/
|
|
182
182
|
export const Hero = defineComponent<HeroProps>(async ({ title, className }, ctx) => (
|
|
183
183
|
<HeroCard title={title ?? ctx.report.title} data={await heroData(ctx.scope)} className={className} />
|
|
@@ -188,7 +188,7 @@ Hero.displayName = "Hero";
|
|
|
188
188
|
* `PoweredBy`:唯一的品牌件,无 props 双面组件。web 面渲染指向 niceeval 官网的一行品牌色
|
|
189
189
|
* 小字(`utm_source=report&utm_medium=powered-by`,`rel` 仅 `noopener` 以保留 Referer);
|
|
190
190
|
* text 面零输出。没有任何配置——品牌契约是「提供一个组件,不给开关」:不想要品牌就不用
|
|
191
|
-
* 这些组件、自己写替代组件(docs/feature/reports/
|
|
191
|
+
* 这些组件、自己写替代组件(docs/feature/reports/components/site/powered-by.md)。
|
|
192
192
|
*/
|
|
193
193
|
export const PoweredBy = defineComponent<Record<never, never>>({
|
|
194
194
|
web: () => <PoweredByWeb />,
|
|
@@ -204,7 +204,7 @@ export type ScopeWarningsProps = DataProps<readonly ScopeWarning[], Record<never
|
|
|
204
204
|
* 按「下一步动作」聚合渲染(带 experimentId 的按实验聚合、非实验作用域按 kind 聚合;
|
|
205
205
|
* 实验作用域组在前、非实验作用域组在后);web 面组头带去重后的可复制命令、明细收原生
|
|
206
206
|
* `<details>`(总条数 ≤ 3 默认展开),text 面同构但不折叠。空警告集与裸 `Snapshot[]` 输入
|
|
207
|
-
* 两面零输出(docs/feature/reports/
|
|
207
|
+
* 两面零输出(docs/feature/reports/components/site/sample-warnings.md)。
|
|
208
208
|
*/
|
|
209
209
|
export const ScopeWarnings = makeDataComponent<readonly ScopeWarning[], Record<never, never>, ChromeProps>({
|
|
210
210
|
name: "ScopeWarnings",
|
|
@@ -229,7 +229,7 @@ export type SnapshotDiagnosticsProps = DataProps<SnapshotDiagnosticsData, Record
|
|
|
229
229
|
* web 面整个区域是默认折叠的原生 `<details>`,`<summary>` 恒可见汇总涉及的 experiment 数、
|
|
230
230
|
* Snapshot 数、记录数(按 count 计数)与最高严重度;单诊断快照退化成一行,不摆空壳层级;
|
|
231
231
|
* text 面同构但不折叠。空诊断集两面零输出
|
|
232
|
-
* (docs/feature/reports/
|
|
232
|
+
* (docs/feature/reports/components/site/run-diagnostics.md)。
|
|
233
233
|
*/
|
|
234
234
|
export const SnapshotDiagnostics = makeDataComponent<SnapshotDiagnosticsData, Record<never, never>, ChromeProps>({
|
|
235
235
|
name: "SnapshotDiagnostics",
|
|
@@ -252,7 +252,7 @@ export type CopyFixPromptProps = DataProps<CopyFixPromptData, Record<never, neve
|
|
|
252
252
|
* `CopyFixPrompt`:把当前范围的全部失败整理成一段可交给 coding agent 的修复 prompt。
|
|
253
253
|
* prompt 在 resolve 阶段算好、烘进静态 HTML,无 JS 时在折叠块里完整可读,「复制」是增强层
|
|
254
254
|
* 行为;`failures` 为 0 时两面零输出;text 面恒零输出——终端里的等价能力是 `show` 的
|
|
255
|
-
* attempt 下钻命令本身(docs/feature/reports/
|
|
255
|
+
* attempt 下钻命令本身(docs/feature/reports/components/site/copy-fix-prompt.md)。
|
|
256
256
|
*/
|
|
257
257
|
export const CopyFixPrompt = makeDataComponent<CopyFixPromptData, Record<never, never>, ChromeProps>({
|
|
258
258
|
name: "CopyFixPrompt",
|
|
@@ -280,7 +280,7 @@ export type TraceWaterfallProps = DataProps<
|
|
|
280
280
|
* 的原始 span(agent / model / tool)。web 面静态渲染顶层 span 分解条(失败 span 带失败
|
|
281
281
|
* 标记),行链接 attempt 详情;text 面每 attempt 一行(locator、总耗时、span 计数与失败
|
|
282
282
|
* 标记)+ 可复制的 `--timing` 下钻命令。trace 缺失的行照常出现并如实显示缺失;runner
|
|
283
|
-
* 生命周期节点不进瀑布(docs/feature/reports/
|
|
283
|
+
* 生命周期节点不进瀑布(docs/feature/reports/components/site/trace-waterfall.md)。
|
|
284
284
|
*/
|
|
285
285
|
export const TraceWaterfall = makeDataComponent<
|
|
286
286
|
readonly TraceWaterfallRow[],
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// ScopeWarnings 的聚合层:把 Scope 警告按「下一步动作」组织成组,web / text 两面共用
|
|
2
|
-
// (docs/feature/reports/
|
|
2
|
+
// (docs/feature/reports/components/site/sample-warnings.md「聚合轴是动作,不是发生顺序」)。
|
|
3
3
|
// message 是完整叙述的单源,这里只组织、不改写;徽标 / 组头文案按 kind 表登记的模板
|
|
4
|
-
// (docs/feature/
|
|
4
|
+
// (docs/feature/record/library.md「警告 kind 全集」)经 locale 词典渲染,未登记的 kind
|
|
5
5
|
// 回退为单独成组、逐条 message 原样。
|
|
6
6
|
|
|
7
7
|
import type { ScopeWarning } from "../../../results/types.ts";
|
|
@@ -74,7 +74,7 @@ function dedupeCommand(members: readonly AnyWarning[]): string | null {
|
|
|
74
74
|
* 其余(含未登记 kind)按 kind 聚合。组排序:实验作用域组在前(按实验 id 字典序),
|
|
75
75
|
* 非实验作用域组在后(按 kind);类别两档制(integrity / freshness)已随 stale-snapshot /
|
|
76
76
|
* partial-coverage 一并删除——三种 warning kind 都是同一类完整性事实,不再需要档位区分
|
|
77
|
-
* (docs/feature/reports/
|
|
77
|
+
* (docs/feature/reports/components/site/sample-warnings.md「聚合轴是动作,不是发生顺序」)。
|
|
78
78
|
*/
|
|
79
79
|
export function groupScopeWarnings(input: readonly ScopeWarning[], locale: ReportLocale): GroupedScopeWarnings {
|
|
80
80
|
const warnings = input as readonly AnyWarning[];
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// SnapshotDiagnostics 的聚合层:把 snapshotDiagnosticsData 的产物按来源 experiment 分组,
|
|
2
|
-
// web / text 两面共用(docs/feature/reports/
|
|
2
|
+
// web / text 两面共用(docs/feature/reports/components/site/run-diagnostics.md)。
|
|
3
3
|
// 输入已经由 snapshotDiagnosticsData 按 experiment id 字典序、组内 startedAt 新到旧排好序,
|
|
4
4
|
// 这里只按 experimentId 分桶,不重新排序、不跨来源合并 DiagnosticRecord。
|
|
5
5
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// ScopeSummary:一个范围的摘要卡——快照时间窗、experiment / eval / attempt 数、判定计票、
|
|
2
2
|
// 端到端通过率与总成本。data 恒携带 eval 级与 attempt 级两份计票;呈现 prop `votes`
|
|
3
3
|
// 只选择显示哪一级(默认 "eval"),不改变 data。通过率与总成本只渲染算好的 MetricCell,
|
|
4
|
-
// 不现场重算(docs/feature/reports/
|
|
4
|
+
// 不现场重算(docs/feature/reports/components/summaries/sample-summary.md)。
|
|
5
5
|
|
|
6
6
|
import type { ReactElement } from "react";
|
|
7
7
|
import type { ScopeSummaryData, VerdictTally } from "../../model/types.ts";
|
|
@@ -45,7 +45,7 @@ export function ScopeSummary({
|
|
|
45
45
|
<div className={cx("nre", "nre-scope-summary", className)} data-votes={votes}>
|
|
46
46
|
<dl className="nre-scope-kpis">
|
|
47
47
|
{/* 计分制 Scope 隐藏通过率只显示总分;混型 Scope 两者都显示;纯通过制 Scope 只显示
|
|
48
|
-
通过率(现状不变),见 docs/feature/reports/
|
|
48
|
+
通过率(现状不变),见 docs/feature/reports/components/summaries/sample-summary.md。 */}
|
|
49
49
|
{data.scoringComposition !== "points" && (
|
|
50
50
|
<div className="nre-scope-kpi nre-scope-kpi-rate">
|
|
51
51
|
<dt>{localeText(locale, "scopeSummary.passRate")}</dt>
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// 计算函数(*Data):ReportInput → 一份组件数据。ScopeSummary 的 scopeSummaryData 住在这里
|
|
2
|
-
// (docs/feature/reports/
|
|
2
|
+
// (docs/feature/reports/components/summaries/README.md)。ExperimentComparison 是纯组合组件,不产生
|
|
3
3
|
// 独立的 data,没有对应的计算函数。
|
|
4
4
|
//
|
|
5
5
|
// 共同约定(docs/feature/reports/architecture.md「指标聚合不变量」):
|
|
@@ -26,7 +26,7 @@ const totalCostMetric = defineMetric({
|
|
|
26
26
|
|
|
27
27
|
/**
|
|
28
28
|
* `scopeSummaryData(input)`:范围摘要——快照时间窗、experiment / eval / attempt 数、
|
|
29
|
-
* 两级判定计票、端到端通过率与总成本(docs/feature/reports/
|
|
29
|
+
* 两级判定计票、端到端通过率与总成本(docs/feature/reports/components/summaries/sample-summary.md)。
|
|
30
30
|
* data 恒携带两级计票;通过率来自官方两级指标引擎,不从任一计票重算。
|
|
31
31
|
*/
|
|
32
32
|
export async function scopeSummaryData(input: ReportInput): Promise<ScopeSummaryData> {
|
|
@@ -18,7 +18,7 @@ export function scopeSummaryText(data: ScopeSummaryData, votes: "eval" | "attemp
|
|
|
18
18
|
const tally = votes === "attempt" ? data.attemptVerdicts : data.evalVerdicts;
|
|
19
19
|
const head = [
|
|
20
20
|
// 计分制 Scope 隐藏通过率只显示总分;混型 Scope 两者都显示;纯通过制 Scope 只显示通过率
|
|
21
|
-
// (现状不变),见 docs/feature/reports/
|
|
21
|
+
// (现状不变),见 docs/feature/reports/components/summaries/sample-summary.md。
|
|
22
22
|
...(data.scoringComposition !== "points"
|
|
23
23
|
? [`${localeText(locale, "scopeSummary.passRate")} ${cellText(data.endToEndPassRate, locale)}`]
|
|
24
24
|
: []),
|
|
@@ -127,7 +127,7 @@ function filterInputBySnapshot(input: ReportInput, predicate: (snapshot: Snapsho
|
|
|
127
127
|
* 「同引用 input + 深相等 spec」记忆化保证。主读数按 scoringComposition(input) 切换:
|
|
128
128
|
* "pass" 用 endToEndPassRate、"points" 用 totalScore;"mixed" 按题型把 input 拆成两个
|
|
129
129
|
* 子 Scope,散点与 ExperimentList 每组各一份、各用各的主读数,ScopeSummary 始终是整个
|
|
130
|
-
* input 一份(docs/feature/reports/
|
|
130
|
+
* input 一份(docs/feature/reports/components/summaries/experiment-comparison.md)。
|
|
131
131
|
*/
|
|
132
132
|
export const ExperimentComparison = defineComponent<ExperimentComparisonProps>(async (props, ctx) => {
|
|
133
133
|
const input = props.input ?? ctx.scope;
|
package/src/report/index.ts
CHANGED
|
@@ -163,7 +163,7 @@ export type {
|
|
|
163
163
|
TraceWaterfallProps,
|
|
164
164
|
} from "./components/site-components/index.tsx";
|
|
165
165
|
|
|
166
|
-
// Attempt 详情组件族(docs/feature/reports/
|
|
166
|
+
// Attempt 详情组件族(docs/feature/reports/components/attempt-detail/README.md):11 个叶子 + 2 个
|
|
167
167
|
// 只装配叶子的组合组件(AttemptAssessment / AttemptDetail),都从 niceeval/report 导出。
|
|
168
168
|
export {
|
|
169
169
|
AttemptAssertions,
|
|
@@ -32,7 +32,7 @@ const KEY_SEP = "\u0000";
|
|
|
32
32
|
* `ReportInput` 归一化:Scope 分支直接消费它已经按口径物化好的 `attempts`(不再重新
|
|
33
33
|
* flatten `snapshots`——真实 Snapshot 各自持有完整、未按 Scope 口径收窄的 evals/attempts,
|
|
34
34
|
* 只有 `Scope.attempts` 才是这次选择的真正结果);裸 `Snapshot[]` 分支没有挑选过程,取全部
|
|
35
|
-
* (docs/feature/
|
|
35
|
+
* (docs/feature/record/library.md「官方现刻水位」)。
|
|
36
36
|
*/
|
|
37
37
|
export function resolveInput(input: ReportInput): {
|
|
38
38
|
snapshots: readonly Snapshot[];
|
|
@@ -70,7 +70,7 @@ export function evalIdOf(item: Item): string {
|
|
|
70
70
|
}
|
|
71
71
|
|
|
72
72
|
/**
|
|
73
|
-
* 历史执行判定(docs/feature/
|
|
73
|
+
* 历史执行判定(docs/feature/sample/library.md「时效:新执行与历史执行」):携带条目,或
|
|
74
74
|
* 真实来源(`item.snapshot`)早于该实验在这份输入里的水位基准(`item.watermark`)——
|
|
75
75
|
* latest() 口径下二者是同一个对象,比较恒假,只剩 carried 生效;current() 口径下贡献
|
|
76
76
|
* 来源可能有多个,水位基准是其中 startedAt 最新的一个。
|
|
@@ -231,7 +231,7 @@ export function formatReportDateTimeRange(
|
|
|
231
231
|
* 历史执行的紧凑时距("3d" / "2h" / "5m" / "10s"):自 `startedAt` 起算,渲染时刻由调用方
|
|
232
232
|
* 传入(`nowIso` 缺省当前时刻)——粒度阈值复用 `gapParts`(与曾经的 stale-snapshot message
|
|
233
233
|
* 同一套单源,见 `results/select.ts`),只是这里的呈现是紧凑单字母,不是完整单词
|
|
234
|
-
* (docs/feature/reports/
|
|
234
|
+
* (docs/feature/reports/components/entity-lists/README.md「时效标注」)。
|
|
235
235
|
*/
|
|
236
236
|
export function formatHistoricalGap(startedAtIso: string, nowIso: string = new Date().toISOString()): string {
|
|
237
237
|
const { n, unit } = gapParts(startedAtIso, nowIso);
|
|
@@ -266,7 +266,7 @@ export function fitFailureSummary(summary: string, maxChars: number): string {
|
|
|
266
266
|
|
|
267
267
|
/**
|
|
268
268
|
* 一份 `ExperimentList` data 的题型构成:主读数列该显示 Pass rate、Total score,还是两者
|
|
269
|
-
* 并存(docs/feature/reports/
|
|
269
|
+
* 并存(docs/feature/reports/components/entity-lists/experiment-list.md 主读数列)。与
|
|
270
270
|
* `entity-lists/compute.ts` 里 `experimentListData` 默认排序专用的 `listScoringComposition`
|
|
271
271
|
* 同一套判据——跳过 `attempts === 0` 的行(coverage-only 占位,`scoring` 是占位默认值不是
|
|
272
272
|
* 读到的事实,一屏占位行不该把纯计分制列表误判成 mixed)。web 面与 text 面在这里读同一份
|
|
@@ -16,8 +16,8 @@
|
|
|
16
16
|
// assistantTurns(assistant-turns) null 实测;o11y 缺失→null 同左 同左 lower
|
|
17
17
|
// repeatedFailedCommands(repeated-failed-commands) null 实测;o11y 缺失→null 同左 同左 lower
|
|
18
18
|
//
|
|
19
|
-
// bounds(自然边界,驱动图轴呼吸边距的钳制,见 docs/feature/reports/
|
|
20
|
-
//
|
|
19
|
+
// bounds(自然边界,驱动图轴呼吸边距的钳制,见 docs/feature/reports/components/charts/README.md
|
|
20
|
+
// 「值域」):三个通过率指标与 examScore 是 { min: 0, max: 1 };其余七个(totalScore、
|
|
21
21
|
// durationMs、tokens、costUSD、assistantTurns、repeatedFailedCommands)是 { min: 0 }。
|
|
22
22
|
//
|
|
23
23
|
// 两档指标(docs/feature/reports/library/metrics.md「内置指标」):以上除 assistantTurns 与
|
|
@@ -192,7 +192,7 @@ export const tokens = defineMetric({
|
|
|
192
192
|
if (a.result.verdict === "skipped") return null;
|
|
193
193
|
const usage = a.result.usage;
|
|
194
194
|
// input/output 缺失(协议没提供)→ null,不拿 0 冒充「实测就是 0」
|
|
195
|
-
// (docs/feature/
|
|
195
|
+
// (docs/feature/record/architecture.md#usage)。
|
|
196
196
|
if (!usage || usage.inputTokens === undefined || usage.outputTokens === undefined) return null;
|
|
197
197
|
// 只加 input + output:缓存读写量大但便宜,计进去会把缓存热的 agent 画成 token 大户;
|
|
198
198
|
// 花钱多少本来就有 costUSD 负责。
|
|
@@ -67,7 +67,7 @@ export interface Metric<Name extends string = string> {
|
|
|
67
67
|
/**
|
|
68
68
|
* 指标值的自然边界(如通过率 0–1、成本下界 0)。图轴呼吸边距不越过声明的边界——
|
|
69
69
|
* 贴边数据点如实落在框线上(如通过率 100%),那是指标的自然边界,不是裁剪
|
|
70
|
-
* (docs/feature/reports/
|
|
70
|
+
* (docs/feature/reports/components/charts/README.md「值域」)。
|
|
71
71
|
*/
|
|
72
72
|
bounds?: { min?: number; max?: number };
|
|
73
73
|
/**
|
|
@@ -152,7 +152,7 @@ export interface MetricColumn {
|
|
|
152
152
|
unit?: string;
|
|
153
153
|
/** 渲染提示:排序方向、轴向、涨跌配色。 */
|
|
154
154
|
better?: "higher" | "lower";
|
|
155
|
-
/** = metric.bounds,原样投影;图轴值域推定读这里(docs/feature/reports/
|
|
155
|
+
/** = metric.bounds,原样投影;图轴值域推定读这里(docs/feature/reports/components/charts/README.md「值域」)。 */
|
|
156
156
|
bounds?: { min?: number; max?: number };
|
|
157
157
|
}
|
|
158
158
|
|
|
@@ -176,7 +176,7 @@ export interface MetricCell {
|
|
|
176
176
|
}
|
|
177
177
|
|
|
178
178
|
/**
|
|
179
|
-
* 数据形状的字段命名规则(docs/feature/reports/
|
|
179
|
+
* 数据形状的字段命名规则(docs/feature/reports/components/tables/README.md「共用数据形状」):
|
|
180
180
|
* 维度名字段 = 产生它的选项名 + `Dimension` 后缀,值是解析后的维度 name;
|
|
181
181
|
* 条目数组一律叫 `rows`(Matrix 的稀疏格子叫 `cells`);条目内的 key / series 是维度值,不带后缀。
|
|
182
182
|
*/
|
|
@@ -262,8 +262,7 @@ export interface ScoreboardData {
|
|
|
262
262
|
}
|
|
263
263
|
|
|
264
264
|
/**
|
|
265
|
-
* `DeltaTable` 的一格:同一条件值 × eval 的折叠(docs/feature/reports/
|
|
266
|
-
* 「DeltaTable」)。`verdict` / `totalScore` 用与榜单同一套题目级判定口径(`totalScore` 取各
|
|
265
|
+
* `DeltaTable` 的一格:同一条件值 × eval 的折叠(docs/feature/reports/components/tables/delta-table.md)。`verdict` / `totalScore` 用与榜单同一套题目级判定口径(`totalScore` 取各
|
|
267
266
|
* attempt 的均值);`totalTokens` / `totalCostUSD` 是该题在该条件下全部 attempt 的**合计**,
|
|
268
267
|
* 不是均值。
|
|
269
268
|
*/
|
|
@@ -377,7 +376,7 @@ export type ScoringComposition = "pass" | "points" | "mixed";
|
|
|
377
376
|
/**
|
|
378
377
|
* 一个范围的摘要:快照时间窗、experiment / eval / attempt 数、两级判定计票、端到端通过率
|
|
379
378
|
* 和总成本。eval 的身份键是 experimentId + evalId;data 恒携带两级计票,渲染面显示哪一级
|
|
380
|
-
* 由呈现 prop `votes` 决定,不改变 data(docs/feature/reports/
|
|
379
|
+
* 由呈现 prop `votes` 决定,不改变 data(docs/feature/reports/components/summaries/sample-summary.md)。
|
|
381
380
|
*/
|
|
382
381
|
export interface ScopeSummaryData {
|
|
383
382
|
/** 贡献当前数据的快照时间范围;空范围为 null,不编造当前时间。 */
|
|
@@ -409,7 +408,7 @@ export interface ScopeSummaryData {
|
|
|
409
408
|
// ───────────────────────── 站点组件(Hero / CopyFixPrompt / TraceWaterfall)─────────────────────────
|
|
410
409
|
|
|
411
410
|
/**
|
|
412
|
-
* `HeroCard` 的数据(docs/feature/reports/
|
|
411
|
+
* `HeroCard` 的数据(docs/feature/reports/components/site/hero-card.md):站点标题区的
|
|
413
412
|
* 运行 meta——最后运行时间与快照合成来源。标题不在 data 里,它是站点声明与 Scope 的合成物,
|
|
414
413
|
* 经 `HeroCardProps.title` 传入。
|
|
415
414
|
*/
|
|
@@ -421,7 +420,7 @@ export interface HeroData {
|
|
|
421
420
|
}
|
|
422
421
|
|
|
423
422
|
/**
|
|
424
|
-
* `SnapshotDiagnostics` 一条来源快照的诊断投影(docs/feature/reports/
|
|
423
|
+
* `SnapshotDiagnostics` 一条来源快照的诊断投影(docs/feature/reports/components/site/run-diagnostics.md):
|
|
425
424
|
* 只携带 experimentId / startedAt / DiagnosticRecord,不带 Snapshot 本体、`evals` 或
|
|
426
425
|
* `AttemptHandle`,避免把文件读取能力拖进浏览器边界。
|
|
427
426
|
*/
|
|
@@ -439,7 +438,7 @@ export type SnapshotDiagnosticsData = readonly SnapshotDiagnosticsItem[];
|
|
|
439
438
|
|
|
440
439
|
/**
|
|
441
440
|
* `CopyFixPrompt` 的数据:resolve 期算好的修复 prompt 全文与参与的失败数
|
|
442
|
-
* (docs/feature/reports/
|
|
441
|
+
* (docs/feature/reports/components/site/copy-fix-prompt.md)。
|
|
443
442
|
*/
|
|
444
443
|
export interface CopyFixPromptData {
|
|
445
444
|
/** 修复 prompt 全文;失败逐条含 eval id、主失败摘要与 attempt 下钻命令。 */
|
|
@@ -577,7 +576,7 @@ export interface ExperimentListItem {
|
|
|
577
576
|
|
|
578
577
|
// ───────────────────────── Attempt 详情组件族 ─────────────────────────
|
|
579
578
|
//
|
|
580
|
-
// 11 个叶子组件的 data 契约(docs/feature/reports/
|
|
579
|
+
// 11 个叶子组件的 data 契约(docs/feature/reports/components/attempt-detail/README.md)。每个都由
|
|
581
580
|
// 同名 `attempt*Data(evidence: AttemptEvidence)` 同步派生,不读文件、不 fetch——
|
|
582
581
|
// loadAttemptEvidence 已经一次性装配好全部证据。`AttemptSummary` 恒非空;其余在对应
|
|
583
582
|
// 能力位为空时函数返回 null,两面渲染为空输出。
|
|
@@ -745,7 +744,7 @@ export interface AttemptConversationData {
|
|
|
745
744
|
locator: AttemptLocator;
|
|
746
745
|
rounds: AttemptConversationRound[];
|
|
747
746
|
/**
|
|
748
|
-
* `commands.json` 的投影(见 docs/feature/
|
|
747
|
+
* `commands.json` 的投影(见 docs/feature/record/architecture.md#commandsjson):按关联
|
|
749
748
|
* timing 节点(`timingNodeId` 对应 `PhaseTiming.children` 中 `kind === "command"` 的节点)
|
|
750
749
|
* 的 `startOffsetMs` 排序;关联不到 timing 节点的排在最后,不按数组偶然顺序猜时间。没有失败
|
|
751
750
|
* 命令时字段省略,不摆空数组。
|
|
@@ -760,14 +759,14 @@ export interface AttemptDiagnosticsData {
|
|
|
760
759
|
|
|
761
760
|
/**
|
|
762
761
|
* `UsageTable` 的 data:判定、轮数、工具调用数、token 拆分与成本摊成的单行用量摘要;组装口径单源
|
|
763
|
-
* 见 docs/feature/reports/
|
|
762
|
+
* 见 docs/feature/reports/components/attempt-detail/usage-table.md#组装口径单源。identity 字段
|
|
764
763
|
* (`locator`/`experimentId`/`evalId`/`attempt`/`verdict`)恒有——它们不是「usage 有没有」的一部分,
|
|
765
764
|
* 是这一行归属哪个 attempt 的身份。其余字段各自独立地只在事实真实存在时出现:
|
|
766
765
|
*
|
|
767
766
|
* - `turns`/`toolCalls`:events 派生(与 `o11y.json` 行为摘要同源),只在有非空 events 时出现,
|
|
768
767
|
* 哪怕派生值恰好是 0(有 events 但零轮/零工具调用是观测到的事实,不是缺失)。
|
|
769
768
|
* - `usage`:落盘 `Usage` 原样,只在 `result.usage` 存在时出现。桶恒互斥,`inputTokens` 本身
|
|
770
|
-
* 就是未缓存输入(契约见 docs/feature/
|
|
769
|
+
* 就是未缓存输入(契约见 docs/feature/record/architecture.md#usage);"uncached in" 标注只在
|
|
771
770
|
* `cacheReadTokens` 在场时由 face 层给出,不派生第二个字段。
|
|
772
771
|
* - `estimatedCostUSD`:能算出成本(`usage.costUSD` 或 `result.estimatedCostUSD`)时才出现。
|
|
773
772
|
*
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// cases: docs/engineering/testing/unit/
|
|
1
|
+
// cases: docs/engineering/testing/unit/record.md
|
|
2
2
|
// buildAnnotatedEvalSource 的单测(定稿见 docs/concepts.md「标注 Eval 源码」)。
|
|
3
3
|
// 覆盖:同一行多条断言、无 loc 断言进 unmapped、loc 指向别的文件/越界行也进 unmapped
|
|
4
4
|
// (never silently dropped 的边界情况)、空断言数组、summary 计数、哈希与归一化行为。
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// cases: docs/engineering/testing/unit/
|
|
1
|
+
// cases: docs/engineering/testing/unit/record.md
|
|
2
2
|
// loadAttemptEvidence 单测(定稿契约见 plan/attempt-evidence-feedback-loop.md「中性数据准备」、
|
|
3
3
|
// src/results/attempt-evidence.ts 的头注)。用真实 createResultsWriter → openResults 的读写链路
|
|
4
4
|
// 落一份最小 fixture(不手写 JSON 文件,理由同 loadAnnotatedEvalSource 的既有端到端测试:
|