niceeval 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INDEX.md +31 -33
- package/dist/agents/types.d.ts +20 -17
- package/dist/context/types.d.ts +21 -21
- package/dist/i18n/en.d.ts +10 -2
- package/dist/i18n/zh-CN.d.ts +15 -7
- package/dist/o11y/execution-tree.d.ts +103 -0
- package/dist/o11y/otlp/select.d.ts +22 -0
- package/dist/report/{react → assets}/colors.d.ts +8 -0
- package/dist/report/{react → assets}/colors.js +23 -0
- package/dist/report/built-in/index.d.ts +2 -2
- package/dist/report/built-in/index.js +2 -2
- package/dist/report/built-in/standard.d.ts +8 -1
- package/dist/report/built-in/standard.js +17 -7
- package/dist/report/components/attempt-detail/AttemptAssertions.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptAssertions.js +17 -0
- package/dist/report/components/attempt-detail/AttemptConversation.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptConversation.js +34 -0
- package/dist/report/components/attempt-detail/AttemptDiagnostics.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptDiagnostics.js +7 -0
- package/dist/report/components/attempt-detail/AttemptDiff.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptDiff.js +11 -0
- package/dist/report/components/attempt-detail/AttemptError.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptError.js +16 -0
- package/dist/report/components/attempt-detail/AttemptFixPrompt.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptFixPrompt.js +7 -0
- package/dist/report/components/attempt-detail/AttemptSource.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptSource.js +27 -0
- package/dist/report/components/attempt-detail/AttemptSummary.d.ts +8 -0
- package/dist/report/components/attempt-detail/AttemptSummary.js +20 -0
- package/dist/report/components/attempt-detail/AttemptTimeline.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptTimeline.js +28 -0
- package/dist/report/components/attempt-detail/AttemptTrace.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptTrace.js +29 -0
- package/dist/report/components/attempt-detail/AttemptUsage.d.ts +6 -0
- package/dist/report/components/attempt-detail/AttemptUsage.js +20 -0
- package/dist/report/components/attempt-detail/compute.d.ts +24 -0
- package/dist/report/components/attempt-detail/compute.js +254 -0
- package/dist/report/components/attempt-detail/faces.d.ts +14 -0
- package/dist/report/components/attempt-detail/faces.js +235 -0
- package/dist/report/components/attempt-detail/index.d.ts +38 -0
- package/dist/report/components/attempt-detail/index.js +527 -0
- package/dist/report/{react → components}/cell.d.ts +5 -3
- package/dist/report/{react → components}/cell.js +3 -3
- package/dist/report/{react → components/entity-lists}/AttemptList.d.ts +8 -4
- package/dist/report/{react → components/entity-lists}/AttemptList.js +16 -10
- package/dist/report/{react → components/entity-lists}/EvalList.d.ts +3 -3
- package/dist/report/{react → components/entity-lists}/EvalList.js +0 -0
- package/dist/report/{react → components/entity-lists}/ExperimentList.d.ts +3 -3
- package/dist/report/{react → components/entity-lists}/ExperimentList.js +10 -10
- package/dist/report/components/entity-lists/compute.d.ts +23 -0
- package/dist/report/components/entity-lists/compute.js +171 -0
- package/dist/report/components/entity-lists/faces.d.ts +5 -0
- package/dist/report/components/entity-lists/faces.js +174 -0
- package/dist/report/components/entity-lists/index.d.ts +51 -0
- package/dist/report/components/entity-lists/index.js +165 -0
- package/dist/report/{react → components}/fixtures.d.ts +2 -2
- package/dist/report/{react → components}/fixtures.js +1 -1
- package/dist/report/{react → components/metric-views}/DeltaTable.d.ts +2 -2
- package/dist/report/{react → components/metric-views}/DeltaTable.js +4 -4
- package/dist/report/{react → components/metric-views}/MetricBars.d.ts +3 -3
- package/dist/report/{react → components/metric-views}/MetricBars.js +3 -3
- package/dist/report/{react → components/metric-views}/MetricLine.d.ts +2 -2
- package/dist/report/{react → components/metric-views}/MetricLine.js +8 -5
- package/dist/report/{react → components/metric-views}/MetricMatrix.d.ts +3 -3
- package/dist/report/{react → components/metric-views}/MetricMatrix.js +4 -4
- package/dist/report/components/metric-views/MetricScatter.d.ts +11 -0
- package/dist/report/{react → components/metric-views}/MetricScatter.js +26 -23
- package/dist/report/{react → components/metric-views}/MetricTable.d.ts +3 -3
- package/dist/report/{react → components/metric-views}/MetricTable.js +4 -4
- package/dist/report/{react → components/metric-views}/Scoreboard.d.ts +2 -2
- package/dist/report/{react → components/metric-views}/Scoreboard.js +3 -3
- package/dist/report/components/metric-views/compute.d.ts +89 -0
- package/dist/report/{compute.js → components/metric-views/compute.js} +32 -415
- package/dist/report/components/metric-views/faces.d.ts +13 -0
- package/dist/report/components/metric-views/faces.js +381 -0
- package/dist/report/components/metric-views/index.d.ts +50 -0
- package/dist/report/components/metric-views/index.js +273 -0
- package/dist/report/{text → components/metric-views}/plot.js +1 -1
- package/dist/report/components/shared-compute.d.ts +22 -0
- package/dist/report/components/shared-compute.js +47 -0
- package/dist/report/components/shared-faces.d.ts +9 -0
- package/dist/report/components/shared-faces.js +26 -0
- package/dist/report/components/shared.d.ts +68 -0
- package/dist/report/components/shared.js +125 -0
- package/dist/report/{react → components/site-components}/CopyFixPrompt.d.ts +2 -2
- package/dist/report/{react → components/site-components}/CopyFixPrompt.js +2 -2
- package/dist/report/{react → components/site-components}/HeroCard.d.ts +2 -2
- package/dist/report/{react → components/site-components}/HeroCard.js +2 -2
- package/dist/report/{react → components/site-components}/ScopeWarnings.d.ts +2 -2
- package/dist/report/{react → components/site-components}/ScopeWarnings.js +3 -3
- package/dist/report/{react → components/site-components}/TraceWaterfall.d.ts +3 -3
- package/dist/report/{react → components/site-components}/TraceWaterfall.js +5 -5
- package/dist/report/components/site-components/compute.d.ts +28 -0
- package/dist/report/components/site-components/compute.js +132 -0
- package/dist/report/components/site-components/faces.d.ts +21 -0
- package/dist/report/components/site-components/faces.js +75 -0
- package/dist/report/components/site-components/index.d.ts +74 -0
- package/dist/report/components/site-components/index.js +220 -0
- package/dist/report/{scope-warnings.d.ts → components/site-components/scope-warnings.d.ts} +2 -2
- package/dist/report/{scope-warnings.js → components/site-components/scope-warnings.js} +2 -2
- package/dist/report/{react → components/summaries}/ScopeSummary.d.ts +2 -2
- package/dist/report/{react → components/summaries}/ScopeSummary.js +17 -8
- package/dist/report/components/summaries/compute.d.ts +7 -0
- package/dist/report/components/summaries/compute.js +51 -0
- package/dist/report/components/summaries/faces.d.ts +7 -0
- package/dist/report/components/summaries/faces.js +38 -0
- package/dist/report/components/summaries/index.d.ts +27 -0
- package/dist/report/components/summaries/index.js +78 -0
- package/dist/report/definition/grid-layout.d.ts +50 -0
- package/dist/report/definition/grid-layout.js +89 -0
- package/dist/report/{primitives.d.ts → definition/primitives.d.ts} +34 -4
- package/dist/report/{primitives.js → definition/primitives.js} +101 -10
- package/dist/report/{report.d.ts → definition/report.d.ts} +36 -65
- package/dist/report/{report.js → definition/report.js} +58 -90
- package/dist/report/{text/table.d.ts → definition/table-text.d.ts} +2 -2
- package/dist/report/{text/table.js → definition/table-text.js} +2 -2
- package/dist/report/{tree.d.ts → definition/tree.d.ts} +44 -12
- package/dist/report/{tree.js → definition/tree.js} +10 -12
- package/dist/report/index.d.ts +33 -18
- package/dist/report/index.js +23 -14
- package/dist/report/{aggregate.d.ts → model/aggregate.d.ts} +12 -4
- package/dist/report/{aggregate.js → model/aggregate.js} +28 -8
- package/dist/report/{flag.d.ts → model/flag.d.ts} +14 -1
- package/dist/report/{flag.js → model/flag.js} +35 -0
- package/dist/report/{format.d.ts → model/format.d.ts} +11 -5
- package/dist/report/{format.js → model/format.js} +48 -4
- package/dist/report/{locale.d.ts → model/locale.d.ts} +17 -17
- package/dist/report/{locale.js → model/locale.js} +31 -33
- package/dist/report/{metrics.d.ts → model/metrics.d.ts} +2 -2
- package/dist/report/{metrics.js → model/metrics.js} +2 -2
- package/dist/report/{types.d.ts → model/types.d.ts} +140 -20
- package/dist/report/react/index.d.ts +32 -21
- package/dist/report/react/index.js +33 -21
- package/dist/report/{load.d.ts → runtime/load.d.ts} +1 -1
- package/dist/report/{load.js → runtime/load.js} +1 -1
- package/dist/report/runtime/text.d.ts +67 -0
- package/dist/report/runtime/text.js +114 -0
- package/dist/report/runtime/web.d.ts +35 -0
- package/dist/report/runtime/web.js +70 -0
- package/dist/results/annotated-source.d.ts +87 -0
- package/dist/results/attempt-evidence.d.ts +74 -0
- package/dist/results/attempt-source.d.ts +15 -0
- package/dist/results/select.d.ts +12 -2
- package/dist/results/select.js +27 -5
- package/dist/runner/eval-selection.d.ts +26 -0
- package/dist/runner/feedback/sink.d.ts +26 -1
- package/dist/runner/types.d.ts +149 -23
- package/dist/sandbox/docker.d.ts +6 -0
- package/dist/sandbox/types.d.ts +23 -21
- package/dist/scoring/display.d.ts +7 -2
- package/dist/scoring/display.js +42 -4
- package/dist/scoring/types.d.ts +3 -3
- package/dist/shared/aggregate.d.ts +15 -1
- package/dist/shared/aggregate.js +32 -1
- package/dist/shared/types.d.ts +6 -1
- package/docs-site/images/logo-dark.svg +7 -0
- package/docs-site/images/logo.svg +6 -5
- package/docs-site/zh/README.md +8 -7
- package/docs-site/zh/examples/ai-agent-application.mdx +5 -5
- package/docs-site/zh/examples/coding-agent-extensions.mdx +4 -4
- package/docs-site/zh/examples/index.mdx +3 -3
- package/docs-site/zh/examples/integrations/ai-sdk-v7.mdx +1 -1
- package/docs-site/zh/examples/integrations/claude-sdk.mdx +3 -3
- package/docs-site/zh/examples/integrations/codex-sdk.mdx +3 -3
- package/docs-site/zh/examples/integrations/langgraph.mdx +3 -3
- package/docs-site/zh/examples/integrations/pi-sdk.mdx +2 -2
- package/docs-site/zh/explanation/adapter.mdx +19 -19
- package/docs-site/zh/explanation/assert.mdx +9 -9
- package/docs-site/zh/explanation/drive.mdx +4 -4
- package/docs-site/zh/explanation/evals.mdx +8 -8
- package/docs-site/zh/explanation/experiment.mdx +8 -8
- package/docs-site/zh/explanation/hitl.mdx +13 -13
- package/docs-site/zh/explanation/judge.mdx +3 -3
- package/docs-site/zh/explanation/overview.mdx +7 -7
- package/docs-site/zh/explanation/runner.mdx +11 -11
- package/docs-site/zh/explanation/tier.mdx +6 -6
- package/docs-site/zh/index.mdx +14 -14
- package/docs-site/zh/introduction.mdx +13 -16
- package/docs-site/zh/reference/builtin-agents.mdx +61 -28
- package/docs-site/zh/reference/capabilities.mdx +7 -27
- package/docs-site/zh/reference/cli.mdx +35 -32
- package/docs-site/zh/reference/define-agent.mdx +32 -31
- package/docs-site/zh/reference/define-config.mdx +6 -6
- package/docs-site/zh/reference/define-eval.mdx +33 -20
- package/docs-site/zh/reference/events.mdx +4 -4
- package/docs-site/zh/reference/expect.mdx +3 -3
- package/docs-site/zh/reference/official-adapters.mdx +17 -17
- package/docs-site/zh/reference/report-components.mdx +193 -121
- package/docs-site/zh/reference/results-data.mdx +13 -13
- package/docs-site/zh/troubleshooting/debug-sandbox.mdx +11 -11
- package/docs-site/zh/troubleshooting/debugging.mdx +43 -19
- package/docs-site/zh/{how-to → tutorials}/agent-feedback-loop.mdx +38 -38
- package/docs-site/zh/tutorials/agent-onboarding.mdx +100 -0
- package/docs-site/zh/tutorials/authoring.mdx +211 -0
- package/docs-site/zh/{how-to → tutorials}/ci-integration.mdx +8 -8
- package/docs-site/zh/{how-to → tutorials}/connect-otel.mdx +22 -22
- package/docs-site/zh/{how-to → tutorials}/connect-your-agent.mdx +58 -63
- package/docs-site/zh/tutorials/custom-reports.mdx +453 -0
- package/docs-site/zh/{how-to → tutorials}/dataset-fanout.mdx +4 -4
- package/docs-site/zh/tutorials/experiments.mdx +103 -0
- package/docs-site/zh/{how-to → tutorials}/fixtures.mdx +9 -9
- package/docs-site/zh/{how-to → tutorials}/publish-report.mdx +11 -5
- package/docs-site/zh/tutorials/quickstart.mdx +18 -18
- package/docs-site/zh/{how-to → tutorials}/reporters.mdx +6 -6
- package/docs-site/zh/{how-to → tutorials}/sandbox-agent.mdx +6 -6
- package/docs-site/zh/{how-to → tutorials}/sandbox-providers.mdx +41 -19
- package/docs-site/zh/{how-to → tutorials}/scoring-guide.mdx +4 -4
- package/docs-site/zh/{how-to → tutorials}/viewing-results.mdx +59 -50
- package/docs-site/zh/tutorials/write-experiment.mdx +355 -0
- package/docs-site/zh/{how-to → tutorials}/write-send.mdx +40 -40
- package/package.json +8 -4
- package/src/agents/bub.ts +21 -8
- package/src/agents/claude-code.ts +23 -10
- package/src/agents/codex.test.ts +22 -7
- package/src/agents/codex.ts +22 -9
- package/src/agents/index.ts +2 -2
- package/src/agents/openai-compat.ts +1 -1
- package/src/agents/post-setup.ts +45 -22
- package/src/agents/streaming.ts +1 -1
- package/src/agents/types.ts +20 -17
- package/src/agents/ui-message-stream.test.ts +10 -0
- package/src/agents/ui-message-stream.ts +12 -1
- package/src/cli.ts +77 -64
- package/src/context/types.ts +21 -21
- package/src/define.ts +13 -0
- package/src/i18n/en.ts +30 -15
- package/src/i18n/zh-CN.ts +30 -15
- package/src/index.ts +2 -0
- package/src/report/{react → assets}/colors.ts +22 -0
- package/src/report/{react → assets}/enhance.js +1 -33
- package/src/report/{react → assets}/styles.css +136 -112
- package/src/report/built-in/index.tsx +2 -2
- package/src/report/built-in/standard.tsx +18 -6
- package/src/report/components/attempt-detail/AttemptAssertions.tsx +67 -0
- package/src/report/components/attempt-detail/AttemptConversation.tsx +110 -0
- package/src/report/components/attempt-detail/AttemptDiagnostics.tsx +38 -0
- package/src/report/components/attempt-detail/AttemptDiff.tsx +35 -0
- package/src/report/components/attempt-detail/AttemptError.tsx +31 -0
- package/src/report/components/attempt-detail/AttemptFixPrompt.tsx +26 -0
- package/src/report/components/attempt-detail/AttemptSource.tsx +80 -0
- package/src/report/components/attempt-detail/AttemptSummary.tsx +74 -0
- package/src/report/components/attempt-detail/AttemptTimeline.tsx +108 -0
- package/src/report/components/attempt-detail/AttemptTrace.tsx +63 -0
- package/src/report/components/attempt-detail/AttemptUsage.tsx +29 -0
- package/src/report/components/attempt-detail/attempt-components.test.tsx +680 -0
- package/src/report/components/attempt-detail/compute.ts +287 -0
- package/src/report/components/attempt-detail/faces.ts +257 -0
- package/src/report/components/attempt-detail/index.tsx +583 -0
- package/src/report/components/attempt-detail/validate.test.ts +235 -0
- package/src/report/{react → components}/cell.tsx +6 -3
- package/src/report/{report.test.ts → components/compute.test.ts} +316 -46
- package/src/report/{react → components/entity-lists}/AttemptList.tsx +24 -16
- package/src/report/{react → components/entity-lists}/EvalList.tsx +0 -0
- package/src/report/{react → components/entity-lists}/ExperimentList.tsx +23 -20
- package/src/report/components/entity-lists/compute.ts +196 -0
- package/src/report/components/entity-lists/faces.ts +207 -0
- package/src/report/components/entity-lists/index.tsx +241 -0
- package/src/report/components/entity-lists/validate.test.ts +114 -0
- package/src/report/{react → components}/fixtures.ts +2 -2
- package/src/report/{react → components/metric-views}/DeltaTable.tsx +5 -5
- package/src/report/{react → components/metric-views}/MetricBars.tsx +5 -5
- package/src/report/{react → components/metric-views}/MetricLine.tsx +9 -6
- package/src/report/{react → components/metric-views}/MetricMatrix.tsx +6 -6
- package/src/report/{react → components/metric-views}/MetricScatter.tsx +41 -33
- package/src/report/{react → components/metric-views}/MetricTable.tsx +6 -6
- package/src/report/{react → components/metric-views}/Scoreboard.tsx +4 -4
- package/src/report/{compute.ts → components/metric-views/compute.ts} +51 -477
- package/src/report/components/metric-views/faces.ts +423 -0
- package/src/report/components/metric-views/index.tsx +352 -0
- package/src/report/{text → components/metric-views}/plot.ts +1 -1
- package/src/report/components/metric-views/validate.test.ts +211 -0
- package/src/report/{react → components}/render.test.tsx +47 -17
- package/src/report/components/shared-compute.ts +59 -0
- package/src/report/components/shared-faces.ts +30 -0
- package/src/report/components/shared.ts +192 -0
- package/src/report/{react → components/site-components}/CopyFixPrompt.tsx +3 -3
- package/src/report/{react → components/site-components}/HeroCard.tsx +3 -3
- package/src/report/{react → components/site-components}/ScopeWarnings.tsx +4 -4
- package/src/report/{react → components/site-components}/TraceWaterfall.tsx +13 -10
- package/src/report/components/site-components/compute.ts +144 -0
- package/src/report/components/site-components/faces.ts +80 -0
- package/src/report/components/site-components/index.tsx +265 -0
- package/src/report/{scope-warnings.ts → components/site-components/scope-warnings.ts} +3 -3
- package/src/report/{site-components.test.tsx → components/site-components/site-components.test.tsx} +23 -21
- package/src/report/components/site-components/validate.test.ts +89 -0
- package/src/report/{react → components/summaries}/ScopeSummary.tsx +26 -11
- package/src/report/components/summaries/compute.ts +56 -0
- package/src/report/components/summaries/faces.ts +48 -0
- package/src/report/components/summaries/index.tsx +127 -0
- package/src/report/components/summaries/validate.test.ts +49 -0
- package/src/report/definition/grid-layout.test.ts +124 -0
- package/src/report/definition/grid-layout.ts +146 -0
- package/src/report/{primitives.tsx → definition/primitives.tsx} +168 -18
- package/src/report/{report.ts → definition/report.ts} +98 -165
- package/src/report/{shell-head.test.ts → definition/shell-head.test.ts} +2 -2
- package/src/report/{text/table.ts → definition/table-text.ts} +4 -4
- package/src/report/{tree.ts → definition/tree.ts} +58 -22
- package/src/report/index.ts +126 -66
- package/src/report/{aggregate.ts → model/aggregate.ts} +31 -9
- package/src/report/{flag.ts → model/flag.ts} +39 -1
- package/src/report/{format.ts → model/format.ts} +59 -5
- package/src/report/{locale.ts → model/locale.ts} +34 -36
- package/src/report/{metrics.ts → model/metrics.ts} +3 -3
- package/src/report/{types.ts → model/types.ts} +143 -22
- package/src/report/react/index.tsx +52 -26
- package/src/report/{dual-render.test.tsx → runtime/dual-render.test.tsx} +564 -49
- package/src/report/runtime/host.test.ts +44 -0
- package/src/report/runtime/host.ts +138 -0
- package/src/report/{load.ts → runtime/load.ts} +1 -1
- package/src/report/runtime/text.ts +192 -0
- package/src/report/runtime/web.ts +106 -0
- package/src/results/attempt-evidence.ts +5 -1
- package/src/results/host-equivalence.test.ts +54 -3
- package/src/results/select.ts +30 -6
- package/src/runner/attempt.test.ts +48 -1
- package/src/runner/attempt.ts +62 -42
- package/src/runner/cleanup-timeout.test.ts +16 -0
- package/src/runner/cleanup-timeout.ts +23 -0
- package/src/runner/discover.ts +1 -2
- package/src/runner/eval-selection.test.ts +187 -0
- package/src/runner/eval-selection.ts +109 -0
- package/src/runner/experiment-cleanup-registry.test.ts +89 -0
- package/src/runner/experiment-cleanup-registry.ts +39 -0
- package/src/runner/experiment-labels.test.ts +71 -0
- package/src/runner/feedback/agent.test.ts +33 -1
- package/src/runner/feedback/agent.ts +12 -16
- package/src/runner/feedback/ci.test.ts +35 -1
- package/src/runner/feedback/ci.ts +16 -18
- package/src/runner/feedback/coordinator.ts +28 -0
- package/src/runner/feedback/human.test.ts +108 -1
- package/src/runner/feedback/human.ts +71 -21
- package/src/runner/feedback/reducer.test.ts +64 -0
- package/src/runner/feedback/reducer.ts +26 -0
- package/src/runner/feedback/sink.ts +40 -1
- package/src/runner/fingerprint.ts +2 -1
- package/src/runner/report.test.ts +14 -0
- package/src/runner/report.ts +3 -1
- package/src/runner/run.test.ts +390 -12
- package/src/runner/run.ts +259 -26
- package/src/runner/sandbox-selection.test.ts +13 -3
- package/src/runner/sandbox-selection.ts +4 -2
- package/src/runner/types.ts +158 -23
- package/src/sandbox/docker.ts +7 -0
- package/src/sandbox/e2b.ts +12 -13
- package/src/sandbox/types.ts +23 -21
- package/src/scoring/display.test.ts +28 -2
- package/src/scoring/display.ts +38 -5
- package/src/scoring/types.ts +3 -3
- package/src/shared/aggregate.test.ts +52 -0
- package/src/shared/aggregate.ts +30 -1
- package/src/shared/types.ts +6 -1
- package/src/show/command.test.ts +34 -0
- package/src/show/command.ts +16 -0
- package/src/show/compose.ts +1 -1
- package/src/show/index.ts +58 -22
- package/src/show/render.ts +16 -240
- package/src/show/show.test.ts +128 -35
- package/src/view/app/App.test.tsx +8 -7
- package/src/view/app/App.tsx +135 -51
- package/src/view/app/i18n.ts +7 -147
- package/src/view/app/lib/attempt-dialog.test.ts +65 -0
- package/src/view/app/lib/attempt-dialog.ts +69 -0
- package/src/view/app/main.tsx +0 -1
- package/src/view/app/types.ts +3 -86
- package/src/view/artifact-serving.test.ts +22 -38
- package/src/view/client-dist/app.css +1 -1
- package/src/view/client-dist/app.js +15 -22
- package/src/view/data.test.ts +48 -36
- package/src/view/data.ts +122 -89
- package/src/view/index.ts +3 -13
- package/src/view/server.ts +36 -6
- package/src/view/shared/types.ts +6 -31
- package/src/view/site-parity.test.ts +24 -1
- package/src/view/site.ts +144 -17
- package/src/view/styles.css +18 -806
- package/src/view/view-report.test.ts +182 -34
- package/dist/report/components.d.ts +0 -179
- package/dist/report/components.js +0 -544
- package/dist/report/compute.d.ts +0 -139
- package/dist/report/react/ExperimentComparison.d.ts +0 -10
- package/dist/report/react/ExperimentComparison.js +0 -12
- package/dist/report/react/MetricScatter.d.ts +0 -9
- package/dist/report/react/format.d.ts +0 -3
- package/dist/report/react/format.js +0 -7
- package/dist/report/text/faces.d.ts +0 -45
- package/dist/report/text/faces.js +0 -671
- package/dist/report/web.d.ts +0 -32
- package/dist/report/web.js +0 -48
- package/docs-site/zh/how-to/authoring.mdx +0 -162
- package/docs-site/zh/how-to/custom-reports.mdx +0 -414
- package/docs-site/zh/how-to/experiments.mdx +0 -86
- package/docs-site/zh/how-to/write-experiment.mdx +0 -164
- package/src/report/components.tsx +0 -925
- package/src/report/react/ExperimentComparison.tsx +0 -73
- package/src/report/react/format.ts +0 -9
- package/src/report/text/faces.ts +0 -767
- package/src/report/web.ts +0 -77
- package/src/show/report-host.test.ts +0 -205
- package/src/show/report-host.ts +0 -389
- package/src/view/app/components/AttemptModal.tsx +0 -496
- package/src/view/app/components/CodeView.test.tsx +0 -142
- package/src/view/app/components/CodeView.tsx +0 -310
- package/src/view/app/components/CopyControls.tsx +0 -106
- package/src/view/app/components/Trace.tsx +0 -100
- package/src/view/app/components/Transcript.tsx +0 -157
- package/src/view/app/components/ui/badge.tsx +0 -21
- package/src/view/app/lib/artifact-url.ts +0 -17
- package/src/view/app/lib/attempt-route.test.ts +0 -80
- package/src/view/app/lib/attempt-route.ts +0 -52
- package/src/view/app/lib/format.ts +0 -70
- package/src/view/app/lib/guards.test.ts +0 -108
- package/src/view/app/lib/guards.ts +0 -71
- package/src/view/app/lib/rows.ts +0 -22
- package/src/view/app/lib/transcript-data.tsx +0 -151
- package/src/view/app/lib/verdict.ts +0 -23
- package/src/view/app/shared.ts +0 -8
- /package/dist/report/{react → components/metric-views}/chart-math.d.ts +0 -0
- /package/dist/report/{react → components/metric-views}/chart-math.js +0 -0
- /package/dist/report/{text → components/metric-views}/plot.d.ts +0 -0
- /package/dist/report/{react → components/site-components}/PoweredBy.d.ts +0 -0
- /package/dist/report/{react → components/site-components}/PoweredBy.js +0 -0
- /package/dist/report/{text/layout.d.ts → model/text-layout.d.ts} +0 -0
- /package/dist/report/{text/layout.js → model/text-layout.js} +0 -0
- /package/dist/report/{types.js → model/types.js} +0 -0
- /package/src/report/{react → components/metric-views}/chart-math.test.ts +0 -0
- /package/src/report/{react → components/metric-views}/chart-math.ts +0 -0
- /package/src/report/{react → components/site-components}/PoweredBy.tsx +0 -0
- /package/src/report/{text/layout.ts → model/text-layout.ts} +0 -0
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "查看 NiceEval 结果并调试 agent 行为"
|
|
3
3
|
sidebarTitle: "查看结果"
|
|
4
|
-
description: "
|
|
4
|
+
description: "用 niceeval show 在终端按 @<locator> 查看 Attempt 的评估用例源码、执行记录与 diff,或用 niceeval view 在网页中查看同一份证据。"
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
每次运行后,[NiceEval](https://niceeval.com/) 都会把结构化结果写入该实验的**结果快照**目录 `.niceeval/<experiment
|
|
7
|
+
每次运行后,[NiceEval](https://niceeval.com/) 都会把结构化结果写入该实验的**结果快照**目录 `.niceeval/<experiment>/<快照>/`。快照记录本次运行实际选择的 `selectedEvalIds`;`show` / `view` 直接比较当前 Scope 中的 experiments。
|
|
8
8
|
|
|
9
9
|
## 控制台输出
|
|
10
10
|
|
|
@@ -23,7 +23,7 @@ ACTIVE
|
|
|
23
23
|
… 17 more active
|
|
24
24
|
```
|
|
25
25
|
|
|
26
|
-
Human profile 只原位更新当前总数和 active slots,不把历史帧推入 scrollback。失败、错误和去重后的 diagnostic
|
|
26
|
+
Human profile 只原位更新当前总数和 active slots,不把历史帧推入 scrollback。失败、错误和去重后的 diagnostic 会保留并带 locator。结束块只包含摘要、失败 locator、查看命令和结果路径。完整的运行、读取、修复与重跑流程见 [Coding Agent 反馈闭环](/zh/tutorials/agent-feedback-loop)。
|
|
27
27
|
|
|
28
28
|
## `.niceeval/<experiment>/<快照>/`
|
|
29
29
|
|
|
@@ -45,14 +45,14 @@ Human profile 只原位更新当前总数和 active slots,不把历史帧推
|
|
|
45
45
|
|
|
46
46
|
## `niceeval show`:终端下钻
|
|
47
47
|
|
|
48
|
-
`niceeval show`
|
|
48
|
+
`niceeval show` 的位置参数有两种形态:评估用例 ID 前缀选择评估用例范围,`@<locator>` 精确指定一次 Attempt。Flag 选择证据类型:
|
|
49
49
|
|
|
50
50
|
```bash
|
|
51
|
-
niceeval show #
|
|
51
|
+
niceeval show # 默认报告:比较当前 Scope 并下钻到 Attempt
|
|
52
52
|
niceeval show weather # 前缀过滤:weather/* 下每个 eval 的判定
|
|
53
53
|
niceeval show weather/brooklyn # 单个 eval:各 experiment 的 attempt 行(含 @<locator>)、默认 attempt 的断言明细
|
|
54
|
-
niceeval show @1k2m9qrs # 精确到一次 Attempt
|
|
55
|
-
niceeval show @1k2m9qrs --
|
|
54
|
+
niceeval show @1k2m9qrs # 精确到一次 Attempt:断言、执行、diff 与可用证据摘要
|
|
55
|
+
niceeval show @1k2m9qrs --source # 该 Attempt 运行时保存的 Eval 源码,断言标回源码行
|
|
56
56
|
niceeval show @1k2m9qrs --execution # 该 Attempt 的消息、thinking、Skill 加载、工具调用,有 OTel 时补时间
|
|
57
57
|
niceeval show @1c3h6twx --timing # 有界诊断时间树:phase、hook、operation、shell、turn、OTel 与收尾
|
|
58
58
|
niceeval show @1c3h6twx --timing=full # 同一棵时间树逐节点完整展开
|
|
@@ -64,18 +64,18 @@ niceeval show weather/brooklyn --history # 跨 run 时间轴:抖动与回
|
|
|
64
64
|
|
|
65
65
|
Sandbox 创建、setup 或 teardown 错误不依赖 trace。`result.json` 保存错误的 `code`、`message`、生命周期阶段(`phase`)、有限 cause/stack 和 diagnostics;trace 只在存在时补调用关系与耗时。Attempt 在 cleanup、teardown 和 sandbox stop 结束后才封口写入,因此收尾 diagnostic 也能回顾。
|
|
66
66
|
|
|
67
|
-
|
|
67
|
+
同一个 Experiment 多次运行会留下多份结果。不带 `@<locator>` 的默认视图显示每个 Experiment × 评估用例最新的判定,并可能从同一个 Experiment 的多次运行中组合结果。按前缀只重跑部分评估用例时,其余评估用例的判定从更早的运行补齐。每份判定都带有生成时间,因此可以识别组合结果的来源。组合结果可能包含不同版本的被测代码,最终判定应以 `--force` 全量重跑为准。
|
|
68
68
|
|
|
69
|
-
|
|
69
|
+
单个评估用例视图里,多 experiment、多 Attempt 时默认展开最新一次失败的 Attempt;需要精确查看就复制该行的 `@<locator>`。`--exp agents/codex` 按 experiment id 路径收窄;`--results` 换结果根,`--snapshot` 只看某一次快照。
|
|
70
70
|
|
|
71
71
|
`niceeval view` 的每个视图都有 CLI 对应物:
|
|
72
72
|
|
|
73
73
|
| `niceeval view` 里的视图 | 终端对应 |
|
|
74
74
|
| --- | --- |
|
|
75
75
|
| 当前结果的分组比较报告 | `niceeval show` |
|
|
76
|
-
|
|
|
77
|
-
| 单次 Attempt
|
|
78
|
-
|
|
|
76
|
+
| 该评估用例各 experiment 的判定行 + 默认 attempt 的断言明细 | `niceeval show <eval id>` |
|
|
77
|
+
| 单次 Attempt 摘要(断言、执行、diff 与证据可用性) | `niceeval show @<locator>` |
|
|
78
|
+
| 评估用例源码标注(断言标回源码行) | `niceeval show @<locator> --source` |
|
|
79
79
|
| AI 对话、thinking、Skill 加载与工具调用(有 OTel 时补时间) | `niceeval show @<locator> --execution` |
|
|
80
80
|
| 单次 Attempt 的阶段耗时分解 | `niceeval show @<locator> --timing`;逐节点审计用 `--timing=full` |
|
|
81
81
|
| 文件改动 | `niceeval show @<locator> --diff` |
|
|
@@ -83,11 +83,11 @@ Sandbox 创建、setup 或 teardown 错误不依赖 trace。`result.json` 保存
|
|
|
83
83
|
|
|
84
84
|
### 默认分组比较报告
|
|
85
85
|
|
|
86
|
-
不带 flag 的 `niceeval show`
|
|
86
|
+
不带 flag 的 `niceeval show` 直接输出当前 Scope 的成本 × 端到端成功率图和实验列表。每个 experiment 的 eval 数与指标分母读取其快照 `selectedEvalIds`;未选择的评估用例不补成失败。
|
|
87
87
|
|
|
88
|
-
|
|
88
|
+
实验列表先给固定列汇总,再按 experiment → 评估用例 → Attempt 展开。locator 只保留 `@<id>`;完整断言和 evidence 留在 `niceeval show @<locator>`。只有一个 experiment 时散点仍显示单点。
|
|
89
89
|
|
|
90
|
-
###
|
|
90
|
+
### 单个评估用例
|
|
91
91
|
|
|
92
92
|
```text
|
|
93
93
|
$ niceeval show weather/brooklyn
|
|
@@ -103,14 +103,14 @@ attempt 3 · compare/codex-gpt-5.4 · failed · 41.2s · 12.3k tokens · $0.04
|
|
|
103
103
|
|
|
104
104
|
artifacts: .niceeval/compare_codex-gpt-5.4/2026-07-09T10-00-00-000Z-x1f2/weather/brooklyn/a2/
|
|
105
105
|
attempt locator: @1k2m9qrs
|
|
106
|
-
next: niceeval show @1k2m9qrs [--
|
|
106
|
+
next: niceeval show @1k2m9qrs [--source|--execution]
|
|
107
107
|
```
|
|
108
108
|
|
|
109
109
|
每个 experiment 一行的紧凑索引末尾就带着代表 attempt 的 `@<locator>`(`compare/bub-gpt-5.4` 只有一次 attempt 就代表它自己,`compare/codex-gpt-5.4` 代表的是默认展开的那一次失败);展开的明细块结尾再重复一次精确的 `attempt locator:`,两处给的是同一个值。想看别的 attempt(比如 `compare/codex-gpt-5.4` 更早失败的那次),去 `--history` 或 artifact 目录里找到它的 locator,`niceeval show @<locator>` 直接跳过去,不需要先回到这张列表。
|
|
110
110
|
|
|
111
|
-
### `@<locator>`:一次 Attempt
|
|
111
|
+
### `@<locator>`:一次 Attempt 的摘要
|
|
112
112
|
|
|
113
|
-
不带证据 flag 的 `niceeval show @<locator>` 是失败诊断首页:除断言计票外,失败项直接列 group、matcher、expected、received
|
|
113
|
+
不带证据 flag 的 `niceeval show @<locator>` 是失败诊断首页:除断言计票外,失败项直接列 group、matcher、expected、received、原因和源码位置;随后给出评估用例源码可用性、Agent 事件计数、OTel 计时可用性与工作区 diff 摘要。该页用于定位失败原因;需要追查 Agent 结果的来源时,再打开对应证据视图:
|
|
114
114
|
|
|
115
115
|
```text
|
|
116
116
|
$ niceeval show @1k2m9qrs
|
|
@@ -134,21 +134,21 @@ changes: diff unavailable · no workspace diff was recorded for this attempt
|
|
|
134
134
|
|
|
135
135
|
artifacts: .niceeval/compare_codex-gpt-5.4/2026-07-09T10-00-00-000Z-x1f2/weather/brooklyn/a2/
|
|
136
136
|
available:
|
|
137
|
-
niceeval show @1k2m9qrs --
|
|
137
|
+
niceeval show @1k2m9qrs --source
|
|
138
138
|
niceeval show @1k2m9qrs --execution
|
|
139
139
|
niceeval show @1k2m9qrs --timing
|
|
140
140
|
```
|
|
141
141
|
|
|
142
|
-
|
|
142
|
+
该视图只给摘要,不复现完整证据。完整源码、事件流和文件列表分别由对应的证据 flag 展开。`changes` 行会说明 diff 不可用的具体原因:此处的 `weather/brooklyn` 不是 Sandbox 评估用例,因此从未收集工作区改动;Sandbox 评估用例未修改文件时则显示「没有产生工作区文件改动」。
|
|
143
143
|
|
|
144
|
-
### `--
|
|
144
|
+
### `--source`、`--execution`、`--timing`、`--diff`:四个证据切面
|
|
145
145
|
|
|
146
|
-
四个证据 flag 既可以加在 `@<locator>` 后面(精确到一次 Attempt
|
|
146
|
+
四个证据 flag 既可以加在 `@<locator>` 后面(精确到一次 Attempt),也可以加在能唯一收窄到一个评估用例的前缀后面(挑同一套默认启发式选中的 attempt——最新一次失败,没有失败就挑最新一次,与单个评估用例详情块展开的是同一次);前缀撞到不止一个评估用例时会报错,报错正文直接给出每个候选评估用例的 `@<locator>`,照抄一个继续即可。可以同时传多个证据 flag,一次输出全部要看的切面。
|
|
147
147
|
|
|
148
|
-
`--
|
|
148
|
+
`--source` 是运行时保存的评估用例源码,按行标注每条断言、gate 失败与 soft 分数直接排在对应源码行下面,源码里没能对应到具体行的断言(没有位置信息,或位置指向另一个文件)单独成一段,永不丢弃:
|
|
149
149
|
|
|
150
150
|
```text
|
|
151
|
-
$ niceeval show @1k2m9qrs --
|
|
151
|
+
$ niceeval show @1k2m9qrs --source
|
|
152
152
|
@1k2m9qrs · weather/brooklyn · compare/codex-gpt-5.4 · failed
|
|
153
153
|
|
|
154
154
|
eval source: evals/weather/brooklyn.eval.ts · sha256:a3fa4555…
|
|
@@ -170,7 +170,7 @@ assertions: 1 passed · 1 gate failed · 1 soft below target
|
|
|
170
170
|
full eval source: .niceeval/compare_codex-gpt-5.4/2026-07-09T10-00-00-000Z-x1f2/weather/brooklyn/a2/sources.json
|
|
171
171
|
```
|
|
172
172
|
|
|
173
|
-
它展示的是**这次 Attempt
|
|
173
|
+
它展示的是**这次 Attempt 运行当时保存的源码**,不是当前工作区里同名文件的最新内容——评估用例改过之后再看旧 Attempt,看到的仍是它跑的那一版。没有捕获到源码时如实说明「eval source unavailable for this attempt」,不伪造一份空文档。
|
|
174
174
|
|
|
175
175
|
`--execution` 把标准事件流(消息、thinking、Skill 加载、工具调用/结果、subagent、错误)排成一棵执行树;这次 Attempt 接入过 OTel 时,同一节点能关联到的 span 会补上相对时间与耗时。无法关联到 Agent 事件的 SDK / runtime span 不逐行混入执行记录,只在结尾报告省略数量并指向完整 `trace.json`。没接入 OTel 时骨架、顺序和内容不变,只去掉时间列并标 `timing unavailable`:
|
|
176
176
|
|
|
@@ -189,7 +189,7 @@ full events: .niceeval/compare_codex-gpt-5.4/2026-07-09T10-00-00-000Z-x1f2/weath
|
|
|
189
189
|
full OTel trace: .niceeval/compare_codex-gpt-5.4/2026-07-09T10-00-00-000Z-x1f2/weather/brooklyn/a2/trace.json
|
|
190
190
|
```
|
|
191
191
|
|
|
192
|
-
这次 Attempt 只有两条消息、没有任何工具调用节点——这正是上面 `calledTool("get_weather")` 这条 gate 断言失败的原因,`--execution` 在这里直接印证了 `--
|
|
192
|
+
这次 Attempt 只有两条消息、没有任何工具调用节点——这正是上面 `calledTool("get_weather")` 这条 gate 断言失败的原因,`--execution` 在这里直接印证了 `--source` 那份断言标注。有 Skill 加载和工具调用时,`--execution` 把它们按发生顺序一起排进同一棵树,工具调用拆成「调用」与「结果」两行读:
|
|
193
193
|
|
|
194
194
|
```text
|
|
195
195
|
$ niceeval show @1c3h6twx --execution
|
|
@@ -234,7 +234,7 @@ timing unavailable · OTel trace was not collected
|
|
|
234
234
|
full events: .niceeval/compare_codex-gpt-5.4/2026-07-09T10-00-00-000Z-x1f2/weather/brooklyn/a2/events.json
|
|
235
235
|
```
|
|
236
236
|
|
|
237
|
-
`--timing`
|
|
237
|
+
`--timing` 显示整个 Attempt 的阶段耗时。Runner 把 lifecycle、setup/teardown hook、批量工作的 operation、所有经 Sandbox API 发出的 shell 命令,以及每个 session/turn 的 `send` 墙钟时间记入 `result.json`。某轮有 OTel 时,再按 `traceId` 把 Agent、model 和 tool span 挂到该轮下面。没有 OTel 时,phase、hook、operation、shell 与 Turn 时间仍完整,只缺轮内细分。出错或超时的 Attempt 会在已知的最深节点用 `✗` 标出:
|
|
238
238
|
|
|
239
239
|
```text
|
|
240
240
|
$ niceeval show @1c3h6twx --timing
|
|
@@ -271,9 +271,9 @@ sandbox.stop 1.2s
|
|
|
271
271
|
|
|
272
272
|
裸 `--timing` 会完整列出 phase,并把 phase 下的细节控制在 80 个节点内。超过上限时,它优先保留失败路径、最慢节点与首尾时序,在省略位置显示节点数、未展示的失败数,并给出 `--timing=full`。小树中两种模式输出相同;`--timing=full` 不设节点上限,适合审计旧结果或把完整树重定向到文件。NiceEval 不自动启动 pager,因此管道、CI 和 coding agent 不会等待键盘输入。命令可能并发,省略行不会把子节点耗时相加。
|
|
273
273
|
|
|
274
|
-
|
|
274
|
+
Operation 名称由执行工作的组件在采集时写入。例如一次 Workspace diff 导出可以显示为 `export workspace diff · 1 window · 3,302 files`,并包含一条真实的批量 shell。展示层不会解析 `git show` 等命令文本来推测 `git show ×N` 分组。旧 artifact 记录大量调用时,默认模式提供有界摘要,`--timing=full` 可以逐条核对。`--timing` 会分别显示 Sandbox 创建、安装命令和轮内模型或工具耗时。旧结果或第三方结果没有阶段数据时,两种模式都会提示 `phase timing unavailable`。
|
|
275
275
|
|
|
276
|
-
`--diff` 默认给文件级摘要(落盘的 diff 只有改动后的全文,没有基线可比对增删行数,所以每个改动过的文件都标 `M`、后面跟它现在的行数,不区分新建与修改);看单个文件的完整内容用 `--diff=<文件路径>`——路径必须用 `=`
|
|
276
|
+
`--diff` 默认给文件级摘要(落盘的 diff 只有改动后的全文,没有基线可比对增删行数,所以每个改动过的文件都标 `M`、后面跟它现在的行数,不区分新建与修改);看单个文件的完整内容用 `--diff=<文件路径>`——路径必须用 `=` 连写,位置参数永远留给评估用例 id 前缀 / `@<locator>`:
|
|
277
277
|
|
|
278
278
|
```text
|
|
279
279
|
$ niceeval show @1c3h6twx --diff
|
|
@@ -296,17 +296,17 @@ $ niceeval show @1c3h6twx --diff=src/components/Button.tsx
|
|
|
296
296
|
1. 运行 niceeval show,找到失败或出错的 eval,从紧凑索引里摘一个 @<locator>。
|
|
297
297
|
2. 运行 niceeval show <eval-id>,读取该题各 experiment 的判定、默认 attempt 的断言明细,
|
|
298
298
|
以及每行末尾的 @<locator>。
|
|
299
|
-
3. 运行 niceeval show @<locator
|
|
300
|
-
4. 按问题选择 --
|
|
299
|
+
3. 运行 niceeval show @<locator>,从摘要确认这次 Attempt 捕获的证据类型。
|
|
300
|
+
4. 按问题选择 --source、--execution、--timing 或 --diff;可以同时传多个证据 flag。
|
|
301
301
|
5. 输出被截断时,读取末尾标出的 sources.json、events.json、trace.json 或 diff.json 原始路径。
|
|
302
302
|
6. 修复后重跑该 eval,再用 niceeval show <eval-id> 验证新的 @<locator>;收工前用 --force 全量重跑确认整体结果。
|
|
303
303
|
```
|
|
304
304
|
|
|
305
|
-
`--
|
|
305
|
+
`--source` 显示评估用例检查内容和断言结果,`--execution` 显示 Agent 消息与调用(可关联时附 OTel 时间),`--timing` 显示 Attempt 各阶段耗时,`--diff` 显示 Sandbox 文件改动。按诊断目标选择证据,不需要每次读取全部四种视图。
|
|
306
306
|
|
|
307
307
|
### 历史与抖动:`--history`
|
|
308
308
|
|
|
309
|
-
|
|
309
|
+
默认报告和单个评估用例视图显示最新判定。`--history` 用跨 run 时间轴显示稳定性和回归起点,每次真实执行一行:
|
|
310
310
|
|
|
311
311
|
```text
|
|
312
312
|
$ niceeval show weather/brooklyn --history
|
|
@@ -319,9 +319,9 @@ compare/codex-gpt-5.4 · 5 runs · passed 2/5
|
|
|
319
319
|
2026-07-07T16-40 ✗ failed 3 attempts $0.05 gate calledTool("get_weather")
|
|
320
320
|
```
|
|
321
321
|
|
|
322
|
-
✓✗
|
|
322
|
+
✓✗ 交替说明这个评估用例在抖,该修的是稳定性(被测程序或断言),反复重跑碰运气只会烧钱;连续绿转红的拐点就是回归引入的位置,用 `view --snapshot <snapshot.json>` 钉住拐点前后两次细看。时间轴只列真实执行——缓存携带的旧结果是判定的复印件,不占行,否则趋势会被复印件灌满假数据。不带评估用例 id 的 `niceeval show --history` 给每个 experiment 的 per-run 通过率序列,同一份趋势的榜单视角。
|
|
323
323
|
|
|
324
|
-
两次 run
|
|
324
|
+
两次 run 的精确对比不提供专用 flag。使用 `DeltaTable` 组件编写自定义对比报告,再把文件传给 `--report`;终端和网页会渲染同一份报告。具体写法见[自定义报告](/zh/tutorials/custom-reports)。
|
|
325
325
|
|
|
326
326
|
## `niceeval view`:在网页看证据
|
|
327
327
|
|
|
@@ -329,19 +329,19 @@ compare/codex-gpt-5.4 · 5 runs · passed 2/5
|
|
|
329
329
|
npx niceeval view
|
|
330
330
|
```
|
|
331
331
|
|
|
332
|
-
这会打开本地结果查看器。它的首页报告和 `niceeval show` 选结果的规则一模一样:对每个 experiment
|
|
332
|
+
这会打开本地结果查看器。它的首页报告和 `niceeval show` 选结果的规则一模一样:对每个 experiment 的每道评估用例,取时间上最新的那份判定,同一个 experiment 跨多次运行拼出来;收窄范围(评估用例 ID 前缀、`--exp`)时按同一条规则收窄。你可以浏览评估用例、查看 agent 的对话与工具调用、读 diff、检查断言结果。数据不会上传到外部服务。
|
|
333
333
|
|
|
334
334
|
<Tip>
|
|
335
335
|
失败后立刻运行 `npx niceeval view`,可以直接打开刚刚那次运行的 artifacts。
|
|
336
336
|
</Tip>
|
|
337
337
|
|
|
338
|
-
`view`
|
|
338
|
+
`view` 的首页直接显示当前 Scope 的摘要、成本 × 端到端成功率散点图与实验表。传 `--report` 就换成自己的报告。Attempt 详情里的 transcript、时间树、trace 与 diff 始终可用。
|
|
339
339
|
|
|
340
|
-
|
|
340
|
+
网页版可以排序实验表、筛行、展开 experiment、悬停散点看数值;这些操作不改判定口径。
|
|
341
341
|
|
|
342
342
|
## 导出与静态托管
|
|
343
343
|
|
|
344
|
-
|
|
344
|
+
发布结果使用 `--out <目录>` 导出完整静态站。需要自定义页面时,使用报告组件编写报告,见[自定义报告](/zh/tutorials/custom-reports)。
|
|
345
345
|
|
|
346
346
|
### 内置查看器整站:静态托管
|
|
347
347
|
|
|
@@ -349,7 +349,16 @@ npx niceeval view
|
|
|
349
349
|
npx niceeval view --out site
|
|
350
350
|
```
|
|
351
351
|
|
|
352
|
-
NiceEval 写入 `<dir>/index.html`,并把查看器要读取的 artifact(`sources.json`、`events.json`、`trace.json
|
|
352
|
+
NiceEval 写入 `<dir>/index.html`,并把查看器要读取的 artifact(`sources.json`、`events.json`、`trace.json`,有 `diff.json` 也带上)复制到 `<dir>/artifact/` 下。把整个目录交给任何静态托管(Vercel、GitHub Pages 或 `python3 -m http.server`),代码视图、transcript 和 trace 瀑布图都和本地 `niceeval view` 一致;表格排序、过滤和图表悬停也照常可用——所需脚本已内联进页面,浏览器禁用 JS 时页面仍完整可读,只是少这些浏览操作。
|
|
353
|
+
|
|
354
|
+
只想发布一部分结果时,把本地查看用的收窄直接交给导出:
|
|
355
|
+
|
|
356
|
+
```bash
|
|
357
|
+
npx niceeval view --exp agents/codex --out site # 只发布一个 experiment 路径范围
|
|
358
|
+
npx niceeval view weather --out site # 只发布 weather 开头的 eval
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
出站的就是收窄到的:页面和 `artifact/` 证据都只含收窄后的范围,被滤掉实验的 transcript、源码和 trace 不会跟着出站。不收窄就是整站导出完整结果。
|
|
353
362
|
|
|
354
363
|
`--out` 只接受目录。没有单文件导出:代码视图、transcript 和 trace 依赖 artifact 文件,单个 HTML 装不下完整证据。要把结果发给别人,托管整站发链接即可。
|
|
355
364
|
|
|
@@ -357,10 +366,10 @@ NiceEval 写入 `<dir>/index.html`,并把查看器要读取的 artifact(`sou
|
|
|
357
366
|
|
|
358
367
|
两点限制:
|
|
359
368
|
|
|
360
|
-
- `
|
|
369
|
+
- `o11y.json` 不会被复制——报告数字已经算进页面,查看器不读取它。
|
|
361
370
|
- 用 `file://` 直接打开 `index.html` 时浏览器不允许 fetch artifact,代码视图会提示源码不可用。本地预览用 http 服务打开。
|
|
362
371
|
|
|
363
|
-
|
|
372
|
+
最短流程是在运行评估用例的机器上导出并直接部署产物目录。要让站点随 push 自动更新,先用 `copySnapshots` 生成经过单文件预算检查的发布结果根并提交到仓库,再让 CI 对该目录运行 `view --results <目录> --out`。Workflow 与托管平台配置见[通过 CI 发布报告](/zh/tutorials/publish-report)。
|
|
364
373
|
|
|
365
374
|
## Artifact 说明
|
|
366
375
|
|
|
@@ -378,15 +387,15 @@ NiceEval 写入 `<dir>/index.html`,并把查看器要读取的 artifact(`sou
|
|
|
378
387
|
|
|
379
388
|
### `diff.json`
|
|
380
389
|
|
|
381
|
-
Sandbox
|
|
390
|
+
Sandbox 评估用例中 agent 改动的文件 diff。
|
|
382
391
|
|
|
383
392
|
### `sources.json`
|
|
384
393
|
|
|
385
|
-
|
|
394
|
+
评估用例源码位置和断言位置,用于查看器把失败断言对应回代码。
|
|
386
395
|
|
|
387
396
|
### `trace.json`
|
|
388
397
|
|
|
389
|
-
OTLP trace 或标准化后的 span
|
|
398
|
+
OTLP trace 或标准化后的 span 数据。它是可选的执行关系和耗时证据,不是错误存储:Sandbox 创建(`sandbox.create`)可能早于 telemetry,teardown 可能晚于 trace collect。
|
|
390
399
|
|
|
391
400
|
### `o11y.json`
|
|
392
401
|
|
|
@@ -398,7 +407,7 @@ OTLP trace 或标准化后的 span 数据。它是可选的执行关系和耗时
|
|
|
398
407
|
<Tab title="passed">所有 gate 通过。</Tab>
|
|
399
408
|
<Tab title="failed">至少一个 gate 失败,或 `--strict` 下 soft 断言低于阈值。</Tab>
|
|
400
409
|
<Tab title="errored">环境、超时、adapter 或 agent runtime 出错。</Tab>
|
|
401
|
-
<Tab title="skipped"
|
|
410
|
+
<Tab title="skipped">评估用例被主动跳过。</Tab>
|
|
402
411
|
</Tabs>
|
|
403
412
|
|
|
404
413
|
Soft 断言的分数记录在 `assertions[].score` 里;非 `--strict` 模式下,它们不会产生额外的 verdict。
|
|
@@ -406,8 +415,8 @@ Soft 断言的分数记录在 `assertions[].score` 里;非 `--strict` 模式
|
|
|
406
415
|
## 调试建议
|
|
407
416
|
|
|
408
417
|
- 榜单里失败/错误的题每条自带下钻命令:`niceeval show <eval id>` 先看断言明细,行尾的 `@<locator>` 可以直接精确下钻到某一次 Attempt。
|
|
409
|
-
-
|
|
410
|
-
-
|
|
418
|
+
- 检查评估用例内容和断言结果时使用 `--source`,断言标注会对应到源码行。
|
|
419
|
+
- 检查 Agent 消息和调用时使用 `--execution`;关联成功的 OTel 时间会显示在同一事件旁。
|
|
411
420
|
- coding-agent 失败时看 `--diff` 和 `--execution` 里的工具调用;两者结合能看出「改错了文件」还是「压根没调用该调用的工具」。
|
|
412
|
-
- Sandbox
|
|
413
|
-
-
|
|
421
|
+
- Sandbox 评估用例运行缓慢或超时时,先看 `--timing`。该视图分别列出排队、Sandbox 启动、setup hook 中的 shell、Agent 安装命令、每轮 `send`、可关联的 OTel model/tool 以及收尾阶段耗时。
|
|
422
|
+
- 定位被测程序或评估用例缺陷并完成重跑的流程见 [Coding Agent 反馈闭环](/zh/tutorials/agent-feedback-loop)。
|