niceeval 0.15.1-canary.76 → 0.16.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INDEX.md +45 -40
- package/dist/adapter-execution-trace.cjs.map +1 -1
- package/dist/adapter-execution-trace.d.cts +26 -0
- package/dist/adapter-execution-trace.d.mts +26 -0
- package/dist/adapter-execution-trace.d.ts +26 -0
- package/dist/adapter-usage.cjs.map +1 -1
- package/dist/adapter-usage.d.cts +9 -0
- package/dist/adapter-usage.d.mts +9 -0
- package/dist/adapter-usage.d.ts +9 -0
- package/dist/adapter.cjs +19 -4
- package/dist/adapter.cjs.map +1 -1
- package/dist/adapter.d.cts +37 -12
- package/dist/adapter.d.mts +37 -12
- package/dist/adapter.d.ts +37 -12
- package/dist/agents/ai-sdk.cjs +3 -0
- package/dist/agents/ai-sdk.cjs.map +1 -1
- package/dist/agents/bub.cjs +3 -0
- package/dist/agents/bub.cjs.map +1 -1
- package/dist/agents/claude-code-native.cjs +28 -2
- package/dist/agents/claude-code-native.cjs.map +1 -1
- package/dist/agents/claude-code.cjs +3 -0
- package/dist/agents/claude-code.cjs.map +1 -1
- package/dist/agents/codex.cjs +3 -0
- package/dist/agents/codex.cjs.map +1 -1
- package/dist/agents/coding-cli-versions.cjs +9 -10
- package/dist/agents/coding-cli-versions.cjs.map +1 -1
- package/dist/agents/coding-cli-versions.d.cts +1 -1
- package/dist/agents/coding-cli-versions.d.mts +1 -1
- package/dist/agents/coding-cli-versions.d.ts +1 -1
- package/dist/agents/deepseek-harness.cjs +3 -0
- package/dist/agents/deepseek-harness.cjs.map +1 -1
- package/dist/agents/hermes.cjs +3 -0
- package/dist/agents/hermes.cjs.map +1 -1
- package/dist/agents/index.cjs.map +1 -1
- package/dist/agents/index.d.cts +3 -2
- package/dist/agents/index.d.mts +3 -2
- package/dist/agents/index.d.ts +3 -2
- package/dist/agents/omp.cjs +3 -0
- package/dist/agents/omp.cjs.map +1 -1
- package/dist/agents/openai-compat.cjs +6 -0
- package/dist/agents/openai-compat.cjs.map +1 -1
- package/dist/agents/openclaw.cjs +3 -0
- package/dist/agents/openclaw.cjs.map +1 -1
- package/dist/agents/opencode.cjs +3 -0
- package/dist/agents/opencode.cjs.map +1 -1
- package/dist/agents/sdk-streams.cjs +9 -0
- package/dist/agents/sdk-streams.cjs.map +1 -1
- package/dist/agents/ui-message-stream.cjs +3 -0
- package/dist/agents/ui-message-stream.cjs.map +1 -1
- package/dist/assertions/api.cjs.map +1 -1
- package/dist/assertions/api.d.cts +27 -5
- package/dist/assertions/api.d.mts +27 -5
- package/dist/assertions/api.d.ts +27 -5
- package/dist/assertions/collection.cjs +6 -0
- package/dist/assertions/collection.cjs.map +1 -1
- package/dist/assertions/collection.d.cts +4 -2
- package/dist/assertions/collection.d.mts +4 -2
- package/dist/assertions/collection.d.ts +4 -2
- package/dist/assertions/collection.js +12 -8
- package/dist/assertions/context-match.cjs +70 -0
- package/dist/assertions/context-match.cjs.map +1 -0
- package/dist/assertions/context-match.d.cts +107 -0
- package/dist/assertions/context-match.d.mts +107 -0
- package/dist/assertions/context-match.d.ts +107 -0
- package/dist/assertions/context-match.js +12 -0
- package/dist/assertions/context-match.js.map +1 -0
- package/dist/assertions/full-content.cjs +109 -0
- package/dist/assertions/full-content.cjs.map +1 -0
- package/dist/assertions/full-content.d.cts +6 -0
- package/dist/assertions/full-content.d.mts +6 -0
- package/dist/assertions/full-content.d.ts +6 -0
- package/dist/assertions/full-content.js +12 -0
- package/dist/assertions/full-content.js.map +1 -0
- package/dist/assertions/judge-presets.cjs +2 -34
- package/dist/assertions/judge-presets.cjs.map +1 -1
- package/dist/assertions/judge-presets.d.cts +6 -9
- package/dist/assertions/judge-presets.d.mts +6 -9
- package/dist/assertions/judge-presets.d.ts +6 -9
- package/dist/assertions/judge-presets.js +8 -8
- package/dist/assertions/judge.cjs +107 -29
- package/dist/assertions/judge.cjs.map +1 -1
- package/dist/assertions/judge.d.cts +2 -0
- package/dist/assertions/judge.d.mts +2 -0
- package/dist/assertions/judge.d.ts +2 -0
- package/dist/assertions/match.cjs +244 -5
- package/dist/assertions/match.cjs.map +1 -1
- package/dist/assertions/match.d.cts +43 -3
- package/dist/assertions/match.d.mts +43 -3
- package/dist/assertions/match.d.ts +43 -3
- package/dist/assertions/match.js +88 -78
- package/dist/assertions/material.cjs +520 -0
- package/dist/assertions/material.cjs.map +1 -0
- package/dist/assertions/material.d.cts +56 -0
- package/dist/assertions/material.d.mts +56 -0
- package/dist/assertions/material.d.ts +56 -0
- package/dist/assertions/material.js +20 -0
- package/dist/assertions/material.js.map +1 -0
- package/dist/assertions/record/attachment.cjs +5 -2
- package/dist/assertions/record/attachment.cjs.map +1 -1
- package/dist/assertions/record/codec.cjs +2 -1
- package/dist/assertions/record/codec.cjs.map +1 -1
- package/dist/assertions/record/model.cjs.map +1 -1
- package/dist/assertions/record/model.d.cts +6 -1
- package/dist/assertions/record/model.d.mts +6 -1
- package/dist/assertions/record/model.d.ts +6 -1
- package/dist/assertions/runtime.cjs +207 -14
- package/dist/assertions/runtime.cjs.map +1 -1
- package/dist/assertions/runtime.d.cts +9 -0
- package/dist/assertions/runtime.d.mts +9 -0
- package/dist/assertions/runtime.d.ts +9 -0
- package/dist/assertions/runtime.js +14 -8
- package/dist/assertions/score-match-audit.cjs +18 -8
- package/dist/assertions/score-match-audit.cjs.map +1 -1
- package/dist/assertions/score-match-gateway.cjs +83 -25
- package/dist/assertions/score-match-gateway.cjs.map +1 -1
- package/dist/assertions/score-match-gateway.d.cts +2 -0
- package/dist/assertions/score-match-gateway.d.mts +2 -0
- package/dist/assertions/score-match-gateway.d.ts +2 -0
- package/dist/assertions/value-composition.test-d.cjs +77 -0
- package/dist/assertions/value-composition.test-d.cjs.map +1 -0
- package/dist/assertions/value-composition.test-d.d.cts +1 -0
- package/dist/assertions/value-composition.test-d.d.mts +1 -0
- package/dist/assertions/value-composition.test-d.d.ts +1 -0
- package/dist/assertions/value-composition.test-d.js +4 -0
- package/dist/assertions/value-composition.test-d.js.map +1 -0
- package/dist/context/assert-first.cjs +207 -213
- package/dist/context/assert-first.cjs.map +1 -1
- package/dist/context/assert-first.d.cts +49 -23
- package/dist/context/assert-first.d.mts +49 -23
- package/dist/context/assert-first.d.ts +49 -23
- package/dist/context/session.cjs +111 -9
- package/dist/context/session.cjs.map +1 -1
- package/dist/context/session.d.cts +18 -0
- package/dist/context/session.d.mts +18 -0
- package/dist/context/session.d.ts +18 -0
- package/dist/define.cjs +17 -1
- package/dist/define.cjs.map +1 -1
- package/dist/eval/cli/contribution.cjs +2 -0
- package/dist/eval/cli/contribution.cjs.map +1 -1
- package/dist/eval/host/index.cjs +2 -0
- package/dist/eval/host/index.cjs.map +1 -1
- package/dist/eval/record/evaluation.cjs +3 -0
- package/dist/eval/record/evaluation.cjs.map +1 -1
- package/dist/eval/record/membership-provenance.d.cts +15 -15
- package/dist/eval/record/membership-provenance.d.mts +15 -15
- package/dist/eval/record/membership-provenance.d.ts +15 -15
- package/dist/eval/record/score.cjs +7 -0
- package/dist/eval/record/score.cjs.map +1 -1
- package/dist/eval/record/sealed-assertion.cjs +3 -0
- package/dist/eval/record/sealed-assertion.cjs.map +1 -1
- package/dist/eval/record/verdict.cjs +7 -0
- package/dist/eval/record/verdict.cjs.map +1 -1
- package/dist/expect/index.cjs +8 -2
- package/dist/expect/index.cjs.map +1 -1
- package/dist/expect/index.d.cts +9 -4
- package/dist/expect/index.d.mts +9 -4
- package/dist/expect/index.d.ts +9 -4
- package/dist/expect/index.js +72 -62
- package/dist/expect/index.mjs +72 -62
- package/dist/experiment/host/cli/contribution.cjs +37 -23
- package/dist/experiment/host/cli/contribution.cjs.map +1 -1
- package/dist/experiment/host/cli/contribution.d.cts +9 -1
- package/dist/experiment/host/cli/contribution.d.mts +9 -1
- package/dist/experiment/host/cli/contribution.d.ts +9 -1
- package/dist/experiment/host/current.cjs +74 -0
- package/dist/experiment/host/current.cjs.map +1 -0
- package/dist/experiment/host/current.d.cts +33 -0
- package/dist/experiment/host/current.d.mts +33 -0
- package/dist/experiment/host/current.d.ts +33 -0
- package/dist/experiment/host/current.js +10 -0
- package/dist/experiment/host/current.js.map +1 -0
- package/dist/experiment/host/index.cjs +17 -174
- package/dist/experiment/host/index.cjs.map +1 -1
- package/dist/experiment/host/index.d.cts +8 -54
- package/dist/experiment/host/index.d.mts +8 -54
- package/dist/experiment/host/index.d.ts +8 -54
- package/dist/experiment/host/operations.cjs +33 -10
- package/dist/experiment/host/operations.cjs.map +1 -1
- package/dist/experiment/host/operations.d.cts +29 -1
- package/dist/experiment/host/operations.d.mts +29 -1
- package/dist/experiment/host/operations.d.ts +29 -1
- package/dist/experiment/host/operations.js +6 -4
- package/dist/experiment/host/requirements.cjs +3 -0
- package/dist/experiment/host/requirements.cjs.map +1 -0
- package/dist/experiment/host/requirements.d.cts +3 -0
- package/dist/experiment/host/requirements.d.mts +3 -0
- package/dist/experiment/host/requirements.d.ts +3 -0
- package/dist/experiment/host/requirements.js +4 -0
- package/dist/experiment/host/requirements.js.map +1 -0
- package/dist/experiment/host/runtime.cjs +179 -0
- package/dist/experiment/host/runtime.cjs.map +1 -0
- package/dist/experiment/host/runtime.d.cts +79 -0
- package/dist/experiment/host/runtime.d.mts +79 -0
- package/dist/experiment/host/runtime.d.ts +79 -0
- package/dist/experiment/host/runtime.js +6 -0
- package/dist/experiment/host/runtime.js.map +1 -0
- package/dist/experiment/host/types.cjs.map +1 -1
- package/dist/experiment/host/types.d.cts +17 -16
- package/dist/experiment/host/types.d.mts +17 -16
- package/dist/experiment/host/types.d.ts +17 -16
- package/dist/fresh-import.cjs +2 -1
- package/dist/fresh-import.cjs.map +1 -1
- package/dist/index.cjs +4 -2
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +12 -6
- package/dist/index.d.mts +12 -6
- package/dist/index.d.ts +12 -6
- package/dist/index.js +32 -30
- package/dist/index.mjs +32 -30
- package/dist/inspection/artifacts.cjs +83 -54
- package/dist/inspection/artifacts.cjs.map +1 -1
- package/dist/inspection/artifacts.d.cts +8 -0
- package/dist/inspection/artifacts.d.mts +8 -0
- package/dist/inspection/artifacts.d.ts +8 -0
- package/dist/inspection/artifacts.js +2 -0
- package/dist/inspection/assertions.cjs +4 -1
- package/dist/inspection/assertions.cjs.map +1 -1
- package/dist/inspection/cli/contribution.cjs +27 -4
- package/dist/inspection/cli/contribution.cjs.map +1 -1
- package/dist/inspection/cli/contribution.d.cts +2 -1
- package/dist/inspection/cli/contribution.d.mts +2 -1
- package/dist/inspection/cli/contribution.d.ts +2 -1
- package/dist/inspection/execution-traces.cjs +114 -8
- package/dist/inspection/execution-traces.cjs.map +1 -1
- package/dist/inspection/execution-traces.d.cts +38 -0
- package/dist/inspection/execution-traces.d.mts +38 -0
- package/dist/inspection/execution-traces.d.ts +38 -0
- package/dist/inspection/facts.cjs +12 -0
- package/dist/inspection/facts.cjs.map +1 -1
- package/dist/inspection/limits.cjs +5 -1
- package/dist/inspection/limits.cjs.map +1 -1
- package/dist/inspection/limits.d.cts +4 -0
- package/dist/inspection/limits.d.mts +4 -0
- package/dist/inspection/limits.d.ts +4 -0
- package/dist/inspection/limits.js +6 -2
- package/dist/inspection/overview.cjs +74 -33
- package/dist/inspection/overview.cjs.map +1 -1
- package/dist/inspection/overview.d.cts +19 -1
- package/dist/inspection/overview.d.mts +19 -1
- package/dist/inspection/overview.d.ts +19 -1
- package/dist/inspection/overview.js +4 -2
- package/dist/inspection/project-input.cjs +9 -0
- package/dist/inspection/project-input.cjs.map +1 -0
- package/dist/inspection/project-input.d.cts +20 -0
- package/dist/inspection/project-input.d.mts +20 -0
- package/dist/inspection/project-input.d.ts +20 -0
- package/dist/inspection/project-input.js +8 -0
- package/dist/inspection/project-input.js.map +1 -0
- package/dist/inspection/project-result.cjs +60 -0
- package/dist/inspection/project-result.cjs.map +1 -0
- package/dist/inspection/project-result.d.cts +1264 -0
- package/dist/inspection/project-result.d.mts +1264 -0
- package/dist/inspection/project-result.d.ts +1264 -0
- package/dist/inspection/project-result.js +10 -0
- package/dist/inspection/project-result.js.map +1 -0
- package/dist/inspection/project-source.cjs +109 -0
- package/dist/inspection/project-source.cjs.map +1 -0
- package/dist/inspection/project-source.d.cts +10 -0
- package/dist/inspection/project-source.d.mts +10 -0
- package/dist/inspection/project-source.d.ts +10 -0
- package/dist/inspection/project-source.js +6 -0
- package/dist/inspection/project-source.js.map +1 -0
- package/dist/inspection/project.cjs +100 -0
- package/dist/inspection/project.cjs.map +1 -0
- package/dist/inspection/project.d.cts +3 -0
- package/dist/inspection/project.d.mts +3 -0
- package/dist/inspection/project.d.ts +3 -0
- package/dist/inspection/project.js +6 -0
- package/dist/inspection/project.js.map +1 -0
- package/dist/inspection/protocol.cjs +19 -9
- package/dist/inspection/protocol.cjs.map +1 -1
- package/dist/inspection/protocol.d.cts +5506 -38189
- package/dist/inspection/protocol.d.mts +5506 -38189
- package/dist/inspection/protocol.d.ts +5506 -38189
- package/dist/inspection/public.mjs +896 -84
- package/dist/inspection/query.cjs +2 -0
- package/dist/inspection/query.cjs.map +1 -1
- package/dist/inspection/results.cjs +234 -29
- package/dist/inspection/results.cjs.map +1 -1
- package/dist/inspection/results.d.cts +870 -13
- package/dist/inspection/results.d.mts +870 -13
- package/dist/inspection/results.d.ts +870 -13
- package/dist/inspection/results.js +36 -34
- package/dist/inspection/select.cjs +126 -14
- package/dist/inspection/select.cjs.map +1 -1
- package/dist/inspection/select.d.cts +1 -1
- package/dist/inspection/select.d.mts +1 -1
- package/dist/inspection/select.d.ts +1 -1
- package/dist/inspection/source.cjs +4 -0
- package/dist/inspection/source.cjs.map +1 -1
- package/dist/inspection/sources.cjs +5 -0
- package/dist/inspection/sources.cjs.map +1 -1
- package/dist/inspection/trace.cjs +82 -84
- package/dist/inspection/trace.cjs.map +1 -1
- package/dist/inspection/trace.d.cts +7 -2
- package/dist/inspection/trace.d.mts +7 -2
- package/dist/inspection/trace.d.ts +7 -2
- package/dist/model-slots.cjs +35 -0
- package/dist/model-slots.cjs.map +1 -0
- package/dist/model-slots.d.cts +19 -0
- package/dist/model-slots.d.mts +19 -0
- package/dist/model-slots.d.ts +19 -0
- package/dist/model-slots.js +8 -0
- package/dist/model-slots.js.map +1 -0
- package/dist/o11y/adapter-call-price.cjs +4 -0
- package/dist/o11y/adapter-call-price.cjs.map +1 -1
- package/dist/o11y/adapter-call-price.d.cts +24 -0
- package/dist/o11y/adapter-call-price.d.mts +24 -0
- package/dist/o11y/adapter-call-price.d.ts +24 -0
- package/dist/o11y/adapter-call-price.js +6 -2
- package/dist/o11y/adapter-usage-projection.cjs +158 -0
- package/dist/o11y/adapter-usage-projection.cjs.map +1 -0
- package/dist/o11y/adapter-usage-projection.d.cts +141 -0
- package/dist/o11y/adapter-usage-projection.d.mts +141 -0
- package/dist/o11y/adapter-usage-projection.d.ts +141 -0
- package/dist/o11y/adapter-usage-projection.js +20 -0
- package/dist/o11y/adapter-usage-projection.js.map +1 -0
- package/dist/o11y/eval-usage.cjs +149 -0
- package/dist/o11y/eval-usage.cjs.map +1 -0
- package/dist/o11y/eval-usage.d.cts +46 -0
- package/dist/o11y/eval-usage.d.mts +46 -0
- package/dist/o11y/eval-usage.d.ts +46 -0
- package/dist/o11y/eval-usage.js +8 -0
- package/dist/o11y/eval-usage.js.map +1 -0
- package/dist/o11y/judge-call-price.cjs +62 -0
- package/dist/o11y/judge-call-price.cjs.map +1 -0
- package/dist/o11y/judge-call-price.d.cts +3 -0
- package/dist/o11y/judge-call-price.d.mts +3 -0
- package/dist/o11y/judge-call-price.d.ts +3 -0
- package/dist/o11y/judge-call-price.js +6 -0
- package/dist/o11y/judge-call-price.js.map +1 -0
- package/dist/o11y/judge-usage-projection.cjs +134 -0
- package/dist/o11y/judge-usage-projection.cjs.map +1 -0
- package/dist/o11y/judge-usage-projection.d.cts +18 -0
- package/dist/o11y/judge-usage-projection.d.mts +18 -0
- package/dist/o11y/judge-usage-projection.d.ts +18 -0
- package/dist/o11y/judge-usage-projection.js +20 -0
- package/dist/o11y/judge-usage-projection.js.map +1 -0
- package/dist/o11y/judge-usage-receipt.cjs +135 -0
- package/dist/o11y/judge-usage-receipt.cjs.map +1 -0
- package/dist/o11y/judge-usage-receipt.d.cts +5 -0
- package/dist/o11y/judge-usage-receipt.d.mts +5 -0
- package/dist/o11y/judge-usage-receipt.d.ts +5 -0
- package/dist/o11y/judge-usage-receipt.js +6 -0
- package/dist/o11y/judge-usage-receipt.js.map +1 -0
- package/dist/o11y/judge-usage.cjs +171 -0
- package/dist/o11y/judge-usage.cjs.map +1 -0
- package/dist/o11y/judge-usage.d.cts +35 -0
- package/dist/o11y/judge-usage.d.mts +35 -0
- package/dist/o11y/judge-usage.d.ts +35 -0
- package/dist/o11y/judge-usage.js +6 -0
- package/dist/o11y/judge-usage.js.map +1 -0
- package/dist/plugin/contracts.cjs +2 -0
- package/dist/plugin/contracts.cjs.map +1 -1
- package/dist/plugin/link.cjs +7 -0
- package/dist/plugin/link.cjs.map +1 -1
- package/dist/record/attachment/protocol.cjs +8 -6
- package/dist/record/attachment/protocol.cjs.map +1 -1
- package/dist/record/attachment/protocol.d.cts +10 -0
- package/dist/record/attachment/protocol.d.mts +10 -0
- package/dist/record/attachment/protocol.d.ts +10 -0
- package/dist/record/attachment/protocol.js +38 -36
- package/dist/record/codec/seal-manifest.d.cts +3 -3
- package/dist/record/codec/seal-manifest.d.mts +3 -3
- package/dist/record/codec/seal-manifest.d.ts +3 -3
- package/dist/record/family/adapter-usage/definition.d.cts +1 -0
- package/dist/record/family/adapter-usage/definition.d.mts +1 -0
- package/dist/record/family/adapter-usage/definition.d.ts +1 -0
- package/dist/record/family/adapter-usage/persistence.cjs +1 -1
- package/dist/record/family/adapter-usage/persistence.cjs.map +1 -1
- package/dist/record/family/adapter-usage/persistence.d.cts +2 -1
- package/dist/record/family/adapter-usage/persistence.d.mts +2 -1
- package/dist/record/family/adapter-usage/persistence.d.ts +2 -1
- package/dist/record/family/adapter-usage/schema.cjs +21 -2
- package/dist/record/family/adapter-usage/schema.cjs.map +1 -1
- package/dist/record/family/adapter-usage/schema.d.cts +85 -0
- package/dist/record/family/adapter-usage/schema.d.mts +85 -0
- package/dist/record/family/adapter-usage/schema.d.ts +85 -0
- package/dist/record/family/adapter-usage/schema.js +18 -12
- package/dist/record/family/catalog.cjs +31 -27
- package/dist/record/family/catalog.cjs.map +1 -1
- package/dist/record/family/catalog.d.cts +94 -3
- package/dist/record/family/catalog.d.mts +94 -3
- package/dist/record/family/catalog.d.ts +94 -3
- package/dist/record/family/current.cjs +45 -42
- package/dist/record/family/current.cjs.map +1 -1
- package/dist/record/family/current.d.cts +240 -1
- package/dist/record/family/current.d.mts +240 -1
- package/dist/record/family/current.d.ts +240 -1
- package/dist/record/family/execution-traces/definition.d.cts +26 -0
- package/dist/record/family/execution-traces/definition.d.mts +26 -0
- package/dist/record/family/execution-traces/definition.d.ts +26 -0
- package/dist/record/family/execution-traces/definition.js +20 -14
- package/dist/record/family/execution-traces/schema.cjs +57 -1
- package/dist/record/family/execution-traces/schema.cjs.map +1 -1
- package/dist/record/family/execution-traces/schema.d.cts +114 -0
- package/dist/record/family/execution-traces/schema.d.mts +114 -0
- package/dist/record/family/execution-traces/schema.d.ts +114 -0
- package/dist/record/family/execution-traces/schema.js +18 -12
- package/dist/record/family/judge-usage/definition.cjs +12 -0
- package/dist/record/family/judge-usage/definition.cjs.map +1 -0
- package/dist/record/family/judge-usage/definition.d.cts +91 -0
- package/dist/record/family/judge-usage/definition.d.mts +91 -0
- package/dist/record/family/judge-usage/definition.d.ts +91 -0
- package/dist/record/family/judge-usage/definition.js +6 -0
- package/dist/record/family/judge-usage/definition.js.map +1 -0
- package/dist/record/family/judge-usage/persistence.cjs +10 -0
- package/dist/record/family/judge-usage/persistence.cjs.map +1 -0
- package/dist/record/family/judge-usage/persistence.d.cts +91 -0
- package/dist/record/family/judge-usage/persistence.d.mts +91 -0
- package/dist/record/family/judge-usage/persistence.d.ts +91 -0
- package/dist/record/family/judge-usage/persistence.js +6 -0
- package/dist/record/family/judge-usage/persistence.js.map +1 -0
- package/dist/record/family/judge-usage/schema.cjs +307 -0
- package/dist/record/family/judge-usage/schema.cjs.map +1 -0
- package/dist/record/family/judge-usage/schema.d.cts +351 -0
- package/dist/record/family/judge-usage/schema.d.mts +351 -0
- package/dist/record/family/judge-usage/schema.d.ts +351 -0
- package/dist/record/family/judge-usage/schema.js +18 -0
- package/dist/record/family/judge-usage/schema.js.map +1 -0
- package/dist/record/host/sqlite-host.cjs +55 -9
- package/dist/record/host/sqlite-host.cjs.map +1 -1
- package/dist/record/host/types.cjs.map +1 -1
- package/dist/record/host/types.d.cts +7 -0
- package/dist/record/host/types.d.mts +7 -0
- package/dist/record/host/types.d.ts +7 -0
- package/dist/record/model/run-context.cjs +7 -1
- package/dist/record/model/run-context.cjs.map +1 -1
- package/dist/record/model/run-context.d.cts +4 -0
- package/dist/record/model/run-context.d.mts +4 -0
- package/dist/record/model/run-context.d.ts +4 -0
- package/dist/record/model/seal-manifest.d.cts +1 -1
- package/dist/record/model/seal-manifest.d.mts +1 -1
- package/dist/record/model/seal-manifest.d.ts +1 -1
- package/dist/record/sqlite/client.cjs +18 -5
- package/dist/record/sqlite/client.cjs.map +1 -1
- package/dist/record/sqlite/client.d.cts +1 -1
- package/dist/record/sqlite/client.d.mts +1 -1
- package/dist/record/sqlite/client.d.ts +1 -1
- package/dist/record/sqlite/project-state-database.cjs +29 -8
- package/dist/record/sqlite/project-state-database.cjs.map +1 -1
- package/dist/run/host/runtime.cjs +4 -0
- package/dist/run/host/runtime.cjs.map +1 -1
- package/dist/run/storage/sqlite.cjs +20 -0
- package/dist/run/storage/sqlite.cjs.map +1 -1
- package/dist/runner/adapter-execution-trace.cjs +125 -2
- package/dist/runner/adapter-execution-trace.cjs.map +1 -1
- package/dist/runner/adapter-execution-trace.d.cts +1 -1
- package/dist/runner/adapter-execution-trace.d.mts +1 -1
- package/dist/runner/adapter-execution-trace.d.ts +1 -1
- package/dist/runner/adapter-usage.cjs +55 -7
- package/dist/runner/adapter-usage.cjs.map +1 -1
- package/dist/runner/adapter-usage.d.cts +9 -2
- package/dist/runner/adapter-usage.d.mts +9 -2
- package/dist/runner/adapter-usage.d.ts +9 -2
- package/dist/runner/adoption.cjs +9 -0
- package/dist/runner/adoption.cjs.map +1 -1
- package/dist/runner/attempt-cancellation.cjs +12 -0
- package/dist/runner/attempt-cancellation.cjs.map +1 -0
- package/dist/runner/attempt-cancellation.d.cts +6 -0
- package/dist/runner/attempt-cancellation.d.mts +6 -0
- package/dist/runner/attempt-cancellation.d.ts +6 -0
- package/dist/runner/attempt-cancellation.js +6 -0
- package/dist/runner/attempt-cancellation.js.map +1 -0
- package/dist/runner/attempt-resources.cjs +10 -4
- package/dist/runner/attempt-resources.cjs.map +1 -1
- package/dist/runner/attempt-resources.d.cts +2 -1
- package/dist/runner/attempt-resources.d.mts +2 -1
- package/dist/runner/attempt-resources.d.ts +2 -1
- package/dist/runner/attempt.cjs +1177 -1139
- package/dist/runner/attempt.cjs.map +1 -1
- package/dist/runner/config-identity.cjs +20 -1
- package/dist/runner/config-identity.cjs.map +1 -1
- package/dist/runner/config-identity.d.cts +5 -0
- package/dist/runner/config-identity.d.mts +5 -0
- package/dist/runner/config-identity.d.ts +5 -0
- package/dist/runner/discover.cjs +14 -0
- package/dist/runner/discover.cjs.map +1 -1
- package/dist/runner/discover.d.cts +3 -0
- package/dist/runner/discover.d.mts +3 -0
- package/dist/runner/discover.d.ts +3 -0
- package/dist/runner/feedback/human.cjs +9 -4
- package/dist/runner/feedback/human.cjs.map +1 -1
- package/dist/runner/fingerprint.cjs +1 -0
- package/dist/runner/fingerprint.cjs.map +1 -1
- package/dist/runner/judge-usage.cjs +18 -0
- package/dist/runner/judge-usage.cjs.map +1 -0
- package/dist/runner/judge-usage.d.cts +5 -0
- package/dist/runner/judge-usage.d.mts +5 -0
- package/dist/runner/judge-usage.d.ts +5 -0
- package/dist/runner/judge-usage.js +8 -0
- package/dist/runner/judge-usage.js.map +1 -0
- package/dist/runner/project-target.cjs.map +1 -1
- package/dist/runner/project-target.d.cts +2 -0
- package/dist/runner/project-target.d.mts +2 -0
- package/dist/runner/project-target.d.ts +2 -0
- package/dist/runner/record/planning.cjs +1 -0
- package/dist/runner/record/planning.cjs.map +1 -1
- package/dist/runner/record.cjs +4 -0
- package/dist/runner/record.cjs.map +1 -1
- package/dist/runner/reporters/braintrust.cjs +2 -0
- package/dist/runner/reporters/braintrust.cjs.map +1 -1
- package/dist/runner/reuse-diagnostics.cjs +9 -10
- package/dist/runner/reuse-diagnostics.cjs.map +1 -1
- package/dist/runner/reuse-diagnostics.d.cts +5 -6
- package/dist/runner/reuse-diagnostics.d.mts +5 -6
- package/dist/runner/reuse-diagnostics.d.ts +5 -6
- package/dist/runner/reuse-diagnostics.js +4 -4
- package/dist/runner/reuse-plan.cjs +100 -48
- package/dist/runner/reuse-plan.cjs.map +1 -1
- package/dist/runner/reuse-plan.d.cts +60 -3
- package/dist/runner/reuse-plan.d.mts +60 -3
- package/dist/runner/reuse-plan.d.ts +60 -3
- package/dist/runner/reuse-plan.js +14 -10
- package/dist/runner/run.cjs +8 -6
- package/dist/runner/run.cjs.map +1 -1
- package/dist/runner/session.cjs +1 -1
- package/dist/runner/session.cjs.map +1 -1
- package/dist/runner/session.d.cts +1 -1
- package/dist/runner/session.d.mts +1 -1
- package/dist/runner/session.d.ts +1 -1
- package/dist/runner/source-producer.cjs +3 -1
- package/dist/runner/source-producer.cjs.map +1 -1
- package/dist/runner/types.cjs.map +1 -1
- package/dist/runner/types.d.cts +26 -5
- package/dist/runner/types.d.mts +26 -5
- package/dist/runner/types.d.ts +26 -5
- package/dist/sandbox/action.cjs +3 -0
- package/dist/sandbox/action.cjs.map +1 -1
- package/dist/sandbox/commands.cjs +3 -0
- package/dist/sandbox/commands.cjs.map +1 -1
- package/dist/sandbox/compose.cjs +6 -0
- package/dist/sandbox/compose.cjs.map +1 -1
- package/dist/sandbox/docker-profile/registry.cjs +7 -0
- package/dist/sandbox/docker-profile/registry.cjs.map +1 -1
- package/dist/sandbox/docker-profile/runtime.cjs +3 -0
- package/dist/sandbox/docker-profile/runtime.cjs.map +1 -1
- package/dist/sandbox/docker-profile/schema.cjs +7 -0
- package/dist/sandbox/docker-profile/schema.cjs.map +1 -1
- package/dist/sandbox/incus.cjs +7 -0
- package/dist/sandbox/incus.cjs.map +1 -1
- package/dist/sandbox/layer.cjs +12 -0
- package/dist/sandbox/layer.cjs.map +1 -1
- package/dist/shared/adapter-cleanup.cjs +18 -0
- package/dist/shared/adapter-cleanup.cjs.map +1 -0
- package/dist/shared/adapter-cleanup.d.cts +4 -0
- package/dist/shared/adapter-cleanup.d.mts +4 -0
- package/dist/shared/adapter-cleanup.d.ts +4 -0
- package/dist/shared/adapter-cleanup.js +10 -0
- package/dist/shared/adapter-cleanup.js.map +1 -0
- package/dist/shared/attempt-lifecycle.cjs +3 -0
- package/dist/shared/attempt-lifecycle.cjs.map +1 -0
- package/dist/shared/attempt-lifecycle.d.cts +13 -0
- package/dist/shared/attempt-lifecycle.d.mts +13 -0
- package/dist/shared/attempt-lifecycle.d.ts +13 -0
- package/dist/shared/attempt-lifecycle.js +4 -0
- package/dist/shared/attempt-lifecycle.js.map +1 -0
- package/dist/show/contribution.cjs +52 -14
- package/dist/show/contribution.cjs.map +1 -1
- package/dist/show/contribution.d.cts +23 -1
- package/dist/show/contribution.d.mts +23 -1
- package/dist/show/contribution.d.ts +23 -1
- package/dist/show/model.cjs +8 -1
- package/dist/show/model.cjs.map +1 -1
- package/dist/show/model.d.cts +10 -1
- package/dist/show/model.d.mts +10 -1
- package/dist/show/model.d.ts +10 -1
- package/dist/show/render.cjs +287 -49
- package/dist/show/render.cjs.map +1 -1
- package/dist/show/render.d.cts +3 -0
- package/dist/show/render.d.mts +3 -0
- package/dist/show/render.d.ts +3 -0
- package/dist/sources/projector.cjs +3 -0
- package/dist/sources/projector.cjs.map +1 -1
- package/dist/types.cjs.map +1 -1
- package/dist/types.d.cts +1 -1
- package/dist/types.d.mts +1 -1
- package/dist/types.d.ts +1 -1
- package/dist/view/app-dist/.vite/manifest.json +2 -2
- package/dist/view/app-dist/assets/index-CmVoXbwG.js +63 -0
- package/dist/view/app-dist/assets/{quality-cost-scatter-CFB3wnkN.js → quality-cost-scatter-BuLISaH4.js} +1 -1
- package/dist/view/app-dist/index.html +1 -1
- package/dist/view/cli/contribution.cjs +54 -21
- package/dist/view/cli/contribution.cjs.map +1 -1
- package/dist/view/cli/contribution.d.cts +2 -1
- package/dist/view/cli/contribution.d.mts +2 -1
- package/dist/view/cli/contribution.d.ts +2 -1
- package/dist/view/function-runtime.mjs +13231 -11647
- package/dist/view/http-protocol.cjs +3 -0
- package/dist/view/http-protocol.cjs.map +1 -1
- package/dist/view/http-protocol.d.cts +8 -0
- package/dist/view/http-protocol.d.mts +8 -0
- package/dist/view/http-protocol.d.ts +8 -0
- package/dist/view/inspection-host.cjs +17 -1
- package/dist/view/inspection-host.cjs.map +1 -1
- package/dist/view/render.cjs +1 -0
- package/dist/view/render.cjs.map +1 -1
- package/dist/view/render.d.cts +2 -1
- package/dist/view/render.d.mts +2 -1
- package/dist/view/render.d.ts +2 -1
- package/dist/view/revision.cjs.map +1 -1
- package/dist/view/revision.d.cts +12 -0
- package/dist/view/revision.d.mts +12 -0
- package/dist/view/revision.d.ts +12 -0
- package/dist/view/server.cjs +10 -9
- package/dist/view/server.cjs.map +1 -1
- package/docs-site/zh/examples/ai-agent-application.mdx +1 -1
- package/docs-site/zh/explanation/adapter.mdx +2 -2
- package/docs-site/zh/explanation/assert.mdx +60 -24
- package/docs-site/zh/explanation/drive.mdx +1 -1
- package/docs-site/zh/explanation/evals.mdx +31 -29
- package/docs-site/zh/explanation/experiment.mdx +15 -15
- package/docs-site/zh/explanation/judge.mdx +23 -15
- package/docs-site/zh/explanation/overview.mdx +44 -21
- package/docs-site/zh/explanation/record.mdx +28 -13
- package/docs-site/zh/explanation/results-flow.mdx +37 -0
- package/docs-site/zh/explanation/runner.mdx +27 -6
- package/docs-site/zh/index.mdx +22 -115
- package/docs-site/zh/introduction.mdx +89 -61
- package/docs-site/zh/reference/cli.mdx +50 -26
- package/docs-site/zh/reference/define-agent.mdx +37 -5
- package/docs-site/zh/reference/define-eval.mdx +130 -29
- package/docs-site/zh/reference/define-experiment.mdx +251 -0
- package/docs-site/zh/reference/events.mdx +34 -30
- package/docs-site/zh/reference/expect.mdx +77 -2
- package/docs-site/zh/reference/official-adapters.mdx +2 -2
- package/docs-site/zh/troubleshooting/debug-sandbox.mdx +43 -24
- package/docs-site/zh/troubleshooting/debugging.mdx +65 -42
- package/docs-site/zh/troubleshooting/recover-after-kill.mdx +35 -28
- package/docs-site/zh/tutorials/accept-results.mdx +34 -24
- package/docs-site/zh/tutorials/agent-feedback-loop.mdx +74 -43
- package/docs-site/zh/tutorials/agent-onboarding.mdx +27 -20
- package/docs-site/zh/tutorials/authoring.mdx +138 -121
- package/docs-site/zh/tutorials/ci-integration.mdx +46 -22
- package/docs-site/zh/tutorials/compare-runs.mdx +40 -13
- package/docs-site/zh/tutorials/concurrency.mdx +66 -62
- package/docs-site/zh/tutorials/configuration.mdx +17 -17
- package/docs-site/zh/tutorials/connect-otel.mdx +48 -37
- package/docs-site/zh/tutorials/connect-your-agent.mdx +80 -173
- package/docs-site/zh/tutorials/control-run-cost.mdx +63 -12
- package/docs-site/zh/tutorials/criteria-files.mdx +28 -21
- package/docs-site/zh/tutorials/custom-application.mdx +75 -55
- package/docs-site/zh/tutorials/dataset-fanout.mdx +32 -22
- package/docs-site/zh/tutorials/debug-lifecycle-plan.mdx +38 -31
- package/docs-site/zh/tutorials/docker-in-docker.mdx +83 -89
- package/docs-site/zh/tutorials/eval-groups.mdx +70 -89
- package/docs-site/zh/tutorials/evaluation-kinds.mdx +12 -5
- package/docs-site/zh/tutorials/experiment-services.mdx +243 -0
- package/docs-site/zh/tutorials/fixtures.mdx +42 -26
- package/docs-site/zh/tutorials/handle-execution-failures.mdx +41 -19
- package/docs-site/zh/tutorials/install-custom-sandbox-agent.mdx +21 -16
- package/docs-site/zh/tutorials/niceeval-taste-application.mdx +394 -0
- package/docs-site/zh/tutorials/niceeval-taste.mdx +190 -0
- package/docs-site/zh/tutorials/nixos-managed-dind.mdx +51 -39
- package/docs-site/zh/tutorials/plugins.mdx +55 -31
- package/docs-site/zh/tutorials/prebuilt-sandbox.mdx +204 -0
- package/docs-site/zh/tutorials/quickstart.mdx +91 -47
- package/docs-site/zh/tutorials/reporters.mdx +45 -30
- package/docs-site/zh/tutorials/rerun-and-cache.mdx +39 -40
- package/docs-site/zh/tutorials/sandbox-agent.mdx +65 -40
- package/docs-site/zh/tutorials/sandbox-providers.mdx +104 -282
- package/docs-site/zh/tutorials/sandbox-reuse.mdx +65 -97
- package/docs-site/zh/tutorials/select-and-preview.mdx +25 -20
- package/docs-site/zh/tutorials/verify-judge.mdx +49 -26
- package/docs-site/zh/tutorials/viewing-results.mdx +130 -80
- package/docs-site/zh/tutorials/write-experiment.mdx +116 -298
- package/docs-site/zh/tutorials/write-send.mdx +53 -52
- package/node_modules/undici/docs/docs/api/Agent.md +8 -8
- package/node_modules/undici/docs/docs/api/DiagnosticsChannel.md +9 -6
- package/node_modules/undici/docs/docs/api/Errors.md +25 -0
- package/node_modules/undici/docs/docs/api/Interceptors.md +4 -3
- package/node_modules/undici/docs/docs/api/Pool.md +5 -6
- package/node_modules/undici/lib/api/readable.js +1 -2
- package/node_modules/undici/lib/core/request.js +23 -2
- package/node_modules/undici/lib/core/util.js +0 -39
- package/node_modules/undici/lib/dispatcher/client-h1.js +11 -4
- package/node_modules/undici/lib/dispatcher/client-h2.js +52 -8
- package/node_modules/undici/lib/dispatcher/client.js +1 -1
- package/node_modules/undici/lib/dispatcher/dispatcher1-wrapper.js +3 -2
- package/node_modules/undici/lib/dispatcher/env-http-proxy-agent.js +21 -6
- package/node_modules/undici/lib/dispatcher/pool-base.js +63 -2
- package/node_modules/undici/lib/dispatcher/pool.js +90 -7
- package/node_modules/undici/lib/dispatcher/proxy-agent.js +16 -3
- package/node_modules/undici/lib/handler/deduplication-handler.js +24 -11
- package/node_modules/undici/lib/handler/redirect-handler.js +5 -0
- package/node_modules/undici/lib/handler/retry-handler.js +72 -13
- package/node_modules/undici/lib/interceptor/cache.js +4 -1
- package/node_modules/undici/lib/interceptor/decompress.js +236 -69
- package/node_modules/undici/lib/web/cache/cache.js +17 -8
- package/node_modules/undici/lib/web/cache/cachestorage.js +11 -5
- package/node_modules/undici/lib/web/cookies/index.js +19 -7
- package/node_modules/undici/lib/web/cookies/util.js +3 -1
- package/node_modules/undici/lib/web/eventsource/eventsource.js +25 -1
- package/node_modules/undici/lib/web/fetch/body.js +22 -15
- package/node_modules/undici/lib/web/fetch/data-url.js +6 -6
- package/node_modules/undici/lib/web/fetch/formdata.js +30 -36
- package/node_modules/undici/lib/web/fetch/headers.js +34 -31
- package/node_modules/undici/lib/web/fetch/index.js +38 -14
- package/node_modules/undici/lib/web/fetch/request.js +74 -79
- package/node_modules/undici/lib/web/fetch/response.js +43 -44
- package/node_modules/undici/lib/web/fetch/util.js +7 -7
- package/node_modules/undici/lib/web/infra/index.js +16 -1
- package/node_modules/undici/lib/web/webidl/index.js +2 -14
- package/node_modules/undici/lib/web/websocket/events.js +43 -26
- package/node_modules/undici/lib/web/websocket/receiver.js +5 -0
- package/node_modules/undici/lib/web/websocket/stream/websocketerror.js +13 -10
- package/node_modules/undici/lib/web/websocket/websocket.js +44 -41
- package/node_modules/undici/package.json +1 -1
- package/node_modules/undici/types/diagnostics-channel.d.ts +2 -1
- package/node_modules/undici/types/interceptors.d.ts +1 -1
- package/node_modules/undici/types/webidl.d.ts +13 -7
- package/package.json +4 -4
- package/dist/view/app-dist/assets/index-DFzw1L97.js +0 -63
- package/docs-site/zh/tutorials/experiments.mdx +0 -101
|
@@ -1,48 +1,69 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "
|
|
2
|
+
title: "快速开始:跑通第一条评估用例"
|
|
3
3
|
sidebarTitle: "快速开始"
|
|
4
|
-
description: "安装 NiceEval
|
|
4
|
+
description: "安装 NiceEval,写一个 Adapter、一个 Experiment 和一条评估用例,十分钟左右对你自己的应用跑通第一条评估,并在终端和浏览器里看到结果。"
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
这一页带你对自己的应用跑通第一条评估用例。你会写三个文件:
|
|
8
|
+
|
|
9
|
+
- **Adapter**:告诉 NiceEval 怎样调用你的应用。
|
|
10
|
+
- **Experiment**:说明对着哪个 Adapter、用哪个模型跑。
|
|
11
|
+
- **评估用例**:说明发什么消息、怎样算做对。
|
|
12
|
+
|
|
13
|
+
有两种走法。让 Coding Agent 帮你写最快;自己动手写能更清楚每个文件管什么。
|
|
14
|
+
|
|
15
|
+
## 让 Coding Agent 帮你接入
|
|
8
16
|
|
|
9
|
-
## 使用 Coding Agent 接入(推荐)
|
|
10
17
|
<Steps>
|
|
11
|
-
<Step title="
|
|
12
|
-
|
|
18
|
+
<Step title="把指令发给 Agent">
|
|
19
|
+
在当前项目里打开你正在用的 Coding Agent,例如 Claude Code 或 Codex,发送下面这段话:
|
|
13
20
|
|
|
14
21
|
```text
|
|
15
22
|
阅读 https://niceeval.com/INIT.md,为当前仓库安装并接入 niceeval,端到端跑通第一条评估用例。完成时列出创建的 Experiment ID、运行命令和结果查看命令。
|
|
16
|
-
|
|
17
23
|
```
|
|
24
|
+
|
|
25
|
+
Agent 会先探索你的项目,确认要测哪个接口,再写出三个文件并跑一次。
|
|
18
26
|
</Step>
|
|
19
|
-
<Step title="
|
|
20
|
-
|
|
27
|
+
<Step title="自己再跑一次">
|
|
28
|
+
用 Agent 最后列出的 Experiment ID 运行。它没有列出时,先用 `pnpm exec niceeval exp list` 查看有哪些 Experiment:
|
|
21
29
|
|
|
22
30
|
```bash
|
|
23
31
|
pnpm exec niceeval exp <experiment-id>
|
|
24
32
|
```
|
|
25
33
|
</Step>
|
|
26
|
-
<Step title="
|
|
34
|
+
<Step title="看结果">
|
|
27
35
|
```bash
|
|
28
|
-
pnpm exec niceeval
|
|
29
|
-
pnpm exec niceeval view # 网页审阅已发布的结果
|
|
36
|
+
pnpm exec niceeval show
|
|
30
37
|
```
|
|
38
|
+
|
|
39
|
+
终端里会列出每条评估用例的结果,失败的排在前面。结果怎么读,见下文[读懂结果](#读懂结果)。
|
|
31
40
|
</Step>
|
|
32
41
|
</Steps>
|
|
33
42
|
|
|
43
|
+
## 自己动手接入
|
|
44
|
+
|
|
45
|
+
### 准备被测应用
|
|
34
46
|
|
|
35
|
-
|
|
47
|
+
先把你的应用启动起来,并确认它有一个能调用的接口。下面的示例假设:
|
|
36
48
|
|
|
37
|
-
|
|
49
|
+
- 接口是 `POST http://localhost:3000/chat`;
|
|
50
|
+
- 请求体是 `{ "message": "...", "model": "..." }`;
|
|
51
|
+
- 响应体是 `{ "reply": "..." }`。
|
|
52
|
+
|
|
53
|
+
你的接口不一样也没关系,只需要改 Adapter 里的地址、请求体和读取回复的那一行。手边还没有能启动的应用时,先从 [Examples](/zh/examples) 挑一个可运行项目。
|
|
54
|
+
|
|
55
|
+
### 安装
|
|
38
56
|
|
|
39
57
|
```bash
|
|
40
58
|
pnpm add -D niceeval
|
|
41
59
|
pnpm exec niceeval init
|
|
42
60
|
```
|
|
43
61
|
|
|
44
|
-
|
|
45
|
-
|
|
62
|
+
`init` 会创建 `niceeval.config.ts`。它还会在 `AGENTS.md` 或 `CLAUDE.md` 里写一段说明,让你的 Coding Agent 读取和安装版本匹配的文档。
|
|
63
|
+
|
|
64
|
+
### 写 Adapter
|
|
65
|
+
|
|
66
|
+
Adapter 负责一件事:把评估用例里的消息发给你的应用,再把回复交回给 NiceEval。
|
|
46
67
|
|
|
47
68
|
```ts
|
|
48
69
|
// agents/my-agent.ts
|
|
@@ -52,11 +73,11 @@ export default defineAgent({
|
|
|
52
73
|
name: "my-agent",
|
|
53
74
|
evidenceCoverage: completeEvidenceCoverage,
|
|
54
75
|
async send(input, ctx) {
|
|
55
|
-
//
|
|
76
|
+
// 换成你自己应用的地址
|
|
56
77
|
const r = await fetch("http://localhost:3000/chat", {
|
|
57
78
|
method: "POST",
|
|
58
79
|
headers: { "content-type": "application/json" },
|
|
59
|
-
body: JSON.stringify({ message: input.text, model: ctx.model }), //
|
|
80
|
+
body: JSON.stringify({ message: input.text, model: ctx.model }), // ctx.model 来自 Experiment
|
|
60
81
|
signal: ctx.signal,
|
|
61
82
|
});
|
|
62
83
|
const body = await r.json();
|
|
@@ -68,7 +89,9 @@ export default defineAgent({
|
|
|
68
89
|
});
|
|
69
90
|
```
|
|
70
91
|
|
|
71
|
-
|
|
92
|
+
`events` 是 NiceEval 读懂回复的方式。这里只交回一条助手消息,后面可以再加工具调用、用量等事件。
|
|
93
|
+
|
|
94
|
+
### 写 Experiment
|
|
72
95
|
|
|
73
96
|
```ts
|
|
74
97
|
// experiments/my-agent.ts
|
|
@@ -77,13 +100,15 @@ import myAgent from "../agents/my-agent.ts";
|
|
|
77
100
|
|
|
78
101
|
export default defineExperiment({
|
|
79
102
|
description: "my-agent 基线",
|
|
80
|
-
model: "gpt-4o",
|
|
81
103
|
agent: myAgent,
|
|
104
|
+
model: "gpt-4o",
|
|
82
105
|
attempts: 1,
|
|
83
106
|
});
|
|
84
107
|
```
|
|
85
108
|
|
|
86
|
-
|
|
109
|
+
文件名就是 Experiment ID,这里是 `my-agent`。以后想对比另一个模型,复制一份改 `model` 就行,评估用例不用动。
|
|
110
|
+
|
|
111
|
+
### 写评估用例
|
|
87
112
|
|
|
88
113
|
```ts
|
|
89
114
|
// evals/refund-policy.eval.ts
|
|
@@ -91,44 +116,63 @@ import { defineEval } from "niceeval";
|
|
|
91
116
|
import { pattern } from "niceeval/expect";
|
|
92
117
|
|
|
93
118
|
export default defineEval({
|
|
94
|
-
description: "
|
|
119
|
+
description: "能正确回答退款政策",
|
|
95
120
|
async test(t) {
|
|
96
|
-
await t.send("
|
|
121
|
+
await t.send("你们的退款政策是什么?");
|
|
97
122
|
t.succeeded();
|
|
98
|
-
t.check(t.reply, pattern(/30\s*天|一个月/));
|
|
123
|
+
t.check(t.reply, pattern(/30\s*天|一个月/));
|
|
99
124
|
},
|
|
100
125
|
});
|
|
101
126
|
```
|
|
102
127
|
|
|
128
|
+
模型每次的措辞都会变,所以这里用正则匹配“30 天”或“一个月”,而不是要求一字不差。
|
|
129
|
+
|
|
130
|
+
### 运行
|
|
131
|
+
|
|
103
132
|
```bash
|
|
104
|
-
pnpm exec niceeval exp my-agent
|
|
105
|
-
pnpm exec niceeval query discover # 让机器发现可用 operation
|
|
106
|
-
pnpm exec niceeval view # 在网页审阅当前已发布的结果
|
|
133
|
+
pnpm exec niceeval exp my-agent
|
|
107
134
|
```
|
|
108
135
|
|
|
109
|
-
|
|
136
|
+
运行过程中终端会显示进度。结束后你会看到类似这样的输出:
|
|
110
137
|
|
|
111
|
-
|
|
138
|
+
```text
|
|
139
|
+
╭─ PASSED ─────────────────────────────────────────────────────── 12s ─╮
|
|
140
|
+
│ 1 passed · 0 failed · 0 errored (0 reused) │
|
|
141
|
+
╰──────────────────────────────────────────────────────────────────────╯
|
|
112
142
|
|
|
113
|
-
|
|
143
|
+
╭─ RESULTS ───────────────────────────────────── 1 run configuration ─╮
|
|
144
|
+
│ my-agent │
|
|
145
|
+
│ refund-policy 1/1 passed │
|
|
146
|
+
╰──────────────────────────────────────────────────────────────────────╯
|
|
114
147
|
|
|
115
|
-
|
|
148
|
+
╭─ NEXT ───────────────────────────────────────────────────────────────╮
|
|
149
|
+
│ my-agent │
|
|
150
|
+
│ show: niceeval show --run 8f3d6f62-1d34-4cf3-99c7-84ba3c483706 │
|
|
151
|
+
│ view: niceeval view --run 8f3d6f62-1d34-4cf3-99c7-84ba3c483706 │
|
|
152
|
+
╰──────────────────────────────────────────────────────────────────────╯
|
|
153
|
+
```
|
|
116
154
|
|
|
117
|
-
|
|
155
|
+
到这里,第一条评估用例已经跑通。
|
|
118
156
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
157
|
+
## 读懂结果
|
|
158
|
+
|
|
159
|
+
标题告诉你这次运行的结论:
|
|
160
|
+
|
|
161
|
+
| 标题 | 意思 | 先做什么 |
|
|
162
|
+
|---|---|---|
|
|
163
|
+
| `PASSED` | 所有断言都满足了。 | 继续加评估用例。 |
|
|
164
|
+
| `FAILED` | 应用正常回复了,但至少一条断言没满足。 | 看 Agent 实际回了什么,判断是应用的问题还是断言写得太严。 |
|
|
165
|
+
| `ERRORED` | 没能拿到结果,例如应用没启动、接口报错、Adapter 抛了异常。 | 看 `FAILURES` 面板里的 `error:` 行,通常能直接看出原因;需要更多细节时运行它下面的 `details:` 命令。 |
|
|
166
|
+
|
|
167
|
+
失败的 Attempt 下面都有一行 `details: niceeval show @<locator>`,复制它就能在终端里看到这一次运行的对话和断言。看整次运行,复制 `NEXT` 面板里的 `show:` 命令;想在浏览器里翻对话、看耗时,复制 `view:` 那一行。
|
|
168
|
+
|
|
169
|
+
更多查看方式见[查看运行结果](/zh/tutorials/viewing-results)。
|
|
170
|
+
|
|
171
|
+
## 接下来
|
|
172
|
+
|
|
173
|
+
这个最小 Adapter 只支持单轮对话,第二轮不会带上第一轮的历史,也看不到工具调用。按你的下一步需要选:
|
|
131
174
|
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
175
|
+
- 要测多轮对话、工具调用或用量:按[编写 Send](/zh/tutorials/write-send)扩展 Adapter,已有评估用例不用改。
|
|
176
|
+
- 要写更多、更有区分度的评估用例:读[编写评估用例](/zh/tutorials/authoring)。
|
|
177
|
+
- 要比较两个模型或两版 prompt:读[编写实验](/zh/tutorials/write-experiment)。
|
|
178
|
+
- 要在 PR 上自动跑:读 [CI 集成](/zh/tutorials/ci-integration)。
|
|
@@ -1,63 +1,78 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "
|
|
2
|
+
title: "把运行结果上报给 Braintrust 等外部系统"
|
|
3
3
|
sidebarTitle: "Reporter 上报"
|
|
4
|
-
description: "
|
|
4
|
+
description: "团队已经在用 Braintrust 或自己的看板时,用 Reporter 在运行过程中把结果发过去;CI 平台用 JUnit;后续脚本用 receipt 里的 Run ID 读取结果。"
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
你的团队已经在 Braintrust 上看实验结果,或者有一个内部看板。你希望 NiceEval 跑完后,结果自动出现在那里,不用手动导出。
|
|
8
|
+
|
|
9
|
+
Reporter 做的就是这件事:在这一次运行过程中,把结果发给外部系统。它只负责“这一次”的上报,NiceEval 保存的结果不会因此改变。事后要重新读取、比较或分享结果,仍然用 `show`、`view` 和 `query`。
|
|
10
|
+
|
|
11
|
+
一次 `niceeval exp` 会同时产出这几样东西:
|
|
8
12
|
|
|
9
13
|
```text
|
|
10
|
-
|
|
11
|
-
├─
|
|
14
|
+
这一次运行
|
|
15
|
+
├─ 终端输出,或 --json 的 NDJSON
|
|
12
16
|
├─ Reporter → 外部系统
|
|
13
|
-
├─ --junit → CI
|
|
14
|
-
└─ receipt → invocationId 与
|
|
17
|
+
├─ --junit → 给 CI 平台的 JUnit 文件
|
|
18
|
+
└─ receipt → invocationId 与 createdRunIds
|
|
15
19
|
|
|
16
|
-
|
|
17
|
-
└─ Run、
|
|
20
|
+
保存下来的结果(.niceeval/record.sqlite)
|
|
21
|
+
└─ Run、Attempt 和可以查看的运行记录
|
|
18
22
|
```
|
|
19
23
|
|
|
20
|
-
##
|
|
24
|
+
## 选择合适的上报方式
|
|
25
|
+
|
|
26
|
+
| 你要 | 用 |
|
|
27
|
+
| --- | --- |
|
|
28
|
+
| 把结果推到 Braintrust 或内部看板 | Reporter |
|
|
29
|
+
| 让 CI 平台按测试用例显示结果 | `--junit <path>` |
|
|
30
|
+
| 让后续脚本找到这次运行的结果 | `--json` 输出的 receipt |
|
|
31
|
+
|
|
32
|
+
## 用 Reporter 推送到外部系统
|
|
21
33
|
|
|
22
|
-
|
|
34
|
+
在 `niceeval.config.ts` 的 `reporters` 里配置项目默认的 Reporter,评估用例的 `reporters` 会和它合并。字段说明见 [defineConfig 参考](/zh/reference/define-config)。
|
|
35
|
+
|
|
36
|
+
Reporter 能拿到的是这次运行过程中已知的结果。外部系统可以接收运行摘要、每个 Attempt 的结果或查看链接。要读取完整的结果,用 receipt 里的 Run ID 重新打开保存下来的结果。
|
|
23
37
|
|
|
24
38
|
Reporter 不能:
|
|
25
39
|
|
|
26
|
-
-
|
|
27
|
-
-
|
|
28
|
-
-
|
|
29
|
-
-
|
|
40
|
+
- 把外部系统里的数据当作 NiceEval 的结果;
|
|
41
|
+
- 代替 NiceEval 保存的判定、用量、断言或诊断;
|
|
42
|
+
- 让 `query`、`show` 或 `view` 改从外部系统读数据;
|
|
43
|
+
- 把外部系统的结果写回 `.niceeval/record.sqlite`。
|
|
30
44
|
|
|
31
|
-
## 在 CI
|
|
45
|
+
## 在 CI 中输出 JUnit
|
|
32
46
|
|
|
33
|
-
JUnit
|
|
47
|
+
接入 CI 平台最直接的方式是 JUnit:
|
|
34
48
|
|
|
35
49
|
```sh
|
|
36
|
-
|
|
50
|
+
pnpm exec niceeval exp ci --junit ./artifacts/niceeval-junit.xml
|
|
37
51
|
```
|
|
38
52
|
|
|
39
|
-
NiceEval
|
|
53
|
+
NiceEval 先写临时文件,再一次性替换目标文件,CI 平台不会读到写了一半的内容。JUnit 只描述这一次运行,结果仍以 NiceEval 保存的为准。完整的 CI 配置见[在 CI 里自动跑评估](/zh/tutorials/ci-integration)。
|
|
40
54
|
|
|
41
|
-
##
|
|
55
|
+
## 用 receipt 关联后续步骤
|
|
42
56
|
|
|
43
57
|
```sh
|
|
44
|
-
|
|
58
|
+
pnpm exec niceeval exp checkout --json
|
|
45
59
|
```
|
|
46
60
|
|
|
47
|
-
`--json`
|
|
61
|
+
`--json` 输出这次运行的 NDJSON。最后一行是 receipt,包含 `invocationId`、`createdRunIds`、`publicationCutoff`、开始时间、结束时间和 `completion`。外部自动化可以保存 receipt,再用 `createdRunIds` 打开对应的 Run:
|
|
48
62
|
|
|
49
63
|
```sh
|
|
64
|
+
pnpm exec niceeval show --run 01J9ZK3M6P4T7V9X2C5N8QW0RY
|
|
50
65
|
pnpm exec niceeval view --run 01J9ZK3M6P4T7V9X2C5N8QW0RY
|
|
51
66
|
```
|
|
52
67
|
|
|
53
|
-
|
|
68
|
+
脚本需要结构化结果时,先运行 `pnpm --silent exec niceeval query discover` 找到可用的查询。
|
|
54
69
|
|
|
55
|
-
|
|
70
|
+
不要把进度或诊断行当作长期保存的结果格式。它们只描述这一次运行正在发生什么。
|
|
56
71
|
|
|
57
|
-
|
|
58
|
-
2. 将凭据保留在运行变量中,不写入输出内容。
|
|
59
|
-
3. 用退出状态处理 CI 门禁。
|
|
60
|
-
4. 用 receipt 的 `runIds` 把外部工作流关联到可查看的结果。
|
|
61
|
-
5. 需要审阅已发布事实时,使用 receipt 的 `runIds` 打开 `view`,或用 `query` 交给自动化流程。
|
|
72
|
+
## 接入外部系统时的原则
|
|
62
73
|
|
|
63
|
-
|
|
74
|
+
1. 外部上报只使用这次运行中已知的信息。
|
|
75
|
+
2. 凭据放在环境变量里,不写进任何输出。
|
|
76
|
+
3. CI 门禁看退出状态。
|
|
77
|
+
4. 用 receipt 的 `createdRunIds` 把外部工作流和 NiceEval 保存的结果关联起来。
|
|
78
|
+
5. 需要审阅结果时,人用 `show` 或 `view` 打开那个 Run,自动化流程用 `query`。
|
|
@@ -1,81 +1,80 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "
|
|
2
|
+
title: "修改后只重跑受影响的评估"
|
|
3
3
|
sidebarTitle: "重跑与沿用"
|
|
4
|
-
description: "
|
|
4
|
+
description: "改了 prompt、评估用例或模型之后,用 --dry 看清哪些结果会沿用、哪些会重跑,再用 --rerun 和 accept 控制范围。"
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
你修好了 Agent 的一个工具调用 bug,想确认它真的修好了。一整套评估要跑一个小时、花几十美元,而这次改动只影响其中几道题。
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
| --- | --- |
|
|
11
|
-
| `origin` | Attempt 的实际执行 slot 就是当前 slot。 |
|
|
12
|
-
| `reference` | 采用了其它 Run 中已发布的 Attempt。 |
|
|
9
|
+
NiceEval 会记住每道题上次的结果。再次运行同一个 Experiment 时,没受这次改动影响、上次已经有结论的题直接沿用旧结果,其余的重新跑。你不需要手动挑题。
|
|
13
10
|
|
|
14
|
-
|
|
11
|
+
## 运行前先看计划
|
|
15
12
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
在运行前使用 `--dry`。它根据已发布结果逐个列出执行或采用的原因。
|
|
13
|
+
加上 `--dry`,NiceEval 只列计划,不调用模型,也不创建 Sandbox:
|
|
19
14
|
|
|
20
15
|
```sh
|
|
21
|
-
|
|
16
|
+
pnpm exec niceeval exp checkout --dry
|
|
22
17
|
```
|
|
23
18
|
|
|
24
|
-
示例输出:
|
|
25
|
-
|
|
26
19
|
```text
|
|
27
20
|
PLAN
|
|
28
21
|
checkout cart/add-item ordinal 0 reuse @1K1P0VJAPVJ12
|
|
29
22
|
checkout cart/add-item ordinal 1 execute: input-identity-changed
|
|
30
23
|
```
|
|
31
24
|
|
|
32
|
-
|
|
25
|
+
每一行是一次 Attempt。`reuse` 后面是要沿用的旧结果 locator;`execute` 后面是要重跑的原因。上面第二行的 `input-identity-changed` 表示这道题的输入变了,例如你改了评估用例、Fixture 或模型。
|
|
33
26
|
|
|
34
|
-
|
|
27
|
+
结果和你预期的不一样时,先看原因。例如你只改了报告里的分组名,却看到整批 `execute`,说明某个会影响结果的输入也一起变了。
|
|
35
28
|
|
|
36
|
-
|
|
29
|
+
## 哪些结果会被沿用
|
|
37
30
|
|
|
38
|
-
|
|
39
|
-
- 输入身份与当前输入相同;
|
|
40
|
-
- 执行时长符合当前 timeout;
|
|
41
|
-
- 本次 `--rerun` 允许采用该 Verdict;
|
|
42
|
-
- 本次没有通过 `--keep-sandbox` 要求真实执行。
|
|
31
|
+
一个旧 Attempt 同时满足下面几条才会被自动沿用:
|
|
43
32
|
|
|
44
|
-
|
|
33
|
+
- 它已经有明确结论,`passed` 或 `failed`。`errored` 说明上次没跑成,总会重跑。
|
|
34
|
+
- 评估用例、Fixture、模型、Agent、判分标准和运行条件都和这次相同。
|
|
35
|
+
- 它的耗时没有超过这次的 `timeout`。
|
|
36
|
+
- 这次的 `--rerun` 没有要求重跑它。
|
|
37
|
+
- 这次没有用 `--keep-sandbox` 要求真实执行。
|
|
45
38
|
|
|
46
|
-
|
|
39
|
+
只改 Experiment 的 `description` 或 `labels` 不会让旧结果失效。评估用例文件不同:文件内容一变,这道题就会重跑。
|
|
47
40
|
|
|
48
41
|
## 控制重跑范围
|
|
49
42
|
|
|
50
43
|
```sh
|
|
51
|
-
#
|
|
52
|
-
|
|
44
|
+
# 默认:沿用符合条件的 passed 和 failed
|
|
45
|
+
pnpm exec niceeval exp checkout
|
|
53
46
|
|
|
54
|
-
#
|
|
55
|
-
|
|
47
|
+
# 修复 bug 后复验:重跑上次 failed 的,passed 的继续沿用
|
|
48
|
+
pnpm exec niceeval exp checkout --rerun
|
|
56
49
|
|
|
57
|
-
#
|
|
58
|
-
|
|
50
|
+
# 模型供应商更新了服务、本地文件却没变:全部重跑
|
|
51
|
+
pnpm exec niceeval exp checkout --rerun all
|
|
59
52
|
```
|
|
60
53
|
|
|
61
|
-
|
|
54
|
+
单独写 `--rerun` 等于 `--rerun failed`,已经通过的题不会重跑。要连通过的一起重做,必须写 `--rerun all`。
|
|
62
55
|
|
|
63
|
-
|
|
56
|
+
`--rerun` 只影响这一次运行,不会改动已有的结果。
|
|
64
57
|
|
|
65
|
-
##
|
|
58
|
+
## 确认旧结果在新配置下仍然成立
|
|
66
59
|
|
|
67
|
-
|
|
60
|
+
有时改动会让 NiceEval 判定旧结果不能沿用,但你知道它其实不受影响。例如你给 Agent 加了一个日志开关,这道题的行为完全没变。NiceEval 不会替你猜,需要你明确接受:
|
|
68
61
|
|
|
69
62
|
```sh
|
|
70
|
-
|
|
63
|
+
pnpm exec niceeval accept @1K1P0VJAPVJ12
|
|
71
64
|
```
|
|
72
65
|
|
|
73
|
-
`accept`
|
|
66
|
+
`accept` 会新建一个 Run 引用这个旧 Attempt,原 Attempt 保持不变。要接受多条时,把所有 locator 写进同一条命令,NiceEval 会先检查全部输入,不会只接受一半。
|
|
67
|
+
|
|
68
|
+
怎样取得候选 locator、接受前要核对什么,见[审阅并采用历史 Attempt](/zh/tutorials/accept-results)。
|
|
74
69
|
|
|
75
|
-
|
|
70
|
+
## 跑完之后看结果
|
|
76
71
|
|
|
77
|
-
|
|
72
|
+
运行结束时,终端的 `NEXT` 面板会列出这次新建的 Run,并给出在终端和浏览器里查看它的两条命令:
|
|
73
|
+
|
|
74
|
+
```sh
|
|
75
|
+
pnpm exec niceeval show --run <run-id>
|
|
76
|
+
```
|
|
78
77
|
|
|
79
|
-
|
|
78
|
+
沿用的题和重跑的题都会出现在这个 Run 里,通过率按完整的一批题计算。不要在 `exp` 还没结束时用 `show --experiment` 看结果,那时可能还没有任何已发布的结果。
|
|
80
79
|
|
|
81
|
-
|
|
80
|
+
查看结果的完整方式见[查看运行结果](/zh/tutorials/viewing-results)。沿用和重跑的判定原理见 [Runner](/zh/explanation/runner)。
|