niceeval 0.13.2-canary.35 → 0.13.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INDEX.md +24 -2
- package/dist/agents/bub-install-spec.cjs +19 -5
- package/dist/agents/bub-install-spec.cjs.map +1 -1
- package/dist/agents/bub-install-spec.d.cts +9 -2
- package/dist/agents/bub-install-spec.d.mts +9 -2
- package/dist/agents/bub-install-spec.d.ts +9 -2
- package/dist/agents/bub-install-spec.js +14 -12
- package/dist/agents/bub.cjs +10 -6
- package/dist/agents/bub.cjs.map +1 -1
- package/dist/agents/bub.d.cts +2 -2
- package/dist/agents/bub.d.mts +2 -2
- package/dist/agents/bub.d.ts +2 -2
- package/dist/agents/claude-code.cjs.map +1 -1
- package/dist/agents/claude-code.d.cts +2 -2
- package/dist/agents/claude-code.d.mts +2 -2
- package/dist/agents/claude-code.d.ts +2 -2
- package/dist/agents/codex.cjs +4 -4
- package/dist/agents/codex.cjs.map +1 -1
- package/dist/agents/codex.d.cts +3 -3
- package/dist/agents/codex.d.mts +3 -3
- package/dist/agents/codex.d.ts +3 -3
- package/dist/agents/deepseek-harness.cjs +50 -2
- package/dist/agents/deepseek-harness.cjs.map +1 -1
- package/dist/agents/deepseek-harness.d.cts +2 -0
- package/dist/agents/deepseek-harness.d.mts +2 -0
- package/dist/agents/deepseek-harness.d.ts +2 -0
- package/dist/agents/exact-npm-plugins.cjs +28 -0
- package/dist/agents/exact-npm-plugins.cjs.map +1 -0
- package/dist/agents/exact-npm-plugins.d.cts +10 -0
- package/dist/agents/exact-npm-plugins.d.mts +10 -0
- package/dist/agents/exact-npm-plugins.d.ts +10 -0
- package/dist/agents/exact-npm-plugins.js +8 -0
- package/dist/agents/exact-npm-plugins.js.map +1 -0
- package/dist/agents/openclaw.cjs +51 -2
- package/dist/agents/openclaw.cjs.map +1 -1
- package/dist/agents/openclaw.d.cts +2 -0
- package/dist/agents/openclaw.d.mts +2 -0
- package/dist/agents/openclaw.d.ts +2 -0
- package/dist/agents/types.cjs.map +1 -1
- package/dist/agents/types.d.cts +2 -2
- package/dist/agents/types.d.mts +2 -2
- package/dist/agents/types.d.ts +2 -2
- package/dist/agents/ui-message-stream.cjs +24 -7
- package/dist/agents/ui-message-stream.cjs.map +1 -1
- package/dist/analysis/api.cjs +19 -1
- package/dist/analysis/api.cjs.map +1 -1
- package/dist/analysis/api.d.cts +2 -2
- package/dist/analysis/api.d.mts +2 -2
- package/dist/analysis/api.d.ts +2 -2
- package/dist/analysis/bindings.cjs +11 -18
- package/dist/analysis/bindings.cjs.map +1 -1
- package/dist/analysis/bindings.d.cts +177 -49
- package/dist/analysis/bindings.d.mts +177 -49
- package/dist/analysis/bindings.d.ts +177 -49
- package/dist/analysis/contracts.cjs +1 -0
- package/dist/analysis/contracts.cjs.map +1 -1
- package/dist/analysis/contracts.d.cts +43 -3
- package/dist/analysis/contracts.d.mts +43 -3
- package/dist/analysis/contracts.d.ts +43 -3
- package/dist/analysis/cost-projection.cjs +2 -1
- package/dist/analysis/cost-projection.cjs.map +1 -1
- package/dist/analysis/cost-projection.d.cts +1 -1
- package/dist/analysis/cost-projection.d.mts +1 -1
- package/dist/analysis/cost-projection.d.ts +1 -1
- package/dist/analysis/cost.cjs +13 -3
- package/dist/analysis/cost.cjs.map +1 -1
- package/dist/analysis/cost.d.cts +10 -3
- package/dist/analysis/cost.d.mts +10 -3
- package/dist/analysis/cost.d.ts +10 -3
- package/dist/analysis/experiment-groups.cjs +96 -0
- package/dist/analysis/experiment-groups.cjs.map +1 -0
- package/dist/analysis/experiment-groups.d.cts +9 -0
- package/dist/analysis/experiment-groups.d.mts +9 -0
- package/dist/analysis/experiment-groups.d.ts +9 -0
- package/dist/analysis/experiment-groups.js +10 -0
- package/dist/analysis/experiment-groups.js.map +1 -0
- package/dist/analysis/index.cjs +4 -1
- package/dist/analysis/index.cjs.map +1 -1
- package/dist/analysis/index.d.cts +4 -2
- package/dist/analysis/index.d.mts +4 -2
- package/dist/analysis/index.d.ts +4 -2
- package/dist/analysis/index.js +40 -36
- package/dist/analysis/index.mjs +40 -36
- package/dist/assertions/limits.cjs +4 -0
- package/dist/assertions/limits.cjs.map +1 -1
- package/dist/assertions/limits.d.cts +4 -0
- package/dist/assertions/limits.d.mts +4 -0
- package/dist/assertions/limits.d.ts +4 -0
- package/dist/assertions/match.cjs +63 -25
- package/dist/assertions/match.cjs.map +1 -1
- package/dist/assertions/match.d.cts +7 -0
- package/dist/assertions/match.d.mts +7 -0
- package/dist/assertions/match.d.ts +7 -0
- package/dist/assertions/record/attachment.cjs +32 -4
- package/dist/assertions/record/attachment.cjs.map +1 -1
- package/dist/assertions/record/diff.cjs +10 -4
- package/dist/assertions/record/diff.cjs.map +1 -1
- package/dist/assertions/record/producer.cjs +101 -4
- package/dist/assertions/record/producer.cjs.map +1 -1
- package/dist/assertions/record/producer.d.cts +4 -1
- package/dist/assertions/record/producer.d.mts +4 -1
- package/dist/assertions/record/producer.d.ts +4 -1
- package/dist/assertions/runtime.cjs +45 -5
- package/dist/assertions/runtime.cjs.map +1 -1
- package/dist/cli/record.cjs +113 -8
- package/dist/cli/record.cjs.map +1 -1
- package/dist/cli/record.d.cts +20 -1
- package/dist/cli/record.d.mts +20 -1
- package/dist/cli/record.d.ts +20 -1
- package/dist/cli/record.js +4 -2
- package/dist/cli.cjs +386 -31
- package/dist/cli.cjs.map +1 -1
- package/dist/cli.d.cts +10 -2
- package/dist/cli.d.mts +10 -2
- package/dist/cli.d.ts +10 -2
- package/dist/context/send-retry.cjs +4 -4
- package/dist/context/send-retry.cjs.map +1 -1
- package/dist/coordination/host/runtime.cjs +8 -30
- package/dist/coordination/host/runtime.cjs.map +1 -1
- package/dist/coordination/host/runtime.d.cts +3 -3
- package/dist/coordination/host/runtime.d.mts +3 -3
- package/dist/coordination/host/runtime.d.ts +3 -3
- package/dist/coordination/host/types.cjs.map +1 -1
- package/dist/coordination/host/types.d.cts +0 -5
- package/dist/coordination/host/types.d.mts +0 -5
- package/dist/coordination/host/types.d.ts +0 -5
- package/dist/coordination/platform/node.cjs +90 -9
- package/dist/coordination/platform/node.cjs.map +1 -1
- package/dist/define.cjs +25 -1
- package/dist/define.cjs.map +1 -1
- package/dist/experiment/host/index.cjs +1 -0
- package/dist/experiment/host/index.cjs.map +1 -1
- package/dist/i18n/en.cjs +19 -6
- package/dist/i18n/en.cjs.map +1 -1
- package/dist/i18n/en.d.cts +12 -4
- package/dist/i18n/en.d.mts +12 -4
- package/dist/i18n/en.d.ts +12 -4
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +1 -1
- package/dist/index.d.mts +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/o11y/prices.json +1 -1
- package/dist/o11y/record/families.cjs +12 -3
- package/dist/o11y/record/families.cjs.map +1 -1
- package/dist/o11y/record/families.d.cts +8 -8
- package/dist/o11y/record/families.d.mts +8 -8
- package/dist/o11y/record/families.d.ts +8 -8
- package/dist/o11y/record/family-projectors.d.cts +66 -18
- package/dist/o11y/record/family-projectors.d.mts +66 -18
- package/dist/o11y/record/family-projectors.d.ts +66 -18
- package/dist/o11y/record/family-writers.d.cts +11 -3
- package/dist/o11y/record/family-writers.d.mts +11 -3
- package/dist/o11y/record/family-writers.d.ts +11 -3
- package/dist/o11y/record/runner-producer.cjs +14 -4
- package/dist/o11y/record/runner-producer.cjs.map +1 -1
- package/dist/record/codec/core.d.cts +2 -2
- package/dist/record/codec/core.d.mts +2 -2
- package/dist/record/codec/core.d.ts +2 -2
- package/dist/record/family/catalog.d.cts +44 -12
- package/dist/record/family/catalog.d.mts +44 -12
- package/dist/record/family/catalog.d.ts +44 -12
- package/dist/record/family/index.js +128 -124
- package/dist/record/family/observability-v1.cjs +35 -0
- package/dist/record/family/observability-v1.cjs.map +1 -0
- package/dist/record/family/observability-v1.d.cts +3 -0
- package/dist/record/family/observability-v1.d.mts +3 -0
- package/dist/record/family/observability-v1.d.ts +3 -0
- package/dist/record/family/observability-v1.js +6 -0
- package/dist/record/family/observability-v1.js.map +1 -0
- package/dist/record/family/observability.cjs +58 -9
- package/dist/record/family/observability.cjs.map +1 -1
- package/dist/record/family/observability.d.cts +1014 -15
- package/dist/record/family/observability.d.mts +1014 -15
- package/dist/record/family/observability.d.ts +1014 -15
- package/dist/record/family/observability.js +22 -18
- package/dist/record/host/runtime.cjs +600 -31
- package/dist/record/host/runtime.cjs.map +1 -1
- package/dist/record/host/types.cjs.map +1 -1
- package/dist/record/host/types.d.cts +33 -9
- package/dist/record/host/types.d.mts +33 -9
- package/dist/record/host/types.d.ts +33 -9
- package/dist/record/maintenance/runtime.cjs +1 -2
- package/dist/record/maintenance/runtime.cjs.map +1 -1
- package/dist/record/model/core.cjs.map +1 -1
- package/dist/record/model/core.d.cts +1 -1
- package/dist/record/model/core.d.mts +1 -1
- package/dist/record/model/core.d.ts +1 -1
- package/dist/record/model/definition.d.cts +6 -6
- package/dist/record/model/definition.d.mts +6 -6
- package/dist/record/model/definition.d.ts +6 -6
- package/dist/record/model/identifiers.cjs +1 -1
- package/dist/record/model/identifiers.cjs.map +1 -1
- package/dist/record/model/identifiers.d.cts +1 -1
- package/dist/record/model/identifiers.d.mts +1 -1
- package/dist/record/model/identifiers.d.ts +1 -1
- package/dist/record/platform/node.cjs +159 -23
- package/dist/record/platform/node.cjs.map +1 -1
- package/dist/record/platform/services.cjs.map +1 -1
- package/dist/record/platform/services.d.cts +15 -2
- package/dist/record/platform/services.d.mts +15 -2
- package/dist/record/platform/services.d.ts +15 -2
- package/dist/record/reader/errors.cjs +37 -1
- package/dist/record/reader/errors.cjs.map +1 -1
- package/dist/record/reader/errors.d.cts +49 -1
- package/dist/record/reader/errors.d.mts +49 -1
- package/dist/record/reader/errors.d.ts +49 -1
- package/dist/record/reader/errors.js +22 -12
- package/dist/record/reader/format.cjs +76 -17
- package/dist/record/reader/format.cjs.map +1 -1
- package/dist/record/reader/format.d.cts +11 -4
- package/dist/record/reader/format.d.mts +11 -4
- package/dist/record/reader/format.d.ts +11 -4
- package/dist/record/reader/format.js +2 -0
- package/dist/record/reader/index.js +32 -18
- package/dist/record/reader/runtime.cjs +64 -0
- package/dist/record/reader/runtime.cjs.map +1 -1
- package/dist/record/reader/runtime.d.cts +15 -0
- package/dist/record/reader/runtime.d.mts +15 -0
- package/dist/record/reader/runtime.d.ts +15 -0
- package/dist/record/reader/runtime.js +2 -0
- package/dist/report/assets/styles.css +53 -29
- package/dist/report/built-in/analysis-values.cjs +0 -8
- package/dist/report/built-in/analysis-values.cjs.map +1 -1
- package/dist/report/built-in/analysis-values.d.cts +0 -13
- package/dist/report/built-in/analysis-values.d.mts +0 -13
- package/dist/report/built-in/analysis-values.d.ts +0 -13
- package/dist/report/built-in/analysis-values.js +4 -6
- package/dist/report/built-in/index.cjs +1 -3
- package/dist/report/built-in/index.cjs.map +1 -1
- package/dist/report/built-in/index.d.cts +1 -1
- package/dist/report/built-in/index.d.mts +1 -1
- package/dist/report/built-in/index.d.ts +1 -1
- package/dist/report/built-in/index.js +4 -8
- package/dist/report/built-in/index.mjs +4 -8
- package/dist/report/built-in/machine.cjs +0 -2
- package/dist/report/built-in/machine.cjs.map +1 -1
- package/dist/report/built-in/machine.d.cts +0 -2
- package/dist/report/built-in/machine.d.mts +0 -2
- package/dist/report/built-in/machine.d.ts +0 -2
- package/dist/report/built-in/result-components.cjs +25 -73
- package/dist/report/built-in/result-components.cjs.map +1 -1
- package/dist/report/built-in/result-components.d.cts +3 -19
- package/dist/report/built-in/result-components.d.mts +3 -19
- package/dist/report/built-in/result-components.d.ts +3 -19
- package/dist/report/built-in/result-components.js +0 -4
- package/dist/report/built-in/run-membership-overview.cjs +30 -17
- package/dist/report/built-in/run-membership-overview.cjs.map +1 -1
- package/dist/report/built-in/run-membership-overview.d.cts +14 -0
- package/dist/report/built-in/run-membership-overview.d.mts +14 -0
- package/dist/report/built-in/run-membership-overview.d.ts +14 -0
- package/dist/report/built-in/run-membership-overview.js +4 -2
- package/dist/report/built-in/standard.cjs +80 -46
- package/dist/report/built-in/standard.cjs.map +1 -1
- package/dist/report/built-in/standard.d.cts +19 -42
- package/dist/report/built-in/standard.d.mts +19 -42
- package/dist/report/built-in/standard.d.ts +19 -42
- package/dist/report/built-in/standard.js +6 -6
- package/dist/report/components/attempt-detail/compute.cjs +78 -2
- package/dist/report/components/attempt-detail/compute.cjs.map +1 -1
- package/dist/report/components/attempt-detail/compute.d.cts +1 -1
- package/dist/report/components/attempt-detail/compute.d.mts +1 -1
- package/dist/report/components/attempt-detail/compute.d.ts +1 -1
- package/dist/report/components/attempt-detail/content.cjs +1 -1
- package/dist/report/components/attempt-detail/content.cjs.map +1 -1
- package/dist/report/components/entity-lists/compute.cjs +2 -2
- package/dist/report/components/entity-lists/compute.cjs.map +1 -1
- package/dist/report/components/entity-lists/compute.d.cts +2 -2
- package/dist/report/components/entity-lists/compute.d.mts +2 -2
- package/dist/report/components/entity-lists/compute.d.ts +2 -2
- package/dist/report/components/entity-lists/index.cjs +6 -2
- package/dist/report/components/entity-lists/index.cjs.map +1 -1
- package/dist/report/components/entity-lists/index.d.cts +3 -6
- package/dist/report/components/entity-lists/index.d.mts +3 -6
- package/dist/report/components/entity-lists/index.d.ts +3 -6
- package/dist/report/components/summaries/compute.cjs +8 -1
- package/dist/report/components/summaries/compute.cjs.map +1 -1
- package/dist/report/components/summaries/compute.d.cts +9 -1
- package/dist/report/components/summaries/compute.d.mts +9 -1
- package/dist/report/components/summaries/compute.d.ts +9 -1
- package/dist/report/components/summaries/index.cjs +48 -4
- package/dist/report/components/summaries/index.cjs.map +1 -1
- package/dist/report/components/summaries/index.d.cts +7 -4
- package/dist/report/components/summaries/index.d.mts +7 -4
- package/dist/report/components/summaries/index.d.ts +7 -4
- package/dist/report/definition/primitives/chart.cjs +8 -5
- package/dist/report/definition/primitives/chart.cjs.map +1 -1
- package/dist/report/definition/primitives/points-dataset.cjs +1 -0
- package/dist/report/definition/primitives/points-dataset.cjs.map +1 -1
- package/dist/report/definition/primitives.cjs +6 -0
- package/dist/report/definition/primitives.cjs.map +1 -1
- package/dist/report/definition/report.cjs +27 -3
- package/dist/report/definition/report.cjs.map +1 -1
- package/dist/report/definition/report.d.cts +8 -0
- package/dist/report/definition/report.d.mts +8 -0
- package/dist/report/definition/report.d.ts +8 -0
- package/dist/report/execution/machine.cjs +30 -14
- package/dist/report/execution/machine.cjs.map +1 -1
- package/dist/report/execution/machine.d.cts +20 -10
- package/dist/report/execution/machine.d.mts +20 -10
- package/dist/report/execution/machine.d.ts +20 -10
- package/dist/report/execution/machine.js +6 -6
- package/dist/report/host/execute.cjs +25 -16
- package/dist/report/host/execute.cjs.map +1 -1
- package/dist/report/host/execute.d.cts +4 -0
- package/dist/report/host/execute.d.mts +4 -0
- package/dist/report/host/execute.d.ts +4 -0
- package/dist/report/host/from-record.cjs +22 -5
- package/dist/report/host/from-record.cjs.map +1 -1
- package/dist/report/host/machine.cjs +53 -11
- package/dist/report/host/machine.cjs.map +1 -1
- package/dist/report/host/site-runtime.cjs +9 -8
- package/dist/report/host/site-runtime.cjs.map +1 -1
- package/dist/report/host/site-runtime.d.cts +1 -1
- package/dist/report/host/site-runtime.d.mts +1 -1
- package/dist/report/host/site-runtime.d.ts +1 -1
- package/dist/report/host/static.cjs +56 -9
- package/dist/report/host/static.cjs.map +1 -1
- package/dist/report/index.cjs.map +1 -1
- package/dist/report/index.d.cts +1 -1
- package/dist/report/index.d.mts +1 -1
- package/dist/report/index.d.ts +1 -1
- package/dist/report/model/dataset.cjs +7 -2
- package/dist/report/model/dataset.cjs.map +1 -1
- package/dist/report/model/locale.cjs +2 -0
- package/dist/report/model/locale.cjs.map +1 -1
- package/dist/report/model/locale.d.cts +1 -0
- package/dist/report/model/locale.d.mts +1 -0
- package/dist/report/model/locale.d.ts +1 -0
- package/dist/report/model/metrics.cjs +3 -1
- package/dist/report/model/metrics.cjs.map +1 -1
- package/dist/report/model/pricing.cjs +16 -11
- package/dist/report/model/pricing.cjs.map +1 -1
- package/dist/report/react/styles.css +53 -29
- package/dist/runner/adoption.cjs +1 -0
- package/dist/runner/adoption.cjs.map +1 -1
- package/dist/runner/attempt.cjs +44 -5
- package/dist/runner/attempt.cjs.map +1 -1
- package/dist/runner/attempt.d.cts +13 -1
- package/dist/runner/attempt.d.mts +13 -1
- package/dist/runner/attempt.d.ts +13 -1
- package/dist/runner/config-identity.cjs +10 -2
- package/dist/runner/config-identity.cjs.map +1 -1
- package/dist/runner/config-identity.d.cts +4 -2
- package/dist/runner/config-identity.d.mts +4 -2
- package/dist/runner/config-identity.d.ts +4 -2
- package/dist/runner/feedback/human.cjs +139 -64
- package/dist/runner/feedback/human.cjs.map +1 -1
- package/dist/runner/feedback/json.cjs +14 -6
- package/dist/runner/feedback/json.cjs.map +1 -1
- package/dist/runner/feedback/json.d.cts +3 -2
- package/dist/runner/feedback/json.d.mts +3 -2
- package/dist/runner/feedback/json.d.ts +3 -2
- package/dist/runner/feedback/reducer.cjs +2 -0
- package/dist/runner/feedback/reducer.cjs.map +1 -1
- package/dist/runner/judge-config.cjs +1 -1
- package/dist/runner/judge-config.cjs.map +1 -1
- package/dist/runner/ledger.cjs +124 -12
- package/dist/runner/ledger.cjs.map +1 -1
- package/dist/runner/project-current.cjs +13 -5
- package/dist/runner/project-current.cjs.map +1 -1
- package/dist/runner/run.cjs +521 -256
- package/dist/runner/run.cjs.map +1 -1
- package/dist/runner/sandbox-pool.cjs +161 -30
- package/dist/runner/sandbox-pool.cjs.map +1 -1
- package/dist/runner/sandbox-pool.d.cts +34 -2
- package/dist/runner/sandbox-pool.d.mts +34 -2
- package/dist/runner/sandbox-pool.d.ts +34 -2
- package/dist/runner/shared-state-lease.cjs +803 -0
- package/dist/runner/shared-state-lease.cjs.map +1 -0
- package/dist/runner/shared-state-lease.d.cts +121 -0
- package/dist/runner/shared-state-lease.d.mts +121 -0
- package/dist/runner/shared-state-lease.d.ts +121 -0
- package/dist/runner/shared-state-lease.js +24 -0
- package/dist/runner/shared-state-lease.js.map +1 -0
- package/dist/runner/teardown-registry.cjs +97 -34
- package/dist/runner/teardown-registry.cjs.map +1 -1
- package/dist/runner/teardown-registry.d.cts +27 -1
- package/dist/runner/teardown-registry.d.mts +27 -1
- package/dist/runner/teardown-registry.d.ts +27 -1
- package/dist/runner/teardown-registry.js +20 -16
- package/dist/runner/types.cjs.map +1 -1
- package/dist/runner/types.d.cts +31 -5
- package/dist/runner/types.d.mts +31 -5
- package/dist/runner/types.d.ts +31 -5
- package/dist/sample/analysis.cjs +249 -459
- package/dist/sample/analysis.cjs.map +1 -1
- package/dist/sample/capability.cjs +46 -8
- package/dist/sample/capability.cjs.map +1 -1
- package/dist/sandbox/docker-profile/schema.cjs +283 -534
- package/dist/sandbox/docker-profile/schema.cjs.map +1 -1
- package/dist/sandbox/docker-profile/schema.d.cts +10 -27
- package/dist/sandbox/docker-profile/schema.d.mts +10 -27
- package/dist/sandbox/docker-profile/schema.d.ts +10 -27
- package/dist/sandbox/e2b-agent-template.cjs +5 -2
- package/dist/sandbox/e2b-agent-template.cjs.map +1 -1
- package/dist/sandbox/keep-registry.cjs +28 -66
- package/dist/sandbox/keep-registry.cjs.map +1 -1
- package/dist/sandbox/types.cjs.map +1 -1
- package/dist/sandbox/types.d.cts +1 -1
- package/dist/sandbox/types.d.mts +1 -1
- package/dist/sandbox/types.d.ts +1 -1
- package/dist/shared/aggregate.cjs +16 -6
- package/dist/shared/aggregate.cjs.map +1 -1
- package/dist/shared/aggregate.d.cts +12 -4
- package/dist/shared/aggregate.d.mts +12 -4
- package/dist/shared/aggregate.d.ts +12 -4
- package/dist/shared/turn-label.cjs +20 -2
- package/dist/shared/turn-label.cjs.map +1 -1
- package/dist/shared/turn-label.d.cts +2 -0
- package/dist/shared/turn-label.d.mts +2 -0
- package/dist/shared/turn-label.d.ts +2 -0
- package/dist/shared/turn-label.js +4 -2
- package/dist/vendor/remark/remark-gfm.cjs +7 -7
- package/dist/vendor/remark/remark-gfm.cjs.map +1 -1
- package/dist/vendor/remark/remark-parse.cjs +12 -12
- package/dist/vendor/remark/remark-parse.cjs.map +1 -1
- package/dist/vendor/remark/remark.cjs +13 -13
- package/dist/vendor/remark/remark.cjs.map +1 -1
- package/dist/view/server.cjs +6 -0
- package/dist/view/server.cjs.map +1 -1
- package/docs-site/zh/examples/ai-agent-application.mdx +1 -1
- package/docs-site/zh/examples/coding-agent-extensions.mdx +1 -1
- package/docs-site/zh/examples/integrations/ai-sdk-v7.mdx +1 -0
- package/docs-site/zh/examples/integrations/langgraph.mdx +1 -1
- package/docs-site/zh/explanation/assert.mdx +6 -6
- package/docs-site/zh/explanation/drive.mdx +1 -1
- package/docs-site/zh/explanation/evals.mdx +9 -9
- package/docs-site/zh/explanation/experiment.mdx +2 -2
- package/docs-site/zh/explanation/hitl.mdx +3 -3
- package/docs-site/zh/explanation/judge.mdx +5 -5
- package/docs-site/zh/explanation/overview.mdx +3 -3
- package/docs-site/zh/explanation/runner.mdx +1 -1
- package/docs-site/zh/index.mdx +3 -3
- package/docs-site/zh/reference/builtin-agents.mdx +9 -9
- package/docs-site/zh/reference/cli.mdx +18 -12
- package/docs-site/zh/reference/define-agent.mdx +130 -5
- package/docs-site/zh/reference/define-config.mdx +2 -2
- package/docs-site/zh/reference/define-eval.mdx +4 -4
- package/docs-site/zh/reference/events.mdx +1 -1
- package/docs-site/zh/reference/official-adapters.mdx +5 -5
- package/docs-site/zh/troubleshooting/debugging.mdx +1 -1
- package/docs-site/zh/troubleshooting/recover-after-kill.mdx +37 -5
- package/docs-site/zh/tutorials/accept-results.mdx +72 -0
- package/docs-site/zh/tutorials/agent-onboarding.mdx +6 -6
- package/docs-site/zh/tutorials/authoring.mdx +9 -13
- package/docs-site/zh/tutorials/ci-integration.mdx +6 -6
- package/docs-site/zh/tutorials/compare-runs.mdx +61 -0
- package/docs-site/zh/tutorials/concurrency.mdx +13 -12
- package/docs-site/zh/tutorials/configuration.mdx +4 -6
- package/docs-site/zh/tutorials/connect-your-agent.mdx +1 -1
- package/docs-site/zh/tutorials/control-run-cost.mdx +58 -0
- package/docs-site/zh/tutorials/criteria-files.mdx +2 -2
- package/docs-site/zh/tutorials/custom-reports.mdx +2 -2
- package/docs-site/zh/tutorials/debug-lifecycle-plan.mdx +46 -0
- package/docs-site/zh/tutorials/docker-in-docker.mdx +50 -44
- package/docs-site/zh/tutorials/eval-groups.mdx +6 -6
- package/docs-site/zh/tutorials/evaluation-kinds.mdx +3 -3
- package/docs-site/zh/tutorials/experiments.mdx +2 -3
- package/docs-site/zh/tutorials/handle-execution-failures.mdx +18 -10
- package/docs-site/zh/tutorials/install-custom-sandbox-agent.mdx +16 -8
- package/docs-site/zh/tutorials/plugins.mdx +14 -0
- package/docs-site/zh/tutorials/rerun-and-cache.mdx +3 -17
- package/docs-site/zh/tutorials/sandbox-agent.mdx +17 -11
- package/docs-site/zh/tutorials/sandbox-providers.mdx +17 -13
- package/docs-site/zh/tutorials/sandbox-reuse.mdx +12 -19
- package/docs-site/zh/tutorials/select-and-preview.mdx +65 -0
- package/docs-site/zh/tutorials/verify-judge.mdx +78 -0
- package/docs-site/zh/tutorials/viewing-results.mdx +27 -8
- package/docs-site/zh/tutorials/write-experiment.mdx +7 -5
- package/docs-site/zh/tutorials/write-send.mdx +2 -2
- package/package.json +9 -5
- package/dist/report/built-in/overview.cjs +0 -45
- package/dist/report/built-in/overview.cjs.map +0 -1
- package/dist/report/built-in/overview.d.cts +0 -6
- package/dist/report/built-in/overview.d.mts +0 -6
- package/dist/report/built-in/overview.d.ts +0 -6
- package/dist/report/built-in/overview.js +0 -9
- package/dist/report/built-in/overview.js.map +0 -1
- package/dist/runner/gate-lease.cjs +0 -283
- package/dist/runner/gate-lease.cjs.map +0 -1
- package/dist/runner/gate-lease.d.cts +0 -62
- package/dist/runner/gate-lease.d.mts +0 -62
- package/dist/runner/gate-lease.d.ts +0 -62
- package/dist/runner/gate-lease.js +0 -22
- package/dist/runner/gate-lease.js.map +0 -1
|
@@ -4,7 +4,7 @@ sidebarTitle: "运行器"
|
|
|
4
4
|
description: "了解 Runner 怎样建立 Run、选择执行或采用 Attempt,并返回可供后续读取的 receipt。"
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
Runner 把选中的 Experiment
|
|
7
|
+
Runner 把选中的 Experiment 和评估用例排成一次 Invocation。它在当前进程中安排执行、显示进度,并把停稳后的业务事实写入 Record。
|
|
8
8
|
|
|
9
9
|
## Invocation、Run 与 Attempt
|
|
10
10
|
|
package/docs-site/zh/index.mdx
CHANGED
|
@@ -15,11 +15,11 @@ import { DirectMode } from '/snippets/diagram-direct-mode.jsx';
|
|
|
15
15
|
|
|
16
16
|
</div>
|
|
17
17
|
|
|
18
|
-
[NiceEval](https://niceeval.com/) 是一个受 [eve](https://eve.dev) 启发的 Agent-Native 评估框架与 Harness:`defineEval` 负责写清楚测什么,Harness
|
|
18
|
+
[NiceEval](https://niceeval.com/) 是一个受 [eve](https://eve.dev) 启发的 Agent-Native 评估框架与 Harness:`defineEval` 负责写清楚测什么,Harness 负责连接被测对象、运行、评分并生成报告。两者合在一起,就是从编写评估用例到查看结果、优化 Agent 的完整闭环。它首先追求良好的开发体验:一个项目可以在 10 分钟左右接入,评估用例文件足够直观,结果也足够容易读。
|
|
19
19
|
|
|
20
|
-
它也框架无关、足够通用:可以用来评估给 Claude Code / Codex 写的
|
|
20
|
+
它也框架无关、足够通用:可以用来评估给 Claude Code / Codex 写的 Coding Agent 插件、Hook 和 Skill,也可以评估你自己的 AI Agent 应用。无论你的应用基于 AI SDK、LangGraph、Pi,还是自研 Agent loop,都可以通过 Adapter 接入。
|
|
21
21
|
|
|
22
|
-
评估运行完成后,[NiceEval](https://niceeval.com/) 会生成易读报告和结构化
|
|
22
|
+
评估运行完成后,[NiceEval](https://niceeval.com/) 会生成易读报告和结构化 Artifact,让你能查看 Agent 的行为细节,调试失败原因,并持续优化通过率、成本和延迟。
|
|
23
23
|
|
|
24
24
|
<CardGroup cols={3}>
|
|
25
25
|
<Card title="Agent Framework 接入" icon="code-branch" href="/zh/examples">
|
|
@@ -44,13 +44,13 @@ description: "NiceEval 内置的 claude-code、codex、bub 适配器分别做到
|
|
|
44
44
|
### codex
|
|
45
45
|
|
|
46
46
|
- 连接方式:Sandbox 里跑 `codex exec --json`(续接时是 `codex exec resume <id> --json`),stdout JSONL 当 transcript。命令带 `--dangerously-bypass-approvals-and-sandbox` 与 `--dangerously-bypass-hook-trust`:Sandbox 里没人能回答 codex 的交互确认,插件或 `postSetup` 装好的 hook 因此不需要交互授信就能生效。
|
|
47
|
-
- 鉴权:`
|
|
47
|
+
- 鉴权:`OPENAI_API_KEY`,可选 `OPENAI_BASE_URL` 接 OpenAI 兼容代理。配置项见下方 `CodexConfig`。
|
|
48
48
|
- `tracing` 通过 `~/.codex/config.toml` 的 `[otel.trace_exporter.otlp-http]` 段配置,协议 `http/json`。
|
|
49
49
|
|
|
50
50
|
### bub
|
|
51
51
|
|
|
52
52
|
- 连接方式:Sandbox 里跑 `bub run` + `--session-id`,从 `~/.bub/tapes/<hash>.jsonl` 读 tape 当 transcript。
|
|
53
|
-
- 鉴权:`
|
|
53
|
+
- 鉴权:`OPENAI_API_KEY` + `OPENAI_BASE_URL`(OpenAI 兼容代理)。配置项见下方 `BubConfig`。
|
|
54
54
|
- `tracing` 通过环境变量注入(`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` 等),协议 `http/protobuf`。
|
|
55
55
|
- 安装走 `uv tool install`(非 npm 包),首次安装带 checkpoint 缓存以加速后续 Sandbox。
|
|
56
56
|
|
|
@@ -85,8 +85,8 @@ baseUrl?: string;
|
|
|
85
85
|
env?: Readonly<globalThis.Record<string, string>>;
|
|
86
86
|
```
|
|
87
87
|
|
|
88
|
-
|
|
89
|
-
|
|
88
|
+
Claude Code 进程的额外环境变量。值只保留在运行时,不进入结果沿用身份。
|
|
89
|
+
会改变行为的非敏感值应同时写入 Experiment flags 或所属 Plugin 的身份。
|
|
90
90
|
|
|
91
91
|
#### `maxTurns`
|
|
92
92
|
|
|
@@ -168,7 +168,7 @@ preTeardown?: readonly SandboxCommand[];
|
|
|
168
168
|
apiKey?: string;
|
|
169
169
|
```
|
|
170
170
|
|
|
171
|
-
代理 / OpenAI API key。省略时读
|
|
171
|
+
代理 / OpenAI API key。省略时读 OPENAI_API_KEY env。
|
|
172
172
|
|
|
173
173
|
#### `baseUrl`
|
|
174
174
|
|
|
@@ -176,7 +176,7 @@ apiKey?: string;
|
|
|
176
176
|
baseUrl?: string;
|
|
177
177
|
```
|
|
178
178
|
|
|
179
|
-
OpenAI 兼容代理 base URL(如 `https://s2a.example.com/v1`)。省略时读
|
|
179
|
+
OpenAI 兼容代理 base URL(如 `https://s2a.example.com/v1`)。省略时读 OPENAI_BASE_URL env。
|
|
180
180
|
|
|
181
181
|
#### `env`
|
|
182
182
|
|
|
@@ -187,7 +187,7 @@ env?: Readonly<globalThis.Record<string, string>>;
|
|
|
187
187
|
注入每次 Codex CLI 进程的额外环境变量。首轮 `codex exec` 与续轮 `codex exec resume`
|
|
188
188
|
使用同一份声明;Codex 启动的 lifecycle Hook、MCP 动态 header 与命令子进程都会继承。
|
|
189
189
|
值只经 Sandbox command options 传入,不拼进 shell 文本或写入 setup manifest,并全部按
|
|
190
|
-
潜在敏感值从 timing / execution / error 证据中脱敏。`
|
|
190
|
+
潜在敏感值从 timing / execution / error 证据中脱敏。`OPENAI_API_KEY` 仍由 `apiKey` 或
|
|
191
191
|
宿主同名环境变量提供,Adapter 的鉴权值会覆盖这里的同名键。
|
|
192
192
|
env value 不进入 carry 身份。会改变被测行为的非敏感值必须同时声明在 Experiment flags
|
|
193
193
|
或所属 Plugin identity;只轮换凭据不会让旧结果失效。
|
|
@@ -267,7 +267,7 @@ preTeardown?: readonly SandboxCommand[];
|
|
|
267
267
|
apiKey?: string;
|
|
268
268
|
```
|
|
269
269
|
|
|
270
|
-
OpenAI 兼容代理的 API key。省略时读
|
|
270
|
+
OpenAI 兼容代理的 API key。省略时读 OPENAI_API_KEY env。
|
|
271
271
|
|
|
272
272
|
#### `apiBase`
|
|
273
273
|
|
|
@@ -275,7 +275,7 @@ OpenAI 兼容代理的 API key。省略时读 BUB_API_KEY env。
|
|
|
275
275
|
apiBase?: string;
|
|
276
276
|
```
|
|
277
277
|
|
|
278
|
-
OpenAI 兼容代理的 base URL。省略时读
|
|
278
|
+
OpenAI 兼容代理的 base URL。省略时读 OPENAI_BASE_URL env。
|
|
279
279
|
|
|
280
280
|
#### `skills`
|
|
281
281
|
|
|
@@ -71,7 +71,7 @@ npx niceeval exp list compare/codex
|
|
|
71
71
|
npx niceeval exp list --json
|
|
72
72
|
```
|
|
73
73
|
|
|
74
|
-
默认输出每个配置的 `experimentId`、描述、agent、model、attempts
|
|
74
|
+
默认输出每个配置的 `experimentId`、描述、agent、model、attempts、选中的评估用例数和 labels。`--json` 输出单个 `niceeval.experiments` 文档,并在每项附上完整的 `selectedEvalIds`。
|
|
75
75
|
|
|
76
76
|
### 查看一个配对的完整 Shell 与 lifecycle
|
|
77
77
|
|
|
@@ -80,7 +80,7 @@ npx niceeval debug compare/codex memory/commit0
|
|
|
80
80
|
npx niceeval debug compare/codex memory/commit0 --json
|
|
81
81
|
```
|
|
82
82
|
|
|
83
|
-
两个 selector 都必须唯一。精确 ID
|
|
83
|
+
两个 selector 都必须唯一。精确 ID 优先,否则可以使用唯一前缀。评估用例只能从 Experiment 自己选中的范围里匹配;零命中或多命中会列出精确候选。
|
|
84
84
|
|
|
85
85
|
计划包含 Plugin lifecycle、author prepare、Agent ensure/setup/teardown、test、cleanup 与 Provider finalizer。能静态声明的 `shell()` / `command()` 展开为实际命令,普通 callback 标为 `opaque`。
|
|
86
86
|
|
|
@@ -136,7 +136,7 @@ E2B template name 与 Vercel snapshot ID 是 Provider 管理的任意字符串
|
|
|
136
136
|
|
|
137
137
|
## 常用 flags
|
|
138
138
|
|
|
139
|
-
下表由 CLI 的 flag 解析表生成,表外的 flag 一律按未知 flag 报错并以非零状态退出。**没有**用于选择 Sandbox provider 的 CLI flag,也没有项目级默认 provider
|
|
139
|
+
下表由 CLI 的 flag 解析表生成,表外的 flag 一律按未知 flag 报错并以非零状态退出。**没有**用于选择 Sandbox provider 的 CLI flag,也没有项目级默认 provider。请在评估用例或 Experiment 上声明 template-bearing `SandboxLayer`,并从 `niceeval/sandbox` 导入下列工厂之一:
|
|
140
140
|
|
|
141
141
|
- `dockerSandbox({ source: { type: "image", image } })`
|
|
142
142
|
- `dockerComposeSandbox({ file, workspaceService })`
|
|
@@ -174,15 +174,19 @@ E2B template name 与 Vercel snapshot ID 是 Provider 管理的任意字符串
|
|
|
174
174
|
| `--experiment` | string[] | `show` / `view` 命令专用:按完整 ExperimentId 收窄当前项目结果;可重复,不接受前缀或逗号列表。 |
|
|
175
175
|
| `--record` | string | `show` / `view` / `accept` / `sandbox enter\|list\|stop` 共用:指定实际 Record root;CLI 不补接 `.niceeval/record` 或其它后缀。 |
|
|
176
176
|
| `--run` | string[] | `show` / `view` 可重复传入 `--run`;每次按完整 RunId 增加一个显式 Run,重复 identity 去重。 |
|
|
177
|
-
| `--report` | string | `show` / `view` 命令专用:内建 `
|
|
177
|
+
| `--report` | string | `show` / `view` 命令专用:内建 `standard` 或受信任的 Report module 路径。 |
|
|
178
178
|
| `--page` | string | `show` / `view` 命令专用:选择报告的初始页;`show` 渲染该页并在尾部附其余页索引,`view` 以它作初始路由。未命中的页 id 按用法错误退出并列出可用页 id。 |
|
|
179
179
|
| `--teardown` | boolean | `exp` 命令专用:补齐被强杀打断的实验级 teardown——只对选中的实验各执行一次 teardown(新进程语义),不派发 attempt、不跑 setup;没有遗留登记也照常执行。与 eval 前缀位置参数组合是用法错误。 |
|
|
180
|
+
| `--recover-shared-state` | string | `exp --teardown` 专用:显式恢复一个永不自动接管的 sharedState lease;必须同时提供 --owner-token、--confirm-owner-terminated 和 --confirm-remote-quiesced。 |
|
|
181
|
+
| `--owner-token` | string | `--recover-shared-state` 专用:要恢复的不可变 owner token;错误或过期 token 不会修改 lease。 |
|
|
182
|
+
| `--confirm-owner-terminated` | boolean | `--recover-shared-state` 专用:确认原 owner 已终止。 |
|
|
183
|
+
| `--confirm-remote-quiesced` | boolean | `--recover-shared-state` 专用:确认远端外部状态已静止。 |
|
|
180
184
|
| `--dry` | boolean | 只打印本次会匹配到的 eval × 运行配置,不实际执行(人读文本或 `--json` 单文档,见「机器怎么读:--json」)。 |
|
|
181
185
|
| `--force` | boolean | `sandbox prune` 专用:除 orphan 外也销毁 unverified 实例;`exp` 明确拒绝此 flag,重跑失败项或全部项请用 `--rerun` / `--rerun all`。 |
|
|
182
186
|
| `--rerun` | boolean | `exp` 命令专用:重新运行失败项(裸写/failed)或全部项(all),不改变长期指纹。 |
|
|
183
187
|
| `--early-exit` / `--no-early-exit` | boolean | 某个 eval 的一次 attempt 通过后,停止该 eval 剩余的 attempts;省略默认关(`attempts` 默认跑满、测完整通过率)。 |
|
|
184
188
|
| `--open` / `--no-open` | boolean | `view` 命令专用:启动后自动打开浏览器(默认行为)。 |
|
|
185
|
-
| `--yes` | boolean | `clean` / `migrate` 专用:确认不可逆 maintenance
|
|
189
|
+
| `--yes` | boolean | `clean` / `migrate` 专用:确认不可逆 maintenance 动作或在 clean Git restore point 下执行 migration。 |
|
|
186
190
|
| `--help` | boolean | 打印用法说明并退出。 |
|
|
187
191
|
| `--version` | boolean | 打印 niceeval 的版本号并退出。 |
|
|
188
192
|
|
|
@@ -235,7 +239,7 @@ pnpm exec niceeval view --host 192.168.0.199
|
|
|
235
239
|
|
|
236
240
|
打开本地结果查看器。它和 `show` 共用默认 Report 与选择规则。不带 locator 或 `--run` 的命令读取所有身份仍匹配当前项目的结果;精确 `@<AttemptLocator>` 显示该 Attempt 的默认概览;`--run` 读取指定历史 Run。NiceEval 随后形成固定 Sample、枚举全部 Page 与参数实例,并构建完整站点。
|
|
237
241
|
|
|
238
|
-
没有显式 `--report` 时,`--run` 使用内建 `run-membership-overview`,精确 locator 使用 `attempt-overview`。`project-current` 才使用配置里的 Report
|
|
242
|
+
没有显式 `--report` 时,`--run` 使用内建 `run-membership-overview`,精确 locator 使用 `attempt-overview`。`project-current` 才使用配置里的 Report,并在没有配置时回退到内建 `standard`。显式 `--report` 始终优先。Config 仍会为 Theme 和 `view` rebuild 加载;这里只说明配置里的 Report 不参与显式 Run 的默认选择。
|
|
239
243
|
|
|
240
244
|
省略 `--host` 时,`view` 只监听 `127.0.0.1`。传入具体 hostname / IP 会显式监听该地址;只写 `--host` 时则监听全部 IPv4 接口。wildcard listener 先打印 loopback URL,再打印可用接口 URL;`--page` 应用到每条 URL,自动打开浏览器只使用第一条。
|
|
241
245
|
|
|
@@ -259,23 +263,25 @@ pnpm exec niceeval show --run 01H... --report ./reports/summary.tsx --json
|
|
|
259
263
|
|
|
260
264
|
参数 Page 的 show 路径只对请求 key 调用 `decode()`、canonical `encode()`、`load()` 与 `render()`。它不调用 `enumerate()`,也不为了查找成员资格而执行其它 Page。通过 Page 的关闭 Evidence 边界验证的 locator、identity 或 key 不属于当前 Sample 时,命令返回类型化错误。
|
|
261
265
|
|
|
262
|
-
不带 locator 或 `--run` 的 `show` 规划当前项目身份,并扫描默认 Record 中全部 published Run。只有 Experiment
|
|
266
|
+
不带 locator 或 `--run` 的 `show` 规划当前项目身份,并扫描默认 Record 中全部 published Run。只有 Experiment、评估用例、Attempt 序号、evaluation kind、input identity 与 config identity 仍匹配的 slot 才进入 Sample;命令不会按时间只留一个 Run。没有匹配结果时显示空 Sample,过期结果仍可用完整 `--run` 读取。
|
|
263
267
|
|
|
264
268
|
`--run` 可重复,只读取这些显式历史 Run。`--experiment <id>` 可重复,按完整 ExperimentId 收窄当前项目目标,不能与 `--run` 合用。`exp`、不带 `--record` 的 `show` 与 `view` 默认使用同一份结果集。
|
|
265
269
|
|
|
266
|
-
单个 `--run`
|
|
270
|
+
单个 `--run` 用于核对这一轮计划的 Attempt 与结果;多个 `--run` 比较多个固定历史边界。即使两个 Run 指向同一个 Attempt,它们也可能分别由实际执行、自动沿用或人工采用形成。`exp --json` 最后一条 receipt 的 `runIds` 是机器稳定出口;TTY 完成反馈也显示 Run ID。`accept` 的成功反馈会显示新 Run ID,但这行人读文本不是 JSON receipt 或自动化输入契约。
|
|
267
271
|
|
|
268
272
|
内建 `run-membership-overview` 固定使用 `reportId: "run-membership-overview"`、`pageId: "run-membership"` 和 route `/`。
|
|
269
273
|
|
|
270
|
-
`Run
|
|
274
|
+
默认 Human 页使用 `Experiment`、`Eval`、`Attempt`、`Result` 与 `Details`。如果 Run 在创建 Attempt 前失败,页面先显示真实 `Run errors` 和 `N attempts not started`,不显示空通过率、空 evidence、空 Analysis notes 或内部 membership 字段。
|
|
275
|
+
|
|
276
|
+
`show --json` 继续保留机器 membership 字段,稳定 column keys 分成三组:
|
|
271
277
|
|
|
272
278
|
- row identity:`runId`、`slotId`。
|
|
273
279
|
- Core 与 provenance:`slotState`、`memberRelation`、`sourceAttemptLocator`、`membershipState`、`membershipOutcome`。
|
|
274
280
|
- Attempt 事实:`verdictState`、`verdict`。
|
|
275
281
|
|
|
276
|
-
单 Run 和多 Run 都按 `runId`、`slotId`
|
|
282
|
+
单 Run 和多 Run 的机器 rows 都按 `runId`、`slotId` 排序;Human 的 planned attempts 与机器 rows 都最多显示 200 项,并显示省略数量。
|
|
277
283
|
|
|
278
|
-
`membershipOutcome`
|
|
284
|
+
`membershipOutcome` 是机器面给出的来源结果,值为 `carried`、`accepted`、`executed`、`not-dispatched` 或 `interrupted`。它用于说明本次位置如何得到结果。已知 Attempt 的身份与证据用 Human 页的 `Details` locator 或 `show @<AttemptLocator>` 下钻,需要其它 Run 字段时显式选择自定义 Report。
|
|
279
285
|
|
|
280
286
|
选择完成后,`show` 只执行目标 Page 需要的 Measure 与领域视图。`partial`、`unsupported`、`failed` 等 Analysis 状态会随结果显示;未使用的数据问题不影响当前页面。`show --json` 是同一个单目标读取:内建 Report 返回 Host-owned 领域文档,自定义 Report 返回一个 Page 的 rendered text manifest。完整契约见[查看结果](/zh/tutorials/viewing-results)。
|
|
281
287
|
|
|
@@ -302,4 +308,4 @@ view 与静态导出都从同一完整站点版本读取 bytes。运行中的 vi
|
|
|
302
308
|
| `2` | 运行时崩溃(未捕获异常),CLI 顶层捕获后退出。 |
|
|
303
309
|
| `130` | 收到中断信号(Ctrl-C / SIGTERM)后退出。 |
|
|
304
310
|
|
|
305
|
-
CI
|
|
311
|
+
CI 的退出码直接来自通过制评估的 Boolean conditions 与最终 Verdict;JUnit 使用同一份已封口的断言结果。
|
|
@@ -61,7 +61,7 @@ export default defineSandboxAgent({
|
|
|
61
61
|
name: "my-cli-agent",
|
|
62
62
|
evidenceCoverage: completeEvidenceCoverage,
|
|
63
63
|
ensure: {
|
|
64
|
-
identity: { agent: "my-cli-agent", version: "1.0.0" },
|
|
64
|
+
identity: { agent: "my-cli-agent", version: "1.0.0", revision: "1" },
|
|
65
65
|
probe: command("my-agent", ["--version"]),
|
|
66
66
|
},
|
|
67
67
|
async send(input, ctx) {
|
|
@@ -303,7 +303,7 @@ SandboxLayer command 按实验隔离跨 attempt 的基础设施内容,或 adap
|
|
|
303
303
|
readonly evalId?: string;
|
|
304
304
|
```
|
|
305
305
|
|
|
306
|
-
当前 Attempt 对应的 eval
|
|
306
|
+
当前 Attempt 对应的 eval ID。NiceEval Runner 始终从 discovery 后的评估用例身份填入;
|
|
307
307
|
第三方直接构造 AgentContext 时可省略。Adapter 可用它定位与题目同目录的只读宿主资产,
|
|
308
308
|
但不能据此绕过 Sandbox 的隐藏判据隔离。
|
|
309
309
|
|
|
@@ -316,7 +316,7 @@ readonly evalGroup?: {
|
|
|
316
316
|
};
|
|
317
317
|
```
|
|
318
318
|
|
|
319
|
-
当前 Attempt
|
|
319
|
+
当前 Attempt 的评估组;未分组评估用例省略。
|
|
320
320
|
|
|
321
321
|
#### `attempt`
|
|
322
322
|
|
|
@@ -388,12 +388,137 @@ log(msg: string): void;
|
|
|
388
388
|
|
|
389
389
|
## Sandbox 接口
|
|
390
390
|
|
|
391
|
-
Sandbox 型 agent 的 `ctx.sandbox`(`Sandbox`)是当前隔离环境的句柄。`CommandOptions` 是 `runCommand` / `runShell` 的可选项:
|
|
391
|
+
Sandbox 型 agent 的 `ctx.sandbox`(`Sandbox`)是当前隔离环境的句柄。`Sandbox` 继承 `SandboxOperations` 与 `SandboxTransferOperations`,下方按声明接口列出全部成员。`CommandOptions` 是 `runCommand` / `runShell` 的可选项:
|
|
392
392
|
|
|
393
393
|
{/* GENERATED:BEGIN sandbox-methods */}
|
|
394
394
|
|
|
395
395
|
{/* 本区块由 pnpm docs:reference 从源码注释生成,勿手改;要改文案,改对应源码的 TSDoc/JSDoc(映射见 scripts/generate-reference.ts) */}
|
|
396
396
|
|
|
397
|
+
### `SandboxOperations`
|
|
398
|
+
|
|
399
|
+
#### `workdir`
|
|
400
|
+
|
|
401
|
+
```ts
|
|
402
|
+
readonly workdir: string;
|
|
403
|
+
```
|
|
404
|
+
|
|
405
|
+
Sandbox 内项目/工作区根目录的绝对路径(agent 命令的默认 cwd,也是 git baseline 提交的位置)。各方法的相对路径都以此为基准解析,省略 `cwd`/`targetDir` 时也落到这里。
|
|
406
|
+
|
|
407
|
+
#### `runCommand`
|
|
408
|
+
|
|
409
|
+
```ts
|
|
410
|
+
runCommand(cmd: string, args?: readonly string[], opts?: CommandOptions): Promise<CommandResult>;
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
执行单个命令,`args` 作为独立 argv 传递、不经 shell 解释(无 `&&`、管道、通配符展开)。
|
|
414
|
+
只想跑一个可执行文件、参数来自外部输入、担心注入时优先用它。
|
|
415
|
+
|
|
416
|
+
#### `runShell`
|
|
417
|
+
|
|
418
|
+
```ts
|
|
419
|
+
runShell(script: string, opts?: CommandOptions): Promise<CommandResult>;
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
执行一整段脚本,经 shell(bash)解释,支持 `&&`、管道、`$()`、重定向等。
|
|
423
|
+
需要拼多条命令或做条件判断时用它。
|
|
424
|
+
|
|
425
|
+
#### `runCommandOrThrow`
|
|
426
|
+
|
|
427
|
+
```ts
|
|
428
|
+
runCommandOrThrow(
|
|
429
|
+
cmd: string,
|
|
430
|
+
args?: readonly string[],
|
|
431
|
+
opts?: CommandOptions,
|
|
432
|
+
): Promise<SuccessfulCommandResult>;
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
像 `runCommand` 一样执行单个命令,但非零退出时抛出 `SandboxCommandExitError`。错误消息附带
|
|
436
|
+
有界、已清理和脱敏的 stderr 尾部;stderr 为空时回退 stdout。完整输出保留在异常的
|
|
437
|
+
`result` 字段中。成功结果的 `exitCode` 在类型上固定为 `0`。
|
|
438
|
+
|
|
439
|
+
#### `runShellOrThrow`
|
|
440
|
+
|
|
441
|
+
```ts
|
|
442
|
+
runShellOrThrow(script: string, opts?: CommandOptions): Promise<SuccessfulCommandResult>;
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
像 `runShell` 一样执行 shell 脚本,但非零退出时抛出 `SandboxCommandExitError`。错误摘要、
|
|
446
|
+
完整输出和成功结果的语义与 `runCommandOrThrow` 相同。
|
|
447
|
+
|
|
448
|
+
#### `readText`
|
|
449
|
+
|
|
450
|
+
```ts
|
|
451
|
+
readText(path: string): Promise<string>;
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
读取 Sandbox 内文件的文本内容(UTF-8)。文件不存在时抛错。
|
|
455
|
+
|
|
456
|
+
#### `writeText`
|
|
457
|
+
|
|
458
|
+
```ts
|
|
459
|
+
writeText(path: string, content: string): Promise<void>;
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
写入 Sandbox 内一个 UTF-8 文本文件;父目录不存在时自动创建。
|
|
463
|
+
|
|
464
|
+
#### `readBytes`
|
|
465
|
+
|
|
466
|
+
```ts
|
|
467
|
+
readBytes(path: string): Promise<Uint8Array>;
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
精确读取 Sandbox 内文件字节;公共契约不绑定 Node Buffer。
|
|
471
|
+
|
|
472
|
+
#### `writeBytes`
|
|
473
|
+
|
|
474
|
+
```ts
|
|
475
|
+
writeBytes(path: string, content: Uint8Array): Promise<void>;
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
精确写入 Sandbox 内文件字节;父目录不存在时自动创建。
|
|
479
|
+
|
|
480
|
+
#### `pathExists`
|
|
481
|
+
|
|
482
|
+
```ts
|
|
483
|
+
pathExists(path: string): Promise<boolean>;
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
检查 Sandbox 内文件或目录路径是否存在。
|
|
487
|
+
|
|
488
|
+
### `SandboxTransferOperations`
|
|
489
|
+
|
|
490
|
+
#### `uploadFile`
|
|
491
|
+
|
|
492
|
+
```ts
|
|
493
|
+
uploadFile(source: string | URL, targetPath: string): Promise<void>;
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
#### `uploadDirectory`
|
|
497
|
+
|
|
498
|
+
```ts
|
|
499
|
+
uploadDirectory(
|
|
500
|
+
sourceDir: string | URL,
|
|
501
|
+
targetDir?: string,
|
|
502
|
+
options?: { readonly ignore?: readonly string[] },
|
|
503
|
+
): Promise<void>;
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
#### `downloadFile`
|
|
507
|
+
|
|
508
|
+
```ts
|
|
509
|
+
downloadFile(sourcePath: string, target: string | URL): Promise<void>;
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
#### `downloadDirectory`
|
|
513
|
+
|
|
514
|
+
```ts
|
|
515
|
+
downloadDirectory(
|
|
516
|
+
sourceDir: string,
|
|
517
|
+
targetDir: string | URL,
|
|
518
|
+
options?: { readonly ignore?: readonly string[] },
|
|
519
|
+
): Promise<void>;
|
|
520
|
+
```
|
|
521
|
+
|
|
397
522
|
### `Sandbox`
|
|
398
523
|
|
|
399
524
|
#### `stop`
|
|
@@ -508,7 +633,7 @@ docs/feature/sandbox/library.md「执行身份」)。
|
|
|
508
633
|
|
|
509
634
|
语义跨 provider 一致,各 provider 映射到自己的原生机制(docker:`exec --user`;E2B:
|
|
510
635
|
`{ user }`;Vercel:只认 `"root"`,映射 `{ sudo: true }`,其它值报错;local:任何值都报错)。
|
|
511
|
-
本就全程 root 的 provider视作 no-op
|
|
636
|
+
本就全程 root 的 provider 视作 no-op;完全无法换身份的 provider 可不支持(抛错)——但**省略与
|
|
512
637
|
显式值的语义保持一致**,不因 provider 而变。
|
|
513
638
|
|
|
514
639
|
#### `timeoutMs`
|
|
@@ -128,6 +128,6 @@ pricing?: globalThis.Record<string, PriceOverride>;
|
|
|
128
128
|
|
|
129
129
|
{/* GENERATED:END config-fields */}
|
|
130
130
|
|
|
131
|
-
> 成本边界:这里的 `pricing` 只提供 Runner 的 Config/runtime price table
|
|
131
|
+
> 成本边界:这里的 `pricing` 只提供 Runner 的 Config/runtime price table 输入。Runner 不论是否已有 `Usage.costUSD`,都会从 `model`、上报的 token 用量和这个表独立产生 `estimatedCostUSD`。Experiment 的预算与 `t.maxCost()` 使用该 estimate,`Usage.costUSD` 不驱动它。Report 不读取此配置;它必须在 Report module 以 `definePricingProfile()` 显式声明自己的 USD Profile,并只对 sealed Usage 做 Analysis 投影。
|
|
132
132
|
|
|
133
|
-
[NiceEval](https://niceeval.com/) 把环境准备放在普通代码里:评估用例自己需要的文件写在 `test(t)`,agent 自己的准备写在 adapter 的 setup。`Config` 刻意没有 `sandbox` 字段,也不自动探测 provider
|
|
133
|
+
[NiceEval](https://niceeval.com/) 把环境准备放在普通代码里:评估用例自己需要的文件写在 `test(t)`,agent 自己的准备写在 adapter 的 setup。`Config` 刻意没有 `sandbox` 字段,也不自动探测 provider。请在评估用例或 Experiment 上声明 template-bearing `SandboxLayer`,让 link planning 在创建任何资源前确定它的 owner 与配对结果。
|
|
@@ -51,7 +51,7 @@ sandbox?: SandboxLayer;
|
|
|
51
51
|
```
|
|
52
52
|
|
|
53
53
|
这道题贡献的 Sandbox 声明层。省略等价于空 command-only layer,不提供隐式 template。
|
|
54
|
-
|
|
54
|
+
每个实际评估用例 × Experiment 配对必须恰好一方提供 template-bearing layer。
|
|
55
55
|
|
|
56
56
|
#### `plugins`
|
|
57
57
|
|
|
@@ -59,7 +59,7 @@ sandbox?: SandboxLayer;
|
|
|
59
59
|
plugins?: readonly PluginInstance<"eval">[];
|
|
60
60
|
```
|
|
61
61
|
|
|
62
|
-
|
|
62
|
+
显式且不可变的评估用例 Plugin occurrence;不存在目录继承。
|
|
63
63
|
|
|
64
64
|
#### `judge`
|
|
65
65
|
|
|
@@ -116,7 +116,7 @@ test(t: TestContext): EvalTestReturn;
|
|
|
116
116
|
|
|
117
117
|
`t`(`TestContext`)是评估用例作者拿到的高层上下文。每一次作者入口调用都直接登记一条 Assertion。`t.check(value, match)` 严格接收两个参数。`t.succeeded()`、`session.succeeded()`、`turn.succeeded()`、`calledTool(...)`、`notCalledTool(...)` 与 Judge recipe 也直接登记 Assertion。handle 只配置同一条 entry 的 `key`、`label`、`.atLeast(n)`、`.gate(n)` 与 `.score(n)`。`.orStop()` 也只作用于此 entry,绝不登记第二条 Assertion。
|
|
118
118
|
|
|
119
|
-
|
|
119
|
+
通过制评估用 Boolean condition 折叠 Attempt Verdict。measurement 用 `.gate(n)` 才进入 failed,`.atLeast(n)` 只增加局部 condition;`await handle.orStop()` 在 mismatch 或 below 时停止当前 continuation。
|
|
120
120
|
|
|
121
121
|
`defineScoreEval` 的 `ScoreTestContext` 额外提供 `t.score(n)` 直接登记 contribution,`n` 必须 finite 且不小于零;已有 Assertion 用 `.score(n)` 贡献分数,`n` 必须 finite 且大于零。`calledTool` 与 `notCalledTool` 的完整契约见 [Scoped assertions](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/scoped-assertions.md)。全部成员:
|
|
122
122
|
|
|
@@ -332,7 +332,7 @@ turn.judge.autoevals.closedQA(question).gate(0.7).label("回答质量");
|
|
|
332
332
|
turn.judge.autoevals.summarizes(sourceText).score(20).label("摘要质量");
|
|
333
333
|
```
|
|
334
334
|
|
|
335
|
-
`t.judge.autoevals` 的根级 recipe 额外接收 `{ input, output }` 字符串材料。`turn.judge` 已绑定 immutable Turn
|
|
335
|
+
`t.judge.autoevals` 的根级 recipe 额外接收 `{ input, output }` 字符串材料。`turn.judge` 已绑定 immutable Turn 材料。模型配置按评估用例、Experiment、项目配置解析一次;没有单次模型覆盖或 `{ on }` 选项。通过制评估使用 `.gate(n)`;计分制评估直接 `.score(n)`。
|
|
336
336
|
|
|
337
337
|
## `Turn` 返回类型
|
|
338
338
|
|
|
@@ -87,7 +87,7 @@ token 用量或 OTel span 反推得到)。与顶层 `estimatedCostUSD`(价目表
|
|
|
87
87
|
|
|
88
88
|
{/* GENERATED:END usage-fields */}
|
|
89
89
|
|
|
90
|
-
> 成本边界:`Usage.costUSD` 只表示
|
|
90
|
+
> 成本边界:`Usage.costUSD` 只表示 Provider / Adapter 返回的 USD observed 成本,绝不由 token、模型目录或本地价目表推导。Runner 即使已有 observed 成本,也会从 `model`、上报的 token 用量和 Config/runtime price table 独立计算 `estimatedCostUSD`。Experiment 的预算与 `t.maxCost()` 使用这个 estimate,`Usage.costUSD` 不驱动它。Report 成本投影只接受显式 `PricingProfile` 与 sealed Usage,不读取 Runner estimate。
|
|
91
91
|
|
|
92
92
|
## StreamEvent 变体一览
|
|
93
93
|
|
|
@@ -64,8 +64,8 @@ export default defineExperiment({
|
|
|
64
64
|
|
|
65
65
|
### codex
|
|
66
66
|
|
|
67
|
-
- **鉴权**:`
|
|
68
|
-
- **Agent 进程环境**:`env` 会注入每次 `codex exec` 与 `codex exec resume`。Codex 启动的 Session 生命周期 Hook、MCP 动态请求头和命令子进程都会继承。值不会拼进 shell 文本或写入安装清单,并按潜在敏感值从 timing、execution 与错误证据中脱敏。`
|
|
67
|
+
- **鉴权**:`OPENAI_API_KEY`(工厂参数 `apiKey` 可覆盖),可选 `OPENAI_BASE_URL`(工厂参数 `baseUrl`)接 OpenAI 兼容代理。
|
|
68
|
+
- **Agent 进程环境**:`env` 会注入每次 `codex exec` 与 `codex exec resume`。Codex 启动的 Session 生命周期 Hook、MCP 动态请求头和命令子进程都会继承。值不会拼进 shell 文本或写入安装清单,并按潜在敏感值从 timing、execution 与错误证据中脱敏。`OPENAI_API_KEY` 仍由 `apiKey` 或宿主环境提供,并覆盖 `env` 里的同名键。`PATH` 是 Sandbox 受管变量,不接受经 `env` 声明——出现即在 `codexAgent()` 调用时报错,改用 Sandbox 的 `pathPrepend`(见 [Sandbox provider](/zh/tutorials/sandbox-providers))。
|
|
69
69
|
- **装 MCP server**:`mcpServers` 配置项,`setup` 阶段追加进 `~/.codex/config.toml` 的 `[mcp_servers.<name>]` 段。两种形态按字段区分:本地 stdio 进程写 `command`(可带 `args` / `env`)。远程 Streamable HTTP 端点写 `url`(可带 `headers`,写成 `[mcp_servers.<name>.http_headers]` 子表)。`url` 要 Sandbox 内可达:服务跑在你自己机器上时,先用 cloudflared / tailscale 这类隧道暴露成公网地址。
|
|
70
70
|
|
|
71
71
|
<Warning>
|
|
@@ -120,7 +120,7 @@ export default defineExperiment({
|
|
|
120
120
|
|
|
121
121
|
### bub
|
|
122
122
|
|
|
123
|
-
- **鉴权**:`
|
|
123
|
+
- **鉴权**:`OPENAI_API_KEY` + `OPENAI_BASE_URL`(OpenAI 兼容代理),工厂参数 `apiKey` / `apiBase` 可覆盖。
|
|
124
124
|
- **装 Skill**:`skills: SkillSpec[]`,与另外两个 Adapter 同一个类型。装进 `.agents/skills/<name>/`,发现指引写进 AGENTS.md。
|
|
125
125
|
- **装插件**:`pythonPlugins: PythonPluginSpec[]`(`{ package }`:PyPI 包、版本约束或 git URL),`setup` 阶段进 `uv tool install … --with <package>`。这个类型只属于 bub。package 集合进安装 checkpoint key,插件不同的两个变体不会复用同一份安装缓存。
|
|
126
126
|
- **预制 Bub**:NiceEval 的 E2B 配方会把 Bub、OTel 插件和 Python 插件集合算成安装指纹。Adapter 只复用指纹完全一致的环境。仅在 PATH 里放一个 `bub` 不足以证明兼容。构建入口见 [Sandbox provider · 从官方基线继续构建以提速](/zh/tutorials/sandbox-providers#从官方基线继续构建以提速)。
|
|
@@ -148,7 +148,7 @@ export default defineExperiment({
|
|
|
148
148
|
|
|
149
149
|
| | claude-code | codex | bub |
|
|
150
150
|
|---|---|---|---|
|
|
151
|
-
| 鉴权环境变量 | `ANTHROPIC_API_KEY` | `
|
|
151
|
+
| 鉴权环境变量 | `ANTHROPIC_API_KEY` | `OPENAI_API_KEY` + 可选 `OPENAI_BASE_URL` | `OPENAI_API_KEY` + `OPENAI_BASE_URL` |
|
|
152
152
|
| 额外 Agent 进程环境 | ❌ | ✅ `env` | ❌ |
|
|
153
153
|
| MCP server | ✅ `mcpServers`(stdio + HTTP) | ✅ `mcpServers`(stdio + HTTP) | ❌ |
|
|
154
154
|
| Skill | ✅ `skills: SkillSpec[]`(原生发现) | ✅ `skills: SkillSpec[]`(+ 发现指引) | ✅ `skills: SkillSpec[]`(+ 发现指引) |
|
|
@@ -159,7 +159,7 @@ export default defineExperiment({
|
|
|
159
159
|
| tracing | ✅(beta,仅结构与计时) | ✅ | ✅ |
|
|
160
160
|
| 安装方式 | npm 全局包(需镜像自带 Node) | 自带运行时的原生包(解压即用,镜像无需 Node) | `uv tool install`(PyPI) |
|
|
161
161
|
|
|
162
|
-
装了什么有据可查:Adapter 在 `setup` 收尾把安装清单交给运行器,存成 Attempt Artifact `agent-setup.json`(库里读 `attempt.agentSetup()
|
|
162
|
+
装了什么有据可查:Adapter 在 `setup` 收尾把安装清单交给运行器,存成 Attempt Artifact `agent-setup.json`(库里读 `attempt.agentSetup()`。清单不落 Sandbox 磁盘,Agent 看不到它)。清单只记来源、ref、Skill/Plugin 名、解析出的版本,以及官方配置文件的项目相对路径和 SHA-256。不保存配置正文、API Key 或环境变量值。MCP server 同理只记非敏感字段:stdio 形态记 `name` / `command` / `args` 不记 `env`,HTTP 形态记 `name` / `url` 不记 `headers`。
|
|
163
163
|
|
|
164
164
|
## 非 Sandbox 适配器
|
|
165
165
|
|
|
@@ -66,7 +66,7 @@ Measure 和领域视图会把自己的状态与问题带进页面:
|
|
|
66
66
|
|
|
67
67
|
`npx niceeval exp <experiment> --json` 只有在成功建立 Invocation 后,最后才会输出 receipt。命令在建立 Invocation 前就因输入错误退出时,没有 `runIds` 可读取。
|
|
68
68
|
|
|
69
|
-
先修正 Experiment
|
|
69
|
+
先修正 Experiment、评估用例选择或命令参数,再重新运行。成功时,使用 receipt 的 `runIds` 运行 `show`、`view` 或 `view --out`。
|
|
70
70
|
|
|
71
71
|
## 静态报告站无法导出
|
|
72
72
|
|
|
@@ -1,25 +1,27 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "运行被强杀后恢复"
|
|
3
|
-
description: "进程被 kill -9、CI
|
|
3
|
+
description: "进程被 kill -9、CI 超时或断电杀掉后,从已发布结果重跑、收回遗留 Sandbox、补齐实验收尾,并显式恢复共享状态租约。"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
`niceeval exp` 跑到一半被 `kill -9`、CI 时限或断电直接杀掉时,进程没有机会做任何清理。你会碰到三种残留:没跑完的评估、还在 Provider 侧占资源的 Sandbox 实例、实验 `setup` 起过但没关掉的外部服务(隧道、共享服务、license 席位)。三种各有一个恢复入口,都不需要手工翻 Docker 或云控制台。
|
|
7
7
|
|
|
8
8
|
正常的 Ctrl+C 或 SIGTERM 不在本页范围:那些路径 NiceEval 会自己走完全部收尾,不留残留。
|
|
9
9
|
|
|
10
|
-
##
|
|
10
|
+
## 从已发布结果重跑
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
只有成功 seal、完整发布的 Run 里的 Attempt 才是 Record 事实。重跑可以自动沿用其中仍符合资格的结果,不再花一次 Agent 和 Sandbox 的成本。强杀 Invocation 留下的 incomplete Run directory 会被整体忽略:里面的草稿 Attempt 不展示、不引用、也不沿用。
|
|
13
13
|
|
|
14
14
|
```bash
|
|
15
15
|
niceeval exp compare/bub-e2b memory/commit0
|
|
16
16
|
```
|
|
17
17
|
|
|
18
|
-
-
|
|
18
|
+
- 如果另一个完整 Run 已经为 `--attempts 5` 发布 3 个符合资格的 Attempt,就只再跑 2 次。如果那 3 个草稿只存在于强杀 Run、尚未到达 `complete` 发布点,它们一个也不会沿用,5 个位置都可能重跑。
|
|
19
19
|
- 被强杀的实验如果留了没做完的收尾,重跑会先补一次实验级 `teardown` 再开始,泄漏不会越积越多。
|
|
20
20
|
- 判定为 `errored` 的 Attempt 不复用,照常重跑。
|
|
21
21
|
- 想全部重来,加 `--rerun all`。
|
|
22
22
|
|
|
23
|
+
确认没有活跃 Invocation 占用 Record root 后,运行 `niceeval clean` 删除 incomplete Run 残留。清理不会把草稿 Attempt 变成可复用结果。
|
|
24
|
+
|
|
23
25
|
## 收回没清理的 Sandbox 实例
|
|
24
26
|
|
|
25
27
|
先核对有哪些实例属于已经死掉的运行:
|
|
@@ -57,7 +59,37 @@ niceeval exp compare/bub-e2b --teardown
|
|
|
57
59
|
- 随时可以执行,不依赖上次运行留下的记录。`teardown` 自身要能容忍重复执行。
|
|
58
60
|
- 强杀后原进程的内存已经丢失,`teardown` 里要从容器名、pid 文件或幂等的关停脚本这类持久信息找回要关的资源,不要依赖 `setup` 存在内存里的对象。
|
|
59
61
|
|
|
62
|
+
## 恢复共享状态租约
|
|
63
|
+
|
|
64
|
+
声明了 `sharedState.key` 的实验会把租约持有到 Sandbox teardown、Provider finalizer 和实验 `teardown` 全部成功。进程被强杀或其中一项 cleanup 失败时,租约会保留;同 key 的新 Invocation 会继续等待,不会根据 PID、超时或 heartbeat 自动接管。
|
|
65
|
+
|
|
66
|
+
先只检查当前 owner evidence,不带 token 或确认参数:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
niceeval exp compare/bub-e2b --teardown \
|
|
70
|
+
--recover-shared-state mempal/codex/cohort-a
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
命令会显示 key、Experiment、owner token、host、PID、process identity 和最新 heartbeat,然后以非零状态退出。先确认原 owner 已终止,再确认它操作的 checkpoint、远端服务和回存流程都已静止。这个恢复入口只补跑实验 `teardown`,不会回滚已经写到一半的业务状态。
|
|
74
|
+
|
|
75
|
+
核对完后,复制刚显示的完整 owner token,并显式确认两个条件:
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
niceeval exp compare/bub-e2b --teardown \
|
|
79
|
+
--recover-shared-state mempal/codex/cohort-a \
|
|
80
|
+
--owner-token <displayed-token> \
|
|
81
|
+
--confirm-owner-terminated \
|
|
82
|
+
--confirm-remote-quiesced
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
- 同机的原进程仍然存活时,NiceEval 拒绝恢复;暂停的进程仍算存活。
|
|
86
|
+
- 错误或过期的 token 不会修改租约,也不会释放后来 owner 的租约。
|
|
87
|
+
- 两条恢复命令同时竞争时,只有一条能取得恢复 ownership。
|
|
88
|
+
- 补偿 `teardown` 失败时,租约继续保留。解决 teardown 问题后重新检查,再用当前显示的 token 重试。
|
|
89
|
+
- 成功后,同 key 的等待 Invocation 才能取得租约并开始 Experiment setup 或创建 Sandbox。
|
|
90
|
+
|
|
60
91
|
## 预防:让下一次强杀无害
|
|
61
92
|
|
|
62
|
-
- 长任务放在有时限的环境(CI
|
|
93
|
+
- 长任务放在有时限的环境(CI、外部看门狗)里跑时,要为 Run 留出到达 `complete` 发布点的时间。后续运行只能沿用完整 Run 已经发布的结果。
|
|
63
94
|
- 实验 `teardown` 写成幂等的:重复执行不报错、目标已经关掉也算成功。这是补收尾机制正确工作的前提。
|
|
95
|
+
- 使用 `sharedState.key` 时,让 checkpoint 回存具备原子性或可补偿性。显式恢复协调租约不能修复半次业务写入。
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "审阅并采用历史 Attempt"
|
|
3
|
+
sidebarTitle: "采用历史 Attempt"
|
|
4
|
+
description: "从 --dry 取得候选 locator,核对不可变 Attempt,再用 niceeval accept 原子采用明确结果,并用新 Run ID 验证分母与来源。"
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
配置身份改变后,自动沿用可能被拒绝。如果你已经核对某个历史 Attempt,确认它在新配置下仍然成立,可以用 `niceeval accept` 明确采用它。
|
|
8
|
+
|
|
9
|
+
不要用这个命令跨过真实的评估条件变化。任务、断言、Agent 行为或 Sandbox 起点已经改变时,应重新运行并取得新证据。
|
|
10
|
+
|
|
11
|
+
## 从计划取得候选 locator
|
|
12
|
+
|
|
13
|
+
先预览当前目标:
|
|
14
|
+
|
|
15
|
+
```sh
|
|
16
|
+
pnpm exec niceeval exp checkout --dry
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
被身份差异挡住的历史结果会显示完整 Attempt locator 和可执行的 `accept` 提示。Locator 是本次授权的唯一范围;命令不会隐含选择同一 Run 或同一 Experiment 的其它结果。
|
|
20
|
+
|
|
21
|
+
## 核对历史 Attempt
|
|
22
|
+
|
|
23
|
+
使用完整 locator 查看不可变结果:
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
pnpm exec niceeval show @1K1P0VJAPVJ12
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
核对 Verdict、Assertion、用量、时间和详情页中的证据。需要回看来源 Run 的完整分母时,再使用:
|
|
30
|
+
|
|
31
|
+
```sh
|
|
32
|
+
pnpm exec niceeval show --run <source-run-id>
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
## 一次采用需要的结果
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
pnpm exec niceeval accept @1K1P0VJAPVJ12
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
同一个 Experiment 有多个结果需要采用时,把 locator 放进同一条命令:
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
pnpm exec niceeval accept @1K1P0VJAPVJ12 @1K1P0VJAPVJ13
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
NiceEval 会先预检全部输入。任一 locator 无效、不符合资格或与另一个输入冲突时,整条命令都不会发布业务结果。
|
|
48
|
+
|
|
49
|
+
成功后,每个目标 slot 以 `reference` 指向原 Attempt,并记录 `accepted` action。源 Attempt 的事实、来源 Run 和 locator 都不会被修改或复制。
|
|
50
|
+
|
|
51
|
+
## 核对新 Run
|
|
52
|
+
|
|
53
|
+
成功反馈会显示新 Run ID:
|
|
54
|
+
|
|
55
|
+
```sh
|
|
56
|
+
pnpm exec niceeval show --run <accepted-run-id>
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
这个 Run 保留当前 Experiment 的完整 expected-slot 分母。命令中选择的位置是 `accepted`;没有选择的 sibling slot 是 `not-dispatched`。一次只采用部分位置时,报告会如实显示结果覆盖不足。
|
|
60
|
+
|
|
61
|
+
每次 `accept` 都发布新 Run,不会修改或合并上一次调用。需要同一批结果组成同一个采用边界时,应放在同一条命令中。
|
|
62
|
+
|
|
63
|
+
## 常见失败
|
|
64
|
+
|
|
65
|
+
| 错误 | 下一步 |
|
|
66
|
+
| --- | --- |
|
|
67
|
+
| `malformed-locator` | 从 `--dry` 或 `show` 复制完整的大写 locator,不使用旧 UUID 或模糊前缀。 |
|
|
68
|
+
| `locator-not-found` | 确认读取的是同一 Record;其它结果根显式传 `--record`。 |
|
|
69
|
+
| `accept-ineligible` | 按提示检查 Verdict、timeout、身份或 Observability;不能证明仍成立时重新运行。 |
|
|
70
|
+
| `duplicate-accept-member` | 每个目标 slot 只保留一个 locator,去掉重复或冲突输入。 |
|
|
71
|
+
|
|
72
|
+
自动沿用与强制重跑的选择见[重跑与沿用结果](/zh/tutorials/rerun-and-cache)。
|