@rpamis/comet 0.4.0-beta.16 → 0.4.0-beta.18
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +85 -56
- package/assets/eval-reports/comet-native-vs-beta16-beta17-20260810/native-benchmark-report.html +8 -0
- package/assets/eval-reports/comet-native-vs-beta16-beta17-20260810/native-benchmark-report.json +286 -0
- package/assets/manifest.json +26 -17
- package/assets/skills/comet/SKILL.md +1 -1
- package/assets/skills/comet/agents/openai.yaml +3 -0
- package/assets/skills/comet/rules/comet-phase-guard.en.md +1 -1
- package/assets/skills/comet/rules/comet-phase-guard.md +1 -1
- package/assets/skills/comet/rules/comet-workflow-guard.en.md +2 -2
- package/assets/skills/comet/rules/comet-workflow-guard.md +2 -2
- package/assets/skills/comet/scripts/comet-archive.mjs +128 -128
- package/assets/skills/comet/scripts/comet-entry-runtime.mjs +77 -77
- package/assets/skills/comet/scripts/comet-guard.mjs +144 -144
- package/assets/skills/comet/scripts/comet-handoff.mjs +107 -107
- package/assets/skills/comet/scripts/comet-hook-guard.mjs +128 -128
- package/assets/skills/comet/scripts/comet-hook-router.mjs +143 -141
- package/assets/skills/comet/scripts/comet-resume-probe.mjs +100 -100
- package/assets/skills/comet/scripts/comet-runtime.mjs +194 -193
- package/assets/skills/comet/scripts/comet-state.mjs +118 -118
- package/assets/skills/comet/scripts/comet-yaml-validate.mjs +95 -95
- package/assets/skills/comet-any/SKILL.md +2 -1
- package/assets/skills/comet-any/agents/openai.yaml +5 -0
- package/assets/skills/comet-archive/SKILL.md +7 -6
- package/assets/skills/comet-archive/agents/openai.yaml +5 -0
- package/assets/skills/comet-build/SKILL.md +28 -68
- package/assets/skills/comet-build/agents/openai.yaml +5 -0
- package/assets/skills/comet-classic/SKILL.md +28 -22
- package/assets/skills/comet-classic/agents/openai.yaml +3 -0
- package/assets/skills/{comet → comet-classic}/reference/auto-transition.md +1 -1
- package/assets/skills/{comet → comet-classic}/reference/comet-yaml-fields.md +3 -3
- package/assets/skills/{comet → comet-classic}/reference/context-recovery.md +5 -5
- package/assets/skills/{comet → comet-classic}/reference/debug-gate.md +2 -2
- package/assets/skills/{comet → comet-classic}/reference/decision-point.md +1 -1
- package/assets/skills/{comet → comet-classic}/reference/dirty-worktree.md +1 -1
- package/assets/skills/{comet → comet-classic}/reference/file-structure.md +1 -1
- package/assets/skills/{comet → comet-classic}/reference/intent-frame.md +1 -1
- package/assets/skills/{comet → comet-classic}/reference/scripts.md +4 -2
- package/assets/skills/{comet → comet-classic}/reference/subagent-dispatch.md +1 -1
- package/assets/skills/comet-classic/reference/workspace.md +23 -0
- package/assets/skills/comet-design/SKILL.md +7 -6
- package/assets/skills/comet-design/agents/openai.yaml +5 -0
- package/assets/skills/comet-hotfix/SKILL.md +10 -9
- package/assets/skills/comet-hotfix/agents/openai.yaml +5 -0
- package/assets/skills/comet-native/SKILL.md +60 -171
- package/assets/skills/comet-native/agents/openai.yaml +3 -0
- package/assets/skills/comet-native/reference/artifacts.md +24 -89
- package/assets/skills/comet-native/reference/clarification.md +54 -29
- package/assets/skills/comet-native/reference/commands.md +30 -173
- package/assets/skills/comet-native/reference/recovery.md +38 -88
- package/assets/skills/comet-native/reference/workspace.md +23 -0
- package/assets/skills/comet-native/scripts/comet-native-archive.mjs +184 -152
- package/assets/skills/comet-native/scripts/comet-native-doctor.mjs +214 -158
- package/assets/skills/comet-native/scripts/comet-native-hook-guard.mjs +132 -123
- package/assets/skills/comet-native/scripts/comet-native-init.mjs +134 -124
- package/assets/skills/comet-native/scripts/comet-native-new.mjs +165 -158
- package/assets/skills/comet-native/scripts/comet-native-next.mjs +183 -152
- package/assets/skills/comet-native/scripts/comet-native-root.mjs +152 -147
- package/assets/skills/comet-native/scripts/comet-native-runtime.mjs +247 -202
- package/assets/skills/comet-native/scripts/comet-native-select.mjs +150 -143
- package/assets/skills/comet-native/scripts/comet-native-show.mjs +128 -121
- package/assets/skills/comet-native/scripts/comet-native-spec.mjs +133 -151
- package/assets/skills/comet-native/scripts/comet-native-status.mjs +147 -140
- package/assets/skills/comet-open/SKILL.md +31 -11
- package/assets/skills/comet-open/agents/openai.yaml +5 -0
- package/assets/skills/comet-tweak/SKILL.md +11 -10
- package/assets/skills/comet-tweak/agents/openai.yaml +5 -0
- package/assets/skills/comet-verify/SKILL.md +9 -8
- package/assets/skills/comet-verify/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet/SKILL.md +1 -1
- package/assets/skills-zh/comet/agents/openai.yaml +3 -0
- package/assets/skills-zh/comet-any/SKILL.md +2 -1
- package/assets/skills-zh/comet-any/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet-archive/SKILL.md +7 -6
- package/assets/skills-zh/comet-archive/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet-build/SKILL.md +28 -68
- package/assets/skills-zh/comet-build/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet-classic/SKILL.md +29 -23
- package/assets/skills-zh/comet-classic/agents/openai.yaml +3 -0
- package/assets/skills-zh/{comet → comet-classic}/reference/auto-transition.md +1 -1
- package/assets/skills-zh/{comet → comet-classic}/reference/comet-yaml-fields.md +3 -3
- package/assets/skills-zh/{comet → comet-classic}/reference/context-recovery.md +5 -5
- package/assets/skills-zh/{comet → comet-classic}/reference/debug-gate.md +2 -2
- package/assets/skills-zh/{comet → comet-classic}/reference/decision-point.md +1 -1
- package/assets/skills-zh/{comet → comet-classic}/reference/dirty-worktree.md +1 -1
- package/assets/skills-zh/{comet → comet-classic}/reference/file-structure.md +1 -1
- package/assets/skills-zh/{comet → comet-classic}/reference/intent-frame.md +1 -1
- package/assets/skills-zh/{comet → comet-classic}/reference/scripts.md +4 -2
- package/assets/skills-zh/{comet → comet-classic}/reference/subagent-dispatch.md +1 -1
- package/assets/skills-zh/comet-classic/reference/workspace.md +23 -0
- package/assets/skills-zh/comet-design/SKILL.md +7 -6
- package/assets/skills-zh/comet-design/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet-hotfix/SKILL.md +10 -9
- package/assets/skills-zh/comet-hotfix/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet-native/SKILL.md +60 -171
- package/assets/skills-zh/comet-native/agents/openai.yaml +3 -0
- package/assets/skills-zh/comet-native/reference/artifacts.md +24 -89
- package/assets/skills-zh/comet-native/reference/clarification.md +40 -15
- package/assets/skills-zh/comet-native/reference/commands.md +30 -173
- package/assets/skills-zh/comet-native/reference/recovery.md +36 -86
- package/assets/skills-zh/comet-native/reference/workspace.md +23 -0
- package/assets/skills-zh/comet-open/SKILL.md +31 -11
- package/assets/skills-zh/comet-open/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet-tweak/SKILL.md +11 -10
- package/assets/skills-zh/comet-tweak/agents/openai.yaml +5 -0
- package/assets/skills-zh/comet-verify/SKILL.md +9 -8
- package/assets/skills-zh/comet-verify/agents/openai.yaml +5 -0
- package/bin/fast-runtime-router.js +0 -4
- package/dist/app/cli/index.js +20 -1
- package/dist/app/cli/index.js.map +1 -1
- package/dist/app/commands/classic.d.ts.map +1 -1
- package/dist/app/commands/classic.js +2 -0
- package/dist/app/commands/classic.js.map +1 -1
- package/dist/app/commands/doctor.d.ts.map +1 -1
- package/dist/app/commands/doctor.js +8 -0
- package/dist/app/commands/doctor.js.map +1 -1
- package/dist/app/commands/eval.d.ts +9 -2
- package/dist/app/commands/eval.d.ts.map +1 -1
- package/dist/app/commands/eval.js +293 -48
- package/dist/app/commands/eval.js.map +1 -1
- package/dist/app/commands/init.d.ts.map +1 -1
- package/dist/app/commands/init.js +11 -2
- package/dist/app/commands/init.js.map +1 -1
- package/dist/app/commands/update.d.ts.map +1 -1
- package/dist/app/commands/update.js +13 -3
- package/dist/app/commands/update.js.map +1 -1
- package/dist/config/repository-layout.json +0 -8
- package/dist/domains/bundle/bundle-platform.d.ts.map +1 -1
- package/dist/domains/bundle/bundle-platform.js +8 -0
- package/dist/domains/bundle/bundle-platform.js.map +1 -1
- package/dist/domains/bundle/types.d.ts +1 -1
- package/dist/domains/bundle/types.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-cli.d.ts +1 -1
- package/dist/domains/comet-classic/classic-cli.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-cli.js +3 -0
- package/dist/domains/comet-classic/classic-cli.js.map +1 -1
- package/dist/domains/comet-classic/classic-current-change.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-current-change.js +29 -3
- package/dist/domains/comet-classic/classic-current-change.js.map +1 -1
- package/dist/domains/comet-classic/classic-guard.js +2 -2
- package/dist/domains/comet-classic/classic-guard.js.map +1 -1
- package/dist/domains/comet-classic/classic-hook-guard.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-hook-guard.js +14 -1
- package/dist/domains/comet-classic/classic-hook-guard.js.map +1 -1
- package/dist/domains/comet-classic/classic-state-command.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-state-command.js +31 -8
- package/dist/domains/comet-classic/classic-state-command.js.map +1 -1
- package/dist/domains/comet-classic/classic-workspace-command.d.ts +3 -0
- package/dist/domains/comet-classic/classic-workspace-command.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-workspace-command.js +45 -0
- package/dist/domains/comet-classic/classic-workspace-command.js.map +1 -0
- package/dist/domains/comet-classic/classic-workspace.d.ts +38 -0
- package/dist/domains/comet-classic/classic-workspace.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-workspace.js +315 -0
- package/dist/domains/comet-classic/classic-workspace.js.map +1 -0
- package/dist/domains/comet-entry/hook-adapter.d.ts.map +1 -1
- package/dist/domains/comet-entry/hook-adapter.js +3 -0
- package/dist/domains/comet-entry/hook-adapter.js.map +1 -1
- package/dist/domains/comet-entry/hook-router-entry.d.ts +2 -0
- package/dist/domains/comet-entry/hook-router-entry.d.ts.map +1 -1
- package/dist/domains/comet-entry/hook-router-entry.js +32 -3
- package/dist/domains/comet-entry/hook-router-entry.js.map +1 -1
- package/dist/domains/comet-entry/init-workflow.d.ts.map +1 -1
- package/dist/domains/comet-entry/init-workflow.js +13 -5
- package/dist/domains/comet-entry/init-workflow.js.map +1 -1
- package/dist/domains/comet-entry/project-activation.d.ts.map +1 -1
- package/dist/domains/comet-entry/project-activation.js +2 -1
- package/dist/domains/comet-entry/project-activation.js.map +1 -1
- package/dist/domains/comet-entry/resume-probe.d.ts.map +1 -1
- package/dist/domains/comet-entry/resume-probe.js +48 -4
- package/dist/domains/comet-entry/resume-probe.js.map +1 -1
- package/dist/domains/comet-native/index.d.ts +1 -0
- package/dist/domains/comet-native/index.d.ts.map +1 -1
- package/dist/domains/comet-native/index.js +1 -0
- package/dist/domains/comet-native/index.js.map +1 -1
- package/dist/domains/comet-native/native-acceptance.d.ts +4 -3
- package/dist/domains/comet-native/native-acceptance.d.ts.map +1 -1
- package/dist/domains/comet-native/native-acceptance.js +23 -18
- package/dist/domains/comet-native/native-acceptance.js.map +1 -1
- package/dist/domains/comet-native/native-archive-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-archive-command.js +204 -61
- package/dist/domains/comet-native/native-archive-command.js.map +1 -1
- package/dist/domains/comet-native/native-archive-transaction.d.ts.map +1 -1
- package/dist/domains/comet-native/native-archive-transaction.js +28 -23
- package/dist/domains/comet-native/native-archive-transaction.js.map +1 -1
- package/dist/domains/comet-native/native-archive.d.ts +1 -0
- package/dist/domains/comet-native/native-archive.d.ts.map +1 -1
- package/dist/domains/comet-native/native-archive.js +56 -12
- package/dist/domains/comet-native/native-archive.js.map +1 -1
- package/dist/domains/comet-native/native-bounded-file.d.ts +32 -2
- package/dist/domains/comet-native/native-bounded-file.d.ts.map +1 -1
- package/dist/domains/comet-native/native-bounded-file.js +87 -6
- package/dist/domains/comet-native/native-bounded-file.js.map +1 -1
- package/dist/domains/comet-native/native-build-evidence.d.ts.map +1 -1
- package/dist/domains/comet-native/native-build-evidence.js +23 -1
- package/dist/domains/comet-native/native-build-evidence.js.map +1 -1
- package/dist/domains/comet-native/native-change.d.ts +1 -0
- package/dist/domains/comet-native/native-change.d.ts.map +1 -1
- package/dist/domains/comet-native/native-change.js +96 -8
- package/dist/domains/comet-native/native-change.js.map +1 -1
- package/dist/domains/comet-native/native-check-executor.d.ts +36 -0
- package/dist/domains/comet-native/native-check-executor.d.ts.map +1 -0
- package/dist/domains/comet-native/native-check-executor.js +132 -0
- package/dist/domains/comet-native/native-check-executor.js.map +1 -0
- package/dist/domains/comet-native/native-check-receipt-model.d.ts +9 -0
- package/dist/domains/comet-native/native-check-receipt-model.d.ts.map +1 -1
- package/dist/domains/comet-native/native-check-receipt-model.js +70 -8
- package/dist/domains/comet-native/native-check-receipt-model.js.map +1 -1
- package/dist/domains/comet-native/native-check-receipt-storage.d.ts.map +1 -1
- package/dist/domains/comet-native/native-check-receipt-storage.js +9 -8
- package/dist/domains/comet-native/native-check-receipt-storage.js.map +1 -1
- package/dist/domains/comet-native/native-check-receipt.d.ts.map +1 -1
- package/dist/domains/comet-native/native-check-receipt.js +123 -60
- package/dist/domains/comet-native/native-check-receipt.js.map +1 -1
- package/dist/domains/comet-native/native-checkpoint-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-checkpoint-command.js +0 -6
- package/dist/domains/comet-native/native-checkpoint-command.js.map +1 -1
- package/dist/domains/comet-native/native-checkpoint-storage.d.ts.map +1 -1
- package/dist/domains/comet-native/native-checkpoint-storage.js +27 -19
- package/dist/domains/comet-native/native-checkpoint-storage.js.map +1 -1
- package/dist/domains/comet-native/native-children.d.ts +49 -0
- package/dist/domains/comet-native/native-children.d.ts.map +1 -0
- package/dist/domains/comet-native/native-children.js +388 -0
- package/dist/domains/comet-native/native-children.js.map +1 -0
- package/dist/domains/comet-native/native-cli-help.d.ts +6 -0
- package/dist/domains/comet-native/native-cli-help.d.ts.map +1 -0
- package/dist/domains/comet-native/native-cli-help.js +192 -0
- package/dist/domains/comet-native/native-cli-help.js.map +1 -0
- package/dist/domains/comet-native/native-cli-shared.d.ts +2 -2
- package/dist/domains/comet-native/native-cli-shared.d.ts.map +1 -1
- package/dist/domains/comet-native/native-cli-shared.js +46 -24
- package/dist/domains/comet-native/native-cli-shared.js.map +1 -1
- package/dist/domains/comet-native/native-cli.d.ts.map +1 -1
- package/dist/domains/comet-native/native-cli.js +18 -11
- package/dist/domains/comet-native/native-cli.js.map +1 -1
- package/dist/domains/comet-native/native-config.d.ts +2 -1
- package/dist/domains/comet-native/native-config.d.ts.map +1 -1
- package/dist/domains/comet-native/native-config.js +2 -1
- package/dist/domains/comet-native/native-config.js.map +1 -1
- package/dist/domains/comet-native/native-continuation.d.ts.map +1 -1
- package/dist/domains/comet-native/native-continuation.js +175 -147
- package/dist/domains/comet-native/native-continuation.js.map +1 -1
- package/dist/domains/comet-native/native-diagnostics.d.ts.map +1 -1
- package/dist/domains/comet-native/native-diagnostics.js +96 -13
- package/dist/domains/comet-native/native-diagnostics.js.map +1 -1
- package/dist/domains/comet-native/native-doctor-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-doctor-command.js +332 -0
- package/dist/domains/comet-native/native-doctor-command.js.map +1 -1
- package/dist/domains/comet-native/native-doctor.d.ts +1 -0
- package/dist/domains/comet-native/native-doctor.d.ts.map +1 -1
- package/dist/domains/comet-native/native-doctor.js +116 -22
- package/dist/domains/comet-native/native-doctor.js.map +1 -1
- package/dist/domains/comet-native/native-evidence-projection.d.ts +50 -0
- package/dist/domains/comet-native/native-evidence-projection.d.ts.map +1 -0
- package/dist/domains/comet-native/native-evidence-projection.js +284 -0
- package/dist/domains/comet-native/native-evidence-projection.js.map +1 -0
- package/dist/domains/comet-native/native-evidence-retention.d.ts.map +1 -1
- package/dist/domains/comet-native/native-evidence-retention.js +14 -12
- package/dist/domains/comet-native/native-evidence-retention.js.map +1 -1
- package/dist/domains/comet-native/native-evidence-storage.d.ts.map +1 -1
- package/dist/domains/comet-native/native-evidence-storage.js +14 -11
- package/dist/domains/comet-native/native-evidence-storage.js.map +1 -1
- package/dist/domains/comet-native/native-findings.d.ts.map +1 -1
- package/dist/domains/comet-native/native-findings.js +18 -0
- package/dist/domains/comet-native/native-findings.js.map +1 -1
- package/dist/domains/comet-native/native-git-provenance.d.ts +16 -0
- package/dist/domains/comet-native/native-git-provenance.d.ts.map +1 -0
- package/dist/domains/comet-native/native-git-provenance.js +171 -0
- package/dist/domains/comet-native/native-git-provenance.js.map +1 -0
- package/dist/domains/comet-native/native-hook-guard.d.ts.map +1 -1
- package/dist/domains/comet-native/native-hook-guard.js +199 -19
- package/dist/domains/comet-native/native-hook-guard.js.map +1 -1
- package/dist/domains/comet-native/native-init-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-init-command.js +14 -2
- package/dist/domains/comet-native/native-init-command.js.map +1 -1
- package/dist/domains/comet-native/native-local-execution.d.ts +25 -0
- package/dist/domains/comet-native/native-local-execution.d.ts.map +1 -0
- package/dist/domains/comet-native/native-local-execution.js +241 -0
- package/dist/domains/comet-native/native-local-execution.js.map +1 -0
- package/dist/domains/comet-native/native-lock.d.ts +1 -1
- package/dist/domains/comet-native/native-lock.d.ts.map +1 -1
- package/dist/domains/comet-native/native-lock.js +9 -9
- package/dist/domains/comet-native/native-lock.js.map +1 -1
- package/dist/domains/comet-native/native-loop-runtime.d.ts +63 -0
- package/dist/domains/comet-native/native-loop-runtime.d.ts.map +1 -0
- package/dist/domains/comet-native/native-loop-runtime.js +580 -0
- package/dist/domains/comet-native/native-loop-runtime.js.map +1 -0
- package/dist/domains/comet-native/native-mutation-lock.d.ts +2 -0
- package/dist/domains/comet-native/native-mutation-lock.d.ts.map +1 -1
- package/dist/domains/comet-native/native-mutation-lock.js +23 -2
- package/dist/domains/comet-native/native-mutation-lock.js.map +1 -1
- package/dist/domains/comet-native/native-new-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-new-command.js +27 -20
- package/dist/domains/comet-native/native-new-command.js.map +1 -1
- package/dist/domains/comet-native/native-next-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-next-command.js +193 -62
- package/dist/domains/comet-native/native-next-command.js.map +1 -1
- package/dist/domains/comet-native/native-paths.d.ts +18 -0
- package/dist/domains/comet-native/native-paths.d.ts.map +1 -1
- package/dist/domains/comet-native/native-paths.js +125 -13
- package/dist/domains/comet-native/native-paths.js.map +1 -1
- package/dist/domains/comet-native/native-portable-acceptance.d.ts +28 -0
- package/dist/domains/comet-native/native-portable-acceptance.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-acceptance.js +73 -0
- package/dist/domains/comet-native/native-portable-acceptance.js.map +1 -0
- package/dist/domains/comet-native/native-portable-archive.d.ts +37 -0
- package/dist/domains/comet-native/native-portable-archive.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-archive.js +492 -0
- package/dist/domains/comet-native/native-portable-archive.js.map +1 -0
- package/dist/domains/comet-native/native-portable-continuation.d.ts +30 -0
- package/dist/domains/comet-native/native-portable-continuation.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-continuation.js +403 -0
- package/dist/domains/comet-native/native-portable-continuation.js.map +1 -0
- package/dist/domains/comet-native/native-portable-migration-runtime.d.ts +9 -0
- package/dist/domains/comet-native/native-portable-migration-runtime.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-migration-runtime.js +213 -0
- package/dist/domains/comet-native/native-portable-migration-runtime.js.map +1 -0
- package/dist/domains/comet-native/native-portable-migration.d.ts +42 -0
- package/dist/domains/comet-native/native-portable-migration.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-migration.js +212 -0
- package/dist/domains/comet-native/native-portable-migration.js.map +1 -0
- package/dist/domains/comet-native/native-portable-recovery.d.ts +15 -0
- package/dist/domains/comet-native/native-portable-recovery.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-recovery.js +203 -0
- package/dist/domains/comet-native/native-portable-recovery.js.map +1 -0
- package/dist/domains/comet-native/native-portable-runtime.d.ts +155 -0
- package/dist/domains/comet-native/native-portable-runtime.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-runtime.js +1550 -0
- package/dist/domains/comet-native/native-portable-runtime.js.map +1 -0
- package/dist/domains/comet-native/native-portable-state.d.ts +30 -0
- package/dist/domains/comet-native/native-portable-state.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-state.js +701 -0
- package/dist/domains/comet-native/native-portable-state.js.map +1 -0
- package/dist/domains/comet-native/native-portable-status.d.ts +65 -0
- package/dist/domains/comet-native/native-portable-status.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-status.js +133 -0
- package/dist/domains/comet-native/native-portable-status.js.map +1 -0
- package/dist/domains/comet-native/native-portable-text.d.ts +8 -0
- package/dist/domains/comet-native/native-portable-text.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-text.js +28 -0
- package/dist/domains/comet-native/native-portable-text.js.map +1 -0
- package/dist/domains/comet-native/native-portable-transactions.d.ts +41 -0
- package/dist/domains/comet-native/native-portable-transactions.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-transactions.js +204 -0
- package/dist/domains/comet-native/native-portable-transactions.js.map +1 -0
- package/dist/domains/comet-native/native-portable-types.d.ts +166 -0
- package/dist/domains/comet-native/native-portable-types.d.ts.map +1 -0
- package/dist/domains/comet-native/native-portable-types.js +18 -0
- package/dist/domains/comet-native/native-portable-types.js.map +1 -0
- package/dist/domains/comet-native/native-repair-integration.js +5 -5
- package/dist/domains/comet-native/native-repair-integration.js.map +1 -1
- package/dist/domains/comet-native/native-resume-view.d.ts.map +1 -1
- package/dist/domains/comet-native/native-resume-view.js +1 -5
- package/dist/domains/comet-native/native-resume-view.js.map +1 -1
- package/dist/domains/comet-native/native-root-move.d.ts.map +1 -1
- package/dist/domains/comet-native/native-root-move.js +4 -20
- package/dist/domains/comet-native/native-root-move.js.map +1 -1
- package/dist/domains/comet-native/native-run-consistency.d.ts.map +1 -1
- package/dist/domains/comet-native/native-run-consistency.js +8 -12
- package/dist/domains/comet-native/native-run-consistency.js.map +1 -1
- package/dist/domains/comet-native/native-run-store.js +4 -4
- package/dist/domains/comet-native/native-run-store.js.map +1 -1
- package/dist/domains/comet-native/native-runner-input.d.ts +160 -0
- package/dist/domains/comet-native/native-runner-input.d.ts.map +1 -0
- package/dist/domains/comet-native/native-runner-input.js +347 -0
- package/dist/domains/comet-native/native-runner-input.js.map +1 -0
- package/dist/domains/comet-native/native-runner-protocol.d.ts +40 -0
- package/dist/domains/comet-native/native-runner-protocol.d.ts.map +1 -0
- package/dist/domains/comet-native/native-runner-protocol.js +55 -0
- package/dist/domains/comet-native/native-runner-protocol.js.map +1 -0
- package/dist/domains/comet-native/native-runtime-package.d.ts +3 -0
- package/dist/domains/comet-native/native-runtime-package.d.ts.map +1 -1
- package/dist/domains/comet-native/native-runtime-package.js +5 -1
- package/dist/domains/comet-native/native-runtime-package.js.map +1 -1
- package/dist/domains/comet-native/native-schema-migration.d.ts.map +1 -1
- package/dist/domains/comet-native/native-schema-migration.js +30 -27
- package/dist/domains/comet-native/native-schema-migration.js.map +1 -1
- package/dist/domains/comet-native/native-select-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-select-command.js +8 -4
- package/dist/domains/comet-native/native-select-command.js.map +1 -1
- package/dist/domains/comet-native/native-selection.d.ts.map +1 -1
- package/dist/domains/comet-native/native-selection.js +11 -2
- package/dist/domains/comet-native/native-selection.js.map +1 -1
- package/dist/domains/comet-native/native-show-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-show-command.js +37 -0
- package/dist/domains/comet-native/native-show-command.js.map +1 -1
- package/dist/domains/comet-native/native-snapshot.d.ts.map +1 -1
- package/dist/domains/comet-native/native-snapshot.js +13 -10
- package/dist/domains/comet-native/native-snapshot.js.map +1 -1
- package/dist/domains/comet-native/native-spec-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-spec-command.js +14 -28
- package/dist/domains/comet-native/native-spec-command.js.map +1 -1
- package/dist/domains/comet-native/native-specs.js +3 -3
- package/dist/domains/comet-native/native-specs.js.map +1 -1
- package/dist/domains/comet-native/native-status-command.d.ts.map +1 -1
- package/dist/domains/comet-native/native-status-command.js +7 -17
- package/dist/domains/comet-native/native-status-command.js.map +1 -1
- package/dist/domains/comet-native/native-status-discovery.d.ts +51 -0
- package/dist/domains/comet-native/native-status-discovery.d.ts.map +1 -0
- package/dist/domains/comet-native/native-status-discovery.js +299 -0
- package/dist/domains/comet-native/native-status-discovery.js.map +1 -0
- package/dist/domains/comet-native/native-trajectory-recovery.d.ts.map +1 -1
- package/dist/domains/comet-native/native-trajectory-recovery.js +6 -9
- package/dist/domains/comet-native/native-trajectory-recovery.js.map +1 -1
- package/dist/domains/comet-native/native-transaction.d.ts.map +1 -1
- package/dist/domains/comet-native/native-transaction.js +31 -20
- package/dist/domains/comet-native/native-transaction.js.map +1 -1
- package/dist/domains/comet-native/native-transition-evidence.d.ts.map +1 -1
- package/dist/domains/comet-native/native-transition-evidence.js +1 -0
- package/dist/domains/comet-native/native-transition-evidence.js.map +1 -1
- package/dist/domains/comet-native/native-transition-journal.d.ts.map +1 -1
- package/dist/domains/comet-native/native-transition-journal.js +71 -22
- package/dist/domains/comet-native/native-transition-journal.js.map +1 -1
- package/dist/domains/comet-native/native-transitions.d.ts.map +1 -1
- package/dist/domains/comet-native/native-transitions.js +167 -15
- package/dist/domains/comet-native/native-transitions.js.map +1 -1
- package/dist/domains/comet-native/native-types.d.ts +36 -1
- package/dist/domains/comet-native/native-types.d.ts.map +1 -1
- package/dist/domains/comet-native/native-verification-receipt-runtime.d.ts.map +1 -1
- package/dist/domains/comet-native/native-verification-receipt-runtime.js +18 -100
- package/dist/domains/comet-native/native-verification-receipt-runtime.js.map +1 -1
- package/dist/domains/comet-native/native-verification-report-v2.d.ts +12 -0
- package/dist/domains/comet-native/native-verification-report-v2.d.ts.map +1 -0
- package/dist/domains/comet-native/native-verification-report-v2.js +141 -0
- package/dist/domains/comet-native/native-verification-report-v2.js.map +1 -0
- package/dist/domains/comet-native/native-verification-runtime.d.ts.map +1 -1
- package/dist/domains/comet-native/native-verification-runtime.js +7 -2
- package/dist/domains/comet-native/native-verification-runtime.js.map +1 -1
- package/dist/domains/comet-native/native-verification-scope.d.ts +23 -0
- package/dist/domains/comet-native/native-verification-scope.d.ts.map +1 -1
- package/dist/domains/comet-native/native-verification-scope.js +106 -19
- package/dist/domains/comet-native/native-verification-scope.js.map +1 -1
- package/dist/domains/comet-native/native-verifier-protocol.d.ts +49 -0
- package/dist/domains/comet-native/native-verifier-protocol.d.ts.map +1 -0
- package/dist/domains/comet-native/native-verifier-protocol.js +157 -0
- package/dist/domains/comet-native/native-verifier-protocol.js.map +1 -0
- package/dist/domains/comet-native/native-workspace-finish.d.ts +61 -0
- package/dist/domains/comet-native/native-workspace-finish.d.ts.map +1 -0
- package/dist/domains/comet-native/native-workspace-finish.js +332 -0
- package/dist/domains/comet-native/native-workspace-finish.js.map +1 -0
- package/dist/domains/comet-native/native-workspace-preparation.d.ts +32 -0
- package/dist/domains/comet-native/native-workspace-preparation.d.ts.map +1 -0
- package/dist/domains/comet-native/native-workspace-preparation.js +324 -0
- package/dist/domains/comet-native/native-workspace-preparation.js.map +1 -0
- package/dist/domains/comet-native/native-workspace.d.ts +9 -1
- package/dist/domains/comet-native/native-workspace.d.ts.map +1 -1
- package/dist/domains/comet-native/native-workspace.js +109 -11
- package/dist/domains/comet-native/native-workspace.js.map +1 -1
- package/dist/domains/dashboard/collector.d.ts +1 -1
- package/dist/domains/dashboard/collector.d.ts.map +1 -1
- package/dist/domains/dashboard/collector.js +102 -87
- package/dist/domains/dashboard/collector.js.map +1 -1
- package/dist/domains/dashboard/native-adapter.d.ts +155 -126
- package/dist/domains/dashboard/native-adapter.d.ts.map +1 -1
- package/dist/domains/dashboard/native-adapter.js +165 -268
- package/dist/domains/dashboard/native-adapter.js.map +1 -1
- package/dist/domains/dashboard/native-collector.d.ts +5 -4
- package/dist/domains/dashboard/native-collector.d.ts.map +1 -1
- package/dist/domains/dashboard/native-collector.js +506 -439
- package/dist/domains/dashboard/native-collector.js.map +1 -1
- package/dist/domains/dashboard/native-legacy-adapter.d.ts +37 -0
- package/dist/domains/dashboard/native-legacy-adapter.d.ts.map +1 -0
- package/dist/domains/dashboard/native-legacy-adapter.js +127 -0
- package/dist/domains/dashboard/native-legacy-adapter.js.map +1 -0
- package/dist/domains/dashboard/server.d.ts.map +1 -1
- package/dist/domains/dashboard/server.js +5 -3
- package/dist/domains/dashboard/server.js.map +1 -1
- package/dist/domains/dashboard/types.d.ts +5 -0
- package/dist/domains/dashboard/types.d.ts.map +1 -1
- package/dist/domains/dashboard/web/assets/abnfDiagram-N423BO3Z-B5cFR6hS.js +1 -0
- package/dist/domains/dashboard/web/assets/arc-CVQDtMsu.js +1 -0
- package/dist/domains/dashboard/web/assets/architectureDiagram-T3A2C74G-C0v0kULK.js +36 -0
- package/dist/domains/dashboard/web/assets/blockDiagram-VBNYF7ZC-DpHLuxu4.js +132 -0
- package/dist/domains/dashboard/web/assets/{c4Diagram-AAUBKEIU-Ca5zba3N.js → c4Diagram-5PPSVZJV-BHnFxUs1.js} +1 -1
- package/dist/domains/dashboard/web/assets/channel-eM0x3m1G.js +1 -0
- package/dist/domains/dashboard/web/assets/{chunk-ND2GUHAM-Bj6HWRwi.js → chunk-2GRJ4B5K-CGC9gh5n.js} +1 -1
- package/dist/domains/dashboard/web/assets/{chunk-QZHKN3VN-B1kXfg5N.js → chunk-2Q5K7J3B-l4puLU_O.js} +1 -1
- package/dist/domains/dashboard/web/assets/chunk-5RXB4S5H-ls9jQHZc.js +231 -0
- package/dist/domains/dashboard/web/assets/{chunk-FMBD7UC4-BAJLB5V6.js → chunk-5VM5RSS4-BVavmTEq.js} +1 -1
- package/dist/domains/dashboard/web/assets/chunk-6Q2QTUOP-Dungt2g9.js +88 -0
- package/dist/domains/dashboard/web/assets/{chunk-727SXJPM-CwghIMO9.js → chunk-GF5L2VYU-BidXO_ZP.js} +5 -5
- package/dist/domains/dashboard/web/assets/{chunk-4BX2VUAB-JYluRsP-.js → chunk-JWPE2WC7-57L5Up5X.js} +1 -1
- package/dist/domains/dashboard/web/assets/{chunk-2J33WTMH-CCVPBm0J.js → chunk-KBJHAD2P-C5u5vbwr.js} +1 -1
- package/dist/domains/dashboard/web/assets/chunk-RYQCIY6F-CvLz21I2.js +1 -0
- package/dist/domains/dashboard/web/assets/{chunk-55IACEB6-DfYm7pN7.js → chunk-XXDRQBXY-D5nJpckG.js} +1 -1
- package/dist/domains/dashboard/web/assets/classDiagram-JCYQIIEL-CEFhao2K.js +1 -0
- package/dist/domains/dashboard/web/assets/classDiagram-v2-OCEON4UE-CEFhao2K.js +1 -0
- package/dist/domains/dashboard/web/assets/{cose-bilkent-S5V4N54A-BBrkuiAU.js → cose-bilkent-JH36ORCC-Dp0JAGyE.js} +1 -1
- package/dist/domains/dashboard/web/assets/{cynefin-VYW2F7L2-DlFhgQHt.js → cynefin-VYW2F7L2-EjC54l1g.js} +2 -2
- package/dist/domains/dashboard/web/assets/cynefinDiagram-MW4NZA55-CpYrW3w3.js +62 -0
- package/dist/domains/dashboard/web/assets/dagre-VZM6K2ZE-h5K39rCk.js +4 -0
- package/dist/domains/dashboard/web/assets/demo-BfLsKrIL.js +45 -0
- package/dist/domains/dashboard/web/assets/diagram-7IWD3JNH-CKQj9O4H.js +30 -0
- package/dist/domains/dashboard/web/assets/{diagram-KO2AKTUF-BQahyP-C.js → diagram-B4RE2ZJO-DXRIYcnW.js} +3 -3
- package/dist/domains/dashboard/web/assets/diagram-LBJQPF4R-wWutQYmv.js +24 -0
- package/dist/domains/dashboard/web/assets/diagram-Q27KOJAE-EgXLJQYi.js +24 -0
- package/dist/domains/dashboard/web/assets/diagram-UB23O5K3-CsQRXDQZ.js +41 -0
- package/dist/domains/dashboard/web/assets/ebnfDiagram-BXEA7PRR-gm328Eho.js +1 -0
- package/dist/domains/dashboard/web/assets/erDiagram-JOGREHBK-BbbMVUx_.js +85 -0
- package/dist/domains/dashboard/web/assets/flowDiagram-UKHOOZJN-1Y7scCns.js +156 -0
- package/dist/domains/dashboard/web/assets/ganttDiagram-PKOTCBZU-BAoBqczv.js +292 -0
- package/dist/domains/dashboard/web/assets/{gitGraphDiagram-PVQCEYII-DCxvMAUJ.js → gitGraphDiagram-DS77QQ5N-CMUAyEzl.js} +3 -3
- package/dist/domains/dashboard/web/assets/{graph--OzhPTMs.js → graph-DOmOIIwC.js} +1 -1
- package/dist/domains/dashboard/web/assets/{index-Blit0P5Z.js → index-CCApJ0Yr.js} +1 -1
- package/dist/domains/dashboard/web/assets/index-DWdm_XBi.js +257 -0
- package/dist/domains/dashboard/web/assets/index-DhhbseFJ.css +1 -0
- package/dist/domains/dashboard/web/assets/{infoDiagram-5YYISTIA-9qXEkzDe.js → infoDiagram-6WML65LV-BaCdtYYC.js} +1 -1
- package/dist/domains/dashboard/web/assets/{ishikawaDiagram-YF4QCWOH-Dkf5othW.js → ishikawaDiagram-WSZJBQD7-DQQlNnjU.js} +4 -4
- package/dist/domains/dashboard/web/assets/{journeyDiagram-JHISSGLW-C2_IkO4s.js → journeyDiagram-NVQOT4AX-DW9wUMqn.js} +1 -1
- package/dist/domains/dashboard/web/assets/{kanban-definition-UN3LZRKU-BxWgu4DL.js → kanban-definition-27J2QSJJ-C_ITaZ0q.js} +7 -7
- package/dist/domains/dashboard/web/assets/layout-D-LzfAck.js +1 -0
- package/dist/domains/dashboard/web/assets/{linear-Dkr6p-Ro.js → linear-DXOo6ZRT.js} +1 -1
- package/dist/domains/dashboard/web/assets/map-DxJ2ADlA.js +1 -0
- package/dist/domains/dashboard/web/assets/mermaid.core-ol-0NI97.js +306 -0
- package/dist/domains/dashboard/web/assets/{mindmap-definition-RKZ34NQL-C9xyxGfa.js → mindmap-definition-FAOFIHXS-CHF_HOLV.js} +1 -1
- package/dist/domains/dashboard/web/assets/pegDiagram-VL7TDLO6-wQEvkNF8.js +1 -0
- package/dist/domains/dashboard/web/assets/pieDiagram-7S7Q4E2Y-C3H3kiIP.js +39 -0
- package/dist/domains/dashboard/web/assets/purify.es-Cz4mVeUR.js +3 -0
- package/dist/domains/dashboard/web/assets/{quadrantDiagram-W4KKPZXB-Cy6iOcxc.js → quadrantDiagram-CIZ2JOQS-S0FdVzJb.js} +4 -4
- package/dist/domains/dashboard/web/assets/railroadDiagram-AXF67PYL-DCDXDaVg.js +1 -0
- package/dist/domains/dashboard/web/assets/{requirementDiagram-4Y6WPE33-B_gZONhx.js → requirementDiagram-LRYGKXZP-BW8J4gO_.js} +1 -1
- package/dist/domains/dashboard/web/assets/sankeyDiagram-W5VNT64P-DQQBmLoy.js +40 -0
- package/dist/domains/dashboard/web/assets/sequenceDiagram-SI44F4Z6-2tDcli0Y.js +162 -0
- package/dist/domains/dashboard/web/assets/sizeCapture-X5ZJPWSS-4sEZLBKb.js +1 -0
- package/dist/domains/dashboard/web/assets/stateDiagram-OKZ733FA-OgPb8pyR.js +1 -0
- package/dist/domains/dashboard/web/assets/stateDiagram-v2-UEYNNEHI-CiDNPr4P.js +1 -0
- package/dist/domains/dashboard/web/assets/swimlanes-SLNWSIFB-DR7lwtxd.js +2 -0
- package/dist/domains/dashboard/web/assets/swimlanesDiagram-ULZ7WXOC-DggHFP9q.js +8 -0
- package/dist/domains/dashboard/web/assets/{timeline-definition-PNZ67QCA-DvONX2YQ.js → timeline-definition-Z64GVDOM-DNIJVlQV.js} +4 -4
- package/dist/domains/dashboard/web/assets/vennDiagram-T6HMQDX7-CClhw4QZ.js +34 -0
- package/dist/domains/dashboard/web/assets/{wardleyDiagram-YWT4CUSO-cf39WX2o.js → wardleyDiagram-T6FBY63Y-DzR3aFPS.js} +3 -3
- package/dist/domains/dashboard/web/assets/xychartDiagram-ELKLHX3M-DZhq0ZRj.js +7 -0
- package/dist/domains/dashboard/web/index.html +16 -16
- package/dist/domains/dashboard/workspace.d.ts +24 -0
- package/dist/domains/dashboard/workspace.d.ts.map +1 -0
- package/dist/domains/dashboard/workspace.js +87 -0
- package/dist/domains/dashboard/workspace.js.map +1 -0
- package/dist/domains/eval/index.d.ts +3 -0
- package/dist/domains/eval/index.d.ts.map +1 -1
- package/dist/domains/eval/index.js +3 -0
- package/dist/domains/eval/index.js.map +1 -1
- package/dist/domains/eval/standalone-context.d.ts +19 -0
- package/dist/domains/eval/standalone-context.d.ts.map +1 -0
- package/dist/domains/eval/standalone-context.js +107 -0
- package/dist/domains/eval/standalone-context.js.map +1 -0
- package/dist/domains/eval/standalone-static-collect.d.ts +22 -0
- package/dist/domains/eval/standalone-static-collect.d.ts.map +1 -0
- package/dist/domains/eval/standalone-static-collect.js +779 -0
- package/dist/domains/eval/standalone-static-collect.js.map +1 -0
- package/dist/domains/eval/user-environment.d.ts +6 -0
- package/dist/domains/eval/user-environment.d.ts.map +1 -0
- package/dist/domains/eval/user-environment.js +73 -0
- package/dist/domains/eval/user-environment.js.map +1 -0
- package/dist/domains/integrations/openspec.d.ts +1 -1
- package/dist/domains/integrations/openspec.d.ts.map +1 -1
- package/dist/domains/integrations/openspec.js +110 -26
- package/dist/domains/integrations/openspec.js.map +1 -1
- package/dist/domains/integrations/superpowers.d.ts +5 -1
- package/dist/domains/integrations/superpowers.d.ts.map +1 -1
- package/dist/domains/integrations/superpowers.js +21 -2
- package/dist/domains/integrations/superpowers.js.map +1 -1
- package/dist/domains/skill/platform-inspect.d.ts.map +1 -1
- package/dist/domains/skill/platform-inspect.js +11 -3
- package/dist/domains/skill/platform-inspect.js.map +1 -1
- package/dist/domains/skill/platform-install.d.ts +8 -2
- package/dist/domains/skill/platform-install.d.ts.map +1 -1
- package/dist/domains/skill/platform-install.js +115 -34
- package/dist/domains/skill/platform-install.js.map +1 -1
- package/dist/domains/skill/project-instructions.d.ts.map +1 -1
- package/dist/domains/skill/project-instructions.js +5 -3
- package/dist/domains/skill/project-instructions.js.map +1 -1
- package/dist/domains/skill/uninstall.d.ts.map +1 -1
- package/dist/domains/skill/uninstall.js +69 -15
- package/dist/domains/skill/uninstall.js.map +1 -1
- package/dist/domains/workflow-contract/hook-write-policy.d.ts +9 -0
- package/dist/domains/workflow-contract/hook-write-policy.d.ts.map +1 -0
- package/dist/domains/workflow-contract/hook-write-policy.js +52 -0
- package/dist/domains/workflow-contract/hook-write-policy.js.map +1 -0
- package/dist/domains/workflow-contract/index.d.ts +3 -0
- package/dist/domains/workflow-contract/index.d.ts.map +1 -1
- package/dist/domains/workflow-contract/index.js +2 -0
- package/dist/domains/workflow-contract/index.js.map +1 -1
- package/dist/domains/workflow-contract/project-config.d.ts +3 -1
- package/dist/domains/workflow-contract/project-config.d.ts.map +1 -1
- package/dist/domains/workflow-contract/project-config.js +119 -19
- package/dist/domains/workflow-contract/project-config.js.map +1 -1
- package/dist/domains/workflow-contract/project-gitignore.d.ts +9 -0
- package/dist/domains/workflow-contract/project-gitignore.d.ts.map +1 -0
- package/dist/domains/workflow-contract/project-gitignore.js +108 -0
- package/dist/domains/workflow-contract/project-gitignore.js.map +1 -0
- package/dist/domains/workflow-contract/protected-project-path.d.ts +13 -0
- package/dist/domains/workflow-contract/protected-project-path.d.ts.map +1 -1
- package/dist/domains/workflow-contract/protected-project-path.js +83 -0
- package/dist/domains/workflow-contract/protected-project-path.js.map +1 -1
- package/dist/domains/workflow-contract/types.d.ts +4 -0
- package/dist/domains/workflow-contract/types.d.ts.map +1 -1
- package/dist/platform/install/platforms.d.ts +3 -1
- package/dist/platform/install/platforms.d.ts.map +1 -1
- package/dist/platform/install/platforms.js +22 -2
- package/dist/platform/install/platforms.js.map +1 -1
- package/dist/platform/paths/git-worktree.d.ts +8 -1
- package/dist/platform/paths/git-worktree.d.ts.map +1 -1
- package/dist/platform/paths/git-worktree.js +41 -7
- package/dist/platform/paths/git-worktree.js.map +1 -1
- package/dist/platform/process/external-command.d.ts +13 -0
- package/dist/platform/process/external-command.d.ts.map +1 -0
- package/dist/platform/process/external-command.js +45 -0
- package/dist/platform/process/external-command.js.map +1 -0
- package/dist/platform/process/git.d.ts +12 -0
- package/dist/platform/process/git.d.ts.map +1 -0
- package/dist/platform/process/git.js +86 -0
- package/dist/platform/process/git.js.map +1 -0
- package/dist/platform/process/spawn-command.d.ts +8 -0
- package/dist/platform/process/spawn-command.d.ts.map +1 -0
- package/dist/platform/process/spawn-command.js +113 -0
- package/dist/platform/process/spawn-command.js.map +1 -0
- package/eval/.env.example +115 -0
- package/eval/README.md +237 -14
- package/eval/langfuse/.env.example +9 -0
- package/eval/langfuse/tests/conftest.py +146 -0
- package/eval/langfuse/tests/tasks/test_tasks.py +109 -0
- package/eval/langfuse/tests/test_adapter.py +272 -0
- package/eval/langsmith/.env.example +16 -0
- package/eval/langsmith/tests/conftest.py +10 -1
- package/eval/local/tasks/comet-classic-layout-lifecycle/environment/current-comet-package.json +4 -2
- package/eval/local/tasks/comet-native-clarification/validation/test_native_clarification.py +9 -21
- package/eval/local/tasks/comet-native-clarification-depth/validation/test_native_clarification_depth.py +4 -7
- package/eval/local/tasks/comet-native-clarification-modes/validation/test_native_clarification_modes.py +4 -10
- package/eval/local/tasks/comet-native-interrupted-transition/instruction.md +1 -1
- package/eval/local/tasks/comet-native-interrupted-transition/validation/test_native_interrupted_transition.py +15 -16
- package/eval/local/tasks/comet-native-repository-fact/instruction.md +2 -2
- package/eval/local/tasks/comet-native-repository-fact/validation/test_native_repository_fact.py +8 -2
- package/eval/local/tasks/comet-native-wave-b-decision-resume/instruction.md +8 -4
- package/eval/local/tasks/comet-native-wave-b-decision-resume/task.toml +6 -6
- package/eval/local/tasks/comet-native-wave-b-decision-resume/validation/test_native_wave_b_decision_resume.py +126 -135
- package/eval/local/tasks/comet-native-wave-c-verification-integrity/instruction.md +6 -6
- package/eval/local/tasks/comet-native-wave-c-verification-integrity/task.toml +8 -8
- package/eval/local/tasks/comet-native-wave-c-verification-integrity/validation/test_native_wave_c_verification_integrity.py +97 -317
- package/eval/local/tasks/comet-native-wave-d-stagnation-stop/instruction.md +7 -7
- package/eval/local/tasks/comet-native-wave-d-stagnation-stop/task.toml +4 -4
- package/eval/local/tasks/comet-native-wave-d-stagnation-stop/validation/test_native_wave_d_stagnation_stop.py +130 -199
- package/eval/local/tasks/comet-native-wave-e-parallel-safety/instruction.md +5 -5
- package/eval/local/tasks/comet-native-wave-e-parallel-safety/task.toml +8 -8
- package/eval/local/tasks/comet-native-wave-e-parallel-safety/validation/test_native_wave_e_parallel_safety.py +135 -239
- package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/environment/current-comet-package.json +2 -2
- package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/instruction.md +3 -3
- package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/task.toml +6 -6
- package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/validation/test_native_wave_f_dashboard_readonly.py +81 -439
- package/eval/local/tasks/comet-native-workflow/instruction.md +5 -4
- package/eval/local/tasks/comet-native-workflow/validation/test_native_workflow.py +145 -502
- package/eval/local/tasks/index.yaml +5 -5
- package/eval/local/tests/conftest.py +701 -80
- package/eval/local/tests/scaffold/fixtures/custom-agent/adapter.yaml +19 -0
- package/eval/local/tests/scaffold/test_agents.py +76 -0
- package/eval/local/tests/scaffold/test_aligned_comparison.py +70 -0
- package/eval/local/tests/scaffold/test_auto_tasks.py +350 -0
- package/eval/local/tests/scaffold/test_conftest_helpers.py +267 -18
- package/eval/local/tests/scaffold/test_custom_agents.py +174 -0
- package/eval/local/tests/scaffold/test_eval_context.py +217 -0
- package/eval/local/tests/scaffold/test_execution.py +364 -0
- package/eval/local/tests/scaffold/test_inline_tasks.py +56 -0
- package/eval/local/tests/scaffold/test_langfuse_trajectory.py +263 -0
- package/eval/local/tests/scaffold/test_langsmith_conftest.py +63 -0
- package/eval/local/tests/scaffold/test_logging.py +162 -0
- package/eval/local/tests/scaffold/test_manifest_tasks.py +76 -0
- package/eval/local/tests/scaffold/test_manifests.py +396 -0
- package/eval/local/tests/scaffold/test_native_aligned_eval.py +163 -35
- package/eval/local/tests/scaffold/test_native_wave_evaluations.py +75 -2054
- package/eval/local/tests/scaffold/test_profiles.py +64 -6
- package/eval/local/tests/scaffold/test_task_resolution.py +212 -0
- package/eval/local/tests/scaffold/test_tasks.py +38 -52
- package/eval/local/tests/scaffold/test_utils.py +480 -4
- package/eval/local/tests/tasks/test_tasks.py +219 -17
- package/eval/local/treatments/comet/comet_native_phase1.yaml +7 -4
- package/eval/pyproject.toml +9 -4
- package/eval/scaffold/__init__.py +6 -0
- package/eval/scaffold/docker/agent-overlay.Dockerfile +41 -0
- package/eval/scaffold/python/__init__.py +128 -20
- package/eval/scaffold/python/agents.py +239 -0
- package/eval/scaffold/python/aligned_comparison.py +156 -27
- package/eval/scaffold/python/auto_tasks.py +700 -0
- package/eval/scaffold/python/custom_agents.py +172 -0
- package/eval/scaffold/python/eval_context.py +236 -0
- package/eval/scaffold/python/eval_schema.py +135 -0
- package/eval/scaffold/python/execution.py +544 -0
- package/eval/scaffold/python/generated_task_cache.py +230 -0
- package/eval/scaffold/python/generic_llm_judge.py +41 -14
- package/eval/scaffold/python/inline_tasks.py +118 -0
- package/eval/scaffold/python/judge_config.py +158 -37
- package/eval/scaffold/python/langfuse_adapter.py +1023 -0
- package/eval/scaffold/python/llm_judge.py +68 -11
- package/eval/scaffold/python/logging.py +157 -17
- package/eval/scaffold/python/manifest_tasks.py +69 -0
- package/eval/scaffold/python/manifests.py +443 -22
- package/eval/scaffold/python/native_eval.py +10 -5
- package/eval/scaffold/python/paths.py +11 -1
- package/eval/scaffold/python/task_resolution.py +176 -0
- package/eval/scaffold/python/tasks.py +17 -4
- package/eval/scaffold/python/utils.py +167 -11
- package/eval/scaffold/python/validation/core.py +48 -2
- package/eval/scaffold/python/validation/generic_rubric.py +9 -0
- package/eval/scaffold/python/validation/native_v4.py +249 -0
- package/eval/scaffold/python/validation/native_wave.py +4 -5
- package/eval/scaffold/python/validation/native_workflow.py +199 -98
- package/eval/scaffold/python/validation/rubric.py +130 -2
- package/eval/scaffold/shell/agent-runtime-config.sh +88 -0
- package/eval/scaffold/shell/docker.sh +534 -33
- package/eval/scaffold/shell/run-agent-runtime.sh +49 -0
- package/eval/scaffold/shell/run-claude-loop.sh +153 -14
- package/eval/schemas/comet.eval/v1alpha1.schema.json +244 -0
- package/package.json +9 -3
- package/assets/skills/comet-native/scripts/comet-native-check.mjs +0 -179
- package/assets/skills/comet-native/scripts/comet-native-checkpoint.mjs +0 -179
- package/assets/skills/comet-native/scripts/comet-native-evidence.mjs +0 -151
- package/assets/skills/comet-native/scripts/comet-native-receipt.mjs +0 -187
- package/dist/domains/comet-native/native-check-entry.d.ts +0 -2
- package/dist/domains/comet-native/native-check-entry.d.ts.map +0 -1
- package/dist/domains/comet-native/native-check-entry.js +0 -4
- package/dist/domains/comet-native/native-check-entry.js.map +0 -1
- package/dist/domains/comet-native/native-checkpoint-entry.d.ts +0 -2
- package/dist/domains/comet-native/native-checkpoint-entry.d.ts.map +0 -1
- package/dist/domains/comet-native/native-checkpoint-entry.js +0 -4
- package/dist/domains/comet-native/native-checkpoint-entry.js.map +0 -1
- package/dist/domains/comet-native/native-evidence-entry.d.ts +0 -2
- package/dist/domains/comet-native/native-evidence-entry.d.ts.map +0 -1
- package/dist/domains/comet-native/native-evidence-entry.js +0 -4
- package/dist/domains/comet-native/native-evidence-entry.js.map +0 -1
- package/dist/domains/comet-native/native-receipt-entry.d.ts +0 -2
- package/dist/domains/comet-native/native-receipt-entry.d.ts.map +0 -1
- package/dist/domains/comet-native/native-receipt-entry.js +0 -4
- package/dist/domains/comet-native/native-receipt-entry.js.map +0 -1
- package/dist/domains/dashboard/web/assets/arc-ClrSujVb.js +0 -1
- package/dist/domains/dashboard/web/assets/architectureDiagram-3BPJPVTR-B4A_t84A.js +0 -36
- package/dist/domains/dashboard/web/assets/blockDiagram-GPEHLZMM-D72u2953.js +0 -132
- package/dist/domains/dashboard/web/assets/channel-B_u-_i2t.js +0 -1
- package/dist/domains/dashboard/web/assets/chunk-AQP2D5EJ-BA_zgMXZ.js +0 -231
- package/dist/domains/dashboard/web/assets/classDiagram-4FO5ZUOK-BpA7BPgX.js +0 -1
- package/dist/domains/dashboard/web/assets/classDiagram-v2-Q7XG4LA2-BpA7BPgX.js +0 -1
- package/dist/domains/dashboard/web/assets/dagre-BM42HDAG-B0Bk5oiP.js +0 -4
- package/dist/domains/dashboard/web/assets/demo-DZYPLZpF.js +0 -25
- package/dist/domains/dashboard/web/assets/diagram-2AECGRRQ-tvTUiBA2.js +0 -43
- package/dist/domains/dashboard/web/assets/diagram-5GNKFQAL-BZfcK29q.js +0 -10
- package/dist/domains/dashboard/web/assets/diagram-LMA3HP47-D5bJnW2C.js +0 -24
- package/dist/domains/dashboard/web/assets/diagram-OG6HWLK6-DTwWdzWJ.js +0 -24
- package/dist/domains/dashboard/web/assets/erDiagram-TEJ5UH35-BbvrdCp9.js +0 -85
- package/dist/domains/dashboard/web/assets/flowDiagram-I6XJVG4X-DgbR7Tcc.js +0 -162
- package/dist/domains/dashboard/web/assets/ganttDiagram-6RSMTGT7-wWMOA7F4.js +0 -292
- package/dist/domains/dashboard/web/assets/index-Bk1o-4VO.css +0 -1
- package/dist/domains/dashboard/web/assets/index-g8vyW0R_.js +0 -257
- package/dist/domains/dashboard/web/assets/layout-SsrduOYp.js +0 -1
- package/dist/domains/dashboard/web/assets/mermaid.core-DZUcsoG4.js +0 -301
- package/dist/domains/dashboard/web/assets/pieDiagram-4H26LBE5-Dn67tdOF.js +0 -30
- package/dist/domains/dashboard/web/assets/purify.es-VaSPOPhr.js +0 -3
- package/dist/domains/dashboard/web/assets/sankeyDiagram-5OEKKPKP-C3u-4dOk.js +0 -40
- package/dist/domains/dashboard/web/assets/sequenceDiagram-3UESZ5HK-Xd3ByTVF.js +0 -162
- package/dist/domains/dashboard/web/assets/stateDiagram-AJRCARHV-CJGHL7Up.js +0 -1
- package/dist/domains/dashboard/web/assets/stateDiagram-v2-BHNVJYJU-rKvBgr6I.js +0 -1
- package/dist/domains/dashboard/web/assets/vennDiagram-CIIHVFJN-B-L4qWy4.js +0 -34
- package/dist/domains/dashboard/web/assets/xychartDiagram-2RQKCTM6-Dk1vcm3u.js +0 -7
- /package/assets/skills/{comet → comet-classic}/reference/classic-layout.md +0 -0
- /package/assets/skills-zh/{comet → comet-classic}/reference/classic-layout.md +0 -0
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
import { spawn } from 'node:child_process';
|
|
2
|
+
import { existsSync } from 'node:fs';
|
|
3
|
+
import path from 'node:path';
|
|
4
|
+
const WINDOWS_SHIM_EXTENSIONS = new Set(['.bat', '.cmd', '.ps1']);
|
|
5
|
+
const WINDOWS_BATCH_EXTENSIONS = new Set(['.bat', '.cmd']);
|
|
6
|
+
const WINDOWS_BATCH_UNSAFE_ARGUMENT = /[&|<>()^%"!\r\n]/u;
|
|
7
|
+
const WINDOWS_POWERSHELL_SCRIPT = [
|
|
8
|
+
"$ProgressPreference = 'SilentlyContinue'",
|
|
9
|
+
'$encoded = $env:COMET_COMMAND_PAYLOAD',
|
|
10
|
+
'Remove-Item Env:COMET_COMMAND_PAYLOAD -ErrorAction SilentlyContinue',
|
|
11
|
+
'$json = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($encoded))',
|
|
12
|
+
'$payload = ConvertFrom-Json $json',
|
|
13
|
+
'$commandArgs = @($payload.arguments)',
|
|
14
|
+
'& $payload.command @commandArgs',
|
|
15
|
+
'if ($null -eq $LASTEXITCODE) { if ($?) { exit 0 } else { exit 1 } }',
|
|
16
|
+
'exit $LASTEXITCODE',
|
|
17
|
+
].join('; ');
|
|
18
|
+
function environmentValue(env, name) {
|
|
19
|
+
const expected = name.toLowerCase();
|
|
20
|
+
return Object.entries(env).find(([key, value]) => key.toLowerCase() === expected && value !== undefined)?.[1];
|
|
21
|
+
}
|
|
22
|
+
function windowsExecutableExtensions(env) {
|
|
23
|
+
const configured = (environmentValue(env, 'PATHEXT') ?? '.COM;.EXE;.BAT;.CMD')
|
|
24
|
+
.split(';')
|
|
25
|
+
.map((extension) => extension.trim().toLowerCase())
|
|
26
|
+
.filter(Boolean);
|
|
27
|
+
return [...new Set([...configured, '.ps1'])];
|
|
28
|
+
}
|
|
29
|
+
function windowsCommandCandidates(command, env, cwd) {
|
|
30
|
+
const hasPath = path.win32.isAbsolute(command) || /[\\/]/u.test(command);
|
|
31
|
+
const directories = hasPath
|
|
32
|
+
? ['']
|
|
33
|
+
: (environmentValue(env, 'PATH') ?? '')
|
|
34
|
+
.split(path.delimiter)
|
|
35
|
+
.map((directory) => directory.trim().replace(/^"(.*)"$/u, '$1'))
|
|
36
|
+
.filter(Boolean);
|
|
37
|
+
const extension = path.win32.extname(command);
|
|
38
|
+
const names = extension
|
|
39
|
+
? [command]
|
|
40
|
+
: windowsExecutableExtensions(env).map((candidate) => `${command}${candidate}`);
|
|
41
|
+
return directories.flatMap((directory) => names.map((name) => (directory ? path.join(directory, name) : path.resolve(cwd, name))));
|
|
42
|
+
}
|
|
43
|
+
function resolveWindowsCommand(command, env, cwd) {
|
|
44
|
+
return (windowsCommandCandidates(command, env, cwd).find((candidate) => existsSync(candidate)) ??
|
|
45
|
+
command);
|
|
46
|
+
}
|
|
47
|
+
function powershellExecutable(env) {
|
|
48
|
+
const systemRoot = environmentValue(env, 'SystemRoot');
|
|
49
|
+
if (systemRoot) {
|
|
50
|
+
const bundled = path.join(systemRoot, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe');
|
|
51
|
+
if (existsSync(bundled))
|
|
52
|
+
return bundled;
|
|
53
|
+
}
|
|
54
|
+
return 'powershell.exe';
|
|
55
|
+
}
|
|
56
|
+
function assertSafeWindowsBatchArguments(args) {
|
|
57
|
+
if (args.some((argument) => WINDOWS_BATCH_UNSAFE_ARGUMENT.test(argument))) {
|
|
58
|
+
throw new Error('Windows batch check arguments must not contain shell syntax');
|
|
59
|
+
}
|
|
60
|
+
}
|
|
61
|
+
function spawnWindowsShim(command, args, options) {
|
|
62
|
+
const payload = Buffer.from(JSON.stringify({ command, arguments: [...args] }), 'utf8').toString('base64');
|
|
63
|
+
const encodedScript = Buffer.from(WINDOWS_POWERSHELL_SCRIPT, 'utf16le').toString('base64');
|
|
64
|
+
return spawn(powershellExecutable(options.env), [
|
|
65
|
+
'-NoLogo',
|
|
66
|
+
'-NoProfile',
|
|
67
|
+
'-NonInteractive',
|
|
68
|
+
'-InputFormat',
|
|
69
|
+
'None',
|
|
70
|
+
'-OutputFormat',
|
|
71
|
+
'Text',
|
|
72
|
+
'-ExecutionPolicy',
|
|
73
|
+
'Bypass',
|
|
74
|
+
'-EncodedCommand',
|
|
75
|
+
encodedScript,
|
|
76
|
+
], {
|
|
77
|
+
cwd: options.cwd,
|
|
78
|
+
env: { ...options.env, COMET_COMMAND_PAYLOAD: payload },
|
|
79
|
+
shell: false,
|
|
80
|
+
windowsHide: true,
|
|
81
|
+
detached: false,
|
|
82
|
+
stdio: ['ignore', 'pipe', 'pipe'],
|
|
83
|
+
});
|
|
84
|
+
}
|
|
85
|
+
export function spawnCommand(command, args, options) {
|
|
86
|
+
const env = options.env ?? process.env;
|
|
87
|
+
if (process.platform !== 'win32') {
|
|
88
|
+
return spawn(command, [...args], {
|
|
89
|
+
cwd: options.cwd,
|
|
90
|
+
env,
|
|
91
|
+
shell: false,
|
|
92
|
+
windowsHide: true,
|
|
93
|
+
detached: true,
|
|
94
|
+
stdio: ['ignore', 'pipe', 'pipe'],
|
|
95
|
+
});
|
|
96
|
+
}
|
|
97
|
+
const resolved = resolveWindowsCommand(command, env, options.cwd);
|
|
98
|
+
const extension = path.win32.extname(resolved).toLowerCase();
|
|
99
|
+
if (WINDOWS_BATCH_EXTENSIONS.has(extension))
|
|
100
|
+
assertSafeWindowsBatchArguments(args);
|
|
101
|
+
if (WINDOWS_SHIM_EXTENSIONS.has(extension)) {
|
|
102
|
+
return spawnWindowsShim(resolved, args, { cwd: options.cwd, env });
|
|
103
|
+
}
|
|
104
|
+
return spawn(resolved, [...args], {
|
|
105
|
+
cwd: options.cwd,
|
|
106
|
+
env,
|
|
107
|
+
shell: false,
|
|
108
|
+
windowsHide: true,
|
|
109
|
+
detached: false,
|
|
110
|
+
stdio: ['ignore', 'pipe', 'pipe'],
|
|
111
|
+
});
|
|
112
|
+
}
|
|
113
|
+
//# sourceMappingURL=spawn-command.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"spawn-command.js","sourceRoot":"","sources":["../../../platform/process/spawn-command.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAA4B,MAAM,oBAAoB,CAAC;AACrE,OAAO,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AACrC,OAAO,IAAI,MAAM,WAAW,CAAC;AAK7B,MAAM,uBAAuB,GAAG,IAAI,GAAG,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC;AAClE,MAAM,wBAAwB,GAAG,IAAI,GAAG,CAAC,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC;AAC3D,MAAM,6BAA6B,GAAG,mBAAmB,CAAC;AAC1D,MAAM,yBAAyB,GAAG;IAChC,0CAA0C;IAC1C,uCAAuC;IACvC,qEAAqE;IACrE,gFAAgF;IAChF,mCAAmC;IACnC,sCAAsC;IACtC,iCAAiC;IACjC,qEAAqE;IACrE,oBAAoB;CACrB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAEb,SAAS,gBAAgB,CAAC,GAAsB,EAAE,IAAY;IAC5D,MAAM,QAAQ,GAAG,IAAI,CAAC,WAAW,EAAE,CAAC;IACpC,OAAO,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,IAAI,CAC7B,CAAC,CAAC,GAAG,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,WAAW,EAAE,KAAK,QAAQ,IAAI,KAAK,KAAK,SAAS,CACxE,EAAE,CAAC,CAAC,CAAC,CAAC;AACT,CAAC;AAED,SAAS,2BAA2B,CAAC,GAAsB;IACzD,MAAM,UAAU,GAAG,CAAC,gBAAgB,CAAC,GAAG,EAAE,SAAS,CAAC,IAAI,qBAAqB,CAAC;SAC3E,KAAK,CAAC,GAAG,CAAC;SACV,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC;SAClD,MAAM,CAAC,OAAO,CAAC,CAAC;IACnB,OAAO,CAAC,GAAG,IAAI,GAAG,CAAC,CAAC,GAAG,UAAU,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC;AAC/C,CAAC;AAED,SAAS,wBAAwB,CAAC,OAAe,EAAE,GAAsB,EAAE,GAAW;IACpF,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,UAAU,CAAC,OAAO,CAAC,IAAI,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;IACzE,MAAM,WAAW,GAAG,OAAO;QACzB,CAAC,CAAC,CAAC,EAAE,CAAC;QACN,CAAC,CAAC,CAAC,gBAAgB,CAAC,GAAG,EAAE,MAAM,CAAC,IAAI,EAAE,CAAC;aAClC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC;aACrB,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,OAAO,CAAC,WAAW,EAAE,IAAI,CAAC,CAAC;aAC/D,MAAM,CAAC,OAAO,CAAC,CAAC;IACvB,MAAM,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IAC9C,MAAM,KAAK,GAAG,SAAS;QACrB,CAAC,CAAC,CAAC,OAAO,CAAC;QACX,CAAC,CAAC,2BAA2B,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,GAAG,OAAO,GAAG,SAAS,EAAE,CAAC,CAAC;IAClF,OAAO,WAAW,CAAC,OAAO,CAAC,CAAC,SAAS,EAAE,EAAE,CACvC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC,CAAC,CACxF,CAAC;AACJ,CAAC;AAED,SAAS,qBAAqB,CAAC,OAAe,EAAE,GAAsB,EAAE,GAAW;IACjF,OAAO,CACL,wBAAwB,CAAC,OAAO,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QACtF,OAAO,CACR,CAAC;AACJ,CAAC;AAED,SAAS,oBAAoB,CAAC,GAAsB;IAClD,MAAM,UAAU,GAAG,gBAAgB,CAAC,GAAG,EAAE,YAAY,CAAC,CAAC;IACvD,IAAI,UAAU,EAAE,CAAC;QACf,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,CACvB,UAAU,EACV,UAAU,EACV,mBAAmB,EACnB,MAAM,EACN,gBAAgB,CACjB,CAAC;QACF,IAAI,UAAU,CAAC,OAAO,CAAC;YAAE,OAAO,OAAO,CAAC;IAC1C,CAAC;IACD,OAAO,gBAAgB,CAAC;AAC1B,CAAC;AAED,SAAS,+BAA+B,CAAC,IAAuB;IAC9D,IAAI,IAAI,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,6BAA6B,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC;QAC1E,MAAM,IAAI,KAAK,CAAC,6DAA6D,CAAC,CAAC;IACjF,CAAC;AACH,CAAC;AAED,SAAS,gBAAgB,CACvB,OAAe,EACf,IAAuB,EACvB,OAAgD;IAEhD,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,CAAC,QAAQ,CAC7F,QAAQ,CACT,CAAC;IACF,MAAM,aAAa,GAAG,MAAM,CAAC,IAAI,CAAC,yBAAyB,EAAE,SAAS,CAAC,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;IAC3F,OAAO,KAAK,CACV,oBAAoB,CAAC,OAAO,CAAC,GAAG,CAAC,EACjC;QACE,SAAS;QACT,YAAY;QACZ,iBAAiB;QACjB,cAAc;QACd,MAAM;QACN,eAAe;QACf,MAAM;QACN,kBAAkB;QAClB,QAAQ;QACR,iBAAiB;QACjB,aAAa;KACd,EACD;QACE,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,GAAG,EAAE,EAAE,GAAG,OAAO,CAAC,GAAG,EAAE,qBAAqB,EAAE,OAAO,EAAE;QACvD,KAAK,EAAE,KAAK;QACZ,WAAW,EAAE,IAAI;QACjB,QAAQ,EAAE,KAAK;QACf,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;KAClC,CACF,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,YAAY,CAC1B,OAAe,EACf,IAAuB,EACvB,OAAiD;IAEjD,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC;IACvC,IAAI,OAAO,CAAC,QAAQ,KAAK,OAAO,EAAE,CAAC;QACjC,OAAO,KAAK,CAAC,OAAO,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE;YAC/B,GAAG,EAAE,OAAO,CAAC,GAAG;YAChB,GAAG;YACH,KAAK,EAAE,KAAK;YACZ,WAAW,EAAE,IAAI;YACjB,QAAQ,EAAE,IAAI;YACd,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;SAClC,CAAC,CAAC;IACL,CAAC;IACD,MAAM,QAAQ,GAAG,qBAAqB,CAAC,OAAO,EAAE,GAAG,EAAE,OAAO,CAAC,GAAG,CAAC,CAAC;IAClE,MAAM,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,WAAW,EAAE,CAAC;IAC7D,IAAI,wBAAwB,CAAC,GAAG,CAAC,SAAS,CAAC;QAAE,+BAA+B,CAAC,IAAI,CAAC,CAAC;IACnF,IAAI,uBAAuB,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;QAC3C,OAAO,gBAAgB,CAAC,QAAQ,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,OAAO,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,CAAC;IACrE,CAAC;IACD,OAAO,KAAK,CAAC,QAAQ,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE;QAChC,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,GAAG;QACH,KAAK,EAAE,KAAK;QACZ,WAAW,EAAE,IAAI;QACjB,QAAQ,EAAE,KAAK;QACf,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;KAClC,CAAC,CAAC;AACL,CAAC"}
|
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
# `comet eval` creates this complete template automatically on first startup at:
|
|
2
|
+
# Unix: ~/.comet/eval/.env
|
|
3
|
+
# Windows: %USERPROFILE%\.comet\eval\.env
|
|
4
|
+
# Edit that user-level file as needed. Existing files are never overwritten.
|
|
5
|
+
# Process environment variables take precedence over this file.
|
|
6
|
+
|
|
7
|
+
# Default subject Agent when --agent is not supplied.
|
|
8
|
+
# BENCH_EVAL_AGENT=claude-code
|
|
9
|
+
|
|
10
|
+
# Common subject (Bench) configuration.
|
|
11
|
+
# BENCH_API_KEY=
|
|
12
|
+
# BENCH_BASE_URL=
|
|
13
|
+
# BENCH_MODEL=
|
|
14
|
+
|
|
15
|
+
# Optional independent LLM-as-judge configuration.
|
|
16
|
+
# BENCH_LLM_JUDGE=1
|
|
17
|
+
# BENCH_JUDGE_AGENT=
|
|
18
|
+
# BENCH_JUDGE_API_KEY=
|
|
19
|
+
# BENCH_JUDGE_BASE_URL=
|
|
20
|
+
# BENCH_JUDGE_MODEL=
|
|
21
|
+
|
|
22
|
+
# Provider-specific variables remain supported and override common fallbacks.
|
|
23
|
+
# ANTHROPIC_API_KEY=
|
|
24
|
+
|
|
25
|
+
# Optional Claude Code model override.
|
|
26
|
+
# BENCH_CC_MODEL=
|
|
27
|
+
# BENCH_CC_VERSION=latest
|
|
28
|
+
|
|
29
|
+
# Codex / Qoder / CodeBuddy credentials when selected with --agent.
|
|
30
|
+
# OPENAI_API_KEY=
|
|
31
|
+
# OPENAI_BASE_URL=
|
|
32
|
+
# OPENAI_MODEL=
|
|
33
|
+
# CODEX_API_KEY=
|
|
34
|
+
# CODEX_BASE_URL=
|
|
35
|
+
# CODEX_MODEL=
|
|
36
|
+
# BENCH_CODEX_VERSION=latest
|
|
37
|
+
# QODER_PERSONAL_ACCESS_TOKEN=
|
|
38
|
+
# QODER_BASE_URL=
|
|
39
|
+
# QODER_MODEL=
|
|
40
|
+
# BENCH_QODER_VERSION=latest
|
|
41
|
+
# CODEBUDDY_API_KEY=
|
|
42
|
+
# CODEBUDDY_AUTH_TOKEN=
|
|
43
|
+
# CODEBUDDY_BASE_URL=
|
|
44
|
+
# CODEBUDDY_MODEL=
|
|
45
|
+
# CODEBUDDY_SMALL_FAST_MODEL=
|
|
46
|
+
# CODEBUDDY_BIG_SLOW_MODEL=
|
|
47
|
+
# CODEBUDDY_CODE_SUBAGENT_MODEL=
|
|
48
|
+
# CODEBUDDY_CUSTOM_HEADERS=
|
|
49
|
+
# CODEBUDDY_INTERNET_ENVIRONMENT=
|
|
50
|
+
# BENCH_CODEBUDDY_VERSION=latest
|
|
51
|
+
# BENCH_CODEX_MODEL=
|
|
52
|
+
# BENCH_QODER_MODEL=
|
|
53
|
+
# BENCH_CODEBUDDY_MODEL=
|
|
54
|
+
|
|
55
|
+
# Optional user-installed custom Agent adapter registry.
|
|
56
|
+
# Default: ~/.comet/eval/adapters (Windows: %USERPROFILE%\.comet\eval\adapters)
|
|
57
|
+
# COMET_EVAL_ADAPTERS_DIR=
|
|
58
|
+
# Custom Agent credential/model/base URL names are declared by its adapter.yaml.
|
|
59
|
+
# Add those names here only after installing an adapter, for example:
|
|
60
|
+
# MY_AGENT_API_KEY=
|
|
61
|
+
# MY_AGENT_MODEL=
|
|
62
|
+
# MY_AGENT_BASE_URL=
|
|
63
|
+
|
|
64
|
+
# Optional: path to a custom user-simulator prompt for auto_user workflow evals.
|
|
65
|
+
# Relative paths resolve from eval/. Defaults to eval/simulator-instruction.md.
|
|
66
|
+
# BENCH_SIMULATOR_PROMPT_FILE=simulator-instruction.md
|
|
67
|
+
|
|
68
|
+
# Optional legacy Judge auth token. The common BENCH_JUDGE_API_KEY above is
|
|
69
|
+
# mapped to the selected Agent's native credential when this is not set.
|
|
70
|
+
# BENCH_JUDGE_AUTH_TOKEN=
|
|
71
|
+
|
|
72
|
+
# Claude Code via Anthropic-compatible proxy (BigModel / mimo / OpenRouter etc.).
|
|
73
|
+
# When ANTHROPIC_API_KEY is unset, claude authenticates with these instead.
|
|
74
|
+
# ANTHROPIC_AUTH_TOKEN=
|
|
75
|
+
# ANTHROPIC_BASE_URL=
|
|
76
|
+
# ANTHROPIC_MODEL=
|
|
77
|
+
# ANTHROPIC_DEFAULT_HAIKU_MODEL=
|
|
78
|
+
# ANTHROPIC_DEFAULT_SONNET_MODEL=
|
|
79
|
+
# ANTHROPIC_DEFAULT_OPUS_MODEL=
|
|
80
|
+
# ANTHROPIC_DEFAULT_OPUS_MODEL_NAME=
|
|
81
|
+
# ANTHROPIC_DEFAULT_SONNET_MODEL_NAME=
|
|
82
|
+
# CLAUDE_CODE_SUBAGENT_MODEL=
|
|
83
|
+
|
|
84
|
+
# Eval never writes API keys into published assets, manifests, reports, or the
|
|
85
|
+
# workspace. During a Docker run, Agent-specific config roots are mounted as
|
|
86
|
+
# container-local tmpfs files. Codex config uses env_key, CodeBuddy uses an
|
|
87
|
+
# apiKeyHelper, and Qoder reads its PAT from the isolated process environment.
|
|
88
|
+
|
|
89
|
+
# Required only for eval/langsmith.
|
|
90
|
+
# LANGSMITH_API_KEY=
|
|
91
|
+
# LANGSMITH_TRACING=true
|
|
92
|
+
# LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
|
93
|
+
# LANGSMITH_PROJECT=comet-skill-eval
|
|
94
|
+
# The Claude Code plugin variables (TRACE_TO_LANGSMITH and CC_LANGSMITH_*) are
|
|
95
|
+
# derived from the LANGSMITH_* settings by eval/langsmith/tests/conftest.py.
|
|
96
|
+
# TRACE_TO_LANGSMITH=false
|
|
97
|
+
|
|
98
|
+
# Optional: name the LangSmith experiment (project) per treatment so CONTROL vs
|
|
99
|
+
# skill-injected runs compare in the experiment view.
|
|
100
|
+
# LANGSMITH_EXPERIMENT=COMET_FULL_040_BETA
|
|
101
|
+
|
|
102
|
+
# Optional trajectory tracing via the official langsmith-tracing Claude Code
|
|
103
|
+
# plugin. When LANGSMITH_TRACING=true, the LangSmith suite auto-builds the
|
|
104
|
+
# plugin into eval/.cache/langsmith-cc-plugin via node:20 and reuses that cache.
|
|
105
|
+
# Do not mount a Windows-host-built node_modules directory into the Linux eval container.
|
|
106
|
+
# Unset = rubric + comparison still logged, just without the nested trajectory.
|
|
107
|
+
# CC_LANGSMITH_PLUGIN_DIR=/abs/path/to/comet/eval/.cache/langsmith-cc-plugin
|
|
108
|
+
# CC_LANGSMITH_PLUGIN_AUTO_BUILD=true
|
|
109
|
+
|
|
110
|
+
# Required only for `comet eval --suite langfuse` run mode. Credentials stay in
|
|
111
|
+
# the process environment and are never written to manifests or reports.
|
|
112
|
+
# LANGFUSE_PUBLIC_KEY=
|
|
113
|
+
# LANGFUSE_SECRET_KEY=
|
|
114
|
+
# LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
|
115
|
+
# LANGFUSE_TRACING_ENVIRONMENT=ci
|
package/eval/README.md
CHANGED
|
@@ -23,16 +23,24 @@ eval/
|
|
|
23
23
|
│ ├── treatments/ # Treatment 配置(注入哪些 Skill)
|
|
24
24
|
│ ├── tests/ # pytest 测试入口
|
|
25
25
|
│ └── logs/ # 评估结果日志
|
|
26
|
-
|
|
26
|
+
├── langsmith/ # LangSmith 评估入口(可选)
|
|
27
|
+
└── langfuse/ # Langfuse 评估入口(可选)
|
|
27
28
|
```
|
|
28
29
|
|
|
29
30
|
## 环境配置
|
|
30
31
|
|
|
32
|
+
普通 npm 用户不需要进入 Comet 源码目录。首次运行 `comet eval` 时,CLI 会在用户目录自动创建完整配置模板:
|
|
33
|
+
|
|
34
|
+
- Windows:`%USERPROFILE%\.comet\eval\.env`
|
|
35
|
+
- macOS/Linux:`~/.comet/eval/.env`
|
|
36
|
+
|
|
37
|
+
模板包含主任务(Bench)、独立 Judge、Claude Code、Codex、Qoder、CodeBuddy、LangSmith 和 Langfuse 的全部可配置参数。CLI 只会创建缺失文件,不会覆盖已有 `.env`;当前进程环境变量优先于文件内容。源码仓库下的 `eval/.env` 仅用于 Comet harness 开发和测试,不是普通用户配置入口。
|
|
38
|
+
|
|
31
39
|
### 前置依赖
|
|
32
40
|
|
|
33
41
|
- Python 3.11+
|
|
34
42
|
- Docker
|
|
35
|
-
- Claude Code CLI
|
|
43
|
+
- 一个受支持的评估 Agent CLI:Claude Code、Codex、Qoder 或 CodeBuddy(运行时会在 Docker 镜像中准备对应 CLI)
|
|
36
44
|
- `uv`(推荐)或 pip
|
|
37
45
|
|
|
38
46
|
### 安装
|
|
@@ -44,20 +52,32 @@ uv sync
|
|
|
44
52
|
|
|
45
53
|
### 环境变量
|
|
46
54
|
|
|
47
|
-
|
|
55
|
+
已发布的 `comet eval` 不需要用户拉取源码。评估运行会先读取当前 shell 的环境变量,再按需加载用户目录下、与自定义 Agent 适配器并列的 `~/.comet/eval/.env`(Windows:`%USERPROFILE%\.comet\eval\.env`)。录屏或演示时可以不创建 `.env`,把密钥放在系统环境变量里。
|
|
48
56
|
|
|
49
|
-
|
|
57
|
+
如需本地文件配置,可参考 `.env.example`,创建用户级文件:
|
|
50
58
|
|
|
51
59
|
```bash
|
|
52
|
-
|
|
60
|
+
mkdir -p ~/.comet/eval
|
|
61
|
+
cp .env.example ~/.comet/eval/.env
|
|
53
62
|
```
|
|
54
63
|
|
|
55
|
-
|
|
64
|
+
进程环境变量优先于用户 `.env`;显式 CLI 或 manifest 配置优先于环境默认值。空值不会覆盖已有的有效进程变量。
|
|
56
65
|
|
|
57
66
|
| 变量 | 必填 | 说明 |
|
|
58
67
|
| ----------------------------------------------------------------- | ---- | --------------------------------------------------------------------------------------------------- |
|
|
59
|
-
| `
|
|
60
|
-
| `
|
|
68
|
+
| `BENCH_EVAL_AGENT` | ❌ | 未通过 CLI 或 manifest 指定时的默认 Agent |
|
|
69
|
+
| `BENCH_API_KEY` / `BENCH_BASE_URL` / `BENCH_MODEL` | ✅\* | 主任务统一凭据、API 地址和模型;会映射到所选 Agent 的原生配置 |
|
|
70
|
+
| `BENCH_JUDGE_AGENT` / `BENCH_JUDGE_API_KEY` / `BENCH_JUDGE_BASE_URL` / `BENCH_JUDGE_MODEL` | Judge | LLM-as-judge 的独立 Agent、凭据、API 地址和模型;不会继承主任务配置 |
|
|
71
|
+
| `ANTHROPIC_API_KEY` | ✅\* | Claude Code 原生 API 密钥;显式设置时优先于 `BENCH_API_KEY` |
|
|
72
|
+
| `BENCH_CC_MODEL` / `BENCH_CC_VERSION` | ❌ | Claude 模型和 CLI 版本覆盖(默认用 CLI 配置) |
|
|
73
|
+
| `OPENAI_API_KEY` / `CODEX_API_KEY` | Codex | 使用 `--agent codex` 时的 Codex 认证 |
|
|
74
|
+
| `OPENAI_BASE_URL` / `OPENAI_MODEL` / `CODEX_BASE_URL` / `CODEX_MODEL` / `BENCH_CODEX_VERSION` | ❌ | Codex 原生地址、模型和 CLI 版本覆盖 |
|
|
75
|
+
| `QODER_PERSONAL_ACCESS_TOKEN` | Qoder | 使用 `--agent qoder` 时的 Qoder 认证 |
|
|
76
|
+
| `QODER_BASE_URL` / `QODER_MODEL` / `BENCH_QODER_VERSION` | ❌ | Qoder 原生地址、模型和 CLI 版本覆盖 |
|
|
77
|
+
| `CODEBUDDY_API_KEY` / `CODEBUDDY_AUTH_TOKEN` | CodeBuddy | 使用 `--agent codebuddy` 时的 CodeBuddy 认证 |
|
|
78
|
+
| `CODEBUDDY_BASE_URL` / `CODEBUDDY_MODEL` / `BENCH_CODEBUDDY_VERSION` | ❌ | CodeBuddy API endpoint、模型和 CLI 版本覆盖 |
|
|
79
|
+
| `CODEBUDDY_SMALL_FAST_MODEL` / `CODEBUDDY_BIG_SLOW_MODEL` / `CODEBUDDY_CODE_SUBAGENT_MODEL` | ❌ | CodeBuddy 的 fast、reasoning 和 code-subagent 模型覆盖 |
|
|
80
|
+
| `CODEBUDDY_CUSTOM_HEADERS` / `CODEBUDDY_INTERNET_ENVIRONMENT` | ❌ | CodeBuddy 的自定义请求头和联网环境配置 |
|
|
61
81
|
| `BENCH_SIMULATOR_PROMPT_FILE` | ❌ | 显式覆盖任务级用户模拟器提示词;任务未配置时回退到 `eval/simulator-instruction.md` |
|
|
62
82
|
| `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_BASE_URL` / `ANTHROPIC_MODEL` | ❌ | 用 Anthropic 兼容代理时的认证与模型配置;设置 `ANTHROPIC_AUTH_TOKEN` 后可不设置 `ANTHROPIC_API_KEY` |
|
|
63
83
|
| `BENCH_LLM_JUDGE` | ❌ | 设为 `1` 启用 LLM-as-judge 评分 |
|
|
@@ -66,8 +86,50 @@ cp .env.example .env
|
|
|
66
86
|
| `BENCH_JUDGE_API_KEY` | ❌ | Judge 专用 Anthropic API key;如果设置 `BENCH_JUDGE_AUTH_TOKEN`,优先使用 auth token |
|
|
67
87
|
| `LANGSMITH_API_KEY` | ❌ | 仅 LangSmith 套件需要 |
|
|
68
88
|
| `LANGSMITH_PROJECT` / `LANGSMITH_TRACING` | ❌ | LangSmith 套件项目名与追踪开关;Claude Code 插件变量会从这组配置自动派生 |
|
|
89
|
+
| `TRACE_TO_LANGSMITH` | ❌ | 兼容旧配置的 LangSmith 追踪开关 |
|
|
90
|
+
| `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` | Langfuse | `--suite langfuse` 运行模式的项目凭据;只从当前进程读取,不写入报告或 manifest |
|
|
91
|
+
| `LANGFUSE_BASE_URL` / `LANGFUSE_TRACING_ENVIRONMENT` | ❌ | Langfuse 区域地址与 trace 环境标签 |
|
|
92
|
+
| `COMET_EVAL_ADAPTERS_DIR` | ❌ | 覆盖自定义 Agent 适配器注册目录;默认是用户目录下的 `~/.comet/eval/adapters` |
|
|
93
|
+
|
|
94
|
+
`*` 主任务至少需要 `BENCH_API_KEY` 或所选 Agent 的原生凭据之一。Claude Code、Codex 和 CodeBuddy 支持 `BENCH_BASE_URL`;Qoder 使用其官方支持的认证和服务地址配置。
|
|
95
|
+
|
|
96
|
+
安全边界:Eval 不会把 API key 写入发布包、manifest、报告或 Skill 工作区。Docker 运行 Agent 时,Codex、Qoder、CodeBuddy 使用容器内独立的配置根;Codex 的 `config.toml` 只引用 `env_key`,CodeBuddy 的 `settings.json` 只使用 `apiKeyHelper`,实际密钥只存在于本次容器进程环境中,运行结束后随临时配置目录销毁。
|
|
97
|
+
|
|
98
|
+
`comet eval` 默认使用 `claude-code`。可以通过 CLI 或 manifest 选择执行 Agent;CLI 优先于
|
|
99
|
+
manifest,未指定时回退到 Claude Code:
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
comet eval ./generated-skill/comet/eval.yaml --agent codex
|
|
103
|
+
comet eval ./generated-skill/comet/eval.yaml --agent qoder
|
|
104
|
+
comet eval ./generated-skill/comet/eval.yaml --agent codebuddy
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
CodeBuddy 的自定义模型使用其 OpenAI-compatible 配置:`CODEBUDDY_API_KEY`、
|
|
108
|
+
`CODEBUDDY_BASE_URL` 和 provider 接受的 `CODEBUDDY_MODEL`,也可以直接用 `--model`
|
|
109
|
+
覆盖当前运行的模型。宿主机的 `~/.codebuddy/models.json` 只用于注册模型;Eval 不挂载它,
|
|
110
|
+
而是在容器内生成临时 `settings.json`,通过 `apiKeyHelper` 从进程环境读取凭据。不要把
|
|
111
|
+
Claude Code 的 Anthropic 地址或 Claude 专用模型 ID 直接复用给 CodeBuddy。
|
|
112
|
+
|
|
113
|
+
manifest 可声明默认 Agent:
|
|
114
|
+
|
|
115
|
+
```yaml
|
|
116
|
+
execution:
|
|
117
|
+
agent: codex
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
四种 Agent 共用 subject、auto_user simulator 和可选 Judge 的执行/报告契约;缺失的 token、成本或
|
|
121
|
+
上下文遥测会在报告中保留为 `N/A`,而不是用估算值补齐。
|
|
122
|
+
|
|
123
|
+
### Langfuse 套件
|
|
124
|
+
|
|
125
|
+
使用 `--suite langfuse` 时,local runner 的任务、Agent、Docker、validator 和本地报告保持不变,额外把每个 task/treatment 的核心结果写入 Langfuse,并在结束时 flush:
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... \
|
|
129
|
+
comet eval ./generated-skill/comet/eval.yaml --suite langfuse --html
|
|
130
|
+
```
|
|
69
131
|
|
|
70
|
-
|
|
132
|
+
`comet eval` 会自动选择 Langfuse optional extra。Claude Code 和 Codex 会在隔离的 `.comet/eval/langfuse/plugins/` 缓存中准备带完整性元数据的固定版本官方插件;Qoder 和 CodeBuddy 使用项目级 Stop hook 读取 JSONL transcript,转换失败时只降级详细轨迹,不影响本地评分和核心 trace。`--collect --suite langfuse` 不初始化 SDK、不联网、不下载插件。
|
|
71
133
|
|
|
72
134
|
## 快速开始
|
|
73
135
|
|
|
@@ -94,6 +156,132 @@ Eval 报告同时保留 raw set 和 analysis set。Raw set 包含所有 run,
|
|
|
94
156
|
|
|
95
157
|
如果关键 treatment 的 clean data 不足,报告会输出 `Insufficient clean data` 或 `Inconclusive due to data quality`,而不是给出误导性的胜负结论。
|
|
96
158
|
|
|
159
|
+
### 独立评估任意 Skill
|
|
160
|
+
|
|
161
|
+
独立评估不依赖 `comet-any`。只要有一个 Skill 目录,就可以直接运行;没有
|
|
162
|
+
`comet/eval.yaml` 时,普通运行会自动生成并缓存 2–4 个任务:
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
comet eval ./my-skill
|
|
166
|
+
comet eval ./my-skill --agent codebuddy --model subject-model
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
主模型和 LLM-as-judge 是两套独立配置。Judge 只有在显式启用后才会运行,至少需要自己的
|
|
170
|
+
`BENCH_JUDGE_MODEL` 和 `BENCH_JUDGE_API_KEY`/`BENCH_JUDGE_AUTH_TOKEN`;它可以使用不同的
|
|
171
|
+
Agent、模型和 API 地址:
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
BENCH_LLM_JUDGE=1 \
|
|
175
|
+
BENCH_JUDGE_MODEL=judge-model \
|
|
176
|
+
BENCH_JUDGE_BASE_URL=https://judge.example/v1 \
|
|
177
|
+
BENCH_JUDGE_API_KEY=... \
|
|
178
|
+
comet eval ./my-skill --model subject-model --base-url https://subject.example/v1
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
也可以全部通过命令行或 manifest 配置:
|
|
182
|
+
|
|
183
|
+
```bash
|
|
184
|
+
comet eval ./my-skill \
|
|
185
|
+
--agent codebuddy --model subject-model --base-url https://subject.example/v1 \
|
|
186
|
+
--judge-agent claude-code --judge-model judge-model \
|
|
187
|
+
--judge-base-url https://judge.example/v1
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
如果只想检查任务、配置和快照,不执行 Agent、Docker、插件或网络请求,使用:
|
|
191
|
+
|
|
192
|
+
```bash
|
|
193
|
+
comet eval ./my-skill --collect
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
### 扩展自定义 Agent
|
|
197
|
+
|
|
198
|
+
非预定义的 Agent 不需要修改 Comet 源码,也不需要把适配器放进 npm 包。用户在自己的配置目录
|
|
199
|
+
中显式注册即可,Eval 不会仅因为某个可执行文件在 `PATH` 中就自动发现它:
|
|
200
|
+
|
|
201
|
+
| 系统 | 默认适配器目录 |
|
|
202
|
+
| --- | --- |
|
|
203
|
+
| macOS/Linux | `~/.comet/eval/adapters/<agent-id>/adapter.yaml` |
|
|
204
|
+
| Windows | `%USERPROFILE%\\.comet\\eval\\adapters\\<agent-id>\\adapter.yaml` |
|
|
205
|
+
|
|
206
|
+
也可以在用户级 `~/.comet/eval/.env`(Windows 为 `%USERPROFILE%\\.comet\\eval\\.env`)中设置
|
|
207
|
+
`COMET_EVAL_ADAPTERS_DIR`,把整个适配器注册表放到其他位置。`<agent-id>` 必须是小写字母开头、
|
|
208
|
+
只包含小写字母/数字/连字符、长度 2–32 的标识符。
|
|
209
|
+
|
|
210
|
+
创建 `<agent-id>/adapter.yaml`,例如:
|
|
211
|
+
|
|
212
|
+
```yaml
|
|
213
|
+
apiVersion: comet.eval.agent/v1alpha1
|
|
214
|
+
kind: EvalAgentAdapter
|
|
215
|
+
metadata: { id: my-agent, version: 1.0.0 }
|
|
216
|
+
runtime:
|
|
217
|
+
executable: my-agent
|
|
218
|
+
install: { kind: npm, package: my-agent, version: latest }
|
|
219
|
+
credentials: [MY_AGENT_API_KEY]
|
|
220
|
+
modelEnv: MY_AGENT_MODEL
|
|
221
|
+
baseUrlEnv: MY_AGENT_BASE_URL
|
|
222
|
+
capabilities:
|
|
223
|
+
singleTurn: true
|
|
224
|
+
resume: true
|
|
225
|
+
structuredEvents: true
|
|
226
|
+
telemetry: false
|
|
227
|
+
skillInvocationEvidence: true
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
字段含义:
|
|
231
|
+
|
|
232
|
+
- `runtime.executable` 是容器内实际启动的命令;npm/pip 包必须暴露这个可执行入口。
|
|
233
|
+
`runtime.install.kind` 可为 `npm`、`pip` 或 `none`;使用 `npm`/`pip` 时 Eval 会在本次 Docker
|
|
234
|
+
镜像构建中安装对应包,`none` 表示命令已在基础镜像中准备好。
|
|
235
|
+
- `credentials` 只能填写最多两个环境变量名,不要填写密钥值。主 Agent 运行时,只有这些声明的
|
|
236
|
+
环境变量会被转发到容器;Judge 运行时,`BENCH_JUDGE_API_KEY` 映射到第一个凭据名,
|
|
237
|
+
`BENCH_JUDGE_AUTH_TOKEN` 映射到第二个凭据名(如果声明了第二个)。
|
|
238
|
+
- `modelEnv` 和 `baseUrlEnv` 是可选的环境变量名。主 Agent 可通过这些变量,或 CLI 的
|
|
239
|
+
`--model`/`--base-url`,或 manifest 的 `execution.model`/`execution.baseUrl` 配置;自定义 Agent
|
|
240
|
+
不会自动把 `BENCH_MODEL`、`BENCH_BASE_URL` 或 `BENCH_API_KEY` 映射到自定义变量。
|
|
241
|
+
- `capabilities` 描述适配器的真实能力。`resume` 用于 `auto_user` 多轮交互;`structuredEvents`
|
|
242
|
+
要求 CLI 输出 JSON Lines;`skillInvocationEvidence` 表示输出中能提供明确的 Skill 调用事件;
|
|
243
|
+
`telemetry: false` 时报告中的 token/cost 可以是 `N/A`。运行真实评估至少需要
|
|
244
|
+
`singleTurn`、`resume`、`structuredEvents` 和 `skillInvocationEvidence` 为 `true`。
|
|
245
|
+
|
|
246
|
+
自定义 CLI 必须接受 Eval 生成的调用形态:
|
|
247
|
+
|
|
248
|
+
```text
|
|
249
|
+
<executable> -p "<prompt>" --output-format stream-json --model <model> [--resume <session-id>]
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
并把每条结构化事件写到 stdout。若要让 rubric 认定 Skill 被调用,输出中需要有类似下面的 JSONL
|
|
253
|
+
事件(普通文件产物或路径推断不算自定义 Agent 的调用证据):
|
|
254
|
+
|
|
255
|
+
```json
|
|
256
|
+
{"type":"skill_invocation","skill":"my-skill"}
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
在用户级 `.env` 中填写适配器声明的变量,然后运行:
|
|
260
|
+
|
|
261
|
+
```dotenv
|
|
262
|
+
BENCH_EVAL_AGENT=my-agent
|
|
263
|
+
MY_AGENT_API_KEY=your-subject-key
|
|
264
|
+
MY_AGENT_MODEL=your-model
|
|
265
|
+
MY_AGENT_BASE_URL=https://your-provider.example/v1
|
|
266
|
+
|
|
267
|
+
# 如果让它作为独立 Judge:
|
|
268
|
+
# BENCH_LLM_JUDGE=1
|
|
269
|
+
# BENCH_JUDGE_AGENT=my-agent
|
|
270
|
+
# BENCH_JUDGE_API_KEY=your-judge-key
|
|
271
|
+
# BENCH_JUDGE_MODEL=your-judge-model
|
|
272
|
+
# BENCH_JUDGE_BASE_URL=https://your-judge-provider.example/v1
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
```bash
|
|
276
|
+
comet eval ./my-skill --collect --agent my-agent
|
|
277
|
+
comet eval ./my-skill --quick --agent my-agent --model your-model
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
`--collect` 会校验适配器和配置但不会启动 Agent、Docker 或联网;确认通过后再运行真实评估。
|
|
281
|
+
如果你的 CLI 不能兼容上述参数和 JSONL 事件契约,需要提供一个 wrapper CLI,将它转换为该契约;
|
|
282
|
+
当前 `adapter.yaml` 不支持自定义参数模板或直接挂载宿主机 Agent 配置目录。适配器文件可以分发给
|
|
283
|
+
团队成员,但真实凭据必须留在每个人自己的用户级 `.env` 或当前 shell 中。
|
|
284
|
+
|
|
97
285
|
### 评估任意 Skill
|
|
98
286
|
|
|
99
287
|
```bash
|
|
@@ -111,23 +299,29 @@ uv run pytest local/tests/tasks/test_tasks.py \
|
|
|
111
299
|
--eval-manifest=/path/to/my-skill/comet/eval.yaml -v
|
|
112
300
|
```
|
|
113
301
|
|
|
114
|
-
|
|
302
|
+
如果 manifest 没有 `evaluation.tasks` 或 `recommendedTasks`,普通运行会基于 Skill 的受限快照自动生成 2–4 个任务,并按 Agent、profile、交互配置和快照 hash 缓存到项目的 `.comet/eval/generated/`。缓存 manifest 会写入生成 hash 和元数据,方便复现;`--collect` 只接受已有缓存,不会在收集阶段调用生成 Agent。需要快速冒烟时显式使用:
|
|
115
303
|
|
|
116
|
-
|
|
304
|
+
```bash
|
|
305
|
+
comet eval ./my-skill --quick
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
## 容器中的 Agent 实验工作区
|
|
309
|
+
|
|
310
|
+
每次实验都会先在宿主机创建一个临时隔离目录,然后把它挂载到所选 Agent 的 Docker 容器的 `/workspace`。Agent 的当前工作目录就是 `/workspace`;双 agent 交互模式还会只读挂载 scaffold shell 到 `/opt/scaffold-shell`。
|
|
117
311
|
|
|
118
312
|
典型结构如下(以 `COMET_FULL_040_BETA` 为例,展示 0.4.0 beta 的 Comet Skill + 完整 dependency snapshot;实际任务产物会随 task/treatment 变化):
|
|
119
313
|
|
|
120
314
|
```text
|
|
121
|
-
/workspace #
|
|
315
|
+
/workspace # 所选 Agent 的当前工作目录;宿主机临时隔离目录挂载点
|
|
122
316
|
|-- Dockerfile # 当前 task 的 Docker 构建文件,来自 task/environment/
|
|
123
317
|
|-- requirements.txt # Python 依赖文件;仅 task 提供时存在
|
|
124
318
|
|-- package.json # Node 依赖文件;仅 task 提供时存在
|
|
125
319
|
|-- package-lock.json # Node 锁文件;仅 task 提供时存在
|
|
126
320
|
|-- tsconfig.json # TypeScript 配置;仅 task 提供时存在
|
|
127
|
-
|-- CLAUDE.md #
|
|
321
|
+
|-- CLAUDE.md # Claude 项目级指令;其他 Agent 使用 AGENTS.md
|
|
128
322
|
|-- .eval-task-prompt.txt # auto_user 模式下的任务提示词临时文件,运行后清理
|
|
129
323
|
|-- .eval-simulator-prompt.txt # auto_user 模式下的用户模拟器提示词临时文件,运行后清理
|
|
130
|
-
|-- .claude
|
|
324
|
+
|-- .claude / .agents / .qoder / .codebuddy # 所选 Agent 的本地配置和 Skill 安装目录
|
|
131
325
|
| |-- CLAUDE.md # Claude 专用项目指令副本
|
|
132
326
|
| |-- settings.json # Claude Code settings;包含 Comet PreToolUse hook
|
|
133
327
|
| `-- skills # 当前 treatment 注入的完整 Skill 包
|
|
@@ -229,6 +423,28 @@ uv run pytest local/tests/tasks/test_tasks.py \
|
|
|
229
423
|
|
|
230
424
|
## 定义自己的 Task
|
|
231
425
|
|
|
426
|
+
除了把任务注册到 `eval/local/tasks/`,项目也可以在自己的 `comet/eval.yaml` 中声明 inline task 或引用 Skill 包内的 task package。项目任务会和 `recommendedTasks` 合并;同名任务会直接拒绝,避免覆盖内置任务。
|
|
427
|
+
|
|
428
|
+
```yaml
|
|
429
|
+
evaluation:
|
|
430
|
+
tasks:
|
|
431
|
+
- name: writes-summary
|
|
432
|
+
prompt: Create summary.md with a short result.
|
|
433
|
+
workspace: fixtures/summary
|
|
434
|
+
expect:
|
|
435
|
+
files: [summary.md]
|
|
436
|
+
contains:
|
|
437
|
+
summary.md: ["# Summary"]
|
|
438
|
+
commands:
|
|
439
|
+
- run: test -s summary.md
|
|
440
|
+
timeout: 30
|
|
441
|
+
rubric:
|
|
442
|
+
- The summary is specific and actionable.
|
|
443
|
+
- source: tasks/advanced
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
`workspace`、`source` 和所有期望产物都必须留在 Skill 包/任务工作区内;`files`、`contains`、`json` 和 `commands` 检查在隔离 Docker 工作区中执行。`--collect` 只做解析和发现,不运行 Agent 或用户命令。
|
|
447
|
+
|
|
232
448
|
每个任务是 `local/tasks/` 下的一个目录,包含三个部分:
|
|
233
449
|
|
|
234
450
|
```text
|
|
@@ -531,6 +747,13 @@ uv run pytest local/tests/tasks/test_tasks.py --count=5 -v
|
|
|
531
747
|
| `--task=<name>` | 指定任务 |
|
|
532
748
|
| `--treatment=<name>` | 指定 treatment(逗号分隔多个) |
|
|
533
749
|
| `--profile=<name>` | 覆盖评估 profile |
|
|
750
|
+
| `--agent=<id>` | 主 Agent;可用内置 Agent 或已安装的自定义适配器 |
|
|
751
|
+
| `--model=<name>` | 主模型覆盖 |
|
|
752
|
+
| `--base-url=<url>` | 主模型 API 地址覆盖 |
|
|
753
|
+
| `--judge-agent=<id>` | 独立 Judge Agent |
|
|
754
|
+
| `--judge-model=<name>` | 独立 Judge 模型 |
|
|
755
|
+
| `--judge-base-url=<url>` | 独立 Judge API 地址 |
|
|
756
|
+
| `--collect` | 只解析/发现,不执行工作负载 |
|
|
534
757
|
| `--skill-path=<path>` | 注入本地 Skill 路径 |
|
|
535
758
|
| `--skill-name=<name>` | 注入的 Skill 名称 |
|
|
536
759
|
| `--count=<n>` | 重复运行次数 |
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# Required for `comet eval --suite langfuse` run mode.
|
|
2
|
+
LANGFUSE_PUBLIC_KEY=
|
|
3
|
+
LANGFUSE_SECRET_KEY=
|
|
4
|
+
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
|
5
|
+
LANGFUSE_TRACING_ENVIRONMENT=ci
|
|
6
|
+
|
|
7
|
+
# Claude Code and Codex official integrations are enabled by the suite through
|
|
8
|
+
# TRACE_TO_LANGFUSE=true. Install their official plugins in the agent image or
|
|
9
|
+
# project configuration before a run; collect mode never installs or contacts them.
|