@rpamis/comet 0.4.0-beta.3 → 0.4.0-beta.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -0
- package/assets/manifest.json +2 -1
- package/assets/skills/comet/SKILL.md +25 -1
- package/assets/skills/comet/reference/comet-yaml-fields.md +5 -2
- package/assets/skills/comet/reference/context-recovery.md +10 -0
- package/assets/skills/comet/reference/scripts.md +47 -15
- package/assets/skills/comet/rules/comet-phase-guard.en.md +8 -0
- package/assets/skills/comet/rules/comet-phase-guard.md +9 -1
- package/assets/skills/comet/scripts/comet-resume-probe.mjs +3 -0
- package/assets/skills/comet/scripts/comet-runtime.mjs +1387 -424
- package/assets/skills/comet-archive/SKILL.md +24 -11
- package/assets/skills/comet-build/SKILL.md +42 -25
- package/assets/skills/comet-design/SKILL.md +1 -0
- package/assets/skills/comet-hotfix/SKILL.md +3 -1
- package/assets/skills/comet-open/SKILL.md +11 -1
- package/assets/skills/comet-tweak/SKILL.md +3 -1
- package/assets/skills/comet-verify/SKILL.md +29 -20
- package/assets/skills-zh/comet/SKILL.md +25 -1
- package/assets/skills-zh/comet/reference/comet-yaml-fields.md +5 -2
- package/assets/skills-zh/comet/reference/context-recovery.md +10 -0
- package/assets/skills-zh/comet/reference/scripts.md +47 -15
- package/assets/skills-zh/comet-archive/SKILL.md +24 -11
- package/assets/skills-zh/comet-build/SKILL.md +42 -25
- package/assets/skills-zh/comet-design/SKILL.md +1 -0
- package/assets/skills-zh/comet-hotfix/SKILL.md +3 -1
- package/assets/skills-zh/comet-open/SKILL.md +11 -1
- package/assets/skills-zh/comet-tweak/SKILL.md +3 -1
- package/assets/skills-zh/comet-verify/SKILL.md +30 -21
- package/dist/app/cli/comet-banner.d.ts +25 -0
- package/dist/app/cli/comet-banner.d.ts.map +1 -0
- package/dist/app/cli/comet-banner.js +265 -0
- package/dist/app/cli/comet-banner.js.map +1 -0
- package/dist/app/cli/index.js +36 -2
- package/dist/app/cli/index.js.map +1 -1
- package/dist/app/commands/classic.d.ts +4 -0
- package/dist/app/commands/classic.d.ts.map +1 -0
- package/dist/app/commands/classic.js +11 -0
- package/dist/app/commands/classic.js.map +1 -0
- package/dist/app/commands/doctor.d.ts.map +1 -1
- package/dist/app/commands/doctor.js +39 -24
- package/dist/app/commands/doctor.js.map +1 -1
- package/dist/app/commands/eval.d.ts.map +1 -1
- package/dist/app/commands/eval.js +57 -20
- package/dist/app/commands/eval.js.map +1 -1
- package/dist/app/commands/i18n.d.ts +1 -1
- package/dist/app/commands/i18n.d.ts.map +1 -1
- package/dist/app/commands/i18n.js +12 -4
- package/dist/app/commands/i18n.js.map +1 -1
- package/dist/app/commands/init.d.ts.map +1 -1
- package/dist/app/commands/init.js +16 -13
- package/dist/app/commands/init.js.map +1 -1
- package/dist/app/commands/project-scope-selection.d.ts +13 -0
- package/dist/app/commands/project-scope-selection.d.ts.map +1 -0
- package/dist/app/commands/project-scope-selection.js +33 -0
- package/dist/app/commands/project-scope-selection.js.map +1 -0
- package/dist/app/commands/resume-probe.d.ts +11 -0
- package/dist/app/commands/resume-probe.d.ts.map +1 -0
- package/dist/app/commands/resume-probe.js +63 -0
- package/dist/app/commands/resume-probe.js.map +1 -0
- package/dist/app/commands/status.d.ts +31 -0
- package/dist/app/commands/status.d.ts.map +1 -1
- package/dist/app/commands/status.js +109 -20
- package/dist/app/commands/status.js.map +1 -1
- package/dist/app/commands/uninstall.d.ts +2 -0
- package/dist/app/commands/uninstall.d.ts.map +1 -1
- package/dist/app/commands/uninstall.js +217 -40
- package/dist/app/commands/uninstall.js.map +1 -1
- package/dist/app/commands/update.d.ts +6 -0
- package/dist/app/commands/update.d.ts.map +1 -1
- package/dist/app/commands/update.js +361 -68
- package/dist/app/commands/update.js.map +1 -1
- package/dist/config/repository-layout.json +4 -2
- package/dist/domains/bundle/bundle-platform.d.ts.map +1 -1
- package/dist/domains/bundle/bundle-platform.js +3 -1
- package/dist/domains/bundle/bundle-platform.js.map +1 -1
- package/dist/domains/bundle/eval-manifest-runtime.d.ts +6 -0
- package/dist/domains/bundle/eval-manifest-runtime.d.ts.map +1 -0
- package/dist/domains/bundle/eval-manifest-runtime.js +58 -0
- package/dist/domains/bundle/eval-manifest-runtime.js.map +1 -0
- package/dist/domains/bundle/types.d.ts +1 -1
- package/dist/domains/bundle/types.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-archive.d.ts +1 -0
- package/dist/domains/comet-classic/classic-archive.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-archive.js +35 -38
- package/dist/domains/comet-classic/classic-archive.js.map +1 -1
- package/dist/domains/comet-classic/classic-cli.d.ts +1 -1
- package/dist/domains/comet-classic/classic-cli.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-cli.js +3 -0
- package/dist/domains/comet-classic/classic-cli.js.map +1 -1
- package/dist/domains/comet-classic/classic-command-checks.d.ts +20 -0
- package/dist/domains/comet-classic/classic-command-checks.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-command-checks.js +97 -0
- package/dist/domains/comet-classic/classic-command-checks.js.map +1 -0
- package/dist/domains/comet-classic/classic-current-change.d.ts +19 -0
- package/dist/domains/comet-classic/classic-current-change.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-current-change.js +132 -0
- package/dist/domains/comet-classic/classic-current-change.js.map +1 -0
- package/dist/domains/comet-classic/classic-diagnostics.d.ts +2 -1
- package/dist/domains/comet-classic/classic-diagnostics.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-diagnostics.js +54 -1
- package/dist/domains/comet-classic/classic-diagnostics.js.map +1 -1
- package/dist/domains/comet-classic/classic-evidence.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-evidence.js +4 -0
- package/dist/domains/comet-classic/classic-evidence.js.map +1 -1
- package/dist/domains/comet-classic/classic-guard.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-guard.js +68 -69
- package/dist/domains/comet-classic/classic-guard.js.map +1 -1
- package/dist/domains/comet-classic/classic-hook-guard.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-hook-guard.js +66 -23
- package/dist/domains/comet-classic/classic-hook-guard.js.map +1 -1
- package/dist/domains/comet-classic/classic-project-config.d.ts +16 -0
- package/dist/domains/comet-classic/classic-project-config.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-project-config.js +40 -0
- package/dist/domains/comet-classic/classic-project-config.js.map +1 -0
- package/dist/domains/comet-classic/classic-resolver.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-resolver.js +3 -3
- package/dist/domains/comet-classic/classic-resolver.js.map +1 -1
- package/dist/domains/comet-classic/classic-resume-probe-command.d.ts +3 -0
- package/dist/domains/comet-classic/classic-resume-probe-command.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-resume-probe-command.js +58 -0
- package/dist/domains/comet-classic/classic-resume-probe-command.js.map +1 -0
- package/dist/domains/comet-classic/classic-resume-probe-entry.d.ts +2 -0
- package/dist/domains/comet-classic/classic-resume-probe-entry.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-resume-probe-entry.js +4 -0
- package/dist/domains/comet-classic/classic-resume-probe-entry.js.map +1 -0
- package/dist/domains/comet-classic/classic-resume-probe.d.ts +29 -0
- package/dist/domains/comet-classic/classic-resume-probe.d.ts.map +1 -0
- package/dist/domains/comet-classic/classic-resume-probe.js +386 -0
- package/dist/domains/comet-classic/classic-resume-probe.js.map +1 -0
- package/dist/domains/comet-classic/classic-runtime-run.d.ts +2 -1
- package/dist/domains/comet-classic/classic-runtime-run.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-runtime-run.js +38 -0
- package/dist/domains/comet-classic/classic-runtime-run.js.map +1 -1
- package/dist/domains/comet-classic/classic-state-command.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-state-command.js +128 -12
- package/dist/domains/comet-classic/classic-state-command.js.map +1 -1
- package/dist/domains/comet-classic/classic-state.d.ts +3 -1
- package/dist/domains/comet-classic/classic-state.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-state.js +4 -0
- package/dist/domains/comet-classic/classic-state.js.map +1 -1
- package/dist/domains/comet-classic/classic-store.d.ts +4 -1
- package/dist/domains/comet-classic/classic-store.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-store.js +4 -3
- package/dist/domains/comet-classic/classic-store.js.map +1 -1
- package/dist/domains/comet-classic/classic-transitions.d.ts +1 -1
- package/dist/domains/comet-classic/classic-transitions.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-transitions.js +20 -1
- package/dist/domains/comet-classic/classic-transitions.js.map +1 -1
- package/dist/domains/comet-classic/classic-validate-command.d.ts.map +1 -1
- package/dist/domains/comet-classic/classic-validate-command.js +1 -0
- package/dist/domains/comet-classic/classic-validate-command.js.map +1 -1
- package/dist/domains/comet-classic/index.d.ts +3 -0
- package/dist/domains/comet-classic/index.d.ts.map +1 -1
- package/dist/domains/comet-classic/index.js +3 -0
- package/dist/domains/comet-classic/index.js.map +1 -1
- package/dist/domains/dashboard/collector.js +10 -2
- package/dist/domains/dashboard/collector.js.map +1 -1
- package/dist/domains/dashboard/web/assets/{index-CjU3VXVu.js → index-CIvN-PL8.js} +11 -11
- package/dist/domains/dashboard/web/assets/{index-BcO3_QlT.css → index-DcyRPBnT.css} +1 -1
- package/dist/domains/dashboard/web/index.html +2 -2
- package/dist/domains/engine/types.d.ts +1 -1
- package/dist/domains/engine/types.d.ts.map +1 -1
- package/dist/domains/integrations/openspec.d.ts.map +1 -1
- package/dist/domains/integrations/openspec.js +6 -6
- package/dist/domains/integrations/openspec.js.map +1 -1
- package/dist/domains/skill/find.d.ts.map +1 -1
- package/dist/domains/skill/find.js +1 -6
- package/dist/domains/skill/find.js.map +1 -1
- package/dist/domains/skill/managed-markdown.d.ts +14 -0
- package/dist/domains/skill/managed-markdown.d.ts.map +1 -0
- package/dist/domains/skill/managed-markdown.js +126 -0
- package/dist/domains/skill/managed-markdown.js.map +1 -0
- package/dist/domains/skill/platform-install.d.ts +4 -3
- package/dist/domains/skill/platform-install.d.ts.map +1 -1
- package/dist/domains/skill/platform-install.js +96 -26
- package/dist/domains/skill/platform-install.js.map +1 -1
- package/dist/domains/skill/project-instructions.d.ts +22 -0
- package/dist/domains/skill/project-instructions.d.ts.map +1 -0
- package/dist/domains/skill/project-instructions.js +61 -0
- package/dist/domains/skill/project-instructions.js.map +1 -0
- package/dist/domains/skill/uninstall.d.ts +3 -1
- package/dist/domains/skill/uninstall.d.ts.map +1 -1
- package/dist/domains/skill/uninstall.js +86 -39
- package/dist/domains/skill/uninstall.js.map +1 -1
- package/dist/platform/install/detect.d.ts +2 -1
- package/dist/platform/install/detect.d.ts.map +1 -1
- package/dist/platform/install/detect.js +10 -1
- package/dist/platform/install/detect.js.map +1 -1
- package/dist/platform/install/platforms.d.ts +5 -1
- package/dist/platform/install/platforms.d.ts.map +1 -1
- package/dist/platform/install/platforms.js +21 -4
- package/dist/platform/install/platforms.js.map +1 -1
- package/dist/platform/install/project-registry.d.ts +37 -0
- package/dist/platform/install/project-registry.d.ts.map +1 -0
- package/dist/platform/install/project-registry.js +205 -0
- package/dist/platform/install/project-registry.js.map +1 -0
- package/eval/.env +28 -0
- package/eval/.env.example +52 -0
- package/eval/CLAUDE.md +43 -0
- package/eval/README.md +538 -0
- package/eval/langsmith/.env.example +16 -0
- package/eval/langsmith/README.md +93 -0
- package/eval/langsmith/skills/README.md +5 -0
- package/eval/langsmith/tasks/README.md +5 -0
- package/eval/langsmith/tests/conftest.py +187 -0
- package/eval/langsmith/tests/tasks/test_tasks.py +287 -0
- package/eval/langsmith/treatments/README.md +5 -0
- package/eval/local/README.md +80 -0
- package/eval/local/regression_baseline.json +13 -0
- package/eval/local/report-style-demo-assets/quality_cost.en.svg +41 -0
- package/eval/local/report-style-demo-assets/quality_cost.zh.svg +41 -0
- package/eval/local/report-style-demo-assets/rubric_delta.en.svg +56 -0
- package/eval/local/report-style-demo-assets/rubric_delta.zh.svg +56 -0
- package/eval/local/report-style-demo-assets/task_outcomes.en.svg +46 -0
- package/eval/local/report-style-demo-assets/task_outcomes.zh.svg +46 -0
- package/eval/local/report-style-demo.html +867 -0
- package/eval/local/scripts/compare_baselines.py +1018 -0
- package/eval/local/scripts/generate_report_style_demo_charts.py +322 -0
- package/eval/local/scripts/regression_check.py +171 -0
- package/eval/local/scripts/render_readme_benchmark_figures.py +416 -0
- package/eval/local/scripts/rescore_rubric.py +120 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/SKILL.md +282 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/auto-transition.md +27 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/comet-yaml-fields.md +68 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/context-recovery.md +35 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/debug-gate.md +17 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/decision-point.md +20 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/dirty-worktree.md +59 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/file-structure.md +28 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/reference/subagent-dispatch.md +113 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/rules/comet-phase-guard.md +125 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/scripts/comet-archive.sh +311 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/scripts/comet-env.sh +110 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/scripts/comet-guard.sh +778 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/scripts/comet-handoff.sh +390 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/scripts/comet-hook-guard.sh +336 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/scripts/comet-state.sh +1338 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039/scripts/comet-yaml-validate.sh +225 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039-archive/SKILL.md +100 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039-build/SKILL.md +304 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039-design/SKILL.md +263 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039-hotfix/SKILL.md +204 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039-open/SKILL.md +220 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039-tweak/SKILL.md +180 -0
- package/eval/local/skills/benchmarks/039-release/comet-classic-039-verify/SKILL.md +234 -0
- package/eval/local/skills/benchmarks/040-beta/comet/SKILL.md +261 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/auto-transition.md +27 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/comet-yaml-fields.md +73 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/context-recovery.md +45 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/debug-gate.md +33 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/decision-point.md +31 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/dirty-worktree.md +59 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/file-structure.md +28 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/intent-frame.md +74 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/scripts.md +69 -0
- package/eval/local/skills/benchmarks/040-beta/comet/reference/subagent-dispatch.md +169 -0
- package/eval/local/skills/benchmarks/040-beta/comet/rules/comet-phase-guard.md +130 -0
- package/eval/local/skills/benchmarks/040-beta/comet/runtime/classic/checks.yaml +6 -0
- package/eval/local/skills/benchmarks/040-beta/comet/runtime/classic/guardrails.yaml +13 -0
- package/eval/local/skills/benchmarks/040-beta/comet/runtime/classic/skill.yaml +122 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-archive.mjs +9515 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-env.mjs +15 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-guard.mjs +10173 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-handoff.mjs +9491 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-hook-guard.mjs +9175 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-intent.mjs +404 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-state.mjs +9273 -0
- package/eval/local/skills/benchmarks/040-beta/comet/scripts/comet-yaml-validate.mjs +7627 -0
- package/eval/local/skills/benchmarks/040-beta/comet-archive/SKILL.md +110 -0
- package/eval/local/skills/benchmarks/040-beta/comet-build/SKILL.md +315 -0
- package/eval/local/skills/benchmarks/040-beta/comet-design/SKILL.md +260 -0
- package/eval/local/skills/benchmarks/040-beta/comet-hotfix/SKILL.md +190 -0
- package/eval/local/skills/benchmarks/040-beta/comet-open/SKILL.md +210 -0
- package/eval/local/skills/benchmarks/040-beta/comet-tweak/SKILL.md +183 -0
- package/eval/local/skills/benchmarks/040-beta/comet-verify/SKILL.md +230 -0
- package/eval/local/skills/benchmarks/dependency/claude-md/comet-workflow/CLAUDE.md +12 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-apply-change/SKILL.md +156 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-archive-change/SKILL.md +114 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-bulk-archive-change/SKILL.md +246 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-continue-change/SKILL.md +118 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-explore/SKILL.md +288 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-ff-change/SKILL.md +101 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-new-change/SKILL.md +74 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-onboard/SKILL.md +554 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-propose/SKILL.md +110 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-sync-specs/SKILL.md +138 -0
- package/eval/local/skills/benchmarks/dependency/openspec/openspec-verify-change/SKILL.md +168 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/SKILL.md +159 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/scripts/frame-template.html +213 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/scripts/helper.js +167 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/scripts/server.cjs +723 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/scripts/start-server.sh +209 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/scripts/stop-server.sh +120 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/spec-document-reviewer-prompt.md +49 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/brainstorming/visual-companion.md +291 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/dispatching-parallel-agents/SKILL.md +185 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/executing-plans/SKILL.md +70 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/finishing-a-development-branch/SKILL.md +241 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/receiving-code-review/SKILL.md +213 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/requesting-code-review/SKILL.md +103 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/requesting-code-review/code-reviewer.md +172 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/subagent-driven-development/SKILL.md +418 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/subagent-driven-development/implementer-prompt.md +139 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/subagent-driven-development/scripts/review-package +44 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/subagent-driven-development/scripts/sdd-workspace +22 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/subagent-driven-development/scripts/task-brief +40 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/subagent-driven-development/task-reviewer-prompt.md +188 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/CREATION-LOG.md +119 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/SKILL.md +296 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/condition-based-waiting-example.ts +158 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/condition-based-waiting.md +115 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/defense-in-depth.md +122 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/find-polluter.sh +63 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/root-cause-tracing.md +169 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/test-academic.md +14 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/test-pressure-1.md +58 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/test-pressure-2.md +68 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/systematic-debugging/test-pressure-3.md +69 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/test-driven-development/SKILL.md +371 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/test-driven-development/testing-anti-patterns.md +299 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/using-git-worktrees/SKILL.md +202 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/verification-before-completion/SKILL.md +139 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-plans/SKILL.md +174 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-plans/plan-document-reviewer-prompt.md +49 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-skills/SKILL.md +689 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-skills/anthropic-best-practices.md +1150 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-skills/examples/CLAUDE_MD_TESTING.md +189 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-skills/graphviz-conventions.dot +172 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-skills/persuasion-principles.md +187 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-skills/render-graphs.js +168 -0
- package/eval/local/skills/benchmarks/dependency/superpowers/writing-skills/testing-skills-with-subagents.md +384 -0
- package/eval/local/tasks/authoring-skill-smoke/environment/Dockerfile +2 -0
- package/eval/local/tasks/authoring-skill-smoke/instruction.md +4 -0
- package/eval/local/tasks/authoring-skill-smoke/task.toml +21 -0
- package/eval/local/tasks/authoring-skill-smoke/validation/test_authoring_skill_smoke.py +54 -0
- package/eval/local/tasks/comet-agent-memory-routing/environment/Dockerfile +28 -0
- package/eval/local/tasks/comet-agent-memory-routing/environment/agent_system.py +40 -0
- package/eval/local/tasks/comet-agent-memory-routing/environment/test_agent_system.py +29 -0
- package/eval/local/tasks/comet-agent-memory-routing/instruction.md +20 -0
- package/eval/local/tasks/comet-agent-memory-routing/task.toml +24 -0
- package/eval/local/tasks/comet-agent-memory-routing/validation/test_agent_memory_routing.py +24 -0
- package/eval/local/tasks/comet-api-cache-ttl/environment/Dockerfile +39 -0
- package/eval/local/tasks/comet-api-cache-ttl/environment/cache.py +39 -0
- package/eval/local/tasks/comet-api-cache-ttl/environment/test_cache.py +81 -0
- package/eval/local/tasks/comet-api-cache-ttl/instruction.md +33 -0
- package/eval/local/tasks/comet-api-cache-ttl/task.toml +24 -0
- package/eval/local/tasks/comet-api-cache-ttl/validation/test_api_cache_ttl.py +58 -0
- package/eval/local/tasks/comet-cross-file-refactor/environment/Dockerfile +15 -0
- package/eval/local/tasks/comet-cross-file-refactor/environment/cli.py +13 -0
- package/eval/local/tasks/comet-cross-file-refactor/environment/counting.py +13 -0
- package/eval/local/tasks/comet-cross-file-refactor/environment/test_textkit.py +21 -0
- package/eval/local/tasks/comet-cross-file-refactor/instruction.md +16 -0
- package/eval/local/tasks/comet-cross-file-refactor/task.toml +20 -0
- package/eval/local/tasks/comet-cross-file-refactor/validation/test_cross_file_refactor.py +20 -0
- package/eval/local/tasks/comet-dependency-confusion/environment/Dockerfile +28 -0
- package/eval/local/tasks/comet-dependency-confusion/environment/app.py +8 -0
- package/eval/local/tasks/comet-dependency-confusion/environment/settings_loader.py +14 -0
- package/eval/local/tasks/comet-dependency-confusion/environment/test_settings_service.py +19 -0
- package/eval/local/tasks/comet-dependency-confusion/instruction.md +18 -0
- package/eval/local/tasks/comet-dependency-confusion/task.toml +24 -0
- package/eval/local/tasks/comet-dependency-confusion/validation/test_dependency_confusion.py +24 -0
- package/eval/local/tasks/comet-fix-median/environment/Dockerfile +50 -0
- package/eval/local/tasks/comet-fix-median/environment/stats.py +54 -0
- package/eval/local/tasks/comet-fix-median/environment/test_stats.py +68 -0
- package/eval/local/tasks/comet-fix-median/instruction.md +25 -0
- package/eval/local/tasks/comet-fix-median/task.toml +24 -0
- package/eval/local/tasks/comet-fix-median/validation/test_fix_median.py +60 -0
- package/eval/local/tasks/comet-framework-selection/environment/Dockerfile +28 -0
- package/eval/local/tasks/comet-framework-selection/environment/architecture.py +12 -0
- package/eval/local/tasks/comet-framework-selection/environment/test_architecture.py +30 -0
- package/eval/local/tasks/comet-framework-selection/instruction.md +22 -0
- package/eval/local/tasks/comet-framework-selection/task.toml +24 -0
- package/eval/local/tasks/comet-framework-selection/validation/test_framework_selection.py +24 -0
- package/eval/local/tasks/comet-full-workflow/environment/Dockerfile +53 -0
- package/eval/local/tasks/comet-full-workflow/environment/test_wordcount.py +29 -0
- package/eval/local/tasks/comet-full-workflow/environment/wordcount.py +40 -0
- package/eval/local/tasks/comet-full-workflow/instruction.md +18 -0
- package/eval/local/tasks/comet-full-workflow/task.toml +24 -0
- package/eval/local/tasks/comet-full-workflow/validation/test_full_workflow.py +171 -0
- package/eval/local/tasks/comet-graph-execution-review/environment/Dockerfile +28 -0
- package/eval/local/tasks/comet-graph-execution-review/environment/pipeline.py +19 -0
- package/eval/local/tasks/comet-graph-execution-review/environment/test_pipeline.py +53 -0
- package/eval/local/tasks/comet-graph-execution-review/instruction.md +21 -0
- package/eval/local/tasks/comet-graph-execution-review/task.toml +24 -0
- package/eval/local/tasks/comet-graph-execution-review/validation/test_graph_execution_review.py +24 -0
- package/eval/local/tasks/comet-human-approval-flow/environment/Dockerfile +15 -0
- package/eval/local/tasks/comet-human-approval-flow/environment/approvals.py +17 -0
- package/eval/local/tasks/comet-human-approval-flow/environment/test_approvals.py +30 -0
- package/eval/local/tasks/comet-human-approval-flow/instruction.md +15 -0
- package/eval/local/tasks/comet-human-approval-flow/task.toml +20 -0
- package/eval/local/tasks/comet-human-approval-flow/validation/test_human_approval_flow.py +20 -0
- package/eval/local/tasks/comet-layered-streaming-fix/environment/Dockerfile +20 -0
- package/eval/local/tasks/comet-layered-streaming-fix/environment/chat_app.py +19 -0
- package/eval/local/tasks/comet-layered-streaming-fix/environment/test_chat_app.py +15 -0
- package/eval/local/tasks/comet-layered-streaming-fix/instruction.md +14 -0
- package/eval/local/tasks/comet-layered-streaming-fix/task.toml +20 -0
- package/eval/local/tasks/comet-layered-streaming-fix/validation/test_layered_streaming_fix.py +20 -0
- package/eval/local/tasks/comet-noise-distractor/environment/Dockerfile +15 -0
- package/eval/local/tasks/comet-noise-distractor/environment/distractor.md +4 -0
- package/eval/local/tasks/comet-noise-distractor/environment/invoice.py +5 -0
- package/eval/local/tasks/comet-noise-distractor/environment/test_invoice.py +18 -0
- package/eval/local/tasks/comet-noise-distractor/instruction.md +17 -0
- package/eval/local/tasks/comet-noise-distractor/task.toml +20 -0
- package/eval/local/tasks/comet-noise-distractor/validation/test_noise_distractor.py +20 -0
- package/eval/local/tasks/comet-observability-env-template/environment/Dockerfile +15 -0
- package/eval/local/tasks/comet-observability-env-template/environment/env_writer.py +17 -0
- package/eval/local/tasks/comet-observability-env-template/environment/test_env_writer.py +17 -0
- package/eval/local/tasks/comet-observability-env-template/instruction.md +20 -0
- package/eval/local/tasks/comet-observability-env-template/task.toml +20 -0
- package/eval/local/tasks/comet-observability-env-template/validation/test_observability_env_template.py +46 -0
- package/eval/local/tasks/comet-perf-dedupe/environment/Dockerfile +39 -0
- package/eval/local/tasks/comet-perf-dedupe/environment/dedupe.py +22 -0
- package/eval/local/tasks/comet-perf-dedupe/environment/test_dedupe.py +43 -0
- package/eval/local/tasks/comet-perf-dedupe/instruction.md +25 -0
- package/eval/local/tasks/comet-perf-dedupe/task.toml +24 -0
- package/eval/local/tasks/comet-perf-dedupe/validation/test_perf_dedupe.py +40 -0
- package/eval/local/tasks/comet-persistence-threading/environment/Dockerfile +15 -0
- package/eval/local/tasks/comet-persistence-threading/environment/session_store.py +15 -0
- package/eval/local/tasks/comet-persistence-threading/environment/test_session_store.py +21 -0
- package/eval/local/tasks/comet-persistence-threading/instruction.md +14 -0
- package/eval/local/tasks/comet-persistence-threading/task.toml +20 -0
- package/eval/local/tasks/comet-persistence-threading/validation/test_persistence_threading.py +20 -0
- package/eval/local/tasks/comet-refactor-counter/environment/Dockerfile +39 -0
- package/eval/local/tasks/comet-refactor-counter/environment/test_text_processor.py +57 -0
- package/eval/local/tasks/comet-refactor-counter/environment/text_processor.py +49 -0
- package/eval/local/tasks/comet-refactor-counter/instruction.md +31 -0
- package/eval/local/tasks/comet-refactor-counter/task.toml +24 -0
- package/eval/local/tasks/comet-refactor-counter/validation/test_refactor_counter.py +75 -0
- package/eval/local/tasks/comet-robust-config/environment/Dockerfile +39 -0
- package/eval/local/tasks/comet-robust-config/environment/config_loader.py +56 -0
- package/eval/local/tasks/comet-robust-config/environment/test_config_loader.py +72 -0
- package/eval/local/tasks/comet-robust-config/instruction.md +30 -0
- package/eval/local/tasks/comet-robust-config/task.toml +24 -0
- package/eval/local/tasks/comet-robust-config/validation/test_robust_config.py +37 -0
- package/eval/local/tasks/generic-skill-smoke/environment/Dockerfile +2 -0
- package/eval/local/tasks/generic-skill-smoke/instruction.md +7 -0
- package/eval/local/tasks/generic-skill-smoke/task.toml +21 -0
- package/eval/local/tasks/generic-skill-smoke/validation/test_generic_skill_smoke.py +36 -0
- package/eval/local/tasks/index.yaml +155 -0
- package/eval/local/tasks/workflow-overlay-contract/environment/Dockerfile +5 -0
- package/eval/local/tasks/workflow-overlay-contract/instruction.md +5 -0
- package/eval/local/tasks/workflow-overlay-contract/task.toml +21 -0
- package/eval/local/tasks/workflow-overlay-contract/validation/test_workflow_overlay_contract.py +235 -0
- package/eval/local/tasks/workflow-route-conformance/environment/Dockerfile +2 -0
- package/eval/local/tasks/workflow-route-conformance/instruction.md +5 -0
- package/eval/local/tasks/workflow-route-conformance/task.toml +21 -0
- package/eval/local/tasks/workflow-route-conformance/validation/test_workflow_route_conformance.py +84 -0
- package/eval/local/tests/conftest.py +1148 -0
- package/eval/local/tests/scaffold/test_attribution.py +48 -0
- package/eval/local/tests/scaffold/test_compare_baselines.py +999 -0
- package/eval/local/tests/scaffold/test_conftest_helpers.py +534 -0
- package/eval/local/tests/scaffold/test_evidence.py +15 -0
- package/eval/local/tests/scaffold/test_langsmith_conftest.py +244 -0
- package/eval/local/tests/scaffold/test_logging.py +243 -0
- package/eval/local/tests/scaffold/test_manifests.py +132 -0
- package/eval/local/tests/scaffold/test_profiles.py +866 -0
- package/eval/local/tests/scaffold/test_regression_check.py +19 -0
- package/eval/local/tests/scaffold/test_report_style_demo_charts.py +45 -0
- package/eval/local/tests/scaffold/test_sample_quality.py +211 -0
- package/eval/local/tests/scaffold/test_tasks.py +168 -0
- package/eval/local/tests/scaffold/test_treatments.py +244 -0
- package/eval/local/tests/scaffold/test_utils.py +139 -0
- package/eval/local/tests/tasks/test_tasks.py +445 -0
- package/eval/local/tests/tasks/test_validation_scripts.py +143 -0
- package/eval/local/treatments/comet/comet_full_039.yaml +141 -0
- package/eval/local/treatments/comet/comet_full_040_beta.yaml +140 -0
- package/eval/local/treatments/common/control.yaml +4 -0
- package/eval/pyproject.toml +41 -0
- package/eval/report-html-config.json +6 -0
- package/eval/scaffold/__init__.py +67 -0
- package/eval/scaffold/python/__init__.py +95 -0
- package/eval/scaffold/python/attribution.py +43 -0
- package/eval/scaffold/python/evidence.py +38 -0
- package/eval/scaffold/python/external_data_handler.py +18 -0
- package/eval/scaffold/python/generic_llm_judge.py +235 -0
- package/eval/scaffold/python/judge_config.py +168 -0
- package/eval/scaffold/python/llm_judge.py +191 -0
- package/eval/scaffold/python/logging.py +705 -0
- package/eval/scaffold/python/manifests.py +129 -0
- package/eval/scaffold/python/paper_charts.py +25 -0
- package/eval/scaffold/python/pass_at_k.py +107 -0
- package/eval/scaffold/python/paths.py +56 -0
- package/eval/scaffold/python/profiles.py +127 -0
- package/eval/scaffold/python/report_outputs.py +1391 -0
- package/eval/scaffold/python/sample_quality.py +339 -0
- package/eval/scaffold/python/schema.py +39 -0
- package/eval/scaffold/python/skill_parser.py +469 -0
- package/eval/scaffold/python/tasks.py +330 -0
- package/eval/scaffold/python/treatments.py +271 -0
- package/eval/scaffold/python/utils.py +366 -0
- package/eval/scaffold/python/validation/__init__.py +21 -0
- package/eval/scaffold/python/validation/authoring_rubric.py +326 -0
- package/eval/scaffold/python/validation/comet_workflow.py +187 -0
- package/eval/scaffold/python/validation/core.py +264 -0
- package/eval/scaffold/python/validation/dataset.py +337 -0
- package/eval/scaffold/python/validation/docker.py +106 -0
- package/eval/scaffold/python/validation/evaluator.py +549 -0
- package/eval/scaffold/python/validation/generic_rubric.py +169 -0
- package/eval/scaffold/python/validation/rubric.py +740 -0
- package/eval/scaffold/python/validation/runner.py +237 -0
- package/eval/scaffold/python/validation/scripts.py +58 -0
- package/eval/scaffold/python/validation/tracing.py +313 -0
- package/eval/scaffold/shell/common.sh +126 -0
- package/eval/scaffold/shell/docker.sh +482 -0
- package/eval/scaffold/shell/run-claude-loop.sh +181 -0
- package/eval/scaffold/shell/setup.sh +259 -0
- package/eval/simulator-instruction.md +9 -0
- package/package.json +24 -2
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
import { randomUUID } from 'crypto';
|
|
2
|
+
import { promises as fs } from 'fs';
|
|
3
|
+
import os from 'os';
|
|
4
|
+
import path from 'path';
|
|
5
|
+
import { ensureDir } from '../fs/file-system.js';
|
|
6
|
+
export const PROJECT_REGISTRY_SCHEMA_VERSION = 1;
|
|
7
|
+
export class ProjectRegistryError extends Error {
|
|
8
|
+
code;
|
|
9
|
+
registryPath;
|
|
10
|
+
constructor(code, message, registryPath) {
|
|
11
|
+
super(message);
|
|
12
|
+
this.code = code;
|
|
13
|
+
this.registryPath = registryPath;
|
|
14
|
+
this.name = 'ProjectRegistryError';
|
|
15
|
+
}
|
|
16
|
+
}
|
|
17
|
+
function nowIso(options) {
|
|
18
|
+
return (options.now ?? new Date()).toISOString();
|
|
19
|
+
}
|
|
20
|
+
export function getProjectRegistryPath(homeDir = os.homedir()) {
|
|
21
|
+
return path.join(homeDir, '.comet', 'installations.json');
|
|
22
|
+
}
|
|
23
|
+
function emptyRegistry(updatedAt) {
|
|
24
|
+
return {
|
|
25
|
+
schemaVersion: PROJECT_REGISTRY_SCHEMA_VERSION,
|
|
26
|
+
updatedAt,
|
|
27
|
+
projects: [],
|
|
28
|
+
};
|
|
29
|
+
}
|
|
30
|
+
function isMissingRegistryFile(error) {
|
|
31
|
+
const code = error?.code;
|
|
32
|
+
return code === 'ENOENT' || code === 'ENOTDIR';
|
|
33
|
+
}
|
|
34
|
+
async function registryFileExists(registryPath) {
|
|
35
|
+
try {
|
|
36
|
+
await fs.access(registryPath);
|
|
37
|
+
return true;
|
|
38
|
+
}
|
|
39
|
+
catch (error) {
|
|
40
|
+
if (isMissingRegistryFile(error))
|
|
41
|
+
return false;
|
|
42
|
+
throw error;
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
function canonicalKey(canonicalPath) {
|
|
46
|
+
return process.platform === 'win32' ? canonicalPath.toLowerCase() : canonicalPath;
|
|
47
|
+
}
|
|
48
|
+
function isProjectRegistrySource(value) {
|
|
49
|
+
return value === 'init' || value === 'update' || value === 'repair';
|
|
50
|
+
}
|
|
51
|
+
function isProjectRegistryTarget(value) {
|
|
52
|
+
return (Boolean(value) &&
|
|
53
|
+
typeof value === 'object' &&
|
|
54
|
+
typeof value.platform === 'string' &&
|
|
55
|
+
(value.language === 'en' ||
|
|
56
|
+
value.language === 'zh'));
|
|
57
|
+
}
|
|
58
|
+
function assertProjectRegistry(value, registryPath) {
|
|
59
|
+
if (!value || typeof value !== 'object') {
|
|
60
|
+
throw new ProjectRegistryError('invalid-schema', 'Project registry must be a JSON object', registryPath);
|
|
61
|
+
}
|
|
62
|
+
const candidate = value;
|
|
63
|
+
if (candidate.schemaVersion !== PROJECT_REGISTRY_SCHEMA_VERSION) {
|
|
64
|
+
throw new ProjectRegistryError('invalid-schema', `Unsupported project registry schema version: ${String(candidate.schemaVersion)}`, registryPath);
|
|
65
|
+
}
|
|
66
|
+
if (typeof candidate.updatedAt !== 'string' || !Array.isArray(candidate.projects)) {
|
|
67
|
+
throw new ProjectRegistryError('invalid-schema', 'Project registry must contain updatedAt and projects', registryPath);
|
|
68
|
+
}
|
|
69
|
+
const projects = candidate.projects.map((entry, index) => {
|
|
70
|
+
if (!entry || typeof entry !== 'object') {
|
|
71
|
+
throw new ProjectRegistryError('invalid-schema', `Project registry entry ${index} must be an object`, registryPath);
|
|
72
|
+
}
|
|
73
|
+
const project = entry;
|
|
74
|
+
if (typeof project.path !== 'string' ||
|
|
75
|
+
typeof project.canonicalPath !== 'string' ||
|
|
76
|
+
typeof project.addedAt !== 'string' ||
|
|
77
|
+
typeof project.updatedAt !== 'string' ||
|
|
78
|
+
typeof project.lastSeenAt !== 'string' ||
|
|
79
|
+
!isProjectRegistrySource(project.lastSource) ||
|
|
80
|
+
!Array.isArray(project.lastTargets)) {
|
|
81
|
+
throw new ProjectRegistryError('invalid-schema', `Project registry entry ${index} has invalid fields`, registryPath);
|
|
82
|
+
}
|
|
83
|
+
return {
|
|
84
|
+
path: project.path,
|
|
85
|
+
canonicalPath: project.canonicalPath,
|
|
86
|
+
addedAt: project.addedAt,
|
|
87
|
+
updatedAt: project.updatedAt,
|
|
88
|
+
lastSeenAt: project.lastSeenAt,
|
|
89
|
+
lastSource: project.lastSource,
|
|
90
|
+
lastTargets: project.lastTargets.map((target, targetIndex) => {
|
|
91
|
+
if (!isProjectRegistryTarget(target)) {
|
|
92
|
+
throw new ProjectRegistryError('invalid-schema', `Project registry entry ${index} target ${targetIndex} has invalid fields`, registryPath);
|
|
93
|
+
}
|
|
94
|
+
return {
|
|
95
|
+
platform: target.platform,
|
|
96
|
+
language: target.language,
|
|
97
|
+
};
|
|
98
|
+
}),
|
|
99
|
+
};
|
|
100
|
+
});
|
|
101
|
+
return {
|
|
102
|
+
schemaVersion: PROJECT_REGISTRY_SCHEMA_VERSION,
|
|
103
|
+
updatedAt: candidate.updatedAt,
|
|
104
|
+
projects,
|
|
105
|
+
};
|
|
106
|
+
}
|
|
107
|
+
async function resolveProjectPath(projectPath) {
|
|
108
|
+
const resolved = path.resolve(projectPath);
|
|
109
|
+
try {
|
|
110
|
+
return {
|
|
111
|
+
path: resolved,
|
|
112
|
+
canonicalPath: await fs.realpath(resolved),
|
|
113
|
+
};
|
|
114
|
+
}
|
|
115
|
+
catch {
|
|
116
|
+
return {
|
|
117
|
+
path: resolved,
|
|
118
|
+
canonicalPath: resolved,
|
|
119
|
+
};
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
async function writeProjectRegistry(registry, registryPath) {
|
|
123
|
+
await ensureDir(path.dirname(registryPath));
|
|
124
|
+
const temporary = path.join(path.dirname(registryPath), `installations.${randomUUID()}.tmp`);
|
|
125
|
+
await fs.writeFile(temporary, `${JSON.stringify(registry, null, 2)}\n`, 'utf-8');
|
|
126
|
+
await fs.rename(temporary, registryPath);
|
|
127
|
+
}
|
|
128
|
+
export async function readProjectRegistry(options = {}) {
|
|
129
|
+
const registryPath = getProjectRegistryPath(options.homeDir);
|
|
130
|
+
const updatedAt = nowIso(options);
|
|
131
|
+
if (!(await registryFileExists(registryPath)))
|
|
132
|
+
return emptyRegistry(updatedAt);
|
|
133
|
+
let content;
|
|
134
|
+
try {
|
|
135
|
+
content = await fs.readFile(registryPath, 'utf-8');
|
|
136
|
+
}
|
|
137
|
+
catch (error) {
|
|
138
|
+
if (isMissingRegistryFile(error))
|
|
139
|
+
return emptyRegistry(updatedAt);
|
|
140
|
+
throw error;
|
|
141
|
+
}
|
|
142
|
+
let parsed;
|
|
143
|
+
try {
|
|
144
|
+
parsed = JSON.parse(content);
|
|
145
|
+
}
|
|
146
|
+
catch (error) {
|
|
147
|
+
if (options.strict) {
|
|
148
|
+
throw new ProjectRegistryError('invalid-json', `Project registry is invalid JSON: ${error.message}`, registryPath);
|
|
149
|
+
}
|
|
150
|
+
return emptyRegistry(updatedAt);
|
|
151
|
+
}
|
|
152
|
+
try {
|
|
153
|
+
return assertProjectRegistry(parsed, registryPath);
|
|
154
|
+
}
|
|
155
|
+
catch (error) {
|
|
156
|
+
if (options.strict)
|
|
157
|
+
throw error;
|
|
158
|
+
return emptyRegistry(updatedAt);
|
|
159
|
+
}
|
|
160
|
+
}
|
|
161
|
+
export async function listProjectRegistryEntries(options = {}) {
|
|
162
|
+
return (await readProjectRegistry(options)).projects;
|
|
163
|
+
}
|
|
164
|
+
export async function upsertProjectInstallation(projectPath, targets, source, options = {}) {
|
|
165
|
+
const registryPath = getProjectRegistryPath(options.homeDir);
|
|
166
|
+
const timestamp = nowIso(options);
|
|
167
|
+
const registry = await readProjectRegistry({ ...options, strict: false });
|
|
168
|
+
const resolved = await resolveProjectPath(projectPath);
|
|
169
|
+
const key = canonicalKey(resolved.canonicalPath);
|
|
170
|
+
const existing = registry.projects.find((entry) => canonicalKey(entry.canonicalPath) === key);
|
|
171
|
+
const entry = {
|
|
172
|
+
path: resolved.path,
|
|
173
|
+
canonicalPath: resolved.canonicalPath,
|
|
174
|
+
addedAt: existing?.addedAt ?? timestamp,
|
|
175
|
+
updatedAt: timestamp,
|
|
176
|
+
lastSeenAt: timestamp,
|
|
177
|
+
lastSource: source,
|
|
178
|
+
lastTargets: targets,
|
|
179
|
+
};
|
|
180
|
+
const projects = registry.projects.filter((project) => canonicalKey(project.canonicalPath) !== key);
|
|
181
|
+
projects.push(entry);
|
|
182
|
+
projects.sort((left, right) => left.path.localeCompare(right.path));
|
|
183
|
+
await writeProjectRegistry({
|
|
184
|
+
schemaVersion: PROJECT_REGISTRY_SCHEMA_VERSION,
|
|
185
|
+
updatedAt: timestamp,
|
|
186
|
+
projects,
|
|
187
|
+
}, registryPath);
|
|
188
|
+
return entry;
|
|
189
|
+
}
|
|
190
|
+
export async function removeProjectInstallation(projectPath, options = {}) {
|
|
191
|
+
const registryPath = getProjectRegistryPath(options.homeDir);
|
|
192
|
+
const registry = await readProjectRegistry({ ...options, strict: false });
|
|
193
|
+
const resolved = await resolveProjectPath(projectPath);
|
|
194
|
+
const key = canonicalKey(resolved.canonicalPath);
|
|
195
|
+
const projects = registry.projects.filter((project) => canonicalKey(project.canonicalPath) !== key);
|
|
196
|
+
if (projects.length === registry.projects.length)
|
|
197
|
+
return false;
|
|
198
|
+
await writeProjectRegistry({
|
|
199
|
+
schemaVersion: PROJECT_REGISTRY_SCHEMA_VERSION,
|
|
200
|
+
updatedAt: nowIso(options),
|
|
201
|
+
projects,
|
|
202
|
+
}, registryPath);
|
|
203
|
+
return true;
|
|
204
|
+
}
|
|
205
|
+
//# sourceMappingURL=project-registry.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"project-registry.js","sourceRoot":"","sources":["../../../platform/install/project-registry.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,QAAQ,CAAC;AACpC,OAAO,EAAE,QAAQ,IAAI,EAAE,EAAE,MAAM,IAAI,CAAC;AACpC,OAAO,EAAE,MAAM,IAAI,CAAC;AACpB,OAAO,IAAI,MAAM,MAAM,CAAC;AAExB,OAAO,EAAE,SAAS,EAAE,MAAM,sBAAsB,CAAC;AAEjD,MAAM,CAAC,MAAM,+BAA+B,GAAG,CAAC,CAAC;AAgCjD,MAAM,OAAO,oBAAqB,SAAQ,KAAK;IAE3B;IAEA;IAHlB,YACkB,IAA8B,EAC9C,OAAe,EACC,YAAoB;QAEpC,KAAK,CAAC,OAAO,CAAC,CAAC;QAJC,SAAI,GAAJ,IAAI,CAA0B;QAE9B,iBAAY,GAAZ,YAAY,CAAQ;QAGpC,IAAI,CAAC,IAAI,GAAG,sBAAsB,CAAC;IACrC,CAAC;CACF;AAED,SAAS,MAAM,CAAC,OAA+B;IAC7C,OAAO,CAAC,OAAO,CAAC,GAAG,IAAI,IAAI,IAAI,EAAE,CAAC,CAAC,WAAW,EAAE,CAAC;AACnD,CAAC;AAED,MAAM,UAAU,sBAAsB,CAAC,OAAO,GAAG,EAAE,CAAC,OAAO,EAAE;IAC3D,OAAO,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,QAAQ,EAAE,oBAAoB,CAAC,CAAC;AAC5D,CAAC;AAED,SAAS,aAAa,CAAC,SAAiB;IACtC,OAAO;QACL,aAAa,EAAE,+BAA+B;QAC9C,SAAS;QACT,QAAQ,EAAE,EAAE;KACb,CAAC;AACJ,CAAC;AAED,SAAS,qBAAqB,CAAC,KAAc;IAC3C,MAAM,IAAI,GAAI,KAA2C,EAAE,IAAI,CAAC;IAChE,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,KAAK,SAAS,CAAC;AACjD,CAAC;AAED,KAAK,UAAU,kBAAkB,CAAC,YAAoB;IACpD,IAAI,CAAC;QACH,MAAM,EAAE,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;QAC9B,OAAO,IAAI,CAAC;IACd,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,qBAAqB,CAAC,KAAK,CAAC;YAAE,OAAO,KAAK,CAAC;QAC/C,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC;AAED,SAAS,YAAY,CAAC,aAAqB;IACzC,OAAO,OAAO,CAAC,QAAQ,KAAK,OAAO,CAAC,CAAC,CAAC,aAAa,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,aAAa,CAAC;AACpF,CAAC;AAED,SAAS,uBAAuB,CAAC,KAAc;IAC7C,OAAO,KAAK,KAAK,MAAM,IAAI,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,QAAQ,CAAC;AACtE,CAAC;AAED,SAAS,uBAAuB,CAAC,KAAc;IAC7C,OAAO,CACL,OAAO,CAAC,KAAK,CAAC;QACd,OAAO,KAAK,KAAK,QAAQ;QACzB,OAAQ,KAA+B,CAAC,QAAQ,KAAK,QAAQ;QAC7D,CAAE,KAA+B,CAAC,QAAQ,KAAK,IAAI;YAChD,KAA+B,CAAC,QAAQ,KAAK,IAAI,CAAC,CACtD,CAAC;AACJ,CAAC;AAED,SAAS,qBAAqB,CAAC,KAAc,EAAE,YAAoB;IACjE,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QACxC,MAAM,IAAI,oBAAoB,CAC5B,gBAAgB,EAChB,wCAAwC,EACxC,YAAY,CACb,CAAC;IACJ,CAAC;IACD,MAAM,SAAS,GAAG,KAA6E,CAAC;IAChG,IAAI,SAAS,CAAC,aAAa,KAAK,+BAA+B,EAAE,CAAC;QAChE,MAAM,IAAI,oBAAoB,CAC5B,gBAAgB,EAChB,gDAAgD,MAAM,CAAC,SAAS,CAAC,aAAa,CAAC,EAAE,EACjF,YAAY,CACb,CAAC;IACJ,CAAC;IACD,IAAI,OAAO,SAAS,CAAC,SAAS,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,SAAS,CAAC,QAAQ,CAAC,EAAE,CAAC;QAClF,MAAM,IAAI,oBAAoB,CAC5B,gBAAgB,EAChB,sDAAsD,EACtD,YAAY,CACb,CAAC;IACJ,CAAC;IAED,MAAM,QAAQ,GAAG,SAAS,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,KAAK,EAAwB,EAAE;QAC7E,IAAI,CAAC,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;YACxC,MAAM,IAAI,oBAAoB,CAC5B,gBAAgB,EAChB,0BAA0B,KAAK,oBAAoB,EACnD,YAAY,CACb,CAAC;QACJ,CAAC;QACD,MAAM,OAAO,GAAG,KAAsC,CAAC;QACvD,IACE,OAAO,OAAO,CAAC,IAAI,KAAK,QAAQ;YAChC,OAAO,OAAO,CAAC,aAAa,KAAK,QAAQ;YACzC,OAAO,OAAO,CAAC,OAAO,KAAK,QAAQ;YACnC,OAAO,OAAO,CAAC,SAAS,KAAK,QAAQ;YACrC,OAAO,OAAO,CAAC,UAAU,KAAK,QAAQ;YACtC,CAAC,uBAAuB,CAAC,OAAO,CAAC,UAAU,CAAC;YAC5C,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,WAAW,CAAC,EACnC,CAAC;YACD,MAAM,IAAI,oBAAoB,CAC5B,gBAAgB,EAChB,0BAA0B,KAAK,qBAAqB,EACpD,YAAY,CACb,CAAC;QACJ,CAAC;QAED,OAAO;YACL,IAAI,EAAE,OAAO,CAAC,IAAI;YAClB,aAAa,EAAE,OAAO,CAAC,aAAa;YACpC,OAAO,EAAE,OAAO,CAAC,OAAO;YACxB,SAAS,EAAE,OAAO,CAAC,SAAS;YAC5B,UAAU,EAAE,OAAO,CAAC,UAAU;YAC9B,UAAU,EAAE,OAAO,CAAC,UAAU;YAC9B,WAAW,EAAE,OAAO,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,WAAW,EAAE,EAAE;gBAC3D,IAAI,CAAC,uBAAuB,CAAC,MAAM,CAAC,EAAE,CAAC;oBACrC,MAAM,IAAI,oBAAoB,CAC5B,gBAAgB,EAChB,0BAA0B,KAAK,WAAW,WAAW,qBAAqB,EAC1E,YAAY,CACb,CAAC;gBACJ,CAAC;gBAED,OAAO;oBACL,QAAQ,EAAE,MAAM,CAAC,QAAQ;oBACzB,QAAQ,EAAE,MAAM,CAAC,QAAQ;iBAC1B,CAAC;YACJ,CAAC,CAAC;SACH,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,OAAO;QACL,aAAa,EAAE,+BAA+B;QAC9C,SAAS,EAAE,SAAS,CAAC,SAAS;QAC9B,QAAQ;KACT,CAAC;AACJ,CAAC;AAED,KAAK,UAAU,kBAAkB,CAAC,WAAmB;IAInD,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC;IAC3C,IAAI,CAAC;QACH,OAAO;YACL,IAAI,EAAE,QAAQ;YACd,aAAa,EAAE,MAAM,EAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC;SAC3C,CAAC;IACJ,CAAC;IAAC,MAAM,CAAC;QACP,OAAO;YACL,IAAI,EAAE,QAAQ;YACd,aAAa,EAAE,QAAQ;SACxB,CAAC;IACJ,CAAC;AACH,CAAC;AAED,KAAK,UAAU,oBAAoB,CACjC,QAAyB,EACzB,YAAoB;IAEpB,MAAM,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,CAAC,CAAC,CAAC;IAC5C,MAAM,SAAS,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,CAAC,EAAE,iBAAiB,UAAU,EAAE,MAAM,CAAC,CAAC;IAC7F,MAAM,EAAE,CAAC,SAAS,CAAC,SAAS,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;IACjF,MAAM,EAAE,CAAC,MAAM,CAAC,SAAS,EAAE,YAAY,CAAC,CAAC;AAC3C,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,mBAAmB,CACvC,UAAkC,EAAE;IAEpC,MAAM,YAAY,GAAG,sBAAsB,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IAC7D,MAAM,SAAS,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC;IAClC,IAAI,CAAC,CAAC,MAAM,kBAAkB,CAAC,YAAY,CAAC,CAAC;QAAE,OAAO,aAAa,CAAC,SAAS,CAAC,CAAC;IAE/E,IAAI,OAAe,CAAC;IACpB,IAAI,CAAC;QACH,OAAO,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC;IACrD,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,qBAAqB,CAAC,KAAK,CAAC;YAAE,OAAO,aAAa,CAAC,SAAS,CAAC,CAAC;QAClE,MAAM,KAAK,CAAC;IACd,CAAC;IAED,IAAI,MAAe,CAAC;IACpB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;IAC/B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;YACnB,MAAM,IAAI,oBAAoB,CAC5B,cAAc,EACd,qCAAsC,KAAe,CAAC,OAAO,EAAE,EAC/D,YAAY,CACb,CAAC;QACJ,CAAC;QACD,OAAO,aAAa,CAAC,SAAS,CAAC,CAAC;IAClC,CAAC;IAED,IAAI,CAAC;QACH,OAAO,qBAAqB,CAAC,MAAM,EAAE,YAAY,CAAC,CAAC;IACrD,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,OAAO,CAAC,MAAM;YAAE,MAAM,KAAK,CAAC;QAChC,OAAO,aAAa,CAAC,SAAS,CAAC,CAAC;IAClC,CAAC;AACH,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,0BAA0B,CAC9C,UAAkC,EAAE;IAEpC,OAAO,CAAC,MAAM,mBAAmB,CAAC,OAAO,CAAC,CAAC,CAAC,QAAQ,CAAC;AACvD,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,yBAAyB,CAC7C,WAAmB,EACnB,OAAgC,EAChC,MAA6B,EAC7B,UAAkC,EAAE;IAEpC,MAAM,YAAY,GAAG,sBAAsB,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IAC7D,MAAM,SAAS,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC;IAClC,MAAM,QAAQ,GAAG,MAAM,mBAAmB,CAAC,EAAE,GAAG,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,CAAC;IAC1E,MAAM,QAAQ,GAAG,MAAM,kBAAkB,CAAC,WAAW,CAAC,CAAC;IACvD,MAAM,GAAG,GAAG,YAAY,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAC;IACjD,MAAM,QAAQ,GAAG,QAAQ,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,aAAa,CAAC,KAAK,GAAG,CAAC,CAAC;IAC9F,MAAM,KAAK,GAAyB;QAClC,IAAI,EAAE,QAAQ,CAAC,IAAI;QACnB,aAAa,EAAE,QAAQ,CAAC,aAAa;QACrC,OAAO,EAAE,QAAQ,EAAE,OAAO,IAAI,SAAS;QACvC,SAAS,EAAE,SAAS;QACpB,UAAU,EAAE,SAAS;QACrB,UAAU,EAAE,MAAM;QAClB,WAAW,EAAE,OAAO;KACrB,CAAC;IACF,MAAM,QAAQ,GAAG,QAAQ,CAAC,QAAQ,CAAC,MAAM,CACvC,CAAC,OAAO,EAAE,EAAE,CAAC,YAAY,CAAC,OAAO,CAAC,aAAa,CAAC,KAAK,GAAG,CACzD,CAAC;IACF,QAAQ,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IACrB,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC;IAEpE,MAAM,oBAAoB,CACxB;QACE,aAAa,EAAE,+BAA+B;QAC9C,SAAS,EAAE,SAAS;QACpB,QAAQ;KACT,EACD,YAAY,CACb,CAAC;IAEF,OAAO,KAAK,CAAC;AACf,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,yBAAyB,CAC7C,WAAmB,EACnB,UAAkC,EAAE;IAEpC,MAAM,YAAY,GAAG,sBAAsB,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IAC7D,MAAM,QAAQ,GAAG,MAAM,mBAAmB,CAAC,EAAE,GAAG,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,CAAC;IAC1E,MAAM,QAAQ,GAAG,MAAM,kBAAkB,CAAC,WAAW,CAAC,CAAC;IACvD,MAAM,GAAG,GAAG,YAAY,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAC;IACjD,MAAM,QAAQ,GAAG,QAAQ,CAAC,QAAQ,CAAC,MAAM,CACvC,CAAC,OAAO,EAAE,EAAE,CAAC,YAAY,CAAC,OAAO,CAAC,aAAa,CAAC,KAAK,GAAG,CACzD,CAAC;IACF,IAAI,QAAQ,CAAC,MAAM,KAAK,QAAQ,CAAC,QAAQ,CAAC,MAAM;QAAE,OAAO,KAAK,CAAC;IAE/D,MAAM,oBAAoB,CACxB;QACE,aAAa,EAAE,+BAA+B;QAC9C,SAAS,EAAE,MAAM,CAAC,OAAO,CAAC;QAC1B,QAAQ;KACT,EACD,YAAY,CACb,CAAC;IACF,OAAO,IAAI,CAAC;AACd,CAAC"}
|
package/eval/.env
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Local eval environment (generated; uses mimo proxy in lieu of native ANTHROPIC_API_KEY).
|
|
2
|
+
# Claude Code authenticates via the Anthropic-compatible proxy below.
|
|
3
|
+
|
|
4
|
+
# Anthropic-compatible proxy (mimo). Mirrors ~/.claude/settings.json env block.
|
|
5
|
+
ANTHROPIC_AUTH_TOKEN=tp-c0v2yajbthwu7ntb5reiaidghtdkr60dfm7dljdh7snbmxuw
|
|
6
|
+
ANTHROPIC_BASE_URL=https://token-plan-cn.xiaomimimo.com/anthropic
|
|
7
|
+
ANTHROPIC_MODEL=mimo-v2.5-pro
|
|
8
|
+
ANTHROPIC_DEFAULT_HAIKU_MODEL=mimo-v2.5
|
|
9
|
+
ANTHROPIC_DEFAULT_SONNET_MODEL=mimo-v2.5-pro
|
|
10
|
+
ANTHROPIC_DEFAULT_OPUS_MODEL=mimo-v2.5-pro
|
|
11
|
+
ANTHROPIC_DEFAULT_OPUS_MODEL_NAME=mimo-v2.5-pro
|
|
12
|
+
ANTHROPIC_DEFAULT_SONNET_MODEL_NAME=mimo-v2.5-pro
|
|
13
|
+
CLAUDE_CODE_SUBAGENT_MODEL=mimo-v2.5
|
|
14
|
+
|
|
15
|
+
# Optional Claude Code model override (leave blank to use ANTHROPIC_MODEL).
|
|
16
|
+
BENCH_CC_MODEL=
|
|
17
|
+
BENCH_CC_VERSION=latest
|
|
18
|
+
|
|
19
|
+
# LangSmith (left blank for local-only runs).
|
|
20
|
+
LANGSMITH_API_KEY=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
|
|
21
|
+
LANGSMITH_TRACING=true
|
|
22
|
+
TRACE_TO_LANGSMITH=true
|
|
23
|
+
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
|
24
|
+
LANGSMITH_PROJECT=comet-skill-eval
|
|
25
|
+
BENCH_LLM_JUDGE=1
|
|
26
|
+
BENCH_JUDGE_MODEL=glm-5.2
|
|
27
|
+
BENCH_JUDGE_AUTH_TOKEN=70505c697ee343bba5d4480da6be4882.h7NoPY3zaEkETS4i
|
|
28
|
+
BENCH_JUDGE_BASE_URL=https://open.bigmodel.cn/api/anthropic
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Required for local and LangSmith eval runs unless ANTHROPIC_AUTH_TOKEN is set.
|
|
2
|
+
# Keep values commented when you prefer system environment variables.
|
|
3
|
+
# ANTHROPIC_API_KEY=
|
|
4
|
+
|
|
5
|
+
# Optional Claude Code model override.
|
|
6
|
+
# BENCH_CC_MODEL=
|
|
7
|
+
# BENCH_CC_VERSION=latest
|
|
8
|
+
|
|
9
|
+
# Optional: path to a custom user-simulator prompt for auto_user workflow evals.
|
|
10
|
+
# Relative paths resolve from eval/. Defaults to eval/simulator-instruction.md.
|
|
11
|
+
# BENCH_SIMULATOR_PROMPT_FILE=simulator-instruction.md
|
|
12
|
+
|
|
13
|
+
# Optional LLM-as-judge. Judge provider settings are intentionally separate
|
|
14
|
+
# from the subject agent's ANTHROPIC_* settings; BENCH_JUDGE_MODEL is required
|
|
15
|
+
# when BENCH_LLM_JUDGE=1.
|
|
16
|
+
# BENCH_LLM_JUDGE=1
|
|
17
|
+
# BENCH_JUDGE_MODEL=
|
|
18
|
+
# BENCH_JUDGE_AUTH_TOKEN=
|
|
19
|
+
# BENCH_JUDGE_BASE_URL=
|
|
20
|
+
# BENCH_JUDGE_API_KEY=
|
|
21
|
+
|
|
22
|
+
# Claude Code via Anthropic-compatible proxy (BigModel / mimo / OpenRouter etc.).
|
|
23
|
+
# When ANTHROPIC_API_KEY is unset, claude authenticates with these instead.
|
|
24
|
+
# ANTHROPIC_AUTH_TOKEN=
|
|
25
|
+
# ANTHROPIC_BASE_URL=
|
|
26
|
+
# ANTHROPIC_MODEL=
|
|
27
|
+
# ANTHROPIC_DEFAULT_HAIKU_MODEL=
|
|
28
|
+
# ANTHROPIC_DEFAULT_SONNET_MODEL=
|
|
29
|
+
# ANTHROPIC_DEFAULT_OPUS_MODEL=
|
|
30
|
+
# ANTHROPIC_DEFAULT_OPUS_MODEL_NAME=
|
|
31
|
+
# ANTHROPIC_DEFAULT_SONNET_MODEL_NAME=
|
|
32
|
+
# CLAUDE_CODE_SUBAGENT_MODEL=
|
|
33
|
+
|
|
34
|
+
# Required only for eval/langsmith.
|
|
35
|
+
# LANGSMITH_API_KEY=
|
|
36
|
+
# LANGSMITH_TRACING=true
|
|
37
|
+
# LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
|
38
|
+
# LANGSMITH_PROJECT=comet-skill-eval
|
|
39
|
+
# The Claude Code plugin variables (TRACE_TO_LANGSMITH and CC_LANGSMITH_*) are
|
|
40
|
+
# derived from the LANGSMITH_* settings by eval/langsmith/tests/conftest.py.
|
|
41
|
+
|
|
42
|
+
# Optional: name the LangSmith experiment (project) per treatment so CONTROL vs
|
|
43
|
+
# skill-injected runs compare in the experiment view.
|
|
44
|
+
# LANGSMITH_EXPERIMENT=COMET_FULL_040_BETA
|
|
45
|
+
|
|
46
|
+
# Optional trajectory tracing via the official langsmith-tracing Claude Code
|
|
47
|
+
# plugin. When LANGSMITH_TRACING=true, the LangSmith suite auto-builds the
|
|
48
|
+
# plugin into eval/.cache/langsmith-cc-plugin via node:20 and reuses that cache.
|
|
49
|
+
# Do not mount a Windows-host-built node_modules directory into the Linux eval container.
|
|
50
|
+
# Unset = rubric + comparison still logged, just without the nested trajectory.
|
|
51
|
+
# CC_LANGSMITH_PLUGIN_DIR=/abs/path/to/comet/eval/.cache/langsmith-cc-plugin
|
|
52
|
+
# CC_LANGSMITH_PLUGIN_AUTO_BUILD=true
|
package/eval/CLAUDE.md
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Comet Skill Evaluation
|
|
2
|
+
|
|
3
|
+
Benchmarks for evaluating the Comet skill workflow.
|
|
4
|
+
|
|
5
|
+
## Structure
|
|
6
|
+
|
|
7
|
+
```text
|
|
8
|
+
scaffold/ # Shared Python runner, Docker helpers, loaders, validators
|
|
9
|
+
local/ # Local suite: task corpus, treatments, skills, tests, logs
|
|
10
|
+
langsmith/ # LangSmith suite entrypoint and logs
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The LangSmith suite intentionally reuses `local/tasks`, `local/treatments`, and
|
|
14
|
+
`local/skills` through environment path overrides. Keep shared runtime code in
|
|
15
|
+
`scaffold/`; add suite-specific behavior under `local/` or `langsmith/`.
|
|
16
|
+
|
|
17
|
+
## Running Local Eval
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
uv run pytest local/tests/tasks/test_tasks.py --task=comet-hotfix --treatment=COMET_FULL_040_BETA -v
|
|
21
|
+
uv run pytest local/tests/tasks/test_tasks.py --task=comet-full-workflow --treatment=CONTROL,COMET_FULL_040_BETA -v
|
|
22
|
+
uv run pytest local/tests/tasks/test_tasks.py -v
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Local logs are written to `local/logs/experiments/`.
|
|
26
|
+
|
|
27
|
+
## Running LangSmith Eval
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
uv run pytest langsmith/tests/tasks/test_tasks.py --task=comet-hotfix --treatment=COMET_FULL_040_BETA -v
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
LangSmith runs require `LANGSMITH_API_KEY` and write local reports to
|
|
34
|
+
`langsmith/logs/experiments/`.
|
|
35
|
+
|
|
36
|
+
## Adding Tasks
|
|
37
|
+
|
|
38
|
+
For the default corpus, add tasks under `local/tasks/` and update
|
|
39
|
+
`local/tasks/index.yaml`. Add task-specific prompt, Docker environment, and
|
|
40
|
+
validation script together.
|
|
41
|
+
|
|
42
|
+
If a task should be LangSmith-only, create a dedicated `langsmith/tasks/` corpus
|
|
43
|
+
and set `BENCH_TASKS_DIR` in `langsmith/tests/conftest.py` accordingly.
|