@rpamis/comet 0.4.0-beta.16 → 0.4.0-beta.18

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (761) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +85 -56
  3. package/assets/eval-reports/comet-native-vs-beta16-beta17-20260810/native-benchmark-report.html +8 -0
  4. package/assets/eval-reports/comet-native-vs-beta16-beta17-20260810/native-benchmark-report.json +286 -0
  5. package/assets/manifest.json +26 -17
  6. package/assets/skills/comet/SKILL.md +1 -1
  7. package/assets/skills/comet/agents/openai.yaml +3 -0
  8. package/assets/skills/comet/rules/comet-phase-guard.en.md +1 -1
  9. package/assets/skills/comet/rules/comet-phase-guard.md +1 -1
  10. package/assets/skills/comet/rules/comet-workflow-guard.en.md +2 -2
  11. package/assets/skills/comet/rules/comet-workflow-guard.md +2 -2
  12. package/assets/skills/comet/scripts/comet-archive.mjs +128 -128
  13. package/assets/skills/comet/scripts/comet-entry-runtime.mjs +77 -77
  14. package/assets/skills/comet/scripts/comet-guard.mjs +144 -144
  15. package/assets/skills/comet/scripts/comet-handoff.mjs +107 -107
  16. package/assets/skills/comet/scripts/comet-hook-guard.mjs +128 -128
  17. package/assets/skills/comet/scripts/comet-hook-router.mjs +143 -141
  18. package/assets/skills/comet/scripts/comet-resume-probe.mjs +100 -100
  19. package/assets/skills/comet/scripts/comet-runtime.mjs +194 -193
  20. package/assets/skills/comet/scripts/comet-state.mjs +118 -118
  21. package/assets/skills/comet/scripts/comet-yaml-validate.mjs +95 -95
  22. package/assets/skills/comet-any/SKILL.md +2 -1
  23. package/assets/skills/comet-any/agents/openai.yaml +5 -0
  24. package/assets/skills/comet-archive/SKILL.md +7 -6
  25. package/assets/skills/comet-archive/agents/openai.yaml +5 -0
  26. package/assets/skills/comet-build/SKILL.md +28 -68
  27. package/assets/skills/comet-build/agents/openai.yaml +5 -0
  28. package/assets/skills/comet-classic/SKILL.md +28 -22
  29. package/assets/skills/comet-classic/agents/openai.yaml +3 -0
  30. package/assets/skills/{comet → comet-classic}/reference/auto-transition.md +1 -1
  31. package/assets/skills/{comet → comet-classic}/reference/comet-yaml-fields.md +3 -3
  32. package/assets/skills/{comet → comet-classic}/reference/context-recovery.md +5 -5
  33. package/assets/skills/{comet → comet-classic}/reference/debug-gate.md +2 -2
  34. package/assets/skills/{comet → comet-classic}/reference/decision-point.md +1 -1
  35. package/assets/skills/{comet → comet-classic}/reference/dirty-worktree.md +1 -1
  36. package/assets/skills/{comet → comet-classic}/reference/file-structure.md +1 -1
  37. package/assets/skills/{comet → comet-classic}/reference/intent-frame.md +1 -1
  38. package/assets/skills/{comet → comet-classic}/reference/scripts.md +4 -2
  39. package/assets/skills/{comet → comet-classic}/reference/subagent-dispatch.md +1 -1
  40. package/assets/skills/comet-classic/reference/workspace.md +23 -0
  41. package/assets/skills/comet-design/SKILL.md +7 -6
  42. package/assets/skills/comet-design/agents/openai.yaml +5 -0
  43. package/assets/skills/comet-hotfix/SKILL.md +10 -9
  44. package/assets/skills/comet-hotfix/agents/openai.yaml +5 -0
  45. package/assets/skills/comet-native/SKILL.md +60 -171
  46. package/assets/skills/comet-native/agents/openai.yaml +3 -0
  47. package/assets/skills/comet-native/reference/artifacts.md +24 -89
  48. package/assets/skills/comet-native/reference/clarification.md +54 -29
  49. package/assets/skills/comet-native/reference/commands.md +30 -173
  50. package/assets/skills/comet-native/reference/recovery.md +38 -88
  51. package/assets/skills/comet-native/reference/workspace.md +23 -0
  52. package/assets/skills/comet-native/scripts/comet-native-archive.mjs +184 -152
  53. package/assets/skills/comet-native/scripts/comet-native-doctor.mjs +214 -158
  54. package/assets/skills/comet-native/scripts/comet-native-hook-guard.mjs +132 -123
  55. package/assets/skills/comet-native/scripts/comet-native-init.mjs +134 -124
  56. package/assets/skills/comet-native/scripts/comet-native-new.mjs +165 -158
  57. package/assets/skills/comet-native/scripts/comet-native-next.mjs +183 -152
  58. package/assets/skills/comet-native/scripts/comet-native-root.mjs +152 -147
  59. package/assets/skills/comet-native/scripts/comet-native-runtime.mjs +247 -202
  60. package/assets/skills/comet-native/scripts/comet-native-select.mjs +150 -143
  61. package/assets/skills/comet-native/scripts/comet-native-show.mjs +128 -121
  62. package/assets/skills/comet-native/scripts/comet-native-spec.mjs +133 -151
  63. package/assets/skills/comet-native/scripts/comet-native-status.mjs +147 -140
  64. package/assets/skills/comet-open/SKILL.md +31 -11
  65. package/assets/skills/comet-open/agents/openai.yaml +5 -0
  66. package/assets/skills/comet-tweak/SKILL.md +11 -10
  67. package/assets/skills/comet-tweak/agents/openai.yaml +5 -0
  68. package/assets/skills/comet-verify/SKILL.md +9 -8
  69. package/assets/skills/comet-verify/agents/openai.yaml +5 -0
  70. package/assets/skills-zh/comet/SKILL.md +1 -1
  71. package/assets/skills-zh/comet/agents/openai.yaml +3 -0
  72. package/assets/skills-zh/comet-any/SKILL.md +2 -1
  73. package/assets/skills-zh/comet-any/agents/openai.yaml +5 -0
  74. package/assets/skills-zh/comet-archive/SKILL.md +7 -6
  75. package/assets/skills-zh/comet-archive/agents/openai.yaml +5 -0
  76. package/assets/skills-zh/comet-build/SKILL.md +28 -68
  77. package/assets/skills-zh/comet-build/agents/openai.yaml +5 -0
  78. package/assets/skills-zh/comet-classic/SKILL.md +29 -23
  79. package/assets/skills-zh/comet-classic/agents/openai.yaml +3 -0
  80. package/assets/skills-zh/{comet → comet-classic}/reference/auto-transition.md +1 -1
  81. package/assets/skills-zh/{comet → comet-classic}/reference/comet-yaml-fields.md +3 -3
  82. package/assets/skills-zh/{comet → comet-classic}/reference/context-recovery.md +5 -5
  83. package/assets/skills-zh/{comet → comet-classic}/reference/debug-gate.md +2 -2
  84. package/assets/skills-zh/{comet → comet-classic}/reference/decision-point.md +1 -1
  85. package/assets/skills-zh/{comet → comet-classic}/reference/dirty-worktree.md +1 -1
  86. package/assets/skills-zh/{comet → comet-classic}/reference/file-structure.md +1 -1
  87. package/assets/skills-zh/{comet → comet-classic}/reference/intent-frame.md +1 -1
  88. package/assets/skills-zh/{comet → comet-classic}/reference/scripts.md +4 -2
  89. package/assets/skills-zh/{comet → comet-classic}/reference/subagent-dispatch.md +1 -1
  90. package/assets/skills-zh/comet-classic/reference/workspace.md +23 -0
  91. package/assets/skills-zh/comet-design/SKILL.md +7 -6
  92. package/assets/skills-zh/comet-design/agents/openai.yaml +5 -0
  93. package/assets/skills-zh/comet-hotfix/SKILL.md +10 -9
  94. package/assets/skills-zh/comet-hotfix/agents/openai.yaml +5 -0
  95. package/assets/skills-zh/comet-native/SKILL.md +60 -171
  96. package/assets/skills-zh/comet-native/agents/openai.yaml +3 -0
  97. package/assets/skills-zh/comet-native/reference/artifacts.md +24 -89
  98. package/assets/skills-zh/comet-native/reference/clarification.md +40 -15
  99. package/assets/skills-zh/comet-native/reference/commands.md +30 -173
  100. package/assets/skills-zh/comet-native/reference/recovery.md +36 -86
  101. package/assets/skills-zh/comet-native/reference/workspace.md +23 -0
  102. package/assets/skills-zh/comet-open/SKILL.md +31 -11
  103. package/assets/skills-zh/comet-open/agents/openai.yaml +5 -0
  104. package/assets/skills-zh/comet-tweak/SKILL.md +11 -10
  105. package/assets/skills-zh/comet-tweak/agents/openai.yaml +5 -0
  106. package/assets/skills-zh/comet-verify/SKILL.md +9 -8
  107. package/assets/skills-zh/comet-verify/agents/openai.yaml +5 -0
  108. package/bin/fast-runtime-router.js +0 -4
  109. package/dist/app/cli/index.js +20 -1
  110. package/dist/app/cli/index.js.map +1 -1
  111. package/dist/app/commands/classic.d.ts.map +1 -1
  112. package/dist/app/commands/classic.js +2 -0
  113. package/dist/app/commands/classic.js.map +1 -1
  114. package/dist/app/commands/doctor.d.ts.map +1 -1
  115. package/dist/app/commands/doctor.js +8 -0
  116. package/dist/app/commands/doctor.js.map +1 -1
  117. package/dist/app/commands/eval.d.ts +9 -2
  118. package/dist/app/commands/eval.d.ts.map +1 -1
  119. package/dist/app/commands/eval.js +293 -48
  120. package/dist/app/commands/eval.js.map +1 -1
  121. package/dist/app/commands/init.d.ts.map +1 -1
  122. package/dist/app/commands/init.js +11 -2
  123. package/dist/app/commands/init.js.map +1 -1
  124. package/dist/app/commands/update.d.ts.map +1 -1
  125. package/dist/app/commands/update.js +13 -3
  126. package/dist/app/commands/update.js.map +1 -1
  127. package/dist/config/repository-layout.json +0 -8
  128. package/dist/domains/bundle/bundle-platform.d.ts.map +1 -1
  129. package/dist/domains/bundle/bundle-platform.js +8 -0
  130. package/dist/domains/bundle/bundle-platform.js.map +1 -1
  131. package/dist/domains/bundle/types.d.ts +1 -1
  132. package/dist/domains/bundle/types.d.ts.map +1 -1
  133. package/dist/domains/comet-classic/classic-cli.d.ts +1 -1
  134. package/dist/domains/comet-classic/classic-cli.d.ts.map +1 -1
  135. package/dist/domains/comet-classic/classic-cli.js +3 -0
  136. package/dist/domains/comet-classic/classic-cli.js.map +1 -1
  137. package/dist/domains/comet-classic/classic-current-change.d.ts.map +1 -1
  138. package/dist/domains/comet-classic/classic-current-change.js +29 -3
  139. package/dist/domains/comet-classic/classic-current-change.js.map +1 -1
  140. package/dist/domains/comet-classic/classic-guard.js +2 -2
  141. package/dist/domains/comet-classic/classic-guard.js.map +1 -1
  142. package/dist/domains/comet-classic/classic-hook-guard.d.ts.map +1 -1
  143. package/dist/domains/comet-classic/classic-hook-guard.js +14 -1
  144. package/dist/domains/comet-classic/classic-hook-guard.js.map +1 -1
  145. package/dist/domains/comet-classic/classic-state-command.d.ts.map +1 -1
  146. package/dist/domains/comet-classic/classic-state-command.js +31 -8
  147. package/dist/domains/comet-classic/classic-state-command.js.map +1 -1
  148. package/dist/domains/comet-classic/classic-workspace-command.d.ts +3 -0
  149. package/dist/domains/comet-classic/classic-workspace-command.d.ts.map +1 -0
  150. package/dist/domains/comet-classic/classic-workspace-command.js +45 -0
  151. package/dist/domains/comet-classic/classic-workspace-command.js.map +1 -0
  152. package/dist/domains/comet-classic/classic-workspace.d.ts +38 -0
  153. package/dist/domains/comet-classic/classic-workspace.d.ts.map +1 -0
  154. package/dist/domains/comet-classic/classic-workspace.js +315 -0
  155. package/dist/domains/comet-classic/classic-workspace.js.map +1 -0
  156. package/dist/domains/comet-entry/hook-adapter.d.ts.map +1 -1
  157. package/dist/domains/comet-entry/hook-adapter.js +3 -0
  158. package/dist/domains/comet-entry/hook-adapter.js.map +1 -1
  159. package/dist/domains/comet-entry/hook-router-entry.d.ts +2 -0
  160. package/dist/domains/comet-entry/hook-router-entry.d.ts.map +1 -1
  161. package/dist/domains/comet-entry/hook-router-entry.js +32 -3
  162. package/dist/domains/comet-entry/hook-router-entry.js.map +1 -1
  163. package/dist/domains/comet-entry/init-workflow.d.ts.map +1 -1
  164. package/dist/domains/comet-entry/init-workflow.js +13 -5
  165. package/dist/domains/comet-entry/init-workflow.js.map +1 -1
  166. package/dist/domains/comet-entry/project-activation.d.ts.map +1 -1
  167. package/dist/domains/comet-entry/project-activation.js +2 -1
  168. package/dist/domains/comet-entry/project-activation.js.map +1 -1
  169. package/dist/domains/comet-entry/resume-probe.d.ts.map +1 -1
  170. package/dist/domains/comet-entry/resume-probe.js +48 -4
  171. package/dist/domains/comet-entry/resume-probe.js.map +1 -1
  172. package/dist/domains/comet-native/index.d.ts +1 -0
  173. package/dist/domains/comet-native/index.d.ts.map +1 -1
  174. package/dist/domains/comet-native/index.js +1 -0
  175. package/dist/domains/comet-native/index.js.map +1 -1
  176. package/dist/domains/comet-native/native-acceptance.d.ts +4 -3
  177. package/dist/domains/comet-native/native-acceptance.d.ts.map +1 -1
  178. package/dist/domains/comet-native/native-acceptance.js +23 -18
  179. package/dist/domains/comet-native/native-acceptance.js.map +1 -1
  180. package/dist/domains/comet-native/native-archive-command.d.ts.map +1 -1
  181. package/dist/domains/comet-native/native-archive-command.js +204 -61
  182. package/dist/domains/comet-native/native-archive-command.js.map +1 -1
  183. package/dist/domains/comet-native/native-archive-transaction.d.ts.map +1 -1
  184. package/dist/domains/comet-native/native-archive-transaction.js +28 -23
  185. package/dist/domains/comet-native/native-archive-transaction.js.map +1 -1
  186. package/dist/domains/comet-native/native-archive.d.ts +1 -0
  187. package/dist/domains/comet-native/native-archive.d.ts.map +1 -1
  188. package/dist/domains/comet-native/native-archive.js +56 -12
  189. package/dist/domains/comet-native/native-archive.js.map +1 -1
  190. package/dist/domains/comet-native/native-bounded-file.d.ts +32 -2
  191. package/dist/domains/comet-native/native-bounded-file.d.ts.map +1 -1
  192. package/dist/domains/comet-native/native-bounded-file.js +87 -6
  193. package/dist/domains/comet-native/native-bounded-file.js.map +1 -1
  194. package/dist/domains/comet-native/native-build-evidence.d.ts.map +1 -1
  195. package/dist/domains/comet-native/native-build-evidence.js +23 -1
  196. package/dist/domains/comet-native/native-build-evidence.js.map +1 -1
  197. package/dist/domains/comet-native/native-change.d.ts +1 -0
  198. package/dist/domains/comet-native/native-change.d.ts.map +1 -1
  199. package/dist/domains/comet-native/native-change.js +96 -8
  200. package/dist/domains/comet-native/native-change.js.map +1 -1
  201. package/dist/domains/comet-native/native-check-executor.d.ts +36 -0
  202. package/dist/domains/comet-native/native-check-executor.d.ts.map +1 -0
  203. package/dist/domains/comet-native/native-check-executor.js +132 -0
  204. package/dist/domains/comet-native/native-check-executor.js.map +1 -0
  205. package/dist/domains/comet-native/native-check-receipt-model.d.ts +9 -0
  206. package/dist/domains/comet-native/native-check-receipt-model.d.ts.map +1 -1
  207. package/dist/domains/comet-native/native-check-receipt-model.js +70 -8
  208. package/dist/domains/comet-native/native-check-receipt-model.js.map +1 -1
  209. package/dist/domains/comet-native/native-check-receipt-storage.d.ts.map +1 -1
  210. package/dist/domains/comet-native/native-check-receipt-storage.js +9 -8
  211. package/dist/domains/comet-native/native-check-receipt-storage.js.map +1 -1
  212. package/dist/domains/comet-native/native-check-receipt.d.ts.map +1 -1
  213. package/dist/domains/comet-native/native-check-receipt.js +123 -60
  214. package/dist/domains/comet-native/native-check-receipt.js.map +1 -1
  215. package/dist/domains/comet-native/native-checkpoint-command.d.ts.map +1 -1
  216. package/dist/domains/comet-native/native-checkpoint-command.js +0 -6
  217. package/dist/domains/comet-native/native-checkpoint-command.js.map +1 -1
  218. package/dist/domains/comet-native/native-checkpoint-storage.d.ts.map +1 -1
  219. package/dist/domains/comet-native/native-checkpoint-storage.js +27 -19
  220. package/dist/domains/comet-native/native-checkpoint-storage.js.map +1 -1
  221. package/dist/domains/comet-native/native-children.d.ts +49 -0
  222. package/dist/domains/comet-native/native-children.d.ts.map +1 -0
  223. package/dist/domains/comet-native/native-children.js +388 -0
  224. package/dist/domains/comet-native/native-children.js.map +1 -0
  225. package/dist/domains/comet-native/native-cli-help.d.ts +6 -0
  226. package/dist/domains/comet-native/native-cli-help.d.ts.map +1 -0
  227. package/dist/domains/comet-native/native-cli-help.js +192 -0
  228. package/dist/domains/comet-native/native-cli-help.js.map +1 -0
  229. package/dist/domains/comet-native/native-cli-shared.d.ts +2 -2
  230. package/dist/domains/comet-native/native-cli-shared.d.ts.map +1 -1
  231. package/dist/domains/comet-native/native-cli-shared.js +46 -24
  232. package/dist/domains/comet-native/native-cli-shared.js.map +1 -1
  233. package/dist/domains/comet-native/native-cli.d.ts.map +1 -1
  234. package/dist/domains/comet-native/native-cli.js +18 -11
  235. package/dist/domains/comet-native/native-cli.js.map +1 -1
  236. package/dist/domains/comet-native/native-config.d.ts +2 -1
  237. package/dist/domains/comet-native/native-config.d.ts.map +1 -1
  238. package/dist/domains/comet-native/native-config.js +2 -1
  239. package/dist/domains/comet-native/native-config.js.map +1 -1
  240. package/dist/domains/comet-native/native-continuation.d.ts.map +1 -1
  241. package/dist/domains/comet-native/native-continuation.js +175 -147
  242. package/dist/domains/comet-native/native-continuation.js.map +1 -1
  243. package/dist/domains/comet-native/native-diagnostics.d.ts.map +1 -1
  244. package/dist/domains/comet-native/native-diagnostics.js +96 -13
  245. package/dist/domains/comet-native/native-diagnostics.js.map +1 -1
  246. package/dist/domains/comet-native/native-doctor-command.d.ts.map +1 -1
  247. package/dist/domains/comet-native/native-doctor-command.js +332 -0
  248. package/dist/domains/comet-native/native-doctor-command.js.map +1 -1
  249. package/dist/domains/comet-native/native-doctor.d.ts +1 -0
  250. package/dist/domains/comet-native/native-doctor.d.ts.map +1 -1
  251. package/dist/domains/comet-native/native-doctor.js +116 -22
  252. package/dist/domains/comet-native/native-doctor.js.map +1 -1
  253. package/dist/domains/comet-native/native-evidence-projection.d.ts +50 -0
  254. package/dist/domains/comet-native/native-evidence-projection.d.ts.map +1 -0
  255. package/dist/domains/comet-native/native-evidence-projection.js +284 -0
  256. package/dist/domains/comet-native/native-evidence-projection.js.map +1 -0
  257. package/dist/domains/comet-native/native-evidence-retention.d.ts.map +1 -1
  258. package/dist/domains/comet-native/native-evidence-retention.js +14 -12
  259. package/dist/domains/comet-native/native-evidence-retention.js.map +1 -1
  260. package/dist/domains/comet-native/native-evidence-storage.d.ts.map +1 -1
  261. package/dist/domains/comet-native/native-evidence-storage.js +14 -11
  262. package/dist/domains/comet-native/native-evidence-storage.js.map +1 -1
  263. package/dist/domains/comet-native/native-findings.d.ts.map +1 -1
  264. package/dist/domains/comet-native/native-findings.js +18 -0
  265. package/dist/domains/comet-native/native-findings.js.map +1 -1
  266. package/dist/domains/comet-native/native-git-provenance.d.ts +16 -0
  267. package/dist/domains/comet-native/native-git-provenance.d.ts.map +1 -0
  268. package/dist/domains/comet-native/native-git-provenance.js +171 -0
  269. package/dist/domains/comet-native/native-git-provenance.js.map +1 -0
  270. package/dist/domains/comet-native/native-hook-guard.d.ts.map +1 -1
  271. package/dist/domains/comet-native/native-hook-guard.js +199 -19
  272. package/dist/domains/comet-native/native-hook-guard.js.map +1 -1
  273. package/dist/domains/comet-native/native-init-command.d.ts.map +1 -1
  274. package/dist/domains/comet-native/native-init-command.js +14 -2
  275. package/dist/domains/comet-native/native-init-command.js.map +1 -1
  276. package/dist/domains/comet-native/native-local-execution.d.ts +25 -0
  277. package/dist/domains/comet-native/native-local-execution.d.ts.map +1 -0
  278. package/dist/domains/comet-native/native-local-execution.js +241 -0
  279. package/dist/domains/comet-native/native-local-execution.js.map +1 -0
  280. package/dist/domains/comet-native/native-lock.d.ts +1 -1
  281. package/dist/domains/comet-native/native-lock.d.ts.map +1 -1
  282. package/dist/domains/comet-native/native-lock.js +9 -9
  283. package/dist/domains/comet-native/native-lock.js.map +1 -1
  284. package/dist/domains/comet-native/native-loop-runtime.d.ts +63 -0
  285. package/dist/domains/comet-native/native-loop-runtime.d.ts.map +1 -0
  286. package/dist/domains/comet-native/native-loop-runtime.js +580 -0
  287. package/dist/domains/comet-native/native-loop-runtime.js.map +1 -0
  288. package/dist/domains/comet-native/native-mutation-lock.d.ts +2 -0
  289. package/dist/domains/comet-native/native-mutation-lock.d.ts.map +1 -1
  290. package/dist/domains/comet-native/native-mutation-lock.js +23 -2
  291. package/dist/domains/comet-native/native-mutation-lock.js.map +1 -1
  292. package/dist/domains/comet-native/native-new-command.d.ts.map +1 -1
  293. package/dist/domains/comet-native/native-new-command.js +27 -20
  294. package/dist/domains/comet-native/native-new-command.js.map +1 -1
  295. package/dist/domains/comet-native/native-next-command.d.ts.map +1 -1
  296. package/dist/domains/comet-native/native-next-command.js +193 -62
  297. package/dist/domains/comet-native/native-next-command.js.map +1 -1
  298. package/dist/domains/comet-native/native-paths.d.ts +18 -0
  299. package/dist/domains/comet-native/native-paths.d.ts.map +1 -1
  300. package/dist/domains/comet-native/native-paths.js +125 -13
  301. package/dist/domains/comet-native/native-paths.js.map +1 -1
  302. package/dist/domains/comet-native/native-portable-acceptance.d.ts +28 -0
  303. package/dist/domains/comet-native/native-portable-acceptance.d.ts.map +1 -0
  304. package/dist/domains/comet-native/native-portable-acceptance.js +73 -0
  305. package/dist/domains/comet-native/native-portable-acceptance.js.map +1 -0
  306. package/dist/domains/comet-native/native-portable-archive.d.ts +37 -0
  307. package/dist/domains/comet-native/native-portable-archive.d.ts.map +1 -0
  308. package/dist/domains/comet-native/native-portable-archive.js +492 -0
  309. package/dist/domains/comet-native/native-portable-archive.js.map +1 -0
  310. package/dist/domains/comet-native/native-portable-continuation.d.ts +30 -0
  311. package/dist/domains/comet-native/native-portable-continuation.d.ts.map +1 -0
  312. package/dist/domains/comet-native/native-portable-continuation.js +403 -0
  313. package/dist/domains/comet-native/native-portable-continuation.js.map +1 -0
  314. package/dist/domains/comet-native/native-portable-migration-runtime.d.ts +9 -0
  315. package/dist/domains/comet-native/native-portable-migration-runtime.d.ts.map +1 -0
  316. package/dist/domains/comet-native/native-portable-migration-runtime.js +213 -0
  317. package/dist/domains/comet-native/native-portable-migration-runtime.js.map +1 -0
  318. package/dist/domains/comet-native/native-portable-migration.d.ts +42 -0
  319. package/dist/domains/comet-native/native-portable-migration.d.ts.map +1 -0
  320. package/dist/domains/comet-native/native-portable-migration.js +212 -0
  321. package/dist/domains/comet-native/native-portable-migration.js.map +1 -0
  322. package/dist/domains/comet-native/native-portable-recovery.d.ts +15 -0
  323. package/dist/domains/comet-native/native-portable-recovery.d.ts.map +1 -0
  324. package/dist/domains/comet-native/native-portable-recovery.js +203 -0
  325. package/dist/domains/comet-native/native-portable-recovery.js.map +1 -0
  326. package/dist/domains/comet-native/native-portable-runtime.d.ts +155 -0
  327. package/dist/domains/comet-native/native-portable-runtime.d.ts.map +1 -0
  328. package/dist/domains/comet-native/native-portable-runtime.js +1550 -0
  329. package/dist/domains/comet-native/native-portable-runtime.js.map +1 -0
  330. package/dist/domains/comet-native/native-portable-state.d.ts +30 -0
  331. package/dist/domains/comet-native/native-portable-state.d.ts.map +1 -0
  332. package/dist/domains/comet-native/native-portable-state.js +701 -0
  333. package/dist/domains/comet-native/native-portable-state.js.map +1 -0
  334. package/dist/domains/comet-native/native-portable-status.d.ts +65 -0
  335. package/dist/domains/comet-native/native-portable-status.d.ts.map +1 -0
  336. package/dist/domains/comet-native/native-portable-status.js +133 -0
  337. package/dist/domains/comet-native/native-portable-status.js.map +1 -0
  338. package/dist/domains/comet-native/native-portable-text.d.ts +8 -0
  339. package/dist/domains/comet-native/native-portable-text.d.ts.map +1 -0
  340. package/dist/domains/comet-native/native-portable-text.js +28 -0
  341. package/dist/domains/comet-native/native-portable-text.js.map +1 -0
  342. package/dist/domains/comet-native/native-portable-transactions.d.ts +41 -0
  343. package/dist/domains/comet-native/native-portable-transactions.d.ts.map +1 -0
  344. package/dist/domains/comet-native/native-portable-transactions.js +204 -0
  345. package/dist/domains/comet-native/native-portable-transactions.js.map +1 -0
  346. package/dist/domains/comet-native/native-portable-types.d.ts +166 -0
  347. package/dist/domains/comet-native/native-portable-types.d.ts.map +1 -0
  348. package/dist/domains/comet-native/native-portable-types.js +18 -0
  349. package/dist/domains/comet-native/native-portable-types.js.map +1 -0
  350. package/dist/domains/comet-native/native-repair-integration.js +5 -5
  351. package/dist/domains/comet-native/native-repair-integration.js.map +1 -1
  352. package/dist/domains/comet-native/native-resume-view.d.ts.map +1 -1
  353. package/dist/domains/comet-native/native-resume-view.js +1 -5
  354. package/dist/domains/comet-native/native-resume-view.js.map +1 -1
  355. package/dist/domains/comet-native/native-root-move.d.ts.map +1 -1
  356. package/dist/domains/comet-native/native-root-move.js +4 -20
  357. package/dist/domains/comet-native/native-root-move.js.map +1 -1
  358. package/dist/domains/comet-native/native-run-consistency.d.ts.map +1 -1
  359. package/dist/domains/comet-native/native-run-consistency.js +8 -12
  360. package/dist/domains/comet-native/native-run-consistency.js.map +1 -1
  361. package/dist/domains/comet-native/native-run-store.js +4 -4
  362. package/dist/domains/comet-native/native-run-store.js.map +1 -1
  363. package/dist/domains/comet-native/native-runner-input.d.ts +160 -0
  364. package/dist/domains/comet-native/native-runner-input.d.ts.map +1 -0
  365. package/dist/domains/comet-native/native-runner-input.js +347 -0
  366. package/dist/domains/comet-native/native-runner-input.js.map +1 -0
  367. package/dist/domains/comet-native/native-runner-protocol.d.ts +40 -0
  368. package/dist/domains/comet-native/native-runner-protocol.d.ts.map +1 -0
  369. package/dist/domains/comet-native/native-runner-protocol.js +55 -0
  370. package/dist/domains/comet-native/native-runner-protocol.js.map +1 -0
  371. package/dist/domains/comet-native/native-runtime-package.d.ts +3 -0
  372. package/dist/domains/comet-native/native-runtime-package.d.ts.map +1 -1
  373. package/dist/domains/comet-native/native-runtime-package.js +5 -1
  374. package/dist/domains/comet-native/native-runtime-package.js.map +1 -1
  375. package/dist/domains/comet-native/native-schema-migration.d.ts.map +1 -1
  376. package/dist/domains/comet-native/native-schema-migration.js +30 -27
  377. package/dist/domains/comet-native/native-schema-migration.js.map +1 -1
  378. package/dist/domains/comet-native/native-select-command.d.ts.map +1 -1
  379. package/dist/domains/comet-native/native-select-command.js +8 -4
  380. package/dist/domains/comet-native/native-select-command.js.map +1 -1
  381. package/dist/domains/comet-native/native-selection.d.ts.map +1 -1
  382. package/dist/domains/comet-native/native-selection.js +11 -2
  383. package/dist/domains/comet-native/native-selection.js.map +1 -1
  384. package/dist/domains/comet-native/native-show-command.d.ts.map +1 -1
  385. package/dist/domains/comet-native/native-show-command.js +37 -0
  386. package/dist/domains/comet-native/native-show-command.js.map +1 -1
  387. package/dist/domains/comet-native/native-snapshot.d.ts.map +1 -1
  388. package/dist/domains/comet-native/native-snapshot.js +13 -10
  389. package/dist/domains/comet-native/native-snapshot.js.map +1 -1
  390. package/dist/domains/comet-native/native-spec-command.d.ts.map +1 -1
  391. package/dist/domains/comet-native/native-spec-command.js +14 -28
  392. package/dist/domains/comet-native/native-spec-command.js.map +1 -1
  393. package/dist/domains/comet-native/native-specs.js +3 -3
  394. package/dist/domains/comet-native/native-specs.js.map +1 -1
  395. package/dist/domains/comet-native/native-status-command.d.ts.map +1 -1
  396. package/dist/domains/comet-native/native-status-command.js +7 -17
  397. package/dist/domains/comet-native/native-status-command.js.map +1 -1
  398. package/dist/domains/comet-native/native-status-discovery.d.ts +51 -0
  399. package/dist/domains/comet-native/native-status-discovery.d.ts.map +1 -0
  400. package/dist/domains/comet-native/native-status-discovery.js +299 -0
  401. package/dist/domains/comet-native/native-status-discovery.js.map +1 -0
  402. package/dist/domains/comet-native/native-trajectory-recovery.d.ts.map +1 -1
  403. package/dist/domains/comet-native/native-trajectory-recovery.js +6 -9
  404. package/dist/domains/comet-native/native-trajectory-recovery.js.map +1 -1
  405. package/dist/domains/comet-native/native-transaction.d.ts.map +1 -1
  406. package/dist/domains/comet-native/native-transaction.js +31 -20
  407. package/dist/domains/comet-native/native-transaction.js.map +1 -1
  408. package/dist/domains/comet-native/native-transition-evidence.d.ts.map +1 -1
  409. package/dist/domains/comet-native/native-transition-evidence.js +1 -0
  410. package/dist/domains/comet-native/native-transition-evidence.js.map +1 -1
  411. package/dist/domains/comet-native/native-transition-journal.d.ts.map +1 -1
  412. package/dist/domains/comet-native/native-transition-journal.js +71 -22
  413. package/dist/domains/comet-native/native-transition-journal.js.map +1 -1
  414. package/dist/domains/comet-native/native-transitions.d.ts.map +1 -1
  415. package/dist/domains/comet-native/native-transitions.js +167 -15
  416. package/dist/domains/comet-native/native-transitions.js.map +1 -1
  417. package/dist/domains/comet-native/native-types.d.ts +36 -1
  418. package/dist/domains/comet-native/native-types.d.ts.map +1 -1
  419. package/dist/domains/comet-native/native-verification-receipt-runtime.d.ts.map +1 -1
  420. package/dist/domains/comet-native/native-verification-receipt-runtime.js +18 -100
  421. package/dist/domains/comet-native/native-verification-receipt-runtime.js.map +1 -1
  422. package/dist/domains/comet-native/native-verification-report-v2.d.ts +12 -0
  423. package/dist/domains/comet-native/native-verification-report-v2.d.ts.map +1 -0
  424. package/dist/domains/comet-native/native-verification-report-v2.js +141 -0
  425. package/dist/domains/comet-native/native-verification-report-v2.js.map +1 -0
  426. package/dist/domains/comet-native/native-verification-runtime.d.ts.map +1 -1
  427. package/dist/domains/comet-native/native-verification-runtime.js +7 -2
  428. package/dist/domains/comet-native/native-verification-runtime.js.map +1 -1
  429. package/dist/domains/comet-native/native-verification-scope.d.ts +23 -0
  430. package/dist/domains/comet-native/native-verification-scope.d.ts.map +1 -1
  431. package/dist/domains/comet-native/native-verification-scope.js +106 -19
  432. package/dist/domains/comet-native/native-verification-scope.js.map +1 -1
  433. package/dist/domains/comet-native/native-verifier-protocol.d.ts +49 -0
  434. package/dist/domains/comet-native/native-verifier-protocol.d.ts.map +1 -0
  435. package/dist/domains/comet-native/native-verifier-protocol.js +157 -0
  436. package/dist/domains/comet-native/native-verifier-protocol.js.map +1 -0
  437. package/dist/domains/comet-native/native-workspace-finish.d.ts +61 -0
  438. package/dist/domains/comet-native/native-workspace-finish.d.ts.map +1 -0
  439. package/dist/domains/comet-native/native-workspace-finish.js +332 -0
  440. package/dist/domains/comet-native/native-workspace-finish.js.map +1 -0
  441. package/dist/domains/comet-native/native-workspace-preparation.d.ts +32 -0
  442. package/dist/domains/comet-native/native-workspace-preparation.d.ts.map +1 -0
  443. package/dist/domains/comet-native/native-workspace-preparation.js +324 -0
  444. package/dist/domains/comet-native/native-workspace-preparation.js.map +1 -0
  445. package/dist/domains/comet-native/native-workspace.d.ts +9 -1
  446. package/dist/domains/comet-native/native-workspace.d.ts.map +1 -1
  447. package/dist/domains/comet-native/native-workspace.js +109 -11
  448. package/dist/domains/comet-native/native-workspace.js.map +1 -1
  449. package/dist/domains/dashboard/collector.d.ts +1 -1
  450. package/dist/domains/dashboard/collector.d.ts.map +1 -1
  451. package/dist/domains/dashboard/collector.js +102 -87
  452. package/dist/domains/dashboard/collector.js.map +1 -1
  453. package/dist/domains/dashboard/native-adapter.d.ts +155 -126
  454. package/dist/domains/dashboard/native-adapter.d.ts.map +1 -1
  455. package/dist/domains/dashboard/native-adapter.js +165 -268
  456. package/dist/domains/dashboard/native-adapter.js.map +1 -1
  457. package/dist/domains/dashboard/native-collector.d.ts +5 -4
  458. package/dist/domains/dashboard/native-collector.d.ts.map +1 -1
  459. package/dist/domains/dashboard/native-collector.js +506 -439
  460. package/dist/domains/dashboard/native-collector.js.map +1 -1
  461. package/dist/domains/dashboard/native-legacy-adapter.d.ts +37 -0
  462. package/dist/domains/dashboard/native-legacy-adapter.d.ts.map +1 -0
  463. package/dist/domains/dashboard/native-legacy-adapter.js +127 -0
  464. package/dist/domains/dashboard/native-legacy-adapter.js.map +1 -0
  465. package/dist/domains/dashboard/server.d.ts.map +1 -1
  466. package/dist/domains/dashboard/server.js +5 -3
  467. package/dist/domains/dashboard/server.js.map +1 -1
  468. package/dist/domains/dashboard/types.d.ts +5 -0
  469. package/dist/domains/dashboard/types.d.ts.map +1 -1
  470. package/dist/domains/dashboard/web/assets/abnfDiagram-N423BO3Z-B5cFR6hS.js +1 -0
  471. package/dist/domains/dashboard/web/assets/arc-CVQDtMsu.js +1 -0
  472. package/dist/domains/dashboard/web/assets/architectureDiagram-T3A2C74G-C0v0kULK.js +36 -0
  473. package/dist/domains/dashboard/web/assets/blockDiagram-VBNYF7ZC-DpHLuxu4.js +132 -0
  474. package/dist/domains/dashboard/web/assets/{c4Diagram-AAUBKEIU-Ca5zba3N.js → c4Diagram-5PPSVZJV-BHnFxUs1.js} +1 -1
  475. package/dist/domains/dashboard/web/assets/channel-eM0x3m1G.js +1 -0
  476. package/dist/domains/dashboard/web/assets/{chunk-ND2GUHAM-Bj6HWRwi.js → chunk-2GRJ4B5K-CGC9gh5n.js} +1 -1
  477. package/dist/domains/dashboard/web/assets/{chunk-QZHKN3VN-B1kXfg5N.js → chunk-2Q5K7J3B-l4puLU_O.js} +1 -1
  478. package/dist/domains/dashboard/web/assets/chunk-5RXB4S5H-ls9jQHZc.js +231 -0
  479. package/dist/domains/dashboard/web/assets/{chunk-FMBD7UC4-BAJLB5V6.js → chunk-5VM5RSS4-BVavmTEq.js} +1 -1
  480. package/dist/domains/dashboard/web/assets/chunk-6Q2QTUOP-Dungt2g9.js +88 -0
  481. package/dist/domains/dashboard/web/assets/{chunk-727SXJPM-CwghIMO9.js → chunk-GF5L2VYU-BidXO_ZP.js} +5 -5
  482. package/dist/domains/dashboard/web/assets/{chunk-4BX2VUAB-JYluRsP-.js → chunk-JWPE2WC7-57L5Up5X.js} +1 -1
  483. package/dist/domains/dashboard/web/assets/{chunk-2J33WTMH-CCVPBm0J.js → chunk-KBJHAD2P-C5u5vbwr.js} +1 -1
  484. package/dist/domains/dashboard/web/assets/chunk-RYQCIY6F-CvLz21I2.js +1 -0
  485. package/dist/domains/dashboard/web/assets/{chunk-55IACEB6-DfYm7pN7.js → chunk-XXDRQBXY-D5nJpckG.js} +1 -1
  486. package/dist/domains/dashboard/web/assets/classDiagram-JCYQIIEL-CEFhao2K.js +1 -0
  487. package/dist/domains/dashboard/web/assets/classDiagram-v2-OCEON4UE-CEFhao2K.js +1 -0
  488. package/dist/domains/dashboard/web/assets/{cose-bilkent-S5V4N54A-BBrkuiAU.js → cose-bilkent-JH36ORCC-Dp0JAGyE.js} +1 -1
  489. package/dist/domains/dashboard/web/assets/{cynefin-VYW2F7L2-DlFhgQHt.js → cynefin-VYW2F7L2-EjC54l1g.js} +2 -2
  490. package/dist/domains/dashboard/web/assets/cynefinDiagram-MW4NZA55-CpYrW3w3.js +62 -0
  491. package/dist/domains/dashboard/web/assets/dagre-VZM6K2ZE-h5K39rCk.js +4 -0
  492. package/dist/domains/dashboard/web/assets/demo-BfLsKrIL.js +45 -0
  493. package/dist/domains/dashboard/web/assets/diagram-7IWD3JNH-CKQj9O4H.js +30 -0
  494. package/dist/domains/dashboard/web/assets/{diagram-KO2AKTUF-BQahyP-C.js → diagram-B4RE2ZJO-DXRIYcnW.js} +3 -3
  495. package/dist/domains/dashboard/web/assets/diagram-LBJQPF4R-wWutQYmv.js +24 -0
  496. package/dist/domains/dashboard/web/assets/diagram-Q27KOJAE-EgXLJQYi.js +24 -0
  497. package/dist/domains/dashboard/web/assets/diagram-UB23O5K3-CsQRXDQZ.js +41 -0
  498. package/dist/domains/dashboard/web/assets/ebnfDiagram-BXEA7PRR-gm328Eho.js +1 -0
  499. package/dist/domains/dashboard/web/assets/erDiagram-JOGREHBK-BbbMVUx_.js +85 -0
  500. package/dist/domains/dashboard/web/assets/flowDiagram-UKHOOZJN-1Y7scCns.js +156 -0
  501. package/dist/domains/dashboard/web/assets/ganttDiagram-PKOTCBZU-BAoBqczv.js +292 -0
  502. package/dist/domains/dashboard/web/assets/{gitGraphDiagram-PVQCEYII-DCxvMAUJ.js → gitGraphDiagram-DS77QQ5N-CMUAyEzl.js} +3 -3
  503. package/dist/domains/dashboard/web/assets/{graph--OzhPTMs.js → graph-DOmOIIwC.js} +1 -1
  504. package/dist/domains/dashboard/web/assets/{index-Blit0P5Z.js → index-CCApJ0Yr.js} +1 -1
  505. package/dist/domains/dashboard/web/assets/index-DWdm_XBi.js +257 -0
  506. package/dist/domains/dashboard/web/assets/index-DhhbseFJ.css +1 -0
  507. package/dist/domains/dashboard/web/assets/{infoDiagram-5YYISTIA-9qXEkzDe.js → infoDiagram-6WML65LV-BaCdtYYC.js} +1 -1
  508. package/dist/domains/dashboard/web/assets/{ishikawaDiagram-YF4QCWOH-Dkf5othW.js → ishikawaDiagram-WSZJBQD7-DQQlNnjU.js} +4 -4
  509. package/dist/domains/dashboard/web/assets/{journeyDiagram-JHISSGLW-C2_IkO4s.js → journeyDiagram-NVQOT4AX-DW9wUMqn.js} +1 -1
  510. package/dist/domains/dashboard/web/assets/{kanban-definition-UN3LZRKU-BxWgu4DL.js → kanban-definition-27J2QSJJ-C_ITaZ0q.js} +7 -7
  511. package/dist/domains/dashboard/web/assets/layout-D-LzfAck.js +1 -0
  512. package/dist/domains/dashboard/web/assets/{linear-Dkr6p-Ro.js → linear-DXOo6ZRT.js} +1 -1
  513. package/dist/domains/dashboard/web/assets/map-DxJ2ADlA.js +1 -0
  514. package/dist/domains/dashboard/web/assets/mermaid.core-ol-0NI97.js +306 -0
  515. package/dist/domains/dashboard/web/assets/{mindmap-definition-RKZ34NQL-C9xyxGfa.js → mindmap-definition-FAOFIHXS-CHF_HOLV.js} +1 -1
  516. package/dist/domains/dashboard/web/assets/pegDiagram-VL7TDLO6-wQEvkNF8.js +1 -0
  517. package/dist/domains/dashboard/web/assets/pieDiagram-7S7Q4E2Y-C3H3kiIP.js +39 -0
  518. package/dist/domains/dashboard/web/assets/purify.es-Cz4mVeUR.js +3 -0
  519. package/dist/domains/dashboard/web/assets/{quadrantDiagram-W4KKPZXB-Cy6iOcxc.js → quadrantDiagram-CIZ2JOQS-S0FdVzJb.js} +4 -4
  520. package/dist/domains/dashboard/web/assets/railroadDiagram-AXF67PYL-DCDXDaVg.js +1 -0
  521. package/dist/domains/dashboard/web/assets/{requirementDiagram-4Y6WPE33-B_gZONhx.js → requirementDiagram-LRYGKXZP-BW8J4gO_.js} +1 -1
  522. package/dist/domains/dashboard/web/assets/sankeyDiagram-W5VNT64P-DQQBmLoy.js +40 -0
  523. package/dist/domains/dashboard/web/assets/sequenceDiagram-SI44F4Z6-2tDcli0Y.js +162 -0
  524. package/dist/domains/dashboard/web/assets/sizeCapture-X5ZJPWSS-4sEZLBKb.js +1 -0
  525. package/dist/domains/dashboard/web/assets/stateDiagram-OKZ733FA-OgPb8pyR.js +1 -0
  526. package/dist/domains/dashboard/web/assets/stateDiagram-v2-UEYNNEHI-CiDNPr4P.js +1 -0
  527. package/dist/domains/dashboard/web/assets/swimlanes-SLNWSIFB-DR7lwtxd.js +2 -0
  528. package/dist/domains/dashboard/web/assets/swimlanesDiagram-ULZ7WXOC-DggHFP9q.js +8 -0
  529. package/dist/domains/dashboard/web/assets/{timeline-definition-PNZ67QCA-DvONX2YQ.js → timeline-definition-Z64GVDOM-DNIJVlQV.js} +4 -4
  530. package/dist/domains/dashboard/web/assets/vennDiagram-T6HMQDX7-CClhw4QZ.js +34 -0
  531. package/dist/domains/dashboard/web/assets/{wardleyDiagram-YWT4CUSO-cf39WX2o.js → wardleyDiagram-T6FBY63Y-DzR3aFPS.js} +3 -3
  532. package/dist/domains/dashboard/web/assets/xychartDiagram-ELKLHX3M-DZhq0ZRj.js +7 -0
  533. package/dist/domains/dashboard/web/index.html +16 -16
  534. package/dist/domains/dashboard/workspace.d.ts +24 -0
  535. package/dist/domains/dashboard/workspace.d.ts.map +1 -0
  536. package/dist/domains/dashboard/workspace.js +87 -0
  537. package/dist/domains/dashboard/workspace.js.map +1 -0
  538. package/dist/domains/eval/index.d.ts +3 -0
  539. package/dist/domains/eval/index.d.ts.map +1 -1
  540. package/dist/domains/eval/index.js +3 -0
  541. package/dist/domains/eval/index.js.map +1 -1
  542. package/dist/domains/eval/standalone-context.d.ts +19 -0
  543. package/dist/domains/eval/standalone-context.d.ts.map +1 -0
  544. package/dist/domains/eval/standalone-context.js +107 -0
  545. package/dist/domains/eval/standalone-context.js.map +1 -0
  546. package/dist/domains/eval/standalone-static-collect.d.ts +22 -0
  547. package/dist/domains/eval/standalone-static-collect.d.ts.map +1 -0
  548. package/dist/domains/eval/standalone-static-collect.js +779 -0
  549. package/dist/domains/eval/standalone-static-collect.js.map +1 -0
  550. package/dist/domains/eval/user-environment.d.ts +6 -0
  551. package/dist/domains/eval/user-environment.d.ts.map +1 -0
  552. package/dist/domains/eval/user-environment.js +73 -0
  553. package/dist/domains/eval/user-environment.js.map +1 -0
  554. package/dist/domains/integrations/openspec.d.ts +1 -1
  555. package/dist/domains/integrations/openspec.d.ts.map +1 -1
  556. package/dist/domains/integrations/openspec.js +110 -26
  557. package/dist/domains/integrations/openspec.js.map +1 -1
  558. package/dist/domains/integrations/superpowers.d.ts +5 -1
  559. package/dist/domains/integrations/superpowers.d.ts.map +1 -1
  560. package/dist/domains/integrations/superpowers.js +21 -2
  561. package/dist/domains/integrations/superpowers.js.map +1 -1
  562. package/dist/domains/skill/platform-inspect.d.ts.map +1 -1
  563. package/dist/domains/skill/platform-inspect.js +11 -3
  564. package/dist/domains/skill/platform-inspect.js.map +1 -1
  565. package/dist/domains/skill/platform-install.d.ts +8 -2
  566. package/dist/domains/skill/platform-install.d.ts.map +1 -1
  567. package/dist/domains/skill/platform-install.js +115 -34
  568. package/dist/domains/skill/platform-install.js.map +1 -1
  569. package/dist/domains/skill/project-instructions.d.ts.map +1 -1
  570. package/dist/domains/skill/project-instructions.js +5 -3
  571. package/dist/domains/skill/project-instructions.js.map +1 -1
  572. package/dist/domains/skill/uninstall.d.ts.map +1 -1
  573. package/dist/domains/skill/uninstall.js +69 -15
  574. package/dist/domains/skill/uninstall.js.map +1 -1
  575. package/dist/domains/workflow-contract/hook-write-policy.d.ts +9 -0
  576. package/dist/domains/workflow-contract/hook-write-policy.d.ts.map +1 -0
  577. package/dist/domains/workflow-contract/hook-write-policy.js +52 -0
  578. package/dist/domains/workflow-contract/hook-write-policy.js.map +1 -0
  579. package/dist/domains/workflow-contract/index.d.ts +3 -0
  580. package/dist/domains/workflow-contract/index.d.ts.map +1 -1
  581. package/dist/domains/workflow-contract/index.js +2 -0
  582. package/dist/domains/workflow-contract/index.js.map +1 -1
  583. package/dist/domains/workflow-contract/project-config.d.ts +3 -1
  584. package/dist/domains/workflow-contract/project-config.d.ts.map +1 -1
  585. package/dist/domains/workflow-contract/project-config.js +119 -19
  586. package/dist/domains/workflow-contract/project-config.js.map +1 -1
  587. package/dist/domains/workflow-contract/project-gitignore.d.ts +9 -0
  588. package/dist/domains/workflow-contract/project-gitignore.d.ts.map +1 -0
  589. package/dist/domains/workflow-contract/project-gitignore.js +108 -0
  590. package/dist/domains/workflow-contract/project-gitignore.js.map +1 -0
  591. package/dist/domains/workflow-contract/protected-project-path.d.ts +13 -0
  592. package/dist/domains/workflow-contract/protected-project-path.d.ts.map +1 -1
  593. package/dist/domains/workflow-contract/protected-project-path.js +83 -0
  594. package/dist/domains/workflow-contract/protected-project-path.js.map +1 -1
  595. package/dist/domains/workflow-contract/types.d.ts +4 -0
  596. package/dist/domains/workflow-contract/types.d.ts.map +1 -1
  597. package/dist/platform/install/platforms.d.ts +3 -1
  598. package/dist/platform/install/platforms.d.ts.map +1 -1
  599. package/dist/platform/install/platforms.js +22 -2
  600. package/dist/platform/install/platforms.js.map +1 -1
  601. package/dist/platform/paths/git-worktree.d.ts +8 -1
  602. package/dist/platform/paths/git-worktree.d.ts.map +1 -1
  603. package/dist/platform/paths/git-worktree.js +41 -7
  604. package/dist/platform/paths/git-worktree.js.map +1 -1
  605. package/dist/platform/process/external-command.d.ts +13 -0
  606. package/dist/platform/process/external-command.d.ts.map +1 -0
  607. package/dist/platform/process/external-command.js +45 -0
  608. package/dist/platform/process/external-command.js.map +1 -0
  609. package/dist/platform/process/git.d.ts +12 -0
  610. package/dist/platform/process/git.d.ts.map +1 -0
  611. package/dist/platform/process/git.js +86 -0
  612. package/dist/platform/process/git.js.map +1 -0
  613. package/dist/platform/process/spawn-command.d.ts +8 -0
  614. package/dist/platform/process/spawn-command.d.ts.map +1 -0
  615. package/dist/platform/process/spawn-command.js +113 -0
  616. package/dist/platform/process/spawn-command.js.map +1 -0
  617. package/eval/.env.example +115 -0
  618. package/eval/README.md +237 -14
  619. package/eval/langfuse/.env.example +9 -0
  620. package/eval/langfuse/tests/conftest.py +146 -0
  621. package/eval/langfuse/tests/tasks/test_tasks.py +109 -0
  622. package/eval/langfuse/tests/test_adapter.py +272 -0
  623. package/eval/langsmith/.env.example +16 -0
  624. package/eval/langsmith/tests/conftest.py +10 -1
  625. package/eval/local/tasks/comet-classic-layout-lifecycle/environment/current-comet-package.json +4 -2
  626. package/eval/local/tasks/comet-native-clarification/validation/test_native_clarification.py +9 -21
  627. package/eval/local/tasks/comet-native-clarification-depth/validation/test_native_clarification_depth.py +4 -7
  628. package/eval/local/tasks/comet-native-clarification-modes/validation/test_native_clarification_modes.py +4 -10
  629. package/eval/local/tasks/comet-native-interrupted-transition/instruction.md +1 -1
  630. package/eval/local/tasks/comet-native-interrupted-transition/validation/test_native_interrupted_transition.py +15 -16
  631. package/eval/local/tasks/comet-native-repository-fact/instruction.md +2 -2
  632. package/eval/local/tasks/comet-native-repository-fact/validation/test_native_repository_fact.py +8 -2
  633. package/eval/local/tasks/comet-native-wave-b-decision-resume/instruction.md +8 -4
  634. package/eval/local/tasks/comet-native-wave-b-decision-resume/task.toml +6 -6
  635. package/eval/local/tasks/comet-native-wave-b-decision-resume/validation/test_native_wave_b_decision_resume.py +126 -135
  636. package/eval/local/tasks/comet-native-wave-c-verification-integrity/instruction.md +6 -6
  637. package/eval/local/tasks/comet-native-wave-c-verification-integrity/task.toml +8 -8
  638. package/eval/local/tasks/comet-native-wave-c-verification-integrity/validation/test_native_wave_c_verification_integrity.py +97 -317
  639. package/eval/local/tasks/comet-native-wave-d-stagnation-stop/instruction.md +7 -7
  640. package/eval/local/tasks/comet-native-wave-d-stagnation-stop/task.toml +4 -4
  641. package/eval/local/tasks/comet-native-wave-d-stagnation-stop/validation/test_native_wave_d_stagnation_stop.py +130 -199
  642. package/eval/local/tasks/comet-native-wave-e-parallel-safety/instruction.md +5 -5
  643. package/eval/local/tasks/comet-native-wave-e-parallel-safety/task.toml +8 -8
  644. package/eval/local/tasks/comet-native-wave-e-parallel-safety/validation/test_native_wave_e_parallel_safety.py +135 -239
  645. package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/environment/current-comet-package.json +2 -2
  646. package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/instruction.md +3 -3
  647. package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/task.toml +6 -6
  648. package/eval/local/tasks/comet-native-wave-f-dashboard-readonly/validation/test_native_wave_f_dashboard_readonly.py +81 -439
  649. package/eval/local/tasks/comet-native-workflow/instruction.md +5 -4
  650. package/eval/local/tasks/comet-native-workflow/validation/test_native_workflow.py +145 -502
  651. package/eval/local/tasks/index.yaml +5 -5
  652. package/eval/local/tests/conftest.py +701 -80
  653. package/eval/local/tests/scaffold/fixtures/custom-agent/adapter.yaml +19 -0
  654. package/eval/local/tests/scaffold/test_agents.py +76 -0
  655. package/eval/local/tests/scaffold/test_aligned_comparison.py +70 -0
  656. package/eval/local/tests/scaffold/test_auto_tasks.py +350 -0
  657. package/eval/local/tests/scaffold/test_conftest_helpers.py +267 -18
  658. package/eval/local/tests/scaffold/test_custom_agents.py +174 -0
  659. package/eval/local/tests/scaffold/test_eval_context.py +217 -0
  660. package/eval/local/tests/scaffold/test_execution.py +364 -0
  661. package/eval/local/tests/scaffold/test_inline_tasks.py +56 -0
  662. package/eval/local/tests/scaffold/test_langfuse_trajectory.py +263 -0
  663. package/eval/local/tests/scaffold/test_langsmith_conftest.py +63 -0
  664. package/eval/local/tests/scaffold/test_logging.py +162 -0
  665. package/eval/local/tests/scaffold/test_manifest_tasks.py +76 -0
  666. package/eval/local/tests/scaffold/test_manifests.py +396 -0
  667. package/eval/local/tests/scaffold/test_native_aligned_eval.py +163 -35
  668. package/eval/local/tests/scaffold/test_native_wave_evaluations.py +75 -2054
  669. package/eval/local/tests/scaffold/test_profiles.py +64 -6
  670. package/eval/local/tests/scaffold/test_task_resolution.py +212 -0
  671. package/eval/local/tests/scaffold/test_tasks.py +38 -52
  672. package/eval/local/tests/scaffold/test_utils.py +480 -4
  673. package/eval/local/tests/tasks/test_tasks.py +219 -17
  674. package/eval/local/treatments/comet/comet_native_phase1.yaml +7 -4
  675. package/eval/pyproject.toml +9 -4
  676. package/eval/scaffold/__init__.py +6 -0
  677. package/eval/scaffold/docker/agent-overlay.Dockerfile +41 -0
  678. package/eval/scaffold/python/__init__.py +128 -20
  679. package/eval/scaffold/python/agents.py +239 -0
  680. package/eval/scaffold/python/aligned_comparison.py +156 -27
  681. package/eval/scaffold/python/auto_tasks.py +700 -0
  682. package/eval/scaffold/python/custom_agents.py +172 -0
  683. package/eval/scaffold/python/eval_context.py +236 -0
  684. package/eval/scaffold/python/eval_schema.py +135 -0
  685. package/eval/scaffold/python/execution.py +544 -0
  686. package/eval/scaffold/python/generated_task_cache.py +230 -0
  687. package/eval/scaffold/python/generic_llm_judge.py +41 -14
  688. package/eval/scaffold/python/inline_tasks.py +118 -0
  689. package/eval/scaffold/python/judge_config.py +158 -37
  690. package/eval/scaffold/python/langfuse_adapter.py +1023 -0
  691. package/eval/scaffold/python/llm_judge.py +68 -11
  692. package/eval/scaffold/python/logging.py +157 -17
  693. package/eval/scaffold/python/manifest_tasks.py +69 -0
  694. package/eval/scaffold/python/manifests.py +443 -22
  695. package/eval/scaffold/python/native_eval.py +10 -5
  696. package/eval/scaffold/python/paths.py +11 -1
  697. package/eval/scaffold/python/task_resolution.py +176 -0
  698. package/eval/scaffold/python/tasks.py +17 -4
  699. package/eval/scaffold/python/utils.py +167 -11
  700. package/eval/scaffold/python/validation/core.py +48 -2
  701. package/eval/scaffold/python/validation/generic_rubric.py +9 -0
  702. package/eval/scaffold/python/validation/native_v4.py +249 -0
  703. package/eval/scaffold/python/validation/native_wave.py +4 -5
  704. package/eval/scaffold/python/validation/native_workflow.py +199 -98
  705. package/eval/scaffold/python/validation/rubric.py +130 -2
  706. package/eval/scaffold/shell/agent-runtime-config.sh +88 -0
  707. package/eval/scaffold/shell/docker.sh +534 -33
  708. package/eval/scaffold/shell/run-agent-runtime.sh +49 -0
  709. package/eval/scaffold/shell/run-claude-loop.sh +153 -14
  710. package/eval/schemas/comet.eval/v1alpha1.schema.json +244 -0
  711. package/package.json +9 -3
  712. package/assets/skills/comet-native/scripts/comet-native-check.mjs +0 -179
  713. package/assets/skills/comet-native/scripts/comet-native-checkpoint.mjs +0 -179
  714. package/assets/skills/comet-native/scripts/comet-native-evidence.mjs +0 -151
  715. package/assets/skills/comet-native/scripts/comet-native-receipt.mjs +0 -187
  716. package/dist/domains/comet-native/native-check-entry.d.ts +0 -2
  717. package/dist/domains/comet-native/native-check-entry.d.ts.map +0 -1
  718. package/dist/domains/comet-native/native-check-entry.js +0 -4
  719. package/dist/domains/comet-native/native-check-entry.js.map +0 -1
  720. package/dist/domains/comet-native/native-checkpoint-entry.d.ts +0 -2
  721. package/dist/domains/comet-native/native-checkpoint-entry.d.ts.map +0 -1
  722. package/dist/domains/comet-native/native-checkpoint-entry.js +0 -4
  723. package/dist/domains/comet-native/native-checkpoint-entry.js.map +0 -1
  724. package/dist/domains/comet-native/native-evidence-entry.d.ts +0 -2
  725. package/dist/domains/comet-native/native-evidence-entry.d.ts.map +0 -1
  726. package/dist/domains/comet-native/native-evidence-entry.js +0 -4
  727. package/dist/domains/comet-native/native-evidence-entry.js.map +0 -1
  728. package/dist/domains/comet-native/native-receipt-entry.d.ts +0 -2
  729. package/dist/domains/comet-native/native-receipt-entry.d.ts.map +0 -1
  730. package/dist/domains/comet-native/native-receipt-entry.js +0 -4
  731. package/dist/domains/comet-native/native-receipt-entry.js.map +0 -1
  732. package/dist/domains/dashboard/web/assets/arc-ClrSujVb.js +0 -1
  733. package/dist/domains/dashboard/web/assets/architectureDiagram-3BPJPVTR-B4A_t84A.js +0 -36
  734. package/dist/domains/dashboard/web/assets/blockDiagram-GPEHLZMM-D72u2953.js +0 -132
  735. package/dist/domains/dashboard/web/assets/channel-B_u-_i2t.js +0 -1
  736. package/dist/domains/dashboard/web/assets/chunk-AQP2D5EJ-BA_zgMXZ.js +0 -231
  737. package/dist/domains/dashboard/web/assets/classDiagram-4FO5ZUOK-BpA7BPgX.js +0 -1
  738. package/dist/domains/dashboard/web/assets/classDiagram-v2-Q7XG4LA2-BpA7BPgX.js +0 -1
  739. package/dist/domains/dashboard/web/assets/dagre-BM42HDAG-B0Bk5oiP.js +0 -4
  740. package/dist/domains/dashboard/web/assets/demo-DZYPLZpF.js +0 -25
  741. package/dist/domains/dashboard/web/assets/diagram-2AECGRRQ-tvTUiBA2.js +0 -43
  742. package/dist/domains/dashboard/web/assets/diagram-5GNKFQAL-BZfcK29q.js +0 -10
  743. package/dist/domains/dashboard/web/assets/diagram-LMA3HP47-D5bJnW2C.js +0 -24
  744. package/dist/domains/dashboard/web/assets/diagram-OG6HWLK6-DTwWdzWJ.js +0 -24
  745. package/dist/domains/dashboard/web/assets/erDiagram-TEJ5UH35-BbvrdCp9.js +0 -85
  746. package/dist/domains/dashboard/web/assets/flowDiagram-I6XJVG4X-DgbR7Tcc.js +0 -162
  747. package/dist/domains/dashboard/web/assets/ganttDiagram-6RSMTGT7-wWMOA7F4.js +0 -292
  748. package/dist/domains/dashboard/web/assets/index-Bk1o-4VO.css +0 -1
  749. package/dist/domains/dashboard/web/assets/index-g8vyW0R_.js +0 -257
  750. package/dist/domains/dashboard/web/assets/layout-SsrduOYp.js +0 -1
  751. package/dist/domains/dashboard/web/assets/mermaid.core-DZUcsoG4.js +0 -301
  752. package/dist/domains/dashboard/web/assets/pieDiagram-4H26LBE5-Dn67tdOF.js +0 -30
  753. package/dist/domains/dashboard/web/assets/purify.es-VaSPOPhr.js +0 -3
  754. package/dist/domains/dashboard/web/assets/sankeyDiagram-5OEKKPKP-C3u-4dOk.js +0 -40
  755. package/dist/domains/dashboard/web/assets/sequenceDiagram-3UESZ5HK-Xd3ByTVF.js +0 -162
  756. package/dist/domains/dashboard/web/assets/stateDiagram-AJRCARHV-CJGHL7Up.js +0 -1
  757. package/dist/domains/dashboard/web/assets/stateDiagram-v2-BHNVJYJU-rKvBgr6I.js +0 -1
  758. package/dist/domains/dashboard/web/assets/vennDiagram-CIIHVFJN-B-L4qWy4.js +0 -34
  759. package/dist/domains/dashboard/web/assets/xychartDiagram-2RQKCTM6-Dk1vcm3u.js +0 -7
  760. /package/assets/skills/{comet → comet-classic}/reference/classic-layout.md +0 -0
  761. /package/assets/skills-zh/{comet → comet-classic}/reference/classic-layout.md +0 -0
@@ -0,0 +1,113 @@
1
+ import { spawn } from 'node:child_process';
2
+ import { existsSync } from 'node:fs';
3
+ import path from 'node:path';
4
+ const WINDOWS_SHIM_EXTENSIONS = new Set(['.bat', '.cmd', '.ps1']);
5
+ const WINDOWS_BATCH_EXTENSIONS = new Set(['.bat', '.cmd']);
6
+ const WINDOWS_BATCH_UNSAFE_ARGUMENT = /[&|<>()^%"!\r\n]/u;
7
+ const WINDOWS_POWERSHELL_SCRIPT = [
8
+ "$ProgressPreference = 'SilentlyContinue'",
9
+ '$encoded = $env:COMET_COMMAND_PAYLOAD',
10
+ 'Remove-Item Env:COMET_COMMAND_PAYLOAD -ErrorAction SilentlyContinue',
11
+ '$json = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($encoded))',
12
+ '$payload = ConvertFrom-Json $json',
13
+ '$commandArgs = @($payload.arguments)',
14
+ '& $payload.command @commandArgs',
15
+ 'if ($null -eq $LASTEXITCODE) { if ($?) { exit 0 } else { exit 1 } }',
16
+ 'exit $LASTEXITCODE',
17
+ ].join('; ');
18
+ function environmentValue(env, name) {
19
+ const expected = name.toLowerCase();
20
+ return Object.entries(env).find(([key, value]) => key.toLowerCase() === expected && value !== undefined)?.[1];
21
+ }
22
+ function windowsExecutableExtensions(env) {
23
+ const configured = (environmentValue(env, 'PATHEXT') ?? '.COM;.EXE;.BAT;.CMD')
24
+ .split(';')
25
+ .map((extension) => extension.trim().toLowerCase())
26
+ .filter(Boolean);
27
+ return [...new Set([...configured, '.ps1'])];
28
+ }
29
+ function windowsCommandCandidates(command, env, cwd) {
30
+ const hasPath = path.win32.isAbsolute(command) || /[\\/]/u.test(command);
31
+ const directories = hasPath
32
+ ? ['']
33
+ : (environmentValue(env, 'PATH') ?? '')
34
+ .split(path.delimiter)
35
+ .map((directory) => directory.trim().replace(/^"(.*)"$/u, '$1'))
36
+ .filter(Boolean);
37
+ const extension = path.win32.extname(command);
38
+ const names = extension
39
+ ? [command]
40
+ : windowsExecutableExtensions(env).map((candidate) => `${command}${candidate}`);
41
+ return directories.flatMap((directory) => names.map((name) => (directory ? path.join(directory, name) : path.resolve(cwd, name))));
42
+ }
43
+ function resolveWindowsCommand(command, env, cwd) {
44
+ return (windowsCommandCandidates(command, env, cwd).find((candidate) => existsSync(candidate)) ??
45
+ command);
46
+ }
47
+ function powershellExecutable(env) {
48
+ const systemRoot = environmentValue(env, 'SystemRoot');
49
+ if (systemRoot) {
50
+ const bundled = path.join(systemRoot, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe');
51
+ if (existsSync(bundled))
52
+ return bundled;
53
+ }
54
+ return 'powershell.exe';
55
+ }
56
+ function assertSafeWindowsBatchArguments(args) {
57
+ if (args.some((argument) => WINDOWS_BATCH_UNSAFE_ARGUMENT.test(argument))) {
58
+ throw new Error('Windows batch check arguments must not contain shell syntax');
59
+ }
60
+ }
61
+ function spawnWindowsShim(command, args, options) {
62
+ const payload = Buffer.from(JSON.stringify({ command, arguments: [...args] }), 'utf8').toString('base64');
63
+ const encodedScript = Buffer.from(WINDOWS_POWERSHELL_SCRIPT, 'utf16le').toString('base64');
64
+ return spawn(powershellExecutable(options.env), [
65
+ '-NoLogo',
66
+ '-NoProfile',
67
+ '-NonInteractive',
68
+ '-InputFormat',
69
+ 'None',
70
+ '-OutputFormat',
71
+ 'Text',
72
+ '-ExecutionPolicy',
73
+ 'Bypass',
74
+ '-EncodedCommand',
75
+ encodedScript,
76
+ ], {
77
+ cwd: options.cwd,
78
+ env: { ...options.env, COMET_COMMAND_PAYLOAD: payload },
79
+ shell: false,
80
+ windowsHide: true,
81
+ detached: false,
82
+ stdio: ['ignore', 'pipe', 'pipe'],
83
+ });
84
+ }
85
+ export function spawnCommand(command, args, options) {
86
+ const env = options.env ?? process.env;
87
+ if (process.platform !== 'win32') {
88
+ return spawn(command, [...args], {
89
+ cwd: options.cwd,
90
+ env,
91
+ shell: false,
92
+ windowsHide: true,
93
+ detached: true,
94
+ stdio: ['ignore', 'pipe', 'pipe'],
95
+ });
96
+ }
97
+ const resolved = resolveWindowsCommand(command, env, options.cwd);
98
+ const extension = path.win32.extname(resolved).toLowerCase();
99
+ if (WINDOWS_BATCH_EXTENSIONS.has(extension))
100
+ assertSafeWindowsBatchArguments(args);
101
+ if (WINDOWS_SHIM_EXTENSIONS.has(extension)) {
102
+ return spawnWindowsShim(resolved, args, { cwd: options.cwd, env });
103
+ }
104
+ return spawn(resolved, [...args], {
105
+ cwd: options.cwd,
106
+ env,
107
+ shell: false,
108
+ windowsHide: true,
109
+ detached: false,
110
+ stdio: ['ignore', 'pipe', 'pipe'],
111
+ });
112
+ }
113
+ //# sourceMappingURL=spawn-command.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"spawn-command.js","sourceRoot":"","sources":["../../../platform/process/spawn-command.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAA4B,MAAM,oBAAoB,CAAC;AACrE,OAAO,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AACrC,OAAO,IAAI,MAAM,WAAW,CAAC;AAK7B,MAAM,uBAAuB,GAAG,IAAI,GAAG,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC;AAClE,MAAM,wBAAwB,GAAG,IAAI,GAAG,CAAC,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC;AAC3D,MAAM,6BAA6B,GAAG,mBAAmB,CAAC;AAC1D,MAAM,yBAAyB,GAAG;IAChC,0CAA0C;IAC1C,uCAAuC;IACvC,qEAAqE;IACrE,gFAAgF;IAChF,mCAAmC;IACnC,sCAAsC;IACtC,iCAAiC;IACjC,qEAAqE;IACrE,oBAAoB;CACrB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAEb,SAAS,gBAAgB,CAAC,GAAsB,EAAE,IAAY;IAC5D,MAAM,QAAQ,GAAG,IAAI,CAAC,WAAW,EAAE,CAAC;IACpC,OAAO,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,IAAI,CAC7B,CAAC,CAAC,GAAG,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,WAAW,EAAE,KAAK,QAAQ,IAAI,KAAK,KAAK,SAAS,CACxE,EAAE,CAAC,CAAC,CAAC,CAAC;AACT,CAAC;AAED,SAAS,2BAA2B,CAAC,GAAsB;IACzD,MAAM,UAAU,GAAG,CAAC,gBAAgB,CAAC,GAAG,EAAE,SAAS,CAAC,IAAI,qBAAqB,CAAC;SAC3E,KAAK,CAAC,GAAG,CAAC;SACV,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC;SAClD,MAAM,CAAC,OAAO,CAAC,CAAC;IACnB,OAAO,CAAC,GAAG,IAAI,GAAG,CAAC,CAAC,GAAG,UAAU,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC;AAC/C,CAAC;AAED,SAAS,wBAAwB,CAAC,OAAe,EAAE,GAAsB,EAAE,GAAW;IACpF,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,UAAU,CAAC,OAAO,CAAC,IAAI,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;IACzE,MAAM,WAAW,GAAG,OAAO;QACzB,CAAC,CAAC,CAAC,EAAE,CAAC;QACN,CAAC,CAAC,CAAC,gBAAgB,CAAC,GAAG,EAAE,MAAM,CAAC,IAAI,EAAE,CAAC;aAClC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC;aACrB,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,OAAO,CAAC,WAAW,EAAE,IAAI,CAAC,CAAC;aAC/D,MAAM,CAAC,OAAO,CAAC,CAAC;IACvB,MAAM,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IAC9C,MAAM,KAAK,GAAG,SAAS;QACrB,CAAC,CAAC,CAAC,OAAO,CAAC;QACX,CAAC,CAAC,2BAA2B,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,GAAG,OAAO,GAAG,SAAS,EAAE,CAAC,CAAC;IAClF,OAAO,WAAW,CAAC,OAAO,CAAC,CAAC,SAAS,EAAE,EAAE,CACvC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC,CAAC,CACxF,CAAC;AACJ,CAAC;AAED,SAAS,qBAAqB,CAAC,OAAe,EAAE,GAAsB,EAAE,GAAW;IACjF,OAAO,CACL,wBAAwB,CAAC,OAAO,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QACtF,OAAO,CACR,CAAC;AACJ,CAAC;AAED,SAAS,oBAAoB,CAAC,GAAsB;IAClD,MAAM,UAAU,GAAG,gBAAgB,CAAC,GAAG,EAAE,YAAY,CAAC,CAAC;IACvD,IAAI,UAAU,EAAE,CAAC;QACf,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,CACvB,UAAU,EACV,UAAU,EACV,mBAAmB,EACnB,MAAM,EACN,gBAAgB,CACjB,CAAC;QACF,IAAI,UAAU,CAAC,OAAO,CAAC;YAAE,OAAO,OAAO,CAAC;IAC1C,CAAC;IACD,OAAO,gBAAgB,CAAC;AAC1B,CAAC;AAED,SAAS,+BAA+B,CAAC,IAAuB;IAC9D,IAAI,IAAI,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,6BAA6B,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC;QAC1E,MAAM,IAAI,KAAK,CAAC,6DAA6D,CAAC,CAAC;IACjF,CAAC;AACH,CAAC;AAED,SAAS,gBAAgB,CACvB,OAAe,EACf,IAAuB,EACvB,OAAgD;IAEhD,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,CAAC,QAAQ,CAC7F,QAAQ,CACT,CAAC;IACF,MAAM,aAAa,GAAG,MAAM,CAAC,IAAI,CAAC,yBAAyB,EAAE,SAAS,CAAC,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;IAC3F,OAAO,KAAK,CACV,oBAAoB,CAAC,OAAO,CAAC,GAAG,CAAC,EACjC;QACE,SAAS;QACT,YAAY;QACZ,iBAAiB;QACjB,cAAc;QACd,MAAM;QACN,eAAe;QACf,MAAM;QACN,kBAAkB;QAClB,QAAQ;QACR,iBAAiB;QACjB,aAAa;KACd,EACD;QACE,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,GAAG,EAAE,EAAE,GAAG,OAAO,CAAC,GAAG,EAAE,qBAAqB,EAAE,OAAO,EAAE;QACvD,KAAK,EAAE,KAAK;QACZ,WAAW,EAAE,IAAI;QACjB,QAAQ,EAAE,KAAK;QACf,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;KAClC,CACF,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,YAAY,CAC1B,OAAe,EACf,IAAuB,EACvB,OAAiD;IAEjD,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC;IACvC,IAAI,OAAO,CAAC,QAAQ,KAAK,OAAO,EAAE,CAAC;QACjC,OAAO,KAAK,CAAC,OAAO,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE;YAC/B,GAAG,EAAE,OAAO,CAAC,GAAG;YAChB,GAAG;YACH,KAAK,EAAE,KAAK;YACZ,WAAW,EAAE,IAAI;YACjB,QAAQ,EAAE,IAAI;YACd,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;SAClC,CAAC,CAAC;IACL,CAAC;IACD,MAAM,QAAQ,GAAG,qBAAqB,CAAC,OAAO,EAAE,GAAG,EAAE,OAAO,CAAC,GAAG,CAAC,CAAC;IAClE,MAAM,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,WAAW,EAAE,CAAC;IAC7D,IAAI,wBAAwB,CAAC,GAAG,CAAC,SAAS,CAAC;QAAE,+BAA+B,CAAC,IAAI,CAAC,CAAC;IACnF,IAAI,uBAAuB,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;QAC3C,OAAO,gBAAgB,CAAC,QAAQ,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,OAAO,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,CAAC;IACrE,CAAC;IACD,OAAO,KAAK,CAAC,QAAQ,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE;QAChC,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,GAAG;QACH,KAAK,EAAE,KAAK;QACZ,WAAW,EAAE,IAAI;QACjB,QAAQ,EAAE,KAAK;QACf,KAAK,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC;KAClC,CAAC,CAAC;AACL,CAAC"}
@@ -0,0 +1,115 @@
1
+ # `comet eval` creates this complete template automatically on first startup at:
2
+ # Unix: ~/.comet/eval/.env
3
+ # Windows: %USERPROFILE%\.comet\eval\.env
4
+ # Edit that user-level file as needed. Existing files are never overwritten.
5
+ # Process environment variables take precedence over this file.
6
+
7
+ # Default subject Agent when --agent is not supplied.
8
+ # BENCH_EVAL_AGENT=claude-code
9
+
10
+ # Common subject (Bench) configuration.
11
+ # BENCH_API_KEY=
12
+ # BENCH_BASE_URL=
13
+ # BENCH_MODEL=
14
+
15
+ # Optional independent LLM-as-judge configuration.
16
+ # BENCH_LLM_JUDGE=1
17
+ # BENCH_JUDGE_AGENT=
18
+ # BENCH_JUDGE_API_KEY=
19
+ # BENCH_JUDGE_BASE_URL=
20
+ # BENCH_JUDGE_MODEL=
21
+
22
+ # Provider-specific variables remain supported and override common fallbacks.
23
+ # ANTHROPIC_API_KEY=
24
+
25
+ # Optional Claude Code model override.
26
+ # BENCH_CC_MODEL=
27
+ # BENCH_CC_VERSION=latest
28
+
29
+ # Codex / Qoder / CodeBuddy credentials when selected with --agent.
30
+ # OPENAI_API_KEY=
31
+ # OPENAI_BASE_URL=
32
+ # OPENAI_MODEL=
33
+ # CODEX_API_KEY=
34
+ # CODEX_BASE_URL=
35
+ # CODEX_MODEL=
36
+ # BENCH_CODEX_VERSION=latest
37
+ # QODER_PERSONAL_ACCESS_TOKEN=
38
+ # QODER_BASE_URL=
39
+ # QODER_MODEL=
40
+ # BENCH_QODER_VERSION=latest
41
+ # CODEBUDDY_API_KEY=
42
+ # CODEBUDDY_AUTH_TOKEN=
43
+ # CODEBUDDY_BASE_URL=
44
+ # CODEBUDDY_MODEL=
45
+ # CODEBUDDY_SMALL_FAST_MODEL=
46
+ # CODEBUDDY_BIG_SLOW_MODEL=
47
+ # CODEBUDDY_CODE_SUBAGENT_MODEL=
48
+ # CODEBUDDY_CUSTOM_HEADERS=
49
+ # CODEBUDDY_INTERNET_ENVIRONMENT=
50
+ # BENCH_CODEBUDDY_VERSION=latest
51
+ # BENCH_CODEX_MODEL=
52
+ # BENCH_QODER_MODEL=
53
+ # BENCH_CODEBUDDY_MODEL=
54
+
55
+ # Optional user-installed custom Agent adapter registry.
56
+ # Default: ~/.comet/eval/adapters (Windows: %USERPROFILE%\.comet\eval\adapters)
57
+ # COMET_EVAL_ADAPTERS_DIR=
58
+ # Custom Agent credential/model/base URL names are declared by its adapter.yaml.
59
+ # Add those names here only after installing an adapter, for example:
60
+ # MY_AGENT_API_KEY=
61
+ # MY_AGENT_MODEL=
62
+ # MY_AGENT_BASE_URL=
63
+
64
+ # Optional: path to a custom user-simulator prompt for auto_user workflow evals.
65
+ # Relative paths resolve from eval/. Defaults to eval/simulator-instruction.md.
66
+ # BENCH_SIMULATOR_PROMPT_FILE=simulator-instruction.md
67
+
68
+ # Optional legacy Judge auth token. The common BENCH_JUDGE_API_KEY above is
69
+ # mapped to the selected Agent's native credential when this is not set.
70
+ # BENCH_JUDGE_AUTH_TOKEN=
71
+
72
+ # Claude Code via Anthropic-compatible proxy (BigModel / mimo / OpenRouter etc.).
73
+ # When ANTHROPIC_API_KEY is unset, claude authenticates with these instead.
74
+ # ANTHROPIC_AUTH_TOKEN=
75
+ # ANTHROPIC_BASE_URL=
76
+ # ANTHROPIC_MODEL=
77
+ # ANTHROPIC_DEFAULT_HAIKU_MODEL=
78
+ # ANTHROPIC_DEFAULT_SONNET_MODEL=
79
+ # ANTHROPIC_DEFAULT_OPUS_MODEL=
80
+ # ANTHROPIC_DEFAULT_OPUS_MODEL_NAME=
81
+ # ANTHROPIC_DEFAULT_SONNET_MODEL_NAME=
82
+ # CLAUDE_CODE_SUBAGENT_MODEL=
83
+
84
+ # Eval never writes API keys into published assets, manifests, reports, or the
85
+ # workspace. During a Docker run, Agent-specific config roots are mounted as
86
+ # container-local tmpfs files. Codex config uses env_key, CodeBuddy uses an
87
+ # apiKeyHelper, and Qoder reads its PAT from the isolated process environment.
88
+
89
+ # Required only for eval/langsmith.
90
+ # LANGSMITH_API_KEY=
91
+ # LANGSMITH_TRACING=true
92
+ # LANGSMITH_ENDPOINT=https://api.smith.langchain.com
93
+ # LANGSMITH_PROJECT=comet-skill-eval
94
+ # The Claude Code plugin variables (TRACE_TO_LANGSMITH and CC_LANGSMITH_*) are
95
+ # derived from the LANGSMITH_* settings by eval/langsmith/tests/conftest.py.
96
+ # TRACE_TO_LANGSMITH=false
97
+
98
+ # Optional: name the LangSmith experiment (project) per treatment so CONTROL vs
99
+ # skill-injected runs compare in the experiment view.
100
+ # LANGSMITH_EXPERIMENT=COMET_FULL_040_BETA
101
+
102
+ # Optional trajectory tracing via the official langsmith-tracing Claude Code
103
+ # plugin. When LANGSMITH_TRACING=true, the LangSmith suite auto-builds the
104
+ # plugin into eval/.cache/langsmith-cc-plugin via node:20 and reuses that cache.
105
+ # Do not mount a Windows-host-built node_modules directory into the Linux eval container.
106
+ # Unset = rubric + comparison still logged, just without the nested trajectory.
107
+ # CC_LANGSMITH_PLUGIN_DIR=/abs/path/to/comet/eval/.cache/langsmith-cc-plugin
108
+ # CC_LANGSMITH_PLUGIN_AUTO_BUILD=true
109
+
110
+ # Required only for `comet eval --suite langfuse` run mode. Credentials stay in
111
+ # the process environment and are never written to manifests or reports.
112
+ # LANGFUSE_PUBLIC_KEY=
113
+ # LANGFUSE_SECRET_KEY=
114
+ # LANGFUSE_BASE_URL=https://cloud.langfuse.com
115
+ # LANGFUSE_TRACING_ENVIRONMENT=ci
package/eval/README.md CHANGED
@@ -23,16 +23,24 @@ eval/
23
23
  │ ├── treatments/ # Treatment 配置(注入哪些 Skill)
24
24
  │ ├── tests/ # pytest 测试入口
25
25
  │ └── logs/ # 评估结果日志
26
- └── langsmith/ # LangSmith 评估入口(可选)
26
+ ├── langsmith/ # LangSmith 评估入口(可选)
27
+ └── langfuse/ # Langfuse 评估入口(可选)
27
28
  ```
28
29
 
29
30
  ## 环境配置
30
31
 
32
+ 普通 npm 用户不需要进入 Comet 源码目录。首次运行 `comet eval` 时,CLI 会在用户目录自动创建完整配置模板:
33
+
34
+ - Windows:`%USERPROFILE%\.comet\eval\.env`
35
+ - macOS/Linux:`~/.comet/eval/.env`
36
+
37
+ 模板包含主任务(Bench)、独立 Judge、Claude Code、Codex、Qoder、CodeBuddy、LangSmith 和 Langfuse 的全部可配置参数。CLI 只会创建缺失文件,不会覆盖已有 `.env`;当前进程环境变量优先于文件内容。源码仓库下的 `eval/.env` 仅用于 Comet harness 开发和测试,不是普通用户配置入口。
38
+
31
39
  ### 前置依赖
32
40
 
33
41
  - Python 3.11+
34
42
  - Docker
35
- - Claude Code CLI(`claude` 命令可用)
43
+ - 一个受支持的评估 Agent CLI:Claude Code、Codex、Qoder 或 CodeBuddy(运行时会在 Docker 镜像中准备对应 CLI)
36
44
  - `uv`(推荐)或 pip
37
45
 
38
46
  ### 安装
@@ -44,20 +52,32 @@ uv sync
44
52
 
45
53
  ### 环境变量
46
54
 
47
- 评估运行会先读取当前 shell 的环境变量,也会按需加载 `eval/.env`。录屏或演示时可以不创建 `.env`,或只创建一个完全空的 `.env`,把密钥放在系统环境变量里,避免文件内容出现在画面中。
55
+ 已发布的 `comet eval` 不需要用户拉取源码。评估运行会先读取当前 shell 的环境变量,再按需加载用户目录下、与自定义 Agent 适配器并列的 `~/.comet/eval/.env`(Windows:`%USERPROFILE%\.comet\eval\.env`)。录屏或演示时可以不创建 `.env`,把密钥放在系统环境变量里。
48
56
 
49
- 如需本地文件配置,可复制 `.env.example` 为 `.env` 并填写:
57
+ 如需本地文件配置,可参考 `.env.example`,创建用户级文件:
50
58
 
51
59
  ```bash
52
- cp .env.example .env
60
+ mkdir -p ~/.comet/eval
61
+ cp .env.example ~/.comet/eval/.env
53
62
  ```
54
63
 
55
- > 注意:如果你使用系统环境变量,不要把 `.env.example` 中的 `ANTHROPIC_API_KEY=` 这类空赋值原样保留到 `.env`;空赋值在部分加载路径中可能覆盖已经存在的系统环境变量。
64
+ 进程环境变量优先于用户 `.env`;显式 CLI 或 manifest 配置优先于环境默认值。空值不会覆盖已有的有效进程变量。
56
65
 
57
66
  | 变量 | 必填 | 说明 |
58
67
  | ----------------------------------------------------------------- | ---- | --------------------------------------------------------------------------------------------------- |
59
- | `ANTHROPIC_API_KEY` | ✅\* | Anthropic API 密钥;也可以由系统环境变量提供 |
60
- | `BENCH_CC_MODEL` | ❌ | Claude 模型覆盖(默认用 CLI 配置) |
68
+ | `BENCH_EVAL_AGENT` | ❌ | 未通过 CLI 或 manifest 指定时的默认 Agent |
69
+ | `BENCH_API_KEY` / `BENCH_BASE_URL` / `BENCH_MODEL` | ✅\* | 主任务统一凭据、API 地址和模型;会映射到所选 Agent 的原生配置 |
70
+ | `BENCH_JUDGE_AGENT` / `BENCH_JUDGE_API_KEY` / `BENCH_JUDGE_BASE_URL` / `BENCH_JUDGE_MODEL` | Judge | LLM-as-judge 的独立 Agent、凭据、API 地址和模型;不会继承主任务配置 |
71
+ | `ANTHROPIC_API_KEY` | ✅\* | Claude Code 原生 API 密钥;显式设置时优先于 `BENCH_API_KEY` |
72
+ | `BENCH_CC_MODEL` / `BENCH_CC_VERSION` | ❌ | Claude 模型和 CLI 版本覆盖(默认用 CLI 配置) |
73
+ | `OPENAI_API_KEY` / `CODEX_API_KEY` | Codex | 使用 `--agent codex` 时的 Codex 认证 |
74
+ | `OPENAI_BASE_URL` / `OPENAI_MODEL` / `CODEX_BASE_URL` / `CODEX_MODEL` / `BENCH_CODEX_VERSION` | ❌ | Codex 原生地址、模型和 CLI 版本覆盖 |
75
+ | `QODER_PERSONAL_ACCESS_TOKEN` | Qoder | 使用 `--agent qoder` 时的 Qoder 认证 |
76
+ | `QODER_BASE_URL` / `QODER_MODEL` / `BENCH_QODER_VERSION` | ❌ | Qoder 原生地址、模型和 CLI 版本覆盖 |
77
+ | `CODEBUDDY_API_KEY` / `CODEBUDDY_AUTH_TOKEN` | CodeBuddy | 使用 `--agent codebuddy` 时的 CodeBuddy 认证 |
78
+ | `CODEBUDDY_BASE_URL` / `CODEBUDDY_MODEL` / `BENCH_CODEBUDDY_VERSION` | ❌ | CodeBuddy API endpoint、模型和 CLI 版本覆盖 |
79
+ | `CODEBUDDY_SMALL_FAST_MODEL` / `CODEBUDDY_BIG_SLOW_MODEL` / `CODEBUDDY_CODE_SUBAGENT_MODEL` | ❌ | CodeBuddy 的 fast、reasoning 和 code-subagent 模型覆盖 |
80
+ | `CODEBUDDY_CUSTOM_HEADERS` / `CODEBUDDY_INTERNET_ENVIRONMENT` | ❌ | CodeBuddy 的自定义请求头和联网环境配置 |
61
81
  | `BENCH_SIMULATOR_PROMPT_FILE` | ❌ | 显式覆盖任务级用户模拟器提示词;任务未配置时回退到 `eval/simulator-instruction.md` |
62
82
  | `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_BASE_URL` / `ANTHROPIC_MODEL` | ❌ | 用 Anthropic 兼容代理时的认证与模型配置;设置 `ANTHROPIC_AUTH_TOKEN` 后可不设置 `ANTHROPIC_API_KEY` |
63
83
  | `BENCH_LLM_JUDGE` | ❌ | 设为 `1` 启用 LLM-as-judge 评分 |
@@ -66,8 +86,50 @@ cp .env.example .env
66
86
  | `BENCH_JUDGE_API_KEY` | ❌ | Judge 专用 Anthropic API key;如果设置 `BENCH_JUDGE_AUTH_TOKEN`,优先使用 auth token |
67
87
  | `LANGSMITH_API_KEY` | ❌ | 仅 LangSmith 套件需要 |
68
88
  | `LANGSMITH_PROJECT` / `LANGSMITH_TRACING` | ❌ | LangSmith 套件项目名与追踪开关;Claude Code 插件变量会从这组配置自动派生 |
89
+ | `TRACE_TO_LANGSMITH` | ❌ | 兼容旧配置的 LangSmith 追踪开关 |
90
+ | `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` | Langfuse | `--suite langfuse` 运行模式的项目凭据;只从当前进程读取,不写入报告或 manifest |
91
+ | `LANGFUSE_BASE_URL` / `LANGFUSE_TRACING_ENVIRONMENT` | ❌ | Langfuse 区域地址与 trace 环境标签 |
92
+ | `COMET_EVAL_ADAPTERS_DIR` | ❌ | 覆盖自定义 Agent 适配器注册目录;默认是用户目录下的 `~/.comet/eval/adapters` |
93
+
94
+ `*` 主任务至少需要 `BENCH_API_KEY` 或所选 Agent 的原生凭据之一。Claude Code、Codex 和 CodeBuddy 支持 `BENCH_BASE_URL`;Qoder 使用其官方支持的认证和服务地址配置。
95
+
96
+ 安全边界:Eval 不会把 API key 写入发布包、manifest、报告或 Skill 工作区。Docker 运行 Agent 时,Codex、Qoder、CodeBuddy 使用容器内独立的配置根;Codex 的 `config.toml` 只引用 `env_key`,CodeBuddy 的 `settings.json` 只使用 `apiKeyHelper`,实际密钥只存在于本次容器进程环境中,运行结束后随临时配置目录销毁。
97
+
98
+ `comet eval` 默认使用 `claude-code`。可以通过 CLI 或 manifest 选择执行 Agent;CLI 优先于
99
+ manifest,未指定时回退到 Claude Code:
100
+
101
+ ```bash
102
+ comet eval ./generated-skill/comet/eval.yaml --agent codex
103
+ comet eval ./generated-skill/comet/eval.yaml --agent qoder
104
+ comet eval ./generated-skill/comet/eval.yaml --agent codebuddy
105
+ ```
106
+
107
+ CodeBuddy 的自定义模型使用其 OpenAI-compatible 配置:`CODEBUDDY_API_KEY`、
108
+ `CODEBUDDY_BASE_URL` 和 provider 接受的 `CODEBUDDY_MODEL`,也可以直接用 `--model`
109
+ 覆盖当前运行的模型。宿主机的 `~/.codebuddy/models.json` 只用于注册模型;Eval 不挂载它,
110
+ 而是在容器内生成临时 `settings.json`,通过 `apiKeyHelper` 从进程环境读取凭据。不要把
111
+ Claude Code 的 Anthropic 地址或 Claude 专用模型 ID 直接复用给 CodeBuddy。
112
+
113
+ manifest 可声明默认 Agent:
114
+
115
+ ```yaml
116
+ execution:
117
+ agent: codex
118
+ ```
119
+
120
+ 四种 Agent 共用 subject、auto_user simulator 和可选 Judge 的执行/报告契约;缺失的 token、成本或
121
+ 上下文遥测会在报告中保留为 `N/A`,而不是用估算值补齐。
122
+
123
+ ### Langfuse 套件
124
+
125
+ 使用 `--suite langfuse` 时,local runner 的任务、Agent、Docker、validator 和本地报告保持不变,额外把每个 task/treatment 的核心结果写入 Langfuse,并在结束时 flush:
126
+
127
+ ```bash
128
+ LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... \
129
+ comet eval ./generated-skill/comet/eval.yaml --suite langfuse --html
130
+ ```
69
131
 
70
- `*` 本地 Claude eval 至少需要 `ANTHROPIC_API_KEY` 或 `ANTHROPIC_AUTH_TOKEN` 之一存在于当前进程环境中。
132
+ `comet eval` 会自动选择 Langfuse optional extra。Claude Code 和 Codex 会在隔离的 `.comet/eval/langfuse/plugins/` 缓存中准备带完整性元数据的固定版本官方插件;Qoder 和 CodeBuddy 使用项目级 Stop hook 读取 JSONL transcript,转换失败时只降级详细轨迹,不影响本地评分和核心 trace。`--collect --suite langfuse` 不初始化 SDK、不联网、不下载插件。
71
133
 
72
134
  ## 快速开始
73
135
 
@@ -94,6 +156,132 @@ Eval 报告同时保留 raw set 和 analysis set。Raw set 包含所有 run,
94
156
 
95
157
  如果关键 treatment 的 clean data 不足,报告会输出 `Insufficient clean data` 或 `Inconclusive due to data quality`,而不是给出误导性的胜负结论。
96
158
 
159
+ ### 独立评估任意 Skill
160
+
161
+ 独立评估不依赖 `comet-any`。只要有一个 Skill 目录,就可以直接运行;没有
162
+ `comet/eval.yaml` 时,普通运行会自动生成并缓存 2–4 个任务:
163
+
164
+ ```bash
165
+ comet eval ./my-skill
166
+ comet eval ./my-skill --agent codebuddy --model subject-model
167
+ ```
168
+
169
+ 主模型和 LLM-as-judge 是两套独立配置。Judge 只有在显式启用后才会运行,至少需要自己的
170
+ `BENCH_JUDGE_MODEL` 和 `BENCH_JUDGE_API_KEY`/`BENCH_JUDGE_AUTH_TOKEN`;它可以使用不同的
171
+ Agent、模型和 API 地址:
172
+
173
+ ```bash
174
+ BENCH_LLM_JUDGE=1 \
175
+ BENCH_JUDGE_MODEL=judge-model \
176
+ BENCH_JUDGE_BASE_URL=https://judge.example/v1 \
177
+ BENCH_JUDGE_API_KEY=... \
178
+ comet eval ./my-skill --model subject-model --base-url https://subject.example/v1
179
+ ```
180
+
181
+ 也可以全部通过命令行或 manifest 配置:
182
+
183
+ ```bash
184
+ comet eval ./my-skill \
185
+ --agent codebuddy --model subject-model --base-url https://subject.example/v1 \
186
+ --judge-agent claude-code --judge-model judge-model \
187
+ --judge-base-url https://judge.example/v1
188
+ ```
189
+
190
+ 如果只想检查任务、配置和快照,不执行 Agent、Docker、插件或网络请求,使用:
191
+
192
+ ```bash
193
+ comet eval ./my-skill --collect
194
+ ```
195
+
196
+ ### 扩展自定义 Agent
197
+
198
+ 非预定义的 Agent 不需要修改 Comet 源码,也不需要把适配器放进 npm 包。用户在自己的配置目录
199
+ 中显式注册即可,Eval 不会仅因为某个可执行文件在 `PATH` 中就自动发现它:
200
+
201
+ | 系统 | 默认适配器目录 |
202
+ | --- | --- |
203
+ | macOS/Linux | `~/.comet/eval/adapters/<agent-id>/adapter.yaml` |
204
+ | Windows | `%USERPROFILE%\\.comet\\eval\\adapters\\<agent-id>\\adapter.yaml` |
205
+
206
+ 也可以在用户级 `~/.comet/eval/.env`(Windows 为 `%USERPROFILE%\\.comet\\eval\\.env`)中设置
207
+ `COMET_EVAL_ADAPTERS_DIR`,把整个适配器注册表放到其他位置。`<agent-id>` 必须是小写字母开头、
208
+ 只包含小写字母/数字/连字符、长度 2–32 的标识符。
209
+
210
+ 创建 `<agent-id>/adapter.yaml`,例如:
211
+
212
+ ```yaml
213
+ apiVersion: comet.eval.agent/v1alpha1
214
+ kind: EvalAgentAdapter
215
+ metadata: { id: my-agent, version: 1.0.0 }
216
+ runtime:
217
+ executable: my-agent
218
+ install: { kind: npm, package: my-agent, version: latest }
219
+ credentials: [MY_AGENT_API_KEY]
220
+ modelEnv: MY_AGENT_MODEL
221
+ baseUrlEnv: MY_AGENT_BASE_URL
222
+ capabilities:
223
+ singleTurn: true
224
+ resume: true
225
+ structuredEvents: true
226
+ telemetry: false
227
+ skillInvocationEvidence: true
228
+ ```
229
+
230
+ 字段含义:
231
+
232
+ - `runtime.executable` 是容器内实际启动的命令;npm/pip 包必须暴露这个可执行入口。
233
+ `runtime.install.kind` 可为 `npm`、`pip` 或 `none`;使用 `npm`/`pip` 时 Eval 会在本次 Docker
234
+ 镜像构建中安装对应包,`none` 表示命令已在基础镜像中准备好。
235
+ - `credentials` 只能填写最多两个环境变量名,不要填写密钥值。主 Agent 运行时,只有这些声明的
236
+ 环境变量会被转发到容器;Judge 运行时,`BENCH_JUDGE_API_KEY` 映射到第一个凭据名,
237
+ `BENCH_JUDGE_AUTH_TOKEN` 映射到第二个凭据名(如果声明了第二个)。
238
+ - `modelEnv` 和 `baseUrlEnv` 是可选的环境变量名。主 Agent 可通过这些变量,或 CLI 的
239
+ `--model`/`--base-url`,或 manifest 的 `execution.model`/`execution.baseUrl` 配置;自定义 Agent
240
+ 不会自动把 `BENCH_MODEL`、`BENCH_BASE_URL` 或 `BENCH_API_KEY` 映射到自定义变量。
241
+ - `capabilities` 描述适配器的真实能力。`resume` 用于 `auto_user` 多轮交互;`structuredEvents`
242
+ 要求 CLI 输出 JSON Lines;`skillInvocationEvidence` 表示输出中能提供明确的 Skill 调用事件;
243
+ `telemetry: false` 时报告中的 token/cost 可以是 `N/A`。运行真实评估至少需要
244
+ `singleTurn`、`resume`、`structuredEvents` 和 `skillInvocationEvidence` 为 `true`。
245
+
246
+ 自定义 CLI 必须接受 Eval 生成的调用形态:
247
+
248
+ ```text
249
+ <executable> -p "<prompt>" --output-format stream-json --model <model> [--resume <session-id>]
250
+ ```
251
+
252
+ 并把每条结构化事件写到 stdout。若要让 rubric 认定 Skill 被调用,输出中需要有类似下面的 JSONL
253
+ 事件(普通文件产物或路径推断不算自定义 Agent 的调用证据):
254
+
255
+ ```json
256
+ {"type":"skill_invocation","skill":"my-skill"}
257
+ ```
258
+
259
+ 在用户级 `.env` 中填写适配器声明的变量,然后运行:
260
+
261
+ ```dotenv
262
+ BENCH_EVAL_AGENT=my-agent
263
+ MY_AGENT_API_KEY=your-subject-key
264
+ MY_AGENT_MODEL=your-model
265
+ MY_AGENT_BASE_URL=https://your-provider.example/v1
266
+
267
+ # 如果让它作为独立 Judge:
268
+ # BENCH_LLM_JUDGE=1
269
+ # BENCH_JUDGE_AGENT=my-agent
270
+ # BENCH_JUDGE_API_KEY=your-judge-key
271
+ # BENCH_JUDGE_MODEL=your-judge-model
272
+ # BENCH_JUDGE_BASE_URL=https://your-judge-provider.example/v1
273
+ ```
274
+
275
+ ```bash
276
+ comet eval ./my-skill --collect --agent my-agent
277
+ comet eval ./my-skill --quick --agent my-agent --model your-model
278
+ ```
279
+
280
+ `--collect` 会校验适配器和配置但不会启动 Agent、Docker 或联网;确认通过后再运行真实评估。
281
+ 如果你的 CLI 不能兼容上述参数和 JSONL 事件契约,需要提供一个 wrapper CLI,将它转换为该契约;
282
+ 当前 `adapter.yaml` 不支持自定义参数模板或直接挂载宿主机 Agent 配置目录。适配器文件可以分发给
283
+ 团队成员,但真实凭据必须留在每个人自己的用户级 `.env` 或当前 shell 中。
284
+
97
285
  ### 评估任意 Skill
98
286
 
99
287
  ```bash
@@ -111,23 +299,29 @@ uv run pytest local/tests/tasks/test_tasks.py \
111
299
  --eval-manifest=/path/to/my-skill/comet/eval.yaml -v
112
300
  ```
113
301
 
114
- ## 容器中的 Claude 实验工作区
302
+ 如果 manifest 没有 `evaluation.tasks` 或 `recommendedTasks`,普通运行会基于 Skill 的受限快照自动生成 2–4 个任务,并按 Agent、profile、交互配置和快照 hash 缓存到项目的 `.comet/eval/generated/`。缓存 manifest 会写入生成 hash 和元数据,方便复现;`--collect` 只接受已有缓存,不会在收集阶段调用生成 Agent。需要快速冒烟时显式使用:
115
303
 
116
- 每次实验都会先在宿主机创建一个临时隔离目录,然后把它挂载到 Claude Code 容器的 `/workspace`。Claude 的当前工作目录就是 `/workspace`;双 agent 交互模式还会只读挂载 scaffold shell 到 `/opt/scaffold-shell`。
304
+ ```bash
305
+ comet eval ./my-skill --quick
306
+ ```
307
+
308
+ ## 容器中的 Agent 实验工作区
309
+
310
+ 每次实验都会先在宿主机创建一个临时隔离目录,然后把它挂载到所选 Agent 的 Docker 容器的 `/workspace`。Agent 的当前工作目录就是 `/workspace`;双 agent 交互模式还会只读挂载 scaffold shell 到 `/opt/scaffold-shell`。
117
311
 
118
312
  典型结构如下(以 `COMET_FULL_040_BETA` 为例,展示 0.4.0 beta 的 Comet Skill + 完整 dependency snapshot;实际任务产物会随 task/treatment 变化):
119
313
 
120
314
  ```text
121
- /workspace # Claude Code 的当前工作目录;宿主机临时隔离目录挂载点
315
+ /workspace # 所选 Agent 的当前工作目录;宿主机临时隔离目录挂载点
122
316
  |-- Dockerfile # 当前 task 的 Docker 构建文件,来自 task/environment/
123
317
  |-- requirements.txt # Python 依赖文件;仅 task 提供时存在
124
318
  |-- package.json # Node 依赖文件;仅 task 提供时存在
125
319
  |-- package-lock.json # Node 锁文件;仅 task 提供时存在
126
320
  |-- tsconfig.json # TypeScript 配置;仅 task 提供时存在
127
- |-- CLAUDE.md # 项目级指令;comet-workflow profile 会注入强制 /comet 指令
321
+ |-- CLAUDE.md # Claude 项目级指令;其他 Agent 使用 AGENTS.md
128
322
  |-- .eval-task-prompt.txt # auto_user 模式下的任务提示词临时文件,运行后清理
129
323
  |-- .eval-simulator-prompt.txt # auto_user 模式下的用户模拟器提示词临时文件,运行后清理
130
- |-- .claude # Claude Code 本地配置和 Skill 安装目录
324
+ |-- .claude / .agents / .qoder / .codebuddy # 所选 Agent 的本地配置和 Skill 安装目录
131
325
  | |-- CLAUDE.md # Claude 专用项目指令副本
132
326
  | |-- settings.json # Claude Code settings;包含 Comet PreToolUse hook
133
327
  | `-- skills # 当前 treatment 注入的完整 Skill 包
@@ -229,6 +423,28 @@ uv run pytest local/tests/tasks/test_tasks.py \
229
423
 
230
424
  ## 定义自己的 Task
231
425
 
426
+ 除了把任务注册到 `eval/local/tasks/`,项目也可以在自己的 `comet/eval.yaml` 中声明 inline task 或引用 Skill 包内的 task package。项目任务会和 `recommendedTasks` 合并;同名任务会直接拒绝,避免覆盖内置任务。
427
+
428
+ ```yaml
429
+ evaluation:
430
+ tasks:
431
+ - name: writes-summary
432
+ prompt: Create summary.md with a short result.
433
+ workspace: fixtures/summary
434
+ expect:
435
+ files: [summary.md]
436
+ contains:
437
+ summary.md: ["# Summary"]
438
+ commands:
439
+ - run: test -s summary.md
440
+ timeout: 30
441
+ rubric:
442
+ - The summary is specific and actionable.
443
+ - source: tasks/advanced
444
+ ```
445
+
446
+ `workspace`、`source` 和所有期望产物都必须留在 Skill 包/任务工作区内;`files`、`contains`、`json` 和 `commands` 检查在隔离 Docker 工作区中执行。`--collect` 只做解析和发现,不运行 Agent 或用户命令。
447
+
232
448
  每个任务是 `local/tasks/` 下的一个目录,包含三个部分:
233
449
 
234
450
  ```text
@@ -531,6 +747,13 @@ uv run pytest local/tests/tasks/test_tasks.py --count=5 -v
531
747
  | `--task=<name>` | 指定任务 |
532
748
  | `--treatment=<name>` | 指定 treatment(逗号分隔多个) |
533
749
  | `--profile=<name>` | 覆盖评估 profile |
750
+ | `--agent=<id>` | 主 Agent;可用内置 Agent 或已安装的自定义适配器 |
751
+ | `--model=<name>` | 主模型覆盖 |
752
+ | `--base-url=<url>` | 主模型 API 地址覆盖 |
753
+ | `--judge-agent=<id>` | 独立 Judge Agent |
754
+ | `--judge-model=<name>` | 独立 Judge 模型 |
755
+ | `--judge-base-url=<url>` | 独立 Judge API 地址 |
756
+ | `--collect` | 只解析/发现,不执行工作负载 |
534
757
  | `--skill-path=<path>` | 注入本地 Skill 路径 |
535
758
  | `--skill-name=<name>` | 注入的 Skill 名称 |
536
759
  | `--count=<n>` | 重复运行次数 |
@@ -0,0 +1,9 @@
1
+ # Required for `comet eval --suite langfuse` run mode.
2
+ LANGFUSE_PUBLIC_KEY=
3
+ LANGFUSE_SECRET_KEY=
4
+ LANGFUSE_BASE_URL=https://cloud.langfuse.com
5
+ LANGFUSE_TRACING_ENVIRONMENT=ci
6
+
7
+ # Claude Code and Codex official integrations are enabled by the suite through
8
+ # TRACE_TO_LANGFUSE=true. Install their official plugins in the agent image or
9
+ # project configuration before a run; collect mode never installs or contacts them.