oh-my-knowledge 0.30.0 → 0.32.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (1084) hide show
  1. package/README.md +33 -672
  2. package/README.zh.md +45 -684
  3. package/dist/analysis/coverage-analyzer.d.ts.map +1 -0
  4. package/dist/analysis/coverage-analyzer.js.map +1 -0
  5. package/dist/analysis/failure-clusterer.d.ts.map +1 -0
  6. package/dist/analysis/failure-clusterer.js.map +1 -0
  7. package/dist/analysis/gap-analyzer.d.ts +121 -0
  8. package/dist/analysis/gap-analyzer.d.ts.map +1 -0
  9. package/dist/analysis/gap-analyzer.js +471 -0
  10. package/dist/analysis/gap-analyzer.js.map +1 -0
  11. package/dist/analysis/hedging-classifier.d.ts.map +1 -0
  12. package/dist/analysis/hedging-classifier.js.map +1 -0
  13. package/dist/analysis/report-diagnostics.d.ts +34 -0
  14. package/dist/analysis/report-diagnostics.d.ts.map +1 -0
  15. package/dist/analysis/report-diagnostics.js +753 -0
  16. package/dist/analysis/report-diagnostics.js.map +1 -0
  17. package/dist/analysis/sample-diagnostics.d.ts.map +1 -0
  18. package/dist/analysis/sample-diagnostics.js.map +1 -0
  19. package/dist/analysis/saturation.d.ts.map +1 -0
  20. package/dist/analysis/saturation.js.map +1 -0
  21. package/dist/authoring/evolver.d.ts +102 -0
  22. package/dist/authoring/evolver.d.ts.map +1 -0
  23. package/dist/authoring/evolver.js +681 -0
  24. package/dist/authoring/evolver.js.map +1 -0
  25. package/dist/authoring/generator.d.ts +38 -0
  26. package/dist/authoring/generator.d.ts.map +1 -0
  27. package/dist/authoring/generator.js +720 -0
  28. package/dist/authoring/generator.js.map +1 -0
  29. package/dist/authoring/sample-fixer.d.ts +51 -0
  30. package/dist/authoring/sample-fixer.d.ts.map +1 -0
  31. package/dist/authoring/sample-fixer.js +213 -0
  32. package/dist/authoring/sample-fixer.js.map +1 -0
  33. package/dist/cli/commands/doctor.d.ts +26 -0
  34. package/dist/cli/commands/doctor.d.ts.map +1 -0
  35. package/dist/cli/commands/doctor.js +302 -0
  36. package/dist/cli/commands/doctor.js.map +1 -0
  37. package/dist/cli/commands/eval/gold/compare.d.ts +17 -0
  38. package/dist/cli/commands/eval/gold/compare.d.ts.map +1 -0
  39. package/dist/cli/commands/eval/gold/compare.js +91 -0
  40. package/dist/cli/commands/eval/gold/compare.js.map +1 -0
  41. package/dist/cli/commands/eval/gold/index.d.ts +9 -0
  42. package/dist/cli/commands/eval/gold/index.d.ts.map +1 -0
  43. package/dist/cli/commands/eval/gold/index.js +34 -0
  44. package/dist/cli/commands/eval/gold/index.js.map +1 -0
  45. package/dist/cli/commands/eval/gold/init.d.ts +11 -0
  46. package/dist/cli/commands/eval/gold/init.d.ts.map +1 -0
  47. package/dist/cli/commands/eval/gold/init.js +51 -0
  48. package/dist/cli/commands/eval/gold/init.js.map +1 -0
  49. package/dist/cli/commands/eval/gold/validate.d.ts +12 -0
  50. package/dist/cli/commands/eval/gold/validate.d.ts.map +1 -0
  51. package/dist/cli/commands/eval/gold/validate.js +46 -0
  52. package/dist/cli/commands/eval/gold/validate.js.map +1 -0
  53. package/dist/cli/commands/eval/index.d.ts +54 -0
  54. package/dist/cli/commands/eval/index.d.ts.map +1 -0
  55. package/dist/cli/commands/eval/index.js +465 -0
  56. package/dist/cli/commands/eval/index.js.map +1 -0
  57. package/dist/cli/commands/evolve.d.ts +37 -0
  58. package/dist/cli/commands/evolve.d.ts.map +1 -0
  59. package/dist/cli/commands/evolve.js +283 -0
  60. package/dist/cli/commands/evolve.js.map +1 -0
  61. package/dist/cli/commands/init.d.ts +16 -0
  62. package/dist/cli/commands/init.d.ts.map +1 -0
  63. package/dist/cli/commands/init.js +152 -0
  64. package/dist/cli/commands/init.js.map +1 -0
  65. package/dist/cli/commands/observe/inbox.d.ts +23 -0
  66. package/dist/cli/commands/observe/inbox.d.ts.map +1 -0
  67. package/dist/cli/commands/observe/inbox.js +260 -0
  68. package/dist/cli/commands/observe/inbox.js.map +1 -0
  69. package/dist/cli/commands/observe/index.d.ts +22 -0
  70. package/dist/cli/commands/observe/index.d.ts.map +1 -0
  71. package/dist/cli/commands/observe/index.js +118 -0
  72. package/dist/cli/commands/observe/index.js.map +1 -0
  73. package/dist/cli/commands/observe/ingest.d.ts +13 -0
  74. package/dist/cli/commands/observe/ingest.d.ts.map +1 -0
  75. package/dist/cli/commands/observe/ingest.js +71 -0
  76. package/dist/cli/commands/observe/ingest.js.map +1 -0
  77. package/dist/cli/commands/observe/show.d.ts +13 -0
  78. package/dist/cli/commands/observe/show.d.ts.map +1 -0
  79. package/dist/cli/commands/observe/show.js +49 -0
  80. package/dist/cli/commands/observe/show.js.map +1 -0
  81. package/dist/cli/commands/sample.d.ts +38 -0
  82. package/dist/cli/commands/sample.d.ts.map +1 -0
  83. package/dist/cli/commands/sample.js +487 -0
  84. package/dist/cli/commands/sample.js.map +1 -0
  85. package/dist/cli/commands/studio.d.ts +23 -0
  86. package/dist/cli/commands/studio.d.ts.map +1 -0
  87. package/dist/cli/commands/studio.js +158 -0
  88. package/dist/cli/commands/studio.js.map +1 -0
  89. package/dist/cli/index.d.ts.map +1 -0
  90. package/dist/cli/index.js +29 -0
  91. package/dist/cli/index.js.map +1 -0
  92. package/dist/cli/lib/cli-exit.d.ts +16 -0
  93. package/dist/cli/lib/cli-exit.d.ts.map +1 -0
  94. package/dist/cli/lib/cli-exit.js +20 -0
  95. package/dist/cli/lib/cli-exit.js.map +1 -0
  96. package/dist/cli/lib/cmd-flags.d.ts +224 -0
  97. package/dist/cli/lib/cmd-flags.d.ts.map +1 -0
  98. package/dist/cli/lib/cmd-flags.js +46 -0
  99. package/dist/cli/lib/cmd-flags.js.map +1 -0
  100. package/dist/cli/lib/i18n-dict/common.d.ts +4 -0
  101. package/dist/cli/lib/i18n-dict/common.d.ts.map +1 -0
  102. package/dist/cli/lib/i18n-dict/common.js +67 -0
  103. package/dist/cli/lib/i18n-dict/common.js.map +1 -0
  104. package/dist/cli/lib/i18n-dict/evolve.d.ts +4 -0
  105. package/dist/cli/lib/i18n-dict/evolve.d.ts.map +1 -0
  106. package/dist/cli/lib/i18n-dict/evolve.js +43 -0
  107. package/dist/cli/lib/i18n-dict/evolve.js.map +1 -0
  108. package/dist/cli/lib/i18n-dict/gen.d.ts +4 -0
  109. package/dist/cli/lib/i18n-dict/gen.d.ts.map +1 -0
  110. package/dist/cli/lib/i18n-dict/gen.js +63 -0
  111. package/dist/cli/lib/i18n-dict/gen.js.map +1 -0
  112. package/dist/cli/lib/i18n-dict/help.d.ts +4 -0
  113. package/dist/cli/lib/i18n-dict/help.d.ts.map +1 -0
  114. package/dist/cli/lib/i18n-dict/help.js +281 -0
  115. package/dist/cli/lib/i18n-dict/help.js.map +1 -0
  116. package/dist/cli/lib/i18n-dict/init.d.ts +4 -0
  117. package/dist/cli/lib/i18n-dict/init.d.ts.map +1 -0
  118. package/dist/cli/lib/i18n-dict/init.js +27 -0
  119. package/dist/cli/lib/i18n-dict/init.js.map +1 -0
  120. package/dist/cli/lib/i18n-dict/run.d.ts +4 -0
  121. package/dist/cli/lib/i18n-dict/run.d.ts.map +1 -0
  122. package/dist/cli/lib/i18n-dict/run.js +143 -0
  123. package/dist/cli/lib/i18n-dict/run.js.map +1 -0
  124. package/dist/cli/lib/i18n-dict/types.d.ts +5 -0
  125. package/dist/cli/lib/i18n-dict/types.d.ts.map +1 -0
  126. package/dist/cli/lib/i18n-dict/types.js +2 -0
  127. package/dist/cli/lib/i18n-dict/types.js.map +1 -0
  128. package/dist/cli/lib/i18n-dict.d.ts +63 -0
  129. package/dist/cli/lib/i18n-dict.d.ts.map +1 -0
  130. package/dist/cli/lib/i18n-dict.js +67 -0
  131. package/dist/cli/lib/i18n-dict.js.map +1 -0
  132. package/dist/cli/lib/i18n.d.ts +29 -0
  133. package/dist/cli/lib/i18n.d.ts.map +1 -0
  134. package/dist/cli/lib/i18n.js +58 -0
  135. package/dist/cli/lib/i18n.js.map +1 -0
  136. package/dist/cli/lib/parse-run-config/judge-models.d.ts +24 -0
  137. package/dist/cli/lib/parse-run-config/judge-models.d.ts.map +1 -0
  138. package/dist/cli/lib/parse-run-config/judge-models.js +55 -0
  139. package/dist/cli/lib/parse-run-config/judge-models.js.map +1 -0
  140. package/dist/cli/lib/parse-run-config/samples-discovery.d.ts +19 -0
  141. package/dist/cli/lib/parse-run-config/samples-discovery.d.ts.map +1 -0
  142. package/dist/cli/lib/parse-run-config/samples-discovery.js +53 -0
  143. package/dist/cli/lib/parse-run-config/samples-discovery.js.map +1 -0
  144. package/dist/cli/lib/parse-run-config/variant-resolution.d.ts +18 -0
  145. package/dist/cli/lib/parse-run-config/variant-resolution.d.ts.map +1 -0
  146. package/dist/cli/lib/parse-run-config/variant-resolution.js +59 -0
  147. package/dist/cli/lib/parse-run-config/variant-resolution.js.map +1 -0
  148. package/dist/cli/lib/parse-run-config.d.ts +93 -0
  149. package/dist/cli/lib/parse-run-config.d.ts.map +1 -0
  150. package/dist/cli/lib/parse-run-config.js +157 -0
  151. package/dist/cli/lib/parse-run-config.js.map +1 -0
  152. package/dist/cli/lib/progress.d.ts.map +1 -0
  153. package/dist/cli/lib/progress.js.map +1 -0
  154. package/dist/cli/lib/resolve-skill-input.d.ts +8 -0
  155. package/dist/cli/lib/resolve-skill-input.d.ts.map +1 -0
  156. package/dist/cli/lib/resolve-skill-input.js +38 -0
  157. package/dist/cli/lib/resolve-skill-input.js.map +1 -0
  158. package/dist/cli/lib/run-tally.d.ts +18 -0
  159. package/dist/cli/lib/run-tally.d.ts.map +1 -0
  160. package/dist/cli/lib/run-tally.js.map +1 -0
  161. package/dist/cli/lib/shared.d.ts +12 -0
  162. package/dist/cli/lib/shared.d.ts.map +1 -0
  163. package/dist/cli/lib/shared.js +26 -0
  164. package/dist/cli/lib/shared.js.map +1 -0
  165. package/dist/cli/lib/update-check.d.ts +9 -0
  166. package/dist/cli/lib/update-check.d.ts.map +1 -0
  167. package/dist/cli/lib/update-check.js +113 -0
  168. package/dist/cli/lib/update-check.js.map +1 -0
  169. package/dist/cli/oclif/base-command.d.ts +8 -0
  170. package/dist/cli/oclif/base-command.d.ts.map +1 -0
  171. package/dist/cli/oclif/base-command.js +27 -0
  172. package/dist/cli/oclif/base-command.js.map +1 -0
  173. package/dist/cli/oclif/help.d.ts +16 -0
  174. package/dist/cli/oclif/help.d.ts.map +1 -0
  175. package/dist/cli/oclif/help.js +33 -0
  176. package/dist/cli/oclif/help.js.map +1 -0
  177. package/dist/cli/oclif/i18n.d.ts +16 -0
  178. package/dist/cli/oclif/i18n.d.ts.map +1 -0
  179. package/dist/cli/oclif/i18n.js +62 -0
  180. package/dist/cli/oclif/i18n.js.map +1 -0
  181. package/dist/cli/oclif/parsers.d.ts +10 -0
  182. package/dist/cli/oclif/parsers.d.ts.map +1 -0
  183. package/dist/cli/oclif/parsers.js +102 -0
  184. package/dist/cli/oclif/parsers.js.map +1 -0
  185. package/dist/cli/oclif/projection.d.ts +23 -0
  186. package/dist/cli/oclif/projection.d.ts.map +1 -0
  187. package/dist/cli/oclif/projection.js +52 -0
  188. package/dist/cli/oclif/projection.js.map +1 -0
  189. package/dist/cli/oclif/run.d.ts +2 -0
  190. package/dist/cli/oclif/run.d.ts.map +1 -0
  191. package/dist/cli/oclif/run.js +11 -0
  192. package/dist/cli/oclif/run.js.map +1 -0
  193. package/dist/diagnosis/observe-mapper.d.ts +68 -0
  194. package/dist/diagnosis/observe-mapper.d.ts.map +1 -0
  195. package/dist/diagnosis/observe-mapper.js +276 -0
  196. package/dist/diagnosis/observe-mapper.js.map +1 -0
  197. package/dist/diagnosis/observe-producer.d.ts +9 -0
  198. package/dist/diagnosis/observe-producer.d.ts.map +1 -0
  199. package/dist/diagnosis/observe-producer.js +199 -0
  200. package/dist/diagnosis/observe-producer.js.map +1 -0
  201. package/dist/diagnosis/studio-projection.d.ts +7 -0
  202. package/dist/diagnosis/studio-projection.d.ts.map +1 -0
  203. package/dist/diagnosis/studio-projection.js +84 -0
  204. package/dist/diagnosis/studio-projection.js.map +1 -0
  205. package/dist/diagnosis/types.d.ts +4 -0
  206. package/dist/diagnosis/types.d.ts.map +1 -0
  207. package/dist/diagnosis/types.js +20 -0
  208. package/dist/diagnosis/types.js.map +1 -0
  209. package/dist/doctor/fixer.d.ts +12 -0
  210. package/dist/doctor/fixer.d.ts.map +1 -0
  211. package/dist/doctor/fixer.js +326 -0
  212. package/dist/doctor/fixer.js.map +1 -0
  213. package/dist/doctor/health/builtin-dimensions.d.ts.map +1 -0
  214. package/dist/doctor/health/builtin-dimensions.js.map +1 -0
  215. package/dist/doctor/health/composer.d.ts.map +1 -0
  216. package/dist/doctor/health/composer.js +296 -0
  217. package/dist/doctor/health/composer.js.map +1 -0
  218. package/dist/doctor/health/dimension-registry.d.ts.map +1 -0
  219. package/dist/doctor/health/dimension-registry.js.map +1 -0
  220. package/dist/doctor/health/dimension-spec.d.ts +47 -0
  221. package/dist/doctor/health/dimension-spec.d.ts.map +1 -0
  222. package/dist/doctor/health/dimension-spec.js.map +1 -0
  223. package/dist/doctor/health/parser.d.ts +26 -0
  224. package/dist/doctor/health/parser.d.ts.map +1 -0
  225. package/dist/doctor/health/parser.js +228 -0
  226. package/dist/doctor/health/parser.js.map +1 -0
  227. package/dist/doctor/health/prompt-builder.d.ts +3 -0
  228. package/dist/doctor/health/prompt-builder.d.ts.map +1 -0
  229. package/dist/doctor/health/prompt-builder.js +2 -0
  230. package/dist/doctor/health/prompt-builder.js.map +1 -0
  231. package/dist/doctor/health/register.d.ts.map +1 -0
  232. package/dist/doctor/health/register.js.map +1 -0
  233. package/dist/doctor/index.d.ts.map +1 -0
  234. package/dist/doctor/index.js +246 -0
  235. package/dist/doctor/index.js.map +1 -0
  236. package/dist/doctor/messages.d.ts +21 -0
  237. package/dist/doctor/messages.d.ts.map +1 -0
  238. package/dist/doctor/messages.js +217 -0
  239. package/dist/doctor/messages.js.map +1 -0
  240. package/dist/doctor/preflight.d.ts.map +1 -0
  241. package/dist/doctor/preflight.js.map +1 -0
  242. package/dist/doctor/renderer.d.ts +17 -0
  243. package/dist/doctor/renderer.d.ts.map +1 -0
  244. package/dist/doctor/renderer.js +122 -0
  245. package/dist/doctor/renderer.js.map +1 -0
  246. package/dist/doctor/rules.d.ts.map +1 -0
  247. package/dist/doctor/rules.js +289 -0
  248. package/dist/doctor/rules.js.map +1 -0
  249. package/dist/eval-core/bootstrap.d.ts.map +1 -0
  250. package/dist/eval-core/bootstrap.js.map +1 -0
  251. package/dist/eval-core/cache.d.ts.map +1 -0
  252. package/dist/eval-core/cache.js.map +1 -0
  253. package/dist/eval-core/comparability.d.ts.map +1 -0
  254. package/dist/eval-core/comparability.js.map +1 -0
  255. package/dist/eval-core/dependency-checker.d.ts +37 -0
  256. package/dist/eval-core/dependency-checker.d.ts.map +1 -0
  257. package/dist/eval-core/dependency-checker.js.map +1 -0
  258. package/dist/eval-core/evaluation-execution.d.ts.map +1 -0
  259. package/dist/eval-core/evaluation-execution.js.map +1 -0
  260. package/dist/eval-core/evaluation-job.d.ts.map +1 -0
  261. package/dist/eval-core/evaluation-job.js.map +1 -0
  262. package/dist/eval-core/evaluation-reporting.d.ts.map +1 -0
  263. package/dist/eval-core/evaluation-reporting.js.map +1 -0
  264. package/dist/eval-core/execution-strategy.d.ts.map +1 -0
  265. package/dist/eval-core/execution-strategy.js +165 -0
  266. package/dist/eval-core/execution-strategy.js.map +1 -0
  267. package/dist/eval-core/fact-checker.d.ts.map +1 -0
  268. package/dist/eval-core/fact-checker.js.map +1 -0
  269. package/dist/eval-core/layer-gates.d.ts.map +1 -0
  270. package/dist/eval-core/layer-gates.js.map +1 -0
  271. package/dist/eval-core/mock-hook.cjs +215 -0
  272. package/dist/eval-core/mocks-runtime.d.ts +100 -0
  273. package/dist/eval-core/mocks-runtime.d.ts.map +1 -0
  274. package/dist/eval-core/mocks-runtime.js +559 -0
  275. package/dist/eval-core/mocks-runtime.js.map +1 -0
  276. package/dist/eval-core/schema.d.ts.map +1 -0
  277. package/dist/eval-core/schema.js.map +1 -0
  278. package/dist/eval-core/statistics.d.ts.map +1 -0
  279. package/dist/eval-core/statistics.js.map +1 -0
  280. package/dist/eval-core/task-planner.d.ts.map +1 -0
  281. package/dist/eval-core/task-planner.js.map +1 -0
  282. package/dist/eval-core/verdict.d.ts.map +1 -0
  283. package/dist/eval-core/verdict.js.map +1 -0
  284. package/dist/eval-workflows/batch-evaluation-workflow.d.ts.map +1 -0
  285. package/dist/eval-workflows/batch-evaluation-workflow.js.map +1 -0
  286. package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.d.ts +22 -0
  287. package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.d.ts.map +1 -0
  288. package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.js +67 -0
  289. package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.js.map +1 -0
  290. package/dist/eval-workflows/evaluation-pipeline/report-finalize.d.ts +32 -0
  291. package/dist/eval-workflows/evaluation-pipeline/report-finalize.d.ts.map +1 -0
  292. package/dist/eval-workflows/evaluation-pipeline/report-finalize.js +57 -0
  293. package/dist/eval-workflows/evaluation-pipeline/report-finalize.js.map +1 -0
  294. package/dist/eval-workflows/evaluation-pipeline/run-state.d.ts +60 -0
  295. package/dist/eval-workflows/evaluation-pipeline/run-state.d.ts.map +1 -0
  296. package/dist/eval-workflows/evaluation-pipeline/run-state.js +88 -0
  297. package/dist/eval-workflows/evaluation-pipeline/run-state.js.map +1 -0
  298. package/dist/eval-workflows/evaluation-pipeline/test-set-hash.d.ts +21 -0
  299. package/dist/eval-workflows/evaluation-pipeline/test-set-hash.d.ts.map +1 -0
  300. package/dist/eval-workflows/evaluation-pipeline/test-set-hash.js +46 -0
  301. package/dist/eval-workflows/evaluation-pipeline/test-set-hash.js.map +1 -0
  302. package/dist/eval-workflows/evaluation-pipeline.d.ts +97 -0
  303. package/dist/eval-workflows/evaluation-pipeline.d.ts.map +1 -0
  304. package/dist/eval-workflows/evaluation-pipeline.js +179 -0
  305. package/dist/eval-workflows/evaluation-pipeline.js.map +1 -0
  306. package/dist/eval-workflows/evaluation-preparation.d.ts.map +1 -0
  307. package/dist/eval-workflows/evaluation-preparation.js.map +1 -0
  308. package/dist/eval-workflows/messages.d.ts +4 -0
  309. package/dist/eval-workflows/messages.d.ts.map +1 -0
  310. package/dist/eval-workflows/messages.js +30 -0
  311. package/dist/eval-workflows/messages.js.map +1 -0
  312. package/dist/eval-workflows/run-evaluation.d.ts.map +1 -0
  313. package/dist/eval-workflows/run-evaluation.js +528 -0
  314. package/dist/eval-workflows/run-evaluation.js.map +1 -0
  315. package/dist/executors/anthropic-api.d.ts.map +1 -0
  316. package/dist/executors/anthropic-api.js.map +1 -0
  317. package/dist/executors/claude-cli.d.ts.map +1 -0
  318. package/dist/executors/claude-cli.js +181 -0
  319. package/dist/executors/claude-cli.js.map +1 -0
  320. package/dist/executors/claude-sdk-trace.d.ts.map +1 -0
  321. package/dist/executors/claude-sdk-trace.js.map +1 -0
  322. package/dist/executors/claude-sdk.d.ts.map +1 -0
  323. package/dist/executors/claude-sdk.js.map +1 -0
  324. package/dist/executors/codex-cli-trace.d.ts.map +1 -0
  325. package/dist/executors/codex-cli-trace.js.map +1 -0
  326. package/dist/executors/codex-cli.d.ts.map +1 -0
  327. package/dist/executors/codex-cli.js.map +1 -0
  328. package/dist/executors/codex-sdk.d.ts.map +1 -0
  329. package/dist/executors/codex-sdk.js.map +1 -0
  330. package/dist/executors/gemini.d.ts.map +1 -0
  331. package/dist/executors/gemini.js.map +1 -0
  332. package/dist/executors/index.d.ts.map +1 -0
  333. package/dist/executors/index.js.map +1 -0
  334. package/dist/executors/openai-api.d.ts.map +1 -0
  335. package/dist/executors/openai-api.js.map +1 -0
  336. package/dist/executors/runtime-fingerprint.d.ts.map +1 -0
  337. package/dist/executors/runtime-fingerprint.js.map +1 -0
  338. package/dist/executors/script.d.ts.map +1 -0
  339. package/dist/executors/script.js.map +1 -0
  340. package/dist/executors/shared.d.ts +194 -0
  341. package/dist/executors/shared.d.ts.map +1 -0
  342. package/dist/executors/shared.js +253 -0
  343. package/dist/executors/shared.js.map +1 -0
  344. package/dist/grading/assertions.d.ts.map +1 -0
  345. package/dist/grading/assertions.js.map +1 -0
  346. package/dist/grading/debias-validate.d.ts.map +1 -0
  347. package/dist/grading/debias-validate.js.map +1 -0
  348. package/dist/grading/diagnostic.d.ts.map +1 -0
  349. package/dist/grading/diagnostic.js.map +1 -0
  350. package/dist/grading/gold-cli.d.ts.map +1 -0
  351. package/dist/grading/gold-cli.js.map +1 -0
  352. package/dist/grading/gold-dataset.d.ts.map +1 -0
  353. package/dist/grading/gold-dataset.js.map +1 -0
  354. package/dist/grading/human-gold.d.ts.map +1 -0
  355. package/dist/grading/human-gold.js.map +1 -0
  356. package/dist/grading/index.d.ts.map +1 -0
  357. package/dist/grading/index.js.map +1 -0
  358. package/dist/grading/judge.d.ts.map +1 -0
  359. package/dist/grading/judge.js.map +1 -0
  360. package/dist/grading/layered-scores.d.ts.map +1 -0
  361. package/dist/grading/layered-scores.js.map +1 -0
  362. package/dist/inputs/eval-config.d.ts.map +1 -0
  363. package/dist/inputs/eval-config.js.map +1 -0
  364. package/dist/inputs/load-samples.d.ts.map +1 -0
  365. package/dist/inputs/load-samples.js.map +1 -0
  366. package/dist/inputs/mcp-resolver.d.ts.map +1 -0
  367. package/dist/inputs/mcp-resolver.js.map +1 -0
  368. package/dist/inputs/skill-loader.d.ts.map +1 -0
  369. package/dist/inputs/skill-loader.js +261 -0
  370. package/dist/inputs/skill-loader.js.map +1 -0
  371. package/dist/inputs/url-fetcher.d.ts.map +1 -0
  372. package/dist/inputs/url-fetcher.js.map +1 -0
  373. package/dist/observability/experience-frontmatter.d.ts +37 -0
  374. package/dist/observability/experience-frontmatter.d.ts.map +1 -0
  375. package/dist/observability/experience-frontmatter.js +100 -0
  376. package/dist/observability/experience-frontmatter.js.map +1 -0
  377. package/dist/observability/experience.d.ts +23 -0
  378. package/dist/observability/experience.d.ts.map +1 -0
  379. package/dist/observability/experience.js +3438 -0
  380. package/dist/observability/experience.js.map +1 -0
  381. package/dist/observability/feedback-matchers.d.ts +37 -0
  382. package/dist/observability/feedback-matchers.d.ts.map +1 -0
  383. package/dist/observability/feedback-matchers.js +232 -0
  384. package/dist/observability/feedback-matchers.js.map +1 -0
  385. package/dist/observability/feedback-projection.d.ts +21 -0
  386. package/dist/observability/feedback-projection.d.ts.map +1 -0
  387. package/dist/observability/feedback-projection.js +20 -0
  388. package/dist/observability/feedback-projection.js.map +1 -0
  389. package/dist/observability/inbox-view-model.d.ts +43 -0
  390. package/dist/observability/inbox-view-model.d.ts.map +1 -0
  391. package/dist/observability/inbox-view-model.js +180 -0
  392. package/dist/observability/inbox-view-model.js.map +1 -0
  393. package/dist/observability/inbox.d.ts +28 -0
  394. package/dist/observability/inbox.d.ts.map +1 -0
  395. package/dist/observability/inbox.js +765 -0
  396. package/dist/observability/inbox.js.map +1 -0
  397. package/dist/observability/problem-patterns.d.ts +12 -0
  398. package/dist/observability/problem-patterns.d.ts.map +1 -0
  399. package/dist/observability/problem-patterns.js +244 -0
  400. package/dist/observability/problem-patterns.js.map +1 -0
  401. package/dist/observability/prompts/llm-enhanced-review.prompt.md +224 -0
  402. package/dist/observability/resolved-review.d.ts +37 -0
  403. package/dist/observability/resolved-review.d.ts.map +1 -0
  404. package/dist/observability/resolved-review.js +552 -0
  405. package/dist/observability/resolved-review.js.map +1 -0
  406. package/dist/observability/review-state.d.ts +20 -0
  407. package/dist/observability/review-state.d.ts.map +1 -0
  408. package/dist/observability/review-state.js +169 -0
  409. package/dist/observability/review-state.js.map +1 -0
  410. package/dist/observability/skill-chain-advisories.d.ts +16 -0
  411. package/dist/observability/skill-chain-advisories.d.ts.map +1 -0
  412. package/dist/observability/skill-chain-advisories.js +69 -0
  413. package/dist/observability/skill-chain-advisories.js.map +1 -0
  414. package/dist/observability/skill-chain.d.ts +7 -0
  415. package/dist/observability/skill-chain.d.ts.map +1 -0
  416. package/dist/observability/skill-chain.js +474 -0
  417. package/dist/observability/skill-chain.js.map +1 -0
  418. package/dist/observability/skill-health-analyzer.d.ts.map +1 -0
  419. package/dist/observability/skill-health-analyzer.js.map +1 -0
  420. package/dist/observability/soft-standards/constants.d.ts +5 -0
  421. package/dist/observability/soft-standards/constants.d.ts.map +1 -0
  422. package/dist/observability/soft-standards/constants.js +10 -0
  423. package/dist/observability/soft-standards/constants.js.map +1 -0
  424. package/dist/observability/soft-standards/index.d.ts +16 -0
  425. package/dist/observability/soft-standards/index.d.ts.map +1 -0
  426. package/dist/observability/soft-standards/index.js +15 -0
  427. package/dist/observability/soft-standards/index.js.map +1 -0
  428. package/dist/observability/soft-standards/llm-extractor.d.ts +6 -0
  429. package/dist/observability/soft-standards/llm-extractor.d.ts.map +1 -0
  430. package/dist/observability/soft-standards/llm-extractor.js +692 -0
  431. package/dist/observability/soft-standards/llm-extractor.js.map +1 -0
  432. package/dist/observability/soft-standards/runtime-evaluator.d.ts +12 -0
  433. package/dist/observability/soft-standards/runtime-evaluator.d.ts.map +1 -0
  434. package/dist/observability/soft-standards/runtime-evaluator.js +385 -0
  435. package/dist/observability/soft-standards/runtime-evaluator.js.map +1 -0
  436. package/dist/observability/soft-standards/skill-standards-store.d.ts +11 -0
  437. package/dist/observability/soft-standards/skill-standards-store.d.ts.map +1 -0
  438. package/dist/observability/soft-standards/skill-standards-store.js +158 -0
  439. package/dist/observability/soft-standards/skill-standards-store.js.map +1 -0
  440. package/dist/observability/soft-standards/types.d.ts +212 -0
  441. package/dist/observability/soft-standards/types.d.ts.map +1 -0
  442. package/dist/observability/soft-standards/types.js +2 -0
  443. package/dist/observability/soft-standards/types.js.map +1 -0
  444. package/dist/observability/text-signals.d.ts +14 -0
  445. package/dist/observability/text-signals.d.ts.map +1 -0
  446. package/dist/observability/text-signals.js +116 -0
  447. package/dist/observability/text-signals.js.map +1 -0
  448. package/dist/observability/trace-adapter.d.ts.map +1 -0
  449. package/dist/observability/trace-adapter.js.map +1 -0
  450. package/dist/observability/trace-attribution.d.ts +56 -0
  451. package/dist/observability/trace-attribution.d.ts.map +1 -0
  452. package/dist/observability/trace-attribution.js +200 -0
  453. package/dist/observability/trace-attribution.js.map +1 -0
  454. package/dist/observability/trace-segmenter.d.ts +52 -0
  455. package/dist/observability/trace-segmenter.d.ts.map +1 -0
  456. package/dist/observability/trace-segmenter.js +322 -0
  457. package/dist/observability/trace-segmenter.js.map +1 -0
  458. package/dist/observability/trace-source.d.ts +92 -0
  459. package/dist/observability/trace-source.d.ts.map +1 -0
  460. package/dist/observability/trace-source.js +502 -0
  461. package/dist/observability/trace-source.js.map +1 -0
  462. package/dist/renderer/html-renderer.d.ts.map +1 -0
  463. package/dist/renderer/html-renderer.js.map +1 -0
  464. package/dist/renderer/layout.d.ts.map +1 -0
  465. package/dist/renderer/layout.js.map +1 -0
  466. package/dist/renderer/observation-inbox/helpers.d.ts +14 -0
  467. package/dist/renderer/observation-inbox/helpers.d.ts.map +1 -0
  468. package/dist/renderer/observation-inbox/helpers.js +75 -0
  469. package/dist/renderer/observation-inbox/helpers.js.map +1 -0
  470. package/dist/renderer/observation-inbox/styles.d.ts +2 -0
  471. package/dist/renderer/observation-inbox/styles.d.ts.map +1 -0
  472. package/dist/renderer/observation-inbox/styles.js +5184 -0
  473. package/dist/renderer/observation-inbox/styles.js.map +1 -0
  474. package/dist/renderer/observation-inbox-renderer.d.ts +5 -0
  475. package/dist/renderer/observation-inbox-renderer.d.ts.map +1 -0
  476. package/dist/renderer/observation-inbox-renderer.js +7182 -0
  477. package/dist/renderer/observation-inbox-renderer.js.map +1 -0
  478. package/dist/renderer/skill-detail-renderer.d.ts +13 -0
  479. package/dist/renderer/skill-detail-renderer.d.ts.map +1 -0
  480. package/dist/renderer/skill-detail-renderer.js +1244 -0
  481. package/dist/renderer/skill-detail-renderer.js.map +1 -0
  482. package/dist/renderer/skill-health-renderer.d.ts.map +1 -0
  483. package/dist/renderer/skill-health-renderer.js.map +1 -0
  484. package/dist/renderer/skill-list-renderer.d.ts +3 -0
  485. package/dist/renderer/skill-list-renderer.d.ts.map +1 -0
  486. package/dist/renderer/skill-list-renderer.js +334 -0
  487. package/dist/renderer/skill-list-renderer.js.map +1 -0
  488. package/dist/renderer/summary.d.ts +68 -0
  489. package/dist/renderer/summary.d.ts.map +1 -0
  490. package/dist/renderer/summary.js +1896 -0
  491. package/dist/renderer/summary.js.map +1 -0
  492. package/dist/renderer/table.d.ts.map +1 -0
  493. package/dist/renderer/table.js.map +1 -0
  494. package/dist/renderer/test-view.d.ts.map +1 -0
  495. package/dist/renderer/test-view.js +905 -0
  496. package/dist/renderer/test-view.js.map +1 -0
  497. package/dist/renderer/trends.d.ts.map +1 -0
  498. package/dist/renderer/trends.js.map +1 -0
  499. package/dist/server/job-store.d.ts.map +1 -0
  500. package/dist/server/job-store.js.map +1 -0
  501. package/dist/server/report-server.d.ts.map +1 -0
  502. package/dist/server/report-server.js +892 -0
  503. package/dist/server/report-server.js.map +1 -0
  504. package/dist/server/report-store.d.ts.map +1 -0
  505. package/dist/server/report-store.js.map +1 -0
  506. package/dist/server/skill-index.d.ts +14 -0
  507. package/dist/server/skill-index.d.ts.map +1 -0
  508. package/dist/server/skill-index.js +381 -0
  509. package/dist/server/skill-index.js.map +1 -0
  510. package/dist/server/skill-insights.d.ts +32 -0
  511. package/dist/server/skill-insights.d.ts.map +1 -0
  512. package/dist/server/skill-insights.js +788 -0
  513. package/dist/server/skill-insights.js.map +1 -0
  514. package/dist/shared/hard-rules.d.ts +42 -0
  515. package/dist/shared/hard-rules.d.ts.map +1 -0
  516. package/dist/shared/hard-rules.js +242 -0
  517. package/dist/shared/hard-rules.js.map +1 -0
  518. package/dist/shared/llm-prompts/index.d.ts +14 -0
  519. package/dist/shared/llm-prompts/index.d.ts.map +1 -0
  520. package/dist/shared/llm-prompts/index.js +22 -0
  521. package/dist/shared/llm-prompts/index.js.map +1 -0
  522. package/dist/shared/llm-prompts/skill-health.d.ts +22 -0
  523. package/dist/shared/llm-prompts/skill-health.d.ts.map +1 -0
  524. package/dist/shared/llm-prompts/skill-health.js +170 -0
  525. package/dist/shared/llm-prompts/skill-health.js.map +1 -0
  526. package/dist/shared/time.d.ts.map +1 -0
  527. package/dist/shared/time.js.map +1 -0
  528. package/dist/shared/tool-search.d.ts.map +1 -0
  529. package/dist/shared/tool-search.js.map +1 -0
  530. package/dist/types/dependencies.d.ts +23 -0
  531. package/dist/types/dependencies.d.ts.map +1 -0
  532. package/dist/types/dependencies.js +2 -0
  533. package/dist/types/dependencies.js.map +1 -0
  534. package/dist/types/diagnosis.d.ts +88 -0
  535. package/dist/types/diagnosis.d.ts.map +1 -0
  536. package/dist/types/diagnosis.js +2 -0
  537. package/dist/types/diagnosis.js.map +1 -0
  538. package/dist/types/doctor.d.ts +157 -0
  539. package/dist/types/doctor.d.ts.map +1 -0
  540. package/dist/types/doctor.js.map +1 -0
  541. package/dist/types/eval.d.ts +377 -0
  542. package/dist/types/eval.d.ts.map +1 -0
  543. package/dist/types/eval.js.map +1 -0
  544. package/dist/types/executor.d.ts.map +1 -0
  545. package/dist/types/executor.js.map +1 -0
  546. package/dist/types/index.d.ts +12 -0
  547. package/dist/types/index.d.ts.map +1 -0
  548. package/dist/types/index.js +12 -0
  549. package/dist/types/index.js.map +1 -0
  550. package/dist/types/judge.d.ts.map +1 -0
  551. package/dist/types/judge.js.map +1 -0
  552. package/dist/types/observability.d.ts +865 -0
  553. package/dist/types/observability.d.ts.map +1 -0
  554. package/dist/types/observability.js +14 -0
  555. package/dist/types/observability.js.map +1 -0
  556. package/dist/types/report.d.ts +560 -0
  557. package/dist/types/report.d.ts.map +1 -0
  558. package/dist/types/report.js.map +1 -0
  559. package/dist/types/shared.d.ts.map +1 -0
  560. package/dist/types/shared.js.map +1 -0
  561. package/dist/types/skill-index.d.ts +142 -0
  562. package/dist/types/skill-index.d.ts.map +1 -0
  563. package/dist/types/skill-index.js +2 -0
  564. package/dist/types/skill-index.js.map +1 -0
  565. package/dist/types/storage.d.ts.map +1 -0
  566. package/dist/types/storage.js.map +1 -0
  567. package/dist/util/safe-slice.d.ts +23 -0
  568. package/dist/util/safe-slice.d.ts.map +1 -0
  569. package/dist/util/safe-slice.js +33 -0
  570. package/dist/util/safe-slice.js.map +1 -0
  571. package/package.json +40 -8
  572. package/dist/src/analysis/coverage-analyzer.d.ts.map +0 -1
  573. package/dist/src/analysis/coverage-analyzer.js.map +0 -1
  574. package/dist/src/analysis/failure-clusterer.d.ts.map +0 -1
  575. package/dist/src/analysis/failure-clusterer.js.map +0 -1
  576. package/dist/src/analysis/gap-analyzer.d.ts +0 -121
  577. package/dist/src/analysis/gap-analyzer.d.ts.map +0 -1
  578. package/dist/src/analysis/gap-analyzer.js +0 -471
  579. package/dist/src/analysis/gap-analyzer.js.map +0 -1
  580. package/dist/src/analysis/hedging-classifier.d.ts.map +0 -1
  581. package/dist/src/analysis/hedging-classifier.js.map +0 -1
  582. package/dist/src/analysis/report-diagnostics.d.ts +0 -34
  583. package/dist/src/analysis/report-diagnostics.d.ts.map +0 -1
  584. package/dist/src/analysis/report-diagnostics.js +0 -753
  585. package/dist/src/analysis/report-diagnostics.js.map +0 -1
  586. package/dist/src/analysis/sample-diagnostics.d.ts.map +0 -1
  587. package/dist/src/analysis/sample-diagnostics.js.map +0 -1
  588. package/dist/src/analysis/saturation.d.ts.map +0 -1
  589. package/dist/src/analysis/saturation.js.map +0 -1
  590. package/dist/src/authoring/evolver.d.ts +0 -78
  591. package/dist/src/authoring/evolver.d.ts.map +0 -1
  592. package/dist/src/authoring/evolver.js +0 -320
  593. package/dist/src/authoring/evolver.js.map +0 -1
  594. package/dist/src/authoring/generator.d.ts +0 -35
  595. package/dist/src/authoring/generator.d.ts.map +0 -1
  596. package/dist/src/authoring/generator.js +0 -704
  597. package/dist/src/authoring/generator.js.map +0 -1
  598. package/dist/src/authoring/sample-fixer.d.ts +0 -45
  599. package/dist/src/authoring/sample-fixer.d.ts.map +0 -1
  600. package/dist/src/authoring/sample-fixer.js +0 -186
  601. package/dist/src/authoring/sample-fixer.js.map +0 -1
  602. package/dist/src/cli/cli-exit.d.ts +0 -15
  603. package/dist/src/cli/cli-exit.d.ts.map +0 -1
  604. package/dist/src/cli/cli-exit.js +0 -19
  605. package/dist/src/cli/cli-exit.js.map +0 -1
  606. package/dist/src/cli/commands/_shared.d.ts +0 -12
  607. package/dist/src/cli/commands/_shared.d.ts.map +0 -1
  608. package/dist/src/cli/commands/_shared.js +0 -26
  609. package/dist/src/cli/commands/_shared.js.map +0 -1
  610. package/dist/src/cli/commands/doctor.d.ts +0 -2
  611. package/dist/src/cli/commands/doctor.d.ts.map +0 -1
  612. package/dist/src/cli/commands/doctor.js +0 -174
  613. package/dist/src/cli/commands/doctor.js.map +0 -1
  614. package/dist/src/cli/commands/eval-gold.d.ts +0 -2
  615. package/dist/src/cli/commands/eval-gold.d.ts.map +0 -1
  616. package/dist/src/cli/commands/eval-gold.js +0 -137
  617. package/dist/src/cli/commands/eval-gold.js.map +0 -1
  618. package/dist/src/cli/commands/eval-runner.d.ts +0 -2
  619. package/dist/src/cli/commands/eval-runner.d.ts.map +0 -1
  620. package/dist/src/cli/commands/eval-runner.js +0 -299
  621. package/dist/src/cli/commands/eval-runner.js.map +0 -1
  622. package/dist/src/cli/commands/eval.d.ts +0 -2
  623. package/dist/src/cli/commands/eval.d.ts.map +0 -1
  624. package/dist/src/cli/commands/eval.js +0 -11
  625. package/dist/src/cli/commands/eval.js.map +0 -1
  626. package/dist/src/cli/commands/evolve.d.ts +0 -2
  627. package/dist/src/cli/commands/evolve.d.ts.map +0 -1
  628. package/dist/src/cli/commands/evolve.js +0 -132
  629. package/dist/src/cli/commands/evolve.js.map +0 -1
  630. package/dist/src/cli/commands/init.d.ts +0 -2
  631. package/dist/src/cli/commands/init.d.ts.map +0 -1
  632. package/dist/src/cli/commands/init.js +0 -110
  633. package/dist/src/cli/commands/init.js.map +0 -1
  634. package/dist/src/cli/commands/observe.d.ts +0 -2
  635. package/dist/src/cli/commands/observe.d.ts.map +0 -1
  636. package/dist/src/cli/commands/observe.js +0 -247
  637. package/dist/src/cli/commands/observe.js.map +0 -1
  638. package/dist/src/cli/commands/registry.d.ts +0 -11
  639. package/dist/src/cli/commands/registry.d.ts.map +0 -1
  640. package/dist/src/cli/commands/registry.js +0 -32
  641. package/dist/src/cli/commands/registry.js.map +0 -1
  642. package/dist/src/cli/commands/sample.d.ts +0 -15
  643. package/dist/src/cli/commands/sample.d.ts.map +0 -1
  644. package/dist/src/cli/commands/sample.js +0 -415
  645. package/dist/src/cli/commands/sample.js.map +0 -1
  646. package/dist/src/cli/commands/studio.d.ts +0 -2
  647. package/dist/src/cli/commands/studio.d.ts.map +0 -1
  648. package/dist/src/cli/commands/studio.js +0 -84
  649. package/dist/src/cli/commands/studio.js.map +0 -1
  650. package/dist/src/cli/i18n-dict.d.ts +0 -54
  651. package/dist/src/cli/i18n-dict.d.ts.map +0 -1
  652. package/dist/src/cli/i18n-dict.js +0 -1125
  653. package/dist/src/cli/i18n-dict.js.map +0 -1
  654. package/dist/src/cli/i18n.d.ts +0 -24
  655. package/dist/src/cli/i18n.d.ts.map +0 -1
  656. package/dist/src/cli/i18n.js +0 -53
  657. package/dist/src/cli/i18n.js.map +0 -1
  658. package/dist/src/cli/index.d.ts.map +0 -1
  659. package/dist/src/cli/index.js +0 -61
  660. package/dist/src/cli/index.js.map +0 -1
  661. package/dist/src/cli/parse-run-config.d.ts +0 -90
  662. package/dist/src/cli/parse-run-config.d.ts.map +0 -1
  663. package/dist/src/cli/parse-run-config.js +0 -302
  664. package/dist/src/cli/parse-run-config.js.map +0 -1
  665. package/dist/src/cli/parse-strict.d.ts +0 -7
  666. package/dist/src/cli/parse-strict.d.ts.map +0 -1
  667. package/dist/src/cli/parse-strict.js +0 -26
  668. package/dist/src/cli/parse-strict.js.map +0 -1
  669. package/dist/src/cli/progress.d.ts.map +0 -1
  670. package/dist/src/cli/progress.js.map +0 -1
  671. package/dist/src/cli/run-tally.d.ts +0 -18
  672. package/dist/src/cli/run-tally.d.ts.map +0 -1
  673. package/dist/src/cli/run-tally.js.map +0 -1
  674. package/dist/src/cli/update-check.d.ts +0 -3
  675. package/dist/src/cli/update-check.d.ts.map +0 -1
  676. package/dist/src/cli/update-check.js +0 -37
  677. package/dist/src/cli/update-check.js.map +0 -1
  678. package/dist/src/doctor/health/builtin-dimensions.d.ts.map +0 -1
  679. package/dist/src/doctor/health/builtin-dimensions.js.map +0 -1
  680. package/dist/src/doctor/health/composer.d.ts.map +0 -1
  681. package/dist/src/doctor/health/composer.js +0 -293
  682. package/dist/src/doctor/health/composer.js.map +0 -1
  683. package/dist/src/doctor/health/dimension-registry.d.ts.map +0 -1
  684. package/dist/src/doctor/health/dimension-registry.js.map +0 -1
  685. package/dist/src/doctor/health/dimension-spec.d.ts +0 -47
  686. package/dist/src/doctor/health/dimension-spec.d.ts.map +0 -1
  687. package/dist/src/doctor/health/dimension-spec.js.map +0 -1
  688. package/dist/src/doctor/health/parser.d.ts +0 -27
  689. package/dist/src/doctor/health/parser.d.ts.map +0 -1
  690. package/dist/src/doctor/health/parser.js +0 -191
  691. package/dist/src/doctor/health/parser.js.map +0 -1
  692. package/dist/src/doctor/health/prompt-builder.d.ts +0 -22
  693. package/dist/src/doctor/health/prompt-builder.d.ts.map +0 -1
  694. package/dist/src/doctor/health/prompt-builder.js +0 -170
  695. package/dist/src/doctor/health/prompt-builder.js.map +0 -1
  696. package/dist/src/doctor/health/register.d.ts.map +0 -1
  697. package/dist/src/doctor/health/register.js.map +0 -1
  698. package/dist/src/doctor/html-renderer.d.ts +0 -20
  699. package/dist/src/doctor/html-renderer.d.ts.map +0 -1
  700. package/dist/src/doctor/html-renderer.js +0 -376
  701. package/dist/src/doctor/html-renderer.js.map +0 -1
  702. package/dist/src/doctor/index.d.ts.map +0 -1
  703. package/dist/src/doctor/index.js +0 -245
  704. package/dist/src/doctor/index.js.map +0 -1
  705. package/dist/src/doctor/preflight.d.ts.map +0 -1
  706. package/dist/src/doctor/preflight.js.map +0 -1
  707. package/dist/src/doctor/renderer.d.ts +0 -17
  708. package/dist/src/doctor/renderer.d.ts.map +0 -1
  709. package/dist/src/doctor/renderer.js +0 -123
  710. package/dist/src/doctor/renderer.js.map +0 -1
  711. package/dist/src/doctor/rules.d.ts.map +0 -1
  712. package/dist/src/doctor/rules.js +0 -289
  713. package/dist/src/doctor/rules.js.map +0 -1
  714. package/dist/src/eval-core/bootstrap.d.ts.map +0 -1
  715. package/dist/src/eval-core/bootstrap.js.map +0 -1
  716. package/dist/src/eval-core/cache.d.ts.map +0 -1
  717. package/dist/src/eval-core/cache.js.map +0 -1
  718. package/dist/src/eval-core/comparability.d.ts.map +0 -1
  719. package/dist/src/eval-core/comparability.js.map +0 -1
  720. package/dist/src/eval-core/dependency-checker.d.ts +0 -58
  721. package/dist/src/eval-core/dependency-checker.d.ts.map +0 -1
  722. package/dist/src/eval-core/dependency-checker.js.map +0 -1
  723. package/dist/src/eval-core/evaluation-execution.d.ts.map +0 -1
  724. package/dist/src/eval-core/evaluation-execution.js.map +0 -1
  725. package/dist/src/eval-core/evaluation-job.d.ts.map +0 -1
  726. package/dist/src/eval-core/evaluation-job.js.map +0 -1
  727. package/dist/src/eval-core/evaluation-reporting.d.ts.map +0 -1
  728. package/dist/src/eval-core/evaluation-reporting.js.map +0 -1
  729. package/dist/src/eval-core/execution-strategy.d.ts.map +0 -1
  730. package/dist/src/eval-core/execution-strategy.js +0 -165
  731. package/dist/src/eval-core/execution-strategy.js.map +0 -1
  732. package/dist/src/eval-core/fact-checker.d.ts.map +0 -1
  733. package/dist/src/eval-core/fact-checker.js.map +0 -1
  734. package/dist/src/eval-core/layer-gates.d.ts.map +0 -1
  735. package/dist/src/eval-core/layer-gates.js.map +0 -1
  736. package/dist/src/eval-core/mock-hook.cjs +0 -203
  737. package/dist/src/eval-core/mocks-runtime.d.ts +0 -96
  738. package/dist/src/eval-core/mocks-runtime.d.ts.map +0 -1
  739. package/dist/src/eval-core/mocks-runtime.js +0 -323
  740. package/dist/src/eval-core/mocks-runtime.js.map +0 -1
  741. package/dist/src/eval-core/schema.d.ts.map +0 -1
  742. package/dist/src/eval-core/schema.js.map +0 -1
  743. package/dist/src/eval-core/statistics.d.ts.map +0 -1
  744. package/dist/src/eval-core/statistics.js.map +0 -1
  745. package/dist/src/eval-core/task-planner.d.ts.map +0 -1
  746. package/dist/src/eval-core/task-planner.js.map +0 -1
  747. package/dist/src/eval-core/verdict.d.ts.map +0 -1
  748. package/dist/src/eval-core/verdict.js.map +0 -1
  749. package/dist/src/eval-workflows/batch-evaluation-workflow.d.ts.map +0 -1
  750. package/dist/src/eval-workflows/batch-evaluation-workflow.js.map +0 -1
  751. package/dist/src/eval-workflows/evaluation-pipeline.d.ts +0 -109
  752. package/dist/src/eval-workflows/evaluation-pipeline.d.ts.map +0 -1
  753. package/dist/src/eval-workflows/evaluation-pipeline.js +0 -373
  754. package/dist/src/eval-workflows/evaluation-pipeline.js.map +0 -1
  755. package/dist/src/eval-workflows/evaluation-preparation.d.ts.map +0 -1
  756. package/dist/src/eval-workflows/evaluation-preparation.js.map +0 -1
  757. package/dist/src/eval-workflows/run-evaluation.d.ts.map +0 -1
  758. package/dist/src/eval-workflows/run-evaluation.js +0 -528
  759. package/dist/src/eval-workflows/run-evaluation.js.map +0 -1
  760. package/dist/src/executors/anthropic-api.d.ts.map +0 -1
  761. package/dist/src/executors/anthropic-api.js.map +0 -1
  762. package/dist/src/executors/claude-cli.d.ts.map +0 -1
  763. package/dist/src/executors/claude-cli.js +0 -179
  764. package/dist/src/executors/claude-cli.js.map +0 -1
  765. package/dist/src/executors/claude-sdk-trace.d.ts.map +0 -1
  766. package/dist/src/executors/claude-sdk-trace.js.map +0 -1
  767. package/dist/src/executors/claude-sdk.d.ts.map +0 -1
  768. package/dist/src/executors/claude-sdk.js.map +0 -1
  769. package/dist/src/executors/codex-cli-trace.d.ts.map +0 -1
  770. package/dist/src/executors/codex-cli-trace.js.map +0 -1
  771. package/dist/src/executors/codex-cli.d.ts.map +0 -1
  772. package/dist/src/executors/codex-cli.js.map +0 -1
  773. package/dist/src/executors/codex-sdk.d.ts.map +0 -1
  774. package/dist/src/executors/codex-sdk.js.map +0 -1
  775. package/dist/src/executors/gemini.d.ts.map +0 -1
  776. package/dist/src/executors/gemini.js.map +0 -1
  777. package/dist/src/executors/index.d.ts.map +0 -1
  778. package/dist/src/executors/index.js.map +0 -1
  779. package/dist/src/executors/openai-api.d.ts.map +0 -1
  780. package/dist/src/executors/openai-api.js.map +0 -1
  781. package/dist/src/executors/runtime-fingerprint.d.ts.map +0 -1
  782. package/dist/src/executors/runtime-fingerprint.js.map +0 -1
  783. package/dist/src/executors/script.d.ts.map +0 -1
  784. package/dist/src/executors/script.js.map +0 -1
  785. package/dist/src/executors/shared.d.ts +0 -194
  786. package/dist/src/executors/shared.d.ts.map +0 -1
  787. package/dist/src/executors/shared.js +0 -256
  788. package/dist/src/executors/shared.js.map +0 -1
  789. package/dist/src/grading/assertions.d.ts.map +0 -1
  790. package/dist/src/grading/assertions.js.map +0 -1
  791. package/dist/src/grading/debias-validate.d.ts.map +0 -1
  792. package/dist/src/grading/debias-validate.js.map +0 -1
  793. package/dist/src/grading/diagnostic.d.ts.map +0 -1
  794. package/dist/src/grading/diagnostic.js.map +0 -1
  795. package/dist/src/grading/gold-cli.d.ts.map +0 -1
  796. package/dist/src/grading/gold-cli.js.map +0 -1
  797. package/dist/src/grading/gold-dataset.d.ts.map +0 -1
  798. package/dist/src/grading/gold-dataset.js.map +0 -1
  799. package/dist/src/grading/human-gold.d.ts.map +0 -1
  800. package/dist/src/grading/human-gold.js.map +0 -1
  801. package/dist/src/grading/index.d.ts.map +0 -1
  802. package/dist/src/grading/index.js.map +0 -1
  803. package/dist/src/grading/judge.d.ts.map +0 -1
  804. package/dist/src/grading/judge.js.map +0 -1
  805. package/dist/src/grading/layered-scores.d.ts.map +0 -1
  806. package/dist/src/grading/layered-scores.js.map +0 -1
  807. package/dist/src/inputs/eval-config.d.ts.map +0 -1
  808. package/dist/src/inputs/eval-config.js.map +0 -1
  809. package/dist/src/inputs/load-samples.d.ts.map +0 -1
  810. package/dist/src/inputs/load-samples.js.map +0 -1
  811. package/dist/src/inputs/mcp-resolver.d.ts.map +0 -1
  812. package/dist/src/inputs/mcp-resolver.js.map +0 -1
  813. package/dist/src/inputs/skill-loader.d.ts.map +0 -1
  814. package/dist/src/inputs/skill-loader.js +0 -253
  815. package/dist/src/inputs/skill-loader.js.map +0 -1
  816. package/dist/src/inputs/url-fetcher.d.ts.map +0 -1
  817. package/dist/src/inputs/url-fetcher.js.map +0 -1
  818. package/dist/src/observability/experience.d.ts +0 -260
  819. package/dist/src/observability/experience.d.ts.map +0 -1
  820. package/dist/src/observability/experience.js +0 -1233
  821. package/dist/src/observability/experience.js.map +0 -1
  822. package/dist/src/observability/inbox-view-model.d.ts +0 -27
  823. package/dist/src/observability/inbox-view-model.d.ts.map +0 -1
  824. package/dist/src/observability/inbox-view-model.js +0 -135
  825. package/dist/src/observability/inbox-view-model.js.map +0 -1
  826. package/dist/src/observability/inbox.d.ts +0 -125
  827. package/dist/src/observability/inbox.d.ts.map +0 -1
  828. package/dist/src/observability/inbox.js +0 -741
  829. package/dist/src/observability/inbox.js.map +0 -1
  830. package/dist/src/observability/problem-patterns.d.ts +0 -51
  831. package/dist/src/observability/problem-patterns.d.ts.map +0 -1
  832. package/dist/src/observability/problem-patterns.js +0 -205
  833. package/dist/src/observability/problem-patterns.js.map +0 -1
  834. package/dist/src/observability/review-state.d.ts +0 -54
  835. package/dist/src/observability/review-state.d.ts.map +0 -1
  836. package/dist/src/observability/review-state.js +0 -131
  837. package/dist/src/observability/review-state.js.map +0 -1
  838. package/dist/src/observability/skill-chain-advisories.d.ts +0 -25
  839. package/dist/src/observability/skill-chain-advisories.d.ts.map +0 -1
  840. package/dist/src/observability/skill-chain-advisories.js +0 -69
  841. package/dist/src/observability/skill-chain-advisories.js.map +0 -1
  842. package/dist/src/observability/skill-chain.d.ts +0 -61
  843. package/dist/src/observability/skill-chain.d.ts.map +0 -1
  844. package/dist/src/observability/skill-chain.js +0 -233
  845. package/dist/src/observability/skill-chain.js.map +0 -1
  846. package/dist/src/observability/skill-health-analyzer.d.ts.map +0 -1
  847. package/dist/src/observability/skill-health-analyzer.js.map +0 -1
  848. package/dist/src/observability/text-signals.d.ts +0 -7
  849. package/dist/src/observability/text-signals.d.ts.map +0 -1
  850. package/dist/src/observability/text-signals.js +0 -22
  851. package/dist/src/observability/text-signals.js.map +0 -1
  852. package/dist/src/observability/trace-adapter.d.ts.map +0 -1
  853. package/dist/src/observability/trace-adapter.js.map +0 -1
  854. package/dist/src/observability/trace-attribution.d.ts +0 -56
  855. package/dist/src/observability/trace-attribution.d.ts.map +0 -1
  856. package/dist/src/observability/trace-attribution.js +0 -200
  857. package/dist/src/observability/trace-attribution.js.map +0 -1
  858. package/dist/src/observability/trace-segmenter.d.ts +0 -52
  859. package/dist/src/observability/trace-segmenter.d.ts.map +0 -1
  860. package/dist/src/observability/trace-segmenter.js +0 -320
  861. package/dist/src/observability/trace-segmenter.js.map +0 -1
  862. package/dist/src/observability/trace-source.d.ts +0 -99
  863. package/dist/src/observability/trace-source.d.ts.map +0 -1
  864. package/dist/src/observability/trace-source.js +0 -492
  865. package/dist/src/observability/trace-source.js.map +0 -1
  866. package/dist/src/renderer/html-renderer.d.ts.map +0 -1
  867. package/dist/src/renderer/html-renderer.js.map +0 -1
  868. package/dist/src/renderer/layout.d.ts.map +0 -1
  869. package/dist/src/renderer/layout.js.map +0 -1
  870. package/dist/src/renderer/observation-inbox-renderer.d.ts +0 -4
  871. package/dist/src/renderer/observation-inbox-renderer.d.ts.map +0 -1
  872. package/dist/src/renderer/observation-inbox-renderer.js +0 -5376
  873. package/dist/src/renderer/observation-inbox-renderer.js.map +0 -1
  874. package/dist/src/renderer/skill-detail-renderer.d.ts +0 -15
  875. package/dist/src/renderer/skill-detail-renderer.d.ts.map +0 -1
  876. package/dist/src/renderer/skill-detail-renderer.js +0 -1234
  877. package/dist/src/renderer/skill-detail-renderer.js.map +0 -1
  878. package/dist/src/renderer/skill-health-renderer.d.ts.map +0 -1
  879. package/dist/src/renderer/skill-health-renderer.js.map +0 -1
  880. package/dist/src/renderer/skill-list-renderer.d.ts +0 -4
  881. package/dist/src/renderer/skill-list-renderer.d.ts.map +0 -1
  882. package/dist/src/renderer/skill-list-renderer.js +0 -327
  883. package/dist/src/renderer/skill-list-renderer.js.map +0 -1
  884. package/dist/src/renderer/summary.d.ts +0 -68
  885. package/dist/src/renderer/summary.d.ts.map +0 -1
  886. package/dist/src/renderer/summary.js +0 -1896
  887. package/dist/src/renderer/summary.js.map +0 -1
  888. package/dist/src/renderer/table.d.ts.map +0 -1
  889. package/dist/src/renderer/table.js.map +0 -1
  890. package/dist/src/renderer/test-view.d.ts.map +0 -1
  891. package/dist/src/renderer/test-view.js +0 -905
  892. package/dist/src/renderer/test-view.js.map +0 -1
  893. package/dist/src/renderer/trends.d.ts.map +0 -1
  894. package/dist/src/renderer/trends.js.map +0 -1
  895. package/dist/src/server/job-store.d.ts.map +0 -1
  896. package/dist/src/server/job-store.js.map +0 -1
  897. package/dist/src/server/report-server.d.ts.map +0 -1
  898. package/dist/src/server/report-server.js +0 -801
  899. package/dist/src/server/report-server.js.map +0 -1
  900. package/dist/src/server/report-store.d.ts.map +0 -1
  901. package/dist/src/server/report-store.js.map +0 -1
  902. package/dist/src/server/skill-index.d.ts +0 -77
  903. package/dist/src/server/skill-index.d.ts.map +0 -1
  904. package/dist/src/server/skill-index.js +0 -331
  905. package/dist/src/server/skill-index.js.map +0 -1
  906. package/dist/src/server/skill-insights.d.ts +0 -92
  907. package/dist/src/server/skill-insights.d.ts.map +0 -1
  908. package/dist/src/server/skill-insights.js +0 -676
  909. package/dist/src/server/skill-insights.js.map +0 -1
  910. package/dist/src/shared/hard-rules.d.ts +0 -40
  911. package/dist/src/shared/hard-rules.d.ts.map +0 -1
  912. package/dist/src/shared/hard-rules.js +0 -200
  913. package/dist/src/shared/hard-rules.js.map +0 -1
  914. package/dist/src/shared/time.d.ts.map +0 -1
  915. package/dist/src/shared/time.js.map +0 -1
  916. package/dist/src/shared/tool-search.d.ts.map +0 -1
  917. package/dist/src/shared/tool-search.js.map +0 -1
  918. package/dist/src/types/doctor.d.ts +0 -155
  919. package/dist/src/types/doctor.d.ts.map +0 -1
  920. package/dist/src/types/doctor.js.map +0 -1
  921. package/dist/src/types/eval.d.ts +0 -372
  922. package/dist/src/types/eval.d.ts.map +0 -1
  923. package/dist/src/types/eval.js.map +0 -1
  924. package/dist/src/types/executor.d.ts.map +0 -1
  925. package/dist/src/types/executor.js.map +0 -1
  926. package/dist/src/types/index.d.ts +0 -8
  927. package/dist/src/types/index.d.ts.map +0 -1
  928. package/dist/src/types/index.js +0 -8
  929. package/dist/src/types/index.js.map +0 -1
  930. package/dist/src/types/judge.d.ts.map +0 -1
  931. package/dist/src/types/judge.js.map +0 -1
  932. package/dist/src/types/report.d.ts +0 -551
  933. package/dist/src/types/report.d.ts.map +0 -1
  934. package/dist/src/types/report.js.map +0 -1
  935. package/dist/src/types/shared.d.ts.map +0 -1
  936. package/dist/src/types/shared.js.map +0 -1
  937. package/dist/src/types/storage.d.ts.map +0 -1
  938. package/dist/src/types/storage.js.map +0 -1
  939. package/dist/src/util/safe-slice.d.ts +0 -23
  940. package/dist/src/util/safe-slice.d.ts.map +0 -1
  941. package/dist/src/util/safe-slice.js +0 -33
  942. package/dist/src/util/safe-slice.js.map +0 -1
  943. /package/dist/{src/analysis → analysis}/coverage-analyzer.d.ts +0 -0
  944. /package/dist/{src/analysis → analysis}/coverage-analyzer.js +0 -0
  945. /package/dist/{src/analysis → analysis}/failure-clusterer.d.ts +0 -0
  946. /package/dist/{src/analysis → analysis}/failure-clusterer.js +0 -0
  947. /package/dist/{src/analysis → analysis}/hedging-classifier.d.ts +0 -0
  948. /package/dist/{src/analysis → analysis}/hedging-classifier.js +0 -0
  949. /package/dist/{src/analysis → analysis}/sample-diagnostics.d.ts +0 -0
  950. /package/dist/{src/analysis → analysis}/sample-diagnostics.js +0 -0
  951. /package/dist/{src/analysis → analysis}/saturation.d.ts +0 -0
  952. /package/dist/{src/analysis → analysis}/saturation.js +0 -0
  953. /package/dist/{src/cli → cli}/index.d.ts +0 -0
  954. /package/dist/{src/cli → cli/lib}/progress.d.ts +0 -0
  955. /package/dist/{src/cli → cli/lib}/progress.js +0 -0
  956. /package/dist/{src/cli → cli/lib}/run-tally.js +0 -0
  957. /package/dist/{src/doctor → doctor}/health/builtin-dimensions.d.ts +0 -0
  958. /package/dist/{src/doctor → doctor}/health/builtin-dimensions.js +0 -0
  959. /package/dist/{src/doctor → doctor}/health/composer.d.ts +0 -0
  960. /package/dist/{src/doctor → doctor}/health/dimension-registry.d.ts +0 -0
  961. /package/dist/{src/doctor → doctor}/health/dimension-registry.js +0 -0
  962. /package/dist/{src/doctor → doctor}/health/dimension-spec.js +0 -0
  963. /package/dist/{src/doctor → doctor}/health/register.d.ts +0 -0
  964. /package/dist/{src/doctor → doctor}/health/register.js +0 -0
  965. /package/dist/{src/doctor → doctor}/index.d.ts +0 -0
  966. /package/dist/{src/doctor → doctor}/preflight.d.ts +0 -0
  967. /package/dist/{src/doctor → doctor}/preflight.js +0 -0
  968. /package/dist/{src/doctor → doctor}/rules.d.ts +0 -0
  969. /package/dist/{src/eval-core → eval-core}/bootstrap.d.ts +0 -0
  970. /package/dist/{src/eval-core → eval-core}/bootstrap.js +0 -0
  971. /package/dist/{src/eval-core → eval-core}/cache.d.ts +0 -0
  972. /package/dist/{src/eval-core → eval-core}/cache.js +0 -0
  973. /package/dist/{src/eval-core → eval-core}/comparability.d.ts +0 -0
  974. /package/dist/{src/eval-core → eval-core}/comparability.js +0 -0
  975. /package/dist/{src/eval-core → eval-core}/dependency-checker.js +0 -0
  976. /package/dist/{src/eval-core → eval-core}/evaluation-execution.d.ts +0 -0
  977. /package/dist/{src/eval-core → eval-core}/evaluation-execution.js +0 -0
  978. /package/dist/{src/eval-core → eval-core}/evaluation-job.d.ts +0 -0
  979. /package/dist/{src/eval-core → eval-core}/evaluation-job.js +0 -0
  980. /package/dist/{src/eval-core → eval-core}/evaluation-reporting.d.ts +0 -0
  981. /package/dist/{src/eval-core → eval-core}/evaluation-reporting.js +0 -0
  982. /package/dist/{src/eval-core → eval-core}/execution-strategy.d.ts +0 -0
  983. /package/dist/{src/eval-core → eval-core}/fact-checker.d.ts +0 -0
  984. /package/dist/{src/eval-core → eval-core}/fact-checker.js +0 -0
  985. /package/dist/{src/eval-core → eval-core}/layer-gates.d.ts +0 -0
  986. /package/dist/{src/eval-core → eval-core}/layer-gates.js +0 -0
  987. /package/dist/{src/eval-core → eval-core}/schema.d.ts +0 -0
  988. /package/dist/{src/eval-core → eval-core}/schema.js +0 -0
  989. /package/dist/{src/eval-core → eval-core}/statistics.d.ts +0 -0
  990. /package/dist/{src/eval-core → eval-core}/statistics.js +0 -0
  991. /package/dist/{src/eval-core → eval-core}/task-planner.d.ts +0 -0
  992. /package/dist/{src/eval-core → eval-core}/task-planner.js +0 -0
  993. /package/dist/{src/eval-core → eval-core}/verdict.d.ts +0 -0
  994. /package/dist/{src/eval-core → eval-core}/verdict.js +0 -0
  995. /package/dist/{src/eval-workflows → eval-workflows}/batch-evaluation-workflow.d.ts +0 -0
  996. /package/dist/{src/eval-workflows → eval-workflows}/batch-evaluation-workflow.js +0 -0
  997. /package/dist/{src/eval-workflows → eval-workflows}/evaluation-preparation.d.ts +0 -0
  998. /package/dist/{src/eval-workflows → eval-workflows}/evaluation-preparation.js +0 -0
  999. /package/dist/{src/eval-workflows → eval-workflows}/run-evaluation.d.ts +0 -0
  1000. /package/dist/{src/executors → executors}/anthropic-api.d.ts +0 -0
  1001. /package/dist/{src/executors → executors}/anthropic-api.js +0 -0
  1002. /package/dist/{src/executors → executors}/claude-cli.d.ts +0 -0
  1003. /package/dist/{src/executors → executors}/claude-sdk-trace.d.ts +0 -0
  1004. /package/dist/{src/executors → executors}/claude-sdk-trace.js +0 -0
  1005. /package/dist/{src/executors → executors}/claude-sdk.d.ts +0 -0
  1006. /package/dist/{src/executors → executors}/claude-sdk.js +0 -0
  1007. /package/dist/{src/executors → executors}/codex-cli-trace.d.ts +0 -0
  1008. /package/dist/{src/executors → executors}/codex-cli-trace.js +0 -0
  1009. /package/dist/{src/executors → executors}/codex-cli.d.ts +0 -0
  1010. /package/dist/{src/executors → executors}/codex-cli.js +0 -0
  1011. /package/dist/{src/executors → executors}/codex-sdk.d.ts +0 -0
  1012. /package/dist/{src/executors → executors}/codex-sdk.js +0 -0
  1013. /package/dist/{src/executors → executors}/gemini.d.ts +0 -0
  1014. /package/dist/{src/executors → executors}/gemini.js +0 -0
  1015. /package/dist/{src/executors → executors}/index.d.ts +0 -0
  1016. /package/dist/{src/executors → executors}/index.js +0 -0
  1017. /package/dist/{src/executors → executors}/openai-api.d.ts +0 -0
  1018. /package/dist/{src/executors → executors}/openai-api.js +0 -0
  1019. /package/dist/{src/executors → executors}/runtime-fingerprint.d.ts +0 -0
  1020. /package/dist/{src/executors → executors}/runtime-fingerprint.js +0 -0
  1021. /package/dist/{src/executors → executors}/script.d.ts +0 -0
  1022. /package/dist/{src/executors → executors}/script.js +0 -0
  1023. /package/dist/{src/grading → grading}/assertions.d.ts +0 -0
  1024. /package/dist/{src/grading → grading}/assertions.js +0 -0
  1025. /package/dist/{src/grading → grading}/debias-validate.d.ts +0 -0
  1026. /package/dist/{src/grading → grading}/debias-validate.js +0 -0
  1027. /package/dist/{src/grading → grading}/diagnostic.d.ts +0 -0
  1028. /package/dist/{src/grading → grading}/diagnostic.js +0 -0
  1029. /package/dist/{src/grading → grading}/gold-cli.d.ts +0 -0
  1030. /package/dist/{src/grading → grading}/gold-cli.js +0 -0
  1031. /package/dist/{src/grading → grading}/gold-dataset.d.ts +0 -0
  1032. /package/dist/{src/grading → grading}/gold-dataset.js +0 -0
  1033. /package/dist/{src/grading → grading}/human-gold.d.ts +0 -0
  1034. /package/dist/{src/grading → grading}/human-gold.js +0 -0
  1035. /package/dist/{src/grading → grading}/index.d.ts +0 -0
  1036. /package/dist/{src/grading → grading}/index.js +0 -0
  1037. /package/dist/{src/grading → grading}/judge.d.ts +0 -0
  1038. /package/dist/{src/grading → grading}/judge.js +0 -0
  1039. /package/dist/{src/grading → grading}/layered-scores.d.ts +0 -0
  1040. /package/dist/{src/grading → grading}/layered-scores.js +0 -0
  1041. /package/dist/{src/inputs → inputs}/eval-config.d.ts +0 -0
  1042. /package/dist/{src/inputs → inputs}/eval-config.js +0 -0
  1043. /package/dist/{src/inputs → inputs}/load-samples.d.ts +0 -0
  1044. /package/dist/{src/inputs → inputs}/load-samples.js +0 -0
  1045. /package/dist/{src/inputs → inputs}/mcp-resolver.d.ts +0 -0
  1046. /package/dist/{src/inputs → inputs}/mcp-resolver.js +0 -0
  1047. /package/dist/{src/inputs → inputs}/skill-loader.d.ts +0 -0
  1048. /package/dist/{src/inputs → inputs}/url-fetcher.d.ts +0 -0
  1049. /package/dist/{src/inputs → inputs}/url-fetcher.js +0 -0
  1050. /package/dist/{src/observability → observability}/skill-health-analyzer.d.ts +0 -0
  1051. /package/dist/{src/observability → observability}/skill-health-analyzer.js +0 -0
  1052. /package/dist/{src/observability → observability}/trace-adapter.d.ts +0 -0
  1053. /package/dist/{src/observability → observability}/trace-adapter.js +0 -0
  1054. /package/dist/{src/renderer → renderer}/html-renderer.d.ts +0 -0
  1055. /package/dist/{src/renderer → renderer}/html-renderer.js +0 -0
  1056. /package/dist/{src/renderer → renderer}/layout.d.ts +0 -0
  1057. /package/dist/{src/renderer → renderer}/layout.js +0 -0
  1058. /package/dist/{src/renderer → renderer}/skill-health-renderer.d.ts +0 -0
  1059. /package/dist/{src/renderer → renderer}/skill-health-renderer.js +0 -0
  1060. /package/dist/{src/renderer → renderer}/table.d.ts +0 -0
  1061. /package/dist/{src/renderer → renderer}/table.js +0 -0
  1062. /package/dist/{src/renderer → renderer}/test-view.d.ts +0 -0
  1063. /package/dist/{src/renderer → renderer}/trends.d.ts +0 -0
  1064. /package/dist/{src/renderer → renderer}/trends.js +0 -0
  1065. /package/dist/{src/server → server}/job-store.d.ts +0 -0
  1066. /package/dist/{src/server → server}/job-store.js +0 -0
  1067. /package/dist/{src/server → server}/report-server.d.ts +0 -0
  1068. /package/dist/{src/server → server}/report-store.d.ts +0 -0
  1069. /package/dist/{src/server → server}/report-store.js +0 -0
  1070. /package/dist/{src/shared → shared}/time.d.ts +0 -0
  1071. /package/dist/{src/shared → shared}/time.js +0 -0
  1072. /package/dist/{src/shared → shared}/tool-search.d.ts +0 -0
  1073. /package/dist/{src/shared → shared}/tool-search.js +0 -0
  1074. /package/dist/{src/types → types}/doctor.js +0 -0
  1075. /package/dist/{src/types → types}/eval.js +0 -0
  1076. /package/dist/{src/types → types}/executor.d.ts +0 -0
  1077. /package/dist/{src/types → types}/executor.js +0 -0
  1078. /package/dist/{src/types → types}/judge.d.ts +0 -0
  1079. /package/dist/{src/types → types}/judge.js +0 -0
  1080. /package/dist/{src/types → types}/report.js +0 -0
  1081. /package/dist/{src/types → types}/shared.d.ts +0 -0
  1082. /package/dist/{src/types → types}/shared.js +0 -0
  1083. /package/dist/{src/types → types}/storage.d.ts +0 -0
  1084. /package/dist/{src/types → types}/storage.js +0 -0
package/README.zh.md CHANGED
@@ -8,26 +8,24 @@
8
8
 
9
9
  [English](./README.md) | **简体中文**
10
10
 
11
- **omk** — 你给 LLM 的知识,价值在哪里?
12
- omk 帮你用客观数据回答,而不是凭感觉。
11
+ **你改完 prompt,真的变好了吗?**
12
+ 用统计严谨性 A/B 测试你的 prompt 和 skill —— Bootstrap 置信区间、Krippendorff α、长度去偏,全部默认开。
13
13
 
14
- **面向 LLM 知识输入(prompt / RAG / skill / agent)的评测框架** —— 固定模型,只变知识载体。
15
-
16
- <a id="statistical-rigor"></a>
17
- > 默认带:Bootstrap 置信区间 · Krippendorff α(评委 ↔ 人工)· 长度去偏 · 饱和曲线 · 用例隔离(construct validity)。[这些为什么重要 →](docs/zh/statistical-rigor.md)
18
-
19
- ![omk 报告](./assets/screenshots/report-overview-zh.png)
14
+ ![omk 报告 — verdict pill「v2 明显优于 v1,可以发布」](./assets/screenshots/report-overview-zh.png)
20
15
 
21
16
  ## 快速开始
22
17
 
23
18
  ```bash
24
- npm i oh-my-knowledge -g
25
- omk init my-eval && cd my-eval
26
- # 编辑 skills/code-review-v1/SKILL.md 和 skills/code-review-v2/SKILL.md,填入你的两版内容
27
- omk eval --control code-review-v1 --treatment code-review-v2 # → 5 分钟出 HTML 报告 + verdict
19
+ npm i -g oh-my-knowledge
20
+ omk init demo && cd demo
21
+ omk eval --control code-review-v1 --treatment code-review-v2
28
22
  ```
29
23
 
30
- 深入:[在 Claude Code / Codex 中调用](#在-ai-coding-agent-中使用) · [`omk eval` 全 flag](#omk-eval) · [artifact 目录结构](#artifact-目录结构) · [`--lang` / `OMK_LANG`](#环境变量)
24
+ 不用改任何文件 —— `omk init` 帮你脚手架两版 skill 和三条评测用例;`omk eval` 跑控制变量 A/B,5 分钟内出 HTML 报告 + 一行 verdict。
25
+
26
+ 手把手教程:[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)(推荐第一次跑评测的用户)。
27
+
28
+ 深入:[CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测样本格式](docs/zh/reference/eval-sample-format.md) · [执行器与 artifact 布局](docs/zh/reference/executors.md)
31
29
 
32
30
  ## 在 AI Coding Agent 中使用
33
31
 
@@ -41,7 +39,7 @@ omk eval --control code-review-v1 --treatment code-review-v2 # → 5 分钟
41
39
  /omk sample # 生成或补齐评测用例
42
40
  ```
43
41
 
44
- 这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill,所以一般不用显式传路径。(下面 Codex 段落给出 `omk evolve <skill>` 的裸 CLI 形式。)也可以直接说「帮我评测 v1 和 v2 的差异」、「改进一下这个 artifact」,omk 会自动理解意图并调用对应命令。
42
+ 这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill。也可以直接说「帮我评测 v1 和 v2 的差异」、「改进一下这个 artifact」,omk 会自动理解意图并调用对应命令。
45
43
 
46
44
  ### 在 Codex 中使用
47
45
 
@@ -53,21 +51,11 @@ omk evolve skills/my-skill.md
53
51
  omk sample skills/my-skill.md
54
52
  ```
55
53
 
56
- 也可以直接用自然语言描述目标,例如"比较 v1 和 v2 的评测差异"、"为这个 skill 生成评测用例"。
54
+ 也可以直接用自然语言描述目标,例如「比较 v1 和 v2 的评测差异」、「为这个 skill 生成评测用例」。
57
55
 
58
56
  ## 为什么需要这个工具
59
57
 
60
- 做知识工程的团队会产出大量知识载体(当前常见是 skill,也包括 prompt、agent、workflow 等)。当被问到"v2 比 v1 好在哪"时,需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题:相同模型、相同评测用例,只改变知识载体。
61
-
62
- ## 核心能力
63
-
64
- - **LLM 健康度审计** — `omk doctor` 用单次 LLM 会话产出多维度健康度报告,7 个内置维度(触发与边界 / 文档清晰 / 指令精确性 / 依赖检查 / 工具规范 / 安全与合规 / 示例完备)独立给「健康 / 亚健康 / 不健康 / 不适用」+ findings + 改进建议;维度可扩展,`--html` 产可视化报告。无 LLM 环境(CI 节点 / 断网调试)可加 `--static-only` 跑纯静态检查(可读性 / 元数据 / 依赖 / samples 契约)。`omk eval` 内部仍跑静态可读性、元数据、依赖 gate 把关评测质量(角色分离:doctor=审计,eval=评测)
65
- - **控制变量离线评测** — 固定模型和用例,只变知识载体;兼容 Claude Code skill、CLAUDE.md prompt、RAG 知识库等任何 markdown 形式的指令
66
- - **六维独立打分** — Fact / Behavior / LLM-judge / Cost / Efficiency / Stability 分别出信号,单一维度的回退不会被其他维度的收益掩盖
67
- - **线上 session 观测** — 解析 Claude Code session JSONL,在真实用户会话上测量各 skill 的失败率、耗时、token 成本和知识缺口信号
68
- - **知识缺口识别** — 严重度加权的信号(显式标记 / 搜索失败 / hedging 用语 / 反复失败)量化风险敞口,不宣称完备性
69
- - **合并前 CI 门** — `omk eval` 强制三层 all-pass(fact + behavior + llm-judge),抓复合分掩盖的单层回退
70
- - **一行 ship/no-ship 结论** — `omk eval` 聚合 bootstrap CI / 三层 ci-gate / saturation / human α,给六档 verdict(PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO)+ 行动建议;exit code 反映是否可 ship
58
+ 做知识工程的团队会产出大量知识载体(当前常见是 skill,也包括 prompt、agent、workflow 等)。当被问到「v2 比 v1 好在哪」时,需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题:**相同模型、相同评测用例,只改变知识载体。**
71
59
 
72
60
  ## 为什么选 omk
73
61
 
@@ -82,9 +70,9 @@ omk sample skills/my-skill.md
82
70
  | 原生 Claude Code skill | ✓ | ✗ | ✗ | ✗ |
83
71
  | 托管 SaaS 看板 | ✗ | ✗ | ✓ | ✓ |
84
72
 
85
- omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 评委 ↔ 人工 α / 长度去偏不是 advanced flag,是默认行为。其他工具让你**手动**接置信区间;omk 让你**默认无法忽略**它。需要 SaaS 看板?选 LangSmith。要快速 prompt 迭代不要统计层?选 promptfoo。**要发到生产且会被问"为什么应该相信这个数字"?选 omk。**
73
+ omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 评委 ↔ 人工 α / 长度去偏不是 advanced flag,是默认行为。其他工具让你**手动**接置信区间;omk 让你**默认无法忽略**它。需要 SaaS 看板?选 LangSmith。要快速 prompt 迭代不要统计层?选 promptfoo。**要发到生产且会被问「为什么应该相信这个数字」?选 omk。**
86
74
 
87
- RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比(7 个工具 × 25+ 维度): [docs/zh/comparison.md](docs/zh/comparison.md)
75
+ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比(7 个工具 × 25+ 维度): [docs/zh/reference/comparison.md](docs/zh/reference/comparison.md)
88
76
 
89
77
  ## 特性
90
78
 
@@ -94,687 +82,60 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
94
82
  | **六维评估** | 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性独立展示 |
95
83
  | **多执行器** | 支持 Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini 及自定义命令 |
96
84
  | **21+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
97
- | **统计严谨性** | Bootstrap CI / Krippendorff α / 长度去偏 / 饱和曲线 —— 全部默认开。[详情 →](docs/zh/statistical-rigor.md) |
98
- | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖,自动继承长度去偏 |
99
- | **预算硬阈值** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` 总成本 + 单用例成本/耗时上限,超出中止保留 partial report |
100
- | **用例隔离 (construct validity)** | `--strict-baseline` (默认开) 三堵 baseline 拿到被测 skill 的污染路径:(1) SDK skill auto-discovery (2) subagent Skill 工具调用 (3) cwd 文件系统(避免 baseline 顺 `skills/<name>/` symlink 直接 Read 到 SKILL.md)。eval.yaml `allowedSkills` 支持 per-variant 白名单 |
101
- | **用例设计科学性 (sample design science)** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段(HF Dataset Cards 风),studio 输出 coverage 分桶 + `rubric_clarity_low` / `capability_thin` 两类 issue。`omk sample` 自动给生成的用例打 provenance。详见 [docs/sample-design-spec.md](docs/sample-design-spec.md),含 8 条行业 gap(HELM / MMLU-Pro / Construct Validity / IRT / Dataset Cards / Adversarial)的 omk v1 映射 |
85
+ | **统计严谨性** | Bootstrap CI / Krippendorff α / 长度去偏 / 饱和曲线 —— 全部默认开。[详情 →](docs/zh/explanation/statistical-rigor.md) |
86
+ | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖 |
87
+ | **LLM 健康度审计** | `omk doctor` 给 7 个内置维度独立打分;`--static-only` 可离线无 LLM 调用 |
88
+ | **线上 session 观测** | 解析 Claude Code session JSONL,测量各 skill 的失败率、耗时、token 成本、知识缺口信号 |
89
+ | **知识缺口识别** | 严重度加权的信号量化风险敞口,不宣称完备性 |
90
+ | **用例隔离 (construct validity)** | `--strict-baseline`(默认开)三堵 baseline 拿到被测 skill 的污染路径 |
91
+ | **用例设计科学性** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段(HF Dataset Cards 风),studio 输出 coverage 分桶 + `rubric_clarity_low` / `capability_thin` issue。[docs/specs/sample-design-spec.md](docs/specs/sample-design-spec.md) |
102
92
  | **多评委 ensemble** | `--judge-models claude:opus,openai:gpt-4o` 跨厂商评分 + agreement 度量 |
103
- | **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容(SSO 保护的知识库等) |
104
93
  | **盲测 A/B** | `--blind` 隐藏变体名称,HTML 报告有揭晓按钮 |
105
94
  | **多轮方差分析** | `--repeat N` 重复 N 次,计算均值/标准差/置信区间/t 检验 |
106
- | **并行执行** | `--concurrency N` 并行 N 个任务 |
107
- | **断言取反 + 组合** | 通用 `not: true` 字段 + `assert-set` (any/all) 任意嵌套 |
95
+ | **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容(SSO 保护的知识库等) |
108
96
  | **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本用例 |
109
97
  | **可追溯性** | 报告含 CLI 版本、Node 版本、artifact 版本指纹、judge prompt hash |
110
98
  | **中英切换** | HTML 报告右上角一键切换语言 |
111
99
 
112
- ## 工作原理
113
-
114
- 核心思路:**固定模型 + 固定样本,只变 artifact 和 runtime context**,通过交错调度消除时间漂移,用断言 + LLM 评委双通道评分,再叠加知识缺口信号量化风险敞口。
115
-
116
- ```mermaid
117
- flowchart TD
118
- subgraph Input["① 输入"]
119
- S["eval-samples<br/>(JSON / YAML)"]
120
- A["artifacts<br/>skills/*.md · SKILL.md<br/>baseline · git:name · @cwd"]
121
- end
122
-
123
- subgraph Prep["② 预处理(解析与抓取)"]
124
- V["变体解析<br/>variant → artifact + runtime context<br/>(cwd / 项目级 CLAUDE.md / 本地 skills)"]
125
- U["URL 抓取<br/>prompt / context 中的 URL<br/>MCP Server(私有文档) → HTTP"]
126
- end
127
-
128
- subgraph Schedule["③ 交错调度 + 并发"]
129
- Q["s1-v1 → s1-v2 → s2-v1 → s2-v2 …<br/>--concurrency N · --repeat N"]
130
- end
131
-
132
- subgraph Exec["④ 执行器(固定模型)"]
133
- E["claude / claude-sdk / codex / openai / gemini<br/>anthropic-api / openai-api / 自定义命令"]
134
- T["claude-sdk 抽取<br/>turns / toolCalls trace"]
135
- E -.-> T
136
- end
137
-
138
- subgraph Score["⑤ 双通道评分"]
139
- AS["断言(18 种)<br/>内容 / 结构 / 成本 / 延迟<br/>agent: tools_called · turns_min …"]
140
- LS["LLM 评委<br/>rubric · dimensions(多维独立打分)"]
141
- CS["综合分数<br/>断言 & LLM 有则均值"]
142
- AS --> CS
143
- LS --> CS
144
- end
145
-
146
- subgraph Analyze["⑥ 自动分析 + 知识缺口"]
147
- D["低区分度断言 / 均匀分 / 全通过全失败<br/>高成本样本 · 方差 · t 检验"]
148
- G["知识缺口信号<br/>(风险敞口量化, 不证明完备)"]
149
- end
150
-
151
- subgraph Report["⑦ 报告"]
152
- R["六维: 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性<br/>JSON + HTML · 顶部 verdict pill · 盲测揭晓<br/>CLI/Node/版本指纹可追溯"]
153
- end
154
-
155
- S --> U
156
- A --> V
157
- V --> Q
158
- U --> Q
159
- Q --> E
160
- T --> AS
161
- E --> AS
162
- E --> LS
163
- CS --> D
164
- CS --> G
165
- D --> R
166
- G --> R
167
- ```
168
-
169
- **关键设计:**
170
-
171
- - **交错调度**消除时间漂移:同一样本的不同 variant 交替发出,而非 v1 全跑完再跑 v2,避免模型负载/网络波动被错误归因给 artifact。
172
- - **variant = artifact + runtime context**:`name@cwd` 让对照组可以显式声明"项目目录"这个隐性输入,把"项目级沉淀"和"显式 artifact 注入"拆开测。
173
- - **双通道评分互补**:断言抓确定性缺陷(必须调用某工具/必须包含某字段),LLM 评委抓主观质量(可读性/完整性),两者都存在时取均值。
174
- - **知识缺口信号**不是评分的一部分,而是一个独立追踪项:它告诉你"这次评测覆盖了多少风险敞口",用于追踪收敛,而非断言知识"完备"。
175
-
176
- ## 评测样本格式
177
-
178
- 支持 JSON 和 YAML(`eval-samples.json`、`eval-samples.yaml`、`eval-samples.yml`)。
179
-
180
- ```json
181
- [
182
- {
183
- "sample_id": "s001",
184
- "prompt": "审查这段代码的安全性",
185
- "context": "function auth(u, p) { db.query('SELECT * FROM users WHERE name=' + u); }",
186
- "rubric": "应识别 SQL 注入风险并建议参数化查询",
187
- "assertions": [
188
- { "type": "contains", "value": "SQL 注入", "weight": 1 },
189
- { "type": "contains", "value": "参数化", "weight": 1 },
190
- { "type": "not_contains", "value": "没有问题", "weight": 0.5 }
191
- ],
192
- "dimensions": {
193
- "security": "是否识别出注入漏洞",
194
- "actionability": "是否给出可直接使用的修复代码"
195
- }
196
- }
197
- ]
198
- ```
199
-
200
- ### 字段说明
201
-
202
- | 字段 | 类型 | 必填 | 说明 |
203
- |------|------|------|------|
204
- | `sample_id` | `string` | **是** | 样本唯一标识 |
205
- | `prompt` | `string` | **是** | 发送给模型的用户提示词 |
206
- | `context` | `string` | 否 | 附加上下文(代码片段等),会被包裹在代码块中拼接到 prompt 后。也支持 URL,运行时自动抓取内容 |
207
- | `rubric` | `string` | 否 | LLM 评委的评分标准(1-5 分) |
208
- | `assertions` | `array` | 否 | 断言检查列表,详见[断言类型](#断言类型) |
209
- | `assertions[].type` | `string` | **是** | 断言类型 |
210
- | `assertions[].value` | `string\|number` | 视类型 | 检查值(`contains`、`min_length`、`cost_max` 等必填) |
211
- | `assertions[].values` | `array` | 视类型 | 字符串数组(`contains_all`、`contains_any` 必填) |
212
- | `assertions[].pattern` | `string` | 视类型 | 正则表达式(`regex` 必填) |
213
- | `assertions[].flags` | `string` | 否 | 正则标志(默认 `"i"`) |
214
- | `assertions[].schema` | `object` | 视类型 | JSON Schema 对象(`json_schema` 必填,基于 [ajv](https://ajv.js.org/)) |
215
- | `assertions[].reference` | `string` | 视类型 | 参考文本(`semantic_similarity` 必填) |
216
- | `assertions[].threshold` | `number` | 否 | 语义相似度通过阈值(默认 3) |
217
- | `assertions[].fn` | `string` | 视类型 | 自定义断言 JS 文件路径(`custom` 必填) |
218
- | `assertions[].weight` | `number` | 否 | 权重(默认 1) |
219
- | `dimensions` | `object` | 否 | 多维度评分,key 为维度名,value 为评分标准文本 |
220
-
221
- ### URL 自动抓取
222
-
223
- `prompt` 和 `context` 中的 URL 会在评测前自动抓取内容并内联到文本中。适用于引用在线文档、API 文档等场景:
224
-
225
- ```json
226
- {
227
- "sample_id": "s001",
228
- "prompt": "请根据以下 PRD 文档生成评测用例:https://wiki.example.com/prd/feature-x"
229
- }
230
- ```
231
-
232
- 运行时,URL 会被替换为实际文档内容。获取顺序:先通过 MCP Server 获取匹配的 URL(如 SSO 保护的私有文档),再通过 HTTP 获取剩余 URL。MCP 已成功的 URL 不会重复 HTTP 抓取。
233
-
234
- **私有文档 URL**:在项目目录放一个 `.mcp.json` 配置文件,或通过 `--mcp-config` 指定路径:
235
-
236
- ```json
237
- {
238
- "mcpServers": {
239
- "docs": {
240
- "command": "npx",
241
- "args": ["@example/docs-mcp-server"],
242
- "env": { "DOCS_API_TOKEN": "xxx" },
243
- "urlPatterns": ["docs.example.com"],
244
- "fetchTool": {
245
- "name": "fetch_doc",
246
- "urlTransform": {
247
- "regex": "docs\\.example\\.com/([^/]+/[^/]+)/([^/?#]+)",
248
- "params": { "namespace": "$1", "slug": "$2" }
249
- },
250
- "contentExtract": "data.body"
251
- }
252
- }
253
- }
254
- }
255
- ```
256
-
257
- **公网 URL**:直接 HTTP 获取,如果需要认证请确保命令行环境已配置好网络访问(VPN、代理等)。
258
-
259
- ### 评分策略
260
-
261
- #### 1. 断言评分
262
-
263
- 基于规则的本地检查,每个断言产生通过/失败结果。
264
-
265
- **计算方式:**
266
-
267
- - 通过率 = 通过断言的权重之和 / 总权重(0~1)
268
- - 分数 = 1 + 通过率 × 4(映射到 1~5 分)
269
- - 示例:3 个断言(权重各 1),2 个通过 → 通过率 = 2/3 → 分数 = 1 + 0.67 × 4 = **3.67**
270
-
271
- #### 2. Rubric / Dimensions 评分
272
-
273
- 评委模型(默认 `haiku`)按标准打 1-5 分。`dimensions` 模式下各维度独立评分后取平均。
274
-
275
- #### 3. 综合分数
276
-
277
- | 条件 | 公式 |
278
- |------|------|
279
- | 仅断言 | `assertionScore` |
280
- | 仅 LLM | `llmScore` |
281
- | 两者都有 | `(assertionScore + llmScore) / 2` |
282
- | 都没有 | `0` |
283
-
284
- ### 断言类型
285
-
286
- **确定性断言(21+ 种):**
287
-
288
- | 类型 | 说明 |
289
- |------|------|
290
- | `contains` / `not_contains` | 包含/不包含子串 |
291
- | `regex` | 正则匹配 |
292
- | `min_length` / `max_length` | 长度范围 |
293
- | `json_valid` / `json_schema` | JSON 校验 |
294
- | `starts_with` / `ends_with` | 前缀/后缀匹配 |
295
- | `equals` / `not_equals` | 精确匹配 |
296
- | `word_count_min` / `word_count_max` | 词数范围 |
297
- | `contains_all` / `contains_any` | 多值匹配 |
298
- | `cost_max` / `latency_max` | 成本/延迟限制 |
299
- | `tools_called` / `tools_not_called` / `tools_count_min` / `tools_count_max` | Agent 工具调用断言 |
300
- | `tool_output_contains` / `tool_input_contains` | 工具输入/输出内容匹配 |
301
- | `turns_min` / `turns_max` | 多轮对话轮数限制 |
302
- | `rouge_n_min` | ROUGE-N recall ≥ threshold(`reference` 字段填参考答案,`n` 默认 1,`threshold` 默认 0.5) |
303
- | `levenshtein_max` | 编辑距离 ≤ value(用于"输出跟参考几乎一致"场景) |
304
- | `bleu_min` | BLEU-4 ≥ threshold(unsmoothed,短文本会塌陷到 0) |
305
- | `faithfulness` | 输出是否被 `sample.context` 支持(反幻觉);LLM judge 1-5 评分,threshold 默认 3 |
306
- | `answer_relevancy` | 输出是否切题回答 `sample.prompt`;能抓住跑题、回避、冗余;threshold 默认 3 |
307
- | `context_recall` | `sample.context` 关键事实在输出中的覆盖率;`reference` 可显式指定 gold facts;threshold 默认 3 |
308
- | `semantic_similarity` | LLM 语义相似度(与 reference 的整体相似度,与 RAG 三 metric 互补) |
309
- | `custom` | 自定义 JS 函数(30s 超时) |
310
-
311
- **通用修饰:**
312
-
313
- 任何断言加 `not: true` 即反向(替代 `not_contains` / `not_equals` 等成对类型;老类型保留作 alias):
314
-
315
- ```yaml
316
- - type: regex
317
- pattern: "TODO|FIXME"
318
- not: true # 必须不含 TODO/FIXME
319
- ```
320
-
321
- **断言组合(— assert-set):**
322
-
323
- `assert-set` 类型让多个断言以 `any`(OR)或 `all`(AND)逻辑组合,可嵌套:
324
-
325
- ```yaml
326
- - type: assert-set
327
- mode: any # 任一通过即过 (mode: 'all' 则需全部通过)
328
- children:
329
- - { type: contains, value: "参数化" }
330
- - { type: contains, value: "prepared statement" }
331
- - { type: regex, pattern: "bind\\(.*\\?" }
332
- ```
333
-
334
- 子断言可独立带 `not: true`;嵌套 assert-set 可表达任意布尔逻辑。
335
-
336
- ### 自定义断言
337
-
338
- ```js
339
- // my-assertion.mjs
340
- export default function(output, { sample, assertion }) {
341
- return { pass: output.includes('SQL'), message: '检查了 SQL 关键字' };
342
- }
343
- ```
344
-
345
- ## 六维评估指标
346
-
347
- 评测报告从六个维度独立展示结果。其中评分三层(事实 / 行为 / LLM 评价)分开展示,让你看到**是哪一层拉胯**,而不是只看到一个合成分:
348
-
349
- | 维度 | 指标 | 说明 |
350
- |------|------|------|
351
- | 📋 **事实** | 事实类断言通过率 | `contains` / `json_schema` / `fact_check` 等规则可验证断言的 1-5 分映射 |
352
- | 🛠️ **行为** | 行为类断言通过率 | `tools_called` / `tool_output_contains` / `turns_max` 等执行合规类断言 |
353
- | 💬 **LLM 评价** | rubric 评分 | 由评委模型按预先写好的评分规则(rubric)打的 1-5 分,主观但能抓规则断言之外的"整体好不好" |
354
- | 💰 **成本** | 总成本、输入/输出 Token 数 | 基于 Token 消耗和模型定价的 API 费用 |
355
- | ⚡ **效率** | 平均延迟 (ms) | 从发送请求到收到完整响应的端到端耗时 |
356
- | 🛡️ **稳定性** | CV(变异系数) | 跨重复运行(`--repeat ≥ 2`)分数一致性;单轮评测显示 `—`,**诚实交代测不到什么** |
357
-
358
- ## CLI 参考
359
-
360
- omk 的公开 CLI 由 7 个顶层命令构成完整闭环:`init`(脚手架)·`doctor`(静态检查)·`eval`(离线 A/B 评测)·`observe`(线上 trace 观测)·`evolve`(多轮自动迭代 skill)·`sample`(生成或补齐评测用例)·`studio`(本地 Web 工作台,看报告 / 分析)。
361
-
362
- ### `omk init`
363
-
364
- ```bash
365
- omk init [目录]
366
- ```
367
-
368
- 生成一个评测项目脚手架,包含两版 starter skill 和 `eval-samples.json`。
369
-
370
- ### `omk doctor`
371
-
372
- ```bash
373
- omk doctor # 体检当前目录或 ./skills
374
- omk doctor skills/v1.md # 体检单个 skill
375
- omk doctor skills/ --html report.html # 产 HTML 可视化报告
376
- omk doctor skills/ --json > r.json # JSON 给 CI / 外部工具消费
377
- omk doctor --gate; echo $? # 静默门禁,fatal 问题 exit 1,警告不阻断
378
- omk doctor --static-only # 离线模式:只跑静态检查,不调 LLM
379
- ```
380
-
381
- LLM 健康度审计:单次 LLM 会话产出 7 个内置维度的健康度评分 + findings + 改进建议;HTML 报告按 fail→warn→pass→skipped 排序,错误 finding 优先。维度可扩展(在自己代码里调 `registerHealthDimension`,自动并入同一次 LLM 调用的 prompt 与报告,顺序 = 注册顺序)。
382
-
383
- 离线静态模式(`--static-only`):CI 节点没装 claude / codex、本地断网调试等场景下跑 4 条静态 rule(可读性 / 元数据 / 依赖 / samples 契约),零 LLM 调用、零成本。结果同样进 `DoctorReport`,可与 `--json` / `--gate` / `--html` 组合。
384
-
385
- `omk eval` 内部继续跑静态 readability / metadata / dependency / samples 契约 gate 保护评测质量,这条路径与用户入口的 `omk doctor` 角色分离,互不干扰。
386
-
387
- ### `omk eval`
388
-
389
- ```bash
390
- omk eval --control baseline --treatment my-skill # 单 skill 必要性测试(baseline 是保留 variant,代表「不注入 skill」)
391
- omk eval --control code-review-v1 --treatment code-review-v2 # 多版本 A/B
392
- omk eval --config eval.yaml
393
- omk eval --batch
394
- omk eval gold compare <report-id> --gold-dir gold-dataset
395
- ```
396
-
397
- 运行离线评测,应用 verdict gate,持久化报告,并用 exit code 表示 ship/no-ship。这个工作流默认开启 bootstrap CI。
398
-
399
- 常用选项:
400
-
401
- ```text
402
- --samples <路径> 样本文件(默认:eval-samples.json,也自动检测 .yaml/.yml;--skill-dir 下的 <skill>/.omk/samples.json 会被自动发现)
403
- --skill-dir <路径> artifact 目录(默认:skills)
404
- --control <expr> 对照组 variant 表达式
405
- --treatment <v1,v2> 实验组 variants,逗号分隔
406
- --config <路径> YAML/JSON 评测配置
407
- --model <名称> 任务执行模型(默认:opus alias)
408
- --effort <level> 执行模型扩展思考预算 low/medium/high/xhigh/max(默认:low;高 effort 跨报告不可严格比较)
409
- --judge-models <list> 评委配置,例如 claude:haiku 或 claude:opus,openai:gpt-4o
410
- --executor <名称> claude / claude-sdk / codex / codex-sdk / openai-api / gemini / custom
411
- --no-judge 跳过 LLM 评委主观评分
412
- --no-diagnostic 关闭 diagnostic 诊断 LLM 调用(默认开启,给 failed sample 出"哪错了 + 怎么改"建议)
413
- --skip-doctor escape hatch:跳过 doctor 健康检查门禁(默认强制启用)。沙箱 mock 提供依赖时绕开 doctor 物理路径误报;garbage-in 风险由调用方承担
414
- --dry-run 仅预览
415
- --blind 盲测模式
416
- --concurrency <n> 并行任务数
417
- --timeout <秒> 单任务超时
418
- --repeat <n> 重复 N 次做方差分析
419
- --batch 每个 artifact 独立和 baseline 对比
420
- --bootstrap-samples N bootstrap 重采样次数(默认 1000)
421
- --threshold <number> verdict 分层门禁阈值(默认 3.5)
422
- --trivial-diff <num> 实际可忽略 diff 阈值(默认 0.1)
423
- --report-only 生成报告并打印 verdict,但始终 exit 0
424
- --no-gate --report-only 的别名
425
- --skip-connectivity 跳过模型连通性检查(内部静态 gate 仍强制执行)
426
- --no-serve 评测后不自动启动报告服务
427
- ```
428
-
429
- HTML 报告有两个 tab:
430
- - **📊 评分视角** — verdict 驱动的 A/B 对比(事实/行为/judge 三层、bootstrap CI、length-debias)。
431
- - **✅ 功能视角** — 每条 sample 当一条单测看:用例设计(prompt / rubric / 工具调用 mock / environment)+ 执行轨迹 + 断言结果 + 可操作的 diagnostic 建议。诊断给出归因(skill 文档模糊 / LLM 误读 / sample 设计 bug / 诱错样本 / ...)、工作流校验(rubric 每步 ✓/✗ + 证据)和失败模式标签(工作流跳步 / 硬编码值 / 幻觉输出 / 工具误用 / 环境拦截 / 误读约束 / 其他)。沙箱 mock 字段语义(`mocks` / `environment` / `tripwire` / `mocksStrict`)见 [docs/sample-design-spec.md §三](./docs/sample-design-spec.md)。
432
-
433
- ### `omk observe`
434
-
435
- omk observe 提供两条工作流:默认的 skill 健康度报告,以及 reviewer 闭环用的 observe inbox。
436
-
437
- #### A. skill 健康度报告(默认)
438
-
439
- ```bash
440
- omk observe ~/.claude/projects/-Users-you-Documents-my-project
441
- omk observe ~/.claude/projects/my-project --last 7d
442
- omk observe ~/.claude/projects/my-project --from 2026-04-01T00:00:00Z --to 2026-04-15T23:59:59Z
443
- omk observe ~/.claude/projects/my-project --skills audit,polish
444
- omk observe ~/.claude/projects/my-project --kb /path/to/project
445
- ```
446
-
447
- 把真实 Claude Code session trace 转成 skill 健康度报告:知识使用、gap 信号、执行稳定性、token 和耗时。这是生产观测,不是生产评分。
448
-
449
- #### B. observe inbox:reviewer 闭环
450
-
451
- 把真实 session trace 解析、聚合、降噪,输出可逐条 review 的 observation 列表。整条链路纯本地、零 LLM。
452
-
453
- ```bash
454
- # 1. 把 trace 解析、聚合、落盘到 .omk/observations/
455
- omk observe ingest ~/.claude/projects/my-project
456
- omk observe ingest ~/.claude/projects/my-project --output-dir ./custom-dir
457
-
458
- # 2. 看 inbox(默认 top 20,按 severity / confidence / lastSeen 排序)
459
- omk observe inbox
460
- omk observe inbox --limit 50
461
- omk observe inbox --skill audit # 只看某个 skill
462
- omk observe inbox --by-skill # 按 skill 资产视图
463
- omk observe inbox --explore 10 # 从 medium / low 桶抽 10 条长尾
464
- omk observe inbox --explore 10 --include-noise # 显式包含 noise 桶
465
- omk observe inbox --json # JSON 输出,便于自动化消费
466
-
467
- # 3. 反向查单条 observation 的事件三元组(前后 message 上下文)
468
- omk observe show <inbox_id>
469
- ```
470
-
471
- 每条 observation 自带:
472
-
473
- + `confidence` 与 `attributionConfidence`:信号可信度 + skill 归因可信度,并列展示
474
- + `severityReasonCode`:判断为该 severity 的稳定结构化原因;人类可读说明由 CLI / studio 渲染时生成
475
- + `messageWindow`:前 3 条 / 触发点 / 后 3 条 message 上下文 + `resolutionAfter`(后续是否解决)
476
- + `evidence.{messageIndex,messageUuid,toolUseId}`:可反向回到原始 jsonl 的锚点
477
-
478
- 支持 trace 格式:Claude Code session JSONL(`.jsonl`)、OpenClaw session JSONL(`.jsonl`)、markdown 对话日志(`.log`)。
479
-
480
- ### `omk evolve`
481
-
482
- ```bash
483
- omk evolve <skill> # 多轮自动迭代 skill
484
- omk evolve skills/foo.md --rounds 10 --target 4.5
485
- ```
486
-
487
- 让 skill 跑 eval → judge → 改写 SKILL.md 的多轮闭环,直到达到 `--target` 或 `--rounds` 上限。耗时按 `轮数 × 样本 × 变体` 累加,几分钟到几十分钟级别。原始 skill 文件版本保存在 `skills/evolve/*.r0.md`。
488
-
489
- ### `omk sample`
490
-
491
- ```bash
492
- omk sample <skill> # 为单个 skill 生成或补齐评测用例
493
- omk sample --batch # 为目录下缺评测集的 skill 批量生成
494
- ```
495
-
496
- 一次性生成。自动给生成的用例打 `provenance`。生成的 assertions 使用英文 / 数字 / 代码 token,便于跨中英文输出对比。
497
-
498
- ### `omk studio`
499
-
500
- ```bash
501
- omk studio
502
- omk studio --port 7799
503
- omk studio --host 0.0.0.0 # 局域网访问(默认 127.0.0.1)
504
- omk studio --reports-dir ~/.oh-my-knowledge/reports
505
- omk studio --observations-dir .omk/observations # observe inbox 数据目录
506
- omk studio --no-open
507
- ```
508
-
509
- 启动本地知识工作台浏览报告。verdict、样本回退、跨样本 diff、饱和曲线、单样本 drill-down 全部在 studio UI 里 —— omk 不提供 CLI 导出 / 分析子命令。CI gate 用 `omk eval` 的 exit code(PROGRESS 退 0、其他非 0),需要文字摘要自己 `jq` report JSON。
510
-
511
- Studio 是 skill-centric 信息架构 — 列表页(`/`)按 skill 卡片展示健康等级 / 0-100 参考分 / 待优化数 / 趋势,详情页(`/skills/<name>`)左栏列关键问题清单(skill 优化 / 用例优化 / 工具反馈三档),右栏画 chart.js 健康趋势 + 三个紧凑阶段卡(doctor / eval / observe),细节走 modal。旧的 run 列表挪到 `/runs`。访问 `/observations/inbox` 查看 observe inbox 看板:按 skill 资产视图(rollup)+ reviewer 待办建议 + 当前可观测漏斗 + 单 observation 详情面板(含事件三元组)。
512
-
513
- ## 执行器
514
-
515
- ### 内置执行器
516
-
517
- | 执行器 | 适用场景 | 说明 |
518
- |--------|----------|------|
519
- | `claude` | 默认 | 通过 `claude -p` 调用 Claude CLI |
520
- | `claude-sdk` | 结构化输出 | 通过 Claude Agent SDK 调用,无 stdout 解析,避免 buffer 截断 |
521
- | `codex` | OpenAI agent CLI | 通过 `codex exec --json` 调用,需本地装好登录的 codex(`@openai/codex`);best-effort tool trace,**costUSD 不报**(codex 自身不输出 USD,需外部账单核算) |
522
- | `codex-sdk` | OpenAI agent SDK | 通过 `@openai/codex-sdk` 调用其自带的 `@openai/codex` binary 和 SDK 事件流;**costUSD 不报** |
523
- | `gemini` | 跨厂商对比 | 通过 `gemini` CLI 调用 |
524
- | `anthropic-api` | 无需 CLI | 直接调用 Anthropic HTTP API(需 `ANTHROPIC_API_KEY`) |
525
- | `openai-api` | 无需 CLI | 直接调用 OpenAI HTTP API(需 `OPENAI_API_KEY`) |
526
-
527
- API 直调执行器支持通过环境变量自定义 Base URL:`ANTHROPIC_BASE_URL`、`OPENAI_BASE_URL`。
528
-
529
- Codex construct-validity 说明:(1)`codex` 使用 `PATH` 上找到的 `codex` binary;`codex-sdk` 使用 `@openai/codex-sdk` 解析到的自带 `@openai/codex` binary。报告会持久化 per-variant `meta.executorRuntimes`、`meta.executorRuntime`,以及每个评委的 `meta.judgeModels[].runtime` 指纹(binary 或 SDK 版本 + 能力快照),strict comparability checks 会在 runtime 指纹无法审计时提示。runtime 指纹不一致时,结果应解释为 executor runtime 对比,而不只是 prompt/template 行为对比。(2)两个 executor 都隔离用户级 config:`codex` 传 `--ephemeral` + `--ignore-user-config`,`codex-sdk` 把 `$CODEX_HOME` 重定向到 per-process tmp 目录(auth.json 通过 symlink 透传)。用户的 `~/.codex/config.toml` 不会渗入任意一个 executor 的 eval。
530
-
531
- ### 自定义执行器
532
-
533
- 任何 shell 命令都可以作为执行器,通过 stdin/stdout JSON 协议通信:
534
-
535
- ```bash
536
- omk eval --executor "python my_provider.py"
537
- omk eval --executor "./my-executor.sh"
538
- ```
539
-
540
- **协议约定:**
541
-
542
- - **输入**(stdin):JSON `{"model":"...","system":"...","prompt":"..."}`
543
- - **输出**(stdout):JSON `{"output":"模型回复","inputTokens":0,"outputTokens":0,"costUSD":0}`
544
- - stdout 中只需返回有值的字段,其余默认为 0;也可以直接输出纯文本(不解析 token/成本)
545
- - 非零退出码视为执行失败
546
-
547
- ### Artifact 目录结构
548
-
549
- 默认执行器(claude/openai/gemini)支持两种 artifact 布局,同一次评测中可混用:
550
-
551
- ```
552
- skills/
553
- ├── v1.md # 方式一:直接放 .md 文件
554
- └── my-skill/ # 方式二:完整 artifact 目录
555
- ├── SKILL.md # 工具自动读取此文件作为 system prompt
556
- ├── config.json # 其他文件不参与评测,仅保留完整性
557
- └── scripts/
558
- ```
559
-
560
- **Variant 解析规则:**
561
-
562
- `variant` 是实验分组表达式。解析之后,OMK 会得到一个 `artifact` 与可选的 `runtime context`(当前主要是 `cwd`)。
563
-
564
- | 格式 | 含义 |
565
- |------|------|
566
- | `name` | 从 artifact 目录查找 `name.md` 或 `name/SKILL.md`,解析为一个 artifact |
567
- | `baseline` | 空 artifact,不使用 system prompt;可直接理解为“什么都没有” |
568
- | `project-env@/path/to/project` | 空 artifact,但在指定项目目录运行,用于单独观察项目级 runtime context |
569
- | `git:name` | 从 git HEAD 读取一个 artifact 的上次提交版本 |
570
- | `git:ref:name` | 从 git 指定 commit 读取一个 artifact |
571
- | `./path/to/file.md` | 含 `/` 的路径,直接读取文件作为 artifact |
572
- | `variant@/path/to/project` | 给任意变体附加运行目录,支持 `name@cwd`、`git:name@cwd`、`/file.md@cwd` |
573
-
574
- `--control` 和 `--treatment` 都不传时,用 `--config eval.yaml` 或 `--batch`。`--batch` 模式下会自动用 `baseline` 作对照组,每个被发现的 artifact 作实验组。
575
-
576
- ```bash
577
- # 显式:一个 control,一个或多个 treatment
578
- omk eval --control v1 --treatment v2
579
- omk eval --control baseline --treatment v1,v2,v3
580
-
581
- # 对比空 artifact 和显式 artifact 的效果差异
582
- omk eval --control baseline --treatment my-skill
583
-
584
- # 单独观察项目级 runtime context 的影响(用自描述标签)
585
- omk eval --control baseline --treatment project-env@/path/to/target-project
586
-
587
- # 对比"项目级 runtime context"与"显式 artifact 注入"
588
- omk eval \
589
- --control project-env@/path/to/target-project \
590
- --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
591
-
592
- # 对比修改前后(旧版本从 git 历史读取)
593
- omk eval --control git:my-skill --treatment my-skill
594
-
595
- # 直接指定文件路径
596
- omk eval --control ./old-skill.md --treatment ./new-skill.md
597
-
598
- # 配置文件驱动(evaluation-as-code)
599
- omk eval --config eval.yaml
600
- ```
601
-
602
- **前置要求:**
603
-
604
- - **claude**:安装 [Claude Code](https://claude.ai/code) 并认证
605
- - **claude-sdk**:安装 [Claude Code](https://claude.ai/code) 并认证(使用 Agent SDK,无需 CLI stdout 解析)
606
- - **anthropic-api**:设置 `ANTHROPIC_API_KEY` 环境变量
607
- - **openai**:`pip install openai` 并设置 `OPENAI_API_KEY`
608
- - **openai-api**:设置 `OPENAI_API_KEY` 环境变量
609
- - **gemini**:`npm i -g @google/gemini-cli` 并认证
610
-
611
- ### Agent 评测与项目级 Runtime Context
612
-
613
- 当执行器使用 `claude-sdk` 时,OMK 现在已经支持第一版 agent-aware evaluation。
614
-
615
- 这里建议把几个概念分开理解:
616
-
617
- - `artifact`:被评测对象,例如 baseline、skill、prompt、agent
618
- - `variant`:CLI 里的实验分组表达式
619
- - `runtime context`:运行时上下文,当前主要是 `cwd`;在项目型 agent 场景下,它就包含项目目录、`CLAUDE.md`、本地 skills 等会影响行为的环境因素
620
-
621
- 在 OMK 里,`agent` 不是所有对象的总称,`skill` 也不是所有对象的总称。更稳妥的说法是:你在比较不同 artifact 在不同 runtime context 下的表现。
622
-
623
- - 自动抽取 turns / toolCalls trace
624
- - 支持基于工具调用行为的断言
625
- - 支持在指定 `cwd` 下运行,让 Claude Code 自动加载项目内的 `CLAUDE.md`、skills 和本地 runtime context
626
-
627
- #### 推荐执行器
628
-
629
- ```bash
630
- omk eval --executor claude-sdk
631
- ```
632
-
633
- #### 支持的 agent 相关断言
634
-
635
- | 断言 | 含义 |
636
- |------|------|
637
- | `tools_called` | 必须调用指定工具 |
638
- | `tools_not_called` | 禁止调用指定工具 |
639
- | `tools_count_min` / `tools_count_max` | 工具调用次数上下界 |
640
- | `tool_output_contains` | 指定工具输出必须包含关键内容 |
641
- | `turns_min` / `turns_max` | 交互轮次上下界 |
642
-
643
- #### 三种常见对照组
644
-
645
- **1. 裸模型 baseline**
646
-
647
- 不注入 system prompt,也不进入带知识的项目目录。至少需要一个 treatment 做对比:
648
-
649
- ```bash
650
- omk eval \
651
- --executor claude-sdk \
652
- --control baseline \
653
- --treatment my-skill
654
- ```
655
-
656
- **2. 空 artifact + 项目级 runtime context**
657
-
658
- 不注入 system prompt,但在项目目录运行。它不是严格意义上的"裸 baseline",而是"空 artifact + 项目级 runtime context"。
659
-
660
- ```bash
661
- omk eval \
662
- --executor claude-sdk \
663
- --control baseline \
664
- --treatment project-env@/path/to/target-project
665
- ```
666
-
667
- **3. 显式 artifact 注入**
668
-
669
- 直接把某个外部 `SKILL.md` 作为 artifact 注入,同时保留项目目录上下文。适合对比"项目级 runtime context"与"显式单 artifact 注入"之间的差异。
670
-
671
- ```bash
672
- omk eval \
673
- --executor claude-sdk \
674
- --control project-env@/path/to/target-project \
675
- --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
676
- ```
677
-
678
- #### 推荐的第一轮对照设计
679
-
680
- 对于 PRD / 复杂业务知识场景,建议从下面开始:
681
-
682
- ```bash
683
- omk eval \
684
- --executor claude-sdk \
685
- --samples skills/evaluate-review/eval-samples.yaml \
686
- --control baseline \
687
- --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
688
- ```
689
-
690
- 如果你想证明"项目目录中的知识沉淀本身"是否有效,加第二个 treatment:
691
-
692
- ```bash
693
- omk eval \
694
- --executor claude-sdk \
695
- --samples skills/evaluate-review/eval-samples.yaml \
696
- --control baseline \
697
- --treatment project-env@/path/to/target-project,/path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
698
- ```
699
-
700
- #### 设计建议
701
-
702
- - **先用 `--dry-run`**:确认样本、variant 和 `cwd` 被正确解析
703
- - **项目级对照必须区分 `cwd`**:相同 prompt 在不同项目目录下会走不同 runtime context
704
- - **优先先跑 PRD 场景**:相比 Coding,更容易验证知识完整性、影响面识别和业务正确性
705
-
706
- ### 常见模型配置示例
707
-
708
- **没有 Claude?** 大多数国产模型(GLM、通义千问、Moonshot、DeepSeek 等)都兼容 OpenAI API 格式,可以直接使用 `openai-api` 执行器:
709
-
710
- ```bash
711
- # GLM(智谱)
712
- export OPENAI_API_KEY="你的智谱 API Key"
713
- export OPENAI_BASE_URL="https://open.bigmodel.cn/api/paas/v4"
714
- omk eval --executor openai-api --model glm-4-plus \
715
- --judge-models openai-api:glm-4-plus --no-cache
716
-
717
- # 通义千问
718
- export OPENAI_API_KEY="你的通义 API Key"
719
- export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
720
- omk eval --executor openai-api --model qwen-plus \
721
- --judge-models openai-api:qwen-plus
722
-
723
- # DeepSeek
724
- export OPENAI_API_KEY="你的 DeepSeek API Key"
725
- export OPENAI_BASE_URL="https://api.deepseek.com"
726
- omk eval --executor openai-api --model deepseek-chat \
727
- --judge-models openai-api:deepseek-chat
728
-
729
- # Moonshot(Kimi)
730
- export OPENAI_API_KEY="你的 Moonshot API Key"
731
- export OPENAI_BASE_URL="https://api.moonshot.cn/v1"
732
- omk eval --executor openai-api --model moonshot-v1-8k \
733
- --judge-models openai-api:moonshot-v1-8k
734
- ```
735
-
736
- **Ollama 本地模型:**
737
-
738
- ```bash
739
- omk eval --executor "python examples/custom-executor/ollama-executor.py" \
740
- --model llama3 --no-judge
741
- ```
742
-
743
- **关于评委:**
100
+ ## 文档
744
101
 
745
- - `--judge-models <list>` 指定评委,格式 `executor:model[,executor:model]`。默认 `${executor}:haiku`(没设 `--executor` 时为 `claude:haiku`)
746
- - 1 条 = 单评委;≥ 2 条 = 多评委 ensemble + inter-judge agreement
747
- - 没有 Claude 时把 `--judge-models` 指向你可用的模型,例如 `--judge-models openai-api:glm-4-plus`
748
- - 加 `--no-judge` 可跳过 LLM 评委,仅使用断言评分
102
+ - **[工作原理](docs/zh/explanation/architecture.md)** —— 交错调度、variant 解析、双通道评分、六维报告
103
+ - **[评测样本格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、21+ 断言类型、自定义 JS 断言
104
+ - **[CLI 参考](docs/zh/reference/cli.md)** —— 7 个命令的 bash 示例和 flag 表
105
+ - **[执行器与 artifact 布局](docs/zh/reference/executors.md)** —— 内置 / 自定义执行器、Agent 评测、常见模型配置(Claude / OpenAI / GLM / 通义 / DeepSeek / Moonshot / Ollama)
106
+ - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
107
+ - **[用例设计规范](docs/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据;行业 gap 映射
108
+ - **[统计严谨性](docs/zh/explanation/statistical-rigor.md)** —— 为什么 Bootstrap CI / α / 长度去偏 / 饱和曲线重要
109
+ - **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / LangSmith / Langfuse / Braintrust 等 25+ 维度横评
749
110
 
750
111
  ## 环境变量
751
112
 
752
113
  | 变量 | 说明 |
753
114
  |------|------|
754
- | `CCV_PROXY_URL` | 将请求代理到 cc-viewer,实时可视化评测流量 |
755
- | `OMK_REPORT_PORT` | 报告服务端口(默认:7799) |
115
+ | `CCV_PROXY_URL` | 通过 cc-viewer 代理请求,实时可视化评测流量 |
116
+ | `OMK_REPORT_PORT` | 报告服务端口(默认 7799) |
756
117
 
757
118
  ## 系统要求
758
119
 
759
120
  - Node.js >= 20
760
- - `claude` CLI(用于默认执行器和 LLM 评委,安装方式见 [Claude Code](https://claude.ai/code))
761
- - 使用其他执行器(openai/gemini)且加 `--no-judge` 时可不装
121
+ - `claude` CLI(默认执行器和 LLM 评委需要,参考 [Claude Code](https://claude.ai/code))
122
+ - 如果使用其它执行器(openai / gemini)+ `--no-judge` 则可不需要
762
123
 
763
124
  ## 安全说明
764
125
 
765
126
  本工具设计用于**本地可信环境**(开发机、CI 流水线)。以下功能会执行本地代码,请确保输入来源可信:
766
127
 
767
- | 功能 | 风险说明 | 适用范围 |
768
- |------|----------|----------|
769
- | **自定义断言** (`custom`) | 动态加载并执行用户指定的 `.mjs` 文件 | 仅使用自己编写或审查过的断言文件 |
770
- | **eval-samples.json** | 断言配置中可引用外部文件路径 | 不要使用不可信来源的样本文件 |
128
+ | 功能 | 风险 | 适用场景 |
129
+ |------|------|----------|
130
+ | **自定义断言**(`custom`) | 动态加载并执行用户指定的 `.mjs` 文件 | 仅使用自己编写或审查过的断言文件 |
131
+ | **eval-samples.json** | 断言配置可引用外部文件路径 | 不要使用来源不明的样本文件 |
771
132
 
772
133
  **建议:**
773
134
 
774
- - 不要在公网服务中暴露本地报告服务(无认证)
775
- - 不要用不可信的第三方 eval-samples 文件
776
- - 自定义断言有 30 秒执行超时,但无沙箱隔离
135
+ - 不要将本地报告服务暴露到公网(无身份认证)
136
+ - 不使用未经审查的第三方 eval-samples
137
+ - 自定义断言有 30 秒超时,但无沙箱隔离
777
138
 
778
139
  ---
779
140
 
780
- 版本变更记录见 [GitHub Releases](https://github.com/lizhiyao/oh-my-knowledge/releases)。欢迎贡献 — 详见 [CONTRIBUTING](./CONTRIBUTING.md)。
141
+ 发布日志见 [GitHub Releases](https://github.com/lizhiyao/oh-my-knowledge/releases)。欢迎贡献 —— 见 [CONTRIBUTING](./CONTRIBUTING.md)。