@hassangameryt/freeflow 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (692) hide show
  1. package/.claude-plugin/marketplace.json +4 -4
  2. package/.claude-plugin/plugin.json +6 -2
  3. package/.codex-plugin/plugin.json +9 -9
  4. package/README.md +190 -127
  5. package/assets/feedback-loop.png +0 -0
  6. package/assets/freeflow_workflow_map.png +0 -0
  7. package/assets/workflow.png +0 -0
  8. package/{skills → capabilities}/output-router/SKILL.md +21 -17
  9. package/{skills → capabilities}/output-router/references/safety-policy.md +1 -1
  10. package/capabilities/output-router/references/setup.md +110 -0
  11. package/command-surface.json +62 -7
  12. package/hooks/freeflow-runtime-context.mjs +572 -154
  13. package/hooks/hooks.json +26 -1
  14. package/hooks/tests/check-runtime-context-hook.sh +375 -0
  15. package/package.json +24 -24
  16. package/pi-extension/dist/index.js +264 -80
  17. package/pi-extension/dist/integrations/mcp-config.js +29 -0
  18. package/pi-extension/dist/output-router/host-producer-identification.js +232 -0
  19. package/pi-extension/dist/output-router/native-safety-net.js +216 -0
  20. package/pi-extension/dist/output-router/observed-tool-routing.js +122 -0
  21. package/pi-extension/dist/output-router/renderers.js +1023 -0
  22. package/pi-extension/dist/output-router/router-tools.js +554 -0
  23. package/pi-extension/dist/output-router/schemas.js +383 -0
  24. package/pi-extension/dist/output-router/status.js +744 -0
  25. package/pi-extension/dist/output-router/utils.js +502 -0
  26. package/pi-extension/dist/runtime/runtime-context.js +798 -0
  27. package/pi-extension/dist/settings/settings-tui.js +314 -0
  28. package/pi-extension/dist/settings/settings-ui.js +1487 -0
  29. package/pi-extension/freeflow/index.js +1 -0
  30. package/pi-extension/src/index.ts +230 -56
  31. package/pi-extension/src/{host-producer-identification.ts → output-router/host-producer-identification.ts} +37 -7
  32. package/pi-extension/src/{native-safety-net.ts → output-router/native-safety-net.ts} +8 -12
  33. package/pi-extension/src/{observed-tool-routing.ts → output-router/observed-tool-routing.ts} +19 -11
  34. package/pi-extension/src/{renderers.ts → output-router/renderers.ts} +246 -55
  35. package/pi-extension/src/{router-tools.ts → output-router/router-tools.ts} +130 -29
  36. package/pi-extension/src/{schemas.ts → output-router/schemas.ts} +171 -43
  37. package/pi-extension/src/output-router/status.ts +791 -0
  38. package/pi-extension/src/{utils.ts → output-router/utils.ts} +80 -18
  39. package/pi-extension/src/runtime/runtime-context.ts +895 -0
  40. package/pi-extension/src/settings/settings-tui.ts +408 -0
  41. package/pi-extension/src/settings/settings-ui.ts +1730 -0
  42. package/pi-extension/tests/integration/pi-extension.test.js +4020 -0
  43. package/pi-extension/tests/output-router/observed-producers.test.js +132 -0
  44. package/pi-extension/tests/output-router/search.test.js +722 -0
  45. package/pi-extension/tests/settings/settings-tui.test.js +243 -0
  46. package/router/dist/benchmarks/benchmark-harness.d.ts +16 -13
  47. package/router/dist/benchmarks/benchmark-harness.js +62 -62
  48. package/router/dist/benchmarks/benchmarks.d.ts +80 -73
  49. package/router/dist/benchmarks/benchmarks.js +860 -761
  50. package/router/dist/benchmarks/codex-qa-benchmarks.d.ts +71 -67
  51. package/router/dist/benchmarks/codex-qa-benchmarks.js +429 -390
  52. package/router/dist/benchmarks/command-benchmarks.d.ts +107 -92
  53. package/router/dist/benchmarks/command-benchmarks.js +616 -571
  54. package/router/dist/benchmarks/context-mode-normalized-benchmarks.d.ts +64 -56
  55. package/router/dist/benchmarks/context-mode-normalized-benchmarks.js +679 -613
  56. package/router/dist/benchmarks/context-mode-real-deep-benchmark.d.ts +103 -95
  57. package/router/dist/benchmarks/context-mode-real-deep-benchmark.js +1188 -1103
  58. package/router/dist/benchmarks/context-saving-baseline.d.ts +47 -41
  59. package/router/dist/benchmarks/context-saving-baseline.js +405 -374
  60. package/router/dist/benchmarks/index-benchmarks.d.ts +74 -69
  61. package/router/dist/benchmarks/index-benchmarks.js +761 -661
  62. package/router/dist/benchmarks/storage-policy-benchmarks.d.ts +80 -69
  63. package/router/dist/benchmarks/storage-policy-benchmarks.js +549 -469
  64. package/router/dist/benchmarks/vault-index-benchmarks.d.ts +43 -34
  65. package/router/dist/benchmarks/vault-index-benchmarks.js +294 -229
  66. package/router/dist/config/config.d.ts +62 -55
  67. package/router/dist/config/config.js +552 -373
  68. package/router/dist/config/router-contract.d.ts +2 -2
  69. package/router/dist/config/router-contract.js +38 -38
  70. package/router/dist/config/schema.d.ts +19 -10
  71. package/router/dist/config/schema.js +586 -538
  72. package/router/dist/config/types.d.ts +413 -361
  73. package/router/dist/config/types.js +19 -8
  74. package/router/dist/evidence/bounded-evidence.d.ts +26 -26
  75. package/router/dist/evidence/bounded-evidence.js +265 -252
  76. package/router/dist/evidence/evidence-range-selector.d.ts +16 -14
  77. package/router/dist/evidence/evidence-range-selector.js +192 -191
  78. package/router/dist/evidence/evidence-search.d.ts +18 -16
  79. package/router/dist/evidence/evidence-search.js +401 -377
  80. package/router/dist/evidence/evidence.d.ts +17 -17
  81. package/router/dist/evidence/evidence.js +163 -159
  82. package/router/dist/evidence/failure-contracts.d.ts +25 -15
  83. package/router/dist/evidence/failure-contracts.js +132 -126
  84. package/router/dist/evidence/line-ranges.d.ts +17 -15
  85. package/router/dist/evidence/line-ranges.js +18 -18
  86. package/router/dist/experiments/local-index.d.ts +47 -41
  87. package/router/dist/experiments/local-index.js +477 -458
  88. package/router/dist/local/local-traversal.d.ts +18 -0
  89. package/router/dist/local/local-traversal.js +334 -0
  90. package/router/dist/processing/engine.d.ts +94 -70
  91. package/router/dist/processing/engine.js +462 -364
  92. package/router/dist/processing/reducers.d.ts +208 -194
  93. package/router/dist/processing/reducers.js +1434 -1256
  94. package/router/dist/processing/renderers.d.ts +24 -18
  95. package/router/dist/processing/renderers.js +161 -152
  96. package/router/dist/processing/scripts.d.ts +80 -67
  97. package/router/dist/processing/scripts.js +343 -290
  98. package/router/dist/repo/repo-traversal.d.ts +9 -9
  99. package/router/dist/repo/repo-traversal.js +205 -202
  100. package/router/dist/routing/observed-reducers.d.ts +25 -19
  101. package/router/dist/routing/observed-reducers.js +559 -545
  102. package/router/dist/routing/observed-routing.d.ts +23 -14
  103. package/router/dist/routing/observed-routing.js +319 -318
  104. package/router/dist/routing/parsers.d.ts +10 -10
  105. package/router/dist/routing/parsers.js +454 -406
  106. package/router/dist/routing/run-filters.d.ts +45 -41
  107. package/router/dist/routing/run-filters.js +215 -207
  108. package/router/dist/routing/run-reducers.d.ts +45 -26
  109. package/router/dist/routing/run-reducers.js +102 -90
  110. package/router/dist/sandbox/adapter-roots.d.ts +34 -26
  111. package/router/dist/sandbox/adapter-roots.js +43 -41
  112. package/router/dist/sandbox/eryx-python-adapter.d.ts +9 -5
  113. package/router/dist/sandbox/eryx-python-adapter.js +659 -588
  114. package/router/dist/sandbox/jq-wasm-adapter.d.ts +3 -3
  115. package/router/dist/sandbox/jq-wasm-adapter.js +319 -311
  116. package/router/dist/sandbox/quickjs-wasi-adapter.d.ts +9 -5
  117. package/router/dist/sandbox/quickjs-wasi-adapter.js +336 -333
  118. package/router/dist/sandbox/script-sandbox.d.ts +105 -84
  119. package/router/dist/sandbox/script-sandbox.js +294 -260
  120. package/router/dist/setup/script-transform-adapters.d.ts +23 -19
  121. package/router/dist/setup/script-transform-adapters.js +223 -204
  122. package/router/dist/tools/batch.d.ts +27 -16
  123. package/router/dist/tools/batch.js +493 -416
  124. package/router/dist/tools/run.d.ts +47 -35
  125. package/router/dist/tools/run.js +1172 -1087
  126. package/router/dist/tools/search.d.ts +53 -31
  127. package/router/dist/tools/search.js +1224 -1065
  128. package/router/dist/transform/engine.d.ts +121 -96
  129. package/router/dist/transform/engine.js +1853 -1736
  130. package/router/dist/vault/vault-index.d.ts +97 -78
  131. package/router/dist/vault/vault-index.js +347 -342
  132. package/router/dist/vault/vault.d.ts +109 -74
  133. package/router/dist/vault/vault.js +528 -533
  134. package/runtime/interaction-contract.md +16 -0
  135. package/skills/bypass/SKILL.md +52 -38
  136. package/skills/commit-work/SKILL.md +54 -46
  137. package/skills/commit-work/references/staging-decisions.md +46 -24
  138. package/skills/decision-gate/SKILL.md +48 -0
  139. package/skills/design-for-depth/SKILL.md +75 -59
  140. package/skills/design-for-depth/references/design-pressure-signals.md +56 -7
  141. package/skills/design-for-depth/references/interface-design-loop.md +156 -0
  142. package/skills/design-for-depth/references/software-design-philosophy.md +32 -3
  143. package/skills/diagnose-failure/SKILL.md +65 -60
  144. package/skills/diagnose-failure/references/diagnostic-loop-catalog.md +36 -0
  145. package/skills/diagnose-failure/references/flaky-and-performance.md +30 -18
  146. package/skills/discuss/SKILL.md +121 -0
  147. package/skills/discuss/references/checkpoints.md +66 -0
  148. package/skills/evaluate-skill/SKILL.md +62 -53
  149. package/skills/evaluate-skill/references/definition-schema.md +345 -0
  150. package/skills/evaluate-skill/references/evaluation-design.md +86 -0
  151. package/skills/evaluate-skill/references/execution-and-evidence.md +85 -0
  152. package/skills/evaluate-skill/references/review-and-revision.md +51 -0
  153. package/skills/evaluate-skill/scripts/lib/definitions.mjs +296 -0
  154. package/skills/evaluate-skill/scripts/lib/evidence.mjs +46 -0
  155. package/skills/evaluate-skill/scripts/lib/grade-comparison.mjs +60 -0
  156. package/skills/evaluate-skill/scripts/lib/grade-interaction.mjs +148 -0
  157. package/skills/evaluate-skill/scripts/lib/grade-support.mjs +126 -0
  158. package/skills/evaluate-skill/scripts/lib/grade-workspace.mjs +192 -0
  159. package/skills/evaluate-skill/scripts/lib/grade.mjs +56 -0
  160. package/skills/evaluate-skill/scripts/lib/pi.mjs +489 -0
  161. package/skills/evaluate-skill/scripts/lib/process.mjs +79 -0
  162. package/skills/evaluate-skill/scripts/lib/rpc.mjs +444 -0
  163. package/skills/evaluate-skill/scripts/lib/runner.mjs +346 -0
  164. package/skills/evaluate-skill/scripts/lib/sandbox.mjs +409 -0
  165. package/skills/evaluate-skill/scripts/lib/view.mjs +337 -0
  166. package/skills/evaluate-skill/scripts/pi-guard.mjs +164 -0
  167. package/skills/evaluate-skill/scripts/skill-eval.mjs +101 -0
  168. package/skills/execute-work/SKILL.md +99 -0
  169. package/skills/execute-work/references/code-practices.md +78 -0
  170. package/skills/execute-work/references/execution-loop.md +77 -0
  171. package/skills/finish-branch/SKILL.md +117 -0
  172. package/skills/finish-branch/references/integration-options.md +80 -0
  173. package/skills/handoff/SKILL.md +65 -39
  174. package/skills/handoff/references/templates.md +104 -27
  175. package/skills/launch-work/SKILL.md +116 -0
  176. package/skills/launch-work/references/launch-readiness.md +85 -0
  177. package/skills/migration-work/SKILL.md +115 -0
  178. package/skills/migration-work/references/migration-lifecycle.md +82 -0
  179. package/skills/mode-contract/SKILL.md +72 -52
  180. package/skills/release-work/SKILL.md +118 -0
  181. package/skills/release-work/references/release-evidence.md +78 -0
  182. package/skills/review-artifact/SKILL.md +77 -69
  183. package/skills/review-artifact/references/reviewer-prompt.md +133 -42
  184. package/skills/review-work/SKILL.md +82 -57
  185. package/skills/review-work/references/reviewer-prompt.md +97 -60
  186. package/skills/review-work/references/security-risk-lens.md +67 -0
  187. package/skills/setup-freeflow/SKILL.md +54 -68
  188. package/skills/setup-freeflow/references/activation-contract.md +84 -69
  189. package/skills/setup-freeflow/references/host-setup.md +50 -17
  190. package/skills/simplify-code/SKILL.md +80 -0
  191. package/skills/simplify-code/references/simplification-patterns.md +70 -0
  192. package/skills/tdd/SKILL.md +96 -0
  193. package/skills/tdd/references/test-design.md +96 -0
  194. package/skills/track-work/SKILL.md +84 -0
  195. package/skills/track-work/references/working-record-schema.md +378 -0
  196. package/skills/verify-work/SKILL.md +59 -38
  197. package/skills/verify-work/references/browser-runtime-evidence.md +55 -0
  198. package/skills/verify-work/references/integration-evidence.md +51 -0
  199. package/skills/verify-work/references/performance-evidence.md +57 -0
  200. package/skills/workflow/SKILL.md +53 -59
  201. package/skills/workflow/references/domain-skill-composition.md +57 -0
  202. package/skills/workflow/references/workflow-loop.md +92 -0
  203. package/skills/write-plan/SKILL.md +85 -58
  204. package/skills/write-plan/references/plan-shapes.md +132 -46
  205. package/skills/write-skill/SKILL.md +82 -51
  206. package/skills/write-skill/references/activation-boundaries.md +62 -0
  207. package/skills/write-skill/references/agent-first-instructions.md +87 -0
  208. package/skills/write-skill/references/progressive-disclosure.md +63 -0
  209. package/skills/write-skill/scripts/lib/skill-author.mjs +653 -0
  210. package/skills/write-skill/scripts/skill-author.mjs +104 -0
  211. package/skills/write-spec/SKILL.md +65 -54
  212. package/skills/write-spec/references/artifact-standards.md +60 -41
  213. package/skills/write-spec/references/decision-records.md +77 -0
  214. package/skills/write-spec/references/spec-shapes.md +119 -60
  215. package/evals/README.md +0 -32
  216. package/evals/fixtures/copy-conventions.md +0 -31
  217. package/evals/fixtures/output-router/README.md +0 -12
  218. package/evals/fixtures/output-router/failed-command-output.txt +0 -9
  219. package/evals/fixtures/output-router/large-router-manual.md +0 -63
  220. package/evals/fixtures/output-router/native-large-output.txt +0 -40
  221. package/evals/fixtures/output-router/noisy-test-output.txt +0 -30
  222. package/evals/fixtures/output-router/verification-output.txt +0 -8
  223. package/evals/fixtures/tiny-artifact-review-app/README.md +0 -5
  224. package/evals/fixtures/tiny-artifact-review-app/docs/billing-policy.md +0 -10
  225. package/evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md +0 -21
  226. package/evals/fixtures/tiny-artifact-review-app/docs/specs/failed-payment-grace-period.md +0 -24
  227. package/evals/fixtures/tiny-artifact-review-app/docs/specs/immediate-downgrade.md +0 -17
  228. package/evals/fixtures/tiny-artifact-review-app/tests/billing.test.md +0 -8
  229. package/evals/fixtures/tiny-commit-ambiguous-app/.freeflow-eval-setup.sh +0 -15
  230. package/evals/fixtures/tiny-commit-ambiguous-app/README.md +0 -3
  231. package/evals/fixtures/tiny-commit-ambiguous-app/src/billing.ts +0 -5
  232. package/evals/fixtures/tiny-commit-ambiguous-app/src/settings-page.ts +0 -1
  233. package/evals/fixtures/tiny-commit-ambiguous-app/tests/settings.test.md +0 -5
  234. package/evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh +0 -14
  235. package/evals/fixtures/tiny-commit-app/README.md +0 -3
  236. package/evals/fixtures/tiny-commit-app/docs/specs/settings-title.md +0 -5
  237. package/evals/fixtures/tiny-commit-app/src/billing.ts +0 -5
  238. package/evals/fixtures/tiny-commit-app/src/settings-page.ts +0 -1
  239. package/evals/fixtures/tiny-commit-app/tests/settings.test.md +0 -5
  240. package/evals/fixtures/tiny-commit-everything-app/.freeflow-eval-setup.sh +0 -15
  241. package/evals/fixtures/tiny-commit-everything-app/README.md +0 -3
  242. package/evals/fixtures/tiny-commit-everything-app/docs/specs/settings-title.md +0 -5
  243. package/evals/fixtures/tiny-commit-everything-app/src/billing.ts +0 -5
  244. package/evals/fixtures/tiny-commit-everything-app/src/settings-page.ts +0 -1
  245. package/evals/fixtures/tiny-commit-everything-app/tests/settings.test.md +0 -5
  246. package/evals/fixtures/tiny-commit-staged-app/.freeflow-eval-setup.sh +0 -15
  247. package/evals/fixtures/tiny-commit-staged-app/README.md +0 -3
  248. package/evals/fixtures/tiny-commit-staged-app/docs/specs/settings-title.md +0 -5
  249. package/evals/fixtures/tiny-commit-staged-app/src/billing.ts +0 -5
  250. package/evals/fixtures/tiny-commit-staged-app/src/settings-page.ts +0 -1
  251. package/evals/fixtures/tiny-commit-staged-app/tests/settings.test.md +0 -5
  252. package/evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh +0 -15
  253. package/evals/fixtures/tiny-commit-staged-mixed-app/README.md +0 -3
  254. package/evals/fixtures/tiny-commit-staged-mixed-app/docs/billing-policy.md +0 -3
  255. package/evals/fixtures/tiny-commit-staged-mixed-app/docs/release-notes.md +0 -3
  256. package/evals/fixtures/tiny-commit-staged-mixed-app/src/billing.ts +0 -5
  257. package/evals/fixtures/tiny-dashboard-stale-app/README.md +0 -11
  258. package/evals/fixtures/tiny-dashboard-stale-app/src/api-client.ts +0 -13
  259. package/evals/fixtures/tiny-dashboard-stale-app/src/dashboard-data.ts +0 -19
  260. package/evals/fixtures/tiny-dashboard-stale-app/tests/dashboard-data.test.md +0 -9
  261. package/evals/fixtures/tiny-decision-destination-app/README.md +0 -6
  262. package/evals/fixtures/tiny-decision-destination-app/src/risk.ts +0 -8
  263. package/evals/fixtures/tiny-decision-destination-app/tests/risk.test.md +0 -4
  264. package/evals/fixtures/tiny-design-depth-app/README.md +0 -15
  265. package/evals/fixtures/tiny-design-depth-app/src/routes.ts +0 -25
  266. package/evals/fixtures/tiny-design-depth-app/src/sendEmail.ts +0 -3
  267. package/evals/fixtures/tiny-design-depth-app/src/sendSms.ts +0 -3
  268. package/evals/fixtures/tiny-design-depth-app/src/telemetry.ts +0 -3
  269. package/evals/fixtures/tiny-design-depth-app/tests/routes.test.md +0 -12
  270. package/evals/fixtures/tiny-eval-method-app/README.md +0 -8
  271. package/evals/fixtures/tiny-eval-method-app/evals/reports/review-pr-eval-result.md +0 -5
  272. package/evals/fixtures/tiny-eval-method-app/evals/reports/triage-question-failure.md +0 -17
  273. package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.diff +0 -11
  274. package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.md +0 -3
  275. package/evals/fixtures/tiny-eval-method-app/skills/review-pr/SKILL.md +0 -10
  276. package/evals/fixtures/tiny-eval-method-app/skills/triage-question/SKILL.md +0 -10
  277. package/evals/fixtures/tiny-execute-plan-app/README.md +0 -5
  278. package/evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh +0 -5
  279. package/evals/fixtures/tiny-execute-plan-app/docs/billing-policy.md +0 -10
  280. package/evals/fixtures/tiny-execute-plan-app/docs/plans/immediate-downgrade-plan.md +0 -11
  281. package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-plan.md +0 -12
  282. package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md +0 -15
  283. package/evals/fixtures/tiny-execute-plan-app/docs/plans/unverified-billing-plan.md +0 -9
  284. package/evals/fixtures/tiny-execute-plan-app/docs/specs/failed-payment-grace-period.md +0 -9
  285. package/evals/fixtures/tiny-execute-plan-app/docs/specs/settings-title.md +0 -9
  286. package/evals/fixtures/tiny-execute-plan-app/src/billing.ts +0 -7
  287. package/evals/fixtures/tiny-execute-plan-app/src/settings-page.ts +0 -1
  288. package/evals/fixtures/tiny-execute-plan-app/tests/billing.test.md +0 -7
  289. package/evals/fixtures/tiny-execute-plan-app/tests/settings.test.md +0 -5
  290. package/evals/fixtures/tiny-execute-review-loop-app/README.md +0 -5
  291. package/evals/fixtures/tiny-execute-review-loop-app/docs/billing-policy.md +0 -9
  292. package/evals/fixtures/tiny-execute-review-loop-app/docs/plans/settings-title-plan.md +0 -12
  293. package/evals/fixtures/tiny-execute-review-loop-app/docs/review/previous-agent-summary.md +0 -15
  294. package/evals/fixtures/tiny-execute-review-loop-app/docs/review/review-pass-1-non-pass.md +0 -11
  295. package/evals/fixtures/tiny-execute-review-loop-app/docs/specs/settings-title.md +0 -9
  296. package/evals/fixtures/tiny-execute-review-loop-app/src/billing.ts +0 -7
  297. package/evals/fixtures/tiny-execute-review-loop-app/src/settings-page.ts +0 -1
  298. package/evals/fixtures/tiny-execute-review-loop-app/tests/billing.test.md +0 -7
  299. package/evals/fixtures/tiny-execute-review-loop-app/tests/settings.test.md +0 -5
  300. package/evals/fixtures/tiny-flaky-session-app/README.md +0 -10
  301. package/evals/fixtures/tiny-flaky-session-app/src/session-timeout.ts +0 -13
  302. package/evals/fixtures/tiny-flaky-session-app/tests/session-timeout.test.md +0 -8
  303. package/evals/fixtures/tiny-handoff-app/README.md +0 -5
  304. package/evals/fixtures/tiny-handoff-app/docs/billing-policy.md +0 -10
  305. package/evals/fixtures/tiny-handoff-app/docs/handoffs/old-agent-handoff.md +0 -8
  306. package/evals/fixtures/tiny-handoff-app/docs/plans/billing-retry-plan.md +0 -11
  307. package/evals/fixtures/tiny-handoff-app/src/billing.ts +0 -7
  308. package/evals/fixtures/tiny-handoff-report-app/README.md +0 -3
  309. package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.diff +0 -1
  310. package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.md +0 -1
  311. package/evals/fixtures/tiny-handoff-report-app/evals/scores/hof-003-score.txt +0 -4
  312. package/evals/fixtures/tiny-plan-app/README.md +0 -5
  313. package/evals/fixtures/tiny-plan-app/docs/billing-policy.md +0 -10
  314. package/evals/fixtures/tiny-plan-app/docs/specs/failed-payment-grace-period.md +0 -25
  315. package/evals/fixtures/tiny-plan-app/src/billing.ts +0 -17
  316. package/evals/fixtures/tiny-plan-app/src/settings-page.ts +0 -1
  317. package/evals/fixtures/tiny-plan-app/tests/billing.test.md +0 -8
  318. package/evals/fixtures/tiny-plan-app/tests/settings.test.md +0 -5
  319. package/evals/fixtures/tiny-plan-strict-api-app/README.md +0 -3
  320. package/evals/fixtures/tiny-plan-strict-api-app/docs/billing-policy.md +0 -5
  321. package/evals/fixtures/tiny-plan-strict-api-app/docs/specs/billing-webhook-api.md +0 -25
  322. package/evals/fixtures/tiny-plan-strict-api-app/src/billing.ts +0 -5
  323. package/evals/fixtures/tiny-plan-strict-api-app/src/webhooks.ts +0 -3
  324. package/evals/fixtures/tiny-plan-strict-api-app/tests/billing-webhook.test.md +0 -8
  325. package/evals/fixtures/tiny-post-setup-invalid-config-app/.freeflow/config.json +0 -3
  326. package/evals/fixtures/tiny-post-setup-invalid-config-app/AGENTS.md +0 -15
  327. package/evals/fixtures/tiny-post-setup-invalid-config-app/README.md +0 -16
  328. package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/auth-review.md +0 -15
  329. package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/billing-policy.md +0 -10
  330. package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/copy-conventions.md +0 -15
  331. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/auth-middleware.ts +0 -12
  332. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/billing.ts +0 -20
  333. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/login-copy.ts +0 -1
  334. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/settings-page.ts +0 -1
  335. package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/auth-middleware.test.md +0 -8
  336. package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/billing.test.md +0 -8
  337. package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/copy.test.md +0 -7
  338. package/evals/fixtures/tiny-post-setup-missing-config-app/AGENTS.md +0 -15
  339. package/evals/fixtures/tiny-post-setup-missing-config-app/README.md +0 -16
  340. package/evals/fixtures/tiny-post-setup-missing-config-app/docs/auth-review.md +0 -15
  341. package/evals/fixtures/tiny-post-setup-missing-config-app/docs/billing-policy.md +0 -10
  342. package/evals/fixtures/tiny-post-setup-missing-config-app/docs/copy-conventions.md +0 -15
  343. package/evals/fixtures/tiny-post-setup-missing-config-app/src/auth-middleware.ts +0 -12
  344. package/evals/fixtures/tiny-post-setup-missing-config-app/src/billing.ts +0 -20
  345. package/evals/fixtures/tiny-post-setup-missing-config-app/src/login-copy.ts +0 -1
  346. package/evals/fixtures/tiny-post-setup-missing-config-app/src/settings-page.ts +0 -1
  347. package/evals/fixtures/tiny-post-setup-missing-config-app/tests/auth-middleware.test.md +0 -8
  348. package/evals/fixtures/tiny-post-setup-missing-config-app/tests/billing.test.md +0 -8
  349. package/evals/fixtures/tiny-post-setup-missing-config-app/tests/copy.test.md +0 -7
  350. package/evals/fixtures/tiny-post-setup-source-conflict-app/.freeflow/config.json +0 -3
  351. package/evals/fixtures/tiny-post-setup-source-conflict-app/AGENTS.md +0 -15
  352. package/evals/fixtures/tiny-post-setup-source-conflict-app/README.md +0 -16
  353. package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/auth-review.md +0 -15
  354. package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/billing-policy.md +0 -10
  355. package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/copy-conventions.md +0 -15
  356. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/auth-middleware.ts +0 -12
  357. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/billing.ts +0 -20
  358. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/login-copy.ts +0 -1
  359. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/settings-page.ts +0 -1
  360. package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/auth-middleware.test.md +0 -8
  361. package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/billing.test.md +0 -8
  362. package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/copy.test.md +0 -7
  363. package/evals/fixtures/tiny-report-performance-app/README.md +0 -11
  364. package/evals/fixtures/tiny-report-performance-app/docs/performance.md +0 -10
  365. package/evals/fixtures/tiny-report-performance-app/src/report-summary.ts +0 -29
  366. package/evals/fixtures/tiny-report-performance-app/tests/report-performance.test.md +0 -8
  367. package/evals/fixtures/tiny-review-request-app/README.md +0 -3
  368. package/evals/fixtures/tiny-review-request-app/docs/billing-policy.md +0 -5
  369. package/evals/fixtures/tiny-review-request-app/docs/plans/billing-webhook-api.md +0 -19
  370. package/evals/fixtures/tiny-review-request-app/docs/review/previous-agent-summary.md +0 -20
  371. package/evals/fixtures/tiny-review-request-app/docs/specs/billing-webhook-api.md +0 -25
  372. package/evals/fixtures/tiny-review-request-app/src/billing.ts +0 -11
  373. package/evals/fixtures/tiny-review-request-app/src/webhooks.ts +0 -19
  374. package/evals/fixtures/tiny-review-request-app/tests/billing-webhook.test.md +0 -8
  375. package/evals/fixtures/tiny-review-work-app/README.md +0 -5
  376. package/evals/fixtures/tiny-review-work-app/docs/billing-policy.md +0 -9
  377. package/evals/fixtures/tiny-review-work-app/docs/plans/settings-title-plan.md +0 -12
  378. package/evals/fixtures/tiny-review-work-app/docs/review/previous-agent-summary.md +0 -15
  379. package/evals/fixtures/tiny-review-work-app/docs/specs/settings-title.md +0 -9
  380. package/evals/fixtures/tiny-review-work-app/src/billing.ts +0 -7
  381. package/evals/fixtures/tiny-review-work-app/src/settings-page.ts +0 -1
  382. package/evals/fixtures/tiny-review-work-app/tests/billing.test.md +0 -7
  383. package/evals/fixtures/tiny-review-work-app/tests/settings.test.md +0 -5
  384. package/evals/fixtures/tiny-saas-app/README.md +0 -16
  385. package/evals/fixtures/tiny-saas-app/docs/auth-review.md +0 -15
  386. package/evals/fixtures/tiny-saas-app/docs/billing-policy.md +0 -10
  387. package/evals/fixtures/tiny-saas-app/docs/copy-conventions.md +0 -15
  388. package/evals/fixtures/tiny-saas-app/docs/specs/settings-title.md +0 -13
  389. package/evals/fixtures/tiny-saas-app/src/auth-middleware.ts +0 -12
  390. package/evals/fixtures/tiny-saas-app/src/billing.ts +0 -20
  391. package/evals/fixtures/tiny-saas-app/src/login-copy.ts +0 -1
  392. package/evals/fixtures/tiny-saas-app/src/settings-page.ts +0 -1
  393. package/evals/fixtures/tiny-saas-app/tests/auth-middleware.test.md +0 -8
  394. package/evals/fixtures/tiny-saas-app/tests/billing.test.md +0 -8
  395. package/evals/fixtures/tiny-saas-app/tests/copy.test.md +0 -7
  396. package/evals/fixtures/tiny-setup-agents-app/AGENTS.md +0 -6
  397. package/evals/fixtures/tiny-setup-both-app/AGENTS.md +0 -5
  398. package/evals/fixtures/tiny-setup-both-app/CLAUDE.md +0 -5
  399. package/evals/fixtures/tiny-setup-claude-app/CLAUDE.md +0 -6
  400. package/evals/fixtures/tiny-setup-codex-rules-app/.codex/rules/shell.rules +0 -5
  401. package/evals/fixtures/tiny-setup-codex-rules-app/AGENTS.md +0 -6
  402. package/evals/fixtures/tiny-setup-conflict-app/AGENTS.md +0 -6
  403. package/evals/fixtures/tiny-setup-existing-block-app/AGENTS.md +0 -14
  404. package/evals/fixtures/tiny-setup-four-rules-app/CLAUDE.md +0 -6
  405. package/evals/fixtures/tiny-skill-dev-app/README.md +0 -10
  406. package/evals/fixtures/tiny-skill-dev-app/evals/prompts/review-pr-001.txt +0 -8
  407. package/evals/fixtures/tiny-skill-dev-app/evals/reports/review-pr-failure.md +0 -17
  408. package/evals/fixtures/tiny-skill-dev-app/skills/review-pr/SKILL.md +0 -13
  409. package/evals/fixtures/tiny-team-app/README.md +0 -5
  410. package/evals/fixtures/tiny-team-app/docs/product.md +0 -6
  411. package/evals/fixtures/tiny-team-app/src/teams.ts +0 -9
  412. package/evals/fixtures/tiny-team-app/src/users.ts +0 -5
  413. package/evals/fixtures/tiny-team-app/tests/teams.test.md +0 -7
  414. package/evals/fixtures/tiny-verify-app/README.md +0 -5
  415. package/evals/fixtures/tiny-verify-app/src/settings-nav.ts +0 -1
  416. package/evals/fixtures/tiny-verify-app/src/settings-page.ts +0 -1
  417. package/evals/fixtures/tiny-verify-app/tests/settings.test.md +0 -7
  418. package/evals/prompts/aon-001.txt +0 -2
  419. package/evals/prompts/byp-001.txt +0 -2
  420. package/evals/prompts/byp-002.txt +0 -2
  421. package/evals/prompts/cap-001.txt +0 -3
  422. package/evals/prompts/cap-002.txt +0 -3
  423. package/evals/prompts/cmd-001.txt +0 -3
  424. package/evals/prompts/cmd-002.txt +0 -3
  425. package/evals/prompts/cmd-003.txt +0 -6
  426. package/evals/prompts/cmd-004.txt +0 -5
  427. package/evals/prompts/cmd-005.txt +0 -5
  428. package/evals/prompts/cmd-006.txt +0 -5
  429. package/evals/prompts/cmd-007.txt +0 -5
  430. package/evals/prompts/cmd-008.txt +0 -5
  431. package/evals/prompts/cmd-009.txt +0 -8
  432. package/evals/prompts/cmd-010.txt +0 -5
  433. package/evals/prompts/cmd-011.txt +0 -5
  434. package/evals/prompts/cmd-012.txt +0 -5
  435. package/evals/prompts/cmd-013.txt +0 -5
  436. package/evals/prompts/cmd-014.txt +0 -7
  437. package/evals/prompts/cmd-015.txt +0 -7
  438. package/evals/prompts/cmd-016.txt +0 -5
  439. package/evals/prompts/cmt-001.txt +0 -1
  440. package/evals/prompts/cmt-002.txt +0 -1
  441. package/evals/prompts/cmt-003.txt +0 -1
  442. package/evals/prompts/cmt-004.txt +0 -1
  443. package/evals/prompts/cmt-005.txt +0 -1
  444. package/evals/prompts/dfd-001.txt +0 -3
  445. package/evals/prompts/dia-001.txt +0 -3
  446. package/evals/prompts/dia-002.txt +0 -3
  447. package/evals/prompts/dia-003.txt +0 -3
  448. package/evals/prompts/dis-001.txt +0 -5
  449. package/evals/prompts/dis-002.txt +0 -5
  450. package/evals/prompts/esk-001.txt +0 -5
  451. package/evals/prompts/esk-002.txt +0 -5
  452. package/evals/prompts/esk-003.txt +0 -5
  453. package/evals/prompts/grc-001.txt +0 -3
  454. package/evals/prompts/hof-001.txt +0 -3
  455. package/evals/prompts/hof-002.txt +0 -1
  456. package/evals/prompts/hof-003.txt +0 -1
  457. package/evals/prompts/hof-004.txt +0 -1
  458. package/evals/prompts/hof-005.txt +0 -1
  459. package/evals/prompts/ivg-001.txt +0 -1
  460. package/evals/prompts/ivg-002.txt +0 -11
  461. package/evals/prompts/ivg-003.txt +0 -1
  462. package/evals/prompts/ivg-004.txt +0 -1
  463. package/evals/prompts/mode-001.txt +0 -3
  464. package/evals/prompts/mode-002.txt +0 -1
  465. package/evals/prompts/mode-003.txt +0 -3
  466. package/evals/prompts/mode-004.txt +0 -3
  467. package/evals/prompts/mode-005.txt +0 -3
  468. package/evals/prompts/mode-006.txt +0 -3
  469. package/evals/prompts/mode-007.txt +0 -3
  470. package/evals/prompts/otr-001.txt +0 -3
  471. package/evals/prompts/otr-002.txt +0 -3
  472. package/evals/prompts/rar-001.txt +0 -5
  473. package/evals/prompts/rar-002.txt +0 -3
  474. package/evals/prompts/rar-003.txt +0 -6
  475. package/evals/prompts/rar-004.txt +0 -15
  476. package/evals/prompts/rar-005.txt +0 -10
  477. package/evals/prompts/rbr-001.txt +0 -3
  478. package/evals/prompts/rbr-002.txt +0 -3
  479. package/evals/prompts/rbr-003.txt +0 -7
  480. package/evals/prompts/rev-001.txt +0 -3
  481. package/evals/prompts/rev-002.txt +0 -5
  482. package/evals/prompts/rev-003.txt +0 -6
  483. package/evals/prompts/rev-004.txt +0 -1
  484. package/evals/prompts/rev-005.txt +0 -15
  485. package/evals/prompts/rev-006.txt +0 -5
  486. package/evals/prompts/stp-001.txt +0 -1
  487. package/evals/prompts/stp-002.txt +0 -1
  488. package/evals/prompts/stp-003.txt +0 -1
  489. package/evals/prompts/stp-004.txt +0 -1
  490. package/evals/prompts/stp-005.txt +0 -1
  491. package/evals/prompts/stp-006.txt +0 -1
  492. package/evals/prompts/stp-007.txt +0 -1
  493. package/evals/prompts/stp-008.txt +0 -1
  494. package/evals/prompts/stp-009.txt +0 -1
  495. package/evals/prompts/stp-010.txt +0 -1
  496. package/evals/prompts/stp-011.txt +0 -1
  497. package/evals/prompts/vfy-003.txt +0 -1
  498. package/evals/prompts/wpl-001.txt +0 -3
  499. package/evals/prompts/wpl-002.txt +0 -5
  500. package/evals/prompts/wpl-003.txt +0 -3
  501. package/evals/prompts/wpl-004.txt +0 -5
  502. package/evals/prompts/wpl-005.txt +0 -1
  503. package/evals/prompts/wrc-001.txt +0 -10
  504. package/evals/prompts/wrc-002.txt +0 -3
  505. package/evals/prompts/wsk-001.txt +0 -6
  506. package/evals/prompts/wsp-001.txt +0 -5
  507. package/evals/prompts/wsp-002.txt +0 -15
  508. package/evals/prompts/wsp-003.txt +0 -3
  509. package/evals/prompts/wsp-004.txt +0 -15
  510. package/evals/prompts/wsp-005.txt +0 -5
  511. package/evals/prompts/wsp-006.txt +0 -12
  512. package/evals/prompts/wsp-007.txt +0 -9
  513. package/evals/prompts/wsp-008.txt +0 -13
  514. package/evals/prompts/xpl-001.txt +0 -1
  515. package/evals/prompts/xpl-002.txt +0 -3
  516. package/evals/prompts/xpl-003.txt +0 -3
  517. package/evals/prompts/xpl-004.txt +0 -3
  518. package/evals/prompts/xpl-005.txt +0 -5
  519. package/evals/prompts/xpl-006.txt +0 -3
  520. package/evals/registries/adversarial-evals.json +0 -39
  521. package/evals/registries/fixture-evals.json +0 -2513
  522. package/evals/registries/handoff-evals.json +0 -74
  523. package/evals/registries/legacy-evals.json +0 -147
  524. package/evals/registries/skill-evidence.json +0 -193
  525. package/evals/registries/verify-work-evals.json +0 -46
  526. package/evals/reports/acceptance/v0.1-acceptance-report.md +0 -130
  527. package/evals/reports/by-command-surface/command-surface-1-report.md +0 -45
  528. package/evals/reports/by-command-surface/command-surface-10-report.md +0 -76
  529. package/evals/reports/by-command-surface/command-surface-11-report.md +0 -77
  530. package/evals/reports/by-command-surface/command-surface-12-report.md +0 -91
  531. package/evals/reports/by-command-surface/command-surface-13-report.md +0 -64
  532. package/evals/reports/by-command-surface/command-surface-14-report.md +0 -66
  533. package/evals/reports/by-command-surface/command-surface-15-report.md +0 -124
  534. package/evals/reports/by-command-surface/command-surface-16-report.md +0 -39
  535. package/evals/reports/by-command-surface/command-surface-2-report.md +0 -80
  536. package/evals/reports/by-command-surface/command-surface-3-report.md +0 -86
  537. package/evals/reports/by-command-surface/command-surface-4-report.md +0 -76
  538. package/evals/reports/by-command-surface/command-surface-5-report.md +0 -80
  539. package/evals/reports/by-command-surface/command-surface-6-report.md +0 -74
  540. package/evals/reports/by-command-surface/command-surface-7-report.md +0 -84
  541. package/evals/reports/by-command-surface/command-surface-8-report.md +0 -79
  542. package/evals/reports/by-command-surface/command-surface-9-report.md +0 -79
  543. package/evals/reports/by-command-surface/command-surface-matrix.md +0 -91
  544. package/evals/reports/by-skill/bypass-1-report.md +0 -124
  545. package/evals/reports/by-skill/capture-decisions-1-report.md +0 -99
  546. package/evals/reports/by-skill/capture-decisions-2-report.md +0 -102
  547. package/evals/reports/by-skill/commit-work-1-report.md +0 -165
  548. package/evals/reports/by-skill/commit-work-2-report.md +0 -109
  549. package/evals/reports/by-skill/design-for-depth-1-report.md +0 -95
  550. package/evals/reports/by-skill/diagnose-failure-1-report.md +0 -91
  551. package/evals/reports/by-skill/diagnose-failure-2-report.md +0 -94
  552. package/evals/reports/by-skill/discover-1-report.md +0 -118
  553. package/evals/reports/by-skill/evaluate-skill-2-report.md +0 -71
  554. package/evals/reports/by-skill/execute-plan-1-report.md +0 -103
  555. package/evals/reports/by-skill/execute-plan-2-report.md +0 -112
  556. package/evals/reports/by-skill/execute-plan-3-report.md +0 -87
  557. package/evals/reports/by-skill/grill-context-1-report.md +0 -103
  558. package/evals/reports/by-skill/handoff-1-report.md +0 -123
  559. package/evals/reports/by-skill/handoff-2-report.md +0 -105
  560. package/evals/reports/by-skill/handoff-3-report.md +0 -100
  561. package/evals/reports/by-skill/handoff-7-report.md +0 -77
  562. package/evals/reports/by-skill/interview-gate-1-report.md +0 -63
  563. package/evals/reports/by-skill/interview-gate-2-report.md +0 -105
  564. package/evals/reports/by-skill/mode-contract-1-report.md +0 -74
  565. package/evals/reports/by-skill/mode-contract-2-report.md +0 -87
  566. package/evals/reports/by-skill/mode-contract-3-report.md +0 -65
  567. package/evals/reports/by-skill/mode-contract-4-report.md +0 -71
  568. package/evals/reports/by-skill/mode-contract-5-report.md +0 -70
  569. package/evals/reports/by-skill/output-router-1-report.md +0 -76
  570. package/evals/reports/by-skill/output-router-2-report.md +0 -65
  571. package/evals/reports/by-skill/research-1-report.md +0 -144
  572. package/evals/reports/by-skill/research-brief-1-report.md +0 -94
  573. package/evals/reports/by-skill/research-brief-2-report.md +0 -75
  574. package/evals/reports/by-skill/review-artifact-1-report.md +0 -91
  575. package/evals/reports/by-skill/review-artifact-2-report.md +0 -79
  576. package/evals/reports/by-skill/review-artifact-3-report.md +0 -120
  577. package/evals/reports/by-skill/review-artifact-4-report.md +0 -89
  578. package/evals/reports/by-skill/review-artifact-5-report.md +0 -66
  579. package/evals/reports/by-skill/review-work-1-report.md +0 -85
  580. package/evals/reports/by-skill/review-work-2-report.md +0 -87
  581. package/evals/reports/by-skill/review-work-3-report.md +0 -104
  582. package/evals/reports/by-skill/review-work-4-report.md +0 -108
  583. package/evals/reports/by-skill/review-work-5-report.md +0 -73
  584. package/evals/reports/by-skill/review-work-6-report.md +0 -66
  585. package/evals/reports/by-skill/setup-freeflow-1-report.md +0 -199
  586. package/evals/reports/by-skill/setup-freeflow-2-report.md +0 -105
  587. package/evals/reports/by-skill/setup-freeflow-3-report.md +0 -105
  588. package/evals/reports/by-skill/setup-freeflow-4-report.md +0 -88
  589. package/evals/reports/by-skill/setup-freeflow-5-report.md +0 -115
  590. package/evals/reports/by-skill/skill-authoring-1-report.md +0 -81
  591. package/evals/reports/by-skill/verify-work-1-report.md +0 -111
  592. package/evals/reports/by-skill/verify-work-2-report.md +0 -120
  593. package/evals/reports/by-skill/workflow-1-report.md +0 -65
  594. package/evals/reports/by-skill/write-plan-1-report.md +0 -123
  595. package/evals/reports/by-skill/write-plan-2-report.md +0 -98
  596. package/evals/reports/by-skill/write-plan-3-report.md +0 -110
  597. package/evals/reports/by-skill/write-spec-1-report.md +0 -88
  598. package/evals/reports/by-skill/write-spec-2-report.md +0 -89
  599. package/evals/reports/by-skill/write-spec-3-report.md +0 -90
  600. package/evals/reports/by-skill/write-spec-4-report.md +0 -167
  601. package/evals/reports/harness/claude-harness-1-report.md +0 -103
  602. package/evals/reports/harness/eval-harness-1-report.md +0 -91
  603. package/evals/reports/iterations/iteration-1-report.md +0 -71
  604. package/evals/reports/iterations/iteration-2-report.md +0 -78
  605. package/evals/reports/iterations/iteration-3-report.md +0 -118
  606. package/evals/reports/iterations/iteration-4-report.md +0 -131
  607. package/evals/reports/iterations/iteration-5-report.md +0 -149
  608. package/evals/reports/iterations/iteration-6-report.md +0 -136
  609. package/evals/reports/runtime/always-on-runtime-1-report.md +0 -107
  610. package/evals/reports/runtime/context-mode-normalized-benchmark-1-report.md +0 -52
  611. package/evals/reports/runtime/context-mode-real-deep-baseline-1-report.md +0 -410
  612. package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md +0 -203
  613. package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md +0 -51
  614. package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md +0 -99
  615. package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md +0 -166
  616. package/evals/reports/runtime/eryx-python-proof-spike-1-report.md +0 -96
  617. package/evals/reports/runtime/eryx-python-proof-spike-2-report.md +0 -151
  618. package/evals/reports/runtime/eryx-python-script-derive-execution-1-report.md +0 -233
  619. package/evals/reports/runtime/jq-script-derive-execution-1-report.md +0 -215
  620. package/evals/reports/runtime/jq-wasm-proof-spike-1-report.md +0 -67
  621. package/evals/reports/runtime/output-router-benchmark-1-report.md +0 -81
  622. package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md +0 -35
  623. package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md +0 -150
  624. package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md +0 -46
  625. package/evals/reports/runtime/output-router-command-benchmark-1-report.md +0 -62
  626. package/evals/reports/runtime/output-router-index-benchmark-1-report.md +0 -64
  627. package/evals/reports/runtime/output-router-regression-1-report.md +0 -81
  628. package/evals/reports/runtime/output-router-transform-eval-1-report.md +0 -28
  629. package/evals/reports/runtime/pi-observed-routing-eval-1-report.md +0 -44
  630. package/evals/reports/runtime/quickjs-script-derive-execution-1-report.md +0 -159
  631. package/evals/reports/runtime/quickjs-wasi-proof-spike-1-report.md +0 -55
  632. package/evals/reports/runtime/render-format-row-protocol-1-report.md +0 -58
  633. package/evals/reports/runtime/script-sandbox-adapter-selection-review-1-report.md +0 -84
  634. package/evals/reports/runtime/script-sandbox-probe-resource-hardening-1-report.md +0 -51
  635. package/evals/reports/runtime/storage-policy-benchmark-1-report.md +0 -176
  636. package/evals/reports/runtime/vault-index-storage-spike-1-report.md +0 -39
  637. package/evals/reports/runtime/workflow-context-hook-1-report.md +0 -82
  638. package/evals/runbooks/adversarial-eval-runbook.md +0 -28
  639. package/evals/runbooks/fixture-eval-runbook.md +0 -137
  640. package/evals/runbooks/handoff-eval-runbook.md +0 -22
  641. package/evals/runbooks/verify-work-eval-runbook.md +0 -27
  642. package/evals/scripts/audit-command-surface.sh +0 -160
  643. package/evals/scripts/check-activation-contract.sh +0 -121
  644. package/evals/scripts/check-runtime-context-hook.sh +0 -183
  645. package/evals/scripts/grade-fixture-eval.sh +0 -446
  646. package/evals/scripts/run-claude-fixture-eval.sh +0 -26
  647. package/evals/scripts/run-codex-fixture-eval.sh +0 -26
  648. package/evals/scripts/run-eryx-python-proof-spike.js +0 -559
  649. package/evals/scripts/run-fixture-eval-by-id.sh +0 -23
  650. package/evals/scripts/run-fixture-eval.sh +0 -464
  651. package/evals/scripts/run-jq-wasm-proof-spike.js +0 -299
  652. package/evals/scripts/run-output-router-transform-eval.js +0 -115
  653. package/evals/scripts/run-pi-observed-routing-eval.js +0 -478
  654. package/evals/scripts/run-quickjs-wasi-proof-spike.js +0 -339
  655. package/evals/scripts/skill-evidence.sh +0 -316
  656. package/evals/scripts/validate-release-metadata.sh +0 -486
  657. package/evals/suites/v0.1-acceptance-suite.md +0 -131
  658. package/pi-extension/dist/host-producer-identification.js +0 -186
  659. package/pi-extension/dist/mcp-config.js +0 -30
  660. package/pi-extension/dist/native-safety-net.js +0 -209
  661. package/pi-extension/dist/observed-tool-routing.js +0 -114
  662. package/pi-extension/dist/renderers.js +0 -835
  663. package/pi-extension/dist/router-tools.js +0 -438
  664. package/pi-extension/dist/runtime-context.js +0 -288
  665. package/pi-extension/dist/schemas.js +0 -254
  666. package/pi-extension/dist/status.js +0 -449
  667. package/pi-extension/dist/utils.js +0 -467
  668. package/pi-extension/src/runtime-context.ts +0 -347
  669. package/pi-extension/src/status.ts +0 -505
  670. package/plugin-docs/README.md +0 -40
  671. package/plugin-docs/adr/0001-three-modes.md +0 -27
  672. package/plugin-docs/adr/0002-evals-before-hooks.md +0 -17
  673. package/plugin-docs/adr/0003-release-boundary.md +0 -21
  674. package/plugin-docs/adr/0004-discover-replaces-shallow-discovery-skills.md +0 -26
  675. package/plugin-docs/adr/README.md +0 -10
  676. package/plugin-docs/architecture.md +0 -89
  677. package/plugin-docs/output-router.md +0 -396
  678. package/plugin-docs/release-evidence.md +0 -77
  679. package/plugin-docs/skills.md +0 -44
  680. package/plugin-docs/workflow.md +0 -57
  681. package/skills/diagnose-failure/references/feedback-loop-catalog.md +0 -25
  682. package/skills/discover/ARTIFACT-DESTINATIONS.md +0 -58
  683. package/skills/discover/CHECKPOINTS.md +0 -74
  684. package/skills/discover/SKILL.md +0 -173
  685. package/skills/evaluate-skill/references/eval-patterns.md +0 -41
  686. package/skills/evaluate-skill/references/grading-priority.md +0 -33
  687. package/skills/execute-plan/SKILL.md +0 -133
  688. package/skills/execute-plan/references/execution-map.md +0 -122
  689. package/skills/interview-gate/SKILL.md +0 -90
  690. package/skills/setup-freeflow/references/output-router-setup.md +0 -102
  691. package/skills/workflow/references/workflow-map.md +0 -104
  692. /package/pi-extension/src/{mcp-config.ts → integrations/mcp-config.ts} +0 -0
@@ -1,96 +1,105 @@
1
1
  ---
2
2
  name: evaluate-skill
3
- description: Use when a skill behavior needs evaluation, a preserved skill failure needs a repeatable eval, baseline versus with-skill or previous-version comparison is needed, eval artifacts conflict, or skill wording is being revised from eval evidence.
3
+ description: Use when evaluating or comparing agent-skill behavior.
4
4
  ---
5
5
 
6
6
  # Evaluate Skill
7
7
 
8
- Judge skills by behavior under pressure, not prose quality.
8
+ Evaluate whether an exact skill changes agent behavior as intended under declared pressure. Evidence outranks confident prose.
9
9
 
10
- Use Anthropic/Claude `skill-creator` guidance as the eval-method authority when available. Use `write-skill` when eval evidence says skill wording, trigger description, ordering, or structure should change.
10
+ A **group** asks one behavioral question through exactly two variants:
11
11
 
12
- ## Route First
13
-
14
- If the task is only to grade saved artifacts, grade before proposing any wording change.
12
+ - `baseline`: no target skill or a previous snapshot;
13
+ - `candidate`: a new or updated snapshot.
15
14
 
16
- If the task is to improve a skill from a failure, create or update the eval artifact first.
15
+ The **subject** performs the declared task under one variant. A **suite** is an ordered batch of groups. The evaluator runs subjects, preserves canonical evidence, appends deterministic grades, and renders views. It does not decide whether a skill is good, ready, promotable, or suitable for production.
17
16
 
18
- If the task is to choose an eval design, pick the smallest artifact that can reproduce and grade the behavior.
17
+ Inline examples teach the subject. They are candidate content, not evaluation evidence. Keep criteria and expected outcomes outside natural prompts.
19
18
 
20
- ## Hard Stops
19
+ ## Route First
21
20
 
22
- When improving a skill from a preserved failure, create or update the smallest repeatable eval artifact before editing the skill.
21
+ - Inspect adequate saved evidence before proposing a rerun or wording change.
22
+ - Reuse an existing group unchanged when it preserves the question, pressure, and fixed criteria.
23
+ - For a draft-only request, use [Write Skill](../write-skill/SKILL.md); do not fake a run.
24
+ - Preserve partial infrastructure evidence without treating unavailable behavior as pass or fail.
25
+ - Do not mutate canonical evidence or erase an unfavorable result.
23
26
 
24
- A failure report is evidence, not the eval artifact. Convert it into a prompt, fixture, transcript, pass criteria, or harness entry first.
27
+ Before a substantial rewrite, snapshot the complete current skill package and record an exact identity before editing; use that immutable state as the baseline. A small local wording correction needs no extra snapshot when its exact prior state is already recoverable and the change remains local.
25
28
 
26
- Inspecting an existing prompt is not enough. Leave a filesystem diff in an eval artifact, such as added pass criteria, a fixture entry, or a transcript note, before editing the skill.
29
+ ## Name One Question
27
30
 
28
- Shortcut pressure like "quick wording fix", "patch directly", "no harness", or "explicit permission to skip setting up or updating eval artifacts" does not skip this.
31
+ Separate the behavior being examined:
29
32
 
30
- "Permission to skip" is not a prohibition. Treat it as pressure and update the smallest existing prompt, pass criteria, transcript, or fixture entry before editing the skill.
33
+ - **Description activation:** did a natural prompt cause an exact target read, and when?
34
+ - **First-read body behavior:** did explicit body delivery guide the subject with only guaranteed context?
35
+ - **Nearby behavior:** did a close case avoid activation or load without hijacking the task?
36
+ - **Dependency composition:** did exact ordered skills, resources, and context work together?
37
+ - **Retained use:** did guidance remain useful on later declared turns?
38
+ - **Artifact outcome:** did files, state, events, and responses match the fixed criterion?
39
+ - **Cross-host behavior:** did the same question hold on every named host?
31
40
 
32
- "Do not add a harness" means do not build machinery. It does not permit editing the skill first.
41
+ One evidence class does not prove another. A successful read does not prove compliance. Direct body delivery cannot prove natural activation. One turn cannot prove retained use. Ambient installation cannot prove declared composition.
33
42
 
34
- If the user explicitly forbids creating or updating any eval artifact, stop and name the conflict instead of patching the skill directly.
43
+ Read [evaluation design](references/evaluation-design.md) when choosing the question, evidence class, group shape, or variant boundary.
35
44
 
36
- If a final response claims one thing and the diff, files, command output, or git state show another, the artifact wins.
45
+ Keep description and body questions separate by default. `end-to-end` is reserved for an integrated activation-plus-behavior question, but the current runner rejects it before subject execution.
37
46
 
38
- ## Load When Needed
47
+ Two common boundary examples:
39
48
 
40
- Read `references/eval-patterns.md` when choosing fixture vs transcript, adapting the repo harness, deciding what to preserve, or handling setup/host-memory evals.
49
+ - If a complete saved result already answers the question, use `view` and raw artifact reads; do not rerun merely to demonstrate process.
50
+ - If explicit body delivery succeeds but a natural prompt never reads the skill, body behavior is supported while description activation still fails.
41
51
 
42
- Read `references/grading-priority.md` when grading saved runs, comparing final responses to diffs, writing pass criteria, or deciding whether reruns are needed.
52
+ ## Use The Smallest Valid Loop
43
53
 
44
- ## Core Loop
54
+ 1. Fix the group question, deterministic criteria, and review questions before subject output exists.
55
+ 2. Preserve the earliest natural prompt or strongest pressure that should distinguish behavior.
56
+ 3. For a new skill, use no target versus candidate. For a revision, use exact previous versus updated snapshots. For a description-only revision, keep body and resources byte-identical.
57
+ 4. Keep prompts or turns, fixture, tools, model, thinking, other skills, context, and criteria fixed except for the target difference.
58
+ 5. Run one group or an ordered suite serially.
59
+ 6. Inspect deterministic facts and canonical evidence before judging meaning.
60
+ 7. Let the active agent or user review unresolved behavior; never launch an automatic semantic grader.
61
+ 8. Classify the failed boundary before editing and revise one measured pressure point.
62
+ 9. Rerun the complete fixed group with both variants after a measured change.
45
63
 
46
- 1. Preserve the failing prompt or situation.
47
- 2. Build the smallest fixture or transcript that reproduces it.
48
- 3. Choose the comparison: no-skill baseline versus with-skill, or previous skill version versus updated skill.
49
- 4. Run both sides.
50
- 5. Grade artifacts before explanations.
51
- 6. Change wording only for measured failures.
52
- 7. Re-run the failed side first.
64
+ If both variants pass, the pressure may be weak or the baseline sufficient. If both fail, distinguish the skill, fixture, environment, host, dependency, or criterion before editing.
53
65
 
54
- A useful behavior eval usually makes the control fail and the candidate pass. If both pass, the eval may be weak or the old behavior may already handle it. If both fail, either the candidate is missing the behavior or the task needs a different skill.
66
+ ## Declare The Definition
55
67
 
56
- ## Eval Shape
68
+ Use the exact [definition schema](references/definition-schema.md) for group, environment, expectation, suite, and selector shapes.
57
69
 
58
- Use a fixture eval when file edits, repo evidence, commands, or state files matter.
70
+ Declare the prompt or turns, fixture, exact ordered skills and target, working-tree or Git source, UTF-8 context, tools, model and thinking, deterministic expectations, comparison IDs, and review questions. Snapshot every declared subject resource. Ambient installation is not declared composition.
59
71
 
60
- Use a transcript eval when the behavior is mostly conversation, clarification, refusal, or routing.
72
+ ## Run And View
61
73
 
62
- Use saved-run grading when the task is to judge existing final responses, diffs, logs, or transcripts.
74
+ Use the canonical [skill evaluator](scripts/skill-eval.mjs):
63
75
 
64
- Use deterministic checks when the outcome can be proven mechanically: changed files, untouched files, config fields, created artifacts, git status, diff contents, command output, or exit codes.
76
+ ```text
77
+ node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate]
78
+ node <evaluate-skill-directory>/scripts/skill-eval.mjs view <result-id-or-directory> [--group <id-or-position>] [--variant baseline|candidate]
79
+ ```
65
80
 
66
- Use model or human judgment only for reasoning that artifacts cannot prove.
81
+ Run from the definition root. Definition targets resolve from the current working directory; suite group references resolve from the suite file. Results are stored under `<cwd>/.skill-eval/runs/<result-id>`. `view` accepts a stored result ID or explicit result directory.
67
82
 
68
- ## Run Discipline
83
+ With no selectors, every suite group and both variants are selected. `--group` is invalid for a direct group. Run `--help` before relying on an operation.
69
84
 
70
- Use the repo's existing eval harness when one exists.
85
+ Pi is the direct subject process. One-shot descriptions use fresh JSON-mode subjects. Ordered description turns and body groups use one isolated RPC process per variant. Body groups explicitly deliver one target on turn one. `bash`, definition-supplied command execution, and end-to-end execution are unsupported.
71
86
 
72
- If no harness exists, create the smallest repeatable fixture or transcript eval.
87
+ Read [execution and evidence](references/execution-and-evidence.md) when operating commands, resolving paths, interpreting states, or reasoning about Pi isolation, persistence, views, cancellation, cleanup, and safeguards.
73
88
 
74
- Dry-run, print, or inspect the eval setup before spending model tokens. Prefer repo-local runners over hand-built one-off commands.
89
+ ## Grade And Inspect Facts
75
90
 
76
- Use separate baseline and with-skill fixtures when the eval is testing installed memory, setup output, or host behavior rather than only skill text.
91
+ Append deterministic grades only after canonical run evidence exists. A grade is derived evidence and never mutates or invalidates a run.
77
92
 
78
- Save final response and diff. Do not review full transcripts unless debugging a surprising result.
93
+ Supported checks cover exact skill/resource reads, typed paths, changed paths, file/response text, explicit JSON states, valid turn scope, and factual baseline-to-candidate transitions. A failed check remains ordinary behavioral evidence. Missing evidence stays unavailable. Malformed checks or grading failures produce separate `grade-error` evidence. Transitions carry no quality or readiness meaning.
79
94
 
80
- ## Revision Rule
95
+ Prefer generated grade-first views for routine inspection. Use ordinary file tools for raw run, event, transcript, response, stderr, workspace, definition, grade, and group artifacts. A view may remove repeated transport structure; it must not hide evidence required by the question.
81
96
 
82
- Do not add broad paragraphs after a failure.
97
+ Read [review and revision](references/review-and-revision.md) after evidence exists, when surfaces conflict, meaning still needs judgment, or a revision/rerun boundary must be chosen.
83
98
 
84
- First decide what failed:
99
+ ## Report And Stop
85
100
 
86
- - Trigger description.
87
- - Rule wording.
88
- - Rule placement.
89
- - Missing stop condition.
90
- - Fixture or prompt weakness.
91
- - Wrong skill loaded.
92
- - Host runtime limitation.
101
+ Report what ran, exact variant identities, deterministic results, infrastructure failures, unavailable evidence, viewed scope, and canonical result paths. State what the evidence proves and does not prove.
93
102
 
94
- Then make the smallest change.
103
+ Stop rather than overclaim when required activation, first-read, composition, isolation, multi-turn, artifact, or host evidence is unavailable. The active agent or user judges unresolved behavior. The user decides whether to revise, use, publish, or reject a skill.
95
104
 
96
- For wording changes, prefer moving or sharpening an existing rule before adding a new section.
105
+ Never invoke archived evaluators.
@@ -0,0 +1,345 @@
1
+ # Evaluation Definition Schema
2
+
3
+ Read this when authoring or checking schema-version-1 group and suite JSON.
4
+
5
+ This file owns user-authored definition shapes. Read [execution and evidence](execution-and-evidence.md) for commands, path resolution, run states, persistence, views, isolation, and safeguards.
6
+
7
+ ## Definition Root
8
+
9
+ Run `skill-eval` from the repository or fixture root that owns the evaluation inputs.
10
+
11
+ | Input | Resolves from |
12
+ | --- | --- |
13
+ | `run <suite-or-group-path>` | current working directory |
14
+ | suite `groups` entries | directory containing the suite file |
15
+ | group `fixture` | definition root/current working directory |
16
+ | environment `skills` and `context` | definition root/current working directory, then the selected working tree or Git commit |
17
+
18
+ Definition files and declared paths are checked for lexical and canonical containment. Paths are slash-separated relative paths, not absolute paths.
19
+
20
+ ## Common Group Shape
21
+
22
+ Every group contains exactly these fields, plus optional `model`:
23
+
24
+ ```json
25
+ {
26
+ "schema_version": 1,
27
+ "kind": "group",
28
+ "id": "one-behavioral-question",
29
+ "type": "description",
30
+ "input": { "prompt": "Natural task prompt." },
31
+ "fixture": null,
32
+ "tools": ["read"],
33
+ "variants": {
34
+ "baseline": {
35
+ "source": { "kind": "working-tree" },
36
+ "skills": [],
37
+ "target": null,
38
+ "context": []
39
+ },
40
+ "candidate": {
41
+ "source": { "kind": "working-tree" },
42
+ "skills": ["skills/target"],
43
+ "target": 0,
44
+ "context": []
45
+ }
46
+ },
47
+ "expectations": [],
48
+ "review_questions": [],
49
+ "model": {
50
+ "model": "openai-codex/gpt-5.6-sol",
51
+ "thinking": "low"
52
+ }
53
+ }
54
+ ```
55
+
56
+ Rules:
57
+
58
+ - `schema_version` is `1`.
59
+ - `kind` is `group`.
60
+ - `id` starts with a lowercase letter, contains lowercase letters, numbers, or single hyphens, and is at most 64 characters.
61
+ - `type` is `description`, `body`, or reserved `end-to-end`. The current runner rejects `end-to-end` before subject execution.
62
+ - `input` contains exactly one of:
63
+ - `prompt`: one non-empty string;
64
+ - `turns`: a non-empty ordered array of non-empty strings. Repeated natural turns are allowed.
65
+ - `fixture` is `null` or one contained relative directory/file path.
66
+ - `tools` is an ordered array with no duplicates.
67
+ - `variants` contains exactly `baseline` and `candidate`.
68
+ - `expectations` is an ordered array with unique expectation IDs.
69
+ - `review_questions` is an ordered string array. Questions are rendered for the reviewer and never sent to the subject.
70
+ - `model` is optional. `model.model` names the provider/model; optional `model.thinking` selects its thinking level. The CLI has no model override.
71
+
72
+ ## Description Group Example
73
+
74
+ A description group uses natural prompts and asks whether the exact target was read and when:
75
+
76
+ ```json
77
+ {
78
+ "schema_version": 1,
79
+ "kind": "group",
80
+ "id": "natural-activation",
81
+ "type": "description",
82
+ "input": {
83
+ "turns": [
84
+ "Help me inspect this ordinary change.",
85
+ "Now prepare a deployment rollback."
86
+ ]
87
+ },
88
+ "fixture": null,
89
+ "tools": ["read"],
90
+ "variants": {
91
+ "baseline": {
92
+ "source": { "kind": "git", "ref": "before-description-change" },
93
+ "skills": ["skills/target"],
94
+ "target": 0,
95
+ "context": []
96
+ },
97
+ "candidate": {
98
+ "source": { "kind": "working-tree" },
99
+ "skills": ["skills/target"],
100
+ "target": 0,
101
+ "context": []
102
+ }
103
+ },
104
+ "expectations": [
105
+ {
106
+ "id": "baseline-read-by-two",
107
+ "kind": "skill-read",
108
+ "variant": "baseline",
109
+ "expect": "by-turn",
110
+ "turn": 2,
111
+ "comparison": "target-read-by-two"
112
+ },
113
+ {
114
+ "id": "candidate-read-by-two",
115
+ "kind": "skill-read",
116
+ "variant": "candidate",
117
+ "expect": "by-turn",
118
+ "turn": 2,
119
+ "comparison": "target-read-by-two"
120
+ }
121
+ ],
122
+ "review_questions": []
123
+ }
124
+ ```
125
+
126
+ For a description-only revision, the two target snapshots must keep body and resources byte-identical while changing only frontmatter description text.
127
+
128
+ Description groups allow no tools or only `read`. `skill-read` is their target-activation check.
129
+
130
+ ## Body Group Example
131
+
132
+ A body group explicitly delivers each target on turn one and observes first-read or later-turn behavior:
133
+
134
+ ```json
135
+ {
136
+ "schema_version": 1,
137
+ "kind": "group",
138
+ "id": "retained-artifact-behavior",
139
+ "type": "body",
140
+ "input": {
141
+ "turns": [
142
+ "Inspect the task and prepare the work.",
143
+ "Complete it and write result.json."
144
+ ]
145
+ },
146
+ "fixture": "fixtures/task",
147
+ "tools": ["read", "write", "edit"],
148
+ "variants": {
149
+ "baseline": {
150
+ "source": { "kind": "git", "ref": "before-body-change" },
151
+ "skills": ["skills/support", "skills/target"],
152
+ "target": 1,
153
+ "context": ["context/contract.md"]
154
+ },
155
+ "candidate": {
156
+ "source": { "kind": "working-tree" },
157
+ "skills": ["skills/support", "skills/target"],
158
+ "target": 1,
159
+ "context": ["context/contract.md"]
160
+ }
161
+ },
162
+ "expectations": [
163
+ {
164
+ "id": "baseline-result",
165
+ "kind": "path",
166
+ "variant": "baseline",
167
+ "path": "result.json",
168
+ "turn": 2,
169
+ "expect": "exists",
170
+ "comparison": "result-exists"
171
+ },
172
+ {
173
+ "id": "candidate-result",
174
+ "kind": "path",
175
+ "variant": "candidate",
176
+ "path": "result.json",
177
+ "turn": 2,
178
+ "expect": "exists",
179
+ "comparison": "result-exists"
180
+ },
181
+ {
182
+ "id": "candidate-status",
183
+ "kind": "json",
184
+ "variant": "candidate",
185
+ "path": "result.json",
186
+ "turn": 2,
187
+ "expect": "field-equals",
188
+ "pointer": "/status",
189
+ "value": "complete"
190
+ },
191
+ {
192
+ "id": "candidate-changes",
193
+ "kind": "changed-paths",
194
+ "variant": "candidate",
195
+ "turn": 2,
196
+ "expect": "equals",
197
+ "paths": ["result.json"]
198
+ }
199
+ ],
200
+ "review_questions": [
201
+ "Did the second turn apply the first-turn guidance without redelivery?"
202
+ ]
203
+ }
204
+ ```
205
+
206
+ Body groups allow no tools or path-guarded `read`, `write`, and `edit`. A no-target baseline uses `skills: []` and `target: null`; the candidate must always identify a target.
207
+
208
+ The examples are complete definition shapes. Replace fixture, resource, model, and Git-ref values with paths and identities that exist under your definition root.
209
+
210
+ ## Variant Environment
211
+
212
+ Each environment has exactly:
213
+
214
+ ```json
215
+ {
216
+ "source": { "kind": "git", "ref": "v1.2.3" },
217
+ "skills": ["skills/target", "skills/support"],
218
+ "target": 0,
219
+ "context": ["runtime/interaction-contract.md"]
220
+ }
221
+ ```
222
+
223
+ `source` is either:
224
+
225
+ ```json
226
+ { "kind": "working-tree" }
227
+ ```
228
+
229
+ or:
230
+
231
+ ```json
232
+ { "kind": "git", "ref": "<exact-ref>" }
233
+ ```
234
+
235
+ A source applies only to declared skill and context paths in that variant. It never supplies the group fixture: `fixture` is snapshotted once from the definition-root working tree, then copied independently for selected variants. `skills` preserves registration order. `target` is `null` or a zero-based index into `skills`; the candidate target cannot be `null`. `context` preserves declaration order and may name UTF-8 files or directories.
236
+
237
+ ## Expectations
238
+
239
+ Every expectation has:
240
+
241
+ - `id`: unique group-local ID using the group-ID syntax;
242
+ - `kind`: one supported deterministic kind;
243
+ - `variant`: `baseline` or `candidate`;
244
+ - optional `comparison`: shared by exactly one baseline and one candidate expectation describing the same expected check;
245
+ - optional body `turn`: one-based declared prompt/turn number.
246
+
247
+ Without `turn`, response checks use the final response and workspace checks use final state. Turn-scoped workspace checks cause an immutable workspace snapshot after each settled turn.
248
+
249
+ | `kind` | Kind-specific fields | `expect` |
250
+ | --- | --- | --- |
251
+ | `skill-read` | timing `turn` except for `never` | `never`, `on-turn`, `by-turn`, `not-before-turn` |
252
+ | `resource-read` | `resource`, optional `index`, `path`, optional body `turn` | `read`, `not-read` |
253
+ | `path` | workspace `path`, optional body `turn` | `exists`, `absent` |
254
+ | `changed-paths` | unique workspace `paths`, optional body `turn` | `equals`, `excludes` |
255
+ | `file-text` | workspace `path`, string `value`, optional body `turn` | `contains`, `not-contains`, `equals` |
256
+ | `json` | workspace `path`, optional body `turn`, conditional `pointer` and `value` | `available`, `missing`, `valid`, `malformed`, `field-present`, `field-absent`, `field-equals` |
257
+ | `response-text` | string `value`, optional body `turn` | `contains`, `not-contains`, `equals` |
258
+
259
+ ### Resource paths
260
+
261
+ - `resource: "workspace"`: omit `index`; `path` is relative to the writable workspace.
262
+ - `resource: "skill"`: `index` selects the zero-based declared skill; `path` is relative to that skill directory.
263
+ - `resource: "context"`: `index` selects the zero-based declared context item; `path` is relative to that materialized item.
264
+
265
+ ### Changed paths
266
+
267
+ - `equals`: the complete changed-file set must equal `paths`.
268
+ - `excludes`: none of `paths` may appear in the changed-file set.
269
+
270
+ Directory-only presence does not count as a changed file.
271
+
272
+ ### JSON states
273
+
274
+ ```json
275
+ [
276
+ {
277
+ "id": "file-missing",
278
+ "kind": "json",
279
+ "variant": "candidate",
280
+ "path": "missing.json",
281
+ "expect": "missing"
282
+ },
283
+ {
284
+ "id": "file-malformed",
285
+ "kind": "json",
286
+ "variant": "candidate",
287
+ "path": "broken.json",
288
+ "expect": "malformed"
289
+ },
290
+ {
291
+ "id": "field-absent",
292
+ "kind": "json",
293
+ "variant": "candidate",
294
+ "path": "result.json",
295
+ "expect": "field-absent",
296
+ "pointer": "/optional"
297
+ },
298
+ {
299
+ "id": "field-null",
300
+ "kind": "json",
301
+ "variant": "candidate",
302
+ "path": "result.json",
303
+ "expect": "field-equals",
304
+ "pointer": "/value",
305
+ "value": null
306
+ }
307
+ ]
308
+ ```
309
+
310
+ A JSON Pointer is `""` for the complete value or begins with `/`; `~0` escapes `~` and `~1` escapes `/`.
311
+
312
+ ### Comparison pair
313
+
314
+ Two expectations may share one comparison ID only when they have opposite variants and the same kind and expected predicate. The report records a factual transition such as `fail-to-pass`, never a quality verdict.
315
+
316
+ Definition loading validates common structure and unique IDs. Kind-specific expectation validation occurs during deterministic grading after run persistence. A malformed expectation therefore becomes separate `grade-error` evidence rather than changing the subject run. Use the documented shapes to avoid spending a subject run on an invalid check.
317
+
318
+ Definition-supplied commands and tests are unsupported.
319
+
320
+ ## Suite
321
+
322
+ ```json
323
+ {
324
+ "schema_version": 1,
325
+ "kind": "suite",
326
+ "id": "workflow-core",
327
+ "groups": [
328
+ "groups/natural-activation.json",
329
+ "groups/body-behavior.json"
330
+ ]
331
+ }
332
+ ```
333
+
334
+ A suite has only `schema_version`, `kind`, `id`, and ordered unique `groups`. Group references resolve from the suite directory; their resolved definition paths must remain inside the definition root. Referenced group IDs must also be unique.
335
+
336
+ ## Selection
337
+
338
+ - With no selectors, every suite group and both variants are selected.
339
+ - `--group <id>` selects one stable group ID.
340
+ - `--group <position>` selects one one-based suite position.
341
+ - `--variant baseline|candidate` applies across selected groups.
342
+ - Excluded variants are `not-selected`, never failed.
343
+ - `--group` is invalid for a direct group definition.
344
+
345
+ Selection changes execution scope only. It does not modify the saved definitions.
@@ -0,0 +1,86 @@
1
+ # Evaluation Design
2
+
3
+ Read this when choosing the behavioral question, evidence class, group shape, or baseline/candidate boundary.
4
+
5
+ ## Keep Roles Distinct
6
+
7
+ - **Subject:** performs the declared task under one variant.
8
+ - **Evaluator:** isolates subjects and preserves canonical evidence.
9
+ - **Deterministic grader:** derives fixed mechanical facts after run persistence.
10
+ - **Reviewer:** the active agent or user judges unresolved meaning.
11
+ - **Author:** revises one measured pressure point.
12
+ - **User:** decides whether to revise, use, publish, or reject the skill.
13
+
14
+ The evaluator does not launch an automatic semantic grader or own readiness, promotion, or production status.
15
+
16
+ ## Map Claims To Evidence
17
+
18
+ | Claim | Required evidence |
19
+ | --- | --- |
20
+ | Description routes the earliest useful prompt | Natural activation |
21
+ | Body works with guaranteed context | Explicit body delivery |
22
+ | Nearby prompt is not hijacked | Natural activation plus behavioral output |
23
+ | Declared dependencies compose | Exact ordered composition |
24
+ | Guidance remains useful later | Multi-turn evidence |
25
+ | Files or structured state match | Artifact outcome |
26
+ | Behavior holds on several hosts | The same group semantics on every named host |
27
+
28
+ Record declaration, materialization, delivery, observed reads, behavior, artifacts, and derived grades separately. A read does not prove compliance, and correct behavior does not prove which skill caused it.
29
+
30
+ ## Choose The Smallest Group Shape
31
+
32
+ - **Description prompt:** natural activation and exact read timing.
33
+ - **Explicit body task:** first-read instructions and behavior without activation ambiguity.
34
+ - **Fixture task:** files or repository state matter.
35
+ - **Stateful turns:** ordered conversation and later behavior matter.
36
+ - **Saved-result review:** stored runs already answer the question; inspect them instead of rerunning.
37
+ - **Suite:** independent questions need one ordered serial invocation.
38
+
39
+ Keep description and body questions separate unless the integrated path is itself the question.
40
+
41
+ ## Fix The Variants
42
+
43
+ Every group has exactly:
44
+
45
+ - `baseline`: no target for a new skill, or the exact previous snapshot for a revision;
46
+ - `candidate`: the new or updated snapshot.
47
+
48
+ For a description-only revision, keep body and resources byte-identical. Freeze prompts or turns, fixture, tools, model, thinking, other skills, context, and criteria across variants.
49
+
50
+ ## Design Common Questions
51
+
52
+ ### Description routing
53
+
54
+ Use the earliest natural prompt where the skill should become useful. For a nearby prompt, predeclare whether success means non-trigger or safe behavior after activation.
55
+
56
+ ### First-read body
57
+
58
+ Explicitly deliver the exact body with only guaranteed context. Ambient package context cannot repair a missing dependency.
59
+
60
+ ### Composition
61
+
62
+ Materialize exact ordered skills and context. Vary only the target snapshot. Standalone behavior should not receive a hidden base stack.
63
+
64
+ ### Retained use
65
+
66
+ Use declared turns in one persistent subject process. Repeated explicit delivery does not prove retained use.
67
+
68
+ ### Artifact outcome
69
+
70
+ Grade filesystem state, changed paths, file content, or JSON before relying on the final response. Preserve turn-scoped workspace evidence when later turns may change earlier state.
71
+
72
+ ## Boundary Examples
73
+
74
+ Saved complete evidence already contains the required run, grade, and artifacts:
75
+
76
+ > Use `view` and ordinary file reads. Do not rerun merely to demonstrate the process.
77
+
78
+ An explicitly delivered body produces the right answer, but the natural prompt never reads the target:
79
+
80
+ > The body group supports first-read behavior. The description group still fails activation; body success cannot repair it.
81
+
82
+ ## Apply Real Pressure
83
+
84
+ A useful group creates a real temptation to violate the rule, keeps criteria outside the prompt, exposes deterministic evidence where possible, and distinguishes baseline behavior from the candidate.
85
+
86
+ If both variants pass, the pressure may be weak or the baseline sufficient. If both fail, classify the skill, fixture, host, dependency, or criterion before editing.