@hassangameryt/freeflow 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (692) hide show
  1. package/.claude-plugin/marketplace.json +4 -4
  2. package/.claude-plugin/plugin.json +6 -2
  3. package/.codex-plugin/plugin.json +9 -9
  4. package/README.md +190 -127
  5. package/assets/feedback-loop.png +0 -0
  6. package/assets/freeflow_workflow_map.png +0 -0
  7. package/assets/workflow.png +0 -0
  8. package/{skills → capabilities}/output-router/SKILL.md +21 -17
  9. package/{skills → capabilities}/output-router/references/safety-policy.md +1 -1
  10. package/capabilities/output-router/references/setup.md +110 -0
  11. package/command-surface.json +62 -7
  12. package/hooks/freeflow-runtime-context.mjs +572 -154
  13. package/hooks/hooks.json +26 -1
  14. package/hooks/tests/check-runtime-context-hook.sh +375 -0
  15. package/package.json +24 -24
  16. package/pi-extension/dist/index.js +264 -80
  17. package/pi-extension/dist/integrations/mcp-config.js +29 -0
  18. package/pi-extension/dist/output-router/host-producer-identification.js +232 -0
  19. package/pi-extension/dist/output-router/native-safety-net.js +216 -0
  20. package/pi-extension/dist/output-router/observed-tool-routing.js +122 -0
  21. package/pi-extension/dist/output-router/renderers.js +1023 -0
  22. package/pi-extension/dist/output-router/router-tools.js +554 -0
  23. package/pi-extension/dist/output-router/schemas.js +383 -0
  24. package/pi-extension/dist/output-router/status.js +744 -0
  25. package/pi-extension/dist/output-router/utils.js +502 -0
  26. package/pi-extension/dist/runtime/runtime-context.js +798 -0
  27. package/pi-extension/dist/settings/settings-tui.js +314 -0
  28. package/pi-extension/dist/settings/settings-ui.js +1487 -0
  29. package/pi-extension/freeflow/index.js +1 -0
  30. package/pi-extension/src/index.ts +230 -56
  31. package/pi-extension/src/{host-producer-identification.ts → output-router/host-producer-identification.ts} +37 -7
  32. package/pi-extension/src/{native-safety-net.ts → output-router/native-safety-net.ts} +8 -12
  33. package/pi-extension/src/{observed-tool-routing.ts → output-router/observed-tool-routing.ts} +19 -11
  34. package/pi-extension/src/{renderers.ts → output-router/renderers.ts} +246 -55
  35. package/pi-extension/src/{router-tools.ts → output-router/router-tools.ts} +130 -29
  36. package/pi-extension/src/{schemas.ts → output-router/schemas.ts} +171 -43
  37. package/pi-extension/src/output-router/status.ts +791 -0
  38. package/pi-extension/src/{utils.ts → output-router/utils.ts} +80 -18
  39. package/pi-extension/src/runtime/runtime-context.ts +895 -0
  40. package/pi-extension/src/settings/settings-tui.ts +408 -0
  41. package/pi-extension/src/settings/settings-ui.ts +1730 -0
  42. package/pi-extension/tests/integration/pi-extension.test.js +4020 -0
  43. package/pi-extension/tests/output-router/observed-producers.test.js +132 -0
  44. package/pi-extension/tests/output-router/search.test.js +722 -0
  45. package/pi-extension/tests/settings/settings-tui.test.js +243 -0
  46. package/router/dist/benchmarks/benchmark-harness.d.ts +16 -13
  47. package/router/dist/benchmarks/benchmark-harness.js +62 -62
  48. package/router/dist/benchmarks/benchmarks.d.ts +80 -73
  49. package/router/dist/benchmarks/benchmarks.js +860 -761
  50. package/router/dist/benchmarks/codex-qa-benchmarks.d.ts +71 -67
  51. package/router/dist/benchmarks/codex-qa-benchmarks.js +429 -390
  52. package/router/dist/benchmarks/command-benchmarks.d.ts +107 -92
  53. package/router/dist/benchmarks/command-benchmarks.js +616 -571
  54. package/router/dist/benchmarks/context-mode-normalized-benchmarks.d.ts +64 -56
  55. package/router/dist/benchmarks/context-mode-normalized-benchmarks.js +679 -613
  56. package/router/dist/benchmarks/context-mode-real-deep-benchmark.d.ts +103 -95
  57. package/router/dist/benchmarks/context-mode-real-deep-benchmark.js +1188 -1103
  58. package/router/dist/benchmarks/context-saving-baseline.d.ts +47 -41
  59. package/router/dist/benchmarks/context-saving-baseline.js +405 -374
  60. package/router/dist/benchmarks/index-benchmarks.d.ts +74 -69
  61. package/router/dist/benchmarks/index-benchmarks.js +761 -661
  62. package/router/dist/benchmarks/storage-policy-benchmarks.d.ts +80 -69
  63. package/router/dist/benchmarks/storage-policy-benchmarks.js +549 -469
  64. package/router/dist/benchmarks/vault-index-benchmarks.d.ts +43 -34
  65. package/router/dist/benchmarks/vault-index-benchmarks.js +294 -229
  66. package/router/dist/config/config.d.ts +62 -55
  67. package/router/dist/config/config.js +552 -373
  68. package/router/dist/config/router-contract.d.ts +2 -2
  69. package/router/dist/config/router-contract.js +38 -38
  70. package/router/dist/config/schema.d.ts +19 -10
  71. package/router/dist/config/schema.js +586 -538
  72. package/router/dist/config/types.d.ts +413 -361
  73. package/router/dist/config/types.js +19 -8
  74. package/router/dist/evidence/bounded-evidence.d.ts +26 -26
  75. package/router/dist/evidence/bounded-evidence.js +265 -252
  76. package/router/dist/evidence/evidence-range-selector.d.ts +16 -14
  77. package/router/dist/evidence/evidence-range-selector.js +192 -191
  78. package/router/dist/evidence/evidence-search.d.ts +18 -16
  79. package/router/dist/evidence/evidence-search.js +401 -377
  80. package/router/dist/evidence/evidence.d.ts +17 -17
  81. package/router/dist/evidence/evidence.js +163 -159
  82. package/router/dist/evidence/failure-contracts.d.ts +25 -15
  83. package/router/dist/evidence/failure-contracts.js +132 -126
  84. package/router/dist/evidence/line-ranges.d.ts +17 -15
  85. package/router/dist/evidence/line-ranges.js +18 -18
  86. package/router/dist/experiments/local-index.d.ts +47 -41
  87. package/router/dist/experiments/local-index.js +477 -458
  88. package/router/dist/local/local-traversal.d.ts +18 -0
  89. package/router/dist/local/local-traversal.js +334 -0
  90. package/router/dist/processing/engine.d.ts +94 -70
  91. package/router/dist/processing/engine.js +462 -364
  92. package/router/dist/processing/reducers.d.ts +208 -194
  93. package/router/dist/processing/reducers.js +1434 -1256
  94. package/router/dist/processing/renderers.d.ts +24 -18
  95. package/router/dist/processing/renderers.js +161 -152
  96. package/router/dist/processing/scripts.d.ts +80 -67
  97. package/router/dist/processing/scripts.js +343 -290
  98. package/router/dist/repo/repo-traversal.d.ts +9 -9
  99. package/router/dist/repo/repo-traversal.js +205 -202
  100. package/router/dist/routing/observed-reducers.d.ts +25 -19
  101. package/router/dist/routing/observed-reducers.js +559 -545
  102. package/router/dist/routing/observed-routing.d.ts +23 -14
  103. package/router/dist/routing/observed-routing.js +319 -318
  104. package/router/dist/routing/parsers.d.ts +10 -10
  105. package/router/dist/routing/parsers.js +454 -406
  106. package/router/dist/routing/run-filters.d.ts +45 -41
  107. package/router/dist/routing/run-filters.js +215 -207
  108. package/router/dist/routing/run-reducers.d.ts +45 -26
  109. package/router/dist/routing/run-reducers.js +102 -90
  110. package/router/dist/sandbox/adapter-roots.d.ts +34 -26
  111. package/router/dist/sandbox/adapter-roots.js +43 -41
  112. package/router/dist/sandbox/eryx-python-adapter.d.ts +9 -5
  113. package/router/dist/sandbox/eryx-python-adapter.js +659 -588
  114. package/router/dist/sandbox/jq-wasm-adapter.d.ts +3 -3
  115. package/router/dist/sandbox/jq-wasm-adapter.js +319 -311
  116. package/router/dist/sandbox/quickjs-wasi-adapter.d.ts +9 -5
  117. package/router/dist/sandbox/quickjs-wasi-adapter.js +336 -333
  118. package/router/dist/sandbox/script-sandbox.d.ts +105 -84
  119. package/router/dist/sandbox/script-sandbox.js +294 -260
  120. package/router/dist/setup/script-transform-adapters.d.ts +23 -19
  121. package/router/dist/setup/script-transform-adapters.js +223 -204
  122. package/router/dist/tools/batch.d.ts +27 -16
  123. package/router/dist/tools/batch.js +493 -416
  124. package/router/dist/tools/run.d.ts +47 -35
  125. package/router/dist/tools/run.js +1172 -1087
  126. package/router/dist/tools/search.d.ts +53 -31
  127. package/router/dist/tools/search.js +1224 -1065
  128. package/router/dist/transform/engine.d.ts +121 -96
  129. package/router/dist/transform/engine.js +1853 -1736
  130. package/router/dist/vault/vault-index.d.ts +97 -78
  131. package/router/dist/vault/vault-index.js +347 -342
  132. package/router/dist/vault/vault.d.ts +109 -74
  133. package/router/dist/vault/vault.js +528 -533
  134. package/runtime/interaction-contract.md +16 -0
  135. package/skills/bypass/SKILL.md +52 -38
  136. package/skills/commit-work/SKILL.md +54 -46
  137. package/skills/commit-work/references/staging-decisions.md +46 -24
  138. package/skills/decision-gate/SKILL.md +48 -0
  139. package/skills/design-for-depth/SKILL.md +75 -59
  140. package/skills/design-for-depth/references/design-pressure-signals.md +56 -7
  141. package/skills/design-for-depth/references/interface-design-loop.md +156 -0
  142. package/skills/design-for-depth/references/software-design-philosophy.md +32 -3
  143. package/skills/diagnose-failure/SKILL.md +65 -60
  144. package/skills/diagnose-failure/references/diagnostic-loop-catalog.md +36 -0
  145. package/skills/diagnose-failure/references/flaky-and-performance.md +30 -18
  146. package/skills/discuss/SKILL.md +121 -0
  147. package/skills/discuss/references/checkpoints.md +66 -0
  148. package/skills/evaluate-skill/SKILL.md +62 -53
  149. package/skills/evaluate-skill/references/definition-schema.md +345 -0
  150. package/skills/evaluate-skill/references/evaluation-design.md +86 -0
  151. package/skills/evaluate-skill/references/execution-and-evidence.md +85 -0
  152. package/skills/evaluate-skill/references/review-and-revision.md +51 -0
  153. package/skills/evaluate-skill/scripts/lib/definitions.mjs +296 -0
  154. package/skills/evaluate-skill/scripts/lib/evidence.mjs +46 -0
  155. package/skills/evaluate-skill/scripts/lib/grade-comparison.mjs +60 -0
  156. package/skills/evaluate-skill/scripts/lib/grade-interaction.mjs +148 -0
  157. package/skills/evaluate-skill/scripts/lib/grade-support.mjs +126 -0
  158. package/skills/evaluate-skill/scripts/lib/grade-workspace.mjs +192 -0
  159. package/skills/evaluate-skill/scripts/lib/grade.mjs +56 -0
  160. package/skills/evaluate-skill/scripts/lib/pi.mjs +489 -0
  161. package/skills/evaluate-skill/scripts/lib/process.mjs +79 -0
  162. package/skills/evaluate-skill/scripts/lib/rpc.mjs +444 -0
  163. package/skills/evaluate-skill/scripts/lib/runner.mjs +346 -0
  164. package/skills/evaluate-skill/scripts/lib/sandbox.mjs +409 -0
  165. package/skills/evaluate-skill/scripts/lib/view.mjs +337 -0
  166. package/skills/evaluate-skill/scripts/pi-guard.mjs +164 -0
  167. package/skills/evaluate-skill/scripts/skill-eval.mjs +101 -0
  168. package/skills/execute-work/SKILL.md +99 -0
  169. package/skills/execute-work/references/code-practices.md +78 -0
  170. package/skills/execute-work/references/execution-loop.md +77 -0
  171. package/skills/finish-branch/SKILL.md +117 -0
  172. package/skills/finish-branch/references/integration-options.md +80 -0
  173. package/skills/handoff/SKILL.md +65 -39
  174. package/skills/handoff/references/templates.md +104 -27
  175. package/skills/launch-work/SKILL.md +116 -0
  176. package/skills/launch-work/references/launch-readiness.md +85 -0
  177. package/skills/migration-work/SKILL.md +115 -0
  178. package/skills/migration-work/references/migration-lifecycle.md +82 -0
  179. package/skills/mode-contract/SKILL.md +72 -52
  180. package/skills/release-work/SKILL.md +118 -0
  181. package/skills/release-work/references/release-evidence.md +78 -0
  182. package/skills/review-artifact/SKILL.md +77 -69
  183. package/skills/review-artifact/references/reviewer-prompt.md +133 -42
  184. package/skills/review-work/SKILL.md +82 -57
  185. package/skills/review-work/references/reviewer-prompt.md +97 -60
  186. package/skills/review-work/references/security-risk-lens.md +67 -0
  187. package/skills/setup-freeflow/SKILL.md +54 -68
  188. package/skills/setup-freeflow/references/activation-contract.md +84 -69
  189. package/skills/setup-freeflow/references/host-setup.md +50 -17
  190. package/skills/simplify-code/SKILL.md +80 -0
  191. package/skills/simplify-code/references/simplification-patterns.md +70 -0
  192. package/skills/tdd/SKILL.md +96 -0
  193. package/skills/tdd/references/test-design.md +96 -0
  194. package/skills/track-work/SKILL.md +84 -0
  195. package/skills/track-work/references/working-record-schema.md +378 -0
  196. package/skills/verify-work/SKILL.md +59 -38
  197. package/skills/verify-work/references/browser-runtime-evidence.md +55 -0
  198. package/skills/verify-work/references/integration-evidence.md +51 -0
  199. package/skills/verify-work/references/performance-evidence.md +57 -0
  200. package/skills/workflow/SKILL.md +53 -59
  201. package/skills/workflow/references/domain-skill-composition.md +57 -0
  202. package/skills/workflow/references/workflow-loop.md +92 -0
  203. package/skills/write-plan/SKILL.md +85 -58
  204. package/skills/write-plan/references/plan-shapes.md +132 -46
  205. package/skills/write-skill/SKILL.md +82 -51
  206. package/skills/write-skill/references/activation-boundaries.md +62 -0
  207. package/skills/write-skill/references/agent-first-instructions.md +87 -0
  208. package/skills/write-skill/references/progressive-disclosure.md +63 -0
  209. package/skills/write-skill/scripts/lib/skill-author.mjs +653 -0
  210. package/skills/write-skill/scripts/skill-author.mjs +104 -0
  211. package/skills/write-spec/SKILL.md +65 -54
  212. package/skills/write-spec/references/artifact-standards.md +60 -41
  213. package/skills/write-spec/references/decision-records.md +77 -0
  214. package/skills/write-spec/references/spec-shapes.md +119 -60
  215. package/evals/README.md +0 -32
  216. package/evals/fixtures/copy-conventions.md +0 -31
  217. package/evals/fixtures/output-router/README.md +0 -12
  218. package/evals/fixtures/output-router/failed-command-output.txt +0 -9
  219. package/evals/fixtures/output-router/large-router-manual.md +0 -63
  220. package/evals/fixtures/output-router/native-large-output.txt +0 -40
  221. package/evals/fixtures/output-router/noisy-test-output.txt +0 -30
  222. package/evals/fixtures/output-router/verification-output.txt +0 -8
  223. package/evals/fixtures/tiny-artifact-review-app/README.md +0 -5
  224. package/evals/fixtures/tiny-artifact-review-app/docs/billing-policy.md +0 -10
  225. package/evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md +0 -21
  226. package/evals/fixtures/tiny-artifact-review-app/docs/specs/failed-payment-grace-period.md +0 -24
  227. package/evals/fixtures/tiny-artifact-review-app/docs/specs/immediate-downgrade.md +0 -17
  228. package/evals/fixtures/tiny-artifact-review-app/tests/billing.test.md +0 -8
  229. package/evals/fixtures/tiny-commit-ambiguous-app/.freeflow-eval-setup.sh +0 -15
  230. package/evals/fixtures/tiny-commit-ambiguous-app/README.md +0 -3
  231. package/evals/fixtures/tiny-commit-ambiguous-app/src/billing.ts +0 -5
  232. package/evals/fixtures/tiny-commit-ambiguous-app/src/settings-page.ts +0 -1
  233. package/evals/fixtures/tiny-commit-ambiguous-app/tests/settings.test.md +0 -5
  234. package/evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh +0 -14
  235. package/evals/fixtures/tiny-commit-app/README.md +0 -3
  236. package/evals/fixtures/tiny-commit-app/docs/specs/settings-title.md +0 -5
  237. package/evals/fixtures/tiny-commit-app/src/billing.ts +0 -5
  238. package/evals/fixtures/tiny-commit-app/src/settings-page.ts +0 -1
  239. package/evals/fixtures/tiny-commit-app/tests/settings.test.md +0 -5
  240. package/evals/fixtures/tiny-commit-everything-app/.freeflow-eval-setup.sh +0 -15
  241. package/evals/fixtures/tiny-commit-everything-app/README.md +0 -3
  242. package/evals/fixtures/tiny-commit-everything-app/docs/specs/settings-title.md +0 -5
  243. package/evals/fixtures/tiny-commit-everything-app/src/billing.ts +0 -5
  244. package/evals/fixtures/tiny-commit-everything-app/src/settings-page.ts +0 -1
  245. package/evals/fixtures/tiny-commit-everything-app/tests/settings.test.md +0 -5
  246. package/evals/fixtures/tiny-commit-staged-app/.freeflow-eval-setup.sh +0 -15
  247. package/evals/fixtures/tiny-commit-staged-app/README.md +0 -3
  248. package/evals/fixtures/tiny-commit-staged-app/docs/specs/settings-title.md +0 -5
  249. package/evals/fixtures/tiny-commit-staged-app/src/billing.ts +0 -5
  250. package/evals/fixtures/tiny-commit-staged-app/src/settings-page.ts +0 -1
  251. package/evals/fixtures/tiny-commit-staged-app/tests/settings.test.md +0 -5
  252. package/evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh +0 -15
  253. package/evals/fixtures/tiny-commit-staged-mixed-app/README.md +0 -3
  254. package/evals/fixtures/tiny-commit-staged-mixed-app/docs/billing-policy.md +0 -3
  255. package/evals/fixtures/tiny-commit-staged-mixed-app/docs/release-notes.md +0 -3
  256. package/evals/fixtures/tiny-commit-staged-mixed-app/src/billing.ts +0 -5
  257. package/evals/fixtures/tiny-dashboard-stale-app/README.md +0 -11
  258. package/evals/fixtures/tiny-dashboard-stale-app/src/api-client.ts +0 -13
  259. package/evals/fixtures/tiny-dashboard-stale-app/src/dashboard-data.ts +0 -19
  260. package/evals/fixtures/tiny-dashboard-stale-app/tests/dashboard-data.test.md +0 -9
  261. package/evals/fixtures/tiny-decision-destination-app/README.md +0 -6
  262. package/evals/fixtures/tiny-decision-destination-app/src/risk.ts +0 -8
  263. package/evals/fixtures/tiny-decision-destination-app/tests/risk.test.md +0 -4
  264. package/evals/fixtures/tiny-design-depth-app/README.md +0 -15
  265. package/evals/fixtures/tiny-design-depth-app/src/routes.ts +0 -25
  266. package/evals/fixtures/tiny-design-depth-app/src/sendEmail.ts +0 -3
  267. package/evals/fixtures/tiny-design-depth-app/src/sendSms.ts +0 -3
  268. package/evals/fixtures/tiny-design-depth-app/src/telemetry.ts +0 -3
  269. package/evals/fixtures/tiny-design-depth-app/tests/routes.test.md +0 -12
  270. package/evals/fixtures/tiny-eval-method-app/README.md +0 -8
  271. package/evals/fixtures/tiny-eval-method-app/evals/reports/review-pr-eval-result.md +0 -5
  272. package/evals/fixtures/tiny-eval-method-app/evals/reports/triage-question-failure.md +0 -17
  273. package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.diff +0 -11
  274. package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.md +0 -3
  275. package/evals/fixtures/tiny-eval-method-app/skills/review-pr/SKILL.md +0 -10
  276. package/evals/fixtures/tiny-eval-method-app/skills/triage-question/SKILL.md +0 -10
  277. package/evals/fixtures/tiny-execute-plan-app/README.md +0 -5
  278. package/evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh +0 -5
  279. package/evals/fixtures/tiny-execute-plan-app/docs/billing-policy.md +0 -10
  280. package/evals/fixtures/tiny-execute-plan-app/docs/plans/immediate-downgrade-plan.md +0 -11
  281. package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-plan.md +0 -12
  282. package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md +0 -15
  283. package/evals/fixtures/tiny-execute-plan-app/docs/plans/unverified-billing-plan.md +0 -9
  284. package/evals/fixtures/tiny-execute-plan-app/docs/specs/failed-payment-grace-period.md +0 -9
  285. package/evals/fixtures/tiny-execute-plan-app/docs/specs/settings-title.md +0 -9
  286. package/evals/fixtures/tiny-execute-plan-app/src/billing.ts +0 -7
  287. package/evals/fixtures/tiny-execute-plan-app/src/settings-page.ts +0 -1
  288. package/evals/fixtures/tiny-execute-plan-app/tests/billing.test.md +0 -7
  289. package/evals/fixtures/tiny-execute-plan-app/tests/settings.test.md +0 -5
  290. package/evals/fixtures/tiny-execute-review-loop-app/README.md +0 -5
  291. package/evals/fixtures/tiny-execute-review-loop-app/docs/billing-policy.md +0 -9
  292. package/evals/fixtures/tiny-execute-review-loop-app/docs/plans/settings-title-plan.md +0 -12
  293. package/evals/fixtures/tiny-execute-review-loop-app/docs/review/previous-agent-summary.md +0 -15
  294. package/evals/fixtures/tiny-execute-review-loop-app/docs/review/review-pass-1-non-pass.md +0 -11
  295. package/evals/fixtures/tiny-execute-review-loop-app/docs/specs/settings-title.md +0 -9
  296. package/evals/fixtures/tiny-execute-review-loop-app/src/billing.ts +0 -7
  297. package/evals/fixtures/tiny-execute-review-loop-app/src/settings-page.ts +0 -1
  298. package/evals/fixtures/tiny-execute-review-loop-app/tests/billing.test.md +0 -7
  299. package/evals/fixtures/tiny-execute-review-loop-app/tests/settings.test.md +0 -5
  300. package/evals/fixtures/tiny-flaky-session-app/README.md +0 -10
  301. package/evals/fixtures/tiny-flaky-session-app/src/session-timeout.ts +0 -13
  302. package/evals/fixtures/tiny-flaky-session-app/tests/session-timeout.test.md +0 -8
  303. package/evals/fixtures/tiny-handoff-app/README.md +0 -5
  304. package/evals/fixtures/tiny-handoff-app/docs/billing-policy.md +0 -10
  305. package/evals/fixtures/tiny-handoff-app/docs/handoffs/old-agent-handoff.md +0 -8
  306. package/evals/fixtures/tiny-handoff-app/docs/plans/billing-retry-plan.md +0 -11
  307. package/evals/fixtures/tiny-handoff-app/src/billing.ts +0 -7
  308. package/evals/fixtures/tiny-handoff-report-app/README.md +0 -3
  309. package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.diff +0 -1
  310. package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.md +0 -1
  311. package/evals/fixtures/tiny-handoff-report-app/evals/scores/hof-003-score.txt +0 -4
  312. package/evals/fixtures/tiny-plan-app/README.md +0 -5
  313. package/evals/fixtures/tiny-plan-app/docs/billing-policy.md +0 -10
  314. package/evals/fixtures/tiny-plan-app/docs/specs/failed-payment-grace-period.md +0 -25
  315. package/evals/fixtures/tiny-plan-app/src/billing.ts +0 -17
  316. package/evals/fixtures/tiny-plan-app/src/settings-page.ts +0 -1
  317. package/evals/fixtures/tiny-plan-app/tests/billing.test.md +0 -8
  318. package/evals/fixtures/tiny-plan-app/tests/settings.test.md +0 -5
  319. package/evals/fixtures/tiny-plan-strict-api-app/README.md +0 -3
  320. package/evals/fixtures/tiny-plan-strict-api-app/docs/billing-policy.md +0 -5
  321. package/evals/fixtures/tiny-plan-strict-api-app/docs/specs/billing-webhook-api.md +0 -25
  322. package/evals/fixtures/tiny-plan-strict-api-app/src/billing.ts +0 -5
  323. package/evals/fixtures/tiny-plan-strict-api-app/src/webhooks.ts +0 -3
  324. package/evals/fixtures/tiny-plan-strict-api-app/tests/billing-webhook.test.md +0 -8
  325. package/evals/fixtures/tiny-post-setup-invalid-config-app/.freeflow/config.json +0 -3
  326. package/evals/fixtures/tiny-post-setup-invalid-config-app/AGENTS.md +0 -15
  327. package/evals/fixtures/tiny-post-setup-invalid-config-app/README.md +0 -16
  328. package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/auth-review.md +0 -15
  329. package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/billing-policy.md +0 -10
  330. package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/copy-conventions.md +0 -15
  331. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/auth-middleware.ts +0 -12
  332. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/billing.ts +0 -20
  333. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/login-copy.ts +0 -1
  334. package/evals/fixtures/tiny-post-setup-invalid-config-app/src/settings-page.ts +0 -1
  335. package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/auth-middleware.test.md +0 -8
  336. package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/billing.test.md +0 -8
  337. package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/copy.test.md +0 -7
  338. package/evals/fixtures/tiny-post-setup-missing-config-app/AGENTS.md +0 -15
  339. package/evals/fixtures/tiny-post-setup-missing-config-app/README.md +0 -16
  340. package/evals/fixtures/tiny-post-setup-missing-config-app/docs/auth-review.md +0 -15
  341. package/evals/fixtures/tiny-post-setup-missing-config-app/docs/billing-policy.md +0 -10
  342. package/evals/fixtures/tiny-post-setup-missing-config-app/docs/copy-conventions.md +0 -15
  343. package/evals/fixtures/tiny-post-setup-missing-config-app/src/auth-middleware.ts +0 -12
  344. package/evals/fixtures/tiny-post-setup-missing-config-app/src/billing.ts +0 -20
  345. package/evals/fixtures/tiny-post-setup-missing-config-app/src/login-copy.ts +0 -1
  346. package/evals/fixtures/tiny-post-setup-missing-config-app/src/settings-page.ts +0 -1
  347. package/evals/fixtures/tiny-post-setup-missing-config-app/tests/auth-middleware.test.md +0 -8
  348. package/evals/fixtures/tiny-post-setup-missing-config-app/tests/billing.test.md +0 -8
  349. package/evals/fixtures/tiny-post-setup-missing-config-app/tests/copy.test.md +0 -7
  350. package/evals/fixtures/tiny-post-setup-source-conflict-app/.freeflow/config.json +0 -3
  351. package/evals/fixtures/tiny-post-setup-source-conflict-app/AGENTS.md +0 -15
  352. package/evals/fixtures/tiny-post-setup-source-conflict-app/README.md +0 -16
  353. package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/auth-review.md +0 -15
  354. package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/billing-policy.md +0 -10
  355. package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/copy-conventions.md +0 -15
  356. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/auth-middleware.ts +0 -12
  357. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/billing.ts +0 -20
  358. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/login-copy.ts +0 -1
  359. package/evals/fixtures/tiny-post-setup-source-conflict-app/src/settings-page.ts +0 -1
  360. package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/auth-middleware.test.md +0 -8
  361. package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/billing.test.md +0 -8
  362. package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/copy.test.md +0 -7
  363. package/evals/fixtures/tiny-report-performance-app/README.md +0 -11
  364. package/evals/fixtures/tiny-report-performance-app/docs/performance.md +0 -10
  365. package/evals/fixtures/tiny-report-performance-app/src/report-summary.ts +0 -29
  366. package/evals/fixtures/tiny-report-performance-app/tests/report-performance.test.md +0 -8
  367. package/evals/fixtures/tiny-review-request-app/README.md +0 -3
  368. package/evals/fixtures/tiny-review-request-app/docs/billing-policy.md +0 -5
  369. package/evals/fixtures/tiny-review-request-app/docs/plans/billing-webhook-api.md +0 -19
  370. package/evals/fixtures/tiny-review-request-app/docs/review/previous-agent-summary.md +0 -20
  371. package/evals/fixtures/tiny-review-request-app/docs/specs/billing-webhook-api.md +0 -25
  372. package/evals/fixtures/tiny-review-request-app/src/billing.ts +0 -11
  373. package/evals/fixtures/tiny-review-request-app/src/webhooks.ts +0 -19
  374. package/evals/fixtures/tiny-review-request-app/tests/billing-webhook.test.md +0 -8
  375. package/evals/fixtures/tiny-review-work-app/README.md +0 -5
  376. package/evals/fixtures/tiny-review-work-app/docs/billing-policy.md +0 -9
  377. package/evals/fixtures/tiny-review-work-app/docs/plans/settings-title-plan.md +0 -12
  378. package/evals/fixtures/tiny-review-work-app/docs/review/previous-agent-summary.md +0 -15
  379. package/evals/fixtures/tiny-review-work-app/docs/specs/settings-title.md +0 -9
  380. package/evals/fixtures/tiny-review-work-app/src/billing.ts +0 -7
  381. package/evals/fixtures/tiny-review-work-app/src/settings-page.ts +0 -1
  382. package/evals/fixtures/tiny-review-work-app/tests/billing.test.md +0 -7
  383. package/evals/fixtures/tiny-review-work-app/tests/settings.test.md +0 -5
  384. package/evals/fixtures/tiny-saas-app/README.md +0 -16
  385. package/evals/fixtures/tiny-saas-app/docs/auth-review.md +0 -15
  386. package/evals/fixtures/tiny-saas-app/docs/billing-policy.md +0 -10
  387. package/evals/fixtures/tiny-saas-app/docs/copy-conventions.md +0 -15
  388. package/evals/fixtures/tiny-saas-app/docs/specs/settings-title.md +0 -13
  389. package/evals/fixtures/tiny-saas-app/src/auth-middleware.ts +0 -12
  390. package/evals/fixtures/tiny-saas-app/src/billing.ts +0 -20
  391. package/evals/fixtures/tiny-saas-app/src/login-copy.ts +0 -1
  392. package/evals/fixtures/tiny-saas-app/src/settings-page.ts +0 -1
  393. package/evals/fixtures/tiny-saas-app/tests/auth-middleware.test.md +0 -8
  394. package/evals/fixtures/tiny-saas-app/tests/billing.test.md +0 -8
  395. package/evals/fixtures/tiny-saas-app/tests/copy.test.md +0 -7
  396. package/evals/fixtures/tiny-setup-agents-app/AGENTS.md +0 -6
  397. package/evals/fixtures/tiny-setup-both-app/AGENTS.md +0 -5
  398. package/evals/fixtures/tiny-setup-both-app/CLAUDE.md +0 -5
  399. package/evals/fixtures/tiny-setup-claude-app/CLAUDE.md +0 -6
  400. package/evals/fixtures/tiny-setup-codex-rules-app/.codex/rules/shell.rules +0 -5
  401. package/evals/fixtures/tiny-setup-codex-rules-app/AGENTS.md +0 -6
  402. package/evals/fixtures/tiny-setup-conflict-app/AGENTS.md +0 -6
  403. package/evals/fixtures/tiny-setup-existing-block-app/AGENTS.md +0 -14
  404. package/evals/fixtures/tiny-setup-four-rules-app/CLAUDE.md +0 -6
  405. package/evals/fixtures/tiny-skill-dev-app/README.md +0 -10
  406. package/evals/fixtures/tiny-skill-dev-app/evals/prompts/review-pr-001.txt +0 -8
  407. package/evals/fixtures/tiny-skill-dev-app/evals/reports/review-pr-failure.md +0 -17
  408. package/evals/fixtures/tiny-skill-dev-app/skills/review-pr/SKILL.md +0 -13
  409. package/evals/fixtures/tiny-team-app/README.md +0 -5
  410. package/evals/fixtures/tiny-team-app/docs/product.md +0 -6
  411. package/evals/fixtures/tiny-team-app/src/teams.ts +0 -9
  412. package/evals/fixtures/tiny-team-app/src/users.ts +0 -5
  413. package/evals/fixtures/tiny-team-app/tests/teams.test.md +0 -7
  414. package/evals/fixtures/tiny-verify-app/README.md +0 -5
  415. package/evals/fixtures/tiny-verify-app/src/settings-nav.ts +0 -1
  416. package/evals/fixtures/tiny-verify-app/src/settings-page.ts +0 -1
  417. package/evals/fixtures/tiny-verify-app/tests/settings.test.md +0 -7
  418. package/evals/prompts/aon-001.txt +0 -2
  419. package/evals/prompts/byp-001.txt +0 -2
  420. package/evals/prompts/byp-002.txt +0 -2
  421. package/evals/prompts/cap-001.txt +0 -3
  422. package/evals/prompts/cap-002.txt +0 -3
  423. package/evals/prompts/cmd-001.txt +0 -3
  424. package/evals/prompts/cmd-002.txt +0 -3
  425. package/evals/prompts/cmd-003.txt +0 -6
  426. package/evals/prompts/cmd-004.txt +0 -5
  427. package/evals/prompts/cmd-005.txt +0 -5
  428. package/evals/prompts/cmd-006.txt +0 -5
  429. package/evals/prompts/cmd-007.txt +0 -5
  430. package/evals/prompts/cmd-008.txt +0 -5
  431. package/evals/prompts/cmd-009.txt +0 -8
  432. package/evals/prompts/cmd-010.txt +0 -5
  433. package/evals/prompts/cmd-011.txt +0 -5
  434. package/evals/prompts/cmd-012.txt +0 -5
  435. package/evals/prompts/cmd-013.txt +0 -5
  436. package/evals/prompts/cmd-014.txt +0 -7
  437. package/evals/prompts/cmd-015.txt +0 -7
  438. package/evals/prompts/cmd-016.txt +0 -5
  439. package/evals/prompts/cmt-001.txt +0 -1
  440. package/evals/prompts/cmt-002.txt +0 -1
  441. package/evals/prompts/cmt-003.txt +0 -1
  442. package/evals/prompts/cmt-004.txt +0 -1
  443. package/evals/prompts/cmt-005.txt +0 -1
  444. package/evals/prompts/dfd-001.txt +0 -3
  445. package/evals/prompts/dia-001.txt +0 -3
  446. package/evals/prompts/dia-002.txt +0 -3
  447. package/evals/prompts/dia-003.txt +0 -3
  448. package/evals/prompts/dis-001.txt +0 -5
  449. package/evals/prompts/dis-002.txt +0 -5
  450. package/evals/prompts/esk-001.txt +0 -5
  451. package/evals/prompts/esk-002.txt +0 -5
  452. package/evals/prompts/esk-003.txt +0 -5
  453. package/evals/prompts/grc-001.txt +0 -3
  454. package/evals/prompts/hof-001.txt +0 -3
  455. package/evals/prompts/hof-002.txt +0 -1
  456. package/evals/prompts/hof-003.txt +0 -1
  457. package/evals/prompts/hof-004.txt +0 -1
  458. package/evals/prompts/hof-005.txt +0 -1
  459. package/evals/prompts/ivg-001.txt +0 -1
  460. package/evals/prompts/ivg-002.txt +0 -11
  461. package/evals/prompts/ivg-003.txt +0 -1
  462. package/evals/prompts/ivg-004.txt +0 -1
  463. package/evals/prompts/mode-001.txt +0 -3
  464. package/evals/prompts/mode-002.txt +0 -1
  465. package/evals/prompts/mode-003.txt +0 -3
  466. package/evals/prompts/mode-004.txt +0 -3
  467. package/evals/prompts/mode-005.txt +0 -3
  468. package/evals/prompts/mode-006.txt +0 -3
  469. package/evals/prompts/mode-007.txt +0 -3
  470. package/evals/prompts/otr-001.txt +0 -3
  471. package/evals/prompts/otr-002.txt +0 -3
  472. package/evals/prompts/rar-001.txt +0 -5
  473. package/evals/prompts/rar-002.txt +0 -3
  474. package/evals/prompts/rar-003.txt +0 -6
  475. package/evals/prompts/rar-004.txt +0 -15
  476. package/evals/prompts/rar-005.txt +0 -10
  477. package/evals/prompts/rbr-001.txt +0 -3
  478. package/evals/prompts/rbr-002.txt +0 -3
  479. package/evals/prompts/rbr-003.txt +0 -7
  480. package/evals/prompts/rev-001.txt +0 -3
  481. package/evals/prompts/rev-002.txt +0 -5
  482. package/evals/prompts/rev-003.txt +0 -6
  483. package/evals/prompts/rev-004.txt +0 -1
  484. package/evals/prompts/rev-005.txt +0 -15
  485. package/evals/prompts/rev-006.txt +0 -5
  486. package/evals/prompts/stp-001.txt +0 -1
  487. package/evals/prompts/stp-002.txt +0 -1
  488. package/evals/prompts/stp-003.txt +0 -1
  489. package/evals/prompts/stp-004.txt +0 -1
  490. package/evals/prompts/stp-005.txt +0 -1
  491. package/evals/prompts/stp-006.txt +0 -1
  492. package/evals/prompts/stp-007.txt +0 -1
  493. package/evals/prompts/stp-008.txt +0 -1
  494. package/evals/prompts/stp-009.txt +0 -1
  495. package/evals/prompts/stp-010.txt +0 -1
  496. package/evals/prompts/stp-011.txt +0 -1
  497. package/evals/prompts/vfy-003.txt +0 -1
  498. package/evals/prompts/wpl-001.txt +0 -3
  499. package/evals/prompts/wpl-002.txt +0 -5
  500. package/evals/prompts/wpl-003.txt +0 -3
  501. package/evals/prompts/wpl-004.txt +0 -5
  502. package/evals/prompts/wpl-005.txt +0 -1
  503. package/evals/prompts/wrc-001.txt +0 -10
  504. package/evals/prompts/wrc-002.txt +0 -3
  505. package/evals/prompts/wsk-001.txt +0 -6
  506. package/evals/prompts/wsp-001.txt +0 -5
  507. package/evals/prompts/wsp-002.txt +0 -15
  508. package/evals/prompts/wsp-003.txt +0 -3
  509. package/evals/prompts/wsp-004.txt +0 -15
  510. package/evals/prompts/wsp-005.txt +0 -5
  511. package/evals/prompts/wsp-006.txt +0 -12
  512. package/evals/prompts/wsp-007.txt +0 -9
  513. package/evals/prompts/wsp-008.txt +0 -13
  514. package/evals/prompts/xpl-001.txt +0 -1
  515. package/evals/prompts/xpl-002.txt +0 -3
  516. package/evals/prompts/xpl-003.txt +0 -3
  517. package/evals/prompts/xpl-004.txt +0 -3
  518. package/evals/prompts/xpl-005.txt +0 -5
  519. package/evals/prompts/xpl-006.txt +0 -3
  520. package/evals/registries/adversarial-evals.json +0 -39
  521. package/evals/registries/fixture-evals.json +0 -2513
  522. package/evals/registries/handoff-evals.json +0 -74
  523. package/evals/registries/legacy-evals.json +0 -147
  524. package/evals/registries/skill-evidence.json +0 -193
  525. package/evals/registries/verify-work-evals.json +0 -46
  526. package/evals/reports/acceptance/v0.1-acceptance-report.md +0 -130
  527. package/evals/reports/by-command-surface/command-surface-1-report.md +0 -45
  528. package/evals/reports/by-command-surface/command-surface-10-report.md +0 -76
  529. package/evals/reports/by-command-surface/command-surface-11-report.md +0 -77
  530. package/evals/reports/by-command-surface/command-surface-12-report.md +0 -91
  531. package/evals/reports/by-command-surface/command-surface-13-report.md +0 -64
  532. package/evals/reports/by-command-surface/command-surface-14-report.md +0 -66
  533. package/evals/reports/by-command-surface/command-surface-15-report.md +0 -124
  534. package/evals/reports/by-command-surface/command-surface-16-report.md +0 -39
  535. package/evals/reports/by-command-surface/command-surface-2-report.md +0 -80
  536. package/evals/reports/by-command-surface/command-surface-3-report.md +0 -86
  537. package/evals/reports/by-command-surface/command-surface-4-report.md +0 -76
  538. package/evals/reports/by-command-surface/command-surface-5-report.md +0 -80
  539. package/evals/reports/by-command-surface/command-surface-6-report.md +0 -74
  540. package/evals/reports/by-command-surface/command-surface-7-report.md +0 -84
  541. package/evals/reports/by-command-surface/command-surface-8-report.md +0 -79
  542. package/evals/reports/by-command-surface/command-surface-9-report.md +0 -79
  543. package/evals/reports/by-command-surface/command-surface-matrix.md +0 -91
  544. package/evals/reports/by-skill/bypass-1-report.md +0 -124
  545. package/evals/reports/by-skill/capture-decisions-1-report.md +0 -99
  546. package/evals/reports/by-skill/capture-decisions-2-report.md +0 -102
  547. package/evals/reports/by-skill/commit-work-1-report.md +0 -165
  548. package/evals/reports/by-skill/commit-work-2-report.md +0 -109
  549. package/evals/reports/by-skill/design-for-depth-1-report.md +0 -95
  550. package/evals/reports/by-skill/diagnose-failure-1-report.md +0 -91
  551. package/evals/reports/by-skill/diagnose-failure-2-report.md +0 -94
  552. package/evals/reports/by-skill/discover-1-report.md +0 -118
  553. package/evals/reports/by-skill/evaluate-skill-2-report.md +0 -71
  554. package/evals/reports/by-skill/execute-plan-1-report.md +0 -103
  555. package/evals/reports/by-skill/execute-plan-2-report.md +0 -112
  556. package/evals/reports/by-skill/execute-plan-3-report.md +0 -87
  557. package/evals/reports/by-skill/grill-context-1-report.md +0 -103
  558. package/evals/reports/by-skill/handoff-1-report.md +0 -123
  559. package/evals/reports/by-skill/handoff-2-report.md +0 -105
  560. package/evals/reports/by-skill/handoff-3-report.md +0 -100
  561. package/evals/reports/by-skill/handoff-7-report.md +0 -77
  562. package/evals/reports/by-skill/interview-gate-1-report.md +0 -63
  563. package/evals/reports/by-skill/interview-gate-2-report.md +0 -105
  564. package/evals/reports/by-skill/mode-contract-1-report.md +0 -74
  565. package/evals/reports/by-skill/mode-contract-2-report.md +0 -87
  566. package/evals/reports/by-skill/mode-contract-3-report.md +0 -65
  567. package/evals/reports/by-skill/mode-contract-4-report.md +0 -71
  568. package/evals/reports/by-skill/mode-contract-5-report.md +0 -70
  569. package/evals/reports/by-skill/output-router-1-report.md +0 -76
  570. package/evals/reports/by-skill/output-router-2-report.md +0 -65
  571. package/evals/reports/by-skill/research-1-report.md +0 -144
  572. package/evals/reports/by-skill/research-brief-1-report.md +0 -94
  573. package/evals/reports/by-skill/research-brief-2-report.md +0 -75
  574. package/evals/reports/by-skill/review-artifact-1-report.md +0 -91
  575. package/evals/reports/by-skill/review-artifact-2-report.md +0 -79
  576. package/evals/reports/by-skill/review-artifact-3-report.md +0 -120
  577. package/evals/reports/by-skill/review-artifact-4-report.md +0 -89
  578. package/evals/reports/by-skill/review-artifact-5-report.md +0 -66
  579. package/evals/reports/by-skill/review-work-1-report.md +0 -85
  580. package/evals/reports/by-skill/review-work-2-report.md +0 -87
  581. package/evals/reports/by-skill/review-work-3-report.md +0 -104
  582. package/evals/reports/by-skill/review-work-4-report.md +0 -108
  583. package/evals/reports/by-skill/review-work-5-report.md +0 -73
  584. package/evals/reports/by-skill/review-work-6-report.md +0 -66
  585. package/evals/reports/by-skill/setup-freeflow-1-report.md +0 -199
  586. package/evals/reports/by-skill/setup-freeflow-2-report.md +0 -105
  587. package/evals/reports/by-skill/setup-freeflow-3-report.md +0 -105
  588. package/evals/reports/by-skill/setup-freeflow-4-report.md +0 -88
  589. package/evals/reports/by-skill/setup-freeflow-5-report.md +0 -115
  590. package/evals/reports/by-skill/skill-authoring-1-report.md +0 -81
  591. package/evals/reports/by-skill/verify-work-1-report.md +0 -111
  592. package/evals/reports/by-skill/verify-work-2-report.md +0 -120
  593. package/evals/reports/by-skill/workflow-1-report.md +0 -65
  594. package/evals/reports/by-skill/write-plan-1-report.md +0 -123
  595. package/evals/reports/by-skill/write-plan-2-report.md +0 -98
  596. package/evals/reports/by-skill/write-plan-3-report.md +0 -110
  597. package/evals/reports/by-skill/write-spec-1-report.md +0 -88
  598. package/evals/reports/by-skill/write-spec-2-report.md +0 -89
  599. package/evals/reports/by-skill/write-spec-3-report.md +0 -90
  600. package/evals/reports/by-skill/write-spec-4-report.md +0 -167
  601. package/evals/reports/harness/claude-harness-1-report.md +0 -103
  602. package/evals/reports/harness/eval-harness-1-report.md +0 -91
  603. package/evals/reports/iterations/iteration-1-report.md +0 -71
  604. package/evals/reports/iterations/iteration-2-report.md +0 -78
  605. package/evals/reports/iterations/iteration-3-report.md +0 -118
  606. package/evals/reports/iterations/iteration-4-report.md +0 -131
  607. package/evals/reports/iterations/iteration-5-report.md +0 -149
  608. package/evals/reports/iterations/iteration-6-report.md +0 -136
  609. package/evals/reports/runtime/always-on-runtime-1-report.md +0 -107
  610. package/evals/reports/runtime/context-mode-normalized-benchmark-1-report.md +0 -52
  611. package/evals/reports/runtime/context-mode-real-deep-baseline-1-report.md +0 -410
  612. package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md +0 -203
  613. package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md +0 -51
  614. package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md +0 -99
  615. package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md +0 -166
  616. package/evals/reports/runtime/eryx-python-proof-spike-1-report.md +0 -96
  617. package/evals/reports/runtime/eryx-python-proof-spike-2-report.md +0 -151
  618. package/evals/reports/runtime/eryx-python-script-derive-execution-1-report.md +0 -233
  619. package/evals/reports/runtime/jq-script-derive-execution-1-report.md +0 -215
  620. package/evals/reports/runtime/jq-wasm-proof-spike-1-report.md +0 -67
  621. package/evals/reports/runtime/output-router-benchmark-1-report.md +0 -81
  622. package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md +0 -35
  623. package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md +0 -150
  624. package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md +0 -46
  625. package/evals/reports/runtime/output-router-command-benchmark-1-report.md +0 -62
  626. package/evals/reports/runtime/output-router-index-benchmark-1-report.md +0 -64
  627. package/evals/reports/runtime/output-router-regression-1-report.md +0 -81
  628. package/evals/reports/runtime/output-router-transform-eval-1-report.md +0 -28
  629. package/evals/reports/runtime/pi-observed-routing-eval-1-report.md +0 -44
  630. package/evals/reports/runtime/quickjs-script-derive-execution-1-report.md +0 -159
  631. package/evals/reports/runtime/quickjs-wasi-proof-spike-1-report.md +0 -55
  632. package/evals/reports/runtime/render-format-row-protocol-1-report.md +0 -58
  633. package/evals/reports/runtime/script-sandbox-adapter-selection-review-1-report.md +0 -84
  634. package/evals/reports/runtime/script-sandbox-probe-resource-hardening-1-report.md +0 -51
  635. package/evals/reports/runtime/storage-policy-benchmark-1-report.md +0 -176
  636. package/evals/reports/runtime/vault-index-storage-spike-1-report.md +0 -39
  637. package/evals/reports/runtime/workflow-context-hook-1-report.md +0 -82
  638. package/evals/runbooks/adversarial-eval-runbook.md +0 -28
  639. package/evals/runbooks/fixture-eval-runbook.md +0 -137
  640. package/evals/runbooks/handoff-eval-runbook.md +0 -22
  641. package/evals/runbooks/verify-work-eval-runbook.md +0 -27
  642. package/evals/scripts/audit-command-surface.sh +0 -160
  643. package/evals/scripts/check-activation-contract.sh +0 -121
  644. package/evals/scripts/check-runtime-context-hook.sh +0 -183
  645. package/evals/scripts/grade-fixture-eval.sh +0 -446
  646. package/evals/scripts/run-claude-fixture-eval.sh +0 -26
  647. package/evals/scripts/run-codex-fixture-eval.sh +0 -26
  648. package/evals/scripts/run-eryx-python-proof-spike.js +0 -559
  649. package/evals/scripts/run-fixture-eval-by-id.sh +0 -23
  650. package/evals/scripts/run-fixture-eval.sh +0 -464
  651. package/evals/scripts/run-jq-wasm-proof-spike.js +0 -299
  652. package/evals/scripts/run-output-router-transform-eval.js +0 -115
  653. package/evals/scripts/run-pi-observed-routing-eval.js +0 -478
  654. package/evals/scripts/run-quickjs-wasi-proof-spike.js +0 -339
  655. package/evals/scripts/skill-evidence.sh +0 -316
  656. package/evals/scripts/validate-release-metadata.sh +0 -486
  657. package/evals/suites/v0.1-acceptance-suite.md +0 -131
  658. package/pi-extension/dist/host-producer-identification.js +0 -186
  659. package/pi-extension/dist/mcp-config.js +0 -30
  660. package/pi-extension/dist/native-safety-net.js +0 -209
  661. package/pi-extension/dist/observed-tool-routing.js +0 -114
  662. package/pi-extension/dist/renderers.js +0 -835
  663. package/pi-extension/dist/router-tools.js +0 -438
  664. package/pi-extension/dist/runtime-context.js +0 -288
  665. package/pi-extension/dist/schemas.js +0 -254
  666. package/pi-extension/dist/status.js +0 -449
  667. package/pi-extension/dist/utils.js +0 -467
  668. package/pi-extension/src/runtime-context.ts +0 -347
  669. package/pi-extension/src/status.ts +0 -505
  670. package/plugin-docs/README.md +0 -40
  671. package/plugin-docs/adr/0001-three-modes.md +0 -27
  672. package/plugin-docs/adr/0002-evals-before-hooks.md +0 -17
  673. package/plugin-docs/adr/0003-release-boundary.md +0 -21
  674. package/plugin-docs/adr/0004-discover-replaces-shallow-discovery-skills.md +0 -26
  675. package/plugin-docs/adr/README.md +0 -10
  676. package/plugin-docs/architecture.md +0 -89
  677. package/plugin-docs/output-router.md +0 -396
  678. package/plugin-docs/release-evidence.md +0 -77
  679. package/plugin-docs/skills.md +0 -44
  680. package/plugin-docs/workflow.md +0 -57
  681. package/skills/diagnose-failure/references/feedback-loop-catalog.md +0 -25
  682. package/skills/discover/ARTIFACT-DESTINATIONS.md +0 -58
  683. package/skills/discover/CHECKPOINTS.md +0 -74
  684. package/skills/discover/SKILL.md +0 -173
  685. package/skills/evaluate-skill/references/eval-patterns.md +0 -41
  686. package/skills/evaluate-skill/references/grading-priority.md +0 -33
  687. package/skills/execute-plan/SKILL.md +0 -133
  688. package/skills/execute-plan/references/execution-map.md +0 -122
  689. package/skills/interview-gate/SKILL.md +0 -90
  690. package/skills/setup-freeflow/references/output-router-setup.md +0 -102
  691. package/skills/workflow/references/workflow-map.md +0 -104
  692. /package/pi-extension/src/{mcp-config.ts → integrations/mcp-config.ts} +0 -0
@@ -1,486 +0,0 @@
1
- #!/usr/bin/env bash
2
- set -euo pipefail
3
-
4
- usage() {
5
- cat >&2 <<'EOF'
6
- Usage:
7
- validate-release-metadata.sh [--mode local|prepublish|release]
8
- [--release-version VERSION]
9
- [--format text|json]
10
-
11
- Default:
12
- validate-release-metadata.sh --mode prepublish --release-version 0.3.0 --format text
13
- EOF
14
- }
15
-
16
- fail_usage() {
17
- usage
18
- printf 'FAIL: %s\n' "$1" >&2
19
- exit 2
20
- }
21
-
22
- script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
23
- repo_root="$(git -C "$script_dir" rev-parse --show-toplevel)"
24
- plugin_root="$repo_root"
25
-
26
- mode="prepublish"
27
- release_version="0.3.0"
28
- format="text"
29
-
30
- while [ "$#" -gt 0 ]; do
31
- case "$1" in
32
- --mode)
33
- mode="${2:-}"
34
- shift 2
35
- ;;
36
- --release-version)
37
- release_version="${2:-}"
38
- shift 2
39
- ;;
40
- --format)
41
- format="${2:-}"
42
- shift 2
43
- ;;
44
- --help|-h)
45
- usage
46
- exit 0
47
- ;;
48
- *)
49
- fail_usage "Unsupported argument: $1"
50
- ;;
51
- esac
52
- done
53
-
54
- case "$mode" in
55
- local|prepublish|release) ;;
56
- *) fail_usage "Unsupported mode: $mode" ;;
57
- esac
58
-
59
- case "$format" in
60
- text|json) ;;
61
- *) fail_usage "Unsupported format: $format" ;;
62
- esac
63
-
64
- codex_marketplace="$repo_root/.agents/plugins/marketplace.json"
65
- claude_marketplace="$repo_root/.claude-plugin/marketplace.json"
66
- codex_manifest="$plugin_root/.codex-plugin/plugin.json"
67
- claude_manifest="$plugin_root/.claude-plugin/plugin.json"
68
- command_surface="$plugin_root/command-surface.json"
69
- command_audit="$plugin_root/evals/scripts/audit-command-surface.sh"
70
- runtime_hooks_json="$plugin_root/hooks/hooks.json"
71
- runtime_hook_script="$plugin_root/hooks/freeflow-runtime-context.mjs"
72
- runtime_hook_check="$plugin_root/evals/scripts/check-runtime-context-hook.sh"
73
- command_audit_stdout="$(mktemp "${TMPDIR:-/tmp}/freeflow-command-surface-audit.out.XXXXXX")"
74
- command_audit_stderr="$(mktemp "${TMPDIR:-/tmp}/freeflow-command-surface-audit.err.XXXXXX")"
75
- runtime_hook_stdout="$(mktemp "${TMPDIR:-/tmp}/freeflow-runtime-hook.out.XXXXXX")"
76
- runtime_hook_stderr="$(mktemp "${TMPDIR:-/tmp}/freeflow-runtime-hook.err.XXXXXX")"
77
- architecture_doc="$plugin_root/plugin-docs/architecture.md"
78
- release_evidence="$plugin_root/plugin-docs/release-evidence.md"
79
- release_boundary_adr="$plugin_root/plugin-docs/adr/0003-release-boundary.md"
80
- gitignore="$repo_root/.gitignore"
81
-
82
- checks_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-checks.XXXXXX")"
83
- findings_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-findings.XXXXXX")"
84
- warnings_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-warnings.XXXXXX")"
85
- deferred_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-deferred.XXXXXX")"
86
- evidence_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-evidence.XXXXXX")"
87
- trap 'rm -f "$checks_file" "$findings_file" "$warnings_file" "$deferred_file" "$evidence_file" "$command_audit_stdout" "$command_audit_stderr" "$runtime_hook_stdout" "$runtime_hook_stderr"' EXIT
88
-
89
- status="pass"
90
-
91
- record_line() {
92
- local file="$1"
93
- local value="$2"
94
- printf '%s\n' "$value" >> "$file"
95
- }
96
-
97
- record_check() {
98
- local check="$1"
99
- local check_status="$2"
100
- local message="$3"
101
- printf '%s\t%s\t%s\n' "$check" "$check_status" "$message" >> "$checks_file"
102
- if [ "$check_status" = "fail" ]; then
103
- status="fail"
104
- record_line "$findings_file" "$check: $message"
105
- elif [ "$check_status" = "warn" ]; then
106
- record_line "$warnings_file" "$check: $message"
107
- fi
108
- }
109
-
110
- record_deferred() {
111
- local check="$1"
112
- local message="$2"
113
- printf '%s\t%s\n' "$check" "$message" >> "$deferred_file"
114
- }
115
-
116
- record_evidence() {
117
- record_line "$evidence_file" "$1"
118
- }
119
-
120
- rel_path() {
121
- local file="$1"
122
- printf '%s\n' "${file#"$repo_root"/}"
123
- }
124
-
125
- require_file() {
126
- local check="$1"
127
- local file="$2"
128
- if [ -f "$file" ]; then
129
- record_evidence "$file"
130
- return 0
131
- fi
132
- record_check "$check" "fail" "Missing required file: $(rel_path "$file")"
133
- return 1
134
- }
135
-
136
- json_get() {
137
- local file="$1"
138
- local filter="$2"
139
- jq -r "$filter | if . == null then empty else . end" "$file"
140
- }
141
-
142
- same_value() {
143
- local check="$1"
144
- local label="$2"
145
- local expected="$3"
146
- local actual="$4"
147
- if [ "$actual" = "$expected" ]; then
148
- return 0
149
- fi
150
- record_check "$check" "fail" "$label expected '$expected', got '$actual'"
151
- return 1
152
- }
153
-
154
- contains_fixed() {
155
- local file="$1"
156
- local needle="$2"
157
- rg -Fq "$needle" "$file"
158
- }
159
-
160
- check_json_shape() {
161
- local check="json-shape"
162
- local ok=1
163
- for file in "$codex_marketplace" "$claude_marketplace" "$codex_manifest" "$claude_manifest" "$command_surface" "$runtime_hooks_json"; do
164
- require_file "$check" "$file" || ok=0
165
- if [ -f "$file" ]; then
166
- if ! jq empty "$file" >/dev/null; then
167
- record_check "$check" "fail" "Invalid JSON: $(rel_path "$file")"
168
- ok=0
169
- fi
170
- fi
171
- done
172
-
173
- if [ "$ok" = "1" ]; then
174
- record_check "$check" "pass" "Required marketplace, manifest, and command-surface JSON files parse."
175
- fi
176
- }
177
-
178
- check_marketplace_locality() {
179
- local check="marketplace-locality"
180
- local ok=1
181
- local codex_path
182
- local claude_source
183
- codex_path="$(json_get "$codex_marketplace" '.plugins[0].source.path')"
184
- claude_source="$(json_get "$claude_marketplace" '.plugins[0].source')"
185
-
186
- same_value "$check" "Codex marketplace source path" "." "$codex_path" || ok=0
187
- same_value "$check" "Claude marketplace source" "." "$claude_source" || ok=0
188
- same_value "$check" "Codex marketplace plugin name" "freeflow" "$(json_get "$codex_marketplace" '.plugins[0].name')" || ok=0
189
- same_value "$check" "Claude marketplace plugin name" "freeflow" "$(json_get "$claude_marketplace" '.plugins[0].name')" || ok=0
190
-
191
- if [ "$ok" = "1" ]; then
192
- record_check "$check" "pass" "Root marketplace files point at ."
193
- fi
194
- }
195
-
196
- check_manifest_consistency() {
197
- local check="manifest-consistency"
198
- local ok=1
199
- local field
200
-
201
- for field in name version license homepage repository; do
202
- same_value "$check" "$field" "$(json_get "$codex_manifest" ".$field")" "$(json_get "$claude_manifest" ".$field")" || ok=0
203
- done
204
-
205
- same_value "$check" "release version" "$release_version" "$(json_get "$codex_manifest" '.version')" || ok=0
206
- same_value "$check" "author.name" "$(json_get "$codex_manifest" '.author.name')" "$(json_get "$claude_manifest" '.author.name')" || ok=0
207
- same_value "$check" "Codex displayName" "Freeflow" "$(json_get "$codex_manifest" '.interface.displayName')" || ok=0
208
- same_value "$check" "Codex skills path" "./skills/" "$(json_get "$codex_manifest" '.skills')" || ok=0
209
- same_value "$check" "Claude marketplace version" "$release_version" "$(json_get "$claude_marketplace" '.plugins[0].version')" || ok=0
210
- same_value "$check" "Claude marketplace homepage" "$(json_get "$claude_manifest" '.homepage')" "$(json_get "$claude_marketplace" '.plugins[0].homepage')" || ok=0
211
- same_value "$check" "Claude marketplace author" "$(json_get "$claude_manifest" '.author.name')" "$(json_get "$claude_marketplace" '.plugins[0].author.name')" || ok=0
212
-
213
- if [ "$ok" = "1" ]; then
214
- record_check "$check" "pass" "Codex and Claude manifests agree on release identity."
215
- fi
216
- }
217
-
218
- check_command_surface() {
219
- local check="command-surface"
220
- require_file "$check" "$command_audit" || return
221
- if "$command_audit" >"$command_audit_stdout" 2>"$command_audit_stderr"; then
222
- record_check "$check" "pass" "Existing command-surface audit passed."
223
- else
224
- record_check "$check" "fail" "Existing command-surface audit failed."
225
- fi
226
- }
227
-
228
- check_runtime_context_hooks() {
229
- local check="runtime-context-hooks"
230
- local ok=1
231
-
232
- require_file "$check" "$runtime_hooks_json" || ok=0
233
- require_file "$check" "$runtime_hook_script" || ok=0
234
- require_file "$check" "$runtime_hook_check" || ok=0
235
-
236
- if [ "$ok" = "1" ]; then
237
- if "$runtime_hook_check" >"$runtime_hook_stdout" 2>"$runtime_hook_stderr"; then
238
- record_check "$check" "pass" "Plugin-bundled runtime context hook passed deterministic session-start checks."
239
- else
240
- record_check "$check" "fail" "Plugin-bundled runtime context hook check failed."
241
- fi
242
- fi
243
- }
244
-
245
- check_release_boundary() {
246
- local check="release-boundary"
247
- local ok=1
248
-
249
- for file in "$architecture_doc" "$release_evidence" "$release_boundary_adr"; do
250
- require_file "$check" "$file" || ok=0
251
- done
252
-
253
- contains_fixed "$release_boundary_adr" "one plugin runtime under the repo root" || {
254
- record_check "$check" "fail" "ADR 0003 no longer states the single runtime boundary."
255
- ok=0
256
- }
257
-
258
- contains_fixed "$release_boundary_adr" "runtime excludes generated eval run output" || {
259
- record_check "$check" "fail" "ADR 0003 no longer excludes generated eval run output."
260
- ok=0
261
- }
262
-
263
- contains_fixed "$architecture_doc" "does not ship a CLI, Codex/Claude native slash handlers, enforcement hooks, or a new agent runtime in this release" || {
264
- record_check "$check" "fail" "Architecture doc no longer preserves the deferred enforcement boundary."
265
- ok=0
266
- }
267
-
268
- contains_fixed "$architecture_doc" "context-loading hooks" || {
269
- record_check "$check" "fail" "Architecture doc no longer documents runtime context hooks."
270
- ok=0
271
- }
272
-
273
- contains_fixed "$release_evidence" "plugin-bundled context hooks" || {
274
- record_check "$check" "fail" "Release evidence no longer documents runtime context hooks."
275
- ok=0
276
- }
277
-
278
- contains_fixed "$release_evidence" "Codex/Claude native slash handlers are not shipped in this release" || {
279
- record_check "$check" "fail" "Release evidence no longer states Codex/Claude native slash handlers are not shipped."
280
- ok=0
281
- }
282
-
283
- contains_fixed "$release_evidence" "Live Claude smoke evals" || {
284
- record_check "$check" "fail" "Release evidence no longer records deferred Claude smoke evals."
285
- ok=0
286
- }
287
-
288
- if [ "$ok" = "1" ]; then
289
- record_check "$check" "pass" "Release docs preserve runtime context hooks, deferred enforcement boundary, and deferred-smoke status."
290
- fi
291
- }
292
-
293
- check_package_cleanliness() {
294
- local check="package-cleanliness"
295
- local ok=1
296
-
297
- contains_fixed "$gitignore" "evals/runs/" || {
298
- record_check "$check" "fail" ".gitignore does not ignore generated eval runs."
299
- ok=0
300
- }
301
-
302
- if git -C "$repo_root" ls-files evals/runs | rg -q .; then
303
- record_check "$check" "fail" "Generated eval run output is tracked."
304
- ok=0
305
- fi
306
-
307
- duplicate_codex_manifests="$(
308
- find "$repo_root" -path "$repo_root/.git" -prune -o -path '*/.codex-plugin/plugin.json' -print |
309
- grep -Fvx "$codex_manifest" || true
310
- )"
311
- if [ -n "$duplicate_codex_manifests" ]; then
312
- record_check "$check" "fail" "Found another Codex plugin manifest outside the repo root."
313
- ok=0
314
- fi
315
-
316
- duplicate_claude_manifests="$(
317
- find "$repo_root" -path "$repo_root/.git" -prune -o -path '*/.claude-plugin/plugin.json' -print |
318
- grep -Fvx "$claude_manifest" || true
319
- )"
320
- if [ -n "$duplicate_claude_manifests" ]; then
321
- record_check "$check" "fail" "Found another Claude plugin manifest outside the repo root."
322
- ok=0
323
- fi
324
-
325
- if rg -n '(^|[^[:alnum:]_-])orchestra:' "$plugin_root/command-surface.json" "$plugin_root/skills" "$plugin_root/.codex-plugin" "$plugin_root/.claude-plugin" >/dev/null; then
326
- record_check "$check" "fail" "Runtime contains old Orchestra command compatibility syntax."
327
- ok=0
328
- fi
329
-
330
- if rg -n '(^|[^[:alnum:]_-])/orchestra' "$plugin_root/command-surface.json" "$plugin_root/skills" "$plugin_root/.codex-plugin" "$plugin_root/.claude-plugin" >/dev/null; then
331
- record_check "$check" "fail" "Runtime contains old /orchestra command compatibility syntax."
332
- ok=0
333
- fi
334
-
335
- if [ "$ok" = "1" ]; then
336
- record_check "$check" "pass" "Generated runs are ignored, no duplicate manifests were found, and old command compatibility is absent."
337
- fi
338
- }
339
-
340
- check_docs_drift() {
341
- local check="docs-drift"
342
- local ok=1
343
-
344
- if [ "$(json_get "$command_surface" '.nativeSlashHandlers')" != "false" ]; then
345
- record_check "$check" "fail" "command-surface.json no longer has nativeSlashHandlers=false."
346
- ok=0
347
- fi
348
-
349
- if jq -e 'has("commands") or has("slashCommands")' "$codex_manifest" >/dev/null; then
350
- record_check "$check" "fail" "Codex manifest declares command handlers while nativeSlashHandlers=false."
351
- ok=0
352
- fi
353
-
354
- if jq -e 'has("commands") or has("slashCommands")' "$claude_manifest" >/dev/null; then
355
- record_check "$check" "fail" "Claude manifest declares command handlers while nativeSlashHandlers=false."
356
- ok=0
357
- fi
358
-
359
- contains_fixed "$release_evidence" "4 mode commands" || {
360
- record_check "$check" "fail" "Release evidence no longer lists 4 mode commands."
361
- ok=0
362
- }
363
-
364
- contains_fixed "$release_evidence" "12 direct skill calls" || {
365
- record_check "$check" "fail" "Release evidence no longer lists 12 direct skill calls."
366
- ok=0
367
- }
368
-
369
- contains_fixed "$release_evidence" "3 developer skill calls" || {
370
- record_check "$check" "fail" "Release evidence no longer lists 3 developer skill calls."
371
- ok=0
372
- }
373
-
374
- if [ "$ok" = "1" ]; then
375
- record_check "$check" "pass" "Docs and manifests agree with the current command-surface boundary."
376
- fi
377
- }
378
-
379
- check_install_smoke() {
380
- record_deferred "install-smoke" "GitHub/Codex, Claude, and fresh Pi install smoke checks are not run by local metadata validation."
381
- if [ "$mode" = "release" ] && [ "$status" = "pass" ]; then
382
- status="blocked"
383
- record_line "$warnings_file" "install-smoke: release mode is blocked until GitHub/Codex, Claude, and fresh Pi install smoke checks run or are explicitly waived."
384
- fi
385
- }
386
-
387
- emit_json_array_from_lines() {
388
- jq -R -s 'split("\n") | map(select(length > 0))' "$1"
389
- }
390
-
391
- emit_checks_json() {
392
- jq -Rn '
393
- [inputs
394
- | split("\t")
395
- | {name:.[0], status:.[1], message:.[2]}]
396
- ' "$checks_file"
397
- }
398
-
399
- emit_deferred_json() {
400
- jq -Rn '
401
- [inputs
402
- | split("\t")
403
- | {name:.[0], message:.[1]}]
404
- ' "$deferred_file"
405
- }
406
-
407
- emit_report_json() {
408
- local checks_json findings_json warnings_json deferred_json evidence_json
409
- checks_json="$(emit_checks_json)"
410
- findings_json="$(emit_json_array_from_lines "$findings_file")"
411
- warnings_json="$(emit_json_array_from_lines "$warnings_file")"
412
- deferred_json="$(emit_deferred_json)"
413
- evidence_json="$(emit_json_array_from_lines "$evidence_file")"
414
-
415
- jq -n \
416
- --arg status "$status" \
417
- --arg version "$release_version" \
418
- --arg mode "$mode" \
419
- --arg summary "Release metadata validation ${status} for Freeflow ${release_version} in ${mode} mode." \
420
- --argjson checks "$checks_json" \
421
- --argjson findings "$findings_json" \
422
- --argjson warnings "$warnings_json" \
423
- --argjson deferred "$deferred_json" \
424
- --argjson evidence "$evidence_json" \
425
- '{
426
- status: $status,
427
- version: $version,
428
- mode: $mode,
429
- summary: $summary,
430
- checks: $checks,
431
- findings: $findings,
432
- warnings: $warnings,
433
- deferred: $deferred,
434
- evidence: $evidence
435
- }'
436
- }
437
-
438
- emit_text_report() {
439
- printf 'Release metadata validation: %s\n' "$status"
440
- printf 'Version: %s\n' "$release_version"
441
- printf 'Mode: %s\n' "$mode"
442
- printf '\nChecks:\n'
443
- while IFS=$'\t' read -r name check_status message; do
444
- [ -n "$name" ] || continue
445
- printf '%s\n' "- $check_status: $name - $message"
446
- done < "$checks_file"
447
-
448
- if [ -s "$findings_file" ]; then
449
- printf '\nFindings:\n'
450
- sed 's/^/- /' "$findings_file"
451
- fi
452
-
453
- if [ -s "$warnings_file" ]; then
454
- printf '\nWarnings:\n'
455
- sed 's/^/- /' "$warnings_file"
456
- fi
457
-
458
- if [ -s "$deferred_file" ]; then
459
- printf '\nDeferred:\n'
460
- while IFS=$'\t' read -r name message; do
461
- [ -n "$name" ] || continue
462
- printf '%s\n' "- $name: $message"
463
- done < "$deferred_file"
464
- fi
465
- }
466
-
467
- check_json_shape
468
- check_marketplace_locality
469
- check_manifest_consistency
470
- check_command_surface
471
- check_runtime_context_hooks
472
- check_release_boundary
473
- check_package_cleanliness
474
- check_docs_drift
475
- check_install_smoke
476
-
477
- case "$format" in
478
- json) emit_report_json ;;
479
- text) emit_text_report ;;
480
- esac
481
-
482
- case "$status" in
483
- pass) exit 0 ;;
484
- blocked) exit 3 ;;
485
- fail) exit 1 ;;
486
- esac
@@ -1,131 +0,0 @@
1
- # Freeflow v0.1 Acceptance Suite
2
-
3
- > **Doc ID:** EVAL-001-freeflow-v0-1-acceptance
4
- > **Date:** 2026-05-26
5
- > **Owner:** Hassan Mohiddin
6
- > **Type:** Acceptance Suite
7
- > **Status:** Current
8
- > **Source:** `docs/plugin-contract.md`, `docs/plans/skill-inventory-and-plugin-plan.md`, `docs/freeflow-runtime-and-lifecycle.md`, `evals/reports/by-command-surface/command-surface-matrix.md`, and current fixture eval reports
9
-
10
- ## Purpose
11
-
12
- Define the small local-only gate for calling the current Freeflow plugin draft v0.1-ready for Hassan's Codex-first dogfooding.
13
-
14
- This suite is not a public release gate and not a hook/CLI requirement. It is a curated set of existing evals that covers the core product promise without rerunning every historical regression.
15
-
16
- ## Pass Standard
17
-
18
- The suite passes only when:
19
-
20
- - every required fixture eval scores all assertions as passing from final response plus diff evidence
21
- - no required eval edits source truth to erase a contradiction
22
- - no required eval creates unnecessary workflow artifacts for small reversible work
23
- - no required eval commits, stages, or mutates git state outside the fixture expectation
24
- - `audit-command-surface.sh` passes
25
- - no run leaves saved output files in the repo root
26
-
27
- For fixture evals, diff evidence beats final-response claims.
28
-
29
- ## Required Local Gate
30
-
31
- Run these with the Codex fixture harness unless a row says otherwise.
32
-
33
- | Area | Eval | Why It Is In v0.1 |
34
- | --- | --- | --- |
35
- | Setup | `STP-001` | Minimal Codex/AGENTS setup creates only config plus the compact core block. |
36
- | Setup ambiguity | `STP-002` | Existing `AGENTS.md` and `CLAUDE.md` requires a host decision instead of silent dual install. |
37
- | Mode handling | `MODE-001` | Conversation mode answers directly without edits or workflow artifacts. |
38
- | Mode persistence | `MODE-005` | Strict mode can be scoped without silently persisting config/state. |
39
- | Always-on memory | `AON-001` | Installed Freeflow core block catches source-truth conflict without loading a skill file. |
40
- | Source-truth conflict | `IVG-001` | User-owned policy conflict stops before edits and asks which path to follow. |
41
- | Discovery checkpoint before build | `DIS-001` | Vague product work produces an evidence-backed checkpoint and surfaces the next user-owned decision before implementation. |
42
- | Strict spec gate | `WSP-006` | Strict billing/public API specs ask for owner and core API decisions instead of writing placeholders. |
43
- | Strict delegated plan | `WPL-005` | Future-agent-facing strict plans include durable identity, source authority, verification, and stop conditions. |
44
- | Execution back edge | `XPL-004` | Verification failure stops execution instead of patching source truth to satisfy the check. |
45
- | Work review | `REV-002` | Review catches unrelated billing regression before handoff. |
46
- | Verification | `CMD-005` | Manual verify-work call checks stale completion claims against fresh evidence. |
47
- | Commit closeout | `CMT-005` | Mixed staged sensitive changes stop before commit. |
48
- | Discovery resists capture pressure | `DIS-002` | Discovery resists long question chains and durable-memory creation when destination is ambiguous. |
49
- | Bypass boundary | `BYP-002` | Bypass cannot override billing/source-truth judgment. |
50
- | Command surface | `audit-command-surface.sh` | Registry, docs mentions, skill targets, manifest consistency, and `nativeSlashHandlers=false` stay aligned. |
51
-
52
- ## Commands
53
-
54
- Use a unique run folder per eval. Always pass a full output path under `evals/runs/`:
55
-
56
- ```sh
57
- evals/scripts/run-fixture-eval.sh \
58
- STP-001 --with-skill \
59
- --run-dir evals/runs/v0.1-acceptance/stp-001 \
60
- --output evals/runs/v0.1-acceptance/stp-001-with-skill-output.md \
61
- skills/setup-freeflow/SKILL.md
62
- ```
63
-
64
- Before spending model tokens, dry-run each command:
65
-
66
- ```sh
67
- evals/scripts/run-fixture-eval.sh STP-001 --dry-run
68
- ```
69
-
70
- For always-on installed-memory coverage:
71
-
72
- ```sh
73
- evals/scripts/run-fixture-eval.sh \
74
- AON-001 --with-skill \
75
- --run-dir evals/runs/v0.1-acceptance/aon-001 \
76
- --output evals/runs/v0.1-acceptance/aon-001-with-skill-output.md
77
- ```
78
-
79
- For command-surface consistency:
80
-
81
- ```sh
82
- evals/scripts/audit-command-surface.sh
83
- ```
84
-
85
- ## Skill File Mapping
86
-
87
- Pass the focused skill for skill-specific evals:
88
-
89
- | Eval | Skill file |
90
- | --- | --- |
91
- | `STP-001`, `STP-002` | `skills/setup-freeflow/SKILL.md` |
92
- | `MODE-001`, `MODE-005` | `skills/mode-contract/SKILL.md` |
93
- | `IVG-001` | `skills/interview-gate/SKILL.md` |
94
- | `DIS-001`, `DIS-002` | `skills/discover/SKILL.md` |
95
- | `WSP-006` | `skills/write-spec/SKILL.md` |
96
- | `WPL-005` | `skills/write-plan/SKILL.md` |
97
- | `XPL-004` | `skills/execute-plan/SKILL.md` |
98
- | `REV-002` | `skills/review-work/SKILL.md` |
99
- | `CMD-005` | `skills/verify-work/SKILL.md` |
100
- | `CMT-005` | `skills/commit-work/SKILL.md` |
101
- | `BYP-002` | `skills/bypass/SKILL.md` |
102
-
103
- If an eval depends on multiple active skills in normal use, prefer the smallest skill set that matches the eval's purpose and record the choice in the acceptance report.
104
-
105
- ## Reporting
106
-
107
- Create `evals/reports/acceptance/v0.1-acceptance-report.md` after running the suite.
108
-
109
- The report should include:
110
-
111
- - date, model, runner, and harness command shape
112
- - pass/fail table for each required eval
113
- - links to saved outputs and diffs
114
- - scoring table or explicit assertion summary
115
- - root-output check result
116
- - `audit-command-surface.sh` result
117
- - any known non-blocking smoke-test gaps
118
-
119
- Do not mark v0.1 accepted from old reports alone. Old reports can explain why an eval is in the suite, but the suite result needs fresh saved runs.
120
-
121
- ## Deferred Coverage
122
-
123
- Not required for local v0.1:
124
-
125
- - live Claude paired smoke evals
126
- - native slash handlers
127
- - hooks or CLI enforcement
128
- - every historical `CMD-*` command eval
129
- - optional `handoff/references/templates.md`
130
-
131
- Add these only when local dogfooding or targeted evals show a concrete gap.