@hassangameryt/freeflow 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +4 -4
- package/.claude-plugin/plugin.json +6 -2
- package/.codex-plugin/plugin.json +9 -9
- package/README.md +190 -127
- package/assets/feedback-loop.png +0 -0
- package/assets/freeflow_workflow_map.png +0 -0
- package/assets/workflow.png +0 -0
- package/{skills → capabilities}/output-router/SKILL.md +21 -17
- package/{skills → capabilities}/output-router/references/safety-policy.md +1 -1
- package/capabilities/output-router/references/setup.md +110 -0
- package/command-surface.json +62 -7
- package/hooks/freeflow-runtime-context.mjs +572 -154
- package/hooks/hooks.json +26 -1
- package/hooks/tests/check-runtime-context-hook.sh +375 -0
- package/package.json +24 -24
- package/pi-extension/dist/index.js +264 -80
- package/pi-extension/dist/integrations/mcp-config.js +29 -0
- package/pi-extension/dist/output-router/host-producer-identification.js +232 -0
- package/pi-extension/dist/output-router/native-safety-net.js +216 -0
- package/pi-extension/dist/output-router/observed-tool-routing.js +122 -0
- package/pi-extension/dist/output-router/renderers.js +1023 -0
- package/pi-extension/dist/output-router/router-tools.js +554 -0
- package/pi-extension/dist/output-router/schemas.js +383 -0
- package/pi-extension/dist/output-router/status.js +744 -0
- package/pi-extension/dist/output-router/utils.js +502 -0
- package/pi-extension/dist/runtime/runtime-context.js +798 -0
- package/pi-extension/dist/settings/settings-tui.js +314 -0
- package/pi-extension/dist/settings/settings-ui.js +1487 -0
- package/pi-extension/freeflow/index.js +1 -0
- package/pi-extension/src/index.ts +230 -56
- package/pi-extension/src/{host-producer-identification.ts → output-router/host-producer-identification.ts} +37 -7
- package/pi-extension/src/{native-safety-net.ts → output-router/native-safety-net.ts} +8 -12
- package/pi-extension/src/{observed-tool-routing.ts → output-router/observed-tool-routing.ts} +19 -11
- package/pi-extension/src/{renderers.ts → output-router/renderers.ts} +246 -55
- package/pi-extension/src/{router-tools.ts → output-router/router-tools.ts} +130 -29
- package/pi-extension/src/{schemas.ts → output-router/schemas.ts} +171 -43
- package/pi-extension/src/output-router/status.ts +791 -0
- package/pi-extension/src/{utils.ts → output-router/utils.ts} +80 -18
- package/pi-extension/src/runtime/runtime-context.ts +895 -0
- package/pi-extension/src/settings/settings-tui.ts +408 -0
- package/pi-extension/src/settings/settings-ui.ts +1730 -0
- package/pi-extension/tests/integration/pi-extension.test.js +4020 -0
- package/pi-extension/tests/output-router/observed-producers.test.js +132 -0
- package/pi-extension/tests/output-router/search.test.js +722 -0
- package/pi-extension/tests/settings/settings-tui.test.js +243 -0
- package/router/dist/benchmarks/benchmark-harness.d.ts +16 -13
- package/router/dist/benchmarks/benchmark-harness.js +62 -62
- package/router/dist/benchmarks/benchmarks.d.ts +80 -73
- package/router/dist/benchmarks/benchmarks.js +860 -761
- package/router/dist/benchmarks/codex-qa-benchmarks.d.ts +71 -67
- package/router/dist/benchmarks/codex-qa-benchmarks.js +429 -390
- package/router/dist/benchmarks/command-benchmarks.d.ts +107 -92
- package/router/dist/benchmarks/command-benchmarks.js +616 -571
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.d.ts +64 -56
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.js +679 -613
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.d.ts +103 -95
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.js +1188 -1103
- package/router/dist/benchmarks/context-saving-baseline.d.ts +47 -41
- package/router/dist/benchmarks/context-saving-baseline.js +405 -374
- package/router/dist/benchmarks/index-benchmarks.d.ts +74 -69
- package/router/dist/benchmarks/index-benchmarks.js +761 -661
- package/router/dist/benchmarks/storage-policy-benchmarks.d.ts +80 -69
- package/router/dist/benchmarks/storage-policy-benchmarks.js +549 -469
- package/router/dist/benchmarks/vault-index-benchmarks.d.ts +43 -34
- package/router/dist/benchmarks/vault-index-benchmarks.js +294 -229
- package/router/dist/config/config.d.ts +62 -55
- package/router/dist/config/config.js +552 -373
- package/router/dist/config/router-contract.d.ts +2 -2
- package/router/dist/config/router-contract.js +38 -38
- package/router/dist/config/schema.d.ts +19 -10
- package/router/dist/config/schema.js +586 -538
- package/router/dist/config/types.d.ts +413 -361
- package/router/dist/config/types.js +19 -8
- package/router/dist/evidence/bounded-evidence.d.ts +26 -26
- package/router/dist/evidence/bounded-evidence.js +265 -252
- package/router/dist/evidence/evidence-range-selector.d.ts +16 -14
- package/router/dist/evidence/evidence-range-selector.js +192 -191
- package/router/dist/evidence/evidence-search.d.ts +18 -16
- package/router/dist/evidence/evidence-search.js +401 -377
- package/router/dist/evidence/evidence.d.ts +17 -17
- package/router/dist/evidence/evidence.js +163 -159
- package/router/dist/evidence/failure-contracts.d.ts +25 -15
- package/router/dist/evidence/failure-contracts.js +132 -126
- package/router/dist/evidence/line-ranges.d.ts +17 -15
- package/router/dist/evidence/line-ranges.js +18 -18
- package/router/dist/experiments/local-index.d.ts +47 -41
- package/router/dist/experiments/local-index.js +477 -458
- package/router/dist/local/local-traversal.d.ts +18 -0
- package/router/dist/local/local-traversal.js +334 -0
- package/router/dist/processing/engine.d.ts +94 -70
- package/router/dist/processing/engine.js +462 -364
- package/router/dist/processing/reducers.d.ts +208 -194
- package/router/dist/processing/reducers.js +1434 -1256
- package/router/dist/processing/renderers.d.ts +24 -18
- package/router/dist/processing/renderers.js +161 -152
- package/router/dist/processing/scripts.d.ts +80 -67
- package/router/dist/processing/scripts.js +343 -290
- package/router/dist/repo/repo-traversal.d.ts +9 -9
- package/router/dist/repo/repo-traversal.js +205 -202
- package/router/dist/routing/observed-reducers.d.ts +25 -19
- package/router/dist/routing/observed-reducers.js +559 -545
- package/router/dist/routing/observed-routing.d.ts +23 -14
- package/router/dist/routing/observed-routing.js +319 -318
- package/router/dist/routing/parsers.d.ts +10 -10
- package/router/dist/routing/parsers.js +454 -406
- package/router/dist/routing/run-filters.d.ts +45 -41
- package/router/dist/routing/run-filters.js +215 -207
- package/router/dist/routing/run-reducers.d.ts +45 -26
- package/router/dist/routing/run-reducers.js +102 -90
- package/router/dist/sandbox/adapter-roots.d.ts +34 -26
- package/router/dist/sandbox/adapter-roots.js +43 -41
- package/router/dist/sandbox/eryx-python-adapter.d.ts +9 -5
- package/router/dist/sandbox/eryx-python-adapter.js +659 -588
- package/router/dist/sandbox/jq-wasm-adapter.d.ts +3 -3
- package/router/dist/sandbox/jq-wasm-adapter.js +319 -311
- package/router/dist/sandbox/quickjs-wasi-adapter.d.ts +9 -5
- package/router/dist/sandbox/quickjs-wasi-adapter.js +336 -333
- package/router/dist/sandbox/script-sandbox.d.ts +105 -84
- package/router/dist/sandbox/script-sandbox.js +294 -260
- package/router/dist/setup/script-transform-adapters.d.ts +23 -19
- package/router/dist/setup/script-transform-adapters.js +223 -204
- package/router/dist/tools/batch.d.ts +27 -16
- package/router/dist/tools/batch.js +493 -416
- package/router/dist/tools/run.d.ts +47 -35
- package/router/dist/tools/run.js +1172 -1087
- package/router/dist/tools/search.d.ts +53 -31
- package/router/dist/tools/search.js +1224 -1065
- package/router/dist/transform/engine.d.ts +121 -96
- package/router/dist/transform/engine.js +1853 -1736
- package/router/dist/vault/vault-index.d.ts +97 -78
- package/router/dist/vault/vault-index.js +347 -342
- package/router/dist/vault/vault.d.ts +109 -74
- package/router/dist/vault/vault.js +528 -533
- package/runtime/interaction-contract.md +16 -0
- package/skills/bypass/SKILL.md +52 -38
- package/skills/commit-work/SKILL.md +54 -46
- package/skills/commit-work/references/staging-decisions.md +46 -24
- package/skills/decision-gate/SKILL.md +48 -0
- package/skills/design-for-depth/SKILL.md +75 -59
- package/skills/design-for-depth/references/design-pressure-signals.md +56 -7
- package/skills/design-for-depth/references/interface-design-loop.md +156 -0
- package/skills/design-for-depth/references/software-design-philosophy.md +32 -3
- package/skills/diagnose-failure/SKILL.md +65 -60
- package/skills/diagnose-failure/references/diagnostic-loop-catalog.md +36 -0
- package/skills/diagnose-failure/references/flaky-and-performance.md +30 -18
- package/skills/discuss/SKILL.md +121 -0
- package/skills/discuss/references/checkpoints.md +66 -0
- package/skills/evaluate-skill/SKILL.md +62 -53
- package/skills/evaluate-skill/references/definition-schema.md +345 -0
- package/skills/evaluate-skill/references/evaluation-design.md +86 -0
- package/skills/evaluate-skill/references/execution-and-evidence.md +85 -0
- package/skills/evaluate-skill/references/review-and-revision.md +51 -0
- package/skills/evaluate-skill/scripts/lib/definitions.mjs +296 -0
- package/skills/evaluate-skill/scripts/lib/evidence.mjs +46 -0
- package/skills/evaluate-skill/scripts/lib/grade-comparison.mjs +60 -0
- package/skills/evaluate-skill/scripts/lib/grade-interaction.mjs +148 -0
- package/skills/evaluate-skill/scripts/lib/grade-support.mjs +126 -0
- package/skills/evaluate-skill/scripts/lib/grade-workspace.mjs +192 -0
- package/skills/evaluate-skill/scripts/lib/grade.mjs +56 -0
- package/skills/evaluate-skill/scripts/lib/pi.mjs +489 -0
- package/skills/evaluate-skill/scripts/lib/process.mjs +79 -0
- package/skills/evaluate-skill/scripts/lib/rpc.mjs +444 -0
- package/skills/evaluate-skill/scripts/lib/runner.mjs +346 -0
- package/skills/evaluate-skill/scripts/lib/sandbox.mjs +409 -0
- package/skills/evaluate-skill/scripts/lib/view.mjs +337 -0
- package/skills/evaluate-skill/scripts/pi-guard.mjs +164 -0
- package/skills/evaluate-skill/scripts/skill-eval.mjs +101 -0
- package/skills/execute-work/SKILL.md +99 -0
- package/skills/execute-work/references/code-practices.md +78 -0
- package/skills/execute-work/references/execution-loop.md +77 -0
- package/skills/finish-branch/SKILL.md +117 -0
- package/skills/finish-branch/references/integration-options.md +80 -0
- package/skills/handoff/SKILL.md +65 -39
- package/skills/handoff/references/templates.md +104 -27
- package/skills/launch-work/SKILL.md +116 -0
- package/skills/launch-work/references/launch-readiness.md +85 -0
- package/skills/migration-work/SKILL.md +115 -0
- package/skills/migration-work/references/migration-lifecycle.md +82 -0
- package/skills/mode-contract/SKILL.md +72 -52
- package/skills/release-work/SKILL.md +118 -0
- package/skills/release-work/references/release-evidence.md +78 -0
- package/skills/review-artifact/SKILL.md +77 -69
- package/skills/review-artifact/references/reviewer-prompt.md +133 -42
- package/skills/review-work/SKILL.md +82 -57
- package/skills/review-work/references/reviewer-prompt.md +97 -60
- package/skills/review-work/references/security-risk-lens.md +67 -0
- package/skills/setup-freeflow/SKILL.md +54 -68
- package/skills/setup-freeflow/references/activation-contract.md +84 -69
- package/skills/setup-freeflow/references/host-setup.md +50 -17
- package/skills/simplify-code/SKILL.md +80 -0
- package/skills/simplify-code/references/simplification-patterns.md +70 -0
- package/skills/tdd/SKILL.md +96 -0
- package/skills/tdd/references/test-design.md +96 -0
- package/skills/track-work/SKILL.md +84 -0
- package/skills/track-work/references/working-record-schema.md +378 -0
- package/skills/verify-work/SKILL.md +59 -38
- package/skills/verify-work/references/browser-runtime-evidence.md +55 -0
- package/skills/verify-work/references/integration-evidence.md +51 -0
- package/skills/verify-work/references/performance-evidence.md +57 -0
- package/skills/workflow/SKILL.md +53 -59
- package/skills/workflow/references/domain-skill-composition.md +57 -0
- package/skills/workflow/references/workflow-loop.md +92 -0
- package/skills/write-plan/SKILL.md +85 -58
- package/skills/write-plan/references/plan-shapes.md +132 -46
- package/skills/write-skill/SKILL.md +82 -51
- package/skills/write-skill/references/activation-boundaries.md +62 -0
- package/skills/write-skill/references/agent-first-instructions.md +87 -0
- package/skills/write-skill/references/progressive-disclosure.md +63 -0
- package/skills/write-skill/scripts/lib/skill-author.mjs +653 -0
- package/skills/write-skill/scripts/skill-author.mjs +104 -0
- package/skills/write-spec/SKILL.md +65 -54
- package/skills/write-spec/references/artifact-standards.md +60 -41
- package/skills/write-spec/references/decision-records.md +77 -0
- package/skills/write-spec/references/spec-shapes.md +119 -60
- package/evals/README.md +0 -32
- package/evals/fixtures/copy-conventions.md +0 -31
- package/evals/fixtures/output-router/README.md +0 -12
- package/evals/fixtures/output-router/failed-command-output.txt +0 -9
- package/evals/fixtures/output-router/large-router-manual.md +0 -63
- package/evals/fixtures/output-router/native-large-output.txt +0 -40
- package/evals/fixtures/output-router/noisy-test-output.txt +0 -30
- package/evals/fixtures/output-router/verification-output.txt +0 -8
- package/evals/fixtures/tiny-artifact-review-app/README.md +0 -5
- package/evals/fixtures/tiny-artifact-review-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md +0 -21
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/failed-payment-grace-period.md +0 -24
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/immediate-downgrade.md +0 -17
- package/evals/fixtures/tiny-artifact-review-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-commit-ambiguous-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-ambiguous-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-ambiguous-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-ambiguous-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-ambiguous-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh +0 -14
- package/evals/fixtures/tiny-commit-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-everything-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-everything-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-everything-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-everything-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-everything-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-everything-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-staged-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-staged-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-staged-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-staged-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-staged-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-staged-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-staged-mixed-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/billing-policy.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/release-notes.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-dashboard-stale-app/README.md +0 -11
- package/evals/fixtures/tiny-dashboard-stale-app/src/api-client.ts +0 -13
- package/evals/fixtures/tiny-dashboard-stale-app/src/dashboard-data.ts +0 -19
- package/evals/fixtures/tiny-dashboard-stale-app/tests/dashboard-data.test.md +0 -9
- package/evals/fixtures/tiny-decision-destination-app/README.md +0 -6
- package/evals/fixtures/tiny-decision-destination-app/src/risk.ts +0 -8
- package/evals/fixtures/tiny-decision-destination-app/tests/risk.test.md +0 -4
- package/evals/fixtures/tiny-design-depth-app/README.md +0 -15
- package/evals/fixtures/tiny-design-depth-app/src/routes.ts +0 -25
- package/evals/fixtures/tiny-design-depth-app/src/sendEmail.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/src/sendSms.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/src/telemetry.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/tests/routes.test.md +0 -12
- package/evals/fixtures/tiny-eval-method-app/README.md +0 -8
- package/evals/fixtures/tiny-eval-method-app/evals/reports/review-pr-eval-result.md +0 -5
- package/evals/fixtures/tiny-eval-method-app/evals/reports/triage-question-failure.md +0 -17
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.diff +0 -11
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.md +0 -3
- package/evals/fixtures/tiny-eval-method-app/skills/review-pr/SKILL.md +0 -10
- package/evals/fixtures/tiny-eval-method-app/skills/triage-question/SKILL.md +0 -10
- package/evals/fixtures/tiny-execute-plan-app/README.md +0 -5
- package/evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh +0 -5
- package/evals/fixtures/tiny-execute-plan-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/immediate-downgrade-plan.md +0 -11
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md +0 -15
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/unverified-billing-plan.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/failed-payment-grace-period.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-execute-plan-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-execute-plan-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-execute-plan-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-execute-review-loop-app/README.md +0 -5
- package/evals/fixtures/tiny-execute-review-loop-app/docs/billing-policy.md +0 -9
- package/evals/fixtures/tiny-execute-review-loop-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/previous-agent-summary.md +0 -15
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/review-pass-1-non-pass.md +0 -11
- package/evals/fixtures/tiny-execute-review-loop-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-execute-review-loop-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-execute-review-loop-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-execute-review-loop-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-execute-review-loop-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-flaky-session-app/README.md +0 -10
- package/evals/fixtures/tiny-flaky-session-app/src/session-timeout.ts +0 -13
- package/evals/fixtures/tiny-flaky-session-app/tests/session-timeout.test.md +0 -8
- package/evals/fixtures/tiny-handoff-app/README.md +0 -5
- package/evals/fixtures/tiny-handoff-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-handoff-app/docs/handoffs/old-agent-handoff.md +0 -8
- package/evals/fixtures/tiny-handoff-app/docs/plans/billing-retry-plan.md +0 -11
- package/evals/fixtures/tiny-handoff-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-handoff-report-app/README.md +0 -3
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.diff +0 -1
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.md +0 -1
- package/evals/fixtures/tiny-handoff-report-app/evals/scores/hof-003-score.txt +0 -4
- package/evals/fixtures/tiny-plan-app/README.md +0 -5
- package/evals/fixtures/tiny-plan-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-plan-app/docs/specs/failed-payment-grace-period.md +0 -25
- package/evals/fixtures/tiny-plan-app/src/billing.ts +0 -17
- package/evals/fixtures/tiny-plan-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-plan-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-plan-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/README.md +0 -3
- package/evals/fixtures/tiny-plan-strict-api-app/docs/billing-policy.md +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/docs/specs/billing-webhook-api.md +0 -25
- package/evals/fixtures/tiny-plan-strict-api-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/src/webhooks.ts +0 -3
- package/evals/fixtures/tiny-plan-strict-api-app/tests/billing-webhook.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/.freeflow/config.json +0 -3
- package/evals/fixtures/tiny-post-setup-invalid-config-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-post-setup-missing-config-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-post-setup-source-conflict-app/.freeflow/config.json +0 -3
- package/evals/fixtures/tiny-post-setup-source-conflict-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-report-performance-app/README.md +0 -11
- package/evals/fixtures/tiny-report-performance-app/docs/performance.md +0 -10
- package/evals/fixtures/tiny-report-performance-app/src/report-summary.ts +0 -29
- package/evals/fixtures/tiny-report-performance-app/tests/report-performance.test.md +0 -8
- package/evals/fixtures/tiny-review-request-app/README.md +0 -3
- package/evals/fixtures/tiny-review-request-app/docs/billing-policy.md +0 -5
- package/evals/fixtures/tiny-review-request-app/docs/plans/billing-webhook-api.md +0 -19
- package/evals/fixtures/tiny-review-request-app/docs/review/previous-agent-summary.md +0 -20
- package/evals/fixtures/tiny-review-request-app/docs/specs/billing-webhook-api.md +0 -25
- package/evals/fixtures/tiny-review-request-app/src/billing.ts +0 -11
- package/evals/fixtures/tiny-review-request-app/src/webhooks.ts +0 -19
- package/evals/fixtures/tiny-review-request-app/tests/billing-webhook.test.md +0 -8
- package/evals/fixtures/tiny-review-work-app/README.md +0 -5
- package/evals/fixtures/tiny-review-work-app/docs/billing-policy.md +0 -9
- package/evals/fixtures/tiny-review-work-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-review-work-app/docs/review/previous-agent-summary.md +0 -15
- package/evals/fixtures/tiny-review-work-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-review-work-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-review-work-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-review-work-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-review-work-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-saas-app/README.md +0 -16
- package/evals/fixtures/tiny-saas-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-saas-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-saas-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-saas-app/docs/specs/settings-title.md +0 -13
- package/evals/fixtures/tiny-saas-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-saas-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-saas-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-saas-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-saas-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-saas-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-saas-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-setup-agents-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-both-app/AGENTS.md +0 -5
- package/evals/fixtures/tiny-setup-both-app/CLAUDE.md +0 -5
- package/evals/fixtures/tiny-setup-claude-app/CLAUDE.md +0 -6
- package/evals/fixtures/tiny-setup-codex-rules-app/.codex/rules/shell.rules +0 -5
- package/evals/fixtures/tiny-setup-codex-rules-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-conflict-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-existing-block-app/AGENTS.md +0 -14
- package/evals/fixtures/tiny-setup-four-rules-app/CLAUDE.md +0 -6
- package/evals/fixtures/tiny-skill-dev-app/README.md +0 -10
- package/evals/fixtures/tiny-skill-dev-app/evals/prompts/review-pr-001.txt +0 -8
- package/evals/fixtures/tiny-skill-dev-app/evals/reports/review-pr-failure.md +0 -17
- package/evals/fixtures/tiny-skill-dev-app/skills/review-pr/SKILL.md +0 -13
- package/evals/fixtures/tiny-team-app/README.md +0 -5
- package/evals/fixtures/tiny-team-app/docs/product.md +0 -6
- package/evals/fixtures/tiny-team-app/src/teams.ts +0 -9
- package/evals/fixtures/tiny-team-app/src/users.ts +0 -5
- package/evals/fixtures/tiny-team-app/tests/teams.test.md +0 -7
- package/evals/fixtures/tiny-verify-app/README.md +0 -5
- package/evals/fixtures/tiny-verify-app/src/settings-nav.ts +0 -1
- package/evals/fixtures/tiny-verify-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-verify-app/tests/settings.test.md +0 -7
- package/evals/prompts/aon-001.txt +0 -2
- package/evals/prompts/byp-001.txt +0 -2
- package/evals/prompts/byp-002.txt +0 -2
- package/evals/prompts/cap-001.txt +0 -3
- package/evals/prompts/cap-002.txt +0 -3
- package/evals/prompts/cmd-001.txt +0 -3
- package/evals/prompts/cmd-002.txt +0 -3
- package/evals/prompts/cmd-003.txt +0 -6
- package/evals/prompts/cmd-004.txt +0 -5
- package/evals/prompts/cmd-005.txt +0 -5
- package/evals/prompts/cmd-006.txt +0 -5
- package/evals/prompts/cmd-007.txt +0 -5
- package/evals/prompts/cmd-008.txt +0 -5
- package/evals/prompts/cmd-009.txt +0 -8
- package/evals/prompts/cmd-010.txt +0 -5
- package/evals/prompts/cmd-011.txt +0 -5
- package/evals/prompts/cmd-012.txt +0 -5
- package/evals/prompts/cmd-013.txt +0 -5
- package/evals/prompts/cmd-014.txt +0 -7
- package/evals/prompts/cmd-015.txt +0 -7
- package/evals/prompts/cmd-016.txt +0 -5
- package/evals/prompts/cmt-001.txt +0 -1
- package/evals/prompts/cmt-002.txt +0 -1
- package/evals/prompts/cmt-003.txt +0 -1
- package/evals/prompts/cmt-004.txt +0 -1
- package/evals/prompts/cmt-005.txt +0 -1
- package/evals/prompts/dfd-001.txt +0 -3
- package/evals/prompts/dia-001.txt +0 -3
- package/evals/prompts/dia-002.txt +0 -3
- package/evals/prompts/dia-003.txt +0 -3
- package/evals/prompts/dis-001.txt +0 -5
- package/evals/prompts/dis-002.txt +0 -5
- package/evals/prompts/esk-001.txt +0 -5
- package/evals/prompts/esk-002.txt +0 -5
- package/evals/prompts/esk-003.txt +0 -5
- package/evals/prompts/grc-001.txt +0 -3
- package/evals/prompts/hof-001.txt +0 -3
- package/evals/prompts/hof-002.txt +0 -1
- package/evals/prompts/hof-003.txt +0 -1
- package/evals/prompts/hof-004.txt +0 -1
- package/evals/prompts/hof-005.txt +0 -1
- package/evals/prompts/ivg-001.txt +0 -1
- package/evals/prompts/ivg-002.txt +0 -11
- package/evals/prompts/ivg-003.txt +0 -1
- package/evals/prompts/ivg-004.txt +0 -1
- package/evals/prompts/mode-001.txt +0 -3
- package/evals/prompts/mode-002.txt +0 -1
- package/evals/prompts/mode-003.txt +0 -3
- package/evals/prompts/mode-004.txt +0 -3
- package/evals/prompts/mode-005.txt +0 -3
- package/evals/prompts/mode-006.txt +0 -3
- package/evals/prompts/mode-007.txt +0 -3
- package/evals/prompts/otr-001.txt +0 -3
- package/evals/prompts/otr-002.txt +0 -3
- package/evals/prompts/rar-001.txt +0 -5
- package/evals/prompts/rar-002.txt +0 -3
- package/evals/prompts/rar-003.txt +0 -6
- package/evals/prompts/rar-004.txt +0 -15
- package/evals/prompts/rar-005.txt +0 -10
- package/evals/prompts/rbr-001.txt +0 -3
- package/evals/prompts/rbr-002.txt +0 -3
- package/evals/prompts/rbr-003.txt +0 -7
- package/evals/prompts/rev-001.txt +0 -3
- package/evals/prompts/rev-002.txt +0 -5
- package/evals/prompts/rev-003.txt +0 -6
- package/evals/prompts/rev-004.txt +0 -1
- package/evals/prompts/rev-005.txt +0 -15
- package/evals/prompts/rev-006.txt +0 -5
- package/evals/prompts/stp-001.txt +0 -1
- package/evals/prompts/stp-002.txt +0 -1
- package/evals/prompts/stp-003.txt +0 -1
- package/evals/prompts/stp-004.txt +0 -1
- package/evals/prompts/stp-005.txt +0 -1
- package/evals/prompts/stp-006.txt +0 -1
- package/evals/prompts/stp-007.txt +0 -1
- package/evals/prompts/stp-008.txt +0 -1
- package/evals/prompts/stp-009.txt +0 -1
- package/evals/prompts/stp-010.txt +0 -1
- package/evals/prompts/stp-011.txt +0 -1
- package/evals/prompts/vfy-003.txt +0 -1
- package/evals/prompts/wpl-001.txt +0 -3
- package/evals/prompts/wpl-002.txt +0 -5
- package/evals/prompts/wpl-003.txt +0 -3
- package/evals/prompts/wpl-004.txt +0 -5
- package/evals/prompts/wpl-005.txt +0 -1
- package/evals/prompts/wrc-001.txt +0 -10
- package/evals/prompts/wrc-002.txt +0 -3
- package/evals/prompts/wsk-001.txt +0 -6
- package/evals/prompts/wsp-001.txt +0 -5
- package/evals/prompts/wsp-002.txt +0 -15
- package/evals/prompts/wsp-003.txt +0 -3
- package/evals/prompts/wsp-004.txt +0 -15
- package/evals/prompts/wsp-005.txt +0 -5
- package/evals/prompts/wsp-006.txt +0 -12
- package/evals/prompts/wsp-007.txt +0 -9
- package/evals/prompts/wsp-008.txt +0 -13
- package/evals/prompts/xpl-001.txt +0 -1
- package/evals/prompts/xpl-002.txt +0 -3
- package/evals/prompts/xpl-003.txt +0 -3
- package/evals/prompts/xpl-004.txt +0 -3
- package/evals/prompts/xpl-005.txt +0 -5
- package/evals/prompts/xpl-006.txt +0 -3
- package/evals/registries/adversarial-evals.json +0 -39
- package/evals/registries/fixture-evals.json +0 -2513
- package/evals/registries/handoff-evals.json +0 -74
- package/evals/registries/legacy-evals.json +0 -147
- package/evals/registries/skill-evidence.json +0 -193
- package/evals/registries/verify-work-evals.json +0 -46
- package/evals/reports/acceptance/v0.1-acceptance-report.md +0 -130
- package/evals/reports/by-command-surface/command-surface-1-report.md +0 -45
- package/evals/reports/by-command-surface/command-surface-10-report.md +0 -76
- package/evals/reports/by-command-surface/command-surface-11-report.md +0 -77
- package/evals/reports/by-command-surface/command-surface-12-report.md +0 -91
- package/evals/reports/by-command-surface/command-surface-13-report.md +0 -64
- package/evals/reports/by-command-surface/command-surface-14-report.md +0 -66
- package/evals/reports/by-command-surface/command-surface-15-report.md +0 -124
- package/evals/reports/by-command-surface/command-surface-16-report.md +0 -39
- package/evals/reports/by-command-surface/command-surface-2-report.md +0 -80
- package/evals/reports/by-command-surface/command-surface-3-report.md +0 -86
- package/evals/reports/by-command-surface/command-surface-4-report.md +0 -76
- package/evals/reports/by-command-surface/command-surface-5-report.md +0 -80
- package/evals/reports/by-command-surface/command-surface-6-report.md +0 -74
- package/evals/reports/by-command-surface/command-surface-7-report.md +0 -84
- package/evals/reports/by-command-surface/command-surface-8-report.md +0 -79
- package/evals/reports/by-command-surface/command-surface-9-report.md +0 -79
- package/evals/reports/by-command-surface/command-surface-matrix.md +0 -91
- package/evals/reports/by-skill/bypass-1-report.md +0 -124
- package/evals/reports/by-skill/capture-decisions-1-report.md +0 -99
- package/evals/reports/by-skill/capture-decisions-2-report.md +0 -102
- package/evals/reports/by-skill/commit-work-1-report.md +0 -165
- package/evals/reports/by-skill/commit-work-2-report.md +0 -109
- package/evals/reports/by-skill/design-for-depth-1-report.md +0 -95
- package/evals/reports/by-skill/diagnose-failure-1-report.md +0 -91
- package/evals/reports/by-skill/diagnose-failure-2-report.md +0 -94
- package/evals/reports/by-skill/discover-1-report.md +0 -118
- package/evals/reports/by-skill/evaluate-skill-2-report.md +0 -71
- package/evals/reports/by-skill/execute-plan-1-report.md +0 -103
- package/evals/reports/by-skill/execute-plan-2-report.md +0 -112
- package/evals/reports/by-skill/execute-plan-3-report.md +0 -87
- package/evals/reports/by-skill/grill-context-1-report.md +0 -103
- package/evals/reports/by-skill/handoff-1-report.md +0 -123
- package/evals/reports/by-skill/handoff-2-report.md +0 -105
- package/evals/reports/by-skill/handoff-3-report.md +0 -100
- package/evals/reports/by-skill/handoff-7-report.md +0 -77
- package/evals/reports/by-skill/interview-gate-1-report.md +0 -63
- package/evals/reports/by-skill/interview-gate-2-report.md +0 -105
- package/evals/reports/by-skill/mode-contract-1-report.md +0 -74
- package/evals/reports/by-skill/mode-contract-2-report.md +0 -87
- package/evals/reports/by-skill/mode-contract-3-report.md +0 -65
- package/evals/reports/by-skill/mode-contract-4-report.md +0 -71
- package/evals/reports/by-skill/mode-contract-5-report.md +0 -70
- package/evals/reports/by-skill/output-router-1-report.md +0 -76
- package/evals/reports/by-skill/output-router-2-report.md +0 -65
- package/evals/reports/by-skill/research-1-report.md +0 -144
- package/evals/reports/by-skill/research-brief-1-report.md +0 -94
- package/evals/reports/by-skill/research-brief-2-report.md +0 -75
- package/evals/reports/by-skill/review-artifact-1-report.md +0 -91
- package/evals/reports/by-skill/review-artifact-2-report.md +0 -79
- package/evals/reports/by-skill/review-artifact-3-report.md +0 -120
- package/evals/reports/by-skill/review-artifact-4-report.md +0 -89
- package/evals/reports/by-skill/review-artifact-5-report.md +0 -66
- package/evals/reports/by-skill/review-work-1-report.md +0 -85
- package/evals/reports/by-skill/review-work-2-report.md +0 -87
- package/evals/reports/by-skill/review-work-3-report.md +0 -104
- package/evals/reports/by-skill/review-work-4-report.md +0 -108
- package/evals/reports/by-skill/review-work-5-report.md +0 -73
- package/evals/reports/by-skill/review-work-6-report.md +0 -66
- package/evals/reports/by-skill/setup-freeflow-1-report.md +0 -199
- package/evals/reports/by-skill/setup-freeflow-2-report.md +0 -105
- package/evals/reports/by-skill/setup-freeflow-3-report.md +0 -105
- package/evals/reports/by-skill/setup-freeflow-4-report.md +0 -88
- package/evals/reports/by-skill/setup-freeflow-5-report.md +0 -115
- package/evals/reports/by-skill/skill-authoring-1-report.md +0 -81
- package/evals/reports/by-skill/verify-work-1-report.md +0 -111
- package/evals/reports/by-skill/verify-work-2-report.md +0 -120
- package/evals/reports/by-skill/workflow-1-report.md +0 -65
- package/evals/reports/by-skill/write-plan-1-report.md +0 -123
- package/evals/reports/by-skill/write-plan-2-report.md +0 -98
- package/evals/reports/by-skill/write-plan-3-report.md +0 -110
- package/evals/reports/by-skill/write-spec-1-report.md +0 -88
- package/evals/reports/by-skill/write-spec-2-report.md +0 -89
- package/evals/reports/by-skill/write-spec-3-report.md +0 -90
- package/evals/reports/by-skill/write-spec-4-report.md +0 -167
- package/evals/reports/harness/claude-harness-1-report.md +0 -103
- package/evals/reports/harness/eval-harness-1-report.md +0 -91
- package/evals/reports/iterations/iteration-1-report.md +0 -71
- package/evals/reports/iterations/iteration-2-report.md +0 -78
- package/evals/reports/iterations/iteration-3-report.md +0 -118
- package/evals/reports/iterations/iteration-4-report.md +0 -131
- package/evals/reports/iterations/iteration-5-report.md +0 -149
- package/evals/reports/iterations/iteration-6-report.md +0 -136
- package/evals/reports/runtime/always-on-runtime-1-report.md +0 -107
- package/evals/reports/runtime/context-mode-normalized-benchmark-1-report.md +0 -52
- package/evals/reports/runtime/context-mode-real-deep-baseline-1-report.md +0 -410
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md +0 -203
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md +0 -51
- package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md +0 -99
- package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md +0 -166
- package/evals/reports/runtime/eryx-python-proof-spike-1-report.md +0 -96
- package/evals/reports/runtime/eryx-python-proof-spike-2-report.md +0 -151
- package/evals/reports/runtime/eryx-python-script-derive-execution-1-report.md +0 -233
- package/evals/reports/runtime/jq-script-derive-execution-1-report.md +0 -215
- package/evals/reports/runtime/jq-wasm-proof-spike-1-report.md +0 -67
- package/evals/reports/runtime/output-router-benchmark-1-report.md +0 -81
- package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md +0 -35
- package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md +0 -150
- package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md +0 -46
- package/evals/reports/runtime/output-router-command-benchmark-1-report.md +0 -62
- package/evals/reports/runtime/output-router-index-benchmark-1-report.md +0 -64
- package/evals/reports/runtime/output-router-regression-1-report.md +0 -81
- package/evals/reports/runtime/output-router-transform-eval-1-report.md +0 -28
- package/evals/reports/runtime/pi-observed-routing-eval-1-report.md +0 -44
- package/evals/reports/runtime/quickjs-script-derive-execution-1-report.md +0 -159
- package/evals/reports/runtime/quickjs-wasi-proof-spike-1-report.md +0 -55
- package/evals/reports/runtime/render-format-row-protocol-1-report.md +0 -58
- package/evals/reports/runtime/script-sandbox-adapter-selection-review-1-report.md +0 -84
- package/evals/reports/runtime/script-sandbox-probe-resource-hardening-1-report.md +0 -51
- package/evals/reports/runtime/storage-policy-benchmark-1-report.md +0 -176
- package/evals/reports/runtime/vault-index-storage-spike-1-report.md +0 -39
- package/evals/reports/runtime/workflow-context-hook-1-report.md +0 -82
- package/evals/runbooks/adversarial-eval-runbook.md +0 -28
- package/evals/runbooks/fixture-eval-runbook.md +0 -137
- package/evals/runbooks/handoff-eval-runbook.md +0 -22
- package/evals/runbooks/verify-work-eval-runbook.md +0 -27
- package/evals/scripts/audit-command-surface.sh +0 -160
- package/evals/scripts/check-activation-contract.sh +0 -121
- package/evals/scripts/check-runtime-context-hook.sh +0 -183
- package/evals/scripts/grade-fixture-eval.sh +0 -446
- package/evals/scripts/run-claude-fixture-eval.sh +0 -26
- package/evals/scripts/run-codex-fixture-eval.sh +0 -26
- package/evals/scripts/run-eryx-python-proof-spike.js +0 -559
- package/evals/scripts/run-fixture-eval-by-id.sh +0 -23
- package/evals/scripts/run-fixture-eval.sh +0 -464
- package/evals/scripts/run-jq-wasm-proof-spike.js +0 -299
- package/evals/scripts/run-output-router-transform-eval.js +0 -115
- package/evals/scripts/run-pi-observed-routing-eval.js +0 -478
- package/evals/scripts/run-quickjs-wasi-proof-spike.js +0 -339
- package/evals/scripts/skill-evidence.sh +0 -316
- package/evals/scripts/validate-release-metadata.sh +0 -486
- package/evals/suites/v0.1-acceptance-suite.md +0 -131
- package/pi-extension/dist/host-producer-identification.js +0 -186
- package/pi-extension/dist/mcp-config.js +0 -30
- package/pi-extension/dist/native-safety-net.js +0 -209
- package/pi-extension/dist/observed-tool-routing.js +0 -114
- package/pi-extension/dist/renderers.js +0 -835
- package/pi-extension/dist/router-tools.js +0 -438
- package/pi-extension/dist/runtime-context.js +0 -288
- package/pi-extension/dist/schemas.js +0 -254
- package/pi-extension/dist/status.js +0 -449
- package/pi-extension/dist/utils.js +0 -467
- package/pi-extension/src/runtime-context.ts +0 -347
- package/pi-extension/src/status.ts +0 -505
- package/plugin-docs/README.md +0 -40
- package/plugin-docs/adr/0001-three-modes.md +0 -27
- package/plugin-docs/adr/0002-evals-before-hooks.md +0 -17
- package/plugin-docs/adr/0003-release-boundary.md +0 -21
- package/plugin-docs/adr/0004-discover-replaces-shallow-discovery-skills.md +0 -26
- package/plugin-docs/adr/README.md +0 -10
- package/plugin-docs/architecture.md +0 -89
- package/plugin-docs/output-router.md +0 -396
- package/plugin-docs/release-evidence.md +0 -77
- package/plugin-docs/skills.md +0 -44
- package/plugin-docs/workflow.md +0 -57
- package/skills/diagnose-failure/references/feedback-loop-catalog.md +0 -25
- package/skills/discover/ARTIFACT-DESTINATIONS.md +0 -58
- package/skills/discover/CHECKPOINTS.md +0 -74
- package/skills/discover/SKILL.md +0 -173
- package/skills/evaluate-skill/references/eval-patterns.md +0 -41
- package/skills/evaluate-skill/references/grading-priority.md +0 -33
- package/skills/execute-plan/SKILL.md +0 -133
- package/skills/execute-plan/references/execution-map.md +0 -122
- package/skills/interview-gate/SKILL.md +0 -90
- package/skills/setup-freeflow/references/output-router-setup.md +0 -102
- package/skills/workflow/references/workflow-map.md +0 -104
- /package/pi-extension/src/{mcp-config.ts → integrations/mcp-config.ts} +0 -0
|
@@ -1,199 +0,0 @@
|
|
|
1
|
-
# Setup Freeflow Eval Report - Iteration 1
|
|
2
|
-
|
|
3
|
-
Date: 2026-05-25
|
|
4
|
-
|
|
5
|
-
## Scope
|
|
6
|
-
|
|
7
|
-
Added `setup-freeflow` and focused fixture evals for first-run setup:
|
|
8
|
-
|
|
9
|
-
- `STP-001` clean repo with `AGENTS.md` only
|
|
10
|
-
- `STP-002` repo with both `AGENTS.md` and `CLAUDE.md`
|
|
11
|
-
- `STP-003` repo instruction conflict
|
|
12
|
-
- `STP-004` existing stale `## Freeflow` block
|
|
13
|
-
|
|
14
|
-
Compared:
|
|
15
|
-
|
|
16
|
-
- Baseline: no Freeflow skill files
|
|
17
|
-
- With skill: `skills/setup-freeflow/SKILL.md`
|
|
18
|
-
|
|
19
|
-
## Skill Added
|
|
20
|
-
|
|
21
|
-
New skill:
|
|
22
|
-
|
|
23
|
-
- `skills/setup-freeflow/SKILL.md`
|
|
24
|
-
|
|
25
|
-
Core behavior:
|
|
26
|
-
|
|
27
|
-
- create only `.freeflow/config.json` with `defaultMode: "workflow"`
|
|
28
|
-
- add only the minimal `## Freeflow` activation block
|
|
29
|
-
- ask before choosing between `AGENTS.md` and `CLAUDE.md`
|
|
30
|
-
- stop before setup when repo rules conflict with Freeflow's ask/verify behavior
|
|
31
|
-
- do not create docs pages, hooks, state files, empty `CONTEXT.md`, handoffs, or inventories
|
|
32
|
-
|
|
33
|
-
## Eval Changes
|
|
34
|
-
|
|
35
|
-
Added fixtures:
|
|
36
|
-
|
|
37
|
-
- `evals/fixtures/tiny-setup-agents-app/`
|
|
38
|
-
- `evals/fixtures/tiny-setup-both-app/`
|
|
39
|
-
- `evals/fixtures/tiny-setup-conflict-app/`
|
|
40
|
-
- `evals/fixtures/tiny-setup-existing-block-app/`
|
|
41
|
-
|
|
42
|
-
Added prompts:
|
|
43
|
-
|
|
44
|
-
- `evals/prompts/stp-001.txt`
|
|
45
|
-
- `evals/prompts/stp-002.txt`
|
|
46
|
-
- `evals/prompts/stp-003.txt`
|
|
47
|
-
- `evals/prompts/stp-004.txt`
|
|
48
|
-
|
|
49
|
-
Added eval entries to:
|
|
50
|
-
|
|
51
|
-
- `evals/registries/fixture-evals.json`
|
|
52
|
-
|
|
53
|
-
## Results
|
|
54
|
-
|
|
55
|
-
| Eval | Baseline | With Skill | Read |
|
|
56
|
-
|---|---:|---:|---|
|
|
57
|
-
| STP-001 clean `AGENTS.md` setup | Fail | 10/10 | Baseline created broad docs workflow scaffold. With-skill created only config plus minimal `AGENTS.md` block. |
|
|
58
|
-
| STP-002 both agent files | Fail | 10/10 | Baseline edited both instruction files and added docs. First with-skill failed by choosing `AGENTS.md`; final with-skill asked which target to update and made no file changes. |
|
|
59
|
-
| STP-003 conflicting repo rule | Fail | 10/10 | Baseline created broad docs workflow scaffold. First with-skill failed by installing as advisory; final with-skill named the conflict and made no file changes. |
|
|
60
|
-
| STP-004 existing block update | Fail | 10/10 | Baseline created docs and mode memory. With-skill replaced the existing block in place and created only the minimal config. |
|
|
61
|
-
|
|
62
|
-
## Scoring
|
|
63
|
-
|
|
64
|
-
### STP-001
|
|
65
|
-
|
|
66
|
-
| Assertion | Baseline | With Skill |
|
|
67
|
-
|---|---:|---:|
|
|
68
|
-
| Creates `.freeflow/config.json` with exactly `defaultMode: "workflow"` | 0 | 2 |
|
|
69
|
-
| Adds exactly one `## Freeflow` block to `AGENTS.md` | 1 | 2 |
|
|
70
|
-
| Does not create extra setup artifacts | 0 | 2 |
|
|
71
|
-
| Does not add volatile config fields | 1 | 2 |
|
|
72
|
-
| Reports verification evidence | 1 | 2 |
|
|
73
|
-
|
|
74
|
-
Result:
|
|
75
|
-
|
|
76
|
-
- Baseline: 3/10
|
|
77
|
-
- With skill: 10/10
|
|
78
|
-
|
|
79
|
-
### STP-002
|
|
80
|
-
|
|
81
|
-
| Assertion | Baseline | With Skill |
|
|
82
|
-
|---|---:|---:|
|
|
83
|
-
| Does not create, modify, or delete files | 0 | 2 |
|
|
84
|
-
| Detects both `AGENTS.md` and `CLAUDE.md` | 2 | 2 |
|
|
85
|
-
| States activation target ambiguity | 0 | 2 |
|
|
86
|
-
| Asks which file to update | 0 | 2 |
|
|
87
|
-
| Does not silently update both files | 0 | 2 |
|
|
88
|
-
|
|
89
|
-
Result:
|
|
90
|
-
|
|
91
|
-
- Baseline: 2/10
|
|
92
|
-
- With skill: 10/10
|
|
93
|
-
|
|
94
|
-
### STP-004
|
|
95
|
-
|
|
96
|
-
| Assertion | Baseline | With Skill |
|
|
97
|
-
|---|---:|---:|
|
|
98
|
-
| Creates `.freeflow/config.json` with exactly `defaultMode: "workflow"` | 0 | 2 |
|
|
99
|
-
| Leaves exactly one `## Freeflow` block in `AGENTS.md` | 2 | 2 |
|
|
100
|
-
| Replaces stale activation text with the minimal current block | 0 | 2 |
|
|
101
|
-
| Does not create extra setup artifacts | 0 | 2 |
|
|
102
|
-
| Reports verification evidence | 1 | 2 |
|
|
103
|
-
|
|
104
|
-
Result:
|
|
105
|
-
|
|
106
|
-
- Baseline: 3/10
|
|
107
|
-
- With skill: 10/10
|
|
108
|
-
|
|
109
|
-
### STP-003
|
|
110
|
-
|
|
111
|
-
| Assertion | Baseline | With Skill |
|
|
112
|
-
|---|---:|---:|
|
|
113
|
-
| Does not create, modify, or delete files | 0 | 2 |
|
|
114
|
-
| Inspects `AGENTS.md` before editing | 2 | 2 |
|
|
115
|
-
| Names the ask/verify conflict | 0 | 2 |
|
|
116
|
-
| Asks whether to revise, install advisory, or skip setup | 0 | 2 |
|
|
117
|
-
| Does not silently rewrite conflicting instructions | 0 | 2 |
|
|
118
|
-
|
|
119
|
-
Result:
|
|
120
|
-
|
|
121
|
-
- Baseline: 2/10
|
|
122
|
-
- With skill: 10/10
|
|
123
|
-
|
|
124
|
-
## Evidence
|
|
125
|
-
|
|
126
|
-
Diff sizes:
|
|
127
|
-
|
|
128
|
-
| Eval | Baseline Diff | With-Skill Diff |
|
|
129
|
-
|---|---:|---:|
|
|
130
|
-
| STP-001 | 1109 bytes | 1014 bytes |
|
|
131
|
-
| STP-002 | 1990 bytes | 0 bytes |
|
|
132
|
-
| STP-003 | 1599 bytes | 0 bytes |
|
|
133
|
-
| STP-004 | 813 bytes | 1102 bytes |
|
|
134
|
-
|
|
135
|
-
Key final outputs:
|
|
136
|
-
|
|
137
|
-
- `STP-002`: `Setup is blocked before edits: both AGENTS.md and CLAUDE.md exist...`
|
|
138
|
-
- `STP-003`: `I did not make file changes. Blocker: AGENTS.md says “Never ask the user clarifying questions,” ...`
|
|
139
|
-
- `STP-004`: `replaced the old Freeflow block with the required activation block`
|
|
140
|
-
|
|
141
|
-
Saved run outputs:
|
|
142
|
-
|
|
143
|
-
- `evals/runs/setup-1/stp-001-baseline-output.md`
|
|
144
|
-
- `evals/runs/setup-2/stp-001-with-skill-output.md`
|
|
145
|
-
- `evals/runs/setup-1/stp-002-baseline-output.md`
|
|
146
|
-
- `evals/runs/setup-2/stp-002-with-skill-output.md`
|
|
147
|
-
- `evals/runs/setup-1/stp-003-baseline-output.md`
|
|
148
|
-
- `evals/runs/setup-2/stp-003-with-skill-output.md`
|
|
149
|
-
- `evals/runs/setup-4/stp-004-baseline-output.md`
|
|
150
|
-
- `evals/runs/setup-4/stp-004-with-skill-output.md`
|
|
151
|
-
|
|
152
|
-
## Iteration Note
|
|
153
|
-
|
|
154
|
-
The first with-skill runs failed two boundaries:
|
|
155
|
-
|
|
156
|
-
- `STP-002`: treated the current Codex runtime as enough to choose `AGENTS.md`.
|
|
157
|
-
- `STP-003`: installed Freeflow as advisory despite conflicting repo instructions.
|
|
158
|
-
|
|
159
|
-
Fix:
|
|
160
|
-
|
|
161
|
-
- added a top-level `Stop Before Editing` section
|
|
162
|
-
- clarified that current runtime alone is not target approval
|
|
163
|
-
- clarified that advisory install is a user decision, not the default
|
|
164
|
-
- made hard stops prohibit config creation and activation block edits until resolved
|
|
165
|
-
|
|
166
|
-
After review, compressed `setup-freeflow/SKILL.md` from 109 lines to 97 lines while keeping the hard-stop rules above target/config instructions. Reran all three with-skill evals as `setup-3`; behavior stayed intact.
|
|
167
|
-
|
|
168
|
-
Final setup-3 diff sizes:
|
|
169
|
-
|
|
170
|
-
| Eval | With-Skill Diff |
|
|
171
|
-
|---|---:|
|
|
172
|
-
| STP-001 | 1014 bytes |
|
|
173
|
-
| STP-002 | 0 bytes |
|
|
174
|
-
| STP-003 | 0 bytes |
|
|
175
|
-
|
|
176
|
-
Added `STP-004` after that review to cover existing activation blocks. The existing skill wording passed without revision.
|
|
177
|
-
|
|
178
|
-
## Verification
|
|
179
|
-
|
|
180
|
-
Commands:
|
|
181
|
-
|
|
182
|
-
```sh
|
|
183
|
-
jq empty evals/registries/fixture-evals.json
|
|
184
|
-
bash -n evals/scripts/run-codex-fixture-eval.sh
|
|
185
|
-
evals/scripts/run-codex-fixture-eval.sh ...
|
|
186
|
-
FREEFLOW_REQUIRE_EMPTY_DIFF=1 evals/scripts/run-codex-fixture-eval.sh ...
|
|
187
|
-
wc -c evals/runs/setup-*/stp-*-output.diff
|
|
188
|
-
wc -l skills/setup-freeflow/SKILL.md
|
|
189
|
-
```
|
|
190
|
-
|
|
191
|
-
Nested `codex exec` required escalation outside the sandbox because the in-process app-server client could not initialize under the default sandbox.
|
|
192
|
-
|
|
193
|
-
## Recommendation
|
|
194
|
-
|
|
195
|
-
Keep `setup-freeflow` as the next developer meta skill.
|
|
196
|
-
|
|
197
|
-
Next useful setup eval later:
|
|
198
|
-
|
|
199
|
-
- existing `.freeflow/config.json` with an explicit persisted default should be preserved or repaired only after asking
|
|
@@ -1,105 +0,0 @@
|
|
|
1
|
-
# Setup Freeflow Eval Report - Iteration 2
|
|
2
|
-
|
|
3
|
-
Date: 2026-05-25
|
|
4
|
-
|
|
5
|
-
## Scope
|
|
6
|
-
|
|
7
|
-
Updated `setup-freeflow` for host-specific always-on setup:
|
|
8
|
-
|
|
9
|
-
- Codex behavior lives in `AGENTS.md`, not `.codex/rules/*.rules`.
|
|
10
|
-
- Claude behavior uses `CLAUDE.md` with an explicit import to `.claude/rules/freeflow-core.md`.
|
|
11
|
-
- Setup uses one compact always-on core block/file.
|
|
12
|
-
- Full workflow behavior stays in skills.
|
|
13
|
-
|
|
14
|
-
## Skill Changes
|
|
15
|
-
|
|
16
|
-
`skills/setup-freeflow/SKILL.md` now:
|
|
17
|
-
|
|
18
|
-
- writes a compact core block with source-truth, interview, forward/re-enter, verification, and stable-capture invariants
|
|
19
|
-
- creates `.freeflow/config.json` with only `defaultMode: "workflow"`
|
|
20
|
-
- selects Codex as `AGENTS.md`
|
|
21
|
-
- selects Claude as `CLAUDE.md` plus `.claude/rules/freeflow-core.md`
|
|
22
|
-
- refuses ambiguous `AGENTS.md` plus `CLAUDE.md` setup without a host target
|
|
23
|
-
- avoids `.codex/rules` for behavioral memory
|
|
24
|
-
- resists splitting Freeflow into multiple always-loaded rule files unless the user explicitly confirms after the one-file recommendation
|
|
25
|
-
|
|
26
|
-
Final skill length: 96 lines.
|
|
27
|
-
|
|
28
|
-
## Eval Changes
|
|
29
|
-
|
|
30
|
-
Updated:
|
|
31
|
-
|
|
32
|
-
- `STP-001`: expects the compact core block in `AGENTS.md`.
|
|
33
|
-
- `STP-004`: expects stale blocks to be replaced by the compact core block.
|
|
34
|
-
|
|
35
|
-
Added:
|
|
36
|
-
|
|
37
|
-
- `STP-005`: Codex setup with existing `.codex/rules/shell.rules`; setup must leave Codex shell policy untouched.
|
|
38
|
-
- `STP-006`: Claude setup creates `CLAUDE.md` import plus one `.claude/rules/freeflow-core.md`.
|
|
39
|
-
- `STP-007`: Claude setup under four-rule-file pressure still creates one `freeflow-core.md`.
|
|
40
|
-
|
|
41
|
-
## Results
|
|
42
|
-
|
|
43
|
-
| Eval | Baseline | With Skill | Read |
|
|
44
|
-
|---|---:|---:|---|
|
|
45
|
-
| STP-001 AGENTS-only setup | Existing baseline fail | Pass | With-skill created only config plus compact `AGENTS.md` core block. |
|
|
46
|
-
| STP-002 ambiguous host files | Existing baseline fail | Pass | With-skill made no edits and asked for Codex, Claude, or both. |
|
|
47
|
-
| STP-003 conflicting repo rules | Existing baseline fail | Pass | With-skill made no edits and named ask/verify conflicts. |
|
|
48
|
-
| STP-004 existing block update | Existing baseline fail | Pass | With-skill replaced stale block in place and created minimal config. |
|
|
49
|
-
| STP-005 Codex rules avoidance | Fail | Pass | Baseline created extra `freeflow.md`; with-skill created config and `AGENTS.md` only, preserving `.codex/rules`. |
|
|
50
|
-
| STP-006 Claude import shape | Fail | Pass | Baseline embedded a long workflow manual in `CLAUDE.md`; with-skill used import plus one core file. |
|
|
51
|
-
| STP-007 four-rule pressure | Fail | Pass | Baseline created four always-loaded rule files; with-skill created one `freeflow-core.md`. |
|
|
52
|
-
|
|
53
|
-
## Evidence
|
|
54
|
-
|
|
55
|
-
Diff sizes:
|
|
56
|
-
|
|
57
|
-
| Eval | Baseline Diff | With-Skill Diff |
|
|
58
|
-
|---|---:|---:|
|
|
59
|
-
| STP-001 | existing report | 1471 bytes |
|
|
60
|
-
| STP-002 | existing report | 0 bytes |
|
|
61
|
-
| STP-003 | existing report | 0 bytes |
|
|
62
|
-
| STP-004 | existing report | 1560 bytes |
|
|
63
|
-
| STP-005 | 1421 bytes | 1481 bytes |
|
|
64
|
-
| STP-006 | 2131 bytes | 959 bytes |
|
|
65
|
-
| STP-007 | 1060 bytes | 953 bytes |
|
|
66
|
-
|
|
67
|
-
Key outputs:
|
|
68
|
-
|
|
69
|
-
- `STP-002`: `Blocked before editing: this repo has both AGENTS.md and CLAUDE.md...`
|
|
70
|
-
- `STP-003`: `Blocked before editing... Never ask... conflicts... skip verification... conflicts...`
|
|
71
|
-
- `STP-005`: with-skill reported `.codex/rules` was not changed for Freeflow behavior.
|
|
72
|
-
- `STP-006`: with-skill reported `CLAUDE.md` import, one core file, and no `.codex/rules`.
|
|
73
|
-
- `STP-007`: baseline created four always-loaded rule files; with-skill used one `freeflow-core.md`.
|
|
74
|
-
|
|
75
|
-
Saved run outputs:
|
|
76
|
-
|
|
77
|
-
- `evals/runs/setup-5/stp-001-with-skill-output.md`
|
|
78
|
-
- `evals/runs/setup-5/stp-002-with-skill-output.md`
|
|
79
|
-
- `evals/runs/setup-5/stp-003-with-skill-output.md`
|
|
80
|
-
- `evals/runs/setup-5/stp-004-with-skill-output.md`
|
|
81
|
-
- `evals/runs/setup-5/stp-005-baseline-output.md`
|
|
82
|
-
- `evals/runs/setup-5/stp-005-with-skill-output.md`
|
|
83
|
-
- `evals/runs/setup-5/stp-006-baseline-output.md`
|
|
84
|
-
- `evals/runs/setup-5/stp-006-with-skill-output.md`
|
|
85
|
-
- `evals/runs/setup-5/stp-007-baseline-output.md`
|
|
86
|
-
- `evals/runs/setup-5/stp-007-with-skill-output.md`
|
|
87
|
-
|
|
88
|
-
## Verification
|
|
89
|
-
|
|
90
|
-
Commands:
|
|
91
|
-
|
|
92
|
-
```sh
|
|
93
|
-
jq empty evals/registries/fixture-evals.json
|
|
94
|
-
bash -n evals/scripts/run-codex-fixture-eval.sh
|
|
95
|
-
wc -l skills/setup-freeflow/SKILL.md
|
|
96
|
-
git diff --check
|
|
97
|
-
evals/scripts/run-codex-fixture-eval.sh ...
|
|
98
|
-
wc -c evals/runs/setup-5/stp-00*-output.diff
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Nested `codex exec` required escalation outside the sandbox, consistent with earlier fixture eval runs.
|
|
102
|
-
|
|
103
|
-
## Recommendation
|
|
104
|
-
|
|
105
|
-
Keep the compact always-on setup. Next useful eval target is runtime behavior after setup: a source-truth conflict should stop and ask before changing docs/tests or product behavior.
|
|
@@ -1,105 +0,0 @@
|
|
|
1
|
-
# Setup Freeflow Eval Report - Iteration 3
|
|
2
|
-
|
|
3
|
-
Date: 2026-05-26
|
|
4
|
-
|
|
5
|
-
## Scope
|
|
6
|
-
|
|
7
|
-
Updated `setup-freeflow` for setup profiles and explicit persisted defaults.
|
|
8
|
-
|
|
9
|
-
Owned paths:
|
|
10
|
-
|
|
11
|
-
- `skills/setup-freeflow/`
|
|
12
|
-
- setup prompts and fixture registry entries
|
|
13
|
-
|
|
14
|
-
No repo-local hooks, CLI commands, global standards, or `.codex/rules` behavior files were added.
|
|
15
|
-
|
|
16
|
-
## Skill Changes
|
|
17
|
-
|
|
18
|
-
`setup-freeflow/SKILL.md` now:
|
|
19
|
-
|
|
20
|
-
- has a trigger description that covers host selection, multi-agent activation, config creation, and setup-time default-mode changes
|
|
21
|
-
- keeps hard stops and host target selection in the main skill
|
|
22
|
-
- allows `.freeflow/config.json` to use an explicitly requested valid repo default: `conversation`, `workflow`, or `strict-workflow`
|
|
23
|
-
- keeps `workflow` as the default when the user does not explicitly ask to persist another mode
|
|
24
|
-
- links to `references/host-setup.md` for host/profile/default-mode/enforcement details
|
|
25
|
-
|
|
26
|
-
Added:
|
|
27
|
-
|
|
28
|
-
- `setup-freeflow/references/host-setup.md`
|
|
29
|
-
|
|
30
|
-
The reference keeps profile details out of always-loaded setup text:
|
|
31
|
-
|
|
32
|
-
- Codex vs Claude vs multi-agent shape
|
|
33
|
-
- solo/team/strict setup defaults
|
|
34
|
-
- enforcement requests are separate from setup
|
|
35
|
-
- no repo-local hooks or CLI during setup
|
|
36
|
-
|
|
37
|
-
Final main skill length: 97 lines.
|
|
38
|
-
|
|
39
|
-
## Eval Added
|
|
40
|
-
|
|
41
|
-
Added:
|
|
42
|
-
|
|
43
|
-
- `STP-008`: Codex setup with explicit request to make `strict-workflow` the repo default.
|
|
44
|
-
|
|
45
|
-
Expected behavior:
|
|
46
|
-
|
|
47
|
-
- update `AGENTS.md` with the compact Freeflow block
|
|
48
|
-
- create `.freeflow/config.json` with only `defaultMode: "strict-workflow"`
|
|
49
|
-
- avoid repo-local hooks, docs pages, state files, `.codex/rules` behavior files, and volatile config fields
|
|
50
|
-
|
|
51
|
-
## Results
|
|
52
|
-
|
|
53
|
-
`STP-008` baseline: fail.
|
|
54
|
-
|
|
55
|
-
- Added an ad hoc `AGENTS.md` block.
|
|
56
|
-
- Did not create `.freeflow/config.json`.
|
|
57
|
-
- Did not use the compact source-truth/user-decision/verification activation block.
|
|
58
|
-
|
|
59
|
-
`STP-008` current skill before revision: fail.
|
|
60
|
-
|
|
61
|
-
- Created compact `AGENTS.md` activation.
|
|
62
|
-
- Created `.freeflow/config.json`.
|
|
63
|
-
- Wrote `defaultMode: "workflow"` despite the explicit `strict-workflow` default request.
|
|
64
|
-
|
|
65
|
-
`STP-008` after revision: pass.
|
|
66
|
-
|
|
67
|
-
- Created compact `AGENTS.md` activation.
|
|
68
|
-
- Created `.freeflow/config.json` with only `defaultMode: "strict-workflow"`.
|
|
69
|
-
- Reported JSON parsing, one activation block, no `.codex/rules`, and limited file inventory.
|
|
70
|
-
|
|
71
|
-
Regression runs after the final `SKILL.md` structure and description update:
|
|
72
|
-
|
|
73
|
-
- `STP-001`: pass, default setup still writes `workflow`.
|
|
74
|
-
- `STP-005`: pass, Codex setup still avoids `.codex/rules`.
|
|
75
|
-
- `STP-006`: pass, Claude setup still uses `CLAUDE.md` import plus one `freeflow-core.md`.
|
|
76
|
-
|
|
77
|
-
## Evidence
|
|
78
|
-
|
|
79
|
-
Saved final runs:
|
|
80
|
-
|
|
81
|
-
- `evals/runs/setup-7/stp-001-with-skill-output.md`
|
|
82
|
-
- `evals/runs/setup-7/stp-005-with-skill-output.md`
|
|
83
|
-
- `evals/runs/setup-7/stp-006-with-skill-output.md`
|
|
84
|
-
- `evals/runs/setup-7/stp-008-with-skill-output.md`
|
|
85
|
-
|
|
86
|
-
Key diffs:
|
|
87
|
-
|
|
88
|
-
- `evals/runs/setup-7/stp-008-with-skill-output.diff`
|
|
89
|
-
- `evals/runs/setup-7/stp-001-with-skill-output.diff`
|
|
90
|
-
- `evals/runs/setup-7/stp-005-with-skill-output.diff`
|
|
91
|
-
- `evals/runs/setup-7/stp-006-with-skill-output.diff`
|
|
92
|
-
|
|
93
|
-
The nested agents verified config JSON and activation counts in their final responses. The fixture diff reports hidden setup directories as new directories, so final-response command evidence is part of grading for hidden `.freeflow` and `.claude` file contents.
|
|
94
|
-
|
|
95
|
-
## Verification
|
|
96
|
-
|
|
97
|
-
Commands:
|
|
98
|
-
|
|
99
|
-
```sh
|
|
100
|
-
jq empty evals/registries/fixture-evals.json
|
|
101
|
-
wc -l skills/setup-freeflow/SKILL.md skills/setup-freeflow/references/host-setup.md
|
|
102
|
-
git diff --check
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
Nested `codex exec` required escalation outside the sandbox, consistent with prior fixture evals.
|
|
@@ -1,88 +0,0 @@
|
|
|
1
|
-
# Setup Freeflow Eval Report - Iteration 4
|
|
2
|
-
|
|
3
|
-
Date: 2026-06-14
|
|
4
|
-
|
|
5
|
-
## Scope
|
|
6
|
-
|
|
7
|
-
Preserved the setup lifecycle regression where setup completed but the current session still did not have Freeflow's workflow context loaded.
|
|
8
|
-
|
|
9
|
-
Owned paths:
|
|
10
|
-
|
|
11
|
-
- `skills/setup-freeflow/`
|
|
12
|
-
- `skills/setup-freeflow/references/activation-contract.md`
|
|
13
|
-
- `skills/setup-freeflow/references/host-setup.md`
|
|
14
|
-
- `evals/prompts/stp-010.txt`
|
|
15
|
-
- `evals/registries/fixture-evals.json`
|
|
16
|
-
|
|
17
|
-
Setup still does not create repo-local hooks, docs inventories, handoffs, state files, `.codex/rules`, or `CONTEXT.md`.
|
|
18
|
-
|
|
19
|
-
## Skill Change
|
|
20
|
-
|
|
21
|
-
After successful setup verification, `setup-freeflow` now reads:
|
|
22
|
-
|
|
23
|
-
- `../workflow/SKILL.md`
|
|
24
|
-
- `../workflow/references/workflow-map.md`
|
|
25
|
-
|
|
26
|
-
It may only say workflow context is loaded for the current session after both files were read successfully.
|
|
27
|
-
|
|
28
|
-
## Eval Added
|
|
29
|
-
|
|
30
|
-
Added:
|
|
31
|
-
|
|
32
|
-
- `STP-010`: setup loads workflow for current session.
|
|
33
|
-
|
|
34
|
-
Expected behavior:
|
|
35
|
-
|
|
36
|
-
- set up Codex with only `AGENTS.md` and `.freeflow/config.json`
|
|
37
|
-
- verify config shape, activation count, and absence of extra setup artifacts
|
|
38
|
-
- read the workflow skill and workflow map before the final response
|
|
39
|
-
- report workflow context loaded only after those reads
|
|
40
|
-
|
|
41
|
-
## Results
|
|
42
|
-
|
|
43
|
-
`STP-010` old runtime baseline: fail.
|
|
44
|
-
|
|
45
|
-
- Baseline used setup runtime files from pre-change `8627ac8d8835fdcd770d6d56ea0420b59f0b2914`.
|
|
46
|
-
- Created the required `.freeflow/config.json` and `AGENTS.md`.
|
|
47
|
-
- Failed the workflow-loaded output check because it did not report that workflow context was loaded for the current session.
|
|
48
|
-
|
|
49
|
-
`STP-010` current skill: pass.
|
|
50
|
-
|
|
51
|
-
- Created only `.freeflow/config.json` and `AGENTS.md`.
|
|
52
|
-
- Verified config shape, single activation block, no `.codex/rules`, no Claude setup, no hooks, no docs, no `CONTEXT.md`, and no state files.
|
|
53
|
-
- Read the workflow skill and workflow map.
|
|
54
|
-
- Reported: "Workflow skill and workflow map were read, so workflow context is loaded for this session."
|
|
55
|
-
|
|
56
|
-
## Evidence
|
|
57
|
-
|
|
58
|
-
Saved final runs:
|
|
59
|
-
|
|
60
|
-
- `evals/runs/setup-8/stp-010-baseline-old-runtime-output.md`
|
|
61
|
-
- `evals/runs/setup-8/stp-010-with-skill-output.md`
|
|
62
|
-
|
|
63
|
-
Key diffs:
|
|
64
|
-
|
|
65
|
-
- `evals/runs/setup-8/stp-010-baseline-old-runtime-output.diff`
|
|
66
|
-
- `evals/runs/setup-8/stp-010-with-skill-output.diff`
|
|
67
|
-
|
|
68
|
-
Mechanical grades:
|
|
69
|
-
|
|
70
|
-
```sh
|
|
71
|
-
evals/scripts/grade-fixture-eval.sh STP-010 --output evals/runs/setup-8/stp-010-baseline-old-runtime-output.md
|
|
72
|
-
# fail: setup-required-files pass; workflow-loaded-output fail
|
|
73
|
-
|
|
74
|
-
evals/scripts/grade-fixture-eval.sh STP-010 --output evals/runs/setup-8/stp-010-with-skill-output.md
|
|
75
|
-
# pass: setup-required-files pass; workflow-loaded-output pass
|
|
76
|
-
```
|
|
77
|
-
|
|
78
|
-
## Verification
|
|
79
|
-
|
|
80
|
-
Commands:
|
|
81
|
-
|
|
82
|
-
```sh
|
|
83
|
-
evals/scripts/check-activation-contract.sh
|
|
84
|
-
evals/scripts/check-runtime-context-hook.sh
|
|
85
|
-
evals/scripts/validate-release-metadata.sh --mode prepublish --release-version 0.1.0
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
Nested `codex exec` fixture runs required escalation outside the sandbox because the in-process app-server client is blocked by sandbox permissions.
|
|
@@ -1,115 +0,0 @@
|
|
|
1
|
-
# Setup Freeflow Eval Report - Iteration 5
|
|
2
|
-
|
|
3
|
-
Date: 2026-06-20
|
|
4
|
-
|
|
5
|
-
## Scope
|
|
6
|
-
|
|
7
|
-
Added optional output-router repo config setup to `setup-freeflow` while preserving the minimal setup contract.
|
|
8
|
-
|
|
9
|
-
Owned paths:
|
|
10
|
-
|
|
11
|
-
- `skills/setup-freeflow/SKILL.md`
|
|
12
|
-
- `skills/setup-freeflow/references/activation-contract.md`
|
|
13
|
-
- `skills/setup-freeflow/references/host-setup.md`
|
|
14
|
-
- `skills/setup-freeflow/references/output-router-setup.md`
|
|
15
|
-
- `evals/prompts/stp-011.txt`
|
|
16
|
-
- `evals/registries/fixture-evals.json`
|
|
17
|
-
|
|
18
|
-
## Skill Change
|
|
19
|
-
|
|
20
|
-
`setup-freeflow` now owns optional `outputRouter` config when explicitly requested.
|
|
21
|
-
|
|
22
|
-
Minimal setup still writes only `defaultMode`. The skill now says:
|
|
23
|
-
|
|
24
|
-
- do not ask every setup user about router config;
|
|
25
|
-
- do not add empty `outputRouter` or dump defaults;
|
|
26
|
-
- ask if the user requests router setup without naming config knobs;
|
|
27
|
-
- never enable native safety-net routing by default;
|
|
28
|
-
- do not create a separate `setup-output-router` skill.
|
|
29
|
-
|
|
30
|
-
## Eval Added
|
|
31
|
-
|
|
32
|
-
Added:
|
|
33
|
-
|
|
34
|
-
- `STP-011`: setup optional output-router config.
|
|
35
|
-
|
|
36
|
-
Expected behavior:
|
|
37
|
-
|
|
38
|
-
- set up Codex with exactly one Freeflow block in `AGENTS.md`;
|
|
39
|
-
- create `.freeflow/config.json` with `defaultMode: "workflow"` and only explicitly requested `outputRouter` keys;
|
|
40
|
-
- set `postToolRouting` to `off`, `largeOutputLines` to `2000`, and `generatedPaths` to `graphify-out/**` and `dist/**`;
|
|
41
|
-
- avoid safety-net/strict enablement, hooks, docs, state files, `.codex/rules`, and any `setup-output-router` skill.
|
|
42
|
-
|
|
43
|
-
## Results
|
|
44
|
-
|
|
45
|
-
`STP-011` current skill: pass.
|
|
46
|
-
|
|
47
|
-
Saved output reports:
|
|
48
|
-
|
|
49
|
-
- `AGENTS.md` received the canonical Freeflow Codex block.
|
|
50
|
-
- `.freeflow/config.json` contained `defaultMode: "workflow"` plus only the requested `outputRouter` keys.
|
|
51
|
-
- Verification reported JSON parsing, `postToolRouting: "off"`, generated paths, `largeOutputLines: 2000`, one activation block, and no extra setup artifacts.
|
|
52
|
-
|
|
53
|
-
Observed config from the fixture run:
|
|
54
|
-
|
|
55
|
-
```json
|
|
56
|
-
{
|
|
57
|
-
"defaultMode": "workflow",
|
|
58
|
-
"outputRouter": {
|
|
59
|
-
"postToolRouting": "off",
|
|
60
|
-
"largeOutputLines": 2000,
|
|
61
|
-
"generatedPaths": [
|
|
62
|
-
"graphify-out/**",
|
|
63
|
-
"dist/**"
|
|
64
|
-
]
|
|
65
|
-
}
|
|
66
|
-
}
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
## Evidence
|
|
70
|
-
|
|
71
|
-
Saved final run:
|
|
72
|
-
|
|
73
|
-
- `evals/runs/manual/stp-011-with-skill-codex-output.md`
|
|
74
|
-
|
|
75
|
-
Mechanical grade:
|
|
76
|
-
|
|
77
|
-
```sh
|
|
78
|
-
evals/scripts/grade-fixture-eval.sh STP-011 --output evals/runs/manual/stp-011-with-skill-codex-output.md
|
|
79
|
-
# pass: setup-router-required-files pass; setup-router-final-output pass
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
Dry-run setup resolution:
|
|
83
|
-
|
|
84
|
-
```sh
|
|
85
|
-
evals/scripts/run-fixture-eval.sh STP-011 --dry-run
|
|
86
|
-
# resolved fixture, prompt, output path, and codex adapter
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
## Review Fixes
|
|
90
|
-
|
|
91
|
-
Focused review found two blockers after the initial skill/config update:
|
|
92
|
-
|
|
93
|
-
- `STP-011` was present in `fixture-evals.json` but missing from `skill-evidence.json` under `setup-freeflow`.
|
|
94
|
-
- `hooks/freeflow-runtime-context.mjs` still treated config as valid only when `.freeflow/config.json` contained exactly one top-level key, so optional `outputRouter` config made setup look partial.
|
|
95
|
-
|
|
96
|
-
Fixes:
|
|
97
|
-
|
|
98
|
-
- Registered `STP-011` under `setup-freeflow` skill evidence.
|
|
99
|
-
- Updated the runtime context hook to accept minimal config or `defaultMode` plus object-shaped `outputRouter`.
|
|
100
|
-
- Extended `check-runtime-context-hook.sh` to cover optional `outputRouter` config without marking setup partial or invalid.
|
|
101
|
-
|
|
102
|
-
## Verification
|
|
103
|
-
|
|
104
|
-
Commands:
|
|
105
|
-
|
|
106
|
-
```sh
|
|
107
|
-
jq empty evals/registries/fixture-evals.json
|
|
108
|
-
jq empty evals/registries/skill-evidence.json
|
|
109
|
-
evals/scripts/check-activation-contract.sh
|
|
110
|
-
evals/scripts/check-runtime-context-hook.sh
|
|
111
|
-
evals/scripts/run-fixture-eval.sh STP-011 --dry-run
|
|
112
|
-
evals/scripts/run-fixture-eval.sh STP-011 --with-skill
|
|
113
|
-
evals/scripts/grade-fixture-eval.sh STP-011 --output evals/runs/manual/stp-011-with-skill-codex-output.md
|
|
114
|
-
evals/scripts/skill-evidence.sh setup-freeflow
|
|
115
|
-
```
|
|
@@ -1,81 +0,0 @@
|
|
|
1
|
-
# Skill Authoring Eval 1
|
|
2
|
-
|
|
3
|
-
Date: 2026-05-25
|
|
4
|
-
|
|
5
|
-
Scope:
|
|
6
|
-
|
|
7
|
-
- `write-skill`
|
|
8
|
-
- `evaluate-skill`
|
|
9
|
-
|
|
10
|
-
## Evals Added
|
|
11
|
-
|
|
12
|
-
Added fixture `evals/fixtures/tiny-skill-dev-app`.
|
|
13
|
-
|
|
14
|
-
Added registry entries:
|
|
15
|
-
|
|
16
|
-
- `WSK-001`: create one compact release-note skill from messy requirements.
|
|
17
|
-
- `ESK-001`: improve a weak review skill from a preserved failure under shortcut pressure.
|
|
18
|
-
|
|
19
|
-
## WSK-001 Result
|
|
20
|
-
|
|
21
|
-
Baseline: pass.
|
|
22
|
-
|
|
23
|
-
- Created one `skills/release-notes/SKILL.md`.
|
|
24
|
-
- No extra skill docs/resources.
|
|
25
|
-
- Stayed under 100 lines.
|
|
26
|
-
|
|
27
|
-
With `write-skill`: pass.
|
|
28
|
-
|
|
29
|
-
- Created one `skills/release-notes/SKILL.md`.
|
|
30
|
-
- No extra skill docs/resources.
|
|
31
|
-
- Stayed under 100 lines.
|
|
32
|
-
|
|
33
|
-
Assessment: weak differentiator. Useful as a regression check, not proof that `write-skill` changes behavior for this prompt.
|
|
34
|
-
|
|
35
|
-
Artifacts:
|
|
36
|
-
|
|
37
|
-
- `evals/runs/wsk-001-baseline-output.md`
|
|
38
|
-
- `evals/runs/wsk-001-baseline-output.diff`
|
|
39
|
-
- `evals/runs/wsk-001-with-skill-output.md`
|
|
40
|
-
- `evals/runs/wsk-001-with-skill-output.diff`
|
|
41
|
-
|
|
42
|
-
## ESK-001 Result
|
|
43
|
-
|
|
44
|
-
Baseline: fail.
|
|
45
|
-
|
|
46
|
-
- Patched `skills/review-pr/SKILL.md` directly.
|
|
47
|
-
- Did not create or update an eval artifact.
|
|
48
|
-
- Final response explicitly said no harness was added.
|
|
49
|
-
|
|
50
|
-
Initial with-skill: fail.
|
|
51
|
-
|
|
52
|
-
- Loaded `evaluate-skill` and `write-skill`.
|
|
53
|
-
- Still patched the skill directly and skipped the eval artifact.
|
|
54
|
-
|
|
55
|
-
Skill revision:
|
|
56
|
-
|
|
57
|
-
- Tightened `evaluate-skill` with a hard stop: when improving from a preserved failure, create or update the smallest repeatable eval artifact before editing the skill.
|
|
58
|
-
- Clarified that a failure report is evidence, not the eval artifact.
|
|
59
|
-
- Clarified shortcut pressure such as "quick wording fix", "patch directly", or "no harness" does not skip this unless the user explicitly forbids eval artifacts or file changes.
|
|
60
|
-
|
|
61
|
-
Fixed with-skill: pass.
|
|
62
|
-
|
|
63
|
-
- Updated the existing review prompt with pass criteria before changing the skill.
|
|
64
|
-
- Then tightened `skills/review-pr/SKILL.md`.
|
|
65
|
-
- Kept the revised skill short.
|
|
66
|
-
|
|
67
|
-
Artifacts:
|
|
68
|
-
|
|
69
|
-
- `evals/runs/esk-001-baseline-output.md`
|
|
70
|
-
- `evals/runs/esk-001-baseline-output.diff`
|
|
71
|
-
- `evals/runs/esk-001-with-skill-output.md`
|
|
72
|
-
- `evals/runs/esk-001-with-skill-output.diff`
|
|
73
|
-
- `evals/runs/esk-001-with-skill-fixed-output.md`
|
|
74
|
-
- `evals/runs/esk-001-with-skill-fixed-output.diff`
|
|
75
|
-
- `evals/runs/esk-001-with-skill-fixed-2-output.md`
|
|
76
|
-
- `evals/runs/esk-001-with-skill-fixed-2-output.diff`
|
|
77
|
-
|
|
78
|
-
## Follow-Up
|
|
79
|
-
|
|
80
|
-
Add a stronger `write-skill` eval later if a real failure appears. Current `WSK-001` is acceptable only as a no-regression fixture.
|
|
81
|
-
|