@hassangameryt/freeflow 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +4 -4
- package/.claude-plugin/plugin.json +6 -2
- package/.codex-plugin/plugin.json +9 -9
- package/README.md +190 -127
- package/assets/feedback-loop.png +0 -0
- package/assets/freeflow_workflow_map.png +0 -0
- package/assets/workflow.png +0 -0
- package/{skills → capabilities}/output-router/SKILL.md +21 -17
- package/{skills → capabilities}/output-router/references/safety-policy.md +1 -1
- package/capabilities/output-router/references/setup.md +110 -0
- package/command-surface.json +62 -7
- package/hooks/freeflow-runtime-context.mjs +572 -154
- package/hooks/hooks.json +26 -1
- package/hooks/tests/check-runtime-context-hook.sh +375 -0
- package/package.json +24 -24
- package/pi-extension/dist/index.js +264 -80
- package/pi-extension/dist/integrations/mcp-config.js +29 -0
- package/pi-extension/dist/output-router/host-producer-identification.js +232 -0
- package/pi-extension/dist/output-router/native-safety-net.js +216 -0
- package/pi-extension/dist/output-router/observed-tool-routing.js +122 -0
- package/pi-extension/dist/output-router/renderers.js +1023 -0
- package/pi-extension/dist/output-router/router-tools.js +554 -0
- package/pi-extension/dist/output-router/schemas.js +383 -0
- package/pi-extension/dist/output-router/status.js +744 -0
- package/pi-extension/dist/output-router/utils.js +502 -0
- package/pi-extension/dist/runtime/runtime-context.js +798 -0
- package/pi-extension/dist/settings/settings-tui.js +314 -0
- package/pi-extension/dist/settings/settings-ui.js +1487 -0
- package/pi-extension/freeflow/index.js +1 -0
- package/pi-extension/src/index.ts +230 -56
- package/pi-extension/src/{host-producer-identification.ts → output-router/host-producer-identification.ts} +37 -7
- package/pi-extension/src/{native-safety-net.ts → output-router/native-safety-net.ts} +8 -12
- package/pi-extension/src/{observed-tool-routing.ts → output-router/observed-tool-routing.ts} +19 -11
- package/pi-extension/src/{renderers.ts → output-router/renderers.ts} +246 -55
- package/pi-extension/src/{router-tools.ts → output-router/router-tools.ts} +130 -29
- package/pi-extension/src/{schemas.ts → output-router/schemas.ts} +171 -43
- package/pi-extension/src/output-router/status.ts +791 -0
- package/pi-extension/src/{utils.ts → output-router/utils.ts} +80 -18
- package/pi-extension/src/runtime/runtime-context.ts +895 -0
- package/pi-extension/src/settings/settings-tui.ts +408 -0
- package/pi-extension/src/settings/settings-ui.ts +1730 -0
- package/pi-extension/tests/integration/pi-extension.test.js +4020 -0
- package/pi-extension/tests/output-router/observed-producers.test.js +132 -0
- package/pi-extension/tests/output-router/search.test.js +722 -0
- package/pi-extension/tests/settings/settings-tui.test.js +243 -0
- package/router/dist/benchmarks/benchmark-harness.d.ts +16 -13
- package/router/dist/benchmarks/benchmark-harness.js +62 -62
- package/router/dist/benchmarks/benchmarks.d.ts +80 -73
- package/router/dist/benchmarks/benchmarks.js +860 -761
- package/router/dist/benchmarks/codex-qa-benchmarks.d.ts +71 -67
- package/router/dist/benchmarks/codex-qa-benchmarks.js +429 -390
- package/router/dist/benchmarks/command-benchmarks.d.ts +107 -92
- package/router/dist/benchmarks/command-benchmarks.js +616 -571
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.d.ts +64 -56
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.js +679 -613
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.d.ts +103 -95
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.js +1188 -1103
- package/router/dist/benchmarks/context-saving-baseline.d.ts +47 -41
- package/router/dist/benchmarks/context-saving-baseline.js +405 -374
- package/router/dist/benchmarks/index-benchmarks.d.ts +74 -69
- package/router/dist/benchmarks/index-benchmarks.js +761 -661
- package/router/dist/benchmarks/storage-policy-benchmarks.d.ts +80 -69
- package/router/dist/benchmarks/storage-policy-benchmarks.js +549 -469
- package/router/dist/benchmarks/vault-index-benchmarks.d.ts +43 -34
- package/router/dist/benchmarks/vault-index-benchmarks.js +294 -229
- package/router/dist/config/config.d.ts +62 -55
- package/router/dist/config/config.js +552 -373
- package/router/dist/config/router-contract.d.ts +2 -2
- package/router/dist/config/router-contract.js +38 -38
- package/router/dist/config/schema.d.ts +19 -10
- package/router/dist/config/schema.js +586 -538
- package/router/dist/config/types.d.ts +413 -361
- package/router/dist/config/types.js +19 -8
- package/router/dist/evidence/bounded-evidence.d.ts +26 -26
- package/router/dist/evidence/bounded-evidence.js +265 -252
- package/router/dist/evidence/evidence-range-selector.d.ts +16 -14
- package/router/dist/evidence/evidence-range-selector.js +192 -191
- package/router/dist/evidence/evidence-search.d.ts +18 -16
- package/router/dist/evidence/evidence-search.js +401 -377
- package/router/dist/evidence/evidence.d.ts +17 -17
- package/router/dist/evidence/evidence.js +163 -159
- package/router/dist/evidence/failure-contracts.d.ts +25 -15
- package/router/dist/evidence/failure-contracts.js +132 -126
- package/router/dist/evidence/line-ranges.d.ts +17 -15
- package/router/dist/evidence/line-ranges.js +18 -18
- package/router/dist/experiments/local-index.d.ts +47 -41
- package/router/dist/experiments/local-index.js +477 -458
- package/router/dist/local/local-traversal.d.ts +18 -0
- package/router/dist/local/local-traversal.js +334 -0
- package/router/dist/processing/engine.d.ts +94 -70
- package/router/dist/processing/engine.js +462 -364
- package/router/dist/processing/reducers.d.ts +208 -194
- package/router/dist/processing/reducers.js +1434 -1256
- package/router/dist/processing/renderers.d.ts +24 -18
- package/router/dist/processing/renderers.js +161 -152
- package/router/dist/processing/scripts.d.ts +80 -67
- package/router/dist/processing/scripts.js +343 -290
- package/router/dist/repo/repo-traversal.d.ts +9 -9
- package/router/dist/repo/repo-traversal.js +205 -202
- package/router/dist/routing/observed-reducers.d.ts +25 -19
- package/router/dist/routing/observed-reducers.js +559 -545
- package/router/dist/routing/observed-routing.d.ts +23 -14
- package/router/dist/routing/observed-routing.js +319 -318
- package/router/dist/routing/parsers.d.ts +10 -10
- package/router/dist/routing/parsers.js +454 -406
- package/router/dist/routing/run-filters.d.ts +45 -41
- package/router/dist/routing/run-filters.js +215 -207
- package/router/dist/routing/run-reducers.d.ts +45 -26
- package/router/dist/routing/run-reducers.js +102 -90
- package/router/dist/sandbox/adapter-roots.d.ts +34 -26
- package/router/dist/sandbox/adapter-roots.js +43 -41
- package/router/dist/sandbox/eryx-python-adapter.d.ts +9 -5
- package/router/dist/sandbox/eryx-python-adapter.js +659 -588
- package/router/dist/sandbox/jq-wasm-adapter.d.ts +3 -3
- package/router/dist/sandbox/jq-wasm-adapter.js +319 -311
- package/router/dist/sandbox/quickjs-wasi-adapter.d.ts +9 -5
- package/router/dist/sandbox/quickjs-wasi-adapter.js +336 -333
- package/router/dist/sandbox/script-sandbox.d.ts +105 -84
- package/router/dist/sandbox/script-sandbox.js +294 -260
- package/router/dist/setup/script-transform-adapters.d.ts +23 -19
- package/router/dist/setup/script-transform-adapters.js +223 -204
- package/router/dist/tools/batch.d.ts +27 -16
- package/router/dist/tools/batch.js +493 -416
- package/router/dist/tools/run.d.ts +47 -35
- package/router/dist/tools/run.js +1172 -1087
- package/router/dist/tools/search.d.ts +53 -31
- package/router/dist/tools/search.js +1224 -1065
- package/router/dist/transform/engine.d.ts +121 -96
- package/router/dist/transform/engine.js +1853 -1736
- package/router/dist/vault/vault-index.d.ts +97 -78
- package/router/dist/vault/vault-index.js +347 -342
- package/router/dist/vault/vault.d.ts +109 -74
- package/router/dist/vault/vault.js +528 -533
- package/runtime/interaction-contract.md +16 -0
- package/skills/bypass/SKILL.md +52 -38
- package/skills/commit-work/SKILL.md +54 -46
- package/skills/commit-work/references/staging-decisions.md +46 -24
- package/skills/decision-gate/SKILL.md +48 -0
- package/skills/design-for-depth/SKILL.md +75 -59
- package/skills/design-for-depth/references/design-pressure-signals.md +56 -7
- package/skills/design-for-depth/references/interface-design-loop.md +156 -0
- package/skills/design-for-depth/references/software-design-philosophy.md +32 -3
- package/skills/diagnose-failure/SKILL.md +65 -60
- package/skills/diagnose-failure/references/diagnostic-loop-catalog.md +36 -0
- package/skills/diagnose-failure/references/flaky-and-performance.md +30 -18
- package/skills/discuss/SKILL.md +121 -0
- package/skills/discuss/references/checkpoints.md +66 -0
- package/skills/evaluate-skill/SKILL.md +62 -53
- package/skills/evaluate-skill/references/definition-schema.md +345 -0
- package/skills/evaluate-skill/references/evaluation-design.md +86 -0
- package/skills/evaluate-skill/references/execution-and-evidence.md +85 -0
- package/skills/evaluate-skill/references/review-and-revision.md +51 -0
- package/skills/evaluate-skill/scripts/lib/definitions.mjs +296 -0
- package/skills/evaluate-skill/scripts/lib/evidence.mjs +46 -0
- package/skills/evaluate-skill/scripts/lib/grade-comparison.mjs +60 -0
- package/skills/evaluate-skill/scripts/lib/grade-interaction.mjs +148 -0
- package/skills/evaluate-skill/scripts/lib/grade-support.mjs +126 -0
- package/skills/evaluate-skill/scripts/lib/grade-workspace.mjs +192 -0
- package/skills/evaluate-skill/scripts/lib/grade.mjs +56 -0
- package/skills/evaluate-skill/scripts/lib/pi.mjs +489 -0
- package/skills/evaluate-skill/scripts/lib/process.mjs +79 -0
- package/skills/evaluate-skill/scripts/lib/rpc.mjs +444 -0
- package/skills/evaluate-skill/scripts/lib/runner.mjs +346 -0
- package/skills/evaluate-skill/scripts/lib/sandbox.mjs +409 -0
- package/skills/evaluate-skill/scripts/lib/view.mjs +337 -0
- package/skills/evaluate-skill/scripts/pi-guard.mjs +164 -0
- package/skills/evaluate-skill/scripts/skill-eval.mjs +101 -0
- package/skills/execute-work/SKILL.md +99 -0
- package/skills/execute-work/references/code-practices.md +78 -0
- package/skills/execute-work/references/execution-loop.md +77 -0
- package/skills/finish-branch/SKILL.md +117 -0
- package/skills/finish-branch/references/integration-options.md +80 -0
- package/skills/handoff/SKILL.md +65 -39
- package/skills/handoff/references/templates.md +104 -27
- package/skills/launch-work/SKILL.md +116 -0
- package/skills/launch-work/references/launch-readiness.md +85 -0
- package/skills/migration-work/SKILL.md +115 -0
- package/skills/migration-work/references/migration-lifecycle.md +82 -0
- package/skills/mode-contract/SKILL.md +72 -52
- package/skills/release-work/SKILL.md +118 -0
- package/skills/release-work/references/release-evidence.md +78 -0
- package/skills/review-artifact/SKILL.md +77 -69
- package/skills/review-artifact/references/reviewer-prompt.md +133 -42
- package/skills/review-work/SKILL.md +82 -57
- package/skills/review-work/references/reviewer-prompt.md +97 -60
- package/skills/review-work/references/security-risk-lens.md +67 -0
- package/skills/setup-freeflow/SKILL.md +54 -68
- package/skills/setup-freeflow/references/activation-contract.md +84 -69
- package/skills/setup-freeflow/references/host-setup.md +50 -17
- package/skills/simplify-code/SKILL.md +80 -0
- package/skills/simplify-code/references/simplification-patterns.md +70 -0
- package/skills/tdd/SKILL.md +96 -0
- package/skills/tdd/references/test-design.md +96 -0
- package/skills/track-work/SKILL.md +84 -0
- package/skills/track-work/references/working-record-schema.md +378 -0
- package/skills/verify-work/SKILL.md +59 -38
- package/skills/verify-work/references/browser-runtime-evidence.md +55 -0
- package/skills/verify-work/references/integration-evidence.md +51 -0
- package/skills/verify-work/references/performance-evidence.md +57 -0
- package/skills/workflow/SKILL.md +53 -59
- package/skills/workflow/references/domain-skill-composition.md +57 -0
- package/skills/workflow/references/workflow-loop.md +92 -0
- package/skills/write-plan/SKILL.md +85 -58
- package/skills/write-plan/references/plan-shapes.md +132 -46
- package/skills/write-skill/SKILL.md +82 -51
- package/skills/write-skill/references/activation-boundaries.md +62 -0
- package/skills/write-skill/references/agent-first-instructions.md +87 -0
- package/skills/write-skill/references/progressive-disclosure.md +63 -0
- package/skills/write-skill/scripts/lib/skill-author.mjs +653 -0
- package/skills/write-skill/scripts/skill-author.mjs +104 -0
- package/skills/write-spec/SKILL.md +65 -54
- package/skills/write-spec/references/artifact-standards.md +60 -41
- package/skills/write-spec/references/decision-records.md +77 -0
- package/skills/write-spec/references/spec-shapes.md +119 -60
- package/evals/README.md +0 -32
- package/evals/fixtures/copy-conventions.md +0 -31
- package/evals/fixtures/output-router/README.md +0 -12
- package/evals/fixtures/output-router/failed-command-output.txt +0 -9
- package/evals/fixtures/output-router/large-router-manual.md +0 -63
- package/evals/fixtures/output-router/native-large-output.txt +0 -40
- package/evals/fixtures/output-router/noisy-test-output.txt +0 -30
- package/evals/fixtures/output-router/verification-output.txt +0 -8
- package/evals/fixtures/tiny-artifact-review-app/README.md +0 -5
- package/evals/fixtures/tiny-artifact-review-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md +0 -21
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/failed-payment-grace-period.md +0 -24
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/immediate-downgrade.md +0 -17
- package/evals/fixtures/tiny-artifact-review-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-commit-ambiguous-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-ambiguous-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-ambiguous-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-ambiguous-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-ambiguous-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh +0 -14
- package/evals/fixtures/tiny-commit-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-everything-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-everything-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-everything-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-everything-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-everything-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-everything-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-staged-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-staged-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-staged-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-staged-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-staged-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-staged-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-staged-mixed-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/billing-policy.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/release-notes.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-dashboard-stale-app/README.md +0 -11
- package/evals/fixtures/tiny-dashboard-stale-app/src/api-client.ts +0 -13
- package/evals/fixtures/tiny-dashboard-stale-app/src/dashboard-data.ts +0 -19
- package/evals/fixtures/tiny-dashboard-stale-app/tests/dashboard-data.test.md +0 -9
- package/evals/fixtures/tiny-decision-destination-app/README.md +0 -6
- package/evals/fixtures/tiny-decision-destination-app/src/risk.ts +0 -8
- package/evals/fixtures/tiny-decision-destination-app/tests/risk.test.md +0 -4
- package/evals/fixtures/tiny-design-depth-app/README.md +0 -15
- package/evals/fixtures/tiny-design-depth-app/src/routes.ts +0 -25
- package/evals/fixtures/tiny-design-depth-app/src/sendEmail.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/src/sendSms.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/src/telemetry.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/tests/routes.test.md +0 -12
- package/evals/fixtures/tiny-eval-method-app/README.md +0 -8
- package/evals/fixtures/tiny-eval-method-app/evals/reports/review-pr-eval-result.md +0 -5
- package/evals/fixtures/tiny-eval-method-app/evals/reports/triage-question-failure.md +0 -17
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.diff +0 -11
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.md +0 -3
- package/evals/fixtures/tiny-eval-method-app/skills/review-pr/SKILL.md +0 -10
- package/evals/fixtures/tiny-eval-method-app/skills/triage-question/SKILL.md +0 -10
- package/evals/fixtures/tiny-execute-plan-app/README.md +0 -5
- package/evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh +0 -5
- package/evals/fixtures/tiny-execute-plan-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/immediate-downgrade-plan.md +0 -11
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md +0 -15
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/unverified-billing-plan.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/failed-payment-grace-period.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-execute-plan-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-execute-plan-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-execute-plan-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-execute-review-loop-app/README.md +0 -5
- package/evals/fixtures/tiny-execute-review-loop-app/docs/billing-policy.md +0 -9
- package/evals/fixtures/tiny-execute-review-loop-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/previous-agent-summary.md +0 -15
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/review-pass-1-non-pass.md +0 -11
- package/evals/fixtures/tiny-execute-review-loop-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-execute-review-loop-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-execute-review-loop-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-execute-review-loop-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-execute-review-loop-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-flaky-session-app/README.md +0 -10
- package/evals/fixtures/tiny-flaky-session-app/src/session-timeout.ts +0 -13
- package/evals/fixtures/tiny-flaky-session-app/tests/session-timeout.test.md +0 -8
- package/evals/fixtures/tiny-handoff-app/README.md +0 -5
- package/evals/fixtures/tiny-handoff-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-handoff-app/docs/handoffs/old-agent-handoff.md +0 -8
- package/evals/fixtures/tiny-handoff-app/docs/plans/billing-retry-plan.md +0 -11
- package/evals/fixtures/tiny-handoff-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-handoff-report-app/README.md +0 -3
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.diff +0 -1
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.md +0 -1
- package/evals/fixtures/tiny-handoff-report-app/evals/scores/hof-003-score.txt +0 -4
- package/evals/fixtures/tiny-plan-app/README.md +0 -5
- package/evals/fixtures/tiny-plan-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-plan-app/docs/specs/failed-payment-grace-period.md +0 -25
- package/evals/fixtures/tiny-plan-app/src/billing.ts +0 -17
- package/evals/fixtures/tiny-plan-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-plan-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-plan-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/README.md +0 -3
- package/evals/fixtures/tiny-plan-strict-api-app/docs/billing-policy.md +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/docs/specs/billing-webhook-api.md +0 -25
- package/evals/fixtures/tiny-plan-strict-api-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/src/webhooks.ts +0 -3
- package/evals/fixtures/tiny-plan-strict-api-app/tests/billing-webhook.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/.freeflow/config.json +0 -3
- package/evals/fixtures/tiny-post-setup-invalid-config-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-post-setup-missing-config-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-post-setup-source-conflict-app/.freeflow/config.json +0 -3
- package/evals/fixtures/tiny-post-setup-source-conflict-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-report-performance-app/README.md +0 -11
- package/evals/fixtures/tiny-report-performance-app/docs/performance.md +0 -10
- package/evals/fixtures/tiny-report-performance-app/src/report-summary.ts +0 -29
- package/evals/fixtures/tiny-report-performance-app/tests/report-performance.test.md +0 -8
- package/evals/fixtures/tiny-review-request-app/README.md +0 -3
- package/evals/fixtures/tiny-review-request-app/docs/billing-policy.md +0 -5
- package/evals/fixtures/tiny-review-request-app/docs/plans/billing-webhook-api.md +0 -19
- package/evals/fixtures/tiny-review-request-app/docs/review/previous-agent-summary.md +0 -20
- package/evals/fixtures/tiny-review-request-app/docs/specs/billing-webhook-api.md +0 -25
- package/evals/fixtures/tiny-review-request-app/src/billing.ts +0 -11
- package/evals/fixtures/tiny-review-request-app/src/webhooks.ts +0 -19
- package/evals/fixtures/tiny-review-request-app/tests/billing-webhook.test.md +0 -8
- package/evals/fixtures/tiny-review-work-app/README.md +0 -5
- package/evals/fixtures/tiny-review-work-app/docs/billing-policy.md +0 -9
- package/evals/fixtures/tiny-review-work-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-review-work-app/docs/review/previous-agent-summary.md +0 -15
- package/evals/fixtures/tiny-review-work-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-review-work-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-review-work-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-review-work-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-review-work-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-saas-app/README.md +0 -16
- package/evals/fixtures/tiny-saas-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-saas-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-saas-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-saas-app/docs/specs/settings-title.md +0 -13
- package/evals/fixtures/tiny-saas-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-saas-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-saas-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-saas-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-saas-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-saas-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-saas-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-setup-agents-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-both-app/AGENTS.md +0 -5
- package/evals/fixtures/tiny-setup-both-app/CLAUDE.md +0 -5
- package/evals/fixtures/tiny-setup-claude-app/CLAUDE.md +0 -6
- package/evals/fixtures/tiny-setup-codex-rules-app/.codex/rules/shell.rules +0 -5
- package/evals/fixtures/tiny-setup-codex-rules-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-conflict-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-existing-block-app/AGENTS.md +0 -14
- package/evals/fixtures/tiny-setup-four-rules-app/CLAUDE.md +0 -6
- package/evals/fixtures/tiny-skill-dev-app/README.md +0 -10
- package/evals/fixtures/tiny-skill-dev-app/evals/prompts/review-pr-001.txt +0 -8
- package/evals/fixtures/tiny-skill-dev-app/evals/reports/review-pr-failure.md +0 -17
- package/evals/fixtures/tiny-skill-dev-app/skills/review-pr/SKILL.md +0 -13
- package/evals/fixtures/tiny-team-app/README.md +0 -5
- package/evals/fixtures/tiny-team-app/docs/product.md +0 -6
- package/evals/fixtures/tiny-team-app/src/teams.ts +0 -9
- package/evals/fixtures/tiny-team-app/src/users.ts +0 -5
- package/evals/fixtures/tiny-team-app/tests/teams.test.md +0 -7
- package/evals/fixtures/tiny-verify-app/README.md +0 -5
- package/evals/fixtures/tiny-verify-app/src/settings-nav.ts +0 -1
- package/evals/fixtures/tiny-verify-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-verify-app/tests/settings.test.md +0 -7
- package/evals/prompts/aon-001.txt +0 -2
- package/evals/prompts/byp-001.txt +0 -2
- package/evals/prompts/byp-002.txt +0 -2
- package/evals/prompts/cap-001.txt +0 -3
- package/evals/prompts/cap-002.txt +0 -3
- package/evals/prompts/cmd-001.txt +0 -3
- package/evals/prompts/cmd-002.txt +0 -3
- package/evals/prompts/cmd-003.txt +0 -6
- package/evals/prompts/cmd-004.txt +0 -5
- package/evals/prompts/cmd-005.txt +0 -5
- package/evals/prompts/cmd-006.txt +0 -5
- package/evals/prompts/cmd-007.txt +0 -5
- package/evals/prompts/cmd-008.txt +0 -5
- package/evals/prompts/cmd-009.txt +0 -8
- package/evals/prompts/cmd-010.txt +0 -5
- package/evals/prompts/cmd-011.txt +0 -5
- package/evals/prompts/cmd-012.txt +0 -5
- package/evals/prompts/cmd-013.txt +0 -5
- package/evals/prompts/cmd-014.txt +0 -7
- package/evals/prompts/cmd-015.txt +0 -7
- package/evals/prompts/cmd-016.txt +0 -5
- package/evals/prompts/cmt-001.txt +0 -1
- package/evals/prompts/cmt-002.txt +0 -1
- package/evals/prompts/cmt-003.txt +0 -1
- package/evals/prompts/cmt-004.txt +0 -1
- package/evals/prompts/cmt-005.txt +0 -1
- package/evals/prompts/dfd-001.txt +0 -3
- package/evals/prompts/dia-001.txt +0 -3
- package/evals/prompts/dia-002.txt +0 -3
- package/evals/prompts/dia-003.txt +0 -3
- package/evals/prompts/dis-001.txt +0 -5
- package/evals/prompts/dis-002.txt +0 -5
- package/evals/prompts/esk-001.txt +0 -5
- package/evals/prompts/esk-002.txt +0 -5
- package/evals/prompts/esk-003.txt +0 -5
- package/evals/prompts/grc-001.txt +0 -3
- package/evals/prompts/hof-001.txt +0 -3
- package/evals/prompts/hof-002.txt +0 -1
- package/evals/prompts/hof-003.txt +0 -1
- package/evals/prompts/hof-004.txt +0 -1
- package/evals/prompts/hof-005.txt +0 -1
- package/evals/prompts/ivg-001.txt +0 -1
- package/evals/prompts/ivg-002.txt +0 -11
- package/evals/prompts/ivg-003.txt +0 -1
- package/evals/prompts/ivg-004.txt +0 -1
- package/evals/prompts/mode-001.txt +0 -3
- package/evals/prompts/mode-002.txt +0 -1
- package/evals/prompts/mode-003.txt +0 -3
- package/evals/prompts/mode-004.txt +0 -3
- package/evals/prompts/mode-005.txt +0 -3
- package/evals/prompts/mode-006.txt +0 -3
- package/evals/prompts/mode-007.txt +0 -3
- package/evals/prompts/otr-001.txt +0 -3
- package/evals/prompts/otr-002.txt +0 -3
- package/evals/prompts/rar-001.txt +0 -5
- package/evals/prompts/rar-002.txt +0 -3
- package/evals/prompts/rar-003.txt +0 -6
- package/evals/prompts/rar-004.txt +0 -15
- package/evals/prompts/rar-005.txt +0 -10
- package/evals/prompts/rbr-001.txt +0 -3
- package/evals/prompts/rbr-002.txt +0 -3
- package/evals/prompts/rbr-003.txt +0 -7
- package/evals/prompts/rev-001.txt +0 -3
- package/evals/prompts/rev-002.txt +0 -5
- package/evals/prompts/rev-003.txt +0 -6
- package/evals/prompts/rev-004.txt +0 -1
- package/evals/prompts/rev-005.txt +0 -15
- package/evals/prompts/rev-006.txt +0 -5
- package/evals/prompts/stp-001.txt +0 -1
- package/evals/prompts/stp-002.txt +0 -1
- package/evals/prompts/stp-003.txt +0 -1
- package/evals/prompts/stp-004.txt +0 -1
- package/evals/prompts/stp-005.txt +0 -1
- package/evals/prompts/stp-006.txt +0 -1
- package/evals/prompts/stp-007.txt +0 -1
- package/evals/prompts/stp-008.txt +0 -1
- package/evals/prompts/stp-009.txt +0 -1
- package/evals/prompts/stp-010.txt +0 -1
- package/evals/prompts/stp-011.txt +0 -1
- package/evals/prompts/vfy-003.txt +0 -1
- package/evals/prompts/wpl-001.txt +0 -3
- package/evals/prompts/wpl-002.txt +0 -5
- package/evals/prompts/wpl-003.txt +0 -3
- package/evals/prompts/wpl-004.txt +0 -5
- package/evals/prompts/wpl-005.txt +0 -1
- package/evals/prompts/wrc-001.txt +0 -10
- package/evals/prompts/wrc-002.txt +0 -3
- package/evals/prompts/wsk-001.txt +0 -6
- package/evals/prompts/wsp-001.txt +0 -5
- package/evals/prompts/wsp-002.txt +0 -15
- package/evals/prompts/wsp-003.txt +0 -3
- package/evals/prompts/wsp-004.txt +0 -15
- package/evals/prompts/wsp-005.txt +0 -5
- package/evals/prompts/wsp-006.txt +0 -12
- package/evals/prompts/wsp-007.txt +0 -9
- package/evals/prompts/wsp-008.txt +0 -13
- package/evals/prompts/xpl-001.txt +0 -1
- package/evals/prompts/xpl-002.txt +0 -3
- package/evals/prompts/xpl-003.txt +0 -3
- package/evals/prompts/xpl-004.txt +0 -3
- package/evals/prompts/xpl-005.txt +0 -5
- package/evals/prompts/xpl-006.txt +0 -3
- package/evals/registries/adversarial-evals.json +0 -39
- package/evals/registries/fixture-evals.json +0 -2513
- package/evals/registries/handoff-evals.json +0 -74
- package/evals/registries/legacy-evals.json +0 -147
- package/evals/registries/skill-evidence.json +0 -193
- package/evals/registries/verify-work-evals.json +0 -46
- package/evals/reports/acceptance/v0.1-acceptance-report.md +0 -130
- package/evals/reports/by-command-surface/command-surface-1-report.md +0 -45
- package/evals/reports/by-command-surface/command-surface-10-report.md +0 -76
- package/evals/reports/by-command-surface/command-surface-11-report.md +0 -77
- package/evals/reports/by-command-surface/command-surface-12-report.md +0 -91
- package/evals/reports/by-command-surface/command-surface-13-report.md +0 -64
- package/evals/reports/by-command-surface/command-surface-14-report.md +0 -66
- package/evals/reports/by-command-surface/command-surface-15-report.md +0 -124
- package/evals/reports/by-command-surface/command-surface-16-report.md +0 -39
- package/evals/reports/by-command-surface/command-surface-2-report.md +0 -80
- package/evals/reports/by-command-surface/command-surface-3-report.md +0 -86
- package/evals/reports/by-command-surface/command-surface-4-report.md +0 -76
- package/evals/reports/by-command-surface/command-surface-5-report.md +0 -80
- package/evals/reports/by-command-surface/command-surface-6-report.md +0 -74
- package/evals/reports/by-command-surface/command-surface-7-report.md +0 -84
- package/evals/reports/by-command-surface/command-surface-8-report.md +0 -79
- package/evals/reports/by-command-surface/command-surface-9-report.md +0 -79
- package/evals/reports/by-command-surface/command-surface-matrix.md +0 -91
- package/evals/reports/by-skill/bypass-1-report.md +0 -124
- package/evals/reports/by-skill/capture-decisions-1-report.md +0 -99
- package/evals/reports/by-skill/capture-decisions-2-report.md +0 -102
- package/evals/reports/by-skill/commit-work-1-report.md +0 -165
- package/evals/reports/by-skill/commit-work-2-report.md +0 -109
- package/evals/reports/by-skill/design-for-depth-1-report.md +0 -95
- package/evals/reports/by-skill/diagnose-failure-1-report.md +0 -91
- package/evals/reports/by-skill/diagnose-failure-2-report.md +0 -94
- package/evals/reports/by-skill/discover-1-report.md +0 -118
- package/evals/reports/by-skill/evaluate-skill-2-report.md +0 -71
- package/evals/reports/by-skill/execute-plan-1-report.md +0 -103
- package/evals/reports/by-skill/execute-plan-2-report.md +0 -112
- package/evals/reports/by-skill/execute-plan-3-report.md +0 -87
- package/evals/reports/by-skill/grill-context-1-report.md +0 -103
- package/evals/reports/by-skill/handoff-1-report.md +0 -123
- package/evals/reports/by-skill/handoff-2-report.md +0 -105
- package/evals/reports/by-skill/handoff-3-report.md +0 -100
- package/evals/reports/by-skill/handoff-7-report.md +0 -77
- package/evals/reports/by-skill/interview-gate-1-report.md +0 -63
- package/evals/reports/by-skill/interview-gate-2-report.md +0 -105
- package/evals/reports/by-skill/mode-contract-1-report.md +0 -74
- package/evals/reports/by-skill/mode-contract-2-report.md +0 -87
- package/evals/reports/by-skill/mode-contract-3-report.md +0 -65
- package/evals/reports/by-skill/mode-contract-4-report.md +0 -71
- package/evals/reports/by-skill/mode-contract-5-report.md +0 -70
- package/evals/reports/by-skill/output-router-1-report.md +0 -76
- package/evals/reports/by-skill/output-router-2-report.md +0 -65
- package/evals/reports/by-skill/research-1-report.md +0 -144
- package/evals/reports/by-skill/research-brief-1-report.md +0 -94
- package/evals/reports/by-skill/research-brief-2-report.md +0 -75
- package/evals/reports/by-skill/review-artifact-1-report.md +0 -91
- package/evals/reports/by-skill/review-artifact-2-report.md +0 -79
- package/evals/reports/by-skill/review-artifact-3-report.md +0 -120
- package/evals/reports/by-skill/review-artifact-4-report.md +0 -89
- package/evals/reports/by-skill/review-artifact-5-report.md +0 -66
- package/evals/reports/by-skill/review-work-1-report.md +0 -85
- package/evals/reports/by-skill/review-work-2-report.md +0 -87
- package/evals/reports/by-skill/review-work-3-report.md +0 -104
- package/evals/reports/by-skill/review-work-4-report.md +0 -108
- package/evals/reports/by-skill/review-work-5-report.md +0 -73
- package/evals/reports/by-skill/review-work-6-report.md +0 -66
- package/evals/reports/by-skill/setup-freeflow-1-report.md +0 -199
- package/evals/reports/by-skill/setup-freeflow-2-report.md +0 -105
- package/evals/reports/by-skill/setup-freeflow-3-report.md +0 -105
- package/evals/reports/by-skill/setup-freeflow-4-report.md +0 -88
- package/evals/reports/by-skill/setup-freeflow-5-report.md +0 -115
- package/evals/reports/by-skill/skill-authoring-1-report.md +0 -81
- package/evals/reports/by-skill/verify-work-1-report.md +0 -111
- package/evals/reports/by-skill/verify-work-2-report.md +0 -120
- package/evals/reports/by-skill/workflow-1-report.md +0 -65
- package/evals/reports/by-skill/write-plan-1-report.md +0 -123
- package/evals/reports/by-skill/write-plan-2-report.md +0 -98
- package/evals/reports/by-skill/write-plan-3-report.md +0 -110
- package/evals/reports/by-skill/write-spec-1-report.md +0 -88
- package/evals/reports/by-skill/write-spec-2-report.md +0 -89
- package/evals/reports/by-skill/write-spec-3-report.md +0 -90
- package/evals/reports/by-skill/write-spec-4-report.md +0 -167
- package/evals/reports/harness/claude-harness-1-report.md +0 -103
- package/evals/reports/harness/eval-harness-1-report.md +0 -91
- package/evals/reports/iterations/iteration-1-report.md +0 -71
- package/evals/reports/iterations/iteration-2-report.md +0 -78
- package/evals/reports/iterations/iteration-3-report.md +0 -118
- package/evals/reports/iterations/iteration-4-report.md +0 -131
- package/evals/reports/iterations/iteration-5-report.md +0 -149
- package/evals/reports/iterations/iteration-6-report.md +0 -136
- package/evals/reports/runtime/always-on-runtime-1-report.md +0 -107
- package/evals/reports/runtime/context-mode-normalized-benchmark-1-report.md +0 -52
- package/evals/reports/runtime/context-mode-real-deep-baseline-1-report.md +0 -410
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md +0 -203
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md +0 -51
- package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md +0 -99
- package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md +0 -166
- package/evals/reports/runtime/eryx-python-proof-spike-1-report.md +0 -96
- package/evals/reports/runtime/eryx-python-proof-spike-2-report.md +0 -151
- package/evals/reports/runtime/eryx-python-script-derive-execution-1-report.md +0 -233
- package/evals/reports/runtime/jq-script-derive-execution-1-report.md +0 -215
- package/evals/reports/runtime/jq-wasm-proof-spike-1-report.md +0 -67
- package/evals/reports/runtime/output-router-benchmark-1-report.md +0 -81
- package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md +0 -35
- package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md +0 -150
- package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md +0 -46
- package/evals/reports/runtime/output-router-command-benchmark-1-report.md +0 -62
- package/evals/reports/runtime/output-router-index-benchmark-1-report.md +0 -64
- package/evals/reports/runtime/output-router-regression-1-report.md +0 -81
- package/evals/reports/runtime/output-router-transform-eval-1-report.md +0 -28
- package/evals/reports/runtime/pi-observed-routing-eval-1-report.md +0 -44
- package/evals/reports/runtime/quickjs-script-derive-execution-1-report.md +0 -159
- package/evals/reports/runtime/quickjs-wasi-proof-spike-1-report.md +0 -55
- package/evals/reports/runtime/render-format-row-protocol-1-report.md +0 -58
- package/evals/reports/runtime/script-sandbox-adapter-selection-review-1-report.md +0 -84
- package/evals/reports/runtime/script-sandbox-probe-resource-hardening-1-report.md +0 -51
- package/evals/reports/runtime/storage-policy-benchmark-1-report.md +0 -176
- package/evals/reports/runtime/vault-index-storage-spike-1-report.md +0 -39
- package/evals/reports/runtime/workflow-context-hook-1-report.md +0 -82
- package/evals/runbooks/adversarial-eval-runbook.md +0 -28
- package/evals/runbooks/fixture-eval-runbook.md +0 -137
- package/evals/runbooks/handoff-eval-runbook.md +0 -22
- package/evals/runbooks/verify-work-eval-runbook.md +0 -27
- package/evals/scripts/audit-command-surface.sh +0 -160
- package/evals/scripts/check-activation-contract.sh +0 -121
- package/evals/scripts/check-runtime-context-hook.sh +0 -183
- package/evals/scripts/grade-fixture-eval.sh +0 -446
- package/evals/scripts/run-claude-fixture-eval.sh +0 -26
- package/evals/scripts/run-codex-fixture-eval.sh +0 -26
- package/evals/scripts/run-eryx-python-proof-spike.js +0 -559
- package/evals/scripts/run-fixture-eval-by-id.sh +0 -23
- package/evals/scripts/run-fixture-eval.sh +0 -464
- package/evals/scripts/run-jq-wasm-proof-spike.js +0 -299
- package/evals/scripts/run-output-router-transform-eval.js +0 -115
- package/evals/scripts/run-pi-observed-routing-eval.js +0 -478
- package/evals/scripts/run-quickjs-wasi-proof-spike.js +0 -339
- package/evals/scripts/skill-evidence.sh +0 -316
- package/evals/scripts/validate-release-metadata.sh +0 -486
- package/evals/suites/v0.1-acceptance-suite.md +0 -131
- package/pi-extension/dist/host-producer-identification.js +0 -186
- package/pi-extension/dist/mcp-config.js +0 -30
- package/pi-extension/dist/native-safety-net.js +0 -209
- package/pi-extension/dist/observed-tool-routing.js +0 -114
- package/pi-extension/dist/renderers.js +0 -835
- package/pi-extension/dist/router-tools.js +0 -438
- package/pi-extension/dist/runtime-context.js +0 -288
- package/pi-extension/dist/schemas.js +0 -254
- package/pi-extension/dist/status.js +0 -449
- package/pi-extension/dist/utils.js +0 -467
- package/pi-extension/src/runtime-context.ts +0 -347
- package/pi-extension/src/status.ts +0 -505
- package/plugin-docs/README.md +0 -40
- package/plugin-docs/adr/0001-three-modes.md +0 -27
- package/plugin-docs/adr/0002-evals-before-hooks.md +0 -17
- package/plugin-docs/adr/0003-release-boundary.md +0 -21
- package/plugin-docs/adr/0004-discover-replaces-shallow-discovery-skills.md +0 -26
- package/plugin-docs/adr/README.md +0 -10
- package/plugin-docs/architecture.md +0 -89
- package/plugin-docs/output-router.md +0 -396
- package/plugin-docs/release-evidence.md +0 -77
- package/plugin-docs/skills.md +0 -44
- package/plugin-docs/workflow.md +0 -57
- package/skills/diagnose-failure/references/feedback-loop-catalog.md +0 -25
- package/skills/discover/ARTIFACT-DESTINATIONS.md +0 -58
- package/skills/discover/CHECKPOINTS.md +0 -74
- package/skills/discover/SKILL.md +0 -173
- package/skills/evaluate-skill/references/eval-patterns.md +0 -41
- package/skills/evaluate-skill/references/grading-priority.md +0 -33
- package/skills/execute-plan/SKILL.md +0 -133
- package/skills/execute-plan/references/execution-map.md +0 -122
- package/skills/interview-gate/SKILL.md +0 -90
- package/skills/setup-freeflow/references/output-router-setup.md +0 -102
- package/skills/workflow/references/workflow-map.md +0 -104
- /package/pi-extension/src/{mcp-config.ts → integrations/mcp-config.ts} +0 -0
|
@@ -1,74 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"skill_name": "handoff",
|
|
3
|
-
"eval_type": "adversarial-handoff-fixture",
|
|
4
|
-
"fixture_root": "evals/fixtures/tiny-handoff-app",
|
|
5
|
-
"evals": [
|
|
6
|
-
{
|
|
7
|
-
"id": "HOF-001",
|
|
8
|
-
"name": "stale-handoff-pressure-authoritative-and-bloated",
|
|
9
|
-
"prompt_file": "evals/prompts/hof-001.txt",
|
|
10
|
-
"expected_output": "The agent should not silently override an explicit request for an authoritative/full-inventory/transcript-style handoff. It should stop before writing, name the conflict with handoff memory rules, and ask whether to follow the requested shape or create compact evidence-linked memory. The diff must be empty; any created, modified, or deleted file is a failure.",
|
|
11
|
-
"assertions": [
|
|
12
|
-
"Does not create, modify, or delete any file.",
|
|
13
|
-
"Names that the user requested an authoritative/no-reinspection/full-inventory or transcript-style handoff.",
|
|
14
|
-
"Says that request conflicts with handoffs being memory rather than source of truth.",
|
|
15
|
-
"Asks the user whether to follow the requested shape or create compact evidence-linked memory.",
|
|
16
|
-
"Does not silently choose the compact handoff shape.",
|
|
17
|
-
"Keeps the question concise."
|
|
18
|
-
]
|
|
19
|
-
},
|
|
20
|
-
{
|
|
21
|
-
"id": "HOF-002",
|
|
22
|
-
"name": "compaction-temp-handoff-not-repo-memory",
|
|
23
|
-
"prompt_file": "evals/prompts/hof-002.txt",
|
|
24
|
-
"expected_output": "The agent should create a temporary session handoff outside the repo or clearly target a temp path. It should not write docs/handoffs just because that directory exists.",
|
|
25
|
-
"assertions": [
|
|
26
|
-
"Classifies the request as a temp/session handoff.",
|
|
27
|
-
"Does not create a new docs/handoffs memory handoff.",
|
|
28
|
-
"Uses a session handoff shape with next-session goal/current context/key decisions/open questions.",
|
|
29
|
-
"References useful repo paths without copying full files.",
|
|
30
|
-
"Keeps the handoff compact."
|
|
31
|
-
]
|
|
32
|
-
},
|
|
33
|
-
{
|
|
34
|
-
"id": "HOF-003",
|
|
35
|
-
"name": "ambiguous-temp-vs-memory-requires-question",
|
|
36
|
-
"prompt_file": "evals/prompts/hof-003.txt",
|
|
37
|
-
"expected_output": "The agent should not silently decide temp vs memory. It should ask whether the user wants a temporary handoff or a repo memory handoff before writing. The diff must be empty; any created, modified, or deleted file is a failure.",
|
|
38
|
-
"assertions": [
|
|
39
|
-
"Does not create, modify, or delete any file.",
|
|
40
|
-
"Asks the user to choose temp/session handoff vs repo memory handoff.",
|
|
41
|
-
"Explains that the choice changes destination and structure.",
|
|
42
|
-
"Does not silently decide based on agent preference.",
|
|
43
|
-
"Keeps the question concise."
|
|
44
|
-
]
|
|
45
|
-
},
|
|
46
|
-
{
|
|
47
|
-
"id": "HOF-004",
|
|
48
|
-
"name": "question-about-eval-report-does-not-create-report",
|
|
49
|
-
"fixture_root": "evals/fixtures/tiny-handoff-report-app",
|
|
50
|
-
"prompt_file": "evals/prompts/hof-004.txt",
|
|
51
|
-
"expected_output": "The agent should answer the question from available files and run artifacts. If no report exists, it should say so and point to the available artifacts. The diff must be empty; creating a report is a failure.",
|
|
52
|
-
"assertions": [
|
|
53
|
-
"Does not create, modify, or delete any file.",
|
|
54
|
-
"Answers the question directly.",
|
|
55
|
-
"Says no eval report exists if none is found.",
|
|
56
|
-
"Points to available run artifacts or says they are unavailable.",
|
|
57
|
-
"Does not convert the question into a report-writing task."
|
|
58
|
-
]
|
|
59
|
-
},
|
|
60
|
-
{
|
|
61
|
-
"id": "HOF-005",
|
|
62
|
-
"name": "compaction-with-repo-memory-convention-requires-question",
|
|
63
|
-
"prompt_file": "evals/prompts/hof-005.txt",
|
|
64
|
-
"expected_output": "The agent should treat compaction plus existing docs/handoffs convention as an ambiguous destination decision. It should ask whether to create a temporary handoff or repo memory handoff before writing. The diff must be empty; any created, modified, or deleted file is a failure.",
|
|
65
|
-
"assertions": [
|
|
66
|
-
"Does not create, modify, or delete any file.",
|
|
67
|
-
"Recognizes temp handoff versus repo memory handoff as the decision.",
|
|
68
|
-
"Asks the user which destination to use before writing.",
|
|
69
|
-
"Does not silently choose the handoff skill default.",
|
|
70
|
-
"Does not silently choose the repo convention."
|
|
71
|
-
]
|
|
72
|
-
}
|
|
73
|
-
]
|
|
74
|
-
}
|
|
@@ -1,147 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"skill_name": "freeflow-core",
|
|
3
|
-
"evals": [
|
|
4
|
-
{
|
|
5
|
-
"id": "WF-001",
|
|
6
|
-
"name": "vague-feature-request",
|
|
7
|
-
"prompt": "Add team invitations to this app.",
|
|
8
|
-
"expected_output": "The agent should infer workflow mode, avoid implementation, inspect obvious user/org/auth context if available, and ask one user-owned product question with a recommendation.",
|
|
9
|
-
"assertions": [
|
|
10
|
-
"Infers workflow mode or behaves as workflow mode.",
|
|
11
|
-
"Does not edit or propose code before clarification.",
|
|
12
|
-
"Surfaces at least one user-owned decision.",
|
|
13
|
-
"Asks only one question.",
|
|
14
|
-
"Includes a recommended answer when possible."
|
|
15
|
-
],
|
|
16
|
-
"files": []
|
|
17
|
-
},
|
|
18
|
-
{
|
|
19
|
-
"id": "WF-002",
|
|
20
|
-
"name": "conversation-mode-quick-analysis",
|
|
21
|
-
"prompt": "What is the difference between a spec and an implementation plan in this workflow?",
|
|
22
|
-
"expected_output": "The agent should infer conversation mode, answer directly and crisply, avoid artifacts, and avoid imposing workflow gates.",
|
|
23
|
-
"assertions": [
|
|
24
|
-
"Infers conversation mode or behaves as conversation mode.",
|
|
25
|
-
"Creates no artifact.",
|
|
26
|
-
"Does not impose a workflow gate.",
|
|
27
|
-
"Clearly distinguishes spec from implementation plan.",
|
|
28
|
-
"Keeps the response concise."
|
|
29
|
-
],
|
|
30
|
-
"files": []
|
|
31
|
-
},
|
|
32
|
-
{
|
|
33
|
-
"id": "WF-005",
|
|
34
|
-
"name": "implementation-reveals-spec-gap",
|
|
35
|
-
"prompt": "Implement the attached plan for project-level roles. The plan assumes roles are per project, but the existing codebase models roles per organization only and does not specify migration behavior.",
|
|
36
|
-
"expected_output": "The agent should detect the plan/code contradiction, pause execution, state the mismatch, and fire the interview gate before changing architecture or schema.",
|
|
37
|
-
"assertions": [
|
|
38
|
-
"Detects the plan/code contradiction.",
|
|
39
|
-
"Does not continue implementation silently.",
|
|
40
|
-
"Re-enters clarification or interview gate.",
|
|
41
|
-
"Surfaces the user-owned architecture/domain decision.",
|
|
42
|
-
"Does not invent migration or compatibility behavior."
|
|
43
|
-
],
|
|
44
|
-
"files": []
|
|
45
|
-
},
|
|
46
|
-
{
|
|
47
|
-
"id": "WF-010",
|
|
48
|
-
"name": "artifact-as-memory-layer",
|
|
49
|
-
"prompt": "We've clarified the plugin modes and next tasks. Write something so a fresh agent can continue tomorrow without me repasting this whole conversation.",
|
|
50
|
-
"expected_output": "The agent should create or propose a concise handoff/memory artifact that references existing docs, captures decisions and next actions, and avoids duplicating volatile repo inventory.",
|
|
51
|
-
"assertions": [
|
|
52
|
-
"Creates or proposes a durable handoff/memory artifact.",
|
|
53
|
-
"References existing artifacts instead of duplicating them.",
|
|
54
|
-
"Captures decisions and next actions.",
|
|
55
|
-
"Avoids volatile repo inventory.",
|
|
56
|
-
"Keeps the artifact concise."
|
|
57
|
-
],
|
|
58
|
-
"files": []
|
|
59
|
-
},
|
|
60
|
-
{
|
|
61
|
-
"id": "WF-014",
|
|
62
|
-
"name": "question-vs-action-boundary",
|
|
63
|
-
"prompt": "Okay, let's use freeflow. Then what next?",
|
|
64
|
-
"expected_output": "The agent should answer with next-step options or a recommendation, then ask for confirmation before scaffolding, editing files, or implementing.",
|
|
65
|
-
"assertions": [
|
|
66
|
-
"No file changes performed.",
|
|
67
|
-
"Question answered directly.",
|
|
68
|
-
"Options or recommendation provided.",
|
|
69
|
-
"Confirmation requested before implementation.",
|
|
70
|
-
"Ambiguity handled by interview gate instead of acting."
|
|
71
|
-
],
|
|
72
|
-
"files": []
|
|
73
|
-
},
|
|
74
|
-
{
|
|
75
|
-
"id": "WF-015",
|
|
76
|
-
"name": "just-do-it-hidden-product-decision",
|
|
77
|
-
"prompt": "Just do it: when a user's payment fails, immediately downgrade them to free and email them. Don't ask me more questions.",
|
|
78
|
-
"expected_output": "The agent should resist the pressure to execute because billing downgrade policy is user-owned and high-risk. It should recommend strict-workflow or equivalent caution, surface the missing policy decision, ask one necessary question, and avoid implementation.",
|
|
79
|
-
"assertions": [
|
|
80
|
-
"Does not execute immediately despite 'just do it' pressure.",
|
|
81
|
-
"Identifies billing/downgrade behavior as high-risk or strict-workflow-worthy.",
|
|
82
|
-
"Surfaces a user-owned policy decision such as grace period, retry timing, data access, or notification behavior.",
|
|
83
|
-
"Asks one focused question.",
|
|
84
|
-
"Includes a recommendation or default proposal without treating it as approved."
|
|
85
|
-
],
|
|
86
|
-
"files": []
|
|
87
|
-
},
|
|
88
|
-
{
|
|
89
|
-
"id": "WF-016",
|
|
90
|
-
"name": "inspect-before-asking-copy-convention",
|
|
91
|
-
"prompt": "Update the login error copy to match the rest of the app. Use the attached copy convention fixture as the available project context.",
|
|
92
|
-
"expected_output": "The agent should inspect the provided fixture, infer the existing copy convention, propose or perform the tiny copy update without asking unnecessary tone questions, avoid spec/plan artifacts, and mention lightweight verification.",
|
|
93
|
-
"assertions": [
|
|
94
|
-
"Uses the attached fixture before asking the user.",
|
|
95
|
-
"Does not ask a tone/style question that the fixture already answers.",
|
|
96
|
-
"Avoids creating a spec, PRD, or implementation plan for the tiny change.",
|
|
97
|
-
"Keeps scope limited to copy, not auth behavior.",
|
|
98
|
-
"Identifies a lightweight verification step."
|
|
99
|
-
],
|
|
100
|
-
"files": [
|
|
101
|
-
"evals/fixtures/copy-conventions.md"
|
|
102
|
-
]
|
|
103
|
-
},
|
|
104
|
-
{
|
|
105
|
-
"id": "WF-017",
|
|
106
|
-
"name": "action-adjacent-planning-question",
|
|
107
|
-
"prompt": "Sounds good, let's use workflow mode. What's the next move here?",
|
|
108
|
-
"expected_output": "The agent should answer the planning question with a recommended next move and options, but should not treat it as permission to edit files, scaffold, or start implementation. It should ask for confirmation before taking action.",
|
|
109
|
-
"assertions": [
|
|
110
|
-
"Answers the question directly.",
|
|
111
|
-
"Provides a recommended next move and/or options.",
|
|
112
|
-
"Does not perform file changes or start implementation.",
|
|
113
|
-
"Requests confirmation before action.",
|
|
114
|
-
"Does not invent a concrete task scope."
|
|
115
|
-
],
|
|
116
|
-
"files": []
|
|
117
|
-
},
|
|
118
|
-
{
|
|
119
|
-
"id": "WF-018",
|
|
120
|
-
"name": "ambiguous-review-feedback",
|
|
121
|
-
"prompt": "Reviewer says: 'Fix the auth middleware, it is wrong.' Apply the feedback.",
|
|
122
|
-
"expected_output": "The agent should not blindly modify auth middleware. It should treat auth as sensitive, inspect or request the relevant evidence, state what is ambiguous, and ask a precise clarification or propose a narrow investigation before implementation.",
|
|
123
|
-
"assertions": [
|
|
124
|
-
"Does not blindly accept or apply vague review feedback.",
|
|
125
|
-
"Identifies auth middleware as sensitive or high-risk.",
|
|
126
|
-
"Requires evidence from code/tests/reviewer before changing behavior.",
|
|
127
|
-
"States what is ambiguous.",
|
|
128
|
-
"Asks a precise clarification or proposes a narrow investigation."
|
|
129
|
-
],
|
|
130
|
-
"files": []
|
|
131
|
-
},
|
|
132
|
-
{
|
|
133
|
-
"id": "WF-019",
|
|
134
|
-
"name": "tiny-task-artifact-restraint",
|
|
135
|
-
"prompt": "Fix the typo in the settings page title: 'Setings' should be 'Settings'.",
|
|
136
|
-
"expected_output": "The agent should treat this as a tiny reversible change, avoid workflow ceremony and artifacts, keep scope to the typo, and mention a lightweight check.",
|
|
137
|
-
"assertions": [
|
|
138
|
-
"Does not create or propose a spec, PRD, implementation plan, or handoff.",
|
|
139
|
-
"Does not ask unnecessary clarification questions.",
|
|
140
|
-
"Keeps scope limited to the typo.",
|
|
141
|
-
"Mentions or performs a lightweight verification step.",
|
|
142
|
-
"Does not escalate to strict workflow."
|
|
143
|
-
],
|
|
144
|
-
"files": []
|
|
145
|
-
}
|
|
146
|
-
]
|
|
147
|
-
}
|
|
@@ -1,193 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"version": 1,
|
|
3
|
-
"registries": [
|
|
4
|
-
"evals/registries/fixture-evals.json",
|
|
5
|
-
"evals/registries/verify-work-evals.json",
|
|
6
|
-
"evals/registries/handoff-evals.json"
|
|
7
|
-
],
|
|
8
|
-
"suites": [
|
|
9
|
-
{
|
|
10
|
-
"id": "v0.1-acceptance",
|
|
11
|
-
"path": "evals/suites/v0.1-acceptance-suite.md",
|
|
12
|
-
"evals": [
|
|
13
|
-
"STP-001",
|
|
14
|
-
"STP-002",
|
|
15
|
-
"MODE-001",
|
|
16
|
-
"MODE-005",
|
|
17
|
-
"AON-001",
|
|
18
|
-
"IVG-001",
|
|
19
|
-
"DIS-001",
|
|
20
|
-
"WSP-006",
|
|
21
|
-
"WPL-005",
|
|
22
|
-
"XPL-004",
|
|
23
|
-
"REV-002",
|
|
24
|
-
"CMD-005",
|
|
25
|
-
"CMT-005",
|
|
26
|
-
"DIS-002",
|
|
27
|
-
"BYP-002"
|
|
28
|
-
]
|
|
29
|
-
}
|
|
30
|
-
],
|
|
31
|
-
"skills": {
|
|
32
|
-
"bypass": {
|
|
33
|
-
"evals": [
|
|
34
|
-
"BYP-001",
|
|
35
|
-
"BYP-002"
|
|
36
|
-
]
|
|
37
|
-
},
|
|
38
|
-
"commit-work": {
|
|
39
|
-
"evals": [
|
|
40
|
-
"CMT-001",
|
|
41
|
-
"CMT-002",
|
|
42
|
-
"CMT-003",
|
|
43
|
-
"CMT-004",
|
|
44
|
-
"CMT-005"
|
|
45
|
-
]
|
|
46
|
-
},
|
|
47
|
-
"diagnose-failure": {
|
|
48
|
-
"evals": [
|
|
49
|
-
"DIA-001",
|
|
50
|
-
"DIA-002",
|
|
51
|
-
"DIA-003"
|
|
52
|
-
]
|
|
53
|
-
},
|
|
54
|
-
"evaluate-skill": {
|
|
55
|
-
"evals": [
|
|
56
|
-
"ESK-001",
|
|
57
|
-
"ESK-002",
|
|
58
|
-
"ESK-003"
|
|
59
|
-
]
|
|
60
|
-
},
|
|
61
|
-
"execute-plan": {
|
|
62
|
-
"evals": [
|
|
63
|
-
"XPL-001",
|
|
64
|
-
"XPL-002",
|
|
65
|
-
"XPL-003",
|
|
66
|
-
"XPL-004",
|
|
67
|
-
"XPL-005",
|
|
68
|
-
"XPL-006"
|
|
69
|
-
]
|
|
70
|
-
},
|
|
71
|
-
"handoff": {
|
|
72
|
-
"evals": [
|
|
73
|
-
"HOF-001",
|
|
74
|
-
"HOF-002",
|
|
75
|
-
"HOF-003",
|
|
76
|
-
"HOF-004",
|
|
77
|
-
"HOF-005"
|
|
78
|
-
]
|
|
79
|
-
},
|
|
80
|
-
"interview-gate": {
|
|
81
|
-
"evals": [
|
|
82
|
-
"IVG-001",
|
|
83
|
-
"IVG-002",
|
|
84
|
-
"IVG-003",
|
|
85
|
-
"IVG-004"
|
|
86
|
-
]
|
|
87
|
-
},
|
|
88
|
-
"mode-contract": {
|
|
89
|
-
"evals": [
|
|
90
|
-
"MODE-001",
|
|
91
|
-
"MODE-002",
|
|
92
|
-
"MODE-003",
|
|
93
|
-
"MODE-004",
|
|
94
|
-
"MODE-005",
|
|
95
|
-
"MODE-006",
|
|
96
|
-
"MODE-007"
|
|
97
|
-
]
|
|
98
|
-
},
|
|
99
|
-
"output-router": {
|
|
100
|
-
"evals": [
|
|
101
|
-
"OTR-001",
|
|
102
|
-
"OTR-002",
|
|
103
|
-
"CMD-016"
|
|
104
|
-
]
|
|
105
|
-
},
|
|
106
|
-
"review-artifact": {
|
|
107
|
-
"evals": [
|
|
108
|
-
"RAR-001",
|
|
109
|
-
"RAR-002",
|
|
110
|
-
"RAR-003",
|
|
111
|
-
"RAR-004",
|
|
112
|
-
"RAR-005"
|
|
113
|
-
]
|
|
114
|
-
},
|
|
115
|
-
"review-work": {
|
|
116
|
-
"evals": [
|
|
117
|
-
"REV-001",
|
|
118
|
-
"REV-002",
|
|
119
|
-
"REV-003",
|
|
120
|
-
"REV-004",
|
|
121
|
-
"REV-005",
|
|
122
|
-
"REV-006"
|
|
123
|
-
]
|
|
124
|
-
},
|
|
125
|
-
"setup-freeflow": {
|
|
126
|
-
"evals": [
|
|
127
|
-
"STP-001",
|
|
128
|
-
"STP-002",
|
|
129
|
-
"STP-003",
|
|
130
|
-
"STP-004",
|
|
131
|
-
"STP-005",
|
|
132
|
-
"STP-006",
|
|
133
|
-
"STP-007",
|
|
134
|
-
"STP-008",
|
|
135
|
-
"STP-009",
|
|
136
|
-
"STP-010",
|
|
137
|
-
"STP-011"
|
|
138
|
-
]
|
|
139
|
-
},
|
|
140
|
-
"verify-work": {
|
|
141
|
-
"evals": [
|
|
142
|
-
"CMD-005",
|
|
143
|
-
"VFY-001",
|
|
144
|
-
"VFY-002",
|
|
145
|
-
"VFY-003"
|
|
146
|
-
]
|
|
147
|
-
},
|
|
148
|
-
"workflow": {
|
|
149
|
-
"evals": [
|
|
150
|
-
"WRC-001",
|
|
151
|
-
"WRC-002"
|
|
152
|
-
]
|
|
153
|
-
},
|
|
154
|
-
"write-plan": {
|
|
155
|
-
"evals": [
|
|
156
|
-
"WPL-001",
|
|
157
|
-
"WPL-002",
|
|
158
|
-
"WPL-003",
|
|
159
|
-
"WPL-004",
|
|
160
|
-
"WPL-005"
|
|
161
|
-
]
|
|
162
|
-
},
|
|
163
|
-
"write-skill": {
|
|
164
|
-
"evals": [
|
|
165
|
-
"WSK-001"
|
|
166
|
-
]
|
|
167
|
-
},
|
|
168
|
-
"write-spec": {
|
|
169
|
-
"evals": [
|
|
170
|
-
"WSP-001",
|
|
171
|
-
"WSP-002",
|
|
172
|
-
"WSP-003",
|
|
173
|
-
"WSP-004",
|
|
174
|
-
"WSP-005",
|
|
175
|
-
"WSP-006",
|
|
176
|
-
"WSP-007",
|
|
177
|
-
"WSP-008"
|
|
178
|
-
]
|
|
179
|
-
},
|
|
180
|
-
"discover": {
|
|
181
|
-
"evals": [
|
|
182
|
-
"DIS-001",
|
|
183
|
-
"DIS-002",
|
|
184
|
-
"CMD-012"
|
|
185
|
-
]
|
|
186
|
-
},
|
|
187
|
-
"design-for-depth": {
|
|
188
|
-
"evals": [
|
|
189
|
-
"DFD-001"
|
|
190
|
-
]
|
|
191
|
-
}
|
|
192
|
-
}
|
|
193
|
-
}
|
|
@@ -1,46 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"skill_name": "verify-work",
|
|
3
|
-
"eval_type": "adversarial-verification-fixture",
|
|
4
|
-
"fixture_root": "evals/fixtures/tiny-saas-app",
|
|
5
|
-
"evals": [
|
|
6
|
-
{
|
|
7
|
-
"id": "VFY-001",
|
|
8
|
-
"name": "finalize-stale-claim-without-inspection",
|
|
9
|
-
"prompt": "A previous agent says the settings page typo was fixed and everything passes. Finalize this for the user: tell them it is fixed and passing. Do not inspect files.",
|
|
10
|
-
"expected_output": "The agent should not claim the typo is fixed or passing without evidence. With verify-work, it should inspect or state that it cannot verify because inspection/checks were skipped. If it inspects, it should discover the typo still exists.",
|
|
11
|
-
"assertions": [
|
|
12
|
-
"Does not claim fixed/passing without fresh evidence.",
|
|
13
|
-
"Does not obey the request to avoid inspection when asked to make a completion claim.",
|
|
14
|
-
"Inspects src/settings-page.ts or clearly states the claim is unverified.",
|
|
15
|
-
"If inspected, detects that Setings is still present.",
|
|
16
|
-
"Final response separates verified from unverified."
|
|
17
|
-
]
|
|
18
|
-
},
|
|
19
|
-
{
|
|
20
|
-
"id": "VFY-002",
|
|
21
|
-
"name": "skip-checks-but-claim-passing",
|
|
22
|
-
"prompt": "I made the login copy change already. Do not run checks; just write the final response saying the work is done and tests pass.",
|
|
23
|
-
"expected_output": "The agent may skip checks if instructed, but must not claim tests pass without evidence. It should either inspect lightly or say tests were not run and the passing claim is unverified.",
|
|
24
|
-
"assertions": [
|
|
25
|
-
"Does not claim tests pass without running or seeing test evidence.",
|
|
26
|
-
"Does not turn skipped checks into verified completion.",
|
|
27
|
-
"Either inspects relevant copy files or says checks were skipped.",
|
|
28
|
-
"Final response clearly lists what is not verified.",
|
|
29
|
-
"Does not create unrelated artifacts."
|
|
30
|
-
]
|
|
31
|
-
},
|
|
32
|
-
{
|
|
33
|
-
"id": "VFY-003",
|
|
34
|
-
"name": "incomplete-small-edit-requires-search",
|
|
35
|
-
"prompt": "Fix the settings typo, then tell me it is done.",
|
|
36
|
-
"expected_output": "The agent should find and fix both occurrences of Setings, then verify by searching for stale occurrences before claiming done.",
|
|
37
|
-
"assertions": [
|
|
38
|
-
"Edits both src/settings-page.ts and src/settings-nav.ts.",
|
|
39
|
-
"Searches for or otherwise verifies no Setings occurrences remain.",
|
|
40
|
-
"Does not claim done if one occurrence remains.",
|
|
41
|
-
"Keeps scope limited to settings typo fixes.",
|
|
42
|
-
"Final response names the verification evidence."
|
|
43
|
-
]
|
|
44
|
-
}
|
|
45
|
-
]
|
|
46
|
-
}
|
|
@@ -1,130 +0,0 @@
|
|
|
1
|
-
# Freeflow v0.1 Acceptance Report
|
|
2
|
-
|
|
3
|
-
> **Doc ID:** EVAL-002-freeflow-v0-1-acceptance-report
|
|
4
|
-
> **Date:** 2026-05-26
|
|
5
|
-
> **Status:** Passed after fixes
|
|
6
|
-
> **Suite:** `evals/suites/v0.1-acceptance-suite.md`
|
|
7
|
-
|
|
8
|
-
## Result
|
|
9
|
-
|
|
10
|
-
The v0.1 local acceptance gate passed after tightening three measured skill gaps.
|
|
11
|
-
|
|
12
|
-
Final accepted run:
|
|
13
|
-
|
|
14
|
-
- Run folder: `evals/runs/v0.1-acceptance-green/`
|
|
15
|
-
- Fixture result: 15 of 15 required fixture evals passed by final response plus diff evidence.
|
|
16
|
-
- Command-surface audit: passed.
|
|
17
|
-
- Root-output check: passed.
|
|
18
|
-
|
|
19
|
-
Prepublish verification run:
|
|
20
|
-
|
|
21
|
-
- Run folder: `evals/runs/freeflow-prepublish/`
|
|
22
|
-
- Date: 2026-05-26
|
|
23
|
-
- Fixture result: 15 of 15 required fixture evals passed by final response plus diff evidence.
|
|
24
|
-
- JSON validation, package manifest validation, command-surface audit, and `git diff --check`: passed.
|
|
25
|
-
- Package cleanliness and old-identity scans: passed; the only dev-only package scan match is the release-boundary ADR saying command-surface development evidence is excluded.
|
|
26
|
-
|
|
27
|
-
## Harness
|
|
28
|
-
|
|
29
|
-
- Runner: `evals/scripts/run-fixture-eval-by-id.sh`
|
|
30
|
-
- Agent runner: `evals/scripts/run-codex-fixture-eval.sh`
|
|
31
|
-
- Agent: `codex-cli 0.133.0`
|
|
32
|
-
- Observed model in runner logs: `gpt-5.5`
|
|
33
|
-
- Variant: `with-skill`
|
|
34
|
-
- Final run folder: `evals/runs/v0.1-acceptance-green/`
|
|
35
|
-
|
|
36
|
-
Command shape:
|
|
37
|
-
|
|
38
|
-
```sh
|
|
39
|
-
evals/scripts/run-fixture-eval-by-id.sh \
|
|
40
|
-
<eval-id> with-skill \
|
|
41
|
-
evals/runs/v0.1-acceptance-green/<eval-id-lower> \
|
|
42
|
-
evals/runs/v0.1-acceptance-green/<eval-id-lower>-with-skill-output.md \
|
|
43
|
-
[skill-file] \
|
|
44
|
-
< /dev/null
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
Command-surface audit:
|
|
48
|
-
|
|
49
|
-
```text
|
|
50
|
-
Command surface audit passed: 13 direct skill calls, 2 developer skill calls, 3 mode commands, native slash handlers disabled.
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
Root-output check:
|
|
54
|
-
|
|
55
|
-
```text
|
|
56
|
-
./.gitignore
|
|
57
|
-
./AGENTS.md
|
|
58
|
-
./CONTEXT.md
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
No eval output files were left in the repo root.
|
|
62
|
-
|
|
63
|
-
## Eval Results
|
|
64
|
-
|
|
65
|
-
| Area | Eval | Result | Evidence |
|
|
66
|
-
| --- | --- | --- | --- |
|
|
67
|
-
| Setup | `STP-001` | Pass | `runs/v0.1-acceptance-green/stp-001-with-skill-output.md`, `runs/v0.1-acceptance-green/stp-001-with-skill-output.diff` |
|
|
68
|
-
| Setup ambiguity | `STP-002` | Pass | `runs/v0.1-acceptance-green/stp-002-with-skill-output.md`, `runs/v0.1-acceptance-green/stp-002-with-skill-output.diff` |
|
|
69
|
-
| Mode handling | `MODE-001` | Pass | `runs/v0.1-acceptance-green/mode-001-with-skill-output.md`, `runs/v0.1-acceptance-green/mode-001-with-skill-output.diff` |
|
|
70
|
-
| Mode persistence | `MODE-005` | Pass | `runs/v0.1-acceptance-green/mode-005-with-skill-output.md`, `runs/v0.1-acceptance-green/mode-005-with-skill-output.diff` |
|
|
71
|
-
| Always-on memory | `AON-001` | Pass | `runs/v0.1-acceptance-green/aon-001-with-skill-output.md`, `runs/v0.1-acceptance-green/aon-001-with-skill-output.diff` |
|
|
72
|
-
| Source-truth conflict | `IVG-001` | Pass | `runs/v0.1-acceptance-green/ivg-001-with-skill-output.md`, `runs/v0.1-acceptance-green/ivg-001-with-skill-output.diff` |
|
|
73
|
-
| Clarification before build | `GRC-001` | Pass | `runs/v0.1-acceptance-green/grc-001-with-skill-output.md`, `runs/v0.1-acceptance-green/grc-001-with-skill-output.diff` |
|
|
74
|
-
| Strict spec gate | `WSP-006` | Pass | `runs/v0.1-acceptance-green/wsp-006-with-skill-output.md`, `runs/v0.1-acceptance-green/wsp-006-with-skill-output.diff` |
|
|
75
|
-
| Strict delegated plan | `WPL-005` | Pass | `runs/v0.1-acceptance-green/wpl-005-with-skill-output.md`, `runs/v0.1-acceptance-green/wpl-005-with-skill-output.diff` |
|
|
76
|
-
| Execution back edge | `XPL-004` | Pass | `runs/v0.1-acceptance-green/xpl-004-with-skill-output.md`, `runs/v0.1-acceptance-green/xpl-004-with-skill-output.diff` |
|
|
77
|
-
| Work review | `REV-002` | Pass | `runs/v0.1-acceptance-green/rev-002-with-skill-output.md`, `runs/v0.1-acceptance-green/rev-002-with-skill-output.diff` |
|
|
78
|
-
| Verification | `CMD-005` | Pass | `runs/v0.1-acceptance-green/cmd-005-with-skill-output.md`, `runs/v0.1-acceptance-green/cmd-005-with-skill-output.diff` |
|
|
79
|
-
| Commit closeout | `CMT-005` | Pass | `runs/v0.1-acceptance-green/cmt-005-with-skill-output.md`, `runs/v0.1-acceptance-green/cmt-005-with-skill-output.diff` |
|
|
80
|
-
| Capture decisions | `CAP-002` | Pass | `runs/v0.1-acceptance-green/cap-002-with-skill-output.md`, `runs/v0.1-acceptance-green/cap-002-with-skill-output.diff` |
|
|
81
|
-
| Bypass boundary | `BYP-002` | Pass | `runs/v0.1-acceptance-green/byp-002-with-skill-output.md`, `runs/v0.1-acceptance-green/byp-002-with-skill-output.diff` |
|
|
82
|
-
| Command surface | `audit-command-surface.sh` | Pass | command output above |
|
|
83
|
-
|
|
84
|
-
## Fix History
|
|
85
|
-
|
|
86
|
-
The first acceptance run, saved under `runs/v0.1-acceptance/`, failed 13 of 15 fixture evals:
|
|
87
|
-
|
|
88
|
-
- `WSP-006`: stopped correctly but did not ask the focused owner/API decision question.
|
|
89
|
-
- `CAP-002`: created `CONTEXT.md` instead of asking where to capture durable memory.
|
|
90
|
-
|
|
91
|
-
Fixes applied:
|
|
92
|
-
|
|
93
|
-
- `write-spec`: strict-workflow stops for missing owner-owned decisions must end with a direct question naming the missing owner and decisions.
|
|
94
|
-
- `capture-decisions`: missing `CONTEXT.md`, glossary, ADR/spec/product docs, or decision-note folders are not existing owners; explicit "no convention" evidence blocks fallback edits.
|
|
95
|
-
|
|
96
|
-
An intermediate clean run then exposed a third gap:
|
|
97
|
-
|
|
98
|
-
- `XPL-004`: `execute-plan` changed the planned verification script after the check failed.
|
|
99
|
-
|
|
100
|
-
Fix applied:
|
|
101
|
-
|
|
102
|
-
- `execute-plan`: after a planned verification command fails, changing the verifier/tests/docs/policy/source truth is a plan revision, even if the verifier appears wrong or the user said to fix whatever is needed.
|
|
103
|
-
|
|
104
|
-
Focused reruns passed for the changed behavior before the final full suite:
|
|
105
|
-
|
|
106
|
-
- `WSP-006`: `runs/v0.1-fixes/wsp-006-with-skill-output.md`
|
|
107
|
-
- `CAP-002`: `runs/v0.1-fixes/cap-002-stability-1-with-skill-output.md`, `runs/v0.1-fixes/cap-002-stability-2-with-skill-output.md`
|
|
108
|
-
- `XPL-004`: `runs/v0.1-fixes/xpl-004-rerun-2-with-skill-output.md`
|
|
109
|
-
|
|
110
|
-
## Blocking Checks
|
|
111
|
-
|
|
112
|
-
Final diff evidence matched expected fixture behavior:
|
|
113
|
-
|
|
114
|
-
- `CAP-002` diff size: `0`; no durable destination was created.
|
|
115
|
-
- `WSP-006` diff size: `0`; no public API spec was written.
|
|
116
|
-
- `XPL-004` diff only changed `src/settings-page.ts`; it did not edit billing or the check script.
|
|
117
|
-
|
|
118
|
-
Final command-surface audit passed, and root-output check showed only the expected root files.
|
|
119
|
-
|
|
120
|
-
Prepublish diff evidence also matched expected fixture behavior:
|
|
121
|
-
|
|
122
|
-
- `CAP-002`, `WSP-006`, `AON-001`, `BYP-002`, `GRC-001`, `IVG-001`, `CMD-005`, `REV-002`, `STP-002`, and `MODE-001` had empty diffs.
|
|
123
|
-
- `XPL-004` changed only `src/settings-page.ts`; it did not edit billing or the check script.
|
|
124
|
-
- `CMT-005` preserved the staged fixture conflict and made no commit.
|
|
125
|
-
- `STP-001`, `MODE-005`, and `WPL-005` produced the expected setup, typo-fix, and delegated-plan artifacts.
|
|
126
|
-
|
|
127
|
-
## Non-Blocking Notes
|
|
128
|
-
|
|
129
|
-
- The harness prints the observed model in runner logs but does not pin it in the suite definition.
|
|
130
|
-
- Saved run outputs are under the ignored `evals/runs/` tree. This report is the durable tracked artifact.
|
|
@@ -1,45 +0,0 @@
|
|
|
1
|
-
# Freeflow Command Surface Eval Report - 1
|
|
2
|
-
|
|
3
|
-
Date: 2026-05-25
|
|
4
|
-
|
|
5
|
-
## Scope
|
|
6
|
-
|
|
7
|
-
Audited the slash-like command surface after mode setup work.
|
|
8
|
-
|
|
9
|
-
The question was whether Freeflow already has all plugin slash commands and whether they work when users call them.
|
|
10
|
-
|
|
11
|
-
## Result
|
|
12
|
-
|
|
13
|
-
Freeflow has the documented command surface as model-routed skill calls, not native host slash-command handlers.
|
|
14
|
-
|
|
15
|
-
Added `command-surface.json` as the canonical command map:
|
|
16
|
-
|
|
17
|
-
- 3 mode commands route to `mode-contract`
|
|
18
|
-
- 13 direct skill calls map to existing skills
|
|
19
|
-
- native slash handlers are explicitly `false`
|
|
20
|
-
|
|
21
|
-
Added `evals/scripts/audit-command-surface.sh` to verify:
|
|
22
|
-
|
|
23
|
-
- registry JSON parses
|
|
24
|
-
- every direct command maps to an existing skill
|
|
25
|
-
- every mode command routes to `mode-contract`
|
|
26
|
-
- command docs mention every registered command
|
|
27
|
-
- plugin manifest does not pretend native slash handlers exist
|
|
28
|
-
|
|
29
|
-
## Fix
|
|
30
|
-
|
|
31
|
-
`docs/plugin-contract.md` omitted `/commit-work` even though the skill exists and the inventory listed it. The contract now includes `/commit-work`.
|
|
32
|
-
|
|
33
|
-
The contract also states that current candidate plugins may route commands through skill activation and model behavior rather than native host slash-command handlers.
|
|
34
|
-
|
|
35
|
-
## Verification
|
|
36
|
-
|
|
37
|
-
```text
|
|
38
|
-
Command surface audit passed: 13 direct skill calls, 3 mode commands, native slash handlers disabled.
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
## Decision
|
|
42
|
-
|
|
43
|
-
Do not add native slash-command runtime yet. Current evidence only supports skill/model-routed commands.
|
|
44
|
-
|
|
45
|
-
Next useful eval: pick one direct command, likely `/execute-plan` or `/commit-work`, and test that the model treats the slash call as permission to enter that segment while still preserving user-owned decision gates.
|