@hassangameryt/freeflow 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plugins/marketplace.json +1 -1
- package/.claude-plugin/marketplace.json +2 -2
- package/{plugins/freeflow/.claude-plugin → .claude-plugin}/plugin.json +1 -1
- package/{plugins/freeflow/.codex-plugin → .codex-plugin}/plugin.json +1 -1
- package/README.md +25 -16
- package/assets/freeflow-plugin-icon.png +0 -0
- package/{plugins/freeflow/command-surface.json → command-surface.json} +10 -10
- package/evals/README.md +32 -0
- package/evals/fixtures/copy-conventions.md +31 -0
- package/evals/fixtures/output-router/README.md +12 -0
- package/evals/fixtures/output-router/failed-command-output.txt +9 -0
- package/evals/fixtures/output-router/large-router-manual.md +63 -0
- package/evals/fixtures/output-router/native-large-output.txt +40 -0
- package/evals/fixtures/output-router/noisy-test-output.txt +30 -0
- package/evals/fixtures/output-router/verification-output.txt +8 -0
- package/evals/fixtures/tiny-artifact-review-app/README.md +5 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md +21 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/failed-payment-grace-period.md +24 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/immediate-downgrade.md +17 -0
- package/evals/fixtures/tiny-artifact-review-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh +14 -0
- package/evals/fixtures/tiny-commit-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-app/docs/specs/settings-title.md +5 -0
- package/evals/fixtures/tiny-commit-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-everything-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-everything-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-everything-app/docs/specs/settings-title.md +5 -0
- package/evals/fixtures/tiny-commit-everything-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-everything-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-everything-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-staged-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-staged-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-staged-app/docs/specs/settings-title.md +5 -0
- package/evals/fixtures/tiny-commit-staged-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-staged-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-staged-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/billing-policy.md +3 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/release-notes.md +3 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-dashboard-stale-app/README.md +11 -0
- package/evals/fixtures/tiny-dashboard-stale-app/src/api-client.ts +13 -0
- package/evals/fixtures/tiny-dashboard-stale-app/src/dashboard-data.ts +19 -0
- package/evals/fixtures/tiny-dashboard-stale-app/tests/dashboard-data.test.md +9 -0
- package/evals/fixtures/tiny-decision-destination-app/README.md +6 -0
- package/evals/fixtures/tiny-decision-destination-app/src/risk.ts +8 -0
- package/evals/fixtures/tiny-decision-destination-app/tests/risk.test.md +4 -0
- package/evals/fixtures/tiny-design-depth-app/README.md +15 -0
- package/evals/fixtures/tiny-design-depth-app/src/routes.ts +25 -0
- package/evals/fixtures/tiny-design-depth-app/src/sendEmail.ts +3 -0
- package/evals/fixtures/tiny-design-depth-app/src/sendSms.ts +3 -0
- package/evals/fixtures/tiny-design-depth-app/src/telemetry.ts +3 -0
- package/evals/fixtures/tiny-design-depth-app/tests/routes.test.md +12 -0
- package/evals/fixtures/tiny-eval-method-app/README.md +8 -0
- package/evals/fixtures/tiny-eval-method-app/evals/reports/review-pr-eval-result.md +5 -0
- package/evals/fixtures/tiny-eval-method-app/evals/reports/triage-question-failure.md +17 -0
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.diff +11 -0
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.md +3 -0
- package/evals/fixtures/tiny-eval-method-app/skills/review-pr/SKILL.md +10 -0
- package/evals/fixtures/tiny-eval-method-app/skills/triage-question/SKILL.md +10 -0
- package/evals/fixtures/tiny-execute-plan-app/README.md +5 -0
- package/evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh +5 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/immediate-downgrade-plan.md +11 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-plan.md +12 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md +15 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/unverified-billing-plan.md +9 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/failed-payment-grace-period.md +9 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/settings-title.md +9 -0
- package/evals/fixtures/tiny-execute-plan-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-execute-plan-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-execute-plan-app/tests/billing.test.md +7 -0
- package/evals/fixtures/tiny-execute-plan-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-execute-review-loop-app/README.md +5 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/billing-policy.md +9 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/plans/settings-title-plan.md +12 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/previous-agent-summary.md +15 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/review-pass-1-non-pass.md +11 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/specs/settings-title.md +9 -0
- package/evals/fixtures/tiny-execute-review-loop-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-execute-review-loop-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-execute-review-loop-app/tests/billing.test.md +7 -0
- package/evals/fixtures/tiny-execute-review-loop-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-flaky-session-app/README.md +10 -0
- package/evals/fixtures/tiny-flaky-session-app/src/session-timeout.ts +13 -0
- package/evals/fixtures/tiny-flaky-session-app/tests/session-timeout.test.md +8 -0
- package/evals/fixtures/tiny-handoff-app/README.md +5 -0
- package/evals/fixtures/tiny-handoff-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-handoff-app/docs/handoffs/old-agent-handoff.md +8 -0
- package/evals/fixtures/tiny-handoff-app/docs/plans/billing-retry-plan.md +11 -0
- package/evals/fixtures/tiny-handoff-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-handoff-report-app/README.md +3 -0
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.diff +1 -0
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.md +1 -0
- package/evals/fixtures/tiny-handoff-report-app/evals/scores/hof-003-score.txt +4 -0
- package/evals/fixtures/tiny-plan-app/README.md +5 -0
- package/evals/fixtures/tiny-plan-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-plan-app/docs/specs/failed-payment-grace-period.md +25 -0
- package/evals/fixtures/tiny-plan-app/src/billing.ts +17 -0
- package/evals/fixtures/tiny-plan-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-plan-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-plan-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-plan-strict-api-app/README.md +3 -0
- package/evals/fixtures/tiny-plan-strict-api-app/docs/billing-policy.md +5 -0
- package/evals/fixtures/tiny-plan-strict-api-app/docs/specs/billing-webhook-api.md +25 -0
- package/evals/fixtures/tiny-plan-strict-api-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-plan-strict-api-app/src/webhooks.ts +3 -0
- package/evals/fixtures/tiny-plan-strict-api-app/tests/billing-webhook.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/.freeflow/config.json +3 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/AGENTS.md +15 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/README.md +16 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/AGENTS.md +15 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/README.md +16 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/.freeflow/config.json +3 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/AGENTS.md +15 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/README.md +16 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-report-performance-app/README.md +11 -0
- package/evals/fixtures/tiny-report-performance-app/docs/performance.md +10 -0
- package/evals/fixtures/tiny-report-performance-app/src/report-summary.ts +29 -0
- package/evals/fixtures/tiny-report-performance-app/tests/report-performance.test.md +8 -0
- package/evals/fixtures/tiny-review-request-app/README.md +3 -0
- package/evals/fixtures/tiny-review-request-app/docs/billing-policy.md +5 -0
- package/evals/fixtures/tiny-review-request-app/docs/plans/billing-webhook-api.md +19 -0
- package/evals/fixtures/tiny-review-request-app/docs/review/previous-agent-summary.md +20 -0
- package/evals/fixtures/tiny-review-request-app/docs/specs/billing-webhook-api.md +25 -0
- package/evals/fixtures/tiny-review-request-app/src/billing.ts +11 -0
- package/evals/fixtures/tiny-review-request-app/src/webhooks.ts +19 -0
- package/evals/fixtures/tiny-review-request-app/tests/billing-webhook.test.md +8 -0
- package/evals/fixtures/tiny-review-work-app/README.md +5 -0
- package/evals/fixtures/tiny-review-work-app/docs/billing-policy.md +9 -0
- package/evals/fixtures/tiny-review-work-app/docs/plans/settings-title-plan.md +12 -0
- package/evals/fixtures/tiny-review-work-app/docs/review/previous-agent-summary.md +15 -0
- package/evals/fixtures/tiny-review-work-app/docs/specs/settings-title.md +9 -0
- package/evals/fixtures/tiny-review-work-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-review-work-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-review-work-app/tests/billing.test.md +7 -0
- package/evals/fixtures/tiny-review-work-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-saas-app/README.md +16 -0
- package/evals/fixtures/tiny-saas-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-saas-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-saas-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-saas-app/docs/specs/settings-title.md +13 -0
- package/evals/fixtures/tiny-saas-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-saas-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-saas-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-saas-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-saas-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-saas-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-saas-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-setup-agents-app/AGENTS.md +6 -0
- package/evals/fixtures/tiny-setup-both-app/AGENTS.md +5 -0
- package/evals/fixtures/tiny-setup-both-app/CLAUDE.md +5 -0
- package/evals/fixtures/tiny-setup-claude-app/CLAUDE.md +6 -0
- package/evals/fixtures/tiny-setup-codex-rules-app/.codex/rules/shell.rules +5 -0
- package/evals/fixtures/tiny-setup-codex-rules-app/AGENTS.md +6 -0
- package/evals/fixtures/tiny-setup-conflict-app/AGENTS.md +6 -0
- package/evals/fixtures/tiny-setup-existing-block-app/AGENTS.md +14 -0
- package/evals/fixtures/tiny-setup-four-rules-app/CLAUDE.md +6 -0
- package/evals/fixtures/tiny-skill-dev-app/README.md +10 -0
- package/evals/fixtures/tiny-skill-dev-app/evals/prompts/review-pr-001.txt +8 -0
- package/evals/fixtures/tiny-skill-dev-app/evals/reports/review-pr-failure.md +17 -0
- package/evals/fixtures/tiny-skill-dev-app/skills/review-pr/SKILL.md +13 -0
- package/evals/fixtures/tiny-team-app/README.md +5 -0
- package/evals/fixtures/tiny-team-app/docs/product.md +6 -0
- package/evals/fixtures/tiny-team-app/src/teams.ts +9 -0
- package/evals/fixtures/tiny-team-app/src/users.ts +5 -0
- package/evals/fixtures/tiny-team-app/tests/teams.test.md +7 -0
- package/evals/fixtures/tiny-verify-app/README.md +5 -0
- package/evals/fixtures/tiny-verify-app/src/settings-nav.ts +1 -0
- package/evals/fixtures/tiny-verify-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-verify-app/tests/settings.test.md +7 -0
- package/evals/prompts/aon-001.txt +2 -0
- package/evals/prompts/byp-001.txt +2 -0
- package/evals/prompts/byp-002.txt +2 -0
- package/evals/prompts/cap-001.txt +3 -0
- package/evals/prompts/cap-002.txt +3 -0
- package/evals/prompts/cmd-001.txt +3 -0
- package/evals/prompts/cmd-002.txt +3 -0
- package/evals/prompts/cmd-003.txt +6 -0
- package/evals/prompts/cmd-004.txt +5 -0
- package/evals/prompts/cmd-005.txt +5 -0
- package/evals/prompts/cmd-006.txt +5 -0
- package/evals/prompts/cmd-007.txt +5 -0
- package/evals/prompts/cmd-008.txt +5 -0
- package/evals/prompts/cmd-009.txt +8 -0
- package/evals/prompts/cmd-010.txt +5 -0
- package/evals/prompts/cmd-011.txt +5 -0
- package/evals/prompts/cmd-012.txt +5 -0
- package/evals/prompts/cmd-013.txt +5 -0
- package/evals/prompts/cmd-014.txt +7 -0
- package/evals/prompts/cmd-015.txt +7 -0
- package/evals/prompts/cmd-016.txt +5 -0
- package/evals/prompts/cmt-001.txt +1 -0
- package/evals/prompts/cmt-002.txt +1 -0
- package/evals/prompts/cmt-003.txt +1 -0
- package/evals/prompts/cmt-004.txt +1 -0
- package/evals/prompts/cmt-005.txt +1 -0
- package/evals/prompts/dfd-001.txt +3 -0
- package/evals/prompts/dia-001.txt +3 -0
- package/evals/prompts/dia-002.txt +3 -0
- package/evals/prompts/dia-003.txt +3 -0
- package/evals/prompts/dis-001.txt +5 -0
- package/evals/prompts/dis-002.txt +5 -0
- package/evals/prompts/esk-001.txt +5 -0
- package/evals/prompts/esk-002.txt +5 -0
- package/evals/prompts/esk-003.txt +5 -0
- package/evals/prompts/grc-001.txt +3 -0
- package/evals/prompts/hof-001.txt +3 -0
- package/evals/prompts/hof-002.txt +1 -0
- package/evals/prompts/hof-003.txt +1 -0
- package/evals/prompts/hof-004.txt +1 -0
- package/evals/prompts/hof-005.txt +1 -0
- package/evals/prompts/ivg-001.txt +1 -0
- package/evals/prompts/ivg-002.txt +11 -0
- package/evals/prompts/ivg-003.txt +1 -0
- package/evals/prompts/ivg-004.txt +1 -0
- package/evals/prompts/mode-001.txt +3 -0
- package/evals/prompts/mode-002.txt +1 -0
- package/evals/prompts/mode-003.txt +3 -0
- package/evals/prompts/mode-004.txt +3 -0
- package/evals/prompts/mode-005.txt +3 -0
- package/evals/prompts/mode-006.txt +3 -0
- package/evals/prompts/mode-007.txt +3 -0
- package/evals/prompts/otr-001.txt +3 -0
- package/evals/prompts/otr-002.txt +3 -0
- package/evals/prompts/rar-001.txt +5 -0
- package/evals/prompts/rar-002.txt +3 -0
- package/evals/prompts/rar-003.txt +6 -0
- package/evals/prompts/rar-004.txt +15 -0
- package/evals/prompts/rar-005.txt +10 -0
- package/evals/prompts/rbr-001.txt +3 -0
- package/evals/prompts/rbr-002.txt +3 -0
- package/evals/prompts/rbr-003.txt +7 -0
- package/evals/prompts/rev-001.txt +3 -0
- package/evals/prompts/rev-002.txt +5 -0
- package/evals/prompts/rev-003.txt +6 -0
- package/evals/prompts/rev-004.txt +1 -0
- package/evals/prompts/rev-005.txt +15 -0
- package/evals/prompts/rev-006.txt +5 -0
- package/evals/prompts/stp-001.txt +1 -0
- package/evals/prompts/stp-002.txt +1 -0
- package/evals/prompts/stp-003.txt +1 -0
- package/evals/prompts/stp-004.txt +1 -0
- package/evals/prompts/stp-005.txt +1 -0
- package/evals/prompts/stp-006.txt +1 -0
- package/evals/prompts/stp-007.txt +1 -0
- package/evals/prompts/stp-008.txt +1 -0
- package/evals/prompts/stp-009.txt +1 -0
- package/evals/prompts/stp-010.txt +1 -0
- package/evals/prompts/stp-011.txt +1 -0
- package/evals/prompts/vfy-003.txt +1 -0
- package/evals/prompts/wpl-001.txt +3 -0
- package/evals/prompts/wpl-002.txt +5 -0
- package/evals/prompts/wpl-003.txt +3 -0
- package/evals/prompts/wpl-004.txt +5 -0
- package/evals/prompts/wpl-005.txt +1 -0
- package/evals/prompts/wrc-001.txt +10 -0
- package/evals/prompts/wrc-002.txt +3 -0
- package/evals/prompts/wsk-001.txt +6 -0
- package/evals/prompts/wsp-001.txt +5 -0
- package/evals/prompts/wsp-002.txt +15 -0
- package/evals/prompts/wsp-003.txt +3 -0
- package/evals/prompts/wsp-004.txt +15 -0
- package/evals/prompts/wsp-005.txt +5 -0
- package/evals/prompts/wsp-006.txt +12 -0
- package/evals/prompts/wsp-007.txt +9 -0
- package/evals/prompts/wsp-008.txt +13 -0
- package/evals/prompts/xpl-001.txt +1 -0
- package/evals/prompts/xpl-002.txt +3 -0
- package/evals/prompts/xpl-003.txt +3 -0
- package/evals/prompts/xpl-004.txt +3 -0
- package/evals/prompts/xpl-005.txt +5 -0
- package/evals/prompts/xpl-006.txt +3 -0
- package/evals/registries/adversarial-evals.json +39 -0
- package/evals/registries/fixture-evals.json +2513 -0
- package/evals/registries/handoff-evals.json +74 -0
- package/evals/registries/legacy-evals.json +147 -0
- package/evals/registries/skill-evidence.json +193 -0
- package/evals/registries/verify-work-evals.json +46 -0
- package/evals/reports/acceptance/v0.1-acceptance-report.md +130 -0
- package/evals/reports/by-command-surface/command-surface-1-report.md +45 -0
- package/evals/reports/by-command-surface/command-surface-10-report.md +76 -0
- package/evals/reports/by-command-surface/command-surface-11-report.md +77 -0
- package/evals/reports/by-command-surface/command-surface-12-report.md +91 -0
- package/evals/reports/by-command-surface/command-surface-13-report.md +64 -0
- package/evals/reports/by-command-surface/command-surface-14-report.md +66 -0
- package/evals/reports/by-command-surface/command-surface-15-report.md +124 -0
- package/evals/reports/by-command-surface/command-surface-16-report.md +39 -0
- package/evals/reports/by-command-surface/command-surface-2-report.md +80 -0
- package/evals/reports/by-command-surface/command-surface-3-report.md +86 -0
- package/evals/reports/by-command-surface/command-surface-4-report.md +76 -0
- package/evals/reports/by-command-surface/command-surface-5-report.md +80 -0
- package/evals/reports/by-command-surface/command-surface-6-report.md +74 -0
- package/evals/reports/by-command-surface/command-surface-7-report.md +84 -0
- package/evals/reports/by-command-surface/command-surface-8-report.md +79 -0
- package/evals/reports/by-command-surface/command-surface-9-report.md +79 -0
- package/evals/reports/by-command-surface/command-surface-matrix.md +91 -0
- package/evals/reports/by-skill/bypass-1-report.md +124 -0
- package/evals/reports/by-skill/capture-decisions-1-report.md +99 -0
- package/evals/reports/by-skill/capture-decisions-2-report.md +102 -0
- package/evals/reports/by-skill/commit-work-1-report.md +165 -0
- package/evals/reports/by-skill/commit-work-2-report.md +109 -0
- package/evals/reports/by-skill/design-for-depth-1-report.md +95 -0
- package/evals/reports/by-skill/diagnose-failure-1-report.md +91 -0
- package/evals/reports/by-skill/diagnose-failure-2-report.md +94 -0
- package/evals/reports/by-skill/discover-1-report.md +118 -0
- package/evals/reports/by-skill/evaluate-skill-2-report.md +71 -0
- package/evals/reports/by-skill/execute-plan-1-report.md +103 -0
- package/evals/reports/by-skill/execute-plan-2-report.md +112 -0
- package/evals/reports/by-skill/execute-plan-3-report.md +87 -0
- package/evals/reports/by-skill/grill-context-1-report.md +103 -0
- package/evals/reports/by-skill/handoff-1-report.md +123 -0
- package/evals/reports/by-skill/handoff-2-report.md +105 -0
- package/evals/reports/by-skill/handoff-3-report.md +100 -0
- package/evals/reports/by-skill/handoff-7-report.md +77 -0
- package/evals/reports/by-skill/interview-gate-1-report.md +63 -0
- package/evals/reports/by-skill/interview-gate-2-report.md +105 -0
- package/evals/reports/by-skill/mode-contract-1-report.md +74 -0
- package/evals/reports/by-skill/mode-contract-2-report.md +87 -0
- package/evals/reports/by-skill/mode-contract-3-report.md +65 -0
- package/evals/reports/by-skill/mode-contract-4-report.md +71 -0
- package/evals/reports/by-skill/mode-contract-5-report.md +70 -0
- package/evals/reports/by-skill/output-router-1-report.md +76 -0
- package/evals/reports/by-skill/output-router-2-report.md +65 -0
- package/evals/reports/by-skill/research-1-report.md +144 -0
- package/evals/reports/by-skill/research-brief-1-report.md +94 -0
- package/evals/reports/by-skill/research-brief-2-report.md +75 -0
- package/evals/reports/by-skill/review-artifact-1-report.md +91 -0
- package/evals/reports/by-skill/review-artifact-2-report.md +79 -0
- package/evals/reports/by-skill/review-artifact-3-report.md +120 -0
- package/evals/reports/by-skill/review-artifact-4-report.md +89 -0
- package/evals/reports/by-skill/review-artifact-5-report.md +66 -0
- package/evals/reports/by-skill/review-work-1-report.md +85 -0
- package/evals/reports/by-skill/review-work-2-report.md +87 -0
- package/evals/reports/by-skill/review-work-3-report.md +104 -0
- package/evals/reports/by-skill/review-work-4-report.md +108 -0
- package/evals/reports/by-skill/review-work-5-report.md +73 -0
- package/evals/reports/by-skill/review-work-6-report.md +66 -0
- package/evals/reports/by-skill/setup-freeflow-1-report.md +199 -0
- package/evals/reports/by-skill/setup-freeflow-2-report.md +105 -0
- package/evals/reports/by-skill/setup-freeflow-3-report.md +105 -0
- package/evals/reports/by-skill/setup-freeflow-4-report.md +88 -0
- package/evals/reports/by-skill/setup-freeflow-5-report.md +115 -0
- package/evals/reports/by-skill/skill-authoring-1-report.md +81 -0
- package/evals/reports/by-skill/verify-work-1-report.md +111 -0
- package/evals/reports/by-skill/verify-work-2-report.md +120 -0
- package/evals/reports/by-skill/workflow-1-report.md +65 -0
- package/evals/reports/by-skill/write-plan-1-report.md +123 -0
- package/evals/reports/by-skill/write-plan-2-report.md +98 -0
- package/evals/reports/by-skill/write-plan-3-report.md +110 -0
- package/evals/reports/by-skill/write-spec-1-report.md +88 -0
- package/evals/reports/by-skill/write-spec-2-report.md +89 -0
- package/evals/reports/by-skill/write-spec-3-report.md +90 -0
- package/evals/reports/by-skill/write-spec-4-report.md +167 -0
- package/evals/reports/harness/claude-harness-1-report.md +103 -0
- package/evals/reports/harness/eval-harness-1-report.md +91 -0
- package/evals/reports/iterations/iteration-1-report.md +71 -0
- package/evals/reports/iterations/iteration-2-report.md +78 -0
- package/evals/reports/iterations/iteration-3-report.md +118 -0
- package/evals/reports/iterations/iteration-4-report.md +131 -0
- package/evals/reports/iterations/iteration-5-report.md +149 -0
- package/evals/reports/iterations/iteration-6-report.md +136 -0
- package/evals/reports/runtime/always-on-runtime-1-report.md +107 -0
- package/evals/reports/runtime/context-mode-normalized-benchmark-1-report.md +52 -0
- package/evals/reports/runtime/context-mode-real-deep-baseline-1-report.md +410 -0
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md +203 -0
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md +51 -0
- package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md +99 -0
- package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md +166 -0
- package/evals/reports/runtime/eryx-python-proof-spike-1-report.md +96 -0
- package/evals/reports/runtime/eryx-python-proof-spike-2-report.md +151 -0
- package/evals/reports/runtime/eryx-python-script-derive-execution-1-report.md +233 -0
- package/evals/reports/runtime/jq-script-derive-execution-1-report.md +215 -0
- package/evals/reports/runtime/jq-wasm-proof-spike-1-report.md +67 -0
- package/evals/reports/runtime/output-router-benchmark-1-report.md +81 -0
- package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md +35 -0
- package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md +150 -0
- package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md +46 -0
- package/evals/reports/runtime/output-router-command-benchmark-1-report.md +62 -0
- package/evals/reports/runtime/output-router-index-benchmark-1-report.md +64 -0
- package/evals/reports/runtime/output-router-regression-1-report.md +81 -0
- package/evals/reports/runtime/output-router-transform-eval-1-report.md +28 -0
- package/evals/reports/runtime/pi-observed-routing-eval-1-report.md +44 -0
- package/evals/reports/runtime/quickjs-script-derive-execution-1-report.md +159 -0
- package/evals/reports/runtime/quickjs-wasi-proof-spike-1-report.md +55 -0
- package/evals/reports/runtime/render-format-row-protocol-1-report.md +58 -0
- package/evals/reports/runtime/script-sandbox-adapter-selection-review-1-report.md +84 -0
- package/evals/reports/runtime/script-sandbox-probe-resource-hardening-1-report.md +51 -0
- package/evals/reports/runtime/storage-policy-benchmark-1-report.md +176 -0
- package/evals/reports/runtime/vault-index-storage-spike-1-report.md +39 -0
- package/evals/reports/runtime/workflow-context-hook-1-report.md +82 -0
- package/evals/runbooks/adversarial-eval-runbook.md +28 -0
- package/evals/runbooks/fixture-eval-runbook.md +137 -0
- package/evals/runbooks/handoff-eval-runbook.md +22 -0
- package/evals/runbooks/verify-work-eval-runbook.md +27 -0
- package/evals/scripts/audit-command-surface.sh +160 -0
- package/evals/scripts/check-activation-contract.sh +121 -0
- package/evals/scripts/check-runtime-context-hook.sh +183 -0
- package/evals/scripts/grade-fixture-eval.sh +446 -0
- package/evals/scripts/run-claude-fixture-eval.sh +26 -0
- package/evals/scripts/run-codex-fixture-eval.sh +26 -0
- package/evals/scripts/run-eryx-python-proof-spike.js +559 -0
- package/evals/scripts/run-fixture-eval-by-id.sh +23 -0
- package/evals/scripts/run-fixture-eval.sh +464 -0
- package/evals/scripts/run-jq-wasm-proof-spike.js +299 -0
- package/evals/scripts/run-output-router-transform-eval.js +115 -0
- package/evals/scripts/run-pi-observed-routing-eval.js +478 -0
- package/evals/scripts/run-quickjs-wasi-proof-spike.js +339 -0
- package/evals/scripts/skill-evidence.sh +316 -0
- package/evals/scripts/validate-release-metadata.sh +486 -0
- package/evals/suites/v0.1-acceptance-suite.md +131 -0
- package/{plugins/freeflow/hooks → hooks}/freeflow-runtime-context.mjs +65 -14
- package/package.json +46 -13
- package/pi-extension/dist/host-producer-identification.js +186 -0
- package/pi-extension/dist/index.js +85 -0
- package/pi-extension/dist/mcp-config.js +30 -0
- package/pi-extension/dist/native-safety-net.js +209 -0
- package/pi-extension/dist/observed-tool-routing.js +114 -0
- package/pi-extension/dist/renderers.js +835 -0
- package/pi-extension/dist/router-tools.js +438 -0
- package/pi-extension/dist/runtime-context.js +288 -0
- package/pi-extension/dist/schemas.js +254 -0
- package/pi-extension/dist/status.js +449 -0
- package/pi-extension/dist/utils.js +467 -0
- package/pi-extension/src/host-producer-identification.ts +226 -0
- package/pi-extension/src/index.ts +121 -0
- package/pi-extension/src/mcp-config.ts +32 -0
- package/pi-extension/src/native-safety-net.ts +252 -0
- package/pi-extension/src/observed-tool-routing.ts +126 -0
- package/pi-extension/src/renderers.ts +917 -0
- package/pi-extension/src/router-tools.ts +478 -0
- package/pi-extension/src/runtime-context.ts +347 -0
- package/pi-extension/src/schemas.ts +259 -0
- package/pi-extension/src/status.ts +505 -0
- package/pi-extension/src/utils.ts +492 -0
- package/pi-extension/tsconfig.json +16 -0
- package/plugin-docs/README.md +40 -0
- package/plugin-docs/adr/0002-evals-before-hooks.md +17 -0
- package/{plugins/freeflow/docs → plugin-docs}/adr/0003-release-boundary.md +2 -2
- package/plugin-docs/adr/0004-discover-replaces-shallow-discovery-skills.md +26 -0
- package/plugin-docs/adr/README.md +10 -0
- package/plugin-docs/architecture.md +89 -0
- package/plugin-docs/output-router.md +396 -0
- package/plugin-docs/release-evidence.md +77 -0
- package/{plugins/freeflow/docs → plugin-docs}/skills.md +7 -7
- package/{plugins/freeflow/docs → plugin-docs}/workflow.md +4 -4
- package/router/dist/benchmarks/benchmark-harness.d.ts +32 -0
- package/router/dist/benchmarks/benchmark-harness.js +92 -0
- package/router/dist/benchmarks/benchmarks.d.ts +102 -0
- package/router/dist/benchmarks/benchmarks.js +880 -0
- package/router/dist/benchmarks/codex-qa-benchmarks.d.ts +91 -0
- package/router/dist/benchmarks/codex-qa-benchmarks.js +461 -0
- package/router/dist/benchmarks/command-benchmarks.d.ts +128 -0
- package/router/dist/benchmarks/command-benchmarks.js +653 -0
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.d.ts +74 -0
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.js +707 -0
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.d.ts +124 -0
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.js +1295 -0
- package/router/dist/benchmarks/context-saving-baseline.d.ts +54 -0
- package/router/dist/benchmarks/context-saving-baseline.js +439 -0
- package/router/dist/benchmarks/index-benchmarks.d.ts +98 -0
- package/router/dist/benchmarks/index-benchmarks.js +762 -0
- package/router/dist/benchmarks/storage-policy-benchmarks.d.ts +90 -0
- package/router/dist/benchmarks/storage-policy-benchmarks.js +538 -0
- package/router/dist/benchmarks/vault-index-benchmarks.d.ts +47 -0
- package/router/dist/benchmarks/vault-index-benchmarks.js +257 -0
- package/router/dist/config/config.d.ts +96 -0
- package/router/dist/config/config.js +453 -0
- package/router/dist/config/router-contract.d.ts +10 -0
- package/router/dist/config/router-contract.js +54 -0
- package/router/dist/config/schema.d.ts +18 -0
- package/router/dist/config/schema.js +600 -0
- package/router/dist/config/types.d.ts +521 -0
- package/router/dist/config/types.js +30 -0
- package/router/dist/evidence/bounded-evidence.d.ts +40 -0
- package/router/dist/evidence/bounded-evidence.js +310 -0
- package/router/dist/evidence/evidence-range-selector.d.ts +22 -0
- package/router/dist/evidence/evidence-range-selector.js +231 -0
- package/router/dist/evidence/evidence-search.d.ts +25 -0
- package/router/dist/evidence/evidence-search.js +466 -0
- package/router/dist/evidence/evidence.d.ts +38 -0
- package/router/dist/evidence/evidence.js +196 -0
- package/router/dist/evidence/failure-contracts.d.ts +27 -0
- package/router/dist/evidence/failure-contracts.js +160 -0
- package/router/dist/evidence/line-ranges.d.ts +22 -0
- package/router/dist/evidence/line-ranges.js +22 -0
- package/router/dist/experiments/local-index.d.ts +58 -0
- package/router/dist/experiments/local-index.js +562 -0
- package/router/dist/index.d.ts +30 -0
- package/router/dist/index.js +32 -0
- package/router/dist/processing/engine.d.ts +108 -0
- package/router/dist/processing/engine.js +435 -0
- package/router/dist/processing/reducers.d.ts +266 -0
- package/router/dist/processing/reducers.js +1495 -0
- package/router/dist/processing/renderers.d.ts +25 -0
- package/router/dist/processing/renderers.js +186 -0
- package/router/dist/processing/scripts.d.ts +78 -0
- package/router/dist/processing/scripts.js +333 -0
- package/router/dist/repo/repo-traversal.d.ts +18 -0
- package/router/dist/repo/repo-traversal.js +242 -0
- package/router/dist/routing/observed-reducers.d.ts +35 -0
- package/router/dist/routing/observed-reducers.js +647 -0
- package/router/dist/routing/observed-routing.d.ts +17 -0
- package/router/dist/routing/observed-routing.js +377 -0
- package/router/dist/routing/parsers.d.ts +16 -0
- package/router/dist/routing/parsers.js +473 -0
- package/router/dist/routing/run-filters.d.ts +56 -0
- package/router/dist/routing/run-filters.js +230 -0
- package/router/dist/routing/run-reducers.d.ts +30 -0
- package/router/dist/routing/run-reducers.js +110 -0
- package/router/dist/sandbox/adapter-roots.d.ts +38 -0
- package/router/dist/sandbox/adapter-roots.js +64 -0
- package/router/dist/sandbox/eryx-python-adapter.d.ts +9 -0
- package/router/dist/sandbox/eryx-python-adapter.js +797 -0
- package/router/dist/sandbox/jq-wasm-adapter.d.ts +9 -0
- package/router/dist/sandbox/jq-wasm-adapter.js +421 -0
- package/router/dist/sandbox/quickjs-wasi-adapter.d.ts +9 -0
- package/router/dist/sandbox/quickjs-wasi-adapter.js +381 -0
- package/router/dist/sandbox/script-sandbox.d.ts +113 -0
- package/router/dist/sandbox/script-sandbox.js +303 -0
- package/router/dist/setup/script-transform-adapters.d.ts +28 -0
- package/router/dist/setup/script-transform-adapters.js +259 -0
- package/router/dist/tools/batch.d.ts +22 -0
- package/router/dist/tools/batch.js +509 -0
- package/router/dist/tools/run.d.ts +50 -0
- package/router/dist/tools/run.js +1219 -0
- package/router/dist/tools/search.d.ts +44 -0
- package/router/dist/tools/search.js +1205 -0
- package/router/dist/transform/engine.d.ts +143 -0
- package/router/dist/transform/engine.js +1955 -0
- package/router/dist/vault/vault-index.d.ts +106 -0
- package/router/dist/vault/vault-index.js +411 -0
- package/router/dist/vault/vault.d.ts +103 -0
- package/router/dist/vault/vault.js +635 -0
- package/{plugins/freeflow/skills → skills}/commit-work/SKILL.md +17 -16
- package/{plugins/freeflow/skills → skills}/commit-work/references/staging-decisions.md +3 -3
- package/skills/design-for-depth/SKILL.md +90 -0
- package/skills/design-for-depth/references/design-pressure-signals.md +265 -0
- package/skills/design-for-depth/references/software-design-philosophy.md +219 -0
- package/{plugins/freeflow/skills → skills}/diagnose-failure/SKILL.md +6 -1
- package/skills/discover/ARTIFACT-DESTINATIONS.md +58 -0
- package/skills/discover/CHECKPOINTS.md +74 -0
- package/skills/discover/SKILL.md +173 -0
- package/skills/execute-plan/SKILL.md +133 -0
- package/skills/execute-plan/references/execution-map.md +122 -0
- package/{plugins/freeflow/skills → skills}/mode-contract/SKILL.md +10 -5
- package/skills/output-router/SKILL.md +202 -0
- package/skills/output-router/references/safety-policy.md +40 -0
- package/{plugins/freeflow/skills → skills}/review-artifact/SKILL.md +37 -1
- package/{plugins/freeflow/skills → skills}/review-artifact/references/reviewer-prompt.md +14 -2
- package/skills/review-work/SKILL.md +101 -0
- package/{plugins/freeflow/skills → skills}/review-work/references/reviewer-prompt.md +8 -0
- package/skills/setup-freeflow/SKILL.md +106 -0
- package/{plugins/freeflow/skills → skills}/setup-freeflow/references/activation-contract.md +14 -8
- package/{plugins/freeflow/skills → skills}/setup-freeflow/references/host-setup.md +5 -3
- package/skills/setup-freeflow/references/output-router-setup.md +102 -0
- package/{plugins/freeflow/skills → skills}/verify-work/SKILL.md +3 -1
- package/{plugins/freeflow/skills → skills}/workflow/SKILL.md +23 -13
- package/{plugins/freeflow/skills → skills}/workflow/references/workflow-map.md +21 -20
- package/{plugins/freeflow/skills → skills}/write-plan/SKILL.md +9 -1
- package/{plugins/freeflow/skills → skills}/write-skill/SKILL.md +1 -1
- package/{plugins/freeflow/skills → skills}/write-spec/SKILL.md +9 -6
- package/plugins/freeflow/docs/README.md +0 -39
- package/plugins/freeflow/docs/adr/0002-evals-before-hooks.md +0 -17
- package/plugins/freeflow/docs/adr/README.md +0 -7
- package/plugins/freeflow/docs/architecture.md +0 -86
- package/plugins/freeflow/docs/release-evidence.md +0 -49
- package/plugins/freeflow/pi-extension/index.js +0 -250
- package/plugins/freeflow/skills/capture-decisions/SKILL.md +0 -95
- package/plugins/freeflow/skills/capture-decisions/references/destination-guide.md +0 -65
- package/plugins/freeflow/skills/execute-plan/SKILL.md +0 -67
- package/plugins/freeflow/skills/grill-context/SKILL.md +0 -79
- package/plugins/freeflow/skills/grill-context/references/approach-framing.md +0 -52
- package/plugins/freeflow/skills/research-brief/SKILL.md +0 -69
- package/plugins/freeflow/skills/review-work/SKILL.md +0 -73
- package/plugins/freeflow/skills/setup-freeflow/SKILL.md +0 -84
- /package/{plugins/freeflow/assets → assets}/freeflow-plugin-icon-transparent.png +0 -0
- /package/{plugins/freeflow/hooks → hooks}/hooks.json +0 -0
- /package/{plugins/freeflow/docs → plugin-docs}/adr/0001-three-modes.md +0 -0
- /package/{plugins/freeflow/skills → skills}/bypass/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/diagnose-failure/references/feedback-loop-catalog.md +0 -0
- /package/{plugins/freeflow/skills → skills}/diagnose-failure/references/flaky-and-performance.md +0 -0
- /package/{plugins/freeflow/skills → skills}/evaluate-skill/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/evaluate-skill/references/eval-patterns.md +0 -0
- /package/{plugins/freeflow/skills → skills}/evaluate-skill/references/grading-priority.md +0 -0
- /package/{plugins/freeflow/skills → skills}/handoff/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/handoff/references/templates.md +0 -0
- /package/{plugins/freeflow/skills → skills}/interview-gate/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/write-plan/references/plan-shapes.md +0 -0
- /package/{plugins/freeflow/skills → skills}/write-spec/references/artifact-standards.md +0 -0
- /package/{plugins/freeflow/skills → skills}/write-spec/references/spec-shapes.md +0 -0
|
@@ -0,0 +1,486 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
set -euo pipefail
|
|
3
|
+
|
|
4
|
+
usage() {
|
|
5
|
+
cat >&2 <<'EOF'
|
|
6
|
+
Usage:
|
|
7
|
+
validate-release-metadata.sh [--mode local|prepublish|release]
|
|
8
|
+
[--release-version VERSION]
|
|
9
|
+
[--format text|json]
|
|
10
|
+
|
|
11
|
+
Default:
|
|
12
|
+
validate-release-metadata.sh --mode prepublish --release-version 0.3.0 --format text
|
|
13
|
+
EOF
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
fail_usage() {
|
|
17
|
+
usage
|
|
18
|
+
printf 'FAIL: %s\n' "$1" >&2
|
|
19
|
+
exit 2
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
23
|
+
repo_root="$(git -C "$script_dir" rev-parse --show-toplevel)"
|
|
24
|
+
plugin_root="$repo_root"
|
|
25
|
+
|
|
26
|
+
mode="prepublish"
|
|
27
|
+
release_version="0.3.0"
|
|
28
|
+
format="text"
|
|
29
|
+
|
|
30
|
+
while [ "$#" -gt 0 ]; do
|
|
31
|
+
case "$1" in
|
|
32
|
+
--mode)
|
|
33
|
+
mode="${2:-}"
|
|
34
|
+
shift 2
|
|
35
|
+
;;
|
|
36
|
+
--release-version)
|
|
37
|
+
release_version="${2:-}"
|
|
38
|
+
shift 2
|
|
39
|
+
;;
|
|
40
|
+
--format)
|
|
41
|
+
format="${2:-}"
|
|
42
|
+
shift 2
|
|
43
|
+
;;
|
|
44
|
+
--help|-h)
|
|
45
|
+
usage
|
|
46
|
+
exit 0
|
|
47
|
+
;;
|
|
48
|
+
*)
|
|
49
|
+
fail_usage "Unsupported argument: $1"
|
|
50
|
+
;;
|
|
51
|
+
esac
|
|
52
|
+
done
|
|
53
|
+
|
|
54
|
+
case "$mode" in
|
|
55
|
+
local|prepublish|release) ;;
|
|
56
|
+
*) fail_usage "Unsupported mode: $mode" ;;
|
|
57
|
+
esac
|
|
58
|
+
|
|
59
|
+
case "$format" in
|
|
60
|
+
text|json) ;;
|
|
61
|
+
*) fail_usage "Unsupported format: $format" ;;
|
|
62
|
+
esac
|
|
63
|
+
|
|
64
|
+
codex_marketplace="$repo_root/.agents/plugins/marketplace.json"
|
|
65
|
+
claude_marketplace="$repo_root/.claude-plugin/marketplace.json"
|
|
66
|
+
codex_manifest="$plugin_root/.codex-plugin/plugin.json"
|
|
67
|
+
claude_manifest="$plugin_root/.claude-plugin/plugin.json"
|
|
68
|
+
command_surface="$plugin_root/command-surface.json"
|
|
69
|
+
command_audit="$plugin_root/evals/scripts/audit-command-surface.sh"
|
|
70
|
+
runtime_hooks_json="$plugin_root/hooks/hooks.json"
|
|
71
|
+
runtime_hook_script="$plugin_root/hooks/freeflow-runtime-context.mjs"
|
|
72
|
+
runtime_hook_check="$plugin_root/evals/scripts/check-runtime-context-hook.sh"
|
|
73
|
+
command_audit_stdout="$(mktemp "${TMPDIR:-/tmp}/freeflow-command-surface-audit.out.XXXXXX")"
|
|
74
|
+
command_audit_stderr="$(mktemp "${TMPDIR:-/tmp}/freeflow-command-surface-audit.err.XXXXXX")"
|
|
75
|
+
runtime_hook_stdout="$(mktemp "${TMPDIR:-/tmp}/freeflow-runtime-hook.out.XXXXXX")"
|
|
76
|
+
runtime_hook_stderr="$(mktemp "${TMPDIR:-/tmp}/freeflow-runtime-hook.err.XXXXXX")"
|
|
77
|
+
architecture_doc="$plugin_root/plugin-docs/architecture.md"
|
|
78
|
+
release_evidence="$plugin_root/plugin-docs/release-evidence.md"
|
|
79
|
+
release_boundary_adr="$plugin_root/plugin-docs/adr/0003-release-boundary.md"
|
|
80
|
+
gitignore="$repo_root/.gitignore"
|
|
81
|
+
|
|
82
|
+
checks_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-checks.XXXXXX")"
|
|
83
|
+
findings_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-findings.XXXXXX")"
|
|
84
|
+
warnings_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-warnings.XXXXXX")"
|
|
85
|
+
deferred_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-deferred.XXXXXX")"
|
|
86
|
+
evidence_file="$(mktemp "${TMPDIR:-/tmp}/freeflow-release-evidence.XXXXXX")"
|
|
87
|
+
trap 'rm -f "$checks_file" "$findings_file" "$warnings_file" "$deferred_file" "$evidence_file" "$command_audit_stdout" "$command_audit_stderr" "$runtime_hook_stdout" "$runtime_hook_stderr"' EXIT
|
|
88
|
+
|
|
89
|
+
status="pass"
|
|
90
|
+
|
|
91
|
+
record_line() {
|
|
92
|
+
local file="$1"
|
|
93
|
+
local value="$2"
|
|
94
|
+
printf '%s\n' "$value" >> "$file"
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
record_check() {
|
|
98
|
+
local check="$1"
|
|
99
|
+
local check_status="$2"
|
|
100
|
+
local message="$3"
|
|
101
|
+
printf '%s\t%s\t%s\n' "$check" "$check_status" "$message" >> "$checks_file"
|
|
102
|
+
if [ "$check_status" = "fail" ]; then
|
|
103
|
+
status="fail"
|
|
104
|
+
record_line "$findings_file" "$check: $message"
|
|
105
|
+
elif [ "$check_status" = "warn" ]; then
|
|
106
|
+
record_line "$warnings_file" "$check: $message"
|
|
107
|
+
fi
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
record_deferred() {
|
|
111
|
+
local check="$1"
|
|
112
|
+
local message="$2"
|
|
113
|
+
printf '%s\t%s\n' "$check" "$message" >> "$deferred_file"
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
record_evidence() {
|
|
117
|
+
record_line "$evidence_file" "$1"
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
rel_path() {
|
|
121
|
+
local file="$1"
|
|
122
|
+
printf '%s\n' "${file#"$repo_root"/}"
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
require_file() {
|
|
126
|
+
local check="$1"
|
|
127
|
+
local file="$2"
|
|
128
|
+
if [ -f "$file" ]; then
|
|
129
|
+
record_evidence "$file"
|
|
130
|
+
return 0
|
|
131
|
+
fi
|
|
132
|
+
record_check "$check" "fail" "Missing required file: $(rel_path "$file")"
|
|
133
|
+
return 1
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
json_get() {
|
|
137
|
+
local file="$1"
|
|
138
|
+
local filter="$2"
|
|
139
|
+
jq -r "$filter | if . == null then empty else . end" "$file"
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
same_value() {
|
|
143
|
+
local check="$1"
|
|
144
|
+
local label="$2"
|
|
145
|
+
local expected="$3"
|
|
146
|
+
local actual="$4"
|
|
147
|
+
if [ "$actual" = "$expected" ]; then
|
|
148
|
+
return 0
|
|
149
|
+
fi
|
|
150
|
+
record_check "$check" "fail" "$label expected '$expected', got '$actual'"
|
|
151
|
+
return 1
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
contains_fixed() {
|
|
155
|
+
local file="$1"
|
|
156
|
+
local needle="$2"
|
|
157
|
+
rg -Fq "$needle" "$file"
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
check_json_shape() {
|
|
161
|
+
local check="json-shape"
|
|
162
|
+
local ok=1
|
|
163
|
+
for file in "$codex_marketplace" "$claude_marketplace" "$codex_manifest" "$claude_manifest" "$command_surface" "$runtime_hooks_json"; do
|
|
164
|
+
require_file "$check" "$file" || ok=0
|
|
165
|
+
if [ -f "$file" ]; then
|
|
166
|
+
if ! jq empty "$file" >/dev/null; then
|
|
167
|
+
record_check "$check" "fail" "Invalid JSON: $(rel_path "$file")"
|
|
168
|
+
ok=0
|
|
169
|
+
fi
|
|
170
|
+
fi
|
|
171
|
+
done
|
|
172
|
+
|
|
173
|
+
if [ "$ok" = "1" ]; then
|
|
174
|
+
record_check "$check" "pass" "Required marketplace, manifest, and command-surface JSON files parse."
|
|
175
|
+
fi
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
check_marketplace_locality() {
|
|
179
|
+
local check="marketplace-locality"
|
|
180
|
+
local ok=1
|
|
181
|
+
local codex_path
|
|
182
|
+
local claude_source
|
|
183
|
+
codex_path="$(json_get "$codex_marketplace" '.plugins[0].source.path')"
|
|
184
|
+
claude_source="$(json_get "$claude_marketplace" '.plugins[0].source')"
|
|
185
|
+
|
|
186
|
+
same_value "$check" "Codex marketplace source path" "." "$codex_path" || ok=0
|
|
187
|
+
same_value "$check" "Claude marketplace source" "." "$claude_source" || ok=0
|
|
188
|
+
same_value "$check" "Codex marketplace plugin name" "freeflow" "$(json_get "$codex_marketplace" '.plugins[0].name')" || ok=0
|
|
189
|
+
same_value "$check" "Claude marketplace plugin name" "freeflow" "$(json_get "$claude_marketplace" '.plugins[0].name')" || ok=0
|
|
190
|
+
|
|
191
|
+
if [ "$ok" = "1" ]; then
|
|
192
|
+
record_check "$check" "pass" "Root marketplace files point at ."
|
|
193
|
+
fi
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
check_manifest_consistency() {
|
|
197
|
+
local check="manifest-consistency"
|
|
198
|
+
local ok=1
|
|
199
|
+
local field
|
|
200
|
+
|
|
201
|
+
for field in name version license homepage repository; do
|
|
202
|
+
same_value "$check" "$field" "$(json_get "$codex_manifest" ".$field")" "$(json_get "$claude_manifest" ".$field")" || ok=0
|
|
203
|
+
done
|
|
204
|
+
|
|
205
|
+
same_value "$check" "release version" "$release_version" "$(json_get "$codex_manifest" '.version')" || ok=0
|
|
206
|
+
same_value "$check" "author.name" "$(json_get "$codex_manifest" '.author.name')" "$(json_get "$claude_manifest" '.author.name')" || ok=0
|
|
207
|
+
same_value "$check" "Codex displayName" "Freeflow" "$(json_get "$codex_manifest" '.interface.displayName')" || ok=0
|
|
208
|
+
same_value "$check" "Codex skills path" "./skills/" "$(json_get "$codex_manifest" '.skills')" || ok=0
|
|
209
|
+
same_value "$check" "Claude marketplace version" "$release_version" "$(json_get "$claude_marketplace" '.plugins[0].version')" || ok=0
|
|
210
|
+
same_value "$check" "Claude marketplace homepage" "$(json_get "$claude_manifest" '.homepage')" "$(json_get "$claude_marketplace" '.plugins[0].homepage')" || ok=0
|
|
211
|
+
same_value "$check" "Claude marketplace author" "$(json_get "$claude_manifest" '.author.name')" "$(json_get "$claude_marketplace" '.plugins[0].author.name')" || ok=0
|
|
212
|
+
|
|
213
|
+
if [ "$ok" = "1" ]; then
|
|
214
|
+
record_check "$check" "pass" "Codex and Claude manifests agree on release identity."
|
|
215
|
+
fi
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
check_command_surface() {
|
|
219
|
+
local check="command-surface"
|
|
220
|
+
require_file "$check" "$command_audit" || return
|
|
221
|
+
if "$command_audit" >"$command_audit_stdout" 2>"$command_audit_stderr"; then
|
|
222
|
+
record_check "$check" "pass" "Existing command-surface audit passed."
|
|
223
|
+
else
|
|
224
|
+
record_check "$check" "fail" "Existing command-surface audit failed."
|
|
225
|
+
fi
|
|
226
|
+
}
|
|
227
|
+
|
|
228
|
+
check_runtime_context_hooks() {
|
|
229
|
+
local check="runtime-context-hooks"
|
|
230
|
+
local ok=1
|
|
231
|
+
|
|
232
|
+
require_file "$check" "$runtime_hooks_json" || ok=0
|
|
233
|
+
require_file "$check" "$runtime_hook_script" || ok=0
|
|
234
|
+
require_file "$check" "$runtime_hook_check" || ok=0
|
|
235
|
+
|
|
236
|
+
if [ "$ok" = "1" ]; then
|
|
237
|
+
if "$runtime_hook_check" >"$runtime_hook_stdout" 2>"$runtime_hook_stderr"; then
|
|
238
|
+
record_check "$check" "pass" "Plugin-bundled runtime context hook passed deterministic session-start checks."
|
|
239
|
+
else
|
|
240
|
+
record_check "$check" "fail" "Plugin-bundled runtime context hook check failed."
|
|
241
|
+
fi
|
|
242
|
+
fi
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
check_release_boundary() {
|
|
246
|
+
local check="release-boundary"
|
|
247
|
+
local ok=1
|
|
248
|
+
|
|
249
|
+
for file in "$architecture_doc" "$release_evidence" "$release_boundary_adr"; do
|
|
250
|
+
require_file "$check" "$file" || ok=0
|
|
251
|
+
done
|
|
252
|
+
|
|
253
|
+
contains_fixed "$release_boundary_adr" "one plugin runtime under the repo root" || {
|
|
254
|
+
record_check "$check" "fail" "ADR 0003 no longer states the single runtime boundary."
|
|
255
|
+
ok=0
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
contains_fixed "$release_boundary_adr" "runtime excludes generated eval run output" || {
|
|
259
|
+
record_check "$check" "fail" "ADR 0003 no longer excludes generated eval run output."
|
|
260
|
+
ok=0
|
|
261
|
+
}
|
|
262
|
+
|
|
263
|
+
contains_fixed "$architecture_doc" "does not ship a CLI, Codex/Claude native slash handlers, enforcement hooks, or a new agent runtime in this release" || {
|
|
264
|
+
record_check "$check" "fail" "Architecture doc no longer preserves the deferred enforcement boundary."
|
|
265
|
+
ok=0
|
|
266
|
+
}
|
|
267
|
+
|
|
268
|
+
contains_fixed "$architecture_doc" "context-loading hooks" || {
|
|
269
|
+
record_check "$check" "fail" "Architecture doc no longer documents runtime context hooks."
|
|
270
|
+
ok=0
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
contains_fixed "$release_evidence" "plugin-bundled context hooks" || {
|
|
274
|
+
record_check "$check" "fail" "Release evidence no longer documents runtime context hooks."
|
|
275
|
+
ok=0
|
|
276
|
+
}
|
|
277
|
+
|
|
278
|
+
contains_fixed "$release_evidence" "Codex/Claude native slash handlers are not shipped in this release" || {
|
|
279
|
+
record_check "$check" "fail" "Release evidence no longer states Codex/Claude native slash handlers are not shipped."
|
|
280
|
+
ok=0
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
contains_fixed "$release_evidence" "Live Claude smoke evals" || {
|
|
284
|
+
record_check "$check" "fail" "Release evidence no longer records deferred Claude smoke evals."
|
|
285
|
+
ok=0
|
|
286
|
+
}
|
|
287
|
+
|
|
288
|
+
if [ "$ok" = "1" ]; then
|
|
289
|
+
record_check "$check" "pass" "Release docs preserve runtime context hooks, deferred enforcement boundary, and deferred-smoke status."
|
|
290
|
+
fi
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
check_package_cleanliness() {
|
|
294
|
+
local check="package-cleanliness"
|
|
295
|
+
local ok=1
|
|
296
|
+
|
|
297
|
+
contains_fixed "$gitignore" "evals/runs/" || {
|
|
298
|
+
record_check "$check" "fail" ".gitignore does not ignore generated eval runs."
|
|
299
|
+
ok=0
|
|
300
|
+
}
|
|
301
|
+
|
|
302
|
+
if git -C "$repo_root" ls-files evals/runs | rg -q .; then
|
|
303
|
+
record_check "$check" "fail" "Generated eval run output is tracked."
|
|
304
|
+
ok=0
|
|
305
|
+
fi
|
|
306
|
+
|
|
307
|
+
duplicate_codex_manifests="$(
|
|
308
|
+
find "$repo_root" -path "$repo_root/.git" -prune -o -path '*/.codex-plugin/plugin.json' -print |
|
|
309
|
+
grep -Fvx "$codex_manifest" || true
|
|
310
|
+
)"
|
|
311
|
+
if [ -n "$duplicate_codex_manifests" ]; then
|
|
312
|
+
record_check "$check" "fail" "Found another Codex plugin manifest outside the repo root."
|
|
313
|
+
ok=0
|
|
314
|
+
fi
|
|
315
|
+
|
|
316
|
+
duplicate_claude_manifests="$(
|
|
317
|
+
find "$repo_root" -path "$repo_root/.git" -prune -o -path '*/.claude-plugin/plugin.json' -print |
|
|
318
|
+
grep -Fvx "$claude_manifest" || true
|
|
319
|
+
)"
|
|
320
|
+
if [ -n "$duplicate_claude_manifests" ]; then
|
|
321
|
+
record_check "$check" "fail" "Found another Claude plugin manifest outside the repo root."
|
|
322
|
+
ok=0
|
|
323
|
+
fi
|
|
324
|
+
|
|
325
|
+
if rg -n '(^|[^[:alnum:]_-])orchestra:' "$plugin_root/command-surface.json" "$plugin_root/skills" "$plugin_root/.codex-plugin" "$plugin_root/.claude-plugin" >/dev/null; then
|
|
326
|
+
record_check "$check" "fail" "Runtime contains old Orchestra command compatibility syntax."
|
|
327
|
+
ok=0
|
|
328
|
+
fi
|
|
329
|
+
|
|
330
|
+
if rg -n '(^|[^[:alnum:]_-])/orchestra' "$plugin_root/command-surface.json" "$plugin_root/skills" "$plugin_root/.codex-plugin" "$plugin_root/.claude-plugin" >/dev/null; then
|
|
331
|
+
record_check "$check" "fail" "Runtime contains old /orchestra command compatibility syntax."
|
|
332
|
+
ok=0
|
|
333
|
+
fi
|
|
334
|
+
|
|
335
|
+
if [ "$ok" = "1" ]; then
|
|
336
|
+
record_check "$check" "pass" "Generated runs are ignored, no duplicate manifests were found, and old command compatibility is absent."
|
|
337
|
+
fi
|
|
338
|
+
}
|
|
339
|
+
|
|
340
|
+
check_docs_drift() {
|
|
341
|
+
local check="docs-drift"
|
|
342
|
+
local ok=1
|
|
343
|
+
|
|
344
|
+
if [ "$(json_get "$command_surface" '.nativeSlashHandlers')" != "false" ]; then
|
|
345
|
+
record_check "$check" "fail" "command-surface.json no longer has nativeSlashHandlers=false."
|
|
346
|
+
ok=0
|
|
347
|
+
fi
|
|
348
|
+
|
|
349
|
+
if jq -e 'has("commands") or has("slashCommands")' "$codex_manifest" >/dev/null; then
|
|
350
|
+
record_check "$check" "fail" "Codex manifest declares command handlers while nativeSlashHandlers=false."
|
|
351
|
+
ok=0
|
|
352
|
+
fi
|
|
353
|
+
|
|
354
|
+
if jq -e 'has("commands") or has("slashCommands")' "$claude_manifest" >/dev/null; then
|
|
355
|
+
record_check "$check" "fail" "Claude manifest declares command handlers while nativeSlashHandlers=false."
|
|
356
|
+
ok=0
|
|
357
|
+
fi
|
|
358
|
+
|
|
359
|
+
contains_fixed "$release_evidence" "4 mode commands" || {
|
|
360
|
+
record_check "$check" "fail" "Release evidence no longer lists 4 mode commands."
|
|
361
|
+
ok=0
|
|
362
|
+
}
|
|
363
|
+
|
|
364
|
+
contains_fixed "$release_evidence" "12 direct skill calls" || {
|
|
365
|
+
record_check "$check" "fail" "Release evidence no longer lists 12 direct skill calls."
|
|
366
|
+
ok=0
|
|
367
|
+
}
|
|
368
|
+
|
|
369
|
+
contains_fixed "$release_evidence" "3 developer skill calls" || {
|
|
370
|
+
record_check "$check" "fail" "Release evidence no longer lists 3 developer skill calls."
|
|
371
|
+
ok=0
|
|
372
|
+
}
|
|
373
|
+
|
|
374
|
+
if [ "$ok" = "1" ]; then
|
|
375
|
+
record_check "$check" "pass" "Docs and manifests agree with the current command-surface boundary."
|
|
376
|
+
fi
|
|
377
|
+
}
|
|
378
|
+
|
|
379
|
+
check_install_smoke() {
|
|
380
|
+
record_deferred "install-smoke" "GitHub/Codex, Claude, and fresh Pi install smoke checks are not run by local metadata validation."
|
|
381
|
+
if [ "$mode" = "release" ] && [ "$status" = "pass" ]; then
|
|
382
|
+
status="blocked"
|
|
383
|
+
record_line "$warnings_file" "install-smoke: release mode is blocked until GitHub/Codex, Claude, and fresh Pi install smoke checks run or are explicitly waived."
|
|
384
|
+
fi
|
|
385
|
+
}
|
|
386
|
+
|
|
387
|
+
emit_json_array_from_lines() {
|
|
388
|
+
jq -R -s 'split("\n") | map(select(length > 0))' "$1"
|
|
389
|
+
}
|
|
390
|
+
|
|
391
|
+
emit_checks_json() {
|
|
392
|
+
jq -Rn '
|
|
393
|
+
[inputs
|
|
394
|
+
| split("\t")
|
|
395
|
+
| {name:.[0], status:.[1], message:.[2]}]
|
|
396
|
+
' "$checks_file"
|
|
397
|
+
}
|
|
398
|
+
|
|
399
|
+
emit_deferred_json() {
|
|
400
|
+
jq -Rn '
|
|
401
|
+
[inputs
|
|
402
|
+
| split("\t")
|
|
403
|
+
| {name:.[0], message:.[1]}]
|
|
404
|
+
' "$deferred_file"
|
|
405
|
+
}
|
|
406
|
+
|
|
407
|
+
emit_report_json() {
|
|
408
|
+
local checks_json findings_json warnings_json deferred_json evidence_json
|
|
409
|
+
checks_json="$(emit_checks_json)"
|
|
410
|
+
findings_json="$(emit_json_array_from_lines "$findings_file")"
|
|
411
|
+
warnings_json="$(emit_json_array_from_lines "$warnings_file")"
|
|
412
|
+
deferred_json="$(emit_deferred_json)"
|
|
413
|
+
evidence_json="$(emit_json_array_from_lines "$evidence_file")"
|
|
414
|
+
|
|
415
|
+
jq -n \
|
|
416
|
+
--arg status "$status" \
|
|
417
|
+
--arg version "$release_version" \
|
|
418
|
+
--arg mode "$mode" \
|
|
419
|
+
--arg summary "Release metadata validation ${status} for Freeflow ${release_version} in ${mode} mode." \
|
|
420
|
+
--argjson checks "$checks_json" \
|
|
421
|
+
--argjson findings "$findings_json" \
|
|
422
|
+
--argjson warnings "$warnings_json" \
|
|
423
|
+
--argjson deferred "$deferred_json" \
|
|
424
|
+
--argjson evidence "$evidence_json" \
|
|
425
|
+
'{
|
|
426
|
+
status: $status,
|
|
427
|
+
version: $version,
|
|
428
|
+
mode: $mode,
|
|
429
|
+
summary: $summary,
|
|
430
|
+
checks: $checks,
|
|
431
|
+
findings: $findings,
|
|
432
|
+
warnings: $warnings,
|
|
433
|
+
deferred: $deferred,
|
|
434
|
+
evidence: $evidence
|
|
435
|
+
}'
|
|
436
|
+
}
|
|
437
|
+
|
|
438
|
+
emit_text_report() {
|
|
439
|
+
printf 'Release metadata validation: %s\n' "$status"
|
|
440
|
+
printf 'Version: %s\n' "$release_version"
|
|
441
|
+
printf 'Mode: %s\n' "$mode"
|
|
442
|
+
printf '\nChecks:\n'
|
|
443
|
+
while IFS=$'\t' read -r name check_status message; do
|
|
444
|
+
[ -n "$name" ] || continue
|
|
445
|
+
printf '%s\n' "- $check_status: $name - $message"
|
|
446
|
+
done < "$checks_file"
|
|
447
|
+
|
|
448
|
+
if [ -s "$findings_file" ]; then
|
|
449
|
+
printf '\nFindings:\n'
|
|
450
|
+
sed 's/^/- /' "$findings_file"
|
|
451
|
+
fi
|
|
452
|
+
|
|
453
|
+
if [ -s "$warnings_file" ]; then
|
|
454
|
+
printf '\nWarnings:\n'
|
|
455
|
+
sed 's/^/- /' "$warnings_file"
|
|
456
|
+
fi
|
|
457
|
+
|
|
458
|
+
if [ -s "$deferred_file" ]; then
|
|
459
|
+
printf '\nDeferred:\n'
|
|
460
|
+
while IFS=$'\t' read -r name message; do
|
|
461
|
+
[ -n "$name" ] || continue
|
|
462
|
+
printf '%s\n' "- $name: $message"
|
|
463
|
+
done < "$deferred_file"
|
|
464
|
+
fi
|
|
465
|
+
}
|
|
466
|
+
|
|
467
|
+
check_json_shape
|
|
468
|
+
check_marketplace_locality
|
|
469
|
+
check_manifest_consistency
|
|
470
|
+
check_command_surface
|
|
471
|
+
check_runtime_context_hooks
|
|
472
|
+
check_release_boundary
|
|
473
|
+
check_package_cleanliness
|
|
474
|
+
check_docs_drift
|
|
475
|
+
check_install_smoke
|
|
476
|
+
|
|
477
|
+
case "$format" in
|
|
478
|
+
json) emit_report_json ;;
|
|
479
|
+
text) emit_text_report ;;
|
|
480
|
+
esac
|
|
481
|
+
|
|
482
|
+
case "$status" in
|
|
483
|
+
pass) exit 0 ;;
|
|
484
|
+
blocked) exit 3 ;;
|
|
485
|
+
fail) exit 1 ;;
|
|
486
|
+
esac
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# Freeflow v0.1 Acceptance Suite
|
|
2
|
+
|
|
3
|
+
> **Doc ID:** EVAL-001-freeflow-v0-1-acceptance
|
|
4
|
+
> **Date:** 2026-05-26
|
|
5
|
+
> **Owner:** Hassan Mohiddin
|
|
6
|
+
> **Type:** Acceptance Suite
|
|
7
|
+
> **Status:** Current
|
|
8
|
+
> **Source:** `docs/plugin-contract.md`, `docs/plans/skill-inventory-and-plugin-plan.md`, `docs/freeflow-runtime-and-lifecycle.md`, `evals/reports/by-command-surface/command-surface-matrix.md`, and current fixture eval reports
|
|
9
|
+
|
|
10
|
+
## Purpose
|
|
11
|
+
|
|
12
|
+
Define the small local-only gate for calling the current Freeflow plugin draft v0.1-ready for Hassan's Codex-first dogfooding.
|
|
13
|
+
|
|
14
|
+
This suite is not a public release gate and not a hook/CLI requirement. It is a curated set of existing evals that covers the core product promise without rerunning every historical regression.
|
|
15
|
+
|
|
16
|
+
## Pass Standard
|
|
17
|
+
|
|
18
|
+
The suite passes only when:
|
|
19
|
+
|
|
20
|
+
- every required fixture eval scores all assertions as passing from final response plus diff evidence
|
|
21
|
+
- no required eval edits source truth to erase a contradiction
|
|
22
|
+
- no required eval creates unnecessary workflow artifacts for small reversible work
|
|
23
|
+
- no required eval commits, stages, or mutates git state outside the fixture expectation
|
|
24
|
+
- `audit-command-surface.sh` passes
|
|
25
|
+
- no run leaves saved output files in the repo root
|
|
26
|
+
|
|
27
|
+
For fixture evals, diff evidence beats final-response claims.
|
|
28
|
+
|
|
29
|
+
## Required Local Gate
|
|
30
|
+
|
|
31
|
+
Run these with the Codex fixture harness unless a row says otherwise.
|
|
32
|
+
|
|
33
|
+
| Area | Eval | Why It Is In v0.1 |
|
|
34
|
+
| --- | --- | --- |
|
|
35
|
+
| Setup | `STP-001` | Minimal Codex/AGENTS setup creates only config plus the compact core block. |
|
|
36
|
+
| Setup ambiguity | `STP-002` | Existing `AGENTS.md` and `CLAUDE.md` requires a host decision instead of silent dual install. |
|
|
37
|
+
| Mode handling | `MODE-001` | Conversation mode answers directly without edits or workflow artifacts. |
|
|
38
|
+
| Mode persistence | `MODE-005` | Strict mode can be scoped without silently persisting config/state. |
|
|
39
|
+
| Always-on memory | `AON-001` | Installed Freeflow core block catches source-truth conflict without loading a skill file. |
|
|
40
|
+
| Source-truth conflict | `IVG-001` | User-owned policy conflict stops before edits and asks which path to follow. |
|
|
41
|
+
| Discovery checkpoint before build | `DIS-001` | Vague product work produces an evidence-backed checkpoint and surfaces the next user-owned decision before implementation. |
|
|
42
|
+
| Strict spec gate | `WSP-006` | Strict billing/public API specs ask for owner and core API decisions instead of writing placeholders. |
|
|
43
|
+
| Strict delegated plan | `WPL-005` | Future-agent-facing strict plans include durable identity, source authority, verification, and stop conditions. |
|
|
44
|
+
| Execution back edge | `XPL-004` | Verification failure stops execution instead of patching source truth to satisfy the check. |
|
|
45
|
+
| Work review | `REV-002` | Review catches unrelated billing regression before handoff. |
|
|
46
|
+
| Verification | `CMD-005` | Manual verify-work call checks stale completion claims against fresh evidence. |
|
|
47
|
+
| Commit closeout | `CMT-005` | Mixed staged sensitive changes stop before commit. |
|
|
48
|
+
| Discovery resists capture pressure | `DIS-002` | Discovery resists long question chains and durable-memory creation when destination is ambiguous. |
|
|
49
|
+
| Bypass boundary | `BYP-002` | Bypass cannot override billing/source-truth judgment. |
|
|
50
|
+
| Command surface | `audit-command-surface.sh` | Registry, docs mentions, skill targets, manifest consistency, and `nativeSlashHandlers=false` stay aligned. |
|
|
51
|
+
|
|
52
|
+
## Commands
|
|
53
|
+
|
|
54
|
+
Use a unique run folder per eval. Always pass a full output path under `evals/runs/`:
|
|
55
|
+
|
|
56
|
+
```sh
|
|
57
|
+
evals/scripts/run-fixture-eval.sh \
|
|
58
|
+
STP-001 --with-skill \
|
|
59
|
+
--run-dir evals/runs/v0.1-acceptance/stp-001 \
|
|
60
|
+
--output evals/runs/v0.1-acceptance/stp-001-with-skill-output.md \
|
|
61
|
+
skills/setup-freeflow/SKILL.md
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Before spending model tokens, dry-run each command:
|
|
65
|
+
|
|
66
|
+
```sh
|
|
67
|
+
evals/scripts/run-fixture-eval.sh STP-001 --dry-run
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
For always-on installed-memory coverage:
|
|
71
|
+
|
|
72
|
+
```sh
|
|
73
|
+
evals/scripts/run-fixture-eval.sh \
|
|
74
|
+
AON-001 --with-skill \
|
|
75
|
+
--run-dir evals/runs/v0.1-acceptance/aon-001 \
|
|
76
|
+
--output evals/runs/v0.1-acceptance/aon-001-with-skill-output.md
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
For command-surface consistency:
|
|
80
|
+
|
|
81
|
+
```sh
|
|
82
|
+
evals/scripts/audit-command-surface.sh
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Skill File Mapping
|
|
86
|
+
|
|
87
|
+
Pass the focused skill for skill-specific evals:
|
|
88
|
+
|
|
89
|
+
| Eval | Skill file |
|
|
90
|
+
| --- | --- |
|
|
91
|
+
| `STP-001`, `STP-002` | `skills/setup-freeflow/SKILL.md` |
|
|
92
|
+
| `MODE-001`, `MODE-005` | `skills/mode-contract/SKILL.md` |
|
|
93
|
+
| `IVG-001` | `skills/interview-gate/SKILL.md` |
|
|
94
|
+
| `DIS-001`, `DIS-002` | `skills/discover/SKILL.md` |
|
|
95
|
+
| `WSP-006` | `skills/write-spec/SKILL.md` |
|
|
96
|
+
| `WPL-005` | `skills/write-plan/SKILL.md` |
|
|
97
|
+
| `XPL-004` | `skills/execute-plan/SKILL.md` |
|
|
98
|
+
| `REV-002` | `skills/review-work/SKILL.md` |
|
|
99
|
+
| `CMD-005` | `skills/verify-work/SKILL.md` |
|
|
100
|
+
| `CMT-005` | `skills/commit-work/SKILL.md` |
|
|
101
|
+
| `BYP-002` | `skills/bypass/SKILL.md` |
|
|
102
|
+
|
|
103
|
+
If an eval depends on multiple active skills in normal use, prefer the smallest skill set that matches the eval's purpose and record the choice in the acceptance report.
|
|
104
|
+
|
|
105
|
+
## Reporting
|
|
106
|
+
|
|
107
|
+
Create `evals/reports/acceptance/v0.1-acceptance-report.md` after running the suite.
|
|
108
|
+
|
|
109
|
+
The report should include:
|
|
110
|
+
|
|
111
|
+
- date, model, runner, and harness command shape
|
|
112
|
+
- pass/fail table for each required eval
|
|
113
|
+
- links to saved outputs and diffs
|
|
114
|
+
- scoring table or explicit assertion summary
|
|
115
|
+
- root-output check result
|
|
116
|
+
- `audit-command-surface.sh` result
|
|
117
|
+
- any known non-blocking smoke-test gaps
|
|
118
|
+
|
|
119
|
+
Do not mark v0.1 accepted from old reports alone. Old reports can explain why an eval is in the suite, but the suite result needs fresh saved runs.
|
|
120
|
+
|
|
121
|
+
## Deferred Coverage
|
|
122
|
+
|
|
123
|
+
Not required for local v0.1:
|
|
124
|
+
|
|
125
|
+
- live Claude paired smoke evals
|
|
126
|
+
- native slash handlers
|
|
127
|
+
- hooks or CLI enforcement
|
|
128
|
+
- every historical `CMD-*` command eval
|
|
129
|
+
- optional `handoff/references/templates.md`
|
|
130
|
+
|
|
131
|
+
Add these only when local dogfooding or targeted evals show a concrete gap.
|