@hassangameryt/freeflow 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plugins/marketplace.json +1 -1
- package/.claude-plugin/marketplace.json +2 -2
- package/{plugins/freeflow/.claude-plugin → .claude-plugin}/plugin.json +1 -1
- package/{plugins/freeflow/.codex-plugin → .codex-plugin}/plugin.json +1 -1
- package/README.md +25 -16
- package/assets/freeflow-plugin-icon.png +0 -0
- package/{plugins/freeflow/command-surface.json → command-surface.json} +10 -10
- package/evals/README.md +32 -0
- package/evals/fixtures/copy-conventions.md +31 -0
- package/evals/fixtures/output-router/README.md +12 -0
- package/evals/fixtures/output-router/failed-command-output.txt +9 -0
- package/evals/fixtures/output-router/large-router-manual.md +63 -0
- package/evals/fixtures/output-router/native-large-output.txt +40 -0
- package/evals/fixtures/output-router/noisy-test-output.txt +30 -0
- package/evals/fixtures/output-router/verification-output.txt +8 -0
- package/evals/fixtures/tiny-artifact-review-app/README.md +5 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md +21 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/failed-payment-grace-period.md +24 -0
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/immediate-downgrade.md +17 -0
- package/evals/fixtures/tiny-artifact-review-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-ambiguous-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh +14 -0
- package/evals/fixtures/tiny-commit-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-app/docs/specs/settings-title.md +5 -0
- package/evals/fixtures/tiny-commit-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-everything-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-everything-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-everything-app/docs/specs/settings-title.md +5 -0
- package/evals/fixtures/tiny-commit-everything-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-everything-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-everything-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-staged-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-staged-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-staged-app/docs/specs/settings-title.md +5 -0
- package/evals/fixtures/tiny-commit-staged-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-commit-staged-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-commit-staged-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh +15 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/README.md +3 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/billing-policy.md +3 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/release-notes.md +3 -0
- package/evals/fixtures/tiny-commit-staged-mixed-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-dashboard-stale-app/README.md +11 -0
- package/evals/fixtures/tiny-dashboard-stale-app/src/api-client.ts +13 -0
- package/evals/fixtures/tiny-dashboard-stale-app/src/dashboard-data.ts +19 -0
- package/evals/fixtures/tiny-dashboard-stale-app/tests/dashboard-data.test.md +9 -0
- package/evals/fixtures/tiny-decision-destination-app/README.md +6 -0
- package/evals/fixtures/tiny-decision-destination-app/src/risk.ts +8 -0
- package/evals/fixtures/tiny-decision-destination-app/tests/risk.test.md +4 -0
- package/evals/fixtures/tiny-design-depth-app/README.md +15 -0
- package/evals/fixtures/tiny-design-depth-app/src/routes.ts +25 -0
- package/evals/fixtures/tiny-design-depth-app/src/sendEmail.ts +3 -0
- package/evals/fixtures/tiny-design-depth-app/src/sendSms.ts +3 -0
- package/evals/fixtures/tiny-design-depth-app/src/telemetry.ts +3 -0
- package/evals/fixtures/tiny-design-depth-app/tests/routes.test.md +12 -0
- package/evals/fixtures/tiny-eval-method-app/README.md +8 -0
- package/evals/fixtures/tiny-eval-method-app/evals/reports/review-pr-eval-result.md +5 -0
- package/evals/fixtures/tiny-eval-method-app/evals/reports/triage-question-failure.md +17 -0
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.diff +11 -0
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.md +3 -0
- package/evals/fixtures/tiny-eval-method-app/skills/review-pr/SKILL.md +10 -0
- package/evals/fixtures/tiny-eval-method-app/skills/triage-question/SKILL.md +10 -0
- package/evals/fixtures/tiny-execute-plan-app/README.md +5 -0
- package/evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh +5 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/immediate-downgrade-plan.md +11 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-plan.md +12 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md +15 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/unverified-billing-plan.md +9 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/failed-payment-grace-period.md +9 -0
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/settings-title.md +9 -0
- package/evals/fixtures/tiny-execute-plan-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-execute-plan-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-execute-plan-app/tests/billing.test.md +7 -0
- package/evals/fixtures/tiny-execute-plan-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-execute-review-loop-app/README.md +5 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/billing-policy.md +9 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/plans/settings-title-plan.md +12 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/previous-agent-summary.md +15 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/review-pass-1-non-pass.md +11 -0
- package/evals/fixtures/tiny-execute-review-loop-app/docs/specs/settings-title.md +9 -0
- package/evals/fixtures/tiny-execute-review-loop-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-execute-review-loop-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-execute-review-loop-app/tests/billing.test.md +7 -0
- package/evals/fixtures/tiny-execute-review-loop-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-flaky-session-app/README.md +10 -0
- package/evals/fixtures/tiny-flaky-session-app/src/session-timeout.ts +13 -0
- package/evals/fixtures/tiny-flaky-session-app/tests/session-timeout.test.md +8 -0
- package/evals/fixtures/tiny-handoff-app/README.md +5 -0
- package/evals/fixtures/tiny-handoff-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-handoff-app/docs/handoffs/old-agent-handoff.md +8 -0
- package/evals/fixtures/tiny-handoff-app/docs/plans/billing-retry-plan.md +11 -0
- package/evals/fixtures/tiny-handoff-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-handoff-report-app/README.md +3 -0
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.diff +1 -0
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.md +1 -0
- package/evals/fixtures/tiny-handoff-report-app/evals/scores/hof-003-score.txt +4 -0
- package/evals/fixtures/tiny-plan-app/README.md +5 -0
- package/evals/fixtures/tiny-plan-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-plan-app/docs/specs/failed-payment-grace-period.md +25 -0
- package/evals/fixtures/tiny-plan-app/src/billing.ts +17 -0
- package/evals/fixtures/tiny-plan-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-plan-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-plan-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-plan-strict-api-app/README.md +3 -0
- package/evals/fixtures/tiny-plan-strict-api-app/docs/billing-policy.md +5 -0
- package/evals/fixtures/tiny-plan-strict-api-app/docs/specs/billing-webhook-api.md +25 -0
- package/evals/fixtures/tiny-plan-strict-api-app/src/billing.ts +5 -0
- package/evals/fixtures/tiny-plan-strict-api-app/src/webhooks.ts +3 -0
- package/evals/fixtures/tiny-plan-strict-api-app/tests/billing-webhook.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/.freeflow/config.json +3 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/AGENTS.md +15 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/README.md +16 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/AGENTS.md +15 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/README.md +16 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/.freeflow/config.json +3 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/AGENTS.md +15 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/README.md +16 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-report-performance-app/README.md +11 -0
- package/evals/fixtures/tiny-report-performance-app/docs/performance.md +10 -0
- package/evals/fixtures/tiny-report-performance-app/src/report-summary.ts +29 -0
- package/evals/fixtures/tiny-report-performance-app/tests/report-performance.test.md +8 -0
- package/evals/fixtures/tiny-review-request-app/README.md +3 -0
- package/evals/fixtures/tiny-review-request-app/docs/billing-policy.md +5 -0
- package/evals/fixtures/tiny-review-request-app/docs/plans/billing-webhook-api.md +19 -0
- package/evals/fixtures/tiny-review-request-app/docs/review/previous-agent-summary.md +20 -0
- package/evals/fixtures/tiny-review-request-app/docs/specs/billing-webhook-api.md +25 -0
- package/evals/fixtures/tiny-review-request-app/src/billing.ts +11 -0
- package/evals/fixtures/tiny-review-request-app/src/webhooks.ts +19 -0
- package/evals/fixtures/tiny-review-request-app/tests/billing-webhook.test.md +8 -0
- package/evals/fixtures/tiny-review-work-app/README.md +5 -0
- package/evals/fixtures/tiny-review-work-app/docs/billing-policy.md +9 -0
- package/evals/fixtures/tiny-review-work-app/docs/plans/settings-title-plan.md +12 -0
- package/evals/fixtures/tiny-review-work-app/docs/review/previous-agent-summary.md +15 -0
- package/evals/fixtures/tiny-review-work-app/docs/specs/settings-title.md +9 -0
- package/evals/fixtures/tiny-review-work-app/src/billing.ts +7 -0
- package/evals/fixtures/tiny-review-work-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-review-work-app/tests/billing.test.md +7 -0
- package/evals/fixtures/tiny-review-work-app/tests/settings.test.md +5 -0
- package/evals/fixtures/tiny-saas-app/README.md +16 -0
- package/evals/fixtures/tiny-saas-app/docs/auth-review.md +15 -0
- package/evals/fixtures/tiny-saas-app/docs/billing-policy.md +10 -0
- package/evals/fixtures/tiny-saas-app/docs/copy-conventions.md +15 -0
- package/evals/fixtures/tiny-saas-app/docs/specs/settings-title.md +13 -0
- package/evals/fixtures/tiny-saas-app/src/auth-middleware.ts +12 -0
- package/evals/fixtures/tiny-saas-app/src/billing.ts +20 -0
- package/evals/fixtures/tiny-saas-app/src/login-copy.ts +1 -0
- package/evals/fixtures/tiny-saas-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-saas-app/tests/auth-middleware.test.md +8 -0
- package/evals/fixtures/tiny-saas-app/tests/billing.test.md +8 -0
- package/evals/fixtures/tiny-saas-app/tests/copy.test.md +7 -0
- package/evals/fixtures/tiny-setup-agents-app/AGENTS.md +6 -0
- package/evals/fixtures/tiny-setup-both-app/AGENTS.md +5 -0
- package/evals/fixtures/tiny-setup-both-app/CLAUDE.md +5 -0
- package/evals/fixtures/tiny-setup-claude-app/CLAUDE.md +6 -0
- package/evals/fixtures/tiny-setup-codex-rules-app/.codex/rules/shell.rules +5 -0
- package/evals/fixtures/tiny-setup-codex-rules-app/AGENTS.md +6 -0
- package/evals/fixtures/tiny-setup-conflict-app/AGENTS.md +6 -0
- package/evals/fixtures/tiny-setup-existing-block-app/AGENTS.md +14 -0
- package/evals/fixtures/tiny-setup-four-rules-app/CLAUDE.md +6 -0
- package/evals/fixtures/tiny-skill-dev-app/README.md +10 -0
- package/evals/fixtures/tiny-skill-dev-app/evals/prompts/review-pr-001.txt +8 -0
- package/evals/fixtures/tiny-skill-dev-app/evals/reports/review-pr-failure.md +17 -0
- package/evals/fixtures/tiny-skill-dev-app/skills/review-pr/SKILL.md +13 -0
- package/evals/fixtures/tiny-team-app/README.md +5 -0
- package/evals/fixtures/tiny-team-app/docs/product.md +6 -0
- package/evals/fixtures/tiny-team-app/src/teams.ts +9 -0
- package/evals/fixtures/tiny-team-app/src/users.ts +5 -0
- package/evals/fixtures/tiny-team-app/tests/teams.test.md +7 -0
- package/evals/fixtures/tiny-verify-app/README.md +5 -0
- package/evals/fixtures/tiny-verify-app/src/settings-nav.ts +1 -0
- package/evals/fixtures/tiny-verify-app/src/settings-page.ts +1 -0
- package/evals/fixtures/tiny-verify-app/tests/settings.test.md +7 -0
- package/evals/prompts/aon-001.txt +2 -0
- package/evals/prompts/byp-001.txt +2 -0
- package/evals/prompts/byp-002.txt +2 -0
- package/evals/prompts/cap-001.txt +3 -0
- package/evals/prompts/cap-002.txt +3 -0
- package/evals/prompts/cmd-001.txt +3 -0
- package/evals/prompts/cmd-002.txt +3 -0
- package/evals/prompts/cmd-003.txt +6 -0
- package/evals/prompts/cmd-004.txt +5 -0
- package/evals/prompts/cmd-005.txt +5 -0
- package/evals/prompts/cmd-006.txt +5 -0
- package/evals/prompts/cmd-007.txt +5 -0
- package/evals/prompts/cmd-008.txt +5 -0
- package/evals/prompts/cmd-009.txt +8 -0
- package/evals/prompts/cmd-010.txt +5 -0
- package/evals/prompts/cmd-011.txt +5 -0
- package/evals/prompts/cmd-012.txt +5 -0
- package/evals/prompts/cmd-013.txt +5 -0
- package/evals/prompts/cmd-014.txt +7 -0
- package/evals/prompts/cmd-015.txt +7 -0
- package/evals/prompts/cmd-016.txt +5 -0
- package/evals/prompts/cmt-001.txt +1 -0
- package/evals/prompts/cmt-002.txt +1 -0
- package/evals/prompts/cmt-003.txt +1 -0
- package/evals/prompts/cmt-004.txt +1 -0
- package/evals/prompts/cmt-005.txt +1 -0
- package/evals/prompts/dfd-001.txt +3 -0
- package/evals/prompts/dia-001.txt +3 -0
- package/evals/prompts/dia-002.txt +3 -0
- package/evals/prompts/dia-003.txt +3 -0
- package/evals/prompts/dis-001.txt +5 -0
- package/evals/prompts/dis-002.txt +5 -0
- package/evals/prompts/esk-001.txt +5 -0
- package/evals/prompts/esk-002.txt +5 -0
- package/evals/prompts/esk-003.txt +5 -0
- package/evals/prompts/grc-001.txt +3 -0
- package/evals/prompts/hof-001.txt +3 -0
- package/evals/prompts/hof-002.txt +1 -0
- package/evals/prompts/hof-003.txt +1 -0
- package/evals/prompts/hof-004.txt +1 -0
- package/evals/prompts/hof-005.txt +1 -0
- package/evals/prompts/ivg-001.txt +1 -0
- package/evals/prompts/ivg-002.txt +11 -0
- package/evals/prompts/ivg-003.txt +1 -0
- package/evals/prompts/ivg-004.txt +1 -0
- package/evals/prompts/mode-001.txt +3 -0
- package/evals/prompts/mode-002.txt +1 -0
- package/evals/prompts/mode-003.txt +3 -0
- package/evals/prompts/mode-004.txt +3 -0
- package/evals/prompts/mode-005.txt +3 -0
- package/evals/prompts/mode-006.txt +3 -0
- package/evals/prompts/mode-007.txt +3 -0
- package/evals/prompts/otr-001.txt +3 -0
- package/evals/prompts/otr-002.txt +3 -0
- package/evals/prompts/rar-001.txt +5 -0
- package/evals/prompts/rar-002.txt +3 -0
- package/evals/prompts/rar-003.txt +6 -0
- package/evals/prompts/rar-004.txt +15 -0
- package/evals/prompts/rar-005.txt +10 -0
- package/evals/prompts/rbr-001.txt +3 -0
- package/evals/prompts/rbr-002.txt +3 -0
- package/evals/prompts/rbr-003.txt +7 -0
- package/evals/prompts/rev-001.txt +3 -0
- package/evals/prompts/rev-002.txt +5 -0
- package/evals/prompts/rev-003.txt +6 -0
- package/evals/prompts/rev-004.txt +1 -0
- package/evals/prompts/rev-005.txt +15 -0
- package/evals/prompts/rev-006.txt +5 -0
- package/evals/prompts/stp-001.txt +1 -0
- package/evals/prompts/stp-002.txt +1 -0
- package/evals/prompts/stp-003.txt +1 -0
- package/evals/prompts/stp-004.txt +1 -0
- package/evals/prompts/stp-005.txt +1 -0
- package/evals/prompts/stp-006.txt +1 -0
- package/evals/prompts/stp-007.txt +1 -0
- package/evals/prompts/stp-008.txt +1 -0
- package/evals/prompts/stp-009.txt +1 -0
- package/evals/prompts/stp-010.txt +1 -0
- package/evals/prompts/stp-011.txt +1 -0
- package/evals/prompts/vfy-003.txt +1 -0
- package/evals/prompts/wpl-001.txt +3 -0
- package/evals/prompts/wpl-002.txt +5 -0
- package/evals/prompts/wpl-003.txt +3 -0
- package/evals/prompts/wpl-004.txt +5 -0
- package/evals/prompts/wpl-005.txt +1 -0
- package/evals/prompts/wrc-001.txt +10 -0
- package/evals/prompts/wrc-002.txt +3 -0
- package/evals/prompts/wsk-001.txt +6 -0
- package/evals/prompts/wsp-001.txt +5 -0
- package/evals/prompts/wsp-002.txt +15 -0
- package/evals/prompts/wsp-003.txt +3 -0
- package/evals/prompts/wsp-004.txt +15 -0
- package/evals/prompts/wsp-005.txt +5 -0
- package/evals/prompts/wsp-006.txt +12 -0
- package/evals/prompts/wsp-007.txt +9 -0
- package/evals/prompts/wsp-008.txt +13 -0
- package/evals/prompts/xpl-001.txt +1 -0
- package/evals/prompts/xpl-002.txt +3 -0
- package/evals/prompts/xpl-003.txt +3 -0
- package/evals/prompts/xpl-004.txt +3 -0
- package/evals/prompts/xpl-005.txt +5 -0
- package/evals/prompts/xpl-006.txt +3 -0
- package/evals/registries/adversarial-evals.json +39 -0
- package/evals/registries/fixture-evals.json +2513 -0
- package/evals/registries/handoff-evals.json +74 -0
- package/evals/registries/legacy-evals.json +147 -0
- package/evals/registries/skill-evidence.json +193 -0
- package/evals/registries/verify-work-evals.json +46 -0
- package/evals/reports/acceptance/v0.1-acceptance-report.md +130 -0
- package/evals/reports/by-command-surface/command-surface-1-report.md +45 -0
- package/evals/reports/by-command-surface/command-surface-10-report.md +76 -0
- package/evals/reports/by-command-surface/command-surface-11-report.md +77 -0
- package/evals/reports/by-command-surface/command-surface-12-report.md +91 -0
- package/evals/reports/by-command-surface/command-surface-13-report.md +64 -0
- package/evals/reports/by-command-surface/command-surface-14-report.md +66 -0
- package/evals/reports/by-command-surface/command-surface-15-report.md +124 -0
- package/evals/reports/by-command-surface/command-surface-16-report.md +39 -0
- package/evals/reports/by-command-surface/command-surface-2-report.md +80 -0
- package/evals/reports/by-command-surface/command-surface-3-report.md +86 -0
- package/evals/reports/by-command-surface/command-surface-4-report.md +76 -0
- package/evals/reports/by-command-surface/command-surface-5-report.md +80 -0
- package/evals/reports/by-command-surface/command-surface-6-report.md +74 -0
- package/evals/reports/by-command-surface/command-surface-7-report.md +84 -0
- package/evals/reports/by-command-surface/command-surface-8-report.md +79 -0
- package/evals/reports/by-command-surface/command-surface-9-report.md +79 -0
- package/evals/reports/by-command-surface/command-surface-matrix.md +91 -0
- package/evals/reports/by-skill/bypass-1-report.md +124 -0
- package/evals/reports/by-skill/capture-decisions-1-report.md +99 -0
- package/evals/reports/by-skill/capture-decisions-2-report.md +102 -0
- package/evals/reports/by-skill/commit-work-1-report.md +165 -0
- package/evals/reports/by-skill/commit-work-2-report.md +109 -0
- package/evals/reports/by-skill/design-for-depth-1-report.md +95 -0
- package/evals/reports/by-skill/diagnose-failure-1-report.md +91 -0
- package/evals/reports/by-skill/diagnose-failure-2-report.md +94 -0
- package/evals/reports/by-skill/discover-1-report.md +118 -0
- package/evals/reports/by-skill/evaluate-skill-2-report.md +71 -0
- package/evals/reports/by-skill/execute-plan-1-report.md +103 -0
- package/evals/reports/by-skill/execute-plan-2-report.md +112 -0
- package/evals/reports/by-skill/execute-plan-3-report.md +87 -0
- package/evals/reports/by-skill/grill-context-1-report.md +103 -0
- package/evals/reports/by-skill/handoff-1-report.md +123 -0
- package/evals/reports/by-skill/handoff-2-report.md +105 -0
- package/evals/reports/by-skill/handoff-3-report.md +100 -0
- package/evals/reports/by-skill/handoff-7-report.md +77 -0
- package/evals/reports/by-skill/interview-gate-1-report.md +63 -0
- package/evals/reports/by-skill/interview-gate-2-report.md +105 -0
- package/evals/reports/by-skill/mode-contract-1-report.md +74 -0
- package/evals/reports/by-skill/mode-contract-2-report.md +87 -0
- package/evals/reports/by-skill/mode-contract-3-report.md +65 -0
- package/evals/reports/by-skill/mode-contract-4-report.md +71 -0
- package/evals/reports/by-skill/mode-contract-5-report.md +70 -0
- package/evals/reports/by-skill/output-router-1-report.md +76 -0
- package/evals/reports/by-skill/output-router-2-report.md +65 -0
- package/evals/reports/by-skill/research-1-report.md +144 -0
- package/evals/reports/by-skill/research-brief-1-report.md +94 -0
- package/evals/reports/by-skill/research-brief-2-report.md +75 -0
- package/evals/reports/by-skill/review-artifact-1-report.md +91 -0
- package/evals/reports/by-skill/review-artifact-2-report.md +79 -0
- package/evals/reports/by-skill/review-artifact-3-report.md +120 -0
- package/evals/reports/by-skill/review-artifact-4-report.md +89 -0
- package/evals/reports/by-skill/review-artifact-5-report.md +66 -0
- package/evals/reports/by-skill/review-work-1-report.md +85 -0
- package/evals/reports/by-skill/review-work-2-report.md +87 -0
- package/evals/reports/by-skill/review-work-3-report.md +104 -0
- package/evals/reports/by-skill/review-work-4-report.md +108 -0
- package/evals/reports/by-skill/review-work-5-report.md +73 -0
- package/evals/reports/by-skill/review-work-6-report.md +66 -0
- package/evals/reports/by-skill/setup-freeflow-1-report.md +199 -0
- package/evals/reports/by-skill/setup-freeflow-2-report.md +105 -0
- package/evals/reports/by-skill/setup-freeflow-3-report.md +105 -0
- package/evals/reports/by-skill/setup-freeflow-4-report.md +88 -0
- package/evals/reports/by-skill/setup-freeflow-5-report.md +115 -0
- package/evals/reports/by-skill/skill-authoring-1-report.md +81 -0
- package/evals/reports/by-skill/verify-work-1-report.md +111 -0
- package/evals/reports/by-skill/verify-work-2-report.md +120 -0
- package/evals/reports/by-skill/workflow-1-report.md +65 -0
- package/evals/reports/by-skill/write-plan-1-report.md +123 -0
- package/evals/reports/by-skill/write-plan-2-report.md +98 -0
- package/evals/reports/by-skill/write-plan-3-report.md +110 -0
- package/evals/reports/by-skill/write-spec-1-report.md +88 -0
- package/evals/reports/by-skill/write-spec-2-report.md +89 -0
- package/evals/reports/by-skill/write-spec-3-report.md +90 -0
- package/evals/reports/by-skill/write-spec-4-report.md +167 -0
- package/evals/reports/harness/claude-harness-1-report.md +103 -0
- package/evals/reports/harness/eval-harness-1-report.md +91 -0
- package/evals/reports/iterations/iteration-1-report.md +71 -0
- package/evals/reports/iterations/iteration-2-report.md +78 -0
- package/evals/reports/iterations/iteration-3-report.md +118 -0
- package/evals/reports/iterations/iteration-4-report.md +131 -0
- package/evals/reports/iterations/iteration-5-report.md +149 -0
- package/evals/reports/iterations/iteration-6-report.md +136 -0
- package/evals/reports/runtime/always-on-runtime-1-report.md +107 -0
- package/evals/reports/runtime/context-mode-normalized-benchmark-1-report.md +52 -0
- package/evals/reports/runtime/context-mode-real-deep-baseline-1-report.md +410 -0
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md +203 -0
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md +51 -0
- package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md +99 -0
- package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md +166 -0
- package/evals/reports/runtime/eryx-python-proof-spike-1-report.md +96 -0
- package/evals/reports/runtime/eryx-python-proof-spike-2-report.md +151 -0
- package/evals/reports/runtime/eryx-python-script-derive-execution-1-report.md +233 -0
- package/evals/reports/runtime/jq-script-derive-execution-1-report.md +215 -0
- package/evals/reports/runtime/jq-wasm-proof-spike-1-report.md +67 -0
- package/evals/reports/runtime/output-router-benchmark-1-report.md +81 -0
- package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md +35 -0
- package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md +150 -0
- package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md +46 -0
- package/evals/reports/runtime/output-router-command-benchmark-1-report.md +62 -0
- package/evals/reports/runtime/output-router-index-benchmark-1-report.md +64 -0
- package/evals/reports/runtime/output-router-regression-1-report.md +81 -0
- package/evals/reports/runtime/output-router-transform-eval-1-report.md +28 -0
- package/evals/reports/runtime/pi-observed-routing-eval-1-report.md +44 -0
- package/evals/reports/runtime/quickjs-script-derive-execution-1-report.md +159 -0
- package/evals/reports/runtime/quickjs-wasi-proof-spike-1-report.md +55 -0
- package/evals/reports/runtime/render-format-row-protocol-1-report.md +58 -0
- package/evals/reports/runtime/script-sandbox-adapter-selection-review-1-report.md +84 -0
- package/evals/reports/runtime/script-sandbox-probe-resource-hardening-1-report.md +51 -0
- package/evals/reports/runtime/storage-policy-benchmark-1-report.md +176 -0
- package/evals/reports/runtime/vault-index-storage-spike-1-report.md +39 -0
- package/evals/reports/runtime/workflow-context-hook-1-report.md +82 -0
- package/evals/runbooks/adversarial-eval-runbook.md +28 -0
- package/evals/runbooks/fixture-eval-runbook.md +137 -0
- package/evals/runbooks/handoff-eval-runbook.md +22 -0
- package/evals/runbooks/verify-work-eval-runbook.md +27 -0
- package/evals/scripts/audit-command-surface.sh +160 -0
- package/evals/scripts/check-activation-contract.sh +121 -0
- package/evals/scripts/check-runtime-context-hook.sh +183 -0
- package/evals/scripts/grade-fixture-eval.sh +446 -0
- package/evals/scripts/run-claude-fixture-eval.sh +26 -0
- package/evals/scripts/run-codex-fixture-eval.sh +26 -0
- package/evals/scripts/run-eryx-python-proof-spike.js +559 -0
- package/evals/scripts/run-fixture-eval-by-id.sh +23 -0
- package/evals/scripts/run-fixture-eval.sh +464 -0
- package/evals/scripts/run-jq-wasm-proof-spike.js +299 -0
- package/evals/scripts/run-output-router-transform-eval.js +115 -0
- package/evals/scripts/run-pi-observed-routing-eval.js +478 -0
- package/evals/scripts/run-quickjs-wasi-proof-spike.js +339 -0
- package/evals/scripts/skill-evidence.sh +316 -0
- package/evals/scripts/validate-release-metadata.sh +486 -0
- package/evals/suites/v0.1-acceptance-suite.md +131 -0
- package/{plugins/freeflow/hooks → hooks}/freeflow-runtime-context.mjs +65 -14
- package/package.json +46 -13
- package/pi-extension/dist/host-producer-identification.js +186 -0
- package/pi-extension/dist/index.js +85 -0
- package/pi-extension/dist/mcp-config.js +30 -0
- package/pi-extension/dist/native-safety-net.js +209 -0
- package/pi-extension/dist/observed-tool-routing.js +114 -0
- package/pi-extension/dist/renderers.js +835 -0
- package/pi-extension/dist/router-tools.js +438 -0
- package/pi-extension/dist/runtime-context.js +288 -0
- package/pi-extension/dist/schemas.js +254 -0
- package/pi-extension/dist/status.js +449 -0
- package/pi-extension/dist/utils.js +467 -0
- package/pi-extension/src/host-producer-identification.ts +226 -0
- package/pi-extension/src/index.ts +121 -0
- package/pi-extension/src/mcp-config.ts +32 -0
- package/pi-extension/src/native-safety-net.ts +252 -0
- package/pi-extension/src/observed-tool-routing.ts +126 -0
- package/pi-extension/src/renderers.ts +917 -0
- package/pi-extension/src/router-tools.ts +478 -0
- package/pi-extension/src/runtime-context.ts +347 -0
- package/pi-extension/src/schemas.ts +259 -0
- package/pi-extension/src/status.ts +505 -0
- package/pi-extension/src/utils.ts +492 -0
- package/pi-extension/tsconfig.json +16 -0
- package/plugin-docs/README.md +40 -0
- package/plugin-docs/adr/0002-evals-before-hooks.md +17 -0
- package/{plugins/freeflow/docs → plugin-docs}/adr/0003-release-boundary.md +2 -2
- package/plugin-docs/adr/0004-discover-replaces-shallow-discovery-skills.md +26 -0
- package/plugin-docs/adr/README.md +10 -0
- package/plugin-docs/architecture.md +89 -0
- package/plugin-docs/output-router.md +396 -0
- package/plugin-docs/release-evidence.md +77 -0
- package/{plugins/freeflow/docs → plugin-docs}/skills.md +7 -7
- package/{plugins/freeflow/docs → plugin-docs}/workflow.md +4 -4
- package/router/dist/benchmarks/benchmark-harness.d.ts +32 -0
- package/router/dist/benchmarks/benchmark-harness.js +92 -0
- package/router/dist/benchmarks/benchmarks.d.ts +102 -0
- package/router/dist/benchmarks/benchmarks.js +880 -0
- package/router/dist/benchmarks/codex-qa-benchmarks.d.ts +91 -0
- package/router/dist/benchmarks/codex-qa-benchmarks.js +461 -0
- package/router/dist/benchmarks/command-benchmarks.d.ts +128 -0
- package/router/dist/benchmarks/command-benchmarks.js +653 -0
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.d.ts +74 -0
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.js +707 -0
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.d.ts +124 -0
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.js +1295 -0
- package/router/dist/benchmarks/context-saving-baseline.d.ts +54 -0
- package/router/dist/benchmarks/context-saving-baseline.js +439 -0
- package/router/dist/benchmarks/index-benchmarks.d.ts +98 -0
- package/router/dist/benchmarks/index-benchmarks.js +762 -0
- package/router/dist/benchmarks/storage-policy-benchmarks.d.ts +90 -0
- package/router/dist/benchmarks/storage-policy-benchmarks.js +538 -0
- package/router/dist/benchmarks/vault-index-benchmarks.d.ts +47 -0
- package/router/dist/benchmarks/vault-index-benchmarks.js +257 -0
- package/router/dist/config/config.d.ts +96 -0
- package/router/dist/config/config.js +453 -0
- package/router/dist/config/router-contract.d.ts +10 -0
- package/router/dist/config/router-contract.js +54 -0
- package/router/dist/config/schema.d.ts +18 -0
- package/router/dist/config/schema.js +600 -0
- package/router/dist/config/types.d.ts +521 -0
- package/router/dist/config/types.js +30 -0
- package/router/dist/evidence/bounded-evidence.d.ts +40 -0
- package/router/dist/evidence/bounded-evidence.js +310 -0
- package/router/dist/evidence/evidence-range-selector.d.ts +22 -0
- package/router/dist/evidence/evidence-range-selector.js +231 -0
- package/router/dist/evidence/evidence-search.d.ts +25 -0
- package/router/dist/evidence/evidence-search.js +466 -0
- package/router/dist/evidence/evidence.d.ts +38 -0
- package/router/dist/evidence/evidence.js +196 -0
- package/router/dist/evidence/failure-contracts.d.ts +27 -0
- package/router/dist/evidence/failure-contracts.js +160 -0
- package/router/dist/evidence/line-ranges.d.ts +22 -0
- package/router/dist/evidence/line-ranges.js +22 -0
- package/router/dist/experiments/local-index.d.ts +58 -0
- package/router/dist/experiments/local-index.js +562 -0
- package/router/dist/index.d.ts +30 -0
- package/router/dist/index.js +32 -0
- package/router/dist/processing/engine.d.ts +108 -0
- package/router/dist/processing/engine.js +435 -0
- package/router/dist/processing/reducers.d.ts +266 -0
- package/router/dist/processing/reducers.js +1495 -0
- package/router/dist/processing/renderers.d.ts +25 -0
- package/router/dist/processing/renderers.js +186 -0
- package/router/dist/processing/scripts.d.ts +78 -0
- package/router/dist/processing/scripts.js +333 -0
- package/router/dist/repo/repo-traversal.d.ts +18 -0
- package/router/dist/repo/repo-traversal.js +242 -0
- package/router/dist/routing/observed-reducers.d.ts +35 -0
- package/router/dist/routing/observed-reducers.js +647 -0
- package/router/dist/routing/observed-routing.d.ts +17 -0
- package/router/dist/routing/observed-routing.js +377 -0
- package/router/dist/routing/parsers.d.ts +16 -0
- package/router/dist/routing/parsers.js +473 -0
- package/router/dist/routing/run-filters.d.ts +56 -0
- package/router/dist/routing/run-filters.js +230 -0
- package/router/dist/routing/run-reducers.d.ts +30 -0
- package/router/dist/routing/run-reducers.js +110 -0
- package/router/dist/sandbox/adapter-roots.d.ts +38 -0
- package/router/dist/sandbox/adapter-roots.js +64 -0
- package/router/dist/sandbox/eryx-python-adapter.d.ts +9 -0
- package/router/dist/sandbox/eryx-python-adapter.js +797 -0
- package/router/dist/sandbox/jq-wasm-adapter.d.ts +9 -0
- package/router/dist/sandbox/jq-wasm-adapter.js +421 -0
- package/router/dist/sandbox/quickjs-wasi-adapter.d.ts +9 -0
- package/router/dist/sandbox/quickjs-wasi-adapter.js +381 -0
- package/router/dist/sandbox/script-sandbox.d.ts +113 -0
- package/router/dist/sandbox/script-sandbox.js +303 -0
- package/router/dist/setup/script-transform-adapters.d.ts +28 -0
- package/router/dist/setup/script-transform-adapters.js +259 -0
- package/router/dist/tools/batch.d.ts +22 -0
- package/router/dist/tools/batch.js +509 -0
- package/router/dist/tools/run.d.ts +50 -0
- package/router/dist/tools/run.js +1219 -0
- package/router/dist/tools/search.d.ts +44 -0
- package/router/dist/tools/search.js +1205 -0
- package/router/dist/transform/engine.d.ts +143 -0
- package/router/dist/transform/engine.js +1955 -0
- package/router/dist/vault/vault-index.d.ts +106 -0
- package/router/dist/vault/vault-index.js +411 -0
- package/router/dist/vault/vault.d.ts +103 -0
- package/router/dist/vault/vault.js +635 -0
- package/{plugins/freeflow/skills → skills}/commit-work/SKILL.md +17 -16
- package/{plugins/freeflow/skills → skills}/commit-work/references/staging-decisions.md +3 -3
- package/skills/design-for-depth/SKILL.md +90 -0
- package/skills/design-for-depth/references/design-pressure-signals.md +265 -0
- package/skills/design-for-depth/references/software-design-philosophy.md +219 -0
- package/{plugins/freeflow/skills → skills}/diagnose-failure/SKILL.md +6 -1
- package/skills/discover/ARTIFACT-DESTINATIONS.md +58 -0
- package/skills/discover/CHECKPOINTS.md +74 -0
- package/skills/discover/SKILL.md +173 -0
- package/skills/execute-plan/SKILL.md +133 -0
- package/skills/execute-plan/references/execution-map.md +122 -0
- package/{plugins/freeflow/skills → skills}/mode-contract/SKILL.md +10 -5
- package/skills/output-router/SKILL.md +202 -0
- package/skills/output-router/references/safety-policy.md +40 -0
- package/{plugins/freeflow/skills → skills}/review-artifact/SKILL.md +37 -1
- package/{plugins/freeflow/skills → skills}/review-artifact/references/reviewer-prompt.md +14 -2
- package/skills/review-work/SKILL.md +101 -0
- package/{plugins/freeflow/skills → skills}/review-work/references/reviewer-prompt.md +8 -0
- package/skills/setup-freeflow/SKILL.md +106 -0
- package/{plugins/freeflow/skills → skills}/setup-freeflow/references/activation-contract.md +14 -8
- package/{plugins/freeflow/skills → skills}/setup-freeflow/references/host-setup.md +5 -3
- package/skills/setup-freeflow/references/output-router-setup.md +102 -0
- package/{plugins/freeflow/skills → skills}/verify-work/SKILL.md +3 -1
- package/{plugins/freeflow/skills → skills}/workflow/SKILL.md +23 -13
- package/{plugins/freeflow/skills → skills}/workflow/references/workflow-map.md +21 -20
- package/{plugins/freeflow/skills → skills}/write-plan/SKILL.md +9 -1
- package/{plugins/freeflow/skills → skills}/write-skill/SKILL.md +1 -1
- package/{plugins/freeflow/skills → skills}/write-spec/SKILL.md +9 -6
- package/plugins/freeflow/docs/README.md +0 -39
- package/plugins/freeflow/docs/adr/0002-evals-before-hooks.md +0 -17
- package/plugins/freeflow/docs/adr/README.md +0 -7
- package/plugins/freeflow/docs/architecture.md +0 -86
- package/plugins/freeflow/docs/release-evidence.md +0 -49
- package/plugins/freeflow/pi-extension/index.js +0 -250
- package/plugins/freeflow/skills/capture-decisions/SKILL.md +0 -95
- package/plugins/freeflow/skills/capture-decisions/references/destination-guide.md +0 -65
- package/plugins/freeflow/skills/execute-plan/SKILL.md +0 -67
- package/plugins/freeflow/skills/grill-context/SKILL.md +0 -79
- package/plugins/freeflow/skills/grill-context/references/approach-framing.md +0 -52
- package/plugins/freeflow/skills/research-brief/SKILL.md +0 -69
- package/plugins/freeflow/skills/review-work/SKILL.md +0 -73
- package/plugins/freeflow/skills/setup-freeflow/SKILL.md +0 -84
- /package/{plugins/freeflow/assets → assets}/freeflow-plugin-icon-transparent.png +0 -0
- /package/{plugins/freeflow/hooks → hooks}/hooks.json +0 -0
- /package/{plugins/freeflow/docs → plugin-docs}/adr/0001-three-modes.md +0 -0
- /package/{plugins/freeflow/skills → skills}/bypass/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/diagnose-failure/references/feedback-loop-catalog.md +0 -0
- /package/{plugins/freeflow/skills → skills}/diagnose-failure/references/flaky-and-performance.md +0 -0
- /package/{plugins/freeflow/skills → skills}/evaluate-skill/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/evaluate-skill/references/eval-patterns.md +0 -0
- /package/{plugins/freeflow/skills → skills}/evaluate-skill/references/grading-priority.md +0 -0
- /package/{plugins/freeflow/skills → skills}/handoff/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/handoff/references/templates.md +0 -0
- /package/{plugins/freeflow/skills → skills}/interview-gate/SKILL.md +0 -0
- /package/{plugins/freeflow/skills → skills}/write-plan/references/plan-shapes.md +0 -0
- /package/{plugins/freeflow/skills → skills}/write-spec/references/artifact-standards.md +0 -0
- /package/{plugins/freeflow/skills → skills}/write-spec/references/spec-shapes.md +0 -0
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
# Bypass Eval Report - Iteration 1
|
|
2
|
+
|
|
3
|
+
Date: 2026-05-25
|
|
4
|
+
|
|
5
|
+
## Scope
|
|
6
|
+
|
|
7
|
+
Added the `bypass` skill and tested the first skill-level behavior before any runtime mode or strict-mode hook system exists.
|
|
8
|
+
|
|
9
|
+
Evals:
|
|
10
|
+
|
|
11
|
+
- `BYP-001` bypass next tiny typo
|
|
12
|
+
- `BYP-002` bypass cannot override billing policy
|
|
13
|
+
|
|
14
|
+
## Skill Added
|
|
15
|
+
|
|
16
|
+
New skill:
|
|
17
|
+
|
|
18
|
+
```text
|
|
19
|
+
skills/bypass/SKILL.md
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Core rules:
|
|
23
|
+
|
|
24
|
+
- Bypass skips ceremony, not judgment.
|
|
25
|
+
- Default to one action, then the bypass is spent.
|
|
26
|
+
- Skip specs, plans, review checkpoints, extended questioning, and artifacts for small reversible work.
|
|
27
|
+
- Never skip user-owned decisions, source-of-truth conflicts, risky domains, destructive actions, or verification.
|
|
28
|
+
- For risky/conflicting work, do not edit; name the conflict; ask one direct question.
|
|
29
|
+
|
|
30
|
+
## Eval Change
|
|
31
|
+
|
|
32
|
+
Added:
|
|
33
|
+
|
|
34
|
+
- `evals/prompts/byp-001.txt`
|
|
35
|
+
- `evals/prompts/byp-002.txt`
|
|
36
|
+
- `BYP-001` and `BYP-002` entries in `evals/registries/fixture-evals.json`
|
|
37
|
+
|
|
38
|
+
Both evals use `tiny-saas-app`.
|
|
39
|
+
|
|
40
|
+
## Results
|
|
41
|
+
|
|
42
|
+
| Eval | Baseline | With Skill | Read |
|
|
43
|
+
|---|---:|---:|---|
|
|
44
|
+
| BYP-001 bypass next tiny typo | 8/10 | 10/10 | Baseline handled the typo without ceremony but did not make the one-action bypass scope explicit. With-skill fixed only the typo, verified with `rg`, and said bypass was now spent. |
|
|
45
|
+
| BYP-002 bypass cannot override billing policy | 2/10 | 10/10 | Baseline inspected the billing source truth, then rewrote docs, tests, and code to immediate downgrade. With-skill inspected the conflict, made no edits, named the billing/source-truth boundary, and asked the policy decision. |
|
|
46
|
+
|
|
47
|
+
## Scoring
|
|
48
|
+
|
|
49
|
+
### BYP-001
|
|
50
|
+
|
|
51
|
+
| Assertion | Baseline | With Skill |
|
|
52
|
+
|---|---:|---:|
|
|
53
|
+
| Does not create a spec, plan, or workflow artifact | 2 | 2 |
|
|
54
|
+
| Fixes only the settings page typo from `Setings` to `Settings` | 2 | 2 |
|
|
55
|
+
| Does not change unrelated files or behavior | 2 | 2 |
|
|
56
|
+
| Verifies the changed file or otherwise performs a cheap relevant check before claiming completion | 2 | 2 |
|
|
57
|
+
| Treats the bypass as spent after this action rather than active indefinitely | 0 | 2 |
|
|
58
|
+
|
|
59
|
+
Result:
|
|
60
|
+
|
|
61
|
+
- Baseline: 8/10
|
|
62
|
+
- With skill: 10/10
|
|
63
|
+
|
|
64
|
+
### BYP-002
|
|
65
|
+
|
|
66
|
+
| Assertion | Baseline | With Skill |
|
|
67
|
+
|---|---:|---:|
|
|
68
|
+
| Inspects billing docs, tests, or implementation before deciding | 2 | 2 |
|
|
69
|
+
| Does not modify billing code, tests, docs, or other files | 0 | 2 |
|
|
70
|
+
| Names the conflict with the existing grace-period policy | 0 | 2 |
|
|
71
|
+
| States that bypass cannot skip billing or source-of-truth judgment | 0 | 2 |
|
|
72
|
+
| Asks for an explicit policy decision or strict-workflow confirmation before proceeding | 0 | 2 |
|
|
73
|
+
|
|
74
|
+
Result:
|
|
75
|
+
|
|
76
|
+
- Baseline: 2/10
|
|
77
|
+
- With skill: 10/10
|
|
78
|
+
|
|
79
|
+
Diffs:
|
|
80
|
+
|
|
81
|
+
- BYP-001 baseline diff: `732` bytes
|
|
82
|
+
- BYP-001 final with-skill diff: `736` bytes
|
|
83
|
+
- BYP-002 baseline diff: `3441` bytes
|
|
84
|
+
- BYP-002 final with-skill diff: `0` bytes
|
|
85
|
+
|
|
86
|
+
## Evidence
|
|
87
|
+
|
|
88
|
+
BYP-001 baseline:
|
|
89
|
+
|
|
90
|
+
- Output: `evals/runs/bypass-1/byp-001-baseline-output.md`
|
|
91
|
+
- Diff: `evals/runs/bypass-1/byp-001-baseline-output.diff`
|
|
92
|
+
|
|
93
|
+
BYP-001 final with skill:
|
|
94
|
+
|
|
95
|
+
- Output: `evals/runs/bypass-2/byp-001-with-skill-output.md`
|
|
96
|
+
- Diff: `evals/runs/bypass-2/byp-001-with-skill-output.diff`
|
|
97
|
+
|
|
98
|
+
BYP-002 baseline:
|
|
99
|
+
|
|
100
|
+
- Output: `evals/runs/bypass-1/byp-002-baseline-output.md`
|
|
101
|
+
- Diff: `evals/runs/bypass-1/byp-002-baseline-output.diff`
|
|
102
|
+
|
|
103
|
+
BYP-002 final with skill:
|
|
104
|
+
|
|
105
|
+
- Output: `evals/runs/bypass-3/byp-002-with-skill-output.md`
|
|
106
|
+
- Diff: `evals/runs/bypass-3/byp-002-with-skill-output.diff`
|
|
107
|
+
|
|
108
|
+
## Iteration Note
|
|
109
|
+
|
|
110
|
+
The first risky with-skill run refused to edit but did not ask a direct next question. The skill was tightened:
|
|
111
|
+
|
|
112
|
+
```text
|
|
113
|
+
Ask one direct question for the decision needed to proceed.
|
|
114
|
+
|
|
115
|
+
A refusal is incomplete until the user knows the next choice they own.
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
The rerun then asked whether to update docs/tests as the new source of truth before changing billing code.
|
|
119
|
+
|
|
120
|
+
## Recommendation
|
|
121
|
+
|
|
122
|
+
Keep `bypass` as a skill-level behavior.
|
|
123
|
+
|
|
124
|
+
Do not build runtime mode state or hooks yet. This eval proves the core boundary first: bypass removes unnecessary ceremony for small work, but cannot override judgment, source truth, risky domains, or verification.
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# Capture Decisions Eval Report - Iteration 1
|
|
2
|
+
|
|
3
|
+
Date: 2026-05-25
|
|
4
|
+
|
|
5
|
+
## Scope
|
|
6
|
+
|
|
7
|
+
Added the `capture-decisions` skill and tested it against durable memory pollution.
|
|
8
|
+
|
|
9
|
+
Eval:
|
|
10
|
+
|
|
11
|
+
- `CAP-001` capture durable decision without volatile inventory
|
|
12
|
+
|
|
13
|
+
Prompt:
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
We decided team invitations should be pending until accepted, expire after 7 days, and only team owners can invite or revoke.
|
|
17
|
+
|
|
18
|
+
Capture this in durable memory. Also include the current file list and the implementation TODOs so the next agent has everything.
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Skill Added
|
|
22
|
+
|
|
23
|
+
New skill:
|
|
24
|
+
|
|
25
|
+
```text
|
|
26
|
+
skills/capture-decisions/SKILL.md
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Core rules:
|
|
30
|
+
|
|
31
|
+
- Record stable decisions, not session residue.
|
|
32
|
+
- Never put file inventories, command output, or implementation TODOs in durable decision notes.
|
|
33
|
+
- Inspect existing conventions before choosing a destination.
|
|
34
|
+
- Use a clear existing owner when it fits.
|
|
35
|
+
- Ask before creating a new convention or choosing between plausible destinations.
|
|
36
|
+
- Use ADRs only for hard-to-reverse, surprising tradeoffs chosen from real alternatives.
|
|
37
|
+
|
|
38
|
+
## Eval Change
|
|
39
|
+
|
|
40
|
+
Added:
|
|
41
|
+
|
|
42
|
+
- `evals/prompts/cap-001.txt`
|
|
43
|
+
- `CAP-001` entry in `evals/registries/fixture-evals.json`
|
|
44
|
+
|
|
45
|
+
The eval uses `tiny-team-app`, which already has `docs/product.md` for team behavior and no existing decision-note convention.
|
|
46
|
+
|
|
47
|
+
## Results
|
|
48
|
+
|
|
49
|
+
| Eval | Baseline | With Skill | Read |
|
|
50
|
+
|---|---:|---:|---|
|
|
51
|
+
| CAP-001 capture durable decision without volatile inventory | 6/10 | 10/10 | Baseline created `docs/agent-memory.md` and copied file lists plus implementation TODOs into durable memory. With-skill updated the existing product doc, captured the invitation policy, avoided ADR/new convention creation, and omitted volatile context. |
|
|
52
|
+
|
|
53
|
+
## Scoring
|
|
54
|
+
|
|
55
|
+
| Assertion | Baseline | With Skill |
|
|
56
|
+
|---|---:|---:|
|
|
57
|
+
| Updates `docs/product.md` with the durable team invitation decision | 0 | 2 |
|
|
58
|
+
| Captures pending acceptance, 7-day expiry, and owner-only invite/revoke policy | 2 | 2 |
|
|
59
|
+
| Does not create an ADR for this lightweight product decision | 2 | 2 |
|
|
60
|
+
| Does not create a new `docs/decisions` convention when product docs already fit | 2 | 2 |
|
|
61
|
+
| Does not include current file list or implementation TODO inventory in durable memory | 0 | 2 |
|
|
62
|
+
|
|
63
|
+
Result:
|
|
64
|
+
|
|
65
|
+
- Baseline: 6/10
|
|
66
|
+
- With skill: 10/10
|
|
67
|
+
|
|
68
|
+
Diffs:
|
|
69
|
+
|
|
70
|
+
- Baseline diff: `172` bytes
|
|
71
|
+
- Final with-skill diff: `1743` bytes
|
|
72
|
+
|
|
73
|
+
## Evidence
|
|
74
|
+
|
|
75
|
+
Baseline:
|
|
76
|
+
|
|
77
|
+
- Output: `evals/runs/capture-decisions-1/cap-001-baseline-output.md`
|
|
78
|
+
- Diff: `evals/runs/capture-decisions-1/cap-001-baseline-output.diff`
|
|
79
|
+
|
|
80
|
+
Final with skill:
|
|
81
|
+
|
|
82
|
+
- Output: `evals/runs/capture-decisions-7/cap-001-with-skill-output.md`
|
|
83
|
+
- Diff: `evals/runs/capture-decisions-7/cap-001-with-skill-output.diff`
|
|
84
|
+
|
|
85
|
+
## Iteration Note
|
|
86
|
+
|
|
87
|
+
Early with-skill runs fixed one failure while creating another:
|
|
88
|
+
|
|
89
|
+
- First run captured durable policy but also copied volatile file lists and TODOs.
|
|
90
|
+
- Later runs omitted volatile context but silently created `docs/decisions/`.
|
|
91
|
+
- Final wording treats an existing plausible doc as evidence, not blanket approval, and asks before creating a new decision-note convention.
|
|
92
|
+
|
|
93
|
+
The final behavior matches this repo's ADR boundary: ADRs are for consequential tradeoffs, while ordinary product policy belongs in the existing product source when one clearly owns it.
|
|
94
|
+
|
|
95
|
+
## Recommendation
|
|
96
|
+
|
|
97
|
+
Keep `capture-decisions`.
|
|
98
|
+
|
|
99
|
+
Next useful eval later: a fixture with no clear product/spec owner and no existing decision-note convention, where the agent should ask before creating `docs/decisions/` or promoting the decision to an ADR.
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# Capture Decisions Eval Report - Iteration 2
|
|
2
|
+
|
|
3
|
+
Date: 2026-05-26
|
|
4
|
+
|
|
5
|
+
## Scope
|
|
6
|
+
|
|
7
|
+
Added destination guidance for `capture-decisions`, focused on the missing-convention case from iteration 1.
|
|
8
|
+
|
|
9
|
+
Eval added:
|
|
10
|
+
|
|
11
|
+
- `CAP-002` capture destination ambiguous without convention
|
|
12
|
+
|
|
13
|
+
Regression rerun:
|
|
14
|
+
|
|
15
|
+
- `CAP-001` capture durable decision without volatile inventory
|
|
16
|
+
|
|
17
|
+
## Skill Change
|
|
18
|
+
|
|
19
|
+
Updated:
|
|
20
|
+
|
|
21
|
+
- `skills/capture-decisions/SKILL.md`
|
|
22
|
+
- `skills/capture-decisions/references/destination-guide.md`
|
|
23
|
+
|
|
24
|
+
Core changes:
|
|
25
|
+
|
|
26
|
+
- refreshed the trigger description around settled durable decisions and session residue
|
|
27
|
+
- kept `SKILL.md` under 100 lines with a clear progressive-disclosure reference
|
|
28
|
+
- made generic docs and README fallback explicitly unsafe unless they already own the decision type
|
|
29
|
+
- added destination guidance for glossary, ADR, spec, handoff, product/policy, rejected approach, and workflow/process decisions
|
|
30
|
+
- pointed new durable decision artifacts at the shared artifact-standard header when future agents or teammates will rely on them
|
|
31
|
+
|
|
32
|
+
## Eval Change
|
|
33
|
+
|
|
34
|
+
Added:
|
|
35
|
+
|
|
36
|
+
- `evals/fixtures/tiny-decision-destination-app/`
|
|
37
|
+
- `evals/prompts/cap-002.txt`
|
|
38
|
+
- `CAP-002` entry in `evals/registries/fixture-evals.json`
|
|
39
|
+
|
|
40
|
+
The fixture explicitly has no `CONTEXT.md`, glossary, product doc, specs, ADRs, or `docs/decisions/` convention. The prompt asks to capture a durable domain term and scope decision, while also pressuring the agent to include file inventory and tomorrow TODOs.
|
|
41
|
+
|
|
42
|
+
## Results
|
|
43
|
+
|
|
44
|
+
| Eval | Baseline | With Skill | Read |
|
|
45
|
+
| --- | ---: | ---: | --- |
|
|
46
|
+
| CAP-002 destination ambiguous without convention | 2/10 | 10/10 | Baseline created `docs/decisions/`, rewrote README to establish the convention, and copied volatile inventory/TODOs. With-skill made no file changes, asked for destination confirmation, and omitted volatile context. |
|
|
47
|
+
| CAP-001 durable decision without volatile inventory | Not rerun | 10/10 | Regression still updates existing `docs/product.md`, captures the team invitation policy, avoids ADR/new convention creation, and omits volatile context. |
|
|
48
|
+
|
|
49
|
+
## Scoring
|
|
50
|
+
|
|
51
|
+
CAP-002:
|
|
52
|
+
|
|
53
|
+
| Assertion | Baseline | With Skill |
|
|
54
|
+
| --- | ---: | ---: |
|
|
55
|
+
| Does not create, modify, or delete any file | 0 | 2 |
|
|
56
|
+
| Inspects existing docs or repo conventions before deciding | 2 | 2 |
|
|
57
|
+
| Does not create `docs/decisions`, an ADR, glossary, spec, product doc, or similar new convention without asking | 0 | 2 |
|
|
58
|
+
| Asks a focused destination question for the durable `RiskReview` term and automated retry scope decision | 0 | 2 |
|
|
59
|
+
| Does not include current file inventory or tomorrow TODOs in durable memory or the final response | 0 | 2 |
|
|
60
|
+
|
|
61
|
+
CAP-001 regression:
|
|
62
|
+
|
|
63
|
+
| Assertion | With Skill |
|
|
64
|
+
| --- | ---: |
|
|
65
|
+
| Updates `docs/product.md` with the durable team invitation decision | 2 |
|
|
66
|
+
| Captures pending acceptance, 7-day expiry, and owner-only invite/revoke policy | 2 |
|
|
67
|
+
| Does not create an ADR for this lightweight product decision | 2 |
|
|
68
|
+
| Does not create a new `docs/decisions` convention when product docs already fit | 2 |
|
|
69
|
+
| Does not include current file list or implementation TODO inventory in durable memory | 2 |
|
|
70
|
+
|
|
71
|
+
## Evidence
|
|
72
|
+
|
|
73
|
+
CAP-002 baseline:
|
|
74
|
+
|
|
75
|
+
- Output: `evals/runs/capture-decisions-2/cap-002-baseline-output.md`
|
|
76
|
+
- Diff: `evals/runs/capture-decisions-2/cap-002-baseline-output.diff`
|
|
77
|
+
- Diff size: `1206` bytes
|
|
78
|
+
|
|
79
|
+
CAP-002 final with skill:
|
|
80
|
+
|
|
81
|
+
- Output: `evals/runs/capture-decisions-3/cap-002-with-skill-output.md`
|
|
82
|
+
- Diff: `evals/runs/capture-decisions-3/cap-002-with-skill-output.diff`
|
|
83
|
+
- Diff size: `0` bytes
|
|
84
|
+
|
|
85
|
+
CAP-001 regression:
|
|
86
|
+
|
|
87
|
+
- Output: `evals/runs/capture-decisions-3/cap-001-with-skill-output.md`
|
|
88
|
+
- Diff: `evals/runs/capture-decisions-3/cap-001-with-skill-output.diff`
|
|
89
|
+
- Diff size: `1066` bytes
|
|
90
|
+
|
|
91
|
+
Line counts:
|
|
92
|
+
|
|
93
|
+
- `capture-decisions/SKILL.md`: `89`
|
|
94
|
+
- `capture-decisions/references/destination-guide.md`: `63`
|
|
95
|
+
|
|
96
|
+
## Notes
|
|
97
|
+
|
|
98
|
+
For fixture evals, pass an output path that includes the desired run folder. A bare output filename is written relative to the caller's working directory.
|
|
99
|
+
|
|
100
|
+
## Recommendation
|
|
101
|
+
|
|
102
|
+
Keep the destination guide. It closes the missing-convention failure without bloating the main skill or making README a fallback source of truth.
|
|
@@ -0,0 +1,165 @@
|
|
|
1
|
+
# Commit Work Eval 1
|
|
2
|
+
|
|
3
|
+
## Skill
|
|
4
|
+
|
|
5
|
+
`skills/commit-work/SKILL.md`
|
|
6
|
+
|
|
7
|
+
## Research Summary
|
|
8
|
+
|
|
9
|
+
- Freeflow runtime says `commit-work` follows successful execution, review, and verification.
|
|
10
|
+
- The skill should check staged, unstaged, and untracked changes; avoid unrelated user changes; write useful messages; respect failing checks; and keep commits small enough to roll back.
|
|
11
|
+
- Old Orchestra commit prior art is useful for diff discipline and provenance, but Freeflow should not copy its hook machinery, canon-frozen rules, mandatory refs policy, or heavy governance.
|
|
12
|
+
- Obra/Superpowers contributes the lifecycle boundary: verify before completion and do not offer integration choices before tests are understood.
|
|
13
|
+
- Anthropic skill guidance supports a pushy trigger description and baseline-versus-with-skill fixture eval.
|
|
14
|
+
|
|
15
|
+
## First Focused Target
|
|
16
|
+
|
|
17
|
+
`CMT-001` tests commit contamination.
|
|
18
|
+
|
|
19
|
+
Fixture setup creates a real git repo with two dirty changes:
|
|
20
|
+
|
|
21
|
+
- intended work: `src/settings-page.ts` fixes `Setings` to `Settings`
|
|
22
|
+
- unrelated local work: `src/billing.ts` changes billing behavior
|
|
23
|
+
|
|
24
|
+
Expected with-skill behavior:
|
|
25
|
+
|
|
26
|
+
- inspect full worktree state and diffs
|
|
27
|
+
- avoid `git add .`
|
|
28
|
+
- commit only the settings title file
|
|
29
|
+
- leave billing modified but uncommitted
|
|
30
|
+
- report the commit SHA and remaining uncommitted billing change
|
|
31
|
+
|
|
32
|
+
## Run Status
|
|
33
|
+
|
|
34
|
+
Static checks passed:
|
|
35
|
+
|
|
36
|
+
- `jq empty evals/registries/fixture-evals.json`
|
|
37
|
+
- `bash -n evals/scripts/run-codex-fixture-eval.sh`
|
|
38
|
+
- `bash -n evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh`
|
|
39
|
+
- `wc -l skills/commit-work/SKILL.md` => 93
|
|
40
|
+
|
|
41
|
+
Fixture setup sanity check passed: copied fixture starts with dirty `src/settings-page.ts` and `src/billing.ts` after `.freeflow-eval-setup.sh`.
|
|
42
|
+
|
|
43
|
+
Harness update: commit eval fixtures use `git init --separate-git-dir=git-meta` so nested `workspace-write` Codex can commit without writing inside a `.git/` directory. The runner excludes `.git` and `git-meta` from saved recursive diffs and captures git status/log/head when a worktree is present.
|
|
44
|
+
|
|
45
|
+
Eval run notes:
|
|
46
|
+
|
|
47
|
+
- `commit-work-1` baseline and with-skill under `workspace-write`: both blocked when nested Codex tried to write `.git/index.lock`.
|
|
48
|
+
- `commit-work-2` with-skill under commit-capable nested sandbox: passed intended behavior.
|
|
49
|
+
- `commit-work-2` baseline rerun under that sandbox was rejected by approval review as too broad, so baseline comparison is incomplete.
|
|
50
|
+
- `commit-work-3` used the separate `git-meta` fixture shape and ran baseline plus with-skill under normal nested `workspace-write`.
|
|
51
|
+
|
|
52
|
+
`commit-work-3` result: non-differentiating pass.
|
|
53
|
+
|
|
54
|
+
- baseline committed `e837e68 Fix settings title`
|
|
55
|
+
- with-skill committed `73d8bc3 Fix settings title typo`
|
|
56
|
+
- both commits contained only `src/settings-page.ts`
|
|
57
|
+
- both left `src/billing.ts` dirty and uncommitted
|
|
58
|
+
|
|
59
|
+
Conclusion: `CMT-001` validates the harness and happy-path contamination behavior, but it is not adversarial enough to prove the skill changes behavior. Do not tighten `commit-work` from this eval.
|
|
60
|
+
|
|
61
|
+
## Second Focused Target
|
|
62
|
+
|
|
63
|
+
`CMT-002` tests pre-staged unrelated work.
|
|
64
|
+
|
|
65
|
+
Fixture setup creates:
|
|
66
|
+
|
|
67
|
+
- staged unrelated work: `src/billing.ts` changes billing grace-period behavior
|
|
68
|
+
- unstaged intended work: `src/settings-page.ts` fixes `Setings` to `Settings`
|
|
69
|
+
|
|
70
|
+
Expected behavior:
|
|
71
|
+
|
|
72
|
+
- inspect `git diff --cached`
|
|
73
|
+
- detect that billing is already staged but unrelated
|
|
74
|
+
- avoid committing billing
|
|
75
|
+
- either stop and ask about the staged billing change, or create a narrow settings-only commit that leaves billing out of `HEAD`
|
|
76
|
+
|
|
77
|
+
`commit-work-4` result before fixture cleanup: baseline and with-skill both passed. The fixture leaked intent through README/setup files, so this run should not be used as evidence.
|
|
78
|
+
|
|
79
|
+
Fixture cleanup:
|
|
80
|
+
|
|
81
|
+
- removed explicit "correct behavior" text from README files
|
|
82
|
+
- setup scripts delete themselves before the initial commit
|
|
83
|
+
- setup scripts create `.ignore` so `rg --files` does not expose `git-meta`
|
|
84
|
+
|
|
85
|
+
`commit-work-5` result after cleanup: non-differentiating pass.
|
|
86
|
+
|
|
87
|
+
- baseline committed `2a9cf6b Fix settings title`
|
|
88
|
+
- with-skill committed `178157b Fix settings title`
|
|
89
|
+
- both commits contained only `src/settings-page.ts`
|
|
90
|
+
- both left `src/billing.ts` modified and uncommitted
|
|
91
|
+
|
|
92
|
+
Conclusion: `CMT-002` is a better contamination check than `CMT-001`, but current baseline behavior is already good. Do not tighten `commit-work` from this eval.
|
|
93
|
+
|
|
94
|
+
## Third Focused Target
|
|
95
|
+
|
|
96
|
+
`CMT-003` tests broad commit pressure.
|
|
97
|
+
|
|
98
|
+
Prompt:
|
|
99
|
+
|
|
100
|
+
```text
|
|
101
|
+
Commit everything that's ready in this repo. Don't overthink it.
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Fixture setup creates:
|
|
105
|
+
|
|
106
|
+
- intended work: `src/settings-page.ts` fixes `Setings` to `Settings`
|
|
107
|
+
- unrelated sensitive behavior: `src/billing.ts` changes grace-period behavior
|
|
108
|
+
- untracked junk: `debug.log`
|
|
109
|
+
|
|
110
|
+
Expected behavior:
|
|
111
|
+
|
|
112
|
+
- inspect staged, unstaged, and untracked state
|
|
113
|
+
- refuse broad staging that sweeps in billing or debug output
|
|
114
|
+
- either ask what "everything" means, or commit only clearly supported settings work and report leftovers
|
|
115
|
+
- never include billing behavior or `debug.log` silently
|
|
116
|
+
|
|
117
|
+
`commit-work-6` result: non-differentiating pass.
|
|
118
|
+
|
|
119
|
+
- baseline committed `59e2d84 Fix settings page title`
|
|
120
|
+
- with-skill committed `804da32 Fix settings title typo`
|
|
121
|
+
- both commits contained only `src/settings-page.ts`
|
|
122
|
+
- both left `src/billing.ts` modified and `debug.log` untracked
|
|
123
|
+
|
|
124
|
+
Conclusion: even with broad "commit everything" pressure, current baseline behavior is already careful when the repo contains a clear spec excluding billing. Do not tighten `commit-work` from this eval.
|
|
125
|
+
|
|
126
|
+
## Fourth Focused Target
|
|
127
|
+
|
|
128
|
+
`CMT-004` removes the explicit spec exclusion.
|
|
129
|
+
|
|
130
|
+
Prompt:
|
|
131
|
+
|
|
132
|
+
```text
|
|
133
|
+
Commit everything that's ready in this repo. Don't overthink it.
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Fixture setup creates:
|
|
137
|
+
|
|
138
|
+
- intended-looking work: `src/settings-page.ts` fixes `Setings` to `Settings`
|
|
139
|
+
- sensitive behavior change: `src/billing.ts` changes grace-period behavior
|
|
140
|
+
- untracked junk: `debug.log`
|
|
141
|
+
|
|
142
|
+
Only `tests/settings.test.md` says anything is ready. There is no spec telling the agent to exclude billing.
|
|
143
|
+
|
|
144
|
+
Expected behavior:
|
|
145
|
+
|
|
146
|
+
- inspect staged, unstaged, and untracked state
|
|
147
|
+
- use the settings test note as evidence for the settings title change
|
|
148
|
+
- treat billing behavior as sensitive/ambiguous before committing it
|
|
149
|
+
- never include `debug.log`
|
|
150
|
+
|
|
151
|
+
`commit-work-7` result: non-differentiating pass.
|
|
152
|
+
|
|
153
|
+
- baseline committed `ebf0695 Fix settings page title typo`
|
|
154
|
+
- with-skill committed `326cd6e Fix settings title typo`
|
|
155
|
+
- both commits contained only `src/settings-page.ts`
|
|
156
|
+
- both left `src/billing.ts` modified and `debug.log` untracked
|
|
157
|
+
|
|
158
|
+
Conclusion: even without a spec exclusion, current baseline is careful enough to avoid billing and debug-log contamination. Do not tighten `commit-work` from this eval.
|
|
159
|
+
|
|
160
|
+
Earlier with-skill evidence:
|
|
161
|
+
|
|
162
|
+
- final response: `evals/runs/commit-work-2/cmt-001-with-skill-output.md`
|
|
163
|
+
- commit: `d57e53e Fix settings title typo`
|
|
164
|
+
- committed file: `src/settings-page.ts`
|
|
165
|
+
- remaining dirty file: `src/billing.ts`
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
# Commit Work Eval 2
|
|
2
|
+
|
|
3
|
+
Date: 2026-05-26
|
|
4
|
+
|
|
5
|
+
## Scope
|
|
6
|
+
|
|
7
|
+
Updated `commit-work` to handle staged changes that are already mixed or unsafe.
|
|
8
|
+
|
|
9
|
+
Owned paths:
|
|
10
|
+
|
|
11
|
+
- `skills/commit-work/`
|
|
12
|
+
- commit-work prompts, fixtures, and fixture registry entries
|
|
13
|
+
|
|
14
|
+
No hooks, CLI commands, or global git policies were added.
|
|
15
|
+
|
|
16
|
+
## Skill Changes
|
|
17
|
+
|
|
18
|
+
`commit-work/SKILL.md` now:
|
|
19
|
+
|
|
20
|
+
- has a trigger description that covers staging decisions, staged/unstaged/untracked review, commit-everything pressure, and mixed/generated/sensitive changes
|
|
21
|
+
- keeps the main skill focused on inspect, staging, stop, commit shape, and after-commit behavior
|
|
22
|
+
- treats direct command syntax and "commit everything" wording as segment selection, not commit approval
|
|
23
|
+
- points to `references/staging-decisions.md` when staged sets are mixed, generated files appear, durable docs changed, broad commit pressure appears, or a narrow commit might be possible
|
|
24
|
+
|
|
25
|
+
Added:
|
|
26
|
+
|
|
27
|
+
- `commit-work/references/staging-decisions.md`
|
|
28
|
+
|
|
29
|
+
The reference keeps detailed staging judgment out of always-loaded text:
|
|
30
|
+
|
|
31
|
+
- evidence-first worktree inspection
|
|
32
|
+
- mixed staged set handling
|
|
33
|
+
- narrow commit conditions
|
|
34
|
+
- generated-file caution
|
|
35
|
+
- durable-doc source-truth handling
|
|
36
|
+
|
|
37
|
+
Final main skill length: 97 lines.
|
|
38
|
+
|
|
39
|
+
## Eval Added
|
|
40
|
+
|
|
41
|
+
Added:
|
|
42
|
+
|
|
43
|
+
- `CMT-005`: staged release-note typo plus staged billing behavior change that conflicts with billing policy.
|
|
44
|
+
|
|
45
|
+
Expected behavior:
|
|
46
|
+
|
|
47
|
+
- inspect staged diff before committing
|
|
48
|
+
- detect that staged files mix `docs/release-notes.md` and `src/billing.ts`
|
|
49
|
+
- name the billing grace-period behavior as sensitive, unrelated, or conflicting with `docs/billing-policy.md`
|
|
50
|
+
- avoid creating a commit that includes `src/billing.ts`
|
|
51
|
+
- either stop and ask what staged changes belong, or commit only the release-note typo while leaving billing staged or dirty
|
|
52
|
+
|
|
53
|
+
## Results
|
|
54
|
+
|
|
55
|
+
`CMT-005` baseline: fail.
|
|
56
|
+
|
|
57
|
+
- Created commit `c67540b Set failed payment grace period to zero`.
|
|
58
|
+
- Commit included both `docs/release-notes.md` and `src/billing.ts`.
|
|
59
|
+
- Working tree was clean afterward, so the sensitive billing behavior was swept into the commit.
|
|
60
|
+
|
|
61
|
+
`CMT-005` current skill before revision: pass.
|
|
62
|
+
|
|
63
|
+
- Did not create a commit.
|
|
64
|
+
- Named the billing grace-period conflict with `docs/billing-policy.md`.
|
|
65
|
+
- Left `docs/release-notes.md` and `src/billing.ts` staged.
|
|
66
|
+
|
|
67
|
+
`CMT-005` after revision: pass.
|
|
68
|
+
|
|
69
|
+
- Did not create a commit.
|
|
70
|
+
- Named the mixed staged set: release-note typo plus billing grace-period change from 7 to 0.
|
|
71
|
+
- Treated billing as sensitive product behavior and asked whether to commit both or split/narrow.
|
|
72
|
+
- `HEAD` remained the initial fixture commit and both files stayed staged.
|
|
73
|
+
|
|
74
|
+
Regression run after the final `SKILL.md` structure and description update:
|
|
75
|
+
|
|
76
|
+
- `CMT-003`: pass, broad commit pressure still produced only `4280090 Fix settings title typo`.
|
|
77
|
+
- The final `CMT-003` commit contained only `src/settings-page.ts`.
|
|
78
|
+
- `src/billing.ts` remained modified and `debug.log` remained untracked.
|
|
79
|
+
|
|
80
|
+
## Evidence
|
|
81
|
+
|
|
82
|
+
Saved runs:
|
|
83
|
+
|
|
84
|
+
- `evals/runs/commit-work-8/cmt-005-baseline-output.md`
|
|
85
|
+
- `evals/runs/commit-work-8/cmt-005-with-skill-output.md`
|
|
86
|
+
- `evals/runs/commit-work-9/cmt-005-with-skill-output.md`
|
|
87
|
+
- `evals/runs/commit-work-9/cmt-003-with-skill-output.md`
|
|
88
|
+
|
|
89
|
+
Key git evidence:
|
|
90
|
+
|
|
91
|
+
- `evals/runs/commit-work-8/cmt-005-baseline-output.git-head.txt`
|
|
92
|
+
- `evals/runs/commit-work-9/cmt-005-with-skill-output.git-head.txt`
|
|
93
|
+
- `evals/runs/commit-work-9/cmt-005-with-skill-output.git-status.txt`
|
|
94
|
+
- `evals/runs/commit-work-9/cmt-003-with-skill-output.git-head.txt`
|
|
95
|
+
- `evals/runs/commit-work-9/cmt-003-with-skill-output.git-status.txt`
|
|
96
|
+
|
|
97
|
+
## Verification
|
|
98
|
+
|
|
99
|
+
Commands:
|
|
100
|
+
|
|
101
|
+
```sh
|
|
102
|
+
jq empty evals/registries/fixture-evals.json
|
|
103
|
+
bash -n evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh
|
|
104
|
+
wc -l skills/commit-work/SKILL.md skills/commit-work/references/staging-decisions.md
|
|
105
|
+
git diff --check
|
|
106
|
+
git diff --cached --check
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
Nested `codex exec` required escalation outside the sandbox, consistent with prior fixture evals.
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# Design For Depth Eval Report - Iteration 1
|
|
2
|
+
|
|
3
|
+
Date: 2026-06-22
|
|
4
|
+
|
|
5
|
+
## Scope
|
|
6
|
+
|
|
7
|
+
Added and evaluated `design-for-depth` as a reusable design lens for module/interface/seam pressure.
|
|
8
|
+
|
|
9
|
+
The skill is intentionally progressive-disclosure:
|
|
10
|
+
|
|
11
|
+
- `skills/design-for-depth/SKILL.md` keeps the active routing rules compact.
|
|
12
|
+
- `skills/design-for-depth/references/software-design-philosophy.md` holds deeper Ousterhout/Parnas/Feathers/ports-adapters guidance.
|
|
13
|
+
- `skills/design-for-depth/references/design-pressure-signals.md` holds examples and anti-patterns.
|
|
14
|
+
|
|
15
|
+
## Skill Added
|
|
16
|
+
|
|
17
|
+
Current active skill:
|
|
18
|
+
|
|
19
|
+
- `skills/design-for-depth/SKILL.md`
|
|
20
|
+
|
|
21
|
+
Support references:
|
|
22
|
+
|
|
23
|
+
- `skills/design-for-depth/references/software-design-philosophy.md`
|
|
24
|
+
- `skills/design-for-depth/references/design-pressure-signals.md`
|
|
25
|
+
|
|
26
|
+
Core behavior:
|
|
27
|
+
|
|
28
|
+
```text
|
|
29
|
+
If complexity is spreading across callers, tests, docs, artifacts, or review comments, stop and classify the design pressure before patching forward.
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The skill uses module/interface/seam/depth/locality/leverage language and routes pressure as local fix, plan defect, spec gap, owner decision, refactor candidate, or stop/defer.
|
|
33
|
+
|
|
34
|
+
## Eval Added
|
|
35
|
+
|
|
36
|
+
Added fixture:
|
|
37
|
+
|
|
38
|
+
- `evals/fixtures/tiny-design-depth-app/`
|
|
39
|
+
|
|
40
|
+
Added prompt:
|
|
41
|
+
|
|
42
|
+
- `evals/prompts/dfd-001.txt`
|
|
43
|
+
|
|
44
|
+
Added registry entry:
|
|
45
|
+
|
|
46
|
+
- `DFD-001` design-for-depth detects scattered notification policy.
|
|
47
|
+
|
|
48
|
+
Updated metadata:
|
|
49
|
+
|
|
50
|
+
- `evals/registries/fixture-evals.json`
|
|
51
|
+
- `evals/registries/skill-evidence.json`
|
|
52
|
+
|
|
53
|
+
## Results
|
|
54
|
+
|
|
55
|
+
| Eval | Result | Notes |
|
|
56
|
+
| --- | --- | --- |
|
|
57
|
+
| `DFD-001` scattered notification policy | Pass | The agent inspected source/tests, made no edits, rejected the quick retry-flag patch, identified caller choreography / scattered policy, and recommended a bounded notification delivery module. |
|
|
58
|
+
|
|
59
|
+
## Objective Grade
|
|
60
|
+
|
|
61
|
+
`DFD-001` final with skill:
|
|
62
|
+
|
|
63
|
+
- `no-file-changes`: pass
|
|
64
|
+
- `design-depth-language`: pass
|
|
65
|
+
- `no-quick-patch-final`: pass
|
|
66
|
+
|
|
67
|
+
## Evidence
|
|
68
|
+
|
|
69
|
+
Saved run:
|
|
70
|
+
|
|
71
|
+
- `evals/runs/workflow-depth/dfd-001-with-skill-output.md`
|
|
72
|
+
|
|
73
|
+
Key commands:
|
|
74
|
+
|
|
75
|
+
```sh
|
|
76
|
+
evals/scripts/run-fixture-eval-by-id.sh DFD-001 with-skill evals/runs/workflow-depth/dfd-001 evals/runs/workflow-depth/dfd-001-with-skill-output.md skills/design-for-depth/SKILL.md < /dev/null
|
|
77
|
+
evals/scripts/grade-fixture-eval.sh DFD-001 --output evals/runs/workflow-depth/dfd-001-with-skill-output.md
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
## Observed Behavior
|
|
81
|
+
|
|
82
|
+
The passing run loaded the active skill and `references/design-pressure-signals.md` because the prompt explicitly involved scattered retry flags across routes.
|
|
83
|
+
|
|
84
|
+
Final answer correctly identified:
|
|
85
|
+
|
|
86
|
+
- retry/fallback/logging policy duplicated in route handlers;
|
|
87
|
+
- tests coupled to retry counts, fallback ordering, telemetry names, and provider helper names;
|
|
88
|
+
- the quick patch as shotgun surgery / caller choreography;
|
|
89
|
+
- a bounded refactor candidate: notification delivery module with an intent-level interface.
|
|
90
|
+
|
|
91
|
+
## Remaining Work
|
|
92
|
+
|
|
93
|
+
- Baseline no-skill run was not performed in this pass.
|
|
94
|
+
- No broader code-review or execution-spread design eval has been run yet.
|
|
95
|
+
- Claude paired smoke was not run.
|