@hassangameryt/freeflow 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +4 -4
- package/.claude-plugin/plugin.json +6 -2
- package/.codex-plugin/plugin.json +9 -9
- package/README.md +190 -127
- package/assets/feedback-loop.png +0 -0
- package/assets/freeflow_workflow_map.png +0 -0
- package/assets/workflow.png +0 -0
- package/{skills → capabilities}/output-router/SKILL.md +21 -17
- package/{skills → capabilities}/output-router/references/safety-policy.md +1 -1
- package/capabilities/output-router/references/setup.md +110 -0
- package/command-surface.json +62 -7
- package/hooks/freeflow-runtime-context.mjs +572 -154
- package/hooks/hooks.json +26 -1
- package/hooks/tests/check-runtime-context-hook.sh +375 -0
- package/package.json +24 -24
- package/pi-extension/dist/index.js +264 -80
- package/pi-extension/dist/integrations/mcp-config.js +29 -0
- package/pi-extension/dist/output-router/host-producer-identification.js +232 -0
- package/pi-extension/dist/output-router/native-safety-net.js +216 -0
- package/pi-extension/dist/output-router/observed-tool-routing.js +122 -0
- package/pi-extension/dist/output-router/renderers.js +1023 -0
- package/pi-extension/dist/output-router/router-tools.js +554 -0
- package/pi-extension/dist/output-router/schemas.js +383 -0
- package/pi-extension/dist/output-router/status.js +744 -0
- package/pi-extension/dist/output-router/utils.js +502 -0
- package/pi-extension/dist/runtime/runtime-context.js +798 -0
- package/pi-extension/dist/settings/settings-tui.js +314 -0
- package/pi-extension/dist/settings/settings-ui.js +1487 -0
- package/pi-extension/freeflow/index.js +1 -0
- package/pi-extension/src/index.ts +230 -56
- package/pi-extension/src/{host-producer-identification.ts → output-router/host-producer-identification.ts} +37 -7
- package/pi-extension/src/{native-safety-net.ts → output-router/native-safety-net.ts} +8 -12
- package/pi-extension/src/{observed-tool-routing.ts → output-router/observed-tool-routing.ts} +19 -11
- package/pi-extension/src/{renderers.ts → output-router/renderers.ts} +246 -55
- package/pi-extension/src/{router-tools.ts → output-router/router-tools.ts} +130 -29
- package/pi-extension/src/{schemas.ts → output-router/schemas.ts} +171 -43
- package/pi-extension/src/output-router/status.ts +791 -0
- package/pi-extension/src/{utils.ts → output-router/utils.ts} +80 -18
- package/pi-extension/src/runtime/runtime-context.ts +895 -0
- package/pi-extension/src/settings/settings-tui.ts +408 -0
- package/pi-extension/src/settings/settings-ui.ts +1730 -0
- package/pi-extension/tests/integration/pi-extension.test.js +4020 -0
- package/pi-extension/tests/output-router/observed-producers.test.js +132 -0
- package/pi-extension/tests/output-router/search.test.js +722 -0
- package/pi-extension/tests/settings/settings-tui.test.js +243 -0
- package/router/dist/benchmarks/benchmark-harness.d.ts +16 -13
- package/router/dist/benchmarks/benchmark-harness.js +62 -62
- package/router/dist/benchmarks/benchmarks.d.ts +80 -73
- package/router/dist/benchmarks/benchmarks.js +860 -761
- package/router/dist/benchmarks/codex-qa-benchmarks.d.ts +71 -67
- package/router/dist/benchmarks/codex-qa-benchmarks.js +429 -390
- package/router/dist/benchmarks/command-benchmarks.d.ts +107 -92
- package/router/dist/benchmarks/command-benchmarks.js +616 -571
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.d.ts +64 -56
- package/router/dist/benchmarks/context-mode-normalized-benchmarks.js +679 -613
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.d.ts +103 -95
- package/router/dist/benchmarks/context-mode-real-deep-benchmark.js +1188 -1103
- package/router/dist/benchmarks/context-saving-baseline.d.ts +47 -41
- package/router/dist/benchmarks/context-saving-baseline.js +405 -374
- package/router/dist/benchmarks/index-benchmarks.d.ts +74 -69
- package/router/dist/benchmarks/index-benchmarks.js +761 -661
- package/router/dist/benchmarks/storage-policy-benchmarks.d.ts +80 -69
- package/router/dist/benchmarks/storage-policy-benchmarks.js +549 -469
- package/router/dist/benchmarks/vault-index-benchmarks.d.ts +43 -34
- package/router/dist/benchmarks/vault-index-benchmarks.js +294 -229
- package/router/dist/config/config.d.ts +62 -55
- package/router/dist/config/config.js +552 -373
- package/router/dist/config/router-contract.d.ts +2 -2
- package/router/dist/config/router-contract.js +38 -38
- package/router/dist/config/schema.d.ts +19 -10
- package/router/dist/config/schema.js +586 -538
- package/router/dist/config/types.d.ts +413 -361
- package/router/dist/config/types.js +19 -8
- package/router/dist/evidence/bounded-evidence.d.ts +26 -26
- package/router/dist/evidence/bounded-evidence.js +265 -252
- package/router/dist/evidence/evidence-range-selector.d.ts +16 -14
- package/router/dist/evidence/evidence-range-selector.js +192 -191
- package/router/dist/evidence/evidence-search.d.ts +18 -16
- package/router/dist/evidence/evidence-search.js +401 -377
- package/router/dist/evidence/evidence.d.ts +17 -17
- package/router/dist/evidence/evidence.js +163 -159
- package/router/dist/evidence/failure-contracts.d.ts +25 -15
- package/router/dist/evidence/failure-contracts.js +132 -126
- package/router/dist/evidence/line-ranges.d.ts +17 -15
- package/router/dist/evidence/line-ranges.js +18 -18
- package/router/dist/experiments/local-index.d.ts +47 -41
- package/router/dist/experiments/local-index.js +477 -458
- package/router/dist/local/local-traversal.d.ts +18 -0
- package/router/dist/local/local-traversal.js +334 -0
- package/router/dist/processing/engine.d.ts +94 -70
- package/router/dist/processing/engine.js +462 -364
- package/router/dist/processing/reducers.d.ts +208 -194
- package/router/dist/processing/reducers.js +1434 -1256
- package/router/dist/processing/renderers.d.ts +24 -18
- package/router/dist/processing/renderers.js +161 -152
- package/router/dist/processing/scripts.d.ts +80 -67
- package/router/dist/processing/scripts.js +343 -290
- package/router/dist/repo/repo-traversal.d.ts +9 -9
- package/router/dist/repo/repo-traversal.js +205 -202
- package/router/dist/routing/observed-reducers.d.ts +25 -19
- package/router/dist/routing/observed-reducers.js +559 -545
- package/router/dist/routing/observed-routing.d.ts +23 -14
- package/router/dist/routing/observed-routing.js +319 -318
- package/router/dist/routing/parsers.d.ts +10 -10
- package/router/dist/routing/parsers.js +454 -406
- package/router/dist/routing/run-filters.d.ts +45 -41
- package/router/dist/routing/run-filters.js +215 -207
- package/router/dist/routing/run-reducers.d.ts +45 -26
- package/router/dist/routing/run-reducers.js +102 -90
- package/router/dist/sandbox/adapter-roots.d.ts +34 -26
- package/router/dist/sandbox/adapter-roots.js +43 -41
- package/router/dist/sandbox/eryx-python-adapter.d.ts +9 -5
- package/router/dist/sandbox/eryx-python-adapter.js +659 -588
- package/router/dist/sandbox/jq-wasm-adapter.d.ts +3 -3
- package/router/dist/sandbox/jq-wasm-adapter.js +319 -311
- package/router/dist/sandbox/quickjs-wasi-adapter.d.ts +9 -5
- package/router/dist/sandbox/quickjs-wasi-adapter.js +336 -333
- package/router/dist/sandbox/script-sandbox.d.ts +105 -84
- package/router/dist/sandbox/script-sandbox.js +294 -260
- package/router/dist/setup/script-transform-adapters.d.ts +23 -19
- package/router/dist/setup/script-transform-adapters.js +223 -204
- package/router/dist/tools/batch.d.ts +27 -16
- package/router/dist/tools/batch.js +493 -416
- package/router/dist/tools/run.d.ts +47 -35
- package/router/dist/tools/run.js +1172 -1087
- package/router/dist/tools/search.d.ts +53 -31
- package/router/dist/tools/search.js +1224 -1065
- package/router/dist/transform/engine.d.ts +121 -96
- package/router/dist/transform/engine.js +1853 -1736
- package/router/dist/vault/vault-index.d.ts +97 -78
- package/router/dist/vault/vault-index.js +347 -342
- package/router/dist/vault/vault.d.ts +109 -74
- package/router/dist/vault/vault.js +528 -533
- package/runtime/interaction-contract.md +16 -0
- package/skills/bypass/SKILL.md +52 -38
- package/skills/commit-work/SKILL.md +54 -46
- package/skills/commit-work/references/staging-decisions.md +46 -24
- package/skills/decision-gate/SKILL.md +48 -0
- package/skills/design-for-depth/SKILL.md +75 -59
- package/skills/design-for-depth/references/design-pressure-signals.md +56 -7
- package/skills/design-for-depth/references/interface-design-loop.md +156 -0
- package/skills/design-for-depth/references/software-design-philosophy.md +32 -3
- package/skills/diagnose-failure/SKILL.md +65 -60
- package/skills/diagnose-failure/references/diagnostic-loop-catalog.md +36 -0
- package/skills/diagnose-failure/references/flaky-and-performance.md +30 -18
- package/skills/discuss/SKILL.md +121 -0
- package/skills/discuss/references/checkpoints.md +66 -0
- package/skills/evaluate-skill/SKILL.md +62 -53
- package/skills/evaluate-skill/references/definition-schema.md +345 -0
- package/skills/evaluate-skill/references/evaluation-design.md +86 -0
- package/skills/evaluate-skill/references/execution-and-evidence.md +85 -0
- package/skills/evaluate-skill/references/review-and-revision.md +51 -0
- package/skills/evaluate-skill/scripts/lib/definitions.mjs +296 -0
- package/skills/evaluate-skill/scripts/lib/evidence.mjs +46 -0
- package/skills/evaluate-skill/scripts/lib/grade-comparison.mjs +60 -0
- package/skills/evaluate-skill/scripts/lib/grade-interaction.mjs +148 -0
- package/skills/evaluate-skill/scripts/lib/grade-support.mjs +126 -0
- package/skills/evaluate-skill/scripts/lib/grade-workspace.mjs +192 -0
- package/skills/evaluate-skill/scripts/lib/grade.mjs +56 -0
- package/skills/evaluate-skill/scripts/lib/pi.mjs +489 -0
- package/skills/evaluate-skill/scripts/lib/process.mjs +79 -0
- package/skills/evaluate-skill/scripts/lib/rpc.mjs +444 -0
- package/skills/evaluate-skill/scripts/lib/runner.mjs +346 -0
- package/skills/evaluate-skill/scripts/lib/sandbox.mjs +409 -0
- package/skills/evaluate-skill/scripts/lib/view.mjs +337 -0
- package/skills/evaluate-skill/scripts/pi-guard.mjs +164 -0
- package/skills/evaluate-skill/scripts/skill-eval.mjs +101 -0
- package/skills/execute-work/SKILL.md +99 -0
- package/skills/execute-work/references/code-practices.md +78 -0
- package/skills/execute-work/references/execution-loop.md +77 -0
- package/skills/finish-branch/SKILL.md +117 -0
- package/skills/finish-branch/references/integration-options.md +80 -0
- package/skills/handoff/SKILL.md +65 -39
- package/skills/handoff/references/templates.md +104 -27
- package/skills/launch-work/SKILL.md +116 -0
- package/skills/launch-work/references/launch-readiness.md +85 -0
- package/skills/migration-work/SKILL.md +115 -0
- package/skills/migration-work/references/migration-lifecycle.md +82 -0
- package/skills/mode-contract/SKILL.md +72 -52
- package/skills/release-work/SKILL.md +118 -0
- package/skills/release-work/references/release-evidence.md +78 -0
- package/skills/review-artifact/SKILL.md +77 -69
- package/skills/review-artifact/references/reviewer-prompt.md +133 -42
- package/skills/review-work/SKILL.md +82 -57
- package/skills/review-work/references/reviewer-prompt.md +97 -60
- package/skills/review-work/references/security-risk-lens.md +67 -0
- package/skills/setup-freeflow/SKILL.md +54 -68
- package/skills/setup-freeflow/references/activation-contract.md +84 -69
- package/skills/setup-freeflow/references/host-setup.md +50 -17
- package/skills/simplify-code/SKILL.md +80 -0
- package/skills/simplify-code/references/simplification-patterns.md +70 -0
- package/skills/tdd/SKILL.md +96 -0
- package/skills/tdd/references/test-design.md +96 -0
- package/skills/track-work/SKILL.md +84 -0
- package/skills/track-work/references/working-record-schema.md +378 -0
- package/skills/verify-work/SKILL.md +59 -38
- package/skills/verify-work/references/browser-runtime-evidence.md +55 -0
- package/skills/verify-work/references/integration-evidence.md +51 -0
- package/skills/verify-work/references/performance-evidence.md +57 -0
- package/skills/workflow/SKILL.md +53 -59
- package/skills/workflow/references/domain-skill-composition.md +57 -0
- package/skills/workflow/references/workflow-loop.md +92 -0
- package/skills/write-plan/SKILL.md +85 -58
- package/skills/write-plan/references/plan-shapes.md +132 -46
- package/skills/write-skill/SKILL.md +82 -51
- package/skills/write-skill/references/activation-boundaries.md +62 -0
- package/skills/write-skill/references/agent-first-instructions.md +87 -0
- package/skills/write-skill/references/progressive-disclosure.md +63 -0
- package/skills/write-skill/scripts/lib/skill-author.mjs +653 -0
- package/skills/write-skill/scripts/skill-author.mjs +104 -0
- package/skills/write-spec/SKILL.md +65 -54
- package/skills/write-spec/references/artifact-standards.md +60 -41
- package/skills/write-spec/references/decision-records.md +77 -0
- package/skills/write-spec/references/spec-shapes.md +119 -60
- package/evals/README.md +0 -32
- package/evals/fixtures/copy-conventions.md +0 -31
- package/evals/fixtures/output-router/README.md +0 -12
- package/evals/fixtures/output-router/failed-command-output.txt +0 -9
- package/evals/fixtures/output-router/large-router-manual.md +0 -63
- package/evals/fixtures/output-router/native-large-output.txt +0 -40
- package/evals/fixtures/output-router/noisy-test-output.txt +0 -30
- package/evals/fixtures/output-router/verification-output.txt +0 -8
- package/evals/fixtures/tiny-artifact-review-app/README.md +0 -5
- package/evals/fixtures/tiny-artifact-review-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md +0 -21
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/failed-payment-grace-period.md +0 -24
- package/evals/fixtures/tiny-artifact-review-app/docs/specs/immediate-downgrade.md +0 -17
- package/evals/fixtures/tiny-artifact-review-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-commit-ambiguous-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-ambiguous-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-ambiguous-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-ambiguous-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-ambiguous-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-app/.freeflow-eval-setup.sh +0 -14
- package/evals/fixtures/tiny-commit-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-everything-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-everything-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-everything-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-everything-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-everything-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-everything-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-staged-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-staged-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-staged-app/docs/specs/settings-title.md +0 -5
- package/evals/fixtures/tiny-commit-staged-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-commit-staged-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-commit-staged-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-commit-staged-mixed-app/.freeflow-eval-setup.sh +0 -15
- package/evals/fixtures/tiny-commit-staged-mixed-app/README.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/billing-policy.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/docs/release-notes.md +0 -3
- package/evals/fixtures/tiny-commit-staged-mixed-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-dashboard-stale-app/README.md +0 -11
- package/evals/fixtures/tiny-dashboard-stale-app/src/api-client.ts +0 -13
- package/evals/fixtures/tiny-dashboard-stale-app/src/dashboard-data.ts +0 -19
- package/evals/fixtures/tiny-dashboard-stale-app/tests/dashboard-data.test.md +0 -9
- package/evals/fixtures/tiny-decision-destination-app/README.md +0 -6
- package/evals/fixtures/tiny-decision-destination-app/src/risk.ts +0 -8
- package/evals/fixtures/tiny-decision-destination-app/tests/risk.test.md +0 -4
- package/evals/fixtures/tiny-design-depth-app/README.md +0 -15
- package/evals/fixtures/tiny-design-depth-app/src/routes.ts +0 -25
- package/evals/fixtures/tiny-design-depth-app/src/sendEmail.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/src/sendSms.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/src/telemetry.ts +0 -3
- package/evals/fixtures/tiny-design-depth-app/tests/routes.test.md +0 -12
- package/evals/fixtures/tiny-eval-method-app/README.md +0 -8
- package/evals/fixtures/tiny-eval-method-app/evals/reports/review-pr-eval-result.md +0 -5
- package/evals/fixtures/tiny-eval-method-app/evals/reports/triage-question-failure.md +0 -17
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.diff +0 -11
- package/evals/fixtures/tiny-eval-method-app/evals/runs/review-pr-with-skill-output.md +0 -3
- package/evals/fixtures/tiny-eval-method-app/skills/review-pr/SKILL.md +0 -10
- package/evals/fixtures/tiny-eval-method-app/skills/triage-question/SKILL.md +0 -10
- package/evals/fixtures/tiny-execute-plan-app/README.md +0 -5
- package/evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh +0 -5
- package/evals/fixtures/tiny-execute-plan-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/immediate-downgrade-plan.md +0 -11
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md +0 -15
- package/evals/fixtures/tiny-execute-plan-app/docs/plans/unverified-billing-plan.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/failed-payment-grace-period.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-execute-plan-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-execute-plan-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-execute-plan-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-execute-plan-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-execute-review-loop-app/README.md +0 -5
- package/evals/fixtures/tiny-execute-review-loop-app/docs/billing-policy.md +0 -9
- package/evals/fixtures/tiny-execute-review-loop-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/previous-agent-summary.md +0 -15
- package/evals/fixtures/tiny-execute-review-loop-app/docs/review/review-pass-1-non-pass.md +0 -11
- package/evals/fixtures/tiny-execute-review-loop-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-execute-review-loop-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-execute-review-loop-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-execute-review-loop-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-execute-review-loop-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-flaky-session-app/README.md +0 -10
- package/evals/fixtures/tiny-flaky-session-app/src/session-timeout.ts +0 -13
- package/evals/fixtures/tiny-flaky-session-app/tests/session-timeout.test.md +0 -8
- package/evals/fixtures/tiny-handoff-app/README.md +0 -5
- package/evals/fixtures/tiny-handoff-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-handoff-app/docs/handoffs/old-agent-handoff.md +0 -8
- package/evals/fixtures/tiny-handoff-app/docs/plans/billing-retry-plan.md +0 -11
- package/evals/fixtures/tiny-handoff-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-handoff-report-app/README.md +0 -3
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.diff +0 -1
- package/evals/fixtures/tiny-handoff-report-app/evals/runs/handoff-3/hof-003/with-skill-output.md +0 -1
- package/evals/fixtures/tiny-handoff-report-app/evals/scores/hof-003-score.txt +0 -4
- package/evals/fixtures/tiny-plan-app/README.md +0 -5
- package/evals/fixtures/tiny-plan-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-plan-app/docs/specs/failed-payment-grace-period.md +0 -25
- package/evals/fixtures/tiny-plan-app/src/billing.ts +0 -17
- package/evals/fixtures/tiny-plan-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-plan-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-plan-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/README.md +0 -3
- package/evals/fixtures/tiny-plan-strict-api-app/docs/billing-policy.md +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/docs/specs/billing-webhook-api.md +0 -25
- package/evals/fixtures/tiny-plan-strict-api-app/src/billing.ts +0 -5
- package/evals/fixtures/tiny-plan-strict-api-app/src/webhooks.ts +0 -3
- package/evals/fixtures/tiny-plan-strict-api-app/tests/billing-webhook.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/.freeflow/config.json +0 -3
- package/evals/fixtures/tiny-post-setup-invalid-config-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-invalid-config-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-invalid-config-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-invalid-config-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-post-setup-missing-config-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-missing-config-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-missing-config-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-missing-config-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-post-setup-source-conflict-app/.freeflow/config.json +0 -3
- package/evals/fixtures/tiny-post-setup-source-conflict-app/AGENTS.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/README.md +0 -16
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-post-setup-source-conflict-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-post-setup-source-conflict-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-post-setup-source-conflict-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-report-performance-app/README.md +0 -11
- package/evals/fixtures/tiny-report-performance-app/docs/performance.md +0 -10
- package/evals/fixtures/tiny-report-performance-app/src/report-summary.ts +0 -29
- package/evals/fixtures/tiny-report-performance-app/tests/report-performance.test.md +0 -8
- package/evals/fixtures/tiny-review-request-app/README.md +0 -3
- package/evals/fixtures/tiny-review-request-app/docs/billing-policy.md +0 -5
- package/evals/fixtures/tiny-review-request-app/docs/plans/billing-webhook-api.md +0 -19
- package/evals/fixtures/tiny-review-request-app/docs/review/previous-agent-summary.md +0 -20
- package/evals/fixtures/tiny-review-request-app/docs/specs/billing-webhook-api.md +0 -25
- package/evals/fixtures/tiny-review-request-app/src/billing.ts +0 -11
- package/evals/fixtures/tiny-review-request-app/src/webhooks.ts +0 -19
- package/evals/fixtures/tiny-review-request-app/tests/billing-webhook.test.md +0 -8
- package/evals/fixtures/tiny-review-work-app/README.md +0 -5
- package/evals/fixtures/tiny-review-work-app/docs/billing-policy.md +0 -9
- package/evals/fixtures/tiny-review-work-app/docs/plans/settings-title-plan.md +0 -12
- package/evals/fixtures/tiny-review-work-app/docs/review/previous-agent-summary.md +0 -15
- package/evals/fixtures/tiny-review-work-app/docs/specs/settings-title.md +0 -9
- package/evals/fixtures/tiny-review-work-app/src/billing.ts +0 -7
- package/evals/fixtures/tiny-review-work-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-review-work-app/tests/billing.test.md +0 -7
- package/evals/fixtures/tiny-review-work-app/tests/settings.test.md +0 -5
- package/evals/fixtures/tiny-saas-app/README.md +0 -16
- package/evals/fixtures/tiny-saas-app/docs/auth-review.md +0 -15
- package/evals/fixtures/tiny-saas-app/docs/billing-policy.md +0 -10
- package/evals/fixtures/tiny-saas-app/docs/copy-conventions.md +0 -15
- package/evals/fixtures/tiny-saas-app/docs/specs/settings-title.md +0 -13
- package/evals/fixtures/tiny-saas-app/src/auth-middleware.ts +0 -12
- package/evals/fixtures/tiny-saas-app/src/billing.ts +0 -20
- package/evals/fixtures/tiny-saas-app/src/login-copy.ts +0 -1
- package/evals/fixtures/tiny-saas-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-saas-app/tests/auth-middleware.test.md +0 -8
- package/evals/fixtures/tiny-saas-app/tests/billing.test.md +0 -8
- package/evals/fixtures/tiny-saas-app/tests/copy.test.md +0 -7
- package/evals/fixtures/tiny-setup-agents-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-both-app/AGENTS.md +0 -5
- package/evals/fixtures/tiny-setup-both-app/CLAUDE.md +0 -5
- package/evals/fixtures/tiny-setup-claude-app/CLAUDE.md +0 -6
- package/evals/fixtures/tiny-setup-codex-rules-app/.codex/rules/shell.rules +0 -5
- package/evals/fixtures/tiny-setup-codex-rules-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-conflict-app/AGENTS.md +0 -6
- package/evals/fixtures/tiny-setup-existing-block-app/AGENTS.md +0 -14
- package/evals/fixtures/tiny-setup-four-rules-app/CLAUDE.md +0 -6
- package/evals/fixtures/tiny-skill-dev-app/README.md +0 -10
- package/evals/fixtures/tiny-skill-dev-app/evals/prompts/review-pr-001.txt +0 -8
- package/evals/fixtures/tiny-skill-dev-app/evals/reports/review-pr-failure.md +0 -17
- package/evals/fixtures/tiny-skill-dev-app/skills/review-pr/SKILL.md +0 -13
- package/evals/fixtures/tiny-team-app/README.md +0 -5
- package/evals/fixtures/tiny-team-app/docs/product.md +0 -6
- package/evals/fixtures/tiny-team-app/src/teams.ts +0 -9
- package/evals/fixtures/tiny-team-app/src/users.ts +0 -5
- package/evals/fixtures/tiny-team-app/tests/teams.test.md +0 -7
- package/evals/fixtures/tiny-verify-app/README.md +0 -5
- package/evals/fixtures/tiny-verify-app/src/settings-nav.ts +0 -1
- package/evals/fixtures/tiny-verify-app/src/settings-page.ts +0 -1
- package/evals/fixtures/tiny-verify-app/tests/settings.test.md +0 -7
- package/evals/prompts/aon-001.txt +0 -2
- package/evals/prompts/byp-001.txt +0 -2
- package/evals/prompts/byp-002.txt +0 -2
- package/evals/prompts/cap-001.txt +0 -3
- package/evals/prompts/cap-002.txt +0 -3
- package/evals/prompts/cmd-001.txt +0 -3
- package/evals/prompts/cmd-002.txt +0 -3
- package/evals/prompts/cmd-003.txt +0 -6
- package/evals/prompts/cmd-004.txt +0 -5
- package/evals/prompts/cmd-005.txt +0 -5
- package/evals/prompts/cmd-006.txt +0 -5
- package/evals/prompts/cmd-007.txt +0 -5
- package/evals/prompts/cmd-008.txt +0 -5
- package/evals/prompts/cmd-009.txt +0 -8
- package/evals/prompts/cmd-010.txt +0 -5
- package/evals/prompts/cmd-011.txt +0 -5
- package/evals/prompts/cmd-012.txt +0 -5
- package/evals/prompts/cmd-013.txt +0 -5
- package/evals/prompts/cmd-014.txt +0 -7
- package/evals/prompts/cmd-015.txt +0 -7
- package/evals/prompts/cmd-016.txt +0 -5
- package/evals/prompts/cmt-001.txt +0 -1
- package/evals/prompts/cmt-002.txt +0 -1
- package/evals/prompts/cmt-003.txt +0 -1
- package/evals/prompts/cmt-004.txt +0 -1
- package/evals/prompts/cmt-005.txt +0 -1
- package/evals/prompts/dfd-001.txt +0 -3
- package/evals/prompts/dia-001.txt +0 -3
- package/evals/prompts/dia-002.txt +0 -3
- package/evals/prompts/dia-003.txt +0 -3
- package/evals/prompts/dis-001.txt +0 -5
- package/evals/prompts/dis-002.txt +0 -5
- package/evals/prompts/esk-001.txt +0 -5
- package/evals/prompts/esk-002.txt +0 -5
- package/evals/prompts/esk-003.txt +0 -5
- package/evals/prompts/grc-001.txt +0 -3
- package/evals/prompts/hof-001.txt +0 -3
- package/evals/prompts/hof-002.txt +0 -1
- package/evals/prompts/hof-003.txt +0 -1
- package/evals/prompts/hof-004.txt +0 -1
- package/evals/prompts/hof-005.txt +0 -1
- package/evals/prompts/ivg-001.txt +0 -1
- package/evals/prompts/ivg-002.txt +0 -11
- package/evals/prompts/ivg-003.txt +0 -1
- package/evals/prompts/ivg-004.txt +0 -1
- package/evals/prompts/mode-001.txt +0 -3
- package/evals/prompts/mode-002.txt +0 -1
- package/evals/prompts/mode-003.txt +0 -3
- package/evals/prompts/mode-004.txt +0 -3
- package/evals/prompts/mode-005.txt +0 -3
- package/evals/prompts/mode-006.txt +0 -3
- package/evals/prompts/mode-007.txt +0 -3
- package/evals/prompts/otr-001.txt +0 -3
- package/evals/prompts/otr-002.txt +0 -3
- package/evals/prompts/rar-001.txt +0 -5
- package/evals/prompts/rar-002.txt +0 -3
- package/evals/prompts/rar-003.txt +0 -6
- package/evals/prompts/rar-004.txt +0 -15
- package/evals/prompts/rar-005.txt +0 -10
- package/evals/prompts/rbr-001.txt +0 -3
- package/evals/prompts/rbr-002.txt +0 -3
- package/evals/prompts/rbr-003.txt +0 -7
- package/evals/prompts/rev-001.txt +0 -3
- package/evals/prompts/rev-002.txt +0 -5
- package/evals/prompts/rev-003.txt +0 -6
- package/evals/prompts/rev-004.txt +0 -1
- package/evals/prompts/rev-005.txt +0 -15
- package/evals/prompts/rev-006.txt +0 -5
- package/evals/prompts/stp-001.txt +0 -1
- package/evals/prompts/stp-002.txt +0 -1
- package/evals/prompts/stp-003.txt +0 -1
- package/evals/prompts/stp-004.txt +0 -1
- package/evals/prompts/stp-005.txt +0 -1
- package/evals/prompts/stp-006.txt +0 -1
- package/evals/prompts/stp-007.txt +0 -1
- package/evals/prompts/stp-008.txt +0 -1
- package/evals/prompts/stp-009.txt +0 -1
- package/evals/prompts/stp-010.txt +0 -1
- package/evals/prompts/stp-011.txt +0 -1
- package/evals/prompts/vfy-003.txt +0 -1
- package/evals/prompts/wpl-001.txt +0 -3
- package/evals/prompts/wpl-002.txt +0 -5
- package/evals/prompts/wpl-003.txt +0 -3
- package/evals/prompts/wpl-004.txt +0 -5
- package/evals/prompts/wpl-005.txt +0 -1
- package/evals/prompts/wrc-001.txt +0 -10
- package/evals/prompts/wrc-002.txt +0 -3
- package/evals/prompts/wsk-001.txt +0 -6
- package/evals/prompts/wsp-001.txt +0 -5
- package/evals/prompts/wsp-002.txt +0 -15
- package/evals/prompts/wsp-003.txt +0 -3
- package/evals/prompts/wsp-004.txt +0 -15
- package/evals/prompts/wsp-005.txt +0 -5
- package/evals/prompts/wsp-006.txt +0 -12
- package/evals/prompts/wsp-007.txt +0 -9
- package/evals/prompts/wsp-008.txt +0 -13
- package/evals/prompts/xpl-001.txt +0 -1
- package/evals/prompts/xpl-002.txt +0 -3
- package/evals/prompts/xpl-003.txt +0 -3
- package/evals/prompts/xpl-004.txt +0 -3
- package/evals/prompts/xpl-005.txt +0 -5
- package/evals/prompts/xpl-006.txt +0 -3
- package/evals/registries/adversarial-evals.json +0 -39
- package/evals/registries/fixture-evals.json +0 -2513
- package/evals/registries/handoff-evals.json +0 -74
- package/evals/registries/legacy-evals.json +0 -147
- package/evals/registries/skill-evidence.json +0 -193
- package/evals/registries/verify-work-evals.json +0 -46
- package/evals/reports/acceptance/v0.1-acceptance-report.md +0 -130
- package/evals/reports/by-command-surface/command-surface-1-report.md +0 -45
- package/evals/reports/by-command-surface/command-surface-10-report.md +0 -76
- package/evals/reports/by-command-surface/command-surface-11-report.md +0 -77
- package/evals/reports/by-command-surface/command-surface-12-report.md +0 -91
- package/evals/reports/by-command-surface/command-surface-13-report.md +0 -64
- package/evals/reports/by-command-surface/command-surface-14-report.md +0 -66
- package/evals/reports/by-command-surface/command-surface-15-report.md +0 -124
- package/evals/reports/by-command-surface/command-surface-16-report.md +0 -39
- package/evals/reports/by-command-surface/command-surface-2-report.md +0 -80
- package/evals/reports/by-command-surface/command-surface-3-report.md +0 -86
- package/evals/reports/by-command-surface/command-surface-4-report.md +0 -76
- package/evals/reports/by-command-surface/command-surface-5-report.md +0 -80
- package/evals/reports/by-command-surface/command-surface-6-report.md +0 -74
- package/evals/reports/by-command-surface/command-surface-7-report.md +0 -84
- package/evals/reports/by-command-surface/command-surface-8-report.md +0 -79
- package/evals/reports/by-command-surface/command-surface-9-report.md +0 -79
- package/evals/reports/by-command-surface/command-surface-matrix.md +0 -91
- package/evals/reports/by-skill/bypass-1-report.md +0 -124
- package/evals/reports/by-skill/capture-decisions-1-report.md +0 -99
- package/evals/reports/by-skill/capture-decisions-2-report.md +0 -102
- package/evals/reports/by-skill/commit-work-1-report.md +0 -165
- package/evals/reports/by-skill/commit-work-2-report.md +0 -109
- package/evals/reports/by-skill/design-for-depth-1-report.md +0 -95
- package/evals/reports/by-skill/diagnose-failure-1-report.md +0 -91
- package/evals/reports/by-skill/diagnose-failure-2-report.md +0 -94
- package/evals/reports/by-skill/discover-1-report.md +0 -118
- package/evals/reports/by-skill/evaluate-skill-2-report.md +0 -71
- package/evals/reports/by-skill/execute-plan-1-report.md +0 -103
- package/evals/reports/by-skill/execute-plan-2-report.md +0 -112
- package/evals/reports/by-skill/execute-plan-3-report.md +0 -87
- package/evals/reports/by-skill/grill-context-1-report.md +0 -103
- package/evals/reports/by-skill/handoff-1-report.md +0 -123
- package/evals/reports/by-skill/handoff-2-report.md +0 -105
- package/evals/reports/by-skill/handoff-3-report.md +0 -100
- package/evals/reports/by-skill/handoff-7-report.md +0 -77
- package/evals/reports/by-skill/interview-gate-1-report.md +0 -63
- package/evals/reports/by-skill/interview-gate-2-report.md +0 -105
- package/evals/reports/by-skill/mode-contract-1-report.md +0 -74
- package/evals/reports/by-skill/mode-contract-2-report.md +0 -87
- package/evals/reports/by-skill/mode-contract-3-report.md +0 -65
- package/evals/reports/by-skill/mode-contract-4-report.md +0 -71
- package/evals/reports/by-skill/mode-contract-5-report.md +0 -70
- package/evals/reports/by-skill/output-router-1-report.md +0 -76
- package/evals/reports/by-skill/output-router-2-report.md +0 -65
- package/evals/reports/by-skill/research-1-report.md +0 -144
- package/evals/reports/by-skill/research-brief-1-report.md +0 -94
- package/evals/reports/by-skill/research-brief-2-report.md +0 -75
- package/evals/reports/by-skill/review-artifact-1-report.md +0 -91
- package/evals/reports/by-skill/review-artifact-2-report.md +0 -79
- package/evals/reports/by-skill/review-artifact-3-report.md +0 -120
- package/evals/reports/by-skill/review-artifact-4-report.md +0 -89
- package/evals/reports/by-skill/review-artifact-5-report.md +0 -66
- package/evals/reports/by-skill/review-work-1-report.md +0 -85
- package/evals/reports/by-skill/review-work-2-report.md +0 -87
- package/evals/reports/by-skill/review-work-3-report.md +0 -104
- package/evals/reports/by-skill/review-work-4-report.md +0 -108
- package/evals/reports/by-skill/review-work-5-report.md +0 -73
- package/evals/reports/by-skill/review-work-6-report.md +0 -66
- package/evals/reports/by-skill/setup-freeflow-1-report.md +0 -199
- package/evals/reports/by-skill/setup-freeflow-2-report.md +0 -105
- package/evals/reports/by-skill/setup-freeflow-3-report.md +0 -105
- package/evals/reports/by-skill/setup-freeflow-4-report.md +0 -88
- package/evals/reports/by-skill/setup-freeflow-5-report.md +0 -115
- package/evals/reports/by-skill/skill-authoring-1-report.md +0 -81
- package/evals/reports/by-skill/verify-work-1-report.md +0 -111
- package/evals/reports/by-skill/verify-work-2-report.md +0 -120
- package/evals/reports/by-skill/workflow-1-report.md +0 -65
- package/evals/reports/by-skill/write-plan-1-report.md +0 -123
- package/evals/reports/by-skill/write-plan-2-report.md +0 -98
- package/evals/reports/by-skill/write-plan-3-report.md +0 -110
- package/evals/reports/by-skill/write-spec-1-report.md +0 -88
- package/evals/reports/by-skill/write-spec-2-report.md +0 -89
- package/evals/reports/by-skill/write-spec-3-report.md +0 -90
- package/evals/reports/by-skill/write-spec-4-report.md +0 -167
- package/evals/reports/harness/claude-harness-1-report.md +0 -103
- package/evals/reports/harness/eval-harness-1-report.md +0 -91
- package/evals/reports/iterations/iteration-1-report.md +0 -71
- package/evals/reports/iterations/iteration-2-report.md +0 -78
- package/evals/reports/iterations/iteration-3-report.md +0 -118
- package/evals/reports/iterations/iteration-4-report.md +0 -131
- package/evals/reports/iterations/iteration-5-report.md +0 -149
- package/evals/reports/iterations/iteration-6-report.md +0 -136
- package/evals/reports/runtime/always-on-runtime-1-report.md +0 -107
- package/evals/reports/runtime/context-mode-normalized-benchmark-1-report.md +0 -52
- package/evals/reports/runtime/context-mode-real-deep-baseline-1-report.md +0 -410
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md +0 -203
- package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md +0 -51
- package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md +0 -99
- package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md +0 -166
- package/evals/reports/runtime/eryx-python-proof-spike-1-report.md +0 -96
- package/evals/reports/runtime/eryx-python-proof-spike-2-report.md +0 -151
- package/evals/reports/runtime/eryx-python-script-derive-execution-1-report.md +0 -233
- package/evals/reports/runtime/jq-script-derive-execution-1-report.md +0 -215
- package/evals/reports/runtime/jq-wasm-proof-spike-1-report.md +0 -67
- package/evals/reports/runtime/output-router-benchmark-1-report.md +0 -81
- package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md +0 -35
- package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md +0 -150
- package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md +0 -46
- package/evals/reports/runtime/output-router-command-benchmark-1-report.md +0 -62
- package/evals/reports/runtime/output-router-index-benchmark-1-report.md +0 -64
- package/evals/reports/runtime/output-router-regression-1-report.md +0 -81
- package/evals/reports/runtime/output-router-transform-eval-1-report.md +0 -28
- package/evals/reports/runtime/pi-observed-routing-eval-1-report.md +0 -44
- package/evals/reports/runtime/quickjs-script-derive-execution-1-report.md +0 -159
- package/evals/reports/runtime/quickjs-wasi-proof-spike-1-report.md +0 -55
- package/evals/reports/runtime/render-format-row-protocol-1-report.md +0 -58
- package/evals/reports/runtime/script-sandbox-adapter-selection-review-1-report.md +0 -84
- package/evals/reports/runtime/script-sandbox-probe-resource-hardening-1-report.md +0 -51
- package/evals/reports/runtime/storage-policy-benchmark-1-report.md +0 -176
- package/evals/reports/runtime/vault-index-storage-spike-1-report.md +0 -39
- package/evals/reports/runtime/workflow-context-hook-1-report.md +0 -82
- package/evals/runbooks/adversarial-eval-runbook.md +0 -28
- package/evals/runbooks/fixture-eval-runbook.md +0 -137
- package/evals/runbooks/handoff-eval-runbook.md +0 -22
- package/evals/runbooks/verify-work-eval-runbook.md +0 -27
- package/evals/scripts/audit-command-surface.sh +0 -160
- package/evals/scripts/check-activation-contract.sh +0 -121
- package/evals/scripts/check-runtime-context-hook.sh +0 -183
- package/evals/scripts/grade-fixture-eval.sh +0 -446
- package/evals/scripts/run-claude-fixture-eval.sh +0 -26
- package/evals/scripts/run-codex-fixture-eval.sh +0 -26
- package/evals/scripts/run-eryx-python-proof-spike.js +0 -559
- package/evals/scripts/run-fixture-eval-by-id.sh +0 -23
- package/evals/scripts/run-fixture-eval.sh +0 -464
- package/evals/scripts/run-jq-wasm-proof-spike.js +0 -299
- package/evals/scripts/run-output-router-transform-eval.js +0 -115
- package/evals/scripts/run-pi-observed-routing-eval.js +0 -478
- package/evals/scripts/run-quickjs-wasi-proof-spike.js +0 -339
- package/evals/scripts/skill-evidence.sh +0 -316
- package/evals/scripts/validate-release-metadata.sh +0 -486
- package/evals/suites/v0.1-acceptance-suite.md +0 -131
- package/pi-extension/dist/host-producer-identification.js +0 -186
- package/pi-extension/dist/mcp-config.js +0 -30
- package/pi-extension/dist/native-safety-net.js +0 -209
- package/pi-extension/dist/observed-tool-routing.js +0 -114
- package/pi-extension/dist/renderers.js +0 -835
- package/pi-extension/dist/router-tools.js +0 -438
- package/pi-extension/dist/runtime-context.js +0 -288
- package/pi-extension/dist/schemas.js +0 -254
- package/pi-extension/dist/status.js +0 -449
- package/pi-extension/dist/utils.js +0 -467
- package/pi-extension/src/runtime-context.ts +0 -347
- package/pi-extension/src/status.ts +0 -505
- package/plugin-docs/README.md +0 -40
- package/plugin-docs/adr/0001-three-modes.md +0 -27
- package/plugin-docs/adr/0002-evals-before-hooks.md +0 -17
- package/plugin-docs/adr/0003-release-boundary.md +0 -21
- package/plugin-docs/adr/0004-discover-replaces-shallow-discovery-skills.md +0 -26
- package/plugin-docs/adr/README.md +0 -10
- package/plugin-docs/architecture.md +0 -89
- package/plugin-docs/output-router.md +0 -396
- package/plugin-docs/release-evidence.md +0 -77
- package/plugin-docs/skills.md +0 -44
- package/plugin-docs/workflow.md +0 -57
- package/skills/diagnose-failure/references/feedback-loop-catalog.md +0 -25
- package/skills/discover/ARTIFACT-DESTINATIONS.md +0 -58
- package/skills/discover/CHECKPOINTS.md +0 -74
- package/skills/discover/SKILL.md +0 -173
- package/skills/evaluate-skill/references/eval-patterns.md +0 -41
- package/skills/evaluate-skill/references/grading-priority.md +0 -33
- package/skills/execute-plan/SKILL.md +0 -133
- package/skills/execute-plan/references/execution-map.md +0 -122
- package/skills/interview-gate/SKILL.md +0 -90
- package/skills/setup-freeflow/references/output-router-setup.md +0 -102
- package/skills/workflow/references/workflow-map.md +0 -104
- /package/pi-extension/src/{mcp-config.ts → integrations/mcp-config.ts} +0 -0
|
@@ -1,96 +1,105 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: evaluate-skill
|
|
3
|
-
description: Use when
|
|
3
|
+
description: Use when evaluating or comparing agent-skill behavior.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Evaluate Skill
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Evaluate whether an exact skill changes agent behavior as intended under declared pressure. Evidence outranks confident prose.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
A **group** asks one behavioral question through exactly two variants:
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
If the task is only to grade saved artifacts, grade before proposing any wording change.
|
|
12
|
+
- `baseline`: no target skill or a previous snapshot;
|
|
13
|
+
- `candidate`: a new or updated snapshot.
|
|
15
14
|
|
|
16
|
-
|
|
15
|
+
The **subject** performs the declared task under one variant. A **suite** is an ordered batch of groups. The evaluator runs subjects, preserves canonical evidence, appends deterministic grades, and renders views. It does not decide whether a skill is good, ready, promotable, or suitable for production.
|
|
17
16
|
|
|
18
|
-
|
|
17
|
+
Inline examples teach the subject. They are candidate content, not evaluation evidence. Keep criteria and expected outcomes outside natural prompts.
|
|
19
18
|
|
|
20
|
-
##
|
|
19
|
+
## Route First
|
|
21
20
|
|
|
22
|
-
|
|
21
|
+
- Inspect adequate saved evidence before proposing a rerun or wording change.
|
|
22
|
+
- Reuse an existing group unchanged when it preserves the question, pressure, and fixed criteria.
|
|
23
|
+
- For a draft-only request, use [Write Skill](../write-skill/SKILL.md); do not fake a run.
|
|
24
|
+
- Preserve partial infrastructure evidence without treating unavailable behavior as pass or fail.
|
|
25
|
+
- Do not mutate canonical evidence or erase an unfavorable result.
|
|
23
26
|
|
|
24
|
-
|
|
27
|
+
Before a substantial rewrite, snapshot the complete current skill package and record an exact identity before editing; use that immutable state as the baseline. A small local wording correction needs no extra snapshot when its exact prior state is already recoverable and the change remains local.
|
|
25
28
|
|
|
26
|
-
|
|
29
|
+
## Name One Question
|
|
27
30
|
|
|
28
|
-
|
|
31
|
+
Separate the behavior being examined:
|
|
29
32
|
|
|
30
|
-
|
|
33
|
+
- **Description activation:** did a natural prompt cause an exact target read, and when?
|
|
34
|
+
- **First-read body behavior:** did explicit body delivery guide the subject with only guaranteed context?
|
|
35
|
+
- **Nearby behavior:** did a close case avoid activation or load without hijacking the task?
|
|
36
|
+
- **Dependency composition:** did exact ordered skills, resources, and context work together?
|
|
37
|
+
- **Retained use:** did guidance remain useful on later declared turns?
|
|
38
|
+
- **Artifact outcome:** did files, state, events, and responses match the fixed criterion?
|
|
39
|
+
- **Cross-host behavior:** did the same question hold on every named host?
|
|
31
40
|
|
|
32
|
-
|
|
41
|
+
One evidence class does not prove another. A successful read does not prove compliance. Direct body delivery cannot prove natural activation. One turn cannot prove retained use. Ambient installation cannot prove declared composition.
|
|
33
42
|
|
|
34
|
-
|
|
43
|
+
Read [evaluation design](references/evaluation-design.md) when choosing the question, evidence class, group shape, or variant boundary.
|
|
35
44
|
|
|
36
|
-
|
|
45
|
+
Keep description and body questions separate by default. `end-to-end` is reserved for an integrated activation-plus-behavior question, but the current runner rejects it before subject execution.
|
|
37
46
|
|
|
38
|
-
|
|
47
|
+
Two common boundary examples:
|
|
39
48
|
|
|
40
|
-
|
|
49
|
+
- If a complete saved result already answers the question, use `view` and raw artifact reads; do not rerun merely to demonstrate process.
|
|
50
|
+
- If explicit body delivery succeeds but a natural prompt never reads the skill, body behavior is supported while description activation still fails.
|
|
41
51
|
|
|
42
|
-
|
|
52
|
+
## Use The Smallest Valid Loop
|
|
43
53
|
|
|
44
|
-
|
|
54
|
+
1. Fix the group question, deterministic criteria, and review questions before subject output exists.
|
|
55
|
+
2. Preserve the earliest natural prompt or strongest pressure that should distinguish behavior.
|
|
56
|
+
3. For a new skill, use no target versus candidate. For a revision, use exact previous versus updated snapshots. For a description-only revision, keep body and resources byte-identical.
|
|
57
|
+
4. Keep prompts or turns, fixture, tools, model, thinking, other skills, context, and criteria fixed except for the target difference.
|
|
58
|
+
5. Run one group or an ordered suite serially.
|
|
59
|
+
6. Inspect deterministic facts and canonical evidence before judging meaning.
|
|
60
|
+
7. Let the active agent or user review unresolved behavior; never launch an automatic semantic grader.
|
|
61
|
+
8. Classify the failed boundary before editing and revise one measured pressure point.
|
|
62
|
+
9. Rerun the complete fixed group with both variants after a measured change.
|
|
45
63
|
|
|
46
|
-
|
|
47
|
-
2. Build the smallest fixture or transcript that reproduces it.
|
|
48
|
-
3. Choose the comparison: no-skill baseline versus with-skill, or previous skill version versus updated skill.
|
|
49
|
-
4. Run both sides.
|
|
50
|
-
5. Grade artifacts before explanations.
|
|
51
|
-
6. Change wording only for measured failures.
|
|
52
|
-
7. Re-run the failed side first.
|
|
64
|
+
If both variants pass, the pressure may be weak or the baseline sufficient. If both fail, distinguish the skill, fixture, environment, host, dependency, or criterion before editing.
|
|
53
65
|
|
|
54
|
-
|
|
66
|
+
## Declare The Definition
|
|
55
67
|
|
|
56
|
-
|
|
68
|
+
Use the exact [definition schema](references/definition-schema.md) for group, environment, expectation, suite, and selector shapes.
|
|
57
69
|
|
|
58
|
-
|
|
70
|
+
Declare the prompt or turns, fixture, exact ordered skills and target, working-tree or Git source, UTF-8 context, tools, model and thinking, deterministic expectations, comparison IDs, and review questions. Snapshot every declared subject resource. Ambient installation is not declared composition.
|
|
59
71
|
|
|
60
|
-
|
|
72
|
+
## Run And View
|
|
61
73
|
|
|
62
|
-
Use
|
|
74
|
+
Use the canonical [skill evaluator](scripts/skill-eval.mjs):
|
|
63
75
|
|
|
64
|
-
|
|
76
|
+
```text
|
|
77
|
+
node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate]
|
|
78
|
+
node <evaluate-skill-directory>/scripts/skill-eval.mjs view <result-id-or-directory> [--group <id-or-position>] [--variant baseline|candidate]
|
|
79
|
+
```
|
|
65
80
|
|
|
66
|
-
|
|
81
|
+
Run from the definition root. Definition targets resolve from the current working directory; suite group references resolve from the suite file. Results are stored under `<cwd>/.skill-eval/runs/<result-id>`. `view` accepts a stored result ID or explicit result directory.
|
|
67
82
|
|
|
68
|
-
|
|
83
|
+
With no selectors, every suite group and both variants are selected. `--group` is invalid for a direct group. Run `--help` before relying on an operation.
|
|
69
84
|
|
|
70
|
-
|
|
85
|
+
Pi is the direct subject process. One-shot descriptions use fresh JSON-mode subjects. Ordered description turns and body groups use one isolated RPC process per variant. Body groups explicitly deliver one target on turn one. `bash`, definition-supplied command execution, and end-to-end execution are unsupported.
|
|
71
86
|
|
|
72
|
-
|
|
87
|
+
Read [execution and evidence](references/execution-and-evidence.md) when operating commands, resolving paths, interpreting states, or reasoning about Pi isolation, persistence, views, cancellation, cleanup, and safeguards.
|
|
73
88
|
|
|
74
|
-
|
|
89
|
+
## Grade And Inspect Facts
|
|
75
90
|
|
|
76
|
-
|
|
91
|
+
Append deterministic grades only after canonical run evidence exists. A grade is derived evidence and never mutates or invalidates a run.
|
|
77
92
|
|
|
78
|
-
|
|
93
|
+
Supported checks cover exact skill/resource reads, typed paths, changed paths, file/response text, explicit JSON states, valid turn scope, and factual baseline-to-candidate transitions. A failed check remains ordinary behavioral evidence. Missing evidence stays unavailable. Malformed checks or grading failures produce separate `grade-error` evidence. Transitions carry no quality or readiness meaning.
|
|
79
94
|
|
|
80
|
-
|
|
95
|
+
Prefer generated grade-first views for routine inspection. Use ordinary file tools for raw run, event, transcript, response, stderr, workspace, definition, grade, and group artifacts. A view may remove repeated transport structure; it must not hide evidence required by the question.
|
|
81
96
|
|
|
82
|
-
|
|
97
|
+
Read [review and revision](references/review-and-revision.md) after evidence exists, when surfaces conflict, meaning still needs judgment, or a revision/rerun boundary must be chosen.
|
|
83
98
|
|
|
84
|
-
|
|
99
|
+
## Report And Stop
|
|
85
100
|
|
|
86
|
-
|
|
87
|
-
- Rule wording.
|
|
88
|
-
- Rule placement.
|
|
89
|
-
- Missing stop condition.
|
|
90
|
-
- Fixture or prompt weakness.
|
|
91
|
-
- Wrong skill loaded.
|
|
92
|
-
- Host runtime limitation.
|
|
101
|
+
Report what ran, exact variant identities, deterministic results, infrastructure failures, unavailable evidence, viewed scope, and canonical result paths. State what the evidence proves and does not prove.
|
|
93
102
|
|
|
94
|
-
|
|
103
|
+
Stop rather than overclaim when required activation, first-read, composition, isolation, multi-turn, artifact, or host evidence is unavailable. The active agent or user judges unresolved behavior. The user decides whether to revise, use, publish, or reject a skill.
|
|
95
104
|
|
|
96
|
-
|
|
105
|
+
Never invoke archived evaluators.
|
|
@@ -0,0 +1,345 @@
|
|
|
1
|
+
# Evaluation Definition Schema
|
|
2
|
+
|
|
3
|
+
Read this when authoring or checking schema-version-1 group and suite JSON.
|
|
4
|
+
|
|
5
|
+
This file owns user-authored definition shapes. Read [execution and evidence](execution-and-evidence.md) for commands, path resolution, run states, persistence, views, isolation, and safeguards.
|
|
6
|
+
|
|
7
|
+
## Definition Root
|
|
8
|
+
|
|
9
|
+
Run `skill-eval` from the repository or fixture root that owns the evaluation inputs.
|
|
10
|
+
|
|
11
|
+
| Input | Resolves from |
|
|
12
|
+
| --- | --- |
|
|
13
|
+
| `run <suite-or-group-path>` | current working directory |
|
|
14
|
+
| suite `groups` entries | directory containing the suite file |
|
|
15
|
+
| group `fixture` | definition root/current working directory |
|
|
16
|
+
| environment `skills` and `context` | definition root/current working directory, then the selected working tree or Git commit |
|
|
17
|
+
|
|
18
|
+
Definition files and declared paths are checked for lexical and canonical containment. Paths are slash-separated relative paths, not absolute paths.
|
|
19
|
+
|
|
20
|
+
## Common Group Shape
|
|
21
|
+
|
|
22
|
+
Every group contains exactly these fields, plus optional `model`:
|
|
23
|
+
|
|
24
|
+
```json
|
|
25
|
+
{
|
|
26
|
+
"schema_version": 1,
|
|
27
|
+
"kind": "group",
|
|
28
|
+
"id": "one-behavioral-question",
|
|
29
|
+
"type": "description",
|
|
30
|
+
"input": { "prompt": "Natural task prompt." },
|
|
31
|
+
"fixture": null,
|
|
32
|
+
"tools": ["read"],
|
|
33
|
+
"variants": {
|
|
34
|
+
"baseline": {
|
|
35
|
+
"source": { "kind": "working-tree" },
|
|
36
|
+
"skills": [],
|
|
37
|
+
"target": null,
|
|
38
|
+
"context": []
|
|
39
|
+
},
|
|
40
|
+
"candidate": {
|
|
41
|
+
"source": { "kind": "working-tree" },
|
|
42
|
+
"skills": ["skills/target"],
|
|
43
|
+
"target": 0,
|
|
44
|
+
"context": []
|
|
45
|
+
}
|
|
46
|
+
},
|
|
47
|
+
"expectations": [],
|
|
48
|
+
"review_questions": [],
|
|
49
|
+
"model": {
|
|
50
|
+
"model": "openai-codex/gpt-5.6-sol",
|
|
51
|
+
"thinking": "low"
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Rules:
|
|
57
|
+
|
|
58
|
+
- `schema_version` is `1`.
|
|
59
|
+
- `kind` is `group`.
|
|
60
|
+
- `id` starts with a lowercase letter, contains lowercase letters, numbers, or single hyphens, and is at most 64 characters.
|
|
61
|
+
- `type` is `description`, `body`, or reserved `end-to-end`. The current runner rejects `end-to-end` before subject execution.
|
|
62
|
+
- `input` contains exactly one of:
|
|
63
|
+
- `prompt`: one non-empty string;
|
|
64
|
+
- `turns`: a non-empty ordered array of non-empty strings. Repeated natural turns are allowed.
|
|
65
|
+
- `fixture` is `null` or one contained relative directory/file path.
|
|
66
|
+
- `tools` is an ordered array with no duplicates.
|
|
67
|
+
- `variants` contains exactly `baseline` and `candidate`.
|
|
68
|
+
- `expectations` is an ordered array with unique expectation IDs.
|
|
69
|
+
- `review_questions` is an ordered string array. Questions are rendered for the reviewer and never sent to the subject.
|
|
70
|
+
- `model` is optional. `model.model` names the provider/model; optional `model.thinking` selects its thinking level. The CLI has no model override.
|
|
71
|
+
|
|
72
|
+
## Description Group Example
|
|
73
|
+
|
|
74
|
+
A description group uses natural prompts and asks whether the exact target was read and when:
|
|
75
|
+
|
|
76
|
+
```json
|
|
77
|
+
{
|
|
78
|
+
"schema_version": 1,
|
|
79
|
+
"kind": "group",
|
|
80
|
+
"id": "natural-activation",
|
|
81
|
+
"type": "description",
|
|
82
|
+
"input": {
|
|
83
|
+
"turns": [
|
|
84
|
+
"Help me inspect this ordinary change.",
|
|
85
|
+
"Now prepare a deployment rollback."
|
|
86
|
+
]
|
|
87
|
+
},
|
|
88
|
+
"fixture": null,
|
|
89
|
+
"tools": ["read"],
|
|
90
|
+
"variants": {
|
|
91
|
+
"baseline": {
|
|
92
|
+
"source": { "kind": "git", "ref": "before-description-change" },
|
|
93
|
+
"skills": ["skills/target"],
|
|
94
|
+
"target": 0,
|
|
95
|
+
"context": []
|
|
96
|
+
},
|
|
97
|
+
"candidate": {
|
|
98
|
+
"source": { "kind": "working-tree" },
|
|
99
|
+
"skills": ["skills/target"],
|
|
100
|
+
"target": 0,
|
|
101
|
+
"context": []
|
|
102
|
+
}
|
|
103
|
+
},
|
|
104
|
+
"expectations": [
|
|
105
|
+
{
|
|
106
|
+
"id": "baseline-read-by-two",
|
|
107
|
+
"kind": "skill-read",
|
|
108
|
+
"variant": "baseline",
|
|
109
|
+
"expect": "by-turn",
|
|
110
|
+
"turn": 2,
|
|
111
|
+
"comparison": "target-read-by-two"
|
|
112
|
+
},
|
|
113
|
+
{
|
|
114
|
+
"id": "candidate-read-by-two",
|
|
115
|
+
"kind": "skill-read",
|
|
116
|
+
"variant": "candidate",
|
|
117
|
+
"expect": "by-turn",
|
|
118
|
+
"turn": 2,
|
|
119
|
+
"comparison": "target-read-by-two"
|
|
120
|
+
}
|
|
121
|
+
],
|
|
122
|
+
"review_questions": []
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
For a description-only revision, the two target snapshots must keep body and resources byte-identical while changing only frontmatter description text.
|
|
127
|
+
|
|
128
|
+
Description groups allow no tools or only `read`. `skill-read` is their target-activation check.
|
|
129
|
+
|
|
130
|
+
## Body Group Example
|
|
131
|
+
|
|
132
|
+
A body group explicitly delivers each target on turn one and observes first-read or later-turn behavior:
|
|
133
|
+
|
|
134
|
+
```json
|
|
135
|
+
{
|
|
136
|
+
"schema_version": 1,
|
|
137
|
+
"kind": "group",
|
|
138
|
+
"id": "retained-artifact-behavior",
|
|
139
|
+
"type": "body",
|
|
140
|
+
"input": {
|
|
141
|
+
"turns": [
|
|
142
|
+
"Inspect the task and prepare the work.",
|
|
143
|
+
"Complete it and write result.json."
|
|
144
|
+
]
|
|
145
|
+
},
|
|
146
|
+
"fixture": "fixtures/task",
|
|
147
|
+
"tools": ["read", "write", "edit"],
|
|
148
|
+
"variants": {
|
|
149
|
+
"baseline": {
|
|
150
|
+
"source": { "kind": "git", "ref": "before-body-change" },
|
|
151
|
+
"skills": ["skills/support", "skills/target"],
|
|
152
|
+
"target": 1,
|
|
153
|
+
"context": ["context/contract.md"]
|
|
154
|
+
},
|
|
155
|
+
"candidate": {
|
|
156
|
+
"source": { "kind": "working-tree" },
|
|
157
|
+
"skills": ["skills/support", "skills/target"],
|
|
158
|
+
"target": 1,
|
|
159
|
+
"context": ["context/contract.md"]
|
|
160
|
+
}
|
|
161
|
+
},
|
|
162
|
+
"expectations": [
|
|
163
|
+
{
|
|
164
|
+
"id": "baseline-result",
|
|
165
|
+
"kind": "path",
|
|
166
|
+
"variant": "baseline",
|
|
167
|
+
"path": "result.json",
|
|
168
|
+
"turn": 2,
|
|
169
|
+
"expect": "exists",
|
|
170
|
+
"comparison": "result-exists"
|
|
171
|
+
},
|
|
172
|
+
{
|
|
173
|
+
"id": "candidate-result",
|
|
174
|
+
"kind": "path",
|
|
175
|
+
"variant": "candidate",
|
|
176
|
+
"path": "result.json",
|
|
177
|
+
"turn": 2,
|
|
178
|
+
"expect": "exists",
|
|
179
|
+
"comparison": "result-exists"
|
|
180
|
+
},
|
|
181
|
+
{
|
|
182
|
+
"id": "candidate-status",
|
|
183
|
+
"kind": "json",
|
|
184
|
+
"variant": "candidate",
|
|
185
|
+
"path": "result.json",
|
|
186
|
+
"turn": 2,
|
|
187
|
+
"expect": "field-equals",
|
|
188
|
+
"pointer": "/status",
|
|
189
|
+
"value": "complete"
|
|
190
|
+
},
|
|
191
|
+
{
|
|
192
|
+
"id": "candidate-changes",
|
|
193
|
+
"kind": "changed-paths",
|
|
194
|
+
"variant": "candidate",
|
|
195
|
+
"turn": 2,
|
|
196
|
+
"expect": "equals",
|
|
197
|
+
"paths": ["result.json"]
|
|
198
|
+
}
|
|
199
|
+
],
|
|
200
|
+
"review_questions": [
|
|
201
|
+
"Did the second turn apply the first-turn guidance without redelivery?"
|
|
202
|
+
]
|
|
203
|
+
}
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Body groups allow no tools or path-guarded `read`, `write`, and `edit`. A no-target baseline uses `skills: []` and `target: null`; the candidate must always identify a target.
|
|
207
|
+
|
|
208
|
+
The examples are complete definition shapes. Replace fixture, resource, model, and Git-ref values with paths and identities that exist under your definition root.
|
|
209
|
+
|
|
210
|
+
## Variant Environment
|
|
211
|
+
|
|
212
|
+
Each environment has exactly:
|
|
213
|
+
|
|
214
|
+
```json
|
|
215
|
+
{
|
|
216
|
+
"source": { "kind": "git", "ref": "v1.2.3" },
|
|
217
|
+
"skills": ["skills/target", "skills/support"],
|
|
218
|
+
"target": 0,
|
|
219
|
+
"context": ["runtime/interaction-contract.md"]
|
|
220
|
+
}
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
`source` is either:
|
|
224
|
+
|
|
225
|
+
```json
|
|
226
|
+
{ "kind": "working-tree" }
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
or:
|
|
230
|
+
|
|
231
|
+
```json
|
|
232
|
+
{ "kind": "git", "ref": "<exact-ref>" }
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
A source applies only to declared skill and context paths in that variant. It never supplies the group fixture: `fixture` is snapshotted once from the definition-root working tree, then copied independently for selected variants. `skills` preserves registration order. `target` is `null` or a zero-based index into `skills`; the candidate target cannot be `null`. `context` preserves declaration order and may name UTF-8 files or directories.
|
|
236
|
+
|
|
237
|
+
## Expectations
|
|
238
|
+
|
|
239
|
+
Every expectation has:
|
|
240
|
+
|
|
241
|
+
- `id`: unique group-local ID using the group-ID syntax;
|
|
242
|
+
- `kind`: one supported deterministic kind;
|
|
243
|
+
- `variant`: `baseline` or `candidate`;
|
|
244
|
+
- optional `comparison`: shared by exactly one baseline and one candidate expectation describing the same expected check;
|
|
245
|
+
- optional body `turn`: one-based declared prompt/turn number.
|
|
246
|
+
|
|
247
|
+
Without `turn`, response checks use the final response and workspace checks use final state. Turn-scoped workspace checks cause an immutable workspace snapshot after each settled turn.
|
|
248
|
+
|
|
249
|
+
| `kind` | Kind-specific fields | `expect` |
|
|
250
|
+
| --- | --- | --- |
|
|
251
|
+
| `skill-read` | timing `turn` except for `never` | `never`, `on-turn`, `by-turn`, `not-before-turn` |
|
|
252
|
+
| `resource-read` | `resource`, optional `index`, `path`, optional body `turn` | `read`, `not-read` |
|
|
253
|
+
| `path` | workspace `path`, optional body `turn` | `exists`, `absent` |
|
|
254
|
+
| `changed-paths` | unique workspace `paths`, optional body `turn` | `equals`, `excludes` |
|
|
255
|
+
| `file-text` | workspace `path`, string `value`, optional body `turn` | `contains`, `not-contains`, `equals` |
|
|
256
|
+
| `json` | workspace `path`, optional body `turn`, conditional `pointer` and `value` | `available`, `missing`, `valid`, `malformed`, `field-present`, `field-absent`, `field-equals` |
|
|
257
|
+
| `response-text` | string `value`, optional body `turn` | `contains`, `not-contains`, `equals` |
|
|
258
|
+
|
|
259
|
+
### Resource paths
|
|
260
|
+
|
|
261
|
+
- `resource: "workspace"`: omit `index`; `path` is relative to the writable workspace.
|
|
262
|
+
- `resource: "skill"`: `index` selects the zero-based declared skill; `path` is relative to that skill directory.
|
|
263
|
+
- `resource: "context"`: `index` selects the zero-based declared context item; `path` is relative to that materialized item.
|
|
264
|
+
|
|
265
|
+
### Changed paths
|
|
266
|
+
|
|
267
|
+
- `equals`: the complete changed-file set must equal `paths`.
|
|
268
|
+
- `excludes`: none of `paths` may appear in the changed-file set.
|
|
269
|
+
|
|
270
|
+
Directory-only presence does not count as a changed file.
|
|
271
|
+
|
|
272
|
+
### JSON states
|
|
273
|
+
|
|
274
|
+
```json
|
|
275
|
+
[
|
|
276
|
+
{
|
|
277
|
+
"id": "file-missing",
|
|
278
|
+
"kind": "json",
|
|
279
|
+
"variant": "candidate",
|
|
280
|
+
"path": "missing.json",
|
|
281
|
+
"expect": "missing"
|
|
282
|
+
},
|
|
283
|
+
{
|
|
284
|
+
"id": "file-malformed",
|
|
285
|
+
"kind": "json",
|
|
286
|
+
"variant": "candidate",
|
|
287
|
+
"path": "broken.json",
|
|
288
|
+
"expect": "malformed"
|
|
289
|
+
},
|
|
290
|
+
{
|
|
291
|
+
"id": "field-absent",
|
|
292
|
+
"kind": "json",
|
|
293
|
+
"variant": "candidate",
|
|
294
|
+
"path": "result.json",
|
|
295
|
+
"expect": "field-absent",
|
|
296
|
+
"pointer": "/optional"
|
|
297
|
+
},
|
|
298
|
+
{
|
|
299
|
+
"id": "field-null",
|
|
300
|
+
"kind": "json",
|
|
301
|
+
"variant": "candidate",
|
|
302
|
+
"path": "result.json",
|
|
303
|
+
"expect": "field-equals",
|
|
304
|
+
"pointer": "/value",
|
|
305
|
+
"value": null
|
|
306
|
+
}
|
|
307
|
+
]
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
A JSON Pointer is `""` for the complete value or begins with `/`; `~0` escapes `~` and `~1` escapes `/`.
|
|
311
|
+
|
|
312
|
+
### Comparison pair
|
|
313
|
+
|
|
314
|
+
Two expectations may share one comparison ID only when they have opposite variants and the same kind and expected predicate. The report records a factual transition such as `fail-to-pass`, never a quality verdict.
|
|
315
|
+
|
|
316
|
+
Definition loading validates common structure and unique IDs. Kind-specific expectation validation occurs during deterministic grading after run persistence. A malformed expectation therefore becomes separate `grade-error` evidence rather than changing the subject run. Use the documented shapes to avoid spending a subject run on an invalid check.
|
|
317
|
+
|
|
318
|
+
Definition-supplied commands and tests are unsupported.
|
|
319
|
+
|
|
320
|
+
## Suite
|
|
321
|
+
|
|
322
|
+
```json
|
|
323
|
+
{
|
|
324
|
+
"schema_version": 1,
|
|
325
|
+
"kind": "suite",
|
|
326
|
+
"id": "workflow-core",
|
|
327
|
+
"groups": [
|
|
328
|
+
"groups/natural-activation.json",
|
|
329
|
+
"groups/body-behavior.json"
|
|
330
|
+
]
|
|
331
|
+
}
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
A suite has only `schema_version`, `kind`, `id`, and ordered unique `groups`. Group references resolve from the suite directory; their resolved definition paths must remain inside the definition root. Referenced group IDs must also be unique.
|
|
335
|
+
|
|
336
|
+
## Selection
|
|
337
|
+
|
|
338
|
+
- With no selectors, every suite group and both variants are selected.
|
|
339
|
+
- `--group <id>` selects one stable group ID.
|
|
340
|
+
- `--group <position>` selects one one-based suite position.
|
|
341
|
+
- `--variant baseline|candidate` applies across selected groups.
|
|
342
|
+
- Excluded variants are `not-selected`, never failed.
|
|
343
|
+
- `--group` is invalid for a direct group definition.
|
|
344
|
+
|
|
345
|
+
Selection changes execution scope only. It does not modify the saved definitions.
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
# Evaluation Design
|
|
2
|
+
|
|
3
|
+
Read this when choosing the behavioral question, evidence class, group shape, or baseline/candidate boundary.
|
|
4
|
+
|
|
5
|
+
## Keep Roles Distinct
|
|
6
|
+
|
|
7
|
+
- **Subject:** performs the declared task under one variant.
|
|
8
|
+
- **Evaluator:** isolates subjects and preserves canonical evidence.
|
|
9
|
+
- **Deterministic grader:** derives fixed mechanical facts after run persistence.
|
|
10
|
+
- **Reviewer:** the active agent or user judges unresolved meaning.
|
|
11
|
+
- **Author:** revises one measured pressure point.
|
|
12
|
+
- **User:** decides whether to revise, use, publish, or reject the skill.
|
|
13
|
+
|
|
14
|
+
The evaluator does not launch an automatic semantic grader or own readiness, promotion, or production status.
|
|
15
|
+
|
|
16
|
+
## Map Claims To Evidence
|
|
17
|
+
|
|
18
|
+
| Claim | Required evidence |
|
|
19
|
+
| --- | --- |
|
|
20
|
+
| Description routes the earliest useful prompt | Natural activation |
|
|
21
|
+
| Body works with guaranteed context | Explicit body delivery |
|
|
22
|
+
| Nearby prompt is not hijacked | Natural activation plus behavioral output |
|
|
23
|
+
| Declared dependencies compose | Exact ordered composition |
|
|
24
|
+
| Guidance remains useful later | Multi-turn evidence |
|
|
25
|
+
| Files or structured state match | Artifact outcome |
|
|
26
|
+
| Behavior holds on several hosts | The same group semantics on every named host |
|
|
27
|
+
|
|
28
|
+
Record declaration, materialization, delivery, observed reads, behavior, artifacts, and derived grades separately. A read does not prove compliance, and correct behavior does not prove which skill caused it.
|
|
29
|
+
|
|
30
|
+
## Choose The Smallest Group Shape
|
|
31
|
+
|
|
32
|
+
- **Description prompt:** natural activation and exact read timing.
|
|
33
|
+
- **Explicit body task:** first-read instructions and behavior without activation ambiguity.
|
|
34
|
+
- **Fixture task:** files or repository state matter.
|
|
35
|
+
- **Stateful turns:** ordered conversation and later behavior matter.
|
|
36
|
+
- **Saved-result review:** stored runs already answer the question; inspect them instead of rerunning.
|
|
37
|
+
- **Suite:** independent questions need one ordered serial invocation.
|
|
38
|
+
|
|
39
|
+
Keep description and body questions separate unless the integrated path is itself the question.
|
|
40
|
+
|
|
41
|
+
## Fix The Variants
|
|
42
|
+
|
|
43
|
+
Every group has exactly:
|
|
44
|
+
|
|
45
|
+
- `baseline`: no target for a new skill, or the exact previous snapshot for a revision;
|
|
46
|
+
- `candidate`: the new or updated snapshot.
|
|
47
|
+
|
|
48
|
+
For a description-only revision, keep body and resources byte-identical. Freeze prompts or turns, fixture, tools, model, thinking, other skills, context, and criteria across variants.
|
|
49
|
+
|
|
50
|
+
## Design Common Questions
|
|
51
|
+
|
|
52
|
+
### Description routing
|
|
53
|
+
|
|
54
|
+
Use the earliest natural prompt where the skill should become useful. For a nearby prompt, predeclare whether success means non-trigger or safe behavior after activation.
|
|
55
|
+
|
|
56
|
+
### First-read body
|
|
57
|
+
|
|
58
|
+
Explicitly deliver the exact body with only guaranteed context. Ambient package context cannot repair a missing dependency.
|
|
59
|
+
|
|
60
|
+
### Composition
|
|
61
|
+
|
|
62
|
+
Materialize exact ordered skills and context. Vary only the target snapshot. Standalone behavior should not receive a hidden base stack.
|
|
63
|
+
|
|
64
|
+
### Retained use
|
|
65
|
+
|
|
66
|
+
Use declared turns in one persistent subject process. Repeated explicit delivery does not prove retained use.
|
|
67
|
+
|
|
68
|
+
### Artifact outcome
|
|
69
|
+
|
|
70
|
+
Grade filesystem state, changed paths, file content, or JSON before relying on the final response. Preserve turn-scoped workspace evidence when later turns may change earlier state.
|
|
71
|
+
|
|
72
|
+
## Boundary Examples
|
|
73
|
+
|
|
74
|
+
Saved complete evidence already contains the required run, grade, and artifacts:
|
|
75
|
+
|
|
76
|
+
> Use `view` and ordinary file reads. Do not rerun merely to demonstrate the process.
|
|
77
|
+
|
|
78
|
+
An explicitly delivered body produces the right answer, but the natural prompt never reads the target:
|
|
79
|
+
|
|
80
|
+
> The body group supports first-read behavior. The description group still fails activation; body success cannot repair it.
|
|
81
|
+
|
|
82
|
+
## Apply Real Pressure
|
|
83
|
+
|
|
84
|
+
A useful group creates a real temptation to violate the rule, keeps criteria outside the prompt, exposes deterministic evidence where possible, and distinguishes baseline behavior from the candidate.
|
|
85
|
+
|
|
86
|
+
If both variants pass, the pressure may be weak or the baseline sufficient. If both fail, classify the skill, fixture, host, dependency, or criterion before editing.
|