prizmkit 1.1.190 → 1.1.192
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/create-prizmkit.js +4 -4
- package/bundled/VERSION.json +3 -3
- package/bundled/adapters/antigravity/assets/prizmkit-hook.cjs +150 -0
- package/bundled/adapters/antigravity/hooks-adapter.js +195 -0
- package/bundled/adapters/antigravity/paths.js +14 -0
- package/bundled/adapters/antigravity/rules-adapter.js +54 -0
- package/bundled/adapters/antigravity/skill-adapter.js +38 -0
- package/bundled/adapters/codex/skill-adapter.js +28 -8
- package/bundled/adapters/cursor/skill-adapter.js +13 -4
- package/bundled/adapters/pi/skill-adapter.js +28 -7
- package/bundled/dev-pipeline/.env.example +4 -12
- package/bundled/dev-pipeline/README.md +21 -15
- package/bundled/dev-pipeline/assets/skill-subagent-integration.md +3 -6
- package/bundled/dev-pipeline/prizmkit_runtime/atomic_io.py +48 -0
- package/bundled/dev-pipeline/prizmkit_runtime/checkpoint_state.py +66 -23
- package/bundled/dev-pipeline/prizmkit_runtime/cli.py +10 -7
- package/bundled/dev-pipeline/prizmkit_runtime/commands.py +2 -21
- package/bundled/dev-pipeline/prizmkit_runtime/config.py +29 -70
- package/bundled/dev-pipeline/prizmkit_runtime/daemon.py +125 -67
- package/bundled/dev-pipeline/prizmkit_runtime/gitops.py +89 -442
- package/bundled/dev-pipeline/prizmkit_runtime/heartbeat.py +93 -49
- package/bundled/dev-pipeline/prizmkit_runtime/interoperability.py +1 -3
- package/bundled/dev-pipeline/prizmkit_runtime/paths.py +0 -6
- package/bundled/dev-pipeline/prizmkit_runtime/platform_detection.py +49 -18
- package/bundled/dev-pipeline/prizmkit_runtime/processes.py +206 -151
- package/bundled/dev-pipeline/prizmkit_runtime/reset.py +2 -23
- package/bundled/dev-pipeline/prizmkit_runtime/reset_preserve.py +15 -134
- package/bundled/dev-pipeline/prizmkit_runtime/results.py +25 -0
- package/bundled/dev-pipeline/prizmkit_runtime/runner_bookkeeping.py +68 -83
- package/bundled/dev-pipeline/prizmkit_runtime/runner_classification.py +15 -5
- package/bundled/dev-pipeline/prizmkit_runtime/runner_models.py +1 -29
- package/bundled/dev-pipeline/prizmkit_runtime/runner_prompts.py +0 -2
- package/bundled/dev-pipeline/prizmkit_runtime/runner_status.py +7 -2
- package/bundled/dev-pipeline/prizmkit_runtime/runners.py +282 -217
- package/bundled/dev-pipeline/prizmkit_runtime/runtime_commit.py +124 -122
- package/bundled/dev-pipeline/prizmkit_runtime/runtime_helper.py +90 -9
- package/bundled/dev-pipeline/prizmkit_runtime/sessions.py +369 -154
- package/bundled/dev-pipeline/prizmkit_runtime/status.py +3 -3
- package/bundled/dev-pipeline/prizmkit_runtime/stream_progress.py +2780 -0
- package/bundled/dev-pipeline/prizmkit_runtime/task_checkout.py +27 -101
- package/bundled/dev-pipeline/prizmkit_runtime/test_result.py +58 -4
- package/bundled/dev-pipeline/scripts/check-session-status.py +43 -19
- package/bundled/dev-pipeline/scripts/continuation.py +6 -4
- package/bundled/dev-pipeline/scripts/detect-stuck.py +28 -28
- package/bundled/dev-pipeline/scripts/generate-bootstrap-prompt.py +7 -2
- package/bundled/dev-pipeline/scripts/generate-bugfix-prompt.py +7 -2
- package/bundled/dev-pipeline/scripts/init-bugfix-pipeline.py +10 -7
- package/bundled/dev-pipeline/scripts/init-change-artifact.py +8 -2
- package/bundled/dev-pipeline/scripts/init-pipeline.py +14 -69
- package/bundled/dev-pipeline/scripts/parse-stream-progress.py +31 -2328
- package/bundled/dev-pipeline/scripts/patch-completion-notes.py +4 -24
- package/bundled/dev-pipeline/scripts/prompt_framework.py +56 -9
- package/bundled/dev-pipeline/scripts/status_common.py +591 -0
- package/bundled/dev-pipeline/scripts/update-bug-status.py +85 -463
- package/bundled/dev-pipeline/scripts/update-checkpoint.py +227 -65
- package/bundled/dev-pipeline/scripts/update-feature-status.py +76 -509
- package/bundled/dev-pipeline/scripts/utils.py +10 -15
- package/bundled/dev-pipeline/templates/bootstrap-tier3.md +4 -4
- package/bundled/dev-pipeline/templates/bug-fix-list-schema.json +7 -0
- package/bundled/dev-pipeline/templates/bugfix-bootstrap-prompt.md +1 -1
- package/bundled/dev-pipeline/templates/sections/artifact-manifest.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-manual-verification.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-review.md +1 -1
- package/bundled/dev-pipeline/templates/sections/bugfix-phase-test.md +2 -1
- package/bundled/dev-pipeline/templates/sections/checkpoint-system.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-browser-verification-auto.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-browser-verification-opencli.md +2 -2
- package/bundled/dev-pipeline/templates/sections/phase-browser-verification.md +2 -2
- package/bundled/dev-pipeline/templates/sections/phase-prizmkit-test.md +2 -1
- package/bundled/dev-pipeline/templates/sections/phase-review-full.md +1 -1
- package/bundled/dev-pipeline/templates/sections/phase-specify-plan-full.md +3 -6
- package/bundled/dev-pipeline/templates/session-status-schema.json +2 -5
- package/bundled/dev-pipeline/tests/_helpers.py +418 -0
- package/bundled/dev-pipeline/tests/conftest.py +45 -1
- package/bundled/dev-pipeline/tests/live_ai_cli_log_smoke.py +1 -0
- package/bundled/dev-pipeline/tests/test_auto_skip.py +4 -69
- package/bundled/dev-pipeline/tests/test_checkpoint_state.py +126 -1
- package/bundled/dev-pipeline/tests/test_daemon_and_scripts.py +257 -0
- package/bundled/dev-pipeline/tests/test_e2e_pipeline_run.py +500 -0
- package/bundled/dev-pipeline/tests/test_generate_bootstrap_prompt.py +59 -117
- package/bundled/dev-pipeline/tests/test_generate_bugfix_prompt.py +12 -10
- package/bundled/dev-pipeline/tests/test_python_runner_parity.py +561 -3094
- package/bundled/dev-pipeline/tests/test_recovery_workflow.py +7 -8
- package/bundled/dev-pipeline/tests/test_repository_bootstrap.py +0 -1
- package/bundled/dev-pipeline/tests/test_reset_modes.py +15 -27
- package/bundled/dev-pipeline/tests/test_reset_preserve.py +54 -139
- package/bundled/dev-pipeline/tests/test_runner_bookkeeping.py +766 -0
- package/bundled/dev-pipeline/tests/test_runner_lifecycle.py +1257 -0
- package/bundled/dev-pipeline/tests/test_runtime_helper.py +97 -20
- package/bundled/dev-pipeline/tests/test_session_launch.py +1101 -0
- package/bundled/dev-pipeline/tests/test_stream_progress.py +585 -0
- package/bundled/dev-pipeline/tests/test_unified_cli.py +490 -2006
- package/bundled/dev-pipeline/tests/test_utils.py +3 -3
- package/bundled/rules/USAGE.md +1 -0
- package/bundled/skills/_metadata.json +9 -63
- package/bundled/skills/app-planner/SKILL.md +32 -303
- package/bundled/skills/app-planner/references/app-design-guide.md +0 -4
- package/bundled/skills/app-planner/references/generated-plan-review.md +2 -1
- package/bundled/skills/app-planner/references/project-brief-guide.md +14 -0
- package/bundled/skills/app-planner/references/project-conventions-discovery.md +4 -6
- package/bundled/skills/app-planner/references/project-state-detection.md +2 -3
- package/bundled/skills/bug-fix-workflow/SKILL.md +1 -2
- package/bundled/skills/bug-fix-workflow/references/bug-diagnosis.md +4 -7
- package/bundled/skills/bug-planner/SKILL.md +54 -435
- package/bundled/skills/bug-planner/references/generated-plan-review.md +3 -2
- package/bundled/skills/bug-planner/references/input-formats.md +32 -0
- package/bundled/skills/bugfix-pipeline-launcher/SKILL.md +4 -6
- package/bundled/skills/bugfix-pipeline-launcher/references/configuration.md +4 -28
- package/bundled/skills/feature-pipeline-launcher/SKILL.md +4 -6
- package/bundled/skills/feature-pipeline-launcher/references/configuration.md +4 -28
- package/bundled/skills/feature-planner/SKILL.md +60 -439
- package/bundled/skills/feature-planner/assets/planning-guide.md +1 -1
- package/bundled/skills/feature-planner/references/completeness-review.md +34 -0
- package/bundled/skills/feature-planner/references/decomposition-patterns.md +2 -2
- package/bundled/skills/feature-planner/references/error-recovery.md +1 -1
- package/bundled/skills/feature-planner/references/generated-plan-review.md +3 -2
- package/bundled/skills/feature-planner/references/incremental-feature-planning.md +3 -1
- package/bundled/skills/feature-workflow/SKILL.md +1 -2
- package/bundled/skills/prizmkit/SKILL.md +7 -1
- package/bundled/skills/prizmkit-code-review/SKILL.md +14 -6
- package/bundled/skills/prizmkit-code-review/references/independent-code-review.md +7 -4
- package/bundled/skills/prizmkit-code-review/references/review-report-template.md +16 -0
- package/bundled/skills/prizmkit-code-review/scripts/render_review_report.py +58 -20
- package/bundled/skills/prizmkit-implement/SKILL.md +1 -1
- package/bundled/skills/prizmkit-init/SKILL.md +25 -128
- package/bundled/skills/prizmkit-init/assets/project-brief-template.md +3 -3
- package/bundled/skills/prizmkit-init/references/config-schema.md +2 -2
- package/bundled/skills/prizmkit-plan/SKILL.md +6 -8
- package/bundled/skills/prizmkit-plan/references/clarify-guide.md +1 -1
- package/bundled/skills/prizmkit-plan/references/independent-plan-review.md +5 -2
- package/bundled/skills/prizmkit-plan/references/review-plan-spec-loop.md +6 -5
- package/bundled/skills/prizmkit-prizm-docs/SKILL.md +1 -13
- package/bundled/skills/prizmkit-prizm-docs/assets/prizm-docs-format.md +1 -1
- package/bundled/skills/prizmkit-prizm-docs/references/op-init.md +1 -1
- package/bundled/skills/prizmkit-prizm-docs/references/op-update.md +1 -1
- package/bundled/skills/prizmkit-retrospective/SKILL.md +6 -4
- package/bundled/skills/prizmkit-test/SKILL.md +25 -133
- package/bundled/skills/prizmkit-test/references/independent-test-review.md +5 -2
- package/bundled/skills/prizmkit-test/references/test-coverage-model.md +6 -0
- package/bundled/skills/prizmkit-workflow/SKILL.md +2 -3
- package/bundled/skills/prizmkit-workflow/references/workflow-state-protocol.md +4 -3
- package/bundled/skills/recovery-workflow/SKILL.md +7 -7
- package/bundled/skills/recovery-workflow/evals/evals.json +1 -1
- package/bundled/skills/recovery-workflow/references/detection.md +1 -3
- package/bundled/skills/recovery-workflow/scripts/detect-recovery-state.py +0 -6
- package/bundled/templates/project-memory-template.md +1 -2
- package/package.json +1 -1
- package/src/ai-cli-launch.js +1 -1
- package/src/clean.js +40 -4
- package/src/config.js +64 -4
- package/src/detect-platform.js +10 -6
- package/src/external-skills.js +2 -2
- package/src/index.js +1 -0
- package/src/manifest.js +1 -1
- package/src/platforms.js +6 -2
- package/src/prompts.js +11 -3
- package/src/scaffold.js +87 -11
- package/src/upgrade.js +9 -3
- package/bundled/dev-pipeline/SCHEMA_ANALYSIS.md +0 -53
- package/bundled/dev-pipeline/scripts/generate-refactor-prompt.py +0 -649
- package/bundled/dev-pipeline/scripts/init-refactor-pipeline.py +0 -463
- package/bundled/dev-pipeline/scripts/update-refactor-status.py +0 -1512
- package/bundled/dev-pipeline/templates/refactor-bootstrap-prompt.md +0 -240
- package/bundled/dev-pipeline/templates/refactor-list-schema.json +0 -266
- package/bundled/dev-pipeline/templates/sections/refactor-critical-paths.md +0 -15
- package/bundled/dev-pipeline/templates/sections/refactor-mission.md +0 -9
- package/bundled/dev-pipeline/templates/sections/refactor-phase-commit-report.md +0 -24
- package/bundled/dev-pipeline/templates/sections/refactor-phase-implement.md +0 -16
- package/bundled/dev-pipeline/templates/sections/refactor-phase-init.md +0 -9
- package/bundled/dev-pipeline/templates/sections/refactor-phase-plan.md +0 -22
- package/bundled/dev-pipeline/templates/sections/refactor-phase-review.md +0 -19
- package/bundled/dev-pipeline/templates/sections/refactor-phase-test.md +0 -24
- package/bundled/dev-pipeline/templates/sections/refactor-reminders.md +0 -12
- package/bundled/dev-pipeline/templates/sections/refactor-session-context.md +0 -6
- package/bundled/dev-pipeline/templates/sections/refactor-task-contract.md +0 -52
- package/bundled/dev-pipeline/tests/test_generate_refactor_prompt.py +0 -478
- package/bundled/skills/feature-planner/assets/evaluation-guide.md +0 -44
- package/bundled/skills/refactor-pipeline-launcher/SKILL.md +0 -219
- package/bundled/skills/refactor-pipeline-launcher/references/configuration.md +0 -78
- package/bundled/skills/refactor-planner/SKILL.md +0 -467
- package/bundled/skills/refactor-planner/assets/planning-guide.md +0 -291
- package/bundled/skills/refactor-planner/references/behavior-preservation.md +0 -187
- package/bundled/skills/refactor-planner/references/error-recovery.md +0 -45
- package/bundled/skills/refactor-planner/references/fast-path.md +0 -59
- package/bundled/skills/refactor-planner/references/generated-plan-review.md +0 -107
- package/bundled/skills/refactor-planner/references/planning-phases.md +0 -172
- package/bundled/skills/refactor-planner/references/refactor-scoping-guide.md +0 -230
- package/bundled/skills/refactor-planner/scripts/validate-and-generate-refactor.py +0 -1120
- package/bundled/skills/refactor-workflow/SKILL.md +0 -205
|
@@ -1,52 +0,0 @@
|
|
|
1
|
-
## Refactor Task Contract
|
|
2
|
-
|
|
3
|
-
### Objective
|
|
4
|
-
|
|
5
|
-
Execute {{REFACTOR_ID}}: "{{REFACTOR_TITLE}}".
|
|
6
|
-
|
|
7
|
-
{{REFACTOR_DESCRIPTION}}
|
|
8
|
-
|
|
9
|
-
{{USER_CONTEXT}}
|
|
10
|
-
|
|
11
|
-
### Scope
|
|
12
|
-
|
|
13
|
-
**Files:**
|
|
14
|
-
{{SCOPE_FILES}}
|
|
15
|
-
|
|
16
|
-
**Modules:**
|
|
17
|
-
{{SCOPE_MODULES}}
|
|
18
|
-
|
|
19
|
-
### Behavior Preservation
|
|
20
|
-
|
|
21
|
-
- **Strategy**: {{BEHAVIOR_STRATEGY}}
|
|
22
|
-
|
|
23
|
-
{{BEHAVIOR_PRESERVATION}}
|
|
24
|
-
|
|
25
|
-
### Existing Tests
|
|
26
|
-
|
|
27
|
-
{{EXISTING_TESTS}}
|
|
28
|
-
|
|
29
|
-
### New Tests Needed
|
|
30
|
-
|
|
31
|
-
{{NEW_TESTS_NEEDED}}
|
|
32
|
-
|
|
33
|
-
### Acceptance Criteria
|
|
34
|
-
|
|
35
|
-
{{ACCEPTANCE_CRITERIA}}
|
|
36
|
-
|
|
37
|
-
### Gate Checklist
|
|
38
|
-
|
|
39
|
-
{{AC_CHECKLIST}}
|
|
40
|
-
|
|
41
|
-
Gate rule:
|
|
42
|
-
- `[x]` means verified with behavior-preservation evidence.
|
|
43
|
-
- Any remaining `[ ]` means the refactor is incomplete.
|
|
44
|
-
- Blocked gates must be documented in `failure-log.md`; blocked is not success.
|
|
45
|
-
|
|
46
|
-
### Dependencies
|
|
47
|
-
|
|
48
|
-
{{DEPENDENCIES}}
|
|
49
|
-
|
|
50
|
-
### App Global Context
|
|
51
|
-
|
|
52
|
-
{{GLOBAL_CONTEXT}}
|
|
@@ -1,478 +0,0 @@
|
|
|
1
|
-
"""Tests for generate-refactor-prompt.py continuation behavior."""
|
|
2
|
-
|
|
3
|
-
import json
|
|
4
|
-
from argparse import Namespace
|
|
5
|
-
from pathlib import Path
|
|
6
|
-
import os
|
|
7
|
-
|
|
8
|
-
from continuation import append_continuation_handoff
|
|
9
|
-
from generate_refactor_prompt import (
|
|
10
|
-
build_replacements,
|
|
11
|
-
format_behavior_preservation,
|
|
12
|
-
format_existing_tests,
|
|
13
|
-
generate_refactor_checkpoint,
|
|
14
|
-
merge_refactor_checkpoint_state,
|
|
15
|
-
refactor_checkpoint_metadata,
|
|
16
|
-
route_refactor_repair,
|
|
17
|
-
)
|
|
18
|
-
|
|
19
|
-
REMOVED_MAX_LOG_ENV = "MAX_" + "LOG_SIZE"
|
|
20
|
-
REMOVED_MAX_LOG_HUMAN = REMOVED_MAX_LOG_ENV + "_HUMAN"
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
class TestRefactorContinuationHandoff:
|
|
24
|
-
def test_writes_specs_and_task_local_summaries(self, tmp_path):
|
|
25
|
-
args = Namespace(
|
|
26
|
-
output=str(tmp_path / ".prizmkit/state/refactor/R-007/sessions/R-007-new/bootstrap-prompt.md"),
|
|
27
|
-
state_dir=str(tmp_path / ".prizmkit/state/refactor"),
|
|
28
|
-
continuation_mode="ai_error",
|
|
29
|
-
failure_classification="ai_runtime_or_provider_error",
|
|
30
|
-
code_retry_count="1",
|
|
31
|
-
infra_error_count="2",
|
|
32
|
-
max_code_retries="3",
|
|
33
|
-
max_infra_retries="3",
|
|
34
|
-
no_progress_count="0",
|
|
35
|
-
progress_fingerprint=json.dumps({"git_diff_fingerprint": "diff", "head_commit": "abc", "checkpoint_cursor": "S03:pending"}),
|
|
36
|
-
previous_session_id="R-007-old",
|
|
37
|
-
continuation_count="1",
|
|
38
|
-
context_overflow_count="1",
|
|
39
|
-
active_dev_branch="refactor/R-007",
|
|
40
|
-
base_branch="main",
|
|
41
|
-
continuation_summary_path=".prizmkit/specs/R-007/continuation-summary.md",
|
|
42
|
-
)
|
|
43
|
-
prompt = append_continuation_handoff(
|
|
44
|
-
"REFACTOR PROMPT", args, str(tmp_path), "refactor", "R-007", "R-007",
|
|
45
|
-
{"steps": [{"id": "S01", "skill": "plan", "name": "Plan", "status": "pending"}]},
|
|
46
|
-
)
|
|
47
|
-
|
|
48
|
-
assert prompt.startswith("REFACTOR PROMPT")
|
|
49
|
-
assert "This is a fresh headless AI CLI session for the same task." in prompt
|
|
50
|
-
assert "Do not attempt `/compact` as the recovery mechanism." not in prompt
|
|
51
|
-
assert ("Log " + "monitor") not in prompt
|
|
52
|
-
assert ("log-" + "monitor") not in prompt
|
|
53
|
-
assert (tmp_path / ".prizmkit/specs/R-007/continuation-summary.md").exists()
|
|
54
|
-
assert (tmp_path / ".prizmkit/refactor/R-007/continuation-summary.md").exists()
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
class TestRefactorBehaviorPreservation:
|
|
58
|
-
def test_schema_boolean_existing_tests_renders_without_iteration(self):
|
|
59
|
-
assert format_existing_tests(True) == (
|
|
60
|
-
"- Coverage declared: yes (specific test paths not supplied)"
|
|
61
|
-
)
|
|
62
|
-
assert format_existing_tests(False) == (
|
|
63
|
-
"- Coverage declared: no (specific test paths not supplied)"
|
|
64
|
-
)
|
|
65
|
-
assert "**Existing Tests Cover Scope**: yes" in format_behavior_preservation({
|
|
66
|
-
"strategy": "test-gate",
|
|
67
|
-
"existing_tests": True,
|
|
68
|
-
})
|
|
69
|
-
|
|
70
|
-
def test_historical_test_path_list_remains_supported(self):
|
|
71
|
-
value = ["test/unit.test.js", "test/integration.test.js"]
|
|
72
|
-
rendered = format_existing_tests(value)
|
|
73
|
-
assert "`test/unit.test.js`" in rendered
|
|
74
|
-
assert "`test/integration.test.js`" in rendered
|
|
75
|
-
preservation = format_behavior_preservation({
|
|
76
|
-
"strategy": "test-gate",
|
|
77
|
-
"existing_tests": value,
|
|
78
|
-
})
|
|
79
|
-
assert "**Existing Tests**" in preservation
|
|
80
|
-
assert "`test/unit.test.js`" in preservation
|
|
81
|
-
|
|
82
|
-
def test_invalid_test_shapes_degrade_to_unspecified(self):
|
|
83
|
-
assert format_existing_tests("test/unit.test.js") == "- (none specified)"
|
|
84
|
-
assert "Existing Tests" not in format_behavior_preservation({
|
|
85
|
-
"strategy": "test-gate",
|
|
86
|
-
"existing_tests": {"path": "test/unit.test.js"},
|
|
87
|
-
})
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
class TestRefactorCheckpointMerge:
|
|
91
|
-
def test_preserves_skipped_without_artifacts(self, tmp_path):
|
|
92
|
-
fresh = generate_refactor_checkpoint("R-008", "R-008-new")
|
|
93
|
-
existing = generate_refactor_checkpoint("R-008", "R-008-old")
|
|
94
|
-
existing["refactor_state"] = refactor_checkpoint_metadata(
|
|
95
|
-
".prizmkit/refactor/R-008",
|
|
96
|
-
stage="test",
|
|
97
|
-
status="failed",
|
|
98
|
-
stage_result="TEST_NEEDS_FIXES",
|
|
99
|
-
repair_scope="public-interface",
|
|
100
|
-
repair_round=2,
|
|
101
|
-
next_stage=None,
|
|
102
|
-
resume_from=None,
|
|
103
|
-
)
|
|
104
|
-
existing["steps"][0]["status"] = "skipped"
|
|
105
|
-
|
|
106
|
-
merged = merge_refactor_checkpoint_state(existing, fresh, str(tmp_path))
|
|
107
|
-
|
|
108
|
-
assert merged["steps"][0]["status"] == "skipped"
|
|
109
|
-
assert merged["steps"][1]["status"] == "pending"
|
|
110
|
-
assert merged["refactor_state"]["status"] == "failed"
|
|
111
|
-
assert merged["refactor_state"]["repair_scope"] == "public-interface"
|
|
112
|
-
assert merged["refactor_state"]["repair_round"] == 2
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
class TestExplicitProjectRoot:
|
|
116
|
-
def test_build_replacements_infers_project_root_from_refactor_list(self, tmp_path, monkeypatch):
|
|
117
|
-
monkeypatch.delenv("PRIZMKIT_PLATFORM", raising=False)
|
|
118
|
-
project = tmp_path / "project"
|
|
119
|
-
plan_path = project / ".prizmkit" / "plans" / "refactor-list.json"
|
|
120
|
-
plan_path.parent.mkdir(parents=True)
|
|
121
|
-
(project / ".prizmkit" / "config.json").write_text(
|
|
122
|
-
'{"platform":"claude","ai_cli":"claude"}', encoding="utf-8"
|
|
123
|
-
)
|
|
124
|
-
args = Namespace(
|
|
125
|
-
refactor_list=str(plan_path),
|
|
126
|
-
refactor_id="R-001",
|
|
127
|
-
session_id="session-1",
|
|
128
|
-
run_id="run-1",
|
|
129
|
-
retry_count="0",
|
|
130
|
-
resume_phase="null",
|
|
131
|
-
state_dir=str(project / ".prizmkit" / "state" / "refactor"),
|
|
132
|
-
output=str(project / "out.md"),
|
|
133
|
-
template=None,
|
|
134
|
-
)
|
|
135
|
-
refactor = {"id": "R-001", "title": "Restructure", "description": "d"}
|
|
136
|
-
|
|
137
|
-
replacements = build_replacements(
|
|
138
|
-
args, refactor, [refactor], {}, str(Path("dev-pipeline/scripts").resolve())
|
|
139
|
-
)
|
|
140
|
-
|
|
141
|
-
assert replacements["{{PROJECT_ROOT}}"] == str(project.resolve())
|
|
142
|
-
assert "{{" + "REVIEWER_" + "SUBAGENT_PATH" + "}}" not in replacements
|
|
143
|
-
assert "{{SESSION_STATUS_PATH}}" not in replacements
|
|
144
|
-
|
|
145
|
-
def test_build_replacements_uses_cli_project_root_for_rendered_paths(self, tmp_path, monkeypatch):
|
|
146
|
-
monkeypatch.setenv("PRIZMKIT_PLATFORM", "claude")
|
|
147
|
-
execution_root = tmp_path / "worktree"
|
|
148
|
-
main_checkout = tmp_path / "main"
|
|
149
|
-
execution_root.mkdir()
|
|
150
|
-
main_checkout.mkdir()
|
|
151
|
-
args = Namespace(
|
|
152
|
-
refactor_list=str(main_checkout / ".prizmkit" / "plans" / "refactor-list.json"),
|
|
153
|
-
refactor_id="R-001",
|
|
154
|
-
session_id="session-1",
|
|
155
|
-
run_id="run-1",
|
|
156
|
-
retry_count="0",
|
|
157
|
-
resume_phase="null",
|
|
158
|
-
state_dir=str(main_checkout / ".prizmkit" / "state" / "refactor"),
|
|
159
|
-
output=str(main_checkout / ".prizmkit" / "state" / "refactor" / "R-001" / "sessions" / "session-1" / "bootstrap-prompt.md"),
|
|
160
|
-
template=None,
|
|
161
|
-
project_root=str(execution_root),
|
|
162
|
-
)
|
|
163
|
-
refactor = {"id": "R-001", "title": "Worktree Refactor", "description": "d"}
|
|
164
|
-
|
|
165
|
-
replacements = build_replacements(
|
|
166
|
-
args,
|
|
167
|
-
refactor,
|
|
168
|
-
[refactor],
|
|
169
|
-
{},
|
|
170
|
-
str(main_checkout / "dev-pipeline" / "scripts"),
|
|
171
|
-
)
|
|
172
|
-
|
|
173
|
-
assert replacements["{{PROJECT_ROOT}}"] == str(execution_root.resolve())
|
|
174
|
-
assert "{{SESSION_STATUS_PATH}}" not in replacements
|
|
175
|
-
assert "{{" + "DEV_" + "SUBAGENT_PATH" + "}}" not in replacements
|
|
176
|
-
assert "### Checkpoint Update Helper" in replacements["{{CHECKPOINT_SYSTEM}}"]
|
|
177
|
-
assert "{{CHECKPOINT_PYTHON_CMD}}" in replacements["{{CHECKPOINT_SYSTEM}}"]
|
|
178
|
-
assert "pipeline progress mechanism" not in replacements["{{CHECKPOINT_SYSTEM}}"]
|
|
179
|
-
assert "prizmkit-runtime-helper.py" in replacements["{{RUNTIME_HELPER_CMD}}"]
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
class TestRefactorPromptMaxLogRemoval:
|
|
183
|
-
def test_refactor_source_has_no_deleted_log_size_injection(self):
|
|
184
|
-
deleted_section = "log-size-" + "awareness.md"
|
|
185
|
-
loader = "load_" + "log_size_section"
|
|
186
|
-
placeholder = "{{LOG_" + "SIZE_AWARENESS}}"
|
|
187
|
-
deleted_title = "Session " + "Checkpoint Awareness"
|
|
188
|
-
source = Path("dev-pipeline/scripts/generate-refactor-prompt.py").read_text(encoding="utf-8")
|
|
189
|
-
template = Path("dev-pipeline/templates/refactor-bootstrap-prompt.md").read_text(encoding="utf-8")
|
|
190
|
-
|
|
191
|
-
assert (Path("dev-pipeline/templates/sections") / deleted_section).exists() is False
|
|
192
|
-
for content in (source, template):
|
|
193
|
-
for forbidden in (deleted_section, loader, placeholder, deleted_title):
|
|
194
|
-
assert forbidden not in content
|
|
195
|
-
|
|
196
|
-
def test_refactor_replacements_keep_session_log_without_max_log_placeholders(self, monkeypatch):
|
|
197
|
-
monkeypatch.setenv("PRIZMKIT_PLATFORM", "claude")
|
|
198
|
-
args = Namespace(
|
|
199
|
-
refactor_id="R-001",
|
|
200
|
-
session_id="session-1",
|
|
201
|
-
run_id="run-1",
|
|
202
|
-
retry_count="0",
|
|
203
|
-
resume_phase="null",
|
|
204
|
-
state_dir=None,
|
|
205
|
-
output="out.md",
|
|
206
|
-
template=None,
|
|
207
|
-
)
|
|
208
|
-
refactor = {"id": "R-001", "title": "Restructure", "description": "d"}
|
|
209
|
-
|
|
210
|
-
replacements = build_replacements(
|
|
211
|
-
args, refactor, [refactor], {}, str(Path("dev-pipeline/scripts").resolve())
|
|
212
|
-
)
|
|
213
|
-
|
|
214
|
-
assert replacements["{{SESSION_LOG_PATH}}"].endswith(
|
|
215
|
-
".prizmkit/state/refactor/R-001/sessions/session-1/logs/session.log"
|
|
216
|
-
)
|
|
217
|
-
assert "{{" + REMOVED_MAX_LOG_ENV + "}}" not in replacements
|
|
218
|
-
assert "{{" + REMOVED_MAX_LOG_HUMAN + "}}" not in replacements
|
|
219
|
-
assert "{{LOG_" + "SIZE_AWARENESS}}" not in replacements
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
HEADLESS_FORBIDDEN_PROMPT_STRINGS = [
|
|
223
|
-
"### Interactive Claude Code Note",
|
|
224
|
-
"### Project Conventions",
|
|
225
|
-
"Scaffold / generated file awareness",
|
|
226
|
-
"### Resume Behavior",
|
|
227
|
-
"[NEEDS CLARIFICATION]",
|
|
228
|
-
"pause for interactive input",
|
|
229
|
-
"prizmkit-test-gate.py",
|
|
230
|
-
"/compact",
|
|
231
|
-
"Session " + "Checkpoint Awareness",
|
|
232
|
-
"{{LOG_" + "SIZE_AWARENESS}}",
|
|
233
|
-
]
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
def test_template_source_is_clean_for_headless_prompt_contract():
|
|
237
|
-
content = Path("dev-pipeline/templates/refactor-bootstrap-prompt.md").read_text(encoding="utf-8")
|
|
238
|
-
for forbidden in HEADLESS_FORBIDDEN_PROMPT_STRINGS:
|
|
239
|
-
assert forbidden not in content
|
|
240
|
-
assert "prizmkit-workflow" not in content
|
|
241
|
-
assert "/prizmkit-test scope=this-change artifact_dir=.prizmkit/refactor/{{REFACTOR_ID}}/" in content
|
|
242
|
-
assert "operation=prepare-runtime-commit" in content
|
|
243
|
-
assert "local_commit_authorized" not in content
|
|
244
|
-
assert "push_authorized" not in content
|
|
245
|
-
assert "Behavior-preservation context remains supporting input" in content
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
# ---------------------------------------------------------------------------
|
|
249
|
-
# Anti-Bash regression guardrails (F-039)
|
|
250
|
-
# Note: grep -n, sed -n, grep -q, wc -l excluded — may appear in cross-pipeline
|
|
251
|
-
# context budget rules as AI tool hints, not as operational shell snippets.
|
|
252
|
-
# | sed (trailing space) only matches pipe-to-sed operational patterns.
|
|
253
|
-
# ---------------------------------------------------------------------------
|
|
254
|
-
|
|
255
|
-
RETIRED_EXTRACTED_SNIPPETS_REFACTOR = [
|
|
256
|
-
"find . -maxdepth",
|
|
257
|
-
"| sed ",
|
|
258
|
-
r"grep -c '^\- \[ \]'",
|
|
259
|
-
"2>/dev/null",
|
|
260
|
-
"/dev/null",
|
|
261
|
-
"$DEV_PORT",
|
|
262
|
-
"$!",
|
|
263
|
-
"$HOME",
|
|
264
|
-
"if [ ",
|
|
265
|
-
"which playwright-cli",
|
|
266
|
-
"which opencli",
|
|
267
|
-
"lsof -ti",
|
|
268
|
-
"xargs",
|
|
269
|
-
" kill -",
|
|
270
|
-
"kill -",
|
|
271
|
-
"|| echo",
|
|
272
|
-
"mkdir -p",
|
|
273
|
-
"touch .prizmkit",
|
|
274
|
-
]
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
def _render_refactor_prompt_full(tmp_path, platform="claude"):
|
|
278
|
-
"""Render a complete refactor prompt through the generator."""
|
|
279
|
-
import json, os
|
|
280
|
-
from generate_refactor_prompt import main as refactor_main
|
|
281
|
-
|
|
282
|
-
project = tmp_path / "refactor-anti-bash"
|
|
283
|
-
project.mkdir(parents=True)
|
|
284
|
-
(project / ".prizmkit").mkdir(exist_ok=True)
|
|
285
|
-
(project / ".prizmkit" / "config.json").write_text(
|
|
286
|
-
json.dumps({"platform": platform, "ai_cli": platform}), encoding="utf-8"
|
|
287
|
-
)
|
|
288
|
-
|
|
289
|
-
refactor_list = project / ".prizmkit" / "plans" / "refactor-list.json"
|
|
290
|
-
refactor_list.parent.mkdir(parents=True)
|
|
291
|
-
refactor_list.write_text(json.dumps({
|
|
292
|
-
"refactors": [{
|
|
293
|
-
"id": "R-001", "title": "Restructure", "description": "Preserve behavior",
|
|
294
|
-
"acceptance_criteria": ["Behavior preserved"],
|
|
295
|
-
"scope": {"files": ["src/a.py"], "modules": ["src"]},
|
|
296
|
-
}],
|
|
297
|
-
"global_context": {"language": "Python"},
|
|
298
|
-
}), encoding="utf-8")
|
|
299
|
-
output = project / "refactor-prompt.md"
|
|
300
|
-
old_argv = os.sys.argv
|
|
301
|
-
old_platform = os.environ.get("PRIZMKIT_PLATFORM")
|
|
302
|
-
try:
|
|
303
|
-
os.environ.pop("PRIZMKIT_PLATFORM", None)
|
|
304
|
-
os.sys.argv = [
|
|
305
|
-
"generate-refactor-prompt.py",
|
|
306
|
-
"--refactor-list", str(refactor_list),
|
|
307
|
-
"--refactor-id", "R-001",
|
|
308
|
-
"--session-id", "session-1",
|
|
309
|
-
"--run-id", "run-1",
|
|
310
|
-
"--retry-count", "0",
|
|
311
|
-
"--resume-phase", "null",
|
|
312
|
-
"--state-dir", str(project / ".prizmkit" / "state" / "refactor"),
|
|
313
|
-
"--output", str(output),
|
|
314
|
-
]
|
|
315
|
-
old_cwd = os.getcwd()
|
|
316
|
-
os.chdir(project)
|
|
317
|
-
try:
|
|
318
|
-
try:
|
|
319
|
-
refactor_main()
|
|
320
|
-
except SystemExit as exc:
|
|
321
|
-
if exc.code not in (0, None):
|
|
322
|
-
raise
|
|
323
|
-
finally:
|
|
324
|
-
os.chdir(old_cwd)
|
|
325
|
-
finally:
|
|
326
|
-
os.sys.argv = old_argv
|
|
327
|
-
if old_platform is None:
|
|
328
|
-
os.environ.pop("PRIZMKIT_PLATFORM", None)
|
|
329
|
-
else:
|
|
330
|
-
os.environ["PRIZMKIT_PLATFORM"] = old_platform
|
|
331
|
-
return output.read_text(encoding="utf-8")
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
class TestRefactorBootstrapShellExtraction:
|
|
335
|
-
"""Regression guardrails: refactor rendered prompts and source use helper, not shell."""
|
|
336
|
-
|
|
337
|
-
def test_rendered_refactor_prompt_uses_runtime_helper_not_extracted_bash(self, tmp_path):
|
|
338
|
-
prompt = _render_refactor_prompt_full(tmp_path)
|
|
339
|
-
for snippet in RETIRED_EXTRACTED_SNIPPETS_REFACTOR:
|
|
340
|
-
assert snippet not in prompt, f"Retired shell snippet found in refactor prompt: {snippet}"
|
|
341
|
-
assert "{{RUNTIME_HELPER_CMD}}" not in prompt
|
|
342
|
-
assert "python" in prompt.lower()
|
|
343
|
-
assert ".prizmkit/dev-pipeline/scripts/prizmkit-runtime-helper.py" in prompt
|
|
344
|
-
assert "artifact ensure-dir .prizmkit/refactor/R-001" in prompt
|
|
345
|
-
assert "text final-verdict .prizmkit/refactor/R-001/review-report.md" in prompt
|
|
346
|
-
|
|
347
|
-
def test_refactor_template_source_is_clean_of_helper_covered_snippets(self):
|
|
348
|
-
content = Path("dev-pipeline/templates/refactor-bootstrap-prompt.md").read_text(encoding="utf-8")
|
|
349
|
-
for snippet in RETIRED_EXTRACTED_SNIPPETS_REFACTOR:
|
|
350
|
-
assert snippet not in content, f"Shell snippet in refactor template: {snippet}"
|
|
351
|
-
|
|
352
|
-
def test_continuation_source_is_clean_of_helper_covered_snippets(self):
|
|
353
|
-
content = Path("dev-pipeline/scripts/continuation.py").read_text(encoding="utf-8")
|
|
354
|
-
for snippet in RETIRED_EXTRACTED_SNIPPETS_REFACTOR:
|
|
355
|
-
assert snippet not in content, f"Shell snippet in continuation.py: {snippet}"
|
|
356
|
-
|
|
357
|
-
def test_refactor_generator_source_is_clean_of_helper_covered_snippets(self):
|
|
358
|
-
content = Path("dev-pipeline/scripts/generate-refactor-prompt.py").read_text(encoding="utf-8")
|
|
359
|
-
for snippet in RETIRED_EXTRACTED_SNIPPETS_REFACTOR:
|
|
360
|
-
assert snippet not in content, f"Shell snippet in generate-refactor-prompt.py: {snippet}"
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
class TestRefactorCheckpointGeneration:
|
|
364
|
-
def test_default_checkpoint_has_all_atomic_gates_in_order(self):
|
|
365
|
-
checkpoint = generate_refactor_checkpoint("R-001", "session-1")
|
|
366
|
-
|
|
367
|
-
assert [step["skill"] for step in checkpoint["steps"]] == [
|
|
368
|
-
"prizmkit-init",
|
|
369
|
-
"prizmkit-plan",
|
|
370
|
-
"prizmkit-implement",
|
|
371
|
-
"prizmkit-code-review",
|
|
372
|
-
"prizmkit-test",
|
|
373
|
-
"refactor-report",
|
|
374
|
-
"completion-summary",
|
|
375
|
-
"prizmkit-retrospective",
|
|
376
|
-
"prizmkit-committer",
|
|
377
|
-
]
|
|
378
|
-
assert checkpoint["steps"][4]["depends_on"] == "S04"
|
|
379
|
-
assert checkpoint["steps"][4]["required_artifacts"] == [
|
|
380
|
-
".prizmkit/refactor/R-001/test-report.md",
|
|
381
|
-
".prizmkit/refactor/R-001/test-result.json",
|
|
382
|
-
]
|
|
383
|
-
assert checkpoint["steps"][5]["depends_on"] == "S05"
|
|
384
|
-
|
|
385
|
-
def test_refactor_result_routing_owns_bounded_test_repair(self):
|
|
386
|
-
review = route_refactor_repair(
|
|
387
|
-
"REVIEW_NEEDS_FIXES", repair_scope="production", repair_round=0,
|
|
388
|
-
)
|
|
389
|
-
needs_fixes = route_refactor_repair(
|
|
390
|
-
"TEST_NEEDS_FIXES", repair_scope="production", repair_round=1,
|
|
391
|
-
)
|
|
392
|
-
blocked = route_refactor_repair("TEST_BLOCKED", repair_round=1)
|
|
393
|
-
|
|
394
|
-
assert review["route"] == [
|
|
395
|
-
"prizmkit-implement", "prizmkit-code-review", "prizmkit-test",
|
|
396
|
-
]
|
|
397
|
-
assert needs_fixes["blocked"] is False
|
|
398
|
-
assert needs_fixes["route"] == [
|
|
399
|
-
"prizmkit-implement", "prizmkit-code-review", "prizmkit-test",
|
|
400
|
-
]
|
|
401
|
-
assert needs_fixes["repair_round"] == 2
|
|
402
|
-
assert blocked["blocked"] is True
|
|
403
|
-
assert blocked["route"] == []
|
|
404
|
-
assert blocked["next_stage"] is None
|
|
405
|
-
assert blocked["terminal_status"] == "WORKFLOW_BLOCKED"
|
|
406
|
-
assert blocked["blocked_reason"] == "test_blocked"
|
|
407
|
-
unsafe = route_refactor_repair("TEST_NEEDS_FIXES", repair_round=0)
|
|
408
|
-
assert unsafe["blocked_reason"] == "unsafe_repair_scope"
|
|
409
|
-
|
|
410
|
-
def test_refactor_checkpoint_metadata_has_no_evidence_handoff(self):
|
|
411
|
-
metadata = refactor_checkpoint_metadata(
|
|
412
|
-
".prizmkit/refactor/R-001",
|
|
413
|
-
stage="test",
|
|
414
|
-
status="completed",
|
|
415
|
-
stage_result="TEST_PASS",
|
|
416
|
-
next_stage="prizmkit-retrospective",
|
|
417
|
-
resume_from="prizmkit-retrospective",
|
|
418
|
-
)
|
|
419
|
-
|
|
420
|
-
assert metadata["artifact_dir"] == ".prizmkit/refactor/R-001"
|
|
421
|
-
assert metadata["status"] == "completed"
|
|
422
|
-
assert metadata["stage_result"] == "TEST_PASS"
|
|
423
|
-
assert "authoritative_evidence_paths" not in metadata
|
|
424
|
-
assert metadata["next_stage"] == "prizmkit-retrospective"
|
|
425
|
-
assert metadata["resume_from"] == "prizmkit-retrospective"
|
|
426
|
-
|
|
427
|
-
def test_rendered_refactor_prompt_is_atomic_headless_and_local_commit_only(self, tmp_path):
|
|
428
|
-
prompt = _render_refactor_prompt_full(tmp_path)
|
|
429
|
-
|
|
430
|
-
assert prompt.index("/prizmkit-code-review") < prompt.index(
|
|
431
|
-
"/prizmkit-test scope=this-change artifact_dir=.prizmkit/refactor/R-001/"
|
|
432
|
-
) < prompt.index("### Refactor Report") < prompt.index("### Retrospective") < prompt.index("### Runtime Commit Handoff")
|
|
433
|
-
assert "prizmkit-workflow" not in prompt
|
|
434
|
-
assert "Existing behavior-preservation context is input to testing" in prompt
|
|
435
|
-
assert "local_commit_authorized" not in prompt
|
|
436
|
-
assert "operation=prepare-runtime-commit" in prompt
|
|
437
|
-
assert "The Python runtime, not the AI session, executes" in prompt
|
|
438
|
-
assert "complete final Git-visible workspace" in prompt
|
|
439
|
-
assert '"schema_version": 2' in prompt
|
|
440
|
-
assert '"intended_paths"' not in prompt
|
|
441
|
-
assert "Ignored paths remain local" in prompt
|
|
442
|
-
assert "automatic push" not in prompt.lower()
|
|
443
|
-
assert "push_authorized\": true" not in prompt
|
|
444
|
-
assert "write the session status" not in prompt.lower()
|
|
445
|
-
|
|
446
|
-
def test_compatibility_checkpoint_keeps_configured_browser_before_test(self):
|
|
447
|
-
checkpoint = generate_refactor_checkpoint(
|
|
448
|
-
"R-001", "session", browser_enabled=True,
|
|
449
|
-
)
|
|
450
|
-
skills = [step["skill"] for step in checkpoint["steps"]]
|
|
451
|
-
assert skills.index("prizmkit-code-review") < skills.index(
|
|
452
|
-
"browser-verification"
|
|
453
|
-
) < skills.index("prizmkit-test")
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
class TestRefactorCheckpointGenerationLegacy:
|
|
457
|
-
def test_setup_section_is_step_and_finalization_group_expands(self):
|
|
458
|
-
checkpoint = generate_refactor_checkpoint(
|
|
459
|
-
"R-001",
|
|
460
|
-
"session-1",
|
|
461
|
-
sections=[
|
|
462
|
-
("refactor-phase-init", "setup"),
|
|
463
|
-
("refactor-phase-implement", "implement"),
|
|
464
|
-
("refactor-phase-commit-report", "report"),
|
|
465
|
-
("retrospective", "retrospective"),
|
|
466
|
-
("runtime-commit-handoff", "commit"),
|
|
467
|
-
],
|
|
468
|
-
)
|
|
469
|
-
|
|
470
|
-
assert [step["skill"] for step in checkpoint["steps"]] == [
|
|
471
|
-
"prizmkit-init",
|
|
472
|
-
"prizmkit-implement",
|
|
473
|
-
"refactor-report",
|
|
474
|
-
"completion-summary",
|
|
475
|
-
"prizmkit-retrospective",
|
|
476
|
-
"prizmkit-committer",
|
|
477
|
-
]
|
|
478
|
-
assert checkpoint["steps"][2]["depends_on"] == "S02"
|
|
@@ -1,44 +0,0 @@
|
|
|
1
|
-
# feature planner Evaluation Guide
|
|
2
|
-
|
|
3
|
-
This guide is for maintainers who evaluate and iterate on the `feature-planner` skill quality.
|
|
4
|
-
|
|
5
|
-
## Evaluation & Quality Gates (Optional but Recommended)
|
|
6
|
-
|
|
7
|
-
After multiple planning cycles or before committing refined skill logic, run standardized evaluation.
|
|
8
|
-
|
|
9
|
-
### One-Command Evaluation
|
|
10
|
-
|
|
11
|
-
Requires npm setup:
|
|
12
|
-
|
|
13
|
-
```bash
|
|
14
|
-
npm run skill:review -- \
|
|
15
|
-
--workspace .prizmkit/skill-evals/feature-planner-workspace \
|
|
16
|
-
--iteration iteration-N \
|
|
17
|
-
--skill-name feature-planner \
|
|
18
|
-
--skill-path ${SKILL_DIR} \
|
|
19
|
-
--runs 3 \
|
|
20
|
-
--grader-cmd "python3 ${SKILL_DIR}/scripts/validate-and-generate.py grade --workspace {workspace} --iteration {iteration}"
|
|
21
|
-
```
|
|
22
|
-
|
|
23
|
-
Produces:
|
|
24
|
-
- `benchmark.json` — quantitative metrics (pass rate, feature quality, time)
|
|
25
|
-
- `benchmark.md` — human-readable summary
|
|
26
|
-
- `review.html` — interactive evaluation viewer
|
|
27
|
-
|
|
28
|
-
### Metrics Tracked
|
|
29
|
-
|
|
30
|
-
| Metric | Computation | Target | Interpretation |
|
|
31
|
-
|--------|-------------|--------|-----------------|
|
|
32
|
-
| `plan_validity` | % runs with validation pass | >95% | Higher = more robust planning |
|
|
33
|
-
| `avg_features_per_run` | avg feature count | ±20% consistency | Should be stable across runs |
|
|
34
|
-
| `avg_acceptance_criteria` | AC count per feature | 4-6 | Target sweet spot for test coverage |
|
|
35
|
-
| `dependency_complexity` | max DAG depth, cycle count | depth < 5 | Manageable dependency graph |
|
|
36
|
-
| `description_quality` | word count, keyword coverage | min 20 words | Sufficient AC detail |
|
|
37
|
-
| `latency_sec` | wall-clock execution time | <120s per run | UX acceptable |
|
|
38
|
-
|
|
39
|
-
### When to Run Evaluation
|
|
40
|
-
|
|
41
|
-
- After major SKILL.md revisions
|
|
42
|
-
- Before releasing new skill updates
|
|
43
|
-
- Quarterly quality assurance
|
|
44
|
-
- Post-optimization to measure improvement
|