@complexthings/superpowers-agent 10.0.3 → 11.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/superpowers-agent +17 -222
- package/.github/copilot-instructions.md +0 -9
- package/AGENTS.md +19 -140
- package/README.md +79 -507
- package/package.json +1 -7
- package/.agents/docs/SUPERPOWERS.md +0 -37
- package/.agents/skills/.gitkeep +0 -0
- package/.agents/skills/claude-handoff/SKILL.md +0 -18
- package/.agents/skills/code-review/SKILL.md +0 -89
- package/.agents/skills/codebase-design/DEEPENING.md +0 -37
- package/.agents/skills/codebase-design/DESIGN-IT-TWICE.md +0 -44
- package/.agents/skills/codebase-design/SKILL.md +0 -114
- package/.agents/skills/design-an-interface/SKILL.md +0 -94
- package/.agents/skills/diagnosing-bugs/SKILL.md +0 -134
- package/.agents/skills/diagnosing-bugs/scripts/hitl-loop.template.sh +0 -41
- package/.agents/skills/domain-modeling/ADR-FORMAT.md +0 -47
- package/.agents/skills/domain-modeling/CONTEXT-FORMAT.md +0 -60
- package/.agents/skills/domain-modeling/SKILL.md +0 -74
- package/.agents/skills/fable-mode/SKILL.md +0 -95
- package/.agents/skills/git-guardrails-claude-code/SKILL.md +0 -95
- package/.agents/skills/git-guardrails-claude-code/scripts/block-dangerous-git.sh +0 -25
- package/.agents/skills/grill-me/SKILL.md +0 -7
- package/.agents/skills/grill-with-docs/SKILL.md +0 -7
- package/.agents/skills/grilling/SKILL.md +0 -14
- package/.agents/skills/handoff/SKILL.md +0 -16
- package/.agents/skills/i-have-adhd/SKILL.md +0 -120
- package/.agents/skills/implement/SKILL.md +0 -11
- package/.agents/skills/improve-codebase-architecture/HTML-REPORT.md +0 -123
- package/.agents/skills/improve-codebase-architecture/SKILL.md +0 -66
- package/.agents/skills/loop-me/SKILL.md +0 -32
- package/.agents/skills/prototype/LOGIC.md +0 -79
- package/.agents/skills/prototype/SKILL.md +0 -30
- package/.agents/skills/prototype/UI.md +0 -112
- package/.agents/skills/qa/SKILL.md +0 -130
- package/.agents/skills/request-refactor-plan/SKILL.md +0 -68
- package/.agents/skills/research/SKILL.md +0 -12
- package/.agents/skills/resolving-merge-conflicts/SKILL.md +0 -14
- package/.agents/skills/scaffold-exercises/SKILL.md +0 -106
- package/.agents/skills/setup-matt-pocock-skills/SKILL.md +0 -127
- package/.agents/skills/setup-matt-pocock-skills/domain.md +0 -51
- package/.agents/skills/setup-matt-pocock-skills/issue-tracker-github.md +0 -45
- package/.agents/skills/setup-matt-pocock-skills/issue-tracker-gitlab.md +0 -46
- package/.agents/skills/setup-matt-pocock-skills/issue-tracker-local.md +0 -30
- package/.agents/skills/setup-matt-pocock-skills/triage-labels.md +0 -15
- package/.agents/skills/skill-creator/LICENSE.txt +0 -202
- package/.agents/skills/skill-creator/SKILL.md +0 -485
- package/.agents/skills/skill-creator/agents/analyzer.md +0 -274
- package/.agents/skills/skill-creator/agents/comparator.md +0 -202
- package/.agents/skills/skill-creator/agents/grader.md +0 -223
- package/.agents/skills/skill-creator/assets/eval_review.html +0 -146
- package/.agents/skills/skill-creator/eval-viewer/generate_review.py +0 -471
- package/.agents/skills/skill-creator/eval-viewer/viewer.html +0 -1325
- package/.agents/skills/skill-creator/references/schemas.md +0 -430
- package/.agents/skills/skill-creator/scripts/__init__.py +0 -0
- package/.agents/skills/skill-creator/scripts/__pycache__/__init__.cpython-314.pyc +0 -0
- package/.agents/skills/skill-creator/scripts/__pycache__/run_eval.cpython-314.pyc +0 -0
- package/.agents/skills/skill-creator/scripts/__pycache__/utils.cpython-314.pyc +0 -0
- package/.agents/skills/skill-creator/scripts/aggregate_benchmark.py +0 -401
- package/.agents/skills/skill-creator/scripts/generate_report.py +0 -326
- package/.agents/skills/skill-creator/scripts/improve_description.py +0 -247
- package/.agents/skills/skill-creator/scripts/package_skill.py +0 -136
- package/.agents/skills/skill-creator/scripts/quick_validate.py +0 -103
- package/.agents/skills/skill-creator/scripts/run_eval.py +0 -310
- package/.agents/skills/skill-creator/scripts/run_loop.py +0 -328
- package/.agents/skills/skill-creator/scripts/utils.py +0 -47
- package/.agents/skills/tdd/SKILL.md +0 -36
- package/.agents/skills/tdd/mocking.md +0 -59
- package/.agents/skills/tdd/tests.md +0 -77
- package/.agents/skills/teach/GLOSSARY-FORMAT.md +0 -35
- package/.agents/skills/teach/LEARNING-RECORD-FORMAT.md +0 -46
- package/.agents/skills/teach/MISSION-FORMAT.md +0 -31
- package/.agents/skills/teach/RESOURCES-FORMAT.md +0 -32
- package/.agents/skills/teach/SKILL.md +0 -140
- package/.agents/skills/to-spec/SKILL.md +0 -75
- package/.agents/skills/to-tickets/SKILL.md +0 -114
- package/.agents/skills/triage/AGENT-BRIEF.md +0 -207
- package/.agents/skills/triage/OUT-OF-SCOPE.md +0 -105
- package/.agents/skills/triage/SKILL.md +0 -112
- package/.agents/skills/ubiquitous-language/SKILL.md +0 -93
- package/.agents/skills/wayfinder/SKILL.md +0 -127
- package/.agents/skills/writing-great-skills/GLOSSARY.md +0 -201
- package/.agents/skills/writing-great-skills/SKILL.md +0 -83
- package/.agents/templates/AGENTS.md.template +0 -27
- package/.agents/templates/SUPERPOWERS.md.template +0 -37
- package/skills/collaboration/brainstorming/SKILL.md +0 -66
- package/skills/collaboration/brainstorming/skill.json +0 -9
- package/skills/collaboration/leveraging-cli-tools/SKILL.md +0 -110
- package/skills/collaboration/leveraging-cli-tools/references/copilot-instructions.md +0 -30
- package/skills/collaboration/leveraging-cli-tools/scripts/setup-ponytail.sh +0 -185
- package/skills/collaboration/leveraging-cli-tools/scripts/setup-rtk.sh +0 -217
- package/skills/collaboration/leveraging-cli-tools/skill.json +0 -9
- package/skills/meta/create-skill-json/SKILL.md +0 -370
- package/skills/meta/create-skill-json/skill.json +0 -9
- package/skills/meta/create-skill-json/test-scenarios.md +0 -29
- package/skills/setup-skills/SKILL.md +0 -54
- package/skills/setup-skills/skill.json +0 -8
|
@@ -1,328 +0,0 @@
|
|
|
1
|
-
#!/usr/bin/env python3
|
|
2
|
-
"""Run the eval + improve loop until all pass or max iterations reached.
|
|
3
|
-
|
|
4
|
-
Combines run_eval.py and improve_description.py in a loop, tracking history
|
|
5
|
-
and returning the best description found. Supports train/test split to prevent
|
|
6
|
-
overfitting.
|
|
7
|
-
"""
|
|
8
|
-
|
|
9
|
-
import argparse
|
|
10
|
-
import json
|
|
11
|
-
import random
|
|
12
|
-
import sys
|
|
13
|
-
import tempfile
|
|
14
|
-
import time
|
|
15
|
-
import webbrowser
|
|
16
|
-
from pathlib import Path
|
|
17
|
-
|
|
18
|
-
from scripts.generate_report import generate_html
|
|
19
|
-
from scripts.improve_description import improve_description
|
|
20
|
-
from scripts.run_eval import find_project_root, run_eval
|
|
21
|
-
from scripts.utils import parse_skill_md
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
def split_eval_set(eval_set: list[dict], holdout: float, seed: int = 42) -> tuple[list[dict], list[dict]]:
|
|
25
|
-
"""Split eval set into train and test sets, stratified by should_trigger."""
|
|
26
|
-
random.seed(seed)
|
|
27
|
-
|
|
28
|
-
# Separate by should_trigger
|
|
29
|
-
trigger = [e for e in eval_set if e["should_trigger"]]
|
|
30
|
-
no_trigger = [e for e in eval_set if not e["should_trigger"]]
|
|
31
|
-
|
|
32
|
-
# Shuffle each group
|
|
33
|
-
random.shuffle(trigger)
|
|
34
|
-
random.shuffle(no_trigger)
|
|
35
|
-
|
|
36
|
-
# Calculate split points
|
|
37
|
-
n_trigger_test = max(1, int(len(trigger) * holdout))
|
|
38
|
-
n_no_trigger_test = max(1, int(len(no_trigger) * holdout))
|
|
39
|
-
|
|
40
|
-
# Split
|
|
41
|
-
test_set = trigger[:n_trigger_test] + no_trigger[:n_no_trigger_test]
|
|
42
|
-
train_set = trigger[n_trigger_test:] + no_trigger[n_no_trigger_test:]
|
|
43
|
-
|
|
44
|
-
return train_set, test_set
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
def run_loop(
|
|
48
|
-
eval_set: list[dict],
|
|
49
|
-
skill_path: Path,
|
|
50
|
-
description_override: str | None,
|
|
51
|
-
num_workers: int,
|
|
52
|
-
timeout: int,
|
|
53
|
-
max_iterations: int,
|
|
54
|
-
runs_per_query: int,
|
|
55
|
-
trigger_threshold: float,
|
|
56
|
-
holdout: float,
|
|
57
|
-
model: str,
|
|
58
|
-
verbose: bool,
|
|
59
|
-
live_report_path: Path | None = None,
|
|
60
|
-
log_dir: Path | None = None,
|
|
61
|
-
) -> dict:
|
|
62
|
-
"""Run the eval + improvement loop."""
|
|
63
|
-
project_root = find_project_root()
|
|
64
|
-
name, original_description, content = parse_skill_md(skill_path)
|
|
65
|
-
current_description = description_override or original_description
|
|
66
|
-
|
|
67
|
-
# Split into train/test if holdout > 0
|
|
68
|
-
if holdout > 0:
|
|
69
|
-
train_set, test_set = split_eval_set(eval_set, holdout)
|
|
70
|
-
if verbose:
|
|
71
|
-
print(f"Split: {len(train_set)} train, {len(test_set)} test (holdout={holdout})", file=sys.stderr)
|
|
72
|
-
else:
|
|
73
|
-
train_set = eval_set
|
|
74
|
-
test_set = []
|
|
75
|
-
|
|
76
|
-
history = []
|
|
77
|
-
exit_reason = "unknown"
|
|
78
|
-
|
|
79
|
-
for iteration in range(1, max_iterations + 1):
|
|
80
|
-
if verbose:
|
|
81
|
-
print(f"\n{'='*60}", file=sys.stderr)
|
|
82
|
-
print(f"Iteration {iteration}/{max_iterations}", file=sys.stderr)
|
|
83
|
-
print(f"Description: {current_description}", file=sys.stderr)
|
|
84
|
-
print(f"{'='*60}", file=sys.stderr)
|
|
85
|
-
|
|
86
|
-
# Evaluate train + test together in one batch for parallelism
|
|
87
|
-
all_queries = train_set + test_set
|
|
88
|
-
t0 = time.time()
|
|
89
|
-
all_results = run_eval(
|
|
90
|
-
eval_set=all_queries,
|
|
91
|
-
skill_name=name,
|
|
92
|
-
description=current_description,
|
|
93
|
-
num_workers=num_workers,
|
|
94
|
-
timeout=timeout,
|
|
95
|
-
project_root=project_root,
|
|
96
|
-
runs_per_query=runs_per_query,
|
|
97
|
-
trigger_threshold=trigger_threshold,
|
|
98
|
-
model=model,
|
|
99
|
-
)
|
|
100
|
-
eval_elapsed = time.time() - t0
|
|
101
|
-
|
|
102
|
-
# Split results back into train/test by matching queries
|
|
103
|
-
train_queries_set = {q["query"] for q in train_set}
|
|
104
|
-
train_result_list = [r for r in all_results["results"] if r["query"] in train_queries_set]
|
|
105
|
-
test_result_list = [r for r in all_results["results"] if r["query"] not in train_queries_set]
|
|
106
|
-
|
|
107
|
-
train_passed = sum(1 for r in train_result_list if r["pass"])
|
|
108
|
-
train_total = len(train_result_list)
|
|
109
|
-
train_summary = {"passed": train_passed, "failed": train_total - train_passed, "total": train_total}
|
|
110
|
-
train_results = {"results": train_result_list, "summary": train_summary}
|
|
111
|
-
|
|
112
|
-
if test_set:
|
|
113
|
-
test_passed = sum(1 for r in test_result_list if r["pass"])
|
|
114
|
-
test_total = len(test_result_list)
|
|
115
|
-
test_summary = {"passed": test_passed, "failed": test_total - test_passed, "total": test_total}
|
|
116
|
-
test_results = {"results": test_result_list, "summary": test_summary}
|
|
117
|
-
else:
|
|
118
|
-
test_results = None
|
|
119
|
-
test_summary = None
|
|
120
|
-
|
|
121
|
-
history.append({
|
|
122
|
-
"iteration": iteration,
|
|
123
|
-
"description": current_description,
|
|
124
|
-
"train_passed": train_summary["passed"],
|
|
125
|
-
"train_failed": train_summary["failed"],
|
|
126
|
-
"train_total": train_summary["total"],
|
|
127
|
-
"train_results": train_results["results"],
|
|
128
|
-
"test_passed": test_summary["passed"] if test_summary else None,
|
|
129
|
-
"test_failed": test_summary["failed"] if test_summary else None,
|
|
130
|
-
"test_total": test_summary["total"] if test_summary else None,
|
|
131
|
-
"test_results": test_results["results"] if test_results else None,
|
|
132
|
-
# For backward compat with report generator
|
|
133
|
-
"passed": train_summary["passed"],
|
|
134
|
-
"failed": train_summary["failed"],
|
|
135
|
-
"total": train_summary["total"],
|
|
136
|
-
"results": train_results["results"],
|
|
137
|
-
})
|
|
138
|
-
|
|
139
|
-
# Write live report if path provided
|
|
140
|
-
if live_report_path:
|
|
141
|
-
partial_output = {
|
|
142
|
-
"original_description": original_description,
|
|
143
|
-
"best_description": current_description,
|
|
144
|
-
"best_score": "in progress",
|
|
145
|
-
"iterations_run": len(history),
|
|
146
|
-
"holdout": holdout,
|
|
147
|
-
"train_size": len(train_set),
|
|
148
|
-
"test_size": len(test_set),
|
|
149
|
-
"history": history,
|
|
150
|
-
}
|
|
151
|
-
live_report_path.write_text(generate_html(partial_output, auto_refresh=True, skill_name=name))
|
|
152
|
-
|
|
153
|
-
if verbose:
|
|
154
|
-
def print_eval_stats(label, results, elapsed):
|
|
155
|
-
pos = [r for r in results if r["should_trigger"]]
|
|
156
|
-
neg = [r for r in results if not r["should_trigger"]]
|
|
157
|
-
tp = sum(r["triggers"] for r in pos)
|
|
158
|
-
pos_runs = sum(r["runs"] for r in pos)
|
|
159
|
-
fn = pos_runs - tp
|
|
160
|
-
fp = sum(r["triggers"] for r in neg)
|
|
161
|
-
neg_runs = sum(r["runs"] for r in neg)
|
|
162
|
-
tn = neg_runs - fp
|
|
163
|
-
total = tp + tn + fp + fn
|
|
164
|
-
precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0
|
|
165
|
-
recall = tp / (tp + fn) if (tp + fn) > 0 else 1.0
|
|
166
|
-
accuracy = (tp + tn) / total if total > 0 else 0.0
|
|
167
|
-
print(f"{label}: {tp+tn}/{total} correct, precision={precision:.0%} recall={recall:.0%} accuracy={accuracy:.0%} ({elapsed:.1f}s)", file=sys.stderr)
|
|
168
|
-
for r in results:
|
|
169
|
-
status = "PASS" if r["pass"] else "FAIL"
|
|
170
|
-
rate_str = f"{r['triggers']}/{r['runs']}"
|
|
171
|
-
print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:60]}", file=sys.stderr)
|
|
172
|
-
|
|
173
|
-
print_eval_stats("Train", train_results["results"], eval_elapsed)
|
|
174
|
-
if test_summary:
|
|
175
|
-
print_eval_stats("Test ", test_results["results"], 0)
|
|
176
|
-
|
|
177
|
-
if train_summary["failed"] == 0:
|
|
178
|
-
exit_reason = f"all_passed (iteration {iteration})"
|
|
179
|
-
if verbose:
|
|
180
|
-
print(f"\nAll train queries passed on iteration {iteration}!", file=sys.stderr)
|
|
181
|
-
break
|
|
182
|
-
|
|
183
|
-
if iteration == max_iterations:
|
|
184
|
-
exit_reason = f"max_iterations ({max_iterations})"
|
|
185
|
-
if verbose:
|
|
186
|
-
print(f"\nMax iterations reached ({max_iterations}).", file=sys.stderr)
|
|
187
|
-
break
|
|
188
|
-
|
|
189
|
-
# Improve the description based on train results
|
|
190
|
-
if verbose:
|
|
191
|
-
print(f"\nImproving description...", file=sys.stderr)
|
|
192
|
-
|
|
193
|
-
t0 = time.time()
|
|
194
|
-
# Strip test scores from history so improvement model can't see them
|
|
195
|
-
blinded_history = [
|
|
196
|
-
{k: v for k, v in h.items() if not k.startswith("test_")}
|
|
197
|
-
for h in history
|
|
198
|
-
]
|
|
199
|
-
new_description = improve_description(
|
|
200
|
-
skill_name=name,
|
|
201
|
-
skill_content=content,
|
|
202
|
-
current_description=current_description,
|
|
203
|
-
eval_results=train_results,
|
|
204
|
-
history=blinded_history,
|
|
205
|
-
model=model,
|
|
206
|
-
log_dir=log_dir,
|
|
207
|
-
iteration=iteration,
|
|
208
|
-
)
|
|
209
|
-
improve_elapsed = time.time() - t0
|
|
210
|
-
|
|
211
|
-
if verbose:
|
|
212
|
-
print(f"Proposed ({improve_elapsed:.1f}s): {new_description}", file=sys.stderr)
|
|
213
|
-
|
|
214
|
-
current_description = new_description
|
|
215
|
-
|
|
216
|
-
# Find the best iteration by TEST score (or train if no test set)
|
|
217
|
-
if test_set:
|
|
218
|
-
best = max(history, key=lambda h: h["test_passed"] or 0)
|
|
219
|
-
best_score = f"{best['test_passed']}/{best['test_total']}"
|
|
220
|
-
else:
|
|
221
|
-
best = max(history, key=lambda h: h["train_passed"])
|
|
222
|
-
best_score = f"{best['train_passed']}/{best['train_total']}"
|
|
223
|
-
|
|
224
|
-
if verbose:
|
|
225
|
-
print(f"\nExit reason: {exit_reason}", file=sys.stderr)
|
|
226
|
-
print(f"Best score: {best_score} (iteration {best['iteration']})", file=sys.stderr)
|
|
227
|
-
|
|
228
|
-
return {
|
|
229
|
-
"exit_reason": exit_reason,
|
|
230
|
-
"original_description": original_description,
|
|
231
|
-
"best_description": best["description"],
|
|
232
|
-
"best_score": best_score,
|
|
233
|
-
"best_train_score": f"{best['train_passed']}/{best['train_total']}",
|
|
234
|
-
"best_test_score": f"{best['test_passed']}/{best['test_total']}" if test_set else None,
|
|
235
|
-
"final_description": current_description,
|
|
236
|
-
"iterations_run": len(history),
|
|
237
|
-
"holdout": holdout,
|
|
238
|
-
"train_size": len(train_set),
|
|
239
|
-
"test_size": len(test_set),
|
|
240
|
-
"history": history,
|
|
241
|
-
}
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
def main():
|
|
245
|
-
parser = argparse.ArgumentParser(description="Run eval + improve loop")
|
|
246
|
-
parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file")
|
|
247
|
-
parser.add_argument("--skill-path", required=True, help="Path to skill directory")
|
|
248
|
-
parser.add_argument("--description", default=None, help="Override starting description")
|
|
249
|
-
parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers")
|
|
250
|
-
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
|
|
251
|
-
parser.add_argument("--max-iterations", type=int, default=5, help="Max improvement iterations")
|
|
252
|
-
parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query")
|
|
253
|
-
parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold")
|
|
254
|
-
parser.add_argument("--holdout", type=float, default=0.4, help="Fraction of eval set to hold out for testing (0 to disable)")
|
|
255
|
-
parser.add_argument("--model", required=True, help="Model for improvement")
|
|
256
|
-
parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")
|
|
257
|
-
parser.add_argument("--report", default="auto", help="Generate HTML report at this path (default: 'auto' for temp file, 'none' to disable)")
|
|
258
|
-
parser.add_argument("--results-dir", default=None, help="Save all outputs (results.json, report.html, log.txt) to a timestamped subdirectory here")
|
|
259
|
-
args = parser.parse_args()
|
|
260
|
-
|
|
261
|
-
eval_set = json.loads(Path(args.eval_set).read_text())
|
|
262
|
-
skill_path = Path(args.skill_path)
|
|
263
|
-
|
|
264
|
-
if not (skill_path / "SKILL.md").exists():
|
|
265
|
-
print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)
|
|
266
|
-
sys.exit(1)
|
|
267
|
-
|
|
268
|
-
name, _, _ = parse_skill_md(skill_path)
|
|
269
|
-
|
|
270
|
-
# Set up live report path
|
|
271
|
-
if args.report != "none":
|
|
272
|
-
if args.report == "auto":
|
|
273
|
-
timestamp = time.strftime("%Y%m%d_%H%M%S")
|
|
274
|
-
live_report_path = Path(tempfile.gettempdir()) / f"skill_description_report_{skill_path.name}_{timestamp}.html"
|
|
275
|
-
else:
|
|
276
|
-
live_report_path = Path(args.report)
|
|
277
|
-
# Open the report immediately so the user can watch
|
|
278
|
-
live_report_path.write_text("<html><body><h1>Starting optimization loop...</h1><meta http-equiv='refresh' content='5'></body></html>")
|
|
279
|
-
webbrowser.open(str(live_report_path))
|
|
280
|
-
else:
|
|
281
|
-
live_report_path = None
|
|
282
|
-
|
|
283
|
-
# Determine output directory (create before run_loop so logs can be written)
|
|
284
|
-
if args.results_dir:
|
|
285
|
-
timestamp = time.strftime("%Y-%m-%d_%H%M%S")
|
|
286
|
-
results_dir = Path(args.results_dir) / timestamp
|
|
287
|
-
results_dir.mkdir(parents=True, exist_ok=True)
|
|
288
|
-
else:
|
|
289
|
-
results_dir = None
|
|
290
|
-
|
|
291
|
-
log_dir = results_dir / "logs" if results_dir else None
|
|
292
|
-
|
|
293
|
-
output = run_loop(
|
|
294
|
-
eval_set=eval_set,
|
|
295
|
-
skill_path=skill_path,
|
|
296
|
-
description_override=args.description,
|
|
297
|
-
num_workers=args.num_workers,
|
|
298
|
-
timeout=args.timeout,
|
|
299
|
-
max_iterations=args.max_iterations,
|
|
300
|
-
runs_per_query=args.runs_per_query,
|
|
301
|
-
trigger_threshold=args.trigger_threshold,
|
|
302
|
-
holdout=args.holdout,
|
|
303
|
-
model=args.model,
|
|
304
|
-
verbose=args.verbose,
|
|
305
|
-
live_report_path=live_report_path,
|
|
306
|
-
log_dir=log_dir,
|
|
307
|
-
)
|
|
308
|
-
|
|
309
|
-
# Save JSON output
|
|
310
|
-
json_output = json.dumps(output, indent=2)
|
|
311
|
-
print(json_output)
|
|
312
|
-
if results_dir:
|
|
313
|
-
(results_dir / "results.json").write_text(json_output)
|
|
314
|
-
|
|
315
|
-
# Write final HTML report (without auto-refresh)
|
|
316
|
-
if live_report_path:
|
|
317
|
-
live_report_path.write_text(generate_html(output, auto_refresh=False, skill_name=name))
|
|
318
|
-
print(f"\nReport: {live_report_path}", file=sys.stderr)
|
|
319
|
-
|
|
320
|
-
if results_dir and live_report_path:
|
|
321
|
-
(results_dir / "report.html").write_text(generate_html(output, auto_refresh=False, skill_name=name))
|
|
322
|
-
|
|
323
|
-
if results_dir:
|
|
324
|
-
print(f"Results saved to: {results_dir}", file=sys.stderr)
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
if __name__ == "__main__":
|
|
328
|
-
main()
|
|
@@ -1,47 +0,0 @@
|
|
|
1
|
-
"""Shared utilities for skill-creator scripts."""
|
|
2
|
-
|
|
3
|
-
from pathlib import Path
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
def parse_skill_md(skill_path: Path) -> tuple[str, str, str]:
|
|
8
|
-
"""Parse a SKILL.md file, returning (name, description, full_content)."""
|
|
9
|
-
content = (skill_path / "SKILL.md").read_text()
|
|
10
|
-
lines = content.split("\n")
|
|
11
|
-
|
|
12
|
-
if lines[0].strip() != "---":
|
|
13
|
-
raise ValueError("SKILL.md missing frontmatter (no opening ---)")
|
|
14
|
-
|
|
15
|
-
end_idx = None
|
|
16
|
-
for i, line in enumerate(lines[1:], start=1):
|
|
17
|
-
if line.strip() == "---":
|
|
18
|
-
end_idx = i
|
|
19
|
-
break
|
|
20
|
-
|
|
21
|
-
if end_idx is None:
|
|
22
|
-
raise ValueError("SKILL.md missing frontmatter (no closing ---)")
|
|
23
|
-
|
|
24
|
-
name = ""
|
|
25
|
-
description = ""
|
|
26
|
-
frontmatter_lines = lines[1:end_idx]
|
|
27
|
-
i = 0
|
|
28
|
-
while i < len(frontmatter_lines):
|
|
29
|
-
line = frontmatter_lines[i]
|
|
30
|
-
if line.startswith("name:"):
|
|
31
|
-
name = line[len("name:"):].strip().strip('"').strip("'")
|
|
32
|
-
elif line.startswith("description:"):
|
|
33
|
-
value = line[len("description:"):].strip()
|
|
34
|
-
# Handle YAML multiline indicators (>, |, >-, |-)
|
|
35
|
-
if value in (">", "|", ">-", "|-"):
|
|
36
|
-
continuation_lines: list[str] = []
|
|
37
|
-
i += 1
|
|
38
|
-
while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith(" ") or frontmatter_lines[i].startswith("\t")):
|
|
39
|
-
continuation_lines.append(frontmatter_lines[i].strip())
|
|
40
|
-
i += 1
|
|
41
|
-
description = " ".join(continuation_lines)
|
|
42
|
-
continue
|
|
43
|
-
else:
|
|
44
|
-
description = value.strip('"').strip("'")
|
|
45
|
-
i += 1
|
|
46
|
-
|
|
47
|
-
return name, description, content
|
|
@@ -1,36 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: tdd
|
|
3
|
-
description: Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Test-Driven Development
|
|
7
|
-
|
|
8
|
-
TDD is the red → green loop. This skill is the reference that makes that loop produce tests worth keeping: what a good test is, where tests go, the anti-patterns, and the rules of the loop. Every section applies on every cycle — consult them before and during the loop, not after.
|
|
9
|
-
|
|
10
|
-
When exploring the codebase, read `CONTEXT.md` (if it exists) so test names and interface vocabulary match the project's domain language, and respect ADRs in the area you're touching.
|
|
11
|
-
|
|
12
|
-
## What a good test is
|
|
13
|
-
|
|
14
|
-
Tests verify behavior through public interfaces, not implementation details. Code can change entirely; tests shouldn't. A good test reads like a specification — "user can checkout with valid cart" tells you exactly what capability exists — and survives refactors because it doesn't care about internal structure.
|
|
15
|
-
|
|
16
|
-
See [tests.md](tests.md) for examples and [mocking.md](mocking.md) for mocking guidelines.
|
|
17
|
-
|
|
18
|
-
## Seams — where tests go
|
|
19
|
-
|
|
20
|
-
A **seam** is the public boundary you test at: the interface where you observe behavior without reaching inside. Tests live at seams, never against internals.
|
|
21
|
-
|
|
22
|
-
**Test only at pre-agreed seams.** Before writing any test, write down the seams under test and confirm them with the user. No test is written at an unconfirmed seam. You can't test everything — agreeing the seams up front is how testing effort lands on the critical paths and complex logic instead of every edge case.
|
|
23
|
-
|
|
24
|
-
Ask: "What's the public interface, and which seams should we test?"
|
|
25
|
-
|
|
26
|
-
## Anti-patterns
|
|
27
|
-
|
|
28
|
-
- **Implementation-coupled** — mocks internal collaborators, tests private methods, or verifies through a side channel (querying the database instead of using the interface). The tell: the test breaks when you refactor but behavior hasn't changed.
|
|
29
|
-
- **Tautological** — the assertion recomputes the expected value the way the code does (`expect(add(a, b)).toBe(a + b)`, a snapshot derived by hand the same way, a constant asserted equal to itself), so it passes by construction and can never disagree with the code. Expected values must come from an independent source of truth — a known-good literal, a worked example, the spec.
|
|
30
|
-
- **Horizontal slicing** — writing all tests first, then all implementation. Bulk tests verify _imagined_ behavior: you test the _shape_ of things rather than user-facing behavior, the tests go insensitive to real changes, and you commit to test structure before understanding the implementation. Work in **vertical slices** instead — one test → one implementation → repeat, each test a **tracer bullet** that responds to what the last cycle taught you.
|
|
31
|
-
|
|
32
|
-
## Rules of the loop
|
|
33
|
-
|
|
34
|
-
- **Red before green.** Write the failing test first, then only enough code to pass it. Don't anticipate future tests or add speculative features.
|
|
35
|
-
- **One slice at a time.** One seam, one test, one minimal implementation per cycle.
|
|
36
|
-
- **Refactoring is not part of the loop.** It belongs to the review stage (see the `code-review` skill), not the red → green implementation cycle.
|
|
@@ -1,59 +0,0 @@
|
|
|
1
|
-
# When to Mock
|
|
2
|
-
|
|
3
|
-
Mock at **system boundaries** only:
|
|
4
|
-
|
|
5
|
-
- External APIs (payment, email, etc.)
|
|
6
|
-
- Databases (sometimes - prefer test DB)
|
|
7
|
-
- Time/randomness
|
|
8
|
-
- File system (sometimes)
|
|
9
|
-
|
|
10
|
-
Don't mock:
|
|
11
|
-
|
|
12
|
-
- Your own classes/modules
|
|
13
|
-
- Internal collaborators
|
|
14
|
-
- Anything you control
|
|
15
|
-
|
|
16
|
-
## Designing for Mockability
|
|
17
|
-
|
|
18
|
-
At system boundaries, design interfaces that are easy to mock:
|
|
19
|
-
|
|
20
|
-
**1. Use dependency injection**
|
|
21
|
-
|
|
22
|
-
Pass external dependencies in rather than creating them internally:
|
|
23
|
-
|
|
24
|
-
```typescript
|
|
25
|
-
// Easy to mock
|
|
26
|
-
function processPayment(order, paymentClient) {
|
|
27
|
-
return paymentClient.charge(order.total);
|
|
28
|
-
}
|
|
29
|
-
|
|
30
|
-
// Hard to mock
|
|
31
|
-
function processPayment(order) {
|
|
32
|
-
const client = new StripeClient(process.env.STRIPE_KEY);
|
|
33
|
-
return client.charge(order.total);
|
|
34
|
-
}
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
**2. Prefer SDK-style interfaces over generic fetchers**
|
|
38
|
-
|
|
39
|
-
Create specific functions for each external operation instead of one generic function with conditional logic:
|
|
40
|
-
|
|
41
|
-
```typescript
|
|
42
|
-
// GOOD: Each function is independently mockable
|
|
43
|
-
const api = {
|
|
44
|
-
getUser: (id) => fetch(`/users/${id}`),
|
|
45
|
-
getOrders: (userId) => fetch(`/users/${userId}/orders`),
|
|
46
|
-
createOrder: (data) => fetch('/orders', { method: 'POST', body: data }),
|
|
47
|
-
};
|
|
48
|
-
|
|
49
|
-
// BAD: Mocking requires conditional logic inside the mock
|
|
50
|
-
const api = {
|
|
51
|
-
fetch: (endpoint, options) => fetch(endpoint, options),
|
|
52
|
-
};
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
The SDK approach means:
|
|
56
|
-
- Each mock returns one specific shape
|
|
57
|
-
- No conditional logic in test setup
|
|
58
|
-
- Easier to see which endpoints a test exercises
|
|
59
|
-
- Type safety per endpoint
|
|
@@ -1,77 +0,0 @@
|
|
|
1
|
-
# Good and Bad Tests
|
|
2
|
-
|
|
3
|
-
## Good Tests
|
|
4
|
-
|
|
5
|
-
**Integration-style**: Test through real interfaces, not mocks of internal parts.
|
|
6
|
-
|
|
7
|
-
```typescript
|
|
8
|
-
// GOOD: Tests observable behavior
|
|
9
|
-
test("user can checkout with valid cart", async () => {
|
|
10
|
-
const cart = createCart();
|
|
11
|
-
cart.add(product);
|
|
12
|
-
const result = await checkout(cart, paymentMethod);
|
|
13
|
-
expect(result.status).toBe("confirmed");
|
|
14
|
-
});
|
|
15
|
-
```
|
|
16
|
-
|
|
17
|
-
Characteristics:
|
|
18
|
-
|
|
19
|
-
- Tests behavior users/callers care about
|
|
20
|
-
- Uses public API only
|
|
21
|
-
- Survives internal refactors
|
|
22
|
-
- Describes WHAT, not HOW
|
|
23
|
-
- One logical assertion per test
|
|
24
|
-
|
|
25
|
-
## Bad Tests
|
|
26
|
-
|
|
27
|
-
**Implementation-detail tests**: Coupled to internal structure.
|
|
28
|
-
|
|
29
|
-
```typescript
|
|
30
|
-
// BAD: Tests implementation details
|
|
31
|
-
test("checkout calls paymentService.process", async () => {
|
|
32
|
-
const mockPayment = jest.mock(paymentService);
|
|
33
|
-
await checkout(cart, payment);
|
|
34
|
-
expect(mockPayment.process).toHaveBeenCalledWith(cart.total);
|
|
35
|
-
});
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
Red flags:
|
|
39
|
-
|
|
40
|
-
- Mocking internal collaborators
|
|
41
|
-
- Testing private methods
|
|
42
|
-
- Asserting on call counts/order
|
|
43
|
-
- Test breaks when refactoring without behavior change
|
|
44
|
-
- Test name describes HOW not WHAT
|
|
45
|
-
- Verifying through external means instead of interface
|
|
46
|
-
|
|
47
|
-
```typescript
|
|
48
|
-
// BAD: Bypasses interface to verify
|
|
49
|
-
test("createUser saves to database", async () => {
|
|
50
|
-
await createUser({ name: "Alice" });
|
|
51
|
-
const row = await db.query("SELECT * FROM users WHERE name = ?", ["Alice"]);
|
|
52
|
-
expect(row).toBeDefined();
|
|
53
|
-
});
|
|
54
|
-
|
|
55
|
-
// GOOD: Verifies through interface
|
|
56
|
-
test("createUser makes user retrievable", async () => {
|
|
57
|
-
const user = await createUser({ name: "Alice" });
|
|
58
|
-
const retrieved = await getUser(user.id);
|
|
59
|
-
expect(retrieved.name).toBe("Alice");
|
|
60
|
-
});
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
**Tautological tests**: Expected value restates the implementation, so the test passes by construction.
|
|
64
|
-
|
|
65
|
-
```typescript
|
|
66
|
-
// BAD: Expected value is recomputed the way the code computes it
|
|
67
|
-
test("calculateTotal sums line items", () => {
|
|
68
|
-
const items = [{ price: 10 }, { price: 5 }];
|
|
69
|
-
const expected = items.reduce((sum, i) => sum + i.price, 0);
|
|
70
|
-
expect(calculateTotal(items)).toBe(expected);
|
|
71
|
-
});
|
|
72
|
-
|
|
73
|
-
// GOOD: Expected value is an independent, known literal
|
|
74
|
-
test("calculateTotal sums line items", () => {
|
|
75
|
-
expect(calculateTotal([{ price: 10 }, { price: 5 }])).toBe(15);
|
|
76
|
-
});
|
|
77
|
-
```
|
|
@@ -1,35 +0,0 @@
|
|
|
1
|
-
# GLOSSARY.md Format
|
|
2
|
-
|
|
3
|
-
`GLOSSARY.md` is the canonical language for this teaching workspace. All explainers, exercises, and learning records should adhere to its terminology. Building it is itself part of learning: compressing a concept into a tight definition is evidence the user understands it.
|
|
4
|
-
|
|
5
|
-
## Structure
|
|
6
|
-
|
|
7
|
-
```md
|
|
8
|
-
# {Topic} Glossary
|
|
9
|
-
|
|
10
|
-
{One or two sentence description of the topic this glossary covers.}
|
|
11
|
-
|
|
12
|
-
## Terms
|
|
13
|
-
|
|
14
|
-
**Hypertrophy**:
|
|
15
|
-
Muscle growth driven by mechanical tension and metabolic stress over repeated training sessions.
|
|
16
|
-
_Avoid_: Bulking, getting big
|
|
17
|
-
|
|
18
|
-
**Progressive overload**:
|
|
19
|
-
Systematically increasing the demand on a muscle over time — via load, volume, or intensity.
|
|
20
|
-
_Avoid_: Pushing harder, levelling up
|
|
21
|
-
|
|
22
|
-
**RPE (Rate of Perceived Exertion)**:
|
|
23
|
-
A 1–10 self-rating of how hard a set felt, where 10 is failure and 8 means two reps left in the tank.
|
|
24
|
-
_Avoid_: Effort score, intensity rating
|
|
25
|
-
```
|
|
26
|
-
|
|
27
|
-
## Rules
|
|
28
|
-
|
|
29
|
-
- **Add a term only when the user understands it.** The glossary is a record of compressed knowledge, not a dictionary the user reads to learn. If the user has just been introduced to a concept, wait until they can use it correctly before promoting it here.
|
|
30
|
-
- **Be opinionated.** When several words exist for the same concept, pick the best one and list the rest as aliases to avoid. This is how language compresses.
|
|
31
|
-
- **Keep definitions tight.** One or two sentences. Define what the term IS, not what it does or how to do it.
|
|
32
|
-
- **Use the glossary's own terms inside definitions.** Once a term is in the glossary, prefer it everywhere — including inside other definitions. This is what makes complex terms easier to grasp later.
|
|
33
|
-
- **Group under subheadings** when natural clusters emerge (e.g. `## Anatomy`, `## Programming`). A flat list is fine when terms cohere.
|
|
34
|
-
- **Flag ambiguities explicitly.** If a term is used loosely in the wider field, note the resolution: "In this workspace, 'set' always means a working set — warm-ups are tracked separately."
|
|
35
|
-
- **Revise as understanding deepens.** A definition the user wrote in week one may be wrong by week six. Update in place; do not leave stale entries.
|
|
@@ -1,46 +0,0 @@
|
|
|
1
|
-
# Learning Record Format
|
|
2
|
-
|
|
3
|
-
Learning records live in `./learning-records/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. Create the directory lazily — only when the first record is written.
|
|
4
|
-
|
|
5
|
-
They are the teaching equivalent of ADRs: they capture non-obvious lessons, key insights, and stated prior knowledge that will steer future sessions. They are used to calculate the zone of proximal development.
|
|
6
|
-
|
|
7
|
-
## Template
|
|
8
|
-
|
|
9
|
-
```md
|
|
10
|
-
# {Short title of what was learned or established}
|
|
11
|
-
|
|
12
|
-
{1-3 sentences: what was learned (or what prior knowledge was established), and why it matters for future sessions.}
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
That is the whole format. A learning record can be a single paragraph. The value is recording _that_ this is now known and _why_ it changes what to teach next — not in filling out sections.
|
|
16
|
-
|
|
17
|
-
## Optional sections
|
|
18
|
-
|
|
19
|
-
Only include these when they add genuine value. Most records won't need them.
|
|
20
|
-
|
|
21
|
-
- **Status** frontmatter (`active | superseded by LR-NNNN`) — useful when an earlier understanding turns out to be wrong and is replaced.
|
|
22
|
-
- **Evidence** — how the user demonstrated the understanding (a question answered, an exercise completed, prior experience cited). Useful when the claim might be revisited.
|
|
23
|
-
- **Implications** — what this unlocks or rules out for future sessions. Worth recording when non-obvious.
|
|
24
|
-
|
|
25
|
-
## Numbering
|
|
26
|
-
|
|
27
|
-
Scan `./learning-records/` for the highest existing number and increment by one.
|
|
28
|
-
|
|
29
|
-
## When to write a learning record
|
|
30
|
-
|
|
31
|
-
Write one when any of these is true:
|
|
32
|
-
|
|
33
|
-
1. **The user demonstrated genuine understanding of something non-trivial** — not just exposure, but evidence they can use the concept correctly. This sets a new floor for what to teach next.
|
|
34
|
-
2. **The user disclosed prior knowledge** — "I already know X." Record it so future sessions don't re-teach it. Also record the _depth_ claimed.
|
|
35
|
-
3. **A misconception was corrected** — the user previously believed something wrong and now sees why. These are high-value: they predict future stumbling blocks for related topics.
|
|
36
|
-
4. **The mission shifted in response to learning** — the user discovered they cared about something different than they thought. Cross-link to [[MISSION.md]] and update it.
|
|
37
|
-
|
|
38
|
-
### What does _not_ qualify
|
|
39
|
-
|
|
40
|
-
- Material that was merely covered. Coverage is not learning. Wait for evidence.
|
|
41
|
-
- Anything already captured tersely in [[GLOSSARY.md]] as a term definition. Don't duplicate.
|
|
42
|
-
- Session-by-session activity logs. Learning records are not a journal — they are decision-grade insights.
|
|
43
|
-
|
|
44
|
-
## Supersession
|
|
45
|
-
|
|
46
|
-
When a later record contradicts an earlier one (the user's understanding deepened or corrected), mark the old record `Status: superseded by LR-NNNN` rather than deleting it. The history of how understanding evolved is itself useful signal.
|
|
@@ -1,31 +0,0 @@
|
|
|
1
|
-
# MISSION.md Format
|
|
2
|
-
|
|
3
|
-
`MISSION.md` lives at the workspace root. It captures the _reason_ the user is learning this topic. Every teaching decision — what to teach next, which resources to surface, which exercises to design — should trace back to this document.
|
|
4
|
-
|
|
5
|
-
## Template
|
|
6
|
-
|
|
7
|
-
```md
|
|
8
|
-
# Mission: {Topic}
|
|
9
|
-
|
|
10
|
-
## Why
|
|
11
|
-
{1-3 sentences. The concrete real-world goal the user is chasing. What changes in their life or work when they have this skill? Avoid abstract framings like "to understand X" — push for the underlying outcome.}
|
|
12
|
-
|
|
13
|
-
## Success looks like
|
|
14
|
-
- {A specific, observable thing the user will be able to do}
|
|
15
|
-
- {Another specific thing}
|
|
16
|
-
- {…}
|
|
17
|
-
|
|
18
|
-
## Constraints
|
|
19
|
-
- {Time, budget, prior commitments, learning preferences, anything that bounds the approach}
|
|
20
|
-
|
|
21
|
-
## Out of scope
|
|
22
|
-
- {Adjacent topics the user explicitly does not want to chase right now — protects the zone of proximal development}
|
|
23
|
-
```
|
|
24
|
-
|
|
25
|
-
## Rules
|
|
26
|
-
|
|
27
|
-
- **One mission per workspace.** If the user wants to learn two unrelated things, that is two workspaces.
|
|
28
|
-
- **Concrete over abstract.** "Run a half marathon by October" beats "get fitter." "Ship a Rust CLI to my team" beats "learn Rust."
|
|
29
|
-
- **Push back on vagueness.** If the user cannot articulate why, interview them before writing anything. A bad mission is worse than no mission.
|
|
30
|
-
- **Revise when reality shifts.** Missions change. When the user's goal moves, update this file — don't leave a stale mission steering future sessions.
|
|
31
|
-
- **Keep it short.** If `MISSION.md` runs past a screen, it has stopped being a compass and started being a plan.
|