@benjaminthomas/spec-driven-dev 1.10.0 → 1.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +78 -8
- package/bin/cli.mjs +131 -10
- package/bin/risk-eval.mjs +152 -0
- package/bin/risk.mjs +137 -0
- package/package.json +3 -2
- package/plugin/.claude-plugin/icon.png +0 -0
- package/plugin/.claude-plugin/plugin.json +20 -0
- package/plugin/skills/spec-checkpoint/SKILL.md +3 -1
- package/plugin/skills/spec-create/SKILL.md +12 -0
- package/plugin/skills/spec-create/references/task-template.md +6 -0
- package/plugin/skills/spec-implement/references/review-prompt-template.md +2 -1
- package/plugin/skills/spec-investigate/SKILL.md +16 -0
- package/plugin/skills/spec-investigate/references/report-template.md +5 -0
- package/plugin/skills/spec-review/SKILL.md +5 -1
- package/plugin/skills/spec-setup/SKILL.md +23 -3
- package/plugin/skills/spec-setup/references/config-template.yml +12 -0
package/README.md
CHANGED
|
@@ -46,6 +46,7 @@ Asking an AI agent to "just build it" tends to fail in predictable ways:
|
|
|
46
46
|
| One agent working alone is slow | **`spec-implement`** runs several coder agents in parallel, each in its own copy of the repo |
|
|
47
47
|
| "It compiles" gets treated as "it works" | **`spec-verify`** drives the real running app and checks every acceptance criterion, with evidence |
|
|
48
48
|
| Problems slip through because nobody reviews the whole change | **`spec-review`** checks the finished branch against the spec, your conventions, and the code, and blocks shipping on real problems |
|
|
49
|
+
| The agent treats a file that broke five times like any other | **GitRisk** scores files from git history (churn, bug fixes, dependents, security fixes), so risky files get a deeper look, a higher tier and targeted tests. Switch it on or off any time |
|
|
49
50
|
| A new session forgets what was decided last time | **Project memory**: a small `specs/CONTEXT.md`, plus a "Resume here" block and a log in every spec, kept short so reloading them is cheap |
|
|
50
51
|
| Broken code gets committed, or a PR is pushed by accident | **`spec-checkpoint`** commits only once lint, types, tests, and build all pass. **`spec-ship`** asks before it pushes |
|
|
51
52
|
|
|
@@ -246,6 +247,54 @@ lines that point to files that no longer exist. If you install
|
|
|
246
247
|
[graphify](https://github.com/Graphify-Labs/graphify), the skills use it for cheaper code
|
|
247
248
|
navigation. It's optional (`context.graph: off` turns it off), and nothing is installed for you.
|
|
248
249
|
|
|
250
|
+
### GitRisk: which files have a history of breaking
|
|
251
|
+
|
|
252
|
+
`spec-driven-dev risk` scores every tracked code file from git history alone, with no model and no
|
|
253
|
+
tokens, and writes the top files to `specs/context/risk.md`:
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
Risk = 0.30 × Churn + 0.35 × Bug Density + 0.20 × Fan-out + 0.15 × Security (each factor 0–100)
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
| Factor | Weight | What it measures |
|
|
260
|
+
|---|---|---|
|
|
261
|
+
| Churn | 30% | how often the file changes, against the repo's 90th percentile |
|
|
262
|
+
| Bug Density | 35% | share of its commits that were bug fixes (`fix:` type or fix-like wording) |
|
|
263
|
+
| Fan-out | 20% | how many other files import it, against the repo's 90th percentile |
|
|
264
|
+
| Security | 15% | share of its commits that were security fixes (security, CVE, XSS, injection…) |
|
|
265
|
+
|
|
266
|
+
Tiers: critical ≥ 75, high ≥ 50, medium ≥ 25. When `risk.enabled` is on, the skills use it:
|
|
267
|
+
- `spec-investigate` digs into high and critical files in the change surface (past fix commits,
|
|
268
|
+
dependents) and lists them as **Risk Hotspots**;
|
|
269
|
+
- `spec-create` raises those tasks one tier and requires a test aimed at each file's past failure
|
|
270
|
+
pattern;
|
|
271
|
+
- `spec-implement`'s reviewer and `spec-review` check those tests and read the risky files line
|
|
272
|
+
by line.
|
|
273
|
+
|
|
274
|
+
**Turning it on or off, no YAML needed:**
|
|
275
|
+
- **In your AI tool:** say "turn GitRisk off" or "turn GitRisk on". `spec-setup` changes just that
|
|
276
|
+
setting and tells you what it means.
|
|
277
|
+
- **In a terminal:** run `spec-driven-dev settings` and pick with the arrow keys. It also switches
|
|
278
|
+
run-through mode. For scripts, use `spec-driven-dev settings --risk=off`.
|
|
279
|
+
- **By hand:** set `risk.enabled: on` or `off` in `.spec-driven.yml`.
|
|
280
|
+
|
|
281
|
+
Off means every skill ignores risk completely.
|
|
282
|
+
|
|
283
|
+
`spec-driven-dev risk --files=a.ts,b.ts` prints scores for chosen files. The weights are a starting
|
|
284
|
+
hypothesis: change them under `risk.weights` in `.spec-driven.yml` once you've backtested them
|
|
285
|
+
against your own past bugs.
|
|
286
|
+
|
|
287
|
+
**Does it change what the agent does?** `spec-driven-dev risk-eval --tickets=tickets.md` runs each
|
|
288
|
+
ticket twice, in fresh clones with risk on and off, through investigate and create on the same
|
|
289
|
+
model. It then compares what the agent did:
|
|
290
|
+
- risky files read and named in the investigation;
|
|
291
|
+
- risky files in the plan, and whether tests were planned for them;
|
|
292
|
+
- task tiers;
|
|
293
|
+
- cost.
|
|
294
|
+
|
|
295
|
+
Results go to `experiments/` with the transcripts, and every run, including failures, is appended to
|
|
296
|
+
`experiments/LOG.md`. It needs Claude Code.
|
|
297
|
+
|
|
249
298
|
## How the workflow fits together
|
|
250
299
|
|
|
251
300
|
```
|
|
@@ -314,6 +363,9 @@ It starts from the project memory (`specs/CONTEXT.md` and only the area files th
|
|
|
314
363
|
treats it as a lead rather than evidence, and notes where the code contradicts it. If graphify is
|
|
315
364
|
installed, it uses it to map the code more cheaply.
|
|
316
365
|
|
|
366
|
+
With GitRisk on, it also looks harder at high-risk files in the change surface (their past fix
|
|
367
|
+
commits and dependents) and lists them as **Risk Hotspots**.
|
|
368
|
+
|
|
317
369
|
**You get:** `specs/<feature>/investigation.md`, with every finding labelled **VERIFIED**,
|
|
318
370
|
**INFERRED**, **ASSUMPTION**, or **UNKNOWN**, a short **Relevant Context** digest for the next
|
|
319
371
|
stages, and requirements numbered R1, R2… so they can be traced through to verification and review.
|
|
@@ -335,6 +387,9 @@ to write**, acceptance criteria, the requirements it **covers**, and the few **p
|
|
|
335
387
|
that apply to it. Every requirement must map to a task, and the spec isn't finished until it does.
|
|
336
388
|
It also starts the spec's `log.md` and **Resume here** block.
|
|
337
389
|
|
|
390
|
+
With GitRisk on, a task that touches a high-risk file goes up one tier and must include a test
|
|
391
|
+
aimed at that file's past failures.
|
|
392
|
+
|
|
338
393
|
**Then:** read it, edit anything you disagree with, and only then run `spec-implement`.
|
|
339
394
|
|
|
340
395
|
### 3. `spec-implement`: build it, batch by batch
|
|
@@ -380,6 +435,8 @@ nothing out of scope), the project's recorded conventions, and the code itself:
|
|
|
380
435
|
security, error handling, tests, and hot-path performance. It checks each finding against the code
|
|
381
436
|
before reporting it, and reads the diff rather than the whole repo.
|
|
382
437
|
|
|
438
|
+
With GitRisk on, high-risk files get a line-by-line read of their error paths and callers.
|
|
439
|
+
|
|
383
440
|
**You get:** `specs/<feature>/review.md`, with findings ranked blocker, should-fix, nit, or question,
|
|
384
441
|
each with `file:line` and a suggested fix. Blockers stop `spec-ship` and run-through. It then asks
|
|
385
442
|
whether to fix them, and fixes go through `spec-implement`'s fix loop only after you say yes. It also
|
|
@@ -443,6 +500,8 @@ specs/
|
|
|
443
500
|
└── review.md ← spec-review: ranked findings; open blockers stop spec-ship
|
|
444
501
|
specs/CONTEXT.md ← spec-setup / spec-checkpoint: project memory index (≤60 lines)
|
|
445
502
|
specs/context/<area>.md ← details per area, loaded only when needed (≤150 lines)
|
|
503
|
+
specs/context/risk.md ← spec-driven-dev risk: GitRisk scores (regenerate; don't edit)
|
|
504
|
+
experiments/ ← spec-driven-dev risk-eval: on/off reports, transcripts, LOG.md
|
|
446
505
|
```
|
|
447
506
|
|
|
448
507
|
These are plain Markdown files, so you can review them in a PR, edit them, or hand them to a
|
|
@@ -475,6 +534,11 @@ Built into the skills, not left to the agent's judgment:
|
|
|
475
534
|
|
|
476
535
|
## FAQ
|
|
477
536
|
|
|
537
|
+
**How do I turn GitRisk off (or on)?**
|
|
538
|
+
Say "turn GitRisk off" in your AI tool, or run `spec-driven-dev settings` and pick from the menu.
|
|
539
|
+
Off means every skill ignores risk scores. Turning it on again offers to score the files if that
|
|
540
|
+
hasn't been done yet.
|
|
541
|
+
|
|
478
542
|
**Do I need Jira?**
|
|
479
543
|
No. Use GitHub Issues, Linear, a transcript, a document, or a sentence of plain text. Jira is just
|
|
480
544
|
one of the options.
|
|
@@ -565,6 +629,10 @@ spec-driven-dev --yes # accept every detected defaul
|
|
|
565
629
|
spec-driven-dev doctor # check .spec-driven.yml against the real repos
|
|
566
630
|
spec-driven-dev status # list unfinished specs and each one's next step
|
|
567
631
|
spec-driven-dev context --areas=billing # print the project memory (index, or chosen areas)
|
|
632
|
+
spec-driven-dev settings # switch GitRisk and run-through on/off from a menu
|
|
633
|
+
spec-driven-dev risk # GitRisk: score files from git history → specs/context/risk.md
|
|
634
|
+
spec-driven-dev risk-eval --tickets=tickets.md # compare agent behaviour with the risk signal on vs off
|
|
635
|
+
# (--runs=3 --arms=on --timeout=20 --model=sonnet)
|
|
568
636
|
spec-driven-dev --help # show all of the above
|
|
569
637
|
```
|
|
570
638
|
|
|
@@ -658,16 +726,16 @@ skips that step rather than failing.
|
|
|
658
726
|
|
|
659
727
|
| Skill | Stage | SKILL.md lines | Words | Supporting files |
|
|
660
728
|
|---|---|---:|---:|---:|
|
|
661
|
-
| `spec-setup` | 0 · configure |
|
|
662
|
-
| `spec-investigate` | 1 · understand |
|
|
663
|
-
| `spec-create` | 2 · plan |
|
|
729
|
+
| `spec-setup` | 0 · configure | 162 | 1,534 | 3 |
|
|
730
|
+
| `spec-investigate` | 1 · understand | 370 | 3,152 | 1 |
|
|
731
|
+
| `spec-create` | 2 · plan | 195 | 1,746 | 6 |
|
|
664
732
|
| `spec-implement` | 3 · build | 322 | 2,683 | 3 |
|
|
665
733
|
| `spec-verify` | 4 · prove | 163 | 1,557 | 1 |
|
|
666
|
-
| `spec-review` | 5 · review |
|
|
667
|
-
| `spec-checkpoint` | 6 · commit |
|
|
734
|
+
| `spec-review` | 5 · review | 135 | 1,270 | 1 |
|
|
735
|
+
| `spec-checkpoint` | 6 · commit | 133 | 1,046 | 0 |
|
|
668
736
|
| `spec-ship` | 7 · PR | 230 | 1,388 | 0 |
|
|
669
737
|
| `generative-seo` | standalone | 255 | 2,209 | 7 |
|
|
670
|
-
| **Total** | | **1,
|
|
738
|
+
| **Total** | | **1,965** | **16,585** | **22** |
|
|
671
739
|
|
|
672
740
|
- **8** skills: **1** setup + **6** workflow stages + **1** standalone
|
|
673
741
|
- **4** tools supported natively (Claude Code, Codex CLI, Cursor, Antigravity), plus an MCP fallback
|
|
@@ -688,7 +756,9 @@ plugin/skills/<name>/references/ templates and prompts the skill loads when need
|
|
|
688
756
|
plugin/ the Claude Code plugin (manifest + skills only, no scripts or hooks)
|
|
689
757
|
.agents/skills/ symlinks so Codex/Antigravity pick the skills up inside this repo
|
|
690
758
|
bin/spec-driven-dev.cjs launcher (runs on any Node, explains if it's too old)
|
|
691
|
-
bin/cli.mjs the `spec-driven-dev` CLI: installer, setup menus, doctor, status, context
|
|
759
|
+
bin/cli.mjs the `spec-driven-dev` CLI: installer, setup menus, doctor, status, context, risk
|
|
760
|
+
bin/risk.mjs GitRisk scoring (churn, bug density, fan-out, security)
|
|
761
|
+
bin/risk-eval.mjs risk on/off experiment harness
|
|
692
762
|
package.json npm package (ships bin/ and plugin/skills/)
|
|
693
763
|
mcp-server/ MCP fallback server
|
|
694
764
|
CHANGELOG.md what changed in each version
|
|
@@ -717,7 +787,7 @@ plugin/evals/ `claude plugin eval` cases
|
|
|
717
787
|
1. Increase `version` in both `package.json` and `plugin/.claude-plugin/plugin.json` (the check fails if
|
|
718
788
|
they differ).
|
|
719
789
|
2. Add a `CHANGELOG.md` entry, then commit and push.
|
|
720
|
-
3. Tag the release and push the tag: `git tag v1.
|
|
790
|
+
3. Tag the release and push the tag: `git tag v1.11.0 && git push origin v1.11.0`.
|
|
721
791
|
4. Watch the **publish** run in the repo's Actions tab. When it's green, the new version is on
|
|
722
792
|
npm. Claude Code plugin users get it from the marketplace once the commit is on `main`.
|
|
723
793
|
|
package/bin/cli.mjs
CHANGED
|
@@ -7,15 +7,19 @@
|
|
|
7
7
|
// spec-driven-dev doctor check .spec-driven.yml against the real repos (exit 1 on problems)
|
|
8
8
|
// spec-driven-dev status list specs with unfinished work and each one's "Resume here" next step
|
|
9
9
|
// spec-driven-dev context print specs/CONTEXT.md; --areas=billing,api prints only those area files
|
|
10
|
+
// spec-driven-dev risk GitRisk: score files from git history into specs/context/risk.md (--files=a,b --json --top=N --since=…)
|
|
11
|
+
// spec-driven-dev settings switch features on or off from a menu (GitRisk, run-through); or --risk=on|off --after-approval=…
|
|
12
|
+
// spec-driven-dev risk-eval U8 experiment: run tickets with risk on vs off and compare agent behaviour (--tickets=<file> --runs=N --model=… --arms=on,off --timeout=20)
|
|
10
13
|
// flags: --yes --global --tools=claude,cursor --skills=spec-create,... --no-setup --dir=<path>
|
|
11
14
|
// --layout=single|monorepo|multi-repo --repos=~/Projects/frontend,/srv/backend --workspace=<config dir>
|
|
12
|
-
// --packages=apps/web,packages/ui --after-approval=stop|run-through --project=new|existing --version
|
|
15
|
+
// --packages=apps/web,packages/ui --after-approval=stop|run-through --project=new|existing --risk=on|off --version
|
|
13
16
|
import fs from "node:fs";
|
|
14
17
|
import os from "node:os";
|
|
15
18
|
import path from "node:path";
|
|
16
19
|
import { execFileSync } from "node:child_process";
|
|
17
20
|
import prompts from "prompts"; // arrow-key menus that work in PowerShell, cmd, macOS and Linux terminals
|
|
18
21
|
import { fileURLToPath } from "node:url";
|
|
22
|
+
import { scoreRepo, riskMarkdown, normWeights, weightProblems, setConfigValue } from "./risk.mjs";
|
|
19
23
|
|
|
20
24
|
const ROOT = path.dirname(path.dirname(fileURLToPath(import.meta.url)));
|
|
21
25
|
const SKILLS_DIR = path.join(ROOT, "plugin", "skills");
|
|
@@ -34,7 +38,7 @@ const args = process.argv.slice(2);
|
|
|
34
38
|
const flag = (n) => args.includes(`--${n}`);
|
|
35
39
|
const opt = (n) => args.find((a) => a.startsWith(`--${n}=`))?.split("=")[1];
|
|
36
40
|
const command = args.find((a) => !a.startsWith("-")) || "install";
|
|
37
|
-
const YES = flag("yes") || flag("y") || !process.stdin.isTTY || ["doctor", "status", "context"].includes(command);
|
|
41
|
+
const YES = flag("yes") || flag("y") || !process.stdin.isTTY || ["doctor", "status", "context", "risk", "risk-eval"].includes(command);
|
|
38
42
|
const DIR = path.resolve(opt("dir") || process.cwd());
|
|
39
43
|
|
|
40
44
|
if (flag("version") || flag("v")) {
|
|
@@ -43,22 +47,23 @@ if (flag("version") || flag("v")) {
|
|
|
43
47
|
}
|
|
44
48
|
// No terminal to answer in (e.g. `!` in Claude Code, an IDE task, a pipe) and nothing passed explicitly:
|
|
45
49
|
// stop rather than write a config from guesses.
|
|
46
|
-
const ANSWER_FLAGS = ["yes", "y", "layout", "repos", "packages", "tools", "workspace", "after-approval", "skills", "global", "no-setup", "project"];
|
|
47
|
-
if (["install", "setup"].includes(command) && !process.stdin.isTTY && !args.some((a) => ANSWER_FLAGS.some((f) => a === `--${f}` || a.startsWith(`--${f}=`)))) {
|
|
48
|
-
console.log(`✗ No interactive terminal, so setup can't ask its questions. Nothing was changed.
|
|
50
|
+
const ANSWER_FLAGS = ["risk", "yes", "y", "layout", "repos", "packages", "tools", "workspace", "after-approval", "skills", "global", "no-setup", "project"];
|
|
51
|
+
if (["install", "setup", "settings"].includes(command) && !process.stdin.isTTY && !args.some((a) => ANSWER_FLAGS.some((f) => a === `--${f}` || a.startsWith(`--${f}=`)))) {
|
|
52
|
+
console.log(`✗ No interactive terminal, so ${command === "settings" ? "the settings menu can't open" : "setup can't ask its questions"}. Nothing was changed.
|
|
49
53
|
|
|
50
54
|
This happens when it runs through another program: a "!" command in Claude Code, an editor task,
|
|
51
55
|
or piped input. Either:
|
|
52
56
|
• run it yourself in a terminal window (Terminal, iTerm, or your editor's terminal panel):
|
|
53
|
-
npx @benjaminthomas/spec-driven-dev@latest ${command === "
|
|
57
|
+
npx @benjaminthomas/spec-driven-dev@latest ${command === "install" ? "" : command}
|
|
54
58
|
• or give the answers as flags, for example:
|
|
55
|
-
npx @benjaminthomas/spec-driven-dev@latest setup --layout=multi-repo --repos=~/Projects/frontend,~/Projects/backend
|
|
59
|
+
npx @benjaminthomas/spec-driven-dev@latest ${command === "settings" ? "settings --risk=off" : "setup --layout=multi-repo --repos=~/Projects/frontend,~/Projects/backend"}
|
|
56
60
|
(add --yes to accept everything it detects)`);
|
|
57
61
|
process.exit(1);
|
|
58
62
|
}
|
|
59
63
|
|
|
60
64
|
if (flag("help") || flag("h")) {
|
|
61
|
-
|
|
65
|
+
const lines = fs.readFileSync(fileURLToPath(import.meta.url), "utf8").split("\n").slice(1);
|
|
66
|
+
console.log(lines.slice(0, lines.findIndex((l) => !l.startsWith("//"))).join("\n").replace(/^\/\/ ?/gm, "")); // the whole header comment
|
|
62
67
|
process.exit(0);
|
|
63
68
|
}
|
|
64
69
|
|
|
@@ -241,6 +246,7 @@ function toYaml(c) {
|
|
|
241
246
|
out += "effort: # effort / reasoning level per tier (Claude Code, Codex): low | medium | high | xhigh | max | default\n"
|
|
242
247
|
+ MODEL_KEYS.map((t) => ` ${t}: ${EFFORT_DEFAULTS[t]}\n`).join("");
|
|
243
248
|
out += `context: # project memory: {specs_dir}/CONTEXT.md (index) + context/<area>.md\n project: ${c.project || "existing"} # existing | new\n graph: auto # auto = use graphify if installed | off\n`;
|
|
249
|
+
out += "risk: # GitRisk: historical risk per file from git (spec-driven-dev risk). Weights are a hypothesis to calibrate\n enabled: on # on | off (off = skills ignore risk; risk-eval compares the two)\n since: \"2 years ago\"\n weights:\n churn: 0.30\n bug_density: 0.35\n fan_out: 0.20\n security: 0.15\n";
|
|
244
250
|
out += `pipeline:\n after_approval: ${c.afterApproval || "stop"} # stop | run-through (implement → verify → review → checkpoint; ship always asks)\n`;
|
|
245
251
|
return out + `specs_dir: ${y(c.specsDir)}\n`;
|
|
246
252
|
}
|
|
@@ -515,10 +521,20 @@ function parseYaml(text) {
|
|
|
515
521
|
return root;
|
|
516
522
|
}
|
|
517
523
|
|
|
524
|
+
// A multi-repo config applies only in its own folder and inside the repos it lists, so a repo that
|
|
525
|
+
// merely sits under that folder (e.g. ~/Projects/other-project) doesn't pick up the workspace's rules.
|
|
526
|
+
function configApplies(f, dir) {
|
|
527
|
+
let c;
|
|
528
|
+
try { c = parseYaml(fs.readFileSync(f, "utf8")); } catch { return true; }
|
|
529
|
+
if (c.layout !== "multi-repo" || !Array.isArray(c.repos)) return true;
|
|
530
|
+
const here = real(dir), base = path.dirname(f);
|
|
531
|
+
return here === real(base) || c.repos.some((r) => { const p = real(path.resolve(base, expandHome(String(r.path || "")))); return here === p || here.startsWith(p + path.sep); });
|
|
532
|
+
}
|
|
533
|
+
|
|
518
534
|
function findConfig(dir) {
|
|
519
535
|
for (let d = dir; ; d = path.dirname(d)) {
|
|
520
536
|
const f = path.join(d, ".spec-driven.yml");
|
|
521
|
-
if (fs.existsSync(f)) return f;
|
|
537
|
+
if (fs.existsSync(f) && configApplies(f, dir)) return f;
|
|
522
538
|
if (d === path.dirname(d) || d === HOME) break;
|
|
523
539
|
}
|
|
524
540
|
// Not above us: a multi-repo workspace elsewhere may list this repo.
|
|
@@ -577,6 +593,11 @@ function doctor() {
|
|
|
577
593
|
if (c.context?.graph && !["auto", "off"].includes(c.context.graph)) problems.push(`context.graph "${c.context.graph}" must be auto or off`);
|
|
578
594
|
const repoRoots = c.layout === "multi-repo" ? (c.repos || []).map((r) => path.resolve(root, expandHome(r.path))) : [root];
|
|
579
595
|
checkContext(specs, c.specs_dir || "specs", repoRoots, problems, notes);
|
|
596
|
+
if (c.risk) {
|
|
597
|
+
if (c.risk.enabled && !["on", "off"].includes(c.risk.enabled)) problems.push(`risk.enabled "${c.risk.enabled}" must be on or off`);
|
|
598
|
+
problems.push(...weightProblems(c.risk.weights));
|
|
599
|
+
if (c.risk.enabled !== "off" && !fs.existsSync(path.join(specs, "context", "risk.md"))) notes.push(`risk is on but ${c.specs_dir || "specs"}/context/risk.md doesn't exist — run: spec-driven-dev risk`);
|
|
600
|
+
}
|
|
580
601
|
console.log(`Checked ${file}`);
|
|
581
602
|
for (const n of notes) console.log(` • ${n}`);
|
|
582
603
|
for (const p of problems) console.log(` ✗ ${p}`);
|
|
@@ -620,6 +641,103 @@ function context() {
|
|
|
620
641
|
}
|
|
621
642
|
}
|
|
622
643
|
|
|
644
|
+
function loadConfig() {
|
|
645
|
+
const file = findConfig(DIR);
|
|
646
|
+
return { file, root: file ? path.dirname(file) : gitTop(DIR), c: file ? parseYaml(fs.readFileSync(file, "utf8")) : {} };
|
|
647
|
+
}
|
|
648
|
+
|
|
649
|
+
function risk() {
|
|
650
|
+
const { root, c } = loadConfig();
|
|
651
|
+
const bad = weightProblems(c.risk?.weights);
|
|
652
|
+
if (bad.length) throw new Error(bad.join("; "));
|
|
653
|
+
const weights = normWeights(c.risk?.weights), since = opt("since") || c.risk?.since || "2 years ago";
|
|
654
|
+
const repoDirs = c.layout === "multi-repo" ? (c.repos || []).map((r) => ({ dir: path.resolve(root, expandHome(r.path)), name: r.path })) : [{ dir: root, name: path.basename(root) }];
|
|
655
|
+
const scored = repoDirs.filter((r) => isRepo(r.dir)).map((r) => ({ ...r, ...scoreRepo(r.dir, { since, weights, prefix: c.layout === "multi-repo" ? r.name : "" }) }));
|
|
656
|
+
if (!scored.length) throw new Error(`No git repo found at ${root}`);
|
|
657
|
+
const rows = scored.flatMap((s) => s.rows).sort((a, b) => b.score - a.score || b.commits - a.commits);
|
|
658
|
+
const wanted = splitList(opt("files"));
|
|
659
|
+
if (wanted.length) {
|
|
660
|
+
const norm = (f) => f.replace(/\\/g, "/").replace(/^\.\//, "");
|
|
661
|
+
const hits = wanted.map((f) => rows.find((r) => r.file === norm(f) || r.file.endsWith("/" + norm(f))) || { file: norm(f), score: null, tier: "no history" });
|
|
662
|
+
if (flag("json")) console.log(JSON.stringify(hits, null, 2));
|
|
663
|
+
else for (const h of hits) console.log(h.score === null ? `${h.file}: no history (new, untracked or not code)` : `${h.file}: ${h.score} ${h.tier} (churn ${h.factors.churn}, bug density ${h.factors.bug_density}, fan-out ${h.factors.fan_out}, security ${h.factors.security})`);
|
|
664
|
+
return;
|
|
665
|
+
}
|
|
666
|
+
if (flag("json")) { console.log(JSON.stringify({ weights, since, rows }, null, 2)); return; }
|
|
667
|
+
const specs = path.join(root, c.specs_dir || "specs");
|
|
668
|
+
fs.mkdirSync(path.join(specs, "context"), { recursive: true });
|
|
669
|
+
const out = path.join(specs, "context", "risk.md");
|
|
670
|
+
fs.writeFileSync(out, riskMarkdown({ rows, repos: scored, weights, since, top: Number(opt("top")) || 40 }));
|
|
671
|
+
const count = (t) => rows.filter((r) => r.tier === t).length;
|
|
672
|
+
console.log(`✓ Scored ${rows.length} files from ${scored.reduce((t, s) => t + s.commits, 0)} commits → ${path.relative(DIR, out) || out}`);
|
|
673
|
+
console.log(` ${count("critical")} critical · ${count("high")} high · ${count("medium")} medium. Top 5:`);
|
|
674
|
+
for (const r of rows.slice(0, 5)) console.log(` ${String(r.score).padStart(5)} ${r.tier.padEnd(8)} ${r.file}`);
|
|
675
|
+
if (c.risk?.enabled === "off") console.log(" Note: risk.enabled is off in .spec-driven.yml, so the skills won't use this until you turn it on.");
|
|
676
|
+
}
|
|
677
|
+
|
|
678
|
+
async function riskEvalCmd() {
|
|
679
|
+
const { file, root, c } = loadConfig();
|
|
680
|
+
const repo = gitTop(DIR);
|
|
681
|
+
if (c.layout === "multi-repo" && real(repo) === real(root)) throw new Error("risk-eval runs on one repo: run it inside one of the workspace's repos");
|
|
682
|
+
const ticketsFile = opt("tickets");
|
|
683
|
+
if (!ticketsFile) throw new Error("Pass --tickets=<file> with one ticket per line");
|
|
684
|
+
const tickets = fs.readFileSync(path.resolve(DIR, ticketsFile), "utf8").split("\n").map((l) => l.replace(/^\s*[-*]\s+/, "").trim()).filter((l) => l && !l.startsWith("#"));
|
|
685
|
+
const claude = opt("claude") || "claude";
|
|
686
|
+
if (!opt("claude") && !onPath("claude")) throw new Error("risk-eval drives Claude Code headlessly; install it first (https://code.claude.com)");
|
|
687
|
+
const { riskEval } = await import("./risk-eval.mjs");
|
|
688
|
+
const runs = Number(opt("runs")) || 1, stages = splitList(opt("stages") || "investigate,create");
|
|
689
|
+
console.log(`GitRisk U8 experiment: ${tickets.length} ticket(s) × ${runs} run(s) × 2 arms (risk on / off) on ${root}`);
|
|
690
|
+
// A workspace config's repo paths don't resolve inside a lone clone, so a multi-repo member gets a single-repo config.
|
|
691
|
+
const configText = c.layout === "multi-repo" ? "version: 1\nlayout: single\n" : file ? fs.readFileSync(file, "utf8") : "";
|
|
692
|
+
const arms = splitList(opt("arms") || "on,off");
|
|
693
|
+
if (!arms.length || arms.some((a) => !["on", "off"].includes(a))) throw new Error("--arms must be on, off, or on,off");
|
|
694
|
+
const { dir } = await riskEval({ repo, tickets, arms, timeoutMin: Number(opt("timeout")) || 20, runs, model: opt("model") || "sonnet", stages, claude, out: path.resolve(DIR, opt("out") || "experiments"),
|
|
695
|
+
pluginDir: path.join(ROOT, "plugin"), since: c.risk?.since || "2 years ago", weights: c.risk?.weights, configText });
|
|
696
|
+
console.log(`✓ Report: ${path.join(dir, "report.md")}`);
|
|
697
|
+
}
|
|
698
|
+
|
|
699
|
+
// Plain-language switches, so anyone can change them without editing YAML.
|
|
700
|
+
const SETTINGS = [
|
|
701
|
+
{ block: "risk", key: "enabled", flag: "risk", name: "GitRisk", about: "extra care on files with a history of bugs",
|
|
702
|
+
get: (c) => c.risk?.enabled || "on", options: [
|
|
703
|
+
{ value: "on", title: "On", description: "investigations, plans and reviews look harder at historically risky files" },
|
|
704
|
+
{ value: "off", title: "Off", description: "the skills ignore risk completely" }] },
|
|
705
|
+
{ block: "pipeline", key: "after_approval", flag: "after-approval", name: "After you approve a spec", about: "how far it goes on its own",
|
|
706
|
+
get: (c) => c.pipeline?.after_approval || "stop", options: [
|
|
707
|
+
{ value: "stop", title: "Stop after each stage", description: "you start implement, verify, review and commit yourself" },
|
|
708
|
+
{ value: "run-through", title: "Run straight through", description: "implement → verify → review → commit; stops at any failure, never pushes" }] },
|
|
709
|
+
];
|
|
710
|
+
|
|
711
|
+
async function settings() {
|
|
712
|
+
const file = findConfig(DIR);
|
|
713
|
+
if (!file) throw new Error(`No .spec-driven.yml in ${DIR} or above. Run: spec-driven-dev setup`);
|
|
714
|
+
let text = fs.readFileSync(file, "utf8");
|
|
715
|
+
const label = (s, v) => s.options.find((o) => o.value === v)?.title || v;
|
|
716
|
+
const save = (s, v) => {
|
|
717
|
+
if (!s.options.some((o) => o.value === v)) throw new Error(`--${s.flag} must be ${s.options.map((o) => o.value).join(" or ")}`);
|
|
718
|
+
text = setConfigValue(text, s.block, s.key, v);
|
|
719
|
+
fs.writeFileSync(file, text);
|
|
720
|
+
console.log(`✓ ${s.name}: ${label(s, v)} (saved in ${file})`);
|
|
721
|
+
};
|
|
722
|
+
const given = SETTINGS.filter((s) => opt(s.flag));
|
|
723
|
+
if (given.length) given.forEach((s) => save(s, opt(s.flag)));
|
|
724
|
+
else for (;;) {
|
|
725
|
+
const c = parseYaml(text);
|
|
726
|
+
const pick = await askSelect("Settings: pick one to change", [
|
|
727
|
+
...SETTINGS.map((s) => ({ title: `${s.name}: ${label(s, s.get(c))}`, description: s.about, value: s })),
|
|
728
|
+
{ title: "Done", value: null }], null);
|
|
729
|
+
if (!pick) break;
|
|
730
|
+
const v = await askSelect(`${pick.name}`, pick.options, pick.get(c));
|
|
731
|
+
if (v !== pick.get(c)) save(pick, v);
|
|
732
|
+
}
|
|
733
|
+
const c = parseYaml(text);
|
|
734
|
+
if (c.risk?.enabled !== "off" && !fs.existsSync(path.join(path.dirname(file), c.specs_dir || "specs", "context", "risk.md"))) {
|
|
735
|
+
if (given.length || !(await askConfirm("GitRisk is on but files haven't been scored yet. Score them now? (uses git only, takes seconds)", true)))
|
|
736
|
+
console.log(" GitRisk is on, but nothing is scored yet. Run: spec-driven-dev risk");
|
|
737
|
+
else risk();
|
|
738
|
+
}
|
|
739
|
+
}
|
|
740
|
+
|
|
623
741
|
// Silent unless something is unfinished, so it's cheap to run at the start of a session.
|
|
624
742
|
function status() {
|
|
625
743
|
const file = findConfig(DIR);
|
|
@@ -648,11 +766,14 @@ try {
|
|
|
648
766
|
if (command === "doctor") doctor();
|
|
649
767
|
else if (command === "status") status();
|
|
650
768
|
else if (command === "context") context();
|
|
769
|
+
else if (command === "risk") risk();
|
|
770
|
+
else if (command === "settings") await settings();
|
|
771
|
+
else if (command === "risk-eval") await riskEvalCmd();
|
|
651
772
|
else if (command === "install") {
|
|
652
773
|
await install();
|
|
653
774
|
if (!flag("no-setup") && (await askConfirm("Set up this project now?", true))) await setup();
|
|
654
775
|
} else if (command === "setup") await setup();
|
|
655
|
-
else throw new Error(`Unknown command "${command}" — use install (default), setup, doctor, status, or
|
|
776
|
+
else throw new Error(`Unknown command "${command}" — use install (default), setup, settings, doctor, status, context, risk, or risk-eval`);
|
|
656
777
|
if (["install", "setup"].includes(command)) console.log('\nNext: in your AI tool, say "investigate <ticket>" or "create a spec".');
|
|
657
778
|
} catch (e) {
|
|
658
779
|
console.error(`✗ ${e.message}`);
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
// GitRisk U8 experiment: does giving the agent the risk signal change what it does?
|
|
2
|
+
// Each ticket runs twice in fresh clones of the repo, risk ON and risk OFF, with the same model and
|
|
3
|
+
// prompt. We then measure behaviour from what the agent read and wrote, and keep a dated log.
|
|
4
|
+
import fs from "node:fs";
|
|
5
|
+
import os from "node:os";
|
|
6
|
+
import path from "node:path";
|
|
7
|
+
import { execFileSync, spawnSync } from "node:child_process";
|
|
8
|
+
import { scoreRepo, riskMarkdown, normWeights, setConfigValue } from "./risk.mjs";
|
|
9
|
+
|
|
10
|
+
const TIER_RANK = { light: 1, standard: 2, deep: 3 };
|
|
11
|
+
const sh = (cwd, cmd, args) => execFileSync(cmd, args, { cwd, stdio: ["ignore", "pipe", "pipe"], maxBuffer: 512 * 1024 * 1024 }).toString();
|
|
12
|
+
|
|
13
|
+
export const setRiskEnabled = (text, on) => setConfigValue(text, "risk", "enabled", on ? "on" : "off");
|
|
14
|
+
|
|
15
|
+
// Behaviour metrics from the agent's transcript (stream-json) and the spec it wrote.
|
|
16
|
+
export function measure({ transcript, clone, risky, tracked = [] }) {
|
|
17
|
+
const reads = new Set();
|
|
18
|
+
let cost = 0, turns = 0, inTok = 0, outTok = 0, error = "";
|
|
19
|
+
for (const line of transcript.split("\n")) {
|
|
20
|
+
let e;
|
|
21
|
+
try { e = JSON.parse(line); } catch { continue; }
|
|
22
|
+
if (e.type === "assistant") for (const c of e.message?.content || []) {
|
|
23
|
+
// A file counts as looked at when a Read, Grep or Bash call names it (agents often use grep/sed/cat, not Read).
|
|
24
|
+
if (c.type !== "tool_use" || !["Read", "Grep", "Bash", "Glob"].includes(c.name)) continue;
|
|
25
|
+
const text = JSON.stringify(c.input || {}).split(clone + "/").join("").split(clone.replace(/^\/private/, "") + "/").join("");
|
|
26
|
+
for (const f of tracked) if (text.includes(f)) reads.add(f);
|
|
27
|
+
}
|
|
28
|
+
if (e.type === "result") {
|
|
29
|
+
cost = e.total_cost_usd || 0; turns = e.num_turns || 0;
|
|
30
|
+
inTok = (e.usage?.input_tokens || 0) + (e.usage?.cache_read_input_tokens || 0) + (e.usage?.cache_creation_input_tokens || 0);
|
|
31
|
+
outTok = e.usage?.output_tokens || 0;
|
|
32
|
+
if (e.is_error) error = String(e.result || "error").slice(0, 200);
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
const specsDir = path.join(clone, "specs");
|
|
36
|
+
const feature = (fs.existsSync(specsDir) ? fs.readdirSync(specsDir) : []).find((d) => fs.existsSync(path.join(specsDir, d, "investigation.md")) || fs.existsSync(path.join(specsDir, d, "tasks")));
|
|
37
|
+
const read = (f) => { try { return fs.readFileSync(f, "utf8"); } catch { return ""; } };
|
|
38
|
+
const inv = feature ? read(path.join(specsDir, feature, "investigation.md")) : "";
|
|
39
|
+
const tasks = [];
|
|
40
|
+
const tdir = feature && path.join(specsDir, feature, "tasks");
|
|
41
|
+
if (tdir && fs.existsSync(tdir)) for (const t of fs.readdirSync(tdir).filter((f) => f.endsWith(".md"))) {
|
|
42
|
+
const text = read(path.join(tdir, t));
|
|
43
|
+
const section = (name) => (text.match(new RegExp(`^## ${name}\\s*\\n([\\s\\S]*?)(?=^## |$(?![\\s\\S]))`, "m")) || [])[1] || "";
|
|
44
|
+
const files = [...(section("Files to Create") + section("Files to Modify")).matchAll(/`([^`\s]+\.[\w]+)`/g)].map((m) => m[1]);
|
|
45
|
+
const tests = section("Tests").trim();
|
|
46
|
+
tasks.push({ name: t, files, tier: (text.match(/\*\*Tier:\*\*\s*(light|standard|deep)/i) || [])[1]?.toLowerCase(), hasTests: !!tests && !/^none\b/i.test(tests) });
|
|
47
|
+
}
|
|
48
|
+
const isRisky = (f) => risky.has(f);
|
|
49
|
+
const planFiles = new Set(tasks.flatMap((t) => t.files));
|
|
50
|
+
const riskyPlan = [...planFiles].filter(isRisky);
|
|
51
|
+
const riskyTasks = tasks.filter((t) => t.files.some(isRisky));
|
|
52
|
+
const avg = (xs) => (xs.length ? Math.round((xs.reduce((a, b) => a + b, 0) / xs.length) * 100) / 100 : null);
|
|
53
|
+
return {
|
|
54
|
+
feature: feature || null,
|
|
55
|
+
files_read: reads.size,
|
|
56
|
+
risky_files_read: [...reads].filter(isRisky).length,
|
|
57
|
+
investigation_risky_mentions: [...risky].filter((f) => inv.includes(f)).length,
|
|
58
|
+
plan_files: planFiles.size,
|
|
59
|
+
risky_plan_files: riskyPlan.length,
|
|
60
|
+
risky_files_with_tests: riskyPlan.filter((f) => tasks.some((t) => t.files.includes(f) && t.hasTests)).length,
|
|
61
|
+
tier_on_risky_tasks: avg(riskyTasks.map((t) => TIER_RANK[t.tier]).filter(Boolean)),
|
|
62
|
+
tier_all_tasks: avg(tasks.map((t) => TIER_RANK[t.tier]).filter(Boolean)),
|
|
63
|
+
tasks: tasks.length,
|
|
64
|
+
cost_usd: Math.round(cost * 10000) / 10000, turns, input_tokens: inTok, output_tokens: outTok, error,
|
|
65
|
+
};
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
const METRICS = [
|
|
69
|
+
["risky_files_read", "Risky files looked at"], ["investigation_risky_mentions", "Risky files named in investigation"],
|
|
70
|
+
["risky_plan_files", "Risky files in the plan"], ["risky_files_with_tests", "…of those, with tests planned"],
|
|
71
|
+
["tier_on_risky_tasks", "Avg tier of tasks on risky files (1–3)"], ["files_read", "Repo files looked at (Read/Grep/Bash)"],
|
|
72
|
+
["tasks", "Tasks"], ["cost_usd", "Cost (USD)"], ["output_tokens", "Output tokens"],
|
|
73
|
+
];
|
|
74
|
+
|
|
75
|
+
function report({ meta, results }) {
|
|
76
|
+
const lines = [`# GitRisk U8 experiment — ${meta.started}`, "",
|
|
77
|
+
"Question: does feeding GitRisk's historical-risk signal into the spec-driven-dev workflow change what the AI agent does?", "",
|
|
78
|
+
`- Repo: \`${meta.repo}\` at ${meta.head}; risky set = ${meta.risky} files scored high or critical (weights ${JSON.stringify(meta.weights)})`,
|
|
79
|
+
`- Model: ${meta.model} · stages: ${meta.stages.join(" → ")} · runs per arm: ${meta.runs} · tickets: ${meta.tickets.length}`,
|
|
80
|
+
"- Arms: **on** = risk.enabled on + specs/context/risk.md present; **off** = risk.enabled off, no risk file. Fresh clone per run, same prompt.",
|
|
81
|
+
"", "## Results (mean per arm)", "", "| Ticket | Metric | Off | On | Δ (on − off) |", "|---|---|---|---|---|"];
|
|
82
|
+
const mean = (rs, k) => { const v = rs.map((r) => r.metrics?.[k]).filter((x) => typeof x === "number"); return v.length ? Math.round((v.reduce((a, b) => a + b, 0) / v.length) * 100) / 100 : null; };
|
|
83
|
+
meta.tickets.forEach((t, i) => {
|
|
84
|
+
const off = results.filter((r) => r.ticket === i && r.arm === "off" && !r.metrics?.error), on = results.filter((r) => r.ticket === i && r.arm === "on" && !r.metrics?.error);
|
|
85
|
+
for (const [k, label] of METRICS) {
|
|
86
|
+
const a = mean(off, k), b = mean(on, k);
|
|
87
|
+
lines.push(`| ${i + 1}. ${t.slice(0, 40).replace(/\|/g, "/")} | ${label} | ${a ?? "—"} | ${b ?? "—"} | ${a !== null && b !== null ? Math.round((b - a) * 100) / 100 : "—"} |`);
|
|
88
|
+
}
|
|
89
|
+
});
|
|
90
|
+
const failed = results.filter((r) => r.metrics?.error || r.exit !== 0);
|
|
91
|
+
lines.push("", "## Runs", "", "| Ticket | Arm | Run | Exit | Feature folder | Error |", "|---|---|---|---|---|---|",
|
|
92
|
+
...results.map((r) => `| ${r.ticket + 1} | ${r.arm} | ${r.run} | ${r.exit} | ${r.metrics?.feature || "—"} | ${(r.metrics?.error || "").replace(/\|/g, "/")} |`));
|
|
93
|
+
lines.push("", "## Caveats", "",
|
|
94
|
+
`- ${meta.runs} run(s) per arm is a pilot, not a significance test. Agent behaviour varies run to run; increase --runs before drawing conclusions.`,
|
|
95
|
+
"- Metrics are proxies read from the transcript and the written spec (files the agent opened, files and tests it planned, task tiers). They don't measure whether the eventual code has fewer bugs.",
|
|
96
|
+
"- The risky set comes from the same repo history the ON arm sees, so a higher \"risky files read\" in ON is expected; the interesting signal is whether that changes the plan (tests, tiers) without raising cost much.",
|
|
97
|
+
`- Failed runs: ${failed.length}. They are excluded from the means but kept in results.json.`);
|
|
98
|
+
return lines.join("\n") + "\n";
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
export async function riskEval(opts) {
|
|
102
|
+
const { repo, tickets, runs = 1, arms = ["on", "off"], timeoutMin = 20, model = "sonnet", stages = ["investigate", "create"], claude = "claude", out, pluginDir, since, weights, configText = "" } = opts;
|
|
103
|
+
if (!tickets.length) throw new Error("No tickets: pass --tickets=<file> with one ticket per line");
|
|
104
|
+
const started = new Date().toISOString().replace(/\.\d+Z$/, "Z");
|
|
105
|
+
const dir = path.join(out, `risk-eval-${started.replace(/[:]/g, "-")}`);
|
|
106
|
+
fs.mkdirSync(dir, { recursive: true });
|
|
107
|
+
const w = normWeights(weights);
|
|
108
|
+
const scored = scoreRepo(repo, { since, weights: w });
|
|
109
|
+
scored.rows.sort((a, b) => b.score - a.score);
|
|
110
|
+
const risky = new Set(scored.rows.filter((r) => r.tier === "high" || r.tier === "critical").map((r) => r.file));
|
|
111
|
+
const riskMd = riskMarkdown({ rows: scored.rows, repos: [{ name: path.basename(repo), head: scored.head, commits: scored.commits }], weights: w, since, top: 60 });
|
|
112
|
+
fs.writeFileSync(path.join(dir, "risk.md"), riskMd);
|
|
113
|
+
const meta = { started, repo, head: scored.head, risky: risky.size, weights: w, model, stages, runs, tickets };
|
|
114
|
+
const stageText = stages.map((s) => `spec-${s}`).join(", then ");
|
|
115
|
+
const results = [];
|
|
116
|
+
for (let t = 0; t < tickets.length; t++) for (let r = 1; r <= runs; r++) for (const arm of ((t + r) % 2 ? ["off", "on"] : ["on", "off"]).filter((a) => arms.includes(a))) {
|
|
117
|
+
const clone = fs.mkdtempSync(path.join(os.tmpdir(), `risk-eval-${arm}-`));
|
|
118
|
+
sh(os.tmpdir(), "git", ["clone", "-q", "--no-hardlinks", repo, clone]);
|
|
119
|
+
for (const extra of ["specs/CONTEXT.md", "specs/context"]) {
|
|
120
|
+
const src = path.join(repo, extra);
|
|
121
|
+
if (fs.existsSync(src) && !fs.existsSync(path.join(clone, extra))) fs.cpSync ? fs.cpSync(src, path.join(clone, extra), { recursive: true }) : sh(repo, "cp", ["-R", src, path.join(clone, extra)]);
|
|
122
|
+
}
|
|
123
|
+
fs.rmSync(path.join(clone, "specs/context/risk.md"), { force: true });
|
|
124
|
+
fs.writeFileSync(path.join(clone, ".spec-driven.yml"), setRiskEnabled(configText || "version: 1\nlayout: single\n", arm === "on"));
|
|
125
|
+
if (arm === "on") { fs.mkdirSync(path.join(clone, "specs/context"), { recursive: true }); fs.writeFileSync(path.join(clone, "specs/context/risk.md"), riskMd); }
|
|
126
|
+
const prompt = `Use the ticket-to-pr plugin skills. This is a non-interactive run: don't ask questions, make reasonable assumptions and record them. Ticket (plain text, no tracker): ${tickets[t]}\nRun ${stageText}. Work on the current branch; don't create branches, commit, push, or implement anything.`;
|
|
127
|
+
const res = spawnSync(claude, ["-p", "--output-format", "stream-json", "--verbose", "--model", model, "--plugin-dir", pluginDir, "--permission-mode", "acceptEdits",
|
|
128
|
+
"--allowedTools", "Read Write Edit Glob Grep Skill Bash(git:*) Bash(ls:*) Bash(cat:*) Bash(head:*) Bash(tail:*) Bash(wc:*) Bash(find:*) Bash(grep:*) Bash(node:*) Bash(npx:*) Bash(spec-driven-dev:*)"],
|
|
129
|
+
{ cwd: clone, input: prompt, encoding: "utf8", maxBuffer: 512 * 1024 * 1024, timeout: timeoutMin * 60 * 1000 });
|
|
130
|
+
const transcript = res.stdout || "";
|
|
131
|
+
const tag = `t${t + 1}-${arm}-r${r}`;
|
|
132
|
+
fs.writeFileSync(path.join(dir, `${tag}.jsonl`), transcript);
|
|
133
|
+
const tracked = sh(clone, "git", ["ls-files"]).split("\n").filter((f) => f.includes("/") || f.includes("."));
|
|
134
|
+
const metrics = measure({ transcript, clone, risky, tracked });
|
|
135
|
+
if (res.status !== 0 && !metrics.error) {
|
|
136
|
+
// Say where it stopped, so a stall (usage limit, API wait) is told apart from a skill problem.
|
|
137
|
+
const events = transcript.split("\n").filter(Boolean).length;
|
|
138
|
+
const lastTool = [...transcript.matchAll(/"type":"tool_use","id":"[^"]+","name":"(\w+)"/g)].pop()?.[1] || "none";
|
|
139
|
+
metrics.error = `${res.error?.code === "ETIMEDOUT" ? `timed out after ${timeoutMin} min` : `exit ${res.status}`} · ${events} events · last tool ${lastTool} ${(res.stderr || "").slice(0, 120)}`.trim();
|
|
140
|
+
}
|
|
141
|
+
if (metrics.feature) fs.cpSync ? fs.cpSync(path.join(clone, "specs", metrics.feature), path.join(dir, tag), { recursive: true }) : sh(clone, "cp", ["-R", path.join("specs", metrics.feature), path.join(dir, tag)]);
|
|
142
|
+
results.push({ ticket: t, arm, run: r, exit: res.status, metrics });
|
|
143
|
+
fs.rmSync(clone, { recursive: true, force: true });
|
|
144
|
+
console.log(` ${tag}: ${metrics.error ? `failed (${metrics.error.slice(0, 80)})` : `read ${metrics.risky_files_read} risky / ${metrics.files_read} files, ${metrics.risky_plan_files} risky in plan, $${metrics.cost_usd}`}`);
|
|
145
|
+
}
|
|
146
|
+
fs.writeFileSync(path.join(dir, "results.json"), JSON.stringify({ meta, results }, null, 2));
|
|
147
|
+
const md = report({ meta, results });
|
|
148
|
+
fs.writeFileSync(path.join(dir, "report.md"), md);
|
|
149
|
+
const failed = results.filter((x) => x.metrics.error).length;
|
|
150
|
+
fs.appendFileSync(path.join(out, "LOG.md"), `${fs.existsSync(path.join(out, "LOG.md")) ? "" : "# GitRisk experiment log\n\nAppend-only record of every run, including failures.\n\n"}- ${started} · ${tickets.length} ticket(s) × ${runs} run(s) × 2 arms on \`${path.basename(repo)}\` at ${scored.head}, ${model}, ${stages.join("+")} · ${results.length - failed} ok, ${failed} failed · [report](${path.basename(dir)}/report.md)\n`);
|
|
151
|
+
return { dir, report: md, results };
|
|
152
|
+
}
|
package/bin/risk.mjs
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
// GitRisk: per-file historical risk from git alone (no model, no tokens).
|
|
2
|
+
// Risk = 0.30·Churn + 0.35·BugDensity + 0.20·FanOut + 0.15·Security (each factor 0–100)
|
|
3
|
+
// The weights are a hypothesis to calibrate by backtest, so .spec-driven.yml `risk.weights` overrides them.
|
|
4
|
+
import fs from "node:fs";
|
|
5
|
+
import path from "node:path";
|
|
6
|
+
import { execFileSync } from "node:child_process";
|
|
7
|
+
|
|
8
|
+
export const DEFAULT_WEIGHTS = { churn: 0.3, bug_density: 0.35, fan_out: 0.2, security: 0.15 };
|
|
9
|
+
export const TIERS = [[75, "critical"], [50, "high"], [25, "medium"], [0, "low"]];
|
|
10
|
+
const MIN_COMMITS = 3; // shortcut: a file with 1 commit that was a fix would score 100% bug density; divide by at least 3. Replace with Bayesian smoothing if calibration shows it matters.
|
|
11
|
+
|
|
12
|
+
// Classification uses the conventional-commit type when the repo uses one, and message keywords otherwise.
|
|
13
|
+
const BUG = /^(fix|hotfix|bugfix|revert)(\(.+\))?!?:|\b(fix(e[sd]|ing)?|bug|hotfix|regression|broken|crash(es|ed)?|patch(ed)?|defect)\b/i;
|
|
14
|
+
const SECURITY = /\b(security|cve-\d+|xss|csrf|ssrf|sqli|injection|vuln(erability)?|exploit|sanitiz(e|ation)|escap(e|ing) (html|input|output)|auth(entication|orization)? (bypass|fix)|privilege|secret(s)? (leak|exposed)|leak(ed|s)? (token|key|secret|credential)|rate.?limit|helmet|csp)\b/i;
|
|
15
|
+
const SKIP = /(^|\/)(node_modules|dist|build|out|vendor|coverage|\.next|graphify-out|specs)\/|\.(md|mdx|txt|lock|log|png|jpe?g|gif|svg|ico|webp|pdf|zip|map|snap|csv)$|(^|\/)(package-lock\.json|yarn\.lock|pnpm-lock\.yaml|Cargo\.lock|go\.sum|poetry\.lock|composer\.lock)$/i;
|
|
16
|
+
const CODE = /\.(m?[jt]sx?|cjs|py|rb|go|rs|java|kt|php|cs|swift|scala|vue|svelte)$/i;
|
|
17
|
+
|
|
18
|
+
const git = (cwd, args) => {
|
|
19
|
+
try { return execFileSync("git", args, { cwd, stdio: ["ignore", "pipe", "ignore"], maxBuffer: 512 * 1024 * 1024 }).toString(); } catch { return ""; }
|
|
20
|
+
};
|
|
21
|
+
export const classify = (subject) => ({ bug: BUG.test(subject), security: SECURITY.test(subject) });
|
|
22
|
+
export const tierOf = (score) => TIERS.find(([min]) => score >= min)[1];
|
|
23
|
+
|
|
24
|
+
// Value at the 90th percentile; "expected" in min(100, actual / expected × 100), so one outlier can't flatten everyone else.
|
|
25
|
+
function p90(values) {
|
|
26
|
+
const v = values.filter((x) => x > 0).sort((a, b) => a - b);
|
|
27
|
+
return v.length ? v[Math.min(v.length - 1, Math.floor(v.length * 0.9))] : 1;
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
// Within-repo dependents: how many other files import this one. Relative JS/TS imports and Python module paths.
|
|
31
|
+
// shortcut: no tsconfig path aliases or cross-repo consumers (GitRisk open question U7); add when a repo needs them.
|
|
32
|
+
function fanOut(repo, files) {
|
|
33
|
+
const set = new Set(files), dependents = new Map(files.map((f) => [f, new Set()]));
|
|
34
|
+
const exts = ["", ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs", "/index.ts", "/index.tsx", "/index.js", "/index.jsx", ".py", "/__init__.py"];
|
|
35
|
+
const pyIndex = new Map(files.filter((f) => f.endsWith(".py")).map((f) => [f.replace(/\/__init__\.py$|\.py$/, "").split("/").join("."), f]));
|
|
36
|
+
for (const f of files) {
|
|
37
|
+
if (!CODE.test(f)) continue;
|
|
38
|
+
let text = "";
|
|
39
|
+
try { text = fs.readFileSync(path.join(repo, f), "utf8"); } catch { continue; }
|
|
40
|
+
if (text.length > 2e6) continue;
|
|
41
|
+
const targets = new Set();
|
|
42
|
+
for (const m of text.matchAll(/(?:from\s+|require\(\s*|import\(\s*|import\s+)["'](\.{1,2}\/[^"']+)["']/g)) {
|
|
43
|
+
const base = path.posix.normalize(path.posix.join(path.posix.dirname(f), m[1]));
|
|
44
|
+
const hit = exts.map((e) => base + e).find((c) => set.has(c));
|
|
45
|
+
if (hit) targets.add(hit);
|
|
46
|
+
}
|
|
47
|
+
if (f.endsWith(".py")) for (const m of text.matchAll(/^\s*(?:from\s+([\w.]+)\s+import|import\s+([\w.]+))/gm)) {
|
|
48
|
+
const hit = pyIndex.get(m[1] || m[2]);
|
|
49
|
+
if (hit) targets.add(hit);
|
|
50
|
+
}
|
|
51
|
+
for (const t of targets) if (t !== f) dependents.get(t).add(f);
|
|
52
|
+
}
|
|
53
|
+
return new Map([...dependents].map(([f, s]) => [f, s.size]));
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
// Score every tracked code file in one repo. `prefix` labels files in multi-repo workspaces.
|
|
57
|
+
export function scoreRepo(repo, { since = "2 years ago", weights = DEFAULT_WEIGHTS, prefix = "" } = {}) {
|
|
58
|
+
const files = git(repo, ["ls-files"]).split("\n").filter((f) => f && !SKIP.test(f));
|
|
59
|
+
const tracked = new Set(files);
|
|
60
|
+
const stats = new Map(files.map((f) => [f, { commits: 0, bug: 0, security: 0 }]));
|
|
61
|
+
let commits = 0;
|
|
62
|
+
// \x1e starts each commit; the subject follows, then the changed file names.
|
|
63
|
+
for (const block of git(repo, ["log", "--no-merges", `--since=${since}`, "--name-only", "--format=%x1e%s"]).split("\x1e").slice(1)) {
|
|
64
|
+
const [subject, ...changed] = block.split("\n");
|
|
65
|
+
const c = classify(subject);
|
|
66
|
+
commits++;
|
|
67
|
+
for (const f of changed) {
|
|
68
|
+
const s = tracked.has(f) && stats.get(f);
|
|
69
|
+
if (!s) continue;
|
|
70
|
+
s.commits++;
|
|
71
|
+
if (c.bug) s.bug++;
|
|
72
|
+
if (c.security) s.security++;
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
const fan = fanOut(repo, files);
|
|
76
|
+
const expChurn = p90([...stats.values()].map((s) => s.commits)), expFan = p90([...fan.values()]);
|
|
77
|
+
const rows = files.map((f) => {
|
|
78
|
+
const s = stats.get(f), denom = Math.max(s.commits, MIN_COMMITS);
|
|
79
|
+
const factors = {
|
|
80
|
+
churn: Math.min(100, (s.commits / expChurn) * 100),
|
|
81
|
+
bug_density: s.commits ? (s.bug / denom) * 100 : 0,
|
|
82
|
+
fan_out: Math.min(100, ((fan.get(f) || 0) / expFan) * 100),
|
|
83
|
+
security: s.commits ? (s.security / denom) * 100 : 0,
|
|
84
|
+
};
|
|
85
|
+
const score = Object.keys(DEFAULT_WEIGHTS).reduce((t, k) => t + (Number(weights[k]) || 0) * factors[k], 0);
|
|
86
|
+
const r = (x) => Math.round(x * 10) / 10;
|
|
87
|
+
return { file: prefix ? `${prefix}/${f}` : f, score: r(score), tier: tierOf(score), commits: s.commits, bug_fixes: s.bug, security_fixes: s.security, dependents: fan.get(f) || 0,
|
|
88
|
+
factors: { churn: r(factors.churn), bug_density: r(factors.bug_density), fan_out: r(factors.fan_out), security: r(factors.security) } };
|
|
89
|
+
});
|
|
90
|
+
return { commits, head: git(repo, ["rev-parse", "--short", "HEAD"]).trim(), rows };
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
export function normWeights(w) {
|
|
94
|
+
const out = { ...DEFAULT_WEIGHTS };
|
|
95
|
+
for (const k of Object.keys(DEFAULT_WEIGHTS)) if (w && w[k] !== undefined && w[k] !== "") out[k] = Number(w[k]);
|
|
96
|
+
return out;
|
|
97
|
+
}
|
|
98
|
+
export function weightProblems(w) {
|
|
99
|
+
const p = [];
|
|
100
|
+
for (const [k, v] of Object.entries(w || {})) {
|
|
101
|
+
if (!(k in DEFAULT_WEIGHTS)) p.push(`risk.weights.${k}: unknown factor (use ${Object.keys(DEFAULT_WEIGHTS).join(", ")})`);
|
|
102
|
+
else if (!(Number(v) >= 0)) p.push(`risk.weights.${k} "${v}" must be a number ≥ 0`);
|
|
103
|
+
}
|
|
104
|
+
const sum = Object.values(normWeights(w)).reduce((a, b) => a + b, 0);
|
|
105
|
+
if (Math.abs(sum - 1) > 0.001) p.push(`risk.weights add up to ${Math.round(sum * 1000) / 1000}, not 1`);
|
|
106
|
+
return p;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
// The area file skills read: top N files, within the 150-line area-file limit.
|
|
110
|
+
export function riskMarkdown({ rows, repos, weights, since, top }) {
|
|
111
|
+
const pct = (x) => `${Math.round(x * 100)}%`;
|
|
112
|
+
const w = weights;
|
|
113
|
+
const shown = rows.filter((r) => r.score > 0).slice(0, Math.min(top, 120));
|
|
114
|
+
const lines = [
|
|
115
|
+
"# Risk",
|
|
116
|
+
"<!-- Generated by `spec-driven-dev risk`. Don't edit by hand: re-run it. Skills read this only when risk.enabled is on. -->",
|
|
117
|
+
`Scored ${repos.map((r) => `${r.name} at ${r.head} (${r.commits} commits)`).join(", ")} · since ${since} · ${new Date().toISOString().slice(0, 10)}`,
|
|
118
|
+
"",
|
|
119
|
+
`Risk = ${w.churn} × Churn + ${w.bug_density} × Bug Density + ${w.fan_out} × Fan-out + ${w.security} × Security (each 0–100; weights ${pct(w.churn)} / ${pct(w.bug_density)} / ${pct(w.fan_out)} / ${pct(w.security)}).`,
|
|
120
|
+
"Tiers: critical ≥ 75 · high ≥ 50 · medium ≥ 25. A historical signal, not proof of a bug.",
|
|
121
|
+
"",
|
|
122
|
+
"| File | Score | Tier | Churn | Bug density | Fan-out | Security | Commits (fixes / security) |",
|
|
123
|
+
"|---|---|---|---|---|---|---|---|",
|
|
124
|
+
...shown.map((r) => `| \`${r.file}\` | ${r.score} | ${r.tier} | ${r.factors.churn} | ${r.factors.bug_density} | ${r.factors.fan_out} | ${r.factors.security} | ${r.commits} (${r.bug_fixes} / ${r.security_fixes}) |`),
|
|
125
|
+
];
|
|
126
|
+
if (rows.length > shown.length) lines.push("", `${rows.length - shown.length} more files not shown. \`spec-driven-dev risk --files=a,b\` prints any file's score.`);
|
|
127
|
+
return lines.join("\n") + "\n";
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
// Change one `block: key: value` line in .spec-driven.yml text, keeping every comment and other line as is.
|
|
131
|
+
export function setConfigValue(text, blockName, key, value) {
|
|
132
|
+
const block = text.match(new RegExp(`^${blockName}:.*\\n(?:(?:[ \\t]+.*|[ \\t]*)\\n)*`, "m"));
|
|
133
|
+
const line = new RegExp(`^([ \\t]+${key}:[ \\t]*)[^\\s#]+`, "m");
|
|
134
|
+
if (!block) return `${text.replace(/\n*$/, "\n")}${blockName}:\n ${key}: ${value}\n`;
|
|
135
|
+
const body = line.test(block[0]) ? block[0].replace(line, `$1${value}`) : block[0].replace(new RegExp(`^${blockName}:.*\\n`), `$& ${key}: ${value}\n`);
|
|
136
|
+
return text.replace(block[0], body);
|
|
137
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@benjaminthomas/spec-driven-dev",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.11.0",
|
|
4
4
|
"description": "Spec-driven AI development workflow (investigate → spec → implement → verify → review → ship) as Agent Skills for Claude Code, Cursor, Codex CLI and Antigravity.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -8,7 +8,8 @@
|
|
|
8
8
|
},
|
|
9
9
|
"files": [
|
|
10
10
|
"bin",
|
|
11
|
-
"plugin/skills"
|
|
11
|
+
"plugin/skills",
|
|
12
|
+
"plugin/.claude-plugin"
|
|
12
13
|
],
|
|
13
14
|
"scripts": {
|
|
14
15
|
"test": "python3 scripts/check.py && node scripts/test-cli.mjs"
|
|
Binary file
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "ticket-to-pr",
|
|
3
|
+
"version": "1.11.0",
|
|
4
|
+
"description": "Parallel-batch spec-driven feature workflow (spec-setup, spec-investigate, spec-create, spec-implement, spec-verify, spec-review, spec-checkpoint, spec-ship) plus a portable SEO/GEO operations skill (generative-seo) for any codebase.",
|
|
5
|
+
"author": {
|
|
6
|
+
"name": "Benjamin Thomas",
|
|
7
|
+
"url": "https://github.com/benjaminthomas"
|
|
8
|
+
},
|
|
9
|
+
"keywords": [
|
|
10
|
+
"spec-driven-development",
|
|
11
|
+
"feature-workflow",
|
|
12
|
+
"parallel-agents",
|
|
13
|
+
"seo",
|
|
14
|
+
"geo",
|
|
15
|
+
"productivity"
|
|
16
|
+
],
|
|
17
|
+
"homepage": "https://github.com/benjaminthomas/spec-driven-dev",
|
|
18
|
+
"repository": "https://github.com/benjaminthomas/spec-driven-dev",
|
|
19
|
+
"icon": ".claude-plugin/icon.png"
|
|
20
|
+
}
|
|
@@ -67,7 +67,9 @@ Skip this step unless the changes belong to a spec (`{specs_dir}/{feature}/` exi
|
|
|
67
67
|
references/context-template.md inside the installed `spec-setup` skill folder (not this repo): dated, with a source, added at the end, within the size
|
|
68
68
|
limits, superseded lines moved to `context/archive.md`.
|
|
69
69
|
- Show the context changes in the report so the user can correct them.
|
|
70
|
-
3.
|
|
70
|
+
3. If `risk.enabled` is on and the CLI is available, re-run `spec-driven-dev risk` so the scores
|
|
71
|
+
include this work. It's git-only, so it costs no tokens.
|
|
72
|
+
4. No CONTEXT.md? Skip the distilling and suggest running `spec-setup` to create one.
|
|
71
73
|
|
|
72
74
|
### Step 3: Stage Everything
|
|
73
75
|
|
|
@@ -45,6 +45,18 @@ the answers to `.spec-driven.yml` (keys: `layout`, `repos`, `tracker`, `git`, `c
|
|
|
45
45
|
- **Start the memory.** Create or extend `log.md` (`references/log-template.md`) and fill the
|
|
46
46
|
README's **Resume here** block.
|
|
47
47
|
|
|
48
|
+
## Risk signal (GitRisk)
|
|
49
|
+
|
|
50
|
+
Same rule as `spec-investigate`: use it only when `risk.enabled` is on (or absent) and
|
|
51
|
+
`{specs_dir}/context/risk.md` exists, and ignore risk completely when it's `off`. For each task that
|
|
52
|
+
creates or modifies a **high** or **critical** file (from the investigation's **Risk Hotspots**, or
|
|
53
|
+
`spec-driven-dev risk --files=…`):
|
|
54
|
+
|
|
55
|
+
- **Raise its tier by one** (light → standard → deep), and say why in Complexity's Signals.
|
|
56
|
+
- Fill the task's **Risk** section: the file, its score, and the past failure pattern.
|
|
57
|
+
- Its **Tests** must include at least one test aimed at that past failure pattern, not just the new
|
|
58
|
+
behaviour.
|
|
59
|
+
|
|
48
60
|
## When to Use
|
|
49
61
|
|
|
50
62
|
- After a planning conversation where requirements and technical details have been discussed
|
|
@@ -35,6 +35,12 @@ pending
|
|
|
35
35
|
|
|
36
36
|
**Context from dependencies:** {Prose summary of what the dependency tasks produce that this task needs. For example: "task-01-setup-database.md creates the PostgreSQL schema with users and sessions tables. This task builds the API routes that query those tables." This is what makes each file self-contained — the agent does not need to read other task files to understand its inputs.}
|
|
37
37
|
|
|
38
|
+
## Risk
|
|
39
|
+
|
|
40
|
+
{Only when the task touches a high/critical file and risk.enabled is on; otherwise write "None".}
|
|
41
|
+
|
|
42
|
+
- `path/to/file.ts`: {score} {tier}. Past failures: {what the fix commits show keeps breaking}. Test: {the test in Tests aimed at it}
|
|
43
|
+
|
|
38
44
|
## Project Context
|
|
39
45
|
|
|
40
46
|
{Only the 3–5 lines from the project context (CONTEXT.md / context/<area>.md, via the investigation's Relevant Context) that apply to THIS task: conventions, pitfalls, off-limits areas. The coder never reads the context files itself. Write "None" if nothing applies.}
|
|
@@ -25,7 +25,8 @@ You are reviewing the combined output of Batch {batch_number} of a feature imple
|
|
|
25
25
|
- No duplicate or conflicting definitions
|
|
26
26
|
- Shared state (env vars, config) is consistent
|
|
27
27
|
4. **Code Quality**: Check for security issues, performance problems, and adherence to project conventions.
|
|
28
|
-
5. **
|
|
28
|
+
5. **Risk**: for every task with a filled **Risk** section, check the code handles the past failure pattern it names, and that the test aimed at it exists and would fail without the fix.
|
|
29
|
+
6. **Scope**: Verify each task stayed within its defined scope (Files to Create/Modify). Flag any unexpected file changes.
|
|
29
30
|
|
|
30
31
|
## Verdict
|
|
31
32
|
|
|
@@ -65,6 +65,22 @@ Cheap sources first, code second:
|
|
|
65
65
|
|
|
66
66
|
No CONTEXT.md? Investigate as usual, and suggest running `spec-setup` afterwards to create one.
|
|
67
67
|
|
|
68
|
+
## Risk signal (GitRisk)
|
|
69
|
+
|
|
70
|
+
Use it only when `.spec-driven.yml` has `risk.enabled: on`, or has no `risk:` block, **and**
|
|
71
|
+
`{specs_dir}/context/risk.md` exists. If `risk.enabled: off`, ignore risk entirely: don't read the
|
|
72
|
+
file, compute scores, or mention risk. That keeps on/off comparisons (`spec-driven-dev risk-eval`)
|
|
73
|
+
clean.
|
|
74
|
+
|
|
75
|
+
1. Once you know the change surface, look up its files: `spec-driven-dev risk --files=a,b` (or
|
|
76
|
+
`npx -y @benjaminthomas/spec-driven-dev risk --files=…`), or find them in `risk.md`. Without the
|
|
77
|
+
CLI, approximate a file's history with `git log --since="2 years ago" --format=%s -- <file>`.
|
|
78
|
+
2. For each **high** or **critical** file in the surface, look harder: read its last ~10 commit
|
|
79
|
+
subjects and the diffs of its last 1–2 fix commits, and check its dependents. Write what keeps
|
|
80
|
+
breaking there in the report's **Risk Hotspots** section.
|
|
81
|
+
3. A score is a reason to look harder, never evidence. Hotspot findings still need their
|
|
82
|
+
VERIFIED / INFERRED labels.
|
|
83
|
+
|
|
68
84
|
## Invocation
|
|
69
85
|
|
|
70
86
|
```
|
|
@@ -22,6 +22,11 @@ Copy this structure into `specs/{feature-slug}/investigation.md`.
|
|
|
22
22
|
- Conventions / decisions / pitfalls that apply: (one line each, with the source)
|
|
23
23
|
- Context corrections: (where the code contradicts CONTEXT.md, with evidence; or "none")
|
|
24
24
|
|
|
25
|
+
## Risk Hotspots
|
|
26
|
+
<!-- Only when risk.enabled is on. One row per high/critical file in the change surface; else "none" or omit. -->
|
|
27
|
+
| File | Score / tier | What has broken here before (from fix commits) | What to watch in this change |
|
|
28
|
+
|---|---|---|---|
|
|
29
|
+
|
|
25
30
|
## Business Context
|
|
26
31
|
## Problem Statement
|
|
27
32
|
## Expected Behaviour
|
|
@@ -74,7 +74,11 @@ review on a weaker model than the code was written with misses things.
|
|
|
74
74
|
3. **The code.** Correctness (logic, edge cases, error paths, races), security (injection, authz,
|
|
75
75
|
secrets, unsafe input), data safety (migrations, deletes, irreversible writes), tests (behaviour
|
|
76
76
|
without a test, tests weakened, skipped or deleted), and performance on hot paths only.
|
|
77
|
-
|
|
77
|
+
**Risk.** When `risk.enabled` is on and `{specs_dir}/context/risk.md` exists, look up the changed
|
|
78
|
+
files (`spec-driven-dev risk --files=…`). Give **high** and **critical** files a line-by-line read
|
|
79
|
+
of their error paths and callers, check that the tests target their past failure patterns, and
|
|
80
|
+
put the score in any finding on them. When risk is `off`, skip this and don't mention it.
|
|
81
|
+
4. **Run the checks.**4. **Run the checks.** Run `commands.lint`, `typecheck`, and `test` once if verify didn't just run
|
|
78
82
|
them on this exact commit. A failing check is a blocker.
|
|
79
83
|
|
|
80
84
|
**Verify every finding before reporting it.** Re-read the code around it and ask "what concrete
|
|
@@ -8,7 +8,7 @@ description: >
|
|
|
8
8
|
plus one round of questions, for a new one the intended stack and conventions, saved to a small
|
|
9
9
|
specs/CONTEXT.md that later skills load cheaply. Use when the user says "set up spec-driven",
|
|
10
10
|
"configure the pipeline", "spec-setup", "audit this repo", "refresh the project context",
|
|
11
|
-
"change our branch rules", or when any other spec-* skill finds no .spec-driven.yml or no
|
|
11
|
+
"change our branch rules", "turn GitRisk on/off", "turn on run-through", or when any other spec-* skill finds no .spec-driven.yml or no
|
|
12
12
|
CONTEXT.md. Detects everything it can first and only asks about what is ambiguous. Writes config
|
|
13
13
|
and context files only — never changes code, branches, or commits.
|
|
14
14
|
license: MIT
|
|
@@ -118,13 +118,33 @@ Skip if `{specs_dir}/CONTEXT.md` already exists (offer a refresh instead, see `r
|
|
|
118
118
|
intended stack and conventions for a new one. One round of questions, no more.
|
|
119
119
|
2. Write `{specs_dir}/CONTEXT.md` (and any area files) using `references/context-template.md`.
|
|
120
120
|
Keep the index to 60 lines and area files to 150.
|
|
121
|
-
3. **
|
|
121
|
+
3. **Risk scores.** For an existing project, run `spec-driven-dev risk` (or
|
|
122
|
+
`npx -y @benjaminthomas/spec-driven-dev risk`) to write `{specs_dir}/context/risk.md`. It uses
|
|
123
|
+
git only, no model. Skip it for a new project with no history. Mention the hotspot count in the
|
|
124
|
+
report. Set `risk.enabled: off` only if the user doesn't want it.
|
|
125
|
+
4. **Offer, don't do:** ask whether to add this one line to `CLAUDE.md` / `AGENTS.md` (whichever
|
|
122
126
|
the project uses), so hosts reload the pointer after compaction and in every new session:
|
|
123
127
|
`For spec work, read specs/CONTEXT.md first, then the "Resume here" block of any unfinished spec.`
|
|
124
128
|
Add it only on a yes.
|
|
125
|
-
|
|
129
|
+
5. Re-run `npx -y @benjaminthomas/spec-driven-dev doctor` if available. It checks the size limits
|
|
126
130
|
and flags context lines that point to files which no longer exist.
|
|
127
131
|
|
|
132
|
+
## Quick switches (plain words, no YAML)
|
|
133
|
+
|
|
134
|
+
When the user asks to switch a feature on or off, change only that one line of `.spec-driven.yml`,
|
|
135
|
+
keep every other line and comment, and confirm in one sentence. Don't re-run discovery.
|
|
136
|
+
|
|
137
|
+
| They say | Set |
|
|
138
|
+
|---|---|
|
|
139
|
+
| "turn GitRisk off", "stop using risk scores", "ignore risky files" | `risk.enabled: off` |
|
|
140
|
+
| "turn GitRisk on", "use the risk scores" | `risk.enabled: on`, then run `spec-driven-dev risk` if `{specs_dir}/context/risk.md` doesn't exist yet |
|
|
141
|
+
| "run straight through after I approve", "fewer stops" | `pipeline.after_approval: run-through` |
|
|
142
|
+
| "stop after each stage" | `pipeline.after_approval: stop` |
|
|
143
|
+
|
|
144
|
+
Reply with what changed and what it means, for example: "GitRisk is now off. Investigations, plans
|
|
145
|
+
and reviews will ignore risk scores until you turn it back on." People who prefer a menu can run
|
|
146
|
+
`spec-driven-dev settings` in a terminal.
|
|
147
|
+
|
|
128
148
|
## Updating
|
|
129
149
|
|
|
130
150
|
Re-running `spec-setup` with an existing config: load it, re-detect, and show only what changed or
|
|
@@ -75,6 +75,18 @@ effort:
|
|
|
75
75
|
deep: high
|
|
76
76
|
escalate: xhigh
|
|
77
77
|
|
|
78
|
+
# GitRisk: historical risk per file from git alone (`spec-driven-dev risk` writes {specs_dir}/context/risk.md).
|
|
79
|
+
# Risk = churn × 0.30 + bug_density × 0.35 + fan_out × 0.20 + security × 0.15 (each factor 0–100).
|
|
80
|
+
# The weights are a starting hypothesis; calibrate them against your own past bugs.
|
|
81
|
+
risk:
|
|
82
|
+
enabled: on # on | off — off makes every skill ignore risk (used by risk-eval's A/B)
|
|
83
|
+
since: "2 years ago" # how much history to score
|
|
84
|
+
weights:
|
|
85
|
+
churn: 0.30 # how often the file changes, vs the repo's 90th percentile
|
|
86
|
+
bug_density: 0.35 # share of its commits that were bug fixes
|
|
87
|
+
fan_out: 0.20 # how many other files import it, vs the repo's 90th percentile
|
|
88
|
+
security: 0.15 # share of its commits that were security fixes
|
|
89
|
+
|
|
78
90
|
pipeline:
|
|
79
91
|
# stop — every stage waits for you (default)
|
|
80
92
|
# run-through — after you approve a spec: implement → verify → review → checkpoint in one go,
|