@benjaminthomas/spec-driven-dev 1.10.0 → 1.11.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +94 -9
- package/bin/cli.mjs +150 -10
- package/bin/risk-eval.mjs +152 -0
- package/bin/risk.mjs +169 -0
- package/package.json +3 -2
- package/plugin/.claude-plugin/icon.png +0 -0
- package/plugin/.claude-plugin/plugin.json +20 -0
- package/plugin/skills/spec-checkpoint/SKILL.md +10 -1
- package/plugin/skills/spec-create/SKILL.md +15 -0
- package/plugin/skills/spec-create/references/task-template.md +6 -0
- package/plugin/skills/spec-implement/SKILL.md +5 -0
- package/plugin/skills/spec-implement/references/review-prompt-template.md +2 -1
- package/plugin/skills/spec-investigate/SKILL.md +19 -0
- package/plugin/skills/spec-investigate/references/report-template.md +5 -0
- package/plugin/skills/spec-review/SKILL.md +7 -1
- package/plugin/skills/spec-setup/SKILL.md +23 -3
- package/plugin/skills/spec-setup/references/config-template.yml +12 -0
package/README.md
CHANGED
|
@@ -46,6 +46,7 @@ Asking an AI agent to "just build it" tends to fail in predictable ways:
|
|
|
46
46
|
| One agent working alone is slow | **`spec-implement`** runs several coder agents in parallel, each in its own copy of the repo |
|
|
47
47
|
| "It compiles" gets treated as "it works" | **`spec-verify`** drives the real running app and checks every acceptance criterion, with evidence |
|
|
48
48
|
| Problems slip through because nobody reviews the whole change | **`spec-review`** checks the finished branch against the spec, your conventions, and the code, and blocks shipping on real problems |
|
|
49
|
+
| The agent treats a file that broke five times like any other | **GitRisk** scores files from git history (churn, bug fixes, dependents, security fixes), so risky files get a deeper look, a higher tier and targeted tests. Switch it on or off any time |
|
|
49
50
|
| A new session forgets what was decided last time | **Project memory**: a small `specs/CONTEXT.md`, plus a "Resume here" block and a log in every spec, kept short so reloading them is cheap |
|
|
50
51
|
| Broken code gets committed, or a PR is pushed by accident | **`spec-checkpoint`** commits only once lint, types, tests, and build all pass. **`spec-ship`** asks before it pushes |
|
|
51
52
|
|
|
@@ -246,6 +247,69 @@ lines that point to files that no longer exist. If you install
|
|
|
246
247
|
[graphify](https://github.com/Graphify-Labs/graphify), the skills use it for cheaper code
|
|
247
248
|
navigation. It's optional (`context.graph: off` turns it off), and nothing is installed for you.
|
|
248
249
|
|
|
250
|
+
### GitRisk: which files have a history of breaking
|
|
251
|
+
|
|
252
|
+
`spec-driven-dev risk` scores every tracked code file from git history alone, with no model and no
|
|
253
|
+
tokens, and writes the top files to `specs/context/risk.md`:
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
Risk = 0.30 × Churn + 0.35 × Bug Density + 0.20 × Fan-out + 0.15 × Security (each factor 0–100)
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
| Factor | Weight | What it measures |
|
|
260
|
+
|---|---|---|
|
|
261
|
+
| Churn | 30% | how often the file changes, against the repo's 90th percentile |
|
|
262
|
+
| Bug Density | 35% | share of its commits that were bug fixes (`fix:` type or fix-like wording) |
|
|
263
|
+
| Fan-out | 20% | how many other files import it, against the repo's 90th percentile |
|
|
264
|
+
| Security | 15% | share of its commits that were security fixes (security, CVE, XSS, injection…) |
|
|
265
|
+
|
|
266
|
+
Tiers: critical ≥ 75, high ≥ 50, medium ≥ 25. Files added in the last 90 days with fewer than 3
|
|
267
|
+
commits, or not committed yet, are labelled **new** instead of being scored, because too little history isn't the same as
|
|
268
|
+
safe. Each new file shows the riskiest file it imports or is imported by, and inherits that file's
|
|
269
|
+
care when it's high or critical.
|
|
270
|
+
|
|
271
|
+
**When it updates:** setup scores the whole history once. After that it re-scores at every batch
|
|
272
|
+
commit in `spec-implement` and at `spec-checkpoint`. Each run takes under a second and uses no model,
|
|
273
|
+
so later batches see the new files and the extra fan-out they add. Lookups for the files a ticket
|
|
274
|
+
touches (`--files=`) always read the live history. Commits made outside the pipeline are picked up
|
|
275
|
+
at the next run. It doesn't hook into every commit.
|
|
276
|
+
|
|
277
|
+
`risk.md` is regenerated on every run, so the first run adds it to `.gitignore`, the way you'd ignore
|
|
278
|
+
a build output. Committing it would cause merge conflicts between branches. `doctor` flags it if
|
|
279
|
+
it's committed.
|
|
280
|
+
|
|
281
|
+
When `risk.enabled` is on, the skills use it:
|
|
282
|
+
- `spec-investigate` digs into high and critical files in the change surface (past fix commits,
|
|
283
|
+
dependents) and lists them as **Risk Hotspots**;
|
|
284
|
+
- `spec-create` raises those tasks one tier and requires a test aimed at each file's past failure
|
|
285
|
+
pattern;
|
|
286
|
+
- `spec-implement`'s reviewer and `spec-review` check those tests and read the risky files line
|
|
287
|
+
by line.
|
|
288
|
+
|
|
289
|
+
**Turning it on or off, no YAML needed:**
|
|
290
|
+
- **In your AI tool:** say "turn GitRisk off" or "turn GitRisk on". `spec-setup` changes just that
|
|
291
|
+
setting and tells you what it means.
|
|
292
|
+
- **In a terminal:** run `spec-driven-dev settings` and pick with the arrow keys. It also switches
|
|
293
|
+
run-through mode. For scripts, use `spec-driven-dev settings --risk=off`.
|
|
294
|
+
- **By hand:** set `risk.enabled: on` or `off` in `.spec-driven.yml`.
|
|
295
|
+
|
|
296
|
+
Off means every skill ignores risk completely.
|
|
297
|
+
|
|
298
|
+
`spec-driven-dev risk --files=a.ts,b.ts` prints scores for chosen files. The weights are a starting
|
|
299
|
+
hypothesis: change them under `risk.weights` in `.spec-driven.yml` once you've backtested them
|
|
300
|
+
against your own past bugs.
|
|
301
|
+
|
|
302
|
+
**Does it change what the agent does?** `spec-driven-dev risk-eval --tickets=tickets.md` runs each
|
|
303
|
+
ticket twice, in fresh clones with risk on and off, through investigate and create on the same
|
|
304
|
+
model. It then compares what the agent did:
|
|
305
|
+
- risky files read and named in the investigation;
|
|
306
|
+
- risky files in the plan, and whether tests were planned for them;
|
|
307
|
+
- task tiers;
|
|
308
|
+
- cost.
|
|
309
|
+
|
|
310
|
+
Results go to `experiments/` with the transcripts, and every run, including failures, is appended to
|
|
311
|
+
`experiments/LOG.md`. It needs Claude Code.
|
|
312
|
+
|
|
249
313
|
## How the workflow fits together
|
|
250
314
|
|
|
251
315
|
```
|
|
@@ -314,6 +378,9 @@ It starts from the project memory (`specs/CONTEXT.md` and only the area files th
|
|
|
314
378
|
treats it as a lead rather than evidence, and notes where the code contradicts it. If graphify is
|
|
315
379
|
installed, it uses it to map the code more cheaply.
|
|
316
380
|
|
|
381
|
+
With GitRisk on, it also looks harder at high-risk files in the change surface (their past fix
|
|
382
|
+
commits and dependents) and lists them as **Risk Hotspots**.
|
|
383
|
+
|
|
317
384
|
**You get:** `specs/<feature>/investigation.md`, with every finding labelled **VERIFIED**,
|
|
318
385
|
**INFERRED**, **ASSUMPTION**, or **UNKNOWN**, a short **Relevant Context** digest for the next
|
|
319
386
|
stages, and requirements numbered R1, R2… so they can be traced through to verification and review.
|
|
@@ -335,6 +402,9 @@ to write**, acceptance criteria, the requirements it **covers**, and the few **p
|
|
|
335
402
|
that apply to it. Every requirement must map to a task, and the spec isn't finished until it does.
|
|
336
403
|
It also starts the spec's `log.md` and **Resume here** block.
|
|
337
404
|
|
|
405
|
+
With GitRisk on, a task that touches a high-risk file goes up one tier and must include a test
|
|
406
|
+
aimed at that file's past failures.
|
|
407
|
+
|
|
338
408
|
**Then:** read it, edit anything you disagree with, and only then run `spec-implement`.
|
|
339
409
|
|
|
340
410
|
### 3. `spec-implement`: build it, batch by batch
|
|
@@ -380,6 +450,8 @@ nothing out of scope), the project's recorded conventions, and the code itself:
|
|
|
380
450
|
security, error handling, tests, and hot-path performance. It checks each finding against the code
|
|
381
451
|
before reporting it, and reads the diff rather than the whole repo.
|
|
382
452
|
|
|
453
|
+
With GitRisk on, high-risk files get a line-by-line read of their error paths and callers.
|
|
454
|
+
|
|
383
455
|
**You get:** `specs/<feature>/review.md`, with findings ranked blocker, should-fix, nit, or question,
|
|
384
456
|
each with `file:line` and a suggested fix. Blockers stop `spec-ship` and run-through. It then asks
|
|
385
457
|
whether to fix them, and fixes go through `spec-implement`'s fix loop only after you say yes. It also
|
|
@@ -443,6 +515,8 @@ specs/
|
|
|
443
515
|
└── review.md ← spec-review: ranked findings; open blockers stop spec-ship
|
|
444
516
|
specs/CONTEXT.md ← spec-setup / spec-checkpoint: project memory index (≤60 lines)
|
|
445
517
|
specs/context/<area>.md ← details per area, loaded only when needed (≤150 lines)
|
|
518
|
+
specs/context/risk.md ← spec-driven-dev risk: GitRisk scores (regenerate; don't edit)
|
|
519
|
+
experiments/ ← spec-driven-dev risk-eval: on/off reports, transcripts, LOG.md
|
|
446
520
|
```
|
|
447
521
|
|
|
448
522
|
These are plain Markdown files, so you can review them in a PR, edit them, or hand them to a
|
|
@@ -475,6 +549,11 @@ Built into the skills, not left to the agent's judgment:
|
|
|
475
549
|
|
|
476
550
|
## FAQ
|
|
477
551
|
|
|
552
|
+
**How do I turn GitRisk off (or on)?**
|
|
553
|
+
Say "turn GitRisk off" in your AI tool, or run `spec-driven-dev settings` and pick from the menu.
|
|
554
|
+
Off means every skill ignores risk scores. Turning it on again offers to score the files if that
|
|
555
|
+
hasn't been done yet.
|
|
556
|
+
|
|
478
557
|
**Do I need Jira?**
|
|
479
558
|
No. Use GitHub Issues, Linear, a transcript, a document, or a sentence of plain text. Jira is just
|
|
480
559
|
one of the options.
|
|
@@ -565,6 +644,10 @@ spec-driven-dev --yes # accept every detected defaul
|
|
|
565
644
|
spec-driven-dev doctor # check .spec-driven.yml against the real repos
|
|
566
645
|
spec-driven-dev status # list unfinished specs and each one's next step
|
|
567
646
|
spec-driven-dev context --areas=billing # print the project memory (index, or chosen areas)
|
|
647
|
+
spec-driven-dev settings # switch GitRisk and run-through on/off from a menu
|
|
648
|
+
spec-driven-dev risk # GitRisk: score files from git history → specs/context/risk.md
|
|
649
|
+
spec-driven-dev risk-eval --tickets=tickets.md # compare agent behaviour with the risk signal on vs off
|
|
650
|
+
# (--runs=3 --arms=on --timeout=20 --model=sonnet)
|
|
568
651
|
spec-driven-dev --help # show all of the above
|
|
569
652
|
```
|
|
570
653
|
|
|
@@ -658,16 +741,16 @@ skips that step rather than failing.
|
|
|
658
741
|
|
|
659
742
|
| Skill | Stage | SKILL.md lines | Words | Supporting files |
|
|
660
743
|
|---|---|---:|---:|---:|
|
|
661
|
-
| `spec-setup` | 0 · configure |
|
|
662
|
-
| `spec-investigate` | 1 · understand |
|
|
663
|
-
| `spec-create` | 2 · plan |
|
|
664
|
-
| `spec-implement` | 3 · build |
|
|
744
|
+
| `spec-setup` | 0 · configure | 162 | 1,545 | 3 |
|
|
745
|
+
| `spec-investigate` | 1 · understand | 373 | 3,211 | 1 |
|
|
746
|
+
| `spec-create` | 2 · plan | 198 | 1,782 | 6 |
|
|
747
|
+
| `spec-implement` | 3 · build | 327 | 2,753 | 3 |
|
|
665
748
|
| `spec-verify` | 4 · prove | 163 | 1,557 | 1 |
|
|
666
|
-
| `spec-review` | 5 · review |
|
|
667
|
-
| `spec-checkpoint` | 6 · commit |
|
|
749
|
+
| `spec-review` | 5 · review | 137 | 1,294 | 1 |
|
|
750
|
+
| `spec-checkpoint` | 6 · commit | 140 | 1,144 | 0 |
|
|
668
751
|
| `spec-ship` | 7 · PR | 230 | 1,388 | 0 |
|
|
669
752
|
| `generative-seo` | standalone | 255 | 2,209 | 7 |
|
|
670
|
-
| **Total** | | **1,
|
|
753
|
+
| **Total** | | **1,985** | **16,883** | **22** |
|
|
671
754
|
|
|
672
755
|
- **8** skills: **1** setup + **6** workflow stages + **1** standalone
|
|
673
756
|
- **4** tools supported natively (Claude Code, Codex CLI, Cursor, Antigravity), plus an MCP fallback
|
|
@@ -688,7 +771,9 @@ plugin/skills/<name>/references/ templates and prompts the skill loads when need
|
|
|
688
771
|
plugin/ the Claude Code plugin (manifest + skills only, no scripts or hooks)
|
|
689
772
|
.agents/skills/ symlinks so Codex/Antigravity pick the skills up inside this repo
|
|
690
773
|
bin/spec-driven-dev.cjs launcher (runs on any Node, explains if it's too old)
|
|
691
|
-
bin/cli.mjs the `spec-driven-dev` CLI: installer, setup menus, doctor, status, context
|
|
774
|
+
bin/cli.mjs the `spec-driven-dev` CLI: installer, setup menus, doctor, status, context, risk
|
|
775
|
+
bin/risk.mjs GitRisk scoring (churn, bug density, fan-out, security)
|
|
776
|
+
bin/risk-eval.mjs risk on/off experiment harness
|
|
692
777
|
package.json npm package (ships bin/ and plugin/skills/)
|
|
693
778
|
mcp-server/ MCP fallback server
|
|
694
779
|
CHANGELOG.md what changed in each version
|
|
@@ -717,7 +802,7 @@ plugin/evals/ `claude plugin eval` cases
|
|
|
717
802
|
1. Increase `version` in both `package.json` and `plugin/.claude-plugin/plugin.json` (the check fails if
|
|
718
803
|
they differ).
|
|
719
804
|
2. Add a `CHANGELOG.md` entry, then commit and push.
|
|
720
|
-
3. Tag the release and push the tag: `git tag v1.
|
|
805
|
+
3. Tag the release and push the tag: `git tag v1.11.1 && git push origin v1.11.1`.
|
|
721
806
|
4. Watch the **publish** run in the repo's Actions tab. When it's green, the new version is on
|
|
722
807
|
npm. Claude Code plugin users get it from the marketplace once the commit is on `main`.
|
|
723
808
|
|
package/bin/cli.mjs
CHANGED
|
@@ -7,15 +7,19 @@
|
|
|
7
7
|
// spec-driven-dev doctor check .spec-driven.yml against the real repos (exit 1 on problems)
|
|
8
8
|
// spec-driven-dev status list specs with unfinished work and each one's "Resume here" next step
|
|
9
9
|
// spec-driven-dev context print specs/CONTEXT.md; --areas=billing,api prints only those area files
|
|
10
|
+
// spec-driven-dev risk GitRisk: score files from git history into specs/context/risk.md (--files=a,b --json --top=N --since=…)
|
|
11
|
+
// spec-driven-dev settings switch features on or off from a menu (GitRisk, run-through); or --risk=on|off --after-approval=…
|
|
12
|
+
// spec-driven-dev risk-eval U8 experiment: run tickets with risk on vs off and compare agent behaviour (--tickets=<file> --runs=N --model=… --arms=on,off --timeout=20)
|
|
10
13
|
// flags: --yes --global --tools=claude,cursor --skills=spec-create,... --no-setup --dir=<path>
|
|
11
14
|
// --layout=single|monorepo|multi-repo --repos=~/Projects/frontend,/srv/backend --workspace=<config dir>
|
|
12
|
-
// --packages=apps/web,packages/ui --after-approval=stop|run-through --project=new|existing --version
|
|
15
|
+
// --packages=apps/web,packages/ui --after-approval=stop|run-through --project=new|existing --risk=on|off --version
|
|
13
16
|
import fs from "node:fs";
|
|
14
17
|
import os from "node:os";
|
|
15
18
|
import path from "node:path";
|
|
16
19
|
import { execFileSync } from "node:child_process";
|
|
17
20
|
import prompts from "prompts"; // arrow-key menus that work in PowerShell, cmd, macOS and Linux terminals
|
|
18
21
|
import { fileURLToPath } from "node:url";
|
|
22
|
+
import { scoreRepo, riskMarkdown, normWeights, weightProblems, setConfigValue } from "./risk.mjs";
|
|
19
23
|
|
|
20
24
|
const ROOT = path.dirname(path.dirname(fileURLToPath(import.meta.url)));
|
|
21
25
|
const SKILLS_DIR = path.join(ROOT, "plugin", "skills");
|
|
@@ -34,7 +38,7 @@ const args = process.argv.slice(2);
|
|
|
34
38
|
const flag = (n) => args.includes(`--${n}`);
|
|
35
39
|
const opt = (n) => args.find((a) => a.startsWith(`--${n}=`))?.split("=")[1];
|
|
36
40
|
const command = args.find((a) => !a.startsWith("-")) || "install";
|
|
37
|
-
const YES = flag("yes") || flag("y") || !process.stdin.isTTY || ["doctor", "status", "context"].includes(command);
|
|
41
|
+
const YES = flag("yes") || flag("y") || !process.stdin.isTTY || ["doctor", "status", "context", "risk", "risk-eval"].includes(command);
|
|
38
42
|
const DIR = path.resolve(opt("dir") || process.cwd());
|
|
39
43
|
|
|
40
44
|
if (flag("version") || flag("v")) {
|
|
@@ -43,22 +47,23 @@ if (flag("version") || flag("v")) {
|
|
|
43
47
|
}
|
|
44
48
|
// No terminal to answer in (e.g. `!` in Claude Code, an IDE task, a pipe) and nothing passed explicitly:
|
|
45
49
|
// stop rather than write a config from guesses.
|
|
46
|
-
const ANSWER_FLAGS = ["yes", "y", "layout", "repos", "packages", "tools", "workspace", "after-approval", "skills", "global", "no-setup", "project"];
|
|
47
|
-
if (["install", "setup"].includes(command) && !process.stdin.isTTY && !args.some((a) => ANSWER_FLAGS.some((f) => a === `--${f}` || a.startsWith(`--${f}=`)))) {
|
|
48
|
-
console.log(`✗ No interactive terminal, so setup can't ask its questions. Nothing was changed.
|
|
50
|
+
const ANSWER_FLAGS = ["risk", "yes", "y", "layout", "repos", "packages", "tools", "workspace", "after-approval", "skills", "global", "no-setup", "project"];
|
|
51
|
+
if (["install", "setup", "settings"].includes(command) && !process.stdin.isTTY && !args.some((a) => ANSWER_FLAGS.some((f) => a === `--${f}` || a.startsWith(`--${f}=`)))) {
|
|
52
|
+
console.log(`✗ No interactive terminal, so ${command === "settings" ? "the settings menu can't open" : "setup can't ask its questions"}. Nothing was changed.
|
|
49
53
|
|
|
50
54
|
This happens when it runs through another program: a "!" command in Claude Code, an editor task,
|
|
51
55
|
or piped input. Either:
|
|
52
56
|
• run it yourself in a terminal window (Terminal, iTerm, or your editor's terminal panel):
|
|
53
|
-
npx @benjaminthomas/spec-driven-dev@latest ${command === "
|
|
57
|
+
npx @benjaminthomas/spec-driven-dev@latest ${command === "install" ? "" : command}
|
|
54
58
|
• or give the answers as flags, for example:
|
|
55
|
-
npx @benjaminthomas/spec-driven-dev@latest setup --layout=multi-repo --repos=~/Projects/frontend,~/Projects/backend
|
|
59
|
+
npx @benjaminthomas/spec-driven-dev@latest ${command === "settings" ? "settings --risk=off" : "setup --layout=multi-repo --repos=~/Projects/frontend,~/Projects/backend"}
|
|
56
60
|
(add --yes to accept everything it detects)`);
|
|
57
61
|
process.exit(1);
|
|
58
62
|
}
|
|
59
63
|
|
|
60
64
|
if (flag("help") || flag("h")) {
|
|
61
|
-
|
|
65
|
+
const lines = fs.readFileSync(fileURLToPath(import.meta.url), "utf8").split("\n").slice(1);
|
|
66
|
+
console.log(lines.slice(0, lines.findIndex((l) => !l.startsWith("//"))).join("\n").replace(/^\/\/ ?/gm, "")); // the whole header comment
|
|
62
67
|
process.exit(0);
|
|
63
68
|
}
|
|
64
69
|
|
|
@@ -241,6 +246,7 @@ function toYaml(c) {
|
|
|
241
246
|
out += "effort: # effort / reasoning level per tier (Claude Code, Codex): low | medium | high | xhigh | max | default\n"
|
|
242
247
|
+ MODEL_KEYS.map((t) => ` ${t}: ${EFFORT_DEFAULTS[t]}\n`).join("");
|
|
243
248
|
out += `context: # project memory: {specs_dir}/CONTEXT.md (index) + context/<area>.md\n project: ${c.project || "existing"} # existing | new\n graph: auto # auto = use graphify if installed | off\n`;
|
|
249
|
+
out += "risk: # GitRisk: historical risk per file from git (spec-driven-dev risk). Weights are a hypothesis to calibrate\n enabled: on # on | off (off = skills ignore risk; risk-eval compares the two)\n since: \"2 years ago\"\n weights:\n churn: 0.30\n bug_density: 0.35\n fan_out: 0.20\n security: 0.15\n";
|
|
244
250
|
out += `pipeline:\n after_approval: ${c.afterApproval || "stop"} # stop | run-through (implement → verify → review → checkpoint; ship always asks)\n`;
|
|
245
251
|
return out + `specs_dir: ${y(c.specsDir)}\n`;
|
|
246
252
|
}
|
|
@@ -515,10 +521,20 @@ function parseYaml(text) {
|
|
|
515
521
|
return root;
|
|
516
522
|
}
|
|
517
523
|
|
|
524
|
+
// A multi-repo config applies only in its own folder and inside the repos it lists, so a repo that
|
|
525
|
+
// merely sits under that folder (e.g. ~/Projects/other-project) doesn't pick up the workspace's rules.
|
|
526
|
+
function configApplies(f, dir) {
|
|
527
|
+
let c;
|
|
528
|
+
try { c = parseYaml(fs.readFileSync(f, "utf8")); } catch { return true; }
|
|
529
|
+
if (c.layout !== "multi-repo" || !Array.isArray(c.repos)) return true;
|
|
530
|
+
const here = real(dir), base = path.dirname(f);
|
|
531
|
+
return here === real(base) || c.repos.some((r) => { const p = real(path.resolve(base, expandHome(String(r.path || "")))); return here === p || here.startsWith(p + path.sep); });
|
|
532
|
+
}
|
|
533
|
+
|
|
518
534
|
function findConfig(dir) {
|
|
519
535
|
for (let d = dir; ; d = path.dirname(d)) {
|
|
520
536
|
const f = path.join(d, ".spec-driven.yml");
|
|
521
|
-
if (fs.existsSync(f)) return f;
|
|
537
|
+
if (fs.existsSync(f) && configApplies(f, dir)) return f;
|
|
522
538
|
if (d === path.dirname(d) || d === HOME) break;
|
|
523
539
|
}
|
|
524
540
|
// Not above us: a multi-repo workspace elsewhere may list this repo.
|
|
@@ -577,6 +593,14 @@ function doctor() {
|
|
|
577
593
|
if (c.context?.graph && !["auto", "off"].includes(c.context.graph)) problems.push(`context.graph "${c.context.graph}" must be auto or off`);
|
|
578
594
|
const repoRoots = c.layout === "multi-repo" ? (c.repos || []).map((r) => path.resolve(root, expandHome(r.path))) : [root];
|
|
579
595
|
checkContext(specs, c.specs_dir || "specs", repoRoots, problems, notes);
|
|
596
|
+
const rmd = path.join(specs, "context", "risk.md"), rtop = run("git", ["rev-parse", "--show-toplevel"], root);
|
|
597
|
+
if (rtop && fs.existsSync(rmd) && run("git", ["ls-files", path.relative(path.resolve(rtop), real(rmd))], rtop))
|
|
598
|
+
problems.push(`${c.specs_dir || "specs"}/context/risk.md is committed to git. It's regenerated on every run, so it causes merge conflicts. Run: git rm --cached ${path.relative(path.resolve(rtop), real(rmd))} (spec-driven-dev risk adds it to .gitignore)`);
|
|
599
|
+
if (c.risk) {
|
|
600
|
+
if (c.risk.enabled && !["on", "off"].includes(c.risk.enabled)) problems.push(`risk.enabled "${c.risk.enabled}" must be on or off`);
|
|
601
|
+
problems.push(...weightProblems(c.risk.weights));
|
|
602
|
+
if (c.risk.enabled !== "off" && !fs.existsSync(path.join(specs, "context", "risk.md"))) notes.push(`risk is on but ${c.specs_dir || "specs"}/context/risk.md doesn't exist — run: spec-driven-dev risk`);
|
|
603
|
+
}
|
|
580
604
|
console.log(`Checked ${file}`);
|
|
581
605
|
for (const n of notes) console.log(` • ${n}`);
|
|
582
606
|
for (const p of problems) console.log(` ✗ ${p}`);
|
|
@@ -620,6 +644,119 @@ function context() {
|
|
|
620
644
|
}
|
|
621
645
|
}
|
|
622
646
|
|
|
647
|
+
// risk.md is regenerated on every run (it carries a date), so committing it means noisy diffs and merge
|
|
648
|
+
// conflicts between branches. Keep it out of git like a build output; anyone can recreate it in a second.
|
|
649
|
+
function ignoreGenerated(file) {
|
|
650
|
+
const top = run("git", ["rev-parse", "--show-toplevel"], path.dirname(file));
|
|
651
|
+
if (!top) return; // e.g. a multi-repo workspace folder that isn't a repo
|
|
652
|
+
const rel = path.relative(path.resolve(top), real(file)).split(path.sep).join("/");
|
|
653
|
+
if (rel.startsWith("..") || run("git", ["check-ignore", "--no-index", rel], top)) return;
|
|
654
|
+
const gi = path.join(top, ".gitignore");
|
|
655
|
+
const text = fs.existsSync(gi) ? fs.readFileSync(gi, "utf8") : "";
|
|
656
|
+
fs.writeFileSync(gi, `${text}${text && !text.endsWith("\n") ? "\n" : ""}# GitRisk scores, regenerated by spec-driven-dev risk\n${rel}\n`);
|
|
657
|
+
console.log(` Added ${rel} to .gitignore (it's regenerated on every run, so it shouldn't be committed).`);
|
|
658
|
+
}
|
|
659
|
+
|
|
660
|
+
function loadConfig() {
|
|
661
|
+
const file = findConfig(DIR);
|
|
662
|
+
return { file, root: file ? path.dirname(file) : gitTop(DIR), c: file ? parseYaml(fs.readFileSync(file, "utf8")) : {} };
|
|
663
|
+
}
|
|
664
|
+
|
|
665
|
+
function risk() {
|
|
666
|
+
const { root, c } = loadConfig();
|
|
667
|
+
const bad = weightProblems(c.risk?.weights);
|
|
668
|
+
if (bad.length) throw new Error(bad.join("; "));
|
|
669
|
+
const weights = normWeights(c.risk?.weights), since = opt("since") || c.risk?.since || "2 years ago";
|
|
670
|
+
const repoDirs = c.layout === "multi-repo" ? (c.repos || []).map((r) => ({ dir: path.resolve(root, expandHome(r.path)), name: r.path })) : [{ dir: root, name: path.basename(root) }];
|
|
671
|
+
const scored = repoDirs.filter((r) => isRepo(r.dir)).map((r) => ({ ...r, ...scoreRepo(r.dir, { since, weights, prefix: c.layout === "multi-repo" ? r.name : "" }) }));
|
|
672
|
+
if (!scored.length) throw new Error(`No git repo found at ${root}`);
|
|
673
|
+
const rows = scored.flatMap((s) => s.rows).sort((a, b) => b.score - a.score || b.commits - a.commits);
|
|
674
|
+
const wanted = splitList(opt("files"));
|
|
675
|
+
if (wanted.length) {
|
|
676
|
+
const norm = (f) => f.replace(/\\/g, "/").replace(/^\.\//, "");
|
|
677
|
+
const hits = wanted.map((f) => rows.find((r) => r.file === norm(f) || r.file.endsWith("/" + norm(f))) || { file: norm(f), score: null, tier: "no history" });
|
|
678
|
+
if (flag("json")) console.log(JSON.stringify(hits, null, 2));
|
|
679
|
+
else for (const h of hits) console.log(h.score === null ? `${h.file}: not scored (not a code file, or doesn't exist)`
|
|
680
|
+
: h.tier === "new" ? `${h.file}: new${h.committed ? "" : ", not committed yet"} — too little history to score.${h.near ? ` Connects to ${h.near.file} (${h.near.score} ${h.near.tier})${["high", "critical"].includes(h.near.tier) ? ": treat it as " + h.near.tier : ""}.` : " Give it standard care."}`
|
|
681
|
+
: `${h.file}: ${h.score} ${h.tier} (churn ${h.factors.churn}, bug density ${h.factors.bug_density}, fan-out ${h.factors.fan_out}, security ${h.factors.security})`);
|
|
682
|
+
return;
|
|
683
|
+
}
|
|
684
|
+
if (flag("json")) { console.log(JSON.stringify({ weights, since, rows }, null, 2)); return; }
|
|
685
|
+
const specs = path.join(root, c.specs_dir || "specs");
|
|
686
|
+
fs.mkdirSync(path.join(specs, "context"), { recursive: true });
|
|
687
|
+
const out = path.join(specs, "context", "risk.md");
|
|
688
|
+
fs.writeFileSync(out, riskMarkdown({ rows, repos: scored, weights, since, top: Number(opt("top")) || 40 }));
|
|
689
|
+
ignoreGenerated(out);
|
|
690
|
+
const count = (t) => rows.filter((r) => r.tier === t).length;
|
|
691
|
+
console.log(`✓ Scored ${rows.length} files from ${scored.reduce((t, s) => t + s.commits, 0)} commits → ${path.relative(DIR, out) || out}`);
|
|
692
|
+
console.log(` ${count("critical")} critical · ${count("high")} high · ${count("medium")} medium · ${count("new")} new. Top 5:`);
|
|
693
|
+
for (const r of rows.filter((x) => x.tier !== "new").slice(0, 5)) console.log(` ${String(r.score).padStart(5)} ${r.tier.padEnd(8)} ${r.file}`);
|
|
694
|
+
if (c.risk?.enabled === "off") console.log(" Note: risk.enabled is off in .spec-driven.yml, so the skills won't use this until you turn it on.");
|
|
695
|
+
}
|
|
696
|
+
|
|
697
|
+
async function riskEvalCmd() {
|
|
698
|
+
const { file, root, c } = loadConfig();
|
|
699
|
+
const repo = gitTop(DIR);
|
|
700
|
+
if (c.layout === "multi-repo" && real(repo) === real(root)) throw new Error("risk-eval runs on one repo: run it inside one of the workspace's repos");
|
|
701
|
+
const ticketsFile = opt("tickets");
|
|
702
|
+
if (!ticketsFile) throw new Error("Pass --tickets=<file> with one ticket per line");
|
|
703
|
+
const tickets = fs.readFileSync(path.resolve(DIR, ticketsFile), "utf8").split("\n").map((l) => l.replace(/^\s*[-*]\s+/, "").trim()).filter((l) => l && !l.startsWith("#"));
|
|
704
|
+
const claude = opt("claude") || "claude";
|
|
705
|
+
if (!opt("claude") && !onPath("claude")) throw new Error("risk-eval drives Claude Code headlessly; install it first (https://code.claude.com)");
|
|
706
|
+
const { riskEval } = await import("./risk-eval.mjs");
|
|
707
|
+
const runs = Number(opt("runs")) || 1, stages = splitList(opt("stages") || "investigate,create");
|
|
708
|
+
console.log(`GitRisk U8 experiment: ${tickets.length} ticket(s) × ${runs} run(s) × 2 arms (risk on / off) on ${root}`);
|
|
709
|
+
// A workspace config's repo paths don't resolve inside a lone clone, so a multi-repo member gets a single-repo config.
|
|
710
|
+
const configText = c.layout === "multi-repo" ? "version: 1\nlayout: single\n" : file ? fs.readFileSync(file, "utf8") : "";
|
|
711
|
+
const arms = splitList(opt("arms") || "on,off");
|
|
712
|
+
if (!arms.length || arms.some((a) => !["on", "off"].includes(a))) throw new Error("--arms must be on, off, or on,off");
|
|
713
|
+
const { dir } = await riskEval({ repo, tickets, arms, timeoutMin: Number(opt("timeout")) || 20, runs, model: opt("model") || "sonnet", stages, claude, out: path.resolve(DIR, opt("out") || "experiments"),
|
|
714
|
+
pluginDir: path.join(ROOT, "plugin"), since: c.risk?.since || "2 years ago", weights: c.risk?.weights, configText });
|
|
715
|
+
console.log(`✓ Report: ${path.join(dir, "report.md")}`);
|
|
716
|
+
}
|
|
717
|
+
|
|
718
|
+
// Plain-language switches, so anyone can change them without editing YAML.
|
|
719
|
+
const SETTINGS = [
|
|
720
|
+
{ block: "risk", key: "enabled", flag: "risk", name: "GitRisk", about: "extra care on files with a history of bugs",
|
|
721
|
+
get: (c) => c.risk?.enabled || "on", options: [
|
|
722
|
+
{ value: "on", title: "On", description: "investigations, plans and reviews look harder at historically risky files" },
|
|
723
|
+
{ value: "off", title: "Off", description: "the skills ignore risk completely" }] },
|
|
724
|
+
{ block: "pipeline", key: "after_approval", flag: "after-approval", name: "After you approve a spec", about: "how far it goes on its own",
|
|
725
|
+
get: (c) => c.pipeline?.after_approval || "stop", options: [
|
|
726
|
+
{ value: "stop", title: "Stop after each stage", description: "you start implement, verify, review and commit yourself" },
|
|
727
|
+
{ value: "run-through", title: "Run straight through", description: "implement → verify → review → commit; stops at any failure, never pushes" }] },
|
|
728
|
+
];
|
|
729
|
+
|
|
730
|
+
async function settings() {
|
|
731
|
+
const file = findConfig(DIR);
|
|
732
|
+
if (!file) throw new Error(`No .spec-driven.yml in ${DIR} or above. Run: spec-driven-dev setup`);
|
|
733
|
+
let text = fs.readFileSync(file, "utf8");
|
|
734
|
+
const label = (s, v) => s.options.find((o) => o.value === v)?.title || v;
|
|
735
|
+
const save = (s, v) => {
|
|
736
|
+
if (!s.options.some((o) => o.value === v)) throw new Error(`--${s.flag} must be ${s.options.map((o) => o.value).join(" or ")}`);
|
|
737
|
+
text = setConfigValue(text, s.block, s.key, v);
|
|
738
|
+
fs.writeFileSync(file, text);
|
|
739
|
+
console.log(`✓ ${s.name}: ${label(s, v)} (saved in ${file})`);
|
|
740
|
+
};
|
|
741
|
+
const given = SETTINGS.filter((s) => opt(s.flag));
|
|
742
|
+
if (given.length) given.forEach((s) => save(s, opt(s.flag)));
|
|
743
|
+
else for (;;) {
|
|
744
|
+
const c = parseYaml(text);
|
|
745
|
+
const pick = await askSelect("Settings: pick one to change", [
|
|
746
|
+
...SETTINGS.map((s) => ({ title: `${s.name}: ${label(s, s.get(c))}`, description: s.about, value: s })),
|
|
747
|
+
{ title: "Done", value: null }], null);
|
|
748
|
+
if (!pick) break;
|
|
749
|
+
const v = await askSelect(`${pick.name}`, pick.options, pick.get(c));
|
|
750
|
+
if (v !== pick.get(c)) save(pick, v);
|
|
751
|
+
}
|
|
752
|
+
const c = parseYaml(text);
|
|
753
|
+
if (c.risk?.enabled !== "off" && !fs.existsSync(path.join(path.dirname(file), c.specs_dir || "specs", "context", "risk.md"))) {
|
|
754
|
+
if (given.length || !(await askConfirm("GitRisk is on but files haven't been scored yet. Score them now? (uses git only, takes seconds)", true)))
|
|
755
|
+
console.log(" GitRisk is on, but nothing is scored yet. Run: spec-driven-dev risk");
|
|
756
|
+
else risk();
|
|
757
|
+
}
|
|
758
|
+
}
|
|
759
|
+
|
|
623
760
|
// Silent unless something is unfinished, so it's cheap to run at the start of a session.
|
|
624
761
|
function status() {
|
|
625
762
|
const file = findConfig(DIR);
|
|
@@ -648,11 +785,14 @@ try {
|
|
|
648
785
|
if (command === "doctor") doctor();
|
|
649
786
|
else if (command === "status") status();
|
|
650
787
|
else if (command === "context") context();
|
|
788
|
+
else if (command === "risk") risk();
|
|
789
|
+
else if (command === "settings") await settings();
|
|
790
|
+
else if (command === "risk-eval") await riskEvalCmd();
|
|
651
791
|
else if (command === "install") {
|
|
652
792
|
await install();
|
|
653
793
|
if (!flag("no-setup") && (await askConfirm("Set up this project now?", true))) await setup();
|
|
654
794
|
} else if (command === "setup") await setup();
|
|
655
|
-
else throw new Error(`Unknown command "${command}" — use install (default), setup, doctor, status, or
|
|
795
|
+
else throw new Error(`Unknown command "${command}" — use install (default), setup, settings, doctor, status, context, risk, or risk-eval`);
|
|
656
796
|
if (["install", "setup"].includes(command)) console.log('\nNext: in your AI tool, say "investigate <ticket>" or "create a spec".');
|
|
657
797
|
} catch (e) {
|
|
658
798
|
console.error(`✗ ${e.message}`);
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
// GitRisk U8 experiment: does giving the agent the risk signal change what it does?
|
|
2
|
+
// Each ticket runs twice in fresh clones of the repo, risk ON and risk OFF, with the same model and
|
|
3
|
+
// prompt. We then measure behaviour from what the agent read and wrote, and keep a dated log.
|
|
4
|
+
import fs from "node:fs";
|
|
5
|
+
import os from "node:os";
|
|
6
|
+
import path from "node:path";
|
|
7
|
+
import { execFileSync, spawnSync } from "node:child_process";
|
|
8
|
+
import { scoreRepo, riskMarkdown, normWeights, setConfigValue } from "./risk.mjs";
|
|
9
|
+
|
|
10
|
+
const TIER_RANK = { light: 1, standard: 2, deep: 3 };
|
|
11
|
+
const sh = (cwd, cmd, args) => execFileSync(cmd, args, { cwd, stdio: ["ignore", "pipe", "pipe"], maxBuffer: 512 * 1024 * 1024 }).toString();
|
|
12
|
+
|
|
13
|
+
export const setRiskEnabled = (text, on) => setConfigValue(text, "risk", "enabled", on ? "on" : "off");
|
|
14
|
+
|
|
15
|
+
// Behaviour metrics from the agent's transcript (stream-json) and the spec it wrote.
|
|
16
|
+
export function measure({ transcript, clone, risky, tracked = [] }) {
|
|
17
|
+
const reads = new Set();
|
|
18
|
+
let cost = 0, turns = 0, inTok = 0, outTok = 0, error = "";
|
|
19
|
+
for (const line of transcript.split("\n")) {
|
|
20
|
+
let e;
|
|
21
|
+
try { e = JSON.parse(line); } catch { continue; }
|
|
22
|
+
if (e.type === "assistant") for (const c of e.message?.content || []) {
|
|
23
|
+
// A file counts as looked at when a Read, Grep or Bash call names it (agents often use grep/sed/cat, not Read).
|
|
24
|
+
if (c.type !== "tool_use" || !["Read", "Grep", "Bash", "Glob"].includes(c.name)) continue;
|
|
25
|
+
const text = JSON.stringify(c.input || {}).split(clone + "/").join("").split(clone.replace(/^\/private/, "") + "/").join("");
|
|
26
|
+
for (const f of tracked) if (text.includes(f)) reads.add(f);
|
|
27
|
+
}
|
|
28
|
+
if (e.type === "result") {
|
|
29
|
+
cost = e.total_cost_usd || 0; turns = e.num_turns || 0;
|
|
30
|
+
inTok = (e.usage?.input_tokens || 0) + (e.usage?.cache_read_input_tokens || 0) + (e.usage?.cache_creation_input_tokens || 0);
|
|
31
|
+
outTok = e.usage?.output_tokens || 0;
|
|
32
|
+
if (e.is_error) error = String(e.result || "error").slice(0, 200);
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
const specsDir = path.join(clone, "specs");
|
|
36
|
+
const feature = (fs.existsSync(specsDir) ? fs.readdirSync(specsDir) : []).find((d) => fs.existsSync(path.join(specsDir, d, "investigation.md")) || fs.existsSync(path.join(specsDir, d, "tasks")));
|
|
37
|
+
const read = (f) => { try { return fs.readFileSync(f, "utf8"); } catch { return ""; } };
|
|
38
|
+
const inv = feature ? read(path.join(specsDir, feature, "investigation.md")) : "";
|
|
39
|
+
const tasks = [];
|
|
40
|
+
const tdir = feature && path.join(specsDir, feature, "tasks");
|
|
41
|
+
if (tdir && fs.existsSync(tdir)) for (const t of fs.readdirSync(tdir).filter((f) => f.endsWith(".md"))) {
|
|
42
|
+
const text = read(path.join(tdir, t));
|
|
43
|
+
const section = (name) => (text.match(new RegExp(`^## ${name}\\s*\\n([\\s\\S]*?)(?=^## |$(?![\\s\\S]))`, "m")) || [])[1] || "";
|
|
44
|
+
const files = [...(section("Files to Create") + section("Files to Modify")).matchAll(/`([^`\s]+\.[\w]+)`/g)].map((m) => m[1]);
|
|
45
|
+
const tests = section("Tests").trim();
|
|
46
|
+
tasks.push({ name: t, files, tier: (text.match(/\*\*Tier:\*\*\s*(light|standard|deep)/i) || [])[1]?.toLowerCase(), hasTests: !!tests && !/^none\b/i.test(tests) });
|
|
47
|
+
}
|
|
48
|
+
const isRisky = (f) => risky.has(f);
|
|
49
|
+
const planFiles = new Set(tasks.flatMap((t) => t.files));
|
|
50
|
+
const riskyPlan = [...planFiles].filter(isRisky);
|
|
51
|
+
const riskyTasks = tasks.filter((t) => t.files.some(isRisky));
|
|
52
|
+
const avg = (xs) => (xs.length ? Math.round((xs.reduce((a, b) => a + b, 0) / xs.length) * 100) / 100 : null);
|
|
53
|
+
return {
|
|
54
|
+
feature: feature || null,
|
|
55
|
+
files_read: reads.size,
|
|
56
|
+
risky_files_read: [...reads].filter(isRisky).length,
|
|
57
|
+
investigation_risky_mentions: [...risky].filter((f) => inv.includes(f)).length,
|
|
58
|
+
plan_files: planFiles.size,
|
|
59
|
+
risky_plan_files: riskyPlan.length,
|
|
60
|
+
risky_files_with_tests: riskyPlan.filter((f) => tasks.some((t) => t.files.includes(f) && t.hasTests)).length,
|
|
61
|
+
tier_on_risky_tasks: avg(riskyTasks.map((t) => TIER_RANK[t.tier]).filter(Boolean)),
|
|
62
|
+
tier_all_tasks: avg(tasks.map((t) => TIER_RANK[t.tier]).filter(Boolean)),
|
|
63
|
+
tasks: tasks.length,
|
|
64
|
+
cost_usd: Math.round(cost * 10000) / 10000, turns, input_tokens: inTok, output_tokens: outTok, error,
|
|
65
|
+
};
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
const METRICS = [
|
|
69
|
+
["risky_files_read", "Risky files looked at"], ["investigation_risky_mentions", "Risky files named in investigation"],
|
|
70
|
+
["risky_plan_files", "Risky files in the plan"], ["risky_files_with_tests", "…of those, with tests planned"],
|
|
71
|
+
["tier_on_risky_tasks", "Avg tier of tasks on risky files (1–3)"], ["files_read", "Repo files looked at (Read/Grep/Bash)"],
|
|
72
|
+
["tasks", "Tasks"], ["cost_usd", "Cost (USD)"], ["output_tokens", "Output tokens"],
|
|
73
|
+
];
|
|
74
|
+
|
|
75
|
+
function report({ meta, results }) {
|
|
76
|
+
const lines = [`# GitRisk U8 experiment — ${meta.started}`, "",
|
|
77
|
+
"Question: does feeding GitRisk's historical-risk signal into the spec-driven-dev workflow change what the AI agent does?", "",
|
|
78
|
+
`- Repo: \`${meta.repo}\` at ${meta.head}; risky set = ${meta.risky} files scored high or critical (weights ${JSON.stringify(meta.weights)})`,
|
|
79
|
+
`- Model: ${meta.model} · stages: ${meta.stages.join(" → ")} · runs per arm: ${meta.runs} · tickets: ${meta.tickets.length}`,
|
|
80
|
+
"- Arms: **on** = risk.enabled on + specs/context/risk.md present; **off** = risk.enabled off, no risk file. Fresh clone per run, same prompt.",
|
|
81
|
+
"", "## Results (mean per arm)", "", "| Ticket | Metric | Off | On | Δ (on − off) |", "|---|---|---|---|---|"];
|
|
82
|
+
const mean = (rs, k) => { const v = rs.map((r) => r.metrics?.[k]).filter((x) => typeof x === "number"); return v.length ? Math.round((v.reduce((a, b) => a + b, 0) / v.length) * 100) / 100 : null; };
|
|
83
|
+
meta.tickets.forEach((t, i) => {
|
|
84
|
+
const off = results.filter((r) => r.ticket === i && r.arm === "off" && !r.metrics?.error), on = results.filter((r) => r.ticket === i && r.arm === "on" && !r.metrics?.error);
|
|
85
|
+
for (const [k, label] of METRICS) {
|
|
86
|
+
const a = mean(off, k), b = mean(on, k);
|
|
87
|
+
lines.push(`| ${i + 1}. ${t.slice(0, 40).replace(/\|/g, "/")} | ${label} | ${a ?? "—"} | ${b ?? "—"} | ${a !== null && b !== null ? Math.round((b - a) * 100) / 100 : "—"} |`);
|
|
88
|
+
}
|
|
89
|
+
});
|
|
90
|
+
const failed = results.filter((r) => r.metrics?.error || r.exit !== 0);
|
|
91
|
+
lines.push("", "## Runs", "", "| Ticket | Arm | Run | Exit | Feature folder | Error |", "|---|---|---|---|---|---|",
|
|
92
|
+
...results.map((r) => `| ${r.ticket + 1} | ${r.arm} | ${r.run} | ${r.exit} | ${r.metrics?.feature || "—"} | ${(r.metrics?.error || "").replace(/\|/g, "/")} |`));
|
|
93
|
+
lines.push("", "## Caveats", "",
|
|
94
|
+
`- ${meta.runs} run(s) per arm is a pilot, not a significance test. Agent behaviour varies run to run; increase --runs before drawing conclusions.`,
|
|
95
|
+
"- Metrics are proxies read from the transcript and the written spec (files the agent opened, files and tests it planned, task tiers). They don't measure whether the eventual code has fewer bugs.",
|
|
96
|
+
"- The risky set comes from the same repo history the ON arm sees, so a higher \"risky files read\" in ON is expected; the interesting signal is whether that changes the plan (tests, tiers) without raising cost much.",
|
|
97
|
+
`- Failed runs: ${failed.length}. They are excluded from the means but kept in results.json.`);
|
|
98
|
+
return lines.join("\n") + "\n";
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
export async function riskEval(opts) {
|
|
102
|
+
const { repo, tickets, runs = 1, arms = ["on", "off"], timeoutMin = 20, model = "sonnet", stages = ["investigate", "create"], claude = "claude", out, pluginDir, since, weights, configText = "" } = opts;
|
|
103
|
+
if (!tickets.length) throw new Error("No tickets: pass --tickets=<file> with one ticket per line");
|
|
104
|
+
const started = new Date().toISOString().replace(/\.\d+Z$/, "Z");
|
|
105
|
+
const dir = path.join(out, `risk-eval-${started.replace(/[:]/g, "-")}`);
|
|
106
|
+
fs.mkdirSync(dir, { recursive: true });
|
|
107
|
+
const w = normWeights(weights);
|
|
108
|
+
const scored = scoreRepo(repo, { since, weights: w });
|
|
109
|
+
scored.rows.sort((a, b) => b.score - a.score);
|
|
110
|
+
const risky = new Set(scored.rows.filter((r) => r.tier === "high" || r.tier === "critical").map((r) => r.file));
|
|
111
|
+
const riskMd = riskMarkdown({ rows: scored.rows, repos: [{ name: path.basename(repo), head: scored.head, commits: scored.commits }], weights: w, since, top: 60 });
|
|
112
|
+
fs.writeFileSync(path.join(dir, "risk.md"), riskMd);
|
|
113
|
+
const meta = { started, repo, head: scored.head, risky: risky.size, weights: w, model, stages, runs, tickets };
|
|
114
|
+
const stageText = stages.map((s) => `spec-${s}`).join(", then ");
|
|
115
|
+
const results = [];
|
|
116
|
+
for (let t = 0; t < tickets.length; t++) for (let r = 1; r <= runs; r++) for (const arm of ((t + r) % 2 ? ["off", "on"] : ["on", "off"]).filter((a) => arms.includes(a))) {
|
|
117
|
+
const clone = fs.mkdtempSync(path.join(os.tmpdir(), `risk-eval-${arm}-`));
|
|
118
|
+
sh(os.tmpdir(), "git", ["clone", "-q", "--no-hardlinks", repo, clone]);
|
|
119
|
+
for (const extra of ["specs/CONTEXT.md", "specs/context"]) {
|
|
120
|
+
const src = path.join(repo, extra);
|
|
121
|
+
if (fs.existsSync(src) && !fs.existsSync(path.join(clone, extra))) fs.cpSync ? fs.cpSync(src, path.join(clone, extra), { recursive: true }) : sh(repo, "cp", ["-R", src, path.join(clone, extra)]);
|
|
122
|
+
}
|
|
123
|
+
fs.rmSync(path.join(clone, "specs/context/risk.md"), { force: true });
|
|
124
|
+
fs.writeFileSync(path.join(clone, ".spec-driven.yml"), setRiskEnabled(configText || "version: 1\nlayout: single\n", arm === "on"));
|
|
125
|
+
if (arm === "on") { fs.mkdirSync(path.join(clone, "specs/context"), { recursive: true }); fs.writeFileSync(path.join(clone, "specs/context/risk.md"), riskMd); }
|
|
126
|
+
const prompt = `Use the ticket-to-pr plugin skills. This is a non-interactive run: don't ask questions, make reasonable assumptions and record them. Ticket (plain text, no tracker): ${tickets[t]}\nRun ${stageText}. Work on the current branch; don't create branches, commit, push, or implement anything.`;
|
|
127
|
+
const res = spawnSync(claude, ["-p", "--output-format", "stream-json", "--verbose", "--model", model, "--plugin-dir", pluginDir, "--permission-mode", "acceptEdits",
|
|
128
|
+
"--allowedTools", "Read Write Edit Glob Grep Skill Bash(git:*) Bash(ls:*) Bash(cat:*) Bash(head:*) Bash(tail:*) Bash(wc:*) Bash(find:*) Bash(grep:*) Bash(node:*) Bash(npx:*) Bash(spec-driven-dev:*)"],
|
|
129
|
+
{ cwd: clone, input: prompt, encoding: "utf8", maxBuffer: 512 * 1024 * 1024, timeout: timeoutMin * 60 * 1000 });
|
|
130
|
+
const transcript = res.stdout || "";
|
|
131
|
+
const tag = `t${t + 1}-${arm}-r${r}`;
|
|
132
|
+
fs.writeFileSync(path.join(dir, `${tag}.jsonl`), transcript);
|
|
133
|
+
const tracked = sh(clone, "git", ["ls-files"]).split("\n").filter((f) => f.includes("/") || f.includes("."));
|
|
134
|
+
const metrics = measure({ transcript, clone, risky, tracked });
|
|
135
|
+
if (res.status !== 0 && !metrics.error) {
|
|
136
|
+
// Say where it stopped, so a stall (usage limit, API wait) is told apart from a skill problem.
|
|
137
|
+
const events = transcript.split("\n").filter(Boolean).length;
|
|
138
|
+
const lastTool = [...transcript.matchAll(/"type":"tool_use","id":"[^"]+","name":"(\w+)"/g)].pop()?.[1] || "none";
|
|
139
|
+
metrics.error = `${res.error?.code === "ETIMEDOUT" ? `timed out after ${timeoutMin} min` : `exit ${res.status}`} · ${events} events · last tool ${lastTool} ${(res.stderr || "").slice(0, 120)}`.trim();
|
|
140
|
+
}
|
|
141
|
+
if (metrics.feature) fs.cpSync ? fs.cpSync(path.join(clone, "specs", metrics.feature), path.join(dir, tag), { recursive: true }) : sh(clone, "cp", ["-R", path.join("specs", metrics.feature), path.join(dir, tag)]);
|
|
142
|
+
results.push({ ticket: t, arm, run: r, exit: res.status, metrics });
|
|
143
|
+
fs.rmSync(clone, { recursive: true, force: true });
|
|
144
|
+
console.log(` ${tag}: ${metrics.error ? `failed (${metrics.error.slice(0, 80)})` : `read ${metrics.risky_files_read} risky / ${metrics.files_read} files, ${metrics.risky_plan_files} risky in plan, $${metrics.cost_usd}`}`);
|
|
145
|
+
}
|
|
146
|
+
fs.writeFileSync(path.join(dir, "results.json"), JSON.stringify({ meta, results }, null, 2));
|
|
147
|
+
const md = report({ meta, results });
|
|
148
|
+
fs.writeFileSync(path.join(dir, "report.md"), md);
|
|
149
|
+
const failed = results.filter((x) => x.metrics.error).length;
|
|
150
|
+
fs.appendFileSync(path.join(out, "LOG.md"), `${fs.existsSync(path.join(out, "LOG.md")) ? "" : "# GitRisk experiment log\n\nAppend-only record of every run, including failures.\n\n"}- ${started} · ${tickets.length} ticket(s) × ${runs} run(s) × 2 arms on \`${path.basename(repo)}\` at ${scored.head}, ${model}, ${stages.join("+")} · ${results.length - failed} ok, ${failed} failed · [report](${path.basename(dir)}/report.md)\n`);
|
|
151
|
+
return { dir, report: md, results };
|
|
152
|
+
}
|
package/bin/risk.mjs
ADDED
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
// GitRisk: per-file historical risk from git alone (no model, no tokens).
|
|
2
|
+
// Risk = 0.30·Churn + 0.35·BugDensity + 0.20·FanOut + 0.15·Security (each factor 0–100)
|
|
3
|
+
// The weights are a hypothesis to calibrate by backtest, so .spec-driven.yml `risk.weights` overrides them.
|
|
4
|
+
import fs from "node:fs";
|
|
5
|
+
import path from "node:path";
|
|
6
|
+
import { execFileSync } from "node:child_process";
|
|
7
|
+
|
|
8
|
+
export const DEFAULT_WEIGHTS = { churn: 0.3, bug_density: 0.35, fan_out: 0.2, security: 0.15 };
|
|
9
|
+
export const TIERS = [[75, "critical"], [50, "high"], [25, "medium"], [0, "low"]];
|
|
10
|
+
const NEW_DAYS = 90; // a file added within this many days with fewer than MIN_COMMITS commits (or not committed yet) has too little history to score fairly
|
|
11
|
+
const MIN_COMMITS = 3; // shortcut: a file with 1 commit that was a fix would score 100% bug density; divide by at least 3. Replace with Bayesian smoothing if calibration shows it matters.
|
|
12
|
+
|
|
13
|
+
// Classification uses the conventional-commit type when the repo uses one, and message keywords otherwise.
|
|
14
|
+
const BUG = /^(fix|hotfix|bugfix|revert)(\(.+\))?!?:|\b(fix(e[sd]|ing)?|bug|hotfix|regression|broken|crash(es|ed)?|patch(ed)?|defect)\b/i;
|
|
15
|
+
const SECURITY = /\b(security|cve-\d+|xss|csrf|ssrf|sqli|injection|vuln(erability)?|exploit|sanitiz(e|ation)|escap(e|ing) (html|input|output)|auth(entication|orization)? (bypass|fix)|privilege|secret(s)? (leak|exposed)|leak(ed|s)? (token|key|secret|credential)|rate.?limit|helmet|csp)\b/i;
|
|
16
|
+
const SKIP = /(^|\/)(node_modules|dist|build|out|vendor|coverage|\.next|graphify-out|specs|experiments)\/|(^|\/)(\.spec-driven\.yml|\.gitignore|\.gitattributes|\.gitmodules)$|\.(md|mdx|txt|lock|log|png|jpe?g|gif|svg|ico|webp|pdf|zip|map|snap|csv)$|(^|\/)(package-lock\.json|yarn\.lock|pnpm-lock\.yaml|Cargo\.lock|go\.sum|poetry\.lock|composer\.lock)$/i;
|
|
17
|
+
const CODE = /\.(m?[jt]sx?|cjs|py|rb|go|rs|java|kt|php|cs|swift|scala|vue|svelte)$/i;
|
|
18
|
+
|
|
19
|
+
const git = (cwd, args) => {
|
|
20
|
+
try { return execFileSync("git", args, { cwd, stdio: ["ignore", "pipe", "ignore"], maxBuffer: 512 * 1024 * 1024 }).toString(); } catch { return ""; }
|
|
21
|
+
};
|
|
22
|
+
export const classify = (subject) => ({ bug: BUG.test(subject), security: SECURITY.test(subject) });
|
|
23
|
+
export const tierOf = (score) => TIERS.find(([min]) => score >= min)[1];
|
|
24
|
+
|
|
25
|
+
// Value at the 90th percentile; "expected" in min(100, actual / expected × 100), so one outlier can't flatten everyone else.
|
|
26
|
+
function p90(values) {
|
|
27
|
+
const v = values.filter((x) => x > 0).sort((a, b) => a - b);
|
|
28
|
+
return v.length ? v[Math.min(v.length - 1, Math.floor(v.length * 0.9))] : 1;
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
// Within-repo dependents: how many other files import this one. Relative JS/TS imports and Python module paths.
|
|
32
|
+
// shortcut: no tsconfig path aliases or cross-repo consumers (GitRisk open question U7); add when a repo needs them.
|
|
33
|
+
function fanOut(repo, files) {
|
|
34
|
+
const set = new Set(files), dependents = new Map(files.map((f) => [f, new Set()]));
|
|
35
|
+
const exts = ["", ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs", "/index.ts", "/index.tsx", "/index.js", "/index.jsx", ".py", "/__init__.py"];
|
|
36
|
+
const pyIndex = new Map(files.filter((f) => f.endsWith(".py")).map((f) => [f.replace(/\/__init__\.py$|\.py$/, "").split("/").join("."), f]));
|
|
37
|
+
for (const f of files) {
|
|
38
|
+
if (!CODE.test(f)) continue;
|
|
39
|
+
let text = "";
|
|
40
|
+
try { text = fs.readFileSync(path.join(repo, f), "utf8"); } catch { continue; }
|
|
41
|
+
if (text.length > 2e6) continue;
|
|
42
|
+
const targets = new Set();
|
|
43
|
+
for (const m of text.matchAll(/(?:from\s+|require\(\s*|import\(\s*|import\s+)["'](\.{1,2}\/[^"']+)["']/g)) {
|
|
44
|
+
const base = path.posix.normalize(path.posix.join(path.posix.dirname(f), m[1]));
|
|
45
|
+
const hit = exts.map((e) => base + e).find((c) => set.has(c));
|
|
46
|
+
if (hit) targets.add(hit);
|
|
47
|
+
}
|
|
48
|
+
if (f.endsWith(".py")) for (const m of text.matchAll(/^\s*(?:from\s+([\w.]+)\s+import|import\s+([\w.]+))/gm)) {
|
|
49
|
+
const hit = pyIndex.get(m[1] || m[2]);
|
|
50
|
+
if (hit) targets.add(hit);
|
|
51
|
+
}
|
|
52
|
+
for (const t of targets) if (t !== f) dependents.get(t).add(f);
|
|
53
|
+
}
|
|
54
|
+
const imports = new Map(files.map((f) => [f, new Set()]));
|
|
55
|
+
for (const [t, srcs] of dependents) for (const f of srcs) imports.get(f).add(t);
|
|
56
|
+
return { counts: new Map([...dependents].map(([f, s]) => [f, s.size])), dependents, imports };
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
// Score every code file in one repo, including new files not committed yet. `prefix` labels files in multi-repo workspaces.
|
|
60
|
+
export function scoreRepo(repo, { since = "2 years ago", weights = DEFAULT_WEIGHTS, prefix = "" } = {}) {
|
|
61
|
+
const committed = git(repo, ["ls-files"]).split("\n").filter((f) => f && !SKIP.test(f));
|
|
62
|
+
const untracked = git(repo, ["ls-files", "--others", "--exclude-standard"]).split("\n").filter((f) => f && !SKIP.test(f));
|
|
63
|
+
const files = [...committed, ...untracked];
|
|
64
|
+
const tracked = new Set(files);
|
|
65
|
+
// When each file was first added (all history, not just `since`), to tell new files from quiet old ones.
|
|
66
|
+
const added = new Map();
|
|
67
|
+
for (const block of git(repo, ["log", "--diff-filter=A", "--format=%x1e%ct", "--name-only"]).split("\x1e").slice(1)) {
|
|
68
|
+
const [ts, ...names] = block.split("\n");
|
|
69
|
+
for (const f of names) if (f) added.set(f, Number(ts)); // log is newest first, so the last write is the first add
|
|
70
|
+
}
|
|
71
|
+
const cutoff = Date.now() / 1000 - NEW_DAYS * 86400;
|
|
72
|
+
const recent = (f) => !added.has(f) || added.get(f) >= cutoff;
|
|
73
|
+
const stats = new Map(files.map((f) => [f, { commits: 0, bug: 0, security: 0 }]));
|
|
74
|
+
let commits = 0;
|
|
75
|
+
// \x1e starts each commit; the subject follows, then the changed file names.
|
|
76
|
+
for (const block of git(repo, ["log", "--no-merges", `--since=${since}`, "--name-only", "--format=%x1e%s"]).split("\x1e").slice(1)) {
|
|
77
|
+
const [subject, ...changed] = block.split("\n");
|
|
78
|
+
const c = classify(subject);
|
|
79
|
+
commits++;
|
|
80
|
+
for (const f of changed) {
|
|
81
|
+
const s = tracked.has(f) && stats.get(f);
|
|
82
|
+
if (!s) continue;
|
|
83
|
+
s.commits++;
|
|
84
|
+
if (c.bug) s.bug++;
|
|
85
|
+
if (c.security) s.security++;
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
// "New" = recent AND too few commits to judge. A young file that's already been fixed a lot has real history.
|
|
89
|
+
const isNew = new Set(files.filter((f) => untracked.includes(f) || (recent(f) && stats.get(f).commits < MIN_COMMITS)));
|
|
90
|
+
const graph = fanOut(repo, files), fan = graph.counts;
|
|
91
|
+
const expChurn = p90([...stats.values()].map((s) => s.commits)), expFan = p90([...fan.values()]);
|
|
92
|
+
const rows = files.map((f) => {
|
|
93
|
+
const s = stats.get(f), denom = Math.max(s.commits, MIN_COMMITS);
|
|
94
|
+
const factors = {
|
|
95
|
+
churn: Math.min(100, (s.commits / expChurn) * 100),
|
|
96
|
+
bug_density: s.commits ? (s.bug / denom) * 100 : 0,
|
|
97
|
+
fan_out: Math.min(100, ((fan.get(f) || 0) / expFan) * 100),
|
|
98
|
+
security: s.commits ? (s.security / denom) * 100 : 0,
|
|
99
|
+
};
|
|
100
|
+
const score = Object.keys(DEFAULT_WEIGHTS).reduce((t, k) => t + (Number(weights[k]) || 0) * factors[k], 0);
|
|
101
|
+
const r = (x) => Math.round(x * 10) / 10;
|
|
102
|
+
return { file: prefix ? `${prefix}/${f}` : f, path: f, score: r(score), tier: tierOf(score), commits: s.commits, bug_fixes: s.bug, security_fixes: s.security, dependents: fan.get(f) || 0,
|
|
103
|
+
factors: { churn: r(factors.churn), bug_density: r(factors.bug_density), fan_out: r(factors.fan_out), security: r(factors.security) } };
|
|
104
|
+
});
|
|
105
|
+
// New files: too little history for a fair score, so label them "new" and show the riskiest established
|
|
106
|
+
// file they import or are imported by. New code next to a hotspot deserves the hotspot's care.
|
|
107
|
+
const byPath = new Map(rows.map((x) => [x.path, x]));
|
|
108
|
+
for (const x of rows) {
|
|
109
|
+
if (!isNew.has(x.path)) continue;
|
|
110
|
+
x.tier = "new";
|
|
111
|
+
x.committed = !untracked.includes(x.path);
|
|
112
|
+
const near = [...graph.imports.get(x.path), ...graph.dependents.get(x.path)].map((n) => byPath.get(n)).filter((n) => n && !isNew.has(n.path)).sort((a, b) => b.score - a.score)[0];
|
|
113
|
+
if (near) x.near = { file: near.file, score: near.score, tier: near.tier };
|
|
114
|
+
}
|
|
115
|
+
for (const x of rows) delete x.path;
|
|
116
|
+
return { commits, head: git(repo, ["rev-parse", "--short", "HEAD"]).trim(), rows };
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
export function normWeights(w) {
|
|
120
|
+
const out = { ...DEFAULT_WEIGHTS };
|
|
121
|
+
for (const k of Object.keys(DEFAULT_WEIGHTS)) if (w && w[k] !== undefined && w[k] !== "") out[k] = Number(w[k]);
|
|
122
|
+
return out;
|
|
123
|
+
}
|
|
124
|
+
export function weightProblems(w) {
|
|
125
|
+
const p = [];
|
|
126
|
+
for (const [k, v] of Object.entries(w || {})) {
|
|
127
|
+
if (!(k in DEFAULT_WEIGHTS)) p.push(`risk.weights.${k}: unknown factor (use ${Object.keys(DEFAULT_WEIGHTS).join(", ")})`);
|
|
128
|
+
else if (!(Number(v) >= 0)) p.push(`risk.weights.${k} "${v}" must be a number ≥ 0`);
|
|
129
|
+
}
|
|
130
|
+
const sum = Object.values(normWeights(w)).reduce((a, b) => a + b, 0);
|
|
131
|
+
if (Math.abs(sum - 1) > 0.001) p.push(`risk.weights add up to ${Math.round(sum * 1000) / 1000}, not 1`);
|
|
132
|
+
return p;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
// The area file skills read: top N files, within the 150-line area-file limit.
|
|
136
|
+
export function riskMarkdown({ rows, repos, weights, since, top }) {
|
|
137
|
+
const pct = (x) => `${Math.round(x * 100)}%`;
|
|
138
|
+
const w = weights;
|
|
139
|
+
const fresh = rows.filter((r) => r.tier === "new");
|
|
140
|
+
const freshShown = fresh.filter((r) => r.near).sort((a, b) => b.near.score - a.near.score).slice(0, 15);
|
|
141
|
+
const shown = rows.filter((r) => r.score > 0 && r.tier !== "new").slice(0, Math.min(top, 120 - freshShown.length));
|
|
142
|
+
const lines = [
|
|
143
|
+
"# Risk",
|
|
144
|
+
"<!-- Generated by `spec-driven-dev risk`. Don't edit by hand: re-run it. Skills read this only when risk.enabled is on. -->",
|
|
145
|
+
`Scored ${repos.map((r) => `${r.name} at ${r.head} (${r.commits} commits)`).join(", ")} · since ${since} · ${new Date().toISOString().slice(0, 10)}`,
|
|
146
|
+
"",
|
|
147
|
+
`Risk = ${w.churn} × Churn + ${w.bug_density} × Bug Density + ${w.fan_out} × Fan-out + ${w.security} × Security (each 0–100; weights ${pct(w.churn)} / ${pct(w.bug_density)} / ${pct(w.fan_out)} / ${pct(w.security)}).`,
|
|
148
|
+
"Tiers: critical ≥ 75 · high ≥ 50 · medium ≥ 25. A historical signal, not proof of a bug.",
|
|
149
|
+
"",
|
|
150
|
+
"| File | Score | Tier | Churn | Bug density | Fan-out | Security | Commits (fixes / security) |",
|
|
151
|
+
"|---|---|---|---|---|---|---|---|",
|
|
152
|
+
...shown.map((r) => `| \`${r.file}\` | ${r.score} | ${r.tier} | ${r.factors.churn} | ${r.factors.bug_density} | ${r.factors.fan_out} | ${r.factors.security} | ${r.commits} (${r.bug_fixes} / ${r.security_fixes}) |`),
|
|
153
|
+
];
|
|
154
|
+
if (fresh.length) {
|
|
155
|
+
lines.push("", `## New files (${fresh.length})`, "", `Added in the last ${NEW_DAYS} days with fewer than ${MIN_COMMITS} commits, or not committed yet, so there's too little history to score. Give them standard care, and the care of the riskiest file they connect to when that file is high or critical.`);
|
|
156
|
+
if (freshShown.length) lines.push("", "| New file | Connects to | Its score / tier |", "|---|---|---|", ...freshShown.map((r) => `| \`${r.file}\` | \`${r.near.file}\` | ${r.near.score} ${r.near.tier} |`));
|
|
157
|
+
}
|
|
158
|
+
if (rows.length > shown.length + freshShown.length) lines.push("", `${rows.length - shown.length - freshShown.length} more files not shown. \`spec-driven-dev risk --files=a,b\` prints any file's score.`);
|
|
159
|
+
return lines.join("\n") + "\n";
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
// Change one `block: key: value` line in .spec-driven.yml text, keeping every comment and other line as is.
|
|
163
|
+
export function setConfigValue(text, blockName, key, value) {
|
|
164
|
+
const block = text.match(new RegExp(`^${blockName}:.*\\n(?:(?:[ \\t]+.*|[ \\t]*)\\n)*`, "m"));
|
|
165
|
+
const line = new RegExp(`^([ \\t]+${key}:[ \\t]*)[^\\s#]+`, "m");
|
|
166
|
+
if (!block) return `${text.replace(/\n*$/, "\n")}${blockName}:\n ${key}: ${value}\n`;
|
|
167
|
+
const body = line.test(block[0]) ? block[0].replace(line, `$1${value}`) : block[0].replace(new RegExp(`^${blockName}:.*\\n`), `$& ${key}: ${value}\n`);
|
|
168
|
+
return text.replace(block[0], body);
|
|
169
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@benjaminthomas/spec-driven-dev",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.11.1",
|
|
4
4
|
"description": "Spec-driven AI development workflow (investigate → spec → implement → verify → review → ship) as Agent Skills for Claude Code, Cursor, Codex CLI and Antigravity.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -8,7 +8,8 @@
|
|
|
8
8
|
},
|
|
9
9
|
"files": [
|
|
10
10
|
"bin",
|
|
11
|
-
"plugin/skills"
|
|
11
|
+
"plugin/skills",
|
|
12
|
+
"plugin/.claude-plugin"
|
|
12
13
|
],
|
|
13
14
|
"scripts": {
|
|
14
15
|
"test": "python3 scripts/check.py && node scripts/test-cli.mjs"
|
|
Binary file
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "ticket-to-pr",
|
|
3
|
+
"version": "1.11.1",
|
|
4
|
+
"description": "Parallel-batch spec-driven feature workflow (spec-setup, spec-investigate, spec-create, spec-implement, spec-verify, spec-review, spec-checkpoint, spec-ship) plus a portable SEO/GEO operations skill (generative-seo) for any codebase.",
|
|
5
|
+
"author": {
|
|
6
|
+
"name": "Benjamin Thomas",
|
|
7
|
+
"url": "https://github.com/benjaminthomas"
|
|
8
|
+
},
|
|
9
|
+
"keywords": [
|
|
10
|
+
"spec-driven-development",
|
|
11
|
+
"feature-workflow",
|
|
12
|
+
"parallel-agents",
|
|
13
|
+
"seo",
|
|
14
|
+
"geo",
|
|
15
|
+
"productivity"
|
|
16
|
+
],
|
|
17
|
+
"homepage": "https://github.com/benjaminthomas/spec-driven-dev",
|
|
18
|
+
"repository": "https://github.com/benjaminthomas/spec-driven-dev",
|
|
19
|
+
"icon": ".claude-plugin/icon.png"
|
|
20
|
+
}
|
|
@@ -66,8 +66,17 @@ Skip this step unless the changes belong to a spec (`{specs_dir}/{feature}/` exi
|
|
|
66
66
|
one line to CONTEXT.md's **Specs** list. Follow the rules in
|
|
67
67
|
references/context-template.md inside the installed `spec-setup` skill folder (not this repo): dated, with a source, added at the end, within the size
|
|
68
68
|
limits, superseded lines moved to `context/archive.md`.
|
|
69
|
+
- **New areas and files.** Compare `git diff --name-status --diff-filter=A <base>...HEAD` with
|
|
70
|
+
CONTEXT.md's **Areas** table. Add a row for any new folder that holds a real area (not a
|
|
71
|
+
single helper), and one line per important new file (an entry point, a shared module others
|
|
72
|
+
will import) to its area file. Stay within the size limits.
|
|
73
|
+
- **Areas that grew.** If the feature changed what an existing area does (new commands,
|
|
74
|
+
responsibilities, or sub-folders), update that row's "What it does" in **Areas** so the
|
|
75
|
+
map stays true. Keep it to one line.
|
|
69
76
|
- Show the context changes in the report so the user can correct them.
|
|
70
|
-
3.
|
|
77
|
+
3. If `risk.enabled` is on and the CLI is available, re-run `spec-driven-dev risk` so the scores
|
|
78
|
+
include this work. It's git-only, so it costs no tokens. `risk.md` is gitignored: don't stage it.
|
|
79
|
+
4. No CONTEXT.md? Skip the distilling and suggest running `spec-setup` to create one.
|
|
71
80
|
|
|
72
81
|
### Step 3: Stage Everything
|
|
73
82
|
|
|
@@ -45,6 +45,21 @@ the answers to `.spec-driven.yml` (keys: `layout`, `repos`, `tracker`, `git`, `c
|
|
|
45
45
|
- **Start the memory.** Create or extend `log.md` (`references/log-template.md`) and fill the
|
|
46
46
|
README's **Resume here** block.
|
|
47
47
|
|
|
48
|
+
## Risk signal (GitRisk)
|
|
49
|
+
|
|
50
|
+
Same rule as `spec-investigate`: use it only when `risk.enabled` is on (or absent) and
|
|
51
|
+
`{specs_dir}/context/risk.md` exists, and ignore risk completely when it's `off`. For each task that
|
|
52
|
+
creates or modifies a **high** or **critical** file (from the investigation's **Risk Hotspots**, or
|
|
53
|
+
`spec-driven-dev risk --files=…`):
|
|
54
|
+
|
|
55
|
+
- **Raise its tier by one** (light → standard → deep), and say why in Complexity's Signals.
|
|
56
|
+
- Fill the task's **Risk** section: the file, its score, and the past failure pattern.
|
|
57
|
+
- Its **Tests** must include at least one test aimed at that past failure pattern, not just the new
|
|
58
|
+
behaviour.
|
|
59
|
+
|
|
60
|
+
A task that creates a **new** file next to a high or critical one (it imports it, or the hotspot will
|
|
61
|
+
import it) gets the same treatment, because the new code inherits that file's failure modes.
|
|
62
|
+
|
|
48
63
|
## When to Use
|
|
49
64
|
|
|
50
65
|
- After a planning conversation where requirements and technical details have been discussed
|
|
@@ -35,6 +35,12 @@ pending
|
|
|
35
35
|
|
|
36
36
|
**Context from dependencies:** {Prose summary of what the dependency tasks produce that this task needs. For example: "task-01-setup-database.md creates the PostgreSQL schema with users and sessions tables. This task builds the API routes that query those tables." This is what makes each file self-contained — the agent does not need to read other task files to understand its inputs.}
|
|
37
37
|
|
|
38
|
+
## Risk
|
|
39
|
+
|
|
40
|
+
{Only when the task touches a high/critical file and risk.enabled is on; otherwise write "None".}
|
|
41
|
+
|
|
42
|
+
- `path/to/file.ts`: {score} {tier}. Past failures: {what the fix commits show keeps breaking}. Test: {the test in Tests aimed at it}
|
|
43
|
+
|
|
38
44
|
## Project Context
|
|
39
45
|
|
|
40
46
|
{Only the 3–5 lines from the project context (CONTEXT.md / context/<area>.md, via the investigation's Relevant Context) that apply to THIS task: conventions, pitfalls, off-limits areas. The coder never reads the context files itself. Write "None" if nothing applies.}
|
|
@@ -225,6 +225,11 @@ After the batch passes review (or the user chooses to proceed):
|
|
|
225
225
|
2. **Update README.md**: change `- [ ]` to `- [x]` for each completed task
|
|
226
226
|
3. **Update memory**: append a log entry (what this batch decided or changed from the spec) and
|
|
227
227
|
overwrite the README's **Resume here** (`Stage: batch {N}/{total} done`, `Next: …`).
|
|
228
|
+
**Refresh risk** after the commit when `risk.enabled` is on and the CLI is available: run
|
|
229
|
+
`spec-driven-dev risk`. It takes well under a second with no model. New files from this batch
|
|
230
|
+
show as `new`, and files they import gain fan-out, so later batches and the reviewers see
|
|
231
|
+
current scores, not the ones from before this feature started. `risk.md` is gitignored (the
|
|
232
|
+
command adds it), so never stage or commit it.
|
|
228
233
|
4. **Commit the batch**:
|
|
229
234
|
```
|
|
230
235
|
git add -A
|
|
@@ -25,7 +25,8 @@ You are reviewing the combined output of Batch {batch_number} of a feature imple
|
|
|
25
25
|
- No duplicate or conflicting definitions
|
|
26
26
|
- Shared state (env vars, config) is consistent
|
|
27
27
|
4. **Code Quality**: Check for security issues, performance problems, and adherence to project conventions.
|
|
28
|
-
5. **
|
|
28
|
+
5. **Risk**: new files show as tier `new`, which means unproven, not safe; review them at standard care, or at the tier of the hotspot they connect to. For every task with a filled **Risk** section, check the code handles the past failure pattern it names, and that the test aimed at it exists and would fail without the fix.
|
|
29
|
+
6. **Scope**: Verify each task stayed within its defined scope (Files to Create/Modify). Flag any unexpected file changes.
|
|
29
30
|
|
|
30
31
|
## Verdict
|
|
31
32
|
|
|
@@ -65,6 +65,25 @@ Cheap sources first, code second:
|
|
|
65
65
|
|
|
66
66
|
No CONTEXT.md? Investigate as usual, and suggest running `spec-setup` afterwards to create one.
|
|
67
67
|
|
|
68
|
+
## Risk signal (GitRisk)
|
|
69
|
+
|
|
70
|
+
Use it only when `.spec-driven.yml` has `risk.enabled: on`, or has no `risk:` block, **and**
|
|
71
|
+
`{specs_dir}/context/risk.md` exists. If `risk.enabled: off`, ignore risk entirely: don't read the
|
|
72
|
+
file, compute scores, or mention risk. That keeps on/off comparisons (`spec-driven-dev risk-eval`)
|
|
73
|
+
clean.
|
|
74
|
+
|
|
75
|
+
1. Once you know the change surface, look up its files: `spec-driven-dev risk --files=a,b` (or
|
|
76
|
+
`npx -y @benjaminthomas/spec-driven-dev risk --files=…`), or find them in `risk.md`. Without the
|
|
77
|
+
CLI, approximate a file's history with `git log --since="2 years ago" --format=%s -- <file>`.
|
|
78
|
+
2. For each **high** or **critical** file in the surface, look harder: read its last ~10 commit
|
|
79
|
+
subjects and the diffs of its last 1–2 fix commits, and check its dependents. Write what keeps
|
|
80
|
+
breaking there in the report's **Risk Hotspots** section.
|
|
81
|
+
3. A score is a reason to look harder, never evidence. Hotspot findings still need their
|
|
82
|
+
VERIFIED / INFERRED labels.
|
|
83
|
+
4. **New files** (added in the last 90 days with fewer than 3 commits, or not committed yet) show as tier `new`: too little
|
|
84
|
+
history to score, which is not the same as safe. Give them standard care. When one connects to a
|
|
85
|
+
high or critical file (`Connects to …` in the lookup), treat it at that file's tier.
|
|
86
|
+
|
|
68
87
|
## Invocation
|
|
69
88
|
|
|
70
89
|
```
|
|
@@ -22,6 +22,11 @@ Copy this structure into `specs/{feature-slug}/investigation.md`.
|
|
|
22
22
|
- Conventions / decisions / pitfalls that apply: (one line each, with the source)
|
|
23
23
|
- Context corrections: (where the code contradicts CONTEXT.md, with evidence; or "none")
|
|
24
24
|
|
|
25
|
+
## Risk Hotspots
|
|
26
|
+
<!-- Only when risk.enabled is on. One row per high/critical file in the change surface; else "none" or omit. -->
|
|
27
|
+
| File | Score / tier | What has broken here before (from fix commits) | What to watch in this change |
|
|
28
|
+
|---|---|---|---|
|
|
29
|
+
|
|
25
30
|
## Business Context
|
|
26
31
|
## Problem Statement
|
|
27
32
|
## Expected Behaviour
|
|
@@ -74,7 +74,13 @@ review on a weaker model than the code was written with misses things.
|
|
|
74
74
|
3. **The code.** Correctness (logic, edge cases, error paths, races), security (injection, authz,
|
|
75
75
|
secrets, unsafe input), data safety (migrations, deletes, irreversible writes), tests (behaviour
|
|
76
76
|
without a test, tests weakened, skipped or deleted), and performance on hot paths only.
|
|
77
|
-
|
|
77
|
+
**Risk.** When `risk.enabled` is on and `{specs_dir}/context/risk.md` exists, look up the changed
|
|
78
|
+
files (`spec-driven-dev risk --files=…`). Give **high** and **critical** files a line-by-line read
|
|
79
|
+
of their error paths and callers, check that the tests target their past failure patterns, and
|
|
80
|
+
put the score in any finding on them. New files show as tier `new`: review them at standard care,
|
|
81
|
+
or at the tier of the high or critical file they connect to. When risk is `off`, skip this and
|
|
82
|
+
don't mention it.
|
|
83
|
+
4. **Run the checks.**4. **Run the checks.** Run `commands.lint`, `typecheck`, and `test` once if verify didn't just run
|
|
78
84
|
them on this exact commit. A failing check is a blocker.
|
|
79
85
|
|
|
80
86
|
**Verify every finding before reporting it.** Re-read the code around it and ask "what concrete
|
|
@@ -8,7 +8,7 @@ description: >
|
|
|
8
8
|
plus one round of questions, for a new one the intended stack and conventions, saved to a small
|
|
9
9
|
specs/CONTEXT.md that later skills load cheaply. Use when the user says "set up spec-driven",
|
|
10
10
|
"configure the pipeline", "spec-setup", "audit this repo", "refresh the project context",
|
|
11
|
-
"change our branch rules", or when any other spec-* skill finds no .spec-driven.yml or no
|
|
11
|
+
"change our branch rules", "turn GitRisk on/off", "turn on run-through", or when any other spec-* skill finds no .spec-driven.yml or no
|
|
12
12
|
CONTEXT.md. Detects everything it can first and only asks about what is ambiguous. Writes config
|
|
13
13
|
and context files only — never changes code, branches, or commits.
|
|
14
14
|
license: MIT
|
|
@@ -118,13 +118,33 @@ Skip if `{specs_dir}/CONTEXT.md` already exists (offer a refresh instead, see `r
|
|
|
118
118
|
intended stack and conventions for a new one. One round of questions, no more.
|
|
119
119
|
2. Write `{specs_dir}/CONTEXT.md` (and any area files) using `references/context-template.md`.
|
|
120
120
|
Keep the index to 60 lines and area files to 150.
|
|
121
|
-
3. **
|
|
121
|
+
3. **Risk scores.** For an existing project, run `spec-driven-dev risk` (or
|
|
122
|
+
`npx -y @benjaminthomas/spec-driven-dev risk`) to write `{specs_dir}/context/risk.md`. It uses
|
|
123
|
+
git only, no model, and adds `risk.md` to `.gitignore` because it's regenerated on every run. Skip it for a new project with no history. Mention the hotspot count in the
|
|
124
|
+
report. Set `risk.enabled: off` only if the user doesn't want it.
|
|
125
|
+
4. **Offer, don't do:** ask whether to add this one line to `CLAUDE.md` / `AGENTS.md` (whichever
|
|
122
126
|
the project uses), so hosts reload the pointer after compaction and in every new session:
|
|
123
127
|
`For spec work, read specs/CONTEXT.md first, then the "Resume here" block of any unfinished spec.`
|
|
124
128
|
Add it only on a yes.
|
|
125
|
-
|
|
129
|
+
5. Re-run `npx -y @benjaminthomas/spec-driven-dev doctor` if available. It checks the size limits
|
|
126
130
|
and flags context lines that point to files which no longer exist.
|
|
127
131
|
|
|
132
|
+
## Quick switches (plain words, no YAML)
|
|
133
|
+
|
|
134
|
+
When the user asks to switch a feature on or off, change only that one line of `.spec-driven.yml`,
|
|
135
|
+
keep every other line and comment, and confirm in one sentence. Don't re-run discovery.
|
|
136
|
+
|
|
137
|
+
| They say | Set |
|
|
138
|
+
|---|---|
|
|
139
|
+
| "turn GitRisk off", "stop using risk scores", "ignore risky files" | `risk.enabled: off` |
|
|
140
|
+
| "turn GitRisk on", "use the risk scores" | `risk.enabled: on`, then run `spec-driven-dev risk` if `{specs_dir}/context/risk.md` doesn't exist yet |
|
|
141
|
+
| "run straight through after I approve", "fewer stops" | `pipeline.after_approval: run-through` |
|
|
142
|
+
| "stop after each stage" | `pipeline.after_approval: stop` |
|
|
143
|
+
|
|
144
|
+
Reply with what changed and what it means, for example: "GitRisk is now off. Investigations, plans
|
|
145
|
+
and reviews will ignore risk scores until you turn it back on." People who prefer a menu can run
|
|
146
|
+
`spec-driven-dev settings` in a terminal.
|
|
147
|
+
|
|
128
148
|
## Updating
|
|
129
149
|
|
|
130
150
|
Re-running `spec-setup` with an existing config: load it, re-detect, and show only what changed or
|
|
@@ -75,6 +75,18 @@ effort:
|
|
|
75
75
|
deep: high
|
|
76
76
|
escalate: xhigh
|
|
77
77
|
|
|
78
|
+
# GitRisk: historical risk per file from git alone (`spec-driven-dev risk` writes {specs_dir}/context/risk.md).
|
|
79
|
+
# Risk = churn × 0.30 + bug_density × 0.35 + fan_out × 0.20 + security × 0.15 (each factor 0–100).
|
|
80
|
+
# The weights are a starting hypothesis; calibrate them against your own past bugs.
|
|
81
|
+
risk:
|
|
82
|
+
enabled: on # on | off — off makes every skill ignore risk (used by risk-eval's A/B)
|
|
83
|
+
since: "2 years ago" # how much history to score
|
|
84
|
+
weights:
|
|
85
|
+
churn: 0.30 # how often the file changes, vs the repo's 90th percentile
|
|
86
|
+
bug_density: 0.35 # share of its commits that were bug fixes
|
|
87
|
+
fan_out: 0.20 # how many other files import it, vs the repo's 90th percentile
|
|
88
|
+
security: 0.15 # share of its commits that were security fixes
|
|
89
|
+
|
|
78
90
|
pipeline:
|
|
79
91
|
# stop — every stage waits for you (default)
|
|
80
92
|
# run-through — after you approve a spec: implement → verify → review → checkpoint in one go,
|