@jagreehal/shepherd 0.14.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -56,7 +56,12 @@ to all of them.
56
56
  nothing risky, no choice of what the product does. A resolution changes the
57
57
  PR's diff, so it ends the iteration `pending` until swarm has read it.
58
58
  - **Behaviour changes made during review are disclosed** in one marked section
59
- of the PR description; the rest of the body is the author's.
59
+ of the PR description; the rest of the body is the author's. A PR `/shepherd`
60
+ opens says in its first line that an agent wrote it and the author has not
61
+ reviewed it; only the author changes that line.
62
+ - **A security failure on a public repository stays private.** Incident opens
63
+ no issue, branch or PR for one and writes no test: a reproducing test is a
64
+ working exploit.
60
65
  - **In CI the model holds no write credential.** Handoff mode commits and
61
66
  never pushes; a job with no model pushes the bundle after its fast-forward
62
67
  and path checks. Never name `SHEPHERD_TOKEN` anywhere in the model's job (a secret any
@@ -89,7 +94,7 @@ to all of them.
89
94
  style) report MEDIUM at most. Personal lenses (`~/.config/shepherd`) are a
90
95
  local preview: their findings never post, so they are never fixed.
91
96
  - **House rules bind fixers too.** Triage, simplify, and ci-repair read
92
- `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, and `docs/adr/` from the default
97
+ `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, `REVIEW.md`, and `docs/adr/` from the default
93
98
  branch before editing.
94
99
  - **CI repair never weakens a test** to go green: no loosened assertions, no
95
100
  skips, no wholesale snapshot updates, no casts to `any`.
@@ -110,6 +115,9 @@ to all of them.
110
115
  post under the author's account, and a run once read shepherd's own revert
111
116
  as the author's decision; only commits without the trailer, and threads the
112
117
  author wrote, say what the author decided.
118
+ - A finding names every place its mistake repeats in the PR's added lines, so
119
+ one round closes the class. Each round costs a commit when an agent is the
120
+ author; garden tracks rounds to pass and how each rule's threads ended.
113
121
  - Reviewers (the router, every lens) start no agents and write no files; the
114
122
  loop checks the tree is clean after them.
115
123
  - GitHub is the record, not the session: swarm dedupes against open threads
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@jagreehal/shepherd",
3
- "version": "0.14.0",
3
+ "version": "0.15.0",
4
4
  "description": "Agent skills that drive a PR to merge-ready: a review swarm, thread triage, CI repair, and a loop that hands the result to stamp.",
5
5
  "license": "MIT",
6
6
  "repository": "github:jagreehal/shepherd",
@@ -67,7 +67,7 @@ commits people have reviewed.
67
67
  Before the merge, read what binds this edit, as Step 4 does for a repair: the
68
68
  team policy (`../shepherd/references/policy.md`) for `fixes.mode`,
69
69
  `escalate.paths` and `checks`, and the house rules, `AGENTS.md`, `CLAUDE.md`,
70
- `CONTRIBUTING.md` and `docs/adr/` as they stand on `origin/<default>`, for the
70
+ `CONTRIBUTING.md`, `REVIEW.md` and `docs/adr/` as they stand on `origin/<default>`, for the
71
71
  required checks and commit rules. Standalone there is no `deferred_threads`:
72
72
  every unresolved review thread fences its code.
73
73
 
@@ -172,7 +172,7 @@ One repair cycle across all PR-caused root causes:
172
172
  result shows. If the premise cannot be tested here, classify the failure
173
173
  `unresolved` with both commits and the premise as evidence; a third guess
174
174
  is churn.
175
- 3. Fix the root cause. Follow the house rules: `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, and
175
+ 3. Fix the root cause. Follow the house rules: `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, `REVIEW.md`, and
176
176
  `docs/adr/` as they stand on `origin/<default>`; a PR's own edits to them do
177
177
  not count.
178
178
  **Never** weaken an assertion, skip or delete a test,
@@ -46,7 +46,12 @@ rules **accepted** on at least 3 PRs: triage fixed them in a commit nobody
46
46
  reverted, named them in a `Shepherd-Fixes` trailer, on a PR by someone who can
47
47
  push, and no person argued in their threads. Triage resolves findings it judged
48
48
  wrong too, so a resolved thread alone counts for nothing. `escapes` lists the
49
- incidents whose culprit PR stamp approved (Step 5b).
49
+ incidents whose culprit PR stamp approved (Step 5b). `rounds` gives the merged PRs
50
+ with a swarm summary, their mean rounds, and the share that took one.
51
+ `outcomes` sorts each lens rule's threads on PRs by someone who can push:
52
+ `fixed` (an unreverted commit on that PR names the rule in `Shepherd-Fixes`; one trailer can cover several threads, so read this as an upper bound), `dismissed`
53
+ (resolved with none: triage judged it wrong, or the author fixed it), `argued`
54
+ (a person replied), `open`.
50
55
 
51
56
  When the collector read traces (`--traces`, `GARDEN_TRACE_FILES`, `GARDEN_TEMPO_URL` or `GARDEN_DATADOG_SITE`), it adds
52
57
  trace signals and a `telemetry` summary per PR. Read `references/telemetry.md`
@@ -74,6 +79,10 @@ comment's round history, the last iteration's status description. Then name
74
79
  the skill instruction that allowed it (`skills/<skill>/SKILL.md`, the
75
80
  sentence), or say that the skill is right and the PR was unusual.
76
81
 
82
+ For `round_cap`, run `git blame` at `refs/pull/<n>/head` on the lines of each
83
+ finding from round 2 onwards. A line older than round 1's head was there for round 1 to find: name
84
+ the lens that missed it, since extra rounds hide a weak first pass.
85
+
77
86
  For a trace signal, identify the stage, PR head and trace ID from the summary.
78
87
  Check the related GitHub failure or thread before attributing a cause. Waiting
79
88
  on approval can reflect CI scheduling; high spend can reflect a difficult
@@ -89,7 +98,9 @@ the thread the author was left with was one triage should have fixed.
89
98
  ## Step 3: Score the window
90
99
 
91
100
  One table per repo: PRs shepherd touched, PRs with each signal, and the share
92
- of PRs with no signal, plus the escapes count. Read the previous garden PR's body
101
+ of PRs with no signal, the mean rounds and one-round share, plus the escapes
102
+ count. A rule with more `dismissed` and `argued` than `fixed` threads is noise:
103
+ read those threads, and treat it as a cause in Step 2. Read the previous garden PR's body
93
104
  (`gh pr list -R jagreehal/shepherd --search "garden:" --state all --limit 1`)
94
105
  and give the change since then, so a merged edit shows whether it helped.
95
106
 
@@ -160,6 +171,14 @@ and stamp's review on it, and the fix PR with its reproducing test.
160
171
  the incident ("a function that now dereferences an argument its callers can
161
172
  pass as null needs a test with that input").
162
173
 
174
+ **Each `coverage_gap` escape also becomes an eval case.** The culprit PR's base
175
+ and head are the pair; the reproducing test gives the bug's `claim`, `checked`,
176
+ `at` and `match` (`evals/README.md` in shepherd). Set `split` to `heldout`. A
177
+ public repository's case joins `evals/cases/` in a shepherd PR titled
178
+ `garden: eval cases <date>`, with the finding and decoys `src/eval.test.ts`
179
+ wants for a new bug, and whoever merges it is the person confirming the bug. A private repository's case stays in that repository's own `evals/cases/`,
180
+ in the same PR as its rule.
181
+
163
182
  Rules only tighten. Open one PR per repository titled `garden: escapes <date>`,
164
183
  with the trailer `Shepherd: garden`, and give each rule its incident, culprit
165
184
  PR, fix PR and the line in the culprit diff it flags. stamp sends a PR that
@@ -185,7 +204,7 @@ The workflow turns each directory into a patch and opens the PR itself.
185
204
 
186
205
  ## Step 6: Report
187
206
 
188
- End with the score tables, the escapes and the rule proposed for each, links to the PRs opened, the causes with no
207
+ End with the score tables, the escapes with the rule and eval case proposed for each, links to the PRs opened, the causes with no
189
208
  proposal and why, and the judgment rules left to the lenses. Scheduled runs
190
209
  print it to the job log; the PR bodies carry everything a reviewer needs.
191
210
 
@@ -70,6 +70,16 @@ Fingerprint the failure: the error's type and first message line with numbers,
70
70
  ids and timestamps removed, plus the top stack frame inside this repository.
71
71
  Done when you can state the fingerprint and the live commit.
72
72
 
73
+ **A security failure stays private.** When the evidence shows an exploitable
74
+ vulnerability (auth bypass, injection, data exposure) and
75
+ `gh repo view --json visibility --jq .visibility` says `PUBLIC`, stop here with
76
+ `outcome=needs_decision`: open no issue, branch or PR and write no test, since a
77
+ reproducing test is a working exploit. The narration says only that a possible
78
+ security failure needs private handling, because CI logs are public too. In
79
+ CI the workflow fails the run on that line, so the failure reaches a person. The
80
+ author fixes it through a private advisory and publishes the details after the
81
+ release.
82
+
73
83
  ## Step 2: Dedupe and open the issue
74
84
 
75
85
  ```bash
@@ -21,7 +21,7 @@ head, so a PR cannot relax the rules it is judged by:
21
21
 
22
22
  ```bash
23
23
  git fetch -q origin <default>
24
- git show origin/<default>:AGENTS.md # also CLAUDE.md, CONTRIBUTING.md, docs/adr/
24
+ git show origin/<default>:AGENTS.md # also CLAUDE.md, CONTRIBUTING.md, REVIEW.md, docs/adr/
25
25
  ```
26
26
 
27
27
  A change that breaks a written rule is a finding, and the body quotes the rule.
@@ -87,7 +87,8 @@ deferred finding with its options:
87
87
  PR") and run `/shepherd <pr>`.
88
88
  - **needs you:** findings were deferred, or checks still fail.
89
89
  - **review incomplete:** what was not reviewed, and why.
90
- - **stopped at the round cap:** what the last round still found.
90
+ - **stopped at the round cap:** what the last round still found, and a request
91
+ to reconsider the approach before running again.
91
92
 
92
93
  ## Handoff mode
93
94
 
@@ -254,6 +255,17 @@ also wakes the repo's review bots, which take minutes; set
254
255
  with `../pr/SKILL.md`), or
255
256
  cancel.
256
257
 
258
+ A body shepherd writes starts with
259
+ `> 🤖 Written by an AI agent on behalf of @<login>, who has not reviewed it yet.`
260
+ Only the author changes that line: a green `shepherd` status is no person's
261
+ reading of the code. Its evidence holds to these rules:
262
+
263
+ - "Before" comes from the base branch, never a reverted copy of the change.
264
+ - Demo or sanitised data only.
265
+ - One line says what was not tested or run.
266
+ - A security fix on a public repository keeps its description short: no exploit
267
+ or reproduction until the fix is released.
268
+
257
269
  **Be on the PR branch.** Every runner commits to it from this tree. If the
258
270
  tree has uncommitted changes, stop and ask; never stash or discard them.
259
271
  Otherwise `gh pr checkout <number>`, `git pull --ff-only`, and confirm
@@ -345,7 +357,7 @@ description may not mention, and stamp refuses undisclosed behaviour in risky
345
357
  territory. When this round's commits changed what the code accepts, returns,
346
358
  throws, or writes, upsert one section at the end of the PR body, between
347
359
  `<!-- shepherd:review-changes -->` markers, headed "Changes made during review"
348
- with one line per behaviour change and its commit. Re-check the lines already
360
+ with one line per behaviour change, its commit, and the test that covers it. Re-check the lines already
349
361
  there against the head: correct or drop one a later commit undid, so the
350
362
  section never claims a change the code no longer makes. Edit only that section;
351
363
  the rest of the body is the author's. `gh pr view --json body`, then
@@ -507,7 +519,8 @@ shepherd trace flush --pr <n> --repo <owner/repo> --head <H2> --base <base_sha>
507
519
  - `failure`: the loop can do nothing more on its own (the "needs the author"
508
520
  terminal condition below). The description says what is needed, most
509
521
  important first: `stamp refused: <reason>`, `stamp escalated`, `changes
510
- requested by <login>`, `3 threads need you`, `base conflict`.
522
+ requested by <login>`, `3 threads need you`, `base conflict`. A loop that
523
+ stopped at its cap adds `round cap: reconsider the approach`.
511
524
  - `pending`: anything else: work remains, stamp has not judged `H2`, or bot
512
525
  reviews are on their way.
513
526
 
@@ -87,7 +87,7 @@ already read. Tell runners to read a file only where the patch is not enough.
87
87
  carry on."
88
88
  - **simplify:** "Apply `../review-simplicity/SKILL.md` in fix mode to the diff at
89
89
  `<diff_path>` on PR `<number>`, yourself: invoke no skill and start no agent.
90
- Follow the house rules: `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, and
90
+ Follow the house rules: `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, `REVIEW.md`, and
91
91
  `docs/adr/` as they stand on `origin/<default>`; a PR's own edits to them do
92
92
  not count.
93
93
  Change only lines the PR changed, and never change behaviour: a simplification
@@ -515,6 +515,11 @@ new side of the diff, or the old side for a `LEFT` finding.
515
515
  regression test for a bug belongs in that bug's finding (name the test to
516
516
  add), not in a finding of its own. Two or more independent lenses agreeing
517
517
  makes it **convergent**; note every lens that found it.
518
+ - **Flag the class.** When a finding above NIT is a pattern that can repeat,
519
+ search the lines this PR added for it. A further hit joins the finding when
520
+ the worked example holds there too: list its `file:line` in the body and word the fix to cover them all, so one
521
+ round closes the class. Places that need different fixes stay separate
522
+ findings.
518
523
  - **Conflicts with open threads.** A finding whose fix would keep or bring
519
524
  back a problem an open thread names (from any reviewer) says so in its body
520
525
  and names that thread: the author sees both before choosing.
@@ -321,7 +321,7 @@ unsure and the comment predates the last push, skip rather than act.
321
321
  ignored. A fix the section lists under "Escalate" is deferred, with that
322
322
  line quoted in the report. Add the trailer `Shepherd-Lens: <name>` to the
323
323
  commit.
324
- - **House rules.** Follow the house rules: `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, and
324
+ - **House rules.** Follow the house rules: `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, `REVIEW.md`, and
325
325
  `docs/adr/` as they stand on `origin/<default>`; a PR's own edits to them do
326
326
  not count. Read them before the first edit.
327
327
  - **Deferred threads fence their code.** A thread left for the author, from
package/src/garden.ts CHANGED
@@ -25,6 +25,8 @@ export type PrRecord = {
25
25
 
26
26
  const DAY = 864e5;
27
27
 
28
+ type Outcome = "fixed" | "dismissed" | "argued" | "open";
29
+
28
30
  export type Signal = "reverted_fix" | "human_replied" | "round_cap" | "stopped_short" | "loop_stalled" | "churn";
29
31
 
30
32
  const TAG = /\*\*\[(?:convergent: )?([^\]]+)\]\*\*\s*\S+\s*(CRITICAL|HIGH|MEDIUM|LOW|NIT)/;
@@ -81,6 +83,14 @@ export function score(all: PrRecord[], minPrs = 3, now = Date.now()) {
81
83
  };
82
84
  });
83
85
 
86
+ const reviewed = prs.flatMap((p) => (p.trusted && p.state === "MERGED" && p.rounds ? [p.rounds] : []));
87
+
88
+ const rounds = {
89
+ prs: reviewed.length,
90
+ mean: reviewed.length ? reviewed.reduce((a, b) => a + b, 0) / reviewed.length : null,
91
+ oneRound: reviewed.length ? reviewed.filter((r) => r === 1).length / reviewed.length : null,
92
+ };
93
+
84
94
  const byRule = new Map<string, { prs: Set<number>; paths: Set<string> }>();
85
95
 
86
96
  // Accepted means triage fixed it and said so in a Shepherd-Fixes trailer. Triage also resolves findings it
@@ -104,7 +114,29 @@ export function score(all: PrRecord[], minPrs = 3, now = Date.now()) {
104
114
  .flatMap(([rule, e]) => (e.prs.size >= minPrs ? [{ rule, prs: [...e.prs].sort((a, b) => a - b), paths: [...e.paths].sort() }] : []))
105
115
  .sort((a, b) => b.prs.length - a.prs.length || a.rule.localeCompare(b.rule));
106
116
 
107
- return { prs: scored, recurring };
117
+ return { prs: scored, recurring, rounds, outcomes: outcomesOf(prs.filter((p) => p.trusted)) };
118
+ }
119
+
120
+ /** Each lens rule's threads by how they ended, noisiest first. Only a rule can match a trailer, so a thread with a bare lens tag is left out. */
121
+ function outcomesOf(prs: PrRecord[]) {
122
+ const byRule = new Map<string, Record<Outcome, number>>();
123
+
124
+ for (const pr of prs) {
125
+ const fixed = new Set(pr.commits.filter((c) => !isReverted(c, pr.commits)).flatMap((c) => c.fixes).flatMap(rulesOf));
126
+
127
+ for (const t of pr.threads) {
128
+ const rules = rulesOf(t.tag);
129
+ const outcome: Outcome = t.humanReplied ? "argued" : !t.resolved ? "open" : rules.some((r) => fixed.has(r)) ? "fixed" : "dismissed";
130
+
131
+ for (const rule of rules) {
132
+ const entry = byRule.get(rule) ?? { fixed: 0, dismissed: 0, argued: 0, open: 0 };
133
+ entry[outcome] += 1;
134
+ byRule.set(rule, entry);
135
+ }
136
+ }
137
+ }
138
+
139
+ return [...byRule].map(([rule, counts]) => ({ rule, ...counts })).sort((a, b) => b.dismissed + b.argued - a.dismissed - a.argued || a.rule.localeCompare(b.rule));
108
140
  }
109
141
 
110
142
  /** Parse one shepherd thread's first comment and its replies; null when the thread is not shepherd's. */
@@ -175,7 +207,7 @@ const Commits = z.array(z.object({ sha: z.string(), commit: z.object({ message:
175
207
 
176
208
  const Statuses = z.object({ statuses: z.array(z.object({ context: z.string(), state: z.string(), description: z.string().nullable(), updated_at: z.string() })) });
177
209
 
178
- const Comments = z.array(z.object({ body: z.string() }));
210
+ const Comments = z.array(z.object({ body: z.string(), user: z.object({ login: z.string(), type: z.string() }).nullable() }));
179
211
 
180
212
  const Issues = z.array(z.object({ number: z.number(), title: z.string(), url: z.string(), body: z.string() }));
181
213
 
@@ -208,16 +240,19 @@ export function collect(repo: string, since: string): PrRecord[] {
208
240
  return canPush.get(login) === true;
209
241
  };
210
242
 
243
+ // Anyone can paste the header on a public PR, so shepherd's marks count only from a bot or an account that can push.
244
+ const canPost = (login: string | undefined, bot: boolean) => bot || (login !== undefined && trusted(login));
245
+
211
246
  return prs.map((pr) => {
212
247
  const query = "query($owner:String!,$name:String!,$n:Int!){repository(owner:$owner,name:$name){pullRequest(number:$n){reviewThreads(first:100){nodes{isResolved path comments(first:50){nodes{author{login __typename} body}}}}}}}";
213
248
 
214
249
  const threads = Threads.parse(JSON.parse(gh(["api", "graphql", "-f", `query=${query}`, "-f", `owner=${owner}`, "-f", `name=${name}`, "-F", `n=${pr.number}`])))
215
- .data.repository.pullRequest.reviewThreads.nodes.flatMap((t) => parseThread(t.path, t.isResolved, t.comments.nodes.map((c) => ({ author: c.author?.login ?? "ghost", bot: c.author?.__typename === "Bot", body: c.body }))) ?? []);
250
+ .data.repository.pullRequest.reviewThreads.nodes.filter((t) => canPost(t.comments.nodes[0]?.author?.login, t.comments.nodes[0]?.author?.__typename === "Bot")).flatMap((t) => parseThread(t.path, t.isResolved, t.comments.nodes.map((c) => ({ author: c.author?.login ?? "ghost", bot: c.author?.__typename === "Bot", body: c.body }))) ?? []);
216
251
 
217
252
  const commits = Commits.parse(JSON.parse(gh(["api", "--paginate", "--slurp", `repos/${repo}/pulls/${pr.number}/commits`])).flat()).map((c) => parseCommit(c.sha, c.commit.message, c.commit.committer?.date ?? ""));
218
253
 
219
254
  const status = Statuses.parse(JSON.parse(gh(["api", `repos/${repo}/commits/${pr.headRefOid}/status`]))).statuses.find((s) => s.context === "shepherd");
220
- const summary = Comments.parse(JSON.parse(gh(["api", "--paginate", "--slurp", `repos/${repo}/issues/${pr.number}/comments`])).flat()).find((c) => c.body.includes("<!-- shepherd-swarm-summary -->"));
255
+ const summary = Comments.parse(JSON.parse(gh(["api", "--paginate", "--slurp", `repos/${repo}/issues/${pr.number}/comments`])).flat()).find((c) => c.body.includes("<!-- shepherd-swarm-summary -->") && canPost(c.user?.login, c.user?.type === "Bot"));
221
256
 
222
257
  return { number: pr.number, headSha: pr.headRefOid, title: pr.title, state: pr.state, threads, commits, status: status ? { state: status.state, description: status.description ?? "", updatedAt: status.updated_at } : null, rounds: roundsOf(summary?.body), trusted: trusted(pr.author.login) };
223
258
  });
@@ -126,11 +126,13 @@ jobs:
126
126
  You are running in GitHub Actions on the default branch, already checked out, with no push access.
127
127
  At Step 6 do not push or open the PR: commit on incident/<issue>, write the PR title to $HANDOFF/pr-title.txt
128
128
  and the body to $HANDOFF/pr-body.md; the next job pushes and opens it.
129
+ When the skill stops because a security failure stays private, create the empty file $HANDOFF/private.
129
130
  Nobody is watching this session: never wait for an answer. Anything you would ask is reported in the final JSON."
130
131
  - name: Bundle the fix branch
131
132
  env:
132
133
  DEFAULT: ${{ github.event.repository.default_branch }}
133
134
  run: |
135
+ [ ! -e "$RUNNER_TEMP/handoff/private" ] || { echo "::error::a possible security failure needs private handling"; exit 1; }
134
136
  branch=$(git branch --list 'incident/*' --format='%(refname:short)' | head -1)
135
137
  [ -n "$branch" ] || { echo "no fix branch: see the incident issue"; exit 0; }
136
138
  echo "$branch" > "$RUNNER_TEMP/handoff/branch"