@oakgreencc/cambium 0.0.18 → 0.0.20

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -37,8 +37,8 @@ Every command takes `--config <file>`; the default is `$CAMBIUM_CONFIG`, then `~
37
37
 
38
38
  Each step is bounded by `stepTimeoutSeconds`. A step that throws or times out is logged and recorded in the heartbeat, and the tick moves on. Nothing waits on a worker.
39
39
 
40
- 1. **Read.** One GraphQL query per repo (`graft graphql`): open issues with any `q:` label, open and recently merged PRs on `agent/*` with their check rollup, merge-queue state and the issues they close.
41
- 2. **Reconcile.** For each tracked worker: alive and under its wall-clock cap → nothing. Over the cap → kill its process group. Gone → exit check (a PR on its branch, or closing its issue, that is open or merged and not still on the commit CI already failed) → `submitted`; otherwise requeue with the stderr tail (a failed prepare step requeues as `spawn-failed`). Then the worktree is torn down if `graft status` says GitHub holds everything in it, else kept with a `warn` saying why.
40
+ 1. **Read.** One GraphQL query per repo (`graft graphql`): open issues with any `q:` label, open and recently merged PRs on `agent/*` with their check rollup, merge-queue state and the issues they close. GitHub links closing references only for a PR into the default branch, so cambium also reads `Closes/Fixes/Resolves #n` from the PR body, fetching issues the first query did not read in one more. A worker's PR that names no issue (found by its branch) gets `Closes #n` added at submission.
41
+ 2. **Reconcile.** For each tracked worker: alive and under its wall-clock cap → nothing. Over the cap → kill its process group and every descendant (Claude Code runs each Bash command in a group of its own, so a group kill alone left a test run loading the next attempt). Gone → exit check (a PR on its branch, or closing its issue, that is open or merged and not still on the commit CI already failed) → `submitted`; otherwise requeue with the stderr tail (a failed prepare step requeues as `spawn-failed`). Then the worktree is torn down if `graft status` says GitHub holds everything in it, else kept with a `warn` saying why.
42
42
  3. **Watch** (no agent), for agent PRs whose issue is in no queue: merged → close the issue, delete the branch; red → requeue with the failing checks' summaries; left the merge queue without merging → requeue; not merged `mergeQueueMinutes` after submission → `q:human`.
43
43
  4. **Sweep.** An issue labelled `running` with this host (or no host) in its marker and no tracked worker → requeue. This is what makes losing the SQLite file cost one requeue.
44
44
  5. **Schedule.** Start Task workers for `q:task` issues without `running`, by priority label then age, up to `phases.task.cap` counted from live workers. A task requeued earlier in the same tick can start in that tick. Then, with `engine: "graph"` and `phases.dispatch.cap > 0`, start triage workers for `q:intake` issues the same way ([the dispatch graph](#the-dispatch-graph)).
@@ -81,7 +81,7 @@ A worker is `claude -p --output-format stream-json --verbose --max-turns <maxTur
81
81
 
82
82
  - in its own worktree, by the target repo's [worktree convention](#target-repo-convention);
83
83
  - behind the prepare step: `node src/prepare.mjs -- claude …` runs install-fresh, hydrate and the repo's `prepare` commands in the worktree, writing to the worker's stderr log, then becomes the worker. It runs in the detached process so a long install never blocks a tick;
84
- - detached, leading its own process group (a kill takes its children), stdin from the prompt file;
84
+ - detached, leading its own process group (a kill takes the group and, by a `ps` snapshot, every descendant), stdin from the prompt file;
85
85
  - with `CAMBIUM_REPO`, `CAMBIUM_ISSUE`, `CAMBIUM_PHASE`, `CAMBIUM_BRANCH` set, `GRAFT_APP=<worker.graftApp>` when configured, then `worker.env`, then `TRACEPARENT` (the starting node's span) when [tracing](#tracing) is on;
86
86
  - with its prompt, stdout and stderr at `<paths.logs>/<owner>__<name>/issue-<n>/a<attempt>-<stamp>.{prompt.md,out.log,err.log}`, keeping the newest `logs.keepPerIssue` runs per issue.
87
87
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@oakgreencc/cambium",
3
- "version": "0.0.18",
3
+ "version": "0.0.20",
4
4
  "description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
5
5
  "license": "AGPL-3.0-only",
6
6
  "type": "module",
@@ -51,7 +51,7 @@
51
51
  "@opentelemetry/sdk-metrics": "^3.1.1",
52
52
  "@opentelemetry/sdk-trace": "^3.1.1",
53
53
  "zod": "^4.6.7",
54
- "@oakgreencc/mycelium": "0.0.0"
54
+ "@oakgreencc/mycelium": "0.0.6"
55
55
  },
56
56
  "devDependencies": {
57
57
  "@langchain/core": "^1.4.1",
package/src/github.mjs CHANGED
@@ -47,10 +47,12 @@ import { Q, QUEUE_LABELS } from './labels.mjs'
47
47
  * editPull(slug: string, number: number, change: { title: string, body: string }): Promise<void>,
48
48
  * closePull(slug: string, number: number, comment: string): Promise<void>,
49
49
  * mergePull(slug: string, number: number): Promise<void>,
50
+ * linkIssue(slug: string, pr: number, issue: number): Promise<void>,
50
51
  * }} GitHub
52
+ * `linkIssue`: put `Closes #<issue>` at the top of the PR's body, so the PR names the issue it closes.
51
53
  */
52
54
 
53
- const PR_FIELDS = `number url headRefName state headRefOid isInMergeQueue
55
+ const PR_FIELDS = `number url headRefName state headRefOid isInMergeQueue body
54
56
  headRef { id }
55
57
  autoMergeRequest { enabledAt }
56
58
  closingIssuesReferences(first: 5) { nodes { ...I } }
@@ -93,6 +95,26 @@ fragment P on PullRequest {
93
95
  comments(first: 100) { nodes { databaseId body author { login } } }
94
96
  }`
95
97
 
98
+ /** GitHub's closing keywords before a same-repo `#n` (graft's closing-refs.mjs, without cross-repo refs). */
99
+ const CLOSING = /\b(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\b\s*:?\s+#(\d+)\b/gi
100
+
101
+ /**
102
+ * The issues a PR body says it closes. GitHub links these (`closingIssuesReferences`) only for a PR into the
103
+ * DEFAULT branch, so a repo whose agents target another base (sektorapp: `develop`) gets nothing from GitHub;
104
+ * cambium reads the body itself.
105
+ * @param {string | null | undefined} body @returns {number[]}
106
+ */
107
+ export function bodyCloses(body) {
108
+ return [...new Set([...(body ?? '').matchAll(CLOSING)].map((m) => Number(m[1])))]
109
+ }
110
+
111
+ /** The issues `numbers`, by alias, for {@link addIssues}. @param {number[]} numbers */
112
+ export const issuesQuery = (numbers) =>
113
+ `query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { ${numbers
114
+ .map((n) => `i${n}: issueOrPullRequest(number: ${n}) { __typename ... on Issue { ...I } }`)
115
+ .join(' ')} } }
116
+ ${SNAPSHOT_QUERY.slice(SNAPSHOT_QUERY.indexOf('fragment I on Issue'), SNAPSHOT_QUERY.indexOf('fragment P on'))}`
117
+
96
118
  const FAILED = new Set(['FAILURE', 'ERROR', 'TIMED_OUT', 'CANCELLED', 'ACTION_REQUIRED', 'STARTUP_FAILURE'])
97
119
 
98
120
  /**
@@ -156,6 +178,38 @@ function normalizeRelease(repo, { from, to }, botLogin) {
156
178
  }
157
179
  }
158
180
 
181
+ /** An issue node → {@link Issue}. @param {any} n @param {string | null} botLogin @returns {Issue} */
182
+ function toIssue(n, botLogin) {
183
+ return {
184
+ number: n.number,
185
+ title: n.title ?? '',
186
+ url: n.url ?? '',
187
+ state: n.state,
188
+ createdAt: n.createdAt,
189
+ body: n.body ?? '',
190
+ labels: labelsOf(n),
191
+ stateComment: stateCommentOf(n, botLogin),
192
+ }
193
+ }
194
+
195
+ /** The PRs' closed issues the snapshot has not read. @param {Snapshot} snap @returns {number[]} */
196
+ export const unreadCloses = (snap) => [...new Set(snap.prs.flatMap((p) => p.closes).filter((n) => !snap.issues.has(n)))]
197
+
198
+ /**
199
+ * Add the answer to {@link issuesQuery} to the snapshot; a number that is not an issue (a PR, or gone) is dropped
200
+ * from every PR's `closes`.
201
+ * @param {Snapshot} snap @param {any} data @param {number[]} asked @param {string | null} [botLogin]
202
+ */
203
+ export function addIssues(snap, data, asked, botLogin = null) {
204
+ const repo = (data?.data ?? data)?.repository ?? {}
205
+ for (const n of asked) {
206
+ const node = repo[`i${n}`]
207
+ if (node?.__typename === 'Issue') snap.issues.set(n, toIssue(node, botLogin))
208
+ }
209
+ for (const p of snap.prs) p.closes = p.closes.filter((n) => snap.issues.has(n))
210
+ return snap
211
+ }
212
+
159
213
  /**
160
214
  * GraphQL `data` → {@link Snapshot}. Only `agent/*` PRs are kept.
161
215
  *
@@ -170,18 +224,7 @@ export function normalize(data, { botLogin = null, release = null } = {}) {
170
224
  /** @type {Map<number, Issue>} */
171
225
  const issues = new Map()
172
226
  const addIssue = (/** @type {any} */ n) => {
173
- if (!issues.has(n.number)) {
174
- issues.set(n.number, {
175
- number: n.number,
176
- title: n.title ?? '',
177
- url: n.url ?? '',
178
- state: n.state,
179
- createdAt: n.createdAt,
180
- body: n.body ?? '',
181
- labels: labelsOf(n),
182
- stateComment: stateCommentOf(n, botLogin),
183
- })
184
- }
227
+ if (!issues.has(n.number)) issues.set(n.number, toIssue(n, botLogin))
185
228
  return n.number
186
229
  }
187
230
  for (const n of [...(repo.issues?.nodes ?? []), ...(repo.awaiting?.nodes ?? [])]) addIssue(n)
@@ -208,7 +251,7 @@ export function normalize(data, { botLogin = null, release = null } = {}) {
208
251
  autoMerge: Boolean(p.autoMergeRequest),
209
252
  checks: rollup?.state ?? null,
210
253
  failing,
211
- closes: (p.closingIssuesReferences?.nodes ?? []).map(addIssue),
254
+ closes: [...new Set([...(p.closingIssuesReferences?.nodes ?? []).map(addIssue), ...bodyCloses(p.body)])],
212
255
  })
213
256
  }
214
257
  return { issues, prs, ...(release ? { release: normalizeRelease(repo, release, botLogin) } : {}) }
@@ -262,7 +305,18 @@ export function graftGitHub({ config, exec: run = exec }) {
262
305
  const branches = release ?? { from: '', to: '' }
263
306
  const vars = JSON.stringify({ owner, name, labels: QUEUE_LABELS, release: Boolean(release), ...branches })
264
307
  const data = await json(graft(slug, ['graphql', SNAPSHOT_QUERY, '--vars', vars]))
265
- return normalize(data, { botLogin: config.graft.botLogin, release })
308
+ const snap = normalize(data, { botLogin: config.graft.botLogin, release })
309
+ const unread = unreadCloses(snap)
310
+ if (!unread.length) return snap
311
+ // Issues named only in PR bodies (in no queue): one more query, for those.
312
+ const ask = (/** @type {number[]} */ ns) =>
313
+ json(graft(slug, ['graphql', issuesQuery(ns), '--vars', JSON.stringify({ owner, name })]))
314
+ // A number that resolves to nothing fails the whole query: then one at a time, skipping those.
315
+ const more = await ask(unread).catch(async () => {
316
+ const one = await Promise.all(unread.map((n) => ask([n]).catch(() => null)))
317
+ return { repository: Object.assign({}, ...one.map((d) => d?.repository ?? {})) }
318
+ })
319
+ return addIssues(snap, more, unread, config.graft.botLogin)
266
320
  },
267
321
  async label(slug, number, { add, remove }) {
268
322
  if (!add.length && !remove.length) return
@@ -312,5 +366,12 @@ export function graftGitHub({ config, exec: run = exec }) {
312
366
  async mergePull(slug, number) {
313
367
  await graft(slug, ['pr', 'merge', String(number), '--auto'])
314
368
  },
369
+ async linkIssue(slug, pr, issue) {
370
+ const path = `/repos/${slug}/pulls/${pr}`
371
+ const { body } = await json(graft(slug, ['api', 'GET', path]))
372
+ if (bodyCloses(body).includes(issue)) return
373
+ const linked = `Closes #${issue}${body ? `\n\n${body}` : ''}`
374
+ await graft(slug, ['api', 'PATCH', path, '--body', JSON.stringify({ body: linked })])
375
+ },
315
376
  }
316
377
  }
@@ -13,7 +13,7 @@ import { branchName } from '../convention.mjs'
13
13
  import { failureSummary, iso, key, markerOf, MIN, setLabels, setMarker } from '../issue-state.mjs'
14
14
  import { move, Q, WHY } from '../labels.mjs'
15
15
  import { withHistory } from '../marker.mjs'
16
- import { prFor, taskExit, taskPrompt } from '../phases/task.mjs'
16
+ import { linkIssue, prFor, taskExit, taskPrompt } from '../phases/task.mjs'
17
17
  import { PREPARE_FAILED_LINE } from '../prepare.mjs'
18
18
  import { traceWorker } from '../tracing.mjs'
19
19
 
@@ -418,6 +418,7 @@ const ciRunning = declare(
418
418
  input: { pr: input.pr, branch: input.branch },
419
419
  })
420
420
  ctx.log('info', `submitted PR #${input.pr}; watching CI`, { repo: ctx.subject.slug, issue: ctx.subject.number })
421
+ await linkIssue(ctx, ctx.subject.slug, prOf(ctx, input), ctx.subject.number)
421
422
  },
422
423
  },
423
424
  )
@@ -72,3 +72,29 @@ export function taskExit(pr, marker = {}) {
72
72
  reason: pr ? `PR #${pr.number} was closed without merging` : 'worker exited without a published PR',
73
73
  }
74
74
  }
75
+
76
+ /**
77
+ * Make a submitted PR name its issue (`Closes #n`) when it does not: Watch finds a PR's issue through the PR's
78
+ * closing references, so a PR matched only by branch at exit would otherwise never be watched (no merge, no
79
+ * stall). A failure is logged, not fatal: the submission stands.
80
+ *
81
+ * @param {{ github: import('../github.mjs').GitHub, log: (level: string, msg: string, where?: object) => void }} deps
82
+ * @param {string} slug @param {import('../github.mjs').PullRequest | null | undefined} pr @param {number} issue
83
+ */
84
+ export async function linkIssue(deps, slug, pr, issue) {
85
+ if (!pr || pr.closes.includes(issue)) return
86
+ try {
87
+ await deps.github.linkIssue(slug, pr.number, issue)
88
+ pr.closes.push(issue)
89
+ deps.log('info', `PR #${pr.number} did not name its issue; added Closes #${issue}`, { repo: slug, issue })
90
+ } catch (err) {
91
+ deps.log(
92
+ 'warn',
93
+ `could not add Closes #${issue} to PR #${pr.number}: ${/** @type {Error} */ (err)?.message ?? err}`,
94
+ {
95
+ repo: slug,
96
+ issue,
97
+ },
98
+ )
99
+ }
100
+ }
package/src/tick.mjs CHANGED
@@ -10,7 +10,7 @@ import { byPriorityThenAge, move, Q, queueOf, RUNNING, WHY } from './labels.mjs'
10
10
  import { withHistory } from './marker.mjs'
11
11
  import { failureSummary, iso, key, markerOf, MIN, setLabels, setMarker } from './issue-state.mjs'
12
12
  import { GRAPH_ENGINE } from './graph-engine.mjs'
13
- import { prFor, taskExit, taskPrompt } from './phases/task.mjs'
13
+ import { linkIssue, prFor, taskExit, taskPrompt } from './phases/task.mjs'
14
14
  import { branchName } from './convention.mjs'
15
15
  import { PREPARE_FAILED_LINE } from './prepare.mjs'
16
16
  import { releaseDue, releaseNotBuilt } from './phases/release.mjs'
@@ -289,6 +289,7 @@ export async function finish(deps, ctx, w, snap) {
289
289
  await setLabels(deps, w.repo, issue, move(issue.labels, null))
290
290
  ctx.touched.add(key(w.repo, w.issue))
291
291
  deps.log('info', `submitted PR #${pr.number}; watching CI`, { repo: w.repo, issue: w.issue })
292
+ await linkIssue(deps, w.repo, pr, w.issue)
292
293
  } else {
293
294
  const tail = deps.spawner.tail(w.stderr, 15)
294
295
  const reason = `${'reason' in verdict ? verdict.reason : 'worker failed'}${tail ? `\n--- stderr (last lines, ${w.stderr})\n${tail}` : ''}`
package/src/workers.mjs CHANGED
@@ -5,7 +5,7 @@
5
5
  * group, behind the prepare step, with its output in rotated log files on this
6
6
  * host, and {@link Procs} to probe and kill by pid.
7
7
  */
8
- import { spawn } from 'node:child_process'
8
+ import { spawn, spawnSync } from 'node:child_process'
9
9
  import {
10
10
  closeSync,
11
11
  existsSync,
@@ -311,6 +311,40 @@ export function processSpawner({
311
311
  }
312
312
  }
313
313
 
314
+ /**
315
+ * Every pid under `root` in a `pid ppid` process table (`ps -A -o pid=,ppid=`), deepest last.
316
+ * @param {number} root @param {string} table @returns {number[]}
317
+ */
318
+ export function descendants(root, table) {
319
+ /** @type {Map<number, number[]>} */
320
+ const children = new Map()
321
+ for (const line of table.split('\n')) {
322
+ const [pid, ppid] = line.trim().split(/\s+/).map(Number)
323
+ if (!Number.isInteger(pid) || !Number.isInteger(ppid) || pid === ppid) continue
324
+ children.set(ppid, [...(children.get(ppid) ?? []), pid])
325
+ }
326
+ const out = []
327
+ const seen = new Set([root])
328
+ for (let i = -1; i < out.length; i++) {
329
+ for (const c of children.get(i < 0 ? root : out[i]) ?? []) {
330
+ if (seen.has(c)) continue
331
+ seen.add(c)
332
+ out.push(c)
333
+ }
334
+ }
335
+ return out
336
+ }
337
+
338
+ /** @param {number} target */
339
+ function sigkill(target) {
340
+ try {
341
+ process.kill(target, 'SIGKILL')
342
+ return true
343
+ } catch {
344
+ return false
345
+ }
346
+ }
347
+
314
348
  /** @type {Procs} */
315
349
  export const realProcs = {
316
350
  alive(pid) {
@@ -322,12 +356,13 @@ export const realProcs = {
322
356
  }
323
357
  },
324
358
  kill(pid) {
359
+ // The process group is not enough: Claude Code runs each Bash command in a group of its own, so a test
360
+ // run the worker started outlives the group kill and loads the next attempt. Snapshot the tree first
361
+ // (killed children reparent to launchd and drop out of it), then kill it and the group.
362
+ const ps = spawnSync('ps', ['-A', '-o', 'pid=,ppid='], { encoding: 'utf8' })
363
+ const tree = ps.status === 0 ? descendants(pid, ps.stdout) : []
325
364
  // A detached worker leads its own process group: the negative pid takes its children with it.
326
- for (const target of [-pid, pid]) {
327
- try {
328
- process.kill(target, 'SIGKILL')
329
- return
330
- } catch {}
331
- }
365
+ if (!sigkill(-pid)) sigkill(pid)
366
+ for (const p of tree) sigkill(p)
332
367
  },
333
368
  }