@oakgreencc/cambium 0.0.29 → 0.0.31
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -2
- package/package.json +2 -2
- package/src/config.mjs +5 -0
- package/src/github.mjs +38 -4
- package/src/graph-engine.mjs +2 -0
- package/src/launchd.mjs +12 -2
- package/src/tick.mjs +3 -2
- package/src/upkeep.mjs +144 -0
package/README.md
CHANGED
|
@@ -38,13 +38,14 @@ Every command takes `--config <file>`; the default is `$CAMBIUM_CONFIG`, then `~
|
|
|
38
38
|
|
|
39
39
|
Each step is bounded by `stepTimeoutSeconds`. A step that throws or times out is logged and recorded in the heartbeat, and the tick moves on. Nothing waits on a worker.
|
|
40
40
|
|
|
41
|
-
1. **Read.** One GraphQL query per repo (`graft graphql`): open issues with any `q:` label, open and recently merged PRs on `agent/*` with their check rollup, merge-queue state, `mergeable`, `reviewDecision` and the issues they close. GitHub links closing references only for a PR into the default branch, so cambium also reads `Closes/Fixes/Resolves #n` from the PR body, fetching issues the first query did not read in one more. A worker's PR that names no issue (found by its branch) gets `Closes #n` added at submission.
|
|
41
|
+
1. **Read.** One GraphQL query per repo (`graft graphql`): open issues with any `q:` label (100 per page; more pages follow the cursor, so a busy queue's newest issues are not invisible), open and recently merged PRs on `agent/*` with their check rollup, merge-queue state, `mergeable`, `reviewDecision` and the issues they close. GitHub links closing references only for a PR into the default branch, so cambium also reads `Closes/Fixes/Resolves #n` from the PR body, fetching issues the first query did not read in one more (also the issue an open PR that closes none names by its branch, `agent/<n>-…`, for the [reaper](#upkeep)). A worker's PR that names no issue (found by its branch) gets `Closes #n` added at submission.
|
|
42
42
|
2. **Reconcile.** For each tracked worker: alive and under its wall-clock cap → nothing. Over the cap → terminate it: every process below `claude` (its Bash tools, a test run) is SIGKILLed at once, and `claude` gets SIGTERM, on which Claude Code ends its interaction span and flushes its exporter (a bare SIGKILL left `claude_code.*` spans orphaned under a root that was never exported). It usually exits within a second; still alive 30 s past the cap (and at least one tick later) → kill its process group and every descendant (Claude Code runs each Bash command in a group of its own, so a group kill alone left a test run loading the next attempt). Gone → exit check (a PR on its branch, or closing its issue, that is open or merged and not still on the commit CI already failed) → `submitted`; otherwise requeue with the stderr tail (a failed prepare step requeues as `spawn-failed`). Then the worktree is torn down if `graft status` says GitHub holds everything in it and no process has its cwd inside it (`lsof`; the worktree's own nx daemon excepted), else kept with a `warn` saying why. Teardown stops the worktree's nx daemon (nx keys each daemon to its workspace root, so no other checkout's is touched; left running, it served the next client from a deleted tree until `nx reset`), moves the tree to `<paths.home>/trash` with one rename, prunes git's record, and erases the trash in the background: a worktree with CocoaPods in it took `git worktree remove` past the step timeout.
|
|
43
43
|
3. **Watch** (no agent). Graph engine: the [PR graph](#the-pr-graph), once per tick for each agent PR whose CI has settled and whose issue has no Task queued or running: merge, rework, close, or `q:human`. Legacy, for agent PRs whose issue is in no queue: merged → close the issue, delete the branch; red → requeue with the failing checks' summaries; left the merge queue without merging → requeue; not merged `mergeQueueMinutes` after submission → `q:human`.
|
|
44
44
|
4. **Sweep.** An issue labelled `running` with this host (or no host) in its marker and no tracked worker → requeue. This is what makes losing the SQLite file cost one requeue.
|
|
45
45
|
5. **Schedule.** Start Task workers for `q:task` issues without `running`, by priority label then age, up to `phases.task.cap` counted from live workers. A task requeued earlier in the same tick can start in that tick. Then, with `engine: "graph"` and `phases.dispatch.cap > 0`, start triage workers for `q:intake` issues the same way ([the dispatch graph](#the-dispatch-graph)).
|
|
46
46
|
6. **Release.** Legacy: once a day at `release.time` when `release.enabled`; the run itself is a seam. Graph: the [release graphs](#the-release-graphs), per repo.
|
|
47
|
-
7. **
|
|
47
|
+
7. **Upkeep** (graph engine). The hourly [reaper](#upkeep) of untracked agent PRs, then the daily [re-triage](#upkeep) of `q:human`.
|
|
48
|
+
8. **Heartbeat.** Write the state file.
|
|
48
49
|
|
|
49
50
|
A requeue is `q:task` with attempt + 1, or, once `maxAttempts` are spent, `q:human` + `why:budget`. Who merges a Task PR is the repo's `merge` setting ([The PR graph](#the-pr-graph)): by default not cambium (a merge queue, auto-merge or a human); with `"merge": "green"` cambium merges it once CI is green. The other merge cambium asks for is an approved promotion PR, through `graft pr merge`; see [The release graphs](#the-release-graphs).
|
|
50
51
|
|
|
@@ -115,6 +116,8 @@ One JSON file. Everything but `repos` is optional; these are the defaults:
|
|
|
115
116
|
"merge": "none",
|
|
116
117
|
"priorityLabels": ["priority:critical", "priority:high", "priority:medium", "priority:low"],
|
|
117
118
|
"release": { "enabled": false, "time": "06:00" },
|
|
119
|
+
"reaper": { "everyMinutes": 60, "staleMinutes": 30, "orphanDays": 3 },
|
|
120
|
+
"retriage": { "enabled": false, "time": "05:00" },
|
|
118
121
|
"paths": {
|
|
119
122
|
"home": "~/.cambium",
|
|
120
123
|
"db": "~/.cambium/cambium.db",
|
|
@@ -168,6 +171,21 @@ How a repo's worktrees are made is the repo's call: a `"cambium"` key in its roo
|
|
|
168
171
|
|
|
169
172
|
The key decides which commands run on this host, so list `package.json` as protected (CODEOWNERS) in the target repo, next to its `"graft"` policy.
|
|
170
173
|
|
|
174
|
+
## Upkeep
|
|
175
|
+
|
|
176
|
+
Two clocks in the graph engine's `upkeep` step, after Watch (`src/upkeep.mjs`).
|
|
177
|
+
|
|
178
|
+
**Reaper** (every `reaper.everyMinutes`, default 60; 0 turns it off). It looks at each open `agent/*` PR that nothing has updated for `reaper.staleMinutes` (default 30) and that cambium is not tracking. Such a PR comes from a worker killed at its wall-clock cap whose `graft publish` still landed, or from a PR body that never said `Closes #n`. Its issue is the one it closes, else the one its branch names (`agent/<n>-…`):
|
|
179
|
+
|
|
180
|
+
| Found | Reaper does |
|
|
181
|
+
| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
182
|
+
| Issue open, no live worker, its marker's `pr` is not this PR | Adopt: `Closes #n` added if missing, then the task graph's `submit` (marker `pr`, out of every queue, `q:human` included) and a comment. The PR graph takes it from the next tick: merge when green, rework when red or conflicting, close when superseded. A fresh Task never redoes work a PR already holds. |
|
|
183
|
+
| Issue closed | Close the PR with a comment. |
|
|
184
|
+
| No issue (none named, or not readable) | Close the PR after `reaper.orphanDays` (default 3) untouched. The branch stays. |
|
|
185
|
+
| A live worker on the issue, or the issue already tracks this PR | Leave it. |
|
|
186
|
+
|
|
187
|
+
**Re-triage** (once a day at `retriage.time` when `retriage.enabled`; off by default). Every `q:human` issue goes back to `q:intake` without its `why:*`, so [triage](#the-dispatch-graph) looks again under its current rubric: done since (obsolete, closed), now checkable (defer), or now a task. Not `why:approve`, a merge only a human may make, where a re-triage would start a duplicate Task. Not one holding `running`. The reaper runs first, so an issue whose PR it adopted is not re-triaged.
|
|
188
|
+
|
|
171
189
|
## Worktree GC
|
|
172
190
|
|
|
173
191
|
Every `gc.everyMinutes` (0: off; `cambium gc` still works) the daemon looks at each worktree under a repo's convention root, the checkout itself never. Skipped: a cambium worker's, a locked one (a Claude Code session lock counts only while its pid is alive with the start it names), and one touched within `gc.graceMinutes` (its directory, HEAD, reflog or a dirty path). The rest:
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@oakgreencc/cambium",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.31",
|
|
4
4
|
"description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
|
|
5
5
|
"license": "AGPL-3.0-only",
|
|
6
6
|
"type": "module",
|
|
@@ -51,7 +51,7 @@
|
|
|
51
51
|
"@opentelemetry/sdk-metrics": "^3.1.1",
|
|
52
52
|
"@opentelemetry/sdk-trace": "^3.1.1",
|
|
53
53
|
"zod": "^4.6.7",
|
|
54
|
-
"@oakgreencc/mycelium": "0.0.
|
|
54
|
+
"@oakgreencc/mycelium": "0.0.8"
|
|
55
55
|
},
|
|
56
56
|
"devDependencies": {
|
|
57
57
|
"@langchain/core": "^1.4.1",
|
package/src/config.mjs
CHANGED
|
@@ -37,6 +37,8 @@ import { join, resolve } from 'node:path'
|
|
|
37
37
|
* merge: MergePolicy,
|
|
38
38
|
* priorityLabels: string[],
|
|
39
39
|
* release: { enabled: boolean, time: string },
|
|
40
|
+
* reaper: { everyMinutes: number, staleMinutes: number, orphanDays: number },
|
|
41
|
+
* retriage: { enabled: boolean, time: string },
|
|
40
42
|
* paths: { home: string, db: string, heartbeat: string, logs: string },
|
|
41
43
|
* logs: { keepPerIssue: number, daemonMaxBytes: number },
|
|
42
44
|
* graft: { command: string, app: string | null, botLogin: string | null },
|
|
@@ -79,6 +81,8 @@ export function defaults(home = DEFAULT_HOME) {
|
|
|
79
81
|
merge: 'none',
|
|
80
82
|
priorityLabels: ['priority:critical', 'priority:high', 'priority:medium', 'priority:low'],
|
|
81
83
|
release: { enabled: false, time: '06:00' },
|
|
84
|
+
reaper: { everyMinutes: 60, staleMinutes: 30, orphanDays: 3 },
|
|
85
|
+
retriage: { enabled: false, time: '05:00' },
|
|
82
86
|
paths: {
|
|
83
87
|
home,
|
|
84
88
|
db: join(home, 'cambium.db'),
|
|
@@ -120,6 +124,7 @@ export function resolveConfig(raw = {}) {
|
|
|
120
124
|
const config = merge(defaults(home), raw)
|
|
121
125
|
for (const k of /** @type {const} */ (['home', 'db', 'heartbeat', 'logs'])) config.paths[k] = untilde(config.paths[k])
|
|
122
126
|
if (!/^\d{2}:\d{2}$/.test(config.release.time)) throw new Error(`config: release.time must be HH:MM`)
|
|
127
|
+
if (!/^\d{2}:\d{2}$/.test(config.retriage.time)) throw new Error(`config: retriage.time must be HH:MM`)
|
|
123
128
|
config.repos = (config.repos ?? []).map((r) => {
|
|
124
129
|
if (!r?.slug || !/^[^/\s]+\/[^/\s]+$/.test(r.slug))
|
|
125
130
|
throw new Error(`config: repo slug must be owner/name, got ${r?.slug}`)
|
package/src/github.mjs
CHANGED
|
@@ -22,6 +22,7 @@ import { BASE_FIELDS, baseHealthOf } from './base-health.mjs'
|
|
|
22
22
|
* checks: 'SUCCESS' | 'FAILURE' | 'ERROR' | 'PENDING' | 'EXPECTED' | null, failing: FailingCheck[], closes: number[],
|
|
23
23
|
* mergeable: 'MERGEABLE' | 'CONFLICTING' | 'UNKNOWN' | null,
|
|
24
24
|
* reviewDecision: 'APPROVED' | 'CHANGES_REQUESTED' | 'REVIEW_REQUIRED' | null, createdAt: string | null,
|
|
25
|
+
* updatedAt?: string | null,
|
|
25
26
|
* }} PullRequest
|
|
26
27
|
* `mergeable`: GitHub's conflict check against the base (UNKNOWN while it computes). `reviewDecision`: null
|
|
27
28
|
* when the base requires no review.
|
|
@@ -66,7 +67,7 @@ import { BASE_FIELDS, baseHealthOf } from './base-health.mjs'
|
|
|
66
67
|
* `linkIssue`: put `Closes #<issue>` at the top of the PR's body, so the PR names the issue it closes.
|
|
67
68
|
*/
|
|
68
69
|
|
|
69
|
-
const PR_FIELDS = `number url headRefName state headRefOid isInMergeQueue body mergeable reviewDecision createdAt
|
|
70
|
+
const PR_FIELDS = `number url headRefName state headRefOid isInMergeQueue body mergeable reviewDecision createdAt updatedAt
|
|
70
71
|
headRef { id }
|
|
71
72
|
autoMergeRequest { enabledAt }
|
|
72
73
|
closingIssuesReferences(first: 5) { nodes { ...I } }
|
|
@@ -82,7 +83,7 @@ const PR_FIELDS = `number url headRefName state headRefOid isInMergeQueue body m
|
|
|
82
83
|
*/
|
|
83
84
|
export const SNAPSHOT_QUERY = `query($owner: String!, $name: String!, $labels: [String!], $release: Boolean!, $from: String!, $to: String!, $health: Boolean!, $baseRef: String!, $check: String!) {
|
|
84
85
|
repository(owner: $owner, name: $name) {
|
|
85
|
-
issues(labels: $labels, states: OPEN, first: 100, orderBy: { field: CREATED_AT, direction: ASC }) { nodes { ...I } }
|
|
86
|
+
issues(labels: $labels, states: OPEN, first: 100, orderBy: { field: CREATED_AT, direction: ASC }) { pageInfo { hasNextPage endCursor } nodes { ...I } }
|
|
86
87
|
open: pullRequests(states: OPEN, first: 100, orderBy: { field: UPDATED_AT, direction: DESC }) { nodes { ${PR_FIELDS} } }
|
|
87
88
|
merged: pullRequests(states: MERGED, first: 30, orderBy: { field: UPDATED_AT, direction: DESC }) { nodes { ${PR_FIELDS} } }
|
|
88
89
|
promoteRef: pullRequests(headRefName: $from, baseRefName: $to, states: OPEN, first: 10) @include(if: $release) { nodes { ...P } }
|
|
@@ -130,6 +131,15 @@ export const issuesQuery = (numbers) =>
|
|
|
130
131
|
.join(' ')} } }
|
|
131
132
|
${SNAPSHOT_QUERY.slice(SNAPSHOT_QUERY.indexOf('fragment I on Issue'), SNAPSHOT_QUERY.indexOf('fragment P on'))}`
|
|
132
133
|
|
|
134
|
+
/** The queued issues after `$after`: the snapshot's first page holds 100, a busy repo has more. */
|
|
135
|
+
export const ISSUES_PAGE_QUERY = `query($owner: String!, $name: String!, $labels: [String!], $after: String) { repository(owner: $owner, name: $name) {
|
|
136
|
+
issues(labels: $labels, states: OPEN, first: 100, after: $after, orderBy: { field: CREATED_AT, direction: ASC }) { pageInfo { hasNextPage endCursor } nodes { ...I } }
|
|
137
|
+
} }
|
|
138
|
+
${SNAPSHOT_QUERY.slice(SNAPSHOT_QUERY.indexOf('fragment I on Issue'), SNAPSHOT_QUERY.indexOf('fragment P on'))}`
|
|
139
|
+
|
|
140
|
+
/** Pages read past the first: 10 × 100 queued issues per repo is far past any real queue. */
|
|
141
|
+
const MAX_ISSUE_PAGES = 10
|
|
142
|
+
|
|
133
143
|
const FAILED = new Set(['FAILURE', 'ERROR', 'TIMED_OUT', 'CANCELLED', 'ACTION_REQUIRED', 'STARTUP_FAILURE'])
|
|
134
144
|
|
|
135
145
|
/**
|
|
@@ -207,8 +217,22 @@ function toIssue(n, botLogin) {
|
|
|
207
217
|
}
|
|
208
218
|
}
|
|
209
219
|
|
|
210
|
-
/** The
|
|
211
|
-
export const
|
|
220
|
+
/** The issue an agent branch names: `agent/<n>-<slug>` or `agent/issue-<n>`. @param {string} branch */
|
|
221
|
+
export const branchIssue = (branch) => {
|
|
222
|
+
const m = /^agent\/(?:issue-)?(\d+)(?:-|$)/.exec(branch)
|
|
223
|
+
return m ? Number(m[1]) : null
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
/**
|
|
227
|
+
* The issues the snapshot has not read that PRs name: the ones they close, and for an open PR that closes none, the
|
|
228
|
+
* one its branch names (the reaper pairs those). @param {Snapshot} snap @returns {number[]}
|
|
229
|
+
*/
|
|
230
|
+
export const unreadCloses = (snap) => {
|
|
231
|
+
const named = snap.prs.flatMap((p) =>
|
|
232
|
+
p.closes.length || p.state !== 'OPEN' ? p.closes : [branchIssue(p.branch)].filter((n) => n !== null),
|
|
233
|
+
)
|
|
234
|
+
return [...new Set(named.filter((n) => !snap.issues.has(n)))]
|
|
235
|
+
}
|
|
212
236
|
|
|
213
237
|
/**
|
|
214
238
|
* Add the answer to {@link issuesQuery} to the snapshot; a number that is not an issue (a PR, or gone) is dropped
|
|
@@ -270,6 +294,7 @@ export function normalize(data, { botLogin = null, release = null, health = null
|
|
|
270
294
|
mergeable: p.mergeable ?? null,
|
|
271
295
|
reviewDecision: p.reviewDecision ?? null,
|
|
272
296
|
createdAt: p.createdAt ?? null,
|
|
297
|
+
updatedAt: p.updatedAt ?? null,
|
|
273
298
|
})
|
|
274
299
|
}
|
|
275
300
|
return {
|
|
@@ -341,6 +366,15 @@ export function graftGitHub({ config, exec: run = exec }) {
|
|
|
341
366
|
check: health?.check ?? '',
|
|
342
367
|
})
|
|
343
368
|
const data = await json(graft(slug, ['graphql', SNAPSHOT_QUERY, '--vars', vars]))
|
|
369
|
+
// More than 100 queued issues: follow the cursor, or the newest ones are never scheduled or watched.
|
|
370
|
+
const issues = (data?.data ?? data)?.repository?.issues
|
|
371
|
+
for (let page = 0, info = issues?.pageInfo; info?.hasNextPage && page < MAX_ISSUE_PAGES; page++) {
|
|
372
|
+
const pageVars = JSON.stringify({ owner, name, labels: QUEUE_LABELS, after: info.endCursor })
|
|
373
|
+
const next = await json(graft(slug, ['graphql', ISSUES_PAGE_QUERY, '--vars', pageVars]))
|
|
374
|
+
const more = (next?.data ?? next)?.repository?.issues
|
|
375
|
+
issues.nodes.push(...(more?.nodes ?? []))
|
|
376
|
+
info = more?.pageInfo
|
|
377
|
+
}
|
|
344
378
|
const snap = normalize(data, { botLogin: config.graft.botLogin, release, health })
|
|
345
379
|
const unread = unreadCloses(snap)
|
|
346
380
|
if (!unread.length) return snap
|
package/src/graph-engine.mjs
CHANGED
|
@@ -15,6 +15,7 @@
|
|
|
15
15
|
* Schedule admits entries), when it enters a wait (polled next tick, unless the wait is `pollOnEntry`), at END,
|
|
16
16
|
* or at a wait that is not ready.
|
|
17
17
|
*/
|
|
18
|
+
import { upkeep } from './upkeep.mjs'
|
|
18
19
|
import { advance, END } from './graph/index.mjs'
|
|
19
20
|
import { deleteMergedBranch, prGraph } from './graphs/pr.mjs'
|
|
20
21
|
import { HANDED_OFF, inputFor, lostLeaseOf, task, TASK_QUEUES, TERMINAL } from './graphs/task.mjs'
|
|
@@ -258,6 +259,7 @@ export const GRAPH_ENGINE = {
|
|
|
258
259
|
sweep: inOrder(sweep, DISPATCH_STEPS.sweep),
|
|
259
260
|
scheduleTasks,
|
|
260
261
|
scheduleDispatch: DISPATCH_STEPS.schedule,
|
|
262
|
+
upkeep,
|
|
261
263
|
}
|
|
262
264
|
|
|
263
265
|
// ---- `cambium run <graph> --issue <n> [--follow]`
|
package/src/launchd.mjs
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
// @ts-check
|
|
2
|
-
/**
|
|
2
|
+
/**
|
|
3
|
+
* The launchd agent: KeepAlive restarts the daemon whenever it exits.
|
|
4
|
+
*
|
|
5
|
+
* Workers inherit the daemon's launchd context. `ProcessType` Background throttles the whole tree
|
|
6
|
+
* (builds, test runs) to background QoS, so it is Standard. launchd sets no locale, and CocoaPods
|
|
7
|
+
* fails under the ASCII-8BIT default, so LANG and LC_ALL are set here.
|
|
8
|
+
*/
|
|
3
9
|
import { join } from 'node:path'
|
|
4
10
|
|
|
5
11
|
export const LABEL = 'com.oakgreencc.cambium'
|
|
@@ -29,7 +35,7 @@ ${args}
|
|
|
29
35
|
<key>ThrottleInterval</key>
|
|
30
36
|
<integer>30</integer>
|
|
31
37
|
<key>ProcessType</key>
|
|
32
|
-
<string>
|
|
38
|
+
<string>Standard</string>
|
|
33
39
|
<key>WorkingDirectory</key>
|
|
34
40
|
<string>${xml(home)}</string>
|
|
35
41
|
<key>EnvironmentVariables</key>
|
|
@@ -38,6 +44,10 @@ ${args}
|
|
|
38
44
|
<string>${xml(path)}</string>
|
|
39
45
|
<key>HOME</key>
|
|
40
46
|
<string>${xml(home)}</string>
|
|
47
|
+
<key>LANG</key>
|
|
48
|
+
<string>en_US.UTF-8</string>
|
|
49
|
+
<key>LC_ALL</key>
|
|
50
|
+
<string>en_US.UTF-8</string>
|
|
41
51
|
</dict>
|
|
42
52
|
<key>StandardOutPath</key>
|
|
43
53
|
<string>/dev/null</string>
|
package/src/tick.mjs
CHANGED
|
@@ -52,7 +52,7 @@ import { stopOverdue } from './workers.mjs'
|
|
|
52
52
|
|
|
53
53
|
/**
|
|
54
54
|
* @typedef {(deps: Deps, ctx: Ctx) => Promise<void>} StepFn
|
|
55
|
-
* @typedef {{ reconcile: StepFn, watch: StepFn, sweep: StepFn, scheduleTasks: StepFn, scheduleDispatch?: StepFn }} Engine
|
|
55
|
+
* @typedef {{ reconcile: StepFn, watch: StepFn, sweep: StepFn, scheduleTasks: StepFn, scheduleDispatch?: StepFn, upkeep?: StepFn }} Engine
|
|
56
56
|
* The Task-phase steps. `config.engine` picks the legacy stage tick (this file) or the `task` graph
|
|
57
57
|
* (graph-engine.mjs). The graph engine also folds the Dispatch phase into reconcile and sweep and brings its own
|
|
58
58
|
* `scheduleDispatch` (without one, the dispatch stub runs). Read and Heartbeat are shared.
|
|
@@ -95,6 +95,7 @@ export async function tick(deps) {
|
|
|
95
95
|
])),
|
|
96
96
|
...(ctx.mode === 'running' ? /** @type {[string, StepFn][]} */ ([['schedule', schedule]]) : []),
|
|
97
97
|
...(paused ? [] : /** @type {[string, any][]} */ ([['release', release]])),
|
|
98
|
+
...(paused || !engine.upkeep ? [] : /** @type {[string, StepFn][]} */ ([['upkeep', engine.upkeep]])),
|
|
98
99
|
...(paused || !deps.gc ? [] : /** @type {[string, StepFn][]} */ ([['gc', gc]])),
|
|
99
100
|
]
|
|
100
101
|
for (const [name, fn] of steps) await runStep(deps, ctx, name, () => fn(deps, ctx))
|
|
@@ -104,7 +105,7 @@ export async function tick(deps) {
|
|
|
104
105
|
}
|
|
105
106
|
|
|
106
107
|
/** @type {Engine} */
|
|
107
|
-
const LEGACY = { reconcile, watch, sweep, scheduleTasks }
|
|
108
|
+
const LEGACY = { reconcile, watch, sweep, scheduleTasks, upkeep: async () => {} }
|
|
108
109
|
|
|
109
110
|
/** @param {Deps} deps @returns {Engine} */
|
|
110
111
|
export const engineOf = (deps) => (deps.config.engine === 'graph' ? GRAPH_ENGINE : LEGACY)
|
package/src/upkeep.mjs
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/**
|
|
3
|
+
* Clock-driven upkeep for the graph engine, run after Watch:
|
|
4
|
+
*
|
|
5
|
+
* - **Reap** (every `reaper.everyMinutes`, default 60): an open `agent/*` PR left alone `reaper.staleMinutes`
|
|
6
|
+
* (default 30) that cambium is not tracking — a worker killed at its wall-clock cap whose publish still landed, a
|
|
7
|
+
* PR whose body never said `Closes #n` — is adopted. Its issue (from `Closes #n`, else the branch `agent/<n>-…`) is
|
|
8
|
+
* handed to the PR graph exactly as a worker's submit would (`submit`: marker `pr`, out of every queue), so Watch
|
|
9
|
+
* merges it when green or reworks it when red, instead of a fresh Task redoing the work. A PR whose issue is closed
|
|
10
|
+
* is closed; one with no issue at all is closed after `reaper.orphanDays` (default 3). The branch stays.
|
|
11
|
+
* - **Re-triage** (daily at `retriage.time` when `retriage.enabled`): every `q:human` issue goes back to `q:intake`,
|
|
12
|
+
* so triage looks again under its current rubric: done since (obsolete), now checkable (defer), or now a task.
|
|
13
|
+
* Not `why:approve` (a merge only a human may make: a re-triage would start a duplicate Task) and not one holding
|
|
14
|
+
* `running`.
|
|
15
|
+
*/
|
|
16
|
+
import { drive } from './graph-engine.mjs'
|
|
17
|
+
import { branchIssue } from './github.mjs'
|
|
18
|
+
import { iso, key, MIN, setLabels } from './issue-state.mjs'
|
|
19
|
+
import { move, Q, RUNNING, WHY } from './labels.mjs'
|
|
20
|
+
import { releaseDue } from './phases/release.mjs'
|
|
21
|
+
import { each } from './tick.mjs'
|
|
22
|
+
|
|
23
|
+
/**
|
|
24
|
+
* @typedef {import('./tick.mjs').Deps} Deps
|
|
25
|
+
* @typedef {import('./tick.mjs').Ctx} Ctx
|
|
26
|
+
* @typedef {import('./github.mjs').PullRequest} PullRequest
|
|
27
|
+
* @typedef {import('./github.mjs').Issue} Issue
|
|
28
|
+
* @typedef {import('./github.mjs').Snapshot} Snapshot
|
|
29
|
+
*/
|
|
30
|
+
|
|
31
|
+
const DAY = 24 * 60 * MIN
|
|
32
|
+
|
|
33
|
+
/**
|
|
34
|
+
* What the reaper does with one open agent PR. Pure: the caller acts.
|
|
35
|
+
*
|
|
36
|
+
* @param {PullRequest} pr @param {Snapshot} snap @param {{ now: number, staleMinutes: number, orphanDays: number,
|
|
37
|
+
* busy: (issue: Issue) => boolean }} o
|
|
38
|
+
* @returns {{ act: 'adopt', issue: Issue, link: boolean } | { act: 'close', reason: string } | { act: 'leave', why: string }}
|
|
39
|
+
*/
|
|
40
|
+
export function reapVerdict(pr, snap, { now, staleMinutes, orphanDays, busy }) {
|
|
41
|
+
if (pr.state !== 'OPEN') return { act: 'leave', why: 'not open' }
|
|
42
|
+
const touched = Date.parse(pr.updatedAt ?? pr.createdAt ?? '') || now
|
|
43
|
+
if (now - touched < staleMinutes * MIN) return { act: 'leave', why: 'recently updated' }
|
|
44
|
+
const linked = pr.closes.find((n) => snap.issues.has(n))
|
|
45
|
+
const n = linked ?? branchIssue(pr.branch)
|
|
46
|
+
const issue = n === null ? undefined : snap.issues.get(n)
|
|
47
|
+
if (!issue) {
|
|
48
|
+
if (now - touched < orphanDays * DAY) return { act: 'leave', why: 'no issue yet; within orphanDays' }
|
|
49
|
+
return {
|
|
50
|
+
act: 'close',
|
|
51
|
+
reason: `Abandoned: no issue names this PR${n ? ` (#${n} is not readable)` : ''} and nothing has touched it for ${orphanDays} days. The branch stays, so it can be reopened.`,
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
if (issue.state !== 'OPEN')
|
|
55
|
+
return { act: 'close', reason: `Issue #${issue.number} is closed, so this PR has nothing to do.` }
|
|
56
|
+
if (busy(issue)) return { act: 'leave', why: `#${issue.number} has a worker` }
|
|
57
|
+
const m = issue.stateComment?.marker
|
|
58
|
+
// Tracked: the PR graph (or a rework queued for this very PR) already owns it.
|
|
59
|
+
if (m?.pr === pr.number) return { act: 'leave', why: `#${issue.number} already tracks it` }
|
|
60
|
+
// A newer agent PR for the same issue is the live one; the PR graph closes this one once it is tracked.
|
|
61
|
+
return { act: 'adopt', issue, link: linked === undefined }
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/** Both clocks, in order: adopt first, so an issue whose PR the reaper takes is not re-triaged. @param {Deps} deps @param {Ctx} ctx */
|
|
65
|
+
export async function upkeep(deps, ctx) {
|
|
66
|
+
await reap(deps, ctx)
|
|
67
|
+
await retriage(deps, ctx)
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
/** @param {Deps} deps @param {Ctx} ctx */
|
|
71
|
+
export async function reap(deps, ctx) {
|
|
72
|
+
const { everyMinutes, staleMinutes, orphanDays } = deps.config.reaper
|
|
73
|
+
const last = Number(deps.store.get('reaper.lastRun') ?? 0)
|
|
74
|
+
if (!everyMinutes || ctx.now - last < everyMinutes * MIN) return
|
|
75
|
+
deps.store.set('reaper.lastRun', String(ctx.now))
|
|
76
|
+
for (const [slug, snap] of ctx.snaps) {
|
|
77
|
+
const busy = (/** @type {Issue} */ i) => i.labels.includes(RUNNING) || Boolean(deps.store.worker(slug, i.number))
|
|
78
|
+
await each(
|
|
79
|
+
deps,
|
|
80
|
+
snap.prs.filter((p) => p.state === 'OPEN'),
|
|
81
|
+
async (pr) => {
|
|
82
|
+
const v = reapVerdict(pr, snap, { now: ctx.now, staleMinutes, orphanDays, busy })
|
|
83
|
+
if (v.act === 'leave') return
|
|
84
|
+
if (v.act === 'close') {
|
|
85
|
+
await deps.github.closePull(slug, pr.number, `${v.reason}\n\nClosed by cambium's reaper.`)
|
|
86
|
+
pr.state = 'CLOSED'
|
|
87
|
+
deps.log('info', `reaper: closed PR #${pr.number}: ${v.reason}`, { repo: slug })
|
|
88
|
+
return
|
|
89
|
+
}
|
|
90
|
+
const { issue } = v
|
|
91
|
+
if (v.link) await deps.github.linkIssue(slug, pr.number, issue.number)
|
|
92
|
+
if (!pr.closes.includes(issue.number)) pr.closes.push(issue.number)
|
|
93
|
+
const from = issue.labels.find((l) => l.startsWith('q:')) ?? 'no queue'
|
|
94
|
+
// The task graph's `submit`: marker `pr`, out of every queue; the PR graph takes it from the next tick.
|
|
95
|
+
await drive(
|
|
96
|
+
deps,
|
|
97
|
+
ctx,
|
|
98
|
+
{ slug, number: issue.number, snap, issue },
|
|
99
|
+
{ node: 'submit', input: { pr: pr.number, branch: pr.branch } },
|
|
100
|
+
)
|
|
101
|
+
await deps.github.comment(
|
|
102
|
+
slug,
|
|
103
|
+
issue.number,
|
|
104
|
+
`**Reaper:** adopted PR #${pr.number}, which nothing was tracking (was ${from}). cambium merges it when green, or reworks it.`,
|
|
105
|
+
)
|
|
106
|
+
deps.log('info', `reaper: adopted PR #${pr.number} (was ${from})`, { repo: slug, issue: issue.number })
|
|
107
|
+
},
|
|
108
|
+
(pr) => ({ repo: slug, issue: pr.number }),
|
|
109
|
+
)
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
/** @param {Deps} deps @param {Ctx} ctx */
|
|
114
|
+
export async function retriage(deps, ctx) {
|
|
115
|
+
const { enabled, time } = deps.config.retriage
|
|
116
|
+
if (!enabled) return
|
|
117
|
+
const last = deps.store.get('retriage.lastRun')
|
|
118
|
+
if (!releaseDue(ctx.now, last === null ? null : Number(last), time)) return
|
|
119
|
+
deps.store.set('retriage.lastRun', String(ctx.now))
|
|
120
|
+
for (const [slug, snap] of ctx.snaps) {
|
|
121
|
+
const human = [...snap.issues.values()].filter(
|
|
122
|
+
(i) =>
|
|
123
|
+
i.state === 'OPEN' &&
|
|
124
|
+
i.labels.includes(Q.human) &&
|
|
125
|
+
!i.labels.includes(WHY.approve) &&
|
|
126
|
+
!i.labels.includes(RUNNING) &&
|
|
127
|
+
!ctx.touched.has(key(slug, i.number)),
|
|
128
|
+
)
|
|
129
|
+
let moved = 0
|
|
130
|
+
await each(
|
|
131
|
+
deps,
|
|
132
|
+
human,
|
|
133
|
+
async (issue) => {
|
|
134
|
+
await setLabels(deps, slug, issue, move(issue.labels, Q.intake))
|
|
135
|
+
moved++
|
|
136
|
+
},
|
|
137
|
+
(issue) => ({ repo: slug, issue: issue.number }),
|
|
138
|
+
)
|
|
139
|
+
if (human.length)
|
|
140
|
+
deps.log('info', `re-triage: ${moved} of ${human.length} q:human issues back to q:intake (${iso(ctx.now)})`, {
|
|
141
|
+
repo: slug,
|
|
142
|
+
})
|
|
143
|
+
}
|
|
144
|
+
}
|