@oakgreencc/cambium 0.0.23 → 0.0.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -0
- package/package.json +2 -2
- package/src/base-health.mjs +208 -0
- package/src/config.mjs +10 -2
- package/src/github.mjs +41 -6
- package/src/graph-engine.mjs +2 -1
- package/src/graphs/task.mjs +20 -6
- package/src/tick.mjs +27 -9
package/README.md
CHANGED
|
@@ -127,6 +127,7 @@ One JSON file. Everything but `repos` is optional; these are the defaults:
|
|
|
127
127
|
- `repos[].path` is a local clone; worktrees are added from it and `graft` runs in it.
|
|
128
128
|
- `graft.app` is the App cambium acts as (`GRAFT_APP` for its own `graft` calls); `worker.graftApp` the one workers act as. Neither leaks into the other.
|
|
129
129
|
- `merge` (top level, or per repo in `repos[]`): who merges a Task PR. See [Merging](#merging).
|
|
130
|
+
- `repos[].requiredCheck` (default `"ci"`): the check whose runs on `baseBranch` say whether the base is green; `null` turns the [red-base](#red-base) gate and repair off for that repo.
|
|
130
131
|
- `engine` picks how the Task phase runs: `"legacy"` (the stage tick, the default) or `"graph"` (the [`task` graph](#the-task-graph)). Both make the same moves; see [The task graph](#the-task-graph). Only `"graph"` runs the [`dispatch` graph](#the-dispatch-graph) (gated by `phases.dispatch.cap`); legacy leaves `q:intake` alone.
|
|
131
132
|
- Setting `paths.home` moves every default path under it.
|
|
132
133
|
- `release.time` is host-local `HH:MM`.
|
|
@@ -323,6 +324,14 @@ Neither of our repos runs a GitHub merge queue (it costs), so `merge` says who m
|
|
|
323
324
|
|
|
324
325
|
The worker prompt says which: "Green: stop; cambium merges the PR once CI is green" or "…the PR is merged outside this session".
|
|
325
326
|
|
|
327
|
+
## Red base
|
|
328
|
+
|
|
329
|
+
Every Task PR lands on `baseBranch`, so a red base fails every worker for a reason that is not its own. Without a gate each one spends its attempts and lands in `q:human` (sektorapp/monorepo#3972: three attempts killed at the cap while #4624 + #4625, green apart, had broken `develop` together). `src/base-health.mjs`, graph engine:
|
|
330
|
+
|
|
331
|
+
- **Observe.** The tick's snapshot reads `baseBranch`'s last 20 commits with their `requiredCheck` run. The newest commit with a verdict decides: `green`, `red` (with the last green commit and the merges since it, the suspects), or `unknown` (no completed run in the window; a pending or cancelled run says nothing, so a red base stays red until a run passes). The heartbeat carries it per repo (`repos[].base`). The base needs its check to run on push: a repo whose CI runs only on pull requests is always `unknown`.
|
|
332
|
+
- **Gate.** While red, Schedule starts no Task worker on that repo except the repair, which takes no slot. A failure during a red base requeues at the same attempt (history `base-red`, "not charged") and never goes to `q:human`; it waits in `q:task` for green. `unknown` gates nothing.
|
|
333
|
+
- **Resolve.** Red with no open repair issue → cambium opens one: labelled `red-base`, `q:task` and the first `priorityLabels` entry, its body the worker's spec (the failing run, the suspects, fix forward or revert the culprit, `pnpm verify`). It runs as an ordinary Task issue and is charged like one, so a repair that spends its attempts goes to `q:human`. Its PR merges through graft's red-base rescue (up to date with the base, its own check green). Green again → a repair issue with no open PR and no worker is closed with a comment.
|
|
334
|
+
|
|
326
335
|
## The dispatch graph
|
|
327
336
|
|
|
328
337
|
With `"engine": "graph"` and `phases.dispatch.cap > 0`, each `q:intake` issue is triaged by a one-shot `claude -p` worker (`src/graphs/dispatch.mjs`, steps in `src/dispatch-engine.mjs`, prompt and result in `src/phases/dispatch.mjs`), then moved to `q:task`, moved to `q:human`, or closed as a duplicate.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@oakgreencc/cambium",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.25",
|
|
4
4
|
"description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
|
|
5
5
|
"license": "AGPL-3.0-only",
|
|
6
6
|
"type": "module",
|
|
@@ -51,7 +51,7 @@
|
|
|
51
51
|
"@opentelemetry/sdk-metrics": "^3.1.1",
|
|
52
52
|
"@opentelemetry/sdk-trace": "^3.1.1",
|
|
53
53
|
"zod": "^4.6.7",
|
|
54
|
-
"@oakgreencc/mycelium": "0.0.
|
|
54
|
+
"@oakgreencc/mycelium": "0.0.7"
|
|
55
55
|
},
|
|
56
56
|
"devDependencies": {
|
|
57
57
|
"@langchain/core": "^1.4.1",
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/**
|
|
3
|
+
* The base branch's health, and the repair of a red one.
|
|
4
|
+
*
|
|
5
|
+
* Every Task PR lands on the repo's `baseBranch`. When that branch is red, every worker's verify fails for a
|
|
6
|
+
* reason that is not its own, and without this module each one spends its attempts and lands in `q:human`
|
|
7
|
+
* (sektorapp/monorepo#3972: three attempts, three kills, on a break #4624 + #4625 made together). So:
|
|
8
|
+
*
|
|
9
|
+
* - **Observe.** The snapshot reads the base's last {@link HISTORY} commits with their `requiredCheck` run
|
|
10
|
+
* ({@link BASE_FIELDS}); {@link baseHealthOf} turns them into green / red / unknown, the last green commit and
|
|
11
|
+
* the merges since it.
|
|
12
|
+
* - **Gate.** While red, Schedule starts no Task worker for that repo except the repair (tick.mjs
|
|
13
|
+
* `taskCandidates`), and a failure is requeued without spending an attempt (graphs/task.mjs `requeue`).
|
|
14
|
+
* - **Resolve.** {@link trackBaseHealth} opens one repair issue per red base ({@link REPAIR_LABEL}, `q:task`,
|
|
15
|
+
* the top priority): its body is the worker's spec, and its PR merges through graft's red-base rescue. A base
|
|
16
|
+
* that goes green again closes a repair issue nobody is working.
|
|
17
|
+
*
|
|
18
|
+
* Unknown (no completed run in the window: the base has no `requiredCheck` on push, or every run is cancelled or
|
|
19
|
+
* pending) gates nothing.
|
|
20
|
+
*/
|
|
21
|
+
import { iso, key } from './issue-state.mjs'
|
|
22
|
+
import { Q } from './labels.mjs'
|
|
23
|
+
|
|
24
|
+
/** Marks the repair issue. Not a `why:` label: those leave with `q:human` (labels.mjs `move`). */
|
|
25
|
+
export const REPAIR_LABEL = 'red-base'
|
|
26
|
+
|
|
27
|
+
/** How many base commits the snapshot reads, newest first. */
|
|
28
|
+
export const HISTORY = 20
|
|
29
|
+
|
|
30
|
+
/**
|
|
31
|
+
* The snapshot's base read, under `repository`. Needs `$baseRef`, `$check` and `$health`.
|
|
32
|
+
*/
|
|
33
|
+
export const BASE_FIELDS = `baseRef: ref(qualifiedName: $baseRef) @include(if: $health) {
|
|
34
|
+
target { ... on Commit { history(first: ${HISTORY}) { nodes {
|
|
35
|
+
oid messageHeadline
|
|
36
|
+
associatedPullRequests(first: 1) { nodes { number } }
|
|
37
|
+
checkSuites(first: 30) { nodes { branch { name } checkRuns(first: 3, filterBy: { checkName: $check }) { nodes { status conclusion detailsUrl } } } }
|
|
38
|
+
} } } }
|
|
39
|
+
}
|
|
40
|
+
repairs: issues(labels: ["${REPAIR_LABEL}"], states: OPEN, first: 5) @include(if: $health) { nodes { number } }`
|
|
41
|
+
|
|
42
|
+
/**
|
|
43
|
+
* @typedef {{ sha: string, headline: string, pr: number | null }} BaseCommit
|
|
44
|
+
* @typedef {{
|
|
45
|
+
* branch: string, check: string, state: 'green' | 'red' | 'unknown', head: string | null,
|
|
46
|
+
* red: (BaseCommit & { url: string | null }) | null, lastGreen: BaseCommit | null, suspects: BaseCommit[],
|
|
47
|
+
* repairs: number[],
|
|
48
|
+
* }} BaseHealth
|
|
49
|
+
* `red`: the newest commit whose run failed (the verdict); `lastGreen`: the newest passing commit older than
|
|
50
|
+
* it; `suspects`: the commits between them, newest first, `red` included, one per PR. `repairs`: open repair
|
|
51
|
+
* issues.
|
|
52
|
+
*/
|
|
53
|
+
|
|
54
|
+
const PASSED = new Set(['SUCCESS', 'NEUTRAL', 'SKIPPED'])
|
|
55
|
+
const FAILED = new Set(['FAILURE', 'TIMED_OUT', 'STARTUP_FAILURE'])
|
|
56
|
+
|
|
57
|
+
/**
|
|
58
|
+
* A commit's verdict from its `check` runs on `branch`: the first completed run that says anything. Only a suite
|
|
59
|
+
* run for the base branch counts: the history also holds the merged PRs' own commits, whose runs judged the PR
|
|
60
|
+
* against an older base (PR CI on sektorapp is affected-only), not the base.
|
|
61
|
+
* @param {any} node @param {string} branch
|
|
62
|
+
*/
|
|
63
|
+
function verdictOf(node, branch) {
|
|
64
|
+
for (const suite of node?.checkSuites?.nodes ?? []) {
|
|
65
|
+
if (suite?.branch?.name !== branch) continue
|
|
66
|
+
for (const run of suite?.checkRuns?.nodes ?? []) {
|
|
67
|
+
if (run?.status !== 'COMPLETED') continue
|
|
68
|
+
if (PASSED.has(run.conclusion)) return { pass: true, url: run.detailsUrl ?? null }
|
|
69
|
+
if (FAILED.has(run.conclusion)) return { pass: false, url: run.detailsUrl ?? null }
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
return null
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
/** @param {any} n @returns {BaseCommit} */
|
|
76
|
+
const commitOf = (n) => ({
|
|
77
|
+
sha: n.oid,
|
|
78
|
+
headline: n.messageHeadline ?? '',
|
|
79
|
+
pr: n.associatedPullRequests?.nodes?.[0]?.number ?? null,
|
|
80
|
+
})
|
|
81
|
+
|
|
82
|
+
/**
|
|
83
|
+
* The base's health from its history, newest first. The newest commit with a verdict decides: a pending or
|
|
84
|
+
* cancelled run on the head says nothing, so a red base stays red until a run passes.
|
|
85
|
+
*
|
|
86
|
+
* @param {any} repo the snapshot's `repository` @param {{ branch: string, check: string }} at
|
|
87
|
+
* @returns {BaseHealth}
|
|
88
|
+
*/
|
|
89
|
+
export function baseHealthOf(repo, { branch, check }) {
|
|
90
|
+
const nodes = repo?.baseRef?.target?.history?.nodes ?? []
|
|
91
|
+
const repairs = (repo?.repairs?.nodes ?? []).map((/** @type {any} */ n) => n.number)
|
|
92
|
+
const base = { branch, check, head: nodes[0]?.oid ?? null, repairs }
|
|
93
|
+
const verdict = (/** @type {any} */ n) => verdictOf(n, branch)
|
|
94
|
+
const decided = nodes.findIndex(verdict)
|
|
95
|
+
if (decided === -1) return { ...base, state: 'unknown', red: null, lastGreen: null, suspects: [] }
|
|
96
|
+
const v = /** @type {{ pass: boolean, url: string | null }} */ (verdict(nodes[decided]))
|
|
97
|
+
if (v.pass) return { ...base, state: 'green', red: null, lastGreen: commitOf(nodes[decided]), suspects: [] }
|
|
98
|
+
const greenAt = nodes.findIndex((/** @type {any} */ n, /** @type {number} */ i) => i > decided && verdict(n)?.pass)
|
|
99
|
+
/** @type {Set<number>} */
|
|
100
|
+
const seen = new Set()
|
|
101
|
+
const suspects = nodes
|
|
102
|
+
.slice(decided, greenAt === -1 ? undefined : greenAt)
|
|
103
|
+
.map(commitOf)
|
|
104
|
+
.filter((c) => c.pr === null || (!seen.has(c.pr) && seen.add(c.pr)))
|
|
105
|
+
return {
|
|
106
|
+
...base,
|
|
107
|
+
state: 'red',
|
|
108
|
+
red: { ...commitOf(nodes[decided]), url: v.url },
|
|
109
|
+
lastGreen: greenAt === -1 ? null : commitOf(nodes[greenAt]),
|
|
110
|
+
suspects,
|
|
111
|
+
}
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/** Red, per the snapshot. @param {import('./github.mjs').Snapshot | undefined} snap */
|
|
115
|
+
export const baseRed = (snap) => snap?.base?.state === 'red'
|
|
116
|
+
|
|
117
|
+
/** @param {import('./github.mjs').Issue} issue */
|
|
118
|
+
export const isRepair = (issue) => issue.labels.includes(REPAIR_LABEL)
|
|
119
|
+
|
|
120
|
+
const short = (/** @type {string} */ sha) => sha.slice(0, 7)
|
|
121
|
+
|
|
122
|
+
/** @param {BaseHealth} h */
|
|
123
|
+
export const repairTitle = (h) => `Red base: ${h.branch} \`${h.check}\` failing since ${short(h.red?.sha ?? '')}`
|
|
124
|
+
|
|
125
|
+
/**
|
|
126
|
+
* The repair issue's body: the worker's spec.
|
|
127
|
+
* @param {BaseHealth} h
|
|
128
|
+
*/
|
|
129
|
+
export function repairBody(h) {
|
|
130
|
+
const red = /** @type {NonNullable<BaseHealth['red']>} */ (h.red)
|
|
131
|
+
const line = (/** @type {BaseCommit} */ c) => `- ${short(c.sha)}${c.pr ? ` #${c.pr}` : ''} ${c.headline}`
|
|
132
|
+
return [
|
|
133
|
+
`\`${h.branch}\` is red: its \`${h.check}\` check failed on ${short(red.sha)}${red.url ? ` (${red.url})` : ''}.`,
|
|
134
|
+
'',
|
|
135
|
+
'Opened by cambium. While this is open cambium starts no other Task worker on this repo, and their failures cost no attempts.',
|
|
136
|
+
'',
|
|
137
|
+
'## Suspects',
|
|
138
|
+
'',
|
|
139
|
+
h.lastGreen
|
|
140
|
+
? `Last green: ${short(h.lastGreen.sha)} ${h.lastGreen.headline}. Merged since, newest first:`
|
|
141
|
+
: `No green commit in the last ${HISTORY}. Newest first:`,
|
|
142
|
+
'',
|
|
143
|
+
...h.suspects.map(line),
|
|
144
|
+
'',
|
|
145
|
+
'## Work',
|
|
146
|
+
'',
|
|
147
|
+
`1. Read why it failed: \`graft jobs ${red.url ?? '<run>'}\`. Reproduce with \`pnpm verify\` in your worktree, which starts at \`${h.branch}\`.`,
|
|
148
|
+
'2. Two PRs that are each green can be red together, so the cause may span suspects.',
|
|
149
|
+
'3. Fix forward when the fix is clear and small. Otherwise revert the merge that broke it, and say on the PR which one and why.',
|
|
150
|
+
'4. Change only what makes the check green; do not fold other work into this PR.',
|
|
151
|
+
'',
|
|
152
|
+
`Your PR closes this issue. cambium merges it through graft's red-base rescue, once it is up to date with \`${h.branch}\` and its own \`${h.check}\` is green.`,
|
|
153
|
+
'',
|
|
154
|
+
'## Done when',
|
|
155
|
+
'',
|
|
156
|
+
`- \`pnpm verify\` passes on \`${h.branch}\` with your change.`,
|
|
157
|
+
`- \`${h.branch}\`'s \`${h.check}\` is green after the merge.`,
|
|
158
|
+
].join('\n')
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/**
|
|
162
|
+
* The repair issue's lifecycle, once per tick before Schedule. Red with no open repair issue → open one
|
|
163
|
+
* (`q:task`, the top priority label, {@link REPAIR_LABEL}). Green, with an open repair issue that has no open PR
|
|
164
|
+
* and no worker → close it. Unknown → nothing.
|
|
165
|
+
*
|
|
166
|
+
* @param {import('./tick.mjs').Deps} deps @param {import('./tick.mjs').Ctx} ctx
|
|
167
|
+
*/
|
|
168
|
+
export async function trackBaseHealth(deps, ctx) {
|
|
169
|
+
for (const [slug, snap] of ctx.snaps) {
|
|
170
|
+
const h = snap.base
|
|
171
|
+
if (!h) continue
|
|
172
|
+
try {
|
|
173
|
+
if (h.state === 'red' && !h.repairs.length) {
|
|
174
|
+
const made = await deps.github.createIssue(slug, {
|
|
175
|
+
title: repairTitle(h),
|
|
176
|
+
body: repairBody(h),
|
|
177
|
+
labels: [Q.task, deps.config.priorityLabels[0], REPAIR_LABEL].filter(Boolean),
|
|
178
|
+
})
|
|
179
|
+
h.repairs = [made.number]
|
|
180
|
+
deps.log('warn', `${h.branch} is red at ${short(h.red?.sha ?? '')}; opened repair #${made.number}`, {
|
|
181
|
+
repo: slug,
|
|
182
|
+
issue: made.number,
|
|
183
|
+
})
|
|
184
|
+
continue
|
|
185
|
+
}
|
|
186
|
+
if (h.state !== 'green') continue
|
|
187
|
+
/** @type {number[]} */
|
|
188
|
+
const closed = []
|
|
189
|
+
for (const n of h.repairs) {
|
|
190
|
+
const worked = deps.store.worker(slug, n) || snap.prs.some((p) => p.state === 'OPEN' && p.closes.includes(n))
|
|
191
|
+
if (worked || ctx.touched.has(key(slug, n))) continue
|
|
192
|
+
await deps.github.close(
|
|
193
|
+
slug,
|
|
194
|
+
n,
|
|
195
|
+
`\`${h.branch}\` is green again at ${short(h.lastGreen?.sha ?? h.head ?? '')} (${iso(ctx.now)}); nothing left to repair.`,
|
|
196
|
+
{ reason: 'completed' },
|
|
197
|
+
)
|
|
198
|
+
const issue = snap.issues.get(n)
|
|
199
|
+
if (issue) issue.state = 'CLOSED'
|
|
200
|
+
closed.push(n)
|
|
201
|
+
deps.log('info', `${h.branch} is green again; closed repair #${n}`, { repo: slug, issue: n })
|
|
202
|
+
}
|
|
203
|
+
h.repairs = h.repairs.filter((n) => !closed.includes(n))
|
|
204
|
+
} catch (err) {
|
|
205
|
+
deps.log('error', `base health: ${/** @type {Error} */ (err)?.message ?? err}`, { repo: slug })
|
|
206
|
+
}
|
|
207
|
+
}
|
|
208
|
+
}
|
package/src/config.mjs
CHANGED
|
@@ -16,7 +16,12 @@ import { join, resolve } from 'node:path'
|
|
|
16
16
|
* Who merges a Task PR. `none`: not cambium (a merge queue, auto-merge, or a human; past `mergeQueueMinutes`
|
|
17
17
|
* the issue goes to `q:human`). `green`: cambium, once its checks are green, through `graft pr merge --auto`
|
|
18
18
|
* (so graft's merge policy still decides). `green` needs `engine: "graph"`.
|
|
19
|
-
* @typedef {{
|
|
19
|
+
* @typedef {{
|
|
20
|
+
* slug: string, path: string, baseBranch: string, release: RepoRelease, merge: MergePolicy,
|
|
21
|
+
* requiredCheck: string | null,
|
|
22
|
+
* }} Repo
|
|
23
|
+
* `requiredCheck` (default `ci`): the check whose runs on `baseBranch` say whether the base is green (see
|
|
24
|
+
* base-health.mjs); `null` turns the red-base gate and repair off for the repo.
|
|
20
25
|
* @typedef {{ cap: number, wallClockMinutes: number, maxTurns: number, maxAttempts: number }} PhaseCaps
|
|
21
26
|
* @typedef {{
|
|
22
27
|
* host: string,
|
|
@@ -121,7 +126,10 @@ export function resolveConfig(raw = {}) {
|
|
|
121
126
|
const merge = r.merge ?? config.merge
|
|
122
127
|
if (!MERGE_POLICIES.includes(merge))
|
|
123
128
|
throw new Error(`config: repo ${r.slug} merge must be "none" or "green", got ${JSON.stringify(merge)}`)
|
|
124
|
-
|
|
129
|
+
const requiredCheck = r.requiredCheck === undefined ? 'ci' : r.requiredCheck
|
|
130
|
+
if (requiredCheck !== null && (typeof requiredCheck !== 'string' || !requiredCheck))
|
|
131
|
+
throw new Error(`config: repo ${r.slug} requiredCheck must be a check name or null`)
|
|
132
|
+
return { slug: r.slug, path: untilde(r.path), baseBranch, release, merge, requiredCheck }
|
|
125
133
|
})
|
|
126
134
|
if (!MERGE_POLICIES.includes(config.merge))
|
|
127
135
|
throw new Error(`config: merge must be "none" or "green", got ${JSON.stringify(config.merge)}`)
|
package/src/github.mjs
CHANGED
|
@@ -8,6 +8,7 @@
|
|
|
8
8
|
import { spawn } from 'node:child_process'
|
|
9
9
|
import { parseMarker } from './marker.mjs'
|
|
10
10
|
import { Q, QUEUE_LABELS } from './labels.mjs'
|
|
11
|
+
import { BASE_FIELDS, baseHealthOf } from './base-health.mjs'
|
|
11
12
|
|
|
12
13
|
/**
|
|
13
14
|
* @typedef {{
|
|
@@ -35,13 +36,18 @@ import { Q, QUEUE_LABELS } from './labels.mjs'
|
|
|
35
36
|
* The repo's promotion state (when its `release` is on): how far `from` is ahead of `to` (null: `to` does not
|
|
36
37
|
* exist), the PRs merged into `from` since `to` (oldest first, from the last 100 commits ahead), and the
|
|
37
38
|
* promotion PRs: open ones from `from` to `to` plus any (open, merged or closed) still labelled `q:promote`.
|
|
38
|
-
* @typedef {{
|
|
39
|
+
* @typedef {{
|
|
40
|
+
* issues: Map<number, Issue>, prs: PullRequest[], release?: Release | null,
|
|
41
|
+
* base?: import('./base-health.mjs').BaseHealth,
|
|
42
|
+
* }} Snapshot
|
|
43
|
+
* `base`: the base branch's health, when the repo has a `requiredCheck`.
|
|
39
44
|
* @typedef {{
|
|
40
45
|
* read(slug: string): Promise<Snapshot>,
|
|
41
46
|
* label(slug: string, number: number, change: { add: string[], remove: string[] }): Promise<void>,
|
|
42
47
|
* writeState(slug: string, number: number, commentId: number | null, body: string): Promise<number>,
|
|
43
48
|
* close(slug: string, number: number, body: string, opts?: { reason?: 'completed' | 'not_planned' | 'duplicate' }): Promise<void>,
|
|
44
49
|
* comment(slug: string, number: number, body: string): Promise<void>,
|
|
50
|
+
* createIssue(slug: string, issue: { title: string, body: string, labels: string[] }): Promise<{ number: number, url: string }>,
|
|
45
51
|
* deleteBranch(slug: string, branch: string): Promise<void>,
|
|
46
52
|
* openPull(slug: string, pr: { head: string, base: string, title: string, body: string }): Promise<{ number: number, url: string }>,
|
|
47
53
|
* editPull(slug: string, number: number, change: { title: string, body: string }): Promise<void>,
|
|
@@ -70,7 +76,7 @@ const PR_FIELDS = `number url headRefName state headRefOid isInMergeQueue body
|
|
|
70
76
|
* One query per repo per tick: the queued issues and the agent PRs with their checks; with `$release`, also the
|
|
71
77
|
* promotion PRs and how far `$from` is ahead of `$to` (see {@link Release}).
|
|
72
78
|
*/
|
|
73
|
-
export const SNAPSHOT_QUERY = `query($owner: String!, $name: String!, $labels: [String!], $release: Boolean!, $from: String!, $to: String!) {
|
|
79
|
+
export const SNAPSHOT_QUERY = `query($owner: String!, $name: String!, $labels: [String!], $release: Boolean!, $from: String!, $to: String!, $health: Boolean!, $baseRef: String!, $check: String!) {
|
|
74
80
|
repository(owner: $owner, name: $name) {
|
|
75
81
|
issues(labels: $labels, states: OPEN, first: 100, orderBy: { field: CREATED_AT, direction: ASC }) { nodes { ...I } }
|
|
76
82
|
open: pullRequests(states: OPEN, first: 100, orderBy: { field: UPDATED_AT, direction: DESC }) { nodes { ${PR_FIELDS} } }
|
|
@@ -84,6 +90,7 @@ export const SNAPSHOT_QUERY = `query($owner: String!, $name: String!, $labels: [
|
|
|
84
90
|
commits(last: 100) { nodes { associatedPullRequests(first: 3) { nodes { number title baseRefName state } } } }
|
|
85
91
|
}
|
|
86
92
|
}
|
|
93
|
+
${BASE_FIELDS}
|
|
87
94
|
}
|
|
88
95
|
}
|
|
89
96
|
fragment I on Issue {
|
|
@@ -222,7 +229,7 @@ export function addIssues(snap, data, asked, botLogin = null) {
|
|
|
222
229
|
* `botLogin`: when set, only that login's markers count. `release`: the branches the query asked about.
|
|
223
230
|
* @returns {Snapshot}
|
|
224
231
|
*/
|
|
225
|
-
export function normalize(data, { botLogin = null, release = null } = {}) {
|
|
232
|
+
export function normalize(data, { botLogin = null, release = null, health = null } = {}) {
|
|
226
233
|
const repo = (data?.data ?? data)?.repository
|
|
227
234
|
if (!repo) throw new Error('snapshot: no repository in the answer')
|
|
228
235
|
/** @type {Map<number, Issue>} */
|
|
@@ -258,7 +265,12 @@ export function normalize(data, { botLogin = null, release = null } = {}) {
|
|
|
258
265
|
closes: [...new Set([...(p.closingIssuesReferences?.nodes ?? []).map(addIssue), ...bodyCloses(p.body)])],
|
|
259
266
|
})
|
|
260
267
|
}
|
|
261
|
-
return {
|
|
268
|
+
return {
|
|
269
|
+
issues,
|
|
270
|
+
prs,
|
|
271
|
+
...(release ? { release: normalizeRelease(repo, release, botLogin) } : {}),
|
|
272
|
+
...(health ? { base: baseHealthOf(repo, health) } : {}),
|
|
273
|
+
}
|
|
262
274
|
}
|
|
263
275
|
|
|
264
276
|
/**
|
|
@@ -309,9 +321,20 @@ export function graftGitHub({ config, exec: run = exec }) {
|
|
|
309
321
|
// (promotion.mjs); the release graphs check `enabled` themselves.
|
|
310
322
|
const release = r && r.from !== r.to ? { from: r.from, to: r.to } : null
|
|
311
323
|
const branches = release ?? { from: '', to: '' }
|
|
312
|
-
const
|
|
324
|
+
const repo = config.repos.find((x) => x.slug === slug)
|
|
325
|
+
const health = repo?.requiredCheck ? { branch: repo.baseBranch, check: repo.requiredCheck } : null
|
|
326
|
+
const vars = JSON.stringify({
|
|
327
|
+
owner,
|
|
328
|
+
name,
|
|
329
|
+
labels: QUEUE_LABELS,
|
|
330
|
+
release: Boolean(release),
|
|
331
|
+
...branches,
|
|
332
|
+
health: Boolean(health),
|
|
333
|
+
baseRef: health ? `refs/heads/${health.branch}` : '',
|
|
334
|
+
check: health?.check ?? '',
|
|
335
|
+
})
|
|
313
336
|
const data = await json(graft(slug, ['graphql', SNAPSHOT_QUERY, '--vars', vars]))
|
|
314
|
-
const snap = normalize(data, { botLogin: config.graft.botLogin, release })
|
|
337
|
+
const snap = normalize(data, { botLogin: config.graft.botLogin, release, health })
|
|
315
338
|
const unread = unreadCloses(snap)
|
|
316
339
|
if (!unread.length) return snap
|
|
317
340
|
// Issues named only in PR bodies (in no queue): one more query, for those.
|
|
@@ -353,6 +376,18 @@ export function graftGitHub({ config, exec: run = exec }) {
|
|
|
353
376
|
async comment(slug, number, body) {
|
|
354
377
|
await graft(slug, ['issue', 'comment', String(number), '--body-file', '-'], body)
|
|
355
378
|
},
|
|
379
|
+
async createIssue(slug, { title, body, labels }) {
|
|
380
|
+
const args = [
|
|
381
|
+
'issue',
|
|
382
|
+
'create',
|
|
383
|
+
'--title',
|
|
384
|
+
title,
|
|
385
|
+
'--body-file',
|
|
386
|
+
'-',
|
|
387
|
+
...(labels.length ? ['--labels', labels.join(',')] : []),
|
|
388
|
+
]
|
|
389
|
+
return json(graft(slug, args, body))
|
|
390
|
+
},
|
|
356
391
|
async deleteBranch(slug, branch) {
|
|
357
392
|
await graft(slug, ['api', 'DELETE', `/repos/${slug}/git/refs/heads/${branch}`])
|
|
358
393
|
},
|
package/src/graph-engine.mjs
CHANGED
|
@@ -25,6 +25,7 @@ import { move, Q } from './labels.mjs'
|
|
|
25
25
|
import { markerTrace, turnTracer } from './tracing.mjs'
|
|
26
26
|
import { DISPATCH_STEPS } from './dispatch-engine.mjs'
|
|
27
27
|
import { trackPromotion } from './promotion.mjs'
|
|
28
|
+
import { trackBaseHealth } from './base-health.mjs'
|
|
28
29
|
|
|
29
30
|
/**
|
|
30
31
|
* @typedef {import('./tick.mjs').Deps} Deps
|
|
@@ -237,7 +238,7 @@ const inOrder =
|
|
|
237
238
|
/** @type {import('./tick.mjs').Engine} */
|
|
238
239
|
export const GRAPH_ENGINE = {
|
|
239
240
|
reconcile: inOrder(reconcile, DISPATCH_STEPS.reconcile),
|
|
240
|
-
watch: inOrder(watch, trackPromotion),
|
|
241
|
+
watch: inOrder(watch, trackPromotion, trackBaseHealth),
|
|
241
242
|
sweep: inOrder(sweep, DISPATCH_STEPS.sweep),
|
|
242
243
|
scheduleTasks,
|
|
243
244
|
scheduleDispatch: DISPATCH_STEPS.schedule,
|
package/src/graphs/task.mjs
CHANGED
|
@@ -16,6 +16,7 @@ import { withHistory } from '../marker.mjs'
|
|
|
16
16
|
import { linkIssue, prFor, taskExit, taskPrompt } from '../phases/task.mjs'
|
|
17
17
|
import { PREPARE_FAILED_LINE } from '../prepare.mjs'
|
|
18
18
|
import { traceWorker } from '../tracing.mjs'
|
|
19
|
+
import { baseRed, isRepair } from '../base-health.mjs'
|
|
19
20
|
|
|
20
21
|
/**
|
|
21
22
|
* @typedef {import('../github.mjs').Issue} Issue
|
|
@@ -607,27 +608,40 @@ const requeue = declare(
|
|
|
607
608
|
z.object({ to: z.literal('start'), attempt: z.number().int() }),
|
|
608
609
|
z.object({ to: z.literal('human'), ...failure }),
|
|
609
610
|
]),
|
|
610
|
-
/**
|
|
611
|
+
/**
|
|
612
|
+
* Back to `q:task` with attempt + 1, or to the Human Queue once the attempts are spent. While the base is red
|
|
613
|
+
* (base-health.mjs) a failure is not the work's own: it requeues at the same attempt, never to the Human
|
|
614
|
+
* Queue, and Schedule holds it until the base is green. The repair issue itself is charged as usual.
|
|
615
|
+
* @param {z.infer<typeof Failure>} what @param {TaskCtx} ctx
|
|
616
|
+
*/
|
|
611
617
|
run: async (what, ctx) => {
|
|
612
618
|
const issue = issueOf(ctx)
|
|
613
619
|
const m = markerOf(issue)
|
|
614
620
|
const max = ctx.config.phases.task.maxAttempts
|
|
615
|
-
|
|
616
|
-
|
|
621
|
+
const free = baseRed(ctx.subject.snap) && !isRepair(issue)
|
|
622
|
+
if (m.attempt >= max && !free)
|
|
623
|
+
return { ...what, to: 'human', reason: `${max} attempts spent. Last: ${what.reason}` }
|
|
624
|
+
const { pr, log, failedSha } = what
|
|
625
|
+
const base = ctx.subject.snap?.base
|
|
626
|
+
const outcome = free ? 'base-red' : what.outcome
|
|
627
|
+
const reason = free
|
|
628
|
+
? `${base?.branch} is red at ${(base?.red?.sha ?? '').slice(0, 7)}; not charged. ${what.reason}`
|
|
629
|
+
: what.reason
|
|
630
|
+
const next = free ? m.attempt : m.attempt + 1
|
|
617
631
|
await land(ctx, 'start-worker', {
|
|
618
632
|
...withHistory(m, { ...nodeAt(what), at: iso(ctx.tick.now), attempt: m.attempt, outcome, reason, pr, log }),
|
|
619
|
-
attempt:
|
|
633
|
+
attempt: next,
|
|
620
634
|
lastFailure: reason,
|
|
621
635
|
host: null,
|
|
622
636
|
start: null,
|
|
623
637
|
input: null,
|
|
624
638
|
...(failedSha !== undefined ? { failedSha } : {}),
|
|
625
639
|
})
|
|
626
|
-
ctx.log('info', `requeued (attempt ${
|
|
640
|
+
ctx.log('info', `requeued (attempt ${next}/${max}): ${reason.split('\n')[0]}`, {
|
|
627
641
|
repo: ctx.subject.slug,
|
|
628
642
|
issue: issue.number,
|
|
629
643
|
})
|
|
630
|
-
return { to: 'start', attempt:
|
|
644
|
+
return { to: 'start', attempt: next }
|
|
631
645
|
},
|
|
632
646
|
}),
|
|
633
647
|
{ queue: Q.task, running: false },
|
package/src/tick.mjs
CHANGED
|
@@ -16,6 +16,7 @@ import { PREPARE_FAILED_LINE } from './prepare.mjs'
|
|
|
16
16
|
import { releaseDue, releaseNotBuilt } from './phases/release.mjs'
|
|
17
17
|
import { releaseGraphs } from './graphs/release.mjs'
|
|
18
18
|
import { DISPATCH_NOT_BUILT, dispatchNotBuilt } from './phases/dispatch.mjs'
|
|
19
|
+
import { isRepair } from './base-health.mjs'
|
|
19
20
|
|
|
20
21
|
/**
|
|
21
22
|
* @typedef {import('./github.mjs').Issue} Issue
|
|
@@ -438,6 +439,10 @@ async function scheduleTasks(deps, ctx) {
|
|
|
438
439
|
/**
|
|
439
440
|
* The `q:task` issues to start this tick: no `running`, no tracked worker, by priority then age, up to the free
|
|
440
441
|
* slots counted from live task workers.
|
|
442
|
+
*
|
|
443
|
+
* The base's health gates it (base-health.mjs): a repo whose base is red starts only its repair issue, which
|
|
444
|
+
* takes no slot (it is what frees the others); a repair issue on a green base is left for `trackBaseHealth` to
|
|
445
|
+
* close.
|
|
441
446
|
* @param {Deps} deps @param {Ctx} ctx
|
|
442
447
|
*/
|
|
443
448
|
export function taskCandidates(deps, ctx) {
|
|
@@ -445,18 +450,27 @@ export function taskCandidates(deps, ctx) {
|
|
|
445
450
|
const free = config.phases.task.cap - store.workers().filter((w) => w.phase === 'task').length
|
|
446
451
|
/** @type {{ slug: string, issue: Issue, snap: Snapshot }[]} */
|
|
447
452
|
const candidates = []
|
|
448
|
-
|
|
449
|
-
|
|
453
|
+
/** @type {{ slug: string, issue: Issue, snap: Snapshot }[]} */
|
|
454
|
+
const repairs = []
|
|
455
|
+
for (const [slug, snap] of ctx.snaps) {
|
|
456
|
+
const health = snap.base?.state ?? 'unknown'
|
|
457
|
+
for (const issue of snap.issues.values()) {
|
|
450
458
|
if (
|
|
451
|
-
issue.state
|
|
452
|
-
issue.labels.includes(Q.task)
|
|
453
|
-
|
|
454
|
-
|
|
459
|
+
issue.state !== 'OPEN' ||
|
|
460
|
+
!issue.labels.includes(Q.task) ||
|
|
461
|
+
issue.labels.includes(RUNNING) ||
|
|
462
|
+
store.worker(slug, issue.number)
|
|
455
463
|
)
|
|
456
|
-
|
|
464
|
+
continue
|
|
465
|
+
if (isRepair(issue)) {
|
|
466
|
+
if (health !== 'green') repairs.push({ slug, issue, snap })
|
|
467
|
+
} else if (health !== 'red') candidates.push({ slug, issue, snap })
|
|
468
|
+
}
|
|
469
|
+
}
|
|
457
470
|
const order = byPriorityThenAge(config.priorityLabels)
|
|
458
471
|
candidates.sort((a, b) => order(a.issue, b.issue))
|
|
459
|
-
|
|
472
|
+
repairs.sort((a, b) => order(a.issue, b.issue))
|
|
473
|
+
return [...repairs, ...candidates.slice(0, Math.max(0, free))]
|
|
460
474
|
}
|
|
461
475
|
|
|
462
476
|
/** @param {Deps} deps @param {Ctx} ctx */
|
|
@@ -624,7 +638,10 @@ async function release(deps, ctx) {
|
|
|
624
638
|
* steps: StepResult[],
|
|
625
639
|
* counts: Counts & { workers: number },
|
|
626
640
|
* caps: { task: number, dispatch: number },
|
|
627
|
-
* repos: {
|
|
641
|
+
* repos: {
|
|
642
|
+
* slug: string, ok: boolean, error: string | null, counts: Counts | null,
|
|
643
|
+
* base: import('./base-health.mjs').BaseHealth | null,
|
|
644
|
+
* }[],
|
|
628
645
|
* workers: {
|
|
629
646
|
* repo: string, issue: number, phase: string, pid: number, alive: boolean, attempt: number, branch: string,
|
|
630
647
|
* worktree: string, startedAt: string, elapsedMs: number, wallClockMs: number, killedReason: string | null,
|
|
@@ -665,6 +682,7 @@ export function heartbeatState(deps, ctx) {
|
|
|
665
682
|
ok: Boolean(snap),
|
|
666
683
|
error: ctx.readErrors.get(r.slug) ?? null,
|
|
667
684
|
counts: snap ? countsOf(snap) : null,
|
|
685
|
+
base: snap?.base ?? null,
|
|
668
686
|
}
|
|
669
687
|
})
|
|
670
688
|
const total = { intake: 0, task: 0, human: 0, promote: 0, running: 0, review: 0 }
|