@oakgreencc/cambium 0.0.24 → 0.0.25

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -127,6 +127,7 @@ One JSON file. Everything but `repos` is optional; these are the defaults:
127
127
  - `repos[].path` is a local clone; worktrees are added from it and `graft` runs in it.
128
128
  - `graft.app` is the App cambium acts as (`GRAFT_APP` for its own `graft` calls); `worker.graftApp` the one workers act as. Neither leaks into the other.
129
129
  - `merge` (top level, or per repo in `repos[]`): who merges a Task PR. See [Merging](#merging).
130
+ - `repos[].requiredCheck` (default `"ci"`): the check whose runs on `baseBranch` say whether the base is green; `null` turns the [red-base](#red-base) gate and repair off for that repo.
130
131
  - `engine` picks how the Task phase runs: `"legacy"` (the stage tick, the default) or `"graph"` (the [`task` graph](#the-task-graph)). Both make the same moves; see [The task graph](#the-task-graph). Only `"graph"` runs the [`dispatch` graph](#the-dispatch-graph) (gated by `phases.dispatch.cap`); legacy leaves `q:intake` alone.
131
132
  - Setting `paths.home` moves every default path under it.
132
133
  - `release.time` is host-local `HH:MM`.
@@ -323,6 +324,14 @@ Neither of our repos runs a GitHub merge queue (it costs), so `merge` says who m
323
324
 
324
325
  The worker prompt says which: "Green: stop; cambium merges the PR once CI is green" or "…the PR is merged outside this session".
325
326
 
327
+ ## Red base
328
+
329
+ Every Task PR lands on `baseBranch`, so a red base fails every worker for a reason that is not its own. Without a gate each one spends its attempts and lands in `q:human` (sektorapp/monorepo#3972: three attempts killed at the cap while #4624 + #4625, green apart, had broken `develop` together). `src/base-health.mjs`, graph engine:
330
+
331
+ - **Observe.** The tick's snapshot reads `baseBranch`'s last 20 commits with their `requiredCheck` run. The newest commit with a verdict decides: `green`, `red` (with the last green commit and the merges since it, the suspects), or `unknown` (no completed run in the window; a pending or cancelled run says nothing, so a red base stays red until a run passes). The heartbeat carries it per repo (`repos[].base`). The base needs its check to run on push: a repo whose CI runs only on pull requests is always `unknown`.
332
+ - **Gate.** While red, Schedule starts no Task worker on that repo except the repair, which takes no slot. A failure during a red base requeues at the same attempt (history `base-red`, "not charged") and never goes to `q:human`; it waits in `q:task` for green. `unknown` gates nothing.
333
+ - **Resolve.** Red with no open repair issue → cambium opens one: labelled `red-base`, `q:task` and the first `priorityLabels` entry, its body the worker's spec (the failing run, the suspects, fix forward or revert the culprit, `pnpm verify`). It runs as an ordinary Task issue and is charged like one, so a repair that spends its attempts goes to `q:human`. Its PR merges through graft's red-base rescue (up to date with the base, its own check green). Green again → a repair issue with no open PR and no worker is closed with a comment.
334
+
326
335
  ## The dispatch graph
327
336
 
328
337
  With `"engine": "graph"` and `phases.dispatch.cap > 0`, each `q:intake` issue is triaged by a one-shot `claude -p` worker (`src/graphs/dispatch.mjs`, steps in `src/dispatch-engine.mjs`, prompt and result in `src/phases/dispatch.mjs`), then moved to `q:task`, moved to `q:human`, or closed as a duplicate.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@oakgreencc/cambium",
3
- "version": "0.0.24",
3
+ "version": "0.0.25",
4
4
  "description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
5
5
  "license": "AGPL-3.0-only",
6
6
  "type": "module",
@@ -0,0 +1,208 @@
1
+ // @ts-check
2
+ /**
3
+ * The base branch's health, and the repair of a red one.
4
+ *
5
+ * Every Task PR lands on the repo's `baseBranch`. When that branch is red, every worker's verify fails for a
6
+ * reason that is not its own, and without this module each one spends its attempts and lands in `q:human`
7
+ * (sektorapp/monorepo#3972: three attempts, three kills, on a break #4624 + #4625 made together). So:
8
+ *
9
+ * - **Observe.** The snapshot reads the base's last {@link HISTORY} commits with their `requiredCheck` run
10
+ * ({@link BASE_FIELDS}); {@link baseHealthOf} turns them into green / red / unknown, the last green commit and
11
+ * the merges since it.
12
+ * - **Gate.** While red, Schedule starts no Task worker for that repo except the repair (tick.mjs
13
+ * `taskCandidates`), and a failure is requeued without spending an attempt (graphs/task.mjs `requeue`).
14
+ * - **Resolve.** {@link trackBaseHealth} opens one repair issue per red base ({@link REPAIR_LABEL}, `q:task`,
15
+ * the top priority): its body is the worker's spec, and its PR merges through graft's red-base rescue. A base
16
+ * that goes green again closes a repair issue nobody is working.
17
+ *
18
+ * Unknown (no completed run in the window: the base has no `requiredCheck` on push, or every run is cancelled or
19
+ * pending) gates nothing.
20
+ */
21
+ import { iso, key } from './issue-state.mjs'
22
+ import { Q } from './labels.mjs'
23
+
24
+ /** Marks the repair issue. Not a `why:` label: those leave with `q:human` (labels.mjs `move`). */
25
+ export const REPAIR_LABEL = 'red-base'
26
+
27
+ /** How many base commits the snapshot reads, newest first. */
28
+ export const HISTORY = 20
29
+
30
+ /**
31
+ * The snapshot's base read, under `repository`. Needs `$baseRef`, `$check` and `$health`.
32
+ */
33
+ export const BASE_FIELDS = `baseRef: ref(qualifiedName: $baseRef) @include(if: $health) {
34
+ target { ... on Commit { history(first: ${HISTORY}) { nodes {
35
+ oid messageHeadline
36
+ associatedPullRequests(first: 1) { nodes { number } }
37
+ checkSuites(first: 30) { nodes { branch { name } checkRuns(first: 3, filterBy: { checkName: $check }) { nodes { status conclusion detailsUrl } } } }
38
+ } } } }
39
+ }
40
+ repairs: issues(labels: ["${REPAIR_LABEL}"], states: OPEN, first: 5) @include(if: $health) { nodes { number } }`
41
+
42
+ /**
43
+ * @typedef {{ sha: string, headline: string, pr: number | null }} BaseCommit
44
+ * @typedef {{
45
+ * branch: string, check: string, state: 'green' | 'red' | 'unknown', head: string | null,
46
+ * red: (BaseCommit & { url: string | null }) | null, lastGreen: BaseCommit | null, suspects: BaseCommit[],
47
+ * repairs: number[],
48
+ * }} BaseHealth
49
+ * `red`: the newest commit whose run failed (the verdict); `lastGreen`: the newest passing commit older than
50
+ * it; `suspects`: the commits between them, newest first, `red` included, one per PR. `repairs`: open repair
51
+ * issues.
52
+ */
53
+
54
+ const PASSED = new Set(['SUCCESS', 'NEUTRAL', 'SKIPPED'])
55
+ const FAILED = new Set(['FAILURE', 'TIMED_OUT', 'STARTUP_FAILURE'])
56
+
57
+ /**
58
+ * A commit's verdict from its `check` runs on `branch`: the first completed run that says anything. Only a suite
59
+ * run for the base branch counts: the history also holds the merged PRs' own commits, whose runs judged the PR
60
+ * against an older base (PR CI on sektorapp is affected-only), not the base.
61
+ * @param {any} node @param {string} branch
62
+ */
63
+ function verdictOf(node, branch) {
64
+ for (const suite of node?.checkSuites?.nodes ?? []) {
65
+ if (suite?.branch?.name !== branch) continue
66
+ for (const run of suite?.checkRuns?.nodes ?? []) {
67
+ if (run?.status !== 'COMPLETED') continue
68
+ if (PASSED.has(run.conclusion)) return { pass: true, url: run.detailsUrl ?? null }
69
+ if (FAILED.has(run.conclusion)) return { pass: false, url: run.detailsUrl ?? null }
70
+ }
71
+ }
72
+ return null
73
+ }
74
+
75
+ /** @param {any} n @returns {BaseCommit} */
76
+ const commitOf = (n) => ({
77
+ sha: n.oid,
78
+ headline: n.messageHeadline ?? '',
79
+ pr: n.associatedPullRequests?.nodes?.[0]?.number ?? null,
80
+ })
81
+
82
+ /**
83
+ * The base's health from its history, newest first. The newest commit with a verdict decides: a pending or
84
+ * cancelled run on the head says nothing, so a red base stays red until a run passes.
85
+ *
86
+ * @param {any} repo the snapshot's `repository` @param {{ branch: string, check: string }} at
87
+ * @returns {BaseHealth}
88
+ */
89
+ export function baseHealthOf(repo, { branch, check }) {
90
+ const nodes = repo?.baseRef?.target?.history?.nodes ?? []
91
+ const repairs = (repo?.repairs?.nodes ?? []).map((/** @type {any} */ n) => n.number)
92
+ const base = { branch, check, head: nodes[0]?.oid ?? null, repairs }
93
+ const verdict = (/** @type {any} */ n) => verdictOf(n, branch)
94
+ const decided = nodes.findIndex(verdict)
95
+ if (decided === -1) return { ...base, state: 'unknown', red: null, lastGreen: null, suspects: [] }
96
+ const v = /** @type {{ pass: boolean, url: string | null }} */ (verdict(nodes[decided]))
97
+ if (v.pass) return { ...base, state: 'green', red: null, lastGreen: commitOf(nodes[decided]), suspects: [] }
98
+ const greenAt = nodes.findIndex((/** @type {any} */ n, /** @type {number} */ i) => i > decided && verdict(n)?.pass)
99
+ /** @type {Set<number>} */
100
+ const seen = new Set()
101
+ const suspects = nodes
102
+ .slice(decided, greenAt === -1 ? undefined : greenAt)
103
+ .map(commitOf)
104
+ .filter((c) => c.pr === null || (!seen.has(c.pr) && seen.add(c.pr)))
105
+ return {
106
+ ...base,
107
+ state: 'red',
108
+ red: { ...commitOf(nodes[decided]), url: v.url },
109
+ lastGreen: greenAt === -1 ? null : commitOf(nodes[greenAt]),
110
+ suspects,
111
+ }
112
+ }
113
+
114
+ /** Red, per the snapshot. @param {import('./github.mjs').Snapshot | undefined} snap */
115
+ export const baseRed = (snap) => snap?.base?.state === 'red'
116
+
117
+ /** @param {import('./github.mjs').Issue} issue */
118
+ export const isRepair = (issue) => issue.labels.includes(REPAIR_LABEL)
119
+
120
+ const short = (/** @type {string} */ sha) => sha.slice(0, 7)
121
+
122
+ /** @param {BaseHealth} h */
123
+ export const repairTitle = (h) => `Red base: ${h.branch} \`${h.check}\` failing since ${short(h.red?.sha ?? '')}`
124
+
125
+ /**
126
+ * The repair issue's body: the worker's spec.
127
+ * @param {BaseHealth} h
128
+ */
129
+ export function repairBody(h) {
130
+ const red = /** @type {NonNullable<BaseHealth['red']>} */ (h.red)
131
+ const line = (/** @type {BaseCommit} */ c) => `- ${short(c.sha)}${c.pr ? ` #${c.pr}` : ''} ${c.headline}`
132
+ return [
133
+ `\`${h.branch}\` is red: its \`${h.check}\` check failed on ${short(red.sha)}${red.url ? ` (${red.url})` : ''}.`,
134
+ '',
135
+ 'Opened by cambium. While this is open cambium starts no other Task worker on this repo, and their failures cost no attempts.',
136
+ '',
137
+ '## Suspects',
138
+ '',
139
+ h.lastGreen
140
+ ? `Last green: ${short(h.lastGreen.sha)} ${h.lastGreen.headline}. Merged since, newest first:`
141
+ : `No green commit in the last ${HISTORY}. Newest first:`,
142
+ '',
143
+ ...h.suspects.map(line),
144
+ '',
145
+ '## Work',
146
+ '',
147
+ `1. Read why it failed: \`graft jobs ${red.url ?? '<run>'}\`. Reproduce with \`pnpm verify\` in your worktree, which starts at \`${h.branch}\`.`,
148
+ '2. Two PRs that are each green can be red together, so the cause may span suspects.',
149
+ '3. Fix forward when the fix is clear and small. Otherwise revert the merge that broke it, and say on the PR which one and why.',
150
+ '4. Change only what makes the check green; do not fold other work into this PR.',
151
+ '',
152
+ `Your PR closes this issue. cambium merges it through graft's red-base rescue, once it is up to date with \`${h.branch}\` and its own \`${h.check}\` is green.`,
153
+ '',
154
+ '## Done when',
155
+ '',
156
+ `- \`pnpm verify\` passes on \`${h.branch}\` with your change.`,
157
+ `- \`${h.branch}\`'s \`${h.check}\` is green after the merge.`,
158
+ ].join('\n')
159
+ }
160
+
161
+ /**
162
+ * The repair issue's lifecycle, once per tick before Schedule. Red with no open repair issue → open one
163
+ * (`q:task`, the top priority label, {@link REPAIR_LABEL}). Green, with an open repair issue that has no open PR
164
+ * and no worker → close it. Unknown → nothing.
165
+ *
166
+ * @param {import('./tick.mjs').Deps} deps @param {import('./tick.mjs').Ctx} ctx
167
+ */
168
+ export async function trackBaseHealth(deps, ctx) {
169
+ for (const [slug, snap] of ctx.snaps) {
170
+ const h = snap.base
171
+ if (!h) continue
172
+ try {
173
+ if (h.state === 'red' && !h.repairs.length) {
174
+ const made = await deps.github.createIssue(slug, {
175
+ title: repairTitle(h),
176
+ body: repairBody(h),
177
+ labels: [Q.task, deps.config.priorityLabels[0], REPAIR_LABEL].filter(Boolean),
178
+ })
179
+ h.repairs = [made.number]
180
+ deps.log('warn', `${h.branch} is red at ${short(h.red?.sha ?? '')}; opened repair #${made.number}`, {
181
+ repo: slug,
182
+ issue: made.number,
183
+ })
184
+ continue
185
+ }
186
+ if (h.state !== 'green') continue
187
+ /** @type {number[]} */
188
+ const closed = []
189
+ for (const n of h.repairs) {
190
+ const worked = deps.store.worker(slug, n) || snap.prs.some((p) => p.state === 'OPEN' && p.closes.includes(n))
191
+ if (worked || ctx.touched.has(key(slug, n))) continue
192
+ await deps.github.close(
193
+ slug,
194
+ n,
195
+ `\`${h.branch}\` is green again at ${short(h.lastGreen?.sha ?? h.head ?? '')} (${iso(ctx.now)}); nothing left to repair.`,
196
+ { reason: 'completed' },
197
+ )
198
+ const issue = snap.issues.get(n)
199
+ if (issue) issue.state = 'CLOSED'
200
+ closed.push(n)
201
+ deps.log('info', `${h.branch} is green again; closed repair #${n}`, { repo: slug, issue: n })
202
+ }
203
+ h.repairs = h.repairs.filter((n) => !closed.includes(n))
204
+ } catch (err) {
205
+ deps.log('error', `base health: ${/** @type {Error} */ (err)?.message ?? err}`, { repo: slug })
206
+ }
207
+ }
208
+ }
package/src/config.mjs CHANGED
@@ -16,7 +16,12 @@ import { join, resolve } from 'node:path'
16
16
  * Who merges a Task PR. `none`: not cambium (a merge queue, auto-merge, or a human; past `mergeQueueMinutes`
17
17
  * the issue goes to `q:human`). `green`: cambium, once its checks are green, through `graft pr merge --auto`
18
18
  * (so graft's merge policy still decides). `green` needs `engine: "graph"`.
19
- * @typedef {{ slug: string, path: string, baseBranch: string, release: RepoRelease, merge: MergePolicy }} Repo
19
+ * @typedef {{
20
+ * slug: string, path: string, baseBranch: string, release: RepoRelease, merge: MergePolicy,
21
+ * requiredCheck: string | null,
22
+ * }} Repo
23
+ * `requiredCheck` (default `ci`): the check whose runs on `baseBranch` say whether the base is green (see
24
+ * base-health.mjs); `null` turns the red-base gate and repair off for the repo.
20
25
  * @typedef {{ cap: number, wallClockMinutes: number, maxTurns: number, maxAttempts: number }} PhaseCaps
21
26
  * @typedef {{
22
27
  * host: string,
@@ -121,7 +126,10 @@ export function resolveConfig(raw = {}) {
121
126
  const merge = r.merge ?? config.merge
122
127
  if (!MERGE_POLICIES.includes(merge))
123
128
  throw new Error(`config: repo ${r.slug} merge must be "none" or "green", got ${JSON.stringify(merge)}`)
124
- return { slug: r.slug, path: untilde(r.path), baseBranch, release, merge }
129
+ const requiredCheck = r.requiredCheck === undefined ? 'ci' : r.requiredCheck
130
+ if (requiredCheck !== null && (typeof requiredCheck !== 'string' || !requiredCheck))
131
+ throw new Error(`config: repo ${r.slug} requiredCheck must be a check name or null`)
132
+ return { slug: r.slug, path: untilde(r.path), baseBranch, release, merge, requiredCheck }
125
133
  })
126
134
  if (!MERGE_POLICIES.includes(config.merge))
127
135
  throw new Error(`config: merge must be "none" or "green", got ${JSON.stringify(config.merge)}`)
package/src/github.mjs CHANGED
@@ -8,6 +8,7 @@
8
8
  import { spawn } from 'node:child_process'
9
9
  import { parseMarker } from './marker.mjs'
10
10
  import { Q, QUEUE_LABELS } from './labels.mjs'
11
+ import { BASE_FIELDS, baseHealthOf } from './base-health.mjs'
11
12
 
12
13
  /**
13
14
  * @typedef {{
@@ -35,13 +36,18 @@ import { Q, QUEUE_LABELS } from './labels.mjs'
35
36
  * The repo's promotion state (when its `release` is on): how far `from` is ahead of `to` (null: `to` does not
36
37
  * exist), the PRs merged into `from` since `to` (oldest first, from the last 100 commits ahead), and the
37
38
  * promotion PRs: open ones from `from` to `to` plus any (open, merged or closed) still labelled `q:promote`.
38
- * @typedef {{ issues: Map<number, Issue>, prs: PullRequest[], release?: Release | null }} Snapshot
39
+ * @typedef {{
40
+ * issues: Map<number, Issue>, prs: PullRequest[], release?: Release | null,
41
+ * base?: import('./base-health.mjs').BaseHealth,
42
+ * }} Snapshot
43
+ * `base`: the base branch's health, when the repo has a `requiredCheck`.
39
44
  * @typedef {{
40
45
  * read(slug: string): Promise<Snapshot>,
41
46
  * label(slug: string, number: number, change: { add: string[], remove: string[] }): Promise<void>,
42
47
  * writeState(slug: string, number: number, commentId: number | null, body: string): Promise<number>,
43
48
  * close(slug: string, number: number, body: string, opts?: { reason?: 'completed' | 'not_planned' | 'duplicate' }): Promise<void>,
44
49
  * comment(slug: string, number: number, body: string): Promise<void>,
50
+ * createIssue(slug: string, issue: { title: string, body: string, labels: string[] }): Promise<{ number: number, url: string }>,
45
51
  * deleteBranch(slug: string, branch: string): Promise<void>,
46
52
  * openPull(slug: string, pr: { head: string, base: string, title: string, body: string }): Promise<{ number: number, url: string }>,
47
53
  * editPull(slug: string, number: number, change: { title: string, body: string }): Promise<void>,
@@ -70,7 +76,7 @@ const PR_FIELDS = `number url headRefName state headRefOid isInMergeQueue body
70
76
  * One query per repo per tick: the queued issues and the agent PRs with their checks; with `$release`, also the
71
77
  * promotion PRs and how far `$from` is ahead of `$to` (see {@link Release}).
72
78
  */
73
- export const SNAPSHOT_QUERY = `query($owner: String!, $name: String!, $labels: [String!], $release: Boolean!, $from: String!, $to: String!) {
79
+ export const SNAPSHOT_QUERY = `query($owner: String!, $name: String!, $labels: [String!], $release: Boolean!, $from: String!, $to: String!, $health: Boolean!, $baseRef: String!, $check: String!) {
74
80
  repository(owner: $owner, name: $name) {
75
81
  issues(labels: $labels, states: OPEN, first: 100, orderBy: { field: CREATED_AT, direction: ASC }) { nodes { ...I } }
76
82
  open: pullRequests(states: OPEN, first: 100, orderBy: { field: UPDATED_AT, direction: DESC }) { nodes { ${PR_FIELDS} } }
@@ -84,6 +90,7 @@ export const SNAPSHOT_QUERY = `query($owner: String!, $name: String!, $labels: [
84
90
  commits(last: 100) { nodes { associatedPullRequests(first: 3) { nodes { number title baseRefName state } } } }
85
91
  }
86
92
  }
93
+ ${BASE_FIELDS}
87
94
  }
88
95
  }
89
96
  fragment I on Issue {
@@ -222,7 +229,7 @@ export function addIssues(snap, data, asked, botLogin = null) {
222
229
  * `botLogin`: when set, only that login's markers count. `release`: the branches the query asked about.
223
230
  * @returns {Snapshot}
224
231
  */
225
- export function normalize(data, { botLogin = null, release = null } = {}) {
232
+ export function normalize(data, { botLogin = null, release = null, health = null } = {}) {
226
233
  const repo = (data?.data ?? data)?.repository
227
234
  if (!repo) throw new Error('snapshot: no repository in the answer')
228
235
  /** @type {Map<number, Issue>} */
@@ -258,7 +265,12 @@ export function normalize(data, { botLogin = null, release = null } = {}) {
258
265
  closes: [...new Set([...(p.closingIssuesReferences?.nodes ?? []).map(addIssue), ...bodyCloses(p.body)])],
259
266
  })
260
267
  }
261
- return { issues, prs, ...(release ? { release: normalizeRelease(repo, release, botLogin) } : {}) }
268
+ return {
269
+ issues,
270
+ prs,
271
+ ...(release ? { release: normalizeRelease(repo, release, botLogin) } : {}),
272
+ ...(health ? { base: baseHealthOf(repo, health) } : {}),
273
+ }
262
274
  }
263
275
 
264
276
  /**
@@ -309,9 +321,20 @@ export function graftGitHub({ config, exec: run = exec }) {
309
321
  // (promotion.mjs); the release graphs check `enabled` themselves.
310
322
  const release = r && r.from !== r.to ? { from: r.from, to: r.to } : null
311
323
  const branches = release ?? { from: '', to: '' }
312
- const vars = JSON.stringify({ owner, name, labels: QUEUE_LABELS, release: Boolean(release), ...branches })
324
+ const repo = config.repos.find((x) => x.slug === slug)
325
+ const health = repo?.requiredCheck ? { branch: repo.baseBranch, check: repo.requiredCheck } : null
326
+ const vars = JSON.stringify({
327
+ owner,
328
+ name,
329
+ labels: QUEUE_LABELS,
330
+ release: Boolean(release),
331
+ ...branches,
332
+ health: Boolean(health),
333
+ baseRef: health ? `refs/heads/${health.branch}` : '',
334
+ check: health?.check ?? '',
335
+ })
313
336
  const data = await json(graft(slug, ['graphql', SNAPSHOT_QUERY, '--vars', vars]))
314
- const snap = normalize(data, { botLogin: config.graft.botLogin, release })
337
+ const snap = normalize(data, { botLogin: config.graft.botLogin, release, health })
315
338
  const unread = unreadCloses(snap)
316
339
  if (!unread.length) return snap
317
340
  // Issues named only in PR bodies (in no queue): one more query, for those.
@@ -353,6 +376,18 @@ export function graftGitHub({ config, exec: run = exec }) {
353
376
  async comment(slug, number, body) {
354
377
  await graft(slug, ['issue', 'comment', String(number), '--body-file', '-'], body)
355
378
  },
379
+ async createIssue(slug, { title, body, labels }) {
380
+ const args = [
381
+ 'issue',
382
+ 'create',
383
+ '--title',
384
+ title,
385
+ '--body-file',
386
+ '-',
387
+ ...(labels.length ? ['--labels', labels.join(',')] : []),
388
+ ]
389
+ return json(graft(slug, args, body))
390
+ },
356
391
  async deleteBranch(slug, branch) {
357
392
  await graft(slug, ['api', 'DELETE', `/repos/${slug}/git/refs/heads/${branch}`])
358
393
  },
@@ -25,6 +25,7 @@ import { move, Q } from './labels.mjs'
25
25
  import { markerTrace, turnTracer } from './tracing.mjs'
26
26
  import { DISPATCH_STEPS } from './dispatch-engine.mjs'
27
27
  import { trackPromotion } from './promotion.mjs'
28
+ import { trackBaseHealth } from './base-health.mjs'
28
29
 
29
30
  /**
30
31
  * @typedef {import('./tick.mjs').Deps} Deps
@@ -237,7 +238,7 @@ const inOrder =
237
238
  /** @type {import('./tick.mjs').Engine} */
238
239
  export const GRAPH_ENGINE = {
239
240
  reconcile: inOrder(reconcile, DISPATCH_STEPS.reconcile),
240
- watch: inOrder(watch, trackPromotion),
241
+ watch: inOrder(watch, trackPromotion, trackBaseHealth),
241
242
  sweep: inOrder(sweep, DISPATCH_STEPS.sweep),
242
243
  scheduleTasks,
243
244
  scheduleDispatch: DISPATCH_STEPS.schedule,
@@ -16,6 +16,7 @@ import { withHistory } from '../marker.mjs'
16
16
  import { linkIssue, prFor, taskExit, taskPrompt } from '../phases/task.mjs'
17
17
  import { PREPARE_FAILED_LINE } from '../prepare.mjs'
18
18
  import { traceWorker } from '../tracing.mjs'
19
+ import { baseRed, isRepair } from '../base-health.mjs'
19
20
 
20
21
  /**
21
22
  * @typedef {import('../github.mjs').Issue} Issue
@@ -607,27 +608,40 @@ const requeue = declare(
607
608
  z.object({ to: z.literal('start'), attempt: z.number().int() }),
608
609
  z.object({ to: z.literal('human'), ...failure }),
609
610
  ]),
610
- /** Back to `q:task` with attempt + 1, or to the Human Queue once the attempts are spent. @param {z.infer<typeof Failure>} what @param {TaskCtx} ctx */
611
+ /**
612
+ * Back to `q:task` with attempt + 1, or to the Human Queue once the attempts are spent. While the base is red
613
+ * (base-health.mjs) a failure is not the work's own: it requeues at the same attempt, never to the Human
614
+ * Queue, and Schedule holds it until the base is green. The repair issue itself is charged as usual.
615
+ * @param {z.infer<typeof Failure>} what @param {TaskCtx} ctx
616
+ */
611
617
  run: async (what, ctx) => {
612
618
  const issue = issueOf(ctx)
613
619
  const m = markerOf(issue)
614
620
  const max = ctx.config.phases.task.maxAttempts
615
- if (m.attempt >= max) return { ...what, to: 'human', reason: `${max} attempts spent. Last: ${what.reason}` }
616
- const { reason, outcome, pr, log, failedSha } = what
621
+ const free = baseRed(ctx.subject.snap) && !isRepair(issue)
622
+ if (m.attempt >= max && !free)
623
+ return { ...what, to: 'human', reason: `${max} attempts spent. Last: ${what.reason}` }
624
+ const { pr, log, failedSha } = what
625
+ const base = ctx.subject.snap?.base
626
+ const outcome = free ? 'base-red' : what.outcome
627
+ const reason = free
628
+ ? `${base?.branch} is red at ${(base?.red?.sha ?? '').slice(0, 7)}; not charged. ${what.reason}`
629
+ : what.reason
630
+ const next = free ? m.attempt : m.attempt + 1
617
631
  await land(ctx, 'start-worker', {
618
632
  ...withHistory(m, { ...nodeAt(what), at: iso(ctx.tick.now), attempt: m.attempt, outcome, reason, pr, log }),
619
- attempt: m.attempt + 1,
633
+ attempt: next,
620
634
  lastFailure: reason,
621
635
  host: null,
622
636
  start: null,
623
637
  input: null,
624
638
  ...(failedSha !== undefined ? { failedSha } : {}),
625
639
  })
626
- ctx.log('info', `requeued (attempt ${m.attempt + 1}/${max}): ${reason.split('\n')[0]}`, {
640
+ ctx.log('info', `requeued (attempt ${next}/${max}): ${reason.split('\n')[0]}`, {
627
641
  repo: ctx.subject.slug,
628
642
  issue: issue.number,
629
643
  })
630
- return { to: 'start', attempt: m.attempt + 1 }
644
+ return { to: 'start', attempt: next }
631
645
  },
632
646
  }),
633
647
  { queue: Q.task, running: false },
package/src/tick.mjs CHANGED
@@ -16,6 +16,7 @@ import { PREPARE_FAILED_LINE } from './prepare.mjs'
16
16
  import { releaseDue, releaseNotBuilt } from './phases/release.mjs'
17
17
  import { releaseGraphs } from './graphs/release.mjs'
18
18
  import { DISPATCH_NOT_BUILT, dispatchNotBuilt } from './phases/dispatch.mjs'
19
+ import { isRepair } from './base-health.mjs'
19
20
 
20
21
  /**
21
22
  * @typedef {import('./github.mjs').Issue} Issue
@@ -438,6 +439,10 @@ async function scheduleTasks(deps, ctx) {
438
439
  /**
439
440
  * The `q:task` issues to start this tick: no `running`, no tracked worker, by priority then age, up to the free
440
441
  * slots counted from live task workers.
442
+ *
443
+ * The base's health gates it (base-health.mjs): a repo whose base is red starts only its repair issue, which
444
+ * takes no slot (it is what frees the others); a repair issue on a green base is left for `trackBaseHealth` to
445
+ * close.
441
446
  * @param {Deps} deps @param {Ctx} ctx
442
447
  */
443
448
  export function taskCandidates(deps, ctx) {
@@ -445,18 +450,27 @@ export function taskCandidates(deps, ctx) {
445
450
  const free = config.phases.task.cap - store.workers().filter((w) => w.phase === 'task').length
446
451
  /** @type {{ slug: string, issue: Issue, snap: Snapshot }[]} */
447
452
  const candidates = []
448
- for (const [slug, snap] of ctx.snaps)
449
- for (const issue of snap.issues.values())
453
+ /** @type {{ slug: string, issue: Issue, snap: Snapshot }[]} */
454
+ const repairs = []
455
+ for (const [slug, snap] of ctx.snaps) {
456
+ const health = snap.base?.state ?? 'unknown'
457
+ for (const issue of snap.issues.values()) {
450
458
  if (
451
- issue.state === 'OPEN' &&
452
- issue.labels.includes(Q.task) &&
453
- !issue.labels.includes(RUNNING) &&
454
- !store.worker(slug, issue.number)
459
+ issue.state !== 'OPEN' ||
460
+ !issue.labels.includes(Q.task) ||
461
+ issue.labels.includes(RUNNING) ||
462
+ store.worker(slug, issue.number)
455
463
  )
456
- candidates.push({ slug, issue, snap })
464
+ continue
465
+ if (isRepair(issue)) {
466
+ if (health !== 'green') repairs.push({ slug, issue, snap })
467
+ } else if (health !== 'red') candidates.push({ slug, issue, snap })
468
+ }
469
+ }
457
470
  const order = byPriorityThenAge(config.priorityLabels)
458
471
  candidates.sort((a, b) => order(a.issue, b.issue))
459
- return candidates.slice(0, Math.max(0, free))
472
+ repairs.sort((a, b) => order(a.issue, b.issue))
473
+ return [...repairs, ...candidates.slice(0, Math.max(0, free))]
460
474
  }
461
475
 
462
476
  /** @param {Deps} deps @param {Ctx} ctx */
@@ -624,7 +638,10 @@ async function release(deps, ctx) {
624
638
  * steps: StepResult[],
625
639
  * counts: Counts & { workers: number },
626
640
  * caps: { task: number, dispatch: number },
627
- * repos: { slug: string, ok: boolean, error: string | null, counts: Counts | null }[],
641
+ * repos: {
642
+ * slug: string, ok: boolean, error: string | null, counts: Counts | null,
643
+ * base: import('./base-health.mjs').BaseHealth | null,
644
+ * }[],
628
645
  * workers: {
629
646
  * repo: string, issue: number, phase: string, pid: number, alive: boolean, attempt: number, branch: string,
630
647
  * worktree: string, startedAt: string, elapsedMs: number, wallClockMs: number, killedReason: string | null,
@@ -665,6 +682,7 @@ export function heartbeatState(deps, ctx) {
665
682
  ok: Boolean(snap),
666
683
  error: ctx.readErrors.get(r.slug) ?? null,
667
684
  counts: snap ? countsOf(snap) : null,
685
+ base: snap?.base ?? null,
668
686
  }
669
687
  })
670
688
  const total = { intake: 0, task: 0, human: 0, promote: 0, running: 0, review: 0 }