@oakgreencc/cambium 0.0.44 → 0.0.45
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -2
- package/package.json +1 -1
- package/src/config.mjs +6 -0
- package/src/cost.mjs +161 -0
- package/src/graphs/dispatch.mjs +2 -0
- package/src/graphs/rescue.mjs +2 -0
- package/src/graphs/task.mjs +2 -0
- package/src/issue-state.mjs +3 -0
- package/src/marker.mjs +4 -0
- package/src/store.mjs +35 -1
- package/src/tick.mjs +49 -0
- package/src/tracing.mjs +11 -0
package/README.md
CHANGED
|
@@ -83,6 +83,8 @@ Each issue cambium touches gets one comment it owns and edits in place. Its firs
|
|
|
83
83
|
|
|
84
84
|
Set `graft.botLogin` to the Clock App's login and only markers written by it are trusted.
|
|
85
85
|
|
|
86
|
+
**Cost.** When a task, triage or rescue worker exits, cambium reads the `result` event at the end of its stream-json log (`total_cost_usd`, `usage`) and adds it to the marker's `cost`: `{ total, triage, task, rescue, lastLog }`, each phase a tally `{ usd, input, output, cacheRead, cacheCreation, runs }` (`src/cost.mjs`). It lives on GitHub, so it survives a restart, and every marker write keeps it. `lastLog` is the last run counted, so a re-checked exit is never counted twice. A worker with no `result` (killed, truncated log) adds nothing; a marker from before `cost` counts from zero. Each run also goes into a per-repo, per-UTC-day ledger in SQLite (`costs` table) for the heartbeat's daily figures; losing the database loses only that history. The rendered comment shows `cost $<total>`.
|
|
87
|
+
|
|
86
88
|
## Workers
|
|
87
89
|
|
|
88
90
|
A worker is `claude -p --output-format stream-json --verbose --thinking-display summarized --max-turns <maxTurns> --allowedTools Read,Edit,Write,Glob,Grep,Bash --disallowedTools "Bash(graft issue label:*),Bash(graft issue edit:*),Bash(gh:*)" <worker.args>`:
|
|
@@ -149,10 +151,12 @@ One JSON file. Everything but `repos` is optional; these are the defaults:
|
|
|
149
151
|
"graft": { "command": "graft", "app": null, "botLogin": null },
|
|
150
152
|
"worker": { "command": "claude", "args": [], "graftApp": null, "fetch": true, "env": {} },
|
|
151
153
|
"update": { "auto": false, "checkMinutes": 360 },
|
|
152
|
-
"gc": { "everyMinutes": 60, "graceMinutes": 60, "staleDays": 14 }
|
|
154
|
+
"gc": { "everyMinutes": 60, "graceMinutes": 60, "staleDays": 14 },
|
|
155
|
+
"costFlagUsd": 10
|
|
153
156
|
}
|
|
154
157
|
```
|
|
155
158
|
|
|
159
|
+
- `costFlagUsd`: an open issue whose task-phase cost is over this many US dollars with no merged PR closing it is `flagged` in the heartbeat; canopy lists it in the Inbox as a session candidate. `0` turns the flag off.
|
|
156
160
|
- `worker.env` is extra environment for every worker (string values only), e.g. Claude Code's own telemetry ([Tracing](#claude-codes-own-spans-under-the-worker)).
|
|
157
161
|
|
|
158
162
|
- The daemon re-reads the file before every tick, so caps, limits, `tickSeconds`, `repos`, priority labels, `release` and `worker` take effect without a restart (logged as `config reloaded: <keys>`). `host`, `engine`, `paths`, `logs` and `graft` are baked in at start: a change to them logs `restart cambium to apply` once and is not applied. A file that does not load, or has no `repos`, keeps the running config and logs one warning until it is fixed.
|
|
@@ -311,7 +315,9 @@ Written at the end of every tick to `paths.heartbeat` (default `~/.cambium/state
|
|
|
311
315
|
}
|
|
312
316
|
```
|
|
313
317
|
|
|
314
|
-
`
|
|
318
|
+
`repos[].cost` is `{ issues, days }`. `issues`: open issues in `q:intake`, `q:task`, `q:human` or `running` whose marker has a cost, most expensive first, each `{ number, title, url, labels, cost, flagged }` (`cost` as on the marker; `flagged` per `costFlagUsd`). `days`: the last 14 UTC days of the repo's ledger with any spend, oldest first, each `{ day, triage, task, rescue, total }` in US dollars. The top-level `costFlagUsd` echoes the config.
|
|
319
|
+
|
|
320
|
+
`engine`, `graphs`, `costFlagUsd` and `repos[].cost` were added within `version: 1`; a reader of an older heartbeat finds none of them.
|
|
315
321
|
|
|
316
322
|
The mode lives in the SQLite file (`paths.db`, table `control`), so `cambium status --json` returns `{ "mode", "heartbeat" }` even when the daemon is down. The daemon's own log is `<paths.logs>/cambium.log` (one `.1` kept); launchd's stderr goes to `<paths.logs>/cambium.launchd.err.log`.
|
|
317
323
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@oakgreencc/cambium",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.45",
|
|
4
4
|
"description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
|
|
5
5
|
"license": "AGPL-3.0-only",
|
|
6
6
|
"type": "module",
|
package/src/config.mjs
CHANGED
|
@@ -49,7 +49,10 @@ import { join, resolve } from 'node:path'
|
|
|
49
49
|
* worker: { command: string, args: string[], graftApp: string | null, fetch: boolean, env: Record<string, string> },
|
|
50
50
|
* update: { auto: boolean, checkMinutes: number },
|
|
51
51
|
* gc: GcConfig,
|
|
52
|
+
* costFlagUsd: number,
|
|
52
53
|
* }} Config
|
|
54
|
+
* `costFlagUsd`: an issue whose task-phase spend (cost.mjs) is over this many US dollars with no merged PR is
|
|
55
|
+
* flagged in the heartbeat, and canopy's Inbox, as a session candidate. 0 turns the flag off.
|
|
53
56
|
* @typedef {{
|
|
54
57
|
* everyMinutes: number, staleHours: number, graceMinutes: number, cooldownHours: number, maxRescues: number,
|
|
55
58
|
* causeRepo: string | null,
|
|
@@ -109,6 +112,7 @@ export function defaults(home = DEFAULT_HOME) {
|
|
|
109
112
|
worker: { command: 'claude', args: [], graftApp: null, fetch: true, env: {} },
|
|
110
113
|
update: { auto: false, checkMinutes: 360 },
|
|
111
114
|
gc: { everyMinutes: 60, graceMinutes: 60, staleDays: 14 },
|
|
115
|
+
costFlagUsd: 10,
|
|
112
116
|
}
|
|
113
117
|
}
|
|
114
118
|
|
|
@@ -185,6 +189,8 @@ export function resolveConfig(raw = {}) {
|
|
|
185
189
|
const causeRepo = config.rescue.causeRepo
|
|
186
190
|
if (causeRepo !== null && (typeof causeRepo !== 'string' || !/^[^/\s]+\/[^/\s]+$/.test(causeRepo)))
|
|
187
191
|
throw new Error(`config: rescue.causeRepo must be owner/name or null, got ${JSON.stringify(causeRepo)}`)
|
|
192
|
+
if (!(typeof config.costFlagUsd === 'number' && config.costFlagUsd >= 0))
|
|
193
|
+
throw new Error(`config: costFlagUsd must be a number, at least 0, got ${JSON.stringify(config.costFlagUsd)}`)
|
|
188
194
|
if (config.engine !== 'legacy' && config.engine !== 'graph')
|
|
189
195
|
throw new Error(`config: engine must be "legacy" or "graph", got ${JSON.stringify(config.engine)}`)
|
|
190
196
|
const merging = config.repos.filter((r) => r.merge === 'green')
|
package/src/cost.mjs
ADDED
|
@@ -0,0 +1,161 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/**
|
|
3
|
+
* Per-issue cost accounting. When a worker exits, its stream-json log's `result` event (`total_cost_usd`, `usage`)
|
|
4
|
+
* is added to the issue's marker, per phase and overall, so the totals live on GitHub and survive a restart. Each
|
|
5
|
+
* run is also added to a per-repo, per-day ledger in SQLite for the heartbeat's daily figures; losing that file
|
|
6
|
+
* loses only the daily history.
|
|
7
|
+
*/
|
|
8
|
+
import { setMarker } from './issue-state.mjs'
|
|
9
|
+
import { resultEvent } from './tracing.mjs'
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* @typedef {{ usd: number, input: number, output: number, cacheRead: number, cacheCreation: number, runs: number }} Tally
|
|
13
|
+
* @typedef {'triage' | 'task' | 'rescue'} CostPhase
|
|
14
|
+
* @typedef {{ total: Tally, triage: Tally, task: Tally, rescue: Tally, lastLog: string | null }} Cost
|
|
15
|
+
* `lastLog`: the stdout log of the last run counted, so a run is never counted twice.
|
|
16
|
+
* @typedef {{ day: string, triage: number, task: number, rescue: number, total: number }} CostDay
|
|
17
|
+
*/
|
|
18
|
+
|
|
19
|
+
export const COST_PHASES = /** @type {const} */ (['triage', 'task', 'rescue'])
|
|
20
|
+
|
|
21
|
+
/** A worker row's `phase` → the cost phase it counts under. */
|
|
22
|
+
const PHASE_OF = /** @type {Record<string, CostPhase>} */ ({ dispatch: 'triage', task: 'task', rescue: 'rescue' })
|
|
23
|
+
|
|
24
|
+
/** @returns {Tally} */
|
|
25
|
+
export const zeroTally = () => ({ usd: 0, input: 0, output: 0, cacheRead: 0, cacheCreation: 0, runs: 0 })
|
|
26
|
+
|
|
27
|
+
const num = (/** @type {unknown} */ v) => (typeof v === 'number' && Number.isFinite(v) && v > 0 ? v : 0)
|
|
28
|
+
|
|
29
|
+
/** @param {any} t @returns {Tally} */
|
|
30
|
+
const tallyOf = (t) => ({
|
|
31
|
+
usd: num(t?.usd),
|
|
32
|
+
input: num(t?.input),
|
|
33
|
+
output: num(t?.output),
|
|
34
|
+
cacheRead: num(t?.cacheRead),
|
|
35
|
+
cacheCreation: num(t?.cacheCreation),
|
|
36
|
+
runs: num(t?.runs),
|
|
37
|
+
})
|
|
38
|
+
|
|
39
|
+
/** @param {Tally} a @param {Tally} b @returns {Tally} */
|
|
40
|
+
const plus = (a, b) => ({
|
|
41
|
+
usd: Math.round((a.usd + b.usd) * 1e6) / 1e6,
|
|
42
|
+
input: a.input + b.input,
|
|
43
|
+
output: a.output + b.output,
|
|
44
|
+
cacheRead: a.cacheRead + b.cacheRead,
|
|
45
|
+
cacheCreation: a.cacheCreation + b.cacheCreation,
|
|
46
|
+
runs: a.runs + b.runs,
|
|
47
|
+
})
|
|
48
|
+
|
|
49
|
+
/**
|
|
50
|
+
* A marker's cost, zeros when it has none (a marker from before cost accounting) or it is malformed.
|
|
51
|
+
* @param {{ cost?: unknown } | null | undefined} marker @returns {Cost}
|
|
52
|
+
*/
|
|
53
|
+
export function costOf(marker) {
|
|
54
|
+
const c = /** @type {any} */ (marker?.cost)
|
|
55
|
+
return {
|
|
56
|
+
total: tallyOf(c?.total),
|
|
57
|
+
triage: tallyOf(c?.triage),
|
|
58
|
+
task: tallyOf(c?.task),
|
|
59
|
+
rescue: tallyOf(c?.rescue),
|
|
60
|
+
lastLog: typeof c?.lastLog === 'string' ? c.lastLog : null,
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/**
|
|
65
|
+
* One run's cost and tokens from its stream-json log, or null when the log has no `result` event (a killed
|
|
66
|
+
* worker, a truncated log) or it carries no cost. Never throws.
|
|
67
|
+
* @param {string | Iterable<string | Record<string, any>>} stream @returns {Tally | null}
|
|
68
|
+
*/
|
|
69
|
+
export function runCost(stream) {
|
|
70
|
+
let r
|
|
71
|
+
try {
|
|
72
|
+
r = resultEvent(stream)
|
|
73
|
+
} catch {
|
|
74
|
+
return null
|
|
75
|
+
}
|
|
76
|
+
if (!r || typeof r.total_cost_usd !== 'number' || !Number.isFinite(r.total_cost_usd)) return null
|
|
77
|
+
const u = r.usage ?? {}
|
|
78
|
+
return {
|
|
79
|
+
usd: num(r.total_cost_usd),
|
|
80
|
+
input: num(u.input_tokens),
|
|
81
|
+
output: num(u.output_tokens),
|
|
82
|
+
cacheRead: num(u.cache_read_input_tokens),
|
|
83
|
+
cacheCreation: num(u.cache_creation_input_tokens),
|
|
84
|
+
runs: 1,
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/**
|
|
89
|
+
* `cost` with one more run added to `phase` and the total. A run whose log was the last one counted is not added
|
|
90
|
+
* again.
|
|
91
|
+
* @param {unknown} cost a marker's `cost`, possibly missing @param {CostPhase} phase @param {Tally} run
|
|
92
|
+
* @param {string | null} log @returns {Cost}
|
|
93
|
+
*/
|
|
94
|
+
export function addCost(cost, phase, run, log) {
|
|
95
|
+
const c = costOf({ cost })
|
|
96
|
+
if (log && c.lastLog === log) return c
|
|
97
|
+
return { ...c, total: plus(c.total, run), [phase]: plus(c[phase], run), lastLog: log ?? c.lastLog }
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
/** The UTC day of a time, `YYYY-MM-DD`. @param {number} ms */
|
|
101
|
+
export const dayOf = (ms) => new Date(ms).toISOString().slice(0, 10)
|
|
102
|
+
|
|
103
|
+
/**
|
|
104
|
+
* A finished worker's cost onto its issue's marker (and the day's ledger). Nothing happens when the worker phase
|
|
105
|
+
* is not counted, the log has no `result` (a killed worker), or this run was already counted. Never throws:
|
|
106
|
+
* accounting must not fail a tick.
|
|
107
|
+
*
|
|
108
|
+
* @param {{ config: import('./config.mjs').Config, github: import('./github.mjs').GitHub,
|
|
109
|
+
* spawner: { readLog(path: string): string },
|
|
110
|
+
* store: { addCost?(repo: string, day: string, phase: string, run: Tally): void },
|
|
111
|
+
* log: (level: 'info' | 'warn' | 'error', msg: string, where?: { repo?: string, issue?: number }) => void }} deps
|
|
112
|
+
* @param {import('./github.mjs').Issue | null | undefined} issue
|
|
113
|
+
* @param {{ repo: string, issue: number, phase: string, stdout: string }} w @param {number} now
|
|
114
|
+
* @returns {Promise<Cost | null>} the marker's new cost, when one was recorded
|
|
115
|
+
*/
|
|
116
|
+
export async function recordCost(deps, issue, w, now) {
|
|
117
|
+
const phase = PHASE_OF[w.phase]
|
|
118
|
+
if (!phase) return null
|
|
119
|
+
try {
|
|
120
|
+
const run = runCost(deps.spawner.readLog(w.stdout))
|
|
121
|
+
if (!run) return null
|
|
122
|
+
const marker = issue?.stateComment?.marker
|
|
123
|
+
if (marker && costOf(marker).lastLog === w.stdout) return null
|
|
124
|
+
let cost = null
|
|
125
|
+
if (issue && marker) {
|
|
126
|
+
cost = addCost(marker.cost, phase, run, w.stdout)
|
|
127
|
+
await setMarker(deps, w.repo, issue, { ...marker, cost })
|
|
128
|
+
}
|
|
129
|
+
deps.store.addCost?.(w.repo, dayOf(now), phase, run)
|
|
130
|
+
return cost
|
|
131
|
+
} catch (err) {
|
|
132
|
+
deps.log('warn', `cost: ${/** @type {Error} */ (err)?.message ?? err}`, { repo: w.repo, issue: w.issue })
|
|
133
|
+
return null
|
|
134
|
+
}
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
/**
|
|
138
|
+
* Whether an issue's task spend is over `flagUsd` with no merged PR for it: a candidate for a human session.
|
|
139
|
+
* `flagUsd` 0 switches the flag off.
|
|
140
|
+
* @param {Cost} cost @param {number} flagUsd @param {boolean} merged
|
|
141
|
+
*/
|
|
142
|
+
export const overBudget = (cost, flagUsd, merged) => flagUsd > 0 && cost.task.usd > flagUsd && !merged
|
|
143
|
+
|
|
144
|
+
/**
|
|
145
|
+
* Ledger rows → one entry per day (oldest first), each phase's spend and the total.
|
|
146
|
+
* @param {{ day: string, phase: string, usd: number }[]} rows @returns {CostDay[]}
|
|
147
|
+
*/
|
|
148
|
+
export function costDays(rows) {
|
|
149
|
+
/** @type {Map<string, CostDay>} */
|
|
150
|
+
const days = new Map()
|
|
151
|
+
for (const r of rows) {
|
|
152
|
+
const d = days.get(r.day) ?? { day: r.day, triage: 0, task: 0, rescue: 0, total: 0 }
|
|
153
|
+
const usd = num(r.usd)
|
|
154
|
+
if (r.phase === 'triage' || r.phase === 'task' || r.phase === 'rescue') d[r.phase] = round(d[r.phase] + usd)
|
|
155
|
+
d.total = round(d.total + usd)
|
|
156
|
+
days.set(r.day, d)
|
|
157
|
+
}
|
|
158
|
+
return [...days.values()].sort((a, b) => a.day.localeCompare(b.day))
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
const round = (/** @type {number} */ n) => Math.round(n * 1e6) / 1e6
|
package/src/graphs/dispatch.mjs
CHANGED
|
@@ -18,6 +18,7 @@ import { readTriage, TriageResult, triageComment, triagePrompt } from '../phases
|
|
|
18
18
|
import { humanReplies, quoted, replyContextOf } from '../human-reply.mjs'
|
|
19
19
|
import { traceWorker } from '../tracing.mjs'
|
|
20
20
|
import { fingerprintOf } from '../triage-inputs.mjs'
|
|
21
|
+
import { recordCost } from '../cost.mjs'
|
|
21
22
|
import { stopOverdue } from '../workers.mjs'
|
|
22
23
|
import { task } from './task.mjs'
|
|
23
24
|
|
|
@@ -278,6 +279,7 @@ const triaging = declare(
|
|
|
278
279
|
)
|
|
279
280
|
return { ready: false }
|
|
280
281
|
if (!snap) return { ready: false }
|
|
282
|
+
await recordCost(ctx, issue, w, ctx.tick.now)
|
|
281
283
|
if (issue?.state !== 'OPEN' || !issue.labels.includes(Q.intake)) {
|
|
282
284
|
ctx.log('info', 'triage worker gone; issue is closed or left q:intake', { repo: w.repo, issue: w.issue })
|
|
283
285
|
return { ready: true, output: { exit: 'gone' } }
|
package/src/graphs/rescue.mjs
CHANGED
|
@@ -32,6 +32,7 @@ import {
|
|
|
32
32
|
seenAgain,
|
|
33
33
|
} from '../phases/rescue.mjs'
|
|
34
34
|
import { traceWorker } from '../tracing.mjs'
|
|
35
|
+
import { recordCost } from '../cost.mjs'
|
|
35
36
|
import { stopOverdue } from '../workers.mjs'
|
|
36
37
|
import { dispatch } from './dispatch.mjs'
|
|
37
38
|
import { issueOf, task } from './task.mjs'
|
|
@@ -308,6 +309,7 @@ const diagnosing = defineWait({
|
|
|
308
309
|
return { ready: false }
|
|
309
310
|
if (!snap || !issue) return { ready: false }
|
|
310
311
|
traceWorker(ctx, w)
|
|
312
|
+
await recordCost(ctx, issue, w, ctx.tick.now)
|
|
311
313
|
if (!stuck) {
|
|
312
314
|
ctx.log('info', 'rescue: no longer stuck; verdict not applied', { repo: slug, issue: number })
|
|
313
315
|
return { ready: true, output: { exit: 'resolved' } }
|
package/src/graphs/task.mjs
CHANGED
|
@@ -17,6 +17,7 @@ import { withHistory } from '../marker.mjs'
|
|
|
17
17
|
import { linkIssue, prFor, taskExit, taskPrompt } from '../phases/task.mjs'
|
|
18
18
|
import { PREPARE_FAILED_LINE } from '../prepare.mjs'
|
|
19
19
|
import { traceWorker } from '../tracing.mjs'
|
|
20
|
+
import { recordCost } from '../cost.mjs'
|
|
20
21
|
import { baseRed, isRepair } from '../base-health.mjs'
|
|
21
22
|
import { stopOverdue } from '../workers.mjs'
|
|
22
23
|
|
|
@@ -277,6 +278,7 @@ const workerRunning = declare(
|
|
|
277
278
|
)
|
|
278
279
|
return { ready: false }
|
|
279
280
|
if (!snap) return { ready: false } // a dead worker waits for a readable snapshot
|
|
281
|
+
await recordCost(ctx, issue, w, ctx.tick.now)
|
|
280
282
|
if (issue?.state !== 'OPEN') {
|
|
281
283
|
ctx.log('info', 'worker gone; issue is closed or no longer queued', { repo: w.repo, issue: w.issue })
|
|
282
284
|
return { ready: true, output: { exit: 'gone' } }
|
package/src/issue-state.mjs
CHANGED
|
@@ -28,6 +28,9 @@ export async function setMarker(deps, slug, issue, marker) {
|
|
|
28
28
|
// takes it, so later ticks parent on the same trace. One that already has a trace keeps it.
|
|
29
29
|
const tp = /** @type {any} */ (deps).subject?.traceparent
|
|
30
30
|
if (tp && !marker.traceparent) marker = { ...marker, traceparent: tp }
|
|
31
|
+
// Cost accumulates for the life of the issue (cost.mjs): a write that starts a fresh marker keeps it.
|
|
32
|
+
const cost = issue.stateComment?.marker?.cost
|
|
33
|
+
if (cost && !marker.cost) marker = { ...marker, cost }
|
|
31
34
|
const maxAttempts = deps.config.phases[marker.phase]?.maxAttempts
|
|
32
35
|
const id = await deps.github.writeState(
|
|
33
36
|
slug,
|
package/src/marker.mjs
CHANGED
|
@@ -51,7 +51,10 @@
|
|
|
51
51
|
* reply?: { since: string, question: string | null } | null,
|
|
52
52
|
* ask?: import('./ask.mjs').AskT | null,
|
|
53
53
|
* fingerprint?: { hash: string, at: string } | null,
|
|
54
|
+
* cost?: import('./cost.mjs').Cost | null,
|
|
54
55
|
* }} Marker
|
|
56
|
+
* `cost`: what this issue's workers have cost so far, per phase (`triage`, `task`, `rescue`) and in `total`, from
|
|
57
|
+
* each worker's `result` event (cost.mjs). Every write keeps it; a marker without one counts from zero.
|
|
55
58
|
* `mergeRefusal`: the last `graft pr merge` cambium was refused (PR graph `merge`), and on which head.
|
|
56
59
|
* `reworks`: how many times the PR graph sent this issue's PR back to a worker.
|
|
57
60
|
* `defers`, `deferUntil`, `deferReason`: the dispatch graph's `deferred` node: how many times triage deferred, until
|
|
@@ -191,6 +194,7 @@ export function renderMarker(input, { maxAttempts } = {}) {
|
|
|
191
194
|
(marker.pr ? ` · PR #${marker.pr}` : '') +
|
|
192
195
|
(marker.host ? ` · host \`${marker.host}\`` : '') +
|
|
193
196
|
(marker.start ? ` · started ${marker.start}` : '') +
|
|
197
|
+
(marker.cost?.total?.usd ? ` · cost $${marker.cost.total.usd.toFixed(2)}` : '') +
|
|
194
198
|
(marker.node === 'deferred' && marker.deferUntil ? ` · re-triage after ${marker.deferUntil}` : '') +
|
|
195
199
|
(marker.node === 'blocked' && marker.blockedBy?.length
|
|
196
200
|
? ` · re-triage once ${marker.blockedBy.map((n) => `#${n}`).join(', ')} close`
|
package/src/store.mjs
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
// @ts-check
|
|
2
2
|
/**
|
|
3
|
-
* Local bookkeeping in SQLite: live worker processes
|
|
3
|
+
* Local bookkeeping in SQLite: live worker processes, the daemon's mode, and the per-day cost ledger (cost.mjs).
|
|
4
4
|
* Never a queue — GitHub labels are the queue. Losing this file costs at most
|
|
5
5
|
* a requeue (the sweep returns `running` issues with no tracked worker).
|
|
6
6
|
*/
|
|
@@ -28,6 +28,12 @@ CREATE TABLE IF NOT EXISTS workers (
|
|
|
28
28
|
PRIMARY KEY (repo, issue)
|
|
29
29
|
);
|
|
30
30
|
CREATE TABLE IF NOT EXISTS control (key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
|
31
|
+
CREATE TABLE IF NOT EXISTS costs (
|
|
32
|
+
repo TEXT NOT NULL, day TEXT NOT NULL, phase TEXT NOT NULL, usd REAL NOT NULL DEFAULT 0,
|
|
33
|
+
input INTEGER NOT NULL DEFAULT 0, output INTEGER NOT NULL DEFAULT 0, cache_read INTEGER NOT NULL DEFAULT 0,
|
|
34
|
+
cache_creation INTEGER NOT NULL DEFAULT 0, runs INTEGER NOT NULL DEFAULT 0,
|
|
35
|
+
PRIMARY KEY (repo, day, phase)
|
|
36
|
+
);
|
|
31
37
|
`
|
|
32
38
|
|
|
33
39
|
/** @param {any} r @returns {Worker} */
|
|
@@ -99,6 +105,34 @@ export function openStore(path) {
|
|
|
99
105
|
removeWorker(repo, issue) {
|
|
100
106
|
db.prepare('DELETE FROM workers WHERE repo = ? AND issue = ?').run(repo, issue)
|
|
101
107
|
},
|
|
108
|
+
/**
|
|
109
|
+
* Add one worker run to a repo's ledger for `day` (`YYYY-MM-DD`, UTC) and phase.
|
|
110
|
+
* @param {string} repo @param {string} day @param {string} phase @param {import('./cost.mjs').Tally} run
|
|
111
|
+
*/
|
|
112
|
+
addCost(repo, day, phase, run) {
|
|
113
|
+
db.prepare(
|
|
114
|
+
`INSERT INTO costs (repo, day, phase, usd, input, output, cache_read, cache_creation, runs)
|
|
115
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
116
|
+
ON CONFLICT(repo, day, phase) DO UPDATE SET usd = usd + excluded.usd, input = input + excluded.input,
|
|
117
|
+
output = output + excluded.output, cache_read = cache_read + excluded.cache_read,
|
|
118
|
+
cache_creation = cache_creation + excluded.cache_creation, runs = runs + excluded.runs`,
|
|
119
|
+
).run(repo, day, phase, run.usd, run.input, run.output, run.cacheRead, run.cacheCreation, run.runs)
|
|
120
|
+
},
|
|
121
|
+
/**
|
|
122
|
+
* The ledger from `since` (`YYYY-MM-DD`) on, oldest first.
|
|
123
|
+
* @param {string} since @returns {{ repo: string, day: string, phase: string, usd: number, runs: number }[]}
|
|
124
|
+
*/
|
|
125
|
+
costs: (since) =>
|
|
126
|
+
db
|
|
127
|
+
.prepare('SELECT repo, day, phase, usd, runs FROM costs WHERE day >= ? ORDER BY day, repo, phase')
|
|
128
|
+
.all(since)
|
|
129
|
+
.map((/** @type {any} */ r) => ({
|
|
130
|
+
repo: r.repo,
|
|
131
|
+
day: r.day,
|
|
132
|
+
phase: r.phase,
|
|
133
|
+
usd: Number(r.usd),
|
|
134
|
+
runs: Number(r.runs),
|
|
135
|
+
})),
|
|
102
136
|
/** @returns {Mode} */
|
|
103
137
|
mode: () => /** @type {Mode} */ (get('mode') ?? 'running'),
|
|
104
138
|
/** @param {Mode} mode */
|
package/src/tick.mjs
CHANGED
|
@@ -20,6 +20,7 @@ import { DISPATCH_NOT_BUILT, dispatchNotBuilt } from './phases/dispatch.mjs'
|
|
|
20
20
|
import { isRepair } from './base-health.mjs'
|
|
21
21
|
import { stopOverdue } from './workers.mjs'
|
|
22
22
|
import { syncBriefs } from './briefs.mjs'
|
|
23
|
+
import { costDays, costOf, dayOf, overBudget, recordCost } from './cost.mjs'
|
|
23
24
|
|
|
24
25
|
/**
|
|
25
26
|
* @typedef {import('./github.mjs').Issue} Issue
|
|
@@ -272,6 +273,7 @@ export async function finish(deps, ctx, w, snap) {
|
|
|
272
273
|
const t = await deps.spawner.cleanup({ repo, worktree: w.worktree, branch: w.branch })
|
|
273
274
|
if (!t.removed) deps.log('warn', `kept worktree ${w.worktree}: ${t.reason}`, { repo: w.repo, issue: w.issue })
|
|
274
275
|
}
|
|
276
|
+
await recordCost(deps, issue, w, ctx.now)
|
|
275
277
|
if (issue?.state !== 'OPEN') {
|
|
276
278
|
deps.log('info', 'worker gone; issue is closed or no longer queued', { repo: w.repo, issue: w.issue })
|
|
277
279
|
return done()
|
|
@@ -650,6 +652,7 @@ async function release(deps, ctx) {
|
|
|
650
652
|
* repos: {
|
|
651
653
|
* slug: string, ok: boolean, error: string | null, counts: Counts | null,
|
|
652
654
|
* base: import('./base-health.mjs').BaseHealth | null,
|
|
655
|
+
* cost: { issues: IssueCost[], days: import('./cost.mjs').CostDay[] },
|
|
653
656
|
* }[],
|
|
654
657
|
* workers: {
|
|
655
658
|
* repo: string, issue: number, phase: string, pid: number, alive: boolean, attempt: number, branch: string,
|
|
@@ -659,8 +662,47 @@ async function release(deps, ctx) {
|
|
|
659
662
|
* events: Event[],
|
|
660
663
|
* engine: 'legacy' | 'graph',
|
|
661
664
|
* graphs: import('./graph/describe.mjs').GraphDescription[],
|
|
665
|
+
* costFlagUsd: number,
|
|
662
666
|
* }} HeartbeatState
|
|
667
|
+
* @typedef {{
|
|
668
|
+
* number: number, title: string, url: string, labels: string[], cost: import('./cost.mjs').Cost, flagged: boolean,
|
|
669
|
+
* }} IssueCost
|
|
670
|
+
* An active (`q:intake`, `q:task`, `running`) or `q:human` issue that has cost something. `flagged`: its task
|
|
671
|
+
* spend is over `costFlagUsd` and it has no merged PR (cost.mjs `overBudget`).
|
|
672
|
+
*/
|
|
673
|
+
|
|
674
|
+
/** Days of the per-repo cost ledger the heartbeat carries, today included. */
|
|
675
|
+
export const COST_DAYS = 14
|
|
676
|
+
|
|
677
|
+
/**
|
|
678
|
+
* The repo's issues worth a cost line, most expensive first.
|
|
679
|
+
* @param {Snapshot} snap @param {number} flagUsd @returns {IssueCost[]}
|
|
663
680
|
*/
|
|
681
|
+
export function issueCosts(snap, flagUsd) {
|
|
682
|
+
/** @type {IssueCost[]} */
|
|
683
|
+
const out = []
|
|
684
|
+
for (const issue of snap.issues.values()) {
|
|
685
|
+
if (issue.state !== 'OPEN') continue
|
|
686
|
+
const q = queueOf(issue.labels)
|
|
687
|
+
if (q !== Q.intake && q !== Q.task && q !== Q.human && !issue.labels.includes(RUNNING)) continue
|
|
688
|
+
const marker = issue.stateComment?.marker
|
|
689
|
+
const cost = costOf(marker)
|
|
690
|
+
if (!cost.total.runs) continue
|
|
691
|
+
const merged = snap.prs.some(
|
|
692
|
+
(p) =>
|
|
693
|
+
p.state === 'MERGED' && (p.closes.includes(issue.number) || (marker?.pr != null && p.number === marker.pr)),
|
|
694
|
+
)
|
|
695
|
+
out.push({
|
|
696
|
+
number: issue.number,
|
|
697
|
+
title: issue.title,
|
|
698
|
+
url: issue.url,
|
|
699
|
+
labels: issue.labels,
|
|
700
|
+
cost,
|
|
701
|
+
flagged: overBudget(cost, flagUsd, merged),
|
|
702
|
+
})
|
|
703
|
+
}
|
|
704
|
+
return out.sort((a, b) => b.cost.total.usd - a.cost.total.usd || a.number - b.number)
|
|
705
|
+
}
|
|
664
706
|
|
|
665
707
|
/** @param {Snapshot} snap @returns {Counts} */
|
|
666
708
|
function countsOf(snap) {
|
|
@@ -694,6 +736,8 @@ const graphDescriptions = () => {
|
|
|
694
736
|
export function heartbeatState(deps, ctx) {
|
|
695
737
|
const { config } = deps
|
|
696
738
|
const finished = deps.clock.now()
|
|
739
|
+
const since = dayOf(ctx.now - (COST_DAYS - 1) * 86_400_000)
|
|
740
|
+
const ledger = deps.store.costs(since)
|
|
697
741
|
const repos = config.repos.map((r) => {
|
|
698
742
|
const snap = ctx.snaps.get(r.slug)
|
|
699
743
|
return {
|
|
@@ -702,6 +746,10 @@ export function heartbeatState(deps, ctx) {
|
|
|
702
746
|
error: ctx.readErrors.get(r.slug) ?? null,
|
|
703
747
|
counts: snap ? countsOf(snap) : null,
|
|
704
748
|
base: snap?.base ?? null,
|
|
749
|
+
cost: {
|
|
750
|
+
issues: snap ? issueCosts(snap, config.costFlagUsd) : [],
|
|
751
|
+
days: costDays(ledger.filter((row) => row.repo === r.slug)),
|
|
752
|
+
},
|
|
705
753
|
}
|
|
706
754
|
})
|
|
707
755
|
const total = { intake: 0, task: 0, human: 0, promote: 0, running: 0, review: 0 }
|
|
@@ -746,6 +794,7 @@ export function heartbeatState(deps, ctx) {
|
|
|
746
794
|
events: deps.events?.() ?? [],
|
|
747
795
|
engine: config.engine,
|
|
748
796
|
graphs: graphDescriptions(),
|
|
797
|
+
costFlagUsd: config.costFlagUsd,
|
|
749
798
|
}
|
|
750
799
|
}
|
|
751
800
|
|
package/src/tracing.mjs
CHANGED
|
@@ -391,6 +391,17 @@ function* events(stream) {
|
|
|
391
391
|
}
|
|
392
392
|
}
|
|
393
393
|
|
|
394
|
+
/**
|
|
395
|
+
* The last `result` event in a worker's stream-json log (cost, usage, turns), or null when there is none: a killed
|
|
396
|
+
* worker or a truncated log. Never throws.
|
|
397
|
+
* @param {string | Iterable<string | Record<string, any>>} stream @returns {Record<string, any> | null}
|
|
398
|
+
*/
|
|
399
|
+
export function resultEvent(stream) {
|
|
400
|
+
let result = null
|
|
401
|
+
for (const e of events(stream)) if (e.type === 'result') result = e
|
|
402
|
+
return result
|
|
403
|
+
}
|
|
404
|
+
|
|
394
405
|
/**
|
|
395
406
|
* @typedef {{ id: string, name: string, input: any, start: number, end: number | null, error: boolean, parentToolId: string | null }} ToolUse
|
|
396
407
|
* @typedef {{ id: string, toolUseId: string | null, type?: string, description?: string, start: number, end: number | null, status?: string }} BgTask
|