@oakgreencc/cambium 0.0.14 → 0.0.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +49 -4
- package/package.json +1 -1
- package/src/cli.mjs +25 -2
- package/src/config.mjs +10 -2
- package/src/daemon.mjs +2 -0
- package/src/dispatch-engine.mjs +13 -4
- package/src/graph-engine.mjs +11 -1
- package/src/graphs/dispatch.mjs +5 -4
- package/src/graphs/release.mjs +18 -5
- package/src/graphs/task.mjs +8 -6
- package/src/issue-state.mjs +4 -0
- package/src/store.mjs +24 -5
- package/src/tick.mjs +1 -0
- package/src/tracing.mjs +56 -0
- package/src/workers.mjs +6 -1
package/README.md
CHANGED
|
@@ -80,7 +80,7 @@ A worker is `claude -p --output-format stream-json --verbose --max-turns <maxTur
|
|
|
80
80
|
- in its own worktree, by the target repo's [worktree convention](#target-repo-convention);
|
|
81
81
|
- behind the prepare step: `node src/prepare.mjs -- claude …` runs install-fresh, hydrate and the repo's `prepare` commands in the worktree, writing to the worker's stderr log, then becomes the worker. It runs in the detached process so a long install never blocks a tick;
|
|
82
82
|
- detached, leading its own process group (a kill takes its children), stdin from the prompt file;
|
|
83
|
-
- with `CAMBIUM_REPO`, `CAMBIUM_ISSUE`, `CAMBIUM_PHASE`, `CAMBIUM_BRANCH` set,
|
|
83
|
+
- with `CAMBIUM_REPO`, `CAMBIUM_ISSUE`, `CAMBIUM_PHASE`, `CAMBIUM_BRANCH` set, `GRAFT_APP=<worker.graftApp>` when configured, then `worker.env`, then `TRACEPARENT` (the starting node's span) when [tracing](#tracing) is on;
|
|
84
84
|
- with its prompt, stdout and stderr at `<paths.logs>/<owner>__<name>/issue-<n>/a<attempt>-<stamp>.{prompt.md,out.log,err.log}`, keeping the newest `logs.keepPerIssue` runs per issue.
|
|
85
85
|
|
|
86
86
|
The prompt states the four-point contract (task issue, `pnpm verify`, `graft publish <branch> … --closes <n>`, `graft ci wait`), the attempt, the existing PR if any (`graft advance` onto it), the last failure, the issue's history, the issue body, and the worktree's `AGENTS.md`.
|
|
@@ -111,10 +111,12 @@ One JSON file. Everything but `repos` is optional; these are the defaults:
|
|
|
111
111
|
},
|
|
112
112
|
"logs": { "keepPerIssue": 20, "daemonMaxBytes": 10485760 },
|
|
113
113
|
"graft": { "command": "graft", "app": null, "botLogin": null },
|
|
114
|
-
"worker": { "command": "claude", "args": [], "graftApp": null, "fetch": true }
|
|
114
|
+
"worker": { "command": "claude", "args": [], "graftApp": null, "fetch": true, "env": {} }
|
|
115
115
|
}
|
|
116
116
|
```
|
|
117
117
|
|
|
118
|
+
- `worker.env` is extra environment for every worker (string values only), e.g. Claude Code's own telemetry ([Tracing](#claude-codes-own-spans-under-the-worker)).
|
|
119
|
+
|
|
118
120
|
- The daemon re-reads the file before every tick, so caps, limits, `tickSeconds`, `repos`, priority labels, `release` and `worker` take effect without a restart (logged as `config reloaded: <keys>`). `host`, `engine`, `paths`, `logs` and `graft` are baked in at start: a change to them logs `restart cambium to apply` once and is not applied. A file that does not load, or has no `repos`, keeps the running config and logs one warning until it is fixed.
|
|
119
121
|
- `repos[].path` is a local clone; worktrees are added from it and `graft` runs in it.
|
|
120
122
|
- `graft.app` is the App cambium acts as (`GRAFT_APP` for its own `graft` calls); `worker.graftApp` the one workers act as. Neither leaks into the other.
|
|
@@ -452,6 +454,16 @@ Why not a shared package: cambium is published to npm, so it can only depend on
|
|
|
452
454
|
|
|
453
455
|
Node and wait spans carry `cambium.graph`, `cambium.graph.version`, `cambium.node`, `cambium.attempt`, `cambium.repo`, `cambium.issue.number`, `cambium.edge`, `cambium.outcome`.
|
|
454
456
|
|
|
457
|
+
### What is traced
|
|
458
|
+
|
|
459
|
+
With `engine: "graph"`, every graph run is traced: the `task` and `dispatch` graphs per issue and both release graphs (the legacy stage tick is not). [canopy](../canopy)'s Traces tab shows them from the local collector's file.
|
|
460
|
+
|
|
461
|
+
- **Lifecycle.** `cambium start` and `cambium run` create the telemetry from their environment (a no-op without OTLP config), flush it after every tick and shut it down on exit.
|
|
462
|
+
- **One trace per issue, across ticks.** Each turn (`drive`, `driveDispatch`, release `drive`) runs `advance` with `turnTracer`'s `instrument` (`advanceHooks` underneath). The first span of an issue starts its `cambium.issue` root, a child of mycelium's intake span when the body carries one; the marker written in that turn takes the root's traceparent (`setMarker` fills `traceparent` from the turn's subject when the marker has none), and every later tick parents on it. Requeues and later attempts stay in the trace, and the dispatch → task hand-off carries it, so one issue reads intake → triage → each task attempt → CI → close-out.
|
|
463
|
+
- **Release.** A release-prepare run is a root of its own; the promotion PR it opens or refreshes takes that trace into its marker when it has none, and its release-promote turns (`merge`, `merging`) continue it, with `cambium.issue.number` the PR's number.
|
|
464
|
+
- **Workers.** start-worker and start-triage start their worker with `TRACEPARENT` set to their own `cambium.node` span and record it on the worker row (`workers.traceparent` in SQLite; older databases gain the column on open). When the worker is found exited, its stdout log becomes `claude.*` spans under that span ([below](#worker-spans-from-stream-json)); a log that cannot be read is a warning, never a failed tick.
|
|
465
|
+
- **Off is unchanged.** Without OTLP config there is no `instrument`, no `TRACEPARENT`, and markers keep `traceparent: null`; the legacy/graph differential test covers that the moves are identical.
|
|
466
|
+
|
|
455
467
|
```js
|
|
456
468
|
import { advanceHooks, createTelemetry } from './tracing.mjs' // inside cambium
|
|
457
469
|
|
|
@@ -495,7 +507,40 @@ telemetry.workerSpans(stream, parent) // same, with cambium's tracer
|
|
|
495
507
|
- `claude.task`: each background task, `task_started` → `task_notification` (or `task_updated.patch.end_time`), under the tool that started it. These events have no timestamp, so their times are those of the latest timestamped event.
|
|
496
508
|
- Malformed lines are counted (`malformed`) and skipped; `system/thinking_tokens`, `tool_progress`, hooks and rate-limit events are ignored.
|
|
497
509
|
|
|
498
|
-
###
|
|
510
|
+
### Claude Code's own spans under the worker
|
|
511
|
+
|
|
512
|
+
**`claude -p` honours `TRACEPARENT`: verified live** (Claude Code 2.1.327, 2026-11-18). A `claude -p` run with a Bash call, given `TRACEPARENT=00-<trace>-<span>-01` and the variables below, exported into `<trace>`:
|
|
513
|
+
|
|
514
|
+
```
|
|
515
|
+
<the TRACEPARENT span>
|
|
516
|
+
└ claude_code.interaction parent.source=env
|
|
517
|
+
├ claude_code.llm_request model, input/output/cache tokens, ttft_ms, stop_reason
|
|
518
|
+
├ claude_code.tool tool_name=Bash
|
|
519
|
+
│ ├ claude_code.tool.blocked_on_user
|
|
520
|
+
│ └ claude_code.tool.execution
|
|
521
|
+
└ claude_code.llm_request
|
|
522
|
+
```
|
|
523
|
+
|
|
524
|
+
Workers already get `TRACEPARENT`, so turning these on is config only:
|
|
525
|
+
|
|
526
|
+
```jsonc
|
|
527
|
+
"worker": {
|
|
528
|
+
"env": {
|
|
529
|
+
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
|
|
530
|
+
"CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
|
|
531
|
+
"OTEL_TRACES_EXPORTER": "otlp",
|
|
532
|
+
"OTEL_METRICS_EXPORTER": "none",
|
|
533
|
+
"OTEL_LOGS_EXPORTER": "none",
|
|
534
|
+
"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
|
|
535
|
+
"OTEL_EXPORTER_OTLP_ENDPOINT": "http://127.0.0.1:4318"
|
|
536
|
+
}
|
|
537
|
+
}
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
- They complement the stream-json spans rather than replace them: they carry per-request timing and tokens but no cost, and need telemetry on in the worker; `claude.session` carries the session's `claude.cost_usd` and works without it. With both on, a worker has `claude.*` and `claude_code.*` spans side by side under start-worker.
|
|
541
|
+
- **Identity in spans.** Every `claude_code.*` span carries `user.email`, `user.account_uuid`, `user.account_id` and `organization.id`. Fine for the local collector file; mind it before exporting elsewhere (strip them in the collector with an `attributes` processor).
|
|
542
|
+
|
|
543
|
+
### Does `claude -p` export traces that honour `TRACEPARENT`? (the docs)
|
|
499
544
|
|
|
500
545
|
**Yes, opt-in and beta**; cambium does not depend on it (worker spans come from stream-json above). Per the [Claude Code monitoring docs](https://code.claude.com/docs/en/monitoring-usage) (checked with Claude Code 2.1.327; `claude --help` lists no telemetry flags, it is env-only):
|
|
501
546
|
|
|
@@ -504,7 +549,7 @@ telemetry.workerSpans(stream, parent) // same, with cambium's tracer
|
|
|
504
549
|
- Bash subprocesses it runs inherit a `TRACEPARENT` for the active tool span, so a `graft` call inside a worker could continue the trace.
|
|
505
550
|
- Prompts and tool inputs are redacted from spans unless `OTEL_LOG_USER_PROMPTS` / `OTEL_LOG_TOOL_DETAILS` / `OTEL_LOG_TOOL_CONTENT` are set.
|
|
506
551
|
|
|
507
|
-
|
|
552
|
+
Confirmed against a live run: see [Claude Code's own spans under the worker](#claude-codes-own-spans-under-the-worker).
|
|
508
553
|
|
|
509
554
|
### Local collector
|
|
510
555
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@oakgreencc/cambium",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.15",
|
|
4
4
|
"description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
|
|
5
5
|
"license": "AGPL-3.0-only",
|
|
6
6
|
"type": "module",
|
package/src/cli.mjs
CHANGED
|
@@ -14,6 +14,7 @@ import { plist, plistPath } from './launchd.mjs'
|
|
|
14
14
|
import { openStore } from './store.mjs'
|
|
15
15
|
import { finish, newCtx, startTask } from './tick.mjs'
|
|
16
16
|
import { runIssue } from './graph-engine.mjs'
|
|
17
|
+
import { createTelemetry } from './tracing.mjs'
|
|
17
18
|
|
|
18
19
|
export const VERSION = '0.0.0'
|
|
19
20
|
|
|
@@ -41,6 +42,24 @@ const flag = (argv, name) => {
|
|
|
41
42
|
/** @param {string[]} argv @param {string} name */
|
|
42
43
|
const has = (argv, name) => argv.includes(name)
|
|
43
44
|
|
|
45
|
+
/**
|
|
46
|
+
* Run `fn` with `deps.telemetry` set (from the OTEL_* vars in `io.env`, a no-op without them, unless the deps
|
|
47
|
+
* already carry one), shutting it down afterwards so queued spans are exported before the process exits.
|
|
48
|
+
*
|
|
49
|
+
* @template T
|
|
50
|
+
* @param {import('./tick.mjs').Deps} deps @param {IO} io @param {() => Promise<T>} fn
|
|
51
|
+
* @returns {Promise<T>}
|
|
52
|
+
*/
|
|
53
|
+
async function withTelemetry(deps, io, fn) {
|
|
54
|
+
const own = !deps.telemetry
|
|
55
|
+
if (own) deps.telemetry = await createTelemetry({ env: io.env })
|
|
56
|
+
try {
|
|
57
|
+
return await fn()
|
|
58
|
+
} finally {
|
|
59
|
+
if (own) await deps.telemetry?.shutdown().catch(() => {})
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
|
|
44
63
|
/**
|
|
45
64
|
* @typedef {{
|
|
46
65
|
* out: (line: string) => void,
|
|
@@ -73,7 +92,9 @@ export async function main(argv, io) {
|
|
|
73
92
|
if (!config.repos.length) return fail(io, `no repos in ${cfgPath}; see the README for the config file`)
|
|
74
93
|
const deps = depsFor(config)
|
|
75
94
|
acquireLock(deps.store, deps.procs, deps.pid ?? process.pid)
|
|
76
|
-
await
|
|
95
|
+
await withTelemetry(deps, io, () =>
|
|
96
|
+
runDaemon(deps, { signal: io.signal, once: has(rest, '--once'), reload: () => loadConfig(cfgPath) }),
|
|
97
|
+
)
|
|
77
98
|
return 0
|
|
78
99
|
}
|
|
79
100
|
case 'status':
|
|
@@ -169,7 +190,9 @@ async function runOne(config, argv, io, depsFor) {
|
|
|
169
190
|
)
|
|
170
191
|
const deps = depsFor(config)
|
|
171
192
|
if (config.engine === 'graph') {
|
|
172
|
-
const r = await
|
|
193
|
+
const r = await withTelemetry(deps, io, () =>
|
|
194
|
+
runIssue(deps, { graph, slug, number: n, follow, out: io.out, sleep: io.sleep, signal: io.signal }),
|
|
195
|
+
)
|
|
173
196
|
return r.ok ? 0 : fail(io, /** @type {string} */ (r.error))
|
|
174
197
|
}
|
|
175
198
|
if (deps.store.worker(slug, n)) return fail(io, `${slug}#${n} already has a tracked worker`)
|
package/src/config.mjs
CHANGED
|
@@ -27,7 +27,7 @@ import { join, resolve } from 'node:path'
|
|
|
27
27
|
* paths: { home: string, db: string, heartbeat: string, logs: string },
|
|
28
28
|
* logs: { keepPerIssue: number, daemonMaxBytes: number },
|
|
29
29
|
* graft: { command: string, app: string | null, botLogin: string | null },
|
|
30
|
-
* worker: { command: string, args: string[], graftApp: string | null, fetch: boolean },
|
|
30
|
+
* worker: { command: string, args: string[], graftApp: string | null, fetch: boolean, env: Record<string, string> },
|
|
31
31
|
* }} Config
|
|
32
32
|
*/
|
|
33
33
|
|
|
@@ -56,7 +56,7 @@ export function defaults(home = DEFAULT_HOME) {
|
|
|
56
56
|
},
|
|
57
57
|
logs: { keepPerIssue: 20, daemonMaxBytes: 10 * 1024 * 1024 },
|
|
58
58
|
graft: { command: 'graft', app: null, botLogin: null },
|
|
59
|
-
worker: { command: 'claude', args: [], graftApp: null, fetch: true },
|
|
59
|
+
worker: { command: 'claude', args: [], graftApp: null, fetch: true, env: {} },
|
|
60
60
|
}
|
|
61
61
|
}
|
|
62
62
|
|
|
@@ -103,6 +103,14 @@ export function resolveConfig(raw = {}) {
|
|
|
103
103
|
if (!/^\d{2}:\d{2}$/.test(release.time)) throw new Error(`config: repo ${r.slug} release.time must be HH:MM`)
|
|
104
104
|
return { slug: r.slug, path: untilde(r.path), baseBranch, release }
|
|
105
105
|
})
|
|
106
|
+
const wenv = config.worker.env
|
|
107
|
+
if (
|
|
108
|
+
!wenv ||
|
|
109
|
+
typeof wenv !== 'object' ||
|
|
110
|
+
Array.isArray(wenv) ||
|
|
111
|
+
Object.values(wenv).some((v) => typeof v !== 'string')
|
|
112
|
+
)
|
|
113
|
+
throw new Error('config: worker.env must be an object of string values')
|
|
106
114
|
if (config.engine !== 'legacy' && config.engine !== 'graph')
|
|
107
115
|
throw new Error(`config: engine must be "legacy" or "graph", got ${JSON.stringify(config.engine)}`)
|
|
108
116
|
return config
|
package/src/daemon.mjs
CHANGED
|
@@ -134,6 +134,8 @@ export async function runDaemon(
|
|
|
134
134
|
} catch (err) {
|
|
135
135
|
deps.log('error', `tick ${n}: ${/** @type {Error} */ (err)?.message ?? err}`)
|
|
136
136
|
}
|
|
137
|
+
// Every span is ended within its tick; export them now rather than when the batch fills.
|
|
138
|
+
await deps.telemetry?.flush().catch((err) => deps.log('warn', `telemetry flush: ${err?.message ?? err}`))
|
|
137
139
|
if (once) break
|
|
138
140
|
try {
|
|
139
141
|
await wait(deps.config.tickSeconds * 1000, signal)
|
package/src/dispatch-engine.mjs
CHANGED
|
@@ -15,6 +15,7 @@ import { advance } from './graph/index.mjs'
|
|
|
15
15
|
import { dispatch, lostLeaseOf, nodeOf } from './graphs/dispatch.mjs'
|
|
16
16
|
import { byPriorityThenAge, Q, RUNNING } from './labels.mjs'
|
|
17
17
|
import { each, staleLeases } from './tick.mjs'
|
|
18
|
+
import { markerTrace, turnTracer } from './tracing.mjs'
|
|
18
19
|
|
|
19
20
|
/**
|
|
20
21
|
* @typedef {import('./tick.mjs').Deps} Deps
|
|
@@ -33,15 +34,23 @@ export const PHASE = 'dispatch'
|
|
|
33
34
|
*/
|
|
34
35
|
export async function driveDispatch(deps, ctx, subject, at) {
|
|
35
36
|
const c = { ...deps, tick: ctx, subject }
|
|
37
|
+
// A new issue's cambium.issue root continues mycelium's intake span from the body, so these node and wait spans
|
|
38
|
+
// join the intake trace (tracing.mjs `issueTrace`); the marker carries it on into the task graph.
|
|
39
|
+
const traced = turnTracer(deps.telemetry, subject, {
|
|
40
|
+
graph: dispatch.name,
|
|
41
|
+
graphVersion: dispatch.version,
|
|
42
|
+
repo: subject.slug,
|
|
43
|
+
issue: subject.number,
|
|
44
|
+
issueBody: subject.issue?.body ?? null,
|
|
45
|
+
...markerTrace(subject.issue?.stateComment?.marker),
|
|
46
|
+
})
|
|
36
47
|
/** @type {Transition[]} */
|
|
37
48
|
const transitions = []
|
|
38
49
|
let pos = at
|
|
39
50
|
for (;;) {
|
|
40
|
-
|
|
41
|
-
// here; the issue's cambium.issue root continues mycelium's intake span from the body, so these node and wait
|
|
42
|
-
// spans join the intake trace (tracing.mjs `issueTrace`).
|
|
43
|
-
const r = await advance(dispatch, pos, c, { maxTransitions: 1 })
|
|
51
|
+
const r = await advance(dispatch, pos, c, { maxTransitions: 1, instrument: traced.instrument })
|
|
44
52
|
if (r.error) throw r.error.cause instanceof Error ? r.error.cause : r.error
|
|
53
|
+
traced.sync()
|
|
45
54
|
transitions.push(...r.transitions)
|
|
46
55
|
pos = { node: r.node, input: r.input }
|
|
47
56
|
if (r.reason !== 'limit' || r.node === dispatch.entry.node) break
|
package/src/graph-engine.mjs
CHANGED
|
@@ -22,6 +22,7 @@ import { iso, key, markerOf, setLabels, setMarker } from './issue-state.mjs'
|
|
|
22
22
|
import { effectivePosition, graphVersionDecision, withHistory } from './marker.mjs'
|
|
23
23
|
import { each, newCtx, staleLeases, taskCandidates, watched, watchPairs } from './tick.mjs'
|
|
24
24
|
import { move, Q } from './labels.mjs'
|
|
25
|
+
import { markerTrace, turnTracer } from './tracing.mjs'
|
|
25
26
|
import { DISPATCH_STEPS } from './dispatch-engine.mjs'
|
|
26
27
|
|
|
27
28
|
/**
|
|
@@ -45,16 +46,25 @@ const taskCtx = (deps, ctx, subject) => ({ ...deps, tick: ctx, subject })
|
|
|
45
46
|
*/
|
|
46
47
|
export async function drive(deps, ctx, subject, at) {
|
|
47
48
|
const c = taskCtx(deps, ctx, subject)
|
|
49
|
+
const traced = turnTracer(deps.telemetry, subject, {
|
|
50
|
+
graph: task.name,
|
|
51
|
+
graphVersion: task.version,
|
|
52
|
+
repo: subject.slug,
|
|
53
|
+
issue: subject.number,
|
|
54
|
+
issueBody: subject.issue?.body ?? null,
|
|
55
|
+
...markerTrace(subject.issue?.stateComment?.marker),
|
|
56
|
+
})
|
|
48
57
|
/** @type {Transition[]} */
|
|
49
58
|
const transitions = []
|
|
50
59
|
let pos = at
|
|
51
60
|
/** @type {Turn['reason']} */
|
|
52
61
|
let reason
|
|
53
62
|
for (;;) {
|
|
54
|
-
const r = await advance(task, pos, c, { maxTransitions: 1 })
|
|
63
|
+
const r = await advance(task, pos, c, { maxTransitions: 1, instrument: traced.instrument })
|
|
55
64
|
if (r.error) throw r.error.cause instanceof Error ? r.error.cause : r.error
|
|
56
65
|
transitions.push(...r.transitions)
|
|
57
66
|
const t = r.transitions[0]
|
|
67
|
+
traced.sync()
|
|
58
68
|
if (t && t.to !== END) await nodeOf(t.to).enter?.(t.next, c)
|
|
59
69
|
pos = { node: r.node, input: r.input }
|
|
60
70
|
if (r.reason !== 'limit') {
|
package/src/graphs/dispatch.mjs
CHANGED
|
@@ -14,6 +14,7 @@ import { iso, key, markerOf, MIN, setLabels, setMarker } from '../issue-state.mj
|
|
|
14
14
|
import { move, Q, WHY } from '../labels.mjs'
|
|
15
15
|
import { newMarker, withHistory } from '../marker.mjs'
|
|
16
16
|
import { readTriage, TriageResult, triageComment, triagePrompt } from '../phases/dispatch.mjs'
|
|
17
|
+
import { traceWorker } from '../tracing.mjs'
|
|
17
18
|
import { task } from './task.mjs'
|
|
18
19
|
|
|
19
20
|
/**
|
|
@@ -167,8 +168,8 @@ const startTriage = declare(
|
|
|
167
168
|
phase: 'dispatch',
|
|
168
169
|
maxTurns: caps.maxTurns,
|
|
169
170
|
readOnly: true,
|
|
170
|
-
//
|
|
171
|
-
|
|
171
|
+
// This node's cambium.node span, so Claude Code's own spans (when worker.env turns them on) join the trace.
|
|
172
|
+
env: ctx.traceparent ? { TRACEPARENT: ctx.traceparent } : undefined,
|
|
172
173
|
buildPrompt: (agentsMd) => triagePrompt({ repo: slug, baseBranch: repo.baseBranch, issue, queued, agentsMd }),
|
|
173
174
|
})
|
|
174
175
|
} catch (err) {
|
|
@@ -191,6 +192,7 @@ const startTriage = declare(
|
|
|
191
192
|
host: config.host,
|
|
192
193
|
stdout: s.stdout,
|
|
193
194
|
stderr: s.stderr,
|
|
195
|
+
traceparent: ctx.traceparent ?? null,
|
|
194
196
|
})
|
|
195
197
|
ctx.log('info', `started triage worker pid ${s.pid} (attempt ${marker.attempt})`, {
|
|
196
198
|
repo: slug,
|
|
@@ -241,8 +243,7 @@ const triaging = declare(
|
|
|
241
243
|
ctx.log('info', 'triage worker gone; issue is closed or left q:intake', { repo: w.repo, issue: w.issue })
|
|
242
244
|
return { ready: true, output: { exit: 'gone' } }
|
|
243
245
|
}
|
|
244
|
-
|
|
245
|
-
// `workerSpans(<w.stdout log>, { tracer, parent: <start-triage's traceparent> })`.
|
|
246
|
+
traceWorker(ctx, w)
|
|
246
247
|
const read = w.killedReason ? { ok: false, error: w.killedReason } : readTriage(spawner.readLog(w.stdout), number)
|
|
247
248
|
if (read.ok && 'result' in read)
|
|
248
249
|
return { ready: true, output: { exit: 'triaged', result: read.result, log: w.stdout } }
|
package/src/graphs/release.mjs
CHANGED
|
@@ -19,13 +19,17 @@ import { iso, setLabels, setMarker } from '../issue-state.mjs'
|
|
|
19
19
|
import { move, Q } from '../labels.mjs'
|
|
20
20
|
import { effectivePosition, newMarker, withHistory } from '../marker.mjs'
|
|
21
21
|
import { releaseDue } from '../phases/release.mjs'
|
|
22
|
+
import { markerTrace, turnTracer } from '../tracing.mjs'
|
|
22
23
|
|
|
23
24
|
/**
|
|
24
25
|
* @typedef {import('../github.mjs').PromotionPr} PromotionPr
|
|
25
26
|
* @typedef {import('../github.mjs').Release} Release
|
|
26
27
|
* @typedef {import('../marker.mjs').Marker} Marker
|
|
27
|
-
* @typedef {{
|
|
28
|
-
*
|
|
28
|
+
* @typedef {{
|
|
29
|
+
* slug: string, repo: import('../config.mjs').Repo, release: Release, pr?: PromotionPr, traceparent?: string | null,
|
|
30
|
+
* }} ReleaseSubject
|
|
31
|
+
* One repo's turn; `pr` is the promotion PR a `release-promote` turn is about; `traceparent` the turn's trace
|
|
32
|
+
* while it is traced (tracing.mjs `turnTracer`), which markers written in the turn take when they have none.
|
|
29
33
|
* @typedef {import('../tick.mjs').Deps & { tick: import('../tick.mjs').Ctx, subject: ReleaseSubject }} ReleaseCtx
|
|
30
34
|
* @typedef {{ queue: string | null, enter?: (input: any, ctx: ReleaseCtx) => Promise<void> }} Declared
|
|
31
35
|
* `queue`: the label a PR resting here carries. `enter`: the write that lands a PR on this wait.
|
|
@@ -248,7 +252,6 @@ const merge = declare(
|
|
|
248
252
|
*/
|
|
249
253
|
run: async (input, ctx) => {
|
|
250
254
|
const pr = prOf(ctx)
|
|
251
|
-
// TRACING (follow-up): a span per merge attempt here, parented on the PR marker's traceparent.
|
|
252
255
|
try {
|
|
253
256
|
await ctx.github.mergePull(ctx.subject.slug, input.pr)
|
|
254
257
|
} catch (err) {
|
|
@@ -364,11 +367,21 @@ export const RELEASE_QUEUES = {
|
|
|
364
367
|
* @param {{ node: string, input: unknown }} at
|
|
365
368
|
*/
|
|
366
369
|
async function drive(graph, ctx, at) {
|
|
370
|
+
const { pr } = ctx.subject
|
|
371
|
+
// A promote turn is traced on its PR (the marker's trace); a prepare run is a root of its own, which the PR it
|
|
372
|
+
// opens or updates takes into its marker when that has none.
|
|
373
|
+
const traced = turnTracer(ctx.telemetry, ctx.subject, {
|
|
374
|
+
graph: graph.name,
|
|
375
|
+
graphVersion: graph.version,
|
|
376
|
+
repo: ctx.subject.slug,
|
|
377
|
+
issue: pr?.number,
|
|
378
|
+
...markerTrace(pr?.stateComment?.marker),
|
|
379
|
+
})
|
|
367
380
|
let pos = at
|
|
368
381
|
for (;;) {
|
|
369
|
-
|
|
370
|
-
const r = await advance(graph, pos, ctx, { maxTransitions: 1 })
|
|
382
|
+
const r = await advance(graph, pos, ctx, { maxTransitions: 1, instrument: traced.instrument })
|
|
371
383
|
if (r.error) throw r.error.cause instanceof Error ? r.error.cause : r.error
|
|
384
|
+
traced.sync()
|
|
372
385
|
const t = r.transitions[0]
|
|
373
386
|
const next = /** @type {(import('../graph/define.mjs').GraphNode & Declared) | undefined} */ (
|
|
374
387
|
graph.nodes.find((n) => n.name === r.node)
|
package/src/graphs/task.mjs
CHANGED
|
@@ -15,6 +15,7 @@ import { move, Q, WHY } from '../labels.mjs'
|
|
|
15
15
|
import { withHistory } from '../marker.mjs'
|
|
16
16
|
import { prFor, taskExit, taskPrompt } from '../phases/task.mjs'
|
|
17
17
|
import { PREPARE_FAILED_LINE } from '../prepare.mjs'
|
|
18
|
+
import { traceWorker } from '../tracing.mjs'
|
|
18
19
|
|
|
19
20
|
/**
|
|
20
21
|
* @typedef {import('../github.mjs').Issue} Issue
|
|
@@ -24,10 +25,11 @@ import { PREPARE_FAILED_LINE } from '../prepare.mjs'
|
|
|
24
25
|
* @typedef {{
|
|
25
26
|
* slug: string, number: number, issue?: Issue, snap?: Snapshot,
|
|
26
27
|
* pr?: PullRequest | null, deleted?: Set<string>, foreground?: boolean, reaped?: Set<number>,
|
|
27
|
-
* started?: import('../workers.mjs').Started & { attempt: number },
|
|
28
|
+
* started?: import('../workers.mjs').Started & { attempt: number }, traceparent?: string | null,
|
|
28
29
|
* }} Subject
|
|
29
30
|
* One issue's turn. `pr` is the PR Watch is looking at; `deleted` the branches already deleted this tick;
|
|
30
|
-
* `foreground`/`reaped`/`started` are for `cambium run
|
|
31
|
+
* `foreground`/`reaped`/`started` are for `cambium run`; `traceparent` is the issue trace while the turn is
|
|
32
|
+
* traced (tracing.mjs `turnTracer`), which markers written in the turn take.
|
|
31
33
|
* @typedef {import('../tick.mjs').Deps & {
|
|
32
34
|
* tick: import('../tick.mjs').Ctx, subject: Subject, traceparent?: string | null,
|
|
33
35
|
* }} TaskCtx
|
|
@@ -240,8 +242,8 @@ const startWorker = declare(
|
|
|
240
242
|
phase: 'task',
|
|
241
243
|
maxTurns: caps.maxTurns,
|
|
242
244
|
foreground,
|
|
243
|
-
//
|
|
244
|
-
|
|
245
|
+
// This node's cambium.node span, so Claude Code's own spans (when worker.env turns them on) join the trace.
|
|
246
|
+
env: ctx.traceparent ? { TRACEPARENT: ctx.traceparent } : undefined,
|
|
245
247
|
buildPrompt: (agentsMd) =>
|
|
246
248
|
taskPrompt({ repo: slug, issue, marker, branch, pr, agentsMd, maxAttempts: caps.maxAttempts }),
|
|
247
249
|
})
|
|
@@ -266,6 +268,7 @@ const startWorker = declare(
|
|
|
266
268
|
host: config.host,
|
|
267
269
|
stdout: s.stdout,
|
|
268
270
|
stderr: s.stderr,
|
|
271
|
+
traceparent: ctx.traceparent ?? null,
|
|
269
272
|
})
|
|
270
273
|
const where = { repo: slug, issue: issue.number }
|
|
271
274
|
if (s.reused) ctx.log('warn', `continuing in the previous worktree ${s.worktree}: ${s.reused}`, where)
|
|
@@ -314,8 +317,7 @@ const workerRunning = declare(
|
|
|
314
317
|
ctx.log('info', 'worker gone; issue is closed or no longer queued', { repo: w.repo, issue: w.issue })
|
|
315
318
|
return { ready: true, output: { exit: 'gone' } }
|
|
316
319
|
}
|
|
317
|
-
|
|
318
|
-
// `workerSpans(<w.stdout log>, { tracer, parent: <start-worker's traceparent> })`.
|
|
320
|
+
traceWorker(ctx, w)
|
|
319
321
|
const pr = prFor(snap, w)
|
|
320
322
|
const m = markerOf(issue)
|
|
321
323
|
const prepFailed = !w.killedReason && spawner.tail(w.stderr, 15).includes(PREPARE_FAILED_LINE)
|
package/src/issue-state.mjs
CHANGED
|
@@ -24,6 +24,10 @@ export async function setLabels(deps, slug, issue, change) {
|
|
|
24
24
|
|
|
25
25
|
/** @param {Writers} deps @param {string} slug @param {Issue} issue @param {import('./marker.mjs').Marker} marker */
|
|
26
26
|
export async function setMarker(deps, slug, issue, marker) {
|
|
27
|
+
// A traced graph turn keeps its issue trace on the subject (tracing.mjs `turnTracer`); a marker without one
|
|
28
|
+
// takes it, so later ticks parent on the same trace. One that already has a trace keeps it.
|
|
29
|
+
const tp = /** @type {any} */ (deps).subject?.traceparent
|
|
30
|
+
if (tp && !marker.traceparent) marker = { ...marker, traceparent: tp }
|
|
27
31
|
const maxAttempts = deps.config.phases[marker.phase]?.maxAttempts
|
|
28
32
|
const id = await deps.github.writeState(
|
|
29
33
|
slug,
|
package/src/store.mjs
CHANGED
|
@@ -13,7 +13,9 @@ import { DatabaseSync } from 'node:sqlite'
|
|
|
13
13
|
* @typedef {{
|
|
14
14
|
* repo: string, issue: number, phase: string, pid: number, worktree: string, branch: string,
|
|
15
15
|
* attempt: number, startedAt: number, host: string, stdout: string, stderr: string, killedReason: string | null,
|
|
16
|
+
* traceparent: string | null,
|
|
16
17
|
* }} Worker
|
|
18
|
+
* `traceparent`: the span of the step that started it, when traced; its `claude.*` spans go under it on exit.
|
|
17
19
|
*/
|
|
18
20
|
|
|
19
21
|
export const MODES = /** @type {const} */ (['running', 'paused', 'draining'])
|
|
@@ -22,7 +24,7 @@ const SCHEMA = `
|
|
|
22
24
|
CREATE TABLE IF NOT EXISTS workers (
|
|
23
25
|
repo TEXT NOT NULL, issue INTEGER NOT NULL, phase TEXT NOT NULL, pid INTEGER NOT NULL,
|
|
24
26
|
worktree TEXT NOT NULL, branch TEXT NOT NULL, attempt INTEGER NOT NULL, started_at INTEGER NOT NULL,
|
|
25
|
-
host TEXT NOT NULL, stdout TEXT NOT NULL, stderr TEXT NOT NULL, killed_reason TEXT,
|
|
27
|
+
host TEXT NOT NULL, stdout TEXT NOT NULL, stderr TEXT NOT NULL, killed_reason TEXT, traceparent TEXT,
|
|
26
28
|
PRIMARY KEY (repo, issue)
|
|
27
29
|
);
|
|
28
30
|
CREATE TABLE IF NOT EXISTS control (key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
|
@@ -42,6 +44,7 @@ const toWorker = (r) => ({
|
|
|
42
44
|
stdout: r.stdout,
|
|
43
45
|
stderr: r.stderr,
|
|
44
46
|
killedReason: r.killed_reason ?? null,
|
|
47
|
+
traceparent: r.traceparent ?? null,
|
|
45
48
|
})
|
|
46
49
|
|
|
47
50
|
/** @param {string} path a file, or `:memory:` */
|
|
@@ -50,6 +53,9 @@ export function openStore(path) {
|
|
|
50
53
|
const db = new DatabaseSync(path)
|
|
51
54
|
db.exec('PRAGMA journal_mode = WAL; PRAGMA busy_timeout = 5000;')
|
|
52
55
|
db.exec(SCHEMA)
|
|
56
|
+
// Databases from before `traceparent` existed.
|
|
57
|
+
const cols = /** @type {{ name: string }[]} */ (db.prepare('PRAGMA table_info(workers)').all())
|
|
58
|
+
if (!cols.some((c) => c.name === 'traceparent')) db.exec('ALTER TABLE workers ADD COLUMN traceparent TEXT')
|
|
53
59
|
const get = (/** @type {string} */ key) =>
|
|
54
60
|
/** @type {any} */ (db.prepare('SELECT value FROM control WHERE key = ?').get(key))?.value ?? null
|
|
55
61
|
const set = (/** @type {string} */ key, /** @type {string} */ value) =>
|
|
@@ -65,12 +71,25 @@ export function openStore(path) {
|
|
|
65
71
|
const r = db.prepare('SELECT * FROM workers WHERE repo = ? AND issue = ?').get(repo, issue)
|
|
66
72
|
return r ? toWorker(r) : null
|
|
67
73
|
},
|
|
68
|
-
/** @param {Omit<Worker, 'killedReason'>} w */
|
|
74
|
+
/** @param {Omit<Worker, 'killedReason' | 'traceparent'> & { traceparent?: string | null }} w */
|
|
69
75
|
addWorker(w) {
|
|
70
76
|
db.prepare(
|
|
71
|
-
`INSERT OR REPLACE INTO workers (repo, issue, phase, pid, worktree, branch, attempt, started_at, host, stdout, stderr, killed_reason)
|
|
72
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL)`,
|
|
73
|
-
).run(
|
|
77
|
+
`INSERT OR REPLACE INTO workers (repo, issue, phase, pid, worktree, branch, attempt, started_at, host, stdout, stderr, killed_reason, traceparent)
|
|
78
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, ?)`,
|
|
79
|
+
).run(
|
|
80
|
+
w.repo,
|
|
81
|
+
w.issue,
|
|
82
|
+
w.phase,
|
|
83
|
+
w.pid,
|
|
84
|
+
w.worktree,
|
|
85
|
+
w.branch,
|
|
86
|
+
w.attempt,
|
|
87
|
+
w.startedAt,
|
|
88
|
+
w.host,
|
|
89
|
+
w.stdout,
|
|
90
|
+
w.stderr,
|
|
91
|
+
w.traceparent ?? null,
|
|
92
|
+
)
|
|
74
93
|
},
|
|
75
94
|
/** @param {string} repo @param {number} issue @param {string} reason */
|
|
76
95
|
markKilled(repo, issue, reason) {
|
package/src/tick.mjs
CHANGED
|
@@ -35,6 +35,7 @@ import { DISPATCH_NOT_BUILT, dispatchNotBuilt } from './phases/dispatch.mjs'
|
|
|
35
35
|
* dispatch?: import('./phases/dispatch.mjs').Dispatch,
|
|
36
36
|
* release?: import('./phases/release.mjs').Release,
|
|
37
37
|
* tickNumber?: number,
|
|
38
|
+
* telemetry?: import('./tracing.mjs').Telemetry,
|
|
38
39
|
* daemonStartedAt?: number,
|
|
39
40
|
* pid?: number,
|
|
40
41
|
* }} Deps
|
package/src/tracing.mjs
CHANGED
|
@@ -290,6 +290,62 @@ export function advanceHooks(telemetry, base) {
|
|
|
290
290
|
}
|
|
291
291
|
}
|
|
292
292
|
|
|
293
|
+
/**
|
|
294
|
+
* Tracing for one turn of a graph driver (graph-engine `drive`, dispatch-engine `driveDispatch`, release `drive`).
|
|
295
|
+
* Wraps {@link advanceHooks} and keeps `holder.traceparent` (the turn's subject) on the issue trace, so a marker
|
|
296
|
+
* written during the turn stores it (issue-state.mjs `setMarker`). Call `sync()` before writes made outside
|
|
297
|
+
* `advance` (a wait's `enter`). With telemetry off, or none, `instrument` is undefined and `advance` runs exactly
|
|
298
|
+
* as it does untraced.
|
|
299
|
+
*
|
|
300
|
+
* @param {Telemetry | undefined} telemetry
|
|
301
|
+
* @param {{ traceparent?: string | null }} holder
|
|
302
|
+
* @param {TraceInfo & { enteredAt?: string | null }} base as for {@link advanceHooks}
|
|
303
|
+
* @returns {{ instrument: import('./graph/advance.mjs').Instrument | undefined, sync: () => void }}
|
|
304
|
+
*/
|
|
305
|
+
export function turnTracer(telemetry, holder, base) {
|
|
306
|
+
if (!telemetry?.enabled) return { instrument: undefined, sync: () => {} }
|
|
307
|
+
const hooks = advanceHooks(telemetry, base)
|
|
308
|
+
const sync = () => {
|
|
309
|
+
if (hooks.traceparent) holder.traceparent = hooks.traceparent
|
|
310
|
+
}
|
|
311
|
+
return {
|
|
312
|
+
instrument: (visit, step) =>
|
|
313
|
+
hooks.instrument(visit, (extra) => {
|
|
314
|
+
sync()
|
|
315
|
+
return step(extra)
|
|
316
|
+
}),
|
|
317
|
+
sync,
|
|
318
|
+
}
|
|
319
|
+
}
|
|
320
|
+
|
|
321
|
+
/**
|
|
322
|
+
* The marker fields a turn's {@link turnTracer} starts from. No marker yet is a fresh issue: attempt 1, as
|
|
323
|
+
* marker.mjs `newMarker` has it.
|
|
324
|
+
* @param {{ traceparent?: string | null, enteredAt?: string | null, attempt?: number } | null | undefined} marker
|
|
325
|
+
*/
|
|
326
|
+
export const markerTrace = (marker) => ({
|
|
327
|
+
traceparent: marker?.traceparent ?? null,
|
|
328
|
+
enteredAt: marker?.enteredAt ?? null,
|
|
329
|
+
attempt: marker?.attempt ?? 1,
|
|
330
|
+
})
|
|
331
|
+
|
|
332
|
+
/**
|
|
333
|
+
* A finished worker's `claude.*` spans from its stream-json stdout log, under the step span that started it
|
|
334
|
+
* (`parent`, kept on the worker row). Never throws: tracing must not fail a tick.
|
|
335
|
+
*
|
|
336
|
+
* @param {{ telemetry?: Telemetry, spawner: { readLog(path: string): string },
|
|
337
|
+
* log: (level: 'info' | 'warn' | 'error', msg: string, where?: { repo?: string, issue?: number }) => void }} ctx
|
|
338
|
+
* @param {{ repo: string, issue: number, stdout: string, traceparent?: string | null }} w
|
|
339
|
+
*/
|
|
340
|
+
export function traceWorker(ctx, w) {
|
|
341
|
+
if (!ctx.telemetry?.enabled || !w.traceparent) return
|
|
342
|
+
try {
|
|
343
|
+
ctx.telemetry.workerSpans(ctx.spawner.readLog(w.stdout), w.traceparent)
|
|
344
|
+
} catch (err) {
|
|
345
|
+
ctx.log('warn', `worker spans: ${/** @type {Error} */ (err)?.message ?? err}`, { repo: w.repo, issue: w.issue })
|
|
346
|
+
}
|
|
347
|
+
}
|
|
348
|
+
|
|
293
349
|
/** A node output's own `outcome` string when it has one, else `ok`. @param {unknown} output */
|
|
294
350
|
const outcomeOf = (output) => {
|
|
295
351
|
const o = /** @type {any} */ (output)?.outcome
|
package/src/workers.mjs
CHANGED
|
@@ -28,10 +28,11 @@ import { exec as defaultExec } from './github.mjs'
|
|
|
28
28
|
* @typedef {{
|
|
29
29
|
* repo: import('./config.mjs').Repo, convention: Convention, issue: number, attempt: number, branch: string,
|
|
30
30
|
* phase: string, maxTurns: number, buildPrompt: (agentsMd: string | null) => string, foreground?: boolean,
|
|
31
|
-
* readOnly?: boolean,
|
|
31
|
+
* readOnly?: boolean, env?: Record<string, string>,
|
|
32
32
|
* }} StartSpec
|
|
33
33
|
* `readOnly`: a disposable checkout for a worker that only reads (triage): detached at the base branch, no
|
|
34
34
|
* install, hydrate or prepare, no Edit/Write; `branch` only names its directory (see {@link topicOf}).
|
|
35
|
+
* `env`: added to the worker's environment after `config.worker.env` (the step's `TRACEPARENT`).
|
|
35
36
|
* @typedef {{
|
|
36
37
|
* pid: number, worktree: string, stdout: string, stderr: string, prompt: string, exited: Promise<number | null>,
|
|
37
38
|
* reused: string | null,
|
|
@@ -221,6 +222,7 @@ export function processSpawner({
|
|
|
221
222
|
buildPrompt,
|
|
222
223
|
foreground = false,
|
|
223
224
|
readOnly = false,
|
|
225
|
+
env: extraEnv = {},
|
|
224
226
|
}) {
|
|
225
227
|
const worktree = join(convention.root, topicOf(branch))
|
|
226
228
|
if (config.worker.fetch) {
|
|
@@ -280,6 +282,9 @@ export function processSpawner({
|
|
|
280
282
|
},
|
|
281
283
|
),
|
|
282
284
|
...(config.worker.graftApp ? { GRAFT_APP: config.worker.graftApp } : {}),
|
|
285
|
+
// config.worker.env (e.g. Claude Code's own telemetry), then this start's (TRACEPARENT).
|
|
286
|
+
...config.worker.env,
|
|
287
|
+
...extraEnv,
|
|
283
288
|
}
|
|
284
289
|
/** @type {import('node:child_process').ChildProcess} */
|
|
285
290
|
let child
|