@oakgreencc/cambium 0.0.13 → 0.0.15

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -80,7 +80,7 @@ A worker is `claude -p --output-format stream-json --verbose --max-turns <maxTur
80
80
  - in its own worktree, by the target repo's [worktree convention](#target-repo-convention);
81
81
  - behind the prepare step: `node src/prepare.mjs -- claude …` runs install-fresh, hydrate and the repo's `prepare` commands in the worktree, writing to the worker's stderr log, then becomes the worker. It runs in the detached process so a long install never blocks a tick;
82
82
  - detached, leading its own process group (a kill takes its children), stdin from the prompt file;
83
- - with `CAMBIUM_REPO`, `CAMBIUM_ISSUE`, `CAMBIUM_PHASE`, `CAMBIUM_BRANCH` set, and `GRAFT_APP=<worker.graftApp>` when configured;
83
+ - with `CAMBIUM_REPO`, `CAMBIUM_ISSUE`, `CAMBIUM_PHASE`, `CAMBIUM_BRANCH` set, `GRAFT_APP=<worker.graftApp>` when configured, then `worker.env`, then `TRACEPARENT` (the starting node's span) when [tracing](#tracing) is on;
84
84
  - with its prompt, stdout and stderr at `<paths.logs>/<owner>__<name>/issue-<n>/a<attempt>-<stamp>.{prompt.md,out.log,err.log}`, keeping the newest `logs.keepPerIssue` runs per issue.
85
85
 
86
86
  The prompt states the four-point contract (task issue, `pnpm verify`, `graft publish <branch> … --closes <n>`, `graft ci wait`), the attempt, the existing PR if any (`graft advance` onto it), the last failure, the issue's history, the issue body, and the worktree's `AGENTS.md`.
@@ -111,10 +111,12 @@ One JSON file. Everything but `repos` is optional; these are the defaults:
111
111
  },
112
112
  "logs": { "keepPerIssue": 20, "daemonMaxBytes": 10485760 },
113
113
  "graft": { "command": "graft", "app": null, "botLogin": null },
114
- "worker": { "command": "claude", "args": [], "graftApp": null, "fetch": true }
114
+ "worker": { "command": "claude", "args": [], "graftApp": null, "fetch": true, "env": {} }
115
115
  }
116
116
  ```
117
117
 
118
+ - `worker.env` is extra environment for every worker (string values only), e.g. Claude Code's own telemetry ([Tracing](#claude-codes-own-spans-under-the-worker)).
119
+
118
120
  - The daemon re-reads the file before every tick, so caps, limits, `tickSeconds`, `repos`, priority labels, `release` and `worker` take effect without a restart (logged as `config reloaded: <keys>`). `host`, `engine`, `paths`, `logs` and `graft` are baked in at start: a change to them logs `restart cambium to apply` once and is not applied. A file that does not load, or has no `repos`, keeps the running config and logs one warning until it is fixed.
119
121
  - `repos[].path` is a local clone; worktrees are added from it and `graft` runs in it.
120
122
  - `graft.app` is the App cambium acts as (`GRAFT_APP` for its own `graft` calls); `worker.graftApp` the one workers act as. Neither leaks into the other.
@@ -137,7 +139,7 @@ How a repo's worktrees are made is the repo's call: a `"cambium"` key in its roo
137
139
  ```
138
140
 
139
141
  - **Branch**: `{issue}` (required) and `{slug}`, the issue title in kebab case, at most 40 characters cut at a word. A retry keeps the branch its first attempt chose (it is in the marker), and an existing PR's branch wins over both.
140
- - **Worktree**: `<repos[].path>/<root>/<topic>`, the topic being the branch less its first segment (`agent/12-fix-the-tick` → `12-fix-the-tick`), so every attempt of an issue uses the same directory. Created with `git fetch origin <baseBranch>`, then `git worktree add --no-track -B <branch> <dir> origin/<baseBranch>`: always from the fetched remote branch, never a local one. Add `<root>` to the repo's `.gitignore`.
142
+ - **Worktree**: `<repos[].path>/<root>/<topic>`, the topic being the branch less its first segment (`agent/12-fix-the-tick` → `12-fix-the-tick`), so every attempt of an issue uses the same directory. Created with `git fetch origin <baseBranch>`, then `git worktree add --no-track -B <branch> <dir> origin/<baseBranch>`: always from the fetched remote branch, never a local one. Between the two, the central checkout is fast-forwarded to `origin/<baseBranch>` when it sits on that branch with no tracked changes, and the convention re-read, so install, hydrate and prepare come from current content; otherwise it is left as it is. Add `<root>` to the repo's `.gitignore`.
141
143
  - **Teardown**: once the worker is reconciled, `graft status <branch> -C <dir>` (as `graft.app`) must end with `nothing local that GitHub does not have`; then the worktree and its local branch are removed. Anything else (local commits, uncommitted paths, graft failing) keeps it and logs why.
142
144
  - **Retry**: a leftover worktree in the topic directory is removed under the same rule and re-created fresh; one that still holds local-only work is continued in instead (logged as `continuing in the previous worktree`), so nothing is lost.
143
145
  - **Prepare**, after the worktree is created and before the worker starts, in order:
@@ -452,6 +454,16 @@ Why not a shared package: cambium is published to npm, so it can only depend on
452
454
 
453
455
  Node and wait spans carry `cambium.graph`, `cambium.graph.version`, `cambium.node`, `cambium.attempt`, `cambium.repo`, `cambium.issue.number`, `cambium.edge`, `cambium.outcome`.
454
456
 
457
+ ### What is traced
458
+
459
+ With `engine: "graph"`, every graph run is traced: the `task` and `dispatch` graphs per issue and both release graphs (the legacy stage tick is not). [canopy](../canopy)'s Traces tab shows them from the local collector's file.
460
+
461
+ - **Lifecycle.** `cambium start` and `cambium run` create the telemetry from their environment (a no-op without OTLP config), flush it after every tick and shut it down on exit.
462
+ - **One trace per issue, across ticks.** Each turn (`drive`, `driveDispatch`, release `drive`) runs `advance` with `turnTracer`'s `instrument` (`advanceHooks` underneath). The first span of an issue starts its `cambium.issue` root, a child of mycelium's intake span when the body carries one; the marker written in that turn takes the root's traceparent (`setMarker` fills `traceparent` from the turn's subject when the marker has none), and every later tick parents on it. Requeues and later attempts stay in the trace, and the dispatch → task hand-off carries it, so one issue reads intake → triage → each task attempt → CI → close-out.
463
+ - **Release.** A release-prepare run is a root of its own; the promotion PR it opens or refreshes takes that trace into its marker when it has none, and its release-promote turns (`merge`, `merging`) continue it, with `cambium.issue.number` the PR's number.
464
+ - **Workers.** start-worker and start-triage start their worker with `TRACEPARENT` set to their own `cambium.node` span and record it on the worker row (`workers.traceparent` in SQLite; older databases gain the column on open). When the worker is found exited, its stdout log becomes `claude.*` spans under that span ([below](#worker-spans-from-stream-json)); a log that cannot be read is a warning, never a failed tick.
465
+ - **Off is unchanged.** Without OTLP config there is no `instrument`, no `TRACEPARENT`, and markers keep `traceparent: null`; the legacy/graph differential test covers that the moves are identical.
466
+
455
467
  ```js
456
468
  import { advanceHooks, createTelemetry } from './tracing.mjs' // inside cambium
457
469
 
@@ -495,7 +507,40 @@ telemetry.workerSpans(stream, parent) // same, with cambium's tracer
495
507
  - `claude.task`: each background task, `task_started` → `task_notification` (or `task_updated.patch.end_time`), under the tool that started it. These events have no timestamp, so their times are those of the latest timestamped event.
496
508
  - Malformed lines are counted (`malformed`) and skipped; `system/thinking_tokens`, `tool_progress`, hooks and rate-limit events are ignored.
497
509
 
498
- ### Does `claude -p` export traces that honour `TRACEPARENT`?
510
+ ### Claude Code's own spans under the worker
511
+
512
+ **`claude -p` honours `TRACEPARENT`: verified live** (Claude Code 2.1.327, 2026-11-18). A `claude -p` run with a Bash call, given `TRACEPARENT=00-<trace>-<span>-01` and the variables below, exported into `<trace>`:
513
+
514
+ ```
515
+ <the TRACEPARENT span>
516
+ └ claude_code.interaction parent.source=env
517
+ ├ claude_code.llm_request model, input/output/cache tokens, ttft_ms, stop_reason
518
+ ├ claude_code.tool tool_name=Bash
519
+ │ ├ claude_code.tool.blocked_on_user
520
+ │ └ claude_code.tool.execution
521
+ └ claude_code.llm_request
522
+ ```
523
+
524
+ Workers already get `TRACEPARENT`, so turning these on is config only:
525
+
526
+ ```jsonc
527
+ "worker": {
528
+ "env": {
529
+ "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
530
+ "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
531
+ "OTEL_TRACES_EXPORTER": "otlp",
532
+ "OTEL_METRICS_EXPORTER": "none",
533
+ "OTEL_LOGS_EXPORTER": "none",
534
+ "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
535
+ "OTEL_EXPORTER_OTLP_ENDPOINT": "http://127.0.0.1:4318"
536
+ }
537
+ }
538
+ ```
539
+
540
+ - They complement the stream-json spans rather than replace them: they carry per-request timing and tokens but no cost, and need telemetry on in the worker; `claude.session` carries the session's `claude.cost_usd` and works without it. With both on, a worker has `claude.*` and `claude_code.*` spans side by side under start-worker.
541
+ - **Identity in spans.** Every `claude_code.*` span carries `user.email`, `user.account_uuid`, `user.account_id` and `organization.id`. Fine for the local collector file; mind it before exporting elsewhere (strip them in the collector with an `attributes` processor).
542
+
543
+ ### Does `claude -p` export traces that honour `TRACEPARENT`? (the docs)
499
544
 
500
545
  **Yes, opt-in and beta**; cambium does not depend on it (worker spans come from stream-json above). Per the [Claude Code monitoring docs](https://code.claude.com/docs/en/monitoring-usage) (checked with Claude Code 2.1.327; `claude --help` lists no telemetry flags, it is env-only):
501
546
 
@@ -504,7 +549,7 @@ telemetry.workerSpans(stream, parent) // same, with cambium's tracer
504
549
  - Bash subprocesses it runs inherit a `TRACEPARENT` for the active tool span, so a `graft` call inside a worker could continue the trace.
505
550
  - Prompts and tool inputs are redacted from spans unless `OTEL_LOG_USER_PROMPTS` / `OTEL_LOG_TOOL_DETAILS` / `OTEL_LOG_TOOL_CONTENT` are set.
506
551
 
507
- So to also get Claude's own spans in the issue trace, add `CLAUDE_CODE_ENABLE_TELEMETRY=1`, `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`, `OTEL_TRACES_EXPORTER=otlp` and the endpoint to `worker.env`; start-worker already sets `TRACEPARENT`. Not verified against a live run here.
552
+ Confirmed against a live run: see [Claude Code's own spans under the worker](#claude-codes-own-spans-under-the-worker).
508
553
 
509
554
  ### Local collector
510
555
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@oakgreencc/cambium",
3
- "version": "0.0.13",
3
+ "version": "0.0.15",
4
4
  "description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
5
5
  "license": "AGPL-3.0-only",
6
6
  "type": "module",
package/src/cli.mjs CHANGED
@@ -14,6 +14,7 @@ import { plist, plistPath } from './launchd.mjs'
14
14
  import { openStore } from './store.mjs'
15
15
  import { finish, newCtx, startTask } from './tick.mjs'
16
16
  import { runIssue } from './graph-engine.mjs'
17
+ import { createTelemetry } from './tracing.mjs'
17
18
 
18
19
  export const VERSION = '0.0.0'
19
20
 
@@ -41,6 +42,24 @@ const flag = (argv, name) => {
41
42
  /** @param {string[]} argv @param {string} name */
42
43
  const has = (argv, name) => argv.includes(name)
43
44
 
45
+ /**
46
+ * Run `fn` with `deps.telemetry` set (from the OTEL_* vars in `io.env`, a no-op without them, unless the deps
47
+ * already carry one), shutting it down afterwards so queued spans are exported before the process exits.
48
+ *
49
+ * @template T
50
+ * @param {import('./tick.mjs').Deps} deps @param {IO} io @param {() => Promise<T>} fn
51
+ * @returns {Promise<T>}
52
+ */
53
+ async function withTelemetry(deps, io, fn) {
54
+ const own = !deps.telemetry
55
+ if (own) deps.telemetry = await createTelemetry({ env: io.env })
56
+ try {
57
+ return await fn()
58
+ } finally {
59
+ if (own) await deps.telemetry?.shutdown().catch(() => {})
60
+ }
61
+ }
62
+
44
63
  /**
45
64
  * @typedef {{
46
65
  * out: (line: string) => void,
@@ -73,7 +92,9 @@ export async function main(argv, io) {
73
92
  if (!config.repos.length) return fail(io, `no repos in ${cfgPath}; see the README for the config file`)
74
93
  const deps = depsFor(config)
75
94
  acquireLock(deps.store, deps.procs, deps.pid ?? process.pid)
76
- await runDaemon(deps, { signal: io.signal, once: has(rest, '--once'), reload: () => loadConfig(cfgPath) })
95
+ await withTelemetry(deps, io, () =>
96
+ runDaemon(deps, { signal: io.signal, once: has(rest, '--once'), reload: () => loadConfig(cfgPath) }),
97
+ )
77
98
  return 0
78
99
  }
79
100
  case 'status':
@@ -169,7 +190,9 @@ async function runOne(config, argv, io, depsFor) {
169
190
  )
170
191
  const deps = depsFor(config)
171
192
  if (config.engine === 'graph') {
172
- const r = await runIssue(deps, { graph, slug, number: n, follow, out: io.out, sleep: io.sleep, signal: io.signal })
193
+ const r = await withTelemetry(deps, io, () =>
194
+ runIssue(deps, { graph, slug, number: n, follow, out: io.out, sleep: io.sleep, signal: io.signal }),
195
+ )
173
196
  return r.ok ? 0 : fail(io, /** @type {string} */ (r.error))
174
197
  }
175
198
  if (deps.store.worker(slug, n)) return fail(io, `${slug}#${n} already has a tracked worker`)
package/src/config.mjs CHANGED
@@ -27,7 +27,7 @@ import { join, resolve } from 'node:path'
27
27
  * paths: { home: string, db: string, heartbeat: string, logs: string },
28
28
  * logs: { keepPerIssue: number, daemonMaxBytes: number },
29
29
  * graft: { command: string, app: string | null, botLogin: string | null },
30
- * worker: { command: string, args: string[], graftApp: string | null, fetch: boolean },
30
+ * worker: { command: string, args: string[], graftApp: string | null, fetch: boolean, env: Record<string, string> },
31
31
  * }} Config
32
32
  */
33
33
 
@@ -56,7 +56,7 @@ export function defaults(home = DEFAULT_HOME) {
56
56
  },
57
57
  logs: { keepPerIssue: 20, daemonMaxBytes: 10 * 1024 * 1024 },
58
58
  graft: { command: 'graft', app: null, botLogin: null },
59
- worker: { command: 'claude', args: [], graftApp: null, fetch: true },
59
+ worker: { command: 'claude', args: [], graftApp: null, fetch: true, env: {} },
60
60
  }
61
61
  }
62
62
 
@@ -103,6 +103,14 @@ export function resolveConfig(raw = {}) {
103
103
  if (!/^\d{2}:\d{2}$/.test(release.time)) throw new Error(`config: repo ${r.slug} release.time must be HH:MM`)
104
104
  return { slug: r.slug, path: untilde(r.path), baseBranch, release }
105
105
  })
106
+ const wenv = config.worker.env
107
+ if (
108
+ !wenv ||
109
+ typeof wenv !== 'object' ||
110
+ Array.isArray(wenv) ||
111
+ Object.values(wenv).some((v) => typeof v !== 'string')
112
+ )
113
+ throw new Error('config: worker.env must be an object of string values')
106
114
  if (config.engine !== 'legacy' && config.engine !== 'graph')
107
115
  throw new Error(`config: engine must be "legacy" or "graph", got ${JSON.stringify(config.engine)}`)
108
116
  return config
package/src/daemon.mjs CHANGED
@@ -134,6 +134,8 @@ export async function runDaemon(
134
134
  } catch (err) {
135
135
  deps.log('error', `tick ${n}: ${/** @type {Error} */ (err)?.message ?? err}`)
136
136
  }
137
+ // Every span is ended within its tick; export them now rather than when the batch fills.
138
+ await deps.telemetry?.flush().catch((err) => deps.log('warn', `telemetry flush: ${err?.message ?? err}`))
137
139
  if (once) break
138
140
  try {
139
141
  await wait(deps.config.tickSeconds * 1000, signal)
@@ -15,6 +15,7 @@ import { advance } from './graph/index.mjs'
15
15
  import { dispatch, lostLeaseOf, nodeOf } from './graphs/dispatch.mjs'
16
16
  import { byPriorityThenAge, Q, RUNNING } from './labels.mjs'
17
17
  import { each, staleLeases } from './tick.mjs'
18
+ import { markerTrace, turnTracer } from './tracing.mjs'
18
19
 
19
20
  /**
20
21
  * @typedef {import('./tick.mjs').Deps} Deps
@@ -33,15 +34,23 @@ export const PHASE = 'dispatch'
33
34
  */
34
35
  export async function driveDispatch(deps, ctx, subject, at) {
35
36
  const c = { ...deps, tick: ctx, subject }
37
+ // A new issue's cambium.issue root continues mycelium's intake span from the body, so these node and wait spans
38
+ // join the intake trace (tracing.mjs `issueTrace`); the marker carries it on into the task graph.
39
+ const traced = turnTracer(deps.telemetry, subject, {
40
+ graph: dispatch.name,
41
+ graphVersion: dispatch.version,
42
+ repo: subject.slug,
43
+ issue: subject.number,
44
+ issueBody: subject.issue?.body ?? null,
45
+ ...markerTrace(subject.issue?.stateComment?.marker),
46
+ })
36
47
  /** @type {Transition[]} */
37
48
  const transitions = []
38
49
  let pos = at
39
50
  for (;;) {
40
- // TRACING (follow-up): pass `instrument: advanceHooks(telemetry, { graph: 'dispatch', …, issueBody }).instrument`
41
- // here; the issue's cambium.issue root continues mycelium's intake span from the body, so these node and wait
42
- // spans join the intake trace (tracing.mjs `issueTrace`).
43
- const r = await advance(dispatch, pos, c, { maxTransitions: 1 })
51
+ const r = await advance(dispatch, pos, c, { maxTransitions: 1, instrument: traced.instrument })
44
52
  if (r.error) throw r.error.cause instanceof Error ? r.error.cause : r.error
53
+ traced.sync()
45
54
  transitions.push(...r.transitions)
46
55
  pos = { node: r.node, input: r.input }
47
56
  if (r.reason !== 'limit' || r.node === dispatch.entry.node) break
@@ -22,6 +22,7 @@ import { iso, key, markerOf, setLabels, setMarker } from './issue-state.mjs'
22
22
  import { effectivePosition, graphVersionDecision, withHistory } from './marker.mjs'
23
23
  import { each, newCtx, staleLeases, taskCandidates, watched, watchPairs } from './tick.mjs'
24
24
  import { move, Q } from './labels.mjs'
25
+ import { markerTrace, turnTracer } from './tracing.mjs'
25
26
  import { DISPATCH_STEPS } from './dispatch-engine.mjs'
26
27
 
27
28
  /**
@@ -45,16 +46,25 @@ const taskCtx = (deps, ctx, subject) => ({ ...deps, tick: ctx, subject })
45
46
  */
46
47
  export async function drive(deps, ctx, subject, at) {
47
48
  const c = taskCtx(deps, ctx, subject)
49
+ const traced = turnTracer(deps.telemetry, subject, {
50
+ graph: task.name,
51
+ graphVersion: task.version,
52
+ repo: subject.slug,
53
+ issue: subject.number,
54
+ issueBody: subject.issue?.body ?? null,
55
+ ...markerTrace(subject.issue?.stateComment?.marker),
56
+ })
48
57
  /** @type {Transition[]} */
49
58
  const transitions = []
50
59
  let pos = at
51
60
  /** @type {Turn['reason']} */
52
61
  let reason
53
62
  for (;;) {
54
- const r = await advance(task, pos, c, { maxTransitions: 1 })
63
+ const r = await advance(task, pos, c, { maxTransitions: 1, instrument: traced.instrument })
55
64
  if (r.error) throw r.error.cause instanceof Error ? r.error.cause : r.error
56
65
  transitions.push(...r.transitions)
57
66
  const t = r.transitions[0]
67
+ traced.sync()
58
68
  if (t && t.to !== END) await nodeOf(t.to).enter?.(t.next, c)
59
69
  pos = { node: r.node, input: r.input }
60
70
  if (r.reason !== 'limit') {
@@ -14,6 +14,7 @@ import { iso, key, markerOf, MIN, setLabels, setMarker } from '../issue-state.mj
14
14
  import { move, Q, WHY } from '../labels.mjs'
15
15
  import { newMarker, withHistory } from '../marker.mjs'
16
16
  import { readTriage, TriageResult, triageComment, triagePrompt } from '../phases/dispatch.mjs'
17
+ import { traceWorker } from '../tracing.mjs'
17
18
  import { task } from './task.mjs'
18
19
 
19
20
  /**
@@ -167,8 +168,8 @@ const startTriage = declare(
167
168
  phase: 'dispatch',
168
169
  maxTurns: caps.maxTurns,
169
170
  readOnly: true,
170
- // TRACING (follow-up): as start-worker, the triage worker's env gets `TRACEPARENT: ctx.traceparent`
171
- // (this node's cambium.node span, under the cambium.issue root that continues mycelium's intake span).
171
+ // This node's cambium.node span, so Claude Code's own spans (when worker.env turns them on) join the trace.
172
+ env: ctx.traceparent ? { TRACEPARENT: ctx.traceparent } : undefined,
172
173
  buildPrompt: (agentsMd) => triagePrompt({ repo: slug, baseBranch: repo.baseBranch, issue, queued, agentsMd }),
173
174
  })
174
175
  } catch (err) {
@@ -191,6 +192,7 @@ const startTriage = declare(
191
192
  host: config.host,
192
193
  stdout: s.stdout,
193
194
  stderr: s.stderr,
195
+ traceparent: ctx.traceparent ?? null,
194
196
  })
195
197
  ctx.log('info', `started triage worker pid ${s.pid} (attempt ${marker.attempt})`, {
196
198
  repo: slug,
@@ -241,8 +243,7 @@ const triaging = declare(
241
243
  ctx.log('info', 'triage worker gone; issue is closed or left q:intake', { repo: w.repo, issue: w.issue })
242
244
  return { ready: true, output: { exit: 'gone' } }
243
245
  }
244
- // TRACING (follow-up): the worker has ended; emit its spans here with tracing.mjs
245
- // `workerSpans(<w.stdout log>, { tracer, parent: <start-triage's traceparent> })`.
246
+ traceWorker(ctx, w)
246
247
  const read = w.killedReason ? { ok: false, error: w.killedReason } : readTriage(spawner.readLog(w.stdout), number)
247
248
  if (read.ok && 'result' in read)
248
249
  return { ready: true, output: { exit: 'triaged', result: read.result, log: w.stdout } }
@@ -19,13 +19,17 @@ import { iso, setLabels, setMarker } from '../issue-state.mjs'
19
19
  import { move, Q } from '../labels.mjs'
20
20
  import { effectivePosition, newMarker, withHistory } from '../marker.mjs'
21
21
  import { releaseDue } from '../phases/release.mjs'
22
+ import { markerTrace, turnTracer } from '../tracing.mjs'
22
23
 
23
24
  /**
24
25
  * @typedef {import('../github.mjs').PromotionPr} PromotionPr
25
26
  * @typedef {import('../github.mjs').Release} Release
26
27
  * @typedef {import('../marker.mjs').Marker} Marker
27
- * @typedef {{ slug: string, repo: import('../config.mjs').Repo, release: Release, pr?: PromotionPr }} ReleaseSubject
28
- * One repo's turn; `pr` is the promotion PR a `release-promote` turn is about.
28
+ * @typedef {{
29
+ * slug: string, repo: import('../config.mjs').Repo, release: Release, pr?: PromotionPr, traceparent?: string | null,
30
+ * }} ReleaseSubject
31
+ * One repo's turn; `pr` is the promotion PR a `release-promote` turn is about; `traceparent` the turn's trace
32
+ * while it is traced (tracing.mjs `turnTracer`), which markers written in the turn take when they have none.
29
33
  * @typedef {import('../tick.mjs').Deps & { tick: import('../tick.mjs').Ctx, subject: ReleaseSubject }} ReleaseCtx
30
34
  * @typedef {{ queue: string | null, enter?: (input: any, ctx: ReleaseCtx) => Promise<void> }} Declared
31
35
  * `queue`: the label a PR resting here carries. `enter`: the write that lands a PR on this wait.
@@ -248,7 +252,6 @@ const merge = declare(
248
252
  */
249
253
  run: async (input, ctx) => {
250
254
  const pr = prOf(ctx)
251
- // TRACING (follow-up): a span per merge attempt here, parented on the PR marker's traceparent.
252
255
  try {
253
256
  await ctx.github.mergePull(ctx.subject.slug, input.pr)
254
257
  } catch (err) {
@@ -364,11 +367,21 @@ export const RELEASE_QUEUES = {
364
367
  * @param {{ node: string, input: unknown }} at
365
368
  */
366
369
  async function drive(graph, ctx, at) {
370
+ const { pr } = ctx.subject
371
+ // A promote turn is traced on its PR (the marker's trace); a prepare run is a root of its own, which the PR it
372
+ // opens or updates takes into its marker when that has none.
373
+ const traced = turnTracer(ctx.telemetry, ctx.subject, {
374
+ graph: graph.name,
375
+ graphVersion: graph.version,
376
+ repo: ctx.subject.slug,
377
+ issue: pr?.number,
378
+ ...markerTrace(pr?.stateComment?.marker),
379
+ })
367
380
  let pos = at
368
381
  for (;;) {
369
- // TRACING (follow-up): pass tracing.mjs `advanceHooks` as `instrument` here, as the task graph will.
370
- const r = await advance(graph, pos, ctx, { maxTransitions: 1 })
382
+ const r = await advance(graph, pos, ctx, { maxTransitions: 1, instrument: traced.instrument })
371
383
  if (r.error) throw r.error.cause instanceof Error ? r.error.cause : r.error
384
+ traced.sync()
372
385
  const t = r.transitions[0]
373
386
  const next = /** @type {(import('../graph/define.mjs').GraphNode & Declared) | undefined} */ (
374
387
  graph.nodes.find((n) => n.name === r.node)
@@ -15,6 +15,7 @@ import { move, Q, WHY } from '../labels.mjs'
15
15
  import { withHistory } from '../marker.mjs'
16
16
  import { prFor, taskExit, taskPrompt } from '../phases/task.mjs'
17
17
  import { PREPARE_FAILED_LINE } from '../prepare.mjs'
18
+ import { traceWorker } from '../tracing.mjs'
18
19
 
19
20
  /**
20
21
  * @typedef {import('../github.mjs').Issue} Issue
@@ -24,10 +25,11 @@ import { PREPARE_FAILED_LINE } from '../prepare.mjs'
24
25
  * @typedef {{
25
26
  * slug: string, number: number, issue?: Issue, snap?: Snapshot,
26
27
  * pr?: PullRequest | null, deleted?: Set<string>, foreground?: boolean, reaped?: Set<number>,
27
- * started?: import('../workers.mjs').Started & { attempt: number },
28
+ * started?: import('../workers.mjs').Started & { attempt: number }, traceparent?: string | null,
28
29
  * }} Subject
29
30
  * One issue's turn. `pr` is the PR Watch is looking at; `deleted` the branches already deleted this tick;
30
- * `foreground`/`reaped`/`started` are for `cambium run`.
31
+ * `foreground`/`reaped`/`started` are for `cambium run`; `traceparent` is the issue trace while the turn is
32
+ * traced (tracing.mjs `turnTracer`), which markers written in the turn take.
31
33
  * @typedef {import('../tick.mjs').Deps & {
32
34
  * tick: import('../tick.mjs').Ctx, subject: Subject, traceparent?: string | null,
33
35
  * }} TaskCtx
@@ -240,8 +242,8 @@ const startWorker = declare(
240
242
  phase: 'task',
241
243
  maxTurns: caps.maxTurns,
242
244
  foreground,
243
- // TRACING (follow-up): the worker's env gets `TRACEPARENT: ctx.traceparent` here (set by tracing.mjs
244
- // `advanceHooks` when advance runs with `instrument`), once the Spawner takes an env.
245
+ // This node's cambium.node span, so Claude Code's own spans (when worker.env turns them on) join the trace.
246
+ env: ctx.traceparent ? { TRACEPARENT: ctx.traceparent } : undefined,
245
247
  buildPrompt: (agentsMd) =>
246
248
  taskPrompt({ repo: slug, issue, marker, branch, pr, agentsMd, maxAttempts: caps.maxAttempts }),
247
249
  })
@@ -266,6 +268,7 @@ const startWorker = declare(
266
268
  host: config.host,
267
269
  stdout: s.stdout,
268
270
  stderr: s.stderr,
271
+ traceparent: ctx.traceparent ?? null,
269
272
  })
270
273
  const where = { repo: slug, issue: issue.number }
271
274
  if (s.reused) ctx.log('warn', `continuing in the previous worktree ${s.worktree}: ${s.reused}`, where)
@@ -314,8 +317,7 @@ const workerRunning = declare(
314
317
  ctx.log('info', 'worker gone; issue is closed or no longer queued', { repo: w.repo, issue: w.issue })
315
318
  return { ready: true, output: { exit: 'gone' } }
316
319
  }
317
- // TRACING (follow-up): the worker has ended; emit its spans here with tracing.mjs
318
- // `workerSpans(<w.stdout log>, { tracer, parent: <start-worker's traceparent> })`.
320
+ traceWorker(ctx, w)
319
321
  const pr = prFor(snap, w)
320
322
  const m = markerOf(issue)
321
323
  const prepFailed = !w.killedReason && spawner.tail(w.stderr, 15).includes(PREPARE_FAILED_LINE)
@@ -24,6 +24,10 @@ export async function setLabels(deps, slug, issue, change) {
24
24
 
25
25
  /** @param {Writers} deps @param {string} slug @param {Issue} issue @param {import('./marker.mjs').Marker} marker */
26
26
  export async function setMarker(deps, slug, issue, marker) {
27
+ // A traced graph turn keeps its issue trace on the subject (tracing.mjs `turnTracer`); a marker without one
28
+ // takes it, so later ticks parent on the same trace. One that already has a trace keeps it.
29
+ const tp = /** @type {any} */ (deps).subject?.traceparent
30
+ if (tp && !marker.traceparent) marker = { ...marker, traceparent: tp }
27
31
  const maxAttempts = deps.config.phases[marker.phase]?.maxAttempts
28
32
  const id = await deps.github.writeState(
29
33
  slug,
package/src/store.mjs CHANGED
@@ -13,7 +13,9 @@ import { DatabaseSync } from 'node:sqlite'
13
13
  * @typedef {{
14
14
  * repo: string, issue: number, phase: string, pid: number, worktree: string, branch: string,
15
15
  * attempt: number, startedAt: number, host: string, stdout: string, stderr: string, killedReason: string | null,
16
+ * traceparent: string | null,
16
17
  * }} Worker
18
+ * `traceparent`: the span of the step that started it, when traced; its `claude.*` spans go under it on exit.
17
19
  */
18
20
 
19
21
  export const MODES = /** @type {const} */ (['running', 'paused', 'draining'])
@@ -22,7 +24,7 @@ const SCHEMA = `
22
24
  CREATE TABLE IF NOT EXISTS workers (
23
25
  repo TEXT NOT NULL, issue INTEGER NOT NULL, phase TEXT NOT NULL, pid INTEGER NOT NULL,
24
26
  worktree TEXT NOT NULL, branch TEXT NOT NULL, attempt INTEGER NOT NULL, started_at INTEGER NOT NULL,
25
- host TEXT NOT NULL, stdout TEXT NOT NULL, stderr TEXT NOT NULL, killed_reason TEXT,
27
+ host TEXT NOT NULL, stdout TEXT NOT NULL, stderr TEXT NOT NULL, killed_reason TEXT, traceparent TEXT,
26
28
  PRIMARY KEY (repo, issue)
27
29
  );
28
30
  CREATE TABLE IF NOT EXISTS control (key TEXT PRIMARY KEY, value TEXT NOT NULL);
@@ -42,6 +44,7 @@ const toWorker = (r) => ({
42
44
  stdout: r.stdout,
43
45
  stderr: r.stderr,
44
46
  killedReason: r.killed_reason ?? null,
47
+ traceparent: r.traceparent ?? null,
45
48
  })
46
49
 
47
50
  /** @param {string} path a file, or `:memory:` */
@@ -50,6 +53,9 @@ export function openStore(path) {
50
53
  const db = new DatabaseSync(path)
51
54
  db.exec('PRAGMA journal_mode = WAL; PRAGMA busy_timeout = 5000;')
52
55
  db.exec(SCHEMA)
56
+ // Databases from before `traceparent` existed.
57
+ const cols = /** @type {{ name: string }[]} */ (db.prepare('PRAGMA table_info(workers)').all())
58
+ if (!cols.some((c) => c.name === 'traceparent')) db.exec('ALTER TABLE workers ADD COLUMN traceparent TEXT')
53
59
  const get = (/** @type {string} */ key) =>
54
60
  /** @type {any} */ (db.prepare('SELECT value FROM control WHERE key = ?').get(key))?.value ?? null
55
61
  const set = (/** @type {string} */ key, /** @type {string} */ value) =>
@@ -65,12 +71,25 @@ export function openStore(path) {
65
71
  const r = db.prepare('SELECT * FROM workers WHERE repo = ? AND issue = ?').get(repo, issue)
66
72
  return r ? toWorker(r) : null
67
73
  },
68
- /** @param {Omit<Worker, 'killedReason'>} w */
74
+ /** @param {Omit<Worker, 'killedReason' | 'traceparent'> & { traceparent?: string | null }} w */
69
75
  addWorker(w) {
70
76
  db.prepare(
71
- `INSERT OR REPLACE INTO workers (repo, issue, phase, pid, worktree, branch, attempt, started_at, host, stdout, stderr, killed_reason)
72
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL)`,
73
- ).run(w.repo, w.issue, w.phase, w.pid, w.worktree, w.branch, w.attempt, w.startedAt, w.host, w.stdout, w.stderr)
77
+ `INSERT OR REPLACE INTO workers (repo, issue, phase, pid, worktree, branch, attempt, started_at, host, stdout, stderr, killed_reason, traceparent)
78
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, ?)`,
79
+ ).run(
80
+ w.repo,
81
+ w.issue,
82
+ w.phase,
83
+ w.pid,
84
+ w.worktree,
85
+ w.branch,
86
+ w.attempt,
87
+ w.startedAt,
88
+ w.host,
89
+ w.stdout,
90
+ w.stderr,
91
+ w.traceparent ?? null,
92
+ )
74
93
  },
75
94
  /** @param {string} repo @param {number} issue @param {string} reason */
76
95
  markKilled(repo, issue, reason) {
package/src/tick.mjs CHANGED
@@ -35,6 +35,7 @@ import { DISPATCH_NOT_BUILT, dispatchNotBuilt } from './phases/dispatch.mjs'
35
35
  * dispatch?: import('./phases/dispatch.mjs').Dispatch,
36
36
  * release?: import('./phases/release.mjs').Release,
37
37
  * tickNumber?: number,
38
+ * telemetry?: import('./tracing.mjs').Telemetry,
38
39
  * daemonStartedAt?: number,
39
40
  * pid?: number,
40
41
  * }} Deps
package/src/tracing.mjs CHANGED
@@ -290,6 +290,62 @@ export function advanceHooks(telemetry, base) {
290
290
  }
291
291
  }
292
292
 
293
+ /**
294
+ * Tracing for one turn of a graph driver (graph-engine `drive`, dispatch-engine `driveDispatch`, release `drive`).
295
+ * Wraps {@link advanceHooks} and keeps `holder.traceparent` (the turn's subject) on the issue trace, so a marker
296
+ * written during the turn stores it (issue-state.mjs `setMarker`). Call `sync()` before writes made outside
297
+ * `advance` (a wait's `enter`). With telemetry off, or none, `instrument` is undefined and `advance` runs exactly
298
+ * as it does untraced.
299
+ *
300
+ * @param {Telemetry | undefined} telemetry
301
+ * @param {{ traceparent?: string | null }} holder
302
+ * @param {TraceInfo & { enteredAt?: string | null }} base as for {@link advanceHooks}
303
+ * @returns {{ instrument: import('./graph/advance.mjs').Instrument | undefined, sync: () => void }}
304
+ */
305
+ export function turnTracer(telemetry, holder, base) {
306
+ if (!telemetry?.enabled) return { instrument: undefined, sync: () => {} }
307
+ const hooks = advanceHooks(telemetry, base)
308
+ const sync = () => {
309
+ if (hooks.traceparent) holder.traceparent = hooks.traceparent
310
+ }
311
+ return {
312
+ instrument: (visit, step) =>
313
+ hooks.instrument(visit, (extra) => {
314
+ sync()
315
+ return step(extra)
316
+ }),
317
+ sync,
318
+ }
319
+ }
320
+
321
+ /**
322
+ * The marker fields a turn's {@link turnTracer} starts from. No marker yet is a fresh issue: attempt 1, as
323
+ * marker.mjs `newMarker` has it.
324
+ * @param {{ traceparent?: string | null, enteredAt?: string | null, attempt?: number } | null | undefined} marker
325
+ */
326
+ export const markerTrace = (marker) => ({
327
+ traceparent: marker?.traceparent ?? null,
328
+ enteredAt: marker?.enteredAt ?? null,
329
+ attempt: marker?.attempt ?? 1,
330
+ })
331
+
332
+ /**
333
+ * A finished worker's `claude.*` spans from its stream-json stdout log, under the step span that started it
334
+ * (`parent`, kept on the worker row). Never throws: tracing must not fail a tick.
335
+ *
336
+ * @param {{ telemetry?: Telemetry, spawner: { readLog(path: string): string },
337
+ * log: (level: 'info' | 'warn' | 'error', msg: string, where?: { repo?: string, issue?: number }) => void }} ctx
338
+ * @param {{ repo: string, issue: number, stdout: string, traceparent?: string | null }} w
339
+ */
340
+ export function traceWorker(ctx, w) {
341
+ if (!ctx.telemetry?.enabled || !w.traceparent) return
342
+ try {
343
+ ctx.telemetry.workerSpans(ctx.spawner.readLog(w.stdout), w.traceparent)
344
+ } catch (err) {
345
+ ctx.log('warn', `worker spans: ${/** @type {Error} */ (err)?.message ?? err}`, { repo: w.repo, issue: w.issue })
346
+ }
347
+ }
348
+
293
349
  /** A node output's own `outcome` string when it has one, else `ok`. @param {unknown} output */
294
350
  const outcomeOf = (output) => {
295
351
  const o = /** @type {any} */ (output)?.outcome
package/src/workers.mjs CHANGED
@@ -28,10 +28,11 @@ import { exec as defaultExec } from './github.mjs'
28
28
  * @typedef {{
29
29
  * repo: import('./config.mjs').Repo, convention: Convention, issue: number, attempt: number, branch: string,
30
30
  * phase: string, maxTurns: number, buildPrompt: (agentsMd: string | null) => string, foreground?: boolean,
31
- * readOnly?: boolean,
31
+ * readOnly?: boolean, env?: Record<string, string>,
32
32
  * }} StartSpec
33
33
  * `readOnly`: a disposable checkout for a worker that only reads (triage): detached at the base branch, no
34
34
  * install, hydrate or prepare, no Edit/Write; `branch` only names its directory (see {@link topicOf}).
35
+ * `env`: added to the worker's environment after `config.worker.env` (the step's `TRACEPARENT`).
35
36
  * @typedef {{
36
37
  * pid: number, worktree: string, stdout: string, stderr: string, prompt: string, exited: Promise<number | null>,
37
38
  * reused: string | null,
@@ -149,6 +150,22 @@ export function processSpawner({
149
150
  const git = (/** @type {string} */ repoPath, /** @type {string[]} */ args) => exec('git', ['-C', repoPath, ...args])
150
151
  const graftEnv = { ...process.env, ...(config.graft.app ? { GRAFT_APP: config.graft.app } : {}) }
151
152
 
153
+ /**
154
+ * Fast-forward the central checkout to the just-fetched base, so the convention
155
+ * and hydrate sources a worker gets are current. Only when that checkout sits on
156
+ * the base branch with no tracked changes; `--ff-only` refuses rather than discards.
157
+ *
158
+ * @param {import('./config.mjs').Repo} repo @returns {Promise<boolean>} whether it moved
159
+ */
160
+ async function syncCentral(repo) {
161
+ const head = (await git(repo.path, ['symbolic-ref', '--short', '-q', 'HEAD']).catch(() => '')).trim()
162
+ if (head !== repo.baseBranch) return false
163
+ if ((await git(repo.path, ['status', '--porcelain', '--untracked-files=no'])).trim()) return false
164
+ const before = (await git(repo.path, ['rev-parse', 'HEAD'])).trim()
165
+ await git(repo.path, ['merge', '--ff-only', '--quiet', `origin/${repo.baseBranch}`]).catch(() => {})
166
+ return (await git(repo.path, ['rev-parse', 'HEAD'])).trim() !== before
167
+ }
168
+
152
169
  /**
153
170
  * Remove a worktree only once `graft status` says GitHub holds everything in
154
171
  * it; otherwise keep it and say why. Its local branch goes with it.
@@ -205,9 +222,19 @@ export function processSpawner({
205
222
  buildPrompt,
206
223
  foreground = false,
207
224
  readOnly = false,
225
+ env: extraEnv = {},
208
226
  }) {
209
227
  const worktree = join(convention.root, topicOf(branch))
210
- if (config.worker.fetch) await git(repo.path, ['fetch', 'origin', repo.baseBranch])
228
+ if (config.worker.fetch) {
229
+ await git(repo.path, ['fetch', 'origin', repo.baseBranch])
230
+ if (await syncCentral(repo)) {
231
+ try {
232
+ convention = readConvention(repo.path)
233
+ } catch {
234
+ // The updated package.json does not parse: keep the convention this tick read.
235
+ }
236
+ }
237
+ }
211
238
  /** @type {string | null} */
212
239
  let reused = null
213
240
  if (readOnly) await dispose(repo, worktree)
@@ -255,6 +282,9 @@ export function processSpawner({
255
282
  },
256
283
  ),
257
284
  ...(config.worker.graftApp ? { GRAFT_APP: config.worker.graftApp } : {}),
285
+ // config.worker.env (e.g. Claude Code's own telemetry), then this start's (TRACEPARENT).
286
+ ...config.worker.env,
287
+ ...extraEnv,
258
288
  }
259
289
  /** @type {import('node:child_process').ChildProcess} */
260
290
  let child