@oakgreencc/cambium 0.0.6 → 0.0.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -4,7 +4,7 @@ The scheduler daemon. Every tick it reads the `q:*` queues from GitHub, starts h
4
4
 
5
5
  This is the **Task phase** MVP. Dispatch and the daily Release run are seams (see [Follow-ups](#follow-ups)).
6
6
 
7
- One runtime dependency (`zod`, for the [graph module](#graph-module-preview)), Node ≥ 22.13 (`node:sqlite`). GitHub access goes through [`graft`](../graft), which must be on `PATH`.
7
+ Runtime dependencies: `zod` (the [graph module](#graph-module-preview)) and OpenTelemetry via `@oakgreencc/mycelium` (loaded only when [tracing](#tracing) is on); Node ≥ 22.13 (`node:sqlite`). GitHub access goes through [`graft`](../graft), which must be on `PATH`.
8
8
 
9
9
  ## Quick start
10
10
 
@@ -258,6 +258,105 @@ const { transitions, reason, node, input } = await advance(g, { node: 'open-pr',
258
258
  - **`validateGraph(graph)`** returns every problem as a string (`defineGraph` throws on any): unknown edge endpoints, nodes unreachable from `entry.node`, a node with outgoing edges whose last edge is conditional (no fallback), duplicate names, a non-integer `version`.
259
259
  - **`advance(graph, { node, input }, ctx, { maxTransitions = 8, signal })`** runs step nodes and ready waits and returns `{ transitions, reason, node, input, error? }`: `reason` is `waiting` (a wait that is not ready), `end`, `limit`, `aborted` (`signal`, or `ctx.signal`, checked before each node) or `error`; `node`/`input` is where it stopped. Each transition is `{ from, to, edge, input, output, next }`. `ctx` is passed to `run`/`poll` untouched apart from `ctx.signal`; edges see `ctx.subject`.
260
260
  - **`toMermaid(graph)`** renders a flowchart: boxes for steps, hexagons for waits, conditional edges labelled with `when`'s function name.
261
+ - **`opts.instrument(visit, step)`** (optional) wraps each node visit: `visit` is `{ node, kind: 'node' | 'wait', input }`; `step(extraCtx?)` runs the node (or poll) and picks the edge, returning the transition, or `null` for a wait that is not ready, and throws on a node error. `extraCtx` is merged into that node's `ctx`. Return what `step` returned. Without it, `advance` behaves exactly as before. [`advanceHooks`](#tracing) is the tracing implementation.
262
+
263
+ ## Tracing
264
+
265
+ `src/tracing.mjs` is OpenTelemetry for cambium, configured only by the standard `OTEL_*` variables with `service.name=cambium`. It reuses mycelium's `createTracing` (same packages, same on/off rule): **on only when `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set**, unless `OTEL_SDK_DISABLED=true` or `OTEL_TRACES_EXPORTER=none`; otherwise nothing is imported and every call is a no-op. Metrics follow the same rule for their signal (`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_METRICS_EXPORTER`). With an endpoint but no collector, shutdown can take ~8 s; set `OTEL_EXPORTER_OTLP_TIMEOUT=2000` for the daemon.
266
+
267
+ Why not a shared package: cambium is published to npm, so it can only depend on published packages; a new publishable `@oakgreencc/telemetry` needs a protected `nx.json` release change. mycelium already publishes `createTracing`, so it gained a `serviceName` option and cambium depends on `@oakgreencc/mycelium` (which also gives it `parseTraceparent` for the intake trace).
268
+
269
+ | Span / metric | When | Parent |
270
+ | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
271
+ | `cambium.issue` | Once per issue, when the marker has no `traceparent`. Ended at once; its traceparent goes in the marker. | mycelium's intake span from the issue body, else none (a root) |
272
+ | `cambium.node` | Each step-node run | The marker's `traceparent` |
273
+ | `cambium.wait` | When a wait ends; `startTime` = the marker's `enteredAt`. No span stays open across ticks. | The marker's `traceparent` |
274
+ | `cambium.wait.not_ready` | Counter, +1 per not-ready poll (attributes graph, node, repo) | |
275
+ | `claude.session` / `.tool` / `.task` | From the worker's stream-json log, see below | The start-worker `cambium.node` span |
276
+
277
+ Node and wait spans carry `cambium.graph`, `cambium.graph.version`, `cambium.node`, `cambium.attempt`, `cambium.repo`, `cambium.issue.number`, `cambium.edge`, `cambium.outcome`.
278
+
279
+ ```js
280
+ import { advanceHooks, createTelemetry } from './tracing.mjs' // inside cambium
281
+
282
+ const telemetry = await createTelemetry() // { env, exporter, metricReader, now } are for tests
283
+ try {
284
+ const hooks = advanceHooks(telemetry, {
285
+ graph,
286
+ graphVersion,
287
+ attempt,
288
+ traceparent,
289
+ enteredAt,
290
+ repo,
291
+ issue,
292
+ issueBody,
293
+ })
294
+ const r = await advance(g, at, ctx, { instrument: hooks.instrument })
295
+ marker.traceparent = hooks.traceparent // new when a cambium.issue root was started
296
+ } finally {
297
+ await telemetry.shutdown()
298
+ }
299
+ ```
300
+
301
+ - `telemetry.nodeSpan(info, fn)`: runs `fn(run)` inside a `cambium.node` span. `run` is `{ span, issueTraceparent, traceparent, set({ edge, outcome }) }`: `issueTraceparent` is what the marker should hold; `traceparent` is this span's, to give a worker as `TRACEPARENT`. A throw sets ERROR and outcome `error` and is rethrown.
302
+ - `telemetry.waitSpan({ ...info, enteredAt, polls? })`: emits the `cambium.wait` span; returns the issue traceparent.
303
+ - `telemetry.notReady(info)`, `telemetry.issueTrace(info)` (the marker traceparent, starting a root if needed), `traceparentFor(spanContext)`, `flush()`, `shutdown()`.
304
+ - `advanceHooks(telemetry, base)` → `{ instrument, traceparent }`: step nodes run in `nodeSpan` and get **`ctx.traceparent`** (start-worker sets the worker's `TRACEPARENT` from it); `cambium.outcome` is the output's `outcome` string, else `ok`. A ready wait emits `waitSpan` (from `enteredAt`, or from the previous transition within one advance); a not-ready one bumps the counter.
305
+
306
+ ### Worker spans from stream-json
307
+
308
+ Worker telemetry comes from the worker's own stdout log (`claude -p --output-format stream-json --verbose`), not from Claude Code's OTel export, so it works with telemetry off on the worker side:
309
+
310
+ ```js
311
+ workerSpans(stream, { tracer, parent }) → { sessionId, complete, ok, start, end, tools, tasks, malformed, costUsd }
312
+ telemetry.workerSpans(stream, parent) // same, with cambium's tracer
313
+ ```
314
+
315
+ `stream` is the log text, its lines, or parsed events; `parent` is the start-worker span's context or traceparent. Call it once the worker has exited; every span it makes is already ended.
316
+
317
+ - `claude.session`: first to last `timestamp`. Attributes from `result` and `system/init`: `claude.session_id`, `claude.model`, `claude_code.version`, `claude.cost_usd`, `claude.tokens.{input,output,cache_read,cache_creation}`, `claude.turns`, `claude.duration_ms`, `claude.terminal_reason`, `claude.result`, `claude.permission_denials`. No `result` (killed, truncated) or an error result → ERROR.
318
+ - `claude.tool`: each `tool_use` → `tool_result` by id (a subagent's tools nest under its Agent tool span); `claude.tool.name`, `claude.tool.outcome` (`ok` / `error` / `unfinished`), `claude.tool.command` (Bash, clipped to 200 chars). Unpaired → ends at the last timestamp, ERROR.
319
+ - `claude.task`: each background task, `task_started` → `task_notification` (or `task_updated.patch.end_time`), under the tool that started it. These events have no timestamp, so their times are those of the latest timestamped event.
320
+ - Malformed lines are counted (`malformed`) and skipped; `system/thinking_tokens`, `tool_progress`, hooks and rate-limit events are ignored.
321
+
322
+ ### Does `claude -p` export traces that honour `TRACEPARENT`?
323
+
324
+ **Yes, opt-in and beta**; cambium does not depend on it (worker spans come from stream-json above). Per the [Claude Code monitoring docs](https://code.claude.com/docs/en/monitoring-usage) (checked with Claude Code 2.1.327; `claude --help` lists no telemetry flags, it is env-only):
325
+
326
+ - Traces need `CLAUDE_CODE_ENABLE_TELEMETRY=1` **and** `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` plus `OTEL_TRACES_EXPORTER=otlp` and the usual `OTEL_EXPORTER_OTLP_*` endpoint/protocol (use `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` against this collector). Spans: `claude_code.interaction` → `claude_code.llm_request`, `claude_code.tool` (→ `.execution`, `.blocked_on_user`).
327
+ - "In Agent SDK and non-interactive sessions started with `-p`, Claude Code also reads `TRACEPARENT` and `TRACESTATE` from its own environment when starting each interaction span": the interaction span becomes a child of the caller's span (`parent.source=env`). Interactive sessions ignore an inbound `TRACEPARENT`.
328
+ - Bash subprocesses it runs inherit a `TRACEPARENT` for the active tool span, so a `graft` call inside a worker could continue the trace.
329
+ - Prompts and tool inputs are redacted from spans unless `OTEL_LOG_USER_PROMPTS` / `OTEL_LOG_TOOL_DETAILS` / `OTEL_LOG_TOOL_CONTENT` are set.
330
+
331
+ So to also get Claude's own spans in the issue trace, add `CLAUDE_CODE_ENABLE_TELEMETRY=1`, `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`, `OTEL_TRACES_EXPORTER=otlp` and the endpoint to `worker.env`; start-worker already sets `TRACEPARENT`. Not verified against a live run here.
332
+
333
+ ### Local collector
334
+
335
+ [`otel/`](otel/) holds an [otelcol-contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) config (OTLP/HTTP on `127.0.0.1:4318` → `~/.cambium/otel/{traces,metrics}.jsonl` + Jaeger) and a launchd agent, `com.oakgreencc.otelcol`. Nothing here loads it.
336
+
337
+ ```sh
338
+ # 1. otelcol-contrib on PATH (darwin_arm64 release tarball; or darwin_amd64)
339
+ V=0.x.y # latest from the releases page
340
+ curl -fsSLO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v$V/otelcol-contrib_${V}_darwin_arm64.tar.gz
341
+ tar -xzf otelcol-contrib_${V}_darwin_arm64.tar.gz otelcol-contrib && mv otelcol-contrib /opt/homebrew/bin/
342
+
343
+ # 2. config + agent
344
+ mkdir -p ~/.cambium/otel
345
+ cp otel/otelcol.yaml ~/.cambium/otel/otelcol.yaml
346
+ otelcol-contrib validate --config ~/.cambium/otel/otelcol.yaml
347
+ cp otel/com.oakgreencc.otelcol.plist ~/Library/LaunchAgents/
348
+ launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.oakgreencc.otelcol.plist
349
+
350
+ # 3. Jaeger UI on :16686, OTLP/gRPC on 14317 (its 4318 would clash with the collector)
351
+ docker run -d --name jaeger -p 16686:16686 -p 14317:4317 jaegertracing/jaeger:latest
352
+
353
+ # 4. point cambium (and mycelium) at it, e.g. in the cambium launchd plist's environment
354
+ OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 OTEL_EXPORTER_OTLP_TIMEOUT=2000
355
+ ```
356
+
357
+ Without Jaeger running the collector still writes the files; the Jaeger exporter just logs failures. Stop it with `launchctl bootout gui/$(id -u)/com.oakgreencc.otelcol`.
358
+
359
+ **AWS** is a collector-config change only: [`otel/otelcol-aws.yaml`](otel/otelcol-aws.yaml) keeps the same receiver and sends traces to X-Ray's OTLP endpoint (`https://xray.<region>.amazonaws.com/v1/traces`, `otlphttp` + `sigv4auth` with `service: xray`; needs X-Ray Transaction Search enabled) and metrics to Amazon Managed Service for Prometheus (`prometheusremotewrite` + `sigv4auth` with `service: aps`). Credentials come from the default AWS chain of the collector's process. cambium's env stays `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318`.
261
360
 
262
361
  ## Lessons from the old Clock
263
362
 
@@ -0,0 +1,33 @@
1
+ <?xml version="1.0" encoding="UTF-8"?>
2
+ <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3
+ <!--
4
+ The local OpenTelemetry collector as a launchd user agent. Not loaded by
5
+ anything in this repo: copy it to ~/Library/LaunchAgents and bootstrap it
6
+ yourself (see ../README.md#tracing). It reads
7
+ ~/.cambium/otel/otelcol.yaml and logs to ~/.cambium/otel/otelcol.log.
8
+ -->
9
+ <plist version="1.0">
10
+ <dict>
11
+ <key>Label</key>
12
+ <string>com.oakgreencc.otelcol</string>
13
+ <key>ProgramArguments</key>
14
+ <array>
15
+ <string>/bin/sh</string>
16
+ <string>-c</string>
17
+ <string>exec otelcol-contrib --config "$HOME/.cambium/otel/otelcol.yaml" &gt;&gt;"$HOME/.cambium/otel/otelcol.log" 2&gt;&amp;1</string>
18
+ </array>
19
+ <key>EnvironmentVariables</key>
20
+ <dict>
21
+ <key>PATH</key>
22
+ <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
23
+ </dict>
24
+ <key>RunAtLoad</key>
25
+ <true/>
26
+ <key>KeepAlive</key>
27
+ <true/>
28
+ <key>ThrottleInterval</key>
29
+ <integer>10</integer>
30
+ <key>ProcessType</key>
31
+ <string>Background</string>
32
+ </dict>
33
+ </plist>
@@ -0,0 +1,52 @@
1
+ # AWS variant: the same OTLP/HTTP receiver; traces go to AWS X-Ray's OTLP
2
+ # endpoint and metrics to Amazon Managed Service for Prometheus, both signed
3
+ # with SigV4 from the default AWS credential chain. cambium itself does not
4
+ # change: only the collector config does. Fill in REGION and WORKSPACE_ID.
5
+ #
6
+ # Prerequisites: X-Ray Transaction Search enabled in the account (the OTLP
7
+ # endpoint requires it); an AMP workspace; credentials allowed
8
+ # xray:PutTraceSegments (and the OTLP span ingest permission for Transaction
9
+ # Search) plus aps:RemoteWrite on the workspace.
10
+ extensions:
11
+ sigv4auth/xray:
12
+ region: REGION
13
+ service: xray
14
+ sigv4auth/aps:
15
+ region: REGION
16
+ service: aps
17
+
18
+ receivers:
19
+ otlp:
20
+ protocols:
21
+ http:
22
+ endpoint: 127.0.0.1:4318
23
+
24
+ processors:
25
+ batch: {}
26
+ memory_limiter:
27
+ check_interval: 1s
28
+ limit_mib: 256
29
+
30
+ exporters:
31
+ otlphttp/xray:
32
+ traces_endpoint: https://xray.REGION.amazonaws.com/v1/traces
33
+ auth:
34
+ authenticator: sigv4auth/xray
35
+ prometheusremotewrite/amp:
36
+ endpoint: https://aps-workspaces.REGION.amazonaws.com/workspaces/WORKSPACE_ID/api/v1/remote_write
37
+ auth:
38
+ authenticator: sigv4auth/aps
39
+ resource_to_telemetry_conversion:
40
+ enabled: true
41
+
42
+ service:
43
+ extensions: [sigv4auth/xray, sigv4auth/aps]
44
+ pipelines:
45
+ traces:
46
+ receivers: [otlp]
47
+ processors: [memory_limiter, batch]
48
+ exporters: [otlphttp/xray]
49
+ metrics:
50
+ receivers: [otlp]
51
+ processors: [memory_limiter, batch]
52
+ exporters: [prometheusremotewrite/amp]
@@ -0,0 +1,50 @@
1
+ # Local collector for cambium (and mycelium): OTLP/HTTP in on :4318,
2
+ # traces and metrics out to JSON-lines files, traces also to a local Jaeger.
3
+ # See ../README.md#tracing for install and the AWS variant (otelcol-aws.yaml).
4
+ receivers:
5
+ otlp:
6
+ protocols:
7
+ http:
8
+ endpoint: 127.0.0.1:4318
9
+
10
+ processors:
11
+ batch: {}
12
+ memory_limiter:
13
+ check_interval: 1s
14
+ limit_mib: 256
15
+
16
+ exporters:
17
+ file/traces:
18
+ path: ${env:HOME}/.cambium/otel/traces.jsonl
19
+ rotation:
20
+ max_megabytes: 50
21
+ max_backups: 5
22
+ file/metrics:
23
+ path: ${env:HOME}/.cambium/otel/metrics.jsonl
24
+ rotation:
25
+ max_megabytes: 50
26
+ max_backups: 5
27
+ # Jaeger takes OTLP natively. Its own OTLP/HTTP port (4318) would clash with
28
+ # this collector, so run it with OTLP/gRPC published on 14317 (see README).
29
+ otlp/jaeger:
30
+ endpoint: 127.0.0.1:14317
31
+ tls:
32
+ insecure: true
33
+ retry_on_failure:
34
+ enabled: false
35
+ sending_queue:
36
+ enabled: true
37
+
38
+ service:
39
+ telemetry:
40
+ logs:
41
+ level: warn
42
+ pipelines:
43
+ traces:
44
+ receivers: [otlp]
45
+ processors: [memory_limiter, batch]
46
+ exporters: [file/traces, otlp/jaeger]
47
+ metrics:
48
+ receivers: [otlp]
49
+ processors: [memory_limiter, batch]
50
+ exporters: [file/metrics]
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@oakgreencc/cambium",
3
- "version": "0.0.6",
3
+ "version": "0.0.7",
4
4
  "description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
5
5
  "license": "AGPL-3.0-only",
6
6
  "type": "module",
@@ -30,6 +30,7 @@
30
30
  },
31
31
  "files": [
32
32
  "bin/",
33
+ "otel/",
33
34
  "src/"
34
35
  ],
35
36
  "engines": {
@@ -43,7 +44,13 @@
43
44
  "name": "cambium"
44
45
  },
45
46
  "dependencies": {
46
- "zod": "^4.6.7"
47
+ "@opentelemetry/api": "^1.10.0",
48
+ "@opentelemetry/exporter-metrics-otlp-http": "^0.301.2",
49
+ "@opentelemetry/resources": "^3.1.1",
50
+ "@opentelemetry/sdk-metrics": "^3.1.1",
51
+ "@opentelemetry/sdk-trace": "^3.1.1",
52
+ "zod": "^4.6.7",
53
+ "@oakgreencc/mycelium": "0.0.5"
47
54
  },
48
55
  "scripts": {
49
56
  "lint": "biome check .",
@@ -9,6 +9,10 @@ import { check, END, GraphNodeError } from './define.mjs'
9
9
  * @typedef {'waiting' | 'end' | 'limit' | 'aborted' | 'error'} StopReason
10
10
  * @typedef {{ transitions: Transition[], reason: StopReason, node: string, input: unknown, error?: GraphNodeError }} Advanced
11
11
  * `node`/`input` is where it stopped: the held wait, END with the last output, the next node to run, or the failed node.
12
+ * @typedef {{ node: string, kind: 'node' | 'wait', input: unknown }} Visit
13
+ * @typedef {(visit: Visit, step: (extraCtx?: Record<string, unknown>) => Promise<Transition | null>) => Promise<Transition | null>} Instrument
14
+ * Wraps each node visit (tracing). `step` runs the node (or poll) and picks the edge: a Transition, or null for a
15
+ * wait that is not ready; it throws on a node error. `extraCtx` is merged into the node's ctx. Return what `step` returned.
12
16
  */
13
17
 
14
18
  /** @param {unknown} err @param {string} node @param {'run' | 'poll' | 'edge'} phase */
@@ -37,7 +41,7 @@ async function call(node, phase, fn) {
37
41
  * @param {import('./define.mjs').Graph} graph
38
42
  * @param {Position} at
39
43
  * @param {Record<string, unknown>} [ctx]
40
- * @param {{ maxTransitions?: number, signal?: AbortSignal }} [opts]
44
+ * @param {{ maxTransitions?: number, signal?: AbortSignal, instrument?: Instrument }} [opts]
41
45
  * @returns {Promise<Advanced>}
42
46
  */
43
47
  export async function advance(graph, at, ctx = {}, opts = {}) {
@@ -56,22 +60,29 @@ export async function advance(graph, at, ctx = {}, opts = {}) {
56
60
  if (transitions.length >= maxTransitions) return stop('limit')
57
61
  const node = byName.get(name)
58
62
  if (!node) return stop('error', new GraphNodeError(name, 'run', 'no such node in the graph'))
59
- try {
63
+ /** @type {(extraCtx?: Record<string, unknown>) => Promise<Transition | null>} */
64
+ const step = async (extraCtx) => {
65
+ const c = extraCtx ? { ...nodeCtx, ...extraCtx } : nodeCtx
60
66
  const valid = await check(node.input, input, name, 'input')
61
67
  let output
62
68
  if (node.kind === 'wait') {
63
- const polled = await call(name, 'poll', () => node.poll(valid, nodeCtx))
69
+ const polled = await call(name, 'poll', () => node.poll(valid, c))
64
70
  if (typeof polled?.ready !== 'boolean')
65
71
  throw new GraphNodeError(name, 'poll', 'poll must return { ready: false } or { ready: true, output }')
66
- if (!polled.ready) return stop('waiting')
72
+ if (!polled.ready) return null
67
73
  output = await check(node.output, polled.output, name, 'output')
68
74
  } else {
69
- output = await check(node.output, await call(name, 'run', () => node.run(valid, nodeCtx)), name, 'output')
75
+ output = await check(node.output, await call(name, 'run', () => node.run(valid, c)), name, 'output')
70
76
  }
71
77
  const { edge, to, next } = pick(graph, name, output, ctx.subject)
72
- transitions.push({ from: name, to, edge, input, output, next })
73
- name = to
74
- input = next
78
+ return { from: name, to, edge, input, output, next }
79
+ }
80
+ try {
81
+ const t = opts.instrument ? await opts.instrument({ node: name, kind: node.kind, input }, step) : await step()
82
+ if (!t) return stop('waiting')
83
+ transitions.push(t)
84
+ name = t.to
85
+ input = t.next
75
86
  } catch (err) {
76
87
  return stop('error', asNodeError(err, name, 'edge'))
77
88
  }
@@ -0,0 +1,522 @@
1
+ // @ts-check
2
+ /**
3
+ * OpenTelemetry for cambium: `cambium.node` spans per step-node run,
4
+ * `cambium.wait` spans per wait period (emitted when it ends), a root
5
+ * `cambium.issue` span per issue trace, and a `cambium.wait.not_ready` counter.
6
+ *
7
+ * Configured only by the standard `OTEL_*` env vars, with mycelium's on/off
8
+ * rule (its `createTracing` is reused with `service.name=cambium`): no OTLP
9
+ * endpoint, `OTEL_SDK_DISABLED=true` or `OTEL_TRACES_EXPORTER=none` → no-op,
10
+ * and the SDK is never imported. Metrics follow the same rule for their own
11
+ * signal (`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_METRICS_EXPORTER`).
12
+ *
13
+ * No span stays open across ticks: the issue root is ended as soon as it is
14
+ * started (its traceparent, stored in the marker, is what later ticks parent
15
+ * on), and a wait span is created only once the wait is over, back-dated to
16
+ * the marker's `enteredAt`.
17
+ */
18
+ import { metrics, ROOT_CONTEXT, SpanStatusCode, trace } from '@opentelemetry/api'
19
+ import {
20
+ createTracing,
21
+ formatTraceparent,
22
+ NO_TRACING,
23
+ otlpConfigured,
24
+ parseTraceparent,
25
+ spanContextOf,
26
+ } from '@oakgreencc/mycelium'
27
+
28
+ export const SERVICE_NAME = 'cambium'
29
+
30
+ /**
31
+ * @typedef {import('@opentelemetry/api').Span} Span
32
+ * @typedef {import('@opentelemetry/api').SpanContext} SpanContext
33
+ * @typedef {import('@opentelemetry/api').Counter} Counter
34
+ * @typedef {{
35
+ * graph?: string, graphVersion?: number, node?: string, attempt?: number,
36
+ * repo?: string, issue?: number, edge?: number | string | null, outcome?: string,
37
+ * }} NodeAttrs
38
+ * @typedef {NodeAttrs & {
39
+ * traceparent?: string | null,
40
+ * issueBody?: string | null,
41
+ * }} TraceInfo
42
+ * `traceparent` is the marker's; `issueBody` lets a new issue trace continue mycelium's intake trace.
43
+ * @typedef {{
44
+ * span: Span,
45
+ * issueTraceparent: string | null,
46
+ * traceparent: string | null,
47
+ * set: (attrs: NodeAttrs) => void,
48
+ * }} NodeRun
49
+ * `issueTraceparent` is what the marker should hold (new when a root was started); `traceparent` is this
50
+ * span's, for a worker's `TRACEPARENT`.
51
+ */
52
+
53
+ /** W3C `traceparent` of a span context, or null for an invalid (no-op) one. @param {SpanContext | undefined} sc */
54
+ export const traceparentFor = (sc) => (sc ? formatTraceparent(sc) : null)
55
+
56
+ /** @param {NodeAttrs} a */
57
+ function attributes(a) {
58
+ /** @type {Record<string, string | number>} */
59
+ const out = {}
60
+ const put = (/** @type {string} */ k, /** @type {unknown} */ v) => {
61
+ if (v !== undefined && v !== null) out[k] = /** @type {string | number} */ (v)
62
+ }
63
+ put('cambium.graph', a.graph)
64
+ put('cambium.graph.version', a.graphVersion)
65
+ put('cambium.node', a.node)
66
+ put('cambium.attempt', a.attempt)
67
+ put('cambium.repo', a.repo)
68
+ put('cambium.issue.number', a.issue)
69
+ put('cambium.edge', a.edge)
70
+ put('cambium.outcome', a.outcome)
71
+ return out
72
+ }
73
+
74
+ /** Metrics on/off for the metrics signal, mirroring mycelium's traces rule. @param {Record<string, string | undefined>} env */
75
+ export function otlpMetricsConfigured(env) {
76
+ if (
77
+ String(env.OTEL_SDK_DISABLED ?? '')
78
+ .trim()
79
+ .toLowerCase() === 'true'
80
+ )
81
+ return false
82
+ const exporters = String(env.OTEL_METRICS_EXPORTER ?? 'otlp')
83
+ .split(',')
84
+ .map((s) => s.trim().toLowerCase())
85
+ if (exporters.includes('none')) return false
86
+ return Boolean(env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT || env.OTEL_EXPORTER_OTLP_ENDPOINT)
87
+ }
88
+
89
+ /**
90
+ * @param {{
91
+ * env?: Record<string, string | undefined>,
92
+ * exporter?: import('@opentelemetry/sdk-trace').SpanExporter,
93
+ * metricReader?: import('@opentelemetry/sdk-metrics').MetricReader,
94
+ * }} [opts]
95
+ */
96
+ async function createMeter({ env = process.env, metricReader } = {}) {
97
+ if (!metricReader && !otlpMetricsConfigured(env)) return null
98
+ const [sdk, res] = await Promise.all([import('@opentelemetry/sdk-metrics'), import('@opentelemetry/resources')])
99
+ const reader =
100
+ metricReader ??
101
+ new sdk.PeriodicExportingMetricReader({
102
+ exporter: new (await import('@opentelemetry/exporter-metrics-otlp-http')).OTLPMetricExporter(),
103
+ })
104
+ const resource = res
105
+ .defaultResource()
106
+ .merge(res.resourceFromAttributes({ 'service.name': SERVICE_NAME }))
107
+ .merge(res.detectResources({ detectors: [res.envDetector] }))
108
+ const provider = new sdk.MeterProvider({ resource, readers: [reader] })
109
+ return { provider, meter: provider.getMeter(SERVICE_NAME) }
110
+ }
111
+
112
+ /**
113
+ * cambium's telemetry. Without OTLP config (and no injected exporter/reader) everything is a no-op that costs a
114
+ * function call: `nodeSpan` just runs `fn`, traceparents are null, so nothing is written to the marker.
115
+ *
116
+ * @param {{
117
+ * env?: Record<string, string | undefined>,
118
+ * exporter?: import('@opentelemetry/sdk-trace').SpanExporter,
119
+ * metricReader?: import('@opentelemetry/sdk-metrics').MetricReader,
120
+ * now?: () => number,
121
+ * }} [opts]
122
+ */
123
+ export async function createTelemetry(opts = {}) {
124
+ const env = opts.env ?? process.env
125
+ const [tracing, metered] = await Promise.all([
126
+ opts.exporter || otlpConfigured(env)
127
+ ? createTracing({ env, exporter: opts.exporter, serviceName: SERVICE_NAME })
128
+ : NO_TRACING,
129
+ createMeter({ ...opts, env }),
130
+ ])
131
+ const { tracer } = tracing
132
+ const now = opts.now ?? Date.now
133
+ /** @type {Counter} */
134
+ const notReadyCounter = (metered?.meter ?? metrics.getMeter('noop')).createCounter('cambium.wait.not_ready', {
135
+ description: 'Polls of a wait node that found it not ready',
136
+ })
137
+
138
+ /**
139
+ * The context to parent on: the marker's traceparent; failing that a new `cambium.issue` root (a child of the
140
+ * intake trace when the issue body carries mycelium's traceparent), ended at once.
141
+ * @param {TraceInfo} info
142
+ */
143
+ function issueTrace(info) {
144
+ const known = spanContextOf(info.traceparent)
145
+ if (known) return { parent: trace.setSpanContext(ROOT_CONTEXT, known), traceparent: known.traceparent }
146
+ const intake = parseTraceparent(info.issueBody)
147
+ const root = tracer.startSpan(
148
+ 'cambium.issue',
149
+ {
150
+ root: !intake,
151
+ attributes: attributes({ graph: info.graph, repo: info.repo, issue: info.issue }),
152
+ startTime: now(),
153
+ },
154
+ intake ? trace.setSpanContext(ROOT_CONTEXT, intake) : ROOT_CONTEXT,
155
+ )
156
+ root.end()
157
+ const traceparent = traceparentFor(root.spanContext())
158
+ return { parent: trace.setSpan(ROOT_CONTEXT, root), traceparent }
159
+ }
160
+
161
+ /**
162
+ * One `cambium.node` span around a step-node run. `fn` gets the span, the issue traceparent the marker should
163
+ * hold, and this span's traceparent (pass it to a worker as `TRACEPARENT`); `run.set({ edge, outcome })` adds
164
+ * attributes known only after the run. A throw marks the span an error (outcome `error`) and is rethrown.
165
+ *
166
+ * @template T
167
+ * @param {TraceInfo} info
168
+ * @param {(run: NodeRun) => T | Promise<T>} fn
169
+ * @returns {Promise<T>}
170
+ */
171
+ async function nodeSpan(info, fn) {
172
+ const { parent, traceparent: issueTraceparent } = issueTrace(info)
173
+ const span = tracer.startSpan('cambium.node', { attributes: attributes(info), startTime: now() }, parent)
174
+ let outcome = info.outcome !== undefined
175
+ /** @type {NodeRun} */
176
+ const run = {
177
+ span,
178
+ issueTraceparent,
179
+ traceparent: traceparentFor(span.spanContext()),
180
+ set: (a) => {
181
+ if (a.outcome !== undefined) outcome = true
182
+ span.setAttributes(attributes(a))
183
+ },
184
+ }
185
+ try {
186
+ const out = await fn(run)
187
+ if (!outcome) span.setAttribute('cambium.outcome', 'ok')
188
+ return out
189
+ } catch (err) {
190
+ span.recordException(/** @type {Error} */ (err))
191
+ span.setStatus({ code: SpanStatusCode.ERROR, message: String(/** @type {any} */ (err)?.message ?? err) })
192
+ span.setAttribute('cambium.outcome', 'error')
193
+ throw err
194
+ } finally {
195
+ span.end(now())
196
+ }
197
+ }
198
+
199
+ /**
200
+ * A `cambium.wait` span for a wait period that just ended: `startTime` = `enteredAt` (the marker's), end = now.
201
+ * Returns the issue traceparent (non-null and new when a root had to be started).
202
+ *
203
+ * @param {TraceInfo & { enteredAt?: string | number | Date | null, polls?: number }} info
204
+ */
205
+ function waitSpan(info) {
206
+ const { parent, traceparent } = issueTrace(info)
207
+ const end = now()
208
+ const start = info.enteredAt == null ? end : new Date(info.enteredAt).getTime()
209
+ const span = tracer.startSpan(
210
+ 'cambium.wait',
211
+ { attributes: attributes(info), startTime: Number.isNaN(start) ? end : Math.min(start, end) },
212
+ parent,
213
+ )
214
+ if (info.polls !== undefined) span.setAttribute('cambium.wait.polls', info.polls)
215
+ span.end(end)
216
+ return traceparent
217
+ }
218
+
219
+ /** A wait was polled and is not ready. @param {NodeAttrs} [info] */
220
+ const notReady = (info = {}) =>
221
+ notReadyCounter.add(1, attributes({ graph: info.graph, node: info.node, repo: info.repo }))
222
+
223
+ return {
224
+ enabled: tracing.enabled,
225
+ metricsEnabled: Boolean(metered),
226
+ tracer,
227
+ issueTrace: (/** @type {TraceInfo} */ info) => issueTrace(info).traceparent,
228
+ nodeSpan,
229
+ waitSpan,
230
+ notReady,
231
+ traceparentFor,
232
+ /** {@link workerSpans} with this tracer. @param {Parameters<typeof workerSpans>[0]} stream @param {string | null} [parent] */
233
+ workerSpans: (stream, parent) => workerSpans(stream, { tracer, parent }),
234
+ now,
235
+ /** Exports what is queued now. */
236
+ flush: async () => {
237
+ await Promise.all([tracing.flush(), metered?.provider.forceFlush()])
238
+ },
239
+ /** Flushes, then stops; call it in a `finally`. Safe to call twice. */
240
+ shutdown: async () => {
241
+ await Promise.all([tracing.shutdown(), metered?.provider.shutdown().catch(() => {})])
242
+ },
243
+ }
244
+ }
245
+
246
+ /**
247
+ * @typedef {Awaited<ReturnType<typeof createTelemetry>>} Telemetry
248
+ */
249
+
250
+ /**
251
+ * An `advance` `instrument` hook (see `graph/advance.mjs`) that traces one advance from a marker position: step
252
+ * nodes run inside `nodeSpan` (and get `ctx.traceparent` for a worker's `TRACEPARENT`), a ready wait emits its
253
+ * `waitSpan` (from `enteredAt` for the first node, else from when the previous transition landed), a not-ready
254
+ * wait bumps the counter. `hooks.traceparent` is the issue traceparent to store in the marker afterwards.
255
+ *
256
+ * @param {Telemetry} telemetry
257
+ * @param {TraceInfo & { enteredAt?: string | null }} base the marker's graph, graphVersion, attempt, traceparent,
258
+ * enteredAt, plus repo, issue and optionally the issue body
259
+ */
260
+ export function advanceHooks(telemetry, base) {
261
+ let traceparent = base.traceparent ?? null
262
+ /** @type {string | number | null | undefined} */
263
+ let enteredAt = base.enteredAt
264
+ /** @type {import('./graph/advance.mjs').Instrument} */
265
+ const instrument = async (visit, step) => {
266
+ const info = { ...base, traceparent, node: visit.node }
267
+ if (visit.kind === 'wait') {
268
+ const res = await step()
269
+ if (!res) telemetry.notReady(info)
270
+ else {
271
+ traceparent = telemetry.waitSpan({ ...info, enteredAt, edge: res.edge, outcome: 'ready' }) ?? traceparent
272
+ enteredAt = telemetry.now()
273
+ }
274
+ return res
275
+ }
276
+ const res = await telemetry.nodeSpan(info, async (run) => {
277
+ traceparent = run.issueTraceparent ?? traceparent
278
+ const t = await step({ traceparent: run.traceparent })
279
+ run.set({ edge: t?.edge ?? null, outcome: outcomeOf(t?.output) })
280
+ return t
281
+ })
282
+ enteredAt = telemetry.now()
283
+ return res
284
+ }
285
+ return {
286
+ instrument,
287
+ get traceparent() {
288
+ return traceparent
289
+ },
290
+ }
291
+ }
292
+
293
+ /** A node output's own `outcome` string when it has one, else `ok`. @param {unknown} output */
294
+ const outcomeOf = (output) => {
295
+ const o = /** @type {any} */ (output)?.outcome
296
+ return typeof o === 'string' ? o : 'ok'
297
+ }
298
+
299
+ const COMMAND_MAX = 200
300
+
301
+ /** @param {unknown} s @param {number} [max] */
302
+ const clip = (s, max = COMMAND_MAX) => {
303
+ if (typeof s !== 'string') return undefined
304
+ return s.length > max ? `${s.slice(0, max - 1)}…` : s
305
+ }
306
+
307
+ /** @param {Record<string, unknown>} a */
308
+ const defined = (a) =>
309
+ /** @type {import('@opentelemetry/api').Attributes} */ (
310
+ Object.fromEntries(Object.entries(a).filter(([, v]) => v !== undefined && v !== null))
311
+ )
312
+
313
+ /** @param {string | Iterable<string | Record<string, any>>} stream @returns {Generator<Record<string, any>>} */
314
+ function* events(stream) {
315
+ const items = typeof stream === 'string' ? stream.split('\n') : stream
316
+ for (const item of items) {
317
+ if (typeof item !== 'string') {
318
+ if (item && typeof item === 'object') yield item
319
+ continue
320
+ }
321
+ if (!item.trim()) continue
322
+ try {
323
+ const e = JSON.parse(item)
324
+ yield e && typeof e === 'object' ? e : { type: '__malformed' }
325
+ } catch {
326
+ yield { type: '__malformed' }
327
+ }
328
+ }
329
+ }
330
+
331
+ /**
332
+ * @typedef {{ id: string, name: string, input: any, start: number, end: number | null, error: boolean, parentToolId: string | null }} ToolUse
333
+ * @typedef {{ id: string, toolUseId: string | null, type?: string, description?: string, start: number, end: number | null, status?: string }} BgTask
334
+ * @typedef {{
335
+ * sessionId: string | null, complete: boolean, ok: boolean, start: number | null, end: number | null,
336
+ * tools: number, tasks: number, malformed: number, costUsd: number | null,
337
+ * }} WorkerSummary
338
+ */
339
+
340
+ /**
341
+ * Spans for one `claude -p --output-format stream-json --verbose` worker run, from its stdout log. Pure apart from
342
+ * the spans it emits (all already ended); call it once the worker has exited.
343
+ *
344
+ * - `claude.session`, child of `parent`, from the first to the last `timestamp` in the stream; attributes from the
345
+ * `result` event (cost, tokens, turns, terminal_reason, model) and `system/init` (claude_code version, session id).
346
+ * No `result` (a killed worker, a truncated log) or an error result → ERROR status.
347
+ * - `claude.tool` per `tool_use` → matching `tool_result` (a subagent's tools nest under its Agent/Task tool span),
348
+ * with the tool name, ok/error and, for Bash, the command clipped to 200 chars. Unpaired → ends at the last
349
+ * timestamp, ERROR.
350
+ * - `claude.task` per background task, `task_started` → `task_notification` (or `task_updated`'s `end_time`). These
351
+ * system events carry no timestamp, so their times are those of the latest timestamped event before them.
352
+ *
353
+ * Malformed lines are counted and skipped. Without any timestamp nothing is emitted.
354
+ *
355
+ * @param {string | Iterable<string | Record<string, any>>} stream the log text, its lines, or parsed events
356
+ * @param {{ tracer: import('@opentelemetry/api').Tracer, parent?: import('@opentelemetry/api').Context | string | null }} opts
357
+ * `parent`: the start-worker span's context or its traceparent
358
+ * @returns {WorkerSummary}
359
+ */
360
+ export function workerSpans(stream, { tracer, parent }) {
361
+ /** @type {number | null} */
362
+ let first = null
363
+ /** @type {number | null} */
364
+ let last = null
365
+ let malformed = 0
366
+ /** @type {any} */
367
+ let init = null
368
+ /** @type {any} */
369
+ let result = null
370
+ /** @type {Map<string, ToolUse>} */
371
+ const tools = new Map()
372
+ /** @type {Map<string, BgTask>} */
373
+ const tasks = new Map()
374
+
375
+ for (const e of events(stream)) {
376
+ if (e.type === '__malformed') {
377
+ malformed++
378
+ continue
379
+ }
380
+ const ts = typeof e.timestamp === 'string' ? Date.parse(e.timestamp) : Number.NaN
381
+ if (!Number.isNaN(ts)) {
382
+ first ??= ts
383
+ last = Math.max(last ?? ts, ts)
384
+ }
385
+ const at = Number.isNaN(ts) ? last : ts
386
+ const content = Array.isArray(e.message?.content) ? e.message.content : []
387
+ if (e.type === 'system' && e.subtype === 'init') init ??= e
388
+ else if (e.type === 'result') result = e
389
+ else if (e.type === 'assistant' && at !== null) {
390
+ for (const c of content)
391
+ if (c?.type === 'tool_use' && typeof c.id === 'string' && !tools.has(c.id))
392
+ tools.set(c.id, {
393
+ id: c.id,
394
+ name: String(c.name ?? 'unknown'),
395
+ input: c.input,
396
+ start: at,
397
+ end: null,
398
+ error: false,
399
+ parentToolId: typeof e.parent_tool_use_id === 'string' ? e.parent_tool_use_id : null,
400
+ })
401
+ } else if (e.type === 'user' && at !== null) {
402
+ for (const c of content) {
403
+ const t = c?.type === 'tool_result' ? tools.get(c.tool_use_id) : undefined
404
+ if (t && t.end === null) {
405
+ t.end = at
406
+ t.error = c.is_error === true
407
+ }
408
+ }
409
+ } else if (e.type === 'system' && typeof e.task_id === 'string' && last !== null) {
410
+ const known = tasks.get(e.task_id)
411
+ if (e.subtype === 'task_started' && !known)
412
+ tasks.set(e.task_id, {
413
+ id: e.task_id,
414
+ toolUseId: typeof e.tool_use_id === 'string' ? e.tool_use_id : null,
415
+ type: e.task_type,
416
+ description: clip(e.description),
417
+ start: last,
418
+ end: null,
419
+ })
420
+ else if (known && e.subtype === 'task_updated') {
421
+ const end = Number(e.patch?.end_time)
422
+ if (Number.isFinite(end) && end > 0) known.end = end
423
+ if (typeof e.patch?.status === 'string') known.status = e.patch.status
424
+ } else if (known && e.subtype === 'task_notification') {
425
+ known.end ??= last
426
+ if (typeof e.status === 'string') known.status = e.status
427
+ }
428
+ }
429
+ }
430
+
431
+ const sessionId = result?.session_id ?? init?.session_id ?? null
432
+ const ok = Boolean(result) && result.is_error !== true && result.subtype === 'success'
433
+ /** @type {WorkerSummary} */
434
+ const summary = {
435
+ sessionId,
436
+ complete: Boolean(result),
437
+ ok,
438
+ start: first,
439
+ end: last,
440
+ tools: tools.size,
441
+ tasks: tasks.size,
442
+ malformed,
443
+ costUsd: typeof result?.total_cost_usd === 'number' ? result.total_cost_usd : null,
444
+ }
445
+ if (first === null || last === null) return summary
446
+ const end = last
447
+
448
+ const sc = typeof parent === 'string' || parent == null ? spanContextOf(parent) : null
449
+ const parentCtx =
450
+ typeof parent === 'object' && parent !== null ? parent : sc ? trace.setSpanContext(ROOT_CONTEXT, sc) : ROOT_CONTEXT
451
+ const usage = result?.usage ?? {}
452
+ const session = tracer.startSpan(
453
+ 'claude.session',
454
+ {
455
+ startTime: first,
456
+ attributes: defined({
457
+ 'claude.session_id': sessionId,
458
+ 'claude.model': Object.keys(result?.modelUsage ?? {})[0] ?? init?.model,
459
+ 'claude_code.version': init?.claude_code_version,
460
+ 'claude.cost_usd': result?.total_cost_usd,
461
+ 'claude.tokens.input': usage.input_tokens,
462
+ 'claude.tokens.output': usage.output_tokens,
463
+ 'claude.tokens.cache_read': usage.cache_read_input_tokens,
464
+ 'claude.tokens.cache_creation': usage.cache_creation_input_tokens,
465
+ 'claude.turns': result?.num_turns,
466
+ 'claude.duration_ms': result?.duration_ms,
467
+ 'claude.terminal_reason': result?.terminal_reason,
468
+ 'claude.result': result?.subtype ?? 'missing',
469
+ 'claude.permission_denials': Array.isArray(result?.permission_denials)
470
+ ? result.permission_denials.length
471
+ : undefined,
472
+ 'claude.stream.malformed': malformed || undefined,
473
+ }),
474
+ },
475
+ parentCtx,
476
+ )
477
+ if (!result) session.setStatus({ code: SpanStatusCode.ERROR, message: 'no result: worker killed or log truncated' })
478
+ else if (!ok) session.setStatus({ code: SpanStatusCode.ERROR, message: String(result.subtype ?? 'error') })
479
+ const sessionCtx = trace.setSpan(parentCtx, session)
480
+
481
+ /** @type {Map<string, import('@opentelemetry/api').Context>} */
482
+ const toolCtx = new Map()
483
+ for (const t of [...tools.values()].sort((a, b) => a.start - b.start)) {
484
+ const span = tracer.startSpan(
485
+ 'claude.tool',
486
+ {
487
+ startTime: t.start,
488
+ attributes: defined({
489
+ 'claude.tool.name': t.name,
490
+ 'claude.tool.use_id': t.id,
491
+ 'claude.tool.outcome': t.end === null ? 'unfinished' : t.error ? 'error' : 'ok',
492
+ 'claude.tool.command': t.name === 'Bash' ? clip(t.input?.command) : undefined,
493
+ }),
494
+ },
495
+ (t.parentToolId && toolCtx.get(t.parentToolId)) || sessionCtx,
496
+ )
497
+ if (t.end === null) span.setStatus({ code: SpanStatusCode.ERROR, message: 'no tool_result' })
498
+ else if (t.error) span.setStatus({ code: SpanStatusCode.ERROR })
499
+ span.end(t.end ?? end)
500
+ toolCtx.set(t.id, trace.setSpan(sessionCtx, span))
501
+ }
502
+ for (const k of tasks.values()) {
503
+ const span = tracer.startSpan(
504
+ 'claude.task',
505
+ {
506
+ startTime: k.start,
507
+ attributes: defined({
508
+ 'claude.task.id': k.id,
509
+ 'claude.task.type': k.type,
510
+ 'claude.task.status': k.status ?? 'unfinished',
511
+ 'claude.task.description': k.description,
512
+ }),
513
+ },
514
+ (k.toolUseId && toolCtx.get(k.toolUseId)) || sessionCtx,
515
+ )
516
+ if (k.end === null || (k.status && k.status !== 'completed'))
517
+ span.setStatus({ code: SpanStatusCode.ERROR, message: k.status ?? 'no task_notification' })
518
+ span.end(Math.max(k.start, k.end ?? end))
519
+ }
520
+ session.end(end)
521
+ return summary
522
+ }