@oakgreencc/cambium 0.0.5 → 0.0.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +141 -1
- package/otel/com.oakgreencc.otelcol.plist +33 -0
- package/otel/otelcol-aws.yaml +52 -0
- package/otel/otelcol.yaml +50 -0
- package/package.json +15 -1
- package/src/graph/advance.mjs +107 -0
- package/src/graph/define.mjs +103 -0
- package/src/graph/index.mjs +6 -0
- package/src/graph/mermaid.mjs +25 -0
- package/src/graph/validate.mjs +77 -0
- package/src/tracing.mjs +522 -0
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@ The scheduler daemon. Every tick it reads the `q:*` queues from GitHub, starts h
|
|
|
4
4
|
|
|
5
5
|
This is the **Task phase** MVP. Dispatch and the daily Release run are seams (see [Follow-ups](#follow-ups)).
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Runtime dependencies: `zod` (the [graph module](#graph-module-preview)) and OpenTelemetry via `@oakgreencc/mycelium` (loaded only when [tracing](#tracing) is on); Node ≥ 22.13 (`node:sqlite`). GitHub access goes through [`graft`](../graft), which must be on `PATH`.
|
|
8
8
|
|
|
9
9
|
## Quick start
|
|
10
10
|
|
|
@@ -218,6 +218,146 @@ Written at the end of every tick to `paths.heartbeat` (default `~/.cambium/state
|
|
|
218
218
|
|
|
219
219
|
The mode lives in the SQLite file (`paths.db`, table `control`), so `cambium status --json` returns `{ "mode", "heartbeat" }` even when the daemon is down. The daemon's own log is `<paths.logs>/cambium.log` (one `.1` kept); launchd's stderr goes to `<paths.logs>/cambium.launchd.err.log`.
|
|
220
220
|
|
|
221
|
+
## Graph module (preview)
|
|
222
|
+
|
|
223
|
+
`@oakgreencc/cambium/graph` is the start of typed workflows: nodes with schemas, edges as data, and a loop that advances a position through them. Nothing in the tick uses it yet.
|
|
224
|
+
|
|
225
|
+
```js
|
|
226
|
+
import { z } from 'zod'
|
|
227
|
+
import { advance, defineGraph, defineNode, defineWait, END, toMermaid } from '@oakgreencc/cambium/graph'
|
|
228
|
+
|
|
229
|
+
const Pr = z.object({ pr: z.number() })
|
|
230
|
+
const open = defineNode({
|
|
231
|
+
name: 'open-pr',
|
|
232
|
+
input: z.object({ issue: z.number() }),
|
|
233
|
+
output: Pr,
|
|
234
|
+
run: async (i, ctx) => ({ pr: 1 }),
|
|
235
|
+
})
|
|
236
|
+
const ci = defineWait({
|
|
237
|
+
name: 'ci',
|
|
238
|
+
input: Pr,
|
|
239
|
+
output: z.object({ pr: z.number(), green: z.boolean() }),
|
|
240
|
+
poll: async ({ pr }, ctx) => (done ? { ready: true, output: { pr, green } } : { ready: false }),
|
|
241
|
+
})
|
|
242
|
+
const g = defineGraph({
|
|
243
|
+
name: 'demo',
|
|
244
|
+
version: 1,
|
|
245
|
+
entry: { queue: 'q:task', node: 'open-pr' },
|
|
246
|
+
nodes: [open, ci],
|
|
247
|
+
edges: [
|
|
248
|
+
['open-pr', 'ci'],
|
|
249
|
+
['ci', END, (out) => out.green],
|
|
250
|
+
['ci', 'open-pr', null, (out) => ({ issue: out.pr })],
|
|
251
|
+
],
|
|
252
|
+
})
|
|
253
|
+
const { transitions, reason, node, input } = await advance(g, { node: 'open-pr', input: { issue: 7 } }, ctx)
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
- **Nodes.** `defineNode({ name, input, output, run })` and `defineWait({ name, input, output, poll })`; `poll` returns `{ ready: false }` or `{ ready: true, output }`. `input`/`output` are any [Standard Schema](https://standardschema.dev) (zod v4 here). Input is validated before `run`/`poll` and output after; a failure is a `GraphNodeError` naming the node, the phase and the schema issues.
|
|
257
|
+
- **Edges** are `[from, to, when?, map?]`, tried in order; the first whose `when(output, subject)` is true (or that has none) wins. `map(output, subject)` builds `to`'s input and defaults to identity. `to` may be `END`; a node with no outgoing edges also ends the graph.
|
|
258
|
+
- **`validateGraph(graph)`** returns every problem as a string (`defineGraph` throws on any): unknown edge endpoints, nodes unreachable from `entry.node`, a node with outgoing edges whose last edge is conditional (no fallback), duplicate names, a non-integer `version`.
|
|
259
|
+
- **`advance(graph, { node, input }, ctx, { maxTransitions = 8, signal })`** runs step nodes and ready waits and returns `{ transitions, reason, node, input, error? }`: `reason` is `waiting` (a wait that is not ready), `end`, `limit`, `aborted` (`signal`, or `ctx.signal`, checked before each node) or `error`; `node`/`input` is where it stopped. Each transition is `{ from, to, edge, input, output, next }`. `ctx` is passed to `run`/`poll` untouched apart from `ctx.signal`; edges see `ctx.subject`.
|
|
260
|
+
- **`toMermaid(graph)`** renders a flowchart: boxes for steps, hexagons for waits, conditional edges labelled with `when`'s function name.
|
|
261
|
+
- **`opts.instrument(visit, step)`** (optional) wraps each node visit: `visit` is `{ node, kind: 'node' | 'wait', input }`; `step(extraCtx?)` runs the node (or poll) and picks the edge, returning the transition, or `null` for a wait that is not ready, and throws on a node error. `extraCtx` is merged into that node's `ctx`. Return what `step` returned. Without it, `advance` behaves exactly as before. [`advanceHooks`](#tracing) is the tracing implementation.
|
|
262
|
+
|
|
263
|
+
## Tracing
|
|
264
|
+
|
|
265
|
+
`src/tracing.mjs` is OpenTelemetry for cambium, configured only by the standard `OTEL_*` variables with `service.name=cambium`. It reuses mycelium's `createTracing` (same packages, same on/off rule): **on only when `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set**, unless `OTEL_SDK_DISABLED=true` or `OTEL_TRACES_EXPORTER=none`; otherwise nothing is imported and every call is a no-op. Metrics follow the same rule for their signal (`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_METRICS_EXPORTER`). With an endpoint but no collector, shutdown can take ~8 s; set `OTEL_EXPORTER_OTLP_TIMEOUT=2000` for the daemon.
|
|
266
|
+
|
|
267
|
+
Why not a shared package: cambium is published to npm, so it can only depend on published packages; a new publishable `@oakgreencc/telemetry` needs a protected `nx.json` release change. mycelium already publishes `createTracing`, so it gained a `serviceName` option and cambium depends on `@oakgreencc/mycelium` (which also gives it `parseTraceparent` for the intake trace).
|
|
268
|
+
|
|
269
|
+
| Span / metric | When | Parent |
|
|
270
|
+
| ------------------------------------ | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
|
|
271
|
+
| `cambium.issue` | Once per issue, when the marker has no `traceparent`. Ended at once; its traceparent goes in the marker. | mycelium's intake span from the issue body, else none (a root) |
|
|
272
|
+
| `cambium.node` | Each step-node run | The marker's `traceparent` |
|
|
273
|
+
| `cambium.wait` | When a wait ends; `startTime` = the marker's `enteredAt`. No span stays open across ticks. | The marker's `traceparent` |
|
|
274
|
+
| `cambium.wait.not_ready` | Counter, +1 per not-ready poll (attributes graph, node, repo) | |
|
|
275
|
+
| `claude.session` / `.tool` / `.task` | From the worker's stream-json log, see below | The start-worker `cambium.node` span |
|
|
276
|
+
|
|
277
|
+
Node and wait spans carry `cambium.graph`, `cambium.graph.version`, `cambium.node`, `cambium.attempt`, `cambium.repo`, `cambium.issue.number`, `cambium.edge`, `cambium.outcome`.
|
|
278
|
+
|
|
279
|
+
```js
|
|
280
|
+
import { advanceHooks, createTelemetry } from './tracing.mjs' // inside cambium
|
|
281
|
+
|
|
282
|
+
const telemetry = await createTelemetry() // { env, exporter, metricReader, now } are for tests
|
|
283
|
+
try {
|
|
284
|
+
const hooks = advanceHooks(telemetry, {
|
|
285
|
+
graph,
|
|
286
|
+
graphVersion,
|
|
287
|
+
attempt,
|
|
288
|
+
traceparent,
|
|
289
|
+
enteredAt,
|
|
290
|
+
repo,
|
|
291
|
+
issue,
|
|
292
|
+
issueBody,
|
|
293
|
+
})
|
|
294
|
+
const r = await advance(g, at, ctx, { instrument: hooks.instrument })
|
|
295
|
+
marker.traceparent = hooks.traceparent // new when a cambium.issue root was started
|
|
296
|
+
} finally {
|
|
297
|
+
await telemetry.shutdown()
|
|
298
|
+
}
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
- `telemetry.nodeSpan(info, fn)`: runs `fn(run)` inside a `cambium.node` span. `run` is `{ span, issueTraceparent, traceparent, set({ edge, outcome }) }`: `issueTraceparent` is what the marker should hold; `traceparent` is this span's, to give a worker as `TRACEPARENT`. A throw sets ERROR and outcome `error` and is rethrown.
|
|
302
|
+
- `telemetry.waitSpan({ ...info, enteredAt, polls? })`: emits the `cambium.wait` span; returns the issue traceparent.
|
|
303
|
+
- `telemetry.notReady(info)`, `telemetry.issueTrace(info)` (the marker traceparent, starting a root if needed), `traceparentFor(spanContext)`, `flush()`, `shutdown()`.
|
|
304
|
+
- `advanceHooks(telemetry, base)` → `{ instrument, traceparent }`: step nodes run in `nodeSpan` and get **`ctx.traceparent`** (start-worker sets the worker's `TRACEPARENT` from it); `cambium.outcome` is the output's `outcome` string, else `ok`. A ready wait emits `waitSpan` (from `enteredAt`, or from the previous transition within one advance); a not-ready one bumps the counter.
|
|
305
|
+
|
|
306
|
+
### Worker spans from stream-json
|
|
307
|
+
|
|
308
|
+
Worker telemetry comes from the worker's own stdout log (`claude -p --output-format stream-json --verbose`), not from Claude Code's OTel export, so it works with telemetry off on the worker side:
|
|
309
|
+
|
|
310
|
+
```js
|
|
311
|
+
workerSpans(stream, { tracer, parent }) → { sessionId, complete, ok, start, end, tools, tasks, malformed, costUsd }
|
|
312
|
+
telemetry.workerSpans(stream, parent) // same, with cambium's tracer
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
`stream` is the log text, its lines, or parsed events; `parent` is the start-worker span's context or traceparent. Call it once the worker has exited; every span it makes is already ended.
|
|
316
|
+
|
|
317
|
+
- `claude.session`: first to last `timestamp`. Attributes from `result` and `system/init`: `claude.session_id`, `claude.model`, `claude_code.version`, `claude.cost_usd`, `claude.tokens.{input,output,cache_read,cache_creation}`, `claude.turns`, `claude.duration_ms`, `claude.terminal_reason`, `claude.result`, `claude.permission_denials`. No `result` (killed, truncated) or an error result → ERROR.
|
|
318
|
+
- `claude.tool`: each `tool_use` → `tool_result` by id (a subagent's tools nest under its Agent tool span); `claude.tool.name`, `claude.tool.outcome` (`ok` / `error` / `unfinished`), `claude.tool.command` (Bash, clipped to 200 chars). Unpaired → ends at the last timestamp, ERROR.
|
|
319
|
+
- `claude.task`: each background task, `task_started` → `task_notification` (or `task_updated.patch.end_time`), under the tool that started it. These events have no timestamp, so their times are those of the latest timestamped event.
|
|
320
|
+
- Malformed lines are counted (`malformed`) and skipped; `system/thinking_tokens`, `tool_progress`, hooks and rate-limit events are ignored.
|
|
321
|
+
|
|
322
|
+
### Does `claude -p` export traces that honour `TRACEPARENT`?
|
|
323
|
+
|
|
324
|
+
**Yes, opt-in and beta**; cambium does not depend on it (worker spans come from stream-json above). Per the [Claude Code monitoring docs](https://code.claude.com/docs/en/monitoring-usage) (checked with Claude Code 2.1.327; `claude --help` lists no telemetry flags, it is env-only):
|
|
325
|
+
|
|
326
|
+
- Traces need `CLAUDE_CODE_ENABLE_TELEMETRY=1` **and** `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` plus `OTEL_TRACES_EXPORTER=otlp` and the usual `OTEL_EXPORTER_OTLP_*` endpoint/protocol (use `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` against this collector). Spans: `claude_code.interaction` → `claude_code.llm_request`, `claude_code.tool` (→ `.execution`, `.blocked_on_user`).
|
|
327
|
+
- "In Agent SDK and non-interactive sessions started with `-p`, Claude Code also reads `TRACEPARENT` and `TRACESTATE` from its own environment when starting each interaction span": the interaction span becomes a child of the caller's span (`parent.source=env`). Interactive sessions ignore an inbound `TRACEPARENT`.
|
|
328
|
+
- Bash subprocesses it runs inherit a `TRACEPARENT` for the active tool span, so a `graft` call inside a worker could continue the trace.
|
|
329
|
+
- Prompts and tool inputs are redacted from spans unless `OTEL_LOG_USER_PROMPTS` / `OTEL_LOG_TOOL_DETAILS` / `OTEL_LOG_TOOL_CONTENT` are set.
|
|
330
|
+
|
|
331
|
+
So to also get Claude's own spans in the issue trace, add `CLAUDE_CODE_ENABLE_TELEMETRY=1`, `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`, `OTEL_TRACES_EXPORTER=otlp` and the endpoint to `worker.env`; start-worker already sets `TRACEPARENT`. Not verified against a live run here.
|
|
332
|
+
|
|
333
|
+
### Local collector
|
|
334
|
+
|
|
335
|
+
[`otel/`](otel/) holds an [otelcol-contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) config (OTLP/HTTP on `127.0.0.1:4318` → `~/.cambium/otel/{traces,metrics}.jsonl` + Jaeger) and a launchd agent, `com.oakgreencc.otelcol`. Nothing here loads it.
|
|
336
|
+
|
|
337
|
+
```sh
|
|
338
|
+
# 1. otelcol-contrib on PATH (darwin_arm64 release tarball; or darwin_amd64)
|
|
339
|
+
V=0.x.y # latest from the releases page
|
|
340
|
+
curl -fsSLO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v$V/otelcol-contrib_${V}_darwin_arm64.tar.gz
|
|
341
|
+
tar -xzf otelcol-contrib_${V}_darwin_arm64.tar.gz otelcol-contrib && mv otelcol-contrib /opt/homebrew/bin/
|
|
342
|
+
|
|
343
|
+
# 2. config + agent
|
|
344
|
+
mkdir -p ~/.cambium/otel
|
|
345
|
+
cp otel/otelcol.yaml ~/.cambium/otel/otelcol.yaml
|
|
346
|
+
otelcol-contrib validate --config ~/.cambium/otel/otelcol.yaml
|
|
347
|
+
cp otel/com.oakgreencc.otelcol.plist ~/Library/LaunchAgents/
|
|
348
|
+
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.oakgreencc.otelcol.plist
|
|
349
|
+
|
|
350
|
+
# 3. Jaeger UI on :16686, OTLP/gRPC on 14317 (its 4318 would clash with the collector)
|
|
351
|
+
docker run -d --name jaeger -p 16686:16686 -p 14317:4317 jaegertracing/jaeger:latest
|
|
352
|
+
|
|
353
|
+
# 4. point cambium (and mycelium) at it, e.g. in the cambium launchd plist's environment
|
|
354
|
+
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 OTEL_EXPORTER_OTLP_TIMEOUT=2000
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
Without Jaeger running the collector still writes the files; the Jaeger exporter just logs failures. Stop it with `launchctl bootout gui/$(id -u)/com.oakgreencc.otelcol`.
|
|
358
|
+
|
|
359
|
+
**AWS** is a collector-config change only: [`otel/otelcol-aws.yaml`](otel/otelcol-aws.yaml) keeps the same receiver and sends traces to X-Ray's OTLP endpoint (`https://xray.<region>.amazonaws.com/v1/traces`, `otlphttp` + `sigv4auth` with `service: xray`; needs X-Ray Transaction Search enabled) and metrics to Amazon Managed Service for Prometheus (`prometheusremotewrite` + `sigv4auth` with `service: aps`). Credentials come from the default AWS chain of the collector's process. cambium's env stays `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318`.
|
|
360
|
+
|
|
221
361
|
## Lessons from the old Clock
|
|
222
362
|
|
|
223
363
|
| Old failure | Here | Test |
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
2
|
+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
|
3
|
+
<!--
|
|
4
|
+
The local OpenTelemetry collector as a launchd user agent. Not loaded by
|
|
5
|
+
anything in this repo: copy it to ~/Library/LaunchAgents and bootstrap it
|
|
6
|
+
yourself (see ../README.md#tracing). It reads
|
|
7
|
+
~/.cambium/otel/otelcol.yaml and logs to ~/.cambium/otel/otelcol.log.
|
|
8
|
+
-->
|
|
9
|
+
<plist version="1.0">
|
|
10
|
+
<dict>
|
|
11
|
+
<key>Label</key>
|
|
12
|
+
<string>com.oakgreencc.otelcol</string>
|
|
13
|
+
<key>ProgramArguments</key>
|
|
14
|
+
<array>
|
|
15
|
+
<string>/bin/sh</string>
|
|
16
|
+
<string>-c</string>
|
|
17
|
+
<string>exec otelcol-contrib --config "$HOME/.cambium/otel/otelcol.yaml" >>"$HOME/.cambium/otel/otelcol.log" 2>&1</string>
|
|
18
|
+
</array>
|
|
19
|
+
<key>EnvironmentVariables</key>
|
|
20
|
+
<dict>
|
|
21
|
+
<key>PATH</key>
|
|
22
|
+
<string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
|
|
23
|
+
</dict>
|
|
24
|
+
<key>RunAtLoad</key>
|
|
25
|
+
<true/>
|
|
26
|
+
<key>KeepAlive</key>
|
|
27
|
+
<true/>
|
|
28
|
+
<key>ThrottleInterval</key>
|
|
29
|
+
<integer>10</integer>
|
|
30
|
+
<key>ProcessType</key>
|
|
31
|
+
<string>Background</string>
|
|
32
|
+
</dict>
|
|
33
|
+
</plist>
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# AWS variant: the same OTLP/HTTP receiver; traces go to AWS X-Ray's OTLP
|
|
2
|
+
# endpoint and metrics to Amazon Managed Service for Prometheus, both signed
|
|
3
|
+
# with SigV4 from the default AWS credential chain. cambium itself does not
|
|
4
|
+
# change: only the collector config does. Fill in REGION and WORKSPACE_ID.
|
|
5
|
+
#
|
|
6
|
+
# Prerequisites: X-Ray Transaction Search enabled in the account (the OTLP
|
|
7
|
+
# endpoint requires it); an AMP workspace; credentials allowed
|
|
8
|
+
# xray:PutTraceSegments (and the OTLP span ingest permission for Transaction
|
|
9
|
+
# Search) plus aps:RemoteWrite on the workspace.
|
|
10
|
+
extensions:
|
|
11
|
+
sigv4auth/xray:
|
|
12
|
+
region: REGION
|
|
13
|
+
service: xray
|
|
14
|
+
sigv4auth/aps:
|
|
15
|
+
region: REGION
|
|
16
|
+
service: aps
|
|
17
|
+
|
|
18
|
+
receivers:
|
|
19
|
+
otlp:
|
|
20
|
+
protocols:
|
|
21
|
+
http:
|
|
22
|
+
endpoint: 127.0.0.1:4318
|
|
23
|
+
|
|
24
|
+
processors:
|
|
25
|
+
batch: {}
|
|
26
|
+
memory_limiter:
|
|
27
|
+
check_interval: 1s
|
|
28
|
+
limit_mib: 256
|
|
29
|
+
|
|
30
|
+
exporters:
|
|
31
|
+
otlphttp/xray:
|
|
32
|
+
traces_endpoint: https://xray.REGION.amazonaws.com/v1/traces
|
|
33
|
+
auth:
|
|
34
|
+
authenticator: sigv4auth/xray
|
|
35
|
+
prometheusremotewrite/amp:
|
|
36
|
+
endpoint: https://aps-workspaces.REGION.amazonaws.com/workspaces/WORKSPACE_ID/api/v1/remote_write
|
|
37
|
+
auth:
|
|
38
|
+
authenticator: sigv4auth/aps
|
|
39
|
+
resource_to_telemetry_conversion:
|
|
40
|
+
enabled: true
|
|
41
|
+
|
|
42
|
+
service:
|
|
43
|
+
extensions: [sigv4auth/xray, sigv4auth/aps]
|
|
44
|
+
pipelines:
|
|
45
|
+
traces:
|
|
46
|
+
receivers: [otlp]
|
|
47
|
+
processors: [memory_limiter, batch]
|
|
48
|
+
exporters: [otlphttp/xray]
|
|
49
|
+
metrics:
|
|
50
|
+
receivers: [otlp]
|
|
51
|
+
processors: [memory_limiter, batch]
|
|
52
|
+
exporters: [prometheusremotewrite/amp]
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# Local collector for cambium (and mycelium): OTLP/HTTP in on :4318,
|
|
2
|
+
# traces and metrics out to JSON-lines files, traces also to a local Jaeger.
|
|
3
|
+
# See ../README.md#tracing for install and the AWS variant (otelcol-aws.yaml).
|
|
4
|
+
receivers:
|
|
5
|
+
otlp:
|
|
6
|
+
protocols:
|
|
7
|
+
http:
|
|
8
|
+
endpoint: 127.0.0.1:4318
|
|
9
|
+
|
|
10
|
+
processors:
|
|
11
|
+
batch: {}
|
|
12
|
+
memory_limiter:
|
|
13
|
+
check_interval: 1s
|
|
14
|
+
limit_mib: 256
|
|
15
|
+
|
|
16
|
+
exporters:
|
|
17
|
+
file/traces:
|
|
18
|
+
path: ${env:HOME}/.cambium/otel/traces.jsonl
|
|
19
|
+
rotation:
|
|
20
|
+
max_megabytes: 50
|
|
21
|
+
max_backups: 5
|
|
22
|
+
file/metrics:
|
|
23
|
+
path: ${env:HOME}/.cambium/otel/metrics.jsonl
|
|
24
|
+
rotation:
|
|
25
|
+
max_megabytes: 50
|
|
26
|
+
max_backups: 5
|
|
27
|
+
# Jaeger takes OTLP natively. Its own OTLP/HTTP port (4318) would clash with
|
|
28
|
+
# this collector, so run it with OTLP/gRPC published on 14317 (see README).
|
|
29
|
+
otlp/jaeger:
|
|
30
|
+
endpoint: 127.0.0.1:14317
|
|
31
|
+
tls:
|
|
32
|
+
insecure: true
|
|
33
|
+
retry_on_failure:
|
|
34
|
+
enabled: false
|
|
35
|
+
sending_queue:
|
|
36
|
+
enabled: true
|
|
37
|
+
|
|
38
|
+
service:
|
|
39
|
+
telemetry:
|
|
40
|
+
logs:
|
|
41
|
+
level: warn
|
|
42
|
+
pipelines:
|
|
43
|
+
traces:
|
|
44
|
+
receivers: [otlp]
|
|
45
|
+
processors: [memory_limiter, batch]
|
|
46
|
+
exporters: [file/traces, otlp/jaeger]
|
|
47
|
+
metrics:
|
|
48
|
+
receivers: [otlp]
|
|
49
|
+
processors: [memory_limiter, batch]
|
|
50
|
+
exporters: [file/metrics]
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@oakgreencc/cambium",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.7",
|
|
4
4
|
"description": "The scheduler daemon: reads the q:* queues from GitHub every tick, runs headless claude workers in their own worktrees up to a cap, and makes every queue transition itself.",
|
|
5
5
|
"license": "AGPL-3.0-only",
|
|
6
6
|
"type": "module",
|
|
@@ -24,8 +24,13 @@
|
|
|
24
24
|
"bin": {
|
|
25
25
|
"cambium": "bin/cambium.mjs"
|
|
26
26
|
},
|
|
27
|
+
"exports": {
|
|
28
|
+
"./graph": "./src/graph/index.mjs",
|
|
29
|
+
"./package.json": "./package.json"
|
|
30
|
+
},
|
|
27
31
|
"files": [
|
|
28
32
|
"bin/",
|
|
33
|
+
"otel/",
|
|
29
34
|
"src/"
|
|
30
35
|
],
|
|
31
36
|
"engines": {
|
|
@@ -38,6 +43,15 @@
|
|
|
38
43
|
"nx": {
|
|
39
44
|
"name": "cambium"
|
|
40
45
|
},
|
|
46
|
+
"dependencies": {
|
|
47
|
+
"@opentelemetry/api": "^1.10.0",
|
|
48
|
+
"@opentelemetry/exporter-metrics-otlp-http": "^0.301.2",
|
|
49
|
+
"@opentelemetry/resources": "^3.1.1",
|
|
50
|
+
"@opentelemetry/sdk-metrics": "^3.1.1",
|
|
51
|
+
"@opentelemetry/sdk-trace": "^3.1.1",
|
|
52
|
+
"zod": "^4.6.7",
|
|
53
|
+
"@oakgreencc/mycelium": "0.0.5"
|
|
54
|
+
},
|
|
41
55
|
"scripts": {
|
|
42
56
|
"lint": "biome check .",
|
|
43
57
|
"test": "node --test 'test/*.test.mjs'"
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/** The advance loop: run a graph from a position until a wait holds, the graph ends, or a bound is hit. */
|
|
3
|
+
import { check, END, GraphNodeError } from './define.mjs'
|
|
4
|
+
|
|
5
|
+
/**
|
|
6
|
+
* @typedef {{ node: string, input: unknown }} Position
|
|
7
|
+
* @typedef {{ from: string, to: string, edge: number | null, input: unknown, output: unknown, next: unknown }} Transition
|
|
8
|
+
* `edge` is the index into `graph.edges` (null when a terminal node ends the graph); `next` is `to`'s input.
|
|
9
|
+
* @typedef {'waiting' | 'end' | 'limit' | 'aborted' | 'error'} StopReason
|
|
10
|
+
* @typedef {{ transitions: Transition[], reason: StopReason, node: string, input: unknown, error?: GraphNodeError }} Advanced
|
|
11
|
+
* `node`/`input` is where it stopped: the held wait, END with the last output, the next node to run, or the failed node.
|
|
12
|
+
* @typedef {{ node: string, kind: 'node' | 'wait', input: unknown }} Visit
|
|
13
|
+
* @typedef {(visit: Visit, step: (extraCtx?: Record<string, unknown>) => Promise<Transition | null>) => Promise<Transition | null>} Instrument
|
|
14
|
+
* Wraps each node visit (tracing). `step` runs the node (or poll) and picks the edge: a Transition, or null for a
|
|
15
|
+
* wait that is not ready; it throws on a node error. `extraCtx` is merged into the node's ctx. Return what `step` returned.
|
|
16
|
+
*/
|
|
17
|
+
|
|
18
|
+
/** @param {unknown} err @param {string} node @param {'run' | 'poll' | 'edge'} phase */
|
|
19
|
+
const asNodeError = (err, node, phase) =>
|
|
20
|
+
err instanceof GraphNodeError
|
|
21
|
+
? err
|
|
22
|
+
: new GraphNodeError(node, phase, String(/** @type {any} */ (err)?.message ?? err), { cause: err })
|
|
23
|
+
|
|
24
|
+
/**
|
|
25
|
+
* @template T @param {string} node @param {'run' | 'poll'} phase @param {() => T} fn @returns {Promise<Awaited<T>>}
|
|
26
|
+
*/
|
|
27
|
+
async function call(node, phase, fn) {
|
|
28
|
+
try {
|
|
29
|
+
return await fn()
|
|
30
|
+
} catch (err) {
|
|
31
|
+
throw asNodeError(err, node, phase)
|
|
32
|
+
}
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* Advance `graph` from `at`. Step nodes and ready waits each take one transition; a wait that is not ready stops
|
|
37
|
+
* the loop without one. Also stops after `maxTransitions`, when `signal` is aborted (checked before each node),
|
|
38
|
+
* at END, or on a node error, which is returned, not thrown, so the transitions already taken are kept.
|
|
39
|
+
*
|
|
40
|
+
* `ctx` is opaque here: each node gets it as `{ ...ctx, signal }`. Edge `when`/`map` get `ctx.subject`.
|
|
41
|
+
* @param {import('./define.mjs').Graph} graph
|
|
42
|
+
* @param {Position} at
|
|
43
|
+
* @param {Record<string, unknown>} [ctx]
|
|
44
|
+
* @param {{ maxTransitions?: number, signal?: AbortSignal, instrument?: Instrument }} [opts]
|
|
45
|
+
* @returns {Promise<Advanced>}
|
|
46
|
+
*/
|
|
47
|
+
export async function advance(graph, at, ctx = {}, opts = {}) {
|
|
48
|
+
const { maxTransitions = 8 } = opts
|
|
49
|
+
const signal = opts.signal ?? /** @type {AbortSignal | undefined} */ (ctx.signal)
|
|
50
|
+
const nodeCtx = { ...ctx, signal }
|
|
51
|
+
const byName = new Map(graph.nodes.map((n) => [n.name, n]))
|
|
52
|
+
/** @type {Transition[]} */
|
|
53
|
+
const transitions = []
|
|
54
|
+
let { node: name, input } = at
|
|
55
|
+
/** @param {StopReason} reason @param {GraphNodeError} [error] @returns {Advanced} */
|
|
56
|
+
const stop = (reason, error) => ({ transitions, reason, node: name, input, ...(error && { error }) })
|
|
57
|
+
|
|
58
|
+
while (name !== END) {
|
|
59
|
+
if (signal?.aborted) return stop('aborted')
|
|
60
|
+
if (transitions.length >= maxTransitions) return stop('limit')
|
|
61
|
+
const node = byName.get(name)
|
|
62
|
+
if (!node) return stop('error', new GraphNodeError(name, 'run', 'no such node in the graph'))
|
|
63
|
+
/** @type {(extraCtx?: Record<string, unknown>) => Promise<Transition | null>} */
|
|
64
|
+
const step = async (extraCtx) => {
|
|
65
|
+
const c = extraCtx ? { ...nodeCtx, ...extraCtx } : nodeCtx
|
|
66
|
+
const valid = await check(node.input, input, name, 'input')
|
|
67
|
+
let output
|
|
68
|
+
if (node.kind === 'wait') {
|
|
69
|
+
const polled = await call(name, 'poll', () => node.poll(valid, c))
|
|
70
|
+
if (typeof polled?.ready !== 'boolean')
|
|
71
|
+
throw new GraphNodeError(name, 'poll', 'poll must return { ready: false } or { ready: true, output }')
|
|
72
|
+
if (!polled.ready) return null
|
|
73
|
+
output = await check(node.output, polled.output, name, 'output')
|
|
74
|
+
} else {
|
|
75
|
+
output = await check(node.output, await call(name, 'run', () => node.run(valid, c)), name, 'output')
|
|
76
|
+
}
|
|
77
|
+
const { edge, to, next } = pick(graph, name, output, ctx.subject)
|
|
78
|
+
return { from: name, to, edge, input, output, next }
|
|
79
|
+
}
|
|
80
|
+
try {
|
|
81
|
+
const t = opts.instrument ? await opts.instrument({ node: name, kind: node.kind, input }, step) : await step()
|
|
82
|
+
if (!t) return stop('waiting')
|
|
83
|
+
transitions.push(t)
|
|
84
|
+
name = t.to
|
|
85
|
+
input = t.next
|
|
86
|
+
} catch (err) {
|
|
87
|
+
return stop('error', asNodeError(err, name, 'edge'))
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
return stop('end')
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
/**
|
|
94
|
+
* The first edge out of `from` whose `when` matches `output`; a node with no outgoing edges goes to END.
|
|
95
|
+
* @param {import('./define.mjs').Graph} graph @param {string} from @param {unknown} output @param {unknown} subject
|
|
96
|
+
*/
|
|
97
|
+
function pick(graph, from, output, subject) {
|
|
98
|
+
let any = false
|
|
99
|
+
for (const [i, [f, to, when, map]] of graph.edges.entries()) {
|
|
100
|
+
if (f !== from) continue
|
|
101
|
+
any = true
|
|
102
|
+
if (when && !when(output, subject)) continue
|
|
103
|
+
return { edge: i, to, next: map ? map(output, subject) : output }
|
|
104
|
+
}
|
|
105
|
+
if (!any) return { edge: null, to: END, next: output }
|
|
106
|
+
throw new GraphNodeError(from, 'edge', 'no edge matched the output')
|
|
107
|
+
}
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/**
|
|
3
|
+
* Graph values: typed nodes, wait nodes and the graph that wires them with data edges.
|
|
4
|
+
* Schemas are any Standard Schema (zod v4 in cambium); validation goes through `schema['~standard'].validate`.
|
|
5
|
+
*/
|
|
6
|
+
import { END, validateGraph } from './validate.mjs'
|
|
7
|
+
|
|
8
|
+
export { END }
|
|
9
|
+
|
|
10
|
+
/**
|
|
11
|
+
* @typedef {{ '~standard': { validate: (value: unknown) => StandardResult | Promise<StandardResult> } }} StandardSchema
|
|
12
|
+
* @typedef {{ value: unknown, issues?: undefined } | { issues: ReadonlyArray<StandardIssue> }} StandardResult
|
|
13
|
+
* @typedef {{ message: string, path?: ReadonlyArray<PropertyKey | { key: PropertyKey }> }} StandardIssue
|
|
14
|
+
* @typedef {{ kind: 'node', name: string, input: StandardSchema, output: StandardSchema,
|
|
15
|
+
* run: (input: any, ctx: any) => unknown }} StepNode
|
|
16
|
+
* @typedef {{ ready: false } | { ready: true, output: unknown }} PollResult
|
|
17
|
+
* @typedef {{ kind: 'wait', name: string, input: StandardSchema, output: StandardSchema,
|
|
18
|
+
* poll: (input: any, ctx: any) => PollResult | Promise<PollResult> }} WaitNode
|
|
19
|
+
* @typedef {StepNode | WaitNode} GraphNode
|
|
20
|
+
* @typedef {(out: any, subject: unknown) => unknown} EdgeFn
|
|
21
|
+
* @typedef {[from: string, to: string, when?: EdgeFn | null, map?: EdgeFn]} Edge
|
|
22
|
+
* @typedef {{ queue?: string, node: string, input?: (...args: any[]) => unknown }} Entry
|
|
23
|
+
* @typedef {{ name: string, version: number, entry: Entry, nodes: GraphNode[], edges: Edge[] }} Graph
|
|
24
|
+
*/
|
|
25
|
+
|
|
26
|
+
/** A failure inside a node: bad input or output, or a throw from run, poll, when or map. */
|
|
27
|
+
export class GraphNodeError extends Error {
|
|
28
|
+
/**
|
|
29
|
+
* @param {string} node
|
|
30
|
+
* @param {'input' | 'output' | 'run' | 'poll' | 'edge'} phase
|
|
31
|
+
* @param {string} detail
|
|
32
|
+
* @param {{ issues?: ReadonlyArray<StandardIssue>, cause?: unknown }} [extra]
|
|
33
|
+
*/
|
|
34
|
+
constructor(node, phase, detail, extra = {}) {
|
|
35
|
+
const what = phase === 'input' || phase === 'output' ? `invalid ${phase}` : `${phase} failed`
|
|
36
|
+
super(`node "${node}": ${what}: ${detail}`, { cause: extra.cause })
|
|
37
|
+
this.name = 'GraphNodeError'
|
|
38
|
+
this.node = node
|
|
39
|
+
this.phase = phase
|
|
40
|
+
this.issues = extra.issues ?? []
|
|
41
|
+
}
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
/** @param {StandardIssue} issue */
|
|
45
|
+
const formatIssue = (issue) => {
|
|
46
|
+
const path = (issue.path ?? []).map((p) => String(typeof p === 'object' ? p.key : p)).join('.')
|
|
47
|
+
return path ? `${path}: ${issue.message}` : issue.message
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/**
|
|
51
|
+
* Validate `value` against `schema`: the parsed value, or a GraphNodeError naming the node and every issue.
|
|
52
|
+
* @param {StandardSchema} schema @param {unknown} value @param {string} node @param {'input' | 'output'} phase
|
|
53
|
+
*/
|
|
54
|
+
export async function check(schema, value, node, phase) {
|
|
55
|
+
const result = await schema['~standard'].validate(value)
|
|
56
|
+
if (result.issues) throw new GraphNodeError(node, phase, result.issues.map(formatIssue).join('; '), result)
|
|
57
|
+
return result.value
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
/** @param {unknown} s */
|
|
61
|
+
const isSchema = (s) =>
|
|
62
|
+
typeof s === 'object' && s !== null && typeof (/** @type {any} */ (s)['~standard']?.validate) === 'function'
|
|
63
|
+
|
|
64
|
+
/** @param {string} what @param {any} def @param {string} fn */
|
|
65
|
+
function checkDef(what, def, fn) {
|
|
66
|
+
if (typeof def?.name !== 'string' || !def.name) throw new Error(`${what}: name must be a non-empty string`)
|
|
67
|
+
if (def.name === END) throw new Error(`${what} "${def.name}": the name is reserved`)
|
|
68
|
+
for (const k of ['input', 'output'])
|
|
69
|
+
if (!isSchema(def[k])) throw new Error(`${what} "${def.name}": ${k} must be a Standard Schema (e.g. zod v4)`)
|
|
70
|
+
if (typeof def[fn] !== 'function') throw new Error(`${what} "${def.name}": ${fn} must be a function`)
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* A step node: input is validated, `run(input, ctx)` returns the output, the output is validated.
|
|
75
|
+
* @param {{ name: string, input: StandardSchema, output: StandardSchema, run: StepNode['run'] }} def
|
|
76
|
+
* @returns {StepNode}
|
|
77
|
+
*/
|
|
78
|
+
export function defineNode(def) {
|
|
79
|
+
checkDef('defineNode', def, 'run')
|
|
80
|
+
return Object.freeze({ kind: 'node', name: def.name, input: def.input, output: def.output, run: def.run })
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
/**
|
|
84
|
+
* A wait node: `poll(input, ctx)` either holds (`{ ready: false }`) or yields `{ ready: true, output }`.
|
|
85
|
+
* @param {{ name: string, input: StandardSchema, output: StandardSchema, poll: WaitNode['poll'] }} def
|
|
86
|
+
* @returns {WaitNode}
|
|
87
|
+
*/
|
|
88
|
+
export function defineWait(def) {
|
|
89
|
+
checkDef('defineWait', def, 'poll')
|
|
90
|
+
return Object.freeze({ kind: 'wait', name: def.name, input: def.input, output: def.output, poll: def.poll })
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
/**
|
|
94
|
+
* A graph value. Throws, listing every problem, when `validateGraph` finds any.
|
|
95
|
+
* @param {Graph} def
|
|
96
|
+
* @returns {Graph}
|
|
97
|
+
*/
|
|
98
|
+
export function defineGraph(def) {
|
|
99
|
+
const problems = validateGraph(def)
|
|
100
|
+
if (problems.length) throw new Error(`graph "${def?.name}" is invalid:\n- ${problems.join('\n- ')}`)
|
|
101
|
+
const { name, version, entry, nodes, edges } = def
|
|
102
|
+
return Object.freeze({ name, version, entry: { ...entry }, nodes: [...nodes], edges: edges.map((e) => e) })
|
|
103
|
+
}
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/** `@oakgreencc/cambium/graph`: typed nodes, data edges, load-time validation, Mermaid and the advance loop. */
|
|
3
|
+
export { defineGraph, defineNode, defineWait, END, GraphNodeError } from './define.mjs'
|
|
4
|
+
export { validateGraph } from './validate.mjs'
|
|
5
|
+
export { toMermaid } from './mermaid.mjs'
|
|
6
|
+
export { advance } from './advance.mjs'
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/** A graph as a Mermaid flowchart: step nodes are boxes, waits are hexagons, conditional edges carry `when`'s name. */
|
|
3
|
+
import { END } from './validate.mjs'
|
|
4
|
+
|
|
5
|
+
/** @param {string} s */
|
|
6
|
+
const label = (s) => `"${s.replaceAll('"', '#quot;')}"`
|
|
7
|
+
|
|
8
|
+
/**
|
|
9
|
+
* @param {import('./define.mjs').Graph} graph
|
|
10
|
+
* @returns {string}
|
|
11
|
+
*/
|
|
12
|
+
export function toMermaid(graph) {
|
|
13
|
+
const id = new Map(graph.nodes.map((n, i) => [n.name, `n${i}`]))
|
|
14
|
+
id.set(END, 'end_')
|
|
15
|
+
const lines = [`flowchart TD`, ` start_([${label(graph.entry.queue ?? 'start')}])`]
|
|
16
|
+
for (const n of graph.nodes)
|
|
17
|
+
lines.push(n.kind === 'wait' ? ` ${id.get(n.name)}{{${label(n.name)}}}` : ` ${id.get(n.name)}[${label(n.name)}]`)
|
|
18
|
+
lines.push(` end_((${label('end')}))`)
|
|
19
|
+
lines.push(` start_ --> ${id.get(graph.entry.node)}`)
|
|
20
|
+
for (const [from, to, when] of graph.edges) {
|
|
21
|
+
const arrow = when ? `-->|${label(when.name || 'when')}|` : '-->'
|
|
22
|
+
lines.push(` ${id.get(from)} ${arrow} ${id.get(to)}`)
|
|
23
|
+
}
|
|
24
|
+
return `${lines.join('\n')}\n`
|
|
25
|
+
}
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/** Load-time checks on a graph value. */
|
|
3
|
+
|
|
4
|
+
/** The terminal target of an edge. A string, so a graph stays plain data. */
|
|
5
|
+
export const END = '__end__'
|
|
6
|
+
|
|
7
|
+
/**
|
|
8
|
+
* Every problem with `graph`, as readable strings; empty when it is valid.
|
|
9
|
+
*
|
|
10
|
+
* - `name` is a non-empty string and `version` an integer;
|
|
11
|
+
* - node names are unique (and not END);
|
|
12
|
+
* - `entry.node` and every edge endpoint name a node (`to` may be END; `from` may not);
|
|
13
|
+
* - `when` and `map`, when given, are functions;
|
|
14
|
+
* - every node is reachable from `entry.node`;
|
|
15
|
+
* - every non-terminal node (one with outgoing edges) ends its edges with an unconditional fallback.
|
|
16
|
+
* A node with no outgoing edges is terminal: the graph ends after it.
|
|
17
|
+
* @param {import('./define.mjs').Graph} graph
|
|
18
|
+
* @returns {string[]}
|
|
19
|
+
*/
|
|
20
|
+
export function validateGraph(graph) {
|
|
21
|
+
/** @type {string[]} */
|
|
22
|
+
const problems = []
|
|
23
|
+
if (typeof graph !== 'object' || graph === null) return ['graph must be an object']
|
|
24
|
+
const { name, version, entry, nodes, edges } = graph
|
|
25
|
+
if (typeof name !== 'string' || !name) problems.push('name must be a non-empty string')
|
|
26
|
+
if (!Number.isInteger(version)) problems.push(`version must be an integer, got ${JSON.stringify(version)}`)
|
|
27
|
+
if (!Array.isArray(nodes) || !nodes.length) return [...problems, 'nodes must be a non-empty array']
|
|
28
|
+
if (!Array.isArray(edges)) return [...problems, 'edges must be an array']
|
|
29
|
+
|
|
30
|
+
const names = new Set()
|
|
31
|
+
for (const n of nodes) {
|
|
32
|
+
if (n?.name === END) problems.push(`node name "${END}" is reserved`)
|
|
33
|
+
else if (names.has(n?.name)) problems.push(`duplicate node name "${n?.name}"`)
|
|
34
|
+
names.add(n?.name)
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
if (typeof entry?.node !== 'string' || !names.has(entry.node))
|
|
38
|
+
problems.push(`entry.node ${JSON.stringify(entry?.node)} is not a node`)
|
|
39
|
+
if (entry?.input !== undefined && typeof entry.input !== 'function') problems.push('entry.input must be a function')
|
|
40
|
+
|
|
41
|
+
/** @type {Map<string, import('./define.mjs').Edge[]>} */
|
|
42
|
+
const out = new Map()
|
|
43
|
+
for (const [i, e] of edges.entries()) {
|
|
44
|
+
if (!Array.isArray(e) || e.length < 2) {
|
|
45
|
+
problems.push(`edge ${i} must be [from, to, when?, map?]`)
|
|
46
|
+
continue
|
|
47
|
+
}
|
|
48
|
+
const [from, to, when, map] = e
|
|
49
|
+
if (!names.has(from)) problems.push(`edge ${i} (${from} -> ${to}): unknown from "${from}"`)
|
|
50
|
+
if (to !== END && !names.has(to)) problems.push(`edge ${i} (${from} -> ${to}): unknown to "${to}"`)
|
|
51
|
+
if (when != null && typeof when !== 'function')
|
|
52
|
+
problems.push(`edge ${i} (${from} -> ${to}): when must be a function`)
|
|
53
|
+
if (map !== undefined && typeof map !== 'function')
|
|
54
|
+
problems.push(`edge ${i} (${from} -> ${to}): map must be a function`)
|
|
55
|
+
out.set(from, [...(out.get(from) ?? []), e])
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
for (const [from, list] of out) {
|
|
59
|
+
if (!names.has(from)) continue
|
|
60
|
+
if (list[list.length - 1][2] != null)
|
|
61
|
+
problems.push(`node "${from}" has no unconditional fallback edge (its last edge must have no when)`)
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
if (names.has(entry?.node)) {
|
|
65
|
+
const seen = new Set([entry.node])
|
|
66
|
+
const todo = [entry.node]
|
|
67
|
+
while (todo.length) {
|
|
68
|
+
for (const [, to] of out.get(/** @type {string} */ (todo.pop())) ?? [])
|
|
69
|
+
if (names.has(to) && !seen.has(to)) {
|
|
70
|
+
seen.add(to)
|
|
71
|
+
todo.push(to)
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
for (const n of names) if (!seen.has(n)) problems.push(`node "${n}" is unreachable from entry "${entry.node}"`)
|
|
75
|
+
}
|
|
76
|
+
return problems
|
|
77
|
+
}
|
package/src/tracing.mjs
ADDED
|
@@ -0,0 +1,522 @@
|
|
|
1
|
+
// @ts-check
|
|
2
|
+
/**
|
|
3
|
+
* OpenTelemetry for cambium: `cambium.node` spans per step-node run,
|
|
4
|
+
* `cambium.wait` spans per wait period (emitted when it ends), a root
|
|
5
|
+
* `cambium.issue` span per issue trace, and a `cambium.wait.not_ready` counter.
|
|
6
|
+
*
|
|
7
|
+
* Configured only by the standard `OTEL_*` env vars, with mycelium's on/off
|
|
8
|
+
* rule (its `createTracing` is reused with `service.name=cambium`): no OTLP
|
|
9
|
+
* endpoint, `OTEL_SDK_DISABLED=true` or `OTEL_TRACES_EXPORTER=none` → no-op,
|
|
10
|
+
* and the SDK is never imported. Metrics follow the same rule for their own
|
|
11
|
+
* signal (`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_METRICS_EXPORTER`).
|
|
12
|
+
*
|
|
13
|
+
* No span stays open across ticks: the issue root is ended as soon as it is
|
|
14
|
+
* started (its traceparent, stored in the marker, is what later ticks parent
|
|
15
|
+
* on), and a wait span is created only once the wait is over, back-dated to
|
|
16
|
+
* the marker's `enteredAt`.
|
|
17
|
+
*/
|
|
18
|
+
import { metrics, ROOT_CONTEXT, SpanStatusCode, trace } from '@opentelemetry/api'
|
|
19
|
+
import {
|
|
20
|
+
createTracing,
|
|
21
|
+
formatTraceparent,
|
|
22
|
+
NO_TRACING,
|
|
23
|
+
otlpConfigured,
|
|
24
|
+
parseTraceparent,
|
|
25
|
+
spanContextOf,
|
|
26
|
+
} from '@oakgreencc/mycelium'
|
|
27
|
+
|
|
28
|
+
export const SERVICE_NAME = 'cambium'
|
|
29
|
+
|
|
30
|
+
/**
|
|
31
|
+
* @typedef {import('@opentelemetry/api').Span} Span
|
|
32
|
+
* @typedef {import('@opentelemetry/api').SpanContext} SpanContext
|
|
33
|
+
* @typedef {import('@opentelemetry/api').Counter} Counter
|
|
34
|
+
* @typedef {{
|
|
35
|
+
* graph?: string, graphVersion?: number, node?: string, attempt?: number,
|
|
36
|
+
* repo?: string, issue?: number, edge?: number | string | null, outcome?: string,
|
|
37
|
+
* }} NodeAttrs
|
|
38
|
+
* @typedef {NodeAttrs & {
|
|
39
|
+
* traceparent?: string | null,
|
|
40
|
+
* issueBody?: string | null,
|
|
41
|
+
* }} TraceInfo
|
|
42
|
+
* `traceparent` is the marker's; `issueBody` lets a new issue trace continue mycelium's intake trace.
|
|
43
|
+
* @typedef {{
|
|
44
|
+
* span: Span,
|
|
45
|
+
* issueTraceparent: string | null,
|
|
46
|
+
* traceparent: string | null,
|
|
47
|
+
* set: (attrs: NodeAttrs) => void,
|
|
48
|
+
* }} NodeRun
|
|
49
|
+
* `issueTraceparent` is what the marker should hold (new when a root was started); `traceparent` is this
|
|
50
|
+
* span's, for a worker's `TRACEPARENT`.
|
|
51
|
+
*/
|
|
52
|
+
|
|
53
|
+
/** W3C `traceparent` of a span context, or null for an invalid (no-op) one. @param {SpanContext | undefined} sc */
|
|
54
|
+
export const traceparentFor = (sc) => (sc ? formatTraceparent(sc) : null)
|
|
55
|
+
|
|
56
|
+
/** @param {NodeAttrs} a */
|
|
57
|
+
function attributes(a) {
|
|
58
|
+
/** @type {Record<string, string | number>} */
|
|
59
|
+
const out = {}
|
|
60
|
+
const put = (/** @type {string} */ k, /** @type {unknown} */ v) => {
|
|
61
|
+
if (v !== undefined && v !== null) out[k] = /** @type {string | number} */ (v)
|
|
62
|
+
}
|
|
63
|
+
put('cambium.graph', a.graph)
|
|
64
|
+
put('cambium.graph.version', a.graphVersion)
|
|
65
|
+
put('cambium.node', a.node)
|
|
66
|
+
put('cambium.attempt', a.attempt)
|
|
67
|
+
put('cambium.repo', a.repo)
|
|
68
|
+
put('cambium.issue.number', a.issue)
|
|
69
|
+
put('cambium.edge', a.edge)
|
|
70
|
+
put('cambium.outcome', a.outcome)
|
|
71
|
+
return out
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
/** Metrics on/off for the metrics signal, mirroring mycelium's traces rule. @param {Record<string, string | undefined>} env */
|
|
75
|
+
export function otlpMetricsConfigured(env) {
|
|
76
|
+
if (
|
|
77
|
+
String(env.OTEL_SDK_DISABLED ?? '')
|
|
78
|
+
.trim()
|
|
79
|
+
.toLowerCase() === 'true'
|
|
80
|
+
)
|
|
81
|
+
return false
|
|
82
|
+
const exporters = String(env.OTEL_METRICS_EXPORTER ?? 'otlp')
|
|
83
|
+
.split(',')
|
|
84
|
+
.map((s) => s.trim().toLowerCase())
|
|
85
|
+
if (exporters.includes('none')) return false
|
|
86
|
+
return Boolean(env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT || env.OTEL_EXPORTER_OTLP_ENDPOINT)
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
/**
|
|
90
|
+
* @param {{
|
|
91
|
+
* env?: Record<string, string | undefined>,
|
|
92
|
+
* exporter?: import('@opentelemetry/sdk-trace').SpanExporter,
|
|
93
|
+
* metricReader?: import('@opentelemetry/sdk-metrics').MetricReader,
|
|
94
|
+
* }} [opts]
|
|
95
|
+
*/
|
|
96
|
+
async function createMeter({ env = process.env, metricReader } = {}) {
|
|
97
|
+
if (!metricReader && !otlpMetricsConfigured(env)) return null
|
|
98
|
+
const [sdk, res] = await Promise.all([import('@opentelemetry/sdk-metrics'), import('@opentelemetry/resources')])
|
|
99
|
+
const reader =
|
|
100
|
+
metricReader ??
|
|
101
|
+
new sdk.PeriodicExportingMetricReader({
|
|
102
|
+
exporter: new (await import('@opentelemetry/exporter-metrics-otlp-http')).OTLPMetricExporter(),
|
|
103
|
+
})
|
|
104
|
+
const resource = res
|
|
105
|
+
.defaultResource()
|
|
106
|
+
.merge(res.resourceFromAttributes({ 'service.name': SERVICE_NAME }))
|
|
107
|
+
.merge(res.detectResources({ detectors: [res.envDetector] }))
|
|
108
|
+
const provider = new sdk.MeterProvider({ resource, readers: [reader] })
|
|
109
|
+
return { provider, meter: provider.getMeter(SERVICE_NAME) }
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
/**
|
|
113
|
+
* cambium's telemetry. Without OTLP config (and no injected exporter/reader) everything is a no-op that costs a
|
|
114
|
+
* function call: `nodeSpan` just runs `fn`, traceparents are null, so nothing is written to the marker.
|
|
115
|
+
*
|
|
116
|
+
* @param {{
|
|
117
|
+
* env?: Record<string, string | undefined>,
|
|
118
|
+
* exporter?: import('@opentelemetry/sdk-trace').SpanExporter,
|
|
119
|
+
* metricReader?: import('@opentelemetry/sdk-metrics').MetricReader,
|
|
120
|
+
* now?: () => number,
|
|
121
|
+
* }} [opts]
|
|
122
|
+
*/
|
|
123
|
+
export async function createTelemetry(opts = {}) {
|
|
124
|
+
const env = opts.env ?? process.env
|
|
125
|
+
const [tracing, metered] = await Promise.all([
|
|
126
|
+
opts.exporter || otlpConfigured(env)
|
|
127
|
+
? createTracing({ env, exporter: opts.exporter, serviceName: SERVICE_NAME })
|
|
128
|
+
: NO_TRACING,
|
|
129
|
+
createMeter({ ...opts, env }),
|
|
130
|
+
])
|
|
131
|
+
const { tracer } = tracing
|
|
132
|
+
const now = opts.now ?? Date.now
|
|
133
|
+
/** @type {Counter} */
|
|
134
|
+
const notReadyCounter = (metered?.meter ?? metrics.getMeter('noop')).createCounter('cambium.wait.not_ready', {
|
|
135
|
+
description: 'Polls of a wait node that found it not ready',
|
|
136
|
+
})
|
|
137
|
+
|
|
138
|
+
/**
|
|
139
|
+
* The context to parent on: the marker's traceparent; failing that a new `cambium.issue` root (a child of the
|
|
140
|
+
* intake trace when the issue body carries mycelium's traceparent), ended at once.
|
|
141
|
+
* @param {TraceInfo} info
|
|
142
|
+
*/
|
|
143
|
+
function issueTrace(info) {
|
|
144
|
+
const known = spanContextOf(info.traceparent)
|
|
145
|
+
if (known) return { parent: trace.setSpanContext(ROOT_CONTEXT, known), traceparent: known.traceparent }
|
|
146
|
+
const intake = parseTraceparent(info.issueBody)
|
|
147
|
+
const root = tracer.startSpan(
|
|
148
|
+
'cambium.issue',
|
|
149
|
+
{
|
|
150
|
+
root: !intake,
|
|
151
|
+
attributes: attributes({ graph: info.graph, repo: info.repo, issue: info.issue }),
|
|
152
|
+
startTime: now(),
|
|
153
|
+
},
|
|
154
|
+
intake ? trace.setSpanContext(ROOT_CONTEXT, intake) : ROOT_CONTEXT,
|
|
155
|
+
)
|
|
156
|
+
root.end()
|
|
157
|
+
const traceparent = traceparentFor(root.spanContext())
|
|
158
|
+
return { parent: trace.setSpan(ROOT_CONTEXT, root), traceparent }
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/**
|
|
162
|
+
* One `cambium.node` span around a step-node run. `fn` gets the span, the issue traceparent the marker should
|
|
163
|
+
* hold, and this span's traceparent (pass it to a worker as `TRACEPARENT`); `run.set({ edge, outcome })` adds
|
|
164
|
+
* attributes known only after the run. A throw marks the span an error (outcome `error`) and is rethrown.
|
|
165
|
+
*
|
|
166
|
+
* @template T
|
|
167
|
+
* @param {TraceInfo} info
|
|
168
|
+
* @param {(run: NodeRun) => T | Promise<T>} fn
|
|
169
|
+
* @returns {Promise<T>}
|
|
170
|
+
*/
|
|
171
|
+
async function nodeSpan(info, fn) {
|
|
172
|
+
const { parent, traceparent: issueTraceparent } = issueTrace(info)
|
|
173
|
+
const span = tracer.startSpan('cambium.node', { attributes: attributes(info), startTime: now() }, parent)
|
|
174
|
+
let outcome = info.outcome !== undefined
|
|
175
|
+
/** @type {NodeRun} */
|
|
176
|
+
const run = {
|
|
177
|
+
span,
|
|
178
|
+
issueTraceparent,
|
|
179
|
+
traceparent: traceparentFor(span.spanContext()),
|
|
180
|
+
set: (a) => {
|
|
181
|
+
if (a.outcome !== undefined) outcome = true
|
|
182
|
+
span.setAttributes(attributes(a))
|
|
183
|
+
},
|
|
184
|
+
}
|
|
185
|
+
try {
|
|
186
|
+
const out = await fn(run)
|
|
187
|
+
if (!outcome) span.setAttribute('cambium.outcome', 'ok')
|
|
188
|
+
return out
|
|
189
|
+
} catch (err) {
|
|
190
|
+
span.recordException(/** @type {Error} */ (err))
|
|
191
|
+
span.setStatus({ code: SpanStatusCode.ERROR, message: String(/** @type {any} */ (err)?.message ?? err) })
|
|
192
|
+
span.setAttribute('cambium.outcome', 'error')
|
|
193
|
+
throw err
|
|
194
|
+
} finally {
|
|
195
|
+
span.end(now())
|
|
196
|
+
}
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
/**
|
|
200
|
+
* A `cambium.wait` span for a wait period that just ended: `startTime` = `enteredAt` (the marker's), end = now.
|
|
201
|
+
* Returns the issue traceparent (non-null and new when a root had to be started).
|
|
202
|
+
*
|
|
203
|
+
* @param {TraceInfo & { enteredAt?: string | number | Date | null, polls?: number }} info
|
|
204
|
+
*/
|
|
205
|
+
function waitSpan(info) {
|
|
206
|
+
const { parent, traceparent } = issueTrace(info)
|
|
207
|
+
const end = now()
|
|
208
|
+
const start = info.enteredAt == null ? end : new Date(info.enteredAt).getTime()
|
|
209
|
+
const span = tracer.startSpan(
|
|
210
|
+
'cambium.wait',
|
|
211
|
+
{ attributes: attributes(info), startTime: Number.isNaN(start) ? end : Math.min(start, end) },
|
|
212
|
+
parent,
|
|
213
|
+
)
|
|
214
|
+
if (info.polls !== undefined) span.setAttribute('cambium.wait.polls', info.polls)
|
|
215
|
+
span.end(end)
|
|
216
|
+
return traceparent
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
/** A wait was polled and is not ready. @param {NodeAttrs} [info] */
|
|
220
|
+
const notReady = (info = {}) =>
|
|
221
|
+
notReadyCounter.add(1, attributes({ graph: info.graph, node: info.node, repo: info.repo }))
|
|
222
|
+
|
|
223
|
+
return {
|
|
224
|
+
enabled: tracing.enabled,
|
|
225
|
+
metricsEnabled: Boolean(metered),
|
|
226
|
+
tracer,
|
|
227
|
+
issueTrace: (/** @type {TraceInfo} */ info) => issueTrace(info).traceparent,
|
|
228
|
+
nodeSpan,
|
|
229
|
+
waitSpan,
|
|
230
|
+
notReady,
|
|
231
|
+
traceparentFor,
|
|
232
|
+
/** {@link workerSpans} with this tracer. @param {Parameters<typeof workerSpans>[0]} stream @param {string | null} [parent] */
|
|
233
|
+
workerSpans: (stream, parent) => workerSpans(stream, { tracer, parent }),
|
|
234
|
+
now,
|
|
235
|
+
/** Exports what is queued now. */
|
|
236
|
+
flush: async () => {
|
|
237
|
+
await Promise.all([tracing.flush(), metered?.provider.forceFlush()])
|
|
238
|
+
},
|
|
239
|
+
/** Flushes, then stops; call it in a `finally`. Safe to call twice. */
|
|
240
|
+
shutdown: async () => {
|
|
241
|
+
await Promise.all([tracing.shutdown(), metered?.provider.shutdown().catch(() => {})])
|
|
242
|
+
},
|
|
243
|
+
}
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
/**
|
|
247
|
+
* @typedef {Awaited<ReturnType<typeof createTelemetry>>} Telemetry
|
|
248
|
+
*/
|
|
249
|
+
|
|
250
|
+
/**
|
|
251
|
+
* An `advance` `instrument` hook (see `graph/advance.mjs`) that traces one advance from a marker position: step
|
|
252
|
+
* nodes run inside `nodeSpan` (and get `ctx.traceparent` for a worker's `TRACEPARENT`), a ready wait emits its
|
|
253
|
+
* `waitSpan` (from `enteredAt` for the first node, else from when the previous transition landed), a not-ready
|
|
254
|
+
* wait bumps the counter. `hooks.traceparent` is the issue traceparent to store in the marker afterwards.
|
|
255
|
+
*
|
|
256
|
+
* @param {Telemetry} telemetry
|
|
257
|
+
* @param {TraceInfo & { enteredAt?: string | null }} base the marker's graph, graphVersion, attempt, traceparent,
|
|
258
|
+
* enteredAt, plus repo, issue and optionally the issue body
|
|
259
|
+
*/
|
|
260
|
+
export function advanceHooks(telemetry, base) {
|
|
261
|
+
let traceparent = base.traceparent ?? null
|
|
262
|
+
/** @type {string | number | null | undefined} */
|
|
263
|
+
let enteredAt = base.enteredAt
|
|
264
|
+
/** @type {import('./graph/advance.mjs').Instrument} */
|
|
265
|
+
const instrument = async (visit, step) => {
|
|
266
|
+
const info = { ...base, traceparent, node: visit.node }
|
|
267
|
+
if (visit.kind === 'wait') {
|
|
268
|
+
const res = await step()
|
|
269
|
+
if (!res) telemetry.notReady(info)
|
|
270
|
+
else {
|
|
271
|
+
traceparent = telemetry.waitSpan({ ...info, enteredAt, edge: res.edge, outcome: 'ready' }) ?? traceparent
|
|
272
|
+
enteredAt = telemetry.now()
|
|
273
|
+
}
|
|
274
|
+
return res
|
|
275
|
+
}
|
|
276
|
+
const res = await telemetry.nodeSpan(info, async (run) => {
|
|
277
|
+
traceparent = run.issueTraceparent ?? traceparent
|
|
278
|
+
const t = await step({ traceparent: run.traceparent })
|
|
279
|
+
run.set({ edge: t?.edge ?? null, outcome: outcomeOf(t?.output) })
|
|
280
|
+
return t
|
|
281
|
+
})
|
|
282
|
+
enteredAt = telemetry.now()
|
|
283
|
+
return res
|
|
284
|
+
}
|
|
285
|
+
return {
|
|
286
|
+
instrument,
|
|
287
|
+
get traceparent() {
|
|
288
|
+
return traceparent
|
|
289
|
+
},
|
|
290
|
+
}
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
/** A node output's own `outcome` string when it has one, else `ok`. @param {unknown} output */
|
|
294
|
+
const outcomeOf = (output) => {
|
|
295
|
+
const o = /** @type {any} */ (output)?.outcome
|
|
296
|
+
return typeof o === 'string' ? o : 'ok'
|
|
297
|
+
}
|
|
298
|
+
|
|
299
|
+
const COMMAND_MAX = 200
|
|
300
|
+
|
|
301
|
+
/** @param {unknown} s @param {number} [max] */
|
|
302
|
+
const clip = (s, max = COMMAND_MAX) => {
|
|
303
|
+
if (typeof s !== 'string') return undefined
|
|
304
|
+
return s.length > max ? `${s.slice(0, max - 1)}…` : s
|
|
305
|
+
}
|
|
306
|
+
|
|
307
|
+
/** @param {Record<string, unknown>} a */
|
|
308
|
+
const defined = (a) =>
|
|
309
|
+
/** @type {import('@opentelemetry/api').Attributes} */ (
|
|
310
|
+
Object.fromEntries(Object.entries(a).filter(([, v]) => v !== undefined && v !== null))
|
|
311
|
+
)
|
|
312
|
+
|
|
313
|
+
/** @param {string | Iterable<string | Record<string, any>>} stream @returns {Generator<Record<string, any>>} */
|
|
314
|
+
function* events(stream) {
|
|
315
|
+
const items = typeof stream === 'string' ? stream.split('\n') : stream
|
|
316
|
+
for (const item of items) {
|
|
317
|
+
if (typeof item !== 'string') {
|
|
318
|
+
if (item && typeof item === 'object') yield item
|
|
319
|
+
continue
|
|
320
|
+
}
|
|
321
|
+
if (!item.trim()) continue
|
|
322
|
+
try {
|
|
323
|
+
const e = JSON.parse(item)
|
|
324
|
+
yield e && typeof e === 'object' ? e : { type: '__malformed' }
|
|
325
|
+
} catch {
|
|
326
|
+
yield { type: '__malformed' }
|
|
327
|
+
}
|
|
328
|
+
}
|
|
329
|
+
}
|
|
330
|
+
|
|
331
|
+
/**
|
|
332
|
+
* @typedef {{ id: string, name: string, input: any, start: number, end: number | null, error: boolean, parentToolId: string | null }} ToolUse
|
|
333
|
+
* @typedef {{ id: string, toolUseId: string | null, type?: string, description?: string, start: number, end: number | null, status?: string }} BgTask
|
|
334
|
+
* @typedef {{
|
|
335
|
+
* sessionId: string | null, complete: boolean, ok: boolean, start: number | null, end: number | null,
|
|
336
|
+
* tools: number, tasks: number, malformed: number, costUsd: number | null,
|
|
337
|
+
* }} WorkerSummary
|
|
338
|
+
*/
|
|
339
|
+
|
|
340
|
+
/**
|
|
341
|
+
* Spans for one `claude -p --output-format stream-json --verbose` worker run, from its stdout log. Pure apart from
|
|
342
|
+
* the spans it emits (all already ended); call it once the worker has exited.
|
|
343
|
+
*
|
|
344
|
+
* - `claude.session`, child of `parent`, from the first to the last `timestamp` in the stream; attributes from the
|
|
345
|
+
* `result` event (cost, tokens, turns, terminal_reason, model) and `system/init` (claude_code version, session id).
|
|
346
|
+
* No `result` (a killed worker, a truncated log) or an error result → ERROR status.
|
|
347
|
+
* - `claude.tool` per `tool_use` → matching `tool_result` (a subagent's tools nest under its Agent/Task tool span),
|
|
348
|
+
* with the tool name, ok/error and, for Bash, the command clipped to 200 chars. Unpaired → ends at the last
|
|
349
|
+
* timestamp, ERROR.
|
|
350
|
+
* - `claude.task` per background task, `task_started` → `task_notification` (or `task_updated`'s `end_time`). These
|
|
351
|
+
* system events carry no timestamp, so their times are those of the latest timestamped event before them.
|
|
352
|
+
*
|
|
353
|
+
* Malformed lines are counted and skipped. Without any timestamp nothing is emitted.
|
|
354
|
+
*
|
|
355
|
+
* @param {string | Iterable<string | Record<string, any>>} stream the log text, its lines, or parsed events
|
|
356
|
+
* @param {{ tracer: import('@opentelemetry/api').Tracer, parent?: import('@opentelemetry/api').Context | string | null }} opts
|
|
357
|
+
* `parent`: the start-worker span's context or its traceparent
|
|
358
|
+
* @returns {WorkerSummary}
|
|
359
|
+
*/
|
|
360
|
+
export function workerSpans(stream, { tracer, parent }) {
|
|
361
|
+
/** @type {number | null} */
|
|
362
|
+
let first = null
|
|
363
|
+
/** @type {number | null} */
|
|
364
|
+
let last = null
|
|
365
|
+
let malformed = 0
|
|
366
|
+
/** @type {any} */
|
|
367
|
+
let init = null
|
|
368
|
+
/** @type {any} */
|
|
369
|
+
let result = null
|
|
370
|
+
/** @type {Map<string, ToolUse>} */
|
|
371
|
+
const tools = new Map()
|
|
372
|
+
/** @type {Map<string, BgTask>} */
|
|
373
|
+
const tasks = new Map()
|
|
374
|
+
|
|
375
|
+
for (const e of events(stream)) {
|
|
376
|
+
if (e.type === '__malformed') {
|
|
377
|
+
malformed++
|
|
378
|
+
continue
|
|
379
|
+
}
|
|
380
|
+
const ts = typeof e.timestamp === 'string' ? Date.parse(e.timestamp) : Number.NaN
|
|
381
|
+
if (!Number.isNaN(ts)) {
|
|
382
|
+
first ??= ts
|
|
383
|
+
last = Math.max(last ?? ts, ts)
|
|
384
|
+
}
|
|
385
|
+
const at = Number.isNaN(ts) ? last : ts
|
|
386
|
+
const content = Array.isArray(e.message?.content) ? e.message.content : []
|
|
387
|
+
if (e.type === 'system' && e.subtype === 'init') init ??= e
|
|
388
|
+
else if (e.type === 'result') result = e
|
|
389
|
+
else if (e.type === 'assistant' && at !== null) {
|
|
390
|
+
for (const c of content)
|
|
391
|
+
if (c?.type === 'tool_use' && typeof c.id === 'string' && !tools.has(c.id))
|
|
392
|
+
tools.set(c.id, {
|
|
393
|
+
id: c.id,
|
|
394
|
+
name: String(c.name ?? 'unknown'),
|
|
395
|
+
input: c.input,
|
|
396
|
+
start: at,
|
|
397
|
+
end: null,
|
|
398
|
+
error: false,
|
|
399
|
+
parentToolId: typeof e.parent_tool_use_id === 'string' ? e.parent_tool_use_id : null,
|
|
400
|
+
})
|
|
401
|
+
} else if (e.type === 'user' && at !== null) {
|
|
402
|
+
for (const c of content) {
|
|
403
|
+
const t = c?.type === 'tool_result' ? tools.get(c.tool_use_id) : undefined
|
|
404
|
+
if (t && t.end === null) {
|
|
405
|
+
t.end = at
|
|
406
|
+
t.error = c.is_error === true
|
|
407
|
+
}
|
|
408
|
+
}
|
|
409
|
+
} else if (e.type === 'system' && typeof e.task_id === 'string' && last !== null) {
|
|
410
|
+
const known = tasks.get(e.task_id)
|
|
411
|
+
if (e.subtype === 'task_started' && !known)
|
|
412
|
+
tasks.set(e.task_id, {
|
|
413
|
+
id: e.task_id,
|
|
414
|
+
toolUseId: typeof e.tool_use_id === 'string' ? e.tool_use_id : null,
|
|
415
|
+
type: e.task_type,
|
|
416
|
+
description: clip(e.description),
|
|
417
|
+
start: last,
|
|
418
|
+
end: null,
|
|
419
|
+
})
|
|
420
|
+
else if (known && e.subtype === 'task_updated') {
|
|
421
|
+
const end = Number(e.patch?.end_time)
|
|
422
|
+
if (Number.isFinite(end) && end > 0) known.end = end
|
|
423
|
+
if (typeof e.patch?.status === 'string') known.status = e.patch.status
|
|
424
|
+
} else if (known && e.subtype === 'task_notification') {
|
|
425
|
+
known.end ??= last
|
|
426
|
+
if (typeof e.status === 'string') known.status = e.status
|
|
427
|
+
}
|
|
428
|
+
}
|
|
429
|
+
}
|
|
430
|
+
|
|
431
|
+
const sessionId = result?.session_id ?? init?.session_id ?? null
|
|
432
|
+
const ok = Boolean(result) && result.is_error !== true && result.subtype === 'success'
|
|
433
|
+
/** @type {WorkerSummary} */
|
|
434
|
+
const summary = {
|
|
435
|
+
sessionId,
|
|
436
|
+
complete: Boolean(result),
|
|
437
|
+
ok,
|
|
438
|
+
start: first,
|
|
439
|
+
end: last,
|
|
440
|
+
tools: tools.size,
|
|
441
|
+
tasks: tasks.size,
|
|
442
|
+
malformed,
|
|
443
|
+
costUsd: typeof result?.total_cost_usd === 'number' ? result.total_cost_usd : null,
|
|
444
|
+
}
|
|
445
|
+
if (first === null || last === null) return summary
|
|
446
|
+
const end = last
|
|
447
|
+
|
|
448
|
+
const sc = typeof parent === 'string' || parent == null ? spanContextOf(parent) : null
|
|
449
|
+
const parentCtx =
|
|
450
|
+
typeof parent === 'object' && parent !== null ? parent : sc ? trace.setSpanContext(ROOT_CONTEXT, sc) : ROOT_CONTEXT
|
|
451
|
+
const usage = result?.usage ?? {}
|
|
452
|
+
const session = tracer.startSpan(
|
|
453
|
+
'claude.session',
|
|
454
|
+
{
|
|
455
|
+
startTime: first,
|
|
456
|
+
attributes: defined({
|
|
457
|
+
'claude.session_id': sessionId,
|
|
458
|
+
'claude.model': Object.keys(result?.modelUsage ?? {})[0] ?? init?.model,
|
|
459
|
+
'claude_code.version': init?.claude_code_version,
|
|
460
|
+
'claude.cost_usd': result?.total_cost_usd,
|
|
461
|
+
'claude.tokens.input': usage.input_tokens,
|
|
462
|
+
'claude.tokens.output': usage.output_tokens,
|
|
463
|
+
'claude.tokens.cache_read': usage.cache_read_input_tokens,
|
|
464
|
+
'claude.tokens.cache_creation': usage.cache_creation_input_tokens,
|
|
465
|
+
'claude.turns': result?.num_turns,
|
|
466
|
+
'claude.duration_ms': result?.duration_ms,
|
|
467
|
+
'claude.terminal_reason': result?.terminal_reason,
|
|
468
|
+
'claude.result': result?.subtype ?? 'missing',
|
|
469
|
+
'claude.permission_denials': Array.isArray(result?.permission_denials)
|
|
470
|
+
? result.permission_denials.length
|
|
471
|
+
: undefined,
|
|
472
|
+
'claude.stream.malformed': malformed || undefined,
|
|
473
|
+
}),
|
|
474
|
+
},
|
|
475
|
+
parentCtx,
|
|
476
|
+
)
|
|
477
|
+
if (!result) session.setStatus({ code: SpanStatusCode.ERROR, message: 'no result: worker killed or log truncated' })
|
|
478
|
+
else if (!ok) session.setStatus({ code: SpanStatusCode.ERROR, message: String(result.subtype ?? 'error') })
|
|
479
|
+
const sessionCtx = trace.setSpan(parentCtx, session)
|
|
480
|
+
|
|
481
|
+
/** @type {Map<string, import('@opentelemetry/api').Context>} */
|
|
482
|
+
const toolCtx = new Map()
|
|
483
|
+
for (const t of [...tools.values()].sort((a, b) => a.start - b.start)) {
|
|
484
|
+
const span = tracer.startSpan(
|
|
485
|
+
'claude.tool',
|
|
486
|
+
{
|
|
487
|
+
startTime: t.start,
|
|
488
|
+
attributes: defined({
|
|
489
|
+
'claude.tool.name': t.name,
|
|
490
|
+
'claude.tool.use_id': t.id,
|
|
491
|
+
'claude.tool.outcome': t.end === null ? 'unfinished' : t.error ? 'error' : 'ok',
|
|
492
|
+
'claude.tool.command': t.name === 'Bash' ? clip(t.input?.command) : undefined,
|
|
493
|
+
}),
|
|
494
|
+
},
|
|
495
|
+
(t.parentToolId && toolCtx.get(t.parentToolId)) || sessionCtx,
|
|
496
|
+
)
|
|
497
|
+
if (t.end === null) span.setStatus({ code: SpanStatusCode.ERROR, message: 'no tool_result' })
|
|
498
|
+
else if (t.error) span.setStatus({ code: SpanStatusCode.ERROR })
|
|
499
|
+
span.end(t.end ?? end)
|
|
500
|
+
toolCtx.set(t.id, trace.setSpan(sessionCtx, span))
|
|
501
|
+
}
|
|
502
|
+
for (const k of tasks.values()) {
|
|
503
|
+
const span = tracer.startSpan(
|
|
504
|
+
'claude.task',
|
|
505
|
+
{
|
|
506
|
+
startTime: k.start,
|
|
507
|
+
attributes: defined({
|
|
508
|
+
'claude.task.id': k.id,
|
|
509
|
+
'claude.task.type': k.type,
|
|
510
|
+
'claude.task.status': k.status ?? 'unfinished',
|
|
511
|
+
'claude.task.description': k.description,
|
|
512
|
+
}),
|
|
513
|
+
},
|
|
514
|
+
(k.toolUseId && toolCtx.get(k.toolUseId)) || sessionCtx,
|
|
515
|
+
)
|
|
516
|
+
if (k.end === null || (k.status && k.status !== 'completed'))
|
|
517
|
+
span.setStatus({ code: SpanStatusCode.ERROR, message: k.status ?? 'no task_notification' })
|
|
518
|
+
span.end(Math.max(k.start, k.end ?? end))
|
|
519
|
+
}
|
|
520
|
+
session.end(end)
|
|
521
|
+
return summary
|
|
522
|
+
}
|