@cohortapp/agent-sdk 2.3.0

package/docs/guides/observability-otel.md

@@ -0,0 +1,265 @@
+# Observability — trace IDs, the event spine, and Claude Code OTLP
+
+Maestro's per-node substrate is strong, but historically there was no way to
+answer two questions an enterprise operator asks every day: *"show me everything
+that happened for this one user message"* and *"is the fleet meeting its SLO?"*.
+There were no correlation IDs, no metrics, and the daemon's counters lived in
+memory and reset on every restart (gap analysis G7).
+
+This guide covers the agent-side observability spine that fixes that:
+
+1. a `trace_id` minted at the start of every interaction and carried through
+   every hop, so one user message is reconstructable end-to-end;
+2. a typed diagnostic-event stream (`logs/diagnostics/<date>.jsonl`) that any
+   hot path can emit onto without risk of crashing;
+3. Claude Code's **native** OpenTelemetry export — the cheap 70% — turned on
+   fleet-wide via managed settings and stamped with each agent's identity.
+
+The first two are local and always-on (failure-tolerant); the third is opt-in
+and exports to your collector.
+
+---
+
+## 1. Trace IDs: correlating one interaction end-to-end
+
+`lib/diagnostics/trace.mjs` mints a ULID-shaped `trace_id` (26 chars,
+time-sortable, no SDK dependency) and propagates it through the canonical hops
+of an interaction:
+
+```
+item_received  →  classified  →  dispatched  →  session_opened …
+       ↑ trace_id minted here, carried unchanged through every hop below
+```
+
+Mint once at the boundary where an interaction begins — a poller item or a
+cadence tick — and run the rest of the work inside that context. (The examples
+below import directly from `lib/diagnostics/` — the direct-`lib` path CLAUDE.md
+sanctions for in-tree development; add short `./diagnostics/*` entries to the
+package `exports` map when the daemon starts consuming these as a published
+subpath.)
+
+```js
+import { mintTraceId, withTrace, childSpan, currentTraceId } from "@cohortapp/agent-sdk/lib/diagnostics/trace.mjs";
+
+const traceId = mintTraceId();
+await withTrace(traceId, async () => {
+  // anywhere beneath here, however deep the await chain:
+  currentTraceId(); // → traceId
+
+  // each hop extends the span path without changing the trace_id:
+  await withTrace(childSpan("classify"), async () => { /* … */ });
+  await withTrace(childSpan("dispatch"), async () => { /* … */ });
+});
+```
+
+Propagation uses `AsyncLocalStorage`, so deep callees read the current trace with
+`currentTrace()` / `currentTraceId()` without threading it through every
+function signature. Span paths are dotted and human-readable
+(`root.classify.send`), giving a cheap parent/child lattice.
+
+When work crosses a process or queue boundary — a spawned `claude --print`
+sub-session, a serialized inbox item — carry the context as plain fields and
+rebuild it on the far side:
+
+```js
+import { traceHeaders, traceFromHeaders } from "@cohortapp/agent-sdk/lib/diagnostics/trace.mjs";
+
+// before the boundary:
+const headers = traceHeaders();          // { trace_id, span } (or {} if none)
+
+// after the boundary (child process / consumer):
+const ctx = traceFromHeaders(received);  // { trace_id, span } or null
+await withTrace(ctx ?? mintTraceId(), async () => { /* … */ });
+```
+
+Nothing here throws: an un-instrumented call site that asks for the current
+trace simply gets `null`, and a boundary that didn't carry a trace mints a fresh
+one rather than failing. Correlation degrades gracefully; it never bricks the
+path it observes.
+
+---
+
+## 2. The diagnostic-event stream
+
+`lib/diagnostics/events.mjs` is the one typed event contract every hot path can
+emit onto. Each event appends a single JSON line to a date-stamped stream at
+`logs/diagnostics/<YYYY-MM-DD>.jsonl` (via the shared atomic appender), matching
+maestro's existing JSONL logging idiom.
+
+The event-type set is **closed** — eight types that trace the interaction:
+
+| type             | emitted when                                   |
+| ---------------- | ---------------------------------------------- |
+| `item_received`  | inbound message/tick admitted (interaction start) |
+| `classified`     | classifier verdict produced                    |
+| `dispatched`     | a session spawn is decided/launched            |
+| `session_opened` | a `claude` sub-session starts                  |
+| `session_closed` | a `claude` sub-session ends                    |
+| `sent`           | an outbound message leaves the agent (interaction end) |
+| `guard_fail`     | a governor/breaker/budget/permission gate refused work |
+| `error`          | an unexpected failure on the path              |
+
+An unknown type is coerced to `error` (with the offending value preserved at
+`attrs._invalid_type`) rather than silently widening the schema.
+
+```js
+import { emitEvent, EVENT_TYPES } from "@cohortapp/agent-sdk/lib/diagnostics/events.mjs";
+
+emitEvent({
+  type: EVENT_TYPES.ITEM_RECEIVED,
+  attrs: { source: "slack", item_id: "C123.456" },
+});
+// trace_id/span are filled from the ambient withTrace context automatically.
+```
+
+Every row has the shape `{ ts, type, trace_id, span, attrs }`. Because
+`trace_id` is the same across every hop of one interaction, the whole thing is a
+`grep`/`jq` away:
+
+```bash
+# everything that happened for one interaction, in order:
+jq -c 'select(.trace_id == "01J…")' logs/diagnostics/2026-06-11.jsonl
+```
+
+### Measuring the <2 min DM SLO
+
+The SLO is now a computable number: per `trace_id`, take the first
+`item_received` and the last `sent`, and the delta is the user-perceived
+latency.
+
+```bash
+jq -s '
+  group_by(.trace_id)[]
+  | { trace_id: .[0].trace_id,
+      start: (map(select(.type=="item_received")) | min_by(.ts) | .ts),
+      end:   (map(select(.type=="sent"))          | max_by(.ts) | .ts) }
+  | select(.start and .end)
+  | .latency_s = ((.end|fromdateiso8601) - (.start|fromdateiso8601))
+' logs/diagnostics/2026-06-11.jsonl
+```
+
+`emitEvent` **never throws**. A bad directory, an unwritable disk, a
+non-serializable attr — all degrade to a `false` return. Diagnostics must not be
+able to take down the path they observe, so it is safe to sprinkle `emitEvent`
+liberally through the daemon's hot paths.
+
+---
+
+## 3. Claude Code native OTLP — the cheap 70%
+
+For a Claude-Code fleet, most observability is "turn on the env vars". Claude
+Code natively emits OpenTelemetry **metrics, logs, and traces** — including
+cost-per-session in USD, `api_error` / `api_refusal` / `tool_decision` events,
+and `TRACEPARENT` inheritance — once telemetry is enabled. There is no SDK to
+add; `lib/diagnostics/otel.mjs` is a pure builder that produces the env block.
+
+```js
+import { buildOtelEnv } from "@cohortapp/agent-sdk/lib/diagnostics/otel.mjs";
+
+// identity is the config/agent.json record; config is the observability block.
+const otelEnv = buildOtelEnv({ identity, config: observabilityConfig });
+
+// spread into a spawned sub-session's env, or render into managed settings:
+spawn(claudeBin, args, { env: { ...process.env, ...otelEnv } });
+```
+
+It is **disabled by default**: with no config and `MAESTRO_OTEL` unset,
+`buildOtelEnv` returns `{}`, so spreading it is a no-op and the agent emits
+nothing. Enable it with `MAESTRO_OTEL=1` (or `config.enabled: true`). When
+enabled it returns:
+
+| env var                        | value                                            |
+| ------------------------------ | ------------------------------------------------ |
+| `CLAUDE_CODE_ENABLE_TELEMETRY` | `1`                                              |
+| `OTEL_METRICS_EXPORTER`        | `otlp` (override via config/env)                 |
+| `OTEL_LOGS_EXPORTER`           | `otlp`                                            |
+| `OTEL_TRACES_EXPORTER`         | `otlp`                                            |
+| `OTEL_EXPORTER_OTLP_PROTOCOL`  | `http/protobuf` (or `grpc` / `http/json`)        |
+| `OTEL_EXPORTER_OTLP_ENDPOINT`  | your collector (only when configured)            |
+| `OTEL_EXPORTER_OTLP_HEADERS`   | auth headers (only when configured)              |
+| `OTEL_METRIC_EXPORT_INTERVAL`  | ms between metric exports (only when configured) |
+| `OTEL_LOGS_EXPORT_INTERVAL`    | ms between log exports (only when configured)    |
+| `OTEL_RESOURCE_ATTRIBUTES`     | `service.name`, `service.version`, the agent id  |
+
+Optional knobs (endpoint, headers, intervals) are emitted **only** when you set
+them, so Claude Code's own OTLP defaults apply otherwise — the builder never
+clobbers a default with a blank.
+
+### Resource attributes: attributing signals to the right agent
+
+`OTEL_RESOURCE_ATTRIBUTES` is what lets a fleet collector tell 60 Mac minis
+apart. The builder stamps `service.name` (default `maestro-agent`) plus the agent
+id — derived from `config/agent.json` (`agentId` → `repoSlug` → slugified name) —
+as both `service.instance.id` and a friendly `agent.id`. Pass extra attributes
+(e.g. `deployment.environment`) via `config.resourceAttributes`.
+
+Values are sanitized so a stray `,` or `=` inside an attribute can't corrupt the
+`k=v,k=v` list.
+
+---
+
+## 4. Enabling OTLP fleet-wide via managed settings
+
+The point of a fleet is that you flip telemetry on **once**, centrally, not by
+SSHing into 60 machines. Claude Code reads an `env` block from its settings
+files, and managed/admin settings take precedence over per-user settings — so
+the operator-controlled managed settings are where fleet-wide telemetry belongs.
+
+The env map `buildOtelEnv` returns is exactly what goes in that `env` block:
+
+```jsonc
+// managed settings (operator-controlled, highest precedence)
+{
+  "env": {
+    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
+    "OTEL_METRICS_EXPORTER": "otlp",
+    "OTEL_LOGS_EXPORTER": "otlp",
+    "OTEL_TRACES_EXPORTER": "otlp",
+    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
+    "OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel-collector.fleet.internal:4318",
+    "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer <collector-token>",
+    "OTEL_RESOURCE_ATTRIBUTES": "service.name=maestro-agent,service.instance.id=alex-ai,agent.id=alex-ai"
+  }
+}
+```
+
+Because the agent id belongs in `OTEL_RESOURCE_ATTRIBUTES` and differs per
+machine, the practical pattern is: keep the shared exporter knobs (endpoint,
+protocol, headers) in the org-wide managed template, and let each agent render
+its own `OTEL_RESOURCE_ATTRIBUTES` from `buildOtelEnv({ identity })` at provision
+time. The builder reads config first, then env, then defaults — so either layout
+works.
+
+A second, lighter path is per-spawn injection: the daemon calls `buildOtelEnv`
+once and spreads the result into every `claude --print` sub-session it launches.
+That keeps telemetry scoped to the agent's own sessions (Claude Code already
+declines to leak `OTEL_*` into Bash/MCP/hook subprocesses, so an OTEL-instrumented
+tool run via Bash won't accidentally inherit the agent's collector endpoint).
+
+### What you get on the collector
+
+- **Per-session cost in USD** and token usage — the missing fleet cost signal.
+- **Behavioral events** — `api_error` rates, `api_refusal` spikes, per-tool
+  `tool_decision` baselines. Alert on patterns, not single failures: agents fail
+  silently by completing with the wrong output.
+- **Traces** with `TRACEPARENT` inheritance, attributable per agent via the
+  resource attributes above.
+
+Where the metrics land (collector → Prometheus/VictoriaMetrics + Grafana +
+Alertmanager) and the GenAI-semconv export are the export-stack layer (gap G14)
+and out of scope here; this guide gets the signal *off the agent* with the right
+identity stamped on it.
+
+---
+
+## How the three layers fit together
+
+The local event stream and the native OTLP export are complementary, not
+redundant. The diagnostic-event JSONL is the always-on, never-throw, on-disk
+record that survives with no collector and feeds the local SLO math and the
+(forthcoming) off-machine audit copy. The OTLP export is the rich, fleet-level
+signal — cost, refusals, traces — that a central collector aggregates across all
+60 agents. The `trace_id` is the thread that ties them: stamp it into your
+diagnostic events, and carry it in `OTEL_RESOURCE_ATTRIBUTES` /
+`TRACEPARENT` context so a trace in your collector and the JSONL rows on a given
+mini describe the same interaction.

package/docs/guides/org-onboarding.md

@@ -0,0 +1,132 @@
+# Org onboarding — admin checklist
+
+The condensed version of [`DEPLOYMENT.md`](../../DEPLOYMENT.md). Use that for the
+full story (architecture, Railway specifics, the credential broker in depth);
+this is the running order an admin follows to bring an org online.
+
+The shape: one Cohort server is the org source of truth (registry, hierarchy,
+board, approvals, audit, and the encrypted credential vault); the agent Mac minis
+lease from it. No web portal is needed for v1 — every step below is CLI- and
+admin-token-driven against the deployed Cohort.
+
+
+## Prerequisites
+
+- Cohort deployed (Railway service from `services/cohort/`, or self-hosted via
+  `services/cohort/docker-compose.yml`). Build context is the maestro repo root.
+- Three secrets set on the server, each via `openssl rand -hex 32` (or
+  `openssl genpkey -algorithm ed25519` for the signing key):
+  - `COHORT_ADMIN_TOKEN` — the admin bearer (pairing, kill switch, credentials).
+  - `COHORT_SIGNING_KEY` — signs snapshots / policy bundles / checkpoints.
+  - `COHORT_CRED_KEY` — AES-256-GCM key for credentials at rest (required for HA;
+    auto-generated and persisted `0600` on the volume for a single-node pilot).
+- A reachable public HTTPS URL for the server, and its pinned TLS fingerprint.
+
+
+## Steps
+
+1. **Health check.** `cohort doctor` — confirm the registry loads, the schema is
+   present, the chain verifies, and the admin token is set.
+
+2. **Ingest governance.** `cohort governance ingest --repo <org-governance>` —
+   populates hierarchy, decision rights, budgets, and policy bundles. The server
+   refuses to route handoffs and board assignment until decision rights exist, so
+   this comes before any agent does real work.
+
+3. **Set org credentials ONCE.** For each provider, pipe the key on stdin so it
+   never hits shell history:
+   ```bash
+   printf '%s' "$DEEPSEEK_API_KEY" | cohort credential put --provider deepseek
+   printf '%s' "$MOONSHOT_API_KEY" | cohort credential put --provider moonshot
+   printf '%s' "$OPENAI_API_KEY"   | cohort credential put --provider openai
+   ```
+   The value is encrypted at rest (AES-256-GCM under `COHORT_CRED_KEY`), never
+   logged. Add `--allowed-roles` / `--allowed-agents` to scope who may lease it.
+   This replaces putting provider keys on every mini.
+
+4. **Pair the agents.** As each agent first boots and requests pairing:
+   ```bash
+   cohort pair list
+   cohort pair approve <agentId> --scopes default
+   ```
+   This mints the agent's short-lived token (printed once — capture it) carrying
+   `credential.use`, so the agent can lease provider keys on demand.
+
+5. **Per agent:** drop `config/org.yaml` (Cohort URL + pinned fingerprint +
+   `roleRequested`) on the mini. The paired token is the only org secret it
+   holds. It needs its Anthropic billing
+   ([`billing-console-keys.md`](billing-console-keys.md)) but **no** third-party
+   provider keys — those are leased from Cohort and never persisted.
+
+
+## Cohort-driven agent init (what `maestro setup` pulls)
+
+When an agent boots with the Cohort pull env set, `maestro setup` (and the
+re-runnable `maestro sync`) treat the org as the source of truth: instead of
+prompting the operator field-by-field, they PULL the agent's record + the org
+context from Cohort and shape the agent's config from it.
+
+### Env contract
+
+| Var | Required | Purpose |
+| --- | --- | --- |
+| `COHORT_API_KEY` | yes | Bearer OrgApiKey for the org-data DTO reads AND the `/v1` read/RPC surface. hq's documented agent var. (`COHORT_API_TOKEN` / `COHORT_TOKEN` are accepted for back-compat.) |
+| `COHORT_ORG_ID` | yes | Org id / slug — the tenant. Sent as `?slug=` on org-data reads and as the `x-org-id` pin on `/v1`. |
+| `COHORT_AGENT_ID` | optional | This agent's member slug. When set it selects `members/<slug>` + keys relationships; when absent the pull resolves via `whoami`. |
+| `COHORT_BASE` / `COHORT_API_URL` | optional | Server origin. Falls back to the persisted `config/org.yaml` endpoint. |
+| `COHORT_AGENT_EMAIL` | optional | Email hint for `whoami` resolution when no `COHORT_AGENT_ID` is given. |
+
+The pull is gated on `COHORT_API_KEY` + `COHORT_ORG_ID` (see
+`lib/setup/enroll-from-cohort.cohortPullEnv`). Everything is **fail-open**: a
+missing var or an unreachable server is a no-op and the manual prompts stand — a
+degraded Cohort can never block setup.
+
+### What init pulls — and how it SHAPES the agent
+
+1. **Persona → `config/agent.json` (source of truth).** The MemberProfileDTO
+   maps firstName/lastName/title/email, the function/altitude archetype (derived
+   from role+team), and the supervisor (`principal`).
+   (`enroll-from-cohort.mapProfileToAgentConfig`)
+2. **Mandates / responsibilities → `config/agent.json`.** From the profile
+   sections (`responsibilities` / `operating-principles` / communication tone).
+3. **Charter(s) → `config/agent.json.charter`, rendered to
+   `config/operating-charter.md`.** The Cohort-authored `CharterDTO` is the
+   source `generate-charter.mjs` renders from (overriding the archetype skeleton).
+4. **Org context → `config/org-context.json`.** One pass fetches self profile,
+   org chart, relationships, strategy (north star / levers / streams), hierarchy,
+   adopted decisions, policy bundle, and the ready board.
+   (`lib/org/bootstrap-context.bootstrapContext`)
+5. **Work backlog → `state/backlog/wbs.yaml` + `state/queues/backlog.yaml` +
+   `config/priorities.yaml`.** `generate-backlog.mjs` reads
+   `config/org-context.json` and lets the fetched org context **shape** the
+   backlog: each org **strategy stream** becomes a leading WBS stream (intent =
+   its mission, north star woven in), and every **ready-board** item becomes a
+   concrete high-priority day-30 task. A standalone (un-enrolled) agent with no
+   cached context gets the archetype-only skeleton unchanged.
+
+Ordering note: the operating-model section (order 60) seeds the archetype backlog
+first; the org section (order 75) then caches `config/org-context.json` and
+**re-runs `generate-backlog.mjs`** so the org context actually shapes the backlog
+on the first pass. `maestro sync` re-pulls and re-materialises to keep the agent
+in step as the org evolves.
+
+
+## Ongoing
+
+- **Rotate a key:** re-run `cohort credential put --provider <p>` with the new
+  value. Every agent picks it up on its next lease — nothing to push to the fleet.
+- **Revoke a key:** `cohort credential revoke --provider <p>` — subsequent leases
+  fail closed.
+- **Contain an agent:** the kill switch deactivates an agent and revokes its
+  token on the next presence beat (≤ ~60s), so it can no longer lease anything.
+- **Audit:** every `credential.put` / `lease` / `revoke` appends one row to the
+  hash-chained `org_events` log recording provider + agent + time — never the
+  secret. `cohort credential list` shows metadata only.
+
+
+## Need a portal?
+
+Not for v1. When admin toil justifies it, add a thin admin UI over the
+read-mostly `/ops` dashboard (same method registry, same auth, no new trust
+boundary): self-serve org creation, a credential/keys form, and an agent-pairing
+approval screen. Until then the steps above are the supported path.