@autter/runtime-node 1.0.0 → 1.2.0

package/README.md

@@ -61,6 +61,14 @@ autter.captureException(err, { "order.id": "…" });
 // errors, just a lower severity ("fatal" | "error" | "warning" | "info"):
 autter.captureMessage("Legacy /orders lookup used", "warning");
 
+// named process spans — background jobs, queue consumers, cron ticks,
+// DB-heavy calls. Always recorded (never head-sampled), so Autter's
+// slow-process monitor sees accurate run counts and durations, and can
+// flag the process when it is slow and repeating a lot:
+await autter.withProcessSpan("invoice.rebuild", async () => {
+  await rebuildInvoices();
+});
+
 // graceful shutdown flushes exporters:
 await autter.shutdown();
 ```
@@ -75,9 +83,24 @@ Defaults (cheap by construction):
 | Signal | Default |
 | --- | --- |
 | Captured/unhandled exceptions | 100% (dedicated always-on tracer) |
-| Traces | 1% head sampling (`traceSampleRate`) |
+| Traces containing an error | 100% (tail retention, `retainTracesOnError`) |
+| `withProcessSpan` spans | 100% (same always-on tracer) |
+| LLM/GenAI call spans | 100% (`llmTracing`, on by default) |
+| Healthy traces | 1% head sampling (`traceSampleRate`) |
 | Request metrics | exported every 60 s |
 | Logs | not collected |
+| PII in custom attributes | redacted before export (`redactAttributes`) |
+| Forgotten `shutdown()` | exporters still flushed on exit (`autoFlush`) |
+
+**Error-linked trace retention.** Errors export at 100% while traces are
+head-sampled — on its own that strands a retained error without the trace
+that explains it. So unsampled spans are kept briefly in an in-process
+buffer, and the moment a trace shows an error — a 5xx response, a recorded
+exception, `captureException`, or an error-severity `captureMessage` — the
+whole trace is exported, sampling lottery notwithstanding. The buffer is
+bounded (256 spans per trace, 5 000 spans total, dropped as soon as the
+request ends healthy, 30 s TTL), degrades to plain head sampling on
+overflow, and never blocks. Disable with `retainTracesOnError: false`.
 
 Crashes are observed via `process.uncaughtExceptionMonitor`, which does
 **not** change your process's exit behaviour; the final flush is
@@ -88,7 +111,171 @@ import { ExpressInstrumentation } from "@opentelemetry/instrumentation-express";
 initAutterServer({ ..., instrumentations: [new ExpressInstrumentation()] });
 ```
 
+### Lifecycle: never lose telemetry to a forgotten shutdown()
+
+Telemetry is batched (errors every ~2 s, healthy traces every ~5 s, metrics
+every 60 s) — so exiting without flushing loses whatever is still buffered.
+`initAutterServer` therefore installs an exit flush **by default**: on
+`beforeExit`, SIGINT, and SIGTERM it force-flushes every exporter, then lets
+your process die as it would have (conventional 130/143 codes). If your own
+code also handles those signals, Autter only flushes alongside it and never
+touches your exit path. Opt out with `autoFlush: false`.
+
+Prefer explicit control? Do the same yourself and get a handle back:
+
+```ts
+import { installAutterAutoFlush } from "@autter/runtime-node";
+
+const handle = installAutterAutoFlush(); // uses the active server's exporters
+// …later: handle.flush("deploy-drain") or handle.dispose()
+```
+
+If the process still exits with captures that were never confirmed
+exported (e.g. a flush timed out), you get a one-line stderr warning — not
+silent loss. While wiring things up, set `debug: true` (or `AUTTER_DEBUG=1`)
+to see `[autter] exported N span(s)` lines on stderr as batches leave.
+
 Note on usage rollups: requests are counted from the `http.server.duration`
 metric (100% accurate) and additionally from sampled server spans. At the
 default 1% sampling the span contribution is negligible; if you set
 `traceSampleRate: 1` in development, expect request counts roughly doubled.
+Tail-retained error traces don't distort this: their spans carry
+`autter.tail_retained` and the ingester keeps them out of span-fed rollups.
+
+### Privacy: custom attributes are redacted before they leave the process
+
+The browser relay whitelist-sanitises everything a client posts — but the
+server tracker accepts free-form attributes from *your* code, where a stray
+`captureException(err, { "user.email": … })` would otherwise go out
+verbatim. Custom attributes are therefore scrubbed at capture time by
+default:
+
+- values that look like emails, JWTs, `sk-…`/`ghp_…`/AWS/Slack tokens,
+  `Bearer …` headers, or `scheme://user:pass@host` URLs are masked;
+- attributes whose **key** looks sensitive (`password`, `token`, `secret`,
+  `api_key`, `authorization`, `cookie`, `ssn`, `card_number`, …) are masked
+  wholesale;
+- non-sensitive keys and primitives pass through untouched, so grouping and
+  dashboards keep working.
+
+Disable or extend it per service:
+
+```ts
+initAutterServer({
+  ...,
+  // redactAttributes: false,            // opt out entirely
+  redactAttributes: {
+    additionalKeyPatterns: ["employee_id"],
+    additionalValuePatterns: [/^ACC-\d+$/],
+  },
+});
+```
+
+Libraries that must never forward PII regardless of host configuration can
+wrap once:
+
+```ts
+import { makeSafeCapture } from "@autter/runtime-node";
+const safe = makeSafeCapture();
+safe.captureException(err, { "user.email": email }); // masked before export
+```
+
+The raw primitive is exported too (`redactAttributes(attrs, options)`).
+This closes the server-side gap to match the browser relay's payload
+whitelist; it is best-effort scrubbing of obvious PII shapes, not a DLP
+engine — keep secrets out of attributes in the first place.
+
+## 3. LLM tracing
+
+`initAutterServer` initialises LLM tracing automatically: any GenAI span —
+`gen_ai.*` semconv attributes or the Vercel AI SDK's `ai.*` spans — bypasses
+head sampling, so **every** model call is recorded with model, tokens,
+latency, and a USD cost. Opt out with `llmTracing: false`.
+
+**Easiest: wrap the client once** — works with the OpenAI, Anthropic, and
+Google GenAI SDKs (or anything with the same call shapes), streaming
+included; every call through it is traced with no per-call code:
+
+```ts
+import { instrumentLlmClient } from "@autter/runtime-node";
+
+const openai = instrumentLlmClient(new OpenAI());
+// use it exactly as before — chat, embeddings, streams are all recorded
+const out = await openai.chat.completions.create({ model: "gpt-5-mini", ... });
+```
+
+Provider is detected from the client (override with
+`{ provider, userId, attributes }` as the second argument). For streamed
+responses the span closes when the stream is consumed; OpenAI streams only
+report token usage when you pass
+`stream_options: { include_usage: true }`.
+
+**Vercel AI SDK** — just turn on its telemetry, nothing else:
+
+```ts
+const { text } = await generateText({
+  model: openai("gpt-5-mini"),
+  prompt,
+  experimental_telemetry: { isEnabled: true, metadata: { userId: user.id } },
+});
+```
+
+**Manual control** (raw fetch, unusual clients) — wrap the call:
+
+```ts
+import { withLlmCall } from "@autter/runtime-node";
+
+const res = await withLlmCall(
+  { provider: "openai", model: "gpt-5-mini", userId: user.id },
+  async (llm) => {
+    const out = await openai.chat.completions.create({ ... });
+    llm.setUsage({
+      inputTokens: out.usage?.prompt_tokens,
+      outputTokens: out.usage?.completion_tokens,
+    });
+    return out;
+  },
+);
+```
+
+Errors thrown inside are rethrown after marking the span failed — failing
+model calls surface both as error issues and as `status: "error"` LLM calls.
+Costs are estimated ingest-side from a built-in price table; report exact
+figures with `llm.setCost(usd)` (the `autter.llm.cost_usd` attribute).
+
+Where wrapping is awkward (queues, callbacks, batch results), report after
+the fact with
+`trackLlmCall({ provider, model, inputTokens, outputTokens, durationMs })`.
+
+To verify the pipe end-to-end without calling a real model:
+
+```ts
+import { emitLlmSelftestTrace } from "@autter/runtime-node";
+
+const { traceId } = await emitLlmSelftestTrace();
+// one fake "autter-selftest" call is flushed to the ingester; look it up by
+// traceId in the dashboard's LLM tab (or runtime_llm_calls when self-hosting)
+```
+
+## 4. Process spans (jobs, consumers, crons)
+
+Non-HTTP work is only visible to the slow-process monitor where a span
+exists — and regular traces are 1% sampled. `withProcessSpan` records a
+span **always**:
+
+```ts
+import { withProcessSpan } from "@autter/runtime-node";
+
+await withProcessSpan("invoice.rebuild", async () => {
+  await rebuildInvoices();
+});
+```
+
+Use stable, low-cardinality names; put ids in attributes
+(`withProcessSpan("email.digest", fn, { "user.id": id })`).
+
+Note on the slow-process monitor: Autter flags HTTP routes from the
+unsampled request metrics, so route detection works out of the box. Non-HTTP
+work is only visible where a span exists — relying on 1%-sampled regular
+traces there would undercount ~100×, which is why these spans skip head
+sampling.