@foam-ai/node 0.1.0-alpha.1 → 0.1.0-alpha.11

package/README.md

@@ -0,0 +1,656 @@
+# Foam OpenTelemetry SDK for Node.js
+
+`@foam-ai/node` is Foam's OpenTelemetry distribution for Node.js.
+
+## Requirements
+
+- Node.js `^18.19.0 || >=20.6.0`, matching the supported engine range of
+    `@opentelemetry/auto-instrumentations-node`
+- Foam ingest token, stored as `FOAM_OTEL_TOKEN`
+
+## Install
+
+```sh
+npm install @foam-ai/node
+```
+
+## Initialize
+
+Call `init()` before your app loads so OpenTelemetry can patch libraries as
+they import. Put `init()` in its own file and preload it with `--import` or
+`--require` (below).
+
+Set `enabled: false` to turn Foam off.
+
+Store the ingest token as `FOAM_OTEL_TOKEN` and pass
+`process.env.FOAM_OTEL_TOKEN` to `init()`.
+
+### TypeScript
+
+Create `instrumentation.ts`:
+
+```ts
+import { init } from "@foam-ai/node";
+
+init({
+  name: "checkout-api",
+  environment: "production",
+  enabled: true,
+  token: process.env.FOAM_OTEL_TOKEN,
+  version: "1.0.0",
+});
+```
+
+Compile it with the app. If TypeScript emits ESM, preload with `--import`
+(same rule as ESM below):
+
+```sh
+node --import ./dist/instrumentation.js ./dist/app.js
+```
+
+If TypeScript emits CommonJS, preload with `--require`:
+
+```sh
+node --require ./dist/instrumentation.js ./dist/app.js
+```
+
+To run TypeScript without compiling first:
+
+```sh
+node --import tsx --import ./instrumentation.ts ./src/app.ts
+```
+
+### ESM
+
+Create `instrumentation.mjs` with the same `init()` call.
+
+`import` lines always run first, even if you write `init()` above them.
+Don't import your app from that file. Use `--import`, or
+`await import("./app.js")` after `init()`:
+
+```sh
+node --import ./instrumentation.mjs ./app.js
+```
+
+### CommonJS
+
+Create `instrumentation.cjs`. `require()` runs in order, so `init()` then
+`require("./app")` in this file is fine. `--require` is still the better
+preload:
+
+```js
+const { init } = require("@foam-ai/node");
+
+init({
+  name: "checkout-api",
+  environment: "production",
+  enabled: true,
+  token: process.env.FOAM_OTEL_TOKEN,
+  version: "1.0.0",
+});
+```
+
+```sh
+node --require ./instrumentation.cjs ./app.js
+```
+
+## Public API
+
+### `init(options)`
+
+Sets up tracing, metrics, logs, auto-instrumentation, OTLP export to Foam,
+and W3C context propagation.
+
+Call this once. Use it when Foam should run OpenTelemetry for the process.
+
+```js
+import { init } from "@foam-ai/node";
+
+init({
+  name: "checkout-api",
+  environment: "production",
+  enabled: true,
+  token: process.env.FOAM_OTEL_TOKEN,
+  version: "1.0.0",
+});
+```
+
+```ts
+init({
+  name,
+  environment,
+  enabled,
+  token,
+  endpoint?,
+  version?,
+  sampleRate?,
+  additionalInstrumentations?,
+  additionalSpanProcessors?,
+  additionalLogRecordProcessors?,
+  additionalMetricReaders?,
+  additionalResourceAttributes?,
+  disableLogSending?,
+  networkCapture?,
+  ignoredOutboundHosts?,
+  redact?,
+  beforeSend?,
+}): void
+```
+
+`name`, `environment`, and `enabled` are required.
+`token` is required only when Foam is enabled; store it as `FOAM_OTEL_TOKEN`.
+
+`endpoint` defaults to Foam's OTLP endpoint and is where all telemetry is
+delivered. Override it only to route through a Foam-compatible OTLP
+gateway, such as an egress proxy or a local collector. It must be an
+`http(s)` URL; its host is always excluded from outbound tracing so
+export calls do not become client spans.
+
+`version` sets `service.version`. 
+
+`sampleRate` is a number from `0` to `1` and defaults to `1`. Rarely if ever updated. Only change it if you are sending telemetry in the petabytes range.
+
+`additionalInstrumentations` adds extra OpenTelemetry instrumentations next
+to Foam's Node auto bundle and `ConsoleInstrumentation`.
+
+`additionalSpanProcessors`, `additionalLogRecordProcessors`, and
+`additionalMetricReaders` add extra processors/readers from other vendors next to Foam's
+exporters. Note: Never add Foam's own processors/readers helpers.
+
+`additionalResourceAttributes` adds extra resource attributes (for example
+`team` or `cloud.region`) to all telemetry.
+
+`disableLogSending` defaults to `false`. Set `true` to stop Foam from
+adding its own sending path to supported loggers; trace correlation in
+log records is unaffected (see Loggers).
+
+`networkCapture` defaults to `"basic"`. Upgrade to `"advanced"` unless you
+cannot; it gives much more powerful HTTP and Undici capture. See Network capture.
+
+`redact` adds keys to Foam's built-in redaction. `secrets` preserves the last
+four characters of longer values; `pii` fully masks values.
+
+```js
+init({
+  // ...
+  redact: {
+    secrets: ["internalId", "internalReference"],
+    pii: ["customerEmail", "homeAddress"],
+  },
+});
+```
+
+`beforeSend` can edit or drop spans and logs before Foam exports them. Return
+the event to send it or `null` to drop it. Span events carry `name`,
+`attributes`, `events` (span events, e.g. recorded exceptions), and `links`;
+log events carry `body`, `severityText`, and `attributes`.
+
+```js
+init({
+  // ...
+  beforeSend: (event) => {
+    if (event.type === "span" && event.attributes["url.path"] === "/health") {
+      return null; // drop health checks
+    }
+    delete event.attributes["user.email"];
+    return event;
+  },
+});
+```
+
+Metrics are not passed through `beforeSend`. See Redaction for coverage.
+
+`ignoredOutboundHosts` skips outbound HTTP tracing for those hostnames.
+The configured `endpoint` host is always skipped so export calls do not
+become client spans. Add extra hosts for high-volume clients that would drown traces,
+for example a sidecar health-check host (`localhost`) or another vendor's
+OTLP ingest host you already export to.
+
+TODO(pcga11): Add other common Gen AI providers.
+
+If another SDK already registered traces, metrics, logs, or the propagator,
+`init()` leaves the signal's slot untouched and only takes what is available. When a slot is taken, use its ingest helper instead.
+
+### `createFoamIngestSpanProcessor(name, environment, token, options?)`
+
+The helper returns a span processor but does not attach it to the
+application's provider. Include it in the `spanProcessors` option when
+constructing `BasicTracerProvider`. It cannot be added after provider
+construction in `@opentelemetry/sdk-trace-base` 2.10; the old
+`addSpanProcessor()` API is obsolete. If another SDK creates the provider,
+configure the processor through that SDK's startup options.
+
+```js
+import { trace } from "@opentelemetry/api";
+import { BasicTracerProvider } from "@opentelemetry/sdk-trace-base";
+import { createFoamIngestSpanProcessor } from "@foam-ai/node";
+
+const spanProcessor = createFoamIngestSpanProcessor(
+  "checkout-api",
+  "production",
+  process.env.FOAM_OTEL_TOKEN,
+);
+const tracerProvider = new BasicTracerProvider({
+  spanProcessors: [spanProcessor],
+});
+trace.setGlobalTracerProvider(tracerProvider);
+```
+
+### `createFoamIngestLogRecordProcessor(name, environment, token, options?)`
+
+The helper returns a log-record processor but does not attach it to the
+application's provider. Include it in the `processors` option when constructing
+`LoggerProvider`. It cannot be added after provider construction in
+`@opentelemetry/sdk-logs` 0.221; the old `addLogRecordProcessor()` API is
+obsolete. If another SDK creates the provider, configure the processor through
+that SDK's startup options.
+
+```js
+import { logs } from "@opentelemetry/api-logs";
+import { LoggerProvider } from "@opentelemetry/sdk-logs";
+import { createFoamIngestLogRecordProcessor } from "@foam-ai/node";
+
+const logRecordProcessor = createFoamIngestLogRecordProcessor(
+  "checkout-api",
+  "production",
+  process.env.FOAM_OTEL_TOKEN,
+);
+const loggerProvider = new LoggerProvider({
+  processors: [logRecordProcessor],
+});
+logs.setGlobalLoggerProvider(loggerProvider);
+```
+
+All ingest helpers accept the same `redact` option as `init()`; the span and
+log helpers also accept `beforeSend`.
+
+### `createFoamIngestMetricReader(name, environment, token, options?)`
+
+The helper returns a metric reader but does not attach it to the application's
+provider. Include it in the `readers` option when constructing `MeterProvider`.
+It cannot be added after provider construction in
+`@opentelemetry/sdk-metrics` 2.10; the old `addMetricReader()` API is obsolete.
+If another SDK creates the provider, configure the reader through that SDK's
+startup options.
+
+```js
+import { metrics } from "@opentelemetry/api";
+import { MeterProvider } from "@opentelemetry/sdk-metrics";
+import { createFoamIngestMetricReader } from "@foam-ai/node";
+
+const metricReader = createFoamIngestMetricReader(
+  "checkout-api",
+  "production",
+  process.env.FOAM_OTEL_TOKEN,
+);
+const meterProvider = new MeterProvider({
+  readers: [metricReader],
+});
+metrics.setGlobalMeterProvider(meterProvider);
+```
+
+If a vendor SDK does not expose a processor or reader startup option, add Foam
+at the collector instead.
+
+If the other SDK already set `service.name` or environment,
+Foam's values overwrite them on that copy only. The original spans, logs,
+and metrics are left unchanged. For the Foam SDK, ensure all of the signals (spans, logs and metrics)
+carry identical `service.name`.
+
+Don't use `init()` and an ingest helper for the same signal. If another
+SDK already owns traces (or another signal), call `init()` for the rest
+and attach ingest only to the provider you don't own. Don't pass ingest
+helpers as `additionalSpanProcessors` or extra readers: `init()` already
+installs Foam exporters.
+
+Ingest helpers do not register instrumentations. If you cannot call
+`init()`, register them with the official OpenTelemetry API
+before the app imports those libraries:
+
+```js
+import { registerInstrumentations } from "@opentelemetry/instrumentation";
+import { RedisInstrumentation } from "@opentelemetry/instrumentation-redis";
+import { createFoamIngestSpanProcessor } from "@foam-ai/node";
+
+registerInstrumentations({
+  instrumentations: [new RedisInstrumentation()],
+});
+```
+
+Attach the ingest helper to the other SDK's provider as in the examples
+above. If that SDK already instruments the same library, don't register a
+second copy.
+
+OpenTelemetry allows only one global propagator per signal. Foam registers W3C Trace
+Context and W3C Baggage (`traceparent`, `tracestate`, `baggage`), not B3,
+Jaeger, or AWS X-Ray. Auto-instrumented HTTP uses whichever propagator
+won. If another SDK already registered a propagator, Foam keeps it and
+`getState().signals.baggage` is `"none"`. A non-W3C propagator will not
+send `traceparent`, so a Foam peer starts a new trace. Foam does not add
+W3C headers on top.
+
+A work around is to initialize Foam first, or include W3C in the other
+SDK's composite, if those headers must be on HTTP.
+
+TODO(pcga11): Handle traceparent conflict^
+
+### `injectTraceContext(headers)`
+
+Writes W3C `traceparent`, `tracestate`, and `baggage` onto a header map.
+
+Use this when you send a message yourself (Kafka, a queue, a custom
+socket). Auto-instrumented HTTP, Express, Undici, and Fetch already inject
+these headers; don't call this on ordinary HTTP. This uses Foam's local
+W3C propagator so it works even if Foam did not get the global one.
+
+```js
+const headers = {};
+injectTraceContext(headers);
+await kafka.send({ value: payload, headers });
+```
+
+### `extractTraceContext(headers)`
+
+Reads W3C trace and baggage headers and returns a parent OpenTelemetry
+context.
+
+Use this when receiving a message on a custom transport. Without extraction,
+the handler loses the propagated parent context.
+
+```js
+import { context } from "@opentelemetry/api";
+
+const parent = extractTraceContext(message.headers);
+await context.with(parent, () => handle(message));
+```
+
+### `setBaggage(key, value, callback)`
+
+Runs a callback with a request-scoped string such as a tenant or user ID.
+The value follows work started by the callback across `await`s without leaking
+to later handlers. Call `injectTraceContext` to send it on custom transports.
+
+```js
+await setBaggage("tenant.id", "acme", () => handleRequest());
+```
+
+### `getBaggage(key)`
+
+Returns a baggage value from the current request, or `undefined`.
+
+Use this later in the same request to read a value set with `setBaggage`,
+or a key that arrived on an inbound `baggage` header.
+
+```js
+const tenant = getBaggage("tenant.id"); // "acme"
+```
+
+### `incrementCounter(name, value?, attributes?, options?)`
+
+Creates the counter if it does not exist, then increments it.
+
+This counter can only increases. Use this for event counts.
+
+```js
+incrementCounter("orders.created", 1, { "cloud.region": "us-west-1" });
+```
+
+### `recordHistogram(name, value, attributes?, options?)`
+
+Creates the histogram if it does not exist, then records a measurement.
+
+Use this for values you want percentiles or averages of: latency, duration, payload size.
+
+```js
+recordHistogram("checkout.duration", 0.142, undefined, { unit: "s" });
+```
+
+### `addUpDownCounter(name, value, attributes?, options?)`
+
+Creates the up-down counter if it does not exist, then adds a signed
+delta. The value can go up or down.
+
+Use this for occupancy: active jobs, open connections, items in a pool.
+
+In other terms, pass +value when something starts occupying a slot, -value when it leaves. The metric is the current state of occupancy.
+
+```js
+addUpDownCounter("jobs.active", -1);
+```
+
+### `setMetric(name, value, attributes?, options?)`
+
+Creates the gauge if it does not exist, then sets its current value.
+
+Use this for a point-in-time level: queue depth, heap used, cache size.
+
+```js
+setMetric("queue.depth", 27);
+```
+
+TODO(pcga11): Investigate if it's worth support observable (pull) instruments API.
+
+### `log(body, severity?, attributes?)`
+
+Sends a log record to Foam on Foam's own export path. It does not write to the app's logger, and it does not emit on a LoggerProvider Foam does
+not own.
+
+Use this to send logs directly. Typically used for Pino 5–6, Winston 1–2 as they do not support sending logs.
+
+```js
+import { log, SeverityNumber } from "@foam-ai/node";
+
+log("checkout failed");
+log("checkout failed", SeverityNumber.ERROR);
+```
+
+### `recordException(error)`
+
+Records an exception on the current active span and sets the span status
+to `ERROR`. Accepts an `Error` or any value (stringified). Never throws;
+does nothing when no span is active or the span is not recording.
+
+Use this in `catch` blocks inside auto-instrumented handlers (HTTP
+routes, message consumers) where you did not create the span yourself.
+The exception lands on whichever SDK owns the active span, so it works
+even when another SDK owns tracing.
+
+```js
+import { recordException } from "@foam-ai/node";
+
+try {
+  await chargeCard(order);
+} catch (err) {
+  recordException(err);
+  throw err;
+}
+```
+
+### `getState()`
+
+Returns whether Foam initialized, which instrumentations registered, and
+which signals have an export path. 
+
+Use this in health checks, tests, or diagnostics to confirm Foam is
+running.
+
+```js
+import { getState } from "@foam-ai/node";
+
+getState();
+// {
+//   initialized: true,
+//   instrumentations: ["http", "express", "pg", ...],
+//   signals: {
+//     traces: "global",
+//     metrics: "global",
+//     logs: "global",
+//     baggage: "global",
+//     profile: "none",
+//   },
+// }
+```
+
+`instrumentations` lists only the instrumentations registered with a
+successfully started Foam SDK, after upstream defaults and
+`OTEL_NODE_ENABLED_INSTRUMENTATIONS` /
+`OTEL_NODE_DISABLED_INSTRUMENTATIONS` filtering. Registered means the
+instrumentation is ready to patch a supported module when that module
+loads.
+
+Each `signals` value names the source of the Foam export path for that
+signal:
+
+- `"global"` — `init()` registered Foam's provider in the global
+  OpenTelemetry slot.
+- `"ingest"` — a `createFoamIngest*` processor/reader was constructed for
+  another SDK's pipeline. Foam cannot tell whether you actually registered
+  the returned object.
+- `"local"` — logs only: another SDK owns the global LoggerProvider, but
+  Foam keeps its own local provider so `log()` still delivers to Foam.
+- `"none"` — no Foam export path.
+
+A value other than `"none"` means telemetry is expected, not that the app
+has already produced or exported it.
+
+## Custom spans and other OpenTelemetry APIs
+
+Foam does not wrap span creation. To start your own spans, read the
+current span, or work with context, import `@opentelemetry/api` directly.
+`init()` registers Foam's providers in the global OpenTelemetry slots, so
+the official API routes to Foam automatically.
+
+Install the API in your app so it resolves one compatible `1.x` copy:
+
+```sh
+npm install @opentelemetry/api
+```
+
+```js
+import { trace, SpanStatusCode } from "@opentelemetry/api";
+
+const tracer = trace.getTracer("checkout");
+
+await tracer.startActiveSpan("charge-card", async (span) => {
+  try {
+    await chargeCard(order);
+  } catch (err) {
+    span.recordException(err);
+    span.setStatus({ code: SpanStatusCode.ERROR });
+    throw err;
+  } finally {
+    span.end();
+  }
+});
+```
+
+Prefer `startActiveSpan` over `startSpan`: it makes the span current, so
+auto-instrumented HTTP/database spans and nested custom spans parent under
+it. Always call `span.end()` (the `finally` above), or the span never
+exports. When you did not create the span yourself, Foam's
+`recordException(err)` does the exception-plus-ERROR-status pair on the
+current active span for you.
+
+The same applies to the rest of the API: `trace.getActiveSpan()` to tag
+the current request (`span.setAttribute(...)`), and `context.with(...)`
+to run code under a specific context.
+
+This is safe in every state: if another SDK owns the global tracer slot,
+these spans go to that SDK; before `init()` (or when Foam is disabled)
+the API returns a no-op tracer and nothing is recorded. Custom spans
+export wherever `getState().signals.traces` points.
+
+## Network capture
+
+`networkCapture` defaults to `"basic"` and controls extra HTTP detail.
+
+- `"off"` keeps standard OpenTelemetry HTTP/Undici telemetry, with no extra
+  capture.
+- `"basic"` adds allowlisted `content-type`, `content-length`, and
+  `content-encoding` headers.
+- `"advanced"` adds those headers plus full Node `http`/`https` and
+  Fetch/Undici request and response bodies as raw wire bytes, up to 1 MiB. Always set to `"advanced"` when security and compliance allows.
+
+## Redaction
+
+Foam always masks common credentials in captured request and response data,
+URLs, and exported span and log fields. The built-in floor matches both exact
+well-known names and any key containing a sensitive term (`legacy_api_key_2`,
+`stripeToken`), so credential-bearing keys do not need to be enumerated. Add
+application-specific keys with the `redact` option shown under `init(options)`;
+customer keys match exactly after normalization.
+
+Redaction applies to everything Foam exports — span, span event, and link
+attributes, log bodies and attributes, metric data-point attributes, and
+resource attributes — including the ingest helpers. It does not modify
+telemetry sent through other exporters.
+
+## Loggers
+
+Initialize Foam before you import or construct the logger.
+
+What Foam does depends on the logger version:
+
+- Bunyan 1, Pino 7–10, and Winston 3: Foam adds trace IDs to every
+  record and automatically sends a copy to Foam.
+- Pino 5.14–6 and Winston 1–2: Foam adds trace IDs, but these versions
+  have no way to attach a sender, so no copy reaches Foam. Upgrade the
+  logger, or call `log()` yourself.
+- Everything else: Foam does nothing. Use `console.*` (captured
+  below), call `log()`, or upgrade the logger.
+
+### If the app already ships logs somewhere
+
+- To a non-OTel destination (files, another vendor's transport): no
+  conflict. Keep the defaults; Foam sends its own copy alongside.
+- Through its own OTel stream/transport (a Bunyan OTel stream,
+  Winston `OpenTelemetryTransportV3`): those already deliver to Foam
+  via the global LoggerProvider. Set `disableLogSending: true` so
+  records don't arrive twice.
+- Through another OTel SDK's LoggerProvider: set
+  `disableLogSending: true` and attach
+  `createFoamIngestLogRecordProcessor` to that provider (see Public
+  API) so Foam gets a copy.
+- Through `pino-opentelemetry-transport`: it runs in a worker
+  thread and never touches the in-process LoggerProvider, so nothing
+  conflicts. Keep `disableLogSending: false`, or Foam gets no Pino
+  records at all.
+- Through a collector or agent reading stdout/files: no app changes;
+  add a Foam OTLP exporter to the collector config.
+
+`log()` sends a single record straight to Foam in any setup. It is a
+manual per-record call, not a mirror of the logger. Use it for
+targeted, high-value records.
+
+`ConsoleInstrumentation` turns calls to `console.*` into OTel log records.
+It does not capture arbitrary stdout/stderr writes.
+
+```js
+console.log("hi");              // captured
+process.stdout.write("hi\n");   // not captured
+```
+
+## OpenTelemetry configuration and compliance
+
+Foam uses the official OpenTelemetry JavaScript API, SDK, resource,
+propagation, instrumentation, and OTLP exporter packages.
+
+The [OpenTelemetry JavaScript compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix/js.yaml)
+
+## TODO(pcga11): OpenTelemetry profiling
+
+The official OpenTelemetry JavaScript SDK has no in-process Profiles provider, processor, or exporter yet. Track upstream progress:
+
+- [OpenTelemetry JS profiling implementation issue](https://github.com/open-telemetry/opentelemetry-js/issues/6500)
+- [Profiling SIG language SDK support tracker](https://github.com/open-telemetry/sig-profiling/issues/106)
+- [OpenTelemetry Profiles public alpha announcement](https://opentelemetry.io/blog/2026/profiles-alpha/)
+- [OpenTelemetry eBPF profiler](https://github.com/open-telemetry/opentelemetry-ebpf-profiler)
+
+## TODO(pcga11): Anthropic instrumentation
+
+There is no released official OpenTelemetry JavaScript instrumentation for the Anthropic SDK yet, but one is actively in progress in js-contrib, based on the OpenInference donation from Arize. Once `@opentelemetry/instrumentation-anthropic` is released (and picked up by `auto-instrumentations-node`), bundle it here like the OpenAI and aws-sdk ones. Until then, Anthropic coverage is handled case by case with the FDE (see FDE.md). Track upstream progress:
+
+- [feat(instrumentation-anthropic): add basic messages instrumentation](https://github.com/open-telemetry/opentelemetry-js-contrib/pull/3664)
+- [refactor(instrumentation-anthropic): use genai-util library](https://github.com/open-telemetry/opentelemetry-js-contrib/pull/3698)
+- [Tracking: Migrate OpenInference JS instrumentations into opentelemetry-js-contrib](https://github.com/open-telemetry/opentelemetry-js-contrib/issues/3668)
+- [feat: Add @opentelemetry/genai-util package for GenAI instrumentations](https://github.com/open-telemetry/opentelemetry-js-contrib/issues/3681)

package/dist/before-send.d.ts

@@ -0,0 +1,27 @@
+export interface BeforeSendSpanEventEntry {
+    name: string;
+    attributes?: Record<string, unknown>;
+    time?: unknown;
+}
+export interface BeforeSendSpanLink {
+    context: Record<string, unknown>;
+    attributes?: Record<string, unknown>;
+}
+interface BeforeSendSpanEvent {
+    readonly type: "span";
+    name: string;
+    attributes: Record<string, unknown>;
+    events: BeforeSendSpanEventEntry[];
+    links: BeforeSendSpanLink[];
+}
+interface BeforeSendLogEvent {
+    readonly type: "log";
+    body?: unknown;
+    severityText?: string;
+    attributes: Record<string, unknown>;
+}
+export type BeforeSendEvent = BeforeSendSpanEvent | BeforeSendLogEvent;
+export type BeforeSendHook = (event: BeforeSendEvent) => BeforeSendEvent | null;
+export declare function resolveBeforeSend(hook: unknown): BeforeSendHook | undefined;
+export declare function invokeBeforeSend(hook: BeforeSendHook, event: BeforeSendEvent): BeforeSendEvent | null | undefined;
+export {};