@foam-ai/node 0.1.0-alpha.3 → 0.1.0-alpha.5

package/README.md

@@ -1,123 +1,586 @@
-# @foam-ai/node
+# Foam OpenTelemetry SDK for Node.js
 
-Foam observability SDK for Node.js. Sends traces, logs, and metrics to Foam with zero configuration overhead.
+`@foam-ai/node` is Foam's OpenTelemetry distribution for Node.js.
+
+## Requirements
+
+- Node.js `^18.19.0 || >=20.6.0`, matching the supported engine range of
+    `@opentelemetry/auto-instrumentations-node`
+- Foam ingest token, stored as `FOAM_OTEL_TOKEN`
 
 ## Install
 
-```bash
+```sh
 npm install @foam-ai/node
 ```
 
-## Quick start
+## Initialize
+
+Call `init()` before your app loads so OpenTelemetry can patch libraries as
+they import. Put `init()` in its own file and preload it with `--import` or
+`--require` (below).
+
+Set `enabled: false` to turn Foam off.
+
+Store the ingest token as `FOAM_OTEL_TOKEN` and pass
+`process.env.FOAM_OTEL_TOKEN` to `init()`.
+
+### TypeScript
+
+Create `instrumentation.ts`:
+
+```ts
+import { init } from "@foam-ai/node";
+
+init({
+  name: "checkout-api",
+  environment: "production",
+  enabled: true,
+  token: process.env.FOAM_OTEL_TOKEN,
+  version: "1.0.0",
+});
+```
+
+Compile it with the app. If TypeScript emits ESM, preload with `--import`
+(same rule as ESM below):
+
+```sh
+node --import ./dist/instrumentation.js ./dist/app.js
+```
+
+If TypeScript emits CommonJS, preload with `--require`:
+
+```sh
+node --require ./dist/instrumentation.js ./dist/app.js
+```
+
+To run TypeScript without compiling first:
+
+```sh
+node --import tsx --import ./instrumentation.ts ./src/app.ts
+```
+
+### ESM
+
+Create `instrumentation.mjs` with the same `init()` call.
+
+`import` lines always run first, even if you write `init()` above them.
+Don't import your app from that file. Use `--import`, or
+`await import("./app.js")` after `init()`:
+
+```sh
+node --import ./instrumentation.mjs ./app.js
+```
+
+### CommonJS
+
+Create `instrumentation.cjs`. `require()` runs in order, so `init()` then
+`require("./app")` in this file is fine. `--require` is still the better
+preload:
+
+```js
+const { init } = require("@foam-ai/node");
+
+init({
+  name: "checkout-api",
+  environment: "production",
+  enabled: true,
+  token: process.env.FOAM_OTEL_TOKEN,
+  version: "1.0.0",
+});
+```
+
+```sh
+node --require ./instrumentation.cjs ./app.js
+```
+
+## Public API
+
+### `init(options)`
+
+Sets up tracing, metrics, logs, auto-instrumentation, OTLP export to Foam,
+and W3C context propagation.
+
+Call this once. Use it when Foam should run OpenTelemetry for the process.
+
+```js
+import { init } from "@foam-ai/node";
+
+init({
+  name: "checkout-api",
+  environment: "production",
+  enabled: true,
+  token: process.env.FOAM_OTEL_TOKEN,
+  version: "1.0.0",
+});
+```
+
+```ts
+init({
+  name,
+  environment,
+  enabled,
+  token,
+  endpoint?,
+  version?,
+  sampleRate?,
+  additionalInstrumentations?,
+  additionalSpanProcessors?,
+  additionalLogRecordProcessors?,
+  additionalMetricReaders?,
+  additionalResourceAttributes?,
+  disableLogSending?,
+  networkCapture?,
+  ignoredOutboundHosts?,
+  redact?,
+  beforeSend?,
+}): void
+```
+
+`name`, `environment`, and `enabled` are required.
+`token` is required only when Foam is enabled; store it as `FOAM_OTEL_TOKEN`.
+
+`endpoint` defaults to Foam's OTLP endpoint and is where all telemetry is
+delivered. Override it only to route through a Foam-compatible OTLP
+gateway, such as an egress proxy or a local collector. It must be an
+`http(s)` URL; its host is always excluded from outbound tracing so
+export calls do not become client spans.
+
+`version` sets `service.version`. 
+
+`sampleRate` is a number from `0` to `1` and defaults to `1`. Rarely if ever updated. Only change it if you are sending telemetry in the petabytes range.
+
+`additionalInstrumentations` adds extra OpenTelemetry instrumentations next
+to Foam's Node auto bundle and `ConsoleInstrumentation`.
+
+`additionalSpanProcessors`, `additionalLogRecordProcessors`, and
+`additionalMetricReaders` add extra processors/readers from other vendors next to Foam's
+exporters. Note: Never add Foam's own processors/readers helpers.
+
+`additionalResourceAttributes` adds extra resource attributes (for example
+`team` or `cloud.region`) to all telemetry.
+
+`disableLogSending` defaults to `false`. Set `true` to stop copying logger
+records into Foam (see Loggers).
+
+`networkCapture` defaults to `"basic"`. Upgrade to `"advanced"` unless you
+cannot; it gives much more powerful HTTP and Undici capture. See Network capture.
+
+`redact` and `beforeSend` are accepted but **not implemented yet**; passing
+them logs a warning and they are ignored. `redact` will add extra
+sensitive keys (strings or RegExps) to the built-in body redaction list.
+`beforeSend` will run on every captured body before it leaves the
+process, returning the (possibly edited) text or `null` to drop the
+capture. Until they ship, rely on the built-in redaction (see Network
+capture).
+
+`ignoredOutboundHosts` skips outbound HTTP tracing for those hostnames.
+The configured `endpoint` host is always skipped so export calls do not
+become client spans. Add extra hosts for high-volume clients that would drown traces,
+for example a sidecar health-check host (`localhost`) or another vendor's
+OTLP ingest host you already export to.
+
+TODO(pcga11): Add other common Gen AI providers.
+
+If another SDK already registered traces, metrics, logs, or the propagator,
+`init()` leaves the signal's slot untouched and only takes what is available. When a slot is taken, use its ingest helper instead.
+
+### `createFoamIngestSpanProcessor(name, environment, token)`
+
+Returns a span processor that exports traces to Foam without taking over
+the global TracerProvider.
+
+Use this when another OpenTelemetry SDK already owns traces and you only
+need Foam as an extra export destination.
+
+```js
+import { createFoamIngestSpanProcessor } from "@foam-ai/node";
+
+const spanProcessor = createFoamIngestSpanProcessor(
+  "checkout-api",
+  "production",
+  process.env.FOAM_OTEL_TOKEN,
+);
+tracerProvider.addSpanProcessor(spanProcessor);
+```
+
+### `createFoamIngestLogRecordProcessor(name, environment, token)`
+
+Returns a log-record processor that exports logs to Foam without taking over
+the global LoggerProvider.
+
+Use this when another SDK already owns logs and you want those records in
+Foam.
+
+```js
+import { createFoamIngestLogRecordProcessor } from "@foam-ai/node";
+
+const logRecordProcessor = createFoamIngestLogRecordProcessor(
+  "checkout-api",
+  "production",
+  process.env.FOAM_OTEL_TOKEN,
+);
+loggerProvider.addLogRecordProcessor(logRecordProcessor);
+```
+
+### `createFoamIngestMetricReader(name, environment, token)`
+
+Returns a metric reader that exports metrics to Foam without taking over
+the global MeterProvider.
+
+Use this when another SDK already owns metrics and you want those meters in
+Foam.
+
+```js
+import { createFoamIngestMetricReader } from "@foam-ai/node";
+
+const metricReader = createFoamIngestMetricReader(
+  "checkout-api",
+  "production",
+  process.env.FOAM_OTEL_TOKEN,
+);
+meterProvider.addMetricReader(metricReader);
+```
+
+If the other SDK already set `service.name` or environment,
+Foam's values overwrite them on that copy only. The original spans, logs,
+and metrics are left unchanged. For the Foam SDK, ensure all of the signals (spans, logs and metrics)
+carry identical `service.name`.
+
+Don't use `init()` and an ingest helper for the same signal. If another
+SDK already owns traces (or another signal), call `init()` for the rest
+and attach ingest only to the provider you don't own. Don't pass ingest
+helpers as `additionalSpanProcessors` or extra readers: `init()` already
+installs Foam exporters.
 
-Call `init()` as early as possible in your application entry point:
+Ingest helpers do not register instrumentations. If you cannot call
+`init()`, register them with the official OpenTelemetry API
+before the app imports those libraries:
 
-```typescript
-import foam from '@foam-ai/node';
+```js
+import { registerInstrumentations } from "@opentelemetry/instrumentation";
+import { RedisInstrumentation } from "@opentelemetry/instrumentation-redis";
+import { createFoamIngestSpanProcessor } from "@foam-ai/node";
 
-foam.init({
-  apiKey: process.env.FOAM_API_KEY,
-  serviceName: 'my-service',
-  isProduction: process.env.NODE_ENV === 'production',
+registerInstrumentations({
+  instrumentations: [new RedisInstrumentation()],
 });
 ```
 
-That's it. In production, Foam will automatically collect:
+Attach the ingest helper to the other SDK's provider as in the examples
+above. If that SDK already instruments the same library, don't register a
+second copy.
+
+OpenTelemetry allows only one global propagator per signal. Foam registers W3C Trace
+Context and W3C Baggage (`traceparent`, `tracestate`, `baggage`), not B3,
+Jaeger, or AWS X-Ray. Auto-instrumented HTTP uses whichever propagator
+won. If another SDK already registered a propagator, Foam keeps it and
+`getState().signals.baggage` is `false`. A non-W3C propagator will not
+send `traceparent`, so a Foam peer starts a new trace. Foam does not add
+W3C headers on top.
+
+A work around is to initialize Foam first, or include W3C in the other
+SDK's composite, if those headers must be on HTTP.
+
+TODO(pcga11): Handle traceparent conflict^
+
+### `injectTraceContext(headers)`
+
+Writes W3C `traceparent`, `tracestate`, and `baggage` onto a header map.
+
+Use this when you send a message yourself (Kafka, a queue, a custom
+socket). Auto-instrumented HTTP, Express, Undici, and Fetch already inject
+these headers; don't call this on ordinary HTTP. This uses Foam's local
+W3C propagator so it works even if Foam did not get the global one.
+
+```js
+const headers = {};
+injectTraceContext(headers);
+await kafka.send({ value: payload, headers });
+```
+
+### `extractTraceContext(headers)`
+
+Reads W3C trace and baggage headers and returns a parent OpenTelemetry
+context.
+
+Use this when you receive a message on a custom transport so the tracing carries over. Missing to extact resets the context on these headers effectively losing the context previously injected.
+
+```js
+import { context } from "@opentelemetry/api";
+
+const parent = extractTraceContext(message.headers);
+await context.with(parent, () => handle(message));
+```
+
+### `setBaggage(key, value)`
+
+Stores a small string on the current request context.
+
+Use this for request-scoped identity (tenant, user id) that should follow
+the request across `await`s without being passed through every function.
+
+Values do not leak to other concurrent requests. Caveat: auto-instrumented HTTP
+does not send these keys until you call `injectTraceContext`.
+
+```js
+setBaggage("tenant.id", "acme");
+```
+
+### `getBaggage(key)`
+
+Returns a baggage value from the current request, or `undefined`.
+
+Use this later in the same request to read a value set with `setBaggage`,
+or a key that arrived on an inbound `baggage` header.
+
+```js
+const tenant = getBaggage("tenant.id"); // "acme"
+```
+
+### `incrementCounter(name, value?, attributes?, options?)`
 
-- **Traces** — distributed traces via OpenTelemetry
-- **Logs** — console.log/info/error/warn/debug output
-- **Metrics** — runtime and custom metrics
-- **Uncaught errors** — process-level `uncaughtException` and `unhandledRejection`
+Creates the counter if it does not exist, then increments it.
 
-In non-production environments, `init()` is a silent no-op.
+This counter can only increases. Use this for event counts.
 
-## Capturing errors
+```js
+incrementCounter("orders.created", 1, { "cloud.region": "us-west-1" });
+```
 
-Unhandled exceptions and rejections are captured automatically. For caught errors you want to report, use `captureException`:
+### `recordHistogram(name, value, attributes?, options?)`
 
-```typescript
-import { captureException } from '@foam-ai/node';
+Creates the histogram if it does not exist, then records a measurement.
+
+Use this for values you want percentiles or averages of: latency, duration, payload size.
+
+```js
+recordHistogram("checkout.duration", 0.142, undefined, { unit: "s" });
+```
+
+### `addUpDownCounter(name, value, attributes?, options?)`
+
+Creates the up-down counter if it does not exist, then adds a signed
+delta. The value can go up or down.
+
+Use this for occupancy: active jobs, open connections, items in a pool.
+
+In other terms, pass +value when something starts occupying a slot, -value when it leaves. The metric is the current state of occupancy.
+
+```js
+addUpDownCounter("jobs.active", -1);
+```
+
+### `setMetric(name, value, attributes?, options?)`
+
+Creates the gauge if it does not exist, then sets its current value.
+
+Use this for a point-in-time level: queue depth, heap used, cache size.
+
+```js
+setMetric("queue.depth", 27);
+```
+
+TODO(pcga11): Investigate if it's worth support observable (pull) instruments API.
+
+### `log(body, severity?, attributes?)`
+
+Sends a log record to Foam on Foam's own export path. It does not write to the app's logger, and it does not emit on a LoggerProvider Foam does
+not own.
+
+Use this to send logs directly. Typically used for Pino 5–6, Winston 1–2 as they do not support sending logs.
+
+```js
+import { log, SeverityNumber } from "@foam-ai/node";
+
+log("checkout failed");
+log("checkout failed", SeverityNumber.ERROR);
+```
+
+### `recordException(error)`
+
+Records an exception on the current active span and sets the span status
+to `ERROR`. Accepts an `Error` or any value (stringified). Never throws;
+does nothing when no span is active or the span is not recording.
+
+Use this in `catch` blocks inside auto-instrumented handlers (HTTP
+routes, message consumers) where you did not create the span yourself.
+The exception lands on whichever SDK owns the active span, so it works
+even when another SDK owns tracing.
+
+```js
+import { recordException } from "@foam-ai/node";
 
 try {
-  await chargeCustomer(order);
+  await chargeCard(order);
 } catch (err) {
-  captureException(err);
-  return res.status(500).json({ error: 'Payment failed' });
+  recordException(err);
+  throw err;
 }
 ```
 
-`captureException` works with or without an active trace span. It always emits an OpenTelemetry log record so errors reach Foam regardless of instrumentation setup.
+### `getState()`
+
+Returns whether Foam initialized, which instrumentations registered, and
+which signals have an export path. 
+
+Use this in health checks, tests, or diagnostics to confirm Foam is
+running.
+
+```js
+import { getState } from "@foam-ai/node";
+
+getState();
+// {
+//   initialized: true,
+//   instrumentations: ["http", "express", "pg", ...],
+//   signals: {
+//     traces: "global",
+//     metrics: "global",
+//     logs: "global",
+//     baggage: "global",
+//     profile: "none",
+//   },
+// }
+```
 
-## Auto-instrumentation (optional)
+`instrumentations` lists only the instrumentations registered with a
+successfully started Foam SDK, after upstream defaults and
+`OTEL_NODE_ENABLED_INSTRUMENTATIONS` /
+`OTEL_NODE_DISABLED_INSTRUMENTATIONS` filtering. Registered means the
+instrumentation is ready to patch a supported module when that module
+loads.
 
-By default, no auto-instrumentations are registered. This keeps the SDK webpack/Turbopack-safe with no bundler configuration needed.
+Each `signals` value names the source of the Foam export path for that
+signal:
 
-For plain Node.js servers (Express, Fastify, etc.), you can opt into automatic HTTP, DNS, and framework instrumentation:
+- `"global"` — `init()` registered Foam's provider in the global
+  OpenTelemetry slot.
+- `"ingest"` — a `createFoamIngest*` processor/reader was constructed for
+  another SDK's pipeline. Foam cannot tell whether you actually registered
+  the returned object.
+- `"local"` — logs only: another SDK owns the global LoggerProvider, but
+  Foam keeps its own local provider so `log()` still delivers to Foam.
+- `"none"` — no Foam export path.
 
-```typescript
-import foam from '@foam-ai/node';
-import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
+A value other than `"none"` means telemetry is expected, not that the app
+has already produced or exported it.
 
-foam.init({
-  apiKey: process.env.FOAM_API_KEY,
-  serviceName: 'my-api',
-  isProduction: process.env.NODE_ENV === 'production',
-  instrumentations: getNodeAutoInstrumentations(),
-});
-```
+## Custom spans and other OpenTelemetry APIs
 
-```bash
-npm install @opentelemetry/auto-instrumentations-node
+Foam does not wrap span creation. To start your own spans, read the
+current span, or work with context, import `@opentelemetry/api` directly.
+`init()` registers Foam's providers in the global OpenTelemetry slots, so
+the official API routes to Foam automatically.
+
+Install the API in your app so it resolves one compatible `1.x` copy:
+
+```sh
+npm install @opentelemetry/api
 ```
 
-## Next.js
+```js
+import { trace, SpanStatusCode } from "@opentelemetry/api";
 
-In a Next.js app, create an `instrumentation.ts` file at the project root:
+const tracer = trace.getTracer("checkout");
 
-```typescript
-export async function register() {
-  if (process.env.NEXT_RUNTIME === 'nodejs') {
-    const foam = await import('@foam-ai/node');
-    foam.init({
-      apiKey: process.env.FOAM_API_KEY!,
-      serviceName: 'my-nextjs-app',
-      isProduction: process.env.NODE_ENV === 'production',
-    });
+await tracer.startActiveSpan("charge-card", async (span) => {
+  try {
+    await chargeCard(order);
+  } catch (err) {
+    span.recordException(err);
+    span.setStatus({ code: SpanStatusCode.ERROR });
+    throw err;
+  } finally {
+    span.end();
   }
-}
+});
 ```
 
-No webpack externals or bundler config needed.
+Prefer `startActiveSpan` over `startSpan`: it makes the span current, so
+auto-instrumented HTTP/database spans and nested custom spans parent under
+it. Always call `span.end()` (the `finally` above), or the span never
+exports. When you did not create the span yourself, Foam's
+`recordException(err)` does the exception-plus-ERROR-status pair on the
+current active span for you.
 
-## Coexistence with other SDKs
+The same applies to the rest of the API: `trace.getActiveSpan()` to tag
+the current request (`span.setAttribute(...)`), and `context.with(...)`
+to run code under a specific context.
 
-Foam detects each OpenTelemetry signal independently. If another SDK (Sentry, @vercel/otel, Datadog, etc.) has already registered a TracerProvider, LoggerProvider, or MeterProvider, Foam attaches its exporters on top rather than replacing them. Both SDKs receive data.
+This is safe in every state: if another SDK owns the global tracer slot,
+these spans go to that SDK; before `init()` (or when Foam is disabled)
+the API returns a no-op tracer and nothing is recorded. Custom spans
+export wherever `getState().signals.traces` points.
 
-If no providers exist, Foam creates its own. This means Foam works in any combination:
+## Network capture
 
-- Foam alone
-- Foam + Sentry
-- Foam + @vercel/otel
-- Foam + any OpenTelemetry-compatible SDK
+`networkCapture` defaults to `"basic"` and controls extra HTTP detail.
 
-## API
+- `"off"` keeps standard OpenTelemetry HTTP/Undici telemetry, with no extra
+  capture.
+- `"basic"` adds allowlisted `content-type`, `content-length`, and
+  `content-encoding` headers.
+- `"advanced"` adds those headers plus full Node `http`/`https` and
+  Fetch/Undici request and response bodies as raw wire bytes, up to 1 MiB. Always set to `"advanced"` when security and compliance allows.
 
-### `init(options)`
 
-| Option | Type | Required | Description |
-|---|---|---|---|
-| `apiKey` | `string` | Yes | Your Foam API key |
-| `serviceName` | `string` | Yes | Name of your service (appears in Foam dashboard) |
-| `isProduction` | `boolean` | Yes | Set to `true` to enable telemetry. `false` = no-op. |
-| `instrumentations` | `InstrumentationBase[]` | No | OpenTelemetry instrumentations to register |
+Headers are captured from an allowlist only, so sensitive headers such as
+`authorization` and `cookie` never leave the process. Captured JSON and
+form-urlencoded bodies are redacted before emission: values of sensitive
+keys (`password`, `token`, `api_key`, `client_secret`, `ssn`, and similar,
+matched case-insensitively across `-`, `_`, and `.` separators) are
+replaced with `[REDACTED]` on both the span body content and the emitted
+body chunks. When redaction changes a compressed body, the chunks carry
+the redacted text instead of the raw wire bytes. Other text and binary
+media types are captured unmodified.
+
+## Loggers
+
+Foam follows the compatibility ranges of its OpenTelemetry logger
+instrumentations:
+
+- Bunyan `>=1 <2`
+- Pino `>=5.14 <11`
+- Winston `>=1 <4`
+
+Initialize Foam before you import or construct the logger. Registration alone does not mean a logger
+was loaded, patched, or produced records.
+
+With `disableLogSending: false` (the default), Foam SDK sends:
+
+- Bunyan 1 records through an added OTel stream
+- Pino 7–10 records through an added main-thread `multistream`
+- Winston 3 records through the bundled `OpenTelemetryTransportV3`
+
+Pino 5–6 and Winston 1–2 still get correlation. But they have no sending
+path (`pino.multistream` starts at 7; Winston transports start at 3), so Foam never receives a copy. Call `log()` to send a record yourself.
+
+Set `disableLogSending: true` when the app already exports logger records
+itself (a Bunyan OTel stream, `pino-opentelemetry-transport`, a Winston
+OTel transport, or another pipeline). That turns off sending for Bunyan
+1, Pino 7–10, and Winston 3.
+
+`ConsoleInstrumentation` turns calls to `console.*` into OTel log records.
+It does not capture arbitrary stdout/stderr writes.
+
+```js
+console.log("hi");              // captured
+process.stdout.write("hi\n");   // not captured
+```
+
+## OpenTelemetry configuration and compliance
+
+Foam uses the official OpenTelemetry JavaScript API, SDK, resource,
+propagation, instrumentation, and OTLP exporter packages.
 
-### `captureException(error)`
+The [OpenTelemetry JavaScript compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix/js.yaml)
 
-Captures a caught error and sends it to Foam. Works with or without an active span.
+## TODO(pcga11): OpenTelemetry profiling
 
-## Design
+The official OpenTelemetry JavaScript SDK has no in-process Profiles provider, processor, or exporter yet. Track upstream progress:
 
-- **HTTP-only exporters** — no gRPC, no Node.js built-in dependencies that break bundlers
-- **Zero latency** — all setup is synchronous; exporting happens in background batch intervals
-- **Crash-safe** — each signal (traces, logs, metrics, console, process handlers) is independently wrapped in try/catch
-- **Idempotent** — safe to call `init()` multiple times; console patching and process handlers use Symbols to prevent double-registration
+- [OpenTelemetry JS profiling implementation issue](https://github.com/open-telemetry/opentelemetry-js/issues/6500)
+- [Profiling SIG language SDK support tracker](https://github.com/open-telemetry/sig-profiling/issues/106)
+- [OpenTelemetry Profiles public alpha announcement](https://opentelemetry.io/blog/2026/profiles-alpha/)
+- [OpenTelemetry eBPF profiler](https://github.com/open-telemetry/opentelemetry-ebpf-profiler)