@diveinto/obs 1.0.4 → 1.1.0

package/README.md

@@ -35,12 +35,33 @@ of the four, and each of those behaviours is pinned by a test.
   writer on the same file, and the two then rename it out from under each
   other at the rotation boundary. Rotation also refuses to overwrite an
   existing rotated file, so a day of records cannot be lost to a rename.
-- **Trace context** on `AsyncLocalStorage`, continuing an inbound W3C
-  `traceparent` and propagating it outbound, so one `trace_id` spans every
-  hop.
-- **Optional OTLP export** to any OpenTelemetry collector. Off unless
-  `OTEL_EXPORTER_OTLP_ENDPOINT` is set, and the first-party logs never depend
-  on it.
+  A single file also has a **ceiling** (256 MB by default,
+  `<PREFIX>_LOG_JSON_MAX_MB` / `_TEXT_MAX_MB`, 0 to remove): daily rotation
+  bounds how many files there are, not how large one can get, and a service
+  that drops to DEBUG or loops on an error can write more in an afternoon
+  than the fortnight around it. Worst case per service is now a number an
+  operator can reason about rather than the free space on the disk.
+- **Real spans, always** (v1.1.0). `startOtel` registers the tracer whether
+  or not anything is exported, so span ids come from it either way: a
+  request's SERVER span (named by its route template, continuing an inbound
+  W3C `traceparent`, and using the framework's own SERVER span when it has
+  opened one), `withSpan` and `withTrace` for units of work, and
+  `continueFrom` for work another process or a later moment does on a
+  request's behalf. Every record carries the ids of the span it was written
+  in, and `traceHeaders()` sends them on, so one `trace_id` spans every hop.
+- **The span tree in the files.** Each span at a process boundary (server,
+  client, producer, consumer) and each span this service opens is written
+  to the files when it ends, as one record: `event: span.end` with
+  `span.name`, `span.kind`, `span.status`, `dur_ms` and its links. A tool
+  that reads only the files can draw the tree an OTLP backend shows.
+- **Optional OTLP export** to any OpenTelemetry collector, when
+  `OTEL_EXPORTER_OTLP_ENDPOINT` is set; the first-party logs never depend on
+  it. What leaves the process passes an allowlist (no credentials, network
+  addresses, user agents, query strings, full URLs or personal identifiers;
+  more with `exportDenyKeys`; messages, and an error's message, through
+  `exportScrub`); the files keep everything. Export is seen, not assumed: an `obs.export` record every
+  minute counts what was sent and what failed, and a failure is noted in the
+  files at most once a minute, never over the exporter that is failing.
 - **Serverless-aware**: on Vercel or Lambda the file sinks switch themselves
   off rather than failing, because there is no persistent writable disk.
 
@@ -81,6 +102,23 @@ export const GET = (req: Request) =>
 the response. `withAction` does the same for a server action in a fresh root
 trace.
 
+Work done later, or by another process, on a request's behalf: store the
+request's context with it, and continue from it.
+
+```ts
+import { continueFrom, currentTraceparent } from "@diveinto/obs";
+
+// in the request: keep its context with the job
+await db.insert(jobs).values({ id, traceparent: currentTraceparent() });
+
+// in the worker: the job's span is a child of the request's
+await continueFrom(job.traceparent, "process jobs", () => run(job), { kind: "consumer" });
+```
+
+A stored or forwarded context is untrusted input: one that is missing or
+malformed starts a new trace and never fails the work. `links` ties a span to
+the other requests it serves (a batch) without making it their child.
+
 ## Configuration
 
 | field | required | what it does |
@@ -92,6 +130,8 @@ trace.
 | `routeTemplate` | no | Collapse dynamic path segments before a path becomes a metric label: `/orders/abc123` to `/orders/{id}`. Without it every id is its own time series |
 | `traceSeed` | no | Where a request's trace continues from when there is no W3C `traceparent`, for a front end that passes a correlation id of its own |
 | `quietPathPrefixes` | no | Request paths whose completion logs at debug rather than info. Health checks and internal polling otherwise bury what matters |
+| `exportDenyKeys` | no | More attribute and field keys that never leave the process over OTLP, beside the built-in ones. The files keep them |
+| `exportScrub` | no | Rewrites a record's message before it is exported, for a service whose messages can name a person. The files keep the message as written |
 
 ### Environment
 
@@ -100,8 +140,10 @@ Generic, never prefixed: `LOG_LEVEL` (`trace`/`debug`/`info`/`warn`/`error`/
 `OTEL_*` set.
 
 Per service, prefixed: `<PREFIX>_VAR_DIR`, `<PREFIX>_LOG_TEXT_FILE`,
-`_TEXT_ROTATION`, `_TEXT_RETENTION`, and the same three for `_JSON_`. A
-`*_FILE` value is a path, or `1` for the default path, or `0`/`off`.
+`_TEXT_ROTATION`, `_TEXT_RETENTION`, `_TEXT_MAX_MB`, and the same four for
+`_JSON_`. A `*_FILE` value is a path, or `1` for the default path, or
+`0`/`off`; `*_MAX_MB` is the per-file ceiling, 256 by default and 0 for
+none.
 
 ## Cardinality
 
@@ -123,10 +165,13 @@ from the Node branch of `instrumentation.ts` and from route handlers, not
 from Edge middleware.
 
 `@opentelemetry/api` and `@opentelemetry/api-logs` are required peers. The
-SDK packages (`@vercel/otel`, `@opentelemetry/sdk-logs`,
-`@opentelemetry/exporter-logs-otlp-proto`, `@opentelemetry/resources`) are
-optional peers, imported dynamically and needed only if you use OTLP export
-via `@diveinto/obs/start`.
+SDK packages (`@vercel/otel`, `@opentelemetry/sdk-trace-base`,
+`@opentelemetry/sdk-logs`, `@opentelemetry/exporter-logs-otlp-proto`,
+`@opentelemetry/resources`) are optional peers, imported dynamically by
+`@diveinto/obs/start`: install them to have real spans (`@vercel/otel`, and
+`@opentelemetry/sdk-trace-base` for export) and OTLP logs. Without them the
+library falls back to ids of its own on `AsyncLocalStorage`, as before
+v1.1.0.
 
 ## License
 

package/dist/api.d.ts

@@ -1,3 +1,4 @@
+import type { SpanKindName } from "./spans.js";
 import type { Fields, LevelInput } from "./types.js";
 /**
  * Emit one named story beat.
@@ -15,6 +16,7 @@ type SpanOpts = {
     fields?: Fields;
     logger?: string;
     level?: LevelInput;
+    kind?: SpanKindName;
 };
 /**
  * Open a child span around a unit of work: mints a child span id, binds

package/dist/api.js

@@ -9,7 +9,7 @@
  * fifth logEvent to one function, you probably want a span instead.
  */
 import { bind, runWithSpan, runWithTrace } from "./context.js";
-import { otelEnabled, withOtelSpan } from "./otel.js";
+import { withOtelRootSpan, withOtelSpan } from "./otel.js";
 import { emit, normalizeLevel } from "./record.js";
 /** The default logger name for one-off events not tied to a module logger. */
 const DEFAULT_LOGGER = "event";
@@ -48,9 +48,9 @@ export async function withSpan(name, fn, opts) {
             throw err;
         }
     });
-    // Open a real OTel child span when export is on, so the domain story appears
-    // in the backend; the logs inside adopt the OTel span ids via context.current.
-    return otelEnabled() ? withOtelSpan(name, opts?.fields ?? {}, inner) : inner();
+    // A real span, exported or not: the records inside adopt its ids via context.current, and its
+    // own record lands in the files when it ends.
+    return withOtelSpan(name, opts?.fields ?? {}, inner, opts?.kind);
 }
 /**
  * Like withSpan, but opens a fresh ROOT trace (for background work with no
@@ -75,6 +75,6 @@ export async function withTrace(name, fn, opts) {
             throw err;
         }
     });
-    return otelEnabled() ? withOtelSpan(name, opts?.fields ?? {}, inner) : inner();
+    return withOtelRootSpan(name, opts?.fields ?? {}, inner, opts?.seed, opts?.kind);
 }
 export { bind } from "./context.js";

package/dist/config.d.ts

@@ -65,9 +65,21 @@ export type ObsConfig = {
      * Matched by prefix.
      */
     quietPathPrefixes?: string[];
+    /**
+     * More attribute and field keys that never leave the process over OTLP, beside the built-in
+     * ones (credentials, network addresses, user agents, query strings and personal identifiers;
+     * see `exportable` in spans.ts). The files keep everything: they stay on the machine.
+     */
+    exportDenyKeys?: string[];
+    /**
+     * Rewrites a record's message before it is exported over OTLP, for a service whose messages
+     * can name a person (an id, an address). The files keep the message as written.
+     */
+    exportScrub?: (msg: string) => string;
 };
-type Resolved = Required<Pick<ObsConfig, "service" | "envPrefix" | "defaultVarDir" | "fileStem">> & Pick<ObsConfig, "routeTemplate" | "traceSeed"> & {
+type Resolved = Required<Pick<ObsConfig, "service" | "envPrefix" | "defaultVarDir" | "fileStem">> & Pick<ObsConfig, "routeTemplate" | "traceSeed" | "exportScrub"> & {
     quietPathPrefixes: string[];
+    exportDenyKeys: string[];
 };
 export declare function configure(cfg: ObsConfig): void;
 /**

package/dist/config.js

@@ -50,6 +50,8 @@ function resolve(cfg) {
         routeTemplate: cfg.routeTemplate,
         traceSeed: cfg.traceSeed,
         quietPathPrefixes: cfg.quietPathPrefixes ?? [],
+        exportDenyKeys: cfg.exportDenyKeys ?? [],
+        exportScrub: cfg.exportScrub,
     };
 }
 export function configure(cfg) {

package/dist/http.js

@@ -17,7 +17,7 @@ import { bind, currentSpanId, currentTraceId, formatTraceparent, isSampled, pars
 import { logEvent } from "./api.js";
 import { obsConfig } from "./config.js";
 import { metricCounter, metricHistogram } from "./metrics.js";
-import { otelEnabled, withOtelServerSpan, withOtelSpan } from "./otel.js";
+import { withOtelRootSpan, withOtelServerSpan } from "./otel.js";
 /**
  * The headers to attach to an outbound request to the agent so it CONTINUES
  * this trace. Returns `{ traceparent }` when a trace is active, else `{}`. The
@@ -111,10 +111,10 @@ export async function withApiTrace(req, handler, opts) {
             throw err;
         }
     });
-    // When OTLP export is on, open a real SERVER span that CONTINUES the inbound
-    // W3C traceparent, then run the logging body inside it (which now adopts the
-    // OTel ids). When off, just run the body on the AsyncLocalStorage path.
-    return otelEnabled() ? withOtelServerSpan(`${method} ${path}`, req.headers, fields, inner) : inner();
+    // The request's SERVER span, continuing an inbound W3C traceparent, exported or not; the
+    // logging body runs inside it and adopts its ids. Named by the route's template, never the raw
+    // path: a span name is a grouping key, and an id in it makes every request its own group.
+    return withOtelServerSpan(`${method} ${routeLabel(path)}`, req.headers, { ...fields, "http.route": routeLabel(path) }, inner);
 }
 /**
  * Wrap a server action (which has no NextRequest) in a FRESH root trace named
@@ -149,9 +149,8 @@ export async function withAction(name, fn, fields) {
             throw err;
         }
     });
-    // A server action has no inbound request, so when export is on this opens a
-    // fresh root OTel span (startActiveSpan with no active parent).
-    return otelEnabled() ? withOtelSpan(`action ${name}`, { "action.name": name, ...(fields ?? {}) }, inner) : inner();
+    // A server action has no inbound request: a fresh root span of its own.
+    return withOtelRootSpan(`action ${name}`, { "action.name": name, ...(fields ?? {}) }, inner);
 }
 /**
  * Record the action duration histogram + invocation counter with the SAME

package/dist/index.d.ts

@@ -53,7 +53,8 @@ export { metricCounter, metricHistogram, timed } from "./metrics.js";
 export { renderJson, renderPretty } from "./render.js";
 export { configureSinks, makeFileSink, makeStdoutSink } from "./sinks.js";
 export { isServerless, resetLoggingForTests, resolveFileTargets, setupLogging, type FileTarget, } from "./setup.js";
-export { currentIds, makeOtelLogSink, otelEnabled, withOtelServerSpan, withOtelSpan } from "./otel.js";
+export { currentIds, makeOtelLogSink, otelEnabled, withOtelRootSpan, withOtelServerSpan, withOtelSpan, } from "./otel.js";
+export { continueFrom, currentTraceparent, exportable, exportCounters, spanContextFromTraceparent, traceparentOf, type ExportCounters, type SpanKindName, } from "./spans.js";
 export { resolveTraceSeed, traceHeaders, withAction, withApiTrace } from "./http.js";
 /**
  * Configure the library and hand back the surface an app actually uses.

package/dist/index.js

@@ -52,7 +52,8 @@ export { metricCounter, metricHistogram, timed } from "./metrics.js";
 export { renderJson, renderPretty } from "./render.js";
 export { configureSinks, makeFileSink, makeStdoutSink } from "./sinks.js";
 export { isServerless, resetLoggingForTests, resolveFileTargets, setupLogging, } from "./setup.js";
-export { currentIds, makeOtelLogSink, otelEnabled, withOtelServerSpan, withOtelSpan } from "./otel.js";
+export { currentIds, makeOtelLogSink, otelEnabled, withOtelRootSpan, withOtelServerSpan, withOtelSpan, } from "./otel.js";
+export { continueFrom, currentTraceparent, exportable, exportCounters, spanContextFromTraceparent, traceparentOf, } from "./spans.js";
 export { resolveTraceSeed, traceHeaders, withAction, withApiTrace } from "./http.js";
 /**
  * Configure the library and hand back the surface an app actually uses.

package/dist/otel-start.d.ts

@@ -1,6 +1,17 @@
 /**
- * Start OTLP export iff an endpoint is configured. Called once from
- * instrumentation.ts on the Node runtime. Idempotent; returns whether export is
- * live. Never throws - on any failure we stay first-party only.
+ * Register the tracer, always, and OTLP export when an endpoint is configured.
+ * Called once from instrumentation.ts on the Node runtime. Idempotent; returns
+ * whether export is live. Never throws - on any failure we stay first-party
+ * only.
  */
 export declare function startOtel(serviceName: string): Promise<boolean>;
+/**
+ * Where a signal is exported, from the standard environment: the signal's own
+ * endpoint (`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, used as given), else the
+ * shared one with `/v1/<signal>` appended; and the headers, shared then the
+ * signal's own, each `key=value` comma separated and percent-decoded.
+ */
+export declare function otlpTarget(signal: "traces" | "logs", env?: NodeJS.ProcessEnv): {
+    url: string;
+    headers: Record<string, string>;
+};

package/dist/otel-start.js

@@ -9,46 +9,98 @@
  *
  * Published as the `@diveinto/obs/start` subpath for the same reason.
  *
+ * The tracer is registered whether or not an endpoint is configured: span ids
+ * come from it either way, and each span's record goes to the files when it
+ * ends (spans.ts), so the files hold the same tree with export on and off.
+ * Only the exporters depend on the endpoint.
+ *
  * Never throws: on any failure the service stays first-party only, still
  * writing stdout and its two files. Losing export is not worth losing logs.
  */
 import { logs } from "@opentelemetry/api-logs";
+import { FileSpanProcessor, GuardedLogExporter, GuardedSpanExporter, startExportHeartbeat, } from "./spans.js";
 // Module-local to the start path: `otelEnabled()` in otel.ts reads the env
 // instead, because Next can run instrumentation.ts in a different module
 // context from the route handlers, where a flag set here would read false.
 let enabled = false;
 let started = false;
+// Batching for both exporters: a bounded queue that drops when full rather than
+// blocking the work that made the span, and an export that gives up rather than
+// holding a request or a shutdown behind a slow backend.
+const BATCH = { maxQueueSize: 2048, maxExportBatchSize: 512, scheduledDelayMillis: 2000, exportTimeoutMillis: 10_000 };
 /**
- * Start OTLP export iff an endpoint is configured. Called once from
- * instrumentation.ts on the Node runtime. Idempotent; returns whether export is
- * live. Never throws - on any failure we stay first-party only.
+ * Register the tracer, always, and OTLP export when an endpoint is configured.
+ * Called once from instrumentation.ts on the Node runtime. Idempotent; returns
+ * whether export is live. Never throws - on any failure we stay first-party
+ * only.
  */
 export async function startOtel(serviceName) {
     if (started)
         return enabled;
     started = true;
-    if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT)
-        return false;
+    const exportOn = !!process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
     try {
-        const { registerOTel } = await import("@vercel/otel");
-        // registerOTel reads OTEL_EXPORTER_OTLP_* (endpoint/headers/protocol) and
-        // OTEL_RESOURCE_ATTRIBUTES from the env. serviceName is the fallback label.
+        const { registerOTel, OTLPHttpProtoTraceExporter } = await import("@vercel/otel");
+        const processors = [new FileSpanProcessor()];
+        if (exportOn) {
+            const { BatchSpanProcessor } = await import("@opentelemetry/sdk-trace-base");
+            const exporter = new GuardedSpanExporter(new OTLPHttpProtoTraceExporter(otlpTarget("traces")));
+            processors.push(new BatchSpanProcessor(exporter, BATCH));
+        }
         // TRACES + LOGS only, on purpose: no metricReader is passed here, so no
         // metrics leave the process regardless of env vars. This is a scar, not a
         // preference: auto-exporting metrics alongside traces once produced tens
         // of millions of samples a month that nobody read. Do not add a
         // metricReader without a named consumer, an export allowlist, and a
         // cardinality cap on every attribute.
-        registerOTel({ serviceName });
-        await startLogExport(serviceName);
-        enabled = true;
-        return true;
+        //
+        // Every trace is sampled: an inbound "not sampled" flag from outside must
+        // not turn tracing off for the work it reaches.
+        registerOTel({
+            serviceName: process.env.OTEL_SERVICE_NAME || serviceName,
+            traceSampler: "always_on",
+            spanProcessors: processors,
+            traceExporter: undefined,
+        });
+        if (exportOn)
+            await startLogExport(serviceName);
+        startExportHeartbeat(exportOn);
+        enabled = exportOn;
+        return exportOn;
     }
     catch (e) {
         process.stderr.write(`[obs] OTel init failed (${String(e)}); staying first-party only\n`);
         return false;
     }
 }
+/**
+ * Where a signal is exported, from the standard environment: the signal's own
+ * endpoint (`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, used as given), else the
+ * shared one with `/v1/<signal>` appended; and the headers, shared then the
+ * signal's own, each `key=value` comma separated and percent-decoded.
+ */
+export function otlpTarget(signal, env = process.env) {
+    const own = env[`OTEL_EXPORTER_OTLP_${signal.toUpperCase()}_ENDPOINT`];
+    const shared = (env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "").replace(/\/+$/, "");
+    const url = own || `${shared}/v1/${signal}`;
+    const headers = {};
+    for (const raw of [env.OTEL_EXPORTER_OTLP_HEADERS, env[`OTEL_EXPORTER_OTLP_${signal.toUpperCase()}_HEADERS`]]) {
+        for (const pair of (raw ?? "").split(",")) {
+            const eq = pair.indexOf("=");
+            if (eq <= 0)
+                continue;
+            const k = pair.slice(0, eq).trim();
+            const v = pair.slice(eq + 1).trim();
+            try {
+                headers[k] = decodeURIComponent(v);
+            }
+            catch {
+                headers[k] = v;
+            }
+        }
+    }
+    return { url, headers };
+}
 /**
  * Set up the OTLP LOGS signal (separate from traces, which @vercel/otel owns).
  * Installs a global LoggerProvider + OTLP/protobuf log exporter, so the log sink
@@ -67,9 +119,10 @@ async function startLogExport(serviceName) {
         // also pull service.version etc. from OTEL_RESOURCE_ATTRIBUTES so the LOGS
         // resource matches the TRACES resource registerOTel builds.
         const resource = defaultResource().merge(resourceFromAttributes(logResourceAttrs(serviceName)));
+        const exporter = new GuardedLogExporter(new OTLPLogExporter());
         const provider = new LoggerProvider({
             resource,
-            processors: [new BatchLogRecordProcessor(new OTLPLogExporter())],
+            processors: [new BatchLogRecordProcessor(exporter, BATCH)],
         });
         logs.setGlobalLoggerProvider(provider);
     }

package/dist/otel.d.ts

@@ -1,10 +1,21 @@
+import { type SpanKindName } from "./spans.js";
 import { type Fields, type Sink } from "./types.js";
+/**
+ * Whether OTLP export is configured: what the log sink and the metrics read. Spans never ask; the
+ * tracer runs either way. Env-based (process-global) on purpose: Next can run instrumentation.ts in
+ * a different module context than the route handlers, so a per-module flag set by startOtel would
+ * read false in handlers.
+ */
 export declare function otelEnabled(): boolean;
 /**
- * A Sink that ships each obs record to the OTLP backend as a log, correlated to
- * the active span (emit() reads the current context, and our records are emitted
- * inside the request/domain span). Returns null when export is off. The internal
- * export fetch does not pass back through our sinks, so there is no feedback loop.
+ * A Sink that ships each obs record to the OTLP backend as a log carrying the record's own trace
+ * and span ids, not only an active span's: a record written for a span after it ended, or in work
+ * continued from a stored traceparent, still sits in its trace. Returns null when export is off.
+ *
+ * Two kinds of record stay in the files: a span's own record (the span itself is exported) and one
+ * marked `obs.local` (an export failure, which would otherwise travel over the failing exporter).
+ * What does leave passes `exportable`, and the message passes the service's `exportScrub`. The
+ * internal export fetch does not pass back through our sinks, so there is no feedback loop.
  */
 export declare function makeOtelLogSink(): Sink | null;
 /** The active OTel span's ids as W3C hex, or null when there is no valid span.
@@ -16,11 +27,20 @@ export declare function currentIds(): {
     parentSpanId: string;
 } | null;
 /**
- * Open a SERVER span continuing the inbound W3C traceparent (extracted from the
- * request headers), so this service's span nests under the caller's trace. A
- * no-op passthrough when export is off.
+ * The request's SERVER span. When the framework has already opened one for this request (Next.js
+ * does, continuing an inbound traceparent itself), that span is named `name` and given `attrs`,
+ * and `fn` runs in it: one SERVER span per request, never two siblings. Otherwise a SERVER span is
+ * opened here, continuing the inbound W3C traceparent extracted from the headers.
  */
 export declare function withOtelServerSpan<T>(name: string, headers: Headers, attrs: Fields, fn: () => Promise<T>): Promise<T>;
-/** Open a child span around a unit of work (domain story / outbound). A no-op
- *  passthrough when export is off. */
-export declare function withOtelSpan<T>(name: string, attrs: Fields, fn: () => Promise<T>): Promise<T>;
+/** Open a child span around a unit of work: internal by default, `kind` for a call out or a message. */
+export declare function withOtelSpan<T>(name: string, attrs: Fields, fn: () => Promise<T>, kind?: SpanKindName): Promise<T>;
+/**
+ * Open a span that starts a trace of its own, whatever span is in force: background work, a server
+ * action. `seed` continues a trace whose ids arrived another way (a stored or forwarded context):
+ * its trace id and, as the parent, its span id.
+ */
+export declare function withOtelRootSpan<T>(name: string, attrs: Fields, fn: () => Promise<T>, seed?: {
+    traceId?: string;
+    parentSpanId?: string;
+}, kind?: SpanKindName): Promise<T>;