@eventferry/kafka 3.3.1 → 3.5.0

package/CHANGELOG.md

@@ -1,5 +1,65 @@
 # @eventferry/kafka
 
+## 3.5.0
+
+### Minor Changes
+
+- fb0549d: Producer-fenced restart. `PRODUCER_FENCED` and `INVALID_PRODUCER_EPOCH` errors now classify as `errorKind: "fenced"` (previously bundled into `"fatal"`). The new kind is documented as **transient by default** — fences also fire on broker restart and network partition recovery, not only on multi-instance collisions.
+
+  New publisher option `autoRecoverFromFence: boolean` (default `false`): when on, a publish batch reporting at least one fenced result triggers exactly one `disconnect → connect → re-send same batch` cycle. Transactional producers re-run `initTransactions` as part of the reconnect. If the second send still reports any fenced record, the publisher gives up — silently retrying again would mask a real misconfiguration. Concurrent fenced publishes share a single in-flight reconnect so the producer is not torn down twice mid-restart.
+
+  New `KafkaPublisherHooks.onProducerFenced(error)` hook fires regardless of the recovery flag — informational signal so dashboards can track fence rates whether or not the publisher attempts recovery.
+
+  `@eventferry/core` minor: `PublishErrorKind` union gains `"fenced"`. The relay treats unknown / `"retriable"` / `"fenced"` identically (retry per backoff, DLQ on `attempts > maxAttempts`) — no relay-level changes required, but the new kind shows up in logs and the `errorKind` field of `PublishResult`.
+
+  Multi-instance EOS guidance: leave `autoRecoverFromFence` OFF and use a callable `transactionalId` that derives a stable, unique id per instance (pod name + replica index). Cross-instance fence is the broker telling the loser instance to stop — recovering silently creates a thrashing leadership flip. The README now spells this out in a `Producer-fenced restart` section.
+
+- 08d3384: `publisher.healthCheck({ timeoutMs })` — cheap reachability probe usable as the body of `/healthz` or `/readyz`. Borrows a fresh admin client, calls `listTopics`, and returns a stable `HealthStatus` shape: `{ ok, latencyMs, timestamp, error? }`. Default timeout 5000 ms (long enough to ride out a single broker leader election, short enough to fail a liveness probe meaningfully); `timeoutMs: 0` disables the timer entirely.
+
+  What it proves: the broker is reachable AND the configured credentials still authenticate. What it does NOT prove: the producer's send path is fully operational — a fenced transactional producer would still answer healthy here. Documented as "broker reachable + auth still good", not "publisher fully operational".
+
+  The borrowed admin is always closed (success, failure, timeout — try/finally). Admin-side close failures are swallowed; health checks aren't the place to crash. Custom drivers without an `admin()` method return `{ ok: false, error: ... }` instead of the throw `publisher.admin()` would surface.
+
+- 90b69c6: librdkafka stats hook on the confluent driver. New `onStats: (stats) => void` callback receives the librdkafka periodic statistics JSON, already parsed to a plain object — pipe queue depth, broker latencies, txmsgs counters, per-topic/per-partition stats into your metrics stack without a second client. The wrapper swallows callback exceptions and JSON parse failures so a misbehaving observer cannot take down the producer's event loop. `statsIntervalMs` controls the polling interval; defaults to 30000 ms when `onStats` is set, stays OFF otherwise (librdkafka CPU-bills the JSON serialization every tick — we don't enable it silently). `rawProducerConfig` still wins on precedence. kafkajs driver warns once and ignores both options — kafkajs has no equivalent surface.
+
+### Patch Changes
+
+- 715523f: Consumer-side documentation. No API change. The root README gains:
+
+  - **`Consuming what eventferry produced`** — canonical loop showing `decode(message)` → `extractTraceContext(headers)` → `defineOutbox(registry).decode(topic, bytes)`. Same registry the producer used, in reverse, returns the typed validated payload.
+  - **`Consuming the DLQ`** — copy-paste handler that routes by `dlq-error-class` (cleaner than parsing `dlq-reason`), pulls `dlq-attempts` for retry-queue accounting, and shows the alert-vs-retry split.
+
+  The `@eventferry/kafka` README adds matching subsections under the existing `Consumer helpers` block: **`Typed payload via the producer-side registry`** and **`DLQ recipe`**.
+
+  `defineOutbox(registry).decode()` was already shipped — the round just makes the symmetric "same registry, both sides" pattern discoverable.
+
+- ba81a78: Hardened TLS configuration documentation. No API change — `ssl.ca`, `ssl.servername`, and the rest of `TlsConfig` were already on the surface. This round:
+
+  - Expanded the `TlsConfig` JSDoc with the driver-parity gap: `servername` is honored by the **kafkajs** driver (Node `tls.connect` reads it directly) but is a documented **no-op on the confluent driver** — librdkafka v1.x's kafkaJS-compat layer doesn't expose an SNI override.
+  - README gained explicit "Dev cluster with a self-signed cert" and "IP-literal brokers (cert hostname mismatch)" sections with copy-paste examples covering CA pinning + `servername` for SNI/SAN alignment.
+  - Reaffirmed that `rejectUnauthorized: false` is **never** going to ship on this surface. TLS verification is non-negotiable. For dev clusters with self-signed certs, the supported pattern is to pass the cluster CA via `ssl.ca` so verification still happens — just against your CA instead of the system trust store.
+
+  Companion library updates (changesets, dependabot) on the way; this patch only touches comments + README, so the change is safe to consume immediately.
+
+- Updated dependencies [715523f]
+- Updated dependencies [fb0549d]
+  - @eventferry/core@3.4.0
+
+## 3.4.0
+
+### Minor Changes
+
+- 3e0c5ee: Add typed admin surface to `KafkaPublisher`: `publisher.admin()` borrows a connected admin client (caller closes it), `publisher.ensureTopics()` idempotently provisions topics with an optional `growPartitions` flag, and a new `validateTopicsOnConnect` option fails fast at startup when expected topics are missing. Implemented on both the kafkajs and confluent drivers; custom drivers that don't implement the optional `admin()` method get a clear error message instead of a silent surprise.
+- ac7a964: Add consumer-side helpers via the new `@eventferry/kafka/consume` subpath import: `decode(message, { decoder })` normalizes the raw message shape (key, value, headers, offset, timestamp, partition) both kafkajs and confluent deliver — with built-in `json` / `utf8` / `none` decoders plus a custom-function escape hatch; `extractTraceContext(headers)` parses the W3C `traceparent` / `tracestate` headers (strict validation per the W3C Trace Context spec) and accepts both raw (Buffer) and decoded (string) header shapes. Paired on the producer side with a new optional `KafkaTracer.inject(span, headers)` hook so OpenTelemetry users can complete the publish→consume trace propagation in two lines. The publisher clones each message before invoking `inject` — the caller's `PublishableMessage` references are never mutated, keeping the relay's retry path safe.
+- 4007a8e: Power-user escape hatches for both drivers. The high-level options cover ~95% of cases; these let you reach into the native client when you need a knob we don't expose typed.
+
+  - `compressionLevel`: per-codec level (confluent only, e.g. `zstd` level 1-22). Maps to librdkafka's `compression.level`. The kafkajs driver warns once and ignores it (kafkajs has no codec-level config).
+  - `rawProducerConfig`: raw librdkafka keys merged into the confluent producer config. Native keys **win** against eventferry's translated ones — use this to override defaults or to tune surface area (queue buffering, statistics interval, socket keepalive, …) we don't expose.
+  - `rawKafkaJsProducerConfig`: same idea for kafkajs — raw keys merged into `kafka.producer({...})` with last-write-wins precedence.
+  - `customPartitioner`: kafkajs partitioner factory (`() => (args) => number`). Overrides the `partitioner` preset entirely. Confluent ignores it — librdkafka's partitioner is a C-level extension point.
+
+  Native config takes precedence over eventferry's translated keys in every case — that's the contract of an escape hatch.
+
 ## 3.3.1
 
 ### Patch Changes

package/README.md

@@ -67,6 +67,40 @@ new KafkaPublisher({
 > non-negotiable. For dev clusters with self-signed certs, pass the cluster
 > CA via `ca` so verification succeeds.
 
+### Dev cluster with a self-signed cert
+
+The right pattern is to pin **your** CA. Verification still happens — just against your CA instead of the system trust store.
+
+```ts
+new KafkaPublisher({
+  brokers: ["dev-broker.internal:9093"],
+  ssl: {
+    ca: readFileSync("/path/to/dev-cluster-ca.pem"),
+    // Cluster reachable via DNS that doesn't match the cert SAN?
+    // Pin the SNI host the cert was issued for:
+    servername: "kafka.dev.internal",
+  },
+});
+```
+
+**Never** add `rejectUnauthorized: false` (TS would reject it anyway — it's not in the type). That disables verification entirely and opens every connection to a man-in-the-middle.
+
+### IP-literal brokers (cert hostname mismatch)
+
+When the broker address is an IP and the cert was issued for a hostname, set `servername`:
+
+```ts
+new KafkaPublisher({
+  brokers: ["10.0.5.12:9093"],          // IP literal
+  ssl: {
+    ca: readFileSync("/etc/ssl/kafka-ca.pem"),
+    servername: "broker.example.com",   // hostname the cert was issued for
+  },
+});
+```
+
+`servername` is honored by the **kafkajs** driver (Node `tls.connect` reads `servername` directly). It's a **documented no-op on the confluent driver** — librdkafka v1.x's kafkaJS-compat layer doesn't expose an SNI override, and SNI is derived from the broker address. Use the kafkajs driver when you need the SNI lever.
+
 ### SASL — username + password (PLAIN / SCRAM)
 
 ```ts
@@ -271,6 +305,190 @@ Per the spec, eventferry emits **one span per `publish()` call**, named `"{topic
 
 The user-supplied tracer SHOULD set `SpanKind.PRODUCER` on the span; the adapter above does this explicitly.
 
+#### Propagating trace context to consumers
+
+Add an optional `inject` method on the tracer to write the W3C `traceparent` / `tracestate` headers into each outgoing message. Pair this with `extractTraceContext` on the consumer side (see [Consumer helpers](#consumer-helpers--eventferrykafkaconsume)).
+
+```ts
+import { context as otelContext, propagation, trace } from "@opentelemetry/api";
+
+const tracer: KafkaTracer = {
+  startPublishSpan: /* …as above… */,
+  inject(_span, headers) {
+    // The publisher wraps the active span context for us before calling this.
+    propagation.inject(otelContext.active(), headers);
+  },
+};
+```
+
+The publisher clones each outbound message before injecting (the caller's `PublishableMessage` is never mutated, so the relay's retry path stays correct).
+
+## Health check
+
+Cheap reachability probe — useful as the body of a `/healthz` or `/readyz` endpoint:
+
+```ts
+import express from "express";
+const app = express();
+
+app.get("/healthz", async (_req, res) => {
+  const status = await publisher.healthCheck({ timeoutMs: 3_000 });
+  res.status(status.ok ? 200 : 503).json({
+    ok: status.ok,
+    latencyMs: status.latencyMs,
+    error: status.error?.message,
+  });
+});
+```
+
+`publisher.healthCheck()` opens a fresh admin, calls `listTopics`, and returns:
+
+```ts
+interface HealthStatus {
+  ok: boolean;          // broker answered within timeout
+  latencyMs: number;    // probe wall-clock
+  timestamp: number;    // epoch ms when the probe started
+  error?: Error;        // present when ok === false
+}
+```
+
+Default `timeoutMs: 5_000` — long enough to ride out a single broker leader election, short enough to fail a liveness probe meaningfully. Set `timeoutMs: 0` to disable the timer.
+
+**What this proves**: the broker is reachable AND the configured credentials still authenticate. **What this does NOT prove**: the producer's send path is fully operational — a fenced transactional producer would still answer healthy here. Treat the result as "broker reachable + auth still good", not "publisher fully operational".
+
+The borrowed admin is always closed (success or failure). Admin-side close failures don't change the outcome — health checks aren't the place to crash.
+
+## Producer-fenced restart
+
+`PRODUCER_FENCED` and `INVALID_PRODUCER_EPOCH` errors classify as `errorKind: "fenced"` — a distinct kind from `fatal` because some fences are **transient** (broker restart, network partition recovery) rather than a permanent multi-instance collision.
+
+### `autoRecoverFromFence: true`
+
+Opt in to a single transparent reconnect-and-retry when a publish batch reports a fence:
+
+```ts
+new KafkaPublisher({
+  brokers,
+  transactional: true,
+  transactionalId: "orders-publisher",
+  autoRecoverFromFence: true,
+});
+```
+
+What happens on a fenced batch:
+
+1. The `onProducerFenced(error)` hook fires (regardless of the recovery flag — informational).
+2. The driver is disconnected and reconnected (re-running `initTransactions` for transactional producers).
+3. The same batch is resent **once**.
+4. If the second send still reports any fenced record, the publisher gives up and surfaces those failures unchanged — silently retrying again would mask a misconfiguration.
+
+Concurrent fenced publishes share a single in-flight reconnect — the producer is not torn down twice while a recovery is in progress.
+
+**Default is `false`** to preserve the previous "fenced → propagate to relay" behavior. The relay will retry fenced records under the configured backoff and DLQ them when `attempts > retry.maxAttempts`.
+
+### `transactional.id` strategy for multi-instance EOS
+
+When running multiple producer instances against the same logical workload, each instance MUST have a stable, unique `transactionalId`. Use the callable form to derive it from runtime context:
+
+```ts
+new KafkaPublisher({
+  brokers,
+  transactional: true,
+  transactionalId: () => `${process.env.POD_NAME}-${process.env.HOSTNAME}`,
+  // Leave autoRecoverFromFence OFF — a fence means a real collision
+  // worth surfacing.
+});
+```
+
+Cross-instance fence is **not** a transient blip — it's the broker telling one of you that the other is now the canonical producer. Auto-recovery would create a thrashing leadership flip. Keep the option off in multi-instance setups and let the loser instance fail loudly.
+
+## librdkafka stats hook
+
+The confluent driver exposes librdkafka's periodic statistics stream as a typed callback. Useful for piping queue depth, broker latency, broker timeout counts, and per-topic/per-partition counters into your metrics stack.
+
+```ts
+new KafkaPublisher({
+  brokers,
+  driver: "confluent",
+  onStats: (stats) => {
+    // stats is opaque librdkafka JSON. Reach for the fields you care about.
+    promClient.gauge("kafka_msg_cnt").set(stats.msg_cnt as number);
+    promClient.gauge("kafka_txmsgs").set(stats.txmsgs as number);
+  },
+  statsIntervalMs: 30_000, // optional; defaults to 30s when onStats is set
+});
+```
+
+- **`onStats`** receives the librdkafka stats JSON, already parsed to a plain object. The schema is opaque (`Record<string, unknown>`) — librdkafka's stats are huge and evolve across versions. Reference: [librdkafka STATISTICS.md](https://github.com/confluentinc/librdkafka/blob/master/STATISTICS.md).
+- **`statsIntervalMs`** maps to librdkafka's `statistics.interval.ms`. **Defaults to 30000 ms when `onStats` is set; otherwise stays off** (librdkafka CPU-bills the JSON serialization every tick — we don't enable it silently).
+- The wrapper swallows callback exceptions and JSON parse failures — a single dropped sample is preferable to taking down the producer's event loop.
+- **No-op on the kafkajs driver** — kafkajs has no equivalent surface. Logs a one-time warning and ignores both options.
+
+## Power-user escape hatches
+
+When the high-level options don't reach a knob you need, drop down to the native client config.
+
+### Compression level
+
+```ts
+new KafkaPublisher({
+  brokers,
+  driver: "confluent",
+  compression: "zstd",
+  compressionLevel: 9, // librdkafka compression.level
+});
+```
+
+Confluent only. The kafkajs driver logs a one-time warning and ignores it (kafkajs does not expose codec levels). Default level is the codec's broker-friendly default.
+
+### Raw librdkafka producer config (confluent driver)
+
+```ts
+new KafkaPublisher({
+  brokers,
+  driver: "confluent",
+  rawProducerConfig: {
+    "queue.buffering.max.messages": 100_000,
+    "statistics.interval.ms": 5_000,
+    "socket.keepalive.enable": true,
+  },
+});
+```
+
+Merged on TOP of eventferry's translated config — raw keys **win** against the translated ones. Use this to override defaults (set `linger.ms` directly) or to tune surface area we don't expose (`queue.buffering.max.kbytes`, etc.).
+
+### Raw kafkajs producer config (kafkajs driver)
+
+```ts
+new KafkaPublisher({
+  brokers,
+  driver: "kafkajs",
+  rawKafkaJsProducerConfig: {
+    retry: { retries: 7, initialRetryTime: 250 },
+    metadataMaxAge: 5_000,
+  },
+});
+```
+
+Same precedence — raw keys win. Use for kafkajs-internal knobs (`retry`, `metadataMaxAge`) or to override defaults like `idempotent`.
+
+### Custom partitioner (kafkajs driver)
+
+```ts
+const tenantAwarePartitioner = () => ({ topic, partitionMetadata, message }) => {
+  const tenant = message.headers["x-tenant"]?.toString();
+  return hashToPartition(tenant, partitionMetadata.length);
+};
+
+new KafkaPublisher({
+  brokers,
+  driver: "kafkajs",
+  customPartitioner: tenantAwarePartitioner,
+});
+```
+
+Overrides the `partitioner` preset. Confluent ignores this — librdkafka's partitioner is a C-level extension point, not a JS callback.
+
 ### Logger
 
 Pass a `Logger` (the same interface used by `@eventferry/core`) to route the publisher's own diagnostics — driver warnings, hook failures — through your logging stack:
@@ -284,6 +502,151 @@ new KafkaPublisher({
 
 When omitted, the publisher is silent and the driver falls back to `console.warn` for its diagnostics (preserves prior behavior).
 
+## Admin operations
+
+The publisher exposes a typed admin surface for listing/describing/creating topics — handy for provisioning in CI, integration tests, or app boot.
+
+### `publisher.admin()`
+
+Borrow a fresh admin client. The returned client is connected and ready; the **caller** is responsible for closing it.
+
+```ts
+const admin = await publisher.admin();
+try {
+  const topics = await admin.listTopics();
+  const desc = await admin.describeTopics(["orders"]);
+  console.log(desc[0].partitions.length); // partition count
+} finally {
+  await admin.close();
+}
+```
+
+Methods on the returned `KafkaAdmin`:
+
+- `listTopics(): Promise<string[]>` — all topic names visible to this principal.
+- `describeTopics(topics): Promise<TopicMetadata[]>` — partition / leader / ISR per topic. Missing topics come back with an empty `partitions` array (no try/catch needed to detect absence).
+- `createTopics(specs)` — idempotent: existing topics are silently skipped.
+- `createPartitions(specs)` — grow a topic's partition count (Kafka does not support shrinking).
+- `close()` — disconnect.
+
+### `publisher.ensureTopics()`
+
+One-shot, idempotent provisioning built on top of the admin surface:
+
+```ts
+await publisher.ensureTopics([
+  { topic: "orders", numPartitions: 12, replicationFactor: 3 },
+  { topic: "orders.dlq", numPartitions: 3, replicationFactor: 3, configEntries: { "retention.ms": "604800000" } },
+]);
+
+// Optionally grow existing topics whose partition count is below the requested numPartitions:
+await publisher.ensureTopics(
+  [{ topic: "orders", numPartitions: 24 }],
+  { growPartitions: true },
+);
+```
+
+What it does:
+
+- Creates topics that don't exist.
+- Skips topics that already exist (no error, no surprise alter).
+- With `growPartitions: true`, calls `createPartitions` for existing topics whose current partition count is **below** the requested `numPartitions`.
+
+What it does NOT do (by design):
+
+- Reconcile replication factor on existing topics — Kafka has no safe in-place alter (use partition reassignment for that).
+- Reconcile `configEntries` on existing topics — use `kafka-configs.sh` or the raw admin client (kafkajs's `alterConfigs`) if you need that.
+
+### Consumer helpers — `@eventferry/kafka/consume`
+
+eventferry is publisher-only, but the records it produces are consumed somewhere downstream. The `consume` subpath ships zero-dep helpers for the typical decode + trace-continuation glue, and pulls in no kafkajs/confluent code:
+
+```ts
+import { decode, extractTraceContext } from "@eventferry/kafka/consume";
+
+await consumer.run({
+  eachMessage: async ({ message }) => {
+    // Normalize key/headers/value; default decoder is JSON.
+    const { key, value, headers, offset } = decode<{ orderId: string }>(message);
+
+    // Continue the producer's W3C trace context (if the publisher's tracer
+    // injects it — see "OpenTelemetry tracing" above for the inject hook).
+    const trace = extractTraceContext(message.headers);
+    if (trace) {
+      // → start a CONSUMER span as a child of trace.traceId / trace.spanId
+    }
+
+    await handle(value!);
+  },
+});
+```
+
+`decode` options:
+
+- `decoder: "json"` (default) — `JSON.parse(value.toString("utf8"))`. Empty/null value → `null` (handles compaction tombstones).
+- `decoder: "utf8"` — raw text.
+- `decoder: "none"` — raw `Buffer`.
+- `decoder: (bytes) => …` — custom (Avro, Protobuf, MessagePack, …).
+
+`extractTraceContext` returns `null` if no `traceparent` header is present or it fails W3C validation (all-zero IDs, `version: ff`, malformed hex). It accepts both raw consumer headers (Buffer values) and already-decoded headers (string values).
+
+#### Typed payload via the producer-side registry
+
+When your consumer lives in the same monorepo as the producer, hand the decoded bytes to the **same `defineOutbox(registry)`** you used to enqueue. `decode` validates against the topic's Standard Schema and returns the typed payload:
+
+```ts
+import { defineOutbox } from "@eventferry/core";
+import { decode } from "@eventferry/kafka/consume";
+import { registry } from "./outbox-registry";
+
+const events = defineOutbox(registry); // no store — consumer side
+
+await consumer.run({
+  eachMessage: async ({ message }) => {
+    const m = decode(message, { decoder: "utf8" });
+    const event = await events.decode("orders.created", m.value!);
+    //    ^? { orderId: string; total: number }
+    await handle(event);
+  },
+});
+```
+
+`events.decode(topic, bytes)` throws `OutboxValidationError` if the topic isn't in the registry or the payload doesn't match the schema. Cross-language consumers (Go, Java, Python) skip the companion and use their own schema tooling — Confluent Schema Registry for typed wire formats.
+
+#### DLQ recipe
+
+Records that exhaust retries land on `${topic}.dlq` (or your configured DLQ topic) carrying enriched headers `dlq-reason`, `dlq-error-class`, `dlq-original-topic`, `dlq-failed-at`, `dlq-attempts` (and optionally `dlq-stack` when you opt in). Route them with `dlq-error-class` rather than parsing `dlq-reason`:
+
+```ts
+await dlqConsumer.run({
+  eachMessage: async ({ message }) => {
+    const m = decode(message);
+    const errClass = m.headers["dlq-error-class"];
+    if (errClass === "KafkaJSProtocolError" && m.headers["dlq-reason"]?.includes("MESSAGE_TOO_LARGE")) {
+      await ticket.create({ title: `Oversized DLQ from ${m.headers["dlq-original-topic"]}` });
+    } else {
+      await retryQueue.put({
+        payload: m.value,
+        attemptsSoFar: Number(m.headers["dlq-attempts"] ?? "0"),
+      });
+    }
+  },
+});
+```
+
+### `validateTopicsOnConnect`
+
+Fail-fast at startup if expected topics are missing:
+
+```ts
+new KafkaPublisher({
+  brokers,
+  validateTopicsOnConnect: ["orders", "orders.dlq", "events"],
+});
+```
+
+`connect()` opens an admin, runs `listTopics`, and throws a single descriptive error naming **every** missing topic. The admin is always closed (success or failure). Skip the check entirely with an empty list or by omitting the option.
+
 📖 **Full documentation:** [github.com/SametGoktepe/eventferry](https://github.com/SametGoktepe/eventferry#readme)
 
 ## License

package/dist/consume.cjs

@@ -0,0 +1,115 @@
+"use strict";
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+
+// src/consume.ts
+var consume_exports = {};
+__export(consume_exports, {
+  decode: () => decode,
+  decodeHeaders: () => decodeHeaders,
+  extractTraceContext: () => extractTraceContext
+});
+module.exports = __toCommonJS(consume_exports);
+function decodeHeaders(raw) {
+  if (!raw) return {};
+  const out = {};
+  for (const [k, v] of Object.entries(raw)) {
+    if (v === void 0 || v === null) continue;
+    out[k] = Buffer.isBuffer(v) ? v.toString("utf8") : v;
+  }
+  return out;
+}
+function decode(msg, opts = {}) {
+  const headers = decodeHeaders(msg.headers);
+  const key = normalizeKey(msg.key);
+  const value = decodeValue(msg.value, opts.decoder ?? "json");
+  const timestamp = msg.timestamp !== void 0 ? Number(msg.timestamp) : void 0;
+  const offset = msg.offset !== void 0 ? String(msg.offset) : void 0;
+  return {
+    key,
+    value,
+    headers,
+    timestamp,
+    offset,
+    partition: msg.partition
+  };
+}
+function normalizeKey(key) {
+  if (key === null || key === void 0) return null;
+  return Buffer.isBuffer(key) ? key.toString("utf8") : key;
+}
+function decodeValue(value, decoder) {
+  if (value === null || value === void 0) return null;
+  const buf = Buffer.isBuffer(value) ? value : Buffer.from(value);
+  if (buf.length === 0) return null;
+  if (typeof decoder === "function") return decoder(buf);
+  switch (decoder) {
+    case "utf8":
+      return buf.toString("utf8");
+    case "none":
+      return buf;
+    case "json":
+    case void 0:
+    default: {
+      const text = buf.toString("utf8");
+      try {
+        return JSON.parse(text);
+      } catch (err) {
+        throw new Error(
+          `decode: JSON.parse failed on message value: ${err.message}`
+        );
+      }
+    }
+  }
+}
+var TRACEPARENT_RE = /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;
+var INVALID_TRACE_ID = "0".repeat(32);
+var INVALID_SPAN_ID = "0".repeat(16);
+function extractTraceContext(headers) {
+  if (!headers) return null;
+  const tp = readHeader(headers, "traceparent");
+  if (!tp) return null;
+  const match = TRACEPARENT_RE.exec(tp);
+  if (!match) return null;
+  const [, version, traceId, spanId, flags] = match;
+  if (version === "ff") return null;
+  if (traceId === INVALID_TRACE_ID || spanId === INVALID_SPAN_ID) return null;
+  const sampled = (parseInt(flags, 16) & 1) === 1;
+  const ts = readHeader(headers, "tracestate");
+  return {
+    traceparent: tp,
+    tracestate: ts && ts.length > 0 ? ts : void 0,
+    traceId,
+    spanId,
+    sampled
+  };
+}
+function readHeader(headers, name) {
+  const v = headers[name];
+  if (v === void 0 || v === null) return void 0;
+  if (typeof v === "string") return v;
+  if (Buffer.isBuffer(v)) return v.toString("utf8");
+  return void 0;
+}
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  decode,
+  decodeHeaders,
+  extractTraceContext
+});
+//# sourceMappingURL=consume.cjs.map

package/dist/consume.cjs.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/consume.ts"],"sourcesContent":["/**\n * Consumer-side helpers — paired with the publisher's outbound surface.\n *\n * eventferry is a publisher-only library, but the messages it produces are\n * consumed somewhere downstream. This module normalizes the message shape\n * that both `kafkajs` and `@confluentinc/kafka-javascript` deliver to\n * consumer callbacks, decodes the payload, and extracts the W3C trace\n * context the publisher injected (see {@link KafkaTracer.inject}).\n *\n * Imported via subpath so consumer code paths don't pull in the producer:\n *\n *   import { decode, extractTraceContext } from \"@eventferry/kafka/consume\";\n *\n * There is intentionally NO Kafka client here — bring your own consumer\n * (kafkajs's `Consumer`, librdkafka's, whatever) and call `decode()` /\n * `extractTraceContext()` on the message you receive.\n */\n\n/**\n * Raw incoming Kafka message — structural subset both kafkajs and confluent\n * (via the kafkaJS-compat layer) deliver. Fields are optional because\n * different consumer APIs surface different subsets.\n */\nexport interface IncomingKafkaMessage {\n  key?: Buffer | string | null;\n  value?: Buffer | string | null;\n  headers?: IncomingHeaders;\n  /** ISO ms string or numeric epoch ms — depends on the client. */\n  timestamp?: string | number;\n  /** Numeric or string per client. */\n  offset?: string | number;\n  partition?: number;\n}\n\n/** Headers as the underlying clients deliver them: bytes, strings, or undefined. */\nexport type IncomingHeaders = Record<string, Buffer | string | undefined>;\n\n/** Headers normalized to UTF-8 strings (the form most application code wants). */\nexport type DecodedHeaders = Record<string, string>;\n\n/** Payload decoder. Buffer in, decoded value out. */\nexport type Decoder<T> = (bytes: Buffer) => T;\n\n/** Decoded message wrapper — value plus normalized headers, key, metadata. */\nexport interface DecodedMessage<V = unknown> {\n  key: string | null;\n  value: V | null;\n  headers: DecodedHeaders;\n  /** Epoch ms when the broker stamped the record. */\n  timestamp?: number;\n  /** Stringified offset (Kafka offsets exceed 2^53 — strings stay safe). */\n  offset?: string;\n  partition?: number;\n}\n\nexport interface DecodeOptions<V> {\n  /**\n   * Decoder for the payload bytes. Built-ins:\n   *\n   * - `\"json\"` (default) — `JSON.parse(value.toString(\"utf8\"))`. Empty\n   *   value returns `null` (matches Kafka tombstones on compacted topics).\n   * - `\"utf8\"` — raw text. Returns the string as-is.\n   * - `\"none\"` — returns the raw `Buffer` unchanged.\n   *\n   * Or pass your own `(bytes: Buffer) => V` for Avro / Protobuf / MessagePack.\n   */\n  decoder?: \"json\" | \"utf8\" | \"none\" | Decoder<V>;\n}\n\n/**\n * Normalize headers to a plain string→string map. Buffers are read as UTF-8;\n * `undefined` entries are dropped (consumers occasionally surface absent\n * headers as `undefined` values).\n */\nexport function decodeHeaders(raw?: IncomingHeaders): DecodedHeaders {\n  if (!raw) return {};\n  const out: DecodedHeaders = {};\n  for (const [k, v] of Object.entries(raw)) {\n    if (v === undefined || v === null) continue;\n    out[k] = Buffer.isBuffer(v) ? v.toString(\"utf8\") : v;\n  }\n  return out;\n}\n\n/**\n * Decode a Kafka message: normalize the key + headers, decode the value\n * with the chosen decoder, and surface the broker metadata.\n *\n * Tombstones (null/empty value) come back with `value: null` regardless of\n * the decoder — compaction-friendly.\n *\n * @throws when `decoder: \"json\"` (the default) and the payload is non-empty\n *   but not valid JSON. Catch the error and decide whether to DLQ the\n *   record or skip it — eventferry does not assume.\n */\nexport function decode<V = unknown>(\n  msg: IncomingKafkaMessage,\n  opts: DecodeOptions<V> = {},\n): DecodedMessage<V> {\n  const headers = decodeHeaders(msg.headers);\n  const key = normalizeKey(msg.key);\n  const value = decodeValue<V>(msg.value, opts.decoder ?? \"json\");\n  const timestamp =\n    msg.timestamp !== undefined ? Number(msg.timestamp) : undefined;\n  const offset = msg.offset !== undefined ? String(msg.offset) : undefined;\n  return {\n    key,\n    value,\n    headers,\n    timestamp,\n    offset,\n    partition: msg.partition,\n  };\n}\n\nfunction normalizeKey(key?: Buffer | string | null): string | null {\n  if (key === null || key === undefined) return null;\n  return Buffer.isBuffer(key) ? key.toString(\"utf8\") : key;\n}\n\nfunction decodeValue<V>(\n  value: Buffer | string | null | undefined,\n  decoder: DecodeOptions<V>[\"decoder\"],\n): V | null {\n  if (value === null || value === undefined) return null;\n  const buf = Buffer.isBuffer(value) ? value : Buffer.from(value);\n  // Tombstones: an empty buffer is a kafka \"delete me\" on compacted\n  // topics. Surface as null for every decoder — applications usually\n  // want the same null-handling for both.\n  if (buf.length === 0) return null;\n  if (typeof decoder === \"function\") return decoder(buf);\n  switch (decoder) {\n    case \"utf8\":\n      return buf.toString(\"utf8\") as unknown as V;\n    case \"none\":\n      return buf as unknown as V;\n    case \"json\":\n    case undefined:\n    default: {\n      const text = buf.toString(\"utf8\");\n      try {\n        return JSON.parse(text) as V;\n      } catch (err) {\n        throw new Error(\n          `decode: JSON.parse failed on message value: ${(err as Error).message}`,\n        );\n      }\n    }\n  }\n}\n\n/**\n * W3C Trace Context extracted from message headers.\n *\n * - `traceparent`: full header value, format `version-traceId-spanId-flags`.\n * - `tracestate`: optional vendor-specific state (W3C `tracestate` header).\n * - `traceId`: 32 hex chars, parsed from `traceparent`.\n * - `spanId`: 16 hex chars (the PARENT span id from the producer).\n * - `sampled`: parsed from the `traceparent` flags (bit 0 = sampled).\n *\n * Returns `null` when no `traceparent` header is present or the value\n * fails W3C validation.\n *\n * Spec: https://www.w3.org/TR/trace-context/\n */\nexport interface TraceContext {\n  traceparent: string;\n  tracestate?: string;\n  traceId: string;\n  spanId: string;\n  sampled: boolean;\n}\n\nconst TRACEPARENT_RE =\n  /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;\nconst INVALID_TRACE_ID = \"0\".repeat(32);\nconst INVALID_SPAN_ID = \"0\".repeat(16);\n\n/**\n * Extract the W3C trace context the publisher injected into headers.\n * Headers may be raw (Buffer values) or already-decoded (string values) —\n * both shapes work, so you can call this before OR after `decode()`.\n *\n * Validation follows the W3C spec strictly: invalid all-zero trace/span\n * IDs are rejected, version `ff` is rejected, malformed hex is rejected.\n * On any of these, the function returns `null` rather than throwing —\n * consumer code should fall back to starting a fresh trace.\n */\nexport function extractTraceContext(\n  headers: IncomingHeaders | DecodedHeaders | undefined,\n): TraceContext | null {\n  if (!headers) return null;\n  const tp = readHeader(headers, \"traceparent\");\n  if (!tp) return null;\n  const match = TRACEPARENT_RE.exec(tp);\n  if (!match) return null;\n  const [, version, traceId, spanId, flags] = match as unknown as [\n    string,\n    string,\n    string,\n    string,\n    string,\n  ];\n  // Spec §3.2.2.5: version \"ff\" is forbidden (reserved sentinel).\n  if (version === \"ff\") return null;\n  if (traceId === INVALID_TRACE_ID || spanId === INVALID_SPAN_ID) return null;\n  const sampled = (parseInt(flags, 16) & 0x01) === 1;\n  const ts = readHeader(headers, \"tracestate\");\n  return {\n    traceparent: tp,\n    tracestate: ts && ts.length > 0 ? ts : undefined,\n    traceId,\n    spanId,\n    sampled,\n  };\n}\n\nfunction readHeader(\n  headers: IncomingHeaders | DecodedHeaders,\n  name: string,\n): string | undefined {\n  const v = (headers as Record<string, Buffer | string | undefined>)[name];\n  if (v === undefined || v === null) return undefined;\n  if (typeof v === \"string\") return v;\n  if (Buffer.isBuffer(v)) return v.toString(\"utf8\");\n  return undefined;\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AA0EO,SAAS,cAAc,KAAuC;AACnE,MAAI,CAAC,IAAK,QAAO,CAAC;AAClB,QAAM,MAAsB,CAAC;AAC7B,aAAW,CAAC,GAAG,CAAC,KAAK,OAAO,QAAQ,GAAG,GAAG;AACxC,QAAI,MAAM,UAAa,MAAM,KAAM;AACnC,QAAI,CAAC,IAAI,OAAO,SAAS,CAAC,IAAI,EAAE,SAAS,MAAM,IAAI;AAAA,EACrD;AACA,SAAO;AACT;AAaO,SAAS,OACd,KACA,OAAyB,CAAC,GACP;AACnB,QAAM,UAAU,cAAc,IAAI,OAAO;AACzC,QAAM,MAAM,aAAa,IAAI,GAAG;AAChC,QAAM,QAAQ,YAAe,IAAI,OAAO,KAAK,WAAW,MAAM;AAC9D,QAAM,YACJ,IAAI,cAAc,SAAY,OAAO,IAAI,SAAS,IAAI;AACxD,QAAM,SAAS,IAAI,WAAW,SAAY,OAAO,IAAI,MAAM,IAAI;AAC/D,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA,WAAW,IAAI;AAAA,EACjB;AACF;AAEA,SAAS,aAAa,KAA6C;AACjE,MAAI,QAAQ,QAAQ,QAAQ,OAAW,QAAO;AAC9C,SAAO,OAAO,SAAS,GAAG,IAAI,IAAI,SAAS,MAAM,IAAI;AACvD;AAEA,SAAS,YACP,OACA,SACU;AACV,MAAI,UAAU,QAAQ,UAAU,OAAW,QAAO;AAClD,QAAM,MAAM,OAAO,SAAS,KAAK,IAAI,QAAQ,OAAO,KAAK,KAAK;AAI9D,MAAI,IAAI,WAAW,EAAG,QAAO;AAC7B,MAAI,OAAO,YAAY,WAAY,QAAO,QAAQ,GAAG;AACrD,UAAQ,SAAS;AAAA,IACf,KAAK;AACH,aAAO,IAAI,SAAS,MAAM;AAAA,IAC5B,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AAAA,IACL,KAAK;AAAA,IACL,SAAS;AACP,YAAM,OAAO,IAAI,SAAS,MAAM;AAChC,UAAI;AACF,eAAO,KAAK,MAAM,IAAI;AAAA,MACxB,SAAS,KAAK;AACZ,cAAM,IAAI;AAAA,UACR,+CAAgD,IAAc,OAAO;AAAA,QACvE;AAAA,MACF;AAAA,IACF;AAAA,EACF;AACF;AAwBA,IAAM,iBACJ;AACF,IAAM,mBAAmB,IAAI,OAAO,EAAE;AACtC,IAAM,kBAAkB,IAAI,OAAO,EAAE;AAY9B,SAAS,oBACd,SACqB;AACrB,MAAI,CAAC,QAAS,QAAO;AACrB,QAAM,KAAK,WAAW,SAAS,aAAa;AAC5C,MAAI,CAAC,GAAI,QAAO;AAChB,QAAM,QAAQ,eAAe,KAAK,EAAE;AACpC,MAAI,CAAC,MAAO,QAAO;AACnB,QAAM,CAAC,EAAE,SAAS,SAAS,QAAQ,KAAK,IAAI;AAQ5C,MAAI,YAAY,KAAM,QAAO;AAC7B,MAAI,YAAY,oBAAoB,WAAW,gBAAiB,QAAO;AACvE,QAAM,WAAW,SAAS,OAAO,EAAE,IAAI,OAAU;AACjD,QAAM,KAAK,WAAW,SAAS,YAAY;AAC3C,SAAO;AAAA,IACL,aAAa;AAAA,IACb,YAAY,MAAM,GAAG,SAAS,IAAI,KAAK;AAAA,IACvC;AAAA,IACA;AAAA,IACA;AAAA,EACF;AACF;AAEA,SAAS,WACP,SACA,MACoB;AACpB,QAAM,IAAK,QAAwD,IAAI;AACvE,MAAI,MAAM,UAAa,MAAM,KAAM,QAAO;AAC1C,MAAI,OAAO,MAAM,SAAU,QAAO;AAClC,MAAI,OAAO,SAAS,CAAC,EAAG,QAAO,EAAE,SAAS,MAAM;AAChD,SAAO;AACT;","names":[]}