@eventferry/kafka 3.3.1 → 3.4.0

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # @eventferry/kafka
 
+## 3.4.0
+
+### Minor Changes
+
+- 3e0c5ee: Add typed admin surface to `KafkaPublisher`: `publisher.admin()` borrows a connected admin client (caller closes it), `publisher.ensureTopics()` idempotently provisions topics with an optional `growPartitions` flag, and a new `validateTopicsOnConnect` option fails fast at startup when expected topics are missing. Implemented on both the kafkajs and confluent drivers; custom drivers that don't implement the optional `admin()` method get a clear error message instead of a silent surprise.
+- ac7a964: Add consumer-side helpers via the new `@eventferry/kafka/consume` subpath import: `decode(message, { decoder })` normalizes the raw message shape (key, value, headers, offset, timestamp, partition) both kafkajs and confluent deliver — with built-in `json` / `utf8` / `none` decoders plus a custom-function escape hatch; `extractTraceContext(headers)` parses the W3C `traceparent` / `tracestate` headers (strict validation per the W3C Trace Context spec) and accepts both raw (Buffer) and decoded (string) header shapes. Paired on the producer side with a new optional `KafkaTracer.inject(span, headers)` hook so OpenTelemetry users can complete the publish→consume trace propagation in two lines. The publisher clones each message before invoking `inject` — the caller's `PublishableMessage` references are never mutated, keeping the relay's retry path safe.
+- 4007a8e: Power-user escape hatches for both drivers. The high-level options cover ~95% of cases; these let you reach into the native client when you need a knob we don't expose typed.
+
+  - `compressionLevel`: per-codec level (confluent only, e.g. `zstd` level 1-22). Maps to librdkafka's `compression.level`. The kafkajs driver warns once and ignores it (kafkajs has no codec-level config).
+  - `rawProducerConfig`: raw librdkafka keys merged into the confluent producer config. Native keys **win** against eventferry's translated ones — use this to override defaults or to tune surface area (queue buffering, statistics interval, socket keepalive, …) we don't expose.
+  - `rawKafkaJsProducerConfig`: same idea for kafkajs — raw keys merged into `kafka.producer({...})` with last-write-wins precedence.
+  - `customPartitioner`: kafkajs partitioner factory (`() => (args) => number`). Overrides the `partitioner` preset entirely. Confluent ignores it — librdkafka's partitioner is a C-level extension point.
+
+  Native config takes precedence over eventferry's translated keys in every case — that's the contract of an escape hatch.
+
 ## 3.3.1
 
 ### Patch Changes

package/README.md

@@ -271,6 +271,89 @@ Per the spec, eventferry emits **one span per `publish()` call**, named `"{topic
 
 The user-supplied tracer SHOULD set `SpanKind.PRODUCER` on the span; the adapter above does this explicitly.
 
+#### Propagating trace context to consumers
+
+Add an optional `inject` method on the tracer to write the W3C `traceparent` / `tracestate` headers into each outgoing message. Pair this with `extractTraceContext` on the consumer side (see [Consumer helpers](#consumer-helpers--eventferrykafkaconsume)).
+
+```ts
+import { context as otelContext, propagation, trace } from "@opentelemetry/api";
+
+const tracer: KafkaTracer = {
+  startPublishSpan: /* …as above… */,
+  inject(_span, headers) {
+    // The publisher wraps the active span context for us before calling this.
+    propagation.inject(otelContext.active(), headers);
+  },
+};
+```
+
+The publisher clones each outbound message before injecting (the caller's `PublishableMessage` is never mutated, so the relay's retry path stays correct).
+
+## Power-user escape hatches
+
+When the high-level options don't reach a knob you need, drop down to the native client config.
+
+### Compression level
+
+```ts
+new KafkaPublisher({
+  brokers,
+  driver: "confluent",
+  compression: "zstd",
+  compressionLevel: 9, // librdkafka compression.level
+});
+```
+
+Confluent only. The kafkajs driver logs a one-time warning and ignores it (kafkajs does not expose codec levels). Default level is the codec's broker-friendly default.
+
+### Raw librdkafka producer config (confluent driver)
+
+```ts
+new KafkaPublisher({
+  brokers,
+  driver: "confluent",
+  rawProducerConfig: {
+    "queue.buffering.max.messages": 100_000,
+    "statistics.interval.ms": 5_000,
+    "socket.keepalive.enable": true,
+  },
+});
+```
+
+Merged on TOP of eventferry's translated config — raw keys **win** against the translated ones. Use this to override defaults (set `linger.ms` directly) or to tune surface area we don't expose (`queue.buffering.max.kbytes`, etc.).
+
+### Raw kafkajs producer config (kafkajs driver)
+
+```ts
+new KafkaPublisher({
+  brokers,
+  driver: "kafkajs",
+  rawKafkaJsProducerConfig: {
+    retry: { retries: 7, initialRetryTime: 250 },
+    metadataMaxAge: 5_000,
+  },
+});
+```
+
+Same precedence — raw keys win. Use for kafkajs-internal knobs (`retry`, `metadataMaxAge`) or to override defaults like `idempotent`.
+
+### Custom partitioner (kafkajs driver)
+
+```ts
+const tenantAwarePartitioner = () => ({ topic, partitionMetadata, message }) => {
+  const tenant = message.headers["x-tenant"]?.toString();
+  return hashToPartition(tenant, partitionMetadata.length);
+};
+
+new KafkaPublisher({
+  brokers,
+  driver: "kafkajs",
+  customPartitioner: tenantAwarePartitioner,
+});
+```
+
+Overrides the `partitioner` preset. Confluent ignores this — librdkafka's partitioner is a C-level extension point, not a JS callback.
+
 ### Logger
 
 Pass a `Logger` (the same interface used by `@eventferry/core`) to route the publisher's own diagnostics — driver warnings, hook failures — through your logging stack:
@@ -284,6 +367,107 @@ new KafkaPublisher({
 
 When omitted, the publisher is silent and the driver falls back to `console.warn` for its diagnostics (preserves prior behavior).
 
+## Admin operations
+
+The publisher exposes a typed admin surface for listing/describing/creating topics — handy for provisioning in CI, integration tests, or app boot.
+
+### `publisher.admin()`
+
+Borrow a fresh admin client. The returned client is connected and ready; the **caller** is responsible for closing it.
+
+```ts
+const admin = await publisher.admin();
+try {
+  const topics = await admin.listTopics();
+  const desc = await admin.describeTopics(["orders"]);
+  console.log(desc[0].partitions.length); // partition count
+} finally {
+  await admin.close();
+}
+```
+
+Methods on the returned `KafkaAdmin`:
+
+- `listTopics(): Promise<string[]>` — all topic names visible to this principal.
+- `describeTopics(topics): Promise<TopicMetadata[]>` — partition / leader / ISR per topic. Missing topics come back with an empty `partitions` array (no try/catch needed to detect absence).
+- `createTopics(specs)` — idempotent: existing topics are silently skipped.
+- `createPartitions(specs)` — grow a topic's partition count (Kafka does not support shrinking).
+- `close()` — disconnect.
+
+### `publisher.ensureTopics()`
+
+One-shot, idempotent provisioning built on top of the admin surface:
+
+```ts
+await publisher.ensureTopics([
+  { topic: "orders", numPartitions: 12, replicationFactor: 3 },
+  { topic: "orders.dlq", numPartitions: 3, replicationFactor: 3, configEntries: { "retention.ms": "604800000" } },
+]);
+
+// Optionally grow existing topics whose partition count is below the requested numPartitions:
+await publisher.ensureTopics(
+  [{ topic: "orders", numPartitions: 24 }],
+  { growPartitions: true },
+);
+```
+
+What it does:
+
+- Creates topics that don't exist.
+- Skips topics that already exist (no error, no surprise alter).
+- With `growPartitions: true`, calls `createPartitions` for existing topics whose current partition count is **below** the requested `numPartitions`.
+
+What it does NOT do (by design):
+
+- Reconcile replication factor on existing topics — Kafka has no safe in-place alter (use partition reassignment for that).
+- Reconcile `configEntries` on existing topics — use `kafka-configs.sh` or the raw admin client (kafkajs's `alterConfigs`) if you need that.
+
+### Consumer helpers — `@eventferry/kafka/consume`
+
+eventferry is publisher-only, but the records it produces are consumed somewhere downstream. The `consume` subpath ships zero-dep helpers for the typical decode + trace-continuation glue, and pulls in no kafkajs/confluent code:
+
+```ts
+import { decode, extractTraceContext } from "@eventferry/kafka/consume";
+
+await consumer.run({
+  eachMessage: async ({ message }) => {
+    // Normalize key/headers/value; default decoder is JSON.
+    const { key, value, headers, offset } = decode<{ orderId: string }>(message);
+
+    // Continue the producer's W3C trace context (if the publisher's tracer
+    // injects it — see "OpenTelemetry tracing" above for the inject hook).
+    const trace = extractTraceContext(message.headers);
+    if (trace) {
+      // → start a CONSUMER span as a child of trace.traceId / trace.spanId
+    }
+
+    await handle(value!);
+  },
+});
+```
+
+`decode` options:
+
+- `decoder: "json"` (default) — `JSON.parse(value.toString("utf8"))`. Empty/null value → `null` (handles compaction tombstones).
+- `decoder: "utf8"` — raw text.
+- `decoder: "none"` — raw `Buffer`.
+- `decoder: (bytes) => …` — custom (Avro, Protobuf, MessagePack, …).
+
+`extractTraceContext` returns `null` if no `traceparent` header is present or it fails W3C validation (all-zero IDs, `version: ff`, malformed hex). It accepts both raw consumer headers (Buffer values) and already-decoded headers (string values).
+
+### `validateTopicsOnConnect`
+
+Fail-fast at startup if expected topics are missing:
+
+```ts
+new KafkaPublisher({
+  brokers,
+  validateTopicsOnConnect: ["orders", "orders.dlq", "events"],
+});
+```
+
+`connect()` opens an admin, runs `listTopics`, and throws a single descriptive error naming **every** missing topic. The admin is always closed (success or failure). Skip the check entirely with an empty list or by omitting the option.
+
 📖 **Full documentation:** [github.com/SametGoktepe/eventferry](https://github.com/SametGoktepe/eventferry#readme)
 
 ## License

package/dist/consume.cjs

@@ -0,0 +1,115 @@
+"use strict";
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+
+// src/consume.ts
+var consume_exports = {};
+__export(consume_exports, {
+  decode: () => decode,
+  decodeHeaders: () => decodeHeaders,
+  extractTraceContext: () => extractTraceContext
+});
+module.exports = __toCommonJS(consume_exports);
+function decodeHeaders(raw) {
+  if (!raw) return {};
+  const out = {};
+  for (const [k, v] of Object.entries(raw)) {
+    if (v === void 0 || v === null) continue;
+    out[k] = Buffer.isBuffer(v) ? v.toString("utf8") : v;
+  }
+  return out;
+}
+function decode(msg, opts = {}) {
+  const headers = decodeHeaders(msg.headers);
+  const key = normalizeKey(msg.key);
+  const value = decodeValue(msg.value, opts.decoder ?? "json");
+  const timestamp = msg.timestamp !== void 0 ? Number(msg.timestamp) : void 0;
+  const offset = msg.offset !== void 0 ? String(msg.offset) : void 0;
+  return {
+    key,
+    value,
+    headers,
+    timestamp,
+    offset,
+    partition: msg.partition
+  };
+}
+function normalizeKey(key) {
+  if (key === null || key === void 0) return null;
+  return Buffer.isBuffer(key) ? key.toString("utf8") : key;
+}
+function decodeValue(value, decoder) {
+  if (value === null || value === void 0) return null;
+  const buf = Buffer.isBuffer(value) ? value : Buffer.from(value);
+  if (buf.length === 0) return null;
+  if (typeof decoder === "function") return decoder(buf);
+  switch (decoder) {
+    case "utf8":
+      return buf.toString("utf8");
+    case "none":
+      return buf;
+    case "json":
+    case void 0:
+    default: {
+      const text = buf.toString("utf8");
+      try {
+        return JSON.parse(text);
+      } catch (err) {
+        throw new Error(
+          `decode: JSON.parse failed on message value: ${err.message}`
+        );
+      }
+    }
+  }
+}
+var TRACEPARENT_RE = /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;
+var INVALID_TRACE_ID = "0".repeat(32);
+var INVALID_SPAN_ID = "0".repeat(16);
+function extractTraceContext(headers) {
+  if (!headers) return null;
+  const tp = readHeader(headers, "traceparent");
+  if (!tp) return null;
+  const match = TRACEPARENT_RE.exec(tp);
+  if (!match) return null;
+  const [, version, traceId, spanId, flags] = match;
+  if (version === "ff") return null;
+  if (traceId === INVALID_TRACE_ID || spanId === INVALID_SPAN_ID) return null;
+  const sampled = (parseInt(flags, 16) & 1) === 1;
+  const ts = readHeader(headers, "tracestate");
+  return {
+    traceparent: tp,
+    tracestate: ts && ts.length > 0 ? ts : void 0,
+    traceId,
+    spanId,
+    sampled
+  };
+}
+function readHeader(headers, name) {
+  const v = headers[name];
+  if (v === void 0 || v === null) return void 0;
+  if (typeof v === "string") return v;
+  if (Buffer.isBuffer(v)) return v.toString("utf8");
+  return void 0;
+}
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  decode,
+  decodeHeaders,
+  extractTraceContext
+});
+//# sourceMappingURL=consume.cjs.map

package/dist/consume.cjs.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/consume.ts"],"sourcesContent":["/**\n * Consumer-side helpers — paired with the publisher's outbound surface.\n *\n * eventferry is a publisher-only library, but the messages it produces are\n * consumed somewhere downstream. This module normalizes the message shape\n * that both `kafkajs` and `@confluentinc/kafka-javascript` deliver to\n * consumer callbacks, decodes the payload, and extracts the W3C trace\n * context the publisher injected (see {@link KafkaTracer.inject}).\n *\n * Imported via subpath so consumer code paths don't pull in the producer:\n *\n *   import { decode, extractTraceContext } from \"@eventferry/kafka/consume\";\n *\n * There is intentionally NO Kafka client here — bring your own consumer\n * (kafkajs's `Consumer`, librdkafka's, whatever) and call `decode()` /\n * `extractTraceContext()` on the message you receive.\n */\n\n/**\n * Raw incoming Kafka message — structural subset both kafkajs and confluent\n * (via the kafkaJS-compat layer) deliver. Fields are optional because\n * different consumer APIs surface different subsets.\n */\nexport interface IncomingKafkaMessage {\n  key?: Buffer | string | null;\n  value?: Buffer | string | null;\n  headers?: IncomingHeaders;\n  /** ISO ms string or numeric epoch ms — depends on the client. */\n  timestamp?: string | number;\n  /** Numeric or string per client. */\n  offset?: string | number;\n  partition?: number;\n}\n\n/** Headers as the underlying clients deliver them: bytes, strings, or undefined. */\nexport type IncomingHeaders = Record<string, Buffer | string | undefined>;\n\n/** Headers normalized to UTF-8 strings (the form most application code wants). */\nexport type DecodedHeaders = Record<string, string>;\n\n/** Payload decoder. Buffer in, decoded value out. */\nexport type Decoder<T> = (bytes: Buffer) => T;\n\n/** Decoded message wrapper — value plus normalized headers, key, metadata. */\nexport interface DecodedMessage<V = unknown> {\n  key: string | null;\n  value: V | null;\n  headers: DecodedHeaders;\n  /** Epoch ms when the broker stamped the record. */\n  timestamp?: number;\n  /** Stringified offset (Kafka offsets exceed 2^53 — strings stay safe). */\n  offset?: string;\n  partition?: number;\n}\n\nexport interface DecodeOptions<V> {\n  /**\n   * Decoder for the payload bytes. Built-ins:\n   *\n   * - `\"json\"` (default) — `JSON.parse(value.toString(\"utf8\"))`. Empty\n   *   value returns `null` (matches Kafka tombstones on compacted topics).\n   * - `\"utf8\"` — raw text. Returns the string as-is.\n   * - `\"none\"` — returns the raw `Buffer` unchanged.\n   *\n   * Or pass your own `(bytes: Buffer) => V` for Avro / Protobuf / MessagePack.\n   */\n  decoder?: \"json\" | \"utf8\" | \"none\" | Decoder<V>;\n}\n\n/**\n * Normalize headers to a plain string→string map. Buffers are read as UTF-8;\n * `undefined` entries are dropped (consumers occasionally surface absent\n * headers as `undefined` values).\n */\nexport function decodeHeaders(raw?: IncomingHeaders): DecodedHeaders {\n  if (!raw) return {};\n  const out: DecodedHeaders = {};\n  for (const [k, v] of Object.entries(raw)) {\n    if (v === undefined || v === null) continue;\n    out[k] = Buffer.isBuffer(v) ? v.toString(\"utf8\") : v;\n  }\n  return out;\n}\n\n/**\n * Decode a Kafka message: normalize the key + headers, decode the value\n * with the chosen decoder, and surface the broker metadata.\n *\n * Tombstones (null/empty value) come back with `value: null` regardless of\n * the decoder — compaction-friendly.\n *\n * @throws when `decoder: \"json\"` (the default) and the payload is non-empty\n *   but not valid JSON. Catch the error and decide whether to DLQ the\n *   record or skip it — eventferry does not assume.\n */\nexport function decode<V = unknown>(\n  msg: IncomingKafkaMessage,\n  opts: DecodeOptions<V> = {},\n): DecodedMessage<V> {\n  const headers = decodeHeaders(msg.headers);\n  const key = normalizeKey(msg.key);\n  const value = decodeValue<V>(msg.value, opts.decoder ?? \"json\");\n  const timestamp =\n    msg.timestamp !== undefined ? Number(msg.timestamp) : undefined;\n  const offset = msg.offset !== undefined ? String(msg.offset) : undefined;\n  return {\n    key,\n    value,\n    headers,\n    timestamp,\n    offset,\n    partition: msg.partition,\n  };\n}\n\nfunction normalizeKey(key?: Buffer | string | null): string | null {\n  if (key === null || key === undefined) return null;\n  return Buffer.isBuffer(key) ? key.toString(\"utf8\") : key;\n}\n\nfunction decodeValue<V>(\n  value: Buffer | string | null | undefined,\n  decoder: DecodeOptions<V>[\"decoder\"],\n): V | null {\n  if (value === null || value === undefined) return null;\n  const buf = Buffer.isBuffer(value) ? value : Buffer.from(value);\n  // Tombstones: an empty buffer is a kafka \"delete me\" on compacted\n  // topics. Surface as null for every decoder — applications usually\n  // want the same null-handling for both.\n  if (buf.length === 0) return null;\n  if (typeof decoder === \"function\") return decoder(buf);\n  switch (decoder) {\n    case \"utf8\":\n      return buf.toString(\"utf8\") as unknown as V;\n    case \"none\":\n      return buf as unknown as V;\n    case \"json\":\n    case undefined:\n    default: {\n      const text = buf.toString(\"utf8\");\n      try {\n        return JSON.parse(text) as V;\n      } catch (err) {\n        throw new Error(\n          `decode: JSON.parse failed on message value: ${(err as Error).message}`,\n        );\n      }\n    }\n  }\n}\n\n/**\n * W3C Trace Context extracted from message headers.\n *\n * - `traceparent`: full header value, format `version-traceId-spanId-flags`.\n * - `tracestate`: optional vendor-specific state (W3C `tracestate` header).\n * - `traceId`: 32 hex chars, parsed from `traceparent`.\n * - `spanId`: 16 hex chars (the PARENT span id from the producer).\n * - `sampled`: parsed from the `traceparent` flags (bit 0 = sampled).\n *\n * Returns `null` when no `traceparent` header is present or the value\n * fails W3C validation.\n *\n * Spec: https://www.w3.org/TR/trace-context/\n */\nexport interface TraceContext {\n  traceparent: string;\n  tracestate?: string;\n  traceId: string;\n  spanId: string;\n  sampled: boolean;\n}\n\nconst TRACEPARENT_RE =\n  /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;\nconst INVALID_TRACE_ID = \"0\".repeat(32);\nconst INVALID_SPAN_ID = \"0\".repeat(16);\n\n/**\n * Extract the W3C trace context the publisher injected into headers.\n * Headers may be raw (Buffer values) or already-decoded (string values) —\n * both shapes work, so you can call this before OR after `decode()`.\n *\n * Validation follows the W3C spec strictly: invalid all-zero trace/span\n * IDs are rejected, version `ff` is rejected, malformed hex is rejected.\n * On any of these, the function returns `null` rather than throwing —\n * consumer code should fall back to starting a fresh trace.\n */\nexport function extractTraceContext(\n  headers: IncomingHeaders | DecodedHeaders | undefined,\n): TraceContext | null {\n  if (!headers) return null;\n  const tp = readHeader(headers, \"traceparent\");\n  if (!tp) return null;\n  const match = TRACEPARENT_RE.exec(tp);\n  if (!match) return null;\n  const [, version, traceId, spanId, flags] = match as unknown as [\n    string,\n    string,\n    string,\n    string,\n    string,\n  ];\n  // Spec §3.2.2.5: version \"ff\" is forbidden (reserved sentinel).\n  if (version === \"ff\") return null;\n  if (traceId === INVALID_TRACE_ID || spanId === INVALID_SPAN_ID) return null;\n  const sampled = (parseInt(flags, 16) & 0x01) === 1;\n  const ts = readHeader(headers, \"tracestate\");\n  return {\n    traceparent: tp,\n    tracestate: ts && ts.length > 0 ? ts : undefined,\n    traceId,\n    spanId,\n    sampled,\n  };\n}\n\nfunction readHeader(\n  headers: IncomingHeaders | DecodedHeaders,\n  name: string,\n): string | undefined {\n  const v = (headers as Record<string, Buffer | string | undefined>)[name];\n  if (v === undefined || v === null) return undefined;\n  if (typeof v === \"string\") return v;\n  if (Buffer.isBuffer(v)) return v.toString(\"utf8\");\n  return undefined;\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AA0EO,SAAS,cAAc,KAAuC;AACnE,MAAI,CAAC,IAAK,QAAO,CAAC;AAClB,QAAM,MAAsB,CAAC;AAC7B,aAAW,CAAC,GAAG,CAAC,KAAK,OAAO,QAAQ,GAAG,GAAG;AACxC,QAAI,MAAM,UAAa,MAAM,KAAM;AACnC,QAAI,CAAC,IAAI,OAAO,SAAS,CAAC,IAAI,EAAE,SAAS,MAAM,IAAI;AAAA,EACrD;AACA,SAAO;AACT;AAaO,SAAS,OACd,KACA,OAAyB,CAAC,GACP;AACnB,QAAM,UAAU,cAAc,IAAI,OAAO;AACzC,QAAM,MAAM,aAAa,IAAI,GAAG;AAChC,QAAM,QAAQ,YAAe,IAAI,OAAO,KAAK,WAAW,MAAM;AAC9D,QAAM,YACJ,IAAI,cAAc,SAAY,OAAO,IAAI,SAAS,IAAI;AACxD,QAAM,SAAS,IAAI,WAAW,SAAY,OAAO,IAAI,MAAM,IAAI;AAC/D,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA,WAAW,IAAI;AAAA,EACjB;AACF;AAEA,SAAS,aAAa,KAA6C;AACjE,MAAI,QAAQ,QAAQ,QAAQ,OAAW,QAAO;AAC9C,SAAO,OAAO,SAAS,GAAG,IAAI,IAAI,SAAS,MAAM,IAAI;AACvD;AAEA,SAAS,YACP,OACA,SACU;AACV,MAAI,UAAU,QAAQ,UAAU,OAAW,QAAO;AAClD,QAAM,MAAM,OAAO,SAAS,KAAK,IAAI,QAAQ,OAAO,KAAK,KAAK;AAI9D,MAAI,IAAI,WAAW,EAAG,QAAO;AAC7B,MAAI,OAAO,YAAY,WAAY,QAAO,QAAQ,GAAG;AACrD,UAAQ,SAAS;AAAA,IACf,KAAK;AACH,aAAO,IAAI,SAAS,MAAM;AAAA,IAC5B,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AAAA,IACL,KAAK;AAAA,IACL,SAAS;AACP,YAAM,OAAO,IAAI,SAAS,MAAM;AAChC,UAAI;AACF,eAAO,KAAK,MAAM,IAAI;AAAA,MACxB,SAAS,KAAK;AACZ,cAAM,IAAI;AAAA,UACR,+CAAgD,IAAc,OAAO;AAAA,QACvE;AAAA,MACF;AAAA,IACF;AAAA,EACF;AACF;AAwBA,IAAM,iBACJ;AACF,IAAM,mBAAmB,IAAI,OAAO,EAAE;AACtC,IAAM,kBAAkB,IAAI,OAAO,EAAE;AAY9B,SAAS,oBACd,SACqB;AACrB,MAAI,CAAC,QAAS,QAAO;AACrB,QAAM,KAAK,WAAW,SAAS,aAAa;AAC5C,MAAI,CAAC,GAAI,QAAO;AAChB,QAAM,QAAQ,eAAe,KAAK,EAAE;AACpC,MAAI,CAAC,MAAO,QAAO;AACnB,QAAM,CAAC,EAAE,SAAS,SAAS,QAAQ,KAAK,IAAI;AAQ5C,MAAI,YAAY,KAAM,QAAO;AAC7B,MAAI,YAAY,oBAAoB,WAAW,gBAAiB,QAAO;AACvE,QAAM,WAAW,SAAS,OAAO,EAAE,IAAI,OAAU;AACjD,QAAM,KAAK,WAAW,SAAS,YAAY;AAC3C,SAAO;AAAA,IACL,aAAa;AAAA,IACb,YAAY,MAAM,GAAG,SAAS,IAAI,KAAK;AAAA,IACvC;AAAA,IACA;AAAA,IACA;AAAA,EACF;AACF;AAEA,SAAS,WACP,SACA,MACoB;AACpB,QAAM,IAAK,QAAwD,IAAI;AACvE,MAAI,MAAM,UAAa,MAAM,KAAM,QAAO;AAC1C,MAAI,OAAO,MAAM,SAAU,QAAO;AAClC,MAAI,OAAO,SAAS,CAAC,EAAG,QAAO,EAAE,SAAS,MAAM;AAChD,SAAO;AACT;","names":[]}

package/dist/consume.d.cts

@@ -0,0 +1,114 @@
+/**
+ * Consumer-side helpers — paired with the publisher's outbound surface.
+ *
+ * eventferry is a publisher-only library, but the messages it produces are
+ * consumed somewhere downstream. This module normalizes the message shape
+ * that both `kafkajs` and `@confluentinc/kafka-javascript` deliver to
+ * consumer callbacks, decodes the payload, and extracts the W3C trace
+ * context the publisher injected (see {@link KafkaTracer.inject}).
+ *
+ * Imported via subpath so consumer code paths don't pull in the producer:
+ *
+ *   import { decode, extractTraceContext } from "@eventferry/kafka/consume";
+ *
+ * There is intentionally NO Kafka client here — bring your own consumer
+ * (kafkajs's `Consumer`, librdkafka's, whatever) and call `decode()` /
+ * `extractTraceContext()` on the message you receive.
+ */
+/**
+ * Raw incoming Kafka message — structural subset both kafkajs and confluent
+ * (via the kafkaJS-compat layer) deliver. Fields are optional because
+ * different consumer APIs surface different subsets.
+ */
+interface IncomingKafkaMessage {
+    key?: Buffer | string | null;
+    value?: Buffer | string | null;
+    headers?: IncomingHeaders;
+    /** ISO ms string or numeric epoch ms — depends on the client. */
+    timestamp?: string | number;
+    /** Numeric or string per client. */
+    offset?: string | number;
+    partition?: number;
+}
+/** Headers as the underlying clients deliver them: bytes, strings, or undefined. */
+type IncomingHeaders = Record<string, Buffer | string | undefined>;
+/** Headers normalized to UTF-8 strings (the form most application code wants). */
+type DecodedHeaders = Record<string, string>;
+/** Payload decoder. Buffer in, decoded value out. */
+type Decoder<T> = (bytes: Buffer) => T;
+/** Decoded message wrapper — value plus normalized headers, key, metadata. */
+interface DecodedMessage<V = unknown> {
+    key: string | null;
+    value: V | null;
+    headers: DecodedHeaders;
+    /** Epoch ms when the broker stamped the record. */
+    timestamp?: number;
+    /** Stringified offset (Kafka offsets exceed 2^53 — strings stay safe). */
+    offset?: string;
+    partition?: number;
+}
+interface DecodeOptions<V> {
+    /**
+     * Decoder for the payload bytes. Built-ins:
+     *
+     * - `"json"` (default) — `JSON.parse(value.toString("utf8"))`. Empty
+     *   value returns `null` (matches Kafka tombstones on compacted topics).
+     * - `"utf8"` — raw text. Returns the string as-is.
+     * - `"none"` — returns the raw `Buffer` unchanged.
+     *
+     * Or pass your own `(bytes: Buffer) => V` for Avro / Protobuf / MessagePack.
+     */
+    decoder?: "json" | "utf8" | "none" | Decoder<V>;
+}
+/**
+ * Normalize headers to a plain string→string map. Buffers are read as UTF-8;
+ * `undefined` entries are dropped (consumers occasionally surface absent
+ * headers as `undefined` values).
+ */
+declare function decodeHeaders(raw?: IncomingHeaders): DecodedHeaders;
+/**
+ * Decode a Kafka message: normalize the key + headers, decode the value
+ * with the chosen decoder, and surface the broker metadata.
+ *
+ * Tombstones (null/empty value) come back with `value: null` regardless of
+ * the decoder — compaction-friendly.
+ *
+ * @throws when `decoder: "json"` (the default) and the payload is non-empty
+ *   but not valid JSON. Catch the error and decide whether to DLQ the
+ *   record or skip it — eventferry does not assume.
+ */
+declare function decode<V = unknown>(msg: IncomingKafkaMessage, opts?: DecodeOptions<V>): DecodedMessage<V>;
+/**
+ * W3C Trace Context extracted from message headers.
+ *
+ * - `traceparent`: full header value, format `version-traceId-spanId-flags`.
+ * - `tracestate`: optional vendor-specific state (W3C `tracestate` header).
+ * - `traceId`: 32 hex chars, parsed from `traceparent`.
+ * - `spanId`: 16 hex chars (the PARENT span id from the producer).
+ * - `sampled`: parsed from the `traceparent` flags (bit 0 = sampled).
+ *
+ * Returns `null` when no `traceparent` header is present or the value
+ * fails W3C validation.
+ *
+ * Spec: https://www.w3.org/TR/trace-context/
+ */
+interface TraceContext {
+    traceparent: string;
+    tracestate?: string;
+    traceId: string;
+    spanId: string;
+    sampled: boolean;
+}
+/**
+ * Extract the W3C trace context the publisher injected into headers.
+ * Headers may be raw (Buffer values) or already-decoded (string values) —
+ * both shapes work, so you can call this before OR after `decode()`.
+ *
+ * Validation follows the W3C spec strictly: invalid all-zero trace/span
+ * IDs are rejected, version `ff` is rejected, malformed hex is rejected.
+ * On any of these, the function returns `null` rather than throwing —
+ * consumer code should fall back to starting a fresh trace.
+ */
+declare function extractTraceContext(headers: IncomingHeaders | DecodedHeaders | undefined): TraceContext | null;
+
+export { type DecodeOptions, type DecodedHeaders, type DecodedMessage, type Decoder, type IncomingHeaders, type IncomingKafkaMessage, type TraceContext, decode, decodeHeaders, extractTraceContext };

package/dist/consume.d.ts

@@ -0,0 +1,114 @@
+/**
+ * Consumer-side helpers — paired with the publisher's outbound surface.
+ *
+ * eventferry is a publisher-only library, but the messages it produces are
+ * consumed somewhere downstream. This module normalizes the message shape
+ * that both `kafkajs` and `@confluentinc/kafka-javascript` deliver to
+ * consumer callbacks, decodes the payload, and extracts the W3C trace
+ * context the publisher injected (see {@link KafkaTracer.inject}).
+ *
+ * Imported via subpath so consumer code paths don't pull in the producer:
+ *
+ *   import { decode, extractTraceContext } from "@eventferry/kafka/consume";
+ *
+ * There is intentionally NO Kafka client here — bring your own consumer
+ * (kafkajs's `Consumer`, librdkafka's, whatever) and call `decode()` /
+ * `extractTraceContext()` on the message you receive.
+ */
+/**
+ * Raw incoming Kafka message — structural subset both kafkajs and confluent
+ * (via the kafkaJS-compat layer) deliver. Fields are optional because
+ * different consumer APIs surface different subsets.
+ */
+interface IncomingKafkaMessage {
+    key?: Buffer | string | null;
+    value?: Buffer | string | null;
+    headers?: IncomingHeaders;
+    /** ISO ms string or numeric epoch ms — depends on the client. */
+    timestamp?: string | number;
+    /** Numeric or string per client. */
+    offset?: string | number;
+    partition?: number;
+}
+/** Headers as the underlying clients deliver them: bytes, strings, or undefined. */
+type IncomingHeaders = Record<string, Buffer | string | undefined>;
+/** Headers normalized to UTF-8 strings (the form most application code wants). */
+type DecodedHeaders = Record<string, string>;
+/** Payload decoder. Buffer in, decoded value out. */
+type Decoder<T> = (bytes: Buffer) => T;
+/** Decoded message wrapper — value plus normalized headers, key, metadata. */
+interface DecodedMessage<V = unknown> {
+    key: string | null;
+    value: V | null;
+    headers: DecodedHeaders;
+    /** Epoch ms when the broker stamped the record. */
+    timestamp?: number;
+    /** Stringified offset (Kafka offsets exceed 2^53 — strings stay safe). */
+    offset?: string;
+    partition?: number;
+}
+interface DecodeOptions<V> {
+    /**
+     * Decoder for the payload bytes. Built-ins:
+     *
+     * - `"json"` (default) — `JSON.parse(value.toString("utf8"))`. Empty
+     *   value returns `null` (matches Kafka tombstones on compacted topics).
+     * - `"utf8"` — raw text. Returns the string as-is.
+     * - `"none"` — returns the raw `Buffer` unchanged.
+     *
+     * Or pass your own `(bytes: Buffer) => V` for Avro / Protobuf / MessagePack.
+     */
+    decoder?: "json" | "utf8" | "none" | Decoder<V>;
+}
+/**
+ * Normalize headers to a plain string→string map. Buffers are read as UTF-8;
+ * `undefined` entries are dropped (consumers occasionally surface absent
+ * headers as `undefined` values).
+ */
+declare function decodeHeaders(raw?: IncomingHeaders): DecodedHeaders;
+/**
+ * Decode a Kafka message: normalize the key + headers, decode the value
+ * with the chosen decoder, and surface the broker metadata.
+ *
+ * Tombstones (null/empty value) come back with `value: null` regardless of
+ * the decoder — compaction-friendly.
+ *
+ * @throws when `decoder: "json"` (the default) and the payload is non-empty
+ *   but not valid JSON. Catch the error and decide whether to DLQ the
+ *   record or skip it — eventferry does not assume.
+ */
+declare function decode<V = unknown>(msg: IncomingKafkaMessage, opts?: DecodeOptions<V>): DecodedMessage<V>;
+/**
+ * W3C Trace Context extracted from message headers.
+ *
+ * - `traceparent`: full header value, format `version-traceId-spanId-flags`.
+ * - `tracestate`: optional vendor-specific state (W3C `tracestate` header).
+ * - `traceId`: 32 hex chars, parsed from `traceparent`.
+ * - `spanId`: 16 hex chars (the PARENT span id from the producer).
+ * - `sampled`: parsed from the `traceparent` flags (bit 0 = sampled).
+ *
+ * Returns `null` when no `traceparent` header is present or the value
+ * fails W3C validation.
+ *
+ * Spec: https://www.w3.org/TR/trace-context/
+ */
+interface TraceContext {
+    traceparent: string;
+    tracestate?: string;
+    traceId: string;
+    spanId: string;
+    sampled: boolean;
+}
+/**
+ * Extract the W3C trace context the publisher injected into headers.
+ * Headers may be raw (Buffer values) or already-decoded (string values) —
+ * both shapes work, so you can call this before OR after `decode()`.
+ *
+ * Validation follows the W3C spec strictly: invalid all-zero trace/span
+ * IDs are rejected, version `ff` is rejected, malformed hex is rejected.
+ * On any of these, the function returns `null` rather than throwing —
+ * consumer code should fall back to starting a fresh trace.
+ */
+declare function extractTraceContext(headers: IncomingHeaders | DecodedHeaders | undefined): TraceContext | null;
+
+export { type DecodeOptions, type DecodedHeaders, type DecodedMessage, type Decoder, type IncomingHeaders, type IncomingKafkaMessage, type TraceContext, decode, decodeHeaders, extractTraceContext };

package/dist/consume.js

@@ -0,0 +1,88 @@
+// src/consume.ts
+function decodeHeaders(raw) {
+  if (!raw) return {};
+  const out = {};
+  for (const [k, v] of Object.entries(raw)) {
+    if (v === void 0 || v === null) continue;
+    out[k] = Buffer.isBuffer(v) ? v.toString("utf8") : v;
+  }
+  return out;
+}
+function decode(msg, opts = {}) {
+  const headers = decodeHeaders(msg.headers);
+  const key = normalizeKey(msg.key);
+  const value = decodeValue(msg.value, opts.decoder ?? "json");
+  const timestamp = msg.timestamp !== void 0 ? Number(msg.timestamp) : void 0;
+  const offset = msg.offset !== void 0 ? String(msg.offset) : void 0;
+  return {
+    key,
+    value,
+    headers,
+    timestamp,
+    offset,
+    partition: msg.partition
+  };
+}
+function normalizeKey(key) {
+  if (key === null || key === void 0) return null;
+  return Buffer.isBuffer(key) ? key.toString("utf8") : key;
+}
+function decodeValue(value, decoder) {
+  if (value === null || value === void 0) return null;
+  const buf = Buffer.isBuffer(value) ? value : Buffer.from(value);
+  if (buf.length === 0) return null;
+  if (typeof decoder === "function") return decoder(buf);
+  switch (decoder) {
+    case "utf8":
+      return buf.toString("utf8");
+    case "none":
+      return buf;
+    case "json":
+    case void 0:
+    default: {
+      const text = buf.toString("utf8");
+      try {
+        return JSON.parse(text);
+      } catch (err) {
+        throw new Error(
+          `decode: JSON.parse failed on message value: ${err.message}`
+        );
+      }
+    }
+  }
+}
+var TRACEPARENT_RE = /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;
+var INVALID_TRACE_ID = "0".repeat(32);
+var INVALID_SPAN_ID = "0".repeat(16);
+function extractTraceContext(headers) {
+  if (!headers) return null;
+  const tp = readHeader(headers, "traceparent");
+  if (!tp) return null;
+  const match = TRACEPARENT_RE.exec(tp);
+  if (!match) return null;
+  const [, version, traceId, spanId, flags] = match;
+  if (version === "ff") return null;
+  if (traceId === INVALID_TRACE_ID || spanId === INVALID_SPAN_ID) return null;
+  const sampled = (parseInt(flags, 16) & 1) === 1;
+  const ts = readHeader(headers, "tracestate");
+  return {
+    traceparent: tp,
+    tracestate: ts && ts.length > 0 ? ts : void 0,
+    traceId,
+    spanId,
+    sampled
+  };
+}
+function readHeader(headers, name) {
+  const v = headers[name];
+  if (v === void 0 || v === null) return void 0;
+  if (typeof v === "string") return v;
+  if (Buffer.isBuffer(v)) return v.toString("utf8");
+  return void 0;
+}
+export {
+  decode,
+  decodeHeaders,
+  extractTraceContext
+};
+//# sourceMappingURL=consume.js.map