@relaybus/kafka 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. package/README.md +86 -48
  2. package/dist/index.cjs +267 -169
  3. package/dist/index.cjs.map +1 -1
  4. package/dist/index.d.cts +123 -62
  5. package/dist/index.d.ts +123 -62
  6. package/dist/index.js +256 -168
  7. package/dist/index.js.map +1 -1
  8. package/package.json +4 -4
package/README.md CHANGED
@@ -53,26 +53,27 @@ const consuming = transport.isConsuming();
53
53
 
54
54
  ## Options
55
55
 
56
- | Option | Default | Meaning |
57
- | ------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
58
- | `kafka` | required | A `kafkajs` `Kafka` or Confluent `KafkaJS.Kafka` instance. It is the producer/consumer factory. |
59
- | `name` | `'kafka'` | Transport name used by routes and subscriptions. |
60
- | `groupId` | none | Consumer group. Required as soon as `subscribe` is used; publish-only services can omit it. |
61
- | `fromBeginning` | `true` | Where a group without committed offsets starts. `true` so events published before the first start are not skipped. |
62
- | `maxInFlightRequests` | `1` | Producer in-flight requests, 1 to 5. `1` keeps retries from reordering events of one key. |
63
- | `producerConfig` | `{}` | Extra producer settings (partitioner, compression, ...). `idempotent` and `maxInFlightRequests` always win. |
64
- | `consumerConfig` | `{}` | Extra consumer settings (`sessionTimeout`, `retry`, `heartbeatInterval`, ...). `groupId` always wins. |
65
- | `processing.maxAttempts` | `5` | In-process attempts per delivery of a record. |
66
- | `processing.initialBackoff` | `'200ms'` | Delay before attempt 2, doubling per attempt. |
67
- | `processing.maxBackoff` | `'5s'` | Upper bound of the delay between attempts. |
68
- | `processing.heartbeatInterval` | `'1s'` | Heartbeat interval while sleeping between attempts, so a long backoff never exceeds `sessionTimeout`. |
69
- | `startupTimeout` | `'30s'` | How long `subscribe` waits for the group join. A failed start-up disconnects the consumer it started. |
70
- | `publishTimeout` | `'30s'` | How long `publish` waits for the broker ack before rejecting with `PublishTimeoutError`. |
71
- | `restart.initialBackoff` | `'1s'` | First delay of the consumer restart loop after an unrecoverable crash; doubles per failure. |
72
- | `restart.maxBackoff` | `'30s'` | Upper bound of the restart delay. |
73
- | `onInvalidMessage` | `'dead-letter'` | `'dead-letter'` (to `<topic>.dead`), `{ deadLetterTopic: string \| (sourceTopic) => string }`, a callback `(message) => void \| Promise<void>`, or `'skip'` (drops the record; explicit opt-in). |
74
- | `decode` | strict binary mode | How consumed records become events. `decodeCloudEvent` (binary and structured mode), `plainJsonDecoder({...})` (plain JSON from producers without relaybus) or your own function. See [Interoperating with services that do not use relaybus](#interoperating-with-services-that-do-not-use-relaybus). |
75
- | `logger` | `console` | `log`, `warn`, `error`. Messages start with `[KafkaTransport](method)`. |
56
+ | Option | Default | Meaning |
57
+ | ------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
58
+ | `kafka` | required | A `kafkajs` `Kafka` or Confluent `KafkaJS.Kafka` instance. It is the producer/consumer factory. |
59
+ | `name` | `'kafka'` | Transport name used by routes and subscriptions. |
60
+ | `groupId` | none | Consumer group. Required as soon as `subscribe` is used; publish-only services can omit it. |
61
+ | `fromBeginning` | `true` | Where a group without committed offsets starts. `true` so events published before the first start are not skipped. |
62
+ | `maxInFlightRequests` | `1` | Producer in-flight requests, 1 to 5. `1` keeps retries from reordering events of one key. |
63
+ | `producerConfig` | `{}` | Extra producer settings (partitioner, compression, ...). `idempotent` and `maxInFlightRequests` always win. |
64
+ | `consumerConfig` | `{}` | Extra consumer settings (`sessionTimeout`, `retry`, `heartbeatInterval`, ...). `groupId` always wins. |
65
+ | `processing.maxAttempts` | `5` | In-process attempts per delivery of a record. |
66
+ | `processing.initialBackoff` | `'200ms'` | Delay before attempt 2, doubling per attempt. |
67
+ | `processing.maxBackoff` | `'5s'` | Upper bound of the delay between attempts. |
68
+ | `processing.heartbeatInterval` | `'1s'` | Heartbeat interval while sleeping between attempts, so a long backoff never exceeds `sessionTimeout`. |
69
+ | `startupTimeout` | `'30s'` | How long `subscribe` waits for the group join. A failed start-up disconnects the consumer it started. |
70
+ | `publishTimeout` | `'30s'` | How long `publish` waits for the broker ack before rejecting with `PublishTimeoutError`. |
71
+ | `restart.initialBackoff` | `'1s'` | First delay of the consumer restart loop after an unrecoverable crash; doubles per failure. |
72
+ | `restart.maxBackoff` | `'30s'` | Upper bound of the restart delay. |
73
+ | `onInvalidMessage` | `'dead-letter'` | `'dead-letter'` (to `<topic>.dead`), `{ deadLetterTopic: string \| (sourceTopic) => string }`, a callback `(message) => void \| Promise<void>`, or `'skip'` (drops the record; explicit opt-in). |
74
+ | `codec` | `cloudEventsBinary` | The wire format, both ways: `cloudEventsBinary` (CloudEvents binary mode, strict on read), `cloudEventsStructured`, `jsonEnvelope({...})` (your JSON envelope, or the bare payload) or `{ encode, decode }`. See [Interoperating with services that do not use relaybus](#interoperating-with-services-that-do-not-use-relaybus). |
75
+ | `decode` | none | Deprecated: reads with your function, writes binary mode. Same as `codec: { encode: encodeEnvelope, decode }`; not together with `codec`. |
76
+ | `logger` | `console` | `log`, `warn`, `error`. Messages start with `[KafkaTransport](method)`. |
76
77
 
77
78
  ## Wire format
78
79
 
@@ -115,41 +116,73 @@ binding: Java `io.cloudevents:cloudevents-kafka` (`CloudEventDeserializer`), Go
115
116
  (`protocol/kafka_sarama`), and Spring Cloud Function / Spring Cloud Stream, which read the
116
117
  `ce_`-prefixed headers. Use `ce_id` to deduplicate: an outbox delivers at least once.
117
118
 
118
- **Other producer, relaybus consumer.** By default the transport accepts only binary-mode
119
- CloudEvents; anything else is malformed and dead-lettered, so a misconfigured producer cannot
120
- slip events in. Opt in per transport (one transport is one subscription) with `decode`:
119
+ **Other formats, both ways.** By default the transport reads only binary-mode CloudEvents;
120
+ anything else is malformed and dead-lettered, so a misconfigured producer cannot slip events in.
121
+ The `codec` option sets another wire format for publishing and consuming (one transport is one
122
+ subscription):
121
123
 
122
124
  ```ts
123
125
  import {
124
126
  KafkaTransport,
125
- decodeCloudEvent,
127
+ cloudEventsStructured,
128
+ encodeEnvelope,
126
129
  fromHeader,
127
- fromPayloadField,
130
+ jsonEnvelope,
128
131
  plainJsonDecoder,
129
- type KafkaMessageDecoder,
132
+ type KafkaCodec,
130
133
  } from '@relaybus/kafka';
131
134
 
132
- // A Spring / Go / plain kafkajs producer writing JSON values with an `event-type` header.
133
- const transport = new KafkaTransport({
134
- kafka,
135
- groupId: 'billing',
136
- decode: plainJsonDecoder({ type: fromHeader('event-type'), id: fromHeader('message-id') }),
135
+ // CloudEvents structured mode (`content-type: application/cloudevents+json`); reads binary too.
136
+ new KafkaTransport({ kafka, groupId: 'billing', codec: cloudEventsStructured });
137
+
138
+ // A service envelope: `{ "eventName": ..., "messageId": ..., "data": {...}, "meta": {...} }`.
139
+ const envelope = jsonEnvelope({
140
+ type: { body: 'eventName' },
141
+ id: { body: 'messageId' },
142
+ payload: { body: 'data' },
143
+ correlationId: { body: 'meta.correlationId' },
144
+ occurredAt: { body: 'meta.occurredAt' },
145
+ headers: { body: 'meta.headers' },
137
146
  });
138
147
 
139
- // A topic that carries one kind of event; the ordering key from the payload.
140
- const oneEventType = plainJsonDecoder({
141
- type: 'order.placed.v1',
142
- key: fromPayloadField('orderId'),
148
+ // The bare JSON payload of a Spring / Go / plain kafkajs producer, the type in a header.
149
+ const plain = jsonEnvelope({
150
+ type: { header: 'event-type' },
151
+ id: 'record-position', // kafka:<topic>:<partition>:<offset>, or { header: 'message-id' }
152
+ payload: 'whole-body',
143
153
  });
144
154
 
145
- // A CloudEvents SDK producer in structured mode (`content-type: application/cloudevents+json`).
146
- const cloudEvents: KafkaMessageDecoder = decodeCloudEvent;
147
-
148
- // Different formats per topic: switch on the record.
149
- const perTopic: KafkaMessageDecoder = (record) =>
150
- record.topic === 'legacy.orders' ? oneEventType(record) : decodeCloudEvent(record);
155
+ // Your own: binary mode out, plain JSON and CloudEvents in.
156
+ const custom: KafkaCodec = {
157
+ encode: encodeEnvelope,
158
+ decode: plainJsonDecoder({ type: fromHeader('event-type') }),
159
+ };
151
160
  ```
152
161
 
162
+ | Codec | Writes | Reads |
163
+ | ----------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------ |
164
+ | `cloudEventsBinary` | Binary mode, as above. Default, byte for byte unchanged. | Binary mode only, strictly. |
165
+ | `cloudEventsStructured` | One `application/cloudevents+json` document. | Structured and binary mode (`decodeCloudEvent`). |
166
+ | `jsonEnvelope(mapping)` | The mapped JSON body and headers. | The same mapping. A missing type, mapped id or payload is malformed. |
167
+ | `{ encode, decode }` | Whatever `encode` returns. | Whatever `decode` returns; missing fields are filled as for a decoder (table below). |
168
+
169
+ `jsonEnvelope` mappings name a body path (`{ body: 'meta.correlationId' }`, `'messageType.0'`
170
+ indexes an array) or a header (`{ header: 'event-type' }`) for `type`, `id`, `payload` (or
171
+ `'whole-body'`), `key`, `source`, `occurredAt`, `correlationId`, `causationId` and `headers`, plus a
172
+ `contentType`. `type`, `id` and `payload` are required by the types; `id: 'record-position'` uses the
173
+ record position (records relaybus writes carry the event id in a `relaybus-event-id` header, which
174
+ wins on read, so outbox retries stay deduplicated). The record key is always the event key, and when `occurredAt` is not mapped the
175
+ record timestamp carries it. Event headers that are record headers keep their names, so a name the
176
+ mapping uses, `content-type` or `relaybus-event-id`, is refused with `UnencodableEventError`. The
177
+ [interoperability recipe](https://berkcansavur.github.io/relaybus/recipes/interoperating-with-other-services)
178
+ has a full table and a MassTransit example.
179
+
180
+ Decoded payloads go through the event's schema (`defineEvent(type, { schema })`) as relaybus's own
181
+ do; a payload that fails it makes the delivery dead with `EventValidationError`.
182
+
183
+ The decoders below remain available for a custom codec's `decode`, and for the deprecated `decode`
184
+ option:
185
+
153
186
  | Decoder | Accepts |
154
187
  | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
155
188
  | default (`decodeEnvelope`) | Binary-mode CloudEvents only, strictly as described above. Unchanged. |
@@ -172,17 +205,18 @@ Every record header becomes an event header (the last value of a repeated header
172
205
  not valid UTF-8 is left out, so binary ids can never collide).
173
206
 
174
207
  **Guarantees stay the same.** Decoding runs again on every redelivery and must be deterministic;
175
- the inbox deduplicates on the id, so a redelivered record is stored and handled once. A record that
176
- cannot be decoded (not JSON, a tombstone, no type, an invalid type, an extractor returning a
177
- non-string) and anything a custom decoder throws go to `onInvalidMessage` (dead-letter by default)
208
+ the inbox deduplicates on the id, so a redelivered record is stored and handled once. Under every
209
+ codec, a record that cannot be decoded (not JSON, a tombstone, no type, an invalid type, a missing
210
+ mapped id, an extractor returning a non-string) and anything a custom `decode` throws go to
211
+ `onInvalidMessage` (dead-letter by default)
178
212
  with the reason in `relaybus-invalid-reason`; the consumer keeps running. Per-key order is the
179
213
  Kafka partition order, as for relaybus producers.
180
214
 
181
215
  **Limits.** The derived id assumes offsets are never reused: deleting and recreating a topic
182
216
  restarts its offsets, so records after a recreation could collide with ids already in the inbox
183
217
  (within its retention). Two sends of one event by a producer are two offsets; only an id set by the
184
- producer (`id: fromHeader(...)`) dedupes them. Payloads must be JSON (Avro or Protobuf need your own
185
- decoder function, which must stay synchronous).
218
+ producer (`id: fromHeader(...)` or `{ header }` / `{ body }` in a mapping) dedupes them. Avro or
219
+ Protobuf need your own codec, whose functions must stay synchronous.
186
220
 
187
221
  ## Partitioning and ordering
188
222
 
@@ -219,8 +253,12 @@ one key can map to different partitions depending on who wrote it.
219
253
  - **Malformed or rejected record**: a record that cannot be decoded, or that the bus rejects with
220
254
  any `NonRetryableError` (such as `InvalidEnvelopeError` or `EventValidationError`), is handled by
221
255
  `onInvalidMessage`. The default copies the untouched record to `<topic>.dead` with its original
222
- headers plus `relaybus-invalid-reason`, `relaybus-source-topic`, `relaybus-source-partition` and
223
- `relaybus-source-offset`, using the same idempotent producer, and commits only after the ack.
256
+ headers plus `relaybus-invalid-reason`, `relaybus-source-topic`, `relaybus-source-partition`,
257
+ `relaybus-source-offset`, `relaybus-error-name` (the error class, such as
258
+ `InvalidKafkaMessageError`) and, for relaybus errors, `relaybus-error-code` (such as
259
+ `RELAYBUS_KAFKA_INVALID_MESSAGE`, see [Error codes](https://berkcansavur.github.io/relaybus/concepts/error-codes)),
260
+ using the same idempotent producer, and commits only after the ack. The log line carries the
261
+ same `errorName=` and `code=` fields.
224
262
  A failing callback or dead-letter send is retried like a handler failure, so a record is dropped
225
263
  only when you chose `'skip'` or your callback returned. A subscription that consumes its own
226
264
  dead-letter topic is rejected as a loop.