@relaybus/kafka 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +166 -21
- package/dist/index.cjs +458 -96
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +245 -23
- package/dist/index.d.ts +245 -23
- package/dist/index.js +446 -98
- package/dist/index.js.map +1 -1
- package/package.json +22 -9
package/README.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# @relaybus/kafka
|
|
2
2
|
|
|
3
|
+
@relaybus/kafka is the Apache Kafka transport of relaybus, a reliable, storage-agnostic event bus for NestJS and Node.js: it forwards committed outbox events to Kafka topics and stores consumed events once in an idempotent inbox, so events are not lost between your database and Kafka.
|
|
4
|
+
|
|
3
5
|
Apache Kafka transport for [relaybus](../../README.md). It forwards committed
|
|
4
6
|
outbox events to Kafka topics and feeds events consumed from Kafka into the
|
|
5
7
|
bus inbox, with at-least-once delivery and idempotent storage.
|
|
@@ -51,25 +53,27 @@ const consuming = transport.isConsuming();
|
|
|
51
53
|
|
|
52
54
|
## Options
|
|
53
55
|
|
|
54
|
-
| Option | Default
|
|
55
|
-
| ------------------------------ |
|
|
56
|
-
| `kafka` | required
|
|
57
|
-
| `name` | `'kafka'`
|
|
58
|
-
| `groupId` | none
|
|
59
|
-
| `fromBeginning` | `true`
|
|
60
|
-
| `maxInFlightRequests` | `1`
|
|
61
|
-
| `producerConfig` | `{}`
|
|
62
|
-
| `consumerConfig` | `{}`
|
|
63
|
-
| `processing.maxAttempts` | `5`
|
|
64
|
-
| `processing.initialBackoff` | `'200ms'`
|
|
65
|
-
| `processing.maxBackoff` | `'5s'`
|
|
66
|
-
| `processing.heartbeatInterval` | `'1s'`
|
|
67
|
-
| `startupTimeout` | `'30s'`
|
|
68
|
-
| `publishTimeout` | `'30s'`
|
|
69
|
-
| `restart.initialBackoff` | `'1s'`
|
|
70
|
-
| `restart.maxBackoff` | `'30s'`
|
|
71
|
-
| `onInvalidMessage` | `'dead-letter'`
|
|
72
|
-
| `
|
|
56
|
+
| Option | Default | Meaning |
|
|
57
|
+
| ------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
58
|
+
| `kafka` | required | A `kafkajs` `Kafka` or Confluent `KafkaJS.Kafka` instance. It is the producer/consumer factory. |
|
|
59
|
+
| `name` | `'kafka'` | Transport name used by routes and subscriptions. |
|
|
60
|
+
| `groupId` | none | Consumer group. Required as soon as `subscribe` is used; publish-only services can omit it. |
|
|
61
|
+
| `fromBeginning` | `true` | Where a group without committed offsets starts. `true` so events published before the first start are not skipped. |
|
|
62
|
+
| `maxInFlightRequests` | `1` | Producer in-flight requests, 1 to 5. `1` keeps retries from reordering events of one key. |
|
|
63
|
+
| `producerConfig` | `{}` | Extra producer settings (partitioner, compression, ...). `idempotent` and `maxInFlightRequests` always win. |
|
|
64
|
+
| `consumerConfig` | `{}` | Extra consumer settings (`sessionTimeout`, `retry`, `heartbeatInterval`, ...). `groupId` always wins. |
|
|
65
|
+
| `processing.maxAttempts` | `5` | In-process attempts per delivery of a record. |
|
|
66
|
+
| `processing.initialBackoff` | `'200ms'` | Delay before attempt 2, doubling per attempt. |
|
|
67
|
+
| `processing.maxBackoff` | `'5s'` | Upper bound of the delay between attempts. |
|
|
68
|
+
| `processing.heartbeatInterval` | `'1s'` | Heartbeat interval while sleeping between attempts, so a long backoff never exceeds `sessionTimeout`. |
|
|
69
|
+
| `startupTimeout` | `'30s'` | How long `subscribe` waits for the group join. A failed start-up disconnects the consumer it started. |
|
|
70
|
+
| `publishTimeout` | `'30s'` | How long `publish` waits for the broker ack before rejecting with `PublishTimeoutError`. |
|
|
71
|
+
| `restart.initialBackoff` | `'1s'` | First delay of the consumer restart loop after an unrecoverable crash; doubles per failure. |
|
|
72
|
+
| `restart.maxBackoff` | `'30s'` | Upper bound of the restart delay. |
|
|
73
|
+
| `onInvalidMessage` | `'dead-letter'` | `'dead-letter'` (to `<topic>.dead`), `{ deadLetterTopic: string \| (sourceTopic) => string }`, a callback `(message) => void \| Promise<void>`, or `'skip'` (drops the record; explicit opt-in). |
|
|
74
|
+
| `codec` | `cloudEventsBinary` | The wire format, both ways: `cloudEventsBinary` (CloudEvents binary mode, strict on read), `cloudEventsStructured`, `jsonEnvelope({...})` (your JSON envelope, or the bare payload) or `{ encode, decode }`. See [Interoperating with services that do not use relaybus](#interoperating-with-services-that-do-not-use-relaybus). |
|
|
75
|
+
| `decode` | none | Deprecated: reads with your function, writes binary mode. Same as `codec: { encode: encodeEnvelope, decode }`; not together with `codec`. |
|
|
76
|
+
| `logger` | `console` | `log`, `warn`, `error`. Messages start with `[KafkaTransport](method)`. |
|
|
73
77
|
|
|
74
78
|
## Wire format
|
|
75
79
|
|
|
@@ -102,6 +106,118 @@ RFC 3339), are rejected on publish with `UnencodableEventError`, a `NonRetryable
|
|
|
102
106
|
retrying cannot help. `Date` payload values become ISO strings. A header named `__proto__` is kept
|
|
103
107
|
as ordinary data by the codec (kafkajs itself drops it when it builds the consumer's header object).
|
|
104
108
|
|
|
109
|
+
## Interoperating with services that do not use relaybus
|
|
110
|
+
|
|
111
|
+
**relaybus producer, other consumer.** Any Kafka client reads the record as is: the value is the
|
|
112
|
+
JSON payload (UTF-8), the key is the event key, and the `ce_*` headers above carry the attributes.
|
|
113
|
+
A plain consumer only needs `JSON.parse(value)` and the `ce_type` / `ce_id` headers. CloudEvents
|
|
114
|
+
SDKs decode the record natively, because it is CloudEvents binary mode of the Kafka protocol
|
|
115
|
+
binding: Java `io.cloudevents:cloudevents-kafka` (`CloudEventDeserializer`), Go `sdk-go`
|
|
116
|
+
(`protocol/kafka_sarama`), and Spring Cloud Function / Spring Cloud Stream, which read the
|
|
117
|
+
`ce_`-prefixed headers. Use `ce_id` to deduplicate: an outbox delivers at least once.
|
|
118
|
+
|
|
119
|
+
**Other formats, both ways.** By default the transport reads only binary-mode CloudEvents;
|
|
120
|
+
anything else is malformed and dead-lettered, so a misconfigured producer cannot slip events in.
|
|
121
|
+
The `codec` option sets another wire format for publishing and consuming (one transport is one
|
|
122
|
+
subscription):
|
|
123
|
+
|
|
124
|
+
```ts
|
|
125
|
+
import {
|
|
126
|
+
KafkaTransport,
|
|
127
|
+
cloudEventsStructured,
|
|
128
|
+
encodeEnvelope,
|
|
129
|
+
fromHeader,
|
|
130
|
+
jsonEnvelope,
|
|
131
|
+
plainJsonDecoder,
|
|
132
|
+
type KafkaCodec,
|
|
133
|
+
} from '@relaybus/kafka';
|
|
134
|
+
|
|
135
|
+
// CloudEvents structured mode (`content-type: application/cloudevents+json`); reads binary too.
|
|
136
|
+
new KafkaTransport({ kafka, groupId: 'billing', codec: cloudEventsStructured });
|
|
137
|
+
|
|
138
|
+
// A service envelope: `{ "eventName": ..., "messageId": ..., "data": {...}, "meta": {...} }`.
|
|
139
|
+
const envelope = jsonEnvelope({
|
|
140
|
+
type: { body: 'eventName' },
|
|
141
|
+
id: { body: 'messageId' },
|
|
142
|
+
payload: { body: 'data' },
|
|
143
|
+
correlationId: { body: 'meta.correlationId' },
|
|
144
|
+
occurredAt: { body: 'meta.occurredAt' },
|
|
145
|
+
headers: { body: 'meta.headers' },
|
|
146
|
+
});
|
|
147
|
+
|
|
148
|
+
// The bare JSON payload of a Spring / Go / plain kafkajs producer, the type in a header.
|
|
149
|
+
const plain = jsonEnvelope({
|
|
150
|
+
type: { header: 'event-type' },
|
|
151
|
+
id: 'record-position', // kafka:<topic>:<partition>:<offset>, or { header: 'message-id' }
|
|
152
|
+
payload: 'whole-body',
|
|
153
|
+
});
|
|
154
|
+
|
|
155
|
+
// Your own: binary mode out, plain JSON and CloudEvents in.
|
|
156
|
+
const custom: KafkaCodec = {
|
|
157
|
+
encode: encodeEnvelope,
|
|
158
|
+
decode: plainJsonDecoder({ type: fromHeader('event-type') }),
|
|
159
|
+
};
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
| Codec | Writes | Reads |
|
|
163
|
+
| ----------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
|
164
|
+
| `cloudEventsBinary` | Binary mode, as above. Default, byte for byte unchanged. | Binary mode only, strictly. |
|
|
165
|
+
| `cloudEventsStructured` | One `application/cloudevents+json` document. | Structured and binary mode (`decodeCloudEvent`). |
|
|
166
|
+
| `jsonEnvelope(mapping)` | The mapped JSON body and headers. | The same mapping. A missing type, mapped id or payload is malformed. |
|
|
167
|
+
| `{ encode, decode }` | Whatever `encode` returns. | Whatever `decode` returns; missing fields are filled as for a decoder (table below). |
|
|
168
|
+
|
|
169
|
+
`jsonEnvelope` mappings name a body path (`{ body: 'meta.correlationId' }`, `'messageType.0'`
|
|
170
|
+
indexes an array) or a header (`{ header: 'event-type' }`) for `type`, `id`, `payload` (or
|
|
171
|
+
`'whole-body'`), `key`, `source`, `occurredAt`, `correlationId`, `causationId` and `headers`, plus a
|
|
172
|
+
`contentType`. `type`, `id` and `payload` are required by the types; `id: 'record-position'` uses the
|
|
173
|
+
record position (records relaybus writes carry the event id in a `relaybus-event-id` header, which
|
|
174
|
+
wins on read, so outbox retries stay deduplicated). The record key is always the event key, and when `occurredAt` is not mapped the
|
|
175
|
+
record timestamp carries it. Event headers that are record headers keep their names, so a name the
|
|
176
|
+
mapping uses, `content-type` or `relaybus-event-id`, is refused with `UnencodableEventError`. The
|
|
177
|
+
[interoperability recipe](https://berkcansavur.github.io/relaybus/recipes/interoperating-with-other-services)
|
|
178
|
+
has a full table and a MassTransit example.
|
|
179
|
+
|
|
180
|
+
Decoded payloads go through the event's schema (`defineEvent(type, { schema })`) as relaybus's own
|
|
181
|
+
do; a payload that fails it makes the delivery dead with `EventValidationError`.
|
|
182
|
+
|
|
183
|
+
The decoders below remain available for a custom codec's `decode`, and for the deprecated `decode`
|
|
184
|
+
option:
|
|
185
|
+
|
|
186
|
+
| Decoder | Accepts |
|
|
187
|
+
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
188
|
+
| default (`decodeEnvelope`) | Binary-mode CloudEvents only, strictly as described above. Unchanged. |
|
|
189
|
+
| `decodeCloudEvent` | Binary mode as the default, plus structured mode: a JSON value with `specversion` `1.0`, `id`, `type`, `source`, optional `time` (else the record timestamp), `subject` (else the record key), `data` (the payload; `data_base64` is refused). |
|
|
190
|
+
| `plainJsonDecoder(opts)` | Any JSON value. A record that is a CloudEvent anyway (a `ce_specversion` header or a structured content type) goes through `decodeCloudEvent`, so relaybus and foreign producers can share a topic. |
|
|
191
|
+
| your function | `(record) => ({ id, type, payload, ...optional })`. Missing fields are filled: `source` = topic, `key` = record key, `occurredAt` = record timestamp, `headers` = `{}`. |
|
|
192
|
+
|
|
193
|
+
`plainJsonDecoder` options (each extractor gets `{ topic, partition, offset, key, headers, payload, timestamp }`;
|
|
194
|
+
an extractor returning `undefined` or `''` selects the default):
|
|
195
|
+
|
|
196
|
+
| Option | Default | Meaning |
|
|
197
|
+
| ------------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
|
|
198
|
+
| `type` | required | A fixed event type, or an extractor such as `fromHeader('event-type')` or `fromPayloadField('type')`. |
|
|
199
|
+
| `id` | `kafka:<topic>:<partition>:<offset>` | Inbox dedup id. The default is stable across redeliveries of the record. Prefer a producer id when it may send an event twice. |
|
|
200
|
+
| `key` | the record key | Ordering key. A key that is not UTF-8 (Java `LongSerializer`) is base64-encoded. |
|
|
201
|
+
| `source` | the topic | Fixed text or an extractor. |
|
|
202
|
+
| `occurredAt` | the record timestamp | Extractor returning a `Date`. |
|
|
203
|
+
|
|
204
|
+
Every record header becomes an event header (the last value of a repeated header; a value that is
|
|
205
|
+
not valid UTF-8 is left out, so binary ids can never collide).
|
|
206
|
+
|
|
207
|
+
**Guarantees stay the same.** Decoding runs again on every redelivery and must be deterministic;
|
|
208
|
+
the inbox deduplicates on the id, so a redelivered record is stored and handled once. Under every
|
|
209
|
+
codec, a record that cannot be decoded (not JSON, a tombstone, no type, an invalid type, a missing
|
|
210
|
+
mapped id, an extractor returning a non-string) and anything a custom `decode` throws go to
|
|
211
|
+
`onInvalidMessage` (dead-letter by default)
|
|
212
|
+
with the reason in `relaybus-invalid-reason`; the consumer keeps running. Per-key order is the
|
|
213
|
+
Kafka partition order, as for relaybus producers.
|
|
214
|
+
|
|
215
|
+
**Limits.** The derived id assumes offsets are never reused: deleting and recreating a topic
|
|
216
|
+
restarts its offsets, so records after a recreation could collide with ids already in the inbox
|
|
217
|
+
(within its retention). Two sends of one event by a producer are two offsets; only an id set by the
|
|
218
|
+
producer (`id: fromHeader(...)` or `{ header }` / `{ body }` in a mapping) dedupes them. Avro or
|
|
219
|
+
Protobuf need your own codec, whose functions must stay synchronous.
|
|
220
|
+
|
|
105
221
|
## Partitioning and ordering
|
|
106
222
|
|
|
107
223
|
The key decides the partition. Events with the same key are consumed in publish order; events
|
|
@@ -137,8 +253,12 @@ one key can map to different partitions depending on who wrote it.
|
|
|
137
253
|
- **Malformed or rejected record**: a record that cannot be decoded, or that the bus rejects with
|
|
138
254
|
any `NonRetryableError` (such as `InvalidEnvelopeError` or `EventValidationError`), is handled by
|
|
139
255
|
`onInvalidMessage`. The default copies the untouched record to `<topic>.dead` with its original
|
|
140
|
-
headers plus `relaybus-invalid-reason`, `relaybus-source-topic`, `relaybus-source-partition
|
|
141
|
-
`relaybus-source-offset`,
|
|
256
|
+
headers plus `relaybus-invalid-reason`, `relaybus-source-topic`, `relaybus-source-partition`,
|
|
257
|
+
`relaybus-source-offset`, `relaybus-error-name` (the error class, such as
|
|
258
|
+
`InvalidKafkaMessageError`) and, for relaybus errors, `relaybus-error-code` (such as
|
|
259
|
+
`RELAYBUS_KAFKA_INVALID_MESSAGE`, see [Error codes](https://berkcansavur.github.io/relaybus/concepts/error-codes)),
|
|
260
|
+
using the same idempotent producer, and commits only after the ack. The log line carries the
|
|
261
|
+
same `errorName=` and `code=` fields.
|
|
142
262
|
A failing callback or dead-letter send is retried like a handler failure, so a record is dropped
|
|
143
263
|
only when you chose `'skip'` or your callback returned. A subscription that consumes its own
|
|
144
264
|
dead-letter topic is rejected as a loop.
|
|
@@ -214,3 +334,28 @@ kafkajs as the supported client.
|
|
|
214
334
|
transport recovered it. Repeated `Restart failed` logs mean the broker or credentials are the problem.
|
|
215
335
|
- `PublishTimeoutError`: no broker ack within `publishTimeout`; the outbox retries.
|
|
216
336
|
- Events appear twice in logs but handlers run once: expected, the inbox deduplicates on the id.
|
|
337
|
+
|
|
338
|
+
## When to use @relaybus/kafka, and when not to
|
|
339
|
+
|
|
340
|
+
Use it to send events between services through Kafka without a dual write between your database and the broker. It is a transport, not a store: it needs `@relaybus/core` and a store, and `@confluentinc/kafka-javascript` support is experimental and untested.
|
|
341
|
+
|
|
342
|
+
Documentation: <https://berkcansavur.github.io/relaybus/packages/kafka>
|
|
343
|
+
|
|
344
|
+
### relaybus vs alternatives
|
|
345
|
+
|
|
346
|
+
<!-- relaybus:snippet:comparison -->
|
|
347
|
+
|
|
348
|
+
| | What it is | Survives a restart | Written in your database transaction | Choose it when |
|
|
349
|
+
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
|
350
|
+
| **relaybus** | Event bus built on the transactional outbox: PostgreSQL and MongoDB stores, Kafka and RabbitMQ transports, per-handler deliveries | Yes, events live in your database | Yes | Events must never be lost or phantom, and handlers need retries, dead letters and ordering keys |
|
|
351
|
+
| [`@nestjs/event-emitter`](https://docs.nestjs.com/techniques/events) | In-process event emitter built on `eventemitter2` | No persistence is documented | No | Simple in-process notifications where losing an event on restart is acceptable |
|
|
352
|
+
| [`@nestjs/cqrs`](https://docs.nestjs.com/recipes/cqrs) | CommandBus, QueryBus, EventBus and sagas; the docs say sagas and event handlers run in memory | No built-in persistence | No | You want CQRS structure; it can sit on top of an outbox |
|
|
353
|
+
| [`@nestjs/outbox`](https://github.com/nestjs/outbox) | Transactional outbox module from the NestJS organisation: PostgreSQL, MySQL and in-memory stores, relay with leases and retries, dead-letter table, inbox | Yes with the database stores | Yes | You need MySQL, or a NestJS-organisation module; compare features with relaybus before choosing |
|
|
354
|
+
| [BullMQ](https://docs.bullmq.io/) | Job queue on Redis with retries, delayed jobs and flows; at-least-once in the worst case | Yes, in Redis | No, jobs are stored in Redis and cannot join a SQL or MongoDB transaction | Your stack is Redis-based and background jobs matter more than event atomicity |
|
|
355
|
+
| [pg-boss](https://github.com/timgit/pg-boss) | Job queue on PostgreSQL using `SKIP LOCKED`, with retries, dead letter queues, scheduling and pub/sub; jobs can be created in your own transaction | Yes, in PostgreSQL | Yes, through its database adapters | You want a PostgreSQL job queue with scheduling; relaybus is an event bus with per-handler deliveries and transports |
|
|
356
|
+
| Kafka client directly | A distributed log; the idempotent producer and transactions give exactly-once within Kafka | Yes, in Kafka | No, producing to Kafka is a second write | You consume and produce only inside Kafka; add an outbox when a database write must be announced |
|
|
357
|
+
|
|
358
|
+
This table summarises the vendors' own documentation as of October 2026; check their docs for current details.
|
|
359
|
+
Full comparison and sources: <https://berkcansavur.github.io/relaybus/comparison>.
|
|
360
|
+
|
|
361
|
+
<!-- relaybus:end -->
|