@relaybus/mongodb 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +251 -88
- package/dist/index.cjs +1957 -510
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +159 -20
- package/dist/index.d.ts +159 -20
- package/dist/index.js +1958 -511
- package/dist/index.js.map +1 -1
- package/package.json +22 -9
package/README.md
CHANGED
|
@@ -1,7 +1,10 @@
|
|
|
1
1
|
# @relaybus/mongodb
|
|
2
2
|
|
|
3
|
+
@relaybus/mongodb is the MongoDB event store of relaybus, a reliable, storage-agnostic event bus for NestJS and Node.js: it writes the transactional outbox in your own multi-document transaction, with the MongoDB driver or Mongoose sessions.
|
|
4
|
+
|
|
3
5
|
MongoDB event store for [relaybus](https://github.com/berkcansavur/relaybus): a transactional
|
|
4
|
-
outbox on multi-document transactions, with
|
|
6
|
+
outbox on multi-document transactions, with lane leases, batched claims and completions, and
|
|
7
|
+
per-key ordering.
|
|
5
8
|
|
|
6
9
|
Events are written in the **same MongoDB transaction** as your business data, so a committed
|
|
7
10
|
change always has its event and a rolled-back change never leaks one. Delivery is at-least-once;
|
|
@@ -92,29 +95,34 @@ The projection write and the delivery's completion commit together or not at all
|
|
|
92
95
|
|
|
93
96
|
## Options
|
|
94
97
|
|
|
95
|
-
| Option | Default | Meaning
|
|
96
|
-
| -------------------- | ----------------------------------------------- |
|
|
97
|
-
| `client` | required | A connected `MongoClient`. The store never closes it.
|
|
98
|
-
| `database` | the client's default database | Database for the collections. 1-63 characters of letters, digits, `_`, `-`.
|
|
99
|
-
| `collectionPrefix` | `'relaybus_'` | Collections are `<prefix>events`, `<prefix>deliveries`, `<prefix>key_sequences`.
|
|
100
|
-
| `
|
|
101
|
-
| `
|
|
102
|
-
| `
|
|
98
|
+
| Option | Default | Meaning |
|
|
99
|
+
| -------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
|
100
|
+
| `client` | required | A connected `MongoClient`. The store never closes it. |
|
|
101
|
+
| `database` | the client's default database | Database for the collections. 1-63 characters of letters, digits, `_`, `-`. |
|
|
102
|
+
| `collectionPrefix` | `'relaybus_'` | Collections are `<prefix>events`, `<prefix>deliveries`, `<prefix>key_sequences`, `<prefix>lanes`. |
|
|
103
|
+
| `completionHistory` | `false` | Keep an append-only record of completed deliveries (`<prefix>completed_deliveries`). Same value in every process. |
|
|
104
|
+
| `transactionOptions` | snapshot read concern, `w: 'majority'`, primary | Passed to `withTransaction`. |
|
|
105
|
+
| `purgeBatchSize` | `500` | Documents deleted per purge step. |
|
|
106
|
+
| `logger` | `console` | Receives warnings about best-effort steps that failed (parking). |
|
|
107
|
+
| `generateIdentifier` | - | Deprecated and ignored: a lease token is `<laneId>:<epoch>:<claims>`. |
|
|
103
108
|
|
|
104
|
-
Non-transactional writes (claims,
|
|
105
|
-
|
|
106
|
-
never reads a stale secondary.
|
|
109
|
+
Non-transactional writes (claims, renewals) use write concern `majority` and primary reads, whatever the client
|
|
110
|
+
default is, so a lease survives a failover and the ordering check never reads a stale secondary.
|
|
107
111
|
|
|
108
112
|
## How it works
|
|
109
113
|
|
|
110
|
-
**Collections.** `events` (`_id` = event id), `deliveries` (one per event and handler),
|
|
111
|
-
`key_sequences` (one counter document per ordering key)
|
|
114
|
+
**Collections.** `events` (`_id` = event id), `deliveries` (one per unfinished event and handler),
|
|
115
|
+
`key_sequences` (one counter document per ordering key), `lanes` (256 lane documents), and
|
|
116
|
+
`completed_deliveries` with `completionHistory`.
|
|
112
117
|
|
|
113
118
|
**Idempotent append.** Event ids are deduplicated by reading what exists inside the transaction
|
|
114
119
|
and writing with upserts. A plain insert would raise a duplicate-key error (11000) when two
|
|
115
120
|
transactions race on the same id, and a duplicate-key error aborts the whole transaction without
|
|
116
|
-
a retry label. An upsert race surfaces as a retryable write conflict instead.
|
|
117
|
-
|
|
121
|
+
a retry label. An upsert race surfaces as a retryable write conflict instead. Every event lists the
|
|
122
|
+
handlers it ever had a delivery for (`handlers`), completed ones included: a redelivered broker
|
|
123
|
+
message or a publish retry adds deliveries only for handlers the event does not list yet, so a
|
|
124
|
+
handler that already completed never runs twice although its delivery was deleted. The unique index
|
|
125
|
+
on `(eventId, handler)` is a backstop; the code never relies on catching it.
|
|
118
126
|
|
|
119
127
|
**Per-key sequence equals commit order.** Appending a keyed event runs
|
|
120
128
|
`findOneAndUpdate({ _id: key }, { $inc: { value: 1 } }, { upsert: true })` on the counter inside
|
|
@@ -123,57 +131,135 @@ appender of that key hit a write conflict; the loser restarts after the winner c
|
|
|
123
131
|
the next value. Two transactions can never both hold sequence `n`, and a lower sequence is always
|
|
124
132
|
committed before a higher one, so ordering by sequence is ordering by commit.
|
|
125
133
|
|
|
126
|
-
**
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
134
|
+
**Lane leases.** Every delivery belongs to one of 256 lanes, `laneId`: the 32-bit FNV-1a hash of
|
|
135
|
+
the UTF-8 ordering key, or of the event id when there is none, modulo 256. All deliveries of a key
|
|
136
|
+
share a lane. Workers lease lanes, not deliveries. A lane document holds the owner, the lease
|
|
137
|
+
expiry, an `epoch` and `inFlight`, the ids of the deliveries the owner claimed. A delivery runs while
|
|
138
|
+
its document exists as `pending` and its lane lists it; `processing` is derived, never stored.
|
|
139
|
+
A claim:
|
|
140
|
+
|
|
141
|
+
1. claims from the lanes the worker already holds under a live lease;
|
|
142
|
+
2. if it still needs more, takes free lanes (no owner, or an expired lease) that have due deliveries
|
|
143
|
+
of its handlers: 4, then 8, 16, ... lanes per round, the lane with the oldest due delivery first,
|
|
144
|
+
so a worker holds only as many lanes as its batch needs. Taking a lane is one conditional
|
|
145
|
+
`findOneAndUpdate` that increments the epoch; a lane another claimer took first is skipped, never
|
|
146
|
+
waited for;
|
|
147
|
+
3. reads the due, ready deliveries of those lanes in `(dueAt, _id)` order, leaving out the ones it
|
|
148
|
+
already runs, applies the per-key rules below, and stops at `limit`;
|
|
149
|
+
4. records the batch with one conditional write per lane, all in one `bulkWrite` (`inFlight` gains
|
|
150
|
+
the new ids and loses the ids whose documents are gone; the lease never shrinks), and hands back
|
|
151
|
+
the lanes that yielded nothing and run nothing.
|
|
152
|
+
|
|
153
|
+
A claim of an unkeyed batch therefore costs a small, fixed number of round trips and writes **no
|
|
154
|
+
delivery document**. A completion deletes the delivery documents and writes no lane document.
|
|
155
|
+
Delivery documents are inserted once and deleted once, and no indexed field of theirs changes on
|
|
156
|
+
the normal path.
|
|
157
|
+
|
|
158
|
+
Lanes are held per store instance as well as per `leaseOwner`: a process restarted with a fixed
|
|
159
|
+
owner name does not adopt the lanes of its dead predecessor, it takes them over once their leases
|
|
160
|
+
expired, like any other worker. A claim reads the events of its batch before recording it, and a
|
|
161
|
+
pick that a dead letter requeued during the claim now precedes is given back right after recording,
|
|
162
|
+
without an attempt charged; parking is best effort and never fails a claim (failures go to the
|
|
163
|
+
`logger` option, default `console`).
|
|
164
|
+
|
|
165
|
+
**Lane exclusivity.** While a worker holds a lane under a live lease, other workers do not see the
|
|
166
|
+
pending deliveries of that lane, even idle ones. A worker that dies delays its lanes by at most the
|
|
167
|
+
lease duration; a worker that stops gracefully hands them back at once (`relinquish`). Lanes are
|
|
168
|
+
chosen per claim, so a backlog spreads over workers; a few hot keys live in few lanes and therefore
|
|
169
|
+
run on few workers at a time.
|
|
170
|
+
|
|
171
|
+
**Per-key order.** A keyed delivery is claimed only when no earlier delivery (lower sequence) of its
|
|
172
|
+
handler and key is pending (a running one is still a pending document) and no other delivery of them
|
|
173
|
+
runs. Because all deliveries of a key share one lane and only the lane owner claims from it, the
|
|
174
|
+
running ones are exactly the ones listed by the owner's lanes, which the claim loads once. No
|
|
175
|
+
delivery is locked or written to check this. One claim call never takes two deliveries of the same
|
|
176
|
+
key, and what it learns about a key answers every later delivery of that key without another
|
|
177
|
+
query. The scan continues past blocked deliveries until `limit` is reached, so a hot blocked key
|
|
178
|
+
cannot starve unrelated work.
|
|
152
179
|
|
|
153
180
|
**Parked backlogs.** A key whose head is slow can accumulate thousands of deliveries that are all
|
|
154
181
|
due and all blocked. Scanning them on every claim would cost time proportional to the backlog.
|
|
155
182
|
When one claim finds 32 or more deliveries of a key waiting behind an unfinished predecessor,
|
|
156
|
-
it moves them to the `parked` lane, which the claim index leaves out.
|
|
183
|
+
it moves them to the `parked` claim lane (the `lane` field), which the claim index leaves out. A
|
|
184
|
+
claim that hands a lane back also parks the waiting deliveries it found there, however few: they
|
|
185
|
+
are all the due work of that lane (typically followers of a head whose retry is not due yet), and
|
|
186
|
+
leaving them ready would make every later claim take the lane for nothing. At the start of every
|
|
157
187
|
claim, the lowest parked delivery of each key with no unfinished predecessor is moved back to
|
|
158
|
-
`ready
|
|
188
|
+
`ready` (one aggregation per handler, a distinct scan of the small parked index, then one
|
|
189
|
+
predecessor lookup per parked key, in parallel, and one update). This is deliberately level-triggered: it looks at the current state, never at an event,
|
|
159
190
|
so a crash while parking, or a predecessor finishing while its followers were being parked, can
|
|
160
|
-
delay a key until the next claim but never strand it.
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
simultaneously backlogged keys in the hundreds, not millions.
|
|
191
|
+
delay a key until the next claim but never strand it. Parked deliveries keep their `dueAt`, so
|
|
192
|
+
`statistics().oldestDueAt` still counts them. The per-claim cost of the move-back step is one
|
|
193
|
+
index probe and one parallel lookup per key that currently has parked deliveries, so keep the number of such
|
|
194
|
+
simultaneously backlogged keys in the hundreds, not millions. Parking and unparking write delivery
|
|
195
|
+
documents; they are the slow path for blocked keys only.
|
|
166
196
|
|
|
167
197
|
`requeue` is an operator action and makes a dead delivery pending again even if later deliveries
|
|
168
198
|
of its key already ran; it blocks the ones that have not been claimed yet, and it cannot start
|
|
169
199
|
while a later delivery of its key is running.
|
|
170
200
|
|
|
171
|
-
**Fencing.**
|
|
172
|
-
|
|
173
|
-
lease
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
201
|
+
**Fencing.** A lease token is `<laneId>:<epoch>:<claims>`. `claims` is the delivery's claim
|
|
202
|
+
generation: it grows whenever a claim of the delivery ends (retry, release, burial, lost lease), so
|
|
203
|
+
the lease of an earlier claim never matches a later claim of the same delivery, even by the same
|
|
204
|
+
owner in the same lane epoch. `complete`, `completeMany`, `reschedule`, `release`, `bury` and
|
|
205
|
+
`extendLease` apply only while the lane still has that epoch and lists the delivery, and the delivery
|
|
206
|
+
still has that generation (a missing generation, as in documents of 0.1.0, counts as 0); a worker
|
|
207
|
+
that lost its lane changes nothing. Claims of one owner on one store instance run one after the
|
|
208
|
+
other, so overlapping claims (`bus.runOnce()` next to a running dispatcher) never read the same
|
|
209
|
+
in-flight list. Renewals check the claim generation and extend the lane in one transaction, so a
|
|
210
|
+
stale lease never extends the lane of a newer claim. A claim that fails at any point, even after
|
|
211
|
+
its record reached the database without the reply reaching the process, reconciles the owner's
|
|
212
|
+
lanes on this instance in one transaction: a listed delivery stays listed only if this instance
|
|
213
|
+
handed out its current claim (current lane epoch and claim generation, exact lease token) and has
|
|
214
|
+
not seen it settled; every other one is taken out and its claim generation moves on, so no old
|
|
215
|
+
lease can match a later claim of it. A worker taking such a lane over meanwhile conflicts with the
|
|
216
|
+
reconciliation, which then leaves its claims alone. A token this store cannot have issued is
|
|
217
|
+
refused without a database call. An expired lease nobody took over is still honoured.
|
|
218
|
+
|
|
219
|
+
**Completion and takeover.** A worker that takes a lane over first writes every delivery the
|
|
220
|
+
previous owner still had in flight (it charges the lost attempt, see below) and only then reads
|
|
221
|
+
what to claim. A completion never writes a lane document, so concurrent completions of one lane
|
|
222
|
+
never conflict with each other; it relies on that charge instead:
|
|
223
|
+
|
|
224
|
+
- `complete(lease, now, session)` (transactional handlers) and every completion when
|
|
225
|
+
`completionHistory` is on run in a transaction that reads the lanes and deletes the deliveries
|
|
226
|
+
(and inserts the history records). The completion and a takeover write the same delivery
|
|
227
|
+
document, so whichever commits second either hits a write conflict and is retried against the
|
|
228
|
+
new state (seeing the new epoch, or the delivery gone), or waits for the first to finish.
|
|
229
|
+
- Otherwise `complete` and `completeMany` need no transaction: they read the deliveries' claim
|
|
230
|
+
generations, then the lanes, and delete only the deliveries that still have the lease's
|
|
231
|
+
generation. A takeover before the lane read shows up in it as a new epoch; a takeover, retry or
|
|
232
|
+
release after it changed the generation, so the delete matches nothing. A batch of any size from
|
|
233
|
+
any number of lanes costs four commands (the fourth records the completion time on the events).
|
|
234
|
+
When fewer deliveries were deleted than leases were fenced (a claim ended or changed hands
|
|
235
|
+
meanwhile), the call reports no lease as completed: a missing document cannot be told apart
|
|
236
|
+
from one the new claim completed, so success is reported only when it is proven.
|
|
237
|
+
|
|
238
|
+
A completion and a takeover can never both succeed, which is what makes a transactional handler's
|
|
239
|
+
writes effectively-once. Two concurrent completions of the same lease without a transaction may both
|
|
240
|
+
report success; the dispatcher never issues them.
|
|
241
|
+
|
|
242
|
+
**Attempts.** `attempts` in a delivery document counts attempts that **ended**: `reschedule` and
|
|
243
|
+
`bury` add the failed one, `release` (graceful shutdown) adds nothing, and the claim result reports
|
|
244
|
+
`attempts + 1`. When a lane is taken over, every delivery the previous owner still had in flight is
|
|
245
|
+
charged one attempt, so a worker that keeps crashing on a poison message still runs it into the
|
|
246
|
+
retry limit and the dead letters. The take moves the previous owner's in-flight list to the lane's
|
|
247
|
+
`lost` list in the same write; `lost` is charged before anything is read from the lane and cleared
|
|
248
|
+
afterwards, and whoever holds the lane next charges a `lost` list it finds first. A claim that fails
|
|
249
|
+
anywhere after the take therefore loses no charge and leaves nothing listed as running. Charging
|
|
250
|
+
and clearing are separate writes, not a transaction, but each charge is recorded on the delivery
|
|
251
|
+
(`lostAt`: the lane epoch of the last lost claim charged, which only grows), so charging the same
|
|
252
|
+
list again after a failure between them, or a delayed charge of an older lost claim, charges
|
|
253
|
+
nothing twice. The charge also moves the claim generation on.
|
|
254
|
+
|
|
255
|
+
**Completion history.** Completed deliveries are deleted. With `completionHistory: true` every
|
|
256
|
+
completion also inserts a record into `completed_deliveries` in the same transaction. Records are
|
|
257
|
+
listed by `listDeliveries` as `completed` (all filters apply), counted by `statistics().completed`
|
|
258
|
+
and removed by `purge({ completedBefore })`. Without it, `listDeliveries({ status: 'completed' })` is
|
|
259
|
+
empty and `statistics().completed` is 0 (except for `completed` documents left by 0.1.0).
|
|
260
|
+
|
|
261
|
+
**Time.** The store uses only the `now` it is given, never the server clock. Lease expiry is
|
|
262
|
+
compared with the `now` of the claimer, so keep worker clocks synchronized.
|
|
177
263
|
|
|
178
264
|
**Payloads.** Payload and headers are stored as the JSON-normalized value: `Date` becomes an ISO
|
|
179
265
|
string, `undefined` members disappear, as in every other relaybus store. Keys containing `.` or
|
|
@@ -205,40 +291,89 @@ core does not renew the lease of a transactional handler while it runs (its comp
|
|
|
205
291
|
conflict with a renewal), so the lease must outlast the handler: keep transactional handlers
|
|
206
292
|
short, as the core documentation recommends.
|
|
207
293
|
|
|
294
|
+
### Transaction time budget
|
|
295
|
+
|
|
296
|
+
`bus.transaction` passes the bus `transactionTimeout` (default `'5s'`) to the store, which bounds the commit on the
|
|
297
|
+
server with `maxCommitTimeMS`. A transaction that ran past the budget is aborted before its commit, with all its writes,
|
|
298
|
+
and the caller gets a `TransactionTimeoutError`. The driver's `timeoutMS` is not set on the transaction, because it would
|
|
299
|
+
make operations that pass their own `timeoutMS` with the session throw. The store's own transactions are not affected.
|
|
300
|
+
|
|
208
301
|
## Indexes
|
|
209
302
|
|
|
210
303
|
Created idempotently by `setup()` (also run by `bus.start()` unless `setupStore: false`):
|
|
211
304
|
|
|
212
|
-
| Collection
|
|
213
|
-
|
|
|
214
|
-
| `deliveries`
|
|
215
|
-
| `deliveries`
|
|
216
|
-
| `deliveries`
|
|
217
|
-
| `deliveries`
|
|
218
|
-
| `deliveries`
|
|
219
|
-
| `deliveries`
|
|
220
|
-
| `deliveries`
|
|
221
|
-
| `deliveries`
|
|
222
|
-
| `deliveries`
|
|
223
|
-
| `deliveries`
|
|
224
|
-
| `
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
the
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
305
|
+
| Collection | Index | Used by |
|
|
306
|
+
| ---------------------- | ----------------------------------------------------- | ---------------------------------------------- |
|
|
307
|
+
| `deliveries` | `(eventId, handler)` unique | idempotent append, lookup by event |
|
|
308
|
+
| `deliveries` | `(laneId, dueAt, _id, handler)` partial | claim, free-lane discovery (`ready` lane only) |
|
|
309
|
+
| `deliveries` | `(handler, key, sequence)` partial | predecessor check of keyed deliveries |
|
|
310
|
+
| `deliveries` | `(handler, key, sequence)` partial | lowest parked delivery of each key (`parked`) |
|
|
311
|
+
| `deliveries` | `(dueAt)` partial | `statistics().oldestDueAt` |
|
|
312
|
+
| `deliveries` | `(completedAt)` partial | purge of `completed` documents left by 0.1.0 |
|
|
313
|
+
| `deliveries` | `(status, createdAt)` | status counts, listing by status |
|
|
314
|
+
| `deliveries` | `(status, updatedAt)` | purge of dead deliveries |
|
|
315
|
+
| `deliveries` | `(handler, createdAt)` | listing by handler |
|
|
316
|
+
| `deliveries` | `(eventType, createdAt)` | listing by event type |
|
|
317
|
+
| `events` | `(createdAt, _id)` | purge of events nothing references |
|
|
318
|
+
| `completed_deliveries` | `(completedAt, _id)`, `(eventId)`, `(createdAt, _id)` | purge, event references, listing |
|
|
319
|
+
|
|
320
|
+
The claim index ends with `handler` so that a query over many lanes stays a merge of one index scan
|
|
321
|
+
per lane (the planner merges at most 200), and a claim queries at most 128 lanes at once. The lane
|
|
322
|
+
collection has 256 small documents and no secondary index.
|
|
323
|
+
|
|
324
|
+
`setup()` also creates the collections explicitly, because collections created implicitly inside
|
|
325
|
+
concurrent transactions conflict with each other, and seeds the lane documents. If you manage the
|
|
326
|
+
schema yourself, run `setup()` once from a deployment script instead.
|
|
327
|
+
|
|
328
|
+
Partial indexes contain only unfinished (or legacy completed) deliveries, so they stay small. If you
|
|
329
|
+
never use `listDeliveries` filters, the three listing indexes can be dropped at the cost of slower
|
|
330
|
+
operator queries.
|
|
331
|
+
|
|
332
|
+
## Upgrading from 0.1.0
|
|
333
|
+
|
|
334
|
+
The step-by-step guide, including `completionHistory` and PgBouncer settings, is [Upgrading from 0.1 to 0.2](https://berkcansavur.github.io/relaybus/upgrading).
|
|
335
|
+
|
|
336
|
+
`setup()` upgrades a 0.1.0 database in place, idempotently, and is safe to run again: `processing`
|
|
337
|
+
deliveries become `pending` and due at once (their attempt stays counted, as a lost lease would count
|
|
338
|
+
it), the per-delivery lease fields are removed, every delivery gets its `laneId` and every event its
|
|
339
|
+
`handlers` and `createdAt` (from its deliveries, in batches of 1,000), the `running` index and the
|
|
340
|
+
0.1.0 `claim` index are dropped, and the lane documents are created. The lane documents mark the
|
|
341
|
+
upgrade as done, so later runs skip it. `completed` documents left by 0.1.0 stay until a purge with
|
|
342
|
+
`completedBefore` removes them. Stop all 0.1.0 workers before the first `setup()`: they cannot work
|
|
343
|
+
with the upgraded documents.
|
|
344
|
+
|
|
345
|
+
## Tuning
|
|
346
|
+
|
|
347
|
+
- **Batch size.** A claim takes 4, 8, 16, ... lanes until it has `limit` deliveries, so a worker's
|
|
348
|
+
lane count follows its batch size. Unkeyed deliveries spread over all 256 lanes; very small batches
|
|
349
|
+
with many workers keep lanes changing hands often, very large ones let one worker hold most lanes.
|
|
350
|
+
- **Lease duration.** A lane whose owner died is unavailable for at most the lease duration. Keep it
|
|
351
|
+
well above the handler run time (the dispatcher renews it while handlers run).
|
|
352
|
+
- **Completion batching.** The dispatcher completes deliveries that finish close together with one
|
|
353
|
+
`completeMany`: three commands per batch (one transaction with `completionHistory`).
|
|
354
|
+
- **Transactional handlers** commit their writes with the deletion of the delivery. They conflict
|
|
355
|
+
only with a takeover of their own delivery, not with each other.
|
|
356
|
+
- **Document size.** New events leave out `key`, `sequence`, `correlationId`, `causationId` when they
|
|
357
|
+
are null and `revision` while it is 0, and new deliveries leave out `updatedAt` until it differs from
|
|
358
|
+
`createdAt`: every byte of a new document is written to the oplog too. Documents of earlier versions
|
|
359
|
+
with these fields are read the same way.
|
|
233
360
|
|
|
234
361
|
## Retention
|
|
235
362
|
|
|
236
|
-
`bus.purgeExpired()` (run on a schedule by the bus) calls `store.purge`, which deletes
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
363
|
+
`bus.purgeExpired()` (run on a schedule by the bus) calls `store.purge`, which deletes, in batches of
|
|
364
|
+
`purgeBatchSize`, dead deliveries (`deadBefore`), completion history records and legacy `completed`
|
|
365
|
+
documents (`completedBefore`), each batch together with the events nothing references any more **in
|
|
366
|
+
one transaction**, so a crash can never leave a delivery without its event. Then it deletes events
|
|
367
|
+
nothing references. Every completion records its time on the event (`completedAt`, the latest
|
|
368
|
+
one), and such an event is kept as deduplication record until `completedBefore` passes that time,
|
|
369
|
+
however old the event is; `deadBefore` alone never removes it. Events that never had a completion
|
|
370
|
+
(no subscribers, or only dead letters) go once they were created before the earliest given cutoff. An append that adds a delivery to an event while
|
|
371
|
+
it is being purged writes the event, conflicts with the purge, and one of them restarts, so a
|
|
372
|
+
delivery never loses its event.
|
|
373
|
+
|
|
374
|
+
An event deleted by purge takes its `handlers` list with it: a broker redelivery or publish retry
|
|
375
|
+
of that event id after the purge is appended as a new event and runs its handlers again (the same
|
|
376
|
+
holds for the PostgreSQL store). Keep completed retention longer than any redelivery window.
|
|
242
377
|
|
|
243
378
|
One counter document per distinct ordering key stays in `key_sequences`. They are tiny; if your
|
|
244
379
|
keys are unbounded (for example one per request), clean up counters of long-finished keys yourself.
|
|
@@ -254,12 +389,40 @@ keys are unbounded (for example one per request), clean up counters of long-fini
|
|
|
254
389
|
publishes with `publishAll`.
|
|
255
390
|
- **Deliveries stay pending**: check `statistics().oldestDueAt`, and that the process that publishes
|
|
256
391
|
also registered the handler (deliveries are created from the publisher's registrations).
|
|
257
|
-
- **
|
|
258
|
-
|
|
259
|
-
|
|
392
|
+
- **Deliveries wait although workers are idle**: another worker holds their lane under a live lease
|
|
393
|
+
(look at `<prefix>lanes`: `owner`, `leaseExpiresAt`); it is released when that worker finishes,
|
|
394
|
+
stops, or the lease expires.
|
|
395
|
+
- **A key is stuck**: a pending, delayed, running or requeued delivery earlier in the same key
|
|
396
|
+
blocks later ones, and so does any running delivery of the key; inspect it with
|
|
397
|
+
`listDeliveries({ handler })` and `bus.requeue` or bury it.
|
|
260
398
|
- **Slow claims with a very large backlog of one key**: backlogs of 32 or more waiting deliveries
|
|
261
|
-
are parked automatically; check that the `claim`, `ordering
|
|
262
|
-
|
|
399
|
+
are parked automatically; check that the `claim`, `ordering` and `parked` indexes exist
|
|
400
|
+
(`setup()`). Deliveries with `lane: 'parked'` are expected while their key is blocked.
|
|
263
401
|
- **`MongoEventStore.append needs a ClientSession inside a started transaction`**: the session
|
|
264
402
|
passed to `publish({ transaction })` or `append` has no transaction. Start one with
|
|
265
403
|
`session.withTransaction` or `session.startTransaction()`.
|
|
404
|
+
|
|
405
|
+
## When to use @relaybus/mongodb, and when not to
|
|
406
|
+
|
|
407
|
+
Use it when your data lives in MongoDB 6 or newer running as a replica set or sharded cluster. MongoDB transactions need a replica set: a standalone `mongod` is rejected.
|
|
408
|
+
|
|
409
|
+
Documentation: <https://berkcansavur.github.io/relaybus/packages/mongodb>
|
|
410
|
+
|
|
411
|
+
### relaybus vs alternatives
|
|
412
|
+
|
|
413
|
+
<!-- relaybus:snippet:comparison -->
|
|
414
|
+
|
|
415
|
+
| | What it is | Survives a restart | Written in your database transaction | Choose it when |
|
|
416
|
+
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
|
417
|
+
| **relaybus** | Event bus built on the transactional outbox: PostgreSQL and MongoDB stores, Kafka and RabbitMQ transports, per-handler deliveries | Yes, events live in your database | Yes | Events must never be lost or phantom, and handlers need retries, dead letters and ordering keys |
|
|
418
|
+
| [`@nestjs/event-emitter`](https://docs.nestjs.com/techniques/events) | In-process event emitter built on `eventemitter2` | No persistence is documented | No | Simple in-process notifications where losing an event on restart is acceptable |
|
|
419
|
+
| [`@nestjs/cqrs`](https://docs.nestjs.com/recipes/cqrs) | CommandBus, QueryBus, EventBus and sagas; the docs say sagas and event handlers run in memory | No built-in persistence | No | You want CQRS structure; it can sit on top of an outbox |
|
|
420
|
+
| [`@nestjs/outbox`](https://github.com/nestjs/outbox) | Transactional outbox module from the NestJS organisation: PostgreSQL, MySQL and in-memory stores, relay with leases and retries, dead-letter table, inbox | Yes with the database stores | Yes | You need MySQL, or a NestJS-organisation module; compare features with relaybus before choosing |
|
|
421
|
+
| [BullMQ](https://docs.bullmq.io/) | Job queue on Redis with retries, delayed jobs and flows; at-least-once in the worst case | Yes, in Redis | No, jobs are stored in Redis and cannot join a SQL or MongoDB transaction | Your stack is Redis-based and background jobs matter more than event atomicity |
|
|
422
|
+
| [pg-boss](https://github.com/timgit/pg-boss) | Job queue on PostgreSQL using `SKIP LOCKED`, with retries, dead letter queues, scheduling and pub/sub; jobs can be created in your own transaction | Yes, in PostgreSQL | Yes, through its database adapters | You want a PostgreSQL job queue with scheduling; relaybus is an event bus with per-handler deliveries and transports |
|
|
423
|
+
| Kafka client directly | A distributed log; the idempotent producer and transactions give exactly-once within Kafka | Yes, in Kafka | No, producing to Kafka is a second write | You consume and produce only inside Kafka; add an outbox when a database write must be announced |
|
|
424
|
+
|
|
425
|
+
This table summarises the vendors' own documentation as of October 2026; check their docs for current details.
|
|
426
|
+
Full comparison and sources: <https://berkcansavur.github.io/relaybus/comparison>.
|
|
427
|
+
|
|
428
|
+
<!-- relaybus:end -->
|