queen-mq 2.0.4 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +135 -2
- package/client-v2/Queen.js +93 -3
- package/client-v2/README.md +66 -1
- package/client-v2/builders/TransactionBuilder.js +105 -17
- package/client-v2/http/HttpClient.js +67 -3
- package/client-v2/http/popFrames.js +211 -0
- package/client-v2/http/pushBinary.js +367 -0
- package/client-v2/http/zstd.js +41 -0
- package/client-v2/index.js +5 -0
- package/client-v2/kv/Kv.js +41 -0
- package/client-v2/locks/Locks.js +544 -0
- package/client-v2/utils/defaults.js +17 -1
- package/package.json +2 -2
- package/test-v2/kv-unit/locksWire.test.js +541 -0
- package/test-v2/locks.js +277 -0
- package/test-v2/run.js +2 -0
- package/test-v2/wire-unit/_wireServer.js +47 -0
- package/test-v2/wire-unit/popFrames.test.js +264 -0
- package/test-v2/wire-unit/pushBinary.test.js +235 -0
- package/test-v2/wire-unit/transactionFrames.test.js +222 -0
package/README.md
CHANGED
|
@@ -229,7 +229,11 @@ const queen = new Queen({
|
|
|
229
229
|
timeoutMillis: 30000,
|
|
230
230
|
retryAttempts: 3,
|
|
231
231
|
loadBalancingStrategy: 'affinity', // or 'round-robin', 'session'
|
|
232
|
-
enableFailover: true
|
|
232
|
+
enableFailover: true,
|
|
233
|
+
// Opt-in wire forms, off by default (see Binary Push, Push Compression and Framed Pop)
|
|
234
|
+
binaryPush: false,
|
|
235
|
+
pushCompression: false,
|
|
236
|
+
framedPop: false
|
|
233
237
|
})
|
|
234
238
|
```
|
|
235
239
|
|
|
@@ -515,6 +519,54 @@ rate, and a broker outage shows up as slow pushes with bounded memory rather
|
|
|
515
519
|
than as messages that quietly disappeared. `close()` flushes with a 30 second
|
|
516
520
|
deadline and logs how many messages were left unsent if it expires.
|
|
517
521
|
|
|
522
|
+
### Binary Push, Push Compression and Framed Pop
|
|
523
|
+
|
|
524
|
+
Three opt-in forms of the push, pop and transaction requests. They change how
|
|
525
|
+
the messages travel between the client and the broker and nothing else: the
|
|
526
|
+
calls, their results, deduplication and the delivery guarantees stay the same.
|
|
527
|
+
All three are off by default, and with them off the client sends and reads the
|
|
528
|
+
JSON every broker knows.
|
|
529
|
+
|
|
530
|
+
```javascript
|
|
531
|
+
const queen = new Queen({
|
|
532
|
+
url: 'http://localhost:6632',
|
|
533
|
+
binaryPush: true, // a push travels as a binary body
|
|
534
|
+
pushCompression: true, // with binaryPush: framed and compressed here
|
|
535
|
+
framedPop: true // a pop returns the stored records, unpacked here
|
|
536
|
+
})
|
|
537
|
+
```
|
|
538
|
+
|
|
539
|
+
| Option | What it does |
|
|
540
|
+
| --- | --- |
|
|
541
|
+
| `binaryPush` | A push, direct or buffered, sends its items as one length-prefixed binary body instead of a JSON document. The answer is the same list, one entry per item. |
|
|
542
|
+
| `pushCompression` | Needs `binaryPush`. The client frames each partition's batch the way the broker stores it, mints the message ids (UUIDv7) and compresses the batch with zstd, once. The broker checks the frames and, when it has nothing to add to them, stores those bytes without compressing them again. A transaction that pushes sends its pushes the same way, with the rest of the bundle as JSON in front of them. |
|
|
543
|
+
| `framedPop` | `pop()` and `consume()` ask for the stored records (`frames=true`) and unpack them here into the same message objects, instead of a JSON answer that the broker renders message by message. |
|
|
544
|
+
|
|
545
|
+
Turn them on when the broker's CPU or the network limits a high message rate:
|
|
546
|
+
the framing, the compression and the unpacking then run in the clients, and the
|
|
547
|
+
batches cross the network compressed. zstd is the one of Node.js (`node:zlib`),
|
|
548
|
+
so there is nothing to install.
|
|
549
|
+
|
|
550
|
+
**What stays JSON**, with no action from you:
|
|
551
|
+
|
|
552
|
+
- A push with a `traceId` on one of its items.
|
|
553
|
+
- A transaction with no push, or with a `traceId` on one of its pushes.
|
|
554
|
+
- A pop the broker cannot answer in frames: a pop of a conflating group, and
|
|
555
|
+
every pop on a broker that runs with an encryption key
|
|
556
|
+
(`QUEEN_ENCRYPTION_KEY`). The client reads the JSON answer as always.
|
|
557
|
+
- Acks, KV, timers, locks and the admin calls. Ephemeral queues have their own
|
|
558
|
+
routes and do not change either.
|
|
559
|
+
|
|
560
|
+
A transaction's `results` come back in the order you added the operations, as
|
|
561
|
+
with JSON. The broker answers a binary transaction with the acks first and the
|
|
562
|
+
pushes after them, and the client puts them back.
|
|
563
|
+
|
|
564
|
+
**Requires broker 2.3.0 or later** for `binaryPush` and `pushCompression`.
|
|
565
|
+
A broker up to 2.2.0 answers 400 to the binary bodies and the client does not
|
|
566
|
+
fall back to JSON, so turn the two on after every broker of the cluster is
|
|
567
|
+
upgraded. `framedPop` is safe against any broker: an older one ignores the
|
|
568
|
+
parameter and answers JSON.
|
|
569
|
+
|
|
518
570
|
### Dead Letter Queue
|
|
519
571
|
|
|
520
572
|
```javascript
|
|
@@ -636,6 +688,49 @@ Otherwise the lanes do not serialise it for you: use `incr`, or carry `expect`.
|
|
|
636
688
|
**`putIfAbsent` plus a TTL is not a distributed lock.** A lock that expires is not revoked: the old
|
|
637
689
|
holder keeps working, it simply no longer has the row. Carry your `version` as `expect` on every
|
|
638
690
|
later write so a lapsed holder fails with `reason: 'version'` instead of overwriting the new one.
|
|
691
|
+
`queen.lock()` below is that, done for you.
|
|
692
|
+
|
|
693
|
+
**`check` asserts a version and writes nothing.** `kv.check(ns, key, { expect })` holds while the
|
|
694
|
+
key is at that version (`expect: 0`: while it is absent). With `required: true` in a transaction
|
|
695
|
+
it gates the commit on a key the transaction does not write.
|
|
696
|
+
|
|
697
|
+
### Locks and semaphores
|
|
698
|
+
|
|
699
|
+
A lock is a lease: one holder at a time, for a lifetime the holder renews, with a token that
|
|
700
|
+
fences a holder that outlived it. `queen.semaphore(name, n, opts)` is the same with `n` permits.
|
|
701
|
+
|
|
702
|
+
```javascript
|
|
703
|
+
const lock = queen.lock('daily-report', { ttl: '30s' })
|
|
704
|
+
if (!(await lock.acquire({ wait: '5s' }))) return // somebody else holds it
|
|
705
|
+
|
|
706
|
+
try {
|
|
707
|
+
await queen.transaction()
|
|
708
|
+
.guard(lock) // commits only while the lock is ours
|
|
709
|
+
.queue('reports').push([{ data: report }])
|
|
710
|
+
.commit()
|
|
711
|
+
} finally {
|
|
712
|
+
await lock.release()
|
|
713
|
+
}
|
|
714
|
+
|
|
715
|
+
// Or: acquire, run, release.
|
|
716
|
+
const { acquired, value } = await queen.lock('sync:crm', { ttl: '1m' }).run(() => sync())
|
|
717
|
+
```
|
|
718
|
+
|
|
719
|
+
**It expires, and nobody tells the holder.** A paused or partitioned process carries on past its
|
|
720
|
+
lifetime while somebody else acquires. `.guard(lock)` is what keeps its work out: the transaction
|
|
721
|
+
rolls back with `reason: 'kv_precondition'` (returned from `commit()`, not thrown) when the lock is
|
|
722
|
+
no longer this handle's. Outside Queen, fence with `lock.token`, which only rises on a lock.
|
|
723
|
+
|
|
724
|
+
**The handle renews in the background**, every third of the lifetime (`autoRenew: false` to do it
|
|
725
|
+
yourself with `lock.renew()`), and says when the lock is gone: `lock.signal` aborts and
|
|
726
|
+
`lock.onLost(fn)` runs. Nothing stops your code: pass `lock.signal` to what the work awaits.
|
|
727
|
+
|
|
728
|
+
**The token changes at every renew.** Read `lock.token` and `lock.guard()` when you use them; do
|
|
729
|
+
not keep a copy across an `await`.
|
|
730
|
+
|
|
731
|
+
`queen.locks` is the wire, with no state kept: `acquire`, `renew`, `release`, `get(name)` (who
|
|
732
|
+
holds it, since when, until when) and `batch` for several locks in one call. `queen.close()`
|
|
733
|
+
releases the locks its handles still hold.
|
|
639
734
|
|
|
640
735
|
### Timers
|
|
641
736
|
|
|
@@ -954,7 +1049,10 @@ await queen.close() // Flush buffers and close connections
|
|
|
954
1049
|
loadBalancingStrategy: 'affinity', // 'affinity' | 'round-robin' | 'session'
|
|
955
1050
|
affinityHashRing: 128,
|
|
956
1051
|
enableFailover: true,
|
|
957
|
-
healthRetryAfterMillis: 5000
|
|
1052
|
+
healthRetryAfterMillis: 5000,
|
|
1053
|
+
binaryPush: false, // a push as a binary body
|
|
1054
|
+
pushCompression: false, // with binaryPush: frames compressed in the client
|
|
1055
|
+
framedPop: false // a pop as the stored records
|
|
958
1056
|
}
|
|
959
1057
|
```
|
|
960
1058
|
|
|
@@ -1095,6 +1193,41 @@ Each consume invocation publishes its own instance into the broker's
|
|
|
1095
1193
|
60-second TTL), plus a final stopped observation. The credential needs KV write
|
|
1096
1194
|
access. Publication is serialized and best effort with a two-second deadline.
|
|
1097
1195
|
|
|
1196
|
+
#### Match supervision to the consumer lifetime
|
|
1197
|
+
|
|
1198
|
+
The instance belongs to one **`consume()` invocation**, not to the `Queen`
|
|
1199
|
+
client, queue, hostname, process or publication group. Reusing the same client,
|
|
1200
|
+
builder and group does not reuse the instance ID. Each invocation starts new
|
|
1201
|
+
counters, publishes its initial state, then reports `stopped` when all its
|
|
1202
|
+
workers exit, including a normal exit caused by `.limit(...)`, `.idleMillis(...)`
|
|
1203
|
+
or cancellation.
|
|
1204
|
+
|
|
1205
|
+
| Application pattern | Supervision setup |
|
|
1206
|
+
| --- | --- |
|
|
1207
|
+
| One persistent `consume()` with several concurrent workers | Enable `.supervision({ group })` on that invocation. |
|
|
1208
|
+
| Several persistent `consume()` calls in one process | Enable reporting on each; expect several instances with the same hostname. Budget their combined concurrency. |
|
|
1209
|
+
| An outer scheduler repeatedly calling `.limit(1).consume(...)` or rotating queues | Leave per-call supervision off when you need the scheduler's lifetime status; publish one observation for the persistent scheduler instead. |
|
|
1210
|
+
|
|
1211
|
+
For a persistent consumer, omit limits that deliberately end the invocation.
|
|
1212
|
+
Pass an `AbortSignal` to `consume(handler, { signal })`; on shutdown, abort it,
|
|
1213
|
+
await the consume promise so active handlers drain, then close the client.
|
|
1214
|
+
Do not remove an existing scheduler's limits just to change the dashboard:
|
|
1215
|
+
those limits may enforce queue fairness and the application's worker budget.
|
|
1216
|
+
|
|
1217
|
+
Supervising short calls is valid when you want to observe those calls. A loop
|
|
1218
|
+
that starts a new call after every message or idle poll will produce many
|
|
1219
|
+
recent `stopped` instances, with `running: 0`, until their 60-second TTL expires.
|
|
1220
|
+
Expired records can remain visible until the broker sweeps them.
|
|
1221
|
+
That is the last state of each finished invocation, not evidence that its
|
|
1222
|
+
process or pod has stopped. Compare instance IDs and application logs.
|
|
1223
|
+
|
|
1224
|
+
The JavaScript API has no option to attach several consume invocations to one
|
|
1225
|
+
persistent reporter or supply their instance ID. A custom scheduler can publish
|
|
1226
|
+
the [consumer status contract](https://queenmq.com/reference/supervisor-status/#reporting-an-application-owned-scheduler)
|
|
1227
|
+
with one ID per scheduler lifetime, stable worker pool names and counters
|
|
1228
|
+
across turns. Keep SDK supervision disabled on its short calls and preserve
|
|
1229
|
+
their ACK/NACK, lease renewal, cancellation and scheduling behavior.
|
|
1230
|
+
|
|
1098
1231
|
The Supervisors page supporting `queen.consumer.status/v1` shows live async
|
|
1099
1232
|
consumer loops, busy handlers, successful/failed handler calls and progress times.
|
|
1100
1233
|
A batch is one handler call; completion does not imply ACK success. No payloads or
|
package/client-v2/Queen.js
CHANGED
|
@@ -10,6 +10,7 @@ import { QueueBuilder } from './builders/QueueBuilder.js'
|
|
|
10
10
|
import { TransactionBuilder } from './builders/TransactionBuilder.js'
|
|
11
11
|
import { TimerBuilder } from './builders/TimerBuilder.js'
|
|
12
12
|
import { Kv } from './kv/Kv.js'
|
|
13
|
+
import { Locks, Lock } from './locks/Locks.js'
|
|
13
14
|
import { Ephemeral } from './ephemeral/Ephemeral.js'
|
|
14
15
|
import { StreamBuilder } from './stream/StreamBuilder.js'
|
|
15
16
|
import { StreamConsumer } from './stream/StreamConsumer.js'
|
|
@@ -93,6 +94,7 @@ export class Queen {
|
|
|
93
94
|
#shutdownHandlers = []
|
|
94
95
|
#admin = null
|
|
95
96
|
#kv = null
|
|
97
|
+
#locks = null
|
|
96
98
|
#ephemeral = null
|
|
97
99
|
// Process-wide kill switch for pop autopilot, read from
|
|
98
100
|
// QUEEN_SDK_POP_AUTOPILOT once here rather than on every pop: it is a
|
|
@@ -177,7 +179,7 @@ export class Queen {
|
|
|
177
179
|
}
|
|
178
180
|
|
|
179
181
|
#createHttpClient() {
|
|
180
|
-
const { urls, timeoutMillis, retryAttempts, retryDelayMillis, loadBalancingStrategy, affinityHashRing, healthRetryAfterMillis, enableFailover, bearerToken, headers, hostHeader, retry429 } = this.#config
|
|
182
|
+
const { urls, timeoutMillis, retryAttempts, retryDelayMillis, loadBalancingStrategy, affinityHashRing, healthRetryAfterMillis, enableFailover, bearerToken, headers, hostHeader, retry429, binaryPush, pushCompression, framedPop } = this.#config
|
|
181
183
|
|
|
182
184
|
if (urls.length === 1) {
|
|
183
185
|
// Single server
|
|
@@ -189,7 +191,10 @@ export class Queen {
|
|
|
189
191
|
bearerToken,
|
|
190
192
|
headers,
|
|
191
193
|
hostHeader,
|
|
192
|
-
retry429
|
|
194
|
+
retry429,
|
|
195
|
+
binaryPush,
|
|
196
|
+
pushCompression,
|
|
197
|
+
framedPop
|
|
193
198
|
})
|
|
194
199
|
}
|
|
195
200
|
|
|
@@ -207,7 +212,10 @@ export class Queen {
|
|
|
207
212
|
bearerToken,
|
|
208
213
|
headers,
|
|
209
214
|
hostHeader,
|
|
210
|
-
retry429
|
|
215
|
+
retry429,
|
|
216
|
+
binaryPush,
|
|
217
|
+
pushCompression,
|
|
218
|
+
framedPop
|
|
211
219
|
})
|
|
212
220
|
}
|
|
213
221
|
|
|
@@ -298,6 +306,76 @@ export class Queen {
|
|
|
298
306
|
return this.#kv
|
|
299
307
|
}
|
|
300
308
|
|
|
309
|
+
// ===========================
|
|
310
|
+
// Locks API Entry Point
|
|
311
|
+
// ===========================
|
|
312
|
+
|
|
313
|
+
/**
|
|
314
|
+
* A lock: one holder at a time, held as a lease, with a fencing token.
|
|
315
|
+
*
|
|
316
|
+
* const lock = queen.lock('daily-report', { ttl: '30s' })
|
|
317
|
+
* if (!(await lock.acquire())) return // somebody else has it
|
|
318
|
+
* try {
|
|
319
|
+
* await queen.transaction().guard(lock)
|
|
320
|
+
* .queue('reports').push([{ data: report }]).commit()
|
|
321
|
+
* } finally {
|
|
322
|
+
* await lock.release()
|
|
323
|
+
* }
|
|
324
|
+
*
|
|
325
|
+
* Creating the handle sends nothing. `acquire()` resolves a boolean,
|
|
326
|
+
* `acquire({ wait: '10s' })` keeps trying, and while it is held the handle
|
|
327
|
+
* renews it in the background and aborts `lock.signal` if it is lost.
|
|
328
|
+
*
|
|
329
|
+
* It is a lease, not a mutex: it expires, and a holder that outlived it
|
|
330
|
+
* keeps running. `.guard(lock)` on a transaction is what makes the WORK
|
|
331
|
+
* exclusive; see `locks/Locks.js` for the whole argument.
|
|
332
|
+
*
|
|
333
|
+
* @param {string} name - no '#', no control characters, at most 256 bytes
|
|
334
|
+
* @param {{ttl?: string, ttlSeconds?: number, owner?: string, autoRenew?: boolean, renewEvery?: string}} opts
|
|
335
|
+
* a lifetime is mandatory: `ttl: '30s'` or `ttlSeconds: 30`
|
|
336
|
+
* @returns {Lock}
|
|
337
|
+
*/
|
|
338
|
+
lock(name, opts = {}) {
|
|
339
|
+
if (opts.limit !== undefined && opts.limit !== 1) {
|
|
340
|
+
throw new Error('lock: a lock has one permit — queen.semaphore(name, limit, opts) is the one with more')
|
|
341
|
+
}
|
|
342
|
+
return new Lock(this.locks, name, opts)
|
|
343
|
+
}
|
|
344
|
+
|
|
345
|
+
/**
|
|
346
|
+
* A semaphore: at most `limit` holders at a time. One handle is ONE permit;
|
|
347
|
+
* make a handle per holder.
|
|
348
|
+
*
|
|
349
|
+
* const gpu = queen.semaphore('gpu', 4, { ttl: '2m' })
|
|
350
|
+
* const { acquired } = await gpu.run(() => train(job), { wait: '30s' })
|
|
351
|
+
*
|
|
352
|
+
* Everything a lock's handle does, this one does (it is the same class: a
|
|
353
|
+
* lock is the semaphore of one). Every holder of one name passes the same
|
|
354
|
+
* limit — it is the caller's and is stored nowhere, so while a limit is
|
|
355
|
+
* being changed the larger one rules.
|
|
356
|
+
*
|
|
357
|
+
* @returns {Lock}
|
|
358
|
+
*/
|
|
359
|
+
semaphore(name, limit, opts = {}) {
|
|
360
|
+
return new Lock(this.locks, name, { ...opts, limit })
|
|
361
|
+
}
|
|
362
|
+
|
|
363
|
+
/**
|
|
364
|
+
* The four lock operations as the broker speaks them — `acquire`, `renew`,
|
|
365
|
+
* `release`, `get`, and `batch` for several locks in one call — with no
|
|
366
|
+
* state kept: you carry the token. `queen.lock()` is what most code wants;
|
|
367
|
+
* this is for `get` (who holds it?) and for a caller with its own loop.
|
|
368
|
+
*
|
|
369
|
+
* Lazily initialized, singleton, like `kv`.
|
|
370
|
+
* @returns {Locks}
|
|
371
|
+
*/
|
|
372
|
+
get locks() {
|
|
373
|
+
if (!this.#locks) {
|
|
374
|
+
this.#locks = new Locks(this.#httpClient)
|
|
375
|
+
}
|
|
376
|
+
return this.#locks
|
|
377
|
+
}
|
|
378
|
+
|
|
301
379
|
// ===========================
|
|
302
380
|
// Ephemeral API Entry Point
|
|
303
381
|
// ===========================
|
|
@@ -717,6 +795,18 @@ export class Queen {
|
|
|
717
795
|
// Cleanup buffer manager
|
|
718
796
|
this.#bufferManager.cleanup()
|
|
719
797
|
|
|
798
|
+
// Give back the locks this client's handles still hold, so the next
|
|
799
|
+
// holder does not wait out their lifetime. Best effort: one that cannot
|
|
800
|
+
// be released expires by itself.
|
|
801
|
+
if (this.#locks) {
|
|
802
|
+
try {
|
|
803
|
+
const released = await this.#locks.releaseAll()
|
|
804
|
+
if (released > 0) logger.log('Queen.close', { locksReleased: released })
|
|
805
|
+
} catch (error) {
|
|
806
|
+
logger.warn('Queen.close', { error: error.message, phase: 'lock-release' })
|
|
807
|
+
}
|
|
808
|
+
}
|
|
809
|
+
|
|
720
810
|
// Remove shutdown handlers
|
|
721
811
|
for (const cleanup of this.#shutdownHandlers) {
|
|
722
812
|
cleanup()
|
package/client-v2/README.md
CHANGED
|
@@ -20,6 +20,7 @@ Welcome to Queen client! This is your friendly guide to mastering message queues
|
|
|
20
20
|
- [Part 12: Message Tracing - Debug Your Workflows](#part-12-message-tracing---debug-your-workflows)
|
|
21
21
|
- [Part 13: Callbacks & Error Handling](#part-13-callbacks--error-handling)
|
|
22
22
|
- [Part 14: Graceful Shutdown](#part-14-graceful-shutdown)
|
|
23
|
+
- [Part 15: Binary Push, Compression & Framed Pop](#part-15-binary-push-compression--framed-pop)
|
|
23
24
|
- [Cheat Sheet](#cheat-sheet)
|
|
24
25
|
|
|
25
26
|
---
|
|
@@ -1607,6 +1608,67 @@ await queen.close()
|
|
|
1607
1608
|
|
|
1608
1609
|
---
|
|
1609
1610
|
|
|
1611
|
+
## Part 15: Binary Push, Compression & Framed Pop
|
|
1612
|
+
|
|
1613
|
+
Everything so far travels as JSON. For a high message rate, the client has three
|
|
1614
|
+
opt-in forms of the push, pop and transaction requests. They change how the
|
|
1615
|
+
messages travel and nothing else: your calls, their results, deduplication and
|
|
1616
|
+
the delivery guarantees stay the same. All three are off by default.
|
|
1617
|
+
|
|
1618
|
+
```javascript
|
|
1619
|
+
const queen = new Queen({
|
|
1620
|
+
url: 'http://localhost:6632',
|
|
1621
|
+
binaryPush: true, // a push travels as a binary body
|
|
1622
|
+
pushCompression: true, // with binaryPush: framed and compressed here
|
|
1623
|
+
framedPop: true // a pop returns the stored records, unpacked here
|
|
1624
|
+
})
|
|
1625
|
+
|
|
1626
|
+
// The rest of your code does not change.
|
|
1627
|
+
await queen.queue('events').partition('device-7').push([{ data: { n: 1 } }])
|
|
1628
|
+
await queen.queue('events').group('workers').consume(async (message) => {
|
|
1629
|
+
console.log(message.data)
|
|
1630
|
+
})
|
|
1631
|
+
```
|
|
1632
|
+
|
|
1633
|
+
### What Each Option Does
|
|
1634
|
+
|
|
1635
|
+
| Option | What it does |
|
|
1636
|
+
| --- | --- |
|
|
1637
|
+
| `binaryPush` | A push, direct or buffered, sends its items as one length-prefixed binary body instead of a JSON document. The answer is the same list, one entry per item. |
|
|
1638
|
+
| `pushCompression` | Needs `binaryPush`. The client frames each partition's batch the way the broker stores it, mints the message ids (UUIDv7) and compresses the batch with zstd, once. The broker checks the frames and, when it has nothing to add to them, stores those bytes without compressing them again. A transaction that pushes sends its pushes the same way, with the rest of the bundle as JSON in front of them. |
|
|
1639
|
+
| `framedPop` | `pop()` and `consume()` ask for the stored records (`frames=true`) and unpack them here into the same message objects, instead of a JSON answer that the broker renders message by message. |
|
|
1640
|
+
|
|
1641
|
+
Turn them on when the broker's CPU or the network limits a high message rate:
|
|
1642
|
+
the framing, the compression and the unpacking then run in the clients, and the
|
|
1643
|
+
batches cross the network compressed. zstd is the one of Node.js (`node:zlib`),
|
|
1644
|
+
so there is nothing to install.
|
|
1645
|
+
|
|
1646
|
+
### What Stays JSON
|
|
1647
|
+
|
|
1648
|
+
The client falls back by itself in these cases:
|
|
1649
|
+
|
|
1650
|
+
- A push with a `traceId` on one of its items.
|
|
1651
|
+
- A transaction with no push, or with a `traceId` on one of its pushes.
|
|
1652
|
+
- A pop the broker cannot answer in frames: a pop of a conflating group, and
|
|
1653
|
+
every pop on a broker that runs with an encryption key
|
|
1654
|
+
(`QUEEN_ENCRYPTION_KEY`). The client reads the JSON answer as always.
|
|
1655
|
+
- Acks, KV, timers, locks and the admin calls. Ephemeral queues have their own
|
|
1656
|
+
routes and do not change either.
|
|
1657
|
+
|
|
1658
|
+
A transaction's `results` come back in the order you added the operations, as
|
|
1659
|
+
with JSON. The broker answers a binary transaction with the acks first and the
|
|
1660
|
+
pushes after them, and the client puts them back.
|
|
1661
|
+
|
|
1662
|
+
### Which Broker
|
|
1663
|
+
|
|
1664
|
+
`binaryPush` and `pushCompression` need broker 2.3.0 or later. A broker up
|
|
1665
|
+
to 2.2.0 answers 400 to the binary bodies and the client does not fall back to
|
|
1666
|
+
JSON, so turn the two on after every broker of the cluster is upgraded.
|
|
1667
|
+
`framedPop` is safe against any broker: an older one ignores the parameter and
|
|
1668
|
+
answers JSON.
|
|
1669
|
+
|
|
1670
|
+
---
|
|
1671
|
+
|
|
1610
1672
|
## Cheat Sheet
|
|
1611
1673
|
|
|
1612
1674
|
### Connection
|
|
@@ -1883,7 +1945,10 @@ await queen.close()
|
|
|
1883
1945
|
loadBalancingStrategy: 'affinity', // 'affinity' | 'round-robin' | 'session'
|
|
1884
1946
|
affinityHashRing: 128,
|
|
1885
1947
|
enableFailover: true,
|
|
1886
|
-
healthRetryAfterMillis: 5000
|
|
1948
|
+
healthRetryAfterMillis: 5000,
|
|
1949
|
+
binaryPush: false, // a push as a binary body
|
|
1950
|
+
pushCompression: false, // with binaryPush: frames compressed in the client
|
|
1951
|
+
framedPop: false // a pop as the stored records
|
|
1887
1952
|
}
|
|
1888
1953
|
```
|
|
1889
1954
|
|
|
@@ -40,6 +40,10 @@ export class TransactionBuilder {
|
|
|
40
40
|
// was queued would ship a TTL already stale by however long the bundle took
|
|
41
41
|
// to assemble. Materialized in commit(), which is send time.
|
|
42
42
|
#kvEntries = []
|
|
43
|
+
// Lock handles whose permit this bundle commits under (`.guard(lock)`).
|
|
44
|
+
// Kept as handles, not as ops: the token is read at SEND time, because a
|
|
45
|
+
// lock's own renewal changes it.
|
|
46
|
+
#guards = []
|
|
43
47
|
#timerOps = []
|
|
44
48
|
#kvApi = null
|
|
45
49
|
|
|
@@ -202,6 +206,9 @@ export class TransactionBuilder {
|
|
|
202
206
|
putIfAbsent: (ns, key, value, opts = {}) => add(kvOp.putIfAbsent(ns, key, value, opts)),
|
|
203
207
|
delete: (ns, key, opts = {}) => add(kvOp.delete(ns, key, opts)),
|
|
204
208
|
incr: (ns, key, delta = 1, opts = {}) => add(kvOp.incr(ns, key, delta, opts)),
|
|
209
|
+
// A precondition on a key the bundle does not write. With
|
|
210
|
+
// `required: true` it is the bundle's gate; without, only a look.
|
|
211
|
+
check: (ns, key, opts = {}) => add(kvOp.check(ns, key, opts)),
|
|
205
212
|
getPrefix: () => {
|
|
206
213
|
throw new Error(
|
|
207
214
|
'kv: getPrefix is not available inside a transaction — unbounded read work under the outermost ' +
|
|
@@ -236,6 +243,40 @@ export class TransactionBuilder {
|
|
|
236
243
|
return this.kv.putIfAbsent(ns, key, value, { ...opts, ...required })
|
|
237
244
|
}
|
|
238
245
|
|
|
246
|
+
/**
|
|
247
|
+
* Commit this bundle only while a lock is held.
|
|
248
|
+
*
|
|
249
|
+
* const lock = queen.lock('daily-report', { ttl: '30s' })
|
|
250
|
+
* if (!(await lock.acquire())) return
|
|
251
|
+
* const res = await queen.transaction()
|
|
252
|
+
* .guard(lock)
|
|
253
|
+
* .queue('reports').push([{ data: report }])
|
|
254
|
+
* .commit()
|
|
255
|
+
* if (res.success === false) return // the lock is somebody else's now
|
|
256
|
+
*
|
|
257
|
+
* The guard is a `check` of the permit's row at the lock's token, with
|
|
258
|
+
* `required: true`: the broker judges it in the same log entry as the acks,
|
|
259
|
+
* pushes, keys and timers beside it. A holder that was paused past its
|
|
260
|
+
* lifetime and replaced commits NOTHING — which the lock by itself cannot
|
|
261
|
+
* promise, since nobody stops an expired holder from running.
|
|
262
|
+
*
|
|
263
|
+
* The token is read when `commit()` sends, and a lock's own background
|
|
264
|
+
* renewal moves it. A guard that lost to this handle's own renewal is sent
|
|
265
|
+
* again with the new token; one that lost to another holder is the verdict,
|
|
266
|
+
* returned like `once`'s (`success: false, reason: 'kv_precondition'`), and
|
|
267
|
+
* the handle then reports the lock lost.
|
|
268
|
+
*
|
|
269
|
+
* Throws at `commit()`, with `.code === 'LOCK_NOT_HELD'`, when the handle
|
|
270
|
+
* holds nothing: a step that asked for a guard must not go out without one.
|
|
271
|
+
*/
|
|
272
|
+
guard(lock) {
|
|
273
|
+
if (!lock || typeof lock.guard !== 'function' || typeof lock._settled !== 'function') {
|
|
274
|
+
throw new Error('transaction: guard() takes a lock from queen.lock() or queen.semaphore()')
|
|
275
|
+
}
|
|
276
|
+
this.#guards.push(lock)
|
|
277
|
+
return this
|
|
278
|
+
}
|
|
279
|
+
|
|
239
280
|
// ===========================
|
|
240
281
|
// Timer rider (§4, §9.6)
|
|
241
282
|
// ===========================
|
|
@@ -266,20 +307,11 @@ export class TransactionBuilder {
|
|
|
266
307
|
})
|
|
267
308
|
}
|
|
268
309
|
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
}
|
|
275
|
-
|
|
276
|
-
logger.log('TransactionBuilder.commit', {
|
|
277
|
-
operationCount: this.#operations.length,
|
|
278
|
-
requiredLeases: this.#requiredLeases.length,
|
|
279
|
-
kv: this.#kvEntries.length,
|
|
280
|
-
timers: this.#timerOps.length
|
|
281
|
-
})
|
|
282
|
-
|
|
310
|
+
/**
|
|
311
|
+
* The request body, built at send time. The guards go first in `kv`, so
|
|
312
|
+
* guard `i` is op `i` of the rider.
|
|
313
|
+
*/
|
|
314
|
+
#body() {
|
|
283
315
|
// Byte-identity when the riders are absent (§6.3): the keys are added only
|
|
284
316
|
// when there is something in them, so a bundle that uses neither feature
|
|
285
317
|
// produces exactly the body it produced before this feature existed.
|
|
@@ -287,16 +319,72 @@ export class TransactionBuilder {
|
|
|
287
319
|
operations: this.#operations,
|
|
288
320
|
requiredLeases: [...new Set(this.#requiredLeases)] // Unique leases
|
|
289
321
|
}
|
|
290
|
-
if (this.#kvEntries.length > 0) {
|
|
322
|
+
if (this.#guards.length + this.#kvEntries.length > 0) {
|
|
291
323
|
const now = Date.now()
|
|
292
|
-
body.kv =
|
|
324
|
+
body.kv = [
|
|
325
|
+
...this.#guards.map(lock => lock.guard()),
|
|
326
|
+
...this.#kvEntries.map(e => materializeKvOp(e, now))
|
|
327
|
+
]
|
|
293
328
|
}
|
|
294
329
|
if (this.#timerOps.length > 0) {
|
|
295
330
|
body.timers = this.#timerOps
|
|
296
331
|
}
|
|
332
|
+
return body
|
|
333
|
+
}
|
|
334
|
+
|
|
335
|
+
/**
|
|
336
|
+
* The lock whose guard is the precondition a rolled-back bundle names, if
|
|
337
|
+
* it is one of this bundle's. `failedIndex` is in the FLAT space of
|
|
338
|
+
* `results[]`: every pushed item and every ack first, then the `kv` rider.
|
|
339
|
+
*/
|
|
340
|
+
#failedGuard(result) {
|
|
341
|
+
if (this.#guards.length === 0 || !Number.isInteger(result.failedIndex)) return null
|
|
342
|
+
const flatOperations = this.#operations.reduce(
|
|
343
|
+
(n, op) => n + (op.type === 'push' ? op.items.length : 1), 0
|
|
344
|
+
)
|
|
345
|
+
return this.#guards[result.failedIndex - flatOperations] ?? null
|
|
346
|
+
}
|
|
347
|
+
|
|
348
|
+
async commit() {
|
|
349
|
+
const riderCount = this.#guards.length + this.#kvEntries.length + this.#timerOps.length
|
|
350
|
+
if (this.#operations.length === 0 && riderCount === 0) {
|
|
351
|
+
logger.error('TransactionBuilder.commit', 'No operations to commit')
|
|
352
|
+
throw new Error('Transaction has no operations to commit')
|
|
353
|
+
}
|
|
354
|
+
|
|
355
|
+
logger.log('TransactionBuilder.commit', {
|
|
356
|
+
operationCount: this.#operations.length,
|
|
357
|
+
requiredLeases: this.#requiredLeases.length,
|
|
358
|
+
kv: this.#kvEntries.length,
|
|
359
|
+
guards: this.#guards.length,
|
|
360
|
+
timers: this.#timerOps.length
|
|
361
|
+
})
|
|
297
362
|
|
|
298
363
|
try {
|
|
299
|
-
|
|
364
|
+
// A bundle is sent again in ONE case: its guard lost to the lock's own
|
|
365
|
+
// background renewal, which moved the token between the moment the body
|
|
366
|
+
// was built and the moment the broker judged it. The row still names
|
|
367
|
+
// this owner, so the lock is held; nothing committed (a lost required
|
|
368
|
+
// precondition rolls the whole bundle back), so sending it again with
|
|
369
|
+
// the new token is the same step, not a second one.
|
|
370
|
+
let result
|
|
371
|
+
for (let attempt = 0; ; attempt++) {
|
|
372
|
+
await Promise.all(this.#guards.map(lock => lock._settled()))
|
|
373
|
+
result = await this.#httpClient.post('/api/v1/transaction', this.#body())
|
|
374
|
+
if (result.success || result.reason !== 'kv_precondition') break
|
|
375
|
+
const lock = this.#failedGuard(result)
|
|
376
|
+
if (!lock) break
|
|
377
|
+
const ownRenewal = result.kvReason === 'version' && result.value?.owner === lock.owner
|
|
378
|
+
if (!ownRenewal) {
|
|
379
|
+
// Expired, released, or another holder's: the lock is gone.
|
|
380
|
+
lock._lost('guard')
|
|
381
|
+
break
|
|
382
|
+
}
|
|
383
|
+
if (attempt >= 3) break
|
|
384
|
+
await lock._settled()
|
|
385
|
+
if (!lock.held) break
|
|
386
|
+
logger.log('TransactionBuilder.commit', { status: 'guard_renewed', lock: lock.name, attempt })
|
|
387
|
+
}
|
|
300
388
|
|
|
301
389
|
if (!result.success) {
|
|
302
390
|
// THE ONE OUTCOME THAT RETURNS INSTEAD OF THROWING (§8.3).
|