queen-mq 2.0.3 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +100 -0
- package/client-v2/Queen.js +84 -0
- package/client-v2/builders/QueueBuilder.js +8 -0
- package/client-v2/builders/TransactionBuilder.js +105 -17
- package/client-v2/consumer/ConsumerManager.js +8 -0
- package/client-v2/consumer/Supervision.js +78 -0
- package/client-v2/http/HttpClient.js +2 -2
- package/client-v2/index.js +5 -0
- package/client-v2/kv/Kv.js +41 -0
- package/client-v2/locks/Locks.js +544 -0
- package/package.json +2 -2
- package/test-v2/consumer-unit/supervision.test.js +97 -0
- package/test-v2/kv-unit/locksWire.test.js +541 -0
- package/test-v2/locks.js +277 -0
- package/test-v2/run.js +2 -0
package/README.md
CHANGED
|
@@ -636,6 +636,49 @@ Otherwise the lanes do not serialise it for you: use `incr`, or carry `expect`.
|
|
|
636
636
|
**`putIfAbsent` plus a TTL is not a distributed lock.** A lock that expires is not revoked: the old
|
|
637
637
|
holder keeps working, it simply no longer has the row. Carry your `version` as `expect` on every
|
|
638
638
|
later write so a lapsed holder fails with `reason: 'version'` instead of overwriting the new one.
|
|
639
|
+
`queen.lock()` below is that, done for you.
|
|
640
|
+
|
|
641
|
+
**`check` asserts a version and writes nothing.** `kv.check(ns, key, { expect })` holds while the
|
|
642
|
+
key is at that version (`expect: 0`: while it is absent). With `required: true` in a transaction
|
|
643
|
+
it gates the commit on a key the transaction does not write.
|
|
644
|
+
|
|
645
|
+
### Locks and semaphores
|
|
646
|
+
|
|
647
|
+
A lock is a lease: one holder at a time, for a lifetime the holder renews, with a token that
|
|
648
|
+
fences a holder that outlived it. `queen.semaphore(name, n, opts)` is the same with `n` permits.
|
|
649
|
+
|
|
650
|
+
```javascript
|
|
651
|
+
const lock = queen.lock('daily-report', { ttl: '30s' })
|
|
652
|
+
if (!(await lock.acquire({ wait: '5s' }))) return // somebody else holds it
|
|
653
|
+
|
|
654
|
+
try {
|
|
655
|
+
await queen.transaction()
|
|
656
|
+
.guard(lock) // commits only while the lock is ours
|
|
657
|
+
.queue('reports').push([{ data: report }])
|
|
658
|
+
.commit()
|
|
659
|
+
} finally {
|
|
660
|
+
await lock.release()
|
|
661
|
+
}
|
|
662
|
+
|
|
663
|
+
// Or: acquire, run, release.
|
|
664
|
+
const { acquired, value } = await queen.lock('sync:crm', { ttl: '1m' }).run(() => sync())
|
|
665
|
+
```
|
|
666
|
+
|
|
667
|
+
**It expires, and nobody tells the holder.** A paused or partitioned process carries on past its
|
|
668
|
+
lifetime while somebody else acquires. `.guard(lock)` is what keeps its work out: the transaction
|
|
669
|
+
rolls back with `reason: 'kv_precondition'` (returned from `commit()`, not thrown) when the lock is
|
|
670
|
+
no longer this handle's. Outside Queen, fence with `lock.token`, which only rises on a lock.
|
|
671
|
+
|
|
672
|
+
**The handle renews in the background**, every third of the lifetime (`autoRenew: false` to do it
|
|
673
|
+
yourself with `lock.renew()`), and says when the lock is gone: `lock.signal` aborts and
|
|
674
|
+
`lock.onLost(fn)` runs. Nothing stops your code: pass `lock.signal` to what the work awaits.
|
|
675
|
+
|
|
676
|
+
**The token changes at every renew.** Read `lock.token` and `lock.guard()` when you use them; do
|
|
677
|
+
not keep a copy across an `await`.
|
|
678
|
+
|
|
679
|
+
`queen.locks` is the wire, with no state kept: `acquire`, `renew`, `release`, `get(name)` (who
|
|
680
|
+
holds it, since when, until when) and `batch` for several locks in one call. `queen.close()`
|
|
681
|
+
releases the locks its handles still hold.
|
|
639
682
|
|
|
640
683
|
### Timers
|
|
641
684
|
|
|
@@ -1079,3 +1122,60 @@ const messages: Message<OrderData>[] = await queen.queue('orders').pop()
|
|
|
1079
1122
|
## License
|
|
1080
1123
|
|
|
1081
1124
|
Apache 2.0 - See [LICENSE.md](../LICENSE.md)
|
|
1125
|
+
|
|
1126
|
+
### Optional consumer supervision
|
|
1127
|
+
|
|
1128
|
+
```js
|
|
1129
|
+
await queen.queue('orders').group('billing')
|
|
1130
|
+
.supervision({ group: 'billing-production' })
|
|
1131
|
+
.concurrency(4).each().consume(async message => { /* process message */ })
|
|
1132
|
+
```
|
|
1133
|
+
|
|
1134
|
+
Supervision defaults to off; `.supervision(false)` disables it. The group names
|
|
1135
|
+
an application/deployment in the dashboard, independently of the consumer group.
|
|
1136
|
+
Each consume invocation publishes its own instance into the broker's
|
|
1137
|
+
`queen-supervisor` KV namespace every 10 seconds (30-second heartbeat timeout,
|
|
1138
|
+
60-second TTL), plus a final stopped observation. The credential needs KV write
|
|
1139
|
+
access. Publication is serialized and best effort with a two-second deadline.
|
|
1140
|
+
|
|
1141
|
+
#### Match supervision to the consumer lifetime
|
|
1142
|
+
|
|
1143
|
+
The instance belongs to one **`consume()` invocation**, not to the `Queen`
|
|
1144
|
+
client, queue, hostname, process or publication group. Reusing the same client,
|
|
1145
|
+
builder and group does not reuse the instance ID. Each invocation starts new
|
|
1146
|
+
counters, publishes its initial state, then reports `stopped` when all its
|
|
1147
|
+
workers exit, including a normal exit caused by `.limit(...)`, `.idleMillis(...)`
|
|
1148
|
+
or cancellation.
|
|
1149
|
+
|
|
1150
|
+
| Application pattern | Supervision setup |
|
|
1151
|
+
| --- | --- |
|
|
1152
|
+
| One persistent `consume()` with several concurrent workers | Enable `.supervision({ group })` on that invocation. |
|
|
1153
|
+
| Several persistent `consume()` calls in one process | Enable reporting on each; expect several instances with the same hostname. Budget their combined concurrency. |
|
|
1154
|
+
| An outer scheduler repeatedly calling `.limit(1).consume(...)` or rotating queues | Leave per-call supervision off when you need the scheduler's lifetime status; publish one observation for the persistent scheduler instead. |
|
|
1155
|
+
|
|
1156
|
+
For a persistent consumer, omit limits that deliberately end the invocation.
|
|
1157
|
+
Pass an `AbortSignal` to `consume(handler, { signal })`; on shutdown, abort it,
|
|
1158
|
+
await the consume promise so active handlers drain, then close the client.
|
|
1159
|
+
Do not remove an existing scheduler's limits just to change the dashboard:
|
|
1160
|
+
those limits may enforce queue fairness and the application's worker budget.
|
|
1161
|
+
|
|
1162
|
+
Supervising short calls is valid when you want to observe those calls. A loop
|
|
1163
|
+
that starts a new call after every message or idle poll will produce many
|
|
1164
|
+
recent `stopped` instances, with `running: 0`, until their 60-second TTL expires.
|
|
1165
|
+
Expired records can remain visible until the broker sweeps them.
|
|
1166
|
+
That is the last state of each finished invocation, not evidence that its
|
|
1167
|
+
process or pod has stopped. Compare instance IDs and application logs.
|
|
1168
|
+
|
|
1169
|
+
The JavaScript API has no option to attach several consume invocations to one
|
|
1170
|
+
persistent reporter or supply their instance ID. A custom scheduler can publish
|
|
1171
|
+
the [consumer status contract](https://queenmq.com/reference/supervisor-status/#reporting-an-application-owned-scheduler)
|
|
1172
|
+
with one ID per scheduler lifetime, stable worker pool names and counters
|
|
1173
|
+
across turns. Keep SDK supervision disabled on its short calls and preserve
|
|
1174
|
+
their ACK/NACK, lease renewal, cancellation and scheduling behavior.
|
|
1175
|
+
|
|
1176
|
+
The Supervisors page supporting `queen.consumer.status/v1` shows live async
|
|
1177
|
+
consumer loops, busy handlers, successful/failed handler calls and progress times.
|
|
1178
|
+
A batch is one handler call; completion does not imply ACK success. No payloads or
|
|
1179
|
+
error text are published. Event-loop starvation can stop heartbeats. Reporting
|
|
1180
|
+
does not restart processes or tasks, change ACK/lease policies, or enable remote
|
|
1181
|
+
control. With reporting off there is no additional timer or network traffic.
|
package/client-v2/Queen.js
CHANGED
|
@@ -10,6 +10,7 @@ import { QueueBuilder } from './builders/QueueBuilder.js'
|
|
|
10
10
|
import { TransactionBuilder } from './builders/TransactionBuilder.js'
|
|
11
11
|
import { TimerBuilder } from './builders/TimerBuilder.js'
|
|
12
12
|
import { Kv } from './kv/Kv.js'
|
|
13
|
+
import { Locks, Lock } from './locks/Locks.js'
|
|
13
14
|
import { Ephemeral } from './ephemeral/Ephemeral.js'
|
|
14
15
|
import { StreamBuilder } from './stream/StreamBuilder.js'
|
|
15
16
|
import { StreamConsumer } from './stream/StreamConsumer.js'
|
|
@@ -93,6 +94,7 @@ export class Queen {
|
|
|
93
94
|
#shutdownHandlers = []
|
|
94
95
|
#admin = null
|
|
95
96
|
#kv = null
|
|
97
|
+
#locks = null
|
|
96
98
|
#ephemeral = null
|
|
97
99
|
// Process-wide kill switch for pop autopilot, read from
|
|
98
100
|
// QUEEN_SDK_POP_AUTOPILOT once here rather than on every pop: it is a
|
|
@@ -298,6 +300,76 @@ export class Queen {
|
|
|
298
300
|
return this.#kv
|
|
299
301
|
}
|
|
300
302
|
|
|
303
|
+
// ===========================
|
|
304
|
+
// Locks API Entry Point
|
|
305
|
+
// ===========================
|
|
306
|
+
|
|
307
|
+
/**
|
|
308
|
+
* A lock: one holder at a time, held as a lease, with a fencing token.
|
|
309
|
+
*
|
|
310
|
+
* const lock = queen.lock('daily-report', { ttl: '30s' })
|
|
311
|
+
* if (!(await lock.acquire())) return // somebody else has it
|
|
312
|
+
* try {
|
|
313
|
+
* await queen.transaction().guard(lock)
|
|
314
|
+
* .queue('reports').push([{ data: report }]).commit()
|
|
315
|
+
* } finally {
|
|
316
|
+
* await lock.release()
|
|
317
|
+
* }
|
|
318
|
+
*
|
|
319
|
+
* Creating the handle sends nothing. `acquire()` resolves a boolean,
|
|
320
|
+
* `acquire({ wait: '10s' })` keeps trying, and while it is held the handle
|
|
321
|
+
* renews it in the background and aborts `lock.signal` if it is lost.
|
|
322
|
+
*
|
|
323
|
+
* It is a lease, not a mutex: it expires, and a holder that outlived it
|
|
324
|
+
* keeps running. `.guard(lock)` on a transaction is what makes the WORK
|
|
325
|
+
* exclusive; see `locks/Locks.js` for the whole argument.
|
|
326
|
+
*
|
|
327
|
+
* @param {string} name - no '#', no control characters, at most 256 bytes
|
|
328
|
+
* @param {{ttl?: string, ttlSeconds?: number, owner?: string, autoRenew?: boolean, renewEvery?: string}} opts
|
|
329
|
+
* a lifetime is mandatory: `ttl: '30s'` or `ttlSeconds: 30`
|
|
330
|
+
* @returns {Lock}
|
|
331
|
+
*/
|
|
332
|
+
lock(name, opts = {}) {
|
|
333
|
+
if (opts.limit !== undefined && opts.limit !== 1) {
|
|
334
|
+
throw new Error('lock: a lock has one permit — queen.semaphore(name, limit, opts) is the one with more')
|
|
335
|
+
}
|
|
336
|
+
return new Lock(this.locks, name, opts)
|
|
337
|
+
}
|
|
338
|
+
|
|
339
|
+
/**
|
|
340
|
+
* A semaphore: at most `limit` holders at a time. One handle is ONE permit;
|
|
341
|
+
* make a handle per holder.
|
|
342
|
+
*
|
|
343
|
+
* const gpu = queen.semaphore('gpu', 4, { ttl: '2m' })
|
|
344
|
+
* const { acquired } = await gpu.run(() => train(job), { wait: '30s' })
|
|
345
|
+
*
|
|
346
|
+
* Everything a lock's handle does, this one does (it is the same class: a
|
|
347
|
+
* lock is the semaphore of one). Every holder of one name passes the same
|
|
348
|
+
* limit — it is the caller's and is stored nowhere, so while a limit is
|
|
349
|
+
* being changed the larger one rules.
|
|
350
|
+
*
|
|
351
|
+
* @returns {Lock}
|
|
352
|
+
*/
|
|
353
|
+
semaphore(name, limit, opts = {}) {
|
|
354
|
+
return new Lock(this.locks, name, { ...opts, limit })
|
|
355
|
+
}
|
|
356
|
+
|
|
357
|
+
/**
|
|
358
|
+
* The four lock operations as the broker speaks them — `acquire`, `renew`,
|
|
359
|
+
* `release`, `get`, and `batch` for several locks in one call — with no
|
|
360
|
+
* state kept: you carry the token. `queen.lock()` is what most code wants;
|
|
361
|
+
* this is for `get` (who holds it?) and for a caller with its own loop.
|
|
362
|
+
*
|
|
363
|
+
* Lazily initialized, singleton, like `kv`.
|
|
364
|
+
* @returns {Locks}
|
|
365
|
+
*/
|
|
366
|
+
get locks() {
|
|
367
|
+
if (!this.#locks) {
|
|
368
|
+
this.#locks = new Locks(this.#httpClient)
|
|
369
|
+
}
|
|
370
|
+
return this.#locks
|
|
371
|
+
}
|
|
372
|
+
|
|
301
373
|
// ===========================
|
|
302
374
|
// Ephemeral API Entry Point
|
|
303
375
|
// ===========================
|
|
@@ -717,6 +789,18 @@ export class Queen {
|
|
|
717
789
|
// Cleanup buffer manager
|
|
718
790
|
this.#bufferManager.cleanup()
|
|
719
791
|
|
|
792
|
+
// Give back the locks this client's handles still hold, so the next
|
|
793
|
+
// holder does not wait out their lifetime. Best effort: one that cannot
|
|
794
|
+
// be released expires by itself.
|
|
795
|
+
if (this.#locks) {
|
|
796
|
+
try {
|
|
797
|
+
const released = await this.#locks.releaseAll()
|
|
798
|
+
if (released > 0) logger.log('Queen.close', { locksReleased: released })
|
|
799
|
+
} catch (error) {
|
|
800
|
+
logger.warn('Queen.close', { error: error.message, phase: 'lock-release' })
|
|
801
|
+
}
|
|
802
|
+
}
|
|
803
|
+
|
|
720
804
|
// Remove shutdown handlers
|
|
721
805
|
for (const cleanup of this.#shutdownHandlers) {
|
|
722
806
|
cleanup()
|
|
@@ -42,6 +42,7 @@ export class QueueBuilder {
|
|
|
42
42
|
#config = {}
|
|
43
43
|
|
|
44
44
|
// Consume options
|
|
45
|
+
#supervision = false
|
|
45
46
|
#concurrency = CONSUME_DEFAULTS.concurrency
|
|
46
47
|
// batch / maxPartitions hold the USER's value, and null means the setter was
|
|
47
48
|
// never called -- which is the dimension pop autopilot gets to choose. The
|
|
@@ -238,6 +239,12 @@ export class QueueBuilder {
|
|
|
238
239
|
return this
|
|
239
240
|
}
|
|
240
241
|
|
|
242
|
+
/** Opt-in consumer observations in the broker dashboard; false by default. */
|
|
243
|
+
supervision(config = false) {
|
|
244
|
+
this.#supervision = config
|
|
245
|
+
return this
|
|
246
|
+
}
|
|
247
|
+
|
|
241
248
|
concurrency(count) {
|
|
242
249
|
this.#concurrency = Math.max(1, count)
|
|
243
250
|
return this
|
|
@@ -410,6 +417,7 @@ export class QueueBuilder {
|
|
|
410
417
|
// has to survive all the way to #buildParams: it is the ONLY record that
|
|
411
418
|
// the user said nothing about that dimension.
|
|
412
419
|
autopilot: this.#autopilotEnabled(),
|
|
420
|
+
supervision: options.supervision ?? this.#supervision,
|
|
413
421
|
signal: options.signal
|
|
414
422
|
}
|
|
415
423
|
|
|
@@ -40,6 +40,10 @@ export class TransactionBuilder {
|
|
|
40
40
|
// was queued would ship a TTL already stale by however long the bundle took
|
|
41
41
|
// to assemble. Materialized in commit(), which is send time.
|
|
42
42
|
#kvEntries = []
|
|
43
|
+
// Lock handles whose permit this bundle commits under (`.guard(lock)`).
|
|
44
|
+
// Kept as handles, not as ops: the token is read at SEND time, because a
|
|
45
|
+
// lock's own renewal changes it.
|
|
46
|
+
#guards = []
|
|
43
47
|
#timerOps = []
|
|
44
48
|
#kvApi = null
|
|
45
49
|
|
|
@@ -202,6 +206,9 @@ export class TransactionBuilder {
|
|
|
202
206
|
putIfAbsent: (ns, key, value, opts = {}) => add(kvOp.putIfAbsent(ns, key, value, opts)),
|
|
203
207
|
delete: (ns, key, opts = {}) => add(kvOp.delete(ns, key, opts)),
|
|
204
208
|
incr: (ns, key, delta = 1, opts = {}) => add(kvOp.incr(ns, key, delta, opts)),
|
|
209
|
+
// A precondition on a key the bundle does not write. With
|
|
210
|
+
// `required: true` it is the bundle's gate; without, only a look.
|
|
211
|
+
check: (ns, key, opts = {}) => add(kvOp.check(ns, key, opts)),
|
|
205
212
|
getPrefix: () => {
|
|
206
213
|
throw new Error(
|
|
207
214
|
'kv: getPrefix is not available inside a transaction — unbounded read work under the outermost ' +
|
|
@@ -236,6 +243,40 @@ export class TransactionBuilder {
|
|
|
236
243
|
return this.kv.putIfAbsent(ns, key, value, { ...opts, ...required })
|
|
237
244
|
}
|
|
238
245
|
|
|
246
|
+
/**
|
|
247
|
+
* Commit this bundle only while a lock is held.
|
|
248
|
+
*
|
|
249
|
+
* const lock = queen.lock('daily-report', { ttl: '30s' })
|
|
250
|
+
* if (!(await lock.acquire())) return
|
|
251
|
+
* const res = await queen.transaction()
|
|
252
|
+
* .guard(lock)
|
|
253
|
+
* .queue('reports').push([{ data: report }])
|
|
254
|
+
* .commit()
|
|
255
|
+
* if (res.success === false) return // the lock is somebody else's now
|
|
256
|
+
*
|
|
257
|
+
* The guard is a `check` of the permit's row at the lock's token, with
|
|
258
|
+
* `required: true`: the broker judges it in the same log entry as the acks,
|
|
259
|
+
* pushes, keys and timers beside it. A holder that was paused past its
|
|
260
|
+
* lifetime and replaced commits NOTHING — which the lock by itself cannot
|
|
261
|
+
* promise, since nobody stops an expired holder from running.
|
|
262
|
+
*
|
|
263
|
+
* The token is read when `commit()` sends, and a lock's own background
|
|
264
|
+
* renewal moves it. A guard that lost to this handle's own renewal is sent
|
|
265
|
+
* again with the new token; one that lost to another holder is the verdict,
|
|
266
|
+
* returned like `once`'s (`success: false, reason: 'kv_precondition'`), and
|
|
267
|
+
* the handle then reports the lock lost.
|
|
268
|
+
*
|
|
269
|
+
* Throws at `commit()`, with `.code === 'LOCK_NOT_HELD'`, when the handle
|
|
270
|
+
* holds nothing: a step that asked for a guard must not go out without one.
|
|
271
|
+
*/
|
|
272
|
+
guard(lock) {
|
|
273
|
+
if (!lock || typeof lock.guard !== 'function' || typeof lock._settled !== 'function') {
|
|
274
|
+
throw new Error('transaction: guard() takes a lock from queen.lock() or queen.semaphore()')
|
|
275
|
+
}
|
|
276
|
+
this.#guards.push(lock)
|
|
277
|
+
return this
|
|
278
|
+
}
|
|
279
|
+
|
|
239
280
|
// ===========================
|
|
240
281
|
// Timer rider (§4, §9.6)
|
|
241
282
|
// ===========================
|
|
@@ -266,20 +307,11 @@ export class TransactionBuilder {
|
|
|
266
307
|
})
|
|
267
308
|
}
|
|
268
309
|
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
}
|
|
275
|
-
|
|
276
|
-
logger.log('TransactionBuilder.commit', {
|
|
277
|
-
operationCount: this.#operations.length,
|
|
278
|
-
requiredLeases: this.#requiredLeases.length,
|
|
279
|
-
kv: this.#kvEntries.length,
|
|
280
|
-
timers: this.#timerOps.length
|
|
281
|
-
})
|
|
282
|
-
|
|
310
|
+
/**
|
|
311
|
+
* The request body, built at send time. The guards go first in `kv`, so
|
|
312
|
+
* guard `i` is op `i` of the rider.
|
|
313
|
+
*/
|
|
314
|
+
#body() {
|
|
283
315
|
// Byte-identity when the riders are absent (§6.3): the keys are added only
|
|
284
316
|
// when there is something in them, so a bundle that uses neither feature
|
|
285
317
|
// produces exactly the body it produced before this feature existed.
|
|
@@ -287,16 +319,72 @@ export class TransactionBuilder {
|
|
|
287
319
|
operations: this.#operations,
|
|
288
320
|
requiredLeases: [...new Set(this.#requiredLeases)] // Unique leases
|
|
289
321
|
}
|
|
290
|
-
if (this.#kvEntries.length > 0) {
|
|
322
|
+
if (this.#guards.length + this.#kvEntries.length > 0) {
|
|
291
323
|
const now = Date.now()
|
|
292
|
-
body.kv =
|
|
324
|
+
body.kv = [
|
|
325
|
+
...this.#guards.map(lock => lock.guard()),
|
|
326
|
+
...this.#kvEntries.map(e => materializeKvOp(e, now))
|
|
327
|
+
]
|
|
293
328
|
}
|
|
294
329
|
if (this.#timerOps.length > 0) {
|
|
295
330
|
body.timers = this.#timerOps
|
|
296
331
|
}
|
|
332
|
+
return body
|
|
333
|
+
}
|
|
334
|
+
|
|
335
|
+
/**
|
|
336
|
+
* The lock whose guard is the precondition a rolled-back bundle names, if
|
|
337
|
+
* it is one of this bundle's. `failedIndex` is in the FLAT space of
|
|
338
|
+
* `results[]`: every pushed item and every ack first, then the `kv` rider.
|
|
339
|
+
*/
|
|
340
|
+
#failedGuard(result) {
|
|
341
|
+
if (this.#guards.length === 0 || !Number.isInteger(result.failedIndex)) return null
|
|
342
|
+
const flatOperations = this.#operations.reduce(
|
|
343
|
+
(n, op) => n + (op.type === 'push' ? op.items.length : 1), 0
|
|
344
|
+
)
|
|
345
|
+
return this.#guards[result.failedIndex - flatOperations] ?? null
|
|
346
|
+
}
|
|
347
|
+
|
|
348
|
+
async commit() {
|
|
349
|
+
const riderCount = this.#guards.length + this.#kvEntries.length + this.#timerOps.length
|
|
350
|
+
if (this.#operations.length === 0 && riderCount === 0) {
|
|
351
|
+
logger.error('TransactionBuilder.commit', 'No operations to commit')
|
|
352
|
+
throw new Error('Transaction has no operations to commit')
|
|
353
|
+
}
|
|
354
|
+
|
|
355
|
+
logger.log('TransactionBuilder.commit', {
|
|
356
|
+
operationCount: this.#operations.length,
|
|
357
|
+
requiredLeases: this.#requiredLeases.length,
|
|
358
|
+
kv: this.#kvEntries.length,
|
|
359
|
+
guards: this.#guards.length,
|
|
360
|
+
timers: this.#timerOps.length
|
|
361
|
+
})
|
|
297
362
|
|
|
298
363
|
try {
|
|
299
|
-
|
|
364
|
+
// A bundle is sent again in ONE case: its guard lost to the lock's own
|
|
365
|
+
// background renewal, which moved the token between the moment the body
|
|
366
|
+
// was built and the moment the broker judged it. The row still names
|
|
367
|
+
// this owner, so the lock is held; nothing committed (a lost required
|
|
368
|
+
// precondition rolls the whole bundle back), so sending it again with
|
|
369
|
+
// the new token is the same step, not a second one.
|
|
370
|
+
let result
|
|
371
|
+
for (let attempt = 0; ; attempt++) {
|
|
372
|
+
await Promise.all(this.#guards.map(lock => lock._settled()))
|
|
373
|
+
result = await this.#httpClient.post('/api/v1/transaction', this.#body())
|
|
374
|
+
if (result.success || result.reason !== 'kv_precondition') break
|
|
375
|
+
const lock = this.#failedGuard(result)
|
|
376
|
+
if (!lock) break
|
|
377
|
+
const ownRenewal = result.kvReason === 'version' && result.value?.owner === lock.owner
|
|
378
|
+
if (!ownRenewal) {
|
|
379
|
+
// Expired, released, or another holder's: the lock is gone.
|
|
380
|
+
lock._lost('guard')
|
|
381
|
+
break
|
|
382
|
+
}
|
|
383
|
+
if (attempt >= 3) break
|
|
384
|
+
await lock._settled()
|
|
385
|
+
if (!lock.held) break
|
|
386
|
+
logger.log('TransactionBuilder.commit', { status: 'guard_renewed', lock: lock.name, attempt })
|
|
387
|
+
}
|
|
300
388
|
|
|
301
389
|
if (!result.success) {
|
|
302
390
|
// THE ONE OUTCOME THAT RETURNS INSTEAD OF THROWING (§8.3).
|
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
*/
|
|
4
4
|
|
|
5
5
|
import * as logger from '../utils/logger.js'
|
|
6
|
+
import { Supervision } from './Supervision.js'
|
|
6
7
|
import { checkConflationResponse, CONFLATION_UNSUPPORTED } from '../utils/conflation.js'
|
|
7
8
|
import { popSizing, parseAutopilotDecision, emptyPollDelayMillis } from '../utils/autopilot.js'
|
|
8
9
|
import { CONSUME_DEFAULTS } from '../utils/defaults.js'
|
|
@@ -92,9 +93,13 @@ export class ConsumerManager {
|
|
|
92
93
|
// Generate affinity key for consistent routing to same backend
|
|
93
94
|
const affinityKey = this.#getAffinityKey(queue, partition, namespace, task, group)
|
|
94
95
|
|
|
96
|
+
const supervision = options.supervision ? new Supervision(this.#httpClient, options.supervision, options) : null
|
|
97
|
+
if (supervision) handler = supervision.wrap(handler)
|
|
98
|
+
|
|
95
99
|
// Start workers
|
|
96
100
|
const workers = []
|
|
97
101
|
for (let i = 0; i < concurrency; i++) {
|
|
102
|
+
if (supervision) supervision.running++
|
|
98
103
|
workers.push(this.#worker(i, handler, path, baseParams, {
|
|
99
104
|
batch,
|
|
100
105
|
limit,
|
|
@@ -113,9 +118,12 @@ export class ConsumerManager {
|
|
|
113
118
|
// the pop target to key the once-per-(queue,group) conflict warning.
|
|
114
119
|
conflation,
|
|
115
120
|
conflationCtx: { queue, namespace, task, group }
|
|
121
|
+
}).finally(async () => {
|
|
122
|
+
if (supervision && --supervision.running === 0) await supervision.stop()
|
|
116
123
|
}))
|
|
117
124
|
}
|
|
118
125
|
|
|
126
|
+
supervision?.start()
|
|
119
127
|
logger.log('ConsumerManager.start', { status: 'workers-started', count: concurrency })
|
|
120
128
|
|
|
121
129
|
// Wait for all workers to complete
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
import { randomBytes } from 'node:crypto'
|
|
2
|
+
import { hostname } from 'node:os'
|
|
3
|
+
import * as logger from '../utils/logger.js'
|
|
4
|
+
|
|
5
|
+
// One opt-in reporter per consume invocation. Handler calls are not ACKs.
|
|
6
|
+
export class Supervision {
|
|
7
|
+
constructor(http, config, options) {
|
|
8
|
+
if (!config || typeof config !== 'object' || typeof config.group !== 'string' || !/^[A-Za-z0-9][A-Za-z0-9._-]{0,254}$/.test(config.group) || config.group === 'coordination' || /[\r\n]/.test(config.group)) throw new Error('supervision.group must be a valid application/deployment name')
|
|
9
|
+
if (!Number.isInteger(options.concurrency) || options.concurrency < 1 || options.concurrency > 4096) throw new Error('supervision requires concurrency between 1 and 4096')
|
|
10
|
+
this.http = http
|
|
11
|
+
this.options = options
|
|
12
|
+
this.group = config.group
|
|
13
|
+
this.id = randomBytes(16).toString('hex')
|
|
14
|
+
this.started = Date.now()
|
|
15
|
+
this.monotonic = performance.now()
|
|
16
|
+
this.running = this.completed = this.failed = this.sequence = 0
|
|
17
|
+
this.last = this.pending = this.timer = null
|
|
18
|
+
this.active = new Map()
|
|
19
|
+
this.state = 'running'
|
|
20
|
+
this.warned = false
|
|
21
|
+
}
|
|
22
|
+
wrap(handler) {
|
|
23
|
+
return async (...args) => {
|
|
24
|
+
const id = this.sequence++
|
|
25
|
+
this.active.set(id, performance.now())
|
|
26
|
+
try { const result = await handler(...args); this.completed++; return result }
|
|
27
|
+
catch (error) { this.failed++; throw error }
|
|
28
|
+
finally { this.active.delete(id); this.last = Math.floor(Date.now() / 1000) }
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
start() {
|
|
32
|
+
this.publish()
|
|
33
|
+
this.timer = setInterval(() => this.publish(), 10_000)
|
|
34
|
+
this.timer.unref()
|
|
35
|
+
}
|
|
36
|
+
document() {
|
|
37
|
+
const now = performance.now()
|
|
38
|
+
return {
|
|
39
|
+
schema: 'queen.consumer.status/v1', instance_id: this.id, engine: 'js', execution_model: 'async-tasks',
|
|
40
|
+
hostname: hostname(), pid: process.pid, state: this.state, updated_at_epoch: Math.floor(Date.now() / 1000),
|
|
41
|
+
started_at_epoch: Math.floor(this.started / 1000), uptime_seconds: Math.floor((now - this.monotonic) / 1000),
|
|
42
|
+
configuration: { heartbeat_timeout: 30 },
|
|
43
|
+
pool_status: [{ name: 'consumer', queue: this.options.queue || null, namespace: this.options.namespace || null,
|
|
44
|
+
task: this.options.task || null, consumer_group: this.options.group || '__QUEUE_MODE__',
|
|
45
|
+
desired: this.options.concurrency, running: this.running, busy: this.active.size,
|
|
46
|
+
completed: this.completed, failed: this.failed, last_completed_at_epoch: this.last,
|
|
47
|
+
oldest_inflight_seconds: this.active.size ? Math.floor((now - Math.min(...this.active.values())) / 1000) : null }],
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
publish() {
|
|
51
|
+
if (this.pending) return this.pending
|
|
52
|
+
this.pending = this.send().finally(() => { this.pending = null })
|
|
53
|
+
return this.pending
|
|
54
|
+
}
|
|
55
|
+
async send() {
|
|
56
|
+
try {
|
|
57
|
+
const bytes = Buffer.from(JSON.stringify(this.document()))
|
|
58
|
+
if (bytes.length > 45_000) throw new Error('Consumer status exceeds one chunk')
|
|
59
|
+
const write = randomBytes(16).toString('hex'), slot = `${this.group}/${this.id}`
|
|
60
|
+
const ops = [
|
|
61
|
+
{ op: 'put', ns: 'queen-supervisor', key: `${slot}/head`, value: { format: 'queen.supervisor.remote-status/v1', write, chunks: 1, bytes: bytes.length }, ttlSeconds: 60 },
|
|
62
|
+
{ op: 'put', ns: 'queen-supervisor', key: `${slot}/chunk/0000`, value: { write, index: 0, data: bytes.toString('base64') }, ttlSeconds: 60 },
|
|
63
|
+
]
|
|
64
|
+
const result = await this.http.post('/api/v1/kv', { operations: ops }, 2000, null, null, AbortSignal.timeout(2000))
|
|
65
|
+
if (!Array.isArray(result?.results) || result.results.length !== 2 || result.results.some(r => r.applied !== true)) throw new Error('Consumer status publication was not applied')
|
|
66
|
+
this.warned = false
|
|
67
|
+
} catch {
|
|
68
|
+
if (!this.warned) logger.warn('Consumer.supervision', 'Status publication failed; consumption continues')
|
|
69
|
+
this.warned = true
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
async stop() {
|
|
73
|
+
clearInterval(this.timer)
|
|
74
|
+
if (this.pending) await this.pending
|
|
75
|
+
this.state = 'stopped'
|
|
76
|
+
await this.publish()
|
|
77
|
+
}
|
|
78
|
+
}
|
|
@@ -611,8 +611,8 @@ export class HttpClient {
|
|
|
611
611
|
return this.#requestWithFailover('GET', path, null, requestTimeoutMillis, affinityKey, retryKind, signal)
|
|
612
612
|
}
|
|
613
613
|
|
|
614
|
-
async post(path, body = null, requestTimeoutMillis = null, affinityKey = null, retryKind = null) {
|
|
615
|
-
return this.#requestWithFailover('POST', path, body, requestTimeoutMillis, affinityKey, retryKind)
|
|
614
|
+
async post(path, body = null, requestTimeoutMillis = null, affinityKey = null, retryKind = null, signal = null) {
|
|
615
|
+
return this.#requestWithFailover('POST', path, body, requestTimeoutMillis, affinityKey, retryKind, signal)
|
|
616
616
|
}
|
|
617
617
|
|
|
618
618
|
async put(path, body = null, requestTimeoutMillis = null, affinityKey = null, retryKind = null) {
|
package/client-v2/index.js
CHANGED
|
@@ -16,6 +16,11 @@
|
|
|
16
16
|
|
|
17
17
|
export { Queen } from './Queen.js'
|
|
18
18
|
export { Admin } from './admin/Admin.js'
|
|
19
|
+
// Locks and semaphores. Reached as `queen.lock()`, `queen.semaphore()` and
|
|
20
|
+
// `queen.locks`; the classes are exported for typing. `LOCK_NOT_HELD` is the
|
|
21
|
+
// `.code` of the error a guard throws for a handle that holds nothing, and of
|
|
22
|
+
// the reason `lock.signal` aborts with.
|
|
23
|
+
export { Lock, Locks, LOCK_NOT_HELD } from './locks/Locks.js'
|
|
19
24
|
// RAM-class queues (EPHEMERAL_QUEUES.md §4). Reached as `queen.ephemeral`; the
|
|
20
25
|
// class is exported for typing and for embedding it on a client of your own.
|
|
21
26
|
// The two `.code`s the family's 404s carry, which are NOT the same fact:
|
package/client-v2/kv/Kv.js
CHANGED
|
@@ -152,6 +152,21 @@ export const kvOp = {
|
|
|
152
152
|
if (opts.min !== undefined) base.min = opts.min
|
|
153
153
|
if (opts.max !== undefined) base.max = opts.max
|
|
154
154
|
return { base: withRequired(base, opts), expiry: opts }
|
|
155
|
+
},
|
|
156
|
+
|
|
157
|
+
check(ns, key, opts = {}) {
|
|
158
|
+
requireName(ns, 'ns')
|
|
159
|
+
requireName(key, 'key')
|
|
160
|
+
// A check IS its expect. One without says nothing, and the broker's
|
|
161
|
+
// `applied:true` to it would read as a guard that held.
|
|
162
|
+
if (!('expect' in opts)) {
|
|
163
|
+
throw new Error(
|
|
164
|
+
'kv: check needs `expect` — the version the key must be at, or 0 for a key that must not exist'
|
|
165
|
+
)
|
|
166
|
+
}
|
|
167
|
+
const base = { op: 'check', ns, key }
|
|
168
|
+
// No expiry: it writes nothing.
|
|
169
|
+
return { base: withRequired(withExpect(base, opts), opts), expiry: null }
|
|
155
170
|
}
|
|
156
171
|
}
|
|
157
172
|
|
|
@@ -408,6 +423,32 @@ export class Kv {
|
|
|
408
423
|
return res
|
|
409
424
|
}
|
|
410
425
|
|
|
426
|
+
/**
|
|
427
|
+
* Is the key still at this version? Writes nothing.
|
|
428
|
+
*
|
|
429
|
+
* const r = await kv.check('orders', '9137', { expect: row.version })
|
|
430
|
+
* if (!r.applied) console.log(r.reason, r.value) // somebody wrote since
|
|
431
|
+
*
|
|
432
|
+
* `expect: 0` asks the opposite: the key must not exist. A WriteResult like
|
|
433
|
+
* every write's, `applied` being "the precondition held"; when it did not,
|
|
434
|
+
* `reason` (`version`, `absent` or `exists`), `value` and `version` are what
|
|
435
|
+
* a reader would see.
|
|
436
|
+
*
|
|
437
|
+
* On its own it is a linearizable look. Where it earns its place is beside
|
|
438
|
+
* writes, with `required: true`: in a batch, or in a transaction's
|
|
439
|
+
* `.kv.check(...)`, it makes everything else commit only if a key the call
|
|
440
|
+
* does not write is unchanged — which is how a step is tied to a lock
|
|
441
|
+
* (`queen.lock`).
|
|
442
|
+
*
|
|
443
|
+
* Needs a broker at cluster version 5, the first with this operation. While
|
|
444
|
+
* a cluster is being upgraded to it the call answers 503
|
|
445
|
+
* `kv_check_needs_cluster_version_5`; an older broker answers 400
|
|
446
|
+
* `kv_unknown_op`.
|
|
447
|
+
*/
|
|
448
|
+
async check(ns, key, opts = {}) {
|
|
449
|
+
return this.#write(kvOp.check(ns, key, opts))
|
|
450
|
+
}
|
|
451
|
+
|
|
411
452
|
/**
|
|
412
453
|
* The idempotency marker, written the way it is actually used:
|
|
413
454
|
*
|