queen-mq 1.2.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. package/README.md +71 -18
  2. package/client-v2/Queen.js +45 -13
  3. package/client-v2/README.md +26 -11
  4. package/client-v2/admin/Admin.js +0 -47
  5. package/client-v2/builders/QueueBuilder.js +33 -10
  6. package/client-v2/builders/TimerBuilder.js +2 -2
  7. package/client-v2/builders/TransactionBuilder.js +35 -9
  8. package/client-v2/consumer/ConsumerManager.js +67 -46
  9. package/client-v2/ephemeral/Ephemeral.js +7 -9
  10. package/client-v2/kv/Kv.js +4 -4
  11. package/client-v2/kv/expiry.js +1 -1
  12. package/client-v2/streams/Stream.js +8 -3
  13. package/client-v2/streams/helpers/rateLimiter.js +4 -4
  14. package/client-v2/streams/operators/GateOperator.js +9 -2
  15. package/client-v2/streams/operators/ReduceOperator.js +2 -2
  16. package/client-v2/streams/operators/WindowSessionOperator.js +3 -3
  17. package/client-v2/streams/runtime/Runner.js +111 -54
  18. package/client-v2/streams/runtime/cycle.js +4 -4
  19. package/client-v2/streams/runtime/register.js +1 -1
  20. package/client-v2/utils/conflation.js +0 -6
  21. package/client-v2/utils/consumerGroup.js +54 -0
  22. package/package.json +5 -8
  23. package/test-v2/_kvtimers.js +12 -13
  24. package/test-v2/ackwindow.js +12 -192
  25. package/test-v2/bootstrap.js +2 -2
  26. package/test-v2/conflation-unit/conflationWire.test.js +0 -12
  27. package/test-v2/consume.js +41 -1
  28. package/test-v2/consumer-unit/handlerError.test.js +161 -0
  29. package/test-v2/docs.js +5 -4
  30. package/test-v2/http-unit/pushStatus.test.js +142 -0
  31. package/test-v2/http-unit/renew.test.js +96 -0
  32. package/test-v2/kv-unit/timerWire.test.js +1 -1
  33. package/test-v2/kv-unit/txnWire.test.js +101 -2
  34. package/test-v2/kv.js +6 -5
  35. package/test-v2/pop.js +28 -1
  36. package/test-v2/retention.js +28 -7
  37. package/test-v2/run.js +50 -114
  38. package/test-v2/runner-unit/fatalExit.test.js +64 -0
  39. package/test-v2/semantics.js +20 -34
  40. package/test-v2/stream/_helpers.js +10 -19
  41. package/test-v2/stream/cron.js +1 -1
  42. package/test-v2/stream/gate.js +54 -0
  43. package/test-v2/stream/index.js +2 -0
  44. package/test-v2/stream/tumbling.js +13 -9
  45. package/test-v2/streams-unit/ack.test.js +61 -0
  46. package/test-v2/streams-unit/cycle.test.js +1 -1
  47. package/test-v2/streams-unit/e2e.test.js +6 -10
  48. package/test-v2/streams-unit/gate.test.js +189 -0
  49. package/test-v2/timers.js +6 -5
  50. package/test-v2/transaction.js +169 -0
  51. package/test-v2/watermark.js +38 -176
  52. package/test-v2/maintenance.js +0 -277
package/README.md CHANGED
@@ -6,7 +6,7 @@
6
6
 
7
7
  [![npm](https://img.shields.io/npm/v/queen-mq.svg)](https://www.npmjs.com/package/queen-mq)
8
8
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE.md)
9
- [![Node](https://img.shields.io/badge/node-%3E%3D22.0.0-brightgreen.svg)](https://nodejs.org/)
9
+ [![Node](https://img.shields.io/badge/node-%3E%3D24.0.0-brightgreen.svg)](https://nodejs.org/)
10
10
 
11
11
  [Quick Start](#quick-start) • [Complete Guide](client-v2/README.md) • [Examples](#examples) • [API Reference](#api-reference)
12
12
 
@@ -16,7 +16,7 @@
16
16
 
17
17
  ## What is Queen MQ?
18
18
 
19
- Queen MQ is a PostgreSQL-backed message queue system with a powerful feature set:
19
+ Queen MQ is a partitioned message queue broker that keeps its state in its own replicated log, with a powerful feature set:
20
20
 
21
21
  - **FIFO Partitions** - Unlimited ordered partitions within queues
22
22
  - **Consumer Groups** - Kafka-style consumer groups for scalability
@@ -40,7 +40,7 @@ This client provides a fluent, promise-based API for Node.js applications.
40
40
  npm install queen-mq
41
41
  ```
42
42
 
43
- **Requirements:** Node.js 22+
43
+ **Requirements:** Node.js 24+
44
44
 
45
45
  ---
46
46
 
@@ -133,20 +133,21 @@ await queen.queue('emails')
133
133
 
134
134
  ### Subscription Modes
135
135
 
136
- Control whether consumer groups process historical messages:
136
+ Control whether consumer groups process historical messages. A group's mode is fixed the first
137
+ time the group pops a queue; unset, it is the broker's `DEFAULT_SUBSCRIPTION_MODE`, which is `new`.
137
138
 
138
139
  ```javascript
139
- // Default: Process ALL messages (including backlog)
140
- await queen.queue('events')
141
- .group('batch-analytics')
142
- .consume(async (message) => { /* all messages */ })
143
-
144
- // Skip history, only new messages
140
+ // Default ('new'): start where the group first pops, skip what is already there
145
141
  await queen.queue('events')
146
142
  .group('realtime-monitor')
147
- .subscriptionMode('new')
148
143
  .consume(async (message) => { /* new only */ })
149
144
 
145
+ // Process ALL messages, including the backlog
146
+ await queen.queue('events')
147
+ .group('batch-analytics')
148
+ .subscriptionMode('all')
149
+ .consume(async (message) => { /* all messages */ })
150
+
150
151
  // Start from specific timestamp
151
152
  await queen.queue('events')
152
153
  .group('replay')
@@ -278,6 +279,51 @@ await queen.queue('tasks')
278
279
  })
279
280
  ```
280
281
 
282
+ **When the handler throws**, the consumer nacks what it was given (the message, or the whole batch)
283
+ and keeps consuming. The broker redelivers it, and moves it to the dead letter queue once the queue's
284
+ `retryLimit` is spent. With `.each()`, the rest of the popped batch is dropped after a nack: the
285
+ broker redelivers it too.
286
+
287
+ This is the same with `.autoAck(false)`. That setting hands the *success* path to your handler (it
288
+ acks), not the failure path: a handler that threw never got to settle its messages. A message your
289
+ handler acked before throwing is not affected: it is already settled, so the broker refuses
290
+ its nack.
291
+
292
+ To handle failures yourself, add `.onError(async (message, error) => { ... })`. The error then never
293
+ reaches the consumer, nothing is nacked for you, and the message is yours to ack, nack, or leave
294
+ until its lease expires. To stop consuming on an error, abort the `signal` you passed to
295
+ `consume(handler, { signal })` from inside `onError`.
296
+
297
+ ```javascript
298
+ // autoAck(false): the handler acks. A throw before the ack is nacked and retried.
299
+ await queen.queue('tasks')
300
+ .group('workers')
301
+ .autoAck(false)
302
+ .each() // one message per call; without it the handler gets the popped array
303
+ .consume(async (message) => {
304
+ await processTask(message.data)
305
+ await queen.ack(message, true)
306
+ })
307
+
308
+ // Your own failure policy: nack, then stop consuming.
309
+ const stop = new AbortController()
310
+ await queen.queue('tasks')
311
+ .group('workers')
312
+ .autoAck(false)
313
+ .each()
314
+ .consume(async (message) => {
315
+ await processTask(message.data)
316
+ await queen.ack(message, true)
317
+ }, { signal: stop.signal })
318
+ .onError(async (message, error) => {
319
+ await queen.ack(message, false, { error: error.message })
320
+ stop.abort()
321
+ })
322
+ ```
323
+
324
+ Earlier versions stopped the consumer on a throw under `.autoAck(false)`: `consume()` rejected after
325
+ the first failure, and the message stayed leased until its lease expired.
326
+
281
327
  ### Pop Messages (On-Demand Processing)
282
328
 
283
329
  ```javascript
@@ -494,9 +540,9 @@ Both surfaces are **always there**. There is nothing to enable: kv and timers ar
494
540
  broker the way push and pop are, on every cell that runs it. There is no capability to probe and
495
541
  no 404 that means "this cell does not have the feature" — a 404 from these routes is a bug.
496
542
 
497
- What an operator can still do is **pause** them, with the runtime kill switch in
498
- `queen.system_state` (`kv_enabled`, `timers_schedule_enabled`, `timers_fire_enabled`) — the same
499
- class of lever as maintenance mode, pulled live during an incident and expected to be pulled back.
543
+ What an operator can still do is **pause** them, with the broker's runtime kill switches
544
+ (`kv_enabled`, `timers_schedule_enabled`, `timers_fire_enabled`) — a lever pulled live during an
545
+ incident and expected to be pulled back.
500
546
  A paused surface answers `503` with `Retry-After` and `error: 'kv_disabled'` / `'timers_disabled'`,
501
547
  which this client retries like any other 5xx. Inside a transaction it is a `403` on the `kv` or
502
548
  `timers` rider instead, so a bundle holding messages does not spin forever on a paused cell.
@@ -581,7 +627,7 @@ the retry budget.
581
627
 
582
628
  Durations that can be sub-second are in **milliseconds** (`delayMs`), the ones that cannot are in
583
629
  **seconds** (`ttlSeconds`). Only relative delays exist, because there is one clock and it is the
584
- database's. A delay in the past is legal and fires on the first cycle.
630
+ broker's. A delay in the past is legal and fires on the first cycle.
585
631
 
586
632
  `deliverAt` is **"not before"**, never "exactly at".
587
633
 
@@ -768,6 +814,11 @@ await queen.ack(message, false, { error: 'reason' })
768
814
  await queen.ack([msg1, msg2], true) // Batch ack
769
815
  ```
770
816
 
817
+ An ack is judged against the lease of the consumer group the message was popped under. `queen.ack()`
818
+ and `transaction().ack()` take that group from the message (`message.consumerGroup`, which every pop
819
+ returns) unless you name one: `{ group }` on `queen.ack()`, `{ consumerGroup }` or `{ group }` on a
820
+ transaction. A batch ack carries a single group, so `queen.ack()` throws on a batch that mixes groups.
821
+
771
822
  ### Transactions
772
823
 
773
824
  ```javascript
@@ -816,11 +867,14 @@ await queen.timer(q).list({ limit: 50, after }) // {rows, trunc
816
867
  ### Lease Renewal
817
868
 
818
869
  ```javascript
819
- await queen.renew(message)
820
- await queen.renew([msg1, msg2, msg3])
870
+ await queen.renew(message) // {leaseId, success, newExpiresAt, renewed}
871
+ await queen.renew([msg1, msg2, msg3]) // one result per distinct lease
821
872
  await queen.queue('q').renewLease(true, 60000).consume(async (msg) => { /* auto-renew */ })
822
873
  ```
823
874
 
875
+ `success: false` (with an `error`) means nothing was renewed: the lease expired, an ack or nack
876
+ already released it, or it never existed. The messages it covered may already be redelivered.
877
+
824
878
  ### Buffering
825
879
 
826
880
  ```javascript
@@ -970,4 +1024,3 @@ const messages: Message<OrderData>[] = await queen.queue('orders').pop()
970
1024
  ## License
971
1025
 
972
1026
  Apache 2.0 - See [LICENSE.md](../LICENSE.md)
973
-
@@ -15,6 +15,7 @@ import { StreamBuilder } from './stream/StreamBuilder.js'
15
15
  import { StreamConsumer } from './stream/StreamConsumer.js'
16
16
  import { Admin } from './admin/Admin.js'
17
17
  import { popAutopilotDisabledByEnv } from './utils/autopilot.js'
18
+ import { consumerGroupOf, sharedConsumerGroupOf } from './utils/consumerGroup.js'
18
19
  import { CLIENT_DEFAULTS } from './utils/defaults.js'
19
20
  import { validateUrl, validateUrls } from './utils/validation.js'
20
21
  import * as logger from './utils/logger.js'
@@ -28,9 +29,9 @@ const CLOSE_FLUSH_DEADLINE_MILLIS = 30000
28
29
  // Both /api/v1/ack and /api/v1/ack/batch respond with a top-level JSON array,
29
30
  // one item per acknowledgment in request order:
30
31
  // [{index, transactionId, success, error, queueName, partitionName, leaseReleased, dlq}]
31
- // (see queen.ack_messages_v2 / routes/ack.cpp). A rejected ack/nack (e.g.
32
- // "Invalid or expired lease") still arrives as HTTP 200 with success=false on
33
- // the item, so the per-item flag is the only signal that the broker accepted it.
32
+ // A rejected ack/nack (e.g. "Invalid or expired lease") still arrives as HTTP
33
+ // 200 with success=false on the item, so the per-item flag is the only signal
34
+ // that the broker accepted it.
34
35
 
35
36
  function normalizeAckItem(item, index) {
36
37
  if (item === null || typeof item !== 'object') {
@@ -61,6 +62,30 @@ function parseAckResults(result, expected) {
61
62
  throw new Error('Unexpected ack response format: missing per-item result array')
62
63
  }
63
64
 
65
+ // POST /api/v1/lease/:leaseId/extend answers HTTP 200 whether or not anything
66
+ // was renewed, so `success` in the body is the only signal:
67
+ // 1.x and 2.x brokers: {leaseId, success, renewed, newExpiresAt, expiresAt, lease_expires_at}
68
+ // the C++ broker: [{index, leaseId, success, error, expiresAt}]
69
+ // success:false means the lease is gone (expired, released by an ack or nack,
70
+ // or never existed): nothing was extended, and the messages it covered can
71
+ // already be on their way to another consumer.
72
+ function parseRenewResult(result, leaseId) {
73
+ const item = Array.isArray(result) ? result[0] : result
74
+ if (item === null || typeof item !== 'object') {
75
+ return { leaseId, success: false, newExpiresAt: null, error: 'Unexpected lease renewal response' }
76
+ }
77
+ const newExpiresAt = item.newExpiresAt ?? item.expiresAt ?? item.lease_expires_at ?? null
78
+ const success = typeof item.success === 'boolean' ? item.success : newExpiresAt !== null
79
+ const outcome = { leaseId, success, newExpiresAt }
80
+ if (typeof item.renewed === 'number') outcome.renewed = item.renewed
81
+ if (!success) {
82
+ outcome.error = typeof item.error === 'string' && item.error.length > 0
83
+ ? item.error
84
+ : 'Lease not renewed: it expired, was released by an ack or nack, or does not exist'
85
+ }
86
+ return outcome
87
+ }
88
+
64
89
  export class Queen {
65
90
  #httpClient
66
91
  #bufferManager
@@ -438,11 +463,15 @@ export class Queen {
438
463
  })
439
464
  }
440
465
 
466
+ // The group the messages were popped under unless the caller names one:
467
+ // the lease every ack has to match belongs to it (utils/consumerGroup.js).
468
+ const consumerGroup = context.group || sharedConsumerGroupOf(message)
469
+
441
470
  // Call batch ack endpoint
442
471
  try {
443
472
  const result = await this.#httpClient.post('/api/v1/ack/batch', {
444
473
  acknowledgments,
445
- consumerGroup: context.group || null
474
+ consumerGroup
446
475
  })
447
476
 
448
477
  const results = parseAckResults(result, acknowledgments.length)
@@ -487,7 +516,9 @@ export class Queen {
487
516
  partitionId,
488
517
  status: statusStr,
489
518
  error: context.error || null,
490
- consumerGroup: context.group || null
519
+ // The group the message was popped under unless the caller names one
520
+ // (utils/consumerGroup.js).
521
+ consumerGroup: context.group || consumerGroupOf(message)
491
522
  }
492
523
 
493
524
  if (leaseId) body.leaseId = leaseId
@@ -529,8 +560,8 @@ export class Queen {
529
560
  }
530
561
 
531
562
  // Dedupe: with v4 multi-partition pop, all messages in one batch share
532
- // the same leaseId (one renew_lease_v2 call extends every claimed
533
- // partition_consumers row). Without this, callers passing the full
563
+ // the same leaseId (one renew call extends the lease of every claimed
564
+ // partition). Without this, callers passing the full
534
565
  // messages array would issue N redundant identical HTTP calls.
535
566
  leaseIds = [...new Set(leaseIds)]
536
567
 
@@ -545,12 +576,13 @@ export class Queen {
545
576
  for (const leaseId of leaseIds) {
546
577
  try {
547
578
  const result = await this.#httpClient.post(`/api/v1/lease/${leaseId}/extend`, {})
548
- results.push({
549
- leaseId,
550
- success: true,
551
- newExpiresAt: result.leaseId ? result.newExpiresAt : result.lease_expires_at
552
- })
553
- logger.log('Queen.renew', { leaseId, success: true })
579
+ const outcome = parseRenewResult(result, leaseId)
580
+ results.push(outcome)
581
+ if (outcome.success) {
582
+ logger.log('Queen.renew', { leaseId, success: true, renewed: outcome.renewed })
583
+ } else {
584
+ logger.error('Queen.renew', { leaseId, error: outcome.error })
585
+ }
554
586
  } catch (error) {
555
587
  results.push({ leaseId, success: false, error: error.message })
556
588
  logger.error('Queen.renew', { leaseId, error: error.message })
@@ -60,7 +60,7 @@ When connecting to multiple Queen servers, you can choose how requests are distr
60
60
 
61
61
  #### Affinity Mode (Recommended for Production)
62
62
 
63
- Uses consistent hashing with virtual nodes to route consumer groups to the same backend server. This optimizes database queries by consolidating poll intentions.
63
+ Uses consistent hashing with virtual nodes to route consumer groups to the same backend server.
64
64
 
65
65
  ```javascript
66
66
  const queen = new Queen({
@@ -444,28 +444,31 @@ When a consumer group first subscribes to a queue, should it process **all histo
444
444
  - Join a stream at a specific point in time
445
445
  - Skip historical data for new analytics consumers
446
446
 
447
- ### Default Behavior (All Messages)
447
+ ### Default Behavior (New Messages)
448
448
 
449
- By default, consumer groups start from the **beginning** and process all messages:
449
+ By default, a consumer group starts where it first pops the queue and skips the messages that were
450
+ already there. That default is the broker's `DEFAULT_SUBSCRIPTION_MODE`, which is `new`:
450
451
 
451
452
  ```javascript
452
- // This consumer group gets ALL messages, including historical ones
453
+ // This consumer group gets only messages that arrive after it first pops
453
454
  await queen
454
455
  .queue('events')
455
456
  .group('new-analytics')
457
+ .each()
456
458
  .consume(async (message) => {
457
459
  console.log('Processing:', message.data)
458
460
  })
459
461
  ```
460
462
 
461
- **Server Default:** The server can be configured to change this default behavior:
463
+ To process the backlog too, ask for it with `.subscriptionMode('all')`, or start the broker with
464
+ `DEFAULT_SUBSCRIPTION_MODE=all` to make that the default for every new group:
462
465
  ```bash
463
- # Make all new consumer groups skip history by default
464
- export DEFAULT_SUBSCRIPTION_MODE="new"
465
- ./bin/queen-server
466
+ export DEFAULT_SUBSCRIPTION_MODE="all"
467
+ ./bin/queen
466
468
  ```
467
469
 
468
- When `DEFAULT_SUBSCRIPTION_MODE="new"` is set, new consumer groups automatically skip historical messages unless you explicitly override with `.subscriptionMode('all')`.
470
+ The mode is stored with the group the first time it pops; changing the default later does not move
471
+ an existing group.
469
472
 
470
473
  ### Subscription Mode: 'new'
471
474
 
@@ -1004,8 +1007,9 @@ const message = messages[0]
1004
1007
 
1005
1008
  // Start long processing
1006
1009
  const timer = setInterval(async () => {
1007
- await queen.renew(message) // Extend lease
1008
- console.log('Lease renewed!')
1010
+ const { success, newExpiresAt, error } = await queen.renew(message) // Extend lease
1011
+ if (success) console.log('Lease renewed until', newExpiresAt)
1012
+ else console.warn('Lease lost:', error) // expired or already released: it will be redelivered
1009
1013
  }, 30000) // Every 30 seconds
1010
1014
 
1011
1015
  try {
@@ -1432,6 +1436,16 @@ await queen.queue('reports').consume(async (msg) => {
1432
1436
 
1433
1437
  Sometimes you need more control over what happens when messages succeed or fail.
1434
1438
 
1439
+ ### What a Throw Does
1440
+
1441
+ Without callbacks, a handler that throws gets what it was given nacked (the message with `.each()`,
1442
+ otherwise the whole popped batch), and the consumer keeps going. The broker redelivers it, and moves
1443
+ it to the DLQ once the queue's `retryLimit` is spent. That holds with `.autoAck(false)` too: there
1444
+ your handler acks on success, but a handler that threw never got to settle anything.
1445
+
1446
+ With `.onError()` the failure is yours: nothing is nacked on your behalf. Ack, nack or DLQ the
1447
+ message in the callback, or it stays leased until its lease expires.
1448
+
1435
1449
  ### Success Callback
1436
1450
 
1437
1451
  ```javascript
@@ -1457,6 +1471,7 @@ await queen
1457
1471
  .onError(async (message, error) => {
1458
1472
  console.error('Failed:', error.message)
1459
1473
  // Log to external service, send alert, etc.
1474
+ await queen.ack(message, false) // with onError, nothing is nacked for you
1460
1475
  })
1461
1476
  ```
1462
1477
 
@@ -359,44 +359,6 @@ export class Admin {
359
359
  return this.#httpClient.get('/metrics')
360
360
  }
361
361
 
362
- /**
363
- * Get push maintenance mode status
364
- * @returns {Promise<object>}
365
- */
366
- async getMaintenanceMode() {
367
- logger.log('Admin.getMaintenanceMode', {})
368
- return this.#httpClient.get('/api/v1/system/maintenance')
369
- }
370
-
371
- /**
372
- * Set push maintenance mode
373
- * @param {boolean} enabled - Enable or disable maintenance mode
374
- * @returns {Promise<object>}
375
- */
376
- async setMaintenanceMode(enabled) {
377
- logger.log('Admin.setMaintenanceMode', { enabled })
378
- return this.#httpClient.post('/api/v1/system/maintenance', { enabled })
379
- }
380
-
381
- /**
382
- * Get pop maintenance mode status
383
- * @returns {Promise<object>}
384
- */
385
- async getPopMaintenanceMode() {
386
- logger.log('Admin.getPopMaintenanceMode', {})
387
- return this.#httpClient.get('/api/v1/system/maintenance/pop')
388
- }
389
-
390
- /**
391
- * Set pop maintenance mode
392
- * @param {boolean} enabled - Enable or disable pop maintenance mode
393
- * @returns {Promise<object>}
394
- */
395
- async setPopMaintenanceMode(enabled) {
396
- logger.log('Admin.setPopMaintenanceMode', { enabled })
397
- return this.#httpClient.post('/api/v1/system/maintenance/pop', { enabled })
398
- }
399
-
400
362
  /**
401
363
  * Get system metrics (CPU, memory, connections, etc.)
402
364
  * @param {object} params - Query parameters (from, to, etc.)
@@ -419,15 +381,6 @@ export class Admin {
419
381
  return this.#httpClient.get(`/api/v1/analytics/worker-metrics${queryString}`)
420
382
  }
421
383
 
422
- /**
423
- * Get PostgreSQL statistics
424
- * @returns {Promise<object>}
425
- */
426
- async getPostgresStats() {
427
- logger.log('Admin.getPostgresStats', {})
428
- return this.#httpClient.get('/api/v1/analytics/postgres-stats')
429
- }
430
-
431
384
  // ===========================
432
385
  // Helper Methods
433
386
  // ===========================
@@ -16,6 +16,14 @@ import { checkConflationResponse, CONFLATION_UNSUPPORTED } from '../utils/confla
16
16
  import { popSizing, parseAutopilotDecision } from '../utils/autopilot.js'
17
17
  import * as logger from '../utils/logger.js'
18
18
 
19
+ // Per-item push statuses that mean the broker took the message: `queued`
20
+ // (stored), and `buffered` (a 1.x broker wrote it to its failover file and
21
+ // replays it into storage itself). Every other status is a rejection: `error`
22
+ // from a 2.x broker, which carries no per-item message, and `failed` from a
23
+ // 1.x broker, which does. Unknown statuses count as rejections too: a status
24
+ // this client cannot read is not a message it can report as stored.
25
+ const ACCEPTED_PUSH_STATUSES = new Set(['queued', 'buffered'])
26
+
19
27
  export class QueueBuilder {
20
28
  #queen
21
29
  #httpClient
@@ -849,24 +857,40 @@ class PushBuilder {
849
857
  for (let i = 0; i < results.length; i++) {
850
858
  const result = results[i]
851
859
  const originalItem = this.#formattedItems[i]
860
+ const status = result && typeof result === 'object' ? result.status : undefined
852
861
 
853
- if (result.status === 'duplicate') {
862
+ if (status === 'duplicate') {
854
863
  duplicates.push({ ...originalItem, result })
855
- } else if (result.status === 'failed') {
856
- failed.push({ ...originalItem, result, error: result.error })
857
- } else if (result.status === 'queued') {
864
+ } else if (ACCEPTED_PUSH_STATUSES.has(status)) {
858
865
  successful.push({ ...originalItem, result })
866
+ } else {
867
+ const error = result && typeof result.error === 'string' && result.error.length > 0
868
+ ? result.error
869
+ : `push rejected by the broker (item status: ${status === undefined ? 'missing' : JSON.stringify(status)})`
870
+ failed.push({ ...originalItem, result, error })
859
871
  }
860
872
  }
861
873
 
874
+ // One error for the whole call, built once so the callback and the
875
+ // throw describe the failure the same way. `results` is the broker's
876
+ // per-item answer in input order: the items that did go through are
877
+ // in it too, which is what a caller needs to retry only the rest.
878
+ let failure = null
879
+ if (failed.length > 0) {
880
+ failure = new Error(failed.length === 1
881
+ ? failed[0].error
882
+ : `${failed.length} of ${results.length} pushed items rejected: ${failed[0].error}`)
883
+ failure.results = results
884
+ logger.error('PushBuilder.execute', { status: 'failed', count: failed.length, total: results.length, error: failed[0].error })
885
+ }
886
+
862
887
  // Call appropriate callbacks
863
888
  if (duplicates.length > 0 && this.#onDuplicateCallback) {
864
889
  await this.#onDuplicateCallback(duplicates, new Error('Duplicate transaction IDs detected'))
865
890
  }
866
891
 
867
- if (failed.length > 0 && this.#onErrorCallback) {
868
- const error = new Error(failed[0].error || 'Push failed')
869
- await this.#onErrorCallback(failed, error)
892
+ if (failure && this.#onErrorCallback) {
893
+ await this.#onErrorCallback(failed, failure)
870
894
  }
871
895
 
872
896
  if (successful.length > 0 && this.#onSuccessCallback) {
@@ -874,9 +898,8 @@ class PushBuilder {
874
898
  }
875
899
 
876
900
  // Only throw if no error callback is defined
877
- if (failed.length > 0 && !this.#onErrorCallback) {
878
- logger.error('PushBuilder.execute', { status: 'failed', count: failed.length })
879
- throw new Error(failed[0].error || 'Push failed')
901
+ if (failure && !this.#onErrorCallback) {
902
+ throw failure
880
903
  }
881
904
 
882
905
  logger.log('PushBuilder.execute', { status: 'success', successful: successful.length, duplicates: duplicates.length, failed: failed.length })
@@ -24,8 +24,8 @@
24
24
  * bundle's fate; see TransactionBuilder, which says so where it happens.)
25
25
  *
26
26
  * 2. ONLY RELATIVE DURATIONS, IN MILLISECONDS (§4.2, §20.6). `delayMs`, never
27
- * `delaySeconds` and never an absolute instant: `deliver_at` is computed in
28
- * Postgres, so there is ONE clock and no broker's skew can enter. The rule
27
+ * `delaySeconds` and never an absolute instant: `deliver_at` is computed on
28
+ * the broker's clock, so there is ONE clock and no client's skew can enter. The rule
29
29
  * of the product is "durations that can be sub-second are in milliseconds,
30
30
  * the ones that cannot are in seconds" -- a 250 ms retry backoff is a real
31
31
  * and central use of timers, which is why this wire is the millisecond one.
@@ -30,6 +30,7 @@ import { generateUUID } from './QueueBuilder.js'
30
30
  import { isValidUUID } from '../utils/validation.js'
31
31
  import { kvOp, materializeKvOp } from '../kv/Kv.js'
32
32
  import { TimerBuilder } from './TimerBuilder.js'
33
+ import { consumerGroupOf } from '../utils/consumerGroup.js'
33
34
 
34
35
  export class TransactionBuilder {
35
36
  #httpClient
@@ -46,10 +47,21 @@ export class TransactionBuilder {
46
47
  this.#httpClient = httpClient
47
48
  }
48
49
 
50
+ /**
51
+ * Ack popped messages as part of this transaction.
52
+ *
53
+ * The consumer group defaults to the one each message was popped under
54
+ * (`message.consumerGroup`, which every pop answers with), because the lease
55
+ * an ack has to match belongs to that group: an ack that names no group is
56
+ * judged in queue mode, and a message popped by a group then fails with
57
+ * `rejected_ack`. Pass `{ consumerGroup }` (or `{ group }`, the key
58
+ * `queen.ack()` takes) to override it.
59
+ */
49
60
  ack(messages, status = 'completed', context = {}) {
50
61
  const msgs = Array.isArray(messages) ? messages : [messages]
51
-
52
- logger.log('TransactionBuilder.ack', { count: msgs.length, status, consumerGroup: context.consumerGroup })
62
+ const explicitGroup = context.consumerGroup || context.group || null
63
+
64
+ logger.log('TransactionBuilder.ack', { count: msgs.length, status, consumerGroup: explicitGroup })
53
65
 
54
66
  msgs.forEach(msg => {
55
67
  const transactionId = typeof msg === 'string' ? msg : (msg.transactionId || msg.id)
@@ -72,9 +84,18 @@ export class TransactionBuilder {
72
84
  status
73
85
  }
74
86
 
75
- // Add consumerGroup if provided in context
76
- if (context.consumerGroup) {
77
- operation.consumerGroup = context.consumerGroup
87
+ const consumerGroup = explicitGroup || consumerGroupOf(msg)
88
+ if (consumerGroup) {
89
+ operation.consumerGroup = consumerGroup
90
+ }
91
+
92
+ // Each ack names its own lease. A Queen 2 broker fences an ack with the
93
+ // lease the operation carries, and lends it one from requiredLeases only
94
+ // when the bundle names a single lease: in a bundle spanning two leases,
95
+ // an ack without its own would be applied whoever holds its partition
96
+ // now. 1.x brokers read the operation's lease before requiredLeases too.
97
+ if (leaseId) {
98
+ operation.leaseId = leaseId
78
99
  }
79
100
 
80
101
  this.#operations.push(operation)
@@ -204,7 +225,7 @@ export class TransactionBuilder {
204
225
  *
205
226
  * `putIfAbsent` with `required:true`, so a marker that already exists ABORTS
206
227
  * the transaction: the push and the ack roll back together with it. That is
207
- * what makes "the email is sent exactly once" a property of the database
228
+ * what makes "the email is sent exactly once" a property of the broker
208
229
  * rather than a hope about redelivery.
209
230
  *
210
231
  * The verdict is RETURNED by `commit()`, not thrown -- see there.
@@ -303,9 +324,14 @@ export class TransactionBuilder {
303
324
 
304
325
  logger.error('TransactionBuilder.commit', { error: result.error, reason: result.reason })
305
326
  const error = new Error(result.error || 'Transaction failed')
306
- // The closed-taxonomy code travels ON the error, so no caller ever has
307
- // to match the message: bad_request | duplicate | ack_rejected |
308
- // timer_horizon_exceeded | payload_too_large | misaligned | db_error.
327
+ // The broker's reason code travels ON the error, so no caller ever has
328
+ // to match the message. The ones worth branching on from a 2.x broker:
329
+ // bad_request, duplicate (a pushed transactionId is already stored),
330
+ // rejected_ack (an acked message is no longer leased by this worker, or
331
+ // was popped under a different consumer group than the ack names), and
332
+ // too_large. A 1.x broker spells the ack one ack_rejected, and also
333
+ // sends timer_horizon_exceeded, payload_too_large, misaligned and
334
+ // db_error.
309
335
  if (result.reason) error.reason = result.reason
310
336
  error.result = result
311
337
  throw error