queen-mq 2.1.0 → 2.6.0

package/README.md

@@ -229,7 +229,11 @@ const queen = new Queen({
   timeoutMillis: 30000,
   retryAttempts: 3,
   loadBalancingStrategy: 'affinity',  // or 'round-robin', 'session'
-  enableFailover: true
+  enableFailover: true,
+  // Opt-in wire forms, off by default (see Binary Push, Push Compression and Framed Pop)
+  binaryPush: false,
+  pushCompression: false,
+  framedPop: false
 })
 ```
 
@@ -515,6 +519,54 @@ rate, and a broker outage shows up as slow pushes with bounded memory rather
 than as messages that quietly disappeared. `close()` flushes with a 30 second
 deadline and logs how many messages were left unsent if it expires.
 
+### Binary Push, Push Compression and Framed Pop
+
+Three opt-in forms of the push, pop and transaction requests. They change how
+the messages travel between the client and the broker and nothing else: the
+calls, their results, deduplication and the delivery guarantees stay the same.
+All three are off by default, and with them off the client sends and reads the
+JSON every broker knows.
+
+```javascript
+const queen = new Queen({
+  url: 'http://localhost:6632',
+  binaryPush: true,       // a push travels as a binary body
+  pushCompression: true,  // with binaryPush: framed and compressed here
+  framedPop: true         // a pop returns the stored records, unpacked here
+})
+```
+
+| Option | What it does |
+| --- | --- |
+| `binaryPush` | A push, direct or buffered, sends its items as one length-prefixed binary body instead of a JSON document. The answer is the same list, one entry per item. |
+| `pushCompression` | Needs `binaryPush`. The client frames each partition's batch the way the broker stores it, mints the message ids (UUIDv7) and compresses the batch with zstd, once. The broker checks the frames and, when it has nothing to add to them, stores those bytes without compressing them again. A transaction that pushes sends its pushes the same way, with the rest of the bundle as JSON in front of them. |
+| `framedPop` | `pop()` and `consume()` ask for the stored records (`frames=true`) and unpack them here into the same message objects, instead of a JSON answer that the broker renders message by message. |
+
+Turn them on when the broker's CPU or the network limits a high message rate:
+the framing, the compression and the unpacking then run in the clients, and the
+batches cross the network compressed. zstd is the one of Node.js (`node:zlib`),
+so there is nothing to install.
+
+**What stays JSON**, with no action from you:
+
+- A push with a `traceId` on one of its items.
+- A transaction with no push, or with a `traceId` on one of its pushes.
+- A pop the broker cannot answer in frames: a pop of a conflating group, and
+  every pop on a broker that runs with an encryption key
+  (`QUEEN_ENCRYPTION_KEY`). The client reads the JSON answer as always.
+- Acks, KV, timers, locks and the admin calls. Ephemeral queues have their own
+  routes and do not change either.
+
+A transaction's `results` come back in the order you added the operations, as
+with JSON. The broker answers a binary transaction with the acks first and the
+pushes after them, and the client puts them back.
+
+**Requires broker 2.3.0 or later** for `binaryPush` and `pushCompression`.
+A broker up to 2.2.0 answers 400 to the binary bodies and the client does not
+fall back to JSON, so turn the two on after every broker of the cluster is
+upgraded. `framedPop` is safe against any broker: an older one ignores the
+parameter and answers JSON.
+
 ### Dead Letter Queue
 
 ```javascript
@@ -747,6 +799,48 @@ other than the lost precondition still throws.
 
 ---
 
+## Reads by Offset
+
+Two queue methods read a partition's log by position, over `POST /api/v1/fetch` and
+`POST /api/v1/partitions/changed`. Neither takes a lease, moves a cursor or needs an ack: two
+readers asking the same question get the same answer, and no consumer group notices them. They
+suit a replay, an audit, a sink that keeps its own position, and a service that rebuilds an entity
+from its partition.
+
+```javascript
+// Every record of these partitions, from the oldest one still kept.
+const read = await queen.queue('orders').fetch(['customer-42', 'customer-7'])
+const part = read.get('customer-42')
+part.records          // [{ offset, transactionId, payload, ts }, ...] in offset order
+part.nextOffset       // where to start next time to read only what is new
+part.highWatermark    // the next offset the partition will assign
+part.logStartOffset   // the oldest offset retention kept
+
+// Only what came after an earlier read.
+const later = await queen.queue('orders').fetch(['customer-42'], {
+  offsets: { 'customer-42': part.nextOffset }
+})
+
+// The partitions written at or after a time, every page of them, in creation order.
+const { partitions, safeTime } = await queen.queue('orders').changedSince(new Date(Date.now() - 600_000))
+partitions.map(p => p.name)   // each is { name, id, lastOffset, logStart, lastWriteAt }
+
+// Next time, only what changed after that answer.
+const next = await queen.queue('orders').changedSince(safeTime)
+```
+
+`fetch` reads each partition up to where it ended when the call first read it: it asks for 1000
+partitions per request, and asks again until each one is read. A partition whose start retention
+deleted is read from its oldest kept record. It answers a `Map` in the order the partitions were
+named; a partition nobody wrote, or of a queue that does not exist, has no records.
+`options.maxBytes` caps the payload bytes of one request (default 8 MiB).
+
+`changedSince` takes a `Date`, epoch milliseconds or an ISO-8601 string; without one it lists every
+partition of the queue. `safeTime` is the broker's time up to which every write is counted in the
+answer: pass it as the next `since`, and never compare it with your own clock.
+
+---
+
 ## Examples
 
 ### Complete Pipeline with Consumer Groups
@@ -882,6 +976,16 @@ throws before any request when the builder has `.commitOnDelivery()`.
 
 `.autoAck()` is `consume()`'s ack after your handler and has no effect on a pop.
 
+### Reads by Offset
+
+```javascript
+const read = await queen.queue('q').fetch(['p1', 'p2'])                          // Map: partition -> { records, nextOffset, ... }
+const later = await queen.queue('q').fetch(['p1'], { offsets: { p1: 42 }, maxBytes: 1 << 20 })
+const { partitions, safeTime } = await queen.queue('q').changedSince(since)     // Date, epoch ms, ISO string; null = all
+```
+
+No lease, no cursor, nothing to ack. See [Reads by Offset](#reads-by-offset).
+
 ### Consume
 
 ```javascript
@@ -997,7 +1101,10 @@ await queen.close()  // Flush buffers and close connections
   loadBalancingStrategy: 'affinity',   // 'affinity' | 'round-robin' | 'session'
   affinityHashRing: 128,
   enableFailover: true,
-  healthRetryAfterMillis: 5000
+  healthRetryAfterMillis: 5000,
+  binaryPush: false,                   // a push as a binary body
+  pushCompression: false,              // with binaryPush: frames compressed in the client
+  framedPop: false                     // a pop as the stored records
 }
 ```
 

package/client-v2/Queen.js

@@ -179,7 +179,7 @@ export class Queen {
   }
 
   #createHttpClient() {
-    const { urls, timeoutMillis, retryAttempts, retryDelayMillis, loadBalancingStrategy, affinityHashRing, healthRetryAfterMillis, enableFailover, bearerToken, headers, hostHeader, retry429 } = this.#config
+    const { urls, timeoutMillis, retryAttempts, retryDelayMillis, loadBalancingStrategy, affinityHashRing, healthRetryAfterMillis, enableFailover, bearerToken, headers, hostHeader, retry429, binaryPush, pushCompression, framedPop } = this.#config
 
     if (urls.length === 1) {
       // Single server
@@ -191,7 +191,10 @@ export class Queen {
         bearerToken,
         headers,
         hostHeader,
-        retry429
+        retry429,
+        binaryPush,
+        pushCompression,
+        framedPop
       })
     }
 
@@ -209,7 +212,10 @@ export class Queen {
       bearerToken,
       headers,
       hostHeader,
-      retry429
+      retry429,
+      binaryPush,
+      pushCompression,
+      framedPop
     })
   }
 

package/client-v2/README.md

@@ -20,6 +20,8 @@ Welcome to Queen client! This is your friendly guide to mastering message queues
 - [Part 12: Message Tracing - Debug Your Workflows](#part-12-message-tracing---debug-your-workflows)
 - [Part 13: Callbacks & Error Handling](#part-13-callbacks--error-handling)
 - [Part 14: Graceful Shutdown](#part-14-graceful-shutdown)
+- [Part 15: Binary Push, Compression & Framed Pop](#part-15-binary-push-compression--framed-pop)
+- [Part 16: Reads by Offset - Replay Without a Lease](#part-16-reads-by-offset---replay-without-a-lease)
 - [Cheat Sheet](#cheat-sheet)
 
 ---
@@ -1607,6 +1609,116 @@ await queen.close()
 
 ---
 
+## Part 15: Binary Push, Compression & Framed Pop
+
+Everything so far travels as JSON. For a high message rate, the client has three
+opt-in forms of the push, pop and transaction requests. They change how the
+messages travel and nothing else: your calls, their results, deduplication and
+the delivery guarantees stay the same. All three are off by default.
+
+```javascript
+const queen = new Queen({
+  url: 'http://localhost:6632',
+  binaryPush: true,       // a push travels as a binary body
+  pushCompression: true,  // with binaryPush: framed and compressed here
+  framedPop: true         // a pop returns the stored records, unpacked here
+})
+
+// The rest of your code does not change.
+await queen.queue('events').partition('device-7').push([{ data: { n: 1 } }])
+await queen.queue('events').group('workers').consume(async (message) => {
+  console.log(message.data)
+})
+```
+
+### What Each Option Does
+
+| Option | What it does |
+| --- | --- |
+| `binaryPush` | A push, direct or buffered, sends its items as one length-prefixed binary body instead of a JSON document. The answer is the same list, one entry per item. |
+| `pushCompression` | Needs `binaryPush`. The client frames each partition's batch the way the broker stores it, mints the message ids (UUIDv7) and compresses the batch with zstd, once. The broker checks the frames and, when it has nothing to add to them, stores those bytes without compressing them again. A transaction that pushes sends its pushes the same way, with the rest of the bundle as JSON in front of them. |
+| `framedPop` | `pop()` and `consume()` ask for the stored records (`frames=true`) and unpack them here into the same message objects, instead of a JSON answer that the broker renders message by message. |
+
+Turn them on when the broker's CPU or the network limits a high message rate:
+the framing, the compression and the unpacking then run in the clients, and the
+batches cross the network compressed. zstd is the one of Node.js (`node:zlib`),
+so there is nothing to install.
+
+### What Stays JSON
+
+The client falls back by itself in these cases:
+
+- A push with a `traceId` on one of its items.
+- A transaction with no push, or with a `traceId` on one of its pushes.
+- A pop the broker cannot answer in frames: a pop of a conflating group, and
+  every pop on a broker that runs with an encryption key
+  (`QUEEN_ENCRYPTION_KEY`). The client reads the JSON answer as always.
+- Acks, KV, timers, locks and the admin calls. Ephemeral queues have their own
+  routes and do not change either.
+
+A transaction's `results` come back in the order you added the operations, as
+with JSON. The broker answers a binary transaction with the acks first and the
+pushes after them, and the client puts them back.
+
+### Which Broker
+
+`binaryPush` and `pushCompression` need broker 2.3.0 or later. A broker up
+to 2.2.0 answers 400 to the binary bodies and the client does not fall back to
+JSON, so turn the two on after every broker of the cluster is upgraded.
+`framedPop` is safe against any broker: an older one ignores the parameter and
+answers JSON.
+
+---
+
+## Part 16: Reads by Offset - Replay Without a Lease
+
+A pop hands messages to one consumer and waits for its ack. Sometimes you only
+want to read: rebuild an entity from its partition, replay a day for an audit,
+copy a queue somewhere else. Two queue methods do that, and neither takes a
+lease, moves a cursor or needs an ack. Two readers asking the same question get
+the same answer, and no consumer group notices them.
+
+```javascript
+// Every record of these partitions, from the oldest one still kept
+const read = await queen.queue('orders').fetch(['customer-42', 'customer-7'])
+
+for (const [partition, part] of read) {
+  console.log(partition, part.records.map(r => r.payload))
+  // part.records: [{ offset, transactionId, payload, ts }, ...]
+  // part.nextOffset: where to start next time to read only what is new
+}
+
+// Only what came after that read
+const later = await queen.queue('orders').fetch(['customer-42'], {
+  offsets: { 'customer-42': read.get('customer-42').nextOffset }
+})
+```
+
+`fetch` reads each partition up to where it ended when the call first read it.
+It asks for 1000 partitions per request and asks again until each one is read,
+and a partition whose start retention deleted is read from its oldest kept
+record. A partition nobody wrote, or of a queue that does not exist, has no
+records. `maxBytes` caps the payload bytes of one request (default 8 MiB).
+
+To find out which partitions to read, ask which ones were written since a time:
+
+```javascript
+const { partitions, safeTime } = await queen.queue('orders')
+  .changedSince(new Date(Date.now() - 10 * 60_000))
+
+partitions.map(p => p.name)   // each { name, id, lastOffset, logStart, lastWriteAt }
+
+// Next time, only what changed after that answer
+const next = await queen.queue('orders').changedSince(safeTime)
+```
+
+`changedSince` follows every page of the answer and returns the partitions in
+the order they were created; without a time it lists all of them. `safeTime` is
+the broker's time up to which every write is counted in the answer: pass it as
+the next `since`, and never compare it with your own clock.
+
+---
+
 ## Cheat Sheet
 
 ### Connection
@@ -1883,7 +1995,10 @@ await queen.close()
   loadBalancingStrategy: 'affinity',     // 'affinity' | 'round-robin' | 'session'
   affinityHashRing: 128,
   enableFailover: true,
-  healthRetryAfterMillis: 5000
+  healthRetryAfterMillis: 5000,
+  binaryPush: false,                     // a push as a binary body
+  pushCompression: false,                // with binaryPush: frames compressed in the client
+  framedPop: false                       // a pop as the stored records
 }
 ```
 

package/client-v2/builders/QueueBuilder.js

@@ -14,6 +14,7 @@ import { durableAddress } from '../buffer/sinks.js'
 import { QUEUE_DEFAULTS, CONSUME_DEFAULTS, POP_DEFAULTS } from '../utils/defaults.js'
 import { checkConflationResponse, CONFLATION_UNSUPPORTED } from '../utils/conflation.js'
 import { popSizing, parseAutopilotDecision } from '../utils/autopilot.js'
+import { fetchPartitions, partitionsChangedSince } from '../reads/offsetReads.js'
 import * as logger from '../utils/logger.js'
 
 // Per-item push statuses that mean the broker took the message: `queued`
@@ -632,6 +633,39 @@ export class QueueBuilder {
     await this.#bufferManager.flushBuffer(queueAddress)
   }
 
+  // ===========================
+  // Reads by offset
+  // ===========================
+
+  /**
+   * Every record of these partitions, read by offset: no lease, no cursor,
+   * nothing to ack. Answers a Map of partition name to
+   * `{ partition, records, highWatermark, logStartOffset, nextOffset }`.
+   *
+   *   const read = await queen.queue('orders').fetch(['customer-42', 'customer-7'])
+   *   read.get('customer-42').records.map(r => r.payload)
+   *
+   * `options.offsets` ({ name: offset } or a Map) starts a partition later than
+   * its oldest kept record, e.g. at the `nextOffset` of an earlier read.
+   * `options.maxBytes` caps the payload bytes of one call (default 8 MiB).
+   * See reads/offsetReads.js.
+   */
+  async fetch(partitions, options = {}) {
+    return fetchPartitions(this.#httpClient, this.#queueName, partitions, options)
+  }
+
+  /**
+   * The partitions written at or after `since` (a Date, epoch milliseconds or
+   * an ISO-8601 string; null lists them all): `{ partitions, safeTime }`.
+   * Pass `safeTime` as the next `since`.
+   *
+   *   const { partitions } = await queen.queue('orders').changedSince(new Date(Date.now() - 600_000))
+   *   partitions.map(p => p.name)
+   */
+  async changedSince(since = null) {
+    return partitionsChangedSince(this.#httpClient, this.#queueName, since)
+  }
+
   // ===========================
   // Dead Letter Queue Methods
   // ===========================

package/client-v2/http/HttpClient.js

@@ -3,6 +3,8 @@
  */
 
 import * as logger from '../utils/logger.js'
+import { binaryPushBody, transactionFrames, restoreResultOrder, PUSH_BINARY_CONTENT_TYPE } from './pushBinary.js'
+import { decodePopFrames, isPopFrames } from './popFrames.js'
 
 /**
  * The error a request ends with when its caller aborted it (a consumer being
@@ -142,6 +144,12 @@ export class HttpClient {
   // carries the same virtual Host.
   #hostOverride = null
   #pinnedDispatchers = new Map()
+  // The binary push body and the framed pop answer (pushBinary.js,
+  // popFrames.js). All three off by default: the client then sends and reads
+  // the JSON every broker knows.
+  #binaryPush
+  #pushCompression
+  #framedPop
 
   constructor(options = {}) {
     const {
@@ -169,7 +177,21 @@ export class HttpClient {
       // unbounded (retry forever, paced by backoff) unless maxAttempts is
       // explicitly set here, in which case it applies to both.
       // See PLAN_QUEEN_PROXY_CLOUD.md §4/§9 (client 429 backoff, blocker B4).
-      retry429 = {}
+      retry429 = {},
+      // A push to /api/v1/push goes out as the binary push body instead of
+      // JSON. It needs a broker that accepts that body. Items with a trace id
+      // stay JSON.
+      binaryPush = false,
+      // With binaryPush: the messages are framed as the broker stores them and
+      // each partition's batch is zstd-compressed once, here; the broker keeps
+      // those bytes as they are. The client then mints the message ids. A
+      // transaction that pushes sends its pushes the same way (the binary
+      // transaction body), and the rest of the bundle as JSON in front of them.
+      pushCompression = false,
+      // Pops ask the broker for the messages as the stored records
+      // (frames=true) and unpack them here. A broker that does not know the
+      // parameter answers JSON, which is read as always.
+      framedPop = false
     } = options
 
     this.#baseUrl = baseUrl
@@ -180,6 +202,9 @@ export class HttpClient {
     this.#enableFailover = enableFailover
     this.#bearerToken = bearerToken
     this.#retry429 = retry429 || {}
+    this.#binaryPush = binaryPush === true
+    this.#pushCompression = pushCompression === true
+    this.#framedPop = framedPop === true
 
     // A Host in `headers` can never go out over fetch(); rather than let it be
     // dropped, map it onto the supported mechanism and say so out loud.
@@ -408,11 +433,15 @@ export class HttpClient {
         }
       }
 
-      if (body) {
+      if (body instanceof Uint8Array) {
+        // Already encoded (the binary push body): sent as it is.
+        headers['Content-Type'] = PUSH_BINARY_CONTENT_TYPE
+        options.body = body
+      } else if (body) {
         options.body = JSON.stringify(body)
       }
 
-      const response = await fetch(requestUrl, options)
+      let response = await fetch(requestUrl, options)
 
       // The broker granted the leases of a pop when it sent these headers. From
       // here the caller's abort no longer cuts the read: the body says which
@@ -460,6 +489,19 @@ export class HttpClient {
         throw error
       }
 
+      // A pop that asked for frames: its first bytes say which answer came
+      // back, whatever a proxy made of the Content-Type on the way. The framed
+      // one is unpacked into the object the JSON answer parses to, so its
+      // readers do not tell the two apart; a JSON one (a broker that does not
+      // know the parameter) goes on to be read as always.
+      if (this.#framedPop && isPop(method, url)) {
+        const bytes = new Uint8Array(await response.arrayBuffer())
+        if (isPopFrames(bytes)) {
+          return await decodePopFrames(bytes)
+        }
+        response = new Response(bytes, { status: response.status, headers: response.headers })
+      }
+
       // Parse successful response
       const contentType = response.headers.get('content-type')
       const contentLength = response.headers.get('content-length')
@@ -608,10 +650,32 @@ export class HttpClient {
   // `signal`: aborting it closes the request, which then rejects with an
   // error carrying `aborted: true` -- never retried and never failed over.
   async get(path, requestTimeoutMillis = null, affinityKey = null, retryKind = null, signal = null) {
+    if (this.#framedPop && path.startsWith('/api/v1/pop')) {
+      path += `${path.includes('?') ? '&' : '?'}frames=true`
+    }
     return this.#requestWithFailover('GET', path, null, requestTimeoutMillis, affinityKey, retryKind, signal)
   }
 
   async post(path, body = null, requestTimeoutMillis = null, affinityKey = null, retryKind = null, signal = null) {
+    if (this.#binaryPush && path === '/api/v1/push' && body && Array.isArray(body.items)) {
+      // Encoded once, before any retry: every attempt sends the same bytes
+      // (with pushCompression, the same message ids). null = an item the
+      // binary body does not carry, and the push stays JSON.
+      body = (await binaryPushBody(body.items, this.#pushCompression)) ?? body
+    }
+    if (this.#binaryPush && this.#pushCompression && path === '/api/v1/transaction') {
+      // The pushes of a bundle as frames built and compressed here, the rest
+      // of it as the JSON head in front of them. null = no push, or one the
+      // binary body does not carry: the bundle stays JSON.
+      const framed = await transactionFrames(body)
+      if (framed) {
+        const answer = await this.#requestWithFailover('POST', path, framed.body, requestTimeoutMillis, affinityKey, retryKind, signal)
+        // The broker answers the head's operations before the pushes: the
+        // caller gets the results in the order of its own operations.
+        restoreResultOrder(answer, framed.order)
+        return answer
+      }
+    }
     return this.#requestWithFailover('POST', path, body, requestTimeoutMillis, affinityKey, retryKind, signal)
   }