queen-mq 1.0.6 → 1.2.0

package/README.md

@@ -154,6 +154,53 @@ await queen.queue('events')
   .consume(async (message) => { /* from timestamp */ })
 ```
 
+### Conflation (Last-Value Delivery)
+
+For command-style queues where one partition is one logical task key — "recompute
+customer 42", "this entity is dirty" — only the newest pending message matters.
+`.conflation(true)` makes a pop of a partition deliver exactly one message, the
+newest visible one, and commit past everything it skipped:
+
+```javascript
+// A backlog of 4 000 recompute requests across 12 entities becomes
+// 12 handler calls, each with the freshest input.
+await queen.queue('recompute')
+  .group('workers')
+  .conflation(true)
+  .partitions(64)
+  .consume(async (message) => {
+    await recompute(message.data.entityId)
+  })
+```
+
+The guarantee: **after the last push to a partition, at least one handler run
+starts after that push committed.** Nothing is deleted — conflation is a delivery
+policy, not compaction; retention still governs what is stored, and a
+non-conflating group on the same queue still sees every message.
+
+Notes:
+
+- It is a property of the **consumer group**, fixed when that group first
+  registers on the queue. A later consumer declaring the opposite does not flip
+  it — the stored value wins, that consumer keeps working, and the SDK warns
+  once per (queue, group).
+- Skipping is per **partition**, so partitioning is the key: one partition = one
+  logical key is the contract this workload has to hold up.
+- A conflating pop returns at most one message per partition, so **partitions**
+  size the round-trip, not `batch`. Left unset, the broker uses `.batch(N)` as
+  the partition cap; either way it is clamped to 64, so a conflating pop returns
+  at most 64 messages per round-trip whatever `batch` says.
+- Refused with 400 by the broker without a `.group(...)`, and together with
+  `.autoAck(true)` (auto-ack commits at delivery, which would turn the guarantee
+  above into at-most-once).
+- Requires broker **>= 1.1.0**. An older broker ignores the flag and would
+  quietly deliver the whole backlog, so the SDK raises
+  `conflation was requested but this broker did not apply it` on the first
+  response that does not echo it — before any message is processed.
+- `admin.getQueueDepth(queue, group)` reports `effectivePending` (handler calls
+  still owed) next to `pending` (log positions still to retire). For a
+  conflating group `pending: 4000000, effectivePending: 12` is healthy.
+
 ---
 
 ## Connection Options
@@ -286,12 +333,64 @@ per-tenant work queues, per-device telemetry). Reduces network round-trips
 from O(P) to O(P / N) while preserving per-partition FIFO ordering.
 
 **When not to use:** few partitions, or each one busy enough to fill
-`batch(B)` on its own. Default is `partitions(1)` which preserves the
-legacy single-partition behaviour.
+`batch(B)` on its own. Leaving `.partitions()` unset hands the sweep width to
+the broker (see Pop Autopilot below); `.partitions(1)` pins the legacy
+single-partition behaviour.
 
 `.partitions(N)` only applies to **wildcard** pops; specifying
 `.partition('name')` ignores the cap.
 
+### Pop Autopilot (Let the Broker Size the Pop)
+
+Since 1.2, `batch` and `partitions` that you do **not** set are chosen by the
+broker, per pop, from state the client cannot see: how many partitions of the
+group are ready, how old their oldest ready message is, how fast messages are
+arriving. The knobs you *do* set are never touched.
+
+```javascript
+// Both knobs are the broker's: it picks the sweep width and the budget.
+await queen.queue('events').group('workers')
+  .consume(async (msgs) => { /* ... */ })
+
+// One knob pinned, one delegated: this consumer stays on one partition
+// forever, and the broker sizes the batch for it.
+await queen.queue('events').group('workers').partitions(1)
+  .consume(async (msgs) => { /* ... */ })
+```
+
+The request carries `autopilot=true` and simply omits the delegated knobs.
+Setting both leaves nothing to decide, so nothing changes on the wire at all.
+
+**Two ways to switch it off**, both restoring the previous client-side
+defaults (batch 1, partitions 1) byte for byte:
+
+```javascript
+await queen.queue('events').autopilot(false).consume(handler)
+```
+
+```bash
+QUEEN_SDK_POP_AUTOPILOT=off   # whole process, read once at client creation
+```
+
+**What the broker chose** rides back on the response and is there for the
+reading, along with an optional pacing hint the consume loop honours in place
+of its own delay between empty polls:
+
+```javascript
+const { messages, autopilot } = await queen.queue('events').group('workers').popResult()
+if (autopilot) {
+  console.log(`${autopilot.partitions} partitions, batch ${autopilot.batch}, ` +
+              `poll again in ${autopilot.waitMillis}ms`)
+}
+```
+
+**Requires broker >= 1.2.** An older broker ignores the parameter, so the
+omitted knobs take *its* defaults (batch 200, partitions 1) instead of the old
+client-side ones. That is a sizing difference and nothing else — no message is
+lost, reordered or duplicated — so unlike conflation it degrades silently and
+on purpose. Pin the values explicitly, or turn autopilot off, if you need the
+old numbers against an old broker.
+
 ### Transactions (Atomic Operations)
 
 ```javascript
@@ -644,10 +743,11 @@ await queen.queue('q').buffer({ messageCount: 100, timeMillis: 1000 }).push([...
 ### Pop
 
 ```javascript
-const msgs = await queen.queue('q').pop()
+const msgs = await queen.queue('q').pop()                            // broker-sized (see Pop Autopilot)
 const msgs = await queen.queue('q').batch(10).pop()
 const msgs = await queen.queue('q').batch(10).wait(true).pop()
 const msgs = await queen.queue('q').batch(200).partitions(50).pop()  // multi-partition pop
+const { messages, autopilot } = await queen.queue('q').popResult()   // + what the broker chose
 ```
 
 ### Consume
@@ -781,7 +881,8 @@ await queen.close()  // Flush buffers and close connections
 ```javascript
 {
   concurrency: 1,
-  batch: 1,
+  batch: 1,              // autopilot OFF only -- unset means the broker sizes it
+  partitions: 1,         // autopilot OFF only -- unset means the broker sizes it
   autoAck: true,
   wait: true,            // Long polling
   timeoutMillis: 30000,
@@ -790,6 +891,11 @@ await queen.close()  // Flush buffers and close connections
 }
 ```
 
+`batch` and `partitions` are the **autopilot-off** defaults: with autopilot on
+(the default) a knob you never set is not defaulted at all, it is delegated to
+the broker. These values are what comes back with `.autopilot(false)` or
+`QUEEN_SDK_POP_AUTOPILOT=off`.
+
 ---
 
 ## Logging

package/client-v2/Queen.js

@@ -10,9 +10,11 @@ import { QueueBuilder } from './builders/QueueBuilder.js'
 import { TransactionBuilder } from './builders/TransactionBuilder.js'
 import { TimerBuilder } from './builders/TimerBuilder.js'
 import { Kv } from './kv/Kv.js'
+import { Ephemeral } from './ephemeral/Ephemeral.js'
 import { StreamBuilder } from './stream/StreamBuilder.js'
 import { StreamConsumer } from './stream/StreamConsumer.js'
 import { Admin } from './admin/Admin.js'
+import { popAutopilotDisabledByEnv } from './utils/autopilot.js'
 import { CLIENT_DEFAULTS } from './utils/defaults.js'
 import { validateUrl, validateUrls } from './utils/validation.js'
 import * as logger from './utils/logger.js'
@@ -66,6 +68,12 @@ export class Queen {
   #shutdownHandlers = []
   #admin = null
   #kv = null
+  #ephemeral = null
+  // Process-wide kill switch for pop autopilot, read from
+  // QUEEN_SDK_POP_AUTOPILOT once here rather than on every pop: it is a
+  // deployment-level rollback, and re-reading it per request would let a
+  // running process change wire shape halfway through.
+  #autopilotOff = false
 
   constructor(config = {}) {
     // Configure custom logger before anything else.
@@ -81,6 +89,9 @@ export class Queen {
     // Normalize config
     this.#config = this.#normalizeConfig(config)
 
+    // Pop autopilot: on unless the environment rolls it back (utils/autopilot.js).
+    this.#autopilotOff = popAutopilotDisabledByEnv()
+
     // Create HTTP client
     this.#httpClient = this.#createHttpClient()
 
@@ -95,6 +106,15 @@ export class Queen {
     logger.log('Queen.constructor', { status: 'initialized', urls: this.#config.urls.length, handleSignals: this.#config.handleSignals })
   }
 
+  /**
+   * Whether pop autopilot is off for this client because the environment asked
+   * (QUEEN_SDK_POP_AUTOPILOT). Read by the builders; a per-call
+   * `.autopilot(...)` still outranks it.
+   */
+  get autopilotOff() {
+    return this.#autopilotOff
+  }
+
   #normalizeConfig(config) {
     // Handle different input formats
     if (typeof config === 'string') {
@@ -253,6 +273,54 @@ export class Queen {
     return this.#kv
   }
 
+  // ===========================
+  // Ephemeral API Entry Point
+  // ===========================
+
+  /**
+   * RAM-class queues: `/api/v1/ephemeral/*` (EPHEMERAL_QUEUES.md §1, §4).
+   *
+   *     await queen.ephemeral.push('inbox:7', [{ hello: 'world' }])
+   *     const { messages } = await queen.ephemeral.pop('inbox:7', { wait: true })
+   *     await queen.ephemeral.ack('inbox:7', messages, { group: 'workers' })
+   *
+   * A different STORAGE CLASS, not a different API style. What changes:
+   *
+   *   * CONTENTS SURVIVE NOTHING (§1.2) -- restart, crash, deploy, or the
+   *     ownership move a membership change causes. Treat a failover like a
+   *     Redis restart. A declared queue's OPTIONS are durable; it comes back
+   *     configured and EMPTY.
+   *   * a queue does not have to exist: the first push or pop that names one
+   *     creates it, which is what makes thousands of short-lived req/reply
+   *     inboxes cheap (§1.1).
+   *   * delivery is at-least-once while the owning broker lives, at-most-once
+   *     with `autoAck` (§1.3) -- NOT "at most once" as a class. Consumers still
+   *     need idempotency.
+   *   * consumption semantics are the pop's `group`, exactly as on durable
+   *     queues (§1.5): same group competes, own group fans out, no group is
+   *     queue mode. There is no queue-level mode to set.
+   *   * there is no replay, no subscriptionMode, no DLQ, no transactions -- the
+   *     verbs are absent because the concepts have no referent (§9).
+   *
+   * `push(..., {buffered})` shares the durable buffer machinery, so
+   * `queen.close()` drains it on the same deadline (§4.1).
+   *
+   * Requires broker/proxy >= 1.1; an older one 404s the whole family and every
+   * verb here maps that to `.code === EPHEMERAL_UNSUPPORTED`. Not to be
+   * confused with the OTHER 404: `depth` on a queue that does not exist raises
+   * `.code === EPHEMERAL_QUEUE_NOT_FOUND`, which is a missing queue and not a
+   * missing feature.
+   *
+   * Lazily initialized, singleton, like `admin` and `kv`.
+   * @returns {Ephemeral}
+   */
+  get ephemeral() {
+    if (!this.#ephemeral) {
+      this.#ephemeral = new Ephemeral(this.#httpClient, this.#bufferManager)
+    }
+    return this.#ephemeral
+  }
+
   // ===========================
   // Timers API Entry Point
   // ===========================

package/client-v2/admin/Admin.js

@@ -71,11 +71,25 @@ export class Admin {
   /**
    * Per-partition backlog for a queue — the cheap sibling of getQueue:
    * watermark arithmetic only, no segments, no timestamps. Shape:
-   * {queue, group, pending, partitions: [{partition, pending}]}.
+   * {queue, group, pending, partitionsPending, conflation, effectivePending,
+   *  partitions: [{partition, pending}]}.
    * Omitting group gives queue-level pending under the same worst-cursor
    * precedence the dashboard publishes; a named group is that group's own
    * backlog per partition. Requires broker >= 1.0.4 — an older broker
    * answers 404 no_such_route, so fall back to getQueue there.
+   *
+   * The three fields added in 1.1.0 (PLAN_CONFLATION §2.5/§5.3):
+   *   - `partitionsPending`: how many partitions owe work (pending > 0). Useful
+   *     for every group; it is what queenctl used to compute client-side.
+   *   - `conflation`: the group's stored last-value delivery policy.
+   *   - `effectivePending`: WORK depth — handler invocations still owed. For a
+   *     conflating group that is `partitionsPending` (one call per partition,
+   *     newest message only); otherwise it equals `pending`.
+   *
+   * Read them together: for a conflating group `pending` is LOG depth (log
+   * positions still to retire), so `pending: 4000000, effectivePending: 12` is
+   * healthy — the same two numbers on a non-conflating group are an incident.
+   * Absent on brokers older than 1.1.0.
    * @param {string} name - Queue name
    * @param {string|null} [group] - Consumer group (optional)
    * @returns {Promise<object>}

package/client-v2/buffer/BufferManager.js

@@ -19,6 +19,14 @@
  * path that has a SIGTERM grace period to respect. When the deadline expires
  * the messages are still in the buffer -- the error says how many -- so the
  * failure is loud rather than silent.
+ *
+ * WHERE a batch goes is the buffer's DESTINATION (buffer/sinks.js), not
+ * something this loop knows: durable pushes and ephemeral pushes are two routes
+ * with two body shapes and exactly one set of ordering, backpressure and retry
+ * semantics, so the drain is parametrized instead of copied. Addresses are
+ * namespaced per family (`eph:` prefix), so the two never share a buffer, a
+ * drain, or a retry queue. A buffer created without a destination drains to the
+ * durable push exactly as before.
  */
 
 import { MessageBuffer } from './MessageBuffer.js'
@@ -45,9 +53,12 @@ export class BufferManager {
    * @param {string} queueAddress
    * @param {object} formattedMessage
    * @param {object} bufferOptions
-   * @param {{ signal?: AbortSignal }} [opts]
+   * @param {{ signal?: AbortSignal, destination?: object }} [opts] - `destination`
+   *   is `{ sink, queue, partition }` (buffer/sinks.js) and is read ONLY when
+   *   this address's buffer is created: an address belongs to one queue of one
+   *   storage class, so its route cannot change under an in-flight retry.
    */
-  async addMessage(queueAddress, formattedMessage, bufferOptions, { signal } = {}) {
+  async addMessage(queueAddress, formattedMessage, bufferOptions, { signal, destination = null } = {}) {
     // A push after cleanup() would otherwise create a fresh buffer that nothing
     // will ever flush -- messages accepted into a client that is already shut
     // down, which is the same false success the bound exists to remove.
@@ -60,8 +71,8 @@ export class BufferManager {
       // maxSize is derived from the messageCount this caller asked for, and
       // merging defaults here first would hide the difference between "not set"
       // and "set to the default".
-      const created = new MessageBuffer(queueAddress, bufferOptions, (addr) => { this.#startDrain(addr) })
-      logger.log('BufferManager.createBuffer', { queueAddress, options: created.options })
+      const created = new MessageBuffer(queueAddress, bufferOptions, (addr) => { this.#startDrain(addr) }, destination)
+      logger.log('BufferManager.createBuffer', { queueAddress, options: created.options, sink: created.destination.sink.name })
       this.#buffers.set(queueAddress, created)
     }
 
@@ -111,6 +122,7 @@ export class BufferManager {
 
   async #drain(queueAddress, buffer, ctl) {
     const { messageCount, retryDelayMillis } = buffer.options
+    const { sink, queue, partition } = buffer.destination
 
     try {
       while (buffer.messageCount > 0 && !buffer.isStopped) {
@@ -118,8 +130,8 @@ export class BufferManager {
         if (batch.length === 0) break
 
         try {
-          const result = await this.#httpClient.post('/api/v1/push', { items: batch })
-          logger.debug('BufferManager.drain', { queueAddress, sent: batch.length, serverResponse: result ? `${result.length || 'N/A'} items` : 'null' })
+          const result = await this.#httpClient.post(sink.path, sink.format(queue, partition, batch))
+          logger.debug('BufferManager.drain', { queueAddress, sink: sink.name, sent: batch.length, serverResponse: result ? `${result.length || 'N/A'} items` : 'null' })
 
           this.#flushCount++
           // Capacity freed. Woken here rather than at takeBatch, because a

package/client-v2/buffer/MessageBuffer.js

@@ -34,14 +34,24 @@
  * Buffered messages still live only in this process's memory. A crash, or a
  * `process.exit()` that skips `close()`, loses them -- buffering belongs on
  * telemetry-shaped traffic, not on anything that must not be lost.
+ *
+ * A buffer also carries its DESTINATION (buffer/sinks.js): the route its
+ * batches are posted to and the shape they are posted in. It is fixed at
+ * creation and never changes, because it is a property of the address -- one
+ * address is one queue of one storage class -- and because a drain that could
+ * change route mid-retry would post a re-queued batch somewhere its earlier
+ * attempt did not go. Absent, it is the durable push, which is what every
+ * caller that predates ephemeral queues gets without knowing sinks exist.
  */
 
 import { BUFFER_DEFAULTS } from '../utils/defaults.js'
+import { DURABLE_DESTINATION } from './sinks.js'
 
 export class MessageBuffer {
   #queueAddress
   #messages = []
   #options
+  #destination
   #flushCallback
   #timer = null
   #firstMessageTime = null
@@ -58,10 +68,11 @@ export class MessageBuffer {
   #parked = 0
   #stopWaiters = []
 
-  constructor(queueAddress, options, flushCallback) {
+  constructor(queueAddress, options, flushCallback, destination = null) {
     this.#queueAddress = queueAddress
     this.#options = MessageBuffer.normalizeOptions(options)
     this.#flushCallback = flushCallback
+    this.#destination = destination || DURABLE_DESTINATION
   }
 
   /**
@@ -298,6 +309,11 @@ export class MessageBuffer {
     return this.#options
   }
 
+  /** `{ sink, queue, partition }` -- where this buffer's batches are posted. */
+  get destination() {
+    return this.#destination
+  }
+
   get isFlushing() {
     return this.#flushing
   }

package/client-v2/buffer/sinks.js

@@ -0,0 +1,89 @@
+/**
+ * Drain sinks: WHERE a buffered batch goes, and in WHAT shape.
+ *
+ * The buffer machinery -- blocking backpressure at `maxSize`, one drain loop
+ * per address, a failed batch put back at the FRONT and retried until it lands
+ * or a flush deadline expires -- is about ordering, occupancy and loss. None of
+ * that is durable-specific, and none of it is worth writing twice. So the drain
+ * takes a SINK instead of a hardcoded POST:
+ *
+ *     { path, format(queue, partition, batch) -> body }
+ *
+ * `format` receives the queue and partition because the two storage classes
+ * disagree about where that identity lives on the wire, and that disagreement
+ * is the entire reason this parameter exists:
+ *
+ *   * the DURABLE push wire repeats `{queue, partition}` on EVERY item, so the
+ *     envelope is just `{items}` and the sink ignores both arguments;
+ *   * the EPHEMERAL push wire hoists them to the envelope --
+ *     `{queue, partition?, messages:[{payload}...]}` -- so the batch elements
+ *     carry nothing but their payload.
+ *
+ * DURABLE_SINK IS TODAY'S REQUEST, BYTE FOR BYTE. It is the default for a
+ * buffer created without a destination, which is every caller that existed
+ * before ephemeral queues did, and test-v2/ephemeral-unit/durableSinkPin.test.js
+ * exists for no other reason than to fail if that ever stops being true.
+ *
+ * ADDRESSES ARE NAMESPACED. A buffer address is the key of the one-buffer-one-
+ * drain map, so an ephemeral `orders` and a durable `orders` must not hash to
+ * the same entry -- they are unrelated objects (EPHEMERAL_QUEUES.md §10 Q8) and
+ * a shared buffer would post one family's messages to the other family's route.
+ * The `eph:` prefix is the same namespacing the broker applies to its own queue
+ * keys (§3.2), for the same reason.
+ */
+
+/** The durable push wire: identity per item, envelope carries only the batch. */
+export const DURABLE_SINK = {
+  name: 'durable',
+  path: '/api/v1/push',
+  format(_queue, _partition, batch) {
+    return { items: batch }
+  }
+}
+
+/** The ephemeral push wire (EPHEMERAL_QUEUES.md §3.1): identity on the envelope. */
+export const EPHEMERAL_SINK = {
+  name: 'ephemeral',
+  path: '/api/v1/ephemeral/push',
+  format(queue, partition, batch) {
+    const body = { queue }
+    // Omitted, never defaulted client-side: which partition an ephemeral push
+    // without one lands on is the broker's rule, and inventing a 'Default' here
+    // would take that decision away from it in a way the caller never asked for.
+    if (partition !== null && partition !== undefined) body.partition = partition
+    body.messages = batch
+    return body
+  }
+}
+
+/**
+ * What a buffer drains into: the sink, plus the identity that sink formats for.
+ * The default is the durable push, so a buffer created without one behaves
+ * exactly as buffers did before sinks existed.
+ */
+export const DURABLE_DESTINATION = { sink: DURABLE_SINK, queue: null, partition: null }
+
+/** The ephemeral counterpart, bound to one (queue, partition). */
+export function ephemeralDestination(queue, partition = null) {
+  return { sink: EPHEMERAL_SINK, queue, partition }
+}
+
+/**
+ * The durable buffer address, unchanged: `queue/partition`. Kept here next to
+ * its ephemeral sibling so the two can be compared at a glance.
+ */
+export function durableAddress(queue, partition) {
+  return `${queue}/${partition}`
+}
+
+/**
+ * The ephemeral buffer address: `eph:queue/partition`, or `eph:queue` when the
+ * caller named no partition (which is a different destination from any named
+ * one, because the broker picks, and a buffer must not merge the two).
+ *
+ * Same ambiguity as the durable address -- a queue named `a/b` collides with
+ * (`a`, `b`) -- inherited deliberately rather than fixed on one side only.
+ */
+export function ephemeralAddress(queue, partition = null) {
+  return partition === null || partition === undefined ? `eph:${queue}` : `eph:${queue}/${partition}`
+}