@maestro-js/queue 1.0.0-alpha.2 → 1.0.0-alpha.20

package/README.md

@@ -1,279 +1,437 @@
 # @maestro-js/queue
 
-Database-backed job queue that stores messages in MySQL via a pluggable driver and processes them asynchronously. Follows the Provider pattern from `@maestro-js/service-registry` with a producer/consumer split supporting scheduled execution, concurrency, and distributed trace propagation.
+Database-backed job queue with middleware, configurable retries, and backoff strategies.
 
 ## Quick Setup
 
 ```ts
 import { Queue } from '@maestro-js/queue'
 import { Db } from '@maestro-js/db'
-import { Log } from '@maestro-js/log'
-
-// Register the queue service with the db driver
-Queue.Provider.register(
-  'default',
-  Queue.Provider.create({
-    driver: Queue.drivers.db({
-      db: Db.provider('default'),
-      databaseTable: 'jobs'
-    }),
-    logger: Log
-  })
-)
 
-// Create a named queue with a work function
-const orderQueue = Queue.create({
-  name: 'fulfill-orders',
-  workFn: async (body) => {
-    await fulfillOrder(body.orderId)
-    return { fulfilled: true }
-  },
-  concurrency: 3
+// Create and register a queue service with the db driver
+const service = Queue.Provider.create({
+  driver: Queue.drivers.db({ db: Db.provider('default'), databaseTable: 'queue' }),
+  logger: Log.provider('default')
 })
+Queue.Provider.register('default', service)
 
-// Enqueue a message
-await orderQueue.addMessage({ orderId: 'ord_123' }, { scheduledDate: null })
+// Define a queue with a handler
+const emailQueue = Queue.create({
+  name: 'send-emails',
+  handle: async (body) => sendEmail(body.to, body.subject),
+  tries: 3,
+  backoffSeconds: [30, 60, 120]
+})
 
-// Start processing
-orderQueue.startProcessing()
+// Dispatch a message and start processing
+await emailQueue.dispatch({ to: 'alice@example.com', subject: 'Welcome' })
+await Queue.work({ stopWhenEmpty: true })
 ```
 
-## Database Schema
-
-The db driver expects a MySQL table (name configurable via `databaseTable`). The column-to-field mapping used internally:
-
-| Column          | Type          | Description                                              |
-| --------------- | ------------- | -------------------------------------------------------- |
-| `id`            | int (PK, AI)  | Message ID                                               |
-| `batch_id`      | int / NULL    | Lock batch for concurrent dequeue                        |
-| `job_type`      | varchar       | Queue name string                                        |
-| `body`          | text (JSON)   | Serialized message body                                  |
-| `trace`         | text (JSON)   | Serialized `DehydratedTrace` for distributed tracing     |
-| `status`        | enum          | `waiting`, `processing`, `success`, `error`, `stuck`     |
-| `scheduledDate` | datetime      | Earliest time the message can be processed               |
-| `enqueuedDate`  | datetime      | When the message was inserted                            |
-| `started_ts`    | datetime      | When processing began                                    |
-| `result`        | text          | JSON result on success, error stack on failure           |
-| `recovered`     | tinyint       | 1 if the message was recovered from stuck state          |
-| `running_time`  | int / NULL    | Seconds elapsed during processing                        |
-| `unique_key`    | int / NULL    | MD5-derived dedup key (nullable)                         |
-| `priority`      | bigint / NULL | Lower value = higher priority (defaults to `Date.now()`) |
-
-The driver requires a `FORCE INDEX (locking_update)` index on `(batch_id, status, job_type, scheduledDate)` for efficient locking queries.
+## Provider Pattern
 
-## API Reference
+Queue follows the standard Maestro Provider pattern built on `@maestro-js/service-registry`:
 
-### Provider Setup
+1. **`Queue.Provider.create(config)`** -- factory that instantiates a queue service with a driver
+2. **`Queue.Provider.register(name, service)`** -- register the instance in a named registry
+3. **`Queue.provider(key)`** -- resolve a named instance (deferred via Proxy)
+4. **Facade** -- `Queue.create()`, `Queue.work()`, etc. spread from `provider('default')`
+5. **`Queue.drivers`** -- factory functions for pluggable backends (`db`, `local`)
 
-```ts
-Queue.Provider.create(config: {
-  driver: Queue.Driver
-  logger: Log.Logger<[Queue.LogMessage]>
-  tracing?: Queue.Tracing  // optional distributed tracing adapter
-}): Queue.QueueService
-```
+Extend provider keys via declaration merging:
 
 ```ts
-Queue.Provider.register(name: Queue.Provider.Key, service: Queue.QueueService): void
+declare module '@maestro-js/queue' {
+  namespace Queue.Provider {
+    interface Keys {
+      default: unknown
+      highPriority: unknown
+    }
+  }
+}
 ```
 
-### Drivers
+Source: `packages/queue/src/index.ts`
+
+## Drivers
+
+### db (MySQL)
+
+Production driver backed by MySQL. Uses optimistic locking with `UPDATE ... INNER JOIN` for
+concurrent pop operations.
 
 ```ts
-Queue.drivers.db(config: {
-  db: { update, select, insert }  // Db provider interface
-  databaseTable: string
-  extraFields?: string[]           // additional columns to insert
-  fastestPollingRateMs?: number    // default 100
-  slowestPollingRateMs?: number    // default 10_000
-  pollingBackoffRate?: number      // default 1.1
-}): Queue.Driver
+Queue.drivers.db({
+  db: Db.provider('default'),    // must implement select/insert/update
+  databaseTable: 'queue',        // table name (alphanumeric + underscores only)
+  queue: 'send-emails',          // optional: filter to specific queue name(s)
+  batchingMaxWaitSeconds: 0.5    // optional: batch pop requests over this window
+})
+```
+
+Config interface:
+
+| Field                    | Type                 | Default | Description                                    |
+|--------------------------|----------------------|---------|------------------------------------------------|
+| `db`                     | `{ select, insert, update }` | --  | Database connection with query methods          |
+| `databaseTable`          | `string`             | --      | MySQL table name                               |
+| `queue`                  | `string \| string[]` | --      | Optional queue name filter                     |
+| `batchingMaxWaitSeconds` | `number`             | `0`     | Batch pop requests over this window (seconds)  |
+
+Required database schema:
+
+```sql
+CREATE TABLE queue (
+  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
+  attempt INT UNSIGNED NOT NULL DEFAULT 0,
+  queue VARCHAR(255) NOT NULL,
+  status ENUM('waiting', 'processing', 'succeeded', 'failed', 'failedPermanently') NOT NULL DEFAULT 'waiting',
+  scheduledDate DATETIME NOT NULL,
+  enqueuedDate DATETIME NOT NULL DEFAULT NOW(),
+  startedDate DATETIME,
+  completedDate DATETIME,
+  body JSON,
+  result TEXT,
+  lockId BIGINT UNSIGNED DEFAULT NULL,
+  PRIMARY KEY (id, attempt),
+  INDEX locking_pop (lockId, status, scheduledDate),
+  INDEX queue (queue)
+);
 ```
 
-### Queue Instance Methods
+Source: `packages/queue/src/db-queue-driver.ts`
+
+### local (In-Memory)
 
-Create a queue:
+In-memory driver for testing. Same interface as db but stores messages in a plain array:
 
 ```ts
-const queue = Queue.create<MessageBody, Result>({
-  name: string,
-  workFn: (body: MessageBody) => Promise<Result>,
-  logger?: Log.LogFunctions<[Queue.LogMessage]>,  // additional per-queue logger
-  concurrency?: number  // default 1
+Queue.drivers.local({
+  queue: 'send-emails'   // optional: filter to specific queue name(s)
 })
 ```
 
-Dispatch a message:
+Source: `packages/queue/src/local-queue-driver.ts`
+
+## Core Concepts
+
+### Retry and Backoff
+
+Configure retries per queue with `tries` and `backoffSeconds`:
 
 ```ts
-await queue.addMessage(body, {
-  scheduledDate: Iso.Instant | null,   // null = process immediately
-  tracingContext?: DehydratedTrace | null
+const q = Queue.create({
+  name: 'flaky-api',
+  handle: async (body) => callExternalApi(body),
+  tries: 4,                          // total attempts (1 initial + 3 retries)
+  backoffSeconds: [10, 30, 120]      // delay before retry 1, 2, 3
 })
 ```
 
-Update a waiting message:
+- `tries` defaults to `1` (no retries)
+- `backoffSeconds` can be a single `number` (constant delay) or an `number[]` (per-attempt delay)
+- If the array is shorter than the number of retries, the last value is reused
+- If omitted entirely, defaults to `60` seconds between retries
+- `attempts` on the message is 0-indexed -- the first attempt is `0`, the first retry is `1`
+
+### Middleware
+
+Wrap message handling with middleware functions for cross-cutting concerns:
 
 ```ts
-await queue.updateMessage({
-  messageId: number,
-  scheduledDate: Iso.Instant,
-  body: MessageBody
+const loggingMiddleware: Queue.Middleware = async (message, next) => {
+  console.log('Processing:', message)
+  const result = await next()
+  console.log('Done:', result)
+  return result
+}
+
+const q = Queue.create({
+  name: 'with-logging',
+  handle: async (body) => processBody(body),
+  middleware: [loggingMiddleware]
 })
 ```
 
-List messages (paginated):
+Middleware is composed right-to-left (`reduceRight`), so the first middleware in the array is the
+outermost wrapper.
+
+### Fail and Release
+
+Handlers receive `fail` and `release` callbacks as a second argument for explicit message control:
 
 ```ts
-const messages = await queue.listMessages({ pageSize: 50, pageNumber: 0 })
+const q = Queue.create({
+  name: 'validate-first',
+  tries: 3,
+  handle: async (body, { fail, release }) => {
+    if (!body.email) {
+      fail(new Error('Missing email -- will never succeed'))  // permanently fail, skip retries
+    }
+    const result = await sendEmail(body.email)
+    if (result.rateLimited) {
+      release(new Error('Rate limited'), 300)  // retry in 5 minutes
+    }
+  }
+})
 ```
 
-Processing control:
+- **`fail(error)`** -- permanently fails the message, regardless of remaining retries
+- **`release(error, delaySeconds?)`** -- releases the message for retry. Pass `delaySeconds` to override the configured backoff, or omit it to use the default.
+
+Both return `never` -- they halt handler execution at the call site.
+
+Any other thrown error also triggers a retry (or permanent failure on the last attempt), using the
+configured `backoffSeconds`. Middleware also receives these callbacks as a third argument.
+
+### Work Loop
+
+`Queue.work()` starts a polling loop that pops messages from the driver and dispatches them to the
+appropriate handler:
 
 ```ts
-queue.startProcessing()     // begin polling and processing
-queue.stopProcessing()      // stop polling
-await queue.processUntilEmpty({ pollingMs?: number })  // process then resolve when empty
+const workHandle = Queue.work({
+  maxJobs: 100,           // stop after processing 100 messages
+  stopWhenEmpty: false,    // keep polling even when queue is empty
+  maxTimeSeconds: 3600,    // stop after 1 hour
+  sleepSeconds: 1,         // pause between polls when idle (default 1)
+  concurrency: 4           // number of parallel polling workers
+})
+
+// Later: gracefully stop
+workHandle.abort('shutting down')
+
+// Or stop all active work loops
+await Queue.stopAllWork()
 ```
 
-### Service-Level Methods
+The work loop swallows handler errors -- failed messages are logged and released/failed via the
+driver, but the loop continues.
+
+## API Reference
+
+### QueueService Methods
 
 ```ts
-Queue.list() // names of all registered queues
-Queue.get<Body, Result>(name) // retrieve a queue by name
-Queue.startProcessing() // start all queues
-Queue.stopProcessing() // stop all queues
+Queue.Provider.create(config: { driver: Queue.Driver, logger?: Log.Logger }): Queue.QueueService
 ```
+Creates a new queue service with the given driver and optional logger.
 
-### Message Statuses
+```ts
+Queue.create<MessageBody, Result>(config: Queue.Config): Queue.QueueHandle
+```
+Creates a named queue bound to the default service. Throws if a queue with the same name already
+exists.
 
-| Status       | Meaning                                                        |
-| ------------ | -------------------------------------------------------------- |
-| `waiting`    | Queued and eligible for processing when scheduled time arrives |
-| `processing` | Locked by a worker and currently executing                     |
-| `success`    | Work function completed without error                          |
-| `error`      | Work function threw or timed out                               |
-| `stuck`      | Processing exceeded timeout; recovered or marked stuck         |
+```ts
+Queue.work(options: Queue.WorkOptions): Promise<void> & { abort(reason?: string): void }
+```
+Starts a polling work loop. Returns a promise that resolves when the loop stops, with an `abort()`
+method to stop it.
 
-### Log Events
+```ts
+Queue.stopAllWork(): Promise<void>
+```
+Aborts all active work loops and waits for in-flight messages to finish.
 
-The queue emits structured log messages with these `event` values:
+```ts
+Queue.processMessage(queueMessage: Queue.Message): Promise<unknown>
+```
+Processes a single message directly by looking up the queue handler by name. Used by
+`@maestro-js/recurring-jobs` to process messages outside the normal work loop.
 
-- `messageEnqueued` -- message added to the queue
-- `messageDequeued` -- message picked up by a worker
-- `messageProcessed` -- work function completed successfully
-- `messageFailed` -- work function threw an error
-- `queueStarted` -- queue processing started
-- `queueStopped` -- queue processing stopped
+### QueueHandle Methods
 
-## Common Patterns
+```ts
+handle.dispatch(body: MessageBody): Promise<void>
+```
+Enqueues a message for immediate processing.
 
-### Schedule a Future Job
+```ts
+handle.dispatchWithDelay(body: MessageBody, scheduledDate: Iso.Instant): Promise<void>
+```
+Enqueues a message scheduled for a future date.
 
 ```ts
-import { instantFns } from 'iso-fns2'
+handle.dispatchSync(body: MessageBody): Promise<Result>
+```
+Runs the handler synchronously without going through the driver. Useful for testing. Middleware
+still executes.
 
-// Schedule 1 hour from now
-const oneHourLater = instantFns.add(instantFns.now(), { hours: 1 })
-await queue.addMessage({ task: 'reminder' }, { scheduledDate: oneHourLater })
+```ts
+handle.bulk(messages: { body: MessageBody; scheduledDate?: Iso.Instant }[]): Promise<void>
 ```
+Enqueues multiple messages in a single operation. Uses the driver's bulk method for efficiency.
 
-### Deduplication with Unique Keys
+### Queue.Config
 
-The db driver supports unique keys via the `addMessage` options on the driver level. When a duplicate `uniqueKey` is inserted, the row is not duplicated -- instead the priority is updated if the new priority is higher (lower number).
+| Field            | Type                                   | Default | Description                                          |
+|------------------|----------------------------------------|---------|------------------------------------------------------|
+| `name`           | `string`                               | --      | Unique queue name                                    |
+| `handle`         | `(message: MessageBody, actions: Queue.Actions) => Promise<Result>` | --  | Handler function for each message |
+| `middleware`     | `Queue.Middleware[]`                   | `[]`    | Middleware wrapping the handler                       |
+| `tries`          | `number`                               | `1`     | Total attempts (1 = no retries)                      |
+| `backoffSeconds` | `number \| number[]`                   | `60`    | Seconds to wait between retries                      |
+| `logger`         | `Log.LogFunctions`                     | --      | Per-queue logger (in addition to service-level logger)|
 
-### Process Until Empty (Worker Scripts)
+### Queue.WorkOptions
 
-```ts
-// Useful for one-off batch processing or tests
-await queue.processUntilEmpty({ pollingMs: 1000 })
-// Resolves when no waiting/processing messages remain
-```
+| Field            | Type      | Default | Description                                  |
+|------------------|-----------|---------|----------------------------------------------|
+| `maxJobs`        | `number`  | --      | Stop after processing this many messages     |
+| `stopWhenEmpty`  | `boolean` | --      | Stop when no messages are available          |
+| `maxTimeSeconds` | `number`  | --      | Stop after this many seconds                 |
+| `sleepSeconds`   | `number`  | `1`     | Seconds to sleep between polls when idle     |
+| `concurrency`    | `number`  | `1`     | Number of parallel polling workers           |
+
+## Common Patterns
 
-### Concurrency Control
+### Email queue with retries
 
 ```ts
-// High-concurrency queue for lightweight jobs
 const emailQueue = Queue.create({
   name: 'send-emails',
-  workFn: async (body) => sendEmail(body),
-  concurrency: 10
+  handle: async (body, { fail }) => {
+    const result = await sendEmail(body.to, body.subject, body.html)
+    if (result.rejected) {
+      fail(new Error(`Rejected: ${body.to}`))
+    }
+  },
+  tries: 3,
+  backoffSeconds: [60, 300, 900]
 })
+
+await emailQueue.dispatch({ to: 'bob@example.com', subject: 'Hello', html: '<p>Hi!</p>' })
 ```
 
-### Stuck Job Recovery
+### Delayed scheduling
 
-The db driver automatically recovers stuck jobs every 60 seconds. Jobs in `processing` status that exceed the timeout (default 45 minutes) are reset to `waiting` and reprocessed. Set `shouldRecoverStuckJobs: false` in the driver `start` options to mark them as `stuck` instead of recovering.
+```ts
+import { instantFns } from 'iso-fns2'
 
-### Distributed Tracing
+// Schedule a reminder for 24 hours from now
+await reminderQueue.dispatchWithDelay(
+  { userId: 42, message: 'Check in' },
+  instantFns.add(instantFns.now(), { hours: 24 })
+)
+```
 
-Pass a `tracing` adapter when creating the service to propagate trace context across producer/consumer boundaries:
+### Bulk dispatch
 
 ```ts
-Queue.Provider.register(
-  'default',
-  Queue.Provider.create({
-    driver: Queue.drivers.db({ db: Db.provider('default'), databaseTable: 'jobs' }),
-    logger: Log,
-    tracing: {
-      hydrate: (trace, callback) => {
-        /* restore trace context */ return callback()
-      },
-      dehydrate: () => {
-        /* return current trace context */ return null
-      },
-      startActiveSpan: (name, fn) => fn(),
-      getActiveSpan: () => undefined
-    }
-  })
-)
+const messages = users.map((user) => ({
+  body: { userId: user.id, type: 'weekly-digest' }
+}))
+await digestQueue.bulk(messages)
+```
+
+### Middleware (logging, tracing)
+
+```ts
+const tracingMiddleware: Queue.Middleware = async (message, next) => {
+  const span = tracer.startSpan('queue.process')
+  try {
+    return await next()
+  } finally {
+    span.end()
+  }
+}
+
+const q = Queue.create({
+  name: 'traced-queue',
+  handle: async (body) => process(body),
+  middleware: [tracingMiddleware]
+})
 ```
 
-The trace data is serialized as `DehydratedTrace` (traceId, spanId, traceFlags, traceState) and stored in the `trace` column.
+### Sync dispatch for testing
+
+```ts
+// Bypass the driver entirely -- runs the handler (and middleware) inline
+const result = await emailQueue.dispatchSync({ to: 'test@example.com', subject: 'Test' })
+```
 
 ## Testing
 
-This package has no tests yet (`pnpm --filter @maestro-js/queue test` prints "No tests yet"). To test queue behavior in consuming packages, mock the driver interface:
+Use the local driver in tests. The local driver stores messages in-memory:
 
 ```ts
-const mockDriver: Queue.Driver = {
-  start: () => {},
-  stop: () => {},
-  listMessages: async () => [],
-  addMessage: async () => ({ id: 1 }),
-  updateMessage: async () => {},
-  isQueueEmpty: async () => true
-}
+import { Queue } from '@maestro-js/queue'
+import { test } from 'beartest-js'
+import { expect } from 'expect'
 
-Queue.Provider.register(
-  'default',
-  Queue.Provider.create({
-    driver: mockDriver,
-    logger: Log
-  })
-)
+const service = Queue.Provider.create({ driver: Queue.drivers.local() })
+Queue.Provider.register('default', service)
+
+const testQueue = Queue.create({
+  name: 'test-queue',
+  handle: async (body) => ({ processed: body.value })
+})
+
+test('dispatches and processes a message', async () => {
+  await testQueue.dispatch({ value: 42 })
+  await Queue.work({ stopWhenEmpty: true })
+})
+
+test('dispatchSync runs handler inline', async () => {
+  const result = await testQueue.dispatchSync({ value: 99 })
+  expect(result).toEqual({ processed: 99 })
+})
 ```
 
-## Cross-Package Integration
+Run tests:
 
-### Mail (async email sending)
+```bash
+pnpm --filter @maestro-js/queue test
+cd packages/queue && npx beartest ./tests/**/*
+```
 
-The `@maestro-js/mail` package uses Queue to dispatch email-sending jobs asynchronously. Mail enqueues messages; a queue consumer processes them via the configured mail driver.
+## Warnings and Gotchas
+
+- **Queue name uniqueness** -- calling `Queue.create()` with a name that's already registered
+  throws an error. Each queue name must be unique within a service instance.
+- **`attempts` is 0-indexed** -- the first execution has `attempts: 0`, the first retry has
+  `attempts: 1`. The message permanently fails when `attempts + 1 >= tries`.
+- **`backoffSeconds` array indexing** -- `backoffSeconds[attempts]` is used for the delay, where
+  `attempts` is 0-indexed. If the array is shorter than the retry count, the last element is reused.
+- **`dispatchSync` bypasses the driver** -- messages sent with `dispatchSync` are never persisted.
+  The handler runs inline, and `work()` will never pick them up. Middleware still executes.
+- **`work()` swallows handler errors** -- the work loop catches all errors from handlers. Failed
+  messages are released/failed via the driver and logged, but the loop continues polling.
+- **Database schema must be created manually** -- the db driver does not create or migrate the
+  queue table. You must create it before using the driver. See the schema in the Drivers section.
+- **`work()` returns a hybrid promise** -- the returned value is both a `Promise<void>` and has an
+  `abort()` method. Call `abort()` to stop the loop, then `await` the promise.
 
-### Recurring Jobs (cron scheduling)
+## Cross-Package Integration
 
-The `@maestro-js/recurring-jobs` package uses Queue to execute scheduled/cron-based jobs. It dispatches queue messages on a schedule and relies on the queue's processing infrastructure.
+- **Depends on**: `@maestro-js/service-registry` (Provider pattern foundation),
+  `@maestro-js/log` (structured logging)
+- **Used by**: `@maestro-js/mail` (async email sending via queue), `@maestro-js/recurring-jobs`
+  (cron-scheduled jobs backed by queue)
+- **Dev dependency**: `@maestro-js/db` (MySQL driver needs a db connection)
 
-### Db (database driver)
+## Driver Interface
 
-The db driver requires a `Db` provider that exposes `select`, `update`, and `insert` methods. Pass `Db.provider('default')` as the `db` config option.
+Implement `Queue.Driver` to create a custom backend:
 
-### Tracing
+```ts
+interface QueueDriver {
+  size(): Promise<number>
+  push(message: { queue: string; body: Record<string, any>; scheduledDate: Iso.Instant }): Promise<string>
+  bulk(messages: { queue: string; body: Record<string, any>; scheduledDate: Iso.Instant }[]): Promise<void>
+  pop(): Promise<QueueMessage | null>
+  fail(options: { id: string; attempts: number; error: Error }): Promise<void>
+  success(options: { id: string; attempts: number; result: any }): Promise<void>
+  release(options: { id: string; attempts: number; error: Error; delaySeconds: number }): Promise<void>
+}
+```
 
-Integrate with `@maestro-js/tracing` by implementing the `Queue.Tracing` interface to propagate OpenTelemetry-compatible trace context across the producer/consumer boundary.
+Source: `packages/queue/src/queue-driver.ts`
 
-## Source Files
+## Key Source Files
 
-- `packages/queue/src/index.ts` -- Provider pattern, facade, queue creation and lifecycle
-- `packages/queue/src/queue-types.ts` -- TypeScript interfaces for QueueMessage, QueueDriver, QueueTracing, log events
-- `packages/queue/src/db-queue-driver.ts` -- MySQL-backed driver with polling, batched locking, stuck job recovery
+- `packages/queue/src/index.ts` -- main export, Provider pattern, `createService()`, queue handles
+- `packages/queue/src/queue-driver.ts` -- `QueueDriver` and `QueueMessage` interfaces
+- `packages/queue/src/db-queue-driver.ts` -- MySQL driver with optimistic locking
+- `packages/queue/src/local-queue-driver.ts` -- in-memory driver for testing