@maestro-js/queue 1.0.0-alpha.19 → 1.0.0-alpha.2

package/README.md

@@ -1,437 +1,279 @@
 # @maestro-js/queue
 
-Database-backed job queue with middleware, configurable retries, and backoff strategies.
+Database-backed job queue that stores messages in MySQL via a pluggable driver and processes them asynchronously. Follows the Provider pattern from `@maestro-js/service-registry` with a producer/consumer split supporting scheduled execution, concurrency, and distributed trace propagation.
 
 ## Quick Setup
 
 ```ts
 import { Queue } from '@maestro-js/queue'
 import { Db } from '@maestro-js/db'
+import { Log } from '@maestro-js/log'
+
+// Register the queue service with the db driver
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: Queue.drivers.db({
+      db: Db.provider('default'),
+      databaseTable: 'jobs'
+    }),
+    logger: Log
+  })
+)
 
-// Create and register a queue service with the db driver
-const service = Queue.Provider.create({
-  driver: Queue.drivers.db({ db: Db.provider('default'), databaseTable: 'queue' }),
-  logger: Log.provider('default')
+// Create a named queue with a work function
+const orderQueue = Queue.create({
+  name: 'fulfill-orders',
+  workFn: async (body) => {
+    await fulfillOrder(body.orderId)
+    return { fulfilled: true }
+  },
+  concurrency: 3
 })
-Queue.Provider.register('default', service)
 
-// Define a queue with a handler
-const emailQueue = Queue.create({
-  name: 'send-emails',
-  handle: async (body) => sendEmail(body.to, body.subject),
-  tries: 3,
-  backoffSeconds: [30, 60, 120]
-})
+// Enqueue a message
+await orderQueue.addMessage({ orderId: 'ord_123' }, { scheduledDate: null })
 
-// Dispatch a message and start processing
-await emailQueue.dispatch({ to: 'alice@example.com', subject: 'Welcome' })
-await Queue.work({ stopWhenEmpty: true })
+// Start processing
+orderQueue.startProcessing()
 ```
 
-## Provider Pattern
-
-Queue follows the standard Maestro Provider pattern built on `@maestro-js/service-registry`:
+## Database Schema
+
+The db driver expects a MySQL table (name configurable via `databaseTable`). The column-to-field mapping used internally:
+
+| Column          | Type          | Description                                              |
+| --------------- | ------------- | -------------------------------------------------------- |
+| `id`            | int (PK, AI)  | Message ID                                               |
+| `batch_id`      | int / NULL    | Lock batch for concurrent dequeue                        |
+| `job_type`      | varchar       | Queue name string                                        |
+| `body`          | text (JSON)   | Serialized message body                                  |
+| `trace`         | text (JSON)   | Serialized `DehydratedTrace` for distributed tracing     |
+| `status`        | enum          | `waiting`, `processing`, `success`, `error`, `stuck`     |
+| `scheduledDate` | datetime      | Earliest time the message can be processed               |
+| `enqueuedDate`  | datetime      | When the message was inserted                            |
+| `started_ts`    | datetime      | When processing began                                    |
+| `result`        | text          | JSON result on success, error stack on failure           |
+| `recovered`     | tinyint       | 1 if the message was recovered from stuck state          |
+| `running_time`  | int / NULL    | Seconds elapsed during processing                        |
+| `unique_key`    | int / NULL    | MD5-derived dedup key (nullable)                         |
+| `priority`      | bigint / NULL | Lower value = higher priority (defaults to `Date.now()`) |
+
+The driver requires a `FORCE INDEX (locking_update)` index on `(batch_id, status, job_type, scheduledDate)` for efficient locking queries.
 
-1. **`Queue.Provider.create(config)`** -- factory that instantiates a queue service with a driver
-2. **`Queue.Provider.register(name, service)`** -- register the instance in a named registry
-3. **`Queue.provider(key)`** -- resolve a named instance (deferred via Proxy)
-4. **Facade** -- `Queue.create()`, `Queue.work()`, etc. spread from `provider('default')`
-5. **`Queue.drivers`** -- factory functions for pluggable backends (`db`, `local`)
+## API Reference
 
-Extend provider keys via declaration merging:
+### Provider Setup
 
 ```ts
-declare module '@maestro-js/queue' {
-  namespace Queue.Provider {
-    interface Keys {
-      default: unknown
-      highPriority: unknown
-    }
-  }
-}
+Queue.Provider.create(config: {
+  driver: Queue.Driver
+  logger: Log.Logger<[Queue.LogMessage]>
+  tracing?: Queue.Tracing  // optional distributed tracing adapter
+}): Queue.QueueService
 ```
 
-Source: `packages/queue/src/index.ts`
-
-## Drivers
-
-### db (MySQL)
-
-Production driver backed by MySQL. Uses optimistic locking with `UPDATE ... INNER JOIN` for
-concurrent pop operations.
-
 ```ts
-Queue.drivers.db({
-  db: Db.provider('default'),    // must implement select/insert/update
-  databaseTable: 'queue',        // table name (alphanumeric + underscores only)
-  queue: 'send-emails',          // optional: filter to specific queue name(s)
-  batchingMaxWaitSeconds: 0.5    // optional: batch pop requests over this window
-})
-```
-
-Config interface:
-
-| Field                    | Type                 | Default | Description                                    |
-|--------------------------|----------------------|---------|------------------------------------------------|
-| `db`                     | `{ select, insert, update }` | --  | Database connection with query methods          |
-| `databaseTable`          | `string`             | --      | MySQL table name                               |
-| `queue`                  | `string \| string[]` | --      | Optional queue name filter                     |
-| `batchingMaxWaitSeconds` | `number`             | `0`     | Batch pop requests over this window (seconds)  |
-
-Required database schema:
-
-```sql
-CREATE TABLE queue (
-  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
-  attempt INT UNSIGNED NOT NULL DEFAULT 0,
-  queue VARCHAR(255) NOT NULL,
-  status ENUM('waiting', 'processing', 'succeeded', 'failed', 'failedPermanently') NOT NULL DEFAULT 'waiting',
-  scheduledDate DATETIME NOT NULL,
-  enqueuedDate DATETIME NOT NULL DEFAULT NOW(),
-  startedDate DATETIME,
-  completedDate DATETIME,
-  body JSON,
-  result TEXT,
-  lockId BIGINT UNSIGNED DEFAULT NULL,
-  PRIMARY KEY (id, attempt),
-  INDEX locking_pop (lockId, status, scheduledDate),
-  INDEX queue (queue)
-);
+Queue.Provider.register(name: Queue.Provider.Key, service: Queue.QueueService): void
 ```
 
-Source: `packages/queue/src/db-queue-driver.ts`
-
-### local (In-Memory)
-
-In-memory driver for testing. Same interface as db but stores messages in a plain array:
+### Drivers
 
 ```ts
-Queue.drivers.local({
-  queue: 'send-emails'   // optional: filter to specific queue name(s)
-})
+Queue.drivers.db(config: {
+  db: { update, select, insert }  // Db provider interface
+  databaseTable: string
+  extraFields?: string[]           // additional columns to insert
+  fastestPollingRateMs?: number    // default 100
+  slowestPollingRateMs?: number    // default 10_000
+  pollingBackoffRate?: number      // default 1.1
+}): Queue.Driver
 ```
 
-Source: `packages/queue/src/local-queue-driver.ts`
-
-## Core Concepts
-
-### Retry and Backoff
+### Queue Instance Methods
 
-Configure retries per queue with `tries` and `backoffSeconds`:
+Create a queue:
 
 ```ts
-const q = Queue.create({
-  name: 'flaky-api',
-  handle: async (body) => callExternalApi(body),
-  tries: 4,                          // total attempts (1 initial + 3 retries)
-  backoffSeconds: [10, 30, 120]      // delay before retry 1, 2, 3
+const queue = Queue.create<MessageBody, Result>({
+  name: string,
+  workFn: (body: MessageBody) => Promise<Result>,
+  logger?: Log.LogFunctions<[Queue.LogMessage]>,  // additional per-queue logger
+  concurrency?: number  // default 1
 })
 ```
 
-- `tries` defaults to `1` (no retries)
-- `backoffSeconds` can be a single `number` (constant delay) or an `number[]` (per-attempt delay)
-- If the array is shorter than the number of retries, the last value is reused
-- If omitted entirely, defaults to `60` seconds between retries
-- `attempts` on the message is 0-indexed -- the first attempt is `0`, the first retry is `1`
-
-### Middleware
-
-Wrap message handling with middleware functions for cross-cutting concerns:
+Dispatch a message:
 
 ```ts
-const loggingMiddleware: Queue.Middleware = async (message, next) => {
-  console.log('Processing:', message)
-  const result = await next()
-  console.log('Done:', result)
-  return result
-}
-
-const q = Queue.create({
-  name: 'with-logging',
-  handle: async (body) => processBody(body),
-  middleware: [loggingMiddleware]
+await queue.addMessage(body, {
+  scheduledDate: Iso.Instant | null,   // null = process immediately
+  tracingContext?: DehydratedTrace | null
 })
 ```
 
-Middleware is composed right-to-left (`reduceRight`), so the first middleware in the array is the
-outermost wrapper.
-
-### Fail and Release
-
-Handlers receive `fail` and `release` callbacks as a second argument for explicit message control:
+Update a waiting message:
 
 ```ts
-const q = Queue.create({
-  name: 'validate-first',
-  tries: 3,
-  handle: async (body, { fail, release }) => {
-    if (!body.email) {
-      fail(new Error('Missing email -- will never succeed'))  // permanently fail, skip retries
-    }
-    const result = await sendEmail(body.email)
-    if (result.rateLimited) {
-      release(new Error('Rate limited'), 300)  // retry in 5 minutes
-    }
-  }
+await queue.updateMessage({
+  messageId: number,
+  scheduledDate: Iso.Instant,
+  body: MessageBody
 })
 ```
 
-- **`fail(error)`** -- permanently fails the message, regardless of remaining retries
-- **`release(error, delaySeconds?)`** -- releases the message for retry. Pass `delaySeconds` to override the configured backoff, or omit it to use the default.
-
-Both return `never` -- they halt handler execution at the call site.
-
-Any other thrown error also triggers a retry (or permanent failure on the last attempt), using the
-configured `backoffSeconds`. Middleware also receives these callbacks as a third argument.
-
-### Work Loop
-
-`Queue.work()` starts a polling loop that pops messages from the driver and dispatches them to the
-appropriate handler:
+List messages (paginated):
 
 ```ts
-const workHandle = Queue.work({
-  maxJobs: 100,           // stop after processing 100 messages
-  stopWhenEmpty: false,    // keep polling even when queue is empty
-  maxTimeSeconds: 3600,    // stop after 1 hour
-  sleepSeconds: 1,         // pause between polls when idle (default 1)
-  concurrency: 4           // number of parallel polling workers
-})
-
-// Later: gracefully stop
-workHandle.abort('shutting down')
-
-// Or stop all active work loops
-await Queue.stopAllWork()
+const messages = await queue.listMessages({ pageSize: 50, pageNumber: 0 })
 ```
 
-The work loop swallows handler errors -- failed messages are logged and released/failed via the
-driver, but the loop continues.
-
-## API Reference
-
-### QueueService Methods
+Processing control:
 
 ```ts
-Queue.Provider.create(config: { driver: Queue.Driver, logger?: Log.Logger }): Queue.QueueService
+queue.startProcessing()     // begin polling and processing
+queue.stopProcessing()      // stop polling
+await queue.processUntilEmpty({ pollingMs?: number })  // process then resolve when empty
 ```
-Creates a new queue service with the given driver and optional logger.
 
-```ts
-Queue.create<MessageBody, Result>(config: Queue.Config): Queue.QueueHandle
-```
-Creates a named queue bound to the default service. Throws if a queue with the same name already
-exists.
+### Service-Level Methods
 
 ```ts
-Queue.work(options: Queue.WorkOptions): Promise<void> & { abort(reason?: string): void }
+Queue.list() // names of all registered queues
+Queue.get<Body, Result>(name) // retrieve a queue by name
+Queue.startProcessing() // start all queues
+Queue.stopProcessing() // stop all queues
 ```
-Starts a polling work loop. Returns a promise that resolves when the loop stops, with an `abort()`
-method to stop it.
 
-```ts
-Queue.stopAllWork(): Promise<void>
-```
-Aborts all active work loops and waits for in-flight messages to finish.
+### Message Statuses
 
-```ts
-Queue.processMessage(queueMessage: Queue.Message): Promise<unknown>
-```
-Processes a single message directly by looking up the queue handler by name. Used by
-`@maestro-js/recurring-jobs` to process messages outside the normal work loop.
+| Status       | Meaning                                                        |
+| ------------ | -------------------------------------------------------------- |
+| `waiting`    | Queued and eligible for processing when scheduled time arrives |
+| `processing` | Locked by a worker and currently executing                     |
+| `success`    | Work function completed without error                          |
+| `error`      | Work function threw or timed out                               |
+| `stuck`      | Processing exceeded timeout; recovered or marked stuck         |
 
-### QueueHandle Methods
+### Log Events
 
-```ts
-handle.dispatch(body: MessageBody): Promise<void>
-```
-Enqueues a message for immediate processing.
+The queue emits structured log messages with these `event` values:
 
-```ts
-handle.dispatchWithDelay(body: MessageBody, scheduledDate: Iso.Instant): Promise<void>
-```
-Enqueues a message scheduled for a future date.
+- `messageEnqueued` -- message added to the queue
+- `messageDequeued` -- message picked up by a worker
+- `messageProcessed` -- work function completed successfully
+- `messageFailed` -- work function threw an error
+- `queueStarted` -- queue processing started
+- `queueStopped` -- queue processing stopped
 
-```ts
-handle.dispatchSync(body: MessageBody): Promise<Result>
-```
-Runs the handler synchronously without going through the driver. Useful for testing. Middleware
-still executes.
+## Common Patterns
+
+### Schedule a Future Job
 
 ```ts
-handle.bulk(messages: { body: MessageBody; scheduledDate?: Iso.Instant }[]): Promise<void>
-```
-Enqueues multiple messages in a single operation. Uses the driver's bulk method for efficiency.
+import { instantFns } from 'iso-fns2'
 
-### Queue.Config
+// Schedule 1 hour from now
+const oneHourLater = instantFns.add(instantFns.now(), { hours: 1 })
+await queue.addMessage({ task: 'reminder' }, { scheduledDate: oneHourLater })
+```
 
-| Field            | Type                                   | Default | Description                                          |
-|------------------|----------------------------------------|---------|------------------------------------------------------|
-| `name`           | `string`                               | --      | Unique queue name                                    |
-| `handle`         | `(message: MessageBody, actions: Queue.Actions) => Promise<Result>` | --  | Handler function for each message |
-| `middleware`     | `Queue.Middleware[]`                   | `[]`    | Middleware wrapping the handler                       |
-| `tries`          | `number`                               | `1`     | Total attempts (1 = no retries)                      |
-| `backoffSeconds` | `number \| number[]`                   | `60`    | Seconds to wait between retries                      |
-| `logger`         | `Log.LogFunctions`                     | --      | Per-queue logger (in addition to service-level logger)|
+### Deduplication with Unique Keys
 
-### Queue.WorkOptions
+The db driver supports unique keys via the `addMessage` options on the driver level. When a duplicate `uniqueKey` is inserted, the row is not duplicated -- instead the priority is updated if the new priority is higher (lower number).
 
-| Field            | Type      | Default | Description                                  |
-|------------------|-----------|---------|----------------------------------------------|
-| `maxJobs`        | `number`  | --      | Stop after processing this many messages     |
-| `stopWhenEmpty`  | `boolean` | --      | Stop when no messages are available          |
-| `maxTimeSeconds` | `number`  | --      | Stop after this many seconds                 |
-| `sleepSeconds`   | `number`  | `1`     | Seconds to sleep between polls when idle     |
-| `concurrency`    | `number`  | `1`     | Number of parallel polling workers           |
+### Process Until Empty (Worker Scripts)
 
-## Common Patterns
+```ts
+// Useful for one-off batch processing or tests
+await queue.processUntilEmpty({ pollingMs: 1000 })
+// Resolves when no waiting/processing messages remain
+```
 
-### Email queue with retries
+### Concurrency Control
 
 ```ts
+// High-concurrency queue for lightweight jobs
 const emailQueue = Queue.create({
   name: 'send-emails',
-  handle: async (body, { fail }) => {
-    const result = await sendEmail(body.to, body.subject, body.html)
-    if (result.rejected) {
-      fail(new Error(`Rejected: ${body.to}`))
-    }
-  },
-  tries: 3,
-  backoffSeconds: [60, 300, 900]
+  workFn: async (body) => sendEmail(body),
+  concurrency: 10
 })
-
-await emailQueue.dispatch({ to: 'bob@example.com', subject: 'Hello', html: '<p>Hi!</p>' })
 ```
 
-### Delayed scheduling
+### Stuck Job Recovery
 
-```ts
-import { instantFns } from 'iso-fns2'
+The db driver automatically recovers stuck jobs every 60 seconds. Jobs in `processing` status that exceed the timeout (default 45 minutes) are reset to `waiting` and reprocessed. Set `shouldRecoverStuckJobs: false` in the driver `start` options to mark them as `stuck` instead of recovering.
 
-// Schedule a reminder for 24 hours from now
-await reminderQueue.dispatchWithDelay(
-  { userId: 42, message: 'Check in' },
-  instantFns.add(instantFns.now(), { hours: 24 })
-)
-```
-
-### Bulk dispatch
-
-```ts
-const messages = users.map((user) => ({
-  body: { userId: user.id, type: 'weekly-digest' }
-}))
-await digestQueue.bulk(messages)
-```
+### Distributed Tracing
 
-### Middleware (logging, tracing)
+Pass a `tracing` adapter when creating the service to propagate trace context across producer/consumer boundaries:
 
 ```ts
-const tracingMiddleware: Queue.Middleware = async (message, next) => {
-  const span = tracer.startSpan('queue.process')
-  try {
-    return await next()
-  } finally {
-    span.end()
-  }
-}
-
-const q = Queue.create({
-  name: 'traced-queue',
-  handle: async (body) => process(body),
-  middleware: [tracingMiddleware]
-})
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: Queue.drivers.db({ db: Db.provider('default'), databaseTable: 'jobs' }),
+    logger: Log,
+    tracing: {
+      hydrate: (trace, callback) => {
+        /* restore trace context */ return callback()
+      },
+      dehydrate: () => {
+        /* return current trace context */ return null
+      },
+      startActiveSpan: (name, fn) => fn(),
+      getActiveSpan: () => undefined
+    }
+  })
+)
 ```
 
-### Sync dispatch for testing
-
-```ts
-// Bypass the driver entirely -- runs the handler (and middleware) inline
-const result = await emailQueue.dispatchSync({ to: 'test@example.com', subject: 'Test' })
-```
+The trace data is serialized as `DehydratedTrace` (traceId, spanId, traceFlags, traceState) and stored in the `trace` column.
 
 ## Testing
 
-Use the local driver in tests. The local driver stores messages in-memory:
+This package has no tests yet (`pnpm --filter @maestro-js/queue test` prints "No tests yet"). To test queue behavior in consuming packages, mock the driver interface:
 
 ```ts
-import { Queue } from '@maestro-js/queue'
-import { test } from 'beartest-js'
-import { expect } from 'expect'
-
-const service = Queue.Provider.create({ driver: Queue.drivers.local() })
-Queue.Provider.register('default', service)
-
-const testQueue = Queue.create({
-  name: 'test-queue',
-  handle: async (body) => ({ processed: body.value })
-})
-
-test('dispatches and processes a message', async () => {
-  await testQueue.dispatch({ value: 42 })
-  await Queue.work({ stopWhenEmpty: true })
-})
+const mockDriver: Queue.Driver = {
+  start: () => {},
+  stop: () => {},
+  listMessages: async () => [],
+  addMessage: async () => ({ id: 1 }),
+  updateMessage: async () => {},
+  isQueueEmpty: async () => true
+}
 
-test('dispatchSync runs handler inline', async () => {
-  const result = await testQueue.dispatchSync({ value: 99 })
-  expect(result).toEqual({ processed: 99 })
-})
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: mockDriver,
+    logger: Log
+  })
+)
 ```
 
-Run tests:
+## Cross-Package Integration
 
-```bash
-pnpm --filter @maestro-js/queue test
-cd packages/queue && npx beartest ./tests/**/*
-```
+### Mail (async email sending)
 
-## Warnings and Gotchas
-
-- **Queue name uniqueness** -- calling `Queue.create()` with a name that's already registered
-  throws an error. Each queue name must be unique within a service instance.
-- **`attempts` is 0-indexed** -- the first execution has `attempts: 0`, the first retry has
-  `attempts: 1`. The message permanently fails when `attempts + 1 >= tries`.
-- **`backoffSeconds` array indexing** -- `backoffSeconds[attempts]` is used for the delay, where
-  `attempts` is 0-indexed. If the array is shorter than the retry count, the last element is reused.
-- **`dispatchSync` bypasses the driver** -- messages sent with `dispatchSync` are never persisted.
-  The handler runs inline, and `work()` will never pick them up. Middleware still executes.
-- **`work()` swallows handler errors** -- the work loop catches all errors from handlers. Failed
-  messages are released/failed via the driver and logged, but the loop continues polling.
-- **Database schema must be created manually** -- the db driver does not create or migrate the
-  queue table. You must create it before using the driver. See the schema in the Drivers section.
-- **`work()` returns a hybrid promise** -- the returned value is both a `Promise<void>` and has an
-  `abort()` method. Call `abort()` to stop the loop, then `await` the promise.
+The `@maestro-js/mail` package uses Queue to dispatch email-sending jobs asynchronously. Mail enqueues messages; a queue consumer processes them via the configured mail driver.
 
-## Cross-Package Integration
+### Recurring Jobs (cron scheduling)
 
-- **Depends on**: `@maestro-js/service-registry` (Provider pattern foundation),
-  `@maestro-js/log` (structured logging)
-- **Used by**: `@maestro-js/mail` (async email sending via queue), `@maestro-js/recurring-jobs`
-  (cron-scheduled jobs backed by queue)
-- **Dev dependency**: `@maestro-js/db` (MySQL driver needs a db connection)
+The `@maestro-js/recurring-jobs` package uses Queue to execute scheduled/cron-based jobs. It dispatches queue messages on a schedule and relies on the queue's processing infrastructure.
 
-## Driver Interface
+### Db (database driver)
 
-Implement `Queue.Driver` to create a custom backend:
+The db driver requires a `Db` provider that exposes `select`, `update`, and `insert` methods. Pass `Db.provider('default')` as the `db` config option.
 
-```ts
-interface QueueDriver {
-  size(): Promise<number>
-  push(message: { queue: string; body: Record<string, any>; scheduledDate: Iso.Instant }): Promise<string>
-  bulk(messages: { queue: string; body: Record<string, any>; scheduledDate: Iso.Instant }[]): Promise<void>
-  pop(): Promise<QueueMessage | null>
-  fail(options: { id: string; attempts: number; error: Error }): Promise<void>
-  success(options: { id: string; attempts: number; result: any }): Promise<void>
-  release(options: { id: string; attempts: number; error: Error; delaySeconds: number }): Promise<void>
-}
-```
+### Tracing
 
-Source: `packages/queue/src/queue-driver.ts`
+Integrate with `@maestro-js/tracing` by implementing the `Queue.Tracing` interface to propagate OpenTelemetry-compatible trace context across the producer/consumer boundary.
 
-## Key Source Files
+## Source Files
 
-- `packages/queue/src/index.ts` -- main export, Provider pattern, `createService()`, queue handles
-- `packages/queue/src/queue-driver.ts` -- `QueueDriver` and `QueueMessage` interfaces
-- `packages/queue/src/db-queue-driver.ts` -- MySQL driver with optimistic locking
-- `packages/queue/src/local-queue-driver.ts` -- in-memory driver for testing
+- `packages/queue/src/index.ts` -- Provider pattern, facade, queue creation and lifecycle
+- `packages/queue/src/queue-types.ts` -- TypeScript interfaces for QueueMessage, QueueDriver, QueueTracing, log events
+- `packages/queue/src/db-queue-driver.ts` -- MySQL-backed driver with polling, batched locking, stuck job recovery