lonnymq 0.0.18 → 0.0.20

package/README.md

@@ -5,12 +5,11 @@ A high-performance, multi-tenant PostgreSQL message queue implementation for Nod
 ## Features
 
 - High throughput message processing
-- Multi-tenant concurrency and capacity constraints
+- Multi-tenant concurrency and rate limits
 - Durable message processing with automatic recovery
 - Flexible message processing retries and deferrals
-- Message deduplication
 - Queue operations as part of *existing* database transactions
-- Database client agnostic
+- Database client agnostic with optional adapters
 - Granular events via PostgreSQL `NOTIFY`
 - Zero dependencies
 
@@ -34,22 +33,18 @@ for (const migration of queue.migrations()) {
 
 // Create messages
 for (let ix = 0; ix < 500; ix += 1) {
-    await queue
-        .channel("myChannel")
-        .message
-        .create({ 
-            content: Buffer.from("Hello"),
-            databaseClient,
-        })
+    await queue.message.create({ 
+        databaseClient,
+        content: Buffer.from("Hello"),
+        lockMs: 30_000 
+    })
 }
 
 // Process messages
 while (true) {
     const dequeueResult = await queue.dequeue({ databaseClient })
     if (dequeueResult.resultType === "MESSAGE_NOT_AVAILABLE") {
-        const sleepMs = Math.min(1000, dequeueResult.retryMs ?? 5000)
-        await new Promise(resolve => setTimeout(resolve, sleepMs))
-        continue
+        break
     }
 
     console.log(dequeueResult.message.content.toString())
@@ -90,7 +85,7 @@ try {
 
 Channels provide LonnyMQ's multi-tenancy support. They can be considered lightweight sub-queues that are read from in round-robin fashion. There is no performance penalty for using large numbers of channels, so they can be assigned on a highly granular basis (e.g., per-user) to ensure work is scheduled fairly.
 
-Channels can be configured with concurrency, capacity, and rate limits by setting their "channel policy":
+Channels can be configured with concurrency and rate limits by setting their "channel policy":
 
 ```typescript
 await queue
@@ -99,7 +94,6 @@ await queue
     .set({ 
         databaseClient,
         maxConcurrency: 1,
-        maxSize: 100,
         releaseIntervalMs: 1000
     })
 
@@ -112,7 +106,18 @@ await queue
 
 ## Message Creation
 
-You can add a message to the queue and assign it to a particular channel using the `create` function:
+You can add a message to the queue using the `create` function. By default, messages are assigned to a random channel, ensuring fair distribution:
+
+```typescript
+await queue.message.create({
+    databaseClient,
+    content: Buffer.from("Hello, world"),
+    lockMs: 30000,
+    delayMs: 5000
+})
+```
+
+If you need to assign messages to specific channels (for example, to take advantage of concurrency or rate limiting features), you can specify the channel explicitly:
 
 ```typescript
 await queue
@@ -121,25 +126,27 @@ await queue
     .create({
         databaseClient,
         content: Buffer.from("Hello, world"),
-        name: "optional-dedup-key",
-        lockMs: 30000,  // 30 seconds
-        delayMs: 5000   // 5 second delay
+        lockMs: 30000,
+        delayMs: 5000
     })
 ```
 
-The `name` argument can be provided for deduplication purposes: if a message that has *never* been dequeued exists with the same name within the same channel, no new message will be created.
+The `lockMs` parameter is required and specifies how long a message will remain exclusively locked after being dequeued. While locked, the message is **not available** for subsequent `dequeue()` calls, preventing duplicate processing. If your process crashes or takes longer than expected, the message will automatically become available for dequeue again after the lock expires.
+
+The `delayMs` parameter is optional and allows you to delay when the message becomes available for processing.
 
 ## Message Processing
 
 Messages can be fetched for processing by calling `dequeue` on the `Queue` - this locks the message. Once processing is complete, messages must be "finalized" via **deletion** or **deferral** (for further processing in the future).
 
 ```typescript
+const dequeueResult = await queue.dequeue({ databaseClient })
 
 if (dequeueResult.resultType === "MESSAGE_DEQUEUED") {
     const { message } = dequeueResult
     console.log(`Processing message: ${message.id}`)
     console.log(`Content: ${message.content.toString()}`)
-    console.log(`Attempts: ${message.numAttempts}`)
+    console.log(`State: ${message.state?.toString()}`)
     
     try {
         // Process the message...
@@ -148,18 +155,61 @@ if (dequeueResult.resultType === "MESSAGE_DEQUEUED") {
         // Delete on success
         await message.delete({ databaseClient })
     } catch (error) {
-        // Defer for retry with updated state
-        await message.defer({ 
-            databaseClient,
-            delayMs: 30000, // Retry in 30 seconds
-            state: Buffer.from(JSON.stringify({ error: error.message }))
-        })
+        if (message.numAttempts >= 5) {
+            // Too many retries, delete permanently
+            await message.delete({ databaseClient })
+        } else {
+            // Defer for retry with exponential backoff and updated state
+            const backoffMs = Math.pow(2, message.numAttempts) * 1000
+            await message.defer({ 
+                databaseClient,
+                delayMs: backoffMs,
+                state: Buffer.from(JSON.stringify({ 
+                    error: error.message,
+                    lastAttempt: new Date().toISOString()
+                }))
+            })
+        }
     }
+} else {
+    console.log("No messages available")
 }
 ```
 
 When deferring a message, you can optionally specify `delayMs` and `state` arguments. The `delayMs` parameter tells the queue how long to wait before making the message available for reprocessing, and `state` allows you to "save your work" and implement durable and/or repeating/scheduled tasks.
 
+**Note:** The above shows just one processing pattern (defer on failure with retry limits). You have complete flexibility in how you handle message processing - you might delete messages immediately, defer them unconditionally, implement different retry strategies based on error types, or use the message metadata (attempts, state, channel) to make sophisticated routing decisions.
+
+### Message Heartbeats
+
+For long-running message processing, you can use the heartbeat functionality to extend the lock time beyond the initial `lockMs` value. This is useful when you need to process a message for longer than originally anticipated but want to keep a shorter default lock time for faster recovery.
+
+```typescript
+const dequeueResult = await queue.dequeue({ databaseClient })
+
+if (dequeueResult.resultType === "MESSAGE_DEQUEUED") {
+    const { message } = dequeueResult
+    
+    // Start a long-running process
+    const intervalId = setInterval(async () => {
+        // Send heartbeat every 10 seconds to keep the message locked
+        await message.heartbeat({ databaseClient })
+    }, 10000)
+    
+    try {
+        // Process the message (this might take a long time)
+        await longRunningProcessMessage(message.content)
+        await message.delete({ databaseClient })
+    } catch (error) {
+        await message.defer({ databaseClient, delayMs: 30000 })
+    } finally {
+        clearInterval(intervalId)
+    }
+}
+```
+
+Heartbeats reset the unlock time to the current time plus the original `lockMs` value.
+
 ### Graceful Shutdowns and Message Recovery
 
 If your program ends unexpectedly, messages that are currently being processed may become "orphaned" in a locked state - causing channel blockages and reducing throughput. To mitigate this problem, it's essential that you shut down gracefully by catching unhandled exceptions and signals (i.e., `SIGINT`/`SIGTERM`) and finalize all outstanding messages before exiting.
@@ -206,7 +256,7 @@ client.on("notification", (msg) => {
 
 ### Waiting for Job Completion
 
-The `MESSAGE_DELETED` event can be used to create coordination patterns where one part of your application waits for an unrelated job to complete. By listening for deletion events on specific channels or message names, you can implement blocking operations that wait for background work to finish.
+The `MESSAGE_DELETED` event can be used to create coordination patterns where one part of your application waits for an unrelated job to complete. By listening for deletion events on specific message IDs, you can implement blocking operations that wait for background work to finish.
 
 ```typescript
 const client = await databaseClient.connect()
@@ -217,9 +267,7 @@ const wait = (messageId: string) : Promise<void> => {
         const handler = (msg) => {
             if (msg.channel === "EVENTS") {
                 const event = queueEventDecode(msg.payload as string)
-                if (event.eventType === "MESSAGE_DELETED" && 
-                    event.channelName === "background-jobs" && 
-                    event.messageName === messageId) {
+                if (event.eventType === "MESSAGE_DELETED" && event.id === messageId) {
                     client.off("notification", handler)
                     resolve()
                 }
@@ -234,7 +282,7 @@ await wait(messageId)
 
 ## Deadlocks
 
-If all queue actions are isolated to their own transaction, there is zero risk of deadlocks occurring. That being said, it is *possible* to safely bulk-perform the following actions within a single transaction if we ensure they are performed in a consistent lexicographical ordering with respect to channel name and message name (if provided):
+If all queue actions are isolated to their own transaction, there is zero risk of deadlocks occurring. That being said, it is *possible* to safely bulk-perform the following actions within a single transaction if we ensure they are performed in a consistent lexicographical ordering with respect to channel name:
 
 - Message create
 - Channel policy set  
@@ -245,38 +293,43 @@ Beyond the actions specified above, it is manifestly **unsafe** to bulk-perform
 - Message dequeue
 - Message defer
 - Message delete
+- Message heartbeat
 
 ## Batching Operations
 
 When you need to perform multiple safe operations (message creation and channel policy changes) within a single transaction, LonnyMQ provides a batching mechanism that automatically handles proper ordering to prevent deadlocks.
 
-The batch interface mirrors the main queue interface - you call `queue.batch()` to create a batch, then use the same `.channel(name).message.create()` and `.channel(name).policy.set/clear()` methods you're already familiar with. The key difference is that batch operations are queued up and don't execute immediately.
-
-The batch system ensures that all operations are executed in a consistent lexicographical order based on channel name and message name, eliminating the possibility of deadlocks when multiple workers are performing bulk operations simultaneously.
+The batch interface mirrors a subset of the queue interface, supporting only the operations that are safe to perform together: message creation and channel policy management.
 
 ```typescript
 const batch = queue.batch()
 
-batch.channel("user-123").message.create({ 
-    content: Buffer.from("Welcome email") 
+// Create messages (assigned to random channels)
+batch.message.create({ 
+    content: Buffer.from("Welcome email"),
+    lockMs: 30000
 })
 
-batch.channel("user-123").policy.set({
-    maxConcurrency: 5,
-    maxSize: 1000,
-    releaseIntervalMs: 100
+// Create messages on specific channels
+batch.channel("user-notifications").message.create({ 
+    content: Buffer.from("User signup notification"),
+    lockMs: 60000
 })
 
-batch.channel("notifications").message.create({ 
-    content: Buffer.from("Daily digest"),
-    name: "daily-digest-2025-08-29"
+// Set channel policies
+batch.channel("high-priority").policy.set({
+    maxConcurrency: 5,
+    releaseIntervalMs: 100
 })
 
+// Clear channel policies  
 batch.channel("analytics").policy.clear()
 
 await batch.execute({ databaseClient })
 ```
 
+The batch system ensures all operations are executed in a consistent lexicographical order, eliminating the possibility of deadlocks when multiple workers are performing bulk operations simultaneously.
+
 ## Database Clients
 
 LonnyMQ is designed to be database client agnostic, requiring only a minimal interface that most PostgreSQL clients already implement. Your database client must provide a single `query` method with this signature:
@@ -287,4 +340,25 @@ interface DatabaseClient {
         rows: Array<Record<string, unknown>>
     }>
 }
+```
+
+### Database Client Adapters
+
+For database clients that don't match the expected interface exactly, LonnyMQ provides an adapter system to improve the developer experience. You can provide an adapter function when creating a Queue:
+
+```typescript
+import { Queue } from "lonnymq"
+import { Pool } from "pg"
+
+const customClient = new SomeOtherPostgresClient()
+const queue = new Queue({ 
+    schema: "lonny",
+    adaptor: (client) => ({
+        query: async (sql, params) => {
+            // Adapt the client's interface to match DatabaseClient
+            const result = await client.executeQuery(sql, params)
+            return { rows: result.data }
+        }
+    })
+})
 ```