@boringnode/queue 0.5.1 → 0.6.0

package/README.md

@@ -26,6 +26,7 @@ npm install @boringnode/queue
 - **Priority Queues**: Process high-priority jobs first
 - **Bulk Dispatch**: Efficiently dispatch thousands of jobs at once
 - **Job Grouping**: Organize related jobs for monitoring
+- **Job Deduplication**: Prevent duplicate jobs with custom IDs
 - **Retry with Backoff**: Exponential, linear, or fixed backoff strategies
 - **Job Timeout**: Fail or retry jobs that exceed a time limit
 - **Job History**: Retain completed/failed jobs for debugging
@@ -131,6 +132,85 @@ await SendEmailJob.dispatchMany(recipients).group('newsletter-jan-2025')
 
 The `groupId` is stored with job data and accessible via `job.data.groupId`.
 
+## Job Deduplication
+
+Prevent the same job from being pushed multiple times. Four modes, all via `.dedup()`:
+
+### Simple (skip while job exists)
+
+```typescript
+// First dispatch - job is created
+await SendInvoiceJob.dispatch({ orderId: 123 }).dedup({ id: 'order-123' }).run()
+
+// Second dispatch with same dedup ID - silently skipped
+await SendInvoiceJob.dispatch({ orderId: 123 }).dedup({ id: 'order-123' }).run()
+```
+
+### Throttle (skip within TTL window)
+
+```typescript
+// Within 5s, duplicates are skipped. After 5s, a new job is created.
+await SendEmailJob.dispatch({ to: 'user@example.com' })
+  .dedup({ id: 'welcome-123', ttl: '5s' })
+  .run()
+```
+
+### Extend (reset TTL on duplicate)
+
+```typescript
+// Each duplicate push resets the TTL timer.
+await RateLimitJob.dispatch({ userId: 42 }).dedup({ id: 'rate-42', ttl: '1m', extend: true }).run()
+```
+
+### Debounce (replace payload + reset TTL)
+
+```typescript
+// Within the 2s window, the latest payload overwrites the previous pending job.
+await SaveDraftJob.dispatch({ content: 'latest draft' })
+  .dedup({ id: 'draft-42', ttl: '2s', replace: true, extend: true })
+  .run()
+```
+
+### Inspecting the outcome
+
+`DispatchResult` tells you what happened:
+
+```typescript
+const { jobId, deduped } = await SaveDraftJob.dispatch({ content: '...' })
+  .dedup({ id: 'draft-42', ttl: '2s', replace: true })
+  .run()
+
+// deduped: 'added' | 'skipped' | 'replaced' | 'extended'
+// jobId: the UUID of the job (the existing one when deduped)
+```
+
+### How it works
+
+- The dedup ID is automatically prefixed with the job name (`SendInvoiceJob::order-123`), so different job types can reuse the same key.
+- The user-supplied `id` must be ≤ 400 characters, and the combined `<jobName>::<id>` key must be ≤ 510 characters (constrained by the Knex storage column). Both limits are validated at `.dedup()` time.
+- `ttl` accepts a Duration (`'5s'`, `'1m'`) or milliseconds, and must be **positive** when provided. Use `0` or omit `ttl` if you want no expiry — `ttl: 0` is rejected to avoid an ambiguous "expired immediately vs no-expiry" interpretation across engines.
+- `extend` and `replace` **require** `ttl` — calling them without `ttl` throws.
+- `replace` only applies to jobs in `pending` or `delayed` state. Jobs that are active (executing) or retained in history (`completed`/`failed` with retention) are left alone; the dispatch returns `{ deduped: 'skipped' }`.
+- `replace` swaps the **payload only** — priority, queue, delay, groupId, and stored dedup options of the existing job are retained. To change those, use a different dedup id or wait for the TTL to expire.
+- `extend` resets the TTL clock but never changes the window length. The window length is fixed to the `ttl` from the first dispatch that created the dedup slot. Later dispatches that pass a different `ttl` only reset the clock; their `ttl` value is ignored. To resize the window, let the slot expire and start over with a new dispatch.
+- `extend` works in **all states** — even when the existing job is `active` (executing) or retained in history. Unlike `replace` (which is no-op on non-replaceable states), `extend` always refreshes the dedup TTL window. Use this when you want the dedup slot to keep blocking new dispatches for the lifetime of a long-running job.
+- `extend` requires the **first** dispatch to have set a `ttl`. If the slot was created without a `ttl`, later `extend` dispatches have no window to refresh and return `{ deduped: 'skipped' }` instead of `'extended'`.
+- `retryJob` does not touch the dedup entry — a retried job continues to occupy the dedup slot. TTL runs on wall-clock time, so long-running retries may outlive the TTL window. Use a generous TTL or no TTL if retries must stay deduped.
+- Atomic and race-free:
+  - **Redis**: a single Lua script per dispatch performs the dedup-key lookup, state check (pending/delayed ZSCORE), payload swap, and TTL refresh atomically.
+  - **Knex**: transactional `SELECT ... FOR UPDATE` + insert/update inside a transaction. A nested savepoint catches unique-constraint violations under concurrent inserts and returns `{ deduped: 'skipped' }` pointing at the winner.
+  - **SyncAdapter**: executes inline, no dedup support.
+
+### Caveats
+
+- Without `.dedup()`, jobs use auto-generated UUIDs and are never deduplicated.
+- The **Sync adapter** ignores `.dedup()` entirely — every dispatch executes inline and `deduped` is always `undefined` on the result. Use Redis or Knex if you need real deduplication.
+- `.dedup()` is only available on single dispatch. `dispatchMany` / `pushManyOn` reject jobs with a `dedup` field.
+- Scheduled jobs (`.schedule()`) do not support dedup — each cron/interval fire is an independent dispatch.
+- With no `ttl`, dedup persists until the job is removed (completed/failed without retention). When retention keeps the record, re-dispatch stays blocked until the record is pruned.
+- With `ttl`, dedup expires after the window — a new job (new UUID) is created. The old job still runs.
+- Knex MySQL concurrent race: MySQL does not support partial unique indexes, so two `pushOn` calls with the same dedup id firing at the exact same instant can both succeed. Serialize at the app layer if strict guarantees are required, or use Postgres / SQLite / Redis (all of which serialize correctly via the partial unique index or Lua atomicity).
+
 ## Job History & Retention
 
 Keep completed and failed jobs for debugging:
@@ -270,20 +350,22 @@ await QueueManager.init({
   locations: ['./app/jobs/**/*.ts'],
 })
 
-const adapter = QueueManager.fake()
+// The `using` keyword automatically restores the real adapters when
+// the variable goes out of scope (at the end of the test function).
+using fake = QueueManager.fake()
 
 await SendEmailJob.dispatch({ to: 'user@example.com' })
 
-adapter.assertPushed(SendEmailJob)
-adapter.assertPushed(SendEmailJob, {
+fake.assertPushed(SendEmailJob)
+fake.assertPushed(SendEmailJob, {
   queue: 'default',
   payload: (payload) => payload.to === 'user@example.com',
 })
-adapter.assertPushedCount(1)
-
-QueueManager.restore()
+fake.assertPushedCount(1)
 ```
 
+You can also call `QueueManager.restore()` manually if you need more control over when the real adapters are restored.
+
 ### Sync (for testing)
 
 ```typescript
@@ -534,7 +616,7 @@ import * as boringqueue from '@boringnode/queue'
 
 const instrumentation = new QueueInstrumentation({
   messagingSystem: 'boringqueue', // default
-  executionSpanLinkMode: 'link',  // or 'parent'
+  executionSpanLinkMode: 'link', // or 'parent'
 })
 
 instrumentation.enable()
@@ -547,19 +629,20 @@ The instrumentation patches `QueueManager.init()` to automatically inject its wr
 
 The instrumentation uses standard [OTel messaging semantic conventions](https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/) where they map cleanly, plus a few queue-specific custom attributes.
 
-| Attribute                       | Kind    | Description                                |
-| ------------------------------- | ------- | ------------------------------------------ |
-| `messaging.system`              | Semconv | `'boringqueue'` (configurable)             |
-| `messaging.operation.name`      | Semconv | `'publish'` or `'process'`                 |
-| `messaging.destination.name`    | Semconv | Queue name                                 |
-| `messaging.message.id`          | Semconv | Job ID for single-message spans            |
-| `messaging.batch.message_count` | Semconv | Number of jobs in a batch dispatch         |
-| `messaging.message.retry.count` | Custom  | Retry count (0-based) for a job attempt    |
-| `messaging.job.name`            | Custom  | Job class name (e.g. `SendEmailJob`)       |
-| `messaging.job.status`          | Custom  | `'completed'`, `'failed'`, or `'retrying'` |
-| `messaging.job.group_id`        | Custom  | Queue-specific group identifier            |
-| `messaging.job.priority`        | Custom  | Queue-specific job priority                |
-| `messaging.job.delay_ms`        | Custom  | Delay before the job becomes available     |
+| Attribute                       | Kind    | Description                                   |
+| ------------------------------- | ------- | --------------------------------------------- |
+| `messaging.system`              | Semconv | `'boringqueue'` (configurable)                |
+| `messaging.operation.name`      | Semconv | `'publish'` or `'process'`                    |
+| `messaging.destination.name`    | Semconv | Queue name                                    |
+| `messaging.message.id`          | Semconv | Job ID for single-message spans               |
+| `messaging.batch.message_count` | Semconv | Number of jobs in a batch dispatch            |
+| `messaging.message.retry.count` | Custom  | Retry count (0-based) for a job attempt       |
+| `messaging.job.name`            | Custom  | Job class name (e.g. `SendEmailJob`)          |
+| `messaging.job.status`          | Custom  | `'completed'`, `'failed'`, or `'retrying'`    |
+| `messaging.job.group_id`        | Custom  | Queue-specific group identifier               |
+| `messaging.job.priority`        | Custom  | Queue-specific job priority                   |
+| `messaging.job.delay_ms`        | Custom  | Delay before the job becomes available        |
+| `messaging.job.queue_time_ms`   | Custom  | Time spent waiting in queue before processing |
 
 ### Trace Context Propagation
 

package/build/chunk-6IO4P6RB.js

@@ -0,0 +1,145 @@
+// src/drivers/redis_job_storage.ts
+function hydrateRedisJob(data, overlay) {
+  const jobData = JSON.parse(data);
+  const parsedOverlay = overlay ? JSON.parse(overlay) : void 0;
+  if (parsedOverlay?.payloadUndefined) {
+    jobData.payload = void 0;
+  } else if (parsedOverlay?.payload !== void 0) {
+    jobData.payload = JSON.parse(parsedOverlay.payload);
+  }
+  return {
+    ...jobData,
+    ...parsedOverlay?.attempts !== void 0 ? { attempts: parsedOverlay.attempts } : {},
+    ...parsedOverlay?.stalledCount !== void 0 ? { stalledCount: parsedOverlay.stalledCount } : {}
+  };
+}
+function encodeRedisJobPayloadOverlay(payload) {
+  const encoded = JSON.stringify(payload);
+  return encoded === void 0 ? ["", "1"] : [encoded, "0"];
+}
+var REDIS_JOB_STORAGE_LUA = `
+  local function read_job_overlay(overlay_key, job_id)
+    local overlay_json = redis.call('HGET', overlay_key, job_id)
+    if overlay_json then
+      return cjson.decode(overlay_json)
+    end
+    return {}
+  end
+
+  local function write_job_overlay(overlay_key, job_id, overlay)
+    local has_overlay = false
+    for _ in pairs(overlay) do
+      has_overlay = true
+      break
+    end
+
+    if has_overlay then
+      redis.call('HSET', overlay_key, job_id, cjson.encode(overlay))
+    else
+      redis.call('HDEL', overlay_key, job_id)
+    end
+  end
+
+  local function store_job_data(data_key, overlay_key, job_id, job_data)
+    redis.call('HDEL', overlay_key, job_id)
+    redis.call('HSET', data_key, job_id, job_data)
+  end
+
+  local function delete_job_data(data_key, overlay_key, job_id)
+    redis.call('HDEL', data_key, job_id)
+    redis.call('HDEL', overlay_key, job_id)
+  end
+
+  local function delete_jobs_data(data_key, overlay_key, job_ids)
+    if #job_ids > 0 then
+      redis.call('HDEL', data_key, unpack(job_ids))
+      redis.call('HDEL', overlay_key, unpack(job_ids))
+    end
+  end
+
+  local function encode_job_result(job_data, overlay_key, job_id, result)
+    local encoded = result or {}
+    encoded.data = job_data
+
+    local overlay_json = redis.call('HGET', overlay_key, job_id)
+    if overlay_json then
+      encoded.overlay = overlay_json
+    end
+
+    return cjson.encode(encoded)
+  end
+
+  local function apply_payload_overlay(overlay, payload_data, payload_is_undefined)
+    if payload_is_undefined == 1 then
+      overlay.payload = nil
+      overlay.payloadUndefined = true
+    else
+      overlay.payload = payload_data
+      overlay.payloadUndefined = nil
+    end
+  end
+`;
+var REDIS_DEDUP_LUA = `
+  local function resolve_dedup_existing_job(
+    data_key,
+    state_key_a,
+    state_key_b,
+    overlay_key,
+    dedup_key,
+    extend,
+    replace,
+    payload_data,
+    payload_is_undefined
+  )
+    local existing = redis.call('GET', dedup_key)
+    if not existing then
+      return nil
+    end
+
+    local existing_data = redis.call('HGET', data_key, existing)
+    if not existing_data then
+      return nil
+    end
+
+    local in_state_a = redis.call('ZSCORE', state_key_a, existing)
+    local in_state_b = redis.call('ZSCORE', state_key_b, existing)
+    local replaceable = in_state_a or in_state_b
+    local ok, existing_job = pcall(cjson.decode, existing_data)
+    local original_ttl = nil
+
+    if ok and type(existing_job) == 'table' and existing_job.dedup then
+      original_ttl = tonumber(existing_job.dedup.ttl)
+    end
+
+    if replace == 1 and replaceable then
+      if ok and existing_job then
+        local overlay = read_job_overlay(overlay_key, existing)
+        apply_payload_overlay(overlay, payload_data, payload_is_undefined)
+        write_job_overlay(overlay_key, existing, overlay)
+
+        if extend == 1 and original_ttl and original_ttl > 0 then
+          redis.call('PEXPIRE', dedup_key, original_ttl)
+        end
+
+        return {'replaced', existing}
+      end
+
+      return {'skipped', existing}
+    end
+
+    if extend == 1 and original_ttl and original_ttl > 0 then
+      redis.call('PEXPIRE', dedup_key, original_ttl)
+      return {'extended', existing}
+    end
+
+    return {'skipped', existing}
+  end
+`;
+
+export {
+  hydrateRedisJob,
+  encodeRedisJobPayloadOverlay,
+  REDIS_JOB_STORAGE_LUA,
+  REDIS_DEDUP_LUA
+};
+//# sourceMappingURL=chunk-6IO4P6RB.js.map

package/build/chunk-6IO4P6RB.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/drivers/redis_job_storage.ts"],"sourcesContent":["import type { JobData } from '../types/main.js'\n\n/**\n * Redis stores the original job JSON as an opaque payload in the data hash.\n * Lua scripts write only mutable Redis-side overrides in this overlay so\n * legacy jobs without overlays keep decoding from their original data.\n */\nexport type RedisJobOverlay = Partial<Pick<JobData, 'attempts' | 'stalledCount'>> & {\n  payload?: string\n  payloadUndefined?: boolean\n}\n\nexport function hydrateRedisJob(data: string, overlay?: string | null | false): JobData {\n  const jobData = JSON.parse(data) as JobData\n  const parsedOverlay = overlay ? (JSON.parse(overlay) as RedisJobOverlay) : undefined\n\n  if (parsedOverlay?.payloadUndefined) {\n    jobData.payload = undefined\n  } else if (parsedOverlay?.payload !== undefined) {\n    jobData.payload = JSON.parse(parsedOverlay.payload)\n  }\n\n  return {\n    ...jobData,\n    ...(parsedOverlay?.attempts !== undefined ? { attempts: parsedOverlay.attempts } : {}),\n    ...(parsedOverlay?.stalledCount !== undefined\n      ? { stalledCount: parsedOverlay.stalledCount }\n      : {}),\n  }\n}\n\nexport function encodeRedisJobPayloadOverlay(payload: JobData['payload']): [string, string] {\n  const encoded = JSON.stringify(payload)\n\n  return encoded === undefined ? ['', '1'] : [encoded, '0']\n}\n\nexport const REDIS_JOB_STORAGE_LUA = `\n  local function read_job_overlay(overlay_key, job_id)\n    local overlay_json = redis.call('HGET', overlay_key, job_id)\n    if overlay_json then\n      return cjson.decode(overlay_json)\n    end\n    return {}\n  end\n\n  local function write_job_overlay(overlay_key, job_id, overlay)\n    local has_overlay = false\n    for _ in pairs(overlay) do\n      has_overlay = true\n      break\n    end\n\n    if has_overlay then\n      redis.call('HSET', overlay_key, job_id, cjson.encode(overlay))\n    else\n      redis.call('HDEL', overlay_key, job_id)\n    end\n  end\n\n  local function store_job_data(data_key, overlay_key, job_id, job_data)\n    redis.call('HDEL', overlay_key, job_id)\n    redis.call('HSET', data_key, job_id, job_data)\n  end\n\n  local function delete_job_data(data_key, overlay_key, job_id)\n    redis.call('HDEL', data_key, job_id)\n    redis.call('HDEL', overlay_key, job_id)\n  end\n\n  local function delete_jobs_data(data_key, overlay_key, job_ids)\n    if #job_ids > 0 then\n      redis.call('HDEL', data_key, unpack(job_ids))\n      redis.call('HDEL', overlay_key, unpack(job_ids))\n    end\n  end\n\n  local function encode_job_result(job_data, overlay_key, job_id, result)\n    local encoded = result or {}\n    encoded.data = job_data\n\n    local overlay_json = redis.call('HGET', overlay_key, job_id)\n    if overlay_json then\n      encoded.overlay = overlay_json\n    end\n\n    return cjson.encode(encoded)\n  end\n\n  local function apply_payload_overlay(overlay, payload_data, payload_is_undefined)\n    if payload_is_undefined == 1 then\n      overlay.payload = nil\n      overlay.payloadUndefined = true\n    else\n      overlay.payload = payload_data\n      overlay.payloadUndefined = nil\n    end\n  end\n`\n\nexport const REDIS_DEDUP_LUA = `\n  local function resolve_dedup_existing_job(\n    data_key,\n    state_key_a,\n    state_key_b,\n    overlay_key,\n    dedup_key,\n    extend,\n    replace,\n    payload_data,\n    payload_is_undefined\n  )\n    local existing = redis.call('GET', dedup_key)\n    if not existing then\n      return nil\n    end\n\n    local existing_data = redis.call('HGET', data_key, existing)\n    if not existing_data then\n      return nil\n    end\n\n    local in_state_a = redis.call('ZSCORE', state_key_a, existing)\n    local in_state_b = redis.call('ZSCORE', state_key_b, existing)\n    local replaceable = in_state_a or in_state_b\n    local ok, existing_job = pcall(cjson.decode, existing_data)\n    local original_ttl = nil\n\n    if ok and type(existing_job) == 'table' and existing_job.dedup then\n      original_ttl = tonumber(existing_job.dedup.ttl)\n    end\n\n    if replace == 1 and replaceable then\n      if ok and existing_job then\n        local overlay = read_job_overlay(overlay_key, existing)\n        apply_payload_overlay(overlay, payload_data, payload_is_undefined)\n        write_job_overlay(overlay_key, existing, overlay)\n\n        if extend == 1 and original_ttl and original_ttl > 0 then\n          redis.call('PEXPIRE', dedup_key, original_ttl)\n        end\n\n        return {'replaced', existing}\n      end\n\n      return {'skipped', existing}\n    end\n\n    if extend == 1 and original_ttl and original_ttl > 0 then\n      redis.call('PEXPIRE', dedup_key, original_ttl)\n      return {'extended', existing}\n    end\n\n    return {'skipped', existing}\n  end\n`\n"],"mappings":";AAYO,SAAS,gBAAgB,MAAc,SAA0C;AACtF,QAAM,UAAU,KAAK,MAAM,IAAI;AAC/B,QAAM,gBAAgB,UAAW,KAAK,MAAM,OAAO,IAAwB;AAE3E,MAAI,eAAe,kBAAkB;AACnC,YAAQ,UAAU;AAAA,EACpB,WAAW,eAAe,YAAY,QAAW;AAC/C,YAAQ,UAAU,KAAK,MAAM,cAAc,OAAO;AAAA,EACpD;AAEA,SAAO;AAAA,IACL,GAAG;AAAA,IACH,GAAI,eAAe,aAAa,SAAY,EAAE,UAAU,cAAc,SAAS,IAAI,CAAC;AAAA,IACpF,GAAI,eAAe,iBAAiB,SAChC,EAAE,cAAc,cAAc,aAAa,IAC3C,CAAC;AAAA,EACP;AACF;AAEO,SAAS,6BAA6B,SAA+C;AAC1F,QAAM,UAAU,KAAK,UAAU,OAAO;AAEtC,SAAO,YAAY,SAAY,CAAC,IAAI,GAAG,IAAI,CAAC,SAAS,GAAG;AAC1D;AAEO,IAAM,wBAAwB;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AA+D9B,IAAM,kBAAkB;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;","names":[]}