@bymax-one/nest-queue 1.2.14 → 1.2.15

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -11,6 +11,29 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
11
11
 
12
12
  ---
13
13
 
14
+ ## [1.2.15] — 2026-11-14
15
+
16
+ Against 1.2.14 the built declarations gain `ConnectionResolver.awaitReady()`; nothing else in the public API changes.
17
+
18
+ ### Fixed
19
+
20
+ - **A call made while the connection is down now reports the documented error.** With the owned connection's offline queue off, a call on a queue already in use during a refused-connection outage rejected at once with ioredis's plain `Stream isn't writeable and enableOfflineQueue options is false`, so a host retrying on `CONNECTION_TIMEOUT` as documented never retried. It is now `CONNECTION_TIMEOUT` with `details.operation` and `details.reason: 'offline'`, and the ioredis error as `cause`.
21
+ - **The first call on a queue no longer waits for Redis without a deadline.** `getMetrics`, `getJob`, `getJobs`, the scheduler calls, `pauseQueue`, `resumeQueue` and `cleanQueue` on a queue name not yet used waited in BullMQ for the client to be ready, with no bound, for as long as the outage lasted. Every `QueueService` call is now bounded by `enqueueTimeoutMs` and rejects with `CONNECTION_TIMEOUT`.
22
+ - **Shutdown no longer leaves a worker running after its drain budget.** A handler that never settled kept the worker's lock-renewal timer alive after `app.close()`: the process did not exit by itself and logged `Connection is closed.` repeatedly. BullMQ ignores a forced close once a graceful one has started, so the worker is now paused first and closed with `close(true)` as its first close when the budget runs out. What the abandoned handler produces after the budget, a result or a failure, is dropped, so BullMQ cannot resume the stopped worker and reopen its connection. The handler's own promise still keeps running, since user code cannot be cancelled. A sandboxed worker is the exception: its job runs in a child process the registry cannot wrap, and in a measured run the host process did not exit by itself after the budget ended with such a job still running.
23
+
24
+ ### Documentation
25
+
26
+ - `enqueueTimeoutMs` is documented as the per-call deadline of every `QueueService` call, and `queue.connection_timeout` as covering a connection that is down.
27
+
28
+ #### Apply to a derived backend
29
+
30
+ - If you match the ioredis error `Stream isn't writeable and enableOfflineQueue options is false` on a call, match `CONNECTION_TIMEOUT` instead; `details.reason` is `'offline'` and the original error is its `cause`.
31
+ - If a `getMetrics`, `getJob`, `getJobs`, scheduler, pause, resume or clean call relied on waiting out an outage on a queue's first use, it now rejects with `CONNECTION_TIMEOUT` after `enqueueTimeoutMs`; raise the option to wait longer.
32
+ - If you replace `ConnectionResolver` with your own implementation, add `awaitReady(): Promise<void>`: `QueueService` calls it before the first call on each queue name.
33
+ - If you parse the shutdown log lines, the wording changed: `forcing close` is gone and the summary reads `N worker(s) stopped without draining`.
34
+
35
+ ---
36
+
14
37
  ## [1.2.14] — 2026-11-14
15
38
 
16
39
  Several defaults change in this release, so read the Apply note below. A boot that succeeds, an
@@ -1013,7 +1036,8 @@ v6 peer range.
1013
1036
 
1014
1037
  ---
1015
1038
 
1016
- [Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.14...HEAD
1039
+ [Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.15...HEAD
1040
+ [1.2.15]: https://github.com/bymaxone/nest-queue/compare/v1.2.14...v1.2.15
1017
1041
  [1.2.14]: https://github.com/bymaxone/nest-queue/compare/v1.2.13...v1.2.14
1018
1042
  [1.2.13]: https://github.com/bymaxone/nest-queue/compare/v1.2.12...v1.2.13
1019
1043
  [1.2.12]: https://github.com/bymaxone/nest-queue/compare/v1.2.11...v1.2.12
package/README.md CHANGED
@@ -66,7 +66,7 @@ peer dependencies, so you control exact versions and the supply-chain surface st
66
66
  - ✅ **Dual-mode connection** — bring your own `ioredis` client, or let the library open its own from a URL or an options object
67
67
  - ✅ **Per-role retry policy** — `maxRetriesPerRequest: null` reaches only the duplicated Worker and QueueEvents connections; the producer keeps a finite retry limit, a 5 s command deadline and no offline queue on the connection the library opens, and `enqueue` has its own 10 s deadline, so a Redis outage rejects a call in seconds instead of holding the request; a shared client with `maxRetriesPerRequest: null` is the exception, and the library warns about it at boot
68
68
  - ✅ **One namespace, four objects** — the configured `prefix` reaches `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer`, so producers and consumers can never end up on separate keyspaces
69
- - ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under a deadline and then force-closes; delivery is at-least-once by design, so handlers must be idempotent
69
+ - ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under one deadline and then stops any that are still busy, whose jobs stall and are retried; delivery is at-least-once by design, so handlers must be idempotent
70
70
  - ✅ **No unlistened emitter** — every `Queue`, `Worker`, `QueueEvents` and `FlowProducer` the library creates carries a fallback `error` listener, so a transient Redis fault cannot become an uncaught exception in your process; your own `@OnWorkerEvent('error')` or `@OnQueueEvent('error')` still runs alongside it
71
71
 
72
72
  ### 🧩 Developer Experience
@@ -243,7 +243,7 @@ Every option passed to `forRoot` / `forRootAsync`. Only `connection` is required
243
243
  | `shutdown.drainOnShutdown` | `boolean` | `false` | Development and test only: once the HTTP server has stopped, call `drain()` on every cached queue, which **removes** its waiting jobs (delayed and active jobs are left alone). Leave it `false` in production. |
244
244
  | `connectionReadyTimeoutMs` | `number` | `10_000` | Mode B only: the budget for Redis to reach `ready`. A connection error while the client is still reconnecting does not end the wait; when the budget ends, startup fails with `CONNECTION_TIMEOUT` carrying `lastErrorCode` (a code such as `ECONNREFUSED`, never the message) and the client the library opened is closed. |
245
245
  | `commandTimeoutMs` | `number` | `5_000` | Mode B only: the deadline for one Redis command on the connection the library opened (ioredis `commandTimeout`); an `options.commandTimeout` wins. A call that hits it rejects with `CONNECTION_TIMEOUT` (`details` carries `operation` and `timeoutMs`), as when `enqueueTimeoutMs` passes. Worker and QueueEvents connections have none. |
246
- | `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `enqueue` or `enqueueBulk`, in either mode; it rejects with `CONNECTION_TIMEOUT`, and the add may still complete in Redis, so retry with a stable `jobId` or `deduplication`. |
246
+ | `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `QueueService` call that reaches Redis, in either mode, including the wait for the client to be ready on a queue's first call; it rejects with `CONNECTION_TIMEOUT`, and an `enqueue` may still complete in Redis, so retry with a stable `jobId` or `deduplication`. |
247
247
  | `isGlobal` | `boolean` | `true` | Register the module globally, so `QueueService` is injectable without re-importing. |
248
248
 
249
249
  > [!WARNING]
@@ -440,7 +440,7 @@ When a custom `jobId` already exists, or a `deduplication` rule keeps the job th
440
440
 
441
441
  ### A timed-out `enqueue` may still have been added
442
442
 
443
- When Redis does not answer in time, whether within `commandTimeoutMs` on the connection the library opened or within `enqueueTimeoutMs`, `enqueue` and `enqueueBulk` reject with `CONNECTION_TIMEOUT`, but the library cannot cancel the command: if Redis applies it late, the job exists anyway. Retrying a call that timed out can therefore add the job twice. For anything that must not run twice, pass a deterministic `jobId` (for example one derived from the business key): BullMQ ignores a second add with the same `jobId` for as long as the first job is still stored, so a retry inside that window is safe. A completed job is removed by `removeOnComplete`, and from then on the same `jobId` adds a new job, so keep the retry inside that window or keep the handler idempotent. A `deduplication` rule gives the same guarantee for work that is allowed to repeat after a window.
443
+ When Redis does not answer in time, whether within `commandTimeoutMs` on the connection the library opened or within `enqueueTimeoutMs`, or the connection is down, `enqueue` and `enqueueBulk` reject with `CONNECTION_TIMEOUT`, but the library cannot cancel the command: if Redis applies it late, the job exists anyway. Retrying a call that timed out can therefore add the job twice. For anything that must not run twice, pass a deterministic `jobId` (for example one derived from the business key): BullMQ ignores a second add with the same `jobId` for as long as the first job is still stored, so a retry inside that window is safe. A completed job is removed by `removeOnComplete`, and from then on the same `jobId` adds a new job, so keep the retry inside that window or keep the handler idempotent. A `deduplication` rule gives the same guarantee for work that is allowed to repeat after a window.
444
444
 
445
445
  ### OpenTelemetry
446
446
 
@@ -465,10 +465,15 @@ It implements a **bounded drain protocol** with at-least-once semantics:
465
465
  1. All workers are closed **in parallel**, within one budget (`drainTimeoutMs`, default 30 s) that
466
466
  covers steps 1 to 4 together: however many workers are stuck, consumption stops after one budget,
467
467
  not one per worker.
468
- 2. A worker resolves either when it drains normally or when the budget runs out — whichever comes
469
- first.
470
- 3. On timeout, the worker is force-closed (`worker.close(true)`), preserving stalled
471
- jobs for reprocessing by the next instance.
468
+ 2. Each worker is paused (it stops fetching and waits for its active jobs) and then closed, or the
469
+ budget runs out — whichever comes first.
470
+ 3. On timeout, the worker is stopped with `worker.close(true)` as its first close, so it does not
471
+ wait for the job and its lock renewal ends; the lock expires and the job stalls and is retried
472
+ by the next instance. What the abandoned handler produces afterwards, a result or a failure, is
473
+ dropped, so the stopped worker cannot restart its connection. The handler's own promise keeps
474
+ running, because user code cannot be cancelled. A sandboxed (file-based) worker is the
475
+ exception: its job runs in a child process the registry cannot wrap, and in a measured run the
476
+ host process did not exit by itself after the budget ended with such a job still running.
472
477
  4. Queue-events and the connections the library opened for workers and queue-events are closed,
473
478
  each within what is left of the budget; a connection that has not quit by then is disconnected.
474
479
  5. Once NestJS has stopped the HTTP server (`onApplicationShutdown`), the flow producer, the
@@ -478,7 +483,7 @@ It implements a **bounded drain protocol** with at-least-once semantics:
478
483
 
479
484
  **Handlers must be idempotent.** Use `jobId`, `deduplication`, or application-level
480
485
  idempotency keys to guard against reprocessing a job that was partially handled before
481
- a crash or force-close.
486
+ a crash or a shutdown that outlasts the drain budget.
482
487
 
483
488
  **DLQ pattern.** When a job exhausts its `attempts`, move it to a dead-letter queue:
484
489
 
@@ -506,14 +511,14 @@ BymaxQueueModule.forRoot({
506
511
 
507
512
  ## 🤔 Why over `@nestjs/bullmq`
508
513
 
509
- | Concern | `@nestjs/bullmq` (official) | `@bymax-one/nest-queue` |
510
- | ------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------- |
511
- | Module setup | `BullModule.forRoot()` + `registerQueue()` per queue | Single `forRoot`/`forRootAsync`; queues created on demand by name |
512
- | Connection | Manual ioredis + `maxRetriesPerRequest` wiring | Dual mode (BYO / lib-owned); correct per-role retry policy applied automatically |
513
- | Job defaults | Per `registerQueue` | Centralized opinionated defaults with per-queue/per-job overrides |
514
- | Producer API | `@InjectQueue` + raw `queue.add` | Typed `enqueue<TData, TResult>()`, `enqueueBulk`, `upsertJobScheduler` |
515
- | Shutdown | Framework closes workers | Explicit bounded-drain protocol; force-close + stalled accounting |
516
- | Observability | Manual | `telemetry` passthrough (OpenTelemetry) + `MetricsService` |
514
+ | Concern | `@nestjs/bullmq` (official) | `@bymax-one/nest-queue` |
515
+ | ------------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------- |
516
+ | Module setup | `BullModule.forRoot()` + `registerQueue()` per queue | Single `forRoot`/`forRootAsync`; queues created on demand by name |
517
+ | Connection | Manual ioredis + `maxRetriesPerRequest` wiring | Dual mode (BYO / lib-owned); correct per-role retry policy applied automatically |
518
+ | Job defaults | Per `registerQueue` | Centralized opinionated defaults with per-queue/per-job overrides |
519
+ | Producer API | `@InjectQueue` + raw `queue.add` | Typed `enqueue<TData, TResult>()`, `enqueueBulk`, `upsertJobScheduler` |
520
+ | Shutdown | Framework closes workers | Explicit bounded-drain protocol; stops a busy worker at the deadline; stalled accounting |
521
+ | Observability | Manual | `telemetry` passthrough (OpenTelemetry) + `MetricsService` |
517
522
 
518
523
  **Choose this library** when you want a single opinionated setup, the `@bymax-one/nest-cache`
519
524
  connection-sharing story, enforced defaults, and a tested bounded-shutdown protocol.
@@ -552,7 +557,7 @@ wire connection and defaults yourself.
552
557
  (@Processor / @Process / @OnWorkerEvent)
553
558
  │
554
559
  QueueLifecycle
555
- (bounded drain → force close → disconnect)
560
+ (bounded drain → stop → disconnect)
556
561
  ```
557
562
 
558
563
  ### Design Principles
@@ -850,18 +855,18 @@ try {
850
855
  }
851
856
  ```
852
857
 
853
- | Code | HTTP | When thrown |
854
- | ---------------------------------------- | :--: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
855
- | `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
856
- | `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, Mode A's injected client is in an unusable `status`, or a Mode B client was refused by the server (`WRONGPASS`, `NOAUTH`) or stopped reconnecting during startup |
857
- | `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
858
- | `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` (`details` carries `timeoutMs` and the last error's `lastErrorCode`), or a `QueueService`/`FlowService` call got no answer within the connection's `commandTimeoutMs`, or `enqueue`/`enqueueBulk` within `enqueueTimeoutMs` (`details` carries `operation` and `timeoutMs`; an enqueue may still have been added) |
859
- | `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
860
- | `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
861
- | `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
862
- | `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
863
- | `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
864
- | `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
858
+ | Code | HTTP | When thrown |
859
+ | ---------------------------------------- | :--: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
860
+ | `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
861
+ | `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, Mode A's injected client is in an unusable `status`, or a Mode B client was refused by the server (`WRONGPASS`, `NOAUTH`) or stopped reconnecting during startup |
862
+ | `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
863
+ | `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` (`details` carries `timeoutMs` and the last error's `lastErrorCode`), or a `QueueService`/`FlowService` call got no answer from Redis: it timed out (`commandTimeoutMs` on the owned connection, or `enqueueTimeoutMs` for a `QueueService` call; `details` carries `operation` and `timeoutMs`) or the connection is down (`details` carries `operation` and `reason: 'offline'`); an enqueue may still have been added |
864
+ | `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
865
+ | `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
866
+ | `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
867
+ | `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
868
+ | `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
869
+ | `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
865
870
 
866
871
  ### Reserved codes
867
872
 
@@ -869,12 +874,12 @@ These are part of the published catalog and stable, but nothing currently throws
869
874
  They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
870
875
  compiles today and keeps compiling when a future release starts emitting them.
871
876
 
872
- | Code | Why it is not thrown today |
873
- | --------------------------------- | --------------------------------------------------------------------- |
874
- | `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
875
- | `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
876
- | `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
877
- | `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline force-closes workers rather than raising |
877
+ | Code | Why it is not thrown today |
878
+ | --------------------------------- | ----------------------------------------------------------------------------- |
879
+ | `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
880
+ | `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
881
+ | `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
882
+ | `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline stops the workers still busy rather than raising |
878
883
 
879
884
  The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
880
885
  so a frontend can map codes to messages without importing the server surface.
@@ -908,7 +913,7 @@ If you see it, the client you passed is not a plain ioredis client (a wrapper or
908
913
  double whose `duplicate()` ignores its options). Pass the ioredis client itself, or let the
909
914
  library open its own connection with Mode B. The producer keeps the client's own retry
910
915
  policy; with a finite `maxRetriesPerRequest` that is what lets it fail fast during an outage, and
911
- `enqueue` is bounded by `enqueueTimeoutMs` whatever the client does.
916
+ every `QueueService` call is bounded by `enqueueTimeoutMs` whatever the client does.
912
917
 
913
918
  ### A handler never settles (the queue stops moving)
914
919
 
@@ -250,13 +250,14 @@ function duplicateConnection(client) {
250
250
  }
251
251
 
252
252
  // src/server/services/connection-resolver.service.ts
253
- var _logger, _client, _options;
253
+ var _logger, _client, _options, _pendingReady;
254
254
  exports.ConnectionResolver = class ConnectionResolver {
255
255
  constructor(options) {
256
256
  __privateAdd(this, _logger, new common.Logger(exports.ConnectionResolver.name));
257
257
  __privateAdd(this, _client);
258
258
  __privateAdd(this, _options);
259
259
  __publicField(this, "mode");
260
+ __privateAdd(this, _pendingReady);
260
261
  __privateSet(this, _options, options);
261
262
  }
262
263
  async init() {
@@ -312,6 +313,17 @@ exports.ConnectionResolver = class ConnectionResolver {
312
313
  isOwned() {
313
314
  return this.mode === "mode-b-owned";
314
315
  }
316
+ awaitReady() {
317
+ const client = this.getClient();
318
+ if (!WAITING_STATUSES.has(client.status)) return Promise.resolve();
319
+ __privateGet(this, _pendingReady) ?? __privateSet(this, _pendingReady, new Promise((resolve) => {
320
+ client.once("ready", () => {
321
+ __privateSet(this, _pendingReady, void 0);
322
+ resolve();
323
+ });
324
+ }));
325
+ return __privateGet(this, _pendingReady);
326
+ }
315
327
  async teardown() {
316
328
  if (this.isOwned() && __privateGet(this, _client)) {
317
329
  const client = __privateGet(this, _client);
@@ -332,7 +344,7 @@ exports.ConnectionResolver = class ConnectionResolver {
332
344
  }
333
345
  if (client.options.maxRetriesPerRequest === null) {
334
346
  __privateGet(this, _logger).warn(
335
- "The shared client has maxRetriesPerRequest: null, so a producer call made while Redis is unreachable waits for as long as the outage lasts; enqueue is bounded by enqueueTimeoutMs, other QueueService calls are not. Give the client a finite limit to fail fast."
347
+ "The shared client has maxRetriesPerRequest: null, so a command issued while Redis is unreachable stays queued in the client for as long as the outage lasts. QueueService calls are bounded by enqueueTimeoutMs and the caller is released, but the command is not cancelled. Give the client a finite limit to fail fast."
336
348
  );
337
349
  }
338
350
  const probe = duplicateConnection(client);
@@ -382,10 +394,17 @@ exports.ConnectionResolver = class ConnectionResolver {
382
394
  _logger = new WeakMap();
383
395
  _client = new WeakMap();
384
396
  _options = new WeakMap();
397
+ _pendingReady = new WeakMap();
385
398
  exports.ConnectionResolver = __decorateClass([
386
399
  common.Injectable(),
387
400
  __decorateParam(0, common.Inject(BYMAX_QUEUE_OPTIONS))
388
401
  ], exports.ConnectionResolver);
402
+ var WAITING_STATUSES = /* @__PURE__ */ new Set([
403
+ "connecting",
404
+ "connect",
405
+ "reconnecting",
406
+ "close"
407
+ ]);
389
408
  function isReplyError(err) {
390
409
  return typeof err === "object" && err !== null && err.name === "ReplyError";
391
410
  }
@@ -456,6 +475,18 @@ function timeoutException(operation, timeoutMs, cause) {
456
475
  if (cause !== void 0) exception.cause = cause;
457
476
  return exception;
458
477
  }
478
+ function unavailableException(operation, cause) {
479
+ const exception = new QueueException(
480
+ QUEUE_ERROR_CODES.CONNECTION_TIMEOUT,
481
+ common.HttpStatus.INTERNAL_SERVER_ERROR,
482
+ { operation, reason: "offline" }
483
+ );
484
+ exception.cause = cause;
485
+ return exception;
486
+ }
487
+ function isOfflineRejection(err) {
488
+ return err instanceof Error && err.message === "Stream isn't writeable and enableOfflineQueue options is false";
489
+ }
459
490
  function isCommandTimeout(err) {
460
491
  return err instanceof Error && err.message === "Command timed out";
461
492
  }
@@ -463,7 +494,8 @@ async function withCommandTimeoutCode(operation, commandTimeoutMs, work) {
463
494
  try {
464
495
  return await work;
465
496
  } catch (err) {
466
- throw isCommandTimeout(err) ? timeoutException(operation, commandTimeoutMs(), err) : err;
497
+ if (isCommandTimeout(err)) throw timeoutException(operation, commandTimeoutMs(), err);
498
+ throw isOfflineRejection(err) ? unavailableException(operation, err) : err;
467
499
  }
468
500
  }
469
501
 
@@ -479,13 +511,14 @@ function assertValidCron(repeat, queueOptions) {
479
511
  });
480
512
  }
481
513
  }
482
- var _queues;
514
+ var _queues, _warm;
483
515
  exports.QueueService = class QueueService {
484
516
  constructor(connection, options) {
485
517
  this.connection = connection;
486
518
  this.options = options;
487
519
  __publicField(this, "logger", new common.Logger(exports.QueueService.name));
488
520
  __privateAdd(this, _queues, /* @__PURE__ */ new Map());
521
+ __privateAdd(this, _warm, /* @__PURE__ */ new Set());
489
522
  }
490
523
  connection;
491
524
  options;
@@ -504,27 +537,38 @@ exports.QueueService = class QueueService {
504
537
  __privateGet(this, _queues).set(queueName, queue);
505
538
  return queue;
506
539
  }
507
- enqueueTimeout(operation) {
508
- return timeoutException(operation, this.options.enqueueTimeoutMs);
509
- }
510
- call(operation, work) {
540
+ call(operation, queueName, run) {
541
+ let expired = false;
542
+ const guarded = withDeadline(
543
+ this.afterReady(queueName).then(() => {
544
+ if (expired) throw new Error("the call had already timed out");
545
+ return run();
546
+ }),
547
+ this.options.enqueueTimeoutMs,
548
+ () => {
549
+ expired = true;
550
+ return timeoutException(operation, this.options.enqueueTimeoutMs);
551
+ }
552
+ );
511
553
  return withCommandTimeoutCode(
512
554
  operation,
513
555
  () => this.connection.getClient().options.commandTimeout ?? this.options.commandTimeoutMs,
514
- work
556
+ guarded
515
557
  );
516
558
  }
559
+ async afterReady(queueName) {
560
+ if (__privateGet(this, _warm).has(queueName)) return;
561
+ await this.connection.awaitReady();
562
+ __privateGet(this, _warm).add(queueName);
563
+ }
517
564
  queueFor(queueName) {
518
565
  return this.getOrCreateQueue(queueName);
519
566
  }
520
567
  async enqueue(queueName, jobName, data, options) {
521
568
  const job = await this.call(
522
569
  "enqueue",
523
- withDeadline(
524
- this.queueFor(queueName).add(jobName, data, options),
525
- this.options.enqueueTimeoutMs,
526
- () => this.enqueueTimeout("enqueue")
527
- )
570
+ queueName,
571
+ () => this.queueFor(queueName).add(jobName, data, options)
528
572
  );
529
573
  return job;
530
574
  }
@@ -543,16 +587,13 @@ exports.QueueService = class QueueService {
543
587
  try {
544
588
  const created = await this.call(
545
589
  "enqueueBulk",
546
- withDeadline(
547
- this.queueFor(queueName).addBulk(
548
- jobs.map((job) => ({
549
- name: job.name,
550
- data: job.data,
551
- ...job.options === void 0 ? {} : { opts: job.options }
552
- }))
553
- ),
554
- this.options.enqueueTimeoutMs,
555
- () => this.enqueueTimeout("enqueueBulk")
590
+ queueName,
591
+ () => this.queueFor(queueName).addBulk(
592
+ jobs.map((job) => ({
593
+ name: job.name,
594
+ data: job.data,
595
+ ...job.options === void 0 ? {} : { opts: job.options }
596
+ }))
556
597
  )
557
598
  );
558
599
  return created;
@@ -570,7 +611,7 @@ exports.QueueService = class QueueService {
570
611
  }
571
612
  }
572
613
  async getJob(queueName, jobId) {
573
- const job = await this.call("getJob", this.queueFor(queueName).getJob(jobId));
614
+ const job = await this.call("getJob", queueName, () => this.queueFor(queueName).getJob(jobId));
574
615
  return job ?? null;
575
616
  }
576
617
  async getJobs(queueName, status, start = 0, end = 50) {
@@ -579,13 +620,18 @@ exports.QueueService = class QueueService {
579
620
  reason: `getJobs needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
580
621
  });
581
622
  }
582
- const jobs = await this.call("getJobs", this.queueFor(queueName).getJobs([status], start, end));
623
+ const jobs = await this.call(
624
+ "getJobs",
625
+ queueName,
626
+ () => this.queueFor(queueName).getJobs([status], start, end)
627
+ );
583
628
  return jobs;
584
629
  }
585
630
  async getMetrics(queueName) {
586
631
  const counts = await this.call(
587
632
  "getMetrics",
588
- this.queueFor(queueName).getJobCounts("waiting", "active", "completed", "failed", "delayed")
633
+ queueName,
634
+ () => this.queueFor(queueName).getJobCounts("waiting", "active", "completed", "failed", "delayed")
589
635
  );
590
636
  return {
591
637
  queue: queueName,
@@ -604,24 +650,37 @@ exports.QueueService = class QueueService {
604
650
  if ("pattern" in repeat) assertValidCron(repeat, this.options.queueOptions);
605
651
  const job = await this.call(
606
652
  "upsertJobScheduler",
607
- queue.upsertJobScheduler(schedulerId, repeat, jobTemplate)
653
+ queueName,
654
+ () => queue.upsertJobScheduler(schedulerId, repeat, jobTemplate)
608
655
  );
609
656
  return job;
610
657
  }
611
658
  async removeJobScheduler(queueName, schedulerId) {
612
- return this.call("removeJobScheduler", this.queueFor(queueName).removeJobScheduler(schedulerId));
659
+ return this.call(
660
+ "removeJobScheduler",
661
+ queueName,
662
+ () => this.queueFor(queueName).removeJobScheduler(schedulerId)
663
+ );
613
664
  }
614
665
  async getJobSchedulers(queueName, start = 0, end = 50, asc = true) {
615
- return this.call("getJobSchedulers", this.queueFor(queueName).getJobSchedulers(start, end, asc));
666
+ return this.call(
667
+ "getJobSchedulers",
668
+ queueName,
669
+ () => this.queueFor(queueName).getJobSchedulers(start, end, asc)
670
+ );
616
671
  }
617
672
  async pauseQueue(queueName) {
618
- await this.call("pauseQueue", this.queueFor(queueName).pause());
673
+ await this.call("pauseQueue", queueName, () => this.queueFor(queueName).pause());
619
674
  }
620
675
  async resumeQueue(queueName) {
621
- await this.call("resumeQueue", this.queueFor(queueName).resume());
676
+ await this.call("resumeQueue", queueName, () => this.queueFor(queueName).resume());
622
677
  }
623
678
  async cleanQueue(queueName, gracePeriodMs, limit, status) {
624
- return this.call("cleanQueue", this.queueFor(queueName).clean(gracePeriodMs, limit, status));
679
+ return this.call(
680
+ "cleanQueue",
681
+ queueName,
682
+ () => this.queueFor(queueName).clean(gracePeriodMs, limit, status)
683
+ );
625
684
  }
626
685
  getCachedQueues() {
627
686
  return __privateGet(this, _queues);
@@ -631,9 +690,11 @@ exports.QueueService = class QueueService {
631
690
  await queue.close().catch(() => void 0);
632
691
  }
633
692
  __privateGet(this, _queues).clear();
693
+ __privateGet(this, _warm).clear();
634
694
  }
635
695
  };
636
696
  _queues = new WeakMap();
697
+ _warm = new WeakMap();
637
698
  exports.QueueService = __decorateClass([
638
699
  common.Injectable(),
639
700
  __decorateParam(0, common.Inject(exports.ConnectionResolver)),
@@ -736,6 +797,17 @@ exports.MetricsService = class MetricsService {
736
797
  exports.MetricsService = __decorateClass([
737
798
  common.Injectable()
738
799
  ], exports.MetricsService);
800
+
801
+ // src/server/utils/abandoned-workers.ts
802
+ var abandoned = /* @__PURE__ */ new WeakSet();
803
+ function abandonWorker(worker) {
804
+ abandoned.add(worker);
805
+ }
806
+ function isAbandoned(worker) {
807
+ return abandoned.has(worker);
808
+ }
809
+
810
+ // src/server/services/worker-registry.service.ts
739
811
  var _entries;
740
812
  exports.WorkerRegistry = class WorkerRegistry {
741
813
  constructor(connection, options) {
@@ -756,7 +828,7 @@ exports.WorkerRegistry = class WorkerRegistry {
756
828
  try {
757
829
  worker = new bullmq.Worker(
758
830
  queueName,
759
- toProcessor(handler, options?.jobTimeoutMs),
831
+ toProcessor(handler, options?.jobTimeoutMs, () => isAbandoned(worker)),
760
832
  bullOpts
761
833
  );
762
834
  } catch (cause) {
@@ -970,7 +1042,7 @@ var JobTimeoutError = class extends Error {
970
1042
  this.name = "JobTimeoutError";
971
1043
  }
972
1044
  };
973
- function toProcessor(handler, timeoutMs) {
1045
+ function adaptHandler(handler, timeoutMs) {
974
1046
  if (timeoutMs === void 0) return (job, _token, signal) => handler(job, signal);
975
1047
  return (job, _token, signal) => {
976
1048
  const own = new AbortController();
@@ -992,6 +1064,16 @@ function toProcessor(handler, timeoutMs) {
992
1064
  });
993
1065
  };
994
1066
  }
1067
+ function toProcessor(handler, timeoutMs, abandoned2) {
1068
+ const run = adaptHandler(handler, timeoutMs);
1069
+ return (job, token, signal) => run(job, token, signal).then(
1070
+ (value) => abandoned2() ? new Promise(() => void 0) : value,
1071
+ (error) => {
1072
+ if (abandoned2()) return new Promise(() => void 0);
1073
+ throw error;
1074
+ }
1075
+ );
1076
+ }
995
1077
  var _events, _connections;
996
1078
  exports.QueueEventsRegistry = class QueueEventsRegistry {
997
1079
  constructor(connection, options) {
@@ -1170,6 +1252,7 @@ ProcessorDiscoveryService = __decorateClass([
1170
1252
  __decorateParam(1, common.Inject(exports.WorkerRegistry)),
1171
1253
  __decorateParam(2, common.Inject(exports.QueueEventsRegistry))
1172
1254
  ], ProcessorDiscoveryService);
1255
+ var FORCE_CLOSE_GRACE_MS = 1e3;
1173
1256
  var DrainTimeoutError = class extends Error {
1174
1257
  };
1175
1258
  var QueueLifecycle = class {
@@ -1203,25 +1286,33 @@ var QueueLifecycle = class {
1203
1286
  await this.closeQueues();
1204
1287
  await this.teardownConnection();
1205
1288
  this.logger.log(
1206
- `Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (forced ${String(this.forcedWorkers)} worker(s))`
1289
+ `Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (${String(this.forcedWorkers)} worker(s) stopped without draining)`
1207
1290
  );
1208
1291
  }
1209
1292
  async closeWorkers(deadline) {
1210
- let forced = 0;
1293
+ let undrained = 0;
1211
1294
  await Promise.all(
1212
1295
  Array.from(this.workers.getAll(), async ([name, worker]) => {
1213
1296
  try {
1214
- await bounded(worker.close(), deadline);
1297
+ await bounded(
1298
+ worker.pause().then(() => worker.close()),
1299
+ deadline
1300
+ );
1215
1301
  } catch (cause) {
1216
- forced++;
1302
+ undrained++;
1303
+ abandonWorker(worker);
1304
+ const stopped = "it is stopped without waiting for its in-flight job, which becomes stalled and is retried";
1217
1305
  this.logger.warn(
1218
- cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget \u2014 forcing close (${QUEUE_ERROR_CODES.SHUTDOWN_TIMEOUT_EXCEEDED}); the in-flight job becomes stalled and is retried` : `Worker "${name}" failed to close gracefully (${describeError(cause)}) \u2014 forcing close; the in-flight job becomes stalled and is retried`
1306
+ cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget (${QUEUE_ERROR_CODES.SHUTDOWN_TIMEOUT_EXCEEDED}); ${stopped}` : `Worker "${name}" failed to close gracefully (${describeError(cause)}); ${stopped}`
1307
+ );
1308
+ const deadlineError = () => new DrainTimeoutError();
1309
+ await withDeadline(worker.close(true), FORCE_CLOSE_GRACE_MS, deadlineError).catch(
1310
+ () => void 0
1219
1311
  );
1220
- void worker.close(true).catch(() => void 0);
1221
1312
  }
1222
1313
  })
1223
1314
  );
1224
- return forced;
1315
+ return undrained;
1225
1316
  }
1226
1317
  async closeQueueEvents(deadline) {
1227
1318
  await Promise.all(
@@ -105,9 +105,11 @@ interface BymaxQueueModuleOptions {
105
105
  */
106
106
  commandTimeoutMs?: number;
107
107
  /**
108
- * The deadline, in milliseconds, for one `enqueue` or `enqueueBulk` call in either connection mode.
109
- * When it passes, the call rejects with `CONNECTION_TIMEOUT`; the add may still complete in Redis
110
- * afterwards, so retry with a stable `jobId` or `deduplication`. Default: 10_000.
108
+ * The deadline, in milliseconds, for one `QueueService` call that reaches Redis (`enqueue`,
109
+ * `getJob`, `getMetrics`, the scheduler, pause and clean calls, and the rest) in either connection
110
+ * mode, including the wait for the client to be ready on a queue's first call. When it passes, the
111
+ * call rejects with `CONNECTION_TIMEOUT`; an `enqueue` may still complete in Redis afterwards, so
112
+ * retry with a stable `jobId` or `deduplication`. Default: 10_000.
111
113
  */
112
114
  enqueueTimeoutMs?: number;
113
115
  }
@@ -401,6 +403,14 @@ declare class ConnectionResolver {
401
403
  getMode(): QueueConnectionMode;
402
404
  /** Whether the library owns (and must close) the connection. */
403
405
  isOwned(): boolean;
406
+ /**
407
+ * Settle when the client can take a command. While it is still connecting or reconnecting, every
408
+ * caller shares one wait on its `ready` event, so a health check that polls through a long outage
409
+ * adds one listener set to the client, not one per call, and gives it back when Redis returns. A
410
+ * lazy client (`wait`) connects on its first command and a finished one (`end`) never becomes
411
+ * ready, so both are left for the call itself.
412
+ */
413
+ awaitReady(): Promise<void>;
404
414
  /** Close the library-owned connection on shutdown; never touch a BYO client. */
405
415
  teardown(): Promise<void>;
406
416
  /** Validate and adopt a bring-your-own client for the Queue role. */
@@ -446,20 +456,27 @@ declare class QueueService {
446
456
  * @param overrides - Optional per-queue option overrides.
447
457
  */
448
458
  getOrCreateQueue<TData = unknown, TResult = unknown>(queueName: string, overrides?: Partial<Omit<QueueOptions, 'connection' | 'prefix'>>): Queue<TData, TResult>;
449
- /** The error a producer call receives when Redis does not answer within `enqueueTimeoutMs`. */
450
- private enqueueTimeout;
451
459
  /**
452
- * Await a Redis call, reporting an ioredis command timeout as `CONNECTION_TIMEOUT` with the
453
- * operation and the command deadline of the connection in force.
460
+ * Run one Redis call under the per-call deadline (`enqueueTimeoutMs`), reporting every way Redis
461
+ * can fail to answer as the documented `CONNECTION_TIMEOUT`: the deadline passing, an ioredis
462
+ * command timeout, or a call refused because the connection is down.
463
+ *
464
+ * The first call on a queue name waits for the client to be ready through ONE wait shared by all
465
+ * callers, instead of letting BullMQ start its own unbounded wait per call, so a caller that
466
+ * polls through an outage does not pile up listeners. The call runs only if it is still within
467
+ * its deadline when the client is ready; a caller that was told it timed out never has its call
468
+ * run later, which would turn its retry into a duplicate.
454
469
  */
455
470
  private call;
471
+ /** Settle when a call on `queueName` may start: at once for a queue that has been used before. */
472
+ private afterReady;
456
473
  /** Resolve the raw cached queue (default generics) for internal delegation. */
457
474
  private queueFor;
458
475
  /**
459
476
  * Add a single job to a queue. The queue is created lazily if needed.
460
477
  *
461
478
  * Delivery is **at-least-once** — a handler may run more than once for the same
462
- * job (worker crash, lock expiry, or a shutdown force-close). Make handlers
479
+ * job (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Make handlers
463
480
  * idempotent: use an idempotency key on writes, prefer upserts over inserts, or
464
481
  * keep an "already-processed" marker keyed by `job.id`.
465
482
  *
@@ -1087,9 +1104,9 @@ declare class QueueEventsRegistry {
1087
1104
  * HTTP server has stopped, it releases what a request can still be using: the
1088
1105
  * flow producer, the cached queues, and finally the Queue-role connection.
1089
1106
  *
1090
- * Delivery is at-least-once: when a worker exceeds the drain budget it is
1091
- * force-closed and its in-flight job becomes `stalled` and is retried by another
1092
- * worker. Handlers must therefore be idempotent, and `lockDuration` should
1107
+ * Delivery is at-least-once: when a worker exceeds the drain budget it is stopped
1108
+ * without waiting for its job, whose lock then expires so the job becomes `stalled`
1109
+ * and is retried by another worker. Handlers must therefore be idempotent, and `lockDuration` should
1093
1110
  * comfortably exceed the worst-case handler runtime so healthy long jobs are
1094
1111
  * never treated as stalled.
1095
1112
  * @layer server/lifecycle
@@ -1138,11 +1155,16 @@ declare class QueueLifecycle implements OnModuleDestroy, OnApplicationShutdown {
1138
1155
  */
1139
1156
  onApplicationShutdown(): Promise<void>;
1140
1157
  /**
1141
- * Close every worker in parallel, each bounded by what is left of the shutdown budget, and
1142
- * force-close any that exceed it. Returns the number of workers that had to be force-closed.
1158
+ * Stop every worker in parallel, each bounded by what is left of the shutdown budget. A worker
1159
+ * is paused first (it stops fetching and waits for its active jobs) and then closed. One that is
1160
+ * still busy when the budget ends is closed with `close(true)`: because `pause` does not start a
1161
+ * close, that is the worker's FIRST close, so it does not wait for the job and stops the lock
1162
+ * renewal that would otherwise keep running and keep the process alive. The job's lock then
1163
+ * expires, and it stalls and is retried elsewhere. Returns how many workers were stopped
1164
+ * without draining.
1143
1165
  *
1144
1166
  * @param deadline - The epoch millisecond at which the shutdown budget ends.
1145
- * @returns The count of force-closed workers.
1167
+ * @returns The count of workers stopped without draining.
1146
1168
  */
1147
1169
  private closeWorkers;
1148
1170
  /** Close every cached `QueueEvents` in parallel within the budget, swallowing failures. */
@@ -1221,7 +1243,7 @@ declare function Processor(queueName: string, options?: WorkerOptions): ClassDec
1221
1243
  * an entry to the accumulated metadata array.
1222
1244
  *
1223
1245
  * Delivery is at-least-once: a handler may run more than once for the same job
1224
- * (worker crash, lock expiry, or a shutdown force-close). Handlers MUST be
1246
+ * (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Handlers MUST be
1225
1247
  * idempotent — use an idempotency key on writes, prefer upserts over inserts, or
1226
1248
  * keep an "already-processed" marker keyed by `job.id`.
1227
1249
  *
@@ -1273,7 +1295,7 @@ type WorkerEventName = 'completed' | 'failed' | 'progress' | 'active' | 'stalled
1273
1295
  * overwriting each other.
1274
1296
  *
1275
1297
  * Delivery is at-least-once. A `'stalled'` job — for example after a shutdown
1276
- * force-close or a lock expiry — is re-run by another worker: this is the
1298
+ * that outlasts the drain budget, or a lock expiry — is re-run by another worker: this is the
1277
1299
  * visible at-least-once path, so handlers must be idempotent. Set `lockDuration`
1278
1300
  * comfortably above the worst-case handler runtime to avoid false stalls.
1279
1301
  *
@@ -1490,7 +1512,7 @@ declare const DEFAULT_JOB_OPTIONS: {
1490
1512
  };
1491
1513
  /** Default TTL (ms) for the metrics cache. */
1492
1514
  declare const DEFAULT_METRICS_CACHE_TTL_MS: 5000;
1493
- /** Default window (ms) for in-flight jobs to drain before force-close on shutdown. */
1515
+ /** Default window (ms) for in-flight jobs to drain before the worker is stopped on shutdown. */
1494
1516
  declare const DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS: 30000;
1495
1517
 
1496
1518
  /**
@@ -105,9 +105,11 @@ interface BymaxQueueModuleOptions {
105
105
  */
106
106
  commandTimeoutMs?: number;
107
107
  /**
108
- * The deadline, in milliseconds, for one `enqueue` or `enqueueBulk` call in either connection mode.
109
- * When it passes, the call rejects with `CONNECTION_TIMEOUT`; the add may still complete in Redis
110
- * afterwards, so retry with a stable `jobId` or `deduplication`. Default: 10_000.
108
+ * The deadline, in milliseconds, for one `QueueService` call that reaches Redis (`enqueue`,
109
+ * `getJob`, `getMetrics`, the scheduler, pause and clean calls, and the rest) in either connection
110
+ * mode, including the wait for the client to be ready on a queue's first call. When it passes, the
111
+ * call rejects with `CONNECTION_TIMEOUT`; an `enqueue` may still complete in Redis afterwards, so
112
+ * retry with a stable `jobId` or `deduplication`. Default: 10_000.
111
113
  */
112
114
  enqueueTimeoutMs?: number;
113
115
  }
@@ -401,6 +403,14 @@ declare class ConnectionResolver {
401
403
  getMode(): QueueConnectionMode;
402
404
  /** Whether the library owns (and must close) the connection. */
403
405
  isOwned(): boolean;
406
+ /**
407
+ * Settle when the client can take a command. While it is still connecting or reconnecting, every
408
+ * caller shares one wait on its `ready` event, so a health check that polls through a long outage
409
+ * adds one listener set to the client, not one per call, and gives it back when Redis returns. A
410
+ * lazy client (`wait`) connects on its first command and a finished one (`end`) never becomes
411
+ * ready, so both are left for the call itself.
412
+ */
413
+ awaitReady(): Promise<void>;
404
414
  /** Close the library-owned connection on shutdown; never touch a BYO client. */
405
415
  teardown(): Promise<void>;
406
416
  /** Validate and adopt a bring-your-own client for the Queue role. */
@@ -446,20 +456,27 @@ declare class QueueService {
446
456
  * @param overrides - Optional per-queue option overrides.
447
457
  */
448
458
  getOrCreateQueue<TData = unknown, TResult = unknown>(queueName: string, overrides?: Partial<Omit<QueueOptions, 'connection' | 'prefix'>>): Queue<TData, TResult>;
449
- /** The error a producer call receives when Redis does not answer within `enqueueTimeoutMs`. */
450
- private enqueueTimeout;
451
459
  /**
452
- * Await a Redis call, reporting an ioredis command timeout as `CONNECTION_TIMEOUT` with the
453
- * operation and the command deadline of the connection in force.
460
+ * Run one Redis call under the per-call deadline (`enqueueTimeoutMs`), reporting every way Redis
461
+ * can fail to answer as the documented `CONNECTION_TIMEOUT`: the deadline passing, an ioredis
462
+ * command timeout, or a call refused because the connection is down.
463
+ *
464
+ * The first call on a queue name waits for the client to be ready through ONE wait shared by all
465
+ * callers, instead of letting BullMQ start its own unbounded wait per call, so a caller that
466
+ * polls through an outage does not pile up listeners. The call runs only if it is still within
467
+ * its deadline when the client is ready; a caller that was told it timed out never has its call
468
+ * run later, which would turn its retry into a duplicate.
454
469
  */
455
470
  private call;
471
+ /** Settle when a call on `queueName` may start: at once for a queue that has been used before. */
472
+ private afterReady;
456
473
  /** Resolve the raw cached queue (default generics) for internal delegation. */
457
474
  private queueFor;
458
475
  /**
459
476
  * Add a single job to a queue. The queue is created lazily if needed.
460
477
  *
461
478
  * Delivery is **at-least-once** — a handler may run more than once for the same
462
- * job (worker crash, lock expiry, or a shutdown force-close). Make handlers
479
+ * job (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Make handlers
463
480
  * idempotent: use an idempotency key on writes, prefer upserts over inserts, or
464
481
  * keep an "already-processed" marker keyed by `job.id`.
465
482
  *
@@ -1087,9 +1104,9 @@ declare class QueueEventsRegistry {
1087
1104
  * HTTP server has stopped, it releases what a request can still be using: the
1088
1105
  * flow producer, the cached queues, and finally the Queue-role connection.
1089
1106
  *
1090
- * Delivery is at-least-once: when a worker exceeds the drain budget it is
1091
- * force-closed and its in-flight job becomes `stalled` and is retried by another
1092
- * worker. Handlers must therefore be idempotent, and `lockDuration` should
1107
+ * Delivery is at-least-once: when a worker exceeds the drain budget it is stopped
1108
+ * without waiting for its job, whose lock then expires so the job becomes `stalled`
1109
+ * and is retried by another worker. Handlers must therefore be idempotent, and `lockDuration` should
1093
1110
  * comfortably exceed the worst-case handler runtime so healthy long jobs are
1094
1111
  * never treated as stalled.
1095
1112
  * @layer server/lifecycle
@@ -1138,11 +1155,16 @@ declare class QueueLifecycle implements OnModuleDestroy, OnApplicationShutdown {
1138
1155
  */
1139
1156
  onApplicationShutdown(): Promise<void>;
1140
1157
  /**
1141
- * Close every worker in parallel, each bounded by what is left of the shutdown budget, and
1142
- * force-close any that exceed it. Returns the number of workers that had to be force-closed.
1158
+ * Stop every worker in parallel, each bounded by what is left of the shutdown budget. A worker
1159
+ * is paused first (it stops fetching and waits for its active jobs) and then closed. One that is
1160
+ * still busy when the budget ends is closed with `close(true)`: because `pause` does not start a
1161
+ * close, that is the worker's FIRST close, so it does not wait for the job and stops the lock
1162
+ * renewal that would otherwise keep running and keep the process alive. The job's lock then
1163
+ * expires, and it stalls and is retried elsewhere. Returns how many workers were stopped
1164
+ * without draining.
1143
1165
  *
1144
1166
  * @param deadline - The epoch millisecond at which the shutdown budget ends.
1145
- * @returns The count of force-closed workers.
1167
+ * @returns The count of workers stopped without draining.
1146
1168
  */
1147
1169
  private closeWorkers;
1148
1170
  /** Close every cached `QueueEvents` in parallel within the budget, swallowing failures. */
@@ -1221,7 +1243,7 @@ declare function Processor(queueName: string, options?: WorkerOptions): ClassDec
1221
1243
  * an entry to the accumulated metadata array.
1222
1244
  *
1223
1245
  * Delivery is at-least-once: a handler may run more than once for the same job
1224
- * (worker crash, lock expiry, or a shutdown force-close). Handlers MUST be
1246
+ * (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Handlers MUST be
1225
1247
  * idempotent — use an idempotency key on writes, prefer upserts over inserts, or
1226
1248
  * keep an "already-processed" marker keyed by `job.id`.
1227
1249
  *
@@ -1273,7 +1295,7 @@ type WorkerEventName = 'completed' | 'failed' | 'progress' | 'active' | 'stalled
1273
1295
  * overwriting each other.
1274
1296
  *
1275
1297
  * Delivery is at-least-once. A `'stalled'` job — for example after a shutdown
1276
- * force-close or a lock expiry — is re-run by another worker: this is the
1298
+ * that outlasts the drain budget, or a lock expiry — is re-run by another worker: this is the
1277
1299
  * visible at-least-once path, so handlers must be idempotent. Set `lockDuration`
1278
1300
  * comfortably above the worst-case handler runtime to avoid false stalls.
1279
1301
  *
@@ -1490,7 +1512,7 @@ declare const DEFAULT_JOB_OPTIONS: {
1490
1512
  };
1491
1513
  /** Default TTL (ms) for the metrics cache. */
1492
1514
  declare const DEFAULT_METRICS_CACHE_TTL_MS: 5000;
1493
- /** Default window (ms) for in-flight jobs to drain before force-close on shutdown. */
1515
+ /** Default window (ms) for in-flight jobs to drain before the worker is stopped on shutdown. */
1494
1516
  declare const DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS: 30000;
1495
1517
 
1496
1518
  /**
@@ -248,13 +248,14 @@ function duplicateConnection(client) {
248
248
  }
249
249
 
250
250
  // src/server/services/connection-resolver.service.ts
251
- var _logger, _client, _options;
251
+ var _logger, _client, _options, _pendingReady;
252
252
  var ConnectionResolver = class {
253
253
  constructor(options) {
254
254
  __privateAdd(this, _logger, new Logger(ConnectionResolver.name));
255
255
  __privateAdd(this, _client);
256
256
  __privateAdd(this, _options);
257
257
  __publicField(this, "mode");
258
+ __privateAdd(this, _pendingReady);
258
259
  __privateSet(this, _options, options);
259
260
  }
260
261
  async init() {
@@ -310,6 +311,17 @@ var ConnectionResolver = class {
310
311
  isOwned() {
311
312
  return this.mode === "mode-b-owned";
312
313
  }
314
+ awaitReady() {
315
+ const client = this.getClient();
316
+ if (!WAITING_STATUSES.has(client.status)) return Promise.resolve();
317
+ __privateGet(this, _pendingReady) ?? __privateSet(this, _pendingReady, new Promise((resolve) => {
318
+ client.once("ready", () => {
319
+ __privateSet(this, _pendingReady, void 0);
320
+ resolve();
321
+ });
322
+ }));
323
+ return __privateGet(this, _pendingReady);
324
+ }
313
325
  async teardown() {
314
326
  if (this.isOwned() && __privateGet(this, _client)) {
315
327
  const client = __privateGet(this, _client);
@@ -330,7 +342,7 @@ var ConnectionResolver = class {
330
342
  }
331
343
  if (client.options.maxRetriesPerRequest === null) {
332
344
  __privateGet(this, _logger).warn(
333
- "The shared client has maxRetriesPerRequest: null, so a producer call made while Redis is unreachable waits for as long as the outage lasts; enqueue is bounded by enqueueTimeoutMs, other QueueService calls are not. Give the client a finite limit to fail fast."
345
+ "The shared client has maxRetriesPerRequest: null, so a command issued while Redis is unreachable stays queued in the client for as long as the outage lasts. QueueService calls are bounded by enqueueTimeoutMs and the caller is released, but the command is not cancelled. Give the client a finite limit to fail fast."
334
346
  );
335
347
  }
336
348
  const probe = duplicateConnection(client);
@@ -380,10 +392,17 @@ var ConnectionResolver = class {
380
392
  _logger = new WeakMap();
381
393
  _client = new WeakMap();
382
394
  _options = new WeakMap();
395
+ _pendingReady = new WeakMap();
383
396
  ConnectionResolver = __decorateClass([
384
397
  Injectable(),
385
398
  __decorateParam(0, Inject(BYMAX_QUEUE_OPTIONS))
386
399
  ], ConnectionResolver);
400
+ var WAITING_STATUSES = /* @__PURE__ */ new Set([
401
+ "connecting",
402
+ "connect",
403
+ "reconnecting",
404
+ "close"
405
+ ]);
387
406
  function isReplyError(err) {
388
407
  return typeof err === "object" && err !== null && err.name === "ReplyError";
389
408
  }
@@ -454,6 +473,18 @@ function timeoutException(operation, timeoutMs, cause) {
454
473
  if (cause !== void 0) exception.cause = cause;
455
474
  return exception;
456
475
  }
476
+ function unavailableException(operation, cause) {
477
+ const exception = new QueueException(
478
+ QUEUE_ERROR_CODES.CONNECTION_TIMEOUT,
479
+ HttpStatus.INTERNAL_SERVER_ERROR,
480
+ { operation, reason: "offline" }
481
+ );
482
+ exception.cause = cause;
483
+ return exception;
484
+ }
485
+ function isOfflineRejection(err) {
486
+ return err instanceof Error && err.message === "Stream isn't writeable and enableOfflineQueue options is false";
487
+ }
457
488
  function isCommandTimeout(err) {
458
489
  return err instanceof Error && err.message === "Command timed out";
459
490
  }
@@ -461,7 +492,8 @@ async function withCommandTimeoutCode(operation, commandTimeoutMs, work) {
461
492
  try {
462
493
  return await work;
463
494
  } catch (err) {
464
- throw isCommandTimeout(err) ? timeoutException(operation, commandTimeoutMs(), err) : err;
495
+ if (isCommandTimeout(err)) throw timeoutException(operation, commandTimeoutMs(), err);
496
+ throw isOfflineRejection(err) ? unavailableException(operation, err) : err;
465
497
  }
466
498
  }
467
499
 
@@ -477,13 +509,14 @@ function assertValidCron(repeat, queueOptions) {
477
509
  });
478
510
  }
479
511
  }
480
- var _queues;
512
+ var _queues, _warm;
481
513
  var QueueService = class {
482
514
  constructor(connection, options) {
483
515
  this.connection = connection;
484
516
  this.options = options;
485
517
  __publicField(this, "logger", new Logger(QueueService.name));
486
518
  __privateAdd(this, _queues, /* @__PURE__ */ new Map());
519
+ __privateAdd(this, _warm, /* @__PURE__ */ new Set());
487
520
  }
488
521
  connection;
489
522
  options;
@@ -502,27 +535,38 @@ var QueueService = class {
502
535
  __privateGet(this, _queues).set(queueName, queue);
503
536
  return queue;
504
537
  }
505
- enqueueTimeout(operation) {
506
- return timeoutException(operation, this.options.enqueueTimeoutMs);
507
- }
508
- call(operation, work) {
538
+ call(operation, queueName, run) {
539
+ let expired = false;
540
+ const guarded = withDeadline(
541
+ this.afterReady(queueName).then(() => {
542
+ if (expired) throw new Error("the call had already timed out");
543
+ return run();
544
+ }),
545
+ this.options.enqueueTimeoutMs,
546
+ () => {
547
+ expired = true;
548
+ return timeoutException(operation, this.options.enqueueTimeoutMs);
549
+ }
550
+ );
509
551
  return withCommandTimeoutCode(
510
552
  operation,
511
553
  () => this.connection.getClient().options.commandTimeout ?? this.options.commandTimeoutMs,
512
- work
554
+ guarded
513
555
  );
514
556
  }
557
+ async afterReady(queueName) {
558
+ if (__privateGet(this, _warm).has(queueName)) return;
559
+ await this.connection.awaitReady();
560
+ __privateGet(this, _warm).add(queueName);
561
+ }
515
562
  queueFor(queueName) {
516
563
  return this.getOrCreateQueue(queueName);
517
564
  }
518
565
  async enqueue(queueName, jobName, data, options) {
519
566
  const job = await this.call(
520
567
  "enqueue",
521
- withDeadline(
522
- this.queueFor(queueName).add(jobName, data, options),
523
- this.options.enqueueTimeoutMs,
524
- () => this.enqueueTimeout("enqueue")
525
- )
568
+ queueName,
569
+ () => this.queueFor(queueName).add(jobName, data, options)
526
570
  );
527
571
  return job;
528
572
  }
@@ -541,16 +585,13 @@ var QueueService = class {
541
585
  try {
542
586
  const created = await this.call(
543
587
  "enqueueBulk",
544
- withDeadline(
545
- this.queueFor(queueName).addBulk(
546
- jobs.map((job) => ({
547
- name: job.name,
548
- data: job.data,
549
- ...job.options === void 0 ? {} : { opts: job.options }
550
- }))
551
- ),
552
- this.options.enqueueTimeoutMs,
553
- () => this.enqueueTimeout("enqueueBulk")
588
+ queueName,
589
+ () => this.queueFor(queueName).addBulk(
590
+ jobs.map((job) => ({
591
+ name: job.name,
592
+ data: job.data,
593
+ ...job.options === void 0 ? {} : { opts: job.options }
594
+ }))
554
595
  )
555
596
  );
556
597
  return created;
@@ -568,7 +609,7 @@ var QueueService = class {
568
609
  }
569
610
  }
570
611
  async getJob(queueName, jobId) {
571
- const job = await this.call("getJob", this.queueFor(queueName).getJob(jobId));
612
+ const job = await this.call("getJob", queueName, () => this.queueFor(queueName).getJob(jobId));
572
613
  return job ?? null;
573
614
  }
574
615
  async getJobs(queueName, status, start = 0, end = 50) {
@@ -577,13 +618,18 @@ var QueueService = class {
577
618
  reason: `getJobs needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
578
619
  });
579
620
  }
580
- const jobs = await this.call("getJobs", this.queueFor(queueName).getJobs([status], start, end));
621
+ const jobs = await this.call(
622
+ "getJobs",
623
+ queueName,
624
+ () => this.queueFor(queueName).getJobs([status], start, end)
625
+ );
581
626
  return jobs;
582
627
  }
583
628
  async getMetrics(queueName) {
584
629
  const counts = await this.call(
585
630
  "getMetrics",
586
- this.queueFor(queueName).getJobCounts("waiting", "active", "completed", "failed", "delayed")
631
+ queueName,
632
+ () => this.queueFor(queueName).getJobCounts("waiting", "active", "completed", "failed", "delayed")
587
633
  );
588
634
  return {
589
635
  queue: queueName,
@@ -602,24 +648,37 @@ var QueueService = class {
602
648
  if ("pattern" in repeat) assertValidCron(repeat, this.options.queueOptions);
603
649
  const job = await this.call(
604
650
  "upsertJobScheduler",
605
- queue.upsertJobScheduler(schedulerId, repeat, jobTemplate)
651
+ queueName,
652
+ () => queue.upsertJobScheduler(schedulerId, repeat, jobTemplate)
606
653
  );
607
654
  return job;
608
655
  }
609
656
  async removeJobScheduler(queueName, schedulerId) {
610
- return this.call("removeJobScheduler", this.queueFor(queueName).removeJobScheduler(schedulerId));
657
+ return this.call(
658
+ "removeJobScheduler",
659
+ queueName,
660
+ () => this.queueFor(queueName).removeJobScheduler(schedulerId)
661
+ );
611
662
  }
612
663
  async getJobSchedulers(queueName, start = 0, end = 50, asc = true) {
613
- return this.call("getJobSchedulers", this.queueFor(queueName).getJobSchedulers(start, end, asc));
664
+ return this.call(
665
+ "getJobSchedulers",
666
+ queueName,
667
+ () => this.queueFor(queueName).getJobSchedulers(start, end, asc)
668
+ );
614
669
  }
615
670
  async pauseQueue(queueName) {
616
- await this.call("pauseQueue", this.queueFor(queueName).pause());
671
+ await this.call("pauseQueue", queueName, () => this.queueFor(queueName).pause());
617
672
  }
618
673
  async resumeQueue(queueName) {
619
- await this.call("resumeQueue", this.queueFor(queueName).resume());
674
+ await this.call("resumeQueue", queueName, () => this.queueFor(queueName).resume());
620
675
  }
621
676
  async cleanQueue(queueName, gracePeriodMs, limit, status) {
622
- return this.call("cleanQueue", this.queueFor(queueName).clean(gracePeriodMs, limit, status));
677
+ return this.call(
678
+ "cleanQueue",
679
+ queueName,
680
+ () => this.queueFor(queueName).clean(gracePeriodMs, limit, status)
681
+ );
623
682
  }
624
683
  getCachedQueues() {
625
684
  return __privateGet(this, _queues);
@@ -629,9 +688,11 @@ var QueueService = class {
629
688
  await queue.close().catch(() => void 0);
630
689
  }
631
690
  __privateGet(this, _queues).clear();
691
+ __privateGet(this, _warm).clear();
632
692
  }
633
693
  };
634
694
  _queues = new WeakMap();
695
+ _warm = new WeakMap();
635
696
  QueueService = __decorateClass([
636
697
  Injectable(),
637
698
  __decorateParam(0, Inject(ConnectionResolver)),
@@ -734,6 +795,17 @@ var MetricsService = class {
734
795
  MetricsService = __decorateClass([
735
796
  Injectable()
736
797
  ], MetricsService);
798
+
799
+ // src/server/utils/abandoned-workers.ts
800
+ var abandoned = /* @__PURE__ */ new WeakSet();
801
+ function abandonWorker(worker) {
802
+ abandoned.add(worker);
803
+ }
804
+ function isAbandoned(worker) {
805
+ return abandoned.has(worker);
806
+ }
807
+
808
+ // src/server/services/worker-registry.service.ts
737
809
  var _entries;
738
810
  var WorkerRegistry = class {
739
811
  constructor(connection, options) {
@@ -754,7 +826,7 @@ var WorkerRegistry = class {
754
826
  try {
755
827
  worker = new Worker(
756
828
  queueName,
757
- toProcessor(handler, options?.jobTimeoutMs),
829
+ toProcessor(handler, options?.jobTimeoutMs, () => isAbandoned(worker)),
758
830
  bullOpts
759
831
  );
760
832
  } catch (cause) {
@@ -968,7 +1040,7 @@ var JobTimeoutError = class extends Error {
968
1040
  this.name = "JobTimeoutError";
969
1041
  }
970
1042
  };
971
- function toProcessor(handler, timeoutMs) {
1043
+ function adaptHandler(handler, timeoutMs) {
972
1044
  if (timeoutMs === void 0) return (job, _token, signal) => handler(job, signal);
973
1045
  return (job, _token, signal) => {
974
1046
  const own = new AbortController();
@@ -990,6 +1062,16 @@ function toProcessor(handler, timeoutMs) {
990
1062
  });
991
1063
  };
992
1064
  }
1065
+ function toProcessor(handler, timeoutMs, abandoned2) {
1066
+ const run = adaptHandler(handler, timeoutMs);
1067
+ return (job, token, signal) => run(job, token, signal).then(
1068
+ (value) => abandoned2() ? new Promise(() => void 0) : value,
1069
+ (error) => {
1070
+ if (abandoned2()) return new Promise(() => void 0);
1071
+ throw error;
1072
+ }
1073
+ );
1074
+ }
993
1075
  var _events, _connections;
994
1076
  var QueueEventsRegistry = class {
995
1077
  constructor(connection, options) {
@@ -1168,6 +1250,7 @@ ProcessorDiscoveryService = __decorateClass([
1168
1250
  __decorateParam(1, Inject(WorkerRegistry)),
1169
1251
  __decorateParam(2, Inject(QueueEventsRegistry))
1170
1252
  ], ProcessorDiscoveryService);
1253
+ var FORCE_CLOSE_GRACE_MS = 1e3;
1171
1254
  var DrainTimeoutError = class extends Error {
1172
1255
  };
1173
1256
  var QueueLifecycle = class {
@@ -1201,25 +1284,33 @@ var QueueLifecycle = class {
1201
1284
  await this.closeQueues();
1202
1285
  await this.teardownConnection();
1203
1286
  this.logger.log(
1204
- `Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (forced ${String(this.forcedWorkers)} worker(s))`
1287
+ `Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (${String(this.forcedWorkers)} worker(s) stopped without draining)`
1205
1288
  );
1206
1289
  }
1207
1290
  async closeWorkers(deadline) {
1208
- let forced = 0;
1291
+ let undrained = 0;
1209
1292
  await Promise.all(
1210
1293
  Array.from(this.workers.getAll(), async ([name, worker]) => {
1211
1294
  try {
1212
- await bounded(worker.close(), deadline);
1295
+ await bounded(
1296
+ worker.pause().then(() => worker.close()),
1297
+ deadline
1298
+ );
1213
1299
  } catch (cause) {
1214
- forced++;
1300
+ undrained++;
1301
+ abandonWorker(worker);
1302
+ const stopped = "it is stopped without waiting for its in-flight job, which becomes stalled and is retried";
1215
1303
  this.logger.warn(
1216
- cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget \u2014 forcing close (${QUEUE_ERROR_CODES.SHUTDOWN_TIMEOUT_EXCEEDED}); the in-flight job becomes stalled and is retried` : `Worker "${name}" failed to close gracefully (${describeError(cause)}) \u2014 forcing close; the in-flight job becomes stalled and is retried`
1304
+ cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget (${QUEUE_ERROR_CODES.SHUTDOWN_TIMEOUT_EXCEEDED}); ${stopped}` : `Worker "${name}" failed to close gracefully (${describeError(cause)}); ${stopped}`
1305
+ );
1306
+ const deadlineError = () => new DrainTimeoutError();
1307
+ await withDeadline(worker.close(true), FORCE_CLOSE_GRACE_MS, deadlineError).catch(
1308
+ () => void 0
1217
1309
  );
1218
- void worker.close(true).catch(() => void 0);
1219
1310
  }
1220
1311
  })
1221
1312
  );
1222
- return forced;
1313
+ return undrained;
1223
1314
  }
1224
1315
  async closeQueueEvents(deadline) {
1225
1316
  await Promise.all(
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@bymax-one/nest-queue",
3
- "version": "1.2.14",
3
+ "version": "1.2.15",
4
4
  "description": "NestJS dynamic module wrapping BullMQ — typed jobs, flows, job schedulers, deduplication, OpenTelemetry, graceful shutdown",
5
5
  "author": "Bymax One <support@bymax.one>",
6
6
  "license": "MIT",