@bymax-one/nest-queue 1.2.15 → 1.2.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -1
- package/README.md +39 -35
- package/dist/server/index.cjs +88 -67
- package/dist/server/index.d.cts +18 -6
- package/dist/server/index.d.ts +18 -6
- package/dist/server/index.mjs +88 -67
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -11,6 +11,30 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
11
11
|
|
|
12
12
|
---
|
|
13
13
|
|
|
14
|
+
## [1.2.16] — 2026-11-14
|
|
15
|
+
|
|
16
|
+
Against 1.2.15 the built declarations gain a fifth optional `FlowService` constructor parameter, `enqueueTimeoutMs`; nothing else in the public API changes.
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
- **Timer-valued options that Node cannot honour are refused at startup.** `shutdown.drainTimeoutMs` and `connectionReadyTimeoutMs` accepted `NaN`, fractions and values above 2 147 483 647 ms; Node reads `NaN` and anything above that limit as a 1 ms timer, so the drain or the startup wait ended at once. Both must now be whole numbers from 1 to 2 147 483 647, like `commandTimeoutMs` and `enqueueTimeoutMs`.
|
|
21
|
+
- **Worker options are checked before BullMQ sees them.** `concurrency`, `limiter.max` and `limiter.duration` must be whole numbers (`NaN` passed the range check). `lockDuration` and `stalledInterval` must be whole numbers from 1000 to 2 147 483 647: BullMQ renews a lock at half of `lockDuration`, so under one second the renewal falls inside ordinary event-loop delay and a busy process loses locks it still holds.
|
|
22
|
+
- **`getJobSchedulers` has the same window bound as `getJobs`.** BullMQ reads an end index of `-1` as "every scheduler"; the window is now at most 1000 entries and a malformed one is refused with `INVALID_OPTIONS`.
|
|
23
|
+
- **`FlowService.add` and `addBulk` are bounded.** Each is held to `enqueueTimeoutMs` and rejects with `CONNECTION_TIMEOUT` (a flow that timed out may still have been added), and `addBulk` refuses more than 1000 flows with `BULK_ENQUEUE_FAILED`, as `enqueueBulk` does.
|
|
24
|
+
- **A call waiting for the connection is released when the client ends.** `ConnectionResolver.awaitReady()` kept its callers waiting until their deadline after ioredis had given up reconnecting; they are now released at once and the call reports the real failure.
|
|
25
|
+
|
|
26
|
+
### Documentation
|
|
27
|
+
|
|
28
|
+
- A handler that outlives `jobTimeoutMs` no longer counts against `concurrency`; the README says so.
|
|
29
|
+
|
|
30
|
+
#### Apply to a derived backend
|
|
31
|
+
|
|
32
|
+
- If you pass `shutdown.drainTimeoutMs` or `connectionReadyTimeoutMs` as a fraction or above 2 147 483 647, or a worker `lockDuration` or `stalledInterval` below 1000, startup or registration now throws; use a whole number in range.
|
|
33
|
+
- If you call `getJobSchedulers` with an open or oversized window, request at most 1000 entries per call.
|
|
34
|
+
- If you construct `FlowService` yourself, pass `enqueueTimeoutMs` as the fifth argument; it defaults to 10 000.
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
14
38
|
## [1.2.15] — 2026-11-14
|
|
15
39
|
|
|
16
40
|
Against 1.2.14 the built declarations gain `ConnectionResolver.awaitReady()`; nothing else in the public API changes.
|
|
@@ -1036,7 +1060,8 @@ v6 peer range.
|
|
|
1036
1060
|
|
|
1037
1061
|
---
|
|
1038
1062
|
|
|
1039
|
-
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.
|
|
1063
|
+
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.16...HEAD
|
|
1064
|
+
[1.2.16]: https://github.com/bymaxone/nest-queue/compare/v1.2.15...v1.2.16
|
|
1040
1065
|
[1.2.15]: https://github.com/bymaxone/nest-queue/compare/v1.2.14...v1.2.15
|
|
1041
1066
|
[1.2.14]: https://github.com/bymaxone/nest-queue/compare/v1.2.13...v1.2.14
|
|
1042
1067
|
[1.2.13]: https://github.com/bymaxone/nest-queue/compare/v1.2.12...v1.2.13
|
package/README.md
CHANGED
|
@@ -229,22 +229,22 @@ export class EmailModule {}
|
|
|
229
229
|
|
|
230
230
|
Every option passed to `forRoot` / `forRootAsync`. Only `connection` is required.
|
|
231
231
|
|
|
232
|
-
| Option | Type | Default | Description
|
|
233
|
-
| -------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
234
|
-
| `connection` | `{ client } \| { url, options? } \| { options }` | — | **Required.** Mode A supplies an existing `ioredis` client; Mode B lets the library open its own. The arms are mutually exclusive.
|
|
235
|
-
| `prefix` | `string` | `'bull'` | Redis key namespace. Applied to the `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer` — see [Architecture](#-architecture).
|
|
236
|
-
| `defaultJobOptions` | `JobsOptions` | `attempts: 3` · exponential backoff `2000` ms with 50% jitter · `removeOnComplete: { age: 24 h, count: 1000 }` · `removeOnFail: { age: 7 d, count: 5000 }` | Merged into every job enqueued through the service; per-job options win. The bounded `removeOn*` caps are what keep Redis from growing without limit under a failure storm.
|
|
237
|
-
| `queueOptions` | `Partial<QueueOptions>` | `{}` | Extra BullMQ `Queue` options. `connection`, `defaultJobOptions` and `prefix` are owned by the library and cannot be set here.
|
|
238
|
-
| `flows.enabled` | `boolean` | `false` | `FlowService` is always provided; while disabled, every method except `close()` throws `FLOW_DISABLED`.
|
|
239
|
-
| `metrics.enabled` | `boolean` | `false` | `MetricsService` is always provided; while disabled, every method throws `METRICS_DISABLED`.
|
|
240
|
-
| `metrics.cacheTtlMs` | `number` | `5_000` | TTL of the metrics snapshot cache. `0` disables caching. Must be `>= 0`.
|
|
241
|
-
| `telemetry` | `Telemetry` | — | A BullMQ `Telemetry` instance (typically `new BullMQOtel(...)` from the optional `bullmq-otel` peer), attached to every `Queue` and `Worker`.
|
|
242
|
-
| `shutdown.drainTimeoutMs` | `number` | `30_000` | Upper bound on the graceful drain before workers are force-closed.
|
|
243
|
-
| `shutdown.drainOnShutdown` | `boolean` | `false` | Development and test only: once the HTTP server has stopped, call `drain()` on every cached queue, which **removes** its waiting jobs (delayed and active jobs are left alone). Leave it `false` in production.
|
|
244
|
-
| `connectionReadyTimeoutMs` | `number` | `10_000` | Mode B only: the budget for Redis to reach `ready
|
|
245
|
-
| `commandTimeoutMs` | `number` | `5_000` | Mode B only: the deadline for one Redis command on the connection the library opened (ioredis `commandTimeout`); an `options.commandTimeout` wins. A call that hits it rejects with `CONNECTION_TIMEOUT` (`details` carries `operation` and `timeoutMs`), as when `enqueueTimeoutMs` passes. Worker and QueueEvents connections have none.
|
|
246
|
-
| `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `QueueService` call that reaches Redis, in either mode, including the wait for the client to be ready on a queue's first call; it rejects with `CONNECTION_TIMEOUT`, and an `enqueue` may still complete in Redis, so retry with a stable `jobId` or `deduplication`.
|
|
247
|
-
| `isGlobal` | `boolean` | `true` | Register the module globally, so `QueueService` is injectable without re-importing.
|
|
232
|
+
| Option | Type | Default | Description |
|
|
233
|
+
| -------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
234
|
+
| `connection` | `{ client } \| { url, options? } \| { options }` | — | **Required.** Mode A supplies an existing `ioredis` client; Mode B lets the library open its own. The arms are mutually exclusive. |
|
|
235
|
+
| `prefix` | `string` | `'bull'` | Redis key namespace. Applied to the `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer` — see [Architecture](#-architecture). |
|
|
236
|
+
| `defaultJobOptions` | `JobsOptions` | `attempts: 3` · exponential backoff `2000` ms with 50% jitter · `removeOnComplete: { age: 24 h, count: 1000 }` · `removeOnFail: { age: 7 d, count: 5000 }` | Merged into every job enqueued through the service; per-job options win. The bounded `removeOn*` caps are what keep Redis from growing without limit under a failure storm. |
|
|
237
|
+
| `queueOptions` | `Partial<QueueOptions>` | `{}` | Extra BullMQ `Queue` options. `connection`, `defaultJobOptions` and `prefix` are owned by the library and cannot be set here. |
|
|
238
|
+
| `flows.enabled` | `boolean` | `false` | `FlowService` is always provided; while disabled, every method except `close()` throws `FLOW_DISABLED`. |
|
|
239
|
+
| `metrics.enabled` | `boolean` | `false` | `MetricsService` is always provided; while disabled, every method throws `METRICS_DISABLED`. |
|
|
240
|
+
| `metrics.cacheTtlMs` | `number` | `5_000` | TTL of the metrics snapshot cache. `0` disables caching. Must be `>= 0`. |
|
|
241
|
+
| `telemetry` | `Telemetry` | — | A BullMQ `Telemetry` instance (typically `new BullMQOtel(...)` from the optional `bullmq-otel` peer), attached to every `Queue` and `Worker`. |
|
|
242
|
+
| `shutdown.drainTimeoutMs` | `number` | `30_000` | Upper bound on the graceful drain before workers are force-closed. A whole number from 1 to 2 147 483 647. |
|
|
243
|
+
| `shutdown.drainOnShutdown` | `boolean` | `false` | Development and test only: once the HTTP server has stopped, call `drain()` on every cached queue, which **removes** its waiting jobs (delayed and active jobs are left alone). Leave it `false` in production. |
|
|
244
|
+
| `connectionReadyTimeoutMs` | `number` | `10_000` | Mode B only: the budget for Redis to reach `ready`, a whole number from 1 to 2 147 483 647. A connection error while the client is still reconnecting does not end the wait; when the budget ends, startup fails with `CONNECTION_TIMEOUT` carrying `lastErrorCode` (a code such as `ECONNREFUSED`, never the message) and the client the library opened is closed. |
|
|
245
|
+
| `commandTimeoutMs` | `number` | `5_000` | Mode B only: the deadline for one Redis command on the connection the library opened (ioredis `commandTimeout`); an `options.commandTimeout` wins. A call that hits it rejects with `CONNECTION_TIMEOUT` (`details` carries `operation` and `timeoutMs`), as when `enqueueTimeoutMs` passes. Worker and QueueEvents connections have none. |
|
|
246
|
+
| `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `QueueService` call that reaches Redis, in either mode, including the wait for the client to be ready on a queue's first call; it rejects with `CONNECTION_TIMEOUT`, and an `enqueue` may still complete in Redis, so retry with a stable `jobId` or `deduplication`. |
|
|
247
|
+
| `isGlobal` | `boolean` | `true` | Register the module globally, so `QueueService` is injectable without re-importing. |
|
|
248
248
|
|
|
249
249
|
> [!WARNING]
|
|
250
250
|
> **`shutdown.drainOnShutdown` deletes jobs.** When it is `true`, shutdown calls `drain()` on every
|
|
@@ -737,7 +737,7 @@ import type {
|
|
|
737
737
|
| `cleanQueue` | `(queue, gracePeriodMs, limit, status?) → Promise<string[]>` | Remove up to `limit` jobs older than `gracePeriodMs` (`0` = no limit); returns removed ids |
|
|
738
738
|
| `upsertJobScheduler` | `<TData, TResult>(queue, id, repeat, template?) → Promise<Job<TData, TResult> \| undefined>` | Create or update a recurring Job Scheduler; returns the first scheduled job |
|
|
739
739
|
| `removeJobScheduler` | `(queue, id) → Promise<boolean>` | Remove a Job Scheduler by ID |
|
|
740
|
-
| `getJobSchedulers` | `(queue, start?, end?, asc?) → Promise<JobSchedulerJson[]>` | Page the schedulers registered on a queue
|
|
740
|
+
| `getJobSchedulers` | `(queue, start?, end?, asc?) → Promise<JobSchedulerJson[]>` | Page the schedulers registered on a queue (at most 1 000 per call) |
|
|
741
741
|
| `getOrCreateQueue` | `<TData, TResult>(queue, overrides?) → Queue<TData, TResult>` | Return the cached raw `Queue` instance |
|
|
742
742
|
| `getCachedQueues` | `() → ReadonlyMap<string, Queue>` | Read-only view of every queue created so far |
|
|
743
743
|
|
|
@@ -809,22 +809,24 @@ const metrics: QueueMetrics = await this.metricsService.get('email')
|
|
|
809
809
|
const all: readonly QueueMetrics[] = await this.metricsService.getAll()
|
|
810
810
|
```
|
|
811
811
|
|
|
812
|
+
`FlowService.add` and `addBulk` are held to `enqueueTimeoutMs` like every `QueueService` call and reject with `CONNECTION_TIMEOUT` when Redis does not answer in time. A flow that timed out may still have been added, so use a custom `jobId` when you retry.
|
|
813
|
+
|
|
812
814
|
### Other exports
|
|
813
815
|
|
|
814
|
-
| Export | What it is
|
|
815
|
-
| ------------------------------------------------------------------------------------------------------------------------------------- |
|
|
816
|
-
| `WorkerRegistry.unregister(queue)`, `list()` | Stop and remove a worker and its connection; list the queue names that have a worker
|
|
817
|
-
| `FlowService.addBulk(flows)`, `getProducer()`, `close()` | Add
|
|
818
|
-
| `MetricsService.invalidate(queue?)` | Drop the cached snapshot of one queue, or of all of them
|
|
819
|
-
| `QueueEventsRegistry` | The shared `QueueEvents` per queue (`getOrCreate`, `list`), used by `@OnQueueEvent`
|
|
820
|
-
| `BYMAX_QUEUE_OPTIONS`, `BYMAX_QUEUE_RESOLVED_OPTIONS` | Injection tokens for the options as given and with the defaults applied
|
|
821
|
-
| `BYMAX_QUEUE_REDIS_CLIENT`, `BYMAX_QUEUE_CONNECTION_MODE` | Injection tokens for the Queue-role ioredis client and for `'mode-a-byo'` or `'mode-b-owned'`
|
|
822
|
-
| `DEFAULT_WORKER_CONCURRENCY`, `MAX_WORKER_CONCURRENCY` | The worker concurrency used when none is given, and its ceiling
|
|
823
|
-
| `DEFAULT_JOB_OPTIONS`, `DEFAULT_METRICS_CACHE_TTL_MS`, `DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS` | The defaults of the Configuration table
|
|
824
|
-
| `QUEUE_ERROR_MESSAGES` | The default message of each error code
|
|
825
|
-
| `BymaxQueueModuleOptions`, `BymaxQueueModuleAsyncOptions`, `BymaxQueueOptionsFactory`, `QueueConnectionConfig`, `QueueConnectionMode` | The option and connection types behind `forRoot` and `forRootAsync`
|
|
826
|
-
| `ProcessorMetadata`, `ProcessHandlerMetadata`, `QueueEventListenerMetadata` | The reflection metadata the decorators write
|
|
827
|
-
| `JobStatus`, `JOB_STATUS`, `BulkJob`, `SandboxedJob`, `WorkerEventName`, `QueueEventName`, `WorkerOptions` | Constants and types for the APIs above
|
|
816
|
+
| Export | What it is |
|
|
817
|
+
| ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
|
818
|
+
| `WorkerRegistry.unregister(queue)`, `list()` | Stop and remove a worker and its connection; list the queue names that have a worker |
|
|
819
|
+
| `FlowService.addBulk(flows)`, `getProducer()`, `close()` | Add up to 1 000 flows at once; the underlying BullMQ `FlowProducer`; close it (a no-op while flows are disabled) |
|
|
820
|
+
| `MetricsService.invalidate(queue?)` | Drop the cached snapshot of one queue, or of all of them |
|
|
821
|
+
| `QueueEventsRegistry` | The shared `QueueEvents` per queue (`getOrCreate`, `list`), used by `@OnQueueEvent` |
|
|
822
|
+
| `BYMAX_QUEUE_OPTIONS`, `BYMAX_QUEUE_RESOLVED_OPTIONS` | Injection tokens for the options as given and with the defaults applied |
|
|
823
|
+
| `BYMAX_QUEUE_REDIS_CLIENT`, `BYMAX_QUEUE_CONNECTION_MODE` | Injection tokens for the Queue-role ioredis client and for `'mode-a-byo'` or `'mode-b-owned'` |
|
|
824
|
+
| `DEFAULT_WORKER_CONCURRENCY`, `MAX_WORKER_CONCURRENCY` | The worker concurrency used when none is given, and its ceiling |
|
|
825
|
+
| `DEFAULT_JOB_OPTIONS`, `DEFAULT_METRICS_CACHE_TTL_MS`, `DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS` | The defaults of the Configuration table |
|
|
826
|
+
| `QUEUE_ERROR_MESSAGES` | The default message of each error code |
|
|
827
|
+
| `BymaxQueueModuleOptions`, `BymaxQueueModuleAsyncOptions`, `BymaxQueueOptionsFactory`, `QueueConnectionConfig`, `QueueConnectionMode` | The option and connection types behind `forRoot` and `forRootAsync` |
|
|
828
|
+
| `ProcessorMetadata`, `ProcessHandlerMetadata`, `QueueEventListenerMetadata` | The reflection metadata the decorators write |
|
|
829
|
+
| `JobStatus`, `JOB_STATUS`, `BulkJob`, `SandboxedJob`, `WorkerEventName`, `QueueEventName`, `WorkerOptions` | Constants and types for the APIs above |
|
|
828
830
|
|
|
829
831
|
---
|
|
830
832
|
|
|
@@ -857,7 +859,7 @@ try {
|
|
|
857
859
|
|
|
858
860
|
| Code | HTTP | When thrown |
|
|
859
861
|
| ---------------------------------------- | :--: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
860
|
-
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms,
|
|
862
|
+
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, a `shutdown.drainTimeoutMs`, `connectionReadyTimeoutMs`, `commandTimeoutMs` or `enqueueTimeoutMs` that is not a whole number from 1 to 2 147 483 647, or negative `metrics.cacheTtlMs` |
|
|
861
863
|
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, Mode A's injected client is in an unusable `status`, or a Mode B client was refused by the server (`WRONGPASS`, `NOAUTH`) or stopped reconnecting during startup |
|
|
862
864
|
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
863
865
|
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` (`details` carries `timeoutMs` and the last error's `lastErrorCode`), or a `QueueService`/`FlowService` call got no answer from Redis: it timed out (`commandTimeoutMs` on the owned connection, or `enqueueTimeoutMs` for a `QueueService` call; `details` carries `operation` and `timeoutMs`) or the connection is down (`details` carries `operation` and `reason: 'offline'`); an enqueue may still have been added |
|
|
@@ -865,7 +867,7 @@ try {
|
|
|
865
867
|
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
866
868
|
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
867
869
|
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
868
|
-
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000
|
|
870
|
+
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` or `FlowService.addBulk` exceeded the 1 000 cap, or the underlying `addBulk` failed |
|
|
869
871
|
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
870
872
|
|
|
871
873
|
### Reserved codes
|
|
@@ -917,7 +919,7 @@ every `QueueService` call is bounded by `enqueueTimeoutMs` whatever the client d
|
|
|
917
919
|
|
|
918
920
|
### A handler never settles (the queue stops moving)
|
|
919
921
|
|
|
920
|
-
A handler whose promise never settles holds its concurrency slot for as long as the process lives, and BullMQ keeps renewing its lock, so the stalled-job check never reclaims it: with every slot held, the rest of the queue waits. Set `jobTimeoutMs` in the worker `options` to bound it. When the time passes, the job fails with a `queue.job_timeout` error and retries under its `attempts` and `backoff`, and the slot is free again. The handler receives the job's `AbortSignal` as its second argument; it aborts at that moment, or when BullMQ cancels the job (`worker.cancelJob()`, `worker.cancelAllJobs()`), and closing the worker does not abort it; pass it to whatever the handler awaits so the work stops with the job. A handler that ignores the signal keeps running in the background, and its retry can overlap it, so it must stay idempotent. There is no default, and sandboxed workers do not accept the option.
|
|
922
|
+
A handler whose promise never settles holds its concurrency slot for as long as the process lives, and BullMQ keeps renewing its lock, so the stalled-job check never reclaims it: with every slot held, the rest of the queue waits. Set `jobTimeoutMs` in the worker `options` to bound it. When the time passes, the job fails with a `queue.job_timeout` error and retries under its `attempts` and `backoff`, and the slot is free again. The handler receives the job's `AbortSignal` as its second argument; it aborts at that moment, or when BullMQ cancels the job (`worker.cancelJob()`, `worker.cancelAllJobs()`), and closing the worker does not abort it; pass it to whatever the handler awaits so the work stops with the job. A handler that ignores the signal keeps running in the background, and its retry can overlap it, so it must stay idempotent. There is no default, and sandboxed workers do not accept the option. Once `jobTimeoutMs` has fired, `concurrency` no longer bounds how many handlers are alive: the slot is free for the next job while the timed-out handler may still be running, so a handler that ignores the signal can overlap its successors.
|
|
921
923
|
|
|
922
924
|
```typescript
|
|
923
925
|
@Processor('reports', { jobTimeoutMs: 60_000 })
|
|
@@ -935,7 +937,9 @@ Stalled jobs are re-queued automatically by BullMQ's stalled-job check. If worke
|
|
|
935
937
|
do not pick them up:
|
|
936
938
|
|
|
937
939
|
1. Verify `drainTimeoutMs` is larger than the worst-case job runtime.
|
|
938
|
-
2. Verify `lockDuration` (in the `options` you pass to `@Processor` or `register
|
|
940
|
+
2. Verify `lockDuration` (in the `options` you pass to `@Processor` or `register`; a whole
|
|
941
|
+
number from 1000 to 2 147 483 647, as is `stalledInterval`; `concurrency` and the
|
|
942
|
+
limiter values must be whole numbers too) covers
|
|
939
943
|
the handler execution time — a worker that runs longer than `lockDuration` will have its
|
|
940
944
|
lock stolen, marking the job as stalled.
|
|
941
945
|
3. BullMQ re-queues a stalled job up to its `maxStalledCount` times (default `1`) and then
|
package/dist/server/index.cjs
CHANGED
|
@@ -90,6 +90,11 @@ function invalid(reason) {
|
|
|
90
90
|
reason
|
|
91
91
|
});
|
|
92
92
|
}
|
|
93
|
+
function checkTimerDelay(name, value) {
|
|
94
|
+
if (value !== void 0 && !isTimerDelayMs(value)) {
|
|
95
|
+
invalid(`${name} must be a whole number from 1 to ${String(MAX_TIMER_DELAY_MS)}`);
|
|
96
|
+
}
|
|
97
|
+
}
|
|
93
98
|
function isParseableUrl(url) {
|
|
94
99
|
const candidate = /^rediss?:\/\//i.test(url) ? url : url.startsWith("//") ? `redis:${url}` : `redis://${url}`;
|
|
95
100
|
if (!URL.canParse(candidate)) return false;
|
|
@@ -130,15 +135,13 @@ function validateOptions(opts) {
|
|
|
130
135
|
if (drainTimeoutMs !== void 0 && drainTimeoutMs <= 0) {
|
|
131
136
|
invalid("shutdown.drainTimeoutMs must be > 0");
|
|
132
137
|
}
|
|
138
|
+
checkTimerDelay("shutdown.drainTimeoutMs", drainTimeoutMs);
|
|
133
139
|
const cacheTtlMs = opts.metrics?.cacheTtlMs;
|
|
134
140
|
if (cacheTtlMs !== void 0 && cacheTtlMs < 0) {
|
|
135
141
|
invalid("metrics.cacheTtlMs must be >= 0");
|
|
136
142
|
}
|
|
137
|
-
for (const key of ["commandTimeoutMs", "enqueueTimeoutMs"]) {
|
|
138
|
-
|
|
139
|
-
if (value !== void 0 && !isTimerDelayMs(value)) {
|
|
140
|
-
invalid(`${key} must be a whole number from 1 to ${String(MAX_TIMER_DELAY_MS)}`);
|
|
141
|
-
}
|
|
143
|
+
for (const key of ["commandTimeoutMs", "enqueueTimeoutMs", "connectionReadyTimeoutMs"]) {
|
|
144
|
+
checkTimerDelay(key, opts[key]);
|
|
142
145
|
}
|
|
143
146
|
}
|
|
144
147
|
|
|
@@ -157,6 +160,7 @@ var DEFAULT_ENQUEUE_TIMEOUT_MS = 1e4;
|
|
|
157
160
|
var MAX_JOBS_PAGE_SIZE = 1e3;
|
|
158
161
|
var DEFAULT_METRICS_CACHE_TTL_MS = 5e3;
|
|
159
162
|
var DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS = 3e4;
|
|
163
|
+
var MAX_BULK_SIZE = 1e3;
|
|
160
164
|
|
|
161
165
|
// src/server/config/resolved-options.ts
|
|
162
166
|
function applyDefaults(opts) {
|
|
@@ -317,10 +321,14 @@ exports.ConnectionResolver = class ConnectionResolver {
|
|
|
317
321
|
const client = this.getClient();
|
|
318
322
|
if (!WAITING_STATUSES.has(client.status)) return Promise.resolve();
|
|
319
323
|
__privateGet(this, _pendingReady) ?? __privateSet(this, _pendingReady, new Promise((resolve) => {
|
|
320
|
-
|
|
324
|
+
const release = () => {
|
|
325
|
+
client.off("ready", release);
|
|
326
|
+
client.off("end", release);
|
|
321
327
|
__privateSet(this, _pendingReady, void 0);
|
|
322
328
|
resolve();
|
|
323
|
-
}
|
|
329
|
+
};
|
|
330
|
+
client.on("ready", release);
|
|
331
|
+
client.on("end", release);
|
|
324
332
|
}));
|
|
325
333
|
return __privateGet(this, _pendingReady);
|
|
326
334
|
}
|
|
@@ -453,6 +461,13 @@ function attachDefaultErrorListener(emitter, logger, kind, queueName) {
|
|
|
453
461
|
logger.error(`${kind} for queue "${queueName}" emitted an error: ${describeError(err)}`);
|
|
454
462
|
});
|
|
455
463
|
}
|
|
464
|
+
function assertPageWindow(operation, start, end) {
|
|
465
|
+
if (!Number.isInteger(start) || !Number.isInteger(end) || start < 0 || end < start || end - start >= MAX_JOBS_PAGE_SIZE) {
|
|
466
|
+
throw new QueueException(QUEUE_ERROR_CODES.INVALID_OPTIONS, common.HttpStatus.BAD_REQUEST, {
|
|
467
|
+
reason: `${operation} needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
|
|
468
|
+
});
|
|
469
|
+
}
|
|
470
|
+
}
|
|
456
471
|
|
|
457
472
|
// src/server/utils/with-deadline.ts
|
|
458
473
|
function withDeadline(work, timeoutMs, onTimeout) {
|
|
@@ -500,7 +515,6 @@ async function withCommandTimeoutCode(operation, commandTimeoutMs, work) {
|
|
|
500
515
|
}
|
|
501
516
|
|
|
502
517
|
// src/server/services/queue.service.ts
|
|
503
|
-
var MAX_BULK_SIZE = 1e3;
|
|
504
518
|
function assertValidCron(repeat, queueOptions) {
|
|
505
519
|
if (queueOptions.settings?.repeatStrategy !== void 0) return;
|
|
506
520
|
try {
|
|
@@ -615,11 +629,7 @@ exports.QueueService = class QueueService {
|
|
|
615
629
|
return job ?? null;
|
|
616
630
|
}
|
|
617
631
|
async getJobs(queueName, status, start = 0, end = 50) {
|
|
618
|
-
|
|
619
|
-
throw new QueueException(QUEUE_ERROR_CODES.INVALID_OPTIONS, common.HttpStatus.BAD_REQUEST, {
|
|
620
|
-
reason: `getJobs needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
|
|
621
|
-
});
|
|
622
|
-
}
|
|
632
|
+
assertPageWindow("getJobs", start, end);
|
|
623
633
|
const jobs = await this.call(
|
|
624
634
|
"getJobs",
|
|
625
635
|
queueName,
|
|
@@ -663,6 +673,7 @@ exports.QueueService = class QueueService {
|
|
|
663
673
|
);
|
|
664
674
|
}
|
|
665
675
|
async getJobSchedulers(queueName, start = 0, end = 50, asc = true) {
|
|
676
|
+
assertPageWindow("getJobSchedulers", start, end);
|
|
666
677
|
return this.call(
|
|
667
678
|
"getJobSchedulers",
|
|
668
679
|
queueName,
|
|
@@ -703,13 +714,15 @@ exports.QueueService = __decorateClass([
|
|
|
703
714
|
function errorCodeOf(err) {
|
|
704
715
|
return err.getResponse().error.code;
|
|
705
716
|
}
|
|
706
|
-
var _connection;
|
|
717
|
+
var _connection, _deadlineMs;
|
|
707
718
|
exports.FlowService = class FlowService {
|
|
708
|
-
constructor(connection, enabled, prefix, telemetry) {
|
|
719
|
+
constructor(connection, enabled, prefix, telemetry, enqueueTimeoutMs = DEFAULT_ENQUEUE_TIMEOUT_MS) {
|
|
709
720
|
__publicField(this, "logger", new common.Logger(exports.FlowService.name));
|
|
710
721
|
__publicField(this, "producer");
|
|
711
722
|
__privateAdd(this, _connection);
|
|
723
|
+
__privateAdd(this, _deadlineMs);
|
|
712
724
|
__privateSet(this, _connection, connection);
|
|
725
|
+
__privateSet(this, _deadlineMs, enqueueTimeoutMs);
|
|
713
726
|
if (enabled) {
|
|
714
727
|
this.producer = new bullmq.FlowProducer({
|
|
715
728
|
connection: connection.getClient(),
|
|
@@ -720,17 +733,24 @@ exports.FlowService = class FlowService {
|
|
|
720
733
|
}
|
|
721
734
|
}
|
|
722
735
|
async add(flow) {
|
|
723
|
-
return
|
|
724
|
-
"addFlow",
|
|
725
|
-
() => this.commandTimeout(),
|
|
726
|
-
this.requireProducer().add(flow)
|
|
727
|
-
);
|
|
736
|
+
return this.bounded("addFlow", this.requireProducer().add(flow));
|
|
728
737
|
}
|
|
729
738
|
async addBulk(flows) {
|
|
739
|
+
const producer = this.requireProducer();
|
|
740
|
+
if (flows.length > MAX_BULK_SIZE) {
|
|
741
|
+
throw new QueueException(
|
|
742
|
+
QUEUE_ERROR_CODES.BULK_ENQUEUE_FAILED,
|
|
743
|
+
common.HttpStatus.INTERNAL_SERVER_ERROR,
|
|
744
|
+
{ reason: "batch size exceeds limit", limit: MAX_BULK_SIZE, received: flows.length }
|
|
745
|
+
);
|
|
746
|
+
}
|
|
747
|
+
return this.bounded("addFlowBulk", producer.addBulk([...flows]));
|
|
748
|
+
}
|
|
749
|
+
bounded(operation, work) {
|
|
730
750
|
return withCommandTimeoutCode(
|
|
731
|
-
|
|
751
|
+
operation,
|
|
732
752
|
() => this.commandTimeout(),
|
|
733
|
-
this
|
|
753
|
+
withDeadline(work, __privateGet(this, _deadlineMs), () => timeoutException(operation, __privateGet(this, _deadlineMs)))
|
|
734
754
|
);
|
|
735
755
|
}
|
|
736
756
|
commandTimeout() {
|
|
@@ -752,6 +772,7 @@ exports.FlowService = class FlowService {
|
|
|
752
772
|
}
|
|
753
773
|
};
|
|
754
774
|
_connection = new WeakMap();
|
|
775
|
+
_deadlineMs = new WeakMap();
|
|
755
776
|
exports.FlowService = __decorateClass([
|
|
756
777
|
common.Injectable()
|
|
757
778
|
], exports.FlowService);
|
|
@@ -798,6 +819,30 @@ exports.MetricsService = __decorateClass([
|
|
|
798
819
|
common.Injectable()
|
|
799
820
|
], exports.MetricsService);
|
|
800
821
|
|
|
822
|
+
// src/server/utils/worker-option-problem.ts
|
|
823
|
+
var WORKER_TIMER_MIN_MS = 1e3;
|
|
824
|
+
function countProblem(name, value, min, max) {
|
|
825
|
+
if (!Number.isInteger(value)) return { reason: `${name} must be a whole number`, actual: value };
|
|
826
|
+
if (value < min) return { reason: `${name} must be >= ${String(min)}`, actual: value };
|
|
827
|
+
if (value > max) return { reason: `${name} must be <= ${String(max)}`, actual: value };
|
|
828
|
+
return void 0;
|
|
829
|
+
}
|
|
830
|
+
function timerProblem(name, value) {
|
|
831
|
+
if (value === void 0) return void 0;
|
|
832
|
+
if (Number.isInteger(value) && value >= WORKER_TIMER_MIN_MS && value <= MAX_TIMER_DELAY_MS) {
|
|
833
|
+
return void 0;
|
|
834
|
+
}
|
|
835
|
+
return {
|
|
836
|
+
reason: `${name} must be a whole number from ${String(WORKER_TIMER_MIN_MS)} to ${String(MAX_TIMER_DELAY_MS)}`,
|
|
837
|
+
actual: value
|
|
838
|
+
};
|
|
839
|
+
}
|
|
840
|
+
function workerOptionProblem(opts) {
|
|
841
|
+
if (opts === void 0) return void 0;
|
|
842
|
+
const { concurrency, limiter, lockDuration, stalledInterval } = opts;
|
|
843
|
+
return (concurrency === void 0 ? void 0 : countProblem("concurrency", concurrency, 1, MAX_WORKER_CONCURRENCY)) ?? (limiter === void 0 ? void 0 : countProblem("limiter.max", limiter.max, 1, Number.POSITIVE_INFINITY) ?? countProblem("limiter.duration", limiter.duration, 1, Number.POSITIVE_INFINITY)) ?? timerProblem("lockDuration", lockDuration) ?? timerProblem("stalledInterval", stalledInterval);
|
|
844
|
+
}
|
|
845
|
+
|
|
801
846
|
// src/server/utils/abandoned-workers.ts
|
|
802
847
|
var abandoned = /* @__PURE__ */ new WeakSet();
|
|
803
848
|
function abandonWorker(worker) {
|
|
@@ -921,51 +966,15 @@ exports.WorkerRegistry = class WorkerRegistry {
|
|
|
921
966
|
}
|
|
922
967
|
}
|
|
923
968
|
validateOptions(opts) {
|
|
924
|
-
|
|
925
|
-
|
|
926
|
-
|
|
927
|
-
|
|
928
|
-
|
|
929
|
-
|
|
930
|
-
|
|
931
|
-
actual: opts.concurrency
|
|
932
|
-
}
|
|
933
|
-
);
|
|
934
|
-
}
|
|
935
|
-
if (opts.concurrency > MAX_WORKER_CONCURRENCY) {
|
|
936
|
-
throw new QueueException(
|
|
937
|
-
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
938
|
-
common.HttpStatus.INTERNAL_SERVER_ERROR,
|
|
939
|
-
{
|
|
940
|
-
reason: `concurrency must be <= ${MAX_WORKER_CONCURRENCY.toString()}`,
|
|
941
|
-
actual: opts.concurrency
|
|
942
|
-
}
|
|
943
|
-
);
|
|
944
|
-
}
|
|
969
|
+
const problem = workerOptionProblem(opts);
|
|
970
|
+
if (problem !== void 0) {
|
|
971
|
+
throw new QueueException(
|
|
972
|
+
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
973
|
+
common.HttpStatus.INTERNAL_SERVER_ERROR,
|
|
974
|
+
problem
|
|
975
|
+
);
|
|
945
976
|
}
|
|
946
977
|
this.validateJobTimeout(opts);
|
|
947
|
-
if (opts?.limiter !== void 0) {
|
|
948
|
-
if (opts.limiter.max < 1) {
|
|
949
|
-
throw new QueueException(
|
|
950
|
-
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
951
|
-
common.HttpStatus.INTERNAL_SERVER_ERROR,
|
|
952
|
-
{
|
|
953
|
-
reason: "limiter.max must be >= 1",
|
|
954
|
-
actual: opts.limiter.max
|
|
955
|
-
}
|
|
956
|
-
);
|
|
957
|
-
}
|
|
958
|
-
if (opts.limiter.duration < 1) {
|
|
959
|
-
throw new QueueException(
|
|
960
|
-
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
961
|
-
common.HttpStatus.INTERNAL_SERVER_ERROR,
|
|
962
|
-
{
|
|
963
|
-
reason: "limiter.duration must be >= 1",
|
|
964
|
-
actual: opts.limiter.duration
|
|
965
|
-
}
|
|
966
|
-
);
|
|
967
|
-
}
|
|
968
|
-
}
|
|
969
978
|
}
|
|
970
979
|
validateJobTimeout(opts) {
|
|
971
980
|
if (opts?.jobTimeoutMs !== void 0 && !isTimerDelayMs(opts.jobTimeoutMs)) {
|
|
@@ -1387,7 +1396,13 @@ exports.BymaxQueueModule = class BymaxQueueModule extends ConfigurableModuleClas
|
|
|
1387
1396
|
{ provide: BYMAX_QUEUE_RESOLVED_OPTIONS, useValue: resolved },
|
|
1388
1397
|
{
|
|
1389
1398
|
provide: exports.FlowService,
|
|
1390
|
-
useFactory: (conn) => new exports.FlowService(
|
|
1399
|
+
useFactory: (conn) => new exports.FlowService(
|
|
1400
|
+
conn,
|
|
1401
|
+
resolved.flows.enabled,
|
|
1402
|
+
resolved.prefix,
|
|
1403
|
+
resolved.telemetry,
|
|
1404
|
+
resolved.enqueueTimeoutMs
|
|
1405
|
+
),
|
|
1391
1406
|
inject: [exports.ConnectionResolver]
|
|
1392
1407
|
},
|
|
1393
1408
|
{
|
|
@@ -1417,7 +1432,13 @@ exports.BymaxQueueModule = class BymaxQueueModule extends ConfigurableModuleClas
|
|
|
1417
1432
|
},
|
|
1418
1433
|
{
|
|
1419
1434
|
provide: exports.FlowService,
|
|
1420
|
-
useFactory: (conn, resolved) => new exports.FlowService(
|
|
1435
|
+
useFactory: (conn, resolved) => new exports.FlowService(
|
|
1436
|
+
conn,
|
|
1437
|
+
resolved.flows.enabled,
|
|
1438
|
+
resolved.prefix,
|
|
1439
|
+
resolved.telemetry,
|
|
1440
|
+
resolved.enqueueTimeoutMs
|
|
1441
|
+
),
|
|
1421
1442
|
inject: [exports.ConnectionResolver, BYMAX_QUEUE_RESOLVED_OPTIONS]
|
|
1422
1443
|
},
|
|
1423
1444
|
{
|
package/dist/server/index.d.cts
CHANGED
|
@@ -597,9 +597,11 @@ declare class QueueService {
|
|
|
597
597
|
*
|
|
598
598
|
* @param queueName - Target queue.
|
|
599
599
|
* @param start - Page start index (inclusive). Default: 0.
|
|
600
|
-
* @param end - Page end index (inclusive). Default: 50.
|
|
600
|
+
* @param end - Page end index (inclusive). Default: 50. The window may span at most
|
|
601
|
+
* `MAX_JOBS_PAGE_SIZE` (1000) entries, and BullMQ's `-1` (everything) is refused.
|
|
601
602
|
* @param asc - Sort ascending by next run time. Default: true.
|
|
602
603
|
* @returns The registered schedulers in the requested page.
|
|
604
|
+
* @throws {QueueException} `INVALID_OPTIONS` when the window is not a bounded range of integers.
|
|
603
605
|
*/
|
|
604
606
|
getJobSchedulers(queueName: string, start?: number, end?: number, asc?: boolean): Promise<Awaited<ReturnType<Queue['getJobSchedulers']>>>;
|
|
605
607
|
/** Pause a queue, halting the processing of new jobs. */
|
|
@@ -660,7 +662,7 @@ declare class FlowService {
|
|
|
660
662
|
#private;
|
|
661
663
|
private readonly logger;
|
|
662
664
|
private readonly producer?;
|
|
663
|
-
constructor(connection: ConnectionResolver, enabled: boolean, prefix: string, telemetry?: Telemetry);
|
|
665
|
+
constructor(connection: ConnectionResolver, enabled: boolean, prefix: string, telemetry?: Telemetry, enqueueTimeoutMs?: number);
|
|
664
666
|
/**
|
|
665
667
|
* Add a flow tree (a parent with its descendant children). The root job
|
|
666
668
|
* becomes processable only after every descendant has completed.
|
|
@@ -678,7 +680,8 @@ declare class FlowService {
|
|
|
678
680
|
* A job whose custom `jobId` already exists is not added again: its node carries the EXISTING
|
|
679
681
|
* id with the data of this call, so the tree does not describe what is stored; call
|
|
680
682
|
* `QueueService.getJob` when you need it.
|
|
681
|
-
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled
|
|
683
|
+
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled; `CONNECTION_TIMEOUT`
|
|
684
|
+
* when Redis does not answer within `enqueueTimeoutMs` (the flow may still have been added).
|
|
682
685
|
*/
|
|
683
686
|
add(flow: FlowJob): Promise<JobNode>;
|
|
684
687
|
/**
|
|
@@ -687,9 +690,17 @@ declare class FlowService {
|
|
|
687
690
|
* @param flows - Flow tree definitions.
|
|
688
691
|
* @returns The root `JobNode` for each flow, in input order.
|
|
689
692
|
* A job whose custom `jobId` already exists is not added again; see {@link FlowService.add}.
|
|
690
|
-
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled
|
|
693
|
+
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled;
|
|
694
|
+
* `BULK_ENQUEUE_FAILED` for more than 1000 flows; `CONNECTION_TIMEOUT` as for
|
|
695
|
+
* {@link FlowService.add}.
|
|
691
696
|
*/
|
|
692
697
|
addBulk(flows: readonly FlowJob[]): Promise<JobNode[]>;
|
|
698
|
+
/**
|
|
699
|
+
* Hold the caller to `enqueueTimeoutMs`, and report a command timeout or a down connection as
|
|
700
|
+
* `CONNECTION_TIMEOUT`. A call that timed out may still have been added: BullMQ is not asked to
|
|
701
|
+
* cancel it.
|
|
702
|
+
*/
|
|
703
|
+
private bounded;
|
|
693
704
|
/** The command deadline of the producer connection, read when a command has timed out. */
|
|
694
705
|
private commandTimeout;
|
|
695
706
|
/**
|
|
@@ -981,8 +992,9 @@ declare class WorkerRegistry {
|
|
|
981
992
|
private guardDuplicate;
|
|
982
993
|
/**
|
|
983
994
|
* Validate tunable options before constructing a Worker.
|
|
984
|
-
* Enforces `concurrency` in `[1, MAX_WORKER_CONCURRENCY]
|
|
985
|
-
* `limiter.max`/`limiter.duration` `>= 1
|
|
995
|
+
* Enforces whole-number `concurrency` in `[1, MAX_WORKER_CONCURRENCY]`, whole-number
|
|
996
|
+
* `limiter.max`/`limiter.duration` `>= 1`, and `lockDuration`/`stalledInterval` within
|
|
997
|
+
* 1000..2^31 - 1.
|
|
986
998
|
*
|
|
987
999
|
* @param opts - The caller-supplied options (may be undefined).
|
|
988
1000
|
* @throws {QueueException} `WORKER_REGISTRATION_FAILED` when values are out of range.
|
package/dist/server/index.d.ts
CHANGED
|
@@ -597,9 +597,11 @@ declare class QueueService {
|
|
|
597
597
|
*
|
|
598
598
|
* @param queueName - Target queue.
|
|
599
599
|
* @param start - Page start index (inclusive). Default: 0.
|
|
600
|
-
* @param end - Page end index (inclusive). Default: 50.
|
|
600
|
+
* @param end - Page end index (inclusive). Default: 50. The window may span at most
|
|
601
|
+
* `MAX_JOBS_PAGE_SIZE` (1000) entries, and BullMQ's `-1` (everything) is refused.
|
|
601
602
|
* @param asc - Sort ascending by next run time. Default: true.
|
|
602
603
|
* @returns The registered schedulers in the requested page.
|
|
604
|
+
* @throws {QueueException} `INVALID_OPTIONS` when the window is not a bounded range of integers.
|
|
603
605
|
*/
|
|
604
606
|
getJobSchedulers(queueName: string, start?: number, end?: number, asc?: boolean): Promise<Awaited<ReturnType<Queue['getJobSchedulers']>>>;
|
|
605
607
|
/** Pause a queue, halting the processing of new jobs. */
|
|
@@ -660,7 +662,7 @@ declare class FlowService {
|
|
|
660
662
|
#private;
|
|
661
663
|
private readonly logger;
|
|
662
664
|
private readonly producer?;
|
|
663
|
-
constructor(connection: ConnectionResolver, enabled: boolean, prefix: string, telemetry?: Telemetry);
|
|
665
|
+
constructor(connection: ConnectionResolver, enabled: boolean, prefix: string, telemetry?: Telemetry, enqueueTimeoutMs?: number);
|
|
664
666
|
/**
|
|
665
667
|
* Add a flow tree (a parent with its descendant children). The root job
|
|
666
668
|
* becomes processable only after every descendant has completed.
|
|
@@ -678,7 +680,8 @@ declare class FlowService {
|
|
|
678
680
|
* A job whose custom `jobId` already exists is not added again: its node carries the EXISTING
|
|
679
681
|
* id with the data of this call, so the tree does not describe what is stored; call
|
|
680
682
|
* `QueueService.getJob` when you need it.
|
|
681
|
-
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled
|
|
683
|
+
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled; `CONNECTION_TIMEOUT`
|
|
684
|
+
* when Redis does not answer within `enqueueTimeoutMs` (the flow may still have been added).
|
|
682
685
|
*/
|
|
683
686
|
add(flow: FlowJob): Promise<JobNode>;
|
|
684
687
|
/**
|
|
@@ -687,9 +690,17 @@ declare class FlowService {
|
|
|
687
690
|
* @param flows - Flow tree definitions.
|
|
688
691
|
* @returns The root `JobNode` for each flow, in input order.
|
|
689
692
|
* A job whose custom `jobId` already exists is not added again; see {@link FlowService.add}.
|
|
690
|
-
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled
|
|
693
|
+
* @throws {QueueException} `FLOW_DISABLED` (503) when flows are not enabled;
|
|
694
|
+
* `BULK_ENQUEUE_FAILED` for more than 1000 flows; `CONNECTION_TIMEOUT` as for
|
|
695
|
+
* {@link FlowService.add}.
|
|
691
696
|
*/
|
|
692
697
|
addBulk(flows: readonly FlowJob[]): Promise<JobNode[]>;
|
|
698
|
+
/**
|
|
699
|
+
* Hold the caller to `enqueueTimeoutMs`, and report a command timeout or a down connection as
|
|
700
|
+
* `CONNECTION_TIMEOUT`. A call that timed out may still have been added: BullMQ is not asked to
|
|
701
|
+
* cancel it.
|
|
702
|
+
*/
|
|
703
|
+
private bounded;
|
|
693
704
|
/** The command deadline of the producer connection, read when a command has timed out. */
|
|
694
705
|
private commandTimeout;
|
|
695
706
|
/**
|
|
@@ -981,8 +992,9 @@ declare class WorkerRegistry {
|
|
|
981
992
|
private guardDuplicate;
|
|
982
993
|
/**
|
|
983
994
|
* Validate tunable options before constructing a Worker.
|
|
984
|
-
* Enforces `concurrency` in `[1, MAX_WORKER_CONCURRENCY]
|
|
985
|
-
* `limiter.max`/`limiter.duration` `>= 1
|
|
995
|
+
* Enforces whole-number `concurrency` in `[1, MAX_WORKER_CONCURRENCY]`, whole-number
|
|
996
|
+
* `limiter.max`/`limiter.duration` `>= 1`, and `lockDuration`/`stalledInterval` within
|
|
997
|
+
* 1000..2^31 - 1.
|
|
986
998
|
*
|
|
987
999
|
* @param opts - The caller-supplied options (may be undefined).
|
|
988
1000
|
* @throws {QueueException} `WORKER_REGISTRATION_FAILED` when values are out of range.
|
package/dist/server/index.mjs
CHANGED
|
@@ -88,6 +88,11 @@ function invalid(reason) {
|
|
|
88
88
|
reason
|
|
89
89
|
});
|
|
90
90
|
}
|
|
91
|
+
function checkTimerDelay(name, value) {
|
|
92
|
+
if (value !== void 0 && !isTimerDelayMs(value)) {
|
|
93
|
+
invalid(`${name} must be a whole number from 1 to ${String(MAX_TIMER_DELAY_MS)}`);
|
|
94
|
+
}
|
|
95
|
+
}
|
|
91
96
|
function isParseableUrl(url) {
|
|
92
97
|
const candidate = /^rediss?:\/\//i.test(url) ? url : url.startsWith("//") ? `redis:${url}` : `redis://${url}`;
|
|
93
98
|
if (!URL.canParse(candidate)) return false;
|
|
@@ -128,15 +133,13 @@ function validateOptions(opts) {
|
|
|
128
133
|
if (drainTimeoutMs !== void 0 && drainTimeoutMs <= 0) {
|
|
129
134
|
invalid("shutdown.drainTimeoutMs must be > 0");
|
|
130
135
|
}
|
|
136
|
+
checkTimerDelay("shutdown.drainTimeoutMs", drainTimeoutMs);
|
|
131
137
|
const cacheTtlMs = opts.metrics?.cacheTtlMs;
|
|
132
138
|
if (cacheTtlMs !== void 0 && cacheTtlMs < 0) {
|
|
133
139
|
invalid("metrics.cacheTtlMs must be >= 0");
|
|
134
140
|
}
|
|
135
|
-
for (const key of ["commandTimeoutMs", "enqueueTimeoutMs"]) {
|
|
136
|
-
|
|
137
|
-
if (value !== void 0 && !isTimerDelayMs(value)) {
|
|
138
|
-
invalid(`${key} must be a whole number from 1 to ${String(MAX_TIMER_DELAY_MS)}`);
|
|
139
|
-
}
|
|
141
|
+
for (const key of ["commandTimeoutMs", "enqueueTimeoutMs", "connectionReadyTimeoutMs"]) {
|
|
142
|
+
checkTimerDelay(key, opts[key]);
|
|
140
143
|
}
|
|
141
144
|
}
|
|
142
145
|
|
|
@@ -155,6 +158,7 @@ var DEFAULT_ENQUEUE_TIMEOUT_MS = 1e4;
|
|
|
155
158
|
var MAX_JOBS_PAGE_SIZE = 1e3;
|
|
156
159
|
var DEFAULT_METRICS_CACHE_TTL_MS = 5e3;
|
|
157
160
|
var DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS = 3e4;
|
|
161
|
+
var MAX_BULK_SIZE = 1e3;
|
|
158
162
|
|
|
159
163
|
// src/server/config/resolved-options.ts
|
|
160
164
|
function applyDefaults(opts) {
|
|
@@ -315,10 +319,14 @@ var ConnectionResolver = class {
|
|
|
315
319
|
const client = this.getClient();
|
|
316
320
|
if (!WAITING_STATUSES.has(client.status)) return Promise.resolve();
|
|
317
321
|
__privateGet(this, _pendingReady) ?? __privateSet(this, _pendingReady, new Promise((resolve) => {
|
|
318
|
-
|
|
322
|
+
const release = () => {
|
|
323
|
+
client.off("ready", release);
|
|
324
|
+
client.off("end", release);
|
|
319
325
|
__privateSet(this, _pendingReady, void 0);
|
|
320
326
|
resolve();
|
|
321
|
-
}
|
|
327
|
+
};
|
|
328
|
+
client.on("ready", release);
|
|
329
|
+
client.on("end", release);
|
|
322
330
|
}));
|
|
323
331
|
return __privateGet(this, _pendingReady);
|
|
324
332
|
}
|
|
@@ -451,6 +459,13 @@ function attachDefaultErrorListener(emitter, logger, kind, queueName) {
|
|
|
451
459
|
logger.error(`${kind} for queue "${queueName}" emitted an error: ${describeError(err)}`);
|
|
452
460
|
});
|
|
453
461
|
}
|
|
462
|
+
function assertPageWindow(operation, start, end) {
|
|
463
|
+
if (!Number.isInteger(start) || !Number.isInteger(end) || start < 0 || end < start || end - start >= MAX_JOBS_PAGE_SIZE) {
|
|
464
|
+
throw new QueueException(QUEUE_ERROR_CODES.INVALID_OPTIONS, HttpStatus.BAD_REQUEST, {
|
|
465
|
+
reason: `${operation} needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
|
|
466
|
+
});
|
|
467
|
+
}
|
|
468
|
+
}
|
|
454
469
|
|
|
455
470
|
// src/server/utils/with-deadline.ts
|
|
456
471
|
function withDeadline(work, timeoutMs, onTimeout) {
|
|
@@ -498,7 +513,6 @@ async function withCommandTimeoutCode(operation, commandTimeoutMs, work) {
|
|
|
498
513
|
}
|
|
499
514
|
|
|
500
515
|
// src/server/services/queue.service.ts
|
|
501
|
-
var MAX_BULK_SIZE = 1e3;
|
|
502
516
|
function assertValidCron(repeat, queueOptions) {
|
|
503
517
|
if (queueOptions.settings?.repeatStrategy !== void 0) return;
|
|
504
518
|
try {
|
|
@@ -613,11 +627,7 @@ var QueueService = class {
|
|
|
613
627
|
return job ?? null;
|
|
614
628
|
}
|
|
615
629
|
async getJobs(queueName, status, start = 0, end = 50) {
|
|
616
|
-
|
|
617
|
-
throw new QueueException(QUEUE_ERROR_CODES.INVALID_OPTIONS, HttpStatus.BAD_REQUEST, {
|
|
618
|
-
reason: `getJobs needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
|
|
619
|
-
});
|
|
620
|
-
}
|
|
630
|
+
assertPageWindow("getJobs", start, end);
|
|
621
631
|
const jobs = await this.call(
|
|
622
632
|
"getJobs",
|
|
623
633
|
queueName,
|
|
@@ -661,6 +671,7 @@ var QueueService = class {
|
|
|
661
671
|
);
|
|
662
672
|
}
|
|
663
673
|
async getJobSchedulers(queueName, start = 0, end = 50, asc = true) {
|
|
674
|
+
assertPageWindow("getJobSchedulers", start, end);
|
|
664
675
|
return this.call(
|
|
665
676
|
"getJobSchedulers",
|
|
666
677
|
queueName,
|
|
@@ -701,13 +712,15 @@ QueueService = __decorateClass([
|
|
|
701
712
|
function errorCodeOf(err) {
|
|
702
713
|
return err.getResponse().error.code;
|
|
703
714
|
}
|
|
704
|
-
var _connection;
|
|
715
|
+
var _connection, _deadlineMs;
|
|
705
716
|
var FlowService = class {
|
|
706
|
-
constructor(connection, enabled, prefix, telemetry) {
|
|
717
|
+
constructor(connection, enabled, prefix, telemetry, enqueueTimeoutMs = DEFAULT_ENQUEUE_TIMEOUT_MS) {
|
|
707
718
|
__publicField(this, "logger", new Logger(FlowService.name));
|
|
708
719
|
__publicField(this, "producer");
|
|
709
720
|
__privateAdd(this, _connection);
|
|
721
|
+
__privateAdd(this, _deadlineMs);
|
|
710
722
|
__privateSet(this, _connection, connection);
|
|
723
|
+
__privateSet(this, _deadlineMs, enqueueTimeoutMs);
|
|
711
724
|
if (enabled) {
|
|
712
725
|
this.producer = new FlowProducer({
|
|
713
726
|
connection: connection.getClient(),
|
|
@@ -718,17 +731,24 @@ var FlowService = class {
|
|
|
718
731
|
}
|
|
719
732
|
}
|
|
720
733
|
async add(flow) {
|
|
721
|
-
return
|
|
722
|
-
"addFlow",
|
|
723
|
-
() => this.commandTimeout(),
|
|
724
|
-
this.requireProducer().add(flow)
|
|
725
|
-
);
|
|
734
|
+
return this.bounded("addFlow", this.requireProducer().add(flow));
|
|
726
735
|
}
|
|
727
736
|
async addBulk(flows) {
|
|
737
|
+
const producer = this.requireProducer();
|
|
738
|
+
if (flows.length > MAX_BULK_SIZE) {
|
|
739
|
+
throw new QueueException(
|
|
740
|
+
QUEUE_ERROR_CODES.BULK_ENQUEUE_FAILED,
|
|
741
|
+
HttpStatus.INTERNAL_SERVER_ERROR,
|
|
742
|
+
{ reason: "batch size exceeds limit", limit: MAX_BULK_SIZE, received: flows.length }
|
|
743
|
+
);
|
|
744
|
+
}
|
|
745
|
+
return this.bounded("addFlowBulk", producer.addBulk([...flows]));
|
|
746
|
+
}
|
|
747
|
+
bounded(operation, work) {
|
|
728
748
|
return withCommandTimeoutCode(
|
|
729
|
-
|
|
749
|
+
operation,
|
|
730
750
|
() => this.commandTimeout(),
|
|
731
|
-
this
|
|
751
|
+
withDeadline(work, __privateGet(this, _deadlineMs), () => timeoutException(operation, __privateGet(this, _deadlineMs)))
|
|
732
752
|
);
|
|
733
753
|
}
|
|
734
754
|
commandTimeout() {
|
|
@@ -750,6 +770,7 @@ var FlowService = class {
|
|
|
750
770
|
}
|
|
751
771
|
};
|
|
752
772
|
_connection = new WeakMap();
|
|
773
|
+
_deadlineMs = new WeakMap();
|
|
753
774
|
FlowService = __decorateClass([
|
|
754
775
|
Injectable()
|
|
755
776
|
], FlowService);
|
|
@@ -796,6 +817,30 @@ MetricsService = __decorateClass([
|
|
|
796
817
|
Injectable()
|
|
797
818
|
], MetricsService);
|
|
798
819
|
|
|
820
|
+
// src/server/utils/worker-option-problem.ts
|
|
821
|
+
var WORKER_TIMER_MIN_MS = 1e3;
|
|
822
|
+
function countProblem(name, value, min, max) {
|
|
823
|
+
if (!Number.isInteger(value)) return { reason: `${name} must be a whole number`, actual: value };
|
|
824
|
+
if (value < min) return { reason: `${name} must be >= ${String(min)}`, actual: value };
|
|
825
|
+
if (value > max) return { reason: `${name} must be <= ${String(max)}`, actual: value };
|
|
826
|
+
return void 0;
|
|
827
|
+
}
|
|
828
|
+
function timerProblem(name, value) {
|
|
829
|
+
if (value === void 0) return void 0;
|
|
830
|
+
if (Number.isInteger(value) && value >= WORKER_TIMER_MIN_MS && value <= MAX_TIMER_DELAY_MS) {
|
|
831
|
+
return void 0;
|
|
832
|
+
}
|
|
833
|
+
return {
|
|
834
|
+
reason: `${name} must be a whole number from ${String(WORKER_TIMER_MIN_MS)} to ${String(MAX_TIMER_DELAY_MS)}`,
|
|
835
|
+
actual: value
|
|
836
|
+
};
|
|
837
|
+
}
|
|
838
|
+
function workerOptionProblem(opts) {
|
|
839
|
+
if (opts === void 0) return void 0;
|
|
840
|
+
const { concurrency, limiter, lockDuration, stalledInterval } = opts;
|
|
841
|
+
return (concurrency === void 0 ? void 0 : countProblem("concurrency", concurrency, 1, MAX_WORKER_CONCURRENCY)) ?? (limiter === void 0 ? void 0 : countProblem("limiter.max", limiter.max, 1, Number.POSITIVE_INFINITY) ?? countProblem("limiter.duration", limiter.duration, 1, Number.POSITIVE_INFINITY)) ?? timerProblem("lockDuration", lockDuration) ?? timerProblem("stalledInterval", stalledInterval);
|
|
842
|
+
}
|
|
843
|
+
|
|
799
844
|
// src/server/utils/abandoned-workers.ts
|
|
800
845
|
var abandoned = /* @__PURE__ */ new WeakSet();
|
|
801
846
|
function abandonWorker(worker) {
|
|
@@ -919,51 +964,15 @@ var WorkerRegistry = class {
|
|
|
919
964
|
}
|
|
920
965
|
}
|
|
921
966
|
validateOptions(opts) {
|
|
922
|
-
|
|
923
|
-
|
|
924
|
-
|
|
925
|
-
|
|
926
|
-
|
|
927
|
-
|
|
928
|
-
|
|
929
|
-
actual: opts.concurrency
|
|
930
|
-
}
|
|
931
|
-
);
|
|
932
|
-
}
|
|
933
|
-
if (opts.concurrency > MAX_WORKER_CONCURRENCY) {
|
|
934
|
-
throw new QueueException(
|
|
935
|
-
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
936
|
-
HttpStatus.INTERNAL_SERVER_ERROR,
|
|
937
|
-
{
|
|
938
|
-
reason: `concurrency must be <= ${MAX_WORKER_CONCURRENCY.toString()}`,
|
|
939
|
-
actual: opts.concurrency
|
|
940
|
-
}
|
|
941
|
-
);
|
|
942
|
-
}
|
|
967
|
+
const problem = workerOptionProblem(opts);
|
|
968
|
+
if (problem !== void 0) {
|
|
969
|
+
throw new QueueException(
|
|
970
|
+
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
971
|
+
HttpStatus.INTERNAL_SERVER_ERROR,
|
|
972
|
+
problem
|
|
973
|
+
);
|
|
943
974
|
}
|
|
944
975
|
this.validateJobTimeout(opts);
|
|
945
|
-
if (opts?.limiter !== void 0) {
|
|
946
|
-
if (opts.limiter.max < 1) {
|
|
947
|
-
throw new QueueException(
|
|
948
|
-
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
949
|
-
HttpStatus.INTERNAL_SERVER_ERROR,
|
|
950
|
-
{
|
|
951
|
-
reason: "limiter.max must be >= 1",
|
|
952
|
-
actual: opts.limiter.max
|
|
953
|
-
}
|
|
954
|
-
);
|
|
955
|
-
}
|
|
956
|
-
if (opts.limiter.duration < 1) {
|
|
957
|
-
throw new QueueException(
|
|
958
|
-
QUEUE_ERROR_CODES.WORKER_REGISTRATION_FAILED,
|
|
959
|
-
HttpStatus.INTERNAL_SERVER_ERROR,
|
|
960
|
-
{
|
|
961
|
-
reason: "limiter.duration must be >= 1",
|
|
962
|
-
actual: opts.limiter.duration
|
|
963
|
-
}
|
|
964
|
-
);
|
|
965
|
-
}
|
|
966
|
-
}
|
|
967
976
|
}
|
|
968
977
|
validateJobTimeout(opts) {
|
|
969
978
|
if (opts?.jobTimeoutMs !== void 0 && !isTimerDelayMs(opts.jobTimeoutMs)) {
|
|
@@ -1385,7 +1394,13 @@ var BymaxQueueModule = class extends ConfigurableModuleClass {
|
|
|
1385
1394
|
{ provide: BYMAX_QUEUE_RESOLVED_OPTIONS, useValue: resolved },
|
|
1386
1395
|
{
|
|
1387
1396
|
provide: FlowService,
|
|
1388
|
-
useFactory: (conn) => new FlowService(
|
|
1397
|
+
useFactory: (conn) => new FlowService(
|
|
1398
|
+
conn,
|
|
1399
|
+
resolved.flows.enabled,
|
|
1400
|
+
resolved.prefix,
|
|
1401
|
+
resolved.telemetry,
|
|
1402
|
+
resolved.enqueueTimeoutMs
|
|
1403
|
+
),
|
|
1389
1404
|
inject: [ConnectionResolver]
|
|
1390
1405
|
},
|
|
1391
1406
|
{
|
|
@@ -1415,7 +1430,13 @@ var BymaxQueueModule = class extends ConfigurableModuleClass {
|
|
|
1415
1430
|
},
|
|
1416
1431
|
{
|
|
1417
1432
|
provide: FlowService,
|
|
1418
|
-
useFactory: (conn, resolved) => new FlowService(
|
|
1433
|
+
useFactory: (conn, resolved) => new FlowService(
|
|
1434
|
+
conn,
|
|
1435
|
+
resolved.flows.enabled,
|
|
1436
|
+
resolved.prefix,
|
|
1437
|
+
resolved.telemetry,
|
|
1438
|
+
resolved.enqueueTimeoutMs
|
|
1439
|
+
),
|
|
1419
1440
|
inject: [ConnectionResolver, BYMAX_QUEUE_RESOLVED_OPTIONS]
|
|
1420
1441
|
},
|
|
1421
1442
|
{
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@bymax-one/nest-queue",
|
|
3
|
-
"version": "1.2.
|
|
3
|
+
"version": "1.2.16",
|
|
4
4
|
"description": "NestJS dynamic module wrapping BullMQ — typed jobs, flows, job schedulers, deduplication, OpenTelemetry, graceful shutdown",
|
|
5
5
|
"author": "Bymax One <support@bymax.one>",
|
|
6
6
|
"license": "MIT",
|