@oneunit/redis 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +422 -0
- package/CHANGELOG.md +186 -0
- package/CONTRIBUTING.md +353 -0
- package/LICENSE +21 -0
- package/README.md +760 -2
- package/dist/client/check.d.ts +19 -0
- package/dist/client/check.d.ts.map +1 -0
- package/dist/client/check.js +44 -0
- package/dist/client/check.js.map +1 -0
- package/dist/client/client.d.ts +10 -0
- package/dist/client/client.d.ts.map +1 -0
- package/dist/client/client.js +25 -0
- package/dist/client/client.js.map +1 -0
- package/dist/client/events.d.ts +5 -0
- package/dist/client/events.d.ts.map +1 -0
- package/dist/client/events.js +66 -0
- package/dist/client/events.js.map +1 -0
- package/dist/client/index.d.ts +6 -0
- package/dist/client/index.d.ts.map +1 -0
- package/dist/client/index.js +5 -0
- package/dist/client/index.js.map +1 -0
- package/dist/client/shutdown.d.ts +4 -0
- package/dist/client/shutdown.d.ts.map +1 -0
- package/dist/client/shutdown.js +78 -0
- package/dist/client/shutdown.js.map +1 -0
- package/dist/index.d.ts +5 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +5 -0
- package/dist/index.js.map +1 -0
- package/dist/logger.d.ts +23 -0
- package/dist/logger.d.ts.map +1 -0
- package/dist/logger.js +120 -0
- package/dist/logger.js.map +1 -0
- package/dist/pipeline/builder.d.ts +111 -0
- package/dist/pipeline/builder.d.ts.map +1 -0
- package/dist/pipeline/builder.js +197 -0
- package/dist/pipeline/builder.js.map +1 -0
- package/dist/pipeline/index.d.ts +3 -0
- package/dist/pipeline/index.d.ts.map +1 -0
- package/dist/pipeline/index.js +2 -0
- package/dist/pipeline/index.js.map +1 -0
- package/dist/queue/events.d.ts +13 -0
- package/dist/queue/events.d.ts.map +1 -0
- package/dist/queue/events.js +109 -0
- package/dist/queue/events.js.map +1 -0
- package/dist/queue/index.d.ts +7 -0
- package/dist/queue/index.d.ts.map +1 -0
- package/dist/queue/index.js +4 -0
- package/dist/queue/index.js.map +1 -0
- package/dist/queue/queue.d.ts +13 -0
- package/dist/queue/queue.d.ts.map +1 -0
- package/dist/queue/queue.js +37 -0
- package/dist/queue/queue.js.map +1 -0
- package/dist/queue/worker.d.ts +15 -0
- package/dist/queue/worker.d.ts.map +1 -0
- package/dist/queue/worker.js +20 -0
- package/dist/queue/worker.js.map +1 -0
- package/examples/README.md +86 -0
- package/examples/_setup.js +143 -0
- package/examples/cache.js +111 -0
- package/examples/pipeline.js +161 -0
- package/examples/pubsub.js +101 -0
- package/examples/queue-worker.js +189 -0
- package/examples/session.js +145 -0
- package/examples/standalone.js +58 -0
- package/package.json +100 -4
- package/src/client/check.ts +69 -0
- package/src/client/client.ts +45 -0
- package/src/client/events.ts +101 -0
- package/src/client/index.ts +5 -0
- package/src/client/shutdown.ts +97 -0
- package/src/index.ts +4 -0
- package/src/logger.ts +159 -0
- package/src/pipeline/builder.ts +307 -0
- package/src/pipeline/index.ts +7 -0
- package/src/queue/events.ts +158 -0
- package/src/queue/index.ts +6 -0
- package/src/queue/queue.ts +60 -0
- package/src/queue/worker.ts +44 -0
package/README.md
CHANGED
|
@@ -1,3 +1,761 @@
|
|
|
1
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<strong>@oneunit/redis</strong>
|
|
3
|
+
<br/>
|
|
4
|
+
Redis client and BullMQ job queues for Node.js
|
|
5
|
+
</p>
|
|
2
6
|
|
|
3
|
-
|
|
7
|
+
<p align="center">
|
|
8
|
+
<a href="https://www.npmjs.com/package/@oneunit/redis"><img src="https://img.shields.io/npm/v/@oneunit/redis?color=0969da&label=npm" alt="npm version"></a>
|
|
9
|
+
<a href="https://github.com/mayank040902/oneunit/blob/master/packages/redis/LICENSE"><img src="https://img.shields.io/npm/l/@oneunit/redis?color=22863a" alt="license"></a>
|
|
10
|
+
<img src="https://img.shields.io/badge/node-%3E%3D20-417e38" alt="node version">
|
|
11
|
+
<img src="https://img.shields.io/badge/types-included-3178c6" alt="types included">
|
|
12
|
+
</p>
|
|
13
|
+
|
|
14
|
+
<p align="center">
|
|
15
|
+
Shared connections · Safe Redis URL handling · Credential redaction · Bounded health checks<br/>
|
|
16
|
+
Idempotent graceful shutdown · Production BullMQ defaults · Safe pipeline batching · Universal logger support
|
|
17
|
+
</p>
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
Production-ready Redis client wrapper and BullMQ job queue integration for Node.js. Works standalone or with any logger that implements `error`, `warn`, `info`, and `debug`.
|
|
22
|
+
|
|
23
|
+
> **Monorepo** — [github.com/mayank040902/oneunit](https://github.com/mayank040902/oneunit)
|
|
24
|
+
|
|
25
|
+
## Table of Contents
|
|
26
|
+
|
|
27
|
+
- [Highlights](#highlights)
|
|
28
|
+
- [Installation](#installation)
|
|
29
|
+
- [Quick Start](#quick-start)
|
|
30
|
+
- [Subpath Imports](#subpath-imports)
|
|
31
|
+
- [Core API Reference](#core-api-reference)
|
|
32
|
+
- [Client API (`@oneunit/redis/client` or `@oneunit/redis`)](#client-api-oneunitredisclient-or-oneunitredis)
|
|
33
|
+
- [`createClient`](#createclientoptions-logger)
|
|
34
|
+
- [`health`](#healthclient-options)
|
|
35
|
+
- [`shutdown`](#shutdownclient-logger)
|
|
36
|
+
- [`attachEvents`](#attacheventsclient-logger)
|
|
37
|
+
- [`redactError`](#redacterrorerror)
|
|
38
|
+
- [Queue API (`@oneunit/redis/queue` or `@oneunit/redis`)](#queue-api-oneunitredisqueue-or-oneunitredis)
|
|
39
|
+
- [`createQueue`](#createqueueconfig)
|
|
40
|
+
- [`createWorker`](#createworkerconfig)
|
|
41
|
+
- [`attachQueueEvents`](#attachqueueeventsconfig)
|
|
42
|
+
- [BullMQ Re-exports](#bullmq-re-exports)
|
|
43
|
+
- [Logger API (`@oneunit/redis`)](#logger-api-oneunitredis)
|
|
44
|
+
- [Adapters and Helpers](#adapters-and-helpers)
|
|
45
|
+
- [Automatic Pino Detection](#automatic-pino-detection)
|
|
46
|
+
- [Pipeline API (`@oneunit/redis/pipeline` or `@oneunit/redis`)](#pipeline-api-oneunitredispipeline-or-oneunitredis)
|
|
47
|
+
- [`runPipeline`](#runpipelineclient-steps-options)
|
|
48
|
+
- [`pipelineValues`](#pipelinevaluesresults)
|
|
49
|
+
- [Errors](#errors)
|
|
50
|
+
- [Common Patterns](#common-patterns)
|
|
51
|
+
- [1. Read-Through Caching with TTL](#1-read-through-caching-with-ttl)
|
|
52
|
+
- [2. Session Store with Sliding Expiration](#2-session-store-with-sliding-expiration)
|
|
53
|
+
- [3. Pub/Sub Messaging](#3-pubsub-messaging)
|
|
54
|
+
- [4. Background Job Processing with Retries & Cleanup](#4-background-job-processing-with-retries--cleanup)
|
|
55
|
+
- [5. Proper Teardown Order](#5-proper-teardown-order)
|
|
56
|
+
- [Runnable Examples](#runnable-examples)
|
|
57
|
+
- [Environment Variables](#environment-variables)
|
|
58
|
+
- [Development & Verification](#development--verification)
|
|
59
|
+
- [Architecture & Contributing](#architecture--contributing)
|
|
60
|
+
- [License](#license)
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Highlights
|
|
65
|
+
|
|
66
|
+
- **Shared Connection Architecture** — Default `maxRetriesPerRequest: null` allows a single `ioredis` client to be shared cleanly between standard Redis operations and BullMQ `Queue` / `Worker` instances.
|
|
67
|
+
- **Safe Redis URL Handling** — Passes URL positionally to `ioredis` or falls back to `process.env.REDIS_URL`. Prevents the silent fallback to `localhost:6379` caused by passing `{ url }` in an options object.
|
|
68
|
+
- **Credential Redaction** — `redactError` automatically strips sensitive plaintext passwords from `AUTH` and `HELLO` command failures across client events, `QueueEvents`, and `shutdown`.
|
|
69
|
+
- **Bounded Health Checks** — `health(client, { timeout: 1000 })` returns latency and status, preventing health probes from hanging indefinitely on disconnected sockets.
|
|
70
|
+
- **Robust Graceful Shutdown** — `shutdown(client)` issues an idempotent `QUIT` bounded by a 5-second deadline, falls back to forced `disconnect()` if unresponsive, and prevents crashes from duplicate `SIGINT` / `SIGTERM` signals.
|
|
71
|
+
- **Production BullMQ Presets** — `createQueue` defaults to exponential backoff (1s initial delay), 3 retry attempts, and automatic retention pruning (100 completed, 1,000 failed jobs).
|
|
72
|
+
- **Safe Pipeline Batching** — `runPipeline` sends many commands in one round trip and reports per-command failures explicitly, because a failed command inside a raw `client.pipeline()` resolves the batch and is otherwise invisible. Each step must queue exactly one command, enforced at runtime; errors are redacted and the batch is bounded by a timeout.
|
|
73
|
+
- **Prefix Inheritance** — `attachQueueEvents` automatically inherits the queue's custom prefix, preventing lost event subscriptions.
|
|
74
|
+
- **Universal Logger Adapter** — Duck-typed logger support for Console, Pino, Winston, etc. Automatically normalizes missing log levels and inverts argument ordering for Pino (`(bindings, message)` vs `(message, extra)`).
|
|
75
|
+
- **Subpath Exports** — Modular imports via `@oneunit/redis`, `@oneunit/redis/client`, `@oneunit/redis/queue`, and `@oneunit/redis/pipeline`.
|
|
76
|
+
- **Strict TypeScript Types** — Fully typed ESM package targeting Node.js 20+ with re-exported BullMQ and ioredis types.
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## Installation
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
npm install @oneunit/redis
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Requires **Node.js 20+**. Ships as pure ESM.
|
|
87
|
+
|
|
88
|
+
> [!NOTE]
|
|
89
|
+
> `ioredis` and `bullmq` are direct dependencies. You do not need to install them separately.
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Quick Start
|
|
94
|
+
|
|
95
|
+
```typescript
|
|
96
|
+
import {
|
|
97
|
+
createClient,
|
|
98
|
+
createQueue,
|
|
99
|
+
createWorker,
|
|
100
|
+
attachQueueEvents,
|
|
101
|
+
health,
|
|
102
|
+
shutdown,
|
|
103
|
+
} from "@oneunit/redis";
|
|
104
|
+
|
|
105
|
+
// 1. Initialize Redis client (reads REDIS_URL from env by default)
|
|
106
|
+
const redis = createClient({ url: process.env.REDIS_URL });
|
|
107
|
+
|
|
108
|
+
// 2. Perform regular Redis operations
|
|
109
|
+
await redis.set(
|
|
110
|
+
"user:session:123",
|
|
111
|
+
JSON.stringify({ id: 123, role: "admin" }),
|
|
112
|
+
"EX",
|
|
113
|
+
3600,
|
|
114
|
+
);
|
|
115
|
+
const session = await redis.get("user:session:123");
|
|
116
|
+
|
|
117
|
+
// 3. Create a BullMQ Queue using the shared Redis client
|
|
118
|
+
const emailQueue = createQueue({
|
|
119
|
+
name: "email",
|
|
120
|
+
connection: redis,
|
|
121
|
+
});
|
|
122
|
+
|
|
123
|
+
// 4. Attach event listeners for job observability
|
|
124
|
+
const queueEvents = attachQueueEvents({ queue: emailQueue });
|
|
125
|
+
|
|
126
|
+
// 5. Create a Worker to process background jobs
|
|
127
|
+
const worker = createWorker({
|
|
128
|
+
name: "email",
|
|
129
|
+
connection: redis,
|
|
130
|
+
concurrency: 5,
|
|
131
|
+
processor: async (job) => {
|
|
132
|
+
console.log(`Processing email job ${job.id}:`, job.data);
|
|
133
|
+
await job.updateProgress(100);
|
|
134
|
+
return { delivered: true };
|
|
135
|
+
},
|
|
136
|
+
});
|
|
137
|
+
|
|
138
|
+
// 6. Enqueue a job (inherits 3 attempts + exponential backoff)
|
|
139
|
+
await emailQueue.add("welcome", {
|
|
140
|
+
to: "developer@example.com",
|
|
141
|
+
template: "welcome",
|
|
142
|
+
});
|
|
143
|
+
|
|
144
|
+
// 7. Check connectivity
|
|
145
|
+
const status = await health(redis);
|
|
146
|
+
console.log(`Redis status: ${status.status} (${status.latency.value}ms)`);
|
|
147
|
+
|
|
148
|
+
// 8. Graceful teardown
|
|
149
|
+
async function closeApp() {
|
|
150
|
+
await queueEvents.close();
|
|
151
|
+
await worker.close();
|
|
152
|
+
await emailQueue.close();
|
|
153
|
+
await shutdown(redis);
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
process.on("SIGTERM", closeApp);
|
|
157
|
+
process.on("SIGINT", closeApp);
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## Subpath Imports
|
|
163
|
+
|
|
164
|
+
Import only what you need to optimize module loading and boundaries:
|
|
165
|
+
|
|
166
|
+
```typescript
|
|
167
|
+
// Everything (client, queue, worker, logger)
|
|
168
|
+
import { createClient, createQueue, createWorker } from "@oneunit/redis";
|
|
169
|
+
|
|
170
|
+
// Redis client only (zero BullMQ imports)
|
|
171
|
+
import {
|
|
172
|
+
createClient,
|
|
173
|
+
health,
|
|
174
|
+
shutdown,
|
|
175
|
+
attachEvents,
|
|
176
|
+
} from "@oneunit/redis/client";
|
|
177
|
+
|
|
178
|
+
// Queues & workers only
|
|
179
|
+
import {
|
|
180
|
+
createQueue,
|
|
181
|
+
createWorker,
|
|
182
|
+
attachQueueEvents,
|
|
183
|
+
} from "@oneunit/redis/queue";
|
|
184
|
+
|
|
185
|
+
// Command batching only
|
|
186
|
+
import { runPipeline, pipelineValues } from "@oneunit/redis/pipeline";
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
---
|
|
190
|
+
|
|
191
|
+
## Core API Reference
|
|
192
|
+
|
|
193
|
+
### Client API (`@oneunit/redis/client` or `@oneunit/redis`)
|
|
194
|
+
|
|
195
|
+
#### `createClient(options?, logger?)`
|
|
196
|
+
|
|
197
|
+
Creates and returns a standard `ioredis` `Redis` instance configured with safe defaults and event logging.
|
|
198
|
+
|
|
199
|
+
```typescript
|
|
200
|
+
function createClient(
|
|
201
|
+
options?: RedisClientOptions | string,
|
|
202
|
+
logger?: Logger,
|
|
203
|
+
): Redis;
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
You can pass either a connection URL string or an options object:
|
|
207
|
+
|
|
208
|
+
```typescript
|
|
209
|
+
// From environment variable REDIS_URL
|
|
210
|
+
const client1 = createClient();
|
|
211
|
+
|
|
212
|
+
// From explicit URL string
|
|
213
|
+
const client2 = createClient("redis://127.0.0.1:6379");
|
|
214
|
+
|
|
215
|
+
// From options object
|
|
216
|
+
const client3 = createClient({
|
|
217
|
+
url: "redis://127.0.0.1:6379",
|
|
218
|
+
tls: { rejectUnauthorized: false },
|
|
219
|
+
});
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
**Default Settings:**
|
|
223
|
+
|
|
224
|
+
| Option | Default | Rationale |
|
|
225
|
+
| :--------------------- | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
226
|
+
| `url` | `process.env.REDIS_URL` | ioredis constructor only parses URL from positional arguments; passing `{ url }` in options is silently ignored. `createClient` ensures URL is passed positionally. |
|
|
227
|
+
| `lazyConnect` | `true` | Opens no socket on instantiation. First command initiates connection, keeping module imports free of network side effects. |
|
|
228
|
+
| `maxRetriesPerRequest` | `null` | Required by BullMQ. BullMQ throws if this is not `null`. Defaulting to `null` allows the client to be shared with queues and workers. |
|
|
229
|
+
|
|
230
|
+
#### `health(client, options?)`
|
|
231
|
+
|
|
232
|
+
Performs an active `PING` bounded by a strict timeout to assess connectivity and measure round-trip latency.
|
|
233
|
+
|
|
234
|
+
```typescript
|
|
235
|
+
interface HealthOptions {
|
|
236
|
+
timeout?: number; // Milliseconds to wait before reporting down (default: 1000)
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
interface HealthResult {
|
|
240
|
+
status: "up" | "down";
|
|
241
|
+
latency: {
|
|
242
|
+
value: number;
|
|
243
|
+
unit: "ms";
|
|
244
|
+
};
|
|
245
|
+
error?: string;
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
async function health(
|
|
249
|
+
client: Redis,
|
|
250
|
+
options?: HealthOptions,
|
|
251
|
+
): Promise<HealthResult>;
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
```typescript
|
|
255
|
+
const result = await health(redis, { timeout: 1500 });
|
|
256
|
+
if (result.status === "down") {
|
|
257
|
+
console.error("Redis unreachable:", result.error);
|
|
258
|
+
}
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
> [!TIP]
|
|
262
|
+
> Why the timeout matters: When disconnected, `ioredis` queues commands indefinitely in its offline queue while reconnecting. Without a bounded timeout, an unbounded `PING` would hang forever instead of returning `"down"`.
|
|
263
|
+
|
|
264
|
+
#### `shutdown(client, logger?)`
|
|
265
|
+
|
|
266
|
+
Gracefully terminates a Redis connection with timeout protection and idempotency.
|
|
267
|
+
|
|
268
|
+
```typescript
|
|
269
|
+
async function shutdown(
|
|
270
|
+
client: Redis | null | undefined,
|
|
271
|
+
logger?: Logger,
|
|
272
|
+
): Promise<void>;
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
- Sends `QUIT` command to allow pending commands to finish.
|
|
276
|
+
- If the server does not acknowledge `QUIT` within **5,000ms**, forces the socket closed via `client.disconnect()`.
|
|
277
|
+
- Safe to call multiple times or bind across both `SIGINT` and `SIGTERM` without throwing `Connection is closed` errors.
|
|
278
|
+
- Automatically redacts credentials if a shutdown error is logged.
|
|
279
|
+
|
|
280
|
+
#### `attachEvents(client, logger?)`
|
|
281
|
+
|
|
282
|
+
Attaches listeners to the five core `ioredis` connection lifecycle events (`connect`, `ready`, `reconnecting`, `error`, `close`).
|
|
283
|
+
|
|
284
|
+
```typescript
|
|
285
|
+
function attachEvents(client: Redis, logger?: Logger): void;
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
- `createClient` calls this automatically.
|
|
289
|
+
- Guarded by an internal symbol (`Symbol.for("oneunit.redis.eventsAttached")`) so duplicate calls will not attach duplicate listeners or trigger `MaxListenersExceededWarning`.
|
|
290
|
+
- The `error` event listener is always registered, preventing unhandled `error` events from crashing the process even if no logger is provided.
|
|
291
|
+
|
|
292
|
+
#### `redactError(error)`
|
|
293
|
+
|
|
294
|
+
Sanitizes `ioredis` errors that include command payloads, replacing sensitive credentials with `"[redacted]"`.
|
|
295
|
+
|
|
296
|
+
```typescript
|
|
297
|
+
function redactError(error: unknown): unknown;
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
```typescript
|
|
301
|
+
try {
|
|
302
|
+
await client.auth("default", "secret_pass");
|
|
303
|
+
} catch (err) {
|
|
304
|
+
// Command arguments with plaintext password are safe to log
|
|
305
|
+
logger.error("Authentication failed", redactError(err));
|
|
306
|
+
}
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
---
|
|
310
|
+
|
|
311
|
+
### Queue API (`@oneunit/redis/queue` or `@oneunit/redis`)
|
|
312
|
+
|
|
313
|
+
#### `createQueue(config)`
|
|
314
|
+
|
|
315
|
+
Creates and returns a BullMQ `Queue` pre-configured with production-ready retry and retention defaults.
|
|
316
|
+
|
|
317
|
+
```typescript
|
|
318
|
+
interface QueueConfig {
|
|
319
|
+
name: string;
|
|
320
|
+
connection: Redis;
|
|
321
|
+
prefix?: string; // Default: "queue"
|
|
322
|
+
defaultJobOptions?: JobsOptions; // Overrides merged key-by-key
|
|
323
|
+
settings?: QueueOptions["settings"];
|
|
324
|
+
}
|
|
325
|
+
|
|
326
|
+
function createQueue(config: QueueConfig): Queue;
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
**Default Job Options:**
|
|
330
|
+
|
|
331
|
+
| Setting | Value | Behavior |
|
|
332
|
+
| :----------------- | :------------------------------------- | :-------------------------------------------------------------------- |
|
|
333
|
+
| `attempts` | `3` | Retries failed jobs up to 3 times before moving them to failed state. |
|
|
334
|
+
| `backoff` | `{ type: "exponential", delay: 1000 }` | Exponential delay (1s, 2s, 4s...) between retries. |
|
|
335
|
+
| `removeOnComplete` | `100` | Keeps the last 100 completed jobs in Redis for inspection. |
|
|
336
|
+
| `removeOnFail` | `1000` | Keeps the last 1,000 failed jobs for debugging. |
|
|
337
|
+
|
|
338
|
+
> [!NOTE]
|
|
339
|
+
> Options are merged key-by-key. Passing `{ attempts: 5 }` keeps the default exponential backoff and retention settings intact. Explicit `null` values (such as `removeOnComplete: null`) pass through to indicate "keep forever".
|
|
340
|
+
|
|
341
|
+
#### `createWorker(config)`
|
|
342
|
+
|
|
343
|
+
Creates and returns a BullMQ `Worker` instance to process jobs from a queue.
|
|
344
|
+
|
|
345
|
+
```typescript
|
|
346
|
+
interface WorkerConfig<T = unknown> {
|
|
347
|
+
name: string;
|
|
348
|
+
processor: Processor<T>;
|
|
349
|
+
connection: Redis;
|
|
350
|
+
prefix?: string; // Default: "queue"
|
|
351
|
+
concurrency?: number; // Default: 1 (BullMQ default)
|
|
352
|
+
limiter?: WorkerOptions["limiter"];
|
|
353
|
+
settings?: WorkerOptions["settings"];
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
function createWorker<T = unknown>(config: WorkerConfig<T>): Worker<T>;
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
```typescript
|
|
360
|
+
interface EmailPayload {
|
|
361
|
+
to: string;
|
|
362
|
+
body: string;
|
|
363
|
+
}
|
|
364
|
+
|
|
365
|
+
const worker = createWorker<EmailPayload>({
|
|
366
|
+
name: "email",
|
|
367
|
+
connection: redis,
|
|
368
|
+
concurrency: 10,
|
|
369
|
+
processor: async (job) => {
|
|
370
|
+
await sendMail(job.data.to, job.data.body);
|
|
371
|
+
},
|
|
372
|
+
});
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
#### `attachQueueEvents(config)`
|
|
376
|
+
|
|
377
|
+
Attaches a managed BullMQ `QueueEvents` listener to report job lifecycle transitions to a logger.
|
|
378
|
+
|
|
379
|
+
```typescript
|
|
380
|
+
interface QueueEventsConfig {
|
|
381
|
+
queue: Queue;
|
|
382
|
+
logger?: Logger;
|
|
383
|
+
prefix?: string; // Inherits queue prefix if omitted
|
|
384
|
+
connection?: QueueEventsOptions["connection"]; // Defaults to queue connection
|
|
385
|
+
}
|
|
386
|
+
|
|
387
|
+
function attachQueueEvents(config: QueueEventsConfig): QueueEvents;
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
- **Prefix Inheritance** — Automatically reads the prefix from `queue.opts.prefix`. Prevents issues where custom prefixes caused events to be silently dropped.
|
|
391
|
+
- **Lifecycle Logging** — Automatically logs `completed`, `failed` (with reason), `progress`, and `error` (with credential redaction).
|
|
392
|
+
- **Resilient Teardown** — Wraps BullMQ's `QueueEvents.close()` so it cannot throw when the background connection already failed during initialization, and so the duplicated connection is released. That duplicate is one the caller has no reference to, so a failure there would otherwise leave the process unable to exit. Errors unrelated to the connection being gone are still rethrown.
|
|
393
|
+
|
|
394
|
+
#### BullMQ Re-exports
|
|
395
|
+
|
|
396
|
+
For convenience and typing consistency, common BullMQ classes and types are re-exported:
|
|
397
|
+
|
|
398
|
+
- **Classes**: `Queue`, `Worker`, `QueueEvents`
|
|
399
|
+
- **Types**: `QueueOptions`, `JobsOptions`, `WorkerOptions`, `Job`, `Processor`, `QueueEventsOptions`
|
|
400
|
+
|
|
401
|
+
---
|
|
402
|
+
|
|
403
|
+
### Logger API (`@oneunit/redis`)
|
|
404
|
+
|
|
405
|
+
`@oneunit/redis` does not require any specific logging library. It accepts any object that implements the `Logger` interface:
|
|
406
|
+
|
|
407
|
+
```typescript
|
|
408
|
+
interface Logger {
|
|
409
|
+
error(message: unknown, extra?: unknown): void;
|
|
410
|
+
warn(message: unknown, extra?: unknown): void;
|
|
411
|
+
info(message: unknown, extra?: unknown): void;
|
|
412
|
+
debug(message: unknown, extra?: unknown): void;
|
|
413
|
+
child?(bindings?: Record<string, unknown>): Logger;
|
|
414
|
+
}
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
#### Adapters and Helpers
|
|
418
|
+
|
|
419
|
+
- `consoleLogger` — Built-in logger that routes to `console.error`, `console.warn`, `console.info`, and `console.debug`.
|
|
420
|
+
- `silentLogger` — Built-in no-op logger that suppresses all log output.
|
|
421
|
+
- `createLogger(input?)` — Completes a partial logger by routing missing levels to `info` or no-op, preventing `logger.info is not a function` runtime crashes.
|
|
422
|
+
- `normalizeLogger(logger?)` — Normalizes a caller-supplied logger via `createLogger` when provided, or returns `undefined` when absent so unconfigured loggers stay silent.
|
|
423
|
+
- `isLogger(value)` — Type guard that verifies if an unknown object implements logger functions.
|
|
424
|
+
|
|
425
|
+
#### Automatic Pino Detection
|
|
426
|
+
|
|
427
|
+
Pino signatures use `(bindings, message)`, whereas standard loggers use `(message, extra)`.
|
|
428
|
+
|
|
429
|
+
`@oneunit/redis` detects Pino instances by checking for `bindings()` and `levels` properties, and automatically swaps argument positions so that metadata is merged into the structured JSON record rather than becoming the message string.
|
|
430
|
+
|
|
431
|
+
```typescript
|
|
432
|
+
import pino from "pino";
|
|
433
|
+
import { createClient } from "@oneunit/redis";
|
|
434
|
+
|
|
435
|
+
const logger = pino();
|
|
436
|
+
const client = createClient({}, logger); // Pino format handled automatically
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
### Pipeline API (`@oneunit/redis/pipeline` or `@oneunit/redis`)
|
|
440
|
+
|
|
441
|
+
#### `runPipeline(client, steps, options?)`
|
|
442
|
+
|
|
443
|
+
Runs a batch of commands in a single round trip. Use it when you already have
|
|
444
|
+
several independent commands in hand; a pipeline cannot help when each command
|
|
445
|
+
depends on the previous one's result.
|
|
446
|
+
|
|
447
|
+
```typescript
|
|
448
|
+
const result = await runPipeline(client, [
|
|
449
|
+
{ label: "set:a", run: (p) => void p.set("a", "1") },
|
|
450
|
+
{ label: "set:b", run: (p) => void p.set("b", "2") },
|
|
451
|
+
{ label: "get:a", run: (p) => void p.get("a") },
|
|
452
|
+
]);
|
|
453
|
+
|
|
454
|
+
result.failed; // how many commands failed
|
|
455
|
+
result.durationMs; // wall-clock duration of EXEC
|
|
456
|
+
pipelineValues(result.results); // ["OK", "OK", "1"] — throws if any failed
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
**Each step must queue exactly one command.** Results are matched to steps by
|
|
460
|
+
position, so a step that queues two commands would shift every later value onto
|
|
461
|
+
the wrong label, and an `async` step queues nothing before `EXEC` is sent and
|
|
462
|
+
loses its command silently. Neither is a compile error, so `runPipeline` checks
|
|
463
|
+
ioredis's own queue length around every step and raises `PipelineStepError`
|
|
464
|
+
naming the offending label:
|
|
465
|
+
|
|
466
|
+
```typescript
|
|
467
|
+
// Throws: Redis pipeline step "seed" queued 2 commands, expected exactly 1
|
|
468
|
+
await runPipeline(client, [
|
|
469
|
+
{
|
|
470
|
+
label: "seed",
|
|
471
|
+
run: (p) => {
|
|
472
|
+
p.set("a", "1");
|
|
473
|
+
p.set("b", "2");
|
|
474
|
+
},
|
|
475
|
+
},
|
|
476
|
+
]);
|
|
477
|
+
|
|
478
|
+
// Also throws — and the commands after it are never sent
|
|
479
|
+
await runPipeline(client, [
|
|
480
|
+
{
|
|
481
|
+
label: "get",
|
|
482
|
+
run: async (p) => {
|
|
483
|
+
await something();
|
|
484
|
+
void p.get("a");
|
|
485
|
+
},
|
|
486
|
+
},
|
|
487
|
+
]);
|
|
488
|
+
```
|
|
489
|
+
|
|
490
|
+
Write two steps instead of one step with two commands, and do not make a step
|
|
491
|
+
`async` — awaiting inside `run` queues the command too late.
|
|
492
|
+
|
|
493
|
+
| Option | Default | Purpose |
|
|
494
|
+
| :------------- | :------ | :-------------------------------------------------------------------------------------------------------------- |
|
|
495
|
+
| `timeout` | `5000` | Milliseconds to wait for `EXEC` before raising `PipelineTimeoutError`. Invalid values fall back to the default. |
|
|
496
|
+
| `logger` | none | Logger for batch-level warnings and the debug summary. |
|
|
497
|
+
| `throwOnError` | `false` | Raise `PipelineCommandError` if any command failed. |
|
|
498
|
+
|
|
499
|
+
#### Why this wraps `client.pipeline()`
|
|
500
|
+
|
|
501
|
+
A failed command does **not** fail a pipeline. ioredis resolves `EXEC` and
|
|
502
|
+
reports the failure per command:
|
|
503
|
+
|
|
504
|
+
```typescript
|
|
505
|
+
const raw = await client.pipeline().incr("a-string-key").exec();
|
|
506
|
+
// [[Error: ERR value is not an integer, null]] — resolved, not rejected
|
|
507
|
+
```
|
|
508
|
+
|
|
509
|
+
Reading only the values (`raw.map(([, value]) => value)`) yields `null` and the
|
|
510
|
+
batch looks successful while a write was dropped. Here every result carries an
|
|
511
|
+
explicit `error`, labelled with the step that produced it.
|
|
512
|
+
|
|
513
|
+
Two further gaps are closed: `EXEC` is bounded by `timeout`, since ioredis parks
|
|
514
|
+
queued commands while reconnecting and would otherwise never settle; and command
|
|
515
|
+
errors are passed through `redactError` before reaching a logger or an exception,
|
|
516
|
+
because ioredis attaches the failing command's arguments and those are the
|
|
517
|
+
password for `AUTH`. That redaction covers a rejected `EXEC` as well as the
|
|
518
|
+
resolved per-command results.
|
|
519
|
+
|
|
520
|
+
#### `pipelineValues(results)`
|
|
521
|
+
|
|
522
|
+
Returns the successful values in order, throwing `PipelineCommandError` if any
|
|
523
|
+
command failed. Use `result.results` directly when a partial batch is expected
|
|
524
|
+
and worth handling.
|
|
525
|
+
|
|
526
|
+
#### Errors
|
|
527
|
+
|
|
528
|
+
- `PipelineTimeoutError` — the batch did not complete within `timeout`. Carries `steps`, the number of commands queued.
|
|
529
|
+
- `PipelineCommandError` — a command failed and `throwOnError` was set, or `pipelineValues` was called on a failed batch. Carries `results`. The `message` lists step labels only, never command arguments.
|
|
530
|
+
- `PipelineStepError` — a step did not queue exactly one command, so results would be paired with the wrong labels. Carries `label` and `queued`. Raised before `EXEC` is sent, so the batch is never executed.
|
|
531
|
+
|
|
532
|
+
---
|
|
533
|
+
|
|
534
|
+
## Common Patterns
|
|
535
|
+
|
|
536
|
+
### 1. Read-Through Caching with TTL
|
|
537
|
+
|
|
538
|
+
```typescript
|
|
539
|
+
import { createClient } from "@oneunit/redis";
|
|
540
|
+
|
|
541
|
+
const redis = createClient();
|
|
542
|
+
|
|
543
|
+
async function getCachedUser(userId: string) {
|
|
544
|
+
const cacheKey = `user:${userId}`;
|
|
545
|
+
|
|
546
|
+
// 1. Try reading from Redis cache
|
|
547
|
+
const cached = await redis.get(cacheKey);
|
|
548
|
+
if (cached) {
|
|
549
|
+
return JSON.parse(cached);
|
|
550
|
+
}
|
|
551
|
+
|
|
552
|
+
// 2. Fall back to primary database
|
|
553
|
+
const user = await db.users.findById(userId);
|
|
554
|
+
|
|
555
|
+
// 3. Cache with 1-hour expiration (3600 seconds)
|
|
556
|
+
if (user) {
|
|
557
|
+
await redis.set(cacheKey, JSON.stringify(user), "EX", 3600);
|
|
558
|
+
}
|
|
559
|
+
|
|
560
|
+
return user;
|
|
561
|
+
}
|
|
562
|
+
|
|
563
|
+
// Invalidation helper
|
|
564
|
+
async function invalidateUser(userId: string) {
|
|
565
|
+
await redis.del(`user:${userId}`);
|
|
566
|
+
}
|
|
567
|
+
```
|
|
568
|
+
|
|
569
|
+
### 2. Session Store with Sliding Expiration
|
|
570
|
+
|
|
571
|
+
```typescript
|
|
572
|
+
import { createClient } from "@oneunit/redis";
|
|
573
|
+
|
|
574
|
+
const redis = createClient();
|
|
575
|
+
const SESSION_TTL = 1800; // 30 minutes
|
|
576
|
+
|
|
577
|
+
async function touchSession(sessionId: string) {
|
|
578
|
+
// Reset the expiration timer without altering the session data
|
|
579
|
+
const updated = await redis.expire(`session:${sessionId}`, SESSION_TTL);
|
|
580
|
+
return updated === 1;
|
|
581
|
+
}
|
|
582
|
+
|
|
583
|
+
async function updateSession(sessionId: string, data: Record<string, unknown>) {
|
|
584
|
+
await redis.set(
|
|
585
|
+
`session:${sessionId}`,
|
|
586
|
+
JSON.stringify(data),
|
|
587
|
+
"EX",
|
|
588
|
+
SESSION_TTL,
|
|
589
|
+
);
|
|
590
|
+
}
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
### 3. Pub/Sub Messaging
|
|
594
|
+
|
|
595
|
+
> [!IMPORTANT]
|
|
596
|
+
> Redis connections in subscriber mode cannot issue standard commands. Use dedicated clients for publisher and subscriber.
|
|
597
|
+
|
|
598
|
+
```typescript
|
|
599
|
+
import { createClient } from "@oneunit/redis";
|
|
600
|
+
|
|
601
|
+
const publisher = createClient();
|
|
602
|
+
const subscriber = createClient();
|
|
603
|
+
|
|
604
|
+
// Listen on notifications channel
|
|
605
|
+
await subscriber.subscribe("notifications");
|
|
606
|
+
|
|
607
|
+
subscriber.on("message", (channel, message) => {
|
|
608
|
+
console.log(`Received message on ${channel}:`, JSON.parse(message));
|
|
609
|
+
});
|
|
610
|
+
|
|
611
|
+
// Publish from the publisher client
|
|
612
|
+
await publisher.publish(
|
|
613
|
+
"notifications",
|
|
614
|
+
JSON.stringify({ type: "ALERT", text: "System update available" }),
|
|
615
|
+
);
|
|
616
|
+
```
|
|
617
|
+
|
|
618
|
+
### 4. Background Job Processing with Retries & Cleanup
|
|
619
|
+
|
|
620
|
+
```typescript
|
|
621
|
+
import {
|
|
622
|
+
createClient,
|
|
623
|
+
createQueue,
|
|
624
|
+
createWorker,
|
|
625
|
+
shutdown,
|
|
626
|
+
} from "@oneunit/redis";
|
|
627
|
+
|
|
628
|
+
const redis = createClient();
|
|
629
|
+
|
|
630
|
+
const reportQueue = createQueue({
|
|
631
|
+
name: "reports",
|
|
632
|
+
connection: redis,
|
|
633
|
+
defaultJobOptions: {
|
|
634
|
+
attempts: 5, // Override to 5 retries
|
|
635
|
+
backoff: { type: "exponential", delay: 2000 },
|
|
636
|
+
},
|
|
637
|
+
});
|
|
638
|
+
|
|
639
|
+
const reportWorker = createWorker({
|
|
640
|
+
name: "reports",
|
|
641
|
+
connection: redis,
|
|
642
|
+
concurrency: 2,
|
|
643
|
+
processor: async (job) => {
|
|
644
|
+
console.log(
|
|
645
|
+
`Generating report ${job.data.reportId} (attempt ${job.attemptsMade + 1})`,
|
|
646
|
+
);
|
|
647
|
+
|
|
648
|
+
// Simulate generation
|
|
649
|
+
await generatePdfReport(job.data);
|
|
650
|
+
|
|
651
|
+
return {
|
|
652
|
+
fileUrl: `https://storage.example.com/reports/${job.data.reportId}.pdf`,
|
|
653
|
+
};
|
|
654
|
+
},
|
|
655
|
+
});
|
|
656
|
+
|
|
657
|
+
await reportQueue.add("monthly-summary", {
|
|
658
|
+
reportId: "rep-2026-10",
|
|
659
|
+
month: 10,
|
|
660
|
+
});
|
|
661
|
+
```
|
|
662
|
+
|
|
663
|
+
### 5. Proper Teardown Order
|
|
664
|
+
|
|
665
|
+
To prevent dropped jobs or connection errors during deployment and shutdown, tear down components in reverse order of creation:
|
|
666
|
+
|
|
667
|
+
```typescript
|
|
668
|
+
async function gracefulTeardown() {
|
|
669
|
+
console.log("Shutting down workers and queues...");
|
|
670
|
+
|
|
671
|
+
// 1. Close event listeners first
|
|
672
|
+
if (queueEvents) await queueEvents.close();
|
|
673
|
+
|
|
674
|
+
// 2. Stop workers to let active jobs finish
|
|
675
|
+
if (worker) await worker.close();
|
|
676
|
+
|
|
677
|
+
// 3. Close queues
|
|
678
|
+
if (queue) await queue.close();
|
|
679
|
+
|
|
680
|
+
// 4. Finally, disconnect the Redis client
|
|
681
|
+
await shutdown(redis);
|
|
682
|
+
|
|
683
|
+
console.log("Redis teardown complete.");
|
|
684
|
+
}
|
|
685
|
+
```
|
|
686
|
+
|
|
687
|
+
---
|
|
688
|
+
|
|
689
|
+
## Runnable Examples
|
|
690
|
+
|
|
691
|
+
The [`examples/`](./examples) directory contains working, standalone scripts demonstrating real-world patterns.
|
|
692
|
+
|
|
693
|
+
To run them, ensure a Redis instance is listening on `localhost:6379` (or specify `REDIS_URL`):
|
|
694
|
+
|
|
695
|
+
```bash
|
|
696
|
+
npm run example:standalone # Health check, SET/GET, atomic INCRBY, TTL
|
|
697
|
+
npm run example:cache # Read-through cache with TTL and SCAN-based invalidation
|
|
698
|
+
npm run example:session # Session lifecycle: create, update, sliding TTL, delete
|
|
699
|
+
npm run example:pubsub # Publish/subscribe across distinct client connections
|
|
700
|
+
npm run example:queue-worker # BullMQ producer, worker retries, backoff, and event logging
|
|
701
|
+
npm run example:pipeline # Batch many commands into one round trip with runPipeline
|
|
702
|
+
```
|
|
703
|
+
|
|
704
|
+
---
|
|
705
|
+
|
|
706
|
+
## Environment Variables
|
|
707
|
+
|
|
708
|
+
| Variable | Default | Description |
|
|
709
|
+
| :------------------- | :----------------------- | :------------------------------------------------------------------------------ |
|
|
710
|
+
| `REDIS_URL` | `redis://localhost:6379` | Default connection URL used when `url` option is not passed to `createClient`. |
|
|
711
|
+
| `REDIS_SILENT` | _unset_ | Set to `true` to silence connection-event logging when running example scripts. |
|
|
712
|
+
| `EXAMPLE_TIMEOUT_MS` | `5000` | Subscription wait bound used in pub/sub example. |
|
|
713
|
+
|
|
714
|
+
---
|
|
715
|
+
|
|
716
|
+
## Development & Verification
|
|
717
|
+
|
|
718
|
+
```bash
|
|
719
|
+
# Build TypeScript to dist/
|
|
720
|
+
npm run build
|
|
721
|
+
|
|
722
|
+
# Typecheck without emitting files
|
|
723
|
+
npm run typecheck
|
|
724
|
+
|
|
725
|
+
# Run test suite
|
|
726
|
+
npm test
|
|
727
|
+
|
|
728
|
+
# Run ESLint
|
|
729
|
+
npm run lint
|
|
730
|
+
|
|
731
|
+
# Comprehensive verification (build, typecheck, lint, test, pack check)
|
|
732
|
+
npm run verify
|
|
733
|
+
```
|
|
734
|
+
|
|
735
|
+
Two things about `npm test` that are easy to trip over:
|
|
736
|
+
|
|
737
|
+
- **Build first.** The suite imports `../dist/index.js`, not `src`, and `dist/`
|
|
738
|
+
is gitignored — so on a fresh checkout the tests cannot even load until
|
|
739
|
+
`npm run build` has run. `npm run verify` orders this correctly; the
|
|
740
|
+
individual scripts do not.
|
|
741
|
+
- **A Redis on `localhost:6379` gives real coverage.** The queue, worker,
|
|
742
|
+
pipeline, and performance tests need a live server and return early without
|
|
743
|
+
one, so the suite is green either way — but with no server those tests are
|
|
744
|
+
no-ops. CI starts a `redis` service container for exactly this reason. An
|
|
745
|
+
unguarded `await client.ping()` does not merely fail without a server, it
|
|
746
|
+
never settles, because `maxRetriesPerRequest` defaults to `null`; see
|
|
747
|
+
[CONTRIBUTING.md](./CONTRIBUTING.md).
|
|
748
|
+
|
|
749
|
+
---
|
|
750
|
+
|
|
751
|
+
## Architecture & Contributing
|
|
752
|
+
|
|
753
|
+
- For deep-dives into design decisions, timeout guarantees, and BullMQ interoperability, read [ARCHITECTURE.md](./ARCHITECTURE.md).
|
|
754
|
+
- To contribute or run the regression test suites, see [CONTRIBUTING.md](./CONTRIBUTING.md).
|
|
755
|
+
- To see recent changes and bug fixes, see [CHANGELOG.md](./CHANGELOG.md).
|
|
756
|
+
|
|
757
|
+
---
|
|
758
|
+
|
|
759
|
+
## License
|
|
760
|
+
|
|
761
|
+
[MIT](./LICENSE) © 2026 mayank
|