@oneunit/redis 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +422 -0
- package/CHANGELOG.md +186 -0
- package/CONTRIBUTING.md +353 -0
- package/LICENSE +21 -0
- package/README.md +760 -2
- package/dist/client/check.d.ts +19 -0
- package/dist/client/check.d.ts.map +1 -0
- package/dist/client/check.js +44 -0
- package/dist/client/check.js.map +1 -0
- package/dist/client/client.d.ts +10 -0
- package/dist/client/client.d.ts.map +1 -0
- package/dist/client/client.js +25 -0
- package/dist/client/client.js.map +1 -0
- package/dist/client/events.d.ts +5 -0
- package/dist/client/events.d.ts.map +1 -0
- package/dist/client/events.js +66 -0
- package/dist/client/events.js.map +1 -0
- package/dist/client/index.d.ts +6 -0
- package/dist/client/index.d.ts.map +1 -0
- package/dist/client/index.js +5 -0
- package/dist/client/index.js.map +1 -0
- package/dist/client/shutdown.d.ts +4 -0
- package/dist/client/shutdown.d.ts.map +1 -0
- package/dist/client/shutdown.js +78 -0
- package/dist/client/shutdown.js.map +1 -0
- package/dist/index.d.ts +5 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +5 -0
- package/dist/index.js.map +1 -0
- package/dist/logger.d.ts +23 -0
- package/dist/logger.d.ts.map +1 -0
- package/dist/logger.js +120 -0
- package/dist/logger.js.map +1 -0
- package/dist/pipeline/builder.d.ts +111 -0
- package/dist/pipeline/builder.d.ts.map +1 -0
- package/dist/pipeline/builder.js +197 -0
- package/dist/pipeline/builder.js.map +1 -0
- package/dist/pipeline/index.d.ts +3 -0
- package/dist/pipeline/index.d.ts.map +1 -0
- package/dist/pipeline/index.js +2 -0
- package/dist/pipeline/index.js.map +1 -0
- package/dist/queue/events.d.ts +13 -0
- package/dist/queue/events.d.ts.map +1 -0
- package/dist/queue/events.js +109 -0
- package/dist/queue/events.js.map +1 -0
- package/dist/queue/index.d.ts +7 -0
- package/dist/queue/index.d.ts.map +1 -0
- package/dist/queue/index.js +4 -0
- package/dist/queue/index.js.map +1 -0
- package/dist/queue/queue.d.ts +13 -0
- package/dist/queue/queue.d.ts.map +1 -0
- package/dist/queue/queue.js +37 -0
- package/dist/queue/queue.js.map +1 -0
- package/dist/queue/worker.d.ts +15 -0
- package/dist/queue/worker.d.ts.map +1 -0
- package/dist/queue/worker.js +20 -0
- package/dist/queue/worker.js.map +1 -0
- package/examples/README.md +86 -0
- package/examples/_setup.js +143 -0
- package/examples/cache.js +111 -0
- package/examples/pipeline.js +161 -0
- package/examples/pubsub.js +101 -0
- package/examples/queue-worker.js +189 -0
- package/examples/session.js +145 -0
- package/examples/standalone.js +58 -0
- package/package.json +100 -4
- package/src/client/check.ts +69 -0
- package/src/client/client.ts +45 -0
- package/src/client/events.ts +101 -0
- package/src/client/index.ts +5 -0
- package/src/client/shutdown.ts +97 -0
- package/src/index.ts +4 -0
- package/src/logger.ts +159 -0
- package/src/pipeline/builder.ts +307 -0
- package/src/pipeline/index.ts +7 -0
- package/src/queue/events.ts +158 -0
- package/src/queue/index.ts +6 -0
- package/src/queue/queue.ts +60 -0
- package/src/queue/worker.ts +44 -0
package/ARCHITECTURE.md
ADDED
|
@@ -0,0 +1,422 @@
|
|
|
1
|
+
# Architecture
|
|
2
|
+
|
|
3
|
+
`@oneunit/redis` is a thin, opinionated layer over two libraries: `ioredis` for
|
|
4
|
+
the connection and `bullmq` for job queues. It owns very little. Its job is to
|
|
5
|
+
apply defaults that every consumer would otherwise get wrong, keep the surface
|
|
6
|
+
small, and stay out of the way.
|
|
7
|
+
|
|
8
|
+
That thinness is deliberate. There is no connection pool, no cache, no
|
|
9
|
+
serializer, no retry policy of its own. Whatever `ioredis` and `bullmq` do is
|
|
10
|
+
what this package does, and where it differs you can read why below.
|
|
11
|
+
|
|
12
|
+
## Dependencies
|
|
13
|
+
|
|
14
|
+
| Dependency | Why it is a dependency rather than a peer |
|
|
15
|
+
| :--------- | :------------------------------------------------------------------------------------------------------------ |
|
|
16
|
+
| `ioredis` | `createClient` constructs one. Consumers use the returned instance directly and do not install it themselves. |
|
|
17
|
+
| `bullmq` | `createQueue`, `createWorker`, and `attachQueueEvents` construct BullMQ objects. |
|
|
18
|
+
|
|
19
|
+
There are **no peer dependencies**. The package has no opinion about your HTTP
|
|
20
|
+
framework, your logger, or your process manager. It also has no optional
|
|
21
|
+
dependency on `@oneunit/logger` — the logger is duck-typed and injected, which
|
|
22
|
+
is what lets this package be used in a worker with no logging library present.
|
|
23
|
+
|
|
24
|
+
## Module layout
|
|
25
|
+
|
|
26
|
+
```text
|
|
27
|
+
src/
|
|
28
|
+
index.ts Public surface. Re-exports the four subtrees.
|
|
29
|
+
logger.ts Logger interface, silent/console/normalize adapters, duck-typing helpers.
|
|
30
|
+
|
|
31
|
+
client/
|
|
32
|
+
index.ts Barrel for the client subtree.
|
|
33
|
+
client.ts createClient: ioredis instance, URL handling, and BullMQ-safe defaults.
|
|
34
|
+
events.ts attachEvents, redactError: maps ioredis events onto logger, sanitizes secrets.
|
|
35
|
+
check.ts health: bounded PING with latency measurement.
|
|
36
|
+
shutdown.ts shutdown: graceful QUIT, idempotent teardown, WeakSet forced disconnect tracking.
|
|
37
|
+
|
|
38
|
+
queue/
|
|
39
|
+
index.ts Barrel for the queue subtree.
|
|
40
|
+
queue.ts createQueue: BullMQ Queue with safe job defaults and per-key option merging.
|
|
41
|
+
worker.ts createWorker: BullMQ Worker with selective option forwarding.
|
|
42
|
+
events.ts attachQueueEvents: ManagedQueueEvents wired to logger with prefix inheritance.
|
|
43
|
+
|
|
44
|
+
pipeline/
|
|
45
|
+
index.ts Barrel for the pipeline subtree.
|
|
46
|
+
builder.ts runPipeline, pipelineValues: batched commands, positional enforcement, bounded timeout, error redaction.
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
### Subpath Exports & Modular Boundaries
|
|
50
|
+
|
|
51
|
+
The package configures granular subpath exports in `package.json` to allow consumers to import isolated components without loading unnecessary modules:
|
|
52
|
+
|
|
53
|
+
| Subpath | Target File | Purpose & Dependency Surface |
|
|
54
|
+
| :------------------------ | :-------------------------- | :------------------------------------------------------------------------ |
|
|
55
|
+
| `@oneunit/redis` | `./dist/index.js` | Full public surface (client, queues, workers, events, logger, pipeline). |
|
|
56
|
+
| `@oneunit/redis/client` | `./dist/client/index.js` | Redis client, health checks, shutdown, events. **Zero BullMQ imports**. |
|
|
57
|
+
| `@oneunit/redis/queue` | `./dist/queue/index.js` | BullMQ Queue, Worker, and QueueEvents. Accepts an injected Redis client. |
|
|
58
|
+
| `@oneunit/redis/pipeline` | `./dist/pipeline/index.js` | Pipeline execution (`runPipeline`, `pipelineValues`). Isolated batching. |
|
|
59
|
+
|
|
60
|
+
`client/` and `queue/` never import each other. The queue subtree knows nothing
|
|
61
|
+
about health checks or event logging; it takes a connection you hand it and
|
|
62
|
+
configures BullMQ. `client/` never imports BullMQ at all. This is what allows
|
|
63
|
+
the subpath imports (`@oneunit/redis/client`, `@oneunit/redis/queue`) to pull in
|
|
64
|
+
only half the dependency surface.
|
|
65
|
+
|
|
66
|
+
`pipeline/` is layered on top of `client/` and imports `redactError` from it. It
|
|
67
|
+
deliberately sits in its own subtree rather than under `client/`, so
|
|
68
|
+
`@oneunit/redis/pipeline` does not drag in the connection helpers, and
|
|
69
|
+
`@oneunit/redis/client` does not drag in the batching helper.
|
|
70
|
+
|
|
71
|
+
## The client
|
|
72
|
+
|
|
73
|
+
```mermaid
|
|
74
|
+
graph LR
|
|
75
|
+
createClient["createClient(options, logger?)"]
|
|
76
|
+
createClient --> attachEvents["attachEvents<br/>ioredis events to logger"]
|
|
77
|
+
createClient --> Redis["ioredis Redis instance"]
|
|
78
|
+
Redis --> health["health(client)"]
|
|
79
|
+
Redis --> shutdown["shutdown(client)"]
|
|
80
|
+
Redis --> Queue["createQueue / createWorker<br/>same instance"]
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
`createClient` returns a plain `ioredis` `Redis` instance. It is not wrapped,
|
|
84
|
+
proxied, or subclassed, so every ioredis method and command is available and
|
|
85
|
+
`bullmq` accepts the instance directly.
|
|
86
|
+
|
|
87
|
+
Three defaults carry the weight of this package.
|
|
88
|
+
|
|
89
|
+
**`url` is passed positionally.** ioredis only reads `url` from the first
|
|
90
|
+
constructor argument. An options object carrying `url` is silently ignored and
|
|
91
|
+
the client connects to `localhost:6379`. `createClient` therefore calls
|
|
92
|
+
`new Redis(url, options)` when a URL is present, which is also why `url` falls
|
|
93
|
+
back to `REDIS_URL` — otherwise a typo in the environment would fail quietly at
|
|
94
|
+
runtime instead of at construction.
|
|
95
|
+
|
|
96
|
+
For the same reason a URL string is accepted directly, as
|
|
97
|
+
`createClient("redis://host:port")`. That is `new Redis(url)`'s own signature and
|
|
98
|
+
the form people reach for first; destructuring a string as an options object
|
|
99
|
+
produces no `url`, so the client would connect to `localhost:6379` with nothing
|
|
100
|
+
to indicate the mistake.
|
|
101
|
+
|
|
102
|
+
**`maxRetriesPerRequest` defaults to `null`.** ioredis defaults it to `20`;
|
|
103
|
+
bullmq refuses any connection where it is set, with `Your redis options
|
|
104
|
+
maxRetriesPerRequest must be null`. Since `createQueue` and `createWorker` are
|
|
105
|
+
designed to take this same instance, the default has to satisfy bullmq or the
|
|
106
|
+
documented usage does not work. Consumers who want ioredis's own retry ceiling
|
|
107
|
+
can pass it, at the cost of bullmq refusing the client.
|
|
108
|
+
|
|
109
|
+
**`lazyConnect` defaults to `true`.** Constructing a client opens no socket. The
|
|
110
|
+
first command connects. This keeps module import side-effect free and lets a
|
|
111
|
+
process start without Redis being up yet.
|
|
112
|
+
|
|
113
|
+
### Event logging
|
|
114
|
+
|
|
115
|
+
`attachEvents` maps five ioredis events to the logger: `connect`, `ready`,
|
|
116
|
+
`reconnecting`, `error`, and `close`. Two details matter:
|
|
117
|
+
|
|
118
|
+
The `error` listener is **always registered**, even when no logger is passed.
|
|
119
|
+
ioredis emits `error` on a failed connection; an `EventEmitter` with no
|
|
120
|
+
`error` listener throws, so skipping registration would turn every transient
|
|
121
|
+
outage into a process crash.
|
|
122
|
+
|
|
123
|
+
Message and extra are passed as `(message, extra)`, matching the `Logger`
|
|
124
|
+
interface. The `logger` parameter is optional everywhere in this package, so
|
|
125
|
+
every call site uses optional chaining.
|
|
126
|
+
|
|
127
|
+
pino is the one exception. It takes `(bindings, message)` and merges the first
|
|
128
|
+
argument into the record, so those two positions are passed the other way round.
|
|
129
|
+
There is no way to tell the signatures apart from a log call, so the logger has
|
|
130
|
+
to be detected: a pino instance carries both a `bindings()` method and a
|
|
131
|
+
`levels` map, and `child()` loggers inherit them.
|
|
132
|
+
|
|
133
|
+
Detection deliberately avoids `child()`. `child?()` is part of this package's
|
|
134
|
+
own `Logger` interface, so every conforming logger has one and using it as the
|
|
135
|
+
signal swapped arguments for all of them, putting the message in the bindings
|
|
136
|
+
slot of every record. A caller who has a logger with `child()` that takes
|
|
137
|
+
`(message, extra)` gets the documented order.
|
|
138
|
+
|
|
139
|
+
## Health checks
|
|
140
|
+
|
|
141
|
+
```mermaid
|
|
142
|
+
sequenceDiagram
|
|
143
|
+
participant C as Caller
|
|
144
|
+
participant H as health
|
|
145
|
+
participant R as Redis
|
|
146
|
+
C->>H: health(client, { timeout })
|
|
147
|
+
H->>R: PING
|
|
148
|
+
alt PONG before timeout
|
|
149
|
+
R-->>H: PONG
|
|
150
|
+
H-->>C: { status: "up", latency }
|
|
151
|
+
else timeout or error
|
|
152
|
+
H-->>C: { status: "down", latency, error }
|
|
153
|
+
end
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
The timeout is not a nicety. ioredis queues commands while reconnecting, so a
|
|
157
|
+
`PING` against an unreachable server does not reject — it waits in the offline
|
|
158
|
+
queue indefinitely. Without a bound, `health()` never settles and a health
|
|
159
|
+
endpoint wired to it hangs instead of reporting `down`, which is the one
|
|
160
|
+
situation a health check exists for.
|
|
161
|
+
|
|
162
|
+
The deadline timer stays **ref'd**, and is cleared in a `finally`. An `unref`'d
|
|
163
|
+
timer looks tidier but is wrong here: if the timer is the only thing keeping the
|
|
164
|
+
event loop alive, Node exits before the promise settles and the caller never
|
|
165
|
+
learns the result. A non-finite or non-positive `timeout` falls back to the
|
|
166
|
+
default rather than being coerced to 1ms by `setTimeout`, which would also emit
|
|
167
|
+
a `TimeoutNaNWarning` per call.
|
|
168
|
+
|
|
169
|
+
## Graceful shutdown
|
|
170
|
+
|
|
171
|
+
`shutdown` issues `QUIT` and waits for pending replies. It is **idempotent**:
|
|
172
|
+
once ioredis reports status `end`, the connection is gone for good and `QUIT`
|
|
173
|
+
rejects with `Connection is closed.`, so a second call is a no-op. This matters
|
|
174
|
+
because applications routinely register a handler on both `SIGINT` and
|
|
175
|
+
`SIGTERM`, and the second signal should not produce an unhandled rejection
|
|
176
|
+
during teardown.
|
|
177
|
+
|
|
178
|
+
A connection that dies _during_ `QUIT` is also treated as disconnected, and
|
|
179
|
+
logged as a warning rather than rethrown. A genuine failure on a still-live
|
|
180
|
+
connection is logged and rethrown, so a broken shutdown is never silent.
|
|
181
|
+
|
|
182
|
+
`QUIT` is a queued command, so a client stuck in `reconnecting` parks it in the
|
|
183
|
+
offline queue and the promise never settles. After a 5s deadline the connection
|
|
184
|
+
is torn down with `disconnect()` so the retry loop stops and the process can
|
|
185
|
+
exit — the caller asked to disconnect, and it does, one way or another.
|
|
186
|
+
|
|
187
|
+
That forced close needs its own record. `disconnect()` only reaches ioredis's
|
|
188
|
+
`closeHandler` from a live connection; on a client in `reconnecting` the
|
|
189
|
+
connector has nothing to close, so the status never becomes `end` even though
|
|
190
|
+
the retry timer has been cleared and the connection can never come back.
|
|
191
|
+
Without remembering the forced close, a second `shutdown` would wait out the
|
|
192
|
+
full 5s deadline again for a connection that is already gone. The bookkeeping
|
|
193
|
+
lives in a `WeakSet` rather than on the client, so it does not appear in
|
|
194
|
+
`Object.keys`, in a serialised snapshot, or in BullMQ's own inspection.
|
|
195
|
+
|
|
196
|
+
## Queues
|
|
197
|
+
|
|
198
|
+
```mermaid
|
|
199
|
+
graph TD
|
|
200
|
+
C["Redis client"]
|
|
201
|
+
C --> Q["createQueue<br/>prefix, job defaults"]
|
|
202
|
+
C --> W["createWorker<br/>prefix, concurrency"]
|
|
203
|
+
C --> QE["attachQueueEvents<br/>inherits queue prefix"]
|
|
204
|
+
Q --> J["Job defaults: 3 attempts,<br/>exponential backoff, 100/1000 retention"]
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
All three take the same connection and default `prefix` to `"queue"`, so the
|
|
208
|
+
common case needs no prefix at all.
|
|
209
|
+
|
|
210
|
+
**Undefined keys are never forwarded.** bullmq merges its own defaults with
|
|
211
|
+
`Object.assign`, so a key present with the value `undefined` still overrides the
|
|
212
|
+
default. Passing `concurrency: undefined` when the caller omitted it therefore
|
|
213
|
+
tripped bullmq's setter with `concurrency must be a finite number greater than
|
|
214
|
+
0` and made `createWorker` unusable without an explicit concurrency. Both
|
|
215
|
+
factories now include a key only when the caller actually set it.
|
|
216
|
+
|
|
217
|
+
`createQueue` also merges `defaultJobOptions` over the package defaults, so
|
|
218
|
+
overriding `attempts` alone leaves the backoff and retention settings intact.
|
|
219
|
+
The merge is per key rather than a spread, for the same reason as above: a spread
|
|
220
|
+
writes `undefined` for every unset key, which erases the default instead of
|
|
221
|
+
falling through to it. That bit a config assembled by spreading another object,
|
|
222
|
+
`{ ...base, attempts: maybeUndefined }`, which silently lost the 3-attempt retry.
|
|
223
|
+
`null` is a deliberate value and passes through unchanged, since bullmq reads
|
|
224
|
+
`removeOnComplete: null` as "keep this job forever".
|
|
225
|
+
|
|
226
|
+
`attachQueueEvents` derives its prefix from `queue.opts.prefix` rather than
|
|
227
|
+
defaulting to a literal. bullmq keys every event stream on the prefix, so a
|
|
228
|
+
queue created with a custom prefix and a listener defaulting to `"queue"`
|
|
229
|
+
receives **no events at all**, with no error to explain why. An explicit
|
|
230
|
+
`prefix` in the config still wins.
|
|
231
|
+
|
|
232
|
+
Bullmq requires a dedicated blocking connection for `Worker` and `QueueEvents`;
|
|
233
|
+
it duplicates the client you pass for that purpose. Closing the queue or worker
|
|
234
|
+
closes the duplicate, not your client — that is what `shutdown` is for.
|
|
235
|
+
|
|
236
|
+
`QueueEvents` also needs its `close()` wrapped, and the reason is narrow but
|
|
237
|
+
important. BullMQ's own `close()` awaits the connection's `initializing` promise
|
|
238
|
+
before disconnecting; against a server that never came up, that promise rejects
|
|
239
|
+
and `close()` throws, leaving the duplicated client in its reconnect loop. Nothing
|
|
240
|
+
in the caller's scope holds a reference to that duplicate, so the process can
|
|
241
|
+
never exit. The wrapper drives the connection closed directly and rethrows only
|
|
242
|
+
errors that are *not* the connection already being gone.
|
|
243
|
+
|
|
244
|
+
Deciding "already gone" structurally rather than by message is not a style
|
|
245
|
+
preference. BullMQ introduced `ConnectionClosedError` for exactly this reason —
|
|
246
|
+
its own comment on the class says it exists so callers can use `instanceof`
|
|
247
|
+
"rather than fragile message-substring matching" — and the message is not stable:
|
|
248
|
+
some of BullMQ's own construction sites pass ioredis's `"Connection is closed."`,
|
|
249
|
+
some pass their own wording, and some pass nothing, in which case the class
|
|
250
|
+
default `"Connection is closed"` (no trailing period) applies. An exact string
|
|
251
|
+
comparison therefore rethrows precisely the failures the wrapper exists to
|
|
252
|
+
absorb. The string checks remain as a fallback, because `instanceof` cannot match
|
|
253
|
+
across two copies of `bullmq` in one tree.
|
|
254
|
+
|
|
255
|
+
## Pipelines
|
|
256
|
+
|
|
257
|
+
```mermaid
|
|
258
|
+
sequenceDiagram
|
|
259
|
+
participant C as Caller
|
|
260
|
+
participant P as runPipeline
|
|
261
|
+
participant I as ioredis Pipeline
|
|
262
|
+
participant R as Redis Server
|
|
263
|
+
|
|
264
|
+
loop For each step
|
|
265
|
+
P->>I: Record pipeline.length before step
|
|
266
|
+
P->>I: Execute step.run(pipeline)
|
|
267
|
+
P->>I: Verify pipeline.length grew by exactly 1
|
|
268
|
+
end
|
|
269
|
+
P->>R: EXEC (bounded by timeout)
|
|
270
|
+
alt Settled before timeout
|
|
271
|
+
R-->>P: Array of [error, result] tuples
|
|
272
|
+
P->>P: Map results to labels & redact errors
|
|
273
|
+
P-->>C: PipelineResult { results, durationMs, failed }
|
|
274
|
+
else Timed out
|
|
275
|
+
P-->>C: throw PipelineTimeoutError
|
|
276
|
+
end
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
`runPipeline` wraps `client.pipeline()`. It does not reimplement batching; the
|
|
280
|
+
only reason it exists is that ioredis's own pipeline API fails in ways that are
|
|
281
|
+
invisible at the call site.
|
|
282
|
+
|
|
283
|
+
**A failed command does not fail the batch.** `EXEC` resolves with a
|
|
284
|
+
`[error, null]` tuple for the command that failed and `null` for its value:
|
|
285
|
+
|
|
286
|
+
```ts
|
|
287
|
+
const [result] = await client.pipeline().incr("a-string-key").exec();
|
|
288
|
+
// [ [ Error: ERR value is not an integer or out of range, null ] ]
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
The common way to read a pipeline is `results.map(([, value]) => value)`, which
|
|
292
|
+
turns that into `[null]` — indistinguishable from a command that legitimately
|
|
293
|
+
returned null, and from a successful write that never landed. So `runPipeline`
|
|
294
|
+
returns one `PipelineStepResult` per command, each with a `label` the caller
|
|
295
|
+
chose and an explicit `error` field. The label is what makes a failure
|
|
296
|
+
identifiable: a bare `results[7]` says nothing about which of several hundred
|
|
297
|
+
commands broke.
|
|
298
|
+
|
|
299
|
+
**`EXEC` can hang forever.** ioredis parks queued commands while reconnecting
|
|
300
|
+
and does not flush them until the connection is back, so a pipeline against an
|
|
301
|
+
unreachable server never settles — verified directly, a 2.5s race elapsed with
|
|
302
|
+
the client still in `reconnecting` and the promise unresolved. This is the same
|
|
303
|
+
failure `health` and `shutdown` are bounded against, and `runPipeline` bounds it
|
|
304
|
+
the same way, with `PipelineTimeoutError` after `options.timeout` (default 5s).
|
|
305
|
+
|
|
306
|
+
**Command errors carry their arguments.** ioredis attaches the failing command
|
|
307
|
+
to its errors, and for `AUTH` those args are the password. A `results` array is
|
|
308
|
+
a natural thing to log wholesale, so redaction happens once, in `runPipeline`,
|
|
309
|
+
before the array is handed back — rather than being left to each call site to
|
|
310
|
+
remember. A rejected `exec()` takes the same path: it is a connection-level
|
|
311
|
+
failure rather than a per-command one, so it never reaches the tuple mapping
|
|
312
|
+
above, and passing it through unredacted would leave the module promising a
|
|
313
|
+
guarantee it did not keep on that path.
|
|
314
|
+
|
|
315
|
+
**One command per step.** Results are paired with `steps` by position, so the
|
|
316
|
+
queue has to grow by exactly one per step. `runPipeline` reads ioredis's own
|
|
317
|
+
queue length (`pipeline.length`, a getter over the internal queue) before and
|
|
318
|
+
after each step and raises `PipelineStepError` if it did not. This is not
|
|
319
|
+
defensive decoration — it is the only thing standing between a caller and a
|
|
320
|
+
wrong answer, because both ways of breaking the pairing are invisible:
|
|
321
|
+
|
|
322
|
+
- a step queuing two commands shifts every later result by one, so
|
|
323
|
+
`results[3].label` names one command and `results[3].value` is another
|
|
324
|
+
command's;
|
|
325
|
+
- an `async` step queues nothing synchronously, so `exec()` is sent first and
|
|
326
|
+
the command lands in a pipeline that has already gone out.
|
|
327
|
+
|
|
328
|
+
`PipelineStep.run` is typed `(pipeline) => void`, and TypeScript permits
|
|
329
|
+
returning any value from a `void` signature — including a promise — so neither
|
|
330
|
+
compiles as an error. The return value is inspected directly for that reason,
|
|
331
|
+
because an `async` step is the one failure a count alone would misreport as
|
|
332
|
+
"queued 0 commands" rather than naming the cause.
|
|
333
|
+
|
|
334
|
+
Two design decisions worth stating:
|
|
335
|
+
|
|
336
|
+
- **`runPipeline` takes steps, not commands.** A `{ label, run }` pair rather
|
|
337
|
+
than a pre-built command list, because the label and the command have to be
|
|
338
|
+
kept in step; positional pairing is exactly the failure mode the label exists
|
|
339
|
+
to prevent. A `run` that throws fails the whole batch rather than being
|
|
340
|
+
skipped, since dropping the command would shift every later result by one and
|
|
341
|
+
return a value against the wrong label.
|
|
342
|
+
- **`throwOnError` is off by default.** A partial batch is a legitimate outcome
|
|
343
|
+
for a batch of independent writes, and forcing every caller to opt into
|
|
344
|
+
strictness would make the common case noisier. `pipelineValues` is the strict
|
|
345
|
+
path, for when a caller wants values and cannot tolerate a silently failed
|
|
346
|
+
write.
|
|
347
|
+
|
|
348
|
+
## Trust boundaries
|
|
349
|
+
|
|
350
|
+
Redis is a shared, unauthenticated-by-default resource, and this package makes
|
|
351
|
+
no attempt to sandbox it. Specifically:
|
|
352
|
+
|
|
353
|
+
- **`connection` is trusted.** It is whatever the caller passes, and bullmq
|
|
354
|
+
drives it. The package never validates or rewrites connection internals
|
|
355
|
+
beyond the documented defaults.
|
|
356
|
+
- **`keyPrefix` is unsupported.** bullmq rejects an ioredis client configured
|
|
357
|
+
with `keyPrefix` outright (`ioredis does not support ioredis prefixes, use
|
|
358
|
+
the prefix option instead`). Use bullmq's `prefix` on the factories.
|
|
359
|
+
- **`url` is parsed by ioredis, not here.** Credentials in `REDIS_URL` are the
|
|
360
|
+
caller's to protect. Do not log the URL.
|
|
361
|
+
- **Every error that reaches the logger is redacted.** ioredis attaches the
|
|
362
|
+
failing command to its errors, and for `AUTH` (and `HELLO`) that command's args
|
|
363
|
+
are the password in plaintext. `redactError` replaces those args while keeping
|
|
364
|
+
the message, which is what tells an operator _why_ authentication failed.
|
|
365
|
+
All three error paths run it: the client `error` event, the `QueueEvents`
|
|
366
|
+
`error` event, and `shutdown`'s own failure report. `QueueEvents` matters most
|
|
367
|
+
because it duplicates the caller's client, so it authenticates with the same
|
|
368
|
+
password and repeats the failure on every reconnect attempt.
|
|
369
|
+
- **Queue names become Redis key names.** bullmq rejects a name containing
|
|
370
|
+
`:`, which is what prevents one queue from addressing another's keys. Names
|
|
371
|
+
are otherwise not sanitized, so treat them as trusted input.
|
|
372
|
+
- **The logger receives job data.** `attachQueueEvents` logs `failedReason` and
|
|
373
|
+
`data`. If job payloads carry secrets, either do not attach events or use a
|
|
374
|
+
logger that redacts.
|
|
375
|
+
|
|
376
|
+
## Testing
|
|
377
|
+
|
|
378
|
+
Tests use the Node built-in runner (`node:test`) via `tsx`. Cases that need a
|
|
379
|
+
live Redis probe for reachability first and return early when there is none, so
|
|
380
|
+
the suite passes offline; those tests no-op rather than fail when no server is
|
|
381
|
+
listening. That no-op is deliberate for a local run and a trap in CI — a green
|
|
382
|
+
build proves nothing about the queue, worker, or pipeline paths unless a server
|
|
383
|
+
is actually there — so the `verify` job starts a `redis` service container, as
|
|
384
|
+
does the release smoke test.
|
|
385
|
+
|
|
386
|
+
The guard is needed for a bare `client.ping()` too, not only for BullMQ objects.
|
|
387
|
+
`createClient` defaults `maxRetriesPerRequest` to `null` because BullMQ requires
|
|
388
|
+
it, and `null` means ioredis never gives up on a queued command — so an unguarded
|
|
389
|
+
`await client.ping()` against an absent server parks in the offline queue and
|
|
390
|
+
never settles, hanging the run rather than failing it. It lives in one place,
|
|
391
|
+
`test/helpers.ts`, because it used to be copied per file and the copies drifted.
|
|
392
|
+
|
|
393
|
+
The runner executes test files in parallel child processes and every one of them
|
|
394
|
+
imports `../dist/index.js`, so a test that rebuilds `dist/` takes it away from its
|
|
395
|
+
siblings mid-run. That is why the packaging test uses
|
|
396
|
+
`npm pack --dry-run --ignore-scripts`: without the flag, `prepack` runs `build`,
|
|
397
|
+
whose `clean` step deletes `dist/`.
|
|
398
|
+
|
|
399
|
+
## Packaging
|
|
400
|
+
|
|
401
|
+
`files` lists `src` as well as `dist`. The compiler emits `.js.map` and
|
|
402
|
+
`.d.ts.map` that reference `../src/*.ts`, so shipping `dist` alone leaves every
|
|
403
|
+
map pointing at a file the consumer does not have: stack traces fall back to
|
|
404
|
+
compiled JavaScript and editor go-to-definition does nothing, with no warning.
|
|
405
|
+
A CI step resolves each map's `sources` against the installed package and fails
|
|
406
|
+
if any target is missing.
|
|
407
|
+
|
|
408
|
+
`npm run build` removes `dist` and `tsconfig.tsbuildinfo` first. `tsc` never
|
|
409
|
+
deletes output for a source file that has been removed, so a deleted module's
|
|
410
|
+
compiled copy would otherwise stay in the tarball indefinitely — and one was
|
|
411
|
+
published this way before the clean step existed.
|
|
412
|
+
|
|
413
|
+
Regression tests here are written to **fail against the bug they cover**. That
|
|
414
|
+
is verified by reintroducing the defect into the compiled output and confirming
|
|
415
|
+
the suite goes red, because a test that passes against broken code protects
|
|
416
|
+
nothing. Two examples worth knowing about:
|
|
417
|
+
|
|
418
|
+
- The missing-logger test asserts on `console.error`, because bullmq's `emit`
|
|
419
|
+
catches a throwing listener and retries the event as `error`, which swallows
|
|
420
|
+
the throw. A plain "does not throw" assertion passes against the bug.
|
|
421
|
+
- The health-timeout test runs in a child process, because the failure mode is
|
|
422
|
+
Node exiting before the promise settles. In-process it is invisible.
|
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented in this file.
|
|
4
|
+
|
|
5
|
+
## Unreleased
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- `runPipeline(client, steps, options?)` batches many commands into a single
|
|
10
|
+
round trip. A failed command does not fail a pipeline — ioredis resolves
|
|
11
|
+
`EXEC` and reports the failure per command — so reading only the values from
|
|
12
|
+
a raw `client.pipeline()` yields `null` for a write that never landed, with
|
|
13
|
+
nothing to distinguish it from a successful one. `runPipeline` returns one
|
|
14
|
+
labelled result per command with an explicit `error`, bounded by a
|
|
15
|
+
`timeout` (default 5s, since ioredis parks queued commands while reconnecting
|
|
16
|
+
and `EXEC` would otherwise never settle), with command errors passed through
|
|
17
|
+
`redactError` — including on a rejected `EXEC`, which skips the per-command
|
|
18
|
+
mapping and would otherwise reach a logger carrying the failing command's
|
|
19
|
+
arguments.
|
|
20
|
+
|
|
21
|
+
**Each step must queue exactly one command.** Results are matched to steps by
|
|
22
|
+
position, so a step queuing two commands reports every later value against the
|
|
23
|
+
wrong label, and an `async` step queues nothing before `EXEC` is sent and loses
|
|
24
|
+
its command silently. Neither is a compile error: `PipelineStep.run` is typed as
|
|
25
|
+
returning `void`, and TypeScript permits returning any value from a `void`
|
|
26
|
+
signature, so both type-check cleanly. `runPipeline` reads ioredis's queue
|
|
27
|
+
length around each step and raises `PipelineStepError`, naming the label,
|
|
28
|
+
before anything is sent. Write one step per command, and do not make a step
|
|
29
|
+
`async`.
|
|
30
|
+
- `pipelineValues(results)` returns the successful values in order, throwing
|
|
31
|
+
`PipelineCommandError` rather than handing back a sparse array that looks
|
|
32
|
+
complete.
|
|
33
|
+
- `PipelineTimeoutError`, `PipelineCommandError`, and `PipelineStepError`. The
|
|
34
|
+
first two messages list step labels only, never command arguments.
|
|
35
|
+
- `throwOnError` option on `runPipeline` for batches where a partial write is not
|
|
36
|
+
acceptable.
|
|
37
|
+
- `@oneunit/redis/pipeline` subpath export.
|
|
38
|
+
- `examples/pipeline.js` and `npm run example:pipeline`.
|
|
39
|
+
|
|
40
|
+
### Fixed
|
|
41
|
+
|
|
42
|
+
- `attachQueueEvents().close()` could throw instead of closing cleanly. It
|
|
43
|
+
decides whether a failed close is benign by comparing the error message
|
|
44
|
+
against `"Connection is closed."`, but BullMQ now raises a
|
|
45
|
+
`ConnectionClosedError` whose default message has no trailing period, and
|
|
46
|
+
several of its own code paths pass unrelated wording. Any of those variants
|
|
47
|
+
was rethrown, which is the one outcome the `close()` wrapper exists to
|
|
48
|
+
prevent — and the duplicated connection it could leave behind is one the
|
|
49
|
+
caller has no reference to. The class is now checked structurally; the
|
|
50
|
+
message comparison remains as a fallback.
|
|
51
|
+
|
|
52
|
+
## 1.0.0 - 2026-10-01
|
|
53
|
+
|
|
54
|
+
First release under the `@oneunit` name. The `1.0.0` published before it, under
|
|
55
|
+
the `@bootstrap-framework` name, is listed under Previous releases.
|
|
56
|
+
|
|
57
|
+
### Fixed
|
|
58
|
+
|
|
59
|
+
- `createClient` passed `url` inside the options object, which ioredis ignores,
|
|
60
|
+
so every client silently connected to `localhost:6379`. The URL is now passed
|
|
61
|
+
positionally and falls back to `REDIS_URL`, matching the documented default.
|
|
62
|
+
- `createClient` called `connect(url)` when `lazyConnect` was `false`. ioredis
|
|
63
|
+
connects on its own in that mode and `connect()` takes no URL, so the call
|
|
64
|
+
rejected with `Redis is already connecting/connected` and left an unhandled
|
|
65
|
+
rejection.
|
|
66
|
+
- `createClient` left `maxRetriesPerRequest` at the ioredis default of `20`.
|
|
67
|
+
BullMQ rejects any connection where it is set, so `createWorker` and
|
|
68
|
+
`attachQueueEvents` could not accept a client from `createClient`. It now
|
|
69
|
+
defaults to `null` and remains overridable.
|
|
70
|
+
- `health` had no timeout. ioredis queues commands while reconnecting, so a
|
|
71
|
+
`PING` against an unreachable server never settled and health checks hung
|
|
72
|
+
instead of reporting `down`. Now bounded by `options.timeout` (default
|
|
73
|
+
1000ms).
|
|
74
|
+
- `createWorker` forwarded `concurrency: undefined` when the caller omitted it.
|
|
75
|
+
BullMQ only applies its own default for absent keys, so its setter rejected
|
|
76
|
+
the value with `concurrency must be a finite number greater than 0` and the
|
|
77
|
+
worker could not be created at all. `limiter` and `settings` had the same
|
|
78
|
+
latent problem; `createQueue` had it for `settings`.
|
|
79
|
+
- `shutdown` threw `Connection is closed.` when called on an already-ended
|
|
80
|
+
client. Applications commonly handle both `SIGINT` and `SIGTERM`, so the
|
|
81
|
+
second signal produced an unhandled rejection during teardown. It is now
|
|
82
|
+
idempotent, and treats a connection that dies mid-`QUIT` as disconnected.
|
|
83
|
+
- `attachQueueEvents` defaulted `prefix` to the literal `"queue"` instead of the
|
|
84
|
+
queue's own prefix. BullMQ keys event streams on the prefix, so a queue
|
|
85
|
+
created with a custom prefix received no events and reported no error. The
|
|
86
|
+
queue's prefix is now inherited, with an explicit `prefix` still winning.
|
|
87
|
+
- `attachQueueEvents` required a `logger` and threw on the first event when it
|
|
88
|
+
was omitted, unlike every other export in the package. BullMQ swallows the
|
|
89
|
+
listener exception and re-emits it as `error`, so it surfaced as unexplained
|
|
90
|
+
`console.error` output. `logger` is now optional.
|
|
91
|
+
- `health` passed a negative or `NaN` timeout straight to `setTimeout`, which
|
|
92
|
+
coerced it to 1ms and emitted a `TimeoutNaNWarning` or
|
|
93
|
+
`TimeoutNegativeWarning` per call. Invalid values now fall back to the default.
|
|
94
|
+
- The client `error` event logged its arguments in pino order while every other
|
|
95
|
+
call site used `(message, extra)`.
|
|
96
|
+
- Any logger exposing a `child()` method had its arguments swapped into pino's
|
|
97
|
+
`(bindings, message)` order. `child?()` is part of this package's own `Logger`
|
|
98
|
+
interface, so every conforming logger qualified: each record put the message
|
|
99
|
+
in the bindings slot. Loggers are now detected by the `bindings()` method and
|
|
100
|
+
`levels` map a pino instance actually carries, which `child()` loggers inherit.
|
|
101
|
+
- `shutdown` could not tell that it had already forced a stalled connection
|
|
102
|
+
closed. `disconnect()` never runs ioredis's `closeHandler` from a client in
|
|
103
|
+
`reconnecting`, so the status stayed `reconnecting` and a second call waited
|
|
104
|
+
out the full 5s QUIT deadline again for a connection that was already gone.
|
|
105
|
+
Applications handling both `SIGINT` and `SIGTERM` paid that twice.
|
|
106
|
+
- `attachQueueEvents` and `shutdown` logged errors unredacted, while the client
|
|
107
|
+
event listener already ran them through `redactError`. `QueueEvents` duplicates
|
|
108
|
+
the caller's client and therefore authenticates with the same password, and
|
|
109
|
+
BullMQ re-emits the AUTH failure on its emitter; ioredis attaches the failing
|
|
110
|
+
command to that error, whose `AUTH` args are the password in plaintext. Both
|
|
111
|
+
call sites now redact before logging, so a misconfigured or wrong-password
|
|
112
|
+
server no longer writes the credential to the application's log.
|
|
113
|
+
- `createClient("redis://host:port")` ignored the string. ioredis's own
|
|
114
|
+
constructor accepts that form, but destructuring it as an options object
|
|
115
|
+
yielded no `url`, so the client silently connected to `localhost:6379` — the
|
|
116
|
+
wrong server, with no error to notice it by. A string is now accepted and
|
|
117
|
+
treated as the URL.
|
|
118
|
+
- `createQueue` spread `defaultJobOptions` over its defaults unconditionally, so
|
|
119
|
+
a key the caller left `undefined` erased the default instead of falling through
|
|
120
|
+
to it. A config assembled by spreading another object
|
|
121
|
+
(`{ ...base, attempts: maybeUndefined }`) silently lost the 3-attempt retry and
|
|
122
|
+
the exponential backoff. The merge is now per key, and `null` still passes
|
|
123
|
+
through, since BullMQ reads `removeOnComplete: null` as "keep the job".
|
|
124
|
+
- `examples/queue-worker.js` called `shutdown` without importing it, throwing a
|
|
125
|
+
`ReferenceError` on every Ctrl+C.
|
|
126
|
+
- `npm run lint` failed on four errors: a `@ts-ignore` that should have been a
|
|
127
|
+
typed import, two `any` return types, and an unused import.
|
|
128
|
+
- The published tarball shipped sourcemaps pointing at a `src/` directory it did
|
|
129
|
+
not include, so consumer stack traces and editor navigation broke silently.
|
|
130
|
+
`src` is now in `files`, matching the auth and logger packages.
|
|
131
|
+
- `npm run build` did not clear `dist`, so output from a deleted source file was
|
|
132
|
+
published forever. The build now cleans `dist` first.
|
|
133
|
+
- `@oneunit/redis` was imported as `@bootstrap-framework/redis` by the server
|
|
134
|
+
package's redis plugin and declared under the retired name in its
|
|
135
|
+
`package.json`, so `pnpm install --frozen-lockfile` could not resolve it and
|
|
136
|
+
the plugin reported itself disabled instead of naming the actual fault.
|
|
137
|
+
|
|
138
|
+
### Changed
|
|
139
|
+
|
|
140
|
+
- `createClient` returns a typed `ioredis` `Redis` instance instead of `any`.
|
|
141
|
+
- `createClient` accepts a URL string as its first argument, matching
|
|
142
|
+
`new Redis(url)`.
|
|
143
|
+
- `health` accepts an options object and exports `HealthOptions`.
|
|
144
|
+
- `attachQueueEvents` accepts an optional `logger`.
|
|
145
|
+
- Published as `@oneunit/redis`, renamed from `@bootstrap-framework/redis`.
|
|
146
|
+
- The tarball now ships `src`, so the sourcemaps its own `dist` output points at
|
|
147
|
+
resolve in an installed package.
|
|
148
|
+
|
|
149
|
+
### Added
|
|
150
|
+
|
|
151
|
+
- `ARCHITECTURE.md` covering module layout, the reasoning behind each default,
|
|
152
|
+
and the trust boundaries around connections, queue names, and logged job data.
|
|
153
|
+
- `CONTRIBUTING.md` covering the regression-test rule, the bullmq and ioredis
|
|
154
|
+
default traps in this package, and behaviours that look like bugs but are not.
|
|
155
|
+
- Regression tests for every fix above, each verified to fail against the bug it
|
|
156
|
+
covers.
|
|
157
|
+
- Tests that need Redis probe for reachability and no-op when none is
|
|
158
|
+
listening, so the suite passes offline.
|
|
159
|
+
- A `verify` script running build, typecheck, lint, tests, and `pack:check` in
|
|
160
|
+
order.
|
|
161
|
+
- A CI workflow that gates a release on a Node 20/22/24 verify matrix and a
|
|
162
|
+
consumer smoke test that installs the real tarball, exercises both
|
|
163
|
+
`createClient` argument forms, runs a job through a worker, asserts no
|
|
164
|
+
credential reaches the logger, and checks that sourcemap targets and subpath
|
|
165
|
+
exports resolve from the installed package.
|
|
166
|
+
|
|
167
|
+
## Previous releases
|
|
168
|
+
|
|
169
|
+
Released as `@bootstrap-framework/redis`.
|
|
170
|
+
|
|
171
|
+
### 1.0.0 - 2026-09-26
|
|
172
|
+
|
|
173
|
+
#### Added
|
|
174
|
+
|
|
175
|
+
- Standalone ioredis client with event logging and shutdown helpers
|
|
176
|
+
- BullMQ queue and worker factories
|
|
177
|
+
- Built-in console and silent logger adapters
|
|
178
|
+
- TypeScript declarations, tests, examples, and npm package metadata
|
|
179
|
+
|
|
180
|
+
#### Changed
|
|
181
|
+
|
|
182
|
+
- Published independently as `@bootstrap-framework/redis`
|
|
183
|
+
- Logger is injected, not required as a workspace dependency
|
|
184
|
+
|
|
185
|
+
[Unreleased]: https://github.com/mayank040902/oneunit/compare/redis-v1.0.0...HEAD
|
|
186
|
+
[1.0.0]: https://github.com/mayank040902/oneunit/releases/tag/redis-v1.0.0
|