@iskra-bun/kv-kit 0.2.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +65 -0
- package/README.md +9 -4
- package/dist/index.d.ts +42 -0
- package/dist/index.js +284 -39
- package/dist/index.js.map +1 -1
- package/package.json +5 -2
- package/src/adapters/memory.ts +86 -18
- package/src/adapters/redis.ts +214 -23
- package/src/manager.ts +86 -14
- package/src/ttl.ts +12 -0
- package/src/types.ts +16 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,70 @@
|
|
|
1
1
|
# @iskra-bun/kv-kit
|
|
2
2
|
|
|
3
|
+
## 0.3.1
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- Updated dependencies [e86ed55]
|
|
8
|
+
- Updated dependencies [ae7c798]
|
|
9
|
+
- Updated dependencies [ef10372]
|
|
10
|
+
- @iskra-bun/core@0.3.0
|
|
11
|
+
|
|
12
|
+
## 0.3.0
|
|
13
|
+
|
|
14
|
+
### Minor Changes
|
|
15
|
+
|
|
16
|
+
- 87f6de2: `KVManager` throws when its constructor gets `adapter`, `driver` or `connection`: the store is chosen by the App config (`kv: { driver, connection }`), and the README's `new KVManager({ adapter: 'redis' })` was silently ignored, leaving the app on per-process memory. Without a `kv` driver it now logs a warning in production instead of an info line. The READMEs of kv-kit, worker-kit (`connection` and `queueName`, not `queue`), db-kit (the App's `db` config) and process-kit (the App's `processes` config) show working examples.
|
|
17
|
+
- 58d4a8f: `KVManager` registers itself in the app context as `'kv'` (as db-kit does with `'db'`) and exposes the underlying ioredis client as `client` (Redis driver, after `start()`; it bypasses `namespace` and the JSON codec) for commands the KV API does not cover, such as sets. forms-app's services read `app.context.get('kv').client`, which was always `undefined`, so they silently skipped every Redis read and write and no published form could be found.
|
|
18
|
+
- dbf8817: Redis connection fixes for kv-kit (also used by cache-kit's `RedisAdapter`).
|
|
19
|
+
|
|
20
|
+
- `connection: { url }`, as shown in the docs, was ignored by ioredis, so the adapter silently connected to `localhost:6379` db 0. The URL is now honored; a plain URL string or regular ioredis options also work.
|
|
21
|
+
- Fractional TTLs (e.g. `0.5`) use `PX` instead of failing on Redis `EX`, and `mset` reports errors from individual pipelined commands instead of ignoring them.
|
|
22
|
+
- `disconnect()` uses `QUIT`, so in-flight writes are not dropped.
|
|
23
|
+
- **Breaking:** operations before `connect()` now throw instead of silently doing nothing, and an unsupported `kv.driver` (such as `"libsql"`, which never had an adapter and fell back to memory) now throws at `init()`. `AppConfig.kv.driver` is `"memory" | "redis"`.
|
|
24
|
+
|
|
25
|
+
- 938dd41: **Security** fixes from the data-kits audit (round 2).
|
|
26
|
+
|
|
27
|
+
- `storage-kit` (**breaking**): files are stored and served with a type from their extension (`contentTypeFor`), and anything but a raster image as a download. The S3 adapter stores a `Content-Disposition` with each object (`attachment` unless it is a PNG, JPEG, GIF, WebP, AVIF, BMP or ICO image; `put(..., { contentDisposition })` to choose), and `url()` signs `response-content-type` and `response-content-disposition` into presigned URLs whatever the object was stored with (`url(path, expiresIn, { contentType, contentDisposition })` to choose): an upload named `logo.svg` or `invoice.html` was stored as `image/svg+xml`/`text/html` and ran its scripts on the bucket's origin. HTML, SVG, XML and JavaScript are now `application/octet-stream`. `put(..., { overwrite: false })` throws the new `FileExistsError` instead of replacing a stored file (S3 `If-None-Match: *`, an exclusive create locally); `@aws-sdk/client-s3` and `@aws-sdk/s3-request-presigner` now need 3.635 or later, the first releases that send `If-None-Match` on a put (earlier ones dropped it, and the file was replaced). The plaintext-endpoint guard parses the endpoint as a URL, as the SDK does: `http:/minio:9000`, `http:minio:9000` and `http:\\minio:9000` were accepted without `useSSL: false`; an endpoint that is not an `http(s)` URL is rejected.
|
|
28
|
+
- `web-kit` uploads (**breaking**): the upload route stores a file with the type of its extension, never `File.type` (which Bun derives from the name), and `uploadFromRequest()` too. Downloads are streamed with `getStream()` (each one was buffered twice), typed by extension, `attachment` unless a raster image, and sandboxed (`Content-Security-Policy: sandbox`). Without `allowedExtensions`, active web content (`.html`, `.svg`, `.xml`, `.js`...) is refused (400) unless listed. `authorize(c, action, target)` receives what the action touches (`{ key, subfolder, filename, size, type }`), and `upload` is asked again with it before the file is written. An upload no longer replaces a stored file: **409** unless `overwrite: true`.
|
|
29
|
+
- `mailer-kit` (**breaking**): every `to`, `cc`, `bcc` and `replyTo` entry must be one bare address, or a new `{ name, address }` object for a display name, in every adapter (the mock too); only `from` was checked. One value such as `"bob@example.com <attacker@evil.test>, x@example.com"`, a group (`"undisclosed: a@evil.test; b@x.com"`, `"a@evil.test:b@x.com"`) or `{ address: "bob@example.com\r\nBcc: …" }` mailed other recipients than the ones an allowlist checked. Addresses may not contain whitespace, control characters or `<>()[]\,;:"` and need exactly one `@`; `replyTo` takes one recipient. `checkRecipients()` is exported. Mailgun cuts the `subject` at a CR/LF, as it does header values.
|
|
30
|
+
- `web-kit` email: `EmailFeature`'s adapter checks recipients with mailer-kit's rules (object recipients were tested as `"[object Object]"`), and rejects through the returned promise instead of throwing synchronously.
|
|
31
|
+
- `kv-kit`: the `KVAdapter` contract gains an optional `clear(prefix?)` and expiring sets (`sadd(key, member, ttl?)`, `sdrain(key)`), implemented by both adapters and `KVManager`. `KVManager.clear()` deletes its namespace's keys; with Redis it uses `SCAN` + `DEL` (within ioredis' `keyPrefix` too) and, without a namespace, refuses to empty the whole database unless `new KVManager({ flushDb: true })`. Expiring sets are sorted sets scored by expiry, updated by one atomic script: cache-kit's tag index. The memory adapter stores and returns copies (`structuredClone`), as Redis does (**breaking** for values that cannot be cloned, such as functions): it returned the stored object itself, so one request's mutation showed up in every other.
|
|
32
|
+
- `cache-kit` (**breaking**): `clear()` runs the adapter's `clear()` with the cache's namespace instead of `disconnect()`/`connect()` of the shared adapter, which on Redis deleted nothing (cached permissions stayed), failed concurrent operations meanwhile, and left the adapter dead when the reconnect failed during a Redis blip. A namespaced cache now clears its own entries (it used to throw); an adapter without `clear()` makes it throw. The tag index is kv-kit's expiring set when the adapter has one (one atomic `sadd` per tagged `set()`; each one read and rewrote the whole index, and the last 10k of 40k tagged sets took 30 s), and otherwise a JSON list that drops expired keys, expires with its last entry and keeps at most 10,000 (the oldest are deleted with their data); indexes written before are still drained by `invalidateTag()`. A value with a `__proto__`/`constructor`/`prototype` key is a miss (deleted when read; not stored by `set()`), so `remember()` refetches instead of every read throwing until the TTL ran out. Data keys and namespaces containing `__cache_tag__:`/`__cache_tags__:` (at the start or after a `:`) are rejected: a caller-chosen key could rewrite a tag index, and `invalidateTag()` deleted whatever it listed.
|
|
33
|
+
- `db-kit` (**breaking**): `MigrationHelper` and the CLI run only the drizzle-kit installed in the project (`node_modules/.bin` of the working directory or a parent), with `bunx --no-install drizzle-kit`, and fail with a `MigrationError` where it is not installed. drizzle-kit is a devDependency, so in a production install `bunx drizzle-kit` downloaded its latest release from npm and ran it with `DATABASE_URL` in its environment.
|
|
34
|
+
|
|
35
|
+
### Patch Changes
|
|
36
|
+
|
|
37
|
+
- 840439a: Packages declare the runtime they are tested on: `engines.bun` `>=1.3.0` (the monorepo now builds and tests on Bun 1.3). `create-iskra`, a CLI that also runs under `npm create iskra`, declares `engines.node` `>=18`.
|
|
38
|
+
|
|
39
|
+
Every package is published with an npm provenance attestation (`publishConfig.provenance`), linking each version to the commit and CI run that built it.
|
|
40
|
+
|
|
41
|
+
- cb3ec43: Register what each kit puts on the app with core's new registries: `app.context.get('db' | 'kv' | 'oracle')` returns the kit's driver, and the `process:*`, `socket:connected` / `socket:disconnected` and `worker:dead-letter` events have typed payloads. `ProcessManager.send()` takes `unknown` data.
|
|
42
|
+
- 620da18: `RedisAdapter.disconnect()` (and so `KVManager.stop()`) no longer waits for Redis's reconnect attempts when the server is down: the client is closed at once. ioredis queued `QUIT` behind the pending commands, so it took about 10 s (never returned with `maxRetriesPerRequest: null`) and used up the app's shutdown timeout. When connected, `QUIT` is given at most 2 s.
|
|
43
|
+
- c4ff1e3: The Redis adapter's `sdrain()` returns only the members that have not expired, as the memory adapter does. It returned every member still stored, expired ones included, until the next `sadd()` dropped them; it now reads and deletes the set in one script that filters by Redis time.
|
|
44
|
+
- 7e89103: The Redis driver connects at `start()` and fails it when Redis is unreachable or rejects the password, instead of failing (or hanging) on the first command; ioredis connection errors go to the app's logger. The memory adapter keeps keys whose TTL exceeds `setTimeout`'s ~24.8-day limit (they expired at once), and a negative or non-finite TTL is rejected with a `RangeError` by both adapters. The docs no longer claim that `Date` or `Map` values round-trip through Redis.
|
|
45
|
+
- 9bb254d: **Security** fixes from the plugin audit.
|
|
46
|
+
|
|
47
|
+
- `kv-kit`: Redis errors passed to `onError` (and so logged by `KVManager`) and the cause of a failed `connect()` keep only the command's name. ioredis attached its arguments, so a refused login logged the Redis password (`command.args` of AUTH), for example after a password rotation.
|
|
48
|
+
- `socket-kit`: `ctx.broadcast()` and `ctx.join()` refuse a room or topic that is not a string before calling `canPublish`/`canJoin`. A handler passing the client's value on could be given `["global"]`, which passes a deny-list such as `topic !== 'global'` and which Bun's `publish()` turns into `"global"`. The rate limiter logs one warning per connection and window instead of one per dropped frame (a flooding client wrote ~23 bytes of log per byte sent), and a frame that is not JSON is logged at debug level without a stack.
|
|
49
|
+
- `process-kit` (**breaking**): `send()` refuses a string with a line break, which the child read as several messages (send an object to have it JSON-encoded). After a stdout line longer than 1 MiB, the rest of that line is dropped instead of being read as a line of its own, which let text a child echoed come out as a JSON `process:message`. The spawn log line has the command and the number of arguments; the arguments, which can carry credentials, are logged at debug level.
|
|
50
|
+
- `create-iskra`: `scaffold()` only accepts the name of a bundled template (`../x` copied any directory into the new project), and `scripts/sync.ts` refuses symlinks in a template instead of copying their target into the published package.
|
|
51
|
+
|
|
52
|
+
- Updated dependencies [620da18]
|
|
53
|
+
- Updated dependencies [b635a2c]
|
|
54
|
+
- Updated dependencies [5b2b0fd]
|
|
55
|
+
- Updated dependencies [58d4a8f]
|
|
56
|
+
- Updated dependencies [5c70c5b]
|
|
57
|
+
- Updated dependencies [ec198d4]
|
|
58
|
+
- Updated dependencies [cb3ec43]
|
|
59
|
+
- Updated dependencies [ef2009b]
|
|
60
|
+
- Updated dependencies [840439a]
|
|
61
|
+
- Updated dependencies [dbf8817]
|
|
62
|
+
- Updated dependencies [3dc5581]
|
|
63
|
+
- Updated dependencies [9872d30]
|
|
64
|
+
- Updated dependencies [f2346f5]
|
|
65
|
+
- Updated dependencies [3579944]
|
|
66
|
+
- @iskra-bun/core@0.2.0
|
|
67
|
+
|
|
3
68
|
## 0.2.0
|
|
4
69
|
|
|
5
70
|
### Minor Changes
|
package/README.md
CHANGED
|
@@ -14,17 +14,22 @@ bun add @iskra-bun/kv-kit @iskra-bun/core
|
|
|
14
14
|
import { App } from '@iskra-bun/core'
|
|
15
15
|
import { KVManager } from '@iskra-bun/kv-kit'
|
|
16
16
|
|
|
17
|
-
const app = new App({
|
|
18
|
-
|
|
17
|
+
const app = new App({
|
|
18
|
+
name: 'mi-app',
|
|
19
|
+
kv: { driver: 'redis', connection: process.env.REDIS_URL }, // o { driver: 'memory' }
|
|
20
|
+
})
|
|
21
|
+
const kv = new KVManager()
|
|
22
|
+
app.register(kv)
|
|
19
23
|
|
|
20
24
|
await app.start()
|
|
25
|
+
await kv.set('usuario:123', { name: 'Ana' }, 60) // TTL en segundos
|
|
21
26
|
```
|
|
22
27
|
|
|
23
|
-
|
|
28
|
+
El store se elige en la config de la App (`kv.driver`), no en el constructor de `KVManager`. Sin `kv` usa memoria: cada proceso tiene la suya y se pierde al reiniciar (en produccion lo avisa con un warning). La API (`get`/`set`/`del`/`has`, TTL) es identica entre adaptadores.
|
|
24
29
|
|
|
25
30
|
## Documentacion
|
|
26
31
|
|
|
27
|
-
Guia completa: [
|
|
32
|
+
Guia completa: [@iskra-bun/kv-kit](https://iskra-docs.fly.dev/es/packages/kv-kit/)
|
|
28
33
|
|
|
29
34
|
## Licencia
|
|
30
35
|
|
package/dist/index.d.ts
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { Driver, App } from '@iskra-bun/core';
|
|
2
|
+
import { Redis } from 'ioredis';
|
|
2
3
|
|
|
3
4
|
interface KVAdapter {
|
|
4
5
|
id: string;
|
|
@@ -11,6 +12,20 @@ interface KVAdapter {
|
|
|
11
12
|
mget?<T = unknown>(keys: string[]): Promise<(T | undefined)[]>;
|
|
12
13
|
mset?<T = unknown>(entries: Array<[string, T]>, ttl?: number): Promise<void>;
|
|
13
14
|
mdel?(keys: string[]): Promise<void>;
|
|
15
|
+
/**
|
|
16
|
+
* Deletes every key that starts with `prefix` (every key the adapter holds
|
|
17
|
+
* without one), without touching the connection. An adapter that cannot
|
|
18
|
+
* leaves it out, and callers such as cache-kit's `clear()` then fail.
|
|
19
|
+
*/
|
|
20
|
+
clear?(prefix?: string): Promise<void>;
|
|
21
|
+
/**
|
|
22
|
+
* Expiring sets, for cache-kit's tag index: `sadd` adds `member` to the set
|
|
23
|
+
* at `key` for `ttl` seconds (for good without one), and the set lives as
|
|
24
|
+
* long as its longest-lived member. `sdrain` deletes the set and returns
|
|
25
|
+
* its unexpired members in one step, so a member added meanwhile is never lost.
|
|
26
|
+
*/
|
|
27
|
+
sadd?(key: string, member: string, ttl?: number): Promise<void>;
|
|
28
|
+
sdrain?(key: string): Promise<string[]>;
|
|
14
29
|
}
|
|
15
30
|
|
|
16
31
|
interface KVManagerOptions {
|
|
@@ -19,6 +34,11 @@ interface KVManagerOptions {
|
|
|
19
34
|
* Defaults to `""` (no prefix) to preserve existing behavior.
|
|
20
35
|
*/
|
|
21
36
|
namespace?: string;
|
|
37
|
+
/**
|
|
38
|
+
* Lets `clear()` without a `namespace` empty the whole Redis database
|
|
39
|
+
* (FLUSHDB). Only for a database no other app or service writes to.
|
|
40
|
+
*/
|
|
41
|
+
flushDb?: boolean;
|
|
22
42
|
}
|
|
23
43
|
declare class KVManager implements Driver, KVAdapter {
|
|
24
44
|
name: string;
|
|
@@ -26,12 +46,20 @@ declare class KVManager implements Driver, KVAdapter {
|
|
|
26
46
|
private app;
|
|
27
47
|
private adapter;
|
|
28
48
|
private readonly prefix;
|
|
49
|
+
private readonly flushDb;
|
|
29
50
|
constructor(options?: KVManagerOptions);
|
|
30
51
|
init(app: App): void;
|
|
31
52
|
connect(): Promise<void>;
|
|
32
53
|
disconnect(): Promise<void>;
|
|
33
54
|
start(): Promise<void>;
|
|
34
55
|
stop(): Promise<void>;
|
|
56
|
+
/**
|
|
57
|
+
* The underlying ioredis client with the "redis" driver, once started: for
|
|
58
|
+
* commands the KV API does not cover (sets, sorted sets, pipelines). It
|
|
59
|
+
* bypasses `namespace` and the JSON codec. `undefined` with the memory
|
|
60
|
+
* driver or before start().
|
|
61
|
+
*/
|
|
62
|
+
get client(): Redis | undefined;
|
|
35
63
|
private prefixed;
|
|
36
64
|
get<T = unknown>(key: string): Promise<T | undefined>;
|
|
37
65
|
set<T = unknown>(key: string, value: T, ttl?: number): Promise<void>;
|
|
@@ -40,6 +68,20 @@ declare class KVManager implements Driver, KVAdapter {
|
|
|
40
68
|
mget<T = unknown>(keys: string[]): Promise<(T | undefined)[]>;
|
|
41
69
|
mset<T = unknown>(entries: Array<[string, T]> | Record<string, T>, ttl?: number): Promise<void>;
|
|
42
70
|
mdel(keys: string[]): Promise<void>;
|
|
71
|
+
/**
|
|
72
|
+
* Deletes every key under `prefix` in this manager's namespace (the whole
|
|
73
|
+
* namespace without one). With the redis driver and no namespace, only
|
|
74
|
+
* `flushDb: true` lets it empty the database.
|
|
75
|
+
*/
|
|
76
|
+
clear(prefix?: string): Promise<void>;
|
|
77
|
+
/** See {@link KVAdapter.sadd}: the member is kept as is, only `key` is namespaced. */
|
|
78
|
+
sadd(key: string, member: string, ttl?: number): Promise<void>;
|
|
79
|
+
sdrain(key: string): Promise<string[]>;
|
|
80
|
+
}
|
|
81
|
+
declare module '@iskra-bun/core' {
|
|
82
|
+
interface AppContextRegistry {
|
|
83
|
+
kv: KVManager;
|
|
84
|
+
}
|
|
43
85
|
}
|
|
44
86
|
|
|
45
87
|
export { type KVAdapter, KVManager };
|
package/dist/index.js
CHANGED
|
@@ -1,7 +1,21 @@
|
|
|
1
|
+
// src/manager.ts
|
|
2
|
+
import { isProductionEnv } from "@iskra-bun/core";
|
|
3
|
+
|
|
4
|
+
// src/ttl.ts
|
|
5
|
+
function checkTtl(ttl) {
|
|
6
|
+
if (ttl === void 0 || ttl === null || ttl === 0) return void 0;
|
|
7
|
+
if (typeof ttl !== "number" || !Number.isFinite(ttl) || ttl < 0) {
|
|
8
|
+
throw new RangeError(`Invalid TTL ${String(ttl)}: expected a positive number of seconds`);
|
|
9
|
+
}
|
|
10
|
+
return ttl;
|
|
11
|
+
}
|
|
12
|
+
|
|
1
13
|
// src/adapters/memory.ts
|
|
14
|
+
var MAX_TIMEOUT_MS = 2 ** 31 - 1;
|
|
2
15
|
var MemoryAdapter = class {
|
|
3
16
|
id = "memory";
|
|
4
17
|
store = /* @__PURE__ */ new Map();
|
|
18
|
+
sets = /* @__PURE__ */ new Map();
|
|
5
19
|
timers = /* @__PURE__ */ new Map();
|
|
6
20
|
connect() {
|
|
7
21
|
}
|
|
@@ -11,36 +25,84 @@ var MemoryAdapter = class {
|
|
|
11
25
|
}
|
|
12
26
|
this.timers.clear();
|
|
13
27
|
this.store.clear();
|
|
28
|
+
this.sets.clear();
|
|
14
29
|
}
|
|
30
|
+
// Values are copied in and out, as Redis does: a stored object returned by
|
|
31
|
+
// reference let one request's mutation show up in every other one.
|
|
15
32
|
async get(key) {
|
|
16
|
-
return this.store.get(key);
|
|
33
|
+
return structuredClone(this.store.get(key));
|
|
17
34
|
}
|
|
18
35
|
async set(key, value, ttl) {
|
|
19
|
-
const
|
|
20
|
-
|
|
21
|
-
|
|
36
|
+
const seconds = checkTtl(ttl);
|
|
37
|
+
const copy = structuredClone(value);
|
|
38
|
+
this.clearTimer(key);
|
|
39
|
+
this.sets.delete(key);
|
|
40
|
+
this.store.set(key, copy);
|
|
41
|
+
if (seconds !== void 0) this.expireIn(key, seconds * 1e3);
|
|
42
|
+
}
|
|
43
|
+
/** Arms the expiry timer, in steps when the delay exceeds setTimeout's limit. */
|
|
44
|
+
expireIn(key, ms) {
|
|
45
|
+
const step = Math.min(ms, MAX_TIMEOUT_MS);
|
|
46
|
+
const timer = setTimeout(() => {
|
|
47
|
+
if (ms > step) {
|
|
48
|
+
this.expireIn(key, ms - step);
|
|
49
|
+
return;
|
|
50
|
+
}
|
|
51
|
+
this.store.delete(key);
|
|
52
|
+
this.sets.delete(key);
|
|
22
53
|
this.timers.delete(key);
|
|
23
|
-
}
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
const timer = setTimeout(() => {
|
|
27
|
-
this.store.delete(key);
|
|
28
|
-
this.timers.delete(key);
|
|
29
|
-
}, ttl * 1e3);
|
|
30
|
-
timer.unref?.();
|
|
31
|
-
this.timers.set(key, timer);
|
|
32
|
-
}
|
|
54
|
+
}, step);
|
|
55
|
+
timer.unref?.();
|
|
56
|
+
this.timers.set(key, timer);
|
|
33
57
|
}
|
|
34
|
-
|
|
58
|
+
clearTimer(key) {
|
|
35
59
|
const timer = this.timers.get(key);
|
|
36
60
|
if (timer !== void 0) {
|
|
37
61
|
clearTimeout(timer);
|
|
38
62
|
this.timers.delete(key);
|
|
39
63
|
}
|
|
64
|
+
}
|
|
65
|
+
async del(key) {
|
|
66
|
+
this.clearTimer(key);
|
|
40
67
|
this.store.delete(key);
|
|
68
|
+
this.sets.delete(key);
|
|
41
69
|
}
|
|
42
70
|
async has(key) {
|
|
43
|
-
return this.store.has(key);
|
|
71
|
+
return this.store.has(key) || this.sets.has(key);
|
|
72
|
+
}
|
|
73
|
+
async clear(prefix = "") {
|
|
74
|
+
for (const key of [...this.store.keys(), ...this.sets.keys()]) {
|
|
75
|
+
if (key.startsWith(prefix)) await this.del(key);
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
async sadd(key, member, ttl) {
|
|
79
|
+
const seconds = checkTtl(ttl);
|
|
80
|
+
const now = Date.now();
|
|
81
|
+
const expiresAt = seconds === void 0 ? Infinity : now + seconds * 1e3;
|
|
82
|
+
let set = this.sets.get(key);
|
|
83
|
+
if (!set) {
|
|
84
|
+
this.store.delete(key);
|
|
85
|
+
set = { members: /* @__PURE__ */ new Map(), expiresAt: 0, pruneAt: 64 };
|
|
86
|
+
this.sets.set(key, set);
|
|
87
|
+
}
|
|
88
|
+
set.members.set(member, Math.max(set.members.get(member) ?? 0, expiresAt));
|
|
89
|
+
if (set.members.size >= set.pruneAt) {
|
|
90
|
+
for (const [name, at] of set.members) if (at <= now) set.members.delete(name);
|
|
91
|
+
set.pruneAt = Math.max(64, set.members.size * 2);
|
|
92
|
+
}
|
|
93
|
+
if (expiresAt > set.expiresAt) {
|
|
94
|
+
set.expiresAt = expiresAt;
|
|
95
|
+
this.clearTimer(key);
|
|
96
|
+
if (expiresAt !== Infinity) this.expireIn(key, expiresAt - now);
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
async sdrain(key) {
|
|
100
|
+
const set = this.sets.get(key);
|
|
101
|
+
if (!set) return [];
|
|
102
|
+
this.sets.delete(key);
|
|
103
|
+
this.clearTimer(key);
|
|
104
|
+
const now = Date.now();
|
|
105
|
+
return [...set.members].filter(([, at]) => at > now).map(([name]) => name);
|
|
44
106
|
}
|
|
45
107
|
};
|
|
46
108
|
|
|
@@ -58,66 +120,202 @@ function decode(raw) {
|
|
|
58
120
|
return raw;
|
|
59
121
|
}
|
|
60
122
|
}
|
|
123
|
+
function ttlArgs(ttl) {
|
|
124
|
+
return Number.isInteger(ttl) ? ["EX", ttl] : ["PX", Math.max(1, Math.round(ttl * 1e3))];
|
|
125
|
+
}
|
|
126
|
+
var escapeGlob = (prefix) => prefix.replace(/[*?[\]\\]/g, "\\$&");
|
|
127
|
+
var CLEAR_BATCH = 1e3;
|
|
128
|
+
var SADD_SCRIPT = `
|
|
129
|
+
if redis.replicate_commands then redis.replicate_commands() end
|
|
130
|
+
local forever = 9007199254740991
|
|
131
|
+
local t = redis.call('TIME')
|
|
132
|
+
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
|
|
133
|
+
local expires = forever
|
|
134
|
+
if ARGV[2] ~= '' then expires = now + tonumber(ARGV[2]) end
|
|
135
|
+
local current = tonumber(redis.call('ZSCORE', KEYS[1], ARGV[1]) or '0')
|
|
136
|
+
if expires > current then redis.call('ZADD', KEYS[1], expires, ARGV[1]) end
|
|
137
|
+
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', '(' .. now)
|
|
138
|
+
local last = redis.call('ZRANGE', KEYS[1], -1, -1, 'WITHSCORES')
|
|
139
|
+
if tonumber(last[2]) >= forever then
|
|
140
|
+
redis.call('PERSIST', KEYS[1])
|
|
141
|
+
else
|
|
142
|
+
redis.call('PEXPIREAT', KEYS[1], last[2])
|
|
143
|
+
end
|
|
144
|
+
`;
|
|
145
|
+
var SDRAIN_SCRIPT = `
|
|
146
|
+
if redis.replicate_commands then redis.replicate_commands() end
|
|
147
|
+
local t = redis.call('TIME')
|
|
148
|
+
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
|
|
149
|
+
local members = redis.call('ZRANGEBYSCORE', KEYS[1], '(' .. now, '+inf')
|
|
150
|
+
redis.call('DEL', KEYS[1])
|
|
151
|
+
return members
|
|
152
|
+
`;
|
|
153
|
+
function withoutCommandArgs(error) {
|
|
154
|
+
const command = error?.command;
|
|
155
|
+
if (command && typeof command === "object" && "args" in command) {
|
|
156
|
+
error.command = { name: command.name };
|
|
157
|
+
}
|
|
158
|
+
return error;
|
|
159
|
+
}
|
|
61
160
|
var RedisAdapter = class {
|
|
62
161
|
id = "redis";
|
|
63
162
|
client = null;
|
|
64
163
|
options;
|
|
65
|
-
|
|
164
|
+
hooks;
|
|
165
|
+
constructor(options, hooks = {}) {
|
|
66
166
|
this.options = options;
|
|
167
|
+
this.hooks = hooks;
|
|
67
168
|
}
|
|
68
|
-
|
|
69
|
-
|
|
169
|
+
/**
|
|
170
|
+
* Connects and waits for Redis to answer: an unreachable server or a wrong
|
|
171
|
+
* password fails here (and so the app's start) instead of on the first
|
|
172
|
+
* command. Once connected, a dropped connection is retried by ioredis.
|
|
173
|
+
*/
|
|
174
|
+
async connect() {
|
|
175
|
+
let client;
|
|
176
|
+
if (typeof this.options === "string") {
|
|
177
|
+
client = new Redis(this.options, { lazyConnect: true });
|
|
178
|
+
} else if (this.options.url) {
|
|
179
|
+
const { url, ...rest } = this.options;
|
|
180
|
+
client = new Redis(url, { ...rest, lazyConnect: true });
|
|
181
|
+
} else {
|
|
182
|
+
client = new Redis({ ...this.options, lazyConnect: true });
|
|
183
|
+
}
|
|
184
|
+
client.on("error", (error) => this.hooks.onError?.(withoutCommandArgs(error)));
|
|
185
|
+
try {
|
|
186
|
+
await client.connect();
|
|
187
|
+
} catch (error) {
|
|
188
|
+
client.disconnect();
|
|
189
|
+
throw new Error(`Could not connect to Redis: ${error instanceof Error ? error.message : String(error)}`, {
|
|
190
|
+
cause: withoutCommandArgs(error)
|
|
191
|
+
});
|
|
192
|
+
}
|
|
193
|
+
this.client = client;
|
|
70
194
|
}
|
|
71
|
-
|
|
72
|
-
|
|
195
|
+
/** Longest wait for QUIT before the connection is closed anyway. */
|
|
196
|
+
quitTimeoutMs = 2e3;
|
|
197
|
+
/**
|
|
198
|
+
* Closes the connection. When connected, QUIT lets pending replies arrive
|
|
199
|
+
* (in-flight writes are not dropped), for at most `quitTimeoutMs`. When the
|
|
200
|
+
* connection is down, ioredis would queue QUIT behind the offline queue and
|
|
201
|
+
* resolve only after its reconnect attempts run out (about 10 s by default,
|
|
202
|
+
* never with `maxRetriesPerRequest: null`), eating the app's shutdown
|
|
203
|
+
* timeout: the client is closed at once instead.
|
|
204
|
+
*/
|
|
205
|
+
async disconnect() {
|
|
206
|
+
const client = this.client;
|
|
207
|
+
this.client = null;
|
|
208
|
+
if (!client) return;
|
|
209
|
+
if (typeof client.quit !== "function" || client.status !== "ready") {
|
|
210
|
+
client.disconnect();
|
|
211
|
+
return;
|
|
212
|
+
}
|
|
213
|
+
let timer;
|
|
214
|
+
const timedOut = new Promise((resolve) => {
|
|
215
|
+
timer = setTimeout(() => resolve("timeout"), this.quitTimeoutMs);
|
|
216
|
+
});
|
|
217
|
+
const outcome = await Promise.race([
|
|
218
|
+
client.quit().then(
|
|
219
|
+
() => "quit",
|
|
220
|
+
() => "error"
|
|
221
|
+
),
|
|
222
|
+
timedOut
|
|
223
|
+
]);
|
|
224
|
+
clearTimeout(timer);
|
|
225
|
+
if (outcome !== "quit") client.disconnect();
|
|
226
|
+
}
|
|
227
|
+
/** The ioredis client once connected (null before connect() and after disconnect()). */
|
|
228
|
+
get nativeClient() {
|
|
229
|
+
return this.client;
|
|
230
|
+
}
|
|
231
|
+
/** The live client; operations before connect() fail instead of silently doing nothing. */
|
|
232
|
+
get redis() {
|
|
233
|
+
if (!this.client) throw new Error("RedisAdapter is not connected; call connect() first");
|
|
234
|
+
return this.client;
|
|
73
235
|
}
|
|
74
236
|
async get(key) {
|
|
75
|
-
const val = await this.
|
|
237
|
+
const val = await this.redis.get(key);
|
|
76
238
|
if (val === null || val === void 0) return void 0;
|
|
77
239
|
return decode(val);
|
|
78
240
|
}
|
|
79
241
|
async set(key, value, ttl) {
|
|
80
242
|
const val = encode(value);
|
|
81
|
-
|
|
82
|
-
|
|
243
|
+
const seconds = checkTtl(ttl);
|
|
244
|
+
if (seconds !== void 0) {
|
|
245
|
+
await this.redis.set(key, val, ...ttlArgs(seconds));
|
|
83
246
|
} else {
|
|
84
|
-
await this.
|
|
247
|
+
await this.redis.set(key, val);
|
|
85
248
|
}
|
|
86
249
|
}
|
|
87
250
|
async del(key) {
|
|
88
|
-
await this.
|
|
251
|
+
await this.redis.del(key);
|
|
89
252
|
}
|
|
90
253
|
async has(key) {
|
|
91
|
-
const exists = await this.
|
|
254
|
+
const exists = await this.redis.exists(key);
|
|
92
255
|
return exists === 1;
|
|
93
256
|
}
|
|
94
257
|
// Native batch operations. A single MGET / pipelined MSET / variadic DEL
|
|
95
258
|
// replaces the manager's per-key fan-out (avoids the N+1 round-trips).
|
|
96
259
|
async mget(keys) {
|
|
97
260
|
if (keys.length === 0) return [];
|
|
98
|
-
const raws = await this.
|
|
261
|
+
const raws = await this.redis.mget(...keys) ?? [];
|
|
99
262
|
return keys.map((_, i) => {
|
|
100
263
|
const raw = raws[i];
|
|
101
264
|
return raw === null || raw === void 0 ? void 0 : decode(raw);
|
|
102
265
|
});
|
|
103
266
|
}
|
|
104
267
|
async mset(entries, ttl) {
|
|
268
|
+
const seconds = checkTtl(ttl);
|
|
105
269
|
if (entries.length === 0) return;
|
|
106
|
-
const pipeline = this.
|
|
107
|
-
if (!pipeline) return;
|
|
270
|
+
const pipeline = this.redis.pipeline();
|
|
108
271
|
for (const [key, value] of entries) {
|
|
109
272
|
const val = encode(value);
|
|
110
|
-
if (
|
|
111
|
-
pipeline.set(key, val,
|
|
273
|
+
if (seconds !== void 0) {
|
|
274
|
+
pipeline.set(key, val, ...ttlArgs(seconds));
|
|
112
275
|
} else {
|
|
113
276
|
pipeline.set(key, val);
|
|
114
277
|
}
|
|
115
278
|
}
|
|
116
|
-
await pipeline.exec();
|
|
279
|
+
const results = await pipeline.exec();
|
|
280
|
+
const failed = results?.find(([err]) => err);
|
|
281
|
+
if (failed) throw failed[0];
|
|
117
282
|
}
|
|
118
283
|
async mdel(keys) {
|
|
119
284
|
if (keys.length === 0) return;
|
|
120
|
-
await this.
|
|
285
|
+
await this.redis.del(...keys);
|
|
286
|
+
}
|
|
287
|
+
/**
|
|
288
|
+
* Deletes the keys under `prefix` (after ioredis' own `keyPrefix`, if set)
|
|
289
|
+
* with SCAN and DEL. With no prefix at all this would be every key in the
|
|
290
|
+
* database, which only `flushDb: true` allows (as a FLUSHDB).
|
|
291
|
+
*/
|
|
292
|
+
async clear(prefix = "") {
|
|
293
|
+
const redis = this.redis;
|
|
294
|
+
const keyPrefix = redis.options.keyPrefix ?? "";
|
|
295
|
+
if (!keyPrefix && !prefix) {
|
|
296
|
+
if (!this.hooks.flushDb) {
|
|
297
|
+
throw new Error(
|
|
298
|
+
"RedisAdapter.clear() without a key prefix would empty the whole Redis database: give the KVManager a namespace, or pass flushDb: true if the database belongs to this app alone"
|
|
299
|
+
);
|
|
300
|
+
}
|
|
301
|
+
await redis.flushdb();
|
|
302
|
+
return;
|
|
303
|
+
}
|
|
304
|
+
const match = `${escapeGlob(keyPrefix + prefix)}*`;
|
|
305
|
+
let cursor = "0";
|
|
306
|
+
do {
|
|
307
|
+
const [next, keys] = await redis.scan(cursor, "MATCH", match, "COUNT", CLEAR_BATCH);
|
|
308
|
+
cursor = next;
|
|
309
|
+
if (keys.length > 0) await redis.del(...keys.map((key) => key.slice(keyPrefix.length)));
|
|
310
|
+
} while (cursor !== "0");
|
|
311
|
+
}
|
|
312
|
+
async sadd(key, member, ttl) {
|
|
313
|
+
const seconds = checkTtl(ttl);
|
|
314
|
+
const ms = seconds === void 0 ? "" : String(Math.max(1, Math.ceil(seconds * 1e3)));
|
|
315
|
+
await this.redis.eval(SADD_SCRIPT, 1, key, member, ms);
|
|
316
|
+
}
|
|
317
|
+
async sdrain(key) {
|
|
318
|
+
return await this.redis.eval(SDRAIN_SCRIPT, 1, key);
|
|
121
319
|
}
|
|
122
320
|
};
|
|
123
321
|
|
|
@@ -128,19 +326,38 @@ var KVManager = class {
|
|
|
128
326
|
app = null;
|
|
129
327
|
adapter;
|
|
130
328
|
prefix;
|
|
329
|
+
flushDb;
|
|
131
330
|
constructor(options = {}) {
|
|
331
|
+
const misplaced = ["adapter", "driver", "connection"].filter((key) => key in options);
|
|
332
|
+
if (misplaced.length > 0) {
|
|
333
|
+
throw new Error(
|
|
334
|
+
`KVManager does not take ${misplaced.map((k) => `"${k}"`).join(", ")}: choose the store in the App config, e.g. new App({ name, kv: { driver: 'redis', connection: process.env.REDIS_URL } })`
|
|
335
|
+
);
|
|
336
|
+
}
|
|
132
337
|
this.adapter = new MemoryAdapter();
|
|
133
338
|
this.prefix = options.namespace ? `${options.namespace}:` : "";
|
|
339
|
+
this.flushDb = options.flushDb ?? false;
|
|
134
340
|
}
|
|
135
341
|
init(app) {
|
|
136
342
|
this.app = app;
|
|
343
|
+
app.context.set("kv", this);
|
|
137
344
|
const config = app.config.kv;
|
|
138
345
|
if (config?.driver === "redis") {
|
|
139
346
|
app.logger.info("Initializing KV with Redis");
|
|
140
|
-
this.adapter = new RedisAdapter(config.connection
|
|
141
|
-
|
|
347
|
+
this.adapter = new RedisAdapter(config.connection ?? {}, {
|
|
348
|
+
onError: (err) => app.logger.warn({ err }, "KV Redis connection error"),
|
|
349
|
+
flushDb: this.flushDb
|
|
350
|
+
});
|
|
351
|
+
} else if (!config?.driver || config.driver === "memory") {
|
|
352
|
+
if (!config?.driver && isProductionEnv()) {
|
|
353
|
+
app.logger.warn(
|
|
354
|
+
"KV has no driver configured: using the in-memory store, which each process keeps on its own and loses on restart. Set kv: { driver: 'redis', connection } in the App config, or kv: { driver: 'memory' } to keep it."
|
|
355
|
+
);
|
|
356
|
+
}
|
|
142
357
|
app.logger.info("Initializing KV with Memory");
|
|
143
358
|
this.adapter = new MemoryAdapter();
|
|
359
|
+
} else {
|
|
360
|
+
throw new Error(`Unsupported KV driver "${config.driver}" (supported: "memory", "redis")`);
|
|
144
361
|
}
|
|
145
362
|
}
|
|
146
363
|
async connect() {
|
|
@@ -157,6 +374,15 @@ var KVManager = class {
|
|
|
157
374
|
async stop() {
|
|
158
375
|
await this.disconnect();
|
|
159
376
|
}
|
|
377
|
+
/**
|
|
378
|
+
* The underlying ioredis client with the "redis" driver, once started: for
|
|
379
|
+
* commands the KV API does not cover (sets, sorted sets, pipelines). It
|
|
380
|
+
* bypasses `namespace` and the JSON codec. `undefined` with the memory
|
|
381
|
+
* driver or before start().
|
|
382
|
+
*/
|
|
383
|
+
get client() {
|
|
384
|
+
return this.adapter instanceof RedisAdapter ? this.adapter.nativeClient ?? void 0 : void 0;
|
|
385
|
+
}
|
|
160
386
|
prefixed(key) {
|
|
161
387
|
return `${this.prefix}${key}`;
|
|
162
388
|
}
|
|
@@ -187,9 +413,7 @@ var KVManager = class {
|
|
|
187
413
|
}
|
|
188
414
|
async mset(entries, ttl) {
|
|
189
415
|
const pairs = Array.isArray(entries) ? entries : Object.entries(entries);
|
|
190
|
-
const prefixed = pairs.map(
|
|
191
|
-
([k, v]) => [this.prefixed(k), v]
|
|
192
|
-
);
|
|
416
|
+
const prefixed = pairs.map(([k, v]) => [this.prefixed(k), v]);
|
|
193
417
|
if (this.adapter.mset) {
|
|
194
418
|
await this.adapter.mset(prefixed, ttl);
|
|
195
419
|
return;
|
|
@@ -204,7 +428,28 @@ var KVManager = class {
|
|
|
204
428
|
}
|
|
205
429
|
await Promise.all(prefixed.map((k) => this.adapter.del(k)));
|
|
206
430
|
}
|
|
431
|
+
/**
|
|
432
|
+
* Deletes every key under `prefix` in this manager's namespace (the whole
|
|
433
|
+
* namespace without one). With the redis driver and no namespace, only
|
|
434
|
+
* `flushDb: true` lets it empty the database.
|
|
435
|
+
*/
|
|
436
|
+
async clear(prefix = "") {
|
|
437
|
+
if (!this.adapter.clear) throw unsupported(this.adapter, "clear");
|
|
438
|
+
await this.adapter.clear(this.prefixed(prefix));
|
|
439
|
+
}
|
|
440
|
+
/** See {@link KVAdapter.sadd}: the member is kept as is, only `key` is namespaced. */
|
|
441
|
+
async sadd(key, member, ttl) {
|
|
442
|
+
if (!this.adapter.sadd) throw unsupported(this.adapter, "sadd");
|
|
443
|
+
await this.adapter.sadd(this.prefixed(key), member, ttl);
|
|
444
|
+
}
|
|
445
|
+
async sdrain(key) {
|
|
446
|
+
if (!this.adapter.sdrain) throw unsupported(this.adapter, "sdrain");
|
|
447
|
+
return this.adapter.sdrain(this.prefixed(key));
|
|
448
|
+
}
|
|
207
449
|
};
|
|
450
|
+
function unsupported(adapter, method) {
|
|
451
|
+
return new Error(`The "${adapter.id}" KV adapter does not support ${method}()`);
|
|
452
|
+
}
|
|
208
453
|
export {
|
|
209
454
|
KVManager
|
|
210
455
|
};
|