lambder 4.2.3 → 4.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/Readme.md +24 -2
- package/dist/core/Lambder.d.ts +11 -2
- package/dist/core/Lambder.js +3 -1
- package/dist/index.d.ts +1 -0
- package/dist/session/LambderSessionManager.d.ts +9 -1
- package/dist/session/LambderSessionManager.js +36 -2
- package/dist/stores/LambderDdbCache.d.ts +15 -4
- package/dist/stores/LambderDdbCache.js +51 -48
- package/dist/stores/LambderDdbCompression.d.ts +24 -2
- package/dist/stores/LambderDdbCompression.js +31 -9
- package/dist/stores/LambderDdbIdempotency.d.ts +19 -11
- package/dist/stores/LambderDdbIdempotency.js +19 -28
- package/package.json +1 -1
package/Readme.md
CHANGED
|
@@ -2,6 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
Lambder is a highly opinionated dynamic serverless framework designed to facilitate the management and implementation of routes and APIs within AWS Lambda functions, specifically tailored for TypeScript projects. It provides a streamlined approach to handling HTTP requests, managing sessions, and defining API routes, making serverless application development more intuitive and structured.
|
|
4
4
|
|
|
5
|
+
**New in 4.3:**
|
|
6
|
+
|
|
7
|
+
- **Compressed sessions**: `session.data` is stored Brotli-compressed by default, as `dataBr` + `dataBytes` on the record, the same scheme LambderDdbCache and LambderDdbIdempotency use (one shared implementation). A session that caches roles, permissions or product lists shrinks 2-3x and stays within one DynamoDB read unit for longer. `session.compression` is `true` by default (the same as `{ minBytes: 0 }`: every record compressed); `false` turns it off and `{ minBytes }` compresses only from that JSON size. Records written under either setting read back, so it can be switched on or off on a live table.
|
|
8
|
+
|
|
9
|
+
- **One compression option everywhere**: `LambderDdbCache`, `LambderDdbIdempotency` and sessions take the same `compression` option (`true` for that store's defaults, `false` for off, `{ minBytes, quality }` to override), resolved by one shared function, and each store records a value's encoding so the option can be switched on or off on a live table. Defaults keep the previous behavior: the cache compresses everything, the idempotency store from 1KB. `compressionQuality` on the cache and idempotency store is replaced by `compression: { quality }`, and HTTP `compression` accepts `true` as `{ minBytes: 860 }`.
|
|
10
|
+
|
|
5
11
|
**New in 4.2:**
|
|
6
12
|
|
|
7
13
|
- **Rate-limit budgets**: a policy's `budget` is `"perApi"` (default: each referencing API gets its own counter, so the numbers are a per-API ceiling and three APIs on a 60/min policy allow one IP 180/min in total) or `"perPolicy"` (one counter shared by every API referencing the policy). The policy is the group, and two separate shared budgets are two policies.
|
|
@@ -302,6 +308,7 @@ const lambder = initLambder<SessionData>().create({
|
|
|
302
308
|
tableRegion: "us-east-1",
|
|
303
309
|
sessionSalt: "CHANGE-THIS-TO-A-SECURE-RANDOM-STRING",
|
|
304
310
|
enableSlidingExpiration: true, // Optional: extend session on each access
|
|
311
|
+
compression: true, // Optional: Brotli-compress session.data at rest (default true; false to disable, or { minBytes })
|
|
305
312
|
// Optionally customize cookie names (defaults: LMDRSESSIONTKID, LMDRSESSIONCSTK)
|
|
306
313
|
tokenCookieKey: "MY_SESSION_TOKEN",
|
|
307
314
|
csrfCookieKey: "MY_CSRF_TOKEN",
|
|
@@ -353,6 +360,21 @@ Semantics:
|
|
|
353
360
|
- Records created before `dataRefresh` was enabled renew on their first read.
|
|
354
361
|
- `updateSessionData()` marks data fresh (it was just written deliberately); `regenerateSession()` carries the old freshness stamp over.
|
|
355
362
|
|
|
363
|
+
#### Session data at rest (`compression`)
|
|
364
|
+
|
|
365
|
+
`session.data` is stored Brotli-compressed by default: the record carries the data's JSON as Brotli bytes in `dataBr` beside its byte length in `dataBytes`, in place of a plain `data` attribute. It is the scheme `LambderDdbCache` and `LambderDdbIdempotency` already use, from one shared implementation, and the byte length both bounds the decompression and verifies it, so a truncated record fails to decode rather than decoding to something else. Session data that caches roles, permissions or product lists typically shrinks 2-3x, which keeps a growing session within one DynamoDB read unit (4KB for the consistent reads sessions use) and one write unit (1KB) for longer.
|
|
366
|
+
|
|
367
|
+
```typescript
|
|
368
|
+
session: {
|
|
369
|
+
// ...
|
|
370
|
+
compression: true, // default: every record compressed, the same as { minBytes: 0 }
|
|
371
|
+
// compression: { minBytes: 1024 } compresses only records whose JSON is 1KB+
|
|
372
|
+
// compression: false stores data as a plain attribute
|
|
373
|
+
}
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
`quality` (Brotli 0-11, default 5) is also accepted; the option (`LambderCompressionOption`) is the same one `LambderDdbCache` and `LambderDdbIdempotency` take. Reads accept both record shapes, so the setting can be switched on or off on a live table: records written under the other setting keep reading, and each is rewritten in the current shape on its next write (a sliding-expiration or `dataRefresh` write included). A compressed record that fails to decode is treated like any malformed record: no session.
|
|
377
|
+
|
|
356
378
|
#### Session Controller
|
|
357
379
|
|
|
358
380
|
Access the session controller with `lambder.getSessionController(ctx)`:
|
|
@@ -644,13 +666,13 @@ Request flow per API: session (session APIs) → idempotency replay lookup → r
|
|
|
644
666
|
|
|
645
667
|
Preflight input slices (guard `apiInput`/`guardInput` values, rate-limit `apiInput` keys) answer a rejection through the same path as the API's own schema: `setApiInputValidationErrorHandler` when set, otherwise the standard 422 body. One failure, one shape, whichever schema rejected it.
|
|
646
668
|
|
|
647
|
-
**Idempotency semantics**: the client sends an `idempotencyKey` per call (see LambderCaller below); generate it once per logical operation with `LambderCaller.createIdempotencyKey()` and reuse it on retries. Keys must be 16-200 characters and UNGUESSABLE random (shorter keys refuse with 400): on session APIs the scope is session + API name + key, and on public APIs it is the key itself + API name, deliberately NOT the client IP, because the retry idempotency exists for (a timeout followed by a network switch) frequently arrives from a new IP. Concurrent duplicates of an in-flight request refuse with 409, repeats of a completed one replay the stored response verbatim until the TTL (response headers included, so headers set via `res.setHeader`/`res.addHeader` replay too), and a crashed original releases its claim so a retry actually retries. The replay rule for failures: RESPONSES are stored and replayed, refusals returned as envelopes (`res.api(null, { errorMessage })`) and thrown responses (`res.die.*`) included; EXCEPTIONS are not, so a thrown `LambderApiError`/`refuse()` releases the claim and a retry re-executes and decides afresh. Stored bodies of 1KB or more are Brotli-compressed (the same scheme as LambderDdbCache
|
|
669
|
+
**Idempotency semantics**: the client sends an `idempotencyKey` per call (see LambderCaller below); generate it once per logical operation with `LambderCaller.createIdempotencyKey()` and reuse it on retries. Keys must be 16-200 characters and UNGUESSABLE random (shorter keys refuse with 400): on session APIs the scope is session + API name + key, and on public APIs it is the key itself + API name, deliberately NOT the client IP, because the retry idempotency exists for (a timeout followed by a network switch) frequently arrives from a new IP. Concurrent duplicates of an in-flight request refuse with 409, repeats of a completed one replay the stored response verbatim until the TTL (response headers included, so headers set via `res.setHeader`/`res.addHeader` replay too), and a crashed original releases its claim so a retry actually retries. The replay rule for failures: RESPONSES are stored and replayed, refusals returned as envelopes (`res.api(null, { errorMessage })`) and thrown responses (`res.die.*`) included; EXCEPTIONS are not, so a thrown `LambderApiError`/`refuse()` releases the claim and a retry re-executes and decides afresh. Stored bodies of 1KB or more are Brotli-compressed by default (the same scheme and `compression` option as LambderDdbCache: `true`, `false`, or `{ minBytes, quality }`, default `{ minBytes: 1024, quality: 5 }`; records of either shape read back, so it can be switched on a live table): JSON envelopes typically shrink 5-10x, which cuts DynamoDB write cost, and the ~350KB item budget applies to the COMPRESSED bytes, so even large responses usually stay replayable. Responses with status ≥ 500, bodies over the budget even compressed, and responses that set cookies are never stored (replaying one request's Set-Cookie, e.g. session tokens, into another would be wrong; such APIs still get in-flight 409 dedupe, just not replays). Claims are owner-checked, so an original that stalls past the pending window can no longer overwrite or delete the claim a retry has since taken. Requests without a key execute normally.
|
|
648
670
|
|
|
649
671
|
Also enforced at registration: **duplicate API names throw** (dispatch is first-match, so a second registration of the same name would be silently dead code).
|
|
650
672
|
|
|
651
673
|
### DynamoDB Cache (LambderDdbCache)
|
|
652
674
|
|
|
653
|
-
Standalone, persistent JSON cache backed by a DynamoDB table (`pk`/`sk` keys + `expiresAt` TTL attribute, same shape as the session table). Items are prefixed `CACHE#<namespace>#`, and the rate limiter (`RL#`) and idempotency store (`IDEM#`) prefix theirs too, so all three non-session systems can share one table without collisions; keep sessions in their own table for IAM scoping. Brotli-compressed values, in-memory LRU layer, single-flight deduplication, a DynamoDB lease so only one Lambda fills a missing key, and fail-open semantics. Server-only. **Full guide with table setup: [docs/DDB_CACHE.md](./docs/DDB_CACHE.md).**
|
|
675
|
+
Standalone, persistent JSON cache backed by a DynamoDB table (`pk`/`sk` keys + `expiresAt` TTL attribute, same shape as the session table). Items are prefixed `CACHE#<namespace>#`, and the rate limiter (`RL#`) and idempotency store (`IDEM#`) prefix theirs too, so all three non-session systems can share one table without collisions; keep sessions in their own table for IAM scoping. Brotli-compressed values (the shared `compression` option), in-memory LRU layer, single-flight deduplication, a DynamoDB lease so only one Lambda fills a missing key, and fail-open semantics. Server-only. **Full guide with table setup: [docs/DDB_CACHE.md](./docs/DDB_CACHE.md).**
|
|
654
676
|
|
|
655
677
|
```typescript
|
|
656
678
|
import { LambderDdbCache } from "lambder";
|
package/dist/core/Lambder.d.ts
CHANGED
|
@@ -6,6 +6,7 @@ import { LambderResponse, type LambderHttpResponse } from "./LambderResponse.js"
|
|
|
6
6
|
import { type ConditionFunction, type LambderRouteMatcher, type PathParamsOf } from "./LambderRouting.js";
|
|
7
7
|
import { type LambderCorsConfig } from "./LambderCors.js";
|
|
8
8
|
import { type LambderSessionDataRefreshConfig } from "../session/LambderSessionManager.js";
|
|
9
|
+
import type { LambderCompressionOption } from "../stores/LambderDdbCompression.js";
|
|
9
10
|
import LambderSessionController, { type LambderSessionCookieOptions } from "../session/LambderSessionController.js";
|
|
10
11
|
import { type LambderPublicFilesOptions } from "./LambderPublicFiles.js";
|
|
11
12
|
import type { LambderApiGuard, LambderGuardMetaMap, LambderGuardsOption, LambderGuardDataOf, LambderGuardInputsOf } from "../policies/LambderApiGuards.js";
|
|
@@ -88,6 +89,14 @@ export type LambderSessionOptions<TSessionData = any> = {
|
|
|
88
89
|
* semantics.
|
|
89
90
|
*/
|
|
90
91
|
dataRefresh?: LambderSessionDataRefreshConfig<TSessionData>;
|
|
92
|
+
/**
|
|
93
|
+
* Brotli compression of session.data at rest. `true` (the default)
|
|
94
|
+
* compresses every record, the same as `{ minBytes: 0 }`; `false` turns
|
|
95
|
+
* it off; `{ minBytes }` compresses only records whose JSON is at least
|
|
96
|
+
* that many bytes. Records written under either setting read back, so
|
|
97
|
+
* it can be switched on or off on a live table.
|
|
98
|
+
*/
|
|
99
|
+
compression?: LambderCompressionOption;
|
|
91
100
|
};
|
|
92
101
|
/**
|
|
93
102
|
* Everything an instance is configured with, in ONE declaration: base
|
|
@@ -101,8 +110,8 @@ export type LambderCreateOptions<TSessionData = any> = {
|
|
|
101
110
|
publicPath?: string;
|
|
102
111
|
apiPath?: string;
|
|
103
112
|
apiVersion?: string;
|
|
104
|
-
/** Automatic gzip for compressible responses.
|
|
105
|
-
compression?:
|
|
113
|
+
/** Automatic gzip for compressible responses. `true` (the default) is `{ minBytes: 860 }`; `false` disables it. */
|
|
114
|
+
compression?: boolean | {
|
|
106
115
|
minBytes?: number;
|
|
107
116
|
};
|
|
108
117
|
/** Automatic ETag + If-None-Match 304 on GET/HEAD 200 responses. Default: true. */
|
package/dist/core/Lambder.js
CHANGED
|
@@ -72,7 +72,8 @@ export default class Lambder {
|
|
|
72
72
|
this.finalizeOptions = {
|
|
73
73
|
compression: options.compression === false
|
|
74
74
|
? false
|
|
75
|
-
: { minBytes: options.compression
|
|
75
|
+
: { minBytes: (typeof options.compression === "object" ? options.compression.minBytes : undefined)
|
|
76
|
+
?? DEFAULT_FINALIZE_OPTIONS.compression.minBytes },
|
|
76
77
|
etag: options.etag ?? DEFAULT_FINALIZE_OPTIONS.etag,
|
|
77
78
|
maxResponseBytes: options.maxResponseBytes ?? DEFAULT_FINALIZE_OPTIONS.maxResponseBytes,
|
|
78
79
|
};
|
|
@@ -90,6 +91,7 @@ export default class Lambder {
|
|
|
90
91
|
enableSlidingExpiration: session.enableSlidingExpiration,
|
|
91
92
|
slidingWriteIntervalSeconds: session.slidingWriteIntervalSeconds,
|
|
92
93
|
dataRefresh: session.dataRefresh,
|
|
94
|
+
compression: session.compression,
|
|
93
95
|
});
|
|
94
96
|
this.sessionCookieOptions = session.cookie ?? {};
|
|
95
97
|
if (session.tokenCookieKey)
|
package/dist/index.d.ts
CHANGED
|
@@ -19,6 +19,7 @@ export { LambderPublicFilesHandler } from "./core/LambderPublicFiles.js";
|
|
|
19
19
|
export type { LambderPublicFilesOptions } from "./core/LambderPublicFiles.js";
|
|
20
20
|
export type { LambderSessionCookieOptions } from "./session/LambderSessionController.js";
|
|
21
21
|
export type { LambderSessionContext, LambderCreatedSession, LambderSessionDataRefreshConfig } from "./session/LambderSessionManager.js";
|
|
22
|
+
export type { LambderCompressionOption, LambderCompressionConfig } from "./stores/LambderDdbCompression.js";
|
|
22
23
|
export { LambderSessionDataRefreshError, LambderSessionReadError } from "./session/LambderSessionManager.js";
|
|
23
24
|
export { LambderDdbCache } from "./stores/LambderDdbCache.js";
|
|
24
25
|
export type { LambderDdbCacheOptions, LambderDdbCacheSetOptions, LambderDdbCacheGetOrSetOptions, } from "./stores/LambderDdbCache.js";
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { type LambderCompressionOption } from "../stores/LambderDdbCompression.js";
|
|
1
2
|
export type LambderSessionContext<SessionData = any> = {
|
|
2
3
|
[x: string]: any;
|
|
3
4
|
/**
|
|
@@ -81,7 +82,8 @@ export default class LambderSessionManager {
|
|
|
81
82
|
private enableSlidingExpiration;
|
|
82
83
|
private slidingWriteIntervalSeconds;
|
|
83
84
|
private dataRefresh;
|
|
84
|
-
|
|
85
|
+
private compression;
|
|
86
|
+
constructor({ tableName, tableRegion, partitionKey, sortKey, sessionSalt, enableSlidingExpiration, slidingWriteIntervalSeconds, dataRefresh, compression, }: {
|
|
85
87
|
tableName: string;
|
|
86
88
|
tableRegion: string;
|
|
87
89
|
partitionKey: string;
|
|
@@ -90,6 +92,7 @@ export default class LambderSessionManager {
|
|
|
90
92
|
enableSlidingExpiration?: boolean;
|
|
91
93
|
slidingWriteIntervalSeconds?: number;
|
|
92
94
|
dataRefresh?: LambderSessionDataRefreshConfig;
|
|
95
|
+
compression?: LambderCompressionOption;
|
|
93
96
|
});
|
|
94
97
|
private sessionUserKeyHasher;
|
|
95
98
|
/**
|
|
@@ -101,6 +104,11 @@ export default class LambderSessionManager {
|
|
|
101
104
|
private hashToken;
|
|
102
105
|
private constantTimeCompare;
|
|
103
106
|
private ddbGetItem;
|
|
107
|
+
/**
|
|
108
|
+
* Persists a session record. With compression on, `data` is stored as
|
|
109
|
+
* Brotli bytes (`dataBr`) beside its JSON byte length (`dataBytes`)
|
|
110
|
+
* once the JSON reaches minBytes; otherwise it stays a plain attribute.
|
|
111
|
+
*/
|
|
104
112
|
private ddbPutItem;
|
|
105
113
|
private ddbDeleteItem;
|
|
106
114
|
private ddbQueryAllByPartitionKey;
|
|
@@ -1,6 +1,16 @@
|
|
|
1
1
|
import crypto from "crypto";
|
|
2
2
|
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
|
|
3
3
|
import { DynamoDBDocumentClient, QueryCommand, DeleteCommand, PutCommand, GetCommand } from "@aws-sdk/lib-dynamodb";
|
|
4
|
+
import { brotliCompressText, brotliRestoreText, resolveCompressionOption, } from "../stores/LambderDdbCompression.js";
|
|
5
|
+
/**
|
|
6
|
+
* Session compression defaults: every record compressed (see
|
|
7
|
+
* LambderCompressionOption for the option's shape and toggle semantics).
|
|
8
|
+
* A compressed record carries the data's JSON as Brotli bytes (`dataBr`)
|
|
9
|
+
* beside its byte length (`dataBytes`), the scheme LambderDdbCache and
|
|
10
|
+
* LambderDdbIdempotency use; below minBytes, or with compression off, the
|
|
11
|
+
* record keeps a plain `data` attribute.
|
|
12
|
+
*/
|
|
13
|
+
const SESSION_COMPRESSION_DEFAULTS = { minBytes: 0, quality: 5 };
|
|
4
14
|
/**
|
|
5
15
|
* Wraps errors thrown by the dataRefresh callback so they stay
|
|
6
16
|
* distinguishable from "no session": fetchSessionIfExists() swallows missing
|
|
@@ -35,7 +45,8 @@ export default class LambderSessionManager {
|
|
|
35
45
|
enableSlidingExpiration;
|
|
36
46
|
slidingWriteIntervalSeconds;
|
|
37
47
|
dataRefresh;
|
|
38
|
-
|
|
48
|
+
compression;
|
|
49
|
+
constructor({ tableName, tableRegion, partitionKey, sortKey, sessionSalt, enableSlidingExpiration = true, slidingWriteIntervalSeconds, dataRefresh, compression, }) {
|
|
39
50
|
this.tableName = tableName;
|
|
40
51
|
this.sessionSalt = sessionSalt;
|
|
41
52
|
this.partitionKey = partitionKey;
|
|
@@ -43,6 +54,7 @@ export default class LambderSessionManager {
|
|
|
43
54
|
this.enableSlidingExpiration = enableSlidingExpiration;
|
|
44
55
|
this.slidingWriteIntervalSeconds = slidingWriteIntervalSeconds ?? null;
|
|
45
56
|
this.dataRefresh = dataRefresh ?? null;
|
|
57
|
+
this.compression = resolveCompressionOption(compression, SESSION_COMPRESSION_DEFAULTS);
|
|
46
58
|
const ddbClient = new DynamoDBClient({ region: tableRegion });
|
|
47
59
|
this.ddbDocumentClient = DynamoDBDocumentClient.from(ddbClient);
|
|
48
60
|
}
|
|
@@ -74,7 +86,21 @@ export default class LambderSessionManager {
|
|
|
74
86
|
return null;
|
|
75
87
|
}
|
|
76
88
|
;
|
|
77
|
-
|
|
89
|
+
/**
|
|
90
|
+
* Persists a session record. With compression on, `data` is stored as
|
|
91
|
+
* Brotli bytes (`dataBr`) beside its JSON byte length (`dataBytes`)
|
|
92
|
+
* once the JSON reaches minBytes; otherwise it stays a plain attribute.
|
|
93
|
+
*/
|
|
94
|
+
async ddbPutItem(session) {
|
|
95
|
+
const { data, ...item } = session;
|
|
96
|
+
const raw = this.compression && Buffer.from(JSON.stringify(data), "utf8");
|
|
97
|
+
if (this.compression && raw && raw.byteLength >= this.compression.minBytes) {
|
|
98
|
+
item.dataBr = await brotliCompressText(raw, this.compression.quality);
|
|
99
|
+
item.dataBytes = raw.byteLength;
|
|
100
|
+
}
|
|
101
|
+
else {
|
|
102
|
+
item.data = data;
|
|
103
|
+
}
|
|
78
104
|
return await this.ddbDocumentClient.send(new PutCommand({ TableName: this.tableName, Item: item, }));
|
|
79
105
|
}
|
|
80
106
|
;
|
|
@@ -169,6 +195,14 @@ export default class LambderSessionManager {
|
|
|
169
195
|
}
|
|
170
196
|
if (!session)
|
|
171
197
|
return null;
|
|
198
|
+
// A compressed record (see ddbPutItem) decodes back into `data`. One
|
|
199
|
+
// that fails to decode throws, which the controller treats like any
|
|
200
|
+
// malformed record: no session.
|
|
201
|
+
if (session.dataBr) {
|
|
202
|
+
session.data = JSON.parse(await brotliRestoreText(session.dataBr, session.dataBytes));
|
|
203
|
+
delete session.dataBr;
|
|
204
|
+
delete session.dataBytes;
|
|
205
|
+
}
|
|
172
206
|
if (!session.csrfTokenHash)
|
|
173
207
|
return null;
|
|
174
208
|
if (!session.sessionKey)
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
|
|
2
|
+
import { type LambderCompressionOption } from "./LambderDdbCompression.js";
|
|
2
3
|
export interface LambderDdbCacheOptions {
|
|
3
4
|
tableName: string;
|
|
4
5
|
region?: string;
|
|
@@ -7,7 +8,13 @@ export interface LambderDdbCacheOptions {
|
|
|
7
8
|
namespace?: string;
|
|
8
9
|
defaultTtlSeconds?: number;
|
|
9
10
|
chunkBytes?: number;
|
|
10
|
-
|
|
11
|
+
/**
|
|
12
|
+
* Brotli compression of stored values. `true` (the default) is
|
|
13
|
+
* `{ minBytes: 0, quality: 5 }`: every value compressed; `false` stores
|
|
14
|
+
* values plain; an object overrides the defaults. The manifest records
|
|
15
|
+
* each value's encoding, so it can be switched on or off on a live table.
|
|
16
|
+
*/
|
|
17
|
+
compression?: LambderCompressionOption;
|
|
11
18
|
maxValueBytes?: number;
|
|
12
19
|
memoryMaxBytes?: number;
|
|
13
20
|
client?: DynamoDBClient;
|
|
@@ -22,8 +29,10 @@ export interface LambderDdbCacheGetOrSetOptions extends LambderDdbCacheSetOption
|
|
|
22
29
|
/**
|
|
23
30
|
* Persistent JSON cache backed by DynamoDB.
|
|
24
31
|
*
|
|
25
|
-
* Values are Brotli-compressed
|
|
26
|
-
*
|
|
32
|
+
* Values are Brotli-compressed by default (`compression` option; the manifest
|
|
33
|
+
* records each value's encoding, so the option can be switched on a live
|
|
34
|
+
* table). Values within the safe DynamoDB item budget are stored directly in
|
|
35
|
+
* the manifest for a single-request read; larger values are
|
|
27
36
|
* split into versioned binary chunks. A manifest is written only after every
|
|
28
37
|
* chunk succeeds, so readers see either the previous complete version or the
|
|
29
38
|
* new complete version. DynamoDB TTL is cleanup only; every read also checks
|
|
@@ -41,7 +50,7 @@ export declare class LambderDdbCache {
|
|
|
41
50
|
private readonly client;
|
|
42
51
|
private readonly defaultTtlSeconds;
|
|
43
52
|
private readonly chunkBytes;
|
|
44
|
-
private readonly
|
|
53
|
+
private readonly compression;
|
|
45
54
|
private readonly maxValueBytes;
|
|
46
55
|
private readonly memory;
|
|
47
56
|
private readonly inFlight;
|
|
@@ -64,6 +73,8 @@ export declare class LambderDdbCache {
|
|
|
64
73
|
private readChunks;
|
|
65
74
|
private invalidateManifest;
|
|
66
75
|
private batchWrite;
|
|
76
|
+
/** The JSON text of a stored payload. */
|
|
77
|
+
private decode;
|
|
67
78
|
private remember;
|
|
68
79
|
private normalizeKey;
|
|
69
80
|
private partitionKey;
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { BatchWriteItemCommand, DeleteItemCommand, DynamoDBClient, GetItemCommand, PutItemCommand, QueryCommand, } from "@aws-sdk/client-dynamodb";
|
|
2
2
|
import { getCrypto } from "../shared/node-polyfills.js";
|
|
3
|
-
import { brotliCompressText,
|
|
3
|
+
import { brotliCompressText, brotliRestoreText, resolveCompressionOption, } from "./LambderDdbCompression.js";
|
|
4
4
|
import { LRUCache } from "lru-cache";
|
|
5
5
|
const DEFAULT_TTL_SECONDS = 365 * 24 * 60 * 60;
|
|
6
6
|
const DEFAULT_CHUNK_BYTES = 350 * 1024;
|
|
@@ -11,10 +11,13 @@ const META_SORT_KEY = "meta";
|
|
|
11
11
|
const LOCK_SORT_KEY = "lock";
|
|
12
12
|
const BATCH_WRITE_LIMIT = 25;
|
|
13
13
|
const MAX_BATCH_RETRIES = 8;
|
|
14
|
+
/** Every value compressed by default; see the `compression` option. */
|
|
15
|
+
const COMPRESSION_DEFAULTS = { minBytes: 0, quality: 5 };
|
|
14
16
|
// Node builtins are loaded lazily through node-polyfills so this module can
|
|
15
17
|
// sit in a frontend bundle's import graph (via the package root) without
|
|
16
18
|
// breaking; using the cache at runtime still requires Node. Brotli helpers
|
|
17
|
-
// are shared with LambderDdbIdempotency via
|
|
19
|
+
// are shared with LambderDdbIdempotency and LambderSessionManager via
|
|
20
|
+
// ./LambderDdbCompression.js.
|
|
18
21
|
const requireCrypto = async () => {
|
|
19
22
|
const crypto = await getCrypto();
|
|
20
23
|
if (!crypto)
|
|
@@ -39,8 +42,10 @@ const sleep = (milliseconds) => new Promise((resolve) => setTimeout(resolve, mil
|
|
|
39
42
|
/**
|
|
40
43
|
* Persistent JSON cache backed by DynamoDB.
|
|
41
44
|
*
|
|
42
|
-
* Values are Brotli-compressed
|
|
43
|
-
*
|
|
45
|
+
* Values are Brotli-compressed by default (`compression` option; the manifest
|
|
46
|
+
* records each value's encoding, so the option can be switched on a live
|
|
47
|
+
* table). Values within the safe DynamoDB item budget are stored directly in
|
|
48
|
+
* the manifest for a single-request read; larger values are
|
|
44
49
|
* split into versioned binary chunks. A manifest is written only after every
|
|
45
50
|
* chunk succeeds, so readers see either the previous complete version or the
|
|
46
51
|
* new complete version. DynamoDB TTL is cleanup only; every read also checks
|
|
@@ -58,7 +63,7 @@ export class LambderDdbCache {
|
|
|
58
63
|
client;
|
|
59
64
|
defaultTtlSeconds;
|
|
60
65
|
chunkBytes;
|
|
61
|
-
|
|
66
|
+
compression;
|
|
62
67
|
maxValueBytes;
|
|
63
68
|
memory;
|
|
64
69
|
inFlight = new Map();
|
|
@@ -76,17 +81,14 @@ export class LambderDdbCache {
|
|
|
76
81
|
if (this.chunkBytes > MAX_SAFE_CHUNK_BYTES) {
|
|
77
82
|
throw new Error(`chunkBytes must not exceed ${MAX_SAFE_CHUNK_BYTES}`);
|
|
78
83
|
}
|
|
79
|
-
this.
|
|
80
|
-
if (!Number.isInteger(this.compressionQuality) || this.compressionQuality < 0 || this.compressionQuality > 11) {
|
|
81
|
-
throw new Error("compressionQuality must be an integer from 0 to 11");
|
|
82
|
-
}
|
|
84
|
+
this.compression = resolveCompressionOption(options.compression, COMPRESSION_DEFAULTS);
|
|
83
85
|
this.maxValueBytes = positiveInteger(options.maxValueBytes ?? DEFAULT_MAX_VALUE_BYTES, "maxValueBytes");
|
|
84
86
|
const memoryMaxBytes = options.memoryMaxBytes ?? DEFAULT_MEMORY_BYTES;
|
|
85
87
|
this.memory = memoryMaxBytes === 0
|
|
86
88
|
? null
|
|
87
89
|
: new LRUCache({
|
|
88
90
|
maxSize: positiveInteger(memoryMaxBytes, "memoryMaxBytes"),
|
|
89
|
-
sizeCalculation: (entry) => entry.
|
|
91
|
+
sizeCalculation: (entry) => entry.stored.length,
|
|
90
92
|
});
|
|
91
93
|
this.client = options.client ?? new DynamoDBClient({ region: options.region ?? "us-east-1" });
|
|
92
94
|
}
|
|
@@ -96,10 +98,7 @@ export class LambderDdbCache {
|
|
|
96
98
|
const nowSeconds = this.nowSeconds();
|
|
97
99
|
if (cached && cached.expiresAt > nowSeconds) {
|
|
98
100
|
try {
|
|
99
|
-
|
|
100
|
-
if (output.length === cached.uncompressedBytes) {
|
|
101
|
-
return JSON.parse(output.toString("utf8"));
|
|
102
|
-
}
|
|
101
|
+
return JSON.parse(await this.decode(cached.stored, cached.encoding, cached.uncompressedBytes));
|
|
103
102
|
}
|
|
104
103
|
catch {
|
|
105
104
|
// Fall through to DynamoDB; the in-memory copy is disposable.
|
|
@@ -112,20 +111,16 @@ export class LambderDdbCache {
|
|
|
112
111
|
if (!manifest || manifest.expiresAt <= nowSeconds)
|
|
113
112
|
return undefined;
|
|
114
113
|
try {
|
|
115
|
-
const
|
|
116
|
-
if (
|
|
117
|
-
throw new Error("
|
|
114
|
+
const stored = manifest.inlineData ?? await this.readChunks(pk, manifest);
|
|
115
|
+
if (stored.length !== manifest.storedBytes) {
|
|
116
|
+
throw new Error("stored byte length does not match manifest");
|
|
118
117
|
}
|
|
119
|
-
if (await sha256(
|
|
120
|
-
throw new Error("
|
|
118
|
+
if (await sha256(stored) !== manifest.checksum) {
|
|
119
|
+
throw new Error("stored checksum does not match manifest");
|
|
121
120
|
}
|
|
122
|
-
const
|
|
123
|
-
if (output.length !== manifest.uncompressedBytes) {
|
|
124
|
-
throw new Error("uncompressed byte length does not match manifest");
|
|
125
|
-
}
|
|
126
|
-
const json = output.toString("utf8");
|
|
121
|
+
const json = await this.decode(stored, manifest.encoding, manifest.uncompressedBytes);
|
|
127
122
|
const parsed = JSON.parse(json);
|
|
128
|
-
this.remember(normalizedKey,
|
|
123
|
+
this.remember(normalizedKey, stored, manifest.encoding, manifest.uncompressedBytes, manifest.expiresAt);
|
|
129
124
|
return parsed;
|
|
130
125
|
}
|
|
131
126
|
catch (error) {
|
|
@@ -155,18 +150,20 @@ export class LambderDdbCache {
|
|
|
155
150
|
if (input.length > this.maxValueBytes) {
|
|
156
151
|
throw new Error(`Cache value exceeds maxValueBytes (${input.length} > ${this.maxValueBytes})`);
|
|
157
152
|
}
|
|
158
|
-
const
|
|
159
|
-
|
|
160
|
-
|
|
153
|
+
const brotli = this.compression && input.length >= this.compression.minBytes ? this.compression : null;
|
|
154
|
+
const encoding = brotli ? "br" : "identity";
|
|
155
|
+
const stored = brotli ? await brotliCompressText(input, brotli.quality) : input;
|
|
156
|
+
if (stored.length > this.maxValueBytes) {
|
|
157
|
+
throw new Error(`Stored cache value exceeds maxValueBytes (${stored.length} > ${this.maxValueBytes})`);
|
|
161
158
|
}
|
|
162
159
|
const pk = await this.partitionKey(normalizedKey);
|
|
163
160
|
const version = `${Date.now().toString(36)}-${await randomUUID()}`;
|
|
164
161
|
const expiresAt = this.nowSeconds() + ttlSeconds;
|
|
165
162
|
const chunks = [];
|
|
166
|
-
const inline =
|
|
163
|
+
const inline = stored.length <= this.chunkBytes;
|
|
167
164
|
if (!inline) {
|
|
168
|
-
for (let offset = 0; offset <
|
|
169
|
-
chunks.push(
|
|
165
|
+
for (let offset = 0; offset < stored.length; offset += this.chunkBytes) {
|
|
166
|
+
chunks.push(stored.subarray(offset, offset + this.chunkBytes));
|
|
170
167
|
}
|
|
171
168
|
}
|
|
172
169
|
const writes = chunks.map((chunk, index) => ({
|
|
@@ -187,16 +184,16 @@ export class LambderDdbCache {
|
|
|
187
184
|
sk: { S: META_SORT_KEY },
|
|
188
185
|
version: { S: version },
|
|
189
186
|
chunkCount: { N: String(chunks.length) },
|
|
190
|
-
|
|
187
|
+
storedBytes: { N: String(stored.length) },
|
|
191
188
|
uncompressedBytes: { N: String(input.length) },
|
|
192
|
-
checksum: { S: await sha256(
|
|
193
|
-
encoding: { S:
|
|
189
|
+
checksum: { S: await sha256(stored) },
|
|
190
|
+
encoding: { S: encoding },
|
|
194
191
|
createdAt: { N: String(this.nowSeconds()) },
|
|
195
192
|
expiresAt: { N: String(expiresAt) },
|
|
196
|
-
...(inline ? { data: { B:
|
|
193
|
+
...(inline ? { data: { B: stored } } : {}),
|
|
197
194
|
},
|
|
198
195
|
}));
|
|
199
|
-
this.remember(normalizedKey,
|
|
196
|
+
this.remember(normalizedKey, stored, encoding, input.length, expiresAt);
|
|
200
197
|
}
|
|
201
198
|
async delete(key) {
|
|
202
199
|
const normalizedKey = this.normalizeKey(key);
|
|
@@ -350,37 +347,39 @@ export class LambderDdbCache {
|
|
|
350
347
|
const version = item.version?.S;
|
|
351
348
|
const encoding = item.encoding?.S;
|
|
352
349
|
const chunkCount = Number(item.chunkCount?.N);
|
|
353
|
-
const
|
|
350
|
+
const storedBytes = Number(item.storedBytes?.N);
|
|
354
351
|
const uncompressedBytes = Number(item.uncompressedBytes?.N);
|
|
355
352
|
const expiresAt = Number(item.expiresAt?.N);
|
|
356
353
|
const checksum = item.checksum?.S;
|
|
357
354
|
const inlineData = item.data?.B == null ? undefined : Buffer.from(item.data.B);
|
|
358
355
|
const validInline = inlineData !== undefined &&
|
|
359
356
|
chunkCount === 0 &&
|
|
360
|
-
inlineData.length ===
|
|
361
|
-
|
|
357
|
+
inlineData.length === storedBytes &&
|
|
358
|
+
storedBytes <= this.chunkBytes;
|
|
362
359
|
const validChunks = inlineData === undefined &&
|
|
363
360
|
chunkCount > 0 &&
|
|
364
|
-
chunkCount === Math.ceil(
|
|
361
|
+
chunkCount === Math.ceil(storedBytes / this.chunkBytes);
|
|
365
362
|
if (!version ||
|
|
366
|
-
encoding !== "br" ||
|
|
363
|
+
(encoding !== "br" && encoding !== "identity") ||
|
|
367
364
|
!checksum ||
|
|
368
365
|
!Number.isSafeInteger(chunkCount) ||
|
|
369
366
|
chunkCount < 0 ||
|
|
370
|
-
!Number.isSafeInteger(
|
|
371
|
-
|
|
372
|
-
|
|
367
|
+
!Number.isSafeInteger(storedBytes) ||
|
|
368
|
+
storedBytes < 0 ||
|
|
369
|
+
storedBytes > this.maxValueBytes ||
|
|
373
370
|
!Number.isSafeInteger(uncompressedBytes) ||
|
|
374
371
|
uncompressedBytes < 0 ||
|
|
375
372
|
uncompressedBytes > this.maxValueBytes ||
|
|
373
|
+
(encoding === "identity" && storedBytes !== uncompressedBytes) ||
|
|
376
374
|
!Number.isSafeInteger(expiresAt) ||
|
|
377
375
|
(!validInline && !validChunks)) {
|
|
378
376
|
return undefined;
|
|
379
377
|
}
|
|
380
378
|
return {
|
|
381
379
|
version,
|
|
380
|
+
encoding,
|
|
382
381
|
chunkCount,
|
|
383
|
-
|
|
382
|
+
storedBytes,
|
|
384
383
|
uncompressedBytes,
|
|
385
384
|
checksum,
|
|
386
385
|
expiresAt,
|
|
@@ -417,7 +416,7 @@ export class LambderDdbCache {
|
|
|
417
416
|
throw new Error(`DynamoDB cache entry has an invalid chunk index at ${index}`);
|
|
418
417
|
}
|
|
419
418
|
}
|
|
420
|
-
return Buffer.concat(chunks.map((chunk) => chunk.data), manifest.
|
|
419
|
+
return Buffer.concat(chunks.map((chunk) => chunk.data), manifest.storedBytes);
|
|
421
420
|
}
|
|
422
421
|
async invalidateManifest(pk, version) {
|
|
423
422
|
try {
|
|
@@ -451,13 +450,17 @@ export class LambderDdbCache {
|
|
|
451
450
|
}
|
|
452
451
|
}
|
|
453
452
|
}
|
|
454
|
-
|
|
453
|
+
/** The JSON text of a stored payload. */
|
|
454
|
+
async decode(stored, encoding, uncompressedBytes) {
|
|
455
|
+
return encoding === "br" ? await brotliRestoreText(stored, uncompressedBytes) : stored.toString("utf8");
|
|
456
|
+
}
|
|
457
|
+
remember(key, stored, encoding, uncompressedBytes, expiresAt) {
|
|
455
458
|
if (!this.memory)
|
|
456
459
|
return;
|
|
457
460
|
const ttl = expiresAt * 1000 - Date.now();
|
|
458
461
|
if (ttl <= 0)
|
|
459
462
|
return;
|
|
460
|
-
this.memory.set(key, {
|
|
463
|
+
this.memory.set(key, { stored, encoding, uncompressedBytes, expiresAt }, { ttl });
|
|
461
464
|
}
|
|
462
465
|
normalizeKey(key) {
|
|
463
466
|
if (typeof key !== "string" || !key.trim())
|
|
@@ -1,3 +1,25 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The compression option every store shares. `true` is on with the store's
|
|
3
|
+
* defaults, `false` is off, an object overrides the defaults: `minBytes` is
|
|
4
|
+
* the UTF-8 size from which a value is stored compressed (0: always),
|
|
5
|
+
* `quality` is Brotli 0-11. Values below minBytes are stored plain, and a
|
|
6
|
+
* store reads records of either shape, so the option can be switched on or
|
|
7
|
+
* off on a live table: records written under the other setting keep
|
|
8
|
+
* reading, and each is rewritten in the current shape on its next write.
|
|
9
|
+
*/
|
|
10
|
+
export type LambderCompressionConfig = {
|
|
11
|
+
minBytes?: number;
|
|
12
|
+
quality?: number;
|
|
13
|
+
};
|
|
14
|
+
export type LambderCompressionOption = boolean | LambderCompressionConfig;
|
|
15
|
+
export type LambderCompressionSettings = Required<LambderCompressionConfig>;
|
|
16
|
+
/** Resolves a store's compression option against its defaults: null when off. */
|
|
17
|
+
export declare const resolveCompressionOption: (option: LambderCompressionOption | undefined, defaults: LambderCompressionSettings) => LambderCompressionSettings | null;
|
|
1
18
|
export declare const brotliCompressText: (input: Buffer, quality: number) => Promise<Buffer>;
|
|
2
|
-
/**
|
|
3
|
-
|
|
19
|
+
/**
|
|
20
|
+
* Restores text stored as Brotli bytes beside its declared UTF-8 byte length.
|
|
21
|
+
* The length bounds the decompression and the output must match it exactly,
|
|
22
|
+
* so a truncated or tampered record fails instead of decoding to something
|
|
23
|
+
* else.
|
|
24
|
+
*/
|
|
25
|
+
export declare const brotliRestoreText: (compressed: Uint8Array, declaredBytes: number) => Promise<string>;
|
|
@@ -1,8 +1,18 @@
|
|
|
1
1
|
import { getZlib } from "../shared/node-polyfills.js";
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
2
|
+
/** Resolves a store's compression option against its defaults: null when off. */
|
|
3
|
+
export const resolveCompressionOption = (option, defaults) => {
|
|
4
|
+
if (option === false)
|
|
5
|
+
return null;
|
|
6
|
+
const config = option === true || option === undefined ? {} : option;
|
|
7
|
+
const settings = { minBytes: config.minBytes ?? defaults.minBytes, quality: config.quality ?? defaults.quality };
|
|
8
|
+
if (!Number.isSafeInteger(settings.minBytes) || settings.minBytes < 0) {
|
|
9
|
+
throw new Error("compression.minBytes must be a non-negative integer");
|
|
10
|
+
}
|
|
11
|
+
if (!Number.isInteger(settings.quality) || settings.quality < 0 || settings.quality > 11) {
|
|
12
|
+
throw new Error("compression.quality must be an integer from 0 to 11");
|
|
13
|
+
}
|
|
14
|
+
return settings;
|
|
15
|
+
};
|
|
6
16
|
const requireZlib = async () => {
|
|
7
17
|
const zlib = await getZlib();
|
|
8
18
|
if (!zlib)
|
|
@@ -25,15 +35,27 @@ export const brotliCompressText = async (input, quality) => {
|
|
|
25
35
|
});
|
|
26
36
|
});
|
|
27
37
|
};
|
|
28
|
-
/**
|
|
29
|
-
|
|
38
|
+
/**
|
|
39
|
+
* Restores text stored as Brotli bytes beside its declared UTF-8 byte length.
|
|
40
|
+
* The length bounds the decompression and the output must match it exactly,
|
|
41
|
+
* so a truncated or tampered record fails instead of decoding to something
|
|
42
|
+
* else.
|
|
43
|
+
*/
|
|
44
|
+
export const brotliRestoreText = async (compressed, declaredBytes) => {
|
|
45
|
+
if (!Number.isSafeInteger(declaredBytes) || declaredBytes <= 0) {
|
|
46
|
+
throw new Error("compressed record is missing its byte length");
|
|
47
|
+
}
|
|
30
48
|
const zlib = await requireZlib();
|
|
31
|
-
|
|
32
|
-
zlib.brotliDecompress(
|
|
49
|
+
const output = await new Promise((resolve, reject) => {
|
|
50
|
+
zlib.brotliDecompress(Buffer.from(compressed), { maxOutputLength: declaredBytes }, (error, result) => {
|
|
33
51
|
if (error)
|
|
34
52
|
reject(error);
|
|
35
53
|
else
|
|
36
|
-
resolve(
|
|
54
|
+
resolve(result);
|
|
37
55
|
});
|
|
38
56
|
});
|
|
57
|
+
if (output.length !== declaredBytes) {
|
|
58
|
+
throw new Error("decompressed length does not match the record");
|
|
59
|
+
}
|
|
60
|
+
return output.toString("utf8");
|
|
39
61
|
};
|
|
@@ -1,11 +1,17 @@
|
|
|
1
1
|
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
|
|
2
|
+
import { type LambderCompressionOption } from "./LambderDdbCompression.js";
|
|
2
3
|
export interface LambderDdbIdempotencyOptions {
|
|
3
4
|
tableName: string;
|
|
4
5
|
region?: string;
|
|
5
6
|
/** Partition key prefix, keeps records separated from other systems in a shared table. Default: "IDEM". */
|
|
6
7
|
keyPrefix?: string;
|
|
7
|
-
/**
|
|
8
|
-
|
|
8
|
+
/**
|
|
9
|
+
* Brotli compression of stored bodies. `true` (the default) is
|
|
10
|
+
* `{ minBytes: 1024, quality: 5 }`; `false` stores every body plain; an
|
|
11
|
+
* object overrides the defaults. Records of either shape read back, so
|
|
12
|
+
* it can be switched on or off on a live table.
|
|
13
|
+
*/
|
|
14
|
+
compression?: LambderCompressionOption;
|
|
9
15
|
client?: DynamoDBClient;
|
|
10
16
|
}
|
|
11
17
|
export type LambderIdempotencyDoneRecord = {
|
|
@@ -35,10 +41,11 @@ export type LambderIdempotencyBeginResult = {
|
|
|
35
41
|
* and loses the scope to a retry can no longer overwrite or delete the
|
|
36
42
|
* retry's claim (both settle calls become silent no-ops instead).
|
|
37
43
|
*
|
|
38
|
-
* Stored bodies
|
|
39
|
-
* LambderDdbCache): the bodies are JSON
|
|
40
|
-
* 5-10x, which cuts DynamoDB write units
|
|
41
|
-
* item budget instead of skipping replay
|
|
44
|
+
* Stored bodies are Brotli-compressed from 1KB by default (same scheme as
|
|
45
|
+
* LambderDdbCache, see the `compression` option): the bodies are JSON
|
|
46
|
+
* envelopes that typically shrink 5-10x, which cuts DynamoDB write units
|
|
47
|
+
* and lets large responses fit the item budget instead of skipping replay
|
|
48
|
+
* storage.
|
|
42
49
|
*
|
|
43
50
|
* Table shape: string hash key `pk`, string range key `sk`, TTL on
|
|
44
51
|
* `expiresAt`. Items are prefixed `IDEM#` by default, so the table can be
|
|
@@ -48,7 +55,7 @@ export type LambderIdempotencyBeginResult = {
|
|
|
48
55
|
export declare class LambderDdbIdempotency {
|
|
49
56
|
readonly tableName: string;
|
|
50
57
|
readonly keyPrefix: string;
|
|
51
|
-
private readonly
|
|
58
|
+
private readonly compression;
|
|
52
59
|
private readonly client;
|
|
53
60
|
constructor(options: LambderDdbIdempotencyOptions);
|
|
54
61
|
private itemKey;
|
|
@@ -74,10 +81,11 @@ export declare class LambderDdbIdempotency {
|
|
|
74
81
|
}): Promise<LambderIdempotencyBeginResult>;
|
|
75
82
|
/**
|
|
76
83
|
* Store the response for replays, overwriting the pending claim. Bodies
|
|
77
|
-
*
|
|
78
|
-
* JSON envelopes, which typically shrink 5-10x), cutting
|
|
79
|
-
* units and letting large responses fit the item budget;
|
|
80
|
-
* stay plain.
|
|
84
|
+
* from the compression option's minBytes are stored Brotli-compressed
|
|
85
|
+
* (they are JSON envelopes, which typically shrink 5-10x), cutting
|
|
86
|
+
* DynamoDB write units and letting large responses fit the item budget;
|
|
87
|
+
* smaller bodies, or all of them with compression off, stay plain.
|
|
88
|
+
* Returns:
|
|
81
89
|
*
|
|
82
90
|
* - "stored": the record is in place and will replay.
|
|
83
91
|
* - "too-large": even compressed, the body exceeds the item budget;
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
import crypto from "crypto";
|
|
2
2
|
import { DynamoDBClient, PutItemCommand, GetItemCommand, DeleteItemCommand, } from "@aws-sdk/client-dynamodb";
|
|
3
|
-
import { brotliCompressText,
|
|
4
|
-
/** Bodies
|
|
5
|
-
const
|
|
3
|
+
import { brotliCompressText, brotliRestoreText, resolveCompressionOption, } from "./LambderDdbCompression.js";
|
|
4
|
+
/** Bodies of 1KB or more are stored Brotli-compressed by default; smaller ones stay plain. */
|
|
5
|
+
const COMPRESSION_DEFAULTS = { minBytes: 1024, quality: 5 };
|
|
6
6
|
/**
|
|
7
7
|
* Stored-body budget inside DynamoDB's 400KB item limit (headers, keys and
|
|
8
8
|
* attributes need headroom). Applies to the bytes actually stored, so a
|
|
@@ -22,10 +22,11 @@ const MAX_STORED_BODY_BYTES = 350_000;
|
|
|
22
22
|
* and loses the scope to a retry can no longer overwrite or delete the
|
|
23
23
|
* retry's claim (both settle calls become silent no-ops instead).
|
|
24
24
|
*
|
|
25
|
-
* Stored bodies
|
|
26
|
-
* LambderDdbCache): the bodies are JSON
|
|
27
|
-
* 5-10x, which cuts DynamoDB write units
|
|
28
|
-
* item budget instead of skipping replay
|
|
25
|
+
* Stored bodies are Brotli-compressed from 1KB by default (same scheme as
|
|
26
|
+
* LambderDdbCache, see the `compression` option): the bodies are JSON
|
|
27
|
+
* envelopes that typically shrink 5-10x, which cuts DynamoDB write units
|
|
28
|
+
* and lets large responses fit the item budget instead of skipping replay
|
|
29
|
+
* storage.
|
|
29
30
|
*
|
|
30
31
|
* Table shape: string hash key `pk`, string range key `sk`, TTL on
|
|
31
32
|
* `expiresAt`. Items are prefixed `IDEM#` by default, so the table can be
|
|
@@ -35,17 +36,14 @@ const MAX_STORED_BODY_BYTES = 350_000;
|
|
|
35
36
|
export class LambderDdbIdempotency {
|
|
36
37
|
tableName;
|
|
37
38
|
keyPrefix;
|
|
38
|
-
|
|
39
|
+
compression;
|
|
39
40
|
client;
|
|
40
41
|
constructor(options) {
|
|
41
42
|
if (!options.tableName.trim())
|
|
42
43
|
throw new Error("tableName is required");
|
|
43
44
|
this.tableName = options.tableName;
|
|
44
45
|
this.keyPrefix = options.keyPrefix ?? "IDEM";
|
|
45
|
-
this.
|
|
46
|
-
if (!Number.isInteger(this.compressionQuality) || this.compressionQuality < 0 || this.compressionQuality > 11) {
|
|
47
|
-
throw new Error("compressionQuality must be an integer from 0 to 11");
|
|
48
|
-
}
|
|
46
|
+
this.compression = resolveCompressionOption(options.compression, COMPRESSION_DEFAULTS);
|
|
49
47
|
this.client = options.client ?? new DynamoDBClient(options.region ? { region: options.region } : {});
|
|
50
48
|
}
|
|
51
49
|
itemKey(scopeKey) {
|
|
@@ -67,16 +65,8 @@ export class LambderDdbIdempotency {
|
|
|
67
65
|
/** A stored item's response body: plain (`body`) or Brotli (`bodyBr` + `bodyBytes`). */
|
|
68
66
|
static async readItemBody(item) {
|
|
69
67
|
const compressed = item.bodyBr?.B;
|
|
70
|
-
if (compressed)
|
|
71
|
-
|
|
72
|
-
if (!declaredBytes)
|
|
73
|
-
throw new Error("LambderDdbIdempotency: compressed record is missing bodyBytes.");
|
|
74
|
-
const output = await brotliDecompressText(Buffer.from(compressed), declaredBytes);
|
|
75
|
-
if (output.length !== declaredBytes) {
|
|
76
|
-
throw new Error("LambderDdbIdempotency: stored body length does not match its record.");
|
|
77
|
-
}
|
|
78
|
-
return output.toString("utf8");
|
|
79
|
-
}
|
|
68
|
+
if (compressed)
|
|
69
|
+
return await brotliRestoreText(compressed, Number(item.bodyBytes?.N ?? 0));
|
|
80
70
|
return item.body?.S ?? "";
|
|
81
71
|
}
|
|
82
72
|
/**
|
|
@@ -150,10 +140,11 @@ export class LambderDdbIdempotency {
|
|
|
150
140
|
}
|
|
151
141
|
/**
|
|
152
142
|
* Store the response for replays, overwriting the pending claim. Bodies
|
|
153
|
-
*
|
|
154
|
-
* JSON envelopes, which typically shrink 5-10x), cutting
|
|
155
|
-
* units and letting large responses fit the item budget;
|
|
156
|
-
* stay plain.
|
|
143
|
+
* from the compression option's minBytes are stored Brotli-compressed
|
|
144
|
+
* (they are JSON envelopes, which typically shrink 5-10x), cutting
|
|
145
|
+
* DynamoDB write units and letting large responses fit the item budget;
|
|
146
|
+
* smaller bodies, or all of them with compression off, stay plain.
|
|
147
|
+
* Returns:
|
|
157
148
|
*
|
|
158
149
|
* - "stored": the record is in place and will replay.
|
|
159
150
|
* - "too-large": even compressed, the body exceeds the item budget;
|
|
@@ -165,8 +156,8 @@ export class LambderDdbIdempotency {
|
|
|
165
156
|
const nowSeconds = Math.floor(Date.now() / 1000);
|
|
166
157
|
const rawBody = Buffer.from(body, "utf8");
|
|
167
158
|
let bodyAttributes;
|
|
168
|
-
if (rawBody.byteLength >=
|
|
169
|
-
const compressed = await brotliCompressText(rawBody, this.
|
|
159
|
+
if (this.compression && rawBody.byteLength >= this.compression.minBytes) {
|
|
160
|
+
const compressed = await brotliCompressText(rawBody, this.compression.quality);
|
|
170
161
|
if (compressed.byteLength > MAX_STORED_BODY_BYTES)
|
|
171
162
|
return "too-large";
|
|
172
163
|
// bodyBytes bounds and verifies decompression on read.
|