@aws-blocks/bb-distributed-table 0.1.2 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/DESIGN.md +219 -0
- package/README.md +42 -0
- package/dist/errors.d.ts +25 -0
- package/dist/errors.d.ts.map +1 -1
- package/dist/errors.js +71 -0
- package/dist/index.aws.d.ts +10 -2
- package/dist/index.aws.d.ts.map +1 -1
- package/dist/index.aws.js +22 -8
- package/dist/index.cdk.d.ts +1 -1
- package/dist/index.cdk.d.ts.map +1 -1
- package/dist/index.mock.d.ts +10 -2
- package/dist/index.mock.d.ts.map +1 -1
- package/dist/index.mock.js +21 -7
- package/dist/index.test.js +172 -0
- package/dist/parity.test.js +49 -0
- package/dist/types.d.ts +51 -0
- package/dist/types.d.ts.map +1 -1
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/package.json +14 -4
- package/src/errors.ts +85 -0
- package/src/index.aws.ts +27 -8
- package/src/index.cdk.ts +1 -1
- package/src/index.mock.ts +26 -7
- package/src/index.test.ts +205 -0
- package/src/parity.test.ts +76 -0
- package/src/types.ts +54 -0
- package/src/version.ts +1 -1
package/DESIGN.md
ADDED
|
@@ -0,0 +1,219 @@
|
|
|
1
|
+
# DistributedTable — Design
|
|
2
|
+
|
|
3
|
+
Design document for DistributedTable. For usage, see [README.md](./README.md).
|
|
4
|
+
|
|
5
|
+
**Package:** `@aws-blocks/bb-distributed-table`
|
|
6
|
+
**Type:** Primitive (new infrastructure)
|
|
7
|
+
**AWS Service:** DynamoDB (partition key + optional sort key + GSIs)
|
|
8
|
+
|
|
9
|
+
## Design Decisions
|
|
10
|
+
|
|
11
|
+
### D-DT-1: Key object over positional arguments
|
|
12
|
+
|
|
13
|
+
**Decision:** `get()`, `delete()`, `getBatch()`, and `deleteBatch()` accept a key object (`{ userId: 'alice', orderId: '001' }`) rather than positional arguments (`'alice', '001'`).
|
|
14
|
+
|
|
15
|
+
**Rationale:**
|
|
16
|
+
|
|
17
|
+
Every major DynamoDB library uses a key object pattern:
|
|
18
|
+
- **AWS SDK v3 DocumentClient:** `Key: { userId: 'alice', orderId: '001' }`
|
|
19
|
+
- **ElectroDB:** `.get({ cityId: 'Atlanta1', mallId: 'EastPointe' })`
|
|
20
|
+
- **DynamoDB-Toolbox v1:** `.key({ pokemonId: 'pikachu1' })`
|
|
21
|
+
|
|
22
|
+
DynamoDB keys are compound by nature — a partition key and optional sort key that together identify an item. An object communicates the field names at the call site, making the code self-documenting. Positional args like `get('alice', '001')` don't tell you which value is which.
|
|
23
|
+
|
|
24
|
+
The key type is a computed `TableKey<T, K>` — a `Required<Pick<T, KeyFields>>`, not `Partial<T>`. `Partial<T>` would be too loose: it makes non-key fields optional but present, and allows omitting required key fields entirely (`table.get({})` would compile). The computed type picks exactly the key fields and makes them all required. When a sort key is defined, it is required in the key object — you cannot accidentally omit it. The class carries a third generic `K extends TableKeyConfig<T>` so the literal key config is preserved and `TableKey` resolves correctly.
|
|
25
|
+
|
|
26
|
+
This also follows the API's options-object convention (objects over positional parameters) since the key is inherently a multi-field value.
|
|
27
|
+
|
|
28
|
+
### D-DT-2: Schema is required
|
|
29
|
+
|
|
30
|
+
**Decision:** `DistributedTableOptions.schema` is required, not optional.
|
|
31
|
+
|
|
32
|
+
**Rationale:** DistributedTable's key configuration references field names from the schema (`partitionKey: 'userId'`). The schema is what makes the key type-safe — without it, there's no way to validate that key field names exist in the item type. Unlike KVStore (which stores opaque values by string key), DistributedTable operates on structured items where the schema is integral to the type system.
|
|
33
|
+
|
|
34
|
+
### D-DT-3: StandardSchemaV1 over Zod-specific types
|
|
35
|
+
|
|
36
|
+
**Decision:** Accept `StandardSchemaV1` from `@standard-schema/spec` instead of Zod-specific structural types.
|
|
37
|
+
|
|
38
|
+
**Rationale:** Building Blocks accept any StandardSchemaV1 implementation. This avoids vendor lock-in to Zod and lets customers use Valibot, ArkType, or any other conforming library. The `@standard-schema/spec` package is types-only (zero runtime). Validation uses `schema['~standard'].validate()`.
|
|
39
|
+
|
|
40
|
+
### D-DT-4: Conditional operations — ifNotExists, ifExists, ifFieldEquals
|
|
41
|
+
|
|
42
|
+
**Decision:** Support `ifNotExists` on put, `ifExists` on delete, and `ifFieldEquals` on both put and delete.
|
|
43
|
+
|
|
44
|
+
**Rationale:** DistributedTable backs structured application data that often requires coordination — idempotent creates, optimistic locking, and guarded deletes. These map directly to DynamoDB's `ConditionExpression` capabilities:
|
|
45
|
+
|
|
46
|
+
- `ifNotExists` → `attribute_not_exists(pk)` — protects create-only operations from overwriting existing items.
|
|
47
|
+
- `ifExists` → `attribute_exists(pk)` — ensures you're deleting something that's actually there (useful for audit trails, cascading deletes).
|
|
48
|
+
- `ifFieldEquals` → `#field = :value` — optimistic locking / compare-and-swap. Check that a field (e.g., `status`, `version`, `updatedAt`) matches an expected value before writing. This is the DynamoDB equivalent of `UPDATE ... WHERE version = ?` in SQL.
|
|
49
|
+
|
|
50
|
+
`ifFieldEquals` accepts `Partial<T>`, so multiple fields can be checked in a single condition (AND semantics). All condition failures throw with `error.name = 'ConditionalCheckFailedException'`, matching the AWS SDK error name.
|
|
51
|
+
|
|
52
|
+
`ifNotExists` and `ifFieldEquals` are mutually exclusive at the type level (discriminated union), as are `ifExists` and `ifFieldEquals`. DynamoDB's `ConditionExpression` could combine them, but the semantics are confusing — "create only if it doesn't exist AND the existing item's field equals X" is contradictory. The type system prevents this rather than silently picking one.
|
|
53
|
+
|
|
54
|
+
KVStore uses `ifValueEquals` (compare the entire value). DistributedTable uses `ifFieldEquals` (compare individual fields) because items are structured objects with multiple fields — comparing the entire item would be impractical and fragile.
|
|
55
|
+
|
|
56
|
+
### D-DT-5: `scan()` not `list()`
|
|
57
|
+
|
|
58
|
+
**Decision:** The full-table enumeration method is named `scan()`, not `list()`.
|
|
59
|
+
|
|
60
|
+
**Rationale:** `list` is an avoided verb because it understates the cost of a full table enumeration. `scan` communicates that every item is read and scales with total data size. The name is borrowed directly from DynamoDB to reinforce the cost implication.
|
|
61
|
+
|
|
62
|
+
### D-DT-6: Single options object for query
|
|
63
|
+
|
|
64
|
+
**Decision:** `query(options)` takes a single options object with `index`, `where`, `limit`, and `order` fields. Omitting `index` queries the primary key.
|
|
65
|
+
|
|
66
|
+
**Rationale:** A single options object is consistent with the rest of the API. The `index` field determines which key config applies to `where` — when present, `where` is typed against the GSI's key config; when absent, it's typed against the table's primary key. This is implemented as a discriminated union on `index`:
|
|
67
|
+
|
|
68
|
+
```typescript
|
|
69
|
+
// GSI query — where is typed against byDate's key config
|
|
70
|
+
orders.query({
|
|
71
|
+
index: 'byDate',
|
|
72
|
+
where: { userId: { equals: 'alice' }, createdAt: { greaterThan: 1000 } },
|
|
73
|
+
order: 'desc',
|
|
74
|
+
})
|
|
75
|
+
|
|
76
|
+
// Primary key query — where is typed against the table's primary key
|
|
77
|
+
orders.query({
|
|
78
|
+
where: { userId: { equals: 'alice' }, orderId: { beginsWith: '2024-' } },
|
|
79
|
+
})
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
The `order` field maps to DynamoDB's `ScanIndexForward` parameter (`'desc'` → `ScanIndexForward: false`). It defaults to `'asc'`.
|
|
83
|
+
|
|
84
|
+
### D-DT-7: TTL via options field
|
|
85
|
+
|
|
86
|
+
**Decision:** TTL is configured via `ttl: 'fieldName'` in the constructor options, not as a separate method or decorator.
|
|
87
|
+
|
|
88
|
+
**Rationale:** DynamoDB TTL is a table-level setting that designates one attribute as the expiration timestamp. It's a static configuration concern, not a per-item operation, so it belongs in the constructor options alongside `key` and `indexes`. The field must exist in the schema and should contain a Unix epoch timestamp in seconds. DynamoDB automatically deletes expired items in the background (typically within 48 hours of expiration).
|
|
89
|
+
|
|
90
|
+
### D-DT-8: Split `InvalidQuery` and `ItemTooLarge` rather than one `Validation` bucket
|
|
91
|
+
|
|
92
|
+
**Decision:** Pre-flight input failures surface as two intent-revealing error names — `InvalidQueryException` and `ItemTooLargeException` — instead of a single generic `ValidationException` that mirrors DynamoDB's wire name.
|
|
93
|
+
|
|
94
|
+
**Rationale:** These are two genuinely different failure modes that an earlier revision collapsed under one catch:
|
|
95
|
+
|
|
96
|
+
- **`InvalidQuery`** — the request *shape* is wrong: a missing `where` clause, a partition key not given as `{ equals: value }`, an unknown index, more than one sort-key condition, or an empty `ifFieldEquals`. Every one of these is a **caller bug** — something the caller fixes by correcting the call. There's no value in branching on which specific shape error occurred at runtime, so they share one name.
|
|
97
|
+
- **`ItemTooLarge`** — an item exceeds the 400 KB per-item limit. This is **not necessarily a caller bug**: the size of a given record may be outside the caller's control (user-supplied content, accumulated history). A caller may legitimately want to branch on it — skip the item, split it, or store a reference instead — which it cannot do if the only signal is a generic name plus message text.
|
|
98
|
+
|
|
99
|
+
A generic `ValidationException` is exactly the kind of catch-all bucket worth avoiding, and `BatchIncomplete` already establishes the Blocks-specific-name pattern in this same package. Splitting into intent-revealing names lets a customer write `isBlocksError(e, DistributedTableErrors.ItemTooLarge)` and reliably tell "this item is too big" from "my query is malformed" without string-matching the message.
|
|
100
|
+
|
|
101
|
+
**Mock/AWS parity:** the mock checks serialized byte length client-side and throws `ItemTooLarge` directly. On AWS, DynamoDB raises a generic `ValidationException` for an oversized item; the runtime narrows on the size-specific message (`size has exceeded`) and re-maps only that case to `ItemTooLarge`. Other `ValidationException` causes (malformed expressions, type mismatches) propagate unchanged. Both layers therefore surface the same `error.name`, and the shared message lives in `errors.ts` (`DistributedTableMessages.itemTooLarge`) so the two stay byte-for-byte aligned.
|
|
102
|
+
|
|
103
|
+
### D-DT-9: `readValidation` — `off | coerce | strict`, defaulting to `coerce`
|
|
104
|
+
|
|
105
|
+
> The **default choice** (`'coerce'` over `'off'`) is recorded as a cross-cutting architectural decision in [`docs/DECISIONS.md` D-015](../../docs/DECISIONS.md#d-015-distributedtable-reads-default-to-readvalidation-coerce-not-off). This section covers the per-BB mechanics.
|
|
106
|
+
|
|
107
|
+
**Decision:** Writes always validate. Reads reconcile a stored item with the schema per the `readValidation` mode, which defaults to **`'coerce'`**. `get`/`getBatch`/`query`/`scan` pass each stored value through `schema['~standard'].validate()`:
|
|
108
|
+
- **`'coerce'`** (default) — return the schema's output (defaults filled / types narrowed for transform-bearing schemas); on validation failure return the **raw** value with a `warn` log — never throws.
|
|
109
|
+
- **`'strict'`** — throw `ValidationFailed` on any item that doesn't satisfy the schema.
|
|
110
|
+
- **`'off'`** — return the raw stored value, no validation.
|
|
111
|
+
|
|
112
|
+
**Rationale:** The asymmetry "validate on write, return raw on read" breaks the documented read-modify-write update pattern after a schema change, in two ways. A newly added field is absent from the read, so `get()` returns a value that silently violates the declared type `T` and any schema `.default()` is neither applied nor persisted on write-back; and if the new field is required with no default, the `put()` half of the cycle throws `ValidationFailed`, stranding the row. Coercing on read fixes the round-trip for the coercible case (added fields with `.default()`, widened/narrowed types) by handing the caller a value the current schema accepts. It cannot invent a required-no-default value — those rows fall through to the raw + warn path and still need an explicit migration.
|
|
113
|
+
|
|
114
|
+
**Why `coerce` is the default.** The bug is that the *default* read violates `T`; a fix only users who discover a flag can enable isn't a fix. Market research across comparable typed data layers backs a coercing default: Rails ActiveRecord type-casts on load, Mongoose applies defaults/casts when hydrating, ElectroDB runs getters and returns schema-shaped items — none re-throw on stored data, and Postgres itself synthesizes an added column's `DEFAULT` at read time (`ALTER TABLE ADD COLUMN … DEFAULT`), which is coerce-on-read in all but name.
|
|
115
|
+
|
|
116
|
+
**Why not `strict` by default.** Throwing on read makes legacy/corrupt rows unreadable — you couldn't fetch them to migrate — turns one bad row into a whole-`scan`/`getBatch` outage, and violates the project rule that reads return data or `null` and throw only for violated preconditions. Every surveyed library that ships a strict read (DynamoDB-Toolbox `format()`, the `zod-firebase` converter) also ships an escape hatch. So `strict` is offered as an opt-in for tables that want mismatch treated as corruption, not as the default.
|
|
117
|
+
|
|
118
|
+
**Why keep `off`.** The raw escape hatch (ElectroDB's `data:'raw'`, Mongoose's `.lean()`) is needed for hot paths, trusted write-validated data, and reading rows that can't yet be coerced during a migration.
|
|
119
|
+
|
|
120
|
+
**Best-effort coercion caveat.** Standard Schema only guarantees `validate()` *checks*; it does not require transformation. Zod fills defaults/coerces, but a check-only Valibot/ArkType schema returns its input unchanged — so `'coerce'` degrades to pass-through for those validators. Documented as best-effort; coercion never fabricates a required value.
|
|
121
|
+
|
|
122
|
+
**`coerce` preserves unknown keys.** Most validators discard unrecognized keys when they produce their output (Zod `.strip()`s by default), so a stored row carrying attributes beyond the current schema — fields from an older schema version, or columns another writer owns — would lose them on read, and a read-modify-write (`get()` → `put()`) would persist that loss, silently deleting data. To prevent this, `'coerce'` deep-merges the coerced value **over the raw stored item** (`mergeCoercedOverRaw` in `errors.ts`, built on `defu`'s `createDefu`): schema output wins per key (defaults filled, types narrowed), while keys the schema doesn't declare — including nested ones — survive. Arrays are treated as opaque leaves: the coerced array replaces the raw array wholesale (defu concatenates arrays by default, which would *duplicate* elements here since the coerced array is derived from the raw one, so that one behavior is overridden). The merge runs only when both the stored item and the coerced value are plain objects; defu's `__proto__`/`constructor` guard is retained. `'coerce'` is therefore additive-and-lossless for the common cases (added fields, nested unknowns, type coercion). The one residual: a schema whose *transform* deliberately renames or removes a key — the merge resurrects the old key, so transform-heavy schemas should use `'strict'` or `'off'`. `defu` is used rather than a hand-rolled merge because treating arrays as leaves requires custom config in every merge library regardless, and defu is a maintained, prototype-safe, zero-dependency ESM package.
|
|
123
|
+
|
|
124
|
+
**Mock/AWS parity:** both layers call the same `applyReadValidation()` helper in `errors.ts`, so all three modes (coerced output, raw-fallback + warn, strict throw) behave identically. `null` (a missing item) passes through untouched in every mode, preserving not-found semantics.
|
|
125
|
+
|
|
126
|
+
## Infrastructure (CDK)
|
|
127
|
+
|
|
128
|
+
Creates a single DynamoDB table:
|
|
129
|
+
|
|
130
|
+
- **Partition key:** Configurable name and type via `options.key.partitionKey`
|
|
131
|
+
- **Sort key:** Configurable name and type via `options.key.sortKey` (optional)
|
|
132
|
+
- **Global secondary indexes:** Managed by a custom resource (see below)
|
|
133
|
+
- **TTL:** Enabled via `TimeToLiveSpecification` when `options.ttl` is set
|
|
134
|
+
- **Billing mode:** PAY_PER_REQUEST
|
|
135
|
+
- **Table name:** Derived from `scope.fullId` (includes stack name for uniqueness)
|
|
136
|
+
- **Removal policy:** DESTROY (sandbox), configurable for production
|
|
137
|
+
- **Permissions:** `grantReadWriteData` to the parent scope's handler automatically, plus explicit `dynamodb:Query` on `index/*`
|
|
138
|
+
|
|
139
|
+
Attribute types are inferred from the schema at synth time. The CDK layer probes the schema's `StandardSchemaV1.validate()` method with a test value of `0` for each key field — if the field accepts it without issues, it's numeric (`AttributeType.NUMBER`), otherwise string (`AttributeType.STRING`). This is schema-library-agnostic and uses only the standard validation interface.
|
|
140
|
+
|
|
141
|
+
### GSI Management Custom Resource
|
|
142
|
+
|
|
143
|
+
DynamoDB only allows one GSI change per `UpdateTable` call, and each change can take minutes to hours on large tables (DynamoDB must backfill the index). A standard CDK `Table` construct cannot express multi-GSI changes in a single deployment. DistributedTable uses a CloudFormation custom resource with the CDK `Provider` framework's async pattern to manage GSIs declaratively.
|
|
144
|
+
|
|
145
|
+
**Architecture:**
|
|
146
|
+
|
|
147
|
+
- **`onEvent` handler** — Invoked once per CloudFormation Create/Update/Delete. Compares the table's current GSIs against the desired state. If already matching, returns immediately. Otherwise, initiates the first GSI change and returns `IN_PROGRESS`. For sandbox deployments, takes a fast path (see below).
|
|
148
|
+
- **`isCompleteHandler`** — Polled by the Provider framework every 10 seconds (configurable via `queryInterval`), up to a 2-hour total timeout. On each poll:
|
|
149
|
+
1. If the table is busy (a GSI is still creating/deleting), returns `IsComplete: false`.
|
|
150
|
+
2. If the table is idle and matches the desired state, returns `IsComplete: true`.
|
|
151
|
+
3. If the table is idle but doesn't match, initiates the next GSI change and returns `IsComplete: false`.
|
|
152
|
+
|
|
153
|
+
**Ordering:** Creations are performed before deletions when possible. This ensures new access patterns are available before old ones are removed. The one exception is schema-mismatched GSIs — if a desired GSI has the same name as an existing GSI but different key schema, the old one must be deleted first because DynamoDB doesn't support in-place GSI modification.
|
|
154
|
+
|
|
155
|
+
**IAM policies are split by environment:**
|
|
156
|
+
|
|
157
|
+
| Permission | Production | Sandbox |
|
|
158
|
+
|------------|-----------|---------|
|
|
159
|
+
| `dynamodb:DescribeTable` | ✅ | ✅ |
|
|
160
|
+
| `dynamodb:UpdateTable` | ✅ | ✅ |
|
|
161
|
+
| `dynamodb:DeleteTable` | ❌ | ✅ |
|
|
162
|
+
| `dynamodb:CreateTable` | ❌ | ✅ |
|
|
163
|
+
| `dynamodb:Scan` | ❌ | ✅ |
|
|
164
|
+
| `dynamodb:BatchWriteItem` | ❌ | ✅ |
|
|
165
|
+
|
|
166
|
+
Production deployments can only add/remove GSIs via `UpdateTable`. The GSI manager lambda cannot drop or recreate the table, scan its data, or batch-write items. This prevents accidental data loss from a misconfigured custom resource.
|
|
167
|
+
|
|
168
|
+
### Sandbox Deployment Model
|
|
169
|
+
|
|
170
|
+
Sandbox deployments use a **drop-and-recreate** fast path that bypasses the sequential one-at-a-time GSI limitation:
|
|
171
|
+
|
|
172
|
+
1. Scan all items from the existing table (backup to memory)
|
|
173
|
+
2. Delete the table
|
|
174
|
+
3. Wait for deletion to complete
|
|
175
|
+
4. Create a new table with all desired GSIs defined upfront (DynamoDB allows multiple GSIs at table creation time)
|
|
176
|
+
5. Wait for the table and all GSIs to become ACTIVE
|
|
177
|
+
6. Restore all items via `BatchWriteItem`
|
|
178
|
+
|
|
179
|
+
This is dramatically faster than sequential GSI creation (seconds vs. minutes/hours) but **destroys and recreates the table**. It is only used when `SandboxMode` is `true` (set by the CDK layer based on the `sandboxMode` context variable). The IAM policy for production deployments does not grant the permissions needed for this path, so it cannot execute in production even if `SandboxMode` were accidentally set.
|
|
180
|
+
|
|
181
|
+
⚠️ **Data loss risk in sandbox:** If the Lambda times out during step 1 (scan) on a very large table, or if the restore in step 6 fails partway through, data may be lost. This is acceptable for sandbox (ephemeral dev environments) but is why the fast path is never used in production.
|
|
182
|
+
|
|
183
|
+
## Serialization
|
|
184
|
+
|
|
185
|
+
Items are stored as DynamoDB JSON (marshalled via `@aws-sdk/lib-dynamodb` DocumentClient). The type parameter `T` is inferred from the StandardSchemaV1 schema at compile time. Runtime validation occurs on `put` and `putBatch` before writing. Both mock and AWS runtime use the same validation path (`schema['~standard'].validate()`).
|
|
186
|
+
|
|
187
|
+
## Mock Implementation
|
|
188
|
+
|
|
189
|
+
- Data stored in `.bb-data/{scope.fullId}/data.json` via `getMockDataDir()` from core.
|
|
190
|
+
- Data persists across dev server restarts. Customers can wipe with `rm -rf .bb-data`.
|
|
191
|
+
- Index queries implemented via in-memory filtering over the full dataset.
|
|
192
|
+
- Conditional write/delete failures throw with `error.name = 'ConditionalCheckFailedException'`.
|
|
193
|
+
- Schema validation on `put()` and `putBatch()`; throws with `error.name = 'ValidationFailedException'`.
|
|
194
|
+
- Validates 400 KB serialized item size limit.
|
|
195
|
+
- TTL is accepted in options but not enforced — items are not auto-deleted locally.
|
|
196
|
+
- `getBatch`/`putBatch`/`deleteBatch` always process every entry in one pass — the
|
|
197
|
+
in-memory store never returns `UnprocessedKeys`/`UnprocessedItems`, so the AWS
|
|
198
|
+
runtime's retry loop and `BatchIncomplete` exhaustion error have no mock equivalent
|
|
199
|
+
(see parity gaps below).
|
|
200
|
+
- `ifFieldEquals` compares values with an order-independent structural deep-equal.
|
|
201
|
+
Object/Map keys are compared as a set (DynamoDB Maps are an unordered collection
|
|
202
|
+
of name-value pairs), while arrays remain order-sensitive (DynamoDB Lists are
|
|
203
|
+
ordered). The unordered-Map equality of `=` in a DynamoDB condition expression
|
|
204
|
+
was confirmed against real DynamoDB: storing `{ role: 'admin', level: 5 }` and
|
|
205
|
+
issuing a conditional `put` with `ifFieldEquals: { level: 5, role: 'admin' }`
|
|
206
|
+
(keys reversed) passes the condition, so the mock's order-independent compare
|
|
207
|
+
matches AWS.
|
|
208
|
+
|
|
209
|
+
### Mock vs AWS Behavior Differences
|
|
210
|
+
|
|
211
|
+
| Behavior Difference | Impact | Mitigation |
|
|
212
|
+
|------------|--------|------------|
|
|
213
|
+
| No throughput limits | Code that would be throttled in AWS succeeds locally | Document the gap; recommend sandbox testing for throughput-sensitive flows |
|
|
214
|
+
| Batch retry exhaustion (`BatchIncomplete`) is AWS-runtime only | Under sustained throttling, AWS batch ops retry with backoff and throw `DistributedTableErrors.BatchIncomplete` once `MAX_BATCH_ATTEMPTS` is reached; the mock never throttles so this path is unreachable locally | Error name and message are single-sourced in `errors.ts` so catch-site handling (`isBlocksError(e, DistributedTableErrors.BatchIncomplete)`) is identical regardless of backend. Exercise throttling/backoff behavior in sandbox |
|
|
215
|
+
| No item size limit enforcement beyond 400 KB check | Edge cases around DynamoDB marshalling overhead | Mock validates serialized JSON size, which is a close approximation |
|
|
216
|
+
| Immediate consistency (vs eventual for GSIs) | GSI reads always reflect the latest write locally | No mitigation — eventual consistency is inherently non-deterministic. Document the gap; recommend sandbox testing |
|
|
217
|
+
| No IAM enforcement | Permission errors only surface in AWS | No mitigation at mock level — IAM is handled by CDK grants automatically |
|
|
218
|
+
| In-memory index queries vs DynamoDB index reads | Index query performance characteristics differ; no GSI throughput throttling | No mitigation — correctness is preserved. Performance testing requires sandbox |
|
|
219
|
+
| TTL not enforced locally | Items with expired TTL remain in mock data | Document the gap; test TTL behavior in sandbox |
|
package/README.md
CHANGED
|
@@ -6,6 +6,8 @@ Structured data storage backed by DynamoDB with secondary indexes and rich query
|
|
|
6
6
|
|
|
7
7
|
**When NOT to use:** If you only need single-key lookups, use `KVStore`. If you need full SQL (joins, aggregations), use `Database`.
|
|
8
8
|
|
|
9
|
+
> Design & mock parity details: [DESIGN.md](./DESIGN.md)
|
|
10
|
+
|
|
9
11
|
## API
|
|
10
12
|
|
|
11
13
|
```typescript
|
|
@@ -38,6 +40,7 @@ const table = new DistributedTable(scope, id, options)
|
|
|
38
40
|
| `key` | `TableKeyConfig<T>` | Yes | Primary key configuration: `{ partitionKey, sortKey? }`. Field names must exist in the schema. |
|
|
39
41
|
| `indexes` | `Record<string, TableKeyConfig<T>>` | No | Global secondary index definitions. |
|
|
40
42
|
| `ttl` | `keyof T & string` | No | Enable DynamoDB TTL on the specified attribute. The field should contain a Unix epoch timestamp in seconds. |
|
|
43
|
+
| `readValidation` | `'off' \| 'coerce' \| 'strict'` | No | How reads (`get`/`getBatch`/`query`/`scan`) reconcile a stored item with `schema`. `'coerce'` (**default**) returns the coerced value and, on failure, the raw value + a warning (never throws); `'strict'` throws `ValidationFailed` on a non-conforming item; `'off'` returns the raw value with no validation. See [Reads and schema evolution](#reads-and-schema-evolution). |
|
|
41
44
|
| `table` | `ExternalTableRef` | No | Wrap an existing DynamoDB table instead of creating one. |
|
|
42
45
|
| `logger` | `ChildLogger` | No | Optional logger for internal operations. When omitted, a default Logger at error level is created. |
|
|
43
46
|
|
|
@@ -127,6 +130,45 @@ All condition failures throw with `error.name === DistributedTableErrors.Conditi
|
|
|
127
130
|
|
|
128
131
|
> **No partial update:** There is no `update()` or `patch()` method. To change a field, do a read-modify-write — `get()` the item, mutate it, then `put()` the full item back. For safe concurrent updates, pass `{ ifFieldEquals: { version: <previous> } }` to `put()` so the write fails (via `ConditionalCheckFailed`) if another writer changed the item in the meantime (optimistic locking).
|
|
129
132
|
|
|
133
|
+
### Reads and schema evolution
|
|
134
|
+
|
|
135
|
+
Writes always validate against `schema`. Reads reconcile a stored item with the schema according to the **`readValidation`** option, which matters after a schema change: a row written under an older schema may no longer match the declared type `T` — a newly added field is **absent** from the read (so the value silently violates `T`, and a `.default()` is neither applied nor persisted on write-back), and a **required, no-default** field makes the read-modify-write cycle above **fail on the write** as `put()` rejects the legacy shape.
|
|
136
|
+
|
|
137
|
+
`readValidation` has three modes:
|
|
138
|
+
|
|
139
|
+
| Mode | On read | On a non-conforming item |
|
|
140
|
+
|---|---|---|
|
|
141
|
+
| **`'coerce'`** (default) | returns the schema's coerced output (defaults filled, types narrowed) | returns the **raw** value + logs a warning — **never throws** |
|
|
142
|
+
| **`'strict'`** | validates against the schema | **throws** `ValidationFailed` |
|
|
143
|
+
| **`'off'`** | returns the raw stored value, no validation | returns it as-is |
|
|
144
|
+
|
|
145
|
+
The default `'coerce'` closes the schema-evolution gap so a legacy row round-trips cleanly:
|
|
146
|
+
|
|
147
|
+
```typescript
|
|
148
|
+
const orderSchema = z.object({
|
|
149
|
+
orderId: z.string(),
|
|
150
|
+
total: z.number(),
|
|
151
|
+
currency: z.string().default('USD'), // added in a later release
|
|
152
|
+
});
|
|
153
|
+
|
|
154
|
+
const orders = new DistributedTable(scope, 'orders', {
|
|
155
|
+
schema: orderSchema,
|
|
156
|
+
key: { partitionKey: 'orderId' },
|
|
157
|
+
// readValidation: 'coerce' is the default
|
|
158
|
+
});
|
|
159
|
+
|
|
160
|
+
const order = await orders.get({ orderId: 'o1' }); // legacy row → { …, currency: 'USD' }
|
|
161
|
+
await orders.put({ ...order, total: 20 }); // round-trips without ValidationFailed
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
**`'coerce'` never throws:** a value that genuinely can't be coerced (e.g. a required field with no default) is returned **as-is** with a warning, so unrecoverable rows stay readable for migration and `get()` never throws for a bad row.
|
|
165
|
+
|
|
166
|
+
> **Best-effort coercion (validator-dependent).** Coercion depends on the schema *transforming* its input. Zod fills defaults and casts; a check-only Standard Schema validator (some Valibot/ArkType schemas) validates without transforming, so `'coerce'` returns the value unchanged for those — it never invents data.
|
|
167
|
+
|
|
168
|
+
> **`'coerce'` preserves stored keys not in the schema.** Many schemas discard unrecognized keys when they validate (Zod object schemas `.strip()` by default), which would drop attributes a stored row carries beyond the current schema — and a read-modify-write would then persist the loss. To avoid that, `'coerce'` deep-merges the coerced output **over the raw stored item**: schema output wins per key (defaults filled, types narrowed), while attributes the schema doesn't declare — from an older schema version, or columns another writer owns — are **kept**. Arrays are replaced wholesale (the coerced array wins), and nested unknown keys are preserved too. So a routine read-modify-write never silently deletes a field you didn't touch. (Use **`'strict'`** if you instead want a schema mismatch to be rejected, or **`'off'`** to skip the schema pass entirely.)
|
|
169
|
+
|
|
170
|
+
Choose **`'strict'`** for tables where a schema mismatch should be treated as corruption and rejected (note: one bad row then fails the whole `query`/`scan`/`getBatch`). Choose **`'off'`** for hot paths, data you trust was written through this schema, or to read (and preserve) rows you can't yet coerce during a migration.
|
|
171
|
+
|
|
130
172
|
### Error Handling
|
|
131
173
|
|
|
132
174
|
Errors thrown by DistributedTable carry an `error.name` you can match with `isBlocksError`:
|
package/dist/errors.d.ts
CHANGED
|
@@ -1,3 +1,6 @@
|
|
|
1
|
+
import type { StandardSchemaV1 } from '@standard-schema/spec';
|
|
2
|
+
import type { ChildLogger } from '@aws-blocks/bb-logger';
|
|
3
|
+
import type { ReadValidationMode } from './types.js';
|
|
1
4
|
/**
|
|
2
5
|
* Typed error constants for DistributedTable. Use with `isBlocksError()` in catch blocks.
|
|
3
6
|
*
|
|
@@ -108,4 +111,26 @@ export declare const DistributedTableMessages: {
|
|
|
108
111
|
* its stack and requestId remain available for debugging.
|
|
109
112
|
*/
|
|
110
113
|
export declare function remapItemTooLarge(err: unknown): unknown;
|
|
114
|
+
/**
|
|
115
|
+
* @internal Reconcile a stored item with the schema on read, per the
|
|
116
|
+
* `readValidation` mode. Shared by the mock and AWS runtime so all three modes
|
|
117
|
+
* behave identically in both. `null` (a missing item) always passes through
|
|
118
|
+
* untouched, preserving not-found semantics.
|
|
119
|
+
*
|
|
120
|
+
* - `'off'` — return the raw item, no validation.
|
|
121
|
+
* - `'coerce'` — apply the schema (fill defaults / narrow types for
|
|
122
|
+
* transform-bearing schemas) **without dropping data**: the coerced output is
|
|
123
|
+
* deep-merged over the raw item, so schema output wins per key while attributes
|
|
124
|
+
* the schema doesn't declare (from an older schema version or another writer)
|
|
125
|
+
* are preserved. Without this merge, returning the bare validator output would
|
|
126
|
+
* strip unknown keys (Zod `.strip()` default), and a read-modify-write would
|
|
127
|
+
* then persist the stripped item — silently deleting stored data. On validation
|
|
128
|
+
* failure, return the **raw** item and `warn` — never throws — so drifted/legacy
|
|
129
|
+
* rows stay readable and the "reads return data or `null`" contract holds.
|
|
130
|
+
* - `'strict'` — throw `ValidationFailed` on any item that doesn't satisfy the
|
|
131
|
+
* schema.
|
|
132
|
+
*
|
|
133
|
+
* Schemas may validate synchronously or asynchronously; this awaits either.
|
|
134
|
+
*/
|
|
135
|
+
export declare function applyReadValidation<T>(mode: ReadValidationMode, schema: StandardSchemaV1<T>, item: T | null, log: Pick<ChildLogger, 'warn'>, context?: Record<string, unknown>): Promise<T | null>;
|
|
111
136
|
//# sourceMappingURL=errors.d.ts.map
|
package/dist/errors.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,uBAAuB,CAAC;AAC9D,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,uBAAuB,CAAC;AACzD,OAAO,KAAK,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAC;AAwBrD;;;;;;;;;;;;;;;;;GAiBG;AACH,eAAO,MAAM,sBAAsB;;;IAGlC;;;;;;;;;;;OAWG;;IAEH;;;;;;;;;;;;OAYG;;IAEH;;;;;;;;;OASG;;CAEM,CAAC;AAEX;;;;GAIG;AACH,wBAAgB,WAAW,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,KAAK,CAIhE;AAED;;;;;;;;;;;;;;;GAeG;AACH,wBAAgB,yBAAyB,CAAC,CAAC,SAAS,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAC1E,SAAS,EAAE,CAAC,GAAG,SAAS,GACtB,CAAC,GAAG,SAAS,CAQf;AAED;;;;GAIG;AACH,eAAO,MAAM,wBAAwB;oCACb,MAAM,GAAG,SAAS;sCAChB,MAAM;mDAEO,MAAM;wDAED,MAAM,EAAE;;mCAI7B,MAAM;0CAEC,MAAM,aAAa,MAAM,YAAY,MAAM;CAG/D,CAAC;AAEX;;;;;;;;;;;GAWG;AACH,wBAAgB,iBAAiB,CAAC,GAAG,EAAE,OAAO,GAAG,OAAO,CAOvD;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAsB,mBAAmB,CAAC,CAAC,EAC1C,IAAI,EAAE,kBAAkB,EACxB,MAAM,EAAE,gBAAgB,CAAC,CAAC,CAAC,EAC3B,IAAI,EAAE,CAAC,GAAG,IAAI,EACd,GAAG,EAAE,IAAI,CAAC,WAAW,EAAE,MAAM,CAAC,EAC9B,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAC/B,OAAO,CAAC,CAAC,GAAG,IAAI,CAAC,CAsBnB"}
|
package/dist/errors.js
CHANGED
|
@@ -1,5 +1,27 @@
|
|
|
1
1
|
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
|
2
2
|
// SPDX-License-Identifier: Apache-2.0
|
|
3
|
+
import { createDefu } from 'defu';
|
|
4
|
+
/**
|
|
5
|
+
* @internal Right-biased deep merge for the `'coerce'` read path: overlay the
|
|
6
|
+
* schema-coerced item onto the raw stored item so schema output (filled defaults,
|
|
7
|
+
* narrowed types) wins, while keys the schema stripped (e.g. attributes from an
|
|
8
|
+
* older schema version or another writer) are preserved rather than silently
|
|
9
|
+
* dropped on a read-modify-write.
|
|
10
|
+
*
|
|
11
|
+
* Arrays are treated as **opaque leaves** — the coerced array replaces the raw
|
|
12
|
+
* array wholesale. defu concatenates arrays by default, which is wrong here: the
|
|
13
|
+
* coerced array is derived from the same raw array, so concatenation would
|
|
14
|
+
* duplicate every element. This customizer overrides that one behavior; plain
|
|
15
|
+
* objects still deep-merge (so nested unknown keys survive), and defu's built-in
|
|
16
|
+
* `__proto__`/`constructor` guard is retained.
|
|
17
|
+
*/
|
|
18
|
+
const mergeCoercedOverRaw = createDefu((obj, key, value) => {
|
|
19
|
+
if (Array.isArray(obj[key]) || Array.isArray(value)) {
|
|
20
|
+
obj[key] = obj[key] ?? value;
|
|
21
|
+
return true;
|
|
22
|
+
}
|
|
23
|
+
return false;
|
|
24
|
+
});
|
|
3
25
|
/**
|
|
4
26
|
* Typed error constants for DistributedTable. Use with `isBlocksError()` in catch blocks.
|
|
5
27
|
*
|
|
@@ -133,3 +155,52 @@ export function remapItemTooLarge(err) {
|
|
|
133
155
|
}
|
|
134
156
|
return err;
|
|
135
157
|
}
|
|
158
|
+
/**
|
|
159
|
+
* @internal Reconcile a stored item with the schema on read, per the
|
|
160
|
+
* `readValidation` mode. Shared by the mock and AWS runtime so all three modes
|
|
161
|
+
* behave identically in both. `null` (a missing item) always passes through
|
|
162
|
+
* untouched, preserving not-found semantics.
|
|
163
|
+
*
|
|
164
|
+
* - `'off'` — return the raw item, no validation.
|
|
165
|
+
* - `'coerce'` — apply the schema (fill defaults / narrow types for
|
|
166
|
+
* transform-bearing schemas) **without dropping data**: the coerced output is
|
|
167
|
+
* deep-merged over the raw item, so schema output wins per key while attributes
|
|
168
|
+
* the schema doesn't declare (from an older schema version or another writer)
|
|
169
|
+
* are preserved. Without this merge, returning the bare validator output would
|
|
170
|
+
* strip unknown keys (Zod `.strip()` default), and a read-modify-write would
|
|
171
|
+
* then persist the stripped item — silently deleting stored data. On validation
|
|
172
|
+
* failure, return the **raw** item and `warn` — never throws — so drifted/legacy
|
|
173
|
+
* rows stay readable and the "reads return data or `null`" contract holds.
|
|
174
|
+
* - `'strict'` — throw `ValidationFailed` on any item that doesn't satisfy the
|
|
175
|
+
* schema.
|
|
176
|
+
*
|
|
177
|
+
* Schemas may validate synchronously or asynchronously; this awaits either.
|
|
178
|
+
*/
|
|
179
|
+
export async function applyReadValidation(mode, schema, item, log, context) {
|
|
180
|
+
if (item == null || mode === 'off')
|
|
181
|
+
return item;
|
|
182
|
+
const result = schema['~standard'].validate(item);
|
|
183
|
+
const resolved = result instanceof Promise ? await result : result;
|
|
184
|
+
if (resolved.issues) {
|
|
185
|
+
if (mode === 'strict') {
|
|
186
|
+
throw blocksError(DistributedTableErrors.ValidationFailed, resolved.issues[0]?.message ?? 'stored item failed schema validation on read');
|
|
187
|
+
}
|
|
188
|
+
log.warn(`readValidation: stored item failed schema validation, returning the raw value. ${resolved.issues[0]?.message ?? ''}`.trim(), context);
|
|
189
|
+
return item;
|
|
190
|
+
}
|
|
191
|
+
// Merge coerced output over the raw item: schema wins per key, but keys the
|
|
192
|
+
// schema stripped are preserved (see mergeCoercedOverRaw). Only merge when both
|
|
193
|
+
// sides are plain objects — a schema whose output is a primitive/array (rare
|
|
194
|
+
// for a table item) is returned as-is.
|
|
195
|
+
if (isPlainObject(item) && isPlainObject(resolved.value)) {
|
|
196
|
+
return mergeCoercedOverRaw(resolved.value, item);
|
|
197
|
+
}
|
|
198
|
+
return resolved.value;
|
|
199
|
+
}
|
|
200
|
+
/** True for a plain `{}` object (not null, array, Date, or class instance). */
|
|
201
|
+
function isPlainObject(v) {
|
|
202
|
+
if (v === null || typeof v !== 'object' || Array.isArray(v))
|
|
203
|
+
return false;
|
|
204
|
+
const proto = Object.getPrototypeOf(v);
|
|
205
|
+
return proto === Object.prototype || proto === null;
|
|
206
|
+
}
|
package/dist/index.aws.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { Scope } from '@aws-blocks/core';
|
|
2
2
|
import type { ScopeParent } from '@aws-blocks/core';
|
|
3
3
|
export { DistributedTableErrors } from './errors.js';
|
|
4
|
-
export type { TableKeyConfig, DistributedTableOptions, ExternalTableRef, TableKey, PartitionKeyCondition, SortKeyCondition, KeyCondition, QueryOptions, ScanOptions, PutOptions, DeleteOptions, } from './types.js';
|
|
4
|
+
export type { TableKeyConfig, DistributedTableOptions, ReadValidationMode, ExternalTableRef, TableKey, PartitionKeyCondition, SortKeyCondition, KeyCondition, QueryOptions, ScanOptions, PutOptions, DeleteOptions, } from './types.js';
|
|
5
5
|
import type { TableKeyConfig, DistributedTableOptions, ExternalTableRef, ScanOptions, PutOptions, DeleteOptions, TableKey } from './types.js';
|
|
6
6
|
import type { QueryOptions } from './types.js';
|
|
7
7
|
import type { ChildLogger } from '@aws-blocks/bb-logger';
|
|
@@ -11,8 +11,9 @@ export declare class DistributedTable<T, K extends TableKeyConfig<T> = TableKeyC
|
|
|
11
11
|
private schema;
|
|
12
12
|
private keyConfig;
|
|
13
13
|
private indexes;
|
|
14
|
+
private readValidation;
|
|
14
15
|
private docClient;
|
|
15
|
-
/** @internal Logger for internal operations. Defaults to
|
|
16
|
+
/** @internal Logger for internal operations. Defaults to warn-level when not provided. */
|
|
16
17
|
protected log: ChildLogger;
|
|
17
18
|
constructor(scope: ScopeParent, id: string, options: DistributedTableOptions<T, K, Indexes>);
|
|
18
19
|
get(key: TableKey<T, K>): Promise<T | null>;
|
|
@@ -34,6 +35,13 @@ export declare class DistributedTable<T, K extends TableKeyConfig<T> = TableKeyC
|
|
|
34
35
|
deleteBatch(keys: TableKey<T, K>[]): Promise<void>;
|
|
35
36
|
static fromExisting(tableName: string): ExternalTableRef;
|
|
36
37
|
private validateItem;
|
|
38
|
+
/**
|
|
39
|
+
* Reconcile a stored value with the schema per this table's `readValidation`
|
|
40
|
+
* mode (`off` → raw, `coerce` → coerced output / raw+warn on failure, `strict`
|
|
41
|
+
* → throw on mismatch). `null` (a missing item) passes straight through. See
|
|
42
|
+
* {@link applyReadValidation}.
|
|
43
|
+
*/
|
|
44
|
+
private reconcileRead;
|
|
37
45
|
private buildKey;
|
|
38
46
|
private backoff;
|
|
39
47
|
/**
|
package/dist/index.aws.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.aws.d.ts","sourceRoot":"","sources":["../src/index.aws.ts"],"names":[],"mappings":"AAcA,OAAO,EAAE,KAAK,EAA6C,MAAM,kBAAkB,CAAC;AACpF,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAIpD,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,YAAY,EACX,cAAc,EACd,uBAAuB,EACvB,gBAAgB,EAChB,QAAQ,EACR,qBAAqB,EACrB,gBAAgB,EAChB,YAAY,EACZ,YAAY,EACZ,WAAW,EACX,UAAU,EACV,aAAa,GACb,MAAM,YAAY,CAAC;AAEpB,OAAO,KAAK,EACX,cAAc,EACd,uBAAuB,EACvB,gBAAgB,EAChB,WAAW,EACX,UAAU,EACV,aAAa,EAGb,QAAQ,
|
|
1
|
+
{"version":3,"file":"index.aws.d.ts","sourceRoot":"","sources":["../src/index.aws.ts"],"names":[],"mappings":"AAcA,OAAO,EAAE,KAAK,EAA6C,MAAM,kBAAkB,CAAC;AACpF,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAIpD,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,YAAY,EACX,cAAc,EACd,uBAAuB,EACvB,kBAAkB,EAClB,gBAAgB,EAChB,QAAQ,EACR,qBAAqB,EACrB,gBAAgB,EAChB,YAAY,EACZ,YAAY,EACZ,WAAW,EACX,UAAU,EACV,aAAa,GACb,MAAM,YAAY,CAAC;AAEpB,OAAO,KAAK,EACX,cAAc,EACd,uBAAuB,EACvB,gBAAgB,EAChB,WAAW,EACX,UAAU,EACV,aAAa,EAGb,QAAQ,EAER,MAAM,YAAY,CAAC;AAEpB,OAAO,KAAK,EAAgB,YAAY,EAAE,MAAM,YAAY,CAAC;AAE7D,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,uBAAuB,CAAC;AAwBzD,qBAAa,gBAAgB,CAC5B,CAAC,EACD,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,GAAG,cAAc,CAAC,CAAC,CAAC,EAC/C,OAAO,SAAS,MAAM,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,CACpF,SAAQ,KAAK;IAWqC,OAAO,EAAE,uBAAuB,CAAC,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC;IAVlG,QAAQ,CAAC,MAAM,sBAAW;IAC1B,OAAO,CAAC,MAAM,CAAsB;IACpC,OAAO,CAAC,SAAS,CAAI;IACrB,OAAO,CAAC,OAAO,CAAU;IACzB,OAAO,CAAC,cAAc,CAAqB;IAC3C,OAAO,CAAC,SAAS,CAAyB;IAE1C,0FAA0F;IAC1F,SAAS,CAAC,GAAG,EAAE,WAAW,CAAC;gBAEf,KAAK,EAAE,WAAW,EAAE,EAAE,EAAE,MAAM,EAAS,OAAO,EAAE,uBAAuB,CAAC,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC;IAkB5F,GAAG,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC,GAAG,IAAI,CAAC;IAQ3C,GAAG,CAAC,IAAI,EAAE,CAAC,EAAE,OAAO,CAAC,EAAE,UAAU,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC;IAkBpD,MAAM,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,OAAO,CAAC,EAAE,aAAa,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC;IAa5E;;;;;;;;OAQG;IACI,KAAK,CACX,OAAO,EAAE,YAAY,CAAC,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC,GAClC,aAAa,CAAC,CAAC,CAAC;IAsCZ,IAAI,CAAC,OAAO,CAAC,EAAE,WAAW,GAAG,aAAa,CAAC,CAAC,CAAC;IAoB9C,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,GAAG,OAAO,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,EAAE,CAAC;IAwBvD,QAAQ,CAAC,KAAK,EAAE,CAAC,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IAsBnC,WAAW,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IAiBxD,MAAM,CAAC,YAAY,CAAC,SAAS,EAAE,MAAM,GAAG,gBAAgB;YAM1C,YAAY;IAQ1B;;;;;OAKG;IACH,OAAO,CAAC,aAAa;IAIrB,OAAO,CAAC,QAAQ;IAMhB,OAAO,CAAC,OAAO;IAWf;;;;;;;;;;;;;;;OAeG;YACW,gBAAgB;IAkB9B,OAAO,CAAC,yBAAyB;IAkBjC,OAAO,CAAC,iBAAiB;CA4CzB"}
|
package/dist/index.aws.js
CHANGED
|
@@ -5,7 +5,7 @@ import { DynamoDBDocumentClient, GetCommand, PutCommand, DeleteCommand, QueryCom
|
|
|
5
5
|
import { Scope, registerSdkIdentifiers, getSdkIdentifiers } from '@aws-blocks/core';
|
|
6
6
|
import { BB_NAME, BB_VERSION } from './version.js';
|
|
7
7
|
export { DistributedTableErrors } from './errors.js';
|
|
8
|
-
import { DistributedTableErrors, DistributedTableMessages, blocksError, normalizeSortKeyCondition, remapItemTooLarge } from './errors.js';
|
|
8
|
+
import { DistributedTableErrors, DistributedTableMessages, blocksError, normalizeSortKeyCondition, remapItemTooLarge, applyReadValidation } from './errors.js';
|
|
9
9
|
import { Logger } from '@aws-blocks/bb-logger';
|
|
10
10
|
// ── Helpers ─────────────────────────────────────────────────────────────────
|
|
11
11
|
// DynamoDB batch API limits and retry tuning.
|
|
@@ -32,17 +32,22 @@ export class DistributedTable extends Scope {
|
|
|
32
32
|
schema;
|
|
33
33
|
keyConfig;
|
|
34
34
|
indexes;
|
|
35
|
+
readValidation;
|
|
35
36
|
docClient;
|
|
36
|
-
/** @internal Logger for internal operations. Defaults to
|
|
37
|
+
/** @internal Logger for internal operations. Defaults to warn-level when not provided. */
|
|
37
38
|
log;
|
|
38
39
|
constructor(scope, id, options) {
|
|
39
40
|
super(id, { parent: scope, bbName: BB_NAME, bbVersion: BB_VERSION });
|
|
40
41
|
this.options = options;
|
|
41
|
-
|
|
42
|
+
// Default level is 'warn' (not 'error') so the readValidation='coerce'
|
|
43
|
+
// raw-fallback warning actually surfaces — it's the only log the block
|
|
44
|
+
// emits, so this doesn't add noise. Callers can pass their own logger.
|
|
45
|
+
this.log = options?.logger ?? new Logger(this, 'logger', { level: 'warn' });
|
|
42
46
|
const tableName = options.table?.tableName ?? this.fullId.substring(0, 255);
|
|
43
47
|
this.schema = options.schema;
|
|
44
48
|
this.keyConfig = options.key;
|
|
45
49
|
this.indexes = (options.indexes ?? {});
|
|
50
|
+
this.readValidation = options.readValidation ?? 'coerce';
|
|
46
51
|
const client = new DynamoDBClient({
|
|
47
52
|
customUserAgent: this.buildUserAgentChain(),
|
|
48
53
|
});
|
|
@@ -54,7 +59,7 @@ export class DistributedTable extends Scope {
|
|
|
54
59
|
TableName: getSdkIdentifiers(this).tableName,
|
|
55
60
|
Key: this.buildKey(key),
|
|
56
61
|
}));
|
|
57
|
-
return result.Item ?? null;
|
|
62
|
+
return this.reconcileRead(result.Item ?? null);
|
|
58
63
|
}
|
|
59
64
|
async put(item, options) {
|
|
60
65
|
await this.validateItem(item);
|
|
@@ -119,7 +124,7 @@ export class DistributedTable extends Scope {
|
|
|
119
124
|
const command = this.buildQueryCommand(options.index, pkField, pkValue, skField, skCondition, lastEvaluatedKey, options);
|
|
120
125
|
const result = await this.docClient.send(command);
|
|
121
126
|
for (const item of result.Items ?? []) {
|
|
122
|
-
yield item;
|
|
127
|
+
yield (await this.reconcileRead(item));
|
|
123
128
|
if (options.limit && ++count >= options.limit)
|
|
124
129
|
return;
|
|
125
130
|
}
|
|
@@ -136,7 +141,7 @@ export class DistributedTable extends Scope {
|
|
|
136
141
|
Limit: options?.limit,
|
|
137
142
|
}));
|
|
138
143
|
for (const item of result.Items ?? []) {
|
|
139
|
-
yield item;
|
|
144
|
+
yield (await this.reconcileRead(item));
|
|
140
145
|
if (options?.limit && ++count >= options.limit)
|
|
141
146
|
return;
|
|
142
147
|
}
|
|
@@ -157,7 +162,7 @@ export class DistributedTable extends Scope {
|
|
|
157
162
|
return resp.UnprocessedKeys?.[tableName]?.Keys;
|
|
158
163
|
});
|
|
159
164
|
}
|
|
160
|
-
return keys.map(key => results.get(JSON.stringify(this.buildKey(key))) ?? null);
|
|
165
|
+
return Promise.all(keys.map(key => this.reconcileRead(results.get(JSON.stringify(this.buildKey(key))) ?? null)));
|
|
161
166
|
}
|
|
162
167
|
async putBatch(items) {
|
|
163
168
|
for (const item of items)
|
|
@@ -199,6 +204,15 @@ export class DistributedTable extends Scope {
|
|
|
199
204
|
throw blocksError(DistributedTableErrors.ValidationFailed, resolved.issues[0].message);
|
|
200
205
|
}
|
|
201
206
|
}
|
|
207
|
+
/**
|
|
208
|
+
* Reconcile a stored value with the schema per this table's `readValidation`
|
|
209
|
+
* mode (`off` → raw, `coerce` → coerced output / raw+warn on failure, `strict`
|
|
210
|
+
* → throw on mismatch). `null` (a missing item) passes straight through. See
|
|
211
|
+
* {@link applyReadValidation}.
|
|
212
|
+
*/
|
|
213
|
+
reconcileRead(item) {
|
|
214
|
+
return applyReadValidation(this.readValidation, this.schema, item, this.log, { table: this.fullId });
|
|
215
|
+
}
|
|
202
216
|
buildKey(key) {
|
|
203
217
|
const result = { [this.keyConfig.partitionKey]: key[this.keyConfig.partitionKey] };
|
|
204
218
|
if (this.keyConfig.sortKey)
|
|
@@ -211,7 +225,7 @@ export class DistributedTable extends Scope {
|
|
|
211
225
|
// and lets concurrent callers re-collide; equal jitter preserves a minimum
|
|
212
226
|
// spacing while still de-synchronising retries under shared throttling.
|
|
213
227
|
// See: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.BatchOperations
|
|
214
|
-
const capped = Math.min(BASE_BACKOFF_MS *
|
|
228
|
+
const capped = Math.min(BASE_BACKOFF_MS * 2 ** attempt, MAX_BACKOFF_MS);
|
|
215
229
|
const ms = capped / 2 + Math.random() * (capped / 2);
|
|
216
230
|
return new Promise(resolve => setTimeout(resolve, ms));
|
|
217
231
|
}
|
package/dist/index.cdk.d.ts
CHANGED
|
@@ -2,7 +2,7 @@ import { Scope } from '@aws-blocks/core/cdk';
|
|
|
2
2
|
import type { ScopeParent } from '@aws-blocks/core';
|
|
3
3
|
import type { ExternalTableRef } from './types.js';
|
|
4
4
|
export { DistributedTableErrors } from './errors.js';
|
|
5
|
-
export type { DistributedTableOptions, TableKeyConfig, TableKey, PutOptions, DeleteOptions, QueryOptions, ScanOptions, ExternalTableRef } from './types.js';
|
|
5
|
+
export type { DistributedTableOptions, ReadValidationMode, TableKeyConfig, TableKey, PutOptions, DeleteOptions, QueryOptions, ScanOptions, ExternalTableRef } from './types.js';
|
|
6
6
|
export declare class DistributedTable<T = any> extends Scope {
|
|
7
7
|
options: any;
|
|
8
8
|
private table;
|
package/dist/index.cdk.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.cdk.d.ts","sourceRoot":"","sources":["../src/index.cdk.ts"],"names":[],"mappings":"AAUA,OAAO,EAAE,KAAK,EAAoC,MAAM,sBAAsB,CAAC;AAC/E,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AACpD,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAInD,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,YAAY,EAAE,uBAAuB,EAAE,cAAc,EAAE,QAAQ,EAAE,UAAU,EAAE,aAAa,EAAE,YAAY,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;
|
|
1
|
+
{"version":3,"file":"index.cdk.d.ts","sourceRoot":"","sources":["../src/index.cdk.ts"],"names":[],"mappings":"AAUA,OAAO,EAAE,KAAK,EAAoC,MAAM,sBAAsB,CAAC;AAC/E,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AACpD,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAInD,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,YAAY,EAAE,uBAAuB,EAAE,kBAAkB,EAAE,cAAc,EAAE,QAAQ,EAAE,UAAU,EAAE,aAAa,EAAE,YAAY,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAEhL,qBAAa,gBAAgB,CAAC,CAAC,GAAG,GAAG,CAAE,SAAQ,KAAK;IAcA,OAAO,EAAE,GAAG;IAb/D,OAAO,CAAC,KAAK,CAAS;IAEtB;;;;;;OAMG;IACH,MAAM,CAAC,YAAY,CAAC,SAAS,EAAE,MAAM,GAAG,gBAAgB;gBAI5C,KAAK,EAAE,WAAW,EAAE,EAAE,EAAE,MAAM,EAAS,OAAO,EAAE,GAAG;IAmG/D,GAAG,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;IAC/B,GAAG,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;IAC/B,MAAM,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;IAClC,KAAK,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;IACjC,IAAI,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;IAChC,QAAQ,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;IACpC,QAAQ,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;IACpC,WAAW,CAAC,GAAG,KAAK,EAAE,OAAO,EAAE,GAAG,KAAK;CACvC"}
|
package/dist/index.mock.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { Scope } from '@aws-blocks/core';
|
|
2
2
|
import type { ScopeParent } from '@aws-blocks/core';
|
|
3
3
|
export { DistributedTableErrors } from './errors.js';
|
|
4
|
-
export type { TableKeyConfig, DistributedTableOptions, ExternalTableRef, TableKey, PartitionKeyCondition, SortKeyCondition, KeyCondition, QueryOptions, ScanOptions, PutOptions, DeleteOptions, } from './types.js';
|
|
4
|
+
export type { TableKeyConfig, DistributedTableOptions, ReadValidationMode, ExternalTableRef, TableKey, PartitionKeyCondition, SortKeyCondition, KeyCondition, QueryOptions, ScanOptions, PutOptions, DeleteOptions, } from './types.js';
|
|
5
5
|
import type { TableKeyConfig, DistributedTableOptions, ExternalTableRef, ScanOptions, PutOptions, DeleteOptions, TableKey } from './types.js';
|
|
6
6
|
/**
|
|
7
7
|
* Structured data storage backed by DynamoDB with secondary indexes and
|
|
@@ -32,10 +32,18 @@ export declare class DistributedTable<T, K extends TableKeyConfig<T> = TableKeyC
|
|
|
32
32
|
private schema;
|
|
33
33
|
private keyConfig;
|
|
34
34
|
private indexes;
|
|
35
|
-
|
|
35
|
+
private readValidation;
|
|
36
|
+
/** @internal Logger for internal operations. Defaults to warn-level when not provided. */
|
|
36
37
|
protected log: ChildLogger;
|
|
37
38
|
constructor(scope: ScopeParent, id: string, options: DistributedTableOptions<T, K, Indexes>);
|
|
38
39
|
get(key: TableKey<T, K>): Promise<T | null>;
|
|
40
|
+
/**
|
|
41
|
+
* Reconcile a stored value with the schema per this table's `readValidation`
|
|
42
|
+
* mode (`off` → raw, `coerce` → coerced output / raw+warn on failure, `strict`
|
|
43
|
+
* → throw on mismatch). `null` (a missing item) passes straight through. See
|
|
44
|
+
* {@link applyReadValidation}.
|
|
45
|
+
*/
|
|
46
|
+
private reconcileRead;
|
|
39
47
|
put(item: T, options?: PutOptions<T>): Promise<void>;
|
|
40
48
|
delete(key: TableKey<T, K>, options?: DeleteOptions<T>): Promise<void>;
|
|
41
49
|
/**
|
package/dist/index.mock.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.mock.d.ts","sourceRoot":"","sources":["../src/index.mock.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,KAAK,EAA0B,MAAM,kBAAkB,CAAC;AAEjE,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAMpD,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,YAAY,EACX,cAAc,EACd,uBAAuB,EACvB,gBAAgB,EAChB,QAAQ,EACR,qBAAqB,EACrB,gBAAgB,EAChB,YAAY,EACZ,YAAY,EACZ,WAAW,EACX,UAAU,EACV,aAAa,GACb,MAAM,YAAY,CAAC;AAEpB,OAAO,KAAK,EACX,cAAc,EACd,uBAAuB,EACvB,gBAAgB,EAEhB,WAAW,EACX,UAAU,EACV,aAAa,EACb,QAAQ,
|
|
1
|
+
{"version":3,"file":"index.mock.d.ts","sourceRoot":"","sources":["../src/index.mock.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,KAAK,EAA0B,MAAM,kBAAkB,CAAC;AAEjE,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAMpD,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,YAAY,EACX,cAAc,EACd,uBAAuB,EACvB,kBAAkB,EAClB,gBAAgB,EAChB,QAAQ,EACR,qBAAqB,EACrB,gBAAgB,EAChB,YAAY,EACZ,YAAY,EACZ,WAAW,EACX,UAAU,EACV,aAAa,GACb,MAAM,YAAY,CAAC;AAEpB,OAAO,KAAK,EACX,cAAc,EACd,uBAAuB,EACvB,gBAAgB,EAEhB,WAAW,EACX,UAAU,EACV,aAAa,EACb,QAAQ,EAER,MAAM,YAAY,CAAC;AA4DpB;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,qBAAa,gBAAgB,CAC5B,CAAC,EACD,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,GAAG,cAAc,CAAC,CAAC,CAAC,EAC/C,OAAO,SAAS,MAAM,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,CACpF,SAAQ,KAAK;IAWqC,OAAO,EAAE,uBAAuB,CAAC,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC;IAVlG,OAAO,CAAC,QAAQ,CAAS;IACzB,OAAO,CAAC,IAAI,CAAiB;IAC7B,OAAO,CAAC,MAAM,CAAsB;IACpC,OAAO,CAAC,SAAS,CAAI;IACrB,OAAO,CAAC,OAAO,CAAU;IACzB,OAAO,CAAC,cAAc,CAAqB;IAE3C,0FAA0F;IAC1F,SAAS,CAAC,GAAG,EAAE,WAAW,CAAC;gBAEf,KAAK,EAAE,WAAW,EAAE,EAAE,EAAE,MAAM,EAAS,OAAO,EAAE,uBAAuB,CAAC,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC;IAe5F,GAAG,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC,GAAG,IAAI,CAAC;IAIjD;;;;;OAKG;IACH,OAAO,CAAC,aAAa;IAIf,GAAG,CAAC,IAAI,EAAE,CAAC,EAAE,OAAO,CAAC,EAAE,UAAU,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC;IAqBpD,MAAM,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,OAAO,CAAC,EAAE,aAAa,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC;IAc5E;;;;;;;;;;;;;;;;;;;;;;;;OAwBG;IACI,KAAK,CACX,OAAO,EAAE,YAAY,CAAC,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC,GAClC,aAAa,CAAC,CAAC,CAAC;IAoDZ,IAAI,CAAC,OAAO,CAAC,EAAE,WAAW,GAAG,aAAa,CAAC,CAAC,CAAC;IAQpD;;;;;;;OAOG;IACG,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,GAAG,OAAO,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,EAAE,CAAC;IAM7D;;;;;;OAMG;IACG,QAAQ,CAAC,KAAK,EAAE,CAAC,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IAczC;;;;;;OAMG;IACG,WAAW,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IAKxD,MAAM,CAAC,YAAY,CAAC,SAAS,EAAE,MAAM,GAAG,gBAAgB;IAMxD,OAAO,CAAC,gBAAgB;IAiBxB,OAAO,CAAC,YAAY;IAMpB,OAAO,CAAC,YAAY;IAOpB,OAAO,CAAC,WAAW;CAGnB;AAOD,OAAO,KAAK,EAAgE,YAAY,EAAE,MAAM,YAAY,CAAC;AAE7G,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,uBAAuB,CAAC"}
|