cdk-glue-iceberg-table 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +617 -0
- package/dist/lib/iceberg/iceberg-partition-transform.d.ts +78 -0
- package/dist/lib/iceberg/iceberg-partition-transform.js +219 -0
- package/dist/lib/iceberg/iceberg-table-properties.d.ts +54 -0
- package/dist/lib/iceberg/iceberg-table-properties.js +183 -0
- package/dist/lib/iceberg/iceberg-table.d.ts +187 -0
- package/dist/lib/iceberg/iceberg-table.js +549 -0
- package/dist/lib/iceberg/iceberg-type.d.ts +182 -0
- package/dist/lib/iceberg/iceberg-type.js +365 -0
- package/dist/lib/iceberg/index.d.ts +4 -0
- package/dist/lib/iceberg/index.js +18 -0
- package/package.json +70 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Rodrigo Carvajal
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,617 @@
|
|
|
1
|
+
# cdk-glue-iceberg-table
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/cdk-glue-iceberg-table)
|
|
4
|
+
[](LICENSE)
|
|
5
|
+
|
|
6
|
+
A CDK L2 construct for Apache Iceberg tables in the AWS Glue Data
|
|
7
|
+
Catalog. Emits the `AWS::Glue::Table` shape that survives
|
|
8
|
+
CloudFormation `Update`, so `cdk deploy` can create, evolve, and
|
|
9
|
+
destroy Iceberg tables the same way it handles any other resource.
|
|
10
|
+
|
|
11
|
+
The motivating issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
|
|
12
|
+
[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)
|
|
13
|
+
documents the only working CFN shape and the silent-corruption traps
|
|
14
|
+
you can hit by getting it slightly wrong. This construct implements
|
|
15
|
+
that shape and refuses to emit the unsafe alternatives.
|
|
16
|
+
|
|
17
|
+
The upstream CDK PR landing this construct in
|
|
18
|
+
`@aws-cdk/aws-glue-alpha` is [aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988).
|
|
19
|
+
Until that merges, this package is the most current reference
|
|
20
|
+
implementation. Once `@aws-cdk/aws-glue-alpha` ships its own
|
|
21
|
+
`IcebergTable`, prefer the official one and treat this package as a
|
|
22
|
+
stopgap.
|
|
23
|
+
|
|
24
|
+
> **Status: pre-release.** `0.1.0` has not been published to npm yet
|
|
25
|
+
> — the `npm install` command below will 404 until the first
|
|
26
|
+
> `v*.*.*` tag is pushed (which triggers the publish workflow). To
|
|
27
|
+
> use the construct today, install directly from this repo:
|
|
28
|
+
> `npm install github:ksco92/arceus#main`.
|
|
29
|
+
|
|
30
|
+
## Install
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
npm install cdk-glue-iceberg-table
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Peer dependencies (your CDK app must already have these):
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
npm install aws-cdk-lib constructs @aws-cdk/aws-glue-alpha
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Use
|
|
43
|
+
|
|
44
|
+
```ts
|
|
45
|
+
import { Bucket } from 'aws-cdk-lib/aws-s3';
|
|
46
|
+
import { Database } from '@aws-cdk/aws-glue-alpha';
|
|
47
|
+
import {
|
|
48
|
+
IcebergTable,
|
|
49
|
+
IcebergType,
|
|
50
|
+
IcebergPartitionTransform,
|
|
51
|
+
} from 'cdk-glue-iceberg-table';
|
|
52
|
+
|
|
53
|
+
const bucket = new Bucket(this, 'Warehouse');
|
|
54
|
+
const db = new Database(this, 'Db', { databaseName: 'analytics' });
|
|
55
|
+
|
|
56
|
+
new IcebergTable(this, 'OrdersTable', {
|
|
57
|
+
database: db,
|
|
58
|
+
tableName: 'orders',
|
|
59
|
+
location: `s3://${bucket.bucketName}/analytics/orders/`,
|
|
60
|
+
columns: [
|
|
61
|
+
{ name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },
|
|
62
|
+
{ name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },
|
|
63
|
+
{ name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },
|
|
64
|
+
],
|
|
65
|
+
partitionSpec: [
|
|
66
|
+
{ sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY },
|
|
67
|
+
{ sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16) },
|
|
68
|
+
],
|
|
69
|
+
identifierFieldNames: ['order_id'],
|
|
70
|
+
});
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Consumer-facing reference sections below:
|
|
74
|
+
|
|
75
|
+
- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.
|
|
76
|
+
- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.
|
|
77
|
+
- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.
|
|
78
|
+
|
|
79
|
+
## Repo layout
|
|
80
|
+
|
|
81
|
+
This repo is **both** the published package and a CDK demo app:
|
|
82
|
+
|
|
83
|
+
- `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
|
|
84
|
+
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus a bash harness that drives schema + partition evolution through real `cdk deploy`s. **Repo-only, not published to npm.**
|
|
85
|
+
|
|
86
|
+
The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
|
|
87
|
+
[Demo tables](#demo-tables-deployed-by-arceusstack), and
|
|
88
|
+
[Schema + partition evolution](#schema--partition-evolution-via-cdk-only)
|
|
89
|
+
cover the demo app. Skip them if you only want to consume the construct.
|
|
90
|
+
|
|
91
|
+
## Prerequisites
|
|
92
|
+
|
|
93
|
+
Before running the quickstart you need:
|
|
94
|
+
|
|
95
|
+
1. AWS credentials in the default profile with permissions to manage
|
|
96
|
+
CloudFormation, KMS, S3, Glue, Lake Formation, Athena, and IAM
|
|
97
|
+
policies. `aws sts get-caller-identity` must return successfully.
|
|
98
|
+
2. `CDK_DEFAULT_ACCOUNT` and `CDK_DEFAULT_REGION` set in the
|
|
99
|
+
environment (the AWS CLI sets these automatically for most
|
|
100
|
+
profile setups; `cdk` also populates them from the active profile).
|
|
101
|
+
3. `DEVELOPER_IAM_USER` set to the name of an **existing IAM user**
|
|
102
|
+
in this account. The stack adds that user as a Lake Formation
|
|
103
|
+
admin and grants it per-table `SELECT/INSERT/DELETE/ALTER/DESCRIBE`
|
|
104
|
+
on the demo Iceberg tables — without it the deploy fails when LF
|
|
105
|
+
can't resolve the principal.
|
|
106
|
+
4. The Lake Formation service-linked role
|
|
107
|
+
`AWSServiceRoleForLakeFormationDataAccess` must exist in the
|
|
108
|
+
account. Create it once with
|
|
109
|
+
`aws iam create-service-linked-role --aws-service-name lakeformation.amazonaws.com`
|
|
110
|
+
if you haven't already.
|
|
111
|
+
5. `cdk bootstrap aws://<account>/<region>` if the account hasn't
|
|
112
|
+
been bootstrapped for CDK.
|
|
113
|
+
|
|
114
|
+
## What's in the repo
|
|
115
|
+
|
|
116
|
+
Repo source paths (npm consumers import from the package root and get the same exports under `dist/lib/iceberg/`):
|
|
117
|
+
|
|
118
|
+
- **`lib/iceberg/iceberg-table.ts`** — the `IcebergTable` L2 construct.
|
|
119
|
+
- **`lib/iceberg/iceberg-type.ts`** — `IcebergType` with primitives + `list` / `map` / `struct` factories. Renders to the JSON shape Glue's `IcebergStructField.type` expects.
|
|
120
|
+
- **`lib/iceberg/iceberg-partition-transform.ts`** — `IcebergPartitionTransform` (identity / bucket(N) / truncate(W) / year / month / day / hour / void). Each transform validates against the source column type at synth time.
|
|
121
|
+
- **`lib/iceberg/iceberg-table-properties.ts`** — `IcebergDataFormat` (parquet/orc/avro — default parquet), `IcebergFormatVersion` (v1/v2 — default v2), and a validator that catches misconfigured `tableProperties` before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
|
|
122
|
+
- **`lib/arceus-stack.ts`** — the demo stack: KMS-encrypted data lake bucket, Athena results bucket, Glue database, three demo Iceberg tables (`orders`, `events`, `customers`). Repo-only, not in the npm tarball.
|
|
123
|
+
- **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh`** — a parameterized stack and a bash harness that drives four real `cdk deploy`s to prove schema/partition evolution works end-to-end. Repo-only, not in the npm tarball.
|
|
124
|
+
|
|
125
|
+
## Quickstart
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
# DEVELOPER_IAM_USER must be the name of an existing IAM user in
|
|
129
|
+
# this account. `aws iam get-user` works when you're authenticated
|
|
130
|
+
# directly as an IAM user; for SSO / assumed-role / aws-vault setups
|
|
131
|
+
# (the modern default) the call fails with "Must specify userName
|
|
132
|
+
# when calling with non-User credentials" and you should set the
|
|
133
|
+
# variable manually to whichever IAM user you want LF to grant on.
|
|
134
|
+
export DEVELOPER_IAM_USER="$(aws iam get-user --query 'User.UserName' --output text)"
|
|
135
|
+
|
|
136
|
+
npm install
|
|
137
|
+
npx jest # 100% line + branch coverage, 95% gate
|
|
138
|
+
npx cdk deploy ArceusStack --require-approval=never
|
|
139
|
+
./scripts/integration-test-evolution.sh # add + rename + drop, via cdk only
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
`cdk ls` will show two stacks — `ArceusStack` (the demo data lake +
|
|
143
|
+
three Iceberg tables) and `IcebergEvolutionStack` (the evolution
|
|
144
|
+
test target driven by `scripts/integration-test-evolution.sh`).
|
|
145
|
+
Deploy only `ArceusStack` for the quickstart; the evolution stack
|
|
146
|
+
is created on demand by the script.
|
|
147
|
+
|
|
148
|
+
## Using `IcebergTable`
|
|
149
|
+
|
|
150
|
+
A minimal table:
|
|
151
|
+
|
|
152
|
+
```typescript
|
|
153
|
+
import {
|
|
154
|
+
Database,
|
|
155
|
+
} from '@aws-cdk/aws-glue-alpha';
|
|
156
|
+
import {
|
|
157
|
+
IcebergTable,
|
|
158
|
+
IcebergType,
|
|
159
|
+
} from './iceberg';
|
|
160
|
+
|
|
161
|
+
const db = new Database(this, 'Db', {
|
|
162
|
+
databaseName: 'analytics',
|
|
163
|
+
});
|
|
164
|
+
|
|
165
|
+
new IcebergTable(this, 'Users', {
|
|
166
|
+
database: db,
|
|
167
|
+
tableName: 'users',
|
|
168
|
+
columns: [
|
|
169
|
+
{
|
|
170
|
+
name: 'user_id',
|
|
171
|
+
type: IcebergType.LONG,
|
|
172
|
+
required: true,
|
|
173
|
+
id: 1,
|
|
174
|
+
},
|
|
175
|
+
{
|
|
176
|
+
name: 'email',
|
|
177
|
+
type: IcebergType.STRING,
|
|
178
|
+
required: true,
|
|
179
|
+
id: 2,
|
|
180
|
+
},
|
|
181
|
+
{
|
|
182
|
+
name: 'signed_up_at',
|
|
183
|
+
type: IcebergType.TIMESTAMPTZ,
|
|
184
|
+
required: true,
|
|
185
|
+
id: 3,
|
|
186
|
+
},
|
|
187
|
+
],
|
|
188
|
+
location: `s3://${bucket.bucketName}/analytics/users/`,
|
|
189
|
+
});
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
A table that exercises most of the surface — partitions, sort order,
|
|
193
|
+
nested types, identifier fields, table properties, removal policy.
|
|
194
|
+
This is the exact shape `ArceusStack` uses for the `orders` demo
|
|
195
|
+
table, so the column list / partition spec / properties round-trip
|
|
196
|
+
straight to the live metadata.json below.
|
|
197
|
+
|
|
198
|
+
```typescript
|
|
199
|
+
import {
|
|
200
|
+
RemovalPolicy,
|
|
201
|
+
} from 'aws-cdk-lib';
|
|
202
|
+
import {
|
|
203
|
+
Database,
|
|
204
|
+
} from '@aws-cdk/aws-glue-alpha';
|
|
205
|
+
import {
|
|
206
|
+
IcebergDataFormat,
|
|
207
|
+
IcebergFormatVersion,
|
|
208
|
+
IcebergNullOrder,
|
|
209
|
+
IcebergPartitionTransform,
|
|
210
|
+
IcebergSortDirection,
|
|
211
|
+
IcebergTable,
|
|
212
|
+
IcebergType,
|
|
213
|
+
} from './iceberg';
|
|
214
|
+
|
|
215
|
+
new IcebergTable(this, 'OrdersTable', {
|
|
216
|
+
database: db,
|
|
217
|
+
tableName: 'orders',
|
|
218
|
+
comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',
|
|
219
|
+
columns: [
|
|
220
|
+
{
|
|
221
|
+
name: 'order_id',
|
|
222
|
+
type: IcebergType.LONG,
|
|
223
|
+
required: true,
|
|
224
|
+
id: 1,
|
|
225
|
+
},
|
|
226
|
+
{
|
|
227
|
+
name: 'customer_id',
|
|
228
|
+
type: IcebergType.LONG,
|
|
229
|
+
required: true,
|
|
230
|
+
id: 2,
|
|
231
|
+
},
|
|
232
|
+
{
|
|
233
|
+
name: 'order_amount',
|
|
234
|
+
type: IcebergType.decimal(12, 2),
|
|
235
|
+
required: true,
|
|
236
|
+
id: 3,
|
|
237
|
+
},
|
|
238
|
+
{
|
|
239
|
+
name: 'currency',
|
|
240
|
+
type: IcebergType.STRING,
|
|
241
|
+
required: true,
|
|
242
|
+
id: 4,
|
|
243
|
+
},
|
|
244
|
+
{
|
|
245
|
+
name: 'placed_at',
|
|
246
|
+
type: IcebergType.TIMESTAMPTZ,
|
|
247
|
+
required: true,
|
|
248
|
+
id: 5,
|
|
249
|
+
},
|
|
250
|
+
{
|
|
251
|
+
name: 'tags',
|
|
252
|
+
type: IcebergType.list(IcebergType.STRING),
|
|
253
|
+
id: 6,
|
|
254
|
+
},
|
|
255
|
+
{
|
|
256
|
+
name: 'shipping_address',
|
|
257
|
+
type: IcebergType.struct([
|
|
258
|
+
{
|
|
259
|
+
name: 'line1',
|
|
260
|
+
type: IcebergType.STRING,
|
|
261
|
+
required: true,
|
|
262
|
+
},
|
|
263
|
+
{
|
|
264
|
+
name: 'city',
|
|
265
|
+
type: IcebergType.STRING,
|
|
266
|
+
required: true,
|
|
267
|
+
},
|
|
268
|
+
{
|
|
269
|
+
name: 'country',
|
|
270
|
+
type: IcebergType.STRING,
|
|
271
|
+
required: true,
|
|
272
|
+
},
|
|
273
|
+
{
|
|
274
|
+
name: 'postal_code',
|
|
275
|
+
type: IcebergType.STRING,
|
|
276
|
+
},
|
|
277
|
+
]),
|
|
278
|
+
id: 7,
|
|
279
|
+
},
|
|
280
|
+
{
|
|
281
|
+
name: 'metadata',
|
|
282
|
+
type: IcebergType.map(IcebergType.STRING, IcebergType.STRING, false),
|
|
283
|
+
id: 8,
|
|
284
|
+
},
|
|
285
|
+
],
|
|
286
|
+
location: `s3://${bucket.bucketName}/analytics/orders/`,
|
|
287
|
+
partitionSpec: [
|
|
288
|
+
{
|
|
289
|
+
sourceColumn: 'placed_at',
|
|
290
|
+
transform: IcebergPartitionTransform.DAY,
|
|
291
|
+
},
|
|
292
|
+
{
|
|
293
|
+
sourceColumn: 'customer_id',
|
|
294
|
+
transform: IcebergPartitionTransform.bucket(16),
|
|
295
|
+
},
|
|
296
|
+
],
|
|
297
|
+
sortOrder: [
|
|
298
|
+
{
|
|
299
|
+
sourceColumn: 'placed_at',
|
|
300
|
+
direction: IcebergSortDirection.ASC,
|
|
301
|
+
nullOrder: IcebergNullOrder.NULLS_LAST,
|
|
302
|
+
},
|
|
303
|
+
{
|
|
304
|
+
sourceColumn: 'order_id',
|
|
305
|
+
direction: IcebergSortDirection.ASC,
|
|
306
|
+
},
|
|
307
|
+
],
|
|
308
|
+
identifierFieldNames: [
|
|
309
|
+
'order_id',
|
|
310
|
+
],
|
|
311
|
+
dataFormat: IcebergDataFormat.PARQUET,
|
|
312
|
+
formatVersion: IcebergFormatVersion.V2,
|
|
313
|
+
tableProperties: {
|
|
314
|
+
'write.parquet.compression-codec': 'zstd',
|
|
315
|
+
'write.delete.mode': 'merge-on-read',
|
|
316
|
+
'write.update.mode': 'merge-on-read',
|
|
317
|
+
'write.merge.mode': 'merge-on-read',
|
|
318
|
+
'write.target-file-size-bytes': '134217728',
|
|
319
|
+
'history.expire.min-snapshots-to-keep': '5',
|
|
320
|
+
'gc.enabled': 'true',
|
|
321
|
+
},
|
|
322
|
+
removalPolicy: RemovalPolicy.DESTROY,
|
|
323
|
+
});
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
### Granting access
|
|
327
|
+
|
|
328
|
+
```typescript
|
|
329
|
+
table.grantRead(role); // Glue read + S3 read on the table's prefix
|
|
330
|
+
table.grantWrite(role); // Glue write + S3 write
|
|
331
|
+
table.grantReadWrite(role);
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
### Importing an existing table
|
|
335
|
+
|
|
336
|
+
```typescript
|
|
337
|
+
const existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {
|
|
338
|
+
database: db,
|
|
339
|
+
tableName: 'orders',
|
|
340
|
+
location: 's3://my-bucket/analytics/orders/',
|
|
341
|
+
});
|
|
342
|
+
existing.grantRead(role);
|
|
343
|
+
```
|
|
344
|
+
|
|
345
|
+
## Demo tables (deployed by `ArceusStack`)
|
|
346
|
+
|
|
347
|
+
| Table | Format | Columns | Partitions | Sort | Notable properties |
|
|
348
|
+
| --- | --- | --- | --- | --- | --- |
|
|
349
|
+
| `orders` | parquet, v2 | `order_id(1)`, `customer_id(2)`, `order_amount(3)`, `currency(4)`, `placed_at(5)`, `tags(6)` (list), `shipping_address(7)` (struct), `metadata(8)` (map) | `day(placed_at)`, `bucket(16)(customer_id)` | `placed_at ASC NULLS LAST`, `order_id ASC` | `write.{delete,update,merge}.mode = merge-on-read`, `zstd`, `history.expire.min-snapshots-to-keep = 5`, identifier-field-ids = `[order_id]`, nested `list`/`struct`/`map` columns |
|
|
350
|
+
| `events` | parquet, v2 | `event_id(1)`, `event_name(2)`, `session_id(3)`, `occurred_at(4)`, `attributes(5)` (map) | `hour(occurred_at)` | (none) | high-cardinality hourly partitioning |
|
|
351
|
+
| `customers` | parquet, v2 | `customer_id(1)`, `email(2)`, `signed_up_at(4)`, `loyalty_tier(5)` (id 3 retired) | (none) | (none) | identifier-field-ids = `[customer_id]` — the stack's `customers` block carries inline comments narrating the schema-evolution journey that landed here (drop `full_name`, add `loyalty_tier`); the live evolution loop runs against the separate `IcebergEvolutionStack` |
|
|
352
|
+
|
|
353
|
+
After `cdk deploy ArceusStack`, the three tables are queryable from Athena (workgroup `ReadOnly`).
|
|
354
|
+
|
|
355
|
+
### Validating the demo
|
|
356
|
+
|
|
357
|
+
`SHOW TBLPROPERTIES sample_database.orders` returns:
|
|
358
|
+
|
|
359
|
+
```
|
|
360
|
+
format parquet
|
|
361
|
+
write_compression zstd
|
|
362
|
+
write_target_data_file_size_bytes 134217728
|
|
363
|
+
vacuum_min_snapshots_to_keep 5
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
The Iceberg `metadata.json` for `orders` contains every feature you set:
|
|
367
|
+
|
|
368
|
+
```json
|
|
369
|
+
{
|
|
370
|
+
"format-version": 2,
|
|
371
|
+
"table-uuid": "39a948f9-...",
|
|
372
|
+
"current-schema-id": 0,
|
|
373
|
+
"schemas": [
|
|
374
|
+
{
|
|
375
|
+
"schema-id": 0,
|
|
376
|
+
"identifier-field-ids": [1],
|
|
377
|
+
"fields": [
|
|
378
|
+
{ "id": 1, "name": "order_id", "required": true, "type": "long" },
|
|
379
|
+
{ "id": 2, "name": "customer_id", "required": true, "type": "long" },
|
|
380
|
+
{ "id": 3, "name": "order_amount", "required": true, "type": "decimal(12, 2)" },
|
|
381
|
+
{ "id": 4, "name": "currency", "required": true, "type": "string" },
|
|
382
|
+
{ "id": 5, "name": "placed_at", "required": true, "type": "timestamptz" },
|
|
383
|
+
{ "id": 6, "name": "tags", "required": false,
|
|
384
|
+
"type": { "type": "list", "element-id": 9, "element": "string", "element-required": true } },
|
|
385
|
+
{ "id": 7, "name": "shipping_address", "required": false,
|
|
386
|
+
"type": { "type": "struct", "fields": [
|
|
387
|
+
{ "id": 10, "name": "line1", "required": true, "type": "string" },
|
|
388
|
+
{ "id": 11, "name": "city", "required": true, "type": "string" },
|
|
389
|
+
{ "id": 12, "name": "country", "required": true, "type": "string" },
|
|
390
|
+
{ "id": 13, "name": "postal_code", "required": false, "type": "string" }
|
|
391
|
+
] } },
|
|
392
|
+
{ "id": 8, "name": "metadata", "required": false,
|
|
393
|
+
"type": { "type": "map", "key-id": 14, "key": "string", "value-id": 15,
|
|
394
|
+
"value-required": false, "value": "string" } }
|
|
395
|
+
]
|
|
396
|
+
}
|
|
397
|
+
],
|
|
398
|
+
"partition-specs": [
|
|
399
|
+
{ "spec-id": 0, "fields": [
|
|
400
|
+
{ "name": "placed_at_day", "transform": "day", "source-id": 5, "field-id": 1000 },
|
|
401
|
+
{ "name": "customer_id_bucket", "transform": "bucket[16]", "source-id": 2, "field-id": 1001 }
|
|
402
|
+
]}
|
|
403
|
+
],
|
|
404
|
+
"sort-orders": [
|
|
405
|
+
{ "order-id": 1, "fields": [
|
|
406
|
+
{ "transform": "identity", "source-id": 5, "direction": "asc", "null-order": "nulls-last" },
|
|
407
|
+
{ "transform": "identity", "source-id": 1, "direction": "asc", "null-order": "nulls-last" }
|
|
408
|
+
]}
|
|
409
|
+
],
|
|
410
|
+
"properties": {
|
|
411
|
+
"format-version": "2",
|
|
412
|
+
"write.format.default": "parquet",
|
|
413
|
+
"write.parquet.compression-codec": "zstd",
|
|
414
|
+
"write.merge.mode": "merge-on-read",
|
|
415
|
+
"write.update.mode": "merge-on-read",
|
|
416
|
+
"write.delete.mode": "merge-on-read",
|
|
417
|
+
"write.target-file-size-bytes": "134217728",
|
|
418
|
+
"history.expire.min-snapshots-to-keep": "5",
|
|
419
|
+
"gc.enabled": "true",
|
|
420
|
+
"comment": "Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read."
|
|
421
|
+
}
|
|
422
|
+
}
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
### Inserting and querying
|
|
426
|
+
|
|
427
|
+
```sql
|
|
428
|
+
-- INSERT into the orders table
|
|
429
|
+
INSERT INTO sample_database.orders VALUES
|
|
430
|
+
(1001, 5001, DECIMAL '149.99', 'USD',
|
|
431
|
+
TIMESTAMP '2026-05-20 09:15:00 UTC',
|
|
432
|
+
ARRAY['holiday-promo','first-order'],
|
|
433
|
+
CAST(ROW('1 Infinite Loop','Cupertino','US','95014')
|
|
434
|
+
AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),
|
|
435
|
+
MAP(ARRAY['channel','utm'], ARRAY['web','google'])),
|
|
436
|
+
-- ... more rows
|
|
437
|
+
;
|
|
438
|
+
|
|
439
|
+
-- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)
|
|
440
|
+
DELETE FROM sample_database.orders WHERE order_id = 1003;
|
|
441
|
+
|
|
442
|
+
-- merge-on-read UPDATE
|
|
443
|
+
UPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;
|
|
444
|
+
|
|
445
|
+
-- SELECT
|
|
446
|
+
SELECT customer_id, SUM(order_amount) AS total
|
|
447
|
+
FROM sample_database.orders
|
|
448
|
+
GROUP BY 1
|
|
449
|
+
ORDER BY 2 DESC;
|
|
450
|
+
```
|
|
451
|
+
|
|
452
|
+
## Schema + partition evolution via cdk-only
|
|
453
|
+
|
|
454
|
+
`scripts/integration-test-evolution.sh` drives the
|
|
455
|
+
`IcebergEvolutionStack` through four `cdk deploy`s and verifies the
|
|
456
|
+
underlying Iceberg `metadata.json` after each:
|
|
457
|
+
|
|
458
|
+
| Step | Change | Columns | Partitions |
|
|
459
|
+
| ---: | --- | --- | --- |
|
|
460
|
+
| 1 | Initial deploy | `customer_id(1)`, `email(2)`, `signed_up_at(3)` | `day(signed_up_at)` |
|
|
461
|
+
| 2 | **ADD** column `region(4)` | + `region(4)` | unchanged |
|
|
462
|
+
| 3 | **RENAME** `email` → `contact_email` (id 2 preserved), **ADD** partition `bucket(8)(customer_id)` | rename | + `bucket(8)(customer_id)` |
|
|
463
|
+
| 4 | **DROP** column `region` (id 4 stays retired), **DROP** partition `bucket(8)(customer_id)` | − `region` | − `customer_id_bucket` |
|
|
464
|
+
|
|
465
|
+
Last script run output (abridged — `cdk deploy` chatter and the
|
|
466
|
+
per-Athena-query state polling lines are omitted; the assertion
|
|
467
|
+
output is verbatim):
|
|
468
|
+
|
|
469
|
+
```
|
|
470
|
+
=== STEP 1 — cdk deploy ===
|
|
471
|
+
✨ Total time: 10.57s
|
|
472
|
+
|
|
473
|
+
=== VERIFY step 1 ===
|
|
474
|
+
columns ✓ (1:customer_id,2:email,3:signed_up_at)
|
|
475
|
+
partitions ✓ (signed_up_at_day)
|
|
476
|
+
|
|
477
|
+
=== INSERT seed rows ===
|
|
478
|
+
3 rows inserted
|
|
479
|
+
|
|
480
|
+
=== STEP 2 — cdk deploy ===
|
|
481
|
+
=== VERIFY step 2 (ADD column) ===
|
|
482
|
+
columns ✓ (1:customer_id,2:email,3:signed_up_at,4:region)
|
|
483
|
+
partitions ✓ (signed_up_at_day)
|
|
484
|
+
|
|
485
|
+
=== VERIFY old rows are preserved with region=NULL ===
|
|
486
|
+
pre-existing rows readable ✓
|
|
487
|
+
inserted 1 row carrying region='us-east-1'
|
|
488
|
+
|
|
489
|
+
=== STEP 3 — cdk deploy ===
|
|
490
|
+
=== VERIFY step 3 (RENAME column + ADD partition) ===
|
|
491
|
+
columns ✓ (1:customer_id,2:contact_email,3:signed_up_at,4:region)
|
|
492
|
+
partitions ✓ (signed_up_at_day,customer_id_bucket)
|
|
493
|
+
rename preserved data ✓
|
|
494
|
+
|
|
495
|
+
=== STEP 4 — cdk deploy ===
|
|
496
|
+
=== VERIFY step 4 (DROP column + DROP partition) ===
|
|
497
|
+
columns ✓ (1:customer_id,2:contact_email,3:signed_up_at)
|
|
498
|
+
partitions ✓ (signed_up_at_day)
|
|
499
|
+
last-column-id stays at 4 — id reuse protection ✓
|
|
500
|
+
all 4 pre-existing rows queryable after drop ✓
|
|
501
|
+
|
|
502
|
+
=== TEARDOWN ===
|
|
503
|
+
IcebergEvolutionStack | 4 | DELETE_COMPLETE | AWS::CloudFormation::Stack
|
|
504
|
+
✅ IcebergEvolutionStack: destroyed
|
|
505
|
+
|
|
506
|
+
=== ALL EVOLUTION STEPS PASSED ===
|
|
507
|
+
```
|
|
508
|
+
|
|
509
|
+
The construct passes each new column list + partition spec to Glue
|
|
510
|
+
`UpdateTable` via `OpenTableFormatInput`. Glue computes the Iceberg
|
|
511
|
+
metadata delta (new `schema-id`, new `spec-id`) and writes a new
|
|
512
|
+
`metadata.json`. Old data files stay readable because the field IDs
|
|
513
|
+
the construct pins (`id: N` on each `IcebergColumn`) never change
|
|
514
|
+
across deploys.
|
|
515
|
+
|
|
516
|
+
## Two footguns the construct prevents
|
|
517
|
+
|
|
518
|
+
### Footgun #1 — schema under `storageDescriptor.columns`
|
|
519
|
+
|
|
520
|
+
The CREATE succeeds but the first UPDATE silently strips
|
|
521
|
+
`table_type=ICEBERG` from the table's Glue parameters, and Athena
|
|
522
|
+
queries after that fail with `HIVE_UNSUPPORTED_FORMAT`.
|
|
523
|
+
|
|
524
|
+
```typescript
|
|
525
|
+
// DON'T DO THIS — what most StackOverflow / re:Post examples show
|
|
526
|
+
new CfnTable(this, 'OrdersBad', {
|
|
527
|
+
catalogId: this.account,
|
|
528
|
+
databaseName: 'analytics',
|
|
529
|
+
tableInput: {
|
|
530
|
+
name: 'orders',
|
|
531
|
+
tableType: 'EXTERNAL_TABLE',
|
|
532
|
+
parameters: {
|
|
533
|
+
table_type: 'ICEBERG',
|
|
534
|
+
},
|
|
535
|
+
storageDescriptor: {
|
|
536
|
+
location: 's3://.../orders/',
|
|
537
|
+
columns: [
|
|
538
|
+
/* ... */
|
|
539
|
+
],
|
|
540
|
+
},
|
|
541
|
+
},
|
|
542
|
+
openTableFormatInput: {
|
|
543
|
+
icebergInput: {
|
|
544
|
+
metadataOperation: 'CREATE',
|
|
545
|
+
version: '2',
|
|
546
|
+
},
|
|
547
|
+
},
|
|
548
|
+
});
|
|
549
|
+
```
|
|
550
|
+
|
|
551
|
+
`IcebergTable` instead always emits schema/partitions/sort/properties
|
|
552
|
+
under `openTableFormatInput.icebergInput.icebergTableInput`, never
|
|
553
|
+
under `storageDescriptor`.
|
|
554
|
+
|
|
555
|
+
### Footgun #2 — `tableInput` co-present with `openTableFormatInput`
|
|
556
|
+
|
|
557
|
+
Even setting just `tableInput: { name: 'foo' }` next to
|
|
558
|
+
`openTableFormatInput` returns
|
|
559
|
+
`"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput"`.
|
|
560
|
+
The construct never emits `tableInput` — table-level comment goes
|
|
561
|
+
into `tableProperties['comment']`, which lives inside
|
|
562
|
+
`icebergTableInput.properties`.
|
|
563
|
+
|
|
564
|
+
(There is a third footgun — field-id reuse after a column drop — that
|
|
565
|
+
the construct does **not** prevent. See the next section.)
|
|
566
|
+
|
|
567
|
+
## Known limitations
|
|
568
|
+
|
|
569
|
+
- **Field-id reuse is not detected across deploys.** If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the "never reuse a retired id" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate column id N` validator), but it doesn't compare against the live table state. The safe workflow is to always pin `id` explicitly and treat dropped ids as retired forever; never let CDK reassign an id that has ever been used.
|
|
570
|
+
- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.
|
|
571
|
+
- **Format version is immutable after CREATE.** The `formatVersion` prop is read once at table creation; changing it later requires a destroy + recreate.
|
|
572
|
+
- **`merge-on-read` requires v2.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.
|
|
573
|
+
- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.
|
|
574
|
+
- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The integration-test script demonstrates the safe pattern: drop partitions that source from `customer_id` while keeping `customer_id` itself in the schema, and drop the `region` column while it is not partitioning anything.
|
|
575
|
+
|
|
576
|
+
## Tests
|
|
577
|
+
|
|
578
|
+
```
|
|
579
|
+
npx jest # runs every suite under test/, prints coverage at the end
|
|
580
|
+
```
|
|
581
|
+
|
|
582
|
+
Coverage is gated at 95% statements / 95% branches / 95% lines / 95% functions
|
|
583
|
+
on `lib/**/*.ts` via the `coverageThreshold.global` block in `jest.config.js`.
|
|
584
|
+
A failing gate fails the suite, so the README does not paste a transcript that
|
|
585
|
+
would drift after the next refactor — run the command locally for the live
|
|
586
|
+
numbers.
|
|
587
|
+
|
|
588
|
+
## Project layout
|
|
589
|
+
|
|
590
|
+
```
|
|
591
|
+
arceus/
|
|
592
|
+
├── bin/arceus.ts # CDK app entry point
|
|
593
|
+
├── lib/
|
|
594
|
+
│ ├── arceus-stack.ts # Demo stack (buckets, DB, 3 demo tables)
|
|
595
|
+
│ ├── iceberg-evolution-stack.ts # Parameterized stack for the evolution test
|
|
596
|
+
│ └── iceberg/
|
|
597
|
+
│ ├── iceberg-table.ts # The L2 construct itself
|
|
598
|
+
│ ├── iceberg-type.ts # IcebergType + struct/list/map/decimal/fixed
|
|
599
|
+
│ ├── iceberg-partition-transform.ts
|
|
600
|
+
│ ├── iceberg-table-properties.ts # Format/version enums + property validation
|
|
601
|
+
│ └── index.ts # Re-exports
|
|
602
|
+
├── test/
|
|
603
|
+
│ ├── arceus-stack.test.ts
|
|
604
|
+
│ ├── iceberg-evolution-stack.test.ts
|
|
605
|
+
│ └── iceberg/
|
|
606
|
+
│ ├── iceberg-partition-transform.test.ts
|
|
607
|
+
│ ├── iceberg-table-properties.test.ts
|
|
608
|
+
│ ├── iceberg-table.test.ts
|
|
609
|
+
│ └── iceberg-type.test.ts
|
|
610
|
+
├── scripts/
|
|
611
|
+
│ └── integration-test-evolution.sh # End-to-end evolution harness
|
|
612
|
+
├── cdk.json
|
|
613
|
+
├── package.json
|
|
614
|
+
├── jest.config.js
|
|
615
|
+
├── eslint.config.js
|
|
616
|
+
└── tsconfig.json
|
|
617
|
+
```
|