cdk-glue-iceberg-table 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Rodrigo Carvajal
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,617 @@
1
+ # cdk-glue-iceberg-table
2
+
3
+ [![npm](https://img.shields.io/npm/v/cdk-glue-iceberg-table.svg)](https://www.npmjs.com/package/cdk-glue-iceberg-table)
4
+ [![license](https://img.shields.io/npm/l/cdk-glue-iceberg-table.svg)](LICENSE)
5
+
6
+ A CDK L2 construct for Apache Iceberg tables in the AWS Glue Data
7
+ Catalog. Emits the `AWS::Glue::Table` shape that survives
8
+ CloudFormation `Update`, so `cdk deploy` can create, evolve, and
9
+ destroy Iceberg tables the same way it handles any other resource.
10
+
11
+ The motivating issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
12
+ [`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)
13
+ documents the only working CFN shape and the silent-corruption traps
14
+ you can hit by getting it slightly wrong. This construct implements
15
+ that shape and refuses to emit the unsafe alternatives.
16
+
17
+ The upstream CDK PR landing this construct in
18
+ `@aws-cdk/aws-glue-alpha` is [aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988).
19
+ Until that merges, this package is the most current reference
20
+ implementation. Once `@aws-cdk/aws-glue-alpha` ships its own
21
+ `IcebergTable`, prefer the official one and treat this package as a
22
+ stopgap.
23
+
24
+ > **Status: pre-release.** `0.1.0` has not been published to npm yet
25
+ > — the `npm install` command below will 404 until the first
26
+ > `v*.*.*` tag is pushed (which triggers the publish workflow). To
27
+ > use the construct today, install directly from this repo:
28
+ > `npm install github:ksco92/arceus#main`.
29
+
30
+ ## Install
31
+
32
+ ```bash
33
+ npm install cdk-glue-iceberg-table
34
+ ```
35
+
36
+ Peer dependencies (your CDK app must already have these):
37
+
38
+ ```bash
39
+ npm install aws-cdk-lib constructs @aws-cdk/aws-glue-alpha
40
+ ```
41
+
42
+ ## Use
43
+
44
+ ```ts
45
+ import { Bucket } from 'aws-cdk-lib/aws-s3';
46
+ import { Database } from '@aws-cdk/aws-glue-alpha';
47
+ import {
48
+ IcebergTable,
49
+ IcebergType,
50
+ IcebergPartitionTransform,
51
+ } from 'cdk-glue-iceberg-table';
52
+
53
+ const bucket = new Bucket(this, 'Warehouse');
54
+ const db = new Database(this, 'Db', { databaseName: 'analytics' });
55
+
56
+ new IcebergTable(this, 'OrdersTable', {
57
+ database: db,
58
+ tableName: 'orders',
59
+ location: `s3://${bucket.bucketName}/analytics/orders/`,
60
+ columns: [
61
+ { name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },
62
+ { name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },
63
+ { name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },
64
+ ],
65
+ partitionSpec: [
66
+ { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY },
67
+ { sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16) },
68
+ ],
69
+ identifierFieldNames: ['order_id'],
70
+ });
71
+ ```
72
+
73
+ Consumer-facing reference sections below:
74
+
75
+ - [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.
76
+ - [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.
77
+ - [Known limitations](#known-limitations) — what the construct does and doesn't enforce.
78
+
79
+ ## Repo layout
80
+
81
+ This repo is **both** the published package and a CDK demo app:
82
+
83
+ - `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
84
+ - `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus a bash harness that drives schema + partition evolution through real `cdk deploy`s. **Repo-only, not published to npm.**
85
+
86
+ The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
87
+ [Demo tables](#demo-tables-deployed-by-arceusstack), and
88
+ [Schema + partition evolution](#schema--partition-evolution-via-cdk-only)
89
+ cover the demo app. Skip them if you only want to consume the construct.
90
+
91
+ ## Prerequisites
92
+
93
+ Before running the quickstart you need:
94
+
95
+ 1. AWS credentials in the default profile with permissions to manage
96
+ CloudFormation, KMS, S3, Glue, Lake Formation, Athena, and IAM
97
+ policies. `aws sts get-caller-identity` must return successfully.
98
+ 2. `CDK_DEFAULT_ACCOUNT` and `CDK_DEFAULT_REGION` set in the
99
+ environment (the AWS CLI sets these automatically for most
100
+ profile setups; `cdk` also populates them from the active profile).
101
+ 3. `DEVELOPER_IAM_USER` set to the name of an **existing IAM user**
102
+ in this account. The stack adds that user as a Lake Formation
103
+ admin and grants it per-table `SELECT/INSERT/DELETE/ALTER/DESCRIBE`
104
+ on the demo Iceberg tables — without it the deploy fails when LF
105
+ can't resolve the principal.
106
+ 4. The Lake Formation service-linked role
107
+ `AWSServiceRoleForLakeFormationDataAccess` must exist in the
108
+ account. Create it once with
109
+ `aws iam create-service-linked-role --aws-service-name lakeformation.amazonaws.com`
110
+ if you haven't already.
111
+ 5. `cdk bootstrap aws://<account>/<region>` if the account hasn't
112
+ been bootstrapped for CDK.
113
+
114
+ ## What's in the repo
115
+
116
+ Repo source paths (npm consumers import from the package root and get the same exports under `dist/lib/iceberg/`):
117
+
118
+ - **`lib/iceberg/iceberg-table.ts`** — the `IcebergTable` L2 construct.
119
+ - **`lib/iceberg/iceberg-type.ts`** — `IcebergType` with primitives + `list` / `map` / `struct` factories. Renders to the JSON shape Glue's `IcebergStructField.type` expects.
120
+ - **`lib/iceberg/iceberg-partition-transform.ts`** — `IcebergPartitionTransform` (identity / bucket(N) / truncate(W) / year / month / day / hour / void). Each transform validates against the source column type at synth time.
121
+ - **`lib/iceberg/iceberg-table-properties.ts`** — `IcebergDataFormat` (parquet/orc/avro — default parquet), `IcebergFormatVersion` (v1/v2 — default v2), and a validator that catches misconfigured `tableProperties` before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
122
+ - **`lib/arceus-stack.ts`** — the demo stack: KMS-encrypted data lake bucket, Athena results bucket, Glue database, three demo Iceberg tables (`orders`, `events`, `customers`). Repo-only, not in the npm tarball.
123
+ - **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh`** — a parameterized stack and a bash harness that drives four real `cdk deploy`s to prove schema/partition evolution works end-to-end. Repo-only, not in the npm tarball.
124
+
125
+ ## Quickstart
126
+
127
+ ```bash
128
+ # DEVELOPER_IAM_USER must be the name of an existing IAM user in
129
+ # this account. `aws iam get-user` works when you're authenticated
130
+ # directly as an IAM user; for SSO / assumed-role / aws-vault setups
131
+ # (the modern default) the call fails with "Must specify userName
132
+ # when calling with non-User credentials" and you should set the
133
+ # variable manually to whichever IAM user you want LF to grant on.
134
+ export DEVELOPER_IAM_USER="$(aws iam get-user --query 'User.UserName' --output text)"
135
+
136
+ npm install
137
+ npx jest # 100% line + branch coverage, 95% gate
138
+ npx cdk deploy ArceusStack --require-approval=never
139
+ ./scripts/integration-test-evolution.sh # add + rename + drop, via cdk only
140
+ ```
141
+
142
+ `cdk ls` will show two stacks — `ArceusStack` (the demo data lake +
143
+ three Iceberg tables) and `IcebergEvolutionStack` (the evolution
144
+ test target driven by `scripts/integration-test-evolution.sh`).
145
+ Deploy only `ArceusStack` for the quickstart; the evolution stack
146
+ is created on demand by the script.
147
+
148
+ ## Using `IcebergTable`
149
+
150
+ A minimal table:
151
+
152
+ ```typescript
153
+ import {
154
+ Database,
155
+ } from '@aws-cdk/aws-glue-alpha';
156
+ import {
157
+ IcebergTable,
158
+ IcebergType,
159
+ } from './iceberg';
160
+
161
+ const db = new Database(this, 'Db', {
162
+ databaseName: 'analytics',
163
+ });
164
+
165
+ new IcebergTable(this, 'Users', {
166
+ database: db,
167
+ tableName: 'users',
168
+ columns: [
169
+ {
170
+ name: 'user_id',
171
+ type: IcebergType.LONG,
172
+ required: true,
173
+ id: 1,
174
+ },
175
+ {
176
+ name: 'email',
177
+ type: IcebergType.STRING,
178
+ required: true,
179
+ id: 2,
180
+ },
181
+ {
182
+ name: 'signed_up_at',
183
+ type: IcebergType.TIMESTAMPTZ,
184
+ required: true,
185
+ id: 3,
186
+ },
187
+ ],
188
+ location: `s3://${bucket.bucketName}/analytics/users/`,
189
+ });
190
+ ```
191
+
192
+ A table that exercises most of the surface — partitions, sort order,
193
+ nested types, identifier fields, table properties, removal policy.
194
+ This is the exact shape `ArceusStack` uses for the `orders` demo
195
+ table, so the column list / partition spec / properties round-trip
196
+ straight to the live metadata.json below.
197
+
198
+ ```typescript
199
+ import {
200
+ RemovalPolicy,
201
+ } from 'aws-cdk-lib';
202
+ import {
203
+ Database,
204
+ } from '@aws-cdk/aws-glue-alpha';
205
+ import {
206
+ IcebergDataFormat,
207
+ IcebergFormatVersion,
208
+ IcebergNullOrder,
209
+ IcebergPartitionTransform,
210
+ IcebergSortDirection,
211
+ IcebergTable,
212
+ IcebergType,
213
+ } from './iceberg';
214
+
215
+ new IcebergTable(this, 'OrdersTable', {
216
+ database: db,
217
+ tableName: 'orders',
218
+ comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',
219
+ columns: [
220
+ {
221
+ name: 'order_id',
222
+ type: IcebergType.LONG,
223
+ required: true,
224
+ id: 1,
225
+ },
226
+ {
227
+ name: 'customer_id',
228
+ type: IcebergType.LONG,
229
+ required: true,
230
+ id: 2,
231
+ },
232
+ {
233
+ name: 'order_amount',
234
+ type: IcebergType.decimal(12, 2),
235
+ required: true,
236
+ id: 3,
237
+ },
238
+ {
239
+ name: 'currency',
240
+ type: IcebergType.STRING,
241
+ required: true,
242
+ id: 4,
243
+ },
244
+ {
245
+ name: 'placed_at',
246
+ type: IcebergType.TIMESTAMPTZ,
247
+ required: true,
248
+ id: 5,
249
+ },
250
+ {
251
+ name: 'tags',
252
+ type: IcebergType.list(IcebergType.STRING),
253
+ id: 6,
254
+ },
255
+ {
256
+ name: 'shipping_address',
257
+ type: IcebergType.struct([
258
+ {
259
+ name: 'line1',
260
+ type: IcebergType.STRING,
261
+ required: true,
262
+ },
263
+ {
264
+ name: 'city',
265
+ type: IcebergType.STRING,
266
+ required: true,
267
+ },
268
+ {
269
+ name: 'country',
270
+ type: IcebergType.STRING,
271
+ required: true,
272
+ },
273
+ {
274
+ name: 'postal_code',
275
+ type: IcebergType.STRING,
276
+ },
277
+ ]),
278
+ id: 7,
279
+ },
280
+ {
281
+ name: 'metadata',
282
+ type: IcebergType.map(IcebergType.STRING, IcebergType.STRING, false),
283
+ id: 8,
284
+ },
285
+ ],
286
+ location: `s3://${bucket.bucketName}/analytics/orders/`,
287
+ partitionSpec: [
288
+ {
289
+ sourceColumn: 'placed_at',
290
+ transform: IcebergPartitionTransform.DAY,
291
+ },
292
+ {
293
+ sourceColumn: 'customer_id',
294
+ transform: IcebergPartitionTransform.bucket(16),
295
+ },
296
+ ],
297
+ sortOrder: [
298
+ {
299
+ sourceColumn: 'placed_at',
300
+ direction: IcebergSortDirection.ASC,
301
+ nullOrder: IcebergNullOrder.NULLS_LAST,
302
+ },
303
+ {
304
+ sourceColumn: 'order_id',
305
+ direction: IcebergSortDirection.ASC,
306
+ },
307
+ ],
308
+ identifierFieldNames: [
309
+ 'order_id',
310
+ ],
311
+ dataFormat: IcebergDataFormat.PARQUET,
312
+ formatVersion: IcebergFormatVersion.V2,
313
+ tableProperties: {
314
+ 'write.parquet.compression-codec': 'zstd',
315
+ 'write.delete.mode': 'merge-on-read',
316
+ 'write.update.mode': 'merge-on-read',
317
+ 'write.merge.mode': 'merge-on-read',
318
+ 'write.target-file-size-bytes': '134217728',
319
+ 'history.expire.min-snapshots-to-keep': '5',
320
+ 'gc.enabled': 'true',
321
+ },
322
+ removalPolicy: RemovalPolicy.DESTROY,
323
+ });
324
+ ```
325
+
326
+ ### Granting access
327
+
328
+ ```typescript
329
+ table.grantRead(role); // Glue read + S3 read on the table's prefix
330
+ table.grantWrite(role); // Glue write + S3 write
331
+ table.grantReadWrite(role);
332
+ ```
333
+
334
+ ### Importing an existing table
335
+
336
+ ```typescript
337
+ const existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {
338
+ database: db,
339
+ tableName: 'orders',
340
+ location: 's3://my-bucket/analytics/orders/',
341
+ });
342
+ existing.grantRead(role);
343
+ ```
344
+
345
+ ## Demo tables (deployed by `ArceusStack`)
346
+
347
+ | Table | Format | Columns | Partitions | Sort | Notable properties |
348
+ | --- | --- | --- | --- | --- | --- |
349
+ | `orders` | parquet, v2 | `order_id(1)`, `customer_id(2)`, `order_amount(3)`, `currency(4)`, `placed_at(5)`, `tags(6)` (list), `shipping_address(7)` (struct), `metadata(8)` (map) | `day(placed_at)`, `bucket(16)(customer_id)` | `placed_at ASC NULLS LAST`, `order_id ASC` | `write.{delete,update,merge}.mode = merge-on-read`, `zstd`, `history.expire.min-snapshots-to-keep = 5`, identifier-field-ids = `[order_id]`, nested `list`/`struct`/`map` columns |
350
+ | `events` | parquet, v2 | `event_id(1)`, `event_name(2)`, `session_id(3)`, `occurred_at(4)`, `attributes(5)` (map) | `hour(occurred_at)` | (none) | high-cardinality hourly partitioning |
351
+ | `customers` | parquet, v2 | `customer_id(1)`, `email(2)`, `signed_up_at(4)`, `loyalty_tier(5)` (id 3 retired) | (none) | (none) | identifier-field-ids = `[customer_id]` — the stack's `customers` block carries inline comments narrating the schema-evolution journey that landed here (drop `full_name`, add `loyalty_tier`); the live evolution loop runs against the separate `IcebergEvolutionStack` |
352
+
353
+ After `cdk deploy ArceusStack`, the three tables are queryable from Athena (workgroup `ReadOnly`).
354
+
355
+ ### Validating the demo
356
+
357
+ `SHOW TBLPROPERTIES sample_database.orders` returns:
358
+
359
+ ```
360
+ format parquet
361
+ write_compression zstd
362
+ write_target_data_file_size_bytes 134217728
363
+ vacuum_min_snapshots_to_keep 5
364
+ ```
365
+
366
+ The Iceberg `metadata.json` for `orders` contains every feature you set:
367
+
368
+ ```json
369
+ {
370
+ "format-version": 2,
371
+ "table-uuid": "39a948f9-...",
372
+ "current-schema-id": 0,
373
+ "schemas": [
374
+ {
375
+ "schema-id": 0,
376
+ "identifier-field-ids": [1],
377
+ "fields": [
378
+ { "id": 1, "name": "order_id", "required": true, "type": "long" },
379
+ { "id": 2, "name": "customer_id", "required": true, "type": "long" },
380
+ { "id": 3, "name": "order_amount", "required": true, "type": "decimal(12, 2)" },
381
+ { "id": 4, "name": "currency", "required": true, "type": "string" },
382
+ { "id": 5, "name": "placed_at", "required": true, "type": "timestamptz" },
383
+ { "id": 6, "name": "tags", "required": false,
384
+ "type": { "type": "list", "element-id": 9, "element": "string", "element-required": true } },
385
+ { "id": 7, "name": "shipping_address", "required": false,
386
+ "type": { "type": "struct", "fields": [
387
+ { "id": 10, "name": "line1", "required": true, "type": "string" },
388
+ { "id": 11, "name": "city", "required": true, "type": "string" },
389
+ { "id": 12, "name": "country", "required": true, "type": "string" },
390
+ { "id": 13, "name": "postal_code", "required": false, "type": "string" }
391
+ ] } },
392
+ { "id": 8, "name": "metadata", "required": false,
393
+ "type": { "type": "map", "key-id": 14, "key": "string", "value-id": 15,
394
+ "value-required": false, "value": "string" } }
395
+ ]
396
+ }
397
+ ],
398
+ "partition-specs": [
399
+ { "spec-id": 0, "fields": [
400
+ { "name": "placed_at_day", "transform": "day", "source-id": 5, "field-id": 1000 },
401
+ { "name": "customer_id_bucket", "transform": "bucket[16]", "source-id": 2, "field-id": 1001 }
402
+ ]}
403
+ ],
404
+ "sort-orders": [
405
+ { "order-id": 1, "fields": [
406
+ { "transform": "identity", "source-id": 5, "direction": "asc", "null-order": "nulls-last" },
407
+ { "transform": "identity", "source-id": 1, "direction": "asc", "null-order": "nulls-last" }
408
+ ]}
409
+ ],
410
+ "properties": {
411
+ "format-version": "2",
412
+ "write.format.default": "parquet",
413
+ "write.parquet.compression-codec": "zstd",
414
+ "write.merge.mode": "merge-on-read",
415
+ "write.update.mode": "merge-on-read",
416
+ "write.delete.mode": "merge-on-read",
417
+ "write.target-file-size-bytes": "134217728",
418
+ "history.expire.min-snapshots-to-keep": "5",
419
+ "gc.enabled": "true",
420
+ "comment": "Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read."
421
+ }
422
+ }
423
+ ```
424
+
425
+ ### Inserting and querying
426
+
427
+ ```sql
428
+ -- INSERT into the orders table
429
+ INSERT INTO sample_database.orders VALUES
430
+ (1001, 5001, DECIMAL '149.99', 'USD',
431
+ TIMESTAMP '2026-05-20 09:15:00 UTC',
432
+ ARRAY['holiday-promo','first-order'],
433
+ CAST(ROW('1 Infinite Loop','Cupertino','US','95014')
434
+ AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),
435
+ MAP(ARRAY['channel','utm'], ARRAY['web','google'])),
436
+ -- ... more rows
437
+ ;
438
+
439
+ -- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)
440
+ DELETE FROM sample_database.orders WHERE order_id = 1003;
441
+
442
+ -- merge-on-read UPDATE
443
+ UPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;
444
+
445
+ -- SELECT
446
+ SELECT customer_id, SUM(order_amount) AS total
447
+ FROM sample_database.orders
448
+ GROUP BY 1
449
+ ORDER BY 2 DESC;
450
+ ```
451
+
452
+ ## Schema + partition evolution via cdk-only
453
+
454
+ `scripts/integration-test-evolution.sh` drives the
455
+ `IcebergEvolutionStack` through four `cdk deploy`s and verifies the
456
+ underlying Iceberg `metadata.json` after each:
457
+
458
+ | Step | Change | Columns | Partitions |
459
+ | ---: | --- | --- | --- |
460
+ | 1 | Initial deploy | `customer_id(1)`, `email(2)`, `signed_up_at(3)` | `day(signed_up_at)` |
461
+ | 2 | **ADD** column `region(4)` | + `region(4)` | unchanged |
462
+ | 3 | **RENAME** `email` → `contact_email` (id 2 preserved), **ADD** partition `bucket(8)(customer_id)` | rename | + `bucket(8)(customer_id)` |
463
+ | 4 | **DROP** column `region` (id 4 stays retired), **DROP** partition `bucket(8)(customer_id)` | − `region` | − `customer_id_bucket` |
464
+
465
+ Last script run output (abridged — `cdk deploy` chatter and the
466
+ per-Athena-query state polling lines are omitted; the assertion
467
+ output is verbatim):
468
+
469
+ ```
470
+ === STEP 1 — cdk deploy ===
471
+ ✨ Total time: 10.57s
472
+
473
+ === VERIFY step 1 ===
474
+ columns ✓ (1:customer_id,2:email,3:signed_up_at)
475
+ partitions ✓ (signed_up_at_day)
476
+
477
+ === INSERT seed rows ===
478
+ 3 rows inserted
479
+
480
+ === STEP 2 — cdk deploy ===
481
+ === VERIFY step 2 (ADD column) ===
482
+ columns ✓ (1:customer_id,2:email,3:signed_up_at,4:region)
483
+ partitions ✓ (signed_up_at_day)
484
+
485
+ === VERIFY old rows are preserved with region=NULL ===
486
+ pre-existing rows readable ✓
487
+ inserted 1 row carrying region='us-east-1'
488
+
489
+ === STEP 3 — cdk deploy ===
490
+ === VERIFY step 3 (RENAME column + ADD partition) ===
491
+ columns ✓ (1:customer_id,2:contact_email,3:signed_up_at,4:region)
492
+ partitions ✓ (signed_up_at_day,customer_id_bucket)
493
+ rename preserved data ✓
494
+
495
+ === STEP 4 — cdk deploy ===
496
+ === VERIFY step 4 (DROP column + DROP partition) ===
497
+ columns ✓ (1:customer_id,2:contact_email,3:signed_up_at)
498
+ partitions ✓ (signed_up_at_day)
499
+ last-column-id stays at 4 — id reuse protection ✓
500
+ all 4 pre-existing rows queryable after drop ✓
501
+
502
+ === TEARDOWN ===
503
+ IcebergEvolutionStack | 4 | DELETE_COMPLETE | AWS::CloudFormation::Stack
504
+ ✅ IcebergEvolutionStack: destroyed
505
+
506
+ === ALL EVOLUTION STEPS PASSED ===
507
+ ```
508
+
509
+ The construct passes each new column list + partition spec to Glue
510
+ `UpdateTable` via `OpenTableFormatInput`. Glue computes the Iceberg
511
+ metadata delta (new `schema-id`, new `spec-id`) and writes a new
512
+ `metadata.json`. Old data files stay readable because the field IDs
513
+ the construct pins (`id: N` on each `IcebergColumn`) never change
514
+ across deploys.
515
+
516
+ ## Two footguns the construct prevents
517
+
518
+ ### Footgun #1 — schema under `storageDescriptor.columns`
519
+
520
+ The CREATE succeeds but the first UPDATE silently strips
521
+ `table_type=ICEBERG` from the table's Glue parameters, and Athena
522
+ queries after that fail with `HIVE_UNSUPPORTED_FORMAT`.
523
+
524
+ ```typescript
525
+ // DON'T DO THIS — what most StackOverflow / re:Post examples show
526
+ new CfnTable(this, 'OrdersBad', {
527
+ catalogId: this.account,
528
+ databaseName: 'analytics',
529
+ tableInput: {
530
+ name: 'orders',
531
+ tableType: 'EXTERNAL_TABLE',
532
+ parameters: {
533
+ table_type: 'ICEBERG',
534
+ },
535
+ storageDescriptor: {
536
+ location: 's3://.../orders/',
537
+ columns: [
538
+ /* ... */
539
+ ],
540
+ },
541
+ },
542
+ openTableFormatInput: {
543
+ icebergInput: {
544
+ metadataOperation: 'CREATE',
545
+ version: '2',
546
+ },
547
+ },
548
+ });
549
+ ```
550
+
551
+ `IcebergTable` instead always emits schema/partitions/sort/properties
552
+ under `openTableFormatInput.icebergInput.icebergTableInput`, never
553
+ under `storageDescriptor`.
554
+
555
+ ### Footgun #2 — `tableInput` co-present with `openTableFormatInput`
556
+
557
+ Even setting just `tableInput: { name: 'foo' }` next to
558
+ `openTableFormatInput` returns
559
+ `"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput"`.
560
+ The construct never emits `tableInput` — table-level comment goes
561
+ into `tableProperties['comment']`, which lives inside
562
+ `icebergTableInput.properties`.
563
+
564
+ (There is a third footgun — field-id reuse after a column drop — that
565
+ the construct does **not** prevent. See the next section.)
566
+
567
+ ## Known limitations
568
+
569
+ - **Field-id reuse is not detected across deploys.** If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the "never reuse a retired id" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate column id N` validator), but it doesn't compare against the live table state. The safe workflow is to always pin `id` explicitly and treat dropped ids as retired forever; never let CDK reassign an id that has ever been used.
570
+ - **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.
571
+ - **Format version is immutable after CREATE.** The `formatVersion` prop is read once at table creation; changing it later requires a destroy + recreate.
572
+ - **`merge-on-read` requires v2.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.
573
+ - **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.
574
+ - **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The integration-test script demonstrates the safe pattern: drop partitions that source from `customer_id` while keeping `customer_id` itself in the schema, and drop the `region` column while it is not partitioning anything.
575
+
576
+ ## Tests
577
+
578
+ ```
579
+ npx jest # runs every suite under test/, prints coverage at the end
580
+ ```
581
+
582
+ Coverage is gated at 95% statements / 95% branches / 95% lines / 95% functions
583
+ on `lib/**/*.ts` via the `coverageThreshold.global` block in `jest.config.js`.
584
+ A failing gate fails the suite, so the README does not paste a transcript that
585
+ would drift after the next refactor — run the command locally for the live
586
+ numbers.
587
+
588
+ ## Project layout
589
+
590
+ ```
591
+ arceus/
592
+ ├── bin/arceus.ts # CDK app entry point
593
+ ├── lib/
594
+ │ ├── arceus-stack.ts # Demo stack (buckets, DB, 3 demo tables)
595
+ │ ├── iceberg-evolution-stack.ts # Parameterized stack for the evolution test
596
+ │ └── iceberg/
597
+ │ ├── iceberg-table.ts # The L2 construct itself
598
+ │ ├── iceberg-type.ts # IcebergType + struct/list/map/decimal/fixed
599
+ │ ├── iceberg-partition-transform.ts
600
+ │ ├── iceberg-table-properties.ts # Format/version enums + property validation
601
+ │ └── index.ts # Re-exports
602
+ ├── test/
603
+ │ ├── arceus-stack.test.ts
604
+ │ ├── iceberg-evolution-stack.test.ts
605
+ │ └── iceberg/
606
+ │ ├── iceberg-partition-transform.test.ts
607
+ │ ├── iceberg-table-properties.test.ts
608
+ │ ├── iceberg-table.test.ts
609
+ │ └── iceberg-type.test.ts
610
+ ├── scripts/
611
+ │ └── integration-test-evolution.sh # End-to-end evolution harness
612
+ ├── cdk.json
613
+ ├── package.json
614
+ ├── jest.config.js
615
+ ├── eslint.config.js
616
+ └── tsconfig.json
617
+ ```