cdk-glue-iceberg-table 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -3,43 +3,91 @@
3
3
  [![CI](https://img.shields.io/github/actions/workflow/status/ksco92/arceus/ci.yml?branch=main&label=CI)](https://github.com/ksco92/arceus/actions/workflows/ci.yml)
4
4
  [![coverage](https://codecov.io/gh/ksco92/arceus/branch/main/graph/badge.svg)](https://codecov.io/gh/ksco92/arceus)
5
5
  [![npm](https://img.shields.io/npm/v/cdk-glue-iceberg-table.svg)](https://www.npmjs.com/package/cdk-glue-iceberg-table)
6
+ [![PyPI](https://img.shields.io/pypi/v/cdk-glue-iceberg-table.svg)](https://pypi.org/project/cdk-glue-iceberg-table/)
7
+ [![Construct Hub](https://constructs.dev/badge?package=cdk-glue-iceberg-table)](https://constructs.dev/packages/cdk-glue-iceberg-table)
6
8
  [![types](https://img.shields.io/npm/types/cdk-glue-iceberg-table.svg)](https://www.npmjs.com/package/cdk-glue-iceberg-table)
7
9
  [![downloads](https://img.shields.io/npm/dw/cdk-glue-iceberg-table.svg)](https://www.npmjs.com/package/cdk-glue-iceberg-table)
8
10
  [![last commit](https://img.shields.io/github/last-commit/ksco92/arceus.svg)](https://github.com/ksco92/arceus/commits/main)
9
11
  [![license](https://img.shields.io/npm/l/cdk-glue-iceberg-table.svg)](LICENSE)
10
12
 
11
- A CDK L2 construct for Apache Iceberg tables in the AWS Glue Data
12
- Catalog. Emits the `AWS::Glue::Table` shape that survives
13
- CloudFormation `Update`, so `cdk deploy` can create, evolve, and
14
- destroy Iceberg tables the same way it handles any other resource.
15
-
16
- The motivating issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
13
+ **`cdk-glue-iceberg-table` is the AWS CDK L2 construct for creating
14
+ and evolving Apache Iceberg tables in the AWS Glue Data Catalog.** It
15
+ emits the exact `AWS::Glue::Table` / `OpenTableFormatInput` shape that
16
+ survives a CloudFormation `Update`, so a single `cdk deploy` creates a
17
+ table, evolves its schema and partitions, and destroys it, the same
18
+ lifecycle CDK gives any other resource. No custom resource, no Lambda,
19
+ no "`cdk deploy` and then run this SQL by hand" two-step. Table
20
+ changes (new columns, renames, drops, new partition fields) land as
21
+ a reviewed diff in a pull request and apply through `cdk deploy`.
22
+
23
+ Status (June 2026): published on npm (TypeScript/JavaScript) and PyPI
24
+ (Python), pre-1.0, with a public surface guarded by an end-to-end
25
+ consumer test on every PR plus three real-AWS integration suites. The
26
+ multi-language packages are generated from one TypeScript source with
27
+ [jsii](https://aws.github.io/jsii/) and indexed on
28
+ [Construct Hub](https://constructs.dev/packages/cdk-glue-iceberg-table).
29
+ Developed in the
30
+ [`ksco92/arceus`](https://github.com/ksco92/arceus) monorepo, which
31
+ also holds a CDK demo app that dogfoods the construct against a real
32
+ AWS account.
33
+
34
+ ### Why this exists
35
+
36
+ AWS documents the CloudFormation shape for Iceberg tables ([AWS Big
37
+ Data blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),
38
+ but raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the
39
+ *first* `Update` silently strips `table_type=ICEBERG`, after which
40
+ every Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The
41
+ motivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
17
42
  [`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)
18
- documents the only working CFN shape and the silent-corruption traps
19
- you can hit by getting it slightly wrong. This construct implements
20
- that shape and refuses to emit the unsafe alternatives.
21
-
22
- The upstream CDK PR landing this construct in
23
- `@aws-cdk/aws-glue-alpha` is [aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988).
24
- Until that merges, this package is the most current reference
25
- implementation. Once `@aws-cdk/aws-glue-alpha` ships its own
26
- `IcebergTable`, prefer the official one and treat this package as a
27
- stopgap.
43
+ documents the only working shape and the silent-corruption traps you
44
+ hit by getting it slightly wrong. This construct implements that
45
+ shape, refuses to emit the unsafe alternatives, and pins Iceberg field
46
+ IDs so schema evolution never orphans existing data. It is the basis
47
+ of the upstream proposal to land an `IcebergTable` in
48
+ `@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988));
49
+ this package tracks that proposal and stays current with it.
50
+
51
+ ### How it compares
52
+
53
+ | Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |
54
+ | --- | --- | --- | --- | --- |
55
+ | Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |
56
+ | CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |
57
+ | Spark / Athena SQL DDL at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |
58
+ | **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |
28
59
 
29
60
  ## Install
30
61
 
62
+ TypeScript / JavaScript (npm):
63
+
31
64
  ```bash
32
65
  npm install cdk-glue-iceberg-table
33
66
  ```
34
67
 
35
- Peer dependencies (your CDK app must already have these):
68
+ Python (PyPI):
69
+
70
+ ```bash
71
+ pip install cdk-glue-iceberg-table
72
+ ```
73
+
74
+ Peer dependencies (your CDK app must already have these). For
75
+ TypeScript / JavaScript:
36
76
 
37
77
  ```bash
38
78
  npm install aws-cdk-lib constructs @aws-cdk/aws-glue-alpha
39
79
  ```
40
80
 
81
+ For Python:
82
+
83
+ ```bash
84
+ pip install aws-cdk-lib constructs aws-cdk.aws-glue-alpha
85
+ ```
86
+
41
87
  ## Use
42
88
 
89
+ ### TypeScript
90
+
43
91
  ```ts
44
92
  import { Bucket } from 'aws-cdk-lib/aws-s3';
45
93
  import { Database } from '@aws-cdk/aws-glue-alpha';
@@ -69,116 +117,60 @@ new IcebergTable(this, 'OrdersTable', {
69
117
  });
70
118
  ```
71
119
 
72
- Consumer-facing reference sections below:
120
+ ### Python
73
121
 
122
+ The same table in Python — the API mirrors TypeScript with
123
+ `snake_case` props and `PascalCase` types:
124
+
125
+ ```python
126
+ from aws_cdk.aws_s3 import Bucket
127
+ from aws_cdk.aws_glue_alpha import Database
128
+ from cdk_glue_iceberg_table import (
129
+ IcebergTable,
130
+ IcebergType,
131
+ IcebergPartitionTransform,
132
+ )
133
+
134
+ bucket = Bucket(self, "Warehouse")
135
+ db = Database(self, "Db", database_name="analytics")
136
+
137
+ IcebergTable(self, "OrdersTable",
138
+ database=db,
139
+ table_name="orders",
140
+ location=f"s3://{bucket.bucket_name}/analytics/orders/",
141
+ columns=[
142
+ {"name": "order_id", "type": IcebergType.LONG, "required": True, "id": 1},
143
+ {"name": "customer_id", "type": IcebergType.LONG, "required": True, "id": 2},
144
+ {"name": "placed_at", "type": IcebergType.TIMESTAMPTZ, "required": True, "id": 3},
145
+ ],
146
+ partition_spec=[
147
+ {"source_column": "placed_at", "transform": IcebergPartitionTransform.DAY},
148
+ {"source_column": "customer_id", "transform": IcebergPartitionTransform.bucket(16)},
149
+ ],
150
+ identifier_field_names=["order_id"],
151
+ )
152
+ ```
153
+
154
+ Consumer-facing reference sections:
155
+
156
+ - [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.
74
157
  - [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.
75
158
  - [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.
76
159
  - [Known limitations](#known-limitations) — what the construct does and doesn't enforce.
160
+ - [FAQ](#faq) — common "how do I … in CDK / CloudFormation" questions.
77
161
 
78
- ## Repo layout
79
-
80
- This repo is **both** the published package and a CDK demo app:
81
-
82
- - `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
83
- - `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `lib/iceberg-dml-stack.ts`, `lib/iceberg-surface-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus three bash harnesses: one drives schema + partition evolution through real `cdk deploy`s, one exercises the v2 DML surface (UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM), and one covers the remaining surface (every partition transform, multi-field sort order, list / struct / map columns, and `grantRead` at runtime by assuming the grantee role and calling Glue / S3 directly). **Repo-only, not published to npm.**
84
- - `e2e-consumer/` — a standalone CDK app that depends on the **published** `cdk-glue-iceberg-table` from npm. Proves that a fresh install + import + `cdk synth` works for downstream consumers. Runs on every PR via the `e2e-consumer` job in `.github/workflows/ci.yml`. Its `lib/surface-reference.ts` touches every exported symbol so that a rename in the published surface breaks CI. The pin in `e2e-consumer/package-lock.json` tracks the version most recently published to npm; CLAUDE.md asks for it to be bumped after each release.
85
-
86
- How the test gates fit together:
87
-
88
- - **`ci.yml`** runs on every PR — lint, unit tests with the 95% coverage gate, `npm pack`, and the `e2e-consumer` synth against the pinned published npm version.
89
- - **`integ-test.yml`** is the real-AWS gate. Runs three scripts back-to-back: `scripts/integration-test-evolution.sh` (four `cdk deploy`s exercising schema + partition evolution), `scripts/integration-test-dml.sh` (one deploy, then UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM against a v2 merge-on-read table), and `scripts/integration-test-surface.sh` (one deploy, then every partition transform, multi-field sort order, nested-type roundtrip, and `grantRead` at runtime via assume-role + direct Glue/S3 calls). Gated by `run-integ-test` label or `/run-integ-test` collaborator comment. PRs that touch any file under `lib/`, `bin/arceus.ts`, `cdk.json`, or any of the scripts must show a green run before merging (see CLAUDE.md §"Integration test for construct-touching PRs"). Doc-only PRs are exempt.
90
- - **`publish.yml`** runs on push to `main` — trusted-publish to npm when `package.json`'s `version` is newer than the registry.
91
-
92
- The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
93
- [Demo tables](#demo-tables-deployed-by-arceusstack), and
94
- [Schema + partition evolution](#schema--partition-evolution-via-cdk-only)
95
- cover the demo app. Skip them if you only want to consume the construct.
96
-
97
- ## Prerequisites
98
-
99
- Before running the quickstart you need:
100
-
101
- 1. AWS credentials in the default profile with permissions to manage
102
- CloudFormation, KMS, S3, Glue, Lake Formation, Athena, and IAM
103
- policies. `aws sts get-caller-identity` must return successfully.
104
- 2. `CDK_DEFAULT_ACCOUNT` and `CDK_DEFAULT_REGION` set in the
105
- environment (the AWS CLI sets these automatically for most
106
- profile setups; `cdk` also populates them from the active profile).
107
- 3. `PRINCIPAL_ARN` set to the **ARN of an existing IAM principal**
108
- in this account (IAM user, role, or federated identity). The
109
- stack adds that principal as a Lake Formation admin and grants
110
- it per-table `SELECT/INSERT/DELETE/ALTER/DESCRIBE` on the demo
111
- Iceberg tables. Without it the deploy fails when LF can't resolve
112
- the principal. The same ARN must also be the identity running
113
- `cdk deploy` and any subsequent Athena queries. Local devs
114
- typically set this to their IAM user ARN; CI (`integ-test.yml`)
115
- sets it to the OIDC role ARN.
116
- - **Only one principal is privileged at a time.** `ArceusStack`
117
- writes the Lake Formation admin list with REPLACE semantics,
118
- so the most recent `cdk deploy` wins. A local `cdk deploy` with
119
- your user ARN revokes the OIDC role's LF grants (and breaks the
120
- next integ-test run until CI redeploys), and vice versa.
121
- Coordinate accordingly.
122
- - **SSO / `aws-vault` users:** set `PRINCIPAL_ARN` to the
123
- canonical role ARN (`arn:aws:iam::<acct>:role/<RoleName>`), not
124
- the per-session `arn:aws:sts::<acct>:assumed-role/...` you'd
125
- get from `aws sts get-caller-identity`. LF grants on the
126
- session-suffixed ARN go stale at the next SSO refresh.
127
- 4. The Lake Formation service-linked role
128
- `AWSServiceRoleForLakeFormationDataAccess` must exist in the
129
- account. Create it once with
130
- `aws iam create-service-linked-role --aws-service-name lakeformation.amazonaws.com`
131
- if you haven't already.
132
- 5. `cdk bootstrap aws://<account>/<region>` if the account hasn't
133
- been bootstrapped for CDK.
134
-
135
- ## What's in the repo
136
-
137
- Repo source paths (npm consumers import from the package root and get the same exports under `dist/lib/iceberg/`):
138
-
139
- - **`lib/iceberg/iceberg-table.ts`**: the `IcebergTable` L2 construct.
140
- - **`lib/iceberg/iceberg-type.ts`**: `IcebergType` with primitives + `list` / `map` / `struct` factories. Renders to the JSON shape Glue's `IcebergStructField.type` expects.
141
- - **`lib/iceberg/iceberg-partition-transform.ts`**: `IcebergPartitionTransform` (identity / bucket(N) / truncate(W) / year / month / day / hour / void). Each transform validates against the source column type at synth time.
142
- - **`lib/iceberg/iceberg-table-properties.ts`**: `IcebergDataFormat` (parquet/orc/avro, default parquet), `IcebergFormatVersion` (v1/v2, required — set explicitly per table), and a validator that catches misconfigured `tableProperties` before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
143
- - **`lib/arceus-stack.ts`**: the demo stack: KMS-encrypted data lake bucket, Athena results bucket, Glue database, three demo Iceberg tables (`orders`, `events`, `customers`). Repo-only, not in the npm tarball.
144
- - **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh`**: a parameterized stack and a bash harness that drives four real `cdk deploy`s to prove schema/partition evolution works end-to-end. Repo-only, not in the npm tarball.
145
- - **`lib/iceberg-dml-stack.ts`** + **`scripts/integration-test-dml.sh`**: a v2 merge-on-read table and a bash harness that exercises UPDATE, DELETE, MERGE INTO (upsert), time-travel SELECT, OPTIMIZE compaction, and VACUUM snapshot expiration. Single deploy, ~3 minutes. Repo-only, not in the npm tarball.
146
- - **`lib/iceberg-surface-stack.ts`** + **`scripts/integration-test-surface.sh`**: three tables (one per concern) and a harness that verifies every partition transform renders the right `metadata.json`, a three-field sort order with mixed direction + null ordering, list/struct/map columns roundtrip through Athena, and `grantRead`'s four-statement IAM split actually authorizes (and denies cross-prefix) at runtime under an assumed grantee role. The same checks run against `IcebergTable.fromIcebergTableAttributes(...)` to prove the import factory's grant path is symmetric with the native one. Single deploy, ~4 minutes. Repo-only, not in the npm tarball.
147
-
148
- ## Quickstart
162
+ ## Exported surface
149
163
 
150
- ```bash
151
- # PRINCIPAL_ARN is the ARN of the IAM principal (user, role, or
152
- # federated identity) that the stack should make a Lake Formation
153
- # admin and per-table grantee. It must equal the identity running
154
- # `cdk deploy` and any subsequent Athena queries — otherwise the
155
- # integration script's INSERT/SELECT calls fail with `Principal does
156
- # not have any privilege on specified resource`.
157
- #
158
- # For a direct IAM-user session, the line below returns the right ARN
159
- # (`arn:aws:iam::<acct>:user/<name>`).
160
- #
161
- # For SSO / aws-vault / any assumed-role session this returns
162
- # `arn:aws:sts::<acct>:assumed-role/<RoleName>/<SessionName>`, which
163
- # Lake Formation accepts but stales on the next session refresh
164
- # because the SessionName changes. Set PRINCIPAL_ARN to the canonical
165
- # `arn:aws:iam::<acct>:role/<RoleName>` instead — for example:
166
- # export PRINCIPAL_ARN="arn:aws:iam::123456789012:role/MyDevRole"
167
- export PRINCIPAL_ARN="$(aws sts get-caller-identity --query Arn --output text)"
168
-
169
- npm install
170
- npx jest # runs the suite; coverage floor in jest.config.js
171
- npx cdk deploy ArceusStack --require-approval=never
172
- ./scripts/integration-test-evolution.sh # add + rename + drop, via cdk only
173
- ```
164
+ The package entry point re-exports everything you import from
165
+ `cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled
166
+ surface lives under `dist/lib/iceberg/` after `npm install`; in Python
167
+ the same surface is imported from the `cdk_glue_iceberg_table` module
168
+ after `pip install`:
174
169
 
175
- `cdk ls` will show four stacks: `ArceusStack` (the demo data lake +
176
- three Iceberg tables), `IcebergEvolutionStack` (the evolution test
177
- target), `IcebergDmlStack` (the DML test target), and
178
- `IcebergSurfaceStack` (the transforms / sort / nested-types /
179
- grants test target). Deploy only `ArceusStack` for the quickstart;
180
- the three test stacks are created on demand by their respective
181
- scripts under `scripts/`.
170
+ - **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite` and the `fromIcebergTableAttributes(...)` import factory.
171
+ - **`IcebergType`** — primitive types plus `list` / `map` / `struct` / `decimal` / `fixed` factories. Renders to the JSON shape Glue's `IcebergStructField.type` expects.
172
+ - **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.
173
+ - **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2, required — set explicitly per table), **`IcebergSortDirection`**, **`IcebergNullOrder`**, plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
182
174
 
183
175
  ## Using `IcebergTable`
184
176
 
@@ -225,10 +217,7 @@ new IcebergTable(this, 'Users', {
225
217
  ```
226
218
 
227
219
  A table that exercises most of the surface (partitions, sort order,
228
- nested types, identifier fields, table properties, removal policy).
229
- This is the exact shape `ArceusStack` uses for the `orders` demo
230
- table, so the column list / partition spec / properties round-trip
231
- straight to the live metadata.json below.
220
+ nested types, identifier fields, table properties, removal policy):
232
221
 
233
222
  ```typescript
234
223
  import {
@@ -358,47 +347,8 @@ new IcebergTable(this, 'OrdersTable', {
358
347
  });
359
348
  ```
360
349
 
361
- ### Granting access
362
-
363
- ```typescript
364
- table.grantRead(role); // Glue read + S3 read on the table's prefix
365
- table.grantWrite(role); // Glue write + S3 write
366
- table.grantReadWrite(role);
367
- ```
368
-
369
- ### Importing an existing table
370
-
371
- ```typescript
372
- const existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {
373
- database: db,
374
- tableName: 'orders',
375
- location: 's3://my-bucket/analytics/orders/',
376
- });
377
- existing.grantRead(role);
378
- ```
379
-
380
- ## Demo tables (deployed by `ArceusStack`)
381
-
382
- | Table | Format | Columns | Partitions | Sort | Notable properties |
383
- | --- | --- | --- | --- | --- | --- |
384
- | `orders` | parquet, v2 | `order_id(1)`, `customer_id(2)`, `order_amount(3)`, `currency(4)`, `placed_at(5)`, `tags(6)` (list), `shipping_address(7)` (struct), `metadata(8)` (map) | `day(placed_at)`, `bucket(16)(customer_id)` | `placed_at ASC NULLS LAST`, `order_id ASC` | `write.{delete,update,merge}.mode = merge-on-read`, `zstd`, `history.expire.min-snapshots-to-keep = 5`, identifier-field-ids = `[order_id]`, nested `list`/`struct`/`map` columns |
385
- | `events` | parquet, v2 | `event_id(1)`, `event_name(2)`, `session_id(3)`, `occurred_at(4)`, `attributes(5)` (map) | `hour(occurred_at)` | (none) | high-cardinality hourly partitioning |
386
- | `customers` | parquet, v2 | `customer_id(1)`, `email(2)`, `signed_up_at(4)`, `loyalty_tier(5)` (id 3 retired) | (none) | (none) | identifier-field-ids = `[customer_id]` — the stack's `customers` block carries inline comments narrating the schema-evolution journey that landed here (drop `full_name`, add `loyalty_tier`); the live evolution loop runs against the separate `IcebergEvolutionStack` |
387
-
388
- After `cdk deploy ArceusStack`, the three tables are queryable from Athena (workgroup `ReadOnly`).
389
-
390
- ### Validating the demo
391
-
392
- `SHOW TBLPROPERTIES sample_database.orders` returns:
393
-
394
- ```
395
- format parquet
396
- write_compression zstd
397
- write_target_data_file_size_bytes 134217728
398
- vacuum_min_snapshots_to_keep 5
399
- ```
400
-
401
- The Iceberg `metadata.json` for `orders` contains every feature you set:
350
+ The resulting Iceberg `metadata.json` for this table contains every
351
+ feature you set:
402
352
 
403
353
  ```json
404
354
  {
@@ -457,6 +407,39 @@ The Iceberg `metadata.json` for `orders` contains every feature you set:
457
407
  }
458
408
  ```
459
409
 
410
+ ### Granting access
411
+
412
+ ```typescript
413
+ table.grantRead(role); // Glue read + S3 read on the table's prefix
414
+ table.grantWrite(role); // Glue write + S3 write
415
+ table.grantReadWrite(role);
416
+ ```
417
+
418
+ The `grant*` helpers issue **IAM grants only**. Under Lake Formation
419
+ you still add the matching `SELECT` / `INSERT` / `DELETE` LF grants on
420
+ top of the construct's IAM grants for Athena queries to succeed.
421
+
422
+ ### Importing an existing table
423
+
424
+ ```typescript
425
+ const existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {
426
+ database: db,
427
+ tableName: 'orders',
428
+ location: 's3://my-bucket/analytics/orders/',
429
+ });
430
+ existing.grantRead(role);
431
+ ```
432
+
433
+ ### Evolving schema and partitions
434
+
435
+ Change the `columns` array (or `partitionSpec`) and run `cdk deploy`
436
+ again. The construct passes the new schema to Glue's `UpdateTable`,
437
+ which writes a new `metadata.json` with a new `schema-id`; existing
438
+ data files stay readable because each column's `id` is pinned and
439
+ never reused. Adds, renames (same `id`, new `name`), and drops all
440
+ flow through `cdk deploy` alone — no out-of-band SQL DDL. The same
441
+ applies to `partitionSpec`.
442
+
460
443
  ### Inserting and querying
461
444
 
462
445
  ```sql
@@ -484,118 +467,6 @@ SELECT customer_id, SUM(order_amount) AS total
484
467
  ORDER BY 2 DESC;
485
468
  ```
486
469
 
487
- ## Schema + partition evolution via cdk-only
488
-
489
- `scripts/integration-test-evolution.sh` drives the
490
- `IcebergEvolutionStack` through four `cdk deploy`s and verifies the
491
- underlying Iceberg `metadata.json` after each:
492
-
493
- | Step | Change | Columns | Partitions |
494
- | ---: | --- | --- | --- |
495
- | 1 | Initial deploy | `customer_id(1)`, `email(2)`, `signed_up_at(3)` | `day(signed_up_at)` |
496
- | 2 | **ADD** column `region(4)` | + `region(4)` | unchanged |
497
- | 3 | **RENAME** `email` → `contact_email` (id 2 preserved), **ADD** partition `bucket(8)(customer_id)` | rename | + `bucket(8)(customer_id)` |
498
- | 4 | **DROP** column `region` (id 4 stays retired), **DROP** partition `bucket(8)(customer_id)` | − `region` | − `customer_id_bucket` |
499
-
500
- Last script run output (abridged: `cdk deploy` chatter and the
501
- per-Athena-query state polling lines are omitted; the assertion
502
- output is verbatim):
503
-
504
- ```
505
- === STEP 1 — cdk deploy ===
506
- ✨ Total time: 10.57s
507
-
508
- === VERIFY step 1 ===
509
- columns ✓ (1:customer_id,2:email,3:signed_up_at)
510
- partitions ✓ (signed_up_at_day)
511
-
512
- === INSERT seed rows ===
513
- 3 rows inserted
514
-
515
- === STEP 2 — cdk deploy ===
516
- === VERIFY step 2 (ADD column) ===
517
- columns ✓ (1:customer_id,2:email,3:signed_up_at,4:region)
518
- partitions ✓ (signed_up_at_day)
519
-
520
- === VERIFY old rows are preserved with region=NULL ===
521
- pre-existing rows readable ✓
522
- inserted 1 row carrying region='us-east-1'
523
-
524
- === STEP 3 — cdk deploy ===
525
- === VERIFY step 3 (RENAME column + ADD partition) ===
526
- columns ✓ (1:customer_id,2:contact_email,3:signed_up_at,4:region)
527
- partitions ✓ (signed_up_at_day,customer_id_bucket)
528
- rename preserved data ✓
529
-
530
- === STEP 4 — cdk deploy ===
531
- === VERIFY step 4 (DROP column + DROP partition) ===
532
- columns ✓ (1:customer_id,2:contact_email,3:signed_up_at)
533
- partitions ✓ (signed_up_at_day)
534
- last-column-id stays at 4 — id reuse protection ✓
535
- all 4 pre-existing rows queryable after drop ✓
536
-
537
- === TEARDOWN ===
538
- IcebergEvolutionStack | 4 | DELETE_COMPLETE | AWS::CloudFormation::Stack
539
- ✅ IcebergEvolutionStack: destroyed
540
-
541
- === ALL EVOLUTION STEPS PASSED ===
542
- ```
543
-
544
- The construct passes each new column list + partition spec to Glue
545
- `UpdateTable` via `OpenTableFormatInput`. Glue computes the Iceberg
546
- metadata delta (new `schema-id`, new `spec-id`) and writes a new
547
- `metadata.json`. Old data files stay readable because the field IDs
548
- the construct pins (`id: N` on each `IcebergColumn`) never change
549
- across deploys.
550
-
551
- ## DML, time travel, OPTIMIZE, and VACUUM
552
-
553
- `scripts/integration-test-dml.sh` covers the v2 surface that the
554
- evolution test doesn't. The harness deploys `IcebergDmlStack` once,
555
- then runs a sequence of Athena statements against a v2 merge-on-read
556
- table with `identifierFieldNames: ['account_id']`:
557
-
558
- | Step | Statement | Verify |
559
- | ---: | --- | --- |
560
- | 1 | `INSERT` 5 seed rows | row count == 5 |
561
- | 2 | `UPDATE balance WHERE account_id = 2` | balance == 250, row count == 5 |
562
- | 3 | `DELETE WHERE account_id = 4` | row count == 4, account_id=4 gone |
563
- | 4 | capture pre-MERGE snapshot id from `dml_test$snapshots` | — |
564
- | 5 | `MERGE INTO ... USING ... ON account_id` (update id=3, insert id=6 + id=7) | row count == 6, id=3 balance updated, ids 6 and 7 present |
565
- | 6 | `SELECT ... FOR VERSION AS OF <pre-MERGE snapshot>` | time-travel still sees 4 rows, no id=6 |
566
- | 7 | `OPTIMIZE ... REWRITE DATA USING BIN_PACK` | succeeds, row count unchanged |
567
- | 8 | `VACUUM ...` (after a 65-s sleep to clear `max-snapshot-age-ms`) | succeeds, row count unchanged |
568
- | 9 | final `SELECT` | 6 rows with the expected balances |
569
-
570
- The DML table is configured with `history.expire.max-snapshot-age-ms`
571
- of 60 seconds so VACUUM has snapshots to expire on a fresh table.
572
- All three scripts run sequentially in the same `integ-test.yml`
573
- job; evolution takes ~5 minutes, DML ~3, and surface ~4.
574
-
575
- ## Partition transforms, sort order, nested types, and grants
576
-
577
- `scripts/integration-test-surface.sh` covers the slice of the
578
- construct surface the evolution and DML stacks don't reach.
579
- `IcebergSurfaceStack` defines three small tables that each isolate
580
- one concern:
581
-
582
- | Concern | Table | What the script checks |
583
- | --- | --- | --- |
584
- | **Every partition transform** | `transforms_test` (separate `year_source` / `month_source` / `day_source` / `hour_source` timestamps to avoid Iceberg's redundant-temporal-transform rejection, plus `user_id` / `email` / `value`) | `metadata.json` contains all seven transforms (`year`, `month`, `day`, `hour`, `bucket[8]`, `truncate[4]`, `identity`); INSERT one row and verify the resulting S3 prefix contains the expected multi-transform layout (`year_source_year=...`, `month_source_month=...`, etc.) |
585
- | **Sort order** | `sorted_test` (`tenant`, `created_at`, `amount`) | `metadata.json`'s `sort-orders` block has three fields with the expected direction + null-order pairs: `asc/nulls-first`, `desc/nulls-last`, `desc/nulls-last` |
586
- | **Nested types** | `nested_test` (`tags` list, `profile` struct, `attrs` map) | INSERT two rows using `ARRAY[...]`, `CAST(ROW(...) AS ROW(...))`, and `MAP(ARRAY[...], ARRAY[...])`. SELECT verifies `tags[1]`, `profile.first_name`, and `element_at(attrs, 'tier')` all roundtrip correctly |
587
- | **`grantRead` S3 statements at runtime** | `transforms_test` + a `GranteeRole` trusted by the deployer | Assume the role and call S3 directly (Lake Formation doesn't gate S3 calls when the bucket is registered with `hybridAccessEnabled: true`, so the IAM grants the construct produces are what's being tested). `s3:ListBucket` on the table's own prefix succeeds. `s3:ListBucket` on a foreign table's prefix is denied — the `s3:prefix` condition kicks in. The Glue action grants are validated by the unit tests, not at runtime, because LF gates `glue:*` against tables in LF-registered locations regardless of the principal's IAM policy |
588
- | **`fromIcebergTableAttributes(...)` + `grantRead`** | imported handle on `transforms_test` + a second `ImportedGranteeRole` | Same S3 checks under the import-factory grantee. Verifies the import path produces a symmetric IAM split |
589
-
590
- The construct's `grantRead` / `grantWrite` / `grantReadWrite` issue
591
- **IAM grants only**. In a Lake-Formation-managed deployment like
592
- `ArceusStack`, Athena queries still need separate LF `SELECT` /
593
- `INSERT` / `DELETE` grants on top of the construct's IAM grants. The
594
- surface test deliberately bypasses Athena (assumes the role and
595
- calls Glue + S3 directly) so the construct's grant logic is
596
- verified in isolation — under LF, the IAM grants alone are
597
- necessary but not sufficient for Athena queries.
598
-
599
470
  ## Two footguns the construct prevents
600
471
 
601
472
  ### Footgun #1 — schema under `storageDescriptor.columns`
@@ -655,71 +526,71 @@ the construct does **not** prevent. See the next section.)
655
526
  - **Format version is immutable after CREATE.** The `formatVersion` prop is read once at table creation; changing it later requires a destroy + recreate.
656
527
  - **`merge-on-read` requires v2.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.
657
528
  - **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.
658
- - **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The integration-test script demonstrates the safe pattern: drop partitions that source from `customer_id` while keeping `customer_id` itself in the schema, and drop the `region` column while it is not partitioning anything.
529
+ - **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.
659
530
 
660
- ## Tests
531
+ ## FAQ
661
532
 
662
- ```
663
- npx jest # runs every suite under test/, prints coverage at the end
664
- ```
533
+ ### How do I create an Apache Iceberg table in AWS CDK?
665
534
 
666
- Coverage is gated at 95% statements / 95% branches / 95% lines / 95% functions
667
- on `lib/**/*.ts` via the `coverageThreshold.global` block in `jest.config.js`.
668
- A failing gate fails the suite, so the README does not paste a transcript that
669
- would drift after the next refactor; run the command locally for the live
670
- numbers.
535
+ Install `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a
536
+ database, a column list, and an S3 `location` — see [Use](#use).
537
+ `cdk deploy` creates the Glue Data Catalog table and writes the
538
+ Iceberg `metadata.json` under your S3 prefix. No custom resource or
539
+ Lambda is involved; it is a plain `AWS::Glue::Table` in your
540
+ CloudFormation template.
671
541
 
672
- ## Project layout
542
+ ### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?
673
543
 
674
- ```
675
- arceus/
676
- ├── bin/arceus.ts # CDK app entry point
677
- ├── lib/
678
- │ ├── arceus-stack.ts # Demo stack (buckets, DB, 3 demo tables)
679
- │ ├── iceberg-evolution-stack.ts # Parameterized stack for the evolution test
680
- │ ├── iceberg-dml-stack.ts # Stack for the DML / time-travel / OPTIMIZE / VACUUM test
681
- │ ├── iceberg-surface-stack.ts # Stack for transforms / sort / nested types / grants test
682
- │ └── iceberg/
683
- │ ├── iceberg-table.ts # The L2 construct itself
684
- │ ├── iceberg-type.ts # IcebergType + struct/list/map/decimal/fixed
685
- │ ├── iceberg-partition-transform.ts
686
- │ ├── iceberg-table-properties.ts # Format/version enums + property validation
687
- │ └── index.ts # Re-exports (the npm package's entry point)
688
- ├── test/
689
- │ ├── arceus-stack.test.ts
690
- │ ├── iceberg-evolution-stack.test.ts
691
- │ ├── iceberg-dml-stack.test.ts
692
- │ ├── iceberg-surface-stack.test.ts
693
- │ └── iceberg/
694
- │ ├── iceberg-partition-transform.test.ts
695
- │ ├── iceberg-table-properties.test.ts
696
- │ ├── iceberg-table.test.ts
697
- │ └── iceberg-type.test.ts
698
- ├── e2e-consumer/ # Standalone CDK app that consumes the
699
- │ │ # published npm package — proves a fresh
700
- │ │ # install + import + synth works. Runs
701
- │ │ # on every PR via the `e2e-consumer` job
702
- │ │ # in `.github/workflows/ci.yml`.
703
- │ ├── bin/app.ts
704
- │ ├── lib/consumer-stack.ts # Realistic consumer (one IcebergTable)
705
- │ └── lib/surface-reference.ts # Anchors every exported symbol so a
706
- │ # rename in the published surface breaks CI
707
- ├── scripts/
708
- │ ├── integration-test-evolution.sh # End-to-end evolution harness
709
- │ ├── integration-test-dml.sh # End-to-end DML harness
710
- │ └── integration-test-surface.sh # End-to-end surface harness
711
- ├── docs/
712
- │ └── integ-test-setup.md # AWS-side prerequisites for the integ-test
713
- │ # workflow (OIDC provider, IAM role, repo var)
714
- ├── .github/workflows/
715
- │ ├── ci.yml # Lint + test + build + pack + e2e-consumer
716
- │ ├── publish.yml # Trusted-publish to npm on version bump
717
- │ └── integ-test.yml # Real-AWS evolution test (gated by label /
718
- │ # `/run-integ-test` comment)
719
- ├── cdk.json
720
- ├── package.json # The published package's manifest
721
- ├── tsconfig.json # Wide include (used by ESLint and dev)
722
- ├── tsconfig.build.json # Narrow include (used by `npm publish`)
723
- ├── jest.config.js
724
- └── eslint.config.js
725
- ```
544
+ Change the `columns` array and run `cdk deploy` again. The construct
545
+ passes the new schema to Glue's `UpdateTable`, which writes a new
546
+ `metadata.json` with a new `schema-id`; existing data files stay
547
+ readable because each column's `id` is pinned and never reused. The
548
+ same applies to `partitionSpec`. See [Evolving schema and
549
+ partitions](#evolving-schema-and-partitions).
550
+
551
+ ### Does CloudFormation support Iceberg tables natively?
552
+
553
+ Yes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,
554
+ documented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).
555
+ The catch is that the raw shape corrupts the table on the first
556
+ `Update` if you place schema under `storageDescriptor.columns` or set
557
+ `tableInput` alongside `openTableFormatInput`. This construct only ever
558
+ emits the safe shape and gives you no way to express the unsafe ones.
559
+ See [Two footguns the construct prevents](#two-footguns-the-construct-prevents).
560
+
561
+ ### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?
562
+
563
+ A raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON
564
+ and own both silent-corruption footguns. `cdk-glue-iceberg-table` gives
565
+ you a typed `IcebergType` / `IcebergPartitionTransform` API,
566
+ synth-time validation (partition transforms checked against column
567
+ types, format/version mismatches caught before deploy), pinned field
568
+ IDs for safe evolution, and `grantRead` / `grantWrite` helpers. See
569
+ [How it compares](#how-it-compares).
570
+
571
+ ### How do I create a partitioned Iceberg table in CDK?
572
+
573
+ Pass a `partitionSpec` of `IcebergPartitionTransform` entries —
574
+ `identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,
575
+ or `void`. Each transform is validated against its source column's type
576
+ at synth time. See [Using `IcebergTable`](#using-icebergtable).
577
+
578
+ ### Does it work with Athena and Lake Formation?
579
+
580
+ Yes. The demo app in the [`arceus`](https://github.com/ksco92/arceus)
581
+ repo registers the tables with Lake Formation and queries them from
582
+ Athena, including v2 merge-on-read `INSERT` / `UPDATE` / `DELETE` /
583
+ `MERGE`, time travel, `OPTIMIZE`, and `VACUUM`. The construct's
584
+ `grant*` helpers issue IAM grants; under Lake Formation you still add
585
+ the matching `SELECT` / `INSERT` LF grants. See [Granting
586
+ access](#granting-access).
587
+
588
+ ### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?
589
+
590
+ Yes — set `formatVersion: IcebergFormatVersion.V2` and the
591
+ `write.{delete,update,merge}.mode = merge-on-read` table properties.
592
+ The construct rejects merge-on-read on a v1 table at synth time.
593
+
594
+ ## License
595
+
596
+ [MIT](LICENSE).
@@ -74,5 +74,5 @@ export declare class IcebergPartitionTransform {
74
74
  *
75
75
  * @internal
76
76
  */
77
- validateSourceType(sourceColumnName: string, sourceType: IcebergType): void;
77
+ _validateSourceType(sourceColumnName: string, sourceType: IcebergType): void;
78
78
  }