cdk-glue-iceberg-table 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.jsii +11943 -0
- package/README.md +211 -340
- package/dist/lib/iceberg/iceberg-partition-transform.d.ts +1 -1
- package/dist/lib/iceberg/iceberg-partition-transform.js +31 -33
- package/dist/lib/iceberg/iceberg-table-properties.d.ts +30 -16
- package/dist/lib/iceberg/iceberg-table-properties.js +58 -39
- package/dist/lib/iceberg/iceberg-table-render.js +5 -5
- package/dist/lib/iceberg/iceberg-table.js +12 -22
- package/dist/lib/iceberg/iceberg-type.js +54 -92
- package/dist/lib/iceberg/index.d.ts +1 -1
- package/dist/lib/iceberg/index.js +3 -3
- package/package.json +42 -14
package/README.md
CHANGED
|
@@ -3,43 +3,91 @@
|
|
|
3
3
|
[](https://github.com/ksco92/arceus/actions/workflows/ci.yml)
|
|
4
4
|
[](https://codecov.io/gh/ksco92/arceus)
|
|
5
5
|
[](https://www.npmjs.com/package/cdk-glue-iceberg-table)
|
|
6
|
+
[](https://pypi.org/project/cdk-glue-iceberg-table/)
|
|
7
|
+
[](https://constructs.dev/packages/cdk-glue-iceberg-table)
|
|
6
8
|
[](https://www.npmjs.com/package/cdk-glue-iceberg-table)
|
|
7
9
|
[](https://www.npmjs.com/package/cdk-glue-iceberg-table)
|
|
8
10
|
[](https://github.com/ksco92/arceus/commits/main)
|
|
9
11
|
[](LICENSE)
|
|
10
12
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
13
|
+
**`cdk-glue-iceberg-table` is the AWS CDK L2 construct for creating
|
|
14
|
+
and evolving Apache Iceberg tables in the AWS Glue Data Catalog.** It
|
|
15
|
+
emits the exact `AWS::Glue::Table` / `OpenTableFormatInput` shape that
|
|
16
|
+
survives a CloudFormation `Update`, so a single `cdk deploy` creates a
|
|
17
|
+
table, evolves its schema and partitions, and destroys it, the same
|
|
18
|
+
lifecycle CDK gives any other resource. No custom resource, no Lambda,
|
|
19
|
+
no "`cdk deploy` and then run this SQL by hand" two-step. Table
|
|
20
|
+
changes (new columns, renames, drops, new partition fields) land as
|
|
21
|
+
a reviewed diff in a pull request and apply through `cdk deploy`.
|
|
22
|
+
|
|
23
|
+
Status (June 2026): published on npm (TypeScript/JavaScript) and PyPI
|
|
24
|
+
(Python), pre-1.0, with a public surface guarded by an end-to-end
|
|
25
|
+
consumer test on every PR plus three real-AWS integration suites. The
|
|
26
|
+
multi-language packages are generated from one TypeScript source with
|
|
27
|
+
[jsii](https://aws.github.io/jsii/) and indexed on
|
|
28
|
+
[Construct Hub](https://constructs.dev/packages/cdk-glue-iceberg-table).
|
|
29
|
+
Developed in the
|
|
30
|
+
[`ksco92/arceus`](https://github.com/ksco92/arceus) monorepo, which
|
|
31
|
+
also holds a CDK demo app that dogfoods the construct against a real
|
|
32
|
+
AWS account.
|
|
33
|
+
|
|
34
|
+
### Why this exists
|
|
35
|
+
|
|
36
|
+
AWS documents the CloudFormation shape for Iceberg tables ([AWS Big
|
|
37
|
+
Data blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),
|
|
38
|
+
but raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the
|
|
39
|
+
*first* `Update` silently strips `table_type=ICEBERG`, after which
|
|
40
|
+
every Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The
|
|
41
|
+
motivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
|
|
17
42
|
[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)
|
|
18
|
-
documents the only working
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
`@aws-cdk/aws-glue-alpha`
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
43
|
+
documents the only working shape and the silent-corruption traps you
|
|
44
|
+
hit by getting it slightly wrong. This construct implements that
|
|
45
|
+
shape, refuses to emit the unsafe alternatives, and pins Iceberg field
|
|
46
|
+
IDs so schema evolution never orphans existing data. It is the basis
|
|
47
|
+
of the upstream proposal to land an `IcebergTable` in
|
|
48
|
+
`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988));
|
|
49
|
+
this package tracks that proposal and stays current with it.
|
|
50
|
+
|
|
51
|
+
### How it compares
|
|
52
|
+
|
|
53
|
+
| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |
|
|
54
|
+
| --- | --- | --- | --- | --- |
|
|
55
|
+
| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |
|
|
56
|
+
| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |
|
|
57
|
+
| Spark / Athena SQL DDL at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |
|
|
58
|
+
| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |
|
|
28
59
|
|
|
29
60
|
## Install
|
|
30
61
|
|
|
62
|
+
TypeScript / JavaScript (npm):
|
|
63
|
+
|
|
31
64
|
```bash
|
|
32
65
|
npm install cdk-glue-iceberg-table
|
|
33
66
|
```
|
|
34
67
|
|
|
35
|
-
|
|
68
|
+
Python (PyPI):
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
pip install cdk-glue-iceberg-table
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Peer dependencies (your CDK app must already have these). For
|
|
75
|
+
TypeScript / JavaScript:
|
|
36
76
|
|
|
37
77
|
```bash
|
|
38
78
|
npm install aws-cdk-lib constructs @aws-cdk/aws-glue-alpha
|
|
39
79
|
```
|
|
40
80
|
|
|
81
|
+
For Python:
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
pip install aws-cdk-lib constructs aws-cdk.aws-glue-alpha
|
|
85
|
+
```
|
|
86
|
+
|
|
41
87
|
## Use
|
|
42
88
|
|
|
89
|
+
### TypeScript
|
|
90
|
+
|
|
43
91
|
```ts
|
|
44
92
|
import { Bucket } from 'aws-cdk-lib/aws-s3';
|
|
45
93
|
import { Database } from '@aws-cdk/aws-glue-alpha';
|
|
@@ -69,116 +117,60 @@ new IcebergTable(this, 'OrdersTable', {
|
|
|
69
117
|
});
|
|
70
118
|
```
|
|
71
119
|
|
|
72
|
-
|
|
120
|
+
### Python
|
|
73
121
|
|
|
122
|
+
The same table in Python — the API mirrors TypeScript with
|
|
123
|
+
`snake_case` props and `PascalCase` types:
|
|
124
|
+
|
|
125
|
+
```python
|
|
126
|
+
from aws_cdk.aws_s3 import Bucket
|
|
127
|
+
from aws_cdk.aws_glue_alpha import Database
|
|
128
|
+
from cdk_glue_iceberg_table import (
|
|
129
|
+
IcebergTable,
|
|
130
|
+
IcebergType,
|
|
131
|
+
IcebergPartitionTransform,
|
|
132
|
+
)
|
|
133
|
+
|
|
134
|
+
bucket = Bucket(self, "Warehouse")
|
|
135
|
+
db = Database(self, "Db", database_name="analytics")
|
|
136
|
+
|
|
137
|
+
IcebergTable(self, "OrdersTable",
|
|
138
|
+
database=db,
|
|
139
|
+
table_name="orders",
|
|
140
|
+
location=f"s3://{bucket.bucket_name}/analytics/orders/",
|
|
141
|
+
columns=[
|
|
142
|
+
{"name": "order_id", "type": IcebergType.LONG, "required": True, "id": 1},
|
|
143
|
+
{"name": "customer_id", "type": IcebergType.LONG, "required": True, "id": 2},
|
|
144
|
+
{"name": "placed_at", "type": IcebergType.TIMESTAMPTZ, "required": True, "id": 3},
|
|
145
|
+
],
|
|
146
|
+
partition_spec=[
|
|
147
|
+
{"source_column": "placed_at", "transform": IcebergPartitionTransform.DAY},
|
|
148
|
+
{"source_column": "customer_id", "transform": IcebergPartitionTransform.bucket(16)},
|
|
149
|
+
],
|
|
150
|
+
identifier_field_names=["order_id"],
|
|
151
|
+
)
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Consumer-facing reference sections:
|
|
155
|
+
|
|
156
|
+
- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.
|
|
74
157
|
- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.
|
|
75
158
|
- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.
|
|
76
159
|
- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.
|
|
160
|
+
- [FAQ](#faq) — common "how do I … in CDK / CloudFormation" questions.
|
|
77
161
|
|
|
78
|
-
##
|
|
79
|
-
|
|
80
|
-
This repo is **both** the published package and a CDK demo app:
|
|
81
|
-
|
|
82
|
-
- `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
|
|
83
|
-
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `lib/iceberg-dml-stack.ts`, `lib/iceberg-surface-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus three bash harnesses: one drives schema + partition evolution through real `cdk deploy`s, one exercises the v2 DML surface (UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM), and one covers the remaining surface (every partition transform, multi-field sort order, list / struct / map columns, and `grantRead` at runtime by assuming the grantee role and calling Glue / S3 directly). **Repo-only, not published to npm.**
|
|
84
|
-
- `e2e-consumer/` — a standalone CDK app that depends on the **published** `cdk-glue-iceberg-table` from npm. Proves that a fresh install + import + `cdk synth` works for downstream consumers. Runs on every PR via the `e2e-consumer` job in `.github/workflows/ci.yml`. Its `lib/surface-reference.ts` touches every exported symbol so that a rename in the published surface breaks CI. The pin in `e2e-consumer/package-lock.json` tracks the version most recently published to npm; CLAUDE.md asks for it to be bumped after each release.
|
|
85
|
-
|
|
86
|
-
How the test gates fit together:
|
|
87
|
-
|
|
88
|
-
- **`ci.yml`** runs on every PR — lint, unit tests with the 95% coverage gate, `npm pack`, and the `e2e-consumer` synth against the pinned published npm version.
|
|
89
|
-
- **`integ-test.yml`** is the real-AWS gate. Runs three scripts back-to-back: `scripts/integration-test-evolution.sh` (four `cdk deploy`s exercising schema + partition evolution), `scripts/integration-test-dml.sh` (one deploy, then UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM against a v2 merge-on-read table), and `scripts/integration-test-surface.sh` (one deploy, then every partition transform, multi-field sort order, nested-type roundtrip, and `grantRead` at runtime via assume-role + direct Glue/S3 calls). Gated by `run-integ-test` label or `/run-integ-test` collaborator comment. PRs that touch any file under `lib/`, `bin/arceus.ts`, `cdk.json`, or any of the scripts must show a green run before merging (see CLAUDE.md §"Integration test for construct-touching PRs"). Doc-only PRs are exempt.
|
|
90
|
-
- **`publish.yml`** runs on push to `main` — trusted-publish to npm when `package.json`'s `version` is newer than the registry.
|
|
91
|
-
|
|
92
|
-
The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
|
|
93
|
-
[Demo tables](#demo-tables-deployed-by-arceusstack), and
|
|
94
|
-
[Schema + partition evolution](#schema--partition-evolution-via-cdk-only)
|
|
95
|
-
cover the demo app. Skip them if you only want to consume the construct.
|
|
96
|
-
|
|
97
|
-
## Prerequisites
|
|
98
|
-
|
|
99
|
-
Before running the quickstart you need:
|
|
100
|
-
|
|
101
|
-
1. AWS credentials in the default profile with permissions to manage
|
|
102
|
-
CloudFormation, KMS, S3, Glue, Lake Formation, Athena, and IAM
|
|
103
|
-
policies. `aws sts get-caller-identity` must return successfully.
|
|
104
|
-
2. `CDK_DEFAULT_ACCOUNT` and `CDK_DEFAULT_REGION` set in the
|
|
105
|
-
environment (the AWS CLI sets these automatically for most
|
|
106
|
-
profile setups; `cdk` also populates them from the active profile).
|
|
107
|
-
3. `PRINCIPAL_ARN` set to the **ARN of an existing IAM principal**
|
|
108
|
-
in this account (IAM user, role, or federated identity). The
|
|
109
|
-
stack adds that principal as a Lake Formation admin and grants
|
|
110
|
-
it per-table `SELECT/INSERT/DELETE/ALTER/DESCRIBE` on the demo
|
|
111
|
-
Iceberg tables. Without it the deploy fails when LF can't resolve
|
|
112
|
-
the principal. The same ARN must also be the identity running
|
|
113
|
-
`cdk deploy` and any subsequent Athena queries. Local devs
|
|
114
|
-
typically set this to their IAM user ARN; CI (`integ-test.yml`)
|
|
115
|
-
sets it to the OIDC role ARN.
|
|
116
|
-
- **Only one principal is privileged at a time.** `ArceusStack`
|
|
117
|
-
writes the Lake Formation admin list with REPLACE semantics,
|
|
118
|
-
so the most recent `cdk deploy` wins. A local `cdk deploy` with
|
|
119
|
-
your user ARN revokes the OIDC role's LF grants (and breaks the
|
|
120
|
-
next integ-test run until CI redeploys), and vice versa.
|
|
121
|
-
Coordinate accordingly.
|
|
122
|
-
- **SSO / `aws-vault` users:** set `PRINCIPAL_ARN` to the
|
|
123
|
-
canonical role ARN (`arn:aws:iam::<acct>:role/<RoleName>`), not
|
|
124
|
-
the per-session `arn:aws:sts::<acct>:assumed-role/...` you'd
|
|
125
|
-
get from `aws sts get-caller-identity`. LF grants on the
|
|
126
|
-
session-suffixed ARN go stale at the next SSO refresh.
|
|
127
|
-
4. The Lake Formation service-linked role
|
|
128
|
-
`AWSServiceRoleForLakeFormationDataAccess` must exist in the
|
|
129
|
-
account. Create it once with
|
|
130
|
-
`aws iam create-service-linked-role --aws-service-name lakeformation.amazonaws.com`
|
|
131
|
-
if you haven't already.
|
|
132
|
-
5. `cdk bootstrap aws://<account>/<region>` if the account hasn't
|
|
133
|
-
been bootstrapped for CDK.
|
|
134
|
-
|
|
135
|
-
## What's in the repo
|
|
136
|
-
|
|
137
|
-
Repo source paths (npm consumers import from the package root and get the same exports under `dist/lib/iceberg/`):
|
|
138
|
-
|
|
139
|
-
- **`lib/iceberg/iceberg-table.ts`**: the `IcebergTable` L2 construct.
|
|
140
|
-
- **`lib/iceberg/iceberg-type.ts`**: `IcebergType` with primitives + `list` / `map` / `struct` factories. Renders to the JSON shape Glue's `IcebergStructField.type` expects.
|
|
141
|
-
- **`lib/iceberg/iceberg-partition-transform.ts`**: `IcebergPartitionTransform` (identity / bucket(N) / truncate(W) / year / month / day / hour / void). Each transform validates against the source column type at synth time.
|
|
142
|
-
- **`lib/iceberg/iceberg-table-properties.ts`**: `IcebergDataFormat` (parquet/orc/avro, default parquet), `IcebergFormatVersion` (v1/v2, required — set explicitly per table), and a validator that catches misconfigured `tableProperties` before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
|
|
143
|
-
- **`lib/arceus-stack.ts`**: the demo stack: KMS-encrypted data lake bucket, Athena results bucket, Glue database, three demo Iceberg tables (`orders`, `events`, `customers`). Repo-only, not in the npm tarball.
|
|
144
|
-
- **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh`**: a parameterized stack and a bash harness that drives four real `cdk deploy`s to prove schema/partition evolution works end-to-end. Repo-only, not in the npm tarball.
|
|
145
|
-
- **`lib/iceberg-dml-stack.ts`** + **`scripts/integration-test-dml.sh`**: a v2 merge-on-read table and a bash harness that exercises UPDATE, DELETE, MERGE INTO (upsert), time-travel SELECT, OPTIMIZE compaction, and VACUUM snapshot expiration. Single deploy, ~3 minutes. Repo-only, not in the npm tarball.
|
|
146
|
-
- **`lib/iceberg-surface-stack.ts`** + **`scripts/integration-test-surface.sh`**: three tables (one per concern) and a harness that verifies every partition transform renders the right `metadata.json`, a three-field sort order with mixed direction + null ordering, list/struct/map columns roundtrip through Athena, and `grantRead`'s four-statement IAM split actually authorizes (and denies cross-prefix) at runtime under an assumed grantee role. The same checks run against `IcebergTable.fromIcebergTableAttributes(...)` to prove the import factory's grant path is symmetric with the native one. Single deploy, ~4 minutes. Repo-only, not in the npm tarball.
|
|
147
|
-
|
|
148
|
-
## Quickstart
|
|
162
|
+
## Exported surface
|
|
149
163
|
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
# integration script's INSERT/SELECT calls fail with `Principal does
|
|
156
|
-
# not have any privilege on specified resource`.
|
|
157
|
-
#
|
|
158
|
-
# For a direct IAM-user session, the line below returns the right ARN
|
|
159
|
-
# (`arn:aws:iam::<acct>:user/<name>`).
|
|
160
|
-
#
|
|
161
|
-
# For SSO / aws-vault / any assumed-role session this returns
|
|
162
|
-
# `arn:aws:sts::<acct>:assumed-role/<RoleName>/<SessionName>`, which
|
|
163
|
-
# Lake Formation accepts but stales on the next session refresh
|
|
164
|
-
# because the SessionName changes. Set PRINCIPAL_ARN to the canonical
|
|
165
|
-
# `arn:aws:iam::<acct>:role/<RoleName>` instead — for example:
|
|
166
|
-
# export PRINCIPAL_ARN="arn:aws:iam::123456789012:role/MyDevRole"
|
|
167
|
-
export PRINCIPAL_ARN="$(aws sts get-caller-identity --query Arn --output text)"
|
|
168
|
-
|
|
169
|
-
npm install
|
|
170
|
-
npx jest # runs the suite; coverage floor in jest.config.js
|
|
171
|
-
npx cdk deploy ArceusStack --require-approval=never
|
|
172
|
-
./scripts/integration-test-evolution.sh # add + rename + drop, via cdk only
|
|
173
|
-
```
|
|
164
|
+
The package entry point re-exports everything you import from
|
|
165
|
+
`cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled
|
|
166
|
+
surface lives under `dist/lib/iceberg/` after `npm install`; in Python
|
|
167
|
+
the same surface is imported from the `cdk_glue_iceberg_table` module
|
|
168
|
+
after `pip install`:
|
|
174
169
|
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
`
|
|
179
|
-
grants test target). Deploy only `ArceusStack` for the quickstart;
|
|
180
|
-
the three test stacks are created on demand by their respective
|
|
181
|
-
scripts under `scripts/`.
|
|
170
|
+
- **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite` and the `fromIcebergTableAttributes(...)` import factory.
|
|
171
|
+
- **`IcebergType`** — primitive types plus `list` / `map` / `struct` / `decimal` / `fixed` factories. Renders to the JSON shape Glue's `IcebergStructField.type` expects.
|
|
172
|
+
- **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.
|
|
173
|
+
- **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2, required — set explicitly per table), **`IcebergSortDirection`**, **`IcebergNullOrder`**, plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
|
|
182
174
|
|
|
183
175
|
## Using `IcebergTable`
|
|
184
176
|
|
|
@@ -225,10 +217,7 @@ new IcebergTable(this, 'Users', {
|
|
|
225
217
|
```
|
|
226
218
|
|
|
227
219
|
A table that exercises most of the surface (partitions, sort order,
|
|
228
|
-
nested types, identifier fields, table properties, removal policy)
|
|
229
|
-
This is the exact shape `ArceusStack` uses for the `orders` demo
|
|
230
|
-
table, so the column list / partition spec / properties round-trip
|
|
231
|
-
straight to the live metadata.json below.
|
|
220
|
+
nested types, identifier fields, table properties, removal policy):
|
|
232
221
|
|
|
233
222
|
```typescript
|
|
234
223
|
import {
|
|
@@ -358,47 +347,8 @@ new IcebergTable(this, 'OrdersTable', {
|
|
|
358
347
|
});
|
|
359
348
|
```
|
|
360
349
|
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
```typescript
|
|
364
|
-
table.grantRead(role); // Glue read + S3 read on the table's prefix
|
|
365
|
-
table.grantWrite(role); // Glue write + S3 write
|
|
366
|
-
table.grantReadWrite(role);
|
|
367
|
-
```
|
|
368
|
-
|
|
369
|
-
### Importing an existing table
|
|
370
|
-
|
|
371
|
-
```typescript
|
|
372
|
-
const existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {
|
|
373
|
-
database: db,
|
|
374
|
-
tableName: 'orders',
|
|
375
|
-
location: 's3://my-bucket/analytics/orders/',
|
|
376
|
-
});
|
|
377
|
-
existing.grantRead(role);
|
|
378
|
-
```
|
|
379
|
-
|
|
380
|
-
## Demo tables (deployed by `ArceusStack`)
|
|
381
|
-
|
|
382
|
-
| Table | Format | Columns | Partitions | Sort | Notable properties |
|
|
383
|
-
| --- | --- | --- | --- | --- | --- |
|
|
384
|
-
| `orders` | parquet, v2 | `order_id(1)`, `customer_id(2)`, `order_amount(3)`, `currency(4)`, `placed_at(5)`, `tags(6)` (list), `shipping_address(7)` (struct), `metadata(8)` (map) | `day(placed_at)`, `bucket(16)(customer_id)` | `placed_at ASC NULLS LAST`, `order_id ASC` | `write.{delete,update,merge}.mode = merge-on-read`, `zstd`, `history.expire.min-snapshots-to-keep = 5`, identifier-field-ids = `[order_id]`, nested `list`/`struct`/`map` columns |
|
|
385
|
-
| `events` | parquet, v2 | `event_id(1)`, `event_name(2)`, `session_id(3)`, `occurred_at(4)`, `attributes(5)` (map) | `hour(occurred_at)` | (none) | high-cardinality hourly partitioning |
|
|
386
|
-
| `customers` | parquet, v2 | `customer_id(1)`, `email(2)`, `signed_up_at(4)`, `loyalty_tier(5)` (id 3 retired) | (none) | (none) | identifier-field-ids = `[customer_id]` — the stack's `customers` block carries inline comments narrating the schema-evolution journey that landed here (drop `full_name`, add `loyalty_tier`); the live evolution loop runs against the separate `IcebergEvolutionStack` |
|
|
387
|
-
|
|
388
|
-
After `cdk deploy ArceusStack`, the three tables are queryable from Athena (workgroup `ReadOnly`).
|
|
389
|
-
|
|
390
|
-
### Validating the demo
|
|
391
|
-
|
|
392
|
-
`SHOW TBLPROPERTIES sample_database.orders` returns:
|
|
393
|
-
|
|
394
|
-
```
|
|
395
|
-
format parquet
|
|
396
|
-
write_compression zstd
|
|
397
|
-
write_target_data_file_size_bytes 134217728
|
|
398
|
-
vacuum_min_snapshots_to_keep 5
|
|
399
|
-
```
|
|
400
|
-
|
|
401
|
-
The Iceberg `metadata.json` for `orders` contains every feature you set:
|
|
350
|
+
The resulting Iceberg `metadata.json` for this table contains every
|
|
351
|
+
feature you set:
|
|
402
352
|
|
|
403
353
|
```json
|
|
404
354
|
{
|
|
@@ -457,6 +407,39 @@ The Iceberg `metadata.json` for `orders` contains every feature you set:
|
|
|
457
407
|
}
|
|
458
408
|
```
|
|
459
409
|
|
|
410
|
+
### Granting access
|
|
411
|
+
|
|
412
|
+
```typescript
|
|
413
|
+
table.grantRead(role); // Glue read + S3 read on the table's prefix
|
|
414
|
+
table.grantWrite(role); // Glue write + S3 write
|
|
415
|
+
table.grantReadWrite(role);
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
The `grant*` helpers issue **IAM grants only**. Under Lake Formation
|
|
419
|
+
you still add the matching `SELECT` / `INSERT` / `DELETE` LF grants on
|
|
420
|
+
top of the construct's IAM grants for Athena queries to succeed.
|
|
421
|
+
|
|
422
|
+
### Importing an existing table
|
|
423
|
+
|
|
424
|
+
```typescript
|
|
425
|
+
const existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {
|
|
426
|
+
database: db,
|
|
427
|
+
tableName: 'orders',
|
|
428
|
+
location: 's3://my-bucket/analytics/orders/',
|
|
429
|
+
});
|
|
430
|
+
existing.grantRead(role);
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
### Evolving schema and partitions
|
|
434
|
+
|
|
435
|
+
Change the `columns` array (or `partitionSpec`) and run `cdk deploy`
|
|
436
|
+
again. The construct passes the new schema to Glue's `UpdateTable`,
|
|
437
|
+
which writes a new `metadata.json` with a new `schema-id`; existing
|
|
438
|
+
data files stay readable because each column's `id` is pinned and
|
|
439
|
+
never reused. Adds, renames (same `id`, new `name`), and drops all
|
|
440
|
+
flow through `cdk deploy` alone — no out-of-band SQL DDL. The same
|
|
441
|
+
applies to `partitionSpec`.
|
|
442
|
+
|
|
460
443
|
### Inserting and querying
|
|
461
444
|
|
|
462
445
|
```sql
|
|
@@ -484,118 +467,6 @@ SELECT customer_id, SUM(order_amount) AS total
|
|
|
484
467
|
ORDER BY 2 DESC;
|
|
485
468
|
```
|
|
486
469
|
|
|
487
|
-
## Schema + partition evolution via cdk-only
|
|
488
|
-
|
|
489
|
-
`scripts/integration-test-evolution.sh` drives the
|
|
490
|
-
`IcebergEvolutionStack` through four `cdk deploy`s and verifies the
|
|
491
|
-
underlying Iceberg `metadata.json` after each:
|
|
492
|
-
|
|
493
|
-
| Step | Change | Columns | Partitions |
|
|
494
|
-
| ---: | --- | --- | --- |
|
|
495
|
-
| 1 | Initial deploy | `customer_id(1)`, `email(2)`, `signed_up_at(3)` | `day(signed_up_at)` |
|
|
496
|
-
| 2 | **ADD** column `region(4)` | + `region(4)` | unchanged |
|
|
497
|
-
| 3 | **RENAME** `email` → `contact_email` (id 2 preserved), **ADD** partition `bucket(8)(customer_id)` | rename | + `bucket(8)(customer_id)` |
|
|
498
|
-
| 4 | **DROP** column `region` (id 4 stays retired), **DROP** partition `bucket(8)(customer_id)` | − `region` | − `customer_id_bucket` |
|
|
499
|
-
|
|
500
|
-
Last script run output (abridged: `cdk deploy` chatter and the
|
|
501
|
-
per-Athena-query state polling lines are omitted; the assertion
|
|
502
|
-
output is verbatim):
|
|
503
|
-
|
|
504
|
-
```
|
|
505
|
-
=== STEP 1 — cdk deploy ===
|
|
506
|
-
✨ Total time: 10.57s
|
|
507
|
-
|
|
508
|
-
=== VERIFY step 1 ===
|
|
509
|
-
columns ✓ (1:customer_id,2:email,3:signed_up_at)
|
|
510
|
-
partitions ✓ (signed_up_at_day)
|
|
511
|
-
|
|
512
|
-
=== INSERT seed rows ===
|
|
513
|
-
3 rows inserted
|
|
514
|
-
|
|
515
|
-
=== STEP 2 — cdk deploy ===
|
|
516
|
-
=== VERIFY step 2 (ADD column) ===
|
|
517
|
-
columns ✓ (1:customer_id,2:email,3:signed_up_at,4:region)
|
|
518
|
-
partitions ✓ (signed_up_at_day)
|
|
519
|
-
|
|
520
|
-
=== VERIFY old rows are preserved with region=NULL ===
|
|
521
|
-
pre-existing rows readable ✓
|
|
522
|
-
inserted 1 row carrying region='us-east-1'
|
|
523
|
-
|
|
524
|
-
=== STEP 3 — cdk deploy ===
|
|
525
|
-
=== VERIFY step 3 (RENAME column + ADD partition) ===
|
|
526
|
-
columns ✓ (1:customer_id,2:contact_email,3:signed_up_at,4:region)
|
|
527
|
-
partitions ✓ (signed_up_at_day,customer_id_bucket)
|
|
528
|
-
rename preserved data ✓
|
|
529
|
-
|
|
530
|
-
=== STEP 4 — cdk deploy ===
|
|
531
|
-
=== VERIFY step 4 (DROP column + DROP partition) ===
|
|
532
|
-
columns ✓ (1:customer_id,2:contact_email,3:signed_up_at)
|
|
533
|
-
partitions ✓ (signed_up_at_day)
|
|
534
|
-
last-column-id stays at 4 — id reuse protection ✓
|
|
535
|
-
all 4 pre-existing rows queryable after drop ✓
|
|
536
|
-
|
|
537
|
-
=== TEARDOWN ===
|
|
538
|
-
IcebergEvolutionStack | 4 | DELETE_COMPLETE | AWS::CloudFormation::Stack
|
|
539
|
-
✅ IcebergEvolutionStack: destroyed
|
|
540
|
-
|
|
541
|
-
=== ALL EVOLUTION STEPS PASSED ===
|
|
542
|
-
```
|
|
543
|
-
|
|
544
|
-
The construct passes each new column list + partition spec to Glue
|
|
545
|
-
`UpdateTable` via `OpenTableFormatInput`. Glue computes the Iceberg
|
|
546
|
-
metadata delta (new `schema-id`, new `spec-id`) and writes a new
|
|
547
|
-
`metadata.json`. Old data files stay readable because the field IDs
|
|
548
|
-
the construct pins (`id: N` on each `IcebergColumn`) never change
|
|
549
|
-
across deploys.
|
|
550
|
-
|
|
551
|
-
## DML, time travel, OPTIMIZE, and VACUUM
|
|
552
|
-
|
|
553
|
-
`scripts/integration-test-dml.sh` covers the v2 surface that the
|
|
554
|
-
evolution test doesn't. The harness deploys `IcebergDmlStack` once,
|
|
555
|
-
then runs a sequence of Athena statements against a v2 merge-on-read
|
|
556
|
-
table with `identifierFieldNames: ['account_id']`:
|
|
557
|
-
|
|
558
|
-
| Step | Statement | Verify |
|
|
559
|
-
| ---: | --- | --- |
|
|
560
|
-
| 1 | `INSERT` 5 seed rows | row count == 5 |
|
|
561
|
-
| 2 | `UPDATE balance WHERE account_id = 2` | balance == 250, row count == 5 |
|
|
562
|
-
| 3 | `DELETE WHERE account_id = 4` | row count == 4, account_id=4 gone |
|
|
563
|
-
| 4 | capture pre-MERGE snapshot id from `dml_test$snapshots` | — |
|
|
564
|
-
| 5 | `MERGE INTO ... USING ... ON account_id` (update id=3, insert id=6 + id=7) | row count == 6, id=3 balance updated, ids 6 and 7 present |
|
|
565
|
-
| 6 | `SELECT ... FOR VERSION AS OF <pre-MERGE snapshot>` | time-travel still sees 4 rows, no id=6 |
|
|
566
|
-
| 7 | `OPTIMIZE ... REWRITE DATA USING BIN_PACK` | succeeds, row count unchanged |
|
|
567
|
-
| 8 | `VACUUM ...` (after a 65-s sleep to clear `max-snapshot-age-ms`) | succeeds, row count unchanged |
|
|
568
|
-
| 9 | final `SELECT` | 6 rows with the expected balances |
|
|
569
|
-
|
|
570
|
-
The DML table is configured with `history.expire.max-snapshot-age-ms`
|
|
571
|
-
of 60 seconds so VACUUM has snapshots to expire on a fresh table.
|
|
572
|
-
All three scripts run sequentially in the same `integ-test.yml`
|
|
573
|
-
job; evolution takes ~5 minutes, DML ~3, and surface ~4.
|
|
574
|
-
|
|
575
|
-
## Partition transforms, sort order, nested types, and grants
|
|
576
|
-
|
|
577
|
-
`scripts/integration-test-surface.sh` covers the slice of the
|
|
578
|
-
construct surface the evolution and DML stacks don't reach.
|
|
579
|
-
`IcebergSurfaceStack` defines three small tables that each isolate
|
|
580
|
-
one concern:
|
|
581
|
-
|
|
582
|
-
| Concern | Table | What the script checks |
|
|
583
|
-
| --- | --- | --- |
|
|
584
|
-
| **Every partition transform** | `transforms_test` (separate `year_source` / `month_source` / `day_source` / `hour_source` timestamps to avoid Iceberg's redundant-temporal-transform rejection, plus `user_id` / `email` / `value`) | `metadata.json` contains all seven transforms (`year`, `month`, `day`, `hour`, `bucket[8]`, `truncate[4]`, `identity`); INSERT one row and verify the resulting S3 prefix contains the expected multi-transform layout (`year_source_year=...`, `month_source_month=...`, etc.) |
|
|
585
|
-
| **Sort order** | `sorted_test` (`tenant`, `created_at`, `amount`) | `metadata.json`'s `sort-orders` block has three fields with the expected direction + null-order pairs: `asc/nulls-first`, `desc/nulls-last`, `desc/nulls-last` |
|
|
586
|
-
| **Nested types** | `nested_test` (`tags` list, `profile` struct, `attrs` map) | INSERT two rows using `ARRAY[...]`, `CAST(ROW(...) AS ROW(...))`, and `MAP(ARRAY[...], ARRAY[...])`. SELECT verifies `tags[1]`, `profile.first_name`, and `element_at(attrs, 'tier')` all roundtrip correctly |
|
|
587
|
-
| **`grantRead` S3 statements at runtime** | `transforms_test` + a `GranteeRole` trusted by the deployer | Assume the role and call S3 directly (Lake Formation doesn't gate S3 calls when the bucket is registered with `hybridAccessEnabled: true`, so the IAM grants the construct produces are what's being tested). `s3:ListBucket` on the table's own prefix succeeds. `s3:ListBucket` on a foreign table's prefix is denied — the `s3:prefix` condition kicks in. The Glue action grants are validated by the unit tests, not at runtime, because LF gates `glue:*` against tables in LF-registered locations regardless of the principal's IAM policy |
|
|
588
|
-
| **`fromIcebergTableAttributes(...)` + `grantRead`** | imported handle on `transforms_test` + a second `ImportedGranteeRole` | Same S3 checks under the import-factory grantee. Verifies the import path produces a symmetric IAM split |
|
|
589
|
-
|
|
590
|
-
The construct's `grantRead` / `grantWrite` / `grantReadWrite` issue
|
|
591
|
-
**IAM grants only**. In a Lake-Formation-managed deployment like
|
|
592
|
-
`ArceusStack`, Athena queries still need separate LF `SELECT` /
|
|
593
|
-
`INSERT` / `DELETE` grants on top of the construct's IAM grants. The
|
|
594
|
-
surface test deliberately bypasses Athena (assumes the role and
|
|
595
|
-
calls Glue + S3 directly) so the construct's grant logic is
|
|
596
|
-
verified in isolation — under LF, the IAM grants alone are
|
|
597
|
-
necessary but not sufficient for Athena queries.
|
|
598
|
-
|
|
599
470
|
## Two footguns the construct prevents
|
|
600
471
|
|
|
601
472
|
### Footgun #1 — schema under `storageDescriptor.columns`
|
|
@@ -655,71 +526,71 @@ the construct does **not** prevent. See the next section.)
|
|
|
655
526
|
- **Format version is immutable after CREATE.** The `formatVersion` prop is read once at table creation; changing it later requires a destroy + recreate.
|
|
656
527
|
- **`merge-on-read` requires v2.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.
|
|
657
528
|
- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.
|
|
658
|
-
- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The
|
|
529
|
+
- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.
|
|
659
530
|
|
|
660
|
-
##
|
|
531
|
+
## FAQ
|
|
661
532
|
|
|
662
|
-
|
|
663
|
-
npx jest # runs every suite under test/, prints coverage at the end
|
|
664
|
-
```
|
|
533
|
+
### How do I create an Apache Iceberg table in AWS CDK?
|
|
665
534
|
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
535
|
+
Install `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a
|
|
536
|
+
database, a column list, and an S3 `location` — see [Use](#use).
|
|
537
|
+
`cdk deploy` creates the Glue Data Catalog table and writes the
|
|
538
|
+
Iceberg `metadata.json` under your S3 prefix. No custom resource or
|
|
539
|
+
Lambda is involved; it is a plain `AWS::Glue::Table` in your
|
|
540
|
+
CloudFormation template.
|
|
671
541
|
|
|
672
|
-
|
|
542
|
+
### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?
|
|
673
543
|
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
|
|
678
|
-
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
|
|
690
|
-
|
|
691
|
-
|
|
692
|
-
|
|
693
|
-
|
|
694
|
-
|
|
695
|
-
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
|
|
710
|
-
|
|
711
|
-
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
|
|
544
|
+
Change the `columns` array and run `cdk deploy` again. The construct
|
|
545
|
+
passes the new schema to Glue's `UpdateTable`, which writes a new
|
|
546
|
+
`metadata.json` with a new `schema-id`; existing data files stay
|
|
547
|
+
readable because each column's `id` is pinned and never reused. The
|
|
548
|
+
same applies to `partitionSpec`. See [Evolving schema and
|
|
549
|
+
partitions](#evolving-schema-and-partitions).
|
|
550
|
+
|
|
551
|
+
### Does CloudFormation support Iceberg tables natively?
|
|
552
|
+
|
|
553
|
+
Yes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,
|
|
554
|
+
documented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).
|
|
555
|
+
The catch is that the raw shape corrupts the table on the first
|
|
556
|
+
`Update` if you place schema under `storageDescriptor.columns` or set
|
|
557
|
+
`tableInput` alongside `openTableFormatInput`. This construct only ever
|
|
558
|
+
emits the safe shape and gives you no way to express the unsafe ones.
|
|
559
|
+
See [Two footguns the construct prevents](#two-footguns-the-construct-prevents).
|
|
560
|
+
|
|
561
|
+
### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?
|
|
562
|
+
|
|
563
|
+
A raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON
|
|
564
|
+
and own both silent-corruption footguns. `cdk-glue-iceberg-table` gives
|
|
565
|
+
you a typed `IcebergType` / `IcebergPartitionTransform` API,
|
|
566
|
+
synth-time validation (partition transforms checked against column
|
|
567
|
+
types, format/version mismatches caught before deploy), pinned field
|
|
568
|
+
IDs for safe evolution, and `grantRead` / `grantWrite` helpers. See
|
|
569
|
+
[How it compares](#how-it-compares).
|
|
570
|
+
|
|
571
|
+
### How do I create a partitioned Iceberg table in CDK?
|
|
572
|
+
|
|
573
|
+
Pass a `partitionSpec` of `IcebergPartitionTransform` entries —
|
|
574
|
+
`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,
|
|
575
|
+
or `void`. Each transform is validated against its source column's type
|
|
576
|
+
at synth time. See [Using `IcebergTable`](#using-icebergtable).
|
|
577
|
+
|
|
578
|
+
### Does it work with Athena and Lake Formation?
|
|
579
|
+
|
|
580
|
+
Yes. The demo app in the [`arceus`](https://github.com/ksco92/arceus)
|
|
581
|
+
repo registers the tables with Lake Formation and queries them from
|
|
582
|
+
Athena, including v2 merge-on-read `INSERT` / `UPDATE` / `DELETE` /
|
|
583
|
+
`MERGE`, time travel, `OPTIMIZE`, and `VACUUM`. The construct's
|
|
584
|
+
`grant*` helpers issue IAM grants; under Lake Formation you still add
|
|
585
|
+
the matching `SELECT` / `INSERT` LF grants. See [Granting
|
|
586
|
+
access](#granting-access).
|
|
587
|
+
|
|
588
|
+
### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?
|
|
589
|
+
|
|
590
|
+
Yes — set `formatVersion: IcebergFormatVersion.V2` and the
|
|
591
|
+
`write.{delete,update,merge}.mode = merge-on-read` table properties.
|
|
592
|
+
The construct rejects merge-on-read on a v1 table at synth time.
|
|
593
|
+
|
|
594
|
+
## License
|
|
595
|
+
|
|
596
|
+
[MIT](LICENSE).
|