cdk-glue-iceberg-table 0.2.1 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +104 -16
- package/package.json +15 -8
package/README.md
CHANGED
|
@@ -8,23 +8,45 @@
|
|
|
8
8
|
[](https://github.com/ksco92/arceus/commits/main)
|
|
9
9
|
[](LICENSE)
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
11
|
+
**`cdk-glue-iceberg-table` is the AWS CDK L2 construct for creating
|
|
12
|
+
and evolving Apache Iceberg tables in the AWS Glue Data Catalog.** It
|
|
13
|
+
emits the exact `AWS::Glue::Table` / `OpenTableFormatInput` shape that
|
|
14
|
+
survives a CloudFormation `Update`, so a single `cdk deploy` creates a
|
|
15
|
+
table, evolves its schema and partitions, and destroys it, the same
|
|
16
|
+
lifecycle CDK gives any other resource. No custom resource, no Lambda,
|
|
17
|
+
no "`cdk deploy` and then run this SQL by hand" two-step. Table
|
|
18
|
+
changes (new columns, renames, drops, new partition fields) land as
|
|
19
|
+
a reviewed diff in a pull request and apply through `cdk deploy`.
|
|
20
|
+
|
|
21
|
+
Status (June 2026): published on npm, pre-1.0, with a public surface
|
|
22
|
+
guarded by an end-to-end consumer test on every PR plus three
|
|
23
|
+
real-AWS integration suites.
|
|
24
|
+
|
|
25
|
+
### Why this exists
|
|
26
|
+
|
|
27
|
+
AWS documents the CloudFormation shape for Iceberg tables ([AWS Big
|
|
28
|
+
Data blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),
|
|
29
|
+
but raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the
|
|
30
|
+
*first* `Update` silently strips `table_type=ICEBERG`, after which
|
|
31
|
+
every Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The
|
|
32
|
+
motivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
|
|
17
33
|
[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)
|
|
18
|
-
documents the only working
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
`@aws-cdk/aws-glue-alpha`
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
34
|
+
documents the only working shape and the silent-corruption traps you
|
|
35
|
+
hit by getting it slightly wrong. This construct implements that
|
|
36
|
+
shape, refuses to emit the unsafe alternatives, and pins Iceberg field
|
|
37
|
+
IDs so schema evolution never orphans existing data. It is the basis
|
|
38
|
+
of the upstream proposal to land an `IcebergTable` in
|
|
39
|
+
`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988));
|
|
40
|
+
this package tracks that proposal and stays current with it.
|
|
41
|
+
|
|
42
|
+
### How it compares
|
|
43
|
+
|
|
44
|
+
| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |
|
|
45
|
+
| --- | --- | --- | --- | --- |
|
|
46
|
+
| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |
|
|
47
|
+
| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |
|
|
48
|
+
| Spark / Athena SQL DDL at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |
|
|
49
|
+
| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |
|
|
28
50
|
|
|
29
51
|
## Install
|
|
30
52
|
|
|
@@ -71,9 +93,11 @@ new IcebergTable(this, 'OrdersTable', {
|
|
|
71
93
|
|
|
72
94
|
Consumer-facing reference sections below:
|
|
73
95
|
|
|
96
|
+
- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.
|
|
74
97
|
- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.
|
|
75
98
|
- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.
|
|
76
99
|
- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.
|
|
100
|
+
- [FAQ](#faq) — common "how do I … in CDK / CloudFormation" questions.
|
|
77
101
|
|
|
78
102
|
## Repo layout
|
|
79
103
|
|
|
@@ -657,6 +681,70 @@ the construct does **not** prevent. See the next section.)
|
|
|
657
681
|
- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.
|
|
658
682
|
- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The integration-test script demonstrates the safe pattern: drop partitions that source from `customer_id` while keeping `customer_id` itself in the schema, and drop the `region` column while it is not partitioning anything.
|
|
659
683
|
|
|
684
|
+
## FAQ
|
|
685
|
+
|
|
686
|
+
### How do I create an Apache Iceberg table in AWS CDK?
|
|
687
|
+
|
|
688
|
+
Install `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a
|
|
689
|
+
database, a column list, and an S3 `location` — see [Use](#use).
|
|
690
|
+
`cdk deploy` creates the Glue Data Catalog table and writes the
|
|
691
|
+
Iceberg `metadata.json` under your S3 prefix. No custom resource or
|
|
692
|
+
Lambda is involved; it is a plain `AWS::Glue::Table` in your
|
|
693
|
+
CloudFormation template.
|
|
694
|
+
|
|
695
|
+
### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?
|
|
696
|
+
|
|
697
|
+
Change the `columns` array and run `cdk deploy` again. The construct
|
|
698
|
+
passes the new schema to Glue's `UpdateTable`, which writes a new
|
|
699
|
+
`metadata.json` with a new `schema-id`; existing data files stay
|
|
700
|
+
readable because each column's `id` is pinned and never reused. The
|
|
701
|
+
same applies to `partitionSpec`. See [Schema + partition
|
|
702
|
+
evolution](#schema--partition-evolution-via-cdk-only) for a four-step
|
|
703
|
+
run that adds, renames, and drops columns and partitions through
|
|
704
|
+
`cdk deploy` alone.
|
|
705
|
+
|
|
706
|
+
### Does CloudFormation support Iceberg tables natively?
|
|
707
|
+
|
|
708
|
+
Yes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,
|
|
709
|
+
documented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).
|
|
710
|
+
The catch is that the raw shape corrupts the table on the first
|
|
711
|
+
`Update` if you place schema under `storageDescriptor.columns` or set
|
|
712
|
+
`tableInput` alongside `openTableFormatInput`. This construct only ever
|
|
713
|
+
emits the safe shape and gives you no way to express the unsafe ones.
|
|
714
|
+
See [Two footguns the construct prevents](#two-footguns-the-construct-prevents).
|
|
715
|
+
|
|
716
|
+
### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?
|
|
717
|
+
|
|
718
|
+
A raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON
|
|
719
|
+
and own both silent-corruption footguns. `cdk-glue-iceberg-table` gives
|
|
720
|
+
you a typed `IcebergType` / `IcebergPartitionTransform` API,
|
|
721
|
+
synth-time validation (partition transforms checked against column
|
|
722
|
+
types, format/version mismatches caught before deploy), pinned field
|
|
723
|
+
IDs for safe evolution, and `grantRead` / `grantWrite` helpers. See
|
|
724
|
+
[How it compares](#how-it-compares).
|
|
725
|
+
|
|
726
|
+
### How do I create a partitioned Iceberg table in CDK?
|
|
727
|
+
|
|
728
|
+
Pass a `partitionSpec` of `IcebergPartitionTransform` entries —
|
|
729
|
+
`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,
|
|
730
|
+
or `void`. Each transform is validated against its source column's type
|
|
731
|
+
at synth time. See [Using `IcebergTable`](#using-icebergtable).
|
|
732
|
+
|
|
733
|
+
### Does it work with Athena and Lake Formation?
|
|
734
|
+
|
|
735
|
+
Yes. The demo `ArceusStack` registers the tables with Lake Formation
|
|
736
|
+
and queries them from Athena, including v2 merge-on-read `INSERT` /
|
|
737
|
+
`UPDATE` / `DELETE` / `MERGE`, time travel, `OPTIMIZE`, and `VACUUM`.
|
|
738
|
+
The construct's `grant*` helpers issue IAM grants; under Lake Formation
|
|
739
|
+
you still add the matching `SELECT` / `INSERT` LF grants. See
|
|
740
|
+
[Inserting and querying](#inserting-and-querying).
|
|
741
|
+
|
|
742
|
+
### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?
|
|
743
|
+
|
|
744
|
+
Yes — set `formatVersion: IcebergFormatVersion.V2` and the
|
|
745
|
+
`write.{delete,update,merge}.mode = merge-on-read` table properties.
|
|
746
|
+
The construct rejects merge-on-read on a v1 table at synth time.
|
|
747
|
+
|
|
660
748
|
## Tests
|
|
661
749
|
|
|
662
750
|
```
|
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cdk-glue-iceberg-table",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.1",
|
|
4
4
|
"license": "MIT",
|
|
5
|
-
"description": "CDK L2 construct
|
|
5
|
+
"description": "AWS CDK L2 construct to create and evolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and partition changes that survive CloudFormation updates, via cdk deploy.",
|
|
6
6
|
"author": "Rodrigo Carvajal",
|
|
7
7
|
"repository": {
|
|
8
8
|
"type": "git",
|
|
@@ -14,13 +14,20 @@
|
|
|
14
14
|
},
|
|
15
15
|
"keywords": [
|
|
16
16
|
"aws",
|
|
17
|
-
"cdk",
|
|
18
17
|
"aws-cdk",
|
|
18
|
+
"awscdk",
|
|
19
|
+
"cdk",
|
|
20
|
+
"cloudformation",
|
|
19
21
|
"glue",
|
|
22
|
+
"glue-data-catalog",
|
|
20
23
|
"iceberg",
|
|
21
24
|
"apache-iceberg",
|
|
25
|
+
"data-catalog",
|
|
22
26
|
"data-lake",
|
|
23
|
-
"
|
|
27
|
+
"lakehouse",
|
|
28
|
+
"schema-evolution",
|
|
29
|
+
"data-engineering",
|
|
30
|
+
"l2-construct"
|
|
24
31
|
],
|
|
25
32
|
"main": "dist/lib/iceberg/index.js",
|
|
26
33
|
"types": "dist/lib/iceberg/index.d.ts",
|
|
@@ -42,19 +49,19 @@
|
|
|
42
49
|
"prepublishOnly": "npm run build && npm test"
|
|
43
50
|
},
|
|
44
51
|
"peerDependencies": {
|
|
45
|
-
"@aws-cdk/aws-glue-alpha": "^2.
|
|
46
|
-
"aws-cdk-lib": "^2.
|
|
52
|
+
"@aws-cdk/aws-glue-alpha": "^2.258.1-alpha.0",
|
|
53
|
+
"aws-cdk-lib": "^2.258.1",
|
|
47
54
|
"constructs": "^10.0.0"
|
|
48
55
|
},
|
|
49
56
|
"devDependencies": {
|
|
50
|
-
"@aws-cdk/aws-glue-alpha": "^2.
|
|
57
|
+
"@aws-cdk/aws-glue-alpha": "^2.258.1-alpha.0",
|
|
51
58
|
"@eslint/js": "^9.27.0",
|
|
52
59
|
"@types/jest": "^29.5.14",
|
|
53
60
|
"@types/node": "^24.13.2",
|
|
54
61
|
"@typescript-eslint/eslint-plugin": "^8.0.0",
|
|
55
62
|
"@typescript-eslint/parser": "^8.0.0",
|
|
56
63
|
"aws-cdk": "^2.1024.0",
|
|
57
|
-
"aws-cdk-lib": "^2.
|
|
64
|
+
"aws-cdk-lib": "^2.258.1",
|
|
58
65
|
"constructs": "^10.0.0",
|
|
59
66
|
"eslint": "^9.27.0",
|
|
60
67
|
"eslint-plugin-jest": "^28.0.0",
|