cdk-glue-iceberg-table 0.3.0 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +104 -16
  2. package/package.json +11 -4
package/README.md CHANGED
@@ -8,23 +8,45 @@
8
8
  [![last commit](https://img.shields.io/github/last-commit/ksco92/arceus.svg)](https://github.com/ksco92/arceus/commits/main)
9
9
  [![license](https://img.shields.io/npm/l/cdk-glue-iceberg-table.svg)](LICENSE)
10
10
 
11
- A CDK L2 construct for Apache Iceberg tables in the AWS Glue Data
12
- Catalog. Emits the `AWS::Glue::Table` shape that survives
13
- CloudFormation `Update`, so `cdk deploy` can create, evolve, and
14
- destroy Iceberg tables the same way it handles any other resource.
15
-
16
- The motivating issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
11
+ **`cdk-glue-iceberg-table` is the AWS CDK L2 construct for creating
12
+ and evolving Apache Iceberg tables in the AWS Glue Data Catalog.** It
13
+ emits the exact `AWS::Glue::Table` / `OpenTableFormatInput` shape that
14
+ survives a CloudFormation `Update`, so a single `cdk deploy` creates a
15
+ table, evolves its schema and partitions, and destroys it, the same
16
+ lifecycle CDK gives any other resource. No custom resource, no Lambda,
17
+ no "`cdk deploy` and then run this SQL by hand" two-step. Table
18
+ changes (new columns, renames, drops, new partition fields) land as
19
+ a reviewed diff in a pull request and apply through `cdk deploy`.
20
+
21
+ Status (June 2026): published on npm, pre-1.0, with a public surface
22
+ guarded by an end-to-end consumer test on every PR plus three
23
+ real-AWS integration suites.
24
+
25
+ ### Why this exists
26
+
27
+ AWS documents the CloudFormation shape for Iceberg tables ([AWS Big
28
+ Data blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),
29
+ but raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the
30
+ *first* `Update` silently strips `table_type=ICEBERG`, after which
31
+ every Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The
32
+ motivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);
17
33
  [`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)
18
- documents the only working CFN shape and the silent-corruption traps
19
- you can hit by getting it slightly wrong. This construct implements
20
- that shape and refuses to emit the unsafe alternatives.
21
-
22
- The upstream CDK PR landing this construct in
23
- `@aws-cdk/aws-glue-alpha` is [aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988).
24
- Until that merges, this package is the most current reference
25
- implementation. Once `@aws-cdk/aws-glue-alpha` ships its own
26
- `IcebergTable`, prefer the official one and treat this package as a
27
- stopgap.
34
+ documents the only working shape and the silent-corruption traps you
35
+ hit by getting it slightly wrong. This construct implements that
36
+ shape, refuses to emit the unsafe alternatives, and pins Iceberg field
37
+ IDs so schema evolution never orphans existing data. It is the basis
38
+ of the upstream proposal to land an `IcebergTable` in
39
+ `@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988));
40
+ this package tracks that proposal and stays current with it.
41
+
42
+ ### How it compares
43
+
44
+ | Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |
45
+ | --- | --- | --- | --- | --- |
46
+ | Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |
47
+ | CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |
48
+ | Spark / Athena SQL DDL at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |
49
+ | **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |
28
50
 
29
51
  ## Install
30
52
 
@@ -71,9 +93,11 @@ new IcebergTable(this, 'OrdersTable', {
71
93
 
72
94
  Consumer-facing reference sections below:
73
95
 
96
+ - [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.
74
97
  - [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.
75
98
  - [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.
76
99
  - [Known limitations](#known-limitations) — what the construct does and doesn't enforce.
100
+ - [FAQ](#faq) — common "how do I … in CDK / CloudFormation" questions.
77
101
 
78
102
  ## Repo layout
79
103
 
@@ -657,6 +681,70 @@ the construct does **not** prevent. See the next section.)
657
681
  - **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.
658
682
  - **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The integration-test script demonstrates the safe pattern: drop partitions that source from `customer_id` while keeping `customer_id` itself in the schema, and drop the `region` column while it is not partitioning anything.
659
683
 
684
+ ## FAQ
685
+
686
+ ### How do I create an Apache Iceberg table in AWS CDK?
687
+
688
+ Install `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a
689
+ database, a column list, and an S3 `location` — see [Use](#use).
690
+ `cdk deploy` creates the Glue Data Catalog table and writes the
691
+ Iceberg `metadata.json` under your S3 prefix. No custom resource or
692
+ Lambda is involved; it is a plain `AWS::Glue::Table` in your
693
+ CloudFormation template.
694
+
695
+ ### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?
696
+
697
+ Change the `columns` array and run `cdk deploy` again. The construct
698
+ passes the new schema to Glue's `UpdateTable`, which writes a new
699
+ `metadata.json` with a new `schema-id`; existing data files stay
700
+ readable because each column's `id` is pinned and never reused. The
701
+ same applies to `partitionSpec`. See [Schema + partition
702
+ evolution](#schema--partition-evolution-via-cdk-only) for a four-step
703
+ run that adds, renames, and drops columns and partitions through
704
+ `cdk deploy` alone.
705
+
706
+ ### Does CloudFormation support Iceberg tables natively?
707
+
708
+ Yes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,
709
+ documented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).
710
+ The catch is that the raw shape corrupts the table on the first
711
+ `Update` if you place schema under `storageDescriptor.columns` or set
712
+ `tableInput` alongside `openTableFormatInput`. This construct only ever
713
+ emits the safe shape and gives you no way to express the unsafe ones.
714
+ See [Two footguns the construct prevents](#two-footguns-the-construct-prevents).
715
+
716
+ ### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?
717
+
718
+ A raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON
719
+ and own both silent-corruption footguns. `cdk-glue-iceberg-table` gives
720
+ you a typed `IcebergType` / `IcebergPartitionTransform` API,
721
+ synth-time validation (partition transforms checked against column
722
+ types, format/version mismatches caught before deploy), pinned field
723
+ IDs for safe evolution, and `grantRead` / `grantWrite` helpers. See
724
+ [How it compares](#how-it-compares).
725
+
726
+ ### How do I create a partitioned Iceberg table in CDK?
727
+
728
+ Pass a `partitionSpec` of `IcebergPartitionTransform` entries —
729
+ `identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,
730
+ or `void`. Each transform is validated against its source column's type
731
+ at synth time. See [Using `IcebergTable`](#using-icebergtable).
732
+
733
+ ### Does it work with Athena and Lake Formation?
734
+
735
+ Yes. The demo `ArceusStack` registers the tables with Lake Formation
736
+ and queries them from Athena, including v2 merge-on-read `INSERT` /
737
+ `UPDATE` / `DELETE` / `MERGE`, time travel, `OPTIMIZE`, and `VACUUM`.
738
+ The construct's `grant*` helpers issue IAM grants; under Lake Formation
739
+ you still add the matching `SELECT` / `INSERT` LF grants. See
740
+ [Inserting and querying](#inserting-and-querying).
741
+
742
+ ### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?
743
+
744
+ Yes — set `formatVersion: IcebergFormatVersion.V2` and the
745
+ `write.{delete,update,merge}.mode = merge-on-read` table properties.
746
+ The construct rejects merge-on-read on a v1 table at synth time.
747
+
660
748
  ## Tests
661
749
 
662
750
  ```
package/package.json CHANGED
@@ -1,8 +1,8 @@
1
1
  {
2
2
  "name": "cdk-glue-iceberg-table",
3
- "version": "0.3.0",
3
+ "version": "0.3.1",
4
4
  "license": "MIT",
5
- "description": "CDK L2 construct for Apache Iceberg tables in the AWS Glue Data Catalog.",
5
+ "description": "AWS CDK L2 construct to create and evolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and partition changes that survive CloudFormation updates, via cdk deploy.",
6
6
  "author": "Rodrigo Carvajal",
7
7
  "repository": {
8
8
  "type": "git",
@@ -14,13 +14,20 @@
14
14
  },
15
15
  "keywords": [
16
16
  "aws",
17
- "cdk",
18
17
  "aws-cdk",
18
+ "awscdk",
19
+ "cdk",
20
+ "cloudformation",
19
21
  "glue",
22
+ "glue-data-catalog",
20
23
  "iceberg",
21
24
  "apache-iceberg",
25
+ "data-catalog",
22
26
  "data-lake",
23
- "construct"
27
+ "lakehouse",
28
+ "schema-evolution",
29
+ "data-engineering",
30
+ "l2-construct"
24
31
  ],
25
32
  "main": "dist/lib/iceberg/index.js",
26
33
  "types": "dist/lib/iceberg/index.d.ts",