cdk-glue-iceberg-table 0.14.1 → 0.14.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/.jsii
CHANGED
|
@@ -9751,7 +9751,7 @@
|
|
|
9751
9751
|
},
|
|
9752
9752
|
"name": "cdk-glue-iceberg-table",
|
|
9753
9753
|
"readme": {
|
|
9754
|
-
"markdown": "# cdk-glue-iceberg-table\n\n[](https://github.com/ksco92/arceus/actions/workflows/ci.yml)\n[](https://codecov.io/gh/ksco92/arceus)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://pypi.org/project/cdk-glue-iceberg-table/)\n[](https://constructs.dev/packages/cdk-glue-iceberg-table)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://github.com/ksco92/arceus/blob/main/LICENSE)\n\n**`cdk-glue-iceberg-table` is an AWS CDK L2 construct to create and\nevolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and\npartition changes that survive CloudFormation updates, via cdk\ndeploy.** It supports Iceberg format version 3 (`variant`, nanosecond\ntimestamps, `geometry` / `geography`, `unknown`). It emits the exact\n`AWS::Glue::Table` / `OpenTableFormatInput` shape, so a single `cdk\ndeploy` creates, evolves, and destroys a table like any other CDK\nresource. No custom resource, no Lambda, no \"`cdk deploy` and then run\nthis SQL by hand\" two-step. Table changes (new columns, renames, drops,\nnew partition fields) land as a reviewed diff in a pull request and\napply through `cdk deploy`.\n\nStatus (October 2026): published on npm (TypeScript/JavaScript) and\nPyPI (Python), pre-1.0, with a public surface guarded on every PR by two\nconsumer jobs (`e2e-consumer` compiles and synthesizes an app against a\npinned published version, `packed-consumer` does the same against the PR's\nown `npm pack` tarball) plus real-AWS integration suites, including ones\nfor format version 3. The multi-language packages are generated\nfrom one TypeScript source with [jsii](https://aws.github.io/jsii/)\nand indexed on [Construct\nHub](https://constructs.dev/packages/cdk-glue-iceberg-table).\nDeveloped in the [`ksco92/arceus`](https://github.com/ksco92/arceus)\nmonorepo, which also holds a CDK demo app that dogfoods the construct\nagainst a real AWS account.\n\n### Why this exists\n\nAWS documents the CloudFormation shape for Iceberg tables ([AWS Big\nData blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),\nbut raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the\n*first* `Update` silently strips `table_type=ICEBERG`, after which\nevery Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The\nmotivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);\n[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)\ndocuments the only working shape and the silent-corruption traps you\nhit by getting it slightly wrong. This construct implements that\nshape, refuses to emit the unsafe alternatives, and pins Iceberg field\nIDs (declared [in creation order](#using-icebergtable) for a new\ntable) so schema evolution never orphans existing data. It was the\nbasis of an upstream proposal to land an `IcebergTable` in\n`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988)),\nclosed unmerged in September 2026.\n\n### How it compares\n\n| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |\n| --- | --- | --- | --- | --- |\n| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |\n| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |\n| Spark SQL DDL, or Athena SQL DDL, at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |\n| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |\n\n## Install\n\nTypeScript / JavaScript (npm):\n\n```bash\nnpm install cdk-glue-iceberg-table\n```\n\nPython (PyPI):\n\n```bash\npip install cdk-glue-iceberg-table\n```\n\nPeer dependencies (your CDK app must already have these). This\nrelease requires `aws-cdk-lib` 2.273.0 or newer and `constructs`\n10.8.1 or newer, plus Node.js 22.14 or newer for both languages (the\nPython package runs under Node.js through jsii).\n\nFor TypeScript / JavaScript:\n\n```bash\nnpm install aws-cdk-lib@^2.273.0 constructs@^10.8.1\n```\n\nFor Python:\n\n```bash\npip install 'aws-cdk-lib>=2.273.0' 'constructs>=10.8.1'\n```\n\n## Use\n\n### TypeScript\n\n```ts\nimport { Bucket } from 'aws-cdk-lib/aws-s3';\nimport { Database } from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\nconst db = new Database(this, 'Db', { databaseName: 'analytics' });\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n columns: [\n { name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },\n { name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },\n ],\n partitionSpec: [\n { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY, fieldId: 1000 },\n { sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16), fieldId: 1001 },\n ],\n identifierFieldNames: ['order_id'],\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\n### Python\n\nThe same table in Python — the API mirrors TypeScript with\n`snake_case` props and `PascalCase` types:\n\n```python\nfrom aws_cdk.aws_s3 import Bucket\nfrom aws_cdk.aws_glue import Database\nfrom cdk_glue_iceberg_table import (\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n)\n\nbucket = Bucket(self, \"Warehouse\")\ndb = Database(self, \"Db\", database_name=\"analytics\")\n\nIcebergTable(self, \"OrdersTable\",\n database=db,\n table_name=\"orders\",\n location=f\"s3://{bucket.bucket_name}/analytics/orders/\",\n columns=[\n {\"name\": \"order_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 1},\n {\"name\": \"customer_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 2},\n {\"name\": \"placed_at\", \"type\": IcebergType.TIMESTAMPTZ, \"required\": True, \"id\": 3},\n ],\n partition_spec=[\n {\"source_column\": \"placed_at\", \"transform\": IcebergPartitionTransform.DAY, \"field_id\": 1000},\n {\"source_column\": \"customer_id\", \"transform\": IcebergPartitionTransform.bucket(16), \"field_id\": 1001},\n ],\n identifier_field_names=[\"order_id\"],\n format_version=IcebergFormatVersion.V2,\n)\n```\n\nConsumer-facing reference sections:\n\n- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.\n- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.\n- [Format version 3](#format-version-3) — the six v3 types, which engines read them, and the accepted limits.\n- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.\n- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.\n- [FAQ](#faq) — common \"how do I … in CDK / CloudFormation\" questions.\n\n## Exported surface\n\nThe package entry point re-exports the main types you import from\n`cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled\nsurface lives under `dist/lib/iceberg/` after `npm install`; in Python\nthe same surface is imported from the `cdk_glue_iceberg_table` module\nafter `pip install`:\n\n- **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite`, the `fromIcebergTableAttributes(...)` import factory, and `creationOrderFieldIds(columns)` (Python: `creation_order_field_ids`), which returns a map from field path to the id Glue gives that field in a new table; a name containing `.` or a backquote appears in backquotes.\n- **`IcebergType`** — primitive types, the v3 constants `UNKNOWN`, `TIMESTAMP_NS`, `TIMESTAMPTZ_NS`, and `VARIANT`, and the `list` / `map` / `struct` / `decimal` / `fixed` / `geometry(crs?)` / `geography(crs?, algorithm?)` factories. `list` takes an `IcebergListDefinition`, `map` an `IcebergMapDefinition`, and `struct` a list of `IcebergStructFieldDefinition`, each carrying the nested field ids. Renders to the string Glue's `IcebergStructField.type` expects.\n- **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.\n- **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2/v3, required — set explicitly per table; Athena cannot query v3 tables), **`IcebergSortDirection`**, **`IcebergNullOrder`**, **`IcebergEdgeAlgorithm`** (`SPHERICAL`, `VINCENTY`, `THOMAS`, `ANDOYER`, `KARNEY`; the second argument of `IcebergType.geography`), plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).\n\n## Using `IcebergTable`\n\nA minimal table:\n\n```typescript\nimport {\n Bucket,\n} from 'aws-cdk-lib/aws-s3';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\n\nconst db = new Database(this, 'Db', {\n databaseName: 'analytics',\n});\n\nnew IcebergTable(this, 'Users', {\n database: db,\n tableName: 'users',\n columns: [\n {\n name: 'user_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'email',\n type: IcebergType.STRING,\n required: true,\n id: 2,\n },\n {\n name: 'signed_up_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 3,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/users/`,\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\nEvery top-level column must declare an `id`: a stable, unique,\npositive integer you pick once and never reuse or reassign. Because\nIceberg data files reference fields by id rather than by name, a fixed\nid, declared in creation order when the table is created (below), is\nwhat lets you add, rename, and drop columns across deploys without\ncorrupting old data — see\n[How do I evolve an Iceberg table schema](#how-do-i-evolve-an-iceberg-table-schema-add-rename-or-drop-a-column-in-cloudformation).\nOnce a column is dropped, leave its id retired; never hand it to a new\ncolumn. Nested fields follow the same rule: every struct field takes\nan `id`, every list an `elementId`, and every map a `keyId` and a\n`valueId`. All ids in a schema, top-level and nested, must be distinct\nintegers from 1 to 2147483447.\n\nA new table must declare its ids in creation order. When Glue creates\na table it does not keep the declared ids: it numbers the fields from\n1 in creation order and points the partition spec, sort order and\nidentifier fields at the new ids. An update Glue accepts stores the\ndeclared ids as given, so the first such update of a table created\nwith other ids renumbers its fields; for the update Glue rejects, see\n[Known limitations](#known-limitations). To number a new table, number\nthe top-level columns first, in order, then the nested fields of each\ncolumn in turn (a list's element; a map's key, then its value, then\nthe key's nested fields, then the value's; a struct's fields, then\ntheir nested fields). `IcebergTable.creationOrderFieldIds` returns\nthese ids by field path and ignores the ids in its input:\n\n```typescript\nconst columns = [\n { name: 'id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'tags', type: IcebergType.list({ element: IcebergType.STRING, elementId: 4 }), id: 2 },\n {\n name: 'attrs',\n type: IcebergType.map({ key: IcebergType.STRING, keyId: 5, value: IcebergType.STRING, valueId: 6 }),\n id: 3,\n },\n];\nIcebergTable.creationOrderFieldIds(columns);\n// { id: 1, tags: 2, attrs: 3, 'tags.element': 4, 'attrs.key': 5, 'attrs.value': 6 }\n```\n\nIds out of creation order are for fields a later deploy adds; see\n[Evolving schema and partitions](#evolving-schema-and-partitions).\n\nA table that exercises most of the surface (partitions, sort order,\nnested types, identifier fields, table properties, removal policy):\n\n```typescript\nimport {\n RemovalPolicy,\n} from 'aws-cdk-lib';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergDataFormat,\n IcebergFormatVersion,\n IcebergNullOrder,\n IcebergPartitionTransform,\n IcebergSortDirection,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',\n columns: [\n {\n name: 'order_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'customer_id',\n type: IcebergType.LONG,\n required: true,\n id: 2,\n },\n {\n name: 'order_amount',\n type: IcebergType.decimal(12, 2),\n required: true,\n id: 3,\n },\n {\n name: 'currency',\n type: IcebergType.STRING,\n required: true,\n id: 4,\n },\n {\n name: 'placed_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 5,\n },\n {\n name: 'tags',\n type: IcebergType.list({\n element: IcebergType.STRING,\n elementId: 9,\n }),\n id: 6,\n },\n {\n name: 'shipping_address',\n type: IcebergType.struct([\n {\n id: 10,\n name: 'line1',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 11,\n name: 'city',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 12,\n name: 'country',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 13,\n name: 'postal_code',\n type: IcebergType.STRING,\n },\n ]),\n id: 7,\n },\n {\n name: 'metadata',\n type: IcebergType.map({\n key: IcebergType.STRING,\n keyId: 14,\n value: IcebergType.STRING,\n valueId: 15,\n }),\n id: 8,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n partitionSpec: [\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1000,\n },\n {\n sourceColumn: 'customer_id',\n transform: IcebergPartitionTransform.bucket(16),\n fieldId: 1001,\n },\n ],\n sortOrder: [\n {\n sourceColumn: 'placed_at',\n direction: IcebergSortDirection.ASC,\n nullOrder: IcebergNullOrder.NULLS_LAST,\n },\n {\n sourceColumn: 'order_id',\n direction: IcebergSortDirection.ASC,\n },\n ],\n identifierFieldNames: [\n 'order_id',\n ],\n dataFormat: IcebergDataFormat.PARQUET,\n formatVersion: IcebergFormatVersion.V2,\n tableProperties: {\n 'write.parquet.compression-codec': 'zstd',\n 'write.delete.mode': 'merge-on-read',\n 'write.update.mode': 'merge-on-read',\n 'write.merge.mode': 'merge-on-read',\n 'write.target-file-size-bytes': '134217728',\n 'history.expire.min-snapshots-to-keep': '5',\n 'gc.enabled': 'true',\n },\n removalPolicy: RemovalPolicy.DESTROY,\n});\n```\n\nEach partition field also pins a `fieldId`: an integer of 1000 or\nabove (the Iceberg spec reserves lower ids for schema columns), unique\nacross the table's entire partition-spec history. The discipline is\nthe inverse of the column-id rule. A column keeps its id forever; a\npartition field keeps its id only while it stays untouched. Change\nanything about a field — swap `month` for `day`, point it at a\ndifferent source column — and it becomes a *new* field that needs a\nfresh, never-before-used id, while the old id retires with the old\nspec. Reusing a retired id (or reassigning an id to a different field)\nproduces a table that Athena still reads but Spark and Trino reject\nwith `ValidationException: Conflicting partition fields`.\n\n`identifierFieldNames` lists the columns that together identify a row\nand renders as the schema's `identifier-field-ids`. Each one must be a\n`required: true` column (Glue rejects an optional one), and `cdk synth`\nrejects an identifier column of type float, double, list, map, struct,\nunknown, variant, geometry or geography. The construct prefixes each\nidentifier column's `doc` with `(PK) `, so a `doc` you write must not\nstart with `(PK) `; see [Evolving schema and\npartitions](#evolving-schema-and-partitions).\n\nThe resulting Iceberg `metadata.json` for this table contains every\nfeature you set:\n\n```json\n{\n \"format-version\": 2,\n \"table-uuid\": \"39a948f9-...\",\n \"current-schema-id\": 0,\n \"schemas\": [\n {\n \"schema-id\": 0,\n \"identifier-field-ids\": [1],\n \"fields\": [\n { \"id\": 1, \"name\": \"order_id\", \"required\": true, \"type\": \"long\", \"doc\": \"(PK) \" },\n { \"id\": 2, \"name\": \"customer_id\", \"required\": true, \"type\": \"long\" },\n { \"id\": 3, \"name\": \"order_amount\", \"required\": true, \"type\": \"decimal(12, 2)\" },\n { \"id\": 4, \"name\": \"currency\", \"required\": true, \"type\": \"string\" },\n { \"id\": 5, \"name\": \"placed_at\", \"required\": true, \"type\": \"timestamptz\" },\n { \"id\": 6, \"name\": \"tags\", \"required\": false,\n \"type\": { \"type\": \"list\", \"element-id\": 9, \"element\": \"string\", \"element-required\": false } },\n { \"id\": 7, \"name\": \"shipping_address\", \"required\": false,\n \"type\": { \"type\": \"struct\", \"fields\": [\n { \"id\": 10, \"name\": \"line1\", \"required\": true, \"type\": \"string\" },\n { \"id\": 11, \"name\": \"city\", \"required\": true, \"type\": \"string\" },\n { \"id\": 12, \"name\": \"country\", \"required\": true, \"type\": \"string\" },\n { \"id\": 13, \"name\": \"postal_code\", \"required\": false, \"type\": \"string\" }\n ] } },\n { \"id\": 8, \"name\": \"metadata\", \"required\": false,\n \"type\": { \"type\": \"map\", \"key-id\": 14, \"key\": \"string\", \"value-id\": 15,\n \"value-required\": false, \"value\": \"string\" } }\n ]\n }\n ],\n \"partition-specs\": [\n { \"spec-id\": 0, \"fields\": [\n { \"name\": \"placed_at_day\", \"transform\": \"day\", \"source-id\": 5, \"field-id\": 1000 },\n { \"name\": \"customer_id_bucket\", \"transform\": \"bucket[16]\", \"source-id\": 2, \"field-id\": 1001 }\n ]}\n ],\n \"sort-orders\": [\n { \"order-id\": 1, \"fields\": [\n { \"transform\": \"identity\", \"source-id\": 5, \"direction\": \"asc\", \"null-order\": \"nulls-last\" },\n { \"transform\": \"identity\", \"source-id\": 1, \"direction\": \"asc\", \"null-order\": \"nulls-last\" }\n ]}\n ],\n \"properties\": {\n \"format-version\": \"2\",\n \"write.format.default\": \"parquet\",\n \"write.parquet.compression-codec\": \"zstd\",\n \"write.merge.mode\": \"merge-on-read\",\n \"write.update.mode\": \"merge-on-read\",\n \"write.delete.mode\": \"merge-on-read\",\n \"write.target-file-size-bytes\": \"134217728\",\n \"history.expire.min-snapshots-to-keep\": \"5\",\n \"gc.enabled\": \"true\",\n \"comment\": \"Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.\"\n }\n}\n```\n\n### Format version 3\n\nSet `formatVersion: IcebergFormatVersion.V3` to create a v3 table. The\nsix v3 types (below) are only accepted on a V3 table, at any nesting\ndepth: inside a struct, as a list element, or as a map key or value.\nSynth fails on a V1 or V2 table that uses one.\n\n```typescript\nimport {\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'Events', {\n database: db,\n tableName: 'events',\n location: `s3://${bucket.bucketName}/analytics/events/`,\n columns: [\n { name: 'occurred_at', type: IcebergType.TIMESTAMP_NS, id: 1 },\n { name: 'payload', type: IcebergType.VARIANT, id: 2 },\n {\n name: 'origin',\n type: IcebergType.geography('OGC:CRS84', IcebergEdgeAlgorithm.SPHERICAL),\n id: 3,\n },\n ],\n formatVersion: IcebergFormatVersion.V3,\n});\n```\n\nThe same table in Python:\n\n```python\nfrom cdk_glue_iceberg_table import (\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n)\n\nIcebergTable(self, \"Events\",\n database=db,\n table_name=\"events\",\n location=f\"s3://{bucket.bucket_name}/analytics/events/\",\n columns=[\n {\"name\": \"occurred_at\", \"type\": IcebergType.TIMESTAMP_NS, \"id\": 1},\n {\"name\": \"payload\", \"type\": IcebergType.VARIANT, \"id\": 2},\n {\n \"name\": \"origin\",\n \"type\": IcebergType.geography(\"OGC:CRS84\", IcebergEdgeAlgorithm.SPHERICAL),\n \"id\": 3,\n },\n ],\n format_version=IcebergFormatVersion.V3,\n)\n```\n\nThis table has a `timestamp_ns` column; see [Known limitations](#known-limitations)\nbefore reading it from upstream Spark.\n\nThe v3 types, with the rules `cdk synth` enforces:\n\n| Type | Factory | Rendered string | Partition and identifier rules |\n| --- | --- | --- | --- |\n| unknown | `IcebergType.UNKNOWN` | `unknown` | Must be optional, as a column, list element, or map value; never a map key. Partition source only with `void`. Not an identifier field. |\n| timestamp with nanoseconds | `IcebergType.TIMESTAMP_NS` | `timestamp_ns` | `identity`, `year`, `month`, `day`, `hour`, `bucket(N)`, `void`; not `truncate(W)`. Allowed as an identifier field. |\n| timestamptz with nanoseconds | `IcebergType.TIMESTAMPTZ_NS` | `timestamptz_ns` | Same as `timestamp_ns`. |\n| variant | `IcebergType.VARIANT` | `variant` | Never a partition or sort source, `void` included. Not an identifier field. |\n| geometry | `IcebergType.geometry(crs?)` | `geometry`, or `geometry(<crs>)` when a CRS is given | Partition source only with `void`. Not an identifier field. |\n| geography | `IcebergType.geography(crs?, algorithm?)` | `geography`, `geography(<crs>)`, or `geography(<crs>, <algorithm>)` when an algorithm is given, such as `geography(OGC:CRS84, spherical)` | Same as `geometry`. |\n\nA geometry or geography `crs` is an identifier such as `OGC:CRS84` or\n`EPSG:3857`; whitespace, parentheses, braces, and commas are rejected.\nWhen `algorithm` is set without a `crs`, the rendered string spells out\n`OGC:CRS84`.\n\nWhich engines read a v3 table created by the construct:\n\n| Engine | v3 tables | Notes |\n| --- | --- | --- |\n| Spark 4.1 + Iceberg 1.12 (the integration-test runner) | Yes | Reads and writes tables the construct creates, including deletion vectors, row lineage, `variant`, `unknown`, and `geometry` / `geography`, except tables with nanosecond-timestamp columns or a geometry CRS other than `OGC:CRS84` (checked by `scripts/spark/verify_v3_dml.py`; details in [Known limitations](#known-limitations)). |\n| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists \"Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types\". The integration suites do not run on Glue 6.0. |\n| Amazon EMR Spark 8.1.0 (Spark 4.1.1, Iceberg 1.11.0-amzn-0) | Documented | The [emr-spark-8.1.0 release notes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html) state \"Amazon EMR Spark 8.1.0 extends Iceberg v3 spec support for Spark workloads\". The integration suites do not run on EMR 8.1. |\n| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states \"Athena only creates and operates on Iceberg v2 tables.\" |\n\n### Granting access\n\n```typescript\ntable.grantRead(role); // Glue read + S3 read on the table's prefix\ntable.grantWrite(role); // Glue write + S3 write\ntable.grantReadWrite(role);\n```\n\nThe `grant*` helpers issue **IAM grants only**. Under Lake Formation\nyou still add the matching `SELECT` / `INSERT` / `DELETE` LF grants on\ntop of the construct's IAM grants for Athena queries to succeed.\n\n### Importing an existing table\n\n```typescript\nconst existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {\n database: db,\n tableName: 'orders',\n location: 's3://my-bucket/analytics/orders/',\n});\nexisting.grantRead(role);\n```\n\n### Evolving schema and partitions\n\nChange the `columns` array (or `partitionSpec`) and run `cdk deploy`\nagain. The construct passes the new schema to Glue's `UpdateTable`,\nwhich writes a new `metadata.json` with a new `schema-id`; existing\ndata files stay readable because each column's `id` is pinned to the\nid the table holds and never reused. Adds, renames (same `id`, new\n`name`), and drops all flow through `cdk deploy` alone — no\nout-of-band SQL DDL.\n\nNested fields evolve the same way. Each struct field, list element,\nmap key, and map value keeps the id you gave it (the rule is under\n[Using `IcebergTable`](#using-icebergtable)), so adding a column or a\nstruct field anywhere in the schema leaves every other id unchanged. A\nnew nested field takes an id no field of the table has ever used:\n\n```typescript\ntype: IcebergType.struct([\n { id: 10, name: 'line1', type: IcebergType.STRING },\n { id: 16, name: 'line2', type: IcebergType.STRING }, // new field, never-used id\n { id: 11, name: 'city', type: IcebergType.STRING },\n]),\n```\n\nSuch an id is out of creation order, which is expected once a table\nhas evolved: creation order applies only to the ids a table is created\nwith (see [Using `IcebergTable`](#using-icebergtable)).\n\nUpgrading from a version before 0.12.0, which numbered nested fields\nitself: declare the ids the table's current `metadata.json` holds,\ntop-level and nested (each field `id`, `element-id`, `key-id` and\n`value-id`), or the next deploy renumbers the fields that differ and\nexisting data files read under the wrong field. A nested type object\nshared by two columns now fails synth with a duplicate id; give each\ncolumn its own nested type with its own ids. The list element and map\nvalue nullability default also changed: `elementRequired` and\n`valueRequired` now default to `false`. Earlier versions made list\nelements required by default and map values required unless the third\nargument of `map` was `false`; set `elementRequired: true` or\n`valueRequired: true` wherever the deployed schema has\n`\"element-required\": true` or `\"value-required\": true`.\n\nGlue's `UpdateTable` reuses an earlier schema whose fields (name,\ntype, `required`, `doc`) match the request, together with that\nschema's `identifier-field-ids`. The construct therefore prefixes the\n`doc` of every identifier column with `(PK) ` (just `(PK) ` when the\ncolumn has no `doc`), shown as the column comment, so every\nidentifier change is a field change. On a table whose schema history\nwas written with the marker, identifier changes flow through `cdk\ndeploy` like column changes.\n\nPartition evolution flows the same way, with one extra rule on\n`fieldId`: a changed partition field is a new partition field. To move\na table from `month(placed_at)` to `day(placed_at)`, drop the month\nentry and add a day entry under a fresh `fieldId` — never recycle the\nmonth field's id:\n\n```typescript\npartitionSpec: [\n // was: { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.MONTH, fieldId: 1000 },\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1001,\n },\n],\n```\n\nOld data files written under the month spec stay readable; new writes\npartition by day. Reusing `1000` for the day field instead corrupts\nthe spec history for strict engines (Spark, Trino) while Athena keeps\nworking — the failure only surfaces at Spark runtime.\n\n### Inserting and querying\n\n```sql\n-- INSERT into the orders table\nINSERT INTO sample_database.orders VALUES\n (1001, 5001, DECIMAL '149.99', 'USD',\n TIMESTAMP '2026-05-20 09:15:00 UTC',\n ARRAY['holiday-promo','first-order'],\n CAST(ROW('1 Infinite Loop','Cupertino','US','95014')\n AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),\n MAP(ARRAY['channel','utm'], ARRAY['web','google'])),\n -- ... more rows\n;\n\n-- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)\nDELETE FROM sample_database.orders WHERE order_id = 1003;\n\n-- merge-on-read UPDATE\nUPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;\n\n-- SELECT\nSELECT customer_id, SUM(order_amount) AS total\n FROM sample_database.orders\n GROUP BY 1\n ORDER BY 2 DESC;\n```\n\n## Two footguns the construct prevents\n\n### Footgun #1 — schema under `storageDescriptor.columns`\n\nThe CREATE succeeds but the first UPDATE silently strips\n`table_type=ICEBERG` from the table's Glue parameters, and Athena\nqueries after that fail with `HIVE_UNSUPPORTED_FORMAT`.\n\n```typescript\n// DON'T DO THIS — what most StackOverflow / re:Post examples show\nnew CfnTable(this, 'OrdersBad', {\n catalogId: this.account,\n databaseName: 'analytics',\n tableInput: {\n name: 'orders',\n tableType: 'EXTERNAL_TABLE',\n parameters: {\n table_type: 'ICEBERG',\n },\n storageDescriptor: {\n location: 's3://.../orders/',\n columns: [\n /* ... */\n ],\n },\n },\n openTableFormatInput: {\n icebergInput: {\n metadataOperation: 'CREATE',\n version: '2',\n },\n },\n});\n```\n\n`IcebergTable` instead always emits schema/partitions/sort/properties\nunder `openTableFormatInput.icebergInput.icebergTableInput`, never\nunder `storageDescriptor`.\n\n### Footgun #2 — `tableInput` co-present with `openTableFormatInput`\n\nEven setting just `tableInput: { name: 'foo' }` next to\n`openTableFormatInput` returns\n`\"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput\"`.\nThe construct never emits `tableInput`; the table-level comment goes\ninto `tableProperties['comment']`, which lives inside\n`icebergTableInput.properties`.\n\n(There is a third footgun, field-id reuse after a column drop, that\nthe construct does **not** prevent. See the next section.)\n\n## Known limitations\n\n- **Creation order is not checked.** The construct cannot tell at synth whether a table is new or already deployed, so it accepts a new table whose ids are not in creation order (the rule is under [Using `IcebergTable`](#using-icebergtable)). When the first update of such a table changes the id of a column an identity partition is sourced from, Glue rejects the update with `Cannot create identity partition sourced from different field in schema` and the CloudFormation rollback fails the same way, leaving the stack in `UPDATE_ROLLBACK_FAILED`.\n- **Field-id reuse is not detected across deploys.** This applies to nested ids (struct field, list element, map key and value) as well as column ids. If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the \"never reuse a retired id\" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate field id N` validator), but it doesn't compare against the live table state. The safe workflow is to treat dropped ids as retired forever.\n- **Partition field-id reuse is not detected across deploys either.** The same live-state blindness applies to partition `fieldId`s: the construct validates them within one deploy (integer >= 1000, unique in the spec), but it cannot tell that the id you pinned already appeared in an earlier spec of the deployed table. The invariant partition ids must uphold is stricter than the column one — unique across the table's **entire spec history**, so a changed field (different transform or source column) must take a fresh id rather than keep its old one. Violate it and strict engines (Spark, Trino) reject the table with `Conflicting partition fields` while Athena keeps answering, masking the break until Spark runtime. The safe workflow: treat every id that has ever appeared in any spec as retired, and allocate new fields the next never-used id. A partition change needs a new source column or transform; Glue ignores a `fieldId`-only change.\n- **`last-partition-id` is written by Glue, not by the construct.** The construct supplies the spec's field ids; Glue's `UpdateTable` computes the table's `last-partition-id` counter when it writes the new metadata. The repo's integration test asserts the counter keeps up with the highest supplied field id after a partition evolution, but engines that later evolve the spec themselves (e.g. Spark `ALTER TABLE … ADD PARTITION FIELD`) allocate from that counter — if you mix CDK-managed and engine-managed partition evolution, verify the counter in `metadata.json` before trusting engine-side changes.\n- **Removing the last identifier from a table created with identifiers before 0.11.0 can bring the old ids back.** That table's history holds an unmarked schema with ids, and Glue reuses it when the rendered fields match it again. Change a column `doc` in the same deploy that removes the last identifier.\n- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.\n- **Format version is immutable after CREATE.** Glue ignores a V2 to V3 `formatVersion` change, though `properties.format-version` then reads `3`. Keep `formatVersion` at the table's real version, or destroy + recreate the table to change it. An engine can upgrade a v2 table to v3 in place with `ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3')`; after that, set `formatVersion` to `V3` so the next deploy declares the version the table already has.\n- **`merge-on-read` requires v2 or later.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.\n- **Column defaults are not available.** Iceberg v3 `initial-default` and `write-default` cannot be declared through CloudFormation: the `AWS::Glue::Table` resource schema has no `InitialDefault` or `WriteDefault` property. Revisit when that schema gains the two properties.\n- **Nanosecond timestamps are catalog-only for upstream Spark 4.1.** A table with any `timestamp_ns` or `timestamptz_ns` column fails to open in upstream Spark 4.1 with Iceberg 1.12. The integration suite for these columns asserts from `metadata.json` only. The [engine matrix](#format-version-3) lists the engines that document support.\n- **Geometry CRS other than `OGC:CRS84` is catalog-only for upstream Spark 4.1.** Spark 4.1 accepts `OGC:CRS84`, or bare `geometry`, as the geometry CRS; a table declaring another CRS, such as `EPSG:3857`, cannot be read there.\n- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.\n- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.\n\n## FAQ\n\n### How do I create an Apache Iceberg table in AWS CDK?\n\nInstall `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a\ndatabase, a column list, and an S3 `location` — see [Use](#use).\n`cdk deploy` creates the Glue Data Catalog table and writes the\nIceberg `metadata.json` under your S3 prefix. No custom resource or\nLambda is involved; it is a plain `AWS::Glue::Table` in your\nCloudFormation template.\n\n### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?\n\nChange the `columns` array and run `cdk deploy` again. The construct\npasses the new schema to Glue's `UpdateTable`, which writes a new\n`metadata.json` with a new `schema-id`; existing data files stay\nreadable because each column's `id` is pinned to the id the table\nholds and never reused.\nPartition changes work the same way, except a changed partition field\ntakes a fresh `fieldId` instead of keeping its old one. See [Evolving\nschema and partitions](#evolving-schema-and-partitions).\n\n### Does CloudFormation support Iceberg tables natively?\n\nYes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,\ndocumented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).\nThe catch is that the raw shape corrupts the table on the first\n`Update` if you place schema under `storageDescriptor.columns` or set\n`tableInput` alongside `openTableFormatInput`. This construct only ever\nemits the safe shape and gives you no way to express the unsafe ones.\nSee [Two footguns the construct prevents](#two-footguns-the-construct-prevents).\n\n### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?\n\nA raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON\nand own both silent-corruption footguns. `cdk-glue-iceberg-table` gives\nyou a typed `IcebergType` / `IcebergPartitionTransform` API,\nsynth-time validation (partition transforms checked against column\ntypes, format/version mismatches caught before deploy), pinned field\nIDs for safe evolution (declared in creation order for a new table),\nand `grantRead` / `grantWrite` helpers. See\n[How it compares](#how-it-compares).\n\n### How do I create a partitioned Iceberg table in CDK?\n\nPass a `partitionSpec` of `IcebergPartitionTransform` entries —\n`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,\nor `void` — each pinned to a `fieldId` of 1000 or above. Transforms are\nvalidated against their source column's type at synth time. See\n[Using `IcebergTable`](#using-icebergtable).\n\n### Does it work with Athena and Lake Formation?\n\nYes for v2 tables, no for v3 (see the engine matrix in [Format\nversion 3](#format-version-3)). On v2, Athena reads and writes,\nincluding merge-on-read `INSERT` / `UPDATE` / `DELETE` / `MERGE`, time\ntravel, `OPTIMIZE`, and `VACUUM`. The demo app in the\n[`arceus`](https://github.com/ksco92/arceus) repo registers its tables\nwith Lake Formation and exercises the same operations through Spark in\nits integration suite. The construct's `grant*` helpers issue IAM\ngrants; under Lake Formation you still add the matching `SELECT` /\n`INSERT` LF grants. See [Granting access](#granting-access).\n\n### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?\n\nYes — set `formatVersion: IcebergFormatVersion.V2` and the\n`write.{delete,update,merge}.mode = merge-on-read` table properties.\nThe construct rejects merge-on-read on a v1 table at synth time.\n\n### Can I create an Iceberg v3 table?\n\nYes. Set `formatVersion: IcebergFormatVersion.V3`; see [Format\nversion 3](#format-version-3) for the types, the engine matrix and the\nlimits.\n\n## Contributing\n\nDevelopment, the monorepo layout, and the demo app live in [CONTRIBUTING.md](https://github.com/ksco92/arceus/blob/main/CONTRIBUTING.md).\n\n## License\n\n[MIT](https://github.com/ksco92/arceus/blob/main/LICENSE).\n"
|
|
9754
|
+
"markdown": "# cdk-glue-iceberg-table\n\n[](https://github.com/ksco92/arceus/actions/workflows/ci.yml)\n[](https://codecov.io/gh/ksco92/arceus)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://pypi.org/project/cdk-glue-iceberg-table/)\n[](https://constructs.dev/packages/cdk-glue-iceberg-table)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://github.com/ksco92/arceus/blob/main/LICENSE)\n\n**`cdk-glue-iceberg-table` is an AWS CDK L2 construct to create and\nevolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and\npartition changes that survive CloudFormation updates, via cdk\ndeploy.** It supports Iceberg format version 3 (`variant`, nanosecond\ntimestamps, `geometry` / `geography`, `unknown`). It emits the exact\n`AWS::Glue::Table` / `OpenTableFormatInput` shape, so a single `cdk\ndeploy` creates, evolves, and destroys a table like any other CDK\nresource. No custom resource, no Lambda, no \"`cdk deploy` and then run\nthis SQL by hand\" two-step. Table changes (new columns, renames, drops,\nnew partition fields) land as a reviewed diff in a pull request and\napply through `cdk deploy`.\n\nStatus (October 2026): published on npm (TypeScript/JavaScript) and\nPyPI (Python), pre-1.0, with a public surface guarded on every PR by two\nconsumer jobs (`e2e-consumer` compiles and synthesizes an app against a\npinned published version, `packed-consumer` does the same against the PR's\nown `npm pack` tarball) plus real-AWS integration suites, including ones\nfor format version 3. The multi-language packages are generated\nfrom one TypeScript source with [jsii](https://aws.github.io/jsii/)\nand indexed on [Construct\nHub](https://constructs.dev/packages/cdk-glue-iceberg-table).\nDeveloped in the [`ksco92/arceus`](https://github.com/ksco92/arceus)\nmonorepo, which also holds a CDK demo app that dogfoods the construct\nagainst a real AWS account.\n\n### Why this exists\n\nAWS documents the CloudFormation shape for Iceberg tables ([AWS Big\nData blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),\nbut raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the\n*first* `Update` silently strips `table_type=ICEBERG`, after which\nevery Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The\nmotivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);\n[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)\ndocuments the only working shape and the silent-corruption traps you\nhit by getting it slightly wrong. This construct implements that\nshape, refuses to emit the unsafe alternatives, and pins Iceberg field\nIDs (declared [in creation order](#using-icebergtable) for a new\ntable) so schema evolution never orphans existing data. It was the\nbasis of an upstream proposal to land an `IcebergTable` in\n`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988)),\nclosed unmerged in September 2026.\n\n### How it compares\n\n| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |\n| --- | --- | --- | --- | --- |\n| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |\n| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |\n| Spark SQL DDL, or Athena SQL DDL, at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |\n| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |\n\n## Install\n\nTypeScript / JavaScript (npm):\n\n```bash\nnpm install cdk-glue-iceberg-table\n```\n\nPython (PyPI):\n\n```bash\npip install cdk-glue-iceberg-table\n```\n\nPeer dependencies (your CDK app must already have these). This\nrelease requires `aws-cdk-lib` 2.273.0 or newer and `constructs`\n10.8.1 or newer, plus Node.js 22.14 or newer for both languages (the\nPython package runs under Node.js through jsii).\n\nFor TypeScript / JavaScript:\n\n```bash\nnpm install aws-cdk-lib@^2.273.0 constructs@^10.8.1\n```\n\nFor Python:\n\n```bash\npip install 'aws-cdk-lib>=2.273.0' 'constructs>=10.8.1'\n```\n\n## Use\n\n### TypeScript\n\n```ts\nimport { Bucket } from 'aws-cdk-lib/aws-s3';\nimport { Database } from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\nconst db = new Database(this, 'Db', { databaseName: 'analytics' });\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n columns: [\n { name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },\n { name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },\n ],\n partitionSpec: [\n { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY, fieldId: 1000 },\n { sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16), fieldId: 1001 },\n ],\n identifierFieldNames: ['order_id'],\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\n### Python\n\nThe same table in Python — the API mirrors TypeScript with\n`snake_case` props and `PascalCase` types:\n\n```python\nfrom aws_cdk.aws_s3 import Bucket\nfrom aws_cdk.aws_glue import Database\nfrom cdk_glue_iceberg_table import (\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n)\n\nbucket = Bucket(self, \"Warehouse\")\ndb = Database(self, \"Db\", database_name=\"analytics\")\n\nIcebergTable(self, \"OrdersTable\",\n database=db,\n table_name=\"orders\",\n location=f\"s3://{bucket.bucket_name}/analytics/orders/\",\n columns=[\n {\"name\": \"order_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 1},\n {\"name\": \"customer_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 2},\n {\"name\": \"placed_at\", \"type\": IcebergType.TIMESTAMPTZ, \"required\": True, \"id\": 3},\n ],\n partition_spec=[\n {\"source_column\": \"placed_at\", \"transform\": IcebergPartitionTransform.DAY, \"field_id\": 1000},\n {\"source_column\": \"customer_id\", \"transform\": IcebergPartitionTransform.bucket(16), \"field_id\": 1001},\n ],\n identifier_field_names=[\"order_id\"],\n format_version=IcebergFormatVersion.V2,\n)\n```\n\nConsumer-facing reference sections:\n\n- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.\n- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.\n- [Format version 3](#format-version-3) — the six v3 types, which engines read them, and the accepted limits.\n- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.\n- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.\n- [FAQ](#faq) — common \"how do I … in CDK / CloudFormation\" questions.\n\n## Exported surface\n\nThe package entry point re-exports the main types you import from\n`cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled\nsurface lives under `dist/lib/iceberg/` after `npm install`; in Python\nthe same surface is imported from the `cdk_glue_iceberg_table` module\nafter `pip install`:\n\n- **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite`, the `fromIcebergTableAttributes(...)` import factory, and `creationOrderFieldIds(columns)` (Python: `creation_order_field_ids`), which returns a map from field path to the id Glue gives that field in a new table; a name containing `.` or a backquote appears in backquotes.\n- **`IcebergType`** — primitive types, the v3 constants `UNKNOWN`, `TIMESTAMP_NS`, `TIMESTAMPTZ_NS`, and `VARIANT`, and the `list` / `map` / `struct` / `decimal` / `fixed` / `geometry(crs?)` / `geography(crs?, algorithm?)` factories. `list` takes an `IcebergListDefinition`, `map` an `IcebergMapDefinition`, and `struct` a list of `IcebergStructFieldDefinition`, each carrying the nested field ids. Renders to the string Glue's `IcebergStructField.type` expects.\n- **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.\n- **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2/v3, required — set explicitly per table; Athena cannot query v3 tables), **`IcebergSortDirection`**, **`IcebergNullOrder`**, **`IcebergEdgeAlgorithm`** (`SPHERICAL`, `VINCENTY`, `THOMAS`, `ANDOYER`, `KARNEY`; the second argument of `IcebergType.geography`), plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).\n\n## Using `IcebergTable`\n\nA minimal table:\n\n```typescript\nimport {\n Bucket,\n} from 'aws-cdk-lib/aws-s3';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\n\nconst db = new Database(this, 'Db', {\n databaseName: 'analytics',\n});\n\nnew IcebergTable(this, 'Users', {\n database: db,\n tableName: 'users',\n columns: [\n {\n name: 'user_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'email',\n type: IcebergType.STRING,\n required: true,\n id: 2,\n },\n {\n name: 'signed_up_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 3,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/users/`,\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\nEvery top-level column must declare an `id`: a stable, unique,\npositive integer you pick once and never reuse or reassign. Because\nIceberg data files reference fields by id rather than by name, a fixed\nid, declared in creation order when the table is created (below), is\nwhat lets you add, rename, and drop columns across deploys without\ncorrupting old data — see\n[How do I evolve an Iceberg table schema](#how-do-i-evolve-an-iceberg-table-schema-add-rename-or-drop-a-column-in-cloudformation).\nOnce a column is dropped, leave its id retired; never hand it to a new\ncolumn. Nested fields follow the same rule: every struct field takes\nan `id`, every list an `elementId`, and every map a `keyId` and a\n`valueId`. All ids in a schema, top-level and nested, must be distinct\nintegers from 1 to 2147483447.\n\nA new table must declare its ids in creation order. When Glue creates\na table it does not keep the declared ids: it numbers the fields from\n1 in creation order and points the partition spec, sort order and\nidentifier fields at the new ids. An update Glue accepts stores the\ndeclared ids as given, so the first such update of a table created\nwith other ids renumbers its fields; for the update Glue rejects, see\n[Known limitations](#known-limitations). To number a new table, number\nthe top-level columns first, in order, then the nested fields of each\ncolumn in turn (a list's element; a map's key, then its value, then\nthe key's nested fields, then the value's; a struct's fields, then\ntheir nested fields). `IcebergTable.creationOrderFieldIds` returns\nthese ids by field path and ignores the ids in its input:\n\n```typescript\nconst columns = [\n { name: 'id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'tags', type: IcebergType.list({ element: IcebergType.STRING, elementId: 4 }), id: 2 },\n {\n name: 'attrs',\n type: IcebergType.map({ key: IcebergType.STRING, keyId: 5, value: IcebergType.STRING, valueId: 6 }),\n id: 3,\n },\n];\nIcebergTable.creationOrderFieldIds(columns);\n// { id: 1, tags: 2, attrs: 3, 'tags.element': 4, 'attrs.key': 5, 'attrs.value': 6 }\n```\n\nIds out of creation order are for fields a later deploy adds; see\n[Evolving schema and partitions](#evolving-schema-and-partitions).\n\nA table that exercises most of the surface (partitions, sort order,\nnested types, identifier fields, table properties, removal policy):\n\n```typescript\nimport {\n RemovalPolicy,\n} from 'aws-cdk-lib';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergDataFormat,\n IcebergFormatVersion,\n IcebergNullOrder,\n IcebergPartitionTransform,\n IcebergSortDirection,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',\n columns: [\n {\n name: 'order_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'customer_id',\n type: IcebergType.LONG,\n required: true,\n id: 2,\n },\n {\n name: 'order_amount',\n type: IcebergType.decimal(12, 2),\n required: true,\n id: 3,\n },\n {\n name: 'currency',\n type: IcebergType.STRING,\n required: true,\n id: 4,\n },\n {\n name: 'placed_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 5,\n },\n {\n name: 'tags',\n type: IcebergType.list({\n element: IcebergType.STRING,\n elementId: 9,\n }),\n id: 6,\n },\n {\n name: 'shipping_address',\n type: IcebergType.struct([\n {\n id: 10,\n name: 'line1',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 11,\n name: 'city',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 12,\n name: 'country',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 13,\n name: 'postal_code',\n type: IcebergType.STRING,\n },\n ]),\n id: 7,\n },\n {\n name: 'metadata',\n type: IcebergType.map({\n key: IcebergType.STRING,\n keyId: 14,\n value: IcebergType.STRING,\n valueId: 15,\n }),\n id: 8,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n partitionSpec: [\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1000,\n },\n {\n sourceColumn: 'customer_id',\n transform: IcebergPartitionTransform.bucket(16),\n fieldId: 1001,\n },\n ],\n sortOrder: [\n {\n sourceColumn: 'placed_at',\n direction: IcebergSortDirection.ASC,\n nullOrder: IcebergNullOrder.NULLS_LAST,\n },\n {\n sourceColumn: 'order_id',\n direction: IcebergSortDirection.ASC,\n },\n ],\n identifierFieldNames: [\n 'order_id',\n ],\n dataFormat: IcebergDataFormat.PARQUET,\n formatVersion: IcebergFormatVersion.V2,\n tableProperties: {\n 'write.parquet.compression-codec': 'zstd',\n 'write.delete.mode': 'merge-on-read',\n 'write.update.mode': 'merge-on-read',\n 'write.merge.mode': 'merge-on-read',\n 'write.target-file-size-bytes': '134217728',\n 'history.expire.min-snapshots-to-keep': '5',\n 'gc.enabled': 'true',\n },\n removalPolicy: RemovalPolicy.DESTROY,\n});\n```\n\nEach partition field also pins a `fieldId`: an integer of 1000 or\nabove (the Iceberg spec reserves lower ids for schema columns), unique\nacross the table's entire partition-spec history. The discipline is\nthe inverse of the column-id rule. A column keeps its id forever; a\npartition field keeps its id only while it stays untouched. Change\nanything about a field — swap `month` for `day`, point it at a\ndifferent source column — and it becomes a *new* field that needs a\nfresh, never-before-used id, while the old id retires with the old\nspec. Reusing a retired id (or reassigning an id to a different field)\nproduces a table that Athena still reads but Spark and Trino reject\nwith `ValidationException: Conflicting partition fields`.\n\n`identifierFieldNames` lists the columns that together identify a row\nand renders as the schema's `identifier-field-ids`. Each one must be a\n`required: true` column (Glue rejects an optional one), and `cdk synth`\nrejects an identifier column of type float, double, list, map, struct,\nunknown, variant, geometry or geography. The construct prefixes each\nidentifier column's `doc` with `(PK) `, so a `doc` you write must not\nstart with `(PK) `; see [Evolving schema and\npartitions](#evolving-schema-and-partitions).\n\nThe resulting Iceberg `metadata.json` for this table contains every\nfeature you set:\n\n```json\n{\n \"format-version\": 2,\n \"table-uuid\": \"39a948f9-...\",\n \"current-schema-id\": 0,\n \"schemas\": [\n {\n \"schema-id\": 0,\n \"identifier-field-ids\": [1],\n \"fields\": [\n { \"id\": 1, \"name\": \"order_id\", \"required\": true, \"type\": \"long\", \"doc\": \"(PK) \" },\n { \"id\": 2, \"name\": \"customer_id\", \"required\": true, \"type\": \"long\" },\n { \"id\": 3, \"name\": \"order_amount\", \"required\": true, \"type\": \"decimal(12, 2)\" },\n { \"id\": 4, \"name\": \"currency\", \"required\": true, \"type\": \"string\" },\n { \"id\": 5, \"name\": \"placed_at\", \"required\": true, \"type\": \"timestamptz\" },\n { \"id\": 6, \"name\": \"tags\", \"required\": false,\n \"type\": { \"type\": \"list\", \"element-id\": 9, \"element\": \"string\", \"element-required\": false } },\n { \"id\": 7, \"name\": \"shipping_address\", \"required\": false,\n \"type\": { \"type\": \"struct\", \"fields\": [\n { \"id\": 10, \"name\": \"line1\", \"required\": true, \"type\": \"string\" },\n { \"id\": 11, \"name\": \"city\", \"required\": true, \"type\": \"string\" },\n { \"id\": 12, \"name\": \"country\", \"required\": true, \"type\": \"string\" },\n { \"id\": 13, \"name\": \"postal_code\", \"required\": false, \"type\": \"string\" }\n ] } },\n { \"id\": 8, \"name\": \"metadata\", \"required\": false,\n \"type\": { \"type\": \"map\", \"key-id\": 14, \"key\": \"string\", \"value-id\": 15,\n \"value-required\": false, \"value\": \"string\" } }\n ]\n }\n ],\n \"partition-specs\": [\n { \"spec-id\": 0, \"fields\": [\n { \"name\": \"placed_at_day\", \"transform\": \"day\", \"source-id\": 5, \"field-id\": 1000 },\n { \"name\": \"customer_id_bucket\", \"transform\": \"bucket[16]\", \"source-id\": 2, \"field-id\": 1001 }\n ]}\n ],\n \"sort-orders\": [\n { \"order-id\": 1, \"fields\": [\n { \"transform\": \"identity\", \"source-id\": 5, \"direction\": \"asc\", \"null-order\": \"nulls-last\" },\n { \"transform\": \"identity\", \"source-id\": 1, \"direction\": \"asc\", \"null-order\": \"nulls-last\" }\n ]}\n ],\n \"properties\": {\n \"format-version\": \"2\",\n \"write.format.default\": \"parquet\",\n \"write.parquet.compression-codec\": \"zstd\",\n \"write.merge.mode\": \"merge-on-read\",\n \"write.update.mode\": \"merge-on-read\",\n \"write.delete.mode\": \"merge-on-read\",\n \"write.target-file-size-bytes\": \"134217728\",\n \"history.expire.min-snapshots-to-keep\": \"5\",\n \"gc.enabled\": \"true\",\n \"comment\": \"Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.\"\n }\n}\n```\n\n### Format version 3\n\nSet `formatVersion: IcebergFormatVersion.V3` to create a v3 table. The\nsix v3 types (below) are only accepted on a V3 table, at any nesting\ndepth: inside a struct, as a list element, or as a map key or value.\nSynth fails on a V1 or V2 table that uses one.\n\n```typescript\nimport {\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'Events', {\n database: db,\n tableName: 'events',\n location: `s3://${bucket.bucketName}/analytics/events/`,\n columns: [\n { name: 'occurred_at', type: IcebergType.TIMESTAMP_NS, id: 1 },\n { name: 'payload', type: IcebergType.VARIANT, id: 2 },\n {\n name: 'origin',\n type: IcebergType.geography('OGC:CRS84', IcebergEdgeAlgorithm.SPHERICAL),\n id: 3,\n },\n ],\n formatVersion: IcebergFormatVersion.V3,\n});\n```\n\nThe same table in Python:\n\n```python\nfrom cdk_glue_iceberg_table import (\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n)\n\nIcebergTable(self, \"Events\",\n database=db,\n table_name=\"events\",\n location=f\"s3://{bucket.bucket_name}/analytics/events/\",\n columns=[\n {\"name\": \"occurred_at\", \"type\": IcebergType.TIMESTAMP_NS, \"id\": 1},\n {\"name\": \"payload\", \"type\": IcebergType.VARIANT, \"id\": 2},\n {\n \"name\": \"origin\",\n \"type\": IcebergType.geography(\"OGC:CRS84\", IcebergEdgeAlgorithm.SPHERICAL),\n \"id\": 3,\n },\n ],\n format_version=IcebergFormatVersion.V3,\n)\n```\n\nThis table has a `timestamp_ns` column; see [Known limitations](#known-limitations)\nbefore reading it from upstream Spark.\n\nThe v3 types, with the rules `cdk synth` enforces:\n\n| Type | Factory | Rendered string | Partition and identifier rules |\n| --- | --- | --- | --- |\n| unknown | `IcebergType.UNKNOWN` | `unknown` | Must be optional, as a column, list element, or map value; never a map key. Partition source only with `void`. Not an identifier field. |\n| timestamp with nanoseconds | `IcebergType.TIMESTAMP_NS` | `timestamp_ns` | `identity`, `year`, `month`, `day`, `hour`, `bucket(N)`, `void`; not `truncate(W)`. Allowed as an identifier field. |\n| timestamptz with nanoseconds | `IcebergType.TIMESTAMPTZ_NS` | `timestamptz_ns` | Same as `timestamp_ns`. |\n| variant | `IcebergType.VARIANT` | `variant` | Never a partition or sort source, `void` included. Not an identifier field. |\n| geometry | `IcebergType.geometry(crs?)` | `geometry`, or `geometry(<crs>)` when a CRS is given | Partition source only with `void`. Not an identifier field. |\n| geography | `IcebergType.geography(crs?, algorithm?)` | `geography`, `geography(<crs>)`, or `geography(<crs>, <algorithm>)` when an algorithm is given, such as `geography(OGC:CRS84, spherical)` | Same as `geometry`. |\n\nA geometry or geography `crs` is an identifier such as `OGC:CRS84` or\n`EPSG:3857`; whitespace, parentheses, braces, and commas are rejected.\nWhen `algorithm` is set without a `crs`, the rendered string spells out\n`OGC:CRS84`.\n\nWhich engines read a v3 table created by the construct:\n\n| Engine | v3 tables | Notes |\n| --- | --- | --- |\n| Spark 4.1.1 + Iceberg 1.11.0 (the integration-test runner) | Yes | Reads and writes tables the construct creates, including deletion vectors, row lineage, `variant` and `unknown`, except tables with a `geometry`, `geography` or nanosecond-timestamp column ([Known limitations](#known-limitations)). |\n| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists \"Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types\". The integration suites run on Glue 6.0's Spark 4.1.1 and Iceberg 1.11.0 (`scripts/spark/spark-env.sh`); the Glue job runtime itself is not exercised. |\n| Amazon EMR Spark 8.1.0 (Spark 4.1.1, Iceberg 1.11.0-amzn-0) | Documented | The [emr-spark-8.1.0 release notes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html) state \"Amazon EMR Spark 8.1.0 extends Iceberg v3 spec support for Spark workloads\". The integration suites do not run on EMR 8.1. |\n| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states \"Athena only creates and operates on Iceberg v2 tables.\" The v2 integration suites also run through Athena (see [Does it work with Athena and Lake Formation?](#does-it-work-with-athena-and-lake-formation)); no v3 table is queried through Athena. |\n\n### Granting access\n\n```typescript\ntable.grantRead(role); // Glue read + S3 read on the table's prefix\ntable.grantWrite(role); // Glue write + S3 write\ntable.grantReadWrite(role);\n```\n\nThe `grant*` helpers issue **IAM grants only**. Under Lake Formation\nyou still add the matching `SELECT` / `INSERT` / `DELETE` LF grants on\ntop of the construct's IAM grants for Athena queries to succeed.\n\n### Importing an existing table\n\n```typescript\nconst existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {\n database: db,\n tableName: 'orders',\n location: 's3://my-bucket/analytics/orders/',\n});\nexisting.grantRead(role);\n```\n\n### Evolving schema and partitions\n\nChange the `columns` array (or `partitionSpec`) and run `cdk deploy`\nagain. The construct passes the new schema to Glue's `UpdateTable`,\nwhich writes a new `metadata.json` with a new `schema-id`; existing\ndata files stay readable because each column's `id` is pinned to the\nid the table holds and never reused. Adds, renames (same `id`, new\n`name`), and drops all flow through `cdk deploy` alone — no\nout-of-band SQL DDL.\n\nNested fields evolve the same way. Each struct field, list element,\nmap key, and map value keeps the id you gave it (the rule is under\n[Using `IcebergTable`](#using-icebergtable)), so adding a column or a\nstruct field anywhere in the schema leaves every other id unchanged. A\nnew nested field takes an id no field of the table has ever used:\n\n```typescript\ntype: IcebergType.struct([\n { id: 10, name: 'line1', type: IcebergType.STRING },\n { id: 16, name: 'line2', type: IcebergType.STRING }, // new field, never-used id\n { id: 11, name: 'city', type: IcebergType.STRING },\n]),\n```\n\nSuch an id is out of creation order, which is expected once a table\nhas evolved: creation order applies only to the ids a table is created\nwith (see [Using `IcebergTable`](#using-icebergtable)).\n\nUpgrading from a version before 0.12.0, which numbered nested fields\nitself: declare the ids the table's current `metadata.json` holds,\ntop-level and nested (each field `id`, `element-id`, `key-id` and\n`value-id`), or the next deploy renumbers the fields that differ and\nexisting data files read under the wrong field. A nested type object\nshared by two columns now fails synth with a duplicate id; give each\ncolumn its own nested type with its own ids. The list element and map\nvalue nullability default also changed: `elementRequired` and\n`valueRequired` now default to `false`. Earlier versions made list\nelements required by default and map values required unless the third\nargument of `map` was `false`; set `elementRequired: true` or\n`valueRequired: true` wherever the deployed schema has\n`\"element-required\": true` or `\"value-required\": true`.\n\nGlue's `UpdateTable` reuses an earlier schema whose fields (name,\ntype, `required`, `doc`) match the request, together with that\nschema's `identifier-field-ids`. The construct therefore prefixes the\n`doc` of every identifier column with `(PK) ` (just `(PK) ` when the\ncolumn has no `doc`), shown as the column comment, so every\nidentifier change is a field change. On a table whose schema history\nwas written with the marker, identifier changes flow through `cdk\ndeploy` like column changes.\n\nPartition evolution flows the same way, with one extra rule on\n`fieldId`: a changed partition field is a new partition field. To move\na table from `month(placed_at)` to `day(placed_at)`, drop the month\nentry and add a day entry under a fresh `fieldId` — never recycle the\nmonth field's id:\n\n```typescript\npartitionSpec: [\n // was: { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.MONTH, fieldId: 1000 },\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1001,\n },\n],\n```\n\nOld data files written under the month spec stay readable; new writes\npartition by day. Reusing `1000` for the day field instead corrupts\nthe spec history for strict engines (Spark, Trino) while Athena keeps\nworking — the failure only surfaces at Spark runtime.\n\n### Inserting and querying\n\n```sql\n-- INSERT into the orders table\nINSERT INTO sample_database.orders VALUES\n (1001, 5001, DECIMAL '149.99', 'USD',\n TIMESTAMP '2026-05-20 09:15:00 UTC',\n ARRAY['holiday-promo','first-order'],\n CAST(ROW('1 Infinite Loop','Cupertino','US','95014')\n AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),\n MAP(ARRAY['channel','utm'], ARRAY['web','google'])),\n -- ... more rows\n;\n\n-- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)\nDELETE FROM sample_database.orders WHERE order_id = 1003;\n\n-- merge-on-read UPDATE\nUPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;\n\n-- SELECT\nSELECT customer_id, SUM(order_amount) AS total\n FROM sample_database.orders\n GROUP BY 1\n ORDER BY 2 DESC;\n```\n\n## Two footguns the construct prevents\n\n### Footgun #1 — schema under `storageDescriptor.columns`\n\nThe CREATE succeeds but the first UPDATE silently strips\n`table_type=ICEBERG` from the table's Glue parameters, and Athena\nqueries after that fail with `HIVE_UNSUPPORTED_FORMAT`.\n\n```typescript\n// DON'T DO THIS — what most StackOverflow / re:Post examples show\nnew CfnTable(this, 'OrdersBad', {\n catalogId: this.account,\n databaseName: 'analytics',\n tableInput: {\n name: 'orders',\n tableType: 'EXTERNAL_TABLE',\n parameters: {\n table_type: 'ICEBERG',\n },\n storageDescriptor: {\n location: 's3://.../orders/',\n columns: [\n /* ... */\n ],\n },\n },\n openTableFormatInput: {\n icebergInput: {\n metadataOperation: 'CREATE',\n version: '2',\n },\n },\n});\n```\n\n`IcebergTable` instead always emits schema/partitions/sort/properties\nunder `openTableFormatInput.icebergInput.icebergTableInput`, never\nunder `storageDescriptor`.\n\n### Footgun #2 — `tableInput` co-present with `openTableFormatInput`\n\nEven setting just `tableInput: { name: 'foo' }` next to\n`openTableFormatInput` returns\n`\"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput\"`.\nThe construct never emits `tableInput`; the table-level comment goes\ninto `tableProperties['comment']`, which lives inside\n`icebergTableInput.properties`.\n\n(There is a third footgun, field-id reuse after a column drop, that\nthe construct does **not** prevent. See the next section.)\n\n## Known limitations\n\n- **Creation order is not checked.** The construct cannot tell at synth whether a table is new or already deployed, so it accepts a new table whose ids are not in creation order (the rule is under [Using `IcebergTable`](#using-icebergtable)). When the first update of such a table changes the id of a column an identity partition is sourced from, Glue rejects the update with `Cannot create identity partition sourced from different field in schema` and the CloudFormation rollback fails the same way, leaving the stack in `UPDATE_ROLLBACK_FAILED`.\n- **Field-id reuse is not detected across deploys.** This applies to nested ids (struct field, list element, map key and value) as well as column ids. If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the \"never reuse a retired id\" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate field id N` validator), but it doesn't compare against the live table state. The safe workflow is to treat dropped ids as retired forever.\n- **Partition field-id reuse is not detected across deploys either.** The same live-state blindness applies to partition `fieldId`s: the construct validates them within one deploy (integer >= 1000, unique in the spec), but it cannot tell that the id you pinned already appeared in an earlier spec of the deployed table. The invariant partition ids must uphold is stricter than the column one — unique across the table's **entire spec history**, so a changed field (different transform or source column) must take a fresh id rather than keep its old one. Violate it and strict engines (Spark, Trino) reject the table with `Conflicting partition fields` while Athena keeps answering, masking the break until Spark runtime. The safe workflow: treat every id that has ever appeared in any spec as retired, and allocate new fields the next never-used id. A partition change needs a new source column or transform; Glue ignores a `fieldId`-only change.\n- **`last-partition-id` is written by Glue, not by the construct.** The construct supplies the spec's field ids; Glue's `UpdateTable` computes the table's `last-partition-id` counter when it writes the new metadata. The repo's integration test asserts the counter keeps up with the highest supplied field id after a partition evolution, but engines that later evolve the spec themselves (e.g. Spark `ALTER TABLE … ADD PARTITION FIELD`) allocate from that counter — if you mix CDK-managed and engine-managed partition evolution, verify the counter in `metadata.json` before trusting engine-side changes.\n- **Removing the last identifier from a table created with identifiers before 0.11.0 can bring the old ids back.** That table's history holds an unmarked schema with ids, and Glue reuses it when the rendered fields match it again. Change a column `doc` in the same deploy that removes the last identifier.\n- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.\n- **Format version is immutable after CREATE.** Glue ignores a V2 to V3 `formatVersion` change, though `properties.format-version` then reads `3`. Keep `formatVersion` at the table's real version, or destroy + recreate the table to change it. An engine can upgrade a v2 table to v3 in place with `ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3')`; after that, set `formatVersion` to `V3` so the next deploy declares the version the table already has.\n- **`merge-on-read` requires v2 or later.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.\n- **Column defaults are not available.** Iceberg v3 `initial-default` and `write-default` cannot be declared through CloudFormation: the `AWS::Glue::Table` resource schema has no `InitialDefault` or `WriteDefault` property. Revisit when that schema gains the two properties.\n- **Nanosecond timestamps are catalog-only for Spark 4.1.1 with Iceberg 1.11.0.** In [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag, `primitive()` has no case for the `TIMESTAMP_NANO` type id, so a table with a `timestamp_ns` or `timestamptz_ns` column reaches the `Cannot convert unsupported type to Spark` error. The integration suite asserts these columns from `metadata.json` only.\n- **Geometry and geography columns are catalog-only for Spark 4.1.1 with Iceberg 1.11.0, the versions AWS Glue 6.0 ships.** The first Spark read of a table holding a `geography` and a `geometry` column failed with `UnsupportedOperationException: Cannot convert unsupported type to Spark: geography(OGC:CRS84, spherical)` on those versions. [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag has no conversion for `geometry` either. The integration suite deploys these columns on a table no Spark job opens and asserts them from `metadata.json` only.\n- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.\n- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.\n\n## FAQ\n\n### How do I create an Apache Iceberg table in AWS CDK?\n\nInstall `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a\ndatabase, a column list, and an S3 `location` — see [Use](#use).\n`cdk deploy` creates the Glue Data Catalog table and writes the\nIceberg `metadata.json` under your S3 prefix. No custom resource or\nLambda is involved; it is a plain `AWS::Glue::Table` in your\nCloudFormation template.\n\n### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?\n\nChange the `columns` array and run `cdk deploy` again. The construct\npasses the new schema to Glue's `UpdateTable`, which writes a new\n`metadata.json` with a new `schema-id`; existing data files stay\nreadable because each column's `id` is pinned to the id the table\nholds and never reused.\nPartition changes work the same way, except a changed partition field\ntakes a fresh `fieldId` instead of keeping its old one. See [Evolving\nschema and partitions](#evolving-schema-and-partitions).\n\n### Does CloudFormation support Iceberg tables natively?\n\nYes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,\ndocumented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).\nThe catch is that the raw shape corrupts the table on the first\n`Update` if you place schema under `storageDescriptor.columns` or set\n`tableInput` alongside `openTableFormatInput`. This construct only ever\nemits the safe shape and gives you no way to express the unsafe ones.\nSee [Two footguns the construct prevents](#two-footguns-the-construct-prevents).\n\n### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?\n\nA raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON\nand own both silent-corruption footguns. `cdk-glue-iceberg-table` gives\nyou a typed `IcebergType` / `IcebergPartitionTransform` API,\nsynth-time validation (partition transforms checked against column\ntypes, format/version mismatches caught before deploy), pinned field\nIDs for safe evolution (declared in creation order for a new table),\nand `grantRead` / `grantWrite` helpers. See\n[How it compares](#how-it-compares).\n\n### How do I create a partitioned Iceberg table in CDK?\n\nPass a `partitionSpec` of `IcebergPartitionTransform` entries —\n`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,\nor `void` — each pinned to a `fieldId` of 1000 or above. Transforms are\nvalidated against their source column's type at synth time. See\n[Using `IcebergTable`](#using-icebergtable).\n\n### Does it work with Athena and Lake Formation?\n\nYes for v2 tables, no for v3 (see the engine matrix in [Format\nversion 3](#format-version-3)). The demo app in the\n[`arceus`](https://github.com/ksco92/arceus) repo registers its tables\nwith Lake Formation, and its v2 integration suites query them through\nAthena as well as Spark, on the same tables:\n\n- merge-on-read `INSERT`, `UPDATE`, `DELETE` and `MERGE INTO`, time\n travel (`FOR VERSION AS OF` / `FOR TIMESTAMP AS OF`), `OPTIMIZE ...\n REWRITE DATA USING BIN_PACK`, and `VACUUM`;\n- a read after every schema, partition, identifier and nested-field\n evolution step;\n- the `(PK) ` comment on identifier columns, read from\n `information_schema.columns`;\n- rows written by Spark read through Athena, and rows written by Athena\n read through Spark.\n\nThe suites grant the querying principal `SELECT`, `INSERT`, `DELETE`,\n`ALTER` and `DESCRIBE` through Lake Formation. The construct's `grant*`\nhelpers issue IAM grants only; under Lake Formation you add the\nmatching LF grants. See [Granting access](#granting-access).\n\n### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?\n\nYes — set `formatVersion: IcebergFormatVersion.V2` and the\n`write.{delete,update,merge}.mode = merge-on-read` table properties.\nThe construct rejects merge-on-read on a v1 table at synth time.\n\n### Can I create an Iceberg v3 table?\n\nYes. Set `formatVersion: IcebergFormatVersion.V3`; see [Format\nversion 3](#format-version-3) for the types, the engine matrix and the\nlimits.\n\n## Contributing\n\nDevelopment, the monorepo layout, and the demo app live in [CONTRIBUTING.md](https://github.com/ksco92/arceus/blob/main/CONTRIBUTING.md).\n\n## License\n\n[MIT](https://github.com/ksco92/arceus/blob/main/LICENSE).\n"
|
|
9755
9755
|
},
|
|
9756
9756
|
"repository": {
|
|
9757
9757
|
"directory": "packages/cdk-glue-iceberg-table",
|
|
@@ -13232,6 +13232,6 @@
|
|
|
13232
13232
|
"symbolId": "lib/iceberg/iceberg-type:IcebergTypeOptions"
|
|
13233
13233
|
}
|
|
13234
13234
|
},
|
|
13235
|
-
"version": "0.14.
|
|
13236
|
-
"fingerprint": "
|
|
13235
|
+
"version": "0.14.3",
|
|
13236
|
+
"fingerprint": "Oiz0+MdWjiKjR1MAzDn6j4gbdYikHdF83JuuhbYBR0I="
|
|
13237
13237
|
}
|
package/README.md
CHANGED
|
@@ -586,10 +586,10 @@ Which engines read a v3 table created by the construct:
|
|
|
586
586
|
|
|
587
587
|
| Engine | v3 tables | Notes |
|
|
588
588
|
| --- | --- | --- |
|
|
589
|
-
| Spark 4.1 + Iceberg 1.
|
|
590
|
-
| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists "Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types". The integration suites
|
|
589
|
+
| Spark 4.1.1 + Iceberg 1.11.0 (the integration-test runner) | Yes | Reads and writes tables the construct creates, including deletion vectors, row lineage, `variant` and `unknown`, except tables with a `geometry`, `geography` or nanosecond-timestamp column ([Known limitations](#known-limitations)). |
|
|
590
|
+
| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists "Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types". The integration suites run on Glue 6.0's Spark 4.1.1 and Iceberg 1.11.0 (`scripts/spark/spark-env.sh`); the Glue job runtime itself is not exercised. |
|
|
591
591
|
| Amazon EMR Spark 8.1.0 (Spark 4.1.1, Iceberg 1.11.0-amzn-0) | Documented | The [emr-spark-8.1.0 release notes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html) state "Amazon EMR Spark 8.1.0 extends Iceberg v3 spec support for Spark workloads". The integration suites do not run on EMR 8.1. |
|
|
592
|
-
| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states "Athena only creates and operates on Iceberg v2 tables." |
|
|
592
|
+
| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states "Athena only creates and operates on Iceberg v2 tables." The v2 integration suites also run through Athena (see [Does it work with Athena and Lake Formation?](#does-it-work-with-athena-and-lake-formation)); no v3 table is queried through Athena. |
|
|
593
593
|
|
|
594
594
|
### Granting access
|
|
595
595
|
|
|
@@ -776,8 +776,8 @@ the construct does **not** prevent. See the next section.)
|
|
|
776
776
|
- **Format version is immutable after CREATE.** Glue ignores a V2 to V3 `formatVersion` change, though `properties.format-version` then reads `3`. Keep `formatVersion` at the table's real version, or destroy + recreate the table to change it. An engine can upgrade a v2 table to v3 in place with `ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3')`; after that, set `formatVersion` to `V3` so the next deploy declares the version the table already has.
|
|
777
777
|
- **`merge-on-read` requires v2 or later.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.
|
|
778
778
|
- **Column defaults are not available.** Iceberg v3 `initial-default` and `write-default` cannot be declared through CloudFormation: the `AWS::Glue::Table` resource schema has no `InitialDefault` or `WriteDefault` property. Revisit when that schema gains the two properties.
|
|
779
|
-
- **Nanosecond timestamps are catalog-only for
|
|
780
|
-
- **Geometry
|
|
779
|
+
- **Nanosecond timestamps are catalog-only for Spark 4.1.1 with Iceberg 1.11.0.** In [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag, `primitive()` has no case for the `TIMESTAMP_NANO` type id, so a table with a `timestamp_ns` or `timestamptz_ns` column reaches the `Cannot convert unsupported type to Spark` error. The integration suite asserts these columns from `metadata.json` only.
|
|
780
|
+
- **Geometry and geography columns are catalog-only for Spark 4.1.1 with Iceberg 1.11.0, the versions AWS Glue 6.0 ships.** The first Spark read of a table holding a `geography` and a `geometry` column failed with `UnsupportedOperationException: Cannot convert unsupported type to Spark: geography(OGC:CRS84, spherical)` on those versions. [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag has no conversion for `geometry` either. The integration suite deploys these columns on a table no Spark job opens and asserts them from `metadata.json` only.
|
|
781
781
|
- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE WRITE ORDERED BY`, `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.
|
|
782
782
|
- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.
|
|
783
783
|
|
|
@@ -835,14 +835,25 @@ validated against their source column's type at synth time. See
|
|
|
835
835
|
### Does it work with Athena and Lake Formation?
|
|
836
836
|
|
|
837
837
|
Yes for v2 tables, no for v3 (see the engine matrix in [Format
|
|
838
|
-
version 3](#format-version-3)).
|
|
839
|
-
including merge-on-read `INSERT` / `UPDATE` / `DELETE` / `MERGE`, time
|
|
840
|
-
travel, `OPTIMIZE`, and `VACUUM`. The demo app in the
|
|
838
|
+
version 3](#format-version-3)). The demo app in the
|
|
841
839
|
[`arceus`](https://github.com/ksco92/arceus) repo registers its tables
|
|
842
|
-
with Lake Formation and
|
|
843
|
-
|
|
844
|
-
|
|
845
|
-
`INSERT`
|
|
840
|
+
with Lake Formation, and its v2 integration suites query them through
|
|
841
|
+
Athena as well as Spark, on the same tables:
|
|
842
|
+
|
|
843
|
+
- merge-on-read `INSERT`, `UPDATE`, `DELETE` and `MERGE INTO`, time
|
|
844
|
+
travel (`FOR VERSION AS OF` / `FOR TIMESTAMP AS OF`), `OPTIMIZE ...
|
|
845
|
+
REWRITE DATA USING BIN_PACK`, and `VACUUM`;
|
|
846
|
+
- a read after every schema, partition, identifier and nested-field
|
|
847
|
+
evolution step;
|
|
848
|
+
- the `(PK) ` comment on identifier columns, read from
|
|
849
|
+
`information_schema.columns`;
|
|
850
|
+
- rows written by Spark read through Athena, and rows written by Athena
|
|
851
|
+
read through Spark.
|
|
852
|
+
|
|
853
|
+
The suites grant the querying principal `SELECT`, `INSERT`, `DELETE`,
|
|
854
|
+
`ALTER` and `DESCRIBE` through Lake Formation. The construct's `grant*`
|
|
855
|
+
helpers issue IAM grants only; under Lake Formation you add the
|
|
856
|
+
matching LF grants. See [Granting access](#granting-access).
|
|
846
857
|
|
|
847
858
|
### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?
|
|
848
859
|
|
|
@@ -26,7 +26,7 @@ var IcebergPartitionTransformKind;
|
|
|
26
26
|
* it is legal to apply against.
|
|
27
27
|
*/
|
|
28
28
|
class IcebergPartitionTransform {
|
|
29
|
-
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergPartitionTransform", version: "0.14.
|
|
29
|
+
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergPartitionTransform", version: "0.14.3" };
|
|
30
30
|
/** `identity` — source value unmodified. Legal on any column type. */
|
|
31
31
|
static IDENTITY = new IcebergPartitionTransform({
|
|
32
32
|
kind: IcebergPartitionTransformKind.IDENTITY,
|
|
@@ -36,7 +36,7 @@ var IcebergDataFormat;
|
|
|
36
36
|
})(IcebergDataFormat || (exports.IcebergDataFormat = IcebergDataFormat = {}));
|
|
37
37
|
/** Canonical Iceberg property keys the L2 understands. */
|
|
38
38
|
class IcebergPropertyKeys {
|
|
39
|
-
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergPropertyKeys", version: "0.14.
|
|
39
|
+
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergPropertyKeys", version: "0.14.3" };
|
|
40
40
|
/** `format-version`. */
|
|
41
41
|
static FORMAT_VERSION = 'format-version';
|
|
42
42
|
/** `write.format.default`. */
|
|
@@ -40,7 +40,7 @@ var IcebergNullOrder;
|
|
|
40
40
|
* leaves the table un-queryable in Athena).
|
|
41
41
|
*/
|
|
42
42
|
class IcebergTable extends aws_cdk_lib_1.Resource {
|
|
43
|
-
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergTable", version: "0.14.
|
|
43
|
+
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergTable", version: "0.14.3" };
|
|
44
44
|
/** Glue ARN of the table. */
|
|
45
45
|
tableArn;
|
|
46
46
|
/** Table name as it appears in Glue / Athena. */
|
|
@@ -56,7 +56,7 @@ const DEFAULT_GEO_CRS = 'OGC:CRS84';
|
|
|
56
56
|
* primitives, decimals, fixed, geospatial types, lists, maps, and structs.
|
|
57
57
|
*/
|
|
58
58
|
class IcebergType {
|
|
59
|
-
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergType", version: "0.14.
|
|
59
|
+
static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-glue-iceberg-table.IcebergType", version: "0.14.3" };
|
|
60
60
|
/// Primitives — values from https://iceberg.apache.org/spec/#schemas-and-data-types
|
|
61
61
|
/** Boolean. */
|
|
62
62
|
static BOOLEAN = new IcebergType({
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cdk-glue-iceberg-table",
|
|
3
|
-
"version": "0.14.
|
|
3
|
+
"version": "0.14.3",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"description": "AWS CDK L2 construct to create and evolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and partition changes that survive CloudFormation updates, via cdk deploy.",
|
|
6
6
|
"stability": "experimental",
|