cdk-glue-iceberg-table 0.17.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.jsii +65 -65
- package/README.md +50 -6
- package/dist/lib/iceberg/iceberg-partition-transform.js +1 -1
- package/dist/lib/iceberg/iceberg-schema-history.d.ts +3 -2
- package/dist/lib/iceberg/iceberg-schema-history.js +25 -4
- package/dist/lib/iceberg/iceberg-table-properties.js +1 -1
- package/dist/lib/iceberg/iceberg-table.d.ts +3 -4
- package/dist/lib/iceberg/iceberg-table.js +2 -2
- package/dist/lib/iceberg/iceberg-type.js +1 -1
- package/package.json +1 -1
package/.jsii
CHANGED
|
@@ -9751,7 +9751,7 @@
|
|
|
9751
9751
|
},
|
|
9752
9752
|
"name": "cdk-glue-iceberg-table",
|
|
9753
9753
|
"readme": {
|
|
9754
|
-
"markdown": "# cdk-glue-iceberg-table\n\n[](https://github.com/ksco92/arceus/actions/workflows/ci.yml)\n[](https://codecov.io/gh/ksco92/arceus)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://pypi.org/project/cdk-glue-iceberg-table/)\n[](https://constructs.dev/packages/cdk-glue-iceberg-table)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://github.com/ksco92/arceus/blob/main/LICENSE)\n\n**`cdk-glue-iceberg-table` is an AWS CDK L2 construct to create and\nevolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and\npartition changes that survive CloudFormation updates, via cdk\ndeploy.** It supports Iceberg format version 3 (`variant`, nanosecond\ntimestamps, `geometry` / `geography`, `unknown`). It emits the exact\n`AWS::Glue::Table` / `OpenTableFormatInput` shape, so a single `cdk\ndeploy` creates, evolves, and destroys a table like any other CDK\nresource. No custom resource, no Lambda, no \"`cdk deploy` and then run\nthis SQL by hand\" two-step. Table changes (new columns, renames, drops,\nnew partition fields) land as a reviewed diff in a pull request and\napply through `cdk deploy`.\n\nStatus (October 2026): published on npm (TypeScript/JavaScript) and\nPyPI (Python), pre-1.0, with a public surface guarded on every PR by two\nconsumer jobs (`e2e-consumer` compiles and synthesizes an app against a\npinned published version, `packed-consumer` does the same against the PR's\nown `npm pack` tarball) plus real-AWS integration suites, including ones\nfor format version 3. The multi-language packages are generated\nfrom one TypeScript source with [jsii](https://aws.github.io/jsii/)\nand indexed on [Construct\nHub](https://constructs.dev/packages/cdk-glue-iceberg-table).\nDeveloped in the [`ksco92/arceus`](https://github.com/ksco92/arceus)\nmonorepo, which also holds a CDK demo app that dogfoods the construct\nagainst a real AWS account.\n\n### Why this exists\n\nAWS documents the CloudFormation shape for Iceberg tables ([AWS Big\nData blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),\nbut raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the\n*first* `Update` silently strips `table_type=ICEBERG`, after which\nevery Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The\nmotivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);\n[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)\ndocuments the only working shape and the silent-corruption traps you\nhit by getting it slightly wrong. This construct implements that\nshape, refuses to emit the unsafe alternatives, and pins Iceberg field\nIDs (declared [in creation order](#using-icebergtable) for a new\ntable) so schema evolution never orphans existing data. It was the\nbasis of an upstream proposal to land an `IcebergTable` in\n`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988)),\nclosed unmerged in September 2026.\n\n### How it compares\n\n| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |\n| --- | --- | --- | --- | --- |\n| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |\n| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |\n| Spark SQL DDL, or Athena SQL DDL, at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |\n| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |\n\n## Install\n\nTypeScript / JavaScript (npm):\n\n```bash\nnpm install cdk-glue-iceberg-table\n```\n\nPython (PyPI):\n\n```bash\npip install cdk-glue-iceberg-table\n```\n\nPeer dependencies (your CDK app must already have these). This\nrelease requires `aws-cdk-lib` 2.273.0 or newer and `constructs`\n10.8.1 or newer, plus Node.js 22.14 or newer for both languages (the\nPython package runs under Node.js through jsii).\n\nFor TypeScript / JavaScript:\n\n```bash\nnpm install aws-cdk-lib@^2.273.0 constructs@^10.8.1\n```\n\nFor Python:\n\n```bash\npip install 'aws-cdk-lib>=2.273.0' 'constructs>=10.8.1'\n```\n\n## Use\n\n### TypeScript\n\n```ts\nimport { Bucket } from 'aws-cdk-lib/aws-s3';\nimport { Database } from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\nconst db = new Database(this, 'Db', { databaseName: 'analytics' });\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n columns: [\n { name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },\n { name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },\n ],\n partitionSpec: [\n { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY, fieldId: 1000 },\n { sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16), fieldId: 1001 },\n ],\n identifierFieldNames: ['order_id'],\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\n### Python\n\nThe same table in Python — the API mirrors TypeScript with\n`snake_case` props and `PascalCase` types:\n\n```python\nfrom aws_cdk.aws_s3 import Bucket\nfrom aws_cdk.aws_glue import Database\nfrom cdk_glue_iceberg_table import (\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n)\n\nbucket = Bucket(self, \"Warehouse\")\ndb = Database(self, \"Db\", database_name=\"analytics\")\n\nIcebergTable(self, \"OrdersTable\",\n database=db,\n table_name=\"orders\",\n location=f\"s3://{bucket.bucket_name}/analytics/orders/\",\n columns=[\n {\"name\": \"order_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 1},\n {\"name\": \"customer_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 2},\n {\"name\": \"placed_at\", \"type\": IcebergType.TIMESTAMPTZ, \"required\": True, \"id\": 3},\n ],\n partition_spec=[\n {\"source_column\": \"placed_at\", \"transform\": IcebergPartitionTransform.DAY, \"field_id\": 1000},\n {\"source_column\": \"customer_id\", \"transform\": IcebergPartitionTransform.bucket(16), \"field_id\": 1001},\n ],\n identifier_field_names=[\"order_id\"],\n format_version=IcebergFormatVersion.V2,\n)\n```\n\nConsumer-facing reference sections:\n\n- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.\n- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.\n- [Format version 3](#format-version-3) — the six v3 types, which engines read them, and the accepted limits.\n- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.\n- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.\n- [FAQ](#faq) — common \"how do I … in CDK / CloudFormation\" questions.\n\n## Exported surface\n\nThe package entry point re-exports the main types you import from\n`cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled\nsurface lives under `dist/lib/iceberg/` after `npm install`; in Python\nthe same surface is imported from the `cdk_glue_iceberg_table` module\nafter `pip install`:\n\n- **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite` (each taking an optional `IcebergTableGrantOptions`), the `fromIcebergTableAttributes(...)` import factory, and `creationOrderFieldIds(columns)` (Python: `creation_order_field_ids`), which returns a map from field path to the id Glue gives that field in a new table; a name containing `.` or a backquote appears in backquotes.\n- **`IcebergSchemaHistory`** — `validateSchemaEvolution(previous, next)` (Python: `validate_schema_evolution`), which returns the unsafe changes from a table's history record to a definition (`IcebergTableSchema`) as `IcebergSchemaViolation`s, each carrying an `IcebergSchemaViolationKind` that names the class of unsafe change; `append`, `toJson` and `fromJson`, which record a definition in, write and read the history file (`IcebergSchemaHistoryFile`, `IcebergTableHistory`, `IcebergFieldHistory`, `IcebergPartitionFieldHistory`); and `updateFromCloudAssembly(directory)`, which records every table of a synthesized app. See [Checking changes against a schema history](#checking-changes-against-a-schema-history).\n- **`IcebergType`** — primitive types, the v3 constants `UNKNOWN`, `TIMESTAMP_NS`, `TIMESTAMPTZ_NS`, and `VARIANT`, and the `list` / `map` / `struct` / `decimal` / `fixed` / `geometry(crs?)` / `geography(crs?, algorithm?)` factories. `list` takes an `IcebergListDefinition`, `map` an `IcebergMapDefinition`, and `struct` a list of `IcebergStructFieldDefinition`, each carrying the nested field ids. Renders to the string Glue's `IcebergStructField.type` expects.\n- **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.\n- **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2/v3, required — set explicitly per table; Athena cannot query v3 tables), **`IcebergSortDirection`**, **`IcebergNullOrder`**, **`IcebergEdgeAlgorithm`** (`SPHERICAL`, `VINCENTY`, `THOMAS`, `ANDOYER`, `KARNEY`; the second argument of `IcebergType.geography`), plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).\n\n## Using `IcebergTable`\n\nA minimal table:\n\n```typescript\nimport {\n Bucket,\n} from 'aws-cdk-lib/aws-s3';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\n\nconst db = new Database(this, 'Db', {\n databaseName: 'analytics',\n});\n\nnew IcebergTable(this, 'Users', {\n database: db,\n tableName: 'users',\n columns: [\n {\n name: 'user_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'email',\n type: IcebergType.STRING,\n required: true,\n id: 2,\n },\n {\n name: 'signed_up_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 3,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/users/`,\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\nEvery top-level column must declare an `id`: a stable, unique,\npositive integer you pick once and never reuse or reassign. Because\nIceberg data files reference fields by id rather than by name, a fixed\nid, declared in creation order when the table is created (below), is\nwhat lets you add, rename, and drop columns across deploys without\ncorrupting old data — see\n[How do I evolve an Iceberg table schema](#how-do-i-evolve-an-iceberg-table-schema-add-rename-or-drop-a-column-in-cloudformation).\nOnce a column is dropped, leave its id retired; never hand it to a new\ncolumn. Nested fields follow the same rule: every struct field takes\nan `id`, every list an `elementId`, and every map a `keyId` and a\n`valueId`. All ids in a schema, top-level and nested, must be distinct\nintegers from 1 to 2147483447.\n\nA new table must declare its ids in creation order. When Glue creates\na table it does not keep the declared ids: it numbers the fields from\n1 in creation order and points the partition spec, sort order and\nidentifier fields at the new ids. An update Glue accepts stores the\ndeclared ids as given, so the first such update of a table created\nwith other ids renumbers its fields; for the update Glue rejects, see\n[Known limitations](#known-limitations). To number a new table, number\nthe top-level columns first, in order, then the nested fields of each\ncolumn in turn (a list's element; a map's key, then its value, then\nthe key's nested fields, then the value's; a struct's fields, then\ntheir nested fields). `IcebergTable.creationOrderFieldIds` returns\nthese ids by field path and ignores the ids in its input:\n\n```typescript\nconst columns = [\n { name: 'id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'tags', type: IcebergType.list({ element: IcebergType.STRING, elementId: 4 }), id: 2 },\n {\n name: 'attrs',\n type: IcebergType.map({ key: IcebergType.STRING, keyId: 5, value: IcebergType.STRING, valueId: 6 }),\n id: 3,\n },\n];\nIcebergTable.creationOrderFieldIds(columns);\n// { id: 1, tags: 2, attrs: 3, 'tags.element': 4, 'attrs.key': 5, 'attrs.value': 6 }\n```\n\nIds out of creation order are for fields a later deploy adds; see\n[Evolving schema and partitions](#evolving-schema-and-partitions).\n\nA table that exercises most of the surface (partitions, sort order,\nnested types, identifier fields, table properties, removal policy):\n\n```typescript\nimport {\n RemovalPolicy,\n} from 'aws-cdk-lib';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergDataFormat,\n IcebergFormatVersion,\n IcebergNullOrder,\n IcebergPartitionTransform,\n IcebergSortDirection,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',\n columns: [\n {\n name: 'order_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'customer_id',\n type: IcebergType.LONG,\n required: true,\n id: 2,\n },\n {\n name: 'order_amount',\n type: IcebergType.decimal(12, 2),\n required: true,\n id: 3,\n },\n {\n name: 'currency',\n type: IcebergType.STRING,\n required: true,\n id: 4,\n },\n {\n name: 'placed_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 5,\n },\n {\n name: 'tags',\n type: IcebergType.list({\n element: IcebergType.STRING,\n elementId: 9,\n }),\n id: 6,\n },\n {\n name: 'shipping_address',\n type: IcebergType.struct([\n {\n id: 10,\n name: 'line1',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 11,\n name: 'city',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 12,\n name: 'country',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 13,\n name: 'postal_code',\n type: IcebergType.STRING,\n },\n ]),\n id: 7,\n },\n {\n name: 'metadata',\n type: IcebergType.map({\n key: IcebergType.STRING,\n keyId: 14,\n value: IcebergType.STRING,\n valueId: 15,\n }),\n id: 8,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n partitionSpec: [\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1000,\n },\n {\n sourceColumn: 'customer_id',\n transform: IcebergPartitionTransform.bucket(16),\n fieldId: 1001,\n },\n ],\n sortOrder: [\n {\n sourceColumn: 'placed_at',\n direction: IcebergSortDirection.ASC,\n nullOrder: IcebergNullOrder.NULLS_LAST,\n },\n {\n sourceColumn: 'order_id',\n direction: IcebergSortDirection.ASC,\n },\n ],\n identifierFieldNames: [\n 'order_id',\n ],\n dataFormat: IcebergDataFormat.PARQUET,\n formatVersion: IcebergFormatVersion.V2,\n tableProperties: {\n 'write.parquet.compression-codec': 'zstd',\n 'write.delete.mode': 'merge-on-read',\n 'write.update.mode': 'merge-on-read',\n 'write.merge.mode': 'merge-on-read',\n 'write.target-file-size-bytes': '134217728',\n 'history.expire.min-snapshots-to-keep': '5',\n 'gc.enabled': 'true',\n },\n removalPolicy: RemovalPolicy.DESTROY,\n});\n```\n\nEach partition field also pins a `fieldId`: an integer of 1000 or\nabove (the Iceberg spec reserves lower ids for schema columns), unique\nacross the table's entire partition-spec history. The discipline is\nthe inverse of the column-id rule. A column keeps its id forever; a\npartition field keeps its id only while it stays untouched. Change\nanything about a field — swap `month` for `day`, point it at a\ndifferent source column — and it becomes a *new* field that needs a\nfresh, never-before-used id, while the old id retires with the old\nspec. Reusing a retired id (or reassigning an id to a different field)\nproduces a table that Athena still reads but Spark and Trino reject\nwith `ValidationException: Conflicting partition fields`.\n\n`identifierFieldNames` lists the columns that together identify a row\nand renders as the schema's `identifier-field-ids`. Each one must be a\n`required: true` column (Glue rejects an optional one), and `cdk synth`\nrejects an identifier column of type float, double, list, map, struct,\nunknown, variant, geometry or geography. The construct prefixes each\nidentifier column's `doc` with `(PK) `, so a `doc` you write must not\nstart with `(PK) `; see [Evolving schema and\npartitions](#evolving-schema-and-partitions).\n\nEvery string you pass is a literal string, not a token, and an\nenum-typed one must be a member of its enum; the exceptions are\n`location`, `comment`, table property values, and the `tableName` of\nan imported table.\n\nThe resulting Iceberg `metadata.json` for this table contains every\nfeature you set:\n\n```json\n{\n \"format-version\": 2,\n \"table-uuid\": \"39a948f9-...\",\n \"current-schema-id\": 0,\n \"schemas\": [\n {\n \"schema-id\": 0,\n \"identifier-field-ids\": [1],\n \"fields\": [\n { \"id\": 1, \"name\": \"order_id\", \"required\": true, \"type\": \"long\", \"doc\": \"(PK) \" },\n { \"id\": 2, \"name\": \"customer_id\", \"required\": true, \"type\": \"long\" },\n { \"id\": 3, \"name\": \"order_amount\", \"required\": true, \"type\": \"decimal(12, 2)\" },\n { \"id\": 4, \"name\": \"currency\", \"required\": true, \"type\": \"string\" },\n { \"id\": 5, \"name\": \"placed_at\", \"required\": true, \"type\": \"timestamptz\" },\n { \"id\": 6, \"name\": \"tags\", \"required\": false,\n \"type\": { \"type\": \"list\", \"element-id\": 9, \"element\": \"string\", \"element-required\": false } },\n { \"id\": 7, \"name\": \"shipping_address\", \"required\": false,\n \"type\": { \"type\": \"struct\", \"fields\": [\n { \"id\": 10, \"name\": \"line1\", \"required\": true, \"type\": \"string\" },\n { \"id\": 11, \"name\": \"city\", \"required\": true, \"type\": \"string\" },\n { \"id\": 12, \"name\": \"country\", \"required\": true, \"type\": \"string\" },\n { \"id\": 13, \"name\": \"postal_code\", \"required\": false, \"type\": \"string\" }\n ] } },\n { \"id\": 8, \"name\": \"metadata\", \"required\": false,\n \"type\": { \"type\": \"map\", \"key-id\": 14, \"key\": \"string\", \"value-id\": 15,\n \"value-required\": false, \"value\": \"string\" } }\n ]\n }\n ],\n \"partition-specs\": [\n { \"spec-id\": 0, \"fields\": [\n { \"name\": \"placed_at_day\", \"transform\": \"day\", \"source-id\": 5, \"field-id\": 1000 },\n { \"name\": \"customer_id_bucket\", \"transform\": \"bucket[16]\", \"source-id\": 2, \"field-id\": 1001 }\n ]}\n ],\n \"sort-orders\": [\n { \"order-id\": 1, \"fields\": [\n { \"transform\": \"identity\", \"source-id\": 5, \"direction\": \"asc\", \"null-order\": \"nulls-last\" },\n { \"transform\": \"identity\", \"source-id\": 1, \"direction\": \"asc\", \"null-order\": \"nulls-last\" }\n ]}\n ],\n \"properties\": {\n \"format-version\": \"2\",\n \"write.format.default\": \"parquet\",\n \"write.parquet.compression-codec\": \"zstd\",\n \"write.merge.mode\": \"merge-on-read\",\n \"write.update.mode\": \"merge-on-read\",\n \"write.delete.mode\": \"merge-on-read\",\n \"write.target-file-size-bytes\": \"134217728\",\n \"history.expire.min-snapshots-to-keep\": \"5\",\n \"gc.enabled\": \"true\",\n \"comment\": \"Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.\"\n }\n}\n```\n\n### Format version 3\n\nSet `formatVersion: IcebergFormatVersion.V3` to create a v3 table. The\nsix v3 types (below) are only accepted on a V3 table, at any nesting\ndepth: inside a struct, as a list element, or as a map key or value.\nSynth fails on a V1 or V2 table that uses one.\n\n```typescript\nimport {\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'Events', {\n database: db,\n tableName: 'events',\n location: `s3://${bucket.bucketName}/analytics/events/`,\n columns: [\n { name: 'occurred_at', type: IcebergType.TIMESTAMP_NS, id: 1 },\n { name: 'payload', type: IcebergType.VARIANT, id: 2 },\n {\n name: 'origin',\n type: IcebergType.geography('OGC:CRS84', IcebergEdgeAlgorithm.SPHERICAL),\n id: 3,\n },\n ],\n formatVersion: IcebergFormatVersion.V3,\n});\n```\n\nThe same table in Python:\n\n```python\nfrom cdk_glue_iceberg_table import (\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n)\n\nIcebergTable(self, \"Events\",\n database=db,\n table_name=\"events\",\n location=f\"s3://{bucket.bucket_name}/analytics/events/\",\n columns=[\n {\"name\": \"occurred_at\", \"type\": IcebergType.TIMESTAMP_NS, \"id\": 1},\n {\"name\": \"payload\", \"type\": IcebergType.VARIANT, \"id\": 2},\n {\n \"name\": \"origin\",\n \"type\": IcebergType.geography(\"OGC:CRS84\", IcebergEdgeAlgorithm.SPHERICAL),\n \"id\": 3,\n },\n ],\n format_version=IcebergFormatVersion.V3,\n)\n```\n\nThis table has a `timestamp_ns` column; see [Known limitations](#known-limitations)\nbefore reading it from upstream Spark.\n\nThe v3 types, with the rules `cdk synth` enforces:\n\n| Type | Factory | Rendered string | Partition and identifier rules |\n| --- | --- | --- | --- |\n| unknown | `IcebergType.UNKNOWN` | `unknown` | Must be optional, as a column, list element, or map value; never a map key. Partition source only with `void`. Not an identifier field. |\n| timestamp with nanoseconds | `IcebergType.TIMESTAMP_NS` | `timestamp_ns` | `identity`, `year`, `month`, `day`, `hour`, `bucket(N)`, `void`; not `truncate(W)`. Allowed as an identifier field. |\n| timestamptz with nanoseconds | `IcebergType.TIMESTAMPTZ_NS` | `timestamptz_ns` | Same as `timestamp_ns`. |\n| variant | `IcebergType.VARIANT` | `variant` | Never a partition or sort source, `void` included. Not an identifier field. |\n| geometry | `IcebergType.geometry(crs?)` | `geometry`, or `geometry(<crs>)` when a CRS is given | Partition source only with `void`. Not an identifier field. |\n| geography | `IcebergType.geography(crs?, algorithm?)` | `geography`, `geography(<crs>)`, or `geography(<crs>, <algorithm>)` when an algorithm is given, such as `geography(OGC:CRS84, spherical)` | Same as `geometry`. |\n\nA geometry or geography `crs` is an identifier such as `OGC:CRS84` or\n`EPSG:3857`; whitespace, parentheses, braces, and commas are rejected.\nWhen `algorithm` is set without a `crs`, the rendered string spells out\n`OGC:CRS84`.\n\nWhich engines read a v3 table created by the construct:\n\n| Engine | v3 tables | Notes |\n| --- | --- | --- |\n| Spark 4.1.1 + Iceberg 1.11.0 (the integration-test runner) | Yes | Reads and writes tables the construct creates, including deletion vectors, row lineage, `variant` and `unknown`, except tables with a `geometry`, `geography` or nanosecond-timestamp column ([Known limitations](#known-limitations)). |\n| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists \"Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types\". The integration suites run on Glue 6.0's Spark 4.1.1 and Iceberg 1.11.0 (`scripts/spark/spark-env.sh`); the Glue job runtime itself is not exercised. |\n| Amazon EMR Spark 8.1.0 (Spark 4.1.1, Iceberg 1.11.0-amzn-0) | Documented | The [emr-spark-8.1.0 release notes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html) state \"Amazon EMR Spark 8.1.0 extends Iceberg v3 spec support for Spark workloads\". The integration suites do not run on EMR 8.1. |\n| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states \"Athena only creates and operates on Iceberg v2 tables.\" The v2 integration suites also run through Athena (see [Does it work with Athena and Lake Formation?](#does-it-work-with-athena-and-lake-formation)); no v3 table is queried through Athena. |\n\n### Granting access\n\n```typescript\nconst table = new IcebergTable(this, 'Orders', {\n // ...\n encryptionKey: bucketKey, // the key the table's bucket encrypts objects with\n});\ntable.grantRead(role);\ntable.grantWrite(role);\ntable.grantReadWrite(role, {\n catalogEncryptionKey: catalogKey, // the Data Catalog's SSE-KMS key\n lakeFormationDataAccess: true, // Athena on a location registered with Lake Formation\n});\n```\n\nEach method issues these statements and no others:\n\n| Statement | `grantRead` | `grantWrite` | `grantReadWrite` | Resource |\n| --- | --- | --- | --- | --- |\n| Glue | `glue:BatchGetPartition`, `glue:GetDatabase`, `glue:GetPartition`, `glue:GetPartitions`, `glue:GetTable`, `glue:GetTables`, `glue:GetTableVersion`, `glue:GetTableVersions` | `glue:BatchCreatePartition`, `glue:BatchDeletePartition`, `glue:CreatePartition`, `glue:DeletePartition`, `glue:UpdatePartition`, `glue:UpdateTable` | both lists | `arn:<partition>:glue:<region>:<account>:catalog`, the database ARN and the table ARN |\n| S3 list | `s3:ListBucket` | none | `s3:ListBucket` | the bucket, with condition `StringLike` `s3:prefix` = `<prefix>*` or `<prefix>` |\n| S3 bucket | `s3:GetBucketLocation` | `s3:ListBucketMultipartUploads` | both | the bucket |\n| S3 objects | `s3:GetObject` | `s3:PutObject`, `s3:DeleteObject`, `s3:AbortMultipartUpload`, `s3:ListMultipartUploadParts` | both lists | `<bucket>/<prefix>*` |\n| Table key, when `encryptionKey` is set | `kms:Decrypt` | `kms:Decrypt`, `kms:GenerateDataKey` | `kms:Decrypt`, `kms:GenerateDataKey` | the key |\n| Catalog key, when `catalogEncryptionKey` is passed | `kms:Decrypt` | `kms:Decrypt`, `kms:Encrypt`, `kms:GenerateDataKey` | `kms:Decrypt`, `kms:Encrypt`, `kms:GenerateDataKey` | the key |\n| Lake Formation, when `lakeFormationDataAccess` is `true` | `lakeformation:GetDataAccess` | `lakeformation:GetDataAccess` | `lakeformation:GetDataAccess` | `*` |\n\nThe KMS statements are on the whole key: `IKey.grant` adds no\nencryption-context condition, so a key shared with other data (a bucket\nor lake key, the catalog key) is covered for everything it encrypts,\nnot only this table.\n\n`lakeformation:GetDataAccess` is the one statement on `*`, because Lake\nFormation supports no resource for it. It is issued only with\n`lakeFormationDataAccess: true`; with it, Lake Formation vends\ncredentials for every table the principal holds Lake Formation\npermissions on, which includes tables open to `IAM_ALLOWED_PRINCIPALS`.\n\nPass `catalogEncryptionKey` when the Data Catalog is encrypted with a\ncustomer managed key.\n\nLake Formation permissions themselves (`AWS::LakeFormation::Permissions`)\nare yours to create:\n\n```typescript\nimport { CfnPermissions } from 'aws-cdk-lib/aws-lakeformation';\n\nnew CfnPermissions(this, 'OrdersReader', {\n dataLakePrincipal: { dataLakePrincipalIdentifier: role.roleArn },\n resource: {\n tableResource: {\n catalogId: this.account,\n databaseName: table.database.databaseName,\n name: table.tableName,\n },\n },\n permissions: ['SELECT', 'DESCRIBE'],\n});\n```\n\nAdd `INSERT`, `DELETE` and `ALTER` for a principal that writes.\n\nIn Python the new names are `encryption_key`, and the grant options are\nthe keyword arguments `lake_formation_data_access` and\n`catalog_encryption_key`.\n\nUpgrading from a version before 0.17.0: existing `grantRead`,\n`grantWrite` and `grantReadWrite` policies change with no code edit.\nThe Glue statement names the catalog and database ARNs as well as the\ntable, and read grants add `glue:GetDatabase`. Nothing for KMS or Lake\nFormation is issued unless the new prop or options are passed.\n\n### Importing an existing table\n\n```typescript\nconst existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {\n database: db,\n tableName: 'orders',\n location: 's3://my-bucket/analytics/orders/',\n encryptionKey: bucketKey, // optional; grants then cover the key\n});\nexisting.grantRead(role);\n```\n\n### Evolving schema and partitions\n\nChange the `columns` array (or `partitionSpec`) and run `cdk deploy`\nagain. The construct passes the new schema to Glue's `UpdateTable`,\nwhich writes a new `metadata.json` with a new `schema-id`; existing\ndata files stay readable because each column's `id` is pinned to the\nid the table holds and never reused. Adds, renames (same `id`, new\n`name`), and drops all flow through `cdk deploy` alone — no\nout-of-band SQL DDL.\n\nNested fields evolve the same way. Each struct field, list element,\nmap key, and map value keeps the id you gave it (the rule is under\n[Using `IcebergTable`](#using-icebergtable)), so adding a column or a\nstruct field anywhere in the schema leaves every other id unchanged. A\nnew nested field takes an id no field of the table has ever used:\n\n```typescript\ntype: IcebergType.struct([\n { id: 10, name: 'line1', type: IcebergType.STRING },\n { id: 16, name: 'line2', type: IcebergType.STRING }, // new field, never-used id\n { id: 11, name: 'city', type: IcebergType.STRING },\n]),\n```\n\nSuch an id is out of creation order, which is expected once a table\nhas evolved: creation order applies only to the ids a table is created\nwith (see [Using `IcebergTable`](#using-icebergtable)).\n\nUpgrading from a version before 0.12.0, which numbered nested fields\nitself: declare the ids the table's current `metadata.json` holds,\ntop-level and nested (each field `id`, `element-id`, `key-id` and\n`value-id`), or the next deploy renumbers the fields that differ and\nexisting data files read under the wrong field. A nested type object\nshared by two columns now fails synth with a duplicate id; give each\ncolumn its own nested type with its own ids. The list element and map\nvalue nullability default also changed: `elementRequired` and\n`valueRequired` now default to `false`. Earlier versions made list\nelements required by default and map values required unless the third\nargument of `map` was `false`; set `elementRequired: true` or\n`valueRequired: true` wherever the deployed schema has\n`\"element-required\": true` or `\"value-required\": true`.\n\nGlue's `UpdateTable` reuses an earlier schema whose fields (name,\ntype, `required`, `doc`) match the request, together with that\nschema's `identifier-field-ids`. The construct therefore prefixes the\n`doc` of every identifier column with `(PK) ` (just `(PK) ` when the\ncolumn has no `doc`), shown as the column comment, so every\nidentifier change is a field change. On a table whose schema history\nwas written with the marker, identifier changes flow through `cdk\ndeploy` like column changes.\n\nPartition evolution flows the same way, with one extra rule on\n`fieldId`: a changed partition field is a new partition field. To move\na table from `month(placed_at)` to `day(placed_at)`, drop the month\nentry and add a day entry under a fresh `fieldId` — never recycle the\nmonth field's id:\n\n```typescript\npartitionSpec: [\n // was: { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.MONTH, fieldId: 1000 },\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1001,\n },\n],\n```\n\nOld data files written under the month spec stay readable; new writes\npartition by day. Reusing `1000` for the day field instead corrupts\nthe spec history for strict engines (Spark, Trino) while Athena keeps\nworking — the failure only surfaces at Spark runtime.\n\nA changed `sortOrder` also flows through `cdk deploy`. Glue adds it to\n`sort-orders` under the next `order-id` and points\n`default-sort-order-id` at it; the earlier orders stay in the list.\nA change to `tableProperties` alone writes new metadata that keeps the\nsame `current-schema-id`, partition spec and sort order.\n\n### Checking changes against a schema history\n\nThe construct never reads the deployed table, so on its own it cannot\ntell that an id belonged to a field dropped two deploys ago. A schema\nhistory file committed beside the app gives synth that record without\nan AWS call. Point `schemaHistoryFile` at the history file of the\ntable's database:\n\n```typescript\nnew IcebergTable(this, 'Orders', {\n // ...\n schemaHistoryFile: path.join(__dirname, 'analytics.schema-history.json'),\n});\n```\n\nA relative path resolves against the working directory, which is the\ndirectory holding `cdk.json` when the `cdk` CLI runs the app. Create\nthe file once with the content `{\"tables\": []}`. Records are keyed by\ntable name, so give each Glue database its own file.\n\nEvery synth (`cdk synth`, `cdk diff`, `cdk deploy`, and the CI job\nthat runs one) then compares each table with its record and fails,\nlisting every problem, when the definition:\n\n- gives a column or nested field id in the record to a different\n field: a dropped field's id, or the id of a field whose\n path is still in the definition under another id;\n- keeps a field's path but changes its id, as in the nested ids\n `4, 5, 6` becoming `5, 6, 7` after a column is added;\n- changes a field's type, except `int` to `long`, `float` to `double`,\n and a wider decimal precision at the same scale;\n- makes an optional field required;\n- drops a column that any recorded partition spec was sourced from;\n- keeps a partition `fieldId` with a different source column or\n transform.\n\nFields are identified by id, so a rename passes. A table with no\nrecord passes. Synth never writes the file; record a change once it\npasses synth, and commit the file with the change:\n\n```bash\ncdk synth --quiet\nnode -e \"console.log(require('cdk-glue-iceberg-table').IcebergSchemaHistory.updateFromCloudAssembly('cdk.out'))\"\n```\n\nFrom a Python app, the second command is:\n\n```bash\npython -c \"from cdk_glue_iceberg_table import IcebergSchemaHistory; print(IcebergSchemaHistory.update_from_cloud_assembly('cdk.out'))\"\n```\n\n`updateFromCloudAssembly` records every table of the assembly that\nsets `schemaHistoryFile` in that file and returns the files it wrote.\nThe record keeps every id the table has used: a dropped field stays\nwith `\"dropped\": true`, and a partition field stays after it leaves\nthe spec. The file must already exist, and nothing is written when any\ntable is an unsafe change. A column replaced under the same name needs\ntwo recorded changes: drop it, then add it with a new id.\n\n`IcebergSchemaHistory.validateSchemaEvolution(previous, next)` runs\nthe same checks on one table record without the prop, and\n`IcebergSchemaHistory.append`, `toJson` and `fromJson` record a\ndefinition in, write and read the file format.\n\n### Inserting and querying\n\n```sql\n-- INSERT into the orders table\nINSERT INTO sample_database.orders VALUES\n (1001, 5001, DECIMAL '149.99', 'USD',\n TIMESTAMP '2026-05-20 09:15:00 UTC',\n ARRAY['holiday-promo','first-order'],\n CAST(ROW('1 Infinite Loop','Cupertino','US','95014')\n AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),\n MAP(ARRAY['channel','utm'], ARRAY['web','google'])),\n -- ... more rows\n;\n\n-- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)\nDELETE FROM sample_database.orders WHERE order_id = 1003;\n\n-- merge-on-read UPDATE\nUPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;\n\n-- SELECT\nSELECT customer_id, SUM(order_amount) AS total\n FROM sample_database.orders\n GROUP BY 1\n ORDER BY 2 DESC;\n```\n\n## Two footguns the construct prevents\n\n### Footgun #1 — schema under `storageDescriptor.columns`\n\nThe CREATE succeeds but the first UPDATE silently strips\n`table_type=ICEBERG` from the table's Glue parameters, and Athena\nqueries after that fail with `HIVE_UNSUPPORTED_FORMAT`.\n\n```typescript\n// DON'T DO THIS — what most StackOverflow / re:Post examples show\nnew CfnTable(this, 'OrdersBad', {\n catalogId: this.account,\n databaseName: 'analytics',\n tableInput: {\n name: 'orders',\n tableType: 'EXTERNAL_TABLE',\n parameters: {\n table_type: 'ICEBERG',\n },\n storageDescriptor: {\n location: 's3://.../orders/',\n columns: [\n /* ... */\n ],\n },\n },\n openTableFormatInput: {\n icebergInput: {\n metadataOperation: 'CREATE',\n version: '2',\n },\n },\n});\n```\n\n`IcebergTable` instead always emits schema/partitions/sort/properties\nunder `openTableFormatInput.icebergInput.icebergTableInput`, never\nunder `storageDescriptor`.\n\n### Footgun #2 — `tableInput` co-present with `openTableFormatInput`\n\nEven setting just `tableInput: { name: 'foo' }` next to\n`openTableFormatInput` returns\n`\"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput\"`.\nThe construct never emits `tableInput`; the table-level comment goes\ninto `tableProperties['comment']`, which lives inside\n`icebergTableInput.properties`.\n\n(There is a third footgun, field-id reuse after a column drop, that\nthe construct prevents only with a [schema history\nfile](#checking-changes-against-a-schema-history). See the next\nsection.)\n\n## Known limitations\n\n- **Creation order is not checked.** The construct cannot tell at synth whether a table is new or already deployed, so it accepts a new table whose ids are not in creation order (the rule is under [Using `IcebergTable`](#using-icebergtable)). When the first update of such a table changes the id of a column an identity partition is sourced from, Glue rejects the update with `Cannot create identity partition sourced from different field in schema` and the CloudFormation rollback fails the same way, leaving the stack in `UPDATE_ROLLBACK_FAILED`.\n- **Field-id reuse across deploys is detected only with a schema history file.** This applies to nested ids (struct field, list element, map key and value) as well as column ids. If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the \"never reuse a retired id\" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate field id N` validator) and never reads the live table state; across deploys, only a [schema history file](#checking-changes-against-a-schema-history) catches reuse. Without it, treat dropped ids as retired forever.\n- **Partition field-id reuse across deploys is detected only with a schema history file.** The same live-state blindness applies to partition `fieldId`s: the construct validates them within one deploy (integer >= 1000, unique in the spec), and only a [schema history file](#checking-changes-against-a-schema-history) tells it that the id you pinned already appeared in an earlier spec of the deployed table. The invariant partition ids must uphold is stricter than the column one — unique across the table's **entire spec history**, so a changed field (different transform or source column) must take a fresh id rather than keep its old one. Violate it and strict engines (Spark, Trino) reject the table with `Conflicting partition fields` while Athena keeps answering, masking the break until Spark runtime. Without the history file, treat every id that has ever appeared in any spec as retired, and allocate new fields the next never-used id. A partition change needs a new source column or transform; Glue ignores a `fieldId`-only change.\n- **`last-partition-id` is written by Glue, not by the construct.** The construct supplies the spec's field ids; Glue's `UpdateTable` computes the table's `last-partition-id` counter when it writes the new metadata. The repo's integration test asserts the counter keeps up with the highest supplied field id after a partition evolution, but engines that later evolve the spec themselves (e.g. Spark `ALTER TABLE … ADD PARTITION FIELD`) allocate from that counter — if you mix CDK-managed and engine-managed partition evolution, verify the counter in `metadata.json` before trusting engine-side changes.\n- **A rolled-back deploy leaves its schema in the table's history.** When a deploy that adds a column updates the table and then fails on another resource, the stack ends in `UPDATE_ROLLBACK_COMPLETE` and `current-schema-id` is back to its earlier value, so the column is gone and the rows read as before. The schema that held the column stays in `schemas`, and `last-column-id` stays at the column's id. A clean redeploy of the same change makes that schema current again without adding a schema. Treat an id from a rolled-back deploy as used: give it only to the column that deploy added.\n- **Removing the last identifier from a table created with identifiers before 0.11.0 can bring the old ids back.** That table's history holds an unmarked schema with ids, and Glue reuses it when the rendered fields match it again. Change a column `doc` in the same deploy that removes the last identifier.\n- **The schema history check covers only the changes it lists.** It does not see a change an engine makes with `ALTER TABLE` (Spark, Athena), whose ids the file never records, or a deploy whose definition was not recorded in the file. It does not check sort order or identifier field changes or report an added required field, top level or nested, and it treats a field moved into another struct under its id as a rename. It reports the format v3 promotions `unknown` to any type and `date` to `timestamp` or `timestamp_ns` as type changes.\n- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.\n- **Format version is immutable after CREATE.** Glue ignores a V2 to V3 `formatVersion` change, though `properties.format-version` then reads `3`. Keep `formatVersion` at the table's real version, or destroy + recreate the table to change it. An engine can upgrade a v2 table to v3 in place with `ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3')`; after that, set `formatVersion` to `V3` so the next deploy declares the version the table already has.\n- **`merge-on-read` requires v2 or later.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.\n- **Column defaults are not available.** Iceberg v3 `initial-default` and `write-default` cannot be declared through CloudFormation: the `AWS::Glue::Table` resource schema has no `InitialDefault` or `WriteDefault` property. Revisit when that schema gains the two properties.\n- **Nanosecond timestamps are catalog-only for Spark 4.1.1 with Iceberg 1.11.0.** In [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag, `primitive()` has no case for the `TIMESTAMP_NANO` type id, so a table with a `timestamp_ns` or `timestamptz_ns` column reaches the `Cannot convert unsupported type to Spark` error. The integration suite asserts these columns from `metadata.json` only.\n- **Geometry and geography columns are catalog-only for Spark 4.1.1 with Iceberg 1.11.0, the versions AWS Glue 6.0 ships.** The first Spark read of a table holding a `geography` and a `geometry` column failed with `UnsupportedOperationException: Cannot convert unsupported type to Spark: geography(OGC:CRS84, spherical)` on those versions. [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag has no conversion for `geometry` either. The integration suite deploys these columns on a table no Spark job opens and asserts them from `metadata.json` only.\n- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.\n- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change unless a [schema history file](#checking-changes-against-a-schema-history) records the column as a partition source, and Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.\n\n## FAQ\n\n### How do I create an Apache Iceberg table in AWS CDK?\n\nInstall `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a\ndatabase, a column list, and an S3 `location` — see [Use](#use).\n`cdk deploy` creates the Glue Data Catalog table and writes the\nIceberg `metadata.json` under your S3 prefix. No custom resource or\nLambda is involved; it is a plain `AWS::Glue::Table` in your\nCloudFormation template.\n\n### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?\n\nChange the `columns` array and run `cdk deploy` again. The construct\npasses the new schema to Glue's `UpdateTable`, which writes a new\n`metadata.json` with a new `schema-id`; existing data files stay\nreadable because each column's `id` is pinned to the id the table\nholds and never reused.\nPartition changes work the same way, except a changed partition field\ntakes a fresh `fieldId` instead of keeping its old one. See [Evolving\nschema and partitions](#evolving-schema-and-partitions).\n\n### Does CloudFormation support Iceberg tables natively?\n\nYes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,\ndocumented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).\nThe catch is that the raw shape corrupts the table on the first\n`Update` if you place schema under `storageDescriptor.columns` or set\n`tableInput` alongside `openTableFormatInput`. This construct only ever\nemits the safe shape and gives you no way to express the unsafe ones.\nSee [Two footguns the construct prevents](#two-footguns-the-construct-prevents).\n\n### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?\n\nA raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON\nand own both silent-corruption footguns. `cdk-glue-iceberg-table` gives\nyou a typed `IcebergType` / `IcebergPartitionTransform` API,\nsynth-time validation (partition transforms checked against column\ntypes, format/version mismatches caught before deploy), pinned field\nIDs for safe evolution (declared in creation order for a new table),\nand `grantRead` / `grantWrite` helpers. See\n[How it compares](#how-it-compares).\n\n### How do I create a partitioned Iceberg table in CDK?\n\nPass a `partitionSpec` of `IcebergPartitionTransform` entries —\n`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,\nor `void` — each pinned to a `fieldId` of 1000 or above. Transforms are\nvalidated against their source column's type at synth time. See\n[Using `IcebergTable`](#using-icebergtable).\n\n### Does it work with Athena and Lake Formation?\n\nYes for v2 tables, no for v3 (see the engine matrix in [Format\nversion 3](#format-version-3)). The demo app in the\n[`arceus`](https://github.com/ksco92/arceus) repo registers its tables\nwith Lake Formation, and its v2 integration suites query them through\nAthena as well as Spark, on the same tables:\n\n- merge-on-read `INSERT`, `UPDATE`, `DELETE` and `MERGE INTO`, time\n travel (`FOR VERSION AS OF` / `FOR TIMESTAMP AS OF`), `OPTIMIZE ...\n REWRITE DATA USING BIN_PACK`, and `VACUUM`;\n- a read after every schema, partition, identifier and nested-field\n evolution step;\n- the `(PK) ` comment on identifier columns, read from\n `information_schema.columns`;\n- rows written by Spark read through Athena, and rows written by Athena\n read through Spark;\n- reads and writes through Spark and Athena as roles whose table\n permissions come from `grantRead` and `grantReadWrite`, and a denied\n write as the `grantRead` role.\n\nThe suites grant the querying principal `SELECT`, `INSERT`, `DELETE`,\n`ALTER` and `DESCRIBE` through Lake Formation. See [Granting access](#granting-access).\n\n### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?\n\nYes — set `formatVersion: IcebergFormatVersion.V2` and the\n`write.{delete,update,merge}.mode = merge-on-read` table properties.\nThe construct rejects merge-on-read on a v1 table at synth time.\n\n### Can I create an Iceberg v3 table?\n\nYes. Set `formatVersion: IcebergFormatVersion.V3`; see [Format\nversion 3](#format-version-3) for the types, the engine matrix and the\nlimits.\n\n## Contributing\n\nDevelopment, the monorepo layout, and the demo app live in [CONTRIBUTING.md](https://github.com/ksco92/arceus/blob/main/CONTRIBUTING.md).\n\n## License\n\n[MIT](https://github.com/ksco92/arceus/blob/main/LICENSE).\n"
|
|
9754
|
+
"markdown": "# cdk-glue-iceberg-table\n\n[](https://github.com/ksco92/arceus/actions/workflows/ci.yml)\n[](https://codecov.io/gh/ksco92/arceus)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://pypi.org/project/cdk-glue-iceberg-table/)\n[](https://constructs.dev/packages/cdk-glue-iceberg-table)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://github.com/ksco92/arceus/blob/main/LICENSE)\n\n**`cdk-glue-iceberg-table` is an AWS CDK L2 construct to create and\nevolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and\npartition changes that survive CloudFormation updates, via cdk\ndeploy.** It supports Iceberg format version 3 (`variant`, nanosecond\ntimestamps, `geometry` / `geography`, `unknown`). It emits the exact\n`AWS::Glue::Table` / `OpenTableFormatInput` shape, so a single `cdk\ndeploy` creates, evolves, and destroys a table like any other CDK\nresource. No custom resource, no Lambda, no \"`cdk deploy` and then run\nthis SQL by hand\" two-step. Table changes (new columns, renames, drops,\nnew partition fields) land as a reviewed diff in a pull request and\napply through `cdk deploy`.\n\nStatus (October 2026): published on npm (TypeScript/JavaScript) and\nPyPI (Python), pre-1.0, with a public surface guarded on every PR by two\nconsumer jobs (`e2e-consumer` compiles and synthesizes an app against a\npinned published version, `packed-consumer` does the same against the PR's\nown `npm pack` tarball) plus real-AWS integration suites, including ones\nfor format version 3. The multi-language packages are generated\nfrom one TypeScript source with [jsii](https://aws.github.io/jsii/)\nand indexed on [Construct\nHub](https://constructs.dev/packages/cdk-glue-iceberg-table).\nDeveloped in the [`ksco92/arceus`](https://github.com/ksco92/arceus)\nmonorepo, which also holds a CDK demo app that dogfoods the construct\nagainst a real AWS account.\n\n### Why this exists\n\nAWS documents the CloudFormation shape for Iceberg tables ([AWS Big\nData blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),\nbut raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the\n*first* `Update` silently strips `table_type=ICEBERG`, after which\nevery Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The\nmotivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);\n[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)\ndocuments the only working shape and the silent-corruption traps you\nhit by getting it slightly wrong. This construct implements that\nshape, refuses to emit the unsafe alternatives, and pins Iceberg field\nIDs (declared [in creation order](#using-icebergtable) for a new\ntable) so schema evolution never orphans existing data. It was the\nbasis of an upstream proposal to land an `IcebergTable` in\n`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988)),\nclosed unmerged in September 2026.\n\n### How it compares\n\n| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |\n| --- | --- | --- | --- | --- |\n| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |\n| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |\n| Spark SQL DDL, or Athena SQL DDL, at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |\n| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |\n\n## Install\n\nTypeScript / JavaScript (npm):\n\n```bash\nnpm install cdk-glue-iceberg-table\n```\n\nPython (PyPI):\n\n```bash\npip install cdk-glue-iceberg-table\n```\n\nPeer dependencies (your CDK app must already have these). This\nrelease requires `aws-cdk-lib` 2.273.0 or newer and `constructs`\n10.8.1 or newer, plus Node.js 22.14 or newer for both languages (the\nPython package runs under Node.js through jsii).\n\nFor TypeScript / JavaScript:\n\n```bash\nnpm install aws-cdk-lib@^2.273.0 constructs@^10.8.1\n```\n\nFor Python:\n\n```bash\npip install 'aws-cdk-lib>=2.273.0' 'constructs>=10.8.1'\n```\n\n## Use\n\n### TypeScript\n\n```ts\nimport { Bucket } from 'aws-cdk-lib/aws-s3';\nimport { Database } from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\nconst db = new Database(this, 'Db', { databaseName: 'analytics' });\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n columns: [\n { name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },\n { name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },\n ],\n partitionSpec: [\n { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY, fieldId: 1000 },\n { sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16), fieldId: 1001 },\n ],\n identifierFieldNames: ['order_id'],\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\n### Python\n\nThe same table in Python — the API mirrors TypeScript with\n`snake_case` props and `PascalCase` types:\n\n```python\nfrom aws_cdk.aws_s3 import Bucket\nfrom aws_cdk.aws_glue import Database\nfrom cdk_glue_iceberg_table import (\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n)\n\nbucket = Bucket(self, \"Warehouse\")\ndb = Database(self, \"Db\", database_name=\"analytics\")\n\nIcebergTable(self, \"OrdersTable\",\n database=db,\n table_name=\"orders\",\n location=f\"s3://{bucket.bucket_name}/analytics/orders/\",\n columns=[\n {\"name\": \"order_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 1},\n {\"name\": \"customer_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 2},\n {\"name\": \"placed_at\", \"type\": IcebergType.TIMESTAMPTZ, \"required\": True, \"id\": 3},\n ],\n partition_spec=[\n {\"source_column\": \"placed_at\", \"transform\": IcebergPartitionTransform.DAY, \"field_id\": 1000},\n {\"source_column\": \"customer_id\", \"transform\": IcebergPartitionTransform.bucket(16), \"field_id\": 1001},\n ],\n identifier_field_names=[\"order_id\"],\n format_version=IcebergFormatVersion.V2,\n)\n```\n\nConsumer-facing reference sections:\n\n- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.\n- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.\n- [Format version 3](#format-version-3) — the six v3 types, which engines read them, and the accepted limits.\n- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.\n- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.\n- [FAQ](#faq) — common \"how do I … in CDK / CloudFormation\" questions.\n\n## Exported surface\n\nThe package entry point re-exports the main types you import from\n`cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled\nsurface lives under `dist/lib/iceberg/` after `npm install`; in Python\nthe same surface is imported from the `cdk_glue_iceberg_table` module\nafter `pip install`:\n\n- **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite` (each taking an optional `IcebergTableGrantOptions`), the `fromIcebergTableAttributes(...)` import factory, and `creationOrderFieldIds(columns)` (Python: `creation_order_field_ids`), which returns a map from field path to the id Glue gives that field in a new table; a name containing `.` or a backquote appears in backquotes.\n- **`IcebergSchemaHistory`** — `validateSchemaEvolution(previous, next)` (Python: `validate_schema_evolution`), which returns the unsafe changes from a table's history record to a definition (`IcebergTableSchema`) as `IcebergSchemaViolation`s, each carrying an `IcebergSchemaViolationKind` that names the class of unsafe change; `append`, `toJson` and `fromJson`, which record a definition in, write and read the history file (`IcebergSchemaHistoryFile`, `IcebergTableHistory`, `IcebergFieldHistory`, `IcebergPartitionFieldHistory`); and `updateFromCloudAssembly(directory)`, which records every table of a synthesized app. See [Checking changes against a schema history](#checking-changes-against-a-schema-history).\n- **`IcebergType`** — primitive types, the v3 constants `UNKNOWN`, `TIMESTAMP_NS`, `TIMESTAMPTZ_NS`, and `VARIANT`, and the `list` / `map` / `struct` / `decimal` / `fixed` / `geometry(crs?)` / `geography(crs?, algorithm?)` factories. `list` takes an `IcebergListDefinition`, `map` an `IcebergMapDefinition`, and `struct` a list of `IcebergStructFieldDefinition`, each carrying the nested field ids. Renders to the string Glue's `IcebergStructField.type` expects.\n- **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.\n- **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2/v3, required — set explicitly per table; Athena cannot query v3 tables), **`IcebergSortDirection`**, **`IcebergNullOrder`**, **`IcebergEdgeAlgorithm`** (`SPHERICAL`, `VINCENTY`, `THOMAS`, `ANDOYER`, `KARNEY`; the second argument of `IcebergType.geography`), plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).\n\n## Using `IcebergTable`\n\nA minimal table:\n\n```typescript\nimport {\n Bucket,\n} from 'aws-cdk-lib/aws-s3';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\n\nconst db = new Database(this, 'Db', {\n databaseName: 'analytics',\n});\n\nnew IcebergTable(this, 'Users', {\n database: db,\n tableName: 'users',\n columns: [\n {\n name: 'user_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'email',\n type: IcebergType.STRING,\n required: true,\n id: 2,\n },\n {\n name: 'signed_up_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 3,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/users/`,\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\nEvery top-level column must declare an `id`: a stable, unique,\npositive integer you pick once and never reuse or reassign. Because\nIceberg data files reference fields by id rather than by name, a fixed\nid, declared in creation order when the table is created (below), is\nwhat lets you add, rename, and drop columns across deploys without\ncorrupting old data — see\n[How do I evolve an Iceberg table schema](#how-do-i-evolve-an-iceberg-table-schema-add-rename-or-drop-a-column-in-cloudformation).\nOnce a column is dropped, leave its id retired; never hand it to a new\ncolumn. Nested fields follow the same rule: every struct field takes\nan `id`, every list an `elementId`, and every map a `keyId` and a\n`valueId`. All ids in a schema, top-level and nested, must be distinct\nintegers from 1 to 2147483447.\n\nA new table must declare its ids in creation order. When Glue creates\na table it does not keep the declared ids: it numbers the fields from\n1 in creation order and points the partition spec, sort order and\nidentifier fields at the new ids. An update Glue accepts stores the\ndeclared ids as given, so the first such update of a table created\nwith other ids renumbers its fields; for the update Glue rejects, see\n[Known limitations](#known-limitations). To number a new table, number\nthe top-level columns first, in order, then the nested fields of each\ncolumn in turn (a list's element; a map's key, then its value, then\nthe key's nested fields, then the value's; a struct's fields, then\ntheir nested fields). `IcebergTable.creationOrderFieldIds` returns\nthese ids by field path and ignores the ids in its input:\n\n```typescript\nconst columns = [\n { name: 'id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'tags', type: IcebergType.list({ element: IcebergType.STRING, elementId: 4 }), id: 2 },\n {\n name: 'attrs',\n type: IcebergType.map({ key: IcebergType.STRING, keyId: 5, value: IcebergType.STRING, valueId: 6 }),\n id: 3,\n },\n];\nIcebergTable.creationOrderFieldIds(columns);\n// { id: 1, tags: 2, attrs: 3, 'tags.element': 4, 'attrs.key': 5, 'attrs.value': 6 }\n```\n\nIds out of creation order are for fields a later deploy adds; see\n[Evolving schema and partitions](#evolving-schema-and-partitions).\n\nA table that exercises most of the surface (partitions, sort order,\nnested types, identifier fields, table properties, removal policy):\n\n```typescript\nimport {\n RemovalPolicy,\n} from 'aws-cdk-lib';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergDataFormat,\n IcebergFormatVersion,\n IcebergNullOrder,\n IcebergPartitionTransform,\n IcebergSortDirection,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',\n columns: [\n {\n name: 'order_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'customer_id',\n type: IcebergType.LONG,\n required: true,\n id: 2,\n },\n {\n name: 'order_amount',\n type: IcebergType.decimal(12, 2),\n required: true,\n id: 3,\n },\n {\n name: 'currency',\n type: IcebergType.STRING,\n required: true,\n id: 4,\n },\n {\n name: 'placed_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 5,\n },\n {\n name: 'tags',\n type: IcebergType.list({\n element: IcebergType.STRING,\n elementId: 9,\n }),\n id: 6,\n },\n {\n name: 'shipping_address',\n type: IcebergType.struct([\n {\n id: 10,\n name: 'line1',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 11,\n name: 'city',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 12,\n name: 'country',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 13,\n name: 'postal_code',\n type: IcebergType.STRING,\n },\n ]),\n id: 7,\n },\n {\n name: 'metadata',\n type: IcebergType.map({\n key: IcebergType.STRING,\n keyId: 14,\n value: IcebergType.STRING,\n valueId: 15,\n }),\n id: 8,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n partitionSpec: [\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1000,\n },\n {\n sourceColumn: 'customer_id',\n transform: IcebergPartitionTransform.bucket(16),\n fieldId: 1001,\n },\n ],\n sortOrder: [\n {\n sourceColumn: 'placed_at',\n direction: IcebergSortDirection.ASC,\n nullOrder: IcebergNullOrder.NULLS_LAST,\n },\n {\n sourceColumn: 'order_id',\n direction: IcebergSortDirection.ASC,\n },\n ],\n identifierFieldNames: [\n 'order_id',\n ],\n dataFormat: IcebergDataFormat.PARQUET,\n formatVersion: IcebergFormatVersion.V2,\n tableProperties: {\n 'write.parquet.compression-codec': 'zstd',\n 'write.delete.mode': 'merge-on-read',\n 'write.update.mode': 'merge-on-read',\n 'write.merge.mode': 'merge-on-read',\n 'write.target-file-size-bytes': '134217728',\n 'history.expire.min-snapshots-to-keep': '5',\n 'gc.enabled': 'true',\n },\n removalPolicy: RemovalPolicy.DESTROY,\n});\n```\n\nEach partition field also pins a `fieldId`: an integer of 1000 or\nabove (the Iceberg spec reserves lower ids for schema columns), unique\nacross the table's entire partition-spec history. The discipline is\nthe inverse of the column-id rule. A column keeps its id forever; a\npartition field keeps its id only while it stays untouched. Change\nanything about a field — swap `month` for `day`, point it at a\ndifferent source column — and it becomes a *new* field that needs a\nfresh, never-before-used id, while the old id retires with the old\nspec. Reusing a retired id (or reassigning an id to a different field)\nproduces a table that Athena still reads but Spark and Trino reject\nwith `ValidationException: Conflicting partition fields`.\n\n`identifierFieldNames` lists the columns that together identify a row\nand renders as the schema's `identifier-field-ids`. Each one must be a\n`required: true` column (Glue rejects an optional one), and `cdk synth`\nrejects an identifier column of type float, double, list, map, struct,\nunknown, variant, geometry or geography. The construct prefixes each\nidentifier column's `doc` with `(PK) `, so a `doc` you write must not\nstart with `(PK) `; see [Evolving schema and\npartitions](#evolving-schema-and-partitions).\n\nEvery string you pass is a literal string, not a token, and an\nenum-typed one must be a member of its enum; the exceptions are\n`location`, `comment`, table property values, and the `tableName` of\nan imported table.\n\nThe resulting Iceberg `metadata.json` for this table contains every\nfeature you set:\n\n```json\n{\n \"format-version\": 2,\n \"table-uuid\": \"39a948f9-...\",\n \"current-schema-id\": 0,\n \"schemas\": [\n {\n \"schema-id\": 0,\n \"identifier-field-ids\": [1],\n \"fields\": [\n { \"id\": 1, \"name\": \"order_id\", \"required\": true, \"type\": \"long\", \"doc\": \"(PK) \" },\n { \"id\": 2, \"name\": \"customer_id\", \"required\": true, \"type\": \"long\" },\n { \"id\": 3, \"name\": \"order_amount\", \"required\": true, \"type\": \"decimal(12, 2)\" },\n { \"id\": 4, \"name\": \"currency\", \"required\": true, \"type\": \"string\" },\n { \"id\": 5, \"name\": \"placed_at\", \"required\": true, \"type\": \"timestamptz\" },\n { \"id\": 6, \"name\": \"tags\", \"required\": false,\n \"type\": { \"type\": \"list\", \"element-id\": 9, \"element\": \"string\", \"element-required\": false } },\n { \"id\": 7, \"name\": \"shipping_address\", \"required\": false,\n \"type\": { \"type\": \"struct\", \"fields\": [\n { \"id\": 10, \"name\": \"line1\", \"required\": true, \"type\": \"string\" },\n { \"id\": 11, \"name\": \"city\", \"required\": true, \"type\": \"string\" },\n { \"id\": 12, \"name\": \"country\", \"required\": true, \"type\": \"string\" },\n { \"id\": 13, \"name\": \"postal_code\", \"required\": false, \"type\": \"string\" }\n ] } },\n { \"id\": 8, \"name\": \"metadata\", \"required\": false,\n \"type\": { \"type\": \"map\", \"key-id\": 14, \"key\": \"string\", \"value-id\": 15,\n \"value-required\": false, \"value\": \"string\" } }\n ]\n }\n ],\n \"partition-specs\": [\n { \"spec-id\": 0, \"fields\": [\n { \"name\": \"placed_at_day\", \"transform\": \"day\", \"source-id\": 5, \"field-id\": 1000 },\n { \"name\": \"customer_id_bucket\", \"transform\": \"bucket[16]\", \"source-id\": 2, \"field-id\": 1001 }\n ]}\n ],\n \"sort-orders\": [\n { \"order-id\": 1, \"fields\": [\n { \"transform\": \"identity\", \"source-id\": 5, \"direction\": \"asc\", \"null-order\": \"nulls-last\" },\n { \"transform\": \"identity\", \"source-id\": 1, \"direction\": \"asc\", \"null-order\": \"nulls-last\" }\n ]}\n ],\n \"properties\": {\n \"format-version\": \"2\",\n \"write.format.default\": \"parquet\",\n \"write.parquet.compression-codec\": \"zstd\",\n \"write.merge.mode\": \"merge-on-read\",\n \"write.update.mode\": \"merge-on-read\",\n \"write.delete.mode\": \"merge-on-read\",\n \"write.target-file-size-bytes\": \"134217728\",\n \"history.expire.min-snapshots-to-keep\": \"5\",\n \"gc.enabled\": \"true\",\n \"comment\": \"Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.\"\n }\n}\n```\n\n### Format version 3\n\nSet `formatVersion: IcebergFormatVersion.V3` to create a v3 table. The\nsix v3 types (below) are only accepted on a V3 table, at any nesting\ndepth: inside a struct, as a list element, or as a map key or value.\nSynth fails on a V1 or V2 table that uses one.\n\n```typescript\nimport {\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'Events', {\n database: db,\n tableName: 'events',\n location: `s3://${bucket.bucketName}/analytics/events/`,\n columns: [\n { name: 'occurred_at', type: IcebergType.TIMESTAMP_NS, id: 1 },\n { name: 'payload', type: IcebergType.VARIANT, id: 2 },\n {\n name: 'origin',\n type: IcebergType.geography('OGC:CRS84', IcebergEdgeAlgorithm.SPHERICAL),\n id: 3,\n },\n ],\n formatVersion: IcebergFormatVersion.V3,\n});\n```\n\nThe same table in Python:\n\n```python\nfrom cdk_glue_iceberg_table import (\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n)\n\nIcebergTable(self, \"Events\",\n database=db,\n table_name=\"events\",\n location=f\"s3://{bucket.bucket_name}/analytics/events/\",\n columns=[\n {\"name\": \"occurred_at\", \"type\": IcebergType.TIMESTAMP_NS, \"id\": 1},\n {\"name\": \"payload\", \"type\": IcebergType.VARIANT, \"id\": 2},\n {\n \"name\": \"origin\",\n \"type\": IcebergType.geography(\"OGC:CRS84\", IcebergEdgeAlgorithm.SPHERICAL),\n \"id\": 3,\n },\n ],\n format_version=IcebergFormatVersion.V3,\n)\n```\n\nThis table has a `timestamp_ns` column; see [Known limitations](#known-limitations)\nbefore reading it from upstream Spark.\n\nThe v3 types, with the rules `cdk synth` enforces:\n\n| Type | Factory | Rendered string | Partition and identifier rules |\n| --- | --- | --- | --- |\n| unknown | `IcebergType.UNKNOWN` | `unknown` | Must be optional, as a column, list element, or map value; never a map key. Partition source only with `void`. Not an identifier field. |\n| timestamp with nanoseconds | `IcebergType.TIMESTAMP_NS` | `timestamp_ns` | `identity`, `year`, `month`, `day`, `hour`, `bucket(N)`, `void`; not `truncate(W)`. Allowed as an identifier field. |\n| timestamptz with nanoseconds | `IcebergType.TIMESTAMPTZ_NS` | `timestamptz_ns` | Same as `timestamp_ns`. |\n| variant | `IcebergType.VARIANT` | `variant` | Never a partition or sort source, `void` included. Not an identifier field. |\n| geometry | `IcebergType.geometry(crs?)` | `geometry`, or `geometry(<crs>)` when a CRS is given | Partition source only with `void`. Not an identifier field. |\n| geography | `IcebergType.geography(crs?, algorithm?)` | `geography`, `geography(<crs>)`, or `geography(<crs>, <algorithm>)` when an algorithm is given, such as `geography(OGC:CRS84, spherical)` | Same as `geometry`. |\n\nA geometry or geography `crs` is an identifier such as `OGC:CRS84` or\n`EPSG:3857`; whitespace, parentheses, braces, and commas are rejected.\nWhen `algorithm` is set without a `crs`, the rendered string spells out\n`OGC:CRS84`.\n\nWhich engines read a v3 table created by the construct:\n\n| Engine | v3 tables | Notes |\n| --- | --- | --- |\n| Spark 4.1.1 + Iceberg 1.11.0 (the integration-test runner) | Yes | Reads and writes tables the construct creates, including deletion vectors, row lineage, `variant` and `unknown`, except tables with a `geometry`, `geography` or nanosecond-timestamp column ([Known limitations](#known-limitations)). |\n| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists \"Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types\". The integration suites run on Glue 6.0's Spark 4.1.1 and Iceberg 1.11.0 (`scripts/spark/spark-env.sh`); the Glue job runtime itself is not exercised. |\n| Amazon EMR Spark 8.1.0 (Spark 4.1.1, Iceberg 1.11.0-amzn-0) | Documented | The [emr-spark-8.1.0 release notes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html) state \"Amazon EMR Spark 8.1.0 extends Iceberg v3 spec support for Spark workloads\". The integration suites do not run on EMR 8.1. |\n| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states \"Athena only creates and operates on Iceberg v2 tables.\" The v2 integration suites also run through Athena (see [Does it work with Athena and Lake Formation?](#does-it-work-with-athena-and-lake-formation)); no v3 table is queried through Athena. |\n\n### Granting access\n\n```typescript\nconst table = new IcebergTable(this, 'Orders', {\n // ...\n encryptionKey: bucketKey, // the key the table's bucket encrypts objects with\n});\ntable.grantRead(role);\ntable.grantWrite(role);\ntable.grantReadWrite(role, {\n catalogEncryptionKey: catalogKey, // the Data Catalog's SSE-KMS key\n lakeFormationDataAccess: true, // Athena on a location registered with Lake Formation\n});\n```\n\nEach method issues these statements and no others:\n\n| Statement | `grantRead` | `grantWrite` | `grantReadWrite` | Resource |\n| --- | --- | --- | --- | --- |\n| Glue | `glue:BatchGetPartition`, `glue:GetDatabase`, `glue:GetPartition`, `glue:GetPartitions`, `glue:GetTable`, `glue:GetTables`, `glue:GetTableVersion`, `glue:GetTableVersions` | `glue:BatchCreatePartition`, `glue:BatchDeletePartition`, `glue:CreatePartition`, `glue:DeletePartition`, `glue:UpdatePartition`, `glue:UpdateTable` | both lists | `arn:<partition>:glue:<region>:<account>:catalog`, the database ARN and the table ARN |\n| S3 list | `s3:ListBucket` | none | `s3:ListBucket` | the bucket, with condition `StringLike` `s3:prefix` = `<prefix>*` or `<prefix>` |\n| S3 bucket | `s3:GetBucketLocation` | `s3:ListBucketMultipartUploads` | both | the bucket |\n| S3 objects | `s3:GetObject` | `s3:PutObject`, `s3:DeleteObject`, `s3:AbortMultipartUpload`, `s3:ListMultipartUploadParts` | both lists | `<bucket>/<prefix>*` |\n| Table key, when `encryptionKey` is set | `kms:Decrypt` | `kms:Decrypt`, `kms:GenerateDataKey` | `kms:Decrypt`, `kms:GenerateDataKey` | the key |\n| Catalog key, when `catalogEncryptionKey` is passed | `kms:Decrypt` | `kms:Decrypt`, `kms:Encrypt`, `kms:GenerateDataKey` | `kms:Decrypt`, `kms:Encrypt`, `kms:GenerateDataKey` | the key |\n| Lake Formation, when `lakeFormationDataAccess` is `true` | `lakeformation:GetDataAccess` | `lakeformation:GetDataAccess` | `lakeformation:GetDataAccess` | `*` |\n\nThe KMS statements are on the whole key: `IKey.grant` adds no\nencryption-context condition, so a key shared with other data (a bucket\nor lake key, the catalog key) is covered for everything it encrypts,\nnot only this table.\n\n`lakeformation:GetDataAccess` is the one statement on `*`, because Lake\nFormation supports no resource for it. It is issued only with\n`lakeFormationDataAccess: true`; with it, Lake Formation vends\ncredentials for every table the principal holds Lake Formation\npermissions on, which includes tables open to `IAM_ALLOWED_PRINCIPALS`.\n\nPass `catalogEncryptionKey` when the Data Catalog is encrypted with a\ncustomer managed key.\n\nLake Formation permissions themselves (`AWS::LakeFormation::Permissions`)\nare yours to create:\n\n```typescript\nimport { CfnPermissions } from 'aws-cdk-lib/aws-lakeformation';\n\nnew CfnPermissions(this, 'OrdersReader', {\n dataLakePrincipal: { dataLakePrincipalIdentifier: role.roleArn },\n resource: {\n tableResource: {\n catalogId: this.account,\n databaseName: table.database.databaseName,\n name: table.tableName,\n },\n },\n permissions: ['SELECT', 'DESCRIBE'],\n});\n```\n\nAdd `INSERT`, `DELETE` and `ALTER` for a principal that writes.\n\nIn Python the new names are `encryption_key`, and the grant options are\nthe keyword arguments `lake_formation_data_access` and\n`catalog_encryption_key`.\n\nUpgrading from a version before 0.17.0: existing `grantRead`,\n`grantWrite` and `grantReadWrite` policies change with no code edit.\nThe Glue statement names the catalog and database ARNs as well as the\ntable, and read grants add `glue:GetDatabase`. Nothing for KMS or Lake\nFormation is issued unless the new prop or options are passed.\n\n### Importing an existing table\n\n```typescript\nconst existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {\n database: db,\n tableName: 'orders',\n location: 's3://my-bucket/analytics/orders/',\n encryptionKey: bucketKey, // optional; grants then cover the key\n});\nexisting.grantRead(role);\n```\n\n### Evolving schema and partitions\n\nChange the `columns` array (or `partitionSpec`) and run `cdk deploy`\nagain. The construct passes the new schema to Glue's `UpdateTable`,\nwhich writes a new `metadata.json` with a new `schema-id`; existing\ndata files stay readable because each column's `id` is pinned to the\nid the table holds and never reused. Adds, renames (same `id`, new\n`name`), and drops all flow through `cdk deploy` alone — no\nout-of-band SQL DDL.\n\nNested fields evolve the same way. Each struct field, list element,\nmap key, and map value keeps the id you gave it (the rule is under\n[Using `IcebergTable`](#using-icebergtable)), so adding a column or a\nstruct field anywhere in the schema leaves every other id unchanged. A\nnew nested field takes an id no field of the table has ever used:\n\n```typescript\ntype: IcebergType.struct([\n { id: 10, name: 'line1', type: IcebergType.STRING },\n { id: 16, name: 'line2', type: IcebergType.STRING }, // new field, never-used id\n { id: 11, name: 'city', type: IcebergType.STRING },\n]),\n```\n\nSuch an id is out of creation order, which is expected once a table\nhas evolved: creation order applies only to the ids a table is created\nwith (see [Using `IcebergTable`](#using-icebergtable)).\n\nUpgrading from a version before 0.12.0, which numbered nested fields\nitself: declare the ids the table's current `metadata.json` holds,\ntop-level and nested (each field `id`, `element-id`, `key-id` and\n`value-id`), or the next deploy renumbers the fields that differ and\nexisting data files read under the wrong field. A nested type object\nshared by two columns now fails synth with a duplicate id; give each\ncolumn its own nested type with its own ids. The list element and map\nvalue nullability default also changed: `elementRequired` and\n`valueRequired` now default to `false`. Earlier versions made list\nelements required by default and map values required unless the third\nargument of `map` was `false`; set `elementRequired: true` or\n`valueRequired: true` wherever the deployed schema has\n`\"element-required\": true` or `\"value-required\": true`.\n\nGlue's `UpdateTable` reuses an earlier schema whose fields (name,\ntype, `required`, `doc`) match the request, together with that\nschema's `identifier-field-ids`. The construct therefore prefixes the\n`doc` of every identifier column with `(PK) ` (just `(PK) ` when the\ncolumn has no `doc`), shown as the column comment, so every\nidentifier change is a field change. On a table whose schema history\nwas written with the marker, identifier changes flow through `cdk\ndeploy` like column changes.\n\nPartition evolution flows the same way, with one extra rule on\n`fieldId`: a changed partition field is a new partition field. To move\na table from `month(placed_at)` to `day(placed_at)`, drop the month\nentry and add a day entry under a fresh `fieldId` — never recycle the\nmonth field's id:\n\n```typescript\npartitionSpec: [\n // was: { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.MONTH, fieldId: 1000 },\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1001,\n },\n],\n```\n\nOld data files written under the month spec stay readable; new writes\npartition by day. Reusing `1000` for the day field instead corrupts\nthe spec history for strict engines (Spark, Trino) while Athena keeps\nworking — the failure only surfaces at Spark runtime.\n\nA changed `sortOrder` also flows through `cdk deploy`. Glue adds it to\n`sort-orders` under the next `order-id` and points\n`default-sort-order-id` at it; the earlier orders stay in the list.\nA change to `tableProperties` alone writes new metadata that keeps the\nsame `current-schema-id`, partition spec and sort order.\n\n### Checking changes against a schema history\n\nThe construct never reads the deployed table, so on its own it cannot\ntell that an id belonged to a field dropped two deploys ago. A schema\nhistory file committed beside the app gives synth that record without\nan AWS call. Point `schemaHistoryFile` at the history file of the\ntable's database:\n\n```typescript\nnew IcebergTable(this, 'Orders', {\n // ...\n schemaHistoryFile: path.join(__dirname, 'analytics.schema-history.json'),\n});\n```\n\nA relative path resolves against the working directory, which is the\ndirectory holding `cdk.json` when the `cdk` CLI runs the app. Create\nthe file once with the content `{\"tables\": []}`; if a table already\nexists in Glue, read\n[Adopting a table that exists in Glue](#adopting-a-table-that-exists-in-glue)\nfirst. Records are keyed by table name, so give each Glue database its\nown file.\n\nEvery synth (`cdk synth`, `cdk diff`, `cdk deploy`, and the CI job\nthat runs one) then compares each table with its record and fails,\nlisting every problem, when the definition:\n\n- gives a column or nested field id in the record to a different\n field: a dropped field's id, or the id of a field whose\n path is still in the definition under another id;\n- keeps a field's path but changes its id, as in the nested ids\n `4, 5, 6` becoming `5, 6, 7` after a column is added;\n- changes a field's type, except `int` to `long`, `float` to `double`,\n and a wider decimal precision at the same scale;\n- makes an optional field required;\n- drops a column that any recorded partition spec was sourced from;\n- keeps a partition `fieldId` with a different source column or\n transform.\n\nFields are identified by id, so a rename passes. A table with no\nrecord is a new table: synth fails unless its ids are the ones Glue\nassigns at creation (the rule is under\n[Using `IcebergTable`](#using-icebergtable)), listing each field with\nits declared id and the id Glue assigns, so the first record of a table\nis written from a creation-order schema. A table with a record is not\nheld to creation order. Synth never writes the file; record a change\nonce it passes synth, and commit the file with the change:\n\n```bash\ncdk synth --quiet\nnode -e \"console.log(require('cdk-glue-iceberg-table').IcebergSchemaHistory.updateFromCloudAssembly('cdk.out'))\"\n```\n\nFrom a Python app, the second command is:\n\n```bash\npython -c \"from cdk_glue_iceberg_table import IcebergSchemaHistory; print(IcebergSchemaHistory.update_from_cloud_assembly('cdk.out'))\"\n```\n\n`updateFromCloudAssembly` records every table of the assembly that\nsets `schemaHistoryFile` in that file and returns the files it wrote.\nThe record keeps every id the table has used: a dropped field stays\nwith `\"dropped\": true`, and a partition field stays after it leaves\nthe spec. The file must already exist, and nothing is written when any\ntable is an unsafe change. A column replaced under the same name needs\ntwo recorded changes: drop it, then add it with a new id.\n\n`IcebergSchemaHistory.validateSchemaEvolution(previous, next)` runs\nthe same checks on one table record without the prop, and\n`IcebergSchemaHistory.append`, `toJson` and `fromJson` record a\ndefinition in, write and read the file format.\n\n#### Adopting a table that exists in Glue\n\nGlue ignores the ids a table declares when it creates the table and\nstores them as given from the first update of any kind. Before\nrecording anything, make the declared ids, top-level and nested, equal\nthe ids in the table's current `metadata.json`, whose location is the\ntable's `metadata_location` parameter (`aws glue get-table`). If those\nids are in creation order, declare them and write no record by hand:\nthat covers a table created in creation order and a table created out\nof order and never updated, which holds creation-order ids whatever it\ndeclared. Otherwise write the record from the definition that now\ndeclares the table's ids; `IcebergSchemaHistory.append` and `toJson`\ndo not run the creation-order check:\n\n```typescript\nconst history = IcebergSchemaHistory.append({ tables: [] }, 'orders', { columns, partitionSpec });\nfs.writeFileSync('analytics.schema-history.json', IcebergSchemaHistory.toJson(history));\n```\n\nFrom a Python app (`partition_spec` is optional, and the history is\npassed on to the next `append` for another table):\n\n```python\nhistory = IcebergSchemaHistory.append({\"tables\": []}, \"orders\", columns=columns, partition_spec=partition_spec)\nwith open(\"analytics.schema-history.json\", \"w\") as f:\n f.write(IcebergSchemaHistory.to_json(tables=history.tables))\n```\n\nRemove a table's record from the file before the table is dropped and\nrecreated: Glue assigns fresh ids at the new CREATE, so the new table\ndeclares creation-order ids.\n\nUpgrading from a version before 0.18.0: an app that sets\n`schemaHistoryFile` and has a table with no record whose ids are not in\ncreation order now fails synth; follow the steps above.\n\n### Inserting and querying\n\n```sql\n-- INSERT into the orders table\nINSERT INTO sample_database.orders VALUES\n (1001, 5001, DECIMAL '149.99', 'USD',\n TIMESTAMP '2026-05-20 09:15:00 UTC',\n ARRAY['holiday-promo','first-order'],\n CAST(ROW('1 Infinite Loop','Cupertino','US','95014')\n AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),\n MAP(ARRAY['channel','utm'], ARRAY['web','google'])),\n -- ... more rows\n;\n\n-- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)\nDELETE FROM sample_database.orders WHERE order_id = 1003;\n\n-- merge-on-read UPDATE\nUPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;\n\n-- SELECT\nSELECT customer_id, SUM(order_amount) AS total\n FROM sample_database.orders\n GROUP BY 1\n ORDER BY 2 DESC;\n```\n\n## Two footguns the construct prevents\n\n### Footgun #1 — schema under `storageDescriptor.columns`\n\nThe CREATE succeeds but the first UPDATE silently strips\n`table_type=ICEBERG` from the table's Glue parameters, and Athena\nqueries after that fail with `HIVE_UNSUPPORTED_FORMAT`.\n\n```typescript\n// DON'T DO THIS — what most StackOverflow / re:Post examples show\nnew CfnTable(this, 'OrdersBad', {\n catalogId: this.account,\n databaseName: 'analytics',\n tableInput: {\n name: 'orders',\n tableType: 'EXTERNAL_TABLE',\n parameters: {\n table_type: 'ICEBERG',\n },\n storageDescriptor: {\n location: 's3://.../orders/',\n columns: [\n /* ... */\n ],\n },\n },\n openTableFormatInput: {\n icebergInput: {\n metadataOperation: 'CREATE',\n version: '2',\n },\n },\n});\n```\n\n`IcebergTable` instead always emits schema/partitions/sort/properties\nunder `openTableFormatInput.icebergInput.icebergTableInput`, never\nunder `storageDescriptor`.\n\n### Footgun #2 — `tableInput` co-present with `openTableFormatInput`\n\nEven setting just `tableInput: { name: 'foo' }` next to\n`openTableFormatInput` returns\n`\"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput\"`.\nThe construct never emits `tableInput`; the table-level comment goes\ninto `tableProperties['comment']`, which lives inside\n`icebergTableInput.properties`.\n\n(There is a third footgun, field-id reuse after a column drop, that\nthe construct prevents only with a [schema history\nfile](#checking-changes-against-a-schema-history). See the next\nsection.)\n\n## Known limitations\n\n- **Creation order is checked only with a schema history file.** Without the file the construct does not detect ids that Glue will reassign at creation; with it, a table with no record is checked ([rule](#checking-changes-against-a-schema-history)). When the first update of such a table changes the id of a column an identity partition is sourced from, Glue rejects the update with `Cannot create identity partition sourced from different field in schema` and the CloudFormation rollback fails the same way, leaving the stack in `UPDATE_ROLLBACK_FAILED`.\n- **Field-id reuse across deploys is detected only with a schema history file.** This applies to nested ids (struct field, list element, map key and value) as well as column ids. If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the \"never reuse a retired id\" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate field id N` validator) and never reads the live table state; across deploys, only a [schema history file](#checking-changes-against-a-schema-history) catches reuse. Without it, treat dropped ids as retired forever.\n- **Partition field-id reuse across deploys is detected only with a schema history file.** The same live-state blindness applies to partition `fieldId`s: the construct validates them within one deploy (integer >= 1000, unique in the spec), and only a [schema history file](#checking-changes-against-a-schema-history) tells it that the id you pinned already appeared in an earlier spec of the deployed table. The invariant partition ids must uphold is stricter than the column one — unique across the table's **entire spec history**, so a changed field (different transform or source column) must take a fresh id rather than keep its old one. Violate it and strict engines (Spark, Trino) reject the table with `Conflicting partition fields` while Athena keeps answering, masking the break until Spark runtime. Without the history file, treat every id that has ever appeared in any spec as retired, and allocate new fields the next never-used id. A partition change needs a new source column or transform; Glue ignores a `fieldId`-only change.\n- **`last-partition-id` is written by Glue, not by the construct.** The construct supplies the spec's field ids; Glue's `UpdateTable` computes the table's `last-partition-id` counter when it writes the new metadata. The repo's integration test asserts the counter keeps up with the highest supplied field id after a partition evolution, but engines that later evolve the spec themselves (e.g. Spark `ALTER TABLE … ADD PARTITION FIELD`) allocate from that counter — if you mix CDK-managed and engine-managed partition evolution, verify the counter in `metadata.json` before trusting engine-side changes.\n- **A rolled-back deploy leaves its schema in the table's history.** When a deploy that adds a column updates the table and then fails on another resource, the stack ends in `UPDATE_ROLLBACK_COMPLETE` and `current-schema-id` is back to its earlier value, so the column is gone and the rows read as before. The schema that held the column stays in `schemas`, and `last-column-id` stays at the column's id. A clean redeploy of the same change makes that schema current again without adding a schema. Treat an id from a rolled-back deploy as used: give it only to the column that deploy added.\n- **Removing the last identifier from a table created with identifiers before 0.11.0 can bring the old ids back.** That table's history holds an unmarked schema with ids, and Glue reuses it when the rendered fields match it again. Change a column `doc` in the same deploy that removes the last identifier.\n- **The schema history check covers only the changes it lists.** It does not see a change an engine makes with `ALTER TABLE` (Spark, Athena), whose ids the file never records, or a deploy whose definition was not recorded in the file. It does not check sort order or identifier field changes or report an added required field, top level or nested, and it treats a field moved into another struct under its id as a rename. It reports the format v3 promotions `unknown` to any type and `date` to `timestamp` or `timestamp_ns` as type changes.\n- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.\n- **Format version is immutable after CREATE.** Glue ignores a V2 to V3 `formatVersion` change, though `properties.format-version` then reads `3`. Keep `formatVersion` at the table's real version, or destroy + recreate the table to change it (remove its [schema history record](#adopting-a-table-that-exists-in-glue) first). An engine can upgrade a v2 table to v3 in place with `ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3')`; after that, set `formatVersion` to `V3` so the next deploy declares the version the table already has.\n- **`merge-on-read` requires v2 or later.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.\n- **Column defaults are not available.** Iceberg v3 `initial-default` and `write-default` cannot be declared through CloudFormation: the `AWS::Glue::Table` resource schema has no `InitialDefault` or `WriteDefault` property. Revisit when that schema gains the two properties.\n- **Nanosecond timestamps are catalog-only for Spark 4.1.1 with Iceberg 1.11.0.** In [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag, `primitive()` has no case for the `TIMESTAMP_NANO` type id, so a table with a `timestamp_ns` or `timestamptz_ns` column reaches the `Cannot convert unsupported type to Spark` error. The integration suite asserts these columns from `metadata.json` only.\n- **Geometry and geography columns are catalog-only for Spark 4.1.1 with Iceberg 1.11.0, the versions AWS Glue 6.0 ships.** The first Spark read of a table holding a `geography` and a `geometry` column failed with `UnsupportedOperationException: Cannot convert unsupported type to Spark: geography(OGC:CRS84, spherical)` on those versions. [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag has no conversion for `geometry` either. The integration suite deploys these columns on a table no Spark job opens and asserts them from `metadata.json` only.\n- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.\n- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change unless a [schema history file](#checking-changes-against-a-schema-history) records the column as a partition source, and Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.\n\n## FAQ\n\n### How do I create an Apache Iceberg table in AWS CDK?\n\nInstall `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a\ndatabase, a column list, and an S3 `location` — see [Use](#use).\n`cdk deploy` creates the Glue Data Catalog table and writes the\nIceberg `metadata.json` under your S3 prefix. No custom resource or\nLambda is involved; it is a plain `AWS::Glue::Table` in your\nCloudFormation template.\n\n### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?\n\nChange the `columns` array and run `cdk deploy` again. The construct\npasses the new schema to Glue's `UpdateTable`, which writes a new\n`metadata.json` with a new `schema-id`; existing data files stay\nreadable because each column's `id` is pinned to the id the table\nholds and never reused.\nPartition changes work the same way, except a changed partition field\ntakes a fresh `fieldId` instead of keeping its old one. See [Evolving\nschema and partitions](#evolving-schema-and-partitions).\n\n### Does CloudFormation support Iceberg tables natively?\n\nYes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,\ndocumented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).\nThe catch is that the raw shape corrupts the table on the first\n`Update` if you place schema under `storageDescriptor.columns` or set\n`tableInput` alongside `openTableFormatInput`. This construct only ever\nemits the safe shape and gives you no way to express the unsafe ones.\nSee [Two footguns the construct prevents](#two-footguns-the-construct-prevents).\n\n### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?\n\nA raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON\nand own both silent-corruption footguns. `cdk-glue-iceberg-table` gives\nyou a typed `IcebergType` / `IcebergPartitionTransform` API,\nsynth-time validation (partition transforms checked against column\ntypes, format/version mismatches caught before deploy), pinned field\nIDs for safe evolution (declared in creation order for a new table),\nand `grantRead` / `grantWrite` helpers. See\n[How it compares](#how-it-compares).\n\n### How do I create a partitioned Iceberg table in CDK?\n\nPass a `partitionSpec` of `IcebergPartitionTransform` entries —\n`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,\nor `void` — each pinned to a `fieldId` of 1000 or above. Transforms are\nvalidated against their source column's type at synth time. See\n[Using `IcebergTable`](#using-icebergtable).\n\n### Does it work with Athena and Lake Formation?\n\nYes for v2 tables, no for v3 (see the engine matrix in [Format\nversion 3](#format-version-3)). The demo app in the\n[`arceus`](https://github.com/ksco92/arceus) repo registers its tables\nwith Lake Formation, and its v2 integration suites query them through\nAthena as well as Spark, on the same tables:\n\n- merge-on-read `INSERT`, `UPDATE`, `DELETE` and `MERGE INTO`, time\n travel (`FOR VERSION AS OF` / `FOR TIMESTAMP AS OF`), `OPTIMIZE ...\n REWRITE DATA USING BIN_PACK`, and `VACUUM`;\n- a read after every schema, partition, identifier and nested-field\n evolution step;\n- the `(PK) ` comment on identifier columns, read from\n `information_schema.columns`;\n- rows written by Spark read through Athena, and rows written by Athena\n read through Spark;\n- reads and writes through Spark and Athena as roles whose table\n permissions come from `grantRead` and `grantReadWrite`, and a denied\n write as the `grantRead` role.\n\nThe suites grant the querying principal `SELECT`, `INSERT`, `DELETE`,\n`ALTER` and `DESCRIBE` through Lake Formation. See [Granting access](#granting-access).\n\n### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?\n\nYes — set `formatVersion: IcebergFormatVersion.V2` and the\n`write.{delete,update,merge}.mode = merge-on-read` table properties.\nThe construct rejects merge-on-read on a v1 table at synth time.\n\n### Can I create an Iceberg v3 table?\n\nYes. Set `formatVersion: IcebergFormatVersion.V3`; see [Format\nversion 3](#format-version-3) for the types, the engine matrix and the\nlimits.\n\n## Contributing\n\nDevelopment, the monorepo layout, and the demo app live in [CONTRIBUTING.md](https://github.com/ksco92/arceus/blob/main/CONTRIBUTING.md).\n\n## License\n\n[MIT](https://github.com/ksco92/arceus/blob/main/LICENSE).\n"
|
|
9755
9755
|
},
|
|
9756
9756
|
"repository": {
|
|
9757
9757
|
"directory": "packages/cdk-glue-iceberg-table",
|
|
@@ -9779,7 +9779,7 @@
|
|
|
9779
9779
|
"kind": "interface",
|
|
9780
9780
|
"locationInModule": {
|
|
9781
9781
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9782
|
-
"line":
|
|
9782
|
+
"line": 244
|
|
9783
9783
|
},
|
|
9784
9784
|
"methods": [
|
|
9785
9785
|
{
|
|
@@ -9790,7 +9790,7 @@
|
|
|
9790
9790
|
},
|
|
9791
9791
|
"locationInModule": {
|
|
9792
9792
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9793
|
-
"line":
|
|
9793
|
+
"line": 261
|
|
9794
9794
|
},
|
|
9795
9795
|
"name": "grantRead",
|
|
9796
9796
|
"parameters": [
|
|
@@ -9822,7 +9822,7 @@
|
|
|
9822
9822
|
},
|
|
9823
9823
|
"locationInModule": {
|
|
9824
9824
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9825
|
-
"line":
|
|
9825
|
+
"line": 267
|
|
9826
9826
|
},
|
|
9827
9827
|
"name": "grantReadWrite",
|
|
9828
9828
|
"parameters": [
|
|
@@ -9854,7 +9854,7 @@
|
|
|
9854
9854
|
},
|
|
9855
9855
|
"locationInModule": {
|
|
9856
9856
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9857
|
-
"line":
|
|
9857
|
+
"line": 264
|
|
9858
9858
|
},
|
|
9859
9859
|
"name": "grantWrite",
|
|
9860
9860
|
"parameters": [
|
|
@@ -9890,7 +9890,7 @@
|
|
|
9890
9890
|
"immutable": true,
|
|
9891
9891
|
"locationInModule": {
|
|
9892
9892
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9893
|
-
"line":
|
|
9893
|
+
"line": 252
|
|
9894
9894
|
},
|
|
9895
9895
|
"name": "database",
|
|
9896
9896
|
"type": {
|
|
@@ -9906,7 +9906,7 @@
|
|
|
9906
9906
|
"immutable": true,
|
|
9907
9907
|
"locationInModule": {
|
|
9908
9908
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9909
|
-
"line":
|
|
9909
|
+
"line": 255
|
|
9910
9910
|
},
|
|
9911
9911
|
"name": "location",
|
|
9912
9912
|
"type": {
|
|
@@ -9922,7 +9922,7 @@
|
|
|
9922
9922
|
"immutable": true,
|
|
9923
9923
|
"locationInModule": {
|
|
9924
9924
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9925
|
-
"line":
|
|
9925
|
+
"line": 246
|
|
9926
9926
|
},
|
|
9927
9927
|
"name": "tableArn",
|
|
9928
9928
|
"type": {
|
|
@@ -9938,7 +9938,7 @@
|
|
|
9938
9938
|
"immutable": true,
|
|
9939
9939
|
"locationInModule": {
|
|
9940
9940
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9941
|
-
"line":
|
|
9941
|
+
"line": 249
|
|
9942
9942
|
},
|
|
9943
9943
|
"name": "tableName",
|
|
9944
9944
|
"type": {
|
|
@@ -9954,7 +9954,7 @@
|
|
|
9954
9954
|
"immutable": true,
|
|
9955
9955
|
"locationInModule": {
|
|
9956
9956
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9957
|
-
"line":
|
|
9957
|
+
"line": 258
|
|
9958
9958
|
},
|
|
9959
9959
|
"name": "encryptionKey",
|
|
9960
9960
|
"optional": true,
|
|
@@ -10164,7 +10164,7 @@
|
|
|
10164
10164
|
"kind": "interface",
|
|
10165
10165
|
"locationInModule": {
|
|
10166
10166
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10167
|
-
"line":
|
|
10167
|
+
"line": 30
|
|
10168
10168
|
},
|
|
10169
10169
|
"name": "IcebergFieldHistory",
|
|
10170
10170
|
"properties": [
|
|
@@ -10178,7 +10178,7 @@
|
|
|
10178
10178
|
"immutable": true,
|
|
10179
10179
|
"locationInModule": {
|
|
10180
10180
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10181
|
-
"line":
|
|
10181
|
+
"line": 44
|
|
10182
10182
|
},
|
|
10183
10183
|
"name": "dropped",
|
|
10184
10184
|
"type": {
|
|
@@ -10194,7 +10194,7 @@
|
|
|
10194
10194
|
"immutable": true,
|
|
10195
10195
|
"locationInModule": {
|
|
10196
10196
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10197
|
-
"line":
|
|
10197
|
+
"line": 32
|
|
10198
10198
|
},
|
|
10199
10199
|
"name": "id",
|
|
10200
10200
|
"type": {
|
|
@@ -10210,7 +10210,7 @@
|
|
|
10210
10210
|
"immutable": true,
|
|
10211
10211
|
"locationInModule": {
|
|
10212
10212
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10213
|
-
"line":
|
|
10213
|
+
"line": 35
|
|
10214
10214
|
},
|
|
10215
10215
|
"name": "path",
|
|
10216
10216
|
"type": {
|
|
@@ -10226,7 +10226,7 @@
|
|
|
10226
10226
|
"immutable": true,
|
|
10227
10227
|
"locationInModule": {
|
|
10228
10228
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10229
|
-
"line":
|
|
10229
|
+
"line": 41
|
|
10230
10230
|
},
|
|
10231
10231
|
"name": "required",
|
|
10232
10232
|
"type": {
|
|
@@ -10242,7 +10242,7 @@
|
|
|
10242
10242
|
"immutable": true,
|
|
10243
10243
|
"locationInModule": {
|
|
10244
10244
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10245
|
-
"line":
|
|
10245
|
+
"line": 38
|
|
10246
10246
|
},
|
|
10247
10247
|
"name": "type",
|
|
10248
10248
|
"type": {
|
|
@@ -10587,7 +10587,7 @@
|
|
|
10587
10587
|
"kind": "interface",
|
|
10588
10588
|
"locationInModule": {
|
|
10589
10589
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10590
|
-
"line":
|
|
10590
|
+
"line": 48
|
|
10591
10591
|
},
|
|
10592
10592
|
"name": "IcebergPartitionFieldHistory",
|
|
10593
10593
|
"properties": [
|
|
@@ -10600,7 +10600,7 @@
|
|
|
10600
10600
|
"immutable": true,
|
|
10601
10601
|
"locationInModule": {
|
|
10602
10602
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10603
|
-
"line":
|
|
10603
|
+
"line": 50
|
|
10604
10604
|
},
|
|
10605
10605
|
"name": "fieldId",
|
|
10606
10606
|
"type": {
|
|
@@ -10616,7 +10616,7 @@
|
|
|
10616
10616
|
"immutable": true,
|
|
10617
10617
|
"locationInModule": {
|
|
10618
10618
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10619
|
-
"line":
|
|
10619
|
+
"line": 53
|
|
10620
10620
|
},
|
|
10621
10621
|
"name": "name",
|
|
10622
10622
|
"type": {
|
|
@@ -10632,7 +10632,7 @@
|
|
|
10632
10632
|
"immutable": true,
|
|
10633
10633
|
"locationInModule": {
|
|
10634
10634
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10635
|
-
"line":
|
|
10635
|
+
"line": 56
|
|
10636
10636
|
},
|
|
10637
10637
|
"name": "sourceId",
|
|
10638
10638
|
"type": {
|
|
@@ -10648,7 +10648,7 @@
|
|
|
10648
10648
|
"immutable": true,
|
|
10649
10649
|
"locationInModule": {
|
|
10650
10650
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10651
|
-
"line":
|
|
10651
|
+
"line": 59
|
|
10652
10652
|
},
|
|
10653
10653
|
"name": "transform",
|
|
10654
10654
|
"type": {
|
|
@@ -11329,7 +11329,7 @@
|
|
|
11329
11329
|
"kind": "class",
|
|
11330
11330
|
"locationInModule": {
|
|
11331
11331
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11332
|
-
"line":
|
|
11332
|
+
"line": 126
|
|
11333
11333
|
},
|
|
11334
11334
|
"methods": [
|
|
11335
11335
|
{
|
|
@@ -11340,7 +11340,7 @@
|
|
|
11340
11340
|
},
|
|
11341
11341
|
"locationInModule": {
|
|
11342
11342
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11343
|
-
"line":
|
|
11343
|
+
"line": 154
|
|
11344
11344
|
},
|
|
11345
11345
|
"name": "append",
|
|
11346
11346
|
"parameters": [
|
|
@@ -11387,7 +11387,7 @@
|
|
|
11387
11387
|
},
|
|
11388
11388
|
"locationInModule": {
|
|
11389
11389
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11390
|
-
"line":
|
|
11390
|
+
"line": 199
|
|
11391
11391
|
},
|
|
11392
11392
|
"name": "fromJson",
|
|
11393
11393
|
"parameters": [
|
|
@@ -11415,7 +11415,7 @@
|
|
|
11415
11415
|
},
|
|
11416
11416
|
"locationInModule": {
|
|
11417
11417
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11418
|
-
"line":
|
|
11418
|
+
"line": 167
|
|
11419
11419
|
},
|
|
11420
11420
|
"name": "toJson",
|
|
11421
11421
|
"parameters": [
|
|
@@ -11444,7 +11444,7 @@
|
|
|
11444
11444
|
},
|
|
11445
11445
|
"locationInModule": {
|
|
11446
11446
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11447
|
-
"line":
|
|
11447
|
+
"line": 209
|
|
11448
11448
|
},
|
|
11449
11449
|
"name": "updateFromCloudAssembly",
|
|
11450
11450
|
"parameters": [
|
|
@@ -11478,7 +11478,7 @@
|
|
|
11478
11478
|
},
|
|
11479
11479
|
"locationInModule": {
|
|
11480
11480
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11481
|
-
"line":
|
|
11481
|
+
"line": 133
|
|
11482
11482
|
},
|
|
11483
11483
|
"name": "validateSchemaEvolution",
|
|
11484
11484
|
"parameters": [
|
|
@@ -11528,7 +11528,7 @@
|
|
|
11528
11528
|
"kind": "interface",
|
|
11529
11529
|
"locationInModule": {
|
|
11530
11530
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11531
|
-
"line":
|
|
11531
|
+
"line": 75
|
|
11532
11532
|
},
|
|
11533
11533
|
"name": "IcebergSchemaHistoryFile",
|
|
11534
11534
|
"properties": [
|
|
@@ -11541,7 +11541,7 @@
|
|
|
11541
11541
|
"immutable": true,
|
|
11542
11542
|
"locationInModule": {
|
|
11543
11543
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11544
|
-
"line":
|
|
11544
|
+
"line": 77
|
|
11545
11545
|
},
|
|
11546
11546
|
"name": "tables",
|
|
11547
11547
|
"type": {
|
|
@@ -11567,7 +11567,7 @@
|
|
|
11567
11567
|
"kind": "interface",
|
|
11568
11568
|
"locationInModule": {
|
|
11569
11569
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11570
|
-
"line":
|
|
11570
|
+
"line": 111
|
|
11571
11571
|
},
|
|
11572
11572
|
"name": "IcebergSchemaViolation",
|
|
11573
11573
|
"properties": [
|
|
@@ -11580,7 +11580,7 @@
|
|
|
11580
11580
|
"immutable": true,
|
|
11581
11581
|
"locationInModule": {
|
|
11582
11582
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11583
|
-
"line":
|
|
11583
|
+
"line": 113
|
|
11584
11584
|
},
|
|
11585
11585
|
"name": "kind",
|
|
11586
11586
|
"type": {
|
|
@@ -11596,7 +11596,7 @@
|
|
|
11596
11596
|
"immutable": true,
|
|
11597
11597
|
"locationInModule": {
|
|
11598
11598
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11599
|
-
"line":
|
|
11599
|
+
"line": 116
|
|
11600
11600
|
},
|
|
11601
11601
|
"name": "message",
|
|
11602
11602
|
"type": {
|
|
@@ -11616,7 +11616,7 @@
|
|
|
11616
11616
|
"kind": "enum",
|
|
11617
11617
|
"locationInModule": {
|
|
11618
11618
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11619
|
-
"line":
|
|
11619
|
+
"line": 90
|
|
11620
11620
|
},
|
|
11621
11621
|
"members": [
|
|
11622
11622
|
{
|
|
@@ -11900,7 +11900,7 @@
|
|
|
11900
11900
|
},
|
|
11901
11901
|
"locationInModule": {
|
|
11902
11902
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11903
|
-
"line":
|
|
11903
|
+
"line": 311
|
|
11904
11904
|
},
|
|
11905
11905
|
"parameters": [
|
|
11906
11906
|
{
|
|
@@ -11929,7 +11929,7 @@
|
|
|
11929
11929
|
"kind": "class",
|
|
11930
11930
|
"locationInModule": {
|
|
11931
11931
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11932
|
-
"line":
|
|
11932
|
+
"line": 284
|
|
11933
11933
|
},
|
|
11934
11934
|
"methods": [
|
|
11935
11935
|
{
|
|
@@ -11939,7 +11939,7 @@
|
|
|
11939
11939
|
},
|
|
11940
11940
|
"locationInModule": {
|
|
11941
11941
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11942
|
-
"line":
|
|
11942
|
+
"line": 390
|
|
11943
11943
|
},
|
|
11944
11944
|
"name": "creationOrderFieldIds",
|
|
11945
11945
|
"parameters": [
|
|
@@ -11978,7 +11978,7 @@
|
|
|
11978
11978
|
},
|
|
11979
11979
|
"locationInModule": {
|
|
11980
11980
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11981
|
-
"line":
|
|
11981
|
+
"line": 400
|
|
11982
11982
|
},
|
|
11983
11983
|
"name": "fromIcebergTableAttributes",
|
|
11984
11984
|
"parameters": [
|
|
@@ -12015,7 +12015,7 @@
|
|
|
12015
12015
|
},
|
|
12016
12016
|
"locationInModule": {
|
|
12017
12017
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12018
|
-
"line":
|
|
12018
|
+
"line": 447
|
|
12019
12019
|
},
|
|
12020
12020
|
"name": "grantRead",
|
|
12021
12021
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -12047,7 +12047,7 @@
|
|
|
12047
12047
|
},
|
|
12048
12048
|
"locationInModule": {
|
|
12049
12049
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12050
|
-
"line":
|
|
12050
|
+
"line": 457
|
|
12051
12051
|
},
|
|
12052
12052
|
"name": "grantReadWrite",
|
|
12053
12053
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -12079,7 +12079,7 @@
|
|
|
12079
12079
|
},
|
|
12080
12080
|
"locationInModule": {
|
|
12081
12081
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12082
|
-
"line":
|
|
12082
|
+
"line": 452
|
|
12083
12083
|
},
|
|
12084
12084
|
"name": "grantWrite",
|
|
12085
12085
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -12115,7 +12115,7 @@
|
|
|
12115
12115
|
"immutable": true,
|
|
12116
12116
|
"locationInModule": {
|
|
12117
12117
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12118
|
-
"line":
|
|
12118
|
+
"line": 292
|
|
12119
12119
|
},
|
|
12120
12120
|
"name": "database",
|
|
12121
12121
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -12131,7 +12131,7 @@
|
|
|
12131
12131
|
"immutable": true,
|
|
12132
12132
|
"locationInModule": {
|
|
12133
12133
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12134
|
-
"line":
|
|
12134
|
+
"line": 301
|
|
12135
12135
|
},
|
|
12136
12136
|
"name": "dataFormat",
|
|
12137
12137
|
"type": {
|
|
@@ -12146,7 +12146,7 @@
|
|
|
12146
12146
|
"immutable": true,
|
|
12147
12147
|
"locationInModule": {
|
|
12148
12148
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12149
|
-
"line":
|
|
12149
|
+
"line": 304
|
|
12150
12150
|
},
|
|
12151
12151
|
"name": "formatVersion",
|
|
12152
12152
|
"type": {
|
|
@@ -12161,7 +12161,7 @@
|
|
|
12161
12161
|
"immutable": true,
|
|
12162
12162
|
"locationInModule": {
|
|
12163
12163
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12164
|
-
"line":
|
|
12164
|
+
"line": 295
|
|
12165
12165
|
},
|
|
12166
12166
|
"name": "location",
|
|
12167
12167
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -12177,7 +12177,7 @@
|
|
|
12177
12177
|
"immutable": true,
|
|
12178
12178
|
"locationInModule": {
|
|
12179
12179
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12180
|
-
"line":
|
|
12180
|
+
"line": 307
|
|
12181
12181
|
},
|
|
12182
12182
|
"name": "resource",
|
|
12183
12183
|
"type": {
|
|
@@ -12192,7 +12192,7 @@
|
|
|
12192
12192
|
"immutable": true,
|
|
12193
12193
|
"locationInModule": {
|
|
12194
12194
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12195
|
-
"line":
|
|
12195
|
+
"line": 286
|
|
12196
12196
|
},
|
|
12197
12197
|
"name": "tableArn",
|
|
12198
12198
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -12208,7 +12208,7 @@
|
|
|
12208
12208
|
"immutable": true,
|
|
12209
12209
|
"locationInModule": {
|
|
12210
12210
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12211
|
-
"line":
|
|
12211
|
+
"line": 289
|
|
12212
12212
|
},
|
|
12213
12213
|
"name": "tableName",
|
|
12214
12214
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -12224,7 +12224,7 @@
|
|
|
12224
12224
|
"immutable": true,
|
|
12225
12225
|
"locationInModule": {
|
|
12226
12226
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12227
|
-
"line":
|
|
12227
|
+
"line": 298
|
|
12228
12228
|
},
|
|
12229
12229
|
"name": "encryptionKey",
|
|
12230
12230
|
"optional": true,
|
|
@@ -12247,7 +12247,7 @@
|
|
|
12247
12247
|
"kind": "interface",
|
|
12248
12248
|
"locationInModule": {
|
|
12249
12249
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12250
|
-
"line":
|
|
12250
|
+
"line": 463
|
|
12251
12251
|
},
|
|
12252
12252
|
"name": "IcebergTableAttributes",
|
|
12253
12253
|
"properties": [
|
|
@@ -12260,7 +12260,7 @@
|
|
|
12260
12260
|
"immutable": true,
|
|
12261
12261
|
"locationInModule": {
|
|
12262
12262
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12263
|
-
"line":
|
|
12263
|
+
"line": 465
|
|
12264
12264
|
},
|
|
12265
12265
|
"name": "database",
|
|
12266
12266
|
"type": {
|
|
@@ -12276,7 +12276,7 @@
|
|
|
12276
12276
|
"immutable": true,
|
|
12277
12277
|
"locationInModule": {
|
|
12278
12278
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12279
|
-
"line":
|
|
12279
|
+
"line": 471
|
|
12280
12280
|
},
|
|
12281
12281
|
"name": "location",
|
|
12282
12282
|
"type": {
|
|
@@ -12292,7 +12292,7 @@
|
|
|
12292
12292
|
"immutable": true,
|
|
12293
12293
|
"locationInModule": {
|
|
12294
12294
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12295
|
-
"line":
|
|
12295
|
+
"line": 468
|
|
12296
12296
|
},
|
|
12297
12297
|
"name": "tableName",
|
|
12298
12298
|
"type": {
|
|
@@ -12309,7 +12309,7 @@
|
|
|
12309
12309
|
"immutable": true,
|
|
12310
12310
|
"locationInModule": {
|
|
12311
12311
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12312
|
-
"line":
|
|
12312
|
+
"line": 474
|
|
12313
12313
|
},
|
|
12314
12314
|
"name": "encryptionKey",
|
|
12315
12315
|
"optional": true,
|
|
@@ -12385,7 +12385,7 @@
|
|
|
12385
12385
|
"kind": "interface",
|
|
12386
12386
|
"locationInModule": {
|
|
12387
12387
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12388
|
-
"line":
|
|
12388
|
+
"line": 63
|
|
12389
12389
|
},
|
|
12390
12390
|
"name": "IcebergTableHistory",
|
|
12391
12391
|
"properties": [
|
|
@@ -12398,7 +12398,7 @@
|
|
|
12398
12398
|
"immutable": true,
|
|
12399
12399
|
"locationInModule": {
|
|
12400
12400
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12401
|
-
"line":
|
|
12401
|
+
"line": 68
|
|
12402
12402
|
},
|
|
12403
12403
|
"name": "fields",
|
|
12404
12404
|
"type": {
|
|
@@ -12419,7 +12419,7 @@
|
|
|
12419
12419
|
"immutable": true,
|
|
12420
12420
|
"locationInModule": {
|
|
12421
12421
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12422
|
-
"line":
|
|
12422
|
+
"line": 71
|
|
12423
12423
|
},
|
|
12424
12424
|
"name": "partitionFields",
|
|
12425
12425
|
"type": {
|
|
@@ -12440,7 +12440,7 @@
|
|
|
12440
12440
|
"immutable": true,
|
|
12441
12441
|
"locationInModule": {
|
|
12442
12442
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12443
|
-
"line":
|
|
12443
|
+
"line": 65
|
|
12444
12444
|
},
|
|
12445
12445
|
"name": "tableName",
|
|
12446
12446
|
"type": {
|
|
@@ -12675,14 +12675,14 @@
|
|
|
12675
12675
|
"abstract": true,
|
|
12676
12676
|
"docs": {
|
|
12677
12677
|
"default": "- no history check",
|
|
12678
|
-
"remarks": "
|
|
12678
|
+
"remarks": "the README section \"Checking changes against a schema history\"\nlists what synth checks against it.",
|
|
12679
12679
|
"stability": "experimental",
|
|
12680
|
-
"summary": "Path of the committed schema history file of the table's database, relative to the working directory
|
|
12680
|
+
"summary": "Path of the committed schema history file of the table's database, relative to the working directory;"
|
|
12681
12681
|
},
|
|
12682
12682
|
"immutable": true,
|
|
12683
12683
|
"locationInModule": {
|
|
12684
12684
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12685
|
-
"line":
|
|
12685
|
+
"line": 240
|
|
12686
12686
|
},
|
|
12687
12687
|
"name": "schemaHistoryFile",
|
|
12688
12688
|
"optional": true,
|
|
@@ -12750,7 +12750,7 @@
|
|
|
12750
12750
|
"kind": "interface",
|
|
12751
12751
|
"locationInModule": {
|
|
12752
12752
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12753
|
-
"line":
|
|
12753
|
+
"line": 81
|
|
12754
12754
|
},
|
|
12755
12755
|
"name": "IcebergTableSchema",
|
|
12756
12756
|
"properties": [
|
|
@@ -12763,7 +12763,7 @@
|
|
|
12763
12763
|
"immutable": true,
|
|
12764
12764
|
"locationInModule": {
|
|
12765
12765
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12766
|
-
"line":
|
|
12766
|
+
"line": 83
|
|
12767
12767
|
},
|
|
12768
12768
|
"name": "columns",
|
|
12769
12769
|
"type": {
|
|
@@ -12784,7 +12784,7 @@
|
|
|
12784
12784
|
"immutable": true,
|
|
12785
12785
|
"locationInModule": {
|
|
12786
12786
|
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12787
|
-
"line":
|
|
12787
|
+
"line": 86
|
|
12788
12788
|
},
|
|
12789
12789
|
"name": "partitionSpec",
|
|
12790
12790
|
"optional": true,
|
|
@@ -14086,6 +14086,6 @@
|
|
|
14086
14086
|
"symbolId": "lib/iceberg/iceberg-type:IcebergTypeOptions"
|
|
14087
14087
|
}
|
|
14088
14088
|
},
|
|
14089
|
-
"version": "0.
|
|
14090
|
-
"fingerprint": "
|
|
14089
|
+
"version": "0.18.0",
|
|
14090
|
+
"fingerprint": "OWf1rZSmSYIKKStrcsHF/tcYlEnyKUhwpxw+fQNY9QU="
|
|
14091
14091
|
}
|