cdk-glue-iceberg-table 0.15.1 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.jsii +808 -120
- package/README.md +72 -4
- package/dist/lib/iceberg/iceberg-partition-transform.js +1 -1
- package/dist/lib/iceberg/iceberg-schema-history.d.ts +117 -0
- package/dist/lib/iceberg/iceberg-schema-history.js +485 -0
- package/dist/lib/iceberg/iceberg-table-properties.js +1 -1
- package/dist/lib/iceberg/iceberg-table-render.d.ts +8 -1
- package/dist/lib/iceberg/iceberg-table-render.js +44 -30
- package/dist/lib/iceberg/iceberg-table.d.ts +9 -1
- package/dist/lib/iceberg/iceberg-table.js +8 -3
- package/dist/lib/iceberg/iceberg-type.d.ts +6 -0
- package/dist/lib/iceberg/iceberg-type.js +40 -26
- package/dist/lib/iceberg/index.d.ts +1 -0
- package/dist/lib/iceberg/index.js +5 -2
- package/package.json +1 -1
package/.jsii
CHANGED
|
@@ -9751,7 +9751,7 @@
|
|
|
9751
9751
|
},
|
|
9752
9752
|
"name": "cdk-glue-iceberg-table",
|
|
9753
9753
|
"readme": {
|
|
9754
|
-
"markdown": "# cdk-glue-iceberg-table\n\n[](https://github.com/ksco92/arceus/actions/workflows/ci.yml)\n[](https://codecov.io/gh/ksco92/arceus)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://pypi.org/project/cdk-glue-iceberg-table/)\n[](https://constructs.dev/packages/cdk-glue-iceberg-table)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://github.com/ksco92/arceus/blob/main/LICENSE)\n\n**`cdk-glue-iceberg-table` is an AWS CDK L2 construct to create and\nevolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and\npartition changes that survive CloudFormation updates, via cdk\ndeploy.** It supports Iceberg format version 3 (`variant`, nanosecond\ntimestamps, `geometry` / `geography`, `unknown`). It emits the exact\n`AWS::Glue::Table` / `OpenTableFormatInput` shape, so a single `cdk\ndeploy` creates, evolves, and destroys a table like any other CDK\nresource. No custom resource, no Lambda, no \"`cdk deploy` and then run\nthis SQL by hand\" two-step. Table changes (new columns, renames, drops,\nnew partition fields) land as a reviewed diff in a pull request and\napply through `cdk deploy`.\n\nStatus (October 2026): published on npm (TypeScript/JavaScript) and\nPyPI (Python), pre-1.0, with a public surface guarded on every PR by two\nconsumer jobs (`e2e-consumer` compiles and synthesizes an app against a\npinned published version, `packed-consumer` does the same against the PR's\nown `npm pack` tarball) plus real-AWS integration suites, including ones\nfor format version 3. The multi-language packages are generated\nfrom one TypeScript source with [jsii](https://aws.github.io/jsii/)\nand indexed on [Construct\nHub](https://constructs.dev/packages/cdk-glue-iceberg-table).\nDeveloped in the [`ksco92/arceus`](https://github.com/ksco92/arceus)\nmonorepo, which also holds a CDK demo app that dogfoods the construct\nagainst a real AWS account.\n\n### Why this exists\n\nAWS documents the CloudFormation shape for Iceberg tables ([AWS Big\nData blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),\nbut raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the\n*first* `Update` silently strips `table_type=ICEBERG`, after which\nevery Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The\nmotivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);\n[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)\ndocuments the only working shape and the silent-corruption traps you\nhit by getting it slightly wrong. This construct implements that\nshape, refuses to emit the unsafe alternatives, and pins Iceberg field\nIDs (declared [in creation order](#using-icebergtable) for a new\ntable) so schema evolution never orphans existing data. It was the\nbasis of an upstream proposal to land an `IcebergTable` in\n`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988)),\nclosed unmerged in September 2026.\n\n### How it compares\n\n| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |\n| --- | --- | --- | --- | --- |\n| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |\n| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |\n| Spark SQL DDL, or Athena SQL DDL, at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |\n| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |\n\n## Install\n\nTypeScript / JavaScript (npm):\n\n```bash\nnpm install cdk-glue-iceberg-table\n```\n\nPython (PyPI):\n\n```bash\npip install cdk-glue-iceberg-table\n```\n\nPeer dependencies (your CDK app must already have these). This\nrelease requires `aws-cdk-lib` 2.273.0 or newer and `constructs`\n10.8.1 or newer, plus Node.js 22.14 or newer for both languages (the\nPython package runs under Node.js through jsii).\n\nFor TypeScript / JavaScript:\n\n```bash\nnpm install aws-cdk-lib@^2.273.0 constructs@^10.8.1\n```\n\nFor Python:\n\n```bash\npip install 'aws-cdk-lib>=2.273.0' 'constructs>=10.8.1'\n```\n\n## Use\n\n### TypeScript\n\n```ts\nimport { Bucket } from 'aws-cdk-lib/aws-s3';\nimport { Database } from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\nconst db = new Database(this, 'Db', { databaseName: 'analytics' });\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n columns: [\n { name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },\n { name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },\n ],\n partitionSpec: [\n { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY, fieldId: 1000 },\n { sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16), fieldId: 1001 },\n ],\n identifierFieldNames: ['order_id'],\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\n### Python\n\nThe same table in Python — the API mirrors TypeScript with\n`snake_case` props and `PascalCase` types:\n\n```python\nfrom aws_cdk.aws_s3 import Bucket\nfrom aws_cdk.aws_glue import Database\nfrom cdk_glue_iceberg_table import (\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n)\n\nbucket = Bucket(self, \"Warehouse\")\ndb = Database(self, \"Db\", database_name=\"analytics\")\n\nIcebergTable(self, \"OrdersTable\",\n database=db,\n table_name=\"orders\",\n location=f\"s3://{bucket.bucket_name}/analytics/orders/\",\n columns=[\n {\"name\": \"order_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 1},\n {\"name\": \"customer_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 2},\n {\"name\": \"placed_at\", \"type\": IcebergType.TIMESTAMPTZ, \"required\": True, \"id\": 3},\n ],\n partition_spec=[\n {\"source_column\": \"placed_at\", \"transform\": IcebergPartitionTransform.DAY, \"field_id\": 1000},\n {\"source_column\": \"customer_id\", \"transform\": IcebergPartitionTransform.bucket(16), \"field_id\": 1001},\n ],\n identifier_field_names=[\"order_id\"],\n format_version=IcebergFormatVersion.V2,\n)\n```\n\nConsumer-facing reference sections:\n\n- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.\n- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.\n- [Format version 3](#format-version-3) — the six v3 types, which engines read them, and the accepted limits.\n- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.\n- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.\n- [FAQ](#faq) — common \"how do I … in CDK / CloudFormation\" questions.\n\n## Exported surface\n\nThe package entry point re-exports the main types you import from\n`cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled\nsurface lives under `dist/lib/iceberg/` after `npm install`; in Python\nthe same surface is imported from the `cdk_glue_iceberg_table` module\nafter `pip install`:\n\n- **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite`, the `fromIcebergTableAttributes(...)` import factory, and `creationOrderFieldIds(columns)` (Python: `creation_order_field_ids`), which returns a map from field path to the id Glue gives that field in a new table; a name containing `.` or a backquote appears in backquotes.\n- **`IcebergType`** — primitive types, the v3 constants `UNKNOWN`, `TIMESTAMP_NS`, `TIMESTAMPTZ_NS`, and `VARIANT`, and the `list` / `map` / `struct` / `decimal` / `fixed` / `geometry(crs?)` / `geography(crs?, algorithm?)` factories. `list` takes an `IcebergListDefinition`, `map` an `IcebergMapDefinition`, and `struct` a list of `IcebergStructFieldDefinition`, each carrying the nested field ids. Renders to the string Glue's `IcebergStructField.type` expects.\n- **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.\n- **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2/v3, required — set explicitly per table; Athena cannot query v3 tables), **`IcebergSortDirection`**, **`IcebergNullOrder`**, **`IcebergEdgeAlgorithm`** (`SPHERICAL`, `VINCENTY`, `THOMAS`, `ANDOYER`, `KARNEY`; the second argument of `IcebergType.geography`), plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).\n\n## Using `IcebergTable`\n\nA minimal table:\n\n```typescript\nimport {\n Bucket,\n} from 'aws-cdk-lib/aws-s3';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\n\nconst db = new Database(this, 'Db', {\n databaseName: 'analytics',\n});\n\nnew IcebergTable(this, 'Users', {\n database: db,\n tableName: 'users',\n columns: [\n {\n name: 'user_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'email',\n type: IcebergType.STRING,\n required: true,\n id: 2,\n },\n {\n name: 'signed_up_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 3,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/users/`,\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\nEvery top-level column must declare an `id`: a stable, unique,\npositive integer you pick once and never reuse or reassign. Because\nIceberg data files reference fields by id rather than by name, a fixed\nid, declared in creation order when the table is created (below), is\nwhat lets you add, rename, and drop columns across deploys without\ncorrupting old data — see\n[How do I evolve an Iceberg table schema](#how-do-i-evolve-an-iceberg-table-schema-add-rename-or-drop-a-column-in-cloudformation).\nOnce a column is dropped, leave its id retired; never hand it to a new\ncolumn. Nested fields follow the same rule: every struct field takes\nan `id`, every list an `elementId`, and every map a `keyId` and a\n`valueId`. All ids in a schema, top-level and nested, must be distinct\nintegers from 1 to 2147483447.\n\nA new table must declare its ids in creation order. When Glue creates\na table it does not keep the declared ids: it numbers the fields from\n1 in creation order and points the partition spec, sort order and\nidentifier fields at the new ids. An update Glue accepts stores the\ndeclared ids as given, so the first such update of a table created\nwith other ids renumbers its fields; for the update Glue rejects, see\n[Known limitations](#known-limitations). To number a new table, number\nthe top-level columns first, in order, then the nested fields of each\ncolumn in turn (a list's element; a map's key, then its value, then\nthe key's nested fields, then the value's; a struct's fields, then\ntheir nested fields). `IcebergTable.creationOrderFieldIds` returns\nthese ids by field path and ignores the ids in its input:\n\n```typescript\nconst columns = [\n { name: 'id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'tags', type: IcebergType.list({ element: IcebergType.STRING, elementId: 4 }), id: 2 },\n {\n name: 'attrs',\n type: IcebergType.map({ key: IcebergType.STRING, keyId: 5, value: IcebergType.STRING, valueId: 6 }),\n id: 3,\n },\n];\nIcebergTable.creationOrderFieldIds(columns);\n// { id: 1, tags: 2, attrs: 3, 'tags.element': 4, 'attrs.key': 5, 'attrs.value': 6 }\n```\n\nIds out of creation order are for fields a later deploy adds; see\n[Evolving schema and partitions](#evolving-schema-and-partitions).\n\nA table that exercises most of the surface (partitions, sort order,\nnested types, identifier fields, table properties, removal policy):\n\n```typescript\nimport {\n RemovalPolicy,\n} from 'aws-cdk-lib';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergDataFormat,\n IcebergFormatVersion,\n IcebergNullOrder,\n IcebergPartitionTransform,\n IcebergSortDirection,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',\n columns: [\n {\n name: 'order_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'customer_id',\n type: IcebergType.LONG,\n required: true,\n id: 2,\n },\n {\n name: 'order_amount',\n type: IcebergType.decimal(12, 2),\n required: true,\n id: 3,\n },\n {\n name: 'currency',\n type: IcebergType.STRING,\n required: true,\n id: 4,\n },\n {\n name: 'placed_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 5,\n },\n {\n name: 'tags',\n type: IcebergType.list({\n element: IcebergType.STRING,\n elementId: 9,\n }),\n id: 6,\n },\n {\n name: 'shipping_address',\n type: IcebergType.struct([\n {\n id: 10,\n name: 'line1',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 11,\n name: 'city',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 12,\n name: 'country',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 13,\n name: 'postal_code',\n type: IcebergType.STRING,\n },\n ]),\n id: 7,\n },\n {\n name: 'metadata',\n type: IcebergType.map({\n key: IcebergType.STRING,\n keyId: 14,\n value: IcebergType.STRING,\n valueId: 15,\n }),\n id: 8,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n partitionSpec: [\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1000,\n },\n {\n sourceColumn: 'customer_id',\n transform: IcebergPartitionTransform.bucket(16),\n fieldId: 1001,\n },\n ],\n sortOrder: [\n {\n sourceColumn: 'placed_at',\n direction: IcebergSortDirection.ASC,\n nullOrder: IcebergNullOrder.NULLS_LAST,\n },\n {\n sourceColumn: 'order_id',\n direction: IcebergSortDirection.ASC,\n },\n ],\n identifierFieldNames: [\n 'order_id',\n ],\n dataFormat: IcebergDataFormat.PARQUET,\n formatVersion: IcebergFormatVersion.V2,\n tableProperties: {\n 'write.parquet.compression-codec': 'zstd',\n 'write.delete.mode': 'merge-on-read',\n 'write.update.mode': 'merge-on-read',\n 'write.merge.mode': 'merge-on-read',\n 'write.target-file-size-bytes': '134217728',\n 'history.expire.min-snapshots-to-keep': '5',\n 'gc.enabled': 'true',\n },\n removalPolicy: RemovalPolicy.DESTROY,\n});\n```\n\nEach partition field also pins a `fieldId`: an integer of 1000 or\nabove (the Iceberg spec reserves lower ids for schema columns), unique\nacross the table's entire partition-spec history. The discipline is\nthe inverse of the column-id rule. A column keeps its id forever; a\npartition field keeps its id only while it stays untouched. Change\nanything about a field — swap `month` for `day`, point it at a\ndifferent source column — and it becomes a *new* field that needs a\nfresh, never-before-used id, while the old id retires with the old\nspec. Reusing a retired id (or reassigning an id to a different field)\nproduces a table that Athena still reads but Spark and Trino reject\nwith `ValidationException: Conflicting partition fields`.\n\n`identifierFieldNames` lists the columns that together identify a row\nand renders as the schema's `identifier-field-ids`. Each one must be a\n`required: true` column (Glue rejects an optional one), and `cdk synth`\nrejects an identifier column of type float, double, list, map, struct,\nunknown, variant, geometry or geography. The construct prefixes each\nidentifier column's `doc` with `(PK) `, so a `doc` you write must not\nstart with `(PK) `; see [Evolving schema and\npartitions](#evolving-schema-and-partitions).\n\nEvery string you pass is a literal string, not a token, and an\nenum-typed one must be a member of its enum; the exceptions are\n`location`, `comment`, table property values, and the `tableName` of\nan imported table.\n\nThe resulting Iceberg `metadata.json` for this table contains every\nfeature you set:\n\n```json\n{\n \"format-version\": 2,\n \"table-uuid\": \"39a948f9-...\",\n \"current-schema-id\": 0,\n \"schemas\": [\n {\n \"schema-id\": 0,\n \"identifier-field-ids\": [1],\n \"fields\": [\n { \"id\": 1, \"name\": \"order_id\", \"required\": true, \"type\": \"long\", \"doc\": \"(PK) \" },\n { \"id\": 2, \"name\": \"customer_id\", \"required\": true, \"type\": \"long\" },\n { \"id\": 3, \"name\": \"order_amount\", \"required\": true, \"type\": \"decimal(12, 2)\" },\n { \"id\": 4, \"name\": \"currency\", \"required\": true, \"type\": \"string\" },\n { \"id\": 5, \"name\": \"placed_at\", \"required\": true, \"type\": \"timestamptz\" },\n { \"id\": 6, \"name\": \"tags\", \"required\": false,\n \"type\": { \"type\": \"list\", \"element-id\": 9, \"element\": \"string\", \"element-required\": false } },\n { \"id\": 7, \"name\": \"shipping_address\", \"required\": false,\n \"type\": { \"type\": \"struct\", \"fields\": [\n { \"id\": 10, \"name\": \"line1\", \"required\": true, \"type\": \"string\" },\n { \"id\": 11, \"name\": \"city\", \"required\": true, \"type\": \"string\" },\n { \"id\": 12, \"name\": \"country\", \"required\": true, \"type\": \"string\" },\n { \"id\": 13, \"name\": \"postal_code\", \"required\": false, \"type\": \"string\" }\n ] } },\n { \"id\": 8, \"name\": \"metadata\", \"required\": false,\n \"type\": { \"type\": \"map\", \"key-id\": 14, \"key\": \"string\", \"value-id\": 15,\n \"value-required\": false, \"value\": \"string\" } }\n ]\n }\n ],\n \"partition-specs\": [\n { \"spec-id\": 0, \"fields\": [\n { \"name\": \"placed_at_day\", \"transform\": \"day\", \"source-id\": 5, \"field-id\": 1000 },\n { \"name\": \"customer_id_bucket\", \"transform\": \"bucket[16]\", \"source-id\": 2, \"field-id\": 1001 }\n ]}\n ],\n \"sort-orders\": [\n { \"order-id\": 1, \"fields\": [\n { \"transform\": \"identity\", \"source-id\": 5, \"direction\": \"asc\", \"null-order\": \"nulls-last\" },\n { \"transform\": \"identity\", \"source-id\": 1, \"direction\": \"asc\", \"null-order\": \"nulls-last\" }\n ]}\n ],\n \"properties\": {\n \"format-version\": \"2\",\n \"write.format.default\": \"parquet\",\n \"write.parquet.compression-codec\": \"zstd\",\n \"write.merge.mode\": \"merge-on-read\",\n \"write.update.mode\": \"merge-on-read\",\n \"write.delete.mode\": \"merge-on-read\",\n \"write.target-file-size-bytes\": \"134217728\",\n \"history.expire.min-snapshots-to-keep\": \"5\",\n \"gc.enabled\": \"true\",\n \"comment\": \"Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.\"\n }\n}\n```\n\n### Format version 3\n\nSet `formatVersion: IcebergFormatVersion.V3` to create a v3 table. The\nsix v3 types (below) are only accepted on a V3 table, at any nesting\ndepth: inside a struct, as a list element, or as a map key or value.\nSynth fails on a V1 or V2 table that uses one.\n\n```typescript\nimport {\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'Events', {\n database: db,\n tableName: 'events',\n location: `s3://${bucket.bucketName}/analytics/events/`,\n columns: [\n { name: 'occurred_at', type: IcebergType.TIMESTAMP_NS, id: 1 },\n { name: 'payload', type: IcebergType.VARIANT, id: 2 },\n {\n name: 'origin',\n type: IcebergType.geography('OGC:CRS84', IcebergEdgeAlgorithm.SPHERICAL),\n id: 3,\n },\n ],\n formatVersion: IcebergFormatVersion.V3,\n});\n```\n\nThe same table in Python:\n\n```python\nfrom cdk_glue_iceberg_table import (\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n)\n\nIcebergTable(self, \"Events\",\n database=db,\n table_name=\"events\",\n location=f\"s3://{bucket.bucket_name}/analytics/events/\",\n columns=[\n {\"name\": \"occurred_at\", \"type\": IcebergType.TIMESTAMP_NS, \"id\": 1},\n {\"name\": \"payload\", \"type\": IcebergType.VARIANT, \"id\": 2},\n {\n \"name\": \"origin\",\n \"type\": IcebergType.geography(\"OGC:CRS84\", IcebergEdgeAlgorithm.SPHERICAL),\n \"id\": 3,\n },\n ],\n format_version=IcebergFormatVersion.V3,\n)\n```\n\nThis table has a `timestamp_ns` column; see [Known limitations](#known-limitations)\nbefore reading it from upstream Spark.\n\nThe v3 types, with the rules `cdk synth` enforces:\n\n| Type | Factory | Rendered string | Partition and identifier rules |\n| --- | --- | --- | --- |\n| unknown | `IcebergType.UNKNOWN` | `unknown` | Must be optional, as a column, list element, or map value; never a map key. Partition source only with `void`. Not an identifier field. |\n| timestamp with nanoseconds | `IcebergType.TIMESTAMP_NS` | `timestamp_ns` | `identity`, `year`, `month`, `day`, `hour`, `bucket(N)`, `void`; not `truncate(W)`. Allowed as an identifier field. |\n| timestamptz with nanoseconds | `IcebergType.TIMESTAMPTZ_NS` | `timestamptz_ns` | Same as `timestamp_ns`. |\n| variant | `IcebergType.VARIANT` | `variant` | Never a partition or sort source, `void` included. Not an identifier field. |\n| geometry | `IcebergType.geometry(crs?)` | `geometry`, or `geometry(<crs>)` when a CRS is given | Partition source only with `void`. Not an identifier field. |\n| geography | `IcebergType.geography(crs?, algorithm?)` | `geography`, `geography(<crs>)`, or `geography(<crs>, <algorithm>)` when an algorithm is given, such as `geography(OGC:CRS84, spherical)` | Same as `geometry`. |\n\nA geometry or geography `crs` is an identifier such as `OGC:CRS84` or\n`EPSG:3857`; whitespace, parentheses, braces, and commas are rejected.\nWhen `algorithm` is set without a `crs`, the rendered string spells out\n`OGC:CRS84`.\n\nWhich engines read a v3 table created by the construct:\n\n| Engine | v3 tables | Notes |\n| --- | --- | --- |\n| Spark 4.1.1 + Iceberg 1.11.0 (the integration-test runner) | Yes | Reads and writes tables the construct creates, including deletion vectors, row lineage, `variant` and `unknown`, except tables with a `geometry`, `geography` or nanosecond-timestamp column ([Known limitations](#known-limitations)). |\n| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists \"Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types\". The integration suites run on Glue 6.0's Spark 4.1.1 and Iceberg 1.11.0 (`scripts/spark/spark-env.sh`); the Glue job runtime itself is not exercised. |\n| Amazon EMR Spark 8.1.0 (Spark 4.1.1, Iceberg 1.11.0-amzn-0) | Documented | The [emr-spark-8.1.0 release notes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html) state \"Amazon EMR Spark 8.1.0 extends Iceberg v3 spec support for Spark workloads\". The integration suites do not run on EMR 8.1. |\n| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states \"Athena only creates and operates on Iceberg v2 tables.\" The v2 integration suites also run through Athena (see [Does it work with Athena and Lake Formation?](#does-it-work-with-athena-and-lake-formation)); no v3 table is queried through Athena. |\n\n### Granting access\n\n```typescript\ntable.grantRead(role); // Glue read + S3 read on the table's prefix\ntable.grantWrite(role); // Glue write + S3 write\ntable.grantReadWrite(role);\n```\n\nThe `grant*` helpers issue **IAM grants only**. Under Lake Formation\nyou still add the matching `SELECT` / `INSERT` / `DELETE` LF grants on\ntop of the construct's IAM grants for Athena queries to succeed.\n\n### Importing an existing table\n\n```typescript\nconst existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {\n database: db,\n tableName: 'orders',\n location: 's3://my-bucket/analytics/orders/',\n});\nexisting.grantRead(role);\n```\n\n### Evolving schema and partitions\n\nChange the `columns` array (or `partitionSpec`) and run `cdk deploy`\nagain. The construct passes the new schema to Glue's `UpdateTable`,\nwhich writes a new `metadata.json` with a new `schema-id`; existing\ndata files stay readable because each column's `id` is pinned to the\nid the table holds and never reused. Adds, renames (same `id`, new\n`name`), and drops all flow through `cdk deploy` alone — no\nout-of-band SQL DDL.\n\nNested fields evolve the same way. Each struct field, list element,\nmap key, and map value keeps the id you gave it (the rule is under\n[Using `IcebergTable`](#using-icebergtable)), so adding a column or a\nstruct field anywhere in the schema leaves every other id unchanged. A\nnew nested field takes an id no field of the table has ever used:\n\n```typescript\ntype: IcebergType.struct([\n { id: 10, name: 'line1', type: IcebergType.STRING },\n { id: 16, name: 'line2', type: IcebergType.STRING }, // new field, never-used id\n { id: 11, name: 'city', type: IcebergType.STRING },\n]),\n```\n\nSuch an id is out of creation order, which is expected once a table\nhas evolved: creation order applies only to the ids a table is created\nwith (see [Using `IcebergTable`](#using-icebergtable)).\n\nUpgrading from a version before 0.12.0, which numbered nested fields\nitself: declare the ids the table's current `metadata.json` holds,\ntop-level and nested (each field `id`, `element-id`, `key-id` and\n`value-id`), or the next deploy renumbers the fields that differ and\nexisting data files read under the wrong field. A nested type object\nshared by two columns now fails synth with a duplicate id; give each\ncolumn its own nested type with its own ids. The list element and map\nvalue nullability default also changed: `elementRequired` and\n`valueRequired` now default to `false`. Earlier versions made list\nelements required by default and map values required unless the third\nargument of `map` was `false`; set `elementRequired: true` or\n`valueRequired: true` wherever the deployed schema has\n`\"element-required\": true` or `\"value-required\": true`.\n\nGlue's `UpdateTable` reuses an earlier schema whose fields (name,\ntype, `required`, `doc`) match the request, together with that\nschema's `identifier-field-ids`. The construct therefore prefixes the\n`doc` of every identifier column with `(PK) ` (just `(PK) ` when the\ncolumn has no `doc`), shown as the column comment, so every\nidentifier change is a field change. On a table whose schema history\nwas written with the marker, identifier changes flow through `cdk\ndeploy` like column changes.\n\nPartition evolution flows the same way, with one extra rule on\n`fieldId`: a changed partition field is a new partition field. To move\na table from `month(placed_at)` to `day(placed_at)`, drop the month\nentry and add a day entry under a fresh `fieldId` — never recycle the\nmonth field's id:\n\n```typescript\npartitionSpec: [\n // was: { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.MONTH, fieldId: 1000 },\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1001,\n },\n],\n```\n\nOld data files written under the month spec stay readable; new writes\npartition by day. Reusing `1000` for the day field instead corrupts\nthe spec history for strict engines (Spark, Trino) while Athena keeps\nworking — the failure only surfaces at Spark runtime.\n\nA changed `sortOrder` also flows through `cdk deploy`. Glue adds it to\n`sort-orders` under the next `order-id` and points\n`default-sort-order-id` at it; the earlier orders stay in the list.\nA change to `tableProperties` alone writes new metadata that keeps the\nsame `current-schema-id`, partition spec and sort order.\n\n### Inserting and querying\n\n```sql\n-- INSERT into the orders table\nINSERT INTO sample_database.orders VALUES\n (1001, 5001, DECIMAL '149.99', 'USD',\n TIMESTAMP '2026-05-20 09:15:00 UTC',\n ARRAY['holiday-promo','first-order'],\n CAST(ROW('1 Infinite Loop','Cupertino','US','95014')\n AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),\n MAP(ARRAY['channel','utm'], ARRAY['web','google'])),\n -- ... more rows\n;\n\n-- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)\nDELETE FROM sample_database.orders WHERE order_id = 1003;\n\n-- merge-on-read UPDATE\nUPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;\n\n-- SELECT\nSELECT customer_id, SUM(order_amount) AS total\n FROM sample_database.orders\n GROUP BY 1\n ORDER BY 2 DESC;\n```\n\n## Two footguns the construct prevents\n\n### Footgun #1 — schema under `storageDescriptor.columns`\n\nThe CREATE succeeds but the first UPDATE silently strips\n`table_type=ICEBERG` from the table's Glue parameters, and Athena\nqueries after that fail with `HIVE_UNSUPPORTED_FORMAT`.\n\n```typescript\n// DON'T DO THIS — what most StackOverflow / re:Post examples show\nnew CfnTable(this, 'OrdersBad', {\n catalogId: this.account,\n databaseName: 'analytics',\n tableInput: {\n name: 'orders',\n tableType: 'EXTERNAL_TABLE',\n parameters: {\n table_type: 'ICEBERG',\n },\n storageDescriptor: {\n location: 's3://.../orders/',\n columns: [\n /* ... */\n ],\n },\n },\n openTableFormatInput: {\n icebergInput: {\n metadataOperation: 'CREATE',\n version: '2',\n },\n },\n});\n```\n\n`IcebergTable` instead always emits schema/partitions/sort/properties\nunder `openTableFormatInput.icebergInput.icebergTableInput`, never\nunder `storageDescriptor`.\n\n### Footgun #2 — `tableInput` co-present with `openTableFormatInput`\n\nEven setting just `tableInput: { name: 'foo' }` next to\n`openTableFormatInput` returns\n`\"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput\"`.\nThe construct never emits `tableInput`; the table-level comment goes\ninto `tableProperties['comment']`, which lives inside\n`icebergTableInput.properties`.\n\n(There is a third footgun, field-id reuse after a column drop, that\nthe construct does **not** prevent. See the next section.)\n\n## Known limitations\n\n- **Creation order is not checked.** The construct cannot tell at synth whether a table is new or already deployed, so it accepts a new table whose ids are not in creation order (the rule is under [Using `IcebergTable`](#using-icebergtable)). When the first update of such a table changes the id of a column an identity partition is sourced from, Glue rejects the update with `Cannot create identity partition sourced from different field in schema` and the CloudFormation rollback fails the same way, leaving the stack in `UPDATE_ROLLBACK_FAILED`.\n- **Field-id reuse is not detected across deploys.** This applies to nested ids (struct field, list element, map key and value) as well as column ids. If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the \"never reuse a retired id\" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate field id N` validator), but it doesn't compare against the live table state. The safe workflow is to treat dropped ids as retired forever.\n- **Partition field-id reuse is not detected across deploys either.** The same live-state blindness applies to partition `fieldId`s: the construct validates them within one deploy (integer >= 1000, unique in the spec), but it cannot tell that the id you pinned already appeared in an earlier spec of the deployed table. The invariant partition ids must uphold is stricter than the column one — unique across the table's **entire spec history**, so a changed field (different transform or source column) must take a fresh id rather than keep its old one. Violate it and strict engines (Spark, Trino) reject the table with `Conflicting partition fields` while Athena keeps answering, masking the break until Spark runtime. The safe workflow: treat every id that has ever appeared in any spec as retired, and allocate new fields the next never-used id. A partition change needs a new source column or transform; Glue ignores a `fieldId`-only change.\n- **`last-partition-id` is written by Glue, not by the construct.** The construct supplies the spec's field ids; Glue's `UpdateTable` computes the table's `last-partition-id` counter when it writes the new metadata. The repo's integration test asserts the counter keeps up with the highest supplied field id after a partition evolution, but engines that later evolve the spec themselves (e.g. Spark `ALTER TABLE … ADD PARTITION FIELD`) allocate from that counter — if you mix CDK-managed and engine-managed partition evolution, verify the counter in `metadata.json` before trusting engine-side changes.\n- **A rolled-back deploy leaves its schema in the table's history.** When a deploy that adds a column updates the table and then fails on another resource, the stack ends in `UPDATE_ROLLBACK_COMPLETE` and `current-schema-id` is back to its earlier value, so the column is gone and the rows read as before. The schema that held the column stays in `schemas`, and `last-column-id` stays at the column's id. A clean redeploy of the same change makes that schema current again without adding a schema. Treat an id from a rolled-back deploy as used: give it only to the column that deploy added.\n- **Removing the last identifier from a table created with identifiers before 0.11.0 can bring the old ids back.** That table's history holds an unmarked schema with ids, and Glue reuses it when the rendered fields match it again. Change a column `doc` in the same deploy that removes the last identifier.\n- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.\n- **Format version is immutable after CREATE.** Glue ignores a V2 to V3 `formatVersion` change, though `properties.format-version` then reads `3`. Keep `formatVersion` at the table's real version, or destroy + recreate the table to change it. An engine can upgrade a v2 table to v3 in place with `ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3')`; after that, set `formatVersion` to `V3` so the next deploy declares the version the table already has.\n- **`merge-on-read` requires v2 or later.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.\n- **Column defaults are not available.** Iceberg v3 `initial-default` and `write-default` cannot be declared through CloudFormation: the `AWS::Glue::Table` resource schema has no `InitialDefault` or `WriteDefault` property. Revisit when that schema gains the two properties.\n- **Nanosecond timestamps are catalog-only for Spark 4.1.1 with Iceberg 1.11.0.** In [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag, `primitive()` has no case for the `TIMESTAMP_NANO` type id, so a table with a `timestamp_ns` or `timestamptz_ns` column reaches the `Cannot convert unsupported type to Spark` error. The integration suite asserts these columns from `metadata.json` only.\n- **Geometry and geography columns are catalog-only for Spark 4.1.1 with Iceberg 1.11.0, the versions AWS Glue 6.0 ships.** The first Spark read of a table holding a `geography` and a `geometry` column failed with `UnsupportedOperationException: Cannot convert unsupported type to Spark: geography(OGC:CRS84, spherical)` on those versions. [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag has no conversion for `geometry` either. The integration suite deploys these columns on a table no Spark job opens and asserts them from `metadata.json` only.\n- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.\n- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change, but Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.\n\n## FAQ\n\n### How do I create an Apache Iceberg table in AWS CDK?\n\nInstall `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a\ndatabase, a column list, and an S3 `location` — see [Use](#use).\n`cdk deploy` creates the Glue Data Catalog table and writes the\nIceberg `metadata.json` under your S3 prefix. No custom resource or\nLambda is involved; it is a plain `AWS::Glue::Table` in your\nCloudFormation template.\n\n### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?\n\nChange the `columns` array and run `cdk deploy` again. The construct\npasses the new schema to Glue's `UpdateTable`, which writes a new\n`metadata.json` with a new `schema-id`; existing data files stay\nreadable because each column's `id` is pinned to the id the table\nholds and never reused.\nPartition changes work the same way, except a changed partition field\ntakes a fresh `fieldId` instead of keeping its old one. See [Evolving\nschema and partitions](#evolving-schema-and-partitions).\n\n### Does CloudFormation support Iceberg tables natively?\n\nYes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,\ndocumented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).\nThe catch is that the raw shape corrupts the table on the first\n`Update` if you place schema under `storageDescriptor.columns` or set\n`tableInput` alongside `openTableFormatInput`. This construct only ever\nemits the safe shape and gives you no way to express the unsafe ones.\nSee [Two footguns the construct prevents](#two-footguns-the-construct-prevents).\n\n### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?\n\nA raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON\nand own both silent-corruption footguns. `cdk-glue-iceberg-table` gives\nyou a typed `IcebergType` / `IcebergPartitionTransform` API,\nsynth-time validation (partition transforms checked against column\ntypes, format/version mismatches caught before deploy), pinned field\nIDs for safe evolution (declared in creation order for a new table),\nand `grantRead` / `grantWrite` helpers. See\n[How it compares](#how-it-compares).\n\n### How do I create a partitioned Iceberg table in CDK?\n\nPass a `partitionSpec` of `IcebergPartitionTransform` entries —\n`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,\nor `void` — each pinned to a `fieldId` of 1000 or above. Transforms are\nvalidated against their source column's type at synth time. See\n[Using `IcebergTable`](#using-icebergtable).\n\n### Does it work with Athena and Lake Formation?\n\nYes for v2 tables, no for v3 (see the engine matrix in [Format\nversion 3](#format-version-3)). The demo app in the\n[`arceus`](https://github.com/ksco92/arceus) repo registers its tables\nwith Lake Formation, and its v2 integration suites query them through\nAthena as well as Spark, on the same tables:\n\n- merge-on-read `INSERT`, `UPDATE`, `DELETE` and `MERGE INTO`, time\n travel (`FOR VERSION AS OF` / `FOR TIMESTAMP AS OF`), `OPTIMIZE ...\n REWRITE DATA USING BIN_PACK`, and `VACUUM`;\n- a read after every schema, partition, identifier and nested-field\n evolution step;\n- the `(PK) ` comment on identifier columns, read from\n `information_schema.columns`;\n- rows written by Spark read through Athena, and rows written by Athena\n read through Spark.\n\nThe suites grant the querying principal `SELECT`, `INSERT`, `DELETE`,\n`ALTER` and `DESCRIBE` through Lake Formation. The construct's `grant*`\nhelpers issue IAM grants only; under Lake Formation you add the\nmatching LF grants. See [Granting access](#granting-access).\n\n### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?\n\nYes — set `formatVersion: IcebergFormatVersion.V2` and the\n`write.{delete,update,merge}.mode = merge-on-read` table properties.\nThe construct rejects merge-on-read on a v1 table at synth time.\n\n### Can I create an Iceberg v3 table?\n\nYes. Set `formatVersion: IcebergFormatVersion.V3`; see [Format\nversion 3](#format-version-3) for the types, the engine matrix and the\nlimits.\n\n## Contributing\n\nDevelopment, the monorepo layout, and the demo app live in [CONTRIBUTING.md](https://github.com/ksco92/arceus/blob/main/CONTRIBUTING.md).\n\n## License\n\n[MIT](https://github.com/ksco92/arceus/blob/main/LICENSE).\n"
|
|
9754
|
+
"markdown": "# cdk-glue-iceberg-table\n\n[](https://github.com/ksco92/arceus/actions/workflows/ci.yml)\n[](https://codecov.io/gh/ksco92/arceus)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://pypi.org/project/cdk-glue-iceberg-table/)\n[](https://constructs.dev/packages/cdk-glue-iceberg-table)\n[](https://www.npmjs.com/package/cdk-glue-iceberg-table)\n[](https://github.com/ksco92/arceus/blob/main/LICENSE)\n\n**`cdk-glue-iceberg-table` is an AWS CDK L2 construct to create and\nevolve Apache Iceberg tables in the AWS Glue Data Catalog — schema and\npartition changes that survive CloudFormation updates, via cdk\ndeploy.** It supports Iceberg format version 3 (`variant`, nanosecond\ntimestamps, `geometry` / `geography`, `unknown`). It emits the exact\n`AWS::Glue::Table` / `OpenTableFormatInput` shape, so a single `cdk\ndeploy` creates, evolves, and destroys a table like any other CDK\nresource. No custom resource, no Lambda, no \"`cdk deploy` and then run\nthis SQL by hand\" two-step. Table changes (new columns, renames, drops,\nnew partition fields) land as a reviewed diff in a pull request and\napply through `cdk deploy`.\n\nStatus (October 2026): published on npm (TypeScript/JavaScript) and\nPyPI (Python), pre-1.0, with a public surface guarded on every PR by two\nconsumer jobs (`e2e-consumer` compiles and synthesizes an app against a\npinned published version, `packed-consumer` does the same against the PR's\nown `npm pack` tarball) plus real-AWS integration suites, including ones\nfor format version 3. The multi-language packages are generated\nfrom one TypeScript source with [jsii](https://aws.github.io/jsii/)\nand indexed on [Construct\nHub](https://constructs.dev/packages/cdk-glue-iceberg-table).\nDeveloped in the [`ksco92/arceus`](https://github.com/ksco92/arceus)\nmonorepo, which also holds a CDK demo app that dogfoods the construct\nagainst a real AWS account.\n\n### Why this exists\n\nAWS documents the CloudFormation shape for Iceberg tables ([AWS Big\nData blog, December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation)),\nbut raw `AWS::Glue::Table` is a minefield: the CREATE succeeds and the\n*first* `Update` silently strips `table_type=ICEBERG`, after which\nevery Athena query fails with `HIVE_UNSUPPORTED_FORMAT`. The\nmotivating CDK issue is [aws/aws-cdk#29660](https://github.com/aws/aws-cdk/issues/29660);\n[`manmartgarc`'s comment](https://github.com/aws/aws-cdk/issues/29660#issuecomment-4411359248)\ndocuments the only working shape and the silent-corruption traps you\nhit by getting it slightly wrong. This construct implements that\nshape, refuses to emit the unsafe alternatives, and pins Iceberg field\nIDs (declared [in creation order](#using-icebergtable) for a new\ntable) so schema evolution never orphans existing data. It was the\nbasis of an upstream proposal to land an `IcebergTable` in\n`@aws-cdk/aws-glue-alpha` ([aws/aws-cdk#37988](https://github.com/aws/aws-cdk/pull/37988)),\nclosed unmerged in September 2026.\n\n### How it compares\n\n| Approach | Declarative IaC (in the PR) | In-place schema + partition evolution | Prevents the silent-corruption footguns | Typed API + synth-time validation |\n| --- | --- | --- | --- | --- |\n| Raw `CfnTable` (L1) | ✅ | ⚠️ only if you hand-write the exact `OpenTableFormatInput` shape | ❌ you own both footguns | ❌ |\n| CDK custom resource (Lambda + Glue SDK) | ⚠️ via a custom resource you maintain | ⚠️ you write the diff logic | ⚠️ your code's responsibility | ❌ |\n| Spark SQL DDL, or Athena SQL DDL, at job runtime | ❌ imperative, outside the PR | ✅ but outside CloudFormation | n/a | ❌ |\n| **`cdk-glue-iceberg-table`** | ✅ | ✅ via `cdk deploy` | ✅ both prevented by construction | ✅ |\n\n## Install\n\nTypeScript / JavaScript (npm):\n\n```bash\nnpm install cdk-glue-iceberg-table\n```\n\nPython (PyPI):\n\n```bash\npip install cdk-glue-iceberg-table\n```\n\nPeer dependencies (your CDK app must already have these). This\nrelease requires `aws-cdk-lib` 2.273.0 or newer and `constructs`\n10.8.1 or newer, plus Node.js 22.14 or newer for both languages (the\nPython package runs under Node.js through jsii).\n\nFor TypeScript / JavaScript:\n\n```bash\nnpm install aws-cdk-lib@^2.273.0 constructs@^10.8.1\n```\n\nFor Python:\n\n```bash\npip install 'aws-cdk-lib>=2.273.0' 'constructs>=10.8.1'\n```\n\n## Use\n\n### TypeScript\n\n```ts\nimport { Bucket } from 'aws-cdk-lib/aws-s3';\nimport { Database } from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\nconst db = new Database(this, 'Db', { databaseName: 'analytics' });\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n columns: [\n { name: 'order_id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'customer_id', type: IcebergType.LONG, required: true, id: 2 },\n { name: 'placed_at', type: IcebergType.TIMESTAMPTZ, required: true, id: 3 },\n ],\n partitionSpec: [\n { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.DAY, fieldId: 1000 },\n { sourceColumn: 'customer_id', transform: IcebergPartitionTransform.bucket(16), fieldId: 1001 },\n ],\n identifierFieldNames: ['order_id'],\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\n### Python\n\nThe same table in Python — the API mirrors TypeScript with\n`snake_case` props and `PascalCase` types:\n\n```python\nfrom aws_cdk.aws_s3 import Bucket\nfrom aws_cdk.aws_glue import Database\nfrom cdk_glue_iceberg_table import (\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n IcebergPartitionTransform,\n)\n\nbucket = Bucket(self, \"Warehouse\")\ndb = Database(self, \"Db\", database_name=\"analytics\")\n\nIcebergTable(self, \"OrdersTable\",\n database=db,\n table_name=\"orders\",\n location=f\"s3://{bucket.bucket_name}/analytics/orders/\",\n columns=[\n {\"name\": \"order_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 1},\n {\"name\": \"customer_id\", \"type\": IcebergType.LONG, \"required\": True, \"id\": 2},\n {\"name\": \"placed_at\", \"type\": IcebergType.TIMESTAMPTZ, \"required\": True, \"id\": 3},\n ],\n partition_spec=[\n {\"source_column\": \"placed_at\", \"transform\": IcebergPartitionTransform.DAY, \"field_id\": 1000},\n {\"source_column\": \"customer_id\", \"transform\": IcebergPartitionTransform.bucket(16), \"field_id\": 1001},\n ],\n identifier_field_names=[\"order_id\"],\n format_version=IcebergFormatVersion.V2,\n)\n```\n\nConsumer-facing reference sections:\n\n- [How it compares](#how-it-compares) — `cdk-glue-iceberg-table` vs raw `CfnTable`, custom resources, and runtime SQL DDL.\n- [Using `IcebergTable`](#using-icebergtable) — full API reference with examples.\n- [Format version 3](#format-version-3) — the six v3 types, which engines read them, and the accepted limits.\n- [Two footguns the construct prevents](#two-footguns-the-construct-prevents) — the silent-corruption traps that motivated this construct.\n- [Known limitations](#known-limitations) — what the construct does and doesn't enforce.\n- [FAQ](#faq) — common \"how do I … in CDK / CloudFormation\" questions.\n\n## Exported surface\n\nThe package entry point re-exports the main types you import from\n`cdk-glue-iceberg-table`. In TypeScript / JavaScript the compiled\nsurface lives under `dist/lib/iceberg/` after `npm install`; in Python\nthe same surface is imported from the `cdk_glue_iceberg_table` module\nafter `pip install`:\n\n- **`IcebergTable`** — the L2 construct itself, with `grantRead` / `grantWrite` / `grantReadWrite`, the `fromIcebergTableAttributes(...)` import factory, and `creationOrderFieldIds(columns)` (Python: `creation_order_field_ids`), which returns a map from field path to the id Glue gives that field in a new table; a name containing `.` or a backquote appears in backquotes.\n- **`IcebergSchemaHistory`** — `validateSchemaEvolution(previous, next)` (Python: `validate_schema_evolution`), which returns the unsafe changes from a table's history record to a definition (`IcebergTableSchema`) as `IcebergSchemaViolation`s, each carrying an `IcebergSchemaViolationKind` that names the class of unsafe change; `append`, `toJson` and `fromJson`, which record a definition in, write and read the history file (`IcebergSchemaHistoryFile`, `IcebergTableHistory`, `IcebergFieldHistory`, `IcebergPartitionFieldHistory`); and `updateFromCloudAssembly(directory)`, which records every table of a synthesized app. See [Checking changes against a schema history](#checking-changes-against-a-schema-history).\n- **`IcebergType`** — primitive types, the v3 constants `UNKNOWN`, `TIMESTAMP_NS`, `TIMESTAMPTZ_NS`, and `VARIANT`, and the `list` / `map` / `struct` / `decimal` / `fixed` / `geometry(crs?)` / `geography(crs?, algorithm?)` factories. `list` takes an `IcebergListDefinition`, `map` an `IcebergMapDefinition`, and `struct` a list of `IcebergStructFieldDefinition`, each carrying the nested field ids. Renders to the string Glue's `IcebergStructField.type` expects.\n- **`IcebergPartitionTransform`** — identity / `bucket(N)` / `truncate(W)` / year / month / day / hour / void. Each transform validates against the source column type at synth time.\n- **`IcebergDataFormat`** (parquet/orc/avro, default parquet), **`IcebergFormatVersion`** (v1/v2/v3, required — set explicitly per table; Athena cannot query v3 tables), **`IcebergSortDirection`**, **`IcebergNullOrder`**, **`IcebergEdgeAlgorithm`** (`SPHERICAL`, `VINCENTY`, `THOMAS`, `ANDOYER`, `KARNEY`; the second argument of `IcebergType.geography`), plus a `tableProperties` validator that catches misconfigured properties before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).\n\n## Using `IcebergTable`\n\nA minimal table:\n\n```typescript\nimport {\n Bucket,\n} from 'aws-cdk-lib/aws-s3';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nconst bucket = new Bucket(this, 'Warehouse');\n\nconst db = new Database(this, 'Db', {\n databaseName: 'analytics',\n});\n\nnew IcebergTable(this, 'Users', {\n database: db,\n tableName: 'users',\n columns: [\n {\n name: 'user_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'email',\n type: IcebergType.STRING,\n required: true,\n id: 2,\n },\n {\n name: 'signed_up_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 3,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/users/`,\n formatVersion: IcebergFormatVersion.V2,\n});\n```\n\nEvery top-level column must declare an `id`: a stable, unique,\npositive integer you pick once and never reuse or reassign. Because\nIceberg data files reference fields by id rather than by name, a fixed\nid, declared in creation order when the table is created (below), is\nwhat lets you add, rename, and drop columns across deploys without\ncorrupting old data — see\n[How do I evolve an Iceberg table schema](#how-do-i-evolve-an-iceberg-table-schema-add-rename-or-drop-a-column-in-cloudformation).\nOnce a column is dropped, leave its id retired; never hand it to a new\ncolumn. Nested fields follow the same rule: every struct field takes\nan `id`, every list an `elementId`, and every map a `keyId` and a\n`valueId`. All ids in a schema, top-level and nested, must be distinct\nintegers from 1 to 2147483447.\n\nA new table must declare its ids in creation order. When Glue creates\na table it does not keep the declared ids: it numbers the fields from\n1 in creation order and points the partition spec, sort order and\nidentifier fields at the new ids. An update Glue accepts stores the\ndeclared ids as given, so the first such update of a table created\nwith other ids renumbers its fields; for the update Glue rejects, see\n[Known limitations](#known-limitations). To number a new table, number\nthe top-level columns first, in order, then the nested fields of each\ncolumn in turn (a list's element; a map's key, then its value, then\nthe key's nested fields, then the value's; a struct's fields, then\ntheir nested fields). `IcebergTable.creationOrderFieldIds` returns\nthese ids by field path and ignores the ids in its input:\n\n```typescript\nconst columns = [\n { name: 'id', type: IcebergType.LONG, required: true, id: 1 },\n { name: 'tags', type: IcebergType.list({ element: IcebergType.STRING, elementId: 4 }), id: 2 },\n {\n name: 'attrs',\n type: IcebergType.map({ key: IcebergType.STRING, keyId: 5, value: IcebergType.STRING, valueId: 6 }),\n id: 3,\n },\n];\nIcebergTable.creationOrderFieldIds(columns);\n// { id: 1, tags: 2, attrs: 3, 'tags.element': 4, 'attrs.key': 5, 'attrs.value': 6 }\n```\n\nIds out of creation order are for fields a later deploy adds; see\n[Evolving schema and partitions](#evolving-schema-and-partitions).\n\nA table that exercises most of the surface (partitions, sort order,\nnested types, identifier fields, table properties, removal policy):\n\n```typescript\nimport {\n RemovalPolicy,\n} from 'aws-cdk-lib';\nimport {\n Database,\n} from 'aws-cdk-lib/aws-glue';\nimport {\n IcebergDataFormat,\n IcebergFormatVersion,\n IcebergNullOrder,\n IcebergPartitionTransform,\n IcebergSortDirection,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'OrdersTable', {\n database: db,\n tableName: 'orders',\n comment: 'Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.',\n columns: [\n {\n name: 'order_id',\n type: IcebergType.LONG,\n required: true,\n id: 1,\n },\n {\n name: 'customer_id',\n type: IcebergType.LONG,\n required: true,\n id: 2,\n },\n {\n name: 'order_amount',\n type: IcebergType.decimal(12, 2),\n required: true,\n id: 3,\n },\n {\n name: 'currency',\n type: IcebergType.STRING,\n required: true,\n id: 4,\n },\n {\n name: 'placed_at',\n type: IcebergType.TIMESTAMPTZ,\n required: true,\n id: 5,\n },\n {\n name: 'tags',\n type: IcebergType.list({\n element: IcebergType.STRING,\n elementId: 9,\n }),\n id: 6,\n },\n {\n name: 'shipping_address',\n type: IcebergType.struct([\n {\n id: 10,\n name: 'line1',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 11,\n name: 'city',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 12,\n name: 'country',\n type: IcebergType.STRING,\n required: true,\n },\n {\n id: 13,\n name: 'postal_code',\n type: IcebergType.STRING,\n },\n ]),\n id: 7,\n },\n {\n name: 'metadata',\n type: IcebergType.map({\n key: IcebergType.STRING,\n keyId: 14,\n value: IcebergType.STRING,\n valueId: 15,\n }),\n id: 8,\n },\n ],\n location: `s3://${bucket.bucketName}/analytics/orders/`,\n partitionSpec: [\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1000,\n },\n {\n sourceColumn: 'customer_id',\n transform: IcebergPartitionTransform.bucket(16),\n fieldId: 1001,\n },\n ],\n sortOrder: [\n {\n sourceColumn: 'placed_at',\n direction: IcebergSortDirection.ASC,\n nullOrder: IcebergNullOrder.NULLS_LAST,\n },\n {\n sourceColumn: 'order_id',\n direction: IcebergSortDirection.ASC,\n },\n ],\n identifierFieldNames: [\n 'order_id',\n ],\n dataFormat: IcebergDataFormat.PARQUET,\n formatVersion: IcebergFormatVersion.V2,\n tableProperties: {\n 'write.parquet.compression-codec': 'zstd',\n 'write.delete.mode': 'merge-on-read',\n 'write.update.mode': 'merge-on-read',\n 'write.merge.mode': 'merge-on-read',\n 'write.target-file-size-bytes': '134217728',\n 'history.expire.min-snapshots-to-keep': '5',\n 'gc.enabled': 'true',\n },\n removalPolicy: RemovalPolicy.DESTROY,\n});\n```\n\nEach partition field also pins a `fieldId`: an integer of 1000 or\nabove (the Iceberg spec reserves lower ids for schema columns), unique\nacross the table's entire partition-spec history. The discipline is\nthe inverse of the column-id rule. A column keeps its id forever; a\npartition field keeps its id only while it stays untouched. Change\nanything about a field — swap `month` for `day`, point it at a\ndifferent source column — and it becomes a *new* field that needs a\nfresh, never-before-used id, while the old id retires with the old\nspec. Reusing a retired id (or reassigning an id to a different field)\nproduces a table that Athena still reads but Spark and Trino reject\nwith `ValidationException: Conflicting partition fields`.\n\n`identifierFieldNames` lists the columns that together identify a row\nand renders as the schema's `identifier-field-ids`. Each one must be a\n`required: true` column (Glue rejects an optional one), and `cdk synth`\nrejects an identifier column of type float, double, list, map, struct,\nunknown, variant, geometry or geography. The construct prefixes each\nidentifier column's `doc` with `(PK) `, so a `doc` you write must not\nstart with `(PK) `; see [Evolving schema and\npartitions](#evolving-schema-and-partitions).\n\nEvery string you pass is a literal string, not a token, and an\nenum-typed one must be a member of its enum; the exceptions are\n`location`, `comment`, table property values, and the `tableName` of\nan imported table.\n\nThe resulting Iceberg `metadata.json` for this table contains every\nfeature you set:\n\n```json\n{\n \"format-version\": 2,\n \"table-uuid\": \"39a948f9-...\",\n \"current-schema-id\": 0,\n \"schemas\": [\n {\n \"schema-id\": 0,\n \"identifier-field-ids\": [1],\n \"fields\": [\n { \"id\": 1, \"name\": \"order_id\", \"required\": true, \"type\": \"long\", \"doc\": \"(PK) \" },\n { \"id\": 2, \"name\": \"customer_id\", \"required\": true, \"type\": \"long\" },\n { \"id\": 3, \"name\": \"order_amount\", \"required\": true, \"type\": \"decimal(12, 2)\" },\n { \"id\": 4, \"name\": \"currency\", \"required\": true, \"type\": \"string\" },\n { \"id\": 5, \"name\": \"placed_at\", \"required\": true, \"type\": \"timestamptz\" },\n { \"id\": 6, \"name\": \"tags\", \"required\": false,\n \"type\": { \"type\": \"list\", \"element-id\": 9, \"element\": \"string\", \"element-required\": false } },\n { \"id\": 7, \"name\": \"shipping_address\", \"required\": false,\n \"type\": { \"type\": \"struct\", \"fields\": [\n { \"id\": 10, \"name\": \"line1\", \"required\": true, \"type\": \"string\" },\n { \"id\": 11, \"name\": \"city\", \"required\": true, \"type\": \"string\" },\n { \"id\": 12, \"name\": \"country\", \"required\": true, \"type\": \"string\" },\n { \"id\": 13, \"name\": \"postal_code\", \"required\": false, \"type\": \"string\" }\n ] } },\n { \"id\": 8, \"name\": \"metadata\", \"required\": false,\n \"type\": { \"type\": \"map\", \"key-id\": 14, \"key\": \"string\", \"value-id\": 15,\n \"value-required\": false, \"value\": \"string\" } }\n ]\n }\n ],\n \"partition-specs\": [\n { \"spec-id\": 0, \"fields\": [\n { \"name\": \"placed_at_day\", \"transform\": \"day\", \"source-id\": 5, \"field-id\": 1000 },\n { \"name\": \"customer_id_bucket\", \"transform\": \"bucket[16]\", \"source-id\": 2, \"field-id\": 1001 }\n ]}\n ],\n \"sort-orders\": [\n { \"order-id\": 1, \"fields\": [\n { \"transform\": \"identity\", \"source-id\": 5, \"direction\": \"asc\", \"null-order\": \"nulls-last\" },\n { \"transform\": \"identity\", \"source-id\": 1, \"direction\": \"asc\", \"null-order\": \"nulls-last\" }\n ]}\n ],\n \"properties\": {\n \"format-version\": \"2\",\n \"write.format.default\": \"parquet\",\n \"write.parquet.compression-codec\": \"zstd\",\n \"write.merge.mode\": \"merge-on-read\",\n \"write.update.mode\": \"merge-on-read\",\n \"write.delete.mode\": \"merge-on-read\",\n \"write.target-file-size-bytes\": \"134217728\",\n \"history.expire.min-snapshots-to-keep\": \"5\",\n \"gc.enabled\": \"true\",\n \"comment\": \"Demo Iceberg orders table — exercises partitions, sort order, and merge-on-read.\"\n }\n}\n```\n\n### Format version 3\n\nSet `formatVersion: IcebergFormatVersion.V3` to create a v3 table. The\nsix v3 types (below) are only accepted on a V3 table, at any nesting\ndepth: inside a struct, as a list element, or as a map key or value.\nSynth fails on a V1 or V2 table that uses one.\n\n```typescript\nimport {\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n} from 'cdk-glue-iceberg-table';\n\nnew IcebergTable(this, 'Events', {\n database: db,\n tableName: 'events',\n location: `s3://${bucket.bucketName}/analytics/events/`,\n columns: [\n { name: 'occurred_at', type: IcebergType.TIMESTAMP_NS, id: 1 },\n { name: 'payload', type: IcebergType.VARIANT, id: 2 },\n {\n name: 'origin',\n type: IcebergType.geography('OGC:CRS84', IcebergEdgeAlgorithm.SPHERICAL),\n id: 3,\n },\n ],\n formatVersion: IcebergFormatVersion.V3,\n});\n```\n\nThe same table in Python:\n\n```python\nfrom cdk_glue_iceberg_table import (\n IcebergEdgeAlgorithm,\n IcebergFormatVersion,\n IcebergTable,\n IcebergType,\n)\n\nIcebergTable(self, \"Events\",\n database=db,\n table_name=\"events\",\n location=f\"s3://{bucket.bucket_name}/analytics/events/\",\n columns=[\n {\"name\": \"occurred_at\", \"type\": IcebergType.TIMESTAMP_NS, \"id\": 1},\n {\"name\": \"payload\", \"type\": IcebergType.VARIANT, \"id\": 2},\n {\n \"name\": \"origin\",\n \"type\": IcebergType.geography(\"OGC:CRS84\", IcebergEdgeAlgorithm.SPHERICAL),\n \"id\": 3,\n },\n ],\n format_version=IcebergFormatVersion.V3,\n)\n```\n\nThis table has a `timestamp_ns` column; see [Known limitations](#known-limitations)\nbefore reading it from upstream Spark.\n\nThe v3 types, with the rules `cdk synth` enforces:\n\n| Type | Factory | Rendered string | Partition and identifier rules |\n| --- | --- | --- | --- |\n| unknown | `IcebergType.UNKNOWN` | `unknown` | Must be optional, as a column, list element, or map value; never a map key. Partition source only with `void`. Not an identifier field. |\n| timestamp with nanoseconds | `IcebergType.TIMESTAMP_NS` | `timestamp_ns` | `identity`, `year`, `month`, `day`, `hour`, `bucket(N)`, `void`; not `truncate(W)`. Allowed as an identifier field. |\n| timestamptz with nanoseconds | `IcebergType.TIMESTAMPTZ_NS` | `timestamptz_ns` | Same as `timestamp_ns`. |\n| variant | `IcebergType.VARIANT` | `variant` | Never a partition or sort source, `void` included. Not an identifier field. |\n| geometry | `IcebergType.geometry(crs?)` | `geometry`, or `geometry(<crs>)` when a CRS is given | Partition source only with `void`. Not an identifier field. |\n| geography | `IcebergType.geography(crs?, algorithm?)` | `geography`, `geography(<crs>)`, or `geography(<crs>, <algorithm>)` when an algorithm is given, such as `geography(OGC:CRS84, spherical)` | Same as `geometry`. |\n\nA geometry or geography `crs` is an identifier such as `OGC:CRS84` or\n`EPSG:3857`; whitespace, parentheses, braces, and commas are rejected.\nWhen `algorithm` is set without a `crs`, the rendered string spells out\n`OGC:CRS84`.\n\nWhich engines read a v3 table created by the construct:\n\n| Engine | v3 tables | Notes |\n| --- | --- | --- |\n| Spark 4.1.1 + Iceberg 1.11.0 (the integration-test runner) | Yes | Reads and writes tables the construct creates, including deletion vectors, row lineage, `variant` and `unknown`, except tables with a `geometry`, `geography` or nanosecond-timestamp column ([Known limitations](#known-limitations)). |\n| AWS Glue 6.0 | Documented | The [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) page lists \"Iceberg format version 3 with VARIANT data type, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types\". The integration suites run on Glue 6.0's Spark 4.1.1 and Iceberg 1.11.0 (`scripts/spark/spark-env.sh`); the Glue job runtime itself is not exercised. |\n| Amazon EMR Spark 8.1.0 (Spark 4.1.1, Iceberg 1.11.0-amzn-0) | Documented | The [emr-spark-8.1.0 release notes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html) state \"Amazon EMR Spark 8.1.0 extends Iceberg v3 spec support for Spark workloads\". The integration suites do not run on EMR 8.1. |\n| Athena SQL | No | The [Athena Iceberg page](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) states \"Athena only creates and operates on Iceberg v2 tables.\" The v2 integration suites also run through Athena (see [Does it work with Athena and Lake Formation?](#does-it-work-with-athena-and-lake-formation)); no v3 table is queried through Athena. |\n\n### Granting access\n\n```typescript\ntable.grantRead(role); // Glue read + S3 read on the table's prefix\ntable.grantWrite(role); // Glue write + S3 write\ntable.grantReadWrite(role);\n```\n\nThe `grant*` helpers issue **IAM grants only**. Under Lake Formation\nyou still add the matching `SELECT` / `INSERT` / `DELETE` LF grants on\ntop of the construct's IAM grants for Athena queries to succeed.\n\n### Importing an existing table\n\n```typescript\nconst existing = IcebergTable.fromIcebergTableAttributes(this, 'Orders', {\n database: db,\n tableName: 'orders',\n location: 's3://my-bucket/analytics/orders/',\n});\nexisting.grantRead(role);\n```\n\n### Evolving schema and partitions\n\nChange the `columns` array (or `partitionSpec`) and run `cdk deploy`\nagain. The construct passes the new schema to Glue's `UpdateTable`,\nwhich writes a new `metadata.json` with a new `schema-id`; existing\ndata files stay readable because each column's `id` is pinned to the\nid the table holds and never reused. Adds, renames (same `id`, new\n`name`), and drops all flow through `cdk deploy` alone — no\nout-of-band SQL DDL.\n\nNested fields evolve the same way. Each struct field, list element,\nmap key, and map value keeps the id you gave it (the rule is under\n[Using `IcebergTable`](#using-icebergtable)), so adding a column or a\nstruct field anywhere in the schema leaves every other id unchanged. A\nnew nested field takes an id no field of the table has ever used:\n\n```typescript\ntype: IcebergType.struct([\n { id: 10, name: 'line1', type: IcebergType.STRING },\n { id: 16, name: 'line2', type: IcebergType.STRING }, // new field, never-used id\n { id: 11, name: 'city', type: IcebergType.STRING },\n]),\n```\n\nSuch an id is out of creation order, which is expected once a table\nhas evolved: creation order applies only to the ids a table is created\nwith (see [Using `IcebergTable`](#using-icebergtable)).\n\nUpgrading from a version before 0.12.0, which numbered nested fields\nitself: declare the ids the table's current `metadata.json` holds,\ntop-level and nested (each field `id`, `element-id`, `key-id` and\n`value-id`), or the next deploy renumbers the fields that differ and\nexisting data files read under the wrong field. A nested type object\nshared by two columns now fails synth with a duplicate id; give each\ncolumn its own nested type with its own ids. The list element and map\nvalue nullability default also changed: `elementRequired` and\n`valueRequired` now default to `false`. Earlier versions made list\nelements required by default and map values required unless the third\nargument of `map` was `false`; set `elementRequired: true` or\n`valueRequired: true` wherever the deployed schema has\n`\"element-required\": true` or `\"value-required\": true`.\n\nGlue's `UpdateTable` reuses an earlier schema whose fields (name,\ntype, `required`, `doc`) match the request, together with that\nschema's `identifier-field-ids`. The construct therefore prefixes the\n`doc` of every identifier column with `(PK) ` (just `(PK) ` when the\ncolumn has no `doc`), shown as the column comment, so every\nidentifier change is a field change. On a table whose schema history\nwas written with the marker, identifier changes flow through `cdk\ndeploy` like column changes.\n\nPartition evolution flows the same way, with one extra rule on\n`fieldId`: a changed partition field is a new partition field. To move\na table from `month(placed_at)` to `day(placed_at)`, drop the month\nentry and add a day entry under a fresh `fieldId` — never recycle the\nmonth field's id:\n\n```typescript\npartitionSpec: [\n // was: { sourceColumn: 'placed_at', transform: IcebergPartitionTransform.MONTH, fieldId: 1000 },\n {\n sourceColumn: 'placed_at',\n transform: IcebergPartitionTransform.DAY,\n fieldId: 1001,\n },\n],\n```\n\nOld data files written under the month spec stay readable; new writes\npartition by day. Reusing `1000` for the day field instead corrupts\nthe spec history for strict engines (Spark, Trino) while Athena keeps\nworking — the failure only surfaces at Spark runtime.\n\nA changed `sortOrder` also flows through `cdk deploy`. Glue adds it to\n`sort-orders` under the next `order-id` and points\n`default-sort-order-id` at it; the earlier orders stay in the list.\nA change to `tableProperties` alone writes new metadata that keeps the\nsame `current-schema-id`, partition spec and sort order.\n\n### Checking changes against a schema history\n\nThe construct never reads the deployed table, so on its own it cannot\ntell that an id belonged to a field dropped two deploys ago. A schema\nhistory file committed beside the app gives synth that record without\nan AWS call. Point `schemaHistoryFile` at the history file of the\ntable's database:\n\n```typescript\nnew IcebergTable(this, 'Orders', {\n // ...\n schemaHistoryFile: path.join(__dirname, 'analytics.schema-history.json'),\n});\n```\n\nA relative path resolves against the working directory, which is the\ndirectory holding `cdk.json` when the `cdk` CLI runs the app. Create\nthe file once with the content `{\"tables\": []}`. Records are keyed by\ntable name, so give each Glue database its own file.\n\nEvery synth (`cdk synth`, `cdk diff`, `cdk deploy`, and the CI job\nthat runs one) then compares each table with its record and fails,\nlisting every problem, when the definition:\n\n- gives a column or nested field id in the record to a different\n field: a dropped field's id, or the id of a field whose\n path is still in the definition under another id;\n- keeps a field's path but changes its id, as in the nested ids\n `4, 5, 6` becoming `5, 6, 7` after a column is added;\n- changes a field's type, except `int` to `long`, `float` to `double`,\n and a wider decimal precision at the same scale;\n- makes an optional field required;\n- drops a column that any recorded partition spec was sourced from;\n- keeps a partition `fieldId` with a different source column or\n transform.\n\nFields are identified by id, so a rename passes. A table with no\nrecord passes. Synth never writes the file; record a change once it\npasses synth, and commit the file with the change:\n\n```bash\ncdk synth --quiet\nnode -e \"console.log(require('cdk-glue-iceberg-table').IcebergSchemaHistory.updateFromCloudAssembly('cdk.out'))\"\n```\n\nFrom a Python app, the second command is:\n\n```bash\npython -c \"from cdk_glue_iceberg_table import IcebergSchemaHistory; print(IcebergSchemaHistory.update_from_cloud_assembly('cdk.out'))\"\n```\n\n`updateFromCloudAssembly` records every table of the assembly that\nsets `schemaHistoryFile` in that file and returns the files it wrote.\nThe record keeps every id the table has used: a dropped field stays\nwith `\"dropped\": true`, and a partition field stays after it leaves\nthe spec. The file must already exist, and nothing is written when any\ntable is an unsafe change. A column replaced under the same name needs\ntwo recorded changes: drop it, then add it with a new id.\n\n`IcebergSchemaHistory.validateSchemaEvolution(previous, next)` runs\nthe same checks on one table record without the prop, and\n`IcebergSchemaHistory.append`, `toJson` and `fromJson` record a\ndefinition in, write and read the file format.\n\n### Inserting and querying\n\n```sql\n-- INSERT into the orders table\nINSERT INTO sample_database.orders VALUES\n (1001, 5001, DECIMAL '149.99', 'USD',\n TIMESTAMP '2026-05-20 09:15:00 UTC',\n ARRAY['holiday-promo','first-order'],\n CAST(ROW('1 Infinite Loop','Cupertino','US','95014')\n AS ROW(line1 VARCHAR,city VARCHAR,country VARCHAR,postal_code VARCHAR)),\n MAP(ARRAY['channel','utm'], ARRAY['web','google'])),\n -- ... more rows\n;\n\n-- merge-on-read DELETE (only legal because we chose v2 + merge-on-read mode)\nDELETE FROM sample_database.orders WHERE order_id = 1003;\n\n-- merge-on-read UPDATE\nUPDATE sample_database.orders SET currency = 'GBP' WHERE customer_id = 5002;\n\n-- SELECT\nSELECT customer_id, SUM(order_amount) AS total\n FROM sample_database.orders\n GROUP BY 1\n ORDER BY 2 DESC;\n```\n\n## Two footguns the construct prevents\n\n### Footgun #1 — schema under `storageDescriptor.columns`\n\nThe CREATE succeeds but the first UPDATE silently strips\n`table_type=ICEBERG` from the table's Glue parameters, and Athena\nqueries after that fail with `HIVE_UNSUPPORTED_FORMAT`.\n\n```typescript\n// DON'T DO THIS — what most StackOverflow / re:Post examples show\nnew CfnTable(this, 'OrdersBad', {\n catalogId: this.account,\n databaseName: 'analytics',\n tableInput: {\n name: 'orders',\n tableType: 'EXTERNAL_TABLE',\n parameters: {\n table_type: 'ICEBERG',\n },\n storageDescriptor: {\n location: 's3://.../orders/',\n columns: [\n /* ... */\n ],\n },\n },\n openTableFormatInput: {\n icebergInput: {\n metadataOperation: 'CREATE',\n version: '2',\n },\n },\n});\n```\n\n`IcebergTable` instead always emits schema/partitions/sort/properties\nunder `openTableFormatInput.icebergInput.icebergTableInput`, never\nunder `storageDescriptor`.\n\n### Footgun #2 — `tableInput` co-present with `openTableFormatInput`\n\nEven setting just `tableInput: { name: 'foo' }` next to\n`openTableFormatInput` returns\n`\"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput\"`.\nThe construct never emits `tableInput`; the table-level comment goes\ninto `tableProperties['comment']`, which lives inside\n`icebergTableInput.properties`.\n\n(There is a third footgun, field-id reuse after a column drop, that\nthe construct prevents only with a [schema history\nfile](#checking-changes-against-a-schema-history). See the next\nsection.)\n\n## Known limitations\n\n- **Creation order is not checked.** The construct cannot tell at synth whether a table is new or already deployed, so it accepts a new table whose ids are not in creation order (the rule is under [Using `IcebergTable`](#using-icebergtable)). When the first update of such a table changes the id of a column an identity partition is sourced from, Glue rejects the update with `Cannot create identity partition sourced from different field in schema` and the CloudFormation rollback fails the same way, leaving the stack in `UPDATE_ROLLBACK_FAILED`.\n- **Field-id reuse across deploys is detected only with a schema history file.** This applies to nested ids (struct field, list element, map key and value) as well as column ids. If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the \"never reuse a retired id\" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate field id N` validator) and never reads the live table state; across deploys, only a [schema history file](#checking-changes-against-a-schema-history) catches reuse. Without it, treat dropped ids as retired forever.\n- **Partition field-id reuse across deploys is detected only with a schema history file.** The same live-state blindness applies to partition `fieldId`s: the construct validates them within one deploy (integer >= 1000, unique in the spec), and only a [schema history file](#checking-changes-against-a-schema-history) tells it that the id you pinned already appeared in an earlier spec of the deployed table. The invariant partition ids must uphold is stricter than the column one — unique across the table's **entire spec history**, so a changed field (different transform or source column) must take a fresh id rather than keep its old one. Violate it and strict engines (Spark, Trino) reject the table with `Conflicting partition fields` while Athena keeps answering, masking the break until Spark runtime. Without the history file, treat every id that has ever appeared in any spec as retired, and allocate new fields the next never-used id. A partition change needs a new source column or transform; Glue ignores a `fieldId`-only change.\n- **`last-partition-id` is written by Glue, not by the construct.** The construct supplies the spec's field ids; Glue's `UpdateTable` computes the table's `last-partition-id` counter when it writes the new metadata. The repo's integration test asserts the counter keeps up with the highest supplied field id after a partition evolution, but engines that later evolve the spec themselves (e.g. Spark `ALTER TABLE … ADD PARTITION FIELD`) allocate from that counter — if you mix CDK-managed and engine-managed partition evolution, verify the counter in `metadata.json` before trusting engine-side changes.\n- **A rolled-back deploy leaves its schema in the table's history.** When a deploy that adds a column updates the table and then fails on another resource, the stack ends in `UPDATE_ROLLBACK_COMPLETE` and `current-schema-id` is back to its earlier value, so the column is gone and the rows read as before. The schema that held the column stays in `schemas`, and `last-column-id` stays at the column's id. A clean redeploy of the same change makes that schema current again without adding a schema. Treat an id from a rolled-back deploy as used: give it only to the column that deploy added.\n- **Removing the last identifier from a table created with identifiers before 0.11.0 can bring the old ids back.** That table's history holds an unmarked schema with ids, and Glue reuses it when the rendered fields match it again. Change a column `doc` in the same deploy that removes the last identifier.\n- **The schema history check covers only the changes it lists.** It does not see a change an engine makes with `ALTER TABLE` (Spark, Athena), whose ids the file never records, or a deploy whose definition was not recorded in the file. It does not check sort order or identifier field changes or report an added required field, top level or nested, and it treats a field moved into another struct under its id as a rename. It reports the format v3 promotions `unknown` to any type and `date` to `timestamp` or `timestamp_ns` as type changes.\n- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.\n- **Format version is immutable after CREATE.** Glue ignores a V2 to V3 `formatVersion` change, though `properties.format-version` then reads `3`. Keep `formatVersion` at the table's real version, or destroy + recreate the table to change it. An engine can upgrade a v2 table to v3 in place with `ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3')`; after that, set `formatVersion` to `V3` so the next deploy declares the version the table already has.\n- **`merge-on-read` requires v2 or later.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.\n- **Column defaults are not available.** Iceberg v3 `initial-default` and `write-default` cannot be declared through CloudFormation: the `AWS::Glue::Table` resource schema has no `InitialDefault` or `WriteDefault` property. Revisit when that schema gains the two properties.\n- **Nanosecond timestamps are catalog-only for Spark 4.1.1 with Iceberg 1.11.0.** In [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag, `primitive()` has no case for the `TIMESTAMP_NANO` type id, so a table with a `timestamp_ns` or `timestamptz_ns` column reaches the `Cannot convert unsupported type to Spark` error. The integration suite asserts these columns from `metadata.json` only.\n- **Geometry and geography columns are catalog-only for Spark 4.1.1 with Iceberg 1.11.0, the versions AWS Glue 6.0 ships.** The first Spark read of a table holding a `geography` and a `geometry` column failed with `UnsupportedOperationException: Cannot convert unsupported type to Spark: geography(OGC:CRS84, spherical)` on those versions. [`TypeToSparkType`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java) at the `apache-iceberg-1.11.0` tag has no conversion for `geometry` either. The integration suite deploys these columns on a table no Spark job opens and asserts them from `metadata.json` only.\n- **Athena DDL features that don't surface through CFN** (e.g. `ALTER TABLE … SET LOCATION`, `bucketed_by` / `bucket_count` Hive clauses) are not exposed. Use `IcebergPartitionTransform.bucket(N)` instead of Hive bucketing.\n- **Dropping a partition column requires a `void` intermediate per the Iceberg spec**, and the CFN `OpenTableFormatInput` cannot express that. The construct accepts the change unless a [schema history file](#checking-changes-against-a-schema-history) records the column as a partition source, and Athena queries against the result will fail with `Type cannot be null`. The safe pattern is to drop partitions that source from a column while keeping that column in the schema, and only drop a column once it is no longer partitioning anything.\n\n## FAQ\n\n### How do I create an Apache Iceberg table in AWS CDK?\n\nInstall `cdk-glue-iceberg-table`, then declare an `IcebergTable` with a\ndatabase, a column list, and an S3 `location` — see [Use](#use).\n`cdk deploy` creates the Glue Data Catalog table and writes the\nIceberg `metadata.json` under your S3 prefix. No custom resource or\nLambda is involved; it is a plain `AWS::Glue::Table` in your\nCloudFormation template.\n\n### How do I evolve an Iceberg table schema (add, rename, or drop a column) in CloudFormation?\n\nChange the `columns` array and run `cdk deploy` again. The construct\npasses the new schema to Glue's `UpdateTable`, which writes a new\n`metadata.json` with a new `schema-id`; existing data files stay\nreadable because each column's `id` is pinned to the id the table\nholds and never reused.\nPartition changes work the same way, except a changed partition field\ntakes a fresh `fieldId` instead of keeping its old one. See [Evolving\nschema and partitions](#evolving-schema-and-partitions).\n\n### Does CloudFormation support Iceberg tables natively?\n\nYes, through `AWS::Glue::Table` with `OpenTableFormatInput.IcebergInput`,\ndocumented by AWS in [December 2025](https://aws.amazon.com/blogs/big-data/create-and-update-apache-iceberg-tables-with-partitions-in-the-aws-glue-data-catalog-using-the-aws-sdk-and-aws-cloudformation).\nThe catch is that the raw shape corrupts the table on the first\n`Update` if you place schema under `storageDescriptor.columns` or set\n`tableInput` alongside `openTableFormatInput`. This construct only ever\nemits the safe shape and gives you no way to express the unsafe ones.\nSee [Two footguns the construct prevents](#two-footguns-the-construct-prevents).\n\n### What is the difference between cdk-glue-iceberg-table and a raw CfnTable?\n\nA raw `CfnTable` makes you hand-write the `OpenTableFormatInput` JSON\nand own both silent-corruption footguns. `cdk-glue-iceberg-table` gives\nyou a typed `IcebergType` / `IcebergPartitionTransform` API,\nsynth-time validation (partition transforms checked against column\ntypes, format/version mismatches caught before deploy), pinned field\nIDs for safe evolution (declared in creation order for a new table),\nand `grantRead` / `grantWrite` helpers. See\n[How it compares](#how-it-compares).\n\n### How do I create a partitioned Iceberg table in CDK?\n\nPass a `partitionSpec` of `IcebergPartitionTransform` entries —\n`identity`, `bucket(N)`, `truncate(W)`, `year`, `month`, `day`, `hour`,\nor `void` — each pinned to a `fieldId` of 1000 or above. Transforms are\nvalidated against their source column's type at synth time. See\n[Using `IcebergTable`](#using-icebergtable).\n\n### Does it work with Athena and Lake Formation?\n\nYes for v2 tables, no for v3 (see the engine matrix in [Format\nversion 3](#format-version-3)). The demo app in the\n[`arceus`](https://github.com/ksco92/arceus) repo registers its tables\nwith Lake Formation, and its v2 integration suites query them through\nAthena as well as Spark, on the same tables:\n\n- merge-on-read `INSERT`, `UPDATE`, `DELETE` and `MERGE INTO`, time\n travel (`FOR VERSION AS OF` / `FOR TIMESTAMP AS OF`), `OPTIMIZE ...\n REWRITE DATA USING BIN_PACK`, and `VACUUM`;\n- a read after every schema, partition, identifier and nested-field\n evolution step;\n- the `(PK) ` comment on identifier columns, read from\n `information_schema.columns`;\n- rows written by Spark read through Athena, and rows written by Athena\n read through Spark.\n\nThe suites grant the querying principal `SELECT`, `INSERT`, `DELETE`,\n`ALTER` and `DESCRIBE` through Lake Formation. The construct's `grant*`\nhelpers issue IAM grants only; under Lake Formation you add the\nmatching LF grants. See [Granting access](#granting-access).\n\n### Can I use Iceberg v2 merge-on-read (row-level UPDATE and DELETE)?\n\nYes — set `formatVersion: IcebergFormatVersion.V2` and the\n`write.{delete,update,merge}.mode = merge-on-read` table properties.\nThe construct rejects merge-on-read on a v1 table at synth time.\n\n### Can I create an Iceberg v3 table?\n\nYes. Set `formatVersion: IcebergFormatVersion.V3`; see [Format\nversion 3](#format-version-3) for the types, the engine matrix and the\nlimits.\n\n## Contributing\n\nDevelopment, the monorepo layout, and the demo app live in [CONTRIBUTING.md](https://github.com/ksco92/arceus/blob/main/CONTRIBUTING.md).\n\n## License\n\n[MIT](https://github.com/ksco92/arceus/blob/main/LICENSE).\n"
|
|
9755
9755
|
},
|
|
9756
9756
|
"repository": {
|
|
9757
9757
|
"directory": "packages/cdk-glue-iceberg-table",
|
|
@@ -9779,7 +9779,7 @@
|
|
|
9779
9779
|
"kind": "interface",
|
|
9780
9780
|
"locationInModule": {
|
|
9781
9781
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9782
|
-
"line":
|
|
9782
|
+
"line": 236
|
|
9783
9783
|
},
|
|
9784
9784
|
"methods": [
|
|
9785
9785
|
{
|
|
@@ -9790,7 +9790,7 @@
|
|
|
9790
9790
|
},
|
|
9791
9791
|
"locationInModule": {
|
|
9792
9792
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9793
|
-
"line":
|
|
9793
|
+
"line": 250
|
|
9794
9794
|
},
|
|
9795
9795
|
"name": "grantRead",
|
|
9796
9796
|
"parameters": [
|
|
@@ -9815,7 +9815,7 @@
|
|
|
9815
9815
|
},
|
|
9816
9816
|
"locationInModule": {
|
|
9817
9817
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9818
|
-
"line":
|
|
9818
|
+
"line": 256
|
|
9819
9819
|
},
|
|
9820
9820
|
"name": "grantReadWrite",
|
|
9821
9821
|
"parameters": [
|
|
@@ -9840,7 +9840,7 @@
|
|
|
9840
9840
|
},
|
|
9841
9841
|
"locationInModule": {
|
|
9842
9842
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9843
|
-
"line":
|
|
9843
|
+
"line": 253
|
|
9844
9844
|
},
|
|
9845
9845
|
"name": "grantWrite",
|
|
9846
9846
|
"parameters": [
|
|
@@ -9869,7 +9869,7 @@
|
|
|
9869
9869
|
"immutable": true,
|
|
9870
9870
|
"locationInModule": {
|
|
9871
9871
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9872
|
-
"line":
|
|
9872
|
+
"line": 244
|
|
9873
9873
|
},
|
|
9874
9874
|
"name": "database",
|
|
9875
9875
|
"type": {
|
|
@@ -9885,7 +9885,7 @@
|
|
|
9885
9885
|
"immutable": true,
|
|
9886
9886
|
"locationInModule": {
|
|
9887
9887
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9888
|
-
"line":
|
|
9888
|
+
"line": 247
|
|
9889
9889
|
},
|
|
9890
9890
|
"name": "location",
|
|
9891
9891
|
"type": {
|
|
@@ -9901,7 +9901,7 @@
|
|
|
9901
9901
|
"immutable": true,
|
|
9902
9902
|
"locationInModule": {
|
|
9903
9903
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9904
|
-
"line":
|
|
9904
|
+
"line": 238
|
|
9905
9905
|
},
|
|
9906
9906
|
"name": "tableArn",
|
|
9907
9907
|
"type": {
|
|
@@ -9917,7 +9917,7 @@
|
|
|
9917
9917
|
"immutable": true,
|
|
9918
9918
|
"locationInModule": {
|
|
9919
9919
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9920
|
-
"line":
|
|
9920
|
+
"line": 241
|
|
9921
9921
|
},
|
|
9922
9922
|
"name": "tableName",
|
|
9923
9923
|
"type": {
|
|
@@ -9938,7 +9938,7 @@
|
|
|
9938
9938
|
"kind": "interface",
|
|
9939
9939
|
"locationInModule": {
|
|
9940
9940
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9941
|
-
"line":
|
|
9941
|
+
"line": 87
|
|
9942
9942
|
},
|
|
9943
9943
|
"name": "IcebergColumn",
|
|
9944
9944
|
"properties": [
|
|
@@ -9952,7 +9952,7 @@
|
|
|
9952
9952
|
"immutable": true,
|
|
9953
9953
|
"locationInModule": {
|
|
9954
9954
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9955
|
-
"line":
|
|
9955
|
+
"line": 108
|
|
9956
9956
|
},
|
|
9957
9957
|
"name": "id",
|
|
9958
9958
|
"type": {
|
|
@@ -9968,7 +9968,7 @@
|
|
|
9968
9968
|
"immutable": true,
|
|
9969
9969
|
"locationInModule": {
|
|
9970
9970
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9971
|
-
"line":
|
|
9971
|
+
"line": 89
|
|
9972
9972
|
},
|
|
9973
9973
|
"name": "name",
|
|
9974
9974
|
"type": {
|
|
@@ -9984,7 +9984,7 @@
|
|
|
9984
9984
|
"immutable": true,
|
|
9985
9985
|
"locationInModule": {
|
|
9986
9986
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
9987
|
-
"line":
|
|
9987
|
+
"line": 92
|
|
9988
9988
|
},
|
|
9989
9989
|
"name": "type",
|
|
9990
9990
|
"type": {
|
|
@@ -10001,7 +10001,7 @@
|
|
|
10001
10001
|
"immutable": true,
|
|
10002
10002
|
"locationInModule": {
|
|
10003
10003
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10004
|
-
"line":
|
|
10004
|
+
"line": 101
|
|
10005
10005
|
},
|
|
10006
10006
|
"name": "doc",
|
|
10007
10007
|
"optional": true,
|
|
@@ -10019,7 +10019,7 @@
|
|
|
10019
10019
|
"immutable": true,
|
|
10020
10020
|
"locationInModule": {
|
|
10021
10021
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10022
|
-
"line":
|
|
10022
|
+
"line": 98
|
|
10023
10023
|
},
|
|
10024
10024
|
"name": "required",
|
|
10025
10025
|
"optional": true,
|
|
@@ -10115,6 +10115,105 @@
|
|
|
10115
10115
|
"name": "IcebergEdgeAlgorithm",
|
|
10116
10116
|
"symbolId": "lib/iceberg/iceberg-type:IcebergEdgeAlgorithm"
|
|
10117
10117
|
},
|
|
10118
|
+
"cdk-glue-iceberg-table.IcebergFieldHistory": {
|
|
10119
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
10120
|
+
"datatype": true,
|
|
10121
|
+
"docs": {
|
|
10122
|
+
"stability": "experimental",
|
|
10123
|
+
"summary": "One field id a table has used: a top-level column or a nested field."
|
|
10124
|
+
},
|
|
10125
|
+
"fqn": "cdk-glue-iceberg-table.IcebergFieldHistory",
|
|
10126
|
+
"kind": "interface",
|
|
10127
|
+
"locationInModule": {
|
|
10128
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10129
|
+
"line": 29
|
|
10130
|
+
},
|
|
10131
|
+
"name": "IcebergFieldHistory",
|
|
10132
|
+
"properties": [
|
|
10133
|
+
{
|
|
10134
|
+
"abstract": true,
|
|
10135
|
+
"docs": {
|
|
10136
|
+
"remarks": "Its id stays retired.",
|
|
10137
|
+
"stability": "experimental",
|
|
10138
|
+
"summary": "Whether the field is absent from the latest recorded definition."
|
|
10139
|
+
},
|
|
10140
|
+
"immutable": true,
|
|
10141
|
+
"locationInModule": {
|
|
10142
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10143
|
+
"line": 43
|
|
10144
|
+
},
|
|
10145
|
+
"name": "dropped",
|
|
10146
|
+
"type": {
|
|
10147
|
+
"primitive": "boolean"
|
|
10148
|
+
}
|
|
10149
|
+
},
|
|
10150
|
+
{
|
|
10151
|
+
"abstract": true,
|
|
10152
|
+
"docs": {
|
|
10153
|
+
"stability": "experimental",
|
|
10154
|
+
"summary": "Iceberg field id."
|
|
10155
|
+
},
|
|
10156
|
+
"immutable": true,
|
|
10157
|
+
"locationInModule": {
|
|
10158
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10159
|
+
"line": 31
|
|
10160
|
+
},
|
|
10161
|
+
"name": "id",
|
|
10162
|
+
"type": {
|
|
10163
|
+
"primitive": "number"
|
|
10164
|
+
}
|
|
10165
|
+
},
|
|
10166
|
+
{
|
|
10167
|
+
"abstract": true,
|
|
10168
|
+
"docs": {
|
|
10169
|
+
"stability": "experimental",
|
|
10170
|
+
"summary": "Field path when last seen (`addr.city`, `tags.element`, `attrs.key`)."
|
|
10171
|
+
},
|
|
10172
|
+
"immutable": true,
|
|
10173
|
+
"locationInModule": {
|
|
10174
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10175
|
+
"line": 34
|
|
10176
|
+
},
|
|
10177
|
+
"name": "path",
|
|
10178
|
+
"type": {
|
|
10179
|
+
"primitive": "string"
|
|
10180
|
+
}
|
|
10181
|
+
},
|
|
10182
|
+
{
|
|
10183
|
+
"abstract": true,
|
|
10184
|
+
"docs": {
|
|
10185
|
+
"stability": "experimental",
|
|
10186
|
+
"summary": "Whether the field was required when last seen."
|
|
10187
|
+
},
|
|
10188
|
+
"immutable": true,
|
|
10189
|
+
"locationInModule": {
|
|
10190
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10191
|
+
"line": 40
|
|
10192
|
+
},
|
|
10193
|
+
"name": "required",
|
|
10194
|
+
"type": {
|
|
10195
|
+
"primitive": "boolean"
|
|
10196
|
+
}
|
|
10197
|
+
},
|
|
10198
|
+
{
|
|
10199
|
+
"abstract": true,
|
|
10200
|
+
"docs": {
|
|
10201
|
+
"stability": "experimental",
|
|
10202
|
+
"summary": "Type when last seen: the primitive type string, or `list`, `map` or `struct`."
|
|
10203
|
+
},
|
|
10204
|
+
"immutable": true,
|
|
10205
|
+
"locationInModule": {
|
|
10206
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10207
|
+
"line": 37
|
|
10208
|
+
},
|
|
10209
|
+
"name": "type",
|
|
10210
|
+
"type": {
|
|
10211
|
+
"primitive": "string"
|
|
10212
|
+
}
|
|
10213
|
+
}
|
|
10214
|
+
],
|
|
10215
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergFieldHistory"
|
|
10216
|
+
},
|
|
10118
10217
|
"cdk-glue-iceberg-table.IcebergFormatVersion": {
|
|
10119
10218
|
"assembly": "cdk-glue-iceberg-table",
|
|
10120
10219
|
"docs": {
|
|
@@ -10333,7 +10432,7 @@
|
|
|
10333
10432
|
"kind": "enum",
|
|
10334
10433
|
"locationInModule": {
|
|
10335
10434
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10336
|
-
"line":
|
|
10435
|
+
"line": 74
|
|
10337
10436
|
},
|
|
10338
10437
|
"members": [
|
|
10339
10438
|
{
|
|
@@ -10365,21 +10464,21 @@
|
|
|
10365
10464
|
"kind": "interface",
|
|
10366
10465
|
"locationInModule": {
|
|
10367
10466
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10368
|
-
"line":
|
|
10467
|
+
"line": 112
|
|
10369
10468
|
},
|
|
10370
10469
|
"name": "IcebergPartitionField",
|
|
10371
10470
|
"properties": [
|
|
10372
10471
|
{
|
|
10373
10472
|
"abstract": true,
|
|
10374
10473
|
"docs": {
|
|
10375
|
-
"remarks": "Required: every partition field must\nbe pinned to a stable integer >= 1000 (the Iceberg spec reserves\nids below 1000 for schema columns) that is unique across the\ntable's entire partition-spec history — not just the current\nspec. Iceberg's strict engines (Spark, Trino) reject a table\nwhose spec history carries two different partition fields with\nthe same id.\n\nThe rule is the inverse of the column-id rule: a column keeps\nits id forever, but any change to a partition field — a\ndifferent transform, a different source column — makes it a NEW\nfield that needs a fresh, never-before-used id. Only an\nuntouched field keeps its id across deploys. Once a field is\nchanged or dropped, its id is retired for good; the construct\
|
|
10474
|
+
"remarks": "Required: every partition field must\nbe pinned to a stable integer >= 1000 (the Iceberg spec reserves\nids below 1000 for schema columns) that is unique across the\ntable's entire partition-spec history — not just the current\nspec. Iceberg's strict engines (Spark, Trino) reject a table\nwhose spec history carries two different partition fields with\nthe same id.\n\nThe rule is the inverse of the column-id rule: a column keeps\nits id forever, but any change to a partition field — a\ndifferent transform, a different source column — makes it a NEW\nfield that needs a fresh, never-before-used id. Only an\nuntouched field keeps its id across deploys. Once a field is\nchanged or dropped, its id is retired for good; the construct\nchecks retirement only against a `schemaHistoryFile`.",
|
|
10376
10475
|
"stability": "experimental",
|
|
10377
10476
|
"summary": "Iceberg partition field id."
|
|
10378
10477
|
},
|
|
10379
10478
|
"immutable": true,
|
|
10380
10479
|
"locationInModule": {
|
|
10381
10480
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10382
|
-
"line":
|
|
10481
|
+
"line": 143
|
|
10383
10482
|
},
|
|
10384
10483
|
"name": "fieldId",
|
|
10385
10484
|
"type": {
|
|
@@ -10395,7 +10494,7 @@
|
|
|
10395
10494
|
"immutable": true,
|
|
10396
10495
|
"locationInModule": {
|
|
10397
10496
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10398
|
-
"line":
|
|
10497
|
+
"line": 114
|
|
10399
10498
|
},
|
|
10400
10499
|
"name": "sourceColumn",
|
|
10401
10500
|
"type": {
|
|
@@ -10411,7 +10510,7 @@
|
|
|
10411
10510
|
"immutable": true,
|
|
10412
10511
|
"locationInModule": {
|
|
10413
10512
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10414
|
-
"line":
|
|
10513
|
+
"line": 117
|
|
10415
10514
|
},
|
|
10416
10515
|
"name": "transform",
|
|
10417
10516
|
"type": {
|
|
@@ -10428,7 +10527,7 @@
|
|
|
10428
10527
|
"immutable": true,
|
|
10429
10528
|
"locationInModule": {
|
|
10430
10529
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
10431
|
-
"line":
|
|
10530
|
+
"line": 124
|
|
10432
10531
|
},
|
|
10433
10532
|
"name": "name",
|
|
10434
10533
|
"optional": true,
|
|
@@ -10439,6 +10538,88 @@
|
|
|
10439
10538
|
],
|
|
10440
10539
|
"symbolId": "lib/iceberg/iceberg-table:IcebergPartitionField"
|
|
10441
10540
|
},
|
|
10541
|
+
"cdk-glue-iceberg-table.IcebergPartitionFieldHistory": {
|
|
10542
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
10543
|
+
"datatype": true,
|
|
10544
|
+
"docs": {
|
|
10545
|
+
"stability": "experimental",
|
|
10546
|
+
"summary": "One partition field id a table has used in any partition spec."
|
|
10547
|
+
},
|
|
10548
|
+
"fqn": "cdk-glue-iceberg-table.IcebergPartitionFieldHistory",
|
|
10549
|
+
"kind": "interface",
|
|
10550
|
+
"locationInModule": {
|
|
10551
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10552
|
+
"line": 47
|
|
10553
|
+
},
|
|
10554
|
+
"name": "IcebergPartitionFieldHistory",
|
|
10555
|
+
"properties": [
|
|
10556
|
+
{
|
|
10557
|
+
"abstract": true,
|
|
10558
|
+
"docs": {
|
|
10559
|
+
"stability": "experimental",
|
|
10560
|
+
"summary": "Partition field id."
|
|
10561
|
+
},
|
|
10562
|
+
"immutable": true,
|
|
10563
|
+
"locationInModule": {
|
|
10564
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10565
|
+
"line": 49
|
|
10566
|
+
},
|
|
10567
|
+
"name": "fieldId",
|
|
10568
|
+
"type": {
|
|
10569
|
+
"primitive": "number"
|
|
10570
|
+
}
|
|
10571
|
+
},
|
|
10572
|
+
{
|
|
10573
|
+
"abstract": true,
|
|
10574
|
+
"docs": {
|
|
10575
|
+
"stability": "experimental",
|
|
10576
|
+
"summary": "Partition field name when last seen."
|
|
10577
|
+
},
|
|
10578
|
+
"immutable": true,
|
|
10579
|
+
"locationInModule": {
|
|
10580
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10581
|
+
"line": 52
|
|
10582
|
+
},
|
|
10583
|
+
"name": "name",
|
|
10584
|
+
"type": {
|
|
10585
|
+
"primitive": "string"
|
|
10586
|
+
}
|
|
10587
|
+
},
|
|
10588
|
+
{
|
|
10589
|
+
"abstract": true,
|
|
10590
|
+
"docs": {
|
|
10591
|
+
"stability": "experimental",
|
|
10592
|
+
"summary": "Field id of the source column."
|
|
10593
|
+
},
|
|
10594
|
+
"immutable": true,
|
|
10595
|
+
"locationInModule": {
|
|
10596
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10597
|
+
"line": 55
|
|
10598
|
+
},
|
|
10599
|
+
"name": "sourceId",
|
|
10600
|
+
"type": {
|
|
10601
|
+
"primitive": "number"
|
|
10602
|
+
}
|
|
10603
|
+
},
|
|
10604
|
+
{
|
|
10605
|
+
"abstract": true,
|
|
10606
|
+
"docs": {
|
|
10607
|
+
"stability": "experimental",
|
|
10608
|
+
"summary": "Transform string (`identity`, `bucket[16]`, `day`, ...)."
|
|
10609
|
+
},
|
|
10610
|
+
"immutable": true,
|
|
10611
|
+
"locationInModule": {
|
|
10612
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
10613
|
+
"line": 58
|
|
10614
|
+
},
|
|
10615
|
+
"name": "transform",
|
|
10616
|
+
"type": {
|
|
10617
|
+
"primitive": "string"
|
|
10618
|
+
}
|
|
10619
|
+
}
|
|
10620
|
+
],
|
|
10621
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergPartitionFieldHistory"
|
|
10622
|
+
},
|
|
10442
10623
|
"cdk-glue-iceberg-table.IcebergPartitionTransform": {
|
|
10443
10624
|
"assembly": "cdk-glue-iceberg-table",
|
|
10444
10625
|
"docs": {
|
|
@@ -11095,87 +11276,438 @@
|
|
|
11095
11276
|
],
|
|
11096
11277
|
"symbolId": "lib/iceberg/iceberg-table-properties:IcebergPropertyKeys"
|
|
11097
11278
|
},
|
|
11098
|
-
"cdk-glue-iceberg-table.
|
|
11279
|
+
"cdk-glue-iceberg-table.IcebergSchemaHistory": {
|
|
11099
11280
|
"assembly": "cdk-glue-iceberg-table",
|
|
11100
11281
|
"docs": {
|
|
11101
11282
|
"stability": "experimental",
|
|
11102
|
-
"summary": "
|
|
11283
|
+
"summary": "Checks and maintains the committed schema history that `IcebergTable` compares a definition against when `schemaHistoryFile` is set."
|
|
11103
11284
|
},
|
|
11104
|
-
"fqn": "cdk-glue-iceberg-table.
|
|
11105
|
-
"
|
|
11106
|
-
|
|
11107
|
-
|
|
11108
|
-
"line": 62
|
|
11109
|
-
},
|
|
11110
|
-
"members": [
|
|
11111
|
-
{
|
|
11112
|
-
"docs": {
|
|
11113
|
-
"stability": "experimental",
|
|
11114
|
-
"summary": "Ascending — smaller values first."
|
|
11115
|
-
},
|
|
11116
|
-
"name": "ASC"
|
|
11117
|
-
},
|
|
11118
|
-
{
|
|
11119
|
-
"docs": {
|
|
11120
|
-
"stability": "experimental",
|
|
11121
|
-
"summary": "Descending — larger values first."
|
|
11122
|
-
},
|
|
11123
|
-
"name": "DESC"
|
|
11285
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaHistory",
|
|
11286
|
+
"initializer": {
|
|
11287
|
+
"docs": {
|
|
11288
|
+
"stability": "experimental"
|
|
11124
11289
|
}
|
|
11125
|
-
],
|
|
11126
|
-
"name": "IcebergSortDirection",
|
|
11127
|
-
"symbolId": "lib/iceberg/iceberg-table:IcebergSortDirection"
|
|
11128
|
-
},
|
|
11129
|
-
"cdk-glue-iceberg-table.IcebergSortField": {
|
|
11130
|
-
"assembly": "cdk-glue-iceberg-table",
|
|
11131
|
-
"datatype": true,
|
|
11132
|
-
"docs": {
|
|
11133
|
-
"stability": "experimental",
|
|
11134
|
-
"summary": "One write-order (sort) field."
|
|
11135
11290
|
},
|
|
11136
|
-
"
|
|
11137
|
-
"kind": "interface",
|
|
11291
|
+
"kind": "class",
|
|
11138
11292
|
"locationInModule": {
|
|
11139
|
-
"filename": "lib/iceberg/iceberg-
|
|
11140
|
-
"line":
|
|
11293
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11294
|
+
"line": 125
|
|
11141
11295
|
},
|
|
11142
|
-
"
|
|
11143
|
-
"properties": [
|
|
11296
|
+
"methods": [
|
|
11144
11297
|
{
|
|
11145
|
-
"abstract": true,
|
|
11146
11298
|
"docs": {
|
|
11299
|
+
"remarks": "ids\nabsent from `next` stay in the record as dropped. Throws if the change is unsafe.",
|
|
11147
11300
|
"stability": "experimental",
|
|
11148
|
-
"summary": "
|
|
11301
|
+
"summary": "`history` with `next` recorded as the current definition of `tableName`;"
|
|
11149
11302
|
},
|
|
11150
|
-
"immutable": true,
|
|
11151
11303
|
"locationInModule": {
|
|
11152
|
-
"filename": "lib/iceberg/iceberg-
|
|
11153
|
-
"line":
|
|
11304
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11305
|
+
"line": 153
|
|
11154
11306
|
},
|
|
11155
|
-
"name": "
|
|
11156
|
-
"
|
|
11157
|
-
|
|
11158
|
-
|
|
11307
|
+
"name": "append",
|
|
11308
|
+
"parameters": [
|
|
11309
|
+
{
|
|
11310
|
+
"docs": {
|
|
11311
|
+
"summary": "The schema history."
|
|
11312
|
+
},
|
|
11313
|
+
"name": "history",
|
|
11314
|
+
"type": {
|
|
11315
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaHistoryFile"
|
|
11316
|
+
}
|
|
11317
|
+
},
|
|
11318
|
+
{
|
|
11319
|
+
"docs": {
|
|
11320
|
+
"summary": "Glue table name."
|
|
11321
|
+
},
|
|
11322
|
+
"name": "tableName",
|
|
11323
|
+
"type": {
|
|
11324
|
+
"primitive": "string"
|
|
11325
|
+
}
|
|
11326
|
+
},
|
|
11327
|
+
{
|
|
11328
|
+
"docs": {
|
|
11329
|
+
"summary": "The table definition to record."
|
|
11330
|
+
},
|
|
11331
|
+
"name": "next",
|
|
11332
|
+
"type": {
|
|
11333
|
+
"fqn": "cdk-glue-iceberg-table.IcebergTableSchema"
|
|
11334
|
+
}
|
|
11335
|
+
}
|
|
11336
|
+
],
|
|
11337
|
+
"returns": {
|
|
11338
|
+
"type": {
|
|
11339
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaHistoryFile"
|
|
11340
|
+
}
|
|
11341
|
+
},
|
|
11342
|
+
"static": true
|
|
11159
11343
|
},
|
|
11160
11344
|
{
|
|
11161
|
-
"abstract": true,
|
|
11162
11345
|
"docs": {
|
|
11163
|
-
"remarks": "
|
|
11346
|
+
"remarks": "Throws on JSON that is not a schema history.",
|
|
11164
11347
|
"stability": "experimental",
|
|
11165
|
-
"summary": "
|
|
11348
|
+
"summary": "Parse a schema history file's contents."
|
|
11166
11349
|
},
|
|
11167
|
-
"immutable": true,
|
|
11168
11350
|
"locationInModule": {
|
|
11169
|
-
"filename": "lib/iceberg/iceberg-
|
|
11170
|
-
"line":
|
|
11351
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11352
|
+
"line": 198
|
|
11171
11353
|
},
|
|
11172
|
-
"name": "
|
|
11173
|
-
"
|
|
11174
|
-
|
|
11175
|
-
|
|
11176
|
-
|
|
11177
|
-
|
|
11178
|
-
|
|
11354
|
+
"name": "fromJson",
|
|
11355
|
+
"parameters": [
|
|
11356
|
+
{
|
|
11357
|
+
"docs": {
|
|
11358
|
+
"summary": "File contents."
|
|
11359
|
+
},
|
|
11360
|
+
"name": "json",
|
|
11361
|
+
"type": {
|
|
11362
|
+
"primitive": "string"
|
|
11363
|
+
}
|
|
11364
|
+
}
|
|
11365
|
+
],
|
|
11366
|
+
"returns": {
|
|
11367
|
+
"type": {
|
|
11368
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaHistoryFile"
|
|
11369
|
+
}
|
|
11370
|
+
},
|
|
11371
|
+
"static": true
|
|
11372
|
+
},
|
|
11373
|
+
{
|
|
11374
|
+
"docs": {
|
|
11375
|
+
"stability": "experimental",
|
|
11376
|
+
"summary": "Serialize `history` as two-space indented JSON with a trailing newline, tables sorted by name, fields by id and partition fields by field id."
|
|
11377
|
+
},
|
|
11378
|
+
"locationInModule": {
|
|
11379
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11380
|
+
"line": 166
|
|
11381
|
+
},
|
|
11382
|
+
"name": "toJson",
|
|
11383
|
+
"parameters": [
|
|
11384
|
+
{
|
|
11385
|
+
"docs": {
|
|
11386
|
+
"summary": "The schema history."
|
|
11387
|
+
},
|
|
11388
|
+
"name": "history",
|
|
11389
|
+
"type": {
|
|
11390
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaHistoryFile"
|
|
11391
|
+
}
|
|
11392
|
+
}
|
|
11393
|
+
],
|
|
11394
|
+
"returns": {
|
|
11395
|
+
"type": {
|
|
11396
|
+
"primitive": "string"
|
|
11397
|
+
}
|
|
11398
|
+
},
|
|
11399
|
+
"static": true
|
|
11400
|
+
},
|
|
11401
|
+
{
|
|
11402
|
+
"docs": {
|
|
11403
|
+
"remarks": "Run it after\n`cdk synth`; it writes nothing if any table's change is unsafe.",
|
|
11404
|
+
"stability": "experimental",
|
|
11405
|
+
"summary": "Record every table that set `schemaHistoryFile` in a synthesized cloud assembly into its history file, and return the paths of the files written."
|
|
11406
|
+
},
|
|
11407
|
+
"locationInModule": {
|
|
11408
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11409
|
+
"line": 208
|
|
11410
|
+
},
|
|
11411
|
+
"name": "updateFromCloudAssembly",
|
|
11412
|
+
"parameters": [
|
|
11413
|
+
{
|
|
11414
|
+
"docs": {
|
|
11415
|
+
"summary": "The cloud assembly directory, usually `cdk.out`."
|
|
11416
|
+
},
|
|
11417
|
+
"name": "assemblyDirectory",
|
|
11418
|
+
"type": {
|
|
11419
|
+
"primitive": "string"
|
|
11420
|
+
}
|
|
11421
|
+
}
|
|
11422
|
+
],
|
|
11423
|
+
"returns": {
|
|
11424
|
+
"type": {
|
|
11425
|
+
"collection": {
|
|
11426
|
+
"elementtype": {
|
|
11427
|
+
"primitive": "string"
|
|
11428
|
+
},
|
|
11429
|
+
"kind": "array"
|
|
11430
|
+
}
|
|
11431
|
+
}
|
|
11432
|
+
},
|
|
11433
|
+
"static": true
|
|
11434
|
+
},
|
|
11435
|
+
{
|
|
11436
|
+
"docs": {
|
|
11437
|
+
"remarks": "empty when the change is safe. Fields\nare identified by id, so a field whose path changed under the same id is a rename.",
|
|
11438
|
+
"stability": "experimental",
|
|
11439
|
+
"summary": "Unsafe changes from `previous` to `next`;"
|
|
11440
|
+
},
|
|
11441
|
+
"locationInModule": {
|
|
11442
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11443
|
+
"line": 132
|
|
11444
|
+
},
|
|
11445
|
+
"name": "validateSchemaEvolution",
|
|
11446
|
+
"parameters": [
|
|
11447
|
+
{
|
|
11448
|
+
"docs": {
|
|
11449
|
+
"summary": "The table's history record."
|
|
11450
|
+
},
|
|
11451
|
+
"name": "previous",
|
|
11452
|
+
"type": {
|
|
11453
|
+
"fqn": "cdk-glue-iceberg-table.IcebergTableHistory"
|
|
11454
|
+
}
|
|
11455
|
+
},
|
|
11456
|
+
{
|
|
11457
|
+
"docs": {
|
|
11458
|
+
"summary": "The new table definition."
|
|
11459
|
+
},
|
|
11460
|
+
"name": "next",
|
|
11461
|
+
"type": {
|
|
11462
|
+
"fqn": "cdk-glue-iceberg-table.IcebergTableSchema"
|
|
11463
|
+
}
|
|
11464
|
+
}
|
|
11465
|
+
],
|
|
11466
|
+
"returns": {
|
|
11467
|
+
"type": {
|
|
11468
|
+
"collection": {
|
|
11469
|
+
"elementtype": {
|
|
11470
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaViolation"
|
|
11471
|
+
},
|
|
11472
|
+
"kind": "array"
|
|
11473
|
+
}
|
|
11474
|
+
}
|
|
11475
|
+
},
|
|
11476
|
+
"static": true
|
|
11477
|
+
}
|
|
11478
|
+
],
|
|
11479
|
+
"name": "IcebergSchemaHistory",
|
|
11480
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergSchemaHistory"
|
|
11481
|
+
},
|
|
11482
|
+
"cdk-glue-iceberg-table.IcebergSchemaHistoryFile": {
|
|
11483
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
11484
|
+
"datatype": true,
|
|
11485
|
+
"docs": {
|
|
11486
|
+
"stability": "experimental",
|
|
11487
|
+
"summary": "Contents of a schema history file: one record per table."
|
|
11488
|
+
},
|
|
11489
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaHistoryFile",
|
|
11490
|
+
"kind": "interface",
|
|
11491
|
+
"locationInModule": {
|
|
11492
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11493
|
+
"line": 74
|
|
11494
|
+
},
|
|
11495
|
+
"name": "IcebergSchemaHistoryFile",
|
|
11496
|
+
"properties": [
|
|
11497
|
+
{
|
|
11498
|
+
"abstract": true,
|
|
11499
|
+
"docs": {
|
|
11500
|
+
"stability": "experimental",
|
|
11501
|
+
"summary": "Table records, ordered by table name."
|
|
11502
|
+
},
|
|
11503
|
+
"immutable": true,
|
|
11504
|
+
"locationInModule": {
|
|
11505
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11506
|
+
"line": 76
|
|
11507
|
+
},
|
|
11508
|
+
"name": "tables",
|
|
11509
|
+
"type": {
|
|
11510
|
+
"collection": {
|
|
11511
|
+
"elementtype": {
|
|
11512
|
+
"fqn": "cdk-glue-iceberg-table.IcebergTableHistory"
|
|
11513
|
+
},
|
|
11514
|
+
"kind": "array"
|
|
11515
|
+
}
|
|
11516
|
+
}
|
|
11517
|
+
}
|
|
11518
|
+
],
|
|
11519
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergSchemaHistoryFile"
|
|
11520
|
+
},
|
|
11521
|
+
"cdk-glue-iceberg-table.IcebergSchemaViolation": {
|
|
11522
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
11523
|
+
"datatype": true,
|
|
11524
|
+
"docs": {
|
|
11525
|
+
"stability": "experimental",
|
|
11526
|
+
"summary": "One unsafe change between a table's history and a new definition."
|
|
11527
|
+
},
|
|
11528
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaViolation",
|
|
11529
|
+
"kind": "interface",
|
|
11530
|
+
"locationInModule": {
|
|
11531
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11532
|
+
"line": 110
|
|
11533
|
+
},
|
|
11534
|
+
"name": "IcebergSchemaViolation",
|
|
11535
|
+
"properties": [
|
|
11536
|
+
{
|
|
11537
|
+
"abstract": true,
|
|
11538
|
+
"docs": {
|
|
11539
|
+
"stability": "experimental",
|
|
11540
|
+
"summary": "Class of the change."
|
|
11541
|
+
},
|
|
11542
|
+
"immutable": true,
|
|
11543
|
+
"locationInModule": {
|
|
11544
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11545
|
+
"line": 112
|
|
11546
|
+
},
|
|
11547
|
+
"name": "kind",
|
|
11548
|
+
"type": {
|
|
11549
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaViolationKind"
|
|
11550
|
+
}
|
|
11551
|
+
},
|
|
11552
|
+
{
|
|
11553
|
+
"abstract": true,
|
|
11554
|
+
"docs": {
|
|
11555
|
+
"stability": "experimental",
|
|
11556
|
+
"summary": "Description naming the field and the ids involved."
|
|
11557
|
+
},
|
|
11558
|
+
"immutable": true,
|
|
11559
|
+
"locationInModule": {
|
|
11560
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11561
|
+
"line": 115
|
|
11562
|
+
},
|
|
11563
|
+
"name": "message",
|
|
11564
|
+
"type": {
|
|
11565
|
+
"primitive": "string"
|
|
11566
|
+
}
|
|
11567
|
+
}
|
|
11568
|
+
],
|
|
11569
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergSchemaViolation"
|
|
11570
|
+
},
|
|
11571
|
+
"cdk-glue-iceberg-table.IcebergSchemaViolationKind": {
|
|
11572
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
11573
|
+
"docs": {
|
|
11574
|
+
"stability": "experimental",
|
|
11575
|
+
"summary": "Class of an unsafe schema change."
|
|
11576
|
+
},
|
|
11577
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSchemaViolationKind",
|
|
11578
|
+
"kind": "enum",
|
|
11579
|
+
"locationInModule": {
|
|
11580
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
11581
|
+
"line": 89
|
|
11582
|
+
},
|
|
11583
|
+
"members": [
|
|
11584
|
+
{
|
|
11585
|
+
"docs": {
|
|
11586
|
+
"stability": "experimental",
|
|
11587
|
+
"summary": "A field id from the history is used by a different field."
|
|
11588
|
+
},
|
|
11589
|
+
"name": "FIELD_ID_REUSED"
|
|
11590
|
+
},
|
|
11591
|
+
{
|
|
11592
|
+
"docs": {
|
|
11593
|
+
"stability": "experimental",
|
|
11594
|
+
"summary": "A field kept its path but changed its id."
|
|
11595
|
+
},
|
|
11596
|
+
"name": "FIELD_ID_CHANGED"
|
|
11597
|
+
},
|
|
11598
|
+
{
|
|
11599
|
+
"docs": {
|
|
11600
|
+
"stability": "experimental",
|
|
11601
|
+
"summary": "A field changed to a type Iceberg cannot promote it to."
|
|
11602
|
+
},
|
|
11603
|
+
"name": "TYPE_CHANGED"
|
|
11604
|
+
},
|
|
11605
|
+
{
|
|
11606
|
+
"docs": {
|
|
11607
|
+
"stability": "experimental",
|
|
11608
|
+
"summary": "An optional field became required."
|
|
11609
|
+
},
|
|
11610
|
+
"name": "OPTIONAL_TO_REQUIRED"
|
|
11611
|
+
},
|
|
11612
|
+
{
|
|
11613
|
+
"docs": {
|
|
11614
|
+
"stability": "experimental",
|
|
11615
|
+
"summary": "A column that a recorded partition spec is sourced from was dropped."
|
|
11616
|
+
},
|
|
11617
|
+
"name": "PARTITION_SOURCE_DROPPED"
|
|
11618
|
+
},
|
|
11619
|
+
{
|
|
11620
|
+
"docs": {
|
|
11621
|
+
"stability": "experimental",
|
|
11622
|
+
"summary": "A partition field id is used with a different source or transform."
|
|
11623
|
+
},
|
|
11624
|
+
"name": "PARTITION_FIELD_CHANGED"
|
|
11625
|
+
}
|
|
11626
|
+
],
|
|
11627
|
+
"name": "IcebergSchemaViolationKind",
|
|
11628
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergSchemaViolationKind"
|
|
11629
|
+
},
|
|
11630
|
+
"cdk-glue-iceberg-table.IcebergSortDirection": {
|
|
11631
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
11632
|
+
"docs": {
|
|
11633
|
+
"stability": "experimental",
|
|
11634
|
+
"summary": "Sort direction for an Iceberg sort field."
|
|
11635
|
+
},
|
|
11636
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSortDirection",
|
|
11637
|
+
"kind": "enum",
|
|
11638
|
+
"locationInModule": {
|
|
11639
|
+
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11640
|
+
"line": 65
|
|
11641
|
+
},
|
|
11642
|
+
"members": [
|
|
11643
|
+
{
|
|
11644
|
+
"docs": {
|
|
11645
|
+
"stability": "experimental",
|
|
11646
|
+
"summary": "Ascending — smaller values first."
|
|
11647
|
+
},
|
|
11648
|
+
"name": "ASC"
|
|
11649
|
+
},
|
|
11650
|
+
{
|
|
11651
|
+
"docs": {
|
|
11652
|
+
"stability": "experimental",
|
|
11653
|
+
"summary": "Descending — larger values first."
|
|
11654
|
+
},
|
|
11655
|
+
"name": "DESC"
|
|
11656
|
+
}
|
|
11657
|
+
],
|
|
11658
|
+
"name": "IcebergSortDirection",
|
|
11659
|
+
"symbolId": "lib/iceberg/iceberg-table:IcebergSortDirection"
|
|
11660
|
+
},
|
|
11661
|
+
"cdk-glue-iceberg-table.IcebergSortField": {
|
|
11662
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
11663
|
+
"datatype": true,
|
|
11664
|
+
"docs": {
|
|
11665
|
+
"stability": "experimental",
|
|
11666
|
+
"summary": "One write-order (sort) field."
|
|
11667
|
+
},
|
|
11668
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSortField",
|
|
11669
|
+
"kind": "interface",
|
|
11670
|
+
"locationInModule": {
|
|
11671
|
+
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11672
|
+
"line": 147
|
|
11673
|
+
},
|
|
11674
|
+
"name": "IcebergSortField",
|
|
11675
|
+
"properties": [
|
|
11676
|
+
{
|
|
11677
|
+
"abstract": true,
|
|
11678
|
+
"docs": {
|
|
11679
|
+
"stability": "experimental",
|
|
11680
|
+
"summary": "Name of the top-level table column to sort on."
|
|
11681
|
+
},
|
|
11682
|
+
"immutable": true,
|
|
11683
|
+
"locationInModule": {
|
|
11684
|
+
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11685
|
+
"line": 149
|
|
11686
|
+
},
|
|
11687
|
+
"name": "sourceColumn",
|
|
11688
|
+
"type": {
|
|
11689
|
+
"primitive": "string"
|
|
11690
|
+
}
|
|
11691
|
+
},
|
|
11692
|
+
{
|
|
11693
|
+
"abstract": true,
|
|
11694
|
+
"docs": {
|
|
11695
|
+
"remarks": "Defaults to `ASC`.",
|
|
11696
|
+
"stability": "experimental",
|
|
11697
|
+
"summary": "Sort direction."
|
|
11698
|
+
},
|
|
11699
|
+
"immutable": true,
|
|
11700
|
+
"locationInModule": {
|
|
11701
|
+
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11702
|
+
"line": 155
|
|
11703
|
+
},
|
|
11704
|
+
"name": "direction",
|
|
11705
|
+
"optional": true,
|
|
11706
|
+
"type": {
|
|
11707
|
+
"fqn": "cdk-glue-iceberg-table.IcebergSortDirection"
|
|
11708
|
+
}
|
|
11709
|
+
},
|
|
11710
|
+
{
|
|
11179
11711
|
"abstract": true,
|
|
11180
11712
|
"docs": {
|
|
11181
11713
|
"remarks": "Defaults to `NULLS_LAST`.",
|
|
@@ -11185,7 +11717,7 @@
|
|
|
11185
11717
|
"immutable": true,
|
|
11186
11718
|
"locationInModule": {
|
|
11187
11719
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11188
|
-
"line":
|
|
11720
|
+
"line": 158
|
|
11189
11721
|
},
|
|
11190
11722
|
"name": "nullOrder",
|
|
11191
11723
|
"optional": true,
|
|
@@ -11203,7 +11735,7 @@
|
|
|
11203
11735
|
"immutable": true,
|
|
11204
11736
|
"locationInModule": {
|
|
11205
11737
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11206
|
-
"line":
|
|
11738
|
+
"line": 152
|
|
11207
11739
|
},
|
|
11208
11740
|
"name": "transform",
|
|
11209
11741
|
"optional": true,
|
|
@@ -11330,7 +11862,7 @@
|
|
|
11330
11862
|
},
|
|
11331
11863
|
"locationInModule": {
|
|
11332
11864
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11333
|
-
"line":
|
|
11865
|
+
"line": 299
|
|
11334
11866
|
},
|
|
11335
11867
|
"parameters": [
|
|
11336
11868
|
{
|
|
@@ -11359,7 +11891,7 @@
|
|
|
11359
11891
|
"kind": "class",
|
|
11360
11892
|
"locationInModule": {
|
|
11361
11893
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11362
|
-
"line":
|
|
11894
|
+
"line": 273
|
|
11363
11895
|
},
|
|
11364
11896
|
"methods": [
|
|
11365
11897
|
{
|
|
@@ -11369,7 +11901,7 @@
|
|
|
11369
11901
|
},
|
|
11370
11902
|
"locationInModule": {
|
|
11371
11903
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11372
|
-
"line":
|
|
11904
|
+
"line": 379
|
|
11373
11905
|
},
|
|
11374
11906
|
"name": "creationOrderFieldIds",
|
|
11375
11907
|
"parameters": [
|
|
@@ -11408,7 +11940,7 @@
|
|
|
11408
11940
|
},
|
|
11409
11941
|
"locationInModule": {
|
|
11410
11942
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11411
|
-
"line":
|
|
11943
|
+
"line": 389
|
|
11412
11944
|
},
|
|
11413
11945
|
"name": "fromIcebergTableAttributes",
|
|
11414
11946
|
"parameters": [
|
|
@@ -11445,7 +11977,7 @@
|
|
|
11445
11977
|
},
|
|
11446
11978
|
"locationInModule": {
|
|
11447
11979
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11448
|
-
"line":
|
|
11980
|
+
"line": 479
|
|
11449
11981
|
},
|
|
11450
11982
|
"name": "grantRead",
|
|
11451
11983
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -11470,7 +12002,7 @@
|
|
|
11470
12002
|
},
|
|
11471
12003
|
"locationInModule": {
|
|
11472
12004
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11473
|
-
"line":
|
|
12005
|
+
"line": 507
|
|
11474
12006
|
},
|
|
11475
12007
|
"name": "grantReadWrite",
|
|
11476
12008
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -11495,7 +12027,7 @@
|
|
|
11495
12027
|
},
|
|
11496
12028
|
"locationInModule": {
|
|
11497
12029
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11498
|
-
"line":
|
|
12030
|
+
"line": 493
|
|
11499
12031
|
},
|
|
11500
12032
|
"name": "grantWrite",
|
|
11501
12033
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -11524,7 +12056,7 @@
|
|
|
11524
12056
|
"immutable": true,
|
|
11525
12057
|
"locationInModule": {
|
|
11526
12058
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11527
|
-
"line":
|
|
12059
|
+
"line": 281
|
|
11528
12060
|
},
|
|
11529
12061
|
"name": "database",
|
|
11530
12062
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -11540,7 +12072,7 @@
|
|
|
11540
12072
|
"immutable": true,
|
|
11541
12073
|
"locationInModule": {
|
|
11542
12074
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11543
|
-
"line":
|
|
12075
|
+
"line": 287
|
|
11544
12076
|
},
|
|
11545
12077
|
"name": "dataFormat",
|
|
11546
12078
|
"type": {
|
|
@@ -11555,7 +12087,7 @@
|
|
|
11555
12087
|
"immutable": true,
|
|
11556
12088
|
"locationInModule": {
|
|
11557
12089
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11558
|
-
"line":
|
|
12090
|
+
"line": 290
|
|
11559
12091
|
},
|
|
11560
12092
|
"name": "formatVersion",
|
|
11561
12093
|
"type": {
|
|
@@ -11570,7 +12102,7 @@
|
|
|
11570
12102
|
"immutable": true,
|
|
11571
12103
|
"locationInModule": {
|
|
11572
12104
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11573
|
-
"line":
|
|
12105
|
+
"line": 284
|
|
11574
12106
|
},
|
|
11575
12107
|
"name": "location",
|
|
11576
12108
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -11586,7 +12118,7 @@
|
|
|
11586
12118
|
"immutable": true,
|
|
11587
12119
|
"locationInModule": {
|
|
11588
12120
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11589
|
-
"line":
|
|
12121
|
+
"line": 293
|
|
11590
12122
|
},
|
|
11591
12123
|
"name": "resource",
|
|
11592
12124
|
"type": {
|
|
@@ -11601,7 +12133,7 @@
|
|
|
11601
12133
|
"immutable": true,
|
|
11602
12134
|
"locationInModule": {
|
|
11603
12135
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11604
|
-
"line":
|
|
12136
|
+
"line": 275
|
|
11605
12137
|
},
|
|
11606
12138
|
"name": "tableArn",
|
|
11607
12139
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -11617,7 +12149,7 @@
|
|
|
11617
12149
|
"immutable": true,
|
|
11618
12150
|
"locationInModule": {
|
|
11619
12151
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11620
|
-
"line":
|
|
12152
|
+
"line": 278
|
|
11621
12153
|
},
|
|
11622
12154
|
"name": "tableName",
|
|
11623
12155
|
"overrides": "cdk-glue-iceberg-table.IIcebergTable",
|
|
@@ -11639,7 +12171,7 @@
|
|
|
11639
12171
|
"kind": "interface",
|
|
11640
12172
|
"locationInModule": {
|
|
11641
12173
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11642
|
-
"line":
|
|
12174
|
+
"line": 531
|
|
11643
12175
|
},
|
|
11644
12176
|
"name": "IcebergTableAttributes",
|
|
11645
12177
|
"properties": [
|
|
@@ -11652,7 +12184,7 @@
|
|
|
11652
12184
|
"immutable": true,
|
|
11653
12185
|
"locationInModule": {
|
|
11654
12186
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11655
|
-
"line":
|
|
12187
|
+
"line": 533
|
|
11656
12188
|
},
|
|
11657
12189
|
"name": "database",
|
|
11658
12190
|
"type": {
|
|
@@ -11668,7 +12200,7 @@
|
|
|
11668
12200
|
"immutable": true,
|
|
11669
12201
|
"locationInModule": {
|
|
11670
12202
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11671
|
-
"line":
|
|
12203
|
+
"line": 539
|
|
11672
12204
|
},
|
|
11673
12205
|
"name": "location",
|
|
11674
12206
|
"type": {
|
|
@@ -11684,7 +12216,7 @@
|
|
|
11684
12216
|
"immutable": true,
|
|
11685
12217
|
"locationInModule": {
|
|
11686
12218
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11687
|
-
"line":
|
|
12219
|
+
"line": 536
|
|
11688
12220
|
},
|
|
11689
12221
|
"name": "tableName",
|
|
11690
12222
|
"type": {
|
|
@@ -11694,6 +12226,82 @@
|
|
|
11694
12226
|
],
|
|
11695
12227
|
"symbolId": "lib/iceberg/iceberg-table:IcebergTableAttributes"
|
|
11696
12228
|
},
|
|
12229
|
+
"cdk-glue-iceberg-table.IcebergTableHistory": {
|
|
12230
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
12231
|
+
"datatype": true,
|
|
12232
|
+
"docs": {
|
|
12233
|
+
"stability": "experimental",
|
|
12234
|
+
"summary": "Every field and partition field id one table has used."
|
|
12235
|
+
},
|
|
12236
|
+
"fqn": "cdk-glue-iceberg-table.IcebergTableHistory",
|
|
12237
|
+
"kind": "interface",
|
|
12238
|
+
"locationInModule": {
|
|
12239
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12240
|
+
"line": 62
|
|
12241
|
+
},
|
|
12242
|
+
"name": "IcebergTableHistory",
|
|
12243
|
+
"properties": [
|
|
12244
|
+
{
|
|
12245
|
+
"abstract": true,
|
|
12246
|
+
"docs": {
|
|
12247
|
+
"stability": "experimental",
|
|
12248
|
+
"summary": "Fields, ordered by id."
|
|
12249
|
+
},
|
|
12250
|
+
"immutable": true,
|
|
12251
|
+
"locationInModule": {
|
|
12252
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12253
|
+
"line": 67
|
|
12254
|
+
},
|
|
12255
|
+
"name": "fields",
|
|
12256
|
+
"type": {
|
|
12257
|
+
"collection": {
|
|
12258
|
+
"elementtype": {
|
|
12259
|
+
"fqn": "cdk-glue-iceberg-table.IcebergFieldHistory"
|
|
12260
|
+
},
|
|
12261
|
+
"kind": "array"
|
|
12262
|
+
}
|
|
12263
|
+
}
|
|
12264
|
+
},
|
|
12265
|
+
{
|
|
12266
|
+
"abstract": true,
|
|
12267
|
+
"docs": {
|
|
12268
|
+
"stability": "experimental",
|
|
12269
|
+
"summary": "Partition fields, ordered by field id."
|
|
12270
|
+
},
|
|
12271
|
+
"immutable": true,
|
|
12272
|
+
"locationInModule": {
|
|
12273
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12274
|
+
"line": 70
|
|
12275
|
+
},
|
|
12276
|
+
"name": "partitionFields",
|
|
12277
|
+
"type": {
|
|
12278
|
+
"collection": {
|
|
12279
|
+
"elementtype": {
|
|
12280
|
+
"fqn": "cdk-glue-iceberg-table.IcebergPartitionFieldHistory"
|
|
12281
|
+
},
|
|
12282
|
+
"kind": "array"
|
|
12283
|
+
}
|
|
12284
|
+
}
|
|
12285
|
+
},
|
|
12286
|
+
{
|
|
12287
|
+
"abstract": true,
|
|
12288
|
+
"docs": {
|
|
12289
|
+
"stability": "experimental",
|
|
12290
|
+
"summary": "Glue table name."
|
|
12291
|
+
},
|
|
12292
|
+
"immutable": true,
|
|
12293
|
+
"locationInModule": {
|
|
12294
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12295
|
+
"line": 64
|
|
12296
|
+
},
|
|
12297
|
+
"name": "tableName",
|
|
12298
|
+
"type": {
|
|
12299
|
+
"primitive": "string"
|
|
12300
|
+
}
|
|
12301
|
+
}
|
|
12302
|
+
],
|
|
12303
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergTableHistory"
|
|
12304
|
+
},
|
|
11697
12305
|
"cdk-glue-iceberg-table.IcebergTableProps": {
|
|
11698
12306
|
"assembly": "cdk-glue-iceberg-table",
|
|
11699
12307
|
"datatype": true,
|
|
@@ -11705,7 +12313,7 @@
|
|
|
11705
12313
|
"kind": "interface",
|
|
11706
12314
|
"locationInModule": {
|
|
11707
12315
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11708
|
-
"line":
|
|
12316
|
+
"line": 162
|
|
11709
12317
|
},
|
|
11710
12318
|
"name": "IcebergTableProps",
|
|
11711
12319
|
"properties": [
|
|
@@ -11719,7 +12327,7 @@
|
|
|
11719
12327
|
"immutable": true,
|
|
11720
12328
|
"locationInModule": {
|
|
11721
12329
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11722
|
-
"line":
|
|
12330
|
+
"line": 170
|
|
11723
12331
|
},
|
|
11724
12332
|
"name": "columns",
|
|
11725
12333
|
"type": {
|
|
@@ -11740,7 +12348,7 @@
|
|
|
11740
12348
|
"immutable": true,
|
|
11741
12349
|
"locationInModule": {
|
|
11742
12350
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11743
|
-
"line":
|
|
12351
|
+
"line": 164
|
|
11744
12352
|
},
|
|
11745
12353
|
"name": "database",
|
|
11746
12354
|
"type": {
|
|
@@ -11757,7 +12365,7 @@
|
|
|
11757
12365
|
"immutable": true,
|
|
11758
12366
|
"locationInModule": {
|
|
11759
12367
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11760
|
-
"line":
|
|
12368
|
+
"line": 196
|
|
11761
12369
|
},
|
|
11762
12370
|
"name": "formatVersion",
|
|
11763
12371
|
"type": {
|
|
@@ -11774,7 +12382,7 @@
|
|
|
11774
12382
|
"immutable": true,
|
|
11775
12383
|
"locationInModule": {
|
|
11776
12384
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11777
|
-
"line":
|
|
12385
|
+
"line": 173
|
|
11778
12386
|
},
|
|
11779
12387
|
"name": "location",
|
|
11780
12388
|
"type": {
|
|
@@ -11790,7 +12398,7 @@
|
|
|
11790
12398
|
"immutable": true,
|
|
11791
12399
|
"locationInModule": {
|
|
11792
12400
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11793
|
-
"line":
|
|
12401
|
+
"line": 167
|
|
11794
12402
|
},
|
|
11795
12403
|
"name": "tableName",
|
|
11796
12404
|
"type": {
|
|
@@ -11807,7 +12415,7 @@
|
|
|
11807
12415
|
"immutable": true,
|
|
11808
12416
|
"locationInModule": {
|
|
11809
12417
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11810
|
-
"line":
|
|
12418
|
+
"line": 216
|
|
11811
12419
|
},
|
|
11812
12420
|
"name": "comment",
|
|
11813
12421
|
"optional": true,
|
|
@@ -11825,7 +12433,7 @@
|
|
|
11825
12433
|
"immutable": true,
|
|
11826
12434
|
"locationInModule": {
|
|
11827
12435
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11828
|
-
"line":
|
|
12436
|
+
"line": 193
|
|
11829
12437
|
},
|
|
11830
12438
|
"name": "dataFormat",
|
|
11831
12439
|
"optional": true,
|
|
@@ -11843,7 +12451,7 @@
|
|
|
11843
12451
|
"immutable": true,
|
|
11844
12452
|
"locationInModule": {
|
|
11845
12453
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11846
|
-
"line":
|
|
12454
|
+
"line": 190
|
|
11847
12455
|
},
|
|
11848
12456
|
"name": "identifierFieldNames",
|
|
11849
12457
|
"optional": true,
|
|
@@ -11866,7 +12474,7 @@
|
|
|
11866
12474
|
"immutable": true,
|
|
11867
12475
|
"locationInModule": {
|
|
11868
12476
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11869
|
-
"line":
|
|
12477
|
+
"line": 176
|
|
11870
12478
|
},
|
|
11871
12479
|
"name": "partitionSpec",
|
|
11872
12480
|
"optional": true,
|
|
@@ -11889,7 +12497,7 @@
|
|
|
11889
12497
|
"immutable": true,
|
|
11890
12498
|
"locationInModule": {
|
|
11891
12499
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11892
|
-
"line":
|
|
12500
|
+
"line": 223
|
|
11893
12501
|
},
|
|
11894
12502
|
"name": "removalPolicy",
|
|
11895
12503
|
"optional": true,
|
|
@@ -11897,6 +12505,25 @@
|
|
|
11897
12505
|
"fqn": "aws-cdk-lib.RemovalPolicy"
|
|
11898
12506
|
}
|
|
11899
12507
|
},
|
|
12508
|
+
{
|
|
12509
|
+
"abstract": true,
|
|
12510
|
+
"docs": {
|
|
12511
|
+
"default": "- no history check",
|
|
12512
|
+
"remarks": "Synth fails if the file is missing or malformed, or if the\ndefinition is an unsafe change from the table's record in it. See the README\nsection \"Checking changes against a schema history\".",
|
|
12513
|
+
"stability": "experimental",
|
|
12514
|
+
"summary": "Path of the committed schema history file of the table's database, relative to the working directory."
|
|
12515
|
+
},
|
|
12516
|
+
"immutable": true,
|
|
12517
|
+
"locationInModule": {
|
|
12518
|
+
"filename": "lib/iceberg/iceberg-table.ts",
|
|
12519
|
+
"line": 232
|
|
12520
|
+
},
|
|
12521
|
+
"name": "schemaHistoryFile",
|
|
12522
|
+
"optional": true,
|
|
12523
|
+
"type": {
|
|
12524
|
+
"primitive": "string"
|
|
12525
|
+
}
|
|
12526
|
+
},
|
|
11900
12527
|
{
|
|
11901
12528
|
"abstract": true,
|
|
11902
12529
|
"docs": {
|
|
@@ -11907,7 +12534,7 @@
|
|
|
11907
12534
|
"immutable": true,
|
|
11908
12535
|
"locationInModule": {
|
|
11909
12536
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11910
|
-
"line":
|
|
12537
|
+
"line": 179
|
|
11911
12538
|
},
|
|
11912
12539
|
"name": "sortOrder",
|
|
11913
12540
|
"optional": true,
|
|
@@ -11930,7 +12557,7 @@
|
|
|
11930
12557
|
"immutable": true,
|
|
11931
12558
|
"locationInModule": {
|
|
11932
12559
|
"filename": "lib/iceberg/iceberg-table.ts",
|
|
11933
|
-
"line":
|
|
12560
|
+
"line": 205
|
|
11934
12561
|
},
|
|
11935
12562
|
"name": "tableProperties",
|
|
11936
12563
|
"optional": true,
|
|
@@ -11946,6 +12573,67 @@
|
|
|
11946
12573
|
],
|
|
11947
12574
|
"symbolId": "lib/iceberg/iceberg-table:IcebergTableProps"
|
|
11948
12575
|
},
|
|
12576
|
+
"cdk-glue-iceberg-table.IcebergTableSchema": {
|
|
12577
|
+
"assembly": "cdk-glue-iceberg-table",
|
|
12578
|
+
"datatype": true,
|
|
12579
|
+
"docs": {
|
|
12580
|
+
"stability": "experimental",
|
|
12581
|
+
"summary": "The part of a table definition the schema history records."
|
|
12582
|
+
},
|
|
12583
|
+
"fqn": "cdk-glue-iceberg-table.IcebergTableSchema",
|
|
12584
|
+
"kind": "interface",
|
|
12585
|
+
"locationInModule": {
|
|
12586
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12587
|
+
"line": 80
|
|
12588
|
+
},
|
|
12589
|
+
"name": "IcebergTableSchema",
|
|
12590
|
+
"properties": [
|
|
12591
|
+
{
|
|
12592
|
+
"abstract": true,
|
|
12593
|
+
"docs": {
|
|
12594
|
+
"stability": "experimental",
|
|
12595
|
+
"summary": "Top-level columns."
|
|
12596
|
+
},
|
|
12597
|
+
"immutable": true,
|
|
12598
|
+
"locationInModule": {
|
|
12599
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12600
|
+
"line": 82
|
|
12601
|
+
},
|
|
12602
|
+
"name": "columns",
|
|
12603
|
+
"type": {
|
|
12604
|
+
"collection": {
|
|
12605
|
+
"elementtype": {
|
|
12606
|
+
"fqn": "cdk-glue-iceberg-table.IcebergColumn"
|
|
12607
|
+
},
|
|
12608
|
+
"kind": "array"
|
|
12609
|
+
}
|
|
12610
|
+
}
|
|
12611
|
+
},
|
|
12612
|
+
{
|
|
12613
|
+
"abstract": true,
|
|
12614
|
+
"docs": {
|
|
12615
|
+
"stability": "experimental",
|
|
12616
|
+
"summary": "Partition spec."
|
|
12617
|
+
},
|
|
12618
|
+
"immutable": true,
|
|
12619
|
+
"locationInModule": {
|
|
12620
|
+
"filename": "lib/iceberg/iceberg-schema-history.ts",
|
|
12621
|
+
"line": 85
|
|
12622
|
+
},
|
|
12623
|
+
"name": "partitionSpec",
|
|
12624
|
+
"optional": true,
|
|
12625
|
+
"type": {
|
|
12626
|
+
"collection": {
|
|
12627
|
+
"elementtype": {
|
|
12628
|
+
"fqn": "cdk-glue-iceberg-table.IcebergPartitionField"
|
|
12629
|
+
},
|
|
12630
|
+
"kind": "array"
|
|
12631
|
+
}
|
|
12632
|
+
}
|
|
12633
|
+
}
|
|
12634
|
+
],
|
|
12635
|
+
"symbolId": "lib/iceberg/iceberg-schema-history:IcebergTableSchema"
|
|
12636
|
+
},
|
|
11949
12637
|
"cdk-glue-iceberg-table.IcebergType": {
|
|
11950
12638
|
"assembly": "cdk-glue-iceberg-table",
|
|
11951
12639
|
"docs": {
|
|
@@ -12211,7 +12899,7 @@
|
|
|
12211
12899
|
},
|
|
12212
12900
|
"locationInModule": {
|
|
12213
12901
|
"filename": "lib/iceberg/iceberg-type.ts",
|
|
12214
|
-
"line":
|
|
12902
|
+
"line": 593
|
|
12215
12903
|
},
|
|
12216
12904
|
"name": "isDecimal",
|
|
12217
12905
|
"returns": {
|
|
@@ -12227,7 +12915,7 @@
|
|
|
12227
12915
|
},
|
|
12228
12916
|
"locationInModule": {
|
|
12229
12917
|
"filename": "lib/iceberg/iceberg-type.ts",
|
|
12230
|
-
"line":
|
|
12918
|
+
"line": 598
|
|
12231
12919
|
},
|
|
12232
12920
|
"name": "isFixed",
|
|
12233
12921
|
"returns": {
|
|
@@ -12243,7 +12931,7 @@
|
|
|
12243
12931
|
},
|
|
12244
12932
|
"locationInModule": {
|
|
12245
12933
|
"filename": "lib/iceberg/iceberg-type.ts",
|
|
12246
|
-
"line":
|
|
12934
|
+
"line": 588
|
|
12247
12935
|
},
|
|
12248
12936
|
"name": "isPrimitive",
|
|
12249
12937
|
"parameters": [
|
|
@@ -13232,6 +13920,6 @@
|
|
|
13232
13920
|
"symbolId": "lib/iceberg/iceberg-type:IcebergTypeOptions"
|
|
13233
13921
|
}
|
|
13234
13922
|
},
|
|
13235
|
-
"version": "0.
|
|
13236
|
-
"fingerprint": "
|
|
13923
|
+
"version": "0.16.0",
|
|
13924
|
+
"fingerprint": "/1m8HN6vz9krKS3rkQmTIHXLvV30It57KLUcgfslHTc="
|
|
13237
13925
|
}
|