cdk-glue-iceberg-table 0.1.8 → 0.1.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +66 -8
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -75,13 +75,13 @@ Consumer-facing reference sections below:
|
|
|
75
75
|
This repo is **both** the published package and a CDK demo app:
|
|
76
76
|
|
|
77
77
|
- `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
|
|
78
|
-
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus
|
|
78
|
+
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `lib/iceberg-dml-stack.ts`, `lib/iceberg-surface-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus three bash harnesses: one drives schema + partition evolution through real `cdk deploy`s, one exercises the v2 DML surface (UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM), and one covers the remaining surface (every partition transform, multi-field sort order, list / struct / map columns, and `grantRead` at runtime by assuming the grantee role and calling Glue / S3 directly). **Repo-only, not published to npm.**
|
|
79
79
|
- `e2e-consumer/` — a standalone CDK app that depends on the **published** `cdk-glue-iceberg-table` from npm. Proves that a fresh install + import + `cdk synth` works for downstream consumers. Runs on every PR via the `e2e-consumer` job in `.github/workflows/ci.yml`. Its `lib/surface-reference.ts` touches every exported symbol so that a rename in the published surface breaks CI. The pin in `e2e-consumer/package-lock.json` tracks the version most recently published to npm; CLAUDE.md asks for it to be bumped after each release.
|
|
80
80
|
|
|
81
81
|
How the test gates fit together:
|
|
82
82
|
|
|
83
83
|
- **`ci.yml`** runs on every PR — lint, unit tests with the 95% coverage gate, `npm pack`, and the `e2e-consumer` synth against the pinned published npm version.
|
|
84
|
-
- **`integ-test.yml`** is the real-AWS gate
|
|
84
|
+
- **`integ-test.yml`** is the real-AWS gate. Runs three scripts back-to-back: `scripts/integration-test-evolution.sh` (four `cdk deploy`s exercising schema + partition evolution), `scripts/integration-test-dml.sh` (one deploy, then UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM against a v2 merge-on-read table), and `scripts/integration-test-surface.sh` (one deploy, then every partition transform, multi-field sort order, nested-type roundtrip, and `grantRead` at runtime via assume-role + direct Glue/S3 calls). Gated by `run-integ-test` label or `/run-integ-test` collaborator comment. PRs that touch any file under `lib/`, `bin/arceus.ts`, `cdk.json`, or any of the scripts must show a green run before merging (see CLAUDE.md §"Integration test for construct-touching PRs"). Doc-only PRs are exempt.
|
|
85
85
|
- **`publish.yml`** runs on push to `main` — trusted-publish to npm when `package.json`'s `version` is newer than the registry.
|
|
86
86
|
|
|
87
87
|
The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
|
|
@@ -137,6 +137,8 @@ Repo source paths (npm consumers import from the package root and get the same e
|
|
|
137
137
|
- **`lib/iceberg/iceberg-table-properties.ts`**: `IcebergDataFormat` (parquet/orc/avro, default parquet), `IcebergFormatVersion` (v1/v2, default v2), and a validator that catches misconfigured `tableProperties` before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
|
|
138
138
|
- **`lib/arceus-stack.ts`**: the demo stack: KMS-encrypted data lake bucket, Athena results bucket, Glue database, three demo Iceberg tables (`orders`, `events`, `customers`). Repo-only, not in the npm tarball.
|
|
139
139
|
- **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh`**: a parameterized stack and a bash harness that drives four real `cdk deploy`s to prove schema/partition evolution works end-to-end. Repo-only, not in the npm tarball.
|
|
140
|
+
- **`lib/iceberg-dml-stack.ts`** + **`scripts/integration-test-dml.sh`**: a v2 merge-on-read table and a bash harness that exercises UPDATE, DELETE, MERGE INTO (upsert), time-travel SELECT, OPTIMIZE compaction, and VACUUM snapshot expiration. Single deploy, ~3 minutes. Repo-only, not in the npm tarball.
|
|
141
|
+
- **`lib/iceberg-surface-stack.ts`** + **`scripts/integration-test-surface.sh`**: three tables (one per concern) and a harness that verifies every partition transform renders the right `metadata.json`, a three-field sort order with mixed direction + null ordering, list/struct/map columns roundtrip through Athena, and `grantRead`'s four-statement IAM split actually authorizes (and denies cross-prefix) at runtime under an assumed grantee role. The same checks run against `IcebergTable.fromIcebergTableAttributes(...)` to prove the import factory's grant path is symmetric with the native one. Single deploy, ~4 minutes. Repo-only, not in the npm tarball.
|
|
140
142
|
|
|
141
143
|
## Quickstart
|
|
142
144
|
|
|
@@ -165,11 +167,13 @@ npx cdk deploy ArceusStack --require-approval=never
|
|
|
165
167
|
./scripts/integration-test-evolution.sh # add + rename + drop, via cdk only
|
|
166
168
|
```
|
|
167
169
|
|
|
168
|
-
`cdk ls` will show
|
|
169
|
-
three Iceberg tables)
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
170
|
+
`cdk ls` will show four stacks: `ArceusStack` (the demo data lake +
|
|
171
|
+
three Iceberg tables), `IcebergEvolutionStack` (the evolution test
|
|
172
|
+
target), `IcebergDmlStack` (the DML test target), and
|
|
173
|
+
`IcebergSurfaceStack` (the transforms / sort / nested-types /
|
|
174
|
+
grants test target). Deploy only `ArceusStack` for the quickstart;
|
|
175
|
+
the three test stacks are created on demand by their respective
|
|
176
|
+
scripts under `scripts/`.
|
|
173
177
|
|
|
174
178
|
## Using `IcebergTable`
|
|
175
179
|
|
|
@@ -539,6 +543,54 @@ metadata delta (new `schema-id`, new `spec-id`) and writes a new
|
|
|
539
543
|
the construct pins (`id: N` on each `IcebergColumn`) never change
|
|
540
544
|
across deploys.
|
|
541
545
|
|
|
546
|
+
## DML, time travel, OPTIMIZE, and VACUUM
|
|
547
|
+
|
|
548
|
+
`scripts/integration-test-dml.sh` covers the v2 surface that the
|
|
549
|
+
evolution test doesn't. The harness deploys `IcebergDmlStack` once,
|
|
550
|
+
then runs a sequence of Athena statements against a v2 merge-on-read
|
|
551
|
+
table with `identifierFieldNames: ['account_id']`:
|
|
552
|
+
|
|
553
|
+
| Step | Statement | Verify |
|
|
554
|
+
| ---: | --- | --- |
|
|
555
|
+
| 1 | `INSERT` 5 seed rows | row count == 5 |
|
|
556
|
+
| 2 | `UPDATE balance WHERE account_id = 2` | balance == 250, row count == 5 |
|
|
557
|
+
| 3 | `DELETE WHERE account_id = 4` | row count == 4, account_id=4 gone |
|
|
558
|
+
| 4 | capture pre-MERGE snapshot id from `dml_test$snapshots` | — |
|
|
559
|
+
| 5 | `MERGE INTO ... USING ... ON account_id` (update id=3, insert id=6 + id=7) | row count == 6, id=3 balance updated, ids 6 and 7 present |
|
|
560
|
+
| 6 | `SELECT ... FOR VERSION AS OF <pre-MERGE snapshot>` | time-travel still sees 4 rows, no id=6 |
|
|
561
|
+
| 7 | `OPTIMIZE ... REWRITE DATA USING BIN_PACK` | succeeds, row count unchanged |
|
|
562
|
+
| 8 | `VACUUM ...` (after a 65-s sleep to clear `max-snapshot-age-ms`) | succeeds, row count unchanged |
|
|
563
|
+
| 9 | final `SELECT` | 6 rows with the expected balances |
|
|
564
|
+
|
|
565
|
+
The DML table is configured with `history.expire.max-snapshot-age-ms`
|
|
566
|
+
of 60 seconds so VACUUM has snapshots to expire on a fresh table.
|
|
567
|
+
All three scripts run sequentially in the same `integ-test.yml`
|
|
568
|
+
job; evolution takes ~5 minutes, DML ~3, and surface ~4.
|
|
569
|
+
|
|
570
|
+
## Partition transforms, sort order, nested types, and grants
|
|
571
|
+
|
|
572
|
+
`scripts/integration-test-surface.sh` covers the slice of the
|
|
573
|
+
construct surface the evolution and DML stacks don't reach.
|
|
574
|
+
`IcebergSurfaceStack` defines three small tables that each isolate
|
|
575
|
+
one concern:
|
|
576
|
+
|
|
577
|
+
| Concern | Table | What the script checks |
|
|
578
|
+
| --- | --- | --- |
|
|
579
|
+
| **Every partition transform** | `transforms_test` (separate `year_source` / `month_source` / `day_source` / `hour_source` timestamps to avoid Iceberg's redundant-temporal-transform rejection, plus `user_id` / `email` / `value`) | `metadata.json` contains all seven transforms (`year`, `month`, `day`, `hour`, `bucket[8]`, `truncate[4]`, `identity`); INSERT one row and verify the resulting S3 prefix contains the expected multi-transform layout (`year_source_year=...`, `month_source_month=...`, etc.) |
|
|
580
|
+
| **Sort order** | `sorted_test` (`tenant`, `created_at`, `amount`) | `metadata.json`'s `sort-orders` block has three fields with the expected direction + null-order pairs: `asc/nulls-first`, `desc/nulls-last`, `desc/nulls-last` |
|
|
581
|
+
| **Nested types** | `nested_test` (`tags` list, `profile` struct, `attrs` map) | INSERT two rows using `ARRAY[...]`, `CAST(ROW(...) AS ROW(...))`, and `MAP(ARRAY[...], ARRAY[...])`. SELECT verifies `tags[1]`, `profile.first_name`, and `element_at(attrs, 'tier')` all roundtrip correctly |
|
|
582
|
+
| **`grantRead` S3 statements at runtime** | `transforms_test` + a `GranteeRole` trusted by the deployer | Assume the role and call S3 directly (Lake Formation doesn't gate S3 calls when the bucket is registered with `hybridAccessEnabled: true`, so the IAM grants the construct produces are what's being tested). `s3:ListBucket` on the table's own prefix succeeds. `s3:ListBucket` on a foreign table's prefix is denied — the `s3:prefix` condition kicks in. The Glue action grants are validated by the unit tests, not at runtime, because LF gates `glue:*` against tables in LF-registered locations regardless of the principal's IAM policy |
|
|
583
|
+
| **`fromIcebergTableAttributes(...)` + `grantRead`** | imported handle on `transforms_test` + a second `ImportedGranteeRole` | Same S3 checks under the import-factory grantee. Verifies the import path produces a symmetric IAM split |
|
|
584
|
+
|
|
585
|
+
The construct's `grantRead` / `grantWrite` / `grantReadWrite` issue
|
|
586
|
+
**IAM grants only**. In a Lake-Formation-managed deployment like
|
|
587
|
+
`ArceusStack`, Athena queries still need separate LF `SELECT` /
|
|
588
|
+
`INSERT` / `DELETE` grants on top of the construct's IAM grants. The
|
|
589
|
+
surface test deliberately bypasses Athena (assumes the role and
|
|
590
|
+
calls Glue + S3 directly) so the construct's grant logic is
|
|
591
|
+
verified in isolation — under LF, the IAM grants alone are
|
|
592
|
+
necessary but not sufficient for Athena queries.
|
|
593
|
+
|
|
542
594
|
## Two footguns the construct prevents
|
|
543
595
|
|
|
544
596
|
### Footgun #1 — schema under `storageDescriptor.columns`
|
|
@@ -620,6 +672,8 @@ arceus/
|
|
|
620
672
|
├── lib/
|
|
621
673
|
│ ├── arceus-stack.ts # Demo stack (buckets, DB, 3 demo tables)
|
|
622
674
|
│ ├── iceberg-evolution-stack.ts # Parameterized stack for the evolution test
|
|
675
|
+
│ ├── iceberg-dml-stack.ts # Stack for the DML / time-travel / OPTIMIZE / VACUUM test
|
|
676
|
+
│ ├── iceberg-surface-stack.ts # Stack for transforms / sort / nested types / grants test
|
|
623
677
|
│ └── iceberg/
|
|
624
678
|
│ ├── iceberg-table.ts # The L2 construct itself
|
|
625
679
|
│ ├── iceberg-type.ts # IcebergType + struct/list/map/decimal/fixed
|
|
@@ -629,6 +683,8 @@ arceus/
|
|
|
629
683
|
├── test/
|
|
630
684
|
│ ├── arceus-stack.test.ts
|
|
631
685
|
│ ├── iceberg-evolution-stack.test.ts
|
|
686
|
+
│ ├── iceberg-dml-stack.test.ts
|
|
687
|
+
│ ├── iceberg-surface-stack.test.ts
|
|
632
688
|
│ └── iceberg/
|
|
633
689
|
│ ├── iceberg-partition-transform.test.ts
|
|
634
690
|
│ ├── iceberg-table-properties.test.ts
|
|
@@ -644,7 +700,9 @@ arceus/
|
|
|
644
700
|
│ └── lib/surface-reference.ts # Anchors every exported symbol so a
|
|
645
701
|
│ # rename in the published surface breaks CI
|
|
646
702
|
├── scripts/
|
|
647
|
-
│
|
|
703
|
+
│ ├── integration-test-evolution.sh # End-to-end evolution harness
|
|
704
|
+
│ ├── integration-test-dml.sh # End-to-end DML harness
|
|
705
|
+
│ └── integration-test-surface.sh # End-to-end surface harness
|
|
648
706
|
├── docs/
|
|
649
707
|
│ └── integ-test-setup.md # AWS-side prerequisites for the integ-test
|
|
650
708
|
│ # workflow (OIDC provider, IAM role, repo var)
|