cdk-glue-iceberg-table 0.1.8 → 0.1.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +38 -8
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -75,13 +75,13 @@ Consumer-facing reference sections below:
|
|
|
75
75
|
This repo is **both** the published package and a CDK demo app:
|
|
76
76
|
|
|
77
77
|
- `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
|
|
78
|
-
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus
|
|
78
|
+
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `lib/iceberg-dml-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus two bash harnesses: one drives schema + partition evolution through real `cdk deploy`s, the other exercises the v2 DML surface (UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM) on a single deploy. **Repo-only, not published to npm.**
|
|
79
79
|
- `e2e-consumer/` — a standalone CDK app that depends on the **published** `cdk-glue-iceberg-table` from npm. Proves that a fresh install + import + `cdk synth` works for downstream consumers. Runs on every PR via the `e2e-consumer` job in `.github/workflows/ci.yml`. Its `lib/surface-reference.ts` touches every exported symbol so that a rename in the published surface breaks CI. The pin in `e2e-consumer/package-lock.json` tracks the version most recently published to npm; CLAUDE.md asks for it to be bumped after each release.
|
|
80
80
|
|
|
81
81
|
How the test gates fit together:
|
|
82
82
|
|
|
83
83
|
- **`ci.yml`** runs on every PR — lint, unit tests with the 95% coverage gate, `npm pack`, and the `e2e-consumer` synth against the pinned published npm version.
|
|
84
|
-
- **`integ-test.yml`** is the real-AWS gate
|
|
84
|
+
- **`integ-test.yml`** is the real-AWS gate. Runs `scripts/integration-test-evolution.sh` (four `cdk deploy`s exercising schema + partition evolution), then `scripts/integration-test-dml.sh` (one deploy, then UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM against a v2 merge-on-read table). Gated by `run-integ-test` label or `/run-integ-test` collaborator comment. PRs that touch any file under `lib/`, `bin/arceus.ts`, `cdk.json`, or either script must show a green run before merging (see CLAUDE.md §"Integration test for construct-touching PRs"). Doc-only PRs are exempt.
|
|
85
85
|
- **`publish.yml`** runs on push to `main` — trusted-publish to npm when `package.json`'s `version` is newer than the registry.
|
|
86
86
|
|
|
87
87
|
The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
|
|
@@ -137,6 +137,7 @@ Repo source paths (npm consumers import from the package root and get the same e
|
|
|
137
137
|
- **`lib/iceberg/iceberg-table-properties.ts`**: `IcebergDataFormat` (parquet/orc/avro, default parquet), `IcebergFormatVersion` (v1/v2, default v2), and a validator that catches misconfigured `tableProperties` before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
|
|
138
138
|
- **`lib/arceus-stack.ts`**: the demo stack: KMS-encrypted data lake bucket, Athena results bucket, Glue database, three demo Iceberg tables (`orders`, `events`, `customers`). Repo-only, not in the npm tarball.
|
|
139
139
|
- **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh`**: a parameterized stack and a bash harness that drives four real `cdk deploy`s to prove schema/partition evolution works end-to-end. Repo-only, not in the npm tarball.
|
|
140
|
+
- **`lib/iceberg-dml-stack.ts`** + **`scripts/integration-test-dml.sh`**: a v2 merge-on-read table and a bash harness that exercises UPDATE, DELETE, MERGE INTO (upsert), time-travel SELECT, OPTIMIZE compaction, and VACUUM snapshot expiration. Single deploy, ~3 minutes. Repo-only, not in the npm tarball.
|
|
140
141
|
|
|
141
142
|
## Quickstart
|
|
142
143
|
|
|
@@ -165,11 +166,13 @@ npx cdk deploy ArceusStack --require-approval=never
|
|
|
165
166
|
./scripts/integration-test-evolution.sh # add + rename + drop, via cdk only
|
|
166
167
|
```
|
|
167
168
|
|
|
168
|
-
`cdk ls` will show
|
|
169
|
-
three Iceberg tables)
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
169
|
+
`cdk ls` will show three stacks: `ArceusStack` (the demo data lake +
|
|
170
|
+
three Iceberg tables), `IcebergEvolutionStack` (the evolution test
|
|
171
|
+
target driven by `scripts/integration-test-evolution.sh`), and
|
|
172
|
+
`IcebergDmlStack` (the DML test target driven by
|
|
173
|
+
`scripts/integration-test-dml.sh`). Deploy only `ArceusStack` for
|
|
174
|
+
the quickstart; the two test stacks are created on demand by their
|
|
175
|
+
respective scripts.
|
|
173
176
|
|
|
174
177
|
## Using `IcebergTable`
|
|
175
178
|
|
|
@@ -539,6 +542,30 @@ metadata delta (new `schema-id`, new `spec-id`) and writes a new
|
|
|
539
542
|
the construct pins (`id: N` on each `IcebergColumn`) never change
|
|
540
543
|
across deploys.
|
|
541
544
|
|
|
545
|
+
## DML, time travel, OPTIMIZE, and VACUUM
|
|
546
|
+
|
|
547
|
+
`scripts/integration-test-dml.sh` covers the v2 surface that the
|
|
548
|
+
evolution test doesn't. The harness deploys `IcebergDmlStack` once,
|
|
549
|
+
then runs a sequence of Athena statements against a v2 merge-on-read
|
|
550
|
+
table with `identifierFieldNames: ['account_id']`:
|
|
551
|
+
|
|
552
|
+
| Step | Statement | Verify |
|
|
553
|
+
| ---: | --- | --- |
|
|
554
|
+
| 1 | `INSERT` 5 seed rows | row count == 5 |
|
|
555
|
+
| 2 | `UPDATE balance WHERE account_id = 2` | balance == 250, row count == 5 |
|
|
556
|
+
| 3 | `DELETE WHERE account_id = 4` | row count == 4, account_id=4 gone |
|
|
557
|
+
| 4 | capture pre-MERGE snapshot id from `dml_test$snapshots` | — |
|
|
558
|
+
| 5 | `MERGE INTO ... USING ... ON account_id` (update id=3, insert id=6 + id=7) | row count == 6, id=3 balance updated, ids 6 and 7 present |
|
|
559
|
+
| 6 | `SELECT ... FOR VERSION AS OF <pre-MERGE snapshot>` | time-travel still sees 4 rows, no id=6 |
|
|
560
|
+
| 7 | `OPTIMIZE ... REWRITE DATA USING BIN_PACK` | succeeds, row count unchanged |
|
|
561
|
+
| 8 | `VACUUM ...` (after a 65-s sleep to clear `max-snapshot-age-ms`) | succeeds, row count unchanged |
|
|
562
|
+
| 9 | final `SELECT` | 6 rows with the expected balances |
|
|
563
|
+
|
|
564
|
+
The DML table is configured with `history.expire.max-snapshot-age-ms`
|
|
565
|
+
of 60 seconds so VACUUM has snapshots to expire on a fresh table.
|
|
566
|
+
Both scripts run sequentially in the same `integ-test.yml` job; the
|
|
567
|
+
evolution test finishes in ~5 minutes and the DML test in ~3.
|
|
568
|
+
|
|
542
569
|
## Two footguns the construct prevents
|
|
543
570
|
|
|
544
571
|
### Footgun #1 — schema under `storageDescriptor.columns`
|
|
@@ -620,6 +647,7 @@ arceus/
|
|
|
620
647
|
├── lib/
|
|
621
648
|
│ ├── arceus-stack.ts # Demo stack (buckets, DB, 3 demo tables)
|
|
622
649
|
│ ├── iceberg-evolution-stack.ts # Parameterized stack for the evolution test
|
|
650
|
+
│ ├── iceberg-dml-stack.ts # Stack for the DML / time-travel / OPTIMIZE / VACUUM test
|
|
623
651
|
│ └── iceberg/
|
|
624
652
|
│ ├── iceberg-table.ts # The L2 construct itself
|
|
625
653
|
│ ├── iceberg-type.ts # IcebergType + struct/list/map/decimal/fixed
|
|
@@ -629,6 +657,7 @@ arceus/
|
|
|
629
657
|
├── test/
|
|
630
658
|
│ ├── arceus-stack.test.ts
|
|
631
659
|
│ ├── iceberg-evolution-stack.test.ts
|
|
660
|
+
│ ├── iceberg-dml-stack.test.ts
|
|
632
661
|
│ └── iceberg/
|
|
633
662
|
│ ├── iceberg-partition-transform.test.ts
|
|
634
663
|
│ ├── iceberg-table-properties.test.ts
|
|
@@ -644,7 +673,8 @@ arceus/
|
|
|
644
673
|
│ └── lib/surface-reference.ts # Anchors every exported symbol so a
|
|
645
674
|
│ # rename in the published surface breaks CI
|
|
646
675
|
├── scripts/
|
|
647
|
-
│
|
|
676
|
+
│ ├── integration-test-evolution.sh # End-to-end evolution harness
|
|
677
|
+
│ └── integration-test-dml.sh # End-to-end DML harness
|
|
648
678
|
├── docs/
|
|
649
679
|
│ └── integ-test-setup.md # AWS-side prerequisites for the integ-test
|
|
650
680
|
│ # workflow (OIDC provider, IAM role, repo var)
|