cdk-glue-iceberg-table 0.1.5 → 0.1.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -18
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -76,6 +76,13 @@ This repo is **both** the published package and a CDK demo app:
|
|
|
76
76
|
|
|
77
77
|
- `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
|
|
78
78
|
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus a bash harness that drives schema + partition evolution through real `cdk deploy`s. **Repo-only, not published to npm.**
|
|
79
|
+
- `e2e-consumer/` — a standalone CDK app that depends on the **published** `cdk-glue-iceberg-table` from npm. Proves that a fresh install + import + `cdk synth` works for downstream consumers. Runs on every PR via the `e2e-consumer` job in `.github/workflows/ci.yml`. Its `lib/surface-reference.ts` touches every exported symbol so that a rename anywhere in the published surface breaks CI.
|
|
80
|
+
|
|
81
|
+
How the test gates fit together:
|
|
82
|
+
|
|
83
|
+
- **`ci.yml`** runs on every PR — lint, unit tests with 95% coverage gate, `npm pack`, and the `e2e-consumer` synth against the latest npm version.
|
|
84
|
+
- **`integ-test.yml`** is the real-AWS gate (`scripts/integration-test-evolution.sh` driving four `cdk deploy`s). Gated by `run-integ-test` label or `/run-integ-test` collaborator comment. PRs that touch any file under `lib/`, `bin/arceus.ts`, `cdk.json`, or the script must show a green run before merging (see CLAUDE.md §"Integration test for construct-touching PRs"). Doc-only PRs are exempt.
|
|
85
|
+
- **`publish.yml`** runs on push to `main` — trusted-publish to npm when `package.json`'s `version` is newer than the registry.
|
|
79
86
|
|
|
80
87
|
The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
|
|
81
88
|
[Demo tables](#demo-tables-deployed-by-arceusstack), and
|
|
@@ -92,11 +99,26 @@ Before running the quickstart you need:
|
|
|
92
99
|
2. `CDK_DEFAULT_ACCOUNT` and `CDK_DEFAULT_REGION` set in the
|
|
93
100
|
environment (the AWS CLI sets these automatically for most
|
|
94
101
|
profile setups; `cdk` also populates them from the active profile).
|
|
95
|
-
3. `
|
|
96
|
-
in this account
|
|
97
|
-
admin and grants
|
|
98
|
-
|
|
99
|
-
can't resolve
|
|
102
|
+
3. `PRINCIPAL_ARN` set to the **ARN of an existing IAM principal**
|
|
103
|
+
in this account — IAM user, role, or federated identity. The
|
|
104
|
+
stack adds that principal as a Lake Formation admin and grants
|
|
105
|
+
it per-table `SELECT/INSERT/DELETE/ALTER/DESCRIBE` on the demo
|
|
106
|
+
Iceberg tables. Without it the deploy fails when LF can't resolve
|
|
107
|
+
the principal. The same ARN must also be the identity running
|
|
108
|
+
`cdk deploy` and any subsequent Athena queries — local devs
|
|
109
|
+
typically set this to their IAM user ARN; CI (`integ-test.yml`)
|
|
110
|
+
sets it to the OIDC role ARN.
|
|
111
|
+
- **Only one principal is privileged at a time.** `ArceusStack`
|
|
112
|
+
writes the Lake Formation admin list with REPLACE semantics,
|
|
113
|
+
so the most recent `cdk deploy` wins. A local `cdk deploy` with
|
|
114
|
+
your user ARN revokes the OIDC role's LF grants (and breaks the
|
|
115
|
+
next integ-test run until CI redeploys), and vice versa.
|
|
116
|
+
Coordinate accordingly.
|
|
117
|
+
- **SSO / `aws-vault` users:** set `PRINCIPAL_ARN` to the
|
|
118
|
+
canonical role ARN (`arn:aws:iam::<acct>:role/<RoleName>`), not
|
|
119
|
+
the per-session `arn:aws:sts::<acct>:assumed-role/...` you'd
|
|
120
|
+
get from `aws sts get-caller-identity`. LF grants on the
|
|
121
|
+
session-suffixed ARN go stale at the next SSO refresh.
|
|
100
122
|
4. The Lake Formation service-linked role
|
|
101
123
|
`AWSServiceRoleForLakeFormationDataAccess` must exist in the
|
|
102
124
|
account. Create it once with
|
|
@@ -119,13 +141,23 @@ Repo source paths (npm consumers import from the package root and get the same e
|
|
|
119
141
|
## Quickstart
|
|
120
142
|
|
|
121
143
|
```bash
|
|
122
|
-
#
|
|
123
|
-
#
|
|
124
|
-
#
|
|
125
|
-
#
|
|
126
|
-
#
|
|
127
|
-
#
|
|
128
|
-
|
|
144
|
+
# PRINCIPAL_ARN is the ARN of the IAM principal (user, role, or
|
|
145
|
+
# federated identity) that the stack should make a Lake Formation
|
|
146
|
+
# admin and per-table grantee. It must equal the identity running
|
|
147
|
+
# `cdk deploy` and any subsequent Athena queries — otherwise the
|
|
148
|
+
# integration script's INSERT/SELECT calls fail with `Principal does
|
|
149
|
+
# not have any privilege on specified resource`.
|
|
150
|
+
#
|
|
151
|
+
# For a direct IAM-user session, the line below returns the right ARN
|
|
152
|
+
# (`arn:aws:iam::<acct>:user/<name>`).
|
|
153
|
+
#
|
|
154
|
+
# For SSO / aws-vault / any assumed-role session this returns
|
|
155
|
+
# `arn:aws:sts::<acct>:assumed-role/<RoleName>/<SessionName>`, which
|
|
156
|
+
# Lake Formation accepts but stales on the next session refresh
|
|
157
|
+
# because the SessionName changes. Set PRINCIPAL_ARN to the canonical
|
|
158
|
+
# `arn:aws:iam::<acct>:role/<RoleName>` instead — for example:
|
|
159
|
+
# export PRINCIPAL_ARN="arn:aws:iam::123456789012:role/MyDevRole"
|
|
160
|
+
export PRINCIPAL_ARN="$(aws sts get-caller-identity --query Arn --output text)"
|
|
129
161
|
|
|
130
162
|
npm install
|
|
131
163
|
npx jest # 100% line + branch coverage, 95% gate
|
|
@@ -150,7 +182,7 @@ import {
|
|
|
150
182
|
import {
|
|
151
183
|
IcebergTable,
|
|
152
184
|
IcebergType,
|
|
153
|
-
} from '
|
|
185
|
+
} from 'cdk-glue-iceberg-table';
|
|
154
186
|
|
|
155
187
|
const db = new Database(this, 'Db', {
|
|
156
188
|
databaseName: 'analytics',
|
|
@@ -204,7 +236,7 @@ import {
|
|
|
204
236
|
IcebergSortDirection,
|
|
205
237
|
IcebergTable,
|
|
206
238
|
IcebergType,
|
|
207
|
-
} from '
|
|
239
|
+
} from 'cdk-glue-iceberg-table';
|
|
208
240
|
|
|
209
241
|
new IcebergTable(this, 'OrdersTable', {
|
|
210
242
|
database: db,
|
|
@@ -592,7 +624,7 @@ arceus/
|
|
|
592
624
|
│ ├── iceberg-type.ts # IcebergType + struct/list/map/decimal/fixed
|
|
593
625
|
│ ├── iceberg-partition-transform.ts
|
|
594
626
|
│ ├── iceberg-table-properties.ts # Format/version enums + property validation
|
|
595
|
-
│ └── index.ts # Re-exports
|
|
627
|
+
│ └── index.ts # Re-exports (the npm package's entry point)
|
|
596
628
|
├── test/
|
|
597
629
|
│ ├── arceus-stack.test.ts
|
|
598
630
|
│ ├── iceberg-evolution-stack.test.ts
|
|
@@ -601,11 +633,29 @@ arceus/
|
|
|
601
633
|
│ ├── iceberg-table-properties.test.ts
|
|
602
634
|
│ ├── iceberg-table.test.ts
|
|
603
635
|
│ └── iceberg-type.test.ts
|
|
636
|
+
├── e2e-consumer/ # Standalone CDK app that consumes the
|
|
637
|
+
│ │ # published npm package — proves a fresh
|
|
638
|
+
│ │ # install + import + synth works. Runs
|
|
639
|
+
│ │ # on every PR via the `e2e-consumer` job
|
|
640
|
+
│ │ # in `.github/workflows/ci.yml`.
|
|
641
|
+
│ ├── bin/app.ts
|
|
642
|
+
│ ├── lib/consumer-stack.ts # Realistic consumer (one IcebergTable)
|
|
643
|
+
│ └── lib/surface-reference.ts # Anchors every exported symbol so a
|
|
644
|
+
│ # rename in the published surface breaks CI
|
|
604
645
|
├── scripts/
|
|
605
646
|
│ └── integration-test-evolution.sh # End-to-end evolution harness
|
|
647
|
+
├── docs/
|
|
648
|
+
│ └── integ-test-setup.md # AWS-side prerequisites for the integ-test
|
|
649
|
+
│ # workflow (OIDC provider, IAM role, repo var)
|
|
650
|
+
├── .github/workflows/
|
|
651
|
+
│ ├── ci.yml # Lint + test + build + pack + e2e-consumer
|
|
652
|
+
│ ├── publish.yml # Trusted-publish to npm on version bump
|
|
653
|
+
│ └── integ-test.yml # Real-AWS evolution test (gated by label /
|
|
654
|
+
│ # `/run-integ-test` comment)
|
|
606
655
|
├── cdk.json
|
|
607
|
-
├── package.json
|
|
656
|
+
├── package.json # The published package's manifest
|
|
657
|
+
├── tsconfig.json # Wide include (used by ESLint and dev)
|
|
658
|
+
├── tsconfig.build.json # Narrow include (used by `npm publish`)
|
|
608
659
|
├── jest.config.js
|
|
609
|
-
|
|
610
|
-
└── tsconfig.json
|
|
660
|
+
└── eslint.config.mjs
|
|
611
661
|
```
|