cdk-glue-iceberg-table 0.1.7 → 0.1.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -26
- package/dist/lib/iceberg/iceberg-table-grants.d.ts +33 -0
- package/dist/lib/iceberg/iceberg-table-grants.js +112 -0
- package/dist/lib/iceberg/iceberg-table-render.d.ts +41 -0
- package/dist/lib/iceberg/iceberg-table-render.js +241 -0
- package/dist/lib/iceberg/iceberg-table.d.ts +3 -0
- package/dist/lib/iceberg/iceberg-table.js +52 -366
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -75,13 +75,13 @@ Consumer-facing reference sections below:
|
|
|
75
75
|
This repo is **both** the published package and a CDK demo app:
|
|
76
76
|
|
|
77
77
|
- `lib/iceberg/` — the published package (`cdk-glue-iceberg-table` on npm).
|
|
78
|
-
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus
|
|
79
|
-
- `e2e-consumer/` — a standalone CDK app that depends on the **published** `cdk-glue-iceberg-table` from npm. Proves that a fresh install + import + `cdk synth` works for downstream consumers. Runs on every PR via the `e2e-consumer` job in `.github/workflows/ci.yml`. Its `lib/surface-reference.ts` touches every exported symbol so that a rename
|
|
78
|
+
- `lib/arceus-stack.ts`, `lib/iceberg-evolution-stack.ts`, `lib/iceberg-dml-stack.ts`, `bin/`, `scripts/` — a CDK app that dogfoods the construct against a real AWS account, plus two bash harnesses: one drives schema + partition evolution through real `cdk deploy`s, the other exercises the v2 DML surface (UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM) on a single deploy. **Repo-only, not published to npm.**
|
|
79
|
+
- `e2e-consumer/` — a standalone CDK app that depends on the **published** `cdk-glue-iceberg-table` from npm. Proves that a fresh install + import + `cdk synth` works for downstream consumers. Runs on every PR via the `e2e-consumer` job in `.github/workflows/ci.yml`. Its `lib/surface-reference.ts` touches every exported symbol so that a rename in the published surface breaks CI. The pin in `e2e-consumer/package-lock.json` tracks the version most recently published to npm; CLAUDE.md asks for it to be bumped after each release.
|
|
80
80
|
|
|
81
81
|
How the test gates fit together:
|
|
82
82
|
|
|
83
|
-
- **`ci.yml`** runs on every PR — lint, unit tests with 95% coverage gate, `npm pack`, and the `e2e-consumer` synth against the
|
|
84
|
-
- **`integ-test.yml`** is the real-AWS gate
|
|
83
|
+
- **`ci.yml`** runs on every PR — lint, unit tests with the 95% coverage gate, `npm pack`, and the `e2e-consumer` synth against the pinned published npm version.
|
|
84
|
+
- **`integ-test.yml`** is the real-AWS gate. Runs `scripts/integration-test-evolution.sh` (four `cdk deploy`s exercising schema + partition evolution), then `scripts/integration-test-dml.sh` (one deploy, then UPDATE / DELETE / MERGE / time travel / OPTIMIZE / VACUUM against a v2 merge-on-read table). Gated by `run-integ-test` label or `/run-integ-test` collaborator comment. PRs that touch any file under `lib/`, `bin/arceus.ts`, `cdk.json`, or either script must show a green run before merging (see CLAUDE.md §"Integration test for construct-touching PRs"). Doc-only PRs are exempt.
|
|
85
85
|
- **`publish.yml`** runs on push to `main` — trusted-publish to npm when `package.json`'s `version` is newer than the registry.
|
|
86
86
|
|
|
87
87
|
The sections [Prerequisites](#prerequisites), [Quickstart](#quickstart),
|
|
@@ -100,12 +100,12 @@ Before running the quickstart you need:
|
|
|
100
100
|
environment (the AWS CLI sets these automatically for most
|
|
101
101
|
profile setups; `cdk` also populates them from the active profile).
|
|
102
102
|
3. `PRINCIPAL_ARN` set to the **ARN of an existing IAM principal**
|
|
103
|
-
in this account
|
|
103
|
+
in this account (IAM user, role, or federated identity). The
|
|
104
104
|
stack adds that principal as a Lake Formation admin and grants
|
|
105
105
|
it per-table `SELECT/INSERT/DELETE/ALTER/DESCRIBE` on the demo
|
|
106
106
|
Iceberg tables. Without it the deploy fails when LF can't resolve
|
|
107
107
|
the principal. The same ARN must also be the identity running
|
|
108
|
-
`cdk deploy` and any subsequent Athena queries
|
|
108
|
+
`cdk deploy` and any subsequent Athena queries. Local devs
|
|
109
109
|
typically set this to their IAM user ARN; CI (`integ-test.yml`)
|
|
110
110
|
sets it to the OIDC role ARN.
|
|
111
111
|
- **Only one principal is privileged at a time.** `ArceusStack`
|
|
@@ -131,12 +131,13 @@ Before running the quickstart you need:
|
|
|
131
131
|
|
|
132
132
|
Repo source paths (npm consumers import from the package root and get the same exports under `dist/lib/iceberg/`):
|
|
133
133
|
|
|
134
|
-
- **`lib/iceberg/iceberg-table.ts
|
|
135
|
-
- **`lib/iceberg/iceberg-type.ts
|
|
136
|
-
- **`lib/iceberg/iceberg-partition-transform.ts
|
|
137
|
-
- **`lib/iceberg/iceberg-table-properties.ts
|
|
138
|
-
- **`lib/arceus-stack.ts
|
|
139
|
-
- **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh
|
|
134
|
+
- **`lib/iceberg/iceberg-table.ts`**: the `IcebergTable` L2 construct.
|
|
135
|
+
- **`lib/iceberg/iceberg-type.ts`**: `IcebergType` with primitives + `list` / `map` / `struct` factories. Renders to the JSON shape Glue's `IcebergStructField.type` expects.
|
|
136
|
+
- **`lib/iceberg/iceberg-partition-transform.ts`**: `IcebergPartitionTransform` (identity / bucket(N) / truncate(W) / year / month / day / hour / void). Each transform validates against the source column type at synth time.
|
|
137
|
+
- **`lib/iceberg/iceberg-table-properties.ts`**: `IcebergDataFormat` (parquet/orc/avro, default parquet), `IcebergFormatVersion` (v1/v2, default v2), and a validator that catches misconfigured `tableProperties` before they leave your machine (wrong codec for the chosen format, `merge-on-read` on a v1 table, non-positive numeric values, …).
|
|
138
|
+
- **`lib/arceus-stack.ts`**: the demo stack: KMS-encrypted data lake bucket, Athena results bucket, Glue database, three demo Iceberg tables (`orders`, `events`, `customers`). Repo-only, not in the npm tarball.
|
|
139
|
+
- **`lib/iceberg-evolution-stack.ts`** + **`scripts/integration-test-evolution.sh`**: a parameterized stack and a bash harness that drives four real `cdk deploy`s to prove schema/partition evolution works end-to-end. Repo-only, not in the npm tarball.
|
|
140
|
+
- **`lib/iceberg-dml-stack.ts`** + **`scripts/integration-test-dml.sh`**: a v2 merge-on-read table and a bash harness that exercises UPDATE, DELETE, MERGE INTO (upsert), time-travel SELECT, OPTIMIZE compaction, and VACUUM snapshot expiration. Single deploy, ~3 minutes. Repo-only, not in the npm tarball.
|
|
140
141
|
|
|
141
142
|
## Quickstart
|
|
142
143
|
|
|
@@ -160,16 +161,18 @@ Repo source paths (npm consumers import from the package root and get the same e
|
|
|
160
161
|
export PRINCIPAL_ARN="$(aws sts get-caller-identity --query Arn --output text)"
|
|
161
162
|
|
|
162
163
|
npm install
|
|
163
|
-
npx jest #
|
|
164
|
+
npx jest # runs the suite; coverage floor in jest.config.js
|
|
164
165
|
npx cdk deploy ArceusStack --require-approval=never
|
|
165
166
|
./scripts/integration-test-evolution.sh # add + rename + drop, via cdk only
|
|
166
167
|
```
|
|
167
168
|
|
|
168
|
-
`cdk ls` will show
|
|
169
|
-
three Iceberg tables)
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
169
|
+
`cdk ls` will show three stacks: `ArceusStack` (the demo data lake +
|
|
170
|
+
three Iceberg tables), `IcebergEvolutionStack` (the evolution test
|
|
171
|
+
target driven by `scripts/integration-test-evolution.sh`), and
|
|
172
|
+
`IcebergDmlStack` (the DML test target driven by
|
|
173
|
+
`scripts/integration-test-dml.sh`). Deploy only `ArceusStack` for
|
|
174
|
+
the quickstart; the two test stacks are created on demand by their
|
|
175
|
+
respective scripts.
|
|
173
176
|
|
|
174
177
|
## Using `IcebergTable`
|
|
175
178
|
|
|
@@ -215,8 +218,8 @@ new IcebergTable(this, 'Users', {
|
|
|
215
218
|
});
|
|
216
219
|
```
|
|
217
220
|
|
|
218
|
-
A table that exercises most of the surface
|
|
219
|
-
nested types, identifier fields, table properties, removal policy.
|
|
221
|
+
A table that exercises most of the surface (partitions, sort order,
|
|
222
|
+
nested types, identifier fields, table properties, removal policy).
|
|
220
223
|
This is the exact shape `ArceusStack` uses for the `orders` demo
|
|
221
224
|
table, so the column list / partition spec / properties round-trip
|
|
222
225
|
straight to the live metadata.json below.
|
|
@@ -488,7 +491,7 @@ underlying Iceberg `metadata.json` after each:
|
|
|
488
491
|
| 3 | **RENAME** `email` → `contact_email` (id 2 preserved), **ADD** partition `bucket(8)(customer_id)` | rename | + `bucket(8)(customer_id)` |
|
|
489
492
|
| 4 | **DROP** column `region` (id 4 stays retired), **DROP** partition `bucket(8)(customer_id)` | − `region` | − `customer_id_bucket` |
|
|
490
493
|
|
|
491
|
-
Last script run output (abridged
|
|
494
|
+
Last script run output (abridged: `cdk deploy` chatter and the
|
|
492
495
|
per-Athena-query state polling lines are omitted; the assertion
|
|
493
496
|
output is verbatim):
|
|
494
497
|
|
|
@@ -539,6 +542,30 @@ metadata delta (new `schema-id`, new `spec-id`) and writes a new
|
|
|
539
542
|
the construct pins (`id: N` on each `IcebergColumn`) never change
|
|
540
543
|
across deploys.
|
|
541
544
|
|
|
545
|
+
## DML, time travel, OPTIMIZE, and VACUUM
|
|
546
|
+
|
|
547
|
+
`scripts/integration-test-dml.sh` covers the v2 surface that the
|
|
548
|
+
evolution test doesn't. The harness deploys `IcebergDmlStack` once,
|
|
549
|
+
then runs a sequence of Athena statements against a v2 merge-on-read
|
|
550
|
+
table with `identifierFieldNames: ['account_id']`:
|
|
551
|
+
|
|
552
|
+
| Step | Statement | Verify |
|
|
553
|
+
| ---: | --- | --- |
|
|
554
|
+
| 1 | `INSERT` 5 seed rows | row count == 5 |
|
|
555
|
+
| 2 | `UPDATE balance WHERE account_id = 2` | balance == 250, row count == 5 |
|
|
556
|
+
| 3 | `DELETE WHERE account_id = 4` | row count == 4, account_id=4 gone |
|
|
557
|
+
| 4 | capture pre-MERGE snapshot id from `dml_test$snapshots` | — |
|
|
558
|
+
| 5 | `MERGE INTO ... USING ... ON account_id` (update id=3, insert id=6 + id=7) | row count == 6, id=3 balance updated, ids 6 and 7 present |
|
|
559
|
+
| 6 | `SELECT ... FOR VERSION AS OF <pre-MERGE snapshot>` | time-travel still sees 4 rows, no id=6 |
|
|
560
|
+
| 7 | `OPTIMIZE ... REWRITE DATA USING BIN_PACK` | succeeds, row count unchanged |
|
|
561
|
+
| 8 | `VACUUM ...` (after a 65-s sleep to clear `max-snapshot-age-ms`) | succeeds, row count unchanged |
|
|
562
|
+
| 9 | final `SELECT` | 6 rows with the expected balances |
|
|
563
|
+
|
|
564
|
+
The DML table is configured with `history.expire.max-snapshot-age-ms`
|
|
565
|
+
of 60 seconds so VACUUM has snapshots to expire on a fresh table.
|
|
566
|
+
Both scripts run sequentially in the same `integ-test.yml` job; the
|
|
567
|
+
evolution test finishes in ~5 minutes and the DML test in ~3.
|
|
568
|
+
|
|
542
569
|
## Two footguns the construct prevents
|
|
543
570
|
|
|
544
571
|
### Footgun #1 — schema under `storageDescriptor.columns`
|
|
@@ -583,16 +610,17 @@ under `storageDescriptor`.
|
|
|
583
610
|
Even setting just `tableInput: { name: 'foo' }` next to
|
|
584
611
|
`openTableFormatInput` returns
|
|
585
612
|
`"Table metadata is expected only via TableInput or via IcebergTableInputProperties inside OpenTableFormatInput"`.
|
|
586
|
-
The construct never emits `tableInput
|
|
613
|
+
The construct never emits `tableInput`; the table-level comment goes
|
|
587
614
|
into `tableProperties['comment']`, which lives inside
|
|
588
615
|
`icebergTableInput.properties`.
|
|
589
616
|
|
|
590
|
-
(There is a third footgun
|
|
617
|
+
(There is a third footgun, field-id reuse after a column drop, that
|
|
591
618
|
the construct does **not** prevent. See the next section.)
|
|
592
619
|
|
|
593
620
|
## Known limitations
|
|
594
621
|
|
|
595
622
|
- **Field-id reuse is not detected across deploys.** If you drop a column with `id = 5` and then add a different column with `id = 5` in a later deploy, Glue accepts the UPDATE and Iceberg's metadata silently violates the "never reuse a retired id" invariant. Readers projecting old snapshots will surface deleted data under the new field's name. The construct enforces uniqueness **within one deploy** (`duplicate column id N` validator), but it doesn't compare against the live table state. The safe workflow is to always pin `id` explicitly and treat dropped ids as retired forever; never let CDK reassign an id that has ever been used.
|
|
623
|
+
- **Partition field ids are positional and not pinnable.** The construct allocates partition `fieldId` densely from 1000 in the order partitions appear in `partitionSpec`. Reordering the array across deploys reassigns those ids for unchanged logical partitions, which is the partition-spec analog of the column-id-reuse footgun above. There is no `IcebergPartitionField.fieldId` pinning prop today. The safe workflow is append-only: add new partition fields at the end of `partitionSpec`, and only drop the trailing ones.
|
|
596
624
|
- **CREATE-only metadata operation.** The CFN `IcebergInput.metadataOperation` only accepts `CREATE`; the construct always emits that. Subsequent deploys use Glue's normal `UpdateTable` path, which writes new Iceberg metadata in-place.
|
|
597
625
|
- **Format version is immutable after CREATE.** The `formatVersion` prop is read once at table creation; changing it later requires a destroy + recreate.
|
|
598
626
|
- **`merge-on-read` requires v2.** The construct rejects `write.{delete,update,merge}.mode = merge-on-read` on a v1 table at synth time.
|
|
@@ -608,7 +636,7 @@ npx jest # runs every suite under test/, prints coverage at the end
|
|
|
608
636
|
Coverage is gated at 95% statements / 95% branches / 95% lines / 95% functions
|
|
609
637
|
on `lib/**/*.ts` via the `coverageThreshold.global` block in `jest.config.js`.
|
|
610
638
|
A failing gate fails the suite, so the README does not paste a transcript that
|
|
611
|
-
would drift after the next refactor
|
|
639
|
+
would drift after the next refactor; run the command locally for the live
|
|
612
640
|
numbers.
|
|
613
641
|
|
|
614
642
|
## Project layout
|
|
@@ -619,6 +647,7 @@ arceus/
|
|
|
619
647
|
├── lib/
|
|
620
648
|
│ ├── arceus-stack.ts # Demo stack (buckets, DB, 3 demo tables)
|
|
621
649
|
│ ├── iceberg-evolution-stack.ts # Parameterized stack for the evolution test
|
|
650
|
+
│ ├── iceberg-dml-stack.ts # Stack for the DML / time-travel / OPTIMIZE / VACUUM test
|
|
622
651
|
│ └── iceberg/
|
|
623
652
|
│ ├── iceberg-table.ts # The L2 construct itself
|
|
624
653
|
│ ├── iceberg-type.ts # IcebergType + struct/list/map/decimal/fixed
|
|
@@ -628,6 +657,7 @@ arceus/
|
|
|
628
657
|
├── test/
|
|
629
658
|
│ ├── arceus-stack.test.ts
|
|
630
659
|
│ ├── iceberg-evolution-stack.test.ts
|
|
660
|
+
│ ├── iceberg-dml-stack.test.ts
|
|
631
661
|
│ └── iceberg/
|
|
632
662
|
│ ├── iceberg-partition-transform.test.ts
|
|
633
663
|
│ ├── iceberg-table-properties.test.ts
|
|
@@ -643,7 +673,8 @@ arceus/
|
|
|
643
673
|
│ └── lib/surface-reference.ts # Anchors every exported symbol so a
|
|
644
674
|
│ # rename in the published surface breaks CI
|
|
645
675
|
├── scripts/
|
|
646
|
-
│
|
|
676
|
+
│ ├── integration-test-evolution.sh # End-to-end evolution harness
|
|
677
|
+
│ └── integration-test-dml.sh # End-to-end DML harness
|
|
647
678
|
├── docs/
|
|
648
679
|
│ └── integ-test-setup.md # AWS-side prerequisites for the integ-test
|
|
649
680
|
│ # workflow (OIDC provider, IAM role, repo var)
|
|
@@ -657,5 +688,5 @@ arceus/
|
|
|
657
688
|
├── tsconfig.json # Wide include (used by ESLint and dev)
|
|
658
689
|
├── tsconfig.build.json # Narrow include (used by `npm publish`)
|
|
659
690
|
├── jest.config.js
|
|
660
|
-
└── eslint.config.
|
|
691
|
+
└── eslint.config.js
|
|
661
692
|
```
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
import { Grant, IGrantable } from 'aws-cdk-lib/aws-iam';
|
|
2
|
+
export declare const READ_TABLE_ACTIONS: string[];
|
|
3
|
+
export declare const WRITE_TABLE_ACTIONS: string[];
|
|
4
|
+
export declare const READ_S3_LIST_ACTIONS: string[];
|
|
5
|
+
export declare const READ_S3_BUCKET_ACTIONS: string[];
|
|
6
|
+
export declare const WRITE_S3_BUCKET_ACTIONS: string[];
|
|
7
|
+
export declare const READ_S3_OBJECT_ACTIONS: string[];
|
|
8
|
+
export declare const WRITE_S3_OBJECT_ACTIONS: string[];
|
|
9
|
+
/**
|
|
10
|
+
* Issue the four policy statements that scope an Iceberg-table grant
|
|
11
|
+
* correctly: Glue actions on the table ARN, S3 list-bucket actions
|
|
12
|
+
* on the bucket ARN with an `s3:prefix` condition so only the
|
|
13
|
+
* table's own prefix can be listed, S3 bucket-level actions that DO
|
|
14
|
+
* NOT support the `s3:prefix` condition (e.g. `GetBucketLocation`,
|
|
15
|
+
* `ListBucketMultipartUploads`) on the bucket ARN with no condition
|
|
16
|
+
* — including them in the conditioned statement would silently deny
|
|
17
|
+
* them at runtime — and S3 object-level actions on the
|
|
18
|
+
* `bucket/prefix*` ARN. Returns the table-actions grant (any of the
|
|
19
|
+
* four is sufficient for the `Grant` API contract; the rest attach
|
|
20
|
+
* as side effects).
|
|
21
|
+
*
|
|
22
|
+
* @internal
|
|
23
|
+
*/
|
|
24
|
+
export declare function grantSplit(grantee: IGrantable, args: {
|
|
25
|
+
tableArn: string;
|
|
26
|
+
bucketArn: string;
|
|
27
|
+
objectArn: string;
|
|
28
|
+
prefixGlob: string;
|
|
29
|
+
tableActions: string[];
|
|
30
|
+
listActions: string[];
|
|
31
|
+
bucketActions: string[];
|
|
32
|
+
objectActions: string[];
|
|
33
|
+
}): Grant;
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.WRITE_S3_OBJECT_ACTIONS = exports.READ_S3_OBJECT_ACTIONS = exports.WRITE_S3_BUCKET_ACTIONS = exports.READ_S3_BUCKET_ACTIONS = exports.READ_S3_LIST_ACTIONS = exports.WRITE_TABLE_ACTIONS = exports.READ_TABLE_ACTIONS = void 0;
|
|
4
|
+
exports.grantSplit = grantSplit;
|
|
5
|
+
const aws_iam_1 = require("aws-cdk-lib/aws-iam");
|
|
6
|
+
/// /////////////////////////////////////////////////
|
|
7
|
+
/// /////////////////////////////////////////////////
|
|
8
|
+
// Grant action sets
|
|
9
|
+
exports.READ_TABLE_ACTIONS = [
|
|
10
|
+
'glue:BatchGetPartition',
|
|
11
|
+
'glue:GetPartition',
|
|
12
|
+
'glue:GetPartitions',
|
|
13
|
+
'glue:GetTable',
|
|
14
|
+
'glue:GetTables',
|
|
15
|
+
'glue:GetTableVersion',
|
|
16
|
+
'glue:GetTableVersions',
|
|
17
|
+
];
|
|
18
|
+
exports.WRITE_TABLE_ACTIONS = [
|
|
19
|
+
'glue:BatchCreatePartition',
|
|
20
|
+
'glue:BatchDeletePartition',
|
|
21
|
+
'glue:CreatePartition',
|
|
22
|
+
'glue:DeletePartition',
|
|
23
|
+
'glue:UpdatePartition',
|
|
24
|
+
'glue:UpdateTable',
|
|
25
|
+
];
|
|
26
|
+
/// S3 actions that operate at the bucket level AND support the
|
|
27
|
+
/// `s3:prefix` request condition key (per the S3 docs). Granted on
|
|
28
|
+
/// the bucket ARN with a `StringLike s3:prefix = [<prefix>*, <prefix>]`
|
|
29
|
+
/// condition so the grantee can only list the table's own prefix.
|
|
30
|
+
exports.READ_S3_LIST_ACTIONS = [
|
|
31
|
+
's3:ListBucket',
|
|
32
|
+
];
|
|
33
|
+
/// S3 actions that operate at the bucket level but DO NOT support
|
|
34
|
+
/// `s3:prefix`. Granted on the bucket ARN with no condition — adding
|
|
35
|
+
/// one would silently deny these actions at runtime even though they
|
|
36
|
+
/// appear in the policy document.
|
|
37
|
+
exports.READ_S3_BUCKET_ACTIONS = [
|
|
38
|
+
's3:GetBucketLocation',
|
|
39
|
+
];
|
|
40
|
+
exports.WRITE_S3_BUCKET_ACTIONS = [
|
|
41
|
+
's3:ListBucketMultipartUploads',
|
|
42
|
+
];
|
|
43
|
+
/// S3 actions that operate at the object level. Must be granted on
|
|
44
|
+
/// the `bucket/prefix*` ARN.
|
|
45
|
+
exports.READ_S3_OBJECT_ACTIONS = [
|
|
46
|
+
's3:GetObject',
|
|
47
|
+
];
|
|
48
|
+
exports.WRITE_S3_OBJECT_ACTIONS = [
|
|
49
|
+
's3:PutObject',
|
|
50
|
+
's3:DeleteObject',
|
|
51
|
+
's3:AbortMultipartUpload',
|
|
52
|
+
's3:ListMultipartUploadParts',
|
|
53
|
+
];
|
|
54
|
+
/**
|
|
55
|
+
* Issue the four policy statements that scope an Iceberg-table grant
|
|
56
|
+
* correctly: Glue actions on the table ARN, S3 list-bucket actions
|
|
57
|
+
* on the bucket ARN with an `s3:prefix` condition so only the
|
|
58
|
+
* table's own prefix can be listed, S3 bucket-level actions that DO
|
|
59
|
+
* NOT support the `s3:prefix` condition (e.g. `GetBucketLocation`,
|
|
60
|
+
* `ListBucketMultipartUploads`) on the bucket ARN with no condition
|
|
61
|
+
* — including them in the conditioned statement would silently deny
|
|
62
|
+
* them at runtime — and S3 object-level actions on the
|
|
63
|
+
* `bucket/prefix*` ARN. Returns the table-actions grant (any of the
|
|
64
|
+
* four is sufficient for the `Grant` API contract; the rest attach
|
|
65
|
+
* as side effects).
|
|
66
|
+
*
|
|
67
|
+
* @internal
|
|
68
|
+
*/
|
|
69
|
+
function grantSplit(grantee, args) {
|
|
70
|
+
const tableGrant = aws_iam_1.Grant.addToPrincipal({
|
|
71
|
+
grantee,
|
|
72
|
+
actions: args.tableActions,
|
|
73
|
+
resourceArns: [
|
|
74
|
+
args.tableArn,
|
|
75
|
+
],
|
|
76
|
+
});
|
|
77
|
+
if (args.listActions.length > 0) {
|
|
78
|
+
aws_iam_1.Grant.addToPrincipal({
|
|
79
|
+
grantee,
|
|
80
|
+
actions: args.listActions,
|
|
81
|
+
resourceArns: [
|
|
82
|
+
args.bucketArn,
|
|
83
|
+
],
|
|
84
|
+
conditions: {
|
|
85
|
+
StringLike: {
|
|
86
|
+
's3:prefix': [
|
|
87
|
+
args.prefixGlob,
|
|
88
|
+
args.prefixGlob.replace(/\*$/, ''),
|
|
89
|
+
],
|
|
90
|
+
},
|
|
91
|
+
},
|
|
92
|
+
});
|
|
93
|
+
}
|
|
94
|
+
if (args.bucketActions.length > 0) {
|
|
95
|
+
aws_iam_1.Grant.addToPrincipal({
|
|
96
|
+
grantee,
|
|
97
|
+
actions: args.bucketActions,
|
|
98
|
+
resourceArns: [
|
|
99
|
+
args.bucketArn,
|
|
100
|
+
],
|
|
101
|
+
});
|
|
102
|
+
}
|
|
103
|
+
aws_iam_1.Grant.addToPrincipal({
|
|
104
|
+
grantee,
|
|
105
|
+
actions: args.objectActions,
|
|
106
|
+
resourceArns: [
|
|
107
|
+
args.objectArn,
|
|
108
|
+
],
|
|
109
|
+
});
|
|
110
|
+
return tableGrant;
|
|
111
|
+
}
|
|
112
|
+
//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoiaWNlYmVyZy10YWJsZS1ncmFudHMuanMiLCJzb3VyY2VSb290IjoiIiwic291cmNlcyI6WyIuLi8uLi8uLi9saWIvaWNlYmVyZy9pY2ViZXJnLXRhYmxlLWdyYW50cy50cyJdLCJuYW1lcyI6W10sIm1hcHBpbmdzIjoiOzs7QUE0RUEsZ0NBc0RDO0FBbElELGlEQUc2QjtBQUU3QixxREFBcUQ7QUFDckQscURBQXFEO0FBQ3JELG9CQUFvQjtBQUVQLFFBQUEsa0JBQWtCLEdBQUc7SUFDOUIsd0JBQXdCO0lBQ3hCLG1CQUFtQjtJQUNuQixvQkFBb0I7SUFDcEIsZUFBZTtJQUNmLGdCQUFnQjtJQUNoQixzQkFBc0I7SUFDdEIsdUJBQXVCO0NBQzFCLENBQUM7QUFFVyxRQUFBLG1CQUFtQixHQUFHO0lBQy9CLDJCQUEyQjtJQUMzQiwyQkFBMkI7SUFDM0Isc0JBQXNCO0lBQ3RCLHNCQUFzQjtJQUN0QixzQkFBc0I7SUFDdEIsa0JBQWtCO0NBQ3JCLENBQUM7QUFFRiwrREFBK0Q7QUFDL0QsbUVBQW1FO0FBQ25FLHdFQUF3RTtBQUN4RSxrRUFBa0U7QUFDckQsUUFBQSxvQkFBb0IsR0FBRztJQUNoQyxlQUFlO0NBQ2xCLENBQUM7QUFFRixrRUFBa0U7QUFDbEUscUVBQXFFO0FBQ3JFLHFFQUFxRTtBQUNyRSxrQ0FBa0M7QUFDckIsUUFBQSxzQkFBc0IsR0FBRztJQUNsQyxzQkFBc0I7Q0FDekIsQ0FBQztBQUVXLFFBQUEsdUJBQXVCLEdBQUc7SUFDbkMsK0JBQStCO0NBQ2xDLENBQUM7QUFFRixtRUFBbUU7QUFDbkUsNkJBQTZCO0FBQ2hCLFFBQUEsc0JBQXNCLEdBQUc7SUFDbEMsY0FBYztDQUNqQixDQUFDO0FBRVcsUUFBQSx1QkFBdUIsR0FBRztJQUNuQyxjQUFjO0lBQ2QsaUJBQWlCO0lBQ2pCLHlCQUF5QjtJQUN6Qiw2QkFBNkI7Q0FDaEMsQ0FBQztBQUVGOzs7Ozs7Ozs7Ozs7OztHQWNHO0FBQ0gsU0FBZ0IsVUFBVSxDQUN0QixPQUFtQixFQUNuQixJQVNDO0lBRUQsTUFBTSxVQUFVLEdBQUcsZUFBSyxDQUFDLGNBQWMsQ0FBQztRQUNwQyxPQUFPO1FBQ1AsT0FBTyxFQUFFLElBQUksQ0FBQyxZQUFZO1FBQzFCLFlBQVksRUFBRTtZQUNWLElBQUksQ0FBQyxRQUFRO1NBQ2hCO0tBQ0osQ0FBQyxDQUFDO0lBQ0gsSUFBSSxJQUFJLENBQUMsV0FBVyxDQUFDLE1BQU0sR0FBRyxDQUFDLEVBQUUsQ0FBQztRQUM5QixlQUFLLENBQUMsY0FBYyxDQUFDO1lBQ2pCLE9BQU87WUFDUCxPQUFPLEVBQUUsSUFBSSxDQUFDLFdBQVc7WUFDekIsWUFBWSxFQUFFO2dCQUNWLElBQUksQ0FBQyxTQUFTO2FBQ2pCO1lBQ0QsVUFBVSxFQUFFO2dCQUNSLFVBQVUsRUFBRTtvQkFDUixXQUFXLEVBQUU7d0JBQ1QsSUFBSSxDQUFDLFVBQVU7d0JBQ2YsSUFBSSxDQUFDLFVBQVUsQ0FBQyxPQUFPLENBQUMsS0FBSyxFQUFFLEVBQUUsQ0FBQztxQkFDckM7aUJBQ0o7YUFDSjtTQUNKLENBQUMsQ0FBQztJQUNQLENBQUM7SUFDRCxJQUFJLElBQUksQ0FBQyxhQUFhLENBQUMsTUFBTSxHQUFHLENBQUMsRUFBRSxDQUFDO1FBQ2hDLGVBQUssQ0FBQyxjQUFjLENBQUM7WUFDakIsT0FBTztZQUNQLE9BQU8sRUFBRSxJQUFJLENBQUMsYUFBYTtZQUMzQixZQUFZLEVBQUU7Z0JBQ1YsSUFBSSxDQUFDLFNBQVM7YUFDakI7U0FDSixDQUFDLENBQUM7SUFDUCxDQUFDO0lBQ0QsZUFBSyxDQUFDLGNBQWMsQ0FBQztRQUNqQixPQUFPO1FBQ1AsT0FBTyxFQUFFLElBQUksQ0FBQyxhQUFhO1FBQzNCLFlBQVksRUFBRTtZQUNWLElBQUksQ0FBQyxTQUFTO1NBQ2pCO0tBQ0osQ0FBQyxDQUFDO0lBQ0gsT0FBTyxVQUFVLENBQUM7QUFDdEIsQ0FBQyIsInNvdXJjZXNDb250ZW50IjpbImltcG9ydCB7XG4gICAgR3JhbnQsXG4gICAgSUdyYW50YWJsZSxcbn0gZnJvbSAnYXdzLWNkay1saWIvYXdzLWlhbSc7XG5cbi8vLyAvLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vXG4vLy8gLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vL1xuLy8gR3JhbnQgYWN0aW9uIHNldHNcblxuZXhwb3J0IGNvbnN0IFJFQURfVEFCTEVfQUNUSU9OUyA9IFtcbiAgICAnZ2x1ZTpCYXRjaEdldFBhcnRpdGlvbicsXG4gICAgJ2dsdWU6R2V0UGFydGl0aW9uJyxcbiAgICAnZ2x1ZTpHZXRQYXJ0aXRpb25zJyxcbiAgICAnZ2x1ZTpHZXRUYWJsZScsXG4gICAgJ2dsdWU6R2V0VGFibGVzJyxcbiAgICAnZ2x1ZTpHZXRUYWJsZVZlcnNpb24nLFxuICAgICdnbHVlOkdldFRhYmxlVmVyc2lvbnMnLFxuXTtcblxuZXhwb3J0IGNvbnN0IFdSSVRFX1RBQkxFX0FDVElPTlMgPSBbXG4gICAgJ2dsdWU6QmF0Y2hDcmVhdGVQYXJ0aXRpb24nLFxuICAgICdnbHVlOkJhdGNoRGVsZXRlUGFydGl0aW9uJyxcbiAgICAnZ2x1ZTpDcmVhdGVQYXJ0aXRpb24nLFxuICAgICdnbHVlOkRlbGV0ZVBhcnRpdGlvbicsXG4gICAgJ2dsdWU6VXBkYXRlUGFydGl0aW9uJyxcbiAgICAnZ2x1ZTpVcGRhdGVUYWJsZScsXG5dO1xuXG4vLy8gUzMgYWN0aW9ucyB0aGF0IG9wZXJhdGUgYXQgdGhlIGJ1Y2tldCBsZXZlbCBBTkQgc3VwcG9ydCB0aGVcbi8vLyBgczM6cHJlZml4YCByZXF1ZXN0IGNvbmRpdGlvbiBrZXkgKHBlciB0aGUgUzMgZG9jcykuIEdyYW50ZWQgb25cbi8vLyB0aGUgYnVja2V0IEFSTiB3aXRoIGEgYFN0cmluZ0xpa2UgczM6cHJlZml4ID0gWzxwcmVmaXg+KiwgPHByZWZpeD5dYFxuLy8vIGNvbmRpdGlvbiBzbyB0aGUgZ3JhbnRlZSBjYW4gb25seSBsaXN0IHRoZSB0YWJsZSdzIG93biBwcmVmaXguXG5leHBvcnQgY29uc3QgUkVBRF9TM19MSVNUX0FDVElPTlMgPSBbXG4gICAgJ3MzOkxpc3RCdWNrZXQnLFxuXTtcblxuLy8vIFMzIGFjdGlvbnMgdGhhdCBvcGVyYXRlIGF0IHRoZSBidWNrZXQgbGV2ZWwgYnV0IERPIE5PVCBzdXBwb3J0XG4vLy8gYHMzOnByZWZpeGAuIEdyYW50ZWQgb24gdGhlIGJ1Y2tldCBBUk4gd2l0aCBubyBjb25kaXRpb24g4oCUIGFkZGluZ1xuLy8vIG9uZSB3b3VsZCBzaWxlbnRseSBkZW55IHRoZXNlIGFjdGlvbnMgYXQgcnVudGltZSBldmVuIHRob3VnaCB0aGV5XG4vLy8gYXBwZWFyIGluIHRoZSBwb2xpY3kgZG9jdW1lbnQuXG5leHBvcnQgY29uc3QgUkVBRF9TM19CVUNLRVRfQUNUSU9OUyA9IFtcbiAgICAnczM6R2V0QnVja2V0TG9jYXRpb24nLFxuXTtcblxuZXhwb3J0IGNvbnN0IFdSSVRFX1MzX0JVQ0tFVF9BQ1RJT05TID0gW1xuICAgICdzMzpMaXN0QnVja2V0TXVsdGlwYXJ0VXBsb2FkcycsXG5dO1xuXG4vLy8gUzMgYWN0aW9ucyB0aGF0IG9wZXJhdGUgYXQgdGhlIG9iamVjdCBsZXZlbC4gTXVzdCBiZSBncmFudGVkIG9uXG4vLy8gdGhlIGBidWNrZXQvcHJlZml4KmAgQVJOLlxuZXhwb3J0IGNvbnN0IFJFQURfUzNfT0JKRUNUX0FDVElPTlMgPSBbXG4gICAgJ3MzOkdldE9iamVjdCcsXG5dO1xuXG5leHBvcnQgY29uc3QgV1JJVEVfUzNfT0JKRUNUX0FDVElPTlMgPSBbXG4gICAgJ3MzOlB1dE9iamVjdCcsXG4gICAgJ3MzOkRlbGV0ZU9iamVjdCcsXG4gICAgJ3MzOkFib3J0TXVsdGlwYXJ0VXBsb2FkJyxcbiAgICAnczM6TGlzdE11bHRpcGFydFVwbG9hZFBhcnRzJyxcbl07XG5cbi8qKlxuICogSXNzdWUgdGhlIGZvdXIgcG9saWN5IHN0YXRlbWVudHMgdGhhdCBzY29wZSBhbiBJY2ViZXJnLXRhYmxlIGdyYW50XG4gKiBjb3JyZWN0bHk6IEdsdWUgYWN0aW9ucyBvbiB0aGUgdGFibGUgQVJOLCBTMyBsaXN0LWJ1Y2tldCBhY3Rpb25zXG4gKiBvbiB0aGUgYnVja2V0IEFSTiB3aXRoIGFuIGBzMzpwcmVmaXhgIGNvbmRpdGlvbiBzbyBvbmx5IHRoZVxuICogdGFibGUncyBvd24gcHJlZml4IGNhbiBiZSBsaXN0ZWQsIFMzIGJ1Y2tldC1sZXZlbCBhY3Rpb25zIHRoYXQgRE9cbiAqIE5PVCBzdXBwb3J0IHRoZSBgczM6cHJlZml4YCBjb25kaXRpb24gKGUuZy4gYEdldEJ1Y2tldExvY2F0aW9uYCxcbiAqIGBMaXN0QnVja2V0TXVsdGlwYXJ0VXBsb2Fkc2ApIG9uIHRoZSBidWNrZXQgQVJOIHdpdGggbm8gY29uZGl0aW9uXG4gKiDigJQgaW5jbHVkaW5nIHRoZW0gaW4gdGhlIGNvbmRpdGlvbmVkIHN0YXRlbWVudCB3b3VsZCBzaWxlbnRseSBkZW55XG4gKiB0aGVtIGF0IHJ1bnRpbWUg4oCUIGFuZCBTMyBvYmplY3QtbGV2ZWwgYWN0aW9ucyBvbiB0aGVcbiAqIGBidWNrZXQvcHJlZml4KmAgQVJOLiBSZXR1cm5zIHRoZSB0YWJsZS1hY3Rpb25zIGdyYW50IChhbnkgb2YgdGhlXG4gKiBmb3VyIGlzIHN1ZmZpY2llbnQgZm9yIHRoZSBgR3JhbnRgIEFQSSBjb250cmFjdDsgdGhlIHJlc3QgYXR0YWNoXG4gKiBhcyBzaWRlIGVmZmVjdHMpLlxuICpcbiAqIEBpbnRlcm5hbFxuICovXG5leHBvcnQgZnVuY3Rpb24gZ3JhbnRTcGxpdChcbiAgICBncmFudGVlOiBJR3JhbnRhYmxlLFxuICAgIGFyZ3M6IHtcbiAgICAgICAgdGFibGVBcm46IHN0cmluZztcbiAgICAgICAgYnVja2V0QXJuOiBzdHJpbmc7XG4gICAgICAgIG9iamVjdEFybjogc3RyaW5nO1xuICAgICAgICBwcmVmaXhHbG9iOiBzdHJpbmc7XG4gICAgICAgIHRhYmxlQWN0aW9uczogc3RyaW5nW107XG4gICAgICAgIGxpc3RBY3Rpb25zOiBzdHJpbmdbXTtcbiAgICAgICAgYnVja2V0QWN0aW9uczogc3RyaW5nW107XG4gICAgICAgIG9iamVjdEFjdGlvbnM6IHN0cmluZ1tdO1xuICAgIH0sXG4pOiBHcmFudCB7XG4gICAgY29uc3QgdGFibGVHcmFudCA9IEdyYW50LmFkZFRvUHJpbmNpcGFsKHtcbiAgICAgICAgZ3JhbnRlZSxcbiAgICAgICAgYWN0aW9uczogYXJncy50YWJsZUFjdGlvbnMsXG4gICAgICAgIHJlc291cmNlQXJuczogW1xuICAgICAgICAgICAgYXJncy50YWJsZUFybixcbiAgICAgICAgXSxcbiAgICB9KTtcbiAgICBpZiAoYXJncy5saXN0QWN0aW9ucy5sZW5ndGggPiAwKSB7XG4gICAgICAgIEdyYW50LmFkZFRvUHJpbmNpcGFsKHtcbiAgICAgICAgICAgIGdyYW50ZWUsXG4gICAgICAgICAgICBhY3Rpb25zOiBhcmdzLmxpc3RBY3Rpb25zLFxuICAgICAgICAgICAgcmVzb3VyY2VBcm5zOiBbXG4gICAgICAgICAgICAgICAgYXJncy5idWNrZXRBcm4sXG4gICAgICAgICAgICBdLFxuICAgICAgICAgICAgY29uZGl0aW9uczoge1xuICAgICAgICAgICAgICAgIFN0cmluZ0xpa2U6IHtcbiAgICAgICAgICAgICAgICAgICAgJ3MzOnByZWZpeCc6IFtcbiAgICAgICAgICAgICAgICAgICAgICAgIGFyZ3MucHJlZml4R2xvYixcbiAgICAgICAgICAgICAgICAgICAgICAgIGFyZ3MucHJlZml4R2xvYi5yZXBsYWNlKC9cXCokLywgJycpLFxuICAgICAgICAgICAgICAgICAgICBdLFxuICAgICAgICAgICAgICAgIH0sXG4gICAgICAgICAgICB9LFxuICAgICAgICB9KTtcbiAgICB9XG4gICAgaWYgKGFyZ3MuYnVja2V0QWN0aW9ucy5sZW5ndGggPiAwKSB7XG4gICAgICAgIEdyYW50LmFkZFRvUHJpbmNpcGFsKHtcbiAgICAgICAgICAgIGdyYW50ZWUsXG4gICAgICAgICAgICBhY3Rpb25zOiBhcmdzLmJ1Y2tldEFjdGlvbnMsXG4gICAgICAgICAgICByZXNvdXJjZUFybnM6IFtcbiAgICAgICAgICAgICAgICBhcmdzLmJ1Y2tldEFybixcbiAgICAgICAgICAgIF0sXG4gICAgICAgIH0pO1xuICAgIH1cbiAgICBHcmFudC5hZGRUb1ByaW5jaXBhbCh7XG4gICAgICAgIGdyYW50ZWUsXG4gICAgICAgIGFjdGlvbnM6IGFyZ3Mub2JqZWN0QWN0aW9ucyxcbiAgICAgICAgcmVzb3VyY2VBcm5zOiBbXG4gICAgICAgICAgICBhcmdzLm9iamVjdEFybixcbiAgICAgICAgXSxcbiAgICB9KTtcbiAgICByZXR1cm4gdGFibGVHcmFudDtcbn1cbiJdfQ==
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
import { CfnTable } from 'aws-cdk-lib/aws-glue';
|
|
2
|
+
import { IcebergType } from './iceberg-type';
|
|
3
|
+
import { IcebergDataFormat, IcebergFormatVersion } from './iceberg-table-properties';
|
|
4
|
+
import { IcebergColumn, IcebergPartitionField, IcebergSortField, IcebergTableProps } from './iceberg-table';
|
|
5
|
+
/** @internal */
|
|
6
|
+
export declare function validateProps(props: IcebergTableProps): void;
|
|
7
|
+
/** @internal */
|
|
8
|
+
export declare function normalizeLocation(location: string): string;
|
|
9
|
+
/** @internal */
|
|
10
|
+
export declare function parseS3Uri(uri: string): {
|
|
11
|
+
bucket: string;
|
|
12
|
+
key: string;
|
|
13
|
+
};
|
|
14
|
+
interface RenderedColumn {
|
|
15
|
+
readonly id: number;
|
|
16
|
+
readonly type: IcebergType;
|
|
17
|
+
}
|
|
18
|
+
/** @internal */
|
|
19
|
+
export interface RenderedSchema {
|
|
20
|
+
readonly fields: CfnTable.IcebergStructFieldProperty[];
|
|
21
|
+
readonly columnByName: Map<string, RenderedColumn>;
|
|
22
|
+
}
|
|
23
|
+
/** @internal */
|
|
24
|
+
export declare function renderSchema(columns: IcebergColumn[]): RenderedSchema;
|
|
25
|
+
/** @internal */
|
|
26
|
+
export declare function validatePartitionSpec(partitionSpec: IcebergPartitionField[] | undefined, columnByName: Map<string, RenderedColumn>): void;
|
|
27
|
+
/** @internal */
|
|
28
|
+
export declare function validateSortOrder(sortOrder: IcebergSortField[] | undefined, columnByName: Map<string, RenderedColumn>): void;
|
|
29
|
+
/** @internal */
|
|
30
|
+
export declare function renderPartitionSpec(partitionSpec: IcebergPartitionField[] | undefined, columnByName: Map<string, RenderedColumn>): CfnTable.IcebergPartitionSpecProperty | undefined;
|
|
31
|
+
/** @internal */
|
|
32
|
+
export declare function renderSortOrder(sortOrder: IcebergSortField[] | undefined, columnByName: Map<string, RenderedColumn>): CfnTable.IcebergSortOrderProperty | undefined;
|
|
33
|
+
/** @internal */
|
|
34
|
+
export declare function resolveIdentifierFieldIds(identifierFieldNames: string[] | undefined, columnByName: Map<string, RenderedColumn>): number[] | undefined;
|
|
35
|
+
/** @internal */
|
|
36
|
+
export declare function mergeProperties(dataFormat: IcebergDataFormat, formatVersion: IcebergFormatVersion, user: {
|
|
37
|
+
[key: string]: string;
|
|
38
|
+
} | undefined, comment: string | undefined): {
|
|
39
|
+
[key: string]: string;
|
|
40
|
+
};
|
|
41
|
+
export {};
|