@aws-cdk/aws-glue-alpha 2.268.0-alpha.0 → 2.270.0-alpha.0

package/README.md

@@ -3,18 +3,16 @@
 
 ---
 
-![cdk-constructs: Experimental](https://img.shields.io/badge/cdk--constructs-experimental-important.svg?style=for-the-badge)
+![Deprecated](https://img.shields.io/badge/deprecated-critical.svg?style=for-the-badge)
 
-> The APIs of higher level constructs in this module are experimental and under active development.
-> They are subject to non-backward compatible changes or removal in any future version. These are
-> not subject to the [Semantic Versioning](https://semver.org/) model and breaking changes will be
-> announced in the release notes. This means that while you may use them, you may need to update
-> your source code when upgrading to a newer version of this package.
+> This API may emit warnings. Backward compatibility is not guaranteed.
 
 ---
 
 <!--END STABILITY BANNER-->
 
+All constructs moved to aws-cdk-lib/aws-glue.
+
 This module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk) project.
 
 ## README
@@ -81,6 +79,17 @@ The Spark UI (`—enable-spark-ui`) is off by default; enable it by setting the
 You can find more details about version, worker type and other features in
 [Glue's public documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html).
 
+> **Note on continuous logging and encryption:** Because continuous logging is
+> enabled by default, job driver and executor stdout/stderr are streamed to
+> CloudWatch. Unless you attach a [`SecurityConfiguration`](#securityconfiguration)
+> with `cloudWatchEncryption`, these logs are written to the account-shared,
+> default Glue log group (`/aws-glue/jobs/logs-v2/`), which is **not** encrypted
+> with a customer-managed key. Since job logs can contain sensitive runtime data
+> (SQL statements, row values, error stack traces), attach a `SecurityConfiguration`
+> with `cloudWatchEncryption` for regulated workloads. The construct emits a
+> synthesis-time warning when continuous logging is on and no `SecurityConfiguration`
+> is attached.
+
 Reference the pyspark-etl-jobs.test.ts and scalaspark-etl-jobs.test.ts unit tests
 for examples of required-only and optional job parameters when creating these
 types of jobs.
@@ -256,8 +265,9 @@ Python shell jobs support a Python version that depends on the AWS Glue
 version you use. These can be used to schedule and run tasks that don't
 require an Apache Spark environment. Python shell jobs default to
 Python 3.9 and a MaxCapacity of `0.0625`. Python 3.9 supports pre-loaded
-analytics libraries using the `library-set=analytics` flag, which is
-enabled by default.
+analytics libraries, enabled by default (`librarySet: glue.LibrarySet.ANALYTICS`).
+Set `librarySet: glue.LibrarySet.NONE` when your libraries are custom or
+conflict with the pre-installed ones.
 
 Reference the pyspark-shell-job.test.ts unit tests for examples of 
 required-only and optional job parameters when creating these types of jobs.
@@ -342,6 +352,51 @@ new glue.PySparkEtlJob(stack, 'SelectiveJob', {
 
 This feature is available for all Spark job types (ETL, Streaming, Flex).
 
+### Job Arguments
+
+Glue jobs are configured through a map of name-value arguments (`DefaultArguments`). This construct
+manages several of these arguments on your behalf and exposes each one through a dedicated,
+strongly-typed prop:
+
+| Managed argument(s)                                                      | Prop                                                            |
+|--------------------------------------------------------------------------|-----------------------------------------------------------------|
+| `--enable-continuous-cloudwatch-log`, `--continuous-log-*`               | `continuousLogging`                                             |
+| `--enable-metrics`                                                       | `enableMetrics`                                                 |
+| `--enable-observability-metrics`                                         | `enableObservabilityMetrics`                                    |
+| `--enable-spark-ui`, `--spark-event-logs-path`                           | `sparkUI`                                                       |
+| `--job-language`, `--class`                                              | job class / `className`                                         |
+| `--extra-jars`, `--user-jars-first`, `--extra-py-files`, `--extra-files` | `extraJars`, `extraJarsFirst`, `extraPythonFiles`, `extraFiles` |
+| `library-set`                                                            | `librarySet` (Python Shell)                                     |
+
+The `defaultArguments` prop is the escape hatch for arguments this construct does **not** model.
+Use it for any argument without a dedicated prop:
+
+```ts
+import * as cdk from 'aws-cdk-lib';
+import * as iam from 'aws-cdk-lib/aws-iam';
+declare const stack: cdk.Stack;
+declare const role: iam.IRole;
+declare const script: glue.Code;
+
+new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
+  role,
+  script,
+  defaultArguments: {
+    // an argument this construct does not manage
+    '--enable-glue-datacatalog': 'true',
+  },
+});
+```
+
+To keep a single, unambiguous way to express each intent, setting a **construct-managed** argument
+(any argument in the table above) or a **Glue-reserved** argument (`--debug`, `--mode`,
+`--JOB_NAME`, `--endpoint`) through `defaultArguments` throws at synthesis time. This holds even
+when the feature is turned off — for example, `enableMetrics: false` combined with
+`defaultArguments: { '--enable-metrics': '' }` throws rather than silently re-enabling metrics.
+Configure managed arguments through their dedicated prop instead — for example, use
+`continuousLogging: { enabled: false }` rather than
+`defaultArguments: { '--enable-continuous-cloudwatch-log': 'false' }`.
+
 ### Enable Job Run Queuing
 
 AWS Glue job queuing monitors your account level quotas and limits. If quotas or limits are insufficient to start a Glue job run, AWS Glue will automatically queue the job and wait for limits to free up. Once limits become available, AWS Glue will retry the job run. Glue jobs will queue for limits like max concurrent job runs per account, max concurrent Data Processing Units (DPU), and resource unavailable due to IP address exhaustion in Amazon Virtual Private Cloud (Amazon VPC).
@@ -405,7 +460,7 @@ const job = new glue.PySparkEtlJob(stack, 'Job', { role, script });
 // Create a workflow and add a trigger that runs the job
 const workflow = new glue.Workflow(stack, 'Workflow');
 workflow.addOnDemandTrigger('OnDemandTrigger', {
-  actions: [{ job }],
+  actions: [glue.Action.job(job)],
 });
 ```
 
@@ -418,21 +473,35 @@ actions list using the job or crawler objects using conditional types.
 
 #### **2. Scheduled Triggers**
 
-You can create scheduled triggers using cron expressions. This construct
-provides daily and weekly convenience functions,
-as well as a custom function that allows you to create your own
-custom timing using the [existing event Schedule class](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_events.Schedule.html)
-without having to build your own cron expressions. The L2 extracts
-the expression that Glue requires from the Schedule object. The constructor
-takes an optional description and a list of jobs or crawlers as actions.
+Use `addScheduledTrigger` with a `TriggerSchedule` to fire on a cron schedule.
+`TriggerSchedule.daily()` and `TriggerSchedule.weekly()` are convenience
+factories; `TriggerSchedule.cron(...)` lets you build any schedule from the
+[existing event Schedule class](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_events.Schedule.html)
+without writing raw cron expressions. The L2 extracts the expression that Glue
+requires from the `TriggerSchedule`.
 
-#### **3. Notify  Event Triggers**
+```ts
+import * as cdk from 'aws-cdk-lib';
+import * as iam from 'aws-cdk-lib/aws-iam';
+declare const stack: cdk.Stack;
+declare const role: iam.IRole;
+declare const script: glue.Code;
+const job = new glue.PySparkEtlJob(stack, 'Job', { role, script });
+const workflow = new glue.Workflow(stack, 'Workflow');
+
+workflow.addScheduledTrigger('WeeklyTrigger', {
+  actions: [glue.Action.job(job)],
+  schedule: glue.TriggerSchedule.weekly(),
+});
+```
 
-There are two types of notify event triggers: batching and non-batching.
-For batching triggers, you must specify `BatchSize`. For non-batching
-triggers, `BatchSize` defaults to 1. For both triggers, `BatchWindow`
-defaults to 900 seconds, but you can override the window to align with
-your workload's requirements.
+#### **3. Event Triggers**
+
+Use `addEventTrigger` for EventBridge event-based triggers. There are two types:
+batching and non-batching. For batching triggers, you must specify `batchSize`.
+For non-batching triggers, `batchSize` defaults to 1. For both, `batchWindow`
+defaults to 900 seconds, but you can override the window to align with your
+workload's requirements.
 
 #### **4. Conditional Triggers**
 
@@ -451,13 +520,14 @@ certain types of data stores.
 
 * **Networking - the CDK determines the best fit subnet for Glue connection
 configuration**
-    You can specify the exact subnet of the Connection when it's defined, but
-    you are not required to. Instead, you can provide a `vpc` and, optionally, a
-    `vpcSubnets` selection, and the L2 leverages the existing
+    Configure VPC placement through the `network` property, built with
+    `ConnectionNetwork.subnet(subnet)` to pin a specific subnet, or
+    `ConnectionNetwork.vpc(vpc, vpcSubnets?)` to let the L2 select one via the
+    existing
     [EC2 Subnet Selection](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_ec2/SubnetSelection.html)
-    library to make the best choice selection for the subnet. A Glue connection
-    targets a single subnet, so the first subnet of the selection is used.
-    `subnet` and `vpc` are mutually exclusive.
+    library. A Glue connection targets a single subnet, so the first subnet of
+    the selection is used. The two factories are mutually exclusive, so a subnet
+    and a VPC can never be combined.
 
 Pin the connection to a specific subnet:
 
@@ -469,7 +539,7 @@ new glue.Connection(this, 'MyConnection', {
   // The security groups granting AWS Glue inbound access to the data source within the VPC
   securityGroups: [securityGroup],
   // The VPC subnet which contains the data source
-  subnet,
+  network: glue.ConnectionNetwork.subnet(subnet),
 });
 ```
 
@@ -481,9 +551,8 @@ declare const vpc: ec2.Vpc;
 new glue.Connection(this, 'MyConnection', {
   type: glue.ConnectionType.NETWORK,
   securityGroups: [securityGroup],
-  vpc,
-  // Optional - defaults to private subnets
-  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
+  // vpcSubnets is optional - defaults to private subnets
+  network: glue.ConnectionNetwork.vpc(vpc, { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }),
 });
 ```
 
@@ -496,7 +565,7 @@ declare const db: rds.DatabaseCluster;
 new glue.Connection(this, "RdsConnection", {
   type: glue.ConnectionType.JDBC,
   securityGroups: [securityGroup],
-  subnet,
+  network: glue.ConnectionNetwork.subnet(subnet),
   secret: db.secret,
   properties: {
     JDBC_CONNECTION_URL: `jdbc:mysql://${db.clusterEndpoint.socketAddress}/databasename`,
@@ -945,8 +1014,9 @@ new glue.S3Table(this, 'MyTable', {
       min: '2020-01-01',
       max: '2023-12-31',
       format: 'yyyy-MM-dd',
-      interval: 1,  // optional, defaults to 1
-      intervalUnit: glue.DateIntervalUnit.DAYS,  // optional: YEARS, MONTHS, WEEKS, DAYS, HOURS, MINUTES, SECONDS
+      // `step` bundles interval + unit (supply both or neither). Optional at day
+      // precision or coarser; required when the format is sub-day (e.g. hours).
+      step: { interval: 1, intervalUnit: glue.DateIntervalUnit.DAYS },
     }),
   },
 });

package/adr/job-arguments.md

@@ -0,0 +1,212 @@
+# Glue Job Arguments
+
+## Status
+
+accepted
+
+## Context
+
+Every Glue job resource (`AWS::Glue::Job`) accepts a `DefaultArguments` map — a
+flat `string → string` dictionary of `--flag`/value pairs that Glue passes to the
+job script on every run. Some of these arguments are ordinary user configuration
+(`--additional-python-modules`, `--enable-glue-datacatalog`, `--TempDir`, …), but
+others are the wire form of features the CDK L2 models with strongly-typed props.
+
+The L2 job constructs therefore populate `DefaultArguments` from two sources:
+
+1. **Construct-managed arguments** — derived by the construct from typed props or
+   from the job class itself. Examples:
+   - `continuousLogging` → `--enable-continuous-cloudwatch-log`,
+     `--continuous-log-logGroup`, `--continuous-log-logStreamPrefix`,
+     `--continuous-log-conversionPattern`, `--enable-continuous-log-filter`
+   - `enableMetrics` → `--enable-metrics`
+   - `enableObservabilityMetrics` → `--enable-observability-metrics`
+   - `sparkUI` → `--enable-spark-ui`, `--spark-event-logs-path`
+   - `extraJars` / `extraJarsFirst` / `extraPythonFiles` / `extraFiles` →
+     `--extra-jars`, `--user-jars-first`, `--extra-py-files`, `--extra-files`
+   - `className` → `--class` (Scala only)
+   - the job language itself → `--job-language`
+   - `librarySet` → `library-set` (Python Shell only)
+2. **`defaultArguments`** — the untyped escape-hatch map the user supplies directly,
+   for arguments the L2 does *not* model.
+
+The two sources can collide. Before this decision, the collision was resolved
+silently and inconsistently across job types:
+
+- `SparkJob` / `PythonShellJob` merged as `{ ...managed, ...userDefaultArguments }`,
+  so the user value won — a user could pass
+  `defaultArguments: { '--enable-continuous-cloudwatch-log': 'false' }` and silently
+  turn off a secure default.
+- `RayJob` merged the other way, so the construct value won — a user's
+  `defaultArguments` entry for a managed key was silently dropped.
+
+Both behaviors are footguns: one weakens the construct's secure/observable defaults
+without warning, the other ignores explicit user input without warning. Because Glue
+enables continuous CloudWatch logging by default and that data can contain sensitive
+runtime values (SQL, row data, stack traces), the "user silently wins" case is also a
+security concern.
+
+Separately, Glue itself reserves a handful of argument keys for its own internal use
+(`--debug`, `--mode`, `--JOB_NAME`, `--endpoint`). These are never valid user input on
+any job type.
+
+## Constraints
+
+- `DefaultArguments` is a single flat map on the L1; there is no separate channel to
+  distinguish "managed" from "user" keys once they are merged. Whatever the L2 does,
+  it must produce one merged map.
+- Which keys are managed varies by job type: `--class` exists only for Scala jobs,
+  `library-set` only for Python Shell, `--enable-spark-ui` only for Spark, and so on.
+  A single global list would either over-reject (block a key that is a legitimate
+  escape hatch for a job type that doesn't manage it — e.g. `--extra-py-files` on a
+  job with no `extraPythonFiles` prop) or under-reject.
+- Argument keys can be tokens (e.g. produced by `CfnJson`) that only resolve at
+  deploy time. String comparison cannot see through them at synthesis.
+- The set of managed keys must not be tied to the set the construct *happens to emit*
+  for a given configuration: a prop that turns a feature off (`enableMetrics:
+  false`) emits nothing, but the key is still construct-managed and must stay reserved.
+
+## Decision
+
+**A managed argument has exactly one way to be configured: its typed prop.** Passing a
+construct-managed or Glue-reserved key through `defaultArguments` throws a
+`ValidationError` at synthesis time rather than silently winning or being dropped.
+`defaultArguments` remains the escape hatch for every argument the L2 does not model.
+
+### Data flow
+
+There is a single sink for every construct-managed argument — the base-class method:
+
+```ts
+protected setManagedArgument(key: string, value?: string): void
+```
+
+It records `key` in the reserved set and, when `value !== undefined`, emits it. A
+subclass calls it once per managed key, passing `undefined` when the feature is off or
+unset — the key is reserved either way. Subclasses do not build local argument maps, so
+this is the *only* way to emit a managed argument: declaration and emission happen in the
+same call, and the reserved set therefore cannot drift from what is emitted.
+
+Each job subclass, in its constructor:
+
+1. Registers its managed arguments through `setManagedArgument` — directly, or through
+   the shared helpers `setupContinuousLogging` (all job types),
+   `nonExecutableCommonArguments` and `setupExtraCodeArguments` (Spark), and its own
+   `executableArguments` (`--job-language`; plus `--class` for Scala, `library-set` for
+   Python Shell).
+2. Calls the base-class method:
+
+   ```ts
+   protected mergeDefaultArguments(
+     defaultArguments?: { [key: string]: string },
+   ): { [key: string]: string }
+   ```
+
+   which validates the user-supplied `defaultArguments` against the accumulated reserved
+   set and returns the merged map, passed as `DefaultArguments` on the `CfnJob`.
+
+`setManagedArgument` declares managed keys; `mergeDefaultArguments` validates and merges
+them. Each is the sole choke point for its job.
+
+Which keys a job type reserves falls out of which `setManagedArgument` calls its
+constructor makes: only Scala jobs register `--class`, only Python Shell registers
+`library-set`, only Spark registers `--enable-spark-ui`, and so on. Per-job-type scoping
+is automatic — there is no separate list to maintain per type.
+
+### The reserved set
+
+The keys a user may not set through `defaultArguments` are the union of:
+
+- **`GLUE_RESERVED_ARGUMENTS`** — `--debug`, `--mode`, `--JOB_NAME`, `--endpoint`. Owned
+  by the Glue service, reserved on every job type. This is the one static list, because
+  it is external to the constructs — nothing derives it from a prop.
+- **`_managedArgumentKeys`** — every key that any `setManagedArgument` call registered on
+  this instance, whether or not a value was emitted for it.
+
+### Validation rules (per user-supplied key)
+
+For each key in `defaultArguments`:
+
+1. If the key is an unresolved token, the conflict check is skipped (equality is
+   unknowable at synth time) and a warning
+   (`@aws-cdk/aws-glue-alpha:tokenJobArgumentKey`) is emitted. If it resolves to a
+   managed key at deploy time, the construct-managed value wins (see merge order).
+2. If the key is in **`GLUE_RESERVED_ARGUMENTS`** → throw. Glue-reserved keys are
+   never emitted by the construct, so there is no construct value to reconcile against.
+3. If the key is in the reserved set (`_managedArgumentKeys`):
+   - If the construct actually emitted a value for that key
+     (`Object.hasOwn(_managedArguments, key)`) and the supplied value is identical →
+     allowed. Passing the same value the construct would produce is not
+     contradictory; autocorrecting config is preferred over an error.
+   - Otherwise (different value, or the construct emitted nothing because the feature
+     is off) → throw.
+4. Otherwise, the key is genuinely custom → allowed, flows through untouched.
+
+`Object.hasOwn` is used deliberately instead of the `in` operator so that inherited
+`Object.prototype` members (`toString`, `constructor`, `hasOwnProperty`, …) supplied as
+argument keys are treated as ordinary custom keys rather than falsely matching a
+managed key.
+
+### Merge order
+
+After validation, the result is:
+
+```ts
+return { ...defaultArguments, ...this._managedArguments };
+```
+
+Managed arguments are spread last, so they win on any residual overlap. By this point
+the only overlaps that can remain are (a) exact-value matches allowed by rule 3, which
+are indistinguishable either way, and (b) token keys from rule 1, for which
+managed-wins is the documented and warned-about behavior.
+
+### Related synthesis-time warnings
+
+Two other warnings live in the same flow:
+
+- **`@aws-cdk/aws-glue-alpha:unencryptedContinuousLogging`** — continuous logging is on
+  (explicitly or by default) but no `SecurityConfiguration` is attached, so driver /
+  executor logs land in an unencrypted, account-shared CloudWatch log group. We only
+  warn when *no* security configuration is attached at all, because
+  `ISecurityConfiguration` exposes only the name and we cannot introspect whether it
+  actually configures `cloudWatchEncryption` (avoiding false positives).
+- **`@aws-cdk/aws-glue-alpha:plaintextJobArgumentSecret`** — a `defaultArguments` key
+  looks like a credential and holds a plaintext literal. `DefaultArguments` is emitted
+  verbatim into the template; secrets belong in AWS Secrets Manager.
+
+## Alternatives
+
+### Invert precedence so the construct always wins
+
+Merge as `{ ...userDefaultArguments, ...managed }` everywhere (which is what `RayJob`
+already did). This is more secure than "user wins" but still silent: a user who
+deliberately sets a managed key via `defaultArguments` has it dropped with no
+indication. It also still leaves two channels for one setting. Rejected in favor of a
+single, explicit way to express each intent.
+
+### Derive the reserved set from the emitted arguments
+
+Compute conflicts from the keys the construct actually emits. Attractive: adding a typed
+prop reserves its key automatically, with no separate list to maintain. Rejected: it ties
+the reserved set to the current configuration. A prop that turns a feature off emits no
+key, so `defaultArguments` could silently re-enable it (`enableMetrics: false` +
+`defaultArguments: { '--enable-metrics': '' }`). The reserved set must include keys the
+construct manages even when it emits nothing for them. That is why `setManagedArgument`
+records the key regardless of value.
+
+### A per-job-type list of managed keys
+
+Give each job type an explicit `string[]` of the keys it manages, passed to the
+validation step alongside the emitted map. Correct, but it names every managed key in
+two places — the emission site (inside an `enabled ? {...} : {}` expression) and the
+list — which drift: adding a typed prop requires updating the list too, and a forgotten
+entry silently reopens the re-enable bypass with no compile-time signal. The
+`setManagedArgument` accumulator collapses the two into one call, so the key is named
+once.
+
+### One global reserved list on the base class
+
+A single list of every managed key across all job types. Rejected because it
+over-rejects: it would block, for example, `--extra-py-files` on a job type that has no
+`extraPythonFiles` prop, removing a legitimate escape hatch with no typed replacement.
+Managed keys must be scoped per job type.