@aws-cdk/aws-glue-alpha 2.90.0-alpha.0 → 2.91.0-alpha.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.jsii +104 -103
- package/.jsii.tabl.json.gz +0 -0
- package/README.md +20 -0
- package/lib/code.js +3 -3
- package/lib/connection.js +2 -2
- package/lib/data-format.js +5 -5
- package/lib/data-quality-ruleset.js +2 -2
- package/lib/database.js +1 -1
- package/lib/job-executable.js +20 -18
- package/lib/job.d.ts +4 -1
- package/lib/job.js +38 -7
- package/lib/schema.js +1 -1
- package/lib/security-configuration.js +1 -1
- package/lib/storage-parameter.js +1 -1
- package/lib/table.js +1 -1
- package/package.json +8 -8
package/.jsii
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
"url": "https://aws.amazon.com"
|
|
9
9
|
},
|
|
10
10
|
"dependencies": {
|
|
11
|
-
"aws-cdk-lib": "2.
|
|
11
|
+
"aws-cdk-lib": "2.91.0",
|
|
12
12
|
"constructs": "^10.0.0"
|
|
13
13
|
},
|
|
14
14
|
"dependencyClosure": {
|
|
@@ -3513,7 +3513,7 @@
|
|
|
3513
3513
|
},
|
|
3514
3514
|
"name": "@aws-cdk/aws-glue-alpha",
|
|
3515
3515
|
"readme": {
|
|
3516
|
-
"markdown": "# AWS Glue Construct Library\n<!--BEGIN STABILITY BANNER-->\n\n---\n\n\n\n> The APIs of higher level constructs in this module are experimental and under active development.\n> They are subject to non-backward compatible changes or removal in any future version. These are\n> not subject to the [Semantic Versioning](https://semver.org/) model and breaking changes will be\n> announced in the release notes. This means that while you may use them, you may need to update\n> your source code when upgrading to a newer version of this package.\n\n---\n\n<!--END STABILITY BANNER-->\n\nThis module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk) project.\n\n## Job\n\nA `Job` encapsulates a script that connects to data sources, processes them, and then writes output to a data target.\n\nThere are 3 types of jobs supported by AWS Glue: Spark ETL, Spark Streaming, and Python Shell jobs.\n\nThe `glue.JobExecutable` allows you to specify the type of job, the language to use and the code assets required by the job.\n\n`glue.Code` allows you to refer to the different code assets required by the job, either from an existing S3 location or from a local file path.\n\n`glue.ExecutionClass` allows you to specify `FLEX` or `STANDARD`. `FLEX` is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads.\n\n### Spark Jobs\n\nThese jobs run in an Apache Spark environment managed by AWS Glue.\n\n#### ETL Jobs\n\nAn ETL job processes data in batches using Apache Spark.\n\n```ts\ndeclare const bucket: s3.Bucket;\nnew glue.Job(this, 'ScalaSparkEtlJob', {\n executable: glue.JobExecutable.scalaEtl({\n glueVersion: glue.GlueVersion.V4_0,\n script: glue.Code.fromBucket(bucket, 'src/com/example/HelloWorld.scala'),\n className: 'com.example.HelloWorld',\n extraJars: [glue.Code.fromBucket(bucket, 'jars/HelloWorld.jar')],\n }),\n workerType: glue.WorkerType.G_8X,\n description: 'an example Scala ETL job',\n});\n```\n\n#### Streaming Jobs\n\nA Streaming job is similar to an ETL job, except that it performs ETL on data streams. It uses the Apache Spark Structured Streaming framework. Some Spark job features are not available to streaming ETL jobs.\n\n```ts\nnew glue.Job(this, 'PythonSparkStreamingJob', {\n executable: glue.JobExecutable.pythonStreaming({\n glueVersion: glue.GlueVersion.V4_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n description: 'an example Python Streaming job',\n});\n```\n\n### Python Shell Jobs\n\nA Python shell job runs Python scripts as a shell and supports a Python version that depends on the AWS Glue version you are using.\nThis can be used to schedule and run tasks that don't require an Apache Spark environment. Currently, three flavors are supported:\n\n* PythonVersion.TWO (2.7; EOL)\n* PythonVersion.THREE (3.6)\n* PythonVersion.THREE_NINE (3.9)\n\n```ts\ndeclare const bucket: s3.Bucket;\nnew glue.Job(this, 'PythonShellJob', {\n executable: glue.JobExecutable.pythonShell({\n glueVersion: glue.GlueVersion.V1_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromBucket(bucket, 'script.py'),\n }),\n description: 'an example Python Shell job',\n});\n```\n\n### Ray Jobs\n\nThese jobs run in a Ray environment managed by AWS Glue.\n\n```ts\nnew glue.Job(this, 'RayJob', {\n executable: glue.JobExecutable.pythonRay({\n glueVersion: glue.GlueVersion.V4_0,\n pythonVersion: glue.PythonVersion.THREE_NINE,\n runtime: glue.Runtime.RAY_TWO_FOUR,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n workerType: glue.WorkerType.Z_2X,\n workerCount: 2,\n description: 'an example Ray job'\n});\n```\n\nSee [documentation](https://docs.aws.amazon.com/glue/latest/dg/add-job.html) for more information on adding jobs in Glue.\n\n## Connection\n\nA `Connection` allows Glue jobs, crawlers and development endpoints to access certain types of data stores. For example, to create a network connection to connect to a data source within a VPC:\n\n```ts\ndeclare const securityGroup: ec2.SecurityGroup;\ndeclare const subnet: ec2.Subnet;\nnew glue.Connection(this, 'MyConnection', {\n type: glue.ConnectionType.NETWORK,\n // The security groups granting AWS Glue inbound access to the data source within the VPC\n securityGroups: [securityGroup],\n // The VPC subnet which contains the data source\n subnet,\n});\n```\n\nFor RDS `Connection` by JDBC, it is recommended to manage credentials using AWS Secrets Manager. To use Secret, specify `SECRET_ID` in `properties` like the following code. Note that in this case, the subnet must have a route to the AWS Secrets Manager VPC endpoint or to the AWS Secrets Manager endpoint through a NAT gateway.\n\n```ts\ndeclare const securityGroup: ec2.SecurityGroup;\ndeclare const subnet: ec2.Subnet;\ndeclare const db: rds.DatabaseCluster;\nnew glue.Connection(this, \"RdsConnection\", {\n type: glue.ConnectionType.JDBC,\n securityGroups: [securityGroup],\n subnet,\n properties: {\n JDBC_CONNECTION_URL: `jdbc:mysql://${db.clusterEndpoint.socketAddress}/databasename`,\n JDBC_ENFORCE_SSL: \"false\",\n SECRET_ID: db.secret!.secretName,\n },\n});\n```\n\nIf you need to use a connection type that doesn't exist as a static member on `ConnectionType`, you can instantiate a `ConnectionType` object, e.g: `new glue.ConnectionType('NEW_TYPE')`.\n\nSee [Adding a Connection to Your Data Store](https://docs.aws.amazon.com/glue/latest/dg/populate-add-connection.html) and [Connection Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-connections.html#aws-glue-api-catalog-connections-Connection) documentation for more information on the supported data stores and their configurations.\n\n## SecurityConfiguration\n\nA `SecurityConfiguration` is a set of security properties that can be used by AWS Glue to encrypt data at rest.\n\n```ts\nnew glue.SecurityConfiguration(this, 'MySecurityConfiguration', {\n cloudWatchEncryption: {\n mode: glue.CloudWatchEncryptionMode.KMS,\n },\n jobBookmarksEncryption: {\n mode: glue.JobBookmarksEncryptionMode.CLIENT_SIDE_KMS,\n },\n s3Encryption: {\n mode: glue.S3EncryptionMode.KMS,\n },\n});\n```\n\nBy default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key for each encryption config, for example, for CloudWatch encryption:\n\n```ts\ndeclare const key: kms.Key;\nnew glue.SecurityConfiguration(this, 'MySecurityConfiguration', {\n cloudWatchEncryption: {\n mode: glue.CloudWatchEncryptionMode.KMS,\n kmsKey: key,\n },\n});\n```\n\nSee [documentation](https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html) for more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.\n\n## Database\n\nA `Database` is a logical grouping of `Tables` in the Glue Catalog.\n\n```ts\nnew glue.Database(this, 'MyDatabase');\n```\n\n## Table\n\nA Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }, {\n name: 'col2',\n type: glue.Schema.array(glue.Schema.STRING),\n comment: 'col2 is an array of strings' // comment is optional\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\nBy default, a S3 bucket will be created to store the table's data but you can manually pass the `bucket` and `s3Prefix`:\n\n```ts\ndeclare const myBucket: s3.Bucket;\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n bucket: myBucket,\n s3Prefix: 'my-table/',\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\nGlue tables can be configured to contain user-defined properties, to describe the physical storage of table data, through the `storageParameters` property:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n storageParameters: [\n glue.StorageParameter.skipHeaderLineCount(1),\n glue.StorageParameter.compressionType(glue.CompressionType.GZIP),\n glue.StorageParameter.custom('separatorChar', ',')\n ],\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n### Partition Keys\n\nTo improve query performance, a table can specify `partitionKeys` on which data is stored and queried separately. For example, you might partition a table by `year` and `month` to optimize queries based on a time window:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n partitionKeys: [{\n name: 'year',\n type: glue.Schema.SMALL_INT,\n }, {\n name: 'month',\n type: glue.Schema.SMALL_INT,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n### Partition Indexes\n\nAnother way to improve query performance is to specify partition indexes. If no partition indexes are\npresent on the table, AWS Glue loads all partitions of the table and filters the loaded partitions using\nthe query expression. The query takes more time to run as the number of partitions increase. With an\nindex, the query will try to fetch a subset of the partitions instead of loading all partitions of the\ntable.\n\nThe keys of a partition index must be a subset of the partition keys of the table. You can have a\nmaximum of 3 partition indexes per table. To specify a partition index, you can use the `partitionIndexes`\nproperty:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n partitionKeys: [{\n name: 'year',\n type: glue.Schema.SMALL_INT,\n }, {\n name: 'month',\n type: glue.Schema.SMALL_INT,\n }],\n partitionIndexes: [{\n indexName: 'my-index', // optional\n keyNames: ['year'],\n }], // supply up to 3 indexes\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\nAlternatively, you can call the `addPartitionIndex()` function on a table:\n\n```ts\ndeclare const myTable: glue.Table;\nmyTable.addPartitionIndex({\n indexName: 'my-index',\n keyNames: ['year'],\n});\n```\n\n### Partition Filtering\n\nIf you have a table with a large number of partitions that grows over time, consider using AWS Glue partition indexing and filtering.\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n partitionKeys: [{\n name: 'year',\n type: glue.Schema.SMALL_INT,\n }, {\n name: 'month',\n type: glue.Schema.SMALL_INT,\n }],\n dataFormat: glue.DataFormat.JSON,\n enablePartitionFiltering: true,\n});\n```\n\n## [Encryption](https://docs.aws.amazon.com/athena/latest/ug/encryption.html)\n\nYou can enable encryption on a Table's data:\n\n* [S3Managed](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) - (default) Server side encryption (`SSE-S3`) with an Amazon S3-managed key.\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.S3_MANAGED,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n* [Kms](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) - Server-side encryption (`SSE-KMS`) with an AWS KMS Key managed by the account owner.\n\n```ts\ndeclare const myDatabase: glue.Database;\n// KMS key is created automatically\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.KMS,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n\n// with an explicit KMS key\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.KMS,\n encryptionKey: new kms.Key(this, 'MyKey'),\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n* [KmsManaged](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) - Server-side encryption (`SSE-KMS`), like `Kms`, except with an AWS KMS Key managed by the AWS Key Management Service.\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.KMS_MANAGED,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n* [ClientSideKms](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html#client-side-encryption-kms-managed-master-key-intro) - Client-side encryption (`CSE-KMS`) with an AWS KMS Key managed by the account owner.\n\n```ts\ndeclare const myDatabase: glue.Database;\n// KMS key is created automatically\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.CLIENT_SIDE_KMS,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n\n// with an explicit KMS key\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.CLIENT_SIDE_KMS,\n encryptionKey: new kms.Key(this, 'MyKey'),\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n*Note: you cannot provide a `Bucket` when creating the `Table` if you wish to use server-side encryption (`KMS`, `KMS_MANAGED` or `S3_MANAGED`)*.\n\n## Types\n\nA table's schema is a collection of columns, each of which have a `name` and a `type`. Types are recursive structures, consisting of primitive and complex types:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n columns: [{\n name: 'primitive_column',\n type: glue.Schema.STRING,\n }, {\n name: 'array_column',\n type: glue.Schema.array(glue.Schema.INTEGER),\n comment: 'array<integer>',\n }, {\n name: 'map_column',\n type: glue.Schema.map(\n glue.Schema.STRING,\n glue.Schema.TIMESTAMP),\n comment: 'map<string,string>',\n }, {\n name: 'struct_column',\n type: glue.Schema.struct([{\n name: 'nested_column',\n type: glue.Schema.DATE,\n comment: 'nested comment',\n }]),\n comment: \"struct<nested_column:date COMMENT 'nested comment'>\",\n }],\n // ...\n database: myDatabase,\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n### Primitives\n\n#### Numeric\n\n| Name \t| Type \t| Comments |\n|-----------\t|----------\t|------------------------------------------------------------------------------------------------------------------\t|\n| FLOAT \t| Constant \t| A 32-bit single-precision floating point number |\n| INTEGER \t| Constant \t| A 32-bit signed value in two's complement format, with a minimum value of -2^31 and a maximum value of 2^31-1 \t|\n| DOUBLE \t| Constant \t| A 64-bit double-precision floating point number |\n| BIG_INT \t| Constant \t| A 64-bit signed INTEGER in two’s complement format, with a minimum value of -2^63 and a maximum value of 2^63 -1 |\n| SMALL_INT \t| Constant \t| A 16-bit signed INTEGER in two’s complement format, with a minimum value of -2^15 and a maximum value of 2^15-1 |\n| TINY_INT \t| Constant \t| A 8-bit signed INTEGER in two’s complement format, with a minimum value of -2^7 and a maximum value of 2^7-1 |\n\n#### Date and time\n\n| Name \t| Type \t| Comments \t|\n|-----------\t|----------\t|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------\t|\n| DATE \t| Constant \t| A date in UNIX format, such as YYYY-MM-DD. \t|\n| TIMESTAMP \t| Constant \t| Date and time instant in the UNiX format, such as yyyy-mm-dd hh:mm:ss[.f...]. For example, TIMESTAMP '2008-09-15 03:04:05.324'. This format uses the session time zone. \t|\n\n#### String\n\n| Name \t| Type \t| Comments \t|\n|--------------------------------------------\t|----------\t|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\t|\n| STRING \t| Constant \t| A string literal enclosed in single or double quotes \t|\n| decimal(precision: number, scale?: number) \t| Function \t| `precision` is the total number of digits. `scale` (optional) is the number of digits in fractional part with a default of 0. For example, use these type definitions: decimal(11,5), decimal(15) \t|\n| char(length: number) \t| Function \t| Fixed length character data, with a specified length between 1 and 255, such as char(10) \t|\n| varchar(length: number) \t| Function \t| Variable length character data, with a specified length between 1 and 65535, such as varchar(10) \t|\n\n#### Miscellaneous\n\n| Name \t| Type \t| Comments \t|\n|---------\t|----------\t|-------------------------------\t|\n| BOOLEAN \t| Constant \t| Values are `true` and `false` \t|\n| BINARY \t| Constant \t| Value is in binary \t|\n\n### Complex\n\n| Name \t| Type \t| Comments \t|\n|-------------------------------------\t|----------\t|-------------------------------------------------------------------\t|\n| array(itemType: Type) \t| Function \t| An array of some other type \t|\n| map(keyType: Type, valueType: Type) \t| Function \t| A map of some primitive key type to any value type \t|\n| struct(collumns: Column[]) \t| Function \t| Nested structure containing individually named and typed collumns \t|\n\n## Data Quality Ruleset\n\nA `DataQualityRuleset` specifies a data quality ruleset with DQDL rules applied to a specified AWS Glue table. For example, to create a data quality ruleset for a given table:\n\n```ts\nnew glue.DataQualityRuleset(this, 'MyDataQualityRuleset', {\n clientToken: 'client_token',\n description: 'description',\n rulesetName: 'ruleset_name',\n rulesetDqdl: 'ruleset_dqdl',\n tags: {\n key1: 'value1',\n key2: 'value2',\n },\n targetTable: new glue.DataQualityTargetTable('database_name', 'table_name'),\n});\n```\n\nFor more information, see [AWS Glue Data Quality](https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html).\n"
|
|
3516
|
+
"markdown": "# AWS Glue Construct Library\n<!--BEGIN STABILITY BANNER-->\n\n---\n\n\n\n> The APIs of higher level constructs in this module are experimental and under active development.\n> They are subject to non-backward compatible changes or removal in any future version. These are\n> not subject to the [Semantic Versioning](https://semver.org/) model and breaking changes will be\n> announced in the release notes. This means that while you may use them, you may need to update\n> your source code when upgrading to a newer version of this package.\n\n---\n\n<!--END STABILITY BANNER-->\n\nThis module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk) project.\n\n## Job\n\nA `Job` encapsulates a script that connects to data sources, processes them, and then writes output to a data target.\n\nThere are 3 types of jobs supported by AWS Glue: Spark ETL, Spark Streaming, and Python Shell jobs.\n\nThe `glue.JobExecutable` allows you to specify the type of job, the language to use and the code assets required by the job.\n\n`glue.Code` allows you to refer to the different code assets required by the job, either from an existing S3 location or from a local file path.\n\n`glue.ExecutionClass` allows you to specify `FLEX` or `STANDARD`. `FLEX` is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads.\n\n### Spark Jobs\n\nThese jobs run in an Apache Spark environment managed by AWS Glue.\n\n#### ETL Jobs\n\nAn ETL job processes data in batches using Apache Spark.\n\n```ts\ndeclare const bucket: s3.Bucket;\nnew glue.Job(this, 'ScalaSparkEtlJob', {\n executable: glue.JobExecutable.scalaEtl({\n glueVersion: glue.GlueVersion.V4_0,\n script: glue.Code.fromBucket(bucket, 'src/com/example/HelloWorld.scala'),\n className: 'com.example.HelloWorld',\n extraJars: [glue.Code.fromBucket(bucket, 'jars/HelloWorld.jar')],\n }),\n workerType: glue.WorkerType.G_8X,\n description: 'an example Scala ETL job',\n});\n```\n\n#### Streaming Jobs\n\nA Streaming job is similar to an ETL job, except that it performs ETL on data streams. It uses the Apache Spark Structured Streaming framework. Some Spark job features are not available to streaming ETL jobs.\n\n```ts\nnew glue.Job(this, 'PythonSparkStreamingJob', {\n executable: glue.JobExecutable.pythonStreaming({\n glueVersion: glue.GlueVersion.V4_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n description: 'an example Python Streaming job',\n});\n```\n\n### Python Shell Jobs\n\nA Python shell job runs Python scripts as a shell and supports a Python version that depends on the AWS Glue version you are using.\nThis can be used to schedule and run tasks that don't require an Apache Spark environment. Currently, three flavors are supported:\n\n* PythonVersion.TWO (2.7; EOL)\n* PythonVersion.THREE (3.6)\n* PythonVersion.THREE_NINE (3.9)\n\n```ts\ndeclare const bucket: s3.Bucket;\nnew glue.Job(this, 'PythonShellJob', {\n executable: glue.JobExecutable.pythonShell({\n glueVersion: glue.GlueVersion.V1_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromBucket(bucket, 'script.py'),\n }),\n description: 'an example Python Shell job',\n});\n```\n\n### Ray Jobs\n\nThese jobs run in a Ray environment managed by AWS Glue.\n\n```ts\nnew glue.Job(this, 'RayJob', {\n executable: glue.JobExecutable.pythonRay({\n glueVersion: glue.GlueVersion.V4_0,\n pythonVersion: glue.PythonVersion.THREE_NINE,\n runtime: glue.Runtime.RAY_TWO_FOUR,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n workerType: glue.WorkerType.Z_2X,\n workerCount: 2,\n description: 'an example Ray job'\n});\n```\n\n### Enable Spark UI\n\nEnable Spark UI setting the `sparkUI` property.\n\n```ts\nnew glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});\n```\n\nThe `sparkUI` property also allows the specification of an s3 bucket and a bucket prefix.\n\nSee [documentation](https://docs.aws.amazon.com/glue/latest/dg/add-job.html) for more information on adding jobs in Glue.\n\n## Connection\n\nA `Connection` allows Glue jobs, crawlers and development endpoints to access certain types of data stores. For example, to create a network connection to connect to a data source within a VPC:\n\n```ts\ndeclare const securityGroup: ec2.SecurityGroup;\ndeclare const subnet: ec2.Subnet;\nnew glue.Connection(this, 'MyConnection', {\n type: glue.ConnectionType.NETWORK,\n // The security groups granting AWS Glue inbound access to the data source within the VPC\n securityGroups: [securityGroup],\n // The VPC subnet which contains the data source\n subnet,\n});\n```\n\nFor RDS `Connection` by JDBC, it is recommended to manage credentials using AWS Secrets Manager. To use Secret, specify `SECRET_ID` in `properties` like the following code. Note that in this case, the subnet must have a route to the AWS Secrets Manager VPC endpoint or to the AWS Secrets Manager endpoint through a NAT gateway.\n\n```ts\ndeclare const securityGroup: ec2.SecurityGroup;\ndeclare const subnet: ec2.Subnet;\ndeclare const db: rds.DatabaseCluster;\nnew glue.Connection(this, \"RdsConnection\", {\n type: glue.ConnectionType.JDBC,\n securityGroups: [securityGroup],\n subnet,\n properties: {\n JDBC_CONNECTION_URL: `jdbc:mysql://${db.clusterEndpoint.socketAddress}/databasename`,\n JDBC_ENFORCE_SSL: \"false\",\n SECRET_ID: db.secret!.secretName,\n },\n});\n```\n\nIf you need to use a connection type that doesn't exist as a static member on `ConnectionType`, you can instantiate a `ConnectionType` object, e.g: `new glue.ConnectionType('NEW_TYPE')`.\n\nSee [Adding a Connection to Your Data Store](https://docs.aws.amazon.com/glue/latest/dg/populate-add-connection.html) and [Connection Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-connections.html#aws-glue-api-catalog-connections-Connection) documentation for more information on the supported data stores and their configurations.\n\n## SecurityConfiguration\n\nA `SecurityConfiguration` is a set of security properties that can be used by AWS Glue to encrypt data at rest.\n\n```ts\nnew glue.SecurityConfiguration(this, 'MySecurityConfiguration', {\n cloudWatchEncryption: {\n mode: glue.CloudWatchEncryptionMode.KMS,\n },\n jobBookmarksEncryption: {\n mode: glue.JobBookmarksEncryptionMode.CLIENT_SIDE_KMS,\n },\n s3Encryption: {\n mode: glue.S3EncryptionMode.KMS,\n },\n});\n```\n\nBy default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key for each encryption config, for example, for CloudWatch encryption:\n\n```ts\ndeclare const key: kms.Key;\nnew glue.SecurityConfiguration(this, 'MySecurityConfiguration', {\n cloudWatchEncryption: {\n mode: glue.CloudWatchEncryptionMode.KMS,\n kmsKey: key,\n },\n});\n```\n\nSee [documentation](https://docs.aws.amazon.com/glue/latest/dg/encryption-security-configuration.html) for more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.\n\n## Database\n\nA `Database` is a logical grouping of `Tables` in the Glue Catalog.\n\n```ts\nnew glue.Database(this, 'MyDatabase');\n```\n\n## Table\n\nA Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }, {\n name: 'col2',\n type: glue.Schema.array(glue.Schema.STRING),\n comment: 'col2 is an array of strings' // comment is optional\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\nBy default, a S3 bucket will be created to store the table's data but you can manually pass the `bucket` and `s3Prefix`:\n\n```ts\ndeclare const myBucket: s3.Bucket;\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n bucket: myBucket,\n s3Prefix: 'my-table/',\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\nGlue tables can be configured to contain user-defined properties, to describe the physical storage of table data, through the `storageParameters` property:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n storageParameters: [\n glue.StorageParameter.skipHeaderLineCount(1),\n glue.StorageParameter.compressionType(glue.CompressionType.GZIP),\n glue.StorageParameter.custom('separatorChar', ',')\n ],\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n### Partition Keys\n\nTo improve query performance, a table can specify `partitionKeys` on which data is stored and queried separately. For example, you might partition a table by `year` and `month` to optimize queries based on a time window:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n partitionKeys: [{\n name: 'year',\n type: glue.Schema.SMALL_INT,\n }, {\n name: 'month',\n type: glue.Schema.SMALL_INT,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n### Partition Indexes\n\nAnother way to improve query performance is to specify partition indexes. If no partition indexes are\npresent on the table, AWS Glue loads all partitions of the table and filters the loaded partitions using\nthe query expression. The query takes more time to run as the number of partitions increase. With an\nindex, the query will try to fetch a subset of the partitions instead of loading all partitions of the\ntable.\n\nThe keys of a partition index must be a subset of the partition keys of the table. You can have a\nmaximum of 3 partition indexes per table. To specify a partition index, you can use the `partitionIndexes`\nproperty:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n partitionKeys: [{\n name: 'year',\n type: glue.Schema.SMALL_INT,\n }, {\n name: 'month',\n type: glue.Schema.SMALL_INT,\n }],\n partitionIndexes: [{\n indexName: 'my-index', // optional\n keyNames: ['year'],\n }], // supply up to 3 indexes\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\nAlternatively, you can call the `addPartitionIndex()` function on a table:\n\n```ts\ndeclare const myTable: glue.Table;\nmyTable.addPartitionIndex({\n indexName: 'my-index',\n keyNames: ['year'],\n});\n```\n\n### Partition Filtering\n\nIf you have a table with a large number of partitions that grows over time, consider using AWS Glue partition indexing and filtering.\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n partitionKeys: [{\n name: 'year',\n type: glue.Schema.SMALL_INT,\n }, {\n name: 'month',\n type: glue.Schema.SMALL_INT,\n }],\n dataFormat: glue.DataFormat.JSON,\n enablePartitionFiltering: true,\n});\n```\n\n## [Encryption](https://docs.aws.amazon.com/athena/latest/ug/encryption.html)\n\nYou can enable encryption on a Table's data:\n\n* [S3Managed](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) - (default) Server side encryption (`SSE-S3`) with an Amazon S3-managed key.\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.S3_MANAGED,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n* [Kms](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) - Server-side encryption (`SSE-KMS`) with an AWS KMS Key managed by the account owner.\n\n```ts\ndeclare const myDatabase: glue.Database;\n// KMS key is created automatically\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.KMS,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n\n// with an explicit KMS key\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.KMS,\n encryptionKey: new kms.Key(this, 'MyKey'),\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n* [KmsManaged](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) - Server-side encryption (`SSE-KMS`), like `Kms`, except with an AWS KMS Key managed by the AWS Key Management Service.\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.KMS_MANAGED,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n* [ClientSideKms](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html#client-side-encryption-kms-managed-master-key-intro) - Client-side encryption (`CSE-KMS`) with an AWS KMS Key managed by the account owner.\n\n```ts\ndeclare const myDatabase: glue.Database;\n// KMS key is created automatically\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.CLIENT_SIDE_KMS,\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n\n// with an explicit KMS key\nnew glue.Table(this, 'MyTable', {\n encryption: glue.TableEncryption.CLIENT_SIDE_KMS,\n encryptionKey: new kms.Key(this, 'MyKey'),\n // ...\n database: myDatabase,\n columns: [{\n name: 'col1',\n type: glue.Schema.STRING,\n }],\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n*Note: you cannot provide a `Bucket` when creating the `Table` if you wish to use server-side encryption (`KMS`, `KMS_MANAGED` or `S3_MANAGED`)*.\n\n## Types\n\nA table's schema is a collection of columns, each of which have a `name` and a `type`. Types are recursive structures, consisting of primitive and complex types:\n\n```ts\ndeclare const myDatabase: glue.Database;\nnew glue.Table(this, 'MyTable', {\n columns: [{\n name: 'primitive_column',\n type: glue.Schema.STRING,\n }, {\n name: 'array_column',\n type: glue.Schema.array(glue.Schema.INTEGER),\n comment: 'array<integer>',\n }, {\n name: 'map_column',\n type: glue.Schema.map(\n glue.Schema.STRING,\n glue.Schema.TIMESTAMP),\n comment: 'map<string,string>',\n }, {\n name: 'struct_column',\n type: glue.Schema.struct([{\n name: 'nested_column',\n type: glue.Schema.DATE,\n comment: 'nested comment',\n }]),\n comment: \"struct<nested_column:date COMMENT 'nested comment'>\",\n }],\n // ...\n database: myDatabase,\n dataFormat: glue.DataFormat.JSON,\n});\n```\n\n### Primitives\n\n#### Numeric\n\n| Name \t| Type \t| Comments |\n|-----------\t|----------\t|------------------------------------------------------------------------------------------------------------------\t|\n| FLOAT \t| Constant \t| A 32-bit single-precision floating point number |\n| INTEGER \t| Constant \t| A 32-bit signed value in two's complement format, with a minimum value of -2^31 and a maximum value of 2^31-1 \t|\n| DOUBLE \t| Constant \t| A 64-bit double-precision floating point number |\n| BIG_INT \t| Constant \t| A 64-bit signed INTEGER in two’s complement format, with a minimum value of -2^63 and a maximum value of 2^63 -1 |\n| SMALL_INT \t| Constant \t| A 16-bit signed INTEGER in two’s complement format, with a minimum value of -2^15 and a maximum value of 2^15-1 |\n| TINY_INT \t| Constant \t| A 8-bit signed INTEGER in two’s complement format, with a minimum value of -2^7 and a maximum value of 2^7-1 |\n\n#### Date and time\n\n| Name \t| Type \t| Comments \t|\n|-----------\t|----------\t|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------\t|\n| DATE \t| Constant \t| A date in UNIX format, such as YYYY-MM-DD. \t|\n| TIMESTAMP \t| Constant \t| Date and time instant in the UNiX format, such as yyyy-mm-dd hh:mm:ss[.f...]. For example, TIMESTAMP '2008-09-15 03:04:05.324'. This format uses the session time zone. \t|\n\n#### String\n\n| Name \t| Type \t| Comments \t|\n|--------------------------------------------\t|----------\t|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\t|\n| STRING \t| Constant \t| A string literal enclosed in single or double quotes \t|\n| decimal(precision: number, scale?: number) \t| Function \t| `precision` is the total number of digits. `scale` (optional) is the number of digits in fractional part with a default of 0. For example, use these type definitions: decimal(11,5), decimal(15) \t|\n| char(length: number) \t| Function \t| Fixed length character data, with a specified length between 1 and 255, such as char(10) \t|\n| varchar(length: number) \t| Function \t| Variable length character data, with a specified length between 1 and 65535, such as varchar(10) \t|\n\n#### Miscellaneous\n\n| Name \t| Type \t| Comments \t|\n|---------\t|----------\t|-------------------------------\t|\n| BOOLEAN \t| Constant \t| Values are `true` and `false` \t|\n| BINARY \t| Constant \t| Value is in binary \t|\n\n### Complex\n\n| Name \t| Type \t| Comments \t|\n|-------------------------------------\t|----------\t|-------------------------------------------------------------------\t|\n| array(itemType: Type) \t| Function \t| An array of some other type \t|\n| map(keyType: Type, valueType: Type) \t| Function \t| A map of some primitive key type to any value type \t|\n| struct(collumns: Column[]) \t| Function \t| Nested structure containing individually named and typed collumns \t|\n\n## Data Quality Ruleset\n\nA `DataQualityRuleset` specifies a data quality ruleset with DQDL rules applied to a specified AWS Glue table. For example, to create a data quality ruleset for a given table:\n\n```ts\nnew glue.DataQualityRuleset(this, 'MyDataQualityRuleset', {\n clientToken: 'client_token',\n description: 'description',\n rulesetName: 'ruleset_name',\n rulesetDqdl: 'ruleset_dqdl',\n tags: {\n key1: 'value1',\n key2: 'value2',\n },\n targetTable: new glue.DataQualityTargetTable('database_name', 'table_name'),\n});\n```\n\nFor more information, see [AWS Glue Data Quality](https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html).\n"
|
|
3517
3517
|
},
|
|
3518
3518
|
"repository": {
|
|
3519
3519
|
"directory": "packages/@aws-cdk/aws-glue-alpha",
|
|
@@ -3878,7 +3878,7 @@
|
|
|
3878
3878
|
"docs": {
|
|
3879
3879
|
"stability": "experimental",
|
|
3880
3880
|
"summary": "Represents a Glue Job's Code assets (an asset can be a scripts, a jar, a python file or any other file).",
|
|
3881
|
-
"example": "
|
|
3881
|
+
"example": "new glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});",
|
|
3882
3882
|
"custom": {
|
|
3883
3883
|
"exampleMetadata": "infused"
|
|
3884
3884
|
}
|
|
@@ -4765,7 +4765,7 @@
|
|
|
4765
4765
|
"kind": "interface",
|
|
4766
4766
|
"locationInModule": {
|
|
4767
4767
|
"filename": "lib/job.ts",
|
|
4768
|
-
"line":
|
|
4768
|
+
"line": 426
|
|
4769
4769
|
},
|
|
4770
4770
|
"name": "ContinuousLoggingProps",
|
|
4771
4771
|
"properties": [
|
|
@@ -4778,7 +4778,7 @@
|
|
|
4778
4778
|
"immutable": true,
|
|
4779
4779
|
"locationInModule": {
|
|
4780
4780
|
"filename": "lib/job.ts",
|
|
4781
|
-
"line":
|
|
4781
|
+
"line": 430
|
|
4782
4782
|
},
|
|
4783
4783
|
"name": "enabled",
|
|
4784
4784
|
"type": {
|
|
@@ -4796,7 +4796,7 @@
|
|
|
4796
4796
|
"immutable": true,
|
|
4797
4797
|
"locationInModule": {
|
|
4798
4798
|
"filename": "lib/job.ts",
|
|
4799
|
-
"line":
|
|
4799
|
+
"line": 460
|
|
4800
4800
|
},
|
|
4801
4801
|
"name": "conversionPattern",
|
|
4802
4802
|
"optional": true,
|
|
@@ -4814,7 +4814,7 @@
|
|
|
4814
4814
|
"immutable": true,
|
|
4815
4815
|
"locationInModule": {
|
|
4816
4816
|
"filename": "lib/job.ts",
|
|
4817
|
-
"line":
|
|
4817
|
+
"line": 437
|
|
4818
4818
|
},
|
|
4819
4819
|
"name": "logGroup",
|
|
4820
4820
|
"optional": true,
|
|
@@ -4832,7 +4832,7 @@
|
|
|
4832
4832
|
"immutable": true,
|
|
4833
4833
|
"locationInModule": {
|
|
4834
4834
|
"filename": "lib/job.ts",
|
|
4835
|
-
"line":
|
|
4835
|
+
"line": 444
|
|
4836
4836
|
},
|
|
4837
4837
|
"name": "logStreamPrefix",
|
|
4838
4838
|
"optional": true,
|
|
@@ -4850,7 +4850,7 @@
|
|
|
4850
4850
|
"immutable": true,
|
|
4851
4851
|
"locationInModule": {
|
|
4852
4852
|
"filename": "lib/job.ts",
|
|
4853
|
-
"line":
|
|
4853
|
+
"line": 451
|
|
4854
4854
|
},
|
|
4855
4855
|
"name": "quiet",
|
|
4856
4856
|
"optional": true,
|
|
@@ -5829,7 +5829,7 @@
|
|
|
5829
5829
|
"kind": "enum",
|
|
5830
5830
|
"locationInModule": {
|
|
5831
5831
|
"filename": "lib/job.ts",
|
|
5832
|
-
"line":
|
|
5832
|
+
"line": 139
|
|
5833
5833
|
},
|
|
5834
5834
|
"members": [
|
|
5835
5835
|
{
|
|
@@ -5856,7 +5856,7 @@
|
|
|
5856
5856
|
"see": "https://docs.aws.amazon.com/glue/latest/dg/add-job.html.\n\nIf you need to use a GlueVersion that doesn't exist as a static member, you\ncan instantiate a `GlueVersion` object, e.g: `GlueVersion.of('1.5')`.",
|
|
5857
5857
|
"stability": "experimental",
|
|
5858
5858
|
"summary": "AWS Glue version determines the versions of Apache Spark and Python that are available to the job.",
|
|
5859
|
-
"example": "
|
|
5859
|
+
"example": "new glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});",
|
|
5860
5860
|
"custom": {
|
|
5861
5861
|
"exampleMetadata": "infused"
|
|
5862
5862
|
}
|
|
@@ -6220,7 +6220,7 @@
|
|
|
6220
6220
|
"kind": "interface",
|
|
6221
6221
|
"locationInModule": {
|
|
6222
6222
|
"filename": "lib/job.ts",
|
|
6223
|
-
"line":
|
|
6223
|
+
"line": 156
|
|
6224
6224
|
},
|
|
6225
6225
|
"methods": [
|
|
6226
6226
|
{
|
|
@@ -6232,7 +6232,7 @@
|
|
|
6232
6232
|
},
|
|
6233
6233
|
"locationInModule": {
|
|
6234
6234
|
"filename": "lib/job.ts",
|
|
6235
|
-
"line":
|
|
6235
|
+
"line": 213
|
|
6236
6236
|
},
|
|
6237
6237
|
"name": "metric",
|
|
6238
6238
|
"parameters": [
|
|
@@ -6279,7 +6279,7 @@
|
|
|
6279
6279
|
},
|
|
6280
6280
|
"locationInModule": {
|
|
6281
6281
|
"filename": "lib/job.ts",
|
|
6282
|
-
"line":
|
|
6282
|
+
"line": 223
|
|
6283
6283
|
},
|
|
6284
6284
|
"name": "metricFailure",
|
|
6285
6285
|
"parameters": [
|
|
@@ -6305,7 +6305,7 @@
|
|
|
6305
6305
|
},
|
|
6306
6306
|
"locationInModule": {
|
|
6307
6307
|
"filename": "lib/job.ts",
|
|
6308
|
-
"line":
|
|
6308
|
+
"line": 218
|
|
6309
6309
|
},
|
|
6310
6310
|
"name": "metricSuccess",
|
|
6311
6311
|
"parameters": [
|
|
@@ -6331,7 +6331,7 @@
|
|
|
6331
6331
|
},
|
|
6332
6332
|
"locationInModule": {
|
|
6333
6333
|
"filename": "lib/job.ts",
|
|
6334
|
-
"line":
|
|
6334
|
+
"line": 228
|
|
6335
6335
|
},
|
|
6336
6336
|
"name": "metricTimeout",
|
|
6337
6337
|
"parameters": [
|
|
@@ -6358,7 +6358,7 @@
|
|
|
6358
6358
|
},
|
|
6359
6359
|
"locationInModule": {
|
|
6360
6360
|
"filename": "lib/job.ts",
|
|
6361
|
-
"line":
|
|
6361
|
+
"line": 174
|
|
6362
6362
|
},
|
|
6363
6363
|
"name": "onEvent",
|
|
6364
6364
|
"parameters": [
|
|
@@ -6391,7 +6391,7 @@
|
|
|
6391
6391
|
},
|
|
6392
6392
|
"locationInModule": {
|
|
6393
6393
|
"filename": "lib/job.ts",
|
|
6394
|
-
"line":
|
|
6394
|
+
"line": 195
|
|
6395
6395
|
},
|
|
6396
6396
|
"name": "onFailure",
|
|
6397
6397
|
"parameters": [
|
|
@@ -6424,7 +6424,7 @@
|
|
|
6424
6424
|
},
|
|
6425
6425
|
"locationInModule": {
|
|
6426
6426
|
"filename": "lib/job.ts",
|
|
6427
|
-
"line":
|
|
6427
|
+
"line": 181
|
|
6428
6428
|
},
|
|
6429
6429
|
"name": "onStateChange",
|
|
6430
6430
|
"parameters": [
|
|
@@ -6463,7 +6463,7 @@
|
|
|
6463
6463
|
},
|
|
6464
6464
|
"locationInModule": {
|
|
6465
6465
|
"filename": "lib/job.ts",
|
|
6466
|
-
"line":
|
|
6466
|
+
"line": 188
|
|
6467
6467
|
},
|
|
6468
6468
|
"name": "onSuccess",
|
|
6469
6469
|
"parameters": [
|
|
@@ -6496,7 +6496,7 @@
|
|
|
6496
6496
|
},
|
|
6497
6497
|
"locationInModule": {
|
|
6498
6498
|
"filename": "lib/job.ts",
|
|
6499
|
-
"line":
|
|
6499
|
+
"line": 202
|
|
6500
6500
|
},
|
|
6501
6501
|
"name": "onTimeout",
|
|
6502
6502
|
"parameters": [
|
|
@@ -6535,7 +6535,7 @@
|
|
|
6535
6535
|
"immutable": true,
|
|
6536
6536
|
"locationInModule": {
|
|
6537
6537
|
"filename": "lib/job.ts",
|
|
6538
|
-
"line":
|
|
6538
|
+
"line": 167
|
|
6539
6539
|
},
|
|
6540
6540
|
"name": "jobArn",
|
|
6541
6541
|
"type": {
|
|
@@ -6554,7 +6554,7 @@
|
|
|
6554
6554
|
"immutable": true,
|
|
6555
6555
|
"locationInModule": {
|
|
6556
6556
|
"filename": "lib/job.ts",
|
|
6557
|
-
"line":
|
|
6557
|
+
"line": 161
|
|
6558
6558
|
},
|
|
6559
6559
|
"name": "jobName",
|
|
6560
6560
|
"type": {
|
|
@@ -6860,7 +6860,7 @@
|
|
|
6860
6860
|
"docs": {
|
|
6861
6861
|
"stability": "experimental",
|
|
6862
6862
|
"summary": "A Glue Job.",
|
|
6863
|
-
"example": "
|
|
6863
|
+
"example": "new glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});",
|
|
6864
6864
|
"custom": {
|
|
6865
6865
|
"exampleMetadata": "infused"
|
|
6866
6866
|
}
|
|
@@ -6872,7 +6872,7 @@
|
|
|
6872
6872
|
},
|
|
6873
6873
|
"locationInModule": {
|
|
6874
6874
|
"filename": "lib/job.ts",
|
|
6875
|
-
"line":
|
|
6875
|
+
"line": 686
|
|
6876
6876
|
},
|
|
6877
6877
|
"parameters": [
|
|
6878
6878
|
{
|
|
@@ -6901,7 +6901,7 @@
|
|
|
6901
6901
|
"kind": "class",
|
|
6902
6902
|
"locationInModule": {
|
|
6903
6903
|
"filename": "lib/job.ts",
|
|
6904
|
-
"line":
|
|
6904
|
+
"line": 640
|
|
6905
6905
|
},
|
|
6906
6906
|
"methods": [
|
|
6907
6907
|
{
|
|
@@ -6911,7 +6911,7 @@
|
|
|
6911
6911
|
},
|
|
6912
6912
|
"locationInModule": {
|
|
6913
6913
|
"filename": "lib/job.ts",
|
|
6914
|
-
"line":
|
|
6914
|
+
"line": 648
|
|
6915
6915
|
},
|
|
6916
6916
|
"name": "fromJobAttributes",
|
|
6917
6917
|
"parameters": [
|
|
@@ -6958,7 +6958,7 @@
|
|
|
6958
6958
|
},
|
|
6959
6959
|
"locationInModule": {
|
|
6960
6960
|
"filename": "lib/job.ts",
|
|
6961
|
-
"line":
|
|
6961
|
+
"line": 320
|
|
6962
6962
|
},
|
|
6963
6963
|
"name": "metric",
|
|
6964
6964
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7006,7 +7006,7 @@
|
|
|
7006
7006
|
},
|
|
7007
7007
|
"locationInModule": {
|
|
7008
7008
|
"filename": "lib/job.ts",
|
|
7009
|
-
"line":
|
|
7009
|
+
"line": 347
|
|
7010
7010
|
},
|
|
7011
7011
|
"name": "metricFailure",
|
|
7012
7012
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7033,7 +7033,7 @@
|
|
|
7033
7033
|
},
|
|
7034
7034
|
"locationInModule": {
|
|
7035
7035
|
"filename": "lib/job.ts",
|
|
7036
|
-
"line":
|
|
7036
|
+
"line": 338
|
|
7037
7037
|
},
|
|
7038
7038
|
"name": "metricSuccess",
|
|
7039
7039
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7060,7 +7060,7 @@
|
|
|
7060
7060
|
},
|
|
7061
7061
|
"locationInModule": {
|
|
7062
7062
|
"filename": "lib/job.ts",
|
|
7063
|
-
"line":
|
|
7063
|
+
"line": 356
|
|
7064
7064
|
},
|
|
7065
7065
|
"name": "metricTimeout",
|
|
7066
7066
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7087,7 +7087,7 @@
|
|
|
7087
7087
|
},
|
|
7088
7088
|
"locationInModule": {
|
|
7089
7089
|
"filename": "lib/job.ts",
|
|
7090
|
-
"line":
|
|
7090
|
+
"line": 248
|
|
7091
7091
|
},
|
|
7092
7092
|
"name": "onEvent",
|
|
7093
7093
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7126,7 +7126,7 @@
|
|
|
7126
7126
|
},
|
|
7127
7127
|
"locationInModule": {
|
|
7128
7128
|
"filename": "lib/job.ts",
|
|
7129
|
-
"line":
|
|
7129
|
+
"line": 297
|
|
7130
7130
|
},
|
|
7131
7131
|
"name": "onFailure",
|
|
7132
7132
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7165,7 +7165,7 @@
|
|
|
7165
7165
|
},
|
|
7166
7166
|
"locationInModule": {
|
|
7167
7167
|
"filename": "lib/job.ts",
|
|
7168
|
-
"line":
|
|
7168
|
+
"line": 268
|
|
7169
7169
|
},
|
|
7170
7170
|
"name": "onStateChange",
|
|
7171
7171
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7212,7 +7212,7 @@
|
|
|
7212
7212
|
},
|
|
7213
7213
|
"locationInModule": {
|
|
7214
7214
|
"filename": "lib/job.ts",
|
|
7215
|
-
"line":
|
|
7215
|
+
"line": 287
|
|
7216
7216
|
},
|
|
7217
7217
|
"name": "onSuccess",
|
|
7218
7218
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7251,7 +7251,7 @@
|
|
|
7251
7251
|
},
|
|
7252
7252
|
"locationInModule": {
|
|
7253
7253
|
"filename": "lib/job.ts",
|
|
7254
|
-
"line":
|
|
7254
|
+
"line": 307
|
|
7255
7255
|
},
|
|
7256
7256
|
"name": "onTimeout",
|
|
7257
7257
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7294,7 +7294,7 @@
|
|
|
7294
7294
|
"immutable": true,
|
|
7295
7295
|
"locationInModule": {
|
|
7296
7296
|
"filename": "lib/job.ts",
|
|
7297
|
-
"line":
|
|
7297
|
+
"line": 676
|
|
7298
7298
|
},
|
|
7299
7299
|
"name": "grantPrincipal",
|
|
7300
7300
|
"overrides": "aws-cdk-lib.aws_iam.IGrantable",
|
|
@@ -7310,7 +7310,7 @@
|
|
|
7310
7310
|
"immutable": true,
|
|
7311
7311
|
"locationInModule": {
|
|
7312
7312
|
"filename": "lib/job.ts",
|
|
7313
|
-
"line":
|
|
7313
|
+
"line": 661
|
|
7314
7314
|
},
|
|
7315
7315
|
"name": "jobArn",
|
|
7316
7316
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7326,7 +7326,7 @@
|
|
|
7326
7326
|
"immutable": true,
|
|
7327
7327
|
"locationInModule": {
|
|
7328
7328
|
"filename": "lib/job.ts",
|
|
7329
|
-
"line":
|
|
7329
|
+
"line": 666
|
|
7330
7330
|
},
|
|
7331
7331
|
"name": "jobName",
|
|
7332
7332
|
"overrides": "@aws-cdk/aws-glue-alpha.IJob",
|
|
@@ -7342,7 +7342,7 @@
|
|
|
7342
7342
|
"immutable": true,
|
|
7343
7343
|
"locationInModule": {
|
|
7344
7344
|
"filename": "lib/job.ts",
|
|
7345
|
-
"line":
|
|
7345
|
+
"line": 671
|
|
7346
7346
|
},
|
|
7347
7347
|
"name": "role",
|
|
7348
7348
|
"type": {
|
|
@@ -7358,7 +7358,7 @@
|
|
|
7358
7358
|
"immutable": true,
|
|
7359
7359
|
"locationInModule": {
|
|
7360
7360
|
"filename": "lib/job.ts",
|
|
7361
|
-
"line":
|
|
7361
|
+
"line": 684
|
|
7362
7362
|
},
|
|
7363
7363
|
"name": "sparkUILoggingLocation",
|
|
7364
7364
|
"optional": true,
|
|
@@ -7384,7 +7384,7 @@
|
|
|
7384
7384
|
"kind": "interface",
|
|
7385
7385
|
"locationInModule": {
|
|
7386
7386
|
"filename": "lib/job.ts",
|
|
7387
|
-
"line":
|
|
7387
|
+
"line": 466
|
|
7388
7388
|
},
|
|
7389
7389
|
"name": "JobAttributes",
|
|
7390
7390
|
"properties": [
|
|
@@ -7397,7 +7397,7 @@
|
|
|
7397
7397
|
"immutable": true,
|
|
7398
7398
|
"locationInModule": {
|
|
7399
7399
|
"filename": "lib/job.ts",
|
|
7400
|
-
"line":
|
|
7400
|
+
"line": 470
|
|
7401
7401
|
},
|
|
7402
7402
|
"name": "jobName",
|
|
7403
7403
|
"type": {
|
|
@@ -7414,7 +7414,7 @@
|
|
|
7414
7414
|
"immutable": true,
|
|
7415
7415
|
"locationInModule": {
|
|
7416
7416
|
"filename": "lib/job.ts",
|
|
7417
|
-
"line":
|
|
7417
|
+
"line": 477
|
|
7418
7418
|
},
|
|
7419
7419
|
"name": "role",
|
|
7420
7420
|
"optional": true,
|
|
@@ -7516,7 +7516,7 @@
|
|
|
7516
7516
|
"docs": {
|
|
7517
7517
|
"stability": "experimental",
|
|
7518
7518
|
"summary": "The executable properties related to the Glue job's GlueVersion, JobType and code.",
|
|
7519
|
-
"example": "
|
|
7519
|
+
"example": "new glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});",
|
|
7520
7520
|
"custom": {
|
|
7521
7521
|
"exampleMetadata": "infused"
|
|
7522
7522
|
}
|
|
@@ -7731,7 +7731,7 @@
|
|
|
7731
7731
|
},
|
|
7732
7732
|
"locationInModule": {
|
|
7733
7733
|
"filename": "lib/job-executable.ts",
|
|
7734
|
-
"line":
|
|
7734
|
+
"line": 395
|
|
7735
7735
|
},
|
|
7736
7736
|
"name": "bind",
|
|
7737
7737
|
"returns": {
|
|
@@ -7759,7 +7759,7 @@
|
|
|
7759
7759
|
"kind": "interface",
|
|
7760
7760
|
"locationInModule": {
|
|
7761
7761
|
"filename": "lib/job-executable.ts",
|
|
7762
|
-
"line":
|
|
7762
|
+
"line": 403
|
|
7763
7763
|
},
|
|
7764
7764
|
"name": "JobExecutableConfig",
|
|
7765
7765
|
"properties": [
|
|
@@ -7773,7 +7773,7 @@
|
|
|
7773
7773
|
"immutable": true,
|
|
7774
7774
|
"locationInModule": {
|
|
7775
7775
|
"filename": "lib/job-executable.ts",
|
|
7776
|
-
"line":
|
|
7776
|
+
"line": 409
|
|
7777
7777
|
},
|
|
7778
7778
|
"name": "glueVersion",
|
|
7779
7779
|
"type": {
|
|
@@ -7790,7 +7790,7 @@
|
|
|
7790
7790
|
"immutable": true,
|
|
7791
7791
|
"locationInModule": {
|
|
7792
7792
|
"filename": "lib/job-executable.ts",
|
|
7793
|
-
"line":
|
|
7793
|
+
"line": 416
|
|
7794
7794
|
},
|
|
7795
7795
|
"name": "language",
|
|
7796
7796
|
"type": {
|
|
@@ -7806,7 +7806,7 @@
|
|
|
7806
7806
|
"immutable": true,
|
|
7807
7807
|
"locationInModule": {
|
|
7808
7808
|
"filename": "lib/job-executable.ts",
|
|
7809
|
-
"line":
|
|
7809
|
+
"line": 440
|
|
7810
7810
|
},
|
|
7811
7811
|
"name": "script",
|
|
7812
7812
|
"type": {
|
|
@@ -7822,7 +7822,7 @@
|
|
|
7822
7822
|
"immutable": true,
|
|
7823
7823
|
"locationInModule": {
|
|
7824
7824
|
"filename": "lib/job-executable.ts",
|
|
7825
|
-
"line":
|
|
7825
|
+
"line": 421
|
|
7826
7826
|
},
|
|
7827
7827
|
"name": "type",
|
|
7828
7828
|
"type": {
|
|
@@ -7841,7 +7841,7 @@
|
|
|
7841
7841
|
"immutable": true,
|
|
7842
7842
|
"locationInModule": {
|
|
7843
7843
|
"filename": "lib/job-executable.ts",
|
|
7844
|
-
"line":
|
|
7844
|
+
"line": 449
|
|
7845
7845
|
},
|
|
7846
7846
|
"name": "className",
|
|
7847
7847
|
"optional": true,
|
|
@@ -7860,7 +7860,7 @@
|
|
|
7860
7860
|
"immutable": true,
|
|
7861
7861
|
"locationInModule": {
|
|
7862
7862
|
"filename": "lib/job-executable.ts",
|
|
7863
|
-
"line":
|
|
7863
|
+
"line": 476
|
|
7864
7864
|
},
|
|
7865
7865
|
"name": "extraFiles",
|
|
7866
7866
|
"optional": true,
|
|
@@ -7884,7 +7884,7 @@
|
|
|
7884
7884
|
"immutable": true,
|
|
7885
7885
|
"locationInModule": {
|
|
7886
7886
|
"filename": "lib/job-executable.ts",
|
|
7887
|
-
"line":
|
|
7887
|
+
"line": 458
|
|
7888
7888
|
},
|
|
7889
7889
|
"name": "extraJars",
|
|
7890
7890
|
"optional": true,
|
|
@@ -7908,7 +7908,7 @@
|
|
|
7908
7908
|
"immutable": true,
|
|
7909
7909
|
"locationInModule": {
|
|
7910
7910
|
"filename": "lib/job-executable.ts",
|
|
7911
|
-
"line":
|
|
7911
|
+
"line": 485
|
|
7912
7912
|
},
|
|
7913
7913
|
"name": "extraJarsFirst",
|
|
7914
7914
|
"optional": true,
|
|
@@ -7927,7 +7927,7 @@
|
|
|
7927
7927
|
"immutable": true,
|
|
7928
7928
|
"locationInModule": {
|
|
7929
7929
|
"filename": "lib/job-executable.ts",
|
|
7930
|
-
"line":
|
|
7930
|
+
"line": 467
|
|
7931
7931
|
},
|
|
7932
7932
|
"name": "extraPythonFiles",
|
|
7933
7933
|
"optional": true,
|
|
@@ -7950,7 +7950,7 @@
|
|
|
7950
7950
|
"immutable": true,
|
|
7951
7951
|
"locationInModule": {
|
|
7952
7952
|
"filename": "lib/job-executable.ts",
|
|
7953
|
-
"line":
|
|
7953
|
+
"line": 428
|
|
7954
7954
|
},
|
|
7955
7955
|
"name": "pythonVersion",
|
|
7956
7956
|
"optional": true,
|
|
@@ -7968,7 +7968,7 @@
|
|
|
7968
7968
|
"immutable": true,
|
|
7969
7969
|
"locationInModule": {
|
|
7970
7970
|
"filename": "lib/job-executable.ts",
|
|
7971
|
-
"line":
|
|
7971
|
+
"line": 435
|
|
7972
7972
|
},
|
|
7973
7973
|
"name": "runtime",
|
|
7974
7974
|
"optional": true,
|
|
@@ -8016,7 +8016,7 @@
|
|
|
8016
8016
|
"docs": {
|
|
8017
8017
|
"stability": "experimental",
|
|
8018
8018
|
"summary": "Construction properties for `Job`.",
|
|
8019
|
-
"example": "
|
|
8019
|
+
"example": "new glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});",
|
|
8020
8020
|
"custom": {
|
|
8021
8021
|
"exampleMetadata": "infused"
|
|
8022
8022
|
}
|
|
@@ -8025,7 +8025,7 @@
|
|
|
8025
8025
|
"kind": "interface",
|
|
8026
8026
|
"locationInModule": {
|
|
8027
8027
|
"filename": "lib/job.ts",
|
|
8028
|
-
"line":
|
|
8028
|
+
"line": 483
|
|
8029
8029
|
},
|
|
8030
8030
|
"name": "JobProps",
|
|
8031
8031
|
"properties": [
|
|
@@ -8038,7 +8038,7 @@
|
|
|
8038
8038
|
"immutable": true,
|
|
8039
8039
|
"locationInModule": {
|
|
8040
8040
|
"filename": "lib/job.ts",
|
|
8041
|
-
"line":
|
|
8041
|
+
"line": 487
|
|
8042
8042
|
},
|
|
8043
8043
|
"name": "executable",
|
|
8044
8044
|
"type": {
|
|
@@ -8056,7 +8056,7 @@
|
|
|
8056
8056
|
"immutable": true,
|
|
8057
8057
|
"locationInModule": {
|
|
8058
8058
|
"filename": "lib/job.ts",
|
|
8059
|
-
"line":
|
|
8059
|
+
"line": 562
|
|
8060
8060
|
},
|
|
8061
8061
|
"name": "connections",
|
|
8062
8062
|
"optional": true,
|
|
@@ -8080,7 +8080,7 @@
|
|
|
8080
8080
|
"immutable": true,
|
|
8081
8081
|
"locationInModule": {
|
|
8082
8082
|
"filename": "lib/job.ts",
|
|
8083
|
-
"line":
|
|
8083
|
+
"line": 624
|
|
8084
8084
|
},
|
|
8085
8085
|
"name": "continuousLogging",
|
|
8086
8086
|
"optional": true,
|
|
@@ -8099,7 +8099,7 @@
|
|
|
8099
8099
|
"immutable": true,
|
|
8100
8100
|
"locationInModule": {
|
|
8101
8101
|
"filename": "lib/job.ts",
|
|
8102
|
-
"line":
|
|
8102
|
+
"line": 577
|
|
8103
8103
|
},
|
|
8104
8104
|
"name": "defaultArguments",
|
|
8105
8105
|
"optional": true,
|
|
@@ -8122,7 +8122,7 @@
|
|
|
8122
8122
|
"immutable": true,
|
|
8123
8123
|
"locationInModule": {
|
|
8124
8124
|
"filename": "lib/job.ts",
|
|
8125
|
-
"line":
|
|
8125
|
+
"line": 501
|
|
8126
8126
|
},
|
|
8127
8127
|
"name": "description",
|
|
8128
8128
|
"optional": true,
|
|
@@ -8141,7 +8141,7 @@
|
|
|
8141
8141
|
"immutable": true,
|
|
8142
8142
|
"locationInModule": {
|
|
8143
8143
|
"filename": "lib/job.ts",
|
|
8144
|
-
"line":
|
|
8144
|
+
"line": 604
|
|
8145
8145
|
},
|
|
8146
8146
|
"name": "enableProfilingMetrics",
|
|
8147
8147
|
"optional": true,
|
|
@@ -8160,7 +8160,7 @@
|
|
|
8160
8160
|
"immutable": true,
|
|
8161
8161
|
"locationInModule": {
|
|
8162
8162
|
"filename": "lib/job.ts",
|
|
8163
|
-
"line":
|
|
8163
|
+
"line": 634
|
|
8164
8164
|
},
|
|
8165
8165
|
"name": "executionClass",
|
|
8166
8166
|
"optional": true,
|
|
@@ -8178,7 +8178,7 @@
|
|
|
8178
8178
|
"immutable": true,
|
|
8179
8179
|
"locationInModule": {
|
|
8180
8180
|
"filename": "lib/job.ts",
|
|
8181
|
-
"line":
|
|
8181
|
+
"line": 494
|
|
8182
8182
|
},
|
|
8183
8183
|
"name": "jobName",
|
|
8184
8184
|
"optional": true,
|
|
@@ -8197,7 +8197,7 @@
|
|
|
8197
8197
|
"immutable": true,
|
|
8198
8198
|
"locationInModule": {
|
|
8199
8199
|
"filename": "lib/job.ts",
|
|
8200
|
-
"line":
|
|
8200
|
+
"line": 509
|
|
8201
8201
|
},
|
|
8202
8202
|
"name": "maxCapacity",
|
|
8203
8203
|
"optional": true,
|
|
@@ -8216,7 +8216,7 @@
|
|
|
8216
8216
|
"immutable": true,
|
|
8217
8217
|
"locationInModule": {
|
|
8218
8218
|
"filename": "lib/job.ts",
|
|
8219
|
-
"line":
|
|
8219
|
+
"line": 525
|
|
8220
8220
|
},
|
|
8221
8221
|
"name": "maxConcurrentRuns",
|
|
8222
8222
|
"optional": true,
|
|
@@ -8234,7 +8234,7 @@
|
|
|
8234
8234
|
"immutable": true,
|
|
8235
8235
|
"locationInModule": {
|
|
8236
8236
|
"filename": "lib/job.ts",
|
|
8237
|
-
"line":
|
|
8237
|
+
"line": 516
|
|
8238
8238
|
},
|
|
8239
8239
|
"name": "maxRetries",
|
|
8240
8240
|
"optional": true,
|
|
@@ -8252,7 +8252,7 @@
|
|
|
8252
8252
|
"immutable": true,
|
|
8253
8253
|
"locationInModule": {
|
|
8254
8254
|
"filename": "lib/job.ts",
|
|
8255
|
-
"line":
|
|
8255
|
+
"line": 532
|
|
8256
8256
|
},
|
|
8257
8257
|
"name": "notifyDelayAfter",
|
|
8258
8258
|
"optional": true,
|
|
@@ -8272,7 +8272,7 @@
|
|
|
8272
8272
|
"immutable": true,
|
|
8273
8273
|
"locationInModule": {
|
|
8274
8274
|
"filename": "lib/job.ts",
|
|
8275
|
-
"line":
|
|
8275
|
+
"line": 595
|
|
8276
8276
|
},
|
|
8277
8277
|
"name": "role",
|
|
8278
8278
|
"optional": true,
|
|
@@ -8290,7 +8290,7 @@
|
|
|
8290
8290
|
"immutable": true,
|
|
8291
8291
|
"locationInModule": {
|
|
8292
8292
|
"filename": "lib/job.ts",
|
|
8293
|
-
"line":
|
|
8293
|
+
"line": 569
|
|
8294
8294
|
},
|
|
8295
8295
|
"name": "securityConfiguration",
|
|
8296
8296
|
"optional": true,
|
|
@@ -8309,7 +8309,7 @@
|
|
|
8309
8309
|
"immutable": true,
|
|
8310
8310
|
"locationInModule": {
|
|
8311
8311
|
"filename": "lib/job.ts",
|
|
8312
|
-
"line":
|
|
8312
|
+
"line": 614
|
|
8313
8313
|
},
|
|
8314
8314
|
"name": "sparkUI",
|
|
8315
8315
|
"optional": true,
|
|
@@ -8327,7 +8327,7 @@
|
|
|
8327
8327
|
"immutable": true,
|
|
8328
8328
|
"locationInModule": {
|
|
8329
8329
|
"filename": "lib/job.ts",
|
|
8330
|
-
"line":
|
|
8330
|
+
"line": 584
|
|
8331
8331
|
},
|
|
8332
8332
|
"name": "tags",
|
|
8333
8333
|
"optional": true,
|
|
@@ -8350,7 +8350,7 @@
|
|
|
8350
8350
|
"immutable": true,
|
|
8351
8351
|
"locationInModule": {
|
|
8352
8352
|
"filename": "lib/job.ts",
|
|
8353
|
-
"line":
|
|
8353
|
+
"line": 539
|
|
8354
8354
|
},
|
|
8355
8355
|
"name": "timeout",
|
|
8356
8356
|
"optional": true,
|
|
@@ -8368,7 +8368,7 @@
|
|
|
8368
8368
|
"immutable": true,
|
|
8369
8369
|
"locationInModule": {
|
|
8370
8370
|
"filename": "lib/job.ts",
|
|
8371
|
-
"line":
|
|
8371
|
+
"line": 553
|
|
8372
8372
|
},
|
|
8373
8373
|
"name": "workerCount",
|
|
8374
8374
|
"optional": true,
|
|
@@ -8386,7 +8386,7 @@
|
|
|
8386
8386
|
"immutable": true,
|
|
8387
8387
|
"locationInModule": {
|
|
8388
8388
|
"filename": "lib/job.ts",
|
|
8389
|
-
"line":
|
|
8389
|
+
"line": 546
|
|
8390
8390
|
},
|
|
8391
8391
|
"name": "workerType",
|
|
8392
8392
|
"optional": true,
|
|
@@ -8408,7 +8408,7 @@
|
|
|
8408
8408
|
"kind": "enum",
|
|
8409
8409
|
"locationInModule": {
|
|
8410
8410
|
"filename": "lib/job.ts",
|
|
8411
|
-
"line":
|
|
8411
|
+
"line": 79
|
|
8412
8412
|
},
|
|
8413
8413
|
"members": [
|
|
8414
8414
|
{
|
|
@@ -8610,7 +8610,7 @@
|
|
|
8610
8610
|
"kind": "enum",
|
|
8611
8611
|
"locationInModule": {
|
|
8612
8612
|
"filename": "lib/job.ts",
|
|
8613
|
-
"line":
|
|
8613
|
+
"line": 121
|
|
8614
8614
|
},
|
|
8615
8615
|
"members": [
|
|
8616
8616
|
{
|
|
@@ -9360,7 +9360,7 @@
|
|
|
9360
9360
|
"docs": {
|
|
9361
9361
|
"stability": "experimental",
|
|
9362
9362
|
"summary": "Python version.",
|
|
9363
|
-
"example": "new glue.Job(this, '
|
|
9363
|
+
"example": "new glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});",
|
|
9364
9364
|
"custom": {
|
|
9365
9365
|
"exampleMetadata": "infused"
|
|
9366
9366
|
}
|
|
@@ -10716,7 +10716,7 @@
|
|
|
10716
10716
|
"kind": "interface",
|
|
10717
10717
|
"locationInModule": {
|
|
10718
10718
|
"filename": "lib/job.ts",
|
|
10719
|
-
"line":
|
|
10719
|
+
"line": 406
|
|
10720
10720
|
},
|
|
10721
10721
|
"name": "SparkUILoggingLocation",
|
|
10722
10722
|
"properties": [
|
|
@@ -10729,7 +10729,7 @@
|
|
|
10729
10729
|
"immutable": true,
|
|
10730
10730
|
"locationInModule": {
|
|
10731
10731
|
"filename": "lib/job.ts",
|
|
10732
|
-
"line":
|
|
10732
|
+
"line": 410
|
|
10733
10733
|
},
|
|
10734
10734
|
"name": "bucket",
|
|
10735
10735
|
"type": {
|
|
@@ -10746,7 +10746,7 @@
|
|
|
10746
10746
|
"immutable": true,
|
|
10747
10747
|
"locationInModule": {
|
|
10748
10748
|
"filename": "lib/job.ts",
|
|
10749
|
-
"line":
|
|
10749
|
+
"line": 417
|
|
10750
10750
|
},
|
|
10751
10751
|
"name": "prefix",
|
|
10752
10752
|
"optional": true,
|
|
@@ -10764,16 +10764,16 @@
|
|
|
10764
10764
|
"see": "https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html",
|
|
10765
10765
|
"stability": "experimental",
|
|
10766
10766
|
"summary": "Properties for enabling Spark UI monitoring feature for Spark-based Glue jobs.",
|
|
10767
|
-
"example": "
|
|
10767
|
+
"example": "new glue.Job(this, 'EnableSparkUI', {\n jobName: 'EtlJobWithSparkUIPrefix',\n sparkUI: {\n enabled: true,\n },\n executable: glue.JobExecutable.pythonEtl({\n glueVersion: glue.GlueVersion.V3_0,\n pythonVersion: glue.PythonVersion.THREE,\n script: glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.py')),\n }),\n});",
|
|
10768
10768
|
"custom": {
|
|
10769
|
-
"exampleMetadata": "
|
|
10769
|
+
"exampleMetadata": "infused"
|
|
10770
10770
|
}
|
|
10771
10771
|
},
|
|
10772
10772
|
"fqn": "@aws-cdk/aws-glue-alpha.SparkUIProps",
|
|
10773
10773
|
"kind": "interface",
|
|
10774
10774
|
"locationInModule": {
|
|
10775
10775
|
"filename": "lib/job.ts",
|
|
10776
|
-
"line":
|
|
10776
|
+
"line": 378
|
|
10777
10777
|
},
|
|
10778
10778
|
"name": "SparkUIProps",
|
|
10779
10779
|
"properties": [
|
|
@@ -10786,7 +10786,7 @@
|
|
|
10786
10786
|
"immutable": true,
|
|
10787
10787
|
"locationInModule": {
|
|
10788
10788
|
"filename": "lib/job.ts",
|
|
10789
|
-
"line":
|
|
10789
|
+
"line": 382
|
|
10790
10790
|
},
|
|
10791
10791
|
"name": "enabled",
|
|
10792
10792
|
"type": {
|
|
@@ -10803,7 +10803,7 @@
|
|
|
10803
10803
|
"immutable": true,
|
|
10804
10804
|
"locationInModule": {
|
|
10805
10805
|
"filename": "lib/job.ts",
|
|
10806
|
-
"line":
|
|
10806
|
+
"line": 389
|
|
10807
10807
|
},
|
|
10808
10808
|
"name": "bucket",
|
|
10809
10809
|
"optional": true,
|
|
@@ -10814,14 +10814,15 @@
|
|
|
10814
10814
|
{
|
|
10815
10815
|
"abstract": true,
|
|
10816
10816
|
"docs": {
|
|
10817
|
-
"default": "
|
|
10817
|
+
"default": "- the logs will be written at the root of the bucket",
|
|
10818
|
+
"remarks": "Use format `'/foo/bar'`",
|
|
10818
10819
|
"stability": "experimental",
|
|
10819
10820
|
"summary": "The path inside the bucket (objects prefix) where the Glue job stores the logs."
|
|
10820
10821
|
},
|
|
10821
10822
|
"immutable": true,
|
|
10822
10823
|
"locationInModule": {
|
|
10823
10824
|
"filename": "lib/job.ts",
|
|
10824
|
-
"line":
|
|
10825
|
+
"line": 397
|
|
10825
10826
|
},
|
|
10826
10827
|
"name": "prefix",
|
|
10827
10828
|
"optional": true,
|
|
@@ -12591,7 +12592,7 @@
|
|
|
12591
12592
|
"kind": "class",
|
|
12592
12593
|
"locationInModule": {
|
|
12593
12594
|
"filename": "lib/job.ts",
|
|
12594
|
-
"line":
|
|
12595
|
+
"line": 20
|
|
12595
12596
|
},
|
|
12596
12597
|
"methods": [
|
|
12597
12598
|
{
|
|
@@ -12601,7 +12602,7 @@
|
|
|
12601
12602
|
},
|
|
12602
12603
|
"locationInModule": {
|
|
12603
12604
|
"filename": "lib/job.ts",
|
|
12604
|
-
"line":
|
|
12605
|
+
"line": 60
|
|
12605
12606
|
},
|
|
12606
12607
|
"name": "of",
|
|
12607
12608
|
"parameters": [
|
|
@@ -12634,7 +12635,7 @@
|
|
|
12634
12635
|
"immutable": true,
|
|
12635
12636
|
"locationInModule": {
|
|
12636
12637
|
"filename": "lib/job.ts",
|
|
12637
|
-
"line":
|
|
12638
|
+
"line": 49
|
|
12638
12639
|
},
|
|
12639
12640
|
"name": "G_025X",
|
|
12640
12641
|
"static": true,
|
|
@@ -12652,7 +12653,7 @@
|
|
|
12652
12653
|
"immutable": true,
|
|
12653
12654
|
"locationInModule": {
|
|
12654
12655
|
"filename": "lib/job.ts",
|
|
12655
|
-
"line":
|
|
12656
|
+
"line": 29
|
|
12656
12657
|
},
|
|
12657
12658
|
"name": "G_1X",
|
|
12658
12659
|
"static": true,
|
|
@@ -12670,7 +12671,7 @@
|
|
|
12670
12671
|
"immutable": true,
|
|
12671
12672
|
"locationInModule": {
|
|
12672
12673
|
"filename": "lib/job.ts",
|
|
12673
|
-
"line":
|
|
12674
|
+
"line": 34
|
|
12674
12675
|
},
|
|
12675
12676
|
"name": "G_2X",
|
|
12676
12677
|
"static": true,
|
|
@@ -12688,7 +12689,7 @@
|
|
|
12688
12689
|
"immutable": true,
|
|
12689
12690
|
"locationInModule": {
|
|
12690
12691
|
"filename": "lib/job.ts",
|
|
12691
|
-
"line":
|
|
12692
|
+
"line": 39
|
|
12692
12693
|
},
|
|
12693
12694
|
"name": "G_4X",
|
|
12694
12695
|
"static": true,
|
|
@@ -12706,7 +12707,7 @@
|
|
|
12706
12707
|
"immutable": true,
|
|
12707
12708
|
"locationInModule": {
|
|
12708
12709
|
"filename": "lib/job.ts",
|
|
12709
|
-
"line":
|
|
12710
|
+
"line": 44
|
|
12710
12711
|
},
|
|
12711
12712
|
"name": "G_8X",
|
|
12712
12713
|
"static": true,
|
|
@@ -12723,7 +12724,7 @@
|
|
|
12723
12724
|
"immutable": true,
|
|
12724
12725
|
"locationInModule": {
|
|
12725
12726
|
"filename": "lib/job.ts",
|
|
12726
|
-
"line":
|
|
12727
|
+
"line": 24
|
|
12727
12728
|
},
|
|
12728
12729
|
"name": "STANDARD",
|
|
12729
12730
|
"static": true,
|
|
@@ -12741,7 +12742,7 @@
|
|
|
12741
12742
|
"immutable": true,
|
|
12742
12743
|
"locationInModule": {
|
|
12743
12744
|
"filename": "lib/job.ts",
|
|
12744
|
-
"line":
|
|
12745
|
+
"line": 54
|
|
12745
12746
|
},
|
|
12746
12747
|
"name": "Z_2X",
|
|
12747
12748
|
"static": true,
|
|
@@ -12757,7 +12758,7 @@
|
|
|
12757
12758
|
"immutable": true,
|
|
12758
12759
|
"locationInModule": {
|
|
12759
12760
|
"filename": "lib/job.ts",
|
|
12760
|
-
"line":
|
|
12761
|
+
"line": 67
|
|
12761
12762
|
},
|
|
12762
12763
|
"name": "name",
|
|
12763
12764
|
"type": {
|
|
@@ -12801,6 +12802,6 @@
|
|
|
12801
12802
|
"symbolId": "lib/storage-parameter:WriteParallel"
|
|
12802
12803
|
}
|
|
12803
12804
|
},
|
|
12804
|
-
"version": "2.
|
|
12805
|
+
"version": "2.91.0-alpha.0",
|
|
12805
12806
|
"fingerprint": "**********"
|
|
12806
12807
|
}
|