AWSGlueDataplanePython 0.0.2__tar.gz → 5.0.0__tar.gz

awsgluedataplanepython-5.0.0/.github/workflows/publish.yml

@@ -0,0 +1,34 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install build
+      - run: python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

awsgluedataplanepython-5.0.0/.gitignore

@@ -0,0 +1,8 @@
+*.pyc
+PyGlue.zip
+conf/
+jars/
+/jarsv1/
+derby.log
+metastore_db/
+*.DS_Store

awsgluedataplanepython-5.0.0/AWSGlueDataplanePython.egg-info/PKG-INFO

@@ -0,0 +1,178 @@
+Metadata-Version: 2.4
+Name: AWSGlueDataplanePython
+Version: 5.0.0
+Summary: AWS Glue Python library for local ETL script development
+License: Amazon Software License 1.0
+        
+        This Amazon Software License ("License") governs your use, reproduction, and
+        distribution of the accompanying software as specified below.
+        
+        1. Definitions
+        
+          "Licensor" means any person or entity that distributes its Work.
+        
+          "Software" means the original work of authorship made available under this
+          License.
+        
+          "Work" means the Software and any additions to or derivative works of the
+          Software that are made available under this License.
+        
+          The terms "reproduce," "reproduction," "derivative works," and
+          "distribution" have the meaning as provided under U.S. copyright law;
+          provided, however, that for the purposes of this License, derivative works
+          shall not include works that remain separable from, or merely link (or bind
+          by name) to the interfaces of, the Work.
+        
+          Works, including the Software, are "made available" under this License by
+          including in or with the Work either (a) a copyright notice referencing the
+          applicability of this License to the Work, or (b) a copy of this License.
+        
+        2. License Grants
+        
+          2.1 Copyright Grant. Subject to the terms and conditions of this License,
+          each Licensor grants to you a perpetual, worldwide, non-exclusive,
+          royalty-free, copyright license to reproduce, prepare derivative works of,
+          publicly display, publicly perform, sublicense and distribute its Work and
+          any resulting derivative works in any form.
+        
+          2.2 Patent Grant. Subject to the terms and conditions of this License, each
+          Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free
+          patent license to make, have made, use, sell, offer for sale, import, and
+          otherwise transfer its Work, in whole or in part. The foregoing license
+          applies only to the patent claims licensable by Licensor that would be
+          infringed by Licensor's Work (or portion thereof) individually and
+          excluding any combinations with any other materials or technology.
+        
+        3. Limitations
+        
+          3.1 Redistribution. You may reproduce or distribute the Work only if
+          (a) you do so under this License, (b) you include a complete copy of this
+          License with your distribution, and (c) you retain without modification
+          any copyright, patent, trademark, or attribution notices that are present
+          in the Work.
+        
+          3.2 Derivative Works. You may specify that additional or different terms
+          apply to the use, reproduction, and distribution of your derivative works
+          of the Work ("Your Terms") only if (a) Your Terms provide that the use
+          limitation in Section 3.3 applies to your derivative works, and (b) you
+          identify the specific derivative works that are subject to Your Terms.
+          Notwithstanding Your Terms, this License (including the redistribution
+          requirements in Section 3.1) will continue to apply to the Work itself.
+        
+          3.3 Use Limitation. The Work and any derivative works thereof only may be
+          used or intended for use with the web services, computing platforms or
+          applications provided by Amazon.com, Inc. or its affiliates, including
+          Amazon Web Services, Inc.
+        
+          3.4 Patent Claims. If you bring or threaten to bring a patent claim against
+          any Licensor (including any claim, cross-claim or counterclaim in a
+          lawsuit) to enforce any patents that you allege are infringed by any Work,
+          then your rights under this License from such Licensor (including the
+          grants in Sections 2.1 and 2.2) will terminate immediately.
+        
+          3.5 Trademarks. This License does not grant any rights to use any
+          Licensor's or its affiliates' names, logos, or trademarks, except as
+          necessary to reproduce the notices described in this License.
+        
+          3.6 Termination. If you violate any term of this License, then your rights
+          under this License (including the grants in Sections 2.1 and 2.2) will
+          terminate immediately.
+        
+        4. Disclaimer of Warranty.
+        
+          THE WORK IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+          EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF
+          MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR
+          NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER
+          THIS LICENSE. SOME STATES' CONSUMER LAWS DO NOT ALLOW EXCLUSION OF AN
+          IMPLIED WARRANTY, SO THIS DISCLAIMER MAY NOT APPLY TO YOU.
+        
+        5. Limitation of Liability.
+        
+          EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL
+          THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE
+          SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT,
+          INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR
+          RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING
+          BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS
+          OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER COMM ERCIAL DAMAGES
+          OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF
+          SUCH DAMAGES.
+        
+Requires-Python: <3.12,>=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+License-File: NOTICE.txt
+Dynamic: license-file
+
+# aws-glue-libs
+
+This repository supports python libraries for local development of glue pyspark batch jobs. Glue streaming is supported in the separate repository [aws-glue-streaming-libs](https://github.com/awslabs/aws-glue-streaming-libs).
+
+## Contents
+This repository contains:
+ * `awsglue` - the Python libary you can use to author [AWS Glue](https://aws.amazon.com/glue) ETL job. This library extends [Apache Spark](https://spark.apache.org/) with additional data types and operations for ETL workflows. It's an interface for Glue ETL library in Python.
+ * `bin` - this directory hosts several executables that allow you to run the Python library locally or open up a PySpark shell to run Glue Spark code interactively.
+
+## Python versions by Glue Version
+
+Different Glue versions support different Python versions. The following table below is for your reference, which also includes the associated repository's branch for each glue version.
+
+| Glue Version  | Python 3 Version  | aws-glue-libs branch |
+|---|---|----------------------|
+| 2.0  | 3.7  | glue-2.0             |
+| 3.0  | 3.7  | glue-3.0             |
+| 4.0  | 3.10 | glue-4.0             |
+| 5.0  | 3.11 | glue-5.0             |
+
+You may refer to AWS Glue's official [release notes](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) for more information
+
+## Setup guide
+
+If you haven't already, please refer to the [official AWS Glue Python local development documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python) for the official setup documentation. The following is a summary of the AWS documentation:
+
+The `awsglue` library provides only the Python interface to the Glue Spark runtime, you need the Glue ETL jar to run it locally. The jar is now available via the maven build system in a s3 backed maven repository. Here are the steps to set up your dev environment locally.
+
+1. install Apache Maven from the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
+1. use `copy-dependencies` target in Apache Maven to download the jar from S3 to your local dev environment.
+1. download and extract the Apache Spark distribution based on the Glue version you're using:
+   * Glue version 2.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-2.0/spark-2.4.3-bin-hadoop2.8.tgz1`
+   * Glue version 3.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-3.0/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz`
+   * Glue version 4.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-4.0/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0.tgz` 
+   * Glue version 5.0: download the Apache Spark 3.5.4 distribution from `https://spark.apache.org`
+1. export the `SPARK_HOME` environmental variable to the extracted location of the above Spark distribution. For example:
+    ```
+    Glue version 2.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8
+    Glue version 3.0: export SPARK_HOME=/home/$USER/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3
+    Glue version 4.0: export SPARK_HOME=/home/$USER/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0
+    Glue version 5.0: export SPARK_HOME=/home/$USER/spark-3.5.4-bin-hadoop3
+    ```
+1. now you can run the executables in the `bin` directory to start a Glue Shell or submit a Glue Spark application.
+    ```
+    Glue shell: ./bin/gluepyspark
+    Glue submit: ./bin/gluesparksubmit
+    pytest: ./bin/gluepytest
+    ```
+(The `gluepytest` script assumes that the pytest module is installed and available in the `PATH` env variable)
+
+## Licensing
+
+The libraries in this repository licensed under the [Amazon Software License](http://aws.amazon.com/asl/) (the "License"). They may not be used except in compliance with the License, a copy of which is included here in the LICENSE file.
+
+---
+
+# Release Notes
+
+## July 26 2023
+* According to [AWS Glue version support policy](https://docs.aws.amazon.com/glue/latest/dg/glue-version-support-policy.html), branches for Glue 0.9 and 1.0 are removed as they are already deprecated.
+
+
+## August 27 2021
+* The master branch has been modified from representing Glue 0.9 to Glue 3.0, we have also created a glue-0.9 branch to reflect the former state of the master branch with Glue 0.9. To rename your local clone of the older master branch and point to the glue-0.9 branch, you may use the following commands:
+```
+git branch -m master glue-0.9
+git fetch origin
+git branch -u origin/glue-0.9 glue-0.9
+git remote set-head origin -a
+```
+

awsgluedataplanepython-5.0.0/AWSGlueDataplanePython.egg-info/SOURCES.txt

@@ -0,0 +1,57 @@
+.gitignore
+LICENSE.txt
+NOTICE.txt
+README.md
+THIRD-PARTY-LICENSES
+pom.xml
+pyproject.toml
+setup.py
+.github/workflows/publish.yml
+AWSGlueDataplanePython.egg-info/PKG-INFO
+AWSGlueDataplanePython.egg-info/SOURCES.txt
+AWSGlueDataplanePython.egg-info/dependency_links.txt
+AWSGlueDataplanePython.egg-info/not-zip-safe
+AWSGlueDataplanePython.egg-info/top_level.txt
+awsglue/README.md
+awsglue/__init__.py
+awsglue/context.py
+awsglue/data_sink.py
+awsglue/data_source.py
+awsglue/dataframereader.py
+awsglue/dataframewriter.py
+awsglue/devutils.py
+awsglue/dynamicframe.py
+awsglue/functions.py
+awsglue/glue_shell.py
+awsglue/gluetypes.py
+awsglue/job.py
+awsglue/streaming_data_source.py
+awsglue/utils.py
+awsglue/dataframe_transforms/__init__.py
+awsglue/dataframe_transforms/apply_mapping.py
+awsglue/scripts/__init__.py
+awsglue/scripts/activate_etl_connector.py
+awsglue/scripts/connector_activation_util.py
+awsglue/scripts/crawler_redo_from_backup.py
+awsglue/scripts/crawler_undo.py
+awsglue/scripts/scripts_utils.py
+awsglue/transforms/__init__.py
+awsglue/transforms/apply_mapping.py
+awsglue/transforms/coalesce.py
+awsglue/transforms/collection_transforms.py
+awsglue/transforms/drop_nulls.py
+awsglue/transforms/dynamicframe_filter.py
+awsglue/transforms/dynamicframe_map.py
+awsglue/transforms/errors_as_dynamicframe.py
+awsglue/transforms/field_transforms.py
+awsglue/transforms/relationalize.py
+awsglue/transforms/repartition.py
+awsglue/transforms/resolve_choice.py
+awsglue/transforms/transform.py
+awsglue/transforms/unbox.py
+awsglue/transforms/union.py
+awsglue/transforms/unnest_frame.py
+bin/glue-setup.sh
+bin/gluepyspark
+bin/gluepytest
+bin/gluesparksubmit

awsgluedataplanepython-5.0.0/AWSGlueDataplanePython.egg-info/top_level.txt

@@ -0,0 +1 @@
+awsglue

awsgluedataplanepython-5.0.0/LICENSE.txt

@@ -0,0 +1,96 @@
+Amazon Software License 1.0
+
+This Amazon Software License ("License") governs your use, reproduction, and
+distribution of the accompanying software as specified below.
+
+1. Definitions
+
+  "Licensor" means any person or entity that distributes its Work.
+
+  "Software" means the original work of authorship made available under this
+  License.
+
+  "Work" means the Software and any additions to or derivative works of the
+  Software that are made available under this License.
+
+  The terms "reproduce," "reproduction," "derivative works," and
+  "distribution" have the meaning as provided under U.S. copyright law;
+  provided, however, that for the purposes of this License, derivative works
+  shall not include works that remain separable from, or merely link (or bind
+  by name) to the interfaces of, the Work.
+
+  Works, including the Software, are "made available" under this License by
+  including in or with the Work either (a) a copyright notice referencing the
+  applicability of this License to the Work, or (b) a copy of this License.
+
+2. License Grants
+
+  2.1 Copyright Grant. Subject to the terms and conditions of this License,
+  each Licensor grants to you a perpetual, worldwide, non-exclusive,
+  royalty-free, copyright license to reproduce, prepare derivative works of,
+  publicly display, publicly perform, sublicense and distribute its Work and
+  any resulting derivative works in any form.
+
+  2.2 Patent Grant. Subject to the terms and conditions of this License, each
+  Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free
+  patent license to make, have made, use, sell, offer for sale, import, and
+  otherwise transfer its Work, in whole or in part. The foregoing license
+  applies only to the patent claims licensable by Licensor that would be
+  infringed by Licensor's Work (or portion thereof) individually and
+  excluding any combinations with any other materials or technology.
+
+3. Limitations
+
+  3.1 Redistribution. You may reproduce or distribute the Work only if
+  (a) you do so under this License, (b) you include a complete copy of this
+  License with your distribution, and (c) you retain without modification
+  any copyright, patent, trademark, or attribution notices that are present
+  in the Work.
+
+  3.2 Derivative Works. You may specify that additional or different terms
+  apply to the use, reproduction, and distribution of your derivative works
+  of the Work ("Your Terms") only if (a) Your Terms provide that the use
+  limitation in Section 3.3 applies to your derivative works, and (b) you
+  identify the specific derivative works that are subject to Your Terms.
+  Notwithstanding Your Terms, this License (including the redistribution
+  requirements in Section 3.1) will continue to apply to the Work itself.
+
+  3.3 Use Limitation. The Work and any derivative works thereof only may be
+  used or intended for use with the web services, computing platforms or
+  applications provided by Amazon.com, Inc. or its affiliates, including
+  Amazon Web Services, Inc.
+
+  3.4 Patent Claims. If you bring or threaten to bring a patent claim against
+  any Licensor (including any claim, cross-claim or counterclaim in a
+  lawsuit) to enforce any patents that you allege are infringed by any Work,
+  then your rights under this License from such Licensor (including the
+  grants in Sections 2.1 and 2.2) will terminate immediately.
+
+  3.5 Trademarks. This License does not grant any rights to use any
+  Licensor's or its affiliates' names, logos, or trademarks, except as
+  necessary to reproduce the notices described in this License.
+
+  3.6 Termination. If you violate any term of this License, then your rights
+  under this License (including the grants in Sections 2.1 and 2.2) will
+  terminate immediately.
+
+4. Disclaimer of Warranty.
+
+  THE WORK IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+  EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF
+  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR
+  NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER
+  THIS LICENSE. SOME STATES' CONSUMER LAWS DO NOT ALLOW EXCLUSION OF AN
+  IMPLIED WARRANTY, SO THIS DISCLAIMER MAY NOT APPLY TO YOU.
+
+5. Limitation of Liability.
+
+  EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL
+  THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE
+  SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT,
+  INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR
+  RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING
+  BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS
+  OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER COMM ERCIAL DAMAGES
+  OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF
+  SUCH DAMAGES.

awsgluedataplanepython-5.0.0/NOTICE.txt

@@ -0,0 +1,3 @@
+aws-glue-libs
+Copyright 2016-2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+

awsgluedataplanepython-5.0.0/PKG-INFO

@@ -0,0 +1,178 @@
+Metadata-Version: 2.4
+Name: AWSGlueDataplanePython
+Version: 5.0.0
+Summary: AWS Glue Python library for local ETL script development
+License: Amazon Software License 1.0
+        
+        This Amazon Software License ("License") governs your use, reproduction, and
+        distribution of the accompanying software as specified below.
+        
+        1. Definitions
+        
+          "Licensor" means any person or entity that distributes its Work.
+        
+          "Software" means the original work of authorship made available under this
+          License.
+        
+          "Work" means the Software and any additions to or derivative works of the
+          Software that are made available under this License.
+        
+          The terms "reproduce," "reproduction," "derivative works," and
+          "distribution" have the meaning as provided under U.S. copyright law;
+          provided, however, that for the purposes of this License, derivative works
+          shall not include works that remain separable from, or merely link (or bind
+          by name) to the interfaces of, the Work.
+        
+          Works, including the Software, are "made available" under this License by
+          including in or with the Work either (a) a copyright notice referencing the
+          applicability of this License to the Work, or (b) a copy of this License.
+        
+        2. License Grants
+        
+          2.1 Copyright Grant. Subject to the terms and conditions of this License,
+          each Licensor grants to you a perpetual, worldwide, non-exclusive,
+          royalty-free, copyright license to reproduce, prepare derivative works of,
+          publicly display, publicly perform, sublicense and distribute its Work and
+          any resulting derivative works in any form.
+        
+          2.2 Patent Grant. Subject to the terms and conditions of this License, each
+          Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free
+          patent license to make, have made, use, sell, offer for sale, import, and
+          otherwise transfer its Work, in whole or in part. The foregoing license
+          applies only to the patent claims licensable by Licensor that would be
+          infringed by Licensor's Work (or portion thereof) individually and
+          excluding any combinations with any other materials or technology.
+        
+        3. Limitations
+        
+          3.1 Redistribution. You may reproduce or distribute the Work only if
+          (a) you do so under this License, (b) you include a complete copy of this
+          License with your distribution, and (c) you retain without modification
+          any copyright, patent, trademark, or attribution notices that are present
+          in the Work.
+        
+          3.2 Derivative Works. You may specify that additional or different terms
+          apply to the use, reproduction, and distribution of your derivative works
+          of the Work ("Your Terms") only if (a) Your Terms provide that the use
+          limitation in Section 3.3 applies to your derivative works, and (b) you
+          identify the specific derivative works that are subject to Your Terms.
+          Notwithstanding Your Terms, this License (including the redistribution
+          requirements in Section 3.1) will continue to apply to the Work itself.
+        
+          3.3 Use Limitation. The Work and any derivative works thereof only may be
+          used or intended for use with the web services, computing platforms or
+          applications provided by Amazon.com, Inc. or its affiliates, including
+          Amazon Web Services, Inc.
+        
+          3.4 Patent Claims. If you bring or threaten to bring a patent claim against
+          any Licensor (including any claim, cross-claim or counterclaim in a
+          lawsuit) to enforce any patents that you allege are infringed by any Work,
+          then your rights under this License from such Licensor (including the
+          grants in Sections 2.1 and 2.2) will terminate immediately.
+        
+          3.5 Trademarks. This License does not grant any rights to use any
+          Licensor's or its affiliates' names, logos, or trademarks, except as
+          necessary to reproduce the notices described in this License.
+        
+          3.6 Termination. If you violate any term of this License, then your rights
+          under this License (including the grants in Sections 2.1 and 2.2) will
+          terminate immediately.
+        
+        4. Disclaimer of Warranty.
+        
+          THE WORK IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+          EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF
+          MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR
+          NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER
+          THIS LICENSE. SOME STATES' CONSUMER LAWS DO NOT ALLOW EXCLUSION OF AN
+          IMPLIED WARRANTY, SO THIS DISCLAIMER MAY NOT APPLY TO YOU.
+        
+        5. Limitation of Liability.
+        
+          EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL
+          THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE
+          SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT,
+          INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR
+          RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING
+          BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS
+          OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER COMM ERCIAL DAMAGES
+          OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF
+          SUCH DAMAGES.
+        
+Requires-Python: <3.12,>=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+License-File: NOTICE.txt
+Dynamic: license-file
+
+# aws-glue-libs
+
+This repository supports python libraries for local development of glue pyspark batch jobs. Glue streaming is supported in the separate repository [aws-glue-streaming-libs](https://github.com/awslabs/aws-glue-streaming-libs).
+
+## Contents
+This repository contains:
+ * `awsglue` - the Python libary you can use to author [AWS Glue](https://aws.amazon.com/glue) ETL job. This library extends [Apache Spark](https://spark.apache.org/) with additional data types and operations for ETL workflows. It's an interface for Glue ETL library in Python.
+ * `bin` - this directory hosts several executables that allow you to run the Python library locally or open up a PySpark shell to run Glue Spark code interactively.
+
+## Python versions by Glue Version
+
+Different Glue versions support different Python versions. The following table below is for your reference, which also includes the associated repository's branch for each glue version.
+
+| Glue Version  | Python 3 Version  | aws-glue-libs branch |
+|---|---|----------------------|
+| 2.0  | 3.7  | glue-2.0             |
+| 3.0  | 3.7  | glue-3.0             |
+| 4.0  | 3.10 | glue-4.0             |
+| 5.0  | 3.11 | glue-5.0             |
+
+You may refer to AWS Glue's official [release notes](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) for more information
+
+## Setup guide
+
+If you haven't already, please refer to the [official AWS Glue Python local development documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python) for the official setup documentation. The following is a summary of the AWS documentation:
+
+The `awsglue` library provides only the Python interface to the Glue Spark runtime, you need the Glue ETL jar to run it locally. The jar is now available via the maven build system in a s3 backed maven repository. Here are the steps to set up your dev environment locally.
+
+1. install Apache Maven from the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
+1. use `copy-dependencies` target in Apache Maven to download the jar from S3 to your local dev environment.
+1. download and extract the Apache Spark distribution based on the Glue version you're using:
+   * Glue version 2.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-2.0/spark-2.4.3-bin-hadoop2.8.tgz1`
+   * Glue version 3.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-3.0/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz`
+   * Glue version 4.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-4.0/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0.tgz` 
+   * Glue version 5.0: download the Apache Spark 3.5.4 distribution from `https://spark.apache.org`
+1. export the `SPARK_HOME` environmental variable to the extracted location of the above Spark distribution. For example:
+    ```
+    Glue version 2.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8
+    Glue version 3.0: export SPARK_HOME=/home/$USER/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3
+    Glue version 4.0: export SPARK_HOME=/home/$USER/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0
+    Glue version 5.0: export SPARK_HOME=/home/$USER/spark-3.5.4-bin-hadoop3
+    ```
+1. now you can run the executables in the `bin` directory to start a Glue Shell or submit a Glue Spark application.
+    ```
+    Glue shell: ./bin/gluepyspark
+    Glue submit: ./bin/gluesparksubmit
+    pytest: ./bin/gluepytest
+    ```
+(The `gluepytest` script assumes that the pytest module is installed and available in the `PATH` env variable)
+
+## Licensing
+
+The libraries in this repository licensed under the [Amazon Software License](http://aws.amazon.com/asl/) (the "License"). They may not be used except in compliance with the License, a copy of which is included here in the LICENSE file.
+
+---
+
+# Release Notes
+
+## July 26 2023
+* According to [AWS Glue version support policy](https://docs.aws.amazon.com/glue/latest/dg/glue-version-support-policy.html), branches for Glue 0.9 and 1.0 are removed as they are already deprecated.
+
+
+## August 27 2021
+* The master branch has been modified from representing Glue 0.9 to Glue 3.0, we have also created a glue-0.9 branch to reflect the former state of the master branch with Glue 0.9. To rename your local clone of the older master branch and point to the glue-0.9 branch, you may use the following commands:
+```
+git branch -m master glue-0.9
+git fetch origin
+git branch -u origin/glue-0.9 glue-0.9
+git remote set-head origin -a
+```
+

awsgluedataplanepython-5.0.0/README.md

@@ -0,0 +1,71 @@
+# aws-glue-libs
+
+This repository supports python libraries for local development of glue pyspark batch jobs. Glue streaming is supported in the separate repository [aws-glue-streaming-libs](https://github.com/awslabs/aws-glue-streaming-libs).
+
+## Contents
+This repository contains:
+ * `awsglue` - the Python libary you can use to author [AWS Glue](https://aws.amazon.com/glue) ETL job. This library extends [Apache Spark](https://spark.apache.org/) with additional data types and operations for ETL workflows. It's an interface for Glue ETL library in Python.
+ * `bin` - this directory hosts several executables that allow you to run the Python library locally or open up a PySpark shell to run Glue Spark code interactively.
+
+## Python versions by Glue Version
+
+Different Glue versions support different Python versions. The following table below is for your reference, which also includes the associated repository's branch for each glue version.
+
+| Glue Version  | Python 3 Version  | aws-glue-libs branch |
+|---|---|----------------------|
+| 2.0  | 3.7  | glue-2.0             |
+| 3.0  | 3.7  | glue-3.0             |
+| 4.0  | 3.10 | glue-4.0             |
+| 5.0  | 3.11 | glue-5.0             |
+
+You may refer to AWS Glue's official [release notes](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) for more information
+
+## Setup guide
+
+If you haven't already, please refer to the [official AWS Glue Python local development documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python) for the official setup documentation. The following is a summary of the AWS documentation:
+
+The `awsglue` library provides only the Python interface to the Glue Spark runtime, you need the Glue ETL jar to run it locally. The jar is now available via the maven build system in a s3 backed maven repository. Here are the steps to set up your dev environment locally.
+
+1. install Apache Maven from the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
+1. use `copy-dependencies` target in Apache Maven to download the jar from S3 to your local dev environment.
+1. download and extract the Apache Spark distribution based on the Glue version you're using:
+   * Glue version 2.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-2.0/spark-2.4.3-bin-hadoop2.8.tgz1`
+   * Glue version 3.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-3.0/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz`
+   * Glue version 4.0: `https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-4.0/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0.tgz` 
+   * Glue version 5.0: download the Apache Spark 3.5.4 distribution from `https://spark.apache.org`
+1. export the `SPARK_HOME` environmental variable to the extracted location of the above Spark distribution. For example:
+    ```
+    Glue version 2.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8
+    Glue version 3.0: export SPARK_HOME=/home/$USER/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3
+    Glue version 4.0: export SPARK_HOME=/home/$USER/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0
+    Glue version 5.0: export SPARK_HOME=/home/$USER/spark-3.5.4-bin-hadoop3
+    ```
+1. now you can run the executables in the `bin` directory to start a Glue Shell or submit a Glue Spark application.
+    ```
+    Glue shell: ./bin/gluepyspark
+    Glue submit: ./bin/gluesparksubmit
+    pytest: ./bin/gluepytest
+    ```
+(The `gluepytest` script assumes that the pytest module is installed and available in the `PATH` env variable)
+
+## Licensing
+
+The libraries in this repository licensed under the [Amazon Software License](http://aws.amazon.com/asl/) (the "License"). They may not be used except in compliance with the License, a copy of which is included here in the LICENSE file.
+
+---
+
+# Release Notes
+
+## July 26 2023
+* According to [AWS Glue version support policy](https://docs.aws.amazon.com/glue/latest/dg/glue-version-support-policy.html), branches for Glue 0.9 and 1.0 are removed as they are already deprecated.
+
+
+## August 27 2021
+* The master branch has been modified from representing Glue 0.9 to Glue 3.0, we have also created a glue-0.9 branch to reflect the former state of the master branch with Glue 0.9. To rename your local clone of the older master branch and point to the glue-0.9 branch, you may use the following commands:
+```
+git branch -m master glue-0.9
+git fetch origin
+git branch -u origin/glue-0.9 glue-0.9
+git remote set-head origin -a
+```
+