koheesio 0.7.0__tar.gz

koheesio-0.7.0/.github/CODEOWNERS

@@ -0,0 +1 @@
+* @Nike-Inc/koheesio-dev

koheesio-0.7.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,40 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG] Please add your bug title here"
+labels: bug
+assignees: ''
+
+---
+
+<!-- Please provide as much detail as possible to help us understand and reproduce the issue. This will enable us to address it more effectively. -->
+
+## Describe the bug
+<!-- A clear and concise description of what the bug is. -->
+
+## Steps to Reproduce
+<!-- Steps to reproduce the behavior. For example:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+-->
+
+## Expected behavior
+<!-- A clear and concise description of what you expected to happen. -->
+
+## Screenshots
+<!-- If applicable, add screenshots to help explain your problem. -->
+
+## Environment
+<!-- Please provide information about your environment. For example:
+- OS: [e.g. iOS]
+- Browser [e.g. chrome, safari]
+- Version [e.g. 22]
+-->
+
+## Additional context
+<!-- Add any other context about the problem here. -->
+
+## Applicable Stack Trace
+<!-- If applicable, add the full stack trace or the clearly applicable sections of the stack trace here. -->

koheesio-0.7.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,28 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[FEATURE] Please add your feature request title"
+labels: enhancement
+assignees: ''
+
+---
+
+<!-- We follow Design thinking principles to bring the new feature request to life. Please read through [Design thinking](https://www.interaction-design.org/literature/article/5-stages-in-the-design-thinking-process) principles if you are not familiar. -->
+
+<!-- This is the [Board](https://github.com/orgs/Nike-Inc/projects/4) your feature request would go through, so keep in mind that there would be more back and forth on this. If you are very clear with all phases, please describe them here for faster development. -->
+
+## Is your feature request related to a problem? Please describe.
+<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
+...
+
+## Describe the solution you'd like
+<!-- A clear and concise description of what you want to happen. -->
+...
+
+## Describe alternatives you've considered
+<!-- A clear and concise description of any alternative solutions or features you've considered. -->
+...
+
+## Additional context
+<!-- Add any other context or screenshots about the feature request here. -->
+...

koheesio-0.7.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,36 @@
+<!--- Provide a general summary of your changes in the Title above -->
+
+## Description
+<!--- Describe your changes in detail -->
+
+## Related Issue
+<!--- This project only accepts pull requests related to open issues -->
+<!--- If suggesting a new feature or change, please discuss it in an issue first -->
+<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
+<!--- Please link to the issue here: -->
+
+## Motivation and Context
+<!--- Why is this change required? What problem does it solve? -->
+
+## How Has This Been Tested?
+<!--- Please describe in detail how you tested your changes. -->
+<!--- Include details of your testing environment, and the tests you ran to -->
+<!--- see how your change affects other areas of the code, etc. -->
+
+## Screenshots (if appropriate):
+
+## Types of changes
+<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
+- [ ] Bug fix (non-breaking change which fixes an issue)
+- [ ] New feature (non-breaking change which adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to change)
+
+## Checklist:
+<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
+<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
+- [ ] My code follows the code style of this project.
+- [ ] My change requires a change to the documentation.
+- [ ] I have updated the documentation accordingly.
+- [ ] I have read the **CONTRIBUTING** document.
+- [ ] I have added tests to cover my changes.
+- [ ] All new and existing tests passed.

koheesio-0.7.0/.github/workflows/release.yml

@@ -0,0 +1,90 @@
+name: release 
+
+on:
+  push:
+    tags:
+    - koheesio-v*
+    branches:
+    - main
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
+defaults:
+  run:
+    shell: bash
+
+env:
+  APP_NAME: koheesio
+  PYTHON_VERSION: "3.12"
+  PYOXIDIZER_VERSION: "0.24.0"
+  DIST_URL: "https://github.com/Nike-Inc/koheesio/releases"
+
+jobs:
+  python-artifacts:
+    name: Build wheel and source distribution
+    runs-on: ubuntu-latest
+
+    outputs:
+      old-version: ${{ steps.version.outputs.old-version }}
+      version: ${{ steps.version.outputs.version }}
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+      with:
+        fetch-depth: 0
+
+    - name: Set up Python ${{ env.PYTHON_VERSION }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ env.PYTHON_VERSION }}
+
+    - name: Install Hatch
+      uses: pypa/hatch@install
+
+    # Windows installers don't accept non-integer versions so we ubiquitously
+    # perform the following transformation: X.Y.Z.devN -> X.Y.Z.N
+    - name: Set project version
+      id: version
+      run: |-
+        old_version="$(hatch version)"
+        version="${old_version/dev/}"
+
+        echo "old-version=$old_version" >> $GITHUB_OUTPUT
+        echo "version=$version" >> $GITHUB_OUTPUT
+        echo "$version"
+        
+    - name: Build
+      run: hatch build
+
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v4
+      with:
+        name: python-artifacts
+        path: dist/*
+        if-no-files-found: error
+  
+  publish-pypi:
+    name: Publish to PyPI
+    needs: python-artifacts
+    if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
+    runs-on: ubuntu-latest
+    # Specifying a GitHub environment is optional, but strongly encouraged
+    environment: release
+    permissions:
+      # IMPORTANT: this permission is mandatory for trusted publishing
+      id-token: write
+
+    steps:
+      - name: Download Python artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: python-artifacts
+          path: dist
+
+      - name: Publish package to PyPI
+        uses: pypa/gh-action-pypi-publish@v1.8.14
+        with:
+          print-hash: true

koheesio-0.7.0/.github/workflows/test.yaml

@@ -0,0 +1,71 @@
+name: test
+
+on:
+  pull_request:
+    branches:
+    - main
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
+env:
+  STABLE_PYTHON_VERSION: '3.11'
+  PYTHONUNBUFFERED: "1"
+  FORCE_COLOR: "1"
+
+jobs:
+  tests:
+    name: Python ${{ matrix.python-version }} with PySpark ${{ matrix.pyspark-version }} on ${{ startsWith(matrix.os, 'macos-') && 'macOS' || startsWith(matrix.os, 'windows-') && 'Windows' || 'Linux' }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+#        os: [ubuntu-latest, windows-latest, macos-latest]  # FIXME: Add Windows and macOS
+        os: [ubuntu-latest]
+        python-version: ['3.9', '3.10', '3.11', '3.12']
+        pyspark-version: ['33', '34', '35']
+        exclude:
+          - python-version: '3.9'
+            pyspark-version: '35'
+          - python-version: '3.11'
+            pyspark-version: '33'
+          - python-version: '3.11'
+            pyspark-version: '34'
+          - python-version: '3.12'
+            pyspark-version: '33'
+          - python-version: '3.12'
+            pyspark-version: '34'
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install Hatch
+      uses: pypa/hatch@install
+
+#    - name: Check Code Quality
+#      run:  |-
+#        hatch fmt --check --python=${{ matrix.python-version }}
+
+    - name: Run tests
+      run: hatch test --python=${{ matrix.python-version }} -i version=pyspark${{ matrix.pyspark-version }}
+
+  # https://github.com/marketplace/actions/alls-green#why
+  final_check: # This job does nothing and is only used for the branch protection
+    if: always()
+
+    needs:
+    - tests 
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Decide whether the needed jobs succeeded or failed
+      uses: re-actors/alls-green@release/v1
+      with:
+        jobs: ${{ toJSON(needs) }}

koheesio-0.7.0/.gitignore

@@ -0,0 +1,141 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+schema*.json
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+.htmlcov/
+XML_FOLDER
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# sdkman
+.sdkmanrc
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# ruff
+.ruff_cache/
+
+# mypy
+.mypy_cache/
+.DS_Store
+
+# IDE
+.idea/
+metastore_db/
+spark-warehouse
+.run
+.obsidian
+
+# Pre-commit
+.pre-commit-config.yaml
+
+out/**
+*.iml
+
+/tests/integration/**/task_definition/*.yaml
+/.vscode/settings.json
+/.vscode/launch.json
+
+.databricks
+
+# PYTEST
+/tests/_data/spark-warehouse
+/tests/_data/checkpoint
+/tests/_data/snowflake_staging.parq/
+
+# DevContainer
+.devcontainer

koheesio-0.7.0/CONTRIBUTING.md

@@ -0,0 +1,103 @@
+# How to contribute
+
+There are a few guidelines that we need contributors to follow so that we are able to process requests as efficiently as possible. 
+
+<!-- uncomment the below once Open Source is ready -->
+[//]: # (If you have any questions or concerns please feel free to contact us at [opensource@nike.com]&#40;mailto:opensource@nike.com&#41;.)
+
+[//]: # ()
+[//]: # (## Getting Started)
+
+[//]: # ()
+[//]: # (* Review our [Code of Conduct]&#40;https://github.com/Nike-Inc/nike-inc.github.io/blob/master/CONDUCT.md&#41;)
+
+[//]: # (* Submit the [Individual Contributor License Agreement]&#40;https://www.clahub.com/agreements/Nike-Inc/fastbreak&#41;)
+
+[//]: # (* Make sure you have a [GitHub account]&#40;https://github.com/signup/free&#41;)
+
+[//]: # (* Submit a ticket for your issue, assuming one does not already exist.)
+
+[//]: # (    * Clearly describe the issue including steps to reproduce when it is a bug.)
+
+[//]: # (    * Make sure you fill in the earliest version that you know has the issue.)
+
+[//]: # (* Fork the repository on GitHub)
+
+## Making Changes
+
+* Create a feature branch off of `main` before you start your work.
+    * Please avoid working directly on the `main` branch.
+* Setup the required package manager [poetry](#-package-manager)
+* Setup the dev environment [see below](#-dev-environment-setup)
+* Make commits of logical units.
+    * You may be asked to squash unnecessary commits down to logical units.
+* Check for unnecessary whitespace with `git diff --check` before committing.
+* Write meaningful, descriptive commit messages.
+* Please follow existing code conventions when working on a file
+* Make sure to check the standards on the code [see below](#-linting-and-standards)
+* Make sure to test the code before you push changes [see below](#-testing)
+
+## 🤝 Submitting Changes
+
+* Push your changes to a topic branch in your fork of the repository.
+* Submit a pull request to the repository in the Nike-Inc organization.
+* After feedback has been given we expect responses within two weeks. After two weeks we may close the pull request 
+if it isn't showing any activity.
+* Bug fixes or features that lack appropriate tests may not be considered for merge.
+* Changes that lower test coverage may not be considered for merge.
+
+### 📦 Package manager
+
+We use `make` for managing different steps of setup and maintenance in the project. You can install make by following
+the instructions [here](https://formulae.brew.sh/formula/make)
+
+We use `poetry` as our package manager.
+
+Please DO NOT use pip or conda to install the dependencies. Instead, use poetry:
+
+```bash
+make poetry-install
+```
+
+### 📌 Dev Environment Setup
+
+To ensure our standards, make sure to install the required packages.
+
+```bash
+make dev
+```
+
+### 🧹 Linting and Standards
+
+We use `pylint`, and `black` to maintain standards in the codebase  
+<small>(`mypy` to be added later)</small>
+
+```bash
+make check
+```
+
+Make sure that the linter does not report any errors or warnings before submitting a pull request.
+
+### 🧪 Testing
+
+We use `pytest` to test our code. You can run the tests by running the following command:
+
+```bash
+make test
+```
+
+Make sure that all tests pass before submitting a pull request.
+
+## 🚀 Release Process
+
+At the moment, the release process is manual. We try to make frequent releases. Usually, we release a new version when we have a new feature or bugfix. A developer with admin rights to the repository will create a new release on GitHub, and then publish the new version to PyPI.
+
+# Additional Resources
+
+* [General GitHub documentation](https://help.github.com/)
+* [GitHub pull request documentation](https://help.github.com/send-pull-requests/)
+* [Nike's Code of Conduct](https://github.com/Nike-Inc/nike-inc.github.io/blob/master/CONDUCT.md)
+
+[//]: # (* [Nike's Individual Contributor License Agreement]&#40;https://www.clahub.com/agreements/Nike-Inc/fastbreak&#41;)
+
+[//]: # (* [Nike OSS]&#40;https://nike-inc.github.io/&#41;)